2024-09-13T16:12:54.854 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-13T16:12:54.875 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T16:12:54.961 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804 branch: wip-rishabh-testing-20240910.095512-reef description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} email: null first_in_suite: false job_id: '7903804' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: wip-rishabh-testing-20240910.095512-reef ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon warn on pool no app: false osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - online, but wants - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected - object missing on disk - Invalid tag char sha1: 50bbe7f395997dcd871813949c49addbbf630361 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 50bbe7f395997dcd871813949c49addbbf630361 subvolume_version: 2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-rishabh-testing-20240910.095512-reef sha1: 50bbe7f395997dcd871813949c49addbbf630361 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 1152 sha1: 50bbe7f395997dcd871813949c49addbbf630361 sleep_before_teardown: 0 subset: 984/1024 suite: fs suite_branch: wip-rishabh-testing-20240910.095512-reef suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 50bbe7f395997dcd871813949c49addbbf630361 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi119.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-13_14:36:41 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-13T16:12:54.961 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa; will attempt to use it 2024-09-13T16:12:54.962 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa/tasks 2024-09-13T16:12:54.962 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-13T16:12:54.963 INFO:teuthology.task.internal:Checking packages... 2024-09-13T16:12:54.987 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '50bbe7f395997dcd871813949c49addbbf630361' 2024-09-13T16:12:54.988 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-13T16:12:54.988 INFO:teuthology.packaging:ref: None 2024-09-13T16:12:54.988 INFO:teuthology.packaging:tag: None 2024-09-13T16:12:54.988 INFO:teuthology.packaging:branch: wip-rishabh-testing-20240910.095512-reef 2024-09-13T16:12:54.988 INFO:teuthology.packaging:sha1: 50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:12:54.988 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=wip-rishabh-testing-20240910.095512-reef 2024-09-13T16:12:55.145 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:12:55.146 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-13T16:12:55.153 INFO:teuthology.task.internal:no buildpackages task found 2024-09-13T16:12:55.153 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-13T16:12:55.161 INFO:teuthology.task.internal:Saving configuration 2024-09-13T16:12:55.178 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-13T16:12:55.185 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-13T16:12:55.217 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 16:06:59.374808', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-13T16:12:55.252 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-13 16:06:59.376803', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGETOy3xONfbx7kUFCNiUtRLNfkcrcEsWzPdq6Np/6Bw'} 2024-09-13T16:12:55.252 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-13T16:12:55.259 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-09-13T16:12:55.259 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-09-13T16:12:55.259 INFO:teuthology.run_tasks:Running task console_log... 2024-09-13T16:12:55.339 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f59f24cbbe0>, signals=[15]) 2024-09-13T16:12:55.339 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-13T16:12:55.346 INFO:teuthology.task.internal:Opening connections... 2024-09-13T16:12:55.346 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:12:55.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:12:55.449 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:12:55.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:12:55.557 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-13T16:12:55.563 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2024-09-13T16:12:55.570 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2024-09-13T16:12:55.571 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2024-09-13T16:12:55.619 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Ubuntu" 2024-09-13T16:12:55.619 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-13T16:12:55.619 INFO:teuthology.orchestra.run.smithi119.stdout:ID=ubuntu 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE=debian 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="20.04" 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_CODENAME=focal 2024-09-13T16:12:55.620 INFO:teuthology.orchestra.run.smithi119.stdout:UBUNTU_CODENAME=focal 2024-09-13T16:12:55.621 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2024-09-13T16:12:55.652 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-09-13T16:12:55.658 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-09-13T16:12:55.659 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-09-13T16:12:55.707 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="Ubuntu" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:ID=ubuntu 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE=debian 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="20.04" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-13T16:12:55.708 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_CODENAME=focal 2024-09-13T16:12:55.709 INFO:teuthology.orchestra.run.smithi145.stdout:UBUNTU_CODENAME=focal 2024-09-13T16:12:55.709 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-09-13T16:12:55.743 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-13T16:12:55.752 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-13T16:12:55.807 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-13T16:12:55.808 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2024-09-13T16:12:55.810 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-09-13T16:12:55.816 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-13T16:12:55.822 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-13T16:12:55.822 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2024-09-13T16:12:55.826 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-09-13T16:12:55.845 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-13T16:12:55.920 INFO:teuthology.run_tasks:Running task kernel... 2024-09-13T16:12:55.935 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-13T16:12:55.935 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-13T16:12:55.936 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T16:12:55.936 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-13T16:12:55.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:12:55.942 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-13T16:12:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:12:55.943 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-09-13T16:12:55.949 INFO:teuthology.orchestra.run.smithi119.stdout:5.4.0-124-generic 2024-09-13T16:12:55.949 INFO:teuthology.task.kernel:Running kernel on smithi119: 5.4.0-124-generic 2024-09-13T16:12:55.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get clean 2024-09-13T16:12:55.950 INFO:teuthology.orchestra.run.smithi145.stdout:5.4.0-124-generic 2024-09-13T16:12:55.950 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.4.0-124-generic 2024-09-13T16:12:55.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get clean 2024-09-13T16:12:56.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2024-09-13T16:12:56.106 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get update 2024-09-13T16:12:56.182 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-13T16:12:56.299 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T16:12:56.381 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-13T16:12:56.445 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T16:12:56.452 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [128 kB] 2024-09-13T16:12:56.481 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-13T16:12:56.514 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-13T16:12:56.552 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [128 kB] 2024-09-13T16:12:56.564 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-13T16:12:56.606 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-13T16:12:56.610 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-13T16:12:56.610 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-13T16:12:56.611 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-13T16:12:56.631 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-13T16:12:56.634 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-13T16:12:56.634 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-13T16:12:56.638 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-13T16:12:56.644 INFO:teuthology.orchestra.run.smithi119.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-13T16:12:56.646 INFO:teuthology.orchestra.run.smithi119.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-13T16:12:56.646 INFO:teuthology.orchestra.run.smithi119.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-13T16:12:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-13T16:12:56.647 INFO:teuthology.orchestra.run.smithi119.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-13T16:12:56.658 INFO:teuthology.orchestra.run.smithi119.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-13T16:12:56.735 INFO:teuthology.orchestra.run.smithi119.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-backports InRelease [128 kB] 2024-09-13T16:12:56.851 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-13T16:12:56.876 INFO:teuthology.orchestra.run.smithi119.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-13T16:12:56.933 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [3,563 kB] 2024-09-13T16:12:56.973 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-13T16:12:56.977 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-13T16:12:56.978 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-13T16:12:56.978 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-13T16:12:57.001 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-13T16:12:57.006 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-13T16:12:57.006 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-13T16:12:57.015 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-13T16:12:57.020 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-13T16:12:57.022 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-13T16:12:57.023 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-13T16:12:57.023 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-13T16:12:57.023 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-13T16:12:57.025 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-13T16:12:57.025 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-13T16:12:57.025 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-13T16:12:57.042 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-13T16:12:57.042 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-13T16:12:57.042 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-13T16:12:57.043 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-13T16:12:57.172 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [3,195 kB] 2024-09-13T16:12:57.260 INFO:teuthology.orchestra.run.smithi119.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [1,030 kB] 2024-09-13T16:12:57.288 INFO:teuthology.orchestra.run.smithi119.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [550 kB] 2024-09-13T16:12:57.302 INFO:teuthology.orchestra.run.smithi119.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 c-n-f Metadata [17.7 kB] 2024-09-13T16:12:57.302 INFO:teuthology.orchestra.run.smithi119.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [3,236 kB] 2024-09-13T16:12:57.396 INFO:teuthology.orchestra.run.smithi119.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/restricted i386 Packages [39.6 kB] 2024-09-13T16:12:57.397 INFO:teuthology.orchestra.run.smithi119.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [453 kB] 2024-09-13T16:12:57.413 INFO:teuthology.orchestra.run.smithi119.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 c-n-f Metadata [548 B] 2024-09-13T16:12:57.414 INFO:teuthology.orchestra.run.smithi119.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [807 kB] 2024-09-13T16:12:57.438 INFO:teuthology.orchestra.run.smithi119.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [1,230 kB] 2024-09-13T16:12:57.470 INFO:teuthology.orchestra.run.smithi119.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [295 kB] 2024-09-13T16:12:57.478 INFO:teuthology.orchestra.run.smithi119.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [28.0 kB] 2024-09-13T16:12:57.478 INFO:teuthology.orchestra.run.smithi119.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 Packages [27.1 kB] 2024-09-13T16:12:57.479 INFO:teuthology.orchestra.run.smithi119.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/multiverse i386 Packages [8,440 B] 2024-09-13T16:12:57.509 INFO:teuthology.orchestra.run.smithi119.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/multiverse Translation-en [7,936 B] 2024-09-13T16:12:57.509 INFO:teuthology.orchestra.run.smithi119.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/multiverse amd64 c-n-f Metadata [616 B] 2024-09-13T16:12:57.510 INFO:teuthology.orchestra.run.smithi119.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 Packages [45.7 kB] 2024-09-13T16:12:57.510 INFO:teuthology.orchestra.run.smithi119.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-backports/main i386 Packages [36.1 kB] 2024-09-13T16:12:57.594 INFO:teuthology.orchestra.run.smithi119.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-backports/main amd64 c-n-f Metadata [1,420 B] 2024-09-13T16:12:57.595 INFO:teuthology.orchestra.run.smithi119.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-backports/universe i386 Packages [13.8 kB] 2024-09-13T16:12:57.595 INFO:teuthology.orchestra.run.smithi119.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 Packages [25.0 kB] 2024-09-13T16:12:57.596 INFO:teuthology.orchestra.run.smithi119.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal-backports/universe amd64 c-n-f Metadata [880 B] 2024-09-13T16:12:57.658 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [812 kB] 2024-09-13T16:12:57.672 INFO:teuthology.orchestra.run.smithi145.stdout:Get:29 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [472 kB] 2024-09-13T16:12:57.680 INFO:teuthology.orchestra.run.smithi145.stdout:Get:30 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [14.3 kB] 2024-09-13T16:12:57.680 INFO:teuthology.orchestra.run.smithi145.stdout:Get:31 http://security.ubuntu.com/ubuntu focal-security/restricted i386 Packages [38.3 kB] 2024-09-13T16:12:57.681 INFO:teuthology.orchestra.run.smithi145.stdout:Get:32 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 Packages [3,117 kB] 2024-09-13T16:12:57.770 INFO:teuthology.orchestra.run.smithi145.stdout:Get:33 http://security.ubuntu.com/ubuntu focal-security/restricted Translation-en [436 kB] 2024-09-13T16:12:57.776 INFO:teuthology.orchestra.run.smithi145.stdout:Get:34 http://security.ubuntu.com/ubuntu focal-security/restricted amd64 c-n-f Metadata [548 B] 2024-09-13T16:12:57.777 INFO:teuthology.orchestra.run.smithi145.stdout:Get:35 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [681 kB] 2024-09-13T16:12:57.821 INFO:teuthology.orchestra.run.smithi145.stdout:Get:36 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [1,011 kB] 2024-09-13T16:12:57.839 INFO:teuthology.orchestra.run.smithi145.stdout:Get:37 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [213 kB] 2024-09-13T16:12:57.844 INFO:teuthology.orchestra.run.smithi145.stdout:Get:38 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [21.2 kB] 2024-09-13T16:12:57.845 INFO:teuthology.orchestra.run.smithi145.stdout:Get:39 http://security.ubuntu.com/ubuntu focal-security/multiverse i386 Packages [7,204 B] 2024-09-13T16:12:57.845 INFO:teuthology.orchestra.run.smithi145.stdout:Get:40 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 Packages [24.8 kB] 2024-09-13T16:12:57.845 INFO:teuthology.orchestra.run.smithi145.stdout:Get:41 http://security.ubuntu.com/ubuntu focal-security/multiverse Translation-en [5,968 B] 2024-09-13T16:12:57.919 INFO:teuthology.orchestra.run.smithi145.stdout:Get:42 http://security.ubuntu.com/ubuntu focal-security/multiverse amd64 c-n-f Metadata [540 B] 2024-09-13T16:13:01.006 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 21.8 MB in 3s (7,549 kB/s) 2024-09-13T16:13:01.303 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 21.8 MB in 3s (6,714 kB/s) 2024-09-13T16:13:02.556 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2024-09-13T16:13:02.584 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T16:13:02.676 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2024-09-13T16:13:02.862 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-09-13T16:13:02.891 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T16:13:02.896 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2024-09-13T16:13:02.896 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2024-09-13T16:13:02.974 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-09-13T16:13:03.110 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2024-09-13T16:13:03.110 INFO:teuthology.orchestra.run.smithi119.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-13T16:13:03.111 INFO:teuthology.orchestra.run.smithi119.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-13T16:13:03.111 INFO:teuthology.orchestra.run.smithi119.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-13T16:13:03.112 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2024-09-13T16:13:03.112 INFO:teuthology.orchestra.run.smithi119.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-13T16:13:03.112 INFO:teuthology.orchestra.run.smithi119.stdout:Recommended packages: 2024-09-13T16:13:03.112 INFO:teuthology.orchestra.run.smithi119.stdout: thermald 2024-09-13T16:13:03.168 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2024-09-13T16:13:03.168 INFO:teuthology.orchestra.run.smithi119.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-13T16:13:03.168 INFO:teuthology.orchestra.run.smithi119.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-13T16:13:03.169 INFO:teuthology.orchestra.run.smithi119.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-13T16:13:03.170 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages will be upgraded: 2024-09-13T16:13:03.170 INFO:teuthology.orchestra.run.smithi119.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T16:13:03.191 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-09-13T16:13:03.192 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-09-13T16:13:03.223 INFO:teuthology.orchestra.run.smithi119.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-13T16:13:03.223 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 77.2 MB of archives. 2024-09-13T16:13:03.223 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-13T16:13:03.224 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-13T16:13:03.389 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2024-09-13T16:13:03.389 INFO:teuthology.orchestra.run.smithi145.stdout: linux-generic linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-13T16:13:03.389 INFO:teuthology.orchestra.run.smithi145.stdout: linux-headers-generic linux-image-5.4.0-195-generic 2024-09-13T16:13:03.390 INFO:teuthology.orchestra.run.smithi145.stdout: linux-modules-5.4.0-195-generic linux-modules-extra-5.4.0-195-generic 2024-09-13T16:13:03.391 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2024-09-13T16:13:03.391 INFO:teuthology.orchestra.run.smithi145.stdout: fdutils linux-doc | linux-source-5.4.0 linux-tools 2024-09-13T16:13:03.391 INFO:teuthology.orchestra.run.smithi145.stdout:Recommended packages: 2024-09-13T16:13:03.391 INFO:teuthology.orchestra.run.smithi145.stdout: thermald 2024-09-13T16:13:03.448 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2024-09-13T16:13:03.448 INFO:teuthology.orchestra.run.smithi145.stdout: linux-headers-5.4.0-195 linux-headers-5.4.0-195-generic 2024-09-13T16:13:03.448 INFO:teuthology.orchestra.run.smithi145.stdout: linux-image-5.4.0-195-generic linux-modules-5.4.0-195-generic 2024-09-13T16:13:03.448 INFO:teuthology.orchestra.run.smithi145.stdout: linux-modules-extra-5.4.0-195-generic 2024-09-13T16:13:03.450 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages will be upgraded: 2024-09-13T16:13:03.450 INFO:teuthology.orchestra.run.smithi145.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-13T16:13:03.505 INFO:teuthology.orchestra.run.smithi145.stdout:3 upgraded, 5 newly installed, 0 to remove and 316 not upgraded. 2024-09-13T16:13:03.505 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 77.2 MB of archives. 2024-09-13T16:13:03.505 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 381 MB of additional disk space will be used. 2024-09-13T16:13:03.505 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-5.4.0-195-generic amd64 5.4.0-195.215 [15.0 MB] 2024-09-13T16:13:03.574 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-13T16:13:03.671 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-13T16:13:03.808 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-5.4.0-195-generic amd64 5.4.0-195.215 [10.5 MB] 2024-09-13T16:13:03.914 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-modules-extra-5.4.0-195-generic amd64 5.4.0-195.215 [39.2 MB] 2024-09-13T16:13:04.175 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-13T16:13:04.176 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-13T16:13:04.176 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-13T16:13:04.269 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-13T16:13:04.280 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-13T16:13:04.434 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-generic amd64 5.4.0.195.193 [1,900 B] 2024-09-13T16:13:04.492 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-image-generic amd64 5.4.0.195.193 [2,584 B] 2024-09-13T16:13:04.492 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195 all 5.4.0-195.215 [11.0 MB] 2024-09-13T16:13:04.595 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-5.4.0-195-generic amd64 5.4.0-195.215 [1,386 kB] 2024-09-13T16:13:04.607 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-headers-generic amd64 5.4.0.195.193 [2,460 B] 2024-09-13T16:13:05.000 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 77.2 MB in 1s (70.0 MB/s) 2024-09-13T16:13:05.199 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-13T16:13:05.297 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 77.2 MB in 1s (67.1 MB/s) 2024-09-13T16:13:05.484 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-modules-5.4.0-195-generic. 2024-09-13T16:13:06.965 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-13T16:13:06.970 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:07.048 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:07.198 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 104373 files and directories currently installed.) 2024-09-13T16:13:07.203 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../0-linux-modules-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:07.282 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:08.971 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-13T16:13:08.986 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:09.088 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:09.099 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-image-5.4.0-195-generic. 2024-09-13T16:13:09.113 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../1-linux-image-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:09.219 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:10.191 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-13T16:13:10.206 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:10.249 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:10.327 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-modules-extra-5.4.0-195-generic. 2024-09-13T16:13:10.339 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../2-linux-modules-extra-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:10.377 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:15.028 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T16:13:15.154 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T16:13:15.183 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../3-linux-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T16:13:15.308 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T16:13:15.490 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T16:13:15.641 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T16:13:15.649 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../4-linux-image-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T16:13:15.770 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-image-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T16:13:15.969 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-13T16:13:15.985 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-13T16:13:16.027 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-13T16:13:16.115 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-headers-5.4.0-195. 2024-09-13T16:13:16.130 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../5-linux-headers-5.4.0-195_5.4.0-195.215_all.deb ... 2024-09-13T16:13:16.173 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-13T16:13:21.292 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-13T16:13:21.314 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:21.358 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:21.524 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package linux-headers-5.4.0-195-generic. 2024-09-13T16:13:21.546 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../6-linux-headers-5.4.0-195-generic_5.4.0-195.215_amd64.deb ... 2024-09-13T16:13:21.591 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:24.270 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T16:13:24.422 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T16:13:24.753 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:25.156 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../7-linux-headers-generic_5.4.0.195.193_amd64.deb ... 2024-09-13T16:13:25.259 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking linux-headers-generic (5.4.0.195.193) over (5.4.0.124.125) ... 2024-09-13T16:13:25.620 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-modules-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:25.961 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:26.808 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:27.118 INFO:teuthology.orchestra.run.smithi145.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-13T16:13:27.118 INFO:teuthology.orchestra.run.smithi145.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-13T16:13:27.280 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:27.983 INFO:teuthology.orchestra.run.smithi119.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.4.0-195-generic 2024-09-13T16:13:28.010 INFO:teuthology.orchestra.run.smithi119.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.4.0-195-generic 2024-09-13T16:13:28.119 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-modules-extra-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:29.373 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-13T16:13:29.508 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-13T16:13:29.659 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:29.785 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-13T16:13:29.911 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-13T16:13:29.920 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-headers-5.4.0-195 (5.4.0-195.215) ... 2024-09-13T16:13:30.046 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-image-generic (5.4.0.195.193) ... 2024-09-13T16:13:30.063 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:30.133 INFO:teuthology.orchestra.run.smithi145.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T16:13:30.133 INFO:teuthology.orchestra.run.smithi145.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-13T16:13:30.172 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-headers-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:30.299 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-headers-generic (5.4.0.195.193) ... 2024-09-13T16:13:30.425 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up linux-generic (5.4.0.195.193) ... 2024-09-13T16:13:30.551 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for linux-image-5.4.0-195-generic (5.4.0-195.215) ... 2024-09-13T16:13:30.619 INFO:teuthology.orchestra.run.smithi119.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-13T16:13:30.619 INFO:teuthology.orchestra.run.smithi119.stdout:update-initramfs: Generating /boot/initrd.img-5.4.0-195-generic 2024-09-13T16:13:52.521 INFO:teuthology.orchestra.run.smithi145.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T16:13:52.521 INFO:teuthology.orchestra.run.smithi145.stdout:Sourcing file `/etc/default/grub' 2024-09-13T16:13:52.528 INFO:teuthology.orchestra.run.smithi145.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T16:13:52.546 INFO:teuthology.orchestra.run.smithi145.stdout:Generating grub configuration file ... 2024-09-13T16:13:52.652 INFO:teuthology.orchestra.run.smithi119.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-13T16:13:52.652 INFO:teuthology.orchestra.run.smithi119.stdout:Sourcing file `/etc/default/grub' 2024-09-13T16:13:52.673 INFO:teuthology.orchestra.run.smithi119.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T16:13:52.693 INFO:teuthology.orchestra.run.smithi119.stdout:Generating grub configuration file ... 2024-09-13T16:13:52.850 INFO:teuthology.orchestra.run.smithi145.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-13T16:13:52.879 INFO:teuthology.orchestra.run.smithi145.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-13T16:13:53.015 INFO:teuthology.orchestra.run.smithi119.stdout:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-13T16:13:53.039 INFO:teuthology.orchestra.run.smithi119.stdout:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-13T16:13:53.254 INFO:teuthology.orchestra.run.smithi145.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-13T16:13:53.266 INFO:teuthology.orchestra.run.smithi145.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-13T16:13:53.391 INFO:teuthology.orchestra.run.smithi119.stdout:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-13T16:13:53.402 INFO:teuthology.orchestra.run.smithi119.stdout:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-13T16:13:55.431 INFO:teuthology.orchestra.run.smithi145.stdout:done 2024-09-13T16:13:55.632 INFO:teuthology.orchestra.run.smithi119.stdout:done 2024-09-13T16:13:55.885 DEBUG:teuthology.orchestra.run.smithi145:> dpkg -s linux-image-generic 2024-09-13T16:13:55.944 INFO:teuthology.orchestra.run.smithi145.stdout:Package: linux-image-generic 2024-09-13T16:13:55.945 INFO:teuthology.orchestra.run.smithi145.stdout:Status: install ok installed 2024-09-13T16:13:55.945 INFO:teuthology.orchestra.run.smithi145.stdout:Priority: optional 2024-09-13T16:13:55.945 INFO:teuthology.orchestra.run.smithi145.stdout:Section: kernel 2024-09-13T16:13:55.945 INFO:teuthology.orchestra.run.smithi145.stdout:Installed-Size: 20 2024-09-13T16:13:55.945 INFO:teuthology.orchestra.run.smithi145.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T16:13:55.945 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: amd64 2024-09-13T16:13:55.945 INFO:teuthology.orchestra.run.smithi145.stdout:Source: linux-meta 2024-09-13T16:13:55.946 INFO:teuthology.orchestra.run.smithi145.stdout:Version: 5.4.0.195.193 2024-09-13T16:13:55.946 INFO:teuthology.orchestra.run.smithi145.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-13T16:13:55.946 INFO:teuthology.orchestra.run.smithi145.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T16:13:55.946 INFO:teuthology.orchestra.run.smithi145.stdout:Recommends: thermald 2024-09-13T16:13:55.947 INFO:teuthology.orchestra.run.smithi145.stdout:Description: Generic Linux kernel image 2024-09-13T16:13:55.947 INFO:teuthology.orchestra.run.smithi145.stdout: This package will always depend on the latest generic kernel image 2024-09-13T16:13:55.947 INFO:teuthology.orchestra.run.smithi145.stdout: available. 2024-09-13T16:13:55.947 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-13T16:13:55.947 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T16:13:55.947 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T16:13:55.948 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-13T16:13:55.948 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T16:13:55.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get clean 2024-09-13T16:13:56.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get update 2024-09-13T16:13:56.106 DEBUG:teuthology.orchestra.run.smithi119:> dpkg -s linux-image-generic 2024-09-13T16:13:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Package: linux-image-generic 2024-09-13T16:13:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Status: install ok installed 2024-09-13T16:13:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Priority: optional 2024-09-13T16:13:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Section: kernel 2024-09-13T16:13:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Installed-Size: 20 2024-09-13T16:13:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T16:13:56.148 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: amd64 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout:Source: linux-meta 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout:Version: 5.4.0.195.193 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout:Recommends: thermald 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout:Description: Generic Linux kernel image 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout: This package will always depend on the latest generic kernel image 2024-09-13T16:13:56.149 INFO:teuthology.orchestra.run.smithi119.stdout: available. 2024-09-13T16:13:56.149 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.4.0-124-generic Expected: 5.4.0-195-generic 2024-09-13T16:13:56.149 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-13T16:13:56.149 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-13T16:13:56.150 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-13T16:13:56.150 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2024-09-13T16:13:56.150 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get clean 2024-09-13T16:13:56.155 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T16:13:56.155 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-13T16:13:56.163 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-13T16:13:56.180 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-13T16:13:56.264 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2024-09-13T16:13:56.392 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-13T16:13:56.392 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T16:13:56.401 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-13T16:13:56.418 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-13T16:13:58.006 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-09-13T16:13:58.033 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T16:13:58.103 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-09-13T16:13:58.243 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2024-09-13T16:13:58.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-13T16:13:58.322 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-09-13T16:13:58.323 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-09-13T16:13:58.335 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2024-09-13T16:13:58.553 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2024-09-13T16:13:58.554 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2024-09-13T16:13:58.554 INFO:teuthology.orchestra.run.smithi145.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-13T16:13:58.554 INFO:teuthology.orchestra.run.smithi145.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-13T16:13:58.556 DEBUG:teuthology.orchestra.run.smithi145:> dpkg -s linux-image-generic 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Package: linux-image-generic 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Status: install ok installed 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Priority: optional 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Section: kernel 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Installed-Size: 20 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: amd64 2024-09-13T16:13:58.575 INFO:teuthology.orchestra.run.smithi145.stdout:Source: linux-meta 2024-09-13T16:13:58.576 INFO:teuthology.orchestra.run.smithi145.stdout:Version: 5.4.0.195.193 2024-09-13T16:13:58.576 INFO:teuthology.orchestra.run.smithi145.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-13T16:13:58.576 INFO:teuthology.orchestra.run.smithi145.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T16:13:58.576 INFO:teuthology.orchestra.run.smithi145.stdout:Recommends: thermald 2024-09-13T16:13:58.576 INFO:teuthology.orchestra.run.smithi145.stdout:Description: Generic Linux kernel image 2024-09-13T16:13:58.576 INFO:teuthology.orchestra.run.smithi145.stdout: This package will always depend on the latest generic kernel image 2024-09-13T16:13:58.576 INFO:teuthology.orchestra.run.smithi145.stdout: available. 2024-09-13T16:13:58.577 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-09-13T16:13:58.582 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.NbKAjHSJQg 2024-09-13T16:13:58.583 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NbKAjHSJQg 2024-09-13T16:13:58.596 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.NbKAjHSJQg 2024-09-13T16:13:58.640 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.NbKAjHSJQg is 9KB 2024-09-13T16:13:58.655 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.NbKAjHSJQg 2024-09-13T16:13:58.661 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T16:13:58.717 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:13:58.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T16:13:58.717 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T16:13:58.746 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-13T16:13:58.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo update-grub 2024-09-13T16:13:58.793 INFO:teuthology.orchestra.run.smithi119.stdout:linux-image-generic is already the newest version (5.4.0.195.193). 2024-09-13T16:13:58.793 INFO:teuthology.orchestra.run.smithi119.stdout:0 upgraded, 0 newly installed, 0 to remove and 316 not upgraded. 2024-09-13T16:13:58.795 DEBUG:teuthology.orchestra.run.smithi119:> dpkg -s linux-image-generic 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Package: linux-image-generic 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Status: install ok installed 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Priority: optional 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Section: kernel 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Installed-Size: 20 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Maintainer: Ubuntu Kernel Team 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: amd64 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Source: linux-meta 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Version: 5.4.0.195.193 2024-09-13T16:13:58.817 INFO:teuthology.orchestra.run.smithi119.stdout:Provides: virtualbox-guest-modules (= 6.1.50-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.17) 2024-09-13T16:13:58.818 INFO:teuthology.orchestra.run.smithi119.stdout:Depends: linux-image-5.4.0-195-generic, linux-modules-extra-5.4.0-195-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-13T16:13:58.818 INFO:teuthology.orchestra.run.smithi119.stdout:Recommends: thermald 2024-09-13T16:13:58.818 INFO:teuthology.orchestra.run.smithi119.stdout:Description: Generic Linux kernel image 2024-09-13T16:13:58.818 INFO:teuthology.orchestra.run.smithi119.stdout: This package will always depend on the latest generic kernel image 2024-09-13T16:13:58.818 INFO:teuthology.orchestra.run.smithi119.stdout: available. 2024-09-13T16:13:58.818 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-13T16:13:58.863 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.qZliPdu4KU 2024-09-13T16:13:58.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub/grub.cfg /tmp/tmp.qZliPdu4KU 2024-09-13T16:13:58.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.qZliPdu4KU 2024-09-13T16:13:58.965 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.qZliPdu4KU is 9KB 2024-09-13T16:13:58.979 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.qZliPdu4KU 2024-09-13T16:13:58.985 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-13T16:13:59.041 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:13:59.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-13T16:13:59.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-13T16:13:59.078 INFO:teuthology.task.kernel:Distro Kernel Version: 5.4.0-195-generic 2024-09-13T16:13:59.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo update-grub 2024-09-13T16:14:00.131 INFO:teuthology.orchestra.run.smithi145.stderr:Sourcing file `/etc/default/grub' 2024-09-13T16:14:00.137 INFO:teuthology.orchestra.run.smithi145.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T16:14:00.149 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-09-13T16:14:00.395 INFO:teuthology.orchestra.run.smithi145.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-13T16:14:00.407 INFO:teuthology.orchestra.run.smithi145.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-13T16:14:00.416 INFO:teuthology.orchestra.run.smithi119.stderr:Sourcing file `/etc/default/grub' 2024-09-13T16:14:00.420 INFO:teuthology.orchestra.run.smithi119.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-13T16:14:00.432 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2024-09-13T16:14:00.688 INFO:teuthology.orchestra.run.smithi119.stderr:Found linux image: /boot/vmlinuz-5.4.0-195-generic 2024-09-13T16:14:00.700 INFO:teuthology.orchestra.run.smithi119.stderr:Found initrd image: /boot/initrd.img-5.4.0-195-generic 2024-09-13T16:14:00.790 INFO:teuthology.orchestra.run.smithi145.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-13T16:14:00.802 INFO:teuthology.orchestra.run.smithi145.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-13T16:14:01.055 INFO:teuthology.orchestra.run.smithi119.stderr:Found linux image: /boot/vmlinuz-5.4.0-124-generic 2024-09-13T16:14:01.067 INFO:teuthology.orchestra.run.smithi119.stderr:Found initrd image: /boot/initrd.img-5.4.0-124-generic 2024-09-13T16:14:01.838 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-09-13T16:14:01.840 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-09-13T16:14:02.112 INFO:teuthology.orchestra.run.smithi119.stderr:done 2024-09-13T16:14:02.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2024-09-13T16:14:31.871 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-13T16:14:31.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-13T16:14:31.873 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:14:32.117 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-13T16:14:32.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-13T16:14:32.118 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:15:06.557 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-09-13T16:15:15.565 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-13T16:15:15.566 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:15:32.145 DEBUG:teuthology.orchestra.remote:timed out 2024-09-13T16:15:33.953 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-09-13T16:15:41.153 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-13T16:15:41.154 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:15:41.830 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-13T16:15:42.746 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-13T16:15:42.746 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.4.0-195-generic"... 2024-09-13T16:15:42.746 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2024-09-13T16:15:42.753 INFO:teuthology.orchestra.run.smithi119.stdout:5.4.0-195-generic 2024-09-13T16:15:42.753 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-13T16:15:42.753 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T16:15:42.753 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T16:15:43.754 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-13T16:15:43.754 DEBUG:teuthology.orchestra.run.smithi119:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T16:15:43.880 INFO:teuthology.orchestra.run.smithi119.stdout:ttyS1 2024-09-13T16:15:43.909 DEBUG:teuthology.parallel:result is None 2024-09-13T16:15:45.953 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-13T16:15:45.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:15:46.226 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-09-13T16:15:46.815 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-13T16:15:46.815 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.4.0-195-generic"... 2024-09-13T16:15:46.815 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-09-13T16:15:46.822 INFO:teuthology.orchestra.run.smithi145.stdout:5.4.0-195-generic 2024-09-13T16:15:46.822 DEBUG:teuthology.task.kernel:current kernel version is 5.4.0-195-generic vs 5.4.0-195-generic 2024-09-13T16:15:46.822 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-13T16:15:46.822 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-13T16:15:47.823 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-13T16:15:47.823 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-13T16:15:47.912 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2024-09-13T16:15:47.940 DEBUG:teuthology.parallel:result is None 2024-09-13T16:15:47.940 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-13T16:15:47.947 INFO:teuthology.task.internal:Creating test directory... 2024-09-13T16:15:47.947 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T16:15:47.950 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-13T16:15:47.956 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-13T16:15:48.021 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-13T16:15:48.031 INFO:teuthology.task.internal:Creating archive directory... 2024-09-13T16:15:48.031 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T16:15:48.034 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-13T16:15:48.154 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-13T16:15:48.161 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-13T16:15:48.161 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T16:15:48.163 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-13T16:15:48.181 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:15:48.186 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:15:48.193 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:15:48.196 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-13T16:15:48.197 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-13T16:15:48.204 INFO:teuthology.task.internal:Configuring sudo... 2024-09-13T16:15:48.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T16:15:48.206 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-13T16:15:48.221 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-13T16:15:48.229 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-13T16:15:48.229 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T16:15:48.261 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-13T16:15:48.268 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T16:15:48.275 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T16:15:48.282 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:15:48.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T16:15:48.349 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-13T16:15:48.356 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-13T16:15:48.404 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:15:48.405 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-13T16:15:48.436 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2024-09-13T16:15:48.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-09-13T16:15:48.487 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-13T16:15:48.494 INFO:teuthology.task.internal:Starting timer... 2024-09-13T16:15:48.494 INFO:teuthology.run_tasks:Running task pcp... 2024-09-13T16:15:48.503 INFO:teuthology.run_tasks:Running task selinux... 2024-09-13T16:15:48.511 DEBUG:teuthology.task.selinux:Excluding smithi119: OS 'ubuntu' does not support SELinux 2024-09-13T16:15:48.511 DEBUG:teuthology.task.selinux:Excluding smithi145: OS 'ubuntu' does not support SELinux 2024-09-13T16:15:48.511 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-13T16:15:48.511 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-13T16:15:48.511 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-13T16:15:48.512 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-13T16:15:48.520 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-13T16:15:48.523 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-13T16:15:48.726 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-13T16:15:48.736 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-13T16:15:48.737 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi119.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-13T16:21:22.611 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi119.front.sepia.ceph.com'), Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2024-09-13T16:21:22.612 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-13T16:21:22.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:21:22.711 DEBUG:teuthology.orchestra.run.smithi119:> true 2024-09-13T16:21:22.834 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2024-09-13T16:21:22.835 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-13T16:21:22.835 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-13T16:21:22.949 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-09-13T16:21:23.075 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-09-13T16:21:23.075 INFO:teuthology.run_tasks:Running task clock... 2024-09-13T16:21:23.086 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-13T16:21:23.086 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T16:21:23.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:21:23.089 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-13T16:21:23.089 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:21:23.123 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-13T16:21:23.123 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Command line: ntpd -gq 2024-09-13T16:21:23.125 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: proto: precision = 0.092 usec (-23) 2024-09-13T16:21:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T16:21:23.125 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T16:21:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T16:21:23.126 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T16:21:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T16:21:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T16:21:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T16:21:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Listen normally on 3 enp3s0f1 172.21.15.119:123 2024-09-13T16:21:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Listen normally on 4 lo [::1]:123 2024-09-13T16:21:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d5ef%5]:123 2024-09-13T16:21:23.127 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:23 ntpd[13730]: Listening on routing socket on fd #22 for interface updates 2024-09-13T16:21:23.153 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: ntpd 4.2.8p12@1.3728-o (1): Starting 2024-09-13T16:21:23.153 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Command line: ntpd -gq 2024-09-13T16:21:23.155 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: proto: precision = 0.117 usec (-23) 2024-09-13T16:21:23.155 INFO:teuthology.orchestra.run.smithi145.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T16:21:23.155 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-13T16:21:23.156 INFO:teuthology.orchestra.run.smithi145.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: restrict ::: KOD does nothing without LIMITED. 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Listen and drop on 0 v6wildcard [::]:123 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Listen normally on 2 lo 127.0.0.1:123 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Listen normally on 3 enp3s0f1 172.21.15.145:123 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Listen normally on 4 lo [::1]:123 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7791%5]:123 2024-09-13T16:21:23.157 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:23 ntpd[13731]: Listening on routing socket on fd #22 for interface updates 2024-09-13T16:21:24.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:24.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:24 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:24.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:24 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:25.154 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:25 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:25.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:25 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:25.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:25 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:25.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:25 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:25.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:25 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:25.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:25 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:26.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:26.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:26.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:26.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:26 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:26.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:26.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:26.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:26.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:26.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:26.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:26 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:27.124 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:27 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:27.124 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:27 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:27 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:27 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:27 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:27 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:27.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:27 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:27.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:27 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:27.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:27 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:27.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:27 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:27.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:27 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:27.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:27 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:28.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.127 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.127 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:28.127 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:28.127 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.127 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.127 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:28.127 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:28 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:28.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:28.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:28.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:28.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:28.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:28.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:28 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:29 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:29 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:29 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:29 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:29.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:29 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:29.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:29 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:29.154 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:29 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:29.154 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:29 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:29.154 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:29 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:29.154 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:29 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:29.154 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:29 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:29.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:29 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:30.124 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:30.125 INFO:teuthology.orchestra.run.smithi119.stdout:13 Sep 16:21:30 ntpd[13730]: ntpd: time slew -0.000196 s 2024-09-13T16:21:30.126 INFO:teuthology.orchestra.run.smithi119.stdout:ntpd: time slew -0.000196s 2024-09-13T16:21:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T16:21:30.126 INFO:teuthology.orchestra.run.smithi119.stderr:13 Sep 16:21:30 ntpd[13730]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T16:21:30.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:30.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:30.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:30.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:30.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:30.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:30.155 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:30.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:30.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-13T16:21:30.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: can't open /var/log/ntpstats/rawstats.20240913: Permission denied 2024-09-13T16:21:30.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-13T16:21:30.156 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: can't open /var/log/ntpstats/peerstats.20240913: Permission denied 2024-09-13T16:21:30.156 INFO:teuthology.orchestra.run.smithi145.stdout:13 Sep 16:21:30 ntpd[13731]: ntpd: time slew -0.000324 s 2024-09-13T16:21:30.156 INFO:teuthology.orchestra.run.smithi145.stdout:ntpd: time slew -0.000324s 2024-09-13T16:21:30.157 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-13T16:21:30.157 INFO:teuthology.orchestra.run.smithi145.stderr:13 Sep 16:21:30 ntpd[13731]: can't open /var/log/ntpstats/loopstats.20240913: Permission denied 2024-09-13T16:21:30.225 INFO:teuthology.orchestra.run.smithi145.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T16:21:30.225 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================== 2024-09-13T16:21:30.225 INFO:teuthology.orchestra.run.smithi145.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.225 INFO:teuthology.orchestra.run.smithi145.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.225 INFO:teuthology.orchestra.run.smithi145.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.226 INFO:teuthology.orchestra.run.smithi145.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.424 INFO:teuthology.orchestra.run.smithi119.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T16:21:30.424 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================== 2024-09-13T16:21:30.425 INFO:teuthology.orchestra.run.smithi119.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.425 INFO:teuthology.orchestra.run.smithi119.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.425 INFO:teuthology.orchestra.run.smithi119.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.425 INFO:teuthology.orchestra.run.smithi119.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2024-09-13T16:21:30.426 INFO:teuthology.run_tasks:Running task install... 2024-09-13T16:21:30.435 DEBUG:teuthology.task.install:project ceph 2024-09-13T16:21:30.435 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}} 2024-09-13T16:21:30.435 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'} 2024-09-13T16:21:30.436 INFO:teuthology.task.install:Using flavor: default 2024-09-13T16:21:30.441 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-13T16:21:30.441 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2024-09-13T16:21:30.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-key list | grep Ceph 2024-09-13T16:21:30.442 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-key list | grep Ceph 2024-09-13T16:21:30.548 INFO:teuthology.orchestra.run.smithi145.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-13T16:21:30.567 INFO:teuthology.orchestra.run.smithi119.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2024-09-13T16:21:30.607 INFO:teuthology.orchestra.run.smithi145.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T16:21:30.608 INFO:teuthology.orchestra.run.smithi145.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T16:21:30.609 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-13T16:21:30.609 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-13T16:21:30.609 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:21:30.627 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-13T16:21:30.628 INFO:teuthology.orchestra.run.smithi119.stdout:uid [ unknown] Ceph.com (release key) 2024-09-13T16:21:30.629 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror on remote deb x86_64 2024-09-13T16:21:30.629 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, network-manager, iproute2, util-linux, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev, postgresql, postgresql-client, postgresql-common, postgresql-contrib on remote deb x86_64 2024-09-13T16:21:30.629 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:21:30.759 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default/ 2024-09-13T16:21:30.759 INFO:teuthology.task.install.deb:Package version is 18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:21:30.773 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default/ 2024-09-13T16:21:30.773 INFO:teuthology.task.install.deb:Package version is 18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:21:30.879 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:21:30.879 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T16:21:30.891 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:21:30.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-13T16:21:30.895 DEBUG:teuthology.orchestra.run.smithi145:> sudo apt-get update 2024-09-13T16:21:30.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo apt-get update 2024-09-13T16:21:31.048 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T16:21:31.055 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-13T16:21:31.072 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-13T16:21:31.105 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:4 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal InRelease 2024-09-13T16:21:31.127 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-13T16:21:31.138 INFO:teuthology.orchestra.run.smithi145.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal InRelease 2024-09-13T16:21:31.150 INFO:teuthology.orchestra.run.smithi119.stdout:Ign:6 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal Release.gpg 2024-09-13T16:21:31.159 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal Release [7,662 B] 2024-09-13T16:21:31.166 INFO:teuthology.orchestra.run.smithi119.stdout:Hit:7 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-13T16:21:31.171 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main i386 Packages [3,435 B] 2024-09-13T16:21:31.183 INFO:teuthology.orchestra.run.smithi145.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal Release.gpg 2024-09-13T16:21:31.205 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-13T16:21:31.213 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:4 http://security.ubuntu.com/ubuntu focal-security InRelease 2024-09-13T16:21:31.213 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2024-09-13T16:21:31.214 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main i386 Packages [3,435 B] 2024-09-13T16:21:31.247 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 Packages [17.6 kB] 2024-09-13T16:21:31.284 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2024-09-13T16:21:31.366 INFO:teuthology.orchestra.run.smithi145.stdout:Hit:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2024-09-13T16:21:31.714 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 28.7 kB in 1s (44.9 kB/s) 2024-09-13T16:21:31.880 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 28.7 kB in 1s (37.1 kB/s) 2024-09-13T16:21:32.872 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2024-09-13T16:21:32.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-850-g50bbe7f3-1focal cephadm=18.2.4-850-g50bbe7f3-1focal ceph-mds=18.2.4-850-g50bbe7f3-1focal ceph-mgr=18.2.4-850-g50bbe7f3-1focal ceph-common=18.2.4-850-g50bbe7f3-1focal ceph-fuse=18.2.4-850-g50bbe7f3-1focal ceph-test=18.2.4-850-g50bbe7f3-1focal ceph-volume=18.2.4-850-g50bbe7f3-1focal radosgw=18.2.4-850-g50bbe7f3-1focal python3-rados=18.2.4-850-g50bbe7f3-1focal python3-rgw=18.2.4-850-g50bbe7f3-1focal python3-cephfs=18.2.4-850-g50bbe7f3-1focal python3-rbd=18.2.4-850-g50bbe7f3-1focal libcephfs2=18.2.4-850-g50bbe7f3-1focal libcephfs-dev=18.2.4-850-g50bbe7f3-1focal librados2=18.2.4-850-g50bbe7f3-1focal librbd1=18.2.4-850-g50bbe7f3-1focal rbd-fuse=18.2.4-850-g50bbe7f3-1focal python3-cephfs=18.2.4-850-g50bbe7f3-1focal cephfs-shell=18.2.4-850-g50bbe7f3-1focal cephfs-top=18.2.4-850-g50bbe7f3-1focal cephfs-mirror=18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:21:32.973 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2024-09-13T16:21:33.035 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-09-13T16:21:33.063 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=18.2.4-850-g50bbe7f3-1focal cephadm=18.2.4-850-g50bbe7f3-1focal ceph-mds=18.2.4-850-g50bbe7f3-1focal ceph-mgr=18.2.4-850-g50bbe7f3-1focal ceph-common=18.2.4-850-g50bbe7f3-1focal ceph-fuse=18.2.4-850-g50bbe7f3-1focal ceph-test=18.2.4-850-g50bbe7f3-1focal ceph-volume=18.2.4-850-g50bbe7f3-1focal radosgw=18.2.4-850-g50bbe7f3-1focal python3-rados=18.2.4-850-g50bbe7f3-1focal python3-rgw=18.2.4-850-g50bbe7f3-1focal python3-cephfs=18.2.4-850-g50bbe7f3-1focal python3-rbd=18.2.4-850-g50bbe7f3-1focal libcephfs2=18.2.4-850-g50bbe7f3-1focal libcephfs-dev=18.2.4-850-g50bbe7f3-1focal librados2=18.2.4-850-g50bbe7f3-1focal librbd1=18.2.4-850-g50bbe7f3-1focal rbd-fuse=18.2.4-850-g50bbe7f3-1focal python3-cephfs=18.2.4-850-g50bbe7f3-1focal cephfs-shell=18.2.4-850-g50bbe7f3-1focal cephfs-top=18.2.4-850-g50bbe7f3-1focal cephfs-mirror=18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:21:33.131 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-09-13T16:21:33.191 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2024-09-13T16:21:33.191 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2024-09-13T16:21:33.324 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-09-13T16:21:33.325 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-09-13T16:21:33.384 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-13T16:21:33.386 INFO:teuthology.orchestra.run.smithi119.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-13T16:21:33.387 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-13T16:21:33.387 INFO:teuthology.orchestra.run.smithi119.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-13T16:21:33.387 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-13T16:21:33.387 INFO:teuthology.orchestra.run.smithi119.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-13T16:21:33.387 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common python3-cherrypy3 python3-cmd2 python3-colorama 2024-09-13T16:21:33.387 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-13T16:21:33.388 INFO:teuthology.orchestra.run.smithi119.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout:Recommended packages: 2024-09-13T16:21:33.389 INFO:teuthology.orchestra.run.smithi119.stdout: btrfs-tools 2024-09-13T16:21:33.515 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T16:21:33.516 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2024-09-13T16:21:33.516 INFO:teuthology.orchestra.run.smithi119.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T16:21:33.516 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T16:21:33.516 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-13T16:21:33.516 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2024-09-13T16:21:33.516 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2024-09-13T16:21:33.516 INFO:teuthology.orchestra.run.smithi119.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2024-09-13T16:21:33.517 INFO:teuthology.orchestra.run.smithi119.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd fonts-mathjax jq 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc libjs-underscore 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 libpcre2-16-0 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 libradosstriper1 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 nvme-cli 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-13T16:21:33.520 INFO:teuthology.orchestra.run.smithi145.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common python3-cherrypy3 python3-cmd2 python3-colorama 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography python3-decorator python3-google-auth python3-html5lib 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-jinja2 python3-joblib python3-jwt python3-kubernetes 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-logutils python3-lxml python3-mako python3-markupsafe 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-natsort python3-oauthlib python3-openssl python3-packaging 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-repoze.lru python3-requests-oauthlib python3-routes python3-rsa 2024-09-13T16:21:33.521 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout: qttranslations5-l10n socat xmlstarlet 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout: python3-influxdb fonts-mathjax-extras fonts-stix libjs-mathjax-doc 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout: python-attr-doc python-blinker-doc python-cryptography-doc 2024-09-13T16:21:33.522 INFO:teuthology.orchestra.run.smithi145.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout:Recommended packages: 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi145.stdout: btrfs-tools 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi119.stdout: nvme-cli python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-13T16:21:33.523 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cmd2 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-13T16:21:33.524 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-13T16:21:33.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-13T16:21:33.525 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-13T16:21:33.525 INFO:teuthology.orchestra.run.smithi119.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-13T16:21:33.525 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages will be upgraded: 2024-09-13T16:21:33.525 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 librbd1 2024-09-13T16:21:33.647 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2024-09-13T16:21:33.647 INFO:teuthology.orchestra.run.smithi145.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-13T16:21:33.647 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-13T16:21:33.647 INFO:teuthology.orchestra.run.smithi145.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm 2024-09-13T16:21:33.647 INFO:teuthology.orchestra.run.smithi145.stdout: cephfs-mirror cephfs-shell cephfs-top fonts-mathjax jq libcephfs-dev 2024-09-13T16:21:33.647 INFO:teuthology.orchestra.run.smithi145.stdout: libcephfs2 libdouble-conversion3 libjq1 libjs-mathjax libjs-sphinxdoc 2024-09-13T16:21:33.648 INFO:teuthology.orchestra.run.smithi145.stdout: libjs-underscore liblbfgsb0 liblttng-ust0 liblua5.3-0 liboath0 libonig5 2024-09-13T16:21:33.648 INFO:teuthology.orchestra.run.smithi145.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librabbitmq4 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.13.0 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: nvme-cli python-asyncssh-doc python-natsort-doc python-pastedeploy-tpl 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-asyncssh python3-atomicwrites python3-attr python3-bcrypt 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-blinker python3-bs4 python3-cachetools python3-ceph-argparse 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-ceph-common python3-cephfs python3-cherrypy3 python3-cmd2 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-colorama python3-cryptography python3-decorator python3-google-auth 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-html5lib python3-jinja2 python3-joblib python3-jwt 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-kubernetes python3-logutils python3-lxml python3-mako 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-markupsafe python3-natsort python3-oauthlib python3-openssl 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2024-09-13T16:21:33.649 INFO:teuthology.orchestra.run.smithi145.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout: python3-scipy python3-simplegeneric python3-simplejson 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout: qttranslations5-l10n radosgw rbd-fuse socat xmlstarlet 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages will be upgraded: 2024-09-13T16:21:33.650 INFO:teuthology.orchestra.run.smithi145.stdout: librados2 librbd1 2024-09-13T16:21:33.667 INFO:teuthology.orchestra.run.smithi119.stdout:2 upgraded, 116 newly installed, 0 to remove and 295 not upgraded. 2024-09-13T16:21:33.667 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 199 MB of archives. 2024-09-13T16:21:33.667 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 979 MB of additional disk space will be used. 2024-09-13T16:21:33.667 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-850-g50bbe7f3-1focal [3,450 kB] 2024-09-13T16:21:33.711 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2024-09-13T16:21:33.716 INFO:teuthology.orchestra.run.smithi145.stdout:2 upgraded, 116 newly installed, 0 to remove and 295 not upgraded. 2024-09-13T16:21:33.716 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 199 MB of archives. 2024-09-13T16:21:33.716 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 979 MB of additional disk space will be used. 2024-09-13T16:21:33.716 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2024-09-13T16:21:33.735 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-13T16:21:33.742 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2024-09-13T16:21:33.768 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-13T16:21:33.794 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2024-09-13T16:21:33.798 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-13T16:21:33.801 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-13T16:21:33.804 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 18.2.4-850-g50bbe7f3-1focal [3,450 kB] 2024-09-13T16:21:33.815 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-13T16:21:33.828 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2024-09-13T16:21:33.838 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-850-g50bbe7f3-1focal [4,041 kB] 2024-09-13T16:21:33.857 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-13T16:21:33.859 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-13T16:21:33.865 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-13T16:21:33.867 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-13T16:21:33.867 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-13T16:21:33.868 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-13T16:21:33.868 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-13T16:21:33.872 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-13T16:21:33.872 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-13T16:21:33.875 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-13T16:21:33.891 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-13T16:21:33.891 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-13T16:21:33.892 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-13T16:21:33.893 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T16:21:33.894 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-13T16:21:33.894 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-13T16:21:33.896 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-13T16:21:33.897 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-13T16:21:33.898 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-13T16:21:33.899 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-850-g50bbe7f3-1focal [747 kB] 2024-09-13T16:21:33.909 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-13T16:21:33.913 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-850-g50bbe7f3-1focal [386 kB] 2024-09-13T16:21:33.918 INFO:teuthology.orchestra.run.smithi145.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-13T16:21:33.920 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-850-g50bbe7f3-1focal [31.9 kB] 2024-09-13T16:21:33.920 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-850-g50bbe7f3-1focal [201 kB] 2024-09-13T16:21:33.923 INFO:teuthology.orchestra.run.smithi145.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-13T16:21:33.923 INFO:teuthology.orchestra.run.smithi145.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-13T16:21:33.923 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-850-g50bbe7f3-1focal [62.5 kB] 2024-09-13T16:21:33.924 INFO:teuthology.orchestra.run.smithi145.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-13T16:21:33.924 INFO:teuthology.orchestra.run.smithi145.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-13T16:21:33.925 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-850-g50bbe7f3-1focal [367 kB] 2024-09-13T16:21:33.925 INFO:teuthology.orchestra.run.smithi145.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-13T16:21:33.928 INFO:teuthology.orchestra.run.smithi145.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-13T16:21:33.928 INFO:teuthology.orchestra.run.smithi145.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-13T16:21:33.930 INFO:teuthology.orchestra.run.smithi145.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-13T16:21:33.930 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-850-g50bbe7f3-1focal [6,627 kB] 2024-09-13T16:21:33.944 INFO:teuthology.orchestra.run.smithi145.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-13T16:21:33.946 INFO:teuthology.orchestra.run.smithi145.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-13T16:21:33.947 INFO:teuthology.orchestra.run.smithi145.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-13T16:21:33.948 INFO:teuthology.orchestra.run.smithi145.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-13T16:21:33.948 INFO:teuthology.orchestra.run.smithi145.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-13T16:21:33.953 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2024-09-13T16:21:33.961 INFO:teuthology.orchestra.run.smithi145.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-13T16:21:33.962 INFO:teuthology.orchestra.run.smithi145.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-13T16:21:33.962 INFO:teuthology.orchestra.run.smithi145.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-13T16:21:33.963 INFO:teuthology.orchestra.run.smithi145.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-13T16:21:33.966 INFO:teuthology.orchestra.run.smithi145.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T16:21:33.978 INFO:teuthology.orchestra.run.smithi145.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-13T16:21:33.980 INFO:teuthology.orchestra.run.smithi145.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-13T16:21:33.982 INFO:teuthology.orchestra.run.smithi145.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-13T16:21:33.984 INFO:teuthology.orchestra.run.smithi145.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-13T16:21:33.984 INFO:teuthology.orchestra.run.smithi145.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-13T16:21:33.996 INFO:teuthology.orchestra.run.smithi145.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-13T16:21:34.051 INFO:teuthology.orchestra.run.smithi119.stdout:Get:14 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-850-g50bbe7f3-1focal [113 kB] 2024-09-13T16:21:34.053 INFO:teuthology.orchestra.run.smithi119.stdout:Get:15 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-850-g50bbe7f3-1focal [479 kB] 2024-09-13T16:21:34.059 INFO:teuthology.orchestra.run.smithi119.stdout:Get:16 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-850-g50bbe7f3-1focal [25.2 MB] 2024-09-13T16:21:34.076 INFO:teuthology.orchestra.run.smithi145.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-13T16:21:34.082 INFO:teuthology.orchestra.run.smithi119.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/universe amd64 libdouble-conversion3 amd64 3.1.5-4ubuntu1 [37.9 kB] 2024-09-13T16:21:34.093 INFO:teuthology.orchestra.run.smithi145.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-13T16:21:34.094 INFO:teuthology.orchestra.run.smithi145.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-13T16:21:34.098 INFO:teuthology.orchestra.run.smithi119.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpcre2-16-0 amd64 10.34-7ubuntu0.1 [181 kB] 2024-09-13T16:21:34.105 INFO:teuthology.orchestra.run.smithi145.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-13T16:21:34.159 INFO:teuthology.orchestra.run.smithi119.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5core5a amd64 5.12.8+dfsg-0ubuntu2.1 [2,006 kB] 2024-09-13T16:21:34.169 INFO:teuthology.orchestra.run.smithi145.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-13T16:21:34.170 INFO:teuthology.orchestra.run.smithi145.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-13T16:21:34.171 INFO:teuthology.orchestra.run.smithi145.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-13T16:21:34.172 INFO:teuthology.orchestra.run.smithi145.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-13T16:21:34.173 INFO:teuthology.orchestra.run.smithi145.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-13T16:21:34.174 INFO:teuthology.orchestra.run.smithi145.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-13T16:21:34.175 INFO:teuthology.orchestra.run.smithi145.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-13T16:21:34.176 INFO:teuthology.orchestra.run.smithi145.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-13T16:21:34.187 INFO:teuthology.orchestra.run.smithi145.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-13T16:21:34.195 INFO:teuthology.orchestra.run.smithi145.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-13T16:21:34.197 INFO:teuthology.orchestra.run.smithi145.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-13T16:21:34.199 INFO:teuthology.orchestra.run.smithi145.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-13T16:21:34.200 INFO:teuthology.orchestra.run.smithi145.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-13T16:21:34.205 INFO:teuthology.orchestra.run.smithi145.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-13T16:21:34.208 INFO:teuthology.orchestra.run.smithi145.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2024-09-13T16:21:34.208 INFO:teuthology.orchestra.run.smithi145.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-13T16:21:34.222 INFO:teuthology.orchestra.run.smithi145.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-13T16:21:34.225 INFO:teuthology.orchestra.run.smithi145.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-13T16:21:34.227 INFO:teuthology.orchestra.run.smithi145.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-13T16:21:34.265 INFO:teuthology.orchestra.run.smithi145.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-13T16:21:34.266 INFO:teuthology.orchestra.run.smithi145.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-13T16:21:34.266 INFO:teuthology.orchestra.run.smithi145.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-13T16:21:34.266 INFO:teuthology.orchestra.run.smithi145.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-13T16:21:34.278 INFO:teuthology.orchestra.run.smithi145.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T16:21:34.278 INFO:teuthology.orchestra.run.smithi145.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-13T16:21:34.279 INFO:teuthology.orchestra.run.smithi145.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-13T16:21:34.279 INFO:teuthology.orchestra.run.smithi145.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-13T16:21:34.280 INFO:teuthology.orchestra.run.smithi145.stdout:Get:85 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-13T16:21:34.355 INFO:teuthology.orchestra.run.smithi119.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5dbus5 amd64 5.12.8+dfsg-0ubuntu2.1 [208 kB] 2024-09-13T16:21:34.364 INFO:teuthology.orchestra.run.smithi119.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libqt5network5 amd64 5.12.8+dfsg-0ubuntu2.1 [673 kB] 2024-09-13T16:21:34.389 INFO:teuthology.orchestra.run.smithi119.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/universe amd64 libthrift-0.13.0 amd64 0.13.0-2build2 [249 kB] 2024-09-13T16:21:34.397 INFO:teuthology.orchestra.run.smithi119.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2024-09-13T16:21:34.398 INFO:teuthology.orchestra.run.smithi119.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 liblua5.3-0 amd64 5.3.3-1.1ubuntu2 [116 kB] 2024-09-13T16:21:34.403 INFO:teuthology.orchestra.run.smithi119.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2024-09-13T16:21:34.404 INFO:teuthology.orchestra.run.smithi119.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2024-09-13T16:21:34.416 INFO:teuthology.orchestra.run.smithi119.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2024-09-13T16:21:34.416 INFO:teuthology.orchestra.run.smithi119.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2024-09-13T16:21:34.437 INFO:teuthology.orchestra.run.smithi119.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-natsort all 7.0.1-1 [29.2 kB] 2024-09-13T16:21:34.503 INFO:teuthology.orchestra.run.smithi119.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2024-09-13T16:21:34.504 INFO:teuthology.orchestra.run.smithi119.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2024-09-13T16:21:34.504 INFO:teuthology.orchestra.run.smithi119.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2.1 [59.2 kB] 2024-09-13T16:21:34.505 INFO:teuthology.orchestra.run.smithi119.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-13T16:21:34.505 INFO:teuthology.orchestra.run.smithi119.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2024-09-13T16:21:34.506 INFO:teuthology.orchestra.run.smithi119.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-webob all 1:1.8.5-2ubuntu0.1 [84.5 kB] 2024-09-13T16:21:34.507 INFO:teuthology.orchestra.run.smithi119.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2024-09-13T16:21:34.508 INFO:teuthology.orchestra.run.smithi119.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2024-09-13T16:21:34.580 INFO:teuthology.orchestra.run.smithi119.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2024-09-13T16:21:34.582 INFO:teuthology.orchestra.run.smithi145.stdout:Get:86 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 18.2.4-850-g50bbe7f3-1focal [4,041 kB] 2024-09-13T16:21:34.630 INFO:teuthology.orchestra.run.smithi119.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2024-09-13T16:21:34.630 INFO:teuthology.orchestra.run.smithi119.stdout:Get:40 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-850-g50bbe7f3-1focal [9,614 kB] 2024-09-13T16:21:34.631 INFO:teuthology.orchestra.run.smithi119.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2024-09-13T16:21:34.644 INFO:teuthology.orchestra.run.smithi119.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2024-09-13T16:21:34.644 INFO:teuthology.orchestra.run.smithi119.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2024-09-13T16:21:34.645 INFO:teuthology.orchestra.run.smithi119.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2024-09-13T16:21:34.646 INFO:teuthology.orchestra.run.smithi119.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2024-09-13T16:21:34.647 INFO:teuthology.orchestra.run.smithi119.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-werkzeug all 0.16.1+dfsg1-2ubuntu0.2 [183 kB] 2024-09-13T16:21:34.650 INFO:teuthology.orchestra.run.smithi119.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2024-09-13T16:21:34.681 INFO:teuthology.orchestra.run.smithi119.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.3 [211 kB] 2024-09-13T16:21:34.732 INFO:teuthology.orchestra.run.smithi119.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2024-09-13T16:21:34.733 INFO:teuthology.orchestra.run.smithi119.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-asyncssh all 1.12.2-1 [152 kB] 2024-09-13T16:21:34.735 INFO:teuthology.orchestra.run.smithi119.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jinja2 all 2.10.1-2ubuntu0.3 [96.0 kB] 2024-09-13T16:21:34.738 INFO:teuthology.orchestra.run.smithi119.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2024-09-13T16:21:34.738 INFO:teuthology.orchestra.run.smithi119.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2024-09-13T16:21:34.739 INFO:teuthology.orchestra.run.smithi119.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2024-09-13T16:21:34.739 INFO:teuthology.orchestra.run.smithi119.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2024-09-13T16:21:34.813 INFO:teuthology.orchestra.run.smithi119.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2024-09-13T16:21:34.814 INFO:teuthology.orchestra.run.smithi119.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-850-g50bbe7f3-1focal [238 kB] 2024-09-13T16:21:34.814 INFO:teuthology.orchestra.run.smithi119.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2024-09-13T16:21:34.817 INFO:teuthology.orchestra.run.smithi119.stdout:Get:59 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-850-g50bbe7f3-1focal [148 kB] 2024-09-13T16:21:34.820 INFO:teuthology.orchestra.run.smithi119.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-850-g50bbe7f3-1focal [1,575 kB] 2024-09-13T16:21:34.835 INFO:teuthology.orchestra.run.smithi119.stdout:Get:61 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-850-g50bbe7f3-1focal [6,639 kB] 2024-09-13T16:21:34.843 INFO:teuthology.orchestra.run.smithi119.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2024-09-13T16:21:34.845 INFO:teuthology.orchestra.run.smithi119.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-13T16:21:34.846 INFO:teuthology.orchestra.run.smithi119.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2024-09-13T16:21:34.848 INFO:teuthology.orchestra.run.smithi119.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2024-09-13T16:21:34.849 INFO:teuthology.orchestra.run.smithi119.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2024-09-13T16:21:34.850 INFO:teuthology.orchestra.run.smithi119.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2024-09-13T16:21:34.851 INFO:teuthology.orchestra.run.smithi119.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2024-09-13T16:21:34.906 INFO:teuthology.orchestra.run.smithi145.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 18.2.4-850-g50bbe7f3-1focal [747 kB] 2024-09-13T16:21:34.925 INFO:teuthology.orchestra.run.smithi119.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2024-09-13T16:21:34.932 INFO:teuthology.orchestra.run.smithi119.stdout:Get:70 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-850-g50bbe7f3-1focal [24.5 MB] 2024-09-13T16:21:35.017 INFO:teuthology.orchestra.run.smithi145.stdout:Get:88 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 18.2.4-850-g50bbe7f3-1focal [386 kB] 2024-09-13T16:21:35.050 INFO:teuthology.orchestra.run.smithi145.stdout:Get:89 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 18.2.4-850-g50bbe7f3-1focal [31.9 kB] 2024-09-13T16:21:35.051 INFO:teuthology.orchestra.run.smithi145.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 18.2.4-850-g50bbe7f3-1focal [201 kB] 2024-09-13T16:21:35.079 INFO:teuthology.orchestra.run.smithi145.stdout:Get:91 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 18.2.4-850-g50bbe7f3-1focal [62.5 kB] 2024-09-13T16:21:35.080 INFO:teuthology.orchestra.run.smithi145.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 18.2.4-850-g50bbe7f3-1focal [367 kB] 2024-09-13T16:21:35.113 INFO:teuthology.orchestra.run.smithi145.stdout:Get:93 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 18.2.4-850-g50bbe7f3-1focal [6,627 kB] 2024-09-13T16:21:35.205 INFO:teuthology.orchestra.run.smithi119.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2024-09-13T16:21:35.279 INFO:teuthology.orchestra.run.smithi119.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2024-09-13T16:21:35.281 INFO:teuthology.orchestra.run.smithi119.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2024-09-13T16:21:35.350 INFO:teuthology.orchestra.run.smithi119.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2024-09-13T16:21:35.351 INFO:teuthology.orchestra.run.smithi119.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2024-09-13T16:21:35.351 INFO:teuthology.orchestra.run.smithi119.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2024-09-13T16:21:35.352 INFO:teuthology.orchestra.run.smithi119.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2024-09-13T16:21:35.352 INFO:teuthology.orchestra.run.smithi119.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2024-09-13T16:21:35.353 INFO:teuthology.orchestra.run.smithi119.stdout:Get:79 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2024-09-13T16:21:35.353 INFO:teuthology.orchestra.run.smithi119.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2024-09-13T16:21:35.368 INFO:teuthology.orchestra.run.smithi119.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2024-09-13T16:21:35.369 INFO:teuthology.orchestra.run.smithi119.stdout:Get:82 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2024-09-13T16:21:35.450 INFO:teuthology.orchestra.run.smithi119.stdout:Get:83 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2024-09-13T16:21:35.487 INFO:teuthology.orchestra.run.smithi119.stdout:Get:84 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2024-09-13T16:21:35.489 INFO:teuthology.orchestra.run.smithi119.stdout:Get:85 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-850-g50bbe7f3-1focal [13.7 kB] 2024-09-13T16:21:35.490 INFO:teuthology.orchestra.run.smithi119.stdout:Get:86 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2024-09-13T16:21:35.490 INFO:teuthology.orchestra.run.smithi119.stdout:Get:87 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-850-g50bbe7f3-1focal [856 kB] 2024-09-13T16:21:35.493 INFO:teuthology.orchestra.run.smithi119.stdout:Get:88 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2024-09-13T16:21:35.495 INFO:teuthology.orchestra.run.smithi119.stdout:Get:89 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2024-09-13T16:21:35.500 INFO:teuthology.orchestra.run.smithi119.stdout:Get:90 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-850-g50bbe7f3-1focal [2,296 kB] 2024-09-13T16:21:35.543 INFO:teuthology.orchestra.run.smithi119.stdout:Get:91 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2024-09-13T16:21:35.543 INFO:teuthology.orchestra.run.smithi119.stdout:Get:92 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-850-g50bbe7f3-1focal [239 kB] 2024-09-13T16:21:35.547 INFO:teuthology.orchestra.run.smithi119.stdout:Get:93 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2024-09-13T16:21:35.548 INFO:teuthology.orchestra.run.smithi119.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-850-g50bbe7f3-1focal [143 kB] 2024-09-13T16:21:35.548 INFO:teuthology.orchestra.run.smithi119.stdout:Get:95 http://archive.ubuntu.com/ubuntu focal/universe amd64 fonts-mathjax all 2.7.4+dfsg-1 [2,208 kB] 2024-09-13T16:21:35.550 INFO:teuthology.orchestra.run.smithi119.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-850-g50bbe7f3-1focal [7,015 kB] 2024-09-13T16:21:35.570 INFO:teuthology.orchestra.run.smithi119.stdout:Get:97 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.2 [283 kB] 2024-09-13T16:21:35.573 INFO:teuthology.orchestra.run.smithi119.stdout:Get:98 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-asyncssh-doc all 1.12.2-1 [258 kB] 2024-09-13T16:21:35.576 INFO:teuthology.orchestra.run.smithi145.stdout:Get:94 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 18.2.4-850-g50bbe7f3-1focal [113 kB] 2024-09-13T16:21:35.583 INFO:teuthology.orchestra.run.smithi145.stdout:Get:95 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 18.2.4-850-g50bbe7f3-1focal [479 kB] 2024-09-13T16:21:35.601 INFO:teuthology.orchestra.run.smithi145.stdout:Get:96 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 18.2.4-850-g50bbe7f3-1focal [25.2 MB] 2024-09-13T16:21:35.612 INFO:teuthology.orchestra.run.smithi119.stdout:Get:99 http://archive.ubuntu.com/ubuntu focal/universe amd64 libjs-mathjax all 2.7.4+dfsg-1 [5,654 kB] 2024-09-13T16:21:35.670 INFO:teuthology.orchestra.run.smithi119.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-850-g50bbe7f3-1focal [7,873 kB] 2024-09-13T16:21:35.776 INFO:teuthology.orchestra.run.smithi119.stdout:Get:101 http://archive.ubuntu.com/ubuntu focal/universe amd64 python-natsort-doc all 7.0.1-1 [103 kB] 2024-09-13T16:21:35.777 INFO:teuthology.orchestra.run.smithi119.stdout:Get:102 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2024-09-13T16:21:35.777 INFO:teuthology.orchestra.run.smithi119.stdout:Get:103 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2024-09-13T16:21:35.778 INFO:teuthology.orchestra.run.smithi119.stdout:Get:104 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2024-09-13T16:21:35.818 INFO:teuthology.orchestra.run.smithi119.stdout:Get:105 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-13T16:21:35.819 INFO:teuthology.orchestra.run.smithi119.stdout:Get:106 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2024-09-13T16:21:35.820 INFO:teuthology.orchestra.run.smithi119.stdout:Get:107 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2024-09-13T16:21:35.820 INFO:teuthology.orchestra.run.smithi119.stdout:Get:108 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2024-09-13T16:21:35.820 INFO:teuthology.orchestra.run.smithi119.stdout:Get:109 http://archive.ubuntu.com/ubuntu focal/universe amd64 qttranslations5-l10n all 5.12.8-0ubuntu1 [1,486 kB] 2024-09-13T16:21:35.833 INFO:teuthology.orchestra.run.smithi119.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-850-g50bbe7f3-1focal [13.9 kB] 2024-09-13T16:21:35.834 INFO:teuthology.orchestra.run.smithi119.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-850-g50bbe7f3-1focal [46.9 MB] 2024-09-13T16:21:36.947 INFO:teuthology.orchestra.run.smithi145.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 18.2.4-850-g50bbe7f3-1focal [9,614 kB] 2024-09-13T16:21:37.186 INFO:teuthology.orchestra.run.smithi145.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 18.2.4-850-g50bbe7f3-1focal [238 kB] 2024-09-13T16:21:37.189 INFO:teuthology.orchestra.run.smithi145.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libsqlite3-mod-ceph amd64 18.2.4-850-g50bbe7f3-1focal [148 kB] 2024-09-13T16:21:37.191 INFO:teuthology.orchestra.run.smithi145.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 18.2.4-850-g50bbe7f3-1focal [1,575 kB] 2024-09-13T16:21:37.222 INFO:teuthology.orchestra.run.smithi145.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 18.2.4-850-g50bbe7f3-1focal [6,639 kB] 2024-09-13T16:21:37.246 INFO:teuthology.orchestra.run.smithi119.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-850-g50bbe7f3-1focal [120 kB] 2024-09-13T16:21:37.248 INFO:teuthology.orchestra.run.smithi119.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 18.2.4-850-g50bbe7f3-1focal [270 kB] 2024-09-13T16:21:37.254 INFO:teuthology.orchestra.run.smithi119.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 18.2.4-850-g50bbe7f3-1focal [33.0 kB] 2024-09-13T16:21:37.255 INFO:teuthology.orchestra.run.smithi119.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 18.2.4-850-g50bbe7f3-1focal [22.8 kB] 2024-09-13T16:21:37.255 INFO:teuthology.orchestra.run.smithi119.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-850-g50bbe7f3-1focal [37.4 kB] 2024-09-13T16:21:37.257 INFO:teuthology.orchestra.run.smithi119.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-850-g50bbe7f3-1focal [13.5 MB] 2024-09-13T16:21:37.399 INFO:teuthology.orchestra.run.smithi145.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 18.2.4-850-g50bbe7f3-1focal [24.5 MB] 2024-09-13T16:21:37.553 INFO:teuthology.orchestra.run.smithi119.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-850-g50bbe7f3-1focal [107 kB] 2024-09-13T16:21:37.959 INFO:teuthology.orchestra.run.smithi145.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 18.2.4-850-g50bbe7f3-1focal [13.7 kB] 2024-09-13T16:21:38.038 INFO:teuthology.orchestra.run.smithi145.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 18.2.4-850-g50bbe7f3-1focal [856 kB] 2024-09-13T16:21:38.052 INFO:teuthology.orchestra.run.smithi145.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 18.2.4-850-g50bbe7f3-1focal [2,296 kB] 2024-09-13T16:21:38.069 INFO:teuthology.orchestra.run.smithi145.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 18.2.4-850-g50bbe7f3-1focal [239 kB] 2024-09-13T16:21:38.071 INFO:teuthology.orchestra.run.smithi145.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 18.2.4-850-g50bbe7f3-1focal [143 kB] 2024-09-13T16:21:38.072 INFO:teuthology.orchestra.run.smithi145.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 18.2.4-850-g50bbe7f3-1focal [7,015 kB] 2024-09-13T16:21:38.167 INFO:teuthology.orchestra.run.smithi145.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 18.2.4-850-g50bbe7f3-1focal [7,873 kB] 2024-09-13T16:21:38.201 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 199 MB in 4s (49.5 MB/s) 2024-09-13T16:21:38.296 INFO:teuthology.orchestra.run.smithi145.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 18.2.4-850-g50bbe7f3-1focal [13.9 kB] 2024-09-13T16:21:38.297 INFO:teuthology.orchestra.run.smithi145.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 18.2.4-850-g50bbe7f3-1focal [46.9 MB] 2024-09-13T16:21:38.302 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-13T16:21:38.359 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-13T16:21:38.363 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2024-09-13T16:21:38.393 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2024-09-13T16:21:38.670 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T16:21:38.686 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-13T16:21:38.728 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-13T16:21:39.057 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cmd2. 2024-09-13T16:21:39.074 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2024-09-13T16:21:39.115 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2024-09-13T16:21:39.302 INFO:teuthology.orchestra.run.smithi145.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 ceph-volume all 18.2.4-850-g50bbe7f3-1focal [120 kB] 2024-09-13T16:21:39.304 INFO:teuthology.orchestra.run.smithi145.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephfs-mirror amd64 18.2.4-850-g50bbe7f3-1focal [270 kB] 2024-09-13T16:21:39.309 INFO:teuthology.orchestra.run.smithi145.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 18.2.4-850-g50bbe7f3-1focal [33.0 kB] 2024-09-13T16:21:39.310 INFO:teuthology.orchestra.run.smithi145.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 cephfs-top all 18.2.4-850-g50bbe7f3-1focal [22.8 kB] 2024-09-13T16:21:39.311 INFO:teuthology.orchestra.run.smithi145.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 18.2.4-850-g50bbe7f3-1focal [37.4 kB] 2024-09-13T16:21:39.312 INFO:teuthology.orchestra.run.smithi145.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 18.2.4-850-g50bbe7f3-1focal [13.5 MB] 2024-09-13T16:21:39.459 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-13T16:21:39.475 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-13T16:21:39.487 INFO:teuthology.orchestra.run.smithi145.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/wip-rishabh-testing-20240910.095512-reef/50bbe7f395997dcd871813949c49addbbf630361/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 18.2.4-850-g50bbe7f3-1focal [107 kB] 2024-09-13T16:21:39.559 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-13T16:21:39.921 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T16:21:39.938 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-13T16:21:39.979 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-13T16:21:40.118 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 199 MB in 6s (34.2 MB/s) 2024-09-13T16:21:40.260 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyperclip. 2024-09-13T16:21:40.291 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T16:21:40.308 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../005-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-13T16:21:40.314 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 141720 files and directories currently installed.) 2024-09-13T16:21:40.319 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2024-09-13T16:21:40.349 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-13T16:21:40.367 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2024-09-13T16:21:40.662 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-13T16:21:40.680 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2024-09-13T16:21:40.720 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-13T16:21:40.810 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T16:21:40.827 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../006-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T16:21:40.879 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:21:41.056 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cmd2. 2024-09-13T16:21:41.073 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2024-09-13T16:21:41.114 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2024-09-13T16:21:41.413 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T16:21:41.430 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../007-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T16:21:41.471 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:21:41.507 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2024-09-13T16:21:41.525 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2024-09-13T16:21:41.617 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-13T16:21:41.775 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T16:21:41.791 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../008-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T16:21:41.833 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:21:41.938 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-13T16:21:41.956 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../004-libdouble-conversion3_3.1.5-4ubuntu1_amd64.deb ... 2024-09-13T16:21:41.987 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-13T16:21:42.211 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-13T16:21:42.229 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../009-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-13T16:21:42.294 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-13T16:21:42.307 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-13T16:21:42.325 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../005-libpcre2-16-0_10.34-7ubuntu0.1_amd64.deb ... 2024-09-13T16:21:42.365 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-13T16:21:42.638 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../010-librbd1_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:42.772 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librbd1 (18.2.4-850-g50bbe7f3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-13T16:21:42.869 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-13T16:21:42.886 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../006-libqt5core5a_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T16:21:42.958 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:21:43.443 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../011-librados2_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:43.447 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-13T16:21:43.465 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../007-libqt5dbus5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T16:21:43.497 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:21:43.568 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librados2 (18.2.4-850-g50bbe7f3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-13T16:21:43.875 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-13T16:21:43.891 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../008-libqt5network5_5.12.8+dfsg-0ubuntu2.1_amd64.deb ... 2024-09-13T16:21:43.933 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:21:44.288 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T16:21:44.305 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../012-libcephfs2_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:44.337 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libthrift-0.13.0. 2024-09-13T16:21:44.346 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libcephfs2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:44.353 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../009-libthrift-0.13.0_0.13.0-2build2_amd64.deb ... 2024-09-13T16:21:44.395 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-13T16:21:44.649 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rados. 2024-09-13T16:21:44.668 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../013-python3-rados_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:44.707 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rados (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:44.747 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../010-librbd1_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:44.873 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librbd1 (18.2.4-850-g50bbe7f3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-13T16:21:45.035 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T16:21:45.051 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:21:45.093 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-ceph-argparse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:45.346 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T16:21:45.362 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../015-python3-cephfs_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:45.404 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cephfs (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:45.610 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../011-librados2_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:45.736 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librados2 (18.2.4-850-g50bbe7f3-1focal) over (15.2.17-0ubuntu0.20.04.6) ... 2024-09-13T16:21:45.749 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T16:21:45.768 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../016-python3-ceph-common_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:21:45.807 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:46.126 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T16:21:46.142 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../017-python3-prettytable_0.7.2-5_all.deb ... 2024-09-13T16:21:46.185 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-13T16:21:46.437 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T16:21:46.454 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../018-python3-rbd_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:46.456 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libcephfs2. 2024-09-13T16:21:46.474 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../012-libcephfs2_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:46.496 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rbd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:46.515 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libcephfs2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:46.857 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-13T16:21:46.859 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rados. 2024-09-13T16:21:46.873 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../019-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-13T16:21:46.875 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../013-python3-rados_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:46.915 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-13T16:21:46.917 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rados (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:47.259 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-13T16:21:47.270 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-13T16:21:47.275 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../020-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-13T16:21:47.286 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../014-python3-ceph-argparse_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:21:47.318 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-13T16:21:47.336 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-ceph-argparse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:47.581 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cephfs. 2024-09-13T16:21:47.596 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../015-python3-cephfs_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:47.646 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T16:21:47.656 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cephfs (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:47.662 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../021-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-13T16:21:47.729 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-13T16:21:47.942 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-13T16:21:47.960 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../016-python3-ceph-common_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:21:48.000 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:48.065 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package librgw2. 2024-09-13T16:21:48.084 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../022-librgw2_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:48.123 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking librgw2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:48.328 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-prettytable. 2024-09-13T16:21:48.348 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../017-python3-prettytable_0.7.2-5_all.deb ... 2024-09-13T16:21:48.420 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2024-09-13T16:21:48.672 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rbd. 2024-09-13T16:21:48.690 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../018-python3-rbd_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:48.730 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rbd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:49.001 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T16:21:49.018 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../023-python3-rgw_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:49.059 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:49.092 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblua5.3-0:amd64. 2024-09-13T16:21:49.111 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../019-liblua5.3-0_5.3.3-1.1ubuntu2_amd64.deb ... 2024-09-13T16:21:49.175 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-13T16:21:49.412 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liboath0. 2024-09-13T16:21:49.427 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-13T16:21:49.470 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-13T16:21:49.536 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librabbitmq4:amd64. 2024-09-13T16:21:49.553 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../020-librabbitmq4_0.10.0-1_amd64.deb ... 2024-09-13T16:21:49.586 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2024-09-13T16:21:49.773 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T16:21:49.789 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../025-libradosstriper1_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:49.831 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libradosstriper1 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:49.939 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-13T16:21:49.957 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../021-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2024-09-13T16:21:49.997 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-13T16:21:50.276 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-common. 2024-09-13T16:21:50.291 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../026-ceph-common_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:50.334 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:50.367 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package librgw2. 2024-09-13T16:21:50.383 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../022-librgw2_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:50.425 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking librgw2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:51.595 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rgw. 2024-09-13T16:21:51.611 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../023-python3-rgw_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:51.653 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:51.997 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liboath0. 2024-09-13T16:21:52.016 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../024-liboath0_2.6.1-1.3_amd64.deb ... 2024-09-13T16:21:52.056 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2024-09-13T16:21:52.392 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libradosstriper1. 2024-09-13T16:21:52.407 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../025-libradosstriper1_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:52.450 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libradosstriper1 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:52.856 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-base. 2024-09-13T16:21:52.874 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../027-ceph-base_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:52.925 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-base (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:52.937 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-common. 2024-09-13T16:21:52.954 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../026-ceph-common_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:52.995 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:54.144 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T16:21:54.162 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-13T16:21:54.202 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-13T16:21:54.497 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T16:21:54.516 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../029-python3-natsort_7.0.1-1_all.deb ... 2024-09-13T16:21:54.547 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-13T16:21:54.817 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T16:21:54.836 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-7_all.deb ... 2024-09-13T16:21:54.875 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-13T16:21:55.161 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T16:21:55.180 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../031-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-13T16:21:55.219 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-13T16:21:55.547 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-mako. 2024-09-13T16:21:55.550 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-base. 2024-09-13T16:21:55.562 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../032-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-13T16:21:55.566 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../027-ceph-base_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:21:55.605 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-13T16:21:55.619 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-base (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:21:55.908 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T16:21:55.924 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T16:21:55.958 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T16:21:56.211 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T16:21:56.229 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-13T16:21:56.269 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-13T16:21:56.605 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webob. 2024-09-13T16:21:56.622 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../035-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-13T16:21:56.663 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-13T16:21:56.880 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-13T16:21:56.900 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-6_all.deb ... 2024-09-13T16:21:56.930 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2024-09-13T16:21:56.983 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-13T16:21:56.999 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../036-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-13T16:21:57.041 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-13T16:21:57.292 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-natsort. 2024-09-13T16:21:57.308 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../029-python3-natsort_7.0.1-1_all.deb ... 2024-09-13T16:21:57.344 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-bs4. 2024-09-13T16:21:57.350 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-natsort (7.0.1-1) ... 2024-09-13T16:21:57.359 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../037-python3-bs4_4.8.2-1_all.deb ... 2024-09-13T16:21:57.402 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-13T16:21:57.653 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-logutils. 2024-09-13T16:21:57.670 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-7_all.deb ... 2024-09-13T16:21:57.703 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-logutils (0.3.3-7) ... 2024-09-13T16:21:57.788 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T16:21:57.805 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../038-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-13T16:21:57.850 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-13T16:21:58.006 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-13T16:21:58.023 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../031-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2024-09-13T16:21:58.064 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2024-09-13T16:21:58.175 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T16:21:58.191 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-6_all.deb ... 2024-09-13T16:21:58.233 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-13T16:21:58.392 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-mako. 2024-09-13T16:21:58.410 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../032-python3-mako_1.1.0+ds1-1ubuntu2.1_all.deb ... 2024-09-13T16:21:58.450 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-13T16:21:58.536 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-paste. 2024-09-13T16:21:58.556 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../040-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-13T16:21:58.619 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-13T16:21:58.804 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-13T16:21:58.820 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-13T16:21:58.862 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-13T16:21:58.939 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T16:21:58.955 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-13T16:21:58.997 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-13T16:21:59.123 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-13T16:21:59.139 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2024-09-13T16:21:59.173 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2024-09-13T16:21:59.308 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T16:21:59.324 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../042-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-13T16:21:59.366 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-13T16:21:59.509 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webob. 2024-09-13T16:21:59.528 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../035-python3-webob_1%3a1.8.5-2ubuntu0.1_all.deb ... 2024-09-13T16:21:59.567 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-13T16:21:59.703 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T16:21:59.721 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../043-python3-webtest_2.0.32-2_all.deb ... 2024-09-13T16:21:59.761 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-13T16:21:59.929 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-soupsieve. 2024-09-13T16:21:59.948 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../036-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2024-09-13T16:22:00.011 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-13T16:22:00.072 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T16:22:00.090 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../044-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-13T16:22:00.130 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-13T16:22:00.315 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-bs4. 2024-09-13T16:22:00.332 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../037-python3-bs4_4.8.2-1_all.deb ... 2024-09-13T16:22:00.398 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2024-09-13T16:22:00.484 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T16:22:00.501 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../045-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-13T16:22:00.551 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-13T16:22:00.759 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-waitress. 2024-09-13T16:22:00.777 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../038-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2024-09-13T16:22:00.820 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-13T16:22:00.870 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T16:22:00.887 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../046-ceph-mgr-modules-core_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:00.928 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-modules-core (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:01.156 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-tempita. 2024-09-13T16:22:01.173 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../039-python3-tempita_0.5.2-6_all.deb ... 2024-09-13T16:22:01.212 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-tempita (0.5.2-6) ... 2024-09-13T16:22:01.300 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T16:22:01.317 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../047-libsqlite3-mod-ceph_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:01.358 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:01.498 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-paste. 2024-09-13T16:22:01.517 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../040-python3-paste_3.4.0+dfsg1-1_all.deb ... 2024-09-13T16:22:01.548 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2024-09-13T16:22:01.661 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T16:22:01.678 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../048-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-13T16:22:01.761 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-13T16:22:01.918 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-13T16:22:01.935 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../041-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2024-09-13T16:22:01.976 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2024-09-13T16:22:02.088 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cryptography. 2024-09-13T16:22:02.106 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../049-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-13T16:22:02.147 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-13T16:22:02.313 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-13T16:22:02.330 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../042-python3-pastedeploy_2.1.0-1_all.deb ... 2024-09-13T16:22:02.371 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2024-09-13T16:22:02.501 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T16:22:02.520 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../050-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-13T16:22:02.559 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-13T16:22:02.697 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webtest. 2024-09-13T16:22:02.714 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../043-python3-webtest_2.0.32-2_all.deb ... 2024-09-13T16:22:02.755 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webtest (2.0.32-2) ... 2024-09-13T16:22:02.930 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T16:22:02.949 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../051-ceph-mgr_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:02.997 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:03.092 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pecan. 2024-09-13T16:22:03.109 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../044-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2024-09-13T16:22:03.150 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2024-09-13T16:22:03.494 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T16:22:03.510 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../052-ceph-mon_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:03.530 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-13T16:22:03.546 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../045-python3-werkzeug_0.16.1+dfsg1-2ubuntu0.2_all.deb ... 2024-09-13T16:22:03.552 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mon (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:03.588 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-13T16:22:03.917 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-13T16:22:03.933 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../046-ceph-mgr-modules-core_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:04.008 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-modules-core (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:04.405 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-13T16:22:04.424 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../047-libsqlite3-mod-ceph_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:04.463 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libsqlite3-mod-ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:04.463 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T16:22:04.481 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../053-ceph-osd_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:04.522 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-osd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:04.816 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-13T16:22:04.834 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../048-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2024-09-13T16:22:04.874 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-13T16:22:05.186 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cryptography. 2024-09-13T16:22:05.205 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../049-python3-cryptography_2.8-3ubuntu0.3_amd64.deb ... 2024-09-13T16:22:05.244 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-13T16:22:05.618 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-openssl. 2024-09-13T16:22:05.637 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../050-python3-openssl_19.0.0-1build1_all.deb ... 2024-09-13T16:22:05.676 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2024-09-13T16:22:06.038 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr. 2024-09-13T16:22:06.057 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../051-ceph-mgr_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:06.096 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:06.607 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph. 2024-09-13T16:22:06.622 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../054-ceph_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:06.623 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mon. 2024-09-13T16:22:06.644 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../052-ceph-mon_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:06.665 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:06.681 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mon (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:07.692 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T16:22:07.713 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../055-ceph-fuse_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:07.750 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:08.027 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-osd. 2024-09-13T16:22:08.043 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../053-ceph-osd_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:08.127 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-osd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:08.154 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T16:22:08.173 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../056-ceph-mds_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:08.212 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mds (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:08.740 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package cephadm. 2024-09-13T16:22:08.758 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../057-cephadm_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:08.798 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:09.101 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T16:22:09.121 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../058-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-13T16:22:09.159 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-13T16:22:09.554 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T16:22:09.573 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-13T16:22:09.612 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-13T16:22:09.940 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T16:22:09.956 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:09.998 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:10.288 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph. 2024-09-13T16:22:10.304 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../054-ceph_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:10.346 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:10.351 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-jwt. 2024-09-13T16:22:10.370 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../061-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-13T16:22:10.409 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-13T16:22:10.707 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-fuse. 2024-09-13T16:22:10.725 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../055-ceph-fuse_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:10.746 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-attr. 2024-09-13T16:22:10.762 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../062-python3-attr_19.3.0-2_all.deb ... 2024-09-13T16:22:10.787 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:10.804 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-13T16:22:11.140 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T16:22:11.155 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../063-python3-pluggy_0.13.0-2_all.deb ... 2024-09-13T16:22:11.206 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-13T16:22:11.244 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mds. 2024-09-13T16:22:11.262 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../056-ceph-mds_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:11.304 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mds (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:11.501 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-py. 2024-09-13T16:22:11.518 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../064-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-13T16:22:11.567 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-13T16:22:11.808 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephadm. 2024-09-13T16:22:11.823 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../057-cephadm_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:11.874 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:11.912 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-13T16:22:11.928 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../065-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-13T16:22:11.970 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-13T16:22:12.153 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-13T16:22:12.169 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../058-python3-asyncssh_1.12.2-1_all.deb ... 2024-09-13T16:22:12.235 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-asyncssh (1.12.2-1) ... 2024-09-13T16:22:12.273 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-packaging. 2024-09-13T16:22:12.290 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../066-python3-packaging_20.3-1_all.deb ... 2024-09-13T16:22:12.331 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-13T16:22:12.588 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jinja2. 2024-09-13T16:22:12.606 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../059-python3-jinja2_2.10.1-2ubuntu0.3_all.deb ... 2024-09-13T16:22:12.650 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T16:22:12.666 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../067-python3-pytest_4.6.9-1_all.deb ... 2024-09-13T16:22:12.688 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-13T16:22:12.750 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-13T16:22:12.983 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-13T16:22:12.999 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../060-ceph-mgr-cephadm_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:13.033 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:13.112 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T16:22:13.128 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../068-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T16:22:13.170 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T16:22:13.361 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-jwt. 2024-09-13T16:22:13.381 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../061-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2024-09-13T16:22:13.411 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-13T16:22:13.423 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjs-underscore. 2024-09-13T16:22:13.442 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../069-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-13T16:22:13.481 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-13T16:22:13.739 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-attr. 2024-09-13T16:22:13.743 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-13T16:22:13.760 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../062-python3-attr_19.3.0-2_all.deb ... 2024-09-13T16:22:13.761 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../070-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-13T16:22:13.797 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-attr (19.3.0-2) ... 2024-09-13T16:22:13.801 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-13T16:22:14.129 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-routes. 2024-09-13T16:22:14.149 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../071-python3-routes_2.4.1-2_all.deb ... 2024-09-13T16:22:14.150 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pluggy. 2024-09-13T16:22:14.166 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../063-python3-pluggy_0.13.0-2_all.deb ... 2024-09-13T16:22:14.187 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-13T16:22:14.208 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2024-09-13T16:22:14.507 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T16:22:14.523 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../072-ceph-mgr-dashboard_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:14.528 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-py. 2024-09-13T16:22:14.548 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../064-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2024-09-13T16:22:14.565 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-dashboard (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:14.586 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2024-09-13T16:22:14.940 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-atomicwrites. 2024-09-13T16:22:14.960 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../065-python3-atomicwrites_1.1.5-2build1_all.deb ... 2024-09-13T16:22:14.998 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2024-09-13T16:22:15.334 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-packaging. 2024-09-13T16:22:15.350 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../066-python3-packaging_20.3-1_all.deb ... 2024-09-13T16:22:15.392 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-packaging (20.3-1) ... 2024-09-13T16:22:15.728 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pytest. 2024-09-13T16:22:15.744 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../067-python3-pytest_4.6.9-1_all.deb ... 2024-09-13T16:22:15.803 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pytest (4.6.9-1) ... 2024-09-13T16:22:15.920 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-decorator. 2024-09-13T16:22:15.940 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../073-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-13T16:22:15.978 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-13T16:22:16.191 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-13T16:22:16.209 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../068-python3-repoze.lru_0.7-2_all.deb ... 2024-09-13T16:22:16.249 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-13T16:22:16.281 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-13T16:22:16.298 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../074-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-13T16:22:16.339 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-13T16:22:16.527 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libjs-underscore. 2024-09-13T16:22:16.545 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../069-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2024-09-13T16:22:16.585 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-13T16:22:16.658 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-scipy. 2024-09-13T16:22:16.674 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../075-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-13T16:22:16.716 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-13T16:22:16.896 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libjs-sphinxdoc. 2024-09-13T16:22:16.916 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../070-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2024-09-13T16:22:16.954 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-13T16:22:17.265 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-routes. 2024-09-13T16:22:17.282 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../071-python3-routes_2.4.1-2_all.deb ... 2024-09-13T16:22:17.332 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-routes (2.4.1-2) ... 2024-09-13T16:22:17.702 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-13T16:22:17.721 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../072-ceph-mgr-dashboard_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:17.760 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-dashboard (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:17.975 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-13T16:22:17.993 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../076-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-13T16:22:18.033 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-13T16:22:18.578 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T16:22:18.595 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../077-python3-joblib_0.14.0-3_all.deb ... 2024-09-13T16:22:18.636 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-13T16:22:18.990 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T16:22:19.007 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../078-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-13T16:22:19.048 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-13T16:22:19.215 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-decorator. 2024-09-13T16:22:19.233 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../073-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2024-09-13T16:22:19.273 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2024-09-13T16:22:19.601 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2024-09-13T16:22:19.618 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../074-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2024-09-13T16:22:19.659 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-13T16:22:19.722 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T16:22:19.739 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../079-ceph-mgr-diskprediction-local_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:19.780 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:19.995 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-scipy. 2024-09-13T16:22:20.013 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../075-python3-scipy_1.3.3-3build1_amd64.deb ... 2024-09-13T16:22:20.053 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2024-09-13T16:22:20.867 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T16:22:20.884 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../080-python3-cachetools_4.0.0-1_all.deb ... 2024-09-13T16:22:20.925 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-13T16:22:21.314 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T16:22:21.331 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../081-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-13T16:22:21.362 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-sklearn-lib. 2024-09-13T16:22:21.372 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-13T16:22:21.379 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../076-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2024-09-13T16:22:21.420 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-13T16:22:21.694 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T16:22:21.711 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../082-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-13T16:22:21.752 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-13T16:22:21.999 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-joblib. 2024-09-13T16:22:22.018 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../077-python3-joblib_0.14.0-3_all.deb ... 2024-09-13T16:22:22.055 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T16:22:22.057 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-joblib (0.14.0-3) ... 2024-09-13T16:22:22.074 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../083-python3-rsa_4.0-3_all.deb ... 2024-09-13T16:22:22.114 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-13T16:22:22.416 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-sklearn. 2024-09-13T16:22:22.430 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../078-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2024-09-13T16:22:22.474 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-13T16:22:22.481 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T16:22:22.499 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../084-python3-google-auth_1.5.1-2_all.deb ... 2024-09-13T16:22:22.534 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-13T16:22:22.882 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-blinker. 2024-09-13T16:22:22.899 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../085-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-13T16:22:22.965 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-13T16:22:23.552 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-13T16:22:23.568 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../086-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-13T16:22:23.618 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-13T16:22:23.704 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-13T16:22:23.725 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../079-ceph-mgr-diskprediction-local_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:23.762 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-diskprediction-local (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:23.972 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T16:22:23.987 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-13T16:22:24.030 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-13T16:22:24.316 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T16:22:24.332 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../088-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-13T16:22:24.366 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-13T16:22:24.652 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T16:22:24.670 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../089-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-13T16:22:24.710 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-13T16:22:24.865 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-cachetools. 2024-09-13T16:22:24.885 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../080-python3-cachetools_4.0.0-1_all.deb ... 2024-09-13T16:22:24.923 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2024-09-13T16:22:25.317 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T16:22:25.336 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:25.346 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-13T16:22:25.365 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../081-python3-pyasn1_0.4.2-3build1_all.deb ... 2024-09-13T16:22:25.374 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:25.403 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2024-09-13T16:22:25.706 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T16:22:25.722 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../091-libonig5_6.9.4-1_amd64.deb ... 2024-09-13T16:22:25.750 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-13T16:22:25.763 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-13T16:22:25.769 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../082-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2024-09-13T16:22:25.799 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-13T16:22:26.134 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T16:22:26.141 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-rsa. 2024-09-13T16:22:26.151 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../092-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:22:26.158 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../083-python3-rsa_4.0-3_all.deb ... 2024-09-13T16:22:26.192 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:22:26.198 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-rsa (4.0-3) ... 2024-09-13T16:22:26.479 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package jq. 2024-09-13T16:22:26.495 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../093-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:22:26.538 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:22:26.544 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-google-auth. 2024-09-13T16:22:26.564 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../084-python3-google-auth_1.5.1-2_all.deb ... 2024-09-13T16:22:26.602 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2024-09-13T16:22:26.791 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package socat. 2024-09-13T16:22:26.807 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../094-socat_1.7.3.3-2_amd64.deb ... 2024-09-13T16:22:26.841 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-13T16:22:26.955 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-blinker. 2024-09-13T16:22:26.971 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../085-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2024-09-13T16:22:27.013 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-13T16:22:27.186 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T16:22:27.202 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-13T16:22:27.243 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-13T16:22:27.308 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-oauthlib. 2024-09-13T16:22:27.327 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../086-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2024-09-13T16:22:27.366 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-13T16:22:27.581 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-test. 2024-09-13T16:22:27.598 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../096-ceph-test_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:27.631 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-test (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:27.745 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-13T16:22:27.762 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../087-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2024-09-13T16:22:27.803 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-13T16:22:28.157 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-websocket. 2024-09-13T16:22:28.177 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../088-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2024-09-13T16:22:28.215 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2024-09-13T16:22:28.552 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-13T16:22:28.570 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../089-python3-kubernetes_7.0.0~a1-2_all.deb ... 2024-09-13T16:22:28.634 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2024-09-13T16:22:29.203 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-13T16:22:29.221 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../090-ceph-mgr-k8sevents_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:29.252 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-mgr-k8sevents (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:29.582 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-13T16:22:29.599 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../091-libonig5_6.9.4-1_amd64.deb ... 2024-09-13T16:22:29.641 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2024-09-13T16:22:30.019 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-13T16:22:30.039 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../092-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:22:30.094 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:22:30.330 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package jq. 2024-09-13T16:22:30.349 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../093-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:22:30.413 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:22:30.667 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package socat. 2024-09-13T16:22:30.682 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../094-socat_1.7.3.3-2_amd64.deb ... 2024-09-13T16:22:30.724 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking socat (1.7.3.3-2) ... 2024-09-13T16:22:31.036 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package xmlstarlet. 2024-09-13T16:22:31.052 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../095-xmlstarlet_1.6.1-2build1_amd64.deb ... 2024-09-13T16:22:31.119 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2024-09-13T16:22:31.507 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-test. 2024-09-13T16:22:31.526 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../096-ceph-test_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:31.565 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-test (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:31.852 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T16:22:31.870 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:31.909 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ceph-volume (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:32.256 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-13T16:22:32.276 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../098-cephfs-mirror_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:32.314 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking cephfs-mirror (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:32.700 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-colorama. 2024-09-13T16:22:32.716 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../099-python3-colorama_0.4.3-1build1_all.deb ... 2024-09-13T16:22:32.758 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2024-09-13T16:22:33.003 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package cephfs-shell. 2024-09-13T16:22:33.021 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../100-cephfs-shell_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:33.061 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking cephfs-shell (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:33.297 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package cephfs-top. 2024-09-13T16:22:33.317 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../101-cephfs-top_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:33.346 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking cephfs-top (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:33.566 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-13T16:22:33.583 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../102-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-13T16:22:33.624 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:34.130 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T16:22:34.147 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../103-libcephfs-dev_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:34.189 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libcephfs-dev (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:34.583 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T16:22:34.602 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../104-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-13T16:22:34.659 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-13T16:22:35.024 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T16:22:35.044 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../105-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-13T16:22:35.082 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-13T16:22:35.461 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-13T16:22:35.478 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../106-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-13T16:22:35.518 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:35.736 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ceph-volume. 2024-09-13T16:22:35.752 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../097-ceph-volume_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:35.785 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ceph-volume (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:36.192 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephfs-mirror. 2024-09-13T16:22:36.212 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../098-cephfs-mirror_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:36.256 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephfs-mirror (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:36.617 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-colorama. 2024-09-13T16:22:36.635 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../099-python3-colorama_0.4.3-1build1_all.deb ... 2024-09-13T16:22:36.675 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2024-09-13T16:22:36.971 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephfs-shell. 2024-09-13T16:22:36.987 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../100-cephfs-shell_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:37.020 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephfs-shell (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:37.109 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-13T16:22:37.128 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../107-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-13T16:22:37.167 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-13T16:22:37.284 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package cephfs-top. 2024-09-13T16:22:37.302 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../101-cephfs-top_18.2.4-850-g50bbe7f3-1focal_all.deb ... 2024-09-13T16:22:37.342 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking cephfs-top (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:37.495 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-webencodings. 2024-09-13T16:22:37.512 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../108-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-13T16:22:37.553 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-13T16:22:37.621 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package fonts-mathjax. 2024-09-13T16:22:37.640 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../102-fonts-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-13T16:22:37.678 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:37.864 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-html5lib. 2024-09-13T16:22:37.885 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../109-python3-html5lib_1.0.1-2_all.deb ... 2024-09-13T16:22:37.922 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-13T16:22:38.232 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-13T16:22:38.250 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../103-libcephfs-dev_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:38.284 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-13T16:22:38.290 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libcephfs-dev (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:38.300 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../110-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-13T16:22:38.342 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-13T16:22:38.713 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package nvme-cli. 2024-09-13T16:22:38.730 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../104-nvme-cli_1.9-1ubuntu0.2_amd64.deb ... 2024-09-13T16:22:38.781 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking nvme-cli (1.9-1ubuntu0.2) ... 2024-09-13T16:22:38.795 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T16:22:38.814 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../111-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T16:22:38.853 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T16:22:39.135 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-13T16:22:39.151 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../105-python-asyncssh-doc_1.12.2-1_all.deb ... 2024-09-13T16:22:39.192 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-asyncssh-doc (1.12.2-1) ... 2024-09-13T16:22:39.210 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T16:22:39.227 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../112-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-13T16:22:39.268 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-13T16:22:39.573 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libjs-mathjax. 2024-09-13T16:22:39.590 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../106-libjs-mathjax_2.7.4+dfsg-1_all.deb ... 2024-09-13T16:22:39.604 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T16:22:39.621 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../113-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-13T16:22:39.631 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:39.662 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-13T16:22:39.957 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T16:22:39.975 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../114-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-13T16:22:40.015 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-13T16:22:40.309 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T16:22:40.327 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-13T16:22:40.367 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-13T16:22:41.021 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package radosgw. 2024-09-13T16:22:41.039 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../116-radosgw_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:41.079 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking radosgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:41.371 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python-natsort-doc. 2024-09-13T16:22:41.389 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../107-python-natsort-doc_7.0.1-1_all.deb ... 2024-09-13T16:22:41.429 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python-natsort-doc (7.0.1-1) ... 2024-09-13T16:22:41.798 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-webencodings. 2024-09-13T16:22:41.816 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../108-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2024-09-13T16:22:41.856 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-13T16:22:42.184 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-html5lib. 2024-09-13T16:22:42.202 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../109-python3-html5lib_1.0.1-2_all.deb ... 2024-09-13T16:22:42.242 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2024-09-13T16:22:42.493 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T16:22:42.510 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../117-rbd-fuse_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:42.551 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking rbd-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:42.629 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-lxml:amd64. 2024-09-13T16:22:42.649 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../110-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2024-09-13T16:22:42.690 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-13T16:22:42.849 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-13T16:22:43.148 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-13T16:22:43.152 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pastescript. 2024-09-13T16:22:43.172 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../111-python3-pastescript_2.0.2-4_all.deb ... 2024-09-13T16:22:43.210 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-13T16:22:43.274 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-13T16:22:43.391 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-13T16:22:43.564 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-psutil. 2024-09-13T16:22:43.583 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../112-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2024-09-13T16:22:43.618 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:43.638 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2024-09-13T16:22:43.991 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-13T16:22:44.010 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../113-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2024-09-13T16:22:44.049 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-13T16:22:44.204 INFO:teuthology.orchestra.run.smithi119.stdout:Adding system user cephadm....done 2024-09-13T16:22:44.361 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package python3-simplejson. 2024-09-13T16:22:44.367 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:44.381 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../114-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2024-09-13T16:22:44.419 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-13T16:22:44.493 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-13T16:22:44.724 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-13T16:22:44.744 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.12.8-0ubuntu1_all.deb ... 2024-09-13T16:22:44.782 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-13T16:22:44.878 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T16:22:45.154 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:45.280 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-13T16:22:45.415 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-13T16:22:45.486 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package radosgw. 2024-09-13T16:22:45.504 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../116-radosgw_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:45.563 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking radosgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:45.733 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-13T16:22:46.127 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-13T16:22:46.412 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-13T16:22:46.671 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2024-09-13T16:22:46.981 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-13T16:22:47.006 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package rbd-fuse. 2024-09-13T16:22:47.023 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../117-rbd-fuse_18.2.4-850-g50bbe7f3-1focal_amd64.deb ... 2024-09-13T16:22:47.089 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking rbd-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:47.265 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-13T16:22:47.365 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-attr (19.3.0-2) ... 2024-09-13T16:22:47.550 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-ceph-argparse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:47.687 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2024-09-13T16:22:47.830 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libdouble-conversion3:amd64 (3.1.5-4ubuntu1) ... 2024-09-13T16:22:47.859 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-13T16:22:47.956 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up nvme-cli (1.9-1ubuntu0.2) ... 2024-09-13T16:22:47.985 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2024-09-13T16:22:48.176 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:48.289 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-13T16:22:48.549 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-13T16:22:48.743 INFO:teuthology.orchestra.run.smithi145.stdout:Adding system user cephadm....done 2024-09-13T16:22:48.814 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-13T16:22:48.938 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up fonts-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:48.955 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-13T16:22:49.056 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2024-09-13T16:22:49.332 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-13T16:22:49.408 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-13T16:22:49.625 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-13T16:22:49.692 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libjs-mathjax (2.7.4+dfsg-1) ... 2024-09-13T16:22:49.810 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liboath0 (2.6.1-1.3) ... 2024-09-13T16:22:49.901 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-13T16:22:49.919 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2024-09-13T16:22:50.269 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-13T16:22:50.300 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-joblib (0.14.0-3) ... 2024-09-13T16:22:50.553 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-13T16:22:50.715 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2024-09-13T16:22:50.904 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-13T16:22:51.014 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cachetools (4.0.0-1) ... 2024-09-13T16:22:51.205 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-13T16:22:51.301 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2024-09-13T16:22:51.482 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-13T16:22:51.591 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-13T16:22:51.602 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2024-09-13T16:22:51.717 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-13T16:22:51.878 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2024-09-13T16:22:52.018 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-13T16:22:52.171 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-ceph-argparse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:52.420 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2024-09-13T16:22:52.489 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2024-09-13T16:22:52.615 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2024-09-13T16:22:52.721 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-13T16:22:52.899 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2024-09-13T16:22:53.089 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-13T16:22:53.192 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2024-09-13T16:22:53.441 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-13T16:22:53.452 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libpcre2-16-0:amd64 (10.34-7ubuntu0.1) ... 2024-09-13T16:22:53.567 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:53.586 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2024-09-13T16:22:54.135 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:22:54.270 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-13T16:22:54.412 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-natsort (7.0.1-1) ... 2024-09-13T16:22:54.638 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-13T16:22:54.722 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2024-09-13T16:22:54.764 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-13T16:22:54.881 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-13T16:22:54.998 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2024-09-13T16:22:55.158 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-13T16:22:55.374 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2024-09-13T16:22:55.476 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:22:55.602 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-13T16:22:55.684 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-jinja2 (2.10.1-2ubuntu0.3) ... 2024-09-13T16:22:55.728 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-13T16:22:56.036 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-tempita (0.5.2-6) ... 2024-09-13T16:22:56.080 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-13T16:22:56.303 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-packaging (20.3-1) ... 2024-09-13T16:22:56.372 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-13T16:22:56.596 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2024-09-13T16:22:56.665 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-13T16:22:56.722 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up qttranslations5-l10n (5.12.8-0ubuntu1) ... 2024-09-13T16:22:56.767 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T16:22:56.873 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2024-09-13T16:22:57.051 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-13T16:22:57.166 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2024-09-13T16:22:57.352 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-13T16:22:57.626 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2024-09-13T16:22:57.661 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-13T16:22:57.910 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2ubuntu0.2) ... 2024-09-13T16:22:58.074 INFO:teuthology.orchestra.run.smithi119.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-13T16:22:58.278 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cryptography (2.8-3ubuntu0.3) ... 2024-09-13T16:22:58.331 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-13T16:22:58.440 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-13T16:22:58.695 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up socat (1.7.3.3-2) ... 2024-09-13T16:22:58.750 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-13T16:22:58.823 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:22:59.110 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-13T16:22:59.149 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqt5core5a:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:22:59.235 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-13T16:22:59.292 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-html5lib (1.0.1-2) ... 2024-09-13T16:22:59.362 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-13T16:22:59.635 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2024-09-13T16:22:59.688 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-13T16:22:59.761 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2024-09-13T16:22:59.921 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pluggy (0.13.0-2) ... 2024-09-13T16:22:59.981 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-13T16:23:00.213 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2024-09-13T16:23:00.540 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqt5dbus5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:23:00.666 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblua5.3-0:amd64 (5.3.3-1.1ubuntu2) ... 2024-09-13T16:23:00.792 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2024-09-13T16:23:01.144 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2024-09-13T16:23:01.411 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:23:01.420 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-logutils (0.3.3-7) ... 2024-09-13T16:23:01.537 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-13T16:23:01.705 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2024-09-13T16:23:01.831 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-13T16:23:01.922 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-13T16:23:02.115 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-prettytable (0.7.2-5) ... 2024-09-13T16:23:02.240 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-13T16:23:02.408 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2024-09-13T16:23:02.516 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:23:02.642 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-13T16:23:02.726 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2024-09-13T16:23:02.985 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-13T16:23:03.065 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2024-09-13T16:23:03.329 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-13T16:23:03.359 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2024-09-13T16:23:03.463 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2.1) ... 2024-09-13T16:23:03.722 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-13T16:23:03.789 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2024-09-13T16:23:04.057 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-13T16:23:04.149 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2024-09-13T16:23:04.275 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2024-09-13T16:23:04.427 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webob (1:1.8.5-2ubuntu0.1) ... 2024-09-13T16:23:04.786 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2024-09-13T16:23:05.062 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2024-09-13T16:23:05.095 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-13T16:23:05.246 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-13T16:23:05.372 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:23:05.482 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-13T16:23:05.758 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-13T16:23:05.884 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-13T16:23:06.194 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-13T16:23:06.493 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:23:06.503 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T16:23:06.652 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pytest (4.6.9-1) ... 2024-09-13T16:23:06.821 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-13T16:23:06.987 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2024-09-13T16:23:07.182 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librados2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:07.297 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2024-09-13T16:23:07.385 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-13T16:23:07.536 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librgw2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:07.590 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqt5network5:amd64 (5.12.8+dfsg-0ubuntu2.1) ... 2024-09-13T16:23:07.662 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libsqlite3-mod-ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:07.740 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-bs4 (4.8.2-1) ... 2024-09-13T16:23:07.788 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libcephfs2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:07.914 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-13T16:23:08.058 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2024-09-13T16:23:08.224 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libradosstriper1 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:08.325 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-13T16:23:08.377 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-asyncssh (1.12.2-1) ... 2024-09-13T16:23:08.601 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up librbd1 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:08.727 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-modules-core (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:08.754 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2024-09-13T16:23:08.853 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:09.079 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T16:23:09.080 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T16:23:09.088 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2024-09-13T16:23:09.496 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libcephfs-dev (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:09.647 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rados (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:09.773 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up cephfs-top (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:09.874 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-13T16:23:10.127 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2024-09-13T16:23:10.253 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-natsort-doc (7.0.1-1) ... 2024-09-13T16:23:10.379 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2024-09-13T16:23:10.538 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rsa (4.0-3) ... 2024-09-13T16:23:10.770 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rbd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:10.831 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libthrift-0.13.0 (0.13.0-2build2) ... 2024-09-13T16:23:10.895 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up rbd-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:10.940 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-webtest (2.0.32-2) ... 2024-09-13T16:23:11.022 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-rgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:11.147 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up python3-cephfs (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:11.233 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2024-09-13T16:23:11.274 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up cephfs-shell (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:11.392 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:11.526 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-13T16:23:11.894 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2024-09-13T16:23:11.936 INFO:teuthology.orchestra.run.smithi119.stdout:Adding group ceph....done 2024-09-13T16:23:12.263 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librados2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:12.306 INFO:teuthology.orchestra.run.smithi119.stdout:Adding system user ceph....done 2024-09-13T16:23:12.474 INFO:teuthology.orchestra.run.smithi119.stdout:Setting system user ceph properties....done 2024-09-13T16:23:12.475 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python-asyncssh-doc (1.12.2-1) ... 2024-09-13T16:23:12.498 INFO:teuthology.orchestra.run.smithi119.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T16:23:12.642 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librgw2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:12.649 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T16:23:12.768 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libsqlite3-mod-ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:12.894 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libcephfs2 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:12.954 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T16:23:13.054 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-routes (2.4.1-2) ... 2024-09-13T16:23:13.380 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libradosstriper1 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:13.437 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up cephfs-mirror (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:13.506 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-google-auth (1.5.1-2) ... 2024-09-13T16:23:13.654 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:23:13.654 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:23:13.800 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up librbd1 (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:13.926 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-modules-core (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:14.052 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:14.097 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-test (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:14.223 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up radosgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:14.310 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T16:23:14.310 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-13T16:23:14.704 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:23:14.705 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:23:14.787 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libcephfs-dev (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:14.963 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rados (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:15.089 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephfs-top (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:15.141 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-base (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:15.240 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2024-09-13T16:23:15.536 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T16:23:15.943 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mds (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:16.136 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rbd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:16.164 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T16:23:16.164 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T16:23:16.295 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up rbd-fuse (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:16.438 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-rgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:16.564 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up python3-cephfs (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:16.578 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:16.715 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephfs-shell (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:16.809 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T16:23:16.809 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T16:23:16.824 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-common (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:17.271 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-osd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:17.435 INFO:teuthology.orchestra.run.smithi145.stdout:Adding group ceph....done 2024-09-13T16:23:17.780 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T16:23:17.780 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T16:23:17.829 INFO:teuthology.orchestra.run.smithi145.stdout:Adding system user ceph....done 2024-09-13T16:23:17.988 INFO:teuthology.orchestra.run.smithi145.stdout:Setting system user ceph properties....done 2024-09-13T16:23:18.012 INFO:teuthology.orchestra.run.smithi145.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-13T16:23:18.164 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-13T16:23:18.189 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-k8sevents (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:18.331 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:18.475 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mon (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:18.488 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-13T16:23:18.704 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T16:23:18.704 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T16:23:18.951 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up cephfs-mirror (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:19.126 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:19.180 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:23:19.180 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target -> /lib/systemd/system/cephfs-mirror.target. 2024-09-13T16:23:19.269 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:19.420 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-mgr-dashboard (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:19.644 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-test (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:19.757 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ceph-volume (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:19.772 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up radosgw (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:20.117 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-13T16:23:20.260 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-13T16:23:20.279 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:23:20.279 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-13T16:23:20.527 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-13T16:23:20.739 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-base (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:21.115 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-13T16:23:21.565 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mds (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:21.843 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T16:23:21.843 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-13T16:23:22.326 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:22.594 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T16:23:22.594 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-13T16:23:23.069 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-osd (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:23.405 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-13T16:23:23.583 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T16:23:23.583 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-13T16:23:24.018 INFO:teuthology.orchestra.run.smithi119.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T16:23:24.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-13T16:23:24.045 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-k8sevents (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:24.097 INFO:teuthology.orchestra.run.smithi119.stdout:Reading package lists... 2024-09-13T16:23:24.195 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-diskprediction-local (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:24.316 INFO:teuthology.orchestra.run.smithi119.stdout:Building dependency tree... 2024-09-13T16:23:24.317 INFO:teuthology.orchestra.run.smithi119.stdout:Reading state information... 2024-09-13T16:23:24.339 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mon (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:24.540 INFO:teuthology.orchestra.run.smithi119.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2024-09-13T16:23:24.540 INFO:teuthology.orchestra.run.smithi119.stdout:flex is already the newest version (2.6.4-6.2). 2024-09-13T16:23:24.540 INFO:teuthology.orchestra.run.smithi119.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2024-09-13T16:23:24.540 INFO:teuthology.orchestra.run.smithi119.stdout:libtool-bin is already the newest version (2.4.6-14). 2024-09-13T16:23:24.540 INFO:teuthology.orchestra.run.smithi119.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2024-09-13T16:23:24.540 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T16:23:24.541 INFO:teuthology.orchestra.run.smithi119.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-13T16:23:24.541 INFO:teuthology.orchestra.run.smithi119.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T16:23:24.541 INFO:teuthology.orchestra.run.smithi119.stdout:The following additional packages will be installed: 2024-09-13T16:23:24.541 INFO:teuthology.orchestra.run.smithi119.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2024-09-13T16:23:24.542 INFO:teuthology.orchestra.run.smithi119.stdout: libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl1.1 2024-09-13T16:23:24.542 INFO:teuthology.orchestra.run.smithi119.stdout: libteamdctl0 libuuid1 modemmanager network-manager-pptp postgresql-12 2024-09-13T16:23:24.542 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-client-12 postgresql-client-common ppp pptp-linux usb-modeswitch 2024-09-13T16:23:24.542 INFO:teuthology.orchestra.run.smithi119.stdout: usb-modeswitch-data wpasupplicant 2024-09-13T16:23:24.543 INFO:teuthology.orchestra.run.smithi119.stdout:Suggested packages: 2024-09-13T16:23:24.543 INFO:teuthology.orchestra.run.smithi119.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils postgresql-doc 2024-09-13T16:23:24.544 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-doc-12 libjson-perl comgt wvdial util-linux-locales wpagui 2024-09-13T16:23:24.544 INFO:teuthology.orchestra.run.smithi119.stdout: libengine-pkcs11-openssl 2024-09-13T16:23:24.567 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T16:23:24.567 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-13T16:23:24.605 INFO:teuthology.orchestra.run.smithi119.stdout:The following NEW packages will be installed: 2024-09-13T16:23:24.606 INFO:teuthology.orchestra.run.smithi119.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2024-09-13T16:23:24.606 INFO:teuthology.orchestra.run.smithi119.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy 2024-09-13T16:23:24.606 INFO:teuthology.orchestra.run.smithi119.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2024-09-13T16:23:24.606 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql postgresql-12 postgresql-client postgresql-client-12 2024-09-13T16:23:24.607 INFO:teuthology.orchestra.run.smithi119.stdout: postgresql-client-common postgresql-common postgresql-contrib ppp pptp-linux 2024-09-13T16:23:24.607 INFO:teuthology.orchestra.run.smithi119.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-13T16:23:24.607 INFO:teuthology.orchestra.run.smithi119.stdout:The following packages will be upgraded: 2024-09-13T16:23:24.609 INFO:teuthology.orchestra.run.smithi119.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2024-09-13T16:23:24.666 INFO:teuthology.orchestra.run.smithi119.stdout:6 upgraded, 31 newly installed, 0 to remove and 289 not upgraded. 2024-09-13T16:23:24.666 INFO:teuthology.orchestra.run.smithi119.stdout:Need to get 25.4 MB of archives. 2024-09-13T16:23:24.666 INFO:teuthology.orchestra.run.smithi119.stdout:After this operation, 80.0 MB of additional disk space will be used. 2024-09-13T16:23:24.666 INFO:teuthology.orchestra.run.smithi119.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.6 [1,022 kB] 2024-09-13T16:23:24.799 INFO:teuthology.orchestra.run.smithi119.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.6 [33.6 kB] 2024-09-13T16:23:24.800 INFO:teuthology.orchestra.run.smithi119.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.6 [20.0 kB] 2024-09-13T16:23:24.801 INFO:teuthology.orchestra.run.smithi119.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2024-09-13T16:23:24.807 INFO:teuthology.orchestra.run.smithi119.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2024-09-13T16:23:24.808 INFO:teuthology.orchestra.run.smithi119.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.23 [1,323 kB] 2024-09-13T16:23:24.824 INFO:teuthology.orchestra.run.smithi119.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2024-09-13T16:23:24.825 INFO:teuthology.orchestra.run.smithi119.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2024-09-13T16:23:24.826 INFO:teuthology.orchestra.run.smithi119.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2024-09-13T16:23:24.827 INFO:teuthology.orchestra.run.smithi119.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2024-09-13T16:23:24.827 INFO:teuthology.orchestra.run.smithi119.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.8 [60.6 kB] 2024-09-13T16:23:24.833 INFO:teuthology.orchestra.run.smithi119.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2024-09-13T16:23:24.834 INFO:teuthology.orchestra.run.smithi119.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2024-09-13T16:23:24.836 INFO:teuthology.orchestra.run.smithi119.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2024-09-13T16:23:24.836 INFO:teuthology.orchestra.run.smithi119.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2024-09-13T16:23:24.851 INFO:teuthology.orchestra.run.smithi119.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libndp0 amd64 1.7-0ubuntu1.1 [11.1 kB] 2024-09-13T16:23:24.851 INFO:teuthology.orchestra.run.smithi119.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.4 [370 kB] 2024-09-13T16:23:24.857 INFO:teuthology.orchestra.run.smithi119.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpq5 amd64 12.20-0ubuntu0.20.04.1 [117 kB] 2024-09-13T16:23:24.859 INFO:teuthology.orchestra.run.smithi119.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2024-09-13T16:23:24.869 INFO:teuthology.orchestra.run.smithi119.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2024-09-13T16:23:24.869 INFO:teuthology.orchestra.run.smithi119.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.23 [1,586 kB] 2024-09-13T16:23:24.882 INFO:teuthology.orchestra.run.smithi119.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2024-09-13T16:23:24.883 INFO:teuthology.orchestra.run.smithi119.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2024-09-13T16:23:24.891 INFO:teuthology.orchestra.run.smithi119.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.4 [1,184 kB] 2024-09-13T16:23:24.903 INFO:teuthology.orchestra.run.smithi119.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.4 [1,854 kB] 2024-09-13T16:23:24.919 INFO:teuthology.orchestra.run.smithi119.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2024-09-13T16:23:24.928 INFO:teuthology.orchestra.run.smithi119.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2024-09-13T16:23:24.931 INFO:teuthology.orchestra.run.smithi119.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2024-09-13T16:23:24.934 INFO:teuthology.orchestra.run.smithi119.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-client-common all 214ubuntu0.1 [28.2 kB] 2024-09-13T16:23:24.941 INFO:teuthology.orchestra.run.smithi119.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-client-12 amd64 12.20-0ubuntu0.20.04.1 [1,055 kB] 2024-09-13T16:23:24.956 INFO:teuthology.orchestra.run.smithi119.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-common all 214ubuntu0.1 [169 kB] 2024-09-13T16:23:25.007 INFO:teuthology.orchestra.run.smithi119.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-12 amd64 12.20-0ubuntu0.20.04.1 [13.5 MB] 2024-09-13T16:23:25.023 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-cephadm (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:25.204 INFO:teuthology.orchestra.run.smithi119.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql all 12+214ubuntu0.1 [3,924 B] 2024-09-13T16:23:25.253 INFO:teuthology.orchestra.run.smithi119.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-client all 12+214ubuntu0.1 [3,940 B] 2024-09-13T16:23:25.285 INFO:teuthology.orchestra.run.smithi119.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-contrib all 12+214ubuntu0.1 [3,932 B] 2024-09-13T16:23:25.285 INFO:teuthology.orchestra.run.smithi119.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2024-09-13T16:23:25.296 INFO:teuthology.orchestra.run.smithi119.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2024-09-13T16:23:25.566 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:25.730 INFO:teuthology.orchestra.run.smithi119.stdout:Preconfiguring packages ... 2024-09-13T16:23:25.905 INFO:teuthology.orchestra.run.smithi119.stdout:Fetched 25.4 MB in 1s (36.1 MB/s) 2024-09-13T16:23:25.994 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-mgr-dashboard (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:26.140 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153098 files and directories currently installed.) 2024-09-13T16:23:26.144 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.6_amd64.deb ... 2024-09-13T16:23:26.170 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ceph-volume (18.2.4-850-g50bbe7f3-1focal) ... 2024-09-13T16:23:26.291 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking util-linux (2.34-0.1ubuntu9.6) over (2.34-0.1ubuntu9.3) ... 2024-09-13T16:23:26.548 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-13T16:23:26.699 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-13T16:23:26.993 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-13T16:23:27.398 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up util-linux (2.34-0.1ubuntu9.6) ... 2024-09-13T16:23:28.785 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153098 files and directories currently installed.) 2024-09-13T16:23:28.789 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.6_amd64.deb ... 2024-09-13T16:23:28.914 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.6) over (2.34-0.1ubuntu9.3) ... 2024-09-13T16:23:29.328 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.6_amd64.deb ... 2024-09-13T16:23:29.495 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.6) over (2.34-0.1ubuntu9.3) ... 2024-09-13T16:23:29.822 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.6) ... 2024-09-13T16:23:29.853 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for fontconfig (2.13.1-2ubuntu3) ... 2024-09-13T16:23:30.293 INFO:teuthology.orchestra.run.smithi119.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153098 files and directories currently installed.) 2024-09-13T16:23:30.298 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2024-09-13T16:23:30.436 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2024-09-13T16:23:30.466 INFO:teuthology.orchestra.run.smithi145.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T16:23:30.469 DEBUG:teuthology.orchestra.run.smithi145:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev network-manager iproute2 util-linux dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev postgresql postgresql-client postgresql-common postgresql-contrib 2024-09-13T16:23:30.543 INFO:teuthology.orchestra.run.smithi145.stdout:Reading package lists... 2024-09-13T16:23:30.763 INFO:teuthology.orchestra.run.smithi145.stdout:Building dependency tree... 2024-09-13T16:23:30.764 INFO:teuthology.orchestra.run.smithi145.stdout:Reading state information... 2024-09-13T16:23:30.986 INFO:teuthology.orchestra.run.smithi145.stdout:bison is already the newest version (2:3.5.1+dfsg-1). 2024-09-13T16:23:30.986 INFO:teuthology.orchestra.run.smithi145.stdout:flex is already the newest version (2.6.4-6.2). 2024-09-13T16:23:30.986 INFO:teuthology.orchestra.run.smithi145.stdout:iproute2 is already the newest version (5.5.0-1ubuntu1). 2024-09-13T16:23:30.986 INFO:teuthology.orchestra.run.smithi145.stdout:libtool-bin is already the newest version (2.4.6-14). 2024-09-13T16:23:30.986 INFO:teuthology.orchestra.run.smithi145.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2024-09-13T16:23:30.986 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages were automatically installed and are no longer required: 2024-09-13T16:23:30.987 INFO:teuthology.orchestra.run.smithi145.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2024-09-13T16:23:30.987 INFO:teuthology.orchestra.run.smithi145.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-13T16:23:30.987 INFO:teuthology.orchestra.run.smithi145.stdout:The following additional packages will be installed: 2024-09-13T16:23:30.987 INFO:teuthology.orchestra.run.smithi145.stdout: libbluetooth3 libdw1 libelf1 liblzo2-2 libmbim-glib4 libmbim-proxy 2024-09-13T16:23:30.988 INFO:teuthology.orchestra.run.smithi145.stdout: libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy libssl1.1 2024-09-13T16:23:30.988 INFO:teuthology.orchestra.run.smithi145.stdout: libteamdctl0 libuuid1 modemmanager network-manager-pptp postgresql-12 2024-09-13T16:23:30.988 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-client-12 postgresql-client-common ppp pptp-linux usb-modeswitch 2024-09-13T16:23:30.988 INFO:teuthology.orchestra.run.smithi145.stdout: usb-modeswitch-data wpasupplicant 2024-09-13T16:23:30.990 INFO:teuthology.orchestra.run.smithi145.stdout:Suggested packages: 2024-09-13T16:23:30.990 INFO:teuthology.orchestra.run.smithi145.stdout: indent-doc libssl-doc avahi-autoipd libteam-utils postgresql-doc 2024-09-13T16:23:30.990 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-doc-12 libjson-perl comgt wvdial util-linux-locales wpagui 2024-09-13T16:23:30.990 INFO:teuthology.orchestra.run.smithi145.stdout: libengine-pkcs11-openssl 2024-09-13T16:23:31.052 INFO:teuthology.orchestra.run.smithi145.stdout:The following NEW packages will be installed: 2024-09-13T16:23:31.053 INFO:teuthology.orchestra.run.smithi145.stdout: dump indent libaio-dev libbluetooth3 libelf-dev liblzo2-2 libmbim-glib4 2024-09-13T16:23:31.053 INFO:teuthology.orchestra.run.smithi145.stdout: libmbim-proxy libmm-glib0 libndp0 libnm0 libpq5 libqmi-glib5 libqmi-proxy 2024-09-13T16:23:31.053 INFO:teuthology.orchestra.run.smithi145.stdout: libssl-dev libteamdctl0 modemmanager network-manager network-manager-pptp 2024-09-13T16:23:31.053 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql postgresql-12 postgresql-client postgresql-client-12 2024-09-13T16:23:31.053 INFO:teuthology.orchestra.run.smithi145.stdout: postgresql-client-common postgresql-common postgresql-contrib ppp pptp-linux 2024-09-13T16:23:31.054 INFO:teuthology.orchestra.run.smithi145.stdout: usb-modeswitch usb-modeswitch-data wpasupplicant 2024-09-13T16:23:31.054 INFO:teuthology.orchestra.run.smithi145.stdout:The following packages will be upgraded: 2024-09-13T16:23:31.055 INFO:teuthology.orchestra.run.smithi145.stdout: libdw1 libelf1 libssl1.1 libuuid1 util-linux uuid-dev 2024-09-13T16:23:31.240 INFO:teuthology.orchestra.run.smithi145.stdout:6 upgraded, 31 newly installed, 0 to remove and 289 not upgraded. 2024-09-13T16:23:31.240 INFO:teuthology.orchestra.run.smithi145.stdout:Need to get 25.4 MB of archives. 2024-09-13T16:23:31.241 INFO:teuthology.orchestra.run.smithi145.stdout:After this operation, 80.0 MB of additional disk space will be used. 2024-09-13T16:23:31.241 INFO:teuthology.orchestra.run.smithi145.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 util-linux amd64 2.34-0.1ubuntu9.6 [1,022 kB] 2024-09-13T16:23:31.854 INFO:teuthology.orchestra.run.smithi145.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 uuid-dev amd64 2.34-0.1ubuntu9.6 [33.6 kB] 2024-09-13T16:23:31.856 INFO:teuthology.orchestra.run.smithi145.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libuuid1 amd64 2.34-0.1ubuntu9.6 [20.0 kB] 2024-09-13T16:23:31.857 INFO:teuthology.orchestra.run.smithi145.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libdw1 amd64 0.176-1.1ubuntu0.1 [226 kB] 2024-09-13T16:23:31.870 INFO:teuthology.orchestra.run.smithi145.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf1 amd64 0.176-1.1ubuntu0.1 [44.2 kB] 2024-09-13T16:23:31.872 INFO:teuthology.orchestra.run.smithi145.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl1.1 amd64 1.1.1f-1ubuntu2.23 [1,323 kB] 2024-09-13T16:23:31.910 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2024-09-13T16:23:31.969 INFO:teuthology.orchestra.run.smithi145.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2024-09-13T16:23:31.972 INFO:teuthology.orchestra.run.smithi145.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2024-09-13T16:23:31.979 INFO:teuthology.orchestra.run.smithi145.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 indent amd64 2.2.12-1ubuntu0.20.04.1 [86.6 kB] 2024-09-13T16:23:31.984 INFO:teuthology.orchestra.run.smithi145.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2024-09-13T16:23:31.984 INFO:teuthology.orchestra.run.smithi145.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libbluetooth3 amd64 5.53-0ubuntu3.8 [60.6 kB] 2024-09-13T16:23:32.017 INFO:teuthology.orchestra.run.smithi145.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libelf-dev amd64 0.176-1.1ubuntu0.1 [57.1 kB] 2024-09-13T16:23:32.019 INFO:teuthology.orchestra.run.smithi145.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-glib4 amd64 1.26.2-1~ubuntu20.04.1 [123 kB] 2024-09-13T16:23:32.022 INFO:teuthology.orchestra.run.smithi145.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmbim-proxy amd64 1.26.2-1~ubuntu20.04.1 [6,012 B] 2024-09-13T16:23:32.022 INFO:teuthology.orchestra.run.smithi145.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libmm-glib0 amd64 1.18.6-1~ubuntu20.04.1 [203 kB] 2024-09-13T16:23:32.026 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2024-09-13T16:23:32.099 INFO:teuthology.orchestra.run.smithi145.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libndp0 amd64 1.7-0ubuntu1.1 [11.1 kB] 2024-09-13T16:23:32.099 INFO:teuthology.orchestra.run.smithi145.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libnm0 amd64 1.22.10-1ubuntu2.4 [370 kB] 2024-09-13T16:23:32.109 INFO:teuthology.orchestra.run.smithi145.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libpq5 amd64 12.20-0ubuntu0.20.04.1 [117 kB] 2024-09-13T16:23:32.112 INFO:teuthology.orchestra.run.smithi145.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-glib5 amd64 1.30.4-1~ubuntu20.04.1 [642 kB] 2024-09-13T16:23:32.128 INFO:teuthology.orchestra.run.smithi145.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libqmi-proxy amd64 1.30.4-1~ubuntu20.04.1 [5,908 B] 2024-09-13T16:23:32.128 INFO:teuthology.orchestra.run.smithi145.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.23 [1,586 kB] 2024-09-13T16:23:32.180 INFO:teuthology.orchestra.run.smithi145.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 libteamdctl0 amd64 1.30-1 [11.8 kB] 2024-09-13T16:23:32.180 INFO:teuthology.orchestra.run.smithi145.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 modemmanager amd64 1.18.6-1~ubuntu20.04.1 [895 kB] 2024-09-13T16:23:32.203 INFO:teuthology.orchestra.run.smithi145.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 wpasupplicant amd64 2:2.9-1ubuntu4.4 [1,184 kB] 2024-09-13T16:23:32.232 INFO:teuthology.orchestra.run.smithi145.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 network-manager amd64 1.22.10-1ubuntu2.4 [1,854 kB] 2024-09-13T16:23:32.289 INFO:teuthology.orchestra.run.smithi145.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 ppp amd64 2.4.7-2+4.1ubuntu5.1 [328 kB] 2024-09-13T16:23:32.299 INFO:teuthology.orchestra.run.smithi145.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 pptp-linux amd64 1.10.0-1build1 [42.0 kB] 2024-09-13T16:23:32.299 INFO:teuthology.orchestra.run.smithi145.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 network-manager-pptp amd64 1.2.8-2 [30.1 kB] 2024-09-13T16:23:32.300 INFO:teuthology.orchestra.run.smithi145.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-client-common all 214ubuntu0.1 [28.2 kB] 2024-09-13T16:23:32.300 INFO:teuthology.orchestra.run.smithi145.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-client-12 amd64 12.20-0ubuntu0.20.04.1 [1,055 kB] 2024-09-13T16:23:32.327 INFO:teuthology.orchestra.run.smithi145.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-common all 214ubuntu0.1 [169 kB] 2024-09-13T16:23:32.343 INFO:teuthology.orchestra.run.smithi145.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-12 amd64 12.20-0ubuntu0.20.04.1 [13.5 MB] 2024-09-13T16:23:32.485 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.23_amd64.deb ... 2024-09-13T16:23:32.622 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.23) over (1.1.1f-1ubuntu2.16) ... 2024-09-13T16:23:32.730 INFO:teuthology.orchestra.run.smithi145.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql all 12+214ubuntu0.1 [3,924 B] 2024-09-13T16:23:32.807 INFO:teuthology.orchestra.run.smithi145.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-client all 12+214ubuntu0.1 [3,940 B] 2024-09-13T16:23:32.807 INFO:teuthology.orchestra.run.smithi145.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 postgresql-contrib all 12+214ubuntu0.1 [3,932 B] 2024-09-13T16:23:32.807 INFO:teuthology.orchestra.run.smithi145.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch-data all 20191128-3 [32.3 kB] 2024-09-13T16:23:32.808 INFO:teuthology.orchestra.run.smithi145.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 usb-modeswitch amd64 2.5.2+repack0-2ubuntu3 [53.1 kB] 2024-09-13T16:23:33.127 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-13T16:23:33.146 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2024-09-13T16:23:33.185 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2024-09-13T16:23:33.213 INFO:teuthology.orchestra.run.smithi145.stdout:Preconfiguring packages ... 2024-09-13T16:23:33.405 INFO:teuthology.orchestra.run.smithi145.stdout:Fetched 25.4 MB in 2s (14.5 MB/s) 2024-09-13T16:23:33.563 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package dump. 2024-09-13T16:23:33.582 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2024-09-13T16:23:33.621 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking dump (0.4b46-6) ... 2024-09-13T16:23:33.712 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153098 files and directories currently installed.) 2024-09-13T16:23:33.717 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../util-linux_2.34-0.1ubuntu9.6_amd64.deb ... 2024-09-13T16:23:33.881 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking util-linux (2.34-0.1ubuntu9.6) over (2.34-0.1ubuntu9.3) ... 2024-09-13T16:23:33.925 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package indent. 2024-09-13T16:23:33.943 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:33.983 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2024-09-13T16:23:34.286 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-13T16:23:34.303 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2024-09-13T16:23:34.345 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2024-09-13T16:23:34.698 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-13T16:23:34.715 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.8_amd64.deb ... 2024-09-13T16:23:34.757 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.8) ... 2024-09-13T16:23:35.018 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-13T16:23:35.035 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2024-09-13T16:23:35.076 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:23:35.098 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up util-linux (2.34-0.1ubuntu9.6) ... 2024-09-13T16:23:35.429 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-13T16:23:35.446 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:35.487 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:23:35.789 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-13T16:23:35.806 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:35.847 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:23:36.192 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-13T16:23:36.210 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:36.250 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:23:36.603 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-13T16:23:36.620 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1.1_amd64.deb ... 2024-09-13T16:23:36.662 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1.1) ... 2024-09-13T16:23:36.788 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153098 files and directories currently installed.) 2024-09-13T16:23:36.792 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../uuid-dev_2.34-0.1ubuntu9.6_amd64.deb ... 2024-09-13T16:23:36.940 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking uuid-dev:amd64 (2.34-0.1ubuntu9.6) over (2.34-0.1ubuntu9.3) ... 2024-09-13T16:23:37.396 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../libuuid1_2.34-0.1ubuntu9.6_amd64.deb ... 2024-09-13T16:23:37.563 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libuuid1:amd64 (2.34-0.1ubuntu9.6) over (2.34-0.1ubuntu9.3) ... 2024-09-13T16:23:37.767 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-13T16:23:37.783 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.4_amd64.deb ... 2024-09-13T16:23:37.825 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.4) ... 2024-09-13T16:23:37.914 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libuuid1:amd64 (2.34-0.1ubuntu9.6) ... 2024-09-13T16:23:38.228 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-13T16:23:38.244 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../14-libpq5_12.20-0ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:38.286 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libpq5:amd64 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:38.384 INFO:teuthology.orchestra.run.smithi145.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 153098 files and directories currently installed.) 2024-09-13T16:23:38.388 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../00-libdw1_0.176-1.1ubuntu0.1_amd64.deb ... 2024-09-13T16:23:38.521 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libdw1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2024-09-13T16:23:38.647 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-13T16:23:38.664 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../15-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:38.705 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:23:38.986 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../01-libelf1_0.176-1.1ubuntu0.1_amd64.deb ... 2024-09-13T16:23:39.058 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-13T16:23:39.077 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../16-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:39.117 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:23:39.160 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libelf1:amd64 (0.176-1.1ubuntu0.1) over (0.176-1.1build1) ... 2024-09-13T16:23:39.386 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-13T16:23:39.403 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../17-libssl-dev_1.1.1f-1ubuntu2.23_amd64.deb ... 2024-09-13T16:23:39.444 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.23) ... 2024-09-13T16:23:39.607 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../02-libssl1.1_1.1.1f-1ubuntu2.23_amd64.deb ... 2024-09-13T16:23:39.731 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libssl1.1:amd64 (1.1.1f-1ubuntu2.23) over (1.1.1f-1ubuntu2.16) ... 2024-09-13T16:23:39.939 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-13T16:23:39.958 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../18-libteamdctl0_1.30-1_amd64.deb ... 2024-09-13T16:23:39.998 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2024-09-13T16:23:40.235 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package liblzo2-2:amd64. 2024-09-13T16:23:40.254 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../03-liblzo2-2_2.10-2_amd64.deb ... 2024-09-13T16:23:40.293 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2024-09-13T16:23:40.384 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package modemmanager. 2024-09-13T16:23:40.405 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../19-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:40.453 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:23:40.648 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package dump. 2024-09-13T16:23:40.665 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../04-dump_0.4b46-6_amd64.deb ... 2024-09-13T16:23:40.704 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking dump (0.4b46-6) ... 2024-09-13T16:23:40.991 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package indent. 2024-09-13T16:23:41.008 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../05-indent_2.2.12-1ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:41.048 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package wpasupplicant. 2024-09-13T16:23:41.049 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking indent (2.2.12-1ubuntu0.20.04.1) ... 2024-09-13T16:23:41.067 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../20-wpasupplicant_2%3a2.9-1ubuntu4.4_amd64.deb ... 2024-09-13T16:23:41.106 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.4) ... 2024-09-13T16:23:41.394 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libaio-dev:amd64. 2024-09-13T16:23:41.412 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../06-libaio-dev_0.3.112-5_amd64.deb ... 2024-09-13T16:23:41.488 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2024-09-13T16:23:41.645 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package network-manager. 2024-09-13T16:23:41.663 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../21-network-manager_1.22.10-1ubuntu2.4_amd64.deb ... 2024-09-13T16:23:41.728 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking network-manager (1.22.10-1ubuntu2.4) ... 2024-09-13T16:23:42.357 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libbluetooth3:amd64. 2024-09-13T16:23:42.377 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../07-libbluetooth3_5.53-0ubuntu3.8_amd64.deb ... 2024-09-13T16:23:42.438 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libbluetooth3:amd64 (5.53-0ubuntu3.8) ... 2024-09-13T16:23:42.477 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package ppp. 2024-09-13T16:23:42.495 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../22-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2024-09-13T16:23:42.710 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libelf-dev:amd64. 2024-09-13T16:23:42.728 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../08-libelf-dev_0.176-1.1ubuntu0.1_amd64.deb ... 2024-09-13T16:23:42.762 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2024-09-13T16:23:42.768 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:23:43.113 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libmbim-glib4:amd64. 2024-09-13T16:23:43.133 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../09-libmbim-glib4_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:43.204 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:23:43.482 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libmbim-proxy. 2024-09-13T16:23:43.499 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../10-libmbim-proxy_1.26.2-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:43.565 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:23:43.584 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package pptp-linux. 2024-09-13T16:23:43.603 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../23-pptp-linux_1.10.0-1build1_amd64.deb ... 2024-09-13T16:23:43.643 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2024-09-13T16:23:43.935 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libmm-glib0:amd64. 2024-09-13T16:23:43.952 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../11-libmm-glib0_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:43.993 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:23:44.004 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-13T16:23:44.025 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../24-network-manager-pptp_1.2.8-2_amd64.deb ... 2024-09-13T16:23:44.080 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2024-09-13T16:23:44.354 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libndp0:amd64. 2024-09-13T16:23:44.374 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../12-libndp0_1.7-0ubuntu1.1_amd64.deb ... 2024-09-13T16:23:44.382 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-13T16:23:44.399 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../25-postgresql-client-common_214ubuntu0.1_all.deb ... 2024-09-13T16:23:44.412 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libndp0:amd64 (1.7-0ubuntu1.1) ... 2024-09-13T16:23:44.465 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking postgresql-client-common (214ubuntu0.1) ... 2024-09-13T16:23:44.762 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package postgresql-client-12. 2024-09-13T16:23:44.779 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../26-postgresql-client-12_12.20-0ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:44.782 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libnm0:amd64. 2024-09-13T16:23:44.799 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../13-libnm0_1.22.10-1ubuntu2.4_amd64.deb ... 2024-09-13T16:23:44.820 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking postgresql-client-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:44.840 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libnm0:amd64 (1.22.10-1ubuntu2.4) ... 2024-09-13T16:23:45.277 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libpq5:amd64. 2024-09-13T16:23:45.295 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../14-libpq5_12.20-0ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:45.360 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libpq5:amd64 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:45.391 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package postgresql-common. 2024-09-13T16:23:45.411 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../27-postgresql-common_214ubuntu0.1_all.deb ... 2024-09-13T16:23:45.474 INFO:teuthology.orchestra.run.smithi119.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-13T16:23:45.518 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking postgresql-common (214ubuntu0.1) ... 2024-09-13T16:23:45.746 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqmi-glib5:amd64. 2024-09-13T16:23:45.763 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../15-libqmi-glib5_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:45.804 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:23:45.981 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package postgresql-12. 2024-09-13T16:23:45.999 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../28-postgresql-12_12.20-0ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:46.043 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking postgresql-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:46.157 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libqmi-proxy. 2024-09-13T16:23:46.177 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../16-libqmi-proxy_1.30.4-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:46.215 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:23:46.459 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libssl-dev:amd64. 2024-09-13T16:23:46.478 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../17-libssl-dev_1.1.1f-1ubuntu2.23_amd64.deb ... 2024-09-13T16:23:46.526 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.23) ... 2024-09-13T16:23:47.054 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package libteamdctl0:amd64. 2024-09-13T16:23:47.074 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../18-libteamdctl0_1.30-1_amd64.deb ... 2024-09-13T16:23:47.250 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking libteamdctl0:amd64 (1.30-1) ... 2024-09-13T16:23:47.551 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package postgresql. 2024-09-13T16:23:47.572 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../29-postgresql_12+214ubuntu0.1_all.deb ... 2024-09-13T16:23:47.609 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking postgresql (12+214ubuntu0.1) ... 2024-09-13T16:23:47.878 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package postgresql-client. 2024-09-13T16:23:47.896 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../30-postgresql-client_12+214ubuntu0.1_all.deb ... 2024-09-13T16:23:47.937 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking postgresql-client (12+214ubuntu0.1) ... 2024-09-13T16:23:48.060 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package modemmanager. 2024-09-13T16:23:48.077 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../19-modemmanager_1.18.6-1~ubuntu20.04.1_amd64.deb ... 2024-09-13T16:23:48.129 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking modemmanager (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:23:48.180 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-13T16:23:48.199 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../31-postgresql-contrib_12+214ubuntu0.1_all.deb ... 2024-09-13T16:23:48.238 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking postgresql-contrib (12+214ubuntu0.1) ... 2024-09-13T16:23:48.508 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-13T16:23:48.528 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../32-usb-modeswitch-data_20191128-3_all.deb ... 2024-09-13T16:23:48.566 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2024-09-13T16:23:48.791 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package wpasupplicant. 2024-09-13T16:23:48.808 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../20-wpasupplicant_2%3a2.9-1ubuntu4.4_amd64.deb ... 2024-09-13T16:23:48.849 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking wpasupplicant (2:2.9-1ubuntu4.4) ... 2024-09-13T16:23:48.978 INFO:teuthology.orchestra.run.smithi119.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-13T16:23:48.998 INFO:teuthology.orchestra.run.smithi119.stdout:Preparing to unpack .../33-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2024-09-13T16:23:49.046 INFO:teuthology.orchestra.run.smithi119.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2024-09-13T16:23:49.347 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up postgresql-client-common (214ubuntu0.1) ... 2024-09-13T16:23:49.404 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package network-manager. 2024-09-13T16:23:49.425 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../21-network-manager_1.22.10-1ubuntu2.4_amd64.deb ... 2024-09-13T16:23:49.519 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking network-manager (1.22.10-1ubuntu2.4) ... 2024-09-13T16:23:49.533 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.23) ... 2024-09-13T16:23:49.818 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libpq5:amd64 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:49.969 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2024-09-13T16:23:50.095 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2024-09-13T16:23:50.221 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.4) ... 2024-09-13T16:23:50.278 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package ppp. 2024-09-13T16:23:50.295 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../22-ppp_2.4.7-2+4.1ubuntu5.1_amd64.deb ... 2024-09-13T16:23:50.577 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking ppp (2.4.7-2+4.1ubuntu5.1) ... 2024-09-13T16:23:50.621 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-13T16:23:50.621 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-13T16:23:50.891 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package pptp-linux. 2024-09-13T16:23:50.910 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../23-pptp-linux_1.10.0-1build1_amd64.deb ... 2024-09-13T16:23:50.974 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking pptp-linux (1.10.0-1build1) ... 2024-09-13T16:23:51.074 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up postgresql-client-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:51.370 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package network-manager-pptp. 2024-09-13T16:23:51.387 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../24-network-manager-pptp_1.2.8-2_amd64.deb ... 2024-09-13T16:23:51.445 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking network-manager-pptp (1.2.8-2) ... 2024-09-13T16:23:51.748 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-client-common. 2024-09-13T16:23:51.768 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../25-postgresql-client-common_214ubuntu0.1_all.deb ... 2024-09-13T16:23:51.806 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-client-common (214ubuntu0.1) ... 2024-09-13T16:23:51.947 INFO:teuthology.orchestra.run.smithi119.stdout:update-alternatives: using /usr/share/postgresql/12/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-13T16:23:52.063 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.6) ... 2024-09-13T16:23:52.135 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-client-12. 2024-09-13T16:23:52.153 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../26-postgresql-client-12_12.20-0ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:52.181 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2024-09-13T16:23:52.194 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-client-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:52.307 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up postgresql-common (214ubuntu0.1) ... 2024-09-13T16:23:52.865 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-common. 2024-09-13T16:23:52.883 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../27-postgresql-common_214ubuntu0.1_all.deb ... 2024-09-13T16:23:52.946 INFO:teuthology.orchestra.run.smithi145.stdout:Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common' 2024-09-13T16:23:52.987 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-common (214ubuntu0.1) ... 2024-09-13T16:23:53.393 INFO:teuthology.orchestra.run.smithi119.stdout:Adding user postgres to group ssl-cert 2024-09-13T16:23:53.501 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-12. 2024-09-13T16:23:53.522 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../28-postgresql-12_12.20-0ubuntu0.20.04.1_amd64.deb ... 2024-09-13T16:23:53.566 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:53.652 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:23:53.652 INFO:teuthology.orchestra.run.smithi119.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-13T16:23:53.918 INFO:teuthology.orchestra.run.smithi119.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-13T16:23:53.918 INFO:teuthology.orchestra.run.smithi119.stdout:Removing obsolete dictionary files: 2024-09-13T16:23:54.256 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-13T16:23:55.059 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.23) ... 2024-09-13T16:23:55.177 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2024-09-13T16:23:55.217 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql. 2024-09-13T16:23:55.233 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../29-postgresql_12+214ubuntu0.1_all.deb ... 2024-09-13T16:23:55.299 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql (12+214ubuntu0.1) ... 2024-09-13T16:23:55.328 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.4) ... 2024-09-13T16:23:55.454 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2024-09-13T16:23:55.535 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-client. 2024-09-13T16:23:55.552 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../30-postgresql-client_12+214ubuntu0.1_all.deb ... 2024-09-13T16:23:55.593 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-client (12+214ubuntu0.1) ... 2024-09-13T16:23:55.862 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package postgresql-contrib. 2024-09-13T16:23:55.880 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../31-postgresql-contrib_12+214ubuntu0.1_all.deb ... 2024-09-13T16:23:55.920 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking postgresql-contrib (12+214ubuntu0.1) ... 2024-09-13T16:23:56.197 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package usb-modeswitch-data. 2024-09-13T16:23:56.215 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../32-usb-modeswitch-data_20191128-3_all.deb ... 2024-09-13T16:23:56.255 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking usb-modeswitch-data (20191128-3) ... 2024-09-13T16:23:56.369 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2024-09-13T16:23:56.593 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up dump (0.4b46-6) ... 2024-09-13T16:23:56.667 INFO:teuthology.orchestra.run.smithi145.stdout:Selecting previously unselected package usb-modeswitch. 2024-09-13T16:23:56.684 INFO:teuthology.orchestra.run.smithi145.stdout:Preparing to unpack .../33-usb-modeswitch_2.5.2+repack0-2ubuntu3_amd64.deb ... 2024-09-13T16:23:56.690 INFO:teuthology.orchestra.run.smithi119.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-13T16:23:56.753 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:23:56.761 INFO:teuthology.orchestra.run.smithi145.stdout:Unpacking usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2024-09-13T16:23:56.879 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.8) ... 2024-09-13T16:23:57.005 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2024-09-13T16:23:57.047 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-client-common (214ubuntu0.1) ... 2024-09-13T16:23:57.131 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2024-09-13T16:23:57.265 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libssl1.1:amd64 (1.1.1f-1ubuntu2.23) ... 2024-09-13T16:23:57.307 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1.1) ... 2024-09-13T16:23:57.433 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up postgresql-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:57.575 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libpq5:amd64 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:23:57.700 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2024-09-13T16:23:57.831 INFO:teuthology.orchestra.run.smithi119.stdout:Creating new PostgreSQL cluster 12/main ... 2024-09-13T16:23:57.847 INFO:teuthology.orchestra.run.smithi119.stdout:/usr/lib/postgresql/12/bin/initdb -D /var/lib/postgresql/12/main --auth-local peer --auth-host md5 2024-09-13T16:23:57.860 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2024-09-13T16:23:57.894 INFO:teuthology.orchestra.run.smithi119.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-13T16:23:57.894 INFO:teuthology.orchestra.run.smithi119.stdout:This user must also own the server process. 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout:The database cluster will be initialized with locale "en_US". 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout:The default text search configuration will be set to "english". 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout:Data page checksums are disabled. 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:23:57.895 INFO:teuthology.orchestra.run.smithi119.stdout:fixing permissions on existing directory /var/lib/postgresql/12/main ... ok 2024-09-13T16:23:57.896 INFO:teuthology.orchestra.run.smithi119.stdout:creating subdirectories ... ok 2024-09-13T16:23:57.896 INFO:teuthology.orchestra.run.smithi119.stdout:selecting dynamic shared memory implementation ... posix 2024-09-13T16:23:57.913 INFO:teuthology.orchestra.run.smithi119.stdout:selecting default max_connections ... 100 2024-09-13T16:23:57.938 INFO:teuthology.orchestra.run.smithi119.stdout:selecting default shared_buffers ... 128MB 2024-09-13T16:23:57.952 INFO:teuthology.orchestra.run.smithi119.stdout:selecting default time zone ... Etc/UTC 2024-09-13T16:23:57.953 INFO:teuthology.orchestra.run.smithi119.stdout:creating configuration files ... ok 2024-09-13T16:23:57.986 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up wpasupplicant (2:2.9-1ubuntu4.4) ... 2024-09-13T16:23:58.063 INFO:teuthology.orchestra.run.smithi119.stdout:running bootstrap script ... ok 2024-09-13T16:23:58.576 INFO:teuthology.orchestra.run.smithi119.stdout:performing post-bootstrap initialization ... ok 2024-09-13T16:23:58.973 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/dbus-fi.w1.wpa_supplicant1.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-13T16:23:58.973 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/wpa_supplicant.service -> /lib/systemd/system/wpa_supplicant.service. 2024-09-13T16:23:59.456 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-client-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:24:00.331 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/share/postgresql/12/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode 2024-09-13T16:24:00.412 INFO:teuthology.orchestra.run.smithi119.stdout:syncing data to disk ... ok 2024-09-13T16:24:00.412 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:00.412 INFO:teuthology.orchestra.run.smithi119.stdout:Success. You can now start the database server using: 2024-09-13T16:24:00.412 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:00.412 INFO:teuthology.orchestra.run.smithi119.stdout: pg_ctlcluster 12 main start 2024-09-13T16:24:00.412 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:00.445 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up uuid-dev:amd64 (2.34-0.1ubuntu9.6) ... 2024-09-13T16:24:00.571 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libteamdctl0:amd64 (1.30-1) ... 2024-09-13T16:24:00.697 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-common (214ubuntu0.1) ... 2024-09-13T16:24:00.724 INFO:teuthology.orchestra.run.smithi119.stdout:Ver Cluster Port Status Owner Data directory Log file 2024-09-13T16:24:00.724 INFO:teuthology.orchestra.run.smithi119.stdout:12 main 5432 down postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log 2024-09-13T16:24:00.806 INFO:teuthology.orchestra.run.smithi119.stdout:update-alternatives: using /usr/share/postgresql/12/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-13T16:24:01.774 INFO:teuthology.orchestra.run.smithi145.stdout:Adding user postgres to group ssl-cert 2024-09-13T16:24:02.076 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:02.076 INFO:teuthology.orchestra.run.smithi145.stdout:Creating config file /etc/postgresql-common/createcluster.conf with new version 2024-09-13T16:24:02.325 INFO:teuthology.orchestra.run.smithi145.stdout:Building PostgreSQL dictionaries from installed myspell/hunspell packages... 2024-09-13T16:24:02.325 INFO:teuthology.orchestra.run.smithi145.stdout:Removing obsolete dictionary files: 2024-09-13T16:24:02.661 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service -> /lib/systemd/system/postgresql.service. 2024-09-13T16:24:03.457 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.23) ... 2024-09-13T16:24:03.585 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up usb-modeswitch-data (20191128-3) ... 2024-09-13T16:24:03.734 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libnm0:amd64 (1.22.10-1ubuntu2.4) ... 2024-09-13T16:24:03.798 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:24:03.860 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up ppp (2.4.7-2+4.1ubuntu5.1) ... 2024-09-13T16:24:03.924 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:24:04.077 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2024-09-13T16:24:04.245 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:24:04.387 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up postgresql-client (12+214ubuntu0.1) ... 2024-09-13T16:24:04.514 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up network-manager (1.22.10-1ubuntu2.4) ... 2024-09-13T16:24:05.187 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:05.187 INFO:teuthology.orchestra.run.smithi119.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-13T16:24:05.187 INFO:teuthology.orchestra.run.smithi119.stdout:which means they are currently configured by ifupdown: 2024-09-13T16:24:05.187 INFO:teuthology.orchestra.run.smithi119.stdout:- enp3s0f1 2024-09-13T16:24:05.188 INFO:teuthology.orchestra.run.smithi119.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-13T16:24:05.188 INFO:teuthology.orchestra.run.smithi119.stdout:remove their configuration from /etc/network/interfaces. 2024-09-13T16:24:05.188 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:05.297 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/pppd-dns.service -> /lib/systemd/system/pppd-dns.service. 2024-09-13T16:24:05.342 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-13T16:24:05.550 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up dump (0.4b46-6) ... 2024-09-13T16:24:05.646 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2024-09-13T16:24:05.687 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-13T16:24:05.719 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libmm-glib0:amd64 (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:24:05.869 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libbluetooth3:amd64 (5.53-0ubuntu3.8) ... 2024-09-13T16:24:05.995 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up indent (2.2.12-1ubuntu0.20.04.1) ... 2024-09-13T16:24:06.121 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up usb-modeswitch (2.5.2+repack0-2ubuntu3) ... 2024-09-13T16:24:06.197 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-13T16:24:06.330 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libndp0:amd64 (1.7-0ubuntu1.1) ... 2024-09-13T16:24:06.456 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-12 (12.20-0ubuntu0.20.04.1) ... 2024-09-13T16:24:06.683 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:24:06.826 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2024-09-13T16:24:06.869 INFO:teuthology.orchestra.run.smithi145.stdout:Creating new PostgreSQL cluster 12/main ... 2024-09-13T16:24:06.888 INFO:teuthology.orchestra.run.smithi145.stdout:/usr/lib/postgresql/12/bin/initdb -D /var/lib/postgresql/12/main --auth-local peer --auth-host md5 2024-09-13T16:24:06.954 INFO:teuthology.orchestra.run.smithi145.stdout:The files belonging to this database system will be owned by user "postgres". 2024-09-13T16:24:06.954 INFO:teuthology.orchestra.run.smithi145.stdout:This user must also own the server process. 2024-09-13T16:24:06.954 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout:The database cluster will be initialized with locale "en_US". 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout:The default database encoding has accordingly been set to "LATIN1". 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout:The default text search configuration will be set to "english". 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout:Data page checksums are disabled. 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout:fixing permissions on existing directory /var/lib/postgresql/12/main ... ok 2024-09-13T16:24:06.955 INFO:teuthology.orchestra.run.smithi145.stdout:creating subdirectories ... ok 2024-09-13T16:24:06.956 INFO:teuthology.orchestra.run.smithi145.stdout:selecting dynamic shared memory implementation ... posix 2024-09-13T16:24:06.969 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:24:06.973 INFO:teuthology.orchestra.run.smithi145.stdout:selecting default max_connections ... 100 2024-09-13T16:24:06.995 INFO:teuthology.orchestra.run.smithi145.stdout:selecting default shared_buffers ... 128MB 2024-09-13T16:24:07.002 INFO:teuthology.orchestra.run.smithi145.stdout:selecting default time zone ... Etc/UTC 2024-09-13T16:24:07.003 INFO:teuthology.orchestra.run.smithi145.stdout:creating configuration files ... ok 2024-09-13T16:24:07.095 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up postgresql-contrib (12+214ubuntu0.1) ... 2024-09-13T16:24:07.109 INFO:teuthology.orchestra.run.smithi145.stdout:running bootstrap script ... ok 2024-09-13T16:24:07.229 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up postgresql (12+214ubuntu0.1) ... 2024-09-13T16:24:07.389 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:24:07.514 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:24:07.640 INFO:teuthology.orchestra.run.smithi119.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:24:07.645 INFO:teuthology.orchestra.run.smithi145.stdout:performing post-bootstrap initialization ... ok 2024-09-13T16:24:07.915 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-13T16:24:07.915 INFO:teuthology.orchestra.run.smithi119.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-13T16:24:08.442 INFO:teuthology.orchestra.run.smithi145.stdout:syncing data to disk ... ok 2024-09-13T16:24:08.443 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:08.443 INFO:teuthology.orchestra.run.smithi145.stdout:Success. You can now start the database server using: 2024-09-13T16:24:08.443 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:08.443 INFO:teuthology.orchestra.run.smithi145.stdout: pg_ctlcluster 12 main start 2024-09-13T16:24:08.443 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:08.444 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2024-09-13T16:24:08.561 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-13T16:24:08.687 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-13T16:24:08.777 INFO:teuthology.orchestra.run.smithi145.stdout:Ver Cluster Port Status Owner Data directory Log file 2024-09-13T16:24:08.778 INFO:teuthology.orchestra.run.smithi145.stdout:12 main 5432 down postgres /var/lib/postgresql/12/main /var/log/postgresql/postgresql-12-main.log 2024-09-13T16:24:08.838 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-13T16:24:08.865 INFO:teuthology.orchestra.run.smithi145.stdout:update-alternatives: using /usr/share/postgresql/12/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode 2024-09-13T16:24:09.140 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-13T16:24:11.980 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libelf1:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:24:12.105 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libmbim-glib4:amd64 (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:24:12.231 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up pptp-linux (1.10.0-1build1) ... 2024-09-13T16:24:12.424 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libdw1:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:24:12.551 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-client (12+214ubuntu0.1) ... 2024-09-13T16:24:12.702 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up network-manager (1.22.10-1ubuntu2.4) ... 2024-09-13T16:24:13.438 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:13.438 INFO:teuthology.orchestra.run.smithi145.stdout:The following network interfaces were found in /etc/network/interfaces 2024-09-13T16:24:13.438 INFO:teuthology.orchestra.run.smithi145.stdout:which means they are currently configured by ifupdown: 2024-09-13T16:24:13.438 INFO:teuthology.orchestra.run.smithi145.stdout:- enp3s0f1 2024-09-13T16:24:13.438 INFO:teuthology.orchestra.run.smithi145.stdout:If you want to manage those interfaces with NetworkManager instead 2024-09-13T16:24:13.438 INFO:teuthology.orchestra.run.smithi145.stdout:remove their configuration from /etc/network/interfaces. 2024-09-13T16:24:13.438 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:13.595 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.nm-dispatcher.service -> /lib/systemd/system/NetworkManager-dispatcher.service. 2024-09-13T16:24:13.931 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/network-online.target.wants/NetworkManager-wait-online.service -> /lib/systemd/system/NetworkManager-wait-online.service. 2024-09-13T16:24:14.488 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/NetworkManager.service -> /lib/systemd/system/NetworkManager.service. 2024-09-13T16:24:14.988 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libmbim-proxy (1.26.2-1~ubuntu20.04.1) ... 2024-09-13T16:24:15.156 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up network-manager-pptp (1.2.8-2) ... 2024-09-13T16:24:15.298 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libelf-dev:amd64 (0.176-1.1ubuntu0.1) ... 2024-09-13T16:24:15.424 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql-contrib (12+214ubuntu0.1) ... 2024-09-13T16:24:15.576 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up postgresql (12+214ubuntu0.1) ... 2024-09-13T16:24:15.702 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqmi-glib5:amd64 (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:24:15.828 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up libqmi-proxy (1.30.4-1~ubuntu20.04.1) ... 2024-09-13T16:24:15.987 INFO:teuthology.orchestra.run.smithi145.stdout:Setting up modemmanager (1.18.6-1~ubuntu20.04.1) ... 2024-09-13T16:24:16.258 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/dbus-org.freedesktop.ModemManager1.service -> /lib/systemd/system/ModemManager.service. 2024-09-13T16:24:16.258 INFO:teuthology.orchestra.run.smithi145.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ModemManager.service -> /lib/systemd/system/ModemManager.service. 2024-09-13T16:24:16.378 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2024-09-13T16:24:16.478 INFO:teuthology.orchestra.run.smithi119.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2024-09-13T16:24:16.790 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for mime-support (3.64ubuntu1) ... 2024-09-13T16:24:16.899 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for hicolor-icon-theme (0.17-2) ... 2024-09-13T16:24:17.058 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2024-09-13T16:24:17.210 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2024-09-13T16:24:17.372 INFO:teuthology.orchestra.run.smithi119.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T16:24:17.377 DEBUG:teuthology.parallel:result is None 2024-09-13T16:24:17.519 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for man-db (2.9.1-1) ... 2024-09-13T16:24:24.649 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for dbus (1.12.16-2ubuntu2.2) ... 2024-09-13T16:24:24.757 INFO:teuthology.orchestra.run.smithi145.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2024-09-13T16:24:25.611 INFO:teuthology.orchestra.run.smithi145.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-13T16:24:25.616 DEBUG:teuthology.parallel:result is None 2024-09-13T16:24:25.616 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:24:25.773 DEBUG:teuthology.orchestra.run.smithi119:> dpkg-query -W -f '${Version}' ceph 2024-09-13T16:24:25.796 INFO:teuthology.orchestra.run.smithi119.stdout:18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:24:25.797 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:24:25.797 INFO:teuthology.task.install:The correct ceph version 18.2.4-850-g50bbe7f3-1focal is installed. 2024-09-13T16:24:25.799 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=50bbe7f395997dcd871813949c49addbbf630361 2024-09-13T16:24:25.943 DEBUG:teuthology.orchestra.run.smithi145:> dpkg-query -W -f '${Version}' ceph 2024-09-13T16:24:25.968 INFO:teuthology.orchestra.run.smithi145.stdout:18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:24:25.968 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-850-g50bbe7f3-1focal 2024-09-13T16:24:25.968 INFO:teuthology.task.install:The correct ceph version 18.2.4-850-g50bbe7f3-1focal is installed. 2024-09-13T16:24:25.970 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-13T16:24:25.970 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:25.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T16:24:25.988 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:25.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-13T16:24:26.027 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-13T16:24:26.027 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:26.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T16:24:26.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T16:24:26.057 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:26.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/daemon-helper 2024-09-13T16:24:26.071 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-13T16:24:26.087 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-13T16:24:26.087 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:26.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T16:24:26.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T16:24:26.161 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:26.161 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-13T16:24:26.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-13T16:24:26.236 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-13T16:24:26.236 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:26.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T16:24:26.252 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T16:24:26.309 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:26.309 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/usr/bin/stdin-killer 2024-09-13T16:24:26.325 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-13T16:24:26.381 INFO:teuthology.run_tasks:Running task ceph... 2024-09-13T16:24:26.460 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-13T16:24:26.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2024-09-13T16:24:26.463 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /var/log/ceph 2024-09-13T16:24:26.480 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-13T16:24:26.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T16:24:26.483 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-13T16:24:26.493 INFO:tasks.ceph:Creating extra log directories... 2024-09-13T16:24:26.493 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T16:24:26.536 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-13T16:24:26.558 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-09-13T16:24:26.558 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-09-13T16:24:26.558 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-09-13T16:24:26.559 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:26.559 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T16:24:26.564 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T16:24:26.599 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:26.599 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-09-13T16:24:26.606 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-09-13T16:24:26.687 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-13T16:24:26.687 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected', 'object missing on disk', 'Invalid tag char'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-13T16:24:26.687 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804', 'branch': 'wip-rishabh-testing-20240910.095512-reef', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub}', 'email': None, 'first_in_suite': False, 'job_id': '7903804', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'wip-rishabh-testing-20240910.095512-reef'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected', 'object missing on disk', 'Invalid tag char'], 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}}, 'subvolume_version': 2, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-rishabh-testing-20240910.095512-reef', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}}, 'owner': 'scheduled_rishabh@teuthology', 'priority': 70, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 1152, 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'sleep_before_teardown': 0, 'subset': '984/1024', 'suite': 'fs', 'suite_branch': 'wip-rishabh-testing-20240910.095512-reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_50bbe7f395997dcd871813949c49addbbf630361/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi119.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM=', 'smithi145.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFyT/r7xuKfmg50w0OdQebr4A/ptJo1Zk6DyZNZNZM/q5rW+sznh+wvHqeT3e4YtCdZLaXyvMms8o8u8H5nXOYM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '50bbe7f395997dcd871813949c49addbbf630361'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'online, but wants', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected', 'object missing on disk', 'Invalid tag char'], 'sha1': '50bbe7f395997dcd871813949c49addbbf630361', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-13_14:36:41', 'tube': 'smithi', 'user': 'rishabh', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-13T16:24:26.688 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T16:24:26.690 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-13T16:24:26.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T16:24:26.740 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m0777 -- /var/run/ceph 2024-09-13T16:24:26.756 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:26.756 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T16:24:26.760 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T16:24:26.761 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2024-09-13T16:24:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T16:24:26.809 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:26.810 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 368 Links: 1 2024-09-13T16:24:26.810 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:26.810 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-13 16:19:54.618647359 +0000 2024-09-13T16:24:26.810 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-13 16:19:54.390652617 +0000 2024-09-13T16:24:26.810 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-13 16:19:54.390652617 +0000 2024-09-13T16:24:26.810 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-09-13T16:24:26.810 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T16:24:26.821 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-13T16:24:26.822 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-13T16:24:26.822 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000434147 s, 1.2 MB/s 2024-09-13T16:24:26.823 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T16:24:26.831 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2024-09-13T16:24:26.838 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T16:24:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:26.839 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 375 Links: 1 2024-09-13T16:24:26.839 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:26.839 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-13 16:19:54.814642839 +0000 2024-09-13T16:24:26.839 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-13 16:19:54.810642931 +0000 2024-09-13T16:24:26.839 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-13 16:19:54.810642931 +0000 2024-09-13T16:24:26.839 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-09-13T16:24:26.839 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T16:24:26.853 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-13T16:24:26.853 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-13T16:24:26.853 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.00029581 s, 1.7 MB/s 2024-09-13T16:24:26.854 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T16:24:26.860 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2024-09-13T16:24:26.909 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T16:24:26.909 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:26.909 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 383 Links: 1 2024-09-13T16:24:26.909 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:26.909 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-13 16:19:55.446628264 +0000 2024-09-13T16:24:26.910 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-13 16:19:55.242632969 +0000 2024-09-13T16:24:26.910 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-13 16:19:55.242632969 +0000 2024-09-13T16:24:26.910 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-09-13T16:24:26.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T16:24:26.920 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-13T16:24:26.921 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-13T16:24:26.921 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000374175 s, 1.4 MB/s 2024-09-13T16:24:26.921 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T16:24:26.930 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2024-09-13T16:24:26.977 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T16:24:26.977 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:26.977 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 390 Links: 1 2024-09-13T16:24:26.977 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:26.978 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2024-09-13 16:19:55.874618394 +0000 2024-09-13T16:24:26.978 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2024-09-13 16:19:55.666623190 +0000 2024-09-13T16:24:26.978 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2024-09-13 16:19:55.666623190 +0000 2024-09-13T16:24:26.978 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2024-09-13T16:24:26.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T16:24:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2024-09-13T16:24:26.988 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2024-09-13T16:24:26.989 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000386359 s, 1.3 MB/s 2024-09-13T16:24:26.990 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T16:24:26.998 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:26.998 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:26.999 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-09-13T16:24:27.005 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-13T16:24:27.006 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 359 Links: 1 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-13 16:19:56.952136884 +0000 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-13 16:19:56.948136975 +0000 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-13 16:19:56.948136975 +0000 2024-09-13T16:24:27.013 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-09-13T16:24:27.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-13T16:24:27.025 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-13T16:24:27.025 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-13T16:24:27.025 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000342191 s, 1.5 MB/s 2024-09-13T16:24:27.026 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-13T16:24:27.033 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-09-13T16:24:27.081 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-13T16:24:27.081 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:27.082 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 366 Links: 1 2024-09-13T16:24:27.082 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:27.082 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-13 16:19:57.596122063 +0000 2024-09-13T16:24:27.082 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-13 16:19:57.372127219 +0000 2024-09-13T16:24:27.082 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-13 16:19:57.372127219 +0000 2024-09-13T16:24:27.082 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-09-13T16:24:27.082 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-13T16:24:27.093 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-13T16:24:27.093 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-13T16:24:27.093 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000404471 s, 1.3 MB/s 2024-09-13T16:24:27.094 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-13T16:24:27.101 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-09-13T16:24:27.149 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-13T16:24:27.149 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:27.150 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 373 Links: 1 2024-09-13T16:24:27.150 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:27.150 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-13 16:19:57.848116265 +0000 2024-09-13T16:24:27.150 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-13 16:19:57.844116356 +0000 2024-09-13T16:24:27.150 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-13 16:19:57.844116356 +0000 2024-09-13T16:24:27.150 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-09-13T16:24:27.150 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-13T16:24:27.161 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-13T16:24:27.161 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-13T16:24:27.161 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000399272 s, 1.3 MB/s 2024-09-13T16:24:27.162 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-13T16:24:27.170 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-09-13T16:24:27.221 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-13T16:24:27.221 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-13T16:24:27.221 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 6h/6d Inode: 380 Links: 1 2024-09-13T16:24:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-13T16:24:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-09-13 16:19:58.264106691 +0000 2024-09-13T16:24:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-09-13 16:19:58.260106784 +0000 2024-09-13T16:24:27.222 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-09-13 16:19:58.260106784 +0000 2024-09-13T16:24:27.222 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: - 2024-09-13T16:24:27.222 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-13T16:24:27.237 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-09-13T16:24:27.237 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-09-13T16:24:27.237 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000374082 s, 1.4 MB/s 2024-09-13T16:24:27.238 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-13T16:24:27.246 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:27.246 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T16:24:27.246 INFO:tasks.ceph:Generating config... 2024-09-13T16:24:27.247 INFO:tasks.ceph:[client] client mount timeout = 600 2024-09-13T16:24:27.247 INFO:tasks.ceph:[client] debug client = 20 2024-09-13T16:24:27.248 INFO:tasks.ceph:[client] debug ms = 1 2024-09-13T16:24:27.248 INFO:tasks.ceph:[client] pid file = 2024-09-13T16:24:27.248 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-09-13T16:24:27.248 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] debug mds = 20 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] debug ms = 1 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] mds debug frag = True 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-09-13T16:24:27.248 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-09-13T16:24:27.249 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-09-13T16:24:27.249 INFO:tasks.ceph:[mgr] debug client = 20 2024-09-13T16:24:27.249 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-13T16:24:27.249 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-13T16:24:27.249 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-13T16:24:27.249 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-13T16:24:27.249 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-13T16:24:27.250 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-09-13T16:24:27.250 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-13T16:24:27.250 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-13T16:24:27.250 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-13T16:24:27.250 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-13T16:24:27.250 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-13T16:24:27.250 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-13T16:24:27.251 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-09-13T16:24:27.251 INFO:tasks.ceph:Setting up mon.a... 2024-09-13T16:24:27.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-13T16:24:27.293 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.keyring 2024-09-13T16:24:27.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-13T16:24:27.350 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T16:24:27.408 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.119'), ('mon.b', '172.21.15.145'), ('mon.c', '[v2:172.21.15.145:3301,v1:172.21.15.145:6790]')] 2024-09-13T16:24:27.408 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon warn on pool no app': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-09-13T16:24:27.409 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:27.409 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T16:24:27.416 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.119 --add b 172.21.15.145 --addv c '[v2:172.21.15.145:3301,v1:172.21.15.145:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:27.460 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: generated fsid 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:setting min_mon_release = pacific 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:epoch 0 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:fsid 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:last_changed 2024-09-13T16:24:27.460973+0000 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:created 2024-09-13T16:24:27.460973+0000 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:min_mon_release 16 (pacific) 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:election_strategy: 1 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:0: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.a 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:1: [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon.b 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:2: [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] mon.c 2024-09-13T16:24:27.461 INFO:teuthology.orchestra.run.smithi119.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-13T16:24:27.462 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-13T16:24:27.469 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 01de39e4-2965-4196-9df7-d7f8aeef36df... 2024-09-13T16:24:27.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:24:27.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:24:27.545 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-09-13T16:24:27.545 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-09-13T16:24:27.545 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:24:27.545 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-09-13T16:24:27.545 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.545 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:24:27.546 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:24:27.547 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:24:27.548 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-09-13T16:24:27.549 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.550 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-09-13T16:24:27.551 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:24:27.552 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.553 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: mon op complaint time = 120 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on pool no app = False 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-09-13T16:24:27.554 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout: client mount timeout = 600 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = "" 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-09-13T16:24:27.555 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout:[mds] 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds = 20 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds balancer = 20 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug frag = True 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug scatterstat = True 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: mds op complaint time = 180 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: mds verify scatter = True 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:24:27.556 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.558 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-09-13T16:24:27.559 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:24:27.560 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-09-13T16:24:27.561 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.562 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-09-13T16:24:27.563 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.564 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:24:27.565 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:24:27.566 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: mon op complaint time = 120 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on pool no app = False 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-09-13T16:24:27.567 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout: client mount timeout = 600 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = "" 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-13T16:24:27.568 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds = 20 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds balancer = 20 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug frag = True 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug scatterstat = True 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: mds op complaint time = 180 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: mds verify scatter = True 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:24:27.569 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:24:27.569 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-13T16:24:27.570 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-13T16:24:27.623 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-13T16:24:27.623 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:27.623 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-13T16:24:27.628 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:27.628 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-13T16:24:27.634 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:24:27.634 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:27.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T16:24:27.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T16:24:27.655 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:27.655 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:27.660 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:24:27.660 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:27.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-13T16:24:27.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-13T16:24:27.687 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:27.687 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:27.737 INFO:tasks.ceph:Setting up mon nodes... 2024-09-13T16:24:27.737 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-13T16:24:27.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T16:24:27.789 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-13T16:24:27.792 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T16:24:27.843 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-13T16:24:27.857 INFO:tasks.ceph:Setting up mds nodes... 2024-09-13T16:24:27.857 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T16:24:27.908 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-09-13T16:24:27.911 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-09-13T16:24:27.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T16:24:27.969 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-09-13T16:24:27.972 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-09-13T16:24:27.982 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T16:24:28.035 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-09-13T16:24:28.038 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-09-13T16:24:28.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T16:24:28.096 INFO:teuthology.orchestra.run.smithi145.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-09-13T16:24:28.099 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-09-13T16:24:28.110 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-13T16:24:28.110 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-13T16:24:28.150 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-13T16:24:28.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-09-13T16:24:28.202 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-09-13T16:24:28.215 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-13T16:24:28.257 INFO:teuthology.orchestra.run.smithi145.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-13T16:24:28.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-09-13T16:24:28.308 INFO:teuthology.orchestra.run.smithi145.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-09-13T16:24:28.322 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-13T16:24:28.322 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi145.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-13T16:24:28.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-13T16:24:28.339 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:28.339 INFO:tasks.ceph:role: osd.0 2024-09-13T16:24:28.339 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:24:28.339 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T16:24:28.840 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:28.840 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:28.840 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:28.840 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2024-09-13T16:24:28.841 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:28.841 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:28.841 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:28.841 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:28.841 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:28.841 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:28.842 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-13T16:24:28.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-13T16:24:28.940 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-13T16:24:28.951 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:28.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:28.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-13T16:24:28.964 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:28.964 INFO:tasks.ceph:role: osd.1 2024-09-13T16:24:28.964 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:24:28.965 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T16:24:29.477 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:29.477 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:29.478 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:29.478 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2024-09-13T16:24:29.478 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:29.478 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:29.478 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:29.478 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:29.479 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:29.479 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:29.480 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-13T16:24:29.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-13T16:24:29.550 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-13T16:24:29.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:29.561 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:29.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-13T16:24:29.575 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:29.575 INFO:tasks.ceph:role: osd.2 2024-09-13T16:24:29.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:24:29.576 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T16:24:30.031 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:30.032 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:30.034 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-13T16:24:30.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-13T16:24:30.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-13T16:24:30.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:30.096 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:30.096 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-13T16:24:30.106 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:30.106 INFO:tasks.ceph:role: osd.3 2024-09-13T16:24:30.106 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:24:30.106 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T16:24:30.642 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:30.643 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:30.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2024-09-13T16:24:30.645 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-13T16:24:30.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-13T16:24:30.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:30.717 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:30.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:30.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:30.792+0000 7f49892fc880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-13T16:24:30.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:30.796+0000 7f49892fc880 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-13T16:24:30.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:30.796+0000 7f49892fc880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T16:24:30.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:30.796+0000 7f49892fc880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T16:24:30.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:30.796+0000 7f49892fc880 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-13T16:24:30.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:30.800+0000 7f49892fc880 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-13T16:24:33.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-13T16:24:33.798 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:33.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:33.840+0000 7f7855c57880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-13T16:24:33.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:33.840+0000 7f7855c57880 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-13T16:24:33.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:33.840+0000 7f7855c57880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T16:24:33.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:33.840+0000 7f7855c57880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T16:24:33.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:33.840+0000 7f7855c57880 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-13T16:24:33.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:33.840+0000 7f7855c57880 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-13T16:24:36.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-13T16:24:36.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:36.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:36.963+0000 7f3b53c16880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-13T16:24:36.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:36.963+0000 7f3b53c16880 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-13T16:24:36.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:36.967+0000 7f3b53c16880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T16:24:36.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:36.967+0000 7f3b53c16880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T16:24:36.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:36.967+0000 7f3b53c16880 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-13T16:24:36.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:36.967+0000 7f3b53c16880 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-13T16:24:39.957 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-13T16:24:39.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:40.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:40.011+0000 7f97e165a880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-13T16:24:40.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:40.011+0000 7f97e165a880 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-13T16:24:40.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:40.011+0000 7f97e165a880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T16:24:40.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:40.011+0000 7f97e165a880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T16:24:40.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:40.011+0000 7f97e165a880 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-13T16:24:40.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:40.015+0000 7f97e165a880 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-13T16:24:43.021 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-13T16:24:43.078 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-13T16:24:43.095 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:43.096 INFO:tasks.ceph:role: osd.4 2024-09-13T16:24:43.096 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:24:43.096 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:43.606 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:43.609 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-09-13T16:24:43.609 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-13T16:24:43.672 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-13T16:24:43.685 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:43.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:43.686 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-13T16:24:43.697 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:43.697 INFO:tasks.ceph:role: osd.5 2024-09-13T16:24:43.697 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:24:43.697 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:44.231 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:44.232 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:44.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-09-13T16:24:44.233 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-13T16:24:44.312 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-13T16:24:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:44.324 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:44.324 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-13T16:24:44.333 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:44.334 INFO:tasks.ceph:role: osd.6 2024-09-13T16:24:44.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:24:44.334 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-13T16:24:44.785 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:44.785 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:44.785 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:44.786 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 2024-09-13T16:24:44.786 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:44.786 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:44.786 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:44.786 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:44.786 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:44.786 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:44.787 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-09-13T16:24:44.788 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-13T16:24:44.830 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-13T16:24:44.840 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:44.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:44.841 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-13T16:24:44.853 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-13T16:24:44.853 INFO:tasks.ceph:role: osd.7 2024-09-13T16:24:44.853 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:24:44.853 DEBUG:teuthology.orchestra.run.smithi145:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-13T16:24:45.337 INFO:teuthology.orchestra.run.smithi145.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-13T16:24:45.337 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-13T16:24:45.337 INFO:teuthology.orchestra.run.smithi145.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-13T16:24:45.337 INFO:teuthology.orchestra.run.smithi145.stdout: = reflink=1 2024-09-13T16:24:45.337 INFO:teuthology.orchestra.run.smithi145.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-13T16:24:45.337 INFO:teuthology.orchestra.run.smithi145.stdout: = sunit=0 swidth=0 blks 2024-09-13T16:24:45.337 INFO:teuthology.orchestra.run.smithi145.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-13T16:24:45.338 INFO:teuthology.orchestra.run.smithi145.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-13T16:24:45.338 INFO:teuthology.orchestra.run.smithi145.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-13T16:24:45.338 INFO:teuthology.orchestra.run.smithi145.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-13T16:24:45.339 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi145.front.sepia.ceph.com -o noatime 2024-09-13T16:24:45.339 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-13T16:24:45.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-13T16:24:45.408 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: /sbin/restorecon: command not found 2024-09-13T16:24:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:24:45.409 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:45.489 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:45.485+0000 7f76609c2880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-13T16:24:45.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:45.489+0000 7f76609c2880 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-13T16:24:45.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:45.489+0000 7f76609c2880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T16:24:45.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:45.489+0000 7f76609c2880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T16:24:45.493 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:45.489+0000 7f76609c2880 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-13T16:24:45.497 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:45.493+0000 7f76609c2880 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-13T16:24:48.495 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-13T16:24:48.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:48.554 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:48.549+0000 7f3836a5b880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-13T16:24:48.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:48.549+0000 7f3836a5b880 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-13T16:24:48.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:48.549+0000 7f3836a5b880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T16:24:48.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:48.549+0000 7f3836a5b880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T16:24:48.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:48.549+0000 7f3836a5b880 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-13T16:24:48.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:48.553+0000 7f3836a5b880 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-13T16:24:51.590 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-13T16:24:51.607 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:51.701 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:51.697+0000 7fb5a63a7880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-13T16:24:51.701 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:51.697+0000 7fb5a63a7880 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-13T16:24:51.701 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:51.697+0000 7fb5a63a7880 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T16:24:51.701 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:51.697+0000 7fb5a63a7880 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T16:24:51.701 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:51.697+0000 7fb5a63a7880 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-13T16:24:51.702 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:51.697+0000 7fb5a63a7880 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-13T16:24:54.725 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-13T16:24:54.738 DEBUG:teuthology.orchestra.run.smithi145:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:54.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:54.773+0000 7f8ee3398880 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-13T16:24:54.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:54.773+0000 7f8ee3398880 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-13T16:24:54.778 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:54.773+0000 7f8ee3398880 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T16:24:54.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:54.773+0000 7f8ee3398880 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T16:24:54.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:54.773+0000 7f8ee3398880 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-13T16:24:54.779 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:24:54.777+0000 7f8ee3398880 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-13T16:24:56.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:24:56.692 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:24:57.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-13T16:24:57.793 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-13T16:24:57.793 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:57.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-13T16:24:57.809 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:57.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-09-13T16:24:57.819 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:57.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-09-13T16:24:57.878 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:57.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-13T16:24:57.888 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:57.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-13T16:24:57.899 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:57.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-13T16:24:57.910 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:57.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-13T16:24:57.922 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:57.923 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-13T16:24:57.937 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:57.937 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-09-13T16:24:57.948 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:57.948 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-09-13T16:24:58.008 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:58.008 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-13T16:24:58.024 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:58.024 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-13T16:24:58.039 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:58.039 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-13T16:24:58.051 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:58.051 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-13T16:24:58.063 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:58.063 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-13T16:24:58.070 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:24:58.070 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-09-13T16:24:58.077 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:58.077 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-13T16:24:58.117 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:24:58.117 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-09-13T16:24:58.123 INFO:tasks.ceph:Adding keys to all mons... 2024-09-13T16:24:58.124 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T16:24:58.126 DEBUG:teuthology.orchestra.run.smithi145:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-13T16:24:58.134 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.y] 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw== 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA== 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ== 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.0] 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA== 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.1] 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw== 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.2] 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ== 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.3] 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww== 2024-09-13T16:24:58.135 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr.x] 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ== 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g== 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw== 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.4] 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA== 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.5] 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q== 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.6] 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg== 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout:[osd.7] 2024-09-13T16:24:58.136 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ== 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout:[client.0] 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ== 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout:[client.2] 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw== 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout:[client.1] 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ== 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout:[client.3] 2024-09-13T16:24:58.137 INFO:teuthology.orchestra.run.smithi145.stdout: key = AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow== 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.y] 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw== 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA== 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ== 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA== 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw== 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2024-09-13T16:24:58.138 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ== 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww== 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ== 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g== 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw== 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA== 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q== 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.6] 2024-09-13T16:24:58.139 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg== 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.7] 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ== 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ== 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout:[client.2] 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw== 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout:[client.1] 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ== 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout:[client.3] 2024-09-13T16:24:58.140 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow== 2024-09-13T16:24:58.141 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:24:58.185 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:24:58.237 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.241 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.293 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.348 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.395 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.443 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.445 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.499 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.548 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:24:58.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-13T16:24:58.601 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.603 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.658 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-09-13T16:24:58.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.712 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.762 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.764 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.812 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.852 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.907 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-13T16:24:58.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:58.996 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:59.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:59.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:59.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:59.111 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:59.162 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:59.164 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-13T16:24:59.215 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-13T16:24:59.216 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-13T16:24:59.231 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T16:24:59.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-13T16:24:59.396 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-13T16:24:59.411 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T16:24:59.501 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-13T16:24:59.516 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-13T16:24:59.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-13T16:24:59.660 DEBUG:teuthology.orchestra.run.smithi145:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-13T16:24:59.675 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:59.678 DEBUG:teuthology.orchestra.run.smithi145:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-13T16:24:59.683 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-13T16:24:59.683 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-13T16:24:59.683 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-13T16:24:59.728 INFO:tasks.ceph.mon.a:Started 2024-09-13T16:24:59.728 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-13T16:24:59.729 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-13T16:24:59.731 INFO:tasks.ceph.mon.b:Started 2024-09-13T16:24:59.731 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-13T16:24:59.731 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-13T16:24:59.734 INFO:tasks.ceph.mon.c:Started 2024-09-13T16:24:59.734 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-13T16:24:59.734 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-13T16:24:59.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-13T16:24:59.736 INFO:tasks.ceph.mgr.y:Started 2024-09-13T16:24:59.736 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-13T16:24:59.736 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-13T16:24:59.739 INFO:tasks.ceph.mgr.x:Started 2024-09-13T16:24:59.739 INFO:tasks.ceph:Setting crush tunables to default 2024-09-13T16:24:59.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd crush tunables default 2024-09-13T16:24:59.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.839+0000 7fab154cf700 1 Processor -- start 2024-09-13T16:24:59.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.839+0000 7fab154cf700 1 -- start start 2024-09-13T16:24:59.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab154cf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:24:59.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab154cf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab101538e0 con 0x7fab10153500 2024-09-13T16:24:59.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab154cf700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab100709e0 con 0x7fab10070e70 2024-09-13T16:24:59.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab154cf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab1006d0d0 con 0x7fab1006c720 2024-09-13T16:24:59.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab0effd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fab10153500 0x7fab100b21f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58000/0 (socket says 172.21.15.119:58000) 2024-09-13T16:24:59.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab0effd700 1 -- 172.21.15.119:0/1102622409 learned_addr learned my addr 172.21.15.119:0/1102622409 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:24:59.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:24:59.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab0e7fc700 1 -- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 msgr2=0x7fab1006cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:24:59.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab0e7fc700 1 -- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 msgr2=0x7fab1006cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:24:59.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:24:59.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:24:59.843+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-13T16:25:00.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.047+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.047+0000 7fab0e7fc700 1 -- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 msgr2=0x7fab1006cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-13T16:25:00.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.047+0000 7fab0e7fc700 1 -- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 msgr2=0x7fab1006cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-13T16:25:00.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.047+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-13T16:25:00.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.047+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-13T16:25:00.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056614023 0 0) 0x7fab100709e0 con 0x7fab10070e70 2024-09-13T16:25:00.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faafc0046e0 con 0x7fab10070e70 2024-09-13T16:25:00.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 <== mon.1 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2297622580 0 0) 0x7faafc0046e0 con 0x7fab10070e70 2024-09-13T16:25:00.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 msgr2=0x7fab1006cb90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:25:00.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 --2- 172.21.15.119:0/1102622409 >> v2:172.21.15.145:3300/0 conn(0x7fab1006c720 0x7fab1006cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 >> v1:172.21.15.119:6789/0 conn(0x7fab10153500 legacy=0x7fab100b21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab100b2900 con 0x7fab10070e70 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 <== mon.1 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fab08001670 con 0x7fab10070e70 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 >> v1:172.21.15.145:6789/0 conn(0x7fab10070e70 legacy=0x7fab100712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc004b30 0x7faafc004f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faafc005980 0x7faafc005e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faafc006460 0x7faafc00cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1006d0d0 con 0x7faafc006460 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faafc00d330 con 0x7faafc004b30 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faafc003650 con 0x7faafc005980 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faafc006460 0x7faafc00cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faafc005980 0x7faafc005e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0effd700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc004b30 0x7faafc004f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 -- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faafc005980 msgr2=0x7faafc005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faafc005980 0x7faafc005e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 -- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc004b30 msgr2=0x7faafc004f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc004b30 0x7faafc004f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 -- 172.21.15.119:0/1102622409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0046e0 con 0x7faafc006460 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 -- 172.21.15.119:0/1102622409 reap_dead start 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0f7fe700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faafc006460 0x7faafc00cb40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab0800c160 tx=0x7faafc0069b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cedff8a4e8fdb363 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08016070 con 0x7faafc006460 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0800cb00 con 0x7faafc006460 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.071+0000 7fab0dffb700 1 -- 172.21.15.119:0/1102622409 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08007b50 con 0x7faafc006460 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.075+0000 7fab154cf700 1 -- 172.21.15.119:0/1102622409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faafc006460 msgr2=0x7faafc00cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.075+0000 7fab154cf700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faafc006460 0x7faafc00cb40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fab0800c160 tx=0x7faafc0069b0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.075+0000 7fab154cf700 1 -- 172.21.15.119:0/1102622409 shutdown_connections 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.075+0000 7fab154cf700 1 --2- 172.21.15.119:0/1102622409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faafc006460 0x7faafc00cb40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.075+0000 7fab154cf700 1 -- 172.21.15.119:0/1102622409 >> 172.21.15.119:0/1102622409 conn(0x7fab1006a350 msgr2=0x7fab1006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.075+0000 7fab154cf700 1 -- 172.21.15.119:0/1102622409 shutdown_connections 2024-09-13T16:25:00.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.075+0000 7fab154cf700 1 -- 172.21.15.119:0/1102622409 wait complete. 2024-09-13T16:25:00.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 Processor -- start 2024-09-13T16:25:00.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 -- start start 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc006460 0x7fab10154800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab10156ab0 0x7fab10151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab1014dbf0 0x7fab1014e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1014e5e0 con 0x7fab1014dbf0 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab10154e90 con 0x7faafc006460 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab10155170 con 0x7fab10156ab0 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab1014dbf0 0x7fab1014e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab1014dbf0 0x7fab1014e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56974/0 (socket says 172.21.15.119:56974) 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 -- 172.21.15.119:0/3620971994 learned_addr learned my addr 172.21.15.119:0/3620971994 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0e7fc700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab10156ab0 0x7fab10151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0effd700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc006460 0x7fab10154800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 -- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab10156ab0 msgr2=0x7fab10151610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab10156ab0 0x7fab10151610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 -- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc006460 msgr2=0x7fab10154800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc006460 0x7fab10154800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 -- 172.21.15.119:0/3620971994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab10155450 con 0x7fab1014dbf0 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0effd700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc006460 0x7fab10154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab0f7fe700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab1014dbf0 0x7fab1014e0a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab08000c00 tx=0x7fab08001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab077fe700 1 -- 172.21.15.119:0/3620971994 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08016040 con 0x7fab1014dbf0 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab077fe700 1 -- 172.21.15.119:0/3620971994 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab080050c0 con 0x7fab1014dbf0 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab10155730 con 0x7fab1014dbf0 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab077fe700 1 -- 172.21.15.119:0/3620971994 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08005450 con 0x7fab1014dbf0 2024-09-13T16:25:00.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab10156f10 con 0x7fab1014dbf0 2024-09-13T16:25:00.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab077fe700 1 -- 172.21.15.119:0/3620971994 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fab080039a0 con 0x7fab1014dbf0 2024-09-13T16:25:00.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.079+0000 7fab077fe700 1 -- 172.21.15.119:0/3620971994 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fab08028870 con 0x7fab1014dbf0 2024-09-13T16:25:00.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.083+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab10067990 con 0x7fab1014dbf0 2024-09-13T16:25:00.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.083+0000 7fab077fe700 1 -- 172.21.15.119:0/3620971994 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fab08007cb0 con 0x7fab1014dbf0 2024-09-13T16:25:00.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.155+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fab1006cb90 con 0x7fab1014dbf0 2024-09-13T16:25:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab077fe700 1 -- 172.21.15.119:0/3620971994 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7fab08003e30 con 0x7fab1014dbf0 2024-09-13T16:25:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab1014dbf0 msgr2=0x7fab1014e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab1014dbf0 0x7fab1014e0a0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fab08000c00 tx=0x7fab08001a00 comp rx=0 tx=0).stop 2024-09-13T16:25:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 shutdown_connections 2024-09-13T16:25:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab1014dbf0 0x7fab1014e0a0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab10156ab0 0x7fab10151610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 --2- 172.21.15.119:0/3620971994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faafc006460 0x7fab10154800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 >> 172.21.15.119:0/3620971994 conn(0x7fab1006a350 msgr2=0x7fab100ae030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 shutdown_connections 2024-09-13T16:25:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:00.159+0000 7fab154cf700 1 -- 172.21.15.119:0/3620971994 wait complete. 2024-09-13T16:25:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:adjusted tunables profile to default 2024-09-13T16:25:00.174 INFO:tasks.ceph:check_enable_crimson: False 2024-09-13T16:25:00.175 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-13T16:25:00.175 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:25:00.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-13T16:25:00.189 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:25:00.189 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-13T16:25:00.199 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:25:00.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-13T16:25:00.208 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:25:00.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-13T16:25:00.222 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:25:00.222 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-13T16:25:00.235 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:25:00.235 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-13T16:25:00.261 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:00.257+0000 7f76c8a432c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T16:25:00.274 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:25:00.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-13T16:25:00.329 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:25:00.329 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-13T16:25:00.338 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new bfde5ac7-b275-4f11-a763-08c219fbfcdd 0 2024-09-13T16:25:00.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.445+0000 7f00c7689700 1 Processor -- start 2024-09-13T16:25:00.449 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.445+0000 7f00c7689700 1 -- start start 2024-09-13T16:25:00.452 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c015a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.452 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f00c015b310 0x7f00c00b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.452 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00c0072030 con 0x7f00c00ce9c0 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00c00cdcd0 con 0x7f00c015b310 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c00ce530 con 0x7f00c00b3b40 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c015a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c4c24700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f00c015b310 0x7f00c00b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c015a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39638/0 (socket says 172.21.15.145:39638) 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c4c24700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f00c015b310 0x7f00c00b3f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:45472/0 (socket says 172.21.15.145:45472) 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 -- 172.21.15.145:0/3305118630 learned_addr learned my addr 172.21.15.145:0/3305118630 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 -- 172.21.15.145:0/3305118630 >> v2:172.21.15.145:3300/0 conn(0x7f00c015b310 msgr2=0x7f00c00b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 --2- 172.21.15.145:0/3305118630 >> v2:172.21.15.145:3300/0 conn(0x7f00c015b310 0x7f00c00b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 -- 172.21.15.145:0/3305118630 >> v1:172.21.15.145:6789/0 conn(0x7f00c00ce9c0 legacy=0x7f00c00cee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 -- 172.21.15.145:0/3305118630 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c015aec0 con 0x7f00c00b3b40 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c4c24700 1 --2- 172.21.15.145:0/3305118630 >> v2:172.21.15.145:3300/0 conn(0x7f00c015b310 0x7f00c00b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:00.453 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c5425700 1 --2- 172.21.15.145:0/3305118630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c015a980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009910 tx=0x7f00bc009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93e84e47a1379a6d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.454 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00b7fff700 1 -- 172.21.15.145:0/3305118630 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc011830 con 0x7f00c00b3b40 2024-09-13T16:25:00.454 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00b7fff700 1 -- 172.21.15.145:0/3305118630 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc011990 con 0x7f00c00b3b40 2024-09-13T16:25:00.454 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00b7fff700 1 -- 172.21.15.145:0/3305118630 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc011d40 con 0x7f00c00b3b40 2024-09-13T16:25:00.454 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- 172.21.15.145:0/3305118630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 msgr2=0x7f00c015a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.454 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 --2- 172.21.15.145:0/3305118630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c015a980 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f00bc009910 tx=0x7f00bc009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- 172.21.15.145:0/3305118630 shutdown_connections 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 --2- 172.21.15.145:0/3305118630 >> v2:172.21.15.145:3300/0 conn(0x7f00c015b310 0x7f00c00b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 --2- 172.21.15.145:0/3305118630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c015a980 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- 172.21.15.145:0/3305118630 >> 172.21.15.145:0/3305118630 conn(0x7f00c0070dc0 msgr2=0x7f00c00711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- 172.21.15.145:0/3305118630 shutdown_connections 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.449+0000 7f00c7689700 1 -- 172.21.15.145:0/3305118630 wait complete. 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 Processor -- start 2024-09-13T16:25:00.455 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 -- start start 2024-09-13T16:25:00.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c01f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.457 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00c00ce9c0 0x7f00c0158e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c015b310 0x7f00c0134db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c0159960 con 0x7f00c00ce9c0 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c01f0f50 con 0x7f00c015b310 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00c0135990 con 0x7f00c00b3b40 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c01f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c01f0a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39640/0 (socket says 172.21.15.145:39640) 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 -- 172.21.15.145:0/101234201 learned_addr learned my addr 172.21.15.145:0/101234201 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5c26700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c015b310 0x7f00c0134db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c4c24700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00c00ce9c0 0x7f00c0158e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 -- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c015b310 msgr2=0x7f00c0134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c015b310 0x7f00c0134db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.458 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 -- 172.21.15.145:0/101234201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00c00ce9c0 msgr2=0x7f00c0158e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00c00ce9c0 0x7f00c0158e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 -- 172.21.15.145:0/101234201 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b00094b0 con 0x7f00c00b3b40 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c5425700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c01f0a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000c00 tx=0x7f00bc004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00b5ffb700 1 -- 172.21.15.145:0/101234201 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc004510 con 0x7f00c00b3b40 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00b5ffb700 1 -- 172.21.15.145:0/101234201 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00bc01e070 con 0x7f00c00b3b40 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc009570 con 0x7f00c00b3b40 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00b5ffb700 1 -- 172.21.15.145:0/101234201 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00bc0038e0 con 0x7f00c00b3b40 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.453+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c0135f50 con 0x7f00c00b3b40 2024-09-13T16:25:00.459 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.457+0000 7f00b5ffb700 1 -- 172.21.15.145:0/101234201 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f00bc003d40 con 0x7f00c00b3b40 2024-09-13T16:25:00.460 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.457+0000 7f00b5ffb700 1 -- 172.21.15.145:0/101234201 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f00bc02dd40 con 0x7f00c00b3b40 2024-09-13T16:25:00.462 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.461+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00c00cc630 con 0x7f00c00b3b40 2024-09-13T16:25:00.466 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.461+0000 7f00b5ffb700 1 -- 172.21.15.145:0/101234201 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f00bc020070 con 0x7f00c00b3b40 2024-09-13T16:25:00.544 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.541+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bfde5ac7-b275-4f11-a763-08c219fbfcdd", "id": 0} v 0) v1 -- 0x7f00c00b4790 con 0x7f00c00b3b40 2024-09-13T16:25:00.553 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.549+0000 7f00b5ffb700 1 -- 172.21.15.145:0/101234201 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bfde5ac7-b275-4f11-a763-08c219fbfcdd", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f00bc015020 con 0x7f00c00b3b40 2024-09-13T16:25:00.553 INFO:teuthology.orchestra.run.smithi145.stdout:0 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.549+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 msgr2=0x7f00c01f0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.549+0000 7f00c7689700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c01f0a10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f00bc000c00 tx=0x7f00bc004770 comp rx=0 tx=0).stop 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.549+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 shutdown_connections 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.549+0000 7f00c7689700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00c015b310 0x7f00c0134db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.549+0000 7f00c7689700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00c00ce9c0 0x7f00c0158e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.549+0000 7f00c7689700 1 --2- 172.21.15.145:0/101234201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00c00b3b40 0x7f00c01f0a10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.553+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 >> 172.21.15.145:0/101234201 conn(0x7f00c0070dc0 msgr2=0x7f00c006a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.555 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.553+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 shutdown_connections 2024-09-13T16:25:00.556 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.553+0000 7f00c7689700 1 -- 172.21.15.145:0/101234201 wait complete. 2024-09-13T16:25:00.567 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new a2427b3c-c6b9-4481-9816-e6b9a0a8ac96 1 2024-09-13T16:25:00.677 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.673+0000 7f91f9fef700 1 Processor -- start 2024-09-13T16:25:00.677 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.673+0000 7f91f9fef700 1 -- start start 2024-09-13T16:25:00.677 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:00.675+0000 7f7f152612c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-13T16:25:00.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f91f4159530 0x7f91f40cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f40ce700 con 0x7f91f40ce110 2024-09-13T16:25:00.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91f40ce9e0 con 0x7f91f415d8d0 2024-09-13T16:25:00.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91f40cecc0 con 0x7f91f4159530 2024-09-13T16:25:00.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f97ee700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f91f415d8d0 0x7f91f4159910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36264/0 (socket says 172.21.15.145:36264) 2024-09-13T16:25:00.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f97ee700 1 -- 172.21.15.145:0/1152362424 learned_addr learned my addr 172.21.15.145:0/1152362424 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-09-13T16:25:00.679 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 --2- 172.21.15.145:0/1152362424 >> v2:172.21.15.119:3300/0 conn(0x7f91f4159530 0x7f91f40cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3960500823 0 0) 0x7f91f40ce9e0 con 0x7f91f415d8d0 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e40035d0 con 0x7f91f415d8d0 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 <== mon.1 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3539177731 0 0) 0x7f91e40035d0 con 0x7f91f415d8d0 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 >> v2:172.21.15.119:3300/0 conn(0x7f91f4159530 msgr2=0x7f91f40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 --2- 172.21.15.145:0/1152362424 >> v2:172.21.15.119:3300/0 conn(0x7f91f4159530 0x7f91f40cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 >> v1:172.21.15.119:6789/0 conn(0x7f91f40ce110 legacy=0x7f91f415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f40cefa0 con 0x7f91f415d8d0 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 <== mon.1 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f91ec0019f0 con 0x7f91f415d8d0 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 >> v1:172.21.15.145:6789/0 conn(0x7f91f415d8d0 legacy=0x7f91f4159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e4003a20 0x7f91e4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.682 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4004870 0x7f91e4004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91e4005230 0x7f91e400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40cecc0 con 0x7f91e4005230 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91e400c100 con 0x7f91e4004870 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e400c2c0 con 0x7f91e4003a20 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e4003a20 0x7f91e4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f3fff700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4004870 0x7f91e4004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f97ee700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91e4005230 0x7f91e400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 -- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4004870 msgr2=0x7f91e4004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4004870 0x7f91e4004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 -- 172.21.15.145:0/1152362424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91e4005230 msgr2=0x7f91e400ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91e4005230 0x7f91e400ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f97ee700 1 -- 172.21.15.145:0/1152362424 reap_dead start 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 -- 172.21.15.145:0/1152362424 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e40035d0 con 0x7f91e4003a20 2024-09-13T16:25:00.683 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f8fed700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e4003a20 0x7f91e4003e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009b70 tx=0x7f91e40058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7cb68f5f785a6f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e800e6c0 con 0x7f91e4003a20 2024-09-13T16:25:00.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e800e820 con 0x7f91e4003a20 2024-09-13T16:25:00.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f37fe700 1 -- 172.21.15.145:0/1152362424 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e800eb90 con 0x7f91e4003a20 2024-09-13T16:25:00.684 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e4003a20 msgr2=0x7f91e4003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e4003a20 0x7f91e4003e40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f91e8009b70 tx=0x7f91e40058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- 172.21.15.145:0/1152362424 shutdown_connections 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 --2- 172.21.15.145:0/1152362424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91e4003a20 0x7f91e4003e40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- 172.21.15.145:0/1152362424 >> 172.21.15.145:0/1152362424 conn(0x7f91f406b460 msgr2=0x7f91f406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- 172.21.15.145:0/1152362424 shutdown_connections 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.677+0000 7f91f9fef700 1 -- 172.21.15.145:0/1152362424 wait complete. 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 Processor -- start 2024-09-13T16:25:00.685 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 -- start start 2024-09-13T16:25:00.687 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4003a20 0x7f91f4155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.687 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91f4151f50 0x7f91f413c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.687 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f40cb0c0 0x7f91f40cb570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.687 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f40cbab0 con 0x7f91f4151f50 2024-09-13T16:25:00.687 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91f4155ed0 con 0x7f91e4003a20 2024-09-13T16:25:00.687 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f9fef700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91f41561b0 con 0x7f91f40cb0c0 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4003a20 0x7f91f4155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4003a20 0x7f91f4155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:45486/0 (socket says 172.21.15.145:45486) 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 -- 172.21.15.145:0/345226995 learned_addr learned my addr 172.21.15.145:0/345226995 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f97ee700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f40cb0c0 0x7f91f40cb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f3fff700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91f4151f50 0x7f91f413c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 -- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f40cb0c0 msgr2=0x7f91f40cb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f40cb0c0 0x7f91f40cb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 -- 172.21.15.145:0/345226995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91f4151f50 msgr2=0x7f91f413c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91f4151f50 0x7f91f413c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f8fed700 1 -- 172.21.15.145:0/345226995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f4156490 con 0x7f91e4003a20 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.681+0000 7f91f3fff700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91f4151f50 0x7f91f413c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f8fed700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4003a20 0x7f91f4155840 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c450 tx=0x7f91e80047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.688 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f17fa700 1 -- 172.21.15.145:0/345226995 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e800e6c0 con 0x7f91e4003a20 2024-09-13T16:25:00.689 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f17fa700 1 -- 172.21.15.145:0/345226995 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e8003af0 con 0x7f91e4003a20 2024-09-13T16:25:00.689 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f17fa700 1 -- 172.21.15.145:0/345226995 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e8003ea0 con 0x7f91e4003a20 2024-09-13T16:25:00.689 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f4156770 con 0x7f91e4003a20 2024-09-13T16:25:00.689 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91f413ca20 con 0x7f91e4003a20 2024-09-13T16:25:00.689 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f17fa700 1 -- 172.21.15.145:0/345226995 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f91e8013770 con 0x7f91e4003a20 2024-09-13T16:25:00.689 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f17fa700 1 -- 172.21.15.145:0/345226995 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f91e801c8f0 con 0x7f91e4003a20 2024-09-13T16:25:00.689 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.685+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91f40cdbd0 con 0x7f91e4003a20 2024-09-13T16:25:00.692 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.689+0000 7f91f17fa700 1 -- 172.21.15.145:0/345226995 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f91e8029070 con 0x7f91e4003a20 2024-09-13T16:25:00.765 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.761+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a2427b3c-c6b9-4481-9816-e6b9a0a8ac96", "id": 1} v 0) v1 -- 0x7f91f40b4d30 con 0x7f91e4003a20 2024-09-13T16:25:00.773 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f17fa700 1 -- 172.21.15.145:0/345226995 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2427b3c-c6b9-4481-9816-e6b9a0a8ac96", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f91e8017070 con 0x7f91e4003a20 2024-09-13T16:25:00.773 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-09-13T16:25:00.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4003a20 msgr2=0x7f91f4155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f9fef700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4003a20 0x7f91f4155840 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f91e400c450 tx=0x7f91e80047e0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 shutdown_connections 2024-09-13T16:25:00.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f9fef700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91f40cb0c0 0x7f91f40cb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f9fef700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91f4151f50 0x7f91f413c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f9fef700 1 --2- 172.21.15.145:0/345226995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91e4003a20 0x7f91f4155840 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.775 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.769+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 >> 172.21.15.145:0/345226995 conn(0x7f91f406b460 msgr2=0x7f91f415bb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.773+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 shutdown_connections 2024-09-13T16:25:00.776 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.773+0000 7f91f9fef700 1 -- 172.21.15.145:0/345226995 wait complete. 2024-09-13T16:25:00.787 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new a6fbd8ac-061c-4d6a-a81e-bd888adca00d 2 2024-09-13T16:25:00.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.889+0000 7f0011a7d700 1 Processor -- start 2024-09-13T16:25:00.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 -- start start 2024-09-13T16:25:00.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f000c0b3200 0x7f000c15d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.898 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c0d0280 0x7f000c06b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000c15ad50 con 0x7f000c15a970 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f000c0721b0 con 0x7f000c0b3200 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c0c9d60 con 0x7f000c0d0280 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c0d0280 0x7f000c06b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000affd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f000c0b3200 0x7f000c15d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c0d0280 0x7f000c06b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39664/0 (socket says 172.21.15.145:39664) 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000bfff700 1 -- 172.21.15.145:0/2523040920 learned_addr learned my addr 172.21.15.145:0/2523040920 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000affd700 1 -- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c0d0280 msgr2=0x7f000c06b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000affd700 1 --2- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c0d0280 0x7f000c06b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000affd700 1 -- 172.21.15.145:0/2523040920 >> v1:172.21.15.119:6789/0 conn(0x7f000c15a970 legacy=0x7f000c0cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000affd700 1 -- 172.21.15.145:0/2523040920 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000c0ca040 con 0x7f000c0b3200 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000bfff700 1 --2- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c0d0280 0x7f000c06b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:00.899 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000affd700 1 --2- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0b3200 0x7f000c15d2f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0000009850 tx=0x7f0000009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5fe7c5995352a5d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000a7fc700 1 -- 172.21.15.145:0/2523040920 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000011830 con 0x7f000c0b3200 2024-09-13T16:25:00.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000a7fc700 1 -- 172.21.15.145:0/2523040920 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0000011990 con 0x7f000c0b3200 2024-09-13T16:25:00.900 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f000a7fc700 1 -- 172.21.15.145:0/2523040920 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000011d40 con 0x7f000c0b3200 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 -- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0b3200 msgr2=0x7f000c15d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.893+0000 7f0011a7d700 1 --2- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0b3200 0x7f000c15d2f0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f0000009850 tx=0x7f0000009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 -- 172.21.15.145:0/2523040920 shutdown_connections 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 --2- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c0d0280 0x7f000c06b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 --2- 172.21.15.145:0/2523040920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0b3200 0x7f000c15d2f0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 -- 172.21.15.145:0/2523040920 >> 172.21.15.145:0/2523040920 conn(0x7f000c0b26e0 msgr2=0x7f000c0b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 -- 172.21.15.145:0/2523040920 shutdown_connections 2024-09-13T16:25:00.901 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 -- 172.21.15.145:0/2523040920 wait complete. 2024-09-13T16:25:00.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 Processor -- start 2024-09-13T16:25:00.902 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.897+0000 7f0011a7d700 1 -- start start 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f000c0b3200 0x7f000c14f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0d0280 0x7f000c06c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c15a970 0x7f000c06c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c156f90 con 0x7f000c0b3200 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000c1e63e0 con 0x7f000c0d0280 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000c1f4030 con 0x7f000c15a970 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c15a970 0x7f000c06c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000affd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0d0280 0x7f000c06c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c15a970 0x7f000c06c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39668/0 (socket says 172.21.15.145:39668) 2024-09-13T16:25:00.905 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 -- 172.21.15.145:0/1528578511 learned_addr learned my addr 172.21.15.145:0/1528578511 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000b7fe700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f000c0b3200 0x7f000c14f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 -- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0d0280 msgr2=0x7f000c06c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0d0280 0x7f000c06c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 -- 172.21.15.145:0/1528578511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f000c0b3200 msgr2=0x7f000c14f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f000c0b3200 0x7f000c14f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 -- 172.21.15.145:0/1528578511 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0004009b30 con 0x7f000c15a970 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000affd700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0d0280 0x7f000c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f000bfff700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c15a970 0x7f000c06c9b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f000400efc0 tx=0x7f000400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7efff3fff700 1 -- 172.21.15.145:0/1528578511 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004018070 con 0x7f000c15a970 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7efff3fff700 1 -- 172.21.15.145:0/1528578511 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000400f040 con 0x7f000c15a970 2024-09-13T16:25:00.906 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00000094b0 con 0x7f000c15a970 2024-09-13T16:25:00.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7efff3fff700 1 -- 172.21.15.145:0/1528578511 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004003c60 con 0x7f000c15a970 2024-09-13T16:25:00.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000c1f3550 con 0x7f000c15a970 2024-09-13T16:25:00.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7efff3fff700 1 -- 172.21.15.145:0/1528578511 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f000400cb80 con 0x7f000c15a970 2024-09-13T16:25:00.907 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7efff3fff700 1 -- 172.21.15.145:0/1528578511 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f0004014c00 con 0x7f000c15a970 2024-09-13T16:25:00.908 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.901+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000c15d2f0 con 0x7f000c15a970 2024-09-13T16:25:00.912 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.909+0000 7efff3fff700 1 -- 172.21.15.145:0/1528578511 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f00040092e0 con 0x7f000c15a970 2024-09-13T16:25:00.985 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.981+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a6fbd8ac-061c-4d6a-a81e-bd888adca00d", "id": 2} v 0) v1 -- 0x7f000c06d450 con 0x7f000c15a970 2024-09-13T16:25:00.996 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7efff3fff700 1 -- 172.21.15.145:0/1528578511 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a6fbd8ac-061c-4d6a-a81e-bd888adca00d", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f000402c0b0 con 0x7f000c15a970 2024-09-13T16:25:00.997 INFO:teuthology.orchestra.run.smithi145.stdout:2 2024-09-13T16:25:00.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c15a970 msgr2=0x7f000c06c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:00.998 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c15a970 0x7f000c06c9b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f000400efc0 tx=0x7f000400c3a0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 shutdown_connections 2024-09-13T16:25:00.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f000c15a970 0x7f000c06c9b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f000c0d0280 0x7f000c06c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 --2- 172.21.15.145:0/1528578511 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f000c0b3200 0x7f000c14f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:00.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 >> 172.21.15.145:0/1528578511 conn(0x7f000c0b26e0 msgr2=0x7f000c15af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:00.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 shutdown_connections 2024-09-13T16:25:00.999 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:00.993+0000 7f0011a7d700 1 -- 172.21.15.145:0/1528578511 wait complete. 2024-09-13T16:25:01.011 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 0ab479b1-9888-4ffd-b4f3-c4c2873accb1 3 2024-09-13T16:25:01.026 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.025+0000 7f76c8a432c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T16:25:01.088 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.085+0000 7f76c8a432c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T16:25:01.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.117+0000 7fc2ef955700 1 Processor -- start 2024-09-13T16:25:01.121 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.117+0000 7fc2ef955700 1 -- start start 2024-09-13T16:25:01.124 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80b3b40 0x7fc2e815a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc2e815b310 0x7fc2e80b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc2e80ce9c0 0x7fc2e80cee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e80cdcd0 con 0x7fc2e815b310 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2e8072030 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e80ce530 con 0x7fc2e80b3b40 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ed6f1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80b3b40 0x7fc2e815a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ed6f1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80b3b40 0x7fc2e815a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39676/0 (socket says 172.21.15.145:39676) 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ed6f1700 1 -- 172.21.15.145:0/3247754337 learned_addr learned my addr 172.21.15.145:0/3247754337 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2edef2700 1 --2- 172.21.15.145:0/3247754337 >> v2:172.21.15.145:3300/0 conn(0x7fc2e80ce9c0 0x7fc2e80cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ecef0700 1 --2- 172.21.15.145:0/3247754337 >> v2:172.21.15.119:3300/0 conn(0x7fc2e815b310 0x7fc2e80b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2edef2700 1 -- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80b3b40 msgr2=0x7fc2e815a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:01.125 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2edef2700 1 --2- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80b3b40 0x7fc2e815a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2edef2700 1 -- 172.21.15.145:0/3247754337 >> v2:172.21.15.119:3300/0 conn(0x7fc2e815b310 msgr2=0x7fc2e80b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2edef2700 1 --2- 172.21.15.145:0/3247754337 >> v2:172.21.15.119:3300/0 conn(0x7fc2e815b310 0x7fc2e80b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2edef2700 1 -- 172.21.15.145:0/3247754337 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e815aec0 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ed6f1700 1 --2- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80b3b40 0x7fc2e815a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2edef2700 1 --2- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e80ce9c0 0x7fc2e80cee70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8067520 tx=0x7fc2e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=277e69601a77418f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2dffff700 1 -- 172.21.15.145:0/3247754337 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e000e070 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2dffff700 1 -- 172.21.15.145:0/3247754337 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2e000bb20 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2dffff700 1 -- 172.21.15.145:0/3247754337 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2e0012430 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e80ce9c0 msgr2=0x7fc2e80cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e80ce9c0 0x7fc2e80cee70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc2e8067520 tx=0x7fc2e000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- 172.21.15.145:0/3247754337 shutdown_connections 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e80ce9c0 0x7fc2e80cee70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.126 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3247754337 >> v2:172.21.15.119:3300/0 conn(0x7fc2e815b310 0x7fc2e80b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3247754337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80b3b40 0x7fc2e815a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- 172.21.15.145:0/3247754337 >> 172.21.15.145:0/3247754337 conn(0x7fc2e8070dc0 msgr2=0x7fc2e80711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- 172.21.15.145:0/3247754337 shutdown_connections 2024-09-13T16:25:01.127 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- 172.21.15.145:0/3247754337 wait complete. 2024-09-13T16:25:01.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 Processor -- start 2024-09-13T16:25:01.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.121+0000 7fc2ef955700 1 -- start start 2024-09-13T16:25:01.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e80b3b40 0x7fc2e81426f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80ce9c0 0x7fc2e815e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e815b310 0x7fc2e815edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8159940 con 0x7fc2e80b3b40 2024-09-13T16:25:01.129 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e81615e0 con 0x7fc2e815b310 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2e8142c30 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80ce9c0 0x7fc2e815e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2edef2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e815b310 0x7fc2e815edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80ce9c0 0x7fc2e815e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39680/0 (socket says 172.21.15.145:39680) 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 -- 172.21.15.145:0/3131331638 learned_addr learned my addr 172.21.15.145:0/3131331638 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ed6f1700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e80b3b40 0x7fc2e81426f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 -- 172.21.15.145:0/3131331638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e815b310 msgr2=0x7fc2e815edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e815b310 0x7fc2e815edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 -- 172.21.15.145:0/3131331638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e80b3b40 msgr2=0x7fc2e81426f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e80b3b40 0x7fc2e81426f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 -- 172.21.15.145:0/3131331638 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e4009570 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ecef0700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80ce9c0 0x7fc2e815e890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800d7d0 tx=0x7fc2d800dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ddffb700 1 -- 172.21.15.145:0/3131331638 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8017070 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.130 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2e0009a70 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ddffb700 1 -- 172.21.15.145:0/3131331638 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d8004020 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2e8160cb0 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ddffb700 1 -- 172.21.15.145:0/3131331638 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d8004190 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ddffb700 1 -- 172.21.15.145:0/3131331638 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fc2d80036c0 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ddffb700 1 -- 172.21.15.145:0/3131331638 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fc2d8003e00 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.131 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.125+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2e813fd30 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.132 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.129+0000 7fc2ddffb700 1 -- 172.21.15.145:0/3131331638 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fc2e813fd30 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.207 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.205+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0ab479b1-9888-4ffd-b4f3-c4c2873accb1", "id": 3} v 0) v1 -- 0x7fc2e815c930 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.217 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ddffb700 1 -- 172.21.15.145:0/3131331638 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ab479b1-9888-4ffd-b4f3-c4c2873accb1", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc2d8003aa0 con 0x7fc2e80ce9c0 2024-09-13T16:25:01.217 INFO:teuthology.orchestra.run.smithi145.stdout:3 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80ce9c0 msgr2=0x7fc2e815e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80ce9c0 0x7fc2e815e890 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc2d800d7d0 tx=0x7fc2d800dae0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 shutdown_connections 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2e815b310 0x7fc2e815edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2e80ce9c0 0x7fc2e815e890 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ef955700 1 --2- 172.21.15.145:0/3131331638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2e80b3b40 0x7fc2e81426f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.213+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 >> 172.21.15.145:0/3131331638 conn(0x7fc2e8070dc0 msgr2=0x7fc2e815cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.219 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.217+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 shutdown_connections 2024-09-13T16:25:01.220 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.217+0000 7fc2ef955700 1 -- 172.21.15.145:0/3131331638 wait complete. 2024-09-13T16:25:01.230 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new d0b1aaff-0f4b-4f31-9a0c-520e64952b8e 4 2024-09-13T16:25:01.268 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.265+0000 7f76c8a432c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:25:01.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.329+0000 7fba198e3700 1 Processor -- start 2024-09-13T16:25:01.335 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.329+0000 7fba198e3700 1 -- start start 2024-09-13T16:25:01.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fba14159530 0x7fba140cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140ce700 con 0x7fba1415d8d0 2024-09-13T16:25:01.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba140ce9e0 con 0x7fba140ce110 2024-09-13T16:25:01.339 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba140cecc0 con 0x7fba14159530 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba188e1700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fba14159530 0x7fba140cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba13fff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fba140ce110 0x7fba1415d1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36280/0 (socket says 172.21.15.145:36280) 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba13fff700 1 -- 172.21.15.145:0/682654692 learned_addr learned my addr 172.21.15.145:0/682654692 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba188e1700 1 -- 172.21.15.145:0/682654692 >> v1:172.21.15.145:6789/0 conn(0x7fba140ce110 legacy=0x7fba1415d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba188e1700 1 -- 172.21.15.145:0/682654692 >> v1:172.21.15.119:6789/0 conn(0x7fba1415d8d0 legacy=0x7fba14159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba137fe700 1 -- 172.21.15.145:0/682654692 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3224800151 0 0) 0x7fba140ce9e0 con 0x7fba140ce110 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba188e1700 1 -- 172.21.15.145:0/682654692 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba140cefa0 con 0x7fba14159530 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba188e1700 1 --2- 172.21.15.145:0/682654692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba14159530 0x7fba140cdbd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fba08009880 tx=0x7fba08009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bfd9141b8a2dc92 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba137fe700 1 -- 172.21.15.145:0/682654692 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08011830 con 0x7fba14159530 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba137fe700 1 -- 172.21.15.145:0/682654692 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba08011990 con 0x7fba14159530 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba137fe700 1 -- 172.21.15.145:0/682654692 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba08011d20 con 0x7fba14159530 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- 172.21.15.145:0/682654692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba14159530 msgr2=0x7fba140cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.340 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 --2- 172.21.15.145:0/682654692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba14159530 0x7fba140cdbd0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fba08009880 tx=0x7fba08009b90 comp rx=0 tx=0).stop 2024-09-13T16:25:01.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- 172.21.15.145:0/682654692 shutdown_connections 2024-09-13T16:25:01.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 --2- 172.21.15.145:0/682654692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba14159530 0x7fba140cdbd0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- 172.21.15.145:0/682654692 >> 172.21.15.145:0/682654692 conn(0x7fba1406b460 msgr2=0x7fba1406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- 172.21.15.145:0/682654692 shutdown_connections 2024-09-13T16:25:01.341 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.333+0000 7fba198e3700 1 -- 172.21.15.145:0/682654692 wait complete. 2024-09-13T16:25:01.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 Processor -- start 2024-09-13T16:25:01.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 -- start start 2024-09-13T16:25:01.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba140ce110 0x7fba141e81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.343 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba14159530 0x7fba14151b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba1415d8d0 0x7fba140c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba141e3050 con 0x7fba14159530 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba141e8800 con 0x7fba1415d8d0 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba141e8ae0 con 0x7fba140ce110 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba140ce110 0x7fba141e81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba190e2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba1415d8d0 0x7fba140c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba140ce110 0x7fba141e81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39690/0 (socket says 172.21.15.145:39690) 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba190e2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba1415d8d0 0x7fba140c9e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:45530/0 (socket says 172.21.15.145:45530) 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 -- 172.21.15.145:0/1591297955 learned_addr learned my addr 172.21.15.145:0/1591297955 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba13fff700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba14159530 0x7fba14151b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 -- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba1415d8d0 msgr2=0x7fba140c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba1415d8d0 0x7fba140c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.344 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 -- 172.21.15.145:0/1591297955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba14159530 msgr2=0x7fba14151b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba14159530 0x7fba14151b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 -- 172.21.15.145:0/1591297955 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba080094b0 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba190e2700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba1415d8d0 0x7fba140c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba188e1700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba140ce110 0x7fba141e81b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fba080090b0 tx=0x7fba08004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba117fa700 1 -- 172.21.15.145:0/1591297955 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0801aa50 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba117fa700 1 -- 172.21.15.145:0/1591297955 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0801f070 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba141e8dc0 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba117fa700 1 -- 172.21.15.145:0/1591297955 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba0801ae10 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba14152070 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba117fa700 1 -- 172.21.15.145:0/1591297955 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fba0801b730 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.337+0000 7fba117fa700 1 -- 172.21.15.145:0/1591297955 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fba08025cd0 con 0x7fba140ce110 2024-09-13T16:25:01.345 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.341+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba141e3190 con 0x7fba140ce110 2024-09-13T16:25:01.347 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.345+0000 7fba117fa700 1 -- 172.21.15.145:0/1591297955 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fba141e3190 con 0x7fba140ce110 2024-09-13T16:25:01.372 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:01.371+0000 7f7f152612c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-13T16:25:01.424 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.421+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d0b1aaff-0f4b-4f31-9a0c-520e64952b8e", "id": 4} v 0) v1 -- 0x7fba1415de30 con 0x7fba140ce110 2024-09-13T16:25:01.430 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:01.427+0000 7f7f152612c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-13T16:25:01.434 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.429+0000 7fba117fa700 1 -- 172.21.15.145:0/1591297955 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d0b1aaff-0f4b-4f31-9a0c-520e64952b8e", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fba08033060 con 0x7fba140ce110 2024-09-13T16:25:01.434 INFO:teuthology.orchestra.run.smithi145.stdout:4 2024-09-13T16:25:01.435 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba140ce110 msgr2=0x7fba141e81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba140ce110 0x7fba141e81b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fba080090b0 tx=0x7fba08004770 comp rx=0 tx=0).stop 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 shutdown_connections 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fba1415d8d0 0x7fba140c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fba14159530 0x7fba14151b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 --2- 172.21.15.145:0/1591297955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fba140ce110 0x7fba141e81b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 >> 172.21.15.145:0/1591297955 conn(0x7fba1406b460 msgr2=0x7fba1415be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 shutdown_connections 2024-09-13T16:25:01.436 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.433+0000 7fba198e3700 1 -- 172.21.15.145:0/1591297955 wait complete. 2024-09-13T16:25:01.449 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new 651f46b8-548f-48f0-a80d-1e52f8bfe475 5 2024-09-13T16:25:01.505 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.501+0000 7f76c8a432c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T16:25:01.561 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.557+0000 7f76c8a432c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T16:25:01.601 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:01.599+0000 7f7f152612c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:25:01.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fe0f92700 1 Processor -- start 2024-09-13T16:25:01.607 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fe0f92700 1 -- start start 2024-09-13T16:25:01.610 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fe0f92700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0fdc15cad0 0x7f0fdc0abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fe0f92700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fdc0ccd70 con 0x7f0fdc06ba70 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fe0f92700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fdc071ea0 con 0x7f0fdc071450 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fe0f92700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fdc072180 con 0x7f0fdc15cad0 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdaffd700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f0fdc071450 0x7f0fdc06b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36292/0 (socket says 172.21.15.145:36292) 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdaffd700 1 -- 172.21.15.145:0/2753984136 learned_addr learned my addr 172.21.15.145:0/2753984136 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdb7fe700 1 --2- 172.21.15.145:0/2753984136 >> v2:172.21.15.119:3300/0 conn(0x7f0fdc15cad0 0x7f0fdc0abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836495702 0 0) 0x7f0fdc071ea0 con 0x7f0fdc071450 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc80035d0 con 0x7f0fdc071450 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 <== mon.1 v1:172.21.15.145:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2415742435 0 0) 0x7f0fc80035d0 con 0x7f0fdc071450 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 >> v2:172.21.15.119:3300/0 conn(0x7f0fdc15cad0 msgr2=0x7f0fdc0abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 --2- 172.21.15.145:0/2753984136 >> v2:172.21.15.119:3300/0 conn(0x7f0fdc15cad0 0x7f0fdc0abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.611 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 >> v1:172.21.15.119:6789/0 conn(0x7f0fdc06ba70 legacy=0x7f0fdc0cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 --> v1:172.21.15.145:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc072460 con 0x7f0fdc071450 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 <== mon.1 v1:172.21.15.145:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f0fcc001670 con 0x7f0fdc071450 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 >> v1:172.21.15.145:6789/0 conn(0x7f0fdc071450 legacy=0x7f0fdc06b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fc8003a20 0x7f0fc8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc8004420 0x7f0fc80048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fc800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc072180 con 0x7f0fc8004420 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc800c0d0 con 0x7f0fc8005190 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc800c290 con 0x7f0fc8003a20 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdb7fe700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fc8003a20 0x7f0fc8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fc800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdaffd700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc8004420 0x7f0fc80048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.612 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 -- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fc8003a20 msgr2=0x7f0fc8003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fc8003a20 0x7f0fc8003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 -- 172.21.15.145:0/2753984136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc8004420 msgr2=0x7f0fc80048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fc8004420 0x7f0fc80048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 -- 172.21.15.145:0/2753984136 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc80035d0 con 0x7f0fc8005190 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 -- 172.21.15.145:0/2753984136 reap_dead start 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.605+0000 7f0fdbfff700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fc800ba30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400c160 tx=0x7f0fc80058a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbac59b8f1bf6ec4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4001db0 con 0x7f0fc8005190 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fd400ccc0 con 0x7f0fc8005190 2024-09-13T16:25:01.613 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fda7fc700 1 -- 172.21.15.145:0/2753984136 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fd4003c60 con 0x7f0fc8005190 2024-09-13T16:25:01.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 -- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 msgr2=0x7f0fc800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fc800ba30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0fd400c160 tx=0x7f0fc80058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 -- 172.21.15.145:0/2753984136 shutdown_connections 2024-09-13T16:25:01.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 --2- 172.21.15.145:0/2753984136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fc800ba30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.614 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 -- 172.21.15.145:0/2753984136 >> 172.21.15.145:0/2753984136 conn(0x7f0fdc0cb8f0 msgr2=0x7f0fdc0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 -- 172.21.15.145:0/2753984136 shutdown_connections 2024-09-13T16:25:01.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 -- 172.21.15.145:0/2753984136 wait complete. 2024-09-13T16:25:01.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 Processor -- start 2024-09-13T16:25:01.615 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.609+0000 7f0fe0f92700 1 -- start start 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fdc09bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fdc09c230 0x7f0fdc099ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fdc09a530 0x7f0fdc1f3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc09b4a0 con 0x7f0fdc09c230 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc09c690 con 0x7f0fc8005190 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fdc09aee0 con 0x7f0fdc09a530 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fdc09bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdbfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fdc09a530 0x7f0fdc1f3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.617 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fdc09bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:45546/0 (socket says 172.21.15.145:45546) 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 -- 172.21.15.145:0/610506127 learned_addr learned my addr 172.21.15.145:0/610506127 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdaffd700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fdc09c230 0x7f0fdc099ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 -- 172.21.15.145:0/610506127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fdc09a530 msgr2=0x7f0fdc1f3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fdc09a530 0x7f0fdc1f3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 -- 172.21.15.145:0/610506127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fdc09c230 msgr2=0x7f0fdc099ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fdc09c230 0x7f0fdc099ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 -- 172.21.15.145:0/610506127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fdc09b1c0 con 0x7f0fc8005190 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fdb7fe700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fdc09bcf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40119e0 tx=0x7f0fc4011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fd3fff700 1 -- 172.21.15.145:0/610506127 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc4011e70 con 0x7f0fc8005190 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fd3fff700 1 -- 172.21.15.145:0/610506127 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc400f940 con 0x7f0fc8005190 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fdc1f3780 con 0x7f0fc8005190 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fd3fff700 1 -- 172.21.15.145:0/610506127 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc400fcd0 con 0x7f0fc8005190 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fdc0b0600 con 0x7f0fc8005190 2024-09-13T16:25:01.618 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fd3fff700 1 -- 172.21.15.145:0/610506127 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f0fc4014740 con 0x7f0fc8005190 2024-09-13T16:25:01.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fd3fff700 1 -- 172.21.15.145:0/610506127 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f0fc401dac0 con 0x7f0fc8005190 2024-09-13T16:25:01.619 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.613+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fdc0abe60 con 0x7f0fc8005190 2024-09-13T16:25:01.622 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.617+0000 7f0fd3fff700 1 -- 172.21.15.145:0/610506127 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f0fc401dda0 con 0x7f0fc8005190 2024-09-13T16:25:01.700 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.697+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "651f46b8-548f-48f0-a80d-1e52f8bfe475", "id": 5} v 0) v1 -- 0x7f0fdc073b40 con 0x7f0fc8005190 2024-09-13T16:25:01.707 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fd3fff700 1 -- 172.21.15.145:0/610506127 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "651f46b8-548f-48f0-a80d-1e52f8bfe475", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0fc4014c00 con 0x7f0fc8005190 2024-09-13T16:25:01.707 INFO:teuthology.orchestra.run.smithi145.stdout:5 2024-09-13T16:25:01.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 msgr2=0x7f0fdc09bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fdc09bcf0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0fc40119e0 tx=0x7f0fc4011cf0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 shutdown_connections 2024-09-13T16:25:01.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fdc09a530 0x7f0fdc1f3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fdc09c230 0x7f0fdc099ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 --2- 172.21.15.145:0/610506127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fc8005190 0x7f0fdc09bcf0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.709 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 >> 172.21.15.145:0/610506127 conn(0x7f0fdc0cb8f0 msgr2=0x7f0fdc0b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 shutdown_connections 2024-09-13T16:25:01.710 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.705+0000 7f0fe0f92700 1 -- 172.21.15.145:0/610506127 wait complete. 2024-09-13T16:25:01.722 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new c1438d19-cd27-4b99-ac1d-e07136e7ad46 6 2024-09-13T16:25:01.750 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.745+0000 7f76c8a432c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T16:25:01.819 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.817+0000 7f76c8a432c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T16:25:01.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 Processor -- start 2024-09-13T16:25:01.827 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- start start 2024-09-13T16:25:01.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec06aa60 0x7fcfec165720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.828 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfec15c300 con 0x7fcfec0cd3f0 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfec15c000 con 0x7fcfec144260 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec15b4b0 con 0x7fcfec06aa60 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2035700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec06aa60 0x7fcfec165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff1834700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fcfec144260 0x7fcfec15ada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36306/0 (socket says 172.21.15.145:36306) 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff1834700 1 -- 172.21.15.145:0/925064080 learned_addr learned my addr 172.21.15.145:0/925064080 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff0832700 1 -- 172.21.15.145:0/925064080 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702890891 0 0) 0x7fcfec15c000 con 0x7fcfec144260 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2035700 1 -- 172.21.15.145:0/925064080 >> v1:172.21.15.145:6789/0 conn(0x7fcfec144260 legacy=0x7fcfec15ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2035700 1 -- 172.21.15.145:0/925064080 >> v1:172.21.15.119:6789/0 conn(0x7fcfec0cd3f0 legacy=0x7fcfec06a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.829 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2035700 1 -- 172.21.15.145:0/925064080 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec144640 con 0x7fcfec06aa60 2024-09-13T16:25:01.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2035700 1 --2- 172.21.15.145:0/925064080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec06aa60 0x7fcfec165720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009dc0 tx=0x7fcfe400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0e7c904fee11138 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.830 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff0832700 1 -- 172.21.15.145:0/925064080 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe400e070 con 0x7fcfec06aa60 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff0832700 1 -- 172.21.15.145:0/925064080 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe400bbd0 con 0x7fcfec06aa60 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff0832700 1 -- 172.21.15.145:0/925064080 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe4012430 con 0x7fcfec06aa60 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- 172.21.15.145:0/925064080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec06aa60 msgr2=0x7fcfec165720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 --2- 172.21.15.145:0/925064080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec06aa60 0x7fcfec165720 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfe4009dc0 tx=0x7fcfe400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- 172.21.15.145:0/925064080 shutdown_connections 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 --2- 172.21.15.145:0/925064080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec06aa60 0x7fcfec165720 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- 172.21.15.145:0/925064080 >> 172.21.15.145:0/925064080 conn(0x7fcfec0b15e0 msgr2=0x7fcfec0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- 172.21.15.145:0/925064080 shutdown_connections 2024-09-13T16:25:01.831 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.825+0000 7fcff2836700 1 -- 172.21.15.145:0/925064080 wait complete. 2024-09-13T16:25:01.832 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:01.831+0000 7f7f152612c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-13T16:25:01.832 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 Processor -- start 2024-09-13T16:25:01.832 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 -- start start 2024-09-13T16:25:01.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfec06aa60 0x7fcfec13c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec0cd3f0 0x7fcfec06d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfec144260 0x7fcfec06db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:01.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec13cca0 con 0x7fcfec144260 2024-09-13T16:25:01.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec1f0720 con 0x7fcfec06aa60 2024-09-13T16:25:01.835 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2836700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec1f0a00 con 0x7fcfec0cd3f0 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfec06aa60 0x7fcfec13c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfec06aa60 0x7fcfec13c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.145:45556/0 (socket says 172.21.15.145:45556) 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 -- 172.21.15.145:0/1573092320 learned_addr learned my addr 172.21.15.145:0/1573092320 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1033700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec0cd3f0 0x7fcfec06d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff2035700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfec144260 0x7fcfec06db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 -- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec0cd3f0 msgr2=0x7fcfec06d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec0cd3f0 0x7fcfec06d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 -- 172.21.15.145:0/1573092320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfec144260 msgr2=0x7fcfec06db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfec144260 0x7fcfec06db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 -- 172.21.15.145:0/1573092320 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe4009a70 con 0x7fcfec06aa60 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcff1834700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfec06aa60 0x7fcfec13c480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c4a0 tx=0x7fcfe800c7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.829+0000 7fcfe27fc700 1 -- 172.21.15.145:0/1573092320 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8004ca0 con 0x7fcfec06aa60 2024-09-13T16:25:01.836 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.833+0000 7fcfe27fc700 1 -- 172.21.15.145:0/1573092320 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfe8004e00 con 0x7fcfec06aa60 2024-09-13T16:25:01.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.833+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec1f0ce0 con 0x7fcfec06aa60 2024-09-13T16:25:01.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.833+0000 7fcfe27fc700 1 -- 172.21.15.145:0/1573092320 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfe8007310 con 0x7fcfec06aa60 2024-09-13T16:25:01.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.833+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec13c9c0 con 0x7fcfec06aa60 2024-09-13T16:25:01.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.833+0000 7fcfe27fc700 1 -- 172.21.15.145:0/1573092320 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fcfe801d3f0 con 0x7fcfec06aa60 2024-09-13T16:25:01.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.833+0000 7fcfe27fc700 1 -- 172.21.15.145:0/1573092320 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fcfe80139b0 con 0x7fcfec06aa60 2024-09-13T16:25:01.837 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.833+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec06a350 con 0x7fcfec06aa60 2024-09-13T16:25:01.839 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.837+0000 7fcfe27fc700 1 -- 172.21.15.145:0/1573092320 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fcfe8013c70 con 0x7fcfec06aa60 2024-09-13T16:25:01.885 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:01.883+0000 7f7f152612c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-13T16:25:01.890 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:01.885+0000 7f76c8a432c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T16:25:01.916 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.913+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c1438d19-cd27-4b99-ac1d-e07136e7ad46", "id": 6} v 0) v1 -- 0x7fcfec0b2210 con 0x7fcfec06aa60 2024-09-13T16:25:01.926 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.921+0000 7fcfe27fc700 1 -- 172.21.15.145:0/1573092320 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c1438d19-cd27-4b99-ac1d-e07136e7ad46", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcfe8013e10 con 0x7fcfec06aa60 2024-09-13T16:25:01.926 INFO:teuthology.orchestra.run.smithi145.stdout:6 2024-09-13T16:25:01.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfec06aa60 msgr2=0x7fcfec13c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:01.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfec06aa60 0x7fcfec13c480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800c4a0 tx=0x7fcfe800c7b0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 shutdown_connections 2024-09-13T16:25:01.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfec144260 0x7fcfec06db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfec0cd3f0 0x7fcfec06d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 --2- 172.21.15.145:0/1573092320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfec06aa60 0x7fcfec13c480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:01.928 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 >> 172.21.15.145:0/1573092320 conn(0x7fcfec0b15e0 msgr2=0x7fcfec06e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:01.929 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 shutdown_connections 2024-09-13T16:25:01.929 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:01.925+0000 7fcff2836700 1 -- 172.21.15.145:0/1573092320 wait complete. 2024-09-13T16:25:01.940 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --cluster ceph osd new fc8cd1d4-48c7-4624-9510-d7e0971ea575 7 2024-09-13T16:25:02.050 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.045+0000 7fdd54256700 1 Processor -- start 2024-09-13T16:25:02.051 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.045+0000 7fdd54256700 1 -- start start 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdd4c15cf80 0x7fdd4c0d1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdd4c0d3890 0x7fdd4c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4c0cd910 con 0x7fdd4c0cfcc0 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c15d360 con 0x7fdd4c0d3890 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd4c0ce750 con 0x7fdd4c15cf80 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd51ff2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdd4c15cf80 0x7fdd4c0d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd527f3700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fdd4c0cfcc0 0x7fdd4c1609d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.145:36318/0 (socket says 172.21.15.145:36318) 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd527f3700 1 -- 172.21.15.145:0/1033395083 learned_addr learned my addr 172.21.15.145:0/1033395083 (peer_addr_for_me v1:172.21.15.145:0/0) 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd517f1700 1 --2- 172.21.15.145:0/1033395083 >> v2:172.21.15.119:3300/0 conn(0x7fdd4c0d3890 0x7fdd4c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.053 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd50ff0700 1 -- 172.21.15.145:0/1033395083 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643276091 0 0) 0x7fdd4c0cd910 con 0x7fdd4c0cfcc0 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd50ff0700 1 -- 172.21.15.145:0/1033395083 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd380035d0 con 0x7fdd4c0cfcc0 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd517f1700 1 -- 172.21.15.145:0/1033395083 >> v2:172.21.15.145:3300/0 conn(0x7fdd4c15cf80 msgr2=0x7fdd4c0d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd517f1700 1 --2- 172.21.15.145:0/1033395083 >> v2:172.21.15.145:3300/0 conn(0x7fdd4c15cf80 0x7fdd4c0d1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd517f1700 1 -- 172.21.15.145:0/1033395083 >> v1:172.21.15.145:6789/0 conn(0x7fdd4c0cfcc0 legacy=0x7fdd4c1609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd517f1700 1 -- 172.21.15.145:0/1033395083 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c0d00f0 con 0x7fdd4c0d3890 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd51ff2700 1 --2- 172.21.15.145:0/1033395083 >> v2:172.21.15.145:3300/0 conn(0x7fdd4c15cf80 0x7fdd4c0d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd517f1700 1 --2- 172.21.15.145:0/1033395083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 0x7fdd4c0cf780 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c15d4a0 tx=0x7fdd40009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b0158901eebd34 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd50ff0700 1 -- 172.21.15.145:0/1033395083 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40004020 con 0x7fdd4c0d3890 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd50ff0700 1 -- 172.21.15.145:0/1033395083 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd40004180 con 0x7fdd4c0d3890 2024-09-13T16:25:02.054 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd50ff0700 1 -- 172.21.15.145:0/1033395083 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd40004530 con 0x7fdd4c0d3890 2024-09-13T16:25:02.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- 172.21.15.145:0/1033395083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 msgr2=0x7fdd4c0cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 --2- 172.21.15.145:0/1033395083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 0x7fdd4c0cf780 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c15d4a0 tx=0x7fdd40009b30 comp rx=0 tx=0).stop 2024-09-13T16:25:02.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- 172.21.15.145:0/1033395083 shutdown_connections 2024-09-13T16:25:02.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 --2- 172.21.15.145:0/1033395083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 0x7fdd4c0cf780 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 --2- 172.21.15.145:0/1033395083 >> v2:172.21.15.145:3300/0 conn(0x7fdd4c15cf80 0x7fdd4c0d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.055 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- 172.21.15.145:0/1033395083 >> 172.21.15.145:0/1033395083 conn(0x7fdd4c0d2d70 msgr2=0x7fdd4c0d3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:02.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- 172.21.15.145:0/1033395083 shutdown_connections 2024-09-13T16:25:02.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.049+0000 7fdd54256700 1 -- 172.21.15.145:0/1033395083 wait complete. 2024-09-13T16:25:02.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 Processor -- start 2024-09-13T16:25:02.056 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 -- start start 2024-09-13T16:25:02.059 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd4c0cfcc0 0x7fdd4c0c9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 0x7fdd4c0a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd4c15cf80 0x7fdd4c1502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c15a780 con 0x7fdd4c0d3890 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0c6780 con 0x7fdd4c0cfcc0 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd54256700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4c0c6a60 con 0x7fdd4c15cf80 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd4c15cf80 0x7fdd4c1502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd51ff2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd4c0cfcc0 0x7fdd4c0c9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd4c15cf80 0x7fdd4c1502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.145:39742/0 (socket says 172.21.15.145:39742) 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 -- 172.21.15.145:0/190964744 learned_addr learned my addr 172.21.15.145:0/190964744 (peer_addr_for_me v2:172.21.15.145:0/0) 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd517f1700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 0x7fdd4c0a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.060 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 -- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd4c0cfcc0 msgr2=0x7fdd4c0c9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd4c0cfcc0 0x7fdd4c0c9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 -- 172.21.15.145:0/190964744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 msgr2=0x7fdd4c0a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 0x7fdd4c0a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd527f3700 1 -- 172.21.15.145:0/190964744 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480094b0 con 0x7fdd4c15cf80 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.053+0000 7fdd51ff2700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd4c0cfcc0 0x7fdd4c0c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd527f3700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd4c15cf80 0x7fdd4c1502e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400caa0 tx=0x7fdd4400cad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd3e7fc700 1 -- 172.21.15.145:0/190964744 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44007480 con 0x7fdd4c15cf80 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd3e7fc700 1 -- 172.21.15.145:0/190964744 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd440075e0 con 0x7fdd4c15cf80 2024-09-13T16:25:02.061 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd400094b0 con 0x7fdd4c15cf80 2024-09-13T16:25:02.062 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd3e7fc700 1 -- 172.21.15.145:0/190964744 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44007970 con 0x7fdd4c15cf80 2024-09-13T16:25:02.062 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4c0c7020 con 0x7fdd4c15cf80 2024-09-13T16:25:02.062 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd3e7fc700 1 -- 172.21.15.145:0/190964744 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7fdd4401c440 con 0x7fdd4c15cf80 2024-09-13T16:25:02.062 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd3e7fc700 1 -- 172.21.15.145:0/190964744 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fdd4401cd50 con 0x7fdd4c15cf80 2024-09-13T16:25:02.062 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.057+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4c0677a0 con 0x7fdd4c15cf80 2024-09-13T16:25:02.064 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.061+0000 7fdd3e7fc700 1 -- 172.21.15.145:0/190964744 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7fdd44017070 con 0x7fdd4c15cf80 2024-09-13T16:25:02.064 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:02.063+0000 7f7f152612c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-13T16:25:02.128 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:02.123+0000 7f7f152612c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-13T16:25:02.141 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.137+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd new", "uuid": "fc8cd1d4-48c7-4624-9510-d7e0971ea575", "id": 7} v 0) v1 -- 0x7fdd4c0ce3e0 con 0x7fdd4c15cf80 2024-09-13T16:25:02.149 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd3e7fc700 1 -- 172.21.15.145:0/190964744 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fc8cd1d4-48c7-4624-9510-d7e0971ea575", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fdd4401c7c0 con 0x7fdd4c15cf80 2024-09-13T16:25:02.149 INFO:teuthology.orchestra.run.smithi145.stdout:7 2024-09-13T16:25:02.150 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd4c15cf80 msgr2=0x7fdd4c1502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd4c15cf80 0x7fdd4c1502e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400caa0 tx=0x7fdd4400cad0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 shutdown_connections 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd4c15cf80 0x7fdd4c1502e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd4c0d3890 0x7fdd4c0a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 --2- 172.21.15.145:0/190964744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd4c0cfcc0 0x7fdd4c0c9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 >> 172.21.15.145:0/190964744 conn(0x7fdd4c0d2d70 msgr2=0x7fdd4c06c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.145+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 shutdown_connections 2024-09-13T16:25:02.151 INFO:teuthology.orchestra.run.smithi145.stderr:2024-09-13T16:25:02.149+0000 7fdd54256700 1 -- 172.21.15.145:0/190964744 wait complete. 2024-09-13T16:25:02.164 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-13T16:25:02.164 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-13T16:25:02.167 INFO:tasks.ceph.osd.0:Started 2024-09-13T16:25:02.167 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-13T16:25:02.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-13T16:25:02.169 INFO:tasks.ceph.osd.1:Started 2024-09-13T16:25:02.169 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-13T16:25:02.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-13T16:25:02.171 INFO:tasks.ceph.osd.2:Started 2024-09-13T16:25:02.172 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-13T16:25:02.172 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-13T16:25:02.174 INFO:tasks.ceph.osd.3:Started 2024-09-13T16:25:02.174 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-13T16:25:02.174 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-13T16:25:02.176 INFO:tasks.ceph.osd.4:Started 2024-09-13T16:25:02.177 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-13T16:25:02.177 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-13T16:25:02.179 INFO:tasks.ceph.osd.5:Started 2024-09-13T16:25:02.179 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-13T16:25:02.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-13T16:25:02.181 INFO:tasks.ceph.osd.6:Started 2024-09-13T16:25:02.182 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-13T16:25:02.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-13T16:25:02.184 INFO:tasks.ceph.osd.7:Started 2024-09-13T16:25:02.184 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:02.197 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:02.195+0000 7f7f152612c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-13T16:25:02.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f28650be700 1 Processor -- start 2024-09-13T16:25:02.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f28650be700 1 -- start start 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f28650be700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f286015fee0 0x7f286006b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f28650be700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f28600714c0 0x7f2860071900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f28650be700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600ad110 con 0x7f2860070360 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f28650be700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2860071e40 con 0x7f286015fee0 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f28650be700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28600acb50 con 0x7f28600714c0 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f286015fee0 0x7f286006b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f286015fee0 0x7f286006b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57016/0 (socket says 172.21.15.119:57016) 2024-09-13T16:25:02.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 -- 172.21.15.119:0/2159007684 learned_addr learned my addr 172.21.15.119:0/2159007684 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 -- 172.21.15.119:0/2159007684 >> v2:172.21.15.145:3300/0 conn(0x7f28600714c0 msgr2=0x7f2860071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 --2- 172.21.15.119:0/2159007684 >> v2:172.21.15.145:3300/0 conn(0x7f28600714c0 0x7f2860071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 -- 172.21.15.119:0/2159007684 >> v1:172.21.15.145:6789/0 conn(0x7f2860070360 legacy=0x7f28600cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 -- 172.21.15.119:0/2159007684 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600ac2b0 con 0x7f286015fee0 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285ffff700 1 --2- 172.21.15.119:0/2159007684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f286006b4f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f28600cde10 tx=0x7f2848009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bc1cf1dd303ef73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285effd700 1 -- 172.21.15.119:0/2159007684 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848004230 con 0x7f286015fee0 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285effd700 1 -- 172.21.15.119:0/2159007684 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2848004390 con 0x7f286015fee0 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.299+0000 7f285effd700 1 -- 172.21.15.119:0/2159007684 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28480036c0 con 0x7f286015fee0 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- 172.21.15.119:0/2159007684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 msgr2=0x7f286006b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 --2- 172.21.15.119:0/2159007684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f286006b4f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f28600cde10 tx=0x7f2848009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- 172.21.15.119:0/2159007684 shutdown_connections 2024-09-13T16:25:02.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 --2- 172.21.15.119:0/2159007684 >> v2:172.21.15.145:3300/0 conn(0x7f28600714c0 0x7f2860071900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 --2- 172.21.15.119:0/2159007684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f286006b4f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- 172.21.15.119:0/2159007684 >> 172.21.15.119:0/2159007684 conn(0x7f28600ae4b0 msgr2=0x7f28600b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:02.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- 172.21.15.119:0/2159007684 shutdown_connections 2024-09-13T16:25:02.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- 172.21.15.119:0/2159007684 wait complete. 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 Processor -- start 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- start start 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2860070360 0x7f2860159eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600714c0 0x7f286014b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f2860148060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286015b5d0 con 0x7f286015fee0 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286015a530 con 0x7f2860070360 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28650be700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286015a810 con 0x7f28600714c0 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28648bd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f2860148060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28648bd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f2860148060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57022/0 (socket says 172.21.15.119:57022) 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f28648bd700 1 -- 172.21.15.119:0/3008914994 learned_addr learned my addr 172.21.15.119:0/3008914994 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:02.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f285ffff700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2860070360 0x7f2860159eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.303+0000 7f285f7fe700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600714c0 0x7f286014b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28648bd700 1 -- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600714c0 msgr2=0x7f286014b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28648bd700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600714c0 0x7f286014b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28648bd700 1 -- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2860070360 msgr2=0x7f2860159eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28648bd700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2860070360 0x7f2860159eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28648bd700 1 -- 172.21.15.119:0/3008914994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2848009570 con 0x7f286015fee0 2024-09-13T16:25:02.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28648bd700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f2860148060 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f285800c7c0 tx=0x7f285800cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:02.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f285cff9700 1 -- 172.21.15.119:0/3008914994 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858003e30 con 0x7f286015fee0 2024-09-13T16:25:02.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f285cff9700 1 -- 172.21.15.119:0/3008914994 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2858005070 con 0x7f286015fee0 2024-09-13T16:25:02.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286015aaf0 con 0x7f286015fee0 2024-09-13T16:25:02.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f285cff9700 1 -- 172.21.15.119:0/3008914994 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2858005420 con 0x7f286015fee0 2024-09-13T16:25:02.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286015b0b0 con 0x7f286015fee0 2024-09-13T16:25:02.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f285cff9700 1 -- 172.21.15.119:0/3008914994 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f285801c440 con 0x7f286015fee0 2024-09-13T16:25:02.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f285cff9700 1 -- 172.21.15.119:0/3008914994 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2858012ad0 con 0x7f286015fee0 2024-09-13T16:25:02.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.307+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28600742c0 con 0x7f286015fee0 2024-09-13T16:25:02.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.311+0000 7f285cff9700 1 -- 172.21.15.119:0/3008914994 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f2858021060 con 0x7f286015fee0 2024-09-13T16:25:02.346 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:02.341+0000 7f76c8a432c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T16:25:02.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.379+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f286006b9e0 con 0x7f286015fee0 2024-09-13T16:25:02.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.379+0000 7f285cff9700 1 -- 172.21.15.119:0/3008914994 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f285802a090 con 0x7f286015fee0 2024-09-13T16:25:02.382 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:02.382 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":10,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:02.142889+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:02.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 msgr2=0x7f2860148060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:02.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f2860148060 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f285800c7c0 tx=0x7f285800cad0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 shutdown_connections 2024-09-13T16:25:02.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f286015fee0 0x7f2860148060 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600714c0 0x7f286014b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 --2- 172.21.15.119:0/3008914994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2860070360 0x7f2860159eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 >> 172.21.15.119:0/3008914994 conn(0x7f28600ae4b0 msgr2=0x7f28600ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 shutdown_connections 2024-09-13T16:25:02.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:02.383+0000 7f28650be700 1 -- 172.21.15.119:0/3008914994 wait complete. 2024-09-13T16:25:02.401 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-13T16:25:02.401 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-13T16:25:02.472 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:02.469+0000 7f76c8a432c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T16:25:02.545 INFO:tasks.ceph.osd.0.smithi119.stderr:2024-09-13T16:25:02.543+0000 7fa111def880 -1 Falling back to public interface 2024-09-13T16:25:02.548 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-09-13T16:25:02.545+0000 7f3f1406f880 -1 Falling back to public interface 2024-09-13T16:25:02.560 INFO:tasks.ceph.osd.3.smithi119.stderr:2024-09-13T16:25:02.559+0000 7f6ae50f0880 -1 Falling back to public interface 2024-09-13T16:25:02.560 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-09-13T16:25:02.557+0000 7fd6bd0e5880 -1 Falling back to public interface 2024-09-13T16:25:02.575 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-09-13T16:25:02.573+0000 7f48f7a7a880 -1 Falling back to public interface 2024-09-13T16:25:02.642 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:02.639+0000 7f7f152612c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-13T16:25:02.769 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:02.767+0000 7f7f152612c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-13T16:25:02.787 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-09-13T16:25:02.785+0000 7f07b1483880 -1 Falling back to public interface 2024-09-13T16:25:02.793 INFO:tasks.ceph.osd.1.smithi119.stderr:2024-09-13T16:25:02.791+0000 7f58e26db880 -1 Falling back to public interface 2024-09-13T16:25:02.808 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-13T16:25:02.807+0000 7fcf4fbb6880 -1 Falling back to public interface 2024-09-13T16:25:02.846 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:02.841+0000 7f76c8a432c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T16:25:03.026 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:03.020+0000 7f76c8a432c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:25:03.095 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:03.088+0000 7f76c8a432c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T16:25:03.136 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:03.135+0000 7f7f152612c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-13T16:25:03.208 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:03.256 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:03.252+0000 7f76c8a432c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T16:25:03.305 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5adf59e700 1 Processor -- start 2024-09-13T16:25:03.305 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5adf59e700 1 -- start start 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5adf59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5ae00cc220 0x7f5ae015a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5adf59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5ae015a910 0x7f5ae00ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5adf59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae00b44e0 con 0x7f5ae01509e0 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5adf59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae00cddf0 con 0x7f5ae00cc220 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5adf59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae0071a50 con 0x7f5ae015a910 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5addd9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5ae00cc220 0x7f5ae015a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5addd9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5ae00cc220 0x7f5ae015a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37578/0 (socket says 172.21.15.119:37578) 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5addd9b700 1 -- 172.21.15.119:0/943839354 learned_addr learned my addr 172.21.15.119:0/943839354 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:03.307 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5aded9d700 1 --2- 172.21.15.119:0/943839354 >> v2:172.21.15.145:3300/0 conn(0x7f5ae015a910 0x7f5ae00ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5addd9b700 1 -- 172.21.15.119:0/943839354 >> v2:172.21.15.145:3300/0 conn(0x7f5ae015a910 msgr2=0x7f5ae00ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5addd9b700 1 --2- 172.21.15.119:0/943839354 >> v2:172.21.15.145:3300/0 conn(0x7f5ae015a910 0x7f5ae00ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5addd9b700 1 -- 172.21.15.119:0/943839354 >> v1:172.21.15.119:6789/0 conn(0x7f5ae01509e0 legacy=0x7f5ae0159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5add59a700 1 -- 172.21.15.119:0/943839354 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54906183 0 0) 0x7f5ae00b44e0 con 0x7f5ae01509e0 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5addd9b700 1 -- 172.21.15.119:0/943839354 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae00cc750 con 0x7f5ae00cc220 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.303+0000 7f5aded9d700 1 --2- 172.21.15.119:0/943839354 >> v2:172.21.15.145:3300/0 conn(0x7f5ae015a910 0x7f5ae00ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5addd9b700 1 --2- 172.21.15.119:0/943839354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae00cc220 0x7f5ae015a3d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00b2a70 tx=0x7f5ad4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8afd3966c194653d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5add59a700 1 -- 172.21.15.119:0/943839354 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400b8a0 con 0x7f5ae00cc220 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5add59a700 1 -- 172.21.15.119:0/943839354 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad400ba00 con 0x7f5ae00cc220 2024-09-13T16:25:03.308 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5add59a700 1 -- 172.21.15.119:0/943839354 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad400bd90 con 0x7f5ae00cc220 2024-09-13T16:25:03.309 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 -- 172.21.15.119:0/943839354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae00cc220 msgr2=0x7f5ae015a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:03.309 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 --2- 172.21.15.119:0/943839354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae00cc220 0x7f5ae015a3d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5ae00b2a70 tx=0x7f5ad4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.309 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 -- 172.21.15.119:0/943839354 shutdown_connections 2024-09-13T16:25:03.310 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 --2- 172.21.15.119:0/943839354 >> v2:172.21.15.145:3300/0 conn(0x7f5ae015a910 0x7f5ae00ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.310 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 --2- 172.21.15.119:0/943839354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae00cc220 0x7f5ae015a3d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.310 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 -- 172.21.15.119:0/943839354 >> 172.21.15.119:0/943839354 conn(0x7f5ae0070cc0 msgr2=0x7f5ae00710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:03.310 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 -- 172.21.15.119:0/943839354 shutdown_connections 2024-09-13T16:25:03.310 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 -- 172.21.15.119:0/943839354 wait complete. 2024-09-13T16:25:03.310 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.307+0000 7f5adf59e700 1 Processor -- start 2024-09-13T16:25:03.311 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 -- start start 2024-09-13T16:25:03.313 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ae00cc220 0x7f5ae0140c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:03.313 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae01509e0 0x7f5ae013d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5ae015a910 0x7f5ae006a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae013e350 con 0x7f5ae01509e0 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae00c4ab0 con 0x7f5ae00cc220 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae00c4d90 con 0x7f5ae015a910 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae01509e0 0x7f5ae013d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae01509e0 0x7f5ae013d8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37594/0 (socket says 172.21.15.119:37594) 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 -- 172.21.15.119:0/1290739711 learned_addr learned my addr 172.21.15.119:0/1290739711 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5ade59c700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ae00cc220 0x7f5ae0140c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5aded9d700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5ae015a910 0x7f5ae006a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 -- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5ae015a910 msgr2=0x7f5ae006a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:03.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5ae015a910 0x7f5ae006a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 -- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ae00cc220 msgr2=0x7f5ae0140c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:03.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ae00cc220 0x7f5ae0140c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 -- 172.21.15.119:0/1290739711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad4009570 con 0x7f5ae01509e0 2024-09-13T16:25:03.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5addd9b700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae01509e0 0x7f5ae013d8f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4000c00 tx=0x7f5ad4004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:03.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5aceffd700 1 -- 172.21.15.119:0/1290739711 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad40038c0 con 0x7f5ae01509e0 2024-09-13T16:25:03.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5aceffd700 1 -- 172.21.15.119:0/1290739711 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad401f070 con 0x7f5ae01509e0 2024-09-13T16:25:03.316 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae013e5f0 con 0x7f5ae01509e0 2024-09-13T16:25:03.316 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5aceffd700 1 -- 172.21.15.119:0/1290739711 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ad4003c60 con 0x7f5ae01509e0 2024-09-13T16:25:03.316 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae00c5070 con 0x7f5ae01509e0 2024-09-13T16:25:03.316 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5aceffd700 1 -- 172.21.15.119:0/1290739711 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 668+0+0 (secure 0 0 0) 0x7f5ad401a430 con 0x7f5ae01509e0 2024-09-13T16:25:03.316 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5aceffd700 1 -- 172.21.15.119:0/1290739711 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f5ad4021070 con 0x7f5ae01509e0 2024-09-13T16:25:03.317 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.311+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae013e800 con 0x7f5ae01509e0 2024-09-13T16:25:03.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.315+0000 7f5aceffd700 1 -- 172.21.15.119:0/1290739711 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73245 (secure 0 0 0) 0x7f5ad4044e30 con 0x7f5ae01509e0 2024-09-13T16:25:03.330 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:03.327+0000 7f7f152612c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-13T16:25:03.382 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.379+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ae00ce2d0 con 0x7f5ae01509e0 2024-09-13T16:25:03.383 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5aceffd700 1 -- 172.21.15.119:0/1290739711 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f5ad4033070 con 0x7f5ae01509e0 2024-09-13T16:25:03.383 INFO:teuthology.misc.health.smithi119.stdout: 2024-09-13T16:25:03.383 INFO:teuthology.misc.health.smithi119.stdout:{"epoch":10,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:02.142889+0000","last_up_change":"0.000000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:03.384 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae01509e0 msgr2=0x7f5ae013d8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:03.384 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae01509e0 0x7f5ae013d8f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4000c00 tx=0x7f5ad4004770 comp rx=0 tx=0).stop 2024-09-13T16:25:03.385 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 shutdown_connections 2024-09-13T16:25:03.385 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5ae015a910 0x7f5ae006a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.385 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5ae01509e0 0x7f5ae013d8f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.385 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 --2- 172.21.15.119:0/1290739711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5ae00cc220 0x7f5ae0140c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:03.385 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 >> 172.21.15.119:0/1290739711 conn(0x7f5ae0070cc0 msgr2=0x7f5ae00cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:03.385 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 shutdown_connections 2024-09-13T16:25:03.385 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:03.383+0000 7f5adf59e700 1 -- 172.21.15.119:0/1290739711 wait complete. 2024-09-13T16:25:03.394 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-13T16:25:03.395 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:03.395+0000 7f7f152612c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-13T16:25:03.555 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:03.555+0000 7f7f152612c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-13T16:25:03.933 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:03.928+0000 7f76c8a432c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T16:25:04.002 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:03.996+0000 7f76c8a432c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T16:25:04.164 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:04.160+0000 7f76c8a432c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T16:25:04.230 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:04.224+0000 7f76c8a432c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T16:25:04.305 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:04.300+0000 7f76c8a432c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T16:25:04.316 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:04.314+0000 7f7f152612c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-13T16:25:04.382 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:04.378+0000 7f7f152612c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-13T16:25:04.534 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:04.530+0000 7f7f152612c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-13T16:25:04.597 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:04.594+0000 7f7f152612c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-13T16:25:04.623 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:04.620+0000 7f76c8a432c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T16:25:04.669 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:04.666+0000 7f7f152612c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-13T16:25:04.694 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:04.692+0000 7f76c8a432c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T16:25:05.002 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:04.998+0000 7f7f152612c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-13T16:25:05.080 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:05.078+0000 7f7f152612c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-13T16:25:05.381 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:05.376+0000 7f76c8a432c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T16:25:05.802 INFO:tasks.ceph.mgr.y.smithi119.stderr:2024-09-13T16:25:05.798+0000 7f7f152612c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-13T16:25:05.928 INFO:tasks.ceph.osd.1.smithi119.stderr:2024-09-13T16:25:05.926+0000 7f58e26db880 -1 osd.1 0 log_to_monitors true 2024-09-13T16:25:06.137 INFO:tasks.ceph.osd.0.smithi119.stderr:2024-09-13T16:25:06.134+0000 7fa111def880 -1 osd.0 0 log_to_monitors true 2024-09-13T16:25:06.325 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-09-13T16:25:06.320+0000 7fd6bd0e5880 -1 osd.7 0 log_to_monitors true 2024-09-13T16:25:06.365 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-09-13T16:25:06.360+0000 7f3f1406f880 -1 osd.6 0 log_to_monitors true 2024-09-13T16:25:06.394 INFO:tasks.ceph.osd.3.smithi119.stderr:2024-09-13T16:25:06.390+0000 7f6ae50f0880 -1 osd.3 0 log_to_monitors true 2024-09-13T16:25:06.425 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:06.420+0000 7f76af5b8700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-13T16:25:06.425 INFO:tasks.ceph.mgr.x.smithi145.stderr:2024-09-13T16:25:06.420+0000 7f76af5b8700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-13T16:25:06.440 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-09-13T16:25:06.436+0000 7f48f7a7a880 -1 osd.4 0 log_to_monitors true 2024-09-13T16:25:06.633 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-09-13T16:25:06.628+0000 7f07b1483880 -1 osd.5 0 log_to_monitors true 2024-09-13T16:25:06.644 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-13T16:25:06.642+0000 7fcf4fbb6880 -1 osd.2 0 log_to_monitors true 2024-09-13T16:25:09.398 INFO:tasks.ceph.osd.3.smithi119.stderr:2024-09-13T16:25:09.394+0000 7f6ae0f2f700 -1 osd.3 0 waiting for initial osdmap 2024-09-13T16:25:09.406 INFO:tasks.ceph.osd.3.smithi119.stderr:2024-09-13T16:25:09.402+0000 7f6ad89de700 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:09.456 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-09-13T16:25:09.452+0000 7f48f38b9700 -1 osd.4 0 waiting for initial osdmap 2024-09-13T16:25:09.467 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-09-13T16:25:09.464+0000 7f3f0feae700 -1 osd.6 0 waiting for initial osdmap 2024-09-13T16:25:09.470 INFO:tasks.ceph.osd.6.smithi145.stderr:2024-09-13T16:25:09.468+0000 7f3f0795d700 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:09.471 INFO:tasks.ceph.osd.4.smithi145.stderr:2024-09-13T16:25:09.464+0000 7f48eb368700 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:09.483 INFO:tasks.ceph.osd.1.smithi119.stderr:2024-09-13T16:25:09.478+0000 7f58de51a700 -1 osd.1 0 waiting for initial osdmap 2024-09-13T16:25:09.488 INFO:tasks.ceph.osd.1.smithi119.stderr:2024-09-13T16:25:09.486+0000 7f58d5fc9700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:09.522 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-09-13T16:25:09.516+0000 7fd6b8f24700 -1 osd.7 0 waiting for initial osdmap 2024-09-13T16:25:09.528 INFO:tasks.ceph.osd.7.smithi145.stderr:2024-09-13T16:25:09.524+0000 7fd6b09d3700 -1 osd.7 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:09.571 INFO:tasks.ceph.osd.0.smithi119.stderr:2024-09-13T16:25:09.566+0000 7fa10e5b5700 -1 osd.0 0 waiting for initial osdmap 2024-09-13T16:25:09.577 INFO:tasks.ceph.osd.0.smithi119.stderr:2024-09-13T16:25:09.574+0000 7fa1056dd700 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:09.744 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-13T16:25:09.742+0000 7fcf4b9f5700 -1 osd.2 0 waiting for initial osdmap 2024-09-13T16:25:09.752 INFO:tasks.ceph.osd.2.smithi119.stderr:2024-09-13T16:25:09.750+0000 7fcf434a4700 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:09.768 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-09-13T16:25:09.764+0000 7f07adc49700 -1 osd.5 0 waiting for initial osdmap 2024-09-13T16:25:09.775 INFO:tasks.ceph.osd.5.smithi145.stderr:2024-09-13T16:25:09.772+0000 7f07a4d71700 -1 osd.5 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-13T16:25:10.199 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:10.313 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbed24700 1 Processor -- start 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbed24700 1 -- start start 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbed24700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fafb80cc060 0x7fafb80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbed24700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cde70 0x7fafb80b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbed24700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb806cf60 con 0x7fafb80adb30 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbed24700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb80cc4d0 con 0x7fafb80cc060 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbed24700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb806c350 con 0x7fafb80cde70 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbd521700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fafb80cc060 0x7fafb80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbd521700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fafb80cc060 0x7fafb80cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37632/0 (socket says 172.21.15.119:37632) 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.306+0000 7fafbd521700 1 -- 172.21.15.119:0/2491140630 learned_addr learned my addr 172.21.15.119:0/2491140630 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbe523700 1 --2- 172.21.15.119:0/2491140630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cde70 0x7fafb80b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbd521700 1 -- 172.21.15.119:0/2491140630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cde70 msgr2=0x7fafb80b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:10.314 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbd521700 1 --2- 172.21.15.119:0/2491140630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cde70 0x7fafb80b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbd521700 1 -- 172.21.15.119:0/2491140630 >> v1:172.21.15.145:6789/0 conn(0x7fafb80adb30 legacy=0x7fafb806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbd521700 1 -- 172.21.15.119:0/2491140630 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb806c630 con 0x7fafb80cc060 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbd521700 1 --2- 172.21.15.119:0/2491140630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cc060 0x7fafb80cd930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009910 tx=0x7fafa8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e3bda11bdf84e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbcd20700 1 -- 172.21.15.119:0/2491140630 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa800ba60 con 0x7fafb80cc060 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbcd20700 1 -- 172.21.15.119:0/2491140630 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa800bbc0 con 0x7fafb80cc060 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbcd20700 1 -- 172.21.15.119:0/2491140630 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8009e60 con 0x7fafb80cc060 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 -- 172.21.15.119:0/2491140630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cc060 msgr2=0x7fafb80cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 --2- 172.21.15.119:0/2491140630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cc060 0x7fafb80cd930 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009910 tx=0x7fafa8009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 -- 172.21.15.119:0/2491140630 shutdown_connections 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 --2- 172.21.15.119:0/2491140630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cde70 0x7fafb80b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 --2- 172.21.15.119:0/2491140630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cc060 0x7fafb80cd930 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 -- 172.21.15.119:0/2491140630 >> 172.21.15.119:0/2491140630 conn(0x7fafb806a350 msgr2=0x7fafb806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:10.315 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 -- 172.21.15.119:0/2491140630 shutdown_connections 2024-09-13T16:25:10.316 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 -- 172.21.15.119:0/2491140630 wait complete. 2024-09-13T16:25:10.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 Processor -- start 2024-09-13T16:25:10.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.310+0000 7fafbed24700 1 -- start start 2024-09-13T16:25:10.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb80adb30 0x7fafb80ad080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cc060 0x7fafb80aa680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cde70 0x7fafb8159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80ad5c0 con 0x7fafb80cde70 2024-09-13T16:25:10.318 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb80aabc0 con 0x7fafb80adb30 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb81f22a0 con 0x7fafb80cc060 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cde70 0x7fafb8159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cde70 0x7fafb8159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37642/0 (socket says 172.21.15.119:37642) 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 -- 172.21.15.119:0/1783820444 learned_addr learned my addr 172.21.15.119:0/1783820444 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbd521700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cc060 0x7fafb80aa680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbdd22700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb80adb30 0x7fafb80ad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 -- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cc060 msgr2=0x7fafb80aa680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cc060 0x7fafb80aa680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 -- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb80adb30 msgr2=0x7fafb80ad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb80adb30 0x7fafb80ad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 -- 172.21.15.119:0/1783820444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8009570 con 0x7fafb80cde70 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbe523700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cde70 0x7fafb8159f90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafb0007e60 tx=0x7fafb000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.319 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafae7fc700 1 -- 172.21.15.119:0/1783820444 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb000ed30 con 0x7fafb80cde70 2024-09-13T16:25:10.320 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafae7fc700 1 -- 172.21.15.119:0/1783820444 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafb0014420 con 0x7fafb80cde70 2024-09-13T16:25:10.320 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb81f2560 con 0x7fafb80cde70 2024-09-13T16:25:10.320 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafae7fc700 1 -- 172.21.15.119:0/1783820444 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb00147d0 con 0x7fafb80cde70 2024-09-13T16:25:10.320 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb81f2b20 con 0x7fafb80cde70 2024-09-13T16:25:10.320 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.314+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb80adf10 con 0x7fafb80cde70 2024-09-13T16:25:10.320 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.318+0000 7fafae7fc700 1 -- 172.21.15.119:0/1783820444 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 86036+0+0 (secure 0 0 0) 0x7fafb0014930 con 0x7fafb80cde70 2024-09-13T16:25:10.321 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.318+0000 7fafae7fc700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faf9c0666a0 0x7faf9c068b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.321 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.318+0000 7fafae7fc700 1 -- 172.21.15.119:0/1783820444 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2231+0+0 (secure 0 0 0) 0x7fafb00838e0 con 0x7fafb80cde70 2024-09-13T16:25:10.321 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.318+0000 7fafbdd22700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faf9c0666a0 0x7faf9c068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.322 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.318+0000 7fafbdd22700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faf9c0666a0 0x7faf9c068b60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fafb80b3ca0 tx=0x7fafb4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.330 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.326+0000 7fafae7fc700 1 -- 172.21.15.119:0/1783820444 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fafb0051ab0 con 0x7fafb80cde70 2024-09-13T16:25:10.513 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.510+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fafb80b2770 con 0x7fafb80cde70 2024-09-13T16:25:10.516 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.510+0000 7fafae7fc700 1 -- 172.21.15.119:0/1783820444 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10968 (secure 0 0 0) 0x7fafb00550d0 con 0x7fafb80cde70 2024-09-13T16:25:10.516 INFO:teuthology.misc.health.smithi119.stdout: 2024-09-13T16:25:10.516 INFO:teuthology.misc.health.smithi119.stdout:{"epoch":14,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:10.404143+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:10.517 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faf9c0666a0 msgr2=0x7faf9c068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faf9c0666a0 0x7faf9c068b60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fafb80b3ca0 tx=0x7fafb4009040 comp rx=0 tx=0).stop 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cde70 msgr2=0x7fafb8159f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cde70 0x7fafb8159f90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fafb0007e60 tx=0x7fafb000e3c0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 shutdown_connections 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faf9c0666a0 0x7faf9c068b60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb80cde70 0x7fafb8159f90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb80cc060 0x7fafb80aa680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 --2- 172.21.15.119:0/1783820444 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb80adb30 0x7fafb80ad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 >> 172.21.15.119:0/1783820444 conn(0x7fafb806a350 msgr2=0x7fafb8066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 shutdown_connections 2024-09-13T16:25:10.518 INFO:teuthology.misc.health.smithi119.stderr:2024-09-13T16:25:10.514+0000 7fafbed24700 1 -- 172.21.15.119:0/1783820444 wait complete. 2024-09-13T16:25:10.529 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-13T16:25:10.529 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-13T16:25:10.529 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:25:10.530 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:25:10.532 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:25:10.532 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:25:10.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:25:10.534 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:25:10.534 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:25:10.534 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:25:10.537 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:25:10.537 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:25:10.537 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:25:10.539 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:25:10.539 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-09-13T16:25:10.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:10.610 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:25:10.622 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:25:10.622 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:25:10.628 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:25:10.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f1aae700 1 Processor -- start 2024-09-13T16:25:10.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f1aae700 1 -- start start 2024-09-13T16:25:10.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f1aae700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80ec1509e0 0x7f80ec159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f1aae700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0b44e0 con 0x7f80ec15a910 2024-09-13T16:25:10.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f1aae700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0cddf0 con 0x7f80ec0cc220 2024-09-13T16:25:10.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f1aae700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80ec071a50 con 0x7f80ec1509e0 2024-09-13T16:25:10.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f0aac700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80ec1509e0 0x7f80ec159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f0aac700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80ec1509e0 0x7f80ec159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37696/0 (socket says 172.21.15.119:37696) 2024-09-13T16:25:10.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f0aac700 1 -- 172.21.15.119:0/1920284357 learned_addr learned my addr 172.21.15.119:0/1920284357 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:10.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f0aac700 1 -- 172.21.15.119:0/1920284357 >> v1:172.21.15.145:6789/0 conn(0x7f80ec0cc220 legacy=0x7f80ec15a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:10.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f0aac700 1 -- 172.21.15.119:0/1920284357 >> v1:172.21.15.119:6789/0 conn(0x7f80ec15a910 legacy=0x7f80ec0ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f0aac700 1 -- 172.21.15.119:0/1920284357 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0cc750 con 0x7f80ec1509e0 2024-09-13T16:25:10.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80f0aac700 1 --2- 172.21.15.119:0/1920284357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec159cc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f80e000bbb0 tx=0x7f80e000bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78fff416f3e61802 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80eb7fe700 1 -- 172.21.15.119:0/1920284357 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e000dd50 con 0x7f80ec1509e0 2024-09-13T16:25:10.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80eb7fe700 1 -- 172.21.15.119:0/1920284357 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e0004030 con 0x7f80ec1509e0 2024-09-13T16:25:10.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.658+0000 7f80eb7fe700 1 -- 172.21.15.119:0/1920284357 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e00043e0 con 0x7f80ec1509e0 2024-09-13T16:25:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- 172.21.15.119:0/1920284357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 msgr2=0x7f80ec159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 --2- 172.21.15.119:0/1920284357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec159cc0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f80e000bbb0 tx=0x7f80e000bec0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- 172.21.15.119:0/1920284357 shutdown_connections 2024-09-13T16:25:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 --2- 172.21.15.119:0/1920284357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec159cc0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- 172.21.15.119:0/1920284357 >> 172.21.15.119:0/1920284357 conn(0x7f80ec070cc0 msgr2=0x7f80ec0710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- 172.21.15.119:0/1920284357 shutdown_connections 2024-09-13T16:25:10.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- 172.21.15.119:0/1920284357 wait complete. 2024-09-13T16:25:10.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 Processor -- start 2024-09-13T16:25:10.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- start start 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80ec0cc220 0x7f80ec0aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec155290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80ec15a910 0x7f80ec1557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec0ae3f0 con 0x7f80ec1509e0 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec14ca30 con 0x7f80ec15a910 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f1aae700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80ec0ad770 con 0x7f80ec0cc220 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec155290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec155290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37702/0 (socket says 172.21.15.119:37702) 2024-09-13T16:25:10.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 -- 172.21.15.119:0/2723941174 learned_addr learned my addr 172.21.15.119:0/2723941174 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f12ad700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80ec15a910 0x7f80ec1557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f0aac700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80ec0cc220 0x7f80ec0aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 -- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80ec0cc220 msgr2=0x7f80ec0aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80ec0cc220 0x7f80ec0aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 -- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80ec15a910 msgr2=0x7f80ec1557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80ec15a910 0x7f80ec1557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 -- 172.21.15.119:0/2723941174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e000b810 con 0x7f80ec1509e0 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f12ad700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80ec15a910 0x7f80ec1557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80f0aac700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80ec0cc220 0x7f80ec0aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80ebfff700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec155290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f80d800c4e0 tx=0x7f80d800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.662+0000 7f80e97fa700 1 -- 172.21.15.119:0/2723941174 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8011790 con 0x7f80ec1509e0 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80e97fa700 1 -- 172.21.15.119:0/2723941174 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d8011910 con 0x7f80ec1509e0 2024-09-13T16:25:10.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec0ada50 con 0x7f80ec1509e0 2024-09-13T16:25:10.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80e97fa700 1 -- 172.21.15.119:0/2723941174 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8011cc0 con 0x7f80ec1509e0 2024-09-13T16:25:10.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec0ae010 con 0x7f80ec1509e0 2024-09-13T16:25:10.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80e97fa700 1 -- 172.21.15.119:0/2723941174 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 86036+0+0 (secure 0 0 0) 0x7f80d801c460 con 0x7f80ec1509e0 2024-09-13T16:25:10.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80ec0ae530 con 0x7f80ec1509e0 2024-09-13T16:25:10.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80e97fa700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f80d00666a0 0x7f80d0068b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80e97fa700 1 -- 172.21.15.119:0/2723941174 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f80d8083aa0 con 0x7f80ec1509e0 2024-09-13T16:25:10.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80f0aac700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f80d00666a0 0x7f80d0068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.666+0000 7f80f0aac700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f80d00666a0 0x7f80d0068b60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e000b360 tx=0x7f80e0013000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.670+0000 7f80e97fa700 1 -- 172.21.15.119:0/2723941174 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f80d8051320 con 0x7f80ec1509e0 2024-09-13T16:25:10.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.842+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80ec0ce2b0 con 0x7f80ec1509e0 2024-09-13T16:25:10.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.842+0000 7f80e97fa700 1 -- 172.21.15.119:0/2723941174 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10968 (secure 0 0 0) 0x7f80d8054940 con 0x7f80ec1509e0 2024-09-13T16:25:10.847 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:10.847 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":14,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:10.404143+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:10.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f80d00666a0 msgr2=0x7f80d0068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f80d00666a0 0x7f80d0068b60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e000b360 tx=0x7f80e0013000 comp rx=0 tx=0).stop 2024-09-13T16:25:10.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 msgr2=0x7f80ec155290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec155290 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f80d800c4e0 tx=0x7f80d800c8a0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 shutdown_connections 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f80d00666a0 0x7f80d0068b60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80ec15a910 0x7f80ec1557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80ec1509e0 0x7f80ec155290 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 --2- 172.21.15.119:0/2723941174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80ec0cc220 0x7f80ec0aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 >> 172.21.15.119:0/2723941174 conn(0x7f80ec070cc0 msgr2=0x7f80ec0af710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 shutdown_connections 2024-09-13T16:25:10.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.846+0000 7f80f1aae700 1 -- 172.21.15.119:0/2723941174 wait complete. 2024-09-13T16:25:10.862 INFO:tasks.cephfs.filesystem.ceph_manager:[] 2024-09-13T16:25:10.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.970+0000 7ff661907700 1 Processor -- start 2024-09-13T16:25:10.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.970+0000 7ff661907700 1 -- start start 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.970+0000 7ff661907700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff65c0cc060 0x7ff65c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.970+0000 7ff661907700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cde70 0x7ff65c0b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.970+0000 7ff661907700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65c06cf60 con 0x7ff65c0adb30 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.970+0000 7ff661907700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff65c0cc4d0 con 0x7ff65c0cc060 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.970+0000 7ff661907700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c06c350 con 0x7ff65c0cde70 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65bfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff65c0cc060 0x7ff65c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff660905700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff65c0adb30 0x7ff65c06bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:45592/0 (socket says 172.21.15.119:45592) 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff660905700 1 -- 172.21.15.119:0/2198906287 learned_addr learned my addr 172.21.15.119:0/2198906287 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:10.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661106700 1 --2- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cde70 0x7ff65c0b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65bfff700 1 -- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cde70 msgr2=0x7ff65c0b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65bfff700 1 --2- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cde70 0x7ff65c0b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65bfff700 1 -- 172.21.15.119:0/2198906287 >> v1:172.21.15.145:6789/0 conn(0x7ff65c0adb30 legacy=0x7ff65c06bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65bfff700 1 -- 172.21.15.119:0/2198906287 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65c06c630 con 0x7ff65c0cc060 2024-09-13T16:25:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65bfff700 1 --2- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cc060 0x7ff65c0cd930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009850 tx=0x7ff64c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99afd42b66680043 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65b7fe700 1 -- 172.21.15.119:0/2198906287 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c011830 con 0x7ff65c0cc060 2024-09-13T16:25:10.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65b7fe700 1 -- 172.21.15.119:0/2198906287 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c011990 con 0x7ff65c0cc060 2024-09-13T16:25:10.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff65b7fe700 1 -- 172.21.15.119:0/2198906287 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff64c011d40 con 0x7ff65c0cc060 2024-09-13T16:25:10.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 -- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cc060 msgr2=0x7ff65c0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 --2- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cc060 0x7ff65c0cd930 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff64c009850 tx=0x7ff64c009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:10.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 -- 172.21.15.119:0/2198906287 shutdown_connections 2024-09-13T16:25:10.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 --2- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cde70 0x7ff65c0b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 --2- 172.21.15.119:0/2198906287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cc060 0x7ff65c0cd930 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 -- 172.21.15.119:0/2198906287 >> 172.21.15.119:0/2198906287 conn(0x7ff65c06a350 msgr2=0x7ff65c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:10.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 -- 172.21.15.119:0/2198906287 shutdown_connections 2024-09-13T16:25:10.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 -- 172.21.15.119:0/2198906287 wait complete. 2024-09-13T16:25:10.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.974+0000 7ff661907700 1 Processor -- start 2024-09-13T16:25:10.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 -- start start 2024-09-13T16:25:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff65c0adb30 0x7ff65c13f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cc060 0x7ff65c0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cde70 0x7ff65c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c13f990 con 0x7ff65c0adb30 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c0717b0 con 0x7ff65c0cde70 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65c071ca0 con 0x7ff65c0cc060 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff65c0adb30 0x7ff65c13f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff65c0adb30 0x7ff65c13f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37712/0 (socket says 172.21.15.119:37712) 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 -- 172.21.15.119:0/2132023703 learned_addr learned my addr 172.21.15.119:0/2132023703 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661106700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cde70 0x7ff65c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff65bfff700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cc060 0x7ff65c0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 -- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cc060 msgr2=0x7ff65c0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cc060 0x7ff65c0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 -- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cde70 msgr2=0x7ff65c0ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cde70 0x7ff65c0ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 -- 172.21.15.119:0/2132023703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64c0094b0 con 0x7ff65c0adb30 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661106700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cde70 0x7ff65c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff660905700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff65c0adb30 0x7ff65c13f450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff65000c500 tx=0x7ff65000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff6597fa700 1 -- 172.21.15.119:0/2132023703 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650005d90 con 0x7ff65c0adb30 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff6597fa700 1 -- 172.21.15.119:0/2132023703 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6500070d0 con 0x7ff65c0adb30 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff65c0ca690 con 0x7ff65c0adb30 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff6597fa700 1 -- 172.21.15.119:0/2132023703 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff650007460 con 0x7ff65c0adb30 2024-09-13T16:25:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.978+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff65c142cf0 con 0x7ff65c0adb30 2024-09-13T16:25:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.982+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff65c0adf10 con 0x7ff65c0adb30 2024-09-13T16:25:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.982+0000 7ff6597fa700 1 -- 172.21.15.119:0/2132023703 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 4) v1 ==== 86036+0+0 (secure 0 0 0) 0x7ff6500075c0 con 0x7ff65c0adb30 2024-09-13T16:25:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.982+0000 7ff6597fa700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6480666a0 0x7ff648068b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.982+0000 7ff6597fa700 1 -- 172.21.15.119:0/2132023703 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4471+0+0 (secure 0 0 0) 0x7ff650082fe0 con 0x7ff65c0adb30 2024-09-13T16:25:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.982+0000 7ff65bfff700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6480666a0 0x7ff648068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:10.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.982+0000 7ff65bfff700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6480666a0 0x7ff648068b60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0090b0 tx=0x7ff64c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:10.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:10.986+0000 7ff6597fa700 1 -- 172.21.15.119:0/2132023703 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff6500b4db0 con 0x7ff65c0adb30 2024-09-13T16:25:11.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.190+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff65c0ca8e0 con 0x7ff65c0adb30 2024-09-13T16:25:11.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.190+0000 7ff6597fa700 1 -- 172.21.15.119:0/2132023703 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1259 (secure 0 0 0) 0x7ff650017090 con 0x7ff65c0adb30 2024-09-13T16:25:11.194 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:11.194 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2024-09-13T16:25:11.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6480666a0 msgr2=0x7ff648068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:11.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6480666a0 0x7ff648068b60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff64c0090b0 tx=0x7ff64c008030 comp rx=0 tx=0).stop 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff65c0adb30 msgr2=0x7ff65c13f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff65c0adb30 0x7ff65c13f450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff65000c500 tx=0x7ff65000c810 comp rx=0 tx=0).stop 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 shutdown_connections 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6480666a0 0x7ff648068b60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff65c0cde70 0x7ff65c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff65c0cc060 0x7ff65c0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 --2- 172.21.15.119:0/2132023703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff65c0adb30 0x7ff65c13f450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 >> 172.21.15.119:0/2132023703 conn(0x7ff65c06a350 msgr2=0x7ff65c0ae870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 shutdown_connections 2024-09-13T16:25:11.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:11.194+0000 7ff661907700 1 -- 172.21.15.119:0/2132023703 wait complete. 2024-09-13T16:25:11.198 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 2 2024-09-13T16:25:13.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:13.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.326+0000 7f18b759e700 1 Processor -- start 2024-09-13T16:25:13.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.326+0000 7f18b759e700 1 -- start start 2024-09-13T16:25:13.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f18b8154f50 0x7f18b80cdad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18b815e530 0x7f18b815e960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815eea0 0x7f18b815af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b80b03e0 con 0x7f18b8154f50 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18b80ce150 con 0x7f18b815e530 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b80ceec0 con 0x7f18b815eea0 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f18b8154f50 0x7f18b80cdad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b5d9b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18b815e530 0x7f18b815e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b6d9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815eea0 0x7f18b815af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b5d9b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18b815e530 0x7f18b815e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:37148/0 (socket says 172.21.15.119:37148) 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b5d9b700 1 -- 172.21.15.119:0/1994474876 learned_addr learned my addr 172.21.15.119:0/1994474876 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f18b8154f50 0x7f18b80cdad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35870/0 (socket says 172.21.15.119:35870) 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 -- 172.21.15.119:0/1994474876 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815eea0 msgr2=0x7f18b815af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 --2- 172.21.15.119:0/1994474876 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815eea0 0x7f18b815af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 -- 172.21.15.119:0/1994474876 >> v2:172.21.15.145:3300/0 conn(0x7f18b815e530 msgr2=0x7f18b815e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 --2- 172.21.15.119:0/1994474876 >> v2:172.21.15.145:3300/0 conn(0x7f18b815e530 0x7f18b815e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 -- 172.21.15.119:0/1994474876 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b80cf1a0 con 0x7f18b8154f50 2024-09-13T16:25:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b659c700 1 --2- 172.21.15.119:0/1994474876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8154f50 0x7f18b80cdad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18b80710c0 tx=0x7f18b000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54f09568aa01076e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b559a700 1 -- 172.21.15.119:0/1994474876 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b000e070 con 0x7f18b8154f50 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b559a700 1 -- 172.21.15.119:0/1994474876 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b000bb20 con 0x7f18b8154f50 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b559a700 1 -- 172.21.15.119:0/1994474876 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0012430 con 0x7f18b8154f50 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- 172.21.15.119:0/1994474876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8154f50 msgr2=0x7f18b80cdad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 --2- 172.21.15.119:0/1994474876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8154f50 0x7f18b80cdad0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f18b80710c0 tx=0x7f18b000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- 172.21.15.119:0/1994474876 shutdown_connections 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 --2- 172.21.15.119:0/1994474876 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815eea0 0x7f18b815af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 --2- 172.21.15.119:0/1994474876 >> v2:172.21.15.145:3300/0 conn(0x7f18b815e530 0x7f18b815e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 --2- 172.21.15.119:0/1994474876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b8154f50 0x7f18b80cdad0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- 172.21.15.119:0/1994474876 >> 172.21.15.119:0/1994474876 conn(0x7f18b80b2570 msgr2=0x7f18b80b2980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- 172.21.15.119:0/1994474876 shutdown_connections 2024-09-13T16:25:13.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.330+0000 7f18b759e700 1 -- 172.21.15.119:0/1994474876 wait complete. 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 Processor -- start 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 -- start start 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b8154f50 0x7f18b814e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815e530 0x7f18b80b4240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b815eea0 0x7f18b81483d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b814df50 con 0x7f18b815eea0 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18b814d340 con 0x7f18b8154f50 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18b814d620 con 0x7f18b815e530 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b815eea0 0x7f18b81483d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b815eea0 0x7f18b81483d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35884/0 (socket says 172.21.15.119:35884) 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b5d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815e530 0x7f18b80b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b6d9d700 1 -- 172.21.15.119:0/2052562971 learned_addr learned my addr 172.21.15.119:0/2052562971 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:13.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b659c700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b8154f50 0x7f18b814e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b659c700 1 -- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815e530 msgr2=0x7f18b80b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b659c700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815e530 0x7f18b80b4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b659c700 1 -- 172.21.15.119:0/2052562971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b815eea0 msgr2=0x7f18b81483d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b659c700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b815eea0 0x7f18b81483d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b659c700 1 -- 172.21.15.119:0/2052562971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18b0009a70 con 0x7f18b8154f50 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b6d9d700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b815eea0 0x7f18b81483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b659c700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b8154f50 0x7f18b814e3e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18b0016000 tx=0x7f18b0004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18aeffd700 1 -- 172.21.15.119:0/2052562971 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0012430 con 0x7f18b8154f50 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18aeffd700 1 -- 172.21.15.119:0/2052562971 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b0020070 con 0x7f18b8154f50 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18b814d900 con 0x7f18b8154f50 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18aeffd700 1 -- 172.21.15.119:0/2052562971 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18b0007000 con 0x7f18b8154f50 2024-09-13T16:25:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.334+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18b80b4780 con 0x7f18b8154f50 2024-09-13T16:25:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.338+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18b80b4970 con 0x7f18b8154f50 2024-09-13T16:25:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.338+0000 7f18aeffd700 1 -- 172.21.15.119:0/2052562971 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 4) v1 ==== 86036+0+0 (secure 0 0 0) 0x7f18b0004280 con 0x7f18b8154f50 2024-09-13T16:25:13.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.338+0000 7f18aeffd700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f18980666a0 0x7f1898068b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.338+0000 7f18aeffd700 1 -- 172.21.15.119:0/2052562971 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f18b0022070 con 0x7f18b8154f50 2024-09-13T16:25:13.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.338+0000 7f18b5d9b700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f18980666a0 0x7f1898068b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.338+0000 7f18b5d9b700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f18980666a0 0x7f1898068b60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f18b815e490 tx=0x7f18a00092e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:13.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.346+0000 7f18aeffd700 1 -- 172.21.15.119:0/2052562971 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f18b00537c0 con 0x7f18b8154f50 2024-09-13T16:25:13.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.562+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18b814d760 con 0x7f18b8154f50 2024-09-13T16:25:13.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.562+0000 7f18aeffd700 1 -- 172.21.15.119:0/2052562971 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3536 (secure 0 0 0) 0x7f18b814d760 con 0x7f18b8154f50 2024-09-13T16:25:13.565 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:13.566 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4285,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2024-09-13T16:25:13.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f18980666a0 msgr2=0x7f1898068b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f18980666a0 0x7f1898068b60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f18b815e490 tx=0x7f18a00092e0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b8154f50 msgr2=0x7f18b814e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b8154f50 0x7f18b814e3e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f18b0016000 tx=0x7f18b0004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 shutdown_connections 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f18980666a0 0x7f1898068b60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18b815eea0 0x7f18b81483d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18b815e530 0x7f18b80b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 --2- 172.21.15.119:0/2052562971 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18b8154f50 0x7f18b814e3e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 >> 172.21.15.119:0/2052562971 conn(0x7f18b80b2570 msgr2=0x7f18b814b950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 shutdown_connections 2024-09-13T16:25:13.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.566+0000 7f18b759e700 1 -- 172.21.15.119:0/2052562971 wait complete. 2024-09-13T16:25:13.570 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 3 2024-09-13T16:25:13.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:13.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 Processor -- start 2024-09-13T16:25:13.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- start start 2024-09-13T16:25:13.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 0x7f78c40d1c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40d22b0 con 0x7f78c4164440 2024-09-13T16:25:13.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c406b610 con 0x7f78c40b17b0 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c406bf40 con 0x7f78c4164820 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f78c4164440 0x7f78c40d2c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49512/0 (socket says 172.21.15.119:49512) 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 -- 172.21.15.119:0/3830294942 learned_addr learned my addr 172.21.15.119:0/3830294942 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794815914 0 0) 0x7f78c40d22b0 con 0x7f78c4164440 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78ac0035d0 con 0x7f78c4164440 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c8cd4700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 0x7f78c40d1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714729887 0 0) 0x7f78ac0035d0 con 0x7f78c4164440 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 msgr2=0x7f78c40d1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 0x7f78c40d1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 >> v1:172.21.15.145:6789/0 conn(0x7f78c40b17b0 legacy=0x7f78c40d33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c406b8f0 con 0x7f78c4164440 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f78b4001690 con 0x7f78c4164440 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 >> v1:172.21.15.119:6789/0 conn(0x7f78c4164440 legacy=0x7f78c40d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78ac003a50 0x7f78ac003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78ac004870 0x7f78ac004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 0x7f78ac005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c406bf40 con 0x7f78ac003a50 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac00bc20 con 0x7f78ac004870 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78ac00bde0 con 0x7f78c4164820 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78ac003a50 0x7f78ac003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c37fe700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78ac004870 0x7f78ac004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c8cd4700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 0x7f78ac005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 -- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 msgr2=0x7f78ac005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78c4164820 0x7f78ac005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 -- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78ac004870 msgr2=0x7f78ac004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78ac004870 0x7f78ac004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 -- 172.21.15.119:0/3830294942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac0035d0 con 0x7f78ac003a50 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78ac003a50 0x7f78ac003e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78b400c2f0 tx=0x7f78ac0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf0d1c7eef7b2c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b400ce30 con 0x7f78ac003a50 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b4007bc0 con 0x7f78ac003a50 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c2ffd700 1 -- 172.21.15.119:0/3830294942 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b4005bc0 con 0x7f78ac003a50 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- 172.21.15.119:0/3830294942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78ac003a50 msgr2=0x7f78ac003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 --2- 172.21.15.119:0/3830294942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78ac003a50 0x7f78ac003e70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78b400c2f0 tx=0x7f78ac0058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c8cd4700 1 -- 172.21.15.119:0/3830294942 reap_dead start 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- 172.21.15.119:0/3830294942 shutdown_connections 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- 172.21.15.119:0/3830294942 >> 172.21.15.119:0/3830294942 conn(0x7f78c408c240 msgr2=0x7f78c409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- 172.21.15.119:0/3830294942 shutdown_connections 2024-09-13T16:25:13.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.682+0000 7f78c94d5700 1 -- 172.21.15.119:0/3830294942 wait complete. 2024-09-13T16:25:13.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 Processor -- start 2024-09-13T16:25:13.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 -- start start 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78ac003a50 0x7f78c415d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78c40c95c0 0x7f78c40c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78c40cb490 0x7f78c40cb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c40cbe80 con 0x7f78c40cb490 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c40cd5f0 con 0x7f78c40c95c0 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c40ca720 con 0x7f78ac003a50 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78c40cb490 0x7f78c40cb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78c40cb490 0x7f78c40cb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35914/0 (socket says 172.21.15.119:35914) 2024-09-13T16:25:13.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 -- 172.21.15.119:0/3092929573 learned_addr learned my addr 172.21.15.119:0/3092929573 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78ac003a50 0x7f78c415d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c37fe700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78c40c95c0 0x7f78c40c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 -- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78ac003a50 msgr2=0x7f78c415d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78ac003a50 0x7f78c415d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 -- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78c40c95c0 msgr2=0x7f78c40c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78c40c95c0 0x7f78c40c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 -- 172.21.15.119:0/3092929573 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40caa00 con 0x7f78c40cb490 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c37fe700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78c40c95c0 0x7f78c40c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78ac003a50 0x7f78c415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c8cd4700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78c40cb490 0x7f78c40cb940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f78c40d2be0 tx=0x7f78bc0049f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c0ff9700 1 -- 172.21.15.119:0/3092929573 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc007c00 con 0x7f78c40cb490 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c0ff9700 1 -- 172.21.15.119:0/3092929573 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78bc00f050 con 0x7f78c40cb490 2024-09-13T16:25:13.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c40cace0 con 0x7f78c40cb490 2024-09-13T16:25:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c0ff9700 1 -- 172.21.15.119:0/3092929573 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78bc014410 con 0x7f78c40cb490 2024-09-13T16:25:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.686+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c415d5e0 con 0x7f78c40cb490 2024-09-13T16:25:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.690+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c40cd7f0 con 0x7f78c40cb490 2024-09-13T16:25:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.690+0000 7f78c0ff9700 1 -- 172.21.15.119:0/3092929573 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f78bc014570 con 0x7f78c40cb490 2024-09-13T16:25:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.690+0000 7f78c0ff9700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78b0066730 0x7f78b0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.690+0000 7f78c0ff9700 1 -- 172.21.15.119:0/3092929573 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f78bc0840e0 con 0x7f78c40cb490 2024-09-13T16:25:13.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.694+0000 7f78c0ff9700 1 -- 172.21.15.119:0/3092929573 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f78c40cd7f0 con 0x7f78c40cb490 2024-09-13T16:25:13.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.714+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78b0066730 0x7f78b0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:13.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.718+0000 7f78c3fff700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78b0066730 0x7f78b0068bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78b4008fd0 tx=0x7f78b4007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:13.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.866+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f78c40cd7b0 con 0x7f78c40cb490 2024-09-13T16:25:13.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.866+0000 7f78c0ff9700 1 -- 172.21.15.119:0/3092929573 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+12910 (secure 0 0 0) 0x7f78bc051700 con 0x7f78c40cb490 2024-09-13T16:25:13.870 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:13.870 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":17,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:13.634963+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:13.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78b0066730 msgr2=0x7f78b0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78b0066730 0x7f78b0068bf0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f78b4008fd0 tx=0x7f78b4007000 comp rx=0 tx=0).stop 2024-09-13T16:25:13.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78c40cb490 msgr2=0x7f78c40cb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78c40cb490 0x7f78c40cb940 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f78c40d2be0 tx=0x7f78bc0049f0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 shutdown_connections 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78b0066730 0x7f78b0068bf0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78c40cb490 0x7f78c40cb940 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78c40c95c0 0x7f78c40c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 --2- 172.21.15.119:0/3092929573 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78ac003a50 0x7f78c415d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 >> 172.21.15.119:0/3092929573 conn(0x7f78c408c240 msgr2=0x7f78c40b2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 shutdown_connections 2024-09-13T16:25:13.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.870+0000 7f78c94d5700 1 -- 172.21.15.119:0/3092929573 wait complete. 2024-09-13T16:25:13.884 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:25:13.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:14.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.998+0000 7f2cb6524700 1 Processor -- start 2024-09-13T16:25:14.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:13.998+0000 7f2cb6524700 1 -- start start 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb6524700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2cb006a5c0 0x7f2cb00cec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb6524700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 0x7f2cb00cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb6524700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb00cf850 con 0x7f2cb006a9a0 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb6524700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb015f340 con 0x7f2cb006a5c0 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb6524700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb006cb60 con 0x7f2cb0154f60 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb4d21700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2cb006a9a0 0x7f2cb0154850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49518/0 (socket says 172.21.15.119:49518) 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb4d21700 1 -- 172.21.15.119:0/2137492987 learned_addr learned my addr 172.21.15.119:0/2137492987 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5522700 1 --2- 172.21.15.119:0/2137492987 >> v2:172.21.15.145:3300/0 conn(0x7f2cb006a5c0 0x7f2cb00cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077669607 0 0) 0x7f2cb00cf850 con 0x7f2cb006a9a0 2024-09-13T16:25:14.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5d23700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 0x7f2cb00cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c9c0035d0 con 0x7f2cb006a9a0 2024-09-13T16:25:14.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3274944389 0 0) 0x7f2c9c0035d0 con 0x7f2cb006a9a0 2024-09-13T16:25:14.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 msgr2=0x7f2cb00cf1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:14.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 0x7f2cb00cf1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 >> v2:172.21.15.145:3300/0 conn(0x7f2cb006a5c0 msgr2=0x7f2cb00cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:14.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 --2- 172.21.15.119:0/2137492987 >> v2:172.21.15.145:3300/0 conn(0x7f2cb006a5c0 0x7f2cb00cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb015aee0 con 0x7f2cb006a9a0 2024-09-13T16:25:14.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f2ca0001670 con 0x7f2cb006a9a0 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 >> v1:172.21.15.119:6789/0 conn(0x7f2cb006a9a0 legacy=0x7f2cb0154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c9c003a20 0x7f2c9c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 0x7f2c9c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c004db0 0x7f2c9c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb015f340 con 0x7f2c9c004db0 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c00ba30 con 0x7f2c9c003a20 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c00bb70 con 0x7f2cb0154f60 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5522700 1 --2- 172.21.15.119:0/2137492987 >> v2:172.21.15.145:3300/0 conn(0x7f2cb006a5c0 0x7f2cb00cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5d23700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c004db0 0x7f2c9c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb4d21700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 0x7f2c9c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5522700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c9c003a20 0x7f2c9c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5d23700 1 -- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 msgr2=0x7f2c9c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5d23700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2cb0154f60 0x7f2c9c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5d23700 1 -- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c9c003a20 msgr2=0x7f2c9c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5d23700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2c9c003a20 0x7f2c9c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.002+0000 7f2cb5d23700 1 -- 172.21.15.119:0/2137492987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c0035d0 con 0x7f2c9c004db0 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb5d23700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c004db0 0x7f2c9c005230 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009f90 tx=0x7f2c9c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42e5d086702423f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca800ec30 con 0x7f2c9c004db0 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca800f040 con 0x7f2c9c004db0 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2ca7fff700 1 -- 172.21.15.119:0/2137492987 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8014480 con 0x7f2c9c004db0 2024-09-13T16:25:14.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 -- 172.21.15.119:0/2137492987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c004db0 msgr2=0x7f2c9c005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 --2- 172.21.15.119:0/2137492987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2c9c004db0 0x7f2c9c005230 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2ca8009f90 tx=0x7f2c9c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb5d23700 1 -- 172.21.15.119:0/2137492987 reap_dead start 2024-09-13T16:25:14.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 -- 172.21.15.119:0/2137492987 shutdown_connections 2024-09-13T16:25:14.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 -- 172.21.15.119:0/2137492987 >> 172.21.15.119:0/2137492987 conn(0x7f2cb006b460 msgr2=0x7f2cb006b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:14.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 -- 172.21.15.119:0/2137492987 shutdown_connections 2024-09-13T16:25:14.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 -- 172.21.15.119:0/2137492987 wait complete. 2024-09-13T16:25:14.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 Processor -- start 2024-09-13T16:25:14.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.006+0000 7f2cb6524700 1 -- start start 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c004db0 0x7f2cb01507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cb0155440 0x7f2cb0157e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cb01558c0 0x7f2cb0158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00b44a0 con 0x7f2cb0155440 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb00b35d0 con 0x7f2cb01558c0 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb0150d10 con 0x7f2c9c004db0 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cb0155440 0x7f2cb0157e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cb0155440 0x7f2cb0157e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35934/0 (socket says 172.21.15.119:35934) 2024-09-13T16:25:14.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 -- 172.21.15.119:0/4004934617 learned_addr learned my addr 172.21.15.119:0/4004934617 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb5522700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c004db0 0x7f2cb01507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb5d23700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cb01558c0 0x7f2cb0158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 -- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c004db0 msgr2=0x7f2cb01507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c004db0 0x7f2cb01507d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 -- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cb01558c0 msgr2=0x7f2cb0158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cb01558c0 0x7f2cb0158380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 -- 172.21.15.119:0/4004934617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb00b3820 con 0x7f2cb0155440 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb4d21700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cb0155440 0x7f2cb0157e40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0007310 tx=0x7f2ca0007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2ca5ffb700 1 -- 172.21.15.119:0/4004934617 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000f6c0 con 0x7f2cb0155440 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2ca5ffb700 1 -- 172.21.15.119:0/4004934617 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca000f820 con 0x7f2cb0155440 2024-09-13T16:25:14.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb00b3b00 con 0x7f2cb0155440 2024-09-13T16:25:14.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb00b40c0 con 0x7f2cb0155440 2024-09-13T16:25:14.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2ca5ffb700 1 -- 172.21.15.119:0/4004934617 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca000fbd0 con 0x7f2cb0155440 2024-09-13T16:25:14.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.010+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c90005310 con 0x7f2cb0155440 2024-09-13T16:25:14.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.014+0000 7f2ca5ffb700 1 -- 172.21.15.119:0/4004934617 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2ca000fd30 con 0x7f2cb0155440 2024-09-13T16:25:14.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.014+0000 7f2ca5ffb700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2c94066730 0x7f2c94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.014+0000 7f2ca5ffb700 1 -- 172.21.15.119:0/4004934617 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ca0083de0 con 0x7f2cb0155440 2024-09-13T16:25:14.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.014+0000 7f2cb5522700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2c94066730 0x7f2c94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.014+0000 7f2cb5522700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2c94066730 0x7f2c94068bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009d90 tx=0x7f2cac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.022+0000 7f2ca5ffb700 1 -- 172.21.15.119:0/4004934617 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2ca0051420 con 0x7f2cb0155440 2024-09-13T16:25:14.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.182+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2c90005b40 con 0x7f2cb0155440 2024-09-13T16:25:14.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2ca5ffb700 1 -- 172.21.15.119:0/4004934617 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v17) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ca0054a40 con 0x7f2cb0155440 2024-09-13T16:25:14.188 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:14.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2c94066730 msgr2=0x7f2c94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2c94066730 0x7f2c94068bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cac009d90 tx=0x7f2cac008040 comp rx=0 tx=0).stop 2024-09-13T16:25:14.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cb0155440 msgr2=0x7f2cb0157e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cb0155440 0x7f2cb0157e40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2ca0007310 tx=0x7f2ca0007620 comp rx=0 tx=0).stop 2024-09-13T16:25:14.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 shutdown_connections 2024-09-13T16:25:14.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2c94066730 0x7f2c94068bf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2cb01558c0 0x7f2cb0158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2cb0155440 0x7f2cb0157e40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 --2- 172.21.15.119:0/4004934617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2c9c004db0 0x7f2cb01507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 >> 172.21.15.119:0/4004934617 conn(0x7f2cb006b460 msgr2=0x7f2cb006cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 shutdown_connections 2024-09-13T16:25:14.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.186+0000 7f2cb6524700 1 -- 172.21.15.119:0/4004934617 wait complete. 2024-09-13T16:25:14.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:25:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 Processor -- start 2024-09-13T16:25:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 -- start start 2024-09-13T16:25:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f60cc146420 0x7f60cc0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f60cc0ce9c0 0x7f60cc06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60cc0cee20 con 0x7f60cc146420 2024-09-13T16:25:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60cc0ad8e0 con 0x7f60cc0ce9c0 2024-09-13T16:25:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc0adea0 con 0x7f60cc06f2a0 2024-09-13T16:25:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f60cc146420 0x7f60cc0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f60cc146420 0x7f60cc0ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35950/0 (socket says 172.21.15.119:35950) 2024-09-13T16:25:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 -- 172.21.15.119:0/2811680976 learned_addr learned my addr 172.21.15.119:0/2811680976 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60c9d9b700 1 --2- 172.21.15.119:0/2811680976 >> v2:172.21.15.145:3300/0 conn(0x7f60cc0ce9c0 0x7f60cc06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60cad9d700 1 --2- 172.21.15.119:0/2811680976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 -- 172.21.15.119:0/2811680976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 msgr2=0x7f60cc0d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 --2- 172.21.15.119:0/2811680976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0d06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 -- 172.21.15.119:0/2811680976 >> v2:172.21.15.145:3300/0 conn(0x7f60cc0ce9c0 msgr2=0x7f60cc06ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 --2- 172.21.15.119:0/2811680976 >> v2:172.21.15.145:3300/0 conn(0x7f60cc0ce9c0 0x7f60cc06ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60ca59c700 1 -- 172.21.15.119:0/2811680976 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60cc146800 con 0x7f60cc146420 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.306+0000 7f60c9d9b700 1 --2- 172.21.15.119:0/2811680976 >> v2:172.21.15.145:3300/0 conn(0x7f60cc0ce9c0 0x7f60cc06ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60ca59c700 1 --2- 172.21.15.119:0/2811680976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc0ce480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0b0540 tx=0x7f60c0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1a41ed8f5602d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60c959a700 1 -- 172.21.15.119:0/2811680976 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0011830 con 0x7f60cc146420 2024-09-13T16:25:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60c959a700 1 -- 172.21.15.119:0/2811680976 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c0011990 con 0x7f60cc146420 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60c959a700 1 -- 172.21.15.119:0/2811680976 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c0011d20 con 0x7f60cc146420 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 -- 172.21.15.119:0/2811680976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 msgr2=0x7f60cc0ce480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2811680976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc0ce480 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f60cc0b0540 tx=0x7f60c0009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 -- 172.21.15.119:0/2811680976 shutdown_connections 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2811680976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0d06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2811680976 >> v2:172.21.15.145:3300/0 conn(0x7f60cc0ce9c0 0x7f60cc06ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2811680976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc0ce480 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 -- 172.21.15.119:0/2811680976 >> 172.21.15.119:0/2811680976 conn(0x7f60cc099a50 msgr2=0x7f60cc099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 -- 172.21.15.119:0/2811680976 shutdown_connections 2024-09-13T16:25:14.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 -- 172.21.15.119:0/2811680976 wait complete. 2024-09-13T16:25:14.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 Processor -- start 2024-09-13T16:25:14.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.310+0000 7f60cb59e700 1 -- start start 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60cc0ce9c0 0x7f60cc13e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc0b2860 con 0x7f60cc146420 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc0b1c50 con 0x7f60cc0ce9c0 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60cc0b1f30 con 0x7f60cc06f2a0 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc159b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35952/0 (socket says 172.21.15.119:35952) 2024-09-13T16:25:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 -- 172.21.15.119:0/2628675174 learned_addr learned my addr 172.21.15.119:0/2628675174 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60c9d9b700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60cc0ce9c0 0x7f60cc13e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60ca59c700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 -- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 msgr2=0x7f60cc0b15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0b15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 -- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60cc0ce9c0 msgr2=0x7f60cc13e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60cc0ce9c0 0x7f60cc13e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 -- 172.21.15.119:0/2628675174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc0094b0 con 0x7f60cc146420 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60ca59c700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cad9d700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc159b70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60c400ec10 tx=0x7f60c400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60baffd700 1 -- 172.21.15.119:0/2628675174 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c400cab0 con 0x7f60cc146420 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60baffd700 1 -- 172.21.15.119:0/2628675174 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60c400f040 con 0x7f60cc146420 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c0009570 con 0x7f60cc146420 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60cc0b24f0 con 0x7f60cc146420 2024-09-13T16:25:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.314+0000 7f60baffd700 1 -- 172.21.15.119:0/2628675174 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c400ce10 con 0x7f60cc146420 2024-09-13T16:25:14.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.318+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60cc0b29d0 con 0x7f60cc146420 2024-09-13T16:25:14.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.318+0000 7f60baffd700 1 -- 172.21.15.119:0/2628675174 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f60c4014480 con 0x7f60cc146420 2024-09-13T16:25:14.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.318+0000 7f60baffd700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ac066760 0x7f60ac068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.318+0000 7f60baffd700 1 -- 172.21.15.119:0/2628675174 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f60c40844c0 con 0x7f60cc146420 2024-09-13T16:25:14.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.318+0000 7f60ca59c700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ac066760 0x7f60ac068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.318+0000 7f60ca59c700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ac066760 0x7f60ac068c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.326+0000 7f60baffd700 1 -- 172.21.15.119:0/2628675174 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f60c4051b00 con 0x7f60cc146420 2024-09-13T16:25:14.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.498+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f60cc06ed60 con 0x7f60cc146420 2024-09-13T16:25:14.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.506+0000 7f60baffd700 1 -- 172.21.15.119:0/2628675174 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+65 (secure 0 0 0) 0x7f60c4018090 con 0x7f60cc146420 2024-09-13T16:25:14.511 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:14.511 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:25:14.511 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:25:14.511 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:25:14.511 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:25:14.511 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:25:14.512 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:25:14.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ac066760 msgr2=0x7f60ac068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ac066760 0x7f60ac068c20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f60c0000c00 tx=0x7f60c0008030 comp rx=0 tx=0).stop 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 msgr2=0x7f60cc159b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc159b70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f60c400ec10 tx=0x7f60c400ef20 comp rx=0 tx=0).stop 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 shutdown_connections 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ac066760 0x7f60ac068c20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60cc146420 0x7f60cc159b70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f60cc0ce9c0 0x7f60cc13e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 --2- 172.21.15.119:0/2628675174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60cc06f2a0 0x7f60cc0b15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 >> 172.21.15.119:0/2628675174 conn(0x7f60cc099a50 msgr2=0x7f60cc066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 shutdown_connections 2024-09-13T16:25:14.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.510+0000 7f60cb59e700 1 -- 172.21.15.119:0/2628675174 wait complete. 2024-09-13T16:25:14.527 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:25:14.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:25:14.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f639d700 1 Processor -- start 2024-09-13T16:25:14.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f639d700 1 -- start start 2024-09-13T16:25:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f639d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f77f006d160 0x7f77f01607f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f639d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f006f140 0x7f77f00b4e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f639d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77f00b49a0 con 0x7f77f0160d30 2024-09-13T16:25:14.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f639d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77f015d140 con 0x7f77f006d160 2024-09-13T16:25:14.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f639d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f00b4420 con 0x7f77f006f140 2024-09-13T16:25:14.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77effff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f77f006d160 0x7f77f01607f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77ef7fe700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f77f0160d30 0x7f77f006ea30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40426/0 (socket says 172.21.15.119:40426) 2024-09-13T16:25:14.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77ef7fe700 1 -- 172.21.15.119:0/1473526340 learned_addr learned my addr 172.21.15.119:0/1473526340 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:14.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77f493a700 1 --2- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f006f140 0x7f77f00b4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77effff700 1 -- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f006f140 msgr2=0x7f77f00b4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77effff700 1 --2- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f006f140 0x7f77f00b4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77effff700 1 -- 172.21.15.119:0/1473526340 >> v1:172.21.15.145:6789/0 conn(0x7f77f0160d30 legacy=0x7f77f006ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77effff700 1 -- 172.21.15.119:0/1473526340 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77f0161770 con 0x7f77f006d160 2024-09-13T16:25:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77effff700 1 --2- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006d160 0x7f77f01607f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009850 tx=0x7f77e0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60c3e6de37c601ac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77eeffd700 1 -- 172.21.15.119:0/1473526340 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0011830 con 0x7f77f006d160 2024-09-13T16:25:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77eeffd700 1 -- 172.21.15.119:0/1473526340 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e0011990 con 0x7f77f006d160 2024-09-13T16:25:14.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.642+0000 7f77eeffd700 1 -- 172.21.15.119:0/1473526340 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0011d40 con 0x7f77f006d160 2024-09-13T16:25:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 -- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006d160 msgr2=0x7f77f01607f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 --2- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006d160 0x7f77f01607f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009850 tx=0x7f77e0009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 -- 172.21.15.119:0/1473526340 shutdown_connections 2024-09-13T16:25:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 --2- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f006f140 0x7f77f00b4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 --2- 172.21.15.119:0/1473526340 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006d160 0x7f77f01607f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 -- 172.21.15.119:0/1473526340 >> 172.21.15.119:0/1473526340 conn(0x7f77f0099a50 msgr2=0x7f77f0099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 -- 172.21.15.119:0/1473526340 shutdown_connections 2024-09-13T16:25:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 -- 172.21.15.119:0/1473526340 wait complete. 2024-09-13T16:25:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 Processor -- start 2024-09-13T16:25:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.646+0000 7f77f639d700 1 -- start start 2024-09-13T16:25:14.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f006d160 0x7f77f0152390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006f140 0x7f77f009e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f0160d30 0x7f77f009e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f00b2f00 con 0x7f77f006d160 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77f00b2030 con 0x7f77f006f140 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77f01528d0 con 0x7f77f0160d30 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f006d160 0x7f77f0152390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f006d160 0x7f77f0152390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35960/0 (socket says 172.21.15.119:35960) 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 -- 172.21.15.119:0/687395710 learned_addr learned my addr 172.21.15.119:0/687395710 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77ef7fe700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006f140 0x7f77f009e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f493a700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f0160d30 0x7f77f009e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 -- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f0160d30 msgr2=0x7f77f009e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f0160d30 0x7f77f009e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 -- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006f140 msgr2=0x7f77f009e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:14.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006f140 0x7f77f009e2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:14.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 -- 172.21.15.119:0/687395710 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e00094b0 con 0x7f77f006d160 2024-09-13T16:25:14.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77effff700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f006d160 0x7f77f0152390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f77e00090b0 tx=0x7f77e0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77ecff9700 1 -- 172.21.15.119:0/687395710 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e00036c0 con 0x7f77f006d160 2024-09-13T16:25:14.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77ecff9700 1 -- 172.21.15.119:0/687395710 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77e001f070 con 0x7f77f006d160 2024-09-13T16:25:14.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77f00b2280 con 0x7f77f006d160 2024-09-13T16:25:14.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77ecff9700 1 -- 172.21.15.119:0/687395710 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77e0003a60 con 0x7f77f006d160 2024-09-13T16:25:14.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.650+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77f00b2840 con 0x7f77f006d160 2024-09-13T16:25:14.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.654+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77f006b460 con 0x7f77f006d160 2024-09-13T16:25:14.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.654+0000 7f77ecff9700 1 -- 172.21.15.119:0/687395710 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f77e0003bc0 con 0x7f77f006d160 2024-09-13T16:25:14.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.654+0000 7f77ecff9700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f77dc066730 0x7f77dc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:14.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.654+0000 7f77ecff9700 1 -- 172.21.15.119:0/687395710 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f77e0021070 con 0x7f77f006d160 2024-09-13T16:25:14.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.654+0000 7f77ef7fe700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f77dc066730 0x7f77dc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:14.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.654+0000 7f77ef7fe700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f77dc066730 0x7f77dc068bf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f77f01f63c0 tx=0x7f77e4007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:14.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.662+0000 7f77ecff9700 1 -- 172.21.15.119:0/687395710 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f77e0052d00 con 0x7f77f006d160 2024-09-13T16:25:14.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:14.834+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f77f006b040 con 0x7f77f006d160 2024-09-13T16:25:15.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.530+0000 7f77ecff9700 1 -- 172.21.15.119:0/687395710 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) v1 ==== 138+0+0 (secure 0 0 0) 0x7f77e00150b0 con 0x7f77f006d160 2024-09-13T16:25:15.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f77dc066730 msgr2=0x7f77dc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:15.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f77dc066730 0x7f77dc068bf0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f77f01f63c0 tx=0x7f77e4007f30 comp rx=0 tx=0).stop 2024-09-13T16:25:15.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f006d160 msgr2=0x7f77f0152390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:15.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f006d160 0x7f77f0152390 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f77e00090b0 tx=0x7f77e0004770 comp rx=0 tx=0).stop 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 shutdown_connections 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f77dc066730 0x7f77dc068bf0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77f0160d30 0x7f77f009e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f77f006f140 0x7f77f009e2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 --2- 172.21.15.119:0/687395710 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f77f006d160 0x7f77f0152390 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 >> 172.21.15.119:0/687395710 conn(0x7f77f0099a50 msgr2=0x7f77f00adf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 shutdown_connections 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.542+0000 7f77f639d700 1 -- 172.21.15.119:0/687395710 wait complete. 2024-09-13T16:25:15.545 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:25:15.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-09-13T16:25:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.674+0000 7f40ccee4700 1 Processor -- start 2024-09-13T16:25:15.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.674+0000 7f40ccee4700 1 -- start start 2024-09-13T16:25:15.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40ccee4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f40c815abb0 0x7f40c80cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:15.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40ccee4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f40c80b31e0 0x7f40c80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:15.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40ccee4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c815af90 con 0x7f40c80cd750 2024-09-13T16:25:15.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40ccee4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80cdbc0 con 0x7f40c80b31e0 2024-09-13T16:25:15.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40ccee4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c80cb6d0 con 0x7f40c815abb0 2024-09-13T16:25:15.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f40c80b31e0 0x7f40c80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:15.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f40c80b31e0 0x7f40c80cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35962/0 (socket says 172.21.15.119:35962) 2024-09-13T16:25:15.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 -- 172.21.15.119:0/3427961035 learned_addr learned my addr 172.21.15.119:0/3427961035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:15.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40c77fe700 1 --2- 172.21.15.119:0/3427961035 >> v2:172.21.15.145:3300/0 conn(0x7f40c815abb0 0x7f40c80cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 -- 172.21.15.119:0/3427961035 >> v2:172.21.15.145:3300/0 conn(0x7f40c815abb0 msgr2=0x7f40c80cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 --2- 172.21.15.119:0/3427961035 >> v2:172.21.15.145:3300/0 conn(0x7f40c815abb0 0x7f40c80cf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 -- 172.21.15.119:0/3427961035 >> v1:172.21.15.145:6789/0 conn(0x7f40c80cd750 legacy=0x7f40c80c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 -- 172.21.15.119:0/3427961035 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c80d0060 con 0x7f40c80b31e0 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40beffd700 1 --2- 172.21.15.119:0/3427961035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c80cd210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f40c8071120 tx=0x7f40b0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3baa4487e04a27d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40c6ffd700 1 -- 172.21.15.119:0/3427961035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0011830 con 0x7f40c80b31e0 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40c6ffd700 1 -- 172.21.15.119:0/3427961035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b0011990 con 0x7f40c80b31e0 2024-09-13T16:25:15.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.678+0000 7f40c6ffd700 1 -- 172.21.15.119:0/3427961035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0011d20 con 0x7f40c80b31e0 2024-09-13T16:25:15.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 -- 172.21.15.119:0/3427961035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 msgr2=0x7f40c80cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:15.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 --2- 172.21.15.119:0/3427961035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c80cd210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f40c8071120 tx=0x7f40b0009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 -- 172.21.15.119:0/3427961035 shutdown_connections 2024-09-13T16:25:15.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 --2- 172.21.15.119:0/3427961035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c80cd210 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 --2- 172.21.15.119:0/3427961035 >> v2:172.21.15.145:3300/0 conn(0x7f40c815abb0 0x7f40c80cf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 -- 172.21.15.119:0/3427961035 >> 172.21.15.119:0/3427961035 conn(0x7f40c80b25e0 msgr2=0x7f40c80b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:15.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 -- 172.21.15.119:0/3427961035 shutdown_connections 2024-09-13T16:25:15.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.682+0000 7f40ccee4700 1 -- 172.21.15.119:0/3427961035 wait complete. 2024-09-13T16:25:15.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 Processor -- start 2024-09-13T16:25:15.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 -- start start 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c8151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c80cd750 0x7f40c806c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c815abb0 0x7f40c80b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c806d4b0 con 0x7f40c80b31e0 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c806cef0 con 0x7f40c815abb0 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c806d1d0 con 0x7f40c80cd750 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c8151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c8151280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35974/0 (socket says 172.21.15.119:35974) 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 -- 172.21.15.119:0/501457671 learned_addr learned my addr 172.21.15.119:0/501457671 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:15.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40beffd700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c80cd750 0x7f40c806c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c7fff700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c815abb0 0x7f40c80b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 -- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c80cd750 msgr2=0x7f40c806c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c80cd750 0x7f40c806c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 -- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c815abb0 msgr2=0x7f40c80b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c815abb0 0x7f40c80b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 -- 172.21.15.119:0/501457671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b0009570 con 0x7f40c80b31e0 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c7fff700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c815abb0 0x7f40c80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c77fe700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c8151280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e9d0 tx=0x7f40b800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c4ff9700 1 -- 172.21.15.119:0/501457671 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8009720 con 0x7f40c80b31e0 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40c4ff9700 1 -- 172.21.15.119:0/501457671 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b800c850 con 0x7f40c80b31e0 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.686+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c80c8be0 con 0x7f40c80b31e0 2024-09-13T16:25:15.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.690+0000 7f40c4ff9700 1 -- 172.21.15.119:0/501457671 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b800cbc0 con 0x7f40c80b31e0 2024-09-13T16:25:15.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.690+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c815b130 con 0x7f40c80b31e0 2024-09-13T16:25:15.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.690+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8066970 con 0x7f40c80b31e0 2024-09-13T16:25:15.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.690+0000 7f40c4ff9700 1 -- 172.21.15.119:0/501457671 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f40b800cd20 con 0x7f40c80b31e0 2024-09-13T16:25:15.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.690+0000 7f40c4ff9700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a8066730 0x7f40a8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:15.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.690+0000 7f40c4ff9700 1 -- 172.21.15.119:0/501457671 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f40b8084740 con 0x7f40c80b31e0 2024-09-13T16:25:15.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.694+0000 7f40beffd700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a8066730 0x7f40a8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:15.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.694+0000 7f40beffd700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a8066730 0x7f40a8068bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40b0000c00 tx=0x7f40b0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:15.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.694+0000 7f40c4ff9700 1 -- 172.21.15.119:0/501457671 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f40b8051d20 con 0x7f40c80b31e0 2024-09-13T16:25:15.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:15.862+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f40c81517c0 con 0x7f40c80b31e0 2024-09-13T16:25:16.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.550+0000 7f40c4ff9700 1 -- 172.21.15.119:0/501457671 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) v1 ==== 171+0+0 (secure 0 0 0) 0x7f40b80170d0 con 0x7f40c80b31e0 2024-09-13T16:25:16.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a8066730 msgr2=0x7f40a8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a8066730 0x7f40a8068bf0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f40b0000c00 tx=0x7f40b0008030 comp rx=0 tx=0).stop 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 msgr2=0x7f40c8151280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c8151280 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f40b800e9d0 tx=0x7f40b800ece0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 shutdown_connections 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a8066730 0x7f40a8068bf0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c815abb0 0x7f40c80b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c80cd750 0x7f40c806c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 --2- 172.21.15.119:0/501457671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c80b31e0 0x7f40c8151280 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 >> 172.21.15.119:0/501457671 conn(0x7f40c80b25e0 msgr2=0x7f40c80cba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 shutdown_connections 2024-09-13T16:25:16.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.554+0000 7f40ccee4700 1 -- 172.21.15.119:0/501457671 wait complete. 2024-09-13T16:25:16.560 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:25:16.572 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:25:16.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855861700 1 Processor -- start 2024-09-13T16:25:16.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855861700 1 -- start start 2024-09-13T16:25:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855861700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08500d1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855861700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f08500b6610 0x7f08500b6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855861700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08500d0120 con 0x7f085015cc30 2024-09-13T16:25:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855861700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08500d1a80 con 0x7f08500b6610 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855861700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500cfe40 con 0x7f08500d1120 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f085485f700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f085015cc30 0x7f08500cf380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40434/0 (socket says 172.21.15.119:40434) 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f085485f700 1 -- 172.21.15.119:0/738367053 learned_addr learned my addr 172.21.15.119:0/738367053 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f0855060700 1 --2- 172.21.15.119:0/738367053 >> v2:172.21.15.145:3300/0 conn(0x7f08500b6610 0x7f08500b6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.682+0000 7f084ffff700 1 --2- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08500d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855060700 1 -- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 msgr2=0x7f08500d1540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855060700 1 --2- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08500d1540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855060700 1 -- 172.21.15.119:0/738367053 >> v1:172.21.15.145:6789/0 conn(0x7f085015cc30 legacy=0x7f08500cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855060700 1 -- 172.21.15.119:0/738367053 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08500b5b30 con 0x7f08500b6610 2024-09-13T16:25:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855060700 1 --2- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08500b6610 0x7f08500b6a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0848009d90 tx=0x7f084800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e15268464d54d16 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f084f7fe700 1 -- 172.21.15.119:0/738367053 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f084800e070 con 0x7f08500b6610 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f084f7fe700 1 -- 172.21.15.119:0/738367053 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f084800bb20 con 0x7f08500b6610 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f084f7fe700 1 -- 172.21.15.119:0/738367053 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0848012430 con 0x7f08500b6610 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 -- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08500b6610 msgr2=0x7f08500b6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 --2- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08500b6610 0x7f08500b6a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0848009d90 tx=0x7f084800b330 comp rx=0 tx=0).stop 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 -- 172.21.15.119:0/738367053 shutdown_connections 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 --2- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08500b6610 0x7f08500b6a30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 --2- 172.21.15.119:0/738367053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08500d1540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 -- 172.21.15.119:0/738367053 >> 172.21.15.119:0/738367053 conn(0x7f085008c0f0 msgr2=0x7f08500d2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 -- 172.21.15.119:0/738367053 shutdown_connections 2024-09-13T16:25:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 -- 172.21.15.119:0/738367053 wait complete. 2024-09-13T16:25:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 Processor -- start 2024-09-13T16:25:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.686+0000 7f0855861700 1 -- start start 2024-09-13T16:25:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08500b6610 0x7f08500cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08501f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f085015cc30 0x7f08501f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500c76f0 con 0x7f08500b6610 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08500c6ae0 con 0x7f085015cc30 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08500c6dc0 con 0x7f08500d1120 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08500b6610 0x7f08500cbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08500b6610 0x7f08500cbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35990/0 (socket says 172.21.15.119:35990) 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 -- 172.21.15.119:0/862832448 learned_addr learned my addr 172.21.15.119:0/862832448 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855060700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f085015cc30 0x7f08501f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f084ffff700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08501f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 -- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 msgr2=0x7f08501f5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08501f5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 -- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f085015cc30 msgr2=0x7f08501f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f085015cc30 0x7f08501f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 -- 172.21.15.119:0/862832448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848009a70 con 0x7f08500b6610 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f085485f700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08500b6610 0x7f08500cbfc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0840001eb0 tx=0x7f084000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f084d7fa700 1 -- 172.21.15.119:0/862832448 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840005d90 con 0x7f08500b6610 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f084d7fa700 1 -- 172.21.15.119:0/862832448 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08400070d0 con 0x7f08500b6610 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08500c70a0 con 0x7f08500b6610 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f084d7fa700 1 -- 172.21.15.119:0/862832448 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0840007440 con 0x7f08500b6610 2024-09-13T16:25:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.690+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08500c6670 con 0x7f08500b6610 2024-09-13T16:25:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.694+0000 7f084d7fa700 1 -- 172.21.15.119:0/862832448 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f08400075a0 con 0x7f08500b6610 2024-09-13T16:25:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.694+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08500b60b0 con 0x7f08500b6610 2024-09-13T16:25:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.694+0000 7f084d7fa700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0830066730 0x7f0830068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.694+0000 7f084d7fa700 1 -- 172.21.15.119:0/862832448 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f0840083430 con 0x7f08500b6610 2024-09-13T16:25:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.694+0000 7f084ffff700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0830066730 0x7f0830068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:16.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.694+0000 7f084ffff700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0830066730 0x7f0830068bf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08500cea30 tx=0x7f0838009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:16.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.698+0000 7f084d7fa700 1 -- 172.21.15.119:0/862832448 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0840050830 con 0x7f08500b6610 2024-09-13T16:25:16.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.902+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f08500d1540 con 0x7f08500b6610 2024-09-13T16:25:16.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.910+0000 7f084d7fa700 1 -- 172.21.15.119:0/862832448 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f0840053e50 con 0x7f08500b6610 2024-09-13T16:25:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.918+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0830066730 msgr2=0x7f0830068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.918+0000 7f0855861700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0830066730 0x7f0830068bf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f08500cea30 tx=0x7f0838009240 comp rx=0 tx=0).stop 2024-09-13T16:25:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.918+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08500b6610 msgr2=0x7f08500cbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.918+0000 7f0855861700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08500b6610 0x7f08500cbfc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0840001eb0 tx=0x7f084000c7b0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.918+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 shutdown_connections 2024-09-13T16:25:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.922+0000 7f0855861700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0830066730 0x7f0830068bf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.922+0000 7f0855861700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f085015cc30 0x7f08501f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.922+0000 7f0855861700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08500d1120 0x7f08501f5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.922+0000 7f0855861700 1 --2- 172.21.15.119:0/862832448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08500b6610 0x7f08500cbfc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:16.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.922+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 >> 172.21.15.119:0/862832448 conn(0x7f085008c0f0 msgr2=0x7f08500ce020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:16.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.922+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 shutdown_connections 2024-09-13T16:25:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:16.922+0000 7f0855861700 1 -- 172.21.15.119:0/862832448 wait complete. 2024-09-13T16:25:16.930 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 2 and data pool 3 2024-09-13T16:25:16.946 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-09-13T16:25:16.946 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-09-13T16:25:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9549aa2700 1 Processor -- start 2024-09-13T16:25:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9549aa2700 1 -- start start 2024-09-13T16:25:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9549aa2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f9544167920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9549aa2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95440b44d0 con 0x7f95440b35e0 2024-09-13T16:25:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9549aa2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95440d12d0 con 0x7f954415cc60 2024-09-13T16:25:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9549aa2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95440d15b0 con 0x7f95440ce970 2024-09-13T16:25:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9542ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f95440b35e0 0x7f95440cfb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49528/0 (socket says 172.21.15.119:49528) 2024-09-13T16:25:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9542ffd700 1 -- 172.21.15.119:0/1796271839 learned_addr learned my addr 172.21.15.119:0/1796271839 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.054+0000 7f9543fff700 1 --2- 172.21.15.119:0/1796271839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f9544167920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f95427fc700 1 -- 172.21.15.119:0/1796271839 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2021713551 0 0) 0x7f95440b44d0 con 0x7f95440b35e0 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f95427fc700 1 -- 172.21.15.119:0/1796271839 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95300035d0 con 0x7f95440b35e0 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9543fff700 1 -- 172.21.15.119:0/1796271839 >> v1:172.21.15.145:6789/0 conn(0x7f954415cc60 legacy=0x7f95440cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9543fff700 1 -- 172.21.15.119:0/1796271839 >> v1:172.21.15.119:6789/0 conn(0x7f95440b35e0 legacy=0x7f95440cfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f95427fc700 1 -- 172.21.15.119:0/1796271839 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633563381 0 0) 0x7f95440d12d0 con 0x7f954415cc60 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9543fff700 1 -- 172.21.15.119:0/1796271839 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95440d1890 con 0x7f95440ce970 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9543fff700 1 --2- 172.21.15.119:0/1796271839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f9544167920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f953c009880 tx=0x7f953c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2604f7ed884edd73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:17.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f95427fc700 1 -- 172.21.15.119:0/1796271839 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c011830 con 0x7f95440ce970 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f95427fc700 1 -- 172.21.15.119:0/1796271839 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c011990 con 0x7f95440ce970 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f95427fc700 1 -- 172.21.15.119:0/1796271839 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c011d00 con 0x7f95440ce970 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9549aa2700 1 -- 172.21.15.119:0/1796271839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 msgr2=0x7f9544167920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9549aa2700 1 --2- 172.21.15.119:0/1796271839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f9544167920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f953c009880 tx=0x7f953c009b90 comp rx=0 tx=0).stop 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9549aa2700 1 -- 172.21.15.119:0/1796271839 shutdown_connections 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9549aa2700 1 --2- 172.21.15.119:0/1796271839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f9544167920 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9549aa2700 1 -- 172.21.15.119:0/1796271839 >> 172.21.15.119:0/1796271839 conn(0x7f95440d2b70 msgr2=0x7f95440d2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9549aa2700 1 -- 172.21.15.119:0/1796271839 shutdown_connections 2024-09-13T16:25:17.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.058+0000 7f9549aa2700 1 -- 172.21.15.119:0/1796271839 wait complete. 2024-09-13T16:25:17.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 Processor -- start 2024-09-13T16:25:17.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 -- start start 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95440b35e0 0x7f95441f3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f95441efba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f954415cc60 0x7f95440c7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95441f00e0 con 0x7f954415cc60 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95440c84f0 con 0x7f95440b35e0 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95440c87d0 con 0x7f95440ce970 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f954415cc60 0x7f95440c7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:17.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f954415cc60 0x7f95440c7bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36002/0 (socket says 172.21.15.119:36002) 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 -- 172.21.15.119:0/217769088 learned_addr learned my addr 172.21.15.119:0/217769088 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9542ffd700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f95441efba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f95437fe700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95440b35e0 0x7f95441f3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 -- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 msgr2=0x7f95441efba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f95441efba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 -- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95440b35e0 msgr2=0x7f95441f3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95440b35e0 0x7f95441f3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 -- 172.21.15.119:0/217769088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f953c0094b0 con 0x7f954415cc60 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9543fff700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f954415cc60 0x7f95440c7bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f953c013040 tx=0x7f953c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f953bfff700 1 -- 172.21.15.119:0/217769088 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c0044d0 con 0x7f954415cc60 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f953bfff700 1 -- 172.21.15.119:0/217769088 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f953c0036c0 con 0x7f954415cc60 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95441f2960 con 0x7f954415cc60 2024-09-13T16:25:17.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.062+0000 7f953bfff700 1 -- 172.21.15.119:0/217769088 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f953c003a70 con 0x7f954415cc60 2024-09-13T16:25:17.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.066+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95441f2610 con 0x7f954415cc60 2024-09-13T16:25:17.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.066+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95441f2e80 con 0x7f954415cc60 2024-09-13T16:25:17.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.066+0000 7f953bfff700 1 -- 172.21.15.119:0/217769088 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f953c003bd0 con 0x7f954415cc60 2024-09-13T16:25:17.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.066+0000 7f953bfff700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9524066730 0x7f9524068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:17.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.066+0000 7f953bfff700 1 -- 172.21.15.119:0/217769088 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f953c0857f0 con 0x7f954415cc60 2024-09-13T16:25:17.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.066+0000 7f95437fe700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9524066730 0x7f9524068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:17.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.070+0000 7f95437fe700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9524066730 0x7f9524068bf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95340012e0 tx=0x7f9534009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:17.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.078+0000 7f953bfff700 1 -- 172.21.15.119:0/217769088 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f95441f2e80 con 0x7f954415cc60 2024-09-13T16:25:17.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.258+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7f95441f2e80 con 0x7f954415cc60 2024-09-13T16:25:17.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.922+0000 7f953bfff700 1 -- 172.21.15.119:0/217769088 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v21) v1 ==== 157+0+0 (secure 0 0 0) 0x7f95441f2e80 con 0x7f954415cc60 2024-09-13T16:25:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9524066730 msgr2=0x7f9524068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9524066730 0x7f9524068bf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f95340012e0 tx=0x7f9534009040 comp rx=0 tx=0).stop 2024-09-13T16:25:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f954415cc60 msgr2=0x7f95440c7bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f954415cc60 0x7f95440c7bf0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f953c013040 tx=0x7f953c004770 comp rx=0 tx=0).stop 2024-09-13T16:25:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 shutdown_connections 2024-09-13T16:25:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9524066730 0x7f9524068bf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:17.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f954415cc60 0x7f95440c7bf0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:17.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95440ce970 0x7f95441efba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:17.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 --2- 172.21.15.119:0/217769088 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95440b35e0 0x7f95441f3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:17.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 >> 172.21.15.119:0/217769088 conn(0x7f95440d2b70 msgr2=0x7f95440d0720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:17.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 shutdown_connections 2024-09-13T16:25:17.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:17.926+0000 7f9549aa2700 1 -- 172.21.15.119:0/217769088 wait complete. 2024-09-13T16:25:17.946 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:25:18.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.078+0000 7f7e51ce2700 1 Processor -- start 2024-09-13T16:25:18.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.078+0000 7f7e51ce2700 1 -- start start 2024-09-13T16:25:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7e4c0cda80 0x7f7e4c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7e4c0cf280 0x7f7e4c0cde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e4c06cd30 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c0b3ad0 con 0x7f7e4c0cf280 2024-09-13T16:25:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e4c0b3db0 con 0x7f7e4c0cda80 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7e4c0cf280 0x7f7e4c0cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7e4c0cf280 0x7f7e4c0cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36012/0 (socket says 172.21.15.119:36012) 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 -- 172.21.15.119:0/169198187 learned_addr learned my addr 172.21.15.119:0/169198187 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e50ce0700 1 --2- 172.21.15.119:0/169198187 >> v2:172.21.15.145:3300/0 conn(0x7f7e4c0cda80 0x7f7e4c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 -- 172.21.15.119:0/169198187 >> v2:172.21.15.145:3300/0 conn(0x7f7e4c0cda80 msgr2=0x7f7e4c0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 --2- 172.21.15.119:0/169198187 >> v2:172.21.15.145:3300/0 conn(0x7f7e4c0cda80 0x7f7e4c0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 -- 172.21.15.119:0/169198187 >> v1:172.21.15.145:6789/0 conn(0x7f7e4c0ce3a0 legacy=0x7f7e4c06d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 -- 172.21.15.119:0/169198187 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4c0b4090 con 0x7f7e4c0cf280 2024-09-13T16:25:18.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e50ce0700 1 --2- 172.21.15.119:0/169198187 >> v2:172.21.15.145:3300/0 conn(0x7f7e4c0cda80 0x7f7e4c0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:18.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4bfff700 1 --2- 172.21.15.119:0/169198187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0cf280 0x7f7e4c0cde60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0b5820 tx=0x7f7e34009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c93cebc45865215 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:18.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4b7fe700 1 -- 172.21.15.119:0/169198187 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e34004230 con 0x7f7e4c0cf280 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4b7fe700 1 -- 172.21.15.119:0/169198187 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e34004390 con 0x7f7e4c0cf280 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e4b7fe700 1 -- 172.21.15.119:0/169198187 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e340036c0 con 0x7f7e4c0cf280 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- 172.21.15.119:0/169198187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0cf280 msgr2=0x7f7e4c0cde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/169198187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0cf280 0x7f7e4c0cde60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c0b5820 tx=0x7f7e34009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- 172.21.15.119:0/169198187 shutdown_connections 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/169198187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0cf280 0x7f7e4c0cde60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/169198187 >> v2:172.21.15.145:3300/0 conn(0x7f7e4c0cda80 0x7f7e4c0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- 172.21.15.119:0/169198187 >> 172.21.15.119:0/169198187 conn(0x7f7e4c06b460 msgr2=0x7f7e4c06b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- 172.21.15.119:0/169198187 shutdown_connections 2024-09-13T16:25:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.082+0000 7f7e51ce2700 1 -- 172.21.15.119:0/169198187 wait complete. 2024-09-13T16:25:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 Processor -- start 2024-09-13T16:25:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 -- start start 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e4c0cda80 0x7f7e4c157920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0ce3a0 0x7f7e4c14e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e4c0cf280 0x7f7e4c06d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c142720 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c06f6a0 con 0x7f7e4c0cda80 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e51ce2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e4c157f00 con 0x7f7e4c0cf280 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0ce3a0 0x7f7e4c14e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0ce3a0 0x7f7e4c14e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36022/0 (socket says 172.21.15.119:36022) 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 -- 172.21.15.119:0/1819659886 learned_addr learned my addr 172.21.15.119:0/1819659886 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e50ce0700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e4c0cda80 0x7f7e4c157920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e514e1700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e4c0cf280 0x7f7e4c06d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 -- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e4c0cf280 msgr2=0x7f7e4c06d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:18.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e4c0cf280 0x7f7e4c06d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:18.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 -- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e4c0cda80 msgr2=0x7f7e4c157920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:18.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e4c0cda80 0x7f7e4c157920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:18.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.086+0000 7f7e4bfff700 1 -- 172.21.15.119:0/1819659886 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e34009570 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e4bfff700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0ce3a0 0x7f7e4c14e6f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e34004000 tx=0x7f7e34004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:18.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e497fa700 1 -- 172.21.15.119:0/1819659886 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3400b660 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e497fa700 1 -- 172.21.15.119:0/1819659886 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e3400b7c0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e4c1581e0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e497fa700 1 -- 172.21.15.119:0/1819659886 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e3400bb70 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e4c1587a0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e497fa700 1 -- 172.21.15.119:0/1819659886 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7e3400bcd0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e497fa700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7e2c066760 0x7f7e2c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:18.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e4c1428d0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e497fa700 1 -- 172.21.15.119:0/1819659886 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5935+0+0 (secure 0 0 0) 0x7f7e34085930 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.090+0000 7f7e50ce0700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7e2c066760 0x7f7e2c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:18.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.094+0000 7f7e50ce0700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7e2c066760 0x7f7e2c068c20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e44003c30 tx=0x7f7e44009910 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.098+0000 7f7e497fa700 1 -- 172.21.15.119:0/1819659886 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7e4c1428d0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:18.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:18.270+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7e4c1428d0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:19.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.938+0000 7f7e497fa700 1 -- 172.21.15.119:0/1819659886 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v23) v1 ==== 228+0+0 (secure 0 0 0) 0x7f7e4c1428d0 con 0x7f7e4c0ce3a0 2024-09-13T16:25:19.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.946+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7e2c066760 msgr2=0x7f7e2c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:19.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.946+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7e2c066760 0x7f7e2c068c20 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7e44003c30 tx=0x7f7e44009910 comp rx=0 tx=0).stop 2024-09-13T16:25:19.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.946+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0ce3a0 msgr2=0x7f7e4c14e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:19.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.946+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0ce3a0 0x7f7e4c14e6f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7e34004000 tx=0x7f7e34004770 comp rx=0 tx=0).stop 2024-09-13T16:25:19.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 shutdown_connections 2024-09-13T16:25:19.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7e2c066760 0x7f7e2c068c20 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:19.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7e4c0cf280 0x7f7e4c06d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:19.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7e4c0ce3a0 0x7f7e4c14e6f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:19.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 --2- 172.21.15.119:0/1819659886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7e4c0cda80 0x7f7e4c157920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:19.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 >> 172.21.15.119:0/1819659886 conn(0x7f7e4c06b460 msgr2=0x7f7e4c0b05f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:19.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 shutdown_connections 2024-09-13T16:25:19.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:19.950+0000 7f7e51ce2700 1 -- 172.21.15.119:0/1819659886 wait complete. 2024-09-13T16:25:19.953 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data_ec' created 2024-09-13T16:25:19.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-09-13T16:25:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fdf59e700 1 Processor -- start 2024-09-13T16:25:20.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fdf59e700 1 -- start start 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fdf59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3fe01595d0 0x7f3fe0070670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fdf59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01599b0 0x7f3fe006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fdf59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe015a1e0 con 0x7f3fe00cd960 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fdf59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe0070ed0 con 0x7f3fe01595d0 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fdf59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe015a4c0 con 0x7f3fe01599b0 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fde59c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3fe01595d0 0x7f3fe0070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fddd9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01599b0 0x7f3fe006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:20.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fde59c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3fe01595d0 0x7f3fe0070670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:37304/0 (socket says 172.21.15.119:37304) 2024-09-13T16:25:20.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fde59c700 1 -- 172.21.15.119:0/3146235226 learned_addr learned my addr 172.21.15.119:0/3146235226 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:20.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fded9d700 1 --1- 172.21.15.119:0/3146235226 >> v1:172.21.15.145:6789/0 conn(0x7f3fe00cd960 0x7f3fe0071320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40456/0 (socket says 172.21.15.119:40456) 2024-09-13T16:25:20.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fddd9b700 1 -- 172.21.15.119:0/3146235226 >> v2:172.21.15.145:3300/0 conn(0x7f3fe01595d0 msgr2=0x7f3fe0070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:20.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fddd9b700 1 --2- 172.21.15.119:0/3146235226 >> v2:172.21.15.145:3300/0 conn(0x7f3fe01595d0 0x7f3fe0070670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fddd9b700 1 -- 172.21.15.119:0/3146235226 >> v1:172.21.15.145:6789/0 conn(0x7f3fe00cd960 legacy=0x7f3fe0071320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.070+0000 7f3fddd9b700 1 -- 172.21.15.119:0/3146235226 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe015a7a0 con 0x7f3fe01599b0 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fddd9b700 1 --2- 172.21.15.119:0/3146235226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01599b0 0x7f3fe006c420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009d90 tx=0x7f3fd000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68beb8425f6cfe69 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdd59a700 1 -- 172.21.15.119:0/3146235226 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd000e070 con 0x7f3fe01599b0 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdd59a700 1 -- 172.21.15.119:0/3146235226 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd000bb20 con 0x7f3fe01599b0 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdd59a700 1 -- 172.21.15.119:0/3146235226 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0012430 con 0x7f3fe01599b0 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 -- 172.21.15.119:0/3146235226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01599b0 msgr2=0x7f3fe006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/3146235226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01599b0 0x7f3fe006c420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0009d90 tx=0x7f3fd000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 -- 172.21.15.119:0/3146235226 shutdown_connections 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/3146235226 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01599b0 0x7f3fe006c420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/3146235226 >> v2:172.21.15.145:3300/0 conn(0x7f3fe01595d0 0x7f3fe0070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 -- 172.21.15.119:0/3146235226 >> 172.21.15.119:0/3146235226 conn(0x7f3fe006a350 msgr2=0x7f3fe006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 -- 172.21.15.119:0/3146235226 shutdown_connections 2024-09-13T16:25:20.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 -- 172.21.15.119:0/3146235226 wait complete. 2024-09-13T16:25:20.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 Processor -- start 2024-09-13T16:25:20.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.074+0000 7f3fdf59e700 1 -- start start 2024-09-13T16:25:20.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe00cd960 0x7f3fe015cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01595d0 0x7f3fe00acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01599b0 0x7f3fe014bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe0158e30 con 0x7f3fe01599b0 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01f6580 con 0x7f3fe00cd960 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00c3950 con 0x7f3fe01595d0 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01595d0 0x7f3fe00acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01595d0 0x7f3fe00acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:47430/0 (socket says 172.21.15.119:47430) 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 -- 172.21.15.119:0/235066247 learned_addr learned my addr 172.21.15.119:0/235066247 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fded9d700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01599b0 0x7f3fe014bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fde59c700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe00cd960 0x7f3fe015cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 -- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe00cd960 msgr2=0x7f3fe015cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe00cd960 0x7f3fe015cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 -- 172.21.15.119:0/235066247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01599b0 msgr2=0x7f3fe014bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:20.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01599b0 0x7f3fe014bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 -- 172.21.15.119:0/235066247 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fd0009a70 con 0x7f3fe01595d0 2024-09-13T16:25:20.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fddd9b700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01595d0 0x7f3fe00acb30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0014040 tx=0x7f3fd0004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:20.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fd6ffd700 1 -- 172.21.15.119:0/235066247 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0012430 con 0x7f3fe01595d0 2024-09-13T16:25:20.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe00c3c30 con 0x7f3fe01595d0 2024-09-13T16:25:20.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fd6ffd700 1 -- 172.21.15.119:0/235066247 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd0020070 con 0x7f3fe01595d0 2024-09-13T16:25:20.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fd6ffd700 1 -- 172.21.15.119:0/235066247 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd0004280 con 0x7f3fe01595d0 2024-09-13T16:25:20.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.078+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe0158a50 con 0x7f3fe01595d0 2024-09-13T16:25:20.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.082+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe00c3e70 con 0x7f3fe01595d0 2024-09-13T16:25:20.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.082+0000 7f3fd6ffd700 1 -- 172.21.15.119:0/235066247 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3fd00043e0 con 0x7f3fe01595d0 2024-09-13T16:25:20.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.082+0000 7f3fd6ffd700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3fc0066730 0x7f3fc0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:20.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.082+0000 7f3fd6ffd700 1 -- 172.21.15.119:0/235066247 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f3fd0022070 con 0x7f3fe01595d0 2024-09-13T16:25:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.082+0000 7f3fde59c700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3fc0066730 0x7f3fc0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:20.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.082+0000 7f3fde59c700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3fc0066730 0x7f3fc0068bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8006d70 tx=0x7f3fc800b4f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:20.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.094+0000 7f3fd6ffd700 1 -- 172.21.15.119:0/235066247 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3fd0053120 con 0x7f3fe01595d0 2024-09-13T16:25:20.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.262+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f3fe0070670 con 0x7f3fe01595d0 2024-09-13T16:25:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.942+0000 7f3fd6ffd700 1 -- 172.21.15.119:0/235066247 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v24) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3fd0056740 con 0x7f3fe01595d0 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3fc0066730 msgr2=0x7f3fc0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3fc0066730 0x7f3fc0068bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8006d70 tx=0x7f3fc800b4f0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01595d0 msgr2=0x7f3fe00acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01595d0 0x7f3fe00acb30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3fd0014040 tx=0x7f3fd0004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 shutdown_connections 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3fc0066730 0x7f3fc0068bf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3fe01599b0 0x7f3fe014bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3fe01595d0 0x7f3fe00acb30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 --2- 172.21.15.119:0/235066247 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3fe00cd960 0x7f3fe015cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 >> 172.21.15.119:0/235066247 conn(0x7f3fe006a350 msgr2=0x7f3fe006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:20.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 shutdown_connections 2024-09-13T16:25:20.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:20.962+0000 7f3fdf59e700 1 -- 172.21.15.119:0/235066247 wait complete. 2024-09-13T16:25:20.966 INFO:teuthology.orchestra.run.smithi119.stderr:set pool 4 allow_ec_overwrites to true 2024-09-13T16:25:20.979 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-09-13T16:25:21.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c2f03700 1 Processor -- start 2024-09-13T16:25:21.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c2f03700 1 -- start start 2024-09-13T16:25:21.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c2f03700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78bc0b3970 0x7f78bc0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c2f03700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78bc0cd4d0 con 0x7f78bc0add60 2024-09-13T16:25:21.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c2f03700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78bc15e130 con 0x7f78bc15dbe0 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c2f03700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78bc071be0 con 0x7f78bc0b3970 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c1f01700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f78bc0add60 0x7f78bc0cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49544/0 (socket says 172.21.15.119:49544) 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c1f01700 1 -- 172.21.15.119:0/1057441874 learned_addr learned my addr 172.21.15.119:0/1057441874 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.086+0000 7f78c2702700 1 --2- 172.21.15.119:0/1057441874 >> v2:172.21.15.145:3300/0 conn(0x7f78bc0b3970 0x7f78bc0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c0eff700 1 -- 172.21.15.119:0/1057441874 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228623512 0 0) 0x7f78bc0cd4d0 con 0x7f78bc0add60 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c0eff700 1 -- 172.21.15.119:0/1057441874 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78a80035d0 con 0x7f78bc0add60 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2702700 1 -- 172.21.15.119:0/1057441874 >> v1:172.21.15.145:6789/0 conn(0x7f78bc15dbe0 legacy=0x7f78bc0b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2702700 1 -- 172.21.15.119:0/1057441874 >> v1:172.21.15.119:6789/0 conn(0x7f78bc0add60 legacy=0x7f78bc0cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2702700 1 -- 172.21.15.119:0/1057441874 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78bc0ccba0 con 0x7f78bc0b3970 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2702700 1 --2- 172.21.15.119:0/1057441874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 0x7f78bc0ce0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d90 tx=0x7f78b400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=52223bd74285eda4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c0eff700 1 -- 172.21.15.119:0/1057441874 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b400e070 con 0x7f78bc0b3970 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c0eff700 1 -- 172.21.15.119:0/1057441874 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b400bb20 con 0x7f78bc0b3970 2024-09-13T16:25:21.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c0eff700 1 -- 172.21.15.119:0/1057441874 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b400be90 con 0x7f78bc0b3970 2024-09-13T16:25:21.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2f03700 1 -- 172.21.15.119:0/1057441874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 msgr2=0x7f78bc0ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2f03700 1 --2- 172.21.15.119:0/1057441874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 0x7f78bc0ce0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f78b4009d90 tx=0x7f78b400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:21.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2f03700 1 -- 172.21.15.119:0/1057441874 shutdown_connections 2024-09-13T16:25:21.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2f03700 1 --2- 172.21.15.119:0/1057441874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 0x7f78bc0ce0f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2f03700 1 -- 172.21.15.119:0/1057441874 >> 172.21.15.119:0/1057441874 conn(0x7f78bc0b0980 msgr2=0x7f78bc0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:21.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2f03700 1 -- 172.21.15.119:0/1057441874 shutdown_connections 2024-09-13T16:25:21.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.090+0000 7f78c2f03700 1 -- 172.21.15.119:0/1057441874 wait complete. 2024-09-13T16:25:21.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 Processor -- start 2024-09-13T16:25:21.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 -- start start 2024-09-13T16:25:21.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78bc0add60 0x7f78bc147290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 0x7f78bc150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78bc15dbe0 0x7f78bc15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc15c000 con 0x7f78bc0add60 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc15c960 con 0x7f78bc0b3970 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78bc15b380 con 0x7f78bc15dbe0 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78bc0add60 0x7f78bc147290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78bc0add60 0x7f78bc147290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36028/0 (socket says 172.21.15.119:36028) 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 -- 172.21.15.119:0/2851461471 learned_addr learned my addr 172.21.15.119:0/2851461471 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1700700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 0x7f78bc150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2702700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78bc15dbe0 0x7f78bc15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 -- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78bc15dbe0 msgr2=0x7f78bc15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78bc15dbe0 0x7f78bc15c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 -- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 msgr2=0x7f78bc150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 0x7f78bc150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 -- 172.21.15.119:0/2851461471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b4009a70 con 0x7f78bc0add60 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c1f01700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78bc0add60 0x7f78bc147290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c700 tx=0x7f78b800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78ae7fc700 1 -- 172.21.15.119:0/2851461471 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8005d90 con 0x7f78bc0add60 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78bc15b660 con 0x7f78bc0add60 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78ae7fc700 1 -- 172.21.15.119:0/2851461471 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b80070d0 con 0x7f78bc0add60 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78bc15bc20 con 0x7f78bc0add60 2024-09-13T16:25:21.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.094+0000 7f78ae7fc700 1 -- 172.21.15.119:0/2851461471 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8007440 con 0x7f78bc0add60 2024-09-13T16:25:21.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.098+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78bc0ce0f0 con 0x7f78bc0add60 2024-09-13T16:25:21.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.098+0000 7f78ae7fc700 1 -- 172.21.15.119:0/2851461471 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f78b80075a0 con 0x7f78bc0add60 2024-09-13T16:25:21.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.098+0000 7f78ae7fc700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78a0066730 0x7f78a0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.098+0000 7f78ae7fc700 1 -- 172.21.15.119:0/2851461471 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6428+0+0 (secure 0 0 0) 0x7f78b80837a0 con 0x7f78bc0add60 2024-09-13T16:25:21.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.098+0000 7f78c1700700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78a0066730 0x7f78a0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.098+0000 7f78c1700700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78a0066730 0x7f78a0068bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f78bc14f870 tx=0x7f78b0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.110+0000 7f78ae7fc700 1 -- 172.21.15.119:0/2851461471 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f78b8050870 con 0x7f78bc0add60 2024-09-13T16:25:21.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.322+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f78bc15c270 con 0x7f78bc0add60 2024-09-13T16:25:21.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.330+0000 7f78ae7fc700 1 -- 172.21.15.119:0/2851461471 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) v1 ==== 137+0+0 (secure 0 0 0) 0x7f78b8053e90 con 0x7f78bc0add60 2024-09-13T16:25:21.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78a0066730 msgr2=0x7f78a0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78a0066730 0x7f78a0068bf0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f78bc14f870 tx=0x7f78b0009040 comp rx=0 tx=0).stop 2024-09-13T16:25:21.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78bc0add60 msgr2=0x7f78bc147290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78bc0add60 0x7f78bc147290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f78b800c700 tx=0x7f78b800ca10 comp rx=0 tx=0).stop 2024-09-13T16:25:21.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 shutdown_connections 2024-09-13T16:25:21.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f78a0066730 0x7f78a0068bf0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78bc15dbe0 0x7f78bc15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f78bc0b3970 0x7f78bc150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 --2- 172.21.15.119:0/2851461471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78bc0add60 0x7f78bc147290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 >> 172.21.15.119:0/2851461471 conn(0x7f78bc0b0980 msgr2=0x7f78bc066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:21.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 shutdown_connections 2024-09-13T16:25:21.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.334+0000 7f78c2f03700 1 -- 172.21.15.119:0/2851461471 wait complete. 2024-09-13T16:25:21.340 INFO:teuthology.orchestra.run.smithi119.stderr:added data pool 4 to fsmap 2024-09-13T16:25:21.352 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:21.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb182345700 1 Processor -- start 2024-09-13T16:25:21.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb182345700 1 -- start start 2024-09-13T16:25:21.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb182345700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c0d37a0 0x7fb17c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb182345700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0d1190 con 0x7fb17c15ce70 2024-09-13T16:25:21.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb182345700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17c0ceaf0 con 0x7fb17c1607c0 2024-09-13T16:25:21.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb182345700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0d3bc0 con 0x7fb17c0d37a0 2024-09-13T16:25:21.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb17c15ce70 0x7fb17c0d1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49558/0 (socket says 172.21.15.119:49558) 2024-09-13T16:25:21.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17bfff700 1 -- 172.21.15.119:0/3729750990 learned_addr learned my addr 172.21.15.119:0/3729750990 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:21.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17b7fe700 1 --2- 172.21.15.119:0/3729750990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c0d37a0 0x7fb17c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17affd700 1 -- 172.21.15.119:0/3729750990 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299616151 0 0) 0x7fb17c0d1190 con 0x7fb17c15ce70 2024-09-13T16:25:21.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17affd700 1 -- 172.21.15.119:0/3729750990 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1680035d0 con 0x7fb17c15ce70 2024-09-13T16:25:21.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17affd700 1 -- 172.21.15.119:0/3729750990 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436844882 0 0) 0x7fb17c0ceaf0 con 0x7fb17c1607c0 2024-09-13T16:25:21.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17affd700 1 -- 172.21.15.119:0/3729750990 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb17c0d1190 con 0x7fb17c1607c0 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17b7fe700 1 -- 172.21.15.119:0/3729750990 >> v1:172.21.15.145:6789/0 conn(0x7fb17c1607c0 legacy=0x7fb17c0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17b7fe700 1 -- 172.21.15.119:0/3729750990 >> v1:172.21.15.119:6789/0 conn(0x7fb17c15ce70 legacy=0x7fb17c0d1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17b7fe700 1 -- 172.21.15.119:0/3729750990 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17c0d0860 con 0x7fb17c0d37a0 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17b7fe700 1 --2- 172.21.15.119:0/3729750990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c0d37a0 0x7fb17c0ce2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009880 tx=0x7fb16c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d138d836945e25 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17affd700 1 -- 172.21.15.119:0/3729750990 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c011830 con 0x7fb17c0d37a0 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17affd700 1 -- 172.21.15.119:0/3729750990 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16c011990 con 0x7fb17c0d37a0 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.458+0000 7fb17affd700 1 -- 172.21.15.119:0/3729750990 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16c011d20 con 0x7fb17c0d37a0 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 -- 172.21.15.119:0/3729750990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c0d37a0 msgr2=0x7fb17c0ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 --2- 172.21.15.119:0/3729750990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c0d37a0 0x7fb17c0ce2a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb16c009880 tx=0x7fb16c009b90 comp rx=0 tx=0).stop 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 -- 172.21.15.119:0/3729750990 shutdown_connections 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 --2- 172.21.15.119:0/3729750990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c0d37a0 0x7fb17c0ce2a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 -- 172.21.15.119:0/3729750990 >> 172.21.15.119:0/3729750990 conn(0x7fb17c0d2bf0 msgr2=0x7fb17c0d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:21.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 -- 172.21.15.119:0/3729750990 shutdown_connections 2024-09-13T16:25:21.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 -- 172.21.15.119:0/3729750990 wait complete. 2024-09-13T16:25:21.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 Processor -- start 2024-09-13T16:25:21.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.462+0000 7fb182345700 1 -- start start 2024-09-13T16:25:21.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb17c0d37a0 0x7fb17c1544c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c15ce70 0x7fb17c0b5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb17c1607c0 0x7fb17c0b62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0af7c0 con 0x7fb17c1607c0 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0c7da0 con 0x7fb17c0d37a0 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb17c0c8080 con 0x7fb17c15ce70 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb17c1607c0 0x7fb17c0b62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb17c1607c0 0x7fb17c0b62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36032/0 (socket says 172.21.15.119:36032) 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 -- 172.21.15.119:0/894446274 learned_addr learned my addr 172.21.15.119:0/894446274 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb17b7fe700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c15ce70 0x7fb17c0b5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb17bfff700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb17c0d37a0 0x7fb17c1544c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 -- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c15ce70 msgr2=0x7fb17c0b5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c15ce70 0x7fb17c0b5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 -- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb17c0d37a0 msgr2=0x7fb17c1544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb17c0d37a0 0x7fb17c1544c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 -- 172.21.15.119:0/894446274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16c0094b0 con 0x7fb17c1607c0 2024-09-13T16:25:21.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb1808e2700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb17c1607c0 0x7fb17c0b62d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb17400cbc0 tx=0x7fb17400ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb178ff9700 1 -- 172.21.15.119:0/894446274 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb174003c20 con 0x7fb17c1607c0 2024-09-13T16:25:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb178ff9700 1 -- 172.21.15.119:0/894446274 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb174003d80 con 0x7fb17c1607c0 2024-09-13T16:25:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17c0c8360 con 0x7fb17c1607c0 2024-09-13T16:25:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb178ff9700 1 -- 172.21.15.119:0/894446274 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17400aaa0 con 0x7fb17c1607c0 2024-09-13T16:25:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17c0c79c0 con 0x7fb17c1607c0 2024-09-13T16:25:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17c0c7b50 con 0x7fb17c1607c0 2024-09-13T16:25:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.466+0000 7fb178ff9700 1 -- 172.21.15.119:0/894446274 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb17400ac00 con 0x7fb17c1607c0 2024-09-13T16:25:21.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.470+0000 7fb178ff9700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb15c066730 0x7fb15c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.470+0000 7fb178ff9700 1 -- 172.21.15.119:0/894446274 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb174084400 con 0x7fb17c1607c0 2024-09-13T16:25:21.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.470+0000 7fb17bfff700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb15c066730 0x7fb15c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.470+0000 7fb17bfff700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb15c066730 0x7fb15c068bf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb164007fd0 tx=0x7fb164007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.478+0000 7fb178ff9700 1 -- 172.21.15.119:0/894446274 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb174051480 con 0x7fb17c1607c0 2024-09-13T16:25:21.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.682+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17c0ceaf0 con 0x7fb17c1607c0 2024-09-13T16:25:21.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.682+0000 7fb178ff9700 1 -- 172.21.15.119:0/894446274 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4742 (secure 0 0 0) 0x7fb174054aa0 con 0x7fb17c1607c0 2024-09-13T16:25:21.687 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:21.687 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:21.326524+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-09-13T16:25:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb15c066730 msgr2=0x7fb15c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb15c066730 0x7fb15c068bf0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb164007fd0 tx=0x7fb164007de0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb17c1607c0 msgr2=0x7fb17c0b62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb17c1607c0 0x7fb17c0b62d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb17400cbc0 tx=0x7fb17400ced0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 shutdown_connections 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb15c066730 0x7fb15c068bf0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb17c1607c0 0x7fb17c0b62d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb17c15ce70 0x7fb17c0b5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 --2- 172.21.15.119:0/894446274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb17c0d37a0 0x7fb17c1544c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 >> 172.21.15.119:0/894446274 conn(0x7fb17c0d2bf0 msgr2=0x7fb17c15b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 shutdown_connections 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.686+0000 7fb182345700 1 -- 172.21.15.119:0/894446274 wait complete. 2024-09-13T16:25:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 7 2024-09-13T16:25:21.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:21.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 Processor -- start 2024-09-13T16:25:21.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- start start 2024-09-13T16:25:21.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc1594f0 0x7f46fc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f46fc06da70 0x7f46fc0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f46fc0b3630 0x7f46fc15d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc0b2280 con 0x7f46fc0b3630 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc06deb0 con 0x7f46fc06da70 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0cc890 con 0x7f46fc1594f0 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f46fc0b3630 0x7f46fc15d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f46fc0b3630 0x7f46fc15d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36044/0 (socket says 172.21.15.119:36044) 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 -- 172.21.15.119:0/1920657839 learned_addr learned my addr 172.21.15.119:0/1920657839 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46f3fff700 1 --2- 172.21.15.119:0/1920657839 >> v2:172.21.15.145:3300/0 conn(0x7f46fc06da70 0x7f46fc0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fa59c700 1 --2- 172.21.15.119:0/1920657839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc1594f0 0x7f46fc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 -- 172.21.15.119:0/1920657839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc1594f0 msgr2=0x7f46fc06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 --2- 172.21.15.119:0/1920657839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc1594f0 0x7f46fc06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 -- 172.21.15.119:0/1920657839 >> v2:172.21.15.145:3300/0 conn(0x7f46fc06da70 msgr2=0x7f46fc0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:21.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 --2- 172.21.15.119:0/1920657839 >> v2:172.21.15.145:3300/0 conn(0x7f46fc06da70 0x7f46fc0b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 -- 172.21.15.119:0/1920657839 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0b1970 con 0x7f46fc0b3630 2024-09-13T16:25:21.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fa59c700 1 --2- 172.21.15.119:0/1920657839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc1594f0 0x7f46fc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:21.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fad9d700 1 --2- 172.21.15.119:0/1920657839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b3630 0x7f46fc15d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0b1b60 tx=0x7f46f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc049b13b4b2ddd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46f9d9b700 1 -- 172.21.15.119:0/1920657839 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f400e070 con 0x7f46fc0b3630 2024-09-13T16:25:21.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46f9d9b700 1 -- 172.21.15.119:0/1920657839 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f4003a40 con 0x7f46fc0b3630 2024-09-13T16:25:21.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46f9d9b700 1 -- 172.21.15.119:0/1920657839 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4003df0 con 0x7f46fc0b3630 2024-09-13T16:25:21.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- 172.21.15.119:0/1920657839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b3630 msgr2=0x7f46fc15d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 --2- 172.21.15.119:0/1920657839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b3630 0x7f46fc15d150 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0b1b60 tx=0x7f46f400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:21.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- 172.21.15.119:0/1920657839 shutdown_connections 2024-09-13T16:25:21.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 --2- 172.21.15.119:0/1920657839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b3630 0x7f46fc15d150 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 --2- 172.21.15.119:0/1920657839 >> v2:172.21.15.145:3300/0 conn(0x7f46fc06da70 0x7f46fc0b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 --2- 172.21.15.119:0/1920657839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc1594f0 0x7f46fc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- 172.21.15.119:0/1920657839 >> 172.21.15.119:0/1920657839 conn(0x7f46fc06a350 msgr2=0x7f46fc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:21.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- 172.21.15.119:0/1920657839 shutdown_connections 2024-09-13T16:25:21.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.814+0000 7f46fb59e700 1 -- 172.21.15.119:0/1920657839 wait complete. 2024-09-13T16:25:21.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 Processor -- start 2024-09-13T16:25:21.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 -- start start 2024-09-13T16:25:21.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc06da70 0x7f46fc14d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46fc0b3630 0x7f46fc151690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc1594f0 0x7f46fc0ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc14dd30 con 0x7f46fc1594f0 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc14fa60 con 0x7f46fc0b3630 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc14f4a0 con 0x7f46fc06da70 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc1594f0 0x7f46fc0ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc1594f0 0x7f46fc0ae440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36056/0 (socket says 172.21.15.119:36056) 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 -- 172.21.15.119:0/2093311237 learned_addr learned my addr 172.21.15.119:0/2093311237 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46f3fff700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46fc0b3630 0x7f46fc151690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fa59c700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc06da70 0x7f46fc14d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 -- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc06da70 msgr2=0x7f46fc14d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc06da70 0x7f46fc14d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 -- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46fc0b3630 msgr2=0x7f46fc151690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:21.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46fc0b3630 0x7f46fc151690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 -- 172.21.15.119:0/2093311237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f4009b30 con 0x7f46fc1594f0 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fa59c700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc06da70 0x7f46fc14d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fad9d700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc1594f0 0x7f46fc0ae440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f46f400b370 tx=0x7f46f4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46f2ffd700 1 -- 172.21.15.119:0/2093311237 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4004440 con 0x7f46fc1594f0 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46f2ffd700 1 -- 172.21.15.119:0/2093311237 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f40092e0 con 0x7f46fc1594f0 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc14f780 con 0x7f46fc1594f0 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46f2ffd700 1 -- 172.21.15.119:0/2093311237 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f40047e0 con 0x7f46fc1594f0 2024-09-13T16:25:21.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.818+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc14d9e0 con 0x7f46fc1594f0 2024-09-13T16:25:21.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.822+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc150fd0 con 0x7f46fc1594f0 2024-09-13T16:25:21.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.822+0000 7f46f2ffd700 1 -- 172.21.15.119:0/2093311237 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f46f4012430 con 0x7f46fc1594f0 2024-09-13T16:25:21.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.822+0000 7f46f2ffd700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46dc066730 0x7f46dc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:21.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.822+0000 7f46f2ffd700 1 -- 172.21.15.119:0/2093311237 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f46f4085710 con 0x7f46fc1594f0 2024-09-13T16:25:21.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.822+0000 7f46fa59c700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46dc066730 0x7f46dc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:21.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.822+0000 7f46fa59c700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46dc066730 0x7f46dc068bf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00c9a0 tx=0x7f46ec008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:21.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:21.830+0000 7f46f2ffd700 1 -- 172.21.15.119:0/2093311237 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f46fc150fd0 con 0x7f46fc1594f0 2024-09-13T16:25:22.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.002+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46fc1513b0 con 0x7f46fc1594f0 2024-09-13T16:25:22.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46f2ffd700 1 -- 172.21.15.119:0/2093311237 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f46fc1513b0 con 0x7f46fc1594f0 2024-09-13T16:25:22.008 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:22.008 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46dc066730 msgr2=0x7f46dc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46dc066730 0x7f46dc068bf0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00c9a0 tx=0x7f46ec008040 comp rx=0 tx=0).stop 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc1594f0 msgr2=0x7f46fc0ae440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc1594f0 0x7f46fc0ae440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f46f400b370 tx=0x7f46f4004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 shutdown_connections 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46dc066730 0x7f46dc068bf0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc1594f0 0x7f46fc0ae440 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46fc0b3630 0x7f46fc151690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 --2- 172.21.15.119:0/2093311237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46fc06da70 0x7f46fc14d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 >> 172.21.15.119:0/2093311237 conn(0x7f46fc06a350 msgr2=0x7f46fc0af310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 shutdown_connections 2024-09-13T16:25:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.006+0000 7f46fb59e700 1 -- 172.21.15.119:0/2093311237 wait complete. 2024-09-13T16:25:22.023 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:22.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95d32700 1 Processor -- start 2024-09-13T16:25:22.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95d32700 1 -- start start 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95d32700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec901594f0 0x7fec9006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95d32700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 0x7fec900b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95d32700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec900b2280 con 0x7fec900b3630 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95d32700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec90070c10 con 0x7fec901594f0 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95d32700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec9006deb0 con 0x7fec9006da70 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95531700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fec900b3630 0x7fec9015d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49560/0 (socket says 172.21.15.119:49560) 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec95531700 1 -- 172.21.15.119:0/2137271886 learned_addr learned my addr 172.21.15.119:0/2137271886 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:22.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8ffff700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 0x7fec900b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec94d30700 1 --2- 172.21.15.119:0/2137271886 >> v2:172.21.15.145:3300/0 conn(0x7fec901594f0 0x7fec9006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279363372 0 0) 0x7fec900b2280 con 0x7fec900b3630 2024-09-13T16:25:22.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec7c0035d0 con 0x7fec900b3630 2024-09-13T16:25:22.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 255582893 0 0) 0x7fec7c0035d0 con 0x7fec900b3630 2024-09-13T16:25:22.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 msgr2=0x7fec900b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:22.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 0x7fec900b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 >> v2:172.21.15.145:3300/0 conn(0x7fec901594f0 msgr2=0x7fec9006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 --2- 172.21.15.119:0/2137271886 >> v2:172.21.15.145:3300/0 conn(0x7fec901594f0 0x7fec9006d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec900cc890 con 0x7fec900b3630 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fec88001670 con 0x7fec900b3630 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.126+0000 7fec94d30700 1 --2- 172.21.15.119:0/2137271886 >> v2:172.21.15.145:3300/0 conn(0x7fec901594f0 0x7fec9006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 >> v1:172.21.15.119:6789/0 conn(0x7fec900b3630 legacy=0x7fec9015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c003a20 0x7fec7c003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 0x7fec7c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec7c004db0 0x7fec7c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec90070c10 con 0x7fec7c004db0 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9006deb0 con 0x7fec7c003a20 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7c00ba30 con 0x7fec9006da70 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec7c004db0 0x7fec7c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec94d30700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c003a20 0x7fec7c003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8ffff700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 0x7fec7c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 -- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 msgr2=0x7fec7c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec9006da70 0x7fec7c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 -- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c003a20 msgr2=0x7fec7c003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c003a20 0x7fec7c003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 -- 172.21.15.119:0/2137271886 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec7c0035d0 con 0x7fec7c004db0 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec94d30700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec7c003a20 0x7fec7c003e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec7c004db0 0x7fec7c005230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fec8800c320 tx=0x7fec7c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b61d0d6302424f9c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec8800ce60 con 0x7fec7c004db0 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec88007bc0 con 0x7fec7c004db0 2024-09-13T16:25:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec8f7fe700 1 -- 172.21.15.119:0/2137271886 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88005bc0 con 0x7fec7c004db0 2024-09-13T16:25:22.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95d32700 1 -- 172.21.15.119:0/2137271886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec7c004db0 msgr2=0x7fec7c005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95d32700 1 --2- 172.21.15.119:0/2137271886 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec7c004db0 0x7fec7c005230 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fec8800c320 tx=0x7fec7c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95531700 1 -- 172.21.15.119:0/2137271886 reap_dead start 2024-09-13T16:25:22.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95d32700 1 -- 172.21.15.119:0/2137271886 shutdown_connections 2024-09-13T16:25:22.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95d32700 1 -- 172.21.15.119:0/2137271886 >> 172.21.15.119:0/2137271886 conn(0x7fec9006a350 msgr2=0x7fec9006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:22.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95d32700 1 -- 172.21.15.119:0/2137271886 shutdown_connections 2024-09-13T16:25:22.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.130+0000 7fec95d32700 1 -- 172.21.15.119:0/2137271886 wait complete. 2024-09-13T16:25:22.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 Processor -- start 2024-09-13T16:25:22.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 -- start start 2024-09-13T16:25:22.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec90145a20 0x7fec901652c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec90165800 0x7fec90149740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec90149c80 0x7fec9014daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9014f710 con 0x7fec90165800 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec9014d610 con 0x7fec90145a20 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec90166260 con 0x7fec90149c80 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec90165800 0x7fec90149740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec90165800 0x7fec90149740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36072/0 (socket says 172.21.15.119:36072) 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 -- 172.21.15.119:0/1562847940 learned_addr learned my addr 172.21.15.119:0/1562847940 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec94d30700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec90145a20 0x7fec901652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95531700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec90149c80 0x7fec9014daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 -- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec90149c80 msgr2=0x7fec9014daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec90149c80 0x7fec9014daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 -- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec90145a20 msgr2=0x7fec901652c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec90145a20 0x7fec901652c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 -- 172.21.15.119:0/1562847940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec90165ca0 con 0x7fec90165800 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95531700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec90149c80 0x7fec9014daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8ffff700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec90165800 0x7fec90149740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec78011e60 tx=0x7fec7800f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8d7fa700 1 -- 172.21.15.119:0/1562847940 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7800e040 con 0x7fec90165800 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8d7fa700 1 -- 172.21.15.119:0/1562847940 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7800f040 con 0x7fec90165800 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec90165f80 con 0x7fec90165800 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec8d7fa700 1 -- 172.21.15.119:0/1562847940 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec78015650 con 0x7fec90165800 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.134+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec90145e00 con 0x7fec90165800 2024-09-13T16:25:22.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.138+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec9014d7a0 con 0x7fec90165800 2024-09-13T16:25:22.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.138+0000 7fec8d7fa700 1 -- 172.21.15.119:0/1562847940 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fec780157b0 con 0x7fec90165800 2024-09-13T16:25:22.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.138+0000 7fec8d7fa700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.138+0000 7fec8d7fa700 1 -- 172.21.15.119:0/1562847940 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fec78084fd0 con 0x7fec90165800 2024-09-13T16:25:22.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.138+0000 7fec94d30700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.138+0000 7fec94d30700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec800098e0 tx=0x7fec80009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.146+0000 7fec8d7fa700 1 -- 172.21.15.119:0/1562847940 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fec78052040 con 0x7fec90165800 2024-09-13T16:25:22.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.318+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec900b2710 con 0x7fec90165800 2024-09-13T16:25:22.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.322+0000 7fec8d7fa700 1 -- 172.21.15.119:0/1562847940 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7fec78055660 con 0x7fec90165800 2024-09-13T16:25:22.325 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:22.325 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.322+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 msgr2=0x7fec70068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.322+0000 7fec95d32700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec800098e0 tx=0x7fec80009240 comp rx=0 tx=0).stop 2024-09-13T16:25:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.322+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec90165800 msgr2=0x7fec90149740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.322+0000 7fec95d32700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec90165800 0x7fec90149740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fec78011e60 tx=0x7fec7800f490 comp rx=0 tx=0).stop 2024-09-13T16:25:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 shutdown_connections 2024-09-13T16:25:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec90149c80 0x7fec9014daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec90165800 0x7fec90149740 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 --2- 172.21.15.119:0/1562847940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec90145a20 0x7fec901652c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 >> 172.21.15.119:0/1562847940 conn(0x7fec9006a350 msgr2=0x7fec9015c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 shutdown_connections 2024-09-13T16:25:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.326+0000 7fec95d32700 1 -- 172.21.15.119:0/1562847940 wait complete. 2024-09-13T16:25:22.361 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:22.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f73fff700 1 Processor -- start 2024-09-13T16:25:22.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f73fff700 1 -- start start 2024-09-13T16:25:22.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f73fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5f740af670 0x7f5f740ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f73fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f740ce030 con 0x7f5f740afae0 2024-09-13T16:25:22.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f73fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f740b3980 con 0x7f5f740d2a30 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f73fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f740cd9e0 con 0x7f5f740af670 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f727fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5f740af670 0x7f5f740ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f727fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5f740af670 0x7f5f740ce4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53590/0 (socket says 172.21.15.119:53590) 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f727fc700 1 -- 172.21.15.119:0/1697127220 learned_addr learned my addr 172.21.15.119:0/1697127220 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f737fe700 1 --1- 172.21.15.119:0/1697127220 >> v1:172.21.15.119:6789/0 conn(0x7f5f740afae0 0x7f5f740b2c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41878/0 (socket says 172.21.15.119:41878) 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f727fc700 1 -- 172.21.15.119:0/1697127220 >> v1:172.21.15.145:6789/0 conn(0x7f5f740d2a30 legacy=0x7f5f740d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f727fc700 1 -- 172.21.15.119:0/1697127220 >> v1:172.21.15.119:6789/0 conn(0x7f5f740afae0 legacy=0x7f5f740b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f727fc700 1 -- 172.21.15.119:0/1697127220 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740cdcc0 con 0x7f5f740af670 2024-09-13T16:25:22.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f727fc700 1 --2- 172.21.15.119:0/1697127220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f740ce4c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009e50 tx=0x7f5f6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6d61fa9ac440fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f71ffb700 1 -- 172.21.15.119:0/1697127220 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6400e070 con 0x7f5f740af670 2024-09-13T16:25:22.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f71ffb700 1 -- 172.21.15.119:0/1697127220 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f6400b9c0 con 0x7f5f740af670 2024-09-13T16:25:22.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.474+0000 7f5f71ffb700 1 -- 172.21.15.119:0/1697127220 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f6400bd70 con 0x7f5f740af670 2024-09-13T16:25:22.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 -- 172.21.15.119:0/1697127220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 msgr2=0x7f5f740ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 --2- 172.21.15.119:0/1697127220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f740ce4c0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5f64009e50 tx=0x7f5f6400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:22.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 -- 172.21.15.119:0/1697127220 shutdown_connections 2024-09-13T16:25:22.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 --2- 172.21.15.119:0/1697127220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f740ce4c0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 -- 172.21.15.119:0/1697127220 >> 172.21.15.119:0/1697127220 conn(0x7f5f7408c0f0 msgr2=0x7f5f7408c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:22.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 -- 172.21.15.119:0/1697127220 shutdown_connections 2024-09-13T16:25:22.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 -- 172.21.15.119:0/1697127220 wait complete. 2024-09-13T16:25:22.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 Processor -- start 2024-09-13T16:25:22.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.478+0000 7f5f73fff700 1 -- start start 2024-09-13T16:25:22.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f74157110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f740afae0 0x7f5f740cc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740d2a30 0x7f5f740c6d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740c7270 con 0x7f5f740af670 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740c9c10 con 0x7f5f740d2a30 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f740c9eb0 con 0x7f5f740afae0 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f74157110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f74157110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53606/0 (socket says 172.21.15.119:53606) 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 -- 172.21.15.119:0/3355233818 learned_addr learned my addr 172.21.15.119:0/3355233818 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f727fc700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f740afae0 0x7f5f740cc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f737fe700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740d2a30 0x7f5f740c6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 -- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f740afae0 msgr2=0x7f5f740cc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f740afae0 0x7f5f740cc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 -- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740d2a30 msgr2=0x7f5f740c6d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740d2a30 0x7f5f740c6d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 -- 172.21.15.119:0/3355233818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f64009b30 con 0x7f5f740af670 2024-09-13T16:25:22.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f72ffd700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f74157110 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00c500 tx=0x7f5f5c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f6b7fe700 1 -- 172.21.15.119:0/3355233818 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0052d0 con 0x7f5f740af670 2024-09-13T16:25:22.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f6b7fe700 1 -- 172.21.15.119:0/3355233818 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c005430 con 0x7f5f740af670 2024-09-13T16:25:22.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f740ca190 con 0x7f5f740af670 2024-09-13T16:25:22.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f6b7fe700 1 -- 172.21.15.119:0/3355233818 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f5c0055a0 con 0x7f5f740af670 2024-09-13T16:25:22.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.482+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f740ccad0 con 0x7f5f740af670 2024-09-13T16:25:22.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.486+0000 7f5f6b7fe700 1 -- 172.21.15.119:0/3355233818 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5f5c0038e0 con 0x7f5f740af670 2024-09-13T16:25:22.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.486+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f740ca6b0 con 0x7f5f740af670 2024-09-13T16:25:22.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.486+0000 7f5f6b7fe700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f54066730 0x7f5f54068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.486+0000 7f5f6b7fe700 1 -- 172.21.15.119:0/3355233818 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5f5c0837f0 con 0x7f5f740af670 2024-09-13T16:25:22.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.486+0000 7f5f727fc700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f54066730 0x7f5f54068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.486+0000 7f5f727fc700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f54066730 0x7f5f54068bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400b370 tx=0x7f5f64009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.490+0000 7f5f6b7fe700 1 -- 172.21.15.119:0/3355233818 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5f5c050870 con 0x7f5f740af670 2024-09-13T16:25:22.637 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:25:22.634+0000 7f1b3432e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:25:22.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.694+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f7408c6f0 con 0x7f5f740af670 2024-09-13T16:25:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.694+0000 7f5f6b7fe700 1 -- 172.21.15.119:0/3355233818 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f5f5c053e90 con 0x7f5f740af670 2024-09-13T16:25:22.698 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:22.698 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:21.326524+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f54066730 msgr2=0x7f5f54068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f54066730 0x7f5f54068bf0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5f6400b370 tx=0x7f5f64009000 comp rx=0 tx=0).stop 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 msgr2=0x7f5f74157110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f74157110 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5f5c00c500 tx=0x7f5f5c00c810 comp rx=0 tx=0).stop 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 shutdown_connections 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f54066730 0x7f5f54068bf0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740d2a30 0x7f5f740c6d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f740afae0 0x7f5f740cc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 --2- 172.21.15.119:0/3355233818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740af670 0x7f5f74157110 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 >> 172.21.15.119:0/3355233818 conn(0x7f5f7408c0f0 msgr2=0x7f5f7415ea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:22.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 shutdown_connections 2024-09-13T16:25:22.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.698+0000 7f5f73fff700 1 -- 172.21.15.119:0/3355233818 wait complete. 2024-09-13T16:25:22.702 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 7 2024-09-13T16:25:22.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae3fff700 1 Processor -- start 2024-09-13T16:25:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae3fff700 1 -- start start 2024-09-13T16:25:22.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7faae4143f50 0x7faae406c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae40b3410 0x7faae4164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae3fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faae40b1c40 con 0x7faae4144330 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae3fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faae406cdf0 con 0x7faae4143f50 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae406d0d0 con 0x7faae40b3410 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae2ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7faae4143f50 0x7faae406c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae27fc700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7faae4144330 0x7faae40cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:57552/0 (socket says 172.21.15.119:57552) 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae27fc700 1 -- 172.21.15.119:0/2086885669 learned_addr learned my addr 172.21.15.119:0/2086885669 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.822+0000 7faae37fe700 1 --2- 172.21.15.119:0/2086885669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae40b3410 0x7faae4164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae1ffb700 1 -- 172.21.15.119:0/2086885669 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554349875 0 0) 0x7faae40b1c40 con 0x7faae4144330 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae1ffb700 1 -- 172.21.15.119:0/2086885669 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faacc0035d0 con 0x7faae4144330 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae2ffd700 1 -- 172.21.15.119:0/2086885669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae40b3410 msgr2=0x7faae4164330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae2ffd700 1 --2- 172.21.15.119:0/2086885669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae40b3410 0x7faae4164330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae2ffd700 1 -- 172.21.15.119:0/2086885669 >> v1:172.21.15.145:6789/0 conn(0x7faae4144330 legacy=0x7faae40cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae2ffd700 1 -- 172.21.15.119:0/2086885669 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faae406d3b0 con 0x7faae4143f50 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae2ffd700 1 --2- 172.21.15.119:0/2086885669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4143f50 0x7faae406c5a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faad8009850 tx=0x7faad8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26ac28aec695b8e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae1ffb700 1 -- 172.21.15.119:0/2086885669 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8011830 con 0x7faae4143f50 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae1ffb700 1 -- 172.21.15.119:0/2086885669 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faad8011990 con 0x7faae4143f50 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae1ffb700 1 -- 172.21.15.119:0/2086885669 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faad8011d40 con 0x7faae4143f50 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 -- 172.21.15.119:0/2086885669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4143f50 msgr2=0x7faae406c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 --2- 172.21.15.119:0/2086885669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4143f50 0x7faae406c5a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faad8009850 tx=0x7faad8009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 -- 172.21.15.119:0/2086885669 shutdown_connections 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 --2- 172.21.15.119:0/2086885669 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae40b3410 0x7faae4164330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 --2- 172.21.15.119:0/2086885669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4143f50 0x7faae406c5a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 -- 172.21.15.119:0/2086885669 >> 172.21.15.119:0/2086885669 conn(0x7faae406a350 msgr2=0x7faae406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:22.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 -- 172.21.15.119:0/2086885669 shutdown_connections 2024-09-13T16:25:22.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.826+0000 7faae3fff700 1 -- 172.21.15.119:0/2086885669 wait complete. 2024-09-13T16:25:22.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 Processor -- start 2024-09-13T16:25:22.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 -- start start 2024-09-13T16:25:22.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faae40b3410 0x7faae40c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae4143f50 0x7faae413e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4144330 0x7faae40c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae40c4730 con 0x7faae4144330 2024-09-13T16:25:22.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faae40c7b20 con 0x7faae40b3410 2024-09-13T16:25:22.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faae40c4450 con 0x7faae4143f50 2024-09-13T16:25:22.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4144330 0x7faae40c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4144330 0x7faae40c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53628/0 (socket says 172.21.15.119:53628) 2024-09-13T16:25:22.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae37fe700 1 -- 172.21.15.119:0/4240985223 learned_addr learned my addr 172.21.15.119:0/4240985223 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:22.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.830+0000 7faae2ffd700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faae40b3410 0x7faae40c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae27fc700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae4143f50 0x7faae413e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae37fe700 1 -- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae4143f50 msgr2=0x7faae413e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae37fe700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae4143f50 0x7faae413e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae37fe700 1 -- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faae40b3410 msgr2=0x7faae40c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:22.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae37fe700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faae40b3410 0x7faae40c75e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:22.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae37fe700 1 -- 172.21.15.119:0/4240985223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad80094b0 con 0x7faae4144330 2024-09-13T16:25:22.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae37fe700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4144330 0x7faae40c55f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faadc00eb60 tx=0x7faadc00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faacb7fe700 1 -- 172.21.15.119:0/4240985223 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc009740 con 0x7faae4144330 2024-09-13T16:25:22.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faacb7fe700 1 -- 172.21.15.119:0/4240985223 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc009e00 con 0x7faae4144330 2024-09-13T16:25:22.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faae40aa490 con 0x7faae4144330 2024-09-13T16:25:22.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faacb7fe700 1 -- 172.21.15.119:0/4240985223 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc00ca70 con 0x7faae4144330 2024-09-13T16:25:22.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faae40aaa50 con 0x7faae4144330 2024-09-13T16:25:22.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faae40aa2f0 con 0x7faae4144330 2024-09-13T16:25:22.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.834+0000 7faacb7fe700 1 -- 172.21.15.119:0/4240985223 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7faadc00cbd0 con 0x7faae4144330 2024-09-13T16:25:22.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.838+0000 7faacb7fe700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faad0066730 0x7faad0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:22.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.838+0000 7faacb7fe700 1 -- 172.21.15.119:0/4240985223 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7faadc084480 con 0x7faae4144330 2024-09-13T16:25:22.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.838+0000 7faae2ffd700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faad0066730 0x7faad0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:22.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.838+0000 7faae2ffd700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faad0066730 0x7faad0068bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faad80090b0 tx=0x7faad8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:22.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:22.846+0000 7faacb7fe700 1 -- 172.21.15.119:0/4240985223 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7faae40aa2f0 con 0x7faae4144330 2024-09-13T16:25:23.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.014+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faae40aa2f0 con 0x7faae4144330 2024-09-13T16:25:23.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.014+0000 7faacb7fe700 1 -- 172.21.15.119:0/4240985223 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7faae40aa2f0 con 0x7faae4144330 2024-09-13T16:25:23.019 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:23.020 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:23.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faad0066730 msgr2=0x7faad0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faad0066730 0x7faad0068bf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faad80090b0 tx=0x7faad8008030 comp rx=0 tx=0).stop 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4144330 msgr2=0x7faae40c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4144330 0x7faae40c55f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7faadc00eb60 tx=0x7faadc00ee70 comp rx=0 tx=0).stop 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 shutdown_connections 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faad0066730 0x7faad0068bf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faae4144330 0x7faae40c55f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faae4143f50 0x7faae413e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 --2- 172.21.15.119:0/4240985223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faae40b3410 0x7faae40c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 >> 172.21.15.119:0/4240985223 conn(0x7faae406a350 msgr2=0x7faae406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 shutdown_connections 2024-09-13T16:25:23.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.018+0000 7faae3fff700 1 -- 172.21.15.119:0/4240985223 wait complete. 2024-09-13T16:25:23.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:25:16.907443+0000', 'modified': '2024-09-13T16:25:21.326524+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4285': {'gid': 4285, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6835/3556289056', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 3556289056}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 3556289056}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-09-13T16:25:23.439 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:25:23.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:25:23.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32eed6b700 1 Processor -- start 2024-09-13T16:25:23.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32eed6b700 1 -- start start 2024-09-13T16:25:23.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32eed6b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e81594f0 0x7f32e806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32eed6b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f32e80b3800 0x7f32e806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32eed6b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e80b2280 con 0x7f32e806da70 2024-09-13T16:25:23.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32eed6b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e806deb0 con 0x7f32e80b3800 2024-09-13T16:25:23.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32eed6b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80cc890 con 0x7f32e81594f0 2024-09-13T16:25:23.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ee56a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f32e80b3800 0x7f32e806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ed568700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f32e806da70 0x7f32e80b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41890/0 (socket says 172.21.15.119:41890) 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ed568700 1 -- 172.21.15.119:0/770857447 learned_addr learned my addr 172.21.15.119:0/770857447 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32edd69700 1 --2- 172.21.15.119:0/770857447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e81594f0 0x7f32e806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ee56a700 1 -- 172.21.15.119:0/770857447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e81594f0 msgr2=0x7f32e806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ee56a700 1 --2- 172.21.15.119:0/770857447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e81594f0 0x7f32e806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ee56a700 1 -- 172.21.15.119:0/770857447 >> v1:172.21.15.119:6789/0 conn(0x7f32e806da70 legacy=0x7f32e80b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ee56a700 1 -- 172.21.15.119:0/770857447 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e80b1970 con 0x7f32e80b3800 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32edd69700 1 --2- 172.21.15.119:0/770857447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e81594f0 0x7f32e806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:23.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ee56a700 1 --2- 172.21.15.119:0/770857447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e80b3800 0x7f32e806fc00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32e80b1b60 tx=0x7f32e000d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd4221020c267b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ecd67700 1 -- 172.21.15.119:0/770857447 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0010070 con 0x7f32e80b3800 2024-09-13T16:25:23.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ecd67700 1 -- 172.21.15.119:0/770857447 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e000b2e0 con 0x7f32e80b3800 2024-09-13T16:25:23.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.562+0000 7f32ecd67700 1 -- 172.21.15.119:0/770857447 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e000dcc0 con 0x7f32e80b3800 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 -- 172.21.15.119:0/770857447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e80b3800 msgr2=0x7f32e806fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 --2- 172.21.15.119:0/770857447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e80b3800 0x7f32e806fc00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f32e80b1b60 tx=0x7f32e000d330 comp rx=0 tx=0).stop 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 -- 172.21.15.119:0/770857447 shutdown_connections 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 --2- 172.21.15.119:0/770857447 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e80b3800 0x7f32e806fc00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 --2- 172.21.15.119:0/770857447 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e81594f0 0x7f32e806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 -- 172.21.15.119:0/770857447 >> 172.21.15.119:0/770857447 conn(0x7f32e806a350 msgr2=0x7f32e806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 -- 172.21.15.119:0/770857447 shutdown_connections 2024-09-13T16:25:23.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 -- 172.21.15.119:0/770857447 wait complete. 2024-09-13T16:25:23.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 Processor -- start 2024-09-13T16:25:23.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.566+0000 7f32eed6b700 1 -- start start 2024-09-13T16:25:23.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32e806da70 0x7f32e815c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e80b3800 0x7f32e815abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e81594f0 0x7f32e815b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e8141400 con 0x7f32e81594f0 2024-09-13T16:25:23.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e8140dd0 con 0x7f32e806da70 2024-09-13T16:25:23.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e8158c40 con 0x7f32e80b3800 2024-09-13T16:25:23.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e81594f0 0x7f32e815b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e81594f0 0x7f32e815b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53652/0 (socket says 172.21.15.119:53652) 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 -- 172.21.15.119:0/1779939982 learned_addr learned my addr 172.21.15.119:0/1779939982 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ed568700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e80b3800 0x7f32e815abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32edd69700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32e806da70 0x7f32e815c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 -- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e80b3800 msgr2=0x7f32e815abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e80b3800 0x7f32e815abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 -- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32e806da70 msgr2=0x7f32e815c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32e806da70 0x7f32e815c490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 -- 172.21.15.119:0/1779939982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e000bc80 con 0x7f32e81594f0 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ed568700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e80b3800 0x7f32e815abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32ee56a700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e81594f0 0x7f32e815b100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f32e80ce710 tx=0x7f32e0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32de7fc700 1 -- 172.21.15.119:0/1779939982 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e000a8a0 con 0x7f32e81594f0 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32de7fc700 1 -- 172.21.15.119:0/1779939982 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e000aa00 con 0x7f32e81594f0 2024-09-13T16:25:23.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32e80a85e0 con 0x7f32e81594f0 2024-09-13T16:25:23.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32de7fc700 1 -- 172.21.15.119:0/1779939982 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e000adf0 con 0x7f32e81594f0 2024-09-13T16:25:23.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.570+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e8141020 con 0x7f32e81594f0 2024-09-13T16:25:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.574+0000 7f32de7fc700 1 -- 172.21.15.119:0/1779939982 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f32e0004040 con 0x7f32e81594f0 2024-09-13T16:25:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.574+0000 7f32de7fc700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32cc066730 0x7f32cc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.574+0000 7f32de7fc700 1 -- 172.21.15.119:0/1779939982 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f32e0087c10 con 0x7f32e81594f0 2024-09-13T16:25:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.574+0000 7f32edd69700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32cc066730 0x7f32cc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.574+0000 7f32edd69700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32cc066730 0x7f32cc068bf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006fd0 tx=0x7f32d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 Processor -- start 2024-09-13T16:25:23.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- start start 2024-09-13T16:25:23.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82230 0x1a82610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a60e90 0x1a4a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a4add0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x19a9d80 con 0x1a82230 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19a7510 con 0x1a82b50 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x19a2e00 con 0x1a60e90 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82230 0x1a82610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82230 0x1a82610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53664/0 (socket says 172.21.15.119:53664) 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 -- 172.21.15.119:0/2373480631 learned_addr learned my addr 172.21.15.119:0/2373480631 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32ee56a700 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a4add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32ed568700 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a60e90 0x1a4a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 -- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a60e90 msgr2=0x1a4a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a60e90 0x1a4a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 -- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 msgr2=0x1a4add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a4add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 -- 172.21.15.119:0/2373480631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1815960 con 0x1a82230 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32edd69700 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82230 0x1a82610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f32d800a600 tx=0x7f32d8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d830528f385d58d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32cb7fe700 1 -- 172.21.15.119:0/2373480631 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d801a040 con 0x1a82230 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32cb7fe700 1 -- 172.21.15.119:0/2373480631 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d800c040 con 0x1a82230 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32cb7fe700 1 -- 172.21.15.119:0/2373480631 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8015c10 con 0x1a82230 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- 172.21.15.119:0/2373480631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82230 msgr2=0x1a82610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82230 0x1a82610 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f32d800a600 tx=0x7f32d8012000 comp rx=0 tx=0).stop 2024-09-13T16:25:23.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- 172.21.15.119:0/2373480631 shutdown_connections 2024-09-13T16:25:23.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a4add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a60e90 0x1a4a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 --2- 172.21.15.119:0/2373480631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82230 0x1a82610 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- 172.21.15.119:0/2373480631 >> 172.21.15.119:0/2373480631 conn(0x17ca4d0 msgr2=0x1a63e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- 172.21.15.119:0/2373480631 shutdown_connections 2024-09-13T16:25:23.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.578+0000 7f32f327e740 1 -- 172.21.15.119:0/2373480631 wait complete. 2024-09-13T16:25:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 -- start start 2024-09-13T16:25:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 Processor -- start 2024-09-13T16:25:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a60e90 0x1a5cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a3a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82b50 0x1a3c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1960730 con 0x1a82b50 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19665b0 con 0x1a60e90 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x184eaa0 con 0x1a82230 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 10 client.0 _ll_register_callbacks cb 0x199c610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82b50 0x1a3c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82b50 0x1a3c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53672/0 (socket says 172.21.15.119:53672) 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 -- 172.21.15.119:0/2942956452 learned_addr learned my addr 172.21.15.119:0/2942956452 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32edd69700 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a60e90 0x1a5cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ed568700 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a3a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 -- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 msgr2=0x1a3a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a3a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 -- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a60e90 msgr2=0x1a5cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a60e90 0x1a5cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 -- 172.21.15.119:0/2942956452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d800fa00 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32ee56a700 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82b50 0x1a3c880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32e000bc50 tx=0x7f32e0058a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6774c597a8df56a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32c8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32c8ff9700 1 -- 172.21.15.119:0/2942956452 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e0070370 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32c8ff9700 1 -- 172.21.15.119:0/2942956452 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f32e0060d50 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x17d71e0 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32c8ff9700 1 -- 172.21.15.119:0/2942956452 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32e0075120 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 10 client.4398 fetch_fsmap learned FSMap version 7 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x1969220 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32c8ff9700 1 -- 172.21.15.119:0/2942956452 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x17d71e0 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32c8ff9700 1 -- 172.21.15.119:0/2942956452 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f32e0057e50 con 0x1a82b50 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 10 client.4398 fetch_fsmap finished waiting for FSMap version 7 2024-09-13T16:25:23.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 10 client.4398 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:25:23.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 20 client.4398 populate_metadata read hostname 'smithi119' 2024-09-13T16:25:23.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a7e140 0x1a7e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 4 client.4398 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:25:23.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 --> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1821460 con 0x1a7e140 2024-09-13T16:25:23.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.582+0000 7f32edd69700 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a7e140 0x1a7e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32edd69700 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a7e140 0x1a7e520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bccb2ce50ab6f80a server_cookie=2bcde2b30485d0f9 in_seq=0 out_seq=0 2024-09-13T16:25:23.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32c8ff9700 10 client.4398 ms_handle_connect on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:23.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32c8ff9700 1 -- 172.21.15.119:0/2942956452 <== mds.0 v2:172.21.15.119:6834/3556289056 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1821460 con 0x1a7e140 2024-09-13T16:25:23.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32c8ff9700 10 client.4398 handle_command_reply: tid=0 2024-09-13T16:25:23.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 client.4398 shutdown 2024-09-13T16:25:23.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82b50 msgr2=0x1a3c880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82b50 0x1a3c880 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f32e000bc50 tx=0x7f32e0058a30 comp rx=0 tx=0).stop 2024-09-13T16:25:23.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 shutdown_connections 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a7e140 0x1a7e520 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a82b50 0x1a3c880 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a3a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 --2- 172.21.15.119:0/2942956452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a60e90 0x1a5cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 >> 172.21.15.119:0/2942956452 conn(0x17ca4d0 msgr2=0x1a5d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32c8ff9700 0 client.4398 ms_handle_reset on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 shutdown_connections 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 1 -- 172.21.15.119:0/2942956452 wait complete. 2024-09-13T16:25:23.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.586+0000 7f32f327e740 20 client.4398 trim_cache size 0 max 16384 2024-09-13T16:25:23.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 Processor -- start 2024-09-13T16:25:23.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 -- start start 2024-09-13T16:25:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1a8aed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a81160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x195f700 con 0x1a60e90 2024-09-13T16:25:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19c8f50 con 0x1a82b50 2024-09-13T16:25:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a32320 con 0x1a82230 2024-09-13T16:25:23.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1a8aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1a8aed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53686/0 (socket says 172.21.15.119:53686) 2024-09-13T16:25:23.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 -- 172.21.15.119:0/111986919 learned_addr learned my addr 172.21.15.119:0/111986919 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:23.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32ed568700 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a81160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32ee56a700 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 -- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 msgr2=0x1a81160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a81160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 -- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 msgr2=0x1a816a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 -- 172.21.15.119:0/111986919 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x194f450 con 0x1a60e90 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32edd69700 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1a8aed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32d8022040 tx=0x7f32d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c2149198a320da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32cbfff700 1 -- 172.21.15.119:0/111986919 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8034d70 con 0x1a60e90 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32cbfff700 1 -- 172.21.15.119:0/111986919 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d800f810 con 0x1a60e90 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.610+0000 7f32cbfff700 1 -- 172.21.15.119:0/111986919 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8015cf0 con 0x1a60e90 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- 172.21.15.119:0/111986919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 msgr2=0x1a8aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1a8aed0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f32d8022040 tx=0x7f32d8014000 comp rx=0 tx=0).stop 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- 172.21.15.119:0/111986919 shutdown_connections 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a81160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 --2- 172.21.15.119:0/111986919 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1a8aed0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- 172.21.15.119:0/111986919 >> 172.21.15.119:0/111986919 conn(0x17ca4d0 msgr2=0x1a8a5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- 172.21.15.119:0/111986919 shutdown_connections 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- 172.21.15.119:0/111986919 wait complete. 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- start start 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 Processor -- start 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1ae5760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a811a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a86900 con 0x1a60e90 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a87a70 con 0x1a82b50 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a3a3f0 con 0x1a82230 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32f327e740 10 client.0 _ll_register_callbacks cb 0x1954ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32edd69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1ae5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32edd69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1ae5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53688/0 (socket says 172.21.15.119:53688) 2024-09-13T16:25:23.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32edd69700 1 -- 172.21.15.119:0/3224260833 learned_addr learned my addr 172.21.15.119:0/3224260833 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32ed568700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a811a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.614+0000 7f32ee56a700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32edd69700 1 -- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 msgr2=0x1a811a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32edd69700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a811a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32edd69700 1 -- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 msgr2=0x1a816e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32edd69700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32edd69700 1 -- 172.21.15.119:0/3224260833 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1815960 con 0x1a60e90 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32ee56a700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32edd69700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1ae5760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32d800fcf0 tx=0x7f32d80047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16b53470ca37549c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 1 -- 172.21.15.119:0/3224260833 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32d8005a40 con 0x1a60e90 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 1 -- 172.21.15.119:0/3224260833 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f32d8032070 con 0x1a60e90 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f32d800c040 con 0x1a60e90 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 1 -- 172.21.15.119:0/3224260833 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=7) v2 ==== 24+0+0 (secure 0 0 0) 0x7f32d8005ba0 con 0x1a60e90 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 10 client.4404 fetch_fsmap learned FSMap version 7 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=7,monmap=2+}) v3 -- 0x1943720 con 0x1a60e90 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 1 -- 172.21.15.119:0/3224260833 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d800c040 con 0x1a60e90 2024-09-13T16:25:23.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 1 -- 172.21.15.119:0/3224260833 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 7) v1 ==== 3122+0+0 (secure 0 0 0) 0x7f32d8005d00 con 0x1a60e90 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 10 client.4404 fetch_fsmap finished waiting for FSMap version 7 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 10 client.4404 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 20 client.4404 populate_metadata read hostname 'smithi119' 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a84f40 0x1a85320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 4 client.4404 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 --> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] -- command(tid 0: {"prefix": "status"}) v1 -- 0x18461e0 con 0x1a84f40 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32ed568700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a84f40 0x1a85320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 10 client.4404 ms_handle_connect on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32ed568700 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a84f40 0x1a85320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b71217032d047c3 server_cookie=e59b6015611842b9 in_seq=0 out_seq=0 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 1 -- 172.21.15.119:0/3224260833 <== mds.0 v2:172.21.15.119:6834/3556289056 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+317 (crc 0 0 0) 0x18461e0 con 0x1a84f40 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 10 client.4404 handle_command_reply: tid=0 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 client.4404 shutdown 2024-09-13T16:25:23.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 msgr2=0x1ae5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1ae5760 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f32d800fcf0 tx=0x7f32d80047e0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 shutdown_connections 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1a84f40 0x1a85320 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a82b50 0x1a816e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a82230 0x1a811a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 --2- 172.21.15.119:0/3224260833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a60e90 0x1ae5760 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 >> 172.21.15.119:0/3224260833 conn(0x17ca4d0 msgr2=0x1a5c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32c8ff9700 0 client.4404 ms_handle_reset on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 shutdown_connections 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.618+0000 7f32f327e740 1 -- 172.21.15.119:0/3224260833 wait complete. 2024-09-13T16:25:23.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32f327e740 20 client.4404 trim_cache size 0 max 16384 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 4285, 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 6.710217053, 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 7, 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 25, 2024-09-13T16:25:23.628 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 24, 2024-09-13T16:25:23.629 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 12.992238896 2024-09-13T16:25:23.629 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:25:23.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32cc066730 msgr2=0x7f32cc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32cc066730 0x7f32cc068bf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006fd0 tx=0x7f32d8008040 comp rx=0 tx=0).stop 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e81594f0 msgr2=0x7f32e815b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e81594f0 0x7f32e815b100 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f32e80ce710 tx=0x7f32e0004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 shutdown_connections 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32cc066730 0x7f32cc068bf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32e81594f0 0x7f32e815b100 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32e80b3800 0x7f32e815abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 --2- 172.21.15.119:0/1779939982 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32e806da70 0x7f32e815c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 >> 172.21.15.119:0/1779939982 conn(0x7f32e806a350 msgr2=0x7f32e80cc9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 shutdown_connections 2024-09-13T16:25:23.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.622+0000 7f32eed6b700 1 -- 172.21.15.119:0/1779939982 wait complete. 2024-09-13T16:25:23.640 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T16:25:23.640 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:23.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:25:23.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9e7a700 1 Processor -- start 2024-09-13T16:25:23.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9e7a700 1 -- start start 2024-09-13T16:25:23.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9e7a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9e7a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f90a40b2200 0x7f90a40b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9e7a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a40705d0 con 0x7f90a40b17f0 2024-09-13T16:25:23.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9e7a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a40cbf10 con 0x7f90a40b2200 2024-09-13T16:25:23.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9e7a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a40b2bd0 con 0x7f90a40cd8d0 2024-09-13T16:25:23.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f90a40b17f0 0x7f90a40cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41906/0 (socket says 172.21.15.119:41906) 2024-09-13T16:25:23.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a3fff700 1 -- 172.21.15.119:0/463093070 learned_addr learned my addr 172.21.15.119:0/463093070 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:23.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a9679700 1 --2- 172.21.15.119:0/463093070 >> v2:172.21.15.145:3300/0 conn(0x7f90a40b2200 0x7f90a40b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a8e78700 1 --2- 172.21.15.119:0/463093070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a37fe700 1 -- 172.21.15.119:0/463093070 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545485971 0 0) 0x7f90a40705d0 con 0x7f90a40b17f0 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a37fe700 1 -- 172.21.15.119:0/463093070 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f908c0035d0 con 0x7f90a40b17f0 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a8e78700 1 -- 172.21.15.119:0/463093070 >> v2:172.21.15.145:3300/0 conn(0x7f90a40b2200 msgr2=0x7f90a40b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a8e78700 1 --2- 172.21.15.119:0/463093070 >> v2:172.21.15.145:3300/0 conn(0x7f90a40b2200 0x7f90a40b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a8e78700 1 -- 172.21.15.119:0/463093070 >> v1:172.21.15.119:6789/0 conn(0x7f90a40b17f0 legacy=0x7f90a40cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a8e78700 1 -- 172.21.15.119:0/463093070 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a406d0d0 con 0x7f90a40cd8d0 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a8e78700 1 --2- 172.21.15.119:0/463093070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a406cb90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9098009850 tx=0x7f9098009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e06e9ac77939449 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a37fe700 1 -- 172.21.15.119:0/463093070 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098011830 con 0x7f90a40cd8d0 2024-09-13T16:25:23.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a37fe700 1 -- 172.21.15.119:0/463093070 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9098011990 con 0x7f90a40cd8d0 2024-09-13T16:25:23.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.754+0000 7f90a37fe700 1 -- 172.21.15.119:0/463093070 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9098011d40 con 0x7f90a40cd8d0 2024-09-13T16:25:23.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- 172.21.15.119:0/463093070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 msgr2=0x7f90a406cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/463093070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a406cb90 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9098009850 tx=0x7f9098009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:23.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- 172.21.15.119:0/463093070 shutdown_connections 2024-09-13T16:25:23.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/463093070 >> v2:172.21.15.145:3300/0 conn(0x7f90a40b2200 0x7f90a40b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/463093070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a406cb90 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- 172.21.15.119:0/463093070 >> 172.21.15.119:0/463093070 conn(0x7f90a406a350 msgr2=0x7f90a406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- 172.21.15.119:0/463093070 shutdown_connections 2024-09-13T16:25:23.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- 172.21.15.119:0/463093070 wait complete. 2024-09-13T16:25:23.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 Processor -- start 2024-09-13T16:25:23.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- start start 2024-09-13T16:25:23.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a40b17f0 0x7f90a40cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a40b2200 0x7f90a41ec2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a41ec830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a414eef0 con 0x7f90a40b2200 2024-09-13T16:25:23.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a40cb1b0 con 0x7f90a40b17f0 2024-09-13T16:25:23.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.758+0000 7f90a9e7a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a414e270 con 0x7f90a40cd8d0 2024-09-13T16:25:23.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a40b2200 0x7f90a41ec2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a40b2200 0x7f90a41ec2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53704/0 (socket says 172.21.15.119:53704) 2024-09-13T16:25:23.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a8e78700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a40b17f0 0x7f90a40cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a9679700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a41ec830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 -- 172.21.15.119:0/2210957353 learned_addr learned my addr 172.21.15.119:0/2210957353 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:23.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 -- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 msgr2=0x7f90a41ec830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a41ec830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 -- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a40b17f0 msgr2=0x7f90a40cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a40b17f0 0x7f90a40cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 -- 172.21.15.119:0/2210957353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980094b0 con 0x7f90a40b2200 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a3fff700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a40b2200 0x7f90a41ec2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f909400cbc0 tx=0x7f909400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a17fa700 1 -- 172.21.15.119:0/2210957353 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909400a770 con 0x7f90a40b2200 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a17fa700 1 -- 172.21.15.119:0/2210957353 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f909400a8d0 con 0x7f90a40b2200 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a414e550 con 0x7f90a40b2200 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a17fa700 1 -- 172.21.15.119:0/2210957353 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f909400ac40 con 0x7f90a40b2200 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a414eb10 con 0x7f90a40b2200 2024-09-13T16:25:23.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.762+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a414f030 con 0x7f90a40b2200 2024-09-13T16:25:23.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.766+0000 7f90a17fa700 1 -- 172.21.15.119:0/2210957353 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9094012400 con 0x7f90a40b2200 2024-09-13T16:25:23.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.766+0000 7f90a17fa700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9090066760 0x7f9090068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:23.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.766+0000 7f90a17fa700 1 -- 172.21.15.119:0/2210957353 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9094083d30 con 0x7f90a40b2200 2024-09-13T16:25:23.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.766+0000 7f90a8e78700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9090066760 0x7f9090068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:23.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.766+0000 7f90a8e78700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9090066760 0x7f9090068c20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9098000c00 tx=0x7f909801c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:23.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.770+0000 7f90a17fa700 1 -- 172.21.15.119:0/2210957353 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f90a414f030 con 0x7f90a40b2200 2024-09-13T16:25:23.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f90a414f030 con 0x7f90a40b2200 2024-09-13T16:25:23.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a17fa700 1 -- 172.21.15.119:0/2210957353 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v7) v1 ==== 53+0+88 (secure 0 0 0) 0x7f90a414f030 con 0x7f90a40b2200 2024-09-13T16:25:23.980 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9090066760 msgr2=0x7f9090068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9090066760 0x7f9090068c20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9098000c00 tx=0x7f909801c040 comp rx=0 tx=0).stop 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a40b2200 msgr2=0x7f90a41ec2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a40b2200 0x7f90a41ec2f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f909400cbc0 tx=0x7f909400cbf0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 shutdown_connections 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9090066760 0x7f9090068c20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90a40cd8d0 0x7f90a41ec830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f90a40b2200 0x7f90a41ec2f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 --2- 172.21.15.119:0/2210957353 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90a40b17f0 0x7f90a40cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:23.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.978+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 >> 172.21.15.119:0/2210957353 conn(0x7f90a406a350 msgr2=0x7f90a4163470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:23.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.982+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 shutdown_connections 2024-09-13T16:25:23.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:23.982+0000 7f90a9e7a700 1 -- 172.21.15.119:0/2210957353 wait complete. 2024-09-13T16:25:23.995 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-09-13T16:25:23.995 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:25:23.995 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:25:23.995 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-09-13T16:25:23.995 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:23.995 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-09-13T16:25:23.995 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:25:23.996 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:25:23.996 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-09-13T16:25:23.996 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:23.996 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:24.001 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42617sec preferred_lft 42617sec 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:24.002 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:25:24.003 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:24.003 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T16:25:24.003 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.003 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:24.003 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:24.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link add name ceph-brx type bridge 2024-09-13T16:25:24.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip addr flush dev ceph-brx 2024-09-13T16:25:24.004 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set ceph-brx up 2024-09-13T16:25:24.004 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T16:25:24.004 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:24.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.047 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:25:24.106 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:25:24.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.108 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:25:24.139 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:25:24.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.140 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:24.140 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:24.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:25:24.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:25:24.140 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:25:24.140 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:24.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.182 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:25:24.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.230 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list-id 2024-09-13T16:25:24.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.238 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:24.238 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:24.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-13T16:25:24.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-09-13T16:25:24.238 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:24.266 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-09-13T16:25:24.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.266 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:24.266 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:24.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-09-13T16:25:24.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T16:25:24.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-09-13T16:25:24.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-09-13T16:25:24.267 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-09-13T16:25:24.267 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:24.377 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.378 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:24.378 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:24.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:25:24.378 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T16:25:24.378 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:24.403 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-09-13T16:25:24.403 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:24.403 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:24.410 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-09-13T16:25:24.410 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:24.410 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:24.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:24.474 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:24.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:24.490 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:24.491 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:24.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.492 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:24.499 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:25:24.499 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-09-13T16:25:24.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:24.561 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:24.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:24.570 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:24.571 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:24.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:24.571 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:24.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi119.stderr:2024-09-13T16:25:24.630+0000 7fb3d5ffa640 -1 init, newargv = 0x556a91d0d380 newargc=15 2024-09-13T16:25:24.633 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi119.stderr:ceph-fuse[32780]: starting ceph client 2024-09-13T16:25:24.646 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi119.stderr:ceph-fuse[32780]: starting fuse 2024-09-13T16:25:25.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:25.696 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:25.696 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:25.754 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:25.755 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:25.755 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:25.756 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:25.762 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:25:25.762 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:25:25.763 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.admin")) 2024-09-13T16:25:25.764 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:25:25.847 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.admin.32780.asok 2024-09-13T16:25:26.030 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.32780.asok 2024-09-13T16:25:26.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:26.030 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.32780.asok status 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "admin", 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "32780", 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:25:26.134 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 4419, 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 4419 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3616794247 2024-09-13T16:25:26.135 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3616794247 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.4419 192.168.0.1:0/3616794247", 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/3616794247", 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 7, 2024-09-13T16:25:26.136 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 25, 2024-09-13T16:25:26.137 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:25:26.137 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:25:26.137 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:25:26.137 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:25:26.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:26.144 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:25:26.144 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:26.152 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:25:26.152 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:26.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:26.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:26.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:26.168 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-09-13T16:25:26.214 INFO:teuthology.orchestra.run.smithi119.stdout:# file: . 2024-09-13T16:25:26.214 INFO:teuthology.orchestra.run.smithi119.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-09-13T16:25:26.214 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:26.215 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:25:26.215 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:25:26.223 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:26.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff80942d700 1 Processor -- start 2024-09-13T16:25:26.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff80942d700 1 -- start start 2024-09-13T16:25:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff80942d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff8040ce040 0x7ff804165450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff80942d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80406cf60 con 0x7ff8040adb30 2024-09-13T16:25:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff80942d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8040cc4d0 con 0x7ff8040cc060 2024-09-13T16:25:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff80942d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8040b3390 con 0x7ff8040ce040 2024-09-13T16:25:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff808c2c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff8040ce040 0x7ff804165450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff808c2c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff8040ce040 0x7ff804165450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53718/0 (socket says 172.21.15.119:53718) 2024-09-13T16:25:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff808c2c700 1 -- 172.21.15.119:0/282662988 learned_addr learned my addr 172.21.15.119:0/282662988 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff808c2c700 1 -- 172.21.15.119:0/282662988 >> v1:172.21.15.145:6789/0 conn(0x7ff8040cc060 legacy=0x7ff8040cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff808c2c700 1 -- 172.21.15.119:0/282662988 >> v1:172.21.15.119:6789/0 conn(0x7ff8040adb30 legacy=0x7ff80406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff808c2c700 1 -- 172.21.15.119:0/282662988 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80406c350 con 0x7ff8040ce040 2024-09-13T16:25:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff808c2c700 1 --2- 172.21.15.119:0/282662988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040ce040 0x7ff804165450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009910 tx=0x7ff7fc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e8a16ce1d97bb2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff802ffd700 1 -- 172.21.15.119:0/282662988 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc004230 con 0x7ff8040ce040 2024-09-13T16:25:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.338+0000 7ff802ffd700 1 -- 172.21.15.119:0/282662988 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc004390 con 0x7ff8040ce040 2024-09-13T16:25:26.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff802ffd700 1 -- 172.21.15.119:0/282662988 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7fc009e60 con 0x7ff8040ce040 2024-09-13T16:25:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff80942d700 1 -- 172.21.15.119:0/282662988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040ce040 msgr2=0x7ff804165450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff80942d700 1 --2- 172.21.15.119:0/282662988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040ce040 0x7ff804165450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc009910 tx=0x7ff7fc009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff80942d700 1 -- 172.21.15.119:0/282662988 shutdown_connections 2024-09-13T16:25:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff80942d700 1 --2- 172.21.15.119:0/282662988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040ce040 0x7ff804165450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff80942d700 1 -- 172.21.15.119:0/282662988 >> 172.21.15.119:0/282662988 conn(0x7ff80406a350 msgr2=0x7ff80406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff80942d700 1 -- 172.21.15.119:0/282662988 shutdown_connections 2024-09-13T16:25:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.342+0000 7ff80942d700 1 -- 172.21.15.119:0/282662988 wait complete. 2024-09-13T16:25:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 Processor -- start 2024-09-13T16:25:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 -- start start 2024-09-13T16:25:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040adb30 0x7ff804141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8040cc060 0x7ff8040a8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8040ce040 0x7ff80413ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8040aaba0 con 0x7ff8040adb30 2024-09-13T16:25:26.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8041e6b20 con 0x7ff8040cc060 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff80942d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8041e5fa0 con 0x7ff8040ce040 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040adb30 0x7ff804141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040adb30 0x7ff804141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53726/0 (socket says 172.21.15.119:53726) 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 -- 172.21.15.119:0/3359303498 learned_addr learned my addr 172.21.15.119:0/3359303498 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff8037fe700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8040cc060 0x7ff8040a8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff808c2c700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8040ce040 0x7ff80413ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 -- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8040ce040 msgr2=0x7ff80413ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8040ce040 0x7ff80413ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 -- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8040cc060 msgr2=0x7ff8040a8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8040cc060 0x7ff8040a8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff803fff700 1 -- 172.21.15.119:0/3359303498 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7fc009570 con 0x7ff8040adb30 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff8037fe700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8040cc060 0x7ff8040a8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.346+0000 7ff808c2c700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8040ce040 0x7ff80413ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff803fff700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040adb30 0x7ff804141000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800c500 tx=0x7ff7f800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff800ff9700 1 -- 172.21.15.119:0/3359303498 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80052d0 con 0x7ff8040adb30 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff800ff9700 1 -- 172.21.15.119:0/3359303498 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f8005430 con 0x7ff8040adb30 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff800ff9700 1 -- 172.21.15.119:0/3359303498 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f80055a0 con 0x7ff8040adb30 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8041e6280 con 0x7ff8040adb30 2024-09-13T16:25:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8041e6840 con 0x7ff8040adb30 2024-09-13T16:25:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8040adf10 con 0x7ff8040adb30 2024-09-13T16:25:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff800ff9700 1 -- 172.21.15.119:0/3359303498 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff7f80036a0 con 0x7ff8040adb30 2024-09-13T16:25:26.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.350+0000 7ff800ff9700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff7f0066730 0x7ff7f0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.354+0000 7ff800ff9700 1 -- 172.21.15.119:0/3359303498 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff7f80836f0 con 0x7ff8040adb30 2024-09-13T16:25:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.354+0000 7ff8037fe700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff7f0066730 0x7ff7f0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.354+0000 7ff8037fe700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff7f0066730 0x7ff7f0068bf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff80415a810 tx=0x7ff7f4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:26.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.358+0000 7ff800ff9700 1 -- 172.21.15.119:0/3359303498 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff7f8050800 con 0x7ff8040adb30 2024-09-13T16:25:26.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.534+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff80415e600 con 0x7ff8040adb30 2024-09-13T16:25:26.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff800ff9700 1 -- 172.21.15.119:0/3359303498 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7ff7f8053e20 con 0x7ff8040adb30 2024-09-13T16:25:26.540 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:26.540 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff7f0066730 msgr2=0x7ff7f0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff7f0066730 0x7ff7f0068bf0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff80415a810 tx=0x7ff7f4009040 comp rx=0 tx=0).stop 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040adb30 msgr2=0x7ff804141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040adb30 0x7ff804141000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800c500 tx=0x7ff7f800c8c0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 shutdown_connections 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff7f0066730 0x7ff7f0068bf0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8040ce040 0x7ff80413ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8040cc060 0x7ff8040a8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 --2- 172.21.15.119:0/3359303498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8040adb30 0x7ff804141000 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 >> 172.21.15.119:0/3359303498 conn(0x7ff80406a350 msgr2=0x7ff8040b3500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 shutdown_connections 2024-09-13T16:25:26.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.538+0000 7ff80942d700 1 -- 172.21.15.119:0/3359303498 wait complete. 2024-09-13T16:25:26.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:25:26.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:26.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8e998700 1 Processor -- start 2024-09-13T16:25:26.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8e998700 1 -- start start 2024-09-13T16:25:26.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8e998700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8815cfb0 0x7f8a8806ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8e998700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8a8806be20 0x7f8a880d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8e998700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a880ce880 con 0x7f8a8806b2d0 2024-09-13T16:25:26.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8e998700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a880cee40 con 0x7f8a8806be20 2024-09-13T16:25:26.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8e998700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a880d1220 con 0x7f8a8815cfb0 2024-09-13T16:25:26.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a877fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8a8806b2d0 0x7f8a8806b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41918/0 (socket says 172.21.15.119:41918) 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8cf35700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8a8806be20 0x7f8a880d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a877fe700 1 -- 172.21.15.119:0/2873036164 learned_addr learned my addr 172.21.15.119:0/2873036164 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a87fff700 1 --2- 172.21.15.119:0/2873036164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8815cfb0 0x7f8a8806ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8cf35700 1 -- 172.21.15.119:0/2873036164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8815cfb0 msgr2=0x7f8a8806ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8cf35700 1 --2- 172.21.15.119:0/2873036164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8815cfb0 0x7f8a8806ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8cf35700 1 -- 172.21.15.119:0/2873036164 >> v1:172.21.15.119:6789/0 conn(0x7f8a8806b2d0 legacy=0x7f8a8806b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8cf35700 1 -- 172.21.15.119:0/2873036164 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a880ceb60 con 0x7f8a8806be20 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a86ffd700 1 -- 172.21.15.119:0/2873036164 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894858935 0 0) 0x7f8a880ce880 con 0x7f8a8806b2d0 2024-09-13T16:25:26.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.674+0000 7f8a8cf35700 1 --2- 172.21.15.119:0/2873036164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8806be20 0x7f8a880d1640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8a80009d90 tx=0x7f8a8000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378f72eb774676dd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a86ffd700 1 -- 172.21.15.119:0/2873036164 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a8000e070 con 0x7f8a8806be20 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a86ffd700 1 -- 172.21.15.119:0/2873036164 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a8000bb20 con 0x7f8a8806be20 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a86ffd700 1 -- 172.21.15.119:0/2873036164 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80012430 con 0x7f8a8806be20 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 -- 172.21.15.119:0/2873036164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8806be20 msgr2=0x7f8a880d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 --2- 172.21.15.119:0/2873036164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8806be20 0x7f8a880d1640 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8a80009d90 tx=0x7f8a8000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 -- 172.21.15.119:0/2873036164 shutdown_connections 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 --2- 172.21.15.119:0/2873036164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8806be20 0x7f8a880d1640 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 --2- 172.21.15.119:0/2873036164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8815cfb0 0x7f8a8806ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 -- 172.21.15.119:0/2873036164 >> 172.21.15.119:0/2873036164 conn(0x7f8a880d2da0 msgr2=0x7f8a880d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:26.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 -- 172.21.15.119:0/2873036164 shutdown_connections 2024-09-13T16:25:26.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 -- 172.21.15.119:0/2873036164 wait complete. 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 Processor -- start 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.678+0000 7f8a8e998700 1 -- start start 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a8806b2d0 0x7f8a8814e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8806be20 0x7f8a8815b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8815cfb0 0x7f8a881564f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8815a970 con 0x7f8a8815cfb0 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a88159da0 con 0x7f8a8806b2d0 2024-09-13T16:25:26.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8815a040 con 0x7f8a8806be20 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8815cfb0 0x7f8a881564f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8815cfb0 0x7f8a881564f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53742/0 (socket says 172.21.15.119:53742) 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 -- 172.21.15.119:0/188903553 learned_addr learned my addr 172.21.15.119:0/188903553 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a87fff700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a8806b2d0 0x7f8a8814e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a877fe700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8806be20 0x7f8a8815b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 -- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8806be20 msgr2=0x7f8a8815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8806be20 0x7f8a8815b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 -- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a8806b2d0 msgr2=0x7f8a8814e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a8806b2d0 0x7f8a8814e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 -- 172.21.15.119:0/188903553 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a80009a70 con 0x7f8a8815cfb0 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8cf35700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8815cfb0 0x7f8a881564f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8a80014040 tx=0x7f8a80004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a84ff9700 1 -- 172.21.15.119:0/188903553 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80012430 con 0x7f8a8815cfb0 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a84ff9700 1 -- 172.21.15.119:0/188903553 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a80020070 con 0x7f8a8815cfb0 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8815a320 con 0x7f8a8815cfb0 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a84ff9700 1 -- 172.21.15.119:0/188903553 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a80004280 con 0x7f8a8815cfb0 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.682+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a880b6470 con 0x7f8a8815cfb0 2024-09-13T16:25:26.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.686+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8815aae0 con 0x7f8a8815cfb0 2024-09-13T16:25:26.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.686+0000 7f8a84ff9700 1 -- 172.21.15.119:0/188903553 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8a800043e0 con 0x7f8a8815cfb0 2024-09-13T16:25:26.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.686+0000 7f8a84ff9700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8a74066760 0x7f8a74068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:26.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.686+0000 7f8a84ff9700 1 -- 172.21.15.119:0/188903553 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8a80022070 con 0x7f8a8815cfb0 2024-09-13T16:25:26.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.686+0000 7f8a87fff700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8a74066760 0x7f8a74068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:26.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.686+0000 7f8a87fff700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8a74066760 0x7f8a74068c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009b50 tx=0x7f8a78009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:26.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.694+0000 7f8a84ff9700 1 -- 172.21.15.119:0/188903553 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8a80053510 con 0x7f8a8815cfb0 2024-09-13T16:25:26.763 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:25:26.765 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:25:26.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.862+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a8806ad90 con 0x7f8a8815cfb0 2024-09-13T16:25:26.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a84ff9700 1 -- 172.21.15.119:0/188903553 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a80056b30 con 0x7f8a8815cfb0 2024-09-13T16:25:26.867 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:26.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8a74066760 msgr2=0x7f8a74068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8a74066760 0x7f8a74068c20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8a78009b50 tx=0x7f8a78009240 comp rx=0 tx=0).stop 2024-09-13T16:25:26.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8815cfb0 msgr2=0x7f8a881564f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:26.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8815cfb0 0x7f8a881564f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8a80014040 tx=0x7f8a80004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 shutdown_connections 2024-09-13T16:25:26.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8a74066760 0x7f8a74068c20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a8815cfb0 0x7f8a881564f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a8806be20 0x7f8a8815b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 --2- 172.21.15.119:0/188903553 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a8806b2d0 0x7f8a8814e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:26.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 >> 172.21.15.119:0/188903553 conn(0x7f8a880d2da0 msgr2=0x7f8a880d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:26.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 shutdown_connections 2024-09-13T16:25:26.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.866+0000 7f8a8e998700 1 -- 172.21.15.119:0/188903553 wait complete. 2024-09-13T16:25:26.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:25:26.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98b1d06700 1 Processor -- start 2024-09-13T16:25:26.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98b1d06700 1 -- start start 2024-09-13T16:25:26.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98b1d06700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98ac0b3450 0x7f98ac070630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98b1d06700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f98ac070b70 0x7f98ac15b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98b1d06700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98ac0ae340 con 0x7f98ac0adf60 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98b1d06700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac15ae40 con 0x7f98ac0b3450 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98b1d06700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98ac15a3b0 con 0x7f98ac070b70 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98ab7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f98ac0adf60 0x7f98ac159ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41928/0 (socket says 172.21.15.119:41928) 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98ab7fe700 1 -- 172.21.15.119:0/322281873 learned_addr learned my addr 172.21.15.119:0/322281873 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98aaffd700 1 --2- 172.21.15.119:0/322281873 >> v2:172.21.15.119:3300/0 conn(0x7f98ac0b3450 0x7f98ac070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98abfff700 1 --2- 172.21.15.119:0/322281873 >> v2:172.21.15.145:3300/0 conn(0x7f98ac070b70 0x7f98ac15b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.994+0000 7f98aa7fc700 1 -- 172.21.15.119:0/322281873 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575193624 0 0) 0x7f98ac0ae340 con 0x7f98ac0adf60 2024-09-13T16:25:27.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aa7fc700 1 -- 172.21.15.119:0/322281873 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98980035d0 con 0x7f98ac0adf60 2024-09-13T16:25:27.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aaffd700 1 -- 172.21.15.119:0/322281873 >> v2:172.21.15.145:3300/0 conn(0x7f98ac070b70 msgr2=0x7f98ac15b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:27.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aaffd700 1 --2- 172.21.15.119:0/322281873 >> v2:172.21.15.145:3300/0 conn(0x7f98ac070b70 0x7f98ac15b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aaffd700 1 -- 172.21.15.119:0/322281873 >> v1:172.21.15.119:6789/0 conn(0x7f98ac0adf60 legacy=0x7f98ac159ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aaffd700 1 -- 172.21.15.119:0/322281873 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac15b1d0 con 0x7f98ac0b3450 2024-09-13T16:25:27.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aaffd700 1 --2- 172.21.15.119:0/322281873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac0b3450 0x7f98ac070630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f98ac06c8b0 tx=0x7f9894009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43bd2e5363652912 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aa7fc700 1 -- 172.21.15.119:0/322281873 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894011830 con 0x7f98ac0b3450 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aa7fc700 1 -- 172.21.15.119:0/322281873 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9894011990 con 0x7f98ac0b3450 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98aa7fc700 1 -- 172.21.15.119:0/322281873 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9894011d20 con 0x7f98ac0b3450 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 -- 172.21.15.119:0/322281873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac0b3450 msgr2=0x7f98ac070630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 --2- 172.21.15.119:0/322281873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac0b3450 0x7f98ac070630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f98ac06c8b0 tx=0x7f9894009b30 comp rx=0 tx=0).stop 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 -- 172.21.15.119:0/322281873 shutdown_connections 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 --2- 172.21.15.119:0/322281873 >> v2:172.21.15.145:3300/0 conn(0x7f98ac070b70 0x7f98ac15b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 --2- 172.21.15.119:0/322281873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac0b3450 0x7f98ac070630 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 -- 172.21.15.119:0/322281873 >> 172.21.15.119:0/322281873 conn(0x7f98ac0b0560 msgr2=0x7f98ac0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:27.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 -- 172.21.15.119:0/322281873 shutdown_connections 2024-09-13T16:25:27.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 -- 172.21.15.119:0/322281873 wait complete. 2024-09-13T16:25:27.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 Processor -- start 2024-09-13T16:25:27.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:26.998+0000 7f98b1d06700 1 -- start start 2024-09-13T16:25:27.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98ac070b70 0x7f98ac13fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98ac0adf60 0x7f98ac1502b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac1507f0 0x7f98ac143a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac14ece0 con 0x7f98ac1507f0 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac150c90 con 0x7f98ac070b70 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98ac14e340 con 0x7f98ac0adf60 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac1507f0 0x7f98ac143a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac1507f0 0x7f98ac143a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53766/0 (socket says 172.21.15.119:53766) 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 -- 172.21.15.119:0/1996581171 learned_addr learned my addr 172.21.15.119:0/1996581171 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98ab7fe700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98ac070b70 0x7f98ac13fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98aaffd700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98ac0adf60 0x7f98ac1502b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 -- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98ac0adf60 msgr2=0x7f98ac1502b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98ac0adf60 0x7f98ac1502b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 -- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98ac070b70 msgr2=0x7f98ac13fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98ac070b70 0x7f98ac13fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 -- 172.21.15.119:0/1996581171 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98940094b0 con 0x7f98ac1507f0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98abfff700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac1507f0 0x7f98ac143a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98a400efc0 tx=0x7f98a400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98a3fff700 1 -- 172.21.15.119:0/1996581171 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a4009ea0 con 0x7f98ac1507f0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98a3fff700 1 -- 172.21.15.119:0/1996581171 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a400f040 con 0x7f98ac1507f0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98ac14e620 con 0x7f98ac1507f0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98a3fff700 1 -- 172.21.15.119:0/1996581171 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a400ced0 con 0x7f98ac1507f0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98ac1524f0 con 0x7f98ac1507f0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.002+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98ac1526e0 con 0x7f98ac1507f0 2024-09-13T16:25:27.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.006+0000 7f98a3fff700 1 -- 172.21.15.119:0/1996581171 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f98a4014480 con 0x7f98ac1507f0 2024-09-13T16:25:27.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.006+0000 7f98a3fff700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f988c066730 0x7f988c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.006+0000 7f98a3fff700 1 -- 172.21.15.119:0/1996581171 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f98a4084c40 con 0x7f98ac1507f0 2024-09-13T16:25:27.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.006+0000 7f98ab7fe700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f988c066730 0x7f988c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.006+0000 7f98ab7fe700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f988c066730 0x7f988c068bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f989c007fd0 tx=0x7f989c007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.010+0000 7f98a3fff700 1 -- 172.21.15.119:0/1996581171 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f98ac1526e0 con 0x7f98ac1507f0 2024-09-13T16:25:27.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.182+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98ac14ee20 con 0x7f98ac1507f0 2024-09-13T16:25:27.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.182+0000 7f98a3fff700 1 -- 172.21.15.119:0/1996581171 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f98ac14ee20 con 0x7f98ac1507f0 2024-09-13T16:25:27.186 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:27.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f988c066730 msgr2=0x7f988c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f988c066730 0x7f988c068bf0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f989c007fd0 tx=0x7f989c007de0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac1507f0 msgr2=0x7f98ac143a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac1507f0 0x7f98ac143a30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f98a400efc0 tx=0x7f98a400c3a0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 shutdown_connections 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f988c066730 0x7f988c068bf0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98ac1507f0 0x7f98ac143a30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98ac0adf60 0x7f98ac1502b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 --2- 172.21.15.119:0/1996581171 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98ac070b70 0x7f98ac13fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 >> 172.21.15.119:0/1996581171 conn(0x7f98ac0b0560 msgr2=0x7f98ac0af7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 shutdown_connections 2024-09-13T16:25:27.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.186+0000 7f98b1d06700 1 -- 172.21.15.119:0/1996581171 wait complete. 2024-09-13T16:25:27.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:25:27.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.314+0000 7f2f360f2700 1 Processor -- start 2024-09-13T16:25:27.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.314+0000 7f2f360f2700 1 -- start start 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2f3006b710 0x7f2f3006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2f300d1640 0x7f2f300d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f3006c0a0 con 0x7f2f3015cfb0 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f300cee40 con 0x7f2f300d1640 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f300d1220 con 0x7f2f3006b710 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2f300d1640 0x7f2f300d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2f3015cfb0 0x7f2f3006ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41936/0 (socket says 172.21.15.119:41936) 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2f300d1640 0x7f2f300d1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53768/0 (socket says 172.21.15.119:53768) 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2f7fe700 1 -- 172.21.15.119:0/138890821 learned_addr learned my addr 172.21.15.119:0/138890821 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2effd700 1 --2- 172.21.15.119:0/138890821 >> v2:172.21.15.145:3300/0 conn(0x7f2f3006b710 0x7f2f3006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2ffff700 1 -- 172.21.15.119:0/138890821 >> v2:172.21.15.145:3300/0 conn(0x7f2f3006b710 msgr2=0x7f2f3006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:27.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2ffff700 1 --2- 172.21.15.119:0/138890821 >> v2:172.21.15.145:3300/0 conn(0x7f2f3006b710 0x7f2f3006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2ffff700 1 -- 172.21.15.119:0/138890821 >> v1:172.21.15.119:6789/0 conn(0x7f2f3015cfb0 legacy=0x7f2f3006ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2ffff700 1 -- 172.21.15.119:0/138890821 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f300ce880 con 0x7f2f300d1640 2024-09-13T16:25:27.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2ffff700 1 --2- 172.21.15.119:0/138890821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300d1640 0x7f2f300d1ad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f300c64b0 tx=0x7f2f2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06c99acd20b594a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2e7fc700 1 -- 172.21.15.119:0/138890821 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800e070 con 0x7f2f300d1640 2024-09-13T16:25:27.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2e7fc700 1 -- 172.21.15.119:0/138890821 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2800bbd0 con 0x7f2f300d1640 2024-09-13T16:25:27.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f2e7fc700 1 -- 172.21.15.119:0/138890821 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28012430 con 0x7f2f300d1640 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- 172.21.15.119:0/138890821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300d1640 msgr2=0x7f2f300d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 --2- 172.21.15.119:0/138890821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300d1640 0x7f2f300d1ad0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2f300c64b0 tx=0x7f2f2800b330 comp rx=0 tx=0).stop 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- 172.21.15.119:0/138890821 shutdown_connections 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 --2- 172.21.15.119:0/138890821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300d1640 0x7f2f300d1ad0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 --2- 172.21.15.119:0/138890821 >> v2:172.21.15.145:3300/0 conn(0x7f2f3006b710 0x7f2f3006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- 172.21.15.119:0/138890821 >> 172.21.15.119:0/138890821 conn(0x7f2f300d2da0 msgr2=0x7f2f300d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- 172.21.15.119:0/138890821 shutdown_connections 2024-09-13T16:25:27.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.318+0000 7f2f360f2700 1 -- 172.21.15.119:0/138890821 wait complete. 2024-09-13T16:25:27.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.322+0000 7f2f360f2700 1 Processor -- start 2024-09-13T16:25:27.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.322+0000 7f2f360f2700 1 -- start start 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f3006b710 0x7f2f300b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300d1640 0x7f2f300b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f3015cfb0 0x7f2f300ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300b80f0 con 0x7f2f3015cfb0 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300b83d0 con 0x7f2f3006b710 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f300b86b0 con 0x7f2f300d1640 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f3015cfb0 0x7f2f300ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f3015cfb0 0x7f2f300ca0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53784/0 (socket says 172.21.15.119:53784) 2024-09-13T16:25:27.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 -- 172.21.15.119:0/3760651439 learned_addr learned my addr 172.21.15.119:0/3760651439 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2f7fe700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f3006b710 0x7f2f300b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2effd700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300d1640 0x7f2f300b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 -- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300d1640 msgr2=0x7f2f300b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300d1640 0x7f2f300b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 -- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f3006b710 msgr2=0x7f2f300b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f3006b710 0x7f2f300b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 -- 172.21.15.119:0/3760651439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f28009a70 con 0x7f2f3015cfb0 2024-09-13T16:25:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f2ffff700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f3015cfb0 0x7f2f300ca0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f28014040 tx=0x7f2f28004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f27fff700 1 -- 172.21.15.119:0/3760651439 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28012430 con 0x7f2f3015cfb0 2024-09-13T16:25:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f27fff700 1 -- 172.21.15.119:0/3760651439 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f28020070 con 0x7f2f3015cfb0 2024-09-13T16:25:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f300c68f0 con 0x7f2f3015cfb0 2024-09-13T16:25:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f27fff700 1 -- 172.21.15.119:0/3760651439 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28004280 con 0x7f2f3015cfb0 2024-09-13T16:25:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f300b7e80 con 0x7f2f3015cfb0 2024-09-13T16:25:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f300c6ae0 con 0x7f2f3015cfb0 2024-09-13T16:25:27.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.326+0000 7f2f27fff700 1 -- 172.21.15.119:0/3760651439 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2f280043e0 con 0x7f2f3015cfb0 2024-09-13T16:25:27.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.330+0000 7f2f27fff700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f10066730 0x7f2f10068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.330+0000 7f2f27fff700 1 -- 172.21.15.119:0/3760651439 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2f28022070 con 0x7f2f3015cfb0 2024-09-13T16:25:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.330+0000 7f2f2f7fe700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f10066730 0x7f2f10068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.330+0000 7f2f2f7fe700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f10066730 0x7f2f10068bf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f18005930 tx=0x7f2f18009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.338+0000 7f2f27fff700 1 -- 172.21.15.119:0/3760651439 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2f28053670 con 0x7f2f3015cfb0 2024-09-13T16:25:27.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2f300ce490 con 0x7f2f3015cfb0 2024-09-13T16:25:27.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f27fff700 1 -- 172.21.15.119:0/3760651439 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2f28056c90 con 0x7f2f3015cfb0 2024-09-13T16:25:27.512 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:25:27.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f10066730 msgr2=0x7f2f10068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f10066730 0x7f2f10068bf0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f2f18005930 tx=0x7f2f18009040 comp rx=0 tx=0).stop 2024-09-13T16:25:27.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f3015cfb0 msgr2=0x7f2f300ca0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f3015cfb0 0x7f2f300ca0b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2f28014040 tx=0x7f2f28004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 shutdown_connections 2024-09-13T16:25:27.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f10066730 0x7f2f10068bf0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f3015cfb0 0x7f2f300ca0b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300d1640 0x7f2f300b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 --2- 172.21.15.119:0/3760651439 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f3006b710 0x7f2f300b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.506+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 >> 172.21.15.119:0/3760651439 conn(0x7f2f300d2da0 msgr2=0x7f2f300b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:27.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.510+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 shutdown_connections 2024-09-13T16:25:27.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.510+0000 7f2f360f2700 1 -- 172.21.15.119:0/3760651439 wait complete. 2024-09-13T16:25:27.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:25:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c759e700 1 Processor -- start 2024-09-13T16:25:27.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c759e700 1 -- start start 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45c815cc00 0x7f45c80d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c80b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45c80cdb00 con 0x7f45c80d0c20 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45c80cdde0 con 0x7f45c815cc00 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80ce0c0 con 0x7f45c80d17e0 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c659c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45c815cc00 0x7f45c80d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c5d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f45c80d0c20 0x7f45c80d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41948/0 (socket says 172.21.15.119:41948) 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c5d9b700 1 -- 172.21.15.119:0/3215518617 learned_addr learned my addr 172.21.15.119:0/3215518617 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:27.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c6d9d700 1 --2- 172.21.15.119:0/3215518617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c80b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c659c700 1 -- 172.21.15.119:0/3215518617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 msgr2=0x7f45c80b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:27.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c659c700 1 --2- 172.21.15.119:0/3215518617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c80b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c659c700 1 -- 172.21.15.119:0/3215518617 >> v1:172.21.15.119:6789/0 conn(0x7f45c80d0c20 legacy=0x7f45c80d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c659c700 1 -- 172.21.15.119:0/3215518617 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c80ce3a0 con 0x7f45c815cc00 2024-09-13T16:25:27.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c659c700 1 --2- 172.21.15.119:0/3215518617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80d06e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009850 tx=0x7f45b8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a2fb7191ca7b6d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.638+0000 7f45c559a700 1 -- 172.21.15.119:0/3215518617 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b8011830 con 0x7f45c815cc00 2024-09-13T16:25:27.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c559a700 1 -- 172.21.15.119:0/3215518617 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45b8011990 con 0x7f45c815cc00 2024-09-13T16:25:27.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c559a700 1 -- 172.21.15.119:0/3215518617 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45b8011d40 con 0x7f45c815cc00 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 -- 172.21.15.119:0/3215518617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 msgr2=0x7f45c80d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 --2- 172.21.15.119:0/3215518617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80d06e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45b8009850 tx=0x7f45b8009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 -- 172.21.15.119:0/3215518617 shutdown_connections 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 --2- 172.21.15.119:0/3215518617 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c80b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 --2- 172.21.15.119:0/3215518617 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80d06e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 -- 172.21.15.119:0/3215518617 >> 172.21.15.119:0/3215518617 conn(0x7f45c808c080 msgr2=0x7f45c80d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 -- 172.21.15.119:0/3215518617 shutdown_connections 2024-09-13T16:25:27.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 -- 172.21.15.119:0/3215518617 wait complete. 2024-09-13T16:25:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 Processor -- start 2024-09-13T16:25:27.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.642+0000 7f45c759e700 1 -- start start 2024-09-13T16:25:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45c80d0c20 0x7f45c815fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c8151120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80caf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80ccf80 con 0x7f45c815cc00 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80cbe10 con 0x7f45c80d0c20 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45c80cc0f0 con 0x7f45c80d17e0 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80caf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80caf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53806/0 (socket says 172.21.15.119:53806) 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 -- 172.21.15.119:0/2359690812 learned_addr learned my addr 172.21.15.119:0/2359690812 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c659c700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45c80d0c20 0x7f45c815fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c5d9b700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c8151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 -- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 msgr2=0x7f45c8151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c8151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 -- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45c80d0c20 msgr2=0x7f45c815fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45c80d0c20 0x7f45c815fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 -- 172.21.15.119:0/2359690812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45b80094b0 con 0x7f45c815cc00 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c5d9b700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c8151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c6d9d700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80caf00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f45c000ef90 tx=0x7f45c000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45b6ffd700 1 -- 172.21.15.119:0/2359690812 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0009ca0 con 0x7f45c815cc00 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45b6ffd700 1 -- 172.21.15.119:0/2359690812 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c000f040 con 0x7f45c815cc00 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c80c6360 con 0x7f45c815cc00 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45b6ffd700 1 -- 172.21.15.119:0/2359690812 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0014480 con 0x7f45c815cc00 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45c8151660 con 0x7f45c815cc00 2024-09-13T16:25:27.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45b6ffd700 1 -- 172.21.15.119:0/2359690812 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f45c00145e0 con 0x7f45c815cc00 2024-09-13T16:25:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.646+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45c80c61c0 con 0x7f45c815cc00 2024-09-13T16:25:27.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.650+0000 7f45b6ffd700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45a8066730 0x7f45a8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.650+0000 7f45b6ffd700 1 -- 172.21.15.119:0/2359690812 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f45c0084dc0 con 0x7f45c815cc00 2024-09-13T16:25:27.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.650+0000 7f45c659c700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45a8066730 0x7f45a8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.650+0000 7f45c659c700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45a8066730 0x7f45a8068bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f45b80090b0 tx=0x7f45b8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.658+0000 7f45b6ffd700 1 -- 172.21.15.119:0/2359690812 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f45c80c61c0 con 0x7f45c815cc00 2024-09-13T16:25:27.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.826+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f45c80c6880 con 0x7f45c815cc00 2024-09-13T16:25:27.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45b6ffd700 1 -- 172.21.15.119:0/2359690812 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f45c80c6880 con 0x7f45c815cc00 2024-09-13T16:25:27.832 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:27.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45a8066730 msgr2=0x7f45a8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45a8066730 0x7f45a8068bf0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f45b80090b0 tx=0x7f45b8008030 comp rx=0 tx=0).stop 2024-09-13T16:25:27.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 msgr2=0x7f45c80caf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80caf00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f45c000ef90 tx=0x7f45c000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 shutdown_connections 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45a8066730 0x7f45a8068bf0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45c815cc00 0x7f45c80caf00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45c80d17e0 0x7f45c8151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 --2- 172.21.15.119:0/2359690812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45c80d0c20 0x7f45c815fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 >> 172.21.15.119:0/2359690812 conn(0x7f45c808c080 msgr2=0x7f45c80cf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 shutdown_connections 2024-09-13T16:25:27.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.830+0000 7f45c759e700 1 -- 172.21.15.119:0/2359690812 wait complete. 2024-09-13T16:25:27.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:25:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ef59e700 1 Processor -- start 2024-09-13T16:25:27.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ef59e700 1 -- start start 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ef59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f26e800acc0 0x7f26e80e3040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ef59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f26e80507a0 0x7f26e80de2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ef59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26e8005d80 con 0x7f26e8033990 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ef59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e804f9e0 con 0x7f26e80507a0 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ef59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26e804fcc0 con 0x7f26e800acc0 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f26e80507a0 0x7f26e80de2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f26e80507a0 0x7f26e80de2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53810/0 (socket says 172.21.15.119:53810) 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 -- 172.21.15.119:0/3767312387 learned_addr learned my addr 172.21.15.119:0/3767312387 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:27.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26edd9b700 1 --2- 172.21.15.119:0/3767312387 >> v2:172.21.15.145:3300/0 conn(0x7f26e800acc0 0x7f26e80e3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 -- 172.21.15.119:0/3767312387 >> v2:172.21.15.145:3300/0 conn(0x7f26e800acc0 msgr2=0x7f26e80e3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:27.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 --2- 172.21.15.119:0/3767312387 >> v2:172.21.15.145:3300/0 conn(0x7f26e800acc0 0x7f26e80e3040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 -- 172.21.15.119:0/3767312387 >> v1:172.21.15.145:6789/0 conn(0x7f26e8033990 legacy=0x7f26e800a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:27.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 -- 172.21.15.119:0/3767312387 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e804ffa0 con 0x7f26e80507a0 2024-09-13T16:25:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26eed9d700 1 --2- 172.21.15.119:0/3767312387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e80de2a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f26f00698b0 tx=0x7f26f004f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=961729d1bd1af234 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ed59a700 1 -- 172.21.15.119:0/3767312387 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0067690 con 0x7f26e80507a0 2024-09-13T16:25:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ed59a700 1 -- 172.21.15.119:0/3767312387 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f004f010 con 0x7f26e80507a0 2024-09-13T16:25:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.958+0000 7f26ed59a700 1 -- 172.21.15.119:0/3767312387 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f0067b90 con 0x7f26e80507a0 2024-09-13T16:25:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- 172.21.15.119:0/3767312387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 msgr2=0x7f26e80de2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 --2- 172.21.15.119:0/3767312387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e80de2a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f26f00698b0 tx=0x7f26f004f710 comp rx=0 tx=0).stop 2024-09-13T16:25:27.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- 172.21.15.119:0/3767312387 shutdown_connections 2024-09-13T16:25:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 --2- 172.21.15.119:0/3767312387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e80de2a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 --2- 172.21.15.119:0/3767312387 >> v2:172.21.15.145:3300/0 conn(0x7f26e800acc0 0x7f26e80e3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- 172.21.15.119:0/3767312387 >> 172.21.15.119:0/3767312387 conn(0x7f26e8036140 msgr2=0x7f26e8036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- 172.21.15.119:0/3767312387 shutdown_connections 2024-09-13T16:25:27.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- 172.21.15.119:0/3767312387 wait complete. 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 Processor -- start 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- start start 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26e800acc0 0x7f26e80d59d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26e8033990 0x7f26e801e510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e8048590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e8048100 con 0x7f26e80507a0 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e801f3f0 con 0x7f26e800acc0 2024-09-13T16:25:27.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26e801f6d0 con 0x7f26e8033990 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e8048590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e8048590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53818/0 (socket says 172.21.15.119:53818) 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 -- 172.21.15.119:0/1453451655 learned_addr learned my addr 172.21.15.119:0/1453451655 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ee59c700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26e800acc0 0x7f26e80d59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26edd9b700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26e8033990 0x7f26e801e510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 -- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26e8033990 msgr2=0x7f26e801e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26e8033990 0x7f26e801e510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 -- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26e800acc0 msgr2=0x7f26e80d59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26e800acc0 0x7f26e80d59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:27.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 -- 172.21.15.119:0/1453451655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26e801f9b0 con 0x7f26e80507a0 2024-09-13T16:25:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26eed9d700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e8048590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f26f002ce60 tx=0x7f26f006a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26e6ffd700 1 -- 172.21.15.119:0/1453451655 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f009a040 con 0x7f26e80507a0 2024-09-13T16:25:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26e6ffd700 1 -- 172.21.15.119:0/1453451655 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26f0072350 con 0x7f26e80507a0 2024-09-13T16:25:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e80d5f10 con 0x7f26e80507a0 2024-09-13T16:25:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.962+0000 7f26e6ffd700 1 -- 172.21.15.119:0/1453451655 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26f00726c0 con 0x7f26e80507a0 2024-09-13T16:25:27.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.966+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e801ec30 con 0x7f26e80507a0 2024-09-13T16:25:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.966+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e8047ed0 con 0x7f26e80507a0 2024-09-13T16:25:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.966+0000 7f26e6ffd700 1 -- 172.21.15.119:0/1453451655 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f26f0072820 con 0x7f26e80507a0 2024-09-13T16:25:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.966+0000 7f26e6ffd700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f26d0066730 0x7f26d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.966+0000 7f26e6ffd700 1 -- 172.21.15.119:0/1453451655 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f26f00ff2b0 con 0x7f26e80507a0 2024-09-13T16:25:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.966+0000 7f26ee59c700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f26d0066730 0x7f26d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:27.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.966+0000 7f26ee59c700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f26d0066730 0x7f26d0068bf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26e0007fd0 tx=0x7f26e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:27.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:27.970+0000 7f26e6ffd700 1 -- 172.21.15.119:0/1453451655 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f26e8047ed0 con 0x7f26e80507a0 2024-09-13T16:25:28.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f26e8047ed0 con 0x7f26e80507a0 2024-09-13T16:25:28.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26e6ffd700 1 -- 172.21.15.119:0/1453451655 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f26e8047ed0 con 0x7f26e80507a0 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f26d0066730 msgr2=0x7f26d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f26d0066730 0x7f26d0068bf0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f26e0007fd0 tx=0x7f26e0009040 comp rx=0 tx=0).stop 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 msgr2=0x7f26e8048590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e8048590 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f26f002ce60 tx=0x7f26f006a080 comp rx=0 tx=0).stop 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 shutdown_connections 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f26d0066730 0x7f26d0068bf0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26e80507a0 0x7f26e8048590 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26e8033990 0x7f26e801e510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 --2- 172.21.15.119:0/1453451655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26e800acc0 0x7f26e80d59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 >> 172.21.15.119:0/1453451655 conn(0x7f26e8036140 msgr2=0x7f26e801c5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:28.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 shutdown_connections 2024-09-13T16:25:28.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.154+0000 7f26ef59e700 1 -- 172.21.15.119:0/1453451655 wait complete. 2024-09-13T16:25:28.160 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:25:28.174 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:25:28.175 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:25:28.183 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:28.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525b41700 1 Processor -- start 2024-09-13T16:25:28.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525b41700 1 -- start start 2024-09-13T16:25:28.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525b41700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f95201594f0 0x7f952006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525b41700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f95200b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525b41700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95200b2280 con 0x7f95200b3630 2024-09-13T16:25:28.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525b41700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9520070c10 con 0x7f95201594f0 2024-09-13T16:25:28.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525b41700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952006deb0 con 0x7f952006da70 2024-09-13T16:25:28.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525340700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f95200b3630 0x7f952015d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41958/0 (socket says 172.21.15.119:41958) 2024-09-13T16:25:28.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9525340700 1 -- 172.21.15.119:0/1558251438 learned_addr learned my addr 172.21.15.119:0/1558251438 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:28.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9524b3f700 1 --2- 172.21.15.119:0/1558251438 >> v2:172.21.15.119:3300/0 conn(0x7f95201594f0 0x7f952006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f951ffff700 1 --2- 172.21.15.119:0/1558251438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f95200b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9524b3f700 1 -- 172.21.15.119:0/1558251438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 msgr2=0x7f95200b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:28.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9524b3f700 1 --2- 172.21.15.119:0/1558251438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f95200b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9524b3f700 1 -- 172.21.15.119:0/1558251438 >> v1:172.21.15.119:6789/0 conn(0x7f95200b3630 legacy=0x7f952015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:28.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f9524b3f700 1 -- 172.21.15.119:0/1558251438 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95200cc890 con 0x7f95201594f0 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.338+0000 7f951ffff700 1 --2- 172.21.15.119:0/1558251438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f95200b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9524b3f700 1 --2- 172.21.15.119:0/1558251438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95201594f0 0x7f952006d530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9520070d50 tx=0x7f9508009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f0a6d494ef2a26d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f951f7fe700 1 -- 172.21.15.119:0/1558251438 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508011830 con 0x7f95201594f0 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f951f7fe700 1 -- 172.21.15.119:0/1558251438 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9508011990 con 0x7f95201594f0 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f951f7fe700 1 -- 172.21.15.119:0/1558251438 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508011d20 con 0x7f95201594f0 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 -- 172.21.15.119:0/1558251438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95201594f0 msgr2=0x7f952006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 --2- 172.21.15.119:0/1558251438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95201594f0 0x7f952006d530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f9520070d50 tx=0x7f9508009b30 comp rx=0 tx=0).stop 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 -- 172.21.15.119:0/1558251438 shutdown_connections 2024-09-13T16:25:28.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 --2- 172.21.15.119:0/1558251438 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f95200b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 --2- 172.21.15.119:0/1558251438 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95201594f0 0x7f952006d530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 -- 172.21.15.119:0/1558251438 >> 172.21.15.119:0/1558251438 conn(0x7f952006a350 msgr2=0x7f952006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:28.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 -- 172.21.15.119:0/1558251438 shutdown_connections 2024-09-13T16:25:28.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 -- 172.21.15.119:0/1558251438 wait complete. 2024-09-13T16:25:28.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 Processor -- start 2024-09-13T16:25:28.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.342+0000 7f9525b41700 1 -- start start 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f952014cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95200b3630 0x7f95200acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95201594f0 0x7f9520148750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95200c3860 con 0x7f95200b3630 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9520149600 con 0x7f95201594f0 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f952014b250 con 0x7f952006da70 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f951ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95200b3630 0x7f95200acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f952014cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f951ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95200b3630 0x7f95200acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53834/0 (socket says 172.21.15.119:53834) 2024-09-13T16:25:28.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f951ffff700 1 -- 172.21.15.119:0/2823534322 learned_addr learned my addr 172.21.15.119:0/2823534322 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:28.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f952014cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:43104/0 (socket says 172.21.15.119:43104) 2024-09-13T16:25:28.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525340700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95201594f0 0x7f9520148750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 -- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95201594f0 msgr2=0x7f9520148750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95201594f0 0x7f9520148750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 -- 172.21.15.119:0/2823534322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95200b3630 msgr2=0x7f95200acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95200b3630 0x7f95200acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 -- 172.21.15.119:0/2823534322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95100094b0 con 0x7f952006da70 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9524b3f700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f952014cbe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9508016000 tx=0x7f9508004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f951d7fa700 1 -- 172.21.15.119:0/2823534322 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95080038c0 con 0x7f952006da70 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f951d7fa700 1 -- 172.21.15.119:0/2823534322 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f950801e070 con 0x7f952006da70 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95080094b0 con 0x7f952006da70 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f951d7fa700 1 -- 172.21.15.119:0/2823534322 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9508003c60 con 0x7f952006da70 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f952015a990 con 0x7f952006da70 2024-09-13T16:25:28.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.346+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95200b2710 con 0x7f952006da70 2024-09-13T16:25:28.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.350+0000 7f951d7fa700 1 -- 172.21.15.119:0/2823534322 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f950802d450 con 0x7f952006da70 2024-09-13T16:25:28.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.350+0000 7f951d7fa700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9500066730 0x7f9500068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.350+0000 7f951d7fa700 1 -- 172.21.15.119:0/2823534322 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9508085770 con 0x7f952006da70 2024-09-13T16:25:28.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.350+0000 7f951ffff700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9500066730 0x7f9500068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.350+0000 7f951ffff700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9500066730 0x7f9500068bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f95200b01e0 tx=0x7f9510009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.358+0000 7f951d7fa700 1 -- 172.21.15.119:0/2823534322 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9508052860 con 0x7f952006da70 2024-09-13T16:25:28.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.534+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9520150150 con 0x7f952006da70 2024-09-13T16:25:28.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f951d7fa700 1 -- 172.21.15.119:0/2823534322 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f9508055e80 con 0x7f952006da70 2024-09-13T16:25:28.540 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:28.540 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:28.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9500066730 msgr2=0x7f9500068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9500066730 0x7f9500068bf0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f95200b01e0 tx=0x7f9510009240 comp rx=0 tx=0).stop 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 msgr2=0x7f952014cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f952014cbe0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9508016000 tx=0x7f9508004770 comp rx=0 tx=0).stop 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 shutdown_connections 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9500066730 0x7f9500068bf0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95201594f0 0x7f9520148750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95200b3630 0x7f95200acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 --2- 172.21.15.119:0/2823534322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f952006da70 0x7f952014cbe0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 >> 172.21.15.119:0/2823534322 conn(0x7f952006a350 msgr2=0x7f952015c1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 shutdown_connections 2024-09-13T16:25:28.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.538+0000 7f9525b41700 1 -- 172.21.15.119:0/2823534322 wait complete. 2024-09-13T16:25:28.556 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:25:28.557 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:28.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9abfff700 1 Processor -- start 2024-09-13T16:25:28.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9abfff700 1 -- start start 2024-09-13T16:25:28.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9abfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 0x7fe9ac0aead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9abfff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac15d410 con 0x7fe9ac06a350 2024-09-13T16:25:28.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9abfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9ac15d6f0 con 0x7fe9ac06f200 2024-09-13T16:25:28.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9abfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac0b2f10 con 0x7fe9ac15cea0 2024-09-13T16:25:28.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9ab7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe9ac06a350 0x7fe9ac0710c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41974/0 (socket says 172.21.15.119:41974) 2024-09-13T16:25:28.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9ab7fe700 1 -- 172.21.15.119:0/2531628402 learned_addr learned my addr 172.21.15.119:0/2531628402 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:28.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9aa7fc700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 0x7fe9ac0aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706301080 0 0) 0x7fe9ac15d410 con 0x7fe9ac06a350 2024-09-13T16:25:28.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9980035d0 con 0x7fe9ac06a350 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 388029073 0 0) 0x7fe9980035d0 con 0x7fe9ac06a350 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 msgr2=0x7fe9ac0aead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9a9ffb700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 0x7fe9ac0aead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.670+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 >> v1:172.21.15.145:6789/0 conn(0x7fe9ac06f200 legacy=0x7fe9ac0b2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac15d9d0 con 0x7fe9ac06a350 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fe9a40019f0 con 0x7fe9ac06a350 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 >> v1:172.21.15.119:6789/0 conn(0x7fe9ac06a350 legacy=0x7fe9ac0710c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe998003a50 0x7fe998003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 0x7fe998004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe998004db0 0x7fe998005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac0b2f10 con 0x7fe998003a50 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe99800ba60 con 0x7fe998004db0 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe99800bc20 con 0x7fe9ac15cea0 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aaffd700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe998003a50 0x7fe998003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aa7fc700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 0x7fe998004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9ab7fe700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe998004db0 0x7fe998005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aaffd700 1 -- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 msgr2=0x7fe998004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aaffd700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9ac15cea0 0x7fe998004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aaffd700 1 -- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe998004db0 msgr2=0x7fe998005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aaffd700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe998004db0 0x7fe998005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aaffd700 1 -- 172.21.15.119:0/2531628402 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9980035d0 con 0x7fe998003a50 2024-09-13T16:25:28.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9ab7fe700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe998004db0 0x7fe998005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9aaffd700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe998003a50 0x7fe998003e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0018f0 tx=0x7fe9980058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28cd9cc982ab1be6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c001390 con 0x7fe998003a50 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c012620 con 0x7fe998003a50 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9a9ffb700 1 -- 172.21.15.119:0/2531628402 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c0129d0 con 0x7fe998003a50 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9abfff700 1 -- 172.21.15.119:0/2531628402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe998003a50 msgr2=0x7fe998003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9abfff700 1 --2- 172.21.15.119:0/2531628402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe998003a50 0x7fe998003e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe99c0018f0 tx=0x7fe9980058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9ab7fe700 1 -- 172.21.15.119:0/2531628402 reap_dead start 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9abfff700 1 -- 172.21.15.119:0/2531628402 shutdown_connections 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9abfff700 1 -- 172.21.15.119:0/2531628402 >> 172.21.15.119:0/2531628402 conn(0x7fe9ac0b1500 msgr2=0x7fe9ac0b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9abfff700 1 -- 172.21.15.119:0/2531628402 shutdown_connections 2024-09-13T16:25:28.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.674+0000 7fe9abfff700 1 -- 172.21.15.119:0/2531628402 wait complete. 2024-09-13T16:25:28.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 Processor -- start 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 -- start start 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe998003a50 0x7fe9ac0c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9ac159bc0 0x7fe9ac159480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9ac06dd10 0x7fe9ac06e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac1f2150 con 0x7fe9ac06dd10 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac06e6d0 con 0x7fe9ac159bc0 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ac1599c0 con 0x7fe998003a50 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9ac06dd10 0x7fe9ac06e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9ac06dd10 0x7fe9ac06e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53852/0 (socket says 172.21.15.119:53852) 2024-09-13T16:25:28.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 -- 172.21.15.119:0/3350186614 learned_addr learned my addr 172.21.15.119:0/3350186614 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:28.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9aa7fc700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9ac159bc0 0x7fe9ac159480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9aaffd700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe998003a50 0x7fe9ac0c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 -- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe998003a50 msgr2=0x7fe9ac0c5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe998003a50 0x7fe9ac0c5ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 -- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9ac159bc0 msgr2=0x7fe9ac159480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9ac159bc0 0x7fe9ac159480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 -- 172.21.15.119:0/3350186614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac0c6190 con 0x7fe9ac06dd10 2024-09-13T16:25:28.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9ab7fe700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9ac06dd10 0x7fe9ac06e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0ae7d0 tx=0x7fe9a4007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9a37fe700 1 -- 172.21.15.119:0/3350186614 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400e860 con 0x7fe9ac06dd10 2024-09-13T16:25:28.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9a37fe700 1 -- 172.21.15.119:0/3350186614 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a400e9c0 con 0x7fe9ac06dd10 2024-09-13T16:25:28.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.678+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ac15a3e0 con 0x7fe9ac06dd10 2024-09-13T16:25:28.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.682+0000 7fe9a37fe700 1 -- 172.21.15.119:0/3350186614 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a400ed70 con 0x7fe9ac06dd10 2024-09-13T16:25:28.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.682+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ac1f2640 con 0x7fe9ac06dd10 2024-09-13T16:25:28.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.682+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ac0aead0 con 0x7fe9ac06dd10 2024-09-13T16:25:28.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.682+0000 7fe9a37fe700 1 -- 172.21.15.119:0/3350186614 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe9a401d460 con 0x7fe9ac06dd10 2024-09-13T16:25:28.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.682+0000 7fe9a37fe700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe98c066760 0x7fe98c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.686+0000 7fe9a37fe700 1 -- 172.21.15.119:0/3350186614 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe9a40845e0 con 0x7fe9ac06dd10 2024-09-13T16:25:28.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.686+0000 7fe9aaffd700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe98c066760 0x7fe98c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.686+0000 7fe9aaffd700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe98c066760 0x7fe98c068c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe99c014040 tx=0x7fe99c005040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.690+0000 7fe9a37fe700 1 -- 172.21.15.119:0/3350186614 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe9a4051660 con 0x7fe9ac06dd10 2024-09-13T16:25:28.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.858+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe9ac06d7f0 con 0x7fe9ac06dd10 2024-09-13T16:25:28.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.858+0000 7fe9a37fe700 1 -- 172.21.15.119:0/3350186614 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe9a4054c80 con 0x7fe9ac06dd10 2024-09-13T16:25:28.863 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:28.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe98c066760 msgr2=0x7fe98c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe98c066760 0x7fe98c068c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe99c014040 tx=0x7fe99c005040 comp rx=0 tx=0).stop 2024-09-13T16:25:28.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9ac06dd10 msgr2=0x7fe9ac06e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9ac06dd10 0x7fe9ac06e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac0ae7d0 tx=0x7fe9a4007c90 comp rx=0 tx=0).stop 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 shutdown_connections 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe98c066760 0x7fe98c068c20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9ac06dd10 0x7fe9ac06e190 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9ac159bc0 0x7fe9ac159480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 --2- 172.21.15.119:0/3350186614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe998003a50 0x7fe9ac0c5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 >> 172.21.15.119:0/3350186614 conn(0x7fe9ac0b1500 msgr2=0x7fe9ac15ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 shutdown_connections 2024-09-13T16:25:28.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.862+0000 7fe9abfff700 1 -- 172.21.15.119:0/3350186614 wait complete. 2024-09-13T16:25:28.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:25:28.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 Processor -- start 2024-09-13T16:25:28.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 -- start start 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc7e40cf190 0x7fc7e413ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e415d560 0x7fc7e415d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc7e406a350 0x7fc7e406a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e406acf0 con 0x7fc7e406a350 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7e406cc00 con 0x7fc7e40cf190 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.982+0000 7fc7e882e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e406cee0 con 0x7fc7e415d560 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc7e406a350 0x7fc7e406a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc7e40cf190 0x7fc7e413ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc7e40cf190 0x7fc7e413ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:33320/0 (socket says 172.21.15.119:33320) 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 -- 172.21.15.119:0/18846632 learned_addr learned my addr 172.21.15.119:0/18846632 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e359e700 1 --2- 172.21.15.119:0/18846632 >> v2:172.21.15.119:3300/0 conn(0x7fc7e406a350 0x7fc7e406a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53866/0 (socket says 172.21.15.119:53866) 2024-09-13T16:25:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e259c700 1 --2- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e415d560 0x7fc7e415d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 -- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e415d560 msgr2=0x7fc7e415d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 --2- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e415d560 0x7fc7e415d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 -- 172.21.15.119:0/18846632 >> v2:172.21.15.119:3300/0 conn(0x7fc7e406a350 msgr2=0x7fc7e406a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 --2- 172.21.15.119:0/18846632 >> v2:172.21.15.119:3300/0 conn(0x7fc7e406a350 0x7fc7e406a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 -- 172.21.15.119:0/18846632 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7e406d1c0 con 0x7fc7e40cf190 2024-09-13T16:25:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e359e700 1 --2- 172.21.15.119:0/18846632 >> v2:172.21.15.119:3300/0 conn(0x7fc7e406a350 0x7fc7e406a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e2d9d700 1 --2- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e40cf190 0x7fc7e413ff50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4070970 tx=0x7fc7d400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2be2826d98602c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e1d9b700 1 -- 172.21.15.119:0/18846632 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400e070 con 0x7fc7e40cf190 2024-09-13T16:25:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e1d9b700 1 -- 172.21.15.119:0/18846632 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d400b910 con 0x7fc7e40cf190 2024-09-13T16:25:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e1d9b700 1 -- 172.21.15.119:0/18846632 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d400bcc0 con 0x7fc7e40cf190 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 -- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e40cf190 msgr2=0x7fc7e413ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 --2- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e40cf190 0x7fc7e413ff50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc7e4070970 tx=0x7fc7d400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 -- 172.21.15.119:0/18846632 shutdown_connections 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 --2- 172.21.15.119:0/18846632 >> v2:172.21.15.119:3300/0 conn(0x7fc7e406a350 0x7fc7e406a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 --2- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e415d560 0x7fc7e415d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 --2- 172.21.15.119:0/18846632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e40cf190 0x7fc7e413ff50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 -- 172.21.15.119:0/18846632 >> 172.21.15.119:0/18846632 conn(0x7fc7e406b460 msgr2=0x7fc7e406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 -- 172.21.15.119:0/18846632 shutdown_connections 2024-09-13T16:25:28.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.986+0000 7fc7e882e700 1 -- 172.21.15.119:0/18846632 wait complete. 2024-09-13T16:25:28.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 Processor -- start 2024-09-13T16:25:28.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 -- start start 2024-09-13T16:25:28.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e406a350 0x7fc7e413b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e40cf190 0x7fc7e4160600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7e415d560 0x7fc7e41f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e415eab0 con 0x7fc7e415d560 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e415ed90 con 0x7fc7e406a350 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e882e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7e415f070 con 0x7fc7e40cf190 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7e415d560 0x7fc7e41f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7e415d560 0x7fc7e41f2aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53882/0 (socket says 172.21.15.119:53882) 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 -- 172.21.15.119:0/2004548385 learned_addr learned my addr 172.21.15.119:0/2004548385 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e2d9d700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e406a350 0x7fc7e413b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e259c700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e40cf190 0x7fc7e4160600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 -- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e40cf190 msgr2=0x7fc7e4160600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e40cf190 0x7fc7e4160600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 -- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e406a350 msgr2=0x7fc7e413b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e406a350 0x7fc7e413b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 -- 172.21.15.119:0/2004548385 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc0094b0 con 0x7fc7e415d560 2024-09-13T16:25:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.990+0000 7fc7e359e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7e415d560 0x7fc7e41f2aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00e930 tx=0x7fc7dc00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.994+0000 7fc7db7fe700 1 -- 172.21.15.119:0/2004548385 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc009710 con 0x7fc7e415d560 2024-09-13T16:25:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.994+0000 7fc7db7fe700 1 -- 172.21.15.119:0/2004548385 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7dc004020 con 0x7fc7e415d560 2024-09-13T16:25:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.994+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d4009b30 con 0x7fc7e415d560 2024-09-13T16:25:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.994+0000 7fc7db7fe700 1 -- 172.21.15.119:0/2004548385 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7dc0043b0 con 0x7fc7e415d560 2024-09-13T16:25:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.994+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7e415f630 con 0x7fc7e415d560 2024-09-13T16:25:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.994+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7e415f850 con 0x7fc7e415d560 2024-09-13T16:25:28.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.994+0000 7fc7db7fe700 1 -- 172.21.15.119:0/2004548385 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc7dc0036c0 con 0x7fc7e415d560 2024-09-13T16:25:28.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.998+0000 7fc7db7fe700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7c4066760 0x7fc7c4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:28.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.998+0000 7fc7db7fe700 1 -- 172.21.15.119:0/2004548385 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc7dc084020 con 0x7fc7e415d560 2024-09-13T16:25:29.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.998+0000 7fc7e2d9d700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7c4066760 0x7fc7c4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:28.998+0000 7fc7e2d9d700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7c4066760 0x7fc7c4068c20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400b370 tx=0x7fc7d4009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.002+0000 7fc7db7fe700 1 -- 172.21.15.119:0/2004548385 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc7dc0510a0 con 0x7fc7e415d560 2024-09-13T16:25:29.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.174+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc7e406a7b0 con 0x7fc7e415d560 2024-09-13T16:25:29.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7db7fe700 1 -- 172.21.15.119:0/2004548385 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc7dc0546c0 con 0x7fc7e415d560 2024-09-13T16:25:29.180 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:29.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7c4066760 msgr2=0x7fc7c4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7c4066760 0x7fc7c4068c20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc7d400b370 tx=0x7fc7d4009000 comp rx=0 tx=0).stop 2024-09-13T16:25:29.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7e415d560 msgr2=0x7fc7e41f2aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7e415d560 0x7fc7e41f2aa0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc7dc00e930 tx=0x7fc7dc00ec40 comp rx=0 tx=0).stop 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 shutdown_connections 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7c4066760 0x7fc7c4068c20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7e415d560 0x7fc7e41f2aa0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7e40cf190 0x7fc7e4160600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 --2- 172.21.15.119:0/2004548385 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7e406a350 0x7fc7e413b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 >> 172.21.15.119:0/2004548385 conn(0x7fc7e406b460 msgr2=0x7fc7e40b08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 shutdown_connections 2024-09-13T16:25:29.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.178+0000 7fc7e882e700 1 -- 172.21.15.119:0/2004548385 wait complete. 2024-09-13T16:25:29.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:25:29.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.302+0000 7f769359e700 1 Processor -- start 2024-09-13T16:25:29.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.302+0000 7f769359e700 1 -- start start 2024-09-13T16:25:29.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.302+0000 7f769359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f76940ae080 0x7f76940b1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f769415d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f769415dc20 0x7f7694159af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769415e0b0 con 0x7f769415dc20 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76940b1f60 con 0x7f76940ae080 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76940b2240 con 0x7f769415d2a0 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f769415dc20 0x7f7694159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f769415dc20 0x7f7694159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53884/0 (socket says 172.21.15.119:53884) 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 -- 172.21.15.119:0/2115080797 learned_addr learned my addr 172.21.15.119:0/2115080797 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769259c700 1 --2- 172.21.15.119:0/2115080797 >> v2:172.21.15.145:3300/0 conn(0x7f76940ae080 0x7f76940b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7691d9b700 1 --2- 172.21.15.119:0/2115080797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f769415d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 -- 172.21.15.119:0/2115080797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 msgr2=0x7f769415d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 --2- 172.21.15.119:0/2115080797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f769415d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 -- 172.21.15.119:0/2115080797 >> v2:172.21.15.145:3300/0 conn(0x7f76940ae080 msgr2=0x7f76940b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 --2- 172.21.15.119:0/2115080797 >> v2:172.21.15.145:3300/0 conn(0x7f76940ae080 0x7f76940b1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 -- 172.21.15.119:0/2115080797 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76940b2520 con 0x7f769415dc20 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7691d9b700 1 --2- 172.21.15.119:0/2115080797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f769415d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769259c700 1 --2- 172.21.15.119:0/2115080797 >> v2:172.21.15.145:3300/0 conn(0x7f76940ae080 0x7f76940b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f7692d9d700 1 --2- 172.21.15.119:0/2115080797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f7694159af0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f769406b610 tx=0x7f768c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=367eb7fb98c79c65 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769159a700 1 -- 172.21.15.119:0/2115080797 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c010070 con 0x7f769415dc20 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769159a700 1 -- 172.21.15.119:0/2115080797 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c00dd90 con 0x7f769415dc20 2024-09-13T16:25:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769159a700 1 -- 172.21.15.119:0/2115080797 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c00b310 con 0x7f769415dc20 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- 172.21.15.119:0/2115080797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 msgr2=0x7f7694159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 --2- 172.21.15.119:0/2115080797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f7694159af0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f769406b610 tx=0x7f768c00d430 comp rx=0 tx=0).stop 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- 172.21.15.119:0/2115080797 shutdown_connections 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 --2- 172.21.15.119:0/2115080797 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f7694159af0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 --2- 172.21.15.119:0/2115080797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f769415d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 --2- 172.21.15.119:0/2115080797 >> v2:172.21.15.145:3300/0 conn(0x7f76940ae080 0x7f76940b1860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- 172.21.15.119:0/2115080797 >> 172.21.15.119:0/2115080797 conn(0x7f76940b0980 msgr2=0x7f76940b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:29.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- 172.21.15.119:0/2115080797 shutdown_connections 2024-09-13T16:25:29.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.306+0000 7f769359e700 1 -- 172.21.15.119:0/2115080797 wait complete. 2024-09-13T16:25:29.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 Processor -- start 2024-09-13T16:25:29.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 -- start start 2024-09-13T16:25:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76940ae080 0x7f76941463a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f7694066090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f76940665d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941468e0 con 0x7f769415dc20 2024-09-13T16:25:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76941f1f70 con 0x7f76940ae080 2024-09-13T16:25:29.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7694149920 con 0x7f769415d2a0 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f76940665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f76940665d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53892/0 (socket says 172.21.15.119:53892) 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 -- 172.21.15.119:0/3333068011 learned_addr learned my addr 172.21.15.119:0/3333068011 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769259c700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76940ae080 0x7f76941463a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7691d9b700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f7694066090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 -- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 msgr2=0x7f7694066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f7694066090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 -- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76940ae080 msgr2=0x7f76941463a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76940ae080 0x7f76941463a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 -- 172.21.15.119:0/3333068011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c00bd40 con 0x7f769415dc20 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7691d9b700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f7694066090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:29.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f7692d9d700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f76940665d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f769406f490 tx=0x7f768c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f768affd700 1 -- 172.21.15.119:0/3333068011 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c009a10 con 0x7f769415dc20 2024-09-13T16:25:29.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f768affd700 1 -- 172.21.15.119:0/3333068011 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f768c009b90 con 0x7f769415dc20 2024-09-13T16:25:29.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f768affd700 1 -- 172.21.15.119:0/3333068011 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768c003e70 con 0x7f769415dc20 2024-09-13T16:25:29.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76941f21c0 con 0x7f769415dc20 2024-09-13T16:25:29.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.310+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76941f2780 con 0x7f769415dc20 2024-09-13T16:25:29.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.314+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769415d6e0 con 0x7f769415dc20 2024-09-13T16:25:29.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.314+0000 7f768affd700 1 -- 172.21.15.119:0/3333068011 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f768c004040 con 0x7f769415dc20 2024-09-13T16:25:29.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.314+0000 7f768affd700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7674066730 0x7f7674068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.314+0000 7f768affd700 1 -- 172.21.15.119:0/3333068011 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f768c085d00 con 0x7f769415dc20 2024-09-13T16:25:29.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.314+0000 7f769259c700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7674066730 0x7f7674068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.318+0000 7f769259c700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7674066730 0x7f7674068bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7694159460 tx=0x7f767c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.322+0000 7f768affd700 1 -- 172.21.15.119:0/3333068011 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f768c052e10 con 0x7f769415dc20 2024-09-13T16:25:29.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f76941589b0 con 0x7f769415dc20 2024-09-13T16:25:29.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f768affd700 1 -- 172.21.15.119:0/3333068011 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f768c056430 con 0x7f769415dc20 2024-09-13T16:25:29.492 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:25:29.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7674066730 msgr2=0x7f7674068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7674066730 0x7f7674068bf0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f7694159460 tx=0x7f767c008040 comp rx=0 tx=0).stop 2024-09-13T16:25:29.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 msgr2=0x7f76940665d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f76940665d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f769406f490 tx=0x7f768c004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 shutdown_connections 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7674066730 0x7f7674068bf0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f769415dc20 0x7f76940665d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f769415d2a0 0x7f7694066090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 --2- 172.21.15.119:0/3333068011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76940ae080 0x7f76941463a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 >> 172.21.15.119:0/3333068011 conn(0x7f76940b0980 msgr2=0x7f76940b3260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 shutdown_connections 2024-09-13T16:25:29.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.490+0000 7f769359e700 1 -- 172.21.15.119:0/3333068011 wait complete. 2024-09-13T16:25:29.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:25:29.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.622+0000 7f68009d9700 1 Processor -- start 2024-09-13T16:25:29.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.622+0000 7f68009d9700 1 -- start start 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f67fc0ce8c0 0x7f67fc0d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f67fc0d1610 0x7f67fc0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0d1a20 con 0x7f67fc15cc00 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67fc0d0df0 con 0x7f67fc0d1610 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67fc0cdb00 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f67fc0d1610 0x7f67fc0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f67fc0d1610 0x7f67fc0b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53898/0 (socket says 172.21.15.119:53898) 2024-09-13T16:25:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 -- 172.21.15.119:0/1696434684 learned_addr learned my addr 172.21.15.119:0/1696434684 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:29.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67faffd700 1 --2- 172.21.15.119:0/1696434684 >> v2:172.21.15.145:3300/0 conn(0x7f67fc0ce8c0 0x7f67fc0d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 -- 172.21.15.119:0/1696434684 >> v2:172.21.15.145:3300/0 conn(0x7f67fc0ce8c0 msgr2=0x7f67fc0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:29.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 --2- 172.21.15.119:0/1696434684 >> v2:172.21.15.145:3300/0 conn(0x7f67fc0ce8c0 0x7f67fc0d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 -- 172.21.15.119:0/1696434684 >> v1:172.21.15.119:6789/0 conn(0x7f67fc15cc00 legacy=0x7f67fc0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 -- 172.21.15.119:0/1696434684 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc0cdde0 con 0x7f67fc0d1610 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fbfff700 1 --2- 172.21.15.119:0/1696434684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b5960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f67fc08c050 tx=0x7f67f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=976201c6e93fbf59 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fa7fc700 1 -- 172.21.15.119:0/1696434684 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f400e070 con 0x7f67fc0d1610 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fa7fc700 1 -- 172.21.15.119:0/1696434684 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67f400b9c0 con 0x7f67fc0d1610 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f67fa7fc700 1 -- 172.21.15.119:0/1696434684 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f400bd70 con 0x7f67fc0d1610 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- 172.21.15.119:0/1696434684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0d1610 msgr2=0x7f67fc0b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 --2- 172.21.15.119:0/1696434684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b5960 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f67fc08c050 tx=0x7f67f400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- 172.21.15.119:0/1696434684 shutdown_connections 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 --2- 172.21.15.119:0/1696434684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b5960 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 --2- 172.21.15.119:0/1696434684 >> v2:172.21.15.145:3300/0 conn(0x7f67fc0ce8c0 0x7f67fc0d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- 172.21.15.119:0/1696434684 >> 172.21.15.119:0/1696434684 conn(0x7f67fc08c080 msgr2=0x7f67fc0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:29.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- 172.21.15.119:0/1696434684 shutdown_connections 2024-09-13T16:25:29.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- 172.21.15.119:0/1696434684 wait complete. 2024-09-13T16:25:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 Processor -- start 2024-09-13T16:25:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.626+0000 7f68009d9700 1 -- start start 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0ce8c0 0x7f67fc1f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b64d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67fc15cc00 0x7f67fc0b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc1f6d40 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc14c080 con 0x7f67fc0d1610 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67fc14c360 con 0x7f67fc15cc00 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0ce8c0 0x7f67fc1f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0ce8c0 0x7f67fc1f6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53904/0 (socket says 172.21.15.119:53904) 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 -- 172.21.15.119:0/1449531783 learned_addr learned my addr 172.21.15.119:0/1449531783 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67faffd700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fbfff700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67fc15cc00 0x7f67fc0b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 -- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67fc15cc00 msgr2=0x7f67fc0b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67fc15cc00 0x7f67fc0b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 -- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67fc0d1610 msgr2=0x7f67fc0b64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 -- 172.21.15.119:0/1449531783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f4009b30 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67faffd700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fbfff700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67fc15cc00 0x7f67fc0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67fb7fe700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0ce8c0 0x7f67fc1f6800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c500 tx=0x7f67ec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67f3fff700 1 -- 172.21.15.119:0/1449531783 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0052d0 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67fc14c640 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67f3fff700 1 -- 172.21.15.119:0/1449531783 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67ec005430 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67f3fff700 1 -- 172.21.15.119:0/1449531783 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67ec0055a0 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67fc149d10 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67fc0d3670 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.630+0000 7f67f3fff700 1 -- 172.21.15.119:0/1449531783 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f67ec0038e0 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.634+0000 7f67f3fff700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f67dc066730 0x7f67dc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.634+0000 7f67f3fff700 1 -- 172.21.15.119:0/1449531783 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f67ec0838f0 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.634+0000 7f67faffd700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f67dc066730 0x7f67dc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.634+0000 7f67faffd700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f67dc066730 0x7f67dc068bf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67fc1e7d00 tx=0x7f67e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.642+0000 7f67f3fff700 1 -- 172.21.15.119:0/1449531783 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f67ec050910 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.810+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f67fc08c560 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.810+0000 7f67f3fff700 1 -- 172.21.15.119:0/1449531783 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f67ec017020 con 0x7f67fc0ce8c0 2024-09-13T16:25:29.815 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f67dc066730 msgr2=0x7f67dc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f67dc066730 0x7f67dc068bf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f67fc1e7d00 tx=0x7f67e4008040 comp rx=0 tx=0).stop 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0ce8c0 msgr2=0x7f67fc1f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0ce8c0 0x7f67fc1f6800 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f67ec00c500 tx=0x7f67ec00c8c0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 shutdown_connections 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f67dc066730 0x7f67dc068bf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f67fc15cc00 0x7f67fc0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f67fc0d1610 0x7f67fc0b64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 --2- 172.21.15.119:0/1449531783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f67fc0ce8c0 0x7f67fc1f6800 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 >> 172.21.15.119:0/1449531783 conn(0x7f67fc08c080 msgr2=0x7f67fc0b5ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:29.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 shutdown_connections 2024-09-13T16:25:29.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.814+0000 7f68009d9700 1 -- 172.21.15.119:0/1449531783 wait complete. 2024-09-13T16:25:29.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:25:29.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a5d1d700 1 Processor -- start 2024-09-13T16:25:29.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a5d1d700 1 -- start start 2024-09-13T16:25:29.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a5d1d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a00ce950 0x7f62a015d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a5d1d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f62a006a350 0x7f62a006c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a5d1d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62a00ad5b0 con 0x7f62a00abf40 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a5d1d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62a015df10 con 0x7f62a006a350 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a5d1d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a006c8a0 con 0x7f62a00ce950 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a551c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f62a00abf40 0x7f62a0071a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41996/0 (socket says 172.21.15.119:41996) 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a551c700 1 -- 172.21.15.119:0/2949460381 learned_addr learned my addr 172.21.15.119:0/2949460381 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f629ffff700 1 --2- 172.21.15.119:0/2949460381 >> v2:172.21.15.145:3300/0 conn(0x7f62a006a350 0x7f62a006c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f62a4d1b700 1 --2- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a00ce950 0x7f62a015d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f629f7fe700 1 -- 172.21.15.119:0/2949460381 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949899388 0 0) 0x7f62a00ad5b0 con 0x7f62a00abf40 2024-09-13T16:25:29.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.942+0000 7f629f7fe700 1 -- 172.21.15.119:0/2949460381 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62880035d0 con 0x7f62a00abf40 2024-09-13T16:25:29.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629ffff700 1 -- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a00ce950 msgr2=0x7f62a015d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629ffff700 1 --2- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a00ce950 0x7f62a015d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629ffff700 1 -- 172.21.15.119:0/2949460381 >> v1:172.21.15.119:6789/0 conn(0x7f62a00abf40 legacy=0x7f62a0071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629ffff700 1 -- 172.21.15.119:0/2949460381 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62a01573a0 con 0x7f62a006a350 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a4d1b700 1 --2- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a00ce950 0x7f62a015d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629ffff700 1 --2- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a006a350 0x7f62a006c360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6294009850 tx=0x7f6294009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d62243f96eb3bec2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629f7fe700 1 -- 172.21.15.119:0/2949460381 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6294011830 con 0x7f62a006a350 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629f7fe700 1 -- 172.21.15.119:0/2949460381 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6294011990 con 0x7f62a006a350 2024-09-13T16:25:29.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f629f7fe700 1 -- 172.21.15.119:0/2949460381 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6294011d20 con 0x7f62a006a350 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a006a350 msgr2=0x7f62a006c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a006a350 0x7f62a006c360 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6294009850 tx=0x7f6294009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2949460381 shutdown_connections 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a006a350 0x7f62a006c360 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2949460381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a00ce950 0x7f62a015d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2949460381 >> 172.21.15.119:0/2949460381 conn(0x7f62a00ae750 msgr2=0x7f62a00b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2949460381 shutdown_connections 2024-09-13T16:25:29.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.946+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2949460381 wait complete. 2024-09-13T16:25:29.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 Processor -- start 2024-09-13T16:25:29.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 -- start start 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a00abf40 0x7f62a0149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62a00ce950 0x7f62a009b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a009ab10 0x7f62a015a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01f3cd0 con 0x7f62a00ce950 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62a009c2a0 con 0x7f62a00abf40 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a5d1d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62a01f33a0 con 0x7f62a009ab10 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f629ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62a00ce950 0x7f62a009b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f629ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62a00ce950 0x7f62a009b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53916/0 (socket says 172.21.15.119:53916) 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f629ffff700 1 -- 172.21.15.119:0/2524006073 learned_addr learned my addr 172.21.15.119:0/2524006073 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:29.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a4d1b700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a00abf40 0x7f62a0149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.950+0000 7f62a551c700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a009ab10 0x7f62a015a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a4d1b700 1 -- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a009ab10 msgr2=0x7f62a015a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a4d1b700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a009ab10 0x7f62a015a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a4d1b700 1 -- 172.21.15.119:0/2524006073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62a00ce950 msgr2=0x7f62a009b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a4d1b700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62a00ce950 0x7f62a009b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a4d1b700 1 -- 172.21.15.119:0/2524006073 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62900094b0 con 0x7f62a00abf40 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a551c700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a009ab10 0x7f62a015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a4d1b700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a00abf40 0x7f62a0149ae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f629000e9b0 tx=0x7f629000ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f629d7fa700 1 -- 172.21.15.119:0/2524006073 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629000c640 con 0x7f62a00abf40 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62940094b0 con 0x7f62a00abf40 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f629d7fa700 1 -- 172.21.15.119:0/2524006073 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629000c7a0 con 0x7f62a00abf40 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f629d7fa700 1 -- 172.21.15.119:0/2524006073 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629000cb50 con 0x7f62a00abf40 2024-09-13T16:25:29.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62a01f3960 con 0x7f62a00abf40 2024-09-13T16:25:29.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62a006ae40 con 0x7f62a00abf40 2024-09-13T16:25:29.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f629d7fa700 1 -- 172.21.15.119:0/2524006073 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f629000ccb0 con 0x7f62a00abf40 2024-09-13T16:25:29.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.954+0000 7f629d7fa700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f628c066730 0x7f628c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:29.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.958+0000 7f629d7fa700 1 -- 172.21.15.119:0/2524006073 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6290084770 con 0x7f62a00abf40 2024-09-13T16:25:29.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.958+0000 7f629ffff700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f628c066730 0x7f628c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:29.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.958+0000 7f629ffff700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f628c066730 0x7f628c068bf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f62a01f5730 tx=0x7f6294008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:29.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:29.970+0000 7f629d7fa700 1 -- 172.21.15.119:0/2524006073 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f62900518d0 con 0x7f62a00abf40 2024-09-13T16:25:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f62a015d300 con 0x7f62a00abf40 2024-09-13T16:25:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f629d7fa700 1 -- 172.21.15.119:0/2524006073 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v25) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6290017080 con 0x7f62a00abf40 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f628c066730 msgr2=0x7f628c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f628c066730 0x7f628c068bf0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f62a01f5730 tx=0x7f6294008030 comp rx=0 tx=0).stop 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a00abf40 msgr2=0x7f62a0149ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a00abf40 0x7f62a0149ae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f629000e9b0 tx=0x7f629000ecc0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 shutdown_connections 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f628c066730 0x7f628c068bf0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62a009ab10 0x7f62a015a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62a00ce950 0x7f62a009b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 --2- 172.21.15.119:0/2524006073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62a00abf40 0x7f62a0149ae0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 >> 172.21.15.119:0/2524006073 conn(0x7f62a006a350 msgr2=0x7f62a00b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:30.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 shutdown_connections 2024-09-13T16:25:30.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.158+0000 7f62a5d1d700 1 -- 172.21.15.119:0/2524006073 wait complete. 2024-09-13T16:25:30.164 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:25:30.176 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:25:30.177 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:30.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-09-13T16:25:30.194 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi119.stderr:ceph-fuse[32780]: fuse finished with error 0 and tester_r 0 2024-09-13T16:25:30.196 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:25:30.219 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:25:30.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:30.219 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-09-13T16:25:30.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:30.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.390+0000 7fb8396c6700 1 Processor -- start 2024-09-13T16:25:30.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.390+0000 7fb8396c6700 1 -- start start 2024-09-13T16:25:30.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.390+0000 7fb8396c6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb8341509e0 0x7fb834159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.390+0000 7fb8396c6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb83415a910 0x7fb8340ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.390+0000 7fb8396c6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8340b44e0 con 0x7fb8340cc220 2024-09-13T16:25:30.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.390+0000 7fb8396c6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8340cddf0 con 0x7fb8341509e0 2024-09-13T16:25:30.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.390+0000 7fb8396c6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb834071a50 con 0x7fb83415a910 2024-09-13T16:25:30.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb8341509e0 0x7fb834159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb8341509e0 0x7fb834159cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53920/0 (socket says 172.21.15.119:53920) 2024-09-13T16:25:30.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 -- 172.21.15.119:0/3771828983 learned_addr learned my addr 172.21.15.119:0/3771828983 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:30.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb838ec5700 1 --2- 172.21.15.119:0/3771828983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb83415a910 0x7fb8340ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 -- 172.21.15.119:0/3771828983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb83415a910 msgr2=0x7fb8340ce2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 --2- 172.21.15.119:0/3771828983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb83415a910 0x7fb8340ce2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 -- 172.21.15.119:0/3771828983 >> v1:172.21.15.145:6789/0 conn(0x7fb8340cc220 legacy=0x7fb83415a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8337fe700 1 -- 172.21.15.119:0/3771828983 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74137266 0 0) 0x7fb8340b44e0 con 0x7fb8340cc220 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 -- 172.21.15.119:0/3771828983 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8340cc750 con 0x7fb8341509e0 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb833fff700 1 --2- 172.21.15.119:0/3771828983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834159cc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb824009910 tx=0x7fb824009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2a9b61fdd4377a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8337fe700 1 -- 172.21.15.119:0/3771828983 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824011830 con 0x7fb8341509e0 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8337fe700 1 -- 172.21.15.119:0/3771828983 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb824011990 con 0x7fb8341509e0 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8337fe700 1 -- 172.21.15.119:0/3771828983 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb824011d40 con 0x7fb8341509e0 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 -- 172.21.15.119:0/3771828983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 msgr2=0x7fb834159cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3771828983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834159cc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fb824009910 tx=0x7fb824009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 -- 172.21.15.119:0/3771828983 shutdown_connections 2024-09-13T16:25:30.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3771828983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb83415a910 0x7fb8340ce2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3771828983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834159cc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 -- 172.21.15.119:0/3771828983 >> 172.21.15.119:0/3771828983 conn(0x7fb834070cc0 msgr2=0x7fb8340710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:30.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 -- 172.21.15.119:0/3771828983 shutdown_connections 2024-09-13T16:25:30.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.394+0000 7fb8396c6700 1 -- 172.21.15.119:0/3771828983 wait complete. 2024-09-13T16:25:30.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 Processor -- start 2024-09-13T16:25:30.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 -- start start 2024-09-13T16:25:30.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8340cc220 0x7fb8341499d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb83415a910 0x7fb834147a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8341557e0 con 0x7fb8341509e0 2024-09-13T16:25:30.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb834149f10 con 0x7fb83415a910 2024-09-13T16:25:30.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb8396c6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8341552d0 con 0x7fb8340cc220 2024-09-13T16:25:30.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb82b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb82b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53930/0 (socket says 172.21.15.119:53930) 2024-09-13T16:25:30.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.398+0000 7fb82b7fe700 1 -- 172.21.15.119:0/3645275193 learned_addr learned my addr 172.21.15.119:0/3645275193 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:30.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb838ec5700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb83415a910 0x7fb834147a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb833fff700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8340cc220 0x7fb8341499d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb82b7fe700 1 -- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8340cc220 msgr2=0x7fb8341499d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb82b7fe700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8340cc220 0x7fb8341499d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb82b7fe700 1 -- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb83415a910 msgr2=0x7fb834147a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb82b7fe700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb83415a910 0x7fb834147a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb82b7fe700 1 -- 172.21.15.119:0/3645275193 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb824009570 con 0x7fb8341509e0 2024-09-13T16:25:30.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb82b7fe700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834154d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c750 tx=0x7fb81c00ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:30.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb8317fa700 1 -- 172.21.15.119:0/3645275193 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c0054e0 con 0x7fb8341509e0 2024-09-13T16:25:30.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb8317fa700 1 -- 172.21.15.119:0/3645275193 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb81c0036a0 con 0x7fb8341509e0 2024-09-13T16:25:30.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8341515d0 con 0x7fb8341509e0 2024-09-13T16:25:30.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb8317fa700 1 -- 172.21.15.119:0/3645275193 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb81c003a50 con 0x7fb8341509e0 2024-09-13T16:25:30.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb834155500 con 0x7fb8341509e0 2024-09-13T16:25:30.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.402+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb834151760 con 0x7fb8341509e0 2024-09-13T16:25:30.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.406+0000 7fb8317fa700 1 -- 172.21.15.119:0/3645275193 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb81c003bb0 con 0x7fb8341509e0 2024-09-13T16:25:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.406+0000 7fb8317fa700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb814066730 0x7fb814068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.406+0000 7fb8317fa700 1 -- 172.21.15.119:0/3645275193 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb81c0838a0 con 0x7fb8341509e0 2024-09-13T16:25:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.406+0000 7fb833fff700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb814066730 0x7fb814068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.406+0000 7fb833fff700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb814066730 0x7fb814068bf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb824000c00 tx=0x7fb824008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:30.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.414+0000 7fb8317fa700 1 -- 172.21.15.119:0/3645275193 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb81c050920 con 0x7fb8341509e0 2024-09-13T16:25:30.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.586+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb8340ce2b0 con 0x7fb8341509e0 2024-09-13T16:25:30.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.586+0000 7fb8317fa700 1 -- 172.21.15.119:0/3645275193 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7fb81c003e60 con 0x7fb8341509e0 2024-09-13T16:25:30.591 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:30.591 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:30.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb814066730 msgr2=0x7fb814068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb814066730 0x7fb814068bf0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb824000c00 tx=0x7fb824008030 comp rx=0 tx=0).stop 2024-09-13T16:25:30.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 msgr2=0x7fb834154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834154d90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb81c00c750 tx=0x7fb81c00ca60 comp rx=0 tx=0).stop 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 shutdown_connections 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb814066730 0x7fb814068bf0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb83415a910 0x7fb834147a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb8341509e0 0x7fb834154d90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 --2- 172.21.15.119:0/3645275193 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb8340cc220 0x7fb8341499d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 >> 172.21.15.119:0/3645275193 conn(0x7fb834070cc0 msgr2=0x7fb8340b39e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 shutdown_connections 2024-09-13T16:25:30.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.590+0000 7fb8396c6700 1 -- 172.21.15.119:0/3645275193 wait complete. 2024-09-13T16:25:30.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:30.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 Processor -- start 2024-09-13T16:25:30.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- start start 2024-09-13T16:25:30.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbb3415e840 0x7fbb34070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415ec20 0x7fbb34161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb34071210 con 0x7fbb34164a70 2024-09-13T16:25:30.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb340714f0 con 0x7fbb3415e840 2024-09-13T16:25:30.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb340717d0 con 0x7fbb3415ec20 2024-09-13T16:25:30.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbb3415e840 0x7fbb34070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbb3415e840 0x7fbb34070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53934/0 (socket says 172.21.15.119:53934) 2024-09-13T16:25:30.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 -- 172.21.15.119:0/850274258 learned_addr learned my addr 172.21.15.119:0/850274258 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:30.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39128700 1 --2- 172.21.15.119:0/850274258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415ec20 0x7fbb34161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 -- 172.21.15.119:0/850274258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415ec20 msgr2=0x7fbb34161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:30.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 --2- 172.21.15.119:0/850274258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415ec20 0x7fbb34161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 -- 172.21.15.119:0/850274258 >> v1:172.21.15.145:6789/0 conn(0x7fbb34164a70 legacy=0x7fbb340ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 -- 172.21.15.119:0/850274258 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb34071ab0 con 0x7fbb3415e840 2024-09-13T16:25:30.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb39929700 1 --2- 172.21.15.119:0/850274258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3415e840 0x7fbb34070b80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009910 tx=0x7fbb24009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2b18f11c38cf9d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:30.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb38927700 1 -- 172.21.15.119:0/850274258 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24011830 con 0x7fbb3415e840 2024-09-13T16:25:30.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb38927700 1 -- 172.21.15.119:0/850274258 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb24011990 con 0x7fbb3415e840 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb38927700 1 -- 172.21.15.119:0/850274258 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb24011d40 con 0x7fbb3415e840 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/850274258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3415e840 msgr2=0x7fbb34070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/850274258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3415e840 0x7fbb34070b80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbb24009910 tx=0x7fbb24009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/850274258 shutdown_connections 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/850274258 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415ec20 0x7fbb34161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/850274258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb3415e840 0x7fbb34070b80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/850274258 >> 172.21.15.119:0/850274258 conn(0x7fbb3406a350 msgr2=0x7fbb3406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/850274258 shutdown_connections 2024-09-13T16:25:30.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.718+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/850274258 wait complete. 2024-09-13T16:25:30.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 Processor -- start 2024-09-13T16:25:30.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 -- start start 2024-09-13T16:25:30.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415e840 0x7fbb341fa5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb34164a70 0x7fbb341fab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb341f81b0 0x7fbb340ccf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb340a99d0 con 0x7fbb341f81b0 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb340cde50 con 0x7fbb34164a70 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3bb8d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb340cdb70 con 0x7fbb3415e840 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb341f81b0 0x7fbb340ccf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb341f81b0 0x7fbb340ccf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53948/0 (socket says 172.21.15.119:53948) 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 -- 172.21.15.119:0/4111631228 learned_addr learned my addr 172.21.15.119:0/4111631228 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb39929700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415e840 0x7fbb341fa5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb39128700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb34164a70 0x7fbb341fab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 -- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415e840 msgr2=0x7fbb341fa5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415e840 0x7fbb341fa5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 -- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb34164a70 msgr2=0x7fbb341fab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb34164a70 0x7fbb341fab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.722+0000 7fbb3a12a700 1 -- 172.21.15.119:0/4111631228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb24009570 con 0x7fbb341f81b0 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb3a12a700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb341f81b0 0x7fbb340ccf90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000c480 tx=0x7fbb3000c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb2a7fc700 1 -- 172.21.15.119:0/4111631228 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb300052d0 con 0x7fbb341f81b0 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb2a7fc700 1 -- 172.21.15.119:0/4111631228 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3000ce90 con 0x7fbb341f81b0 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb3415f2f0 con 0x7fbb341f81b0 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb2a7fc700 1 -- 172.21.15.119:0/4111631228 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb300036a0 con 0x7fbb341f81b0 2024-09-13T16:25:30.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb3415f8b0 con 0x7fbb341f81b0 2024-09-13T16:25:30.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb3415fad0 con 0x7fbb341f81b0 2024-09-13T16:25:30.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb2a7fc700 1 -- 172.21.15.119:0/4111631228 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbb30003800 con 0x7fbb341f81b0 2024-09-13T16:25:30.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb2a7fc700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb18066730 0x7fbb18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:30.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb2a7fc700 1 -- 172.21.15.119:0/4111631228 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbb30083770 con 0x7fbb341f81b0 2024-09-13T16:25:30.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.726+0000 7fbb39929700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb18066730 0x7fbb18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:30.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.730+0000 7fbb39929700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb18066730 0x7fbb18068bf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb24000c00 tx=0x7fbb24008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:30.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.734+0000 7fbb2a7fc700 1 -- 172.21.15.119:0/4111631228 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbb300507f0 con 0x7fbb341f81b0 2024-09-13T16:25:30.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.906+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb34067760 con 0x7fbb341f81b0 2024-09-13T16:25:30.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.906+0000 7fbb2a7fc700 1 -- 172.21.15.119:0/4111631228 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7fbb30053e10 con 0x7fbb341f81b0 2024-09-13T16:25:30.910 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:30.911 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb18066730 msgr2=0x7fbb18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb18066730 0x7fbb18068bf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb24000c00 tx=0x7fbb24008030 comp rx=0 tx=0).stop 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb341f81b0 msgr2=0x7fbb340ccf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb341f81b0 0x7fbb340ccf90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fbb3000c480 tx=0x7fbb3000c790 comp rx=0 tx=0).stop 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 shutdown_connections 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb18066730 0x7fbb18068bf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb341f81b0 0x7fbb340ccf90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb34164a70 0x7fbb341fab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 --2- 172.21.15.119:0/4111631228 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb3415e840 0x7fbb341fa5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:30.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 >> 172.21.15.119:0/4111631228 conn(0x7fbb3406a350 msgr2=0x7fbb34066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:30.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 shutdown_connections 2024-09-13T16:25:30.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:30.910+0000 7fbb3bb8d700 1 -- 172.21.15.119:0/4111631228 wait complete. 2024-09-13T16:25:30.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:25:31.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f335a53b700 1 Processor -- start 2024-09-13T16:25:31.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f335a53b700 1 -- start start 2024-09-13T16:25:31.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f335a53b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3354144180 0x7f335406c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:31.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f335a53b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540cc010 0x7f33540cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:31.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f335a53b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f335406cb40 con 0x7f33540ce000 2024-09-13T16:25:31.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f335a53b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33540b1c50 con 0x7f3354144180 2024-09-13T16:25:31.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f335a53b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335406ce20 con 0x7f33540cc010 2024-09-13T16:25:31.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3358d38700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540cc010 0x7f33540cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3359539700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3354144180 0x7f335406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3358d38700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540cc010 0x7f33540cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:43214/0 (socket says 172.21.15.119:43214) 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3359d3a700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f33540ce000 0x7f335415e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:57612/0 (socket says 172.21.15.119:57612) 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3358d38700 1 -- 172.21.15.119:0/4027001712 learned_addr learned my addr 172.21.15.119:0/4027001712 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3358d38700 1 -- 172.21.15.119:0/4027001712 >> v2:172.21.15.145:3300/0 conn(0x7f3354144180 msgr2=0x7f335406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3358d38700 1 --2- 172.21.15.119:0/4027001712 >> v2:172.21.15.145:3300/0 conn(0x7f3354144180 0x7f335406c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3358d38700 1 -- 172.21.15.119:0/4027001712 >> v1:172.21.15.145:6789/0 conn(0x7f33540ce000 legacy=0x7f335415e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f334bfff700 1 -- 172.21.15.119:0/4027001712 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214097030 0 0) 0x7f335406cb40 con 0x7f33540ce000 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.042+0000 7f3358d38700 1 -- 172.21.15.119:0/4027001712 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f335406d100 con 0x7f33540cc010 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f3358d38700 1 --2- 172.21.15.119:0/4027001712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540cc010 0x7f33540cdac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f334c009e50 tx=0x7f334c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a4329edf5df4563 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f334bfff700 1 -- 172.21.15.119:0/4027001712 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c00e070 con 0x7f33540cc010 2024-09-13T16:25:31.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f334bfff700 1 -- 172.21.15.119:0/4027001712 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c00b910 con 0x7f33540cc010 2024-09-13T16:25:31.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f334bfff700 1 -- 172.21.15.119:0/4027001712 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c00bcc0 con 0x7f33540cc010 2024-09-13T16:25:31.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 -- 172.21.15.119:0/4027001712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540cc010 msgr2=0x7f33540cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:31.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 --2- 172.21.15.119:0/4027001712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540cc010 0x7f33540cdac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f334c009e50 tx=0x7f334c00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:31.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 -- 172.21.15.119:0/4027001712 shutdown_connections 2024-09-13T16:25:31.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 --2- 172.21.15.119:0/4027001712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540cc010 0x7f33540cdac0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:31.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 --2- 172.21.15.119:0/4027001712 >> v2:172.21.15.145:3300/0 conn(0x7f3354144180 0x7f335406c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:31.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 -- 172.21.15.119:0/4027001712 >> 172.21.15.119:0/4027001712 conn(0x7f33540b0980 msgr2=0x7f33540b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:31.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 -- 172.21.15.119:0/4027001712 shutdown_connections 2024-09-13T16:25:31.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 -- 172.21.15.119:0/4027001712 wait complete. 2024-09-13T16:25:31.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 Processor -- start 2024-09-13T16:25:31.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.046+0000 7f335a53b700 1 -- start start 2024-09-13T16:25:31.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33540cc010 0x7f335412f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:31.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540ce000 0x7f3354159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:31.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3354144180 0x7f335412d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:31.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335412f750 con 0x7f33540cc010 2024-09-13T16:25:31.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335415a2a0 con 0x7f3354144180 2024-09-13T16:25:31.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335412e170 con 0x7f33540ce000 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3359539700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33540cc010 0x7f335412f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540ce000 0x7f3354159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3359d3a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3354144180 0x7f335412d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540ce000 0x7f3354159d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:43220/0 (socket says 172.21.15.119:43220) 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3359539700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33540cc010 0x7f335412f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53964/0 (socket says 172.21.15.119:53964) 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3359d3a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3354144180 0x7f335412d3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:33410/0 (socket says 172.21.15.119:33410) 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 -- 172.21.15.119:0/1638764224 learned_addr learned my addr 172.21.15.119:0/1638764224 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 -- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3354144180 msgr2=0x7f335412d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3354144180 0x7f335412d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 -- 172.21.15.119:0/1638764224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33540cc010 msgr2=0x7f335412f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33540cc010 0x7f335412f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 -- 172.21.15.119:0/1638764224 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c009b30 con 0x7f33540ce000 2024-09-13T16:25:31.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3358d38700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540ce000 0x7f3354159d60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f334c00b370 tx=0x7f334c004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3349ffb700 1 -- 172.21.15.119:0/1638764224 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c0048c0 con 0x7f33540ce000 2024-09-13T16:25:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3349ffb700 1 -- 172.21.15.119:0/1638764224 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f334c0092e0 con 0x7f33540ce000 2024-09-13T16:25:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f335412e450 con 0x7f33540ce000 2024-09-13T16:25:31.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f3349ffb700 1 -- 172.21.15.119:0/1638764224 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334c01c660 con 0x7f33540ce000 2024-09-13T16:25:31.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.050+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f335412ea10 con 0x7f33540ce000 2024-09-13T16:25:31.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.054+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33540cdac0 con 0x7f33540ce000 2024-09-13T16:25:31.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.054+0000 7f3349ffb700 1 -- 172.21.15.119:0/1638764224 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f334c01c7c0 con 0x7f33540ce000 2024-09-13T16:25:31.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.054+0000 7f3349ffb700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3340066730 0x7f3340068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:31.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.054+0000 7f3349ffb700 1 -- 172.21.15.119:0/1638764224 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f334c085db0 con 0x7f33540ce000 2024-09-13T16:25:31.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.054+0000 7f3359539700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3340066730 0x7f3340068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:31.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.054+0000 7f3359539700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3340066730 0x7f3340068bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3350009b50 tx=0x7f3350009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:31.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.062+0000 7f3349ffb700 1 -- 172.21.15.119:0/1638764224 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f334c052fa0 con 0x7f33540ce000 2024-09-13T16:25:31.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:31.270+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f335415a8d0 con 0x7f33540ce000 2024-09-13T16:25:32.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.006+0000 7f3349ffb700 1 -- 172.21.15.119:0/1638764224 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v8) v1 ==== 118+0+0 (secure 0 0 0) 0x7f334c0565c0 con 0x7f33540ce000 2024-09-13T16:25:32.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3340066730 msgr2=0x7f3340068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:32.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3340066730 0x7f3340068bf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3350009b50 tx=0x7f3350009240 comp rx=0 tx=0).stop 2024-09-13T16:25:32.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540ce000 msgr2=0x7f3354159d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:32.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540ce000 0x7f3354159d60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f334c00b370 tx=0x7f334c004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 shutdown_connections 2024-09-13T16:25:32.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3340066730 0x7f3340068bf0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3354144180 0x7f335412d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33540ce000 0x7f3354159d60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 --2- 172.21.15.119:0/1638764224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33540cc010 0x7f335412f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 >> 172.21.15.119:0/1638764224 conn(0x7f33540b0980 msgr2=0x7f33540b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:32.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.010+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 shutdown_connections 2024-09-13T16:25:32.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.014+0000 7f335a53b700 1 -- 172.21.15.119:0/1638764224 wait complete. 2024-09-13T16:25:32.031 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']} 2024-09-13T16:25:32.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:25:32.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.154+0000 7f19f9815700 1 Processor -- start 2024-09-13T16:25:32.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.154+0000 7f19f9815700 1 -- start start 2024-09-13T16:25:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.154+0000 7f19f9815700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f19f40ad9a0 0x7f19f4159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.154+0000 7f19f9815700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 0x7f19f406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.154+0000 7f19f9815700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19f40b1dd0 con 0x7f19f406c890 2024-09-13T16:25:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.154+0000 7f19f9815700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19f40b2fd0 con 0x7f19f40ad9a0 2024-09-13T16:25:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.154+0000 7f19f9815700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f415a520 con 0x7f19f40b1860 2024-09-13T16:25:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f19f406c890 0x7f19f406bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42000/0 (socket says 172.21.15.119:42000) 2024-09-13T16:25:32.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 -- 172.21.15.119:0/3689775312 learned_addr learned my addr 172.21.15.119:0/3689775312 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:32.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f8813700 1 --2- 172.21.15.119:0/3689775312 >> v2:172.21.15.145:3300/0 conn(0x7f19f40ad9a0 0x7f19f4159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f3fff700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 0x7f19f406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40103519 0 0) 0x7f19f40b1dd0 con 0x7f19f406c890 2024-09-13T16:25:32.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19e00035d0 con 0x7f19f406c890 2024-09-13T16:25:32.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1305972494 0 0) 0x7f19e00035d0 con 0x7f19f406c890 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 msgr2=0x7f19f406c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 0x7f19f406c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 >> v2:172.21.15.145:3300/0 conn(0x7f19f40ad9a0 msgr2=0x7f19f4159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 --2- 172.21.15.119:0/3689775312 >> v2:172.21.15.145:3300/0 conn(0x7f19f40ad9a0 0x7f19f4159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f40b20b0 con 0x7f19f406c890 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f19ec001670 con 0x7f19f406c890 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 >> v1:172.21.15.119:6789/0 conn(0x7f19f406c890 legacy=0x7f19f406bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 0x7f19e0003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0003fa0 0x7f19e0004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e0004db0 0x7f19e0005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e000dc40 con 0x7f19e0004db0 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e000dd80 con 0x7f19e0003fa0 2024-09-13T16:25:32.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e000df10 con 0x7f19f40b1860 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e0004db0 0x7f19e0005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f8813700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 0x7f19e0003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f3fff700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0003fa0 0x7f19e0004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 -- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 msgr2=0x7f19e0003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f40b1860 0x7f19e0003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 -- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0003fa0 msgr2=0x7f19e0004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0003fa0 0x7f19e0004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 -- 172.21.15.119:0/3689775312 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e00035d0 con 0x7f19e0004db0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e0004db0 0x7f19e0005230 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c350 tx=0x7f19e00058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9747642d447620 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec00ce50 con 0x7f19e0004db0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec005980 con 0x7f19e0004db0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f37fe700 1 -- 172.21.15.119:0/3689775312 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec005d30 con 0x7f19e0004db0 2024-09-13T16:25:32.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9815700 1 -- 172.21.15.119:0/3689775312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e0004db0 msgr2=0x7f19e0005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:32.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9815700 1 --2- 172.21.15.119:0/3689775312 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19e0004db0 0x7f19e0005230 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f19ec00c350 tx=0x7f19e00058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9014700 1 -- 172.21.15.119:0/3689775312 reap_dead start 2024-09-13T16:25:32.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9815700 1 -- 172.21.15.119:0/3689775312 shutdown_connections 2024-09-13T16:25:32.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9815700 1 -- 172.21.15.119:0/3689775312 >> 172.21.15.119:0/3689775312 conn(0x7f19f40b0980 msgr2=0x7f19f40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:32.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9815700 1 -- 172.21.15.119:0/3689775312 shutdown_connections 2024-09-13T16:25:32.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.158+0000 7f19f9815700 1 -- 172.21.15.119:0/3689775312 wait complete. 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 Processor -- start 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 -- start start 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0004db0 0x7f19f40cc910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f415b490 0x7f19f415b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19f414cba0 0x7f19f414d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f415be50 con 0x7f19f414cba0 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40cd180 con 0x7f19e0004db0 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9815700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19f40cd460 con 0x7f19f415b490 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9014700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19f414cba0 0x7f19f414d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f8813700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0004db0 0x7f19f40cc910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f8813700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0004db0 0x7f19f40cc910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:33422/0 (socket says 172.21.15.119:33422) 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f8813700 1 -- 172.21.15.119:0/299317793 learned_addr learned my addr 172.21.15.119:0/299317793 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f3fff700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f415b490 0x7f19f415b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9014700 1 -- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f415b490 msgr2=0x7f19f415b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9014700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f415b490 0x7f19f415b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9014700 1 -- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0004db0 msgr2=0x7f19f40cc910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9014700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0004db0 0x7f19f40cc910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.162+0000 7f19f9014700 1 -- 172.21.15.119:0/299317793 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19f40cd740 con 0x7f19f414cba0 2024-09-13T16:25:32.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f9014700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19f414cba0 0x7f19f414d020 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f19ec014000 tx=0x7f19ec001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:32.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f17fa700 1 -- 172.21.15.119:0/299317793 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec022070 con 0x7f19f414cba0 2024-09-13T16:25:32.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f17fa700 1 -- 172.21.15.119:0/299317793 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec005e20 con 0x7f19f414cba0 2024-09-13T16:25:32.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f40cda20 con 0x7f19f414cba0 2024-09-13T16:25:32.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f17fa700 1 -- 172.21.15.119:0/299317793 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19ec004aa0 con 0x7f19f414cba0 2024-09-13T16:25:32.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f40cce50 con 0x7f19f414cba0 2024-09-13T16:25:32.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f40cdf40 con 0x7f19f414cba0 2024-09-13T16:25:32.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.166+0000 7f19f17fa700 1 -- 172.21.15.119:0/299317793 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f19ec004c00 con 0x7f19f414cba0 2024-09-13T16:25:32.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.170+0000 7f19f17fa700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f19d4066760 0x7f19d4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:32.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.170+0000 7f19f17fa700 1 -- 172.21.15.119:0/299317793 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f19ec084700 con 0x7f19f414cba0 2024-09-13T16:25:32.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.170+0000 7f19f8813700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f19d4066760 0x7f19d4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:32.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.170+0000 7f19f8813700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f19d4066760 0x7f19d4068c20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f19f41547c0 tx=0x7f19dc009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:32.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.174+0000 7f19f17fa700 1 -- 172.21.15.119:0/299317793 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f19f40cdf40 con 0x7f19f414cba0 2024-09-13T16:25:32.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:32.386+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f19f40cdf40 con 0x7f19f414cba0 2024-09-13T16:25:33.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.010+0000 7f19f17fa700 1 -- 172.21.15.119:0/299317793 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v9) v1 ==== 122+0+0 (secure 0 0 0) 0x7f19f40cdf40 con 0x7f19f414cba0 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f19d4066760 msgr2=0x7f19d4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f19d4066760 0x7f19d4068c20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f19f41547c0 tx=0x7f19dc009240 comp rx=0 tx=0).stop 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19f414cba0 msgr2=0x7f19f414d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19f414cba0 0x7f19f414d020 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f19ec014000 tx=0x7f19ec001a00 comp rx=0 tx=0).stop 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 shutdown_connections 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f19d4066760 0x7f19d4068c20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19f414cba0 0x7f19f414d020 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f19f415b490 0x7f19f415b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 --2- 172.21.15.119:0/299317793 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19e0004db0 0x7f19f40cc910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 >> 172.21.15.119:0/299317793 conn(0x7f19f40b0980 msgr2=0x7f19f406c0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:33.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 shutdown_connections 2024-09-13T16:25:33.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.014+0000 7f19f9815700 1 -- 172.21.15.119:0/299317793 wait complete. 2024-09-13T16:25:33.034 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:25:33.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:33.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.150+0000 7f878728b700 1 Processor -- start 2024-09-13T16:25:33.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.150+0000 7f878728b700 1 -- start start 2024-09-13T16:25:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.150+0000 7f878728b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f878006d3e0 0x7f878006ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.150+0000 7f878728b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 0x7f878006c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878015aa80 con 0x7f87801576c0 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87800d0020 con 0x7f878006d3e0 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8780157aa0 con 0x7f87800cfc20 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f87801576c0 0x7f878006ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33626/0 (socket says 172.21.15.119:33626) 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 -- 172.21.15.119:0/1068901535 learned_addr learned my addr 172.21.15.119:0/1068901535 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8784826700 1 --2- 172.21.15.119:0/1068901535 >> v2:172.21.15.145:3300/0 conn(0x7f878006d3e0 0x7f878006ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785828700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 0x7f878006c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938721126 0 0) 0x7f878015aa80 con 0x7f87801576c0 2024-09-13T16:25:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876c0035d0 con 0x7f87801576c0 2024-09-13T16:25:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635888660 0 0) 0x7f876c0035d0 con 0x7f87801576c0 2024-09-13T16:25:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 msgr2=0x7f878006c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 0x7f878006c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 >> v2:172.21.15.145:3300/0 conn(0x7f878006d3e0 msgr2=0x7f878006ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 --2- 172.21.15.119:0/1068901535 >> v2:172.21.15.145:3300/0 conn(0x7f878006d3e0 0x7f878006ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87800cf790 con 0x7f87801576c0 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f877c001670 con 0x7f87801576c0 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 >> v1:172.21.15.119:6789/0 conn(0x7f87801576c0 legacy=0x7f878006ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f876c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 0x7f876c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c004db0 0x7f876c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87800d0020 con 0x7f876c003a20 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780157aa0 con 0x7f876c004db0 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876c00ba30 con 0x7f87800cfc20 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f876c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8784826700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 0x7f876c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785828700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c004db0 0x7f876c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 -- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 msgr2=0x7f876c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800cfc20 0x7f876c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 -- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c004db0 msgr2=0x7f876c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c004db0 0x7f876c005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 -- 172.21.15.119:0/1068901535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f876c0035d0 con 0x7f876c003a20 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785828700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876c004db0 0x7f876c005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785027700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f876c003e40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f877c00c270 tx=0x7f876c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189797632e4a0824 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c00ce10 con 0x7f876c003a20 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c004920 con 0x7f876c003a20 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8777fff700 1 -- 172.21.15.119:0/1068901535 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c004cb0 con 0x7f876c003a20 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- 172.21.15.119:0/1068901535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 msgr2=0x7f876c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 --2- 172.21.15.119:0/1068901535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f876c003e40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f877c00c270 tx=0x7f876c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f8785828700 1 -- 172.21.15.119:0/1068901535 reap_dead start 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- 172.21.15.119:0/1068901535 shutdown_connections 2024-09-13T16:25:33.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- 172.21.15.119:0/1068901535 >> 172.21.15.119:0/1068901535 conn(0x7f87800b3fd0 msgr2=0x7f87800b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:33.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- 172.21.15.119:0/1068901535 shutdown_connections 2024-09-13T16:25:33.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.154+0000 7f878728b700 1 -- 172.21.15.119:0/1068901535 wait complete. 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 Processor -- start 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 -- start start 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f87801ed7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800b0d90 0x7f87800b1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87800aed40 0x7f87800af1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87800af730 con 0x7f876c003a20 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8780154520 con 0x7f87800aed40 2024-09-13T16:25:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8780145d30 con 0x7f87800b0d90 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f87801ed7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f87801ed7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51966/0 (socket says 172.21.15.119:51966) 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 -- 172.21.15.119:0/509447435 learned_addr learned my addr 172.21.15.119:0/509447435 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8784826700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800b0d90 0x7f87800b1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785828700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87800aed40 0x7f87800af1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 -- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800b0d90 msgr2=0x7f87800b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800b0d90 0x7f87800b1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 -- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87800aed40 msgr2=0x7f87800af1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87800aed40 0x7f87800af1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 -- 172.21.15.119:0/509447435 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8780153910 con 0x7f876c003a20 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8784826700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800b0d90 0x7f87800b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8785027700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f87801ed7e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f877c015040 tx=0x7f878015a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8775ffb700 1 -- 172.21.15.119:0/509447435 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c004ea0 con 0x7f876c003a20 2024-09-13T16:25:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f8775ffb700 1 -- 172.21.15.119:0/509447435 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f877c0060d0 con 0x7f876c003a20 2024-09-13T16:25:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.158+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8780153bf0 con 0x7f876c003a20 2024-09-13T16:25:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.162+0000 7f8775ffb700 1 -- 172.21.15.119:0/509447435 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f877c006480 con 0x7f876c003a20 2024-09-13T16:25:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.162+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87801541b0 con 0x7f876c003a20 2024-09-13T16:25:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.162+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8780067b90 con 0x7f876c003a20 2024-09-13T16:25:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.162+0000 7f8775ffb700 1 -- 172.21.15.119:0/509447435 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f877c0065e0 con 0x7f876c003a20 2024-09-13T16:25:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.162+0000 7f8775ffb700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f87640667b0 0x7f8764068c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.162+0000 7f8775ffb700 1 -- 172.21.15.119:0/509447435 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f877c085a00 con 0x7f876c003a20 2024-09-13T16:25:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.162+0000 7f8784826700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f87640667b0 0x7f8764068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.166+0000 7f8784826700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f87640667b0 0x7f8764068c70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8780154f70 tx=0x7f8770008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.170+0000 7f8775ffb700 1 -- 172.21.15.119:0/509447435 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f877c052cb0 con 0x7f876c003a20 2024-09-13T16:25:33.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.378+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f878006b5b0 con 0x7f876c003a20 2024-09-13T16:25:33.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.378+0000 7f8775ffb700 1 -- 172.21.15.119:0/509447435 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f877c0562d0 con 0x7f876c003a20 2024-09-13T16:25:33.385 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:33.385 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:25:33.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f87640667b0 msgr2=0x7f8764068c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f87640667b0 0x7f8764068c70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f8780154f70 tx=0x7f8770008040 comp rx=0 tx=0).stop 2024-09-13T16:25:33.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 msgr2=0x7f87801ed7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f87801ed7e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f877c015040 tx=0x7f878015a000 comp rx=0 tx=0).stop 2024-09-13T16:25:33.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 shutdown_connections 2024-09-13T16:25:33.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f87640667b0 0x7f8764068c70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87800aed40 0x7f87800af1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87800b0d90 0x7f87800b1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 --2- 172.21.15.119:0/509447435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876c003a20 0x7f87801ed7e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 >> 172.21.15.119:0/509447435 conn(0x7f87800b3fd0 msgr2=0x7f87800997c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:33.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 shutdown_connections 2024-09-13T16:25:33.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.382+0000 7f878728b700 1 -- 172.21.15.119:0/509447435 wait complete. 2024-09-13T16:25:33.390 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:25:33.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:33.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 Processor -- start 2024-09-13T16:25:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 -- start start 2024-09-13T16:25:33.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ab015a8e0 0x7f1ab00cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00ceef0 0x7f1ab00cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab00cbdc0 con 0x7f1ab00afb90 2024-09-13T16:25:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab006e650 con 0x7f1ab015a8e0 2024-09-13T16:25:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab00cf860 con 0x7f1ab00ceef0 2024-09-13T16:25:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab4808700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1ab00afb90 0x7f1ab006df40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33636/0 (socket says 172.21.15.119:33636) 2024-09-13T16:25:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab4808700 1 -- 172.21.15.119:0/4156858159 learned_addr learned my addr 172.21.15.119:0/4156858159 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab5009700 1 --2- 172.21.15.119:0/4156858159 >> v2:172.21.15.145:3300/0 conn(0x7f1ab015a8e0 0x7f1ab00cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab580a700 1 --2- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00ceef0 0x7f1ab00cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1aa7fff700 1 -- 172.21.15.119:0/4156858159 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 413128334 0 0) 0x7f1ab00cbdc0 con 0x7f1ab00afb90 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1aa7fff700 1 -- 172.21.15.119:0/4156858159 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a9c0035d0 con 0x7f1ab00afb90 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab5009700 1 -- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00ceef0 msgr2=0x7f1ab00cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab5009700 1 --2- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00ceef0 0x7f1ab00cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab5009700 1 -- 172.21.15.119:0/4156858159 >> v1:172.21.15.119:6789/0 conn(0x7f1ab00afb90 legacy=0x7f1ab006df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab5009700 1 -- 172.21.15.119:0/4156858159 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab00b00e0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab5009700 1 --2- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00cdc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009850 tx=0x7f1aac009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=562be49fdeaf553d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1aa7fff700 1 -- 172.21.15.119:0/4156858159 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac011830 con 0x7f1ab015a8e0 2024-09-13T16:25:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1aa7fff700 1 -- 172.21.15.119:0/4156858159 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aac011990 con 0x7f1ab015a8e0 2024-09-13T16:25:33.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1aa7fff700 1 -- 172.21.15.119:0/4156858159 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aac011d00 con 0x7f1ab015a8e0 2024-09-13T16:25:33.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 -- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab015a8e0 msgr2=0x7f1ab00cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 --2- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00cdc90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1aac009850 tx=0x7f1aac009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:33.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 -- 172.21.15.119:0/4156858159 shutdown_connections 2024-09-13T16:25:33.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 --2- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00ceef0 0x7f1ab00cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 --2- 172.21.15.119:0/4156858159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00cdc90 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.522+0000 7f1ab600b700 1 -- 172.21.15.119:0/4156858159 >> 172.21.15.119:0/4156858159 conn(0x7f1ab00b25e0 msgr2=0x7f1ab00b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:33.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 -- 172.21.15.119:0/4156858159 shutdown_connections 2024-09-13T16:25:33.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 -- 172.21.15.119:0/4156858159 wait complete. 2024-09-13T16:25:33.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 Processor -- start 2024-09-13T16:25:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 -- start start 2024-09-13T16:25:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00afb90 0x7f1ab014f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab00ceef0 0x7f1ab00b61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00aea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab01546b0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab01554f0 con 0x7f1ab00ceef0 2024-09-13T16:25:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab600b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab00b66f0 con 0x7f1ab00afb90 2024-09-13T16:25:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab580a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00aea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab580a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00aea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51978/0 (socket says 172.21.15.119:51978) 2024-09-13T16:25:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab580a700 1 -- 172.21.15.119:0/2707999236 learned_addr learned my addr 172.21.15.119:0/2707999236 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab4808700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab00ceef0 0x7f1ab00b61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.526+0000 7f1ab5009700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00afb90 0x7f1ab014f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab580a700 1 -- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00afb90 msgr2=0x7f1ab014f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab580a700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00afb90 0x7f1ab014f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab580a700 1 -- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab00ceef0 msgr2=0x7f1ab00b61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab580a700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab00ceef0 0x7f1ab00b61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab580a700 1 -- 172.21.15.119:0/2707999236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aac0094b0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab580a700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00aea00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800efc0 tx=0x7f1aa800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1aa5ffb700 1 -- 172.21.15.119:0/2707999236 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa8009ea0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1aa5ffb700 1 -- 172.21.15.119:0/2707999236 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa800f040 con 0x7f1ab015a8e0 2024-09-13T16:25:33.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab01548e0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1aa5ffb700 1 -- 172.21.15.119:0/2707999236 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aa800ced0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab0154ea0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab01556a0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1aa5ffb700 1 -- 172.21.15.119:0/2707999236 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1aa8014480 con 0x7f1ab015a8e0 2024-09-13T16:25:33.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1aa5ffb700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1a94066730 0x7f1a94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.530+0000 7f1aa5ffb700 1 -- 172.21.15.119:0/2707999236 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1aa8085060 con 0x7f1ab015a8e0 2024-09-13T16:25:33.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.534+0000 7f1ab5009700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1a94066730 0x7f1a94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.534+0000 7f1ab5009700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1a94066730 0x7f1a94068bf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0090b0 tx=0x7f1aac01c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.542+0000 7f1aa5ffb700 1 -- 172.21.15.119:0/2707999236 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1ab01556a0 con 0x7f1ab015a8e0 2024-09-13T16:25:33.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.713+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ab0155660 con 0x7f1ab015a8e0 2024-09-13T16:25:33.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.717+0000 7f1aa5ffb700 1 -- 172.21.15.119:0/2707999236 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f1aa8018090 con 0x7f1ab015a8e0 2024-09-13T16:25:33.722 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:33.722 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1a94066730 msgr2=0x7f1a94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1a94066730 0x7f1a94068bf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1aac0090b0 tx=0x7f1aac01c040 comp rx=0 tx=0).stop 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ab015a8e0 msgr2=0x7f1ab00aea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00aea00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1aa800efc0 tx=0x7f1aa800c3a0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 shutdown_connections 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1a94066730 0x7f1a94068bf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ab015a8e0 0x7f1ab00aea00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ab00ceef0 0x7f1ab00b61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 --2- 172.21.15.119:0/2707999236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ab00afb90 0x7f1ab014f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 >> 172.21.15.119:0/2707999236 conn(0x7f1ab00b25e0 msgr2=0x7f1ab00cc250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:33.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 shutdown_connections 2024-09-13T16:25:33.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.721+0000 7f1ab600b700 1 -- 172.21.15.119:0/2707999236 wait complete. 2024-09-13T16:25:33.737 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:25:33.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:25:33.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a28932700 1 Processor -- start 2024-09-13T16:25:33.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a28932700 1 -- start start 2024-09-13T16:25:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a28932700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0a2415cbe0 0x7f0a24161390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a28932700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0a240d3420 0x7f0a2406ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a28932700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a240cdae0 con 0x7f0a240d1030 2024-09-13T16:25:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a28932700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a240d0760 con 0x7f0a240d3420 2024-09-13T16:25:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a28932700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a240d0a40 con 0x7f0a2415cbe0 2024-09-13T16:25:33.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a23fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0a240d1030 0x7f0a240cdf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33638/0 (socket says 172.21.15.119:33638) 2024-09-13T16:25:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a23fff700 1 -- 172.21.15.119:0/3226818653 learned_addr learned my addr 172.21.15.119:0/3226818653 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a1affd700 1 --2- 172.21.15.119:0/3226818653 >> v2:172.21.15.119:3300/0 conn(0x7f0a240d3420 0x7f0a2406ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a237fe700 1 --2- 172.21.15.119:0/3226818653 >> v2:172.21.15.145:3300/0 conn(0x7f0a2415cbe0 0x7f0a24161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a22ffd700 1 -- 172.21.15.119:0/3226818653 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1136556238 0 0) 0x7f0a240cdae0 con 0x7f0a240d1030 2024-09-13T16:25:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a22ffd700 1 -- 172.21.15.119:0/3226818653 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a100035d0 con 0x7f0a240d1030 2024-09-13T16:25:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a1affd700 1 -- 172.21.15.119:0/3226818653 >> v2:172.21.15.145:3300/0 conn(0x7f0a2415cbe0 msgr2=0x7f0a24161390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a1affd700 1 --2- 172.21.15.119:0/3226818653 >> v2:172.21.15.145:3300/0 conn(0x7f0a2415cbe0 0x7f0a24161390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a1affd700 1 -- 172.21.15.119:0/3226818653 >> v1:172.21.15.119:6789/0 conn(0x7f0a240d1030 legacy=0x7f0a240cdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a1affd700 1 -- 172.21.15.119:0/3226818653 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a240d0d20 con 0x7f0a240d3420 2024-09-13T16:25:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a237fe700 1 --2- 172.21.15.119:0/3226818653 >> v2:172.21.15.145:3300/0 conn(0x7f0a2415cbe0 0x7f0a24161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:33.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a1affd700 1 --2- 172.21.15.119:0/3226818653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d3420 0x7f0a2406ab00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a2408bdf0 tx=0x7f0a0c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9502e7060f113e20 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a22ffd700 1 -- 172.21.15.119:0/3226818653 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c011830 con 0x7f0a240d3420 2024-09-13T16:25:33.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a22ffd700 1 -- 172.21.15.119:0/3226818653 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0c011990 con 0x7f0a240d3420 2024-09-13T16:25:33.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.853+0000 7f0a22ffd700 1 -- 172.21.15.119:0/3226818653 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0c011d20 con 0x7f0a240d3420 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 -- 172.21.15.119:0/3226818653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d3420 msgr2=0x7f0a2406ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 --2- 172.21.15.119:0/3226818653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d3420 0x7f0a2406ab00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0a2408bdf0 tx=0x7f0a0c009b30 comp rx=0 tx=0).stop 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 -- 172.21.15.119:0/3226818653 shutdown_connections 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 --2- 172.21.15.119:0/3226818653 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d3420 0x7f0a2406ab00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 --2- 172.21.15.119:0/3226818653 >> v2:172.21.15.145:3300/0 conn(0x7f0a2415cbe0 0x7f0a24161390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 -- 172.21.15.119:0/3226818653 >> 172.21.15.119:0/3226818653 conn(0x7f0a2408c080 msgr2=0x7f0a240d2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 -- 172.21.15.119:0/3226818653 shutdown_connections 2024-09-13T16:25:33.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 -- 172.21.15.119:0/3226818653 wait complete. 2024-09-13T16:25:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 Processor -- start 2024-09-13T16:25:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.857+0000 7f0a28932700 1 -- start start 2024-09-13T16:25:33.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d1030 0x7f0a240c6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a240d3420 0x7f0a24151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a2415cbe0 0x7f0a240b0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a241f2450 con 0x7f0a240d1030 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240c7a50 con 0x7f0a2415cbe0 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a240c6c00 con 0x7f0a240d3420 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d1030 0x7f0a240c6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d1030 0x7f0a240c6620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51988/0 (socket says 172.21.15.119:51988) 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 -- 172.21.15.119:0/3646839897 learned_addr learned my addr 172.21.15.119:0/3646839897 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a23fff700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a2415cbe0 0x7f0a240b0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a1affd700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a240d3420 0x7f0a24151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 -- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a240d3420 msgr2=0x7f0a24151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a240d3420 0x7f0a24151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 -- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a2415cbe0 msgr2=0x7f0a240b0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a2415cbe0 0x7f0a240b0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:33.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 -- 172.21.15.119:0/3646839897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a140094b0 con 0x7f0a240d1030 2024-09-13T16:25:33.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a1affd700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a240d3420 0x7f0a24151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a237fe700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d1030 0x7f0a240c6620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400ec10 tx=0x7f0a14009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a20ff9700 1 -- 172.21.15.119:0/3646839897 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400c940 con 0x7f0a240d1030 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a20ff9700 1 -- 172.21.15.119:0/3646839897 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a1400caa0 con 0x7f0a240d1030 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0c0094b0 con 0x7f0a240d1030 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a20ff9700 1 -- 172.21.15.119:0/3646839897 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400ce30 con 0x7f0a240d1030 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a240c71c0 con 0x7f0a240d1030 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.861+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a240d3860 con 0x7f0a240d1030 2024-09-13T16:25:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.865+0000 7f0a20ff9700 1 -- 172.21.15.119:0/3646839897 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0a14013470 con 0x7f0a240d1030 2024-09-13T16:25:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.865+0000 7f0a20ff9700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a04066730 0x7f0a04068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.865+0000 7f0a20ff9700 1 -- 172.21.15.119:0/3646839897 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0a14084570 con 0x7f0a240d1030 2024-09-13T16:25:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.865+0000 7f0a1affd700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a04066730 0x7f0a04068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.865+0000 7f0a1affd700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a04066730 0x7f0a04068bf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c000c00 tx=0x7f0a0c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:33.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:33.873+0000 7f0a20ff9700 1 -- 172.21.15.119:0/3646839897 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0a14051680 con 0x7f0a240d1030 2024-09-13T16:25:34.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.041+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a2415cfc0 con 0x7f0a04066730 2024-09-13T16:25:34.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.057+0000 7f0a20ff9700 1 -- 172.21.15.119:0/3646839897 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345162 (secure 0 0 0) 0x7f0a0c006c80 con 0x7f0a04066730 2024-09-13T16:25:34.066 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:34.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a04066730 msgr2=0x7f0a04068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a04066730 0x7f0a04068bf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c000c00 tx=0x7f0a0c008030 comp rx=0 tx=0).stop 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d1030 msgr2=0x7f0a240c6620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d1030 0x7f0a240c6620 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0a1400ec10 tx=0x7f0a14009ed0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 shutdown_connections 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a04066730 0x7f0a04068bf0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a2415cbe0 0x7f0a240b0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a240d3420 0x7f0a24151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 --2- 172.21.15.119:0/3646839897 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a240d1030 0x7f0a240c6620 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 >> 172.21.15.119:0/3646839897 conn(0x7f0a2408c080 msgr2=0x7f0a240d2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 shutdown_connections 2024-09-13T16:25:34.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.065+0000 7f0a28932700 1 -- 172.21.15.119:0/3646839897 wait complete. 2024-09-13T16:25:34.071 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:25:34.084 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-13T16:25:33.423968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219456,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755264,"statfs":{"total":773094113280,"available":772869390336,"internally_reserved":0,"allocated":3350528,"data_stored":2013930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2231,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005383"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954165+0000","last_change":"2024-09-13T16:25:16.560322+0000","last_active":"2024-09-13T16:25:20.954165+0000","last_peered":"2024-09-13T16:25:20.954165+0000","last_clean":"2024-09-13T16:25:20.954165+0000","last_became_active":"2024-09-13T16:25:16.560130+0000","last_became_peered":"2024-09-13T16:25:16.560130+0000","last_unstale":"2024-09-13T16:25:20.954165+0000","last_undegraded":"2024-09-13T16:25:20.954165+0000","last_fullsized":"2024-09-13T16:25:20.954165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941703+0000","last_change":"2024-09-13T16:25:16.937844+0000","last_active":"2024-09-13T16:25:20.941703+0000","last_peered":"2024-09-13T16:25:20.941703+0000","last_clean":"2024-09-13T16:25:20.941703+0000","last_became_active":"2024-09-13T16:25:16.937508+0000","last_became_peered":"2024-09-13T16:25:16.937508+0000","last_unstale":"2024-09-13T16:25:20.941703+0000","last_undegraded":"2024-09-13T16:25:20.941703+0000","last_fullsized":"2024-09-13T16:25:20.941703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:46.073323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163484+0000","last_change":"2024-09-13T16:25:21.162079+0000","last_active":"2024-09-13T16:25:21.163484+0000","last_peered":"2024-09-13T16:25:21.163484+0000","last_clean":"2024-09-13T16:25:21.163484+0000","last_became_active":"2024-09-13T16:25:21.161341+0000","last_became_peered":"2024-09-13T16:25:21.161341+0000","last_unstale":"2024-09-13T16:25:21.163484+0000","last_undegraded":"2024-09-13T16:25:21.163484+0000","last_fullsized":"2024-09-13T16:25:21.163484+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:55.099223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941956+0000","last_change":"2024-09-13T16:25:16.554751+0000","last_active":"2024-09-13T16:25:20.941956+0000","last_peered":"2024-09-13T16:25:20.941956+0000","last_clean":"2024-09-13T16:25:20.941956+0000","last_became_active":"2024-09-13T16:25:16.552873+0000","last_became_peered":"2024-09-13T16:25:16.552873+0000","last_unstale":"2024-09-13T16:25:20.941956+0000","last_undegraded":"2024-09-13T16:25:20.941956+0000","last_fullsized":"2024-09-13T16:25:20.941956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.947694+0000","last_change":"2024-09-13T16:25:16.933368+0000","last_active":"2024-09-13T16:25:20.947694+0000","last_peered":"2024-09-13T16:25:20.947694+0000","last_clean":"2024-09-13T16:25:20.947694+0000","last_became_active":"2024-09-13T16:25:16.932824+0000","last_became_peered":"2024-09-13T16:25:16.932824+0000","last_unstale":"2024-09-13T16:25:20.947694+0000","last_undegraded":"2024-09-13T16:25:20.947694+0000","last_fullsized":"2024-09-13T16:25:20.947694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969023+0000","last_change":"2024-09-13T16:25:20.965564+0000","last_active":"2024-09-13T16:25:20.969023+0000","last_peered":"2024-09-13T16:25:20.969023+0000","last_clean":"2024-09-13T16:25:20.969023+0000","last_became_active":"2024-09-13T16:25:20.965159+0000","last_became_peered":"2024-09-13T16:25:20.965159+0000","last_unstale":"2024-09-13T16:25:20.969023+0000","last_undegraded":"2024-09-13T16:25:20.969023+0000","last_fullsized":"2024-09-13T16:25:20.969023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150484+0000","last_change":"2024-09-13T16:25:16.560318+0000","last_active":"2024-09-13T16:25:21.150484+0000","last_peered":"2024-09-13T16:25:21.150484+0000","last_clean":"2024-09-13T16:25:21.150484+0000","last_became_active":"2024-09-13T16:25:16.560069+0000","last_became_peered":"2024-09-13T16:25:16.560069+0000","last_unstale":"2024-09-13T16:25:21.150484+0000","last_undegraded":"2024-09-13T16:25:21.150484+0000","last_fullsized":"2024-09-13T16:25:21.150484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:12.814872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943883+0000","last_change":"2024-09-13T16:25:16.935607+0000","last_active":"2024-09-13T16:25:20.943883+0000","last_peered":"2024-09-13T16:25:20.943883+0000","last_clean":"2024-09-13T16:25:20.943883+0000","last_became_active":"2024-09-13T16:25:16.935353+0000","last_became_peered":"2024-09-13T16:25:16.935353+0000","last_unstale":"2024-09-13T16:25:20.943883+0000","last_undegraded":"2024-09-13T16:25:20.943883+0000","last_fullsized":"2024-09-13T16:25:20.943883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:35.981913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973971+0000","last_change":"2024-09-13T16:25:20.969324+0000","last_active":"2024-09-13T16:25:20.973971+0000","last_peered":"2024-09-13T16:25:20.973971+0000","last_clean":"2024-09-13T16:25:20.973971+0000","last_became_active":"2024-09-13T16:25:20.968916+0000","last_became_peered":"2024-09-13T16:25:20.968916+0000","last_unstale":"2024-09-13T16:25:20.973971+0000","last_undegraded":"2024-09-13T16:25:20.973971+0000","last_fullsized":"2024-09-13T16:25:20.973971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:03.825482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149974+0000","last_change":"2024-09-13T16:25:16.560280+0000","last_active":"2024-09-13T16:25:21.149974+0000","last_peered":"2024-09-13T16:25:21.149974+0000","last_clean":"2024-09-13T16:25:21.149974+0000","last_became_active":"2024-09-13T16:25:16.559981+0000","last_became_peered":"2024-09-13T16:25:16.559981+0000","last_unstale":"2024-09-13T16:25:21.149974+0000","last_undegraded":"2024-09-13T16:25:21.149974+0000","last_fullsized":"2024-09-13T16:25:21.149974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943411+0000","last_change":"2024-09-13T16:25:16.937364+0000","last_active":"2024-09-13T16:25:20.943411+0000","last_peered":"2024-09-13T16:25:20.943411+0000","last_clean":"2024-09-13T16:25:20.943411+0000","last_became_active":"2024-09-13T16:25:16.937119+0000","last_became_peered":"2024-09-13T16:25:16.937119+0000","last_unstale":"2024-09-13T16:25:20.943411+0000","last_undegraded":"2024-09-13T16:25:20.943411+0000","last_fullsized":"2024-09-13T16:25:20.943411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:31.885317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959982+0000","last_change":"2024-09-13T16:25:20.959414+0000","last_active":"2024-09-13T16:25:20.959982+0000","last_peered":"2024-09-13T16:25:20.959982+0000","last_clean":"2024-09-13T16:25:20.959982+0000","last_became_active":"2024-09-13T16:25:20.958955+0000","last_became_peered":"2024-09-13T16:25:20.958955+0000","last_unstale":"2024-09-13T16:25:20.959982+0000","last_undegraded":"2024-09-13T16:25:20.959982+0000","last_fullsized":"2024-09-13T16:25:20.959982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:35.667608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939712+0000","last_change":"2024-09-13T16:25:16.936925+0000","last_active":"2024-09-13T16:25:20.939712+0000","last_peered":"2024-09-13T16:25:20.939712+0000","last_clean":"2024-09-13T16:25:20.939712+0000","last_became_active":"2024-09-13T16:25:16.936608+0000","last_became_peered":"2024-09-13T16:25:16.936608+0000","last_unstale":"2024-09-13T16:25:20.939712+0000","last_undegraded":"2024-09-13T16:25:20.939712+0000","last_fullsized":"2024-09-13T16:25:20.939712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150049+0000","last_change":"2024-09-13T16:25:16.561316+0000","last_active":"2024-09-13T16:25:21.150049+0000","last_peered":"2024-09-13T16:25:21.150049+0000","last_clean":"2024-09-13T16:25:21.150049+0000","last_became_active":"2024-09-13T16:25:16.560850+0000","last_became_peered":"2024-09-13T16:25:16.560850+0000","last_unstale":"2024-09-13T16:25:21.150049+0000","last_undegraded":"2024-09-13T16:25:21.150049+0000","last_fullsized":"2024-09-13T16:25:21.150049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.961598+0000","last_change":"2024-09-13T16:25:20.960831+0000","last_active":"2024-09-13T16:25:20.961598+0000","last_peered":"2024-09-13T16:25:20.961598+0000","last_clean":"2024-09-13T16:25:20.961598+0000","last_became_active":"2024-09-13T16:25:20.960495+0000","last_became_peered":"2024-09-13T16:25:20.960495+0000","last_unstale":"2024-09-13T16:25:20.961598+0000","last_undegraded":"2024-09-13T16:25:20.961598+0000","last_fullsized":"2024-09-13T16:25:20.961598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940654+0000","last_change":"2024-09-13T16:25:16.937601+0000","last_active":"2024-09-13T16:25:20.940654+0000","last_peered":"2024-09-13T16:25:20.940654+0000","last_clean":"2024-09-13T16:25:20.940654+0000","last_became_active":"2024-09-13T16:25:16.937343+0000","last_became_peered":"2024-09-13T16:25:16.937343+0000","last_unstale":"2024-09-13T16:25:20.940654+0000","last_undegraded":"2024-09-13T16:25:20.940654+0000","last_fullsized":"2024-09-13T16:25:20.940654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950384+0000","last_change":"2024-09-13T16:25:16.550175+0000","last_active":"2024-09-13T16:25:20.950384+0000","last_peered":"2024-09-13T16:25:20.950384+0000","last_clean":"2024-09-13T16:25:20.950384+0000","last_became_active":"2024-09-13T16:25:16.549875+0000","last_became_peered":"2024-09-13T16:25:16.549875+0000","last_unstale":"2024-09-13T16:25:20.950384+0000","last_undegraded":"2024-09-13T16:25:20.950384+0000","last_fullsized":"2024-09-13T16:25:20.950384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973920+0000","last_change":"2024-09-13T16:25:20.968856+0000","last_active":"2024-09-13T16:25:20.973920+0000","last_peered":"2024-09-13T16:25:20.973920+0000","last_clean":"2024-09-13T16:25:20.973920+0000","last_became_active":"2024-09-13T16:25:20.968470+0000","last_became_peered":"2024-09-13T16:25:20.968470+0000","last_unstale":"2024-09-13T16:25:20.973920+0000","last_undegraded":"2024-09-13T16:25:20.973920+0000","last_fullsized":"2024-09-13T16:25:20.973920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940018+0000","last_change":"2024-09-13T16:25:16.936551+0000","last_active":"2024-09-13T16:25:20.940018+0000","last_peered":"2024-09-13T16:25:20.940018+0000","last_clean":"2024-09-13T16:25:20.940018+0000","last_became_active":"2024-09-13T16:25:16.936298+0000","last_became_peered":"2024-09-13T16:25:16.936298+0000","last_unstale":"2024-09-13T16:25:20.940018+0000","last_undegraded":"2024-09-13T16:25:20.940018+0000","last_fullsized":"2024-09-13T16:25:20.940018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949896+0000","last_change":"2024-09-13T16:25:16.556208+0000","last_active":"2024-09-13T16:25:20.949896+0000","last_peered":"2024-09-13T16:25:20.949896+0000","last_clean":"2024-09-13T16:25:20.949896+0000","last_became_active":"2024-09-13T16:25:16.555897+0000","last_became_peered":"2024-09-13T16:25:16.555897+0000","last_unstale":"2024-09-13T16:25:20.949896+0000","last_undegraded":"2024-09-13T16:25:20.949896+0000","last_fullsized":"2024-09-13T16:25:20.949896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.164051+0000","last_change":"2024-09-13T16:25:21.162085+0000","last_active":"2024-09-13T16:25:21.164051+0000","last_peered":"2024-09-13T16:25:21.164051+0000","last_clean":"2024-09-13T16:25:21.164051+0000","last_became_active":"2024-09-13T16:25:21.160633+0000","last_became_peered":"2024-09-13T16:25:21.160633+0000","last_unstale":"2024-09-13T16:25:21.164051+0000","last_undegraded":"2024-09-13T16:25:21.164051+0000","last_fullsized":"2024-09-13T16:25:21.164051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951220+0000","last_change":"2024-09-13T16:25:16.931406+0000","last_active":"2024-09-13T16:25:20.951220+0000","last_peered":"2024-09-13T16:25:20.951220+0000","last_clean":"2024-09-13T16:25:20.951220+0000","last_became_active":"2024-09-13T16:25:16.931073+0000","last_became_peered":"2024-09-13T16:25:16.931073+0000","last_unstale":"2024-09-13T16:25:20.951220+0000","last_undegraded":"2024-09-13T16:25:20.951220+0000","last_fullsized":"2024-09-13T16:25:20.951220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:31.812511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149390+0000","last_change":"2024-09-13T16:25:16.562667+0000","last_active":"2024-09-13T16:25:21.149390+0000","last_peered":"2024-09-13T16:25:21.149390+0000","last_clean":"2024-09-13T16:25:21.149390+0000","last_became_active":"2024-09-13T16:25:16.562361+0000","last_became_peered":"2024-09-13T16:25:16.562361+0000","last_unstale":"2024-09-13T16:25:21.149390+0000","last_undegraded":"2024-09-13T16:25:21.149390+0000","last_fullsized":"2024-09-13T16:25:21.149390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:32.764470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973976+0000","last_change":"2024-09-13T16:25:20.969014+0000","last_active":"2024-09-13T16:25:20.973976+0000","last_peered":"2024-09-13T16:25:20.973976+0000","last_clean":"2024-09-13T16:25:20.973976+0000","last_became_active":"2024-09-13T16:25:20.968528+0000","last_became_peered":"2024-09-13T16:25:20.968528+0000","last_unstale":"2024-09-13T16:25:20.973976+0000","last_undegraded":"2024-09-13T16:25:20.973976+0000","last_fullsized":"2024-09-13T16:25:20.973976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948588+0000","last_change":"2024-09-13T16:25:16.933619+0000","last_active":"2024-09-13T16:25:20.948588+0000","last_peered":"2024-09-13T16:25:20.948588+0000","last_clean":"2024-09-13T16:25:20.948588+0000","last_became_active":"2024-09-13T16:25:16.933324+0000","last_became_peered":"2024-09-13T16:25:16.933324+0000","last_unstale":"2024-09-13T16:25:20.948588+0000","last_undegraded":"2024-09-13T16:25:20.948588+0000","last_fullsized":"2024-09-13T16:25:20.948588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:13.824400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942057+0000","last_change":"2024-09-13T16:25:16.560251+0000","last_active":"2024-09-13T16:25:20.942057+0000","last_peered":"2024-09-13T16:25:20.942057+0000","last_clean":"2024-09-13T16:25:20.942057+0000","last_became_active":"2024-09-13T16:25:16.559836+0000","last_became_peered":"2024-09-13T16:25:16.559836+0000","last_unstale":"2024-09-13T16:25:20.942057+0000","last_undegraded":"2024-09-13T16:25:20.942057+0000","last_fullsized":"2024-09-13T16:25:20.942057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.159122+0000","last_change":"2024-09-13T16:25:21.156691+0000","last_active":"2024-09-13T16:25:21.159122+0000","last_peered":"2024-09-13T16:25:21.159122+0000","last_clean":"2024-09-13T16:25:21.159122+0000","last_became_active":"2024-09-13T16:25:21.156255+0000","last_became_peered":"2024-09-13T16:25:21.156255+0000","last_unstale":"2024-09-13T16:25:21.159122+0000","last_undegraded":"2024-09-13T16:25:21.159122+0000","last_fullsized":"2024-09-13T16:25:21.159122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:12.641280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940196+0000","last_change":"2024-09-13T16:25:16.946326+0000","last_active":"2024-09-13T16:25:20.940196+0000","last_peered":"2024-09-13T16:25:20.940196+0000","last_clean":"2024-09-13T16:25:20.940196+0000","last_became_active":"2024-09-13T16:25:16.946091+0000","last_became_peered":"2024-09-13T16:25:16.946091+0000","last_unstale":"2024-09-13T16:25:20.940196+0000","last_undegraded":"2024-09-13T16:25:20.940196+0000","last_fullsized":"2024-09-13T16:25:20.940196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953612+0000","last_change":"2024-09-13T16:25:16.548181+0000","last_active":"2024-09-13T16:25:20.953612+0000","last_peered":"2024-09-13T16:25:20.953612+0000","last_clean":"2024-09-13T16:25:20.953612+0000","last_became_active":"2024-09-13T16:25:16.546980+0000","last_became_peered":"2024-09-13T16:25:16.546980+0000","last_unstale":"2024-09-13T16:25:20.953612+0000","last_undegraded":"2024-09-13T16:25:20.953612+0000","last_fullsized":"2024-09-13T16:25:20.953612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163366+0000","last_change":"2024-09-13T16:25:21.157394+0000","last_active":"2024-09-13T16:25:21.163366+0000","last_peered":"2024-09-13T16:25:21.163366+0000","last_clean":"2024-09-13T16:25:21.163366+0000","last_became_active":"2024-09-13T16:25:21.156428+0000","last_became_peered":"2024-09-13T16:25:21.156428+0000","last_unstale":"2024-09-13T16:25:21.163366+0000","last_undegraded":"2024-09-13T16:25:21.163366+0000","last_fullsized":"2024-09-13T16:25:21.163366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:31.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950532+0000","last_change":"2024-09-13T16:25:16.932134+0000","last_active":"2024-09-13T16:25:20.950532+0000","last_peered":"2024-09-13T16:25:20.950532+0000","last_clean":"2024-09-13T16:25:20.950532+0000","last_became_active":"2024-09-13T16:25:16.931157+0000","last_became_peered":"2024-09-13T16:25:16.931157+0000","last_unstale":"2024-09-13T16:25:20.950532+0000","last_undegraded":"2024-09-13T16:25:20.950532+0000","last_fullsized":"2024-09-13T16:25:20.950532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150728+0000","last_change":"2024-09-13T16:25:16.560858+0000","last_active":"2024-09-13T16:25:21.150728+0000","last_peered":"2024-09-13T16:25:21.150728+0000","last_clean":"2024-09-13T16:25:21.150728+0000","last_became_active":"2024-09-13T16:25:16.560627+0000","last_became_peered":"2024-09-13T16:25:16.560627+0000","last_unstale":"2024-09-13T16:25:21.150728+0000","last_undegraded":"2024-09-13T16:25:21.150728+0000","last_fullsized":"2024-09-13T16:25:21.150728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:50.678565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158800+0000","last_change":"2024-09-13T16:25:21.155558+0000","last_active":"2024-09-13T16:25:21.158800+0000","last_peered":"2024-09-13T16:25:21.158800+0000","last_clean":"2024-09-13T16:25:21.158800+0000","last_became_active":"2024-09-13T16:25:21.154513+0000","last_became_peered":"2024-09-13T16:25:21.154513+0000","last_unstale":"2024-09-13T16:25:21.158800+0000","last_undegraded":"2024-09-13T16:25:21.158800+0000","last_fullsized":"2024-09-13T16:25:21.158800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943548+0000","last_change":"2024-09-13T16:25:16.934627+0000","last_active":"2024-09-13T16:25:20.943548+0000","last_peered":"2024-09-13T16:25:20.943548+0000","last_clean":"2024-09-13T16:25:20.943548+0000","last_became_active":"2024-09-13T16:25:16.934031+0000","last_became_peered":"2024-09-13T16:25:16.934031+0000","last_unstale":"2024-09-13T16:25:20.943548+0000","last_undegraded":"2024-09-13T16:25:20.943548+0000","last_fullsized":"2024-09-13T16:25:20.943548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:19.244372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950619+0000","last_change":"2024-09-13T16:25:16.550845+0000","last_active":"2024-09-13T16:25:20.950619+0000","last_peered":"2024-09-13T16:25:20.950619+0000","last_clean":"2024-09-13T16:25:20.950619+0000","last_became_active":"2024-09-13T16:25:16.550523+0000","last_became_peered":"2024-09-13T16:25:16.550523+0000","last_unstale":"2024-09-13T16:25:20.950619+0000","last_undegraded":"2024-09-13T16:25:20.950619+0000","last_fullsized":"2024-09-13T16:25:20.950619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971742+0000","last_change":"2024-09-13T16:25:20.963835+0000","last_active":"2024-09-13T16:25:20.971742+0000","last_peered":"2024-09-13T16:25:20.971742+0000","last_clean":"2024-09-13T16:25:20.971742+0000","last_became_active":"2024-09-13T16:25:20.963031+0000","last_became_peered":"2024-09-13T16:25:20.963031+0000","last_unstale":"2024-09-13T16:25:20.971742+0000","last_undegraded":"2024-09-13T16:25:20.971742+0000","last_fullsized":"2024-09-13T16:25:20.971742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:39.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951685+0000","last_change":"2024-09-13T16:25:16.936436+0000","last_active":"2024-09-13T16:25:20.951685+0000","last_peered":"2024-09-13T16:25:20.951685+0000","last_clean":"2024-09-13T16:25:20.951685+0000","last_became_active":"2024-09-13T16:25:16.936131+0000","last_became_peered":"2024-09-13T16:25:16.936131+0000","last_unstale":"2024-09-13T16:25:20.951685+0000","last_undegraded":"2024-09-13T16:25:20.951685+0000","last_fullsized":"2024-09-13T16:25:20.951685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:33.773998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941767+0000","last_change":"2024-09-13T16:25:16.548310+0000","last_active":"2024-09-13T16:25:20.941767+0000","last_peered":"2024-09-13T16:25:20.941767+0000","last_clean":"2024-09-13T16:25:20.941767+0000","last_became_active":"2024-09-13T16:25:16.548056+0000","last_became_peered":"2024-09-13T16:25:16.548056+0000","last_unstale":"2024-09-13T16:25:20.941767+0000","last_undegraded":"2024-09-13T16:25:20.941767+0000","last_fullsized":"2024-09-13T16:25:20.941767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971523+0000","last_change":"2024-09-13T16:25:20.960907+0000","last_active":"2024-09-13T16:25:20.971523+0000","last_peered":"2024-09-13T16:25:20.971523+0000","last_clean":"2024-09-13T16:25:20.971523+0000","last_became_active":"2024-09-13T16:25:20.960504+0000","last_became_peered":"2024-09-13T16:25:20.960504+0000","last_unstale":"2024-09-13T16:25:20.971523+0000","last_undegraded":"2024-09-13T16:25:20.971523+0000","last_fullsized":"2024-09-13T16:25:20.971523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951906+0000","last_change":"2024-09-13T16:25:16.935966+0000","last_active":"2024-09-13T16:25:20.951906+0000","last_peered":"2024-09-13T16:25:20.951906+0000","last_clean":"2024-09-13T16:25:20.951906+0000","last_became_active":"2024-09-13T16:25:16.935728+0000","last_became_peered":"2024-09-13T16:25:16.935728+0000","last_unstale":"2024-09-13T16:25:20.951906+0000","last_undegraded":"2024-09-13T16:25:20.951906+0000","last_fullsized":"2024-09-13T16:25:20.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954151+0000","last_change":"2024-09-13T16:25:16.541632+0000","last_active":"2024-09-13T16:25:20.954151+0000","last_peered":"2024-09-13T16:25:20.954151+0000","last_clean":"2024-09-13T16:25:20.954151+0000","last_became_active":"2024-09-13T16:25:16.541362+0000","last_became_peered":"2024-09-13T16:25:16.541362+0000","last_unstale":"2024-09-13T16:25:20.954151+0000","last_undegraded":"2024-09-13T16:25:20.954151+0000","last_fullsized":"2024-09-13T16:25:20.954151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971468+0000","last_change":"2024-09-13T16:25:20.961579+0000","last_active":"2024-09-13T16:25:20.971468+0000","last_peered":"2024-09-13T16:25:20.971468+0000","last_clean":"2024-09-13T16:25:20.971468+0000","last_became_active":"2024-09-13T16:25:20.961177+0000","last_became_peered":"2024-09-13T16:25:20.961177+0000","last_unstale":"2024-09-13T16:25:20.971468+0000","last_undegraded":"2024-09-13T16:25:20.971468+0000","last_fullsized":"2024-09-13T16:25:20.971468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948340+0000","last_change":"2024-09-13T16:25:16.932105+0000","last_active":"2024-09-13T16:25:20.948340+0000","last_peered":"2024-09-13T16:25:20.948340+0000","last_clean":"2024-09-13T16:25:20.948340+0000","last_became_active":"2024-09-13T16:25:16.931893+0000","last_became_peered":"2024-09-13T16:25:16.931893+0000","last_unstale":"2024-09-13T16:25:20.948340+0000","last_undegraded":"2024-09-13T16:25:20.948340+0000","last_fullsized":"2024-09-13T16:25:20.948340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149350+0000","last_change":"2024-09-13T16:25:16.563013+0000","last_active":"2024-09-13T16:25:21.149350+0000","last_peered":"2024-09-13T16:25:21.149350+0000","last_clean":"2024-09-13T16:25:21.149350+0000","last_became_active":"2024-09-13T16:25:16.562820+0000","last_became_peered":"2024-09-13T16:25:16.562820+0000","last_unstale":"2024-09-13T16:25:21.149350+0000","last_undegraded":"2024-09-13T16:25:21.149350+0000","last_fullsized":"2024-09-13T16:25:21.149350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:43.080203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.161407+0000","last_change":"2024-09-13T16:25:21.156288+0000","last_active":"2024-09-13T16:25:21.161407+0000","last_peered":"2024-09-13T16:25:21.161407+0000","last_clean":"2024-09-13T16:25:21.161407+0000","last_became_active":"2024-09-13T16:25:21.154873+0000","last_became_peered":"2024-09-13T16:25:21.154873+0000","last_unstale":"2024-09-13T16:25:21.161407+0000","last_undegraded":"2024-09-13T16:25:21.161407+0000","last_fullsized":"2024-09-13T16:25:21.161407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:57.390965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941321+0000","last_change":"2024-09-13T16:25:16.934659+0000","last_active":"2024-09-13T16:25:20.941321+0000","last_peered":"2024-09-13T16:25:20.941321+0000","last_clean":"2024-09-13T16:25:20.941321+0000","last_became_active":"2024-09-13T16:25:16.934286+0000","last_became_peered":"2024-09-13T16:25:16.934286+0000","last_unstale":"2024-09-13T16:25:20.941321+0000","last_undegraded":"2024-09-13T16:25:20.941321+0000","last_fullsized":"2024-09-13T16:25:20.941321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941932+0000","last_change":"2024-09-13T16:25:16.548968+0000","last_active":"2024-09-13T16:25:20.941932+0000","last_peered":"2024-09-13T16:25:20.941932+0000","last_clean":"2024-09-13T16:25:20.941932+0000","last_became_active":"2024-09-13T16:25:16.548722+0000","last_became_peered":"2024-09-13T16:25:16.548722+0000","last_unstale":"2024-09-13T16:25:20.941932+0000","last_undegraded":"2024-09-13T16:25:20.941932+0000","last_fullsized":"2024-09-13T16:25:20.941932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:25:08.328680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149477+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:25:21.149477+0000","last_peered":"2024-09-13T16:25:21.149477+0000","last_clean":"2024-09-13T16:25:21.149477+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:25:21.149477+0000","last_undegraded":"2024-09-13T16:25:21.149477+0000","last_fullsized":"2024-09-13T16:25:21.149477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156667+0000","last_change":"2024-09-13T16:25:21.154730+0000","last_active":"2024-09-13T16:25:21.156667+0000","last_peered":"2024-09-13T16:25:21.156667+0000","last_clean":"2024-09-13T16:25:21.156667+0000","last_became_active":"2024-09-13T16:25:21.154226+0000","last_became_peered":"2024-09-13T16:25:21.154226+0000","last_unstale":"2024-09-13T16:25:21.156667+0000","last_undegraded":"2024-09-13T16:25:21.156667+0000","last_fullsized":"2024-09-13T16:25:21.156667+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150204+0000","last_change":"2024-09-13T16:25:16.944787+0000","last_active":"2024-09-13T16:25:21.150204+0000","last_peered":"2024-09-13T16:25:21.150204+0000","last_clean":"2024-09-13T16:25:21.150204+0000","last_became_active":"2024-09-13T16:25:16.944505+0000","last_became_peered":"2024-09-13T16:25:16.944505+0000","last_unstale":"2024-09-13T16:25:21.150204+0000","last_undegraded":"2024-09-13T16:25:21.150204+0000","last_fullsized":"2024-09-13T16:25:21.150204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954746+0000","last_change":"2024-09-13T16:25:16.548195+0000","last_active":"2024-09-13T16:25:20.954746+0000","last_peered":"2024-09-13T16:25:20.954746+0000","last_clean":"2024-09-13T16:25:20.954746+0000","last_became_active":"2024-09-13T16:25:16.547839+0000","last_became_peered":"2024-09-13T16:25:16.547839+0000","last_unstale":"2024-09-13T16:25:20.954746+0000","last_undegraded":"2024-09-13T16:25:20.954746+0000","last_fullsized":"2024-09-13T16:25:20.954746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156778+0000","last_change":"2024-09-13T16:25:21.154890+0000","last_active":"2024-09-13T16:25:21.156778+0000","last_peered":"2024-09-13T16:25:21.156778+0000","last_clean":"2024-09-13T16:25:21.156778+0000","last_became_active":"2024-09-13T16:25:21.154376+0000","last_became_peered":"2024-09-13T16:25:21.154376+0000","last_unstale":"2024-09-13T16:25:21.156778+0000","last_undegraded":"2024-09-13T16:25:21.156778+0000","last_fullsized":"2024-09-13T16:25:21.156778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:09.876152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944021+0000","last_change":"2024-09-13T16:25:16.933523+0000","last_active":"2024-09-13T16:25:20.944021+0000","last_peered":"2024-09-13T16:25:20.944021+0000","last_clean":"2024-09-13T16:25:20.944021+0000","last_became_active":"2024-09-13T16:25:16.932020+0000","last_became_peered":"2024-09-13T16:25:16.932020+0000","last_unstale":"2024-09-13T16:25:20.944021+0000","last_undegraded":"2024-09-13T16:25:20.944021+0000","last_fullsized":"2024-09-13T16:25:20.944021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:34.060136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952497+0000","last_change":"2024-09-13T16:25:16.546857+0000","last_active":"2024-09-13T16:25:20.952497+0000","last_peered":"2024-09-13T16:25:20.952497+0000","last_clean":"2024-09-13T16:25:20.952497+0000","last_became_active":"2024-09-13T16:25:16.546592+0000","last_became_peered":"2024-09-13T16:25:16.546592+0000","last_unstale":"2024-09-13T16:25:20.952497+0000","last_undegraded":"2024-09-13T16:25:20.952497+0000","last_fullsized":"2024-09-13T16:25:20.952497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963301+0000","last_change":"2024-09-13T16:25:20.958682+0000","last_active":"2024-09-13T16:25:20.963301+0000","last_peered":"2024-09-13T16:25:20.963301+0000","last_clean":"2024-09-13T16:25:20.963301+0000","last_became_active":"2024-09-13T16:25:20.958064+0000","last_became_peered":"2024-09-13T16:25:20.958064+0000","last_unstale":"2024-09-13T16:25:20.963301+0000","last_undegraded":"2024-09-13T16:25:20.963301+0000","last_fullsized":"2024-09-13T16:25:20.963301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.502649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150943+0000","last_change":"2024-09-13T16:25:16.951592+0000","last_active":"2024-09-13T16:25:21.150943+0000","last_peered":"2024-09-13T16:25:21.150943+0000","last_clean":"2024-09-13T16:25:21.150943+0000","last_became_active":"2024-09-13T16:25:16.951375+0000","last_became_peered":"2024-09-13T16:25:16.951375+0000","last_unstale":"2024-09-13T16:25:21.150943+0000","last_undegraded":"2024-09-13T16:25:21.150943+0000","last_fullsized":"2024-09-13T16:25:21.150943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:32.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959285+0000","last_change":"2024-09-13T16:25:16.541133+0000","last_active":"2024-09-13T16:25:20.959285+0000","last_peered":"2024-09-13T16:25:20.959285+0000","last_clean":"2024-09-13T16:25:20.959285+0000","last_became_active":"2024-09-13T16:25:16.540843+0000","last_became_peered":"2024-09-13T16:25:16.540843+0000","last_unstale":"2024-09-13T16:25:20.959285+0000","last_undegraded":"2024-09-13T16:25:20.959285+0000","last_fullsized":"2024-09-13T16:25:20.959285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:05.477981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158080+0000","last_change":"2024-09-13T16:25:21.154908+0000","last_active":"2024-09-13T16:25:21.158080+0000","last_peered":"2024-09-13T16:25:21.158080+0000","last_clean":"2024-09-13T16:25:21.158080+0000","last_became_active":"2024-09-13T16:25:21.154365+0000","last_became_peered":"2024-09-13T16:25:21.154365+0000","last_unstale":"2024-09-13T16:25:21.158080+0000","last_undegraded":"2024-09-13T16:25:21.158080+0000","last_fullsized":"2024-09-13T16:25:21.158080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:00.139666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150870+0000","last_change":"2024-09-13T16:25:16.941714+0000","last_active":"2024-09-13T16:25:21.150870+0000","last_peered":"2024-09-13T16:25:21.150870+0000","last_clean":"2024-09-13T16:25:21.150870+0000","last_became_active":"2024-09-13T16:25:16.941465+0000","last_became_peered":"2024-09-13T16:25:16.941465+0000","last_unstale":"2024-09-13T16:25:21.150870+0000","last_undegraded":"2024-09-13T16:25:21.150870+0000","last_fullsized":"2024-09-13T16:25:21.150870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952152+0000","last_change":"2024-09-13T16:25:16.544584+0000","last_active":"2024-09-13T16:25:20.952152+0000","last_peered":"2024-09-13T16:25:20.952152+0000","last_clean":"2024-09-13T16:25:20.952152+0000","last_became_active":"2024-09-13T16:25:16.544275+0000","last_became_peered":"2024-09-13T16:25:16.544275+0000","last_unstale":"2024-09-13T16:25:20.952152+0000","last_undegraded":"2024-09-13T16:25:20.952152+0000","last_fullsized":"2024-09-13T16:25:20.952152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:38.495511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.976715+0000","last_change":"2024-09-13T16:25:20.972282+0000","last_active":"2024-09-13T16:25:20.976715+0000","last_peered":"2024-09-13T16:25:20.976715+0000","last_clean":"2024-09-13T16:25:20.976715+0000","last_became_active":"2024-09-13T16:25:20.971899+0000","last_became_peered":"2024-09-13T16:25:20.971899+0000","last_unstale":"2024-09-13T16:25:20.976715+0000","last_undegraded":"2024-09-13T16:25:20.976715+0000","last_fullsized":"2024-09-13T16:25:20.976715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150689+0000","last_change":"2024-09-13T16:25:16.944388+0000","last_active":"2024-09-13T16:25:21.150689+0000","last_peered":"2024-09-13T16:25:21.150689+0000","last_clean":"2024-09-13T16:25:21.150689+0000","last_became_active":"2024-09-13T16:25:16.943987+0000","last_became_peered":"2024-09-13T16:25:16.943987+0000","last_unstale":"2024-09-13T16:25:21.150689+0000","last_undegraded":"2024-09-13T16:25:21.150689+0000","last_fullsized":"2024-09-13T16:25:21.150689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953615+0000","last_change":"2024-09-13T16:25:16.545908+0000","last_active":"2024-09-13T16:25:20.953615+0000","last_peered":"2024-09-13T16:25:20.953615+0000","last_clean":"2024-09-13T16:25:20.953615+0000","last_became_active":"2024-09-13T16:25:16.545632+0000","last_became_peered":"2024-09-13T16:25:16.545632+0000","last_unstale":"2024-09-13T16:25:20.953615+0000","last_undegraded":"2024-09-13T16:25:20.953615+0000","last_fullsized":"2024-09-13T16:25:20.953615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963693+0000","last_change":"2024-09-13T16:25:20.961076+0000","last_active":"2024-09-13T16:25:20.963693+0000","last_peered":"2024-09-13T16:25:20.963693+0000","last_clean":"2024-09-13T16:25:20.963693+0000","last_became_active":"2024-09-13T16:25:20.960736+0000","last_became_peered":"2024-09-13T16:25:20.960736+0000","last_unstale":"2024-09-13T16:25:20.963693+0000","last_undegraded":"2024-09-13T16:25:20.963693+0000","last_fullsized":"2024-09-13T16:25:20.963693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939386+0000","last_change":"2024-09-13T16:25:16.939543+0000","last_active":"2024-09-13T16:25:20.939386+0000","last_peered":"2024-09-13T16:25:20.939386+0000","last_clean":"2024-09-13T16:25:20.939386+0000","last_became_active":"2024-09-13T16:25:16.939269+0000","last_became_peered":"2024-09-13T16:25:16.939269+0000","last_unstale":"2024-09-13T16:25:20.939386+0000","last_undegraded":"2024-09-13T16:25:20.939386+0000","last_fullsized":"2024-09-13T16:25:20.939386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939280+0000","last_change":"2024-09-13T16:25:16.557172+0000","last_active":"2024-09-13T16:25:20.939280+0000","last_peered":"2024-09-13T16:25:20.939280+0000","last_clean":"2024-09-13T16:25:20.939280+0000","last_became_active":"2024-09-13T16:25:16.556910+0000","last_became_peered":"2024-09-13T16:25:16.556910+0000","last_unstale":"2024-09-13T16:25:20.939280+0000","last_undegraded":"2024-09-13T16:25:20.939280+0000","last_fullsized":"2024-09-13T16:25:20.939280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162791+0000","last_change":"2024-09-13T16:25:21.154750+0000","last_active":"2024-09-13T16:25:21.162791+0000","last_peered":"2024-09-13T16:25:21.162791+0000","last_clean":"2024-09-13T16:25:21.162791+0000","last_became_active":"2024-09-13T16:25:21.153987+0000","last_became_peered":"2024-09-13T16:25:21.153987+0000","last_unstale":"2024-09-13T16:25:21.162791+0000","last_undegraded":"2024-09-13T16:25:21.162791+0000","last_fullsized":"2024-09-13T16:25:21.162791+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149825+0000","last_change":"2024-09-13T16:25:16.944291+0000","last_active":"2024-09-13T16:25:21.149825+0000","last_peered":"2024-09-13T16:25:21.149825+0000","last_clean":"2024-09-13T16:25:21.149825+0000","last_became_active":"2024-09-13T16:25:16.941709+0000","last_became_peered":"2024-09-13T16:25:16.941709+0000","last_unstale":"2024-09-13T16:25:21.149825+0000","last_undegraded":"2024-09-13T16:25:21.149825+0000","last_fullsized":"2024-09-13T16:25:21.149825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:45.537446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941804+0000","last_change":"2024-09-13T16:25:16.553038+0000","last_active":"2024-09-13T16:25:20.941804+0000","last_peered":"2024-09-13T16:25:20.941804+0000","last_clean":"2024-09-13T16:25:20.941804+0000","last_became_active":"2024-09-13T16:25:16.551960+0000","last_became_peered":"2024-09-13T16:25:16.551960+0000","last_unstale":"2024-09-13T16:25:20.941804+0000","last_undegraded":"2024-09-13T16:25:20.941804+0000","last_fullsized":"2024-09-13T16:25:20.941804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:27.632855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163399+0000","last_change":"2024-09-13T16:25:21.158491+0000","last_active":"2024-09-13T16:25:21.163399+0000","last_peered":"2024-09-13T16:25:21.163399+0000","last_clean":"2024-09-13T16:25:21.163399+0000","last_became_active":"2024-09-13T16:25:21.158133+0000","last_became_peered":"2024-09-13T16:25:21.158133+0000","last_unstale":"2024-09-13T16:25:21.163399+0000","last_undegraded":"2024-09-13T16:25:21.163399+0000","last_fullsized":"2024-09-13T16:25:21.163399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946925+0000","last_change":"2024-09-13T16:25:16.941874+0000","last_active":"2024-09-13T16:25:20.946925+0000","last_peered":"2024-09-13T16:25:20.946925+0000","last_clean":"2024-09-13T16:25:20.946925+0000","last_became_active":"2024-09-13T16:25:16.941683+0000","last_became_peered":"2024-09-13T16:25:16.941683+0000","last_unstale":"2024-09-13T16:25:20.946925+0000","last_undegraded":"2024-09-13T16:25:20.946925+0000","last_fullsized":"2024-09-13T16:25:20.946925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953752+0000","last_change":"2024-09-13T16:25:16.551439+0000","last_active":"2024-09-13T16:25:20.953752+0000","last_peered":"2024-09-13T16:25:20.953752+0000","last_clean":"2024-09-13T16:25:20.953752+0000","last_became_active":"2024-09-13T16:25:16.550937+0000","last_became_peered":"2024-09-13T16:25:16.550937+0000","last_unstale":"2024-09-13T16:25:20.953752+0000","last_undegraded":"2024-09-13T16:25:20.953752+0000","last_fullsized":"2024-09-13T16:25:20.953752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:14.983082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158057+0000","last_change":"2024-09-13T16:25:21.155556+0000","last_active":"2024-09-13T16:25:21.158057+0000","last_peered":"2024-09-13T16:25:21.158057+0000","last_clean":"2024-09-13T16:25:21.158057+0000","last_became_active":"2024-09-13T16:25:21.155201+0000","last_became_peered":"2024-09-13T16:25:21.155201+0000","last_unstale":"2024-09-13T16:25:21.158057+0000","last_undegraded":"2024-09-13T16:25:21.158057+0000","last_fullsized":"2024-09-13T16:25:21.158057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:42.823106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150192+0000","last_change":"2024-09-13T16:25:16.944770+0000","last_active":"2024-09-13T16:25:21.150192+0000","last_peered":"2024-09-13T16:25:21.150192+0000","last_clean":"2024-09-13T16:25:21.150192+0000","last_became_active":"2024-09-13T16:25:16.944455+0000","last_became_peered":"2024-09-13T16:25:16.944455+0000","last_unstale":"2024-09-13T16:25:21.150192+0000","last_undegraded":"2024-09-13T16:25:21.150192+0000","last_fullsized":"2024-09-13T16:25:21.150192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:41:40.454800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-13T16:25:26.199825+0000","last_change":"2024-09-13T16:25:16.552517+0000","last_active":"2024-09-13T16:25:26.199825+0000","last_peered":"2024-09-13T16:25:26.199825+0000","last_clean":"2024-09-13T16:25:26.199825+0000","last_became_active":"2024-09-13T16:25:16.552101+0000","last_became_peered":"2024-09-13T16:25:16.552101+0000","last_unstale":"2024-09-13T16:25:26.199825+0000","last_undegraded":"2024-09-13T16:25:26.199825+0000","last_fullsized":"2024-09-13T16:25:26.199825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971715+0000","last_change":"2024-09-13T16:25:20.963516+0000","last_active":"2024-09-13T16:25:20.971715+0000","last_peered":"2024-09-13T16:25:20.971715+0000","last_clean":"2024-09-13T16:25:20.971715+0000","last_became_active":"2024-09-13T16:25:20.962518+0000","last_became_peered":"2024-09-13T16:25:20.962518+0000","last_unstale":"2024-09-13T16:25:20.971715+0000","last_undegraded":"2024-09-13T16:25:20.971715+0000","last_fullsized":"2024-09-13T16:25:20.971715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:05.557712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951494+0000","last_change":"2024-09-13T16:25:16.941833+0000","last_active":"2024-09-13T16:25:20.951494+0000","last_peered":"2024-09-13T16:25:20.951494+0000","last_clean":"2024-09-13T16:25:20.951494+0000","last_became_active":"2024-09-13T16:25:16.941607+0000","last_became_peered":"2024-09-13T16:25:16.941607+0000","last_unstale":"2024-09-13T16:25:20.951494+0000","last_undegraded":"2024-09-13T16:25:20.951494+0000","last_fullsized":"2024-09-13T16:25:20.951494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:44.089730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952414+0000","last_change":"2024-09-13T16:25:16.549878+0000","last_active":"2024-09-13T16:25:20.952414+0000","last_peered":"2024-09-13T16:25:20.952414+0000","last_clean":"2024-09-13T16:25:20.952414+0000","last_became_active":"2024-09-13T16:25:16.548581+0000","last_became_peered":"2024-09-13T16:25:16.548581+0000","last_unstale":"2024-09-13T16:25:20.952414+0000","last_undegraded":"2024-09-13T16:25:20.952414+0000","last_fullsized":"2024-09-13T16:25:20.952414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969433+0000","last_change":"2024-09-13T16:25:20.965523+0000","last_active":"2024-09-13T16:25:20.969433+0000","last_peered":"2024-09-13T16:25:20.969433+0000","last_clean":"2024-09-13T16:25:20.969433+0000","last_became_active":"2024-09-13T16:25:20.965035+0000","last_became_peered":"2024-09-13T16:25:20.965035+0000","last_unstale":"2024-09-13T16:25:20.969433+0000","last_undegraded":"2024-09-13T16:25:20.969433+0000","last_fullsized":"2024-09-13T16:25:20.969433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954066+0000","last_change":"2024-09-13T16:25:16.933522+0000","last_active":"2024-09-13T16:25:20.954066+0000","last_peered":"2024-09-13T16:25:20.954066+0000","last_clean":"2024-09-13T16:25:20.954066+0000","last_became_active":"2024-09-13T16:25:16.933134+0000","last_became_peered":"2024-09-13T16:25:16.933134+0000","last_unstale":"2024-09-13T16:25:20.954066+0000","last_undegraded":"2024-09-13T16:25:20.954066+0000","last_fullsized":"2024-09-13T16:25:20.954066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942876+0000","last_change":"2024-09-13T16:25:16.549756+0000","last_active":"2024-09-13T16:25:20.942876+0000","last_peered":"2024-09-13T16:25:20.942876+0000","last_clean":"2024-09-13T16:25:20.942876+0000","last_became_active":"2024-09-13T16:25:16.549515+0000","last_became_peered":"2024-09-13T16:25:16.549515+0000","last_unstale":"2024-09-13T16:25:20.942876+0000","last_undegraded":"2024-09-13T16:25:20.942876+0000","last_fullsized":"2024-09-13T16:25:20.942876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963576+0000","last_change":"2024-09-13T16:25:20.960084+0000","last_active":"2024-09-13T16:25:20.963576+0000","last_peered":"2024-09-13T16:25:20.963576+0000","last_clean":"2024-09-13T16:25:20.963576+0000","last_became_active":"2024-09-13T16:25:20.957343+0000","last_became_peered":"2024-09-13T16:25:20.957343+0000","last_unstale":"2024-09-13T16:25:20.963576+0000","last_undegraded":"2024-09-13T16:25:20.963576+0000","last_fullsized":"2024-09-13T16:25:20.963576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:06.115184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942738+0000","last_change":"2024-09-13T16:25:16.942162+0000","last_active":"2024-09-13T16:25:20.942738+0000","last_peered":"2024-09-13T16:25:20.942738+0000","last_clean":"2024-09-13T16:25:20.942738+0000","last_became_active":"2024-09-13T16:25:16.941896+0000","last_became_peered":"2024-09-13T16:25:16.941896+0000","last_unstale":"2024-09-13T16:25:20.942738+0000","last_undegraded":"2024-09-13T16:25:20.942738+0000","last_fullsized":"2024-09-13T16:25:20.942738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:24.023742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941934+0000","last_change":"2024-09-13T16:25:16.551442+0000","last_active":"2024-09-13T16:25:20.941934+0000","last_peered":"2024-09-13T16:25:20.941934+0000","last_clean":"2024-09-13T16:25:20.941934+0000","last_became_active":"2024-09-13T16:25:16.551084+0000","last_became_peered":"2024-09-13T16:25:16.551084+0000","last_unstale":"2024-09-13T16:25:20.941934+0000","last_undegraded":"2024-09-13T16:25:20.941934+0000","last_fullsized":"2024-09-13T16:25:20.941934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163416+0000","last_change":"2024-09-13T16:25:21.157528+0000","last_active":"2024-09-13T16:25:21.163416+0000","last_peered":"2024-09-13T16:25:21.163416+0000","last_clean":"2024-09-13T16:25:21.163416+0000","last_became_active":"2024-09-13T16:25:21.155560+0000","last_became_peered":"2024-09-13T16:25:21.155560+0000","last_unstale":"2024-09-13T16:25:21.163416+0000","last_undegraded":"2024-09-13T16:25:21.163416+0000","last_fullsized":"2024-09-13T16:25:21.163416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162596+0000","last_change":"2024-09-13T16:25:21.158060+0000","last_active":"2024-09-13T16:25:21.162596+0000","last_peered":"2024-09-13T16:25:21.162596+0000","last_clean":"2024-09-13T16:25:21.162596+0000","last_became_active":"2024-09-13T16:25:21.156882+0000","last_became_peered":"2024-09-13T16:25:21.156882+0000","last_unstale":"2024-09-13T16:25:21.162596+0000","last_undegraded":"2024-09-13T16:25:21.162596+0000","last_fullsized":"2024-09-13T16:25:21.162596+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949704+0000","last_change":"2024-09-13T16:25:16.932645+0000","last_active":"2024-09-13T16:25:20.949704+0000","last_peered":"2024-09-13T16:25:20.949704+0000","last_clean":"2024-09-13T16:25:20.949704+0000","last_became_active":"2024-09-13T16:25:16.931775+0000","last_became_peered":"2024-09-13T16:25:16.931775+0000","last_unstale":"2024-09-13T16:25:20.949704+0000","last_undegraded":"2024-09-13T16:25:20.949704+0000","last_fullsized":"2024-09-13T16:25:20.949704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942164+0000","last_change":"2024-09-13T16:25:16.547336+0000","last_active":"2024-09-13T16:25:20.942164+0000","last_peered":"2024-09-13T16:25:20.942164+0000","last_clean":"2024-09-13T16:25:20.942164+0000","last_became_active":"2024-09-13T16:25:16.547062+0000","last_became_peered":"2024-09-13T16:25:16.547062+0000","last_unstale":"2024-09-13T16:25:20.942164+0000","last_undegraded":"2024-09-13T16:25:20.942164+0000","last_fullsized":"2024-09-13T16:25:20.942164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.968355+0000","last_change":"2024-09-13T16:25:20.965354+0000","last_active":"2024-09-13T16:25:20.968355+0000","last_peered":"2024-09-13T16:25:20.968355+0000","last_clean":"2024-09-13T16:25:20.968355+0000","last_became_active":"2024-09-13T16:25:20.964985+0000","last_became_peered":"2024-09-13T16:25:20.964985+0000","last_unstale":"2024-09-13T16:25:20.968355+0000","last_undegraded":"2024-09-13T16:25:20.968355+0000","last_fullsized":"2024-09-13T16:25:20.968355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:42.840413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939763+0000","last_change":"2024-09-13T16:25:16.929034+0000","last_active":"2024-09-13T16:25:20.939763+0000","last_peered":"2024-09-13T16:25:20.939763+0000","last_clean":"2024-09-13T16:25:20.939763+0000","last_became_active":"2024-09-13T16:25:16.928782+0000","last_became_peered":"2024-09-13T16:25:16.928782+0000","last_unstale":"2024-09-13T16:25:20.939763+0000","last_undegraded":"2024-09-13T16:25:20.939763+0000","last_fullsized":"2024-09-13T16:25:20.939763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940092+0000","last_change":"2024-09-13T16:25:16.548923+0000","last_active":"2024-09-13T16:25:20.940092+0000","last_peered":"2024-09-13T16:25:20.940092+0000","last_clean":"2024-09-13T16:25:20.940092+0000","last_became_active":"2024-09-13T16:25:16.545843+0000","last_became_peered":"2024-09-13T16:25:16.545843+0000","last_unstale":"2024-09-13T16:25:20.940092+0000","last_undegraded":"2024-09-13T16:25:20.940092+0000","last_fullsized":"2024-09-13T16:25:20.940092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939883+0000","last_change":"2024-09-13T16:25:16.555750+0000","last_active":"2024-09-13T16:25:20.939883+0000","last_peered":"2024-09-13T16:25:20.939883+0000","last_clean":"2024-09-13T16:25:20.939883+0000","last_became_active":"2024-09-13T16:25:16.555480+0000","last_became_peered":"2024-09-13T16:25:16.555480+0000","last_unstale":"2024-09-13T16:25:20.939883+0000","last_undegraded":"2024-09-13T16:25:20.939883+0000","last_fullsized":"2024-09-13T16:25:20.939883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953647+0000","last_change":"2024-09-13T16:25:16.934605+0000","last_active":"2024-09-13T16:25:20.953647+0000","last_peered":"2024-09-13T16:25:20.953647+0000","last_clean":"2024-09-13T16:25:20.953647+0000","last_became_active":"2024-09-13T16:25:16.934365+0000","last_became_peered":"2024-09-13T16:25:16.934365+0000","last_unstale":"2024-09-13T16:25:20.953647+0000","last_undegraded":"2024-09-13T16:25:20.953647+0000","last_fullsized":"2024-09-13T16:25:20.953647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.967920+0000","last_change":"2024-09-13T16:25:20.965561+0000","last_active":"2024-09-13T16:25:20.967920+0000","last_peered":"2024-09-13T16:25:20.967920+0000","last_clean":"2024-09-13T16:25:20.967920+0000","last_became_active":"2024-09-13T16:25:20.965162+0000","last_became_peered":"2024-09-13T16:25:20.965162+0000","last_unstale":"2024-09-13T16:25:20.967920+0000","last_undegraded":"2024-09-13T16:25:20.967920+0000","last_fullsized":"2024-09-13T16:25:20.967920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:35.284928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940864+0000","last_change":"2024-09-13T16:25:16.560967+0000","last_active":"2024-09-13T16:25:20.940864+0000","last_peered":"2024-09-13T16:25:20.940864+0000","last_clean":"2024-09-13T16:25:20.940864+0000","last_became_active":"2024-09-13T16:25:16.560675+0000","last_became_peered":"2024-09-13T16:25:16.560675+0000","last_unstale":"2024-09-13T16:25:20.940864+0000","last_undegraded":"2024-09-13T16:25:20.940864+0000","last_fullsized":"2024-09-13T16:25:20.940864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951187+0000","last_change":"2024-09-13T16:25:16.936540+0000","last_active":"2024-09-13T16:25:20.951187+0000","last_peered":"2024-09-13T16:25:20.951187+0000","last_clean":"2024-09-13T16:25:20.951187+0000","last_became_active":"2024-09-13T16:25:16.936269+0000","last_became_peered":"2024-09-13T16:25:16.936269+0000","last_unstale":"2024-09-13T16:25:20.951187+0000","last_undegraded":"2024-09-13T16:25:20.951187+0000","last_fullsized":"2024-09-13T16:25:20.951187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:18.089801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963605+0000","last_change":"2024-09-13T16:25:20.961207+0000","last_active":"2024-09-13T16:25:20.963605+0000","last_peered":"2024-09-13T16:25:20.963605+0000","last_clean":"2024-09-13T16:25:20.963605+0000","last_became_active":"2024-09-13T16:25:20.960801+0000","last_became_peered":"2024-09-13T16:25:20.960801+0000","last_unstale":"2024-09-13T16:25:20.963605+0000","last_undegraded":"2024-09-13T16:25:20.963605+0000","last_fullsized":"2024-09-13T16:25:20.963605+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:38.874981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939239+0000","last_change":"2024-09-13T16:25:16.939880+0000","last_active":"2024-09-13T16:25:20.939239+0000","last_peered":"2024-09-13T16:25:20.939239+0000","last_clean":"2024-09-13T16:25:20.939239+0000","last_became_active":"2024-09-13T16:25:16.939579+0000","last_became_peered":"2024-09-13T16:25:16.939579+0000","last_unstale":"2024-09-13T16:25:20.939239+0000","last_undegraded":"2024-09-13T16:25:20.939239+0000","last_fullsized":"2024-09-13T16:25:20.939239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:37.417453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953935+0000","last_change":"2024-09-13T16:25:16.938944+0000","last_active":"2024-09-13T16:25:20.953935+0000","last_peered":"2024-09-13T16:25:20.953935+0000","last_clean":"2024-09-13T16:25:20.953935+0000","last_became_active":"2024-09-13T16:25:16.938658+0000","last_became_peered":"2024-09-13T16:25:16.938658+0000","last_unstale":"2024-09-13T16:25:20.953935+0000","last_undegraded":"2024-09-13T16:25:20.953935+0000","last_fullsized":"2024-09-13T16:25:20.953935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:50.130499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941580+0000","last_change":"2024-09-13T16:25:16.933588+0000","last_active":"2024-09-13T16:25:20.941580+0000","last_peered":"2024-09-13T16:25:20.941580+0000","last_clean":"2024-09-13T16:25:20.941580+0000","last_became_active":"2024-09-13T16:25:16.932300+0000","last_became_peered":"2024-09-13T16:25:16.932300+0000","last_unstale":"2024-09-13T16:25:20.941580+0000","last_undegraded":"2024-09-13T16:25:20.941580+0000","last_fullsized":"2024-09-13T16:25:20.941580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:43.437120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953565+0000","last_change":"2024-09-13T16:25:16.934119+0000","last_active":"2024-09-13T16:25:20.953565+0000","last_peered":"2024-09-13T16:25:20.953565+0000","last_clean":"2024-09-13T16:25:20.953565+0000","last_became_active":"2024-09-13T16:25:16.930259+0000","last_became_peered":"2024-09-13T16:25:16.930259+0000","last_unstale":"2024-09-13T16:25:20.953565+0000","last_undegraded":"2024-09-13T16:25:20.953565+0000","last_fullsized":"2024-09-13T16:25:20.953565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941316+0000","last_change":"2024-09-13T16:25:16.931922+0000","last_active":"2024-09-13T16:25:20.941316+0000","last_peered":"2024-09-13T16:25:20.941316+0000","last_clean":"2024-09-13T16:25:20.941316+0000","last_became_active":"2024-09-13T16:25:16.931678+0000","last_became_peered":"2024-09-13T16:25:16.931678+0000","last_unstale":"2024-09-13T16:25:20.941316+0000","last_undegraded":"2024-09-13T16:25:20.941316+0000","last_fullsized":"2024-09-13T16:25:20.941316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:27.822409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952316+0000","last_change":"2024-09-13T16:25:16.932526+0000","last_active":"2024-09-13T16:25:20.952316+0000","last_peered":"2024-09-13T16:25:20.952316+0000","last_clean":"2024-09-13T16:25:20.952316+0000","last_became_active":"2024-09-13T16:25:16.932189+0000","last_became_peered":"2024-09-13T16:25:16.932189+0000","last_unstale":"2024-09-13T16:25:20.952316+0000","last_undegraded":"2024-09-13T16:25:20.952316+0000","last_fullsized":"2024-09-13T16:25:20.952316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941353+0000","last_change":"2024-09-13T16:25:16.947538+0000","last_active":"2024-09-13T16:25:20.941353+0000","last_peered":"2024-09-13T16:25:20.941353+0000","last_clean":"2024-09-13T16:25:20.941353+0000","last_became_active":"2024-09-13T16:25:16.947272+0000","last_became_peered":"2024-09-13T16:25:16.947272+0000","last_unstale":"2024-09-13T16:25:20.941353+0000","last_undegraded":"2024-09-13T16:25:20.941353+0000","last_fullsized":"2024-09-13T16:25:20.941353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940064+0000","last_change":"2024-09-13T16:25:16.932732+0000","last_active":"2024-09-13T16:25:20.940064+0000","last_peered":"2024-09-13T16:25:20.940064+0000","last_clean":"2024-09-13T16:25:20.940064+0000","last_became_active":"2024-09-13T16:25:16.932505+0000","last_became_peered":"2024-09-13T16:25:16.932505+0000","last_unstale":"2024-09-13T16:25:20.940064+0000","last_undegraded":"2024-09-13T16:25:20.940064+0000","last_fullsized":"2024-09-13T16:25:20.940064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942122+0000","last_change":"2024-09-13T16:25:16.933750+0000","last_active":"2024-09-13T16:25:20.942122+0000","last_peered":"2024-09-13T16:25:20.942122+0000","last_clean":"2024-09-13T16:25:20.942122+0000","last_became_active":"2024-09-13T16:25:16.933509+0000","last_became_peered":"2024-09-13T16:25:16.933509+0000","last_unstale":"2024-09-13T16:25:20.942122+0000","last_undegraded":"2024-09-13T16:25:20.942122+0000","last_fullsized":"2024-09-13T16:25:20.942122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:28.433477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939801+0000","last_change":"2024-09-13T16:25:16.932135+0000","last_active":"2024-09-13T16:25:20.939801+0000","last_peered":"2024-09-13T16:25:20.939801+0000","last_clean":"2024-09-13T16:25:20.939801+0000","last_became_active":"2024-09-13T16:25:16.931865+0000","last_became_peered":"2024-09-13T16:25:16.931865+0000","last_unstale":"2024-09-13T16:25:20.939801+0000","last_undegraded":"2024-09-13T16:25:20.939801+0000","last_fullsized":"2024-09-13T16:25:20.939801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941544+0000","last_change":"2024-09-13T16:25:16.942044+0000","last_active":"2024-09-13T16:25:20.941544+0000","last_peered":"2024-09-13T16:25:20.941544+0000","last_clean":"2024-09-13T16:25:20.941544+0000","last_became_active":"2024-09-13T16:25:16.941738+0000","last_became_peered":"2024-09-13T16:25:16.941738+0000","last_unstale":"2024-09-13T16:25:20.941544+0000","last_undegraded":"2024-09-13T16:25:20.941544+0000","last_fullsized":"2024-09-13T16:25:20.941544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:58:43.832593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948993+0000","last_change":"2024-09-13T16:25:16.942581+0000","last_active":"2024-09-13T16:25:20.948993+0000","last_peered":"2024-09-13T16:25:20.948993+0000","last_clean":"2024-09-13T16:25:20.948993+0000","last_became_active":"2024-09-13T16:25:16.942321+0000","last_became_peered":"2024-09-13T16:25:16.942321+0000","last_unstale":"2024-09-13T16:25:20.948993+0000","last_undegraded":"2024-09-13T16:25:20.948993+0000","last_fullsized":"2024-09-13T16:25:20.948993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939626+0000","last_change":"2024-09-13T16:25:16.936450+0000","last_active":"2024-09-13T16:25:20.939626+0000","last_peered":"2024-09-13T16:25:20.939626+0000","last_clean":"2024-09-13T16:25:20.939626+0000","last_became_active":"2024-09-13T16:25:16.936084+0000","last_became_peered":"2024-09-13T16:25:16.936084+0000","last_unstale":"2024-09-13T16:25:20.939626+0000","last_undegraded":"2024-09-13T16:25:20.939626+0000","last_fullsized":"2024-09-13T16:25:20.939626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150470+0000","last_change":"2024-09-13T16:25:16.948876+0000","last_active":"2024-09-13T16:25:21.150470+0000","last_peered":"2024-09-13T16:25:21.150470+0000","last_clean":"2024-09-13T16:25:21.150470+0000","last_became_active":"2024-09-13T16:25:16.948536+0000","last_became_peered":"2024-09-13T16:25:16.948536+0000","last_unstale":"2024-09-13T16:25:21.150470+0000","last_undegraded":"2024-09-13T16:25:21.150470+0000","last_fullsized":"2024-09-13T16:25:21.150470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959182+0000","last_change":"2024-09-13T16:25:16.937371+0000","last_active":"2024-09-13T16:25:20.959182+0000","last_peered":"2024-09-13T16:25:20.959182+0000","last_clean":"2024-09-13T16:25:20.959182+0000","last_became_active":"2024-09-13T16:25:16.937130+0000","last_became_peered":"2024-09-13T16:25:16.937130+0000","last_unstale":"2024-09-13T16:25:20.959182+0000","last_undegraded":"2024-09-13T16:25:20.959182+0000","last_fullsized":"2024-09-13T16:25:20.959182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:29.816675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944314+0000","last_change":"2024-09-13T16:25:16.947064+0000","last_active":"2024-09-13T16:25:20.944314+0000","last_peered":"2024-09-13T16:25:20.944314+0000","last_clean":"2024-09-13T16:25:20.944314+0000","last_became_active":"2024-09-13T16:25:16.946790+0000","last_became_peered":"2024-09-13T16:25:16.946790+0000","last_unstale":"2024-09-13T16:25:20.944314+0000","last_undegraded":"2024-09-13T16:25:20.944314+0000","last_fullsized":"2024-09-13T16:25:20.944314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.978002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942037+0000","last_change":"2024-09-13T16:25:16.931584+0000","last_active":"2024-09-13T16:25:20.942037+0000","last_peered":"2024-09-13T16:25:20.942037+0000","last_clean":"2024-09-13T16:25:20.942037+0000","last_became_active":"2024-09-13T16:25:16.931224+0000","last_became_peered":"2024-09-13T16:25:16.931224+0000","last_unstale":"2024-09-13T16:25:20.942037+0000","last_undegraded":"2024-09-13T16:25:20.942037+0000","last_fullsized":"2024-09-13T16:25:20.942037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:11.767256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949799+0000","last_change":"2024-09-13T16:25:16.936233+0000","last_active":"2024-09-13T16:25:20.949799+0000","last_peered":"2024-09-13T16:25:20.949799+0000","last_clean":"2024-09-13T16:25:20.949799+0000","last_became_active":"2024-09-13T16:25:16.935935+0000","last_became_peered":"2024-09-13T16:25:16.935935+0000","last_unstale":"2024-09-13T16:25:20.949799+0000","last_undegraded":"2024-09-13T16:25:20.949799+0000","last_fullsized":"2024-09-13T16:25:20.949799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946795+0000","last_change":"2024-09-13T16:25:16.936516+0000","last_active":"2024-09-13T16:25:20.946795+0000","last_peered":"2024-09-13T16:25:20.946795+0000","last_clean":"2024-09-13T16:25:20.946795+0000","last_became_active":"2024-09-13T16:25:16.936281+0000","last_became_peered":"2024-09-13T16:25:16.936281+0000","last_unstale":"2024-09-13T16:25:20.946795+0000","last_undegraded":"2024-09-13T16:25:20.946795+0000","last_fullsized":"2024-09-13T16:25:20.946795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940457+0000","last_change":"2024-09-13T16:25:16.938439+0000","last_active":"2024-09-13T16:25:20.940457+0000","last_peered":"2024-09-13T16:25:20.940457+0000","last_clean":"2024-09-13T16:25:20.940457+0000","last_became_active":"2024-09-13T16:25:16.938204+0000","last_became_peered":"2024-09-13T16:25:16.938204+0000","last_unstale":"2024-09-13T16:25:20.940457+0000","last_undegraded":"2024-09-13T16:25:20.940457+0000","last_fullsized":"2024-09-13T16:25:20.940457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951414+0000","last_change":"2024-09-13T16:25:16.931605+0000","last_active":"2024-09-13T16:25:20.951414+0000","last_peered":"2024-09-13T16:25:20.951414+0000","last_clean":"2024-09-13T16:25:20.951414+0000","last_became_active":"2024-09-13T16:25:16.931330+0000","last_became_peered":"2024-09-13T16:25:16.931330+0000","last_unstale":"2024-09-13T16:25:20.951414+0000","last_undegraded":"2024-09-13T16:25:20.951414+0000","last_fullsized":"2024-09-13T16:25:20.951414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:51.688093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940628+0000","last_change":"2024-09-13T16:25:16.936567+0000","last_active":"2024-09-13T16:25:20.940628+0000","last_peered":"2024-09-13T16:25:20.940628+0000","last_clean":"2024-09-13T16:25:20.940628+0000","last_became_active":"2024-09-13T16:25:16.936076+0000","last_became_peered":"2024-09-13T16:25:16.936076+0000","last_unstale":"2024-09-13T16:25:20.940628+0000","last_undegraded":"2024-09-13T16:25:20.940628+0000","last_fullsized":"2024-09-13T16:25:20.940628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941217+0000","last_change":"2024-09-13T16:25:16.936360+0000","last_active":"2024-09-13T16:25:20.941217+0000","last_peered":"2024-09-13T16:25:20.941217+0000","last_clean":"2024-09-13T16:25:20.941217+0000","last_became_active":"2024-09-13T16:25:16.935361+0000","last_became_peered":"2024-09-13T16:25:16.935361+0000","last_unstale":"2024-09-13T16:25:20.941217+0000","last_undegraded":"2024-09-13T16:25:20.941217+0000","last_fullsized":"2024-09-13T16:25:20.941217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150991+0000","last_change":"2024-09-13T16:25:16.951709+0000","last_active":"2024-09-13T16:25:21.150991+0000","last_peered":"2024-09-13T16:25:21.150991+0000","last_clean":"2024-09-13T16:25:21.150991+0000","last_became_active":"2024-09-13T16:25:16.951465+0000","last_became_peered":"2024-09-13T16:25:16.951465+0000","last_unstale":"2024-09-13T16:25:21.150991+0000","last_undegraded":"2024-09-13T16:25:21.150991+0000","last_fullsized":"2024-09-13T16:25:21.150991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:05.832843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.938568+0000","last_change":"2024-09-13T16:25:16.933931+0000","last_active":"2024-09-13T16:25:20.938568+0000","last_peered":"2024-09-13T16:25:20.938568+0000","last_clean":"2024-09-13T16:25:20.938568+0000","last_became_active":"2024-09-13T16:25:16.933646+0000","last_became_peered":"2024-09-13T16:25:16.933646+0000","last_unstale":"2024-09-13T16:25:20.938568+0000","last_undegraded":"2024-09-13T16:25:20.938568+0000","last_fullsized":"2024-09-13T16:25:20.938568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150544+0000","last_change":"2024-09-13T16:25:16.941658+0000","last_active":"2024-09-13T16:25:21.150544+0000","last_peered":"2024-09-13T16:25:21.150544+0000","last_clean":"2024-09-13T16:25:21.150544+0000","last_became_active":"2024-09-13T16:25:16.941375+0000","last_became_peered":"2024-09-13T16:25:16.941375+0000","last_unstale":"2024-09-13T16:25:21.150544+0000","last_undegraded":"2024-09-13T16:25:21.150544+0000","last_fullsized":"2024-09-13T16:25:21.150544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:05.301510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941428+0000","last_change":"2024-09-13T16:25:16.931253+0000","last_active":"2024-09-13T16:25:20.941428+0000","last_peered":"2024-09-13T16:25:20.941428+0000","last_clean":"2024-09-13T16:25:20.941428+0000","last_became_active":"2024-09-13T16:25:16.930844+0000","last_became_peered":"2024-09-13T16:25:16.930844+0000","last_unstale":"2024-09-13T16:25:20.941428+0000","last_undegraded":"2024-09-13T16:25:20.941428+0000","last_fullsized":"2024-09-13T16:25:20.941428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:11.873813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149733+0000","last_change":"2024-09-13T16:25:16.946470+0000","last_active":"2024-09-13T16:25:21.149733+0000","last_peered":"2024-09-13T16:25:21.149733+0000","last_clean":"2024-09-13T16:25:21.149733+0000","last_became_active":"2024-09-13T16:25:16.946228+0000","last_became_peered":"2024-09-13T16:25:16.946228+0000","last_unstale":"2024-09-13T16:25:21.149733+0000","last_undegraded":"2024-09-13T16:25:21.149733+0000","last_fullsized":"2024-09-13T16:25:21.149733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:51.187953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943747+0000","last_change":"2024-09-13T16:25:16.933763+0000","last_active":"2024-09-13T16:25:20.943747+0000","last_peered":"2024-09-13T16:25:20.943747+0000","last_clean":"2024-09-13T16:25:20.943747+0000","last_became_active":"2024-09-13T16:25:16.932396+0000","last_became_peered":"2024-09-13T16:25:16.932396+0000","last_unstale":"2024-09-13T16:25:20.943747+0000","last_undegraded":"2024-09-13T16:25:20.943747+0000","last_fullsized":"2024-09-13T16:25:20.943747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:13.889786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943715+0000","last_change":"2024-09-13T16:25:16.931365+0000","last_active":"2024-09-13T16:25:20.943715+0000","last_peered":"2024-09-13T16:25:20.943715+0000","last_clean":"2024-09-13T16:25:20.943715+0000","last_became_active":"2024-09-13T16:25:16.931117+0000","last_became_peered":"2024-09-13T16:25:16.931117+0000","last_unstale":"2024-09-13T16:25:20.943715+0000","last_undegraded":"2024-09-13T16:25:20.943715+0000","last_fullsized":"2024-09-13T16:25:20.943715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.578965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953517+0000","last_change":"2024-09-13T16:25:16.932889+0000","last_active":"2024-09-13T16:25:20.953517+0000","last_peered":"2024-09-13T16:25:20.953517+0000","last_clean":"2024-09-13T16:25:20.953517+0000","last_became_active":"2024-09-13T16:25:16.931952+0000","last_became_peered":"2024-09-13T16:25:16.931952+0000","last_unstale":"2024-09-13T16:25:20.953517+0000","last_undegraded":"2024-09-13T16:25:20.953517+0000","last_fullsized":"2024-09-13T16:25:20.953517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150702+0000","last_change":"2024-09-13T16:25:16.948699+0000","last_active":"2024-09-13T16:25:21.150702+0000","last_peered":"2024-09-13T16:25:21.150702+0000","last_clean":"2024-09-13T16:25:21.150702+0000","last_became_active":"2024-09-13T16:25:16.948387+0000","last_became_peered":"2024-09-13T16:25:16.948387+0000","last_unstale":"2024-09-13T16:25:21.150702+0000","last_undegraded":"2024-09-13T16:25:21.150702+0000","last_fullsized":"2024-09-13T16:25:21.150702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:28.568881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":693951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":266240,"data_stored":103091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":103127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":103001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":103161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":694007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:25:34.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:25:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921c5c3700 1 Processor -- start 2024-09-13T16:25:34.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921c5c3700 1 -- start start 2024-09-13T16:25:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921c5c3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f92140b3ad0 0x7f9214159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921c5c3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9214159880 0x7f92140cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921c5c3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9214159d00 con 0x7f92140b36f0 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921c5c3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92140b2050 con 0x7f92140b3ad0 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921c5c3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92140b2320 con 0x7f9214159880 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f92140b3ad0 0x7f9214159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f92140b3ad0 0x7f9214159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52000/0 (socket says 172.21.15.119:52000) 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 -- 172.21.15.119:0/1044697448 learned_addr learned my addr 172.21.15.119:0/1044697448 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921ab60700 1 --2- 172.21.15.119:0/1044697448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9214159880 0x7f92140cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 -- 172.21.15.119:0/1044697448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9214159880 msgr2=0x7f92140cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 --2- 172.21.15.119:0/1044697448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9214159880 0x7f92140cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 -- 172.21.15.119:0/1044697448 >> v1:172.21.15.145:6789/0 conn(0x7f92140b36f0 legacy=0x7f92140b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 -- 172.21.15.119:0/1044697448 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92140b2600 con 0x7f92140b3ad0 2024-09-13T16:25:34.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f9219b5e700 1 --2- 172.21.15.119:0/1044697448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f9214159340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9204009e50 tx=0x7f920400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d77cd5592b51f36 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921935d700 1 -- 172.21.15.119:0/1044697448 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920400e070 con 0x7f92140b3ad0 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921935d700 1 -- 172.21.15.119:0/1044697448 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f920400b9c0 con 0x7f92140b3ad0 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.201+0000 7f921935d700 1 -- 172.21.15.119:0/1044697448 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920400bd70 con 0x7f92140b3ad0 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 -- 172.21.15.119:0/1044697448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 msgr2=0x7f9214159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 --2- 172.21.15.119:0/1044697448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f9214159340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9204009e50 tx=0x7f920400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 -- 172.21.15.119:0/1044697448 shutdown_connections 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 --2- 172.21.15.119:0/1044697448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9214159880 0x7f92140cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 --2- 172.21.15.119:0/1044697448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f9214159340 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 -- 172.21.15.119:0/1044697448 >> 172.21.15.119:0/1044697448 conn(0x7f921406a350 msgr2=0x7f921406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:34.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 -- 172.21.15.119:0/1044697448 shutdown_connections 2024-09-13T16:25:34.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 -- 172.21.15.119:0/1044697448 wait complete. 2024-09-13T16:25:34.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 Processor -- start 2024-09-13T16:25:34.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.205+0000 7f921c5c3700 1 -- start start 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92140b36f0 0x7f9214140040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f921413cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9214159880 0x7f921415c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9214152d30 con 0x7f92140b3ad0 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92141528a0 con 0x7f9214159880 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9214140580 con 0x7f92140b36f0 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f921413cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f921413cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52010/0 (socket says 172.21.15.119:52010) 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 -- 172.21.15.119:0/2703117218 learned_addr learned my addr 172.21.15.119:0/2703117218 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921a35f700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92140b36f0 0x7f9214140040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921ab60700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9214159880 0x7f921415c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 -- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92140b36f0 msgr2=0x7f9214140040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92140b36f0 0x7f9214140040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 -- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9214159880 msgr2=0x7f921415c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9214159880 0x7f921415c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 -- 172.21.15.119:0/2703117218 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9204009b30 con 0x7f92140b3ad0 2024-09-13T16:25:34.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f9219b5e700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f921413cd10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f920400b370 tx=0x7f9204004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f920affd700 1 -- 172.21.15.119:0/2703117218 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9204004800 con 0x7f92140b3ad0 2024-09-13T16:25:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f920affd700 1 -- 172.21.15.119:0/2703117218 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92040092e0 con 0x7f92140b3ad0 2024-09-13T16:25:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f920affd700 1 -- 172.21.15.119:0/2703117218 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920401c680 con 0x7f92140b3ad0 2024-09-13T16:25:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9214151c90 con 0x7f92140b3ad0 2024-09-13T16:25:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.209+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9214152250 con 0x7f92140b3ad0 2024-09-13T16:25:34.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.213+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9214152a30 con 0x7f92140b3ad0 2024-09-13T16:25:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.213+0000 7f920affd700 1 -- 172.21.15.119:0/2703117218 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f920401c7e0 con 0x7f92140b3ad0 2024-09-13T16:25:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.213+0000 7f920affd700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9200066760 0x7f9200068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.213+0000 7f920affd700 1 -- 172.21.15.119:0/2703117218 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9204085fe0 con 0x7f92140b3ad0 2024-09-13T16:25:34.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.213+0000 7f921a35f700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9200066760 0x7f9200068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.213+0000 7f921a35f700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9200066760 0x7f9200068c20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9210000f60 tx=0x7f9210009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.221+0000 7f920affd700 1 -- 172.21.15.119:0/2703117218 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9204053050 con 0x7f92140b3ad0 2024-09-13T16:25:34.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.389+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f92140b2bc0 con 0x7f9200066760 2024-09-13T16:25:34.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.405+0000 7f920affd700 1 -- 172.21.15.119:0/2703117218 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345162 (secure 0 0 0) 0x7f92100015e0 con 0x7f9200066760 2024-09-13T16:25:34.409 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:34.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9200066760 msgr2=0x7f9200068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9200066760 0x7f9200068c20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9210000f60 tx=0x7f9210009040 comp rx=0 tx=0).stop 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 msgr2=0x7f921413cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f921413cd10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f920400b370 tx=0x7f9204004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 shutdown_connections 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9200066760 0x7f9200068c20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9214159880 0x7f921415c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f92140b3ad0 0x7f921413cd10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 --2- 172.21.15.119:0/2703117218 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f92140b36f0 0x7f9214140040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 >> 172.21.15.119:0/2703117218 conn(0x7f921406a350 msgr2=0x7f921406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 shutdown_connections 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.409+0000 7f921c5c3700 1 -- 172.21.15.119:0/2703117218 wait complete. 2024-09-13T16:25:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:25:34.426 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2024-09-13T16:25:33.423968+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219456,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755264,"statfs":{"total":773094113280,"available":772869390336,"internally_reserved":0,"allocated":3350528,"data_stored":2013930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2231,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005383"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954165+0000","last_change":"2024-09-13T16:25:16.560322+0000","last_active":"2024-09-13T16:25:20.954165+0000","last_peered":"2024-09-13T16:25:20.954165+0000","last_clean":"2024-09-13T16:25:20.954165+0000","last_became_active":"2024-09-13T16:25:16.560130+0000","last_became_peered":"2024-09-13T16:25:16.560130+0000","last_unstale":"2024-09-13T16:25:20.954165+0000","last_undegraded":"2024-09-13T16:25:20.954165+0000","last_fullsized":"2024-09-13T16:25:20.954165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941703+0000","last_change":"2024-09-13T16:25:16.937844+0000","last_active":"2024-09-13T16:25:20.941703+0000","last_peered":"2024-09-13T16:25:20.941703+0000","last_clean":"2024-09-13T16:25:20.941703+0000","last_became_active":"2024-09-13T16:25:16.937508+0000","last_became_peered":"2024-09-13T16:25:16.937508+0000","last_unstale":"2024-09-13T16:25:20.941703+0000","last_undegraded":"2024-09-13T16:25:20.941703+0000","last_fullsized":"2024-09-13T16:25:20.941703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:46.073323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163484+0000","last_change":"2024-09-13T16:25:21.162079+0000","last_active":"2024-09-13T16:25:21.163484+0000","last_peered":"2024-09-13T16:25:21.163484+0000","last_clean":"2024-09-13T16:25:21.163484+0000","last_became_active":"2024-09-13T16:25:21.161341+0000","last_became_peered":"2024-09-13T16:25:21.161341+0000","last_unstale":"2024-09-13T16:25:21.163484+0000","last_undegraded":"2024-09-13T16:25:21.163484+0000","last_fullsized":"2024-09-13T16:25:21.163484+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:55.099223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941956+0000","last_change":"2024-09-13T16:25:16.554751+0000","last_active":"2024-09-13T16:25:20.941956+0000","last_peered":"2024-09-13T16:25:20.941956+0000","last_clean":"2024-09-13T16:25:20.941956+0000","last_became_active":"2024-09-13T16:25:16.552873+0000","last_became_peered":"2024-09-13T16:25:16.552873+0000","last_unstale":"2024-09-13T16:25:20.941956+0000","last_undegraded":"2024-09-13T16:25:20.941956+0000","last_fullsized":"2024-09-13T16:25:20.941956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.947694+0000","last_change":"2024-09-13T16:25:16.933368+0000","last_active":"2024-09-13T16:25:20.947694+0000","last_peered":"2024-09-13T16:25:20.947694+0000","last_clean":"2024-09-13T16:25:20.947694+0000","last_became_active":"2024-09-13T16:25:16.932824+0000","last_became_peered":"2024-09-13T16:25:16.932824+0000","last_unstale":"2024-09-13T16:25:20.947694+0000","last_undegraded":"2024-09-13T16:25:20.947694+0000","last_fullsized":"2024-09-13T16:25:20.947694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969023+0000","last_change":"2024-09-13T16:25:20.965564+0000","last_active":"2024-09-13T16:25:20.969023+0000","last_peered":"2024-09-13T16:25:20.969023+0000","last_clean":"2024-09-13T16:25:20.969023+0000","last_became_active":"2024-09-13T16:25:20.965159+0000","last_became_peered":"2024-09-13T16:25:20.965159+0000","last_unstale":"2024-09-13T16:25:20.969023+0000","last_undegraded":"2024-09-13T16:25:20.969023+0000","last_fullsized":"2024-09-13T16:25:20.969023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150484+0000","last_change":"2024-09-13T16:25:16.560318+0000","last_active":"2024-09-13T16:25:21.150484+0000","last_peered":"2024-09-13T16:25:21.150484+0000","last_clean":"2024-09-13T16:25:21.150484+0000","last_became_active":"2024-09-13T16:25:16.560069+0000","last_became_peered":"2024-09-13T16:25:16.560069+0000","last_unstale":"2024-09-13T16:25:21.150484+0000","last_undegraded":"2024-09-13T16:25:21.150484+0000","last_fullsized":"2024-09-13T16:25:21.150484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:12.814872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943883+0000","last_change":"2024-09-13T16:25:16.935607+0000","last_active":"2024-09-13T16:25:20.943883+0000","last_peered":"2024-09-13T16:25:20.943883+0000","last_clean":"2024-09-13T16:25:20.943883+0000","last_became_active":"2024-09-13T16:25:16.935353+0000","last_became_peered":"2024-09-13T16:25:16.935353+0000","last_unstale":"2024-09-13T16:25:20.943883+0000","last_undegraded":"2024-09-13T16:25:20.943883+0000","last_fullsized":"2024-09-13T16:25:20.943883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:35.981913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973971+0000","last_change":"2024-09-13T16:25:20.969324+0000","last_active":"2024-09-13T16:25:20.973971+0000","last_peered":"2024-09-13T16:25:20.973971+0000","last_clean":"2024-09-13T16:25:20.973971+0000","last_became_active":"2024-09-13T16:25:20.968916+0000","last_became_peered":"2024-09-13T16:25:20.968916+0000","last_unstale":"2024-09-13T16:25:20.973971+0000","last_undegraded":"2024-09-13T16:25:20.973971+0000","last_fullsized":"2024-09-13T16:25:20.973971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:03.825482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149974+0000","last_change":"2024-09-13T16:25:16.560280+0000","last_active":"2024-09-13T16:25:21.149974+0000","last_peered":"2024-09-13T16:25:21.149974+0000","last_clean":"2024-09-13T16:25:21.149974+0000","last_became_active":"2024-09-13T16:25:16.559981+0000","last_became_peered":"2024-09-13T16:25:16.559981+0000","last_unstale":"2024-09-13T16:25:21.149974+0000","last_undegraded":"2024-09-13T16:25:21.149974+0000","last_fullsized":"2024-09-13T16:25:21.149974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943411+0000","last_change":"2024-09-13T16:25:16.937364+0000","last_active":"2024-09-13T16:25:20.943411+0000","last_peered":"2024-09-13T16:25:20.943411+0000","last_clean":"2024-09-13T16:25:20.943411+0000","last_became_active":"2024-09-13T16:25:16.937119+0000","last_became_peered":"2024-09-13T16:25:16.937119+0000","last_unstale":"2024-09-13T16:25:20.943411+0000","last_undegraded":"2024-09-13T16:25:20.943411+0000","last_fullsized":"2024-09-13T16:25:20.943411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:31.885317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959982+0000","last_change":"2024-09-13T16:25:20.959414+0000","last_active":"2024-09-13T16:25:20.959982+0000","last_peered":"2024-09-13T16:25:20.959982+0000","last_clean":"2024-09-13T16:25:20.959982+0000","last_became_active":"2024-09-13T16:25:20.958955+0000","last_became_peered":"2024-09-13T16:25:20.958955+0000","last_unstale":"2024-09-13T16:25:20.959982+0000","last_undegraded":"2024-09-13T16:25:20.959982+0000","last_fullsized":"2024-09-13T16:25:20.959982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:35.667608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939712+0000","last_change":"2024-09-13T16:25:16.936925+0000","last_active":"2024-09-13T16:25:20.939712+0000","last_peered":"2024-09-13T16:25:20.939712+0000","last_clean":"2024-09-13T16:25:20.939712+0000","last_became_active":"2024-09-13T16:25:16.936608+0000","last_became_peered":"2024-09-13T16:25:16.936608+0000","last_unstale":"2024-09-13T16:25:20.939712+0000","last_undegraded":"2024-09-13T16:25:20.939712+0000","last_fullsized":"2024-09-13T16:25:20.939712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150049+0000","last_change":"2024-09-13T16:25:16.561316+0000","last_active":"2024-09-13T16:25:21.150049+0000","last_peered":"2024-09-13T16:25:21.150049+0000","last_clean":"2024-09-13T16:25:21.150049+0000","last_became_active":"2024-09-13T16:25:16.560850+0000","last_became_peered":"2024-09-13T16:25:16.560850+0000","last_unstale":"2024-09-13T16:25:21.150049+0000","last_undegraded":"2024-09-13T16:25:21.150049+0000","last_fullsized":"2024-09-13T16:25:21.150049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.961598+0000","last_change":"2024-09-13T16:25:20.960831+0000","last_active":"2024-09-13T16:25:20.961598+0000","last_peered":"2024-09-13T16:25:20.961598+0000","last_clean":"2024-09-13T16:25:20.961598+0000","last_became_active":"2024-09-13T16:25:20.960495+0000","last_became_peered":"2024-09-13T16:25:20.960495+0000","last_unstale":"2024-09-13T16:25:20.961598+0000","last_undegraded":"2024-09-13T16:25:20.961598+0000","last_fullsized":"2024-09-13T16:25:20.961598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940654+0000","last_change":"2024-09-13T16:25:16.937601+0000","last_active":"2024-09-13T16:25:20.940654+0000","last_peered":"2024-09-13T16:25:20.940654+0000","last_clean":"2024-09-13T16:25:20.940654+0000","last_became_active":"2024-09-13T16:25:16.937343+0000","last_became_peered":"2024-09-13T16:25:16.937343+0000","last_unstale":"2024-09-13T16:25:20.940654+0000","last_undegraded":"2024-09-13T16:25:20.940654+0000","last_fullsized":"2024-09-13T16:25:20.940654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950384+0000","last_change":"2024-09-13T16:25:16.550175+0000","last_active":"2024-09-13T16:25:20.950384+0000","last_peered":"2024-09-13T16:25:20.950384+0000","last_clean":"2024-09-13T16:25:20.950384+0000","last_became_active":"2024-09-13T16:25:16.549875+0000","last_became_peered":"2024-09-13T16:25:16.549875+0000","last_unstale":"2024-09-13T16:25:20.950384+0000","last_undegraded":"2024-09-13T16:25:20.950384+0000","last_fullsized":"2024-09-13T16:25:20.950384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973920+0000","last_change":"2024-09-13T16:25:20.968856+0000","last_active":"2024-09-13T16:25:20.973920+0000","last_peered":"2024-09-13T16:25:20.973920+0000","last_clean":"2024-09-13T16:25:20.973920+0000","last_became_active":"2024-09-13T16:25:20.968470+0000","last_became_peered":"2024-09-13T16:25:20.968470+0000","last_unstale":"2024-09-13T16:25:20.973920+0000","last_undegraded":"2024-09-13T16:25:20.973920+0000","last_fullsized":"2024-09-13T16:25:20.973920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940018+0000","last_change":"2024-09-13T16:25:16.936551+0000","last_active":"2024-09-13T16:25:20.940018+0000","last_peered":"2024-09-13T16:25:20.940018+0000","last_clean":"2024-09-13T16:25:20.940018+0000","last_became_active":"2024-09-13T16:25:16.936298+0000","last_became_peered":"2024-09-13T16:25:16.936298+0000","last_unstale":"2024-09-13T16:25:20.940018+0000","last_undegraded":"2024-09-13T16:25:20.940018+0000","last_fullsized":"2024-09-13T16:25:20.940018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949896+0000","last_change":"2024-09-13T16:25:16.556208+0000","last_active":"2024-09-13T16:25:20.949896+0000","last_peered":"2024-09-13T16:25:20.949896+0000","last_clean":"2024-09-13T16:25:20.949896+0000","last_became_active":"2024-09-13T16:25:16.555897+0000","last_became_peered":"2024-09-13T16:25:16.555897+0000","last_unstale":"2024-09-13T16:25:20.949896+0000","last_undegraded":"2024-09-13T16:25:20.949896+0000","last_fullsized":"2024-09-13T16:25:20.949896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.164051+0000","last_change":"2024-09-13T16:25:21.162085+0000","last_active":"2024-09-13T16:25:21.164051+0000","last_peered":"2024-09-13T16:25:21.164051+0000","last_clean":"2024-09-13T16:25:21.164051+0000","last_became_active":"2024-09-13T16:25:21.160633+0000","last_became_peered":"2024-09-13T16:25:21.160633+0000","last_unstale":"2024-09-13T16:25:21.164051+0000","last_undegraded":"2024-09-13T16:25:21.164051+0000","last_fullsized":"2024-09-13T16:25:21.164051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951220+0000","last_change":"2024-09-13T16:25:16.931406+0000","last_active":"2024-09-13T16:25:20.951220+0000","last_peered":"2024-09-13T16:25:20.951220+0000","last_clean":"2024-09-13T16:25:20.951220+0000","last_became_active":"2024-09-13T16:25:16.931073+0000","last_became_peered":"2024-09-13T16:25:16.931073+0000","last_unstale":"2024-09-13T16:25:20.951220+0000","last_undegraded":"2024-09-13T16:25:20.951220+0000","last_fullsized":"2024-09-13T16:25:20.951220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:31.812511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149390+0000","last_change":"2024-09-13T16:25:16.562667+0000","last_active":"2024-09-13T16:25:21.149390+0000","last_peered":"2024-09-13T16:25:21.149390+0000","last_clean":"2024-09-13T16:25:21.149390+0000","last_became_active":"2024-09-13T16:25:16.562361+0000","last_became_peered":"2024-09-13T16:25:16.562361+0000","last_unstale":"2024-09-13T16:25:21.149390+0000","last_undegraded":"2024-09-13T16:25:21.149390+0000","last_fullsized":"2024-09-13T16:25:21.149390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:32.764470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973976+0000","last_change":"2024-09-13T16:25:20.969014+0000","last_active":"2024-09-13T16:25:20.973976+0000","last_peered":"2024-09-13T16:25:20.973976+0000","last_clean":"2024-09-13T16:25:20.973976+0000","last_became_active":"2024-09-13T16:25:20.968528+0000","last_became_peered":"2024-09-13T16:25:20.968528+0000","last_unstale":"2024-09-13T16:25:20.973976+0000","last_undegraded":"2024-09-13T16:25:20.973976+0000","last_fullsized":"2024-09-13T16:25:20.973976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948588+0000","last_change":"2024-09-13T16:25:16.933619+0000","last_active":"2024-09-13T16:25:20.948588+0000","last_peered":"2024-09-13T16:25:20.948588+0000","last_clean":"2024-09-13T16:25:20.948588+0000","last_became_active":"2024-09-13T16:25:16.933324+0000","last_became_peered":"2024-09-13T16:25:16.933324+0000","last_unstale":"2024-09-13T16:25:20.948588+0000","last_undegraded":"2024-09-13T16:25:20.948588+0000","last_fullsized":"2024-09-13T16:25:20.948588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:13.824400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942057+0000","last_change":"2024-09-13T16:25:16.560251+0000","last_active":"2024-09-13T16:25:20.942057+0000","last_peered":"2024-09-13T16:25:20.942057+0000","last_clean":"2024-09-13T16:25:20.942057+0000","last_became_active":"2024-09-13T16:25:16.559836+0000","last_became_peered":"2024-09-13T16:25:16.559836+0000","last_unstale":"2024-09-13T16:25:20.942057+0000","last_undegraded":"2024-09-13T16:25:20.942057+0000","last_fullsized":"2024-09-13T16:25:20.942057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.159122+0000","last_change":"2024-09-13T16:25:21.156691+0000","last_active":"2024-09-13T16:25:21.159122+0000","last_peered":"2024-09-13T16:25:21.159122+0000","last_clean":"2024-09-13T16:25:21.159122+0000","last_became_active":"2024-09-13T16:25:21.156255+0000","last_became_peered":"2024-09-13T16:25:21.156255+0000","last_unstale":"2024-09-13T16:25:21.159122+0000","last_undegraded":"2024-09-13T16:25:21.159122+0000","last_fullsized":"2024-09-13T16:25:21.159122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:12.641280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940196+0000","last_change":"2024-09-13T16:25:16.946326+0000","last_active":"2024-09-13T16:25:20.940196+0000","last_peered":"2024-09-13T16:25:20.940196+0000","last_clean":"2024-09-13T16:25:20.940196+0000","last_became_active":"2024-09-13T16:25:16.946091+0000","last_became_peered":"2024-09-13T16:25:16.946091+0000","last_unstale":"2024-09-13T16:25:20.940196+0000","last_undegraded":"2024-09-13T16:25:20.940196+0000","last_fullsized":"2024-09-13T16:25:20.940196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953612+0000","last_change":"2024-09-13T16:25:16.548181+0000","last_active":"2024-09-13T16:25:20.953612+0000","last_peered":"2024-09-13T16:25:20.953612+0000","last_clean":"2024-09-13T16:25:20.953612+0000","last_became_active":"2024-09-13T16:25:16.546980+0000","last_became_peered":"2024-09-13T16:25:16.546980+0000","last_unstale":"2024-09-13T16:25:20.953612+0000","last_undegraded":"2024-09-13T16:25:20.953612+0000","last_fullsized":"2024-09-13T16:25:20.953612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163366+0000","last_change":"2024-09-13T16:25:21.157394+0000","last_active":"2024-09-13T16:25:21.163366+0000","last_peered":"2024-09-13T16:25:21.163366+0000","last_clean":"2024-09-13T16:25:21.163366+0000","last_became_active":"2024-09-13T16:25:21.156428+0000","last_became_peered":"2024-09-13T16:25:21.156428+0000","last_unstale":"2024-09-13T16:25:21.163366+0000","last_undegraded":"2024-09-13T16:25:21.163366+0000","last_fullsized":"2024-09-13T16:25:21.163366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:31.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950532+0000","last_change":"2024-09-13T16:25:16.932134+0000","last_active":"2024-09-13T16:25:20.950532+0000","last_peered":"2024-09-13T16:25:20.950532+0000","last_clean":"2024-09-13T16:25:20.950532+0000","last_became_active":"2024-09-13T16:25:16.931157+0000","last_became_peered":"2024-09-13T16:25:16.931157+0000","last_unstale":"2024-09-13T16:25:20.950532+0000","last_undegraded":"2024-09-13T16:25:20.950532+0000","last_fullsized":"2024-09-13T16:25:20.950532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150728+0000","last_change":"2024-09-13T16:25:16.560858+0000","last_active":"2024-09-13T16:25:21.150728+0000","last_peered":"2024-09-13T16:25:21.150728+0000","last_clean":"2024-09-13T16:25:21.150728+0000","last_became_active":"2024-09-13T16:25:16.560627+0000","last_became_peered":"2024-09-13T16:25:16.560627+0000","last_unstale":"2024-09-13T16:25:21.150728+0000","last_undegraded":"2024-09-13T16:25:21.150728+0000","last_fullsized":"2024-09-13T16:25:21.150728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:50.678565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158800+0000","last_change":"2024-09-13T16:25:21.155558+0000","last_active":"2024-09-13T16:25:21.158800+0000","last_peered":"2024-09-13T16:25:21.158800+0000","last_clean":"2024-09-13T16:25:21.158800+0000","last_became_active":"2024-09-13T16:25:21.154513+0000","last_became_peered":"2024-09-13T16:25:21.154513+0000","last_unstale":"2024-09-13T16:25:21.158800+0000","last_undegraded":"2024-09-13T16:25:21.158800+0000","last_fullsized":"2024-09-13T16:25:21.158800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943548+0000","last_change":"2024-09-13T16:25:16.934627+0000","last_active":"2024-09-13T16:25:20.943548+0000","last_peered":"2024-09-13T16:25:20.943548+0000","last_clean":"2024-09-13T16:25:20.943548+0000","last_became_active":"2024-09-13T16:25:16.934031+0000","last_became_peered":"2024-09-13T16:25:16.934031+0000","last_unstale":"2024-09-13T16:25:20.943548+0000","last_undegraded":"2024-09-13T16:25:20.943548+0000","last_fullsized":"2024-09-13T16:25:20.943548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:19.244372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950619+0000","last_change":"2024-09-13T16:25:16.550845+0000","last_active":"2024-09-13T16:25:20.950619+0000","last_peered":"2024-09-13T16:25:20.950619+0000","last_clean":"2024-09-13T16:25:20.950619+0000","last_became_active":"2024-09-13T16:25:16.550523+0000","last_became_peered":"2024-09-13T16:25:16.550523+0000","last_unstale":"2024-09-13T16:25:20.950619+0000","last_undegraded":"2024-09-13T16:25:20.950619+0000","last_fullsized":"2024-09-13T16:25:20.950619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971742+0000","last_change":"2024-09-13T16:25:20.963835+0000","last_active":"2024-09-13T16:25:20.971742+0000","last_peered":"2024-09-13T16:25:20.971742+0000","last_clean":"2024-09-13T16:25:20.971742+0000","last_became_active":"2024-09-13T16:25:20.963031+0000","last_became_peered":"2024-09-13T16:25:20.963031+0000","last_unstale":"2024-09-13T16:25:20.971742+0000","last_undegraded":"2024-09-13T16:25:20.971742+0000","last_fullsized":"2024-09-13T16:25:20.971742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:39.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951685+0000","last_change":"2024-09-13T16:25:16.936436+0000","last_active":"2024-09-13T16:25:20.951685+0000","last_peered":"2024-09-13T16:25:20.951685+0000","last_clean":"2024-09-13T16:25:20.951685+0000","last_became_active":"2024-09-13T16:25:16.936131+0000","last_became_peered":"2024-09-13T16:25:16.936131+0000","last_unstale":"2024-09-13T16:25:20.951685+0000","last_undegraded":"2024-09-13T16:25:20.951685+0000","last_fullsized":"2024-09-13T16:25:20.951685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:33.773998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941767+0000","last_change":"2024-09-13T16:25:16.548310+0000","last_active":"2024-09-13T16:25:20.941767+0000","last_peered":"2024-09-13T16:25:20.941767+0000","last_clean":"2024-09-13T16:25:20.941767+0000","last_became_active":"2024-09-13T16:25:16.548056+0000","last_became_peered":"2024-09-13T16:25:16.548056+0000","last_unstale":"2024-09-13T16:25:20.941767+0000","last_undegraded":"2024-09-13T16:25:20.941767+0000","last_fullsized":"2024-09-13T16:25:20.941767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971523+0000","last_change":"2024-09-13T16:25:20.960907+0000","last_active":"2024-09-13T16:25:20.971523+0000","last_peered":"2024-09-13T16:25:20.971523+0000","last_clean":"2024-09-13T16:25:20.971523+0000","last_became_active":"2024-09-13T16:25:20.960504+0000","last_became_peered":"2024-09-13T16:25:20.960504+0000","last_unstale":"2024-09-13T16:25:20.971523+0000","last_undegraded":"2024-09-13T16:25:20.971523+0000","last_fullsized":"2024-09-13T16:25:20.971523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951906+0000","last_change":"2024-09-13T16:25:16.935966+0000","last_active":"2024-09-13T16:25:20.951906+0000","last_peered":"2024-09-13T16:25:20.951906+0000","last_clean":"2024-09-13T16:25:20.951906+0000","last_became_active":"2024-09-13T16:25:16.935728+0000","last_became_peered":"2024-09-13T16:25:16.935728+0000","last_unstale":"2024-09-13T16:25:20.951906+0000","last_undegraded":"2024-09-13T16:25:20.951906+0000","last_fullsized":"2024-09-13T16:25:20.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954151+0000","last_change":"2024-09-13T16:25:16.541632+0000","last_active":"2024-09-13T16:25:20.954151+0000","last_peered":"2024-09-13T16:25:20.954151+0000","last_clean":"2024-09-13T16:25:20.954151+0000","last_became_active":"2024-09-13T16:25:16.541362+0000","last_became_peered":"2024-09-13T16:25:16.541362+0000","last_unstale":"2024-09-13T16:25:20.954151+0000","last_undegraded":"2024-09-13T16:25:20.954151+0000","last_fullsized":"2024-09-13T16:25:20.954151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971468+0000","last_change":"2024-09-13T16:25:20.961579+0000","last_active":"2024-09-13T16:25:20.971468+0000","last_peered":"2024-09-13T16:25:20.971468+0000","last_clean":"2024-09-13T16:25:20.971468+0000","last_became_active":"2024-09-13T16:25:20.961177+0000","last_became_peered":"2024-09-13T16:25:20.961177+0000","last_unstale":"2024-09-13T16:25:20.971468+0000","last_undegraded":"2024-09-13T16:25:20.971468+0000","last_fullsized":"2024-09-13T16:25:20.971468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948340+0000","last_change":"2024-09-13T16:25:16.932105+0000","last_active":"2024-09-13T16:25:20.948340+0000","last_peered":"2024-09-13T16:25:20.948340+0000","last_clean":"2024-09-13T16:25:20.948340+0000","last_became_active":"2024-09-13T16:25:16.931893+0000","last_became_peered":"2024-09-13T16:25:16.931893+0000","last_unstale":"2024-09-13T16:25:20.948340+0000","last_undegraded":"2024-09-13T16:25:20.948340+0000","last_fullsized":"2024-09-13T16:25:20.948340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149350+0000","last_change":"2024-09-13T16:25:16.563013+0000","last_active":"2024-09-13T16:25:21.149350+0000","last_peered":"2024-09-13T16:25:21.149350+0000","last_clean":"2024-09-13T16:25:21.149350+0000","last_became_active":"2024-09-13T16:25:16.562820+0000","last_became_peered":"2024-09-13T16:25:16.562820+0000","last_unstale":"2024-09-13T16:25:21.149350+0000","last_undegraded":"2024-09-13T16:25:21.149350+0000","last_fullsized":"2024-09-13T16:25:21.149350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:43.080203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.161407+0000","last_change":"2024-09-13T16:25:21.156288+0000","last_active":"2024-09-13T16:25:21.161407+0000","last_peered":"2024-09-13T16:25:21.161407+0000","last_clean":"2024-09-13T16:25:21.161407+0000","last_became_active":"2024-09-13T16:25:21.154873+0000","last_became_peered":"2024-09-13T16:25:21.154873+0000","last_unstale":"2024-09-13T16:25:21.161407+0000","last_undegraded":"2024-09-13T16:25:21.161407+0000","last_fullsized":"2024-09-13T16:25:21.161407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:57.390965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941321+0000","last_change":"2024-09-13T16:25:16.934659+0000","last_active":"2024-09-13T16:25:20.941321+0000","last_peered":"2024-09-13T16:25:20.941321+0000","last_clean":"2024-09-13T16:25:20.941321+0000","last_became_active":"2024-09-13T16:25:16.934286+0000","last_became_peered":"2024-09-13T16:25:16.934286+0000","last_unstale":"2024-09-13T16:25:20.941321+0000","last_undegraded":"2024-09-13T16:25:20.941321+0000","last_fullsized":"2024-09-13T16:25:20.941321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941932+0000","last_change":"2024-09-13T16:25:16.548968+0000","last_active":"2024-09-13T16:25:20.941932+0000","last_peered":"2024-09-13T16:25:20.941932+0000","last_clean":"2024-09-13T16:25:20.941932+0000","last_became_active":"2024-09-13T16:25:16.548722+0000","last_became_peered":"2024-09-13T16:25:16.548722+0000","last_unstale":"2024-09-13T16:25:20.941932+0000","last_undegraded":"2024-09-13T16:25:20.941932+0000","last_fullsized":"2024-09-13T16:25:20.941932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:25:08.328680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149477+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:25:21.149477+0000","last_peered":"2024-09-13T16:25:21.149477+0000","last_clean":"2024-09-13T16:25:21.149477+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:25:21.149477+0000","last_undegraded":"2024-09-13T16:25:21.149477+0000","last_fullsized":"2024-09-13T16:25:21.149477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156667+0000","last_change":"2024-09-13T16:25:21.154730+0000","last_active":"2024-09-13T16:25:21.156667+0000","last_peered":"2024-09-13T16:25:21.156667+0000","last_clean":"2024-09-13T16:25:21.156667+0000","last_became_active":"2024-09-13T16:25:21.154226+0000","last_became_peered":"2024-09-13T16:25:21.154226+0000","last_unstale":"2024-09-13T16:25:21.156667+0000","last_undegraded":"2024-09-13T16:25:21.156667+0000","last_fullsized":"2024-09-13T16:25:21.156667+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150204+0000","last_change":"2024-09-13T16:25:16.944787+0000","last_active":"2024-09-13T16:25:21.150204+0000","last_peered":"2024-09-13T16:25:21.150204+0000","last_clean":"2024-09-13T16:25:21.150204+0000","last_became_active":"2024-09-13T16:25:16.944505+0000","last_became_peered":"2024-09-13T16:25:16.944505+0000","last_unstale":"2024-09-13T16:25:21.150204+0000","last_undegraded":"2024-09-13T16:25:21.150204+0000","last_fullsized":"2024-09-13T16:25:21.150204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954746+0000","last_change":"2024-09-13T16:25:16.548195+0000","last_active":"2024-09-13T16:25:20.954746+0000","last_peered":"2024-09-13T16:25:20.954746+0000","last_clean":"2024-09-13T16:25:20.954746+0000","last_became_active":"2024-09-13T16:25:16.547839+0000","last_became_peered":"2024-09-13T16:25:16.547839+0000","last_unstale":"2024-09-13T16:25:20.954746+0000","last_undegraded":"2024-09-13T16:25:20.954746+0000","last_fullsized":"2024-09-13T16:25:20.954746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156778+0000","last_change":"2024-09-13T16:25:21.154890+0000","last_active":"2024-09-13T16:25:21.156778+0000","last_peered":"2024-09-13T16:25:21.156778+0000","last_clean":"2024-09-13T16:25:21.156778+0000","last_became_active":"2024-09-13T16:25:21.154376+0000","last_became_peered":"2024-09-13T16:25:21.154376+0000","last_unstale":"2024-09-13T16:25:21.156778+0000","last_undegraded":"2024-09-13T16:25:21.156778+0000","last_fullsized":"2024-09-13T16:25:21.156778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:09.876152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944021+0000","last_change":"2024-09-13T16:25:16.933523+0000","last_active":"2024-09-13T16:25:20.944021+0000","last_peered":"2024-09-13T16:25:20.944021+0000","last_clean":"2024-09-13T16:25:20.944021+0000","last_became_active":"2024-09-13T16:25:16.932020+0000","last_became_peered":"2024-09-13T16:25:16.932020+0000","last_unstale":"2024-09-13T16:25:20.944021+0000","last_undegraded":"2024-09-13T16:25:20.944021+0000","last_fullsized":"2024-09-13T16:25:20.944021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:34.060136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952497+0000","last_change":"2024-09-13T16:25:16.546857+0000","last_active":"2024-09-13T16:25:20.952497+0000","last_peered":"2024-09-13T16:25:20.952497+0000","last_clean":"2024-09-13T16:25:20.952497+0000","last_became_active":"2024-09-13T16:25:16.546592+0000","last_became_peered":"2024-09-13T16:25:16.546592+0000","last_unstale":"2024-09-13T16:25:20.952497+0000","last_undegraded":"2024-09-13T16:25:20.952497+0000","last_fullsized":"2024-09-13T16:25:20.952497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963301+0000","last_change":"2024-09-13T16:25:20.958682+0000","last_active":"2024-09-13T16:25:20.963301+0000","last_peered":"2024-09-13T16:25:20.963301+0000","last_clean":"2024-09-13T16:25:20.963301+0000","last_became_active":"2024-09-13T16:25:20.958064+0000","last_became_peered":"2024-09-13T16:25:20.958064+0000","last_unstale":"2024-09-13T16:25:20.963301+0000","last_undegraded":"2024-09-13T16:25:20.963301+0000","last_fullsized":"2024-09-13T16:25:20.963301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.502649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150943+0000","last_change":"2024-09-13T16:25:16.951592+0000","last_active":"2024-09-13T16:25:21.150943+0000","last_peered":"2024-09-13T16:25:21.150943+0000","last_clean":"2024-09-13T16:25:21.150943+0000","last_became_active":"2024-09-13T16:25:16.951375+0000","last_became_peered":"2024-09-13T16:25:16.951375+0000","last_unstale":"2024-09-13T16:25:21.150943+0000","last_undegraded":"2024-09-13T16:25:21.150943+0000","last_fullsized":"2024-09-13T16:25:21.150943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:32.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959285+0000","last_change":"2024-09-13T16:25:16.541133+0000","last_active":"2024-09-13T16:25:20.959285+0000","last_peered":"2024-09-13T16:25:20.959285+0000","last_clean":"2024-09-13T16:25:20.959285+0000","last_became_active":"2024-09-13T16:25:16.540843+0000","last_became_peered":"2024-09-13T16:25:16.540843+0000","last_unstale":"2024-09-13T16:25:20.959285+0000","last_undegraded":"2024-09-13T16:25:20.959285+0000","last_fullsized":"2024-09-13T16:25:20.959285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:05.477981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158080+0000","last_change":"2024-09-13T16:25:21.154908+0000","last_active":"2024-09-13T16:25:21.158080+0000","last_peered":"2024-09-13T16:25:21.158080+0000","last_clean":"2024-09-13T16:25:21.158080+0000","last_became_active":"2024-09-13T16:25:21.154365+0000","last_became_peered":"2024-09-13T16:25:21.154365+0000","last_unstale":"2024-09-13T16:25:21.158080+0000","last_undegraded":"2024-09-13T16:25:21.158080+0000","last_fullsized":"2024-09-13T16:25:21.158080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:00.139666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150870+0000","last_change":"2024-09-13T16:25:16.941714+0000","last_active":"2024-09-13T16:25:21.150870+0000","last_peered":"2024-09-13T16:25:21.150870+0000","last_clean":"2024-09-13T16:25:21.150870+0000","last_became_active":"2024-09-13T16:25:16.941465+0000","last_became_peered":"2024-09-13T16:25:16.941465+0000","last_unstale":"2024-09-13T16:25:21.150870+0000","last_undegraded":"2024-09-13T16:25:21.150870+0000","last_fullsized":"2024-09-13T16:25:21.150870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952152+0000","last_change":"2024-09-13T16:25:16.544584+0000","last_active":"2024-09-13T16:25:20.952152+0000","last_peered":"2024-09-13T16:25:20.952152+0000","last_clean":"2024-09-13T16:25:20.952152+0000","last_became_active":"2024-09-13T16:25:16.544275+0000","last_became_peered":"2024-09-13T16:25:16.544275+0000","last_unstale":"2024-09-13T16:25:20.952152+0000","last_undegraded":"2024-09-13T16:25:20.952152+0000","last_fullsized":"2024-09-13T16:25:20.952152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:38.495511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.976715+0000","last_change":"2024-09-13T16:25:20.972282+0000","last_active":"2024-09-13T16:25:20.976715+0000","last_peered":"2024-09-13T16:25:20.976715+0000","last_clean":"2024-09-13T16:25:20.976715+0000","last_became_active":"2024-09-13T16:25:20.971899+0000","last_became_peered":"2024-09-13T16:25:20.971899+0000","last_unstale":"2024-09-13T16:25:20.976715+0000","last_undegraded":"2024-09-13T16:25:20.976715+0000","last_fullsized":"2024-09-13T16:25:20.976715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150689+0000","last_change":"2024-09-13T16:25:16.944388+0000","last_active":"2024-09-13T16:25:21.150689+0000","last_peered":"2024-09-13T16:25:21.150689+0000","last_clean":"2024-09-13T16:25:21.150689+0000","last_became_active":"2024-09-13T16:25:16.943987+0000","last_became_peered":"2024-09-13T16:25:16.943987+0000","last_unstale":"2024-09-13T16:25:21.150689+0000","last_undegraded":"2024-09-13T16:25:21.150689+0000","last_fullsized":"2024-09-13T16:25:21.150689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953615+0000","last_change":"2024-09-13T16:25:16.545908+0000","last_active":"2024-09-13T16:25:20.953615+0000","last_peered":"2024-09-13T16:25:20.953615+0000","last_clean":"2024-09-13T16:25:20.953615+0000","last_became_active":"2024-09-13T16:25:16.545632+0000","last_became_peered":"2024-09-13T16:25:16.545632+0000","last_unstale":"2024-09-13T16:25:20.953615+0000","last_undegraded":"2024-09-13T16:25:20.953615+0000","last_fullsized":"2024-09-13T16:25:20.953615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963693+0000","last_change":"2024-09-13T16:25:20.961076+0000","last_active":"2024-09-13T16:25:20.963693+0000","last_peered":"2024-09-13T16:25:20.963693+0000","last_clean":"2024-09-13T16:25:20.963693+0000","last_became_active":"2024-09-13T16:25:20.960736+0000","last_became_peered":"2024-09-13T16:25:20.960736+0000","last_unstale":"2024-09-13T16:25:20.963693+0000","last_undegraded":"2024-09-13T16:25:20.963693+0000","last_fullsized":"2024-09-13T16:25:20.963693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939386+0000","last_change":"2024-09-13T16:25:16.939543+0000","last_active":"2024-09-13T16:25:20.939386+0000","last_peered":"2024-09-13T16:25:20.939386+0000","last_clean":"2024-09-13T16:25:20.939386+0000","last_became_active":"2024-09-13T16:25:16.939269+0000","last_became_peered":"2024-09-13T16:25:16.939269+0000","last_unstale":"2024-09-13T16:25:20.939386+0000","last_undegraded":"2024-09-13T16:25:20.939386+0000","last_fullsized":"2024-09-13T16:25:20.939386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939280+0000","last_change":"2024-09-13T16:25:16.557172+0000","last_active":"2024-09-13T16:25:20.939280+0000","last_peered":"2024-09-13T16:25:20.939280+0000","last_clean":"2024-09-13T16:25:20.939280+0000","last_became_active":"2024-09-13T16:25:16.556910+0000","last_became_peered":"2024-09-13T16:25:16.556910+0000","last_unstale":"2024-09-13T16:25:20.939280+0000","last_undegraded":"2024-09-13T16:25:20.939280+0000","last_fullsized":"2024-09-13T16:25:20.939280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162791+0000","last_change":"2024-09-13T16:25:21.154750+0000","last_active":"2024-09-13T16:25:21.162791+0000","last_peered":"2024-09-13T16:25:21.162791+0000","last_clean":"2024-09-13T16:25:21.162791+0000","last_became_active":"2024-09-13T16:25:21.153987+0000","last_became_peered":"2024-09-13T16:25:21.153987+0000","last_unstale":"2024-09-13T16:25:21.162791+0000","last_undegraded":"2024-09-13T16:25:21.162791+0000","last_fullsized":"2024-09-13T16:25:21.162791+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149825+0000","last_change":"2024-09-13T16:25:16.944291+0000","last_active":"2024-09-13T16:25:21.149825+0000","last_peered":"2024-09-13T16:25:21.149825+0000","last_clean":"2024-09-13T16:25:21.149825+0000","last_became_active":"2024-09-13T16:25:16.941709+0000","last_became_peered":"2024-09-13T16:25:16.941709+0000","last_unstale":"2024-09-13T16:25:21.149825+0000","last_undegraded":"2024-09-13T16:25:21.149825+0000","last_fullsized":"2024-09-13T16:25:21.149825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:45.537446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941804+0000","last_change":"2024-09-13T16:25:16.553038+0000","last_active":"2024-09-13T16:25:20.941804+0000","last_peered":"2024-09-13T16:25:20.941804+0000","last_clean":"2024-09-13T16:25:20.941804+0000","last_became_active":"2024-09-13T16:25:16.551960+0000","last_became_peered":"2024-09-13T16:25:16.551960+0000","last_unstale":"2024-09-13T16:25:20.941804+0000","last_undegraded":"2024-09-13T16:25:20.941804+0000","last_fullsized":"2024-09-13T16:25:20.941804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:27.632855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163399+0000","last_change":"2024-09-13T16:25:21.158491+0000","last_active":"2024-09-13T16:25:21.163399+0000","last_peered":"2024-09-13T16:25:21.163399+0000","last_clean":"2024-09-13T16:25:21.163399+0000","last_became_active":"2024-09-13T16:25:21.158133+0000","last_became_peered":"2024-09-13T16:25:21.158133+0000","last_unstale":"2024-09-13T16:25:21.163399+0000","last_undegraded":"2024-09-13T16:25:21.163399+0000","last_fullsized":"2024-09-13T16:25:21.163399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946925+0000","last_change":"2024-09-13T16:25:16.941874+0000","last_active":"2024-09-13T16:25:20.946925+0000","last_peered":"2024-09-13T16:25:20.946925+0000","last_clean":"2024-09-13T16:25:20.946925+0000","last_became_active":"2024-09-13T16:25:16.941683+0000","last_became_peered":"2024-09-13T16:25:16.941683+0000","last_unstale":"2024-09-13T16:25:20.946925+0000","last_undegraded":"2024-09-13T16:25:20.946925+0000","last_fullsized":"2024-09-13T16:25:20.946925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953752+0000","last_change":"2024-09-13T16:25:16.551439+0000","last_active":"2024-09-13T16:25:20.953752+0000","last_peered":"2024-09-13T16:25:20.953752+0000","last_clean":"2024-09-13T16:25:20.953752+0000","last_became_active":"2024-09-13T16:25:16.550937+0000","last_became_peered":"2024-09-13T16:25:16.550937+0000","last_unstale":"2024-09-13T16:25:20.953752+0000","last_undegraded":"2024-09-13T16:25:20.953752+0000","last_fullsized":"2024-09-13T16:25:20.953752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:14.983082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158057+0000","last_change":"2024-09-13T16:25:21.155556+0000","last_active":"2024-09-13T16:25:21.158057+0000","last_peered":"2024-09-13T16:25:21.158057+0000","last_clean":"2024-09-13T16:25:21.158057+0000","last_became_active":"2024-09-13T16:25:21.155201+0000","last_became_peered":"2024-09-13T16:25:21.155201+0000","last_unstale":"2024-09-13T16:25:21.158057+0000","last_undegraded":"2024-09-13T16:25:21.158057+0000","last_fullsized":"2024-09-13T16:25:21.158057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:42.823106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150192+0000","last_change":"2024-09-13T16:25:16.944770+0000","last_active":"2024-09-13T16:25:21.150192+0000","last_peered":"2024-09-13T16:25:21.150192+0000","last_clean":"2024-09-13T16:25:21.150192+0000","last_became_active":"2024-09-13T16:25:16.944455+0000","last_became_peered":"2024-09-13T16:25:16.944455+0000","last_unstale":"2024-09-13T16:25:21.150192+0000","last_undegraded":"2024-09-13T16:25:21.150192+0000","last_fullsized":"2024-09-13T16:25:21.150192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:41:40.454800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-13T16:25:26.199825+0000","last_change":"2024-09-13T16:25:16.552517+0000","last_active":"2024-09-13T16:25:26.199825+0000","last_peered":"2024-09-13T16:25:26.199825+0000","last_clean":"2024-09-13T16:25:26.199825+0000","last_became_active":"2024-09-13T16:25:16.552101+0000","last_became_peered":"2024-09-13T16:25:16.552101+0000","last_unstale":"2024-09-13T16:25:26.199825+0000","last_undegraded":"2024-09-13T16:25:26.199825+0000","last_fullsized":"2024-09-13T16:25:26.199825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971715+0000","last_change":"2024-09-13T16:25:20.963516+0000","last_active":"2024-09-13T16:25:20.971715+0000","last_peered":"2024-09-13T16:25:20.971715+0000","last_clean":"2024-09-13T16:25:20.971715+0000","last_became_active":"2024-09-13T16:25:20.962518+0000","last_became_peered":"2024-09-13T16:25:20.962518+0000","last_unstale":"2024-09-13T16:25:20.971715+0000","last_undegraded":"2024-09-13T16:25:20.971715+0000","last_fullsized":"2024-09-13T16:25:20.971715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:05.557712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951494+0000","last_change":"2024-09-13T16:25:16.941833+0000","last_active":"2024-09-13T16:25:20.951494+0000","last_peered":"2024-09-13T16:25:20.951494+0000","last_clean":"2024-09-13T16:25:20.951494+0000","last_became_active":"2024-09-13T16:25:16.941607+0000","last_became_peered":"2024-09-13T16:25:16.941607+0000","last_unstale":"2024-09-13T16:25:20.951494+0000","last_undegraded":"2024-09-13T16:25:20.951494+0000","last_fullsized":"2024-09-13T16:25:20.951494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:44.089730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952414+0000","last_change":"2024-09-13T16:25:16.549878+0000","last_active":"2024-09-13T16:25:20.952414+0000","last_peered":"2024-09-13T16:25:20.952414+0000","last_clean":"2024-09-13T16:25:20.952414+0000","last_became_active":"2024-09-13T16:25:16.548581+0000","last_became_peered":"2024-09-13T16:25:16.548581+0000","last_unstale":"2024-09-13T16:25:20.952414+0000","last_undegraded":"2024-09-13T16:25:20.952414+0000","last_fullsized":"2024-09-13T16:25:20.952414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969433+0000","last_change":"2024-09-13T16:25:20.965523+0000","last_active":"2024-09-13T16:25:20.969433+0000","last_peered":"2024-09-13T16:25:20.969433+0000","last_clean":"2024-09-13T16:25:20.969433+0000","last_became_active":"2024-09-13T16:25:20.965035+0000","last_became_peered":"2024-09-13T16:25:20.965035+0000","last_unstale":"2024-09-13T16:25:20.969433+0000","last_undegraded":"2024-09-13T16:25:20.969433+0000","last_fullsized":"2024-09-13T16:25:20.969433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954066+0000","last_change":"2024-09-13T16:25:16.933522+0000","last_active":"2024-09-13T16:25:20.954066+0000","last_peered":"2024-09-13T16:25:20.954066+0000","last_clean":"2024-09-13T16:25:20.954066+0000","last_became_active":"2024-09-13T16:25:16.933134+0000","last_became_peered":"2024-09-13T16:25:16.933134+0000","last_unstale":"2024-09-13T16:25:20.954066+0000","last_undegraded":"2024-09-13T16:25:20.954066+0000","last_fullsized":"2024-09-13T16:25:20.954066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942876+0000","last_change":"2024-09-13T16:25:16.549756+0000","last_active":"2024-09-13T16:25:20.942876+0000","last_peered":"2024-09-13T16:25:20.942876+0000","last_clean":"2024-09-13T16:25:20.942876+0000","last_became_active":"2024-09-13T16:25:16.549515+0000","last_became_peered":"2024-09-13T16:25:16.549515+0000","last_unstale":"2024-09-13T16:25:20.942876+0000","last_undegraded":"2024-09-13T16:25:20.942876+0000","last_fullsized":"2024-09-13T16:25:20.942876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963576+0000","last_change":"2024-09-13T16:25:20.960084+0000","last_active":"2024-09-13T16:25:20.963576+0000","last_peered":"2024-09-13T16:25:20.963576+0000","last_clean":"2024-09-13T16:25:20.963576+0000","last_became_active":"2024-09-13T16:25:20.957343+0000","last_became_peered":"2024-09-13T16:25:20.957343+0000","last_unstale":"2024-09-13T16:25:20.963576+0000","last_undegraded":"2024-09-13T16:25:20.963576+0000","last_fullsized":"2024-09-13T16:25:20.963576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:06.115184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942738+0000","last_change":"2024-09-13T16:25:16.942162+0000","last_active":"2024-09-13T16:25:20.942738+0000","last_peered":"2024-09-13T16:25:20.942738+0000","last_clean":"2024-09-13T16:25:20.942738+0000","last_became_active":"2024-09-13T16:25:16.941896+0000","last_became_peered":"2024-09-13T16:25:16.941896+0000","last_unstale":"2024-09-13T16:25:20.942738+0000","last_undegraded":"2024-09-13T16:25:20.942738+0000","last_fullsized":"2024-09-13T16:25:20.942738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:24.023742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941934+0000","last_change":"2024-09-13T16:25:16.551442+0000","last_active":"2024-09-13T16:25:20.941934+0000","last_peered":"2024-09-13T16:25:20.941934+0000","last_clean":"2024-09-13T16:25:20.941934+0000","last_became_active":"2024-09-13T16:25:16.551084+0000","last_became_peered":"2024-09-13T16:25:16.551084+0000","last_unstale":"2024-09-13T16:25:20.941934+0000","last_undegraded":"2024-09-13T16:25:20.941934+0000","last_fullsized":"2024-09-13T16:25:20.941934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163416+0000","last_change":"2024-09-13T16:25:21.157528+0000","last_active":"2024-09-13T16:25:21.163416+0000","last_peered":"2024-09-13T16:25:21.163416+0000","last_clean":"2024-09-13T16:25:21.163416+0000","last_became_active":"2024-09-13T16:25:21.155560+0000","last_became_peered":"2024-09-13T16:25:21.155560+0000","last_unstale":"2024-09-13T16:25:21.163416+0000","last_undegraded":"2024-09-13T16:25:21.163416+0000","last_fullsized":"2024-09-13T16:25:21.163416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162596+0000","last_change":"2024-09-13T16:25:21.158060+0000","last_active":"2024-09-13T16:25:21.162596+0000","last_peered":"2024-09-13T16:25:21.162596+0000","last_clean":"2024-09-13T16:25:21.162596+0000","last_became_active":"2024-09-13T16:25:21.156882+0000","last_became_peered":"2024-09-13T16:25:21.156882+0000","last_unstale":"2024-09-13T16:25:21.162596+0000","last_undegraded":"2024-09-13T16:25:21.162596+0000","last_fullsized":"2024-09-13T16:25:21.162596+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949704+0000","last_change":"2024-09-13T16:25:16.932645+0000","last_active":"2024-09-13T16:25:20.949704+0000","last_peered":"2024-09-13T16:25:20.949704+0000","last_clean":"2024-09-13T16:25:20.949704+0000","last_became_active":"2024-09-13T16:25:16.931775+0000","last_became_peered":"2024-09-13T16:25:16.931775+0000","last_unstale":"2024-09-13T16:25:20.949704+0000","last_undegraded":"2024-09-13T16:25:20.949704+0000","last_fullsized":"2024-09-13T16:25:20.949704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942164+0000","last_change":"2024-09-13T16:25:16.547336+0000","last_active":"2024-09-13T16:25:20.942164+0000","last_peered":"2024-09-13T16:25:20.942164+0000","last_clean":"2024-09-13T16:25:20.942164+0000","last_became_active":"2024-09-13T16:25:16.547062+0000","last_became_peered":"2024-09-13T16:25:16.547062+0000","last_unstale":"2024-09-13T16:25:20.942164+0000","last_undegraded":"2024-09-13T16:25:20.942164+0000","last_fullsized":"2024-09-13T16:25:20.942164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.968355+0000","last_change":"2024-09-13T16:25:20.965354+0000","last_active":"2024-09-13T16:25:20.968355+0000","last_peered":"2024-09-13T16:25:20.968355+0000","last_clean":"2024-09-13T16:25:20.968355+0000","last_became_active":"2024-09-13T16:25:20.964985+0000","last_became_peered":"2024-09-13T16:25:20.964985+0000","last_unstale":"2024-09-13T16:25:20.968355+0000","last_undegraded":"2024-09-13T16:25:20.968355+0000","last_fullsized":"2024-09-13T16:25:20.968355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:42.840413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939763+0000","last_change":"2024-09-13T16:25:16.929034+0000","last_active":"2024-09-13T16:25:20.939763+0000","last_peered":"2024-09-13T16:25:20.939763+0000","last_clean":"2024-09-13T16:25:20.939763+0000","last_became_active":"2024-09-13T16:25:16.928782+0000","last_became_peered":"2024-09-13T16:25:16.928782+0000","last_unstale":"2024-09-13T16:25:20.939763+0000","last_undegraded":"2024-09-13T16:25:20.939763+0000","last_fullsized":"2024-09-13T16:25:20.939763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940092+0000","last_change":"2024-09-13T16:25:16.548923+0000","last_active":"2024-09-13T16:25:20.940092+0000","last_peered":"2024-09-13T16:25:20.940092+0000","last_clean":"2024-09-13T16:25:20.940092+0000","last_became_active":"2024-09-13T16:25:16.545843+0000","last_became_peered":"2024-09-13T16:25:16.545843+0000","last_unstale":"2024-09-13T16:25:20.940092+0000","last_undegraded":"2024-09-13T16:25:20.940092+0000","last_fullsized":"2024-09-13T16:25:20.940092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939883+0000","last_change":"2024-09-13T16:25:16.555750+0000","last_active":"2024-09-13T16:25:20.939883+0000","last_peered":"2024-09-13T16:25:20.939883+0000","last_clean":"2024-09-13T16:25:20.939883+0000","last_became_active":"2024-09-13T16:25:16.555480+0000","last_became_peered":"2024-09-13T16:25:16.555480+0000","last_unstale":"2024-09-13T16:25:20.939883+0000","last_undegraded":"2024-09-13T16:25:20.939883+0000","last_fullsized":"2024-09-13T16:25:20.939883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953647+0000","last_change":"2024-09-13T16:25:16.934605+0000","last_active":"2024-09-13T16:25:20.953647+0000","last_peered":"2024-09-13T16:25:20.953647+0000","last_clean":"2024-09-13T16:25:20.953647+0000","last_became_active":"2024-09-13T16:25:16.934365+0000","last_became_peered":"2024-09-13T16:25:16.934365+0000","last_unstale":"2024-09-13T16:25:20.953647+0000","last_undegraded":"2024-09-13T16:25:20.953647+0000","last_fullsized":"2024-09-13T16:25:20.953647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.967920+0000","last_change":"2024-09-13T16:25:20.965561+0000","last_active":"2024-09-13T16:25:20.967920+0000","last_peered":"2024-09-13T16:25:20.967920+0000","last_clean":"2024-09-13T16:25:20.967920+0000","last_became_active":"2024-09-13T16:25:20.965162+0000","last_became_peered":"2024-09-13T16:25:20.965162+0000","last_unstale":"2024-09-13T16:25:20.967920+0000","last_undegraded":"2024-09-13T16:25:20.967920+0000","last_fullsized":"2024-09-13T16:25:20.967920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:35.284928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940864+0000","last_change":"2024-09-13T16:25:16.560967+0000","last_active":"2024-09-13T16:25:20.940864+0000","last_peered":"2024-09-13T16:25:20.940864+0000","last_clean":"2024-09-13T16:25:20.940864+0000","last_became_active":"2024-09-13T16:25:16.560675+0000","last_became_peered":"2024-09-13T16:25:16.560675+0000","last_unstale":"2024-09-13T16:25:20.940864+0000","last_undegraded":"2024-09-13T16:25:20.940864+0000","last_fullsized":"2024-09-13T16:25:20.940864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951187+0000","last_change":"2024-09-13T16:25:16.936540+0000","last_active":"2024-09-13T16:25:20.951187+0000","last_peered":"2024-09-13T16:25:20.951187+0000","last_clean":"2024-09-13T16:25:20.951187+0000","last_became_active":"2024-09-13T16:25:16.936269+0000","last_became_peered":"2024-09-13T16:25:16.936269+0000","last_unstale":"2024-09-13T16:25:20.951187+0000","last_undegraded":"2024-09-13T16:25:20.951187+0000","last_fullsized":"2024-09-13T16:25:20.951187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:18.089801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963605+0000","last_change":"2024-09-13T16:25:20.961207+0000","last_active":"2024-09-13T16:25:20.963605+0000","last_peered":"2024-09-13T16:25:20.963605+0000","last_clean":"2024-09-13T16:25:20.963605+0000","last_became_active":"2024-09-13T16:25:20.960801+0000","last_became_peered":"2024-09-13T16:25:20.960801+0000","last_unstale":"2024-09-13T16:25:20.963605+0000","last_undegraded":"2024-09-13T16:25:20.963605+0000","last_fullsized":"2024-09-13T16:25:20.963605+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:38.874981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939239+0000","last_change":"2024-09-13T16:25:16.939880+0000","last_active":"2024-09-13T16:25:20.939239+0000","last_peered":"2024-09-13T16:25:20.939239+0000","last_clean":"2024-09-13T16:25:20.939239+0000","last_became_active":"2024-09-13T16:25:16.939579+0000","last_became_peered":"2024-09-13T16:25:16.939579+0000","last_unstale":"2024-09-13T16:25:20.939239+0000","last_undegraded":"2024-09-13T16:25:20.939239+0000","last_fullsized":"2024-09-13T16:25:20.939239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:37.417453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953935+0000","last_change":"2024-09-13T16:25:16.938944+0000","last_active":"2024-09-13T16:25:20.953935+0000","last_peered":"2024-09-13T16:25:20.953935+0000","last_clean":"2024-09-13T16:25:20.953935+0000","last_became_active":"2024-09-13T16:25:16.938658+0000","last_became_peered":"2024-09-13T16:25:16.938658+0000","last_unstale":"2024-09-13T16:25:20.953935+0000","last_undegraded":"2024-09-13T16:25:20.953935+0000","last_fullsized":"2024-09-13T16:25:20.953935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:50.130499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941580+0000","last_change":"2024-09-13T16:25:16.933588+0000","last_active":"2024-09-13T16:25:20.941580+0000","last_peered":"2024-09-13T16:25:20.941580+0000","last_clean":"2024-09-13T16:25:20.941580+0000","last_became_active":"2024-09-13T16:25:16.932300+0000","last_became_peered":"2024-09-13T16:25:16.932300+0000","last_unstale":"2024-09-13T16:25:20.941580+0000","last_undegraded":"2024-09-13T16:25:20.941580+0000","last_fullsized":"2024-09-13T16:25:20.941580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:43.437120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953565+0000","last_change":"2024-09-13T16:25:16.934119+0000","last_active":"2024-09-13T16:25:20.953565+0000","last_peered":"2024-09-13T16:25:20.953565+0000","last_clean":"2024-09-13T16:25:20.953565+0000","last_became_active":"2024-09-13T16:25:16.930259+0000","last_became_peered":"2024-09-13T16:25:16.930259+0000","last_unstale":"2024-09-13T16:25:20.953565+0000","last_undegraded":"2024-09-13T16:25:20.953565+0000","last_fullsized":"2024-09-13T16:25:20.953565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941316+0000","last_change":"2024-09-13T16:25:16.931922+0000","last_active":"2024-09-13T16:25:20.941316+0000","last_peered":"2024-09-13T16:25:20.941316+0000","last_clean":"2024-09-13T16:25:20.941316+0000","last_became_active":"2024-09-13T16:25:16.931678+0000","last_became_peered":"2024-09-13T16:25:16.931678+0000","last_unstale":"2024-09-13T16:25:20.941316+0000","last_undegraded":"2024-09-13T16:25:20.941316+0000","last_fullsized":"2024-09-13T16:25:20.941316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:27.822409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952316+0000","last_change":"2024-09-13T16:25:16.932526+0000","last_active":"2024-09-13T16:25:20.952316+0000","last_peered":"2024-09-13T16:25:20.952316+0000","last_clean":"2024-09-13T16:25:20.952316+0000","last_became_active":"2024-09-13T16:25:16.932189+0000","last_became_peered":"2024-09-13T16:25:16.932189+0000","last_unstale":"2024-09-13T16:25:20.952316+0000","last_undegraded":"2024-09-13T16:25:20.952316+0000","last_fullsized":"2024-09-13T16:25:20.952316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941353+0000","last_change":"2024-09-13T16:25:16.947538+0000","last_active":"2024-09-13T16:25:20.941353+0000","last_peered":"2024-09-13T16:25:20.941353+0000","last_clean":"2024-09-13T16:25:20.941353+0000","last_became_active":"2024-09-13T16:25:16.947272+0000","last_became_peered":"2024-09-13T16:25:16.947272+0000","last_unstale":"2024-09-13T16:25:20.941353+0000","last_undegraded":"2024-09-13T16:25:20.941353+0000","last_fullsized":"2024-09-13T16:25:20.941353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940064+0000","last_change":"2024-09-13T16:25:16.932732+0000","last_active":"2024-09-13T16:25:20.940064+0000","last_peered":"2024-09-13T16:25:20.940064+0000","last_clean":"2024-09-13T16:25:20.940064+0000","last_became_active":"2024-09-13T16:25:16.932505+0000","last_became_peered":"2024-09-13T16:25:16.932505+0000","last_unstale":"2024-09-13T16:25:20.940064+0000","last_undegraded":"2024-09-13T16:25:20.940064+0000","last_fullsized":"2024-09-13T16:25:20.940064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942122+0000","last_change":"2024-09-13T16:25:16.933750+0000","last_active":"2024-09-13T16:25:20.942122+0000","last_peered":"2024-09-13T16:25:20.942122+0000","last_clean":"2024-09-13T16:25:20.942122+0000","last_became_active":"2024-09-13T16:25:16.933509+0000","last_became_peered":"2024-09-13T16:25:16.933509+0000","last_unstale":"2024-09-13T16:25:20.942122+0000","last_undegraded":"2024-09-13T16:25:20.942122+0000","last_fullsized":"2024-09-13T16:25:20.942122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:28.433477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939801+0000","last_change":"2024-09-13T16:25:16.932135+0000","last_active":"2024-09-13T16:25:20.939801+0000","last_peered":"2024-09-13T16:25:20.939801+0000","last_clean":"2024-09-13T16:25:20.939801+0000","last_became_active":"2024-09-13T16:25:16.931865+0000","last_became_peered":"2024-09-13T16:25:16.931865+0000","last_unstale":"2024-09-13T16:25:20.939801+0000","last_undegraded":"2024-09-13T16:25:20.939801+0000","last_fullsized":"2024-09-13T16:25:20.939801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941544+0000","last_change":"2024-09-13T16:25:16.942044+0000","last_active":"2024-09-13T16:25:20.941544+0000","last_peered":"2024-09-13T16:25:20.941544+0000","last_clean":"2024-09-13T16:25:20.941544+0000","last_became_active":"2024-09-13T16:25:16.941738+0000","last_became_peered":"2024-09-13T16:25:16.941738+0000","last_unstale":"2024-09-13T16:25:20.941544+0000","last_undegraded":"2024-09-13T16:25:20.941544+0000","last_fullsized":"2024-09-13T16:25:20.941544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:58:43.832593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948993+0000","last_change":"2024-09-13T16:25:16.942581+0000","last_active":"2024-09-13T16:25:20.948993+0000","last_peered":"2024-09-13T16:25:20.948993+0000","last_clean":"2024-09-13T16:25:20.948993+0000","last_became_active":"2024-09-13T16:25:16.942321+0000","last_became_peered":"2024-09-13T16:25:16.942321+0000","last_unstale":"2024-09-13T16:25:20.948993+0000","last_undegraded":"2024-09-13T16:25:20.948993+0000","last_fullsized":"2024-09-13T16:25:20.948993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939626+0000","last_change":"2024-09-13T16:25:16.936450+0000","last_active":"2024-09-13T16:25:20.939626+0000","last_peered":"2024-09-13T16:25:20.939626+0000","last_clean":"2024-09-13T16:25:20.939626+0000","last_became_active":"2024-09-13T16:25:16.936084+0000","last_became_peered":"2024-09-13T16:25:16.936084+0000","last_unstale":"2024-09-13T16:25:20.939626+0000","last_undegraded":"2024-09-13T16:25:20.939626+0000","last_fullsized":"2024-09-13T16:25:20.939626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150470+0000","last_change":"2024-09-13T16:25:16.948876+0000","last_active":"2024-09-13T16:25:21.150470+0000","last_peered":"2024-09-13T16:25:21.150470+0000","last_clean":"2024-09-13T16:25:21.150470+0000","last_became_active":"2024-09-13T16:25:16.948536+0000","last_became_peered":"2024-09-13T16:25:16.948536+0000","last_unstale":"2024-09-13T16:25:21.150470+0000","last_undegraded":"2024-09-13T16:25:21.150470+0000","last_fullsized":"2024-09-13T16:25:21.150470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959182+0000","last_change":"2024-09-13T16:25:16.937371+0000","last_active":"2024-09-13T16:25:20.959182+0000","last_peered":"2024-09-13T16:25:20.959182+0000","last_clean":"2024-09-13T16:25:20.959182+0000","last_became_active":"2024-09-13T16:25:16.937130+0000","last_became_peered":"2024-09-13T16:25:16.937130+0000","last_unstale":"2024-09-13T16:25:20.959182+0000","last_undegraded":"2024-09-13T16:25:20.959182+0000","last_fullsized":"2024-09-13T16:25:20.959182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:29.816675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944314+0000","last_change":"2024-09-13T16:25:16.947064+0000","last_active":"2024-09-13T16:25:20.944314+0000","last_peered":"2024-09-13T16:25:20.944314+0000","last_clean":"2024-09-13T16:25:20.944314+0000","last_became_active":"2024-09-13T16:25:16.946790+0000","last_became_peered":"2024-09-13T16:25:16.946790+0000","last_unstale":"2024-09-13T16:25:20.944314+0000","last_undegraded":"2024-09-13T16:25:20.944314+0000","last_fullsized":"2024-09-13T16:25:20.944314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.978002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942037+0000","last_change":"2024-09-13T16:25:16.931584+0000","last_active":"2024-09-13T16:25:20.942037+0000","last_peered":"2024-09-13T16:25:20.942037+0000","last_clean":"2024-09-13T16:25:20.942037+0000","last_became_active":"2024-09-13T16:25:16.931224+0000","last_became_peered":"2024-09-13T16:25:16.931224+0000","last_unstale":"2024-09-13T16:25:20.942037+0000","last_undegraded":"2024-09-13T16:25:20.942037+0000","last_fullsized":"2024-09-13T16:25:20.942037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:11.767256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949799+0000","last_change":"2024-09-13T16:25:16.936233+0000","last_active":"2024-09-13T16:25:20.949799+0000","last_peered":"2024-09-13T16:25:20.949799+0000","last_clean":"2024-09-13T16:25:20.949799+0000","last_became_active":"2024-09-13T16:25:16.935935+0000","last_became_peered":"2024-09-13T16:25:16.935935+0000","last_unstale":"2024-09-13T16:25:20.949799+0000","last_undegraded":"2024-09-13T16:25:20.949799+0000","last_fullsized":"2024-09-13T16:25:20.949799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946795+0000","last_change":"2024-09-13T16:25:16.936516+0000","last_active":"2024-09-13T16:25:20.946795+0000","last_peered":"2024-09-13T16:25:20.946795+0000","last_clean":"2024-09-13T16:25:20.946795+0000","last_became_active":"2024-09-13T16:25:16.936281+0000","last_became_peered":"2024-09-13T16:25:16.936281+0000","last_unstale":"2024-09-13T16:25:20.946795+0000","last_undegraded":"2024-09-13T16:25:20.946795+0000","last_fullsized":"2024-09-13T16:25:20.946795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940457+0000","last_change":"2024-09-13T16:25:16.938439+0000","last_active":"2024-09-13T16:25:20.940457+0000","last_peered":"2024-09-13T16:25:20.940457+0000","last_clean":"2024-09-13T16:25:20.940457+0000","last_became_active":"2024-09-13T16:25:16.938204+0000","last_became_peered":"2024-09-13T16:25:16.938204+0000","last_unstale":"2024-09-13T16:25:20.940457+0000","last_undegraded":"2024-09-13T16:25:20.940457+0000","last_fullsized":"2024-09-13T16:25:20.940457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951414+0000","last_change":"2024-09-13T16:25:16.931605+0000","last_active":"2024-09-13T16:25:20.951414+0000","last_peered":"2024-09-13T16:25:20.951414+0000","last_clean":"2024-09-13T16:25:20.951414+0000","last_became_active":"2024-09-13T16:25:16.931330+0000","last_became_peered":"2024-09-13T16:25:16.931330+0000","last_unstale":"2024-09-13T16:25:20.951414+0000","last_undegraded":"2024-09-13T16:25:20.951414+0000","last_fullsized":"2024-09-13T16:25:20.951414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:51.688093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940628+0000","last_change":"2024-09-13T16:25:16.936567+0000","last_active":"2024-09-13T16:25:20.940628+0000","last_peered":"2024-09-13T16:25:20.940628+0000","last_clean":"2024-09-13T16:25:20.940628+0000","last_became_active":"2024-09-13T16:25:16.936076+0000","last_became_peered":"2024-09-13T16:25:16.936076+0000","last_unstale":"2024-09-13T16:25:20.940628+0000","last_undegraded":"2024-09-13T16:25:20.940628+0000","last_fullsized":"2024-09-13T16:25:20.940628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941217+0000","last_change":"2024-09-13T16:25:16.936360+0000","last_active":"2024-09-13T16:25:20.941217+0000","last_peered":"2024-09-13T16:25:20.941217+0000","last_clean":"2024-09-13T16:25:20.941217+0000","last_became_active":"2024-09-13T16:25:16.935361+0000","last_became_peered":"2024-09-13T16:25:16.935361+0000","last_unstale":"2024-09-13T16:25:20.941217+0000","last_undegraded":"2024-09-13T16:25:20.941217+0000","last_fullsized":"2024-09-13T16:25:20.941217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150991+0000","last_change":"2024-09-13T16:25:16.951709+0000","last_active":"2024-09-13T16:25:21.150991+0000","last_peered":"2024-09-13T16:25:21.150991+0000","last_clean":"2024-09-13T16:25:21.150991+0000","last_became_active":"2024-09-13T16:25:16.951465+0000","last_became_peered":"2024-09-13T16:25:16.951465+0000","last_unstale":"2024-09-13T16:25:21.150991+0000","last_undegraded":"2024-09-13T16:25:21.150991+0000","last_fullsized":"2024-09-13T16:25:21.150991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:05.832843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.938568+0000","last_change":"2024-09-13T16:25:16.933931+0000","last_active":"2024-09-13T16:25:20.938568+0000","last_peered":"2024-09-13T16:25:20.938568+0000","last_clean":"2024-09-13T16:25:20.938568+0000","last_became_active":"2024-09-13T16:25:16.933646+0000","last_became_peered":"2024-09-13T16:25:16.933646+0000","last_unstale":"2024-09-13T16:25:20.938568+0000","last_undegraded":"2024-09-13T16:25:20.938568+0000","last_fullsized":"2024-09-13T16:25:20.938568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150544+0000","last_change":"2024-09-13T16:25:16.941658+0000","last_active":"2024-09-13T16:25:21.150544+0000","last_peered":"2024-09-13T16:25:21.150544+0000","last_clean":"2024-09-13T16:25:21.150544+0000","last_became_active":"2024-09-13T16:25:16.941375+0000","last_became_peered":"2024-09-13T16:25:16.941375+0000","last_unstale":"2024-09-13T16:25:21.150544+0000","last_undegraded":"2024-09-13T16:25:21.150544+0000","last_fullsized":"2024-09-13T16:25:21.150544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:05.301510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941428+0000","last_change":"2024-09-13T16:25:16.931253+0000","last_active":"2024-09-13T16:25:20.941428+0000","last_peered":"2024-09-13T16:25:20.941428+0000","last_clean":"2024-09-13T16:25:20.941428+0000","last_became_active":"2024-09-13T16:25:16.930844+0000","last_became_peered":"2024-09-13T16:25:16.930844+0000","last_unstale":"2024-09-13T16:25:20.941428+0000","last_undegraded":"2024-09-13T16:25:20.941428+0000","last_fullsized":"2024-09-13T16:25:20.941428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:11.873813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149733+0000","last_change":"2024-09-13T16:25:16.946470+0000","last_active":"2024-09-13T16:25:21.149733+0000","last_peered":"2024-09-13T16:25:21.149733+0000","last_clean":"2024-09-13T16:25:21.149733+0000","last_became_active":"2024-09-13T16:25:16.946228+0000","last_became_peered":"2024-09-13T16:25:16.946228+0000","last_unstale":"2024-09-13T16:25:21.149733+0000","last_undegraded":"2024-09-13T16:25:21.149733+0000","last_fullsized":"2024-09-13T16:25:21.149733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:51.187953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943747+0000","last_change":"2024-09-13T16:25:16.933763+0000","last_active":"2024-09-13T16:25:20.943747+0000","last_peered":"2024-09-13T16:25:20.943747+0000","last_clean":"2024-09-13T16:25:20.943747+0000","last_became_active":"2024-09-13T16:25:16.932396+0000","last_became_peered":"2024-09-13T16:25:16.932396+0000","last_unstale":"2024-09-13T16:25:20.943747+0000","last_undegraded":"2024-09-13T16:25:20.943747+0000","last_fullsized":"2024-09-13T16:25:20.943747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:13.889786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943715+0000","last_change":"2024-09-13T16:25:16.931365+0000","last_active":"2024-09-13T16:25:20.943715+0000","last_peered":"2024-09-13T16:25:20.943715+0000","last_clean":"2024-09-13T16:25:20.943715+0000","last_became_active":"2024-09-13T16:25:16.931117+0000","last_became_peered":"2024-09-13T16:25:16.931117+0000","last_unstale":"2024-09-13T16:25:20.943715+0000","last_undegraded":"2024-09-13T16:25:20.943715+0000","last_fullsized":"2024-09-13T16:25:20.943715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.578965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953517+0000","last_change":"2024-09-13T16:25:16.932889+0000","last_active":"2024-09-13T16:25:20.953517+0000","last_peered":"2024-09-13T16:25:20.953517+0000","last_clean":"2024-09-13T16:25:20.953517+0000","last_became_active":"2024-09-13T16:25:16.931952+0000","last_became_peered":"2024-09-13T16:25:16.931952+0000","last_unstale":"2024-09-13T16:25:20.953517+0000","last_undegraded":"2024-09-13T16:25:20.953517+0000","last_fullsized":"2024-09-13T16:25:20.953517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150702+0000","last_change":"2024-09-13T16:25:16.948699+0000","last_active":"2024-09-13T16:25:21.150702+0000","last_peered":"2024-09-13T16:25:21.150702+0000","last_clean":"2024-09-13T16:25:21.150702+0000","last_became_active":"2024-09-13T16:25:16.948387+0000","last_became_peered":"2024-09-13T16:25:16.948387+0000","last_unstale":"2024-09-13T16:25:21.150702+0000","last_undegraded":"2024-09-13T16:25:21.150702+0000","last_fullsized":"2024-09-13T16:25:21.150702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:28.568881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542150,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":693951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99299999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.046}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.026}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1279999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0029999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2689999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.347}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]}]},{"osd":5,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":266240,"data_stored":103091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0409999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89900000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542149,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":103127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82999999999999996}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91700000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]}]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65100000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84299999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.218}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]}]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":103001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0049999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94799999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":103161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.754}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59399999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.075}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.137}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.91100000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542150,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":694007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0609999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.101}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:25:34.436 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:25:34.436 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-13T16:25:34.436 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-13T16:25:34.436 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-13T16:25:34.437 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-13T16:25:34.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 Processor -- start 2024-09-13T16:25:34.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- start start 2024-09-13T16:25:34.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5be4005b10 0x7f5be40e18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5be401c740 0x7f5be401cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be40e1e00 con 0x7f5be400a5b0 2024-09-13T16:25:34.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be4006ce0 con 0x7f5be401c740 2024-09-13T16:25:34.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be40060d0 con 0x7f5be4005b10 2024-09-13T16:25:34.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2768700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5be401c740 0x7f5be401cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf1f67700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5be400a5b0 0x7f5be400a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33640/0 (socket says 172.21.15.119:33640) 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2768700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5be401c740 0x7f5be401cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52016/0 (socket says 172.21.15.119:52016) 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf1f67700 1 -- 172.21.15.119:0/3724483956 learned_addr learned my addr 172.21.15.119:0/3724483956 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf1766700 1 --2- 172.21.15.119:0/3724483956 >> v2:172.21.15.145:3300/0 conn(0x7f5be4005b10 0x7f5be40e18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2768700 1 -- 172.21.15.119:0/3724483956 >> v2:172.21.15.145:3300/0 conn(0x7f5be4005b10 msgr2=0x7f5be40e18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2768700 1 --2- 172.21.15.119:0/3724483956 >> v2:172.21.15.145:3300/0 conn(0x7f5be4005b10 0x7f5be40e18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2768700 1 -- 172.21.15.119:0/3724483956 >> v1:172.21.15.119:6789/0 conn(0x7f5be400a5b0 legacy=0x7f5be400a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2768700 1 -- 172.21.15.119:0/3724483956 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be40063b0 con 0x7f5be401c740 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf0f65700 1 -- 172.21.15.119:0/3724483956 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802884724 0 0) 0x7f5be8001290 con 0x7f5be400a5b0 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2768700 1 --2- 172.21.15.119:0/3724483956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be401cbd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5bec04ff70 tx=0x7f5bec068790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe8aa52b069cd84f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf0f65700 1 -- 172.21.15.119:0/3724483956 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec067690 con 0x7f5be401c740 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf0f65700 1 -- 172.21.15.119:0/3724483956 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec067990 con 0x7f5be401c740 2024-09-13T16:25:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf0f65700 1 -- 172.21.15.119:0/3724483956 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec067d20 con 0x7f5be401c740 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3724483956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 msgr2=0x7f5be401cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3724483956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be401cbd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5bec04ff70 tx=0x7f5bec068790 comp rx=0 tx=0).stop 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3724483956 shutdown_connections 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3724483956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be401cbd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3724483956 >> v2:172.21.15.145:3300/0 conn(0x7f5be4005b10 0x7f5be40e18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3724483956 >> 172.21.15.119:0/3724483956 conn(0x7f5be4036140 msgr2=0x7f5be4036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3724483956 shutdown_connections 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3724483956 wait complete. 2024-09-13T16:25:34.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.549+0000 7f5bf2f69700 1 Processor -- start 2024-09-13T16:25:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 -- start start 2024-09-13T16:25:34.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5be4005b10 0x7f5be404e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5be400a5b0 0x7f5be404b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be41799e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be404bde0 con 0x7f5be401c740 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be404e930 con 0x7f5be4005b10 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be40d9120 con 0x7f5be400a5b0 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be41799e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be41799e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52026/0 (socket says 172.21.15.119:52026) 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 -- 172.21.15.119:0/3304482091 learned_addr learned my addr 172.21.15.119:0/3304482091 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf1f67700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5be4005b10 0x7f5be404e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf1766700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5be400a5b0 0x7f5be404b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 -- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5be400a5b0 msgr2=0x7f5be404b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5be400a5b0 0x7f5be404b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 -- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5be4005b10 msgr2=0x7f5be404e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5be4005b10 0x7f5be404e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 -- 172.21.15.119:0/3304482091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be403a530 con 0x7f5be401c740 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf1f67700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5be4005b10 0x7f5be404e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2768700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be41799e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5bec04f480 tx=0x7f5bec068560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5be27fc700 1 -- 172.21.15.119:0/3304482091 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec09a070 con 0x7f5be401c740 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5be404df60 con 0x7f5be401c740 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5be27fc700 1 -- 172.21.15.119:0/3304482091 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bec072150 con 0x7f5be401c740 2024-09-13T16:25:34.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be403a250 con 0x7f5be401c740 2024-09-13T16:25:34.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5be27fc700 1 -- 172.21.15.119:0/3304482091 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bec0724c0 con 0x7f5be401c740 2024-09-13T16:25:34.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.553+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be404e0f0 con 0x7f5be401c740 2024-09-13T16:25:34.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.557+0000 7f5be27fc700 1 -- 172.21.15.119:0/3304482091 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5bec072620 con 0x7f5be401c740 2024-09-13T16:25:34.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.557+0000 7f5be27fc700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5bd0066730 0x7f5bd0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.557+0000 7f5be27fc700 1 -- 172.21.15.119:0/3304482091 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5bec0ff770 con 0x7f5be401c740 2024-09-13T16:25:34.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.557+0000 7f5bf1f67700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5bd0066730 0x7f5bd0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.557+0000 7f5bf1f67700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5bd0066730 0x7f5bd0068bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5be8007fd0 tx=0x7f5be8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.561+0000 7f5be27fc700 1 -- 172.21.15.119:0/3304482091 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5bec0cc900 con 0x7f5be401c740 2024-09-13T16:25:34.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.773+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f5be40e18c0 con 0x7f5be401c740 2024-09-13T16:25:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.781+0000 7f5be27fc700 1 -- 172.21.15.119:0/3304482091 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+164904 (secure 0 0 0) 0x7f5bec0cff20 con 0x7f5be401c740 2024-09-13T16:25:34.788 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:34.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.781+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5bd0066730 msgr2=0x7f5bd0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.781+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5bd0066730 0x7f5bd0068bf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5be8007fd0 tx=0x7f5be8009040 comp rx=0 tx=0).stop 2024-09-13T16:25:34.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 msgr2=0x7f5be41799e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be41799e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5bec04f480 tx=0x7f5bec068560 comp rx=0 tx=0).stop 2024-09-13T16:25:34.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 shutdown_connections 2024-09-13T16:25:34.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5bd0066730 0x7f5bd0068bf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be401c740 0x7f5be41799e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5be400a5b0 0x7f5be404b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 --2- 172.21.15.119:0/3304482091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5be4005b10 0x7f5be404e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 >> 172.21.15.119:0/3304482091 conn(0x7f5be4036140 msgr2=0x7f5be40e43a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:34.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 shutdown_connections 2024-09-13T16:25:34.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.785+0000 7f5bf2f69700 1 -- 172.21.15.119:0/3304482091 wait complete. 2024-09-13T16:25:34.801 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":5,"flags":0,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6832","nonce":2154272207},{"type":"v1","addr":"172.21.15.145:6833","nonce":2154272207}]},"active_addr":"172.21.15.145:6833/2154272207","active_change":"2024-09-13T16:25:05.402583+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4120,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":656004719}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":88530266}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":3373376912}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.145:0","nonce":2302174072}]}]} 2024-09-13T16:25:34.806 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-13T16:25:34.806 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-13T16:25:34.806 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:34.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac908bd700 1 Processor -- start 2024-09-13T16:25:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac908bd700 1 -- start start 2024-09-13T16:25:34.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac908bd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac908bd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fac8c06bb90 0x7fac8c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac908bd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8c06cf60 con 0x7fac8c0cde70 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac908bd700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8c0b3390 con 0x7fac8c06bb90 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac908bd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c06c350 con 0x7fac8c0adb30 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac8affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fac8c06bb90 0x7fac8c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac8affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fac8c06bb90 0x7fac8c0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52038/0 (socket says 172.21.15.119:52038) 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac8affd700 1 -- 172.21.15.119:0/1898956083 learned_addr learned my addr 172.21.15.119:0/1898956083 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:34.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.917+0000 7fac8b7fe700 1 --2- 172.21.15.119:0/1898956083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8affd700 1 -- 172.21.15.119:0/1898956083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 msgr2=0x7fac8c0cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8affd700 1 --2- 172.21.15.119:0/1898956083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c0cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8affd700 1 -- 172.21.15.119:0/1898956083 >> v1:172.21.15.145:6789/0 conn(0x7fac8c0cde70 legacy=0x7fac8c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8affd700 1 -- 172.21.15.119:0/1898956083 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c06c630 con 0x7fac8c06bb90 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8affd700 1 --2- 172.21.15.119:0/1898956083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c06bb90 0x7fac8c0cd930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009e50 tx=0x7fac7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c75bb4ca0e08fd5f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8a7fc700 1 -- 172.21.15.119:0/1898956083 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c00e070 con 0x7fac8c06bb90 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8a7fc700 1 -- 172.21.15.119:0/1898956083 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7c00b910 con 0x7fac8c06bb90 2024-09-13T16:25:34.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac8a7fc700 1 -- 172.21.15.119:0/1898956083 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac7c00bcc0 con 0x7fac8c06bb90 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 -- 172.21.15.119:0/1898956083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c06bb90 msgr2=0x7fac8c0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 --2- 172.21.15.119:0/1898956083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c06bb90 0x7fac8c0cd930 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009e50 tx=0x7fac7c00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 -- 172.21.15.119:0/1898956083 shutdown_connections 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 --2- 172.21.15.119:0/1898956083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c06bb90 0x7fac8c0cd930 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 --2- 172.21.15.119:0/1898956083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c0cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 -- 172.21.15.119:0/1898956083 >> 172.21.15.119:0/1898956083 conn(0x7fac8c06a350 msgr2=0x7fac8c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 -- 172.21.15.119:0/1898956083 shutdown_connections 2024-09-13T16:25:34.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 -- 172.21.15.119:0/1898956083 wait complete. 2024-09-13T16:25:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 Processor -- start 2024-09-13T16:25:34.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.921+0000 7fac908bd700 1 -- start start 2024-09-13T16:25:34.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8c06bb90 0x7fac8c1f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c0cde70 0x7fac8c142620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c131880 con 0x7fac8c0cde70 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c1f45a0 con 0x7fac8c06bb90 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8c1320e0 con 0x7fac8c0adb30 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c0cde70 0x7fac8c142620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c0cde70 0x7fac8c142620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52052/0 (socket says 172.21.15.119:52052) 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 -- 172.21.15.119:0/1326661108 learned_addr learned my addr 172.21.15.119:0/1326661108 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8b7fe700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8c06bb90 0x7fac8c1f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8affd700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 -- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 msgr2=0x7fac8c159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 -- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8c06bb90 msgr2=0x7fac8c1f4060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8c06bb90 0x7fac8c1f4060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 -- 172.21.15.119:0/1326661108 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7c009b30 con 0x7fac8c0cde70 2024-09-13T16:25:34.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac8bfff700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c0cde70 0x7fac8c142620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c430 tx=0x7fac8400c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac83fff700 1 -- 172.21.15.119:0/1326661108 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840052d0 con 0x7fac8c0cde70 2024-09-13T16:25:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac83fff700 1 -- 172.21.15.119:0/1326661108 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84005430 con 0x7fac8c0cde70 2024-09-13T16:25:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8c1eff30 con 0x7fac8c0cde70 2024-09-13T16:25:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac83fff700 1 -- 172.21.15.119:0/1326661108 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac840055a0 con 0x7fac8c0cde70 2024-09-13T16:25:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.925+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8c0ad1f0 con 0x7fac8c0cde70 2024-09-13T16:25:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.929+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8c06bfd0 con 0x7fac8c0cde70 2024-09-13T16:25:34.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.929+0000 7fac83fff700 1 -- 172.21.15.119:0/1326661108 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fac84003af0 con 0x7fac8c0cde70 2024-09-13T16:25:34.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.929+0000 7fac83fff700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac6c066730 0x7fac6c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:34.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.929+0000 7fac83fff700 1 -- 172.21.15.119:0/1326661108 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fac840838a0 con 0x7fac8c0cde70 2024-09-13T16:25:34.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.929+0000 7fac8b7fe700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac6c066730 0x7fac6c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:34.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.929+0000 7fac8b7fe700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac6c066730 0x7fac6c068bf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fac74009bf0 tx=0x7fac74009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:34.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:34.937+0000 7fac83fff700 1 -- 172.21.15.119:0/1326661108 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fac84050920 con 0x7fac8c0cde70 2024-09-13T16:25:35.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.105+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac8c132220 con 0x7fac8c0cde70 2024-09-13T16:25:35.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac83fff700 1 -- 172.21.15.119:0/1326661108 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7fac84053f40 con 0x7fac8c0cde70 2024-09-13T16:25:35.112 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:35.112 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:35.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac6c066730 msgr2=0x7fac6c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac6c066730 0x7fac6c068bf0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fac74009bf0 tx=0x7fac74009240 comp rx=0 tx=0).stop 2024-09-13T16:25:35.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c0cde70 msgr2=0x7fac8c142620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c0cde70 0x7fac8c142620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac8400c430 tx=0x7fac8400c740 comp rx=0 tx=0).stop 2024-09-13T16:25:35.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 shutdown_connections 2024-09-13T16:25:35.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac6c066730 0x7fac6c068bf0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac8c0cde70 0x7fac8c142620 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac8c0adb30 0x7fac8c159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 --2- 172.21.15.119:0/1326661108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8c06bb90 0x7fac8c1f4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 >> 172.21.15.119:0/1326661108 conn(0x7fac8c06a350 msgr2=0x7fac8c15e500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 shutdown_connections 2024-09-13T16:25:35.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.109+0000 7fac908bd700 1 -- 172.21.15.119:0/1326661108 wait complete. 2024-09-13T16:25:35.128 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-13T16:25:35.128 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.237+0000 7f350fa01700 1 Processor -- start 2024-09-13T16:25:35.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.237+0000 7f350fa01700 1 -- start start 2024-09-13T16:25:35.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3508159260 0x7f350806a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35080ac6c0 0x7f35080704d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350806bc90 con 0x7f35080ceae0 2024-09-13T16:25:35.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3508159640 con 0x7f35080ac6c0 2024-09-13T16:25:35.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350806abf0 con 0x7f3508159260 2024-09-13T16:25:35.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350cf9c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f35080ceae0 0x7f35080abfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33650/0 (socket says 172.21.15.119:33650) 2024-09-13T16:25:35.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350cf9c700 1 -- 172.21.15.119:0/1375024196 learned_addr learned my addr 172.21.15.119:0/1375024196 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350d79d700 1 --2- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3508159260 0x7f350806a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350df9e700 1 --2- 172.21.15.119:0/1375024196 >> v2:172.21.15.145:3300/0 conn(0x7f35080ac6c0 0x7f35080704d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f34fffff700 1 -- 172.21.15.119:0/1375024196 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570999207 0 0) 0x7f350806bc90 con 0x7f35080ceae0 2024-09-13T16:25:35.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f34fffff700 1 -- 172.21.15.119:0/1375024196 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34f40035d0 con 0x7f35080ceae0 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350df9e700 1 -- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3508159260 msgr2=0x7f350806a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350df9e700 1 --2- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3508159260 0x7f350806a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350df9e700 1 -- 172.21.15.119:0/1375024196 >> v1:172.21.15.119:6789/0 conn(0x7f35080ceae0 legacy=0x7f35080abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350df9e700 1 -- 172.21.15.119:0/1375024196 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350806aed0 con 0x7f35080ac6c0 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350d79d700 1 --2- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3508159260 0x7f350806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350df9e700 1 --2- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35080ac6c0 0x7f35080704d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f350000aea0 tx=0x7f350000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c870ff93c37577a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f34fffff700 1 -- 172.21.15.119:0/1375024196 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350000f070 con 0x7f35080ac6c0 2024-09-13T16:25:35.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f34fffff700 1 -- 172.21.15.119:0/1375024196 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f350000a2e0 con 0x7f35080ac6c0 2024-09-13T16:25:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f34fffff700 1 -- 172.21.15.119:0/1375024196 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f350000cb10 con 0x7f35080ac6c0 2024-09-13T16:25:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35080ac6c0 msgr2=0x7f35080704d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 --2- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35080ac6c0 0x7f35080704d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f350000aea0 tx=0x7f350000c330 comp rx=0 tx=0).stop 2024-09-13T16:25:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- 172.21.15.119:0/1375024196 shutdown_connections 2024-09-13T16:25:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 --2- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35080ac6c0 0x7f35080704d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 --2- 172.21.15.119:0/1375024196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3508159260 0x7f350806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- 172.21.15.119:0/1375024196 >> 172.21.15.119:0/1375024196 conn(0x7f35080b0cd0 msgr2=0x7f35080b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- 172.21.15.119:0/1375024196 shutdown_connections 2024-09-13T16:25:35.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.241+0000 7f350fa01700 1 -- 172.21.15.119:0/1375024196 wait complete. 2024-09-13T16:25:35.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 Processor -- start 2024-09-13T16:25:35.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 -- start start 2024-09-13T16:25:35.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35080ac6c0 0x7f350814b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35080ceae0 0x7f35080afe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3508159260 0x7f35080b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3508151a50 con 0x7f35080ceae0 2024-09-13T16:25:35.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3508151d30 con 0x7f3508159260 2024-09-13T16:25:35.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.245+0000 7f350fa01700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3508152010 con 0x7f35080ac6c0 2024-09-13T16:25:35.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35080ceae0 0x7f35080afe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35080ceae0 0x7f35080afe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52056/0 (socket says 172.21.15.119:52056) 2024-09-13T16:25:35.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 -- 172.21.15.119:0/286336610 learned_addr learned my addr 172.21.15.119:0/286336610 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350d79d700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35080ac6c0 0x7f350814b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350df9e700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3508159260 0x7f35080b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 -- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35080ac6c0 msgr2=0x7f350814b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35080ac6c0 0x7f350814b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 -- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3508159260 msgr2=0x7f35080b03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3508159260 0x7f35080b03c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 -- 172.21.15.119:0/286336610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35040094b0 con 0x7f35080ceae0 2024-09-13T16:25:35.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350cf9c700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35080ceae0 0x7f35080afe80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f34f80010e0 tx=0x7f34f800c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f34fdffb700 1 -- 172.21.15.119:0/286336610 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8005d90 con 0x7f35080ceae0 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f34fdffb700 1 -- 172.21.15.119:0/286336610 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f80070d0 con 0x7f35080ceae0 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f34fdffb700 1 -- 172.21.15.119:0/286336610 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34f8007460 con 0x7f35080ceae0 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350000ab80 con 0x7f35080ceae0 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3508159b00 con 0x7f35080ceae0 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3508159d20 con 0x7f35080ceae0 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.249+0000 7f34fdffb700 1 -- 172.21.15.119:0/286336610 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f34f80075c0 con 0x7f35080ceae0 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.253+0000 7f34fdffb700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34ec066730 0x7f34ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.253+0000 7f34fdffb700 1 -- 172.21.15.119:0/286336610 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f34f8083860 con 0x7f35080ceae0 2024-09-13T16:25:35.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.253+0000 7f350d79d700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34ec066730 0x7f34ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.253+0000 7f350d79d700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34ec066730 0x7f34ec068bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35080b14b0 tx=0x7f3504009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.261+0000 7f34fdffb700 1 -- 172.21.15.119:0/286336610 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f34f80508e0 con 0x7f35080ceae0 2024-09-13T16:25:35.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.433+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f350806a350 con 0x7f35080ceae0 2024-09-13T16:25:35.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.437+0000 7f34fdffb700 1 -- 172.21.15.119:0/286336610 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f34f8017090 con 0x7f35080ceae0 2024-09-13T16:25:35.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:35.440 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:35.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34ec066730 msgr2=0x7f34ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34ec066730 0x7f34ec068bf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f35080b14b0 tx=0x7f3504009240 comp rx=0 tx=0).stop 2024-09-13T16:25:35.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35080ceae0 msgr2=0x7f35080afe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35080ceae0 0x7f35080afe80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f34f80010e0 tx=0x7f34f800c7d0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 shutdown_connections 2024-09-13T16:25:35.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34ec066730 0x7f34ec068bf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3508159260 0x7f35080b03c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35080ceae0 0x7f35080afe80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 --2- 172.21.15.119:0/286336610 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35080ac6c0 0x7f350814b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 >> 172.21.15.119:0/286336610 conn(0x7f35080b0cd0 msgr2=0x7f35080b2690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 shutdown_connections 2024-09-13T16:25:35.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.441+0000 7f350fa01700 1 -- 172.21.15.119:0/286336610 wait complete. 2024-09-13T16:25:35.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-13T16:25:35.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-13T16:25:35.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-13T16:25:35.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-13T16:25:35.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-13T16:25:35.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-13T16:25:35.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-13T16:25:35.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-13T16:25:35.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fbc3a114700 1 Processor -- start 2024-09-13T16:25:35.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fbc3a114700 1 -- start start 2024-09-13T16:25:35.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fb23c953700 1 Processor -- start 2024-09-13T16:25:35.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fb23c953700 1 -- start start 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fbc3a114700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbc34153100 0x7fbc34159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fbc3a114700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc34071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fbc3a114700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc34071590 con 0x7fbc34152d20 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fbc3a114700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc340b28a0 con 0x7fbc34153100 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.609+0000 7fbc3a114700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc340b2000 con 0x7fbc34070c10 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc39112700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbc34152d20 0x7fbc340b17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33658/0 (socket says 172.21.15.119:33658) 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc39112700 1 -- 172.21.15.119:0/163713800 learned_addr learned my addr 172.21.15.119:0/163713800 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc38911700 1 --2- 172.21.15.119:0/163713800 >> v2:172.21.15.145:3300/0 conn(0x7fbc34153100 0x7fbc34159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc39913700 1 --2- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc34071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc2bfff700 1 -- 172.21.15.119:0/163713800 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694031643 0 0) 0x7fbc34071590 con 0x7fbc34152d20 2024-09-13T16:25:35.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc2bfff700 1 -- 172.21.15.119:0/163713800 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc200035d0 con 0x7fbc34152d20 2024-09-13T16:25:35.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f8f33700 1 Processor -- start 2024-09-13T16:25:35.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc38911700 1 -- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 msgr2=0x7fbc34071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc38911700 1 --2- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc34071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f8f33700 1 -- start start 2024-09-13T16:25:35.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06797ab700 1 Processor -- start 2024-09-13T16:25:35.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06797ab700 1 -- start start 2024-09-13T16:25:35.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3b18b700 1 Processor -- start 2024-09-13T16:25:35.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3b18b700 1 -- start start 2024-09-13T16:25:35.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23c953700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb238159930 0x7fb238070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23c953700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2380cc080 con 0x7fb23806ea70 2024-09-13T16:25:35.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23c953700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb23806d1b0 con 0x7fb23806e610 2024-09-13T16:25:35.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23c953700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb23806cc60 con 0x7fb238159930 2024-09-13T16:25:35.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23759e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb23806ea70 0x7fb23806b360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33666/0 (socket says 172.21.15.119:33666) 2024-09-13T16:25:35.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23759e700 1 -- 172.21.15.119:0/3768429796 learned_addr learned my addr 172.21.15.119:0/3768429796 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb236d9d700 1 --2- 172.21.15.119:0/3768429796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb238159930 0x7fb238070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f8f33700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f4143f50 0x7f02f40cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f8f33700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f02f4071320 0x7f02f40b26e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f8f33700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02f415aca0 con 0x7f02f406dda0 2024-09-13T16:25:35.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f8f33700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02f40b1920 con 0x7f02f4071320 2024-09-13T16:25:35.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f8f33700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f40b1c00 con 0x7f02f4143f50 2024-09-13T16:25:35.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23659c700 1 -- 172.21.15.119:0/3768429796 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1919212647 0 0) 0x7fb23806d1b0 con 0x7fb23806e610 2024-09-13T16:25:35.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23659c700 1 -- 172.21.15.119:0/3768429796 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2240035d0 con 0x7fb23806e610 2024-09-13T16:25:35.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23659c700 1 -- 172.21.15.119:0/3768429796 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201764442 0 0) 0x7fb230001490 con 0x7fb23806ea70 2024-09-13T16:25:35.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb23659c700 1 -- 172.21.15.119:0/3768429796 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb23806d1b0 con 0x7fb23806ea70 2024-09-13T16:25:35.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f10a92eb700 1 Processor -- start 2024-09-13T16:25:35.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06797ab700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0674158f10 0x7f0674071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06797ab700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f067406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb236d9d700 1 -- 172.21.15.119:0/3768429796 >> v1:172.21.15.145:6789/0 conn(0x7fb23806e610 legacy=0x7fb23806d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb236d9d700 1 -- 172.21.15.119:0/3768429796 >> v1:172.21.15.119:6789/0 conn(0x7fb23806ea70 legacy=0x7fb23806b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb236d9d700 1 -- 172.21.15.119:0/3768429796 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2380708a0 con 0x7fb238159930 2024-09-13T16:25:35.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f4143f50 0x7f02f40cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f4143f50 0x7f02f40cbc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:36966/0 (socket says 172.21.15.119:36966) 2024-09-13T16:25:35.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 -- 172.21.15.119:0/3572671907 learned_addr learned my addr 172.21.15.119:0/3572671907 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc38911700 1 -- 172.21.15.119:0/163713800 >> v1:172.21.15.119:6789/0 conn(0x7fbc34152d20 legacy=0x7fbc340b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc38911700 1 -- 172.21.15.119:0/163713800 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc340b22e0 con 0x7fbc34153100 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc38911700 1 --2- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34153100 0x7fbc34159a40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400bb00 tx=0x7fbc2400be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=164311bec17836b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc2bfff700 1 -- 172.21.15.119:0/163713800 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc24006d30 con 0x7fbc34153100 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc2bfff700 1 -- 172.21.15.119:0/163713800 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc24006e90 con 0x7fbc34153100 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc2bfff700 1 -- 172.21.15.119:0/163713800 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2400dca0 con 0x7fbc34153100 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06797ab700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06740719c0 con 0x7f0674158b30 2024-09-13T16:25:35.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06797ab700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067406c2a0 con 0x7f0674158f10 2024-09-13T16:25:35.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06797ab700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067406de00 con 0x7f0674071570 2024-09-13T16:25:35.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3b18b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7eff341594f0 0x7eff3406d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3b18b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7eff340b3800 0x7eff3406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f3fff700 1 --2- 172.21.15.119:0/3572671907 >> v2:172.21.15.145:3300/0 conn(0x7f02f4071320 0x7f02f40b26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06737fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0674158f10 0x7f0674071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3b18b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff340b2280 con 0x7eff3406da70 2024-09-13T16:25:35.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3b18b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff3406dee0 con 0x7eff341594f0 2024-09-13T16:25:35.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3b18b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff340cc890 con 0x7eff340b3800 2024-09-13T16:25:35.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fb236d9d700 1 --2- 172.21.15.119:0/3768429796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb238159930 0x7fb238070360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb228009880 tx=0x7fb228009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=27cf22e100b9e42c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06737fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0674158f10 0x7f0674071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40672/0 (socket says 172.21.15.119:40672) 2024-09-13T16:25:35.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f06737fe700 1 -- 172.21.15.119:0/701991905 learned_addr learned my addr 172.21.15.119:0/701991905 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 -- 172.21.15.119:0/3572671907 >> v2:172.21.15.145:3300/0 conn(0x7f02f4071320 msgr2=0x7f02f40b26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 --2- 172.21.15.119:0/3572671907 >> v2:172.21.15.145:3300/0 conn(0x7f02f4071320 0x7f02f40b26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 -- 172.21.15.119:0/3572671907 >> v1:172.21.15.119:6789/0 conn(0x7f02f406dda0 legacy=0x7f02f4070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f3fff700 1 --2- 172.21.15.119:0/3572671907 >> v2:172.21.15.145:3300/0 conn(0x7f02f4071320 0x7f02f40b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:35.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 -- 172.21.15.119:0/3572671907 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02f40b1ee0 con 0x7f02f4143f50 2024-09-13T16:25:35.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f0678faa700 1 --2- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f067406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f10a92eb700 1 -- start start 2024-09-13T16:25:35.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3a98a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7eff340b3800 0x7eff3406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff39988700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7eff3406da70 0x7eff340b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33684/0 (socket says 172.21.15.119:33684) 2024-09-13T16:25:35.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc3a114700 1 -- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34153100 msgr2=0x7fbc34159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff3a98a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7eff340b3800 0x7eff3406fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40676/0 (socket says 172.21.15.119:40676) 2024-09-13T16:25:35.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff39988700 1 -- 172.21.15.119:0/323291213 learned_addr learned my addr 172.21.15.119:0/323291213 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc3a114700 1 --2- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34153100 0x7fbc34159a40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fbc2400bb00 tx=0x7fbc2400be10 comp rx=0 tx=0).stop 2024-09-13T16:25:35.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc3a114700 1 -- 172.21.15.119:0/163713800 shutdown_connections 2024-09-13T16:25:35.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc3a114700 1 --2- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc34071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc3a114700 1 --2- 172.21.15.119:0/163713800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34153100 0x7fbc34159a40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc3a114700 1 -- 172.21.15.119:0/163713800 >> 172.21.15.119:0/163713800 conn(0x7fbc340b04f0 msgr2=0x7fbc340b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f02f37fe700 1 --2- 172.21.15.119:0/3572671907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f4143f50 0x7f02f40cbc30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009910 tx=0x7f02dc009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d2fc1a75fe71082 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7fbc3a114700 1 -- 172.21.15.119:0/163713800 shutdown_connections 2024-09-13T16:25:35.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff39187700 1 -- 172.21.15.119:0/323291213 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053905602 0 0) 0x7eff340b2280 con 0x7eff3406da70 2024-09-13T16:25:35.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7eff39187700 1 -- 172.21.15.119:0/323291213 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff200035d0 con 0x7eff3406da70 2024-09-13T16:25:35.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f10a92eb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10a40cd8d0 0x7f10a406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.613+0000 7f10a92eb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10a40b17f0 0x7f10a40cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10a40705d0 con 0x7f10a40b2200 2024-09-13T16:25:35.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10a406d0d0 con 0x7f10a40b17f0 2024-09-13T16:25:35.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a406fca0 con 0x7f10a40cd8d0 2024-09-13T16:25:35.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06737fe700 1 -- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 msgr2=0x7f067406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff3a189700 1 --2- 172.21.15.119:0/323291213 >> v2:172.21.15.119:3300/0 conn(0x7eff341594f0 0x7eff3406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06737fe700 1 --2- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f067406ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06737fe700 1 -- 172.21.15.119:0/701991905 >> v1:172.21.15.119:6789/0 conn(0x7f0674158b30 legacy=0x7f067406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06737fe700 1 -- 172.21.15.119:0/701991905 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067406db20 con 0x7f0674158f10 2024-09-13T16:25:35.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10a40b17f0 0x7f10a40cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 -- 172.21.15.119:0/163713800 wait complete. 2024-09-13T16:25:35.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10a40b17f0 0x7f10a40cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52074/0 (socket says 172.21.15.119:52074) 2024-09-13T16:25:35.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 -- 172.21.15.119:0/76362969 learned_addr learned my addr 172.21.15.119:0/76362969 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f27fc700 1 -- 172.21.15.119:0/3572671907 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc004230 con 0x7f02f4143f50 2024-09-13T16:25:35.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff39187700 1 -- 172.21.15.119:0/323291213 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3879814429 0 0) 0x7eff200035d0 con 0x7eff3406da70 2024-09-13T16:25:35.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff3a98a700 1 -- 172.21.15.119:0/323291213 >> v2:172.21.15.119:3300/0 conn(0x7eff341594f0 msgr2=0x7eff3406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff3a98a700 1 --2- 172.21.15.119:0/323291213 >> v2:172.21.15.119:3300/0 conn(0x7eff341594f0 0x7eff3406d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f27fc700 1 -- 172.21.15.119:0/3572671907 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc004390 con 0x7f02f4143f50 2024-09-13T16:25:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f27fc700 1 -- 172.21.15.119:0/3572671907 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc009e60 con 0x7f02f4143f50 2024-09-13T16:25:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff3a98a700 1 -- 172.21.15.119:0/323291213 >> v1:172.21.15.119:6789/0 conn(0x7eff3406da70 legacy=0x7eff340b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff3a98a700 1 -- 172.21.15.119:0/323291213 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340b1970 con 0x7eff340b3800 2024-09-13T16:25:35.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 -- 172.21.15.119:0/76362969 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10a40cd8d0 msgr2=0x7f10a406cb90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:25:35.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 --2- 172.21.15.119:0/76362969 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10a40cd8d0 0x7f10a406cb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 -- 172.21.15.119:0/76362969 >> v1:172.21.15.145:6789/0 conn(0x7f10a40b2200 legacy=0x7f10a4070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23659c700 1 -- 172.21.15.119:0/3768429796 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228011830 con 0x7fb238159930 2024-09-13T16:25:35.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 -- 172.21.15.119:0/76362969 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a406ff80 con 0x7f10a40b17f0 2024-09-13T16:25:35.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23659c700 1 -- 172.21.15.119:0/3768429796 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb228011990 con 0x7fb238159930 2024-09-13T16:25:35.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23659c700 1 -- 172.21.15.119:0/3768429796 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228011d00 con 0x7fb238159930 2024-09-13T16:25:35.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a37fe700 1 --2- 172.21.15.119:0/76362969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10a40b17f0 0x7f10a40cb800 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1094009910 tx=0x7f1094009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5e775214618b61 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a2ffd700 1 -- 172.21.15.119:0/76362969 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094004230 con 0x7f10a40b17f0 2024-09-13T16:25:35.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a2ffd700 1 -- 172.21.15.119:0/76362969 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1094004390 con 0x7f10a40b17f0 2024-09-13T16:25:35.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a2ffd700 1 -- 172.21.15.119:0/76362969 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094009e60 con 0x7f10a40b17f0 2024-09-13T16:25:35.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 -- 172.21.15.119:0/3572671907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f4143f50 msgr2=0x7f02f40cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 --2- 172.21.15.119:0/3572671907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f4143f50 0x7f02f40cbc30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f02dc009910 tx=0x7f02dc009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:35.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 Processor -- start 2024-09-13T16:25:35.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 -- 172.21.15.119:0/3572671907 shutdown_connections 2024-09-13T16:25:35.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 --2- 172.21.15.119:0/3572671907 >> v2:172.21.15.145:3300/0 conn(0x7f02f4071320 0x7f02f40b26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 --2- 172.21.15.119:0/3572671907 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f4143f50 0x7f02f40cbc30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 -- 172.21.15.119:0/3572671907 >> 172.21.15.119:0/3572671907 conn(0x7f02f40b04f0 msgr2=0x7f02f40b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 -- start start 2024-09-13T16:25:35.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc159ac3700 1 Processor -- start 2024-09-13T16:25:35.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc159ac3700 1 -- start start 2024-09-13T16:25:35.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06737fe700 1 --2- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 0x7f0674071030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0664009910 tx=0x7f0664009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=23edbc3e8a0d0db8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f0672ffd700 1 -- 172.21.15.119:0/701991905 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664011830 con 0x7f0674158f10 2024-09-13T16:25:35.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc159ac3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540ce360 0x7fc154070c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f0672ffd700 1 -- 172.21.15.119:0/701991905 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0664011990 con 0x7f0674158f10 2024-09-13T16:25:35.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f0672ffd700 1 -- 172.21.15.119:0/701991905 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664011d40 con 0x7f0674158f10 2024-09-13T16:25:35.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc159ac3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc154071150 0x7fc1540b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- 172.21.15.119:0/76362969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10a40b17f0 msgr2=0x7f10a40cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 --2- 172.21.15.119:0/76362969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10a40b17f0 0x7f10a40cb800 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1094009910 tx=0x7f1094009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:35.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- 172.21.15.119:0/76362969 shutdown_connections 2024-09-13T16:25:35.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 --2- 172.21.15.119:0/76362969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10a40b17f0 0x7f10a40cb800 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 --2- 172.21.15.119:0/76362969 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10a40cd8d0 0x7f10a406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- 172.21.15.119:0/76362969 >> 172.21.15.119:0/76362969 conn(0x7f10a406a350 msgr2=0x7f10a406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f2e6e201700 1 Processor -- start 2024-09-13T16:25:35.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc159ac3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1540b25a0 con 0x7fc1540cdf80 2024-09-13T16:25:35.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc159ac3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc154071600 con 0x7fc154071150 2024-09-13T16:25:35.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc159ac3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540b1c70 con 0x7fc1540ce360 2024-09-13T16:25:35.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc158ac1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc1540cdf80 0x7fc1541693a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33698/0 (socket says 172.21.15.119:33698) 2024-09-13T16:25:35.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc158ac1700 1 -- 172.21.15.119:0/1883728852 learned_addr learned my addr 172.21.15.119:0/1883728852 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc153fff700 1 --2- 172.21.15.119:0/1883728852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540ce360 0x7fc154070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 -- 172.21.15.119:0/3572671907 shutdown_connections 2024-09-13T16:25:35.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc1592c2700 1 --2- 172.21.15.119:0/1883728852 >> v2:172.21.15.119:3300/0 conn(0x7fc154071150 0x7fc1540b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23c953700 1 -- 172.21.15.119:0/3768429796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb238159930 msgr2=0x7fb238070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23c953700 1 --2- 172.21.15.119:0/3768429796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb238159930 0x7fb238070360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb228009880 tx=0x7fb228009b90 comp rx=0 tx=0).stop 2024-09-13T16:25:35.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f2e6e201700 1 -- start start 2024-09-13T16:25:35.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23c953700 1 -- 172.21.15.119:0/3768429796 shutdown_connections 2024-09-13T16:25:35.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23c953700 1 --2- 172.21.15.119:0/3768429796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb238159930 0x7fb238070360 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fb23c953700 1 -- 172.21.15.119:0/3768429796 >> 172.21.15.119:0/3768429796 conn(0x7fb2380ae4b0 msgr2=0x7fb2380b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc1537fe700 1 -- 172.21.15.119:0/1883728852 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3238733096 0 0) 0x7fc1540b25a0 con 0x7fc1540cdf80 2024-09-13T16:25:35.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc3414afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34152d20 0x7fbc340c82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34153100 0x7fbc34155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc3415c590 con 0x7fbc34153100 2024-09-13T16:25:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341f2f70 con 0x7fbc34152d20 2024-09-13T16:25:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc3a114700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc341f2070 con 0x7fbc34070c10 2024-09-13T16:25:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff3a98a700 1 --2- 172.21.15.119:0/323291213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 0x7eff3406fc00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff340cc040 tx=0x7eff2c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7277c8326241193 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc1537fe700 1 -- 172.21.15.119:0/1883728852 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1400035d0 con 0x7fc1540cdf80 2024-09-13T16:25:35.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc153fff700 1 -- 172.21.15.119:0/1883728852 >> v2:172.21.15.119:3300/0 conn(0x7fc154071150 msgr2=0x7fc1540b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 -- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 msgr2=0x7f0674071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 --2- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 0x7f0674071030 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0664009910 tx=0x7f0664009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:35.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc39112700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc3414afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc38911700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34152d20 0x7fbc340c82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc1537fe700 1 -- 172.21.15.119:0/1883728852 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 892696473 0 0) 0x7fc1400035d0 con 0x7fc1540cdf80 2024-09-13T16:25:35.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 -- 172.21.15.119:0/701991905 shutdown_connections 2024-09-13T16:25:35.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 --2- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f067406ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 --2- 172.21.15.119:0/701991905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 0x7f0674071030 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc39112700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc3414afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:37008/0 (socket says 172.21.15.119:37008) 2024-09-13T16:25:35.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc39112700 1 -- 172.21.15.119:0/2283078566 learned_addr learned my addr 172.21.15.119:0/2283078566 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc39913700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34153100 0x7fbc34155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 -- 172.21.15.119:0/701991905 >> 172.21.15.119:0/701991905 conn(0x7f067406a350 msgr2=0x7f067406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- 172.21.15.119:0/76362969 shutdown_connections 2024-09-13T16:25:35.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc153fff700 1 --2- 172.21.15.119:0/1883728852 >> v2:172.21.15.119:3300/0 conn(0x7fc154071150 0x7fc1540b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc153fff700 1 -- 172.21.15.119:0/1883728852 >> v1:172.21.15.119:6789/0 conn(0x7fc1540cdf80 legacy=0x7fc1541693a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc153fff700 1 -- 172.21.15.119:0/1883728852 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1540b1f50 con 0x7fc1540ce360 2024-09-13T16:25:35.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc1592c2700 1 --2- 172.21.15.119:0/1883728852 >> v2:172.21.15.119:3300/0 conn(0x7fc154071150 0x7fc1540b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:35.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff39187700 1 -- 172.21.15.119:0/323291213 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff2c00e070 con 0x7eff340b3800 2024-09-13T16:25:35.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff39187700 1 -- 172.21.15.119:0/323291213 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff2c003a40 con 0x7eff340b3800 2024-09-13T16:25:35.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 -- 172.21.15.119:0/701991905 shutdown_connections 2024-09-13T16:25:35.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f10a92eb700 1 -- 172.21.15.119:0/76362969 wait complete. 2024-09-13T16:25:35.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f06797ab700 1 -- 172.21.15.119:0/701991905 wait complete. 2024-09-13T16:25:35.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fc153fff700 1 --2- 172.21.15.119:0/1883728852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540ce360 0x7fc154070c10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009d90 tx=0x7fc14c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=231092d91df024fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7fbc39913700 1 -- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 msgr2=0x7fbc3414afc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc39913700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc3414afc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc39913700 1 -- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34152d20 msgr2=0x7fbc340c82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc39913700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc34152d20 0x7fbc340c82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2e68142440 0x7f2e680cbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2e6806d140 0x7f2e6815cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc39913700 1 -- 172.21.15.119:0/2283078566 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2400b760 con 0x7fbc34153100 2024-09-13T16:25:35.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc39112700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc34070c10 0x7fbc3414afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:35.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6815d0b0 con 0x7f2e6815c710 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e680b1940 con 0x7f2e6806d140 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e6806d590 con 0x7f2e68142440 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e66ffd700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2e6815c710 0x7f2e6806ca30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:53172/0 (socket says 172.21.15.119:53172) 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e66ffd700 1 -- 172.21.15.119:0/3135908549 learned_addr learned my addr 172.21.15.119:0/3135908549 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e67fff700 1 --2- 172.21.15.119:0/3135908549 >> v2:172.21.15.119:3300/0 conn(0x7f2e6806d140 0x7f2e6815cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc1537fe700 1 -- 172.21.15.119:0/1883728852 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c00e070 con 0x7fc1540ce360 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc1537fe700 1 -- 172.21.15.119:0/1883728852 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c003a40 con 0x7fc1540ce360 2024-09-13T16:25:35.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc1537fe700 1 -- 172.21.15.119:0/1883728852 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc144001670 con 0x7fc1540ce360 2024-09-13T16:25:35.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc39913700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34153100 0x7fbc34155840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009d90 tx=0x7fbc2c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7f02f8f33700 1 -- 172.21.15.119:0/3572671907 wait complete. 2024-09-13T16:25:35.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 Processor -- start 2024-09-13T16:25:35.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fb23c953700 1 -- 172.21.15.119:0/3768429796 shutdown_connections 2024-09-13T16:25:35.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fb23c953700 1 -- 172.21.15.119:0/3768429796 wait complete. 2024-09-13T16:25:35.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c009c00 con 0x7fbc34153100 2024-09-13T16:25:35.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e67fff700 1 -- 172.21.15.119:0/3135908549 >> v2:172.21.15.145:3300/0 conn(0x7f2e68142440 msgr2=0x7f2e680cbfc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:25:35.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e67fff700 1 --2- 172.21.15.119:0/3135908549 >> v2:172.21.15.145:3300/0 conn(0x7f2e68142440 0x7f2e680cbfc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e67fff700 1 -- 172.21.15.119:0/3135908549 >> v1:172.21.15.145:6789/0 conn(0x7f2e6815c710 legacy=0x7f2e6806ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e67fff700 1 -- 172.21.15.119:0/3135908549 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e680b1b50 con 0x7f2e6806d140 2024-09-13T16:25:35.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc2c00f040 con 0x7fbc34153100 2024-09-13T16:25:35.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc341f2350 con 0x7fbc34153100 2024-09-13T16:25:35.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e67fff700 1 --2- 172.21.15.119:0/3135908549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e6815cb70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2e6806d930 tx=0x7f2e60009df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aebca46e8aaed3de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.617+0000 7eff39187700 1 -- 172.21.15.119:0/323291213 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff24001670 con 0x7eff340b3800 2024-09-13T16:25:35.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 -- 172.21.15.119:0/323291213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 msgr2=0x7eff3406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 --2- 172.21.15.119:0/323291213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 0x7eff3406fc00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7eff340cc040 tx=0x7eff2c00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 -- 172.21.15.119:0/323291213 shutdown_connections 2024-09-13T16:25:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 --2- 172.21.15.119:0/323291213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 0x7eff3406fc00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 --2- 172.21.15.119:0/323291213 >> v2:172.21.15.119:3300/0 conn(0x7eff341594f0 0x7eff3406d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 -- 172.21.15.119:0/323291213 >> 172.21.15.119:0/323291213 conn(0x7eff3406a350 msgr2=0x7eff3406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 -- 172.21.15.119:0/323291213 shutdown_connections 2024-09-13T16:25:35.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f06797ab700 1 Processor -- start 2024-09-13T16:25:35.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 -- 172.21.15.119:0/1883728852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540ce360 msgr2=0x7fc154070c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 --2- 172.21.15.119:0/1883728852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540ce360 0x7fc154070c10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fc14c009d90 tx=0x7fc14c00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:35.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7eff3b18b700 1 -- 172.21.15.119:0/323291213 wait complete. 2024-09-13T16:25:35.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc2c014480 con 0x7fbc34153100 2024-09-13T16:25:35.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc341f2b90 con 0x7fbc34153100 2024-09-13T16:25:35.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e667fc700 1 -- 172.21.15.119:0/3135908549 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e6000de40 con 0x7f2e6806d140 2024-09-13T16:25:35.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 -- start start 2024-09-13T16:25:35.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f406dda0 0x7f02f41ff430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02f4071320 0x7f02f4141690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4143f50 0x7f02f413f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f413d670 con 0x7f02f4071320 2024-09-13T16:25:35.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02f413ca30 con 0x7f02f4143f50 2024-09-13T16:25:35.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f8f33700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02f41ff970 con 0x7f02f406dda0 2024-09-13T16:25:35.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e667fc700 1 -- 172.21.15.119:0/3135908549 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e60010470 con 0x7f2e6806d140 2024-09-13T16:25:35.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fb23c953700 1 Processor -- start 2024-09-13T16:25:35.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 Processor -- start 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 -- start start 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10a40b17f0 0x7f10a40aef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10a40b2200 0x7f10a414e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10a40cd8d0 0x7f10a40aaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40af4c0 con 0x7f10a40b17f0 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40a8550 con 0x7f10a40b2200 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a92eb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10a40af6f0 con 0x7f10a40cd8d0 2024-09-13T16:25:35.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f406dda0 0x7f02f41ff430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10a40b2200 0x7f10a414e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10a40b2200 0x7f10a414e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40680/0 (socket says 172.21.15.119:40680) 2024-09-13T16:25:35.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f406dda0 0x7f02f41ff430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:37022/0 (socket says 172.21.15.119:37022) 2024-09-13T16:25:35.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 -- 172.21.15.119:0/2355019760 learned_addr learned my addr 172.21.15.119:0/2355019760 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 -- 172.21.15.119:0/4235149457 learned_addr learned my addr 172.21.15.119:0/4235149457 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f3fff700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4143f50 0x7f02f413f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbc3406bd30 con 0x7fbc34153100 2024-09-13T16:25:35.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fb23c953700 1 -- start start 2024-09-13T16:25:35.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f06797ab700 1 -- start start 2024-09-13T16:25:35.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 -- 172.21.15.119:0/1883728852 shutdown_connections 2024-09-13T16:25:35.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 --2- 172.21.15.119:0/1883728852 >> v2:172.21.15.119:3300/0 conn(0x7fc154071150 0x7fc1540b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 --2- 172.21.15.119:0/1883728852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc1540ce360 0x7fc154070c10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 -- 172.21.15.119:0/1883728852 >> 172.21.15.119:0/1883728852 conn(0x7fc1540b04f0 msgr2=0x7fc1540b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 -- 172.21.15.119:0/1883728852 shutdown_connections 2024-09-13T16:25:35.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7fc159ac3700 1 -- 172.21.15.119:0/1883728852 wait complete. 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f0674150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0674158b30 0x7f06740aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 0x7f06740af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06740a9ab0 con 0x7f0674158b30 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067415a420 con 0x7f0674158f10 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0674150f80 con 0x7f0674071570 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0674158b30 0x7f06740aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0674158b30 0x7f06740aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52104/0 (socket says 172.21.15.119:52104) 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 -- 172.21.15.119:0/3165677184 learned_addr learned my addr 172.21.15.119:0/3165677184 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f0673fff700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f0674150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f0678faa700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 0x7f06740af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb23806e610 0x7fb2381511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb23806ea70 0x7fb238099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb238159930 0x7fb2380afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2381f7c90 con 0x7fb238159930 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2381f7080 con 0x7fb23806e610 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2381f7360 con 0x7fb23806ea70 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb238159930 0x7fb2380afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb238159930 0x7fb2380afa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52102/0 (socket says 172.21.15.119:52102) 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23759e700 1 -- 172.21.15.119:0/2332721683 learned_addr learned my addr 172.21.15.119:0/2332721683 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb236d9d700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb23806e610 0x7fb2381511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 -- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10a40cd8d0 msgr2=0x7f10a40aaa40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10a40cd8d0 0x7f10a40aaa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 -- 172.21.15.119:0/4235149457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10a40b17f0 msgr2=0x7f10a40aef80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10a40b17f0 0x7f10a40aef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 -- 172.21.15.119:0/4235149457 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1094009570 con 0x7f10a40b2200 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a37fe700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10a40b2200 0x7f10a414e780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1094009c60 tx=0x7f1094004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10940036c0 con 0x7f10a40b2200 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f109401f070 con 0x7f10a40b2200 2024-09-13T16:25:35.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbc2c0145e0 con 0x7fbc34153100 2024-09-13T16:25:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 Processor -- start 2024-09-13T16:25:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 -- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 msgr2=0x7f0674150a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f0674150a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 -- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 msgr2=0x7f06740af520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0674158f10 0x7f06740af520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 -- 172.21.15.119:0/3165677184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0664009570 con 0x7f0674158b30 2024-09-13T16:25:35.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f0673fff700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0674071570 0x7f0674150a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:35.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 Processor -- start 2024-09-13T16:25:35.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1094003a40 con 0x7f10a40b2200 2024-09-13T16:25:35.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 -- start start 2024-09-13T16:25:35.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb236d9d700 1 -- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb23806ea70 msgr2=0x7fb238099a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:25:35.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb236d9d700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb23806ea70 0x7fb238099a50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb236d9d700 1 -- 172.21.15.119:0/2332721683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb238159930 msgr2=0x7fb2380afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb236d9d700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb238159930 0x7fb2380afa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb236d9d700 1 -- 172.21.15.119:0/2332721683 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2280094b0 con 0x7fb23806e610 2024-09-13T16:25:35.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06737fe700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0674158b30 0x7f06740aefe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0664009f70 tx=0x7f0664004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc29ffb700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc18066800 0x7fbc18068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664004520 con 0x7f0674158b30 2024-09-13T16:25:35.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066401e070 con 0x7f0674158b30 2024-09-13T16:25:35.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0664003900 con 0x7f0674158b30 2024-09-13T16:25:35.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbc2c084260 con 0x7fbc34153100 2024-09-13T16:25:35.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb236d9d700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb23806e610 0x7fb2381511d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb228015000 tx=0x7fb228004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06740a87e0 con 0x7f0674158b30 2024-09-13T16:25:35.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067415a950 con 0x7f0674158b30 2024-09-13T16:25:35.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc29ffb700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7fbc1806fcd0 0x7fbc18072110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc39112700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc18066800 0x7fbc18068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbc180727e0 con 0x7fbc1806fcd0 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc2c084560 con 0x7fbc34153100 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc38911700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7fbc1806fcd0 0x7fbc18072110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fbc38911700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7fbc1806fcd0 0x7fbc18072110 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 -- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4143f50 msgr2=0x7f02f413f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02f4143f50 0x7f02f413f7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 -- 172.21.15.119:0/2355019760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02f4071320 msgr2=0x7f02f4141690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02f4071320 0x7f02f4141690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 -- 172.21.15.119:0/2355019760 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc009570 con 0x7f02f406dda0 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f02f37fe700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f406dda0 0x7f02f41ff430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f02dc004000 tx=0x7f02dc004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc0036c0 con 0x7f02f406dda0 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02dc003840 con 0x7f02f406dda0 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02dc003bf0 con 0x7f02f406dda0 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02f413ccd0 con 0x7f02f406dda0 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02f413d290 con 0x7f02f406dda0 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02d8000fc0 con 0x7f02f406dda0 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f02dc021070 con 0x7f02f406dda0 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff3406da70 0x7eff34158390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 0x7eff3414ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff341594f0 0x7eff340ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff3414fb30 con 0x7eff3406da70 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34151450 con 0x7eff340b3800 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3b18b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff3414f850 con 0x7eff341594f0 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff39988700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 0x7eff3414ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff39988700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 0x7eff3414ed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40714/0 (socket says 172.21.15.119:40714) 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff39988700 1 -- 172.21.15.119:0/872056170 learned_addr learned my addr 172.21.15.119:0/872056170 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3a189700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff3406da70 0x7eff34158390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3a98a700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff341594f0 0x7eff340ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3a189700 1 -- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff341594f0 msgr2=0x7eff340ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3a189700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7eff341594f0 0x7eff340ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3a189700 1 -- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 msgr2=0x7eff3414ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3a189700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7eff340b3800 0x7eff3414ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7eff3a189700 1 -- 172.21.15.119:0/872056170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff2c009a70 con 0x7eff3406da70 2024-09-13T16:25:35.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2280198d0 con 0x7fb23806e610 2024-09-13T16:25:35.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb22801e070 con 0x7fb23806e610 2024-09-13T16:25:35.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb228019c30 con 0x7fb23806e610 2024-09-13T16:25:35.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2381f7640 con 0x7fb23806e610 2024-09-13T16:25:35.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2381f6d30 con 0x7fb23806e610 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- 172.21.15.119:0/3135908549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 msgr2=0x7f2e6815cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 --2- 172.21.15.119:0/3135908549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e6815cb70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2e6806d930 tx=0x7f2e60009df0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- 172.21.15.119:0/3135908549 shutdown_connections 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 --2- 172.21.15.119:0/3135908549 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e6815cb70 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 --2- 172.21.15.119:0/3135908549 >> v2:172.21.15.145:3300/0 conn(0x7f2e68142440 0x7f2e680cbfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- 172.21.15.119:0/3135908549 >> 172.21.15.119:0/3135908549 conn(0x7f2e6806a350 msgr2=0x7f2e6806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- 172.21.15.119:0/3135908549 shutdown_connections 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.621+0000 7f2e6e201700 1 -- 172.21.15.119:0/3135908549 wait complete. 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f2e6e201700 1 Processor -- start 2024-09-13T16:25:35.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a40af9d0 con 0x7f10a40b2200 2024-09-13T16:25:35.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a40aff90 con 0x7f10a40b2200 2024-09-13T16:25:35.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02ebfff700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02d4066780 0x7f02d4068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0664003a60 con 0x7f0674158b30 2024-09-13T16:25:35.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f067406c980 con 0x7f0674158b30 2024-09-13T16:25:35.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0670ff9700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0660066730 0x7f0660068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0664020070 con 0x7f0674158b30 2024-09-13T16:25:35.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0673fff700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0660066730 0x7f0660068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0670ff9700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f066006fcf0 0x7f0660072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0660072800 con 0x7f066006fcf0 2024-09-13T16:25:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0664087c60 con 0x7f0674158b30 2024-09-13T16:25:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fbc180727e0 con 0x7fbc1806fcd0 2024-09-13T16:25:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fbc39112700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc18066800 0x7fbc18068cc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbc340c9270 tx=0x7fbc300060e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7eff3a189700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff3406da70 0x7eff34158390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e810 tx=0x7eff3000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff3000c070 con 0x7eff3406da70 2024-09-13T16:25:35.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff30013610 con 0x7eff3406da70 2024-09-13T16:25:35.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff300139a0 con 0x7eff3406da70 2024-09-13T16:25:35.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb2380cbac0 con 0x7fb23806e610 2024-09-13T16:25:35.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7f2e6e201700 1 -- start start 2024-09-13T16:25:35.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e680accb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e68142440 0x7f2e680b0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f10a406d380 con 0x7f10a40b2200 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 -- start start 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc154071150 0x7fc1541fe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1540cdf80 0x7fc15415caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1540ce360 0x7fc15415d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15415c6c0 con 0x7fc1540ce360 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc15420b690 con 0x7fc1540cdf80 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc159ac3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1541fe820 con 0x7fc154071150 2024-09-13T16:25:35.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc153fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1540cdf80 0x7fc15415caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc153fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1540cdf80 0x7fc15415caf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40718/0 (socket says 172.21.15.119:40718) 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.625+0000 7fc153fff700 1 -- 172.21.15.119:0/1309767734 learned_addr learned my addr 172.21.15.119:0/1309767734 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc1592c2700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1540ce360 0x7fc15415d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc153fff700 1 -- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc154071150 msgr2=0x7fc1541fe2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc153fff700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc154071150 0x7fc1541fe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc153fff700 1 -- 172.21.15.119:0/1309767734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1540ce360 msgr2=0x7fc15415d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc153fff700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc1540ce360 0x7fc15415d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc153fff700 1 -- 172.21.15.119:0/1309767734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0094b0 con 0x7fc1540cdf80 2024-09-13T16:25:35.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff34156500 con 0x7eff3406da70 2024-09-13T16:25:35.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff34156ac0 con 0x7eff3406da70 2024-09-13T16:25:35.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0678faa700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f066006fcf0 0x7f0660072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e6815c710 0x7f2e680b06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680ad1f0 con 0x7f2e6806d140 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e680ae0f0 con 0x7f2e6815c710 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e680accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e68203460 con 0x7f2e68142440 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e680accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52126/0 (socket says 172.21.15.119:52126) 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 -- 172.21.15.119:0/1303298520 learned_addr learned my addr 172.21.15.119:0/1303298520 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e66ffd700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e68142440 0x7f2e680b0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e67fff700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e6815c710 0x7f2e680b06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 -- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e68142440 msgr2=0x7f2e680b0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e68142440 0x7f2e680b0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 -- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e6815c710 msgr2=0x7f2e680b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e6815c710 0x7f2e680b06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 -- 172.21.15.119:0/1303298520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e60009770 con 0x7f2e6806d140 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e677fe700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e680accb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800b520 tx=0x7f2e5800b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0678faa700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f066006fcf0 0x7f0660072130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f02dc085c40 con 0x7f02f406dda0 2024-09-13T16:25:35.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02f2ffd700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02d4066780 0x7f02d4068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02ebfff700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f02d406fdc0 0x7f02d4072200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1094021030 con 0x7f10a40b2200 2024-09-13T16:25:35.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e4ffff700 1 -- 172.21.15.119:0/1303298520 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58011830 con 0x7f2e6806d140 2024-09-13T16:25:35.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02d40728d0 con 0x7f02d406fdc0 2024-09-13T16:25:35.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02f3fff700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f02d406fdc0 0x7f02d4072200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e4ffff700 1 -- 172.21.15.119:0/1303298520 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e58011990 con 0x7f2e6806d140 2024-09-13T16:25:35.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e680b0cf0 con 0x7f2e6806d140 2024-09-13T16:25:35.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02dc088350 con 0x7f02f406dda0 2024-09-13T16:25:35.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7eff340b2710 con 0x7eff3406da70 2024-09-13T16:25:35.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02f2ffd700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02d4066780 0x7f02d4068c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02f413c820 tx=0x7f02ec007a60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0673fff700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0660066730 0x7f0660068bf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0668001c90 tx=0x7f0668009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02f3fff700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f02d406fdc0 0x7f02d4072200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a0ff9700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1090066760 0x7f1090068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f10940861e0 con 0x7f10a40b2200 2024-09-13T16:25:35.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a3fff700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1090066760 0x7f1090068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a0ff9700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7f109006fbb0 0x7f1090071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f10900726c0 con 0x7f109006fbb0 2024-09-13T16:25:35.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1094086590 con 0x7f10a40b2200 2024-09-13T16:25:35.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc153fff700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1540cdf80 0x7fc15415caf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc1540b1db0 tx=0x7fc14c004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e4ffff700 1 -- 172.21.15.119:0/1303298520 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e58011d40 con 0x7f2e6806d140 2024-09-13T16:25:35.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e680b12b0 con 0x7f2e6806d140 2024-09-13T16:25:35.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a8aea700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7f109006fbb0 0x7f1090071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f10a3fff700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1090066760 0x7f1090068c20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10a40cb2e0 tx=0x7f1098006be0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f0660072800 con 0x7f066006fcf0 2024-09-13T16:25:35.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f02d40728d0 con 0x7f02d406fdc0 2024-09-13T16:25:35.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7f10a8aea700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7f109006fbb0 0x7f1090071ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb228024400 con 0x7fb23806e610 2024-09-13T16:25:35.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc1517fa700 1 -- 172.21.15.119:0/1309767734 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c004670 con 0x7fc1540cdf80 2024-09-13T16:25:35.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc1517fa700 1 -- 172.21.15.119:0/1309767734 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc14c0047f0 con 0x7fc1540cdf80 2024-09-13T16:25:35.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc1517fa700 1 -- 172.21.15.119:0/1309767734 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc14c012670 con 0x7fc1540cdf80 2024-09-13T16:25:35.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc14c009a70 con 0x7fc1540cdf80 2024-09-13T16:25:35.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.629+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc15415e500 con 0x7fc1540cdf80 2024-09-13T16:25:35.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb22f7fe700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb218066730 0x7fb218068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb228085c80 con 0x7fb23806e610 2024-09-13T16:25:35.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb22f7fe700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fb21806fcf0 0x7fb218072130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb218072800 con 0x7fb21806fcf0 2024-09-13T16:25:35.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb2280860a0 con 0x7fb23806e610 2024-09-13T16:25:35.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb22ffff700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb218066730 0x7fb218068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fb23759e700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fb21806fcf0 0x7fb218072130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7f2e4ffff700 1 -- 172.21.15.119:0/1303298520 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2e5801a400 con 0x7f2e6806d140 2024-09-13T16:25:35.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7f2e4ffff700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e54066730 0x7f2e54068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7f2e4ffff700 1 -- 172.21.15.119:0/1303298520 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2e58083830 con 0x7f2e6806d140 2024-09-13T16:25:35.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7f2e66ffd700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e54066730 0x7f2e54068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fc1517fa700 1 -- 172.21.15.119:0/1309767734 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc14c0127d0 con 0x7fc1540cdf80 2024-09-13T16:25:35.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7fc1517fa700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc134066800 0x7fc134068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7f2e66ffd700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e54066730 0x7f2e54068bf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e680cadb0 tx=0x7f2e5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7eff30013b00 con 0x7eff3406da70 2024-09-13T16:25:35.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7eff2affd700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7eff18066730 0x7eff18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7eff30084900 con 0x7eff3406da70 2024-09-13T16:25:35.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7eff39988700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7eff18066730 0x7eff18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7eff2affd700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7eff1806fd70 0x7eff180721b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7eff18072880 con 0x7eff1806fd70 2024-09-13T16:25:35.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=25) v2 ==== 24+0+0 (secure 0 0 0) 0x7eff30084cb0 con 0x7eff3406da70 2024-09-13T16:25:35.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7fc1517fa700 1 -- 172.21.15.119:0/1309767734 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc14c086530 con 0x7fc1540cdf80 2024-09-13T16:25:35.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7fc158ac1700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc134066800 0x7fc134068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7eff39988700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7eff18066730 0x7eff18068bf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7eff24007fd0 tx=0x7eff24007c80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.637+0000 7fc158ac1700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc134066800 0x7fc134068cc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc1440010e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.633+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f10900726c0 con 0x7f109006fbb0 2024-09-13T16:25:35.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7fc159ac3700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7fc15420bb00 0x7fc1540cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7eff3a98a700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7eff1806fd70 0x7eff180721b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc154070c10 con 0x7fc15420bb00 2024-09-13T16:25:35.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7f2e6e201700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f2e68202670 0x7f2e68202a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2e680b2110 con 0x7f2e68202670 2024-09-13T16:25:35.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7fc1592c2700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7fc15420bb00 0x7fc1540cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7fb23759e700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fb21806fcf0 0x7fb218072130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7fb22ffff700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb218066730 0x7fb218068bf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb2380c8240 tx=0x7fb220007950 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7eff3a98a700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7eff1806fd70 0x7eff180721b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.641+0000 7f2e67fff700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f2e68202670 0x7f2e68202a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.645+0000 7fc1592c2700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7fc15420bb00 0x7fc1540cd2f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.645+0000 7f2e67fff700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f2e68202670 0x7f2e68202a50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.645+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7eff18072880 con 0x7eff1806fd70 2024-09-13T16:25:35.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.645+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fb218072800 con 0x7fb21806fcf0 2024-09-13T16:25:35.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.645+0000 7fc1517fa700 1 -- 172.21.15.119:0/1309767734 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fc154070c10 con 0x7fc15420bb00 2024-09-13T16:25:35.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.645+0000 7f2e4ffff700 1 -- 172.21.15.119:0/1303298520 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f2e6000dc40 con 0x7f2e68202670 2024-09-13T16:25:35.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f06740b1ee0 con 0x7f066006fcf0 2024-09-13T16:25:35.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f02d8002d60 con 0x7f02d406fdc0 2024-09-13T16:25:35.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f02ebfff700 1 -- 172.21.15.119:0/2355019760 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f02d8002d60 con 0x7f02d406fdc0 2024-09-13T16:25:35.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f0670ff9700 1 -- 172.21.15.119:0/3165677184 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f066c00f040 con 0x7f066006fcf0 2024-09-13T16:25:35.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f066006fcf0 msgr2=0x7f0660072130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f06797ab700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f066006fcf0 0x7f0660072130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0660066730 msgr2=0x7f0660068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f06797ab700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0660066730 0x7f0660068bf0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f0668001c90 tx=0x7f0668009040 comp rx=0 tx=0).stop 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0674158b30 msgr2=0x7f06740aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f06797ab700 1 --2- 172.21.15.119:0/3165677184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0674158b30 0x7f06740aefe0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f0664009f70 tx=0x7f0664004770 comp rx=0 tx=0).stop 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.653+0000 7f0678faa700 1 -- 172.21.15.119:0/3165677184 reap_dead start 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 shutdown_connections 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 >> 172.21.15.119:0/3165677184 conn(0x7f067406a350 msgr2=0x7f067406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 shutdown_connections 2024-09-13T16:25:35.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f06797ab700 1 -- 172.21.15.119:0/3165677184 wait complete. 2024-09-13T16:25:35.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f02d406fdc0 msgr2=0x7f02d4072200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f02d406fdc0 0x7f02d4072200 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02d4066780 msgr2=0x7f02d4068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02d4066780 0x7f02d4068c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f02f413c820 tx=0x7f02ec007a60 comp rx=0 tx=0).stop 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f406dda0 msgr2=0x7f02f41ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 --2- 172.21.15.119:0/2355019760 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02f406dda0 0x7f02f41ff430 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f02dc004000 tx=0x7f02dc004770 comp rx=0 tx=0).stop 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f3fff700 1 -- 172.21.15.119:0/2355019760 reap_dead start 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 shutdown_connections 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 >> 172.21.15.119:0/2355019760 conn(0x7f02f40b04f0 msgr2=0x7f02f40cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 shutdown_connections 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.657+0000 7f02f8f33700 1 -- 172.21.15.119:0/2355019760 wait complete. 2024-09-13T16:25:35.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.665+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbc341f26f0 con 0x7fbc1806fcd0 2024-09-13T16:25:35.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.665+0000 7fbc29ffb700 1 -- 172.21.15.119:0/2283078566 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbc341f26f0 con 0x7fbc1806fcd0 2024-09-13T16:25:35.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f10a406d380 con 0x7f109006fbb0 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7fbc1806fcd0 msgr2=0x7fbc18072110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7fbc1806fcd0 0x7fbc18072110 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc18066800 msgr2=0x7fbc18068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc18066800 0x7fbc18068cc0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbc340c9270 tx=0x7fbc300060e0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34153100 msgr2=0x7fbc34155840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 --2- 172.21.15.119:0/2283078566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc34153100 0x7fbc34155840 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c009d90 tx=0x7fbc2c00c3a0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc39913700 1 -- 172.21.15.119:0/2283078566 reap_dead start 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 shutdown_connections 2024-09-13T16:25:35.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 >> 172.21.15.119:0/2283078566 conn(0x7fbc340b04f0 msgr2=0x7fbc3406c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 shutdown_connections 2024-09-13T16:25:35.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7fbc3a114700 1 -- 172.21.15.119:0/2283078566 wait complete. 2024-09-13T16:25:35.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.669+0000 7f10a0ff9700 1 -- 172.21.15.119:0/4235149457 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f109c004410 con 0x7f109006fbb0 2024-09-13T16:25:35.717 INFO:teuthology.orchestra.run.smithi119.stdout:60129542153 2024-09-13T16:25:35.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T16:25:35.718 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:35.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T16:25:35.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb2380cd8d0 con 0x7fb21806fcf0 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7f109006fbb0 msgr2=0x7f1090071ff0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7f109006fbb0 0x7f1090071ff0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1090066760 msgr2=0x7f1090068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1090066760 0x7f1090068c20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f10a40cb2e0 tx=0x7f1098006be0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10a40b2200 msgr2=0x7f10a414e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 --2- 172.21.15.119:0/4235149457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10a40b2200 0x7f10a414e780 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1094009c60 tx=0x7f1094004770 comp rx=0 tx=0).stop 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a8aea700 1 -- 172.21.15.119:0/4235149457 reap_dead start 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 shutdown_connections 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 >> 172.21.15.119:0/4235149457 conn(0x7f10a406a350 msgr2=0x7f10a41606d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 shutdown_connections 2024-09-13T16:25:35.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f10a92eb700 1 -- 172.21.15.119:0/4235149457 wait complete. 2024-09-13T16:25:35.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb22f7fe700 1 -- 172.21.15.119:0/2332721683 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fb230002030 con 0x7fb21806fcf0 2024-09-13T16:25:35.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7eff340b1f10 con 0x7eff1806fd70 2024-09-13T16:25:35.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2e68142820 con 0x7f2e68202670 2024-09-13T16:25:35.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fb21806fcf0 msgr2=0x7fb218072130 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23c953700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fb21806fcf0 0x7fb218072130 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb218066730 msgr2=0x7fb218068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23c953700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb218066730 0x7fb218068bf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb2380c8240 tx=0x7fb220007950 comp rx=0 tx=0).stop 2024-09-13T16:25:35.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb23806e610 msgr2=0x7fb2381511d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23c953700 1 --2- 172.21.15.119:0/2332721683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb23806e610 0x7fb2381511d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb228015000 tx=0x7fb228004770 comp rx=0 tx=0).stop 2024-09-13T16:25:35.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fb23759e700 1 -- 172.21.15.119:0/2332721683 reap_dead start 2024-09-13T16:25:35.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff2affd700 1 -- 172.21.15.119:0/872056170 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7eff2c003d60 con 0x7eff1806fd70 2024-09-13T16:25:35.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc15420b7d0 con 0x7fc15420bb00 2024-09-13T16:25:35.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e4ffff700 1 -- 172.21.15.119:0/1303298520 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2e6000ddb0 con 0x7f2e68202670 2024-09-13T16:25:35.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7eff1806fd70 msgr2=0x7eff180721b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7eff1806fd70 0x7eff180721b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7eff18066730 msgr2=0x7eff18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7eff18066730 0x7eff18068bf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7eff24007fd0 tx=0x7eff24007c80 comp rx=0 tx=0).stop 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff3406da70 msgr2=0x7eff34158390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 --2- 172.21.15.119:0/872056170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7eff3406da70 0x7eff34158390 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff3000e810 tx=0x7eff3000eb20 comp rx=0 tx=0).stop 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3a98a700 1 -- 172.21.15.119:0/872056170 reap_dead start 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 shutdown_connections 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 >> 172.21.15.119:0/872056170 conn(0x7eff3406a350 msgr2=0x7eff340ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 shutdown_connections 2024-09-13T16:25:35.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7eff3b18b700 1 -- 172.21.15.119:0/872056170 wait complete. 2024-09-13T16:25:35.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f2e68202670 msgr2=0x7f2e68202a50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f2e68202670 0x7f2e68202a50 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e54066730 msgr2=0x7f2e54068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e54066730 0x7f2e54068bf0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2e680cadb0 tx=0x7f2e5c009040 comp rx=0 tx=0).stop 2024-09-13T16:25:35.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 msgr2=0x7f2e680accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 --2- 172.21.15.119:0/1303298520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e6806d140 0x7f2e680accb0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2e5800b520 tx=0x7f2e5800b8e0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e67fff700 1 -- 172.21.15.119:0/1303298520 reap_dead start 2024-09-13T16:25:35.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 shutdown_connections 2024-09-13T16:25:35.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 >> 172.21.15.119:0/1303298520 conn(0x7f2e6806a350 msgr2=0x7f2e681585f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 shutdown_connections 2024-09-13T16:25:35.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7f2e6e201700 1 -- 172.21.15.119:0/1303298520 wait complete. 2024-09-13T16:25:35.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.673+0000 7fc1517fa700 1 -- 172.21.15.119:0/1309767734 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc15420b7d0 con 0x7fc15420bb00 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7fc15420bb00 msgr2=0x7fc1540cd2f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7fc15420bb00 0x7fc1540cd2f0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc134066800 msgr2=0x7fc134068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc134066800 0x7fc134068cc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fc144000c00 tx=0x7fc1440010e0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1540cdf80 msgr2=0x7fc15415caf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 --2- 172.21.15.119:0/1309767734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc1540cdf80 0x7fc15415caf0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fc1540b1db0 tx=0x7fc14c004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc1592c2700 1 -- 172.21.15.119:0/1309767734 reap_dead start 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 shutdown_connections 2024-09-13T16:25:35.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 >> 172.21.15.119:0/1309767734 conn(0x7fc1540b04f0 msgr2=0x7fc1540caa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 shutdown_connections 2024-09-13T16:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 >> 172.21.15.119:0/2332721683 conn(0x7fb2380ae4b0 msgr2=0x7fb2380cae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 shutdown_connections 2024-09-13T16:25:35.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fc159ac3700 1 -- 172.21.15.119:0/1309767734 wait complete. 2024-09-13T16:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 shutdown_connections 2024-09-13T16:25:35.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.677+0000 7fb23c953700 1 -- 172.21.15.119:0/2332721683 wait complete. 2024-09-13T16:25:35.730 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:35.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T16:25:35.732 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:35.732 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T16:25:35.733 INFO:teuthology.orchestra.run.smithi119.stdout:60129542153 2024-09-13T16:25:35.733 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T16:25:35.734 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:35.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T16:25:35.735 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:35.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T16:25:35.736 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:35.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T16:25:35.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e967700 1 Processor -- start 2024-09-13T16:25:35.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e967700 1 -- start start 2024-09-13T16:25:35.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6431fa4700 1 Processor -- start 2024-09-13T16:25:35.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6431fa4700 1 -- start start 2024-09-13T16:25:35.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e967700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4e080cd8d0 0x7f4e0806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e967700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4e080b2200 0x7f4e080b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e967700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080705d0 con 0x7f4e080b17f0 2024-09-13T16:25:35.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e967700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e080cbf10 con 0x7f4e080cd8d0 2024-09-13T16:25:35.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e967700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e080b2bb0 con 0x7f4e080b2200 2024-09-13T16:25:35.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4e080cd8d0 0x7f4e0806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4e080cd8d0 0x7f4e0806cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52142/0 (socket says 172.21.15.119:52142) 2024-09-13T16:25:35.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 -- 172.21.15.119:0/3947365071 learned_addr learned my addr 172.21.15.119:0/3947365071 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6431fa4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c160c60 0x7f642c0b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0e166700 1 --2- 172.21.15.119:0/3947365071 >> v2:172.21.15.145:3300/0 conn(0x7f4e080b2200 0x7f4e080b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6431fa4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c164af0 con 0x7f642c06cbd0 2024-09-13T16:25:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6431fa4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c0ce220 con 0x7f642c0b17f0 2024-09-13T16:25:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6431fa4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c06d930 con 0x7f642c160c60 2024-09-13T16:25:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f642bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f642c06cbd0 0x7f642c164f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33702/0 (socket says 172.21.15.119:33702) 2024-09-13T16:25:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f642bfff700 1 -- 172.21.15.119:0/1059463971 learned_addr learned my addr 172.21.15.119:0/1059463971 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0c963700 1 -- 172.21.15.119:0/3947365071 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4290640901 0 0) 0x7f4e080705d0 con 0x7f4e080b17f0 2024-09-13T16:25:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 -- 172.21.15.119:0/3947365071 >> v2:172.21.15.145:3300/0 conn(0x7f4e080b2200 msgr2=0x7f4e080b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 --2- 172.21.15.119:0/3947365071 >> v2:172.21.15.145:3300/0 conn(0x7f4e080b2200 0x7f4e080b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 -- 172.21.15.119:0/3947365071 >> v1:172.21.15.145:6789/0 conn(0x7f4e080b17f0 legacy=0x7f4e080cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 -- 172.21.15.119:0/3947365071 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0806d0d0 con 0x7f4e080cd8d0 2024-09-13T16:25:35.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6430fa2700 1 --2- 172.21.15.119:0/1059463971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c160c60 0x7f642c0b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0d965700 1 --2- 172.21.15.119:0/3947365071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e0806cb90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4e080b3a60 tx=0x7f4e04009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0de25942d9b459 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0c963700 1 -- 172.21.15.119:0/3947365071 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04011830 con 0x7f4e080cd8d0 2024-09-13T16:25:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6430fa2700 1 -- 172.21.15.119:0/1059463971 >> v1:172.21.15.145:6789/0 conn(0x7f642c0b17f0 legacy=0x7f642c070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6430fa2700 1 -- 172.21.15.119:0/1059463971 >> v1:172.21.15.119:6789/0 conn(0x7f642c06cbd0 legacy=0x7f642c164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f642b7fe700 1 -- 172.21.15.119:0/1059463971 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530939106 0 0) 0x7f642c0ce220 con 0x7f642c0b17f0 2024-09-13T16:25:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f6430fa2700 1 -- 172.21.15.119:0/1059463971 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c06d090 con 0x7f642c160c60 2024-09-13T16:25:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0c963700 1 -- 172.21.15.119:0/3947365071 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e04011990 con 0x7f4e080cd8d0 2024-09-13T16:25:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.921+0000 7f4e0c963700 1 -- 172.21.15.119:0/3947365071 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04011d40 con 0x7f4e080cd8d0 2024-09-13T16:25:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f6430fa2700 1 --2- 172.21.15.119:0/1059463971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c160c60 0x7f642c0b38a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f641c009880 tx=0x7f641c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cdd0097cf3042c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f642b7fe700 1 -- 172.21.15.119:0/1059463971 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c011830 con 0x7f642c160c60 2024-09-13T16:25:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f642b7fe700 1 -- 172.21.15.119:0/1059463971 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f641c011990 con 0x7f642c160c60 2024-09-13T16:25:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f642b7fe700 1 -- 172.21.15.119:0/1059463971 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c011d20 con 0x7f642c160c60 2024-09-13T16:25:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4dc87a700 1 Processor -- start 2024-09-13T16:25:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4dc87a700 1 -- start start 2024-09-13T16:25:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f6431fa4700 1 -- 172.21.15.119:0/1059463971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c160c60 msgr2=0x7f642c0b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f6431fa4700 1 --2- 172.21.15.119:0/1059463971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c160c60 0x7f642c0b38a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f641c009880 tx=0x7f641c009b90 comp rx=0 tx=0).stop 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 -- 172.21.15.119:0/3947365071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 msgr2=0x7f4e0806cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 --2- 172.21.15.119:0/3947365071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e0806cb90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f4e080b3a60 tx=0x7f4e04009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 -- 172.21.15.119:0/3947365071 shutdown_connections 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 --2- 172.21.15.119:0/3947365071 >> v2:172.21.15.145:3300/0 conn(0x7f4e080b2200 0x7f4e080b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 --2- 172.21.15.119:0/3947365071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e0806cb90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 -- 172.21.15.119:0/3947365071 >> 172.21.15.119:0/3947365071 conn(0x7f4e0806a350 msgr2=0x7f4e0806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 -- 172.21.15.119:0/3947365071 shutdown_connections 2024-09-13T16:25:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 -- 172.21.15.119:0/3947365071 wait complete. 2024-09-13T16:25:35.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f6431fa4700 1 -- 172.21.15.119:0/1059463971 shutdown_connections 2024-09-13T16:25:35.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f6431fa4700 1 --2- 172.21.15.119:0/1059463971 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c160c60 0x7f642c0b38a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f6431fa4700 1 -- 172.21.15.119:0/1059463971 >> 172.21.15.119:0/1059463971 conn(0x7f642c0b04f0 msgr2=0x7f642c0b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4dc87a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 0x7ff4d80b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8e9d9700 1 Processor -- start 2024-09-13T16:25:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4dc87a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff4d806cbd0 0x7ff4d8164f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d7fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff4d81654c0 0x7ff4d80b17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33710/0 (socket says 172.21.15.119:33710) 2024-09-13T16:25:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d7fff700 1 -- 172.21.15.119:0/3576027598 learned_addr learned my addr 172.21.15.119:0/3576027598 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4d8164af0 con 0x7ff4d81654c0 2024-09-13T16:25:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d77fe700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 0x7ff4d80b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4d806d930 con 0x7ff4d806cbd0 2024-09-13T16:25:35.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8164850 con 0x7ff4d8160c60 2024-09-13T16:25:35.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 Processor -- start 2024-09-13T16:25:35.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 -- start start 2024-09-13T16:25:35.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610725721 0 0) 0x7ff4d8164af0 con 0x7ff4d81654c0 2024-09-13T16:25:35.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4c00035d0 con 0x7ff4d81654c0 2024-09-13T16:25:35.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 431482076 0 0) 0x7ff4c00035d0 con 0x7ff4d81654c0 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 msgr2=0x7ff4d80b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 0x7ff4d80b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 >> v2:172.21.15.119:3300/0 conn(0x7ff4d806cbd0 msgr2=0x7ff4d8164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 --2- 172.21.15.119:0/3576027598 >> v2:172.21.15.119:3300/0 conn(0x7ff4d806cbd0 0x7ff4d8164f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d80ce220 con 0x7ff4d81654c0 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7ff4d00019f0 con 0x7ff4d81654c0 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 >> v1:172.21.15.119:6789/0 conn(0x7ff4d81654c0 legacy=0x7ff4d80b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c0003a20 0x7ff4c0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4c0004870 0x7ff4c0004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 0x7ff4c0005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8164850 con 0x7ff4c0004870 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c000bae0 con 0x7ff4c0003a20 2024-09-13T16:25:35.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c000bc50 con 0x7ff4d8160c60 2024-09-13T16:25:35.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8e9d9700 1 -- start start 2024-09-13T16:25:35.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8e9d9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88152d20 0x7f9e88159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d77fe700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c0003a20 0x7ff4c0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7ff4d7fff700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 0x7ff4c0005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f6431fa4700 1 -- 172.21.15.119:0/1059463971 shutdown_connections 2024-09-13T16:25:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e080b17f0 0x7f4e080aacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8e9d9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e880b2000 con 0x7f9e880b0fe0 2024-09-13T16:25:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8e9d9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e880b22e0 con 0x7f9e88153100 2024-09-13T16:25:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8e9d9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e880b28a0 con 0x7f9e88152d20 2024-09-13T16:25:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f5f48ded700 1 Processor -- start 2024-09-13T16:25:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e080b2200 0x7f4e080ad830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8d1d6700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9e880b0fe0 0x7f9e880b17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33712/0 (socket says 172.21.15.119:33712) 2024-09-13T16:25:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f9e8d1d6700 1 -- 172.21.15.119:0/2920246172 learned_addr learned my addr 172.21.15.119:0/2920246172 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8d9d7700 1 --2- 172.21.15.119:0/2920246172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88152d20 0x7f9e88159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d6ffd700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4c0004870 0x7ff4c0004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f4e0e967700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e081f6e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0e967700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080ab2d0 con 0x7f4e080cd8d0 2024-09-13T16:25:35.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0e967700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080ab5b0 con 0x7f4e080b17f0 2024-09-13T16:25:35.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0e967700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e080ab890 con 0x7f4e080b2200 2024-09-13T16:25:35.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.925+0000 7f5f48ded700 1 -- start start 2024-09-13T16:25:35.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e080b17f0 0x7f4e080aacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 -- 172.21.15.119:0/1059463971 wait complete. 2024-09-13T16:25:35.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e080b17f0 0x7f4e080aacf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40760/0 (socket says 172.21.15.119:40760) 2024-09-13T16:25:35.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 -- 172.21.15.119:0/59183036 learned_addr learned my addr 172.21.15.119:0/59183036 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d7fff700 1 -- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c0003a20 msgr2=0x7ff4c0003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d7fff700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4c0003a20 0x7ff4c0003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d7fff700 1 -- 172.21.15.119:0/3576027598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4c0004870 msgr2=0x7ff4c0004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d7fff700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4c0004870 0x7ff4c0004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d7fff700 1 -- 172.21.15.119:0/3576027598 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d80ce220 con 0x7ff4d8160c60 2024-09-13T16:25:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0e166700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e081f6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8c9d5700 1 -- 172.21.15.119:0/2920246172 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1482089013 0 0) 0x7f9e880b2000 con 0x7f9e880b0fe0 2024-09-13T16:25:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8c9d5700 1 -- 172.21.15.119:0/2920246172 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e740035d0 con 0x7f9e880b0fe0 2024-09-13T16:25:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8c9d5700 1 -- 172.21.15.119:0/2920246172 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642268896 0 0) 0x7f9e880b22e0 con 0x7f9e88153100 2024-09-13T16:25:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8d9d7700 1 -- 172.21.15.119:0/2920246172 >> v1:172.21.15.145:6789/0 conn(0x7f9e88153100 legacy=0x7f9e880ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8d9d7700 1 -- 172.21.15.119:0/2920246172 >> v1:172.21.15.119:6789/0 conn(0x7f9e880b0fe0 legacy=0x7f9e880b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8d9d7700 1 -- 172.21.15.119:0/2920246172 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e880b25c0 con 0x7f9e88152d20 2024-09-13T16:25:35.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d7fff700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 0x7ff4c0005230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000a8a0 tx=0x7ff4c00058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad2222f503c4389a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 -- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e080b2200 msgr2=0x7f4e080ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d164700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e080b2200 0x7f4e080ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e080b2200 0x7f4e080ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 -- 172.21.15.119:0/59183036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 msgr2=0x7f4e081f6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e081f6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 -- 172.21.15.119:0/59183036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04009570 con 0x7f4e080b17f0 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0e166700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e081f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d164700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e080b2200 0x7f4e080ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f48ded700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5f4415a630 0x7f5f4406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f48ded700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4406dd40 con 0x7f5f4415a250 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f48ded700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f4406d430 con 0x7f5f44070c10 2024-09-13T16:25:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f48ded700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f4406d6f0 con 0x7f5f4415a630 2024-09-13T16:25:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d00072d0 con 0x7ff4d8160c60 2024-09-13T16:25:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4d0007450 con 0x7ff4d8160c60 2024-09-13T16:25:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d67fc700 1 -- 172.21.15.119:0/3576027598 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4d0007800 con 0x7ff4d8160c60 2024-09-13T16:25:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8d9d7700 1 --2- 172.21.15.119:0/2920246172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88152d20 0x7f9e88159a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009880 tx=0x7f9e84009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee94e7d4a9b087c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f437fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5f4415a250 0x7f5f440b3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33728/0 (socket says 172.21.15.119:33728) 2024-09-13T16:25:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f437fe700 1 -- 172.21.15.119:0/578502722 learned_addr learned my addr 172.21.15.119:0/578502722 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f43fff700 1 --1- 172.21.15.119:0/578502722 >> v1:172.21.15.145:6789/0 conn(0x7f5f44070c10 0x7f5f440b17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:53210/0 (socket says 172.21.15.119:53210) 2024-09-13T16:25:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f42ffd700 1 --2- 172.21.15.119:0/578502722 >> v2:172.21.15.119:3300/0 conn(0x7f5f4415a630 0x7f5f4406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0d965700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e080b17f0 0x7f4e080aacf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000c00 tx=0x7f4e04004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8c9d5700 1 -- 172.21.15.119:0/2920246172 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84004230 con 0x7f9e88152d20 2024-09-13T16:25:35.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f42ffd700 1 -- 172.21.15.119:0/578502722 >> v1:172.21.15.145:6789/0 conn(0x7f5f44070c10 legacy=0x7f5f440b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f42ffd700 1 -- 172.21.15.119:0/578502722 >> v1:172.21.15.119:6789/0 conn(0x7f5f4415a250 legacy=0x7f5f440b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 Processor -- start 2024-09-13T16:25:35.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f42ffd700 1 -- 172.21.15.119:0/578502722 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f4406d9d0 con 0x7f5f4415a630 2024-09-13T16:25:35.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8c9d5700 1 -- 172.21.15.119:0/2920246172 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e84004390 con 0x7f9e88152d20 2024-09-13T16:25:35.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8c9d5700 1 -- 172.21.15.119:0/2920246172 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84010890 con 0x7f9e88152d20 2024-09-13T16:25:35.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f5f42ffd700 1 --2- 172.21.15.119:0/578502722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a630 0x7f5f4406e1d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009850 tx=0x7f5f34009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43779417b15a2118 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f972a25c700 1 Processor -- start 2024-09-13T16:25:35.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f972a25c700 1 -- start start 2024-09-13T16:25:35.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 msgr2=0x7ff4c0005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4dc87a700 1 --2- 172.21.15.119:0/3576027598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d8160c60 0x7ff4c0005230 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4d000a8a0 tx=0x7ff4c00058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4d7fff700 1 -- 172.21.15.119:0/3576027598 reap_dead start 2024-09-13T16:25:35.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4dfe7fc700 1 -- 172.21.15.119:0/59183036 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04019900 con 0x7f4e080b17f0 2024-09-13T16:25:35.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4dfe7fc700 1 -- 172.21.15.119:0/59183036 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0401e070 con 0x7f4e080b17f0 2024-09-13T16:25:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e080abb70 con 0x7f4e080b17f0 2024-09-13T16:25:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 shutdown_connections 2024-09-13T16:25:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 >> 172.21.15.119:0/3576027598 conn(0x7ff4d80b04f0 msgr2=0x7ff4d80b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4dfe7fc700 1 -- 172.21.15.119:0/59183036 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e04019ca0 con 0x7f4e080b17f0 2024-09-13T16:25:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e080add70 con 0x7f4e080b17f0 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 shutdown_connections 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 -- start start 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c06cbd0 0x7f642c1f9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f642c0b17f0 0x7f642c0c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f642c160c60 0x7f642c0a9760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0c3dc0 con 0x7f642c0b17f0 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0c3800 con 0x7f642c160c60 2024-09-13T16:25:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f6431fa4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c0c3ae0 con 0x7f642c06cbd0 2024-09-13T16:25:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/2920246172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88152d20 msgr2=0x7f9e88159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/2920246172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88152d20 0x7f9e88159a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e84009880 tx=0x7f9e84009b90 comp rx=0 tx=0).stop 2024-09-13T16:25:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f642bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f642c0b17f0 0x7f642c0c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f642bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f642c0b17f0 0x7f642c0c4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52198/0 (socket says 172.21.15.119:52198) 2024-09-13T16:25:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f642bfff700 1 -- 172.21.15.119:0/4020620000 learned_addr learned my addr 172.21.15.119:0/4020620000 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/2920246172 shutdown_connections 2024-09-13T16:25:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/2920246172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88152d20 0x7f9e88159a40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/2920246172 >> 172.21.15.119:0/2920246172 conn(0x7f9e880b04f0 msgr2=0x7f9e880b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f64317a3700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f642c160c60 0x7f642c0a9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/2920246172 shutdown_connections 2024-09-13T16:25:35.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f427fc700 1 -- 172.21.15.119:0/578502722 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34011830 con 0x7f5f4415a630 2024-09-13T16:25:35.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f427fc700 1 -- 172.21.15.119:0/578502722 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f34011990 con 0x7f5f4415a630 2024-09-13T16:25:35.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f427fc700 1 -- 172.21.15.119:0/578502722 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34011d20 con 0x7f5f4415a630 2024-09-13T16:25:35.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7ff4dc87a700 1 -- 172.21.15.119:0/3576027598 wait complete. 2024-09-13T16:25:35.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 Processor -- start 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/2920246172 wait complete. 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f642bfff700 1 -- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c06cbd0 msgr2=0x7f642c1f9e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f642bfff700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c06cbd0 0x7f642c1f9e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f642bfff700 1 -- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f642c160c60 msgr2=0x7f642c0a9760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f642bfff700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f642c160c60 0x7f642c0a9760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f642bfff700 1 -- 172.21.15.119:0/4020620000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f641c0094b0 con 0x7f642c0b17f0 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f642bfff700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f642c0b17f0 0x7f642c0c4150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f642000c520 tx=0x7f642000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f64297fa700 1 -- 172.21.15.119:0/4020620000 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420005d90 con 0x7f642c0b17f0 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f64297fa700 1 -- 172.21.15.119:0/4020620000 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64200070d0 con 0x7f642c0b17f0 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f6431fa4700 1 -- 172.21.15.119:0/4020620000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c0a8f60 con 0x7f642c0b17f0 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f6431fa4700 1 -- 172.21.15.119:0/4020620000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c1fa3b0 con 0x7f642c0b17f0 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f64297fa700 1 -- 172.21.15.119:0/4020620000 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6420007440 con 0x7f642c0b17f0 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f48ded700 1 -- 172.21.15.119:0/578502722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a630 msgr2=0x7f5f4406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f48ded700 1 --2- 172.21.15.119:0/578502722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a630 0x7f5f4406e1d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5f34009850 tx=0x7f5f34009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f48ded700 1 -- 172.21.15.119:0/578502722 shutdown_connections 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f48ded700 1 --2- 172.21.15.119:0/578502722 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a630 0x7f5f4406e1d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f48ded700 1 -- 172.21.15.119:0/578502722 >> 172.21.15.119:0/578502722 conn(0x7f5f440b04f0 msgr2=0x7f5f440b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 -- start start 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4d8160c60 0x7ff4d81fa6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d81f6000 0x7ff4d81f72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4d80708b0 0x7ff4d8070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d8071270 con 0x7ff4d8160c60 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d81f6e20 con 0x7ff4d80708b0 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4dc87a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4d81fabe0 con 0x7ff4d81f6000 2024-09-13T16:25:35.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f48ded700 1 -- 172.21.15.119:0/578502722 shutdown_connections 2024-09-13T16:25:35.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4d8160c60 0x7ff4d81fa6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4d8160c60 0x7ff4d81fa6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52220/0 (socket says 172.21.15.119:52220) 2024-09-13T16:25:35.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 -- 172.21.15.119:0/1833758009 learned_addr learned my addr 172.21.15.119:0/1833758009 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d6ffd700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d81f6000 0x7ff4d81f72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f972a25c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406f0c0 0x7f972406cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f972a25c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f972406d460 0x7f972406a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f972a25c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972415acc0 con 0x7f9724153bf0 2024-09-13T16:25:35.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f6431fa4700 1 -- 172.21.15.119:0/4020620000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640c005310 con 0x7f642c0b17f0 2024-09-13T16:25:35.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d7fff700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4d80708b0 0x7ff4d8070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 -- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d81f6000 msgr2=0x7ff4d81f72b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d81f6000 0x7ff4d81f72b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 -- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4d80708b0 msgr2=0x7ff4d8070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4d80708b0 0x7ff4d8070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f5f48ded700 1 -- 172.21.15.119:0/578502722 wait complete. 2024-09-13T16:25:35.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 -- 172.21.15.119:0/1833758009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4d81591e0 con 0x7ff4d8160c60 2024-09-13T16:25:35.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d7fff700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4d80708b0 0x7ff4d8070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:35.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f972a25c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9724153fd0 con 0x7f972406d460 2024-09-13T16:25:35.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.929+0000 7f972a25c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972406dc90 con 0x7f972406f0c0 2024-09-13T16:25:35.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f972406d460 0x7f972406a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f4dfe7fc700 1 -- 172.21.15.119:0/59183036 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4e0402d450 con 0x7f4e080b17f0 2024-09-13T16:25:35.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f972406d460 0x7f972406a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52214/0 (socket says 172.21.15.119:52214) 2024-09-13T16:25:35.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 -- 172.21.15.119:0/1298935279 learned_addr learned my addr 172.21.15.119:0/1298935279 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7ff4d77fe700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4d8160c60 0x7ff4d81fa6a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8011a80 tx=0x7ff4c8011d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9728a59700 1 --2- 172.21.15.119:0/1298935279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406f0c0 0x7f972406cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df0005310 con 0x7f4e080b17f0 2024-09-13T16:25:35.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f8458de9700 1 Processor -- start 2024-09-13T16:25:35.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 -- 172.21.15.119:0/1298935279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406f0c0 msgr2=0x7f972406cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f64297fa700 1 -- 172.21.15.119:0/4020620000 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f64200075a0 con 0x7f642c0b17f0 2024-09-13T16:25:35.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 --2- 172.21.15.119:0/1298935279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406f0c0 0x7f972406cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 -- 172.21.15.119:0/1298935279 >> v1:172.21.15.119:6789/0 conn(0x7f9724153bf0 legacy=0x7f972406e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:35.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 -- 172.21.15.119:0/1298935279 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972406df70 con 0x7f972406d460 2024-09-13T16:25:35.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9729a5b700 1 --2- 172.21.15.119:0/1298935279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406d460 0x7f972406a350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9720009910 tx=0x7f9720009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a063826948821f0f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f4dfe7fc700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4dec066800 0x7f4dec068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f971bfff700 1 -- 172.21.15.119:0/1298935279 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9720004230 con 0x7f972406d460 2024-09-13T16:25:35.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f971bfff700 1 -- 172.21.15.119:0/1298935279 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9720004390 con 0x7f972406d460 2024-09-13T16:25:35.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f971bfff700 1 -- 172.21.15.119:0/1298935279 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97200036c0 con 0x7f972406d460 2024-09-13T16:25:35.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f4dfe7fc700 1 -- 172.21.15.119:0/59183036 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4e04085360 con 0x7f4e080b17f0 2024-09-13T16:25:35.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f64297fa700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6418066800 0x7f6418068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f64297fa700 1 -- 172.21.15.119:0/4020620000 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f64200837f0 con 0x7f642c0b17f0 2024-09-13T16:25:35.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f972a25c700 1 -- 172.21.15.119:0/1298935279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406d460 msgr2=0x7f972406a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f972a25c700 1 --2- 172.21.15.119:0/1298935279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406d460 0x7f972406a350 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f9720009910 tx=0x7f9720009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:35.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f6430fa2700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6418066800 0x7f6418068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4e0d164700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4dec066800 0x7f4dec068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- 172.21.15.119:0/1298935279 shutdown_connections 2024-09-13T16:25:35.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 --2- 172.21.15.119:0/1298935279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406d460 0x7f972406a350 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 --2- 172.21.15.119:0/1298935279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406f0c0 0x7f972406cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- 172.21.15.119:0/1298935279 >> 172.21.15.119:0/1298935279 conn(0x7f97240b15e0 msgr2=0x7f97240b19f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 Processor -- start 2024-09-13T16:25:35.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9e8e9d9700 1 Processor -- start 2024-09-13T16:25:35.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f9e8e9d9700 1 -- start start 2024-09-13T16:25:35.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f6430fa2700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6418066800 0x7f6418068cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f642c04e960 tx=0x7f641c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.933+0000 7f8458de9700 1 -- start start 2024-09-13T16:25:35.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f8458de9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8454152d20 0x7f8454159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f8458de9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84540b2000 con 0x7f8454070c10 2024-09-13T16:25:35.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f8458de9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84540b22e0 con 0x7f8454153100 2024-09-13T16:25:35.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f8458de9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84540b28a0 con 0x7f8454152d20 2024-09-13T16:25:35.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8e9d9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e880b0fe0 0x7f9e8814b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8e9d9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e88152d20 0x7f9e881550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8e9d9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88153100 0x7f9e88155630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8e9d9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e880a87b0 con 0x7f9e88152d20 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8e9d9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8815d000 con 0x7f9e880b0fe0 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8e9d9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8815bf60 con 0x7f9e88153100 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e880b0fe0 0x7f9e8814b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e880b0fe0 0x7f9e8814b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40794/0 (socket says 172.21.15.119:40794) 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 -- 172.21.15.119:0/346090684 learned_addr learned my addr 172.21.15.119:0/346090684 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 -- start start 2024-09-13T16:25:35.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f44070c10 0x7f5f44147000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a250 0x7f5f44154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f4415a630 0x7f5f440ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f441528c0 con 0x7f5f4415a250 2024-09-13T16:25:35.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f44152000 con 0x7f5f4415a630 2024-09-13T16:25:35.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f48ded700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f44152270 con 0x7f5f44070c10 2024-09-13T16:25:35.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- 172.21.15.119:0/1298935279 shutdown_connections 2024-09-13T16:25:35.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- 172.21.15.119:0/1298935279 wait complete. 2024-09-13T16:25:35.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a250 0x7f5f44154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a250 0x7f5f44154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52222/0 (socket says 172.21.15.119:52222) 2024-09-13T16:25:35.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 -- 172.21.15.119:0/2663856819 learned_addr learned my addr 172.21.15.119:0/2663856819 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f43fff700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f4415a630 0x7f5f440ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f437fe700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f44070c10 0x7f5f44147000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 -- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f44070c10 msgr2=0x7f5f44147000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f44070c10 0x7f5f44147000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 -- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f4415a630 msgr2=0x7f5f440ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f4415a630 0x7f5f440ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 -- 172.21.15.119:0/2663856819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f340094b0 con 0x7f5f4415a250 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 -- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88153100 msgr2=0x7f9e88155630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88153100 0x7f9e88155630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 -- 172.21.15.119:0/346090684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e88152d20 msgr2=0x7f9e881550f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e88152d20 0x7f9e881550f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 -- 172.21.15.119:0/346090684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e840094b0 con 0x7f9e880b0fe0 2024-09-13T16:25:35.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f5f42ffd700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a250 0x7f5f44154930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5f34014040 tx=0x7f5f34004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9e8d9d7700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e880b0fe0 0x7f9e8814b7f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401a040 tx=0x7f9e84004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 Processor -- start 2024-09-13T16:25:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4e0d164700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4dec066800 0x7f4dec068cc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4e080ca9b0 tx=0x7f4df8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- start start 2024-09-13T16:25:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d66a6a700 1 Processor -- start 2024-09-13T16:25:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d66a6a700 1 -- start start 2024-09-13T16:25:35.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406d460 0x7f97241493d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406f0c0 0x7f972415bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9724153bf0 0x7f97241541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9724149910 con 0x7f972406f0c0 2024-09-13T16:25:35.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9724151d10 con 0x7f9724153bf0 2024-09-13T16:25:35.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972a25c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9724151ff0 con 0x7f972406d460 2024-09-13T16:25:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9728a59700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406f0c0 0x7f972415bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d66a6a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4d60159850 0x7f4d600cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9728a59700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406f0c0 0x7f972415bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52238/0 (socket says 172.21.15.119:52238) 2024-09-13T16:25:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9728a59700 1 -- 172.21.15.119:0/2526345267 learned_addr learned my addr 172.21.15.119:0/2526345267 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f972925a700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406d460 0x7f97241493d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f9729a5b700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9724153bf0 0x7f97241541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d66a6a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600ad9f0 con 0x7f4d600b3b00 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d66a6a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600cce20 con 0x7f4d600b2350 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d66a6a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600cd100 con 0x7f4d60159850 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d65007700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4d600b3b00 0x7f4d6006fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33750/0 (socket says 172.21.15.119:33750) 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d65007700 1 -- 172.21.15.119:0/3787204285 learned_addr learned my addr 172.21.15.119:0/3787204285 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.937+0000 7f4d64806700 1 --2- 172.21.15.119:0/3787204285 >> v2:172.21.15.119:3300/0 conn(0x7f4d60159850 0x7f4d600cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9728a59700 1 -- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406d460 msgr2=0x7f97241493d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9728a59700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406d460 0x7f97241493d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9728a59700 1 -- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9724153bf0 msgr2=0x7f97241541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9728a59700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9724153bf0 0x7f97241541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9728a59700 1 -- 172.21.15.119:0/2526345267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9720009570 con 0x7f972406f0c0 2024-09-13T16:25:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f8453fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8454070c10 0x7f84540ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33762/0 (socket says 172.21.15.119:33762) 2024-09-13T16:25:35.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f8453fff700 1 -- 172.21.15.119:0/2515958084 learned_addr learned my addr 172.21.15.119:0/2515958084 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:35.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84537fe700 1 --2- 172.21.15.119:0/2515958084 >> v2:172.21.15.119:3300/0 conn(0x7f8454152d20 0x7f8454159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7ff4bffff700 1 -- 172.21.15.119:0/1833758009 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c800f940 con 0x7ff4d8160c60 2024-09-13T16:25:35.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9728a59700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406f0c0 0x7f972415bec0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f971401f880 tx=0x7f971401fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4d8163480 con 0x7ff4d8160c60 2024-09-13T16:25:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4d81f67d0 con 0x7ff4d8160c60 2024-09-13T16:25:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f4d64806700 1 -- 172.21.15.119:0/3787204285 >> v1:172.21.15.145:6789/0 conn(0x7f4d600b2350 legacy=0x7f4d600b33f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84537fe700 1 -- 172.21.15.119:0/2515958084 >> v1:172.21.15.145:6789/0 conn(0x7f8454153100 legacy=0x7f84540b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84537fe700 1 -- 172.21.15.119:0/2515958084 >> v1:172.21.15.119:6789/0 conn(0x7f8454070c10 legacy=0x7f84540ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84537fe700 1 -- 172.21.15.119:0/2515958084 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84540b25c0 con 0x7f8454152d20 2024-09-13T16:25:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f4d64806700 1 -- 172.21.15.119:0/3787204285 >> v1:172.21.15.119:6789/0 conn(0x7f4d600b3b00 legacy=0x7f4d6006fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f4d64806700 1 -- 172.21.15.119:0/3787204285 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600cd3e0 con 0x7f4d60159850 2024-09-13T16:25:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f4d64806700 1 --2- 172.21.15.119:0/3787204285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60159850 0x7f4d600cdec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009850 tx=0x7f4d50009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4a4727c3e26a49 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84537fe700 1 --2- 172.21.15.119:0/2515958084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454152d20 0x7f8454159a40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8444009260 tx=0x7f8444009570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29c29ee92993d87c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4d80af270 con 0x7ff4d8160c60 2024-09-13T16:25:35.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7ff4bffff700 1 -- 172.21.15.119:0/1833758009 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c800faa0 con 0x7ff4d8160c60 2024-09-13T16:25:35.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7ff4bffff700 1 -- 172.21.15.119:0/1833758009 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c800fe10 con 0x7ff4d8160c60 2024-09-13T16:25:35.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f4d5f7fe700 1 -- 172.21.15.119:0/3787204285 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50011830 con 0x7f4d60159850 2024-09-13T16:25:35.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f5f2bfff700 1 -- 172.21.15.119:0/2663856819 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f340036c0 con 0x7f5f4415a250 2024-09-13T16:25:35.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f44152550 con 0x7f5f4415a250 2024-09-13T16:25:35.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f64297fa700 1 -- 172.21.15.119:0/4020620000 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6420050900 con 0x7f642c0b17f0 2024-09-13T16:25:35.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f44155380 con 0x7f5f4415a250 2024-09-13T16:25:35.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f5f2bfff700 1 -- 172.21.15.119:0/2663856819 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f3401e070 con 0x7f5f4415a250 2024-09-13T16:25:35.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f5f2bfff700 1 -- 172.21.15.119:0/2663856819 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f34003a20 con 0x7f5f4415a250 2024-09-13T16:25:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9719ffb700 1 -- 172.21.15.119:0/2526345267 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714024890 con 0x7f972406f0c0 2024-09-13T16:25:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f9719ffb700 1 -- 172.21.15.119:0/2526345267 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97140249f0 con 0x7f972406f0c0 2024-09-13T16:25:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97241522d0 con 0x7f972406f0c0 2024-09-13T16:25:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84527fc700 1 -- 172.21.15.119:0/2515958084 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844400c940 con 0x7f8454152d20 2024-09-13T16:25:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84527fc700 1 -- 172.21.15.119:0/2515958084 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f844400caa0 con 0x7f8454152d20 2024-09-13T16:25:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f84527fc700 1 -- 172.21.15.119:0/2515958084 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f844400ce30 con 0x7f8454152d20 2024-09-13T16:25:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f4d5f7fe700 1 -- 172.21.15.119:0/3787204285 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50011990 con 0x7f4d60159850 2024-09-13T16:25:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.941+0000 7f4d5f7fe700 1 -- 172.21.15.119:0/3787204285 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50011d00 con 0x7f4d60159850 2024-09-13T16:25:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3787204285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60159850 msgr2=0x7f4d600cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3787204285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60159850 0x7f4d600cdec0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009850 tx=0x7f4d50009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 -- 172.21.15.119:0/2515958084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454152d20 msgr2=0x7f8454159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 --2- 172.21.15.119:0/2515958084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454152d20 0x7f8454159a40 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8444009260 tx=0x7f8444009570 comp rx=0 tx=0).stop 2024-09-13T16:25:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f9719ffb700 1 -- 172.21.15.119:0/2526345267 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714024da0 con 0x7f972406f0c0 2024-09-13T16:25:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972415c910 con 0x7f972406f0c0 2024-09-13T16:25:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f5f2bfff700 1 -- 172.21.15.119:0/2663856819 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5f34003b80 con 0x7f5f4415a250 2024-09-13T16:25:35.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 -- 172.21.15.119:0/2515958084 shutdown_connections 2024-09-13T16:25:35.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 --2- 172.21.15.119:0/2515958084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454152d20 0x7f8454159a40 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 -- 172.21.15.119:0/2515958084 >> 172.21.15.119:0/2515958084 conn(0x7f84540b04f0 msgr2=0x7f84540b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 -- 172.21.15.119:0/2515958084 shutdown_connections 2024-09-13T16:25:35.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 -- 172.21.15.119:0/2515958084 wait complete. 2024-09-13T16:25:35.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f5f2bfff700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f30066760 0x7f5f30068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f5f437fe700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f30066760 0x7f5f30068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f9e7e7fc700 1 -- 172.21.15.119:0/346090684 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84006ea0 con 0x7f9e880b0fe0 2024-09-13T16:25:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f9e7e7fc700 1 -- 172.21.15.119:0/346090684 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e840038f0 con 0x7f9e880b0fe0 2024-09-13T16:25:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8815c240 con 0x7f9e880b0fe0 2024-09-13T16:25:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f9e7e7fc700 1 -- 172.21.15.119:0/346090684 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e84003aa0 con 0x7f9e880b0fe0 2024-09-13T16:25:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8815c800 con 0x7f9e880b0fe0 2024-09-13T16:25:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f5f437fe700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f30066760 0x7f5f30068c20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5f38005930 tx=0x7f5f38009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 Processor -- start 2024-09-13T16:25:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f8458de9700 1 -- start start 2024-09-13T16:25:35.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7ff4bffff700 1 -- 172.21.15.119:0/1833758009 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff4c801d400 con 0x7ff4d8160c60 2024-09-13T16:25:35.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7ff4bffff700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4c4066760 0x7ff4c4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7ff4bffff700 1 -- 172.21.15.119:0/1833758009 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff4c8084e20 con 0x7ff4d8160c60 2024-09-13T16:25:35.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7ff4d6ffd700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4c4066760 0x7ff4c4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7f4dfe7fc700 1 -- 172.21.15.119:0/59183036 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4e0405ae60 con 0x7f4e080b17f0 2024-09-13T16:25:35.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.945+0000 7ff4d6ffd700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4c4066760 0x7ff4c4068c20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff4d81606a0 tx=0x7ff4cc0090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3787204285 shutdown_connections 2024-09-13T16:25:35.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3787204285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d60159850 0x7f4d600cdec0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3787204285 >> 172.21.15.119:0/3787204285 conn(0x7f4d6006a350 msgr2=0x7f4d6006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:35.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e88159a20 con 0x7f9e880b0fe0 2024-09-13T16:25:35.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97240cc130 con 0x7f972406f0c0 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8458de9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454070c10 0x7f84541f35d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8458de9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8454152d20 0x7f8454157b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8458de9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8454153100 0x7f84541f4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8458de9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540c8ab0 con 0x7f8454070c10 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8458de9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84540c7f30 con 0x7f8454152d20 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8458de9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84540c8210 con 0x7f8454153100 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454070c10 0x7f84541f35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454070c10 0x7f84541f35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52252/0 (socket says 172.21.15.119:52252) 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 -- 172.21.15.119:0/335837541 learned_addr learned my addr 172.21.15.119:0/335837541 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:35.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8453fff700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8454153100 0x7f84541f4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f8452ffd700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8454152d20 0x7f8454157b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 -- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8454153100 msgr2=0x7f84541f4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8454153100 0x7f84541f4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 -- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8454152d20 msgr2=0x7f8454157b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8454152d20 0x7f8454157b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 -- 172.21.15.119:0/335837541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8444009d40 con 0x7f8454070c10 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f84537fe700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454070c10 0x7f84541f35d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8444009fd0 tx=0x7f8444004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:35.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7ff4bffff700 1 -- 172.21.15.119:0/1833758009 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff4c8051ff0 con 0x7ff4d8160c60 2024-09-13T16:25:35.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f843bfff700 1 -- 172.21.15.119:0/335837541 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84440038c0 con 0x7f8454070c10 2024-09-13T16:25:35.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f9719ffb700 1 -- 172.21.15.119:0/2526345267 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f97140233e0 con 0x7f972406f0c0 2024-09-13T16:25:35.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f9719ffb700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97080667b0 0x7f9708068c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:35.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f972925a700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97080667b0 0x7f9708068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:35.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f972925a700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97080667b0 0x7f9708068c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f971c00b450 tx=0x7f971c00d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f9e7e7fc700 1 -- 172.21.15.119:0/346090684 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9e84003c00 con 0x7f9e880b0fe0 2024-09-13T16:25:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f9e7e7fc700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e6c066730 0x7f9e6c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f9e7e7fc700 1 -- 172.21.15.119:0/346090684 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9e84085130 con 0x7f9e880b0fe0 2024-09-13T16:25:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f9e8d1d6700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e6c066730 0x7f9e6c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:36.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f9e8d1d6700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e6c066730 0x7f9e6c068bf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9e78007fd0 tx=0x7f9e78007940 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:36.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f843bfff700 1 -- 172.21.15.119:0/335837541 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8444003a40 con 0x7f8454070c10 2024-09-13T16:25:36.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f843bfff700 1 -- 172.21.15.119:0/335837541 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8444003df0 con 0x7f8454070c10 2024-09-13T16:25:36.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f9719ffb700 1 -- 172.21.15.119:0/2526345267 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9714066fa0 con 0x7f972406f0c0 2024-09-13T16:25:36.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.957+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84540c84f0 con 0x7f8454070c10 2024-09-13T16:25:36.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.957+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84541580b0 con 0x7f8454070c10 2024-09-13T16:25:36.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.957+0000 7f9e7e7fc700 1 -- 172.21.15.119:0/346090684 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9e84052240 con 0x7f9e880b0fe0 2024-09-13T16:25:36.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.957+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3787204285 shutdown_connections 2024-09-13T16:25:36.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f24005310 con 0x7f5f4415a250 2024-09-13T16:25:36.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.949+0000 7f5f2bfff700 1 -- 172.21.15.119:0/2663856819 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5f34020070 con 0x7f5f4415a250 2024-09-13T16:25:36.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.953+0000 7f5f2bfff700 1 -- 172.21.15.119:0/2663856819 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5f34015020 con 0x7f5f4415a250 2024-09-13T16:25:36.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.961+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8454159a20 con 0x7f8454070c10 2024-09-13T16:25:36.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.961+0000 7f9719ffb700 1 -- 172.21.15.119:0/2526345267 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9714062be0 con 0x7f972406f0c0 2024-09-13T16:25:36.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.961+0000 7f843bfff700 1 -- 172.21.15.119:0/335837541 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8444018400 con 0x7f8454070c10 2024-09-13T16:25:36.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.961+0000 7f843bfff700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8440066730 0x7f8440068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:36.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.961+0000 7f843bfff700 1 -- 172.21.15.119:0/335837541 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f84440867b0 con 0x7f8454070c10 2024-09-13T16:25:36.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.961+0000 7f8452ffd700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8440066730 0x7f8440068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:36.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.965+0000 7f8452ffd700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8440066730 0x7f8440068bf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84540b2d80 tx=0x7f8448009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:36.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3787204285 wait complete. 2024-09-13T16:25:36.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 Processor -- start 2024-09-13T16:25:36.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 -- start start 2024-09-13T16:25:36.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f843bfff700 1 -- 172.21.15.119:0/335837541 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8444053940 con 0x7f8454070c10 2024-09-13T16:25:36.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600b2350 0x7f4d6015b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:36.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b3b00 0x7f4d600af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:36.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60159850 0x7f4d6014d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:36.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014d170 con 0x7f4d600b2350 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014c560 con 0x7f4d600b3b00 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d66a6a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6014c840 con 0x7f4d60159850 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.969+0000 7f4d64806700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600b2350 0x7f4d6015b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600b2350 0x7f4d6015b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52260/0 (socket says 172.21.15.119:52260) 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 -- 172.21.15.119:0/3547118167 learned_addr learned my addr 172.21.15.119:0/3547118167 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d65007700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60159850 0x7f4d6014d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d5ffff700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b3b00 0x7f4d600af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 -- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60159850 msgr2=0x7f4d6014d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60159850 0x7f4d6014d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 -- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b3b00 msgr2=0x7f4d600af290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b3b00 0x7f4d600af290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 -- 172.21.15.119:0/3547118167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094b0 con 0x7f4d600b2350 2024-09-13T16:25:36.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d64806700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600b2350 0x7f4d6015b4d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4d50014040 tx=0x7f4d50004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d5d7fa700 1 -- 172.21.15.119:0/3547118167 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50004610 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6014cb20 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6014fb60 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d5d7fa700 1 -- 172.21.15.119:0/3547118167 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d500036c0 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.973+0000 7f4d5d7fa700 1 -- 172.21.15.119:0/3547118167 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50003a70 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.977+0000 7f4d5d7fa700 1 -- 172.21.15.119:0/3547118167 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4d50003bd0 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.977+0000 7f4d5d7fa700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066800 0x7f4d4c068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.977+0000 7f4d5d7fa700 1 -- 172.21.15.119:0/3547118167 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4d500859d0 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.977+0000 7f4d5ffff700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066800 0x7f4d4c068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.977+0000 7f4d5ffff700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066800 0x7f4d4c068cc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d600703c0 tx=0x7f4d54009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.985+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d40005310 con 0x7f4d600b2350 2024-09-13T16:25:36.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:35.989+0000 7f4d5d7fa700 1 -- 172.21.15.119:0/3547118167 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4d50052ac0 con 0x7f4d600b2350 2024-09-13T16:25:36.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.261+0000 7f6431fa4700 1 -- 172.21.15.119:0/4020620000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f640c0055c0 con 0x7f642c0b17f0 2024-09-13T16:25:36.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.261+0000 7f64297fa700 1 -- 172.21.15.119:0/4020620000 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6420053f20 con 0x7f642c0b17f0 2024-09-13T16:25:36.268 INFO:teuthology.orchestra.run.smithi119.stdout:60129542149 2024-09-13T16:25:36.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 -- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6418066800 msgr2=0x7f6418068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f4df00055c0 con 0x7f4e080b17f0 2024-09-13T16:25:36.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6418066800 0x7f6418068cc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f642c04e960 tx=0x7f641c008030 comp rx=0 tx=0).stop 2024-09-13T16:25:36.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 -- 172.21.15.119:0/4020620000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f642c0b17f0 msgr2=0x7f642c0c4150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f642c0b17f0 0x7f642c0c4150 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f642000c520 tx=0x7f642000c830 comp rx=0 tx=0).stop 2024-09-13T16:25:36.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 -- 172.21.15.119:0/4020620000 shutdown_connections 2024-09-13T16:25:36.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6418066800 0x7f6418068cc0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f642c160c60 0x7f642c0a9760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f642c0b17f0 0x7f642c0c4150 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 --2- 172.21.15.119:0/4020620000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f642c06cbd0 0x7f642c1f9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f4dfe7fc700 1 -- 172.21.15.119:0/59183036 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4e04015020 con 0x7f4e080b17f0 2024-09-13T16:25:36.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.269+0000 7f6412ffd700 1 -- 172.21.15.119:0/4020620000 >> 172.21.15.119:0/4020620000 conn(0x7f642c0b04f0 msgr2=0x7f642c0cbfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.276 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:36.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f6412ffd700 1 -- 172.21.15.119:0/4020620000 shutdown_connections 2024-09-13T16:25:36.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f6412ffd700 1 -- 172.21.15.119:0/4020620000 wait complete. 2024-09-13T16:25:36.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4dec066800 msgr2=0x7f4dec068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4dec066800 0x7f4dec068cc0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4e080ca9b0 tx=0x7f4df8009040 comp rx=0 tx=0).stop 2024-09-13T16:25:36.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e080b17f0 msgr2=0x7f4e080aacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e080b17f0 0x7f4e080aacf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000c00 tx=0x7f4e04004770 comp rx=0 tx=0).stop 2024-09-13T16:25:36.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 shutdown_connections 2024-09-13T16:25:36.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4dec066800 0x7f4dec068cc0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e080cd8d0 0x7f4e081f6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e080b2200 0x7f4e080ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 --2- 172.21.15.119:0/59183036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e080b17f0 0x7f4e080aacf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 >> 172.21.15.119:0/59183036 conn(0x7f4e0806a350 msgr2=0x7f4e08162540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 shutdown_connections 2024-09-13T16:25:36.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.273+0000 7f4e0e967700 1 -- 172.21.15.119:0/59183036 wait complete. 2024-09-13T16:25:36.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.277+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff4d8164f80 con 0x7ff4d8160c60 2024-09-13T16:25:36.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.277+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f5f240055c0 con 0x7f5f4415a250 2024-09-13T16:25:36.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.277+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9724067990 con 0x7f972406f0c0 2024-09-13T16:25:36.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.277+0000 7f5f2bfff700 1 -- 172.21.15.119:0/2663856819 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5f34052ac0 con 0x7f5f4415a250 2024-09-13T16:25:36.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f9e88159a20 con 0x7f9e880b0fe0 2024-09-13T16:25:36.286 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:36.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.277+0000 7ff4bffff700 1 -- 172.21.15.119:0/1833758009 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff4c8055610 con 0x7ff4d8160c60 2024-09-13T16:25:36.288 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:36.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f30066760 msgr2=0x7f5f30068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f30066760 0x7f5f30068c20 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5f38005930 tx=0x7f5f38009040 comp rx=0 tx=0).stop 2024-09-13T16:25:36.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a250 msgr2=0x7f5f44154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a250 0x7f5f44154930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5f34014040 tx=0x7f5f34004770 comp rx=0 tx=0).stop 2024-09-13T16:25:36.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f9719ffb700 1 -- 172.21.15.119:0/2526345267 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9714032020 con 0x7f972406f0c0 2024-09-13T16:25:36.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f9e7e7fc700 1 -- 172.21.15.119:0/346090684 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9e84055860 con 0x7f9e880b0fe0 2024-09-13T16:25:36.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4c4066760 msgr2=0x7ff4c4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4c4066760 0x7ff4c4068c20 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7ff4d81606a0 tx=0x7ff4cc0090e0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4d8160c60 msgr2=0x7ff4d81fa6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4d8160c60 0x7ff4d81fa6a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8011a80 tx=0x7ff4c8011d90 comp rx=0 tx=0).stop 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 shutdown_connections 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4c4066760 0x7ff4c4068c20 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4d80708b0 0x7ff4d8070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4d81f6000 0x7ff4d81f72b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 --2- 172.21.15.119:0/1833758009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4d8160c60 0x7ff4d81fa6a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 >> 172.21.15.119:0/1833758009 conn(0x7ff4d80b04f0 msgr2=0x7ff4d80af4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 shutdown_connections 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7ff4dc87a700 1 -- 172.21.15.119:0/1833758009 wait complete. 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 shutdown_connections 2024-09-13T16:25:36.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f30066760 0x7f5f30068c20 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f4415a630 0x7f5f440ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f4415a250 0x7f5f44154930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 --2- 172.21.15.119:0/2663856819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f44070c10 0x7f5f44147000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 >> 172.21.15.119:0/2663856819 conn(0x7f5f440b04f0 msgr2=0x7f5f440afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 shutdown_connections 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.281+0000 7f5f48ded700 1 -- 172.21.15.119:0/2663856819 wait complete. 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:36.292 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:36.293 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.6 2024-09-13T16:25:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97080667b0 msgr2=0x7f9708068c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97080667b0 0x7f9708068c70 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f971c00b450 tx=0x7f971c00d040 comp rx=0 tx=0).stop 2024-09-13T16:25:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406f0c0 msgr2=0x7f972415bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406f0c0 0x7f972415bec0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f971401f880 tx=0x7f971401fb90 comp rx=0 tx=0).stop 2024-09-13T16:25:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 shutdown_connections 2024-09-13T16:25:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97080667b0 0x7f9708068c70 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9724153bf0 0x7f97241541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f972406f0c0 0x7f972415bec0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 --2- 172.21.15.119:0/2526345267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f972406d460 0x7f97241493d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 >> 172.21.15.119:0/2526345267 conn(0x7f97240b15e0 msgr2=0x7f972406e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 shutdown_connections 2024-09-13T16:25:36.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.285+0000 7f972a25c700 1 -- 172.21.15.119:0/2526345267 wait complete. 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e6c066730 msgr2=0x7f9e6c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e6c066730 0x7f9e6c068bf0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9e78007fd0 tx=0x7f9e78007940 comp rx=0 tx=0).stop 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e880b0fe0 msgr2=0x7f9e8814b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e880b0fe0 0x7f9e8814b7f0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f9e8401a040 tx=0x7f9e84004770 comp rx=0 tx=0).stop 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 shutdown_connections 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e6c066730 0x7f9e6c068bf0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e88153100 0x7f9e88155630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e88152d20 0x7f9e881550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 --2- 172.21.15.119:0/346090684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e880b0fe0 0x7f9e8814b7f0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 >> 172.21.15.119:0/346090684 conn(0x7f9e880b04f0 msgr2=0x7f9e881520e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 shutdown_connections 2024-09-13T16:25:36.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.289+0000 7f9e8e9d9700 1 -- 172.21.15.119:0/346090684 wait complete. 2024-09-13T16:25:36.297 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.5 2024-09-13T16:25:36.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.293+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f84540b0ae0 con 0x7f8454070c10 2024-09-13T16:25:36.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.293+0000 7f843bfff700 1 -- 172.21.15.119:0/335837541 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8444056f60 con 0x7f8454070c10 2024-09-13T16:25:36.298 INFO:teuthology.orchestra.run.smithi119.stdout:60129542149 2024-09-13T16:25:36.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8440066730 msgr2=0x7f8440068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8440066730 0x7f8440068bf0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f84540b2d80 tx=0x7f8448009040 comp rx=0 tx=0).stop 2024-09-13T16:25:36.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454070c10 msgr2=0x7f84541f35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454070c10 0x7f84541f35d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8444009fd0 tx=0x7f8444004770 comp rx=0 tx=0).stop 2024-09-13T16:25:36.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 shutdown_connections 2024-09-13T16:25:36.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8440066730 0x7f8440068bf0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8454153100 0x7f84541f4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8454152d20 0x7f8454157b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 --2- 172.21.15.119:0/335837541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8454070c10 0x7f84541f35d0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 >> 172.21.15.119:0/335837541 conn(0x7f84540b04f0 msgr2=0x7f8454150fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 shutdown_connections 2024-09-13T16:25:36.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.297+0000 7f8458de9700 1 -- 172.21.15.119:0/335837541 wait complete. 2024-09-13T16:25:36.301 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.2 2024-09-13T16:25:36.305 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.3 2024-09-13T16:25:36.308 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.1 2024-09-13T16:25:36.309 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.7 2024-09-13T16:25:36.312 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.4 2024-09-13T16:25:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.309+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f4d400055c0 con 0x7f4d600b2350 2024-09-13T16:25:36.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.309+0000 7f4d5d7fa700 1 -- 172.21.15.119:0/3547118167 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d50020020 con 0x7f4d600b2350 2024-09-13T16:25:36.313 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:36.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.309+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066800 msgr2=0x7f4d4c068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.309+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066800 0x7f4d4c068cc0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f4d600703c0 tx=0x7f4d54009040 comp rx=0 tx=0).stop 2024-09-13T16:25:36.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.309+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600b2350 msgr2=0x7f4d6015b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:36.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.309+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600b2350 0x7f4d6015b4d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4d50014040 tx=0x7f4d50004770 comp rx=0 tx=0).stop 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 shutdown_connections 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066800 0x7f4d4c068cc0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d60159850 0x7f4d6014d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d600b3b00 0x7f4d600af290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 --2- 172.21.15.119:0/3547118167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600b2350 0x7f4d6015b4d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 >> 172.21.15.119:0/3547118167 conn(0x7f4d6006a350 msgr2=0x7f4d60066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 shutdown_connections 2024-09-13T16:25:36.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:36.313+0000 7f4d66a6a700 1 -- 172.21.15.119:0/3547118167 wait complete. 2024-09-13T16:25:36.325 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.0 2024-09-13T16:25:37.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T16:25:37.298 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T16:25:37.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T16:25:37.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T16:25:37.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T16:25:37.310 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T16:25:37.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T16:25:37.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T16:25:37.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.429+0000 7f97f5829700 1 Processor -- start 2024-09-13T16:25:37.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.429+0000 7f97f5829700 1 -- start start 2024-09-13T16:25:37.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97f5829700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0152d20 0x7f97f0159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97f5829700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f97f0153100 0x7f97f00b17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97f5829700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f97f0070c10 0x7f97f00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97f5829700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f00b28a0 con 0x7f97f0153100 2024-09-13T16:25:37.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97f5829700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f0071600 con 0x7f97f0070c10 2024-09-13T16:25:37.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97f5829700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f00b1d30 con 0x7f97f0152d20 2024-09-13T16:25:37.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ee7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f97f0153100 0x7f97f00b17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f97f0070c10 0x7f97f00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f97f0070c10 0x7f97f00710c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40842/0 (socket says 172.21.15.119:40842) 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 -- 172.21.15.119:0/1214629743 learned_addr learned my addr 172.21.15.119:0/1214629743 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 -- 172.21.15.119:0/1214629743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0152d20 msgr2=0x7f97f0159a40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 --2- 172.21.15.119:0/1214629743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0152d20 0x7f97f0159a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 -- 172.21.15.119:0/1214629743 >> v2:172.21.15.119:3300/0 conn(0x7f97f0153100 msgr2=0x7f97f00b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 --2- 172.21.15.119:0/1214629743 >> v2:172.21.15.119:3300/0 conn(0x7f97f0153100 0x7f97f00b17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 -- 172.21.15.119:0/1214629743 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f00b2000 con 0x7f97f0070c10 2024-09-13T16:25:37.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ee7fc700 1 --2- 172.21.15.119:0/1214629743 >> v2:172.21.15.119:3300/0 conn(0x7f97f0153100 0x7f97f00b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:37.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97ef7fe700 1 --2- 172.21.15.119:0/1214629743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0070c10 0x7f97f00710c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f97e8005dc0 tx=0x7f97e80060d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bdae96812b84fef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97edffb700 1 -- 172.21.15.119:0/1214629743 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e8007440 con 0x7f97f0070c10 2024-09-13T16:25:37.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.433+0000 7f97edffb700 1 -- 172.21.15.119:0/1214629743 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e800c720 con 0x7f97f0070c10 2024-09-13T16:25:37.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 -- 172.21.15.119:0/1214629743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0070c10 msgr2=0x7f97f00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 --2- 172.21.15.119:0/1214629743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0070c10 0x7f97f00710c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f97e8005dc0 tx=0x7f97e80060d0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 -- 172.21.15.119:0/1214629743 shutdown_connections 2024-09-13T16:25:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 --2- 172.21.15.119:0/1214629743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0070c10 0x7f97f00710c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 --2- 172.21.15.119:0/1214629743 >> v2:172.21.15.119:3300/0 conn(0x7f97f0153100 0x7f97f00b17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 --2- 172.21.15.119:0/1214629743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0152d20 0x7f97f0159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 -- 172.21.15.119:0/1214629743 >> 172.21.15.119:0/1214629743 conn(0x7f97f00b04f0 msgr2=0x7f97f00b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 -- 172.21.15.119:0/1214629743 shutdown_connections 2024-09-13T16:25:37.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 -- 172.21.15.119:0/1214629743 wait complete. 2024-09-13T16:25:37.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 Processor -- start 2024-09-13T16:25:37.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.437+0000 7f97f5829700 1 -- start start 2024-09-13T16:25:37.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f0070c10 0x7f97f014a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f0070c10 0x7f97f014a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f0070c10 0x7f97f014a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52280/0 (socket says 172.21.15.119:52280) 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0152d20 0x7f97f006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0153100 0x7f97f0157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f01540d0 con 0x7f97f0070c10 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f01516d0 con 0x7f97f0152d20 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f0151970 con 0x7f97f0153100 2024-09-13T16:25:37.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 -- 172.21.15.119:0/2186027011 learned_addr learned my addr 172.21.15.119:0/2186027011 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97ee7fc700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0152d20 0x7f97f006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97ef7fe700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0153100 0x7f97f0157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 -- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0153100 msgr2=0x7f97f0157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0153100 0x7f97f0157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 -- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0152d20 msgr2=0x7f97f006d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0152d20 0x7f97f006d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 -- 172.21.15.119:0/2186027011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e00094b0 con 0x7f97f0070c10 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97eeffd700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f0070c10 0x7f97f014a5b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f97e400b520 tx=0x7f97e400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97d7fff700 1 -- 172.21.15.119:0/2186027011 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4011830 con 0x7f97f0070c10 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 -- 172.21.15.119:0/2186027011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e800c200 con 0x7f97f0070c10 2024-09-13T16:25:37.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.441+0000 7f97f5829700 1 -- 172.21.15.119:0/2186027011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f0151f30 con 0x7f97f0070c10 2024-09-13T16:25:37.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.445+0000 7f97d7fff700 1 -- 172.21.15.119:0/2186027011 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e40119d0 con 0x7f97f0070c10 2024-09-13T16:25:37.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.445+0000 7f97d7fff700 1 -- 172.21.15.119:0/2186027011 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e4011d80 con 0x7f97f0070c10 2024-09-13T16:25:37.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.445+0000 7f97d7fff700 1 -- 172.21.15.119:0/2186027011 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f97e401a400 con 0x7f97f0070c10 2024-09-13T16:25:37.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.445+0000 7f97d7fff700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97d8066a20 0x7f97d8068ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.445+0000 7f97d7fff700 1 -- 172.21.15.119:0/2186027011 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f97e40845b0 con 0x7f97f0070c10 2024-09-13T16:25:37.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.445+0000 7f97ee7fc700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97d8066a20 0x7f97d8068ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.445+0000 7f97f5829700 1 -- 172.21.15.119:0/2186027011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97dc005310 con 0x7f97f0070c10 2024-09-13T16:25:37.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0ef09700 1 Processor -- start 2024-09-13T16:25:37.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7f97ee7fc700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97d8066a20 0x7f97d8068ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f97f006df60 tx=0x7f97e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0ef09700 1 -- start start 2024-09-13T16:25:37.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0ef09700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08153100 0x7fee08159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0ef09700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fee08070c10 0x7fee080710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0ef09700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080b2000 con 0x7fee08152d20 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0ef09700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee080b28a0 con 0x7fee08070c10 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0ef09700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee08071600 con 0x7fee08153100 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0cca5700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fee08152d20 0x7fee080b17f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33764/0 (socket says 172.21.15.119:33764) 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0cca5700 1 -- 172.21.15.119:0/3542200536 learned_addr learned my addr 172.21.15.119:0/3542200536 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee0d4a6700 1 --2- 172.21.15.119:0/3542200536 >> v2:172.21.15.119:3300/0 conn(0x7fee08070c10 0x7fee080710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.449+0000 7fee07fff700 1 --2- 172.21.15.119:0/3542200536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08153100 0x7fee08159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee0d4a6700 1 -- 172.21.15.119:0/3542200536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08153100 msgr2=0x7fee08159a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee0d4a6700 1 --2- 172.21.15.119:0/3542200536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08153100 0x7fee08159a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee0d4a6700 1 -- 172.21.15.119:0/3542200536 >> v1:172.21.15.119:6789/0 conn(0x7fee08152d20 legacy=0x7fee080b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee0d4a6700 1 -- 172.21.15.119:0/3542200536 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee080b22e0 con 0x7fee08070c10 2024-09-13T16:25:37.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee0d4a6700 1 --2- 172.21.15.119:0/3542200536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08070c10 0x7fee080710c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee0000aea0 tx=0x7fee0000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bd185f1422fab2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee077fe700 1 -- 172.21.15.119:0/3542200536 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0000f070 con 0x7fee08070c10 2024-09-13T16:25:37.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7f97d7fff700 1 -- 172.21.15.119:0/2186027011 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f97e4055130 con 0x7f97f0070c10 2024-09-13T16:25:37.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee077fe700 1 -- 172.21.15.119:0/3542200536 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee0000a2e0 con 0x7fee08070c10 2024-09-13T16:25:37.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.453+0000 7fee077fe700 1 -- 172.21.15.119:0/3542200536 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee0000cb30 con 0x7fee08070c10 2024-09-13T16:25:37.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 -- 172.21.15.119:0/3542200536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08070c10 msgr2=0x7fee080710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 --2- 172.21.15.119:0/3542200536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08070c10 0x7fee080710c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fee0000aea0 tx=0x7fee0000c330 comp rx=0 tx=0).stop 2024-09-13T16:25:37.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 -- 172.21.15.119:0/3542200536 shutdown_connections 2024-09-13T16:25:37.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 --2- 172.21.15.119:0/3542200536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08070c10 0x7fee080710c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 --2- 172.21.15.119:0/3542200536 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08153100 0x7fee08159a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 -- 172.21.15.119:0/3542200536 >> 172.21.15.119:0/3542200536 conn(0x7fee080b04f0 msgr2=0x7fee080b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 -- 172.21.15.119:0/3542200536 shutdown_connections 2024-09-13T16:25:37.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 -- 172.21.15.119:0/3542200536 wait complete. 2024-09-13T16:25:37.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 Processor -- start 2024-09-13T16:25:37.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.457+0000 7fee0ef09700 1 -- start start 2024-09-13T16:25:37.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0ef09700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08070c10 0x7fee081f5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0ef09700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08152d20 0x7fee08163b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0ef09700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee08153100 0x7fee08162850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08152d20 0x7fee08163b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08152d20 0x7fee08163b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52290/0 (socket says 172.21.15.119:52290) 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 -- 172.21.15.119:0/1658906967 learned_addr learned my addr 172.21.15.119:0/1658906967 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081f6270 con 0x7fee08152d20 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081640c0 con 0x7fee08153100 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee08154530 con 0x7fee08070c10 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0cca5700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08070c10 0x7fee081f5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0d4a6700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee08153100 0x7fee08162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 -- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08070c10 msgr2=0x7fee081f5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08070c10 0x7fee081f5d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 -- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee08153100 msgr2=0x7fee08162850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee08153100 0x7fee08162850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0000ab80 con 0x7fee08152d20 2024-09-13T16:25:37.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee07fff700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08152d20 0x7fee08163b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009fd0 tx=0x7fedf800ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee057fa700 1 -- 172.21.15.119:0/1658906967 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8013680 con 0x7fee08152d20 2024-09-13T16:25:37.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.461+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee08154810 con 0x7fee08152d20 2024-09-13T16:25:37.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.465+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee08154dd0 con 0x7fee08152d20 2024-09-13T16:25:37.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.465+0000 7fee057fa700 1 -- 172.21.15.119:0/1658906967 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf80137e0 con 0x7fee08152d20 2024-09-13T16:25:37.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.465+0000 7fee057fa700 1 -- 172.21.15.119:0/1658906967 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf8013b90 con 0x7fee08152d20 2024-09-13T16:25:37.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.465+0000 7fee057fa700 1 -- 172.21.15.119:0/1658906967 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fedf8013cf0 con 0x7fee08152d20 2024-09-13T16:25:37.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.465+0000 7fee057fa700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fedf4066880 0x7fedf4068d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.465+0000 7fee057fa700 1 -- 172.21.15.119:0/1658906967 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fedf8088170 con 0x7fee08152d20 2024-09-13T16:25:37.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.469+0000 7fedeeffd700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fede8005310 con 0x7fee08152d20 2024-09-13T16:25:37.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.469+0000 7fee0cca5700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fedf4066880 0x7fedf4068d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.469+0000 7fee0cca5700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fedf4066880 0x7fedf4068d40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b9b0 tx=0x7fedfc005da0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.477+0000 7fee057fa700 1 -- 172.21.15.119:0/1658906967 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fedf8017080 con 0x7fee08152d20 2024-09-13T16:25:37.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b51b0f700 1 Processor -- start 2024-09-13T16:25:37.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b51b0f700 1 -- start start 2024-09-13T16:25:37.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f95b7d58700 1 Processor -- start 2024-09-13T16:25:37.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f95b7d58700 1 -- start start 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b51b0f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7b4c160c60 0x7f7b4c0b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b51b0f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7b4c0b17f0 0x7f7b4c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b51b0f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c0b2160 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b51b0f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c06d930 con 0x7f7b4c160c60 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b51b0f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c164850 con 0x7f7b4c0b17f0 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b4bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7b4c06cbd0 0x7f7b4c164f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33772/0 (socket says 172.21.15.119:33772) 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b4bfff700 1 -- 172.21.15.119:0/3824863464 learned_addr learned my addr 172.21.15.119:0/3824863464 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b50b0d700 1 --2- 172.21.15.119:0/3824863464 >> v2:172.21.15.119:3300/0 conn(0x7f7b4c160c60 0x7f7b4c0b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b5130e700 1 --2- 172.21.15.119:0/3824863464 >> v2:172.21.15.145:3300/0 conn(0x7f7b4c0b17f0 0x7f7b4c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b4b7fe700 1 -- 172.21.15.119:0/3824863464 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487853301 0 0) 0x7f7b4c0b2160 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b4b7fe700 1 -- 172.21.15.119:0/3824863464 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b340035d0 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b50b0d700 1 -- 172.21.15.119:0/3824863464 >> v2:172.21.15.145:3300/0 conn(0x7f7b4c0b17f0 msgr2=0x7f7b4c0b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b50b0d700 1 --2- 172.21.15.119:0/3824863464 >> v2:172.21.15.145:3300/0 conn(0x7f7b4c0b17f0 0x7f7b4c0b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b50b0d700 1 -- 172.21.15.119:0/3824863464 >> v1:172.21.15.119:6789/0 conn(0x7f7b4c06cbd0 legacy=0x7f7b4c164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b50b0d700 1 -- 172.21.15.119:0/3824863464 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c164af0 con 0x7f7b4c160c60 2024-09-13T16:25:37.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b50b0d700 1 --2- 172.21.15.119:0/3824863464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c160c60 0x7f7b4c0b38a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c071c40 tx=0x7f7b4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78175a77aa0d521e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f95b7d58700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f95b00b1f00 0x7f95b00b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f7b4b7fe700 1 -- 172.21.15.119:0/3824863464 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4400e070 con 0x7f7b4c160c60 2024-09-13T16:25:37.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.485+0000 7f95b5af4700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f95b0160c60 0x7f95b0164f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33786/0 (socket says 172.21.15.119:33786) 2024-09-13T16:25:37.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b4b7fe700 1 -- 172.21.15.119:0/3824863464 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4400bb20 con 0x7f7b4c160c60 2024-09-13T16:25:37.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 -- 172.21.15.119:0/3572166409 learned_addr learned my addr 172.21.15.119:0/3572166409 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:37.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b4b7fe700 1 -- 172.21.15.119:0/3824863464 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b4400be90 con 0x7f7b4c160c60 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b0164af0 con 0x7f95b0160c60 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95b00ce220 con 0x7f95b006cbd0 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95b006d930 con 0x7f95b00b1f00 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052941692 0 0) 0x7f95b0164af0 con 0x7f95b0160c60 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f959c0035d0 con 0x7f95b0160c60 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999902268 0 0) 0x7f95b00ce220 con 0x7f95b006cbd0 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b0164af0 con 0x7f95b006cbd0 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 -- 172.21.15.119:0/3824863464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c160c60 msgr2=0x7f7b4c0b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/3824863464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c160c60 0x7f7b4c0b38a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c071c40 tx=0x7f7b4400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 831259893 0 0) 0x7f959c0035d0 con 0x7f95b0160c60 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 >> v2:172.21.15.119:3300/0 conn(0x7f95b00b1f00 msgr2=0x7f95b00b38a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b62f5700 1 --2- 172.21.15.119:0/3572166409 >> v2:172.21.15.119:3300/0 conn(0x7f95b00b1f00 0x7f95b00b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 --2- 172.21.15.119:0/3572166409 >> v2:172.21.15.119:3300/0 conn(0x7f95b00b1f00 0x7f95b00b38a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 >> v1:172.21.15.145:6789/0 conn(0x7f95b006cbd0 legacy=0x7f95b00b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b006d090 con 0x7f95b0160c60 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b62f5700 1 --2- 172.21.15.119:0/3572166409 >> v2:172.21.15.119:3300/0 conn(0x7f95b00b1f00 0x7f95b00b38a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:37.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f95ac001670 con 0x7f95b0160c60 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 >> v1:172.21.15.119:6789/0 conn(0x7f95b0160c60 legacy=0x7f95b0164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f959c004c40 0x7f959c005020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c003810 0x7f959c003c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 -- 172.21.15.119:0/3824863464 shutdown_connections 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/3824863464 >> v2:172.21.15.145:3300/0 conn(0x7f7b4c0b17f0 0x7f7b4c0b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/3824863464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c160c60 0x7f7b4c0b38a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 -- 172.21.15.119:0/3824863464 >> 172.21.15.119:0/3824863464 conn(0x7f7b4c0b04f0 msgr2=0x7f7b4c0b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 -- 172.21.15.119:0/3824863464 shutdown_connections 2024-09-13T16:25:37.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f959c0042f0 0x7f959c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a80015b0 con 0x7f959c004c40 2024-09-13T16:25:37.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959c00c200 con 0x7f959c003810 2024-09-13T16:25:37.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959c00c3c0 con 0x7f959c0042f0 2024-09-13T16:25:37.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 -- 172.21.15.119:0/3824863464 wait complete. 2024-09-13T16:25:37.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f959c004c40 0x7f959c005020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b52f3700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c003810 0x7f959c003c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b62f5700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f959c0042f0 0x7f959c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 -- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f959c0042f0 msgr2=0x7f959c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f959c0042f0 0x7f959c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 -- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c003810 msgr2=0x7f959c003c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c003810 0x7f959c003c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 -- 172.21.15.119:0/3572166409 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959c0035d0 con 0x7f959c004c40 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b62f5700 1 -- 172.21.15.119:0/3572166409 reap_dead start 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b5af4700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f959c004c40 0x7f959c005020 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f95ac001eb0 tx=0x7f959c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1357d3d908cb9d38 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00cd50 con 0x7f959c004c40 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95ac005980 con 0x7f959c004c40 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b4af2700 1 -- 172.21.15.119:0/3572166409 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95ac005d10 con 0x7f959c004c40 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 Processor -- start 2024-09-13T16:25:37.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f7b51b0f700 1 -- start start 2024-09-13T16:25:37.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f959c004c40 msgr2=0x7f959c005020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.489+0000 7f95b7d58700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f959c004c40 0x7f959c005020 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f95ac001eb0 tx=0x7f959c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 shutdown_connections 2024-09-13T16:25:37.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f95b7d58700 1 --2- 172.21.15.119:0/3572166409 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f959c004c40 0x7f959c005020 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c06cbd0 0x7f7b4c1524a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b4c0b17f0 0x7f7b4c15e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b4c1f9de0 0x7f7b4c0b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 >> 172.21.15.119:0/3572166409 conn(0x7f95b00b04f0 msgr2=0x7f95b00b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 shutdown_connections 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c1fa2e0 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c1fa5c0 con 0x7f7b4c0b17f0 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c1fa8a0 con 0x7f7b4c1f9de0 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c06cbd0 0x7f7b4c1524a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c06cbd0 0x7f7b4c1524a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52330/0 (socket says 172.21.15.119:52330) 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 -- 172.21.15.119:0/1623652968 learned_addr learned my addr 172.21.15.119:0/1623652968 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b4bfff700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b4c0b17f0 0x7f7b4c15e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b5130e700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b4c1f9de0 0x7f7b4c0b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 -- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b4c1f9de0 msgr2=0x7f7b4c0b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b4c1f9de0 0x7f7b4c0b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 -- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b4c0b17f0 msgr2=0x7f7b4c15e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b4c0b17f0 0x7f7b4c15e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 -- 172.21.15.119:0/1623652968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44009a70 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b4bfff700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b4c0b17f0 0x7f7b4c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:37.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b50b0d700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c06cbd0 0x7f7b4c1524a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400b370 tx=0x7f7b44004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f95b7d58700 1 -- 172.21.15.119:0/3572166409 wait complete. 2024-09-13T16:25:37.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b497fa700 1 -- 172.21.15.119:0/1623652968 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44007000 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b497fa700 1 -- 172.21.15.119:0/1623652968 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b44012840 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b497fa700 1 -- 172.21.15.119:0/1623652968 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b44012bf0 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c1fab80 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c1529e0 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.493+0000 7f7b497fa700 1 -- 172.21.15.119:0/1623652968 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7b44012d50 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f7b497fa700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b34066b90 0x7f7b34069050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 Processor -- start 2024-09-13T16:25:37.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 -- start start 2024-09-13T16:25:37.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f7b497fa700 1 -- 172.21.15.119:0/1623652968 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7b44086070 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f7b4bfff700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b34066b90 0x7f7b34069050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f7b4bfff700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b34066b90 0x7f7b34069050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c1566b0 tx=0x7f7b40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c004c40 0x7f95b015b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95b0161230 0x7f95b0163a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95b0162040 0x7f95b01f6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b01629e0 con 0x7f95b0161230 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95b0162cc0 con 0x7f959c004c40 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b7d58700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95b0162fa0 con 0x7f95b0162040 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95b0161230 0x7f95b0163a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95b0161230 0x7f95b0163a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52342/0 (socket says 172.21.15.119:52342) 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 -- 172.21.15.119:0/2924609160 learned_addr learned my addr 172.21.15.119:0/2924609160 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b5af4700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c004c40 0x7f95b015b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b62f5700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95b0162040 0x7f95b01f6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 -- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95b0162040 msgr2=0x7f95b01f6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95b0162040 0x7f95b01f6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 -- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c004c40 msgr2=0x7f95b015b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c004c40 0x7f95b015b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 -- 172.21.15.119:0/2924609160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b0163280 con 0x7f95b0161230 2024-09-13T16:25:37.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b5af4700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c004c40 0x7f95b015b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:37.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f95b52f3700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95b0161230 0x7f95b0163a80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f95a80021a0 tx=0x7f95a800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b2c005310 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f9ba1241700 1 Processor -- start 2024-09-13T16:25:37.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.497+0000 7f9ba1241700 1 -- start start 2024-09-13T16:25:37.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba1241700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9b9c15a250 0x7f9b9c06e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba1241700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9b9c070c10 0x7f9b9c0710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba1241700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b9c06dd40 con 0x7f9b9c15a630 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba1241700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c071600 con 0x7f9b9c070c10 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba1241700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b9c06d430 con 0x7f9b9c15a250 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9b9b7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9b9c15a630 0x7f9b9c0b3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33790/0 (socket says 172.21.15.119:33790) 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba0a40700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9b9c070c10 0x7f9b9c0710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9b9b7fe700 1 -- 172.21.15.119:0/3208664751 learned_addr learned my addr 172.21.15.119:0/3208664751 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9b9bfff700 1 --2- 172.21.15.119:0/3208664751 >> v2:172.21.15.145:3300/0 conn(0x7f9b9c15a250 0x7f9b9c06e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba0a40700 1 -- 172.21.15.119:0/3208664751 >> v2:172.21.15.145:3300/0 conn(0x7f9b9c15a250 msgr2=0x7f9b9c06e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba0a40700 1 --2- 172.21.15.119:0/3208664751 >> v2:172.21.15.145:3300/0 conn(0x7f9b9c15a250 0x7f9b9c06e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba0a40700 1 -- 172.21.15.119:0/3208664751 >> v1:172.21.15.119:6789/0 conn(0x7f9b9c15a630 legacy=0x7f9b9c0b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba0a40700 1 -- 172.21.15.119:0/3208664751 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c06d6f0 con 0x7f9b9c070c10 2024-09-13T16:25:37.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f9ba0a40700 1 --2- 172.21.15.119:0/3208664751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c070c10 0x7f9b9c0710c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0723b0 tx=0x7f9b8c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e7f0a9453257a8c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f2abd4e9700 1 Processor -- start 2024-09-13T16:25:37.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f2abd4e9700 1 -- start start 2024-09-13T16:25:37.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f95a67fc700 1 -- 172.21.15.119:0/2924609160 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a8003c30 con 0x7f95b0161230 2024-09-13T16:25:37.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f95a67fc700 1 -- 172.21.15.119:0/2924609160 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a8003dd0 con 0x7f95b0161230 2024-09-13T16:25:37.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95b0163560 con 0x7f95b0161230 2024-09-13T16:25:37.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f95a67fc700 1 -- 172.21.15.119:0/2924609160 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a800dc80 con 0x7f95b0161230 2024-09-13T16:25:37.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.501+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95b0163fc0 con 0x7f95b0161230 2024-09-13T16:25:37.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f7b497fa700 1 -- 172.21.15.119:0/1623652968 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7b44053230 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2abd4e9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2ab8070c10 0x7f2ab8070ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2abd4e9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab8071cf0 con 0x7f2ab80ad940 2024-09-13T16:25:37.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2abd4e9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab806cac0 con 0x7f2ab8071860 2024-09-13T16:25:37.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2abd4e9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ab80b3820 con 0x7f2ab8070c10 2024-09-13T16:25:37.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2ab7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2ab8070c10 0x7f2ab8070ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2ab7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2ab8070c10 0x7f2ab8070ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52352/0 (socket says 172.21.15.119:52352) 2024-09-13T16:25:37.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9b9affd700 1 -- 172.21.15.119:0/3208664751 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c011830 con 0x7f9b9c070c10 2024-09-13T16:25:37.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2ab7fff700 1 -- 172.21.15.119:0/2351085815 learned_addr learned my addr 172.21.15.119:0/2351085815 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9b9affd700 1 -- 172.21.15.119:0/3208664751 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b8c011990 con 0x7f9b9c070c10 2024-09-13T16:25:37.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9b9affd700 1 -- 172.21.15.119:0/3208664751 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b8c011d40 con 0x7f9b9c070c10 2024-09-13T16:25:37.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2ab7fff700 1 -- 172.21.15.119:0/2351085815 >> v1:172.21.15.145:6789/0 conn(0x7f2ab8071860 legacy=0x7f2ab80b3110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:25:37.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2ab7fff700 1 -- 172.21.15.119:0/2351085815 >> v1:172.21.15.119:6789/0 conn(0x7f2ab80ad940 legacy=0x7f2ab80b2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2ab7fff700 1 -- 172.21.15.119:0/2351085815 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab8071530 con 0x7f2ab8070c10 2024-09-13T16:25:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f2ab7fff700 1 --2- 172.21.15.119:0/2351085815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab8070ff0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009e50 tx=0x7f2ab000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dabe02bc7656b0da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9ba1241700 1 -- 172.21.15.119:0/3208664751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c070c10 msgr2=0x7f9b9c0710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9ba1241700 1 --2- 172.21.15.119:0/3208664751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c070c10 0x7f9b9c0710c0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b9c0723b0 tx=0x7f9b8c009b30 comp rx=0 tx=0).stop 2024-09-13T16:25:37.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f95a67fc700 1 -- 172.21.15.119:0/2924609160 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f95a80050d0 con 0x7f95b0161230 2024-09-13T16:25:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9ba1241700 1 -- 172.21.15.119:0/3208664751 shutdown_connections 2024-09-13T16:25:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9ba1241700 1 --2- 172.21.15.119:0/3208664751 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c070c10 0x7f9b9c0710c0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9ba1241700 1 --2- 172.21.15.119:0/3208664751 >> v2:172.21.15.145:3300/0 conn(0x7f9b9c15a250 0x7f9b9c06e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f9ba1241700 1 -- 172.21.15.119:0/3208664751 >> 172.21.15.119:0/3208664751 conn(0x7f9b9c0b04f0 msgr2=0x7f9b9c0b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f95a67fc700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9594066780 0x7f9594068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f95a67fc700 1 -- 172.21.15.119:0/2924609160 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f95a80846b0 con 0x7f95b0161230 2024-09-13T16:25:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f95b5af4700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9594066780 0x7f9594068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95b01641b0 con 0x7f95b0161230 2024-09-13T16:25:37.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.505+0000 7f95b5af4700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9594066780 0x7f9594068c40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f95b00b1340 tx=0x7f95ac00a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 -- 172.21.15.119:0/3208664751 shutdown_connections 2024-09-13T16:25:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 -- 172.21.15.119:0/3208664751 wait complete. 2024-09-13T16:25:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 Processor -- start 2024-09-13T16:25:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 -- start start 2024-09-13T16:25:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2ab6ffd700 1 -- 172.21.15.119:0/2351085815 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab000e070 con 0x7f2ab8070c10 2024-09-13T16:25:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2ab6ffd700 1 -- 172.21.15.119:0/2351085815 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab0003a40 con 0x7f2ab8070c10 2024-09-13T16:25:37.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2ab6ffd700 1 -- 172.21.15.119:0/2351085815 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0003df0 con 0x7f2ab8070c10 2024-09-13T16:25:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa7013d5700 1 Processor -- start 2024-09-13T16:25:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa7013d5700 1 -- start start 2024-09-13T16:25:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa7013d5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a630 0x7fa6fc06e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa7013d5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa6fc070c10 0x7fa6fc071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa7013d5700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6fc06dd40 con 0x7fa6fc15a250 2024-09-13T16:25:37.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa7013d5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6fc0715c0 con 0x7fa6fc070c10 2024-09-13T16:25:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa7013d5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc06d430 con 0x7fa6fc15a630 2024-09-13T16:25:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c070c10 0x7f9b9c1461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c15a250 0x7f9b9c1541c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c15a630 0x7f9b9c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c157440 con 0x7f9b9c15a250 2024-09-13T16:25:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c151760 con 0x7f9b9c070c10 2024-09-13T16:25:37.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba1241700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c156e80 con 0x7f9b9c15a630 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c070c10 0x7f9b9c1461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c070c10 0x7f9b9c1461a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40906/0 (socket says 172.21.15.119:40906) 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9bfff700 1 -- 172.21.15.119:0/1467896396 learned_addr learned my addr 172.21.15.119:0/1467896396 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9b7fe700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c15a250 0x7f9b9c1541c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9ba0a40700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c15a630 0x7f9b9c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9b7fe700 1 -- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c15a630 msgr2=0x7f9b9c1575f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9b7fe700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c15a630 0x7f9b9c1575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9b7fe700 1 -- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c070c10 msgr2=0x7f9b9c1461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9b7fe700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c070c10 0x7f9b9c1461a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9b7fe700 1 -- 172.21.15.119:0/1467896396 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b8c0094b0 con 0x7f9b9c15a250 2024-09-13T16:25:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2abd4e9700 1 -- 172.21.15.119:0/2351085815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 msgr2=0x7f2ab8070ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/2351085815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab8070ff0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f2ab0009e50 tx=0x7f2ab000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2abd4e9700 1 -- 172.21.15.119:0/2351085815 shutdown_connections 2024-09-13T16:25:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/2351085815 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab8070ff0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2abd4e9700 1 -- 172.21.15.119:0/2351085815 >> 172.21.15.119:0/2351085815 conn(0x7f2ab80b04f0 msgr2=0x7f2ab80b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2abd4e9700 1 -- 172.21.15.119:0/2351085815 shutdown_connections 2024-09-13T16:25:37.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f2abd4e9700 1 -- 172.21.15.119:0/2351085815 wait complete. 2024-09-13T16:25:37.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa6fbfff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa6fc15a250 0x7fa6fc0b3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:53274/0 (socket says 172.21.15.119:53274) 2024-09-13T16:25:37.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa6fbfff700 1 -- 172.21.15.119:0/167006383 learned_addr learned my addr 172.21.15.119:0/167006383 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:37.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7f9b9b7fe700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c15a250 0x7f9b9c1541c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400ca70 tx=0x7f9b9400caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa6fb7fe700 1 --2- 172.21.15.119:0/167006383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a630 0x7fa6fc06e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.509+0000 7fa700bd4700 1 --2- 172.21.15.119:0/167006383 >> v2:172.21.15.119:3300/0 conn(0x7fa6fc070c10 0x7fa6fc071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7fa700bd4700 1 -- 172.21.15.119:0/167006383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a630 msgr2=0x7fa6fc06e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7fa700bd4700 1 --2- 172.21.15.119:0/167006383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a630 0x7fa6fc06e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7fa700bd4700 1 -- 172.21.15.119:0/167006383 >> v1:172.21.15.145:6789/0 conn(0x7fa6fc15a250 legacy=0x7fa6fc0b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7fa700bd4700 1 -- 172.21.15.119:0/167006383 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6fc06d6f0 con 0x7fa6fc070c10 2024-09-13T16:25:37.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7fa700bd4700 1 --2- 172.21.15.119:0/167006383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc071080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009850 tx=0x7fa6ec009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da75a82f5b165cce server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 Processor -- start 2024-09-13T16:25:37.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 -- start start 2024-09-13T16:25:37.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab8159d80 0x7f2ab815c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab8150640 0x7f2ab8150ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab8151000 con 0x7f2ab8070c10 2024-09-13T16:25:37.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab815b040 con 0x7f2ab8159d80 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abd4e9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab815cda0 con 0x7f2ab8150640 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab814e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52394/0 (socket says 172.21.15.119:52394) 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 -- 172.21.15.119:0/895860285 learned_addr learned my addr 172.21.15.119:0/895860285 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab77fe700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab8159d80 0x7f2ab815c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2abcce8700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab8150640 0x7f2ab8150ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 -- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab8150640 msgr2=0x7f2ab8150ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab8150640 0x7f2ab8150ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 -- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab8159d80 msgr2=0x7f2ab815c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab8159d80 0x7f2ab815c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 -- 172.21.15.119:0/895860285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab0009b30 con 0x7f2ab8070c10 2024-09-13T16:25:37.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f2ab7fff700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab814e190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000b730 tx=0x7f2ab0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f9b98ff9700 1 -- 172.21.15.119:0/1467896396 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94007480 con 0x7f9b9c15a250 2024-09-13T16:25:37.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f9b98ff9700 1 -- 172.21.15.119:0/1467896396 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b940075e0 con 0x7f9b9c15a250 2024-09-13T16:25:37.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f9b98ff9700 1 -- 172.21.15.119:0/1467896396 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b94007950 con 0x7f9b9c15a250 2024-09-13T16:25:37.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.513+0000 7f95a67fc700 1 -- 172.21.15.119:0/2924609160 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f95a8051810 con 0x7f95b0161230 2024-09-13T16:25:37.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c157160 con 0x7f9b9c15a250 2024-09-13T16:25:37.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d69b1c700 1 Processor -- start 2024-09-13T16:25:37.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c1466e0 con 0x7f9b9c15a250 2024-09-13T16:25:37.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d69b1c700 1 -- start start 2024-09-13T16:25:37.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7fa6faffd700 1 -- 172.21.15.119:0/167006383 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec004230 con 0x7fa6fc070c10 2024-09-13T16:25:37.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7fa6faffd700 1 -- 172.21.15.119:0/167006383 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ec004390 con 0x7fa6fc070c10 2024-09-13T16:25:37.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7fa6faffd700 1 -- 172.21.15.119:0/167006383 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0036c0 con 0x7fa6fc070c10 2024-09-13T16:25:37.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d69b1c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4d64143f50 0x7f4d640cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d69b1c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d6406dda0 0x7f4d64070c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d69b1c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d6415aca0 con 0x7f4d64071150 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d69b1c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d640715c0 con 0x7f4d64143f50 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d69b1c700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d640b1920 con 0x7f4d6406dda0 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d6931b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4d64071150 0x7f4d6415e290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33812/0 (socket says 172.21.15.119:33812) 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d68b1a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4d64143f50 0x7f4d640cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d6931b700 1 -- 172.21.15.119:0/3004074035 learned_addr learned my addr 172.21.15.119:0/3004074035 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d68b1a700 1 -- 172.21.15.119:0/3004074035 >> v2:172.21.15.145:3300/0 conn(0x7f4d6406dda0 msgr2=0x7f4d64070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d63fff700 1 --2- 172.21.15.119:0/3004074035 >> v2:172.21.15.145:3300/0 conn(0x7f4d6406dda0 0x7f4d64070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d68b1a700 1 --2- 172.21.15.119:0/3004074035 >> v2:172.21.15.145:3300/0 conn(0x7f4d6406dda0 0x7f4d64070c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d68b1a700 1 -- 172.21.15.119:0/3004074035 >> v1:172.21.15.119:6789/0 conn(0x7f4d64071150 legacy=0x7f4d6415e290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d68b1a700 1 -- 172.21.15.119:0/3004074035 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d640b1c00 con 0x7f4d64143f50 2024-09-13T16:25:37.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d63fff700 1 --2- 172.21.15.119:0/3004074035 >> v2:172.21.15.145:3300/0 conn(0x7f4d6406dda0 0x7f4d64070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:37.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7fa7013d5700 1 -- 172.21.15.119:0/167006383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 msgr2=0x7fa6fc071080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7fa7013d5700 1 --2- 172.21.15.119:0/167006383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc071080 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec009850 tx=0x7fa6ec009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:37.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f9b98ff9700 1 -- 172.21.15.119:0/1467896396 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9b94012400 con 0x7f9b9c15a250 2024-09-13T16:25:37.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f9b98ff9700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b88066780 0x7f9b88068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f9b98ff9700 1 -- 172.21.15.119:0/1467896396 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9b94083ab0 con 0x7f9b9c15a250 2024-09-13T16:25:37.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f4d68b1a700 1 --2- 172.21.15.119:0/3004074035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d64143f50 0x7f4d640cbc30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4d640cd490 tx=0x7f4d54009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd958226eed4f7a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f9b9bfff700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b88066780 0x7f9b88068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.517+0000 7f9b9bfff700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b88066780 0x7f9b88068c40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009ab0 tx=0x7f9b90008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- 172.21.15.119:0/167006383 shutdown_connections 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 --2- 172.21.15.119:0/167006383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc071080 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 --2- 172.21.15.119:0/167006383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a630 0x7fa6fc06e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- 172.21.15.119:0/167006383 >> 172.21.15.119:0/167006383 conn(0x7fa6fc0b04f0 msgr2=0x7fa6fc0b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- 172.21.15.119:0/167006383 shutdown_connections 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- 172.21.15.119:0/167006383 wait complete. 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7f2ab4ff9700 1 -- 172.21.15.119:0/895860285 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0004440 con 0x7f2ab8070c10 2024-09-13T16:25:37.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7f2ab4ff9700 1 -- 172.21.15.119:0/895860285 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ab00045a0 con 0x7f2ab8070c10 2024-09-13T16:25:37.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7f2ab4ff9700 1 -- 172.21.15.119:0/895860285 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ab0004950 con 0x7f2ab8070c10 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 Processor -- start 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- start start 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a250 0x7fa6fc0af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6fc15a630 0x7fa6fc0af910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc159480 con 0x7fa6fc070c10 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc1f61d0 con 0x7fa6fc15a630 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa7013d5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6fc1f64b0 con 0x7fa6fc15a250 2024-09-13T16:25:37.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc158f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52426/0 (socket says 172.21.15.119:52426) 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 -- 172.21.15.119:0/74934781 learned_addr learned my addr 172.21.15.119:0/74934781 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa700bd4700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6fc15a630 0x7fa6fc0af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 -- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a250 msgr2=0x7fa6fc0af3d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a250 0x7fa6fc0af3d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 -- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6fc15a630 msgr2=0x7fa6fc0af910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6fc15a630 0x7fa6fc0af910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.521+0000 7fa6fbfff700 1 -- 172.21.15.119:0/74934781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec0094b0 con 0x7fa6fc070c10 2024-09-13T16:25:37.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7fa6fbfff700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc158f40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000c780 tx=0x7fa6f000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d637fe700 1 -- 172.21.15.119:0/3004074035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54011830 con 0x7f4d64143f50 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d637fe700 1 -- 172.21.15.119:0/3004074035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d54011990 con 0x7f4d64143f50 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d637fe700 1 -- 172.21.15.119:0/3004074035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54011d40 con 0x7f4d64143f50 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 -- 172.21.15.119:0/3004074035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d64143f50 msgr2=0x7f4d640cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/3004074035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d64143f50 0x7f4d640cbc30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f4d640cd490 tx=0x7f4d54009b30 comp rx=0 tx=0).stop 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 -- 172.21.15.119:0/3004074035 shutdown_connections 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/3004074035 >> v2:172.21.15.145:3300/0 conn(0x7f4d6406dda0 0x7f4d64070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/3004074035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d64143f50 0x7f4d640cbc30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 -- 172.21.15.119:0/3004074035 >> 172.21.15.119:0/3004074035 conn(0x7f4d640b04f0 msgr2=0x7f4d640b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 -- 172.21.15.119:0/3004074035 shutdown_connections 2024-09-13T16:25:37.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 -- 172.21.15.119:0/3004074035 wait complete. 2024-09-13T16:25:37.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 Processor -- start 2024-09-13T16:25:37.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.525+0000 7f4d69b1c700 1 -- start start 2024-09-13T16:25:37.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab815d080 con 0x7f2ab8070c10 2024-09-13T16:25:37.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab815d640 con 0x7f2ab8070c10 2024-09-13T16:25:37.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c151250 con 0x7f9b9c15a250 2024-09-13T16:25:37.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f2ab4ff9700 1 -- 172.21.15.119:0/895860285 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2ab0012430 con 0x7f2ab8070c10 2024-09-13T16:25:37.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f2ab4ff9700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa40667b0 0x7f2aa4068c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f2ab4ff9700 1 -- 172.21.15.119:0/895860285 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2ab0085ab0 con 0x7f2ab8070c10 2024-09-13T16:25:37.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f2ab77fe700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa40667b0 0x7f2aa4068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.529+0000 7f2ab77fe700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa40667b0 0x7f2aa4068c70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8151410 tx=0x7f2aa80090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.533+0000 7fa6f8ff9700 1 -- 172.21.15.119:0/74934781 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0004ca0 con 0x7fa6fc070c10 2024-09-13T16:25:37.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.533+0000 7fa6f8ff9700 1 -- 172.21.15.119:0/74934781 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6f0004e00 con 0x7fa6fc070c10 2024-09-13T16:25:37.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.533+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6fc1f6790 con 0x7fa6fc070c10 2024-09-13T16:25:37.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.533+0000 7fa6f8ff9700 1 -- 172.21.15.119:0/74934781 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6f0007310 con 0x7fa6fc070c10 2024-09-13T16:25:37.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.533+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6fc14e600 con 0x7fa6fc070c10 2024-09-13T16:25:37.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d69b1c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d6406dda0 0x7f4d6415a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d69b1c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d64071150 0x7f4d6415d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d69b1c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d64143f50 0x7f4d6415d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d69b1c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6413a9e0 con 0x7f4d6406dda0 2024-09-13T16:25:37.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d69b1c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d64139980 con 0x7f4d64143f50 2024-09-13T16:25:37.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab80cda80 con 0x7f2ab8070c10 2024-09-13T16:25:37.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d6406dda0 0x7f4d6415a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d69b1c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d64139c20 con 0x7f4d64071150 2024-09-13T16:25:37.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d6406dda0 0x7f4d6415a0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52442/0 (socket says 172.21.15.119:52442) 2024-09-13T16:25:37.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 -- 172.21.15.119:0/148318000 learned_addr learned my addr 172.21.15.119:0/148318000 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d6931b700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d64143f50 0x7f4d6415d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 -- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d64071150 msgr2=0x7f4d6415d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d63fff700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d64071150 0x7f4d6415d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d64071150 0x7f4d6415d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 -- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d64143f50 msgr2=0x7f4d6415d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d64143f50 0x7f4d6415d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 -- 172.21.15.119:0/148318000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d540094b0 con 0x7f4d6406dda0 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d63fff700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d64071150 0x7f4d6415d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f4d68b1a700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d6406dda0 0x7f4d6415a0a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d54000c00 tx=0x7f4d54004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7f9b98ff9700 1 -- 172.21.15.119:0/1467896396 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9b9c151250 con 0x7f9b9c15a250 2024-09-13T16:25:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d8005310 con 0x7fa6fc070c10 2024-09-13T16:25:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7fa6f8ff9700 1 -- 172.21.15.119:0/74934781 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa6f0007470 con 0x7fa6fc070c10 2024-09-13T16:25:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7fa6f8ff9700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa6e0066780 0x7fa6e0068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.537+0000 7fa6fb7fe700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa6e0066780 0x7fa6e0068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.541+0000 7fa6fb7fe700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa6e0066780 0x7fa6e0068c40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc15be50 tx=0x7fa6f400d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.545+0000 7f2ab4ff9700 1 -- 172.21.15.119:0/895860285 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2ab0052ca0 con 0x7f2ab8070c10 2024-09-13T16:25:37.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.545+0000 7f4d617fa700 1 -- 172.21.15.119:0/148318000 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d54004610 con 0x7f4d6406dda0 2024-09-13T16:25:37.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.545+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d64139f00 con 0x7f4d6406dda0 2024-09-13T16:25:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.545+0000 7fa6f8ff9700 1 -- 172.21.15.119:0/74934781 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa6f0007720 con 0x7fa6fc070c10 2024-09-13T16:25:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.545+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6413a4c0 con 0x7f4d6406dda0 2024-09-13T16:25:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.549+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d6413a6b0 con 0x7f4d6406dda0 2024-09-13T16:25:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.553+0000 7fa6f8ff9700 1 -- 172.21.15.119:0/74934781 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa6f0054990 con 0x7fa6fc070c10 2024-09-13T16:25:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.553+0000 7f4d617fa700 1 -- 172.21.15.119:0/148318000 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5401f070 con 0x7f4d6406dda0 2024-09-13T16:25:37.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.553+0000 7f4d617fa700 1 -- 172.21.15.119:0/148318000 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d540038c0 con 0x7f4d6406dda0 2024-09-13T16:25:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.561+0000 7f4d617fa700 1 -- 172.21.15.119:0/148318000 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4d54003a20 con 0x7f4d6406dda0 2024-09-13T16:25:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.561+0000 7f4d617fa700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d50066730 0x7f4d50068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.561+0000 7f4d617fa700 1 -- 172.21.15.119:0/148318000 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4d54021070 con 0x7f4d6406dda0 2024-09-13T16:25:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.561+0000 7f4d63fff700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d50066730 0x7f4d50068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.561+0000 7f4d63fff700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d50066730 0x7f4d50068bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4d6406c6f0 tx=0x7f4d58008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.565+0000 7f4d617fa700 1 -- 172.21.15.119:0/148318000 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4d54053030 con 0x7f4d6406dda0 2024-09-13T16:25:37.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.749+0000 7f97f5829700 1 -- 172.21.15.119:0/2186027011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f97dc0055c0 con 0x7f97f0070c10 2024-09-13T16:25:37.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.749+0000 7f97d7fff700 1 -- 172.21.15.119:0/2186027011 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f97e4015030 con 0x7f97f0070c10 2024-09-13T16:25:37.757 INFO:teuthology.orchestra.run.smithi119.stdout:60129542149 2024-09-13T16:25:37.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 -- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97d8066a20 msgr2=0x7f97d8068ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97d8066a20 0x7f97d8068ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f97f006df60 tx=0x7f97e0008040 comp rx=0 tx=0).stop 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 -- 172.21.15.119:0/2186027011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f0070c10 msgr2=0x7f97f014a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f0070c10 0x7f97f014a5b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f97e400b520 tx=0x7f97e400b8e0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 -- 172.21.15.119:0/2186027011 shutdown_connections 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f97d8066a20 0x7f97d8068ee0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f97f0153100 0x7f97f0157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f97f0152d20 0x7f97f006d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 --2- 172.21.15.119:0/2186027011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f97f0070c10 0x7f97f014a5b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 -- 172.21.15.119:0/2186027011 >> 172.21.15.119:0/2186027011 conn(0x7f97f00b04f0 msgr2=0x7f97f00ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 -- 172.21.15.119:0/2186027011 shutdown_connections 2024-09-13T16:25:37.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.757+0000 7f97d5ffb700 1 -- 172.21.15.119:0/2186027011 wait complete. 2024-09-13T16:25:37.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.761+0000 7fedeeffd700 1 -- 172.21.15.119:0/1658906967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fede8005890 con 0x7fee08152d20 2024-09-13T16:25:37.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.761+0000 7fee057fa700 1 -- 172.21.15.119:0/1658906967 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fedf8089060 con 0x7fee08152d20 2024-09-13T16:25:37.769 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:37.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fedf4066880 msgr2=0x7fedf4068d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fedf4066880 0x7fedf4068d40 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b9b0 tx=0x7fedfc005da0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08152d20 msgr2=0x7fee08163b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08152d20 0x7fee08163b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fedf8009fd0 tx=0x7fedf800ebd0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 shutdown_connections 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fedf4066880 0x7fedf4068d40 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fee08153100 0x7fee08162850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fee08152d20 0x7fee08163b80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 --2- 172.21.15.119:0/1658906967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fee08070c10 0x7fee081f5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 >> 172.21.15.119:0/1658906967 conn(0x7fee080b04f0 msgr2=0x7fee08158620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 shutdown_connections 2024-09-13T16:25:37.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.769+0000 7fee0ef09700 1 -- 172.21.15.119:0/1658906967 wait complete. 2024-09-13T16:25:37.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.6 2024-09-13T16:25:37.787 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.5 2024-09-13T16:25:37.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.805+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f7b2c005890 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.805+0000 7f7b497fa700 1 -- 172.21.15.119:0/1623652968 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7b4402b030 con 0x7f7b4c06cbd0 2024-09-13T16:25:37.809 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:37.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b34066b90 msgr2=0x7f7b34069050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b34066b90 0x7f7b34069050 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c1566b0 tx=0x7f7b40009040 comp rx=0 tx=0).stop 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c06cbd0 msgr2=0x7f7b4c1524a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c06cbd0 0x7f7b4c1524a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f7b4400b370 tx=0x7f7b44004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 shutdown_connections 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b34066b90 0x7f7b34069050 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b4c1f9de0 0x7f7b4c0b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b4c0b17f0 0x7f7b4c15e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 --2- 172.21.15.119:0/1623652968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b4c06cbd0 0x7f7b4c1524a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 >> 172.21.15.119:0/1623652968 conn(0x7f7b4c0b04f0 msgr2=0x7f7b4c15ff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 shutdown_connections 2024-09-13T16:25:37.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.809+0000 7f7b51b0f700 1 -- 172.21.15.119:0/1623652968 wait complete. 2024-09-13T16:25:37.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.817+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f2ab806be40 con 0x7f2ab8070c10 2024-09-13T16:25:37.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.817+0000 7f2ab4ff9700 1 -- 172.21.15.119:0/895860285 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2ab0017070 con 0x7f2ab8070c10 2024-09-13T16:25:37.822 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:37.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa40667b0 msgr2=0x7f2aa4068c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa40667b0 0x7f2aa4068c70 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2ab8151410 tx=0x7f2aa80090e0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 msgr2=0x7f2ab814e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab814e190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2ab000b730 tx=0x7f2ab0004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 shutdown_connections 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa40667b0 0x7f2aa4068c70 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab8150640 0x7f2ab8150ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab8159d80 0x7f2ab815c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 --2- 172.21.15.119:0/895860285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab8070c10 0x7f2ab814e190 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 >> 172.21.15.119:0/895860285 conn(0x7f2ab80b04f0 msgr2=0x7f2ab80b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 shutdown_connections 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f2abd4e9700 1 -- 172.21.15.119:0/895860285 wait complete. 2024-09-13T16:25:37.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f95b01637a0 con 0x7f95b0161230 2024-09-13T16:25:37.826 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.0 2024-09-13T16:25:37.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.821+0000 7f95a67fc700 1 -- 172.21.15.119:0/2924609160 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f95b01637a0 con 0x7f95b0161230 2024-09-13T16:25:37.827 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:37.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9594066780 msgr2=0x7f9594068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9594066780 0x7f9594068c40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f95b00b1340 tx=0x7f95ac00a000 comp rx=0 tx=0).stop 2024-09-13T16:25:37.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95b0161230 msgr2=0x7f95b0163a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95b0161230 0x7f95b0163a80 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f95a80021a0 tx=0x7f95a800d430 comp rx=0 tx=0).stop 2024-09-13T16:25:37.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 shutdown_connections 2024-09-13T16:25:37.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9594066780 0x7f9594068c40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95b0162040 0x7f95b01f6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95b0161230 0x7f95b0163a80 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 --2- 172.21.15.119:0/2924609160 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f959c004c40 0x7f95b015b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 >> 172.21.15.119:0/2924609160 conn(0x7f95b00b04f0 msgr2=0x7f95b00af9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 shutdown_connections 2024-09-13T16:25:37.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.825+0000 7f95b7d58700 1 -- 172.21.15.119:0/2924609160 wait complete. 2024-09-13T16:25:37.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.829+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fa6d8005890 con 0x7fa6fc070c10 2024-09-13T16:25:37.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.829+0000 7fa6f8ff9700 1 -- 172.21.15.119:0/74934781 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa6f0018030 con 0x7fa6fc070c10 2024-09-13T16:25:37.833 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:37.836 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.2 2024-09-13T16:25:37.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa6e0066780 msgr2=0x7fa6e0068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa6e0066780 0x7fa6e0068c40 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc15be50 tx=0x7fa6f400d040 comp rx=0 tx=0).stop 2024-09-13T16:25:37.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 msgr2=0x7fa6fc158f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc158f40 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa6f000c780 tx=0x7fa6f000cb40 comp rx=0 tx=0).stop 2024-09-13T16:25:37.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 shutdown_connections 2024-09-13T16:25:37.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa6e0066780 0x7fa6e0068c40 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa6fc15a630 0x7fa6fc0af910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa6fc15a250 0x7fa6fc0af3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 --2- 172.21.15.119:0/74934781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa6fc070c10 0x7fa6fc158f40 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 >> 172.21.15.119:0/74934781 conn(0x7fa6fc0b04f0 msgr2=0x7fa6fc0ad480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 shutdown_connections 2024-09-13T16:25:37.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.833+0000 7fa7013d5700 1 -- 172.21.15.119:0/74934781 wait complete. 2024-09-13T16:25:37.846 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.3 2024-09-13T16:25:37.847 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.1 2024-09-13T16:25:37.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.849+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f9b9c151250 con 0x7f9b9c15a250 2024-09-13T16:25:37.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.849+0000 7f9b98ff9700 1 -- 172.21.15.119:0/1467896396 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f9b9c151250 con 0x7f9b9c15a250 2024-09-13T16:25:37.853 INFO:teuthology.orchestra.run.smithi119.stdout:60129542150 2024-09-13T16:25:37.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b88066780 msgr2=0x7f9b88068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b88066780 0x7f9b88068c40 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9b90009ab0 tx=0x7f9b90008040 comp rx=0 tx=0).stop 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c15a250 msgr2=0x7f9b9c1541c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c15a250 0x7f9b9c1541c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b9400ca70 tx=0x7f9b9400caa0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 shutdown_connections 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b88066780 0x7f9b88068c40 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b9c15a630 0x7f9b9c1575f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b9c15a250 0x7f9b9c1541c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 --2- 172.21.15.119:0/1467896396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b9c070c10 0x7f9b9c1461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 >> 172.21.15.119:0/1467896396 conn(0x7f9b9c0b04f0 msgr2=0x7f9b9c066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 shutdown_connections 2024-09-13T16:25:37.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.853+0000 7f9ba1241700 1 -- 172.21.15.119:0/1467896396 wait complete. 2024-09-13T16:25:37.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.857+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f4d64144330 con 0x7f4d6406dda0 2024-09-13T16:25:37.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.857+0000 7f4d617fa700 1 -- 172.21.15.119:0/148318000 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d54015020 con 0x7f4d6406dda0 2024-09-13T16:25:37.861 INFO:teuthology.orchestra.run.smithi119.stdout:60129542149 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d50066730 msgr2=0x7f4d50068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d50066730 0x7f4d50068bf0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f4d6406c6f0 tx=0x7f4d58008040 comp rx=0 tx=0).stop 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d6406dda0 msgr2=0x7f4d6415a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d6406dda0 0x7f4d6415a0a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4d54000c00 tx=0x7f4d54004770 comp rx=0 tx=0).stop 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 shutdown_connections 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d50066730 0x7f4d50068bf0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d64143f50 0x7f4d6415d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d64071150 0x7f4d6415d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 --2- 172.21.15.119:0/148318000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d6406dda0 0x7f4d6415a0a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 >> 172.21.15.119:0/148318000 conn(0x7f4d640b04f0 msgr2=0x7f4d64142f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:37.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 shutdown_connections 2024-09-13T16:25:37.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:37.861+0000 7f4d69b1c700 1 -- 172.21.15.119:0/148318000 wait complete. 2024-09-13T16:25:37.865 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.7 2024-09-13T16:25:37.877 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.4 2024-09-13T16:25:38.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-13T16:25:38.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-13T16:25:38.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-13T16:25:38.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-13T16:25:38.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-13T16:25:38.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-13T16:25:38.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-13T16:25:38.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-13T16:25:38.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c18d3700 1 Processor -- start 2024-09-13T16:25:38.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c18d3700 1 -- start start 2024-09-13T16:25:38.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c18d3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe7bc071320 0x7fe7bc159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c18d3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc0b2000 con 0x7fe7bc153100 2024-09-13T16:25:38.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c18d3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7bc0b22e0 con 0x7fe7bc152d20 2024-09-13T16:25:38.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c18d3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7bc0b28a0 con 0x7fe7bc071320 2024-09-13T16:25:38.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7bbfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe7bc153100 0x7fe7bc070c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33820/0 (socket says 172.21.15.119:33820) 2024-09-13T16:25:38.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c10d2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe7bc071320 0x7fe7bc159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7bbfff700 1 -- 172.21.15.119:0/1735907616 learned_addr learned my addr 172.21.15.119:0/1735907616 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c10d2700 1 -- 172.21.15.119:0/1735907616 >> v1:172.21.15.145:6789/0 conn(0x7fe7bc152d20 legacy=0x7fe7bc0b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7bb7fe700 1 -- 172.21.15.119:0/1735907616 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 145903716 0 0) 0x7fe7bc0b2000 con 0x7fe7bc153100 2024-09-13T16:25:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c10d2700 1 -- 172.21.15.119:0/1735907616 >> v1:172.21.15.119:6789/0 conn(0x7fe7bc153100 legacy=0x7fe7bc070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c10d2700 1 -- 172.21.15.119:0/1735907616 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7bc0b25c0 con 0x7fe7bc071320 2024-09-13T16:25:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.897+0000 7fe7c10d2700 1 --2- 172.21.15.119:0/1735907616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc071320 0x7fe7bc159a40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40061f0 tx=0x7fe7b4006500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4a4f1893f6c207 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.901+0000 7fe7bb7fe700 1 -- 172.21.15.119:0/1735907616 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b4003c40 con 0x7fe7bc071320 2024-09-13T16:25:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.901+0000 7fe7bb7fe700 1 -- 172.21.15.119:0/1735907616 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b4003da0 con 0x7fe7bc071320 2024-09-13T16:25:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.901+0000 7fe7c18d3700 1 -- 172.21.15.119:0/1735907616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc071320 msgr2=0x7fe7bc159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.901+0000 7fe7c18d3700 1 --2- 172.21.15.119:0/1735907616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc071320 0x7fe7bc159a40 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fe7b40061f0 tx=0x7fe7b4006500 comp rx=0 tx=0).stop 2024-09-13T16:25:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.901+0000 7fe7c18d3700 1 -- 172.21.15.119:0/1735907616 shutdown_connections 2024-09-13T16:25:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.901+0000 7fe7c18d3700 1 --2- 172.21.15.119:0/1735907616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc071320 0x7fe7bc159a40 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.901+0000 7fe7c18d3700 1 -- 172.21.15.119:0/1735907616 >> 172.21.15.119:0/1735907616 conn(0x7fe7bc0b04f0 msgr2=0x7fe7bc0b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.905+0000 7fe7c18d3700 1 -- 172.21.15.119:0/1735907616 shutdown_connections 2024-09-13T16:25:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.905+0000 7fe7c18d3700 1 -- 172.21.15.119:0/1735907616 wait complete. 2024-09-13T16:25:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 Processor -- start 2024-09-13T16:25:38.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 -- start start 2024-09-13T16:25:38.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb9759e700 1 Processor -- start 2024-09-13T16:25:38.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7bc152d20 0x7fe7bc150f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7bc153100 0x7fe7bc0abd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc154780 0x7fe7bc14f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc150a90 con 0x7fe7bc154780 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc0ac3e0 con 0x7fe7bc153100 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c18d3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7bc154c10 con 0x7fe7bc152d20 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc154780 0x7fe7bc14f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc154780 0x7fe7bc14f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52472/0 (socket says 172.21.15.119:52472) 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 -- 172.21.15.119:0/2527230639 learned_addr learned my addr 172.21.15.119:0/2527230639 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7bbfff700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7bc153100 0x7fe7bc0abd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c08d1700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7bc152d20 0x7fe7bc150f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb9759e700 1 -- start start 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb9759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcb98071320 0x7fcb98159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb9759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980b2000 con 0x7fcb98152d20 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb9759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb980b22e0 con 0x7fcb98153100 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb9759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb980b28a0 con 0x7fcb98071320 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb96d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcb98071320 0x7fcb98159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 -- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7bc152d20 msgr2=0x7fe7bc150f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7bc152d20 0x7fe7bc150f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 -- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7bc153100 msgr2=0x7fe7bc0abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7bc153100 0x7fe7bc0abd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 -- 172.21.15.119:0/2527230639 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b4005e20 con 0x7fe7bc154780 2024-09-13T16:25:38.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb96d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcb98071320 0x7fcb98159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52478/0 (socket says 172.21.15.119:52478) 2024-09-13T16:25:38.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb96d9d700 1 -- 172.21.15.119:0/2913804042 learned_addr learned my addr 172.21.15.119:0/2913804042 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:38.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fe7c10d2700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc154780 0x7fe7bc14f850 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4006620 tx=0x7fe7b4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb96d9d700 1 -- 172.21.15.119:0/2913804042 >> v1:172.21.15.145:6789/0 conn(0x7fcb98153100 legacy=0x7fcb98070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb96d9d700 1 -- 172.21.15.119:0/2913804042 >> v1:172.21.15.119:6789/0 conn(0x7fcb98152d20 legacy=0x7fcb980b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.909+0000 7fcb96d9d700 1 -- 172.21.15.119:0/2913804042 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb980b25c0 con 0x7fcb98071320 2024-09-13T16:25:38.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fcb96d9d700 1 --2- 172.21.15.119:0/2913804042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98071320 0x7fcb98159a40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009730 tx=0x7fcb90009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0d0eab76482300b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7b97fa700 1 -- 172.21.15.119:0/2527230639 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b400d030 con 0x7fe7bc154780 2024-09-13T16:25:38.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7b97fa700 1 -- 172.21.15.119:0/2527230639 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b4004540 con 0x7fe7bc154780 2024-09-13T16:25:38.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7b97fa700 1 -- 172.21.15.119:0/2527230639 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b401e070 con 0x7fe7bc154780 2024-09-13T16:25:38.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7c18d3700 1 -- 172.21.15.119:0/2527230639 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7bc158ad0 con 0x7fe7bc154780 2024-09-13T16:25:38.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7c18d3700 1 -- 172.21.15.119:0/2527230639 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7bc150160 con 0x7fe7bc154780 2024-09-13T16:25:38.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7b97fa700 1 -- 172.21.15.119:0/2527230639 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe7b40193f0 con 0x7fe7bc154780 2024-09-13T16:25:38.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7b97fa700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7a8066780 0x7fe7a8068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.913+0000 7fe7b97fa700 1 -- 172.21.15.119:0/2527230639 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe7b40856d0 con 0x7fe7bc154780 2024-09-13T16:25:38.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.917+0000 7fe7c08d1700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7a8066780 0x7fe7a8068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.917+0000 7fe7c08d1700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7a8066780 0x7fe7a8068c40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0005150 tx=0x7fe7b0007ee0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9559a700 1 -- 172.21.15.119:0/2913804042 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9000c920 con 0x7fcb98071320 2024-09-13T16:25:38.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9559a700 1 -- 172.21.15.119:0/2913804042 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9000ca80 con 0x7fcb98071320 2024-09-13T16:25:38.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9559a700 1 -- 172.21.15.119:0/2913804042 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb9000ce10 con 0x7fcb98071320 2024-09-13T16:25:38.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fe7c18d3700 1 -- 172.21.15.119:0/2527230639 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe79c005310 con 0x7fe7bc154780 2024-09-13T16:25:38.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9759e700 1 -- 172.21.15.119:0/2913804042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98071320 msgr2=0x7fcb98159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9759e700 1 --2- 172.21.15.119:0/2913804042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98071320 0x7fcb98159a40 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009730 tx=0x7fcb90009760 comp rx=0 tx=0).stop 2024-09-13T16:25:38.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9759e700 1 -- 172.21.15.119:0/2913804042 shutdown_connections 2024-09-13T16:25:38.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9759e700 1 --2- 172.21.15.119:0/2913804042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98071320 0x7fcb98159a40 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9759e700 1 -- 172.21.15.119:0/2913804042 >> 172.21.15.119:0/2913804042 conn(0x7fcb980b04f0 msgr2=0x7fcb980b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:38.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9759e700 1 -- 172.21.15.119:0/2913804042 shutdown_connections 2024-09-13T16:25:38.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.921+0000 7fcb9759e700 1 -- 172.21.15.119:0/2913804042 wait complete. 2024-09-13T16:25:38.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 Processor -- start 2024-09-13T16:25:38.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 -- start start 2024-09-13T16:25:38.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb98071320 0x7fcb98141d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb98152d20 0x7fcb9815a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98153100 0x7fcb981580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98142690 con 0x7fcb98153100 2024-09-13T16:25:38.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb96d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98153100 0x7fcb981580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb96d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98153100 0x7fcb981580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52490/0 (socket says 172.21.15.119:52490) 2024-09-13T16:25:38.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb96d9d700 1 -- 172.21.15.119:0/3654553664 learned_addr learned my addr 172.21.15.119:0/3654553664 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:38.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb95d9b700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb98152d20 0x7fcb9815a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 -- 172.21.15.119:0/3654553664 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98142970 con 0x7fcb98071320 2024-09-13T16:25:38.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.925+0000 7fcb9759e700 1 -- 172.21.15.119:0/3654553664 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb98142c50 con 0x7fcb98152d20 2024-09-13T16:25:38.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb96d9d700 1 -- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb98152d20 msgr2=0x7fcb9815a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb96d9d700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb98152d20 0x7fcb9815a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb96d9d700 1 -- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb98071320 msgr2=0x7fcb98141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb9659c700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb98071320 0x7fcb98141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb96d9d700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb98071320 0x7fcb98141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb96d9d700 1 -- 172.21.15.119:0/3654553664 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9000c3d0 con 0x7fcb98153100 2024-09-13T16:25:38.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb96d9d700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98153100 0x7fcb981580c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009b60 tx=0x7fcb90004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb9659c700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb98071320 0x7fcb98141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:38.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb86ffd700 1 -- 172.21.15.119:0/3654553664 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb90010640 con 0x7fcb98153100 2024-09-13T16:25:38.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb86ffd700 1 -- 172.21.15.119:0/3654553664 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb9001e040 con 0x7fcb98153100 2024-09-13T16:25:38.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb86ffd700 1 -- 172.21.15.119:0/3654553664 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb900109c0 con 0x7fcb98153100 2024-09-13T16:25:38.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fe7b97fa700 1 -- 172.21.15.119:0/2527230639 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe7b40528c0 con 0x7fe7bc154780 2024-09-13T16:25:38.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb9759e700 1 -- 172.21.15.119:0/3654553664 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb98142f30 con 0x7fcb98153100 2024-09-13T16:25:38.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.929+0000 7fcb9759e700 1 -- 172.21.15.119:0/3654553664 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb981edf70 con 0x7fcb98153100 2024-09-13T16:25:38.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.933+0000 7fcb86ffd700 1 -- 172.21.15.119:0/3654553664 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcb90010b20 con 0x7fcb98153100 2024-09-13T16:25:38.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.933+0000 7fcb86ffd700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcb800668d0 0x7fcb80068d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.933+0000 7fcb86ffd700 1 -- 172.21.15.119:0/3654553664 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fcb90014070 con 0x7fcb98153100 2024-09-13T16:25:38.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.933+0000 7fcb9659c700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcb800668d0 0x7fcb80068d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.933+0000 7fcb9659c700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcb800668d0 0x7fcb80068d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c001720 tx=0x7fcb8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.937+0000 7fcb9759e700 1 -- 172.21.15.119:0/3654553664 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb7c005310 con 0x7fcb98153100 2024-09-13T16:25:38.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.945+0000 7fcb86ffd700 1 -- 172.21.15.119:0/3654553664 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcb90054e10 con 0x7fcb98153100 2024-09-13T16:25:38.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce923700 1 Processor -- start 2024-09-13T16:25:38.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce923700 1 -- start start 2024-09-13T16:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce923700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f13c8160c60 0x7f13c80b38a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce923700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f13c80b17f0 0x7f13c80b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce923700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13c80b2160 con 0x7f13c806cbd0 2024-09-13T16:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce923700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c806d930 con 0x7f13c8160c60 2024-09-13T16:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce923700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13c8164850 con 0x7f13c80b17f0 2024-09-13T16:25:38.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd120700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f13c806cbd0 0x7f13c8164f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33852/0 (socket says 172.21.15.119:33852) 2024-09-13T16:25:38.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd120700 1 -- 172.21.15.119:0/4120994364 learned_addr learned my addr 172.21.15.119:0/4120994364 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:38.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd921700 1 --2- 172.21.15.119:0/4120994364 >> v2:172.21.15.119:3300/0 conn(0x7f13c8160c60 0x7f13c80b38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce122700 1 --2- 172.21.15.119:0/4120994364 >> v2:172.21.15.145:3300/0 conn(0x7f13c80b17f0 0x7f13c80b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cc91f700 1 -- 172.21.15.119:0/4120994364 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3565475421 0 0) 0x7f13c80b2160 con 0x7f13c806cbd0 2024-09-13T16:25:38.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cc91f700 1 -- 172.21.15.119:0/4120994364 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13b40035d0 con 0x7f13c806cbd0 2024-09-13T16:25:38.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd921700 1 -- 172.21.15.119:0/4120994364 >> v2:172.21.15.145:3300/0 conn(0x7f13c80b17f0 msgr2=0x7f13c80b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:38.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd921700 1 --2- 172.21.15.119:0/4120994364 >> v2:172.21.15.145:3300/0 conn(0x7f13c80b17f0 0x7f13c80b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd921700 1 -- 172.21.15.119:0/4120994364 >> v1:172.21.15.119:6789/0 conn(0x7f13c806cbd0 legacy=0x7f13c8164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd921700 1 -- 172.21.15.119:0/4120994364 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13c8164af0 con 0x7f13c8160c60 2024-09-13T16:25:38.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13ce122700 1 --2- 172.21.15.119:0/4120994364 >> v2:172.21.15.145:3300/0 conn(0x7f13c80b17f0 0x7f13c80b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:38.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cd921700 1 --2- 172.21.15.119:0/4120994364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c80b38a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f13c8071c40 tx=0x7f13b8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8f70f95a467b267 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cc91f700 1 -- 172.21.15.119:0/4120994364 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b8004230 con 0x7f13c8160c60 2024-09-13T16:25:38.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cc91f700 1 -- 172.21.15.119:0/4120994364 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13b8004390 con 0x7f13c8160c60 2024-09-13T16:25:38.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.969+0000 7f13cc91f700 1 -- 172.21.15.119:0/4120994364 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13b80036c0 con 0x7f13c8160c60 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.973+0000 7f13ce923700 1 -- 172.21.15.119:0/4120994364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 msgr2=0x7f13c80b38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.973+0000 7f13ce923700 1 --2- 172.21.15.119:0/4120994364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c80b38a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f13c8071c40 tx=0x7f13b8009b30 comp rx=0 tx=0).stop 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- 172.21.15.119:0/4120994364 shutdown_connections 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 --2- 172.21.15.119:0/4120994364 >> v2:172.21.15.145:3300/0 conn(0x7f13c80b17f0 0x7f13c80b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 --2- 172.21.15.119:0/4120994364 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c80b38a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- 172.21.15.119:0/4120994364 >> 172.21.15.119:0/4120994364 conn(0x7f13c80b04f0 msgr2=0x7f13c80b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- 172.21.15.119:0/4120994364 shutdown_connections 2024-09-13T16:25:38.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- 172.21.15.119:0/4120994364 wait complete. 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 Processor -- start 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- start start 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c806cbd0 0x7f13c81494e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c80b17f0 0x7f13c8148710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c814b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c8148c50 con 0x7f13c8160c60 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c8148e80 con 0x7f13c80b17f0 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce923700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13c81612f0 con 0x7f13c806cbd0 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c814b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c814b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52520/0 (socket says 172.21.15.119:52520) 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 -- 172.21.15.119:0/1852156324 learned_addr learned my addr 172.21.15.119:0/1852156324 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 -- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c806cbd0 msgr2=0x7f13c81494e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13cd120700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c80b17f0 0x7f13c8148710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13cd921700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c806cbd0 0x7f13c81494e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c806cbd0 0x7f13c81494e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 -- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c80b17f0 msgr2=0x7f13c8148710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c80b17f0 0x7f13c8148710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 -- 172.21.15.119:0/1852156324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b80094b0 con 0x7f13c8160c60 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13cd120700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c80b17f0 0x7f13c8148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13cd921700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c806cbd0 0x7f13c81494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.977+0000 7f13ce122700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c814b0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009fa0 tx=0x7f13c000d460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.981+0000 7f13be7fc700 1 -- 172.21.15.119:0/1852156324 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0003a40 con 0x7f13c8160c60 2024-09-13T16:25:38.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.981+0000 7f13be7fc700 1 -- 172.21.15.119:0/1852156324 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13c000f040 con 0x7f13c8160c60 2024-09-13T16:25:38.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.981+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c81615d0 con 0x7f13c8160c60 2024-09-13T16:25:38.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.981+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c8161b90 con 0x7f13c8160c60 2024-09-13T16:25:38.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.981+0000 7f13be7fc700 1 -- 172.21.15.119:0/1852156324 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13c0003e00 con 0x7f13c8160c60 2024-09-13T16:25:38.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.985+0000 7f13be7fc700 1 -- 172.21.15.119:0/1852156324 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f13c0004030 con 0x7f13c8160c60 2024-09-13T16:25:38.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.985+0000 7f13be7fc700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f13ac066800 0x7f13ac068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:38.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.985+0000 7f13be7fc700 1 -- 172.21.15.119:0/1852156324 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f13c0084680 con 0x7f13c8160c60 2024-09-13T16:25:38.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.985+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c8161db0 con 0x7f13c8160c60 2024-09-13T16:25:38.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.985+0000 7f13cd921700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f13ac066800 0x7f13ac068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:38.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.989+0000 7f13cd921700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f13ac066800 0x7f13ac068cc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f13c8071c40 tx=0x7f13b8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:38.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:38.993+0000 7f13be7fc700 1 -- 172.21.15.119:0/1852156324 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f13c00516e0 con 0x7f13c8160c60 2024-09-13T16:25:39.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5f54700 1 Processor -- start 2024-09-13T16:25:39.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5f54700 1 -- start start 2024-09-13T16:25:39.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5f54700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff9d015a630 0x7ff9d006e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5f54700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff9d0070c10 0x7ff9d00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5f54700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d006dd40 con 0x7ff9d015a250 2024-09-13T16:25:39.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5f54700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d0071600 con 0x7ff9d0070c10 2024-09-13T16:25:39.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5f54700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d006d430 con 0x7ff9d015a630 2024-09-13T16:25:39.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff9d0070c10 0x7ff9d00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff9d0070c10 0x7ff9d00710c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52528/0 (socket says 172.21.15.119:52528) 2024-09-13T16:25:39.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 -- 172.21.15.119:0/3304246522 learned_addr learned my addr 172.21.15.119:0/3304246522 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9cffff700 1 --2- 172.21.15.119:0/3304246522 >> v2:172.21.15.145:3300/0 conn(0x7ff9d015a630 0x7ff9d006e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 -- 172.21.15.119:0/3304246522 >> v2:172.21.15.145:3300/0 conn(0x7ff9d015a630 msgr2=0x7ff9d006e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:39.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 --2- 172.21.15.119:0/3304246522 >> v2:172.21.15.145:3300/0 conn(0x7ff9d015a630 0x7ff9d006e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 -- 172.21.15.119:0/3304246522 >> v1:172.21.15.145:6789/0 conn(0x7ff9d015a250 legacy=0x7ff9d00b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:39.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 -- 172.21.15.119:0/3304246522 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d006d6f0 con 0x7ff9d0070c10 2024-09-13T16:25:39.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9d5753700 1 --2- 172.21.15.119:0/3304246522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d00710c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff9d00718a0 tx=0x7ff9c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9a2b7b3cd0caf94 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9cf7fe700 1 -- 172.21.15.119:0/3304246522 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c800e070 con 0x7ff9d0070c10 2024-09-13T16:25:39.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9cf7fe700 1 -- 172.21.15.119:0/3304246522 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c8003a40 con 0x7ff9d0070c10 2024-09-13T16:25:39.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.009+0000 7ff9cf7fe700 1 -- 172.21.15.119:0/3304246522 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c8003df0 con 0x7ff9d0070c10 2024-09-13T16:25:39.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3304246522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 msgr2=0x7ff9d00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3304246522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d00710c0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff9d00718a0 tx=0x7ff9c800b330 comp rx=0 tx=0).stop 2024-09-13T16:25:39.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3304246522 shutdown_connections 2024-09-13T16:25:39.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3304246522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d00710c0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3304246522 >> v2:172.21.15.145:3300/0 conn(0x7ff9d015a630 0x7ff9d006e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3304246522 >> 172.21.15.119:0/3304246522 conn(0x7ff9d00b04f0 msgr2=0x7ff9d00b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3304246522 shutdown_connections 2024-09-13T16:25:39.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3304246522 wait complete. 2024-09-13T16:25:39.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c9ee6700 1 Processor -- start 2024-09-13T16:25:39.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c9ee6700 1 -- start start 2024-09-13T16:25:39.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c9ee6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c4146230 0x7fb5c406bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c9ee6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb5c40b2a30 0x7fb5c415cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c9ee6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c40b2e70 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c9ee6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c406cd80 con 0x7fb5c40b2a30 2024-09-13T16:25:39.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c9ee6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c406c170 con 0x7fb5c4146230 2024-09-13T16:25:39.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 Processor -- start 2024-09-13T16:25:39.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 -- start start 2024-09-13T16:25:39.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c96e5700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb5c40ac8c0 0x7fb5c406d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33868/0 (socket says 172.21.15.119:33868) 2024-09-13T16:25:39.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c96e5700 1 -- 172.21.15.119:0/135982089 learned_addr learned my addr 172.21.15.119:0/135982089 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:39.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c3fff700 1 --2- 172.21.15.119:0/135982089 >> v2:172.21.15.119:3300/0 conn(0x7fb5c40b2a30 0x7fb5c415cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c3fff700 1 -- 172.21.15.119:0/135982089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c4146230 msgr2=0x7fb5c406bb90 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:25:39.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c3fff700 1 --2- 172.21.15.119:0/135982089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c4146230 0x7fb5c406bb90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c3fff700 1 -- 172.21.15.119:0/135982089 >> v1:172.21.15.119:6789/0 conn(0x7fb5c40ac8c0 legacy=0x7fb5c406d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7fb5c3fff700 1 -- 172.21.15.119:0/135982089 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c406c450 con 0x7fb5c40b2a30 2024-09-13T16:25:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.013+0000 7ff9d5f54700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d01fe930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c3fff700 1 --2- 172.21.15.119:0/135982089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c40b2a30 0x7fb5c415cec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009850 tx=0x7fb5b4009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dec1c7cb8d4f5f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d5f54700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff9d015a250 0x7ff9d00ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d5f54700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9d015a630 0x7ff9d00cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d5f54700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01571d0 con 0x7ff9d0070c10 2024-09-13T16:25:39.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d5f54700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0156c10 con 0x7ff9d015a630 2024-09-13T16:25:39.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d5f54700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d0156ef0 con 0x7ff9d015a250 2024-09-13T16:25:39.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c37fe700 1 -- 172.21.15.119:0/135982089 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4011830 con 0x7fb5c40b2a30 2024-09-13T16:25:39.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d01fe930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9cffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff9d015a250 0x7ff9d00ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d01fe930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52554/0 (socket says 172.21.15.119:52554) 2024-09-13T16:25:39.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9cffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff9d015a250 0x7ff9d00ccfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:37260/0 (socket says 172.21.15.119:37260) 2024-09-13T16:25:39.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 -- 172.21.15.119:0/3865194283 learned_addr learned my addr 172.21.15.119:0/3865194283 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d5753700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9d015a630 0x7ff9d00cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c37fe700 1 -- 172.21.15.119:0/135982089 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b4011990 con 0x7fb5c40b2a30 2024-09-13T16:25:39.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c37fe700 1 -- 172.21.15.119:0/135982089 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b4011d20 con 0x7fb5c40b2a30 2024-09-13T16:25:39.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 -- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff9d015a250 msgr2=0x7ff9d00ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff9d015a250 0x7ff9d00ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 -- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9d015a630 msgr2=0x7ff9d00cd500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9d015a630 0x7ff9d00cd500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 -- 172.21.15.119:0/3865194283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c8009b30 con 0x7ff9d0070c10 2024-09-13T16:25:39.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d4f52700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d01fe930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c4e0 tx=0x7ff9c400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/135982089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c40b2a30 msgr2=0x7fb5c415cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/135982089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c40b2a30 0x7fb5c415cec0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009850 tx=0x7fb5b4009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:39.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9cd7fa700 1 -- 172.21.15.119:0/3865194283 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4011790 con 0x7ff9d0070c10 2024-09-13T16:25:39.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9cd7fa700 1 -- 172.21.15.119:0/3865194283 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c40118f0 con 0x7ff9d0070c10 2024-09-13T16:25:39.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9cd7fa700 1 -- 172.21.15.119:0/3865194283 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4011ca0 con 0x7ff9d0070c10 2024-09-13T16:25:39.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/135982089 shutdown_connections 2024-09-13T16:25:39.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/135982089 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c40b2a30 0x7fb5c415cec0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/135982089 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c4146230 0x7fb5c406bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/135982089 >> 172.21.15.119:0/135982089 conn(0x7fb5c406a350 msgr2=0x7fb5c406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9d01458e0 con 0x7ff9d0070c10 2024-09-13T16:25:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/135982089 shutdown_connections 2024-09-13T16:25:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/135982089 wait complete. 2024-09-13T16:25:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 Processor -- start 2024-09-13T16:25:39.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7fb5c9ee6700 1 -- start start 2024-09-13T16:25:39.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.017+0000 7ff9cd7fa700 1 -- 172.21.15.119:0/3865194283 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff9c401c460 con 0x7ff9d0070c10 2024-09-13T16:25:39.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7ff9cd7fa700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff9bc066780 0x7ff9bc068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d0155ea0 con 0x7ff9d0070c10 2024-09-13T16:25:39.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c40ac8c0 0x7fb5c41e5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c40b2a30 0x7fb5c413f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c4146230 0x7fb5c415c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c413e3d0 con 0x7fb5c4146230 2024-09-13T16:25:39.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c41e6010 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c413f1d0 con 0x7fb5c40b2a30 2024-09-13T16:25:39.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c40ac8c0 0x7fb5c41e5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c40ac8c0 0x7fb5c41e5ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:41010/0 (socket says 172.21.15.119:41010) 2024-09-13T16:25:39.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 -- 172.21.15.119:0/3421208124 learned_addr learned my addr 172.21.15.119:0/3421208124 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c3fff700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c40b2a30 0x7fb5c413f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7ff9cd7fa700 1 -- 172.21.15.119:0/3865194283 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff9c400e280 con 0x7ff9d0070c10 2024-09-13T16:25:39.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7ff9cffff700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff9bc066780 0x7ff9bc068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c96e5700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c4146230 0x7fb5c415c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 -- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c40b2a30 msgr2=0x7fb5c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c40b2a30 0x7fb5c413f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 -- 172.21.15.119:0/3421208124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c4146230 msgr2=0x7fb5c415c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c4146230 0x7fb5c415c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 -- 172.21.15.119:0/3421208124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b40094b0 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c8ee4700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c40ac8c0 0x7fb5c41e5ad0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800b4a0 tx=0x7fb5b800b7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7ff9cffff700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff9bc066780 0x7ff9bc068c40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff9d02022d0 tx=0x7ff9c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c17fa700 1 -- 172.21.15.119:0/3421208124 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8011830 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c413fba0 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c17fa700 1 -- 172.21.15.119:0/3421208124 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b80119b0 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c17fa700 1 -- 172.21.15.119:0/3421208124 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8011d60 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d0145e00 con 0x7ff9d0070c10 2024-09-13T16:25:39.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c413e650 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7f556d75b700 1 Processor -- start 2024-09-13T16:25:39.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.021+0000 7f556d75b700 1 -- start start 2024-09-13T16:25:39.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c415cec0 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556d75b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568071320 0x7f5568165410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556d75b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556815aca0 con 0x7f5568143f50 2024-09-13T16:25:39.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556d75b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55680b1920 con 0x7f556806dda0 2024-09-13T16:25:39.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556d75b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55680b1c00 con 0x7f5568071320 2024-09-13T16:25:39.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7fb5c17fa700 1 -- 172.21.15.119:0/3421208124 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb5b801a400 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556cf5a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568071320 0x7f5568165410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556cf5a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568071320 0x7f5568165410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:37288/0 (socket says 172.21.15.119:37288) 2024-09-13T16:25:39.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f5567fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5568143f50 0x7f55680cbc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33878/0 (socket says 172.21.15.119:33878) 2024-09-13T16:25:39.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556cf5a700 1 -- 172.21.15.119:0/3988529874 learned_addr learned my addr 172.21.15.119:0/3988529874 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7fb5c17fa700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5b0066730 0x7fb5b0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556cf5a700 1 -- 172.21.15.119:0/3988529874 >> v1:172.21.15.145:6789/0 conn(0x7f556806dda0 legacy=0x7f5568070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556cf5a700 1 -- 172.21.15.119:0/3988529874 >> v1:172.21.15.119:6789/0 conn(0x7f5568143f50 legacy=0x7f55680cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556cf5a700 1 -- 172.21.15.119:0/3988529874 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55680b1ee0 con 0x7f5568071320 2024-09-13T16:25:39.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7fb5c17fa700 1 -- 172.21.15.119:0/3421208124 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fb5b8084420 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7fb5c3fff700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5b0066730 0x7fb5b0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f556cf5a700 1 --2- 172.21.15.119:0/3988529874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568071320 0x7f5568165410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f555c009940 tx=0x7f555c009c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce7b059da3729c50 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f5566ffd700 1 -- 172.21.15.119:0/3988529874 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c00ba60 con 0x7f5568071320 2024-09-13T16:25:39.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f5566ffd700 1 -- 172.21.15.119:0/3988529874 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f555c009e90 con 0x7f5568071320 2024-09-13T16:25:39.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7f5566ffd700 1 -- 172.21.15.119:0/3988529874 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f555c00bde0 con 0x7f5568071320 2024-09-13T16:25:39.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.025+0000 7fb5c3fff700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5b0066730 0x7fb5b0068bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009fd0 tx=0x7fb5b4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f556d75b700 1 -- 172.21.15.119:0/3988529874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568071320 msgr2=0x7f5568165410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f556d75b700 1 --2- 172.21.15.119:0/3988529874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568071320 0x7f5568165410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f555c009940 tx=0x7f555c009c50 comp rx=0 tx=0).stop 2024-09-13T16:25:39.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792ab8a700 1 Processor -- start 2024-09-13T16:25:39.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792ab8a700 1 -- start start 2024-09-13T16:25:39.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7ff9cd7fa700 1 -- 172.21.15.119:0/3865194283 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff9d0145e00 con 0x7ff9d0070c10 2024-09-13T16:25:39.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792ab8a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7924158f10 0x7f792406d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792ab8a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792406de00 con 0x7f7924071740 2024-09-13T16:25:39.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792ab8a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792406db20 con 0x7f7924158b30 2024-09-13T16:25:39.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792ab8a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79240b1920 con 0x7f7924158f10 2024-09-13T16:25:39.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f556d75b700 1 -- 172.21.15.119:0/3988529874 shutdown_connections 2024-09-13T16:25:39.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f556d75b700 1 --2- 172.21.15.119:0/3988529874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568071320 0x7f5568165410 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f556d75b700 1 -- 172.21.15.119:0/3988529874 >> 172.21.15.119:0/3988529874 conn(0x7f55680b04f0 msgr2=0x7f55680b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f7929387700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7924071740 0x7f792406ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33890/0 (socket says 172.21.15.119:33890) 2024-09-13T16:25:39.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f7929387700 1 -- 172.21.15.119:0/1612523403 learned_addr learned my addr 172.21.15.119:0/1612523403 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:39.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792a389700 1 --2- 172.21.15.119:0/1612523403 >> v2:172.21.15.145:3300/0 conn(0x7f7924158f10 0x7f792406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792a389700 1 -- 172.21.15.119:0/1612523403 >> v1:172.21.15.145:6789/0 conn(0x7f7924158b30 legacy=0x7f7924071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792a389700 1 -- 172.21.15.119:0/1612523403 >> v1:172.21.15.119:6789/0 conn(0x7f7924071740 legacy=0x7f792406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792a389700 1 -- 172.21.15.119:0/1612523403 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79240b1c00 con 0x7f7924158f10 2024-09-13T16:25:39.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.029+0000 7f792a389700 1 --2- 172.21.15.119:0/1612523403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158f10 0x7f792406d540 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f791c009d90 tx=0x7f791c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359115f0ba2e78c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 -- 172.21.15.119:0/3988529874 shutdown_connections 2024-09-13T16:25:39.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 -- 172.21.15.119:0/3988529874 wait complete. 2024-09-13T16:25:39.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 Processor -- start 2024-09-13T16:25:39.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 -- start start 2024-09-13T16:25:39.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f556806dda0 0x7f55681f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5568071320 0x7f5568159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568143f50 0x7f556815c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556813fbd0 con 0x7f556806dda0 2024-09-13T16:25:39.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f556813a730 con 0x7f5568071320 2024-09-13T16:25:39.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556d75b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55681f1ca0 con 0x7f5568143f50 2024-09-13T16:25:39.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f1ff5507700 1 Processor -- start 2024-09-13T16:25:39.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f556806dda0 0x7f55681f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f1ff5507700 1 -- start start 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f556806dda0 0x7f55681f1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52572/0 (socket says 172.21.15.119:52572) 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 -- 172.21.15.119:0/513351739 learned_addr learned my addr 172.21.15.119:0/513351739 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f55677fe700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5568071320 0x7f5568159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f556cf5a700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568143f50 0x7f556815c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 -- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568143f50 msgr2=0x7f556815c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568143f50 0x7f556815c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 -- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5568071320 msgr2=0x7f5568159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5568071320 0x7f5568159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.033+0000 7f5567fff700 1 -- 172.21.15.119:0/513351739 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f555c009570 con 0x7f556806dda0 2024-09-13T16:25:39.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f5567fff700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f556806dda0 0x7f55681f1760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f556000cbc0 tx=0x7f556000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f5564ff9700 1 -- 172.21.15.119:0/513351739 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f556000aaa0 con 0x7f556806dda0 2024-09-13T16:25:39.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f5564ff9700 1 -- 172.21.15.119:0/513351739 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f556000ac00 con 0x7f556806dda0 2024-09-13T16:25:39.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f7928b86700 1 -- 172.21.15.119:0/1612523403 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c00e070 con 0x7f7924158f10 2024-09-13T16:25:39.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f7928b86700 1 -- 172.21.15.119:0/1612523403 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791c00bad0 con 0x7f7924158f10 2024-09-13T16:25:39.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f7928b86700 1 -- 172.21.15.119:0/1612523403 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791c00be80 con 0x7f7924158f10 2024-09-13T16:25:39.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f556d75b700 1 -- 172.21.15.119:0/513351739 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556815a510 con 0x7f556806dda0 2024-09-13T16:25:39.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1ff5507700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ff0153100 0x7f1ff0159a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1ff5507700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1ff0070c10 0x7f1ff0071080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1ff5507700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff00b2000 con 0x7f1ff0152d20 2024-09-13T16:25:39.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1ff5507700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00b28a0 con 0x7f1ff0070c10 2024-09-13T16:25:39.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1ff5507700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff00715c0 con 0x7f1ff0153100 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ff0153100 0x7f1ff0159a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ff0153100 0x7f1ff0159a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:41032/0 (socket says 172.21.15.119:41032) 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 -- 172.21.15.119:0/698220098 learned_addr learned my addr 172.21.15.119:0/698220098 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 -- 172.21.15.119:0/698220098 >> v2:172.21.15.119:3300/0 conn(0x7f1ff0070c10 msgr2=0x7f1ff0071080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 --2- 172.21.15.119:0/698220098 >> v2:172.21.15.119:3300/0 conn(0x7f1ff0070c10 0x7f1ff0071080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 -- 172.21.15.119:0/698220098 >> v1:172.21.15.119:6789/0 conn(0x7f1ff0152d20 legacy=0x7f1ff00b17f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 -- 172.21.15.119:0/698220098 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff00b22e0 con 0x7f1ff0153100 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f792ab8a700 1 -- 172.21.15.119:0/1612523403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158f10 msgr2=0x7f792406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f792ab8a700 1 --2- 172.21.15.119:0/1612523403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158f10 0x7f792406d540 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f791c009d90 tx=0x7f791c00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:39.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f5564ff9700 1 -- 172.21.15.119:0/513351739 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5560007070 con 0x7f556806dda0 2024-09-13T16:25:39.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f792ab8a700 1 -- 172.21.15.119:0/1612523403 shutdown_connections 2024-09-13T16:25:39.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f792ab8a700 1 --2- 172.21.15.119:0/1612523403 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158f10 0x7f792406d540 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f792ab8a700 1 -- 172.21.15.119:0/1612523403 >> 172.21.15.119:0/1612523403 conn(0x7f792406a350 msgr2=0x7f792406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1fef7fe700 1 --2- 172.21.15.119:0/698220098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0153100 0x7f1ff0159a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0072340 tx=0x7f1fe800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8eaf0c62d22fe58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7fb5c17fa700 1 -- 172.21.15.119:0/3421208124 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb5b80514b0 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f556d75b700 1 -- 172.21.15.119:0/513351739 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5568139e90 con 0x7f556806dda0 2024-09-13T16:25:39.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1feeffd700 1 -- 172.21.15.119:0/698220098 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe8003c40 con 0x7f1ff0153100 2024-09-13T16:25:39.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1feeffd700 1 -- 172.21.15.119:0/698220098 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe8003da0 con 0x7f1ff0153100 2024-09-13T16:25:39.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1feeffd700 1 -- 172.21.15.119:0/698220098 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe800bbc0 con 0x7f1ff0153100 2024-09-13T16:25:39.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1ff5507700 1 -- 172.21.15.119:0/698220098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0153100 msgr2=0x7f1ff0159a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f1ff5507700 1 --2- 172.21.15.119:0/698220098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0153100 0x7f1ff0159a40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0072340 tx=0x7f1fe800b330 comp rx=0 tx=0).stop 2024-09-13T16:25:39.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f792ab8a700 1 -- 172.21.15.119:0/1612523403 shutdown_connections 2024-09-13T16:25:39.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.037+0000 7f792ab8a700 1 -- 172.21.15.119:0/1612523403 wait complete. 2024-09-13T16:25:39.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5568070c10 con 0x7f556806dda0 2024-09-13T16:25:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f5564ff9700 1 -- 172.21.15.119:0/513351739 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f55600071d0 con 0x7f556806dda0 2024-09-13T16:25:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f5564ff9700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5554066730 0x7f5554068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f5564ff9700 1 -- 172.21.15.119:0/513351739 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5560084180 con 0x7f556806dda0 2024-09-13T16:25:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f1ff5507700 1 -- 172.21.15.119:0/698220098 shutdown_connections 2024-09-13T16:25:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f1ff5507700 1 --2- 172.21.15.119:0/698220098 >> v2:172.21.15.119:3300/0 conn(0x7f1ff0070c10 0x7f1ff0071080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f1ff5507700 1 --2- 172.21.15.119:0/698220098 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0153100 0x7f1ff0159a40 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f1ff5507700 1 -- 172.21.15.119:0/698220098 >> 172.21.15.119:0/698220098 conn(0x7f1ff00b04f0 msgr2=0x7f1ff00b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 Processor -- start 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 -- start start 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7924071740 0x7f7924146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158b30 0x7f792414e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7924158f10 0x7f7924150b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79240cd320 con 0x7f7924071740 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79241f1400 con 0x7f7924158b30 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f792ab8a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7924150450 con 0x7f7924158f10 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f7929b88700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7924071740 0x7f7924146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f7929b88700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7924071740 0x7f7924146c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52590/0 (socket says 172.21.15.119:52590) 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.041+0000 7f7929b88700 1 -- 172.21.15.119:0/3764122093 learned_addr learned my addr 172.21.15.119:0/3764122093 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f792a389700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7924158f10 0x7f7924150b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929b88700 1 -- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7924158f10 msgr2=0x7f7924150b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929b88700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7924158f10 0x7f7924150b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929387700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158b30 0x7f792414e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929b88700 1 -- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158b30 msgr2=0x7f792414e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929b88700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158b30 0x7f792414e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929b88700 1 -- 172.21.15.119:0/3764122093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c009a70 con 0x7f7924071740 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929387700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158b30 0x7f792414e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:39.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.045+0000 7f7929b88700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7924071740 0x7f7924146c60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f791400c6b0 tx=0x7f791400ca70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 -- 172.21.15.119:0/698220098 shutdown_connections 2024-09-13T16:25:39.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 -- 172.21.15.119:0/698220098 wait complete. 2024-09-13T16:25:39.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f55677fe700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5554066730 0x7f5554068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f5564ff9700 1 -- 172.21.15.119:0/513351739 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5560051190 con 0x7f556806dda0 2024-09-13T16:25:39.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f55677fe700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5554066730 0x7f5554068bf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f556813cb30 tx=0x7f5558009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 Processor -- start 2024-09-13T16:25:39.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 -- start start 2024-09-13T16:25:39.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ff0070c10 0x7f1ff015cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0152d20 0x7f1ff0155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ff0153100 0x7f1ff0154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff0155a00 con 0x7f1ff0070c10 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00a84b0 con 0x7f1ff0152d20 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff5507700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff00a8770 con 0x7f1ff0153100 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1fef7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0152d20 0x7f1ff0155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff4d06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ff0153100 0x7f1ff0154dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff4d06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ff0153100 0x7f1ff0154dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:37320/0 (socket says 172.21.15.119:37320) 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1fef7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0152d20 0x7f1ff0155e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:41048/0 (socket says 172.21.15.119:41048) 2024-09-13T16:25:39.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1ff4d06700 1 -- 172.21.15.119:0/1980428583 learned_addr learned my addr 172.21.15.119:0/1980428583 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f1feffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ff0070c10 0x7f1ff015cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f1feffff700 1 -- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ff0153100 msgr2=0x7f1ff0154dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f1feffff700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ff0153100 0x7f1ff0154dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f1feffff700 1 -- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0152d20 msgr2=0x7f1ff0155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f1feffff700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0152d20 0x7f1ff0155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f1feffff700 1 -- 172.21.15.119:0/1980428583 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe8009b30 con 0x7f1ff0070c10 2024-09-13T16:25:39.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f1feffff700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ff0070c10 0x7f1ff015cf00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b470 tx=0x7f1fe400b780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.049+0000 7f791a7fc700 1 -- 172.21.15.119:0/3764122093 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79140052d0 con 0x7f7924071740 2024-09-13T16:25:39.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79240cc6a0 con 0x7f7924071740 2024-09-13T16:25:39.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79240ccc60 con 0x7f7924071740 2024-09-13T16:25:39.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.053+0000 7f791a7fc700 1 -- 172.21.15.119:0/3764122093 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7914005450 con 0x7f7924071740 2024-09-13T16:25:39.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.057+0000 7f791a7fc700 1 -- 172.21.15.119:0/3764122093 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79140038e0 con 0x7f7924071740 2024-09-13T16:25:39.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.061+0000 7f1fecff9700 1 -- 172.21.15.119:0/1980428583 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4011830 con 0x7f1ff0070c10 2024-09-13T16:25:39.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.061+0000 7f1fecff9700 1 -- 172.21.15.119:0/1980428583 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4011990 con 0x7f1ff0070c10 2024-09-13T16:25:39.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.061+0000 7f1fecff9700 1 -- 172.21.15.119:0/1980428583 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4011d40 con 0x7f1ff0070c10 2024-09-13T16:25:39.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.061+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff0155690 con 0x7f1ff0070c10 2024-09-13T16:25:39.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.061+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff0144ef0 con 0x7f1ff0070c10 2024-09-13T16:25:39.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.061+0000 7f1fecff9700 1 -- 172.21.15.119:0/1980428583 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1fe401a400 con 0x7f1ff0070c10 2024-09-13T16:25:39.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.061+0000 7f791a7fc700 1 -- 172.21.15.119:0/3764122093 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7914003a40 con 0x7f7924071740 2024-09-13T16:25:39.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f1fecff9700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fd4066780 0x7f1fd4068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f1fecff9700 1 -- 172.21.15.119:0/1980428583 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f1fe40842b0 con 0x7f1ff0070c10 2024-09-13T16:25:39.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f791a7fc700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f791006ad20 0x7f791006d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f1fef7fe700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fd4066780 0x7f1fd4068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f7929387700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f791006ad20 0x7f791006d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f791a7fc700 1 -- 172.21.15.119:0/3764122093 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7914083910 con 0x7f7924071740 2024-09-13T16:25:39.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f1fef7fe700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fd4066780 0x7f1fd4068c40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000f80 tx=0x7f1fe801b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.065+0000 7f7929387700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f791006ad20 0x7f791006d1e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7924158240 tx=0x7f7920009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.069+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff0155820 con 0x7f1ff0070c10 2024-09-13T16:25:39.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.069+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f790c005310 con 0x7f7924071740 2024-09-13T16:25:39.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.077+0000 7f1fecff9700 1 -- 172.21.15.119:0/1980428583 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1fe4051410 con 0x7f1ff0070c10 2024-09-13T16:25:39.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.089+0000 7f791a7fc700 1 -- 172.21.15.119:0/3764122093 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f79140509f0 con 0x7f7924071740 2024-09-13T16:25:39.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.233+0000 7fcb9759e700 1 -- 172.21.15.119:0/3654553664 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fcb7c005890 con 0x7fcb98153100 2024-09-13T16:25:39.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.233+0000 7fcb86ffd700 1 -- 172.21.15.119:0/3654553664 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcb900183e0 con 0x7fcb98153100 2024-09-13T16:25:39.239 INFO:teuthology.orchestra.run.smithi119.stdout:60129542153 2024-09-13T16:25:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 -- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcb800668d0 msgr2=0x7fcb80068d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcb800668d0 0x7fcb80068d90 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c001720 tx=0x7fcb8c009040 comp rx=0 tx=0).stop 2024-09-13T16:25:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 -- 172.21.15.119:0/3654553664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98153100 msgr2=0x7fcb981580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98153100 0x7fcb981580c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fcb90009b60 tx=0x7fcb90004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 -- 172.21.15.119:0/3654553664 shutdown_connections 2024-09-13T16:25:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcb800668d0 0x7fcb80068d90 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcb98153100 0x7fcb981580c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcb98152d20 0x7fcb9815a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 --2- 172.21.15.119:0/3654553664 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcb98071320 0x7fcb98141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 -- 172.21.15.119:0/3654553664 >> 172.21.15.119:0/3654553664 conn(0x7fcb980b04f0 msgr2=0x7fcb981523a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 -- 172.21.15.119:0/3654553664 shutdown_connections 2024-09-13T16:25:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.237+0000 7fcb84ff9700 1 -- 172.21.15.119:0/3654553664 wait complete. 2024-09-13T16:25:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.241+0000 7fe7c18d3700 1 -- 172.21.15.119:0/2527230639 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fe79c005890 con 0x7fe7bc154780 2024-09-13T16:25:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.241+0000 7fe7b97fa700 1 -- 172.21.15.119:0/2527230639 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fe7b4027090 con 0x7fe7bc154780 2024-09-13T16:25:39.249 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:39.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 -- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7a8066780 msgr2=0x7fe7a8068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7a8066780 0x7fe7a8068c40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0005150 tx=0x7fe7b0007ee0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 -- 172.21.15.119:0/2527230639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc154780 msgr2=0x7fe7bc14f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc154780 0x7fe7bc14f850 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe7b4006620 tx=0x7fe7b4004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 -- 172.21.15.119:0/2527230639 shutdown_connections 2024-09-13T16:25:39.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7a8066780 0x7fe7a8068c40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7bc154780 0x7fe7bc14f850 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7bc153100 0x7fe7bc0abd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 --2- 172.21.15.119:0/2527230639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7bc152d20 0x7fe7bc150f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 -- 172.21.15.119:0/2527230639 >> 172.21.15.119:0/2527230639 conn(0x7fe7bc0b04f0 msgr2=0x7fe7bc0cd380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 -- 172.21.15.119:0/2527230639 shutdown_connections 2024-09-13T16:25:39.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.249+0000 7fe7a2ffd700 1 -- 172.21.15.119:0/2527230639 wait complete. 2024-09-13T16:25:39.255 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.5 2024-09-13T16:25:39.256 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.267 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.6 2024-09-13T16:25:39.267 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.289+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f13c80718e0 con 0x7f13c8160c60 2024-09-13T16:25:39.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.289+0000 7f13be7fc700 1 -- 172.21.15.119:0/1852156324 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f13c001a070 con 0x7f13c8160c60 2024-09-13T16:25:39.293 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f13ac066800 msgr2=0x7f13ac068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f13ac066800 0x7f13ac068cc0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f13c8071c40 tx=0x7f13b8008030 comp rx=0 tx=0).stop 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 msgr2=0x7f13c814b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c814b0e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f13c0009fa0 tx=0x7f13c000d460 comp rx=0 tx=0).stop 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 shutdown_connections 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f13ac066800 0x7f13ac068cc0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f13c8160c60 0x7f13c814b0e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f13c80b17f0 0x7f13c8148710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 --2- 172.21.15.119:0/1852156324 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f13c806cbd0 0x7f13c81494e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 >> 172.21.15.119:0/1852156324 conn(0x7f13c80b04f0 msgr2=0x7f13c80adeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 shutdown_connections 2024-09-13T16:25:39.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.293+0000 7f13ce923700 1 -- 172.21.15.119:0/1852156324 wait complete. 2024-09-13T16:25:39.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.301+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7ff9d0145e00 con 0x7ff9d0070c10 2024-09-13T16:25:39.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.301+0000 7ff9cd7fa700 1 -- 172.21.15.119:0/3865194283 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff9d0145e00 con 0x7ff9d0070c10 2024-09-13T16:25:39.307 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff9bc066780 msgr2=0x7ff9bc068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff9bc066780 0x7ff9bc068c40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff9d02022d0 tx=0x7ff9c0008040 comp rx=0 tx=0).stop 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 msgr2=0x7ff9d01fe930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d01fe930 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff9c400c4e0 tx=0x7ff9c400c8a0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 shutdown_connections 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff9bc066780 0x7ff9bc068c40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff9d015a630 0x7ff9d00cd500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff9d015a250 0x7ff9d00ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 --2- 172.21.15.119:0/3865194283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff9d0070c10 0x7ff9d01fe930 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 >> 172.21.15.119:0/3865194283 conn(0x7ff9d00b04f0 msgr2=0x7ff9d015aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 shutdown_connections 2024-09-13T16:25:39.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.305+0000 7ff9d5f54700 1 -- 172.21.15.119:0/3865194283 wait complete. 2024-09-13T16:25:39.310 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.0 2024-09-13T16:25:39.311 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.319 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.7 2024-09-13T16:25:39.319 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.317+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f55680b3c70 con 0x7f556806dda0 2024-09-13T16:25:39.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.317+0000 7f5564ff9700 1 -- 172.21.15.119:0/513351739 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f55600547b0 con 0x7f556806dda0 2024-09-13T16:25:39.322 INFO:teuthology.orchestra.run.smithi119.stdout:60129542153 2024-09-13T16:25:39.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5554066730 msgr2=0x7f5554068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5554066730 0x7f5554068bf0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f556813cb30 tx=0x7f5558009040 comp rx=0 tx=0).stop 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f556806dda0 msgr2=0x7f55681f1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f556806dda0 0x7f55681f1760 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f556000cbc0 tx=0x7f556000ced0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 shutdown_connections 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5554066730 0x7f5554068bf0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5568143f50 0x7f556815c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5568071320 0x7f5568159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 --2- 172.21.15.119:0/513351739 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f556806dda0 0x7f55681f1760 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 >> 172.21.15.119:0/513351739 conn(0x7f55680b04f0 msgr2=0x7f55680cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 shutdown_connections 2024-09-13T16:25:39.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.321+0000 7f554e7fc700 1 -- 172.21.15.119:0/513351739 wait complete. 2024-09-13T16:25:39.334 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.2 2024-09-13T16:25:39.334 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1ff0071080 con 0x7f1ff0070c10 2024-09-13T16:25:39.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1fecff9700 1 -- 172.21.15.119:0/1980428583 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1fe4015070 con 0x7f1ff0070c10 2024-09-13T16:25:39.353 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:39.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fb5c40cdba0 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fd4066780 msgr2=0x7f1fd4068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fd4066780 0x7f1fd4068c40 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f1fe8000f80 tx=0x7f1fe801b040 comp rx=0 tx=0).stop 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ff0070c10 msgr2=0x7f1ff015cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ff0070c10 0x7f1ff015cf00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400b470 tx=0x7f1fe400b780 comp rx=0 tx=0).stop 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 shutdown_connections 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fd4066780 0x7f1fd4068c40 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ff0153100 0x7f1ff0154dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ff0152d20 0x7f1ff0155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 --2- 172.21.15.119:0/1980428583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ff0070c10 0x7f1ff015cf00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 >> 172.21.15.119:0/1980428583 conn(0x7f1ff00b04f0 msgr2=0x7f1ff0157a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 shutdown_connections 2024-09-13T16:25:39.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7f1ff5507700 1 -- 172.21.15.119:0/1980428583 wait complete. 2024-09-13T16:25:39.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.349+0000 7fb5c17fa700 1 -- 172.21.15.119:0/3421208124 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb5b8054ad0 con 0x7fb5c40ac8c0 2024-09-13T16:25:39.356 INFO:teuthology.orchestra.run.smithi119.stdout:60129542151 2024-09-13T16:25:39.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5b0066730 msgr2=0x7fb5b0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5b0066730 0x7fb5b0068bf0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009fd0 tx=0x7fb5b4008030 comp rx=0 tx=0).stop 2024-09-13T16:25:39.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c40ac8c0 msgr2=0x7fb5c41e5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c40ac8c0 0x7fb5c41e5ad0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb5b800b4a0 tx=0x7fb5b800b7b0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 shutdown_connections 2024-09-13T16:25:39.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5b0066730 0x7fb5b0068bf0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c4146230 0x7fb5c415c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c40b2a30 0x7fb5c413f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 --2- 172.21.15.119:0/3421208124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c40ac8c0 0x7fb5c41e5ad0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 >> 172.21.15.119:0/3421208124 conn(0x7fb5c406a350 msgr2=0x7fb5c4144ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 shutdown_connections 2024-09-13T16:25:39.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.353+0000 7fb5c9ee6700 1 -- 172.21.15.119:0/3421208124 wait complete. 2024-09-13T16:25:39.370 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.3 2024-09-13T16:25:39.370 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.372 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.4 2024-09-13T16:25:39.372 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.385+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f790c005890 con 0x7f7924071740 2024-09-13T16:25:39.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.385+0000 7f791a7fc700 1 -- 172.21.15.119:0/3764122093 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7914017070 con 0x7f7924071740 2024-09-13T16:25:39.389 INFO:teuthology.orchestra.run.smithi119.stdout:60129542152 2024-09-13T16:25:39.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f791006ad20 msgr2=0x7f791006d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f791006ad20 0x7f791006d1e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f7924158240 tx=0x7f7920009040 comp rx=0 tx=0).stop 2024-09-13T16:25:39.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7924071740 msgr2=0x7f7924146c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7924071740 0x7f7924146c60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f791400c6b0 tx=0x7f791400ca70 comp rx=0 tx=0).stop 2024-09-13T16:25:39.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 shutdown_connections 2024-09-13T16:25:39.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f791006ad20 0x7f791006d1e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7924158f10 0x7f7924150b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7924158b30 0x7f792414e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 --2- 172.21.15.119:0/3764122093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7924071740 0x7f7924146c60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 >> 172.21.15.119:0/3764122093 conn(0x7f792406a350 msgr2=0x7f79240ae460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 shutdown_connections 2024-09-13T16:25:39.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.389+0000 7f792ab8a700 1 -- 172.21.15.119:0/3764122093 wait complete. 2024-09-13T16:25:39.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-09-13T16:25:39.408 DEBUG:teuthology.parallel:result is None 2024-09-13T16:25:39.408 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T16:25:39.408 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:25:39.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.553+0000 7f4ae225f700 1 Processor -- start 2024-09-13T16:25:39.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.553+0000 7f4ae225f700 1 -- start start 2024-09-13T16:25:39.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4050480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4ad40509c0 0x7f4ad4005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ad40e3310 con 0x7f4ad40df8e0 2024-09-13T16:25:39.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ad4050e50 con 0x7f4ad40509c0 2024-09-13T16:25:39.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad40e35f0 con 0x7f4ad404fe30 2024-09-13T16:25:39.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae1a5e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4ad40509c0 0x7f4ad4005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae125d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4ad40df8e0 0x7f4ad401c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33900/0 (socket says 172.21.15.119:33900) 2024-09-13T16:25:39.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae125d700 1 -- 172.21.15.119:0/1796910808 learned_addr learned my addr 172.21.15.119:0/1796910808 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:39.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae0a5c700 1 --2- 172.21.15.119:0/1796910808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4050480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4acffff700 1 -- 172.21.15.119:0/1796910808 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228059323 0 0) 0x7f4ad40e3310 con 0x7f4ad40df8e0 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4acffff700 1 -- 172.21.15.119:0/1796910808 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ac80057e0 con 0x7f4ad40df8e0 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae1a5e700 1 -- 172.21.15.119:0/1796910808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 msgr2=0x7f4ad4050480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae1a5e700 1 --2- 172.21.15.119:0/1796910808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4050480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae1a5e700 1 -- 172.21.15.119:0/1796910808 >> v1:172.21.15.119:6789/0 conn(0x7f4ad40df8e0 legacy=0x7f4ad401c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae1a5e700 1 -- 172.21.15.119:0/1796910808 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad40e38d0 con 0x7f4ad40509c0 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae1a5e700 1 --2- 172.21.15.119:0/1796910808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40509c0 0x7f4ad4005b10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4adc04ff70 tx=0x7f4adc068790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=385930cfedc34939 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4acffff700 1 -- 172.21.15.119:0/1796910808 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc067690 con 0x7f4ad40509c0 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4acffff700 1 -- 172.21.15.119:0/1796910808 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4adc04f010 con 0x7f4ad40509c0 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4acffff700 1 -- 172.21.15.119:0/1796910808 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc067b90 con 0x7f4ad40509c0 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- 172.21.15.119:0/1796910808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40509c0 msgr2=0x7f4ad4005b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 --2- 172.21.15.119:0/1796910808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40509c0 0x7f4ad4005b10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f4adc04ff70 tx=0x7f4adc068790 comp rx=0 tx=0).stop 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- 172.21.15.119:0/1796910808 shutdown_connections 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 --2- 172.21.15.119:0/1796910808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40509c0 0x7f4ad4005b10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 --2- 172.21.15.119:0/1796910808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4050480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- 172.21.15.119:0/1796910808 >> 172.21.15.119:0/1796910808 conn(0x7f4ad4036140 msgr2=0x7f4ad4036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- 172.21.15.119:0/1796910808 shutdown_connections 2024-09-13T16:25:39.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.557+0000 7f4ae225f700 1 -- 172.21.15.119:0/1796910808 wait complete. 2024-09-13T16:25:39.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 Processor -- start 2024-09-13T16:25:39.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 -- start start 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4032610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ad40509c0 0x7f4ad4178700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40df8e0 0x7f4ad4039d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad4032b50 con 0x7f4ad40df8e0 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad4031570 con 0x7f4ad40509c0 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ad4031850 con 0x7f4ad404fe30 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40df8e0 0x7f4ad4039d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40df8e0 0x7f4ad4039d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52624/0 (socket says 172.21.15.119:52624) 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 -- 172.21.15.119:0/2602123770 learned_addr learned my addr 172.21.15.119:0/2602123770 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae125d700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4032610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae0a5c700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ad40509c0 0x7f4ad4178700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 -- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 msgr2=0x7f4ad4032610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4032610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 -- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ad40509c0 msgr2=0x7f4ad4178700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ad40509c0 0x7f4ad4178700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 -- 172.21.15.119:0/2602123770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad4031b30 con 0x7f4ad40df8e0 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae125d700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4032610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae1a5e700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40df8e0 0x7f4ad4039d10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4adc04f480 tx=0x7f4adc06a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4acdffb700 1 -- 172.21.15.119:0/2602123770 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc09a040 con 0x7f4ad40df8e0 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ad4031e10 con 0x7f4ad40df8e0 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4acdffb700 1 -- 172.21.15.119:0/2602123770 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4adc072350 con 0x7f4ad40df8e0 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ad4178c40 con 0x7f4ad40df8e0 2024-09-13T16:25:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.561+0000 7f4acdffb700 1 -- 172.21.15.119:0/2602123770 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4adc0726c0 con 0x7f4ad40df8e0 2024-09-13T16:25:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.565+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ad40e3310 con 0x7f4ad40df8e0 2024-09-13T16:25:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.565+0000 7f4acdffb700 1 -- 172.21.15.119:0/2602123770 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4adc072820 con 0x7f4ad40df8e0 2024-09-13T16:25:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.565+0000 7f4acdffb700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ac0066730 0x7f4ac0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.565+0000 7f4acdffb700 1 -- 172.21.15.119:0/2602123770 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4adc0ff1e0 con 0x7f4ad40df8e0 2024-09-13T16:25:39.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.565+0000 7f4ae125d700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ac0066730 0x7f4ac0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.565+0000 7f4ae125d700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ac0066730 0x7f4ac0068bf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8001360 tx=0x7f4ad8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.569+0000 7f4acdffb700 1 -- 172.21.15.119:0/2602123770 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4adc0cc380 con 0x7f4ad40df8e0 2024-09-13T16:25:39.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.721+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ad40e3e90 con 0x7f4ac0066730 2024-09-13T16:25:39.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.741+0000 7f4acdffb700 1 -- 172.21.15.119:0/2602123770 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345197 (secure 0 0 0) 0x7f4ad8005850 con 0x7f4ac0066730 2024-09-13T16:25:39.751 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:39.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ac0066730 msgr2=0x7f4ac0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ac0066730 0x7f4ac0068bf0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4ad8001360 tx=0x7f4ad8009040 comp rx=0 tx=0).stop 2024-09-13T16:25:39.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40df8e0 msgr2=0x7f4ad4039d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40df8e0 0x7f4ad4039d10 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f4adc04f480 tx=0x7f4adc06a080 comp rx=0 tx=0).stop 2024-09-13T16:25:39.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 shutdown_connections 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ac0066730 0x7f4ac0068bf0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ad40df8e0 0x7f4ad4039d10 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ad40509c0 0x7f4ad4178700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 --2- 172.21.15.119:0/2602123770 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ad404fe30 0x7f4ad4032610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 >> 172.21.15.119:0/2602123770 conn(0x7f4ad4036140 msgr2=0x7f4ad4034810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 shutdown_connections 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.749+0000 7f4ae225f700 1 -- 172.21.15.119:0/2602123770 wait complete. 2024-09-13T16:25:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:25:39.765 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-13T16:25:39.426441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219456,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755264,"statfs":{"total":773094113280,"available":772869390336,"internally_reserved":0,"allocated":3350528,"data_stored":2013930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005660"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954165+0000","last_change":"2024-09-13T16:25:16.560322+0000","last_active":"2024-09-13T16:25:20.954165+0000","last_peered":"2024-09-13T16:25:20.954165+0000","last_clean":"2024-09-13T16:25:20.954165+0000","last_became_active":"2024-09-13T16:25:16.560130+0000","last_became_peered":"2024-09-13T16:25:16.560130+0000","last_unstale":"2024-09-13T16:25:20.954165+0000","last_undegraded":"2024-09-13T16:25:20.954165+0000","last_fullsized":"2024-09-13T16:25:20.954165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941703+0000","last_change":"2024-09-13T16:25:16.937844+0000","last_active":"2024-09-13T16:25:20.941703+0000","last_peered":"2024-09-13T16:25:20.941703+0000","last_clean":"2024-09-13T16:25:20.941703+0000","last_became_active":"2024-09-13T16:25:16.937508+0000","last_became_peered":"2024-09-13T16:25:16.937508+0000","last_unstale":"2024-09-13T16:25:20.941703+0000","last_undegraded":"2024-09-13T16:25:20.941703+0000","last_fullsized":"2024-09-13T16:25:20.941703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:46.073323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163484+0000","last_change":"2024-09-13T16:25:21.162079+0000","last_active":"2024-09-13T16:25:21.163484+0000","last_peered":"2024-09-13T16:25:21.163484+0000","last_clean":"2024-09-13T16:25:21.163484+0000","last_became_active":"2024-09-13T16:25:21.161341+0000","last_became_peered":"2024-09-13T16:25:21.161341+0000","last_unstale":"2024-09-13T16:25:21.163484+0000","last_undegraded":"2024-09-13T16:25:21.163484+0000","last_fullsized":"2024-09-13T16:25:21.163484+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:55.099223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941956+0000","last_change":"2024-09-13T16:25:16.554751+0000","last_active":"2024-09-13T16:25:20.941956+0000","last_peered":"2024-09-13T16:25:20.941956+0000","last_clean":"2024-09-13T16:25:20.941956+0000","last_became_active":"2024-09-13T16:25:16.552873+0000","last_became_peered":"2024-09-13T16:25:16.552873+0000","last_unstale":"2024-09-13T16:25:20.941956+0000","last_undegraded":"2024-09-13T16:25:20.941956+0000","last_fullsized":"2024-09-13T16:25:20.941956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.947694+0000","last_change":"2024-09-13T16:25:16.933368+0000","last_active":"2024-09-13T16:25:20.947694+0000","last_peered":"2024-09-13T16:25:20.947694+0000","last_clean":"2024-09-13T16:25:20.947694+0000","last_became_active":"2024-09-13T16:25:16.932824+0000","last_became_peered":"2024-09-13T16:25:16.932824+0000","last_unstale":"2024-09-13T16:25:20.947694+0000","last_undegraded":"2024-09-13T16:25:20.947694+0000","last_fullsized":"2024-09-13T16:25:20.947694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969023+0000","last_change":"2024-09-13T16:25:20.965564+0000","last_active":"2024-09-13T16:25:20.969023+0000","last_peered":"2024-09-13T16:25:20.969023+0000","last_clean":"2024-09-13T16:25:20.969023+0000","last_became_active":"2024-09-13T16:25:20.965159+0000","last_became_peered":"2024-09-13T16:25:20.965159+0000","last_unstale":"2024-09-13T16:25:20.969023+0000","last_undegraded":"2024-09-13T16:25:20.969023+0000","last_fullsized":"2024-09-13T16:25:20.969023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150484+0000","last_change":"2024-09-13T16:25:16.560318+0000","last_active":"2024-09-13T16:25:21.150484+0000","last_peered":"2024-09-13T16:25:21.150484+0000","last_clean":"2024-09-13T16:25:21.150484+0000","last_became_active":"2024-09-13T16:25:16.560069+0000","last_became_peered":"2024-09-13T16:25:16.560069+0000","last_unstale":"2024-09-13T16:25:21.150484+0000","last_undegraded":"2024-09-13T16:25:21.150484+0000","last_fullsized":"2024-09-13T16:25:21.150484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:12.814872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943883+0000","last_change":"2024-09-13T16:25:16.935607+0000","last_active":"2024-09-13T16:25:20.943883+0000","last_peered":"2024-09-13T16:25:20.943883+0000","last_clean":"2024-09-13T16:25:20.943883+0000","last_became_active":"2024-09-13T16:25:16.935353+0000","last_became_peered":"2024-09-13T16:25:16.935353+0000","last_unstale":"2024-09-13T16:25:20.943883+0000","last_undegraded":"2024-09-13T16:25:20.943883+0000","last_fullsized":"2024-09-13T16:25:20.943883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:35.981913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973971+0000","last_change":"2024-09-13T16:25:20.969324+0000","last_active":"2024-09-13T16:25:20.973971+0000","last_peered":"2024-09-13T16:25:20.973971+0000","last_clean":"2024-09-13T16:25:20.973971+0000","last_became_active":"2024-09-13T16:25:20.968916+0000","last_became_peered":"2024-09-13T16:25:20.968916+0000","last_unstale":"2024-09-13T16:25:20.973971+0000","last_undegraded":"2024-09-13T16:25:20.973971+0000","last_fullsized":"2024-09-13T16:25:20.973971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:03.825482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149974+0000","last_change":"2024-09-13T16:25:16.560280+0000","last_active":"2024-09-13T16:25:21.149974+0000","last_peered":"2024-09-13T16:25:21.149974+0000","last_clean":"2024-09-13T16:25:21.149974+0000","last_became_active":"2024-09-13T16:25:16.559981+0000","last_became_peered":"2024-09-13T16:25:16.559981+0000","last_unstale":"2024-09-13T16:25:21.149974+0000","last_undegraded":"2024-09-13T16:25:21.149974+0000","last_fullsized":"2024-09-13T16:25:21.149974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943411+0000","last_change":"2024-09-13T16:25:16.937364+0000","last_active":"2024-09-13T16:25:20.943411+0000","last_peered":"2024-09-13T16:25:20.943411+0000","last_clean":"2024-09-13T16:25:20.943411+0000","last_became_active":"2024-09-13T16:25:16.937119+0000","last_became_peered":"2024-09-13T16:25:16.937119+0000","last_unstale":"2024-09-13T16:25:20.943411+0000","last_undegraded":"2024-09-13T16:25:20.943411+0000","last_fullsized":"2024-09-13T16:25:20.943411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:31.885317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959982+0000","last_change":"2024-09-13T16:25:20.959414+0000","last_active":"2024-09-13T16:25:20.959982+0000","last_peered":"2024-09-13T16:25:20.959982+0000","last_clean":"2024-09-13T16:25:20.959982+0000","last_became_active":"2024-09-13T16:25:20.958955+0000","last_became_peered":"2024-09-13T16:25:20.958955+0000","last_unstale":"2024-09-13T16:25:20.959982+0000","last_undegraded":"2024-09-13T16:25:20.959982+0000","last_fullsized":"2024-09-13T16:25:20.959982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:35.667608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939712+0000","last_change":"2024-09-13T16:25:16.936925+0000","last_active":"2024-09-13T16:25:20.939712+0000","last_peered":"2024-09-13T16:25:20.939712+0000","last_clean":"2024-09-13T16:25:20.939712+0000","last_became_active":"2024-09-13T16:25:16.936608+0000","last_became_peered":"2024-09-13T16:25:16.936608+0000","last_unstale":"2024-09-13T16:25:20.939712+0000","last_undegraded":"2024-09-13T16:25:20.939712+0000","last_fullsized":"2024-09-13T16:25:20.939712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150049+0000","last_change":"2024-09-13T16:25:16.561316+0000","last_active":"2024-09-13T16:25:21.150049+0000","last_peered":"2024-09-13T16:25:21.150049+0000","last_clean":"2024-09-13T16:25:21.150049+0000","last_became_active":"2024-09-13T16:25:16.560850+0000","last_became_peered":"2024-09-13T16:25:16.560850+0000","last_unstale":"2024-09-13T16:25:21.150049+0000","last_undegraded":"2024-09-13T16:25:21.150049+0000","last_fullsized":"2024-09-13T16:25:21.150049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.961598+0000","last_change":"2024-09-13T16:25:20.960831+0000","last_active":"2024-09-13T16:25:20.961598+0000","last_peered":"2024-09-13T16:25:20.961598+0000","last_clean":"2024-09-13T16:25:20.961598+0000","last_became_active":"2024-09-13T16:25:20.960495+0000","last_became_peered":"2024-09-13T16:25:20.960495+0000","last_unstale":"2024-09-13T16:25:20.961598+0000","last_undegraded":"2024-09-13T16:25:20.961598+0000","last_fullsized":"2024-09-13T16:25:20.961598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940654+0000","last_change":"2024-09-13T16:25:16.937601+0000","last_active":"2024-09-13T16:25:20.940654+0000","last_peered":"2024-09-13T16:25:20.940654+0000","last_clean":"2024-09-13T16:25:20.940654+0000","last_became_active":"2024-09-13T16:25:16.937343+0000","last_became_peered":"2024-09-13T16:25:16.937343+0000","last_unstale":"2024-09-13T16:25:20.940654+0000","last_undegraded":"2024-09-13T16:25:20.940654+0000","last_fullsized":"2024-09-13T16:25:20.940654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950384+0000","last_change":"2024-09-13T16:25:16.550175+0000","last_active":"2024-09-13T16:25:20.950384+0000","last_peered":"2024-09-13T16:25:20.950384+0000","last_clean":"2024-09-13T16:25:20.950384+0000","last_became_active":"2024-09-13T16:25:16.549875+0000","last_became_peered":"2024-09-13T16:25:16.549875+0000","last_unstale":"2024-09-13T16:25:20.950384+0000","last_undegraded":"2024-09-13T16:25:20.950384+0000","last_fullsized":"2024-09-13T16:25:20.950384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973920+0000","last_change":"2024-09-13T16:25:20.968856+0000","last_active":"2024-09-13T16:25:20.973920+0000","last_peered":"2024-09-13T16:25:20.973920+0000","last_clean":"2024-09-13T16:25:20.973920+0000","last_became_active":"2024-09-13T16:25:20.968470+0000","last_became_peered":"2024-09-13T16:25:20.968470+0000","last_unstale":"2024-09-13T16:25:20.973920+0000","last_undegraded":"2024-09-13T16:25:20.973920+0000","last_fullsized":"2024-09-13T16:25:20.973920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940018+0000","last_change":"2024-09-13T16:25:16.936551+0000","last_active":"2024-09-13T16:25:20.940018+0000","last_peered":"2024-09-13T16:25:20.940018+0000","last_clean":"2024-09-13T16:25:20.940018+0000","last_became_active":"2024-09-13T16:25:16.936298+0000","last_became_peered":"2024-09-13T16:25:16.936298+0000","last_unstale":"2024-09-13T16:25:20.940018+0000","last_undegraded":"2024-09-13T16:25:20.940018+0000","last_fullsized":"2024-09-13T16:25:20.940018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949896+0000","last_change":"2024-09-13T16:25:16.556208+0000","last_active":"2024-09-13T16:25:20.949896+0000","last_peered":"2024-09-13T16:25:20.949896+0000","last_clean":"2024-09-13T16:25:20.949896+0000","last_became_active":"2024-09-13T16:25:16.555897+0000","last_became_peered":"2024-09-13T16:25:16.555897+0000","last_unstale":"2024-09-13T16:25:20.949896+0000","last_undegraded":"2024-09-13T16:25:20.949896+0000","last_fullsized":"2024-09-13T16:25:20.949896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.164051+0000","last_change":"2024-09-13T16:25:21.162085+0000","last_active":"2024-09-13T16:25:21.164051+0000","last_peered":"2024-09-13T16:25:21.164051+0000","last_clean":"2024-09-13T16:25:21.164051+0000","last_became_active":"2024-09-13T16:25:21.160633+0000","last_became_peered":"2024-09-13T16:25:21.160633+0000","last_unstale":"2024-09-13T16:25:21.164051+0000","last_undegraded":"2024-09-13T16:25:21.164051+0000","last_fullsized":"2024-09-13T16:25:21.164051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951220+0000","last_change":"2024-09-13T16:25:16.931406+0000","last_active":"2024-09-13T16:25:20.951220+0000","last_peered":"2024-09-13T16:25:20.951220+0000","last_clean":"2024-09-13T16:25:20.951220+0000","last_became_active":"2024-09-13T16:25:16.931073+0000","last_became_peered":"2024-09-13T16:25:16.931073+0000","last_unstale":"2024-09-13T16:25:20.951220+0000","last_undegraded":"2024-09-13T16:25:20.951220+0000","last_fullsized":"2024-09-13T16:25:20.951220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:31.812511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149390+0000","last_change":"2024-09-13T16:25:16.562667+0000","last_active":"2024-09-13T16:25:21.149390+0000","last_peered":"2024-09-13T16:25:21.149390+0000","last_clean":"2024-09-13T16:25:21.149390+0000","last_became_active":"2024-09-13T16:25:16.562361+0000","last_became_peered":"2024-09-13T16:25:16.562361+0000","last_unstale":"2024-09-13T16:25:21.149390+0000","last_undegraded":"2024-09-13T16:25:21.149390+0000","last_fullsized":"2024-09-13T16:25:21.149390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:32.764470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973976+0000","last_change":"2024-09-13T16:25:20.969014+0000","last_active":"2024-09-13T16:25:20.973976+0000","last_peered":"2024-09-13T16:25:20.973976+0000","last_clean":"2024-09-13T16:25:20.973976+0000","last_became_active":"2024-09-13T16:25:20.968528+0000","last_became_peered":"2024-09-13T16:25:20.968528+0000","last_unstale":"2024-09-13T16:25:20.973976+0000","last_undegraded":"2024-09-13T16:25:20.973976+0000","last_fullsized":"2024-09-13T16:25:20.973976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948588+0000","last_change":"2024-09-13T16:25:16.933619+0000","last_active":"2024-09-13T16:25:20.948588+0000","last_peered":"2024-09-13T16:25:20.948588+0000","last_clean":"2024-09-13T16:25:20.948588+0000","last_became_active":"2024-09-13T16:25:16.933324+0000","last_became_peered":"2024-09-13T16:25:16.933324+0000","last_unstale":"2024-09-13T16:25:20.948588+0000","last_undegraded":"2024-09-13T16:25:20.948588+0000","last_fullsized":"2024-09-13T16:25:20.948588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:13.824400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942057+0000","last_change":"2024-09-13T16:25:16.560251+0000","last_active":"2024-09-13T16:25:20.942057+0000","last_peered":"2024-09-13T16:25:20.942057+0000","last_clean":"2024-09-13T16:25:20.942057+0000","last_became_active":"2024-09-13T16:25:16.559836+0000","last_became_peered":"2024-09-13T16:25:16.559836+0000","last_unstale":"2024-09-13T16:25:20.942057+0000","last_undegraded":"2024-09-13T16:25:20.942057+0000","last_fullsized":"2024-09-13T16:25:20.942057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.159122+0000","last_change":"2024-09-13T16:25:21.156691+0000","last_active":"2024-09-13T16:25:21.159122+0000","last_peered":"2024-09-13T16:25:21.159122+0000","last_clean":"2024-09-13T16:25:21.159122+0000","last_became_active":"2024-09-13T16:25:21.156255+0000","last_became_peered":"2024-09-13T16:25:21.156255+0000","last_unstale":"2024-09-13T16:25:21.159122+0000","last_undegraded":"2024-09-13T16:25:21.159122+0000","last_fullsized":"2024-09-13T16:25:21.159122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:12.641280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940196+0000","last_change":"2024-09-13T16:25:16.946326+0000","last_active":"2024-09-13T16:25:20.940196+0000","last_peered":"2024-09-13T16:25:20.940196+0000","last_clean":"2024-09-13T16:25:20.940196+0000","last_became_active":"2024-09-13T16:25:16.946091+0000","last_became_peered":"2024-09-13T16:25:16.946091+0000","last_unstale":"2024-09-13T16:25:20.940196+0000","last_undegraded":"2024-09-13T16:25:20.940196+0000","last_fullsized":"2024-09-13T16:25:20.940196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953612+0000","last_change":"2024-09-13T16:25:16.548181+0000","last_active":"2024-09-13T16:25:20.953612+0000","last_peered":"2024-09-13T16:25:20.953612+0000","last_clean":"2024-09-13T16:25:20.953612+0000","last_became_active":"2024-09-13T16:25:16.546980+0000","last_became_peered":"2024-09-13T16:25:16.546980+0000","last_unstale":"2024-09-13T16:25:20.953612+0000","last_undegraded":"2024-09-13T16:25:20.953612+0000","last_fullsized":"2024-09-13T16:25:20.953612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163366+0000","last_change":"2024-09-13T16:25:21.157394+0000","last_active":"2024-09-13T16:25:21.163366+0000","last_peered":"2024-09-13T16:25:21.163366+0000","last_clean":"2024-09-13T16:25:21.163366+0000","last_became_active":"2024-09-13T16:25:21.156428+0000","last_became_peered":"2024-09-13T16:25:21.156428+0000","last_unstale":"2024-09-13T16:25:21.163366+0000","last_undegraded":"2024-09-13T16:25:21.163366+0000","last_fullsized":"2024-09-13T16:25:21.163366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:31.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950532+0000","last_change":"2024-09-13T16:25:16.932134+0000","last_active":"2024-09-13T16:25:20.950532+0000","last_peered":"2024-09-13T16:25:20.950532+0000","last_clean":"2024-09-13T16:25:20.950532+0000","last_became_active":"2024-09-13T16:25:16.931157+0000","last_became_peered":"2024-09-13T16:25:16.931157+0000","last_unstale":"2024-09-13T16:25:20.950532+0000","last_undegraded":"2024-09-13T16:25:20.950532+0000","last_fullsized":"2024-09-13T16:25:20.950532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150728+0000","last_change":"2024-09-13T16:25:16.560858+0000","last_active":"2024-09-13T16:25:21.150728+0000","last_peered":"2024-09-13T16:25:21.150728+0000","last_clean":"2024-09-13T16:25:21.150728+0000","last_became_active":"2024-09-13T16:25:16.560627+0000","last_became_peered":"2024-09-13T16:25:16.560627+0000","last_unstale":"2024-09-13T16:25:21.150728+0000","last_undegraded":"2024-09-13T16:25:21.150728+0000","last_fullsized":"2024-09-13T16:25:21.150728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:50.678565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158800+0000","last_change":"2024-09-13T16:25:21.155558+0000","last_active":"2024-09-13T16:25:21.158800+0000","last_peered":"2024-09-13T16:25:21.158800+0000","last_clean":"2024-09-13T16:25:21.158800+0000","last_became_active":"2024-09-13T16:25:21.154513+0000","last_became_peered":"2024-09-13T16:25:21.154513+0000","last_unstale":"2024-09-13T16:25:21.158800+0000","last_undegraded":"2024-09-13T16:25:21.158800+0000","last_fullsized":"2024-09-13T16:25:21.158800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943548+0000","last_change":"2024-09-13T16:25:16.934627+0000","last_active":"2024-09-13T16:25:20.943548+0000","last_peered":"2024-09-13T16:25:20.943548+0000","last_clean":"2024-09-13T16:25:20.943548+0000","last_became_active":"2024-09-13T16:25:16.934031+0000","last_became_peered":"2024-09-13T16:25:16.934031+0000","last_unstale":"2024-09-13T16:25:20.943548+0000","last_undegraded":"2024-09-13T16:25:20.943548+0000","last_fullsized":"2024-09-13T16:25:20.943548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:19.244372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950619+0000","last_change":"2024-09-13T16:25:16.550845+0000","last_active":"2024-09-13T16:25:20.950619+0000","last_peered":"2024-09-13T16:25:20.950619+0000","last_clean":"2024-09-13T16:25:20.950619+0000","last_became_active":"2024-09-13T16:25:16.550523+0000","last_became_peered":"2024-09-13T16:25:16.550523+0000","last_unstale":"2024-09-13T16:25:20.950619+0000","last_undegraded":"2024-09-13T16:25:20.950619+0000","last_fullsized":"2024-09-13T16:25:20.950619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971742+0000","last_change":"2024-09-13T16:25:20.963835+0000","last_active":"2024-09-13T16:25:20.971742+0000","last_peered":"2024-09-13T16:25:20.971742+0000","last_clean":"2024-09-13T16:25:20.971742+0000","last_became_active":"2024-09-13T16:25:20.963031+0000","last_became_peered":"2024-09-13T16:25:20.963031+0000","last_unstale":"2024-09-13T16:25:20.971742+0000","last_undegraded":"2024-09-13T16:25:20.971742+0000","last_fullsized":"2024-09-13T16:25:20.971742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:39.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951685+0000","last_change":"2024-09-13T16:25:16.936436+0000","last_active":"2024-09-13T16:25:20.951685+0000","last_peered":"2024-09-13T16:25:20.951685+0000","last_clean":"2024-09-13T16:25:20.951685+0000","last_became_active":"2024-09-13T16:25:16.936131+0000","last_became_peered":"2024-09-13T16:25:16.936131+0000","last_unstale":"2024-09-13T16:25:20.951685+0000","last_undegraded":"2024-09-13T16:25:20.951685+0000","last_fullsized":"2024-09-13T16:25:20.951685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:33.773998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941767+0000","last_change":"2024-09-13T16:25:16.548310+0000","last_active":"2024-09-13T16:25:20.941767+0000","last_peered":"2024-09-13T16:25:20.941767+0000","last_clean":"2024-09-13T16:25:20.941767+0000","last_became_active":"2024-09-13T16:25:16.548056+0000","last_became_peered":"2024-09-13T16:25:16.548056+0000","last_unstale":"2024-09-13T16:25:20.941767+0000","last_undegraded":"2024-09-13T16:25:20.941767+0000","last_fullsized":"2024-09-13T16:25:20.941767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971523+0000","last_change":"2024-09-13T16:25:20.960907+0000","last_active":"2024-09-13T16:25:20.971523+0000","last_peered":"2024-09-13T16:25:20.971523+0000","last_clean":"2024-09-13T16:25:20.971523+0000","last_became_active":"2024-09-13T16:25:20.960504+0000","last_became_peered":"2024-09-13T16:25:20.960504+0000","last_unstale":"2024-09-13T16:25:20.971523+0000","last_undegraded":"2024-09-13T16:25:20.971523+0000","last_fullsized":"2024-09-13T16:25:20.971523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951906+0000","last_change":"2024-09-13T16:25:16.935966+0000","last_active":"2024-09-13T16:25:20.951906+0000","last_peered":"2024-09-13T16:25:20.951906+0000","last_clean":"2024-09-13T16:25:20.951906+0000","last_became_active":"2024-09-13T16:25:16.935728+0000","last_became_peered":"2024-09-13T16:25:16.935728+0000","last_unstale":"2024-09-13T16:25:20.951906+0000","last_undegraded":"2024-09-13T16:25:20.951906+0000","last_fullsized":"2024-09-13T16:25:20.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954151+0000","last_change":"2024-09-13T16:25:16.541632+0000","last_active":"2024-09-13T16:25:20.954151+0000","last_peered":"2024-09-13T16:25:20.954151+0000","last_clean":"2024-09-13T16:25:20.954151+0000","last_became_active":"2024-09-13T16:25:16.541362+0000","last_became_peered":"2024-09-13T16:25:16.541362+0000","last_unstale":"2024-09-13T16:25:20.954151+0000","last_undegraded":"2024-09-13T16:25:20.954151+0000","last_fullsized":"2024-09-13T16:25:20.954151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971468+0000","last_change":"2024-09-13T16:25:20.961579+0000","last_active":"2024-09-13T16:25:20.971468+0000","last_peered":"2024-09-13T16:25:20.971468+0000","last_clean":"2024-09-13T16:25:20.971468+0000","last_became_active":"2024-09-13T16:25:20.961177+0000","last_became_peered":"2024-09-13T16:25:20.961177+0000","last_unstale":"2024-09-13T16:25:20.971468+0000","last_undegraded":"2024-09-13T16:25:20.971468+0000","last_fullsized":"2024-09-13T16:25:20.971468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948340+0000","last_change":"2024-09-13T16:25:16.932105+0000","last_active":"2024-09-13T16:25:20.948340+0000","last_peered":"2024-09-13T16:25:20.948340+0000","last_clean":"2024-09-13T16:25:20.948340+0000","last_became_active":"2024-09-13T16:25:16.931893+0000","last_became_peered":"2024-09-13T16:25:16.931893+0000","last_unstale":"2024-09-13T16:25:20.948340+0000","last_undegraded":"2024-09-13T16:25:20.948340+0000","last_fullsized":"2024-09-13T16:25:20.948340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149350+0000","last_change":"2024-09-13T16:25:16.563013+0000","last_active":"2024-09-13T16:25:21.149350+0000","last_peered":"2024-09-13T16:25:21.149350+0000","last_clean":"2024-09-13T16:25:21.149350+0000","last_became_active":"2024-09-13T16:25:16.562820+0000","last_became_peered":"2024-09-13T16:25:16.562820+0000","last_unstale":"2024-09-13T16:25:21.149350+0000","last_undegraded":"2024-09-13T16:25:21.149350+0000","last_fullsized":"2024-09-13T16:25:21.149350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:43.080203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.161407+0000","last_change":"2024-09-13T16:25:21.156288+0000","last_active":"2024-09-13T16:25:21.161407+0000","last_peered":"2024-09-13T16:25:21.161407+0000","last_clean":"2024-09-13T16:25:21.161407+0000","last_became_active":"2024-09-13T16:25:21.154873+0000","last_became_peered":"2024-09-13T16:25:21.154873+0000","last_unstale":"2024-09-13T16:25:21.161407+0000","last_undegraded":"2024-09-13T16:25:21.161407+0000","last_fullsized":"2024-09-13T16:25:21.161407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:57.390965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941321+0000","last_change":"2024-09-13T16:25:16.934659+0000","last_active":"2024-09-13T16:25:20.941321+0000","last_peered":"2024-09-13T16:25:20.941321+0000","last_clean":"2024-09-13T16:25:20.941321+0000","last_became_active":"2024-09-13T16:25:16.934286+0000","last_became_peered":"2024-09-13T16:25:16.934286+0000","last_unstale":"2024-09-13T16:25:20.941321+0000","last_undegraded":"2024-09-13T16:25:20.941321+0000","last_fullsized":"2024-09-13T16:25:20.941321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941932+0000","last_change":"2024-09-13T16:25:16.548968+0000","last_active":"2024-09-13T16:25:20.941932+0000","last_peered":"2024-09-13T16:25:20.941932+0000","last_clean":"2024-09-13T16:25:20.941932+0000","last_became_active":"2024-09-13T16:25:16.548722+0000","last_became_peered":"2024-09-13T16:25:16.548722+0000","last_unstale":"2024-09-13T16:25:20.941932+0000","last_undegraded":"2024-09-13T16:25:20.941932+0000","last_fullsized":"2024-09-13T16:25:20.941932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:25:08.328680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149477+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:25:21.149477+0000","last_peered":"2024-09-13T16:25:21.149477+0000","last_clean":"2024-09-13T16:25:21.149477+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:25:21.149477+0000","last_undegraded":"2024-09-13T16:25:21.149477+0000","last_fullsized":"2024-09-13T16:25:21.149477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156667+0000","last_change":"2024-09-13T16:25:21.154730+0000","last_active":"2024-09-13T16:25:21.156667+0000","last_peered":"2024-09-13T16:25:21.156667+0000","last_clean":"2024-09-13T16:25:21.156667+0000","last_became_active":"2024-09-13T16:25:21.154226+0000","last_became_peered":"2024-09-13T16:25:21.154226+0000","last_unstale":"2024-09-13T16:25:21.156667+0000","last_undegraded":"2024-09-13T16:25:21.156667+0000","last_fullsized":"2024-09-13T16:25:21.156667+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150204+0000","last_change":"2024-09-13T16:25:16.944787+0000","last_active":"2024-09-13T16:25:21.150204+0000","last_peered":"2024-09-13T16:25:21.150204+0000","last_clean":"2024-09-13T16:25:21.150204+0000","last_became_active":"2024-09-13T16:25:16.944505+0000","last_became_peered":"2024-09-13T16:25:16.944505+0000","last_unstale":"2024-09-13T16:25:21.150204+0000","last_undegraded":"2024-09-13T16:25:21.150204+0000","last_fullsized":"2024-09-13T16:25:21.150204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954746+0000","last_change":"2024-09-13T16:25:16.548195+0000","last_active":"2024-09-13T16:25:20.954746+0000","last_peered":"2024-09-13T16:25:20.954746+0000","last_clean":"2024-09-13T16:25:20.954746+0000","last_became_active":"2024-09-13T16:25:16.547839+0000","last_became_peered":"2024-09-13T16:25:16.547839+0000","last_unstale":"2024-09-13T16:25:20.954746+0000","last_undegraded":"2024-09-13T16:25:20.954746+0000","last_fullsized":"2024-09-13T16:25:20.954746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156778+0000","last_change":"2024-09-13T16:25:21.154890+0000","last_active":"2024-09-13T16:25:21.156778+0000","last_peered":"2024-09-13T16:25:21.156778+0000","last_clean":"2024-09-13T16:25:21.156778+0000","last_became_active":"2024-09-13T16:25:21.154376+0000","last_became_peered":"2024-09-13T16:25:21.154376+0000","last_unstale":"2024-09-13T16:25:21.156778+0000","last_undegraded":"2024-09-13T16:25:21.156778+0000","last_fullsized":"2024-09-13T16:25:21.156778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:09.876152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944021+0000","last_change":"2024-09-13T16:25:16.933523+0000","last_active":"2024-09-13T16:25:20.944021+0000","last_peered":"2024-09-13T16:25:20.944021+0000","last_clean":"2024-09-13T16:25:20.944021+0000","last_became_active":"2024-09-13T16:25:16.932020+0000","last_became_peered":"2024-09-13T16:25:16.932020+0000","last_unstale":"2024-09-13T16:25:20.944021+0000","last_undegraded":"2024-09-13T16:25:20.944021+0000","last_fullsized":"2024-09-13T16:25:20.944021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:34.060136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952497+0000","last_change":"2024-09-13T16:25:16.546857+0000","last_active":"2024-09-13T16:25:20.952497+0000","last_peered":"2024-09-13T16:25:20.952497+0000","last_clean":"2024-09-13T16:25:20.952497+0000","last_became_active":"2024-09-13T16:25:16.546592+0000","last_became_peered":"2024-09-13T16:25:16.546592+0000","last_unstale":"2024-09-13T16:25:20.952497+0000","last_undegraded":"2024-09-13T16:25:20.952497+0000","last_fullsized":"2024-09-13T16:25:20.952497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963301+0000","last_change":"2024-09-13T16:25:20.958682+0000","last_active":"2024-09-13T16:25:20.963301+0000","last_peered":"2024-09-13T16:25:20.963301+0000","last_clean":"2024-09-13T16:25:20.963301+0000","last_became_active":"2024-09-13T16:25:20.958064+0000","last_became_peered":"2024-09-13T16:25:20.958064+0000","last_unstale":"2024-09-13T16:25:20.963301+0000","last_undegraded":"2024-09-13T16:25:20.963301+0000","last_fullsized":"2024-09-13T16:25:20.963301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.502649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150943+0000","last_change":"2024-09-13T16:25:16.951592+0000","last_active":"2024-09-13T16:25:21.150943+0000","last_peered":"2024-09-13T16:25:21.150943+0000","last_clean":"2024-09-13T16:25:21.150943+0000","last_became_active":"2024-09-13T16:25:16.951375+0000","last_became_peered":"2024-09-13T16:25:16.951375+0000","last_unstale":"2024-09-13T16:25:21.150943+0000","last_undegraded":"2024-09-13T16:25:21.150943+0000","last_fullsized":"2024-09-13T16:25:21.150943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:32.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959285+0000","last_change":"2024-09-13T16:25:16.541133+0000","last_active":"2024-09-13T16:25:20.959285+0000","last_peered":"2024-09-13T16:25:20.959285+0000","last_clean":"2024-09-13T16:25:20.959285+0000","last_became_active":"2024-09-13T16:25:16.540843+0000","last_became_peered":"2024-09-13T16:25:16.540843+0000","last_unstale":"2024-09-13T16:25:20.959285+0000","last_undegraded":"2024-09-13T16:25:20.959285+0000","last_fullsized":"2024-09-13T16:25:20.959285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:05.477981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158080+0000","last_change":"2024-09-13T16:25:21.154908+0000","last_active":"2024-09-13T16:25:21.158080+0000","last_peered":"2024-09-13T16:25:21.158080+0000","last_clean":"2024-09-13T16:25:21.158080+0000","last_became_active":"2024-09-13T16:25:21.154365+0000","last_became_peered":"2024-09-13T16:25:21.154365+0000","last_unstale":"2024-09-13T16:25:21.158080+0000","last_undegraded":"2024-09-13T16:25:21.158080+0000","last_fullsized":"2024-09-13T16:25:21.158080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:00.139666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150870+0000","last_change":"2024-09-13T16:25:16.941714+0000","last_active":"2024-09-13T16:25:21.150870+0000","last_peered":"2024-09-13T16:25:21.150870+0000","last_clean":"2024-09-13T16:25:21.150870+0000","last_became_active":"2024-09-13T16:25:16.941465+0000","last_became_peered":"2024-09-13T16:25:16.941465+0000","last_unstale":"2024-09-13T16:25:21.150870+0000","last_undegraded":"2024-09-13T16:25:21.150870+0000","last_fullsized":"2024-09-13T16:25:21.150870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952152+0000","last_change":"2024-09-13T16:25:16.544584+0000","last_active":"2024-09-13T16:25:20.952152+0000","last_peered":"2024-09-13T16:25:20.952152+0000","last_clean":"2024-09-13T16:25:20.952152+0000","last_became_active":"2024-09-13T16:25:16.544275+0000","last_became_peered":"2024-09-13T16:25:16.544275+0000","last_unstale":"2024-09-13T16:25:20.952152+0000","last_undegraded":"2024-09-13T16:25:20.952152+0000","last_fullsized":"2024-09-13T16:25:20.952152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:38.495511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.976715+0000","last_change":"2024-09-13T16:25:20.972282+0000","last_active":"2024-09-13T16:25:20.976715+0000","last_peered":"2024-09-13T16:25:20.976715+0000","last_clean":"2024-09-13T16:25:20.976715+0000","last_became_active":"2024-09-13T16:25:20.971899+0000","last_became_peered":"2024-09-13T16:25:20.971899+0000","last_unstale":"2024-09-13T16:25:20.976715+0000","last_undegraded":"2024-09-13T16:25:20.976715+0000","last_fullsized":"2024-09-13T16:25:20.976715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150689+0000","last_change":"2024-09-13T16:25:16.944388+0000","last_active":"2024-09-13T16:25:21.150689+0000","last_peered":"2024-09-13T16:25:21.150689+0000","last_clean":"2024-09-13T16:25:21.150689+0000","last_became_active":"2024-09-13T16:25:16.943987+0000","last_became_peered":"2024-09-13T16:25:16.943987+0000","last_unstale":"2024-09-13T16:25:21.150689+0000","last_undegraded":"2024-09-13T16:25:21.150689+0000","last_fullsized":"2024-09-13T16:25:21.150689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953615+0000","last_change":"2024-09-13T16:25:16.545908+0000","last_active":"2024-09-13T16:25:20.953615+0000","last_peered":"2024-09-13T16:25:20.953615+0000","last_clean":"2024-09-13T16:25:20.953615+0000","last_became_active":"2024-09-13T16:25:16.545632+0000","last_became_peered":"2024-09-13T16:25:16.545632+0000","last_unstale":"2024-09-13T16:25:20.953615+0000","last_undegraded":"2024-09-13T16:25:20.953615+0000","last_fullsized":"2024-09-13T16:25:20.953615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963693+0000","last_change":"2024-09-13T16:25:20.961076+0000","last_active":"2024-09-13T16:25:20.963693+0000","last_peered":"2024-09-13T16:25:20.963693+0000","last_clean":"2024-09-13T16:25:20.963693+0000","last_became_active":"2024-09-13T16:25:20.960736+0000","last_became_peered":"2024-09-13T16:25:20.960736+0000","last_unstale":"2024-09-13T16:25:20.963693+0000","last_undegraded":"2024-09-13T16:25:20.963693+0000","last_fullsized":"2024-09-13T16:25:20.963693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939386+0000","last_change":"2024-09-13T16:25:16.939543+0000","last_active":"2024-09-13T16:25:20.939386+0000","last_peered":"2024-09-13T16:25:20.939386+0000","last_clean":"2024-09-13T16:25:20.939386+0000","last_became_active":"2024-09-13T16:25:16.939269+0000","last_became_peered":"2024-09-13T16:25:16.939269+0000","last_unstale":"2024-09-13T16:25:20.939386+0000","last_undegraded":"2024-09-13T16:25:20.939386+0000","last_fullsized":"2024-09-13T16:25:20.939386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939280+0000","last_change":"2024-09-13T16:25:16.557172+0000","last_active":"2024-09-13T16:25:20.939280+0000","last_peered":"2024-09-13T16:25:20.939280+0000","last_clean":"2024-09-13T16:25:20.939280+0000","last_became_active":"2024-09-13T16:25:16.556910+0000","last_became_peered":"2024-09-13T16:25:16.556910+0000","last_unstale":"2024-09-13T16:25:20.939280+0000","last_undegraded":"2024-09-13T16:25:20.939280+0000","last_fullsized":"2024-09-13T16:25:20.939280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162791+0000","last_change":"2024-09-13T16:25:21.154750+0000","last_active":"2024-09-13T16:25:21.162791+0000","last_peered":"2024-09-13T16:25:21.162791+0000","last_clean":"2024-09-13T16:25:21.162791+0000","last_became_active":"2024-09-13T16:25:21.153987+0000","last_became_peered":"2024-09-13T16:25:21.153987+0000","last_unstale":"2024-09-13T16:25:21.162791+0000","last_undegraded":"2024-09-13T16:25:21.162791+0000","last_fullsized":"2024-09-13T16:25:21.162791+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149825+0000","last_change":"2024-09-13T16:25:16.944291+0000","last_active":"2024-09-13T16:25:21.149825+0000","last_peered":"2024-09-13T16:25:21.149825+0000","last_clean":"2024-09-13T16:25:21.149825+0000","last_became_active":"2024-09-13T16:25:16.941709+0000","last_became_peered":"2024-09-13T16:25:16.941709+0000","last_unstale":"2024-09-13T16:25:21.149825+0000","last_undegraded":"2024-09-13T16:25:21.149825+0000","last_fullsized":"2024-09-13T16:25:21.149825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:45.537446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941804+0000","last_change":"2024-09-13T16:25:16.553038+0000","last_active":"2024-09-13T16:25:20.941804+0000","last_peered":"2024-09-13T16:25:20.941804+0000","last_clean":"2024-09-13T16:25:20.941804+0000","last_became_active":"2024-09-13T16:25:16.551960+0000","last_became_peered":"2024-09-13T16:25:16.551960+0000","last_unstale":"2024-09-13T16:25:20.941804+0000","last_undegraded":"2024-09-13T16:25:20.941804+0000","last_fullsized":"2024-09-13T16:25:20.941804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:27.632855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163399+0000","last_change":"2024-09-13T16:25:21.158491+0000","last_active":"2024-09-13T16:25:21.163399+0000","last_peered":"2024-09-13T16:25:21.163399+0000","last_clean":"2024-09-13T16:25:21.163399+0000","last_became_active":"2024-09-13T16:25:21.158133+0000","last_became_peered":"2024-09-13T16:25:21.158133+0000","last_unstale":"2024-09-13T16:25:21.163399+0000","last_undegraded":"2024-09-13T16:25:21.163399+0000","last_fullsized":"2024-09-13T16:25:21.163399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946925+0000","last_change":"2024-09-13T16:25:16.941874+0000","last_active":"2024-09-13T16:25:20.946925+0000","last_peered":"2024-09-13T16:25:20.946925+0000","last_clean":"2024-09-13T16:25:20.946925+0000","last_became_active":"2024-09-13T16:25:16.941683+0000","last_became_peered":"2024-09-13T16:25:16.941683+0000","last_unstale":"2024-09-13T16:25:20.946925+0000","last_undegraded":"2024-09-13T16:25:20.946925+0000","last_fullsized":"2024-09-13T16:25:20.946925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953752+0000","last_change":"2024-09-13T16:25:16.551439+0000","last_active":"2024-09-13T16:25:20.953752+0000","last_peered":"2024-09-13T16:25:20.953752+0000","last_clean":"2024-09-13T16:25:20.953752+0000","last_became_active":"2024-09-13T16:25:16.550937+0000","last_became_peered":"2024-09-13T16:25:16.550937+0000","last_unstale":"2024-09-13T16:25:20.953752+0000","last_undegraded":"2024-09-13T16:25:20.953752+0000","last_fullsized":"2024-09-13T16:25:20.953752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:14.983082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158057+0000","last_change":"2024-09-13T16:25:21.155556+0000","last_active":"2024-09-13T16:25:21.158057+0000","last_peered":"2024-09-13T16:25:21.158057+0000","last_clean":"2024-09-13T16:25:21.158057+0000","last_became_active":"2024-09-13T16:25:21.155201+0000","last_became_peered":"2024-09-13T16:25:21.155201+0000","last_unstale":"2024-09-13T16:25:21.158057+0000","last_undegraded":"2024-09-13T16:25:21.158057+0000","last_fullsized":"2024-09-13T16:25:21.158057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:42.823106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150192+0000","last_change":"2024-09-13T16:25:16.944770+0000","last_active":"2024-09-13T16:25:21.150192+0000","last_peered":"2024-09-13T16:25:21.150192+0000","last_clean":"2024-09-13T16:25:21.150192+0000","last_became_active":"2024-09-13T16:25:16.944455+0000","last_became_peered":"2024-09-13T16:25:16.944455+0000","last_unstale":"2024-09-13T16:25:21.150192+0000","last_undegraded":"2024-09-13T16:25:21.150192+0000","last_fullsized":"2024-09-13T16:25:21.150192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:41:40.454800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-13T16:25:26.199825+0000","last_change":"2024-09-13T16:25:16.552517+0000","last_active":"2024-09-13T16:25:26.199825+0000","last_peered":"2024-09-13T16:25:26.199825+0000","last_clean":"2024-09-13T16:25:26.199825+0000","last_became_active":"2024-09-13T16:25:16.552101+0000","last_became_peered":"2024-09-13T16:25:16.552101+0000","last_unstale":"2024-09-13T16:25:26.199825+0000","last_undegraded":"2024-09-13T16:25:26.199825+0000","last_fullsized":"2024-09-13T16:25:26.199825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971715+0000","last_change":"2024-09-13T16:25:20.963516+0000","last_active":"2024-09-13T16:25:20.971715+0000","last_peered":"2024-09-13T16:25:20.971715+0000","last_clean":"2024-09-13T16:25:20.971715+0000","last_became_active":"2024-09-13T16:25:20.962518+0000","last_became_peered":"2024-09-13T16:25:20.962518+0000","last_unstale":"2024-09-13T16:25:20.971715+0000","last_undegraded":"2024-09-13T16:25:20.971715+0000","last_fullsized":"2024-09-13T16:25:20.971715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:05.557712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951494+0000","last_change":"2024-09-13T16:25:16.941833+0000","last_active":"2024-09-13T16:25:20.951494+0000","last_peered":"2024-09-13T16:25:20.951494+0000","last_clean":"2024-09-13T16:25:20.951494+0000","last_became_active":"2024-09-13T16:25:16.941607+0000","last_became_peered":"2024-09-13T16:25:16.941607+0000","last_unstale":"2024-09-13T16:25:20.951494+0000","last_undegraded":"2024-09-13T16:25:20.951494+0000","last_fullsized":"2024-09-13T16:25:20.951494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:44.089730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952414+0000","last_change":"2024-09-13T16:25:16.549878+0000","last_active":"2024-09-13T16:25:20.952414+0000","last_peered":"2024-09-13T16:25:20.952414+0000","last_clean":"2024-09-13T16:25:20.952414+0000","last_became_active":"2024-09-13T16:25:16.548581+0000","last_became_peered":"2024-09-13T16:25:16.548581+0000","last_unstale":"2024-09-13T16:25:20.952414+0000","last_undegraded":"2024-09-13T16:25:20.952414+0000","last_fullsized":"2024-09-13T16:25:20.952414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969433+0000","last_change":"2024-09-13T16:25:20.965523+0000","last_active":"2024-09-13T16:25:20.969433+0000","last_peered":"2024-09-13T16:25:20.969433+0000","last_clean":"2024-09-13T16:25:20.969433+0000","last_became_active":"2024-09-13T16:25:20.965035+0000","last_became_peered":"2024-09-13T16:25:20.965035+0000","last_unstale":"2024-09-13T16:25:20.969433+0000","last_undegraded":"2024-09-13T16:25:20.969433+0000","last_fullsized":"2024-09-13T16:25:20.969433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954066+0000","last_change":"2024-09-13T16:25:16.933522+0000","last_active":"2024-09-13T16:25:20.954066+0000","last_peered":"2024-09-13T16:25:20.954066+0000","last_clean":"2024-09-13T16:25:20.954066+0000","last_became_active":"2024-09-13T16:25:16.933134+0000","last_became_peered":"2024-09-13T16:25:16.933134+0000","last_unstale":"2024-09-13T16:25:20.954066+0000","last_undegraded":"2024-09-13T16:25:20.954066+0000","last_fullsized":"2024-09-13T16:25:20.954066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942876+0000","last_change":"2024-09-13T16:25:16.549756+0000","last_active":"2024-09-13T16:25:20.942876+0000","last_peered":"2024-09-13T16:25:20.942876+0000","last_clean":"2024-09-13T16:25:20.942876+0000","last_became_active":"2024-09-13T16:25:16.549515+0000","last_became_peered":"2024-09-13T16:25:16.549515+0000","last_unstale":"2024-09-13T16:25:20.942876+0000","last_undegraded":"2024-09-13T16:25:20.942876+0000","last_fullsized":"2024-09-13T16:25:20.942876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963576+0000","last_change":"2024-09-13T16:25:20.960084+0000","last_active":"2024-09-13T16:25:20.963576+0000","last_peered":"2024-09-13T16:25:20.963576+0000","last_clean":"2024-09-13T16:25:20.963576+0000","last_became_active":"2024-09-13T16:25:20.957343+0000","last_became_peered":"2024-09-13T16:25:20.957343+0000","last_unstale":"2024-09-13T16:25:20.963576+0000","last_undegraded":"2024-09-13T16:25:20.963576+0000","last_fullsized":"2024-09-13T16:25:20.963576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:06.115184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942738+0000","last_change":"2024-09-13T16:25:16.942162+0000","last_active":"2024-09-13T16:25:20.942738+0000","last_peered":"2024-09-13T16:25:20.942738+0000","last_clean":"2024-09-13T16:25:20.942738+0000","last_became_active":"2024-09-13T16:25:16.941896+0000","last_became_peered":"2024-09-13T16:25:16.941896+0000","last_unstale":"2024-09-13T16:25:20.942738+0000","last_undegraded":"2024-09-13T16:25:20.942738+0000","last_fullsized":"2024-09-13T16:25:20.942738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:24.023742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941934+0000","last_change":"2024-09-13T16:25:16.551442+0000","last_active":"2024-09-13T16:25:20.941934+0000","last_peered":"2024-09-13T16:25:20.941934+0000","last_clean":"2024-09-13T16:25:20.941934+0000","last_became_active":"2024-09-13T16:25:16.551084+0000","last_became_peered":"2024-09-13T16:25:16.551084+0000","last_unstale":"2024-09-13T16:25:20.941934+0000","last_undegraded":"2024-09-13T16:25:20.941934+0000","last_fullsized":"2024-09-13T16:25:20.941934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163416+0000","last_change":"2024-09-13T16:25:21.157528+0000","last_active":"2024-09-13T16:25:21.163416+0000","last_peered":"2024-09-13T16:25:21.163416+0000","last_clean":"2024-09-13T16:25:21.163416+0000","last_became_active":"2024-09-13T16:25:21.155560+0000","last_became_peered":"2024-09-13T16:25:21.155560+0000","last_unstale":"2024-09-13T16:25:21.163416+0000","last_undegraded":"2024-09-13T16:25:21.163416+0000","last_fullsized":"2024-09-13T16:25:21.163416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162596+0000","last_change":"2024-09-13T16:25:21.158060+0000","last_active":"2024-09-13T16:25:21.162596+0000","last_peered":"2024-09-13T16:25:21.162596+0000","last_clean":"2024-09-13T16:25:21.162596+0000","last_became_active":"2024-09-13T16:25:21.156882+0000","last_became_peered":"2024-09-13T16:25:21.156882+0000","last_unstale":"2024-09-13T16:25:21.162596+0000","last_undegraded":"2024-09-13T16:25:21.162596+0000","last_fullsized":"2024-09-13T16:25:21.162596+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949704+0000","last_change":"2024-09-13T16:25:16.932645+0000","last_active":"2024-09-13T16:25:20.949704+0000","last_peered":"2024-09-13T16:25:20.949704+0000","last_clean":"2024-09-13T16:25:20.949704+0000","last_became_active":"2024-09-13T16:25:16.931775+0000","last_became_peered":"2024-09-13T16:25:16.931775+0000","last_unstale":"2024-09-13T16:25:20.949704+0000","last_undegraded":"2024-09-13T16:25:20.949704+0000","last_fullsized":"2024-09-13T16:25:20.949704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942164+0000","last_change":"2024-09-13T16:25:16.547336+0000","last_active":"2024-09-13T16:25:20.942164+0000","last_peered":"2024-09-13T16:25:20.942164+0000","last_clean":"2024-09-13T16:25:20.942164+0000","last_became_active":"2024-09-13T16:25:16.547062+0000","last_became_peered":"2024-09-13T16:25:16.547062+0000","last_unstale":"2024-09-13T16:25:20.942164+0000","last_undegraded":"2024-09-13T16:25:20.942164+0000","last_fullsized":"2024-09-13T16:25:20.942164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.968355+0000","last_change":"2024-09-13T16:25:20.965354+0000","last_active":"2024-09-13T16:25:20.968355+0000","last_peered":"2024-09-13T16:25:20.968355+0000","last_clean":"2024-09-13T16:25:20.968355+0000","last_became_active":"2024-09-13T16:25:20.964985+0000","last_became_peered":"2024-09-13T16:25:20.964985+0000","last_unstale":"2024-09-13T16:25:20.968355+0000","last_undegraded":"2024-09-13T16:25:20.968355+0000","last_fullsized":"2024-09-13T16:25:20.968355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:42.840413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939763+0000","last_change":"2024-09-13T16:25:16.929034+0000","last_active":"2024-09-13T16:25:20.939763+0000","last_peered":"2024-09-13T16:25:20.939763+0000","last_clean":"2024-09-13T16:25:20.939763+0000","last_became_active":"2024-09-13T16:25:16.928782+0000","last_became_peered":"2024-09-13T16:25:16.928782+0000","last_unstale":"2024-09-13T16:25:20.939763+0000","last_undegraded":"2024-09-13T16:25:20.939763+0000","last_fullsized":"2024-09-13T16:25:20.939763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940092+0000","last_change":"2024-09-13T16:25:16.548923+0000","last_active":"2024-09-13T16:25:20.940092+0000","last_peered":"2024-09-13T16:25:20.940092+0000","last_clean":"2024-09-13T16:25:20.940092+0000","last_became_active":"2024-09-13T16:25:16.545843+0000","last_became_peered":"2024-09-13T16:25:16.545843+0000","last_unstale":"2024-09-13T16:25:20.940092+0000","last_undegraded":"2024-09-13T16:25:20.940092+0000","last_fullsized":"2024-09-13T16:25:20.940092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939883+0000","last_change":"2024-09-13T16:25:16.555750+0000","last_active":"2024-09-13T16:25:20.939883+0000","last_peered":"2024-09-13T16:25:20.939883+0000","last_clean":"2024-09-13T16:25:20.939883+0000","last_became_active":"2024-09-13T16:25:16.555480+0000","last_became_peered":"2024-09-13T16:25:16.555480+0000","last_unstale":"2024-09-13T16:25:20.939883+0000","last_undegraded":"2024-09-13T16:25:20.939883+0000","last_fullsized":"2024-09-13T16:25:20.939883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953647+0000","last_change":"2024-09-13T16:25:16.934605+0000","last_active":"2024-09-13T16:25:20.953647+0000","last_peered":"2024-09-13T16:25:20.953647+0000","last_clean":"2024-09-13T16:25:20.953647+0000","last_became_active":"2024-09-13T16:25:16.934365+0000","last_became_peered":"2024-09-13T16:25:16.934365+0000","last_unstale":"2024-09-13T16:25:20.953647+0000","last_undegraded":"2024-09-13T16:25:20.953647+0000","last_fullsized":"2024-09-13T16:25:20.953647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.967920+0000","last_change":"2024-09-13T16:25:20.965561+0000","last_active":"2024-09-13T16:25:20.967920+0000","last_peered":"2024-09-13T16:25:20.967920+0000","last_clean":"2024-09-13T16:25:20.967920+0000","last_became_active":"2024-09-13T16:25:20.965162+0000","last_became_peered":"2024-09-13T16:25:20.965162+0000","last_unstale":"2024-09-13T16:25:20.967920+0000","last_undegraded":"2024-09-13T16:25:20.967920+0000","last_fullsized":"2024-09-13T16:25:20.967920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:35.284928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940864+0000","last_change":"2024-09-13T16:25:16.560967+0000","last_active":"2024-09-13T16:25:20.940864+0000","last_peered":"2024-09-13T16:25:20.940864+0000","last_clean":"2024-09-13T16:25:20.940864+0000","last_became_active":"2024-09-13T16:25:16.560675+0000","last_became_peered":"2024-09-13T16:25:16.560675+0000","last_unstale":"2024-09-13T16:25:20.940864+0000","last_undegraded":"2024-09-13T16:25:20.940864+0000","last_fullsized":"2024-09-13T16:25:20.940864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951187+0000","last_change":"2024-09-13T16:25:16.936540+0000","last_active":"2024-09-13T16:25:20.951187+0000","last_peered":"2024-09-13T16:25:20.951187+0000","last_clean":"2024-09-13T16:25:20.951187+0000","last_became_active":"2024-09-13T16:25:16.936269+0000","last_became_peered":"2024-09-13T16:25:16.936269+0000","last_unstale":"2024-09-13T16:25:20.951187+0000","last_undegraded":"2024-09-13T16:25:20.951187+0000","last_fullsized":"2024-09-13T16:25:20.951187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:18.089801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963605+0000","last_change":"2024-09-13T16:25:20.961207+0000","last_active":"2024-09-13T16:25:20.963605+0000","last_peered":"2024-09-13T16:25:20.963605+0000","last_clean":"2024-09-13T16:25:20.963605+0000","last_became_active":"2024-09-13T16:25:20.960801+0000","last_became_peered":"2024-09-13T16:25:20.960801+0000","last_unstale":"2024-09-13T16:25:20.963605+0000","last_undegraded":"2024-09-13T16:25:20.963605+0000","last_fullsized":"2024-09-13T16:25:20.963605+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:38.874981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939239+0000","last_change":"2024-09-13T16:25:16.939880+0000","last_active":"2024-09-13T16:25:20.939239+0000","last_peered":"2024-09-13T16:25:20.939239+0000","last_clean":"2024-09-13T16:25:20.939239+0000","last_became_active":"2024-09-13T16:25:16.939579+0000","last_became_peered":"2024-09-13T16:25:16.939579+0000","last_unstale":"2024-09-13T16:25:20.939239+0000","last_undegraded":"2024-09-13T16:25:20.939239+0000","last_fullsized":"2024-09-13T16:25:20.939239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:37.417453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953935+0000","last_change":"2024-09-13T16:25:16.938944+0000","last_active":"2024-09-13T16:25:20.953935+0000","last_peered":"2024-09-13T16:25:20.953935+0000","last_clean":"2024-09-13T16:25:20.953935+0000","last_became_active":"2024-09-13T16:25:16.938658+0000","last_became_peered":"2024-09-13T16:25:16.938658+0000","last_unstale":"2024-09-13T16:25:20.953935+0000","last_undegraded":"2024-09-13T16:25:20.953935+0000","last_fullsized":"2024-09-13T16:25:20.953935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:50.130499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941580+0000","last_change":"2024-09-13T16:25:16.933588+0000","last_active":"2024-09-13T16:25:20.941580+0000","last_peered":"2024-09-13T16:25:20.941580+0000","last_clean":"2024-09-13T16:25:20.941580+0000","last_became_active":"2024-09-13T16:25:16.932300+0000","last_became_peered":"2024-09-13T16:25:16.932300+0000","last_unstale":"2024-09-13T16:25:20.941580+0000","last_undegraded":"2024-09-13T16:25:20.941580+0000","last_fullsized":"2024-09-13T16:25:20.941580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:43.437120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953565+0000","last_change":"2024-09-13T16:25:16.934119+0000","last_active":"2024-09-13T16:25:20.953565+0000","last_peered":"2024-09-13T16:25:20.953565+0000","last_clean":"2024-09-13T16:25:20.953565+0000","last_became_active":"2024-09-13T16:25:16.930259+0000","last_became_peered":"2024-09-13T16:25:16.930259+0000","last_unstale":"2024-09-13T16:25:20.953565+0000","last_undegraded":"2024-09-13T16:25:20.953565+0000","last_fullsized":"2024-09-13T16:25:20.953565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941316+0000","last_change":"2024-09-13T16:25:16.931922+0000","last_active":"2024-09-13T16:25:20.941316+0000","last_peered":"2024-09-13T16:25:20.941316+0000","last_clean":"2024-09-13T16:25:20.941316+0000","last_became_active":"2024-09-13T16:25:16.931678+0000","last_became_peered":"2024-09-13T16:25:16.931678+0000","last_unstale":"2024-09-13T16:25:20.941316+0000","last_undegraded":"2024-09-13T16:25:20.941316+0000","last_fullsized":"2024-09-13T16:25:20.941316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:27.822409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952316+0000","last_change":"2024-09-13T16:25:16.932526+0000","last_active":"2024-09-13T16:25:20.952316+0000","last_peered":"2024-09-13T16:25:20.952316+0000","last_clean":"2024-09-13T16:25:20.952316+0000","last_became_active":"2024-09-13T16:25:16.932189+0000","last_became_peered":"2024-09-13T16:25:16.932189+0000","last_unstale":"2024-09-13T16:25:20.952316+0000","last_undegraded":"2024-09-13T16:25:20.952316+0000","last_fullsized":"2024-09-13T16:25:20.952316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941353+0000","last_change":"2024-09-13T16:25:16.947538+0000","last_active":"2024-09-13T16:25:20.941353+0000","last_peered":"2024-09-13T16:25:20.941353+0000","last_clean":"2024-09-13T16:25:20.941353+0000","last_became_active":"2024-09-13T16:25:16.947272+0000","last_became_peered":"2024-09-13T16:25:16.947272+0000","last_unstale":"2024-09-13T16:25:20.941353+0000","last_undegraded":"2024-09-13T16:25:20.941353+0000","last_fullsized":"2024-09-13T16:25:20.941353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940064+0000","last_change":"2024-09-13T16:25:16.932732+0000","last_active":"2024-09-13T16:25:20.940064+0000","last_peered":"2024-09-13T16:25:20.940064+0000","last_clean":"2024-09-13T16:25:20.940064+0000","last_became_active":"2024-09-13T16:25:16.932505+0000","last_became_peered":"2024-09-13T16:25:16.932505+0000","last_unstale":"2024-09-13T16:25:20.940064+0000","last_undegraded":"2024-09-13T16:25:20.940064+0000","last_fullsized":"2024-09-13T16:25:20.940064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942122+0000","last_change":"2024-09-13T16:25:16.933750+0000","last_active":"2024-09-13T16:25:20.942122+0000","last_peered":"2024-09-13T16:25:20.942122+0000","last_clean":"2024-09-13T16:25:20.942122+0000","last_became_active":"2024-09-13T16:25:16.933509+0000","last_became_peered":"2024-09-13T16:25:16.933509+0000","last_unstale":"2024-09-13T16:25:20.942122+0000","last_undegraded":"2024-09-13T16:25:20.942122+0000","last_fullsized":"2024-09-13T16:25:20.942122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:28.433477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939801+0000","last_change":"2024-09-13T16:25:16.932135+0000","last_active":"2024-09-13T16:25:20.939801+0000","last_peered":"2024-09-13T16:25:20.939801+0000","last_clean":"2024-09-13T16:25:20.939801+0000","last_became_active":"2024-09-13T16:25:16.931865+0000","last_became_peered":"2024-09-13T16:25:16.931865+0000","last_unstale":"2024-09-13T16:25:20.939801+0000","last_undegraded":"2024-09-13T16:25:20.939801+0000","last_fullsized":"2024-09-13T16:25:20.939801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941544+0000","last_change":"2024-09-13T16:25:16.942044+0000","last_active":"2024-09-13T16:25:20.941544+0000","last_peered":"2024-09-13T16:25:20.941544+0000","last_clean":"2024-09-13T16:25:20.941544+0000","last_became_active":"2024-09-13T16:25:16.941738+0000","last_became_peered":"2024-09-13T16:25:16.941738+0000","last_unstale":"2024-09-13T16:25:20.941544+0000","last_undegraded":"2024-09-13T16:25:20.941544+0000","last_fullsized":"2024-09-13T16:25:20.941544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:58:43.832593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948993+0000","last_change":"2024-09-13T16:25:16.942581+0000","last_active":"2024-09-13T16:25:20.948993+0000","last_peered":"2024-09-13T16:25:20.948993+0000","last_clean":"2024-09-13T16:25:20.948993+0000","last_became_active":"2024-09-13T16:25:16.942321+0000","last_became_peered":"2024-09-13T16:25:16.942321+0000","last_unstale":"2024-09-13T16:25:20.948993+0000","last_undegraded":"2024-09-13T16:25:20.948993+0000","last_fullsized":"2024-09-13T16:25:20.948993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939626+0000","last_change":"2024-09-13T16:25:16.936450+0000","last_active":"2024-09-13T16:25:20.939626+0000","last_peered":"2024-09-13T16:25:20.939626+0000","last_clean":"2024-09-13T16:25:20.939626+0000","last_became_active":"2024-09-13T16:25:16.936084+0000","last_became_peered":"2024-09-13T16:25:16.936084+0000","last_unstale":"2024-09-13T16:25:20.939626+0000","last_undegraded":"2024-09-13T16:25:20.939626+0000","last_fullsized":"2024-09-13T16:25:20.939626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150470+0000","last_change":"2024-09-13T16:25:16.948876+0000","last_active":"2024-09-13T16:25:21.150470+0000","last_peered":"2024-09-13T16:25:21.150470+0000","last_clean":"2024-09-13T16:25:21.150470+0000","last_became_active":"2024-09-13T16:25:16.948536+0000","last_became_peered":"2024-09-13T16:25:16.948536+0000","last_unstale":"2024-09-13T16:25:21.150470+0000","last_undegraded":"2024-09-13T16:25:21.150470+0000","last_fullsized":"2024-09-13T16:25:21.150470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959182+0000","last_change":"2024-09-13T16:25:16.937371+0000","last_active":"2024-09-13T16:25:20.959182+0000","last_peered":"2024-09-13T16:25:20.959182+0000","last_clean":"2024-09-13T16:25:20.959182+0000","last_became_active":"2024-09-13T16:25:16.937130+0000","last_became_peered":"2024-09-13T16:25:16.937130+0000","last_unstale":"2024-09-13T16:25:20.959182+0000","last_undegraded":"2024-09-13T16:25:20.959182+0000","last_fullsized":"2024-09-13T16:25:20.959182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:29.816675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944314+0000","last_change":"2024-09-13T16:25:16.947064+0000","last_active":"2024-09-13T16:25:20.944314+0000","last_peered":"2024-09-13T16:25:20.944314+0000","last_clean":"2024-09-13T16:25:20.944314+0000","last_became_active":"2024-09-13T16:25:16.946790+0000","last_became_peered":"2024-09-13T16:25:16.946790+0000","last_unstale":"2024-09-13T16:25:20.944314+0000","last_undegraded":"2024-09-13T16:25:20.944314+0000","last_fullsized":"2024-09-13T16:25:20.944314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.978002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942037+0000","last_change":"2024-09-13T16:25:16.931584+0000","last_active":"2024-09-13T16:25:20.942037+0000","last_peered":"2024-09-13T16:25:20.942037+0000","last_clean":"2024-09-13T16:25:20.942037+0000","last_became_active":"2024-09-13T16:25:16.931224+0000","last_became_peered":"2024-09-13T16:25:16.931224+0000","last_unstale":"2024-09-13T16:25:20.942037+0000","last_undegraded":"2024-09-13T16:25:20.942037+0000","last_fullsized":"2024-09-13T16:25:20.942037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:11.767256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949799+0000","last_change":"2024-09-13T16:25:16.936233+0000","last_active":"2024-09-13T16:25:20.949799+0000","last_peered":"2024-09-13T16:25:20.949799+0000","last_clean":"2024-09-13T16:25:20.949799+0000","last_became_active":"2024-09-13T16:25:16.935935+0000","last_became_peered":"2024-09-13T16:25:16.935935+0000","last_unstale":"2024-09-13T16:25:20.949799+0000","last_undegraded":"2024-09-13T16:25:20.949799+0000","last_fullsized":"2024-09-13T16:25:20.949799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946795+0000","last_change":"2024-09-13T16:25:16.936516+0000","last_active":"2024-09-13T16:25:20.946795+0000","last_peered":"2024-09-13T16:25:20.946795+0000","last_clean":"2024-09-13T16:25:20.946795+0000","last_became_active":"2024-09-13T16:25:16.936281+0000","last_became_peered":"2024-09-13T16:25:16.936281+0000","last_unstale":"2024-09-13T16:25:20.946795+0000","last_undegraded":"2024-09-13T16:25:20.946795+0000","last_fullsized":"2024-09-13T16:25:20.946795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940457+0000","last_change":"2024-09-13T16:25:16.938439+0000","last_active":"2024-09-13T16:25:20.940457+0000","last_peered":"2024-09-13T16:25:20.940457+0000","last_clean":"2024-09-13T16:25:20.940457+0000","last_became_active":"2024-09-13T16:25:16.938204+0000","last_became_peered":"2024-09-13T16:25:16.938204+0000","last_unstale":"2024-09-13T16:25:20.940457+0000","last_undegraded":"2024-09-13T16:25:20.940457+0000","last_fullsized":"2024-09-13T16:25:20.940457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951414+0000","last_change":"2024-09-13T16:25:16.931605+0000","last_active":"2024-09-13T16:25:20.951414+0000","last_peered":"2024-09-13T16:25:20.951414+0000","last_clean":"2024-09-13T16:25:20.951414+0000","last_became_active":"2024-09-13T16:25:16.931330+0000","last_became_peered":"2024-09-13T16:25:16.931330+0000","last_unstale":"2024-09-13T16:25:20.951414+0000","last_undegraded":"2024-09-13T16:25:20.951414+0000","last_fullsized":"2024-09-13T16:25:20.951414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:51.688093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940628+0000","last_change":"2024-09-13T16:25:16.936567+0000","last_active":"2024-09-13T16:25:20.940628+0000","last_peered":"2024-09-13T16:25:20.940628+0000","last_clean":"2024-09-13T16:25:20.940628+0000","last_became_active":"2024-09-13T16:25:16.936076+0000","last_became_peered":"2024-09-13T16:25:16.936076+0000","last_unstale":"2024-09-13T16:25:20.940628+0000","last_undegraded":"2024-09-13T16:25:20.940628+0000","last_fullsized":"2024-09-13T16:25:20.940628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941217+0000","last_change":"2024-09-13T16:25:16.936360+0000","last_active":"2024-09-13T16:25:20.941217+0000","last_peered":"2024-09-13T16:25:20.941217+0000","last_clean":"2024-09-13T16:25:20.941217+0000","last_became_active":"2024-09-13T16:25:16.935361+0000","last_became_peered":"2024-09-13T16:25:16.935361+0000","last_unstale":"2024-09-13T16:25:20.941217+0000","last_undegraded":"2024-09-13T16:25:20.941217+0000","last_fullsized":"2024-09-13T16:25:20.941217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150991+0000","last_change":"2024-09-13T16:25:16.951709+0000","last_active":"2024-09-13T16:25:21.150991+0000","last_peered":"2024-09-13T16:25:21.150991+0000","last_clean":"2024-09-13T16:25:21.150991+0000","last_became_active":"2024-09-13T16:25:16.951465+0000","last_became_peered":"2024-09-13T16:25:16.951465+0000","last_unstale":"2024-09-13T16:25:21.150991+0000","last_undegraded":"2024-09-13T16:25:21.150991+0000","last_fullsized":"2024-09-13T16:25:21.150991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:05.832843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.938568+0000","last_change":"2024-09-13T16:25:16.933931+0000","last_active":"2024-09-13T16:25:20.938568+0000","last_peered":"2024-09-13T16:25:20.938568+0000","last_clean":"2024-09-13T16:25:20.938568+0000","last_became_active":"2024-09-13T16:25:16.933646+0000","last_became_peered":"2024-09-13T16:25:16.933646+0000","last_unstale":"2024-09-13T16:25:20.938568+0000","last_undegraded":"2024-09-13T16:25:20.938568+0000","last_fullsized":"2024-09-13T16:25:20.938568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150544+0000","last_change":"2024-09-13T16:25:16.941658+0000","last_active":"2024-09-13T16:25:21.150544+0000","last_peered":"2024-09-13T16:25:21.150544+0000","last_clean":"2024-09-13T16:25:21.150544+0000","last_became_active":"2024-09-13T16:25:16.941375+0000","last_became_peered":"2024-09-13T16:25:16.941375+0000","last_unstale":"2024-09-13T16:25:21.150544+0000","last_undegraded":"2024-09-13T16:25:21.150544+0000","last_fullsized":"2024-09-13T16:25:21.150544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:05.301510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941428+0000","last_change":"2024-09-13T16:25:16.931253+0000","last_active":"2024-09-13T16:25:20.941428+0000","last_peered":"2024-09-13T16:25:20.941428+0000","last_clean":"2024-09-13T16:25:20.941428+0000","last_became_active":"2024-09-13T16:25:16.930844+0000","last_became_peered":"2024-09-13T16:25:16.930844+0000","last_unstale":"2024-09-13T16:25:20.941428+0000","last_undegraded":"2024-09-13T16:25:20.941428+0000","last_fullsized":"2024-09-13T16:25:20.941428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:11.873813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149733+0000","last_change":"2024-09-13T16:25:16.946470+0000","last_active":"2024-09-13T16:25:21.149733+0000","last_peered":"2024-09-13T16:25:21.149733+0000","last_clean":"2024-09-13T16:25:21.149733+0000","last_became_active":"2024-09-13T16:25:16.946228+0000","last_became_peered":"2024-09-13T16:25:16.946228+0000","last_unstale":"2024-09-13T16:25:21.149733+0000","last_undegraded":"2024-09-13T16:25:21.149733+0000","last_fullsized":"2024-09-13T16:25:21.149733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:51.187953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943747+0000","last_change":"2024-09-13T16:25:16.933763+0000","last_active":"2024-09-13T16:25:20.943747+0000","last_peered":"2024-09-13T16:25:20.943747+0000","last_clean":"2024-09-13T16:25:20.943747+0000","last_became_active":"2024-09-13T16:25:16.932396+0000","last_became_peered":"2024-09-13T16:25:16.932396+0000","last_unstale":"2024-09-13T16:25:20.943747+0000","last_undegraded":"2024-09-13T16:25:20.943747+0000","last_fullsized":"2024-09-13T16:25:20.943747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:13.889786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943715+0000","last_change":"2024-09-13T16:25:16.931365+0000","last_active":"2024-09-13T16:25:20.943715+0000","last_peered":"2024-09-13T16:25:20.943715+0000","last_clean":"2024-09-13T16:25:20.943715+0000","last_became_active":"2024-09-13T16:25:16.931117+0000","last_became_peered":"2024-09-13T16:25:16.931117+0000","last_unstale":"2024-09-13T16:25:20.943715+0000","last_undegraded":"2024-09-13T16:25:20.943715+0000","last_fullsized":"2024-09-13T16:25:20.943715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.578965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953517+0000","last_change":"2024-09-13T16:25:16.932889+0000","last_active":"2024-09-13T16:25:20.953517+0000","last_peered":"2024-09-13T16:25:20.953517+0000","last_clean":"2024-09-13T16:25:20.953517+0000","last_became_active":"2024-09-13T16:25:16.931952+0000","last_became_peered":"2024-09-13T16:25:16.931952+0000","last_unstale":"2024-09-13T16:25:20.953517+0000","last_undegraded":"2024-09-13T16:25:20.953517+0000","last_fullsized":"2024-09-13T16:25:20.953517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150702+0000","last_change":"2024-09-13T16:25:16.948699+0000","last_active":"2024-09-13T16:25:21.150702+0000","last_peered":"2024-09-13T16:25:21.150702+0000","last_clean":"2024-09-13T16:25:21.150702+0000","last_became_active":"2024-09-13T16:25:16.948387+0000","last_became_peered":"2024-09-13T16:25:16.948387+0000","last_unstale":"2024-09-13T16:25:21.150702+0000","last_undegraded":"2024-09-13T16:25:21.150702+0000","last_fullsized":"2024-09-13T16:25:21.150702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:28.568881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":693951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":266240,"data_stored":103091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]}]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":103127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":103001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":103161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":694007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:25:39.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:25:39.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.873+0000 7f87680d4700 1 Processor -- start 2024-09-13T16:25:39.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.873+0000 7f87680d4700 1 -- start start 2024-09-13T16:25:39.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.873+0000 7f87680d4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f876006fd40 0x7f8760070190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.873+0000 7f87680d4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600ac960 con 0x7f876015a620 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.873+0000 7f87680d4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87600acc40 con 0x7f87601612f0 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.873+0000 7f87680d4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87600706d0 con 0x7f876006fd40 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8765e70700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f876015a620 0x7f87600b3720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33910/0 (socket says 172.21.15.119:33910) 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8765e70700 1 -- 172.21.15.119:0/3041015947 learned_addr learned my addr 172.21.15.119:0/3041015947 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8766671700 1 --2- 172.21.15.119:0/3041015947 >> v2:172.21.15.145:3300/0 conn(0x7f876006fd40 0x7f8760070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8764e6e700 1 -- 172.21.15.119:0/3041015947 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137017650 0 0) 0x7f87600ac960 con 0x7f876015a620 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8766671700 1 -- 172.21.15.119:0/3041015947 >> v1:172.21.15.145:6789/0 conn(0x7f87601612f0 legacy=0x7f87600ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8766671700 1 -- 172.21.15.119:0/3041015947 >> v1:172.21.15.119:6789/0 conn(0x7f876015a620 legacy=0x7f87600b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8766671700 1 -- 172.21.15.119:0/3041015947 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87600acf20 con 0x7f876006fd40 2024-09-13T16:25:39.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8766671700 1 --2- 172.21.15.119:0/3041015947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876006fd40 0x7f8760070190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8758009d90 tx=0x7f875800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b77e2dc29e1554 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8764e6e700 1 -- 172.21.15.119:0/3041015947 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875800e070 con 0x7f876006fd40 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8764e6e700 1 -- 172.21.15.119:0/3041015947 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f875800bbd0 con 0x7f876006fd40 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f8764e6e700 1 -- 172.21.15.119:0/3041015947 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8758012430 con 0x7f876006fd40 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f87680d4700 1 -- 172.21.15.119:0/3041015947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876006fd40 msgr2=0x7f8760070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f87680d4700 1 --2- 172.21.15.119:0/3041015947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876006fd40 0x7f8760070190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8758009d90 tx=0x7f875800b330 comp rx=0 tx=0).stop 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f87680d4700 1 -- 172.21.15.119:0/3041015947 shutdown_connections 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f87680d4700 1 --2- 172.21.15.119:0/3041015947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f876006fd40 0x7f8760070190 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f87680d4700 1 -- 172.21.15.119:0/3041015947 >> 172.21.15.119:0/3041015947 conn(0x7f876006a350 msgr2=0x7f876006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f87680d4700 1 -- 172.21.15.119:0/3041015947 shutdown_connections 2024-09-13T16:25:39.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.877+0000 7f87680d4700 1 -- 172.21.15.119:0/3041015947 wait complete. 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 Processor -- start 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 -- start start 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876006fd40 0x7f87601311a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876015a620 0x7f87600cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87601612f0 0x7f876015b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876015c920 con 0x7f876015a620 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f876015bad0 con 0x7f87601612f0 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f876015bda0 con 0x7f876006fd40 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876015a620 0x7f87600cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876015a620 0x7f87600cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52626/0 (socket says 172.21.15.119:52626) 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 -- 172.21.15.119:0/3330318265 learned_addr learned my addr 172.21.15.119:0/3330318265 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f8765e70700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876006fd40 0x7f87601311a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f8766671700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87601612f0 0x7f876015b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 -- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876006fd40 msgr2=0x7f87601311a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876006fd40 0x7f87601311a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 -- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87601612f0 msgr2=0x7f876015b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87601612f0 0x7f876015b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 -- 172.21.15.119:0/3330318265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8758009a70 con 0x7f876015a620 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f876566f700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876015a620 0x7f87600cd590 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f875400c500 tx=0x7f875400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87527fc700 1 -- 172.21.15.119:0/3330318265 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87540052d0 con 0x7f876015a620 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87527fc700 1 -- 172.21.15.119:0/3330318265 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8754005430 con 0x7f876015a620 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f876015c080 con 0x7f876015a620 2024-09-13T16:25:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.881+0000 7f87527fc700 1 -- 172.21.15.119:0/3330318265 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87540055a0 con 0x7f876015a620 2024-09-13T16:25:39.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.885+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f876015c640 con 0x7f876015a620 2024-09-13T16:25:39.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.885+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8760067760 con 0x7f876015a620 2024-09-13T16:25:39.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.885+0000 7f87527fc700 1 -- 172.21.15.119:0/3330318265 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f87540038e0 con 0x7f876015a620 2024-09-13T16:25:39.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.885+0000 7f87527fc700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8744066730 0x7f8744068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:39.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.885+0000 7f87527fc700 1 -- 172.21.15.119:0/3330318265 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f8754083830 con 0x7f876015a620 2024-09-13T16:25:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.885+0000 7f8765e70700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8744066730 0x7f8744068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.889+0000 7f8765e70700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8744066730 0x7f8744068bf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f875c007b30 tx=0x7f875c007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:39.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:39.893+0000 7f87527fc700 1 -- 172.21.15.119:0/3330318265 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8754050810 con 0x7f876015a620 2024-09-13T16:25:40.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.045+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f876015aa00 con 0x7f8744066730 2024-09-13T16:25:40.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.065+0000 7f87527fc700 1 -- 172.21.15.119:0/3330318265 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+345197 (secure 0 0 0) 0x7f875c001350 con 0x7f8744066730 2024-09-13T16:25:40.076 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8744066730 msgr2=0x7f8744068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8744066730 0x7f8744068bf0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f875c007b30 tx=0x7f875c007a80 comp rx=0 tx=0).stop 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876015a620 msgr2=0x7f87600cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876015a620 0x7f87600cd590 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f875400c500 tx=0x7f875400c810 comp rx=0 tx=0).stop 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 shutdown_connections 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8744066730 0x7f8744068bf0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f87601612f0 0x7f876015b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f876015a620 0x7f87600cd590 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 --2- 172.21.15.119:0/3330318265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f876006fd40 0x7f87601311a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 >> 172.21.15.119:0/3330318265 conn(0x7f876006a350 msgr2=0x7f876006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 shutdown_connections 2024-09-13T16:25:40.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.073+0000 7f87680d4700 1 -- 172.21.15.119:0/3330318265 wait complete. 2024-09-13T16:25:40.080 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:25:40.092 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":33,"stamp":"2024-09-13T16:25:39.426441+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":594961,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":207,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":159,"ondisk_log_size":159,"up":322,"acting":322,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":322,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219456,"kb_used_data":3272,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755264,"statfs":{"total":773094113280,"available":772869390336,"internally_reserved":0,"allocated":3350528,"data_stored":2013930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1781,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005660"},"pg_stats":[{"pgid":"2.1f","version":"20'4","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954165+0000","last_change":"2024-09-13T16:25:16.560322+0000","last_active":"2024-09-13T16:25:20.954165+0000","last_peered":"2024-09-13T16:25:20.954165+0000","last_clean":"2024-09-13T16:25:20.954165+0000","last_became_active":"2024-09-13T16:25:16.560130+0000","last_became_peered":"2024-09-13T16:25:16.560130+0000","last_unstale":"2024-09-13T16:25:20.954165+0000","last_undegraded":"2024-09-13T16:25:20.954165+0000","last_fullsized":"2024-09-13T16:25:20.954165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941703+0000","last_change":"2024-09-13T16:25:16.937844+0000","last_active":"2024-09-13T16:25:20.941703+0000","last_peered":"2024-09-13T16:25:20.941703+0000","last_clean":"2024-09-13T16:25:20.941703+0000","last_became_active":"2024-09-13T16:25:16.937508+0000","last_became_peered":"2024-09-13T16:25:16.937508+0000","last_unstale":"2024-09-13T16:25:20.941703+0000","last_undegraded":"2024-09-13T16:25:20.941703+0000","last_fullsized":"2024-09-13T16:25:20.941703+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:46.073323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163484+0000","last_change":"2024-09-13T16:25:21.162079+0000","last_active":"2024-09-13T16:25:21.163484+0000","last_peered":"2024-09-13T16:25:21.163484+0000","last_clean":"2024-09-13T16:25:21.163484+0000","last_became_active":"2024-09-13T16:25:21.161341+0000","last_became_peered":"2024-09-13T16:25:21.161341+0000","last_unstale":"2024-09-13T16:25:21.163484+0000","last_undegraded":"2024-09-13T16:25:21.163484+0000","last_fullsized":"2024-09-13T16:25:21.163484+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:55.099223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7,0],"acting":[3,2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941956+0000","last_change":"2024-09-13T16:25:16.554751+0000","last_active":"2024-09-13T16:25:20.941956+0000","last_peered":"2024-09-13T16:25:20.941956+0000","last_clean":"2024-09-13T16:25:20.941956+0000","last_became_active":"2024-09-13T16:25:16.552873+0000","last_became_peered":"2024-09-13T16:25:16.552873+0000","last_unstale":"2024-09-13T16:25:20.941956+0000","last_undegraded":"2024-09-13T16:25:20.941956+0000","last_fullsized":"2024-09-13T16:25:20.941956+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.947694+0000","last_change":"2024-09-13T16:25:16.933368+0000","last_active":"2024-09-13T16:25:20.947694+0000","last_peered":"2024-09-13T16:25:20.947694+0000","last_clean":"2024-09-13T16:25:20.947694+0000","last_became_active":"2024-09-13T16:25:16.932824+0000","last_became_peered":"2024-09-13T16:25:16.932824+0000","last_unstale":"2024-09-13T16:25:20.947694+0000","last_undegraded":"2024-09-13T16:25:20.947694+0000","last_fullsized":"2024-09-13T16:25:20.947694+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969023+0000","last_change":"2024-09-13T16:25:20.965564+0000","last_active":"2024-09-13T16:25:20.969023+0000","last_peered":"2024-09-13T16:25:20.969023+0000","last_clean":"2024-09-13T16:25:20.969023+0000","last_became_active":"2024-09-13T16:25:20.965159+0000","last_became_peered":"2024-09-13T16:25:20.965159+0000","last_unstale":"2024-09-13T16:25:20.969023+0000","last_undegraded":"2024-09-13T16:25:20.969023+0000","last_fullsized":"2024-09-13T16:25:20.969023+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:30:00.090488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1,3],"acting":[4,6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150484+0000","last_change":"2024-09-13T16:25:16.560318+0000","last_active":"2024-09-13T16:25:21.150484+0000","last_peered":"2024-09-13T16:25:21.150484+0000","last_clean":"2024-09-13T16:25:21.150484+0000","last_became_active":"2024-09-13T16:25:16.560069+0000","last_became_peered":"2024-09-13T16:25:16.560069+0000","last_unstale":"2024-09-13T16:25:21.150484+0000","last_undegraded":"2024-09-13T16:25:21.150484+0000","last_fullsized":"2024-09-13T16:25:21.150484+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:12.814872+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943883+0000","last_change":"2024-09-13T16:25:16.935607+0000","last_active":"2024-09-13T16:25:20.943883+0000","last_peered":"2024-09-13T16:25:20.943883+0000","last_clean":"2024-09-13T16:25:20.943883+0000","last_became_active":"2024-09-13T16:25:16.935353+0000","last_became_peered":"2024-09-13T16:25:16.935353+0000","last_unstale":"2024-09-13T16:25:20.943883+0000","last_undegraded":"2024-09-13T16:25:20.943883+0000","last_fullsized":"2024-09-13T16:25:20.943883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:35.981913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973971+0000","last_change":"2024-09-13T16:25:20.969324+0000","last_active":"2024-09-13T16:25:20.973971+0000","last_peered":"2024-09-13T16:25:20.973971+0000","last_clean":"2024-09-13T16:25:20.973971+0000","last_became_active":"2024-09-13T16:25:20.968916+0000","last_became_peered":"2024-09-13T16:25:20.968916+0000","last_unstale":"2024-09-13T16:25:20.973971+0000","last_undegraded":"2024-09-13T16:25:20.973971+0000","last_fullsized":"2024-09-13T16:25:20.973971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:03.825482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149974+0000","last_change":"2024-09-13T16:25:16.560280+0000","last_active":"2024-09-13T16:25:21.149974+0000","last_peered":"2024-09-13T16:25:21.149974+0000","last_clean":"2024-09-13T16:25:21.149974+0000","last_became_active":"2024-09-13T16:25:16.559981+0000","last_became_peered":"2024-09-13T16:25:16.559981+0000","last_unstale":"2024-09-13T16:25:21.149974+0000","last_undegraded":"2024-09-13T16:25:21.149974+0000","last_fullsized":"2024-09-13T16:25:21.149974+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943411+0000","last_change":"2024-09-13T16:25:16.937364+0000","last_active":"2024-09-13T16:25:20.943411+0000","last_peered":"2024-09-13T16:25:20.943411+0000","last_clean":"2024-09-13T16:25:20.943411+0000","last_became_active":"2024-09-13T16:25:16.937119+0000","last_became_peered":"2024-09-13T16:25:16.937119+0000","last_unstale":"2024-09-13T16:25:20.943411+0000","last_undegraded":"2024-09-13T16:25:20.943411+0000","last_fullsized":"2024-09-13T16:25:20.943411+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:31.885317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959982+0000","last_change":"2024-09-13T16:25:20.959414+0000","last_active":"2024-09-13T16:25:20.959982+0000","last_peered":"2024-09-13T16:25:20.959982+0000","last_clean":"2024-09-13T16:25:20.959982+0000","last_became_active":"2024-09-13T16:25:20.958955+0000","last_became_peered":"2024-09-13T16:25:20.958955+0000","last_unstale":"2024-09-13T16:25:20.959982+0000","last_undegraded":"2024-09-13T16:25:20.959982+0000","last_fullsized":"2024-09-13T16:25:20.959982+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:35.667608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0,5],"acting":[4,3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939712+0000","last_change":"2024-09-13T16:25:16.936925+0000","last_active":"2024-09-13T16:25:20.939712+0000","last_peered":"2024-09-13T16:25:20.939712+0000","last_clean":"2024-09-13T16:25:20.939712+0000","last_became_active":"2024-09-13T16:25:16.936608+0000","last_became_peered":"2024-09-13T16:25:16.936608+0000","last_unstale":"2024-09-13T16:25:20.939712+0000","last_undegraded":"2024-09-13T16:25:20.939712+0000","last_fullsized":"2024-09-13T16:25:20.939712+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.b","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150049+0000","last_change":"2024-09-13T16:25:16.561316+0000","last_active":"2024-09-13T16:25:21.150049+0000","last_peered":"2024-09-13T16:25:21.150049+0000","last_clean":"2024-09-13T16:25:21.150049+0000","last_became_active":"2024-09-13T16:25:16.560850+0000","last_became_peered":"2024-09-13T16:25:16.560850+0000","last_unstale":"2024-09-13T16:25:21.150049+0000","last_undegraded":"2024-09-13T16:25:21.150049+0000","last_fullsized":"2024-09-13T16:25:21.150049+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.961598+0000","last_change":"2024-09-13T16:25:20.960831+0000","last_active":"2024-09-13T16:25:20.961598+0000","last_peered":"2024-09-13T16:25:20.961598+0000","last_clean":"2024-09-13T16:25:20.961598+0000","last_became_active":"2024-09-13T16:25:20.960495+0000","last_became_peered":"2024-09-13T16:25:20.960495+0000","last_unstale":"2024-09-13T16:25:20.961598+0000","last_undegraded":"2024-09-13T16:25:20.961598+0000","last_fullsized":"2024-09-13T16:25:20.961598+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2,1],"acting":[4,0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940654+0000","last_change":"2024-09-13T16:25:16.937601+0000","last_active":"2024-09-13T16:25:20.940654+0000","last_peered":"2024-09-13T16:25:20.940654+0000","last_clean":"2024-09-13T16:25:20.940654+0000","last_became_active":"2024-09-13T16:25:16.937343+0000","last_became_peered":"2024-09-13T16:25:16.937343+0000","last_unstale":"2024-09-13T16:25:20.940654+0000","last_undegraded":"2024-09-13T16:25:20.940654+0000","last_fullsized":"2024-09-13T16:25:20.940654+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950384+0000","last_change":"2024-09-13T16:25:16.550175+0000","last_active":"2024-09-13T16:25:20.950384+0000","last_peered":"2024-09-13T16:25:20.950384+0000","last_clean":"2024-09-13T16:25:20.950384+0000","last_became_active":"2024-09-13T16:25:16.549875+0000","last_became_peered":"2024-09-13T16:25:16.549875+0000","last_unstale":"2024-09-13T16:25:20.950384+0000","last_undegraded":"2024-09-13T16:25:20.950384+0000","last_fullsized":"2024-09-13T16:25:20.950384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973920+0000","last_change":"2024-09-13T16:25:20.968856+0000","last_active":"2024-09-13T16:25:20.973920+0000","last_peered":"2024-09-13T16:25:20.973920+0000","last_clean":"2024-09-13T16:25:20.973920+0000","last_became_active":"2024-09-13T16:25:20.968470+0000","last_became_peered":"2024-09-13T16:25:20.968470+0000","last_unstale":"2024-09-13T16:25:20.973920+0000","last_undegraded":"2024-09-13T16:25:20.973920+0000","last_fullsized":"2024-09-13T16:25:20.973920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940018+0000","last_change":"2024-09-13T16:25:16.936551+0000","last_active":"2024-09-13T16:25:20.940018+0000","last_peered":"2024-09-13T16:25:20.940018+0000","last_clean":"2024-09-13T16:25:20.940018+0000","last_became_active":"2024-09-13T16:25:16.936298+0000","last_became_peered":"2024-09-13T16:25:16.936298+0000","last_unstale":"2024-09-13T16:25:20.940018+0000","last_undegraded":"2024-09-13T16:25:20.940018+0000","last_fullsized":"2024-09-13T16:25:20.940018+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949896+0000","last_change":"2024-09-13T16:25:16.556208+0000","last_active":"2024-09-13T16:25:20.949896+0000","last_peered":"2024-09-13T16:25:20.949896+0000","last_clean":"2024-09-13T16:25:20.949896+0000","last_became_active":"2024-09-13T16:25:16.555897+0000","last_became_peered":"2024-09-13T16:25:16.555897+0000","last_unstale":"2024-09-13T16:25:20.949896+0000","last_undegraded":"2024-09-13T16:25:20.949896+0000","last_fullsized":"2024-09-13T16:25:20.949896+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.164051+0000","last_change":"2024-09-13T16:25:21.162085+0000","last_active":"2024-09-13T16:25:21.164051+0000","last_peered":"2024-09-13T16:25:21.164051+0000","last_clean":"2024-09-13T16:25:21.164051+0000","last_became_active":"2024-09-13T16:25:21.160633+0000","last_became_peered":"2024-09-13T16:25:21.160633+0000","last_unstale":"2024-09-13T16:25:21.164051+0000","last_undegraded":"2024-09-13T16:25:21.164051+0000","last_fullsized":"2024-09-13T16:25:21.164051+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4,7],"acting":[1,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951220+0000","last_change":"2024-09-13T16:25:16.931406+0000","last_active":"2024-09-13T16:25:20.951220+0000","last_peered":"2024-09-13T16:25:20.951220+0000","last_clean":"2024-09-13T16:25:20.951220+0000","last_became_active":"2024-09-13T16:25:16.931073+0000","last_became_peered":"2024-09-13T16:25:16.931073+0000","last_unstale":"2024-09-13T16:25:20.951220+0000","last_undegraded":"2024-09-13T16:25:20.951220+0000","last_fullsized":"2024-09-13T16:25:20.951220+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:31.812511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149390+0000","last_change":"2024-09-13T16:25:16.562667+0000","last_active":"2024-09-13T16:25:21.149390+0000","last_peered":"2024-09-13T16:25:21.149390+0000","last_clean":"2024-09-13T16:25:21.149390+0000","last_became_active":"2024-09-13T16:25:16.562361+0000","last_became_peered":"2024-09-13T16:25:16.562361+0000","last_unstale":"2024-09-13T16:25:21.149390+0000","last_undegraded":"2024-09-13T16:25:21.149390+0000","last_fullsized":"2024-09-13T16:25:21.149390+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:32.764470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.973976+0000","last_change":"2024-09-13T16:25:20.969014+0000","last_active":"2024-09-13T16:25:20.973976+0000","last_peered":"2024-09-13T16:25:20.973976+0000","last_clean":"2024-09-13T16:25:20.973976+0000","last_became_active":"2024-09-13T16:25:20.968528+0000","last_became_peered":"2024-09-13T16:25:20.968528+0000","last_unstale":"2024-09-13T16:25:20.973976+0000","last_undegraded":"2024-09-13T16:25:20.973976+0000","last_fullsized":"2024-09-13T16:25:20.973976+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6,1],"acting":[4,3,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948588+0000","last_change":"2024-09-13T16:25:16.933619+0000","last_active":"2024-09-13T16:25:20.948588+0000","last_peered":"2024-09-13T16:25:20.948588+0000","last_clean":"2024-09-13T16:25:20.948588+0000","last_became_active":"2024-09-13T16:25:16.933324+0000","last_became_peered":"2024-09-13T16:25:16.933324+0000","last_unstale":"2024-09-13T16:25:20.948588+0000","last_undegraded":"2024-09-13T16:25:20.948588+0000","last_fullsized":"2024-09-13T16:25:20.948588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:13.824400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942057+0000","last_change":"2024-09-13T16:25:16.560251+0000","last_active":"2024-09-13T16:25:20.942057+0000","last_peered":"2024-09-13T16:25:20.942057+0000","last_clean":"2024-09-13T16:25:20.942057+0000","last_became_active":"2024-09-13T16:25:16.559836+0000","last_became_peered":"2024-09-13T16:25:16.559836+0000","last_unstale":"2024-09-13T16:25:20.942057+0000","last_undegraded":"2024-09-13T16:25:20.942057+0000","last_fullsized":"2024-09-13T16:25:20.942057+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.159122+0000","last_change":"2024-09-13T16:25:21.156691+0000","last_active":"2024-09-13T16:25:21.159122+0000","last_peered":"2024-09-13T16:25:21.159122+0000","last_clean":"2024-09-13T16:25:21.159122+0000","last_became_active":"2024-09-13T16:25:21.156255+0000","last_became_peered":"2024-09-13T16:25:21.156255+0000","last_unstale":"2024-09-13T16:25:21.159122+0000","last_undegraded":"2024-09-13T16:25:21.159122+0000","last_fullsized":"2024-09-13T16:25:21.159122+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:12.641280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,7],"acting":[4,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940196+0000","last_change":"2024-09-13T16:25:16.946326+0000","last_active":"2024-09-13T16:25:20.940196+0000","last_peered":"2024-09-13T16:25:20.940196+0000","last_clean":"2024-09-13T16:25:20.940196+0000","last_became_active":"2024-09-13T16:25:16.946091+0000","last_became_peered":"2024-09-13T16:25:16.946091+0000","last_unstale":"2024-09-13T16:25:20.940196+0000","last_undegraded":"2024-09-13T16:25:20.940196+0000","last_fullsized":"2024-09-13T16:25:20.940196+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"23'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953612+0000","last_change":"2024-09-13T16:25:16.548181+0000","last_active":"2024-09-13T16:25:20.953612+0000","last_peered":"2024-09-13T16:25:20.953612+0000","last_clean":"2024-09-13T16:25:20.953612+0000","last_became_active":"2024-09-13T16:25:16.546980+0000","last_became_peered":"2024-09-13T16:25:16.546980+0000","last_unstale":"2024-09-13T16:25:20.953612+0000","last_undegraded":"2024-09-13T16:25:20.953612+0000","last_fullsized":"2024-09-13T16:25:20.953612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163366+0000","last_change":"2024-09-13T16:25:21.157394+0000","last_active":"2024-09-13T16:25:21.163366+0000","last_peered":"2024-09-13T16:25:21.163366+0000","last_clean":"2024-09-13T16:25:21.163366+0000","last_became_active":"2024-09-13T16:25:21.156428+0000","last_became_peered":"2024-09-13T16:25:21.156428+0000","last_unstale":"2024-09-13T16:25:21.163366+0000","last_undegraded":"2024-09-13T16:25:21.163366+0000","last_fullsized":"2024-09-13T16:25:21.163366+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:57:31.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0,1],"acting":[3,7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950532+0000","last_change":"2024-09-13T16:25:16.932134+0000","last_active":"2024-09-13T16:25:20.950532+0000","last_peered":"2024-09-13T16:25:20.950532+0000","last_clean":"2024-09-13T16:25:20.950532+0000","last_became_active":"2024-09-13T16:25:16.931157+0000","last_became_peered":"2024-09-13T16:25:16.931157+0000","last_unstale":"2024-09-13T16:25:20.950532+0000","last_undegraded":"2024-09-13T16:25:20.950532+0000","last_fullsized":"2024-09-13T16:25:20.950532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150728+0000","last_change":"2024-09-13T16:25:16.560858+0000","last_active":"2024-09-13T16:25:21.150728+0000","last_peered":"2024-09-13T16:25:21.150728+0000","last_clean":"2024-09-13T16:25:21.150728+0000","last_became_active":"2024-09-13T16:25:16.560627+0000","last_became_peered":"2024-09-13T16:25:16.560627+0000","last_unstale":"2024-09-13T16:25:21.150728+0000","last_undegraded":"2024-09-13T16:25:21.150728+0000","last_fullsized":"2024-09-13T16:25:21.150728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:50.678565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158800+0000","last_change":"2024-09-13T16:25:21.155558+0000","last_active":"2024-09-13T16:25:21.158800+0000","last_peered":"2024-09-13T16:25:21.158800+0000","last_clean":"2024-09-13T16:25:21.158800+0000","last_became_active":"2024-09-13T16:25:21.154513+0000","last_became_peered":"2024-09-13T16:25:21.154513+0000","last_unstale":"2024-09-13T16:25:21.158800+0000","last_undegraded":"2024-09-13T16:25:21.158800+0000","last_fullsized":"2024-09-13T16:25:21.158800+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7,1],"acting":[0,5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943548+0000","last_change":"2024-09-13T16:25:16.934627+0000","last_active":"2024-09-13T16:25:20.943548+0000","last_peered":"2024-09-13T16:25:20.943548+0000","last_clean":"2024-09-13T16:25:20.943548+0000","last_became_active":"2024-09-13T16:25:16.934031+0000","last_became_peered":"2024-09-13T16:25:16.934031+0000","last_unstale":"2024-09-13T16:25:20.943548+0000","last_undegraded":"2024-09-13T16:25:20.943548+0000","last_fullsized":"2024-09-13T16:25:20.943548+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:19.244372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.950619+0000","last_change":"2024-09-13T16:25:16.550845+0000","last_active":"2024-09-13T16:25:20.950619+0000","last_peered":"2024-09-13T16:25:20.950619+0000","last_clean":"2024-09-13T16:25:20.950619+0000","last_became_active":"2024-09-13T16:25:16.550523+0000","last_became_peered":"2024-09-13T16:25:16.550523+0000","last_unstale":"2024-09-13T16:25:20.950619+0000","last_undegraded":"2024-09-13T16:25:20.950619+0000","last_fullsized":"2024-09-13T16:25:20.950619+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:25:11.860490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971742+0000","last_change":"2024-09-13T16:25:20.963835+0000","last_active":"2024-09-13T16:25:20.971742+0000","last_peered":"2024-09-13T16:25:20.971742+0000","last_clean":"2024-09-13T16:25:20.971742+0000","last_became_active":"2024-09-13T16:25:20.963031+0000","last_became_peered":"2024-09-13T16:25:20.963031+0000","last_unstale":"2024-09-13T16:25:20.971742+0000","last_undegraded":"2024-09-13T16:25:20.971742+0000","last_fullsized":"2024-09-13T16:25:20.971742+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:33:39.370983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4,0],"acting":[1,5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951685+0000","last_change":"2024-09-13T16:25:16.936436+0000","last_active":"2024-09-13T16:25:20.951685+0000","last_peered":"2024-09-13T16:25:20.951685+0000","last_clean":"2024-09-13T16:25:20.951685+0000","last_became_active":"2024-09-13T16:25:16.936131+0000","last_became_peered":"2024-09-13T16:25:16.936131+0000","last_unstale":"2024-09-13T16:25:20.951685+0000","last_undegraded":"2024-09-13T16:25:20.951685+0000","last_fullsized":"2024-09-13T16:25:20.951685+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:33.773998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941767+0000","last_change":"2024-09-13T16:25:16.548310+0000","last_active":"2024-09-13T16:25:20.941767+0000","last_peered":"2024-09-13T16:25:20.941767+0000","last_clean":"2024-09-13T16:25:20.941767+0000","last_became_active":"2024-09-13T16:25:16.548056+0000","last_became_peered":"2024-09-13T16:25:16.548056+0000","last_unstale":"2024-09-13T16:25:20.941767+0000","last_undegraded":"2024-09-13T16:25:20.941767+0000","last_fullsized":"2024-09-13T16:25:20.941767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971523+0000","last_change":"2024-09-13T16:25:20.960907+0000","last_active":"2024-09-13T16:25:20.971523+0000","last_peered":"2024-09-13T16:25:20.971523+0000","last_clean":"2024-09-13T16:25:20.971523+0000","last_became_active":"2024-09-13T16:25:20.960504+0000","last_became_peered":"2024-09-13T16:25:20.960504+0000","last_unstale":"2024-09-13T16:25:20.971523+0000","last_undegraded":"2024-09-13T16:25:20.971523+0000","last_fullsized":"2024-09-13T16:25:20.971523+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:45.040342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3,6],"acting":[1,2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951906+0000","last_change":"2024-09-13T16:25:16.935966+0000","last_active":"2024-09-13T16:25:20.951906+0000","last_peered":"2024-09-13T16:25:20.951906+0000","last_clean":"2024-09-13T16:25:20.951906+0000","last_became_active":"2024-09-13T16:25:16.935728+0000","last_became_peered":"2024-09-13T16:25:16.935728+0000","last_unstale":"2024-09-13T16:25:20.951906+0000","last_undegraded":"2024-09-13T16:25:20.951906+0000","last_fullsized":"2024-09-13T16:25:20.951906+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954151+0000","last_change":"2024-09-13T16:25:16.541632+0000","last_active":"2024-09-13T16:25:20.954151+0000","last_peered":"2024-09-13T16:25:20.954151+0000","last_clean":"2024-09-13T16:25:20.954151+0000","last_became_active":"2024-09-13T16:25:16.541362+0000","last_became_peered":"2024-09-13T16:25:16.541362+0000","last_unstale":"2024-09-13T16:25:20.954151+0000","last_undegraded":"2024-09-13T16:25:20.954151+0000","last_fullsized":"2024-09-13T16:25:20.954151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971468+0000","last_change":"2024-09-13T16:25:20.961579+0000","last_active":"2024-09-13T16:25:20.971468+0000","last_peered":"2024-09-13T16:25:20.971468+0000","last_clean":"2024-09-13T16:25:20.971468+0000","last_became_active":"2024-09-13T16:25:20.961177+0000","last_became_peered":"2024-09-13T16:25:20.961177+0000","last_unstale":"2024-09-13T16:25:20.971468+0000","last_undegraded":"2024-09-13T16:25:20.971468+0000","last_fullsized":"2024-09-13T16:25:20.971468+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0,3],"acting":[1,5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948340+0000","last_change":"2024-09-13T16:25:16.932105+0000","last_active":"2024-09-13T16:25:20.948340+0000","last_peered":"2024-09-13T16:25:20.948340+0000","last_clean":"2024-09-13T16:25:20.948340+0000","last_became_active":"2024-09-13T16:25:16.931893+0000","last_became_peered":"2024-09-13T16:25:16.931893+0000","last_unstale":"2024-09-13T16:25:20.948340+0000","last_undegraded":"2024-09-13T16:25:20.948340+0000","last_fullsized":"2024-09-13T16:25:20.948340+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149350+0000","last_change":"2024-09-13T16:25:16.563013+0000","last_active":"2024-09-13T16:25:21.149350+0000","last_peered":"2024-09-13T16:25:21.149350+0000","last_clean":"2024-09-13T16:25:21.149350+0000","last_became_active":"2024-09-13T16:25:16.562820+0000","last_became_peered":"2024-09-13T16:25:16.562820+0000","last_unstale":"2024-09-13T16:25:21.149350+0000","last_undegraded":"2024-09-13T16:25:21.149350+0000","last_fullsized":"2024-09-13T16:25:21.149350+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:43.080203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.161407+0000","last_change":"2024-09-13T16:25:21.156288+0000","last_active":"2024-09-13T16:25:21.161407+0000","last_peered":"2024-09-13T16:25:21.161407+0000","last_clean":"2024-09-13T16:25:21.161407+0000","last_became_active":"2024-09-13T16:25:21.154873+0000","last_became_peered":"2024-09-13T16:25:21.154873+0000","last_unstale":"2024-09-13T16:25:21.161407+0000","last_undegraded":"2024-09-13T16:25:21.161407+0000","last_fullsized":"2024-09-13T16:25:21.161407+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:57.390965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2,7],"acting":[0,1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941321+0000","last_change":"2024-09-13T16:25:16.934659+0000","last_active":"2024-09-13T16:25:20.941321+0000","last_peered":"2024-09-13T16:25:20.941321+0000","last_clean":"2024-09-13T16:25:20.941321+0000","last_became_active":"2024-09-13T16:25:16.934286+0000","last_became_peered":"2024-09-13T16:25:16.934286+0000","last_unstale":"2024-09-13T16:25:20.941321+0000","last_undegraded":"2024-09-13T16:25:20.941321+0000","last_fullsized":"2024-09-13T16:25:20.941321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941932+0000","last_change":"2024-09-13T16:25:16.548968+0000","last_active":"2024-09-13T16:25:20.941932+0000","last_peered":"2024-09-13T16:25:20.941932+0000","last_clean":"2024-09-13T16:25:20.941932+0000","last_became_active":"2024-09-13T16:25:16.548722+0000","last_became_peered":"2024-09-13T16:25:16.548722+0000","last_unstale":"2024-09-13T16:25:20.941932+0000","last_undegraded":"2024-09-13T16:25:20.941932+0000","last_fullsized":"2024-09-13T16:25:20.941932+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:25:08.328680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":167,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149477+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:25:21.149477+0000","last_peered":"2024-09-13T16:25:21.149477+0000","last_clean":"2024-09-13T16:25:21.149477+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:25:21.149477+0000","last_undegraded":"2024-09-13T16:25:21.149477+0000","last_fullsized":"2024-09-13T16:25:21.149477+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156667+0000","last_change":"2024-09-13T16:25:21.154730+0000","last_active":"2024-09-13T16:25:21.156667+0000","last_peered":"2024-09-13T16:25:21.156667+0000","last_clean":"2024-09-13T16:25:21.156667+0000","last_became_active":"2024-09-13T16:25:21.154226+0000","last_became_peered":"2024-09-13T16:25:21.154226+0000","last_unstale":"2024-09-13T16:25:21.156667+0000","last_undegraded":"2024-09-13T16:25:21.156667+0000","last_fullsized":"2024-09-13T16:25:21.156667+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:24:40.936861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,4,7],"acting":[6,3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150204+0000","last_change":"2024-09-13T16:25:16.944787+0000","last_active":"2024-09-13T16:25:21.150204+0000","last_peered":"2024-09-13T16:25:21.150204+0000","last_clean":"2024-09-13T16:25:21.150204+0000","last_became_active":"2024-09-13T16:25:16.944505+0000","last_became_peered":"2024-09-13T16:25:16.944505+0000","last_unstale":"2024-09-13T16:25:21.150204+0000","last_undegraded":"2024-09-13T16:25:21.150204+0000","last_fullsized":"2024-09-13T16:25:21.150204+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:36:09.241669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954746+0000","last_change":"2024-09-13T16:25:16.548195+0000","last_active":"2024-09-13T16:25:20.954746+0000","last_peered":"2024-09-13T16:25:20.954746+0000","last_clean":"2024-09-13T16:25:20.954746+0000","last_became_active":"2024-09-13T16:25:16.547839+0000","last_became_peered":"2024-09-13T16:25:16.547839+0000","last_unstale":"2024-09-13T16:25:20.954746+0000","last_undegraded":"2024-09-13T16:25:20.954746+0000","last_fullsized":"2024-09-13T16:25:20.954746+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:47.062890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.156778+0000","last_change":"2024-09-13T16:25:21.154890+0000","last_active":"2024-09-13T16:25:21.156778+0000","last_peered":"2024-09-13T16:25:21.156778+0000","last_clean":"2024-09-13T16:25:21.156778+0000","last_became_active":"2024-09-13T16:25:21.154376+0000","last_became_peered":"2024-09-13T16:25:21.154376+0000","last_unstale":"2024-09-13T16:25:21.156778+0000","last_undegraded":"2024-09-13T16:25:21.156778+0000","last_fullsized":"2024-09-13T16:25:21.156778+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:50:09.876152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,1,0],"acting":[6,7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944021+0000","last_change":"2024-09-13T16:25:16.933523+0000","last_active":"2024-09-13T16:25:20.944021+0000","last_peered":"2024-09-13T16:25:20.944021+0000","last_clean":"2024-09-13T16:25:20.944021+0000","last_became_active":"2024-09-13T16:25:16.932020+0000","last_became_peered":"2024-09-13T16:25:16.932020+0000","last_unstale":"2024-09-13T16:25:20.944021+0000","last_undegraded":"2024-09-13T16:25:20.944021+0000","last_fullsized":"2024-09-13T16:25:20.944021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:34.060136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.d","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952497+0000","last_change":"2024-09-13T16:25:16.546857+0000","last_active":"2024-09-13T16:25:20.952497+0000","last_peered":"2024-09-13T16:25:20.952497+0000","last_clean":"2024-09-13T16:25:20.952497+0000","last_became_active":"2024-09-13T16:25:16.546592+0000","last_became_peered":"2024-09-13T16:25:16.546592+0000","last_unstale":"2024-09-13T16:25:20.952497+0000","last_undegraded":"2024-09-13T16:25:20.952497+0000","last_fullsized":"2024-09-13T16:25:20.952497+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963301+0000","last_change":"2024-09-13T16:25:20.958682+0000","last_active":"2024-09-13T16:25:20.963301+0000","last_peered":"2024-09-13T16:25:20.963301+0000","last_clean":"2024-09-13T16:25:20.963301+0000","last_became_active":"2024-09-13T16:25:20.958064+0000","last_became_peered":"2024-09-13T16:25:20.958064+0000","last_unstale":"2024-09-13T16:25:20.963301+0000","last_undegraded":"2024-09-13T16:25:20.963301+0000","last_fullsized":"2024-09-13T16:25:20.963301+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:08.502649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,5],"acting":[0,1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150943+0000","last_change":"2024-09-13T16:25:16.951592+0000","last_active":"2024-09-13T16:25:21.150943+0000","last_peered":"2024-09-13T16:25:21.150943+0000","last_clean":"2024-09-13T16:25:21.150943+0000","last_became_active":"2024-09-13T16:25:16.951375+0000","last_became_peered":"2024-09-13T16:25:16.951375+0000","last_unstale":"2024-09-13T16:25:21.150943+0000","last_undegraded":"2024-09-13T16:25:21.150943+0000","last_fullsized":"2024-09-13T16:25:21.150943+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:32.222786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959285+0000","last_change":"2024-09-13T16:25:16.541133+0000","last_active":"2024-09-13T16:25:20.959285+0000","last_peered":"2024-09-13T16:25:20.959285+0000","last_clean":"2024-09-13T16:25:20.959285+0000","last_became_active":"2024-09-13T16:25:16.540843+0000","last_became_peered":"2024-09-13T16:25:16.540843+0000","last_unstale":"2024-09-13T16:25:20.959285+0000","last_undegraded":"2024-09-13T16:25:20.959285+0000","last_fullsized":"2024-09-13T16:25:20.959285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:18:05.477981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158080+0000","last_change":"2024-09-13T16:25:21.154908+0000","last_active":"2024-09-13T16:25:21.158080+0000","last_peered":"2024-09-13T16:25:21.158080+0000","last_clean":"2024-09-13T16:25:21.158080+0000","last_became_active":"2024-09-13T16:25:21.154365+0000","last_became_peered":"2024-09-13T16:25:21.154365+0000","last_unstale":"2024-09-13T16:25:21.158080+0000","last_undegraded":"2024-09-13T16:25:21.158080+0000","last_fullsized":"2024-09-13T16:25:21.158080+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:06:00.139666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7,6],"acting":[5,4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150870+0000","last_change":"2024-09-13T16:25:16.941714+0000","last_active":"2024-09-13T16:25:21.150870+0000","last_peered":"2024-09-13T16:25:21.150870+0000","last_clean":"2024-09-13T16:25:21.150870+0000","last_became_active":"2024-09-13T16:25:16.941465+0000","last_became_peered":"2024-09-13T16:25:16.941465+0000","last_unstale":"2024-09-13T16:25:21.150870+0000","last_undegraded":"2024-09-13T16:25:21.150870+0000","last_fullsized":"2024-09-13T16:25:21.150870+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"20'1","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952152+0000","last_change":"2024-09-13T16:25:16.544584+0000","last_active":"2024-09-13T16:25:20.952152+0000","last_peered":"2024-09-13T16:25:20.952152+0000","last_clean":"2024-09-13T16:25:20.952152+0000","last_became_active":"2024-09-13T16:25:16.544275+0000","last_became_peered":"2024-09-13T16:25:16.544275+0000","last_unstale":"2024-09-13T16:25:20.952152+0000","last_undegraded":"2024-09-13T16:25:20.952152+0000","last_fullsized":"2024-09-13T16:25:20.952152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:17:38.495511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.976715+0000","last_change":"2024-09-13T16:25:20.972282+0000","last_active":"2024-09-13T16:25:20.976715+0000","last_peered":"2024-09-13T16:25:20.976715+0000","last_clean":"2024-09-13T16:25:20.976715+0000","last_became_active":"2024-09-13T16:25:20.971899+0000","last_became_peered":"2024-09-13T16:25:20.971899+0000","last_unstale":"2024-09-13T16:25:20.976715+0000","last_undegraded":"2024-09-13T16:25:20.976715+0000","last_fullsized":"2024-09-13T16:25:20.976715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3,6],"acting":[4,1,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150689+0000","last_change":"2024-09-13T16:25:16.944388+0000","last_active":"2024-09-13T16:25:21.150689+0000","last_peered":"2024-09-13T16:25:21.150689+0000","last_clean":"2024-09-13T16:25:21.150689+0000","last_became_active":"2024-09-13T16:25:16.943987+0000","last_became_peered":"2024-09-13T16:25:16.943987+0000","last_unstale":"2024-09-13T16:25:21.150689+0000","last_undegraded":"2024-09-13T16:25:21.150689+0000","last_fullsized":"2024-09-13T16:25:21.150689+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"20'8","reported_seq":22,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953615+0000","last_change":"2024-09-13T16:25:16.545908+0000","last_active":"2024-09-13T16:25:20.953615+0000","last_peered":"2024-09-13T16:25:20.953615+0000","last_clean":"2024-09-13T16:25:20.953615+0000","last_became_active":"2024-09-13T16:25:16.545632+0000","last_became_peered":"2024-09-13T16:25:16.545632+0000","last_unstale":"2024-09-13T16:25:20.953615+0000","last_undegraded":"2024-09-13T16:25:20.953615+0000","last_fullsized":"2024-09-13T16:25:20.953615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963693+0000","last_change":"2024-09-13T16:25:20.961076+0000","last_active":"2024-09-13T16:25:20.963693+0000","last_peered":"2024-09-13T16:25:20.963693+0000","last_clean":"2024-09-13T16:25:20.963693+0000","last_became_active":"2024-09-13T16:25:20.960736+0000","last_became_peered":"2024-09-13T16:25:20.960736+0000","last_unstale":"2024-09-13T16:25:20.963693+0000","last_undegraded":"2024-09-13T16:25:20.963693+0000","last_fullsized":"2024-09-13T16:25:20.963693+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2,6],"acting":[0,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939386+0000","last_change":"2024-09-13T16:25:16.939543+0000","last_active":"2024-09-13T16:25:20.939386+0000","last_peered":"2024-09-13T16:25:20.939386+0000","last_clean":"2024-09-13T16:25:20.939386+0000","last_became_active":"2024-09-13T16:25:16.939269+0000","last_became_peered":"2024-09-13T16:25:16.939269+0000","last_unstale":"2024-09-13T16:25:20.939386+0000","last_undegraded":"2024-09-13T16:25:20.939386+0000","last_fullsized":"2024-09-13T16:25:20.939386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939280+0000","last_change":"2024-09-13T16:25:16.557172+0000","last_active":"2024-09-13T16:25:20.939280+0000","last_peered":"2024-09-13T16:25:20.939280+0000","last_clean":"2024-09-13T16:25:20.939280+0000","last_became_active":"2024-09-13T16:25:16.556910+0000","last_became_peered":"2024-09-13T16:25:16.556910+0000","last_unstale":"2024-09-13T16:25:20.939280+0000","last_undegraded":"2024-09-13T16:25:20.939280+0000","last_fullsized":"2024-09-13T16:25:20.939280+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:08:30.802983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162791+0000","last_change":"2024-09-13T16:25:21.154750+0000","last_active":"2024-09-13T16:25:21.162791+0000","last_peered":"2024-09-13T16:25:21.162791+0000","last_clean":"2024-09-13T16:25:21.162791+0000","last_became_active":"2024-09-13T16:25:21.153987+0000","last_became_peered":"2024-09-13T16:25:21.153987+0000","last_unstale":"2024-09-13T16:25:21.162791+0000","last_undegraded":"2024-09-13T16:25:21.162791+0000","last_fullsized":"2024-09-13T16:25:21.162791+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:35.042637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,6],"acting":[3,7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149825+0000","last_change":"2024-09-13T16:25:16.944291+0000","last_active":"2024-09-13T16:25:21.149825+0000","last_peered":"2024-09-13T16:25:21.149825+0000","last_clean":"2024-09-13T16:25:21.149825+0000","last_became_active":"2024-09-13T16:25:16.941709+0000","last_became_peered":"2024-09-13T16:25:16.941709+0000","last_unstale":"2024-09-13T16:25:21.149825+0000","last_undegraded":"2024-09-13T16:25:21.149825+0000","last_fullsized":"2024-09-13T16:25:21.149825+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:27:45.537446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941804+0000","last_change":"2024-09-13T16:25:16.553038+0000","last_active":"2024-09-13T16:25:20.941804+0000","last_peered":"2024-09-13T16:25:20.941804+0000","last_clean":"2024-09-13T16:25:20.941804+0000","last_became_active":"2024-09-13T16:25:16.551960+0000","last_became_peered":"2024-09-13T16:25:16.551960+0000","last_unstale":"2024-09-13T16:25:20.941804+0000","last_undegraded":"2024-09-13T16:25:20.941804+0000","last_fullsized":"2024-09-13T16:25:20.941804+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:27.632855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163399+0000","last_change":"2024-09-13T16:25:21.158491+0000","last_active":"2024-09-13T16:25:21.163399+0000","last_peered":"2024-09-13T16:25:21.163399+0000","last_clean":"2024-09-13T16:25:21.163399+0000","last_became_active":"2024-09-13T16:25:21.158133+0000","last_became_peered":"2024-09-13T16:25:21.158133+0000","last_unstale":"2024-09-13T16:25:21.163399+0000","last_undegraded":"2024-09-13T16:25:21.163399+0000","last_fullsized":"2024-09-13T16:25:21.163399+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:15:04.781809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7,2],"acting":[3,1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946925+0000","last_change":"2024-09-13T16:25:16.941874+0000","last_active":"2024-09-13T16:25:20.946925+0000","last_peered":"2024-09-13T16:25:20.946925+0000","last_clean":"2024-09-13T16:25:20.946925+0000","last_became_active":"2024-09-13T16:25:16.941683+0000","last_became_peered":"2024-09-13T16:25:16.941683+0000","last_unstale":"2024-09-13T16:25:20.946925+0000","last_undegraded":"2024-09-13T16:25:20.946925+0000","last_fullsized":"2024-09-13T16:25:20.946925+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.13","version":"20'2","reported_seq":16,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953752+0000","last_change":"2024-09-13T16:25:16.551439+0000","last_active":"2024-09-13T16:25:20.953752+0000","last_peered":"2024-09-13T16:25:20.953752+0000","last_clean":"2024-09-13T16:25:20.953752+0000","last_became_active":"2024-09-13T16:25:16.550937+0000","last_became_peered":"2024-09-13T16:25:16.550937+0000","last_unstale":"2024-09-13T16:25:20.953752+0000","last_undegraded":"2024-09-13T16:25:20.953752+0000","last_fullsized":"2024-09-13T16:25:20.953752+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:47:14.983082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.158057+0000","last_change":"2024-09-13T16:25:21.155556+0000","last_active":"2024-09-13T16:25:21.158057+0000","last_peered":"2024-09-13T16:25:21.158057+0000","last_clean":"2024-09-13T16:25:21.158057+0000","last_became_active":"2024-09-13T16:25:21.155201+0000","last_became_peered":"2024-09-13T16:25:21.155201+0000","last_unstale":"2024-09-13T16:25:21.158057+0000","last_undegraded":"2024-09-13T16:25:21.158057+0000","last_fullsized":"2024-09-13T16:25:21.158057+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:42.823106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3,2],"acting":[5,7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150192+0000","last_change":"2024-09-13T16:25:16.944770+0000","last_active":"2024-09-13T16:25:21.150192+0000","last_peered":"2024-09-13T16:25:21.150192+0000","last_clean":"2024-09-13T16:25:21.150192+0000","last_became_active":"2024-09-13T16:25:16.944455+0000","last_became_peered":"2024-09-13T16:25:16.944455+0000","last_unstale":"2024-09-13T16:25:21.150192+0000","last_undegraded":"2024-09-13T16:25:21.150192+0000","last_fullsized":"2024-09-13T16:25:21.150192+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:41:40.454800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.14","version":"25'9","reported_seq":23,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-09-13T16:25:26.199825+0000","last_change":"2024-09-13T16:25:16.552517+0000","last_active":"2024-09-13T16:25:26.199825+0000","last_peered":"2024-09-13T16:25:26.199825+0000","last_clean":"2024-09-13T16:25:26.199825+0000","last_became_active":"2024-09-13T16:25:16.552101+0000","last_became_peered":"2024-09-13T16:25:16.552101+0000","last_unstale":"2024-09-13T16:25:26.199825+0000","last_undegraded":"2024-09-13T16:25:26.199825+0000","last_fullsized":"2024-09-13T16:25:26.199825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:31:26.386336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3795,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.971715+0000","last_change":"2024-09-13T16:25:20.963516+0000","last_active":"2024-09-13T16:25:20.971715+0000","last_peered":"2024-09-13T16:25:20.971715+0000","last_clean":"2024-09-13T16:25:20.971715+0000","last_became_active":"2024-09-13T16:25:20.962518+0000","last_became_peered":"2024-09-13T16:25:20.962518+0000","last_unstale":"2024-09-13T16:25:20.971715+0000","last_undegraded":"2024-09-13T16:25:20.971715+0000","last_fullsized":"2024-09-13T16:25:20.971715+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:05.557712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2,0],"acting":[1,6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951494+0000","last_change":"2024-09-13T16:25:16.941833+0000","last_active":"2024-09-13T16:25:20.951494+0000","last_peered":"2024-09-13T16:25:20.951494+0000","last_clean":"2024-09-13T16:25:20.951494+0000","last_became_active":"2024-09-13T16:25:16.941607+0000","last_became_peered":"2024-09-13T16:25:16.941607+0000","last_unstale":"2024-09-13T16:25:20.951494+0000","last_undegraded":"2024-09-13T16:25:20.951494+0000","last_fullsized":"2024-09-13T16:25:20.951494+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:44.089730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"20'3","reported_seq":18,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952414+0000","last_change":"2024-09-13T16:25:16.549878+0000","last_active":"2024-09-13T16:25:20.952414+0000","last_peered":"2024-09-13T16:25:20.952414+0000","last_clean":"2024-09-13T16:25:20.952414+0000","last_became_active":"2024-09-13T16:25:16.548581+0000","last_became_peered":"2024-09-13T16:25:16.548581+0000","last_unstale":"2024-09-13T16:25:20.952414+0000","last_undegraded":"2024-09-13T16:25:20.952414+0000","last_fullsized":"2024-09-13T16:25:20.952414+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:17.080274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.969433+0000","last_change":"2024-09-13T16:25:20.965523+0000","last_active":"2024-09-13T16:25:20.969433+0000","last_peered":"2024-09-13T16:25:20.969433+0000","last_clean":"2024-09-13T16:25:20.969433+0000","last_became_active":"2024-09-13T16:25:20.965035+0000","last_became_peered":"2024-09-13T16:25:20.965035+0000","last_unstale":"2024-09-13T16:25:20.969433+0000","last_undegraded":"2024-09-13T16:25:20.969433+0000","last_fullsized":"2024-09-13T16:25:20.969433+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:57.932065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6,1],"acting":[4,5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.954066+0000","last_change":"2024-09-13T16:25:16.933522+0000","last_active":"2024-09-13T16:25:20.954066+0000","last_peered":"2024-09-13T16:25:20.954066+0000","last_clean":"2024-09-13T16:25:20.954066+0000","last_became_active":"2024-09-13T16:25:16.933134+0000","last_became_peered":"2024-09-13T16:25:16.933134+0000","last_unstale":"2024-09-13T16:25:20.954066+0000","last_undegraded":"2024-09-13T16:25:20.954066+0000","last_fullsized":"2024-09-13T16:25:20.954066+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:02.064909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942876+0000","last_change":"2024-09-13T16:25:16.549756+0000","last_active":"2024-09-13T16:25:20.942876+0000","last_peered":"2024-09-13T16:25:20.942876+0000","last_clean":"2024-09-13T16:25:20.942876+0000","last_became_active":"2024-09-13T16:25:16.549515+0000","last_became_peered":"2024-09-13T16:25:16.549515+0000","last_unstale":"2024-09-13T16:25:20.942876+0000","last_undegraded":"2024-09-13T16:25:20.942876+0000","last_fullsized":"2024-09-13T16:25:20.942876+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:29:00.617563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963576+0000","last_change":"2024-09-13T16:25:20.960084+0000","last_active":"2024-09-13T16:25:20.963576+0000","last_peered":"2024-09-13T16:25:20.963576+0000","last_clean":"2024-09-13T16:25:20.963576+0000","last_became_active":"2024-09-13T16:25:20.957343+0000","last_became_peered":"2024-09-13T16:25:20.957343+0000","last_unstale":"2024-09-13T16:25:20.963576+0000","last_undegraded":"2024-09-13T16:25:20.963576+0000","last_fullsized":"2024-09-13T16:25:20.963576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:02:06.115184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6,4],"acting":[3,1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942738+0000","last_change":"2024-09-13T16:25:16.942162+0000","last_active":"2024-09-13T16:25:20.942738+0000","last_peered":"2024-09-13T16:25:20.942738+0000","last_clean":"2024-09-13T16:25:20.942738+0000","last_became_active":"2024-09-13T16:25:16.941896+0000","last_became_peered":"2024-09-13T16:25:16.941896+0000","last_unstale":"2024-09-13T16:25:20.942738+0000","last_undegraded":"2024-09-13T16:25:20.942738+0000","last_fullsized":"2024-09-13T16:25:20.942738+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:24.023742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941934+0000","last_change":"2024-09-13T16:25:16.551442+0000","last_active":"2024-09-13T16:25:20.941934+0000","last_peered":"2024-09-13T16:25:20.941934+0000","last_clean":"2024-09-13T16:25:20.941934+0000","last_became_active":"2024-09-13T16:25:16.551084+0000","last_became_peered":"2024-09-13T16:25:16.551084+0000","last_unstale":"2024-09-13T16:25:20.941934+0000","last_undegraded":"2024-09-13T16:25:20.941934+0000","last_fullsized":"2024-09-13T16:25:20.941934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:05.948320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.163416+0000","last_change":"2024-09-13T16:25:21.157528+0000","last_active":"2024-09-13T16:25:21.163416+0000","last_peered":"2024-09-13T16:25:21.163416+0000","last_clean":"2024-09-13T16:25:21.163416+0000","last_became_active":"2024-09-13T16:25:21.155560+0000","last_became_peered":"2024-09-13T16:25:21.155560+0000","last_unstale":"2024-09-13T16:25:21.163416+0000","last_undegraded":"2024-09-13T16:25:21.163416+0000","last_fullsized":"2024-09-13T16:25:21.163416+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:59.932295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6,5],"acting":[3,7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.162596+0000","last_change":"2024-09-13T16:25:21.158060+0000","last_active":"2024-09-13T16:25:21.162596+0000","last_peered":"2024-09-13T16:25:21.162596+0000","last_clean":"2024-09-13T16:25:21.162596+0000","last_became_active":"2024-09-13T16:25:21.156882+0000","last_became_peered":"2024-09-13T16:25:21.156882+0000","last_unstale":"2024-09-13T16:25:21.162596+0000","last_undegraded":"2024-09-13T16:25:21.162596+0000","last_fullsized":"2024-09-13T16:25:21.162596+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:42.931952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3,1],"acting":[0,7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949704+0000","last_change":"2024-09-13T16:25:16.932645+0000","last_active":"2024-09-13T16:25:20.949704+0000","last_peered":"2024-09-13T16:25:20.949704+0000","last_clean":"2024-09-13T16:25:20.949704+0000","last_became_active":"2024-09-13T16:25:16.931775+0000","last_became_peered":"2024-09-13T16:25:16.931775+0000","last_unstale":"2024-09-13T16:25:20.949704+0000","last_undegraded":"2024-09-13T16:25:20.949704+0000","last_fullsized":"2024-09-13T16:25:20.949704+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942164+0000","last_change":"2024-09-13T16:25:16.547336+0000","last_active":"2024-09-13T16:25:20.942164+0000","last_peered":"2024-09-13T16:25:20.942164+0000","last_clean":"2024-09-13T16:25:20.942164+0000","last_became_active":"2024-09-13T16:25:16.547062+0000","last_became_peered":"2024-09-13T16:25:16.547062+0000","last_unstale":"2024-09-13T16:25:20.942164+0000","last_undegraded":"2024-09-13T16:25:20.942164+0000","last_fullsized":"2024-09-13T16:25:20.942164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:52:40.140428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.968355+0000","last_change":"2024-09-13T16:25:20.965354+0000","last_active":"2024-09-13T16:25:20.968355+0000","last_peered":"2024-09-13T16:25:20.968355+0000","last_clean":"2024-09-13T16:25:20.968355+0000","last_became_active":"2024-09-13T16:25:20.964985+0000","last_became_peered":"2024-09-13T16:25:20.964985+0000","last_unstale":"2024-09-13T16:25:20.968355+0000","last_undegraded":"2024-09-13T16:25:20.968355+0000","last_fullsized":"2024-09-13T16:25:20.968355+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:42.840413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1,0],"acting":[6,5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939763+0000","last_change":"2024-09-13T16:25:16.929034+0000","last_active":"2024-09-13T16:25:20.939763+0000","last_peered":"2024-09-13T16:25:20.939763+0000","last_clean":"2024-09-13T16:25:20.939763+0000","last_became_active":"2024-09-13T16:25:16.928782+0000","last_became_peered":"2024-09-13T16:25:16.928782+0000","last_unstale":"2024-09-13T16:25:20.939763+0000","last_undegraded":"2024-09-13T16:25:20.939763+0000","last_fullsized":"2024-09-13T16:25:20.939763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940092+0000","last_change":"2024-09-13T16:25:16.548923+0000","last_active":"2024-09-13T16:25:20.940092+0000","last_peered":"2024-09-13T16:25:20.940092+0000","last_clean":"2024-09-13T16:25:20.940092+0000","last_became_active":"2024-09-13T16:25:16.545843+0000","last_became_peered":"2024-09-13T16:25:16.545843+0000","last_unstale":"2024-09-13T16:25:20.940092+0000","last_undegraded":"2024-09-13T16:25:20.940092+0000","last_fullsized":"2024-09-13T16:25:20.940092+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:20.597041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1a","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939883+0000","last_change":"2024-09-13T16:25:16.555750+0000","last_active":"2024-09-13T16:25:20.939883+0000","last_peered":"2024-09-13T16:25:20.939883+0000","last_clean":"2024-09-13T16:25:20.939883+0000","last_became_active":"2024-09-13T16:25:16.555480+0000","last_became_peered":"2024-09-13T16:25:16.555480+0000","last_unstale":"2024-09-13T16:25:20.939883+0000","last_undegraded":"2024-09-13T16:25:20.939883+0000","last_fullsized":"2024-09-13T16:25:20.939883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953647+0000","last_change":"2024-09-13T16:25:16.934605+0000","last_active":"2024-09-13T16:25:20.953647+0000","last_peered":"2024-09-13T16:25:20.953647+0000","last_clean":"2024-09-13T16:25:20.953647+0000","last_became_active":"2024-09-13T16:25:16.934365+0000","last_became_peered":"2024-09-13T16:25:16.934365+0000","last_unstale":"2024-09-13T16:25:20.953647+0000","last_undegraded":"2024-09-13T16:25:20.953647+0000","last_fullsized":"2024-09-13T16:25:20.953647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.967920+0000","last_change":"2024-09-13T16:25:20.965561+0000","last_active":"2024-09-13T16:25:20.967920+0000","last_peered":"2024-09-13T16:25:20.967920+0000","last_clean":"2024-09-13T16:25:20.967920+0000","last_became_active":"2024-09-13T16:25:20.965162+0000","last_became_peered":"2024-09-13T16:25:20.965162+0000","last_unstale":"2024-09-13T16:25:20.967920+0000","last_undegraded":"2024-09-13T16:25:20.967920+0000","last_fullsized":"2024-09-13T16:25:20.967920+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:35.284928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3,5],"acting":[2,1,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"20'1","reported_seq":15,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940864+0000","last_change":"2024-09-13T16:25:16.560967+0000","last_active":"2024-09-13T16:25:20.940864+0000","last_peered":"2024-09-13T16:25:20.940864+0000","last_clean":"2024-09-13T16:25:20.940864+0000","last_became_active":"2024-09-13T16:25:16.560675+0000","last_became_peered":"2024-09-13T16:25:16.560675+0000","last_unstale":"2024-09-13T16:25:20.940864+0000","last_undegraded":"2024-09-13T16:25:20.940864+0000","last_fullsized":"2024-09-13T16:25:20.940864+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:15.514395+0000","last_clean_scrub_stamp":"2024-09-13T16:25:15.514395+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:26:01.400837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951187+0000","last_change":"2024-09-13T16:25:16.936540+0000","last_active":"2024-09-13T16:25:20.951187+0000","last_peered":"2024-09-13T16:25:20.951187+0000","last_clean":"2024-09-13T16:25:20.951187+0000","last_became_active":"2024-09-13T16:25:16.936269+0000","last_became_peered":"2024-09-13T16:25:16.936269+0000","last_unstale":"2024-09-13T16:25:20.951187+0000","last_undegraded":"2024-09-13T16:25:20.951187+0000","last_fullsized":"2024-09-13T16:25:20.951187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:18.089801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":12,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.963605+0000","last_change":"2024-09-13T16:25:20.961207+0000","last_active":"2024-09-13T16:25:20.963605+0000","last_peered":"2024-09-13T16:25:20.963605+0000","last_clean":"2024-09-13T16:25:20.963605+0000","last_became_active":"2024-09-13T16:25:20.960801+0000","last_became_peered":"2024-09-13T16:25:20.960801+0000","last_unstale":"2024-09-13T16:25:20.963605+0000","last_undegraded":"2024-09-13T16:25:20.963605+0000","last_fullsized":"2024-09-13T16:25:20.963605+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:19.923534+0000","last_clean_scrub_stamp":"2024-09-13T16:25:19.923534+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:46:38.874981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4,2],"acting":[3,1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939239+0000","last_change":"2024-09-13T16:25:16.939880+0000","last_active":"2024-09-13T16:25:20.939239+0000","last_peered":"2024-09-13T16:25:20.939239+0000","last_clean":"2024-09-13T16:25:20.939239+0000","last_became_active":"2024-09-13T16:25:16.939579+0000","last_became_peered":"2024-09-13T16:25:16.939579+0000","last_unstale":"2024-09-13T16:25:20.939239+0000","last_undegraded":"2024-09-13T16:25:20.939239+0000","last_fullsized":"2024-09-13T16:25:20.939239+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:37.417453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953935+0000","last_change":"2024-09-13T16:25:16.938944+0000","last_active":"2024-09-13T16:25:20.953935+0000","last_peered":"2024-09-13T16:25:20.953935+0000","last_clean":"2024-09-13T16:25:20.953935+0000","last_became_active":"2024-09-13T16:25:16.938658+0000","last_became_peered":"2024-09-13T16:25:16.938658+0000","last_unstale":"2024-09-13T16:25:20.953935+0000","last_undegraded":"2024-09-13T16:25:20.953935+0000","last_fullsized":"2024-09-13T16:25:20.953935+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:47:50.130499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941580+0000","last_change":"2024-09-13T16:25:16.933588+0000","last_active":"2024-09-13T16:25:20.941580+0000","last_peered":"2024-09-13T16:25:20.941580+0000","last_clean":"2024-09-13T16:25:20.941580+0000","last_became_active":"2024-09-13T16:25:16.932300+0000","last_became_peered":"2024-09-13T16:25:16.932300+0000","last_unstale":"2024-09-13T16:25:20.941580+0000","last_undegraded":"2024-09-13T16:25:20.941580+0000","last_fullsized":"2024-09-13T16:25:20.941580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:13:43.437120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953565+0000","last_change":"2024-09-13T16:25:16.934119+0000","last_active":"2024-09-13T16:25:20.953565+0000","last_peered":"2024-09-13T16:25:20.953565+0000","last_clean":"2024-09-13T16:25:20.953565+0000","last_became_active":"2024-09-13T16:25:16.930259+0000","last_became_peered":"2024-09-13T16:25:16.930259+0000","last_unstale":"2024-09-13T16:25:20.953565+0000","last_undegraded":"2024-09-13T16:25:20.953565+0000","last_fullsized":"2024-09-13T16:25:20.953565+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941316+0000","last_change":"2024-09-13T16:25:16.931922+0000","last_active":"2024-09-13T16:25:20.941316+0000","last_peered":"2024-09-13T16:25:20.941316+0000","last_clean":"2024-09-13T16:25:20.941316+0000","last_became_active":"2024-09-13T16:25:16.931678+0000","last_became_peered":"2024-09-13T16:25:16.931678+0000","last_unstale":"2024-09-13T16:25:20.941316+0000","last_undegraded":"2024-09-13T16:25:20.941316+0000","last_fullsized":"2024-09-13T16:25:20.941316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:27.822409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.952316+0000","last_change":"2024-09-13T16:25:16.932526+0000","last_active":"2024-09-13T16:25:20.952316+0000","last_peered":"2024-09-13T16:25:20.952316+0000","last_clean":"2024-09-13T16:25:20.952316+0000","last_became_active":"2024-09-13T16:25:16.932189+0000","last_became_peered":"2024-09-13T16:25:16.932189+0000","last_unstale":"2024-09-13T16:25:20.952316+0000","last_undegraded":"2024-09-13T16:25:20.952316+0000","last_fullsized":"2024-09-13T16:25:20.952316+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941353+0000","last_change":"2024-09-13T16:25:16.947538+0000","last_active":"2024-09-13T16:25:20.941353+0000","last_peered":"2024-09-13T16:25:20.941353+0000","last_clean":"2024-09-13T16:25:20.941353+0000","last_became_active":"2024-09-13T16:25:16.947272+0000","last_became_peered":"2024-09-13T16:25:16.947272+0000","last_unstale":"2024-09-13T16:25:20.941353+0000","last_undegraded":"2024-09-13T16:25:20.941353+0000","last_fullsized":"2024-09-13T16:25:20.941353+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940064+0000","last_change":"2024-09-13T16:25:16.932732+0000","last_active":"2024-09-13T16:25:20.940064+0000","last_peered":"2024-09-13T16:25:20.940064+0000","last_clean":"2024-09-13T16:25:20.940064+0000","last_became_active":"2024-09-13T16:25:16.932505+0000","last_became_peered":"2024-09-13T16:25:16.932505+0000","last_unstale":"2024-09-13T16:25:20.940064+0000","last_undegraded":"2024-09-13T16:25:20.940064+0000","last_fullsized":"2024-09-13T16:25:20.940064+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942122+0000","last_change":"2024-09-13T16:25:16.933750+0000","last_active":"2024-09-13T16:25:20.942122+0000","last_peered":"2024-09-13T16:25:20.942122+0000","last_clean":"2024-09-13T16:25:20.942122+0000","last_became_active":"2024-09-13T16:25:16.933509+0000","last_became_peered":"2024-09-13T16:25:16.933509+0000","last_unstale":"2024-09-13T16:25:20.942122+0000","last_undegraded":"2024-09-13T16:25:20.942122+0000","last_fullsized":"2024-09-13T16:25:20.942122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:28.433477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939801+0000","last_change":"2024-09-13T16:25:16.932135+0000","last_active":"2024-09-13T16:25:20.939801+0000","last_peered":"2024-09-13T16:25:20.939801+0000","last_clean":"2024-09-13T16:25:20.939801+0000","last_became_active":"2024-09-13T16:25:16.931865+0000","last_became_peered":"2024-09-13T16:25:16.931865+0000","last_unstale":"2024-09-13T16:25:20.939801+0000","last_undegraded":"2024-09-13T16:25:20.939801+0000","last_fullsized":"2024-09-13T16:25:20.939801+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:45.336706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941544+0000","last_change":"2024-09-13T16:25:16.942044+0000","last_active":"2024-09-13T16:25:20.941544+0000","last_peered":"2024-09-13T16:25:20.941544+0000","last_clean":"2024-09-13T16:25:20.941544+0000","last_became_active":"2024-09-13T16:25:16.941738+0000","last_became_peered":"2024-09-13T16:25:16.941738+0000","last_unstale":"2024-09-13T16:25:20.941544+0000","last_undegraded":"2024-09-13T16:25:20.941544+0000","last_fullsized":"2024-09-13T16:25:20.941544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:58:43.832593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.948993+0000","last_change":"2024-09-13T16:25:16.942581+0000","last_active":"2024-09-13T16:25:20.948993+0000","last_peered":"2024-09-13T16:25:20.948993+0000","last_clean":"2024-09-13T16:25:20.948993+0000","last_became_active":"2024-09-13T16:25:16.942321+0000","last_became_peered":"2024-09-13T16:25:16.942321+0000","last_unstale":"2024-09-13T16:25:20.948993+0000","last_undegraded":"2024-09-13T16:25:20.948993+0000","last_fullsized":"2024-09-13T16:25:20.948993+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.939626+0000","last_change":"2024-09-13T16:25:16.936450+0000","last_active":"2024-09-13T16:25:20.939626+0000","last_peered":"2024-09-13T16:25:20.939626+0000","last_clean":"2024-09-13T16:25:20.939626+0000","last_became_active":"2024-09-13T16:25:16.936084+0000","last_became_peered":"2024-09-13T16:25:16.936084+0000","last_unstale":"2024-09-13T16:25:20.939626+0000","last_undegraded":"2024-09-13T16:25:20.939626+0000","last_fullsized":"2024-09-13T16:25:20.939626+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:04:03.371165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150470+0000","last_change":"2024-09-13T16:25:16.948876+0000","last_active":"2024-09-13T16:25:21.150470+0000","last_peered":"2024-09-13T16:25:21.150470+0000","last_clean":"2024-09-13T16:25:21.150470+0000","last_became_active":"2024-09-13T16:25:16.948536+0000","last_became_peered":"2024-09-13T16:25:16.948536+0000","last_unstale":"2024-09-13T16:25:21.150470+0000","last_undegraded":"2024-09-13T16:25:21.150470+0000","last_fullsized":"2024-09-13T16:25:21.150470+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:17.422099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.959182+0000","last_change":"2024-09-13T16:25:16.937371+0000","last_active":"2024-09-13T16:25:20.959182+0000","last_peered":"2024-09-13T16:25:20.959182+0000","last_clean":"2024-09-13T16:25:20.959182+0000","last_became_active":"2024-09-13T16:25:16.937130+0000","last_became_peered":"2024-09-13T16:25:16.937130+0000","last_unstale":"2024-09-13T16:25:20.959182+0000","last_undegraded":"2024-09-13T16:25:20.959182+0000","last_fullsized":"2024-09-13T16:25:20.959182+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:35:29.816675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.944314+0000","last_change":"2024-09-13T16:25:16.947064+0000","last_active":"2024-09-13T16:25:20.944314+0000","last_peered":"2024-09-13T16:25:20.944314+0000","last_clean":"2024-09-13T16:25:20.944314+0000","last_became_active":"2024-09-13T16:25:16.946790+0000","last_became_peered":"2024-09-13T16:25:16.946790+0000","last_unstale":"2024-09-13T16:25:20.944314+0000","last_undegraded":"2024-09-13T16:25:20.944314+0000","last_fullsized":"2024-09-13T16:25:20.944314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:49.978002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.942037+0000","last_change":"2024-09-13T16:25:16.931584+0000","last_active":"2024-09-13T16:25:20.942037+0000","last_peered":"2024-09-13T16:25:20.942037+0000","last_clean":"2024-09-13T16:25:20.942037+0000","last_became_active":"2024-09-13T16:25:16.931224+0000","last_became_peered":"2024-09-13T16:25:16.931224+0000","last_unstale":"2024-09-13T16:25:20.942037+0000","last_undegraded":"2024-09-13T16:25:20.942037+0000","last_fullsized":"2024-09-13T16:25:20.942037+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:11.767256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.949799+0000","last_change":"2024-09-13T16:25:16.936233+0000","last_active":"2024-09-13T16:25:20.949799+0000","last_peered":"2024-09-13T16:25:20.949799+0000","last_clean":"2024-09-13T16:25:20.949799+0000","last_became_active":"2024-09-13T16:25:16.935935+0000","last_became_peered":"2024-09-13T16:25:16.935935+0000","last_unstale":"2024-09-13T16:25:20.949799+0000","last_undegraded":"2024-09-13T16:25:20.949799+0000","last_fullsized":"2024-09-13T16:25:20.949799+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.946795+0000","last_change":"2024-09-13T16:25:16.936516+0000","last_active":"2024-09-13T16:25:20.946795+0000","last_peered":"2024-09-13T16:25:20.946795+0000","last_clean":"2024-09-13T16:25:20.946795+0000","last_became_active":"2024-09-13T16:25:16.936281+0000","last_became_peered":"2024-09-13T16:25:16.936281+0000","last_unstale":"2024-09-13T16:25:20.946795+0000","last_undegraded":"2024-09-13T16:25:20.946795+0000","last_fullsized":"2024-09-13T16:25:20.946795+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:42:18.524153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940457+0000","last_change":"2024-09-13T16:25:16.938439+0000","last_active":"2024-09-13T16:25:20.940457+0000","last_peered":"2024-09-13T16:25:20.940457+0000","last_clean":"2024-09-13T16:25:20.940457+0000","last_became_active":"2024-09-13T16:25:16.938204+0000","last_became_peered":"2024-09-13T16:25:16.938204+0000","last_unstale":"2024-09-13T16:25:20.940457+0000","last_undegraded":"2024-09-13T16:25:20.940457+0000","last_fullsized":"2024-09-13T16:25:20.940457+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:21.804311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.951414+0000","last_change":"2024-09-13T16:25:16.931605+0000","last_active":"2024-09-13T16:25:20.951414+0000","last_peered":"2024-09-13T16:25:20.951414+0000","last_clean":"2024-09-13T16:25:20.951414+0000","last_became_active":"2024-09-13T16:25:16.931330+0000","last_became_peered":"2024-09-13T16:25:16.931330+0000","last_unstale":"2024-09-13T16:25:20.951414+0000","last_undegraded":"2024-09-13T16:25:20.951414+0000","last_fullsized":"2024-09-13T16:25:20.951414+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:22:51.688093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.940628+0000","last_change":"2024-09-13T16:25:16.936567+0000","last_active":"2024-09-13T16:25:20.940628+0000","last_peered":"2024-09-13T16:25:20.940628+0000","last_clean":"2024-09-13T16:25:20.940628+0000","last_became_active":"2024-09-13T16:25:16.936076+0000","last_became_peered":"2024-09-13T16:25:16.936076+0000","last_unstale":"2024-09-13T16:25:20.940628+0000","last_undegraded":"2024-09-13T16:25:20.940628+0000","last_fullsized":"2024-09-13T16:25:20.940628+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:21.373436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941217+0000","last_change":"2024-09-13T16:25:16.936360+0000","last_active":"2024-09-13T16:25:20.941217+0000","last_peered":"2024-09-13T16:25:20.941217+0000","last_clean":"2024-09-13T16:25:20.941217+0000","last_became_active":"2024-09-13T16:25:16.935361+0000","last_became_peered":"2024-09-13T16:25:16.935361+0000","last_unstale":"2024-09-13T16:25:20.941217+0000","last_undegraded":"2024-09-13T16:25:20.941217+0000","last_fullsized":"2024-09-13T16:25:20.941217+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:17.148914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150991+0000","last_change":"2024-09-13T16:25:16.951709+0000","last_active":"2024-09-13T16:25:21.150991+0000","last_peered":"2024-09-13T16:25:21.150991+0000","last_clean":"2024-09-13T16:25:21.150991+0000","last_became_active":"2024-09-13T16:25:16.951465+0000","last_became_peered":"2024-09-13T16:25:16.951465+0000","last_unstale":"2024-09-13T16:25:21.150991+0000","last_undegraded":"2024-09-13T16:25:21.150991+0000","last_fullsized":"2024-09-13T16:25:21.150991+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:44:05.832843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.938568+0000","last_change":"2024-09-13T16:25:16.933931+0000","last_active":"2024-09-13T16:25:20.938568+0000","last_peered":"2024-09-13T16:25:20.938568+0000","last_clean":"2024-09-13T16:25:20.938568+0000","last_became_active":"2024-09-13T16:25:16.933646+0000","last_became_peered":"2024-09-13T16:25:16.933646+0000","last_unstale":"2024-09-13T16:25:20.938568+0000","last_undegraded":"2024-09-13T16:25:20.938568+0000","last_fullsized":"2024-09-13T16:25:20.938568+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:09.651976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150544+0000","last_change":"2024-09-13T16:25:16.941658+0000","last_active":"2024-09-13T16:25:21.150544+0000","last_peered":"2024-09-13T16:25:21.150544+0000","last_clean":"2024-09-13T16:25:21.150544+0000","last_became_active":"2024-09-13T16:25:16.941375+0000","last_became_peered":"2024-09-13T16:25:16.941375+0000","last_unstale":"2024-09-13T16:25:21.150544+0000","last_undegraded":"2024-09-13T16:25:21.150544+0000","last_fullsized":"2024-09-13T16:25:21.150544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:06:05.301510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.941428+0000","last_change":"2024-09-13T16:25:16.931253+0000","last_active":"2024-09-13T16:25:20.941428+0000","last_peered":"2024-09-13T16:25:20.941428+0000","last_clean":"2024-09-13T16:25:20.941428+0000","last_became_active":"2024-09-13T16:25:16.930844+0000","last_became_peered":"2024-09-13T16:25:16.930844+0000","last_unstale":"2024-09-13T16:25:20.941428+0000","last_undegraded":"2024-09-13T16:25:20.941428+0000","last_fullsized":"2024-09-13T16:25:20.941428+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:45:11.873813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.149733+0000","last_change":"2024-09-13T16:25:16.946470+0000","last_active":"2024-09-13T16:25:21.149733+0000","last_peered":"2024-09-13T16:25:21.149733+0000","last_clean":"2024-09-13T16:25:21.149733+0000","last_became_active":"2024-09-13T16:25:16.946228+0000","last_became_peered":"2024-09-13T16:25:16.946228+0000","last_unstale":"2024-09-13T16:25:21.149733+0000","last_undegraded":"2024-09-13T16:25:21.149733+0000","last_fullsized":"2024-09-13T16:25:21.149733+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:06:51.187953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943747+0000","last_change":"2024-09-13T16:25:16.933763+0000","last_active":"2024-09-13T16:25:20.943747+0000","last_peered":"2024-09-13T16:25:20.943747+0000","last_clean":"2024-09-13T16:25:20.943747+0000","last_became_active":"2024-09-13T16:25:16.932396+0000","last_became_peered":"2024-09-13T16:25:16.932396+0000","last_unstale":"2024-09-13T16:25:20.943747+0000","last_undegraded":"2024-09-13T16:25:20.943747+0000","last_fullsized":"2024-09-13T16:25:20.943747+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:55:13.889786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.943715+0000","last_change":"2024-09-13T16:25:16.931365+0000","last_active":"2024-09-13T16:25:20.943715+0000","last_peered":"2024-09-13T16:25:20.943715+0000","last_clean":"2024-09-13T16:25:20.943715+0000","last_became_active":"2024-09-13T16:25:16.931117+0000","last_became_peered":"2024-09-13T16:25:16.931117+0000","last_unstale":"2024-09-13T16:25:20.943715+0000","last_undegraded":"2024-09-13T16:25:20.943715+0000","last_fullsized":"2024-09-13T16:25:20.943715+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:20:13.578965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":14,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:20.953517+0000","last_change":"2024-09-13T16:25:16.932889+0000","last_active":"2024-09-13T16:25:20.953517+0000","last_peered":"2024-09-13T16:25:20.953517+0000","last_clean":"2024-09-13T16:25:20.953517+0000","last_became_active":"2024-09-13T16:25:16.931952+0000","last_became_peered":"2024-09-13T16:25:16.931952+0000","last_unstale":"2024-09-13T16:25:20.953517+0000","last_undegraded":"2024-09-13T16:25:20.953517+0000","last_fullsized":"2024-09-13T16:25:20.953517+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:46.008868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":24,"state":"active+clean","last_fresh":"2024-09-13T16:25:21.150702+0000","last_change":"2024-09-13T16:25:16.948699+0000","last_active":"2024-09-13T16:25:21.150702+0000","last_peered":"2024-09-13T16:25:21.150702+0000","last_clean":"2024-09-13T16:25:21.150702+0000","last_became_active":"2024-09-13T16:25:16.948387+0000","last_became_peered":"2024-09-13T16:25:16.948387+0000","last_unstale":"2024-09-13T16:25:21.150702+0000","last_undegraded":"2024-09-13T16:25:21.150702+0000","last_fullsized":"2024-09-13T16:25:21.150702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:16.523923+0000","last_clean_scrub_stamp":"2024-09-13T16:25:16.523923+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:28.568881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4593,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":48,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":9186,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":42,"ondisk_log_size":42,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542152,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27860,"kb_used_data":840,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343980,"statfs":{"total":96636764160,"available":96608235520,"internally_reserved":0,"allocated":860160,"data_stored":693951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85499999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.111}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.145}]}]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71299999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59299999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0269999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542153,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27296,"kb_used_data":260,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344544,"statfs":{"total":96636764160,"available":96608813056,"internally_reserved":0,"allocated":266240,"data_stored":103091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.087}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98499999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0129999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.04}]}]},{"osd":4,"up_from":14,"seq":60129542151,"num_pgs":42,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27292,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344548,"statfs":{"total":96636764160,"available":96608817152,"internally_reserved":0,"allocated":274432,"data_stored":103127,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78500000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58199999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88500000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81599999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":268,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":274432,"data_stored":106796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89500000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.032}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2849999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27284,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344556,"statfs":{"total":96636764160,"available":96608825344,"internally_reserved":0,"allocated":262144,"data_stored":103001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73699999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76200000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.90100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82699999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.024}]}]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27288,"kb_used_data":264,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344552,"statfs":{"total":96636764160,"available":96608821248,"internally_reserved":0,"allocated":270336,"data_stored":103161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53600000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69499999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":0,"up_from":14,"seq":60129542152,"num_pgs":41,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27868,"kb_used_data":848,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343972,"statfs":{"total":96636764160,"available":96608227328,"internally_reserved":0,"allocated":868352,"data_stored":694007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65700000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82099999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.165}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:25:40.097 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T16:25:40.097 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-13T16:25:40.097 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-13T16:25:40.098 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:25:40.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.193+0000 7fa7bca19700 1 Processor -- start 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.193+0000 7fa7bca19700 1 -- start start 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.193+0000 7fa7bca19700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa7b806dc80 0x7fa7b8070af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.193+0000 7fa7bca19700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b815af50 0x7fa7b815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.193+0000 7fa7bca19700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7b8070660 con 0x7fa7b80cdfc0 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.193+0000 7fa7bca19700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7b8071030 con 0x7fa7b806dc80 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.193+0000 7fa7bca19700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b815bca0 con 0x7fa7b815af50 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b815af50 0x7fa7b815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b815af50 0x7fa7b815b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:37366/0 (socket says 172.21.15.119:37366) 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 -- 172.21.15.119:0/2102643424 learned_addr learned my addr 172.21.15.119:0/2102643424 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:40.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7affff700 1 --2- 172.21.15.119:0/2102643424 >> v2:172.21.15.145:3300/0 conn(0x7fa7b806dc80 0x7fa7b8070af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 -- 172.21.15.119:0/2102643424 >> v2:172.21.15.145:3300/0 conn(0x7fa7b806dc80 msgr2=0x7fa7b8070af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 --2- 172.21.15.119:0/2102643424 >> v2:172.21.15.145:3300/0 conn(0x7fa7b806dc80 0x7fa7b8070af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 -- 172.21.15.119:0/2102643424 >> v1:172.21.15.145:6789/0 conn(0x7fa7b80cdfc0 legacy=0x7fa7b806d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 -- 172.21.15.119:0/2102643424 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b806fd30 con 0x7fa7b815af50 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7affff700 1 --2- 172.21.15.119:0/2102643424 >> v2:172.21.15.145:3300/0 conn(0x7fa7b806dc80 0x7fa7b8070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b759e700 1 --2- 172.21.15.119:0/2102643424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b815af50 0x7fa7b815b3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009e50 tx=0x7fa7b000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a35181f9eb26a52 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b659c700 1 -- 172.21.15.119:0/2102643424 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b000e070 con 0x7fa7b815af50 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b659c700 1 -- 172.21.15.119:0/2102643424 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7b000b910 con 0x7fa7b815af50 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7b659c700 1 -- 172.21.15.119:0/2102643424 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7b000bcc0 con 0x7fa7b815af50 2024-09-13T16:25:40.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 -- 172.21.15.119:0/2102643424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b815af50 msgr2=0x7fa7b815b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2102643424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b815af50 0x7fa7b815b3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009e50 tx=0x7fa7b000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 -- 172.21.15.119:0/2102643424 shutdown_connections 2024-09-13T16:25:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2102643424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b815af50 0x7fa7b815b3e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2102643424 >> v2:172.21.15.145:3300/0 conn(0x7fa7b806dc80 0x7fa7b8070af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 -- 172.21.15.119:0/2102643424 >> 172.21.15.119:0/2102643424 conn(0x7fa7b80b0640 msgr2=0x7fa7b80b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 -- 172.21.15.119:0/2102643424 shutdown_connections 2024-09-13T16:25:40.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.197+0000 7fa7bca19700 1 -- 172.21.15.119:0/2102643424 wait complete. 2024-09-13T16:25:40.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 Processor -- start 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 -- start start 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b806dc80 0x7fa7b80a9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b80cdfc0 0x7fa7b814ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7b815af50 0x7fa7b81597d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b814f250 con 0x7fa7b806dc80 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80aa8d0 con 0x7fa7b815af50 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b80a8650 con 0x7fa7b80cdfc0 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b806dc80 0x7fa7b80a9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b806dc80 0x7fa7b80a9410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52630/0 (socket says 172.21.15.119:52630) 2024-09-13T16:25:40.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 -- 172.21.15.119:0/2881015140 learned_addr learned my addr 172.21.15.119:0/2881015140 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7affff700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b80cdfc0 0x7fa7b814ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b759e700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7b815af50 0x7fa7b81597d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 -- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b80cdfc0 msgr2=0x7fa7b814ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b80cdfc0 0x7fa7b814ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 -- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7b815af50 msgr2=0x7fa7b81597d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7b815af50 0x7fa7b81597d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 -- 172.21.15.119:0/2881015140 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a00094b0 con 0x7fa7b806dc80 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7b6d9d700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b806dc80 0x7fa7b80a9410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800c500 tx=0x7fa7a800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7af7fe700 1 -- 172.21.15.119:0/2881015140 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a80052d0 con 0x7fa7b806dc80 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7af7fe700 1 -- 172.21.15.119:0/2881015140 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a8005430 con 0x7fa7b806dc80 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7af7fe700 1 -- 172.21.15.119:0/2881015140 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a80055a0 con 0x7fa7b806dc80 2024-09-13T16:25:40.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b0009b30 con 0x7fa7b806dc80 2024-09-13T16:25:40.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.201+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b80a8c10 con 0x7fa7b806dc80 2024-09-13T16:25:40.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.205+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b806c960 con 0x7fa7b806dc80 2024-09-13T16:25:40.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.205+0000 7fa7af7fe700 1 -- 172.21.15.119:0/2881015140 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa7a80038e0 con 0x7fa7b806dc80 2024-09-13T16:25:40.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.205+0000 7fa7af7fe700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa798066760 0x7fa798068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.205+0000 7fa7af7fe700 1 -- 172.21.15.119:0/2881015140 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa7a8083cc0 con 0x7fa7b806dc80 2024-09-13T16:25:40.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.205+0000 7fa7affff700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa798066760 0x7fa798068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.209+0000 7fa7affff700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa798066760 0x7fa798068c20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009d80 tx=0x7fa7a0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.209+0000 7fa7af7fe700 1 -- 172.21.15.119:0/2881015140 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa7a8050d20 con 0x7fa7b806dc80 2024-09-13T16:25:40.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa7b8070af0 con 0x7fa7b806dc80 2024-09-13T16:25:40.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7af7fe700 1 -- 172.21.15.119:0/2881015140 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fa7a8017040 con 0x7fa7b806dc80 2024-09-13T16:25:40.409 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:40.409 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T16:25:40.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa798066760 msgr2=0x7fa798068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa798066760 0x7fa798068c20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009d80 tx=0x7fa7a0009240 comp rx=0 tx=0).stop 2024-09-13T16:25:40.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b806dc80 msgr2=0x7fa7b80a9410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b806dc80 0x7fa7b80a9410 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa7a800c500 tx=0x7fa7a800c810 comp rx=0 tx=0).stop 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 shutdown_connections 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa798066760 0x7fa798068c20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7b815af50 0x7fa7b81597d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa7b80cdfc0 0x7fa7b814ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 --2- 172.21.15.119:0/2881015140 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7b806dc80 0x7fa7b80a9410 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 >> 172.21.15.119:0/2881015140 conn(0x7fa7b80b0640 msgr2=0x7fa7b815c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 shutdown_connections 2024-09-13T16:25:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.405+0000 7fa7bca19700 1 -- 172.21.15.119:0/2881015140 wait complete. 2024-09-13T16:25:40.422 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-13T16:25:40.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:40.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f0314d14700 1 Processor -- start 2024-09-13T16:25:40.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f0314d14700 1 -- start start 2024-09-13T16:25:40.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f0314d14700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f03100adf00 0x7f03100b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f0314d14700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0310071770 con 0x7f0310159360 2024-09-13T16:25:40.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f0314d14700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f031015e0b0 con 0x7f031006dc40 2024-09-13T16:25:40.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f0314d14700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03100cc890 con 0x7f03100adf00 2024-09-13T16:25:40.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f030f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0310159360 0x7f031006d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33914/0 (socket says 172.21.15.119:33914) 2024-09-13T16:25:40.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f030f7fe700 1 -- 172.21.15.119:0/1285310578 learned_addr learned my addr 172.21.15.119:0/1285310578 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:40.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f030ffff700 1 --2- 172.21.15.119:0/1285310578 >> v2:172.21.15.145:3300/0 conn(0x7f03100adf00 0x7f03100b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f030e7fc700 1 -- 172.21.15.119:0/1285310578 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250646769 0 0) 0x7f0310071770 con 0x7f0310159360 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.525+0000 7f030e7fc700 1 -- 172.21.15.119:0/1285310578 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02f40035d0 con 0x7f0310159360 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030e7fc700 1 -- 172.21.15.119:0/1285310578 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811718489 0 0) 0x7f02f40035d0 con 0x7f0310159360 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030ffff700 1 -- 172.21.15.119:0/1285310578 >> v1:172.21.15.145:6789/0 conn(0x7f031006dc40 legacy=0x7f03100b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030ffff700 1 -- 172.21.15.119:0/1285310578 >> v1:172.21.15.119:6789/0 conn(0x7f0310159360 legacy=0x7f031006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030ffff700 1 -- 172.21.15.119:0/1285310578 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031015d2f0 con 0x7f03100adf00 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030ffff700 1 --2- 172.21.15.119:0/1285310578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03100adf00 0x7f03100b1f00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0308009d90 tx=0x7f030800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91e7455f855c4d4f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030e7fc700 1 -- 172.21.15.119:0/1285310578 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030800e070 con 0x7f03100adf00 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030e7fc700 1 -- 172.21.15.119:0/1285310578 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0308003a40 con 0x7f03100adf00 2024-09-13T16:25:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030e7fc700 1 -- 172.21.15.119:0/1285310578 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300001670 con 0x7f03100adf00 2024-09-13T16:25:40.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- 172.21.15.119:0/1285310578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03100adf00 msgr2=0x7f03100b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 --2- 172.21.15.119:0/1285310578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03100adf00 0x7f03100b1f00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f0308009d90 tx=0x7f030800b330 comp rx=0 tx=0).stop 2024-09-13T16:25:40.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- 172.21.15.119:0/1285310578 shutdown_connections 2024-09-13T16:25:40.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 --2- 172.21.15.119:0/1285310578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f03100adf00 0x7f03100b1f00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- 172.21.15.119:0/1285310578 >> 172.21.15.119:0/1285310578 conn(0x7f031006a350 msgr2=0x7f031006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:40.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- 172.21.15.119:0/1285310578 shutdown_connections 2024-09-13T16:25:40.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- 172.21.15.119:0/1285310578 wait complete. 2024-09-13T16:25:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 Processor -- start 2024-09-13T16:25:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- start start 2024-09-13T16:25:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f031006dc40 0x7f0310145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03100adf00 0x7f03101529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0310159360 0x7f03101f27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0310145640 con 0x7f03100adf00 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03101f3120 con 0x7f031006dc40 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f0314d14700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03101f3400 con 0x7f0310159360 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03100adf00 0x7f03101529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03100adf00 0x7f03101529a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52634/0 (socket says 172.21.15.119:52634) 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030effd700 1 -- 172.21.15.119:0/4078423086 learned_addr learned my addr 172.21.15.119:0/4078423086 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.529+0000 7f030f7fe700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f031006dc40 0x7f0310145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030ffff700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0310159360 0x7f03101f27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030effd700 1 -- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0310159360 msgr2=0x7f03101f27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030effd700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0310159360 0x7f03101f27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030effd700 1 -- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f031006dc40 msgr2=0x7f0310145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030effd700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f031006dc40 0x7f0310145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030effd700 1 -- 172.21.15.119:0/4078423086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0308009a70 con 0x7f03100adf00 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030f7fe700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f031006dc40 0x7f0310145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030effd700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03100adf00 0x7f03101529a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c750 tx=0x7f02fc00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0307fff700 1 -- 172.21.15.119:0/4078423086 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc0054e0 con 0x7f03100adf00 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0307fff700 1 -- 172.21.15.119:0/4078423086 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02fc0036a0 con 0x7f03100adf00 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03101f36e0 con 0x7f03100adf00 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0307fff700 1 -- 172.21.15.119:0/4078423086 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02fc003a30 con 0x7f03100adf00 2024-09-13T16:25:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0310159d60 con 0x7f03100adf00 2024-09-13T16:25:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0310159f80 con 0x7f03100adf00 2024-09-13T16:25:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0307fff700 1 -- 172.21.15.119:0/4078423086 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f02fc003b90 con 0x7f03100adf00 2024-09-13T16:25:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0307fff700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02f8066760 0x7f02f8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f0307fff700 1 -- 172.21.15.119:0/4078423086 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f02fc083940 con 0x7f03100adf00 2024-09-13T16:25:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030f7fe700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02f8066760 0x7f02f8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.533+0000 7f030f7fe700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02f8066760 0x7f02f8068c20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0300005a90 tx=0x7f0300001c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.541+0000 7f0307fff700 1 -- 172.21.15.119:0/4078423086 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f02fc050a50 con 0x7f03100adf00 2024-09-13T16:25:40.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.697+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03100adbc0 con 0x7f03100adf00 2024-09-13T16:25:40.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.697+0000 7f0307fff700 1 -- 172.21.15.119:0/4078423086 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f02fc017090 con 0x7f03100adf00 2024-09-13T16:25:40.703 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:40.703 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:40.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02f8066760 msgr2=0x7f02f8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02f8066760 0x7f02f8068c20 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0300005a90 tx=0x7f0300001c00 comp rx=0 tx=0).stop 2024-09-13T16:25:40.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03100adf00 msgr2=0x7f03101529a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03100adf00 0x7f03101529a0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f02fc00c750 tx=0x7f02fc00cb10 comp rx=0 tx=0).stop 2024-09-13T16:25:40.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 shutdown_connections 2024-09-13T16:25:40.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02f8066760 0x7f02f8068c20 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0310159360 0x7f03101f27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03100adf00 0x7f03101529a0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 --2- 172.21.15.119:0/4078423086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f031006dc40 0x7f0310145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 >> 172.21.15.119:0/4078423086 conn(0x7f031006a350 msgr2=0x7f031006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:40.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 shutdown_connections 2024-09-13T16:25:40.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.701+0000 7f0314d14700 1 -- 172.21.15.119:0/4078423086 wait complete. 2024-09-13T16:25:40.717 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:25:40.717 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:40.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eb2257700 1 Processor -- start 2024-09-13T16:25:40.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eb2257700 1 -- start start 2024-09-13T16:25:40.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eb2257700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4eac06bb60 0x7f4eac0d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eb2257700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac0ce880 con 0x7f4eac15cfb0 2024-09-13T16:25:40.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eb2257700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eac0ceb60 con 0x7f4eac06b710 2024-09-13T16:25:40.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eb2257700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eac0cee40 con 0x7f4eac06bb60 2024-09-13T16:25:40.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eab7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4eac15cfb0 0x7f4eac06ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33920/0 (socket says 172.21.15.119:33920) 2024-09-13T16:25:40.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eabfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4eac06bb60 0x7f4eac0d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eab7fe700 1 -- 172.21.15.119:0/343963382 learned_addr learned my addr 172.21.15.119:0/343963382 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eabfff700 1 -- 172.21.15.119:0/343963382 >> v1:172.21.15.145:6789/0 conn(0x7f4eac06b710 legacy=0x7f4eac0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eabfff700 1 -- 172.21.15.119:0/343963382 >> v1:172.21.15.119:6789/0 conn(0x7f4eac15cfb0 legacy=0x7f4eac06ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eabfff700 1 -- 172.21.15.119:0/343963382 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eac0d0e40 con 0x7f4eac06bb60 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eaa7fc700 1 -- 172.21.15.119:0/343963382 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957180149 0 0) 0x7f4eac0ce880 con 0x7f4eac15cfb0 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eabfff700 1 --2- 172.21.15.119:0/343963382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0d1d50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009d90 tx=0x7f4ea400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c997ac0f62e70a41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eaa7fc700 1 -- 172.21.15.119:0/343963382 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea400e070 con 0x7f4eac06bb60 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eaa7fc700 1 -- 172.21.15.119:0/343963382 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea400bbd0 con 0x7f4eac06bb60 2024-09-13T16:25:40.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.825+0000 7f4eaa7fc700 1 -- 172.21.15.119:0/343963382 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea4012430 con 0x7f4eac06bb60 2024-09-13T16:25:40.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 -- 172.21.15.119:0/343963382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 msgr2=0x7f4eac0d1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 --2- 172.21.15.119:0/343963382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0d1d50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4ea4009d90 tx=0x7f4ea400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:40.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 -- 172.21.15.119:0/343963382 shutdown_connections 2024-09-13T16:25:40.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 --2- 172.21.15.119:0/343963382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0d1d50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 -- 172.21.15.119:0/343963382 >> 172.21.15.119:0/343963382 conn(0x7f4eac0d2da0 msgr2=0x7f4eac0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:40.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 -- 172.21.15.119:0/343963382 shutdown_connections 2024-09-13T16:25:40.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 -- 172.21.15.119:0/343963382 wait complete. 2024-09-13T16:25:40.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 Processor -- start 2024-09-13T16:25:40.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.829+0000 7f4eb2257700 1 -- start start 2024-09-13T16:25:40.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4eac06b710 0x7f4eac0cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0ca2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4eac15cfb0 0x7f4eac1f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0d1220 con 0x7f4eac06bb60 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0b6d40 con 0x7f4eac15cfb0 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eac0cccc0 con 0x7f4eac06b710 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0ca2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0ca2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52652/0 (socket says 172.21.15.119:52652) 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 -- 172.21.15.119:0/2156463400 learned_addr learned my addr 172.21.15.119:0/2156463400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eab7fe700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4eac06b710 0x7f4eac0cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eabfff700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4eac15cfb0 0x7f4eac1f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 -- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4eac06b710 msgr2=0x7f4eac0cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4eac06b710 0x7f4eac0cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 -- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4eac15cfb0 msgr2=0x7f4eac1f2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4eac15cfb0 0x7f4eac1f2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:40.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 -- 172.21.15.119:0/2156463400 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea4009a70 con 0x7f4eac06bb60 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eabfff700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4eac15cfb0 0x7f4eac1f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eaaffd700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0ca2b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c530 tx=0x7f4ea000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4e93fff700 1 -- 172.21.15.119:0/2156463400 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00052d0 con 0x7f4eac06bb60 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4e93fff700 1 -- 172.21.15.119:0/2156463400 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea0005430 con 0x7f4eac06bb60 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eac0b66c0 con 0x7f4eac06bb60 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4e93fff700 1 -- 172.21.15.119:0/2156463400 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea00055a0 con 0x7f4eac06bb60 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eac0ca840 con 0x7f4eac06bb60 2024-09-13T16:25:40.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.833+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eac0b6e80 con 0x7f4eac06bb60 2024-09-13T16:25:40.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.837+0000 7f4e93fff700 1 -- 172.21.15.119:0/2156463400 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4ea00038e0 con 0x7f4eac06bb60 2024-09-13T16:25:40.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.837+0000 7f4e93fff700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e98066730 0x7f4e98068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:40.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.837+0000 7f4e93fff700 1 -- 172.21.15.119:0/2156463400 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4ea0083870 con 0x7f4eac06bb60 2024-09-13T16:25:40.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.837+0000 7f4eab7fe700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e98066730 0x7f4e98068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:40.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.837+0000 7f4eab7fe700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e98066730 0x7f4e98068bf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c000f60 tx=0x7f4e9c004a70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:40.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:40.845+0000 7f4e93fff700 1 -- 172.21.15.119:0/2156463400 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4eac0b6e80 con 0x7f4eac06bb60 2024-09-13T16:25:41.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.009+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4eac0b6e80 con 0x7f4eac06bb60 2024-09-13T16:25:41.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.009+0000 7f4e93fff700 1 -- 172.21.15.119:0/2156463400 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4eac0b6e80 con 0x7f4eac06bb60 2024-09-13T16:25:41.014 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e98066730 msgr2=0x7f4e98068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e98066730 0x7f4e98068bf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c000f60 tx=0x7f4e9c004a70 comp rx=0 tx=0).stop 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 msgr2=0x7f4eac0ca2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0ca2b0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f4ea000c530 tx=0x7f4ea000c840 comp rx=0 tx=0).stop 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 shutdown_connections 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e98066730 0x7f4e98068bf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4eac15cfb0 0x7f4eac1f2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4eac06bb60 0x7f4eac0ca2b0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 --2- 172.21.15.119:0/2156463400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4eac06b710 0x7f4eac0cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 >> 172.21.15.119:0/2156463400 conn(0x7f4eac0d2da0 msgr2=0x7f4eac0b28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 shutdown_connections 2024-09-13T16:25:41.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.013+0000 7f4eb2257700 1 -- 172.21.15.119:0/2156463400 wait complete. 2024-09-13T16:25:41.029 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:25:41.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.137+0000 7fe1b5ee0700 1 Processor -- start 2024-09-13T16:25:41.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.137+0000 7fe1b5ee0700 1 -- start start 2024-09-13T16:25:41.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b015aa50 0x7fe1b015bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe1b015c120 0x7fe1b00b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b015ae30 con 0x7fe1b00b4400 2024-09-13T16:25:41.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b015c580 con 0x7fe1b015c120 2024-09-13T16:25:41.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00b3f30 con 0x7fe1b015aa50 2024-09-13T16:25:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1affff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe1b00b4400 0x7fe1b00af240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33936/0 (socket says 172.21.15.119:33936) 2024-09-13T16:25:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1affff700 1 -- 172.21.15.119:0/1746497451 learned_addr learned my addr 172.21.15.119:0/1746497451 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1af7fe700 1 --2- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b015aa50 0x7fe1b015bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1aeffd700 1 --2- 172.21.15.119:0/1746497451 >> v2:172.21.15.145:3300/0 conn(0x7fe1b015c120 0x7fe1b00b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1ae7fc700 1 -- 172.21.15.119:0/1746497451 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377408848 0 0) 0x7fe1a8001490 con 0x7fe1b00b4400 2024-09-13T16:25:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1ae7fc700 1 -- 172.21.15.119:0/1746497451 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe19c0035d0 con 0x7fe1b00b4400 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1aeffd700 1 -- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b015aa50 msgr2=0x7fe1b015bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1aeffd700 1 --2- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b015aa50 0x7fe1b015bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1aeffd700 1 -- 172.21.15.119:0/1746497451 >> v1:172.21.15.119:6789/0 conn(0x7fe1b00b4400 legacy=0x7fe1b00af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1aeffd700 1 -- 172.21.15.119:0/1746497451 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b00ae480 con 0x7fe1b015c120 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1ae7fc700 1 -- 172.21.15.119:0/1746497451 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2405992130 0 0) 0x7fe19c0035d0 con 0x7fe1b00b4400 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1aeffd700 1 --2- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015c120 0x7fe1b00b39f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009850 tx=0x7fe1a0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1becb9bc35b16bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1ae7fc700 1 -- 172.21.15.119:0/1746497451 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8001b80 con 0x7fe1b015c120 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1ae7fc700 1 -- 172.21.15.119:0/1746497451 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a0004030 con 0x7fe1b015c120 2024-09-13T16:25:41.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1ae7fc700 1 -- 172.21.15.119:0/1746497451 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a00043e0 con 0x7fe1b015c120 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015c120 msgr2=0x7fe1b00b39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015c120 0x7fe1b00b39f0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009850 tx=0x7fe1a0009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1746497451 shutdown_connections 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015c120 0x7fe1b00b39f0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1746497451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b015aa50 0x7fe1b015bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1746497451 >> 172.21.15.119:0/1746497451 conn(0x7fe1b0099a50 msgr2=0x7fe1b0099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1746497451 shutdown_connections 2024-09-13T16:25:41.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.141+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1746497451 wait complete. 2024-09-13T16:25:41.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 Processor -- start 2024-09-13T16:25:41.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 -- start start 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b00b4400 0x7fe1b0154ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015aa50 0x7fe1b0151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1b015c120 0x7fe1b01463c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00b0c20 con 0x7fe1b015c120 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b0152480 con 0x7fe1b015aa50 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1b5ee0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b01526b0 con 0x7fe1b00b4400 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1b015c120 0x7fe1b01463c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1aeffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015aa50 0x7fe1b0151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1af7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b00b4400 0x7fe1b0154ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1b015c120 0x7fe1b01463c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52656/0 (socket says 172.21.15.119:52656) 2024-09-13T16:25:41.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 -- 172.21.15.119:0/1855427415 learned_addr learned my addr 172.21.15.119:0/1855427415 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 -- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b00b4400 msgr2=0x7fe1b0154ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b00b4400 0x7fe1b0154ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 -- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015aa50 msgr2=0x7fe1b0151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015aa50 0x7fe1b0151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 -- 172.21.15.119:0/1855427415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00094b0 con 0x7fe1b015c120 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1af7fe700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b00b4400 0x7fe1b0154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.145+0000 7fe1affff700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1b015c120 0x7fe1b01463c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800cc40 tx=0x7fe1a800cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1a7fff700 1 -- 172.21.15.119:0/1855427415 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a800aac0 con 0x7fe1b015c120 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b00b0280 con 0x7fe1b015c120 2024-09-13T16:25:41.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1a7fff700 1 -- 172.21.15.119:0/1855427415 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a800ac20 con 0x7fe1b015c120 2024-09-13T16:25:41.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1a7fff700 1 -- 172.21.15.119:0/1855427415 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a80070d0 con 0x7fe1b015c120 2024-09-13T16:25:41.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b00b0840 con 0x7fe1b015c120 2024-09-13T16:25:41.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b00b0d60 con 0x7fe1b015c120 2024-09-13T16:25:41.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1a7fff700 1 -- 172.21.15.119:0/1855427415 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe1a8007230 con 0x7fe1b015c120 2024-09-13T16:25:41.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1a7fff700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe190066730 0x7fe190068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1a7fff700 1 -- 172.21.15.119:0/1855427415 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe1a8083e90 con 0x7fe1b015c120 2024-09-13T16:25:41.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1af7fe700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe190066730 0x7fe190068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.149+0000 7fe1af7fe700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe190066730 0x7fe190068bf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0152ab0 tx=0x7fe198009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.161+0000 7fe1a7fff700 1 -- 172.21.15.119:0/1855427415 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe1b00b0d60 con 0x7fe1b015c120 2024-09-13T16:25:41.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.329+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe1b00b0d60 con 0x7fe1b015c120 2024-09-13T16:25:41.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.329+0000 7fe1a7fff700 1 -- 172.21.15.119:0/1855427415 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe1b00b0d60 con 0x7fe1b015c120 2024-09-13T16:25:41.334 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:41.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe190066730 msgr2=0x7fe190068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe190066730 0x7fe190068bf0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe1b0152ab0 tx=0x7fe198009240 comp rx=0 tx=0).stop 2024-09-13T16:25:41.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1b015c120 msgr2=0x7fe1b01463c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1b015c120 0x7fe1b01463c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800cc40 tx=0x7fe1a800cf50 comp rx=0 tx=0).stop 2024-09-13T16:25:41.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 shutdown_connections 2024-09-13T16:25:41.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe190066730 0x7fe190068bf0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1b015c120 0x7fe1b01463c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1b015aa50 0x7fe1b0151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 --2- 172.21.15.119:0/1855427415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1b00b4400 0x7fe1b0154ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 >> 172.21.15.119:0/1855427415 conn(0x7fe1b0099a50 msgr2=0x7fe1b006e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:41.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 shutdown_connections 2024-09-13T16:25:41.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.333+0000 7fe1b5ee0700 1 -- 172.21.15.119:0/1855427415 wait complete. 2024-09-13T16:25:41.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:25:41.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f0661774700 1 Processor -- start 2024-09-13T16:25:41.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f0661774700 1 -- start start 2024-09-13T16:25:41.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f0661774700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f065c15cc00 0x7f065c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f0661774700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f065c0d17e0 0x7f065c0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f0661774700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f065c0cdb00 con 0x7f065c0d0c20 2024-09-13T16:25:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f0661774700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c0cdde0 con 0x7f065c15cc00 2024-09-13T16:25:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f0661774700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f065c0ce0c0 con 0x7f065c0d17e0 2024-09-13T16:25:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.473+0000 7f065bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f065c15cc00 0x7f065c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f065c15cc00 0x7f065c0d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52666/0 (socket says 172.21.15.119:52666) 2024-09-13T16:25:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065bfff700 1 -- 172.21.15.119:0/2880054518 learned_addr learned my addr 172.21.15.119:0/2880054518 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0660f73700 1 --2- 172.21.15.119:0/2880054518 >> v2:172.21.15.145:3300/0 conn(0x7f065c0d17e0 0x7f065c0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065bfff700 1 -- 172.21.15.119:0/2880054518 >> v2:172.21.15.145:3300/0 conn(0x7f065c0d17e0 msgr2=0x7f065c0b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065bfff700 1 --2- 172.21.15.119:0/2880054518 >> v2:172.21.15.145:3300/0 conn(0x7f065c0d17e0 0x7f065c0b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065bfff700 1 -- 172.21.15.119:0/2880054518 >> v1:172.21.15.145:6789/0 conn(0x7f065c0d0c20 legacy=0x7f065c0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065bfff700 1 -- 172.21.15.119:0/2880054518 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f065c0ce3a0 con 0x7f065c15cc00 2024-09-13T16:25:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065bfff700 1 --2- 172.21.15.119:0/2880054518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c15cc00 0x7f065c0d06e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f065c0a0a20 tx=0x7f0644009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19f42fb074fd76cb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065affd700 1 -- 172.21.15.119:0/2880054518 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644011830 con 0x7f065c15cc00 2024-09-13T16:25:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065affd700 1 -- 172.21.15.119:0/2880054518 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0644011990 con 0x7f065c15cc00 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f065affd700 1 -- 172.21.15.119:0/2880054518 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644011d20 con 0x7f065c15cc00 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 -- 172.21.15.119:0/2880054518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c15cc00 msgr2=0x7f065c0d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 --2- 172.21.15.119:0/2880054518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c15cc00 0x7f065c0d06e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f065c0a0a20 tx=0x7f0644009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 -- 172.21.15.119:0/2880054518 shutdown_connections 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 --2- 172.21.15.119:0/2880054518 >> v2:172.21.15.145:3300/0 conn(0x7f065c0d17e0 0x7f065c0b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 --2- 172.21.15.119:0/2880054518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c15cc00 0x7f065c0d06e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 -- 172.21.15.119:0/2880054518 >> 172.21.15.119:0/2880054518 conn(0x7f065c08c080 msgr2=0x7f065c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 -- 172.21.15.119:0/2880054518 shutdown_connections 2024-09-13T16:25:41.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.477+0000 7f0661774700 1 -- 172.21.15.119:0/2880054518 wait complete. 2024-09-13T16:25:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 Processor -- start 2024-09-13T16:25:41.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 -- start start 2024-09-13T16:25:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f065c0d0c20 0x7f065c0c72f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f065c0d17e0 0x7f065c1518e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c0c6450 0x7f065c0b6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1f3f70 con 0x7f065c0c6450 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1f4400 con 0x7f065c0d0c20 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f065c1f3360 con 0x7f065c0d17e0 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c0c6450 0x7f065c0b6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c0c6450 0x7f065c0b6660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52678/0 (socket says 172.21.15.119:52678) 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f065b7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f065c0d17e0 0x7f065c1518e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 -- 172.21.15.119:0/2190922192 learned_addr learned my addr 172.21.15.119:0/2190922192 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f065bfff700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f065c0d0c20 0x7f065c0c72f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 -- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f065c0d17e0 msgr2=0x7f065c1518e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f065c0d17e0 0x7f065c1518e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 -- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f065c0d0c20 msgr2=0x7f065c0c72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f065c0d0c20 0x7f065c0c72f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 -- 172.21.15.119:0/2190922192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0644009570 con 0x7f065c0c6450 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0660f73700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c0c6450 0x7f065c0b6660 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f065400efc0 tx=0x7f0654009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0658ff9700 1 -- 172.21.15.119:0/2190922192 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0654009b20 con 0x7f065c0c6450 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0658ff9700 1 -- 172.21.15.119:0/2190922192 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065400f040 con 0x7f065c0c6450 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f065c1f3640 con 0x7f065c0c6450 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0658ff9700 1 -- 172.21.15.119:0/2190922192 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065400cc90 con 0x7f065c0c6450 2024-09-13T16:25:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.481+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f065c1f3c00 con 0x7f065c0c6450 2024-09-13T16:25:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.485+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f065c1f4120 con 0x7f065c0c6450 2024-09-13T16:25:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.485+0000 7f0658ff9700 1 -- 172.21.15.119:0/2190922192 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0654014480 con 0x7f065c0c6450 2024-09-13T16:25:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.485+0000 7f0658ff9700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f063c066730 0x7f063c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.485+0000 7f0658ff9700 1 -- 172.21.15.119:0/2190922192 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0654084f20 con 0x7f065c0c6450 2024-09-13T16:25:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.485+0000 7f065bfff700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f063c066730 0x7f063c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.489+0000 7f065bfff700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f063c066730 0x7f063c068bf0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f065c08b470 tx=0x7f0644008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.497+0000 7f0658ff9700 1 -- 172.21.15.119:0/2190922192 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f065c1f4120 con 0x7f065c0c6450 2024-09-13T16:25:41.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.669+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f065c0ce8c0 con 0x7f065c0c6450 2024-09-13T16:25:41.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0658ff9700 1 -- 172.21.15.119:0/2190922192 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0654018070 con 0x7f065c0c6450 2024-09-13T16:25:41.676 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f063c066730 msgr2=0x7f063c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f063c066730 0x7f063c068bf0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f065c08b470 tx=0x7f0644008030 comp rx=0 tx=0).stop 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c0c6450 msgr2=0x7f065c0b6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c0c6450 0x7f065c0b6660 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f065400efc0 tx=0x7f0654009f60 comp rx=0 tx=0).stop 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 shutdown_connections 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f063c066730 0x7f063c068bf0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f065c0c6450 0x7f065c0b6660 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f065c0d17e0 0x7f065c1518e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 --2- 172.21.15.119:0/2190922192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f065c0d0c20 0x7f065c0c72f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 >> 172.21.15.119:0/2190922192 conn(0x7f065c08c080 msgr2=0x7f065c15bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:41.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 shutdown_connections 2024-09-13T16:25:41.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.673+0000 7f0661774700 1 -- 172.21.15.119:0/2190922192 wait complete. 2024-09-13T16:25:41.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:25:41.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.801+0000 7fa0c9da4700 1 Processor -- start 2024-09-13T16:25:41.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.801+0000 7fa0c9da4700 1 -- start start 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa0c406b710 0x7fa0c406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa0c40d1640 0x7fa0c40d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c406c0a0 con 0x7fa0c415cfb0 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c40cee40 con 0x7fa0c40d1640 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c40d1220 con 0x7fa0c406b710 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa0c40d1640 0x7fa0c40d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c37fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa0c415cfb0 0x7fa0c406ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33942/0 (socket says 172.21.15.119:33942) 2024-09-13T16:25:41.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c37fe700 1 -- 172.21.15.119:0/2388627831 learned_addr learned my addr 172.21.15.119:0/2388627831 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c2ffd700 1 --2- 172.21.15.119:0/2388627831 >> v2:172.21.15.145:3300/0 conn(0x7fa0c406b710 0x7fa0c406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c3fff700 1 -- 172.21.15.119:0/2388627831 >> v2:172.21.15.145:3300/0 conn(0x7fa0c406b710 msgr2=0x7fa0c406bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c3fff700 1 --2- 172.21.15.119:0/2388627831 >> v2:172.21.15.145:3300/0 conn(0x7fa0c406b710 0x7fa0c406bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c3fff700 1 -- 172.21.15.119:0/2388627831 >> v1:172.21.15.119:6789/0 conn(0x7fa0c415cfb0 legacy=0x7fa0c406ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c3fff700 1 -- 172.21.15.119:0/2388627831 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c40ce880 con 0x7fa0c40d1640 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c3fff700 1 --2- 172.21.15.119:0/2388627831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c40d1640 0x7fa0c40d1ad0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40c64b0 tx=0x7fa0bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62fc5a85bf7c866 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c27fc700 1 -- 172.21.15.119:0/2388627831 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc00e070 con 0x7fa0c40d1640 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c27fc700 1 -- 172.21.15.119:0/2388627831 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0bc00bb20 con 0x7fa0c40d1640 2024-09-13T16:25:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c27fc700 1 -- 172.21.15.119:0/2388627831 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0bc012430 con 0x7fa0c40d1640 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- 172.21.15.119:0/2388627831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c40d1640 msgr2=0x7fa0c40d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/2388627831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c40d1640 0x7fa0c40d1ad0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa0c40c64b0 tx=0x7fa0bc00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- 172.21.15.119:0/2388627831 shutdown_connections 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/2388627831 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c40d1640 0x7fa0c40d1ad0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/2388627831 >> v2:172.21.15.145:3300/0 conn(0x7fa0c406b710 0x7fa0c406bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- 172.21.15.119:0/2388627831 >> 172.21.15.119:0/2388627831 conn(0x7fa0c40d2da0 msgr2=0x7fa0c40d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- 172.21.15.119:0/2388627831 shutdown_connections 2024-09-13T16:25:41.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.805+0000 7fa0c9da4700 1 -- 172.21.15.119:0/2388627831 wait complete. 2024-09-13T16:25:41.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 Processor -- start 2024-09-13T16:25:41.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 -- start start 2024-09-13T16:25:41.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c406b710 0x7fa0c40b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c40d1640 0x7fa0c40b7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c415cfb0 0x7fa0c40ca0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40b80f0 con 0x7fa0c406b710 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40b83d0 con 0x7fa0c415cfb0 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c40b86b0 con 0x7fa0c40d1640 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c406b710 0x7fa0c40b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c406b710 0x7fa0c40b7220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52708/0 (socket says 172.21.15.119:52708) 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 -- 172.21.15.119:0/1197310454 learned_addr learned my addr 172.21.15.119:0/1197310454 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c3fff700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c415cfb0 0x7fa0c40ca0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c2ffd700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c40d1640 0x7fa0c40b7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 -- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c40d1640 msgr2=0x7fa0c40b7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c40d1640 0x7fa0c40b7760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 -- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c415cfb0 msgr2=0x7fa0c40ca0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c415cfb0 0x7fa0c40ca0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 -- 172.21.15.119:0/1197310454 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0bc009a70 con 0x7fa0c406b710 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c37fe700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c406b710 0x7fa0c40b7220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001eb0 tx=0x7fa0b800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0abfff700 1 -- 172.21.15.119:0/1197310454 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8005d90 con 0x7fa0c406b710 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0abfff700 1 -- 172.21.15.119:0/1197310454 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b80070d0 con 0x7fa0c406b710 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0abfff700 1 -- 172.21.15.119:0/1197310454 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b8007440 con 0x7fa0c406b710 2024-09-13T16:25:41.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c40c68f0 con 0x7fa0c406b710 2024-09-13T16:25:41.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.809+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c40b7e80 con 0x7fa0c406b710 2024-09-13T16:25:41.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.813+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c40b88d0 con 0x7fa0c406b710 2024-09-13T16:25:41.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.813+0000 7fa0abfff700 1 -- 172.21.15.119:0/1197310454 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa0b80075a0 con 0x7fa0c406b710 2024-09-13T16:25:41.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.813+0000 7fa0abfff700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0b0066730 0x7fa0b0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:41.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.813+0000 7fa0abfff700 1 -- 172.21.15.119:0/1197310454 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fa0b80835b0 con 0x7fa0c406b710 2024-09-13T16:25:41.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.813+0000 7fa0c2ffd700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0b0066730 0x7fa0b0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:41.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.813+0000 7fa0c2ffd700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0b0066730 0x7fa0b0068bf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4159ef0 tx=0x7fa0b4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:41.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.821+0000 7fa0abfff700 1 -- 172.21.15.119:0/1197310454 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa0b80506e0 con 0x7fa0c406b710 2024-09-13T16:25:41.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.993+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa0c406c0a0 con 0x7fa0c406b710 2024-09-13T16:25:41.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.993+0000 7fa0abfff700 1 -- 172.21.15.119:0/1197310454 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa0c406c0a0 con 0x7fa0c406b710 2024-09-13T16:25:41.998 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:42.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0b0066730 msgr2=0x7fa0b0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0b0066730 0x7fa0b0068bf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa0c4159ef0 tx=0x7fa0b4008040 comp rx=0 tx=0).stop 2024-09-13T16:25:42.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c406b710 msgr2=0x7fa0c40b7220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c406b710 0x7fa0c40b7220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8001eb0 tx=0x7fa0b800c860 comp rx=0 tx=0).stop 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 shutdown_connections 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0b0066730 0x7fa0b0068bf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c415cfb0 0x7fa0c40ca0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c40d1640 0x7fa0c40b7760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 --2- 172.21.15.119:0/1197310454 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c406b710 0x7fa0c40b7220 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 >> 172.21.15.119:0/1197310454 conn(0x7fa0c40d2da0 msgr2=0x7fa0c40b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 shutdown_connections 2024-09-13T16:25:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:41.997+0000 7fa0c9da4700 1 -- 172.21.15.119:0/1197310454 wait complete. 2024-09-13T16:25:42.013 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:42.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50c58ec700 1 Processor -- start 2024-09-13T16:25:42.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50c58ec700 1 -- start start 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50c58ec700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f50c015ec20 0x7f50c0070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50c58ec700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f50c0165180 0x7f50c0161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50c58ec700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c0071210 con 0x7f50c015e840 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50c58ec700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c00714f0 con 0x7f50c015ec20 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50c58ec700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50c00717d0 con 0x7f50c0165180 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f50c015ec20 0x7f50c0070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f50c015ec20 0x7f50c0070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52712/0 (socket says 172.21.15.119:52712) 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 -- 172.21.15.119:0/4040476547 learned_addr learned my addr 172.21.15.119:0/4040476547 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50beffd700 1 --1- 172.21.15.119:0/4040476547 >> v1:172.21.15.119:6789/0 conn(0x7f50c015e840 0x7f50c0164a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33944/0 (socket says 172.21.15.119:33944) 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50bf7fe700 1 --2- 172.21.15.119:0/4040476547 >> v2:172.21.15.145:3300/0 conn(0x7f50c0165180 0x7f50c0161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 -- 172.21.15.119:0/4040476547 >> v2:172.21.15.145:3300/0 conn(0x7f50c0165180 msgr2=0x7f50c0161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:42.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 --2- 172.21.15.119:0/4040476547 >> v2:172.21.15.145:3300/0 conn(0x7f50c0165180 0x7f50c0161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 -- 172.21.15.119:0/4040476547 >> v1:172.21.15.119:6789/0 conn(0x7f50c015e840 legacy=0x7f50c0164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 -- 172.21.15.119:0/4040476547 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0071ab0 con 0x7f50c015ec20 2024-09-13T16:25:42.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50be7fc700 1 --2- 172.21.15.119:0/4040476547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015ec20 0x7f50c0070b80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50c00b3da0 tx=0x7f50a8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=283baaa529926f43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50bdffb700 1 -- 172.21.15.119:0/4040476547 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a800b8a0 con 0x7f50c015ec20 2024-09-13T16:25:42.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50bdffb700 1 -- 172.21.15.119:0/4040476547 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a800ba00 con 0x7f50c015ec20 2024-09-13T16:25:42.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.121+0000 7f50bdffb700 1 -- 172.21.15.119:0/4040476547 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a800bd90 con 0x7f50c015ec20 2024-09-13T16:25:42.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 -- 172.21.15.119:0/4040476547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015ec20 msgr2=0x7f50c0070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 --2- 172.21.15.119:0/4040476547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015ec20 0x7f50c0070b80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f50c00b3da0 tx=0x7f50a8009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 -- 172.21.15.119:0/4040476547 shutdown_connections 2024-09-13T16:25:42.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 --2- 172.21.15.119:0/4040476547 >> v2:172.21.15.145:3300/0 conn(0x7f50c0165180 0x7f50c0161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 --2- 172.21.15.119:0/4040476547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015ec20 0x7f50c0070b80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 -- 172.21.15.119:0/4040476547 >> 172.21.15.119:0/4040476547 conn(0x7f50c006a350 msgr2=0x7f50c006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:42.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 -- 172.21.15.119:0/4040476547 shutdown_connections 2024-09-13T16:25:42.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 -- 172.21.15.119:0/4040476547 wait complete. 2024-09-13T16:25:42.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 Processor -- start 2024-09-13T16:25:42.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.125+0000 7f50c58ec700 1 -- start start 2024-09-13T16:25:42.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015e840 0x7f50c00afc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50c0165180 0x7f50c0160060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50c00aee40 0x7f50c01605a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00ae080 con 0x7f50c015e840 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00ae360 con 0x7f50c0165180 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50c00ae640 con 0x7f50c00aee40 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015e840 0x7f50c00afc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015e840 0x7f50c00afc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:52726/0 (socket says 172.21.15.119:52726) 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 -- 172.21.15.119:0/2351471041 learned_addr learned my addr 172.21.15.119:0/2351471041 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50be7fc700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50c0165180 0x7f50c0160060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50bf7fe700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50c00aee40 0x7f50c01605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 -- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50c00aee40 msgr2=0x7f50c01605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50c00aee40 0x7f50c01605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 -- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50c0165180 msgr2=0x7f50c0160060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50c0165180 0x7f50c0160060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 -- 172.21.15.119:0/2351471041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a8009570 con 0x7f50c015e840 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50beffd700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015e840 0x7f50c00afc80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f50b000c700 tx=0x7f50b000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50b7fff700 1 -- 172.21.15.119:0/2351471041 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b0005cd0 con 0x7f50c015e840 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50b7fff700 1 -- 172.21.15.119:0/2351471041 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50b0005e30 con 0x7f50c015e840 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c00ae920 con 0x7f50c015e840 2024-09-13T16:25:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50b7fff700 1 -- 172.21.15.119:0/2351471041 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50b00072d0 con 0x7f50c015e840 2024-09-13T16:25:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.129+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c0160bd0 con 0x7f50c015e840 2024-09-13T16:25:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.133+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c0067760 con 0x7f50c015e840 2024-09-13T16:25:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.133+0000 7f50b7fff700 1 -- 172.21.15.119:0/2351471041 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f50b0007430 con 0x7f50c015e840 2024-09-13T16:25:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.133+0000 7f50b7fff700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f50a0066760 0x7f50a0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.133+0000 7f50b7fff700 1 -- 172.21.15.119:0/2351471041 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f50b00836a0 con 0x7f50c015e840 2024-09-13T16:25:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.133+0000 7f50be7fc700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f50a0066760 0x7f50a0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.133+0000 7f50be7fc700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f50a0066760 0x7f50a0068c20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000c00 tx=0x7f50a8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.141+0000 7f50b7fff700 1 -- 172.21.15.119:0/2351471041 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f50b00507c0 con 0x7f50c015e840 2024-09-13T16:25:42.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.345+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50c00ae1c0 con 0x7f50c015e840 2024-09-13T16:25:42.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50b7fff700 1 -- 172.21.15.119:0/2351471041 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f50c00ae1c0 con 0x7f50c015e840 2024-09-13T16:25:42.351 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:42.351 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:25:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f50a0066760 msgr2=0x7f50a0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f50a0066760 0x7f50a0068c20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f50a8000c00 tx=0x7f50a8008030 comp rx=0 tx=0).stop 2024-09-13T16:25:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015e840 msgr2=0x7f50c00afc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015e840 0x7f50c00afc80 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f50b000c700 tx=0x7f50b000ca10 comp rx=0 tx=0).stop 2024-09-13T16:25:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 shutdown_connections 2024-09-13T16:25:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f50a0066760 0x7f50a0068c20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f50c00aee40 0x7f50c01605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f50c0165180 0x7f50c0160060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 --2- 172.21.15.119:0/2351471041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f50c015e840 0x7f50c00afc80 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 >> 172.21.15.119:0/2351471041 conn(0x7f50c006a350 msgr2=0x7f50c006cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 shutdown_connections 2024-09-13T16:25:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.349+0000 7f50c58ec700 1 -- 172.21.15.119:0/2351471041 wait complete. 2024-09-13T16:25:42.357 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:25:42.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:42.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.473+0000 7f0b2e2d5700 1 Processor -- start 2024-09-13T16:25:42.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.473+0000 7f0b2e2d5700 1 -- start start 2024-09-13T16:25:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0b280adb30 0x7f0b280cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b2806cf60 con 0x7f0b2806bb90 2024-09-13T16:25:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b2806c350 con 0x7f0b280adb30 2024-09-13T16:25:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b2806c630 con 0x7f0b280ce040 2024-09-13T16:25:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2cad2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0b2806bb90 0x7f0b280cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58588/0 (socket says 172.21.15.119:58588) 2024-09-13T16:25:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2cad2700 1 -- 172.21.15.119:0/1045290045 learned_addr learned my addr 172.21.15.119:0/1045290045 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2dad4700 1 --2- 172.21.15.119:0/1045290045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2d2d3700 1 --2- 172.21.15.119:0/1045290045 >> v2:172.21.15.145:3300/0 conn(0x7f0b280adb30 0x7f0b280cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b27fff700 1 -- 172.21.15.119:0/1045290045 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067223648 0 0) 0x7f0b2806cf60 con 0x7f0b2806bb90 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b27fff700 1 -- 172.21.15.119:0/1045290045 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b140035d0 con 0x7f0b2806bb90 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2dad4700 1 -- 172.21.15.119:0/1045290045 >> v2:172.21.15.145:3300/0 conn(0x7f0b280adb30 msgr2=0x7f0b280cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2dad4700 1 --2- 172.21.15.119:0/1045290045 >> v2:172.21.15.145:3300/0 conn(0x7f0b280adb30 0x7f0b280cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2dad4700 1 -- 172.21.15.119:0/1045290045 >> v1:172.21.15.119:6789/0 conn(0x7f0b2806bb90 legacy=0x7f0b280cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b27fff700 1 -- 172.21.15.119:0/1045290045 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 746458491 0 0) 0x7f0b140035d0 con 0x7f0b2806bb90 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2dad4700 1 -- 172.21.15.119:0/1045290045 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2806c910 con 0x7f0b280ce040 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2dad4700 1 --2- 172.21.15.119:0/1045290045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b2380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009850 tx=0x7f0b20009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=585ba0b5edbaf7d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b27fff700 1 -- 172.21.15.119:0/1045290045 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b100019f0 con 0x7f0b280ce040 2024-09-13T16:25:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b27fff700 1 -- 172.21.15.119:0/1045290045 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b20004030 con 0x7f0b280ce040 2024-09-13T16:25:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b27fff700 1 -- 172.21.15.119:0/1045290045 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b200043a0 con 0x7f0b280ce040 2024-09-13T16:25:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/1045290045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 msgr2=0x7f0b280b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/1045290045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b2380 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0b20009850 tx=0x7f0b20009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/1045290045 shutdown_connections 2024-09-13T16:25:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/1045290045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b2380 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.477+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/1045290045 >> v2:172.21.15.145:3300/0 conn(0x7f0b280adb30 0x7f0b280cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/1045290045 >> 172.21.15.119:0/1045290045 conn(0x7f0b2806a350 msgr2=0x7f0b2806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/1045290045 shutdown_connections 2024-09-13T16:25:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/1045290045 wait complete. 2024-09-13T16:25:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 Processor -- start 2024-09-13T16:25:42.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- start start 2024-09-13T16:25:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2806bb90 0x7f0b280af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b280adb30 0x7f0b280a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b0480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280ae740 con 0x7f0b280adb30 2024-09-13T16:25:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280aea20 con 0x7f0b2806bb90 2024-09-13T16:25:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b280aed00 con 0x7f0b280ce040 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b280adb30 0x7f0b280a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b280adb30 0x7f0b280a8f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43018/0 (socket says 172.21.15.119:43018) 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 -- 172.21.15.119:0/4236276175 learned_addr learned my addr 172.21.15.119:0/4236276175 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2dad4700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2d2d3700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2806bb90 0x7f0b280af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 -- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 msgr2=0x7f0b280b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 -- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2806bb90 msgr2=0x7f0b280af7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2806bb90 0x7f0b280af7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 -- 172.21.15.119:0/4236276175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b200094b0 con 0x7f0b280adb30 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2cad2700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b280adb30 0x7f0b280a8f50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000cc70 tx=0x7f0b1000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b25ffb700 1 -- 172.21.15.119:0/4236276175 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b1000ade0 con 0x7f0b280adb30 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b25ffb700 1 -- 172.21.15.119:0/4236276175 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b1000ce70 con 0x7f0b280adb30 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b280aefe0 con 0x7f0b280adb30 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b25ffb700 1 -- 172.21.15.119:0/4236276175 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b100072f0 con 0x7f0b280adb30 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.481+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b280ae360 con 0x7f0b280adb30 2024-09-13T16:25:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.485+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b280af500 con 0x7f0b280adb30 2024-09-13T16:25:42.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.485+0000 7f0b25ffb700 1 -- 172.21.15.119:0/4236276175 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0b10007450 con 0x7f0b280adb30 2024-09-13T16:25:42.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.485+0000 7f0b25ffb700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b08066730 0x7f0b08068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.485+0000 7f0b25ffb700 1 -- 172.21.15.119:0/4236276175 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0b100843b0 con 0x7f0b280adb30 2024-09-13T16:25:42.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.485+0000 7f0b2d2d3700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b08066730 0x7f0b08068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.489+0000 7f0b2d2d3700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b08066730 0x7f0b08068bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009b80 tx=0x7f0b18008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.497+0000 7f0b25ffb700 1 -- 172.21.15.119:0/4236276175 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0b10051430 con 0x7f0b280adb30 2024-09-13T16:25:42.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.665+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0b280cc060 con 0x7f0b280adb30 2024-09-13T16:25:42.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.665+0000 7f0b25ffb700 1 -- 172.21.15.119:0/4236276175 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f0b10054a50 con 0x7f0b280adb30 2024-09-13T16:25:42.669 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:42.669 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b08066730 msgr2=0x7f0b08068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b08066730 0x7f0b08068bf0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0b18009b80 tx=0x7f0b18008040 comp rx=0 tx=0).stop 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b280adb30 msgr2=0x7f0b280a8f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b280adb30 0x7f0b280a8f50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0b1000cc70 tx=0x7f0b1000a340 comp rx=0 tx=0).stop 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 shutdown_connections 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b08066730 0x7f0b08068bf0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b280ce040 0x7f0b280b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b280adb30 0x7f0b280a8f50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 --2- 172.21.15.119:0/4236276175 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b2806bb90 0x7f0b280af7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 >> 172.21.15.119:0/4236276175 conn(0x7f0b2806a350 msgr2=0x7f0b28142010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:42.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 shutdown_connections 2024-09-13T16:25:42.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.669+0000 7f0b2e2d5700 1 -- 172.21.15.119:0/4236276175 wait complete. 2024-09-13T16:25:42.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:25:42.685 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21359e700 1 Processor -- start 2024-09-13T16:25:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21359e700 1 -- start start 2024-09-13T16:25:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff21415cc00 0x7ff2140d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d17e0 0x7ff2140b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21359e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2140cdb00 con 0x7ff2140d0c20 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2140cdde0 con 0x7ff21415cc00 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140ce0c0 con 0x7ff2140d17e0 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff211d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff2140d0c20 0x7ff2140d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58590/0 (socket says 172.21.15.119:58590) 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff211d9b700 1 -- 172.21.15.119:0/2068142598 learned_addr learned my addr 172.21.15.119:0/2068142598 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.805+0000 7ff21259c700 1 --2- 172.21.15.119:0/2068142598 >> v2:172.21.15.119:3300/0 conn(0x7ff21415cc00 0x7ff2140d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff212d9d700 1 --2- 172.21.15.119:0/2068142598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d17e0 0x7ff2140b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21159a700 1 -- 172.21.15.119:0/2068142598 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342109913 0 0) 0x7ff2140cdb00 con 0x7ff2140d0c20 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21159a700 1 -- 172.21.15.119:0/2068142598 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc0035d0 con 0x7ff2140d0c20 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21259c700 1 -- 172.21.15.119:0/2068142598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d17e0 msgr2=0x7ff2140b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21259c700 1 --2- 172.21.15.119:0/2068142598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d17e0 0x7ff2140b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21259c700 1 -- 172.21.15.119:0/2068142598 >> v1:172.21.15.119:6789/0 conn(0x7ff2140d0c20 legacy=0x7ff2140d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21159a700 1 -- 172.21.15.119:0/2068142598 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984665958 0 0) 0x7ff1fc0035d0 con 0x7ff2140d0c20 2024-09-13T16:25:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21259c700 1 -- 172.21.15.119:0/2068142598 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2140ce3a0 con 0x7ff21415cc00 2024-09-13T16:25:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21259c700 1 --2- 172.21.15.119:0/2068142598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2140d06e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff204009850 tx=0x7ff204009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d6ba4b85669cfba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21159a700 1 -- 172.21.15.119:0/2068142598 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208001670 con 0x7ff21415cc00 2024-09-13T16:25:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21159a700 1 -- 172.21.15.119:0/2068142598 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff204004030 con 0x7ff21415cc00 2024-09-13T16:25:42.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21159a700 1 -- 172.21.15.119:0/2068142598 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2040043e0 con 0x7ff21415cc00 2024-09-13T16:25:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 -- 172.21.15.119:0/2068142598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 msgr2=0x7ff2140d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 --2- 172.21.15.119:0/2068142598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2140d06e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff204009850 tx=0x7ff204009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 -- 172.21.15.119:0/2068142598 shutdown_connections 2024-09-13T16:25:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 --2- 172.21.15.119:0/2068142598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d17e0 0x7ff2140b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 --2- 172.21.15.119:0/2068142598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2140d06e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 -- 172.21.15.119:0/2068142598 >> 172.21.15.119:0/2068142598 conn(0x7ff21408c080 msgr2=0x7ff2140d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:42.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 -- 172.21.15.119:0/2068142598 shutdown_connections 2024-09-13T16:25:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 -- 172.21.15.119:0/2068142598 wait complete. 2024-09-13T16:25:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 Processor -- start 2024-09-13T16:25:42.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.809+0000 7ff21359e700 1 -- start start 2024-09-13T16:25:42.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d0c20 0x7ff2140b3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2140d17e0 0x7ff214157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2141584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff214158a10 con 0x7ff21415cc00 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2140caeb0 con 0x7ff2140d17e0 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141f0130 con 0x7ff2140d0c20 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2141584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2141584d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43044/0 (socket says 172.21.15.119:43044) 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 -- 172.21.15.119:0/1756267601 learned_addr learned my addr 172.21.15.119:0/1756267601 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21259c700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d0c20 0x7ff2140b3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff211d9b700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2140d17e0 0x7ff214157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 -- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d0c20 msgr2=0x7ff2140b3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d0c20 0x7ff2140b3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 -- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2140d17e0 msgr2=0x7ff214157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2140d17e0 0x7ff214157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 -- 172.21.15.119:0/1756267601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2040094b0 con 0x7ff21415cc00 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff212d9d700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2141584d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00efc0 tx=0x7ff20c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff202ffd700 1 -- 172.21.15.119:0/1756267601 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c00cc90 con 0x7ff21415cc00 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff202ffd700 1 -- 172.21.15.119:0/1756267601 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c00f040 con 0x7ff21415cc00 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff202ffd700 1 -- 172.21.15.119:0/1756267601 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c014480 con 0x7ff21415cc00 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2141563a0 con 0x7ff21415cc00 2024-09-13T16:25:42.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.813+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2141f35d0 con 0x7ff21415cc00 2024-09-13T16:25:42.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.817+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2140ce8c0 con 0x7ff21415cc00 2024-09-13T16:25:42.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.817+0000 7ff202ffd700 1 -- 172.21.15.119:0/1756267601 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff20c014600 con 0x7ff21415cc00 2024-09-13T16:25:42.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.817+0000 7ff202ffd700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff1f4066730 0x7ff1f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:42.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.817+0000 7ff202ffd700 1 -- 172.21.15.119:0/1756267601 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff20c084c40 con 0x7ff21415cc00 2024-09-13T16:25:42.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.817+0000 7ff21259c700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff1f4066730 0x7ff1f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:42.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.817+0000 7ff21259c700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff1f4066730 0x7ff1f4068bf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2040090b0 tx=0x7ff204008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:42.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.825+0000 7ff202ffd700 1 -- 172.21.15.119:0/1756267601 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff20c051cd0 con 0x7ff21415cc00 2024-09-13T16:25:43.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.997+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff21408c560 con 0x7ff21415cc00 2024-09-13T16:25:43.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:42.997+0000 7ff202ffd700 1 -- 172.21.15.119:0/1756267601 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff20c0180a0 con 0x7ff21415cc00 2024-09-13T16:25:43.002 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff1f4066730 msgr2=0x7ff1f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff1f4066730 0x7ff1f4068bf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ff2040090b0 tx=0x7ff204008030 comp rx=0 tx=0).stop 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 msgr2=0x7ff2141584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2141584d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00efc0 tx=0x7ff20c00c3a0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 shutdown_connections 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff1f4066730 0x7ff1f4068bf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff21415cc00 0x7ff2141584d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2140d17e0 0x7ff214157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 --2- 172.21.15.119:0/1756267601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2140d0c20 0x7ff2140b3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 >> 172.21.15.119:0/1756267601 conn(0x7ff21408c080 msgr2=0x7ff2140b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 shutdown_connections 2024-09-13T16:25:43.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.001+0000 7ff21359e700 1 -- 172.21.15.119:0/1756267601 wait complete. 2024-09-13T16:25:43.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:25:43.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc342aae700 1 Processor -- start 2024-09-13T16:25:43.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc342aae700 1 -- start start 2024-09-13T16:25:43.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc342aae700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 0x7fc33c06c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc342aae700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c06cb40 con 0x7fc33c0cc010 2024-09-13T16:25:43.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc342aae700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c06ce20 con 0x7fc33c0ce1d0 2024-09-13T16:25:43.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc342aae700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0b1c50 con 0x7fc33c144180 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc3412ab700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc33c0cc010 0x7fc33c0cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58596/0 (socket says 172.21.15.119:58596) 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc3412ab700 1 -- 172.21.15.119:0/2177760548 learned_addr learned my addr 172.21.15.119:0/2177760548 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc341aac700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 0x7fc33c06c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070581821 0 0) 0x7fc33c06cb40 con 0x7fc33c0cc010 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3280035d0 con 0x7fc33c0cc010 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269490879 0 0) 0x7fc3280035d0 con 0x7fc33c0cc010 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 msgr2=0x7fc33c06c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 0x7fc33c06c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 >> v1:172.21.15.145:6789/0 conn(0x7fc33c0ce1d0 legacy=0x7fc33c0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c06d100 con 0x7fc33c0cc010 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fc338001670 con 0x7fc33c0cc010 2024-09-13T16:25:43.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 >> v1:172.21.15.119:6789/0 conn(0x7fc33c0cc010 legacy=0x7fc33c0cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc328003a50 0x7fc328003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328004870 0x7fc328004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 0x7fc328005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0b1c50 con 0x7fc328003a50 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc32800bae0 con 0x7fc328004870 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc32800bca0 con 0x7fc33c144180 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc341aac700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc328003a50 0x7fc328003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc3412ab700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328004870 0x7fc328004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc3422ad700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 0x7fc328005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc341aac700 1 -- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 msgr2=0x7fc328005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:43.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc341aac700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 0x7fc328005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc341aac700 1 -- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328004870 msgr2=0x7fc328004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.133+0000 7fc341aac700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328004870 0x7fc328004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc341aac700 1 -- 172.21.15.119:0/2177760548 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280035d0 con 0x7fc328003a50 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc3422ad700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c144180 0x7fc328005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc341aac700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc328003a50 0x7fc328003e70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009b40 tx=0x7fc3280058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f008f71ffc59f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c00e6c0 con 0x7fc328003a50 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c00e820 con 0x7fc328003a50 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc340aaa700 1 -- 172.21.15.119:0/2177760548 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c00eb90 con 0x7fc328003a50 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc342aae700 1 -- 172.21.15.119:0/2177760548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc328003a50 msgr2=0x7fc328003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc342aae700 1 --2- 172.21.15.119:0/2177760548 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc328003a50 0x7fc328003e70 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fc32c009b40 tx=0x7fc3280058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc3422ad700 1 -- 172.21.15.119:0/2177760548 reap_dead start 2024-09-13T16:25:43.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc342aae700 1 -- 172.21.15.119:0/2177760548 shutdown_connections 2024-09-13T16:25:43.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc342aae700 1 -- 172.21.15.119:0/2177760548 >> 172.21.15.119:0/2177760548 conn(0x7fc33c0b0980 msgr2=0x7fc33c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:43.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc342aae700 1 -- 172.21.15.119:0/2177760548 shutdown_connections 2024-09-13T16:25:43.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.137+0000 7fc342aae700 1 -- 172.21.15.119:0/2177760548 wait complete. 2024-09-13T16:25:43.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 Processor -- start 2024-09-13T16:25:43.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 -- start start 2024-09-13T16:25:43.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328003a50 0x7fc33c0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c13b210 0x7fc33c13c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33c13cc10 0x7fc33c13f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c13ef00 con 0x7fc33c13cc10 2024-09-13T16:25:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c13c020 con 0x7fc328003a50 2024-09-13T16:25:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc342aae700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c13b780 con 0x7fc33c13b210 2024-09-13T16:25:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33c13cc10 0x7fc33c13f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc341aac700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328003a50 0x7fc33c0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3412ab700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c13b210 0x7fc33c13c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33c13cc10 0x7fc33c13f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43058/0 (socket says 172.21.15.119:43058) 2024-09-13T16:25:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 -- 172.21.15.119:0/3307539 learned_addr learned my addr 172.21.15.119:0/3307539 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 -- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c13b210 msgr2=0x7fc33c13c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c13b210 0x7fc33c13c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 -- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328003a50 msgr2=0x7fc33c0a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328003a50 0x7fc33c0a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 -- 172.21.15.119:0/3307539 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c13ba60 con 0x7fc33c13cc10 2024-09-13T16:25:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3422ad700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33c13cc10 0x7fc33c13f390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc334000f80 tx=0x7fc33400ed50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.141+0000 7fc3327fc700 1 -- 172.21.15.119:0/3307539 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc33400eee0 con 0x7fc33c13cc10 2024-09-13T16:25:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc33c13bd40 con 0x7fc33c13cc10 2024-09-13T16:25:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc3327fc700 1 -- 172.21.15.119:0/3307539 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc334003c30 con 0x7fc33c13cc10 2024-09-13T16:25:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc3327fc700 1 -- 172.21.15.119:0/3307539 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3340050d0 con 0x7fc33c13cc10 2024-09-13T16:25:43.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c13d090 con 0x7fc33c13cc10 2024-09-13T16:25:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c13d2b0 con 0x7fc33c13cc10 2024-09-13T16:25:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc3327fc700 1 -- 172.21.15.119:0/3307539 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc334005230 con 0x7fc33c13cc10 2024-09-13T16:25:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc3327fc700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc3327fc700 1 -- 172.21.15.119:0/3307539 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc334084290 con 0x7fc33c13cc10 2024-09-13T16:25:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.145+0000 7fc341aac700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.149+0000 7fc341aac700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0090b0 tx=0x7fc32c01e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.153+0000 7fc3327fc700 1 -- 172.21.15.119:0/3307539 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc3340513a0 con 0x7fc33c13cc10 2024-09-13T16:25:43.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.321+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc33c06c480 con 0x7fc33c13cc10 2024-09-13T16:25:43.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.321+0000 7fc3327fc700 1 -- 172.21.15.119:0/3307539 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc3340549c0 con 0x7fc33c13cc10 2024-09-13T16:25:43.325 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:43.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.321+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 msgr2=0x7fc320068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.321+0000 7fc342aae700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc32c0090b0 tx=0x7fc32c01e040 comp rx=0 tx=0).stop 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.321+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33c13cc10 msgr2=0x7fc33c13f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.321+0000 7fc342aae700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33c13cc10 0x7fc33c13f390 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fc334000f80 tx=0x7fc33400ed50 comp rx=0 tx=0).stop 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 shutdown_connections 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33c13cc10 0x7fc33c13f390 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33c13b210 0x7fc33c13c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 --2- 172.21.15.119:0/3307539 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc328003a50 0x7fc33c0a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 >> 172.21.15.119:0/3307539 conn(0x7fc33c0b0980 msgr2=0x7fc33c066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 shutdown_connections 2024-09-13T16:25:43.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.325+0000 7fc342aae700 1 -- 172.21.15.119:0/3307539 wait complete. 2024-09-13T16:25:43.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:25:43.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 Processor -- start 2024-09-13T16:25:43.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- start start 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc44815cad0 0x7fc4480d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4480cf930 0x7fc4480ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4480d05b0 con 0x7fc4480ce0d0 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4480d0870 con 0x7fc4480cf930 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480d0b50 con 0x7fc44815cad0 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4480cf930 0x7fc4480ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4480cf930 0x7fc4480ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43066/0 (socket says 172.21.15.119:43066) 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 -- 172.21.15.119:0/408294280 learned_addr learned my addr 172.21.15.119:0/408294280 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:43.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc447fff700 1 --2- 172.21.15.119:0/408294280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc44815cad0 0x7fc4480d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 -- 172.21.15.119:0/408294280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc44815cad0 msgr2=0x7fc4480d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 --2- 172.21.15.119:0/408294280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc44815cad0 0x7fc4480d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 -- 172.21.15.119:0/408294280 >> v1:172.21.15.119:6789/0 conn(0x7fc4480ce0d0 legacy=0x7fc4480cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 -- 172.21.15.119:0/408294280 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4480d0e30 con 0x7fc4480cf930 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc447fff700 1 --2- 172.21.15.119:0/408294280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc44815cad0 0x7fc4480d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44c8dc700 1 --2- 172.21.15.119:0/408294280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4480cf930 0x7fc4480ce510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc440009e50 tx=0x7fc44000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af42c2af52eed632 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc446ffd700 1 -- 172.21.15.119:0/408294280 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44000e070 con 0x7fc4480cf930 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc446ffd700 1 -- 172.21.15.119:0/408294280 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44000b9c0 con 0x7fc4480cf930 2024-09-13T16:25:43.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc446ffd700 1 -- 172.21.15.119:0/408294280 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44000bd70 con 0x7fc4480cf930 2024-09-13T16:25:43.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- 172.21.15.119:0/408294280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4480cf930 msgr2=0x7fc4480ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/408294280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4480cf930 0x7fc4480ce510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fc440009e50 tx=0x7fc44000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- 172.21.15.119:0/408294280 shutdown_connections 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/408294280 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4480cf930 0x7fc4480ce510 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/408294280 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc44815cad0 0x7fc4480d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- 172.21.15.119:0/408294280 >> 172.21.15.119:0/408294280 conn(0x7fc44808bf70 msgr2=0x7fc4480d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- 172.21.15.119:0/408294280 shutdown_connections 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.457+0000 7fc44d0dd700 1 -- 172.21.15.119:0/408294280 wait complete. 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 Processor -- start 2024-09-13T16:25:43.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 -- start start 2024-09-13T16:25:43.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4480ce0d0 0x7fc44809c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4480cf930 0x7fc448066300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc44815cad0 0x7fc44809d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44809cba0 con 0x7fc44815cad0 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4480a0650 con 0x7fc4480ce0d0 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44d0dd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44809f5b0 con 0x7fc4480cf930 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc44815cad0 0x7fc44809d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc44815cad0 0x7fc44809d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43080/0 (socket says 172.21.15.119:43080) 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 -- 172.21.15.119:0/3175223086 learned_addr learned my addr 172.21.15.119:0/3175223086 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc4477fe700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4480cf930 0x7fc448066300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc447fff700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4480ce0d0 0x7fc44809c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 -- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4480cf930 msgr2=0x7fc448066300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4480cf930 0x7fc448066300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 -- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4480ce0d0 msgr2=0x7fc44809c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4480ce0d0 0x7fc44809c660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 -- 172.21.15.119:0/3175223086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc440009b30 con 0x7fc44815cad0 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc44c8dc700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc44815cad0 0x7fc44809d440 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc440012000 tx=0x7fc440004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.461+0000 7fc444ff9700 1 -- 172.21.15.119:0/3175223086 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc440004960 con 0x7fc44815cad0 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc444ff9700 1 -- 172.21.15.119:0/3175223086 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4400092e0 con 0x7fc44815cad0 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44809f890 con 0x7fc44815cad0 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc444ff9700 1 -- 172.21.15.119:0/3175223086 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44001c680 con 0x7fc44815cad0 2024-09-13T16:25:43.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc44809fe50 con 0x7fc44815cad0 2024-09-13T16:25:43.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4480ce510 con 0x7fc44815cad0 2024-09-13T16:25:43.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc444ff9700 1 -- 172.21.15.119:0/3175223086 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc44001c7e0 con 0x7fc44815cad0 2024-09-13T16:25:43.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc444ff9700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc428066730 0x7fc428068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc444ff9700 1 -- 172.21.15.119:0/3175223086 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc440085ed0 con 0x7fc44815cad0 2024-09-13T16:25:43.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.465+0000 7fc447fff700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc428066730 0x7fc428068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.469+0000 7fc447fff700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc428066730 0x7fc428068bf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc43000a9a0 tx=0x7fc430008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.473+0000 7fc444ff9700 1 -- 172.21.15.119:0/3175223086 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc440052fe0 con 0x7fc44815cad0 2024-09-13T16:25:43.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.641+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc4480d38f0 con 0x7fc44815cad0 2024-09-13T16:25:43.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.641+0000 7fc444ff9700 1 -- 172.21.15.119:0/3175223086 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc440056600 con 0x7fc44815cad0 2024-09-13T16:25:43.646 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:25:43.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc428066730 msgr2=0x7fc428068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc428066730 0x7fc428068bf0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fc43000a9a0 tx=0x7fc430008040 comp rx=0 tx=0).stop 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc44815cad0 msgr2=0x7fc44809d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc44815cad0 0x7fc44809d440 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fc440012000 tx=0x7fc440004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 shutdown_connections 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc428066730 0x7fc428068bf0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc44815cad0 0x7fc44809d440 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4480cf930 0x7fc448066300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 --2- 172.21.15.119:0/3175223086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4480ce0d0 0x7fc44809c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 >> 172.21.15.119:0/3175223086 conn(0x7fc44808bf70 msgr2=0x7fc4480d1900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 shutdown_connections 2024-09-13T16:25:43.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.645+0000 7fc44d0dd700 1 -- 172.21.15.119:0/3175223086 wait complete. 2024-09-13T16:25:43.666 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:25:43.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4efece700 1 Processor -- start 2024-09-13T16:25:43.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4efece700 1 -- start start 2024-09-13T16:25:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4efece700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc4e806db40 0x7fc4e816c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4efece700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e8162270 con 0x7fc4e8071c10 2024-09-13T16:25:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4efece700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4e806ce40 con 0x7fc4e80b1ad0 2024-09-13T16:25:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4efece700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4e80702f0 con 0x7fc4e806db40 2024-09-13T16:25:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ed469700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc4e8071c10 0x7fc4e8168a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58616/0 (socket says 172.21.15.119:58616) 2024-09-13T16:25:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ed469700 1 -- 172.21.15.119:0/482225813 learned_addr learned my addr 172.21.15.119:0/482225813 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:43.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ee46b700 1 --2- 172.21.15.119:0/482225813 >> v2:172.21.15.145:3300/0 conn(0x7fc4e806db40 0x7fc4e816c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ecc68700 1 -- 172.21.15.119:0/482225813 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740823604 0 0) 0x7fc4e8162270 con 0x7fc4e8071c10 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ecc68700 1 -- 172.21.15.119:0/482225813 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4d40035d0 con 0x7fc4e8071c10 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ee46b700 1 -- 172.21.15.119:0/482225813 >> v1:172.21.15.145:6789/0 conn(0x7fc4e80b1ad0 legacy=0x7fc4e80b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ee46b700 1 -- 172.21.15.119:0/482225813 >> v1:172.21.15.119:6789/0 conn(0x7fc4e8071c10 legacy=0x7fc4e8168a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ee46b700 1 -- 172.21.15.119:0/482225813 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e8071730 con 0x7fc4e806db40 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.781+0000 7fc4ee46b700 1 --2- 172.21.15.119:0/482225813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e806db40 0x7fc4e816c140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009d90 tx=0x7fc4e000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c98989ca2347919b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4ecc68700 1 -- 172.21.15.119:0/482225813 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e000e070 con 0x7fc4e806db40 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4ecc68700 1 -- 172.21.15.119:0/482225813 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e000bb20 con 0x7fc4e806db40 2024-09-13T16:25:43.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4ecc68700 1 -- 172.21.15.119:0/482225813 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e0012430 con 0x7fc4e806db40 2024-09-13T16:25:43.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 -- 172.21.15.119:0/482225813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e806db40 msgr2=0x7fc4e816c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 --2- 172.21.15.119:0/482225813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e806db40 0x7fc4e816c140 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0009d90 tx=0x7fc4e000b330 comp rx=0 tx=0).stop 2024-09-13T16:25:43.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 -- 172.21.15.119:0/482225813 shutdown_connections 2024-09-13T16:25:43.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 --2- 172.21.15.119:0/482225813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e806db40 0x7fc4e816c140 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 -- 172.21.15.119:0/482225813 >> 172.21.15.119:0/482225813 conn(0x7fc4e80b0560 msgr2=0x7fc4e80b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:43.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 -- 172.21.15.119:0/482225813 shutdown_connections 2024-09-13T16:25:43.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 -- 172.21.15.119:0/482225813 wait complete. 2024-09-13T16:25:43.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 Processor -- start 2024-09-13T16:25:43.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.785+0000 7fc4efece700 1 -- start start 2024-09-13T16:25:43.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4e806db40 0x7fc4e80aa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4e8071c10 0x7fc4e8162cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e80b1ad0 0x7fc4e8163230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e80ab2e0 con 0x7fc4e806db40 2024-09-13T16:25:43.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e80ab5c0 con 0x7fc4e80b1ad0 2024-09-13T16:25:43.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4e80ab8a0 con 0x7fc4e8071c10 2024-09-13T16:25:43.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4e806db40 0x7fc4e80aa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4e806db40 0x7fc4e80aa9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43082/0 (socket says 172.21.15.119:43082) 2024-09-13T16:25:43.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 -- 172.21.15.119:0/3153680699 learned_addr learned my addr 172.21.15.119:0/3153680699 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:43.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4ee46b700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e80b1ad0 0x7fc4e8163230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4ed469700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4e8071c10 0x7fc4e8162cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 -- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4e8071c10 msgr2=0x7fc4e8162cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4e8071c10 0x7fc4e8162cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 -- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e80b1ad0 msgr2=0x7fc4e8163230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e80b1ad0 0x7fc4e8163230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 -- 172.21.15.119:0/3153680699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4e0009a70 con 0x7fc4e806db40 2024-09-13T16:25:43.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4ed469700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4e8071c10 0x7fc4e8162cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:43.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4edc6a700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4e806db40 0x7fc4e80aa9b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400c500 tx=0x7fc4e400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4da7fc700 1 -- 172.21.15.119:0/3153680699 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40052d0 con 0x7fc4e806db40 2024-09-13T16:25:43.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4da7fc700 1 -- 172.21.15.119:0/3153680699 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e4005430 con 0x7fc4e806db40 2024-09-13T16:25:43.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4e80abb80 con 0x7fc4e806db40 2024-09-13T16:25:43.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4da7fc700 1 -- 172.21.15.119:0/3153680699 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e40055a0 con 0x7fc4e806db40 2024-09-13T16:25:43.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4e815d790 con 0x7fc4e806db40 2024-09-13T16:25:43.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.789+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4e8073590 con 0x7fc4e806db40 2024-09-13T16:25:43.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.793+0000 7fc4da7fc700 1 -- 172.21.15.119:0/3153680699 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc4e40038e0 con 0x7fc4e806db40 2024-09-13T16:25:43.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.793+0000 7fc4da7fc700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4cc066730 0x7fc4cc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:43.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.793+0000 7fc4da7fc700 1 -- 172.21.15.119:0/3153680699 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc4e4083680 con 0x7fc4e806db40 2024-09-13T16:25:43.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.793+0000 7fc4ed469700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4cc066730 0x7fc4cc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:43.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.793+0000 7fc4ed469700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4cc066730 0x7fc4cc068bf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc4e81f9e60 tx=0x7fc4dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:43.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.797+0000 7fc4da7fc700 1 -- 172.21.15.119:0/3153680699 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc4e40506f0 con 0x7fc4e806db40 2024-09-13T16:25:43.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc4e806ce40 con 0x7fc4e806db40 2024-09-13T16:25:43.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4da7fc700 1 -- 172.21.15.119:0/3153680699 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc4e806ce40 con 0x7fc4e806db40 2024-09-13T16:25:43.973 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:43.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4cc066730 msgr2=0x7fc4cc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4cc066730 0x7fc4cc068bf0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fc4e81f9e60 tx=0x7fc4dc009040 comp rx=0 tx=0).stop 2024-09-13T16:25:43.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4e806db40 msgr2=0x7fc4e80aa9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:43.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4e806db40 0x7fc4e80aa9b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc4e400c500 tx=0x7fc4e400c810 comp rx=0 tx=0).stop 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 shutdown_connections 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4cc066730 0x7fc4cc068bf0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4e80b1ad0 0x7fc4e8163230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4e8071c10 0x7fc4e8162cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 --2- 172.21.15.119:0/3153680699 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4e806db40 0x7fc4e80aa9b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.969+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 >> 172.21.15.119:0/3153680699 conn(0x7fc4e80b0560 msgr2=0x7fc4e8161030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.973+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 shutdown_connections 2024-09-13T16:25:43.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:43.973+0000 7fc4efece700 1 -- 172.21.15.119:0/3153680699 wait complete. 2024-09-13T16:25:43.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:44.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.105+0000 7f5040a09700 1 Processor -- start 2024-09-13T16:25:44.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.105+0000 7f5040a09700 1 -- start start 2024-09-13T16:25:44.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f503c0adb30 0x7f503c0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f503c06bb90 0x7f503c0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f503c06cf60 con 0x7f503c0cde70 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503c0b3390 con 0x7f503c06bb90 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f503c06c350 con 0x7f503c0adb30 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f503c06bb90 0x7f503c0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f503c06bb90 0x7f503c0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43086/0 (socket says 172.21.15.119:43086) 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 -- 172.21.15.119:0/3159847490 learned_addr learned my addr 172.21.15.119:0/3159847490 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503b7fe700 1 --2- 172.21.15.119:0/3159847490 >> v2:172.21.15.145:3300/0 conn(0x7f503c0adb30 0x7f503c0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 -- 172.21.15.119:0/3159847490 >> v2:172.21.15.145:3300/0 conn(0x7f503c0adb30 msgr2=0x7f503c0cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 --2- 172.21.15.119:0/3159847490 >> v2:172.21.15.145:3300/0 conn(0x7f503c0adb30 0x7f503c0cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 -- 172.21.15.119:0/3159847490 >> v1:172.21.15.145:6789/0 conn(0x7f503c0cde70 legacy=0x7f503c15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 -- 172.21.15.119:0/3159847490 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f503c06c630 con 0x7f503c06bb90 2024-09-13T16:25:44.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503affd700 1 --2- 172.21.15.119:0/3159847490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c0cd930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f503c071fa0 tx=0x7f5030009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff01edbab33940c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503a7fc700 1 -- 172.21.15.119:0/3159847490 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5030011830 con 0x7f503c06bb90 2024-09-13T16:25:44.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f503a7fc700 1 -- 172.21.15.119:0/3159847490 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5030011990 con 0x7f503c06bb90 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- 172.21.15.119:0/3159847490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 msgr2=0x7f503c0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 --2- 172.21.15.119:0/3159847490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c0cd930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f503c071fa0 tx=0x7f5030009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- 172.21.15.119:0/3159847490 shutdown_connections 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 --2- 172.21.15.119:0/3159847490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c0cd930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 --2- 172.21.15.119:0/3159847490 >> v2:172.21.15.145:3300/0 conn(0x7f503c0adb30 0x7f503c0cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- 172.21.15.119:0/3159847490 >> 172.21.15.119:0/3159847490 conn(0x7f503c06a350 msgr2=0x7f503c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- 172.21.15.119:0/3159847490 shutdown_connections 2024-09-13T16:25:44.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.109+0000 7f5040a09700 1 -- 172.21.15.119:0/3159847490 wait complete. 2024-09-13T16:25:44.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.113+0000 7f5040a09700 1 Processor -- start 2024-09-13T16:25:44.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.113+0000 7f5040a09700 1 -- start start 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.113+0000 7f5040a09700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c13c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.113+0000 7f5040a09700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f503c0adb30 0x7f503c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5040a09700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503c0cde70 0x7f503c159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5040a09700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c15cbd0 con 0x7f503c06bb90 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5040a09700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503c13a6b0 con 0x7f503c0cde70 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5040a09700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f503c1f4300 con 0x7f503c0adb30 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c13c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c13c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43088/0 (socket says 172.21.15.119:43088) 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 -- 172.21.15.119:0/3653628119 learned_addr learned my addr 172.21.15.119:0/3653628119 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:44.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503affd700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f503c0adb30 0x7f503c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503bfff700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503c0cde70 0x7f503c159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 -- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f503c0adb30 msgr2=0x7f503c15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f503c0adb30 0x7f503c15c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 -- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503c0cde70 msgr2=0x7f503c159df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503c0cde70 0x7f503c159df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 -- 172.21.15.119:0/3653628119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5030009570 con 0x7f503c06bb90 2024-09-13T16:25:44.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f503b7fe700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c13c8e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5034009db0 tx=0x7f503400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5023fff700 1 -- 172.21.15.119:0/3653628119 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034009c20 con 0x7f503c06bb90 2024-09-13T16:25:44.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5023fff700 1 -- 172.21.15.119:0/3653628119 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503400f040 con 0x7f503c06bb90 2024-09-13T16:25:44.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503c1f39d0 con 0x7f503c06bb90 2024-09-13T16:25:44.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5023fff700 1 -- 172.21.15.119:0/3653628119 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5034014480 con 0x7f503c06bb90 2024-09-13T16:25:44.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503c1f3f90 con 0x7f503c06bb90 2024-09-13T16:25:44.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.117+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503c0aef10 con 0x7f503c06bb90 2024-09-13T16:25:44.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.121+0000 7f5023fff700 1 -- 172.21.15.119:0/3653628119 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f50340145e0 con 0x7f503c06bb90 2024-09-13T16:25:44.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.121+0000 7f5023fff700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5028066730 0x7f5028068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.121+0000 7f5023fff700 1 -- 172.21.15.119:0/3653628119 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5034084ca0 con 0x7f503c06bb90 2024-09-13T16:25:44.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.121+0000 7f503affd700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5028066730 0x7f5028068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.121+0000 7f503affd700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5028066730 0x7f5028068bf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5030014040 tx=0x7f5030008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.129+0000 7f5023fff700 1 -- 172.21.15.119:0/3653628119 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5034051d20 con 0x7f503c06bb90 2024-09-13T16:25:44.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.333+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f503c06d3f0 con 0x7f503c06bb90 2024-09-13T16:25:44.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.333+0000 7f5023fff700 1 -- 172.21.15.119:0/3653628119 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f5034055340 con 0x7f503c06bb90 2024-09-13T16:25:44.339 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:44.339 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:25:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5028066730 msgr2=0x7f5028068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5028066730 0x7f5028068bf0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f5030014040 tx=0x7f5030008030 comp rx=0 tx=0).stop 2024-09-13T16:25:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 msgr2=0x7f503c13c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c13c8e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5034009db0 tx=0x7f503400c3a0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 shutdown_connections 2024-09-13T16:25:44.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5028066730 0x7f5028068bf0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f503c0cde70 0x7f503c159df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f503c0adb30 0x7f503c15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 --2- 172.21.15.119:0/3653628119 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f503c06bb90 0x7f503c13c8e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 >> 172.21.15.119:0/3653628119 conn(0x7f503c06a350 msgr2=0x7f503c0b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 shutdown_connections 2024-09-13T16:25:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.337+0000 7f5040a09700 1 -- 172.21.15.119:0/3653628119 wait complete. 2024-09-13T16:25:44.342 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:25:44.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:44.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.457+0000 7f9fe359e700 1 Processor -- start 2024-09-13T16:25:44.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.457+0000 7f9fe359e700 1 -- start start 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158b30 0x7f9fe4071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9fe4158f10 0x7f9fe4071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9fe406bb90 0x7f9fe406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe406de00 con 0x7f9fe406bb90 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fe406db20 con 0x7f9fe4158f10 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe40b1920 con 0x7f9fe4158b30 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9fe406bb90 0x7f9fe406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9fe406bb90 0x7f9fe406ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43104/0 (socket says 172.21.15.119:43104) 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 -- 172.21.15.119:0/2089657694 learned_addr learned my addr 172.21.15.119:0/2089657694 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe259c700 1 --2- 172.21.15.119:0/2089657694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158b30 0x7f9fe4071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe1d9b700 1 --2- 172.21.15.119:0/2089657694 >> v2:172.21.15.145:3300/0 conn(0x7f9fe4158f10 0x7f9fe4071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 -- 172.21.15.119:0/2089657694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158b30 msgr2=0x7f9fe4071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 --2- 172.21.15.119:0/2089657694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158b30 0x7f9fe4071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 -- 172.21.15.119:0/2089657694 >> v2:172.21.15.145:3300/0 conn(0x7f9fe4158f10 msgr2=0x7f9fe4071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 --2- 172.21.15.119:0/2089657694 >> v2:172.21.15.145:3300/0 conn(0x7f9fe4158f10 0x7f9fe4071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 -- 172.21.15.119:0/2089657694 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe40b1c00 con 0x7f9fe406bb90 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe2d9d700 1 --2- 172.21.15.119:0/2089657694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe406bb90 0x7f9fe406ce30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9fe406fca0 tx=0x7f9fdc00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4caa247c39db5514 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe159a700 1 -- 172.21.15.119:0/2089657694 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc010070 con 0x7f9fe406bb90 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe159a700 1 -- 172.21.15.119:0/2089657694 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc00dce0 con 0x7f9fe406bb90 2024-09-13T16:25:44.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe159a700 1 -- 172.21.15.119:0/2089657694 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00b310 con 0x7f9fe406bb90 2024-09-13T16:25:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- 172.21.15.119:0/2089657694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe406bb90 msgr2=0x7f9fe406ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 --2- 172.21.15.119:0/2089657694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe406bb90 0x7f9fe406ce30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f9fe406fca0 tx=0x7f9fdc00d430 comp rx=0 tx=0).stop 2024-09-13T16:25:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- 172.21.15.119:0/2089657694 shutdown_connections 2024-09-13T16:25:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 --2- 172.21.15.119:0/2089657694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe406bb90 0x7f9fe406ce30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 --2- 172.21.15.119:0/2089657694 >> v2:172.21.15.145:3300/0 conn(0x7f9fe4158f10 0x7f9fe4071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 --2- 172.21.15.119:0/2089657694 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158b30 0x7f9fe4071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- 172.21.15.119:0/2089657694 >> 172.21.15.119:0/2089657694 conn(0x7f9fe406a350 msgr2=0x7f9fe406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- 172.21.15.119:0/2089657694 shutdown_connections 2024-09-13T16:25:44.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.461+0000 7f9fe359e700 1 -- 172.21.15.119:0/2089657694 wait complete. 2024-09-13T16:25:44.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 Processor -- start 2024-09-13T16:25:44.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 -- start start 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fe406bb90 0x7f9fe415bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4158b30 0x7f9fe41420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158f10 0x7f9fe40ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe415c320 con 0x7f9fe4158b30 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe4142c00 con 0x7f9fe406bb90 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fe415b950 con 0x7f9fe4158f10 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.465+0000 7f9fe1d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4158b30 0x7f9fe41420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4158b30 0x7f9fe41420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43118/0 (socket says 172.21.15.119:43118) 2024-09-13T16:25:44.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 -- 172.21.15.119:0/3904383665 learned_addr learned my addr 172.21.15.119:0/3904383665 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe259c700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fe406bb90 0x7f9fe415bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe2d9d700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158f10 0x7f9fe40ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 -- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158f10 msgr2=0x7f9fe40ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158f10 0x7f9fe40ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 -- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fe406bb90 msgr2=0x7f9fe415bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fe406bb90 0x7f9fe415bde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 -- 172.21.15.119:0/3904383665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fdc00bd40 con 0x7f9fe4158b30 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe1d9b700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4158b30 0x7f9fe41420d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009fd0 tx=0x7f9fd800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fd2ffd700 1 -- 172.21.15.119:0/3904383665 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd80095a0 con 0x7f9fe4158b30 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fd2ffd700 1 -- 172.21.15.119:0/3904383665 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fd8013680 con 0x7f9fe4158b30 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe4159d20 con 0x7f9fe4158b30 2024-09-13T16:25:44.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fd2ffd700 1 -- 172.21.15.119:0/3904383665 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fd8013a30 con 0x7f9fe4158b30 2024-09-13T16:25:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe413f3c0 con 0x7f9fe4158b30 2024-09-13T16:25:44.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe4073590 con 0x7f9fe4158b30 2024-09-13T16:25:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.469+0000 7f9fd2ffd700 1 -- 172.21.15.119:0/3904383665 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9fd8013b90 con 0x7f9fe4158b30 2024-09-13T16:25:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.473+0000 7f9fd2ffd700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fc4066730 0x7f9fc4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.473+0000 7f9fd2ffd700 1 -- 172.21.15.119:0/3904383665 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9fd80841a0 con 0x7f9fe4158b30 2024-09-13T16:25:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.473+0000 7f9fe259c700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fc4066730 0x7f9fc4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.473+0000 7f9fe259c700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fc4066730 0x7f9fc4068bf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009bf0 tx=0x7f9fd4009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.477+0000 7f9fd2ffd700 1 -- 172.21.15.119:0/3904383665 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9fd8051220 con 0x7f9fe4158b30 2024-09-13T16:25:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.649+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fe415bac0 con 0x7f9fe4158b30 2024-09-13T16:25:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.649+0000 7f9fd2ffd700 1 -- 172.21.15.119:0/3904383665 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f9fd8054840 con 0x7f9fe4158b30 2024-09-13T16:25:44.654 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:44.654 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fc4066730 msgr2=0x7f9fc4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fc4066730 0x7f9fc4068bf0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9fd4009bf0 tx=0x7f9fd4009240 comp rx=0 tx=0).stop 2024-09-13T16:25:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4158b30 msgr2=0x7f9fe41420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4158b30 0x7f9fe41420d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f9fd8009fd0 tx=0x7f9fd800eb20 comp rx=0 tx=0).stop 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 shutdown_connections 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fc4066730 0x7f9fc4068bf0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9fe4158f10 0x7f9fe40ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9fe4158b30 0x7f9fe41420d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 --2- 172.21.15.119:0/3904383665 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9fe406bb90 0x7f9fe415bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 >> 172.21.15.119:0/3904383665 conn(0x7f9fe406a350 msgr2=0x7f9fe40af030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 shutdown_connections 2024-09-13T16:25:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.653+0000 7f9fe359e700 1 -- 172.21.15.119:0/3904383665 wait complete. 2024-09-13T16:25:44.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:25:44.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 Processor -- start 2024-09-13T16:25:44.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 -- start start 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe5a015ec20 0x7fe5a0070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a0161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a0071210 con 0x7fe5a015e840 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a00714f0 con 0x7fe5a015ec20 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a00717d0 con 0x7fe5a0165180 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a4fb1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe5a015e840 0x7fe5a0164a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58622/0 (socket says 172.21.15.119:58622) 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a4fb1700 1 -- 172.21.15.119:0/1618370322 learned_addr learned my addr 172.21.15.119:0/1618370322 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a57b2700 1 --2- 172.21.15.119:0/1618370322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a0161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe59ffff700 1 --2- 172.21.15.119:0/1618370322 >> v2:172.21.15.145:3300/0 conn(0x7fe5a015ec20 0x7fe5a0070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a57b2700 1 -- 172.21.15.119:0/1618370322 >> v2:172.21.15.145:3300/0 conn(0x7fe5a015ec20 msgr2=0x7fe5a0070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:44.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a57b2700 1 --2- 172.21.15.119:0/1618370322 >> v2:172.21.15.145:3300/0 conn(0x7fe5a015ec20 0x7fe5a0070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a57b2700 1 -- 172.21.15.119:0/1618370322 >> v1:172.21.15.119:6789/0 conn(0x7fe5a015e840 legacy=0x7fe5a0164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a57b2700 1 -- 172.21.15.119:0/1618370322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a0071ab0 con 0x7fe5a0165180 2024-09-13T16:25:44.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe59f7fe700 1 -- 172.21.15.119:0/1618370322 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616660864 0 0) 0x7fe5a0071210 con 0x7fe5a015e840 2024-09-13T16:25:44.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a57b2700 1 --2- 172.21.15.119:0/1618370322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a0161f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe590009850 tx=0x7fe590009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd87b88cdcaee312 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe59f7fe700 1 -- 172.21.15.119:0/1618370322 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590011830 con 0x7fe5a0165180 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe59f7fe700 1 -- 172.21.15.119:0/1618370322 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe590011990 con 0x7fe5a0165180 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe59f7fe700 1 -- 172.21.15.119:0/1618370322 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe590011d40 con 0x7fe5a0165180 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 -- 172.21.15.119:0/1618370322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 msgr2=0x7fe5a0161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 --2- 172.21.15.119:0/1618370322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a0161f00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe590009850 tx=0x7fe590009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 -- 172.21.15.119:0/1618370322 shutdown_connections 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 --2- 172.21.15.119:0/1618370322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a0161f00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 --2- 172.21.15.119:0/1618370322 >> v2:172.21.15.145:3300/0 conn(0x7fe5a015ec20 0x7fe5a0070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.781+0000 7fe5a7215700 1 -- 172.21.15.119:0/1618370322 >> 172.21.15.119:0/1618370322 conn(0x7fe5a006a350 msgr2=0x7fe5a006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 -- 172.21.15.119:0/1618370322 shutdown_connections 2024-09-13T16:25:44.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 -- 172.21.15.119:0/1618370322 wait complete. 2024-09-13T16:25:44.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 Processor -- start 2024-09-13T16:25:44.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 -- start start 2024-09-13T16:25:44.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a015e840 0x7fe5a0160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a01f5ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a014f390 0x7fe5a01f9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a014ca30 con 0x7fe5a015e840 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a014cd10 con 0x7fe5a014f390 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a7215700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a014cff0 con 0x7fe5a0165180 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a015e840 0x7fe5a0160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a015e840 0x7fe5a0160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43120/0 (socket says 172.21.15.119:43120) 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 -- 172.21.15.119:0/2832259719 learned_addr learned my addr 172.21.15.119:0/2832259719 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe59ffff700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a01f5ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a57b2700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a014f390 0x7fe5a01f9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 -- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 msgr2=0x7fe5a01f5ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a01f5ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 -- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a014f390 msgr2=0x7fe5a01f9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a014f390 0x7fe5a01f9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.785+0000 7fe5a4fb1700 1 -- 172.21.15.119:0/2832259719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5900094b0 con 0x7fe5a015e840 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe5a4fb1700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a015e840 0x7fe5a0160300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe59400c580 tx=0x7fe59400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe59d7fa700 1 -- 172.21.15.119:0/2832259719 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594004ca0 con 0x7fe5a015e840 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe59d7fa700 1 -- 172.21.15.119:0/2832259719 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe594004e20 con 0x7fe5a015e840 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe59d7fa700 1 -- 172.21.15.119:0/2832259719 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe594007310 con 0x7fe5a015e840 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a014d2d0 con 0x7fe5a015e840 2024-09-13T16:25:44.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a014c6e0 con 0x7fe5a015e840 2024-09-13T16:25:44.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a0067760 con 0x7fe5a015e840 2024-09-13T16:25:44.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe59d7fa700 1 -- 172.21.15.119:0/2832259719 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe594007470 con 0x7fe5a015e840 2024-09-13T16:25:44.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe59d7fa700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066730 0x7fe58c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:44.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe59d7fa700 1 -- 172.21.15.119:0/2832259719 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe594084080 con 0x7fe5a015e840 2024-09-13T16:25:44.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.789+0000 7fe59ffff700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066730 0x7fe58c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:44.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.793+0000 7fe59ffff700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066730 0x7fe58c068bf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00cdf00 tx=0x7fe59800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:44.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:44.801+0000 7fe59d7fa700 1 -- 172.21.15.119:0/2832259719 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe594051100 con 0x7fe5a015e840 2024-09-13T16:25:45.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.009+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5a0071210 con 0x7fe5a015e840 2024-09-13T16:25:45.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.009+0000 7fe59d7fa700 1 -- 172.21.15.119:0/2832259719 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7fe594007720 con 0x7fe5a015e840 2024-09-13T16:25:45.014 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:45.014 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:25:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066730 msgr2=0x7fe58c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066730 0x7fe58c068bf0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00cdf00 tx=0x7fe59800b040 comp rx=0 tx=0).stop 2024-09-13T16:25:45.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a015e840 msgr2=0x7fe5a0160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a015e840 0x7fe5a0160300 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fe59400c580 tx=0x7fe59400c890 comp rx=0 tx=0).stop 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 shutdown_connections 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066730 0x7fe58c068bf0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a014f390 0x7fe5a01f9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a0165180 0x7fe5a01f5ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 --2- 172.21.15.119:0/2832259719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a015e840 0x7fe5a0160300 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 >> 172.21.15.119:0/2832259719 conn(0x7fe5a006a350 msgr2=0x7fe5a0163000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:45.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 shutdown_connections 2024-09-13T16:25:45.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.013+0000 7fe5a7215700 1 -- 172.21.15.119:0/2832259719 wait complete. 2024-09-13T16:25:45.019 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:25:45.431 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 9, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:25:16.907443+0000', 'modified': '2024-09-13T16:25:33.005194+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4285': {'gid': 4285, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6835/3556289056', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 3556289056}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 3556289056}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:25:45.432 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:25:45.432 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:25:45.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.553+0000 7ff713fcd700 1 Processor -- start 2024-09-13T16:25:45.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.553+0000 7ff713fcd700 1 -- start start 2024-09-13T16:25:45.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff70c0d37a0 0x7ff70c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70c0d1190 con 0x7ff70c15ce70 2024-09-13T16:25:45.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff70c0ceaf0 con 0x7ff70c1607c0 2024-09-13T16:25:45.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff70c0d3bc0 con 0x7ff70c0d37a0 2024-09-13T16:25:45.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff711d69700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff70c15ce70 0x7ff70c0d1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58624/0 (socket says 172.21.15.119:58624) 2024-09-13T16:25:45.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff711d69700 1 -- 172.21.15.119:0/2092960772 learned_addr learned my addr 172.21.15.119:0/2092960772 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:45.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff711568700 1 --2- 172.21.15.119:0/2092960772 >> v2:172.21.15.145:3300/0 conn(0x7ff70c0d37a0 0x7ff70c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff710d67700 1 -- 172.21.15.119:0/2092960772 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3448925428 0 0) 0x7ff70c0d1190 con 0x7ff70c15ce70 2024-09-13T16:25:45.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff710d67700 1 -- 172.21.15.119:0/2092960772 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f80035d0 con 0x7ff70c15ce70 2024-09-13T16:25:45.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff711568700 1 -- 172.21.15.119:0/2092960772 >> v1:172.21.15.145:6789/0 conn(0x7ff70c1607c0 legacy=0x7ff70c0b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff711568700 1 -- 172.21.15.119:0/2092960772 >> v1:172.21.15.119:6789/0 conn(0x7ff70c15ce70 legacy=0x7ff70c0d1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff711568700 1 -- 172.21.15.119:0/2092960772 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c0d0860 con 0x7ff70c0d37a0 2024-09-13T16:25:45.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff711568700 1 --2- 172.21.15.119:0/2092960772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0ce2a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff708009d90 tx=0x7ff70800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eeff6a14c921305c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:45.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff710d67700 1 -- 172.21.15.119:0/2092960772 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70800e070 con 0x7ff70c0d37a0 2024-09-13T16:25:45.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff710d67700 1 -- 172.21.15.119:0/2092960772 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff70800bb20 con 0x7ff70c0d37a0 2024-09-13T16:25:45.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff710d67700 1 -- 172.21.15.119:0/2092960772 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff70800be90 con 0x7ff70c0d37a0 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- 172.21.15.119:0/2092960772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c0d37a0 msgr2=0x7ff70c0ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 --2- 172.21.15.119:0/2092960772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0ce2a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff708009d90 tx=0x7ff70800b330 comp rx=0 tx=0).stop 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- 172.21.15.119:0/2092960772 shutdown_connections 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 --2- 172.21.15.119:0/2092960772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0ce2a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- 172.21.15.119:0/2092960772 >> 172.21.15.119:0/2092960772 conn(0x7ff70c0d2bf0 msgr2=0x7ff70c0d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- 172.21.15.119:0/2092960772 shutdown_connections 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.557+0000 7ff713fcd700 1 -- 172.21.15.119:0/2092960772 wait complete. 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 Processor -- start 2024-09-13T16:25:45.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 -- start start 2024-09-13T16:25:45.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff70c15ce70 0x7ff70c1547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c1607c0 0x7ff70c154ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c155220 con 0x7ff70c0d37a0 2024-09-13T16:25:45.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0c6720 con 0x7ff70c1607c0 2024-09-13T16:25:45.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff713fcd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff70c0c6970 con 0x7ff70c15ce70 2024-09-13T16:25:45.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff711d69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff711d69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0b5a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43134/0 (socket says 172.21.15.119:43134) 2024-09-13T16:25:45.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff711d69700 1 -- 172.21.15.119:0/3866047033 learned_addr learned my addr 172.21.15.119:0/3866047033 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:45.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff711568700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff70c15ce70 0x7ff70c1547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff71256a700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c1607c0 0x7ff70c154ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.561+0000 7ff711d69700 1 -- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff70c15ce70 msgr2=0x7ff70c1547a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff711d69700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff70c15ce70 0x7ff70c1547a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff711d69700 1 -- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c1607c0 msgr2=0x7ff70c154ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff711d69700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c1607c0 0x7ff70c154ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff711d69700 1 -- 172.21.15.119:0/3866047033 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff708009a70 con 0x7ff70c0d37a0 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff711d69700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0b5a40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c700 tx=0x7ff6fc00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff7027fc700 1 -- 172.21.15.119:0/3866047033 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc005d90 con 0x7ff70c0d37a0 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff70c0c6c50 con 0x7ff70c0d37a0 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff7027fc700 1 -- 172.21.15.119:0/3866047033 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc0070d0 con 0x7ff70c0d37a0 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff7027fc700 1 -- 172.21.15.119:0/3866047033 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc007480 con 0x7ff70c0d37a0 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff70c0c7210 con 0x7ff70c0d37a0 2024-09-13T16:25:45.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff7027fc700 1 -- 172.21.15.119:0/3866047033 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff6fc0075e0 con 0x7ff70c0d37a0 2024-09-13T16:25:45.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff7027fc700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6f0066730 0x7ff6f0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff7027fc700 1 -- 172.21.15.119:0/3866047033 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff6fc083930 con 0x7ff70c0d37a0 2024-09-13T16:25:45.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.565+0000 7ff711568700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6f0066730 0x7ff6f0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff711568700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6f0066730 0x7ff6f0068bf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff708000f80 tx=0x7ff708011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:45.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 Processor -- start 2024-09-13T16:25:45.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 -- start start 2024-09-13T16:25:45.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x1058180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1020890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1020dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xf7fd80 con 0x1036d00 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xf7d510 con 0x1057da0 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.569+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xf78e00 con 0x10586c0 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1020890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71256a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1020dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711d69700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x1058180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1020890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43146/0 (socket says 172.21.15.119:43146) 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 -- 172.21.15.119:0/2506927198 learned_addr learned my addr 172.21.15.119:0/2506927198 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:45.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 -- 172.21.15.119:0/2506927198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 msgr2=0x1020dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 --2- 172.21.15.119:0/2506927198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1020dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 -- 172.21.15.119:0/2506927198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 msgr2=0x1058180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 --2- 172.21.15.119:0/2506927198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x1058180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 -- 172.21.15.119:0/2506927198 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xdeb960 con 0x1036d00 2024-09-13T16:25:45.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff711568700 1 --2- 172.21.15.119:0/2506927198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1020890 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b370 tx=0x7ff708010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71d4ced94d5ed461 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:45.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff6ef7fe700 1 -- 172.21.15.119:0/2506927198 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708012630 con 0x1036d00 2024-09-13T16:25:45.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff6ef7fe700 1 -- 172.21.15.119:0/2506927198 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff708012790 con 0x1036d00 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff6ef7fe700 1 -- 172.21.15.119:0/2506927198 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708012b00 con 0x1036d00 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 -- 172.21.15.119:0/2506927198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 msgr2=0x1020890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 --2- 172.21.15.119:0/2506927198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1020890 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b370 tx=0x7ff708010000 comp rx=0 tx=0).stop 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 -- 172.21.15.119:0/2506927198 shutdown_connections 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 --2- 172.21.15.119:0/2506927198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1020dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 --2- 172.21.15.119:0/2506927198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1020890 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 --2- 172.21.15.119:0/2506927198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x1058180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 -- 172.21.15.119:0/2506927198 >> 172.21.15.119:0/2506927198 conn(0xda04d0 msgr2=0x1039c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 -- 172.21.15.119:0/2506927198 shutdown_connections 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 -- 172.21.15.119:0/2506927198 wait complete. 2024-09-13T16:25:45.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 -- start start 2024-09-13T16:25:45.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.573+0000 7ff71727e740 1 Processor -- start 2024-09-13T16:25:45.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1013180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x100fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1010d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xf36730 con 0x1036d00 2024-09-13T16:25:45.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xf3c5b0 con 0x1057da0 2024-09-13T16:25:45.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xe24aa0 con 0x10586c0 2024-09-13T16:25:45.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 10 client.0 _ll_register_callbacks cb 0xf72610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1013180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1013180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43162/0 (socket says 172.21.15.119:43162) 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 -- 172.21.15.119:0/1879143989 learned_addr learned my addr 172.21.15.119:0/1879143989 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71256a700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1010d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711568700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x100fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 -- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 msgr2=0x1010d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1010d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 -- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 msgr2=0x100fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x100fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 -- 172.21.15.119:0/1879143989 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff708023070 con 0x1036d00 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711568700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x100fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:45.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711d69700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1013180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc0011e0 tx=0x7ff6fc001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30d55539487297d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff6ecff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1879143989 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc058420 con 0x1036d00 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1879143989 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff6fc05c9e0 con 0x1036d00 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xdad1e0 con 0x1036d00 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1879143989 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff6fc05c550 con 0x1036d00 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 10 client.4905 fetch_fsmap learned FSMap version 9 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0xf3f220 con 0x1036d00 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1879143989 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xdad1e0 con 0x1036d00 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1879143989 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7ff6fc05a3b0 con 0x1036d00 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 10 client.4905 fetch_fsmap finished waiting for FSMap version 9 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 10 client.4905 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 20 client.4905 populate_metadata read hostname 'smithi119' 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1032ad0 0x1032eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 4 client.4905 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:25:45.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 --> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xdf7460 con 0x1032ad0 2024-09-13T16:25:45.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711568700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1032ad0 0x1032eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff711568700 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1032ad0 0x1032eb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ada61b933222f0cd server_cookie=b652fc172affad81 in_seq=0 out_seq=0 2024-09-13T16:25:45.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.577+0000 7ff6ecff9700 10 client.4905 ms_handle_connect on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:45.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1879143989 <== mds.0 v2:172.21.15.119:6834/3556289056 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0xdf7460 con 0x1032ad0 2024-09-13T16:25:45.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff6ecff9700 10 client.4905 handle_command_reply: tid=0 2024-09-13T16:25:45.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 client.4905 shutdown 2024-09-13T16:25:45.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 msgr2=0x1013180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1013180 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc0011e0 tx=0x7ff6fc001150 comp rx=0 tx=0).stop 2024-09-13T16:25:45.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 shutdown_connections 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1032ad0 0x1032eb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10586c0 0x1010d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff6ecff9700 0 client.4905 ms_handle_reset on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x100fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 --2- 172.21.15.119:0/1879143989 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1036d00 0x1013180 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 >> 172.21.15.119:0/1879143989 conn(0xda04d0 msgr2=0x1033680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 shutdown_connections 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 1 -- 172.21.15.119:0/1879143989 wait complete. 2024-09-13T16:25:45.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.581+0000 7ff71727e740 20 client.4905 trim_cache size 0 max 16384 2024-09-13T16:25:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 Processor -- start 2024-09-13T16:25:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 -- start start 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x1055ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1057da0 0x1056530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10586c0 0x1056a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x10115d0 con 0x1057da0 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xfa7ec0 con 0x10586c0 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xfa6d90 con 0x1036d00 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1057da0 0x1056530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1057da0 0x1056530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43174/0 (socket says 172.21.15.119:43174) 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 -- 172.21.15.119:0/4134174775 learned_addr learned my addr 172.21.15.119:0/4134174775 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711d69700 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x1055ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff71256a700 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10586c0 0x1056a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 -- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 msgr2=0x1055ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x1055ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 -- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10586c0 msgr2=0x1056a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10586c0 0x1056a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 -- 172.21.15.119:0/4134174775 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff70c1e8800 con 0x1057da0 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff711568700 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1057da0 0x1056530 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff708009fd0 tx=0x7ff70801f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd3c69153d79f9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff6effff700 1 -- 172.21.15.119:0/4134174775 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708012900 con 0x1057da0 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff6effff700 1 -- 172.21.15.119:0/4134174775 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff708012a60 con 0x1057da0 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.601+0000 7ff6effff700 1 -- 172.21.15.119:0/4134174775 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708012dd0 con 0x1057da0 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- 172.21.15.119:0/4134174775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1057da0 msgr2=0x1056530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1057da0 0x1056530 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff708009fd0 tx=0x7ff70801f040 comp rx=0 tx=0).stop 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- 172.21.15.119:0/4134174775 shutdown_connections 2024-09-13T16:25:45.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10586c0 0x1056a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1057da0 0x1056530 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 --2- 172.21.15.119:0/4134174775 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x1055ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- 172.21.15.119:0/4134174775 >> 172.21.15.119:0/4134174775 conn(0xda04d0 msgr2=0x1066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:45.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- 172.21.15.119:0/4134174775 shutdown_connections 2024-09-13T16:25:45.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- 172.21.15.119:0/4134174775 wait complete. 2024-09-13T16:25:45.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- start start 2024-09-13T16:25:45.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 Processor -- start 2024-09-13T16:25:45.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x10c0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x105d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10586c0 0x105d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x107f450 con 0x10586c0 2024-09-13T16:25:45.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x105e8c0 con 0x1057da0 2024-09-13T16:25:45.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x105e160 con 0x1036d00 2024-09-13T16:25:45.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 10 client.0 _ll_register_callbacks cb 0xfcf710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71256a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10586c0 0x105d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71256a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10586c0 0x105d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43182/0 (socket says 172.21.15.119:43182) 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711d69700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x10c0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711568700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x105d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71256a700 1 -- 172.21.15.119:0/1160444686 learned_addr learned my addr 172.21.15.119:0/1160444686 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711568700 1 -- 172.21.15.119:0/1160444686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 msgr2=0x10c0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711568700 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x10c0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711568700 1 -- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10586c0 msgr2=0x105d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711568700 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10586c0 0x105d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711568700 1 -- 172.21.15.119:0/1160444686 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0xdeb960 con 0x1057da0 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71256a700 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10586c0 0x105d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff711568700 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x105d1d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b370 tx=0x7ff708004b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bf3679dc2623c68 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:45.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff6ecff9700 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1160444686 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff708012900 con 0x1057da0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1160444686 <== mon.1 v2:172.21.15.145:3300/0 2 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff7080124b0 con 0x1057da0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff708012790 con 0x1057da0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1160444686 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=9) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff708012bd0 con 0x1057da0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.605+0000 7ff71727e740 10 client.4828 fetch_fsmap learned FSMap version 9 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=9,monmap=2+}) v3 -- 0xf25450 con 0x1057da0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1160444686 <== mon.1 v2:172.21.15.145:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff708012790 con 0x1057da0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1160444686 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 9) v1 ==== 3122+0+0 (secure 0 0 0) 0x7ff708045070 con 0x1057da0 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 10 client.4828 fetch_fsmap finished waiting for FSMap version 9 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 10 client.4828 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 20 client.4828 populate_metadata read hostname 'smithi119' 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1068030 0x10c1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:45.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 4 client.4828 mds_command: new command op to 4285 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 --> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] -- command(tid 0: {"prefix": "status"}) v1 -- 0xe1c1e0 con 0x1068030 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff711d69700 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1068030 0x10c1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff6ecff9700 10 client.4828 ms_handle_connect on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff711d69700 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1068030 0x10c1fd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=800e2c1844ab55d5 server_cookie=1f268b3d3a6519e2 in_seq=0 out_seq=0 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff6ecff9700 1 -- 172.21.15.119:0/1160444686 <== mds.0 v2:172.21.15.119:6834/3556289056 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0xe1c1e0 con 0x1068030 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff6ecff9700 10 client.4828 handle_command_reply: tid=0 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 client.4828 shutdown 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 msgr2=0x105d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x105d1d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff70800b370 tx=0x7ff708004b60 comp rx=0 tx=0).stop 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 shutdown_connections 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:6834/3556289056,v1:172.21.15.119:6835/3556289056] conn(0x1068030 0x10c1fd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10586c0 0x105d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1057da0 0x105d1d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 --2- 172.21.15.119:0/1160444686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1036d00 0x10c0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 >> 172.21.15.119:0/1160444686 conn(0xda04d0 msgr2=0x10356e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff6ecff9700 0 client.4828 ms_handle_reset on v2:172.21.15.119:6834/3556289056 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 shutdown_connections 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 1 -- 172.21.15.119:0/1160444686 wait complete. 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff71727e740 20 client.4828 trim_cache size 0 max 16384 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:25:45.618 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 4285, 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 28.700780228999999, 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 9, 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 25, 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 25, 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 34.982803597999997 2024-09-13T16:25:45.619 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:25:45.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6f0066730 msgr2=0x7ff6f0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6f0066730 0x7ff6f0068bf0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff708000f80 tx=0x7ff708011000 comp rx=0 tx=0).stop 2024-09-13T16:25:45.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff70c0d37a0 msgr2=0x7ff70c0b5a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:45.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0b5a40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc00c700 tx=0x7ff6fc00ca10 comp rx=0 tx=0).stop 2024-09-13T16:25:45.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 shutdown_connections 2024-09-13T16:25:45.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6f0066730 0x7ff6f0068bf0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff70c1607c0 0x7ff70c154ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff70c15ce70 0x7ff70c1547a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 --2- 172.21.15.119:0/3866047033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff70c0d37a0 0x7ff70c0b5a40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:45.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 >> 172.21.15.119:0/3866047033 conn(0x7ff70c0d2bf0 msgr2=0x7ff70c1e9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:45.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 shutdown_connections 2024-09-13T16:25:45.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:45.609+0000 7ff713fcd700 1 -- 172.21.15.119:0/3866047033 wait complete. 2024-09-13T16:25:45.628 INFO:teuthology.run_tasks:Running task sequential... 2024-09-13T16:25:45.639 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-09-13T16:25:45.639 INFO:teuthology.task.sequential:In sequential, running task print... 2024-09-13T16:25:45.639 INFO:teuthology.task.print:Enabling mgr modules 2024-09-13T16:25:45.639 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-09-13T16:25:45.667 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-09-13T16:25:45.667 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-09-13T16:25:45.667 INFO:tasks.ceph_fuse:client.0 config is {} 2024-09-13T16:25:45.667 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T16:25:45.667 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:25:45.667 INFO:tasks.ceph_fuse:client.2 config is {} 2024-09-13T16:25:45.667 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T16:25:45.667 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-13T16:25:45.668 INFO:tasks.ceph_fuse:client.1 config is {} 2024-09-13T16:25:45.668 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T16:25:45.668 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T16:25:45.668 INFO:tasks.ceph_fuse:client.3 config is {} 2024-09-13T16:25:45.668 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-09-13T16:25:45.668 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-13T16:25:45.668 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:45.668 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:25:45.676 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-09-13T16:25:45.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:45.676 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-09-13T16:25:45.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:45.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link delete ceph-brx 2024-09-13T16:25:45.865 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:45.865 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-09-13T16:25:45.873 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:45.873 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link delete ceph-brx 2024-09-13T16:25:45.890 INFO:teuthology.orchestra.run.smithi145.stderr:Cannot find device "ceph-brx" 2024-09-13T16:25:45.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:25:45.892 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-09-13T16:25:45.892 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-13T16:25:45.892 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:25:46.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff275bcf700 1 Processor -- start 2024-09-13T16:25:46.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff275bcf700 1 -- start start 2024-09-13T16:25:46.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff275bcf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff27015e840 0x7ff270070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff275bcf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff270161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff275bcf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff270071210 con 0x7ff27015ec20 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff275bcf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2700714f0 con 0x7ff27015e840 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff275bcf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700717d0 con 0x7ff270165180 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26effd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff27015ec20 0x7ff270164a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58632/0 (socket says 172.21.15.119:58632) 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26effd700 1 -- 172.21.15.119:0/3429473213 learned_addr learned my addr 172.21.15.119:0/3429473213 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26f7fe700 1 --2- 172.21.15.119:0/3429473213 >> v2:172.21.15.145:3300/0 conn(0x7ff27015e840 0x7ff270070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26ffff700 1 --2- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff270161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26e7fc700 1 -- 172.21.15.119:0/3429473213 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3367277098 0 0) 0x7ff270071210 con 0x7ff27015ec20 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26e7fc700 1 -- 172.21.15.119:0/3429473213 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff25c0035d0 con 0x7ff27015ec20 2024-09-13T16:25:46.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26f7fe700 1 -- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 msgr2=0x7ff270161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:46.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26f7fe700 1 --2- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff270161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26f7fe700 1 -- 172.21.15.119:0/3429473213 >> v1:172.21.15.119:6789/0 conn(0x7ff27015ec20 legacy=0x7ff270164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:46.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26f7fe700 1 -- 172.21.15.119:0/3429473213 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff270071ab0 con 0x7ff27015e840 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26f7fe700 1 --2- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270070b80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff258009850 tx=0x7ff258009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a86ce8be6a977c85 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26e7fc700 1 -- 172.21.15.119:0/3429473213 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258011830 con 0x7ff27015e840 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26e7fc700 1 -- 172.21.15.119:0/3429473213 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff258011990 con 0x7ff27015e840 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.009+0000 7ff26e7fc700 1 -- 172.21.15.119:0/3429473213 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258011d00 con 0x7ff27015e840 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 msgr2=0x7ff270070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 --2- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270070b80 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff258009850 tx=0x7ff258009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- 172.21.15.119:0/3429473213 shutdown_connections 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 --2- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff270161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 --2- 172.21.15.119:0/3429473213 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270070b80 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- 172.21.15.119:0/3429473213 >> 172.21.15.119:0/3429473213 conn(0x7ff27006a350 msgr2=0x7ff27006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:46.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- 172.21.15.119:0/3429473213 shutdown_connections 2024-09-13T16:25:46.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- 172.21.15.119:0/3429473213 wait complete. 2024-09-13T16:25:46.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 Processor -- start 2024-09-13T16:25:46.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- start start 2024-09-13T16:25:46.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270152910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:46.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff2700cbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:46.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2700aa610 0x7ff2700aaac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:46.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700a9060 con 0x7ff2700aa610 2024-09-13T16:25:46.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700ab000 con 0x7ff27015e840 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff275bcf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2700a84b0 con 0x7ff270165180 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2700aa610 0x7ff2700aaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26f7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270152910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2700aa610 0x7ff2700aaac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43194/0 (socket says 172.21.15.119:43194) 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26ffff700 1 -- 172.21.15.119:0/49946684 learned_addr learned my addr 172.21.15.119:0/49946684 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26effd700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff2700cbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26f7fe700 1 -- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 msgr2=0x7ff2700cbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26f7fe700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff2700cbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26f7fe700 1 -- 172.21.15.119:0/49946684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2700aa610 msgr2=0x7ff2700aaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.013+0000 7ff26f7fe700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2700aa610 0x7ff2700aaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff26f7fe700 1 -- 172.21.15.119:0/49946684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2580094b0 con 0x7ff27015e840 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff26ffff700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2700aa610 0x7ff2700aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:46.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff26f7fe700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270152910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff258016000 tx=0x7ff258004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:46.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff267fff700 1 -- 172.21.15.119:0/49946684 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2580044d0 con 0x7ff27015e840 2024-09-13T16:25:46.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff267fff700 1 -- 172.21.15.119:0/49946684 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff25801f070 con 0x7ff27015e840 2024-09-13T16:25:46.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2700a86c0 con 0x7ff27015e840 2024-09-13T16:25:46.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff267fff700 1 -- 172.21.15.119:0/49946684 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff258003900 con 0x7ff27015e840 2024-09-13T16:25:46.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2700a8c80 con 0x7ff27015e840 2024-09-13T16:25:46.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff270067760 con 0x7ff27015e840 2024-09-13T16:25:46.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff267fff700 1 -- 172.21.15.119:0/49946684 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff258003a60 con 0x7ff27015e840 2024-09-13T16:25:46.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff267fff700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff250066730 0x7ff250068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:46.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.017+0000 7ff267fff700 1 -- 172.21.15.119:0/49946684 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ff258021070 con 0x7ff27015e840 2024-09-13T16:25:46.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.021+0000 7ff26effd700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff250066730 0x7ff250068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:46.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.021+0000 7ff26effd700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff250066730 0x7ff250068bf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff270066b90 tx=0x7ff268009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:46.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.025+0000 7ff267fff700 1 -- 172.21.15.119:0/49946684 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff258053090 con 0x7ff27015e840 2024-09-13T16:25:46.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.233+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff270071210 con 0x7ff27015e840 2024-09-13T16:25:46.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.233+0000 7ff267fff700 1 -- 172.21.15.119:0/49946684 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7ff258015020 con 0x7ff27015e840 2024-09-13T16:25:46.237 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff250066730 msgr2=0x7ff250068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff250066730 0x7ff250068bf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff270066b90 tx=0x7ff268009040 comp rx=0 tx=0).stop 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 msgr2=0x7ff270152910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270152910 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff258016000 tx=0x7ff258004770 comp rx=0 tx=0).stop 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 shutdown_connections 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff250066730 0x7ff250068bf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2700aa610 0x7ff2700aaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff270165180 0x7ff2700cbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 --2- 172.21.15.119:0/49946684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff27015e840 0x7ff270152910 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:46.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 >> 172.21.15.119:0/49946684 conn(0x7ff27006a350 msgr2=0x7ff270162770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:46.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 shutdown_connections 2024-09-13T16:25:46.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:46.237+0000 7ff275bcf700 1 -- 172.21.15.119:0/49946684 wait complete. 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:25:46.252 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:25:46.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.253 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:25:46.258 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:46.258 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:46.258 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:46.258 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:46.258 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:46.258 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:46.258 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:46.259 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42595sec preferred_lft 42595sec 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:46.260 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T16:25:46.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.261 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:46.261 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:46.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link add name ceph-brx type bridge 2024-09-13T16:25:46.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip addr flush dev ceph-brx 2024-09-13T16:25:46.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set ceph-brx up 2024-09-13T16:25:46.261 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T16:25:46.261 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:46.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.306 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:25:46.362 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:25:46.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.364 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:25:46.373 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:25:46.373 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:25:46.373 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:25:46.374 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:25:46.374 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:25:46.374 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:25:46.374 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:25:46.374 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:25:46.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.374 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:46.374 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:46.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:25:46.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:25:46.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:25:46.375 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:46.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.416 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:25:46.421 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.421 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list-id 2024-09-13T16:25:46.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.470 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:46.471 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:46.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:25:46.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-09-13T16:25:46.471 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:46.503 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-09-13T16:25:46.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-09-13T16:25:46.504 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:46.558 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.558 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:46.558 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:46.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:25:46.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T16:25:46.558 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:46.586 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:25:46.586 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:46.586 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:25:46.593 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:25:46.594 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:46.594 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:25:46.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:46.613 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:46.613 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:46.623 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:46.624 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:46.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.625 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:46.634 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:25:46.634 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:25:46.681 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:46.698 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:46.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:46.708 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:46.709 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:46.709 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:46.710 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:46.766 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:25:46.761+0000 7f17a57c1640 -1 init, newargv = 0x55ddfe09a5c0 newargc=15 2024-09-13T16:25:46.767 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[34964]: starting ceph client 2024-09-13T16:25:46.775 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[34964]: starting fuse 2024-09-13T16:25:47.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:47.836 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:47.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:47.850 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:47.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:47.852 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:47.859 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:25:47.859 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:25:47.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:25:47.859 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:25:47.860 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:25:47.861 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:47.862 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:25:47.862 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:25:47.942 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.34964.asok 2024-09-13T16:25:48.129 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.34964.asok 2024-09-13T16:25:48.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.34964.asok status 2024-09-13T16:25:48.277 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:25:48.277 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:25:48.277 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:25:48.277 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:25:48.277 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:25:48.277 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "34964", 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 4926, 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:25:48.278 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 4926 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2136421824 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2136421824 2024-09-13T16:25:48.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.4926 192.168.0.1:0/2136421824", 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/2136421824", 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 9, 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 25, 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:25:48.280 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:25:48.281 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:25:48.287 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-13T16:25:48.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:25:48.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c9fb4700 1 Processor -- start 2024-09-13T16:25:48.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c9fb4700 1 -- start start 2024-09-13T16:25:48.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c9fb4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:48.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c9fb4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10c40b17f0 0x7f10c40cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:48.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c9fb4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10c40705d0 con 0x7f10c40b2200 2024-09-13T16:25:48.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c9fb4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10c406d0d0 con 0x7f10c40b17f0 2024-09-13T16:25:48.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c9fb4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c406fca0 con 0x7f10c40cd8d0 2024-09-13T16:25:48.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10c40b17f0 0x7f10c40cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10c40b17f0 0x7f10c40cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43208/0 (socket says 172.21.15.119:43208) 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 -- 172.21.15.119:0/3615544045 learned_addr learned my addr 172.21.15.119:0/3615544045 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c8fb2700 1 --2- 172.21.15.119:0/3615544045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 -- 172.21.15.119:0/3615544045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 msgr2=0x7f10c406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 --2- 172.21.15.119:0/3615544045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 -- 172.21.15.119:0/3615544045 >> v1:172.21.15.119:6789/0 conn(0x7f10c40b2200 legacy=0x7f10c4070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 -- 172.21.15.119:0/3615544045 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c406ff80 con 0x7f10c40b17f0 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c37fe700 1 -- 172.21.15.119:0/3615544045 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289669542 0 0) 0x7f10c40705d0 con 0x7f10c40b2200 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c3fff700 1 --2- 172.21.15.119:0/3615544045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40cb800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009e50 tx=0x7f10b400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1e814982fa38635 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:48.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c37fe700 1 -- 172.21.15.119:0/3615544045 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b400e070 con 0x7f10c40b17f0 2024-09-13T16:25:48.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.401+0000 7f10c37fe700 1 -- 172.21.15.119:0/3615544045 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10b400b910 con 0x7f10c40b17f0 2024-09-13T16:25:48.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c37fe700 1 -- 172.21.15.119:0/3615544045 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10b400bcc0 con 0x7f10c40b17f0 2024-09-13T16:25:48.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 -- 172.21.15.119:0/3615544045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 msgr2=0x7f10c40cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/3615544045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40cb800 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009e50 tx=0x7f10b400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 -- 172.21.15.119:0/3615544045 shutdown_connections 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/3615544045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40cb800 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/3615544045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 -- 172.21.15.119:0/3615544045 >> 172.21.15.119:0/3615544045 conn(0x7f10c406a350 msgr2=0x7f10c406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 -- 172.21.15.119:0/3615544045 shutdown_connections 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 -- 172.21.15.119:0/3615544045 wait complete. 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 Processor -- start 2024-09-13T16:25:48.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.405+0000 7f10c9fb4700 1 -- start start 2024-09-13T16:25:48.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40aa4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:48.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10c40b2200 0x7f10c415c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c40ac8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c40ab370 con 0x7f10c40b17f0 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10c40aaad0 con 0x7f10c40b2200 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10c40aadb0 con 0x7f10c40cd8d0 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40aa4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40aa4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43218/0 (socket says 172.21.15.119:43218) 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 -- 172.21.15.119:0/1596270688 learned_addr learned my addr 172.21.15.119:0/1596270688 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c97b3700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c40ac8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c3fff700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10c40b2200 0x7f10c415c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 -- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 msgr2=0x7f10c40ac8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:48.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c40ac8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 -- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10c40b2200 msgr2=0x7f10c415c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10c40b2200 0x7f10c415c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 -- 172.21.15.119:0/1596270688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10b4009b30 con 0x7f10c40b17f0 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c8fb2700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40aa4f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00e840 tx=0x7f10ac00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c17fa700 1 -- 172.21.15.119:0/1596270688 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac0095a0 con 0x7f10c40b17f0 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c17fa700 1 -- 172.21.15.119:0/1596270688 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10ac013680 con 0x7f10c40b17f0 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c17fa700 1 -- 172.21.15.119:0/1596270688 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10ac0139f0 con 0x7f10c40b17f0 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10c40ab090 con 0x7f10c40b17f0 2024-09-13T16:25:48.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.409+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10c40b0770 con 0x7f10c40b17f0 2024-09-13T16:25:48.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.413+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10c40cdcb0 con 0x7f10c40b17f0 2024-09-13T16:25:48.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.413+0000 7f10c17fa700 1 -- 172.21.15.119:0/1596270688 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f10ac013b50 con 0x7f10c40b17f0 2024-09-13T16:25:48.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.413+0000 7f10c17fa700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f10a4066730 0x7f10a4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:48.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.413+0000 7f10c17fa700 1 -- 172.21.15.119:0/1596270688 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f10ac084290 con 0x7f10c40b17f0 2024-09-13T16:25:48.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.413+0000 7f10c3fff700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f10a4066730 0x7f10a4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:48.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.413+0000 7f10c3fff700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f10a4066730 0x7f10a4068bf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009e50 tx=0x7f10b4009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:48.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.417+0000 7f10c17fa700 1 -- 172.21.15.119:0/1596270688 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f10ac051310 con 0x7f10c40b17f0 2024-09-13T16:25:48.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.625+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f10c40cad30 con 0x7f10c40b17f0 2024-09-13T16:25:48.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.625+0000 7f10c17fa700 1 -- 172.21.15.119:0/1596270688 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f10ac054930 con 0x7f10c40b17f0 2024-09-13T16:25:48.631 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-13T16:25:48.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f10a4066730 msgr2=0x7f10a4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:48.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f10a4066730 0x7f10a4068bf0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f10b4009e50 tx=0x7f10b4009000 comp rx=0 tx=0).stop 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 msgr2=0x7f10c40aa4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40aa4f0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f10ac00e840 tx=0x7f10ac00eb50 comp rx=0 tx=0).stop 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 shutdown_connections 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f10a4066730 0x7f10a4068bf0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10c40cd8d0 0x7f10c40ac8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10c40b2200 0x7f10c415c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 --2- 172.21.15.119:0/1596270688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10c40b17f0 0x7f10c40aa4f0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 >> 172.21.15.119:0/1596270688 conn(0x7f10c406a350 msgr2=0x7f10c40b2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 shutdown_connections 2024-09-13T16:25:48.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:48.629+0000 7f10c9fb4700 1 -- 172.21.15.119:0/1596270688 wait complete. 2024-09-13T16:25:48.645 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:25:48.645 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:25:48.645 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:25:48.646 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-09-13T16:25:48.646 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-09-13T16:25:48.646 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:25:48.646 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:25:48.646 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:25:48.646 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-09-13T16:25:48.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.646 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:25:48.651 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.652 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:48.652 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:48.652 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42592sec preferred_lft 42592sec 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.653 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:25:48.654 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.655 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:25:48.708 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:25:48.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.710 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:25:48.719 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:25:48.719 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:25:48.719 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:25:48.719 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:25:48.720 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:25:48.720 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:25:48.720 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:25:48.720 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:25:48.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.720 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:48.720 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:48.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:25:48.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:25:48.721 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:25:48.721 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:48.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.761 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:25:48.810 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:25:48.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.811 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list-id 2024-09-13T16:25:48.818 INFO:teuthology.orchestra.run.smithi119.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-09-13T16:25:48.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.818 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:48.819 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:48.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:25:48.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-09-13T16:25:48.819 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:48.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-13T16:25:48.908 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:48.908 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:48.908 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:48.908 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 32:27:a0:12:ab:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::3027:a0ff:fe12:abf0/64 scope link 2024-09-13T16:25:48.909 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-09-13T16:25:48.925 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:48.925 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout:2: veth0@if10: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 32:27:a0:12:ab:f0 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T16:25:48.926 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.927 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::3027:a0ff:fe12:abf0/64 scope link 2024-09-13T16:25:48.927 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:48.928 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-09-13T16:25:48.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-09-13T16:25:48.929 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:48.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:48.987 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:25:48.987 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:25:48.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.1 up 2024-09-13T16:25:48.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set dev brx.1 master ceph-brx 2024-09-13T16:25:48.987 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:25:49.013 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-09-13T16:25:49.013 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:49.013 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-09-13T16:25:49.020 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-09-13T16:25:49.021 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:49.021 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-09-13T16:25:49.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:49.040 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:49.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:49.097 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:49.098 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:49.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:49.099 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:49.107 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:25:49.107 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [52] 2024-09-13T16:25:49.107 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-09-13T16:25:49.109 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:49.125 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:49.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:49.187 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:49.188 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:49.189 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:49.189 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:49.195 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:25:49.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi119.stderr:2024-09-13T16:25:49.193+0000 7fdbd87c1640 -1 init, newargv = 0x55dfbd03c510 newargc=15 2024-09-13T16:25:49.196 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi119.stderr:ceph-fuse[35102]: starting ceph client 2024-09-13T16:25:49.205 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi119.stderr:ceph-fuse[35102]: starting fuse 2024-09-13T16:25:50.297 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:25:50.316 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:50.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:50.375 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:50.376 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:50.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:50.376 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:25:50.384 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:25:50.384 INFO:teuthology.orchestra.run.smithi119.stdout:53 2024-09-13T16:25:50.384 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52, 53] 2024-09-13T16:25:50.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:25:50.384 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:25:50.384 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:25:50.384 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:25:50.384 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:25:50.384 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:25:50.385 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.2")) 2024-09-13T16:25:50.386 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:25:50.470 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.2.35102.asok 2024-09-13T16:25:50.658 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.35102.asok 2024-09-13T16:25:50.658 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:50.658 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.35102.asok status 2024-09-13T16:25:50.805 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:25:50.805 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:25:50.805 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:25:50.805 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:25:50.805 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "2", 2024-09-13T16:25:50.805 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:25:50.805 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "35102", 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 4944, 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 4944 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:50.806 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.2:0", 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 1825094135 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.2:0", 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 1825094135 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.4944 192.168.0.2:0/1825094135", 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.2:0/1825094135", 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:25:50.807 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 9, 2024-09-13T16:25:50.808 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 25, 2024-09-13T16:25:50.808 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:25:50.808 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:25:50.808 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:25:50.808 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:25:50.815 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-13T16:25:50.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:25:50.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 Processor -- start 2024-09-13T16:25:50.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- start start 2024-09-13T16:25:50.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f32f815e840 0x7f32f8070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:50.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f8161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:50.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32f8071210 con 0x7f32f815ec20 2024-09-13T16:25:50.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32f80714f0 con 0x7f32f815e840 2024-09-13T16:25:50.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80717d0 con 0x7f32f8165180 2024-09-13T16:25:50.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f32f815e840 0x7f32f8070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:50.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f32f815e840 0x7f32f8070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43228/0 (socket says 172.21.15.119:43228) 2024-09-13T16:25:50.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 -- 172.21.15.119:0/3081305686 learned_addr learned my addr 172.21.15.119:0/3081305686 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:50.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fe6bb700 1 --2- 172.21.15.119:0/3081305686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f8161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:50.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 -- 172.21.15.119:0/3081305686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 msgr2=0x7f32f8161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:50.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 --2- 172.21.15.119:0/3081305686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f8161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:50.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 -- 172.21.15.119:0/3081305686 >> v1:172.21.15.145:6789/0 conn(0x7f32f815ec20 legacy=0x7f32f8164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:50.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 -- 172.21.15.119:0/3081305686 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f8071ab0 con 0x7f32f815e840 2024-09-13T16:25:50.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fdeba700 1 --2- 172.21.15.119:0/3081305686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815e840 0x7f32f8070b80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009910 tx=0x7f32f4009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcee82aae9d2b98a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:50.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fceb8700 1 -- 172.21.15.119:0/3081305686 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f400ba60 con 0x7f32f815e840 2024-09-13T16:25:50.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fceb8700 1 -- 172.21.15.119:0/3081305686 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32f400bbc0 con 0x7f32f815e840 2024-09-13T16:25:50.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f32fceb8700 1 -- 172.21.15.119:0/3081305686 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32f4009e60 con 0x7f32f815e840 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- 172.21.15.119:0/3081305686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815e840 msgr2=0x7f32f8070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 --2- 172.21.15.119:0/3081305686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815e840 0x7f32f8070b80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f32f4009910 tx=0x7f32f4009c20 comp rx=0 tx=0).stop 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- 172.21.15.119:0/3081305686 shutdown_connections 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 --2- 172.21.15.119:0/3081305686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f8161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 --2- 172.21.15.119:0/3081305686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815e840 0x7f32f8070b80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- 172.21.15.119:0/3081305686 >> 172.21.15.119:0/3081305686 conn(0x7f32f806a350 msgr2=0x7f32f806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- 172.21.15.119:0/3081305686 shutdown_connections 2024-09-13T16:25:50.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.929+0000 7f330011e700 1 -- 172.21.15.119:0/3081305686 wait complete. 2024-09-13T16:25:50.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 Processor -- start 2024-09-13T16:25:50.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 -- start start 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32f815e840 0x7f32f80cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f80ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815d6e0 0x7f32f815db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f815e0d0 con 0x7f32f815d6e0 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80a8890 con 0x7f32f815e840 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f330011e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32f80a8b70 con 0x7f32f8165180 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815d6e0 0x7f32f815db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fdeba700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32f815e840 0x7f32f80cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815d6e0 0x7f32f815db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43238/0 (socket says 172.21.15.119:43238) 2024-09-13T16:25:50.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 -- 172.21.15.119:0/3498243551 learned_addr learned my addr 172.21.15.119:0/3498243551 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:50.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fd6b9700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f80ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:50.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 -- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 msgr2=0x7f32f80ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:50.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f80ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:50.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 -- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32f815e840 msgr2=0x7f32f80cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:50.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32f815e840 0x7f32f80cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:50.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.933+0000 7f32fe6bb700 1 -- 172.21.15.119:0/3498243551 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32f4009570 con 0x7f32f815d6e0 2024-09-13T16:25:50.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f32fe6bb700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815d6e0 0x7f32f815db90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f32e800d5d0 tx=0x7f32e800d8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:50.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f32ee7fc700 1 -- 172.21.15.119:0/3498243551 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e80125f0 con 0x7f32f815d6e0 2024-09-13T16:25:50.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f32ee7fc700 1 -- 172.21.15.119:0/3498243551 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32e8012770 con 0x7f32f815d6e0 2024-09-13T16:25:50.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f32ee7fc700 1 -- 172.21.15.119:0/3498243551 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32e8012b20 con 0x7f32f815d6e0 2024-09-13T16:25:50.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f80a8e50 con 0x7f32f815d6e0 2024-09-13T16:25:50.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f80a9410 con 0x7f32f815d6e0 2024-09-13T16:25:50.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f8067760 con 0x7f32f815d6e0 2024-09-13T16:25:50.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.937+0000 7f32ee7fc700 1 -- 172.21.15.119:0/3498243551 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f32e8012c80 con 0x7f32f815d6e0 2024-09-13T16:25:50.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.941+0000 7f32ee7fc700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32e4066760 0x7f32e4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:50.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.941+0000 7f32ee7fc700 1 -- 172.21.15.119:0/3498243551 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f32e8084400 con 0x7f32f815d6e0 2024-09-13T16:25:50.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.941+0000 7f32fdeba700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32e4066760 0x7f32e4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:50.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.941+0000 7f32fdeba700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32e4066760 0x7f32e4068c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f32f4000c00 tx=0x7f32f4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:50.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:50.945+0000 7f32ee7fc700 1 -- 172.21.15.119:0/3498243551 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f32e8051480 con 0x7f32f815d6e0 2024-09-13T16:25:51.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.149+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f32f8070b80 con 0x7f32f815d6e0 2024-09-13T16:25:51.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.149+0000 7f32ee7fc700 1 -- 172.21.15.119:0/3498243551 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f32e8054aa0 con 0x7f32f815d6e0 2024-09-13T16:25:51.154 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-13T16:25:51.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32e4066760 msgr2=0x7f32e4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32e4066760 0x7f32e4068c20 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f32f4000c00 tx=0x7f32f4008030 comp rx=0 tx=0).stop 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815d6e0 msgr2=0x7f32f815db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815d6e0 0x7f32f815db90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f32e800d5d0 tx=0x7f32e800d8e0 comp rx=0 tx=0).stop 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 shutdown_connections 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32e4066760 0x7f32e4068c20 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32f815d6e0 0x7f32f815db90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32f8165180 0x7f32f80ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 --2- 172.21.15.119:0/3498243551 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32f815e840 0x7f32f80cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 >> 172.21.15.119:0/3498243551 conn(0x7f32f806a350 msgr2=0x7f32f8162590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 shutdown_connections 2024-09-13T16:25:51.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:51.153+0000 7f330011e700 1 -- 172.21.15.119:0/3498243551 wait complete. 2024-09-13T16:25:51.168 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:25:51.168 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:25:51.169 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi145.front.sepia.ceph.com 2024-09-13T16:25:51.169 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-09-13T16:25:51.169 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-09-13T16:25:51.169 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:25:51.169 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:25:51.169 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:25:51.169 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-09-13T16:25:51.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.169 DEBUG:teuthology.orchestra.run.smithi145:> ip addr 2024-09-13T16:25:51.177 INFO:teuthology.orchestra.run.smithi145.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:51.177 INFO:teuthology.orchestra.run.smithi145.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:51.177 INFO:teuthology.orchestra.run.smithi145.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:66 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:67 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:90 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:91 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: inet 172.21.15.145/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:25:51.178 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft 42589sec preferred_lft 42589sec 2024-09-13T16:25:51.179 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::ec4:7aff:fe88:7791/64 scope link 2024-09-13T16:25:51.179 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:51.179 INFO:teuthology.orchestra.run.smithi145.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:25:51.179 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 02:42:d5:a1:81:93 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:51.179 INFO:teuthology.orchestra.run.smithi145.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:25:51.179 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:51.179 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-09-13T16:25:51.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.179 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:51.179 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:51.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add name ceph-brx type bridge 2024-09-13T16:25:51.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr flush dev ceph-brx 2024-09-13T16:25:51.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set ceph-brx up 2024-09-13T16:25:51.180 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-09-13T16:25:51.180 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:51.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.224 DEBUG:teuthology.orchestra.run.smithi145:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:25:51.234 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-09-13T16:25:51.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.235 DEBUG:teuthology.orchestra.run.smithi145:> route 2024-09-13T16:25:51.268 INFO:teuthology.orchestra.run.smithi145.stdout:Kernel IP routing table 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run.smithi145.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run.smithi145.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run.smithi145.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run.smithi145.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run.smithi145.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run.smithi145.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:25:51.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.270 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:51.270 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:51.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:25:51.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:25:51.270 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:25:51.270 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:51.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.313 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-09-13T16:25:51.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.361 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list-id 2024-09-13T16:25:51.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.368 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:51.369 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:51.369 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T16:25:51.369 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-09-13T16:25:51.369 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:51.400 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-09-13T16:25:51.400 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.400 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:51.400 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:51.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-09-13T16:25:51.400 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-09-13T16:25:51.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-09-13T16:25:51.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-09-13T16:25:51.401 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-09-13T16:25:51.401 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:51.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.520 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:51.520 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:51.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set brx.0 up 2024-09-13T16:25:51.521 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set dev brx.0 master ceph-brx 2024-09-13T16:25:51.521 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:51.549 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-09-13T16:25:51.549 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:51.549 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-09-13T16:25:51.554 INFO:teuthology.orchestra.run.smithi145.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-09-13T16:25:51.554 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:51.554 DEBUG:teuthology.orchestra.run.smithi145:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-09-13T16:25:51.604 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-09-13T16:25:51.620 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:51.621 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:51.631 INFO:teuthology.orchestra.run.smithi145.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:51.632 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:51.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.633 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-09-13T16:25:51.637 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:25:51.638 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-09-13T16:25:51.640 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-09-13T16:25:51.661 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:51.662 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:51.716 INFO:teuthology.orchestra.run.smithi145.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:51.717 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:51.717 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-09-13T16:25:51.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi145.stderr:2024-09-13T16:25:51.720+0000 7fac2af11640 -1 init, newargv = 0x56058f6015b0 newargc=15 2024-09-13T16:25:51.725 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi145.stderr:ceph-fuse[32509]: starting ceph client 2024-09-13T16:25:51.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi145.stderr:ceph-fuse[32509]: starting fuse 2024-09-13T16:25:52.825 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-09-13T16:25:52.843 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:52.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:52.902 INFO:teuthology.orchestra.run.smithi145.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:52.902 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:52.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:52.903 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-09-13T16:25:52.910 INFO:teuthology.orchestra.run.smithi145.stdout:52 2024-09-13T16:25:52.910 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:25:52.910 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:25:52.910 DEBUG:teuthology.orchestra.run.smithi145:> import glob 2024-09-13T16:25:52.910 DEBUG:teuthology.orchestra.run.smithi145:> import re 2024-09-13T16:25:52.910 DEBUG:teuthology.orchestra.run.smithi145:> import os 2024-09-13T16:25:52.910 DEBUG:teuthology.orchestra.run.smithi145:> import subprocess 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> def _find_admin_socket(client_name): 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> files = glob.glob(asok_path) 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> # Given a non-glob path, it better be there 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> if "*" not in asok_path: 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> assert(len(files) == 1) 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> return files[0] 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> for f in files: 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:25:52.911 DEBUG:teuthology.orchestra.run.smithi145:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> contents = proc_f.read() 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> if mountpoint in contents: 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> return f 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> print(_find_admin_socket("client.1")) 2024-09-13T16:25:52.912 DEBUG:teuthology.orchestra.run.smithi145:> ' 2024-09-13T16:25:52.994 INFO:teuthology.orchestra.run.smithi145.stdout:/var/run/ceph/ceph-client.1.32509.asok 2024-09-13T16:25:53.181 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.32509.asok 2024-09-13T16:25:53.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.32509.asok status 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "metadata": { 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "entity_id": "1", 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "hostname": "smithi145", 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "pid": "32509", 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "root": "/" 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "dentry_count": 0, 2024-09-13T16:25:53.284 INFO:teuthology.orchestra.run.smithi145.stdout: "dentry_pinned_count": 0, 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 4858, 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "inst": { 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "name": { 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "client", 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "num": 4858 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": { 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v1", 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 2654633161 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-13T16:25:53.285 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": { 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v1", 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 2654633161 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "inst_str": "client.4858 192.168.0.1:0/2654633161", 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "addr_str": "192.168.0.1:0/2654633161", 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "inode_count": 1, 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_epoch": 9, 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_epoch": 25, 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "blocklisted": false, 2024-09-13T16:25:53.286 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs" 2024-09-13T16:25:53.287 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-13T16:25:53.293 DEBUG:tasks.ceph_fuse:passing mntargs=[] 2024-09-13T16:25:53.293 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:25:53.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.413+0000 7f7ce9547700 1 Processor -- start 2024-09-13T16:25:53.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.413+0000 7f7ce9547700 1 -- start start 2024-09-13T16:25:53.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce9547700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7ce406c590 0x7f7ce40acf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:53.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce9547700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce40ad640 con 0x7f7ce40abf40 2024-09-13T16:25:53.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce9547700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce40b3140 con 0x7f7ce415a6c0 2024-09-13T16:25:53.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce9547700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce40b3ae0 con 0x7f7ce406c590 2024-09-13T16:25:53.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7cdbfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7ce406c590 0x7f7ce40acf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:53.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce8d46700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7ce40abf40 0x7f7ce406a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58766/0 (socket says 172.21.15.119:58766) 2024-09-13T16:25:53.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce8d46700 1 -- 172.21.15.119:0/3935085838 learned_addr learned my addr 172.21.15.119:0/3935085838 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce37fe700 1 -- 172.21.15.119:0/3935085838 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213213608 0 0) 0x7f7ce40ad640 con 0x7f7ce40abf40 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce37fe700 1 -- 172.21.15.119:0/3935085838 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd00035d0 con 0x7f7ce40abf40 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce37fe700 1 -- 172.21.15.119:0/3935085838 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2656838387 0 0) 0x7f7ce40b3140 con 0x7f7ce415a6c0 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7cdbfff700 1 -- 172.21.15.119:0/3935085838 >> v1:172.21.15.145:6789/0 conn(0x7f7ce415a6c0 legacy=0x7f7ce40b3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7cdbfff700 1 -- 172.21.15.119:0/3935085838 >> v1:172.21.15.119:6789/0 conn(0x7f7ce40abf40 legacy=0x7f7ce406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7cdbfff700 1 -- 172.21.15.119:0/3935085838 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce40b3420 con 0x7f7ce406c590 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7cdbfff700 1 --2- 172.21.15.119:0/3935085838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce406c590 0x7f7ce40acf10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009850 tx=0x7f7ccc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e34df4ab98c0375e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce37fe700 1 -- 172.21.15.119:0/3935085838 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc011830 con 0x7f7ce406c590 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce37fe700 1 -- 172.21.15.119:0/3935085838 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ccc011990 con 0x7f7ce406c590 2024-09-13T16:25:53.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.417+0000 7f7ce37fe700 1 -- 172.21.15.119:0/3935085838 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ccc011d40 con 0x7f7ce406c590 2024-09-13T16:25:53.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 -- 172.21.15.119:0/3935085838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce406c590 msgr2=0x7f7ce40acf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:53.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 --2- 172.21.15.119:0/3935085838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce406c590 0x7f7ce40acf10 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009850 tx=0x7f7ccc009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:53.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 -- 172.21.15.119:0/3935085838 shutdown_connections 2024-09-13T16:25:53.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 --2- 172.21.15.119:0/3935085838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce406c590 0x7f7ce40acf10 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:53.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 -- 172.21.15.119:0/3935085838 >> 172.21.15.119:0/3935085838 conn(0x7f7ce40ae750 msgr2=0x7f7ce40b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:53.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 -- 172.21.15.119:0/3935085838 shutdown_connections 2024-09-13T16:25:53.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 -- 172.21.15.119:0/3935085838 wait complete. 2024-09-13T16:25:53.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 Processor -- start 2024-09-13T16:25:53.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.421+0000 7f7ce9547700 1 -- start start 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ce40abf40 0x7f7ce40caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ce415a6c0 0x7f7ce40cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41f30c0 0x7f7ce4099dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce41574f0 con 0x7f7ce41f30c0 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce41f3640 con 0x7f7ce40abf40 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce41f3920 con 0x7f7ce415a6c0 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41f30c0 0x7f7ce4099dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41f30c0 0x7f7ce4099dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43444/0 (socket says 172.21.15.119:43444) 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 -- 172.21.15.119:0/376762667 learned_addr learned my addr 172.21.15.119:0/376762667 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:53.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7cdbfff700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ce415a6c0 0x7f7ce40cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:53.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce3fff700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ce40abf40 0x7f7ce40caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:53.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 -- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ce415a6c0 msgr2=0x7f7ce40cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:53.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ce415a6c0 0x7f7ce40cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:53.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 -- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ce40abf40 msgr2=0x7f7ce40caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:53.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ce40abf40 0x7f7ce40caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 -- 172.21.15.119:0/376762667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc0094b0 con 0x7f7ce41f30c0 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce8d46700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41f30c0 0x7f7ce4099dd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00cb40 tx=0x7f7cdc00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce17fa700 1 -- 172.21.15.119:0/376762667 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc007500 con 0x7f7ce41f30c0 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce17fa700 1 -- 172.21.15.119:0/376762667 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cdc007660 con 0x7f7ce41f30c0 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce41f3c00 con 0x7f7ce41f30c0 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce17fa700 1 -- 172.21.15.119:0/376762667 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cdc0079f0 con 0x7f7ce41f30c0 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce41efad0 con 0x7f7ce41f30c0 2024-09-13T16:25:53.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.425+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce406df20 con 0x7f7ce41f30c0 2024-09-13T16:25:53.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.429+0000 7f7ce17fa700 1 -- 172.21.15.119:0/376762667 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7cdc01b3b0 con 0x7f7ce41f30c0 2024-09-13T16:25:53.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.429+0000 7f7ce17fa700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cc4066730 0x7f7cc4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:53.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.429+0000 7f7ce17fa700 1 -- 172.21.15.119:0/376762667 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7cdc083e80 con 0x7f7ce41f30c0 2024-09-13T16:25:53.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.429+0000 7f7ce3fff700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cc4066730 0x7f7cc4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:53.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.429+0000 7f7ce3fff700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cc4066730 0x7f7cc4068bf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4000f60 tx=0x7f7cd4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:53.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.437+0000 7f7ce17fa700 1 -- 172.21.15.119:0/376762667 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7ce406df20 con 0x7f7ce41f30c0 2024-09-13T16:25:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7ce406df20 con 0x7f7ce41f30c0 2024-09-13T16:25:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce17fa700 1 -- 172.21.15.119:0/376762667 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v9) v1 ==== 53+0+88 (secure 0 0 0) 0x7f7ce406df20 con 0x7f7ce41f30c0 2024-09-13T16:25:53.648 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-09-13T16:25:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cc4066730 msgr2=0x7f7cc4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cc4066730 0x7f7cc4068bf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4000f60 tx=0x7f7cd4009040 comp rx=0 tx=0).stop 2024-09-13T16:25:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41f30c0 msgr2=0x7f7ce4099dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41f30c0 0x7f7ce4099dd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f7cdc00cb40 tx=0x7f7cdc00cb70 comp rx=0 tx=0).stop 2024-09-13T16:25:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 shutdown_connections 2024-09-13T16:25:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cc4066730 0x7f7cc4068bf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ce41f30c0 0x7f7ce4099dd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ce415a6c0 0x7f7ce40cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 --2- 172.21.15.119:0/376762667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ce40abf40 0x7f7ce40caa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 >> 172.21.15.119:0/376762667 conn(0x7f7ce406c590 msgr2=0x7f7ce409ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 shutdown_connections 2024-09-13T16:25:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:53.645+0000 7f7ce9547700 1 -- 172.21.15.119:0/376762667 wait complete. 2024-09-13T16:25:53.668 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:25:53.668 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:25:53.668 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi145.front.sepia.ceph.com 2024-09-13T16:25:53.668 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-09-13T16:25:53.668 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-09-13T16:25:53.669 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:25:53.669 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:25:53.669 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:25:53.669 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-09-13T16:25:53.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.669 DEBUG:teuthology.orchestra.run.smithi145:> ip addr 2024-09-13T16:25:53.677 INFO:teuthology.orchestra.run.smithi145.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:66 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:53.678 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:d9:38:67 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:90 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 0c:c4:7a:88:77:91 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout: inet 172.21.15.145/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft 42586sec preferred_lft 42586sec 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::ec4:7aff:fe88:7791/64 scope link 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:25:53.679 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 02:42:d5:a1:81:93 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout:7: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 66:d4:66:8f:07:31 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::845f:73ff:fed8:28cb/64 scope link 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout:8: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether 66:d4:66:8f:07:31 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::64d4:66ff:fe8f:731/64 scope link 2024-09-13T16:25:53.680 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.681 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.681 DEBUG:teuthology.orchestra.run.smithi145:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:25:53.733 INFO:teuthology.orchestra.run.smithi145.stdout:1 2024-09-13T16:25:53.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.736 DEBUG:teuthology.orchestra.run.smithi145:> route 2024-09-13T16:25:53.745 INFO:teuthology.orchestra.run.smithi145.stdout:Kernel IP routing table 2024-09-13T16:25:53.745 INFO:teuthology.orchestra.run.smithi145.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:25:53.745 INFO:teuthology.orchestra.run.smithi145.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:25:53.745 INFO:teuthology.orchestra.run.smithi145.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:25:53.746 INFO:teuthology.orchestra.run.smithi145.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:25:53.746 INFO:teuthology.orchestra.run.smithi145.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:25:53.746 INFO:teuthology.orchestra.run.smithi145.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:25:53.746 INFO:teuthology.orchestra.run.smithi145.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:25:53.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.746 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:53.746 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:53.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:25:53.746 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:25:53.747 DEBUG:teuthology.orchestra.run.smithi145:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:25:53.747 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:53.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.784 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-09-13T16:25:53.837 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T16:25:53.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.837 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list-id 2024-09-13T16:25:53.843 INFO:teuthology.orchestra.run.smithi145.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-09-13T16:25:53.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.843 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:53.843 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:53.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T16:25:53.843 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-09-13T16:25:53.843 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:53.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.872 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-13T16:25:53.931 INFO:teuthology.orchestra.run.smithi145.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:53.931 INFO:teuthology.orchestra.run.smithi145.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:53.931 INFO:teuthology.orchestra.run.smithi145.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether ea:3a:58:1a:6b:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.932 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::e83a:58ff:fe1a:6b6c/64 scope link 2024-09-13T16:25:53.933 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.934 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 ::1/128 scope host 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout:2: veth0@if8: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: link/ether ea:3a:58:1a:6b:6c brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: inet6 fe80::e83a:58ff:fe1a:6b6c/64 scope link 2024-09-13T16:25:53.949 INFO:teuthology.orchestra.run.smithi145.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:25:53.950 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-09-13T16:25:53.951 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-09-13T16:25:53.951 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:54.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:54.014 DEBUG:teuthology.orchestra.run.smithi145:> (cd / && exec bash -c ' 2024-09-13T16:25:54.014 DEBUG:teuthology.orchestra.run.smithi145:> set -e 2024-09-13T16:25:54.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set brx.1 up 2024-09-13T16:25:54.014 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link set dev brx.1 master ceph-brx 2024-09-13T16:25:54.014 DEBUG:teuthology.orchestra.run.smithi145:> ') 2024-09-13T16:25:54.042 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-09-13T16:25:54.042 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:54.042 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-09-13T16:25:54.054 INFO:teuthology.orchestra.run.smithi145.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-09-13T16:25:54.054 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:25:54.054 DEBUG:teuthology.orchestra.run.smithi145:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-09-13T16:25:54.112 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-09-13T16:25:54.134 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:54.134 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:54.204 INFO:teuthology.orchestra.run.smithi145.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:54.205 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:54.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:54.206 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-09-13T16:25:54.212 INFO:teuthology.orchestra.run.smithi145.stdout:52 2024-09-13T16:25:54.212 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [52] 2024-09-13T16:25:54.212 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-09-13T16:25:54.258 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-09-13T16:25:54.273 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:54.274 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:54.282 INFO:teuthology.orchestra.run.smithi145.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:54.283 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:54.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:54.284 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-09-13T16:25:54.334 INFO:teuthology.orchestra.run.smithi145.stdout:52 2024-09-13T16:25:54.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi145.stderr:2024-09-13T16:25:54.344+0000 7f0db43ca640 -1 init, newargv = 0x5580be3fc400 newargc=15 2024-09-13T16:25:54.349 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi145.stderr:ceph-fuse[32625]: starting ceph client 2024-09-13T16:25:54.359 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi145.stderr:ceph-fuse[32625]: starting fuse 2024-09-13T16:25:55.437 DEBUG:teuthology.orchestra.run.smithi145:> sudo modprobe fuse 2024-09-13T16:25:55.455 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:25:55.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:25:55.510 INFO:teuthology.orchestra.run.smithi145.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:25:55.511 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:25:55.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.512 DEBUG:teuthology.orchestra.run.smithi145:> ls /sys/fs/fuse/connections 2024-09-13T16:25:55.516 INFO:teuthology.orchestra.run.smithi145.stdout:52 2024-09-13T16:25:55.516 INFO:teuthology.orchestra.run.smithi145.stdout:53 2024-09-13T16:25:55.517 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52, 53] 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> import glob 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> import re 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> import os 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> import subprocess 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> def _find_admin_socket(client_name): 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> files = glob.glob(asok_path) 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-09-13T16:25:55.517 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> # Given a non-glob path, it better be there 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> if "*" not in asok_path: 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> assert(len(files) == 1) 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> return files[0] 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> for f in files: 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> contents = proc_f.read() 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> if mountpoint in contents: 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> return f 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> 2024-09-13T16:25:55.518 DEBUG:teuthology.orchestra.run.smithi145:> print(_find_admin_socket("client.3")) 2024-09-13T16:25:55.519 DEBUG:teuthology.orchestra.run.smithi145:> ' 2024-09-13T16:25:55.597 INFO:teuthology.orchestra.run.smithi145.stdout:/var/run/ceph/ceph-client.3.32625.asok 2024-09-13T16:25:55.782 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.32625.asok 2024-09-13T16:25:55.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.782 DEBUG:teuthology.orchestra.run.smithi145:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.32625.asok status 2024-09-13T16:25:55.925 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-09-13T16:25:55.925 INFO:teuthology.orchestra.run.smithi145.stdout: "metadata": { 2024-09-13T16:25:55.925 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:25:55.925 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:25:55.925 INFO:teuthology.orchestra.run.smithi145.stdout: "entity_id": "3", 2024-09-13T16:25:55.925 INFO:teuthology.orchestra.run.smithi145.stdout: "hostname": "smithi145", 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "pid": "32625", 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "root": "/" 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "dentry_count": 0, 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "dentry_pinned_count": 0, 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "id": 4676, 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "inst": { 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "name": { 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "client", 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "num": 4676 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:55.926 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": { 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v1", 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "192.168.0.2:0", 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 67587920 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: } 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": { 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "type": "v1", 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "addr": "192.168.0.2:0", 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "nonce": 67587920 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: }, 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "inst_str": "client.4676 192.168.0.2:0/67587920", 2024-09-13T16:25:55.927 INFO:teuthology.orchestra.run.smithi145.stdout: "addr_str": "192.168.0.2:0/67587920", 2024-09-13T16:25:55.928 INFO:teuthology.orchestra.run.smithi145.stdout: "inode_count": 1, 2024-09-13T16:25:55.928 INFO:teuthology.orchestra.run.smithi145.stdout: "mds_epoch": 9, 2024-09-13T16:25:55.928 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_epoch": 25, 2024-09-13T16:25:55.928 INFO:teuthology.orchestra.run.smithi145.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:25:55.928 INFO:teuthology.orchestra.run.smithi145.stdout: "blocklisted": false, 2024-09-13T16:25:55.928 INFO:teuthology.orchestra.run.smithi145.stdout: "fs_name": "cephfs" 2024-09-13T16:25:55.928 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-09-13T16:25:55.935 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.935 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:25:55.935 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:25:55.943 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:25:55.943 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:25:55.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:25:55.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.961 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:25:55.961 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.2 2024-09-13T16:25:55.970 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:25:55.970 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-09-13T16:25:55.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-09-13T16:25:55.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.986 DEBUG:teuthology.orchestra.run.smithi145:> stat --file-system '--printf=%T 2024-09-13T16:25:55.986 DEBUG:teuthology.orchestra.run.smithi145:> ' -- /home/ubuntu/cephtest/mnt.1 2024-09-13T16:25:55.994 INFO:teuthology.orchestra.run.smithi145.stdout:fuseblk 2024-09-13T16:25:55.994 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-09-13T16:25:55.995 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:55.995 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-09-13T16:25:56.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:56.008 DEBUG:teuthology.orchestra.run.smithi145:> stat --file-system '--printf=%T 2024-09-13T16:25:56.008 DEBUG:teuthology.orchestra.run.smithi145:> ' -- /home/ubuntu/cephtest/mnt.3 2024-09-13T16:25:56.057 INFO:teuthology.orchestra.run.smithi145.stdout:fuseblk 2024-09-13T16:25:56.057 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-09-13T16:25:56.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:25:56.058 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-09-13T16:25:56.072 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-09-13T16:25:56.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.209+0000 7fc2db59e700 1 Processor -- start 2024-09-13T16:25:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.209+0000 7fc2db59e700 1 -- start start 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2db59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc1594f0 0x7fc2dc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2db59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc2dc06da70 0x7fc2dc0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2db59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2dc0b2280 con 0x7fc2dc0b3630 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2db59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2dc070c10 con 0x7fc2dc06da70 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2db59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc06deb0 con 0x7fc2dc1594f0 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2dad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc2dc0b3630 0x7fc2dc15d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58772/0 (socket says 172.21.15.119:58772) 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2dad9d700 1 -- 172.21.15.119:0/1139588165 learned_addr learned my addr 172.21.15.119:0/1139588165 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2da59c700 1 --2- 172.21.15.119:0/1139588165 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc1594f0 0x7fc2dc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2d9d9b700 1 --2- 172.21.15.119:0/1139588165 >> v2:172.21.15.145:3300/0 conn(0x7fc2dc06da70 0x7fc2dc0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2d959a700 1 -- 172.21.15.119:0/1139588165 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226368353 0 0) 0x7fc2dc0b2280 con 0x7fc2dc0b3630 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2d959a700 1 -- 172.21.15.119:0/1139588165 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2c80035d0 con 0x7fc2dc0b3630 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2da59c700 1 -- 172.21.15.119:0/1139588165 >> v2:172.21.15.145:3300/0 conn(0x7fc2dc06da70 msgr2=0x7fc2dc0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2da59c700 1 --2- 172.21.15.119:0/1139588165 >> v2:172.21.15.145:3300/0 conn(0x7fc2dc06da70 0x7fc2dc0b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2da59c700 1 -- 172.21.15.119:0/1139588165 >> v1:172.21.15.119:6789/0 conn(0x7fc2dc0b3630 legacy=0x7fc2dc15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2da59c700 1 -- 172.21.15.119:0/1139588165 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2dc0cc890 con 0x7fc2dc1594f0 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2da59c700 1 --2- 172.21.15.119:0/1139588165 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc1594f0 0x7fc2dc06d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc070d50 tx=0x7fc2cc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fefde0e31f7b43f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2d959a700 1 -- 172.21.15.119:0/1139588165 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc016070 con 0x7fc2dc1594f0 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2d959a700 1 -- 172.21.15.119:0/1139588165 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2cc00bb20 con 0x7fc2dc1594f0 2024-09-13T16:25:56.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.213+0000 7fc2d959a700 1 -- 172.21.15.119:0/1139588165 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2cc00e070 con 0x7fc2dc1594f0 2024-09-13T16:25:56.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- 172.21.15.119:0/1139588165 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc1594f0 msgr2=0x7fc2dc06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1139588165 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc1594f0 0x7fc2dc06d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc2dc070d50 tx=0x7fc2cc00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:56.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- 172.21.15.119:0/1139588165 shutdown_connections 2024-09-13T16:25:56.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1139588165 >> v2:172.21.15.145:3300/0 conn(0x7fc2dc06da70 0x7fc2dc0b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1139588165 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc1594f0 0x7fc2dc06d530 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- 172.21.15.119:0/1139588165 >> 172.21.15.119:0/1139588165 conn(0x7fc2dc06a350 msgr2=0x7fc2dc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:56.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- 172.21.15.119:0/1139588165 shutdown_connections 2024-09-13T16:25:56.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- 172.21.15.119:0/1139588165 wait complete. 2024-09-13T16:25:56.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 Processor -- start 2024-09-13T16:25:56.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- start start 2024-09-13T16:25:56.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2dc06da70 0x7fc2dc155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc0b3630 0x7fc2dc1f2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2dc1594f0 0x7fc2dc158500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc157ad0 con 0x7fc2dc1594f0 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1f2980 con 0x7fc2dc06da70 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.217+0000 7fc2db59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2dc1565d0 con 0x7fc2dc0b3630 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2dc1594f0 0x7fc2dc158500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2dc1594f0 0x7fc2dc158500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43452/0 (socket says 172.21.15.119:43452) 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2da59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2dc06da70 0x7fc2dc155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2d9d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc0b3630 0x7fc2dc1f2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 -- 172.21.15.119:0/1698266758 learned_addr learned my addr 172.21.15.119:0/1698266758 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:56.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 -- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc0b3630 msgr2=0x7fc2dc1f2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc0b3630 0x7fc2dc1f2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 -- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2dc06da70 msgr2=0x7fc2dc155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2dc06da70 0x7fc2dc155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 -- 172.21.15.119:0/1698266758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2cc009a70 con 0x7fc2dc1594f0 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2dad9d700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2dc1594f0 0x7fc2dc158500 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c520 tx=0x7fc2d400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2d2ffd700 1 -- 172.21.15.119:0/1698266758 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d4005d90 con 0x7fc2dc1594f0 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2d2ffd700 1 -- 172.21.15.119:0/1698266758 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2d400ce90 con 0x7fc2dc1594f0 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2dc156010 con 0x7fc2dc1594f0 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2d2ffd700 1 -- 172.21.15.119:0/1698266758 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2d40072f0 con 0x7fc2dc1594f0 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2dc157d00 con 0x7fc2dc1594f0 2024-09-13T16:25:56.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.221+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2dc071e90 con 0x7fc2dc1594f0 2024-09-13T16:25:56.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.225+0000 7fc2d2ffd700 1 -- 172.21.15.119:0/1698266758 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc2d4007450 con 0x7fc2dc1594f0 2024-09-13T16:25:56.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.225+0000 7fc2d2ffd700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2bc066760 0x7fc2bc068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.225+0000 7fc2d2ffd700 1 -- 172.21.15.119:0/1698266758 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fc2d4083850 con 0x7fc2dc1594f0 2024-09-13T16:25:56.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.225+0000 7fc2da59c700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2bc066760 0x7fc2bc068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.225+0000 7fc2da59c700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2bc066760 0x7fc2bc068c20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc000f80 tx=0x7fc2cc011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.233+0000 7fc2d2ffd700 1 -- 172.21.15.119:0/1698266758 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc2d4050840 con 0x7fc2dc1594f0 2024-09-13T16:25:56.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.401+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc2dc0b26f0 con 0x7fc2dc1594f0 2024-09-13T16:25:56.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.405+0000 7fc2d2ffd700 1 -- 172.21.15.119:0/1698266758 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7fc2d4053e60 con 0x7fc2dc1594f0 2024-09-13T16:25:56.408 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:56.408 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:56.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2bc066760 msgr2=0x7fc2bc068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2bc066760 0x7fc2bc068c20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc2cc000f80 tx=0x7fc2cc011000 comp rx=0 tx=0).stop 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2dc1594f0 msgr2=0x7fc2dc158500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2dc1594f0 0x7fc2dc158500 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc2d400c520 tx=0x7fc2d400c830 comp rx=0 tx=0).stop 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 shutdown_connections 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2bc066760 0x7fc2bc068c20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2dc1594f0 0x7fc2dc158500 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2dc0b3630 0x7fc2dc1f2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 --2- 172.21.15.119:0/1698266758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2dc06da70 0x7fc2dc155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 >> 172.21.15.119:0/1698266758 conn(0x7fc2dc06a350 msgr2=0x7fc2dc0b0710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 shutdown_connections 2024-09-13T16:25:56.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.409+0000 7fc2db59e700 1 -- 172.21.15.119:0/1698266758 wait complete. 2024-09-13T16:25:56.427 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:25:56.427 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:56.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7727e06700 1 Processor -- start 2024-09-13T16:25:56.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7727e06700 1 -- start start 2024-09-13T16:25:56.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7727e06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f7720167b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7727e06700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200b4390 con 0x7f77200b1d50 2024-09-13T16:25:56.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7727e06700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772006c7b0 con 0x7f772015cb90 2024-09-13T16:25:56.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7727e06700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77200b1be0 con 0x7f77200b5860 2024-09-13T16:25:56.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f77253a1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f77200b1d50 0x7f772006c0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58780/0 (socket says 172.21.15.119:58780) 2024-09-13T16:25:56.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f77253a1700 1 -- 172.21.15.119:0/1208376529 learned_addr learned my addr 172.21.15.119:0/1208376529 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:56.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f77263a3700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f7720167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3627134572 0 0) 0x7f77200b4390 con 0x7f77200b1d50 2024-09-13T16:25:56.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f770c0035d0 con 0x7f77200b1d50 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814816642 0 0) 0x7f772006c7b0 con 0x7f772015cb90 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77200b4390 con 0x7f772015cb90 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 259900472 0 0) 0x7f770c0035d0 con 0x7f77200b1d50 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 msgr2=0x7f7720167b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f7720167b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 >> v1:172.21.15.145:6789/0 conn(0x7f772015cb90 legacy=0x7f772006b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77200b1060 con 0x7f77200b1d50 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f771c001670 con 0x7f77200b1d50 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.541+0000 7f77263a3700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f7720167b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 >> v1:172.21.15.119:6789/0 conn(0x7f77200b1d50 legacy=0x7f772006c0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f770c003be0 0x7f770c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f770c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f770c004db0 0x7f770c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c00bae0 con 0x7f770c004db0 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f770c00bc50 con 0x7f770c003be0 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f770c00be10 con 0x7f77200b5860 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f770c004db0 0x7f770c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7725ba2700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f770c003be0 0x7f770c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77253a1700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f770c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 -- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 msgr2=0x7f770c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f770c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 -- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f770c003be0 msgr2=0x7f770c003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f770c003be0 0x7f770c003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 -- 172.21.15.119:0/1208376529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718009a70 con 0x7f770c004db0 2024-09-13T16:25:56.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77253a1700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200b5860 0x7f770c004870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:56.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f770c004db0 0x7f770c005230 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f771800e350 tx=0x7f770c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bca87d3ac0d1d71 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771800edf0 con 0x7f770c004db0 2024-09-13T16:25:56.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771800f040 con 0x7f770c004db0 2024-09-13T16:25:56.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7724ba0700 1 -- 172.21.15.119:0/1208376529 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77180146a0 con 0x7f770c004db0 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7727e06700 1 -- 172.21.15.119:0/1208376529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f770c004db0 msgr2=0x7f770c005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7727e06700 1 --2- 172.21.15.119:0/1208376529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f770c004db0 0x7f770c005230 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f771800e350 tx=0x7f770c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f77263a3700 1 -- 172.21.15.119:0/1208376529 reap_dead start 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7727e06700 1 -- 172.21.15.119:0/1208376529 shutdown_connections 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7727e06700 1 -- 172.21.15.119:0/1208376529 >> 172.21.15.119:0/1208376529 conn(0x7f7720099a50 msgr2=0x7f7720099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7727e06700 1 -- 172.21.15.119:0/1208376529 shutdown_connections 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.545+0000 7f7727e06700 1 -- 172.21.15.119:0/1208376529 wait complete. 2024-09-13T16:25:56.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 Processor -- start 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 -- start start 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f772013dd50 0x7f7720140e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7720154000 0x7f77200aee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200af3a0 0x7f77200cadc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77200af820 con 0x7f7720154000 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772013e130 con 0x7f772013dd50 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77201413d0 con 0x7f77200af3a0 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7720154000 0x7f77200aee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7720154000 0x7f77200aee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43462/0 (socket says 172.21.15.119:43462) 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 -- 172.21.15.119:0/4198993315 learned_addr learned my addr 172.21.15.119:0/4198993315 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7725ba2700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f772013dd50 0x7f7720140e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77263a3700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200af3a0 0x7f77200cadc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 -- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200af3a0 msgr2=0x7f77200cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200af3a0 0x7f77200cadc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 -- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f772013dd50 msgr2=0x7f7720140e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f772013dd50 0x7f7720140e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 -- 172.21.15.119:0/4198993315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77201523a0 con 0x7f7720154000 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77253a1700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7720154000 0x7f77200aee60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f771c000fd0 tx=0x7f771c00ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77167fc700 1 -- 172.21.15.119:0/4198993315 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c004b50 con 0x7f7720154000 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77167fc700 1 -- 172.21.15.119:0/4198993315 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f771c004cb0 con 0x7f7720154000 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7720154550 con 0x7f7720154000 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f77167fc700 1 -- 172.21.15.119:0/4198993315 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771c007310 con 0x7f7720154000 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.549+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7720154b10 con 0x7f7720154000 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.553+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772014e910 con 0x7f7720154000 2024-09-13T16:25:56.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.553+0000 7f77167fc700 1 -- 172.21.15.119:0/4198993315 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f771c007470 con 0x7f7720154000 2024-09-13T16:25:56.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.553+0000 7f77167fc700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7704066760 0x7f7704068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.553+0000 7f77167fc700 1 -- 172.21.15.119:0/4198993315 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f771c083be0 con 0x7f7720154000 2024-09-13T16:25:56.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.553+0000 7f7725ba2700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7704066760 0x7f7704068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.553+0000 7f7725ba2700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7704066760 0x7f7704068c20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77200afd40 tx=0x7f7710009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.557+0000 7f77167fc700 1 -- 172.21.15.119:0/4198993315 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f771c050b40 con 0x7f7720154000 2024-09-13T16:25:56.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.725+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f772015cf70 con 0x7f7720154000 2024-09-13T16:25:56.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f77167fc700 1 -- 172.21.15.119:0/4198993315 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f771c054160 con 0x7f7720154000 2024-09-13T16:25:56.731 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7704066760 msgr2=0x7f7704068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7704066760 0x7f7704068c20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f77200afd40 tx=0x7f7710009040 comp rx=0 tx=0).stop 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7720154000 msgr2=0x7f77200aee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7720154000 0x7f77200aee60 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f771c000fd0 tx=0x7f771c00ddc0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 shutdown_connections 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7704066760 0x7f7704068c20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f77200af3a0 0x7f77200cadc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7720154000 0x7f77200aee60 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 --2- 172.21.15.119:0/4198993315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f772013dd50 0x7f7720140e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 >> 172.21.15.119:0/4198993315 conn(0x7f7720099a50 msgr2=0x7f77200b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:56.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 shutdown_connections 2024-09-13T16:25:56.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.729+0000 7f7727e06700 1 -- 172.21.15.119:0/4198993315 wait complete. 2024-09-13T16:25:56.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:25:56.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:25:56.830 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:25:56.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.857+0000 7f6d4144a700 1 Processor -- start 2024-09-13T16:25:56.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.857+0000 7f6d4144a700 1 -- start start 2024-09-13T16:25:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.857+0000 7f6d4144a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6d3c0ce5f0 0x7f6d3c0cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.857+0000 7f6d4144a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3c0cd600 con 0x7f6d3c0d0fb0 2024-09-13T16:25:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d3c0d0dd0 con 0x7f6d3c15cae0 2024-09-13T16:25:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d3c0ce1d0 con 0x7f6d3c0ce5f0 2024-09-13T16:25:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3b7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6d3c0ce5f0 0x7f6d3c0cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3b7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6d3c0ce5f0 0x7f6d3c0cff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43472/0 (socket says 172.21.15.119:43472) 2024-09-13T16:25:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3b7fe700 1 -- 172.21.15.119:0/2226553565 learned_addr learned my addr 172.21.15.119:0/2226553565 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d40c49700 1 --1- 172.21.15.119:0/2226553565 >> v1:172.21.15.119:6789/0 conn(0x7f6d3c0d0fb0 0x7f6d3c0b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58784/0 (socket says 172.21.15.119:58784) 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3b7fe700 1 -- 172.21.15.119:0/2226553565 >> v1:172.21.15.145:6789/0 conn(0x7f6d3c15cae0 legacy=0x7f6d3c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3b7fe700 1 -- 172.21.15.119:0/2226553565 >> v1:172.21.15.119:6789/0 conn(0x7f6d3c0d0fb0 legacy=0x7f6d3c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3b7fe700 1 -- 172.21.15.119:0/2226553565 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d3c0cd830 con 0x7f6d3c0ce5f0 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3b7fe700 1 --2- 172.21.15.119:0/2226553565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c0ce5f0 0x7f6d3c0cff60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009e50 tx=0x7f6d3400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5bdd0612a89933 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3affd700 1 -- 172.21.15.119:0/2226553565 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3400e070 con 0x7f6d3c0ce5f0 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3affd700 1 -- 172.21.15.119:0/2226553565 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d34003a40 con 0x7f6d3c0ce5f0 2024-09-13T16:25:56.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d3affd700 1 -- 172.21.15.119:0/2226553565 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d34003df0 con 0x7f6d3c0ce5f0 2024-09-13T16:25:56.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- 172.21.15.119:0/2226553565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c0ce5f0 msgr2=0x7f6d3c0cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 --2- 172.21.15.119:0/2226553565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c0ce5f0 0x7f6d3c0cff60 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009e50 tx=0x7f6d3400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:56.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- 172.21.15.119:0/2226553565 shutdown_connections 2024-09-13T16:25:56.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 --2- 172.21.15.119:0/2226553565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c0ce5f0 0x7f6d3c0cff60 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- 172.21.15.119:0/2226553565 >> 172.21.15.119:0/2226553565 conn(0x7f6d3c0d29f0 msgr2=0x7f6d3c0d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:56.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- 172.21.15.119:0/2226553565 shutdown_connections 2024-09-13T16:25:56.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- 172.21.15.119:0/2226553565 wait complete. 2024-09-13T16:25:56.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 Processor -- start 2024-09-13T16:25:56.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.861+0000 7f6d4144a700 1 -- start start 2024-09-13T16:25:56.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d3c0ce5f0 0x7f6d3c0c94e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d3c0d0fb0 0x7f6d3c0b0e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c15cae0 0x7f6d3c15a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c0cea60 con 0x7f6d3c15cae0 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c15e680 con 0x7f6d3c0ce5f0 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d3c160320 con 0x7f6d3c0d0fb0 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c15cae0 0x7f6d3c15a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c15cae0 0x7f6d3c15a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43484/0 (socket says 172.21.15.119:43484) 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 -- 172.21.15.119:0/3977851948 learned_addr learned my addr 172.21.15.119:0/3977851948 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d3b7fe700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d3c0d0fb0 0x7f6d3c0b0e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d3bfff700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d3c0ce5f0 0x7f6d3c0c94e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 -- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d3c0d0fb0 msgr2=0x7f6d3c0b0e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d3c0d0fb0 0x7f6d3c0b0e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 -- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d3c0ce5f0 msgr2=0x7f6d3c0c94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:56.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d3c0ce5f0 0x7f6d3c0c94e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:56.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 -- 172.21.15.119:0/3977851948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d34009b30 con 0x7f6d3c15cae0 2024-09-13T16:25:56.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d3b7fe700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d3c0d0fb0 0x7f6d3c0b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:56.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d40c49700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c15cae0 0x7f6d3c15a7f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00c410 tx=0x7f6d2c00c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d38ff9700 1 -- 172.21.15.119:0/3977851948 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c005cd0 con 0x7f6d3c15cae0 2024-09-13T16:25:56.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d38ff9700 1 -- 172.21.15.119:0/3977851948 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d2c005e30 con 0x7f6d3c15cae0 2024-09-13T16:25:56.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d3c15faa0 con 0x7f6d3c15cae0 2024-09-13T16:25:56.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d38ff9700 1 -- 172.21.15.119:0/3977851948 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d2c0072f0 con 0x7f6d3c15cae0 2024-09-13T16:25:56.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d3c160040 con 0x7f6d3c15cae0 2024-09-13T16:25:56.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.865+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d3c160460 con 0x7f6d3c15cae0 2024-09-13T16:25:56.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.869+0000 7f6d38ff9700 1 -- 172.21.15.119:0/3977851948 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6d2c007450 con 0x7f6d3c15cae0 2024-09-13T16:25:56.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.869+0000 7f6d38ff9700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6d28066730 0x7f6d28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:56.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.869+0000 7f6d38ff9700 1 -- 172.21.15.119:0/3977851948 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6d2c083950 con 0x7f6d3c15cae0 2024-09-13T16:25:56.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.869+0000 7f6d3bfff700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6d28066730 0x7f6d28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:56.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.869+0000 7f6d3bfff700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6d28066730 0x7f6d28068bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d30004f70 tx=0x7f6d30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:56.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:56.873+0000 7f6d38ff9700 1 -- 172.21.15.119:0/3977851948 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6d2c0508b0 con 0x7f6d3c15cae0 2024-09-13T16:25:57.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.045+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6d3c0744c0 con 0x7f6d3c15cae0 2024-09-13T16:25:57.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.045+0000 7f6d38ff9700 1 -- 172.21.15.119:0/3977851948 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6d2c053ed0 con 0x7f6d3c15cae0 2024-09-13T16:25:57.050 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:57.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6d28066730 msgr2=0x7f6d28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6d28066730 0x7f6d28068bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6d30004f70 tx=0x7f6d30009040 comp rx=0 tx=0).stop 2024-09-13T16:25:57.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c15cae0 msgr2=0x7f6d3c15a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c15cae0 0x7f6d3c15a7f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c00c410 tx=0x7f6d2c00c7d0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 shutdown_connections 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6d28066730 0x7f6d28068bf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6d3c15cae0 0x7f6d3c15a7f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6d3c0d0fb0 0x7f6d3c0b0e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 --2- 172.21.15.119:0/3977851948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6d3c0ce5f0 0x7f6d3c0c94e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 >> 172.21.15.119:0/3977851948 conn(0x7f6d3c0d29f0 msgr2=0x7f6d3c0b3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 shutdown_connections 2024-09-13T16:25:57.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.049+0000 7f6d4144a700 1 -- 172.21.15.119:0/3977851948 wait complete. 2024-09-13T16:25:57.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:25:57.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec93523700 1 Processor -- start 2024-09-13T16:25:57.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec93523700 1 -- start start 2024-09-13T16:25:57.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec93523700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c06b710 0x7fec8c06bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec93523700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec8c0d1640 0x7fec8c0d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec93523700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec8c06c070 con 0x7fec8c15cfb0 2024-09-13T16:25:57.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec93523700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8c0cee40 con 0x7fec8c0d1640 2024-09-13T16:25:57.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec93523700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c0d1220 con 0x7fec8c06b710 2024-09-13T16:25:57.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec91ac0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec8c0d1640 0x7fec8c0d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec90abe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c06b710 0x7fec8c06bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec912bf700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fec8c15cfb0 0x7fec8c06ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:49612/0 (socket says 172.21.15.119:49612) 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec912bf700 1 -- 172.21.15.119:0/1950700376 learned_addr learned my addr 172.21.15.119:0/1950700376 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec90abe700 1 -- 172.21.15.119:0/1950700376 >> v2:172.21.15.145:3300/0 conn(0x7fec8c0d1640 msgr2=0x7fec8c0d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec90abe700 1 --2- 172.21.15.119:0/1950700376 >> v2:172.21.15.145:3300/0 conn(0x7fec8c0d1640 0x7fec8c0d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec90abe700 1 -- 172.21.15.119:0/1950700376 >> v1:172.21.15.145:6789/0 conn(0x7fec8c15cfb0 legacy=0x7fec8c06ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec90abe700 1 -- 172.21.15.119:0/1950700376 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec8c0ce880 con 0x7fec8c06b710 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.181+0000 7fec91ac0700 1 --2- 172.21.15.119:0/1950700376 >> v2:172.21.15.145:3300/0 conn(0x7fec8c0d1640 0x7fec8c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec90abe700 1 --2- 172.21.15.119:0/1950700376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c06b710 0x7fec8c06bb30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec88009850 tx=0x7fec88009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a304d82c9a0cd62 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec83fff700 1 -- 172.21.15.119:0/1950700376 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88011830 con 0x7fec8c06b710 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec83fff700 1 -- 172.21.15.119:0/1950700376 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec88011990 con 0x7fec8c06b710 2024-09-13T16:25:57.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec83fff700 1 -- 172.21.15.119:0/1950700376 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec88011d40 con 0x7fec8c06b710 2024-09-13T16:25:57.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 -- 172.21.15.119:0/1950700376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c06b710 msgr2=0x7fec8c06bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 --2- 172.21.15.119:0/1950700376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c06b710 0x7fec8c06bb30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fec88009850 tx=0x7fec88009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 -- 172.21.15.119:0/1950700376 shutdown_connections 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 --2- 172.21.15.119:0/1950700376 >> v2:172.21.15.145:3300/0 conn(0x7fec8c0d1640 0x7fec8c0d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 --2- 172.21.15.119:0/1950700376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c06b710 0x7fec8c06bb30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 -- 172.21.15.119:0/1950700376 >> 172.21.15.119:0/1950700376 conn(0x7fec8c0d2da0 msgr2=0x7fec8c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 -- 172.21.15.119:0/1950700376 shutdown_connections 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 -- 172.21.15.119:0/1950700376 wait complete. 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 Processor -- start 2024-09-13T16:25:57.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.185+0000 7fec93523700 1 -- start start 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec8c06b710 0x7fec8c15b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c0d1640 0x7fec8c15fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8c15cfb0 0x7fec8c160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c15b900 con 0x7fec8c06b710 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c1f6e40 con 0x7fec8c15cfb0 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec8c1f38a0 con 0x7fec8c0d1640 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec912bf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec8c06b710 0x7fec8c15b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec90abe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c0d1640 0x7fec8c15fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec91ac0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8c15cfb0 0x7fec8c160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec90abe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c0d1640 0x7fec8c15fd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:36048/0 (socket says 172.21.15.119:36048) 2024-09-13T16:25:57.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec90abe700 1 -- 172.21.15.119:0/4128183321 learned_addr learned my addr 172.21.15.119:0/4128183321 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec912bf700 1 -- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c0d1640 msgr2=0x7fec8c15fd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec912bf700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c0d1640 0x7fec8c15fd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec912bf700 1 -- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8c15cfb0 msgr2=0x7fec8c160240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec912bf700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8c15cfb0 0x7fec8c160240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec912bf700 1 -- 172.21.15.119:0/4128183321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec84009a70 con 0x7fec8c06b710 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec90abe700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c0d1640 0x7fec8c15fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec912bf700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec8c06b710 0x7fec8c15b3c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00c3c0 tx=0x7fec7c00c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec81ffb700 1 -- 172.21.15.119:0/4128183321 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c005b60 con 0x7fec8c06b710 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec81ffb700 1 -- 172.21.15.119:0/4128183321 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7c005cc0 con 0x7fec8c06b710 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec880094b0 con 0x7fec8c06b710 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec81ffb700 1 -- 172.21.15.119:0/4128183321 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec7c007310 con 0x7fec8c06b710 2024-09-13T16:25:57.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8c1f2dc0 con 0x7fec8c06b710 2024-09-13T16:25:57.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.189+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8c0ce490 con 0x7fec8c06b710 2024-09-13T16:25:57.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.193+0000 7fec81ffb700 1 -- 172.21.15.119:0/4128183321 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fec7c007470 con 0x7fec8c06b710 2024-09-13T16:25:57.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.193+0000 7fec81ffb700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.193+0000 7fec81ffb700 1 -- 172.21.15.119:0/4128183321 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fec7c083720 con 0x7fec8c06b710 2024-09-13T16:25:57.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.193+0000 7fec90abe700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.193+0000 7fec90abe700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0cd350 tx=0x7fec88008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.197+0000 7fec81ffb700 1 -- 172.21.15.119:0/4128183321 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fec7c050700 con 0x7fec8c06b710 2024-09-13T16:25:57.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.369+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fec8c066600 con 0x7fec8c06b710 2024-09-13T16:25:57.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec81ffb700 1 -- 172.21.15.119:0/4128183321 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fec7c053d20 con 0x7fec8c06b710 2024-09-13T16:25:57.376 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:25:57.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 msgr2=0x7fec70068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0cd350 tx=0x7fec88008030 comp rx=0 tx=0).stop 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec8c06b710 msgr2=0x7fec8c15b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec8c06b710 0x7fec8c15b3c0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fec7c00c3c0 tx=0x7fec7c00c6d0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 shutdown_connections 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec70066730 0x7fec70068bf0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec8c15cfb0 0x7fec8c160240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec8c0d1640 0x7fec8c15fd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 --2- 172.21.15.119:0/4128183321 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec8c06b710 0x7fec8c15b3c0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 >> 172.21.15.119:0/4128183321 conn(0x7fec8c0d2da0 msgr2=0x7fec8c0cef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 shutdown_connections 2024-09-13T16:25:57.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.373+0000 7fec93523700 1 -- 172.21.15.119:0/4128183321 wait complete. 2024-09-13T16:25:57.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:25:57.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9323a3e700 1 Processor -- start 2024-09-13T16:25:57.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9323a3e700 1 -- start start 2024-09-13T16:25:57.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9323a3e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f931c15cfb0 0x7f931c06ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9323a3e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f931c06b2d0 0x7f931c06b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9323a3e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f931c0ce880 con 0x7f931c06bc50 2024-09-13T16:25:57.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9323a3e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c0cee40 con 0x7f931c15cfb0 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9323a3e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f931c0d1220 con 0x7f931c06b2d0 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f93217da700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f931c15cfb0 0x7f931c06ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9321fdb700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f931c06bc50 0x7f931c0cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58796/0 (socket says 172.21.15.119:58796) 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9321fdb700 1 -- 172.21.15.119:0/3926171056 learned_addr learned my addr 172.21.15.119:0/3926171056 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f9320fd9700 1 --2- 172.21.15.119:0/3926171056 >> v2:172.21.15.145:3300/0 conn(0x7f931c06b2d0 0x7f931c06b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f93217da700 1 -- 172.21.15.119:0/3926171056 >> v2:172.21.15.145:3300/0 conn(0x7f931c06b2d0 msgr2=0x7f931c06b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f93217da700 1 --2- 172.21.15.119:0/3926171056 >> v2:172.21.15.145:3300/0 conn(0x7f931c06b2d0 0x7f931c06b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f93217da700 1 -- 172.21.15.119:0/3926171056 >> v1:172.21.15.119:6789/0 conn(0x7f931c06bc50 legacy=0x7f931c0cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.509+0000 7f93217da700 1 -- 172.21.15.119:0/3926171056 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931c0ceb60 con 0x7f931c15cfb0 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f93217da700 1 --2- 172.21.15.119:0/3926171056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c15cfb0 0x7f931c06ad90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f931c0c64b0 tx=0x7f9318009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5fd2927a773b0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9313fff700 1 -- 172.21.15.119:0/3926171056 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318011830 con 0x7f931c15cfb0 2024-09-13T16:25:57.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9313fff700 1 -- 172.21.15.119:0/3926171056 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9318011990 con 0x7f931c15cfb0 2024-09-13T16:25:57.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9313fff700 1 -- 172.21.15.119:0/3926171056 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9318011d20 con 0x7f931c15cfb0 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 -- 172.21.15.119:0/3926171056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c15cfb0 msgr2=0x7f931c06ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 --2- 172.21.15.119:0/3926171056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c15cfb0 0x7f931c06ad90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f931c0c64b0 tx=0x7f9318009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 -- 172.21.15.119:0/3926171056 shutdown_connections 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 --2- 172.21.15.119:0/3926171056 >> v2:172.21.15.145:3300/0 conn(0x7f931c06b2d0 0x7f931c06b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 --2- 172.21.15.119:0/3926171056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c15cfb0 0x7f931c06ad90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 -- 172.21.15.119:0/3926171056 >> 172.21.15.119:0/3926171056 conn(0x7f931c0d2da0 msgr2=0x7f931c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 -- 172.21.15.119:0/3926171056 shutdown_connections 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 -- 172.21.15.119:0/3926171056 wait complete. 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 Processor -- start 2024-09-13T16:25:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 -- start start 2024-09-13T16:25:57.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.513+0000 7f9323a3e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c06b2d0 0x7f931c1f1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f931c06bc50 0x7f931c0b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f931c15cfb0 0x7f931c0b1510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0cbf80 con 0x7f931c06b2d0 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0cc260 con 0x7f931c15cfb0 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f931c0cc540 con 0x7f931c06bc50 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f93217da700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c06b2d0 0x7f931c1f1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f93217da700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c06b2d0 0x7f931c1f1b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43496/0 (socket says 172.21.15.119:43496) 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f93217da700 1 -- 172.21.15.119:0/1491277034 learned_addr learned my addr 172.21.15.119:0/1491277034 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9321fdb700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f931c15cfb0 0x7f931c0b1510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9320fd9700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f931c06bc50 0x7f931c0b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9321fdb700 1 -- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f931c06bc50 msgr2=0x7f931c0b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9321fdb700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f931c06bc50 0x7f931c0b0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9321fdb700 1 -- 172.21.15.119:0/1491277034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c06b2d0 msgr2=0x7f931c1f1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9321fdb700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c06b2d0 0x7f931c1f1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9321fdb700 1 -- 172.21.15.119:0/1491277034 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93180094b0 con 0x7f931c15cfb0 2024-09-13T16:25:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9321fdb700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f931c15cfb0 0x7f931c0b1510 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f931400caa0 tx=0x7f931400cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9311ffb700 1 -- 172.21.15.119:0/1491277034 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314007480 con 0x7f931c15cfb0 2024-09-13T16:25:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9311ffb700 1 -- 172.21.15.119:0/1491277034 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93140075e0 con 0x7f931c15cfb0 2024-09-13T16:25:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f931c0cc820 con 0x7f931c15cfb0 2024-09-13T16:25:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9311ffb700 1 -- 172.21.15.119:0/1491277034 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9314007950 con 0x7f931c15cfb0 2024-09-13T16:25:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f931c0ccd40 con 0x7f931c15cfb0 2024-09-13T16:25:57.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9311ffb700 1 -- 172.21.15.119:0/1491277034 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f931401b3b0 con 0x7f931c15cfb0 2024-09-13T16:25:57.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f931c0cca10 con 0x7f931c15cfb0 2024-09-13T16:25:57.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9311ffb700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9300066730 0x7f9300068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f9311ffb700 1 -- 172.21.15.119:0/1491277034 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9314083b90 con 0x7f931c15cfb0 2024-09-13T16:25:57.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.517+0000 7f93217da700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9300066730 0x7f9300068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.521+0000 7f93217da700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9300066730 0x7f9300068bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f93180097d0 tx=0x7f931801c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.525+0000 7f9311ffb700 1 -- 172.21.15.119:0/1491277034 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9314050c20 con 0x7f931c15cfb0 2024-09-13T16:25:57.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.693+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f931c0cc3a0 con 0x7f931c15cfb0 2024-09-13T16:25:57.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.697+0000 7f9311ffb700 1 -- 172.21.15.119:0/1491277034 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f931c0cc3a0 con 0x7f931c15cfb0 2024-09-13T16:25:57.700 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:57.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9300066730 msgr2=0x7f9300068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9300066730 0x7f9300068bf0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f93180097d0 tx=0x7f931801c040 comp rx=0 tx=0).stop 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f931c15cfb0 msgr2=0x7f931c0b1510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f931c15cfb0 0x7f931c0b1510 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f931400caa0 tx=0x7f931400cad0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 shutdown_connections 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9300066730 0x7f9300068bf0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f931c15cfb0 0x7f931c0b1510 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f931c06bc50 0x7f931c0b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 --2- 172.21.15.119:0/1491277034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f931c06b2d0 0x7f931c1f1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 >> 172.21.15.119:0/1491277034 conn(0x7f931c0d2da0 msgr2=0x7f931c0b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:57.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 shutdown_connections 2024-09-13T16:25:57.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.701+0000 7f9323a3e700 1 -- 172.21.15.119:0/1491277034 wait complete. 2024-09-13T16:25:57.722 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:57.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 Processor -- start 2024-09-13T16:25:57.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 -- start start 2024-09-13T16:25:57.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fad6c146420 0x7fad6c0ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c06ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fad6c06f2a0 0x7fad6c0d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6c0cee20 con 0x7fad6c06f2a0 2024-09-13T16:25:57.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad6c0ad8e0 con 0x7fad6c146420 2024-09-13T16:25:57.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad72773700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c0adea0 con 0x7fad6c0ce9c0 2024-09-13T16:25:57.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fad6c06f2a0 0x7fad6c0d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fad6c06f2a0 0x7fad6c0d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43502/0 (socket says 172.21.15.119:43502) 2024-09-13T16:25:57.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 -- 172.21.15.119:0/2896102426 learned_addr learned my addr 172.21.15.119:0/2896102426 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:57.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71771700 1 --2- 172.21.15.119:0/2896102426 >> v2:172.21.15.145:3300/0 conn(0x7fad6c146420 0x7fad6c0ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad70f70700 1 --2- 172.21.15.119:0/2896102426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c06ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 -- 172.21.15.119:0/2896102426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 msgr2=0x7fad6c06ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:57.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 --2- 172.21.15.119:0/2896102426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c06ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 -- 172.21.15.119:0/2896102426 >> v2:172.21.15.145:3300/0 conn(0x7fad6c146420 msgr2=0x7fad6c0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 --2- 172.21.15.119:0/2896102426 >> v2:172.21.15.145:3300/0 conn(0x7fad6c146420 0x7fad6c0ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 -- 172.21.15.119:0/2896102426 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6c06f750 con 0x7fad6c06f2a0 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad70f70700 1 --2- 172.21.15.119:0/2896102426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c06ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad71f72700 1 --2- 172.21.15.119:0/2896102426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c06f2a0 0x7fad6c0d06d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fad6c067610 tx=0x7fad6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8731506350663bd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad63fff700 1 -- 172.21.15.119:0/2896102426 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400e070 con 0x7fad6c06f2a0 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad63fff700 1 -- 172.21.15.119:0/2896102426 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6400b910 con 0x7fad6c06f2a0 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.829+0000 7fad63fff700 1 -- 172.21.15.119:0/2896102426 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad6400bcc0 con 0x7fad6c06f2a0 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- 172.21.15.119:0/2896102426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c06f2a0 msgr2=0x7fad6c0d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 --2- 172.21.15.119:0/2896102426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c06f2a0 0x7fad6c0d06d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fad6c067610 tx=0x7fad6400b330 comp rx=0 tx=0).stop 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- 172.21.15.119:0/2896102426 shutdown_connections 2024-09-13T16:25:57.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 --2- 172.21.15.119:0/2896102426 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c06f2a0 0x7fad6c0d06d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 --2- 172.21.15.119:0/2896102426 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c06ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 --2- 172.21.15.119:0/2896102426 >> v2:172.21.15.145:3300/0 conn(0x7fad6c146420 0x7fad6c0ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- 172.21.15.119:0/2896102426 >> 172.21.15.119:0/2896102426 conn(0x7fad6c099a50 msgr2=0x7fad6c099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:57.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- 172.21.15.119:0/2896102426 shutdown_connections 2024-09-13T16:25:57.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- 172.21.15.119:0/2896102426 wait complete. 2024-09-13T16:25:57.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 Processor -- start 2024-09-13T16:25:57.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- start start 2024-09-13T16:25:57.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad6c06f2a0 0x7fad6c1437b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c15f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c146420 0x7fad6c09c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c09b3d0 con 0x7fad6c146420 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c09b610 con 0x7fad6c06f2a0 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad72773700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad6c09b8f0 con 0x7fad6c0ce9c0 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.833+0000 7fad71f72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c146420 0x7fad6c09c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71771700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad6c06f2a0 0x7fad6c1437b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71771700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad6c06f2a0 0x7fad6c1437b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:38490/0 (socket says 172.21.15.119:38490) 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71771700 1 -- 172.21.15.119:0/2997648443 learned_addr learned my addr 172.21.15.119:0/2997648443 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71f72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c146420 0x7fad6c09c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43514/0 (socket says 172.21.15.119:43514) 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad70f70700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c15f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71f72700 1 -- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 msgr2=0x7fad6c15f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71f72700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c15f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71f72700 1 -- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad6c06f2a0 msgr2=0x7fad6c1437b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:57.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71f72700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad6c06f2a0 0x7fad6c1437b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:57.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71f72700 1 -- 172.21.15.119:0/2997648443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad64009b30 con 0x7fad6c146420 2024-09-13T16:25:57.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad71f72700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c146420 0x7fad6c09c3d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad64015000 tx=0x7fad64004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad61ffb700 1 -- 172.21.15.119:0/2997648443 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad640099b0 con 0x7fad6c146420 2024-09-13T16:25:57.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad61ffb700 1 -- 172.21.15.119:0/2997648443 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad6400ce80 con 0x7fad6c146420 2024-09-13T16:25:57.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad6c09bbd0 con 0x7fad6c146420 2024-09-13T16:25:57.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad61ffb700 1 -- 172.21.15.119:0/2997648443 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad64004490 con 0x7fad6c146420 2024-09-13T16:25:57.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad6c143cf0 con 0x7fad6c146420 2024-09-13T16:25:57.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.837+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad6c09c0f0 con 0x7fad6c146420 2024-09-13T16:25:57.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.841+0000 7fad61ffb700 1 -- 172.21.15.119:0/2997648443 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fad640045f0 con 0x7fad6c146420 2024-09-13T16:25:57.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.841+0000 7fad61ffb700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad50066760 0x7fad50068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:57.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.841+0000 7fad61ffb700 1 -- 172.21.15.119:0/2997648443 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fad64085620 con 0x7fad6c146420 2024-09-13T16:25:57.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.841+0000 7fad71771700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad50066760 0x7fad50068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:57.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.841+0000 7fad71771700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad50066760 0x7fad50068c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad68009c30 tx=0x7fad68009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:57.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:57.845+0000 7fad61ffb700 1 -- 172.21.15.119:0/2997648443 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fad6c09c0f0 con 0x7fad6c146420 2024-09-13T16:25:58.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.021+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad6c09c0f0 con 0x7fad6c146420 2024-09-13T16:25:58.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.025+0000 7fad61ffb700 1 -- 172.21.15.119:0/2997648443 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7fad6c09c0f0 con 0x7fad6c146420 2024-09-13T16:25:58.030 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:58.030 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad50066760 msgr2=0x7fad50068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad50066760 0x7fad50068c20 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fad68009c30 tx=0x7fad68009240 comp rx=0 tx=0).stop 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c146420 msgr2=0x7fad6c09c3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c146420 0x7fad6c09c3d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fad64015000 tx=0x7fad64004af0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 shutdown_connections 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad50066760 0x7fad50068c20 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad6c146420 0x7fad6c09c3d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad6c0ce9c0 0x7fad6c15f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 --2- 172.21.15.119:0/2997648443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad6c06f2a0 0x7fad6c1437b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 >> 172.21.15.119:0/2997648443 conn(0x7fad6c099a50 msgr2=0x7fad6c0b3da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:58.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 shutdown_connections 2024-09-13T16:25:58.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.029+0000 7fad72773700 1 -- 172.21.15.119:0/2997648443 wait complete. 2024-09-13T16:25:58.045 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:25:58.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:58.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.157+0000 7f96e8a9a700 1 Processor -- start 2024-09-13T16:25:58.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.157+0000 7f96e8a9a700 1 -- start start 2024-09-13T16:25:58.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f96e4144180 0x7f96e40cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f96e40ce240 0x7f96e406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96e40b2720 con 0x7f96e40b22b0 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e40b1fe0 con 0x7f96e40ce240 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96e40b29b0 con 0x7f96e4144180 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f96e40ce240 0x7f96e406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f96e40ce240 0x7f96e406bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43524/0 (socket says 172.21.15.119:43524) 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 -- 172.21.15.119:0/717971904 learned_addr learned my addr 172.21.15.119:0/717971904 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e37fe700 1 --2- 172.21.15.119:0/717971904 >> v2:172.21.15.145:3300/0 conn(0x7f96e4144180 0x7f96e40cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 -- 172.21.15.119:0/717971904 >> v2:172.21.15.145:3300/0 conn(0x7f96e4144180 msgr2=0x7f96e40cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 --2- 172.21.15.119:0/717971904 >> v2:172.21.15.145:3300/0 conn(0x7f96e4144180 0x7f96e40cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 -- 172.21.15.119:0/717971904 >> v1:172.21.15.145:6789/0 conn(0x7f96e40b22b0 legacy=0x7f96e40cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 -- 172.21.15.119:0/717971904 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96e40b2c90 con 0x7f96e40ce240 2024-09-13T16:25:58.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e3fff700 1 --2- 172.21.15.119:0/717971904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40ce240 0x7f96e406bb60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f96e406de50 tx=0x7f96dc009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f487bcfc24fac47 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e27fc700 1 -- 172.21.15.119:0/717971904 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc011830 con 0x7f96e40ce240 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e27fc700 1 -- 172.21.15.119:0/717971904 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96dc011990 con 0x7f96e40ce240 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e27fc700 1 -- 172.21.15.119:0/717971904 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96dc011d20 con 0x7f96e40ce240 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- 172.21.15.119:0/717971904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40ce240 msgr2=0x7f96e406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/717971904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40ce240 0x7f96e406bb60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f96e406de50 tx=0x7f96dc009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- 172.21.15.119:0/717971904 shutdown_connections 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/717971904 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40ce240 0x7f96e406bb60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/717971904 >> v2:172.21.15.145:3300/0 conn(0x7f96e4144180 0x7f96e40cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- 172.21.15.119:0/717971904 >> 172.21.15.119:0/717971904 conn(0x7f96e40b0980 msgr2=0x7f96e40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- 172.21.15.119:0/717971904 shutdown_connections 2024-09-13T16:25:58.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.161+0000 7f96e8a9a700 1 -- 172.21.15.119:0/717971904 wait complete. 2024-09-13T16:25:58.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 Processor -- start 2024-09-13T16:25:58.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 -- start start 2024-09-13T16:25:58.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40b22b0 0x7f96e415bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96e40ce240 0x7f96e415c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96e4144180 0x7f96e41354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e415aaf0 con 0x7f96e40b22b0 2024-09-13T16:25:58.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96e4135150 con 0x7f96e40ce240 2024-09-13T16:25:58.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96e40c92b0 con 0x7f96e4144180 2024-09-13T16:25:58.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40b22b0 0x7f96e415bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e2ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96e40ce240 0x7f96e415c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e2ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96e40ce240 0x7f96e415c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:38510/0 (socket says 172.21.15.119:38510) 2024-09-13T16:25:58.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e2ffd700 1 -- 172.21.15.119:0/2728221121 learned_addr learned my addr 172.21.15.119:0/2728221121 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:58.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e3fff700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96e4144180 0x7f96e41354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e37fe700 1 -- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96e4144180 msgr2=0x7f96e41354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e37fe700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96e4144180 0x7f96e41354a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e37fe700 1 -- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96e40ce240 msgr2=0x7f96e415c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e37fe700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96e40ce240 0x7f96e415c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e37fe700 1 -- 172.21.15.119:0/2728221121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96dc009570 con 0x7f96e40b22b0 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e2ffd700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96e40ce240 0x7f96e415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e37fe700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40b22b0 0x7f96e415bfe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00e900 tx=0x7f96cc00ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96dbfff700 1 -- 172.21.15.119:0/2728221121 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc013680 con 0x7f96e40b22b0 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96dbfff700 1 -- 172.21.15.119:0/2728221121 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96cc0137e0 con 0x7f96e40b22b0 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96e40c9590 con 0x7f96e40b22b0 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96dbfff700 1 -- 172.21.15.119:0/2728221121 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc013b50 con 0x7f96e40b22b0 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.165+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96e40c8ed0 con 0x7f96e40b22b0 2024-09-13T16:25:58.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.169+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96e40cc2b0 con 0x7f96e40b22b0 2024-09-13T16:25:58.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.169+0000 7f96dbfff700 1 -- 172.21.15.119:0/2728221121 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f96cc013cb0 con 0x7f96e40b22b0 2024-09-13T16:25:58.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.169+0000 7f96dbfff700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f96c4066730 0x7f96c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.169+0000 7f96dbfff700 1 -- 172.21.15.119:0/2728221121 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f96cc084a70 con 0x7f96e40b22b0 2024-09-13T16:25:58.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.173+0000 7f96e2ffd700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f96c4066730 0x7f96c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.173+0000 7f96e2ffd700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f96c4066730 0x7f96c4068bf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f96e41f60f0 tx=0x7f96d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.173+0000 7f96dbfff700 1 -- 172.21.15.119:0/2728221121 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f96cc051b40 con 0x7f96e40b22b0 2024-09-13T16:25:58.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.341+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f96e40adbf0 con 0x7f96e40b22b0 2024-09-13T16:25:58.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.341+0000 7f96dbfff700 1 -- 172.21.15.119:0/2728221121 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f96cc017020 con 0x7f96e40b22b0 2024-09-13T16:25:58.346 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:58.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f96c4066730 msgr2=0x7f96c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f96c4066730 0x7f96c4068bf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f96e41f60f0 tx=0x7f96d4009040 comp rx=0 tx=0).stop 2024-09-13T16:25:58.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40b22b0 msgr2=0x7f96e415bfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40b22b0 0x7f96e415bfe0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f96cc00e900 tx=0x7f96cc00ec10 comp rx=0 tx=0).stop 2024-09-13T16:25:58.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 shutdown_connections 2024-09-13T16:25:58.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f96c4066730 0x7f96c4068bf0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f96e4144180 0x7f96e41354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f96e40ce240 0x7f96e415c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 --2- 172.21.15.119:0/2728221121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f96e40b22b0 0x7f96e415bfe0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 >> 172.21.15.119:0/2728221121 conn(0x7f96e40b0980 msgr2=0x7f96e40acbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:58.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 shutdown_connections 2024-09-13T16:25:58.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.345+0000 7f96e8a9a700 1 -- 172.21.15.119:0/2728221121 wait complete. 2024-09-13T16:25:58.360 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:25:58.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.473+0000 7f0a0a4f3700 1 Processor -- start 2024-09-13T16:25:58.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.473+0000 7f0a0a4f3700 1 -- start start 2024-09-13T16:25:58.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a0a4f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a040b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a0a4f3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0a04159ca0 0x7f0a04070630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a0a4f3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a040ae340 con 0x7f0a0415a0e0 2024-09-13T16:25:58.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a0a4f3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a0415ae40 con 0x7f0a04159ca0 2024-09-13T16:25:58.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a0a4f3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0415b1d0 con 0x7f0a040adf60 2024-09-13T16:25:58.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a08a90700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0a0415a0e0 0x7f0a0406cc40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58800/0 (socket says 172.21.15.119:58800) 2024-09-13T16:25:58.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a08a90700 1 -- 172.21.15.119:0/1523224698 learned_addr learned my addr 172.21.15.119:0/1523224698 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:58.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a037fe700 1 --2- 172.21.15.119:0/1523224698 >> v2:172.21.15.145:3300/0 conn(0x7f0a04159ca0 0x7f0a04070630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a03fff700 1 --2- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a040b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a02ffd700 1 -- 172.21.15.119:0/1523224698 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664345129 0 0) 0x7f0a040ae340 con 0x7f0a0415a0e0 2024-09-13T16:25:58.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a02ffd700 1 -- 172.21.15.119:0/1523224698 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f00035d0 con 0x7f0a0415a0e0 2024-09-13T16:25:58.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a037fe700 1 -- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 msgr2=0x7f0a040b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:58.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a037fe700 1 --2- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a040b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a037fe700 1 -- 172.21.15.119:0/1523224698 >> v1:172.21.15.119:6789/0 conn(0x7f0a0415a0e0 legacy=0x7f0a0406cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a02ffd700 1 -- 172.21.15.119:0/1523224698 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3514318151 0 0) 0x7f09f00035d0 con 0x7f0a0415a0e0 2024-09-13T16:25:58.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a037fe700 1 -- 172.21.15.119:0/1523224698 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a0415a5a0 con 0x7f0a04159ca0 2024-09-13T16:25:58.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a037fe700 1 --2- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a04159ca0 0x7f0a04070630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009850 tx=0x7f09ec009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0e9ee01186cb612 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a02ffd700 1 -- 172.21.15.119:0/1523224698 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec011830 con 0x7f0a04159ca0 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a02ffd700 1 -- 172.21.15.119:0/1523224698 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec011990 con 0x7f0a04159ca0 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.477+0000 7f0a02ffd700 1 -- 172.21.15.119:0/1523224698 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec011d00 con 0x7f0a04159ca0 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a04159ca0 msgr2=0x7f0a04070630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a04159ca0 0x7f0a04070630 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009850 tx=0x7f09ec009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/1523224698 shutdown_connections 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a04159ca0 0x7f0a04070630 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/1523224698 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a040b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/1523224698 >> 172.21.15.119:0/1523224698 conn(0x7f0a040b0560 msgr2=0x7f0a040b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:58.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/1523224698 shutdown_connections 2024-09-13T16:25:58.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/1523224698 wait complete. 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 Processor -- start 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- start start 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a041525a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a04159ca0 0x7f0a040cd860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a0415a0e0 0x7f0a040ae8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040c57f0 con 0x7f0a04159ca0 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040c4750 con 0x7f0a0415a0e0 2024-09-13T16:25:58.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a0a4f3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a040c4a30 con 0x7f0a040adf60 2024-09-13T16:25:58.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a04159ca0 0x7f0a040cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a04159ca0 0x7f0a040cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43542/0 (socket says 172.21.15.119:43542) 2024-09-13T16:25:58.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a037fe700 1 -- 172.21.15.119:0/528090482 learned_addr learned my addr 172.21.15.119:0/528090482 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:58.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a03fff700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a041525a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.481+0000 7f0a08a90700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a0415a0e0 0x7f0a040ae8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a037fe700 1 -- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 msgr2=0x7f0a041525a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a037fe700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a041525a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a037fe700 1 -- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a0415a0e0 msgr2=0x7f0a040ae8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a037fe700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a0415a0e0 0x7f0a040ae8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a037fe700 1 -- 172.21.15.119:0/528090482 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09ec0094b0 con 0x7f0a04159ca0 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a037fe700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a04159ca0 0x7f0a040cd860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0090b0 tx=0x7f09ec004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a00ff9700 1 -- 172.21.15.119:0/528090482 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec004380 con 0x7f0a04159ca0 2024-09-13T16:25:58.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a00ff9700 1 -- 172.21.15.119:0/528090482 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09ec01f070 con 0x7f0a04159ca0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a040c4d10 con 0x7f0a04159ca0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a040c52d0 con 0x7f0a04159ca0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a00ff9700 1 -- 172.21.15.119:0/528090482 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09ec0036c0 con 0x7f0a04159ca0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a040b3450 con 0x7f0a04159ca0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.485+0000 7f0a00ff9700 1 -- 172.21.15.119:0/528090482 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f09ec021030 con 0x7f0a04159ca0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.489+0000 7f0a00ff9700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f09e4066730 0x7f09e4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.489+0000 7f0a00ff9700 1 -- 172.21.15.119:0/528090482 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f09ec0857e0 con 0x7f0a04159ca0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.489+0000 7f0a03fff700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f09e4066730 0x7f09e4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.489+0000 7f0a03fff700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f09e4066730 0x7f09e4068bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009bf0 tx=0x7f09f4009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.497+0000 7f0a00ff9700 1 -- 172.21.15.119:0/528090482 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f09ec052870 con 0x7f0a04159ca0 2024-09-13T16:25:58.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.669+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0a0415d140 con 0x7f0a04159ca0 2024-09-13T16:25:58.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a00ff9700 1 -- 172.21.15.119:0/528090482 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f09ec055e90 con 0x7f0a04159ca0 2024-09-13T16:25:58.675 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:58.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f09e4066730 msgr2=0x7f09e4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f09e4066730 0x7f09e4068bf0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f09f4009bf0 tx=0x7f09f4009240 comp rx=0 tx=0).stop 2024-09-13T16:25:58.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a04159ca0 msgr2=0x7f0a040cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a04159ca0 0x7f0a040cd860 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f09ec0090b0 tx=0x7f09ec004770 comp rx=0 tx=0).stop 2024-09-13T16:25:58.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 shutdown_connections 2024-09-13T16:25:58.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f09e4066730 0x7f09e4068bf0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a0415a0e0 0x7f0a040ae8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a04159ca0 0x7f0a040cd860 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 --2- 172.21.15.119:0/528090482 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a040adf60 0x7f0a041525a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 >> 172.21.15.119:0/528090482 conn(0x7f0a040b0560 msgr2=0x7f0a040b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:58.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 shutdown_connections 2024-09-13T16:25:58.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.673+0000 7f0a0a4f3700 1 -- 172.21.15.119:0/528090482 wait complete. 2024-09-13T16:25:58.690 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:25:58.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 Processor -- start 2024-09-13T16:25:58.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 -- start start 2024-09-13T16:25:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0ef015b160 0x7f0ef006d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0ef0161d20 0x7f0ef006a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ef01621b0 con 0x7f0ef006a890 2024-09-13T16:25:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef006ad40 con 0x7f0ef015b160 2024-09-13T16:25:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ef0147330 con 0x7f0ef0161d20 2024-09-13T16:25:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0ef015b160 0x7f0ef006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0ef015b160 0x7f0ef006d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43556/0 (socket says 172.21.15.119:43556) 2024-09-13T16:25:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 -- 172.21.15.119:0/2359538046 learned_addr learned my addr 172.21.15.119:0/2359538046 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eedd9b700 1 --2- 172.21.15.119:0/2359538046 >> v2:172.21.15.145:3300/0 conn(0x7f0ef0161d20 0x7f0ef006a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 -- 172.21.15.119:0/2359538046 >> v2:172.21.15.145:3300/0 conn(0x7f0ef0161d20 msgr2=0x7f0ef006a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 --2- 172.21.15.119:0/2359538046 >> v2:172.21.15.145:3300/0 conn(0x7f0ef0161d20 0x7f0ef006a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 -- 172.21.15.119:0/2359538046 >> v1:172.21.15.145:6789/0 conn(0x7f0ef006a890 legacy=0x7f0ef00ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 -- 172.21.15.119:0/2359538046 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef006dc70 con 0x7f0ef015b160 2024-09-13T16:25:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eee59c700 1 --2- 172.21.15.119:0/2359538046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef015b160 0x7f0ef006d570 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0161220 tx=0x7f0ee4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d30c9242707db7b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eed59a700 1 -- 172.21.15.119:0/2359538046 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4011830 con 0x7f0ef015b160 2024-09-13T16:25:58.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eed59a700 1 -- 172.21.15.119:0/2359538046 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee4011990 con 0x7f0ef015b160 2024-09-13T16:25:58.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eed59a700 1 -- 172.21.15.119:0/2359538046 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4011d20 con 0x7f0ef015b160 2024-09-13T16:25:58.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 -- 172.21.15.119:0/2359538046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef015b160 msgr2=0x7f0ef006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.801+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2359538046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef015b160 0x7f0ef006d570 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0161220 tx=0x7f0ee4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- 172.21.15.119:0/2359538046 shutdown_connections 2024-09-13T16:25:58.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2359538046 >> v2:172.21.15.145:3300/0 conn(0x7f0ef0161d20 0x7f0ef006a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2359538046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef015b160 0x7f0ef006d570 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- 172.21.15.119:0/2359538046 >> 172.21.15.119:0/2359538046 conn(0x7f0ef006b460 msgr2=0x7f0ef006b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:58.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- 172.21.15.119:0/2359538046 shutdown_connections 2024-09-13T16:25:58.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- 172.21.15.119:0/2359538046 wait complete. 2024-09-13T16:25:58.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 Processor -- start 2024-09-13T16:25:58.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- start start 2024-09-13T16:25:58.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef006a890 0x7f0ef0145e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ef015b160 0x7f0ef01e9790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ef0161d20 0x7f0ef015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef00b3f20 con 0x7f0ef006a890 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef0149650 con 0x7f0ef015b160 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.805+0000 7f0eef59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ef01463d0 con 0x7f0ef0161d20 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef006a890 0x7f0ef0145e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef006a890 0x7f0ef0145e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43566/0 (socket says 172.21.15.119:43566) 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 -- 172.21.15.119:0/2603723356 learned_addr learned my addr 172.21.15.119:0/2603723356 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eeed9d700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ef0161d20 0x7f0ef015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eedd9b700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ef015b160 0x7f0ef01e9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 -- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ef0161d20 msgr2=0x7f0ef015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ef0161d20 0x7f0ef015ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 -- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ef015b160 msgr2=0x7f0ef01e9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ef015b160 0x7f0ef01e9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 -- 172.21.15.119:0/2603723356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee4009570 con 0x7f0ef006a890 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eedd9b700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ef015b160 0x7f0ef01e9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eee59c700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef006a890 0x7f0ef0145e90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009f40 tx=0x7f0ee4004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0edeffd700 1 -- 172.21.15.119:0/2603723356 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4004550 con 0x7f0ef006a890 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0edeffd700 1 -- 172.21.15.119:0/2603723356 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee401f070 con 0x7f0ef006a890 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ef00b3c40 con 0x7f0ef006a890 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0edeffd700 1 -- 172.21.15.119:0/2603723356 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4003900 con 0x7f0ef006a890 2024-09-13T16:25:58.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ef01492e0 con 0x7f0ef006a890 2024-09-13T16:25:58.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.809+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ef014a820 con 0x7f0ef006a890 2024-09-13T16:25:58.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.813+0000 7f0edeffd700 1 -- 172.21.15.119:0/2603723356 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0ee4003a60 con 0x7f0ef006a890 2024-09-13T16:25:58.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.813+0000 7f0edeffd700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ed0066780 0x7f0ed0068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:58.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.813+0000 7f0edeffd700 1 -- 172.21.15.119:0/2603723356 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0ee4021070 con 0x7f0ef006a890 2024-09-13T16:25:58.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.813+0000 7f0eedd9b700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ed0066780 0x7f0ed0068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:58.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.813+0000 7f0eedd9b700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ed0066780 0x7f0ed0068c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00b40f0 tx=0x7f0ee000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:58.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.817+0000 7f0edeffd700 1 -- 172.21.15.119:0/2603723356 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0ee40529a0 con 0x7f0ef006a890 2024-09-13T16:25:58.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.981+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0ef00afd40 con 0x7f0ef006a890 2024-09-13T16:25:58.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0edeffd700 1 -- 172.21.15.119:0/2603723356 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0ee4055fc0 con 0x7f0ef006a890 2024-09-13T16:25:58.988 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ed0066780 msgr2=0x7f0ed0068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ed0066780 0x7f0ed0068c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00b40f0 tx=0x7f0ee000b040 comp rx=0 tx=0).stop 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef006a890 msgr2=0x7f0ef0145e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef006a890 0x7f0ef0145e90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f0ee4009f40 tx=0x7f0ee4004770 comp rx=0 tx=0).stop 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 shutdown_connections 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ed0066780 0x7f0ed0068c40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ef0161d20 0x7f0ef015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ef015b160 0x7f0ef01e9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 --2- 172.21.15.119:0/2603723356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ef006a890 0x7f0ef0145e90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:58.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 >> 172.21.15.119:0/2603723356 conn(0x7f0ef006b460 msgr2=0x7f0ef00af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:58.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 shutdown_connections 2024-09-13T16:25:58.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:58.985+0000 7f0eef59e700 1 -- 172.21.15.119:0/2603723356 wait complete. 2024-09-13T16:25:59.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:25:59.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb08c34700 1 Processor -- start 2024-09-13T16:25:59.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb08c34700 1 -- start start 2024-09-13T16:25:59.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb08c34700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffb0415cad0 0x7ffb040d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb08c34700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffb040cf930 0x7ffb040ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb08c34700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb040d05b0 con 0x7ffb040ce0d0 2024-09-13T16:25:59.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb08c34700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb040d0870 con 0x7ffb040cf930 2024-09-13T16:25:59.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb08c34700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb040d0b50 con 0x7ffb0415cad0 2024-09-13T16:25:59.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffb040cf930 0x7ffb040ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffb040cf930 0x7ffb040ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43568/0 (socket says 172.21.15.119:43568) 2024-09-13T16:25:59.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 -- 172.21.15.119:0/677440443 learned_addr learned my addr 172.21.15.119:0/677440443 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:59.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb037fe700 1 --2- 172.21.15.119:0/677440443 >> v2:172.21.15.145:3300/0 conn(0x7ffb0415cad0 0x7ffb040d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 -- 172.21.15.119:0/677440443 >> v2:172.21.15.145:3300/0 conn(0x7ffb0415cad0 msgr2=0x7ffb040d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:59.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 --2- 172.21.15.119:0/677440443 >> v2:172.21.15.145:3300/0 conn(0x7ffb0415cad0 0x7ffb040d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 -- 172.21.15.119:0/677440443 >> v1:172.21.15.145:6789/0 conn(0x7ffb040ce0d0 legacy=0x7ffb040cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 -- 172.21.15.119:0/677440443 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb040d0e30 con 0x7ffb040cf930 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb03fff700 1 --2- 172.21.15.119:0/677440443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040cf930 0x7ffb040ce510 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffb040b55b0 tx=0x7ffafc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21ed633a486602dd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.113+0000 7ffb027fc700 1 -- 172.21.15.119:0/677440443 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc00e070 con 0x7ffb040cf930 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb027fc700 1 -- 172.21.15.119:0/677440443 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffafc00b9c0 con 0x7ffb040cf930 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb027fc700 1 -- 172.21.15.119:0/677440443 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffafc00bd70 con 0x7ffb040cf930 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 -- 172.21.15.119:0/677440443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040cf930 msgr2=0x7ffb040ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 --2- 172.21.15.119:0/677440443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040cf930 0x7ffb040ce510 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7ffb040b55b0 tx=0x7ffafc00b330 comp rx=0 tx=0).stop 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 -- 172.21.15.119:0/677440443 shutdown_connections 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 --2- 172.21.15.119:0/677440443 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040cf930 0x7ffb040ce510 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 --2- 172.21.15.119:0/677440443 >> v2:172.21.15.145:3300/0 conn(0x7ffb0415cad0 0x7ffb040d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 -- 172.21.15.119:0/677440443 >> 172.21.15.119:0/677440443 conn(0x7ffb0408bf70 msgr2=0x7ffb040d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:59.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 -- 172.21.15.119:0/677440443 shutdown_connections 2024-09-13T16:25:59.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 -- 172.21.15.119:0/677440443 wait complete. 2024-09-13T16:25:59.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 Processor -- start 2024-09-13T16:25:59.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.117+0000 7ffb08c34700 1 -- start start 2024-09-13T16:25:59.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040ce0d0 0x7ffb041f3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb040cf930 0x7ffb040c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb0415cad0 0x7ffb040c8a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040c5ff0 con 0x7ffb040ce0d0 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040c6890 con 0x7ffb0415cad0 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb040cc0f0 con 0x7ffb040cf930 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040ce0d0 0x7ffb041f3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040ce0d0 0x7ffb041f3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43572/0 (socket says 172.21.15.119:43572) 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 -- 172.21.15.119:0/3580562007 learned_addr learned my addr 172.21.15.119:0/3580562007 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb03fff700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb0415cad0 0x7ffb040c8a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb02ffd700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb040cf930 0x7ffb040c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 -- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb040cf930 msgr2=0x7ffb040c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb040cf930 0x7ffb040c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 -- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb0415cad0 msgr2=0x7ffb040c8a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb0415cad0 0x7ffb040c8a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 -- 172.21.15.119:0/3580562007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffafc009b30 con 0x7ffb040ce0d0 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb03fff700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb0415cad0 0x7ffb040c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb037fe700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040ce0d0 0x7ffb041f3e50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800d800 tx=0x7ffaf800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffaebfff700 1 -- 172.21.15.119:0/3580562007 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80125f0 con 0x7ffb040ce0d0 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffaebfff700 1 -- 172.21.15.119:0/3580562007 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffaf8012750 con 0x7ffb040ce0d0 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb040c6240 con 0x7ffb040ce0d0 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffaebfff700 1 -- 172.21.15.119:0/3580562007 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8012b00 con 0x7ffb040ce0d0 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.121+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb041f5d20 con 0x7ffb040ce0d0 2024-09-13T16:25:59.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.125+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb040ce510 con 0x7ffb040ce0d0 2024-09-13T16:25:59.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.125+0000 7ffaebfff700 1 -- 172.21.15.119:0/3580562007 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ffaf8012c60 con 0x7ffb040ce0d0 2024-09-13T16:25:59.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.125+0000 7ffaebfff700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffaf0066730 0x7ffaf0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.125+0000 7ffaebfff700 1 -- 172.21.15.119:0/3580562007 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7ffaf8084880 con 0x7ffb040ce0d0 2024-09-13T16:25:59.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.125+0000 7ffb02ffd700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffaf0066730 0x7ffaf0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.125+0000 7ffb02ffd700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffaf0066730 0x7ffaf0068bf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffb0415fa90 tx=0x7ffaf4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.133+0000 7ffaebfff700 1 -- 172.21.15.119:0/3580562007 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ffaf80518f0 con 0x7ffb040ce0d0 2024-09-13T16:25:59.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.301+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ffb040d38f0 con 0x7ffb040ce0d0 2024-09-13T16:25:59.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffaebfff700 1 -- 172.21.15.119:0/3580562007 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7ffaf8054f10 con 0x7ffb040ce0d0 2024-09-13T16:25:59.308 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:25:59.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffaf0066730 msgr2=0x7ffaf0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffaf0066730 0x7ffaf0068bf0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ffb0415fa90 tx=0x7ffaf4009040 comp rx=0 tx=0).stop 2024-09-13T16:25:59.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040ce0d0 msgr2=0x7ffb041f3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040ce0d0 0x7ffb041f3e50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800d800 tx=0x7ffaf800dbc0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 shutdown_connections 2024-09-13T16:25:59.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffaf0066730 0x7ffaf0068bf0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb0415cad0 0x7ffb040c8a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb040cf930 0x7ffb040c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 --2- 172.21.15.119:0/3580562007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb040ce0d0 0x7ffb041f3e50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 >> 172.21.15.119:0/3580562007 conn(0x7ffb0408bf70 msgr2=0x7ffb04066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:59.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 shutdown_connections 2024-09-13T16:25:59.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.305+0000 7ffb08c34700 1 -- 172.21.15.119:0/3580562007 wait complete. 2024-09-13T16:25:59.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:25:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.425+0000 7fd44e012700 1 Processor -- start 2024-09-13T16:25:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.425+0000 7fd44e012700 1 -- start start 2024-09-13T16:25:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4481509e0 0x7fd448159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd44815a910 0x7fd4480ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4480b44e0 con 0x7fd4480cc220 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4480cddf0 con 0x7fd44815a910 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd448071a50 con 0x7fd4481509e0 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44c80f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd4480cc220 0x7fd44815a200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58814/0 (socket says 172.21.15.119:58814) 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44c80f700 1 -- 172.21.15.119:0/3416098743 learned_addr learned my addr 172.21.15.119:0/3416098743 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d010700 1 --2- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4481509e0 0x7fd448159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d811700 1 --2- 172.21.15.119:0/3416098743 >> v2:172.21.15.145:3300/0 conn(0x7fd44815a910 0x7fd4480ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd447fff700 1 -- 172.21.15.119:0/3416098743 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367054259 0 0) 0x7fd4480b44e0 con 0x7fd4480cc220 2024-09-13T16:25:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd447fff700 1 -- 172.21.15.119:0/3416098743 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4300035d0 con 0x7fd4480cc220 2024-09-13T16:25:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d811700 1 -- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4481509e0 msgr2=0x7fd448159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d811700 1 --2- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4481509e0 0x7fd448159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d811700 1 -- 172.21.15.119:0/3416098743 >> v1:172.21.15.119:6789/0 conn(0x7fd4480cc220 legacy=0x7fd44815a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d811700 1 -- 172.21.15.119:0/3416098743 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4480cc750 con 0x7fd44815a910 2024-09-13T16:25:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d010700 1 --2- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4481509e0 0x7fd448159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44d811700 1 --2- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd44815a910 0x7fd4480ce2d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd440009850 tx=0x7fd440009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90da26fa01039264 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd447fff700 1 -- 172.21.15.119:0/3416098743 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440011830 con 0x7fd44815a910 2024-09-13T16:25:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd447fff700 1 -- 172.21.15.119:0/3416098743 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd440011990 con 0x7fd44815a910 2024-09-13T16:25:59.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd447fff700 1 -- 172.21.15.119:0/3416098743 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440011d00 con 0x7fd44815a910 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd44815a910 msgr2=0x7fd4480ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 --2- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd44815a910 0x7fd4480ce2d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd440009850 tx=0x7fd440009b60 comp rx=0 tx=0).stop 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- 172.21.15.119:0/3416098743 shutdown_connections 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 --2- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd44815a910 0x7fd4480ce2d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 --2- 172.21.15.119:0/3416098743 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4481509e0 0x7fd448159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- 172.21.15.119:0/3416098743 >> 172.21.15.119:0/3416098743 conn(0x7fd448070cc0 msgr2=0x7fd4480710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- 172.21.15.119:0/3416098743 shutdown_connections 2024-09-13T16:25:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.429+0000 7fd44e012700 1 -- 172.21.15.119:0/3416098743 wait complete. 2024-09-13T16:25:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 Processor -- start 2024-09-13T16:25:59.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 -- start start 2024-09-13T16:25:59.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4480cc220 0x7fd448147730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4481509e0 0x7fd4481568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd44815a910 0x7fd448156e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd448154370 con 0x7fd4481509e0 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd448152d00 con 0x7fd4480cc220 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44e012700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd448154630 con 0x7fd44815a910 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44c80f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4481509e0 0x7fd4481568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44c80f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4481509e0 0x7fd4481568f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43586/0 (socket says 172.21.15.119:43586) 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44d811700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd44815a910 0x7fd448156e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44c80f700 1 -- 172.21.15.119:0/938433474 learned_addr learned my addr 172.21.15.119:0/938433474 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:59.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.433+0000 7fd44d010700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4480cc220 0x7fd448147730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44d811700 1 -- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4480cc220 msgr2=0x7fd448147730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44d811700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4480cc220 0x7fd448147730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44d811700 1 -- 172.21.15.119:0/938433474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4481509e0 msgr2=0x7fd4481568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44d811700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4481509e0 0x7fd4481568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44d811700 1 -- 172.21.15.119:0/938433474 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43800bc80 con 0x7fd44815a910 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44c80f700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4481509e0 0x7fd4481568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44d811700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd44815a910 0x7fd448156e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd440016000 tx=0x7fd440004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd445ffb700 1 -- 172.21.15.119:0/938433474 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440004590 con 0x7fd44815a910 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4400094b0 con 0x7fd44815a910 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd445ffb700 1 -- 172.21.15.119:0/938433474 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4400036c0 con 0x7fd44815a910 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd445ffb700 1 -- 172.21.15.119:0/938433474 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440003a30 con 0x7fd44815a910 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd448147db0 con 0x7fd44815a910 2024-09-13T16:25:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.437+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd448154b50 con 0x7fd44815a910 2024-09-13T16:25:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.441+0000 7fd445ffb700 1 -- 172.21.15.119:0/938433474 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd440003b90 con 0x7fd44815a910 2024-09-13T16:25:59.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.441+0000 7fd445ffb700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd434066730 0x7fd434068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.441+0000 7fd445ffb700 1 -- 172.21.15.119:0/938433474 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fd440021030 con 0x7fd44815a910 2024-09-13T16:25:59.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.441+0000 7fd44d010700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd434066730 0x7fd434068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.441+0000 7fd44d010700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd434066730 0x7fd434068bf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd438003fb0 tx=0x7fd43800d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.449+0000 7fd445ffb700 1 -- 172.21.15.119:0/938433474 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd440052cb0 con 0x7fd44815a910 2024-09-13T16:25:59.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.621+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd448154b50 con 0x7fd44815a910 2024-09-13T16:25:59.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.625+0000 7fd445ffb700 1 -- 172.21.15.119:0/938433474 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7fd448154b50 con 0x7fd44815a910 2024-09-13T16:25:59.630 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:25:59.631 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd434066730 msgr2=0x7fd434068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd434066730 0x7fd434068bf0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd438003fb0 tx=0x7fd43800d040 comp rx=0 tx=0).stop 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd44815a910 msgr2=0x7fd448156e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd44815a910 0x7fd448156e30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fd440016000 tx=0x7fd440004770 comp rx=0 tx=0).stop 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 shutdown_connections 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd434066730 0x7fd434068bf0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd44815a910 0x7fd448156e30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4481509e0 0x7fd4481568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 --2- 172.21.15.119:0/938433474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4480cc220 0x7fd448147730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 >> 172.21.15.119:0/938433474 conn(0x7fd448070cc0 msgr2=0x7fd44806d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 shutdown_connections 2024-09-13T16:25:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.629+0000 7fd44e012700 1 -- 172.21.15.119:0/938433474 wait complete. 2024-09-13T16:25:59.648 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:25:59.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:25:59.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3b59e700 1 Processor -- start 2024-09-13T16:25:59.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3b59e700 1 -- start start 2024-09-13T16:25:59.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3b59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0f3c0b3100 0x7f0f3c0cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f3c0c5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3b59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f3c0cef70 con 0x7f0f3c15cdf0 2024-09-13T16:25:59.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3b59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3c15d1d0 con 0x7f0f3c0b3100 2024-09-13T16:25:59.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c0cf1c0 con 0x7f0f3c0b3ff0 2024-09-13T16:25:59.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3a59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0f3c15cdf0 0x7f0f3c0b38e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58816/0 (socket says 172.21.15.119:58816) 2024-09-13T16:25:59.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.765+0000 7f0f3a59c700 1 -- 172.21.15.119:0/3635600705 learned_addr learned my addr 172.21.15.119:0/3635600705 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f39d9b700 1 --2- 172.21.15.119:0/3635600705 >> v2:172.21.15.145:3300/0 conn(0x7f0f3c0b3100 0x7f0f3c0cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010177972 0 0) 0x7f0f3c0cef70 con 0x7f0f3c15cdf0 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f280035d0 con 0x7f0f3c15cdf0 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3ad9d700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f3c0c5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2158570852 0 0) 0x7f0f280035d0 con 0x7f0f3c15cdf0 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 msgr2=0x7f0f3c0c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f3c0c5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 >> v2:172.21.15.145:3300/0 conn(0x7f0f3c0b3100 msgr2=0x7f0f3c0cff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:59.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 --2- 172.21.15.119:0/3635600705 >> v2:172.21.15.145:3300/0 conn(0x7f0f3c0b3100 0x7f0f3c0cff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c0cf4a0 con 0x7f0f3c15cdf0 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f0f2c001670 con 0x7f0f3c15cdf0 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3ad9d700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f3c0c5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 >> v1:172.21.15.119:6789/0 conn(0x7f0f3c15cdf0 legacy=0x7f0f3c0b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f28003a20 0x7f0f28003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f28004870 0x7f0f28004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f28005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c15d1d0 con 0x7f0f28003a20 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c0cf1c0 con 0x7f0f28004870 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2800ba80 con 0x7f0f3c0b3ff0 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3a59c700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f28003a20 0x7f0f28003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f39d9b700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f28004870 0x7f0f28004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3ad9d700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f28005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3a59c700 1 -- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 msgr2=0x7f0f28005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3a59c700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f28005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3a59c700 1 -- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f28004870 msgr2=0x7f0f28004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3a59c700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f28004870 0x7f0f28004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3a59c700 1 -- 172.21.15.119:0/3635600705 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f280035d0 con 0x7f0f28003a20 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3ad9d700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c0b3ff0 0x7f0f28005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3a59c700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f28003a20 0x7f0f28003e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00c270 tx=0x7f0f280058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5b14b577ed2c51 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c007b90 con 0x7f0f28003a20 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c00cdb0 con 0x7f0f28003a20 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3959a700 1 -- 172.21.15.119:0/3635600705 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c005bc0 con 0x7f0f28003a20 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3635600705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f28003a20 msgr2=0x7f0f28003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3b59e700 1 --2- 172.21.15.119:0/3635600705 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f28003a20 0x7f0f28003e40 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00c270 tx=0x7f0f280058a0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3ad9d700 1 -- 172.21.15.119:0/3635600705 reap_dead start 2024-09-13T16:25:59.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3635600705 shutdown_connections 2024-09-13T16:25:59.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3635600705 >> 172.21.15.119:0/3635600705 conn(0x7f0f3c0b25e0 msgr2=0x7f0f3c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:59.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3635600705 shutdown_connections 2024-09-13T16:25:59.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.769+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3635600705 wait complete. 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 Processor -- start 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 -- start start 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f3c1537f0 0x7f0f3c0cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f3c158970 0x7f0f3c158df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c1f2d90 0x7f0f3c1f3210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c15e300 con 0x7f0f3c158970 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c1542e0 con 0x7f0f3c1537f0 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f3b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f3c159330 con 0x7f0f3c1f2d90 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f39d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f3c158970 0x7f0f3c158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.773+0000 7f0f39d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f3c158970 0x7f0f3c158df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43606/0 (socket says 172.21.15.119:43606) 2024-09-13T16:25:59.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f39d9b700 1 -- 172.21.15.119:0/3784631570 learned_addr learned my addr 172.21.15.119:0/3784631570 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f3a59c700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f3c1537f0 0x7f0f3c0cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f3ad9d700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c1f2d90 0x7f0f3c1f3210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f39d9b700 1 -- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c1f2d90 msgr2=0x7f0f3c1f3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f39d9b700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c1f2d90 0x7f0f3c1f3210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f39d9b700 1 -- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f3c1537f0 msgr2=0x7f0f3c0cb9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f39d9b700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f3c1537f0 0x7f0f3c0cb9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f39d9b700 1 -- 172.21.15.119:0/3784631570 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3c15d3a0 con 0x7f0f3c158970 2024-09-13T16:25:59.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f39d9b700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f3c158970 0x7f0f3c158df0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f240116f0 tx=0x7f0f24011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f32ffd700 1 -- 172.21.15.119:0/3784631570 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24015870 con 0x7f0f3c158970 2024-09-13T16:25:59.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f32ffd700 1 -- 172.21.15.119:0/3784631570 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f240159d0 con 0x7f0f3c158970 2024-09-13T16:25:59.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f32ffd700 1 -- 172.21.15.119:0/3784631570 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f24015d40 con 0x7f0f3c158970 2024-09-13T16:25:59.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f3c15d680 con 0x7f0f3c158970 2024-09-13T16:25:59.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f3c15dc40 con 0x7f0f3c158970 2024-09-13T16:25:59.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.777+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f3c0cff80 con 0x7f0f3c158970 2024-09-13T16:25:59.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.781+0000 7f0f32ffd700 1 -- 172.21.15.119:0/3784631570 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0f24013400 con 0x7f0f3c158970 2024-09-13T16:25:59.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.781+0000 7f0f32ffd700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f1c066730 0x7f0f1c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:25:59.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.781+0000 7f0f32ffd700 1 -- 172.21.15.119:0/3784631570 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f0f240846d0 con 0x7f0f3c158970 2024-09-13T16:25:59.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.781+0000 7f0f3a59c700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f1c066730 0x7f0f1c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:25:59.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.781+0000 7f0f3a59c700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f1c066730 0x7f0f1c068bf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00c270 tx=0x7f0f2c00a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:25:59.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.785+0000 7f0f32ffd700 1 -- 172.21.15.119:0/3784631570 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0f24051750 con 0x7f0f3c158970 2024-09-13T16:25:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0f3c15c420 con 0x7f0f3c158970 2024-09-13T16:25:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f32ffd700 1 -- 172.21.15.119:0/3784631570 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0f24054d70 con 0x7f0f3c158970 2024-09-13T16:25:59.960 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f1c066730 msgr2=0x7f0f1c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f1c066730 0x7f0f1c068bf0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00c270 tx=0x7f0f2c00a000 comp rx=0 tx=0).stop 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f3c158970 msgr2=0x7f0f3c158df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f3c158970 0x7f0f3c158df0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0f240116f0 tx=0x7f0f24011a00 comp rx=0 tx=0).stop 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 shutdown_connections 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f1c066730 0x7f0f1c068bf0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f3c1f2d90 0x7f0f3c1f3210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f3c158970 0x7f0f3c158df0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 --2- 172.21.15.119:0/3784631570 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f3c1537f0 0x7f0f3c0cb9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 >> 172.21.15.119:0/3784631570 conn(0x7f0f3c0b25e0 msgr2=0x7f0f3c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:25:59.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 shutdown_connections 2024-09-13T16:25:59.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:25:59.957+0000 7f0f3b59e700 1 -- 172.21.15.119:0/3784631570 wait complete. 2024-09-13T16:25:59.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:00.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 Processor -- start 2024-09-13T16:26:00.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- start start 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f015cae0 0x7f85f00ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f85f00d0fb0 0x7f85f00d1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f00cd600 con 0x7f85f00cff60 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f00ce1d0 con 0x7f85f00d0fb0 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00d19a0 con 0x7f85f015cae0 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f6b20700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f85f00d0fb0 0x7f85f00d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f5b1e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f85f00cff60 0x7f85f00d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58830/0 (socket says 172.21.15.119:58830) 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f6b20700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f85f00d0fb0 0x7f85f00d1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43612/0 (socket says 172.21.15.119:43612) 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f5b1e700 1 -- 172.21.15.119:0/1832664085 learned_addr learned my addr 172.21.15.119:0/1832664085 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f631f700 1 --2- 172.21.15.119:0/1832664085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f015cae0 0x7f85f00ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f6b20700 1 -- 172.21.15.119:0/1832664085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f015cae0 msgr2=0x7f85f00ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f6b20700 1 --2- 172.21.15.119:0/1832664085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f015cae0 0x7f85f00ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f6b20700 1 -- 172.21.15.119:0/1832664085 >> v1:172.21.15.119:6789/0 conn(0x7f85f00cff60 legacy=0x7f85f00d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f531d700 1 -- 172.21.15.119:0/1832664085 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931156637 0 0) 0x7f85f00cd600 con 0x7f85f00cff60 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f6b20700 1 -- 172.21.15.119:0/1832664085 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f00d0dd0 con 0x7f85f00d0fb0 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f631f700 1 --2- 172.21.15.119:0/1832664085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f015cae0 0x7f85f00ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f6b20700 1 --2- 172.21.15.119:0/1832664085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00d0fb0 0x7f85f00d1460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009d90 tx=0x7f85e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83eabe35c132aed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f531d700 1 -- 172.21.15.119:0/1832664085 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e800e070 con 0x7f85f00d0fb0 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f531d700 1 -- 172.21.15.119:0/1832664085 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e800bbd0 con 0x7f85f00d0fb0 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f531d700 1 -- 172.21.15.119:0/1832664085 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e8012430 con 0x7f85f00d0fb0 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- 172.21.15.119:0/1832664085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00d0fb0 msgr2=0x7f85f00d1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 --2- 172.21.15.119:0/1832664085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00d0fb0 0x7f85f00d1460 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f85e8009d90 tx=0x7f85e800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- 172.21.15.119:0/1832664085 shutdown_connections 2024-09-13T16:26:00.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 --2- 172.21.15.119:0/1832664085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00d0fb0 0x7f85f00d1460 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 --2- 172.21.15.119:0/1832664085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f015cae0 0x7f85f00ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- 172.21.15.119:0/1832664085 >> 172.21.15.119:0/1832664085 conn(0x7f85f00d29f0 msgr2=0x7f85f00d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:00.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- 172.21.15.119:0/1832664085 shutdown_connections 2024-09-13T16:26:00.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.089+0000 7f85f7321700 1 -- 172.21.15.119:0/1832664085 wait complete. 2024-09-13T16:26:00.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 Processor -- start 2024-09-13T16:26:00.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 -- start start 2024-09-13T16:26:00.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00cff60 0x7f85f014c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f00d0fb0 0x7f85f00c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85f015cae0 0x7f85f00c84d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00c9760 con 0x7f85f00cff60 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00c8b50 con 0x7f85f015cae0 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f00c8e30 con 0x7f85f00d0fb0 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00cff60 0x7f85f014c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f5b1e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f00d0fb0 0x7f85f00c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00cff60 0x7f85f014c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43626/0 (socket says 172.21.15.119:43626) 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 -- 172.21.15.119:0/1907091825 learned_addr learned my addr 172.21.15.119:0/1907091825 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f6b20700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85f015cae0 0x7f85f00c84d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 -- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f00d0fb0 msgr2=0x7f85f00c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f00d0fb0 0x7f85f00c7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 -- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85f015cae0 msgr2=0x7f85f00c84d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85f015cae0 0x7f85f00c84d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 -- 172.21.15.119:0/1907091825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85e8009a70 con 0x7f85f00cff60 2024-09-13T16:26:00.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f631f700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00cff60 0x7f85f014c850 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f85e000e760 tx=0x7f85e000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85e6ffd700 1 -- 172.21.15.119:0/1907091825 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e000c070 con 0x7f85f00cff60 2024-09-13T16:26:00.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85e6ffd700 1 -- 172.21.15.119:0/1907091825 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e0004250 con 0x7f85f00cff60 2024-09-13T16:26:00.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f00c9110 con 0x7f85f00cff60 2024-09-13T16:26:00.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85e6ffd700 1 -- 172.21.15.119:0/1907091825 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85e00036c0 con 0x7f85f00cff60 2024-09-13T16:26:00.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.093+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f015c280 con 0x7f85f00cff60 2024-09-13T16:26:00.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.097+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f00cdad0 con 0x7f85f00cff60 2024-09-13T16:26:00.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.097+0000 7f85e6ffd700 1 -- 172.21.15.119:0/1907091825 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f85e0003820 con 0x7f85f00cff60 2024-09-13T16:26:00.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.097+0000 7f85e6ffd700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85d4066760 0x7f85d4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.097+0000 7f85e6ffd700 1 -- 172.21.15.119:0/1907091825 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f85e00840f0 con 0x7f85f00cff60 2024-09-13T16:26:00.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.097+0000 7f85f5b1e700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85d4066760 0x7f85d4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.097+0000 7f85f5b1e700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85d4066760 0x7f85d4068c20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85f00c9c70 tx=0x7f85ec007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.105+0000 7f85e6ffd700 1 -- 172.21.15.119:0/1907091825 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f85e0051170 con 0x7f85f00cff60 2024-09-13T16:26:00.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.273+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f85f00b0c90 con 0x7f85f00cff60 2024-09-13T16:26:00.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.273+0000 7f85e6ffd700 1 -- 172.21.15.119:0/1907091825 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f85e00094e0 con 0x7f85f00cff60 2024-09-13T16:26:00.278 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:00.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85d4066760 msgr2=0x7f85d4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85d4066760 0x7f85d4068c20 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f85f00c9c70 tx=0x7f85ec007e90 comp rx=0 tx=0).stop 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00cff60 msgr2=0x7f85f014c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00cff60 0x7f85f014c850 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f85e000e760 tx=0x7f85e000ea70 comp rx=0 tx=0).stop 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 shutdown_connections 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85d4066760 0x7f85d4068c20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85f015cae0 0x7f85f00c84d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85f00d0fb0 0x7f85f00c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 --2- 172.21.15.119:0/1907091825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85f00cff60 0x7f85f014c850 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 >> 172.21.15.119:0/1907091825 conn(0x7f85f00d29f0 msgr2=0x7f85f006c080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 shutdown_connections 2024-09-13T16:26:00.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.277+0000 7f85f7321700 1 -- 172.21.15.119:0/1907091825 wait complete. 2024-09-13T16:26:00.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:00.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585b02e700 1 Processor -- start 2024-09-13T16:26:00.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585b02e700 1 -- start start 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585b02e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f58540cf190 0x7f585413ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585b02e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f585406aa60 0x7f585415d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585b02e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f585406cc00 con 0x7f585415d560 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585b02e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585406cee0 con 0x7f585406aa60 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585b02e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f585406d1c0 con 0x7f58540cf190 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a82d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f585406aa60 0x7f585415d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a82d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f585406aa60 0x7f585415d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43628/0 (socket says 172.21.15.119:43628) 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a82d700 1 -- 172.21.15.119:0/2611797471 learned_addr learned my addr 172.21.15.119:0/2611797471 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a02c700 1 --2- 172.21.15.119:0/2611797471 >> v2:172.21.15.145:3300/0 conn(0x7f58540cf190 0x7f585413ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a82d700 1 -- 172.21.15.119:0/2611797471 >> v2:172.21.15.145:3300/0 conn(0x7f58540cf190 msgr2=0x7f585413ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:00.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a82d700 1 --2- 172.21.15.119:0/2611797471 >> v2:172.21.15.145:3300/0 conn(0x7f58540cf190 0x7f585413ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a82d700 1 -- 172.21.15.119:0/2611797471 >> v1:172.21.15.145:6789/0 conn(0x7f585415d560 legacy=0x7f585406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.401+0000 7f585a82d700 1 -- 172.21.15.119:0/2611797471 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f585406d4a0 con 0x7f585406aa60 2024-09-13T16:26:00.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585a82d700 1 --2- 172.21.15.119:0/2611797471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585406aa60 0x7f585415d9f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5854070970 tx=0x7f584c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f740c4049d0ac68e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585902a700 1 -- 172.21.15.119:0/2611797471 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c00e070 con 0x7f585406aa60 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585902a700 1 -- 172.21.15.119:0/2611797471 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584c00b910 con 0x7f585406aa60 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585902a700 1 -- 172.21.15.119:0/2611797471 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584c00bcc0 con 0x7f585406aa60 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- 172.21.15.119:0/2611797471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585406aa60 msgr2=0x7f585415d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 --2- 172.21.15.119:0/2611797471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585406aa60 0x7f585415d9f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5854070970 tx=0x7f584c00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- 172.21.15.119:0/2611797471 shutdown_connections 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 --2- 172.21.15.119:0/2611797471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585406aa60 0x7f585415d9f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 --2- 172.21.15.119:0/2611797471 >> v2:172.21.15.145:3300/0 conn(0x7f58540cf190 0x7f585413ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- 172.21.15.119:0/2611797471 >> 172.21.15.119:0/2611797471 conn(0x7f585406b460 msgr2=0x7f585406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:00.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- 172.21.15.119:0/2611797471 shutdown_connections 2024-09-13T16:26:00.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- 172.21.15.119:0/2611797471 wait complete. 2024-09-13T16:26:00.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 Processor -- start 2024-09-13T16:26:00.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- start start 2024-09-13T16:26:00.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f585406aa60 0x7f585413f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58540cf190 0x7f58540b3160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585415d560 0x7f585415e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5854141ea0 con 0x7f585415d560 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5854141220 con 0x7f58540cf190 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585b02e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5854141500 con 0x7f585406aa60 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585a02c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f585406aa60 0x7f585413f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585a82d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585415d560 0x7f585415e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585a02c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f585406aa60 0x7f585413f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:36176/0 (socket says 172.21.15.119:36176) 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.405+0000 7f585a02c700 1 -- 172.21.15.119:0/2778609117 learned_addr learned my addr 172.21.15.119:0/2778609117 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585982b700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58540cf190 0x7f58540b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585a02c700 1 -- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58540cf190 msgr2=0x7f58540b3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585a02c700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58540cf190 0x7f58540b3160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585a02c700 1 -- 172.21.15.119:0/2778609117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585415d560 msgr2=0x7f585415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585a02c700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585415d560 0x7f585415e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585a02c700 1 -- 172.21.15.119:0/2778609117 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f584c009b30 con 0x7f585406aa60 2024-09-13T16:26:00.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585a02c700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f585406aa60 0x7f585413f260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9d0 tx=0x7f584800ece0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f5846ffd700 1 -- 172.21.15.119:0/2778609117 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848009720 con 0x7f585406aa60 2024-09-13T16:26:00.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f5846ffd700 1 -- 172.21.15.119:0/2778609117 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584800ee70 con 0x7f585406aa60 2024-09-13T16:26:00.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58541417e0 con 0x7f585406aa60 2024-09-13T16:26:00.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f5846ffd700 1 -- 172.21.15.119:0/2778609117 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800ca50 con 0x7f585406aa60 2024-09-13T16:26:00.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f585413f7a0 con 0x7f585406aa60 2024-09-13T16:26:00.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f585415d9f0 con 0x7f585406aa60 2024-09-13T16:26:00.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f5846ffd700 1 -- 172.21.15.119:0/2778609117 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f584800cbb0 con 0x7f585406aa60 2024-09-13T16:26:00.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.409+0000 7f5846ffd700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5838066730 0x7f5838068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.413+0000 7f5846ffd700 1 -- 172.21.15.119:0/2778609117 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f5848084420 con 0x7f585406aa60 2024-09-13T16:26:00.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.413+0000 7f585982b700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5838066730 0x7f5838068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.413+0000 7f585982b700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5838066730 0x7f5838068bf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5854157520 tx=0x7f5850009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.421+0000 7f5846ffd700 1 -- 172.21.15.119:0/2778609117 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5848051480 con 0x7f585406aa60 2024-09-13T16:26:00.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.589+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f585406e880 con 0x7f585406aa60 2024-09-13T16:26:00.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.589+0000 7f5846ffd700 1 -- 172.21.15.119:0/2778609117 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f584800ce60 con 0x7f585406aa60 2024-09-13T16:26:00.595 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5838066730 msgr2=0x7f5838068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5838066730 0x7f5838068bf0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5854157520 tx=0x7f5850009040 comp rx=0 tx=0).stop 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f585406aa60 msgr2=0x7f585413f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f585406aa60 0x7f585413f260 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f584800e9d0 tx=0x7f584800ece0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 shutdown_connections 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5838066730 0x7f5838068bf0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f585415d560 0x7f585415e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f58540cf190 0x7f58540b3160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 --2- 172.21.15.119:0/2778609117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f585406aa60 0x7f585413f260 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 >> 172.21.15.119:0/2778609117 conn(0x7f585406b460 msgr2=0x7f585406a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:00.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 shutdown_connections 2024-09-13T16:26:00.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.593+0000 7f585b02e700 1 -- 172.21.15.119:0/2778609117 wait complete. 2024-09-13T16:26:00.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:00.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.721+0000 7f2985889700 1 Processor -- start 2024-09-13T16:26:00.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.721+0000 7f2985889700 1 -- start start 2024-09-13T16:26:00.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2980144180 0x7f298006c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298006cb40 con 0x7f29800cc010 2024-09-13T16:26:00.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f298006ce20 con 0x7f29800ce1d0 2024-09-13T16:26:00.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29800b1c50 con 0x7f2980144180 2024-09-13T16:26:00.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f297ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f29800cc010 0x7f29800cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58846/0 (socket says 172.21.15.119:58846) 2024-09-13T16:26:00.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f297ffff700 1 -- 172.21.15.119:0/3854396866 learned_addr learned my addr 172.21.15.119:0/3854396866 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:00.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2984887700 1 --2- 172.21.15.119:0/3854396866 >> v2:172.21.15.145:3300/0 conn(0x7f2980144180 0x7f298006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f297f7fe700 1 -- 172.21.15.119:0/3854396866 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129683628 0 0) 0x7f298006cb40 con 0x7f29800cc010 2024-09-13T16:26:00.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f297f7fe700 1 -- 172.21.15.119:0/3854396866 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296c0035d0 con 0x7f29800cc010 2024-09-13T16:26:00.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2984887700 1 -- 172.21.15.119:0/3854396866 >> v1:172.21.15.145:6789/0 conn(0x7f29800ce1d0 legacy=0x7f29800b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2984887700 1 -- 172.21.15.119:0/3854396866 >> v1:172.21.15.119:6789/0 conn(0x7f29800cc010 legacy=0x7f29800cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2984887700 1 -- 172.21.15.119:0/3854396866 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298006d100 con 0x7f2980144180 2024-09-13T16:26:00.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2984887700 1 --2- 172.21.15.119:0/3854396866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2980144180 0x7f298006c480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2978009d90 tx=0x7f297800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff570193d4f908d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f297f7fe700 1 -- 172.21.15.119:0/3854396866 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f297800e070 con 0x7f2980144180 2024-09-13T16:26:00.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f297f7fe700 1 -- 172.21.15.119:0/3854396866 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f297800bb20 con 0x7f2980144180 2024-09-13T16:26:00.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f297f7fe700 1 -- 172.21.15.119:0/3854396866 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2978012430 con 0x7f2980144180 2024-09-13T16:26:00.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- 172.21.15.119:0/3854396866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2980144180 msgr2=0x7f298006c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 --2- 172.21.15.119:0/3854396866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2980144180 0x7f298006c480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2978009d90 tx=0x7f297800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:00.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- 172.21.15.119:0/3854396866 shutdown_connections 2024-09-13T16:26:00.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 --2- 172.21.15.119:0/3854396866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2980144180 0x7f298006c480 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- 172.21.15.119:0/3854396866 >> 172.21.15.119:0/3854396866 conn(0x7f29800b0980 msgr2=0x7f29800b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:00.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- 172.21.15.119:0/3854396866 shutdown_connections 2024-09-13T16:26:00.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.725+0000 7f2985889700 1 -- 172.21.15.119:0/3854396866 wait complete. 2024-09-13T16:26:00.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 Processor -- start 2024-09-13T16:26:00.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 -- start start 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29800cc010 0x7f298015a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29800ce1d0 0x7f29800abc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2980144180 0x7f29800ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29800c37d0 con 0x7f29800ce1d0 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29800a9f50 con 0x7f29800cc010 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f298013f650 con 0x7f2980144180 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29800ce1d0 0x7f29800abc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29800ce1d0 0x7f29800abc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43656/0 (socket says 172.21.15.119:43656) 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 -- 172.21.15.119:0/1672964978 learned_addr learned my addr 172.21.15.119:0/1672964978 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2984887700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29800cc010 0x7f298015a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985088700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2980144180 0x7f29800ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 -- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2980144180 msgr2=0x7f29800ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2980144180 0x7f29800ac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 -- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29800cc010 msgr2=0x7f298015a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29800cc010 0x7f298015a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 -- 172.21.15.119:0/1672964978 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2978009a70 con 0x7f29800ce1d0 2024-09-13T16:26:00.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297ffff700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29800ce1d0 0x7f29800abc40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f296800c730 tx=0x7f296800caf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297d7fa700 1 -- 172.21.15.119:0/1672964978 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968005d90 con 0x7f29800ce1d0 2024-09-13T16:26:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297d7fa700 1 -- 172.21.15.119:0/1672964978 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29680070d0 con 0x7f29800ce1d0 2024-09-13T16:26:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29801f1460 con 0x7f29800ce1d0 2024-09-13T16:26:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f297d7fa700 1 -- 172.21.15.119:0/1672964978 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2968007460 con 0x7f29800ce1d0 2024-09-13T16:26:00.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.729+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29800a9900 con 0x7f29800ce1d0 2024-09-13T16:26:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.733+0000 7f297d7fa700 1 -- 172.21.15.119:0/1672964978 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f29680075c0 con 0x7f29800ce1d0 2024-09-13T16:26:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.733+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f298006c480 con 0x7f29800ce1d0 2024-09-13T16:26:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.733+0000 7f297d7fa700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2960066730 0x7f2960068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.733+0000 7f297d7fa700 1 -- 172.21.15.119:0/1672964978 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f2968083bb0 con 0x7f29800ce1d0 2024-09-13T16:26:00.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.733+0000 7f2984887700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2960066730 0x7f2960068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:00.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.733+0000 7f2984887700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2960066730 0x7f2960068bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2978014040 tx=0x7f2978011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:00.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.741+0000 7f297d7fa700 1 -- 172.21.15.119:0/1672964978 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2968050a80 con 0x7f29800ce1d0 2024-09-13T16:26:00.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.901+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f29800aa090 con 0x7f29800ce1d0 2024-09-13T16:26:00.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.901+0000 7f297d7fa700 1 -- 172.21.15.119:0/1672964978 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7f29680170a0 con 0x7f29800ce1d0 2024-09-13T16:26:00.905 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:00.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2960066730 msgr2=0x7f2960068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2960066730 0x7f2960068bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2978014040 tx=0x7f2978011000 comp rx=0 tx=0).stop 2024-09-13T16:26:00.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29800ce1d0 msgr2=0x7f29800abc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:00.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29800ce1d0 0x7f29800abc40 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f296800c730 tx=0x7f296800caf0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 shutdown_connections 2024-09-13T16:26:00.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2960066730 0x7f2960068bf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2980144180 0x7f29800ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29800ce1d0 0x7f29800abc40 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 --2- 172.21.15.119:0/1672964978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29800cc010 0x7f298015a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:00.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 >> 172.21.15.119:0/1672964978 conn(0x7f29800b0980 msgr2=0x7f29800b20a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:00.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 shutdown_connections 2024-09-13T16:26:00.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:00.905+0000 7f2985889700 1 -- 172.21.15.119:0/1672964978 wait complete. 2024-09-13T16:26:00.925 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:01.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 Processor -- start 2024-09-13T16:26:01.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- start start 2024-09-13T16:26:01.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f61a0034cf0 0x7f61a0035c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a0052140 con 0x7f61a0048e70 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a0052420 con 0x7f61a0036140 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a0052700 con 0x7f61a0034cf0 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61ae5ec700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f61a0048e70 0x7f61a0053ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58856/0 (socket says 172.21.15.119:58856) 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61ae5ec700 1 -- 172.21.15.119:0/1474154775 learned_addr learned my addr 172.21.15.119:0/1474154775 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61addeb700 1 --2- 172.21.15.119:0/1474154775 >> v2:172.21.15.145:3300/0 conn(0x7f61a0034cf0 0x7f61a0035c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61acde9700 1 -- 172.21.15.119:0/1474154775 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3136797454 0 0) 0x7f61a0052140 con 0x7f61a0048e70 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61acde9700 1 -- 172.21.15.119:0/1474154775 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61940035d0 con 0x7f61a0048e70 2024-09-13T16:26:01.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61addeb700 1 -- 172.21.15.119:0/1474154775 >> v1:172.21.15.145:6789/0 conn(0x7f61a0036140 legacy=0x7f61a0048760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61addeb700 1 -- 172.21.15.119:0/1474154775 >> v1:172.21.15.119:6789/0 conn(0x7f61a0048e70 legacy=0x7f61a0053ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61addeb700 1 -- 172.21.15.119:0/1474154775 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a00529e0 con 0x7f61a0034cf0 2024-09-13T16:26:01.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61addeb700 1 --2- 172.21.15.119:0/1474154775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0034cf0 0x7f61a0035c00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6198009850 tx=0x7f6198009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ac297bbafe72893 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61acde9700 1 -- 172.21.15.119:0/1474154775 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198011830 con 0x7f61a0034cf0 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61acde9700 1 -- 172.21.15.119:0/1474154775 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6198011990 con 0x7f61a0034cf0 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61acde9700 1 -- 172.21.15.119:0/1474154775 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198011d40 con 0x7f61a0034cf0 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- 172.21.15.119:0/1474154775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0034cf0 msgr2=0x7f61a0035c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 --2- 172.21.15.119:0/1474154775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0034cf0 0x7f61a0035c00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6198009850 tx=0x7f6198009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- 172.21.15.119:0/1474154775 shutdown_connections 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 --2- 172.21.15.119:0/1474154775 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0034cf0 0x7f61a0035c00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- 172.21.15.119:0/1474154775 >> 172.21.15.119:0/1474154775 conn(0x7f61a001eaf0 msgr2=0x7f61a0055790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.033+0000 7f61aeded700 1 -- 172.21.15.119:0/1474154775 shutdown_connections 2024-09-13T16:26:01.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 -- 172.21.15.119:0/1474154775 wait complete. 2024-09-13T16:26:01.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 Processor -- start 2024-09-13T16:26:01.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 -- start start 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61a0034cf0 0x7f61a00e46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0036140 0x7f61a004be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61a0048e70 0x7f61a0007960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a004e4d0 con 0x7f61a0034cf0 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a00e6f10 con 0x7f61a0036140 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a017c9a0 con 0x7f61a0048e70 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61a0034cf0 0x7f61a00e46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61a0034cf0 0x7f61a00e46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43672/0 (socket says 172.21.15.119:43672) 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 -- 172.21.15.119:0/3140492441 learned_addr learned my addr 172.21.15.119:0/3140492441 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61ad5ea700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0036140 0x7f61a004be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61ae5ec700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61a0048e70 0x7f61a0007960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 -- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61a0048e70 msgr2=0x7f61a0007960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61a0048e70 0x7f61a0007960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 -- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0036140 msgr2=0x7f61a004be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0036140 0x7f61a004be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 -- 172.21.15.119:0/3140492441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61980094b0 con 0x7f61a0034cf0 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61ad5ea700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0036140 0x7f61a004be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:01.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61addeb700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61a0034cf0 0x7f61a00e46a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f61a000d540 tx=0x7f6198004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f619e7fc700 1 -- 172.21.15.119:0/3140492441 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198004570 con 0x7f61a0034cf0 2024-09-13T16:26:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f619e7fc700 1 -- 172.21.15.119:0/3140492441 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619801f070 con 0x7f61a0034cf0 2024-09-13T16:26:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f619e7fc700 1 -- 172.21.15.119:0/3140492441 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6198003900 con 0x7f61a0034cf0 2024-09-13T16:26:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a017cc80 con 0x7f61a0034cf0 2024-09-13T16:26:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.037+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a017d240 con 0x7f61a0034cf0 2024-09-13T16:26:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.041+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a0052140 con 0x7f61a0034cf0 2024-09-13T16:26:01.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.041+0000 7f619e7fc700 1 -- 172.21.15.119:0/3140492441 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6198021030 con 0x7f61a0034cf0 2024-09-13T16:26:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.041+0000 7f619e7fc700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f618c066730 0x7f618c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.041+0000 7f619e7fc700 1 -- 172.21.15.119:0/3140492441 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f6198085860 con 0x7f61a0034cf0 2024-09-13T16:26:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.041+0000 7f61ad5ea700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f618c066730 0x7f618c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.041+0000 7f61ad5ea700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f618c066730 0x7f618c068bf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f61a8064010 tx=0x7f61a8065ea0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.049+0000 7f619e7fc700 1 -- 172.21.15.119:0/3140492441 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6198052a90 con 0x7f61a0034cf0 2024-09-13T16:26:01.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.221+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61a0005410 con 0x7f61a0034cf0 2024-09-13T16:26:01.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f619e7fc700 1 -- 172.21.15.119:0/3140492441 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f61980560b0 con 0x7f61a0034cf0 2024-09-13T16:26:01.229 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:01.229 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:01.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f618c066730 msgr2=0x7f618c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f618c066730 0x7f618c068bf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f61a8064010 tx=0x7f61a8065ea0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61a0034cf0 msgr2=0x7f61a00e46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61a0034cf0 0x7f61a00e46a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f61a000d540 tx=0x7f6198004770 comp rx=0 tx=0).stop 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 shutdown_connections 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f618c066730 0x7f618c068bf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f61a0048e70 0x7f61a0007960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61a0036140 0x7f61a004be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 --2- 172.21.15.119:0/3140492441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f61a0034cf0 0x7f61a00e46a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.225+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 >> 172.21.15.119:0/3140492441 conn(0x7f61a001eaf0 msgr2=0x7f61a003b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.229+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 shutdown_connections 2024-09-13T16:26:01.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.229+0000 7f61aeded700 1 -- 172.21.15.119:0/3140492441 wait complete. 2024-09-13T16:26:01.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:01.244 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:01.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 Processor -- start 2024-09-13T16:26:01.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 -- start start 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf28160210 0x7fbf280d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbf280d17e0 0x7fbf280b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf280b2240 con 0x7fbf280d3760 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf281605f0 con 0x7fbf280d17e0 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280b2520 con 0x7fbf28160210 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf267fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbf280d3760 0x7fbf280d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58872/0 (socket says 172.21.15.119:58872) 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf267fc700 1 -- 172.21.15.119:0/899918898 learned_addr learned my addr 172.21.15.119:0/899918898 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:01.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf26ffd700 1 --2- 172.21.15.119:0/899918898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf28160210 0x7fbf280d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf277fe700 1 --2- 172.21.15.119:0/899918898 >> v2:172.21.15.145:3300/0 conn(0x7fbf280d17e0 0x7fbf280b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf25ffb700 1 -- 172.21.15.119:0/899918898 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821441231 0 0) 0x7fbf280b2240 con 0x7fbf280d3760 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf25ffb700 1 -- 172.21.15.119:0/899918898 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf100035d0 con 0x7fbf280d3760 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf26ffd700 1 -- 172.21.15.119:0/899918898 >> v2:172.21.15.145:3300/0 conn(0x7fbf280d17e0 msgr2=0x7fbf280b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf26ffd700 1 --2- 172.21.15.119:0/899918898 >> v2:172.21.15.145:3300/0 conn(0x7fbf280d17e0 0x7fbf280b29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf26ffd700 1 -- 172.21.15.119:0/899918898 >> v1:172.21.15.119:6789/0 conn(0x7fbf280d3760 legacy=0x7fbf280d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf26ffd700 1 -- 172.21.15.119:0/899918898 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf280d0780 con 0x7fbf28160210 2024-09-13T16:26:01.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf26ffd700 1 --2- 172.21.15.119:0/899918898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf28160210 0x7fbf280d3220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009850 tx=0x7fbf18009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4787c6c427f710a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf25ffb700 1 -- 172.21.15.119:0/899918898 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18011830 con 0x7fbf28160210 2024-09-13T16:26:01.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf25ffb700 1 -- 172.21.15.119:0/899918898 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf18011990 con 0x7fbf28160210 2024-09-13T16:26:01.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf25ffb700 1 -- 172.21.15.119:0/899918898 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18011d40 con 0x7fbf28160210 2024-09-13T16:26:01.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 -- 172.21.15.119:0/899918898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf28160210 msgr2=0x7fbf280d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.361+0000 7fbf27fff700 1 --2- 172.21.15.119:0/899918898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf28160210 0x7fbf280d3220 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbf18009850 tx=0x7fbf18009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:01.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 -- 172.21.15.119:0/899918898 shutdown_connections 2024-09-13T16:26:01.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 --2- 172.21.15.119:0/899918898 >> v2:172.21.15.145:3300/0 conn(0x7fbf280d17e0 0x7fbf280b29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 --2- 172.21.15.119:0/899918898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf28160210 0x7fbf280d3220 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 -- 172.21.15.119:0/899918898 >> 172.21.15.119:0/899918898 conn(0x7fbf2808c080 msgr2=0x7fbf280d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:01.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 -- 172.21.15.119:0/899918898 shutdown_connections 2024-09-13T16:26:01.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 -- 172.21.15.119:0/899918898 wait complete. 2024-09-13T16:26:01.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 Processor -- start 2024-09-13T16:26:01.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.365+0000 7fbf27fff700 1 -- start start 2024-09-13T16:26:01.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280d17e0 0x7fbf281555b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf280d3760 0x7fbf280cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf28160210 0x7fbf280b5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280c7010 con 0x7fbf28160210 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280cc440 con 0x7fbf280d3760 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf280c66e0 con 0x7fbf280d17e0 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf28160210 0x7fbf280b5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf28160210 0x7fbf280b5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43682/0 (socket says 172.21.15.119:43682) 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 -- 172.21.15.119:0/251435807 learned_addr learned my addr 172.21.15.119:0/251435807 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf26ffd700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280d17e0 0x7fbf281555b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf267fc700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf280d3760 0x7fbf280cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 -- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280d17e0 msgr2=0x7fbf281555b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280d17e0 0x7fbf281555b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 -- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf280d3760 msgr2=0x7fbf280cbd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf280d3760 0x7fbf280cbd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 -- 172.21.15.119:0/251435807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf180094b0 con 0x7fbf28160210 2024-09-13T16:26:01.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf277fe700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf28160210 0x7fbf280b5e10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000ef90 tx=0x7fbf2000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf0f7fe700 1 -- 172.21.15.119:0/251435807 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000cc90 con 0x7fbf28160210 2024-09-13T16:26:01.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf0f7fe700 1 -- 172.21.15.119:0/251435807 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2000f040 con 0x7fbf28160210 2024-09-13T16:26:01.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf280c69c0 con 0x7fbf28160210 2024-09-13T16:26:01.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf0f7fe700 1 -- 172.21.15.119:0/251435807 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20014480 con 0x7fbf28160210 2024-09-13T16:26:01.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf280b5b50 con 0x7fbf28160210 2024-09-13T16:26:01.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.369+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf280c7180 con 0x7fbf28160210 2024-09-13T16:26:01.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.373+0000 7fbf0f7fe700 1 -- 172.21.15.119:0/251435807 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbf200145e0 con 0x7fbf28160210 2024-09-13T16:26:01.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.373+0000 7fbf0f7fe700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf14066760 0x7fbf14068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.373+0000 7fbf0f7fe700 1 -- 172.21.15.119:0/251435807 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbf20085070 con 0x7fbf28160210 2024-09-13T16:26:01.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.373+0000 7fbf26ffd700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf14066760 0x7fbf14068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.373+0000 7fbf26ffd700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf14066760 0x7fbf14068c20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf180090b0 tx=0x7fbf1801c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.377+0000 7fbf0f7fe700 1 -- 172.21.15.119:0/251435807 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbf200b6db0 con 0x7fbf28160210 2024-09-13T16:26:01.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.549+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbf280d3220 con 0x7fbf28160210 2024-09-13T16:26:01.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.549+0000 7fbf0f7fe700 1 -- 172.21.15.119:0/251435807 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbf200180a0 con 0x7fbf28160210 2024-09-13T16:26:01.553 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:01.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.549+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf14066760 msgr2=0x7fbf14068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.549+0000 7fbf27fff700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf14066760 0x7fbf14068c20 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbf180090b0 tx=0x7fbf1801c040 comp rx=0 tx=0).stop 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.549+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf28160210 msgr2=0x7fbf280b5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.549+0000 7fbf27fff700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf28160210 0x7fbf280b5e10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbf2000ef90 tx=0x7fbf2000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 shutdown_connections 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf14066760 0x7fbf14068c20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf28160210 0x7fbf280b5e10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf280d3760 0x7fbf280cbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 --2- 172.21.15.119:0/251435807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf280d17e0 0x7fbf281555b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 >> 172.21.15.119:0/251435807 conn(0x7fbf2808c080 msgr2=0x7fbf280b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 shutdown_connections 2024-09-13T16:26:01.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.553+0000 7fbf27fff700 1 -- 172.21.15.119:0/251435807 wait complete. 2024-09-13T16:26:01.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:01.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9858700 1 Processor -- start 2024-09-13T16:26:01.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9858700 1 -- start start 2024-09-13T16:26:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9858700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7ee4158950 0x7f7ee415cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9858700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7ee40b1d00 0x7f7ee40b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9858700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee40704d0 con 0x7f7ee406dbf0 2024-09-13T16:26:01.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9858700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee415d4b0 con 0x7f7ee40b1d00 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9858700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee4070720 con 0x7f7ee4158950 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee9057700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7ee40b1d00 0x7f7ee40b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7ee406dbf0 0x7f7ee4159980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58882/0 (socket says 172.21.15.119:58882) 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.677+0000 7f7ee3fff700 1 -- 172.21.15.119:0/729940568 learned_addr learned my addr 172.21.15.119:0/729940568 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee8856700 1 --2- 172.21.15.119:0/729940568 >> v2:172.21.15.145:3300/0 conn(0x7f7ee4158950 0x7f7ee415cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee37fe700 1 -- 172.21.15.119:0/729940568 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2584918369 0 0) 0x7f7ee40704d0 con 0x7f7ee406dbf0 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee37fe700 1 -- 172.21.15.119:0/729940568 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ed00035d0 con 0x7f7ee406dbf0 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9057700 1 -- 172.21.15.119:0/729940568 >> v2:172.21.15.145:3300/0 conn(0x7f7ee4158950 msgr2=0x7f7ee415cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:01.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9057700 1 --2- 172.21.15.119:0/729940568 >> v2:172.21.15.145:3300/0 conn(0x7f7ee4158950 0x7f7ee415cf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9057700 1 -- 172.21.15.119:0/729940568 >> v1:172.21.15.119:6789/0 conn(0x7f7ee406dbf0 legacy=0x7f7ee4159980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9057700 1 -- 172.21.15.119:0/729940568 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee4070a00 con 0x7f7ee40b1d00 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9057700 1 --2- 172.21.15.119:0/729940568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee40b1d00 0x7f7ee40b26a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7ee406f4e0 tx=0x7f7edc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea5c41f4f498ff1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee37fe700 1 -- 172.21.15.119:0/729940568 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc00e070 con 0x7f7ee40b1d00 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee37fe700 1 -- 172.21.15.119:0/729940568 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc00bb20 con 0x7f7ee40b1d00 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee37fe700 1 -- 172.21.15.119:0/729940568 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc012430 con 0x7f7ee40b1d00 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 -- 172.21.15.119:0/729940568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee40b1d00 msgr2=0x7f7ee40b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 --2- 172.21.15.119:0/729940568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee40b1d00 0x7f7ee40b26a0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7ee406f4e0 tx=0x7f7edc00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:01.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 -- 172.21.15.119:0/729940568 shutdown_connections 2024-09-13T16:26:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 --2- 172.21.15.119:0/729940568 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee40b1d00 0x7f7ee40b26a0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 --2- 172.21.15.119:0/729940568 >> v2:172.21.15.145:3300/0 conn(0x7f7ee4158950 0x7f7ee415cf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 -- 172.21.15.119:0/729940568 >> 172.21.15.119:0/729940568 conn(0x7f7ee40b0980 msgr2=0x7f7ee40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 -- 172.21.15.119:0/729940568 shutdown_connections 2024-09-13T16:26:01.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 -- 172.21.15.119:0/729940568 wait complete. 2024-09-13T16:26:01.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 Processor -- start 2024-09-13T16:26:01.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.681+0000 7f7ee9858700 1 -- start start 2024-09-13T16:26:01.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee406dbf0 0x7f7ee4155220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee40b1d00 0x7f7ee41516b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee4158950 0x7f7ee41efac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4151290 con 0x7f7ee4158950 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee414fff0 con 0x7f7ee40b1d00 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee414ec70 con 0x7f7ee406dbf0 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee4158950 0x7f7ee41efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee4158950 0x7f7ee41efac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43708/0 (socket says 172.21.15.119:43708) 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 -- 172.21.15.119:0/1145110509 learned_addr learned my addr 172.21.15.119:0/1145110509 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee8856700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee406dbf0 0x7f7ee4155220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee3fff700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee40b1d00 0x7f7ee41516b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 -- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee406dbf0 msgr2=0x7f7ee4155220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee406dbf0 0x7f7ee4155220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 -- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee40b1d00 msgr2=0x7f7ee41516b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee40b1d00 0x7f7ee41516b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 -- 172.21.15.119:0/1145110509 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7edc009a70 con 0x7f7ee4158950 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9057700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee4158950 0x7f7ee41efac0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00b370 tx=0x7f7edc004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee17fa700 1 -- 172.21.15.119:0/1145110509 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc012430 con 0x7f7ee4158950 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee17fa700 1 -- 172.21.15.119:0/1145110509 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc020070 con 0x7f7ee4158950 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee414fd10 con 0x7f7ee4158950 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee17fa700 1 -- 172.21.15.119:0/1145110509 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7edc004280 con 0x7f7ee4158950 2024-09-13T16:26:01.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee4150eb0 con 0x7f7ee4158950 2024-09-13T16:26:01.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.685+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee4150a30 con 0x7f7ee4158950 2024-09-13T16:26:01.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.689+0000 7f7ee17fa700 1 -- 172.21.15.119:0/1145110509 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7edc0043e0 con 0x7f7ee4158950 2024-09-13T16:26:01.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.689+0000 7f7ee17fa700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ec4066760 0x7f7ec4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.689+0000 7f7ee17fa700 1 -- 172.21.15.119:0/1145110509 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f7edc022070 con 0x7f7ee4158950 2024-09-13T16:26:01.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.689+0000 7f7ee8856700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ec4066760 0x7f7ec4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.689+0000 7f7ee8856700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ec4066760 0x7f7ec4068c20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40719f0 tx=0x7f7ecc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:01.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.697+0000 7f7ee17fa700 1 -- 172.21.15.119:0/1145110509 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7ee4150a30 con 0x7f7ee4158950 2024-09-13T16:26:01.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.861+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7ee41513d0 con 0x7f7ee4158950 2024-09-13T16:26:01.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.861+0000 7f7ee17fa700 1 -- 172.21.15.119:0/1145110509 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7ee41513d0 con 0x7f7ee4158950 2024-09-13T16:26:01.866 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ec4066760 msgr2=0x7f7ec4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ec4066760 0x7f7ec4068c20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7ee40719f0 tx=0x7f7ecc008040 comp rx=0 tx=0).stop 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee4158950 msgr2=0x7f7ee41efac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee4158950 0x7f7ee41efac0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f7edc00b370 tx=0x7f7edc004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 shutdown_connections 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ec4066760 0x7f7ec4068c20 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7ee4158950 0x7f7ee41efac0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7ee40b1d00 0x7f7ee41516b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 --2- 172.21.15.119:0/1145110509 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7ee406dbf0 0x7f7ee4155220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:01.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 >> 172.21.15.119:0/1145110509 conn(0x7f7ee40b0980 msgr2=0x7f7ee40ae9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:01.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 shutdown_connections 2024-09-13T16:26:01.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.865+0000 7f7ee9858700 1 -- 172.21.15.119:0/1145110509 wait complete. 2024-09-13T16:26:01.880 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:01.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3484b03700 1 Processor -- start 2024-09-13T16:26:01.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3484b03700 1 -- start start 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3484b03700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f34800cf190 0x7f348013ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3484b03700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f348015d560 0x7f348015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3484b03700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348006cc00 con 0x7f348006a350 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3484b03700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348006cee0 con 0x7f348015d560 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3484b03700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f348006d1c0 con 0x7f34800cf190 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f3476ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f348015d560 0x7f348015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.993+0000 7f347ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f348006a350 0x7f348006dca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58892/0 (socket says 172.21.15.119:58892) 2024-09-13T16:26:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f347ffff700 1 -- 172.21.15.119:0/89359004 learned_addr learned my addr 172.21.15.119:0/89359004 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f347f7fe700 1 --2- 172.21.15.119:0/89359004 >> v2:172.21.15.145:3300/0 conn(0x7f34800cf190 0x7f348013ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3476ffd700 1 -- 172.21.15.119:0/89359004 >> v2:172.21.15.145:3300/0 conn(0x7f34800cf190 msgr2=0x7f348013ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3476ffd700 1 --2- 172.21.15.119:0/89359004 >> v2:172.21.15.145:3300/0 conn(0x7f34800cf190 0x7f348013ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3476ffd700 1 -- 172.21.15.119:0/89359004 >> v1:172.21.15.119:6789/0 conn(0x7f348006a350 legacy=0x7f348006dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f347effd700 1 -- 172.21.15.119:0/89359004 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817752103 0 0) 0x7f348006cc00 con 0x7f348006a350 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3476ffd700 1 -- 172.21.15.119:0/89359004 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348006d4a0 con 0x7f348015d560 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3476ffd700 1 --2- 172.21.15.119:0/89359004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348015d560 0x7f348015d9f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3480070970 tx=0x7f3468009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2452ef02704806c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f347effd700 1 -- 172.21.15.119:0/89359004 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468011830 con 0x7f348015d560 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f347effd700 1 -- 172.21.15.119:0/89359004 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3468011990 con 0x7f348015d560 2024-09-13T16:26:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f347effd700 1 -- 172.21.15.119:0/89359004 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468011d20 con 0x7f348015d560 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 -- 172.21.15.119:0/89359004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348015d560 msgr2=0x7f348015d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 --2- 172.21.15.119:0/89359004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348015d560 0x7f348015d9f0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f3480070970 tx=0x7f3468009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 -- 172.21.15.119:0/89359004 shutdown_connections 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 --2- 172.21.15.119:0/89359004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348015d560 0x7f348015d9f0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 --2- 172.21.15.119:0/89359004 >> v2:172.21.15.145:3300/0 conn(0x7f34800cf190 0x7f348013ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 -- 172.21.15.119:0/89359004 >> 172.21.15.119:0/89359004 conn(0x7f348006b460 msgr2=0x7f348006b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 -- 172.21.15.119:0/89359004 shutdown_connections 2024-09-13T16:26:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:01.997+0000 7f3484b03700 1 -- 172.21.15.119:0/89359004 wait complete. 2024-09-13T16:26:02.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 Processor -- start 2024-09-13T16:26:02.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 -- start start 2024-09-13T16:26:02.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348006a350 0x7f34801f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cf190 0x7f34801f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f348015d560 0x7f34801f61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f348015f3f0 con 0x7f348006a350 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3480157080 con 0x7f348015d560 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348015f640 con 0x7f34800cf190 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348006a350 0x7f34801f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348006a350 0x7f34801f5760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43726/0 (socket says 172.21.15.119:43726) 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 -- 172.21.15.119:0/797383467 learned_addr learned my addr 172.21.15.119:0/797383467 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3476ffd700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cf190 0x7f34801f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347ffff700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f348015d560 0x7f34801f61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 -- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cf190 msgr2=0x7f34801f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cf190 0x7f34801f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 -- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f348015d560 msgr2=0x7f34801f61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f348015d560 0x7f34801f61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 -- 172.21.15.119:0/797383467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34680094b0 con 0x7f348006a350 2024-09-13T16:26:02.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347ffff700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f348015d560 0x7f34801f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3476ffd700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cf190 0x7f34801f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347f7fe700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348006a350 0x7f34801f5760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f347000e9d0 tx=0x7f347000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347cff9700 1 -- 172.21.15.119:0/797383467 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470009720 con 0x7f348006a350 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347cff9700 1 -- 172.21.15.119:0/797383467 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347000c850 con 0x7f348006a350 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f348015f920 con 0x7f348006a350 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f347cff9700 1 -- 172.21.15.119:0/797383467 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347000cbc0 con 0x7f348006a350 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.001+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3480156820 con 0x7f348006a350 2024-09-13T16:26:02.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.005+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34800b1470 con 0x7f348006a350 2024-09-13T16:26:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.005+0000 7f347cff9700 1 -- 172.21.15.119:0/797383467 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f347000cd20 con 0x7f348006a350 2024-09-13T16:26:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.005+0000 7f347cff9700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3460066730 0x7f3460068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.005+0000 7f347cff9700 1 -- 172.21.15.119:0/797383467 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f34700845e0 con 0x7f348006a350 2024-09-13T16:26:02.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.005+0000 7f3476ffd700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3460066730 0x7f3460068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.005+0000 7f3476ffd700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3460066730 0x7f3460068bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3468009fd0 tx=0x7f3468008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.013+0000 7f347cff9700 1 -- 172.21.15.119:0/797383467 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3470051660 con 0x7f348006a350 2024-09-13T16:26:02.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.173+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f348006cc00 con 0x7f348006a350 2024-09-13T16:26:02.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.173+0000 7f347cff9700 1 -- 172.21.15.119:0/797383467 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3470054c80 con 0x7f348006a350 2024-09-13T16:26:02.179 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3460066730 msgr2=0x7f3460068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3460066730 0x7f3460068bf0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3468009fd0 tx=0x7f3468008030 comp rx=0 tx=0).stop 2024-09-13T16:26:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348006a350 msgr2=0x7f34801f5760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348006a350 0x7f34801f5760 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f347000e9d0 tx=0x7f347000ed90 comp rx=0 tx=0).stop 2024-09-13T16:26:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 shutdown_connections 2024-09-13T16:26:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3460066730 0x7f3460068bf0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f348015d560 0x7f34801f61e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cf190 0x7f34801f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 --2- 172.21.15.119:0/797383467 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f348006a350 0x7f34801f5760 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 >> 172.21.15.119:0/797383467 conn(0x7f348006b460 msgr2=0x7f3480160be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:02.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 shutdown_connections 2024-09-13T16:26:02.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.177+0000 7f3484b03700 1 -- 172.21.15.119:0/797383467 wait complete. 2024-09-13T16:26:02.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:02.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.313+0000 7fce17027700 1 Processor -- start 2024-09-13T16:26:02.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.313+0000 7fce17027700 1 -- start start 2024-09-13T16:26:02.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fce1015cc00 0x7fce100d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fce100d3800 0x7fce100ce1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100cf250 0x7fce100cf6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce100ceb30 con 0x7fce1015cc00 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce100d1b80 con 0x7fce100d3800 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce100cfbe0 con 0x7fce100cf250 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fce1015cc00 0x7fce100d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fce1015cc00 0x7fce100d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43736/0 (socket says 172.21.15.119:43736) 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16826700 1 --2- 172.21.15.119:0/3581783034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100cf250 0x7fce100cf6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 -- 172.21.15.119:0/3581783034 learned_addr learned my addr 172.21.15.119:0/3581783034 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:02.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce15824700 1 --2- 172.21.15.119:0/3581783034 >> v2:172.21.15.145:3300/0 conn(0x7fce100d3800 0x7fce100ce1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 -- 172.21.15.119:0/3581783034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100cf250 msgr2=0x7fce100cf6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 --2- 172.21.15.119:0/3581783034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100cf250 0x7fce100cf6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 -- 172.21.15.119:0/3581783034 >> v2:172.21.15.145:3300/0 conn(0x7fce100d3800 msgr2=0x7fce100ce1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 --2- 172.21.15.119:0/3581783034 >> v2:172.21.15.145:3300/0 conn(0x7fce100d3800 0x7fce100ce1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 -- 172.21.15.119:0/3581783034 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1015cfe0 con 0x7fce1015cc00 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce16025700 1 --2- 172.21.15.119:0/3581783034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce1015cc00 0x7fce100d1640 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fce100cfd50 tx=0x7fce0c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdaeea05d65d048 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce15023700 1 -- 172.21.15.119:0/3581783034 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c004230 con 0x7fce1015cc00 2024-09-13T16:26:02.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce15023700 1 -- 172.21.15.119:0/3581783034 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c004390 con 0x7fce1015cc00 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce15023700 1 -- 172.21.15.119:0/3581783034 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c0036c0 con 0x7fce1015cc00 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- 172.21.15.119:0/3581783034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce1015cc00 msgr2=0x7fce100d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 --2- 172.21.15.119:0/3581783034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce1015cc00 0x7fce100d1640 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fce100cfd50 tx=0x7fce0c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- 172.21.15.119:0/3581783034 shutdown_connections 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 --2- 172.21.15.119:0/3581783034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100cf250 0x7fce100cf6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 --2- 172.21.15.119:0/3581783034 >> v2:172.21.15.145:3300/0 conn(0x7fce100d3800 0x7fce100ce1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 --2- 172.21.15.119:0/3581783034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce1015cc00 0x7fce100d1640 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- 172.21.15.119:0/3581783034 >> 172.21.15.119:0/3581783034 conn(0x7fce1008c080 msgr2=0x7fce100d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- 172.21.15.119:0/3581783034 shutdown_connections 2024-09-13T16:26:02.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.317+0000 7fce17027700 1 -- 172.21.15.119:0/3581783034 wait complete. 2024-09-13T16:26:02.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 Processor -- start 2024-09-13T16:26:02.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 -- start start 2024-09-13T16:26:02.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce100cf250 0x7fce10153040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100d3800 0x7fce1015f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce1015cc00 0x7fce100b7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1015e4f0 con 0x7fce100cf250 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1015e7d0 con 0x7fce1015cc00 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce1015eab0 con 0x7fce100d3800 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce100cf250 0x7fce10153040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce100cf250 0x7fce10153040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:43738/0 (socket says 172.21.15.119:43738) 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 -- 172.21.15.119:0/2944221103 learned_addr learned my addr 172.21.15.119:0/2944221103 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16826700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce1015cc00 0x7fce100b7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce15824700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100d3800 0x7fce1015f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 -- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100d3800 msgr2=0x7fce1015f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100d3800 0x7fce1015f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 -- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce1015cc00 msgr2=0x7fce100b7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce1015cc00 0x7fce100b7a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 -- 172.21.15.119:0/2944221103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce0c009570 con 0x7fce100cf250 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16826700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce1015cc00 0x7fce100b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce16025700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce100cf250 0x7fce10153040 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009c30 tx=0x7fce0c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce02ffd700 1 -- 172.21.15.119:0/2944221103 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c0036c0 con 0x7fce100cf250 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1015ed90 con 0x7fce100cf250 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce02ffd700 1 -- 172.21.15.119:0/2944221103 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce0c011840 con 0x7fce100cf250 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce02ffd700 1 -- 172.21.15.119:0/2944221103 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce0c011a10 con 0x7fce100cf250 2024-09-13T16:26:02.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.321+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1015e1c0 con 0x7fce100cf250 2024-09-13T16:26:02.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.325+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce100cf6a0 con 0x7fce100cf250 2024-09-13T16:26:02.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.325+0000 7fce02ffd700 1 -- 172.21.15.119:0/2944221103 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fce0c011b70 con 0x7fce100cf250 2024-09-13T16:26:02.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.325+0000 7fce02ffd700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdf4066730 0x7fcdf4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.325+0000 7fce02ffd700 1 -- 172.21.15.119:0/2944221103 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fce0c085a50 con 0x7fce100cf250 2024-09-13T16:26:02.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.325+0000 7fce15824700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdf4066730 0x7fcdf4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.325+0000 7fce15824700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdf4066730 0x7fcdf4068bf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fce100b6e00 tx=0x7fce04008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.333+0000 7fce02ffd700 1 -- 172.21.15.119:0/2944221103 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fce0c052ba0 con 0x7fce100cf250 2024-09-13T16:26:02.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.505+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fce100d00a0 con 0x7fce100cf250 2024-09-13T16:26:02.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.505+0000 7fce02ffd700 1 -- 172.21.15.119:0/2944221103 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fce0c0561c0 con 0x7fce100cf250 2024-09-13T16:26:02.508 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:02.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdf4066730 msgr2=0x7fcdf4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdf4066730 0x7fcdf4068bf0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fce100b6e00 tx=0x7fce04008040 comp rx=0 tx=0).stop 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce100cf250 msgr2=0x7fce10153040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce100cf250 0x7fce10153040 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fce0c009c30 tx=0x7fce0c004770 comp rx=0 tx=0).stop 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 shutdown_connections 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdf4066730 0x7fcdf4068bf0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce1015cc00 0x7fce100b7a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce100d3800 0x7fce1015f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 --2- 172.21.15.119:0/2944221103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce100cf250 0x7fce10153040 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 >> 172.21.15.119:0/2944221103 conn(0x7fce1008c080 msgr2=0x7fce100c8650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 shutdown_connections 2024-09-13T16:26:02.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.509+0000 7fce17027700 1 -- 172.21.15.119:0/2944221103 wait complete. 2024-09-13T16:26:02.546 INFO:tasks.cephfs_test_runner:Starting test: test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-09-13T16:26:02.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:02.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 Processor -- start 2024-09-13T16:26:02.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- start start 2024-09-13T16:26:02.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f80640cd8d0 0x7f806406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80640cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80640b2200 0x7f80640b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80640b2bd0 con 0x7f80640b2200 2024-09-13T16:26:02.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80640705d0 con 0x7f80640cd8d0 2024-09-13T16:26:02.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806406d0d0 con 0x7f80640b17f0 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80640b2200 0x7f80640b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80640b2200 0x7f80640b2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44328/0 (socket says 172.21.15.119:44328) 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 -- 172.21.15.119:0/1272960093 learned_addr learned my addr 172.21.15.119:0/1272960093 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8063fff700 1 --2- 172.21.15.119:0/1272960093 >> v2:172.21.15.145:3300/0 conn(0x7f80640cd8d0 0x7f806406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80637fe700 1 --2- 172.21.15.119:0/1272960093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80640cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 -- 172.21.15.119:0/1272960093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 msgr2=0x7f80640cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 --2- 172.21.15.119:0/1272960093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80640cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 -- 172.21.15.119:0/1272960093 >> v2:172.21.15.145:3300/0 conn(0x7f80640cd8d0 msgr2=0x7f806406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 --2- 172.21.15.119:0/1272960093 >> v2:172.21.15.145:3300/0 conn(0x7f80640cd8d0 0x7f806406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 -- 172.21.15.119:0/1272960093 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806406fca0 con 0x7f80640b2200 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8063fff700 1 --2- 172.21.15.119:0/1272960093 >> v2:172.21.15.145:3300/0 conn(0x7f80640cd8d0 0x7f806406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80637fe700 1 --2- 172.21.15.119:0/1272960093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80640cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8068fe6700 1 --2- 172.21.15.119:0/1272960093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640b2200 0x7f80640b2690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8064071850 tx=0x7f8054009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9342dabb47438f4c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8062ffd700 1 -- 172.21.15.119:0/1272960093 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054011830 con 0x7f80640b2200 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8062ffd700 1 -- 172.21.15.119:0/1272960093 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8054011990 con 0x7f80640b2200 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f8062ffd700 1 -- 172.21.15.119:0/1272960093 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054011d20 con 0x7f80640b2200 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- 172.21.15.119:0/1272960093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640b2200 msgr2=0x7f80640b2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 --2- 172.21.15.119:0/1272960093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640b2200 0x7f80640b2690 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8064071850 tx=0x7f8054009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- 172.21.15.119:0/1272960093 shutdown_connections 2024-09-13T16:26:02.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 --2- 172.21.15.119:0/1272960093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640b2200 0x7f80640b2690 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 --2- 172.21.15.119:0/1272960093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80640cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 --2- 172.21.15.119:0/1272960093 >> v2:172.21.15.145:3300/0 conn(0x7f80640cd8d0 0x7f806406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- 172.21.15.119:0/1272960093 >> 172.21.15.119:0/1272960093 conn(0x7f806406a350 msgr2=0x7f806406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:02.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- 172.21.15.119:0/1272960093 shutdown_connections 2024-09-13T16:26:02.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.653+0000 7f80697e7700 1 -- 172.21.15.119:0/1272960093 wait complete. 2024-09-13T16:26:02.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 Processor -- start 2024-09-13T16:26:02.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 -- start start 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80641f67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80640b2200 0x7f80640ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640cd8d0 0x7f80640ab7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80640a8e30 con 0x7f80640cd8d0 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80640a84b0 con 0x7f80640b2200 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80640ac2b0 con 0x7f80640b17f0 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640cd8d0 0x7f80640ab7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8063fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80641f67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640cd8d0 0x7f80640ab7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44342/0 (socket says 172.21.15.119:44342) 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 -- 172.21.15.119:0/4082422081 learned_addr learned my addr 172.21.15.119:0/4082422081 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80637fe700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80640b2200 0x7f80640ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 -- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 msgr2=0x7f80641f67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80641f67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 -- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80640b2200 msgr2=0x7f80640ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80640b2200 0x7f80640ad210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 -- 172.21.15.119:0/4082422081 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805c009a70 con 0x7f80640cd8d0 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8068fe6700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640cd8d0 0x7f80640ab7f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f80640b3ee0 tx=0x7f8054004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8060ff9700 1 -- 172.21.15.119:0/4082422081 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054004450 con 0x7f80640cd8d0 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8060ff9700 1 -- 172.21.15.119:0/4082422081 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80540045b0 con 0x7f80640cd8d0 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f804c0094b0 con 0x7f80640cd8d0 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f8060ff9700 1 -- 172.21.15.119:0/4082422081 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054003900 con 0x7f80640cd8d0 2024-09-13T16:26:02.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.657+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8054009570 con 0x7f80640cd8d0 2024-09-13T16:26:02.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.661+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80640cdcb0 con 0x7f80640cd8d0 2024-09-13T16:26:02.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.661+0000 7f8060ff9700 1 -- 172.21.15.119:0/4082422081 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8054021070 con 0x7f80640cd8d0 2024-09-13T16:26:02.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.661+0000 7f8060ff9700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8044066730 0x7f8044068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.661+0000 7f8060ff9700 1 -- 172.21.15.119:0/4082422081 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f80540856e0 con 0x7f80640cd8d0 2024-09-13T16:26:02.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.661+0000 7f8063fff700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8044066730 0x7f8044068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.665+0000 7f8063fff700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8044066730 0x7f8044068bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f804c009c60 tx=0x7f804c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.669+0000 7f8060ff9700 1 -- 172.21.15.119:0/4082422081 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8054052910 con 0x7f80640cd8d0 2024-09-13T16:26:02.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.841+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80640ccf00 con 0x7f80640cd8d0 2024-09-13T16:26:02.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f8060ff9700 1 -- 172.21.15.119:0/4082422081 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+18406 (secure 0 0 0) 0x7f8054055f30 con 0x7f80640cd8d0 2024-09-13T16:26:02.847 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:02.848 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":25,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:25:21.326158+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8044066730 msgr2=0x7f8044068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8044066730 0x7f8044068bf0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f804c009c60 tx=0x7f804c009240 comp rx=0 tx=0).stop 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640cd8d0 msgr2=0x7f80640ab7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640cd8d0 0x7f80640ab7f0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f80640b3ee0 tx=0x7f8054004770 comp rx=0 tx=0).stop 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 shutdown_connections 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8044066730 0x7f8044068bf0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80640cd8d0 0x7f80640ab7f0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80640b2200 0x7f80640ad210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 --2- 172.21.15.119:0/4082422081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80640b17f0 0x7f80641f67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 >> 172.21.15.119:0/4082422081 conn(0x7f806406a350 msgr2=0x7f8064161b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:02.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 shutdown_connections 2024-09-13T16:26:02.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.845+0000 7f80697e7700 1 -- 172.21.15.119:0/4082422081 wait complete. 2024-09-13T16:26:02.868 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:02.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:02.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.985+0000 7f9ec4c4a700 1 Processor -- start 2024-09-13T16:26:02.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.985+0000 7f9ec4c4a700 1 -- start start 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9ec015cc00 0x7f9ec00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ec00d17e0 0x7f9ec00b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ec00cdb00 con 0x7f9ec00d0c20 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec00cdde0 con 0x7f9ec00d17e0 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ec00ce0c0 con 0x7f9ec015cc00 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ec00d17e0 0x7f9ec00b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9eb6ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9ec00d0c20 0x7f9ec00d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50174/0 (socket says 172.21.15.119:50174) 2024-09-13T16:26:02.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9eb6ffd700 1 -- 172.21.15.119:0/3962882880 learned_addr learned my addr 172.21.15.119:0/3962882880 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:02.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebf7fe700 1 --2- 172.21.15.119:0/3962882880 >> v2:172.21.15.145:3300/0 conn(0x7f9ec015cc00 0x7f9ec00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:02.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebffff700 1 -- 172.21.15.119:0/3962882880 >> v2:172.21.15.145:3300/0 conn(0x7f9ec015cc00 msgr2=0x7f9ec00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:02.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebffff700 1 --2- 172.21.15.119:0/3962882880 >> v2:172.21.15.145:3300/0 conn(0x7f9ec015cc00 0x7f9ec00d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebffff700 1 -- 172.21.15.119:0/3962882880 >> v1:172.21.15.119:6789/0 conn(0x7f9ec00d0c20 legacy=0x7f9ec00d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebffff700 1 -- 172.21.15.119:0/3962882880 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ec00ce3a0 con 0x7f9ec00d17e0 2024-09-13T16:26:02.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebffff700 1 --2- 172.21.15.119:0/3962882880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec00d17e0 0x7f9ec00b5960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00b5610 tx=0x7f9eb800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70b31c743b575160 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebeffd700 1 -- 172.21.15.119:0/3962882880 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb800e070 con 0x7f9ec00d17e0 2024-09-13T16:26:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebeffd700 1 -- 172.21.15.119:0/3962882880 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb800bb20 con 0x7f9ec00d17e0 2024-09-13T16:26:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ebeffd700 1 -- 172.21.15.119:0/3962882880 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8012430 con 0x7f9ec00d17e0 2024-09-13T16:26:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/3962882880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec00d17e0 msgr2=0x7f9ec00b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/3962882880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec00d17e0 0x7f9ec00b5960 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f9ec00b5610 tx=0x7f9eb800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:02.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/3962882880 shutdown_connections 2024-09-13T16:26:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/3962882880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec00d17e0 0x7f9ec00b5960 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/3962882880 >> v2:172.21.15.145:3300/0 conn(0x7f9ec015cc00 0x7f9ec00d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/3962882880 >> 172.21.15.119:0/3962882880 conn(0x7f9ec008c080 msgr2=0x7f9ec00d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/3962882880 shutdown_connections 2024-09-13T16:26:02.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.989+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/3962882880 wait complete. 2024-09-13T16:26:02.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 Processor -- start 2024-09-13T16:26:02.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 -- start start 2024-09-13T16:26:02.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ec00d0c20 0x7f9ec014fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ec00d17e0 0x7f9ec00c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec015cc00 0x7f9ec00cb790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:02.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec015f940 con 0x7f9ec015cc00 2024-09-13T16:26:02.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec015ed30 con 0x7f9ec00d17e0 2024-09-13T16:26:02.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ec4c4a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ec015efa0 con 0x7f9ec00d0c20 2024-09-13T16:26:02.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ebffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec015cc00 0x7f9ec00cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ebffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec015cc00 0x7f9ec00cb790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44358/0 (socket says 172.21.15.119:44358) 2024-09-13T16:26:03.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ebffff700 1 -- 172.21.15.119:0/684485983 learned_addr learned my addr 172.21.15.119:0/684485983 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:03.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9eb6ffd700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ec00d17e0 0x7f9ec00c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.993+0000 7f9ebf7fe700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ec00d0c20 0x7f9ec014fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebffff700 1 -- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ec00d0c20 msgr2=0x7f9ec014fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebffff700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ec00d0c20 0x7f9ec014fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebffff700 1 -- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ec00d17e0 msgr2=0x7f9ec00c91f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebffff700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ec00d17e0 0x7f9ec00c91f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebffff700 1 -- 172.21.15.119:0/684485983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb8009a70 con 0x7f9ec015cc00 2024-09-13T16:26:03.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebf7fe700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ec00d0c20 0x7f9ec014fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:03.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebffff700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec015cc00 0x7f9ec00cb790 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8013000 tx=0x7f9eb8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebcff9700 1 -- 172.21.15.119:0/684485983 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb8012430 con 0x7f9ec015cc00 2024-09-13T16:26:03.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebcff9700 1 -- 172.21.15.119:0/684485983 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eb8020070 con 0x7f9ec015cc00 2024-09-13T16:26:03.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ec015f280 con 0x7f9ec015cc00 2024-09-13T16:26:03.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebcff9700 1 -- 172.21.15.119:0/684485983 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eb80098f0 con 0x7f9ec015cc00 2024-09-13T16:26:03.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ec01f2a70 con 0x7f9ec015cc00 2024-09-13T16:26:03.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ec015fa80 con 0x7f9ec015cc00 2024-09-13T16:26:03.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:02.997+0000 7f9ebcff9700 1 -- 172.21.15.119:0/684485983 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9eb8004040 con 0x7f9ec015cc00 2024-09-13T16:26:03.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.001+0000 7f9ebcff9700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ea0066730 0x7f9ea0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.001+0000 7f9ebcff9700 1 -- 172.21.15.119:0/684485983 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f9eb8022070 con 0x7f9ec015cc00 2024-09-13T16:26:03.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.001+0000 7f9ebf7fe700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ea0066730 0x7f9ea0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.001+0000 7f9ebf7fe700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ea0066730 0x7f9ea0068bf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009f40 tx=0x7f9eb0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.005+0000 7f9ebcff9700 1 -- 172.21.15.119:0/684485983 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9ec015fa80 con 0x7f9ec015cc00 2024-09-13T16:26:03.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.177+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ec015fa80 con 0x7f9ec015cc00 2024-09-13T16:26:03.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.177+0000 7f9ebcff9700 1 -- 172.21.15.119:0/684485983 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v25) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ec015fa80 con 0x7f9ec015cc00 2024-09-13T16:26:03.181 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:03.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.177+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ea0066730 msgr2=0x7f9ea0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.177+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ea0066730 0x7f9ea0068bf0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9eb0009f40 tx=0x7f9eb0009240 comp rx=0 tx=0).stop 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec015cc00 msgr2=0x7f9ec00cb790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec015cc00 0x7f9ec00cb790 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9eb8013000 tx=0x7f9eb8004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 shutdown_connections 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ea0066730 0x7f9ea0068bf0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ec015cc00 0x7f9ec00cb790 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ec00d17e0 0x7f9ec00c91f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 --2- 172.21.15.119:0/684485983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ec00d0c20 0x7f9ec014fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 >> 172.21.15.119:0/684485983 conn(0x7f9ec008c080 msgr2=0x7f9ec00b7220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 shutdown_connections 2024-09-13T16:26:03.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.181+0000 7f9ec4c4a700 1 -- 172.21.15.119:0/684485983 wait complete. 2024-09-13T16:26:03.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:03.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.309+0000 7f4e661dd700 1 Processor -- start 2024-09-13T16:26:03.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.309+0000 7f4e661dd700 1 -- start start 2024-09-13T16:26:03.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.309+0000 7f4e661dd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e6015dbe0 0x7f4e600ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4e600b3260 0x7f4e600b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e600cd4d0 con 0x7f4e600add60 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e60071be0 con 0x7f4e600b3260 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60159af0 con 0x7f4e6015dbe0 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e6015dbe0 0x7f4e600ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e6015dbe0 0x7f4e600ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:51652/0 (socket says 172.21.15.119:51652) 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 -- 172.21.15.119:0/213421932 learned_addr learned my addr 172.21.15.119:0/213421932 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:03.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e659dc700 1 --2- 172.21.15.119:0/213421932 >> v2:172.21.15.145:3300/0 conn(0x7f4e600b3260 0x7f4e600b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 -- 172.21.15.119:0/213421932 >> v2:172.21.15.145:3300/0 conn(0x7f4e600b3260 msgr2=0x7f4e600b36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 --2- 172.21.15.119:0/213421932 >> v2:172.21.15.145:3300/0 conn(0x7f4e600b3260 0x7f4e600b36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 -- 172.21.15.119:0/213421932 >> v1:172.21.15.145:6789/0 conn(0x7f4e600add60 legacy=0x7f4e600cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 -- 172.21.15.119:0/213421932 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e6015e130 con 0x7f4e6015dbe0 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e649da700 1 --2- 172.21.15.119:0/213421932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e6015dbe0 0x7f4e600ce0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009e50 tx=0x7f4e5000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d517468c81d7b52e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e57fff700 1 -- 172.21.15.119:0/213421932 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000e070 con 0x7f4e6015dbe0 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e57fff700 1 -- 172.21.15.119:0/213421932 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5000b910 con 0x7f4e6015dbe0 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e57fff700 1 -- 172.21.15.119:0/213421932 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5000bcc0 con 0x7f4e6015dbe0 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- 172.21.15.119:0/213421932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e6015dbe0 msgr2=0x7f4e600ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 --2- 172.21.15.119:0/213421932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e6015dbe0 0x7f4e600ce0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009e50 tx=0x7f4e5000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- 172.21.15.119:0/213421932 shutdown_connections 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 --2- 172.21.15.119:0/213421932 >> v2:172.21.15.145:3300/0 conn(0x7f4e600b3260 0x7f4e600b36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 --2- 172.21.15.119:0/213421932 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e6015dbe0 0x7f4e600ce0f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- 172.21.15.119:0/213421932 >> 172.21.15.119:0/213421932 conn(0x7f4e600b0980 msgr2=0x7f4e600b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:03.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- 172.21.15.119:0/213421932 shutdown_connections 2024-09-13T16:26:03.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- 172.21.15.119:0/213421932 wait complete. 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 Processor -- start 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.313+0000 7f4e661dd700 1 -- start start 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e600add60 0x7f4e6014bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e600b3260 0x7f4e60150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e6015dbe0 0x7f4e60150ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60135c90 con 0x7f4e600add60 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60151490 con 0x7f4e6015dbe0 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e60151770 con 0x7f4e600b3260 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e600add60 0x7f4e6014bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e649da700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e600b3260 0x7f4e60150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e600add60 0x7f4e6014bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44362/0 (socket says 172.21.15.119:44362) 2024-09-13T16:26:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e649da700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e600b3260 0x7f4e60150960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:51666/0 (socket says 172.21.15.119:51666) 2024-09-13T16:26:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e659dc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e6015dbe0 0x7f4e60150ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 -- 172.21.15.119:0/2440171189 learned_addr learned my addr 172.21.15.119:0/2440171189 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 -- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e600b3260 msgr2=0x7f4e60150960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e600b3260 0x7f4e60150960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 -- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e6015dbe0 msgr2=0x7f4e60150ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e6015dbe0 0x7f4e60150ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 -- 172.21.15.119:0/2440171189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e50009b30 con 0x7f4e600add60 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e651db700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e600add60 0x7f4e6014bb90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00c410 tx=0x7f4e5c00c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e55ffb700 1 -- 172.21.15.119:0/2440171189 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c0052d0 con 0x7f4e600add60 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e55ffb700 1 -- 172.21.15.119:0/2440171189 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e5c005430 con 0x7f4e600add60 2024-09-13T16:26:03.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e55ffb700 1 -- 172.21.15.119:0/2440171189 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e5c0055a0 con 0x7f4e600add60 2024-09-13T16:26:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e60151a50 con 0x7f4e600add60 2024-09-13T16:26:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e6014c0d0 con 0x7f4e600add60 2024-09-13T16:26:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.317+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e600ae140 con 0x7f4e600add60 2024-09-13T16:26:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.321+0000 7f4e55ffb700 1 -- 172.21.15.119:0/2440171189 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4e5c0036a0 con 0x7f4e600add60 2024-09-13T16:26:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.321+0000 7f4e55ffb700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e4c066730 0x7f4e4c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.321+0000 7f4e55ffb700 1 -- 172.21.15.119:0/2440171189 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f4e5c0834f0 con 0x7f4e600add60 2024-09-13T16:26:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.321+0000 7f4e649da700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e4c066730 0x7f4e4c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.321+0000 7f4e649da700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e4c066730 0x7f4e4c068bf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009e50 tx=0x7f4e50009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.325+0000 7f4e55ffb700 1 -- 172.21.15.119:0/2440171189 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4e5c050600 con 0x7f4e600add60 2024-09-13T16:26:03.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.501+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4e60155c70 con 0x7f4e600add60 2024-09-13T16:26:03.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.501+0000 7f4e55ffb700 1 -- 172.21.15.119:0/2440171189 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v25) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4e5c053c20 con 0x7f4e600add60 2024-09-13T16:26:03.506 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:03.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e4c066730 msgr2=0x7f4e4c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e4c066730 0x7f4e4c068bf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4e50009e50 tx=0x7f4e50009000 comp rx=0 tx=0).stop 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e600add60 msgr2=0x7f4e6014bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e600add60 0x7f4e6014bb90 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c00c410 tx=0x7f4e5c00c720 comp rx=0 tx=0).stop 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 shutdown_connections 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e4c066730 0x7f4e4c068bf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e6015dbe0 0x7f4e60150ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e600b3260 0x7f4e60150960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 --2- 172.21.15.119:0/2440171189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e600add60 0x7f4e6014bb90 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 >> 172.21.15.119:0/2440171189 conn(0x7f4e600b0980 msgr2=0x7f4e600cd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 shutdown_connections 2024-09-13T16:26:03.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.505+0000 7f4e661dd700 1 -- 172.21.15.119:0/2440171189 wait complete. 2024-09-13T16:26:03.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5af190700 1 Processor -- start 2024-09-13T16:26:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5af190700 1 -- start start 2024-09-13T16:26:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5af190700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe5a815dbe0 0x7fe5a80ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5af190700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe5a80b3260 0x7fe5a80b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5af190700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5a80cd4d0 con 0x7fe5a80add60 2024-09-13T16:26:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5af190700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a8071be0 con 0x7fe5a815dbe0 2024-09-13T16:26:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5af190700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5a8159af0 con 0x7fe5a80b3260 2024-09-13T16:26:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ad98d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe5a815dbe0 0x7fe5a80ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ae18e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe5a80add60 0x7fe5a80cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50188/0 (socket says 172.21.15.119:50188) 2024-09-13T16:26:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ae18e700 1 -- 172.21.15.119:0/4067218732 learned_addr learned my addr 172.21.15.119:0/4067218732 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ae98f700 1 --2- 172.21.15.119:0/4067218732 >> v2:172.21.15.145:3300/0 conn(0x7fe5a80b3260 0x7fe5a80b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ad98d700 1 -- 172.21.15.119:0/4067218732 >> v2:172.21.15.145:3300/0 conn(0x7fe5a80b3260 msgr2=0x7fe5a80b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ad98d700 1 --2- 172.21.15.119:0/4067218732 >> v2:172.21.15.145:3300/0 conn(0x7fe5a80b3260 0x7fe5a80b3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ad98d700 1 -- 172.21.15.119:0/4067218732 >> v1:172.21.15.119:6789/0 conn(0x7fe5a80add60 legacy=0x7fe5a80cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.633+0000 7fe5ad98d700 1 -- 172.21.15.119:0/4067218732 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a815e130 con 0x7fe5a815dbe0 2024-09-13T16:26:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5ad98d700 1 --2- 172.21.15.119:0/4067218732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a80ce0f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe5a806fa90 tx=0x7fe5a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e8ea056f15214d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5ad18c700 1 -- 172.21.15.119:0/4067218732 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a400e070 con 0x7fe5a815dbe0 2024-09-13T16:26:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5ad18c700 1 -- 172.21.15.119:0/4067218732 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a400bb20 con 0x7fe5a815dbe0 2024-09-13T16:26:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5ad18c700 1 -- 172.21.15.119:0/4067218732 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a4012430 con 0x7fe5a815dbe0 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 -- 172.21.15.119:0/4067218732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 msgr2=0x7fe5a80ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 --2- 172.21.15.119:0/4067218732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a80ce0f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fe5a806fa90 tx=0x7fe5a400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 -- 172.21.15.119:0/4067218732 shutdown_connections 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 --2- 172.21.15.119:0/4067218732 >> v2:172.21.15.145:3300/0 conn(0x7fe5a80b3260 0x7fe5a80b3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 --2- 172.21.15.119:0/4067218732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a80ce0f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 -- 172.21.15.119:0/4067218732 >> 172.21.15.119:0/4067218732 conn(0x7fe5a80b0980 msgr2=0x7fe5a80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 -- 172.21.15.119:0/4067218732 shutdown_connections 2024-09-13T16:26:03.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.637+0000 7fe5af190700 1 -- 172.21.15.119:0/4067218732 wait complete. 2024-09-13T16:26:03.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 Processor -- start 2024-09-13T16:26:03.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 -- start start 2024-09-13T16:26:03.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a80add60 0x7fe5a815c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a80b3260 0x7fe5a81f0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a8152930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a81f0770 con 0x7fe5a815dbe0 2024-09-13T16:26:03.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a8153870 con 0x7fe5a80add60 2024-09-13T16:26:03.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5af190700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5a81f0120 con 0x7fe5a80b3260 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ae98f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a8152930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ae18e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a80add60 0x7fe5a815c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ae18e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a80add60 0x7fe5a815c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:33220/0 (socket says 172.21.15.119:33220) 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ae18e700 1 -- 172.21.15.119:0/2354618476 learned_addr learned my addr 172.21.15.119:0/2354618476 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ae98f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a8152930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44382/0 (socket says 172.21.15.119:44382) 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ad98d700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a80b3260 0x7fe5a81f0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ae98f700 1 -- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a80b3260 msgr2=0x7fe5a81f0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.641+0000 7fe5ae98f700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a80b3260 0x7fe5a81f0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe5ae98f700 1 -- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a80add60 msgr2=0x7fe5a815c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe5ae98f700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a80add60 0x7fe5a815c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe5ae98f700 1 -- 172.21.15.119:0/2354618476 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a4009a70 con 0x7fe5a815dbe0 2024-09-13T16:26:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe5ae98f700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a8152930 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000e860 tx=0x7fe5a000eb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe59effd700 1 -- 172.21.15.119:0/2354618476 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0013680 con 0x7fe5a815dbe0 2024-09-13T16:26:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe59effd700 1 -- 172.21.15.119:0/2354618476 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5a00137e0 con 0x7fe5a815dbe0 2024-09-13T16:26:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5a81f0400 con 0x7fe5a815dbe0 2024-09-13T16:26:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe59effd700 1 -- 172.21.15.119:0/2354618476 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0013b90 con 0x7fe5a815dbe0 2024-09-13T16:26:03.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5a8153100 con 0x7fe5a815dbe0 2024-09-13T16:26:03.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe59effd700 1 -- 172.21.15.119:0/2354618476 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe5a0013cf0 con 0x7fe5a815dbe0 2024-09-13T16:26:03.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5a81f08e0 con 0x7fe5a815dbe0 2024-09-13T16:26:03.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.645+0000 7fe59effd700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066760 0x7fe58c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.649+0000 7fe59effd700 1 -- 172.21.15.119:0/2354618476 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fe5a0084350 con 0x7fe5a815dbe0 2024-09-13T16:26:03.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.649+0000 7fe5ae18e700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066760 0x7fe58c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.649+0000 7fe5ae18e700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066760 0x7fe58c068c20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe598005930 tx=0x7fe598009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.657+0000 7fe59effd700 1 -- 172.21.15.119:0/2354618476 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe5a00513e0 con 0x7fe5a815dbe0 2024-09-13T16:26:03.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.825+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe5a80cda70 con 0x7fe5a815dbe0 2024-09-13T16:26:03.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.825+0000 7fe59effd700 1 -- 172.21.15.119:0/2354618476 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v25) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe5a0054a00 con 0x7fe5a815dbe0 2024-09-13T16:26:03.831 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066760 msgr2=0x7fe58c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066760 0x7fe58c068c20 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fe598005930 tx=0x7fe598009040 comp rx=0 tx=0).stop 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 msgr2=0x7fe5a8152930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a8152930 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe5a000e860 tx=0x7fe5a000eb70 comp rx=0 tx=0).stop 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 shutdown_connections 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe58c066760 0x7fe58c068c20 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5a815dbe0 0x7fe5a8152930 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5a80b3260 0x7fe5a81f0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 --2- 172.21.15.119:0/2354618476 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5a80add60 0x7fe5a815c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 >> 172.21.15.119:0/2354618476 conn(0x7fe5a80b0980 msgr2=0x7fe5a80b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:03.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 shutdown_connections 2024-09-13T16:26:03.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.829+0000 7fe5af190700 1 -- 172.21.15.119:0/2354618476 wait complete. 2024-09-13T16:26:03.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:03.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe63fff700 1 Processor -- start 2024-09-13T16:26:03.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe63fff700 1 -- start start 2024-09-13T16:26:03.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe63fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbe6415e010 0x7fbe6415b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe63fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6415cb70 con 0x7fbe640b4740 2024-09-13T16:26:03.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe63fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe6415bf60 con 0x7fbe640b4360 2024-09-13T16:26:03.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe63fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe6415c240 con 0x7fbe6415e010 2024-09-13T16:26:03.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe637fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbe640b4740 0x7fbe641645a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50202/0 (socket says 172.21.15.119:50202) 2024-09-13T16:26:03.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe637fe700 1 -- 172.21.15.119:0/3970023303 learned_addr learned my addr 172.21.15.119:0/3970023303 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:03.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe627fc700 1 --2- 172.21.15.119:0/3970023303 >> v2:172.21.15.145:3300/0 conn(0x7fbe6415e010 0x7fbe6415b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe61ffb700 1 -- 172.21.15.119:0/3970023303 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 440567169 0 0) 0x7fbe6415cb70 con 0x7fbe640b4740 2024-09-13T16:26:03.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe61ffb700 1 -- 172.21.15.119:0/3970023303 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4c0035d0 con 0x7fbe640b4740 2024-09-13T16:26:03.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe627fc700 1 -- 172.21.15.119:0/3970023303 >> v1:172.21.15.145:6789/0 conn(0x7fbe640b4360 legacy=0x7fbe640b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe61ffb700 1 -- 172.21.15.119:0/3970023303 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688524468 0 0) 0x7fbe6415bf60 con 0x7fbe640b4360 2024-09-13T16:26:03.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe627fc700 1 -- 172.21.15.119:0/3970023303 >> v1:172.21.15.119:6789/0 conn(0x7fbe640b4740 legacy=0x7fbe641645a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.969+0000 7fbe627fc700 1 -- 172.21.15.119:0/3970023303 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe6415c520 con 0x7fbe6415e010 2024-09-13T16:26:03.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe627fc700 1 --2- 172.21.15.119:0/3970023303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6415e010 0x7fbe6415b910 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009880 tx=0x7fbe54009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af9cbdc5f86dee8f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe61ffb700 1 -- 172.21.15.119:0/3970023303 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54011830 con 0x7fbe6415e010 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe61ffb700 1 -- 172.21.15.119:0/3970023303 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe54011990 con 0x7fbe6415e010 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe61ffb700 1 -- 172.21.15.119:0/3970023303 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe54011d00 con 0x7fbe6415e010 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 -- 172.21.15.119:0/3970023303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6415e010 msgr2=0x7fbe6415b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 --2- 172.21.15.119:0/3970023303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6415e010 0x7fbe6415b910 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbe54009880 tx=0x7fbe54009b90 comp rx=0 tx=0).stop 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 -- 172.21.15.119:0/3970023303 shutdown_connections 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 --2- 172.21.15.119:0/3970023303 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe6415e010 0x7fbe6415b910 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 -- 172.21.15.119:0/3970023303 >> 172.21.15.119:0/3970023303 conn(0x7fbe64072240 msgr2=0x7fbe64099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:03.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 -- 172.21.15.119:0/3970023303 shutdown_connections 2024-09-13T16:26:03.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 -- 172.21.15.119:0/3970023303 wait complete. 2024-09-13T16:26:03.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 Processor -- start 2024-09-13T16:26:03.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.973+0000 7fbe63fff700 1 -- start start 2024-09-13T16:26:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe640b4360 0x7fbe641f3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe640b4740 0x7fbe6414e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe6415e010 0x7fbe64153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64152810 con 0x7fbe6415e010 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64154ab0 con 0x7fbe640b4360 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641f3570 con 0x7fbe640b4740 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe6415e010 0x7fbe64153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe6415e010 0x7fbe64153800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44394/0 (socket says 172.21.15.119:44394) 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe627fc700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe640b4740 0x7fbe6414e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 -- 172.21.15.119:0/1971048880 learned_addr learned my addr 172.21.15.119:0/1971048880 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe62ffd700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe640b4360 0x7fbe641f3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 -- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe640b4740 msgr2=0x7fbe6414e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe640b4740 0x7fbe6414e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 -- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe640b4360 msgr2=0x7fbe641f3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe640b4360 0x7fbe641f3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:03.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 -- 172.21.15.119:0/1971048880 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe540094b0 con 0x7fbe6415e010 2024-09-13T16:26:03.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe637fe700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe6415e010 0x7fbe64153800 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00cb40 tx=0x7fbe5c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe4b7fe700 1 -- 172.21.15.119:0/1971048880 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c007500 con 0x7fbe6415e010 2024-09-13T16:26:03.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe4b7fe700 1 -- 172.21.15.119:0/1971048880 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe5c007660 con 0x7fbe6415e010 2024-09-13T16:26:03.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe640b0c20 con 0x7fbe6415e010 2024-09-13T16:26:03.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe4b7fe700 1 -- 172.21.15.119:0/1971048880 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c0079f0 con 0x7fbe6415e010 2024-09-13T16:26:03.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe64152150 con 0x7fbe6415e010 2024-09-13T16:26:03.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.977+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe640d18a0 con 0x7fbe6415e010 2024-09-13T16:26:03.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.981+0000 7fbe4b7fe700 1 -- 172.21.15.119:0/1971048880 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbe5c01b3b0 con 0x7fbe6415e010 2024-09-13T16:26:03.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.981+0000 7fbe4b7fe700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe50066730 0x7fbe50068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:03.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.981+0000 7fbe4b7fe700 1 -- 172.21.15.119:0/1971048880 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7fbe5c083dc0 con 0x7fbe6415e010 2024-09-13T16:26:03.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.981+0000 7fbe62ffd700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe50066730 0x7fbe50068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:03.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.981+0000 7fbe62ffd700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe50066730 0x7fbe50068bf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe58000f60 tx=0x7fbe58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:03.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:03.989+0000 7fbe4b7fe700 1 -- 172.21.15.119:0/1971048880 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbe5c050ed0 con 0x7fbe6415e010 2024-09-13T16:26:04.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.157+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fbe64152950 con 0x7fbe6415e010 2024-09-13T16:26:04.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.157+0000 7fbe4b7fe700 1 -- 172.21.15.119:0/1971048880 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v25) v1 ==== 103+0+11 (secure 0 0 0) 0x7fbe5c017020 con 0x7fbe6415e010 2024-09-13T16:26:04.163 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:04.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe50066730 msgr2=0x7fbe50068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:04.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe50066730 0x7fbe50068bf0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbe58000f60 tx=0x7fbe58009040 comp rx=0 tx=0).stop 2024-09-13T16:26:04.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe6415e010 msgr2=0x7fbe64153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:04.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe6415e010 0x7fbe64153800 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00cb40 tx=0x7fbe5c00cb70 comp rx=0 tx=0).stop 2024-09-13T16:26:04.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 shutdown_connections 2024-09-13T16:26:04.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe50066730 0x7fbe50068bf0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe6415e010 0x7fbe64153800 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe640b4740 0x7fbe6414e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 --2- 172.21.15.119:0/1971048880 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe640b4360 0x7fbe641f3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 >> 172.21.15.119:0/1971048880 conn(0x7fbe64072240 msgr2=0x7fbe640b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:04.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 shutdown_connections 2024-09-13T16:26:04.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.161+0000 7fbe63fff700 1 -- 172.21.15.119:0/1971048880 wait complete. 2024-09-13T16:26:04.178 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-09-13T16:26:04.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec359e700 1 Processor -- start 2024-09-13T16:26:04.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec359e700 1 -- start start 2024-09-13T16:26:04.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3ec40cd8d0 0x7f3ec406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:04.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40b2200 0x7f3ec40b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:04.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec359e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec40705d0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec40cbf10 con 0x7f3ec40cd8d0 2024-09-13T16:26:04.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40b2bb0 con 0x7f3ec40b2200 2024-09-13T16:26:04.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.285+0000 7f3ec259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3ec40cd8d0 0x7f3ec406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec1d9b700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f3ec40b17f0 0x7f3ec40cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:54976/0 (socket says 172.21.15.119:54976) 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3ec40cd8d0 0x7f3ec406cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44408/0 (socket says 172.21.15.119:44408) 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec259c700 1 -- 172.21.15.119:0/1423462875 learned_addr learned my addr 172.21.15.119:0/1423462875 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec2d9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40b2200 0x7f3ec40b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec159a700 1 -- 172.21.15.119:0/1423462875 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178147220 0 0) 0x7f3ec40705d0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec159a700 1 -- 172.21.15.119:0/1423462875 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3eb00035d0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec259c700 1 -- 172.21.15.119:0/1423462875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40b2200 msgr2=0x7f3ec40b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec259c700 1 --2- 172.21.15.119:0/1423462875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40b2200 0x7f3ec40b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec259c700 1 -- 172.21.15.119:0/1423462875 >> v1:172.21.15.145:6789/0 conn(0x7f3ec40b17f0 legacy=0x7f3ec40cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec259c700 1 -- 172.21.15.119:0/1423462875 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec406d0d0 con 0x7f3ec40cd8d0 2024-09-13T16:26:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec2d9d700 1 --2- 172.21.15.119:0/1423462875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40b2200 0x7f3ec40b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec259c700 1 --2- 172.21.15.119:0/1423462875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40cd8d0 0x7f3ec406cb90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009940 tx=0x7f3eac009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f83d534ca35e4c07 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec159a700 1 -- 172.21.15.119:0/1423462875 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac011830 con 0x7f3ec40cd8d0 2024-09-13T16:26:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec159a700 1 -- 172.21.15.119:0/1423462875 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac011990 con 0x7f3ec40cd8d0 2024-09-13T16:26:04.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec159a700 1 -- 172.21.15.119:0/1423462875 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac011d20 con 0x7f3ec40cd8d0 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 -- 172.21.15.119:0/1423462875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40cd8d0 msgr2=0x7f3ec406cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1423462875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40cd8d0 0x7f3ec406cb90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009940 tx=0x7f3eac009c50 comp rx=0 tx=0).stop 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 -- 172.21.15.119:0/1423462875 shutdown_connections 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1423462875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40b2200 0x7f3ec40b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1423462875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40cd8d0 0x7f3ec406cb90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 -- 172.21.15.119:0/1423462875 >> 172.21.15.119:0/1423462875 conn(0x7f3ec406a350 msgr2=0x7f3ec406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 -- 172.21.15.119:0/1423462875 shutdown_connections 2024-09-13T16:26:04.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.289+0000 7f3ec359e700 1 -- 172.21.15.119:0/1423462875 wait complete. 2024-09-13T16:26:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 Processor -- start 2024-09-13T16:26:04.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 -- start start 2024-09-13T16:26:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40b17f0 0x7f3ec40ad400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ec40b2200 0x7f3ec415c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:04.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40cd8d0 0x7f3ec40aa8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40a8550 con 0x7f3ec40b17f0 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40a8830 con 0x7f3ec40b2200 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec40a8b10 con 0x7f3ec40cd8d0 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40b17f0 0x7f3ec40ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40b17f0 0x7f3ec40ad400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44424/0 (socket says 172.21.15.119:44424) 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 -- 172.21.15.119:0/1543114752 learned_addr learned my addr 172.21.15.119:0/1543114752 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec2d9d700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40cd8d0 0x7f3ec40aa8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec1d9b700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ec40b2200 0x7f3ec415c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 -- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40cd8d0 msgr2=0x7f3ec40aa8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40cd8d0 0x7f3ec40aa8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 -- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ec40b2200 msgr2=0x7f3ec415c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ec40b2200 0x7f3ec415c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:04.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 -- 172.21.15.119:0/1543114752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc009a70 con 0x7f3ec40b17f0 2024-09-13T16:26:04.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec1d9b700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ec40b2200 0x7f3ec415c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:04.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec259c700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40b17f0 0x7f3ec40ad400 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009fd0 tx=0x7f3eac004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:04.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ebaffd700 1 -- 172.21.15.119:0/1543114752 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac0044d0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ebaffd700 1 -- 172.21.15.119:0/1543114752 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac0036c0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.293+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eac009570 con 0x7f3ec40b17f0 2024-09-13T16:26:04.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ebaffd700 1 -- 172.21.15.119:0/1543114752 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac003a30 con 0x7f3ec40b17f0 2024-09-13T16:26:04.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec40a90d0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ebaffd700 1 -- 172.21.15.119:0/1543114752 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3eac003b90 con 0x7f3ec40b17f0 2024-09-13T16:26:04.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec40a92c0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ebaffd700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ea4066730 0x7f3ea4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:04.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ebaffd700 1 -- 172.21.15.119:0/1543114752 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6460+0+0 (secure 0 0 0) 0x7f3eac0856d0 con 0x7f3ec40b17f0 2024-09-13T16:26:04.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ec1d9b700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ea4066730 0x7f3ea4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:04.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.297+0000 7f3ec1d9b700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ea4066730 0x7f3ea4068bf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4071190 tx=0x7f3eb4007cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:04.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.305+0000 7f3ebaffd700 1 -- 172.21.15.119:0/1543114752 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3eac052760 con 0x7f3ec40b17f0 2024-09-13T16:26:04.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:04.545+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f3ec40cdcb0 con 0x7f3ec40b17f0 2024-09-13T16:26:05.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.569+0000 7f3ebaffd700 1 -- 172.21.15.119:0/1543114752 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v62) v1 ==== 145+0+0 (secure 0 0 0) 0x7f3eac05c070 con 0x7f3ec40b17f0 2024-09-13T16:26:05.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ea4066730 msgr2=0x7f3ea4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ea4066730 0x7f3ea4068bf0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3ec4071190 tx=0x7f3eb4007cc0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40b17f0 msgr2=0x7f3ec40ad400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40b17f0 0x7f3ec40ad400 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3eac009fd0 tx=0x7f3eac004770 comp rx=0 tx=0).stop 2024-09-13T16:26:05.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 shutdown_connections 2024-09-13T16:26:05.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ea4066730 0x7f3ea4068bf0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ec40cd8d0 0x7f3ec40aa8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ec40b2200 0x7f3ec415c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 --2- 172.21.15.119:0/1543114752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ec40b17f0 0x7f3ec40ad400 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 >> 172.21.15.119:0/1543114752 conn(0x7f3ec406a350 msgr2=0x7f3ec41621f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:05.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 shutdown_connections 2024-09-13T16:26:05.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.573+0000 7f3ec359e700 1 -- 172.21.15.119:0/1543114752 wait complete. 2024-09-13T16:26:05.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T16:26:05.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa194490700 1 Processor -- start 2024-09-13T16:26:05.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa194490700 1 -- start start 2024-09-13T16:26:05.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa194490700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa18c15cf80 0x7fa18c0d1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:05.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa194490700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa18c0d3890 0x7fa18c0cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:05.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa194490700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa18c0cd910 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa194490700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c15d360 con 0x7fa18c0d3890 2024-09-13T16:26:05.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa194490700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa18c0ce750 con 0x7fa18c15cf80 2024-09-13T16:26:05.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa191a2b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa18c0d3890 0x7fa18c0cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:05.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa192a2d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa18c0cfcc0 0x7fa18c1609d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50218/0 (socket says 172.21.15.119:50218) 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa192a2d700 1 -- 172.21.15.119:0/2605605762 learned_addr learned my addr 172.21.15.119:0/2605605762 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa19222c700 1 --2- 172.21.15.119:0/2605605762 >> v2:172.21.15.145:3300/0 conn(0x7fa18c15cf80 0x7fa18c0d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa191a2b700 1 -- 172.21.15.119:0/2605605762 >> v2:172.21.15.145:3300/0 conn(0x7fa18c15cf80 msgr2=0x7fa18c0d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa191a2b700 1 --2- 172.21.15.119:0/2605605762 >> v2:172.21.15.145:3300/0 conn(0x7fa18c15cf80 0x7fa18c0d1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa191a2b700 1 -- 172.21.15.119:0/2605605762 >> v1:172.21.15.119:6789/0 conn(0x7fa18c0cfcc0 legacy=0x7fa18c1609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa191a2b700 1 -- 172.21.15.119:0/2605605762 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa18c0d00f0 con 0x7fa18c0d3890 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa19222c700 1 --2- 172.21.15.119:0/2605605762 >> v2:172.21.15.145:3300/0 conn(0x7fa18c15cf80 0x7fa18c0d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa19122a700 1 -- 172.21.15.119:0/2605605762 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882699709 0 0) 0x7fa18c0cd910 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa191a2b700 1 --2- 172.21.15.119:0/2605605762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0d3890 0x7fa18c0cf780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa18c15d4a0 tx=0x7fa17c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cec00cce0a14821c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:05.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa19122a700 1 -- 172.21.15.119:0/2605605762 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c004020 con 0x7fa18c0d3890 2024-09-13T16:26:05.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa19122a700 1 -- 172.21.15.119:0/2605605762 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa17c004180 con 0x7fa18c0d3890 2024-09-13T16:26:05.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.705+0000 7fa19122a700 1 -- 172.21.15.119:0/2605605762 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa17c004530 con 0x7fa18c0d3890 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 -- 172.21.15.119:0/2605605762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0d3890 msgr2=0x7fa18c0cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 --2- 172.21.15.119:0/2605605762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0d3890 0x7fa18c0cf780 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa18c15d4a0 tx=0x7fa17c009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 -- 172.21.15.119:0/2605605762 shutdown_connections 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 --2- 172.21.15.119:0/2605605762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0d3890 0x7fa18c0cf780 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 --2- 172.21.15.119:0/2605605762 >> v2:172.21.15.145:3300/0 conn(0x7fa18c15cf80 0x7fa18c0d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 -- 172.21.15.119:0/2605605762 >> 172.21.15.119:0/2605605762 conn(0x7fa18c0d2d70 msgr2=0x7fa18c0d3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 -- 172.21.15.119:0/2605605762 shutdown_connections 2024-09-13T16:26:05.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 -- 172.21.15.119:0/2605605762 wait complete. 2024-09-13T16:26:05.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 Processor -- start 2024-09-13T16:26:05.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.709+0000 7fa194490700 1 -- start start 2024-09-13T16:26:05.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0cfcc0 0x7fa18c09e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:05.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c0d3890 0x7fa18c0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:05.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c15cf80 0x7fa18c159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:05.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0cd1b0 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c15a370 con 0x7fa18c15cf80 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa18c0cae90 con 0x7fa18c0d3890 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0cfcc0 0x7fa18c09e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0cfcc0 0x7fa18c09e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44442/0 (socket says 172.21.15.119:44442) 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 -- 172.21.15.119:0/2238599451 learned_addr learned my addr 172.21.15.119:0/2238599451 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa192a2d700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c15cf80 0x7fa18c159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa191a2b700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c0d3890 0x7fa18c0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 -- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c0d3890 msgr2=0x7fa18c0ccc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c0d3890 0x7fa18c0ccc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 -- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c15cf80 msgr2=0x7fa18c159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c15cf80 0x7fa18c159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 -- 172.21.15.119:0/2238599451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0094b0 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa192a2d700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c15cf80 0x7fa18c159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:05.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa19222c700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0cfcc0 0x7fa18c09e880 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa18800e760 tx=0x7fa18800ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa182ffd700 1 -- 172.21.15.119:0/2238599451 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa18800c070 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa182ffd700 1 -- 172.21.15.119:0/2238599451 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa188004250 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa182ffd700 1 -- 172.21.15.119:0/2238599451 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1880036c0 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa18c09ef60 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.713+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18c09f520 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.717+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa18c0677a0 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.717+0000 7fa182ffd700 1 -- 172.21.15.119:0/2238599451 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa188003820 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.717+0000 7fa182ffd700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa170066730 0x7fa170068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:05.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.717+0000 7fa182ffd700 1 -- 172.21.15.119:0/2238599451 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa188084220 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.717+0000 7fa191a2b700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa170066730 0x7fa170068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:05.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.717+0000 7fa191a2b700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa170066730 0x7fa170068bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0090b0 tx=0x7fa17c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:05.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.725+0000 7fa182ffd700 1 -- 172.21.15.119:0/2238599451 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa1880511c0 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.893+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa18c0b4570 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.901+0000 7fa182ffd700 1 -- 172.21.15.119:0/2238599451 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa1880094e0 con 0x7fa18c0cfcc0 2024-09-13T16:26:05.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa170066730 msgr2=0x7fa170068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa170066730 0x7fa170068bf0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa17c0090b0 tx=0x7fa17c008030 comp rx=0 tx=0).stop 2024-09-13T16:26:05.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0cfcc0 msgr2=0x7fa18c09e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:05.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0cfcc0 0x7fa18c09e880 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa18800e760 tx=0x7fa18800ea70 comp rx=0 tx=0).stop 2024-09-13T16:26:05.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 shutdown_connections 2024-09-13T16:26:05.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa170066730 0x7fa170068bf0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa18c15cf80 0x7fa18c159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa18c0d3890 0x7fa18c0ccc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 --2- 172.21.15.119:0/2238599451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa18c0cfcc0 0x7fa18c09e880 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 >> 172.21.15.119:0/2238599451 conn(0x7fa18c0d2d70 msgr2=0x7fa18c15c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 shutdown_connections 2024-09-13T16:26:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:05.905+0000 7fa194490700 1 -- 172.21.15.119:0/2238599451 wait complete. 2024-09-13T16:26:05.923 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:26:05.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:26:05.942 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:26:05.964 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:26:05.964 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:05.964 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:05.971 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:05.971 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:05.978 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.125+0000 7fa437ba6700 1 Processor -- start 2024-09-13T16:26:06.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.125+0000 7fa437ba6700 1 -- start start 2024-09-13T16:26:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa43015a620 0x7fa4300b3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa4300ada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4300ac960 con 0x7fa4300adf40 2024-09-13T16:26:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4300acc40 con 0x7fa43015a620 2024-09-13T16:26:06.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4300acf20 con 0x7fa4301612f0 2024-09-13T16:26:06.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa435942700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa43015a620 0x7fa4300b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa436143700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa4300adf40 0x7fa43006fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:54990/0 (socket says 172.21.15.119:54990) 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa436143700 1 -- 172.21.15.119:0/3640248041 learned_addr learned my addr 172.21.15.119:0/3640248041 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa435141700 1 --2- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa4300ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa434940700 1 -- 172.21.15.119:0/3640248041 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399569418 0 0) 0x7fa4300ac960 con 0x7fa4300adf40 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa434940700 1 -- 172.21.15.119:0/3640248041 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa41c0035d0 con 0x7fa4300adf40 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa435942700 1 -- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 msgr2=0x7fa4300ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa435942700 1 --2- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa4300ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa435942700 1 -- 172.21.15.119:0/3640248041 >> v1:172.21.15.145:6789/0 conn(0x7fa4300adf40 legacy=0x7fa43006fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa435942700 1 -- 172.21.15.119:0/3640248041 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4300ad200 con 0x7fa43015a620 2024-09-13T16:26:06.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa435942700 1 --2- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa43015a620 0x7fa4300b3720 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009850 tx=0x7fa42c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0a5141689021463 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa434940700 1 -- 172.21.15.119:0/3640248041 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c011830 con 0x7fa43015a620 2024-09-13T16:26:06.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa434940700 1 -- 172.21.15.119:0/3640248041 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c011990 con 0x7fa43015a620 2024-09-13T16:26:06.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa434940700 1 -- 172.21.15.119:0/3640248041 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa42c011d40 con 0x7fa43015a620 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa43015a620 msgr2=0x7fa4300b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa43015a620 0x7fa4300b3720 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa42c009850 tx=0x7fa42c009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- 172.21.15.119:0/3640248041 shutdown_connections 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa4300ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3640248041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa43015a620 0x7fa4300b3720 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- 172.21.15.119:0/3640248041 >> 172.21.15.119:0/3640248041 conn(0x7fa43006a350 msgr2=0x7fa43006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- 172.21.15.119:0/3640248041 shutdown_connections 2024-09-13T16:26:06.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.129+0000 7fa437ba6700 1 -- 172.21.15.119:0/3640248041 wait complete. 2024-09-13T16:26:06.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 Processor -- start 2024-09-13T16:26:06.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 -- start start 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4300adf40 0x7fa43015d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43015a620 0x7fa4300c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa43014b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43015de70 con 0x7fa43015a620 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa43015bbc0 con 0x7fa4300adf40 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa437ba6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43015bea0 con 0x7fa4301612f0 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43015a620 0x7fa4300c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435942700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4300adf40 0x7fa43015d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43015a620 0x7fa4300c60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44450/0 (socket says 172.21.15.119:44450) 2024-09-13T16:26:06.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 -- 172.21.15.119:0/3865381172 learned_addr learned my addr 172.21.15.119:0/3865381172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:06.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa436143700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa43014b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 -- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 msgr2=0x7fa43014b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa43014b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 -- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4300adf40 msgr2=0x7fa43015d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4300adf40 0x7fa43015d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 -- 172.21.15.119:0/3865381172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa42c0094b0 con 0x7fa43015a620 2024-09-13T16:26:06.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.133+0000 7fa435141700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43015a620 0x7fa4300c60a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa420009fd0 tx=0x7fa42000eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa4267fc700 1 -- 172.21.15.119:0/3865381172 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4200095a0 con 0x7fa43015a620 2024-09-13T16:26:06.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa4267fc700 1 -- 172.21.15.119:0/3865381172 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa420013680 con 0x7fa43015a620 2024-09-13T16:26:06.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43015c180 con 0x7fa43015a620 2024-09-13T16:26:06.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa4267fc700 1 -- 172.21.15.119:0/3865381172 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa420013a30 con 0x7fa43015a620 2024-09-13T16:26:06.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4300c38b0 con 0x7fa43015a620 2024-09-13T16:26:06.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43014eeb0 con 0x7fa43015a620 2024-09-13T16:26:06.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa4267fc700 1 -- 172.21.15.119:0/3865381172 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa420013b90 con 0x7fa43015a620 2024-09-13T16:26:06.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa4267fc700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa414066760 0x7fa414068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa4267fc700 1 -- 172.21.15.119:0/3865381172 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa4200843b0 con 0x7fa43015a620 2024-09-13T16:26:06.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.137+0000 7fa435942700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa414066760 0x7fa414068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.141+0000 7fa435942700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa414066760 0x7fa414068c20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0090b0 tx=0x7fa42c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.145+0000 7fa4267fc700 1 -- 172.21.15.119:0/3865381172 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa420051240 con 0x7fa43015a620 2024-09-13T16:26:06.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.317+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa430067760 con 0x7fa43015a620 2024-09-13T16:26:06.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.317+0000 7fa4267fc700 1 -- 172.21.15.119:0/3865381172 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) v1 ==== 74+0+18794 (secure 0 0 0) 0x7fa420054860 con 0x7fa43015a620 2024-09-13T16:26:06.323 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:06.323 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":26,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:05.547209+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:06.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa414066760 msgr2=0x7fa414068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa414066760 0x7fa414068c20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fa42c0090b0 tx=0x7fa42c008030 comp rx=0 tx=0).stop 2024-09-13T16:26:06.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43015a620 msgr2=0x7fa4300c60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43015a620 0x7fa4300c60a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa420009fd0 tx=0x7fa42000eb20 comp rx=0 tx=0).stop 2024-09-13T16:26:06.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 shutdown_connections 2024-09-13T16:26:06.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa414066760 0x7fa414068c20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4301612f0 0x7fa43014b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa43015a620 0x7fa4300c60a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 --2- 172.21.15.119:0/3865381172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4300adf40 0x7fa43015d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 >> 172.21.15.119:0/3865381172 conn(0x7fa43006a350 msgr2=0x7fa43006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:06.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 shutdown_connections 2024-09-13T16:26:06.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.321+0000 7fa437ba6700 1 -- 172.21.15.119:0/3865381172 wait complete. 2024-09-13T16:26:06.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:06.339 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.457+0000 7fa96dadf700 1 Processor -- start 2024-09-13T16:26:06.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.457+0000 7fa96dadf700 1 -- start start 2024-09-13T16:26:06.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.457+0000 7fa96dadf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa9681573d0 0x7fa9680aca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.457+0000 7fa96dadf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa9680aeee0 0x7fa96806a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.457+0000 7fa96dadf700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9680ad740 con 0x7fa968156ff0 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.457+0000 7fa96dadf700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa96806e890 con 0x7fa9681573d0 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.457+0000 7fa96dadf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa96815d160 con 0x7fa9680aeee0 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa9681573d0 0x7fa9680aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa9681573d0 0x7fa9680aca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44456/0 (socket says 172.21.15.119:44456) 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 -- 172.21.15.119:0/1732727263 learned_addr learned my addr 172.21.15.119:0/1732727263 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa967fff700 1 --2- 172.21.15.119:0/1732727263 >> v2:172.21.15.145:3300/0 conn(0x7fa9680aeee0 0x7fa96806a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 -- 172.21.15.119:0/1732727263 >> v2:172.21.15.145:3300/0 conn(0x7fa9680aeee0 msgr2=0x7fa96806a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 --2- 172.21.15.119:0/1732727263 >> v2:172.21.15.145:3300/0 conn(0x7fa9680aeee0 0x7fa96806a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 -- 172.21.15.119:0/1732727263 >> v1:172.21.15.145:6789/0 conn(0x7fa968156ff0 legacy=0x7fa96815db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 -- 172.21.15.119:0/1732727263 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9680ad180 con 0x7fa9681573d0 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa966ffd700 1 --2- 172.21.15.119:0/1732727263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9681573d0 0x7fa9680aca80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa9680ae440 tx=0x7fa958009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77945dc67ce171c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa9667fc700 1 -- 172.21.15.119:0/1732727263 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958011830 con 0x7fa9681573d0 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa9667fc700 1 -- 172.21.15.119:0/1732727263 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa958011990 con 0x7fa9681573d0 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa9667fc700 1 -- 172.21.15.119:0/1732727263 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa958011d20 con 0x7fa9681573d0 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 -- 172.21.15.119:0/1732727263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9681573d0 msgr2=0x7fa9680aca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1732727263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9681573d0 0x7fa9680aca80 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fa9680ae440 tx=0x7fa958009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 -- 172.21.15.119:0/1732727263 shutdown_connections 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1732727263 >> v2:172.21.15.145:3300/0 conn(0x7fa9680aeee0 0x7fa96806a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1732727263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9681573d0 0x7fa9680aca80 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 -- 172.21.15.119:0/1732727263 >> 172.21.15.119:0/1732727263 conn(0x7fa96806b460 msgr2=0x7fa96806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 -- 172.21.15.119:0/1732727263 shutdown_connections 2024-09-13T16:26:06.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.461+0000 7fa96dadf700 1 -- 172.21.15.119:0/1732727263 wait complete. 2024-09-13T16:26:06.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 Processor -- start 2024-09-13T16:26:06.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 -- start start 2024-09-13T16:26:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9680aeee0 0x7fa96814d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa968156ff0 0x7fa9681f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9681573d0 0x7fa968155be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96814e900 con 0x7fa9680aeee0 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96814dcf0 con 0x7fa968156ff0 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa96814dfd0 con 0x7fa9681573d0 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9680aeee0 0x7fa96814d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9680aeee0 0x7fa96814d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44458/0 (socket says 172.21.15.119:44458) 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 -- 172.21.15.119:0/1076218397 learned_addr learned my addr 172.21.15.119:0/1076218397 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa966ffd700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa968156ff0 0x7fa9681f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa967fff700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9681573d0 0x7fa968155be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 -- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9681573d0 msgr2=0x7fa968155be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9681573d0 0x7fa968155be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 -- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa968156ff0 msgr2=0x7fa9681f3e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa968156ff0 0x7fa9681f3e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 -- 172.21.15.119:0/1076218397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa958009570 con 0x7fa9680aeee0 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa967fff700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9681573d0 0x7fa968155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa9677fe700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9680aeee0 0x7fa96814d4a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00c500 tx=0x7fa95c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa94ffff700 1 -- 172.21.15.119:0/1076218397 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c0052d0 con 0x7fa9680aeee0 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa94ffff700 1 -- 172.21.15.119:0/1076218397 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa95c005430 con 0x7fa9680aeee0 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.465+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96814e2b0 con 0x7fa9680aeee0 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.469+0000 7fa94ffff700 1 -- 172.21.15.119:0/1076218397 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa95c0055a0 con 0x7fa9680aeee0 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.469+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa968156120 con 0x7fa9680aeee0 2024-09-13T16:26:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.469+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9680b2a50 con 0x7fa9680aeee0 2024-09-13T16:26:06.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.469+0000 7fa94ffff700 1 -- 172.21.15.119:0/1076218397 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa95c0038e0 con 0x7fa9680aeee0 2024-09-13T16:26:06.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.473+0000 7fa94ffff700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa954066730 0x7fa954068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.473+0000 7fa94ffff700 1 -- 172.21.15.119:0/1076218397 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa95c083ac0 con 0x7fa9680aeee0 2024-09-13T16:26:06.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.473+0000 7fa966ffd700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa954066730 0x7fa954068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.473+0000 7fa966ffd700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa954066730 0x7fa954068bf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa96806c270 tx=0x7fa958008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.481+0000 7fa94ffff700 1 -- 172.21.15.119:0/1076218397 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa95c0509d0 con 0x7fa9680aeee0 2024-09-13T16:26:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.657+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa96806d840 con 0x7fa9680aeee0 2024-09-13T16:26:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.657+0000 7fa94ffff700 1 -- 172.21.15.119:0/1076218397 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa95c053ff0 con 0x7fa9680aeee0 2024-09-13T16:26:06.661 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:06.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa954066730 msgr2=0x7fa954068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa954066730 0x7fa954068bf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa96806c270 tx=0x7fa958008030 comp rx=0 tx=0).stop 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9680aeee0 msgr2=0x7fa96814d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9680aeee0 0x7fa96814d4a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00c500 tx=0x7fa95c00c8c0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 shutdown_connections 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa954066730 0x7fa954068bf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa9681573d0 0x7fa968155be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa968156ff0 0x7fa9681f3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 --2- 172.21.15.119:0/1076218397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa9680aeee0 0x7fa96814d4a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 >> 172.21.15.119:0/1076218397 conn(0x7fa96806b460 msgr2=0x7fa96806e9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 shutdown_connections 2024-09-13T16:26:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.661+0000 7fa96dadf700 1 -- 172.21.15.119:0/1076218397 wait complete. 2024-09-13T16:26:06.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:06.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f263cc8b700 1 Processor -- start 2024-09-13T16:26:06.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f263cc8b700 1 -- start start 2024-09-13T16:26:06.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f263cc8b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f263815dbe0 0x7f26380ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f263cc8b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26380cd4d0 con 0x7f26380add60 2024-09-13T16:26:06.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f263cc8b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f263815e130 con 0x7f26380b3260 2024-09-13T16:26:06.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f263cc8b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2638071be0 con 0x7f263815dbe0 2024-09-13T16:26:06.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f262ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f263815dbe0 0x7f26380ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f26377fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f26380add60 0x7f26380cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50224/0 (socket says 172.21.15.119:50224) 2024-09-13T16:26:06.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.785+0000 7f26377fe700 1 -- 172.21.15.119:0/490603070 learned_addr learned my addr 172.21.15.119:0/490603070 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:06.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f262ffff700 1 -- 172.21.15.119:0/490603070 >> v1:172.21.15.145:6789/0 conn(0x7f26380b3260 legacy=0x7f26380b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f262ffff700 1 -- 172.21.15.119:0/490603070 >> v1:172.21.15.119:6789/0 conn(0x7f26380add60 legacy=0x7f26380cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f262ffff700 1 -- 172.21.15.119:0/490603070 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26380ccba0 con 0x7f263815dbe0 2024-09-13T16:26:06.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f262ffff700 1 --2- 172.21.15.119:0/490603070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f26380ce0f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f262800baf0 tx=0x7f262800be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5055297e4cdb1311 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f2636ffd700 1 -- 172.21.15.119:0/490603070 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262800dd30 con 0x7f263815dbe0 2024-09-13T16:26:06.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f2636ffd700 1 -- 172.21.15.119:0/490603070 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262800de90 con 0x7f263815dbe0 2024-09-13T16:26:06.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f2636ffd700 1 -- 172.21.15.119:0/490603070 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2628010690 con 0x7f263815dbe0 2024-09-13T16:26:06.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 -- 172.21.15.119:0/490603070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 msgr2=0x7f26380ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 --2- 172.21.15.119:0/490603070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f26380ce0f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f262800baf0 tx=0x7f262800be00 comp rx=0 tx=0).stop 2024-09-13T16:26:06.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 -- 172.21.15.119:0/490603070 shutdown_connections 2024-09-13T16:26:06.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 --2- 172.21.15.119:0/490603070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f26380ce0f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 -- 172.21.15.119:0/490603070 >> 172.21.15.119:0/490603070 conn(0x7f26380b0980 msgr2=0x7f26380b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:06.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 -- 172.21.15.119:0/490603070 shutdown_connections 2024-09-13T16:26:06.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 -- 172.21.15.119:0/490603070 wait complete. 2024-09-13T16:26:06.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 Processor -- start 2024-09-13T16:26:06.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.789+0000 7f263cc8b700 1 -- start start 2024-09-13T16:26:06.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26380add60 0x7f26380aa040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26380b3260 0x7f26380a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f2638145ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26380acf50 con 0x7f263815dbe0 2024-09-13T16:26:06.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26380ad370 con 0x7f26380b3260 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26380ac2d0 con 0x7f26380add60 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f2638145ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f2638145ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44480/0 (socket says 172.21.15.119:44480) 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 -- 172.21.15.119:0/2670590289 learned_addr learned my addr 172.21.15.119:0/2670590289 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f26377fe700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26380add60 0x7f26380aa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f262ffff700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26380b3260 0x7f26380a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 -- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26380add60 msgr2=0x7f26380aa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26380add60 0x7f26380aa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 -- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26380b3260 msgr2=0x7f26380a8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26380b3260 0x7f26380a8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 -- 172.21.15.119:0/2670590289 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262800b750 con 0x7f263815dbe0 2024-09-13T16:26:06.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2637fff700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f2638145ce0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f263000cc50 tx=0x7f263000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2634ff9700 1 -- 172.21.15.119:0/2670590289 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2630003c20 con 0x7f263815dbe0 2024-09-13T16:26:06.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2634ff9700 1 -- 172.21.15.119:0/2670590289 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2630003d80 con 0x7f263815dbe0 2024-09-13T16:26:06.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26380ac5b0 con 0x7f263815dbe0 2024-09-13T16:26:06.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f2634ff9700 1 -- 172.21.15.119:0/2670590289 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263000aab0 con 0x7f263815dbe0 2024-09-13T16:26:06.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.793+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26380acb70 con 0x7f263815dbe0 2024-09-13T16:26:06.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.797+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26380ad090 con 0x7f263815dbe0 2024-09-13T16:26:06.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.797+0000 7f2634ff9700 1 -- 172.21.15.119:0/2670590289 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f263000ac10 con 0x7f263815dbe0 2024-09-13T16:26:06.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.797+0000 7f2634ff9700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2618066730 0x7f2618068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:06.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.797+0000 7f2634ff9700 1 -- 172.21.15.119:0/2670590289 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2630084440 con 0x7f263815dbe0 2024-09-13T16:26:06.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.797+0000 7f26377fe700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2618066730 0x7f2618068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:06.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.797+0000 7f26377fe700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2618066730 0x7f2618068bf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2620005930 tx=0x7f2620009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:06.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.805+0000 7f2634ff9700 1 -- 172.21.15.119:0/2670590289 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2630051480 con 0x7f263815dbe0 2024-09-13T16:26:06.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.973+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f26380ce0f0 con 0x7f263815dbe0 2024-09-13T16:26:06.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.973+0000 7f2634ff9700 1 -- 172.21.15.119:0/2670590289 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2630054aa0 con 0x7f263815dbe0 2024-09-13T16:26:06.978 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2618066730 msgr2=0x7f2618068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2618066730 0x7f2618068bf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2620005930 tx=0x7f2620009040 comp rx=0 tx=0).stop 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 msgr2=0x7f2638145ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f2638145ce0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f263000cc50 tx=0x7f263000cf60 comp rx=0 tx=0).stop 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 shutdown_connections 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2618066730 0x7f2618068bf0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f263815dbe0 0x7f2638145ce0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26380b3260 0x7f26380a8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 --2- 172.21.15.119:0/2670590289 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26380add60 0x7f26380aa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 >> 172.21.15.119:0/2670590289 conn(0x7f26380b0980 msgr2=0x7f2638066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:06.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 shutdown_connections 2024-09-13T16:26:06.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:06.977+0000 7f263cc8b700 1 -- 172.21.15.119:0/2670590289 wait complete. 2024-09-13T16:26:06.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:07.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a9472700 1 Processor -- start 2024-09-13T16:26:07.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a9472700 1 -- start start 2024-09-13T16:26:07.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a9472700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a4159800 0x7f40a4159be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a9472700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f40a406ee00 0x7f40a40b3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a9472700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40a406f3a0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a9472700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40a40b2a80 con 0x7f40a406ee00 2024-09-13T16:26:07.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a9472700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40b2d60 con 0x7f40a4159800 2024-09-13T16:26:07.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a8c71700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f40a40b3aa0 0x7f40a40b3ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50232/0 (socket says 172.21.15.119:50232) 2024-09-13T16:26:07.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a8c71700 1 -- 172.21.15.119:0/3980496275 learned_addr learned my addr 172.21.15.119:0/3980496275 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:07.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a37fe700 1 --2- 172.21.15.119:0/3980496275 >> v2:172.21.15.145:3300/0 conn(0x7f40a406ee00 0x7f40a40b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a3fff700 1 --2- 172.21.15.119:0/3980496275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a4159800 0x7f40a4159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.101+0000 7f40a2ffd700 1 -- 172.21.15.119:0/3980496275 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2028632493 0 0) 0x7f40a406f3a0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a2ffd700 1 -- 172.21.15.119:0/3980496275 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f408c0035d0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a3fff700 1 -- 172.21.15.119:0/3980496275 >> v2:172.21.15.145:3300/0 conn(0x7f40a406ee00 msgr2=0x7f40a40b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a3fff700 1 --2- 172.21.15.119:0/3980496275 >> v2:172.21.15.145:3300/0 conn(0x7f40a406ee00 0x7f40a40b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a3fff700 1 -- 172.21.15.119:0/3980496275 >> v1:172.21.15.119:6789/0 conn(0x7f40a40b3aa0 legacy=0x7f40a40b3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a3fff700 1 -- 172.21.15.119:0/3980496275 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a40b3040 con 0x7f40a4159800 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a37fe700 1 --2- 172.21.15.119:0/3980496275 >> v2:172.21.15.145:3300/0 conn(0x7f40a406ee00 0x7f40a40b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a3fff700 1 --2- 172.21.15.119:0/3980496275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a4159800 0x7f40a4159be0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4094009850 tx=0x7f4094009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17e7e35b925fe51f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a2ffd700 1 -- 172.21.15.119:0/3980496275 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094011830 con 0x7f40a4159800 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a2ffd700 1 -- 172.21.15.119:0/3980496275 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4094011990 con 0x7f40a4159800 2024-09-13T16:26:07.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a2ffd700 1 -- 172.21.15.119:0/3980496275 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4094011d00 con 0x7f40a4159800 2024-09-13T16:26:07.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 -- 172.21.15.119:0/3980496275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a4159800 msgr2=0x7f40a4159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 --2- 172.21.15.119:0/3980496275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a4159800 0x7f40a4159be0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4094009850 tx=0x7f4094009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:07.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 -- 172.21.15.119:0/3980496275 shutdown_connections 2024-09-13T16:26:07.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 --2- 172.21.15.119:0/3980496275 >> v2:172.21.15.145:3300/0 conn(0x7f40a406ee00 0x7f40a40b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 --2- 172.21.15.119:0/3980496275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a4159800 0x7f40a4159be0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 -- 172.21.15.119:0/3980496275 >> 172.21.15.119:0/3980496275 conn(0x7f40a40b15e0 msgr2=0x7f40a40b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:07.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 -- 172.21.15.119:0/3980496275 shutdown_connections 2024-09-13T16:26:07.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.105+0000 7f40a9472700 1 -- 172.21.15.119:0/3980496275 wait complete. 2024-09-13T16:26:07.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 Processor -- start 2024-09-13T16:26:07.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 -- start start 2024-09-13T16:26:07.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40a406ee00 0x7f40a40b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a40b3aa0 0x7f40a41588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4159800 0x7f40a414fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40af910 con 0x7f40a4159800 2024-09-13T16:26:07.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40afbf0 con 0x7f40a406ee00 2024-09-13T16:26:07.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40a40afed0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a8c71700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4159800 0x7f40a414fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a37fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a40b3aa0 0x7f40a41588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a8c71700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4159800 0x7f40a414fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44484/0 (socket says 172.21.15.119:44484) 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a8c71700 1 -- 172.21.15.119:0/596612686 learned_addr learned my addr 172.21.15.119:0/596612686 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a3fff700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40a406ee00 0x7f40a40b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a37fe700 1 -- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40a406ee00 msgr2=0x7f40a40b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a37fe700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40a406ee00 0x7f40a40b0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a37fe700 1 -- 172.21.15.119:0/596612686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4159800 msgr2=0x7f40a414fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a37fe700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4159800 0x7f40a414fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a37fe700 1 -- 172.21.15.119:0/596612686 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40980094b0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a8c71700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4159800 0x7f40a414fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:07.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a37fe700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a40b3aa0 0x7f40a41588c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f409800e980 tx=0x7f409800ec90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a0ff9700 1 -- 172.21.15.119:0/596612686 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409800c690 con 0x7f40a40b3aa0 2024-09-13T16:26:07.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a0ff9700 1 -- 172.21.15.119:0/596612686 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409800c7f0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40940094b0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a0ff9700 1 -- 172.21.15.119:0/596612686 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409800cb80 con 0x7f40a40b3aa0 2024-09-13T16:26:07.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.109+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40a40b0490 con 0x7f40a40b3aa0 2024-09-13T16:26:07.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.113+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40a40b3560 con 0x7f40a40b3aa0 2024-09-13T16:26:07.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.113+0000 7f40a0ff9700 1 -- 172.21.15.119:0/596612686 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f409800cce0 con 0x7f40a40b3aa0 2024-09-13T16:26:07.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.113+0000 7f40a0ff9700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4090066730 0x7f4090068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.113+0000 7f40a0ff9700 1 -- 172.21.15.119:0/596612686 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f4098084c50 con 0x7f40a40b3aa0 2024-09-13T16:26:07.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.113+0000 7f40a3fff700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4090066730 0x7f4090068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.117+0000 7f40a3fff700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4090066730 0x7f4090068bf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40940090b0 tx=0x7f4094008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.121+0000 7f40a0ff9700 1 -- 172.21.15.119:0/596612686 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4098051c90 con 0x7f40a40b3aa0 2024-09-13T16:26:07.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40a40b0010 con 0x7f40a40b3aa0 2024-09-13T16:26:07.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a0ff9700 1 -- 172.21.15.119:0/596612686 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f40a40b0010 con 0x7f40a40b3aa0 2024-09-13T16:26:07.296 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:07.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4090066730 msgr2=0x7f4090068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4090066730 0x7f4090068bf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40940090b0 tx=0x7f4094008030 comp rx=0 tx=0).stop 2024-09-13T16:26:07.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a40b3aa0 msgr2=0x7f40a41588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a40b3aa0 0x7f40a41588c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f409800e980 tx=0x7f409800ec90 comp rx=0 tx=0).stop 2024-09-13T16:26:07.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 shutdown_connections 2024-09-13T16:26:07.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4090066730 0x7f4090068bf0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40a4159800 0x7f40a414fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40a40b3aa0 0x7f40a41588c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 --2- 172.21.15.119:0/596612686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40a406ee00 0x7f40a40b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 >> 172.21.15.119:0/596612686 conn(0x7f40a40b15e0 msgr2=0x7f40a40cee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:07.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 shutdown_connections 2024-09-13T16:26:07.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.293+0000 7f40a9472700 1 -- 172.21.15.119:0/596612686 wait complete. 2024-09-13T16:26:07.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:07.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.421+0000 7ff27f59e700 1 Processor -- start 2024-09-13T16:26:07.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.421+0000 7ff27f59e700 1 -- start start 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2800ce8c0 0x7ff2800d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff2800d1610 0x7ff2800b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2800d0df0 con 0x7ff28015cc00 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2800d1a80 con 0x7ff2800d1610 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800cdb00 con 0x7ff2800ce8c0 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff2800d1610 0x7ff2800b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff2800d1610 0x7ff2800b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44500/0 (socket says 172.21.15.119:44500) 2024-09-13T16:26:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27e59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff28015cc00 0x7ff2800d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50246/0 (socket says 172.21.15.119:50246) 2024-09-13T16:26:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 -- 172.21.15.119:0/1470830559 learned_addr learned my addr 172.21.15.119:0/1470830559 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27dd9b700 1 --2- 172.21.15.119:0/1470830559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2800ce8c0 0x7ff2800d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 -- 172.21.15.119:0/1470830559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2800ce8c0 msgr2=0x7ff2800d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 --2- 172.21.15.119:0/1470830559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2800ce8c0 0x7ff2800d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 -- 172.21.15.119:0/1470830559 >> v1:172.21.15.119:6789/0 conn(0x7ff28015cc00 legacy=0x7ff2800d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 -- 172.21.15.119:0/1470830559 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2800cdde0 con 0x7ff2800d1610 2024-09-13T16:26:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27ed9d700 1 --2- 172.21.15.119:0/1470830559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800b5960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff278009e50 tx=0x7ff27800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce78265c2a5f8ede server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27d59a700 1 -- 172.21.15.119:0/1470830559 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27800e070 con 0x7ff2800d1610 2024-09-13T16:26:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27d59a700 1 -- 172.21.15.119:0/1470830559 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff27800b9c0 con 0x7ff2800d1610 2024-09-13T16:26:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27d59a700 1 -- 172.21.15.119:0/1470830559 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27800bd70 con 0x7ff2800d1610 2024-09-13T16:26:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- 172.21.15.119:0/1470830559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 msgr2=0x7ff2800b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 --2- 172.21.15.119:0/1470830559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800b5960 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff278009e50 tx=0x7ff27800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- 172.21.15.119:0/1470830559 shutdown_connections 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 --2- 172.21.15.119:0/1470830559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800b5960 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 --2- 172.21.15.119:0/1470830559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2800ce8c0 0x7ff2800d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- 172.21.15.119:0/1470830559 >> 172.21.15.119:0/1470830559 conn(0x7ff28008c080 msgr2=0x7ff2800d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- 172.21.15.119:0/1470830559 shutdown_connections 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.425+0000 7ff27f59e700 1 -- 172.21.15.119:0/1470830559 wait complete. 2024-09-13T16:26:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 Processor -- start 2024-09-13T16:26:07.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 -- start start 2024-09-13T16:26:07.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2800ce8c0 0x7ff280154c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800cbd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff28015cc00 0x7ff2801542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800c7b80 con 0x7ff2800d1610 2024-09-13T16:26:07.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800ca390 con 0x7ff2800ce8c0 2024-09-13T16:26:07.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2800c9d80 con 0x7ff28015cc00 2024-09-13T16:26:07.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800cbd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800cbd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44502/0 (socket says 172.21.15.119:44502) 2024-09-13T16:26:07.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 -- 172.21.15.119:0/3056348430 learned_addr learned my addr 172.21.15.119:0/3056348430 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:07.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27ed9d700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff28015cc00 0x7ff2801542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27e59c700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2800ce8c0 0x7ff280154c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 -- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff28015cc00 msgr2=0x7ff2801542e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff28015cc00 0x7ff2801542e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 -- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2800ce8c0 msgr2=0x7ff280154c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2800ce8c0 0x7ff280154c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 -- 172.21.15.119:0/3056348430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff278009b30 con 0x7ff2800d1610 2024-09-13T16:26:07.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27dd9b700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800cbd60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff27000e730 tx=0x7ff27000ea40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff26effd700 1 -- 172.21.15.119:0/3056348430 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff27000c070 con 0x7ff2800d1610 2024-09-13T16:26:07.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff26effd700 1 -- 172.21.15.119:0/3056348430 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff270013610 con 0x7ff2800d1610 2024-09-13T16:26:07.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2800c78a0 con 0x7ff2800d1610 2024-09-13T16:26:07.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff26effd700 1 -- 172.21.15.119:0/3056348430 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2700139c0 con 0x7ff2800d1610 2024-09-13T16:26:07.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.429+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2800c7020 con 0x7ff2800d1610 2024-09-13T16:26:07.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.433+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2800c9bc0 con 0x7ff2800d1610 2024-09-13T16:26:07.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.433+0000 7ff26effd700 1 -- 172.21.15.119:0/3056348430 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff270013b20 con 0x7ff2800d1610 2024-09-13T16:26:07.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.433+0000 7ff26effd700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff260066730 0x7ff260068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.433+0000 7ff26effd700 1 -- 172.21.15.119:0/3056348430 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff270084300 con 0x7ff2800d1610 2024-09-13T16:26:07.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.437+0000 7ff27e59c700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff260066730 0x7ff260068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.437+0000 7ff27e59c700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff260066730 0x7ff260068bf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff280160330 tx=0x7ff274007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.441+0000 7ff26effd700 1 -- 172.21.15.119:0/3056348430 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff270051210 con 0x7ff2800d1610 2024-09-13T16:26:07.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.605+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ff2800d3670 con 0x7ff2800d1610 2024-09-13T16:26:07.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.605+0000 7ff26effd700 1 -- 172.21.15.119:0/3056348430 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7ff270054830 con 0x7ff2800d1610 2024-09-13T16:26:07.610 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff260066730 msgr2=0x7ff260068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff260066730 0x7ff260068bf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ff280160330 tx=0x7ff274007f30 comp rx=0 tx=0).stop 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 msgr2=0x7ff2800cbd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800cbd60 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ff27000e730 tx=0x7ff27000ea40 comp rx=0 tx=0).stop 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 shutdown_connections 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff260066730 0x7ff260068bf0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff28015cc00 0x7ff2801542e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2800d1610 0x7ff2800cbd60 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 --2- 172.21.15.119:0/3056348430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2800ce8c0 0x7ff280154c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 >> 172.21.15.119:0/3056348430 conn(0x7ff28008c080 msgr2=0x7ff2800b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:07.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 shutdown_connections 2024-09-13T16:26:07.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.609+0000 7ff27f59e700 1 -- 172.21.15.119:0/3056348430 wait complete. 2024-09-13T16:26:07.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:07.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37ffff700 1 Processor -- start 2024-09-13T16:26:07.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37ffff700 1 -- start start 2024-09-13T16:26:07.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe38015cc40 0x7fe3800ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37ffff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe3800cd860 0x7fe38008bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37ffff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38015d020 con 0x7fe38008c4e0 2024-09-13T16:26:07.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37ffff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe38008c990 con 0x7fe3800cd860 2024-09-13T16:26:07.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3800b44c0 con 0x7fe38015cc40 2024-09-13T16:26:07.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37effd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe38015cc40 0x7fe3800ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37e7fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe3800cd860 0x7fe38008bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37e7fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe3800cd860 0x7fe38008bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:33326/0 (socket says 172.21.15.119:33326) 2024-09-13T16:26:07.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37e7fc700 1 -- 172.21.15.119:0/835625785 learned_addr learned my addr 172.21.15.119:0/835625785 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.733+0000 7fe37effd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe38015cc40 0x7fe3800ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:51808/0 (socket says 172.21.15.119:51808) 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37e7fc700 1 -- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe38015cc40 msgr2=0x7fe3800ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37e7fc700 1 --2- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe38015cc40 0x7fe3800ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37e7fc700 1 -- 172.21.15.119:0/835625785 >> v1:172.21.15.145:6789/0 conn(0x7fe38008c4e0 legacy=0x7fe380160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37e7fc700 1 -- 172.21.15.119:0/835625785 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3800d0720 con 0x7fe3800cd860 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37effd700 1 --2- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe38015cc40 0x7fe3800ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37e7fc700 1 --2- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3800cd860 0x7fe38008bfa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe370009940 tx=0x7fe370009c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12998b3872a07c90 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37dffb700 1 -- 172.21.15.119:0/835625785 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370011830 con 0x7fe3800cd860 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37dffb700 1 -- 172.21.15.119:0/835625785 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe370011990 con 0x7fe3800cd860 2024-09-13T16:26:07.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37dffb700 1 -- 172.21.15.119:0/835625785 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370011d20 con 0x7fe3800cd860 2024-09-13T16:26:07.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 -- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3800cd860 msgr2=0x7fe38008bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 --2- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3800cd860 0x7fe38008bfa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fe370009940 tx=0x7fe370009c50 comp rx=0 tx=0).stop 2024-09-13T16:26:07.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 -- 172.21.15.119:0/835625785 shutdown_connections 2024-09-13T16:26:07.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 --2- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3800cd860 0x7fe38008bfa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 --2- 172.21.15.119:0/835625785 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe38015cc40 0x7fe3800ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 -- 172.21.15.119:0/835625785 >> 172.21.15.119:0/835625785 conn(0x7fe3800b4640 msgr2=0x7fe3800b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:07.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 -- 172.21.15.119:0/835625785 shutdown_connections 2024-09-13T16:26:07.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.737+0000 7fe37ffff700 1 -- 172.21.15.119:0/835625785 wait complete. 2024-09-13T16:26:07.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 Processor -- start 2024-09-13T16:26:07.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 -- start start 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe38008c4e0 0x7fe3800c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3800cd860 0x7fe3800cba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe38015cc40 0x7fe3800c9410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3801e6410 con 0x7fe38008c4e0 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3800cc190 con 0x7fe38015cc40 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3801e6620 con 0x7fe3800cd860 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe38008c4e0 0x7fe3800c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3800cd860 0x7fe3800cba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3800cd860 0x7fe3800cba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:51812/0 (socket says 172.21.15.119:51812) 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 -- 172.21.15.119:0/3118270889 learned_addr learned my addr 172.21.15.119:0/3118270889 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:07.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37f7fe700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe38015cc40 0x7fe3800c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 -- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe38015cc40 msgr2=0x7fe3800c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe38015cc40 0x7fe3800c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 -- 172.21.15.119:0/3118270889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe38008c4e0 msgr2=0x7fe3800c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe38008c4e0 0x7fe3800c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 -- 172.21.15.119:0/3118270889 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3740094b0 con 0x7fe3800cd860 2024-09-13T16:26:07.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37effd700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe38008c4e0 0x7fe3800c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:07.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37e7fc700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3800cd860 0x7fe3800cba80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe370009fd0 tx=0x7fe370004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe3677fe700 1 -- 172.21.15.119:0/3118270889 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370004510 con 0x7fe3800cd860 2024-09-13T16:26:07.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe3677fe700 1 -- 172.21.15.119:0/3118270889 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3700036c0 con 0x7fe3800cd860 2024-09-13T16:26:07.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe370009570 con 0x7fe3800cd860 2024-09-13T16:26:07.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe3677fe700 1 -- 172.21.15.119:0/3118270889 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe370003a30 con 0x7fe3800cd860 2024-09-13T16:26:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.741+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3801e6be0 con 0x7fe3800cd860 2024-09-13T16:26:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.745+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3800b42e0 con 0x7fe3800cd860 2024-09-13T16:26:07.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.745+0000 7fe3677fe700 1 -- 172.21.15.119:0/3118270889 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe370003b90 con 0x7fe3800cd860 2024-09-13T16:26:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.745+0000 7fe3677fe700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe36c066760 0x7fe36c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.745+0000 7fe3677fe700 1 -- 172.21.15.119:0/3118270889 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe370085a70 con 0x7fe3800cd860 2024-09-13T16:26:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.745+0000 7fe37effd700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe36c066760 0x7fe36c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:07.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.745+0000 7fe37effd700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe36c066760 0x7fe36c068c20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe3740114b0 tx=0x7fe374009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.757+0000 7fe3677fe700 1 -- 172.21.15.119:0/3118270889 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe370052ab0 con 0x7fe3800cd860 2024-09-13T16:26:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.937+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe3800d1540 con 0x7fe3800cd860 2024-09-13T16:26:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe3677fe700 1 -- 172.21.15.119:0/3118270889 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe3700560d0 con 0x7fe3800cd860 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe36c066760 msgr2=0x7fe36c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe36c066760 0x7fe36c068c20 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe3740114b0 tx=0x7fe374009240 comp rx=0 tx=0).stop 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3800cd860 msgr2=0x7fe3800cba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3800cd860 0x7fe3800cba80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe370009fd0 tx=0x7fe370004770 comp rx=0 tx=0).stop 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 shutdown_connections 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe36c066760 0x7fe36c068c20 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe38015cc40 0x7fe3800c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3800cd860 0x7fe3800cba80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 --2- 172.21.15.119:0/3118270889 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe38008c4e0 0x7fe3800c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 >> 172.21.15.119:0/3118270889 conn(0x7fe3800b4640 msgr2=0x7fe3800ce450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 shutdown_connections 2024-09-13T16:26:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:07.941+0000 7fe37ffff700 1 -- 172.21.15.119:0/3118270889 wait complete. 2024-09-13T16:26:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:07.959 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:07.960 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:07.966 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:08.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.077+0000 7f61108e4700 1 Processor -- start 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.077+0000 7f61108e4700 1 -- start start 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f610c1594f0 0x7f610c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f610c0b3800 0x7f610c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610c0b2280 con 0x7f610c06da70 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610c06dee0 con 0x7f610c0b3800 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610c0cc890 con 0x7f610c1594f0 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610b59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f610c0b3800 0x7f610c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610a59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f610c06da70 0x7f610c0b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50262/0 (socket says 172.21.15.119:50262) 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610a59c700 1 -- 172.21.15.119:0/464970121 learned_addr learned my addr 172.21.15.119:0/464970121 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610ad9d700 1 --2- 172.21.15.119:0/464970121 >> v2:172.21.15.145:3300/0 conn(0x7f610c1594f0 0x7f610c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610b59e700 1 -- 172.21.15.119:0/464970121 >> v2:172.21.15.145:3300/0 conn(0x7f610c1594f0 msgr2=0x7f610c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610b59e700 1 --2- 172.21.15.119:0/464970121 >> v2:172.21.15.145:3300/0 conn(0x7f610c1594f0 0x7f610c06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610b59e700 1 -- 172.21.15.119:0/464970121 >> v1:172.21.15.119:6789/0 conn(0x7f610c06da70 legacy=0x7f610c0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610b59e700 1 -- 172.21.15.119:0/464970121 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0b1970 con 0x7f610c0b3800 2024-09-13T16:26:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610ad9d700 1 --2- 172.21.15.119:0/464970121 >> v2:172.21.15.145:3300/0 conn(0x7f610c1594f0 0x7f610c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f610b59e700 1 --2- 172.21.15.119:0/464970121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c0b3800 0x7f610c06fc00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f610c06e0f0 tx=0x7f6104009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c6f0ae38ae5a20 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f6109d9b700 1 -- 172.21.15.119:0/464970121 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104011830 con 0x7f610c0b3800 2024-09-13T16:26:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f6109d9b700 1 -- 172.21.15.119:0/464970121 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6104011990 con 0x7f610c0b3800 2024-09-13T16:26:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f6109d9b700 1 -- 172.21.15.119:0/464970121 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104011d40 con 0x7f610c0b3800 2024-09-13T16:26:08.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- 172.21.15.119:0/464970121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c0b3800 msgr2=0x7f610c06fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 --2- 172.21.15.119:0/464970121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c0b3800 0x7f610c06fc00 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f610c06e0f0 tx=0x7f6104009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:08.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- 172.21.15.119:0/464970121 shutdown_connections 2024-09-13T16:26:08.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 --2- 172.21.15.119:0/464970121 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c0b3800 0x7f610c06fc00 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 --2- 172.21.15.119:0/464970121 >> v2:172.21.15.145:3300/0 conn(0x7f610c1594f0 0x7f610c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- 172.21.15.119:0/464970121 >> 172.21.15.119:0/464970121 conn(0x7f610c06a350 msgr2=0x7f610c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:08.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- 172.21.15.119:0/464970121 shutdown_connections 2024-09-13T16:26:08.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.081+0000 7f61108e4700 1 -- 172.21.15.119:0/464970121 wait complete. 2024-09-13T16:26:08.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 Processor -- start 2024-09-13T16:26:08.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 -- start start 2024-09-13T16:26:08.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f610c06da70 0x7f610c155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f610c0b3800 0x7f610c13df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c1594f0 0x7f610c1f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c1f15e0 con 0x7f610c1594f0 2024-09-13T16:26:08.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c153270 con 0x7f610c0b3800 2024-09-13T16:26:08.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610c1f1020 con 0x7f610c06da70 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c1594f0 0x7f610c1f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c1594f0 0x7f610c1f05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44526/0 (socket says 172.21.15.119:44526) 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 -- 172.21.15.119:0/3292554934 learned_addr learned my addr 172.21.15.119:0/3292554934 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610a59c700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f610c0b3800 0x7f610c13df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610ad9d700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f610c06da70 0x7f610c155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 -- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f610c06da70 msgr2=0x7f610c155920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f610c06da70 0x7f610c155920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 -- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f610c0b3800 msgr2=0x7f610c13df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f610c0b3800 0x7f610c13df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 -- 172.21.15.119:0/3292554934 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61040094b0 con 0x7f610c1594f0 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610ad9d700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f610c06da70 0x7f610c155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610a59c700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f610c0b3800 0x7f610c13df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:08.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f610b59e700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c1594f0 0x7f610c1f05f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f610c06b1a0 tx=0x7f6104004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61037fe700 1 -- 172.21.15.119:0/3292554934 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104004480 con 0x7f610c1594f0 2024-09-13T16:26:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61037fe700 1 -- 172.21.15.119:0/3292554934 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610401f070 con 0x7f610c1594f0 2024-09-13T16:26:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610c1f1300 con 0x7f610c1594f0 2024-09-13T16:26:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61037fe700 1 -- 172.21.15.119:0/3292554934 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104003900 con 0x7f610c1594f0 2024-09-13T16:26:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.085+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610c155e60 con 0x7f610c1594f0 2024-09-13T16:26:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.089+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610c1533b0 con 0x7f610c1594f0 2024-09-13T16:26:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.089+0000 7f61037fe700 1 -- 172.21.15.119:0/3292554934 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6104021030 con 0x7f610c1594f0 2024-09-13T16:26:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.089+0000 7f61037fe700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ec066730 0x7f60ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.089+0000 7f61037fe700 1 -- 172.21.15.119:0/3292554934 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f61040859f0 con 0x7f610c1594f0 2024-09-13T16:26:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.089+0000 7f610ad9d700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ec066730 0x7f60ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.089+0000 7f610ad9d700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ec066730 0x7f60ec068bf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009a10 tx=0x7f60f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.097+0000 7f61037fe700 1 -- 172.21.15.119:0/3292554934 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f610c1533b0 con 0x7f610c1594f0 2024-09-13T16:26:08.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.273+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f610c1533b0 con 0x7f610c1594f0 2024-09-13T16:26:08.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.273+0000 7f61037fe700 1 -- 172.21.15.119:0/3292554934 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f610c1533b0 con 0x7f610c1594f0 2024-09-13T16:26:08.279 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:08.279 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ec066730 msgr2=0x7f60ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ec066730 0x7f60ec068bf0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f60f4009a10 tx=0x7f60f4008040 comp rx=0 tx=0).stop 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c1594f0 msgr2=0x7f610c1f05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c1594f0 0x7f610c1f05f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f610c06b1a0 tx=0x7f6104004770 comp rx=0 tx=0).stop 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 shutdown_connections 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f60ec066730 0x7f60ec068bf0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f610c1594f0 0x7f610c1f05f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f610c0b3800 0x7f610c13df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 --2- 172.21.15.119:0/3292554934 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f610c06da70 0x7f610c155920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 >> 172.21.15.119:0/3292554934 conn(0x7f610c06a350 msgr2=0x7f610c15bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:08.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 shutdown_connections 2024-09-13T16:26:08.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.277+0000 7f61108e4700 1 -- 172.21.15.119:0/3292554934 wait complete. 2024-09-13T16:26:08.294 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:08.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:08.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.409+0000 7f272671e700 1 Processor -- start 2024-09-13T16:26:08.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.409+0000 7f272671e700 1 -- start start 2024-09-13T16:26:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.409+0000 7f272671e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f27200d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27200d0c20 0x7f27200d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27200cdb00 con 0x7f27200ce8c0 2024-09-13T16:26:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27200cdde0 con 0x7f27200d0c20 2024-09-13T16:26:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27200ce0c0 con 0x7f272015cc00 2024-09-13T16:26:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27200d0c20 0x7f27200d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27200d0c20 0x7f27200d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44534/0 (socket says 172.21.15.119:44534) 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 -- 172.21.15.119:0/77044035 learned_addr learned my addr 172.21.15.119:0/77044035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272571c700 1 --2- 172.21.15.119:0/77044035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f27200d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 -- 172.21.15.119:0/77044035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 msgr2=0x7f27200d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 --2- 172.21.15.119:0/77044035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f27200d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 -- 172.21.15.119:0/77044035 >> v1:172.21.15.145:6789/0 conn(0x7f27200ce8c0 legacy=0x7f2720160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 -- 172.21.15.119:0/77044035 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27200ce3a0 con 0x7f27200d0c20 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272571c700 1 --2- 172.21.15.119:0/77044035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f27200d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2724f1b700 1 --2- 172.21.15.119:0/77044035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f27200d10d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f271c009910 tx=0x7f271c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e72b355deef3685 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2717fff700 1 -- 172.21.15.119:0/77044035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c00ba60 con 0x7f27200d0c20 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2717fff700 1 -- 172.21.15.119:0/77044035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c00bbc0 con 0x7f27200d0c20 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f2717fff700 1 -- 172.21.15.119:0/77044035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c009e60 con 0x7f27200d0c20 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- 172.21.15.119:0/77044035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 msgr2=0x7f27200d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 --2- 172.21.15.119:0/77044035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f27200d10d0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f271c009910 tx=0x7f271c009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- 172.21.15.119:0/77044035 shutdown_connections 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 --2- 172.21.15.119:0/77044035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f27200d10d0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 --2- 172.21.15.119:0/77044035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f27200d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- 172.21.15.119:0/77044035 >> 172.21.15.119:0/77044035 conn(0x7f272008c080 msgr2=0x7f27200d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- 172.21.15.119:0/77044035 shutdown_connections 2024-09-13T16:26:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.413+0000 7f272671e700 1 -- 172.21.15.119:0/77044035 wait complete. 2024-09-13T16:26:08.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 Processor -- start 2024-09-13T16:26:08.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 -- start start 2024-09-13T16:26:08.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27200ce8c0 0x7f27200c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f2720155e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f2720156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27200c61c0 con 0x7f27200d0c20 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27200c94b0 con 0x7f27200ce8c0 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27200c9d80 con 0x7f272015cc00 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f2720155e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f2720155e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44544/0 (socket says 172.21.15.119:44544) 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 -- 172.21.15.119:0/287279841 learned_addr learned my addr 172.21.15.119:0/287279841 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272571c700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27200ce8c0 0x7f27200c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2725f1d700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f2720156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 -- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 msgr2=0x7f2720156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f2720156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 -- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27200ce8c0 msgr2=0x7f27200c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27200ce8c0 0x7f27200c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 -- 172.21.15.119:0/287279841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c009570 con 0x7f27200d0c20 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272571c700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27200ce8c0 0x7f27200c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2724f1b700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f2720155e20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f271c013000 tx=0x7f271c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2715ffb700 1 -- 172.21.15.119:0/287279841 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c004510 con 0x7f27200d0c20 2024-09-13T16:26:08.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2715ffb700 1 -- 172.21.15.119:0/287279841 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f271c01f070 con 0x7f27200d0c20 2024-09-13T16:26:08.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27200ca060 con 0x7f27200d0c20 2024-09-13T16:26:08.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f2715ffb700 1 -- 172.21.15.119:0/287279841 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271c003900 con 0x7f27200d0c20 2024-09-13T16:26:08.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.417+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27200ca620 con 0x7f27200d0c20 2024-09-13T16:26:08.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.421+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27200d06e0 con 0x7f27200d0c20 2024-09-13T16:26:08.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.421+0000 7f2715ffb700 1 -- 172.21.15.119:0/287279841 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f271c003a60 con 0x7f27200d0c20 2024-09-13T16:26:08.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.421+0000 7f2715ffb700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2704066760 0x7f2704068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.421+0000 7f2715ffb700 1 -- 172.21.15.119:0/287279841 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f271c021070 con 0x7f27200d0c20 2024-09-13T16:26:08.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.421+0000 7f272571c700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2704066760 0x7f2704068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.425+0000 7f272571c700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2704066760 0x7f2704068c20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2710009a10 tx=0x7f2710008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.429+0000 7f2715ffb700 1 -- 172.21.15.119:0/287279841 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f271c052ce0 con 0x7f27200d0c20 2024-09-13T16:26:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.593+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f272006c8e0 con 0x7f27200d0c20 2024-09-13T16:26:08.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.593+0000 7f2715ffb700 1 -- 172.21.15.119:0/287279841 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f271c0150a0 con 0x7f27200d0c20 2024-09-13T16:26:08.599 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2704066760 msgr2=0x7f2704068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2704066760 0x7f2704068c20 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2710009a10 tx=0x7f2710008040 comp rx=0 tx=0).stop 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 msgr2=0x7f2720155e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f2720155e20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f271c013000 tx=0x7f271c004770 comp rx=0 tx=0).stop 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 shutdown_connections 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2704066760 0x7f2704068c20 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f272015cc00 0x7f2720156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27200d0c20 0x7f2720155e20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 --2- 172.21.15.119:0/287279841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27200ce8c0 0x7f27200c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 >> 172.21.15.119:0/287279841 conn(0x7f272008c080 msgr2=0x7f272006c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:08.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 shutdown_connections 2024-09-13T16:26:08.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.597+0000 7f272671e700 1 -- 172.21.15.119:0/287279841 wait complete. 2024-09-13T16:26:08.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:08.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.737+0000 7f548e9fc700 1 Processor -- start 2024-09-13T16:26:08.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.737+0000 7f548e9fc700 1 -- start start 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f548815cc00 0x7f54880d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f54880d0c20 0x7f54880d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54880cdb00 con 0x7f54880ce8c0 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54880cdde0 con 0x7f54880d0c20 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54880ce0c0 con 0x7f548815cc00 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f54880d0c20 0x7f54880d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f54880d0c20 0x7f54880d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44558/0 (socket says 172.21.15.119:44558) 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 -- 172.21.15.119:0/2041458903 learned_addr learned my addr 172.21.15.119:0/2041458903 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e1fb700 1 --1- 172.21.15.119:0/2041458903 >> v1:172.21.15.119:6789/0 conn(0x7f54880ce8c0 0x7f5488160880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50264/0 (socket says 172.21.15.119:50264) 2024-09-13T16:26:08.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d9fa700 1 --2- 172.21.15.119:0/2041458903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f548815cc00 0x7f54880d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 -- 172.21.15.119:0/2041458903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f548815cc00 msgr2=0x7f54880d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 --2- 172.21.15.119:0/2041458903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f548815cc00 0x7f54880d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 -- 172.21.15.119:0/2041458903 >> v1:172.21.15.119:6789/0 conn(0x7f54880ce8c0 legacy=0x7f5488160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 -- 172.21.15.119:0/2041458903 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54880ce3a0 con 0x7f54880d0c20 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548d1f9700 1 --2- 172.21.15.119:0/2041458903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54880d0c20 0x7f54880d10d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f547c009910 tx=0x7f547c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22fdef388ed1b706 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548c9f8700 1 -- 172.21.15.119:0/2041458903 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c011830 con 0x7f54880d0c20 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548c9f8700 1 -- 172.21.15.119:0/2041458903 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c011990 con 0x7f54880d0c20 2024-09-13T16:26:08.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548c9f8700 1 -- 172.21.15.119:0/2041458903 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c011d40 con 0x7f54880d0c20 2024-09-13T16:26:08.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 -- 172.21.15.119:0/2041458903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54880d0c20 msgr2=0x7f54880d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 --2- 172.21.15.119:0/2041458903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54880d0c20 0x7f54880d10d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f547c009910 tx=0x7f547c009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:08.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 -- 172.21.15.119:0/2041458903 shutdown_connections 2024-09-13T16:26:08.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 --2- 172.21.15.119:0/2041458903 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f54880d0c20 0x7f54880d10d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 --2- 172.21.15.119:0/2041458903 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f548815cc00 0x7f54880d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 -- 172.21.15.119:0/2041458903 >> 172.21.15.119:0/2041458903 conn(0x7f548808c080 msgr2=0x7f54880d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:08.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.741+0000 7f548e9fc700 1 -- 172.21.15.119:0/2041458903 shutdown_connections 2024-09-13T16:26:08.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.745+0000 7f548e9fc700 1 -- 172.21.15.119:0/2041458903 wait complete. 2024-09-13T16:26:08.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.745+0000 7f548e9fc700 1 Processor -- start 2024-09-13T16:26:08.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.745+0000 7f548e9fc700 1 -- start start 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54880ce8c0 0x7f54881541e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54880d0c20 0x7f54880c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f548815cc00 0x7f54881e4380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54881e5640 con 0x7f548815cc00 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54880c92c0 con 0x7f54880d0c20 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54881e2fe0 con 0x7f54880ce8c0 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f548815cc00 0x7f54881e4380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f548815cc00 0x7f54881e4380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44564/0 (socket says 172.21.15.119:44564) 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 -- 172.21.15.119:0/3469479202 learned_addr learned my addr 172.21.15.119:0/3469479202 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:08.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548d9fa700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54880ce8c0 0x7f54881541e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548d1f9700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54880d0c20 0x7f54880c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 -- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54880ce8c0 msgr2=0x7f54881541e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54880ce8c0 0x7f54881541e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 -- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54880d0c20 msgr2=0x7f54880c8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54880d0c20 0x7f54880c8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 -- 172.21.15.119:0/3469479202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c009570 con 0x7f548815cc00 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e1fb700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f548815cc00 0x7f54881e4380 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f548000cbc0 tx=0x7f548000ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f547a7fc700 1 -- 172.21.15.119:0/3469479202 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548000a9e0 con 0x7f548815cc00 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f547a7fc700 1 -- 172.21.15.119:0/3469479202 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548000ab40 con 0x7f548815cc00 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54881e32c0 con 0x7f548815cc00 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f547a7fc700 1 -- 172.21.15.119:0/3469479202 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5480007070 con 0x7f548815cc00 2024-09-13T16:26:08.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54881e2a20 con 0x7f548815cc00 2024-09-13T16:26:08.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.749+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54881e53a0 con 0x7f548815cc00 2024-09-13T16:26:08.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.753+0000 7f547a7fc700 1 -- 172.21.15.119:0/3469479202 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f54800071d0 con 0x7f548815cc00 2024-09-13T16:26:08.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.753+0000 7f547a7fc700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f546c066730 0x7f546c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:08.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.753+0000 7f547a7fc700 1 -- 172.21.15.119:0/3469479202 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5480083fe0 con 0x7f548815cc00 2024-09-13T16:26:08.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.753+0000 7f548d9fa700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f546c066730 0x7f546c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:08.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.753+0000 7f548d9fa700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f546c066730 0x7f546c068bf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5484009bf0 tx=0x7f5484009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:08.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.757+0000 7f547a7fc700 1 -- 172.21.15.119:0/3469479202 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f54881e53a0 con 0x7f548815cc00 2024-09-13T16:26:08.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.925+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f54881e5780 con 0x7f548815cc00 2024-09-13T16:26:08.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.925+0000 7f547a7fc700 1 -- 172.21.15.119:0/3469479202 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f54881e5780 con 0x7f548815cc00 2024-09-13T16:26:08.930 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:08.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f546c066730 msgr2=0x7f546c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f546c066730 0x7f546c068bf0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f5484009bf0 tx=0x7f5484009240 comp rx=0 tx=0).stop 2024-09-13T16:26:08.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f548815cc00 msgr2=0x7f54881e4380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:08.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f548815cc00 0x7f54881e4380 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f548000cbc0 tx=0x7f548000ced0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 shutdown_connections 2024-09-13T16:26:08.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f546c066730 0x7f546c068bf0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f548815cc00 0x7f54881e4380 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f54880d0c20 0x7f54880c8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 --2- 172.21.15.119:0/3469479202 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f54880ce8c0 0x7f54881541e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:08.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 >> 172.21.15.119:0/3469479202 conn(0x7f548808c080 msgr2=0x7f54880b5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:08.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 shutdown_connections 2024-09-13T16:26:08.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:08.929+0000 7f548e9fc700 1 -- 172.21.15.119:0/3469479202 wait complete. 2024-09-13T16:26:08.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:09.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.053+0000 7fed73fff700 1 Processor -- start 2024-09-13T16:26:09.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.053+0000 7fed73fff700 1 -- start start 2024-09-13T16:26:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.053+0000 7fed73fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed74144180 0x7fed740cc2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.053+0000 7fed73fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fed740ce240 0x7fed7406bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.053+0000 7fed73fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed740b26f0 con 0x7fed740b22b0 2024-09-13T16:26:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.053+0000 7fed73fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed740b1fe0 con 0x7fed740ce240 2024-09-13T16:26:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.053+0000 7fed73fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed740b29b0 con 0x7fed74144180 2024-09-13T16:26:09.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed737fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fed740ce240 0x7fed7406bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed727fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fed740b22b0 0x7fed740cdb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50270/0 (socket says 172.21.15.119:50270) 2024-09-13T16:26:09.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed727fc700 1 -- 172.21.15.119:0/4190902465 learned_addr learned my addr 172.21.15.119:0/4190902465 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:09.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed72ffd700 1 --2- 172.21.15.119:0/4190902465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed74144180 0x7fed740cc2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed737fe700 1 -- 172.21.15.119:0/4190902465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed74144180 msgr2=0x7fed740cc2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed737fe700 1 --2- 172.21.15.119:0/4190902465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed74144180 0x7fed740cc2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed737fe700 1 -- 172.21.15.119:0/4190902465 >> v1:172.21.15.119:6789/0 conn(0x7fed740b22b0 legacy=0x7fed740cdb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed737fe700 1 -- 172.21.15.119:0/4190902465 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed740b2c90 con 0x7fed740ce240 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed71ffb700 1 -- 172.21.15.119:0/4190902465 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368891328 0 0) 0x7fed740b26f0 con 0x7fed740b22b0 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed737fe700 1 --2- 172.21.15.119:0/4190902465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740ce240 0x7fed7406bb60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009850 tx=0x7fed6c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dee9b65a07e2bc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed71ffb700 1 -- 172.21.15.119:0/4190902465 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c011830 con 0x7fed740ce240 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed71ffb700 1 -- 172.21.15.119:0/4190902465 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c011990 con 0x7fed740ce240 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed71ffb700 1 -- 172.21.15.119:0/4190902465 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c011d40 con 0x7fed740ce240 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 -- 172.21.15.119:0/4190902465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740ce240 msgr2=0x7fed7406bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 --2- 172.21.15.119:0/4190902465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740ce240 0x7fed7406bb60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fed6c009850 tx=0x7fed6c009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 -- 172.21.15.119:0/4190902465 shutdown_connections 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 --2- 172.21.15.119:0/4190902465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740ce240 0x7fed7406bb60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 --2- 172.21.15.119:0/4190902465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed74144180 0x7fed740cc2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 -- 172.21.15.119:0/4190902465 >> 172.21.15.119:0/4190902465 conn(0x7fed740b0980 msgr2=0x7fed740b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 -- 172.21.15.119:0/4190902465 shutdown_connections 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.057+0000 7fed73fff700 1 -- 172.21.15.119:0/4190902465 wait complete. 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 Processor -- start 2024-09-13T16:26:09.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 -- start start 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740b22b0 0x7fed741532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed740ce240 0x7fed7413fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed74144180 0x7fed740ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed74139980 con 0x7fed740b22b0 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed74140e60 con 0x7fed74144180 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed741401e0 con 0x7fed740ce240 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740b22b0 0x7fed741532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740b22b0 0x7fed741532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44588/0 (socket says 172.21.15.119:44588) 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 -- 172.21.15.119:0/2763854612 learned_addr learned my addr 172.21.15.119:0/2763854612 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed737fe700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed74144180 0x7fed740ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed727fc700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed740ce240 0x7fed7413fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 -- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed740ce240 msgr2=0x7fed7413fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed740ce240 0x7fed7413fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 -- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed74144180 msgr2=0x7fed740ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed74144180 0x7fed740ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 -- 172.21.15.119:0/2763854612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c0094b0 con 0x7fed740b22b0 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed72ffd700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740b22b0 0x7fed741532e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00e840 tx=0x7fed5c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed6b7fe700 1 -- 172.21.15.119:0/2763854612 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0095a0 con 0x7fed740b22b0 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed6b7fe700 1 -- 172.21.15.119:0/2763854612 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c013680 con 0x7fed740b22b0 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed741404c0 con 0x7fed740b22b0 2024-09-13T16:26:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed6b7fe700 1 -- 172.21.15.119:0/2763854612 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0139f0 con 0x7fed740b22b0 2024-09-13T16:26:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.061+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed74140a80 con 0x7fed740b22b0 2024-09-13T16:26:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.065+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed740b1e00 con 0x7fed740b22b0 2024-09-13T16:26:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.065+0000 7fed6b7fe700 1 -- 172.21.15.119:0/2763854612 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fed5c013b50 con 0x7fed740b22b0 2024-09-13T16:26:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.065+0000 7fed6b7fe700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed54066760 0x7fed54068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.065+0000 7fed6b7fe700 1 -- 172.21.15.119:0/2763854612 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fed5c0842f0 con 0x7fed740b22b0 2024-09-13T16:26:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.065+0000 7fed727fc700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed54066760 0x7fed54068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.065+0000 7fed727fc700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed54066760 0x7fed54068c20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fed74143eb0 tx=0x7fed640018c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.073+0000 7fed6b7fe700 1 -- 172.21.15.119:0/2763854612 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fed5c051330 con 0x7fed740b22b0 2024-09-13T16:26:09.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.245+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed740cc2b0 con 0x7fed740b22b0 2024-09-13T16:26:09.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.245+0000 7fed6b7fe700 1 -- 172.21.15.119:0/2763854612 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed5c054950 con 0x7fed740b22b0 2024-09-13T16:26:09.251 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed54066760 msgr2=0x7fed54068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed54066760 0x7fed54068c20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fed74143eb0 tx=0x7fed640018c0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740b22b0 msgr2=0x7fed741532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740b22b0 0x7fed741532e0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00e840 tx=0x7fed5c00eb50 comp rx=0 tx=0).stop 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 shutdown_connections 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed54066760 0x7fed54068c20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed74144180 0x7fed740ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed740ce240 0x7fed7413fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 --2- 172.21.15.119:0/2763854612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed740b22b0 0x7fed741532e0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 >> 172.21.15.119:0/2763854612 conn(0x7fed740b0980 msgr2=0x7fed7415b3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:09.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 shutdown_connections 2024-09-13T16:26:09.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.249+0000 7fed73fff700 1 -- 172.21.15.119:0/2763854612 wait complete. 2024-09-13T16:26:09.266 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:09.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 Processor -- start 2024-09-13T16:26:09.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- start start 2024-09-13T16:26:09.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f69a00b1ad0 0x7f69a0071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a0162270 con 0x7f69a006db40 2024-09-13T16:26:09.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69a006ce40 con 0x7f69a0168a10 2024-09-13T16:26:09.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69a00702f0 con 0x7f69a00b1ad0 2024-09-13T16:26:09.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699f7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f69a00b1ad0 0x7f69a0071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699f7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f69a00b1ad0 0x7f69a0071c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44602/0 (socket says 172.21.15.119:44602) 2024-09-13T16:26:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699effd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f69a006db40 0x7f69a00b1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50284/0 (socket says 172.21.15.119:50284) 2024-09-13T16:26:09.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699f7fe700 1 -- 172.21.15.119:0/3794418141 learned_addr learned my addr 172.21.15.119:0/3794418141 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699f7fe700 1 -- 172.21.15.119:0/3794418141 >> v1:172.21.15.145:6789/0 conn(0x7f69a0168a10 legacy=0x7f69a016ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699f7fe700 1 -- 172.21.15.119:0/3794418141 >> v1:172.21.15.119:6789/0 conn(0x7f69a006db40 legacy=0x7f69a00b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699f7fe700 1 -- 172.21.15.119:0/3794418141 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a0071730 con 0x7f69a00b1ad0 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699f7fe700 1 --2- 172.21.15.119:0/3794418141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a00b1ad0 0x7f69a0071c10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6988009910 tx=0x7f6988009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ac88cfe3e47095 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699e7fc700 1 -- 172.21.15.119:0/3794418141 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698800ba60 con 0x7f69a00b1ad0 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699e7fc700 1 -- 172.21.15.119:0/3794418141 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698800bbc0 con 0x7f69a00b1ad0 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f699e7fc700 1 -- 172.21.15.119:0/3794418141 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6988009e60 con 0x7f69a00b1ad0 2024-09-13T16:26:09.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- 172.21.15.119:0/3794418141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a00b1ad0 msgr2=0x7f69a0071c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/3794418141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a00b1ad0 0x7f69a0071c10 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6988009910 tx=0x7f6988009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- 172.21.15.119:0/3794418141 shutdown_connections 2024-09-13T16:26:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/3794418141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a00b1ad0 0x7f69a0071c10 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- 172.21.15.119:0/3794418141 >> 172.21.15.119:0/3794418141 conn(0x7f69a00b0560 msgr2=0x7f69a00b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- 172.21.15.119:0/3794418141 shutdown_connections 2024-09-13T16:26:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.381+0000 7f69a5f9b700 1 -- 172.21.15.119:0/3794418141 wait complete. 2024-09-13T16:26:09.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 Processor -- start 2024-09-13T16:26:09.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 -- start start 2024-09-13T16:26:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69a006db40 0x7f69a0160650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69a00b1ad0 0x7f69a0153d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a0168a10 0x7f69a0155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a015c9e0 con 0x7f69a0168a10 2024-09-13T16:26:09.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69a015bd60 con 0x7f69a00b1ad0 2024-09-13T16:26:09.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f69a5f9b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69a015c040 con 0x7f69a006db40 2024-09-13T16:26:09.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a0168a10 0x7f69a0155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a0168a10 0x7f69a0155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44612/0 (socket says 172.21.15.119:44612) 2024-09-13T16:26:09.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 -- 172.21.15.119:0/1372036298 learned_addr learned my addr 172.21.15.119:0/1372036298 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:09.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699effd700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69a00b1ad0 0x7f69a0153d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699f7fe700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69a006db40 0x7f69a0160650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 -- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69a006db40 msgr2=0x7f69a0160650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69a006db40 0x7f69a0160650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 -- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69a00b1ad0 msgr2=0x7f69a0153d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69a00b1ad0 0x7f69a0153d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 -- 172.21.15.119:0/1372036298 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6988009570 con 0x7f69a0168a10 2024-09-13T16:26:09.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.385+0000 7f699ffff700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a0168a10 0x7f69a0155520 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f699800cb50 tx=0x7f699800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.389+0000 7f6997fff700 1 -- 172.21.15.119:0/1372036298 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6998003c20 con 0x7f69a0168a10 2024-09-13T16:26:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.389+0000 7f6997fff700 1 -- 172.21.15.119:0/1372036298 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6998003d80 con 0x7f69a0168a10 2024-09-13T16:26:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.389+0000 7f6997fff700 1 -- 172.21.15.119:0/1372036298 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f699800aa20 con 0x7f69a0168a10 2024-09-13T16:26:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.389+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69a015c320 con 0x7f69a0168a10 2024-09-13T16:26:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.389+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69a0154260 con 0x7f69a0168a10 2024-09-13T16:26:09.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.389+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69a015cb20 con 0x7f69a0168a10 2024-09-13T16:26:09.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.389+0000 7f6997fff700 1 -- 172.21.15.119:0/1372036298 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f699800ab80 con 0x7f69a0168a10 2024-09-13T16:26:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.393+0000 7f6997fff700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6980066730 0x7f6980068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.393+0000 7f6997fff700 1 -- 172.21.15.119:0/1372036298 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f69980842e0 con 0x7f69a0168a10 2024-09-13T16:26:09.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.393+0000 7f699f7fe700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6980066730 0x7f6980068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.393+0000 7f699f7fe700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6980066730 0x7f6980068bf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6988006f80 tx=0x7f6988008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.397+0000 7f6997fff700 1 -- 172.21.15.119:0/1372036298 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f69a015cb20 con 0x7f69a0168a10 2024-09-13T16:26:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.565+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f69a006ce40 con 0x7f69a0168a10 2024-09-13T16:26:09.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f6997fff700 1 -- 172.21.15.119:0/1372036298 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f69a006ce40 con 0x7f69a0168a10 2024-09-13T16:26:09.572 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:09.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6980066730 msgr2=0x7f6980068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6980066730 0x7f6980068bf0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6988006f80 tx=0x7f6988008030 comp rx=0 tx=0).stop 2024-09-13T16:26:09.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a0168a10 msgr2=0x7f69a0155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a0168a10 0x7f69a0155520 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f699800cb50 tx=0x7f699800ce60 comp rx=0 tx=0).stop 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 shutdown_connections 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6980066730 0x7f6980068bf0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69a0168a10 0x7f69a0155520 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69a00b1ad0 0x7f69a0153d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 --2- 172.21.15.119:0/1372036298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69a006db40 0x7f69a0160650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 >> 172.21.15.119:0/1372036298 conn(0x7f69a00b0560 msgr2=0x7f69a00b1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 shutdown_connections 2024-09-13T16:26:09.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.569+0000 7f69a5f9b700 1 -- 172.21.15.119:0/1372036298 wait complete. 2024-09-13T16:26:09.587 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:09.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a3fff700 1 Processor -- start 2024-09-13T16:26:09.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a3fff700 1 -- start start 2024-09-13T16:26:09.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a40ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9a408c6b0 0x7fe9a4166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a3fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a415d020 con 0x7fe9a40cd860 2024-09-13T16:26:09.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a3fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9a40b44c0 con 0x7fe9a408c6b0 2024-09-13T16:26:09.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40d0720 con 0x7fe9a415cc40 2024-09-13T16:26:09.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a27fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe9a40cd860 0x7fe9a408bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50298/0 (socket says 172.21.15.119:50298) 2024-09-13T16:26:09.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9a408c6b0 0x7fe9a4166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a27fc700 1 -- 172.21.15.119:0/2565560711 learned_addr learned my addr 172.21.15.119:0/2565560711 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:09.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a2ffd700 1 --2- 172.21.15.119:0/2565560711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a40ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a37fe700 1 -- 172.21.15.119:0/2565560711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 msgr2=0x7fe9a40ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:09.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a37fe700 1 --2- 172.21.15.119:0/2565560711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a40ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a37fe700 1 -- 172.21.15.119:0/2565560711 >> v1:172.21.15.119:6789/0 conn(0x7fe9a40cd860 legacy=0x7fe9a408bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a37fe700 1 -- 172.21.15.119:0/2565560711 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a40d09c0 con 0x7fe9a408c6b0 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.693+0000 7fe9a37fe700 1 --2- 172.21.15.119:0/2565560711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a4166a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009850 tx=0x7fe99c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c90cb8a5a0521ad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a1ffb700 1 -- 172.21.15.119:0/2565560711 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c011830 con 0x7fe9a408c6b0 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a1ffb700 1 -- 172.21.15.119:0/2565560711 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99c011990 con 0x7fe9a408c6b0 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a1ffb700 1 -- 172.21.15.119:0/2565560711 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99c011d40 con 0x7fe9a408c6b0 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 -- 172.21.15.119:0/2565560711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 msgr2=0x7fe9a4166a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/2565560711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a4166a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fe99c009850 tx=0x7fe99c009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 -- 172.21.15.119:0/2565560711 shutdown_connections 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/2565560711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a4166a50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/2565560711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a40ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 -- 172.21.15.119:0/2565560711 >> 172.21.15.119:0/2565560711 conn(0x7fe9a40b4640 msgr2=0x7fe9a40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:09.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 -- 172.21.15.119:0/2565560711 shutdown_connections 2024-09-13T16:26:09.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 -- 172.21.15.119:0/2565560711 wait complete. 2024-09-13T16:26:09.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 Processor -- start 2024-09-13T16:26:09.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 -- start start 2024-09-13T16:26:09.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.697+0000 7fe9a3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a415e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9a40cd860 0x7fe9a40c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a415f810 con 0x7fe9a408c6b0 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a40c97e0 con 0x7fe9a40cd860 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9a415eb90 con 0x7fe9a415cc40 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a415e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a415e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44638/0 (socket says 172.21.15.119:44638) 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 -- 172.21.15.119:0/193317144 learned_addr learned my addr 172.21.15.119:0/193317144 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a37fe700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a27fc700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9a40cd860 0x7fe9a40c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 -- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 msgr2=0x7fe9a415a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a415a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 -- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9a40cd860 msgr2=0x7fe9a40c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9a40cd860 0x7fe9a40c79f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 -- 172.21.15.119:0/193317144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe99c0094b0 con 0x7fe9a408c6b0 2024-09-13T16:26:09.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a2ffd700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a415e4d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe99400efc0 tx=0x7fe99400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe99bfff700 1 -- 172.21.15.119:0/193317144 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe99400cdc0 con 0x7fe9a408c6b0 2024-09-13T16:26:09.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe99bfff700 1 -- 172.21.15.119:0/193317144 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe99400f040 con 0x7fe9a408c6b0 2024-09-13T16:26:09.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a415ee70 con 0x7fe9a408c6b0 2024-09-13T16:26:09.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe99bfff700 1 -- 172.21.15.119:0/193317144 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9940146a0 con 0x7fe9a408c6b0 2024-09-13T16:26:09.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a415f430 con 0x7fe9a408c6b0 2024-09-13T16:26:09.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a415fa80 con 0x7fe9a408c6b0 2024-09-13T16:26:09.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.701+0000 7fe99bfff700 1 -- 172.21.15.119:0/193317144 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe994014800 con 0x7fe9a408c6b0 2024-09-13T16:26:09.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.705+0000 7fe99bfff700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe984066730 0x7fe984068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:09.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.705+0000 7fe99bfff700 1 -- 172.21.15.119:0/193317144 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe994084ea0 con 0x7fe9a408c6b0 2024-09-13T16:26:09.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.705+0000 7fe9a27fc700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe984066730 0x7fe984068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:09.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.705+0000 7fe9a27fc700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe984066730 0x7fe984068bf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe9a41f5dd0 tx=0x7fe98c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:09.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.713+0000 7fe99bfff700 1 -- 172.21.15.119:0/193317144 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe994051e50 con 0x7fe9a408c6b0 2024-09-13T16:26:09.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe9a40cec70 con 0x7fe9a408c6b0 2024-09-13T16:26:09.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe99bfff700 1 -- 172.21.15.119:0/193317144 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe994055470 con 0x7fe9a408c6b0 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe984066730 msgr2=0x7fe984068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe984066730 0x7fe984068bf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe9a41f5dd0 tx=0x7fe98c009040 comp rx=0 tx=0).stop 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 msgr2=0x7fe9a415e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a415e4d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fe99400efc0 tx=0x7fe99400c3a0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 shutdown_connections 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe984066730 0x7fe984068bf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9a415cc40 0x7fe9a415a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9a40cd860 0x7fe9a40c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 --2- 172.21.15.119:0/193317144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9a408c6b0 0x7fe9a415e4d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:09.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 >> 172.21.15.119:0/193317144 conn(0x7fe9a40b4640 msgr2=0x7fe9a40b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:09.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 shutdown_connections 2024-09-13T16:26:09.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:09.893+0000 7fe9a3fff700 1 -- 172.21.15.119:0/193317144 wait complete. 2024-09-13T16:26:09.900 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:09.915 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:26:09.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:09.915 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:09.926 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:09.926 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:26:09.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:09.927 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:26:09.932 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:26:09.932 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:09.940 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:26:09.940 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:09.947 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:10.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 Processor -- start 2024-09-13T16:26:10.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- start start 2024-09-13T16:26:10.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe1d80cd8d0 0x7fe1d806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d80705d0 con 0x7fe1d80b2200 2024-09-13T16:26:10.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1d806d0d0 con 0x7fe1d80b17f0 2024-09-13T16:26:10.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1d80cbf10 con 0x7fe1d80cd8d0 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe1d80cd8d0 0x7fe1d806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe1d80cd8d0 0x7fe1d806cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44652/0 (socket says 172.21.15.119:44652) 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d6ffd700 1 -- 172.21.15.119:0/126291927 learned_addr learned my addr 172.21.15.119:0/126291927 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d77fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe1d80b2200 0x7fe1d8070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50302/0 (socket says 172.21.15.119:50302) 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d6ffd700 1 -- 172.21.15.119:0/126291927 >> v1:172.21.15.145:6789/0 conn(0x7fe1d80b17f0 legacy=0x7fe1d80cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d6ffd700 1 -- 172.21.15.119:0/126291927 >> v1:172.21.15.119:6789/0 conn(0x7fe1d80b2200 legacy=0x7fe1d8070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d6ffd700 1 -- 172.21.15.119:0/126291927 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d806fca0 con 0x7fe1d80cd8d0 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d6ffd700 1 --2- 172.21.15.119:0/126291927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80cd8d0 0x7fe1d806cb90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009e50 tx=0x7fe1c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae48a6398e23eae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d5ffb700 1 -- 172.21.15.119:0/126291927 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c800e070 con 0x7fe1d80cd8d0 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d5ffb700 1 -- 172.21.15.119:0/126291927 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c800b9c0 con 0x7fe1d80cd8d0 2024-09-13T16:26:10.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d5ffb700 1 -- 172.21.15.119:0/126291927 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c800bd70 con 0x7fe1d80cd8d0 2024-09-13T16:26:10.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- 172.21.15.119:0/126291927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80cd8d0 msgr2=0x7fe1d806cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/126291927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80cd8d0 0x7fe1d806cb90 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8009e50 tx=0x7fe1c800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:10.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- 172.21.15.119:0/126291927 shutdown_connections 2024-09-13T16:26:10.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/126291927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80cd8d0 0x7fe1d806cb90 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- 172.21.15.119:0/126291927 >> 172.21.15.119:0/126291927 conn(0x7fe1d806a350 msgr2=0x7fe1d806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:10.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- 172.21.15.119:0/126291927 shutdown_connections 2024-09-13T16:26:10.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.061+0000 7fe1d7fff700 1 -- 172.21.15.119:0/126291927 wait complete. 2024-09-13T16:26:10.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 Processor -- start 2024-09-13T16:26:10.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 -- start start 2024-09-13T16:26:10.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1d80b17f0 0x7fe1d80b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80b2200 0x7fe1d81f9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1d80cd8d0 0x7fe1d81fa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d814d370 con 0x7fe1d80b2200 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d80b0d50 con 0x7fe1d80b17f0 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d7fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1d814dc00 con 0x7fe1d80cd8d0 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d67fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80b2200 0x7fe1d81f9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d67fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80b2200 0x7fe1d81f9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44654/0 (socket says 172.21.15.119:44654) 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d67fc700 1 -- 172.21.15.119:0/1841224107 learned_addr learned my addr 172.21.15.119:0/1841224107 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.065+0000 7fe1d77fe700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1d80cd8d0 0x7fe1d81fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d6ffd700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1d80b17f0 0x7fe1d80b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d67fc700 1 -- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1d80cd8d0 msgr2=0x7fe1d81fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d67fc700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1d80cd8d0 0x7fe1d81fa090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d67fc700 1 -- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1d80b17f0 msgr2=0x7fe1d80b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d67fc700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1d80b17f0 0x7fe1d80b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d67fc700 1 -- 172.21.15.119:0/1841224107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1c8009b30 con 0x7fe1d80b2200 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d77fe700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1d80cd8d0 0x7fe1d81fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d6ffd700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1d80b17f0 0x7fe1d80b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d67fc700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80b2200 0x7fe1d81f9b50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c500 tx=0x7fe1c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1cf7fe700 1 -- 172.21.15.119:0/1841224107 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00052d0 con 0x7fe1d80b2200 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1cf7fe700 1 -- 172.21.15.119:0/1841224107 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c0005430 con 0x7fe1d80b2200 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1d80c7030 con 0x7fe1d80b2200 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1cf7fe700 1 -- 172.21.15.119:0/1841224107 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1c00055a0 con 0x7fe1d80b2200 2024-09-13T16:26:10.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1d814d5b0 con 0x7fe1d80b2200 2024-09-13T16:26:10.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1d806cb90 con 0x7fe1d80b2200 2024-09-13T16:26:10.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1cf7fe700 1 -- 172.21.15.119:0/1841224107 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe1c00038e0 con 0x7fe1d80b2200 2024-09-13T16:26:10.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1cf7fe700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe1b8066730 0x7fe1b8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1cf7fe700 1 -- 172.21.15.119:0/1841224107 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe1c0083e20 con 0x7fe1d80b2200 2024-09-13T16:26:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.069+0000 7fe1d6ffd700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe1b8066730 0x7fe1b8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.073+0000 7fe1d6ffd700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe1b8066730 0x7fe1b8068bf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8000f80 tx=0x7fe1c8009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.077+0000 7fe1cf7fe700 1 -- 172.21.15.119:0/1841224107 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe1c0050e60 con 0x7fe1d80b2200 2024-09-13T16:26:10.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.245+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1d8072c50 con 0x7fe1d80b2200 2024-09-13T16:26:10.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.245+0000 7fe1cf7fe700 1 -- 172.21.15.119:0/1841224107 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7fe1c0017090 con 0x7fe1d80b2200 2024-09-13T16:26:10.249 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:10.250 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:10.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe1b8066730 msgr2=0x7fe1b8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe1b8066730 0x7fe1b8068bf0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fe1c8000f80 tx=0x7fe1c8009000 comp rx=0 tx=0).stop 2024-09-13T16:26:10.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80b2200 msgr2=0x7fe1d81f9b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80b2200 0x7fe1d81f9b50 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c500 tx=0x7fe1c000c8c0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 shutdown_connections 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe1b8066730 0x7fe1b8068bf0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1d80cd8d0 0x7fe1d81fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1d80b2200 0x7fe1d81f9b50 secure :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fe1c000c500 tx=0x7fe1c000c8c0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 --2- 172.21.15.119:0/1841224107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1d80b17f0 0x7fe1d80b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 >> 172.21.15.119:0/1841224107 conn(0x7fe1d806a350 msgr2=0x7fe1d80cc060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 shutdown_connections 2024-09-13T16:26:10.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.249+0000 7fe1d7fff700 1 -- 172.21.15.119:0/1841224107 wait complete. 2024-09-13T16:26:10.270 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:10.271 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:10.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d43fff700 1 Processor -- start 2024-09-13T16:26:10.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d43fff700 1 -- start start 2024-09-13T16:26:10.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d43fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2d440ade80 0x7f2d441652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d43fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d44071ac0 con 0x7f2d440b24a0 2024-09-13T16:26:10.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d43fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d44070eb0 con 0x7f2d4406cdb0 2024-09-13T16:26:10.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d43fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d441657f0 con 0x7f2d440ade80 2024-09-13T16:26:10.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d427fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2d440b24a0 0x7f2d441612d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50304/0 (socket says 172.21.15.119:50304) 2024-09-13T16:26:10.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d427fc700 1 -- 172.21.15.119:0/11363757 learned_addr learned my addr 172.21.15.119:0/11363757 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:10.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d42ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2d440ade80 0x7f2d441652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d41ffb700 1 -- 172.21.15.119:0/11363757 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172093500 0 0) 0x7f2d44071ac0 con 0x7f2d440b24a0 2024-09-13T16:26:10.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d41ffb700 1 -- 172.21.15.119:0/11363757 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d2c0035d0 con 0x7f2d440b24a0 2024-09-13T16:26:10.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d42ffd700 1 -- 172.21.15.119:0/11363757 >> v1:172.21.15.145:6789/0 conn(0x7f2d4406cdb0 legacy=0x7f2d441619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d42ffd700 1 -- 172.21.15.119:0/11363757 >> v1:172.21.15.119:6789/0 conn(0x7f2d440b24a0 legacy=0x7f2d441612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d42ffd700 1 -- 172.21.15.119:0/11363757 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d44071190 con 0x7f2d440ade80 2024-09-13T16:26:10.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d42ffd700 1 --2- 172.21.15.119:0/11363757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d440ade80 0x7f2d441652b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009850 tx=0x7f2d38009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f9e987d0be11156 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d41ffb700 1 -- 172.21.15.119:0/11363757 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38011830 con 0x7f2d440ade80 2024-09-13T16:26:10.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d41ffb700 1 -- 172.21.15.119:0/11363757 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d38011990 con 0x7f2d440ade80 2024-09-13T16:26:10.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.389+0000 7f2d41ffb700 1 -- 172.21.15.119:0/11363757 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d38011d40 con 0x7f2d440ade80 2024-09-13T16:26:10.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- 172.21.15.119:0/11363757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d440ade80 msgr2=0x7f2d441652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 --2- 172.21.15.119:0/11363757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d440ade80 0x7f2d441652b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f2d38009850 tx=0x7f2d38009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:10.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- 172.21.15.119:0/11363757 shutdown_connections 2024-09-13T16:26:10.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 --2- 172.21.15.119:0/11363757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d440ade80 0x7f2d441652b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- 172.21.15.119:0/11363757 >> 172.21.15.119:0/11363757 conn(0x7f2d440b0980 msgr2=0x7f2d440b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:10.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- 172.21.15.119:0/11363757 shutdown_connections 2024-09-13T16:26:10.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- 172.21.15.119:0/11363757 wait complete. 2024-09-13T16:26:10.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 Processor -- start 2024-09-13T16:26:10.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- start start 2024-09-13T16:26:10.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d4406cdb0 0x7f2d441fa410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d440ade80 0x7f2d441f5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d440b24a0 0x7f2d4406bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d441fa950 con 0x7f2d4406cdb0 2024-09-13T16:26:10.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d4414e3f0 con 0x7f2d440b24a0 2024-09-13T16:26:10.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d43fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d44150bc0 con 0x7f2d440ade80 2024-09-13T16:26:10.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d42ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d4406cdb0 0x7f2d441fa410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d42ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d4406cdb0 0x7f2d441fa410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44668/0 (socket says 172.21.15.119:44668) 2024-09-13T16:26:10.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.393+0000 7f2d42ffd700 1 -- 172.21.15.119:0/4219315478 learned_addr learned my addr 172.21.15.119:0/4219315478 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:10.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d427fc700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d440ade80 0x7f2d441f5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d437fe700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d440b24a0 0x7f2d4406bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d42ffd700 1 -- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d440ade80 msgr2=0x7f2d441f5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d42ffd700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d440ade80 0x7f2d441f5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d42ffd700 1 -- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d440b24a0 msgr2=0x7f2d4406bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d42ffd700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d440b24a0 0x7f2d4406bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d42ffd700 1 -- 172.21.15.119:0/4219315478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d380094b0 con 0x7f2d4406cdb0 2024-09-13T16:26:10.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d42ffd700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d4406cdb0 0x7f2d441fa410 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2d38000c00 tx=0x7f2d38004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d2bfff700 1 -- 172.21.15.119:0/4219315478 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3801a990 con 0x7f2d4406cdb0 2024-09-13T16:26:10.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d2bfff700 1 -- 172.21.15.119:0/4219315478 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d3801f070 con 0x7f2d4406cdb0 2024-09-13T16:26:10.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d44150ea0 con 0x7f2d4406cdb0 2024-09-13T16:26:10.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d2bfff700 1 -- 172.21.15.119:0/4219315478 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d3801ad30 con 0x7f2d4406cdb0 2024-09-13T16:26:10.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d44151460 con 0x7f2d4406cdb0 2024-09-13T16:26:10.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.397+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d44072570 con 0x7f2d4406cdb0 2024-09-13T16:26:10.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.401+0000 7f2d2bfff700 1 -- 172.21.15.119:0/4219315478 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2d3801b430 con 0x7f2d4406cdb0 2024-09-13T16:26:10.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.401+0000 7f2d2bfff700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d30066730 0x7f2d30068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.401+0000 7f2d2bfff700 1 -- 172.21.15.119:0/4219315478 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2d38085500 con 0x7f2d4406cdb0 2024-09-13T16:26:10.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.401+0000 7f2d427fc700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d30066730 0x7f2d30068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.401+0000 7f2d427fc700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d30066730 0x7f2d30068bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2d44164840 tx=0x7f2d34007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.405+0000 7f2d2bfff700 1 -- 172.21.15.119:0/4219315478 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2d38052540 con 0x7f2d4406cdb0 2024-09-13T16:26:10.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.573+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d44151650 con 0x7f2d4406cdb0 2024-09-13T16:26:10.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d2bfff700 1 -- 172.21.15.119:0/4219315478 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2d38021020 con 0x7f2d4406cdb0 2024-09-13T16:26:10.579 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:10.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d30066730 msgr2=0x7f2d30068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d30066730 0x7f2d30068bf0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2d44164840 tx=0x7f2d34007de0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d4406cdb0 msgr2=0x7f2d441fa410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d4406cdb0 0x7f2d441fa410 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f2d38000c00 tx=0x7f2d38004770 comp rx=0 tx=0).stop 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 shutdown_connections 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d30066730 0x7f2d30068bf0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d440b24a0 0x7f2d4406bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d440ade80 0x7f2d441f5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 --2- 172.21.15.119:0/4219315478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d4406cdb0 0x7f2d441fa410 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 >> 172.21.15.119:0/4219315478 conn(0x7f2d440b0980 msgr2=0x7f2d44162bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 shutdown_connections 2024-09-13T16:26:10.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.577+0000 7f2d43fff700 1 -- 172.21.15.119:0/4219315478 wait complete. 2024-09-13T16:26:10.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062955700 1 Processor -- start 2024-09-13T16:26:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062955700 1 -- start start 2024-09-13T16:26:10.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062955700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f005c0ce5f0 0x7f005c0cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062955700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c0d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062955700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f005c0cd600 con 0x7f005c15cae0 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062955700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f005c0ce1d0 con 0x7f005c0ce5f0 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062955700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c0cea60 con 0x7f005c0d0fb0 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062154700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c0d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0061152700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f005c0ce5f0 0x7f005c0cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0061953700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f005c15cae0 0x7f005c0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:55042/0 (socket says 172.21.15.119:55042) 2024-09-13T16:26:10.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0061953700 1 -- 172.21.15.119:0/2103843446 learned_addr learned my addr 172.21.15.119:0/2103843446 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0060951700 1 -- 172.21.15.119:0/2103843446 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048204794 0 0) 0x7f005c0cd600 con 0x7f005c15cae0 2024-09-13T16:26:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0060951700 1 -- 172.21.15.119:0/2103843446 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00480035d0 con 0x7f005c15cae0 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062154700 1 -- 172.21.15.119:0/2103843446 >> v2:172.21.15.145:3300/0 conn(0x7f005c0ce5f0 msgr2=0x7f005c0cff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062154700 1 --2- 172.21.15.119:0/2103843446 >> v2:172.21.15.145:3300/0 conn(0x7f005c0ce5f0 0x7f005c0cff60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062154700 1 -- 172.21.15.119:0/2103843446 >> v1:172.21.15.145:6789/0 conn(0x7f005c15cae0 legacy=0x7f005c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.713+0000 7f0062154700 1 -- 172.21.15.119:0/2103843446 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f005c0d0dd0 con 0x7f005c0d0fb0 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062154700 1 --2- 172.21.15.119:0/2103843446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c0d1420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0054009d90 tx=0x7f005400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4252f6f020597c01 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0060951700 1 -- 172.21.15.119:0/2103843446 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005400e070 con 0x7f005c0d0fb0 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0060951700 1 -- 172.21.15.119:0/2103843446 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005400bb20 con 0x7f005c0d0fb0 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0060951700 1 -- 172.21.15.119:0/2103843446 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054012430 con 0x7f005c0d0fb0 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 -- 172.21.15.119:0/2103843446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 msgr2=0x7f005c0d1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 --2- 172.21.15.119:0/2103843446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c0d1420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0054009d90 tx=0x7f005400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 -- 172.21.15.119:0/2103843446 shutdown_connections 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 --2- 172.21.15.119:0/2103843446 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c0d1420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 --2- 172.21.15.119:0/2103843446 >> v2:172.21.15.145:3300/0 conn(0x7f005c0ce5f0 0x7f005c0cff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 -- 172.21.15.119:0/2103843446 >> 172.21.15.119:0/2103843446 conn(0x7f005c0d29f0 msgr2=0x7f005c0d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 -- 172.21.15.119:0/2103843446 shutdown_connections 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 -- 172.21.15.119:0/2103843446 wait complete. 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 Processor -- start 2024-09-13T16:26:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.717+0000 7f0062955700 1 -- start start 2024-09-13T16:26:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f005c0ce5f0 0x7f005c15d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c1f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f005c15cae0 0x7f005c1f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c156510 con 0x7f005c15cae0 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f005c15fd80 con 0x7f005c0ce5f0 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f005c160060 con 0x7f005c0d0fb0 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f005c15cae0 0x7f005c1f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f005c15cae0 0x7f005c1f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44684/0 (socket says 172.21.15.119:44684) 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 -- 172.21.15.119:0/1135286894 learned_addr learned my addr 172.21.15.119:0/1135286894 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0061953700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f005c0ce5f0 0x7f005c15d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0061152700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c1f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 -- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 msgr2=0x7f005c1f21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c1f21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 -- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f005c0ce5f0 msgr2=0x7f005c15d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f005c0ce5f0 0x7f005c15d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 -- 172.21.15.119:0/1135286894 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0054009a70 con 0x7f005c15cae0 2024-09-13T16:26:10.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062154700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f005c15cae0 0x7f005c1f26e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f005400b680 tx=0x7f0054004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f00527fc700 1 -- 172.21.15.119:0/1135286894 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054012430 con 0x7f005c15cae0 2024-09-13T16:26:10.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f00527fc700 1 -- 172.21.15.119:0/1135286894 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0054020070 con 0x7f005c15cae0 2024-09-13T16:26:10.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f005c0c6680 con 0x7f005c15cae0 2024-09-13T16:26:10.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f00527fc700 1 -- 172.21.15.119:0/1135286894 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0054004280 con 0x7f005c15cae0 2024-09-13T16:26:10.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.721+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f005c160340 con 0x7f005c15cae0 2024-09-13T16:26:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.725+0000 7f00527fc700 1 -- 172.21.15.119:0/1135286894 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f00540043e0 con 0x7f005c15cae0 2024-09-13T16:26:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.725+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f005c0b3ac0 con 0x7f005c15cae0 2024-09-13T16:26:10.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.725+0000 7f00527fc700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0040066760 0x7f0040068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.725+0000 7f00527fc700 1 -- 172.21.15.119:0/1135286894 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0054022070 con 0x7f005c15cae0 2024-09-13T16:26:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.725+0000 7f0061953700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0040066760 0x7f0040068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:10.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.725+0000 7f0061953700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0040066760 0x7f0040068c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f004c007fd0 tx=0x7f004c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:10.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.733+0000 7f00527fc700 1 -- 172.21.15.119:0/1135286894 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0054052fe0 con 0x7f005c15cae0 2024-09-13T16:26:10.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f005c15cec0 con 0x7f005c15cae0 2024-09-13T16:26:10.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f00527fc700 1 -- 172.21.15.119:0/1135286894 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0054056600 con 0x7f005c15cae0 2024-09-13T16:26:10.904 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:10.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0040066760 msgr2=0x7f0040068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0040066760 0x7f0040068c20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f004c007fd0 tx=0x7f004c009040 comp rx=0 tx=0).stop 2024-09-13T16:26:10.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f005c15cae0 msgr2=0x7f005c1f26e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:10.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f005c15cae0 0x7f005c1f26e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f005400b680 tx=0x7f0054004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 shutdown_connections 2024-09-13T16:26:10.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0040066760 0x7f0040068c20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f005c15cae0 0x7f005c1f26e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f005c0d0fb0 0x7f005c1f21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 --2- 172.21.15.119:0/1135286894 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f005c0ce5f0 0x7f005c15d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:10.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 >> 172.21.15.119:0/1135286894 conn(0x7f005c0d29f0 msgr2=0x7f005c0c9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:10.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 shutdown_connections 2024-09-13T16:26:10.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:10.901+0000 7f0062955700 1 -- 172.21.15.119:0/1135286894 wait complete. 2024-09-13T16:26:10.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:11.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a24f9700 1 Processor -- start 2024-09-13T16:26:11.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a24f9700 1 -- start start 2024-09-13T16:26:11.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a24f9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb39c0ce5f0 0x7fb39c0cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a24f9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb39c0d0fb0 0x7fb39c0d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a24f9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb39c0cd600 con 0x7fb39c15cae0 2024-09-13T16:26:11.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a24f9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c0ce1d0 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a24f9700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb39c0cea60 con 0x7fb39c0d0fb0 2024-09-13T16:26:11.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a14f7700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb39c15cae0 0x7fb39c0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50316/0 (socket says 172.21.15.119:50316) 2024-09-13T16:26:11.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a14f7700 1 -- 172.21.15.119:0/1002233168 learned_addr learned my addr 172.21.15.119:0/1002233168 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:11.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a0cf6700 1 --2- 172.21.15.119:0/1002233168 >> v2:172.21.15.119:3300/0 conn(0x7fb39c0ce5f0 0x7fb39c0cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a1cf8700 1 --2- 172.21.15.119:0/1002233168 >> v2:172.21.15.145:3300/0 conn(0x7fb39c0d0fb0 0x7fb39c0d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a0cf6700 1 -- 172.21.15.119:0/1002233168 >> v2:172.21.15.145:3300/0 conn(0x7fb39c0d0fb0 msgr2=0x7fb39c0d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:11.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a0cf6700 1 --2- 172.21.15.119:0/1002233168 >> v2:172.21.15.145:3300/0 conn(0x7fb39c0d0fb0 0x7fb39c0d1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a0cf6700 1 -- 172.21.15.119:0/1002233168 >> v1:172.21.15.119:6789/0 conn(0x7fb39c15cae0 legacy=0x7fb39c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a0cf6700 1 -- 172.21.15.119:0/1002233168 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb39c0d0dd0 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb393fff700 1 -- 172.21.15.119:0/1002233168 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228142803 0 0) 0x7fb39c0cd600 con 0x7fb39c15cae0 2024-09-13T16:26:11.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb3a0cf6700 1 --2- 172.21.15.119:0/1002233168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c0cff60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb39c08baf0 tx=0x7fb38c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb525bdc4ad4372b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.037+0000 7fb393fff700 1 -- 172.21.15.119:0/1002233168 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c011830 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb393fff700 1 -- 172.21.15.119:0/1002233168 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb38c011990 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb393fff700 1 -- 172.21.15.119:0/1002233168 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb38c011d20 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- 172.21.15.119:0/1002233168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 msgr2=0x7fb39c0cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/1002233168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c0cff60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb39c08baf0 tx=0x7fb38c009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:11.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- 172.21.15.119:0/1002233168 shutdown_connections 2024-09-13T16:26:11.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/1002233168 >> v2:172.21.15.145:3300/0 conn(0x7fb39c0d0fb0 0x7fb39c0d1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/1002233168 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c0cff60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- 172.21.15.119:0/1002233168 >> 172.21.15.119:0/1002233168 conn(0x7fb39c0d29f0 msgr2=0x7fb39c0d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:11.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- 172.21.15.119:0/1002233168 shutdown_connections 2024-09-13T16:26:11.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- 172.21.15.119:0/1002233168 wait complete. 2024-09-13T16:26:11.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 Processor -- start 2024-09-13T16:26:11.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- start start 2024-09-13T16:26:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c14a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb39c0d0fb0 0x7fb39c1f19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb39c15cae0 0x7fb39c1586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c149d30 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c149720 con 0x7fb39c0d0fb0 2024-09-13T16:26:11.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.041+0000 7fb3a24f9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39c0b38c0 con 0x7fb39c15cae0 2024-09-13T16:26:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c14a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c14a1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44698/0 (socket says 172.21.15.119:44698) 2024-09-13T16:26:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 -- 172.21.15.119:0/140839587 learned_addr learned my addr 172.21.15.119:0/140839587 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a1cf8700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb39c15cae0 0x7fb39c1586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a0cf6700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb39c0d0fb0 0x7fb39c1f19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 -- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb39c15cae0 msgr2=0x7fb39c1586c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb39c15cae0 0x7fb39c1586c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 -- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb39c0d0fb0 msgr2=0x7fb39c1f19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb39c0d0fb0 0x7fb39c1f19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 -- 172.21.15.119:0/140839587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38c0094b0 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a14f7700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c14a1c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb39800c580 tx=0x7fb39800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb391ffb700 1 -- 172.21.15.119:0/140839587 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398004ca0 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb391ffb700 1 -- 172.21.15.119:0/140839587 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb398004e20 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb39c1f6290 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb391ffb700 1 -- 172.21.15.119:0/140839587 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb398007310 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb39c1f6570 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.045+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb39c15cec0 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.049+0000 7fb391ffb700 1 -- 172.21.15.119:0/140839587 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb398007470 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.049+0000 7fb391ffb700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb380066730 0x7fb380068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.049+0000 7fb391ffb700 1 -- 172.21.15.119:0/140839587 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fb398084170 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.049+0000 7fb3a0cf6700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb380066730 0x7fb380068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.049+0000 7fb3a0cf6700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb380066730 0x7fb380068bf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0090b0 tx=0x7fb38c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.061+0000 7fb391ffb700 1 -- 172.21.15.119:0/140839587 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb398051120 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.229+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb39c0744c0 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.229+0000 7fb391ffb700 1 -- 172.21.15.119:0/140839587 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb398007720 con 0x7fb39c0ce5f0 2024-09-13T16:26:11.235 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:11.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb380066730 msgr2=0x7fb380068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb380066730 0x7fb380068bf0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb38c0090b0 tx=0x7fb38c008030 comp rx=0 tx=0).stop 2024-09-13T16:26:11.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 msgr2=0x7fb39c14a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c14a1c0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fb39800c580 tx=0x7fb39800c890 comp rx=0 tx=0).stop 2024-09-13T16:26:11.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 shutdown_connections 2024-09-13T16:26:11.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb380066730 0x7fb380068bf0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb39c15cae0 0x7fb39c1586c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb39c0d0fb0 0x7fb39c1f19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 --2- 172.21.15.119:0/140839587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb39c0ce5f0 0x7fb39c14a1c0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 >> 172.21.15.119:0/140839587 conn(0x7fb39c0d29f0 msgr2=0x7fb39c15f0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:11.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 shutdown_connections 2024-09-13T16:26:11.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.233+0000 7fb3a24f9700 1 -- 172.21.15.119:0/140839587 wait complete. 2024-09-13T16:26:11.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:11.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.365+0000 7f23c4e5e700 1 Processor -- start 2024-09-13T16:26:11.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.365+0000 7f23c4e5e700 1 -- start start 2024-09-13T16:26:11.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f23b80e09e0 0x7f23b8036200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8037ac0 con 0x7f23b80e0dc0 2024-09-13T16:26:11.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b8057870 con 0x7f23b803bfb0 2024-09-13T16:26:11.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23b8038210 con 0x7f23b80e09e0 2024-09-13T16:26:11.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23bffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f23b80e0dc0 0x7f23b8006ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50320/0 (socket says 172.21.15.119:50320) 2024-09-13T16:26:11.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23bffff700 1 -- 172.21.15.119:0/4186452954 learned_addr learned my addr 172.21.15.119:0/4186452954 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:11.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23bf7fe700 1 --2- 172.21.15.119:0/4186452954 >> v2:172.21.15.145:3300/0 conn(0x7f23b80e09e0 0x7f23b8036200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23be7fc700 1 -- 172.21.15.119:0/4186452954 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918929573 0 0) 0x7f23b8037ac0 con 0x7f23b80e0dc0 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23be7fc700 1 -- 172.21.15.119:0/4186452954 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b00035d0 con 0x7f23b80e0dc0 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23bf7fe700 1 -- 172.21.15.119:0/4186452954 >> v1:172.21.15.145:6789/0 conn(0x7f23b803bfb0 legacy=0x7f23b80544c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23bf7fe700 1 -- 172.21.15.119:0/4186452954 >> v1:172.21.15.119:6789/0 conn(0x7f23b80e0dc0 legacy=0x7f23b8006ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23bf7fe700 1 -- 172.21.15.119:0/4186452954 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b8037d80 con 0x7f23b80e09e0 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23bf7fe700 1 --2- 172.21.15.119:0/4186452954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b80e09e0 0x7f23b8036200 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009850 tx=0x7f23b4009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a2b70dc8b503434 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23be7fc700 1 -- 172.21.15.119:0/4186452954 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4011830 con 0x7f23b80e09e0 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23be7fc700 1 -- 172.21.15.119:0/4186452954 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b4011990 con 0x7f23b80e09e0 2024-09-13T16:26:11.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23be7fc700 1 -- 172.21.15.119:0/4186452954 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b4011d40 con 0x7f23b80e09e0 2024-09-13T16:26:11.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- 172.21.15.119:0/4186452954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b80e09e0 msgr2=0x7f23b8036200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/4186452954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b80e09e0 0x7f23b8036200 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f23b4009850 tx=0x7f23b4009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:11.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- 172.21.15.119:0/4186452954 shutdown_connections 2024-09-13T16:26:11.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/4186452954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b80e09e0 0x7f23b8036200 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- 172.21.15.119:0/4186452954 >> 172.21.15.119:0/4186452954 conn(0x7f23b801fbc0 msgr2=0x7f23b80568f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:11.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- 172.21.15.119:0/4186452954 shutdown_connections 2024-09-13T16:26:11.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.369+0000 7f23c4e5e700 1 -- 172.21.15.119:0/4186452954 wait complete. 2024-09-13T16:26:11.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 Processor -- start 2024-09-13T16:26:11.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 -- start start 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b803bfb0 0x7f23b804a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b80e09e0 0x7f23b8175d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23b80e0dc0 0x7f23b80deea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b8022bc0 con 0x7f23b80e09e0 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b80de590 con 0x7f23b803bfb0 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23c4e5e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b804a8b0 con 0x7f23b80e0dc0 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23beffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b80e09e0 0x7f23b8175d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23beffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b80e09e0 0x7f23b8175d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44712/0 (socket says 172.21.15.119:44712) 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.373+0000 7f23beffd700 1 -- 172.21.15.119:0/2301744840 learned_addr learned my addr 172.21.15.119:0/2301744840 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:11.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23bf7fe700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b803bfb0 0x7f23b804a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23bffff700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23b80e0dc0 0x7f23b80deea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23beffd700 1 -- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23b80e0dc0 msgr2=0x7f23b80deea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23beffd700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23b80e0dc0 0x7f23b80deea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23beffd700 1 -- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b803bfb0 msgr2=0x7f23b804a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23beffd700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b803bfb0 0x7f23b804a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23beffd700 1 -- 172.21.15.119:0/2301744840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b40094b0 con 0x7f23b80e09e0 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23bf7fe700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b803bfb0 0x7f23b804a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:11.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23beffd700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b80e09e0 0x7f23b8175d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c480 tx=0x7f23ac00c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23abfff700 1 -- 172.21.15.119:0/2301744840 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac0052d0 con 0x7f23b80e09e0 2024-09-13T16:26:11.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23abfff700 1 -- 172.21.15.119:0/2301744840 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23ac00ce90 con 0x7f23b80e09e0 2024-09-13T16:26:11.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b804ab90 con 0x7f23b80e09e0 2024-09-13T16:26:11.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23abfff700 1 -- 172.21.15.119:0/2301744840 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ac006880 con 0x7f23b80e09e0 2024-09-13T16:26:11.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b804b150 con 0x7f23b80e09e0 2024-09-13T16:26:11.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.377+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b8035e40 con 0x7f23b80e09e0 2024-09-13T16:26:11.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.381+0000 7f23abfff700 1 -- 172.21.15.119:0/2301744840 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f23ac0036a0 con 0x7f23b80e09e0 2024-09-13T16:26:11.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.381+0000 7f23abfff700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23a4066730 0x7f23a4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.381+0000 7f23abfff700 1 -- 172.21.15.119:0/2301744840 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f23ac0836b0 con 0x7f23b80e09e0 2024-09-13T16:26:11.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.381+0000 7f23bf7fe700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23a4066730 0x7f23a4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.381+0000 7f23bf7fe700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23a4066730 0x7f23a4068bf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f23b40090b0 tx=0x7f23b4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.389+0000 7f23abfff700 1 -- 172.21.15.119:0/2301744840 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f23ac050660 con 0x7f23b80e09e0 2024-09-13T16:26:11.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f23b8006ac0 con 0x7f23b80e09e0 2024-09-13T16:26:11.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23abfff700 1 -- 172.21.15.119:0/2301744840 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f23ac053c80 con 0x7f23b80e09e0 2024-09-13T16:26:11.564 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23a4066730 msgr2=0x7f23a4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23a4066730 0x7f23a4068bf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f23b40090b0 tx=0x7f23b4008030 comp rx=0 tx=0).stop 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b80e09e0 msgr2=0x7f23b8175d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b80e09e0 0x7f23b8175d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f23ac00c480 tx=0x7f23ac00c790 comp rx=0 tx=0).stop 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 shutdown_connections 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23a4066730 0x7f23a4068bf0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f23b80e0dc0 0x7f23b80deea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f23b80e09e0 0x7f23b8175d60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 --2- 172.21.15.119:0/2301744840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f23b803bfb0 0x7f23b804a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 >> 172.21.15.119:0/2301744840 conn(0x7f23b801fbc0 msgr2=0x7f23b8056550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:11.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 shutdown_connections 2024-09-13T16:26:11.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.560+0000 7f23c4e5e700 1 -- 172.21.15.119:0/2301744840 wait complete. 2024-09-13T16:26:11.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:11.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.684+0000 7f27d1288700 1 Processor -- start 2024-09-13T16:26:11.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.684+0000 7f27d1288700 1 -- start start 2024-09-13T16:26:11.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27cc06cd40 0x7f27cc0721f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc071d50 con 0x7f27cc0abf40 2024-09-13T16:26:11.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27cc06d350 con 0x7f27cc158fd0 2024-09-13T16:26:11.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27cc06d150 con 0x7f27cc06cd40 2024-09-13T16:26:11.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d0a87700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27cc06cd40 0x7f27cc0721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27cbfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f27cc0abf40 0x7f27cc0ac320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50324/0 (socket says 172.21.15.119:50324) 2024-09-13T16:26:11.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27cbfff700 1 -- 172.21.15.119:0/2339165363 learned_addr learned my addr 172.21.15.119:0/2339165363 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d0a87700 1 -- 172.21.15.119:0/2339165363 >> v1:172.21.15.145:6789/0 conn(0x7f27cc158fd0 legacy=0x7f27cc06dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d0a87700 1 -- 172.21.15.119:0/2339165363 >> v1:172.21.15.119:6789/0 conn(0x7f27cc0abf40 legacy=0x7f27cc0ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d0a87700 1 -- 172.21.15.119:0/2339165363 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27cc0acec0 con 0x7f27cc06cd40 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27cb7fe700 1 -- 172.21.15.119:0/2339165363 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35926649 0 0) 0x7f27cc071d50 con 0x7f27cc0abf40 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d0a87700 1 --2- 172.21.15.119:0/2339165363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc06cd40 0x7f27cc0721f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009d90 tx=0x7f27c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c6a9cf8c52736f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27cb7fe700 1 -- 172.21.15.119:0/2339165363 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c400e070 con 0x7f27cc06cd40 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27cb7fe700 1 -- 172.21.15.119:0/2339165363 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c400bbd0 con 0x7f27cc06cd40 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27cb7fe700 1 -- 172.21.15.119:0/2339165363 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4012430 con 0x7f27cc06cd40 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 -- 172.21.15.119:0/2339165363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc06cd40 msgr2=0x7f27cc0721f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 --2- 172.21.15.119:0/2339165363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc06cd40 0x7f27cc0721f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f27c4009d90 tx=0x7f27c400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 -- 172.21.15.119:0/2339165363 shutdown_connections 2024-09-13T16:26:11.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 --2- 172.21.15.119:0/2339165363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc06cd40 0x7f27cc0721f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.688+0000 7f27d1288700 1 -- 172.21.15.119:0/2339165363 >> 172.21.15.119:0/2339165363 conn(0x7f27cc0ae750 msgr2=0x7f27cc0b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:11.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 -- 172.21.15.119:0/2339165363 shutdown_connections 2024-09-13T16:26:11.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 -- 172.21.15.119:0/2339165363 wait complete. 2024-09-13T16:26:11.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 Processor -- start 2024-09-13T16:26:11.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 -- start start 2024-09-13T16:26:11.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc0abf40 0x7f27cc13e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27cc158fd0 0x7f27cc15bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc15a0f0 0x7f27cc141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc15bab0 con 0x7f27cc15a0f0 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc1418b0 con 0x7f27cc0abf40 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d1288700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27cc13e5a0 con 0x7f27cc158fd0 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d0a87700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc15a0f0 0x7f27cc141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d0a87700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc15a0f0 0x7f27cc141000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44722/0 (socket says 172.21.15.119:44722) 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27d0a87700 1 -- 172.21.15.119:0/474644649 learned_addr learned my addr 172.21.15.119:0/474644649 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27cbfff700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc0abf40 0x7f27cc13e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.692+0000 7f27c37fe700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27cc158fd0 0x7f27cc15bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d0a87700 1 -- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27cc158fd0 msgr2=0x7f27cc15bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d0a87700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27cc158fd0 0x7f27cc15bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d0a87700 1 -- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc0abf40 msgr2=0x7f27cc13e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d0a87700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc0abf40 0x7f27cc13e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d0a87700 1 -- 172.21.15.119:0/474644649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c4009a70 con 0x7f27cc15a0f0 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27cbfff700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc0abf40 0x7f27cc13e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27c37fe700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27cc158fd0 0x7f27cc15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d0a87700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc15a0f0 0x7f27cc141000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f27c4014040 tx=0x7f27c4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27c97fa700 1 -- 172.21.15.119:0/474644649 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4012430 con 0x7f27cc15a0f0 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27c97fa700 1 -- 172.21.15.119:0/474644649 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27c4020070 con 0x7f27cc15a0f0 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27cc15c410 con 0x7f27cc15a0f0 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27c97fa700 1 -- 172.21.15.119:0/474644649 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c4004280 con 0x7f27cc15a0f0 2024-09-13T16:26:11.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27cc15b3f0 con 0x7f27cc15a0f0 2024-09-13T16:26:11.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27cc0662c0 con 0x7f27cc15a0f0 2024-09-13T16:26:11.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27c97fa700 1 -- 172.21.15.119:0/474644649 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f27c40043e0 con 0x7f27cc15a0f0 2024-09-13T16:26:11.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27c97fa700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27ac066730 0x7f27ac068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:11.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.696+0000 7f27c97fa700 1 -- 172.21.15.119:0/474644649 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f27c4022070 con 0x7f27cc15a0f0 2024-09-13T16:26:11.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.700+0000 7f27cbfff700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27ac066730 0x7f27ac068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:11.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.700+0000 7f27cbfff700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27ac066730 0x7f27ac068bf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f27bc000f60 tx=0x7f27bc007d00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:11.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.704+0000 7f27c97fa700 1 -- 172.21.15.119:0/474644649 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f27c4057250 con 0x7f27cc15a0f0 2024-09-13T16:26:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f27cc0721f0 con 0x7f27cc15a0f0 2024-09-13T16:26:11.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27c97fa700 1 -- 172.21.15.119:0/474644649 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f27c401ae60 con 0x7f27cc15a0f0 2024-09-13T16:26:11.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27ac066730 msgr2=0x7f27ac068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27ac066730 0x7f27ac068bf0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f27bc000f60 tx=0x7f27bc007d00 comp rx=0 tx=0).stop 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc15a0f0 msgr2=0x7f27cc141000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc15a0f0 0x7f27cc141000 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f27c4014040 tx=0x7f27c4004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 shutdown_connections 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27ac066730 0x7f27ac068bf0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27cc15a0f0 0x7f27cc141000 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27cc158fd0 0x7f27cc15bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 --2- 172.21.15.119:0/474644649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27cc0abf40 0x7f27cc13e060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 >> 172.21.15.119:0/474644649 conn(0x7f27cc06cd40 msgr2=0x7f27cc155ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 shutdown_connections 2024-09-13T16:26:11.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:11.888+0000 7f27d1288700 1 -- 172.21.15.119:0/474644649 wait complete. 2024-09-13T16:26:11.896 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:11.907 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:26:11.908 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:11.916 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:12.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a03fff700 1 Processor -- start 2024-09-13T16:26:12.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a03fff700 1 -- start start 2024-09-13T16:26:12.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a03fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a040ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a03fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3a0408c6b0 0x7f3a04166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a03fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a0415d020 con 0x7f3a040cd860 2024-09-13T16:26:12.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a03fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a040b44c0 con 0x7f3a0408c6b0 2024-09-13T16:26:12.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a03fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040d0720 con 0x7f3a0415cc40 2024-09-13T16:26:12.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a037fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3a0408c6b0 0x7f3a04166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a02ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a040ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f39fbfff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f3a040cd860 0x7f3a0408bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:55084/0 (socket says 172.21.15.119:55084) 2024-09-13T16:26:12.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f39fbfff700 1 -- 172.21.15.119:0/1681921378 learned_addr learned my addr 172.21.15.119:0/1681921378 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a037fe700 1 -- 172.21.15.119:0/1681921378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 msgr2=0x7f3a040ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a037fe700 1 --2- 172.21.15.119:0/1681921378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a040ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a037fe700 1 -- 172.21.15.119:0/1681921378 >> v1:172.21.15.145:6789/0 conn(0x7f3a040cd860 legacy=0x7f3a0408bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a037fe700 1 -- 172.21.15.119:0/1681921378 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a040d09c0 con 0x7f3a0408c6b0 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a037fe700 1 --2- 172.21.15.119:0/1681921378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 0x7f3a04166a50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009d90 tx=0x7f39fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=104e0747a448038b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a027fc700 1 -- 172.21.15.119:0/1681921378 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc00e070 con 0x7f3a0408c6b0 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a027fc700 1 -- 172.21.15.119:0/1681921378 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39fc00bb20 con 0x7f3a0408c6b0 2024-09-13T16:26:12.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.028+0000 7f3a027fc700 1 -- 172.21.15.119:0/1681921378 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39fc012430 con 0x7f3a0408c6b0 2024-09-13T16:26:12.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 -- 172.21.15.119:0/1681921378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 msgr2=0x7f3a04166a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 --2- 172.21.15.119:0/1681921378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 0x7f3a04166a50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f39fc009d90 tx=0x7f39fc00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:12.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 -- 172.21.15.119:0/1681921378 shutdown_connections 2024-09-13T16:26:12.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 --2- 172.21.15.119:0/1681921378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 0x7f3a04166a50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 --2- 172.21.15.119:0/1681921378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a040ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 -- 172.21.15.119:0/1681921378 >> 172.21.15.119:0/1681921378 conn(0x7f3a040b4640 msgr2=0x7f3a040b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:12.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 -- 172.21.15.119:0/1681921378 shutdown_connections 2024-09-13T16:26:12.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 -- 172.21.15.119:0/1681921378 wait complete. 2024-09-13T16:26:12.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 Processor -- start 2024-09-13T16:26:12.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 -- start start 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 0x7f3a040b6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a040cd860 0x7f3a0415a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.032+0000 7f3a03fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a0415a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f3a03fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040cb2c0 con 0x7f3a040cd860 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f3a03fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a041f33a0 con 0x7f3a0408c6b0 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f3a03fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a041f2790 con 0x7f3a0415cc40 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a040cd860 0x7f3a0415a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a040cd860 0x7f3a0415a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44734/0 (socket says 172.21.15.119:44734) 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 -- 172.21.15.119:0/3094882612 learned_addr learned my addr 172.21.15.119:0/3094882612 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f3a037fe700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a0415a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 -- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 msgr2=0x7f3a0415a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a0415a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 -- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 msgr2=0x7f3a040b6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 0x7f3a040b6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 -- 172.21.15.119:0/3094882612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc009a70 con 0x7f3a040cd860 2024-09-13T16:26:12.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fbfff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a040cd860 0x7f3a0415a020 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00c430 tx=0x7f39ec00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fb7fe700 1 -- 172.21.15.119:0/3094882612 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec005d90 con 0x7f3a040cd860 2024-09-13T16:26:12.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fb7fe700 1 -- 172.21.15.119:0/3094882612 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ec0070d0 con 0x7f3a040cd860 2024-09-13T16:26:12.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a041f2a70 con 0x7f3a040cd860 2024-09-13T16:26:12.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f39fb7fe700 1 -- 172.21.15.119:0/3094882612 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ec007460 con 0x7f3a040cd860 2024-09-13T16:26:12.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.036+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a041f3030 con 0x7f3a040cd860 2024-09-13T16:26:12.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.040+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a040d1a60 con 0x7f3a040cd860 2024-09-13T16:26:12.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.040+0000 7f39fb7fe700 1 -- 172.21.15.119:0/3094882612 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f39ec0075c0 con 0x7f3a040cd860 2024-09-13T16:26:12.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.040+0000 7f39fb7fe700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39e4066730 0x7f39e4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.040+0000 7f39fb7fe700 1 -- 172.21.15.119:0/3094882612 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f39ec0838c0 con 0x7f3a040cd860 2024-09-13T16:26:12.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.040+0000 7f3a02ffd700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39e4066730 0x7f39e4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.040+0000 7f3a02ffd700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39e4066730 0x7f39e4068bf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39f4006b40 tx=0x7f39f4006ad0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.048+0000 7f39fb7fe700 1 -- 172.21.15.119:0/3094882612 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f39ec050870 con 0x7f3a040cd860 2024-09-13T16:26:12.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.216+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a041f34e0 con 0x7f3a040cd860 2024-09-13T16:26:12.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.220+0000 7f39fb7fe700 1 -- 172.21.15.119:0/3094882612 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f39ec017090 con 0x7f3a040cd860 2024-09-13T16:26:12.225 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:12.225 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:12.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39e4066730 msgr2=0x7f39e4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39e4066730 0x7f39e4068bf0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f39f4006b40 tx=0x7f39f4006ad0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a040cd860 msgr2=0x7f3a0415a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a040cd860 0x7f3a0415a020 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f39ec00c430 tx=0x7f39ec00c740 comp rx=0 tx=0).stop 2024-09-13T16:26:12.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 shutdown_connections 2024-09-13T16:26:12.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39e4066730 0x7f39e4068bf0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3a0415cc40 0x7f3a0415a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3a040cd860 0x7f3a0415a020 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 --2- 172.21.15.119:0/3094882612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3a0408c6b0 0x7f3a040b6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 >> 172.21.15.119:0/3094882612 conn(0x7f3a040b4640 msgr2=0x7f3a0415bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:12.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 shutdown_connections 2024-09-13T16:26:12.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.224+0000 7f3a03fff700 1 -- 172.21.15.119:0/3094882612 wait complete. 2024-09-13T16:26:12.246 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:12.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:12.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.356+0000 7fdb03fff700 1 Processor -- start 2024-09-13T16:26:12.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.356+0000 7fdb03fff700 1 -- start start 2024-09-13T16:26:12.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.356+0000 7fdb03fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdb04144180 0x7fdb0406c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.356+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040cc010 0x7fdb040cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.356+0000 7fdb03fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb0406cb40 con 0x7fdb040ce000 2024-09-13T16:26:12.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.356+0000 7fdb03fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb040b1c50 con 0x7fdb04144180 2024-09-13T16:26:12.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.356+0000 7fdb03fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0406ce20 con 0x7fdb040cc010 2024-09-13T16:26:12.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb037fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fdb040ce000 0x7fdb0415e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50328/0 (socket says 172.21.15.119:50328) 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb037fe700 1 -- 172.21.15.119:0/3581773051 learned_addr learned my addr 172.21.15.119:0/3581773051 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb027fc700 1 --2- 172.21.15.119:0/3581773051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040cc010 0x7fdb040cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb02ffd700 1 --2- 172.21.15.119:0/3581773051 >> v2:172.21.15.145:3300/0 conn(0x7fdb04144180 0x7fdb0406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb01ffb700 1 -- 172.21.15.119:0/3581773051 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525849289 0 0) 0x7fdb0406cb40 con 0x7fdb040ce000 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb01ffb700 1 -- 172.21.15.119:0/3581773051 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaec0035d0 con 0x7fdb040ce000 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb027fc700 1 -- 172.21.15.119:0/3581773051 >> v2:172.21.15.145:3300/0 conn(0x7fdb04144180 msgr2=0x7fdb0406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb027fc700 1 --2- 172.21.15.119:0/3581773051 >> v2:172.21.15.145:3300/0 conn(0x7fdb04144180 0x7fdb0406c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb027fc700 1 -- 172.21.15.119:0/3581773051 >> v1:172.21.15.119:6789/0 conn(0x7fdb040ce000 legacy=0x7fdb0415e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb027fc700 1 -- 172.21.15.119:0/3581773051 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0406d100 con 0x7fdb040cc010 2024-09-13T16:26:12.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb027fc700 1 --2- 172.21.15.119:0/3581773051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040cc010 0x7fdb040cdac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7927fc72fbbf96c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb01ffb700 1 -- 172.21.15.119:0/3581773051 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf400e070 con 0x7fdb040cc010 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb01ffb700 1 -- 172.21.15.119:0/3581773051 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf400bb20 con 0x7fdb040cc010 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb01ffb700 1 -- 172.21.15.119:0/3581773051 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf400be90 con 0x7fdb040cc010 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 -- 172.21.15.119:0/3581773051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040cc010 msgr2=0x7fdb040cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 --2- 172.21.15.119:0/3581773051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040cc010 0x7fdb040cdac0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 -- 172.21.15.119:0/3581773051 shutdown_connections 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 --2- 172.21.15.119:0/3581773051 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040cc010 0x7fdb040cdac0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 --2- 172.21.15.119:0/3581773051 >> v2:172.21.15.145:3300/0 conn(0x7fdb04144180 0x7fdb0406c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 -- 172.21.15.119:0/3581773051 >> 172.21.15.119:0/3581773051 conn(0x7fdb040b0980 msgr2=0x7fdb040b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:12.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 -- 172.21.15.119:0/3581773051 shutdown_connections 2024-09-13T16:26:12.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 -- 172.21.15.119:0/3581773051 wait complete. 2024-09-13T16:26:12.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 Processor -- start 2024-09-13T16:26:12.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.360+0000 7fdb03fff700 1 -- start start 2024-09-13T16:26:12.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb040cc010 0x7fdb0413cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040ce000 0x7fdb040a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb04144180 0x7fdb0414cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0414af50 con 0x7fdb040cc010 2024-09-13T16:26:12.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0414bad0 con 0x7fdb04144180 2024-09-13T16:26:12.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb0413d240 con 0x7fdb040ce000 2024-09-13T16:26:12.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb040cc010 0x7fdb0413cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb040cc010 0x7fdb0413cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44740/0 (socket says 172.21.15.119:44740) 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 -- 172.21.15.119:0/1480166067 learned_addr learned my addr 172.21.15.119:0/1480166067 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb037fe700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb04144180 0x7fdb0414cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb027fc700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040ce000 0x7fdb040a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 -- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040ce000 msgr2=0x7fdb040a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040ce000 0x7fdb040a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 -- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb04144180 msgr2=0x7fdb0414cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb04144180 0x7fdb0414cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 -- 172.21.15.119:0/1480166067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf4009a70 con 0x7fdb040cc010 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb037fe700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb04144180 0x7fdb0414cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb027fc700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040ce000 0x7fdb040a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb02ffd700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb040cc010 0x7fdb0413cd00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800e870 tx=0x7fdaf800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdaeb7fe700 1 -- 172.21.15.119:0/1480166067 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf80095a0 con 0x7fdb040cc010 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb0414b1a0 con 0x7fdb040cc010 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdaeb7fe700 1 -- 172.21.15.119:0/1480166067 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaf800ee70 con 0x7fdb040cc010 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb0414b760 con 0x7fdb040cc010 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdaeb7fe700 1 -- 172.21.15.119:0/1480166067 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdaf800ca50 con 0x7fdb040cc010 2024-09-13T16:26:12.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb040cdac0 con 0x7fdb040cc010 2024-09-13T16:26:12.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.364+0000 7fdaeb7fe700 1 -- 172.21.15.119:0/1480166067 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fdaf800cbb0 con 0x7fdb040cc010 2024-09-13T16:26:12.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.368+0000 7fdaeb7fe700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdaf0066730 0x7fdaf0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.368+0000 7fdaeb7fe700 1 -- 172.21.15.119:0/1480166067 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdaf8084c90 con 0x7fdb040cc010 2024-09-13T16:26:12.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.368+0000 7fdb027fc700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdaf0066730 0x7fdaf0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.368+0000 7fdb027fc700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdaf0066730 0x7fdaf0068bf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf4011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.372+0000 7fdaeb7fe700 1 -- 172.21.15.119:0/1480166067 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdaf8051cd0 con 0x7fdb040cc010 2024-09-13T16:26:12.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.540+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdb0415c720 con 0x7fdb040cc010 2024-09-13T16:26:12.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.540+0000 7fdaeb7fe700 1 -- 172.21.15.119:0/1480166067 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdaf80170a0 con 0x7fdb040cc010 2024-09-13T16:26:12.546 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:12.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdaf0066730 msgr2=0x7fdaf0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdaf0066730 0x7fdaf0068bf0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdaf4009d90 tx=0x7fdaf4011000 comp rx=0 tx=0).stop 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb040cc010 msgr2=0x7fdb0413cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb040cc010 0x7fdb0413cd00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fdaf800e870 tx=0x7fdaf800ec30 comp rx=0 tx=0).stop 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 shutdown_connections 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdaf0066730 0x7fdaf0068bf0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdb04144180 0x7fdb0414cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdb040ce000 0x7fdb040a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 --2- 172.21.15.119:0/1480166067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdb040cc010 0x7fdb0413cd00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 >> 172.21.15.119:0/1480166067 conn(0x7fdb040b0980 msgr2=0x7fdb040af920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 shutdown_connections 2024-09-13T16:26:12.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.544+0000 7fdb03fff700 1 -- 172.21.15.119:0/1480166067 wait complete. 2024-09-13T16:26:12.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:12.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.680+0000 7f899eec1700 1 Processor -- start 2024-09-13T16:26:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.680+0000 7f899eec1700 1 -- start start 2024-09-13T16:26:12.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f89980b3100 0x7f89980cff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980cef70 con 0x7f89980b3ff0 2024-09-13T16:26:12.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f899815d1d0 con 0x7f899815cdf0 2024-09-13T16:26:12.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89980cf1c0 con 0x7f89980b3100 2024-09-13T16:26:12.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899e6c0700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f89980b3ff0 0x7f89980c5850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59366/0 (socket says 172.21.15.119:59366) 2024-09-13T16:26:12.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899e6c0700 1 -- 172.21.15.119:0/3102423049 learned_addr learned my addr 172.21.15.119:0/3102423049 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:12.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899d6be700 1 --2- 172.21.15.119:0/3102423049 >> v2:172.21.15.145:3300/0 conn(0x7f89980b3100 0x7f89980cff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899cebd700 1 -- 172.21.15.119:0/3102423049 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771548963 0 0) 0x7f899815d1d0 con 0x7f899815cdf0 2024-09-13T16:26:12.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899cebd700 1 -- 172.21.15.119:0/3102423049 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89840035d0 con 0x7f899815cdf0 2024-09-13T16:26:12.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899cebd700 1 -- 172.21.15.119:0/3102423049 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2461153068 0 0) 0x7f89980cef70 con 0x7f89980b3ff0 2024-09-13T16:26:12.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899cebd700 1 -- 172.21.15.119:0/3102423049 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f899815d1d0 con 0x7f89980b3ff0 2024-09-13T16:26:12.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899d6be700 1 -- 172.21.15.119:0/3102423049 >> v1:172.21.15.145:6789/0 conn(0x7f899815cdf0 legacy=0x7f89980b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899d6be700 1 -- 172.21.15.119:0/3102423049 >> v1:172.21.15.119:6789/0 conn(0x7f89980b3ff0 legacy=0x7f89980c5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899d6be700 1 -- 172.21.15.119:0/3102423049 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89980cf4a0 con 0x7f89980b3100 2024-09-13T16:26:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899d6be700 1 --2- 172.21.15.119:0/3102423049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 0x7f89980cff80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8988009850 tx=0x7f8988009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=559a5613b8aa7c30 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899cebd700 1 -- 172.21.15.119:0/3102423049 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988011830 con 0x7f89980b3100 2024-09-13T16:26:12.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899cebd700 1 -- 172.21.15.119:0/3102423049 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8988011990 con 0x7f89980b3100 2024-09-13T16:26:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899cebd700 1 -- 172.21.15.119:0/3102423049 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8988011d40 con 0x7f89980b3100 2024-09-13T16:26:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- 172.21.15.119:0/3102423049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 msgr2=0x7f89980cff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 --2- 172.21.15.119:0/3102423049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 0x7f89980cff80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8988009850 tx=0x7f8988009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- 172.21.15.119:0/3102423049 shutdown_connections 2024-09-13T16:26:12.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 --2- 172.21.15.119:0/3102423049 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 0x7f89980cff80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- 172.21.15.119:0/3102423049 >> 172.21.15.119:0/3102423049 conn(0x7f89980b25e0 msgr2=0x7f89980b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- 172.21.15.119:0/3102423049 shutdown_connections 2024-09-13T16:26:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.684+0000 7f899eec1700 1 -- 172.21.15.119:0/3102423049 wait complete. 2024-09-13T16:26:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 Processor -- start 2024-09-13T16:26:12.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 -- start start 2024-09-13T16:26:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 0x7f899814c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89980b3ff0 0x7f89980c7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f899815cdf0 0x7f89980c7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980cc490 con 0x7f899815cdf0 2024-09-13T16:26:12.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89980cbb60 con 0x7f89980b3100 2024-09-13T16:26:12.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89980cbe40 con 0x7f89980b3ff0 2024-09-13T16:26:12.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f899815cdf0 0x7f89980c7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f899815cdf0 0x7f89980c7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41730/0 (socket says 172.21.15.119:41730) 2024-09-13T16:26:12.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 -- 172.21.15.119:0/458023503 learned_addr learned my addr 172.21.15.119:0/458023503 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:12.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899debf700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 0x7f899814c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899d6be700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89980b3ff0 0x7f89980c7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 -- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89980b3ff0 msgr2=0x7f89980c7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89980b3ff0 0x7f89980c7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 -- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 msgr2=0x7f899814c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 0x7f899814c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 -- 172.21.15.119:0/458023503 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89880094b0 con 0x7f899815cdf0 2024-09-13T16:26:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899d6be700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89980b3ff0 0x7f89980c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:12.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899e6c0700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f899815cdf0 0x7f89980c7fa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f899000caa0 tx=0x7f899000cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f898e7fc700 1 -- 172.21.15.119:0/458023503 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990007480 con 0x7f899815cdf0 2024-09-13T16:26:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f898e7fc700 1 -- 172.21.15.119:0/458023503 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f89900075e0 con 0x7f899815cdf0 2024-09-13T16:26:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89980cc120 con 0x7f899815cdf0 2024-09-13T16:26:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f898e7fc700 1 -- 172.21.15.119:0/458023503 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8990007970 con 0x7f899815cdf0 2024-09-13T16:26:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.688+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f899814cb20 con 0x7f899815cdf0 2024-09-13T16:26:12.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.692+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89980cc600 con 0x7f899815cdf0 2024-09-13T16:26:12.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.692+0000 7f898e7fc700 1 -- 172.21.15.119:0/458023503 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f899001b3b0 con 0x7f899815cdf0 2024-09-13T16:26:12.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.692+0000 7f898e7fc700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f897c066730 0x7f897c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:12.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.692+0000 7f898e7fc700 1 -- 172.21.15.119:0/458023503 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8990083ec0 con 0x7f899815cdf0 2024-09-13T16:26:12.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.692+0000 7f899debf700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f897c066730 0x7f897c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:12.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.692+0000 7f899debf700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f897c066730 0x7f897c068bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f89940011e0 tx=0x7f8994009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:12.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.700+0000 7f898e7fc700 1 -- 172.21.15.119:0/458023503 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8990050e70 con 0x7f899815cdf0 2024-09-13T16:26:12.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.860+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89980cff80 con 0x7f899815cdf0 2024-09-13T16:26:12.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f898e7fc700 1 -- 172.21.15.119:0/458023503 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89900170a0 con 0x7f899815cdf0 2024-09-13T16:26:12.868 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:12.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f897c066730 msgr2=0x7f897c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f897c066730 0x7f897c068bf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f89940011e0 tx=0x7f8994009040 comp rx=0 tx=0).stop 2024-09-13T16:26:12.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f899815cdf0 msgr2=0x7f89980c7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f899815cdf0 0x7f89980c7fa0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f899000caa0 tx=0x7f899000cad0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 shutdown_connections 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f897c066730 0x7f897c068bf0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f899815cdf0 0x7f89980c7fa0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89980b3ff0 0x7f89980c7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 --2- 172.21.15.119:0/458023503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89980b3100 0x7f899814c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 >> 172.21.15.119:0/458023503 conn(0x7f89980b25e0 msgr2=0x7f89980b1de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 shutdown_connections 2024-09-13T16:26:12.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:12.864+0000 7f899eec1700 1 -- 172.21.15.119:0/458023503 wait complete. 2024-09-13T16:26:12.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:13.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.000+0000 7ff009e53700 1 Processor -- start 2024-09-13T16:26:13.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.000+0000 7ff009e53700 1 -- start start 2024-09-13T16:26:13.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415a7a0 0x7ff00406a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff0040723c0 0x7ff0040ac950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff00415e320 0x7ff00415e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff00415ed10 con 0x7ff00415e320 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0040ad2c0 con 0x7ff0040723c0 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00406c7a0 con 0x7ff00415a7a0 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff00415e320 0x7ff00415e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff00415e320 0x7ff00415e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41742/0 (socket says 172.21.15.119:41742) 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 -- 172.21.15.119:0/3315246486 learned_addr learned my addr 172.21.15.119:0/3315246486 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff008e51700 1 --2- 172.21.15.119:0/3315246486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415a7a0 0x7ff00406a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7feffbfff700 1 --2- 172.21.15.119:0/3315246486 >> v2:172.21.15.145:3300/0 conn(0x7ff0040723c0 0x7ff0040ac950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 -- 172.21.15.119:0/3315246486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415a7a0 msgr2=0x7ff00406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 --2- 172.21.15.119:0/3315246486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415a7a0 0x7ff00406a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 -- 172.21.15.119:0/3315246486 >> v2:172.21.15.145:3300/0 conn(0x7ff0040723c0 msgr2=0x7ff0040ac950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:13.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 --2- 172.21.15.119:0/3315246486 >> v2:172.21.15.145:3300/0 conn(0x7ff0040723c0 0x7ff0040ac950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 -- 172.21.15.119:0/3315246486 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00406ca80 con 0x7ff00415e320 2024-09-13T16:26:13.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009652700 1 --2- 172.21.15.119:0/3315246486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415e320 0x7ff00415e7d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff00406eef0 tx=0x7feffc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be776734a856c4e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7feffb7fe700 1 -- 172.21.15.119:0/3315246486 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc00e070 con 0x7ff00415e320 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7feffb7fe700 1 -- 172.21.15.119:0/3315246486 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feffc00b910 con 0x7ff00415e320 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7feffb7fe700 1 -- 172.21.15.119:0/3315246486 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feffc00bcc0 con 0x7ff00415e320 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- 172.21.15.119:0/3315246486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415e320 msgr2=0x7ff00415e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 --2- 172.21.15.119:0/3315246486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415e320 0x7ff00415e7d0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff00406eef0 tx=0x7feffc00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- 172.21.15.119:0/3315246486 shutdown_connections 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 --2- 172.21.15.119:0/3315246486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415e320 0x7ff00415e7d0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 --2- 172.21.15.119:0/3315246486 >> v2:172.21.15.145:3300/0 conn(0x7ff0040723c0 0x7ff0040ac950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 --2- 172.21.15.119:0/3315246486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415a7a0 0x7ff00406a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- 172.21.15.119:0/3315246486 >> 172.21.15.119:0/3315246486 conn(0x7ff0040b0cd0 msgr2=0x7ff0040b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:13.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- 172.21.15.119:0/3315246486 shutdown_connections 2024-09-13T16:26:13.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.004+0000 7ff009e53700 1 -- 172.21.15.119:0/3315246486 wait complete. 2024-09-13T16:26:13.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 Processor -- start 2024-09-13T16:26:13.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 -- start start 2024-09-13T16:26:13.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0040723c0 0x7ff00415cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415a7a0 0x7ff004152710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415e320 0x7ff00414fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00415c880 con 0x7ff00415a7a0 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff00415b990 con 0x7ff0040723c0 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff009e53700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff00415bc70 con 0x7ff00415e320 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7feffbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415a7a0 0x7ff004152710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7feffbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415a7a0 0x7ff004152710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41746/0 (socket says 172.21.15.119:41746) 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7feffbfff700 1 -- 172.21.15.119:0/4028468614 learned_addr learned my addr 172.21.15.119:0/4028468614 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.008+0000 7ff008e51700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0040723c0 0x7ff00415cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7ff009652700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415e320 0x7ff00414fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feffbfff700 1 -- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415e320 msgr2=0x7ff00414fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feffbfff700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415e320 0x7ff00414fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feffbfff700 1 -- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0040723c0 msgr2=0x7ff00415cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feffbfff700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0040723c0 0x7ff00415cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feffbfff700 1 -- 172.21.15.119:0/4028468614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feffc009b30 con 0x7ff00415a7a0 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feffbfff700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415a7a0 0x7ff004152710 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feff400e8e0 tx=0x7feff400eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feff97fa700 1 -- 172.21.15.119:0/4028468614 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff400eee0 con 0x7ff00415a7a0 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feff97fa700 1 -- 172.21.15.119:0/4028468614 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff400c850 con 0x7ff00415a7a0 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff00415bf50 con 0x7ff00415a7a0 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feff97fa700 1 -- 172.21.15.119:0/4028468614 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feff400cbc0 con 0x7ff00415a7a0 2024-09-13T16:26:13.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff00415c510 con 0x7ff00415a7a0 2024-09-13T16:26:13.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0040742c0 con 0x7ff00415a7a0 2024-09-13T16:26:13.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feff97fa700 1 -- 172.21.15.119:0/4028468614 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7feff400cd20 con 0x7ff00415a7a0 2024-09-13T16:26:13.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feff97fa700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff0066730 0x7feff0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.012+0000 7feff97fa700 1 -- 172.21.15.119:0/4028468614 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7feff4084540 con 0x7ff00415a7a0 2024-09-13T16:26:13.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.016+0000 7ff008e51700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff0066730 0x7feff0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.016+0000 7ff008e51700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff0066730 0x7feff0068bf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff000009b80 tx=0x7ff000008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.024+0000 7feff97fa700 1 -- 172.21.15.119:0/4028468614 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7feff40514f0 con 0x7ff00415a7a0 2024-09-13T16:26:13.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.192+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff00406a350 con 0x7ff00415a7a0 2024-09-13T16:26:13.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.192+0000 7feff97fa700 1 -- 172.21.15.119:0/4028468614 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7feff4054b10 con 0x7ff00415a7a0 2024-09-13T16:26:13.197 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:13.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff0066730 msgr2=0x7feff0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff0066730 0x7feff0068bf0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff000009b80 tx=0x7ff000008040 comp rx=0 tx=0).stop 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415a7a0 msgr2=0x7ff004152710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415a7a0 0x7ff004152710 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7feff400e8e0 tx=0x7feff400eca0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 shutdown_connections 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff0066730 0x7feff0068bf0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff00415e320 0x7ff00414fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff00415a7a0 0x7ff004152710 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 --2- 172.21.15.119:0/4028468614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0040723c0 0x7ff00415cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 >> 172.21.15.119:0/4028468614 conn(0x7ff0040b0cd0 msgr2=0x7ff0040b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 shutdown_connections 2024-09-13T16:26:13.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.196+0000 7ff009e53700 1 -- 172.21.15.119:0/4028468614 wait complete. 2024-09-13T16:26:13.212 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca1503700 1 Processor -- start 2024-09-13T16:26:13.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca1503700 1 -- start start 2024-09-13T16:26:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca1503700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1c9c0ade80 0x7f1c9c1652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca1503700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 0x7f1c9c0b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca1503700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c9c071ac0 con 0x7f1c9c1612d0 2024-09-13T16:26:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca1503700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9c1657f0 con 0x7f1c9c0ade80 2024-09-13T16:26:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca1503700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0b2e20 con 0x7f1c9c0b24a0 2024-09-13T16:26:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca0d02700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1c9c1612d0 0x7f1c9c161780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59368/0 (socket says 172.21.15.119:59368) 2024-09-13T16:26:13.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1ca0d02700 1 -- 172.21.15.119:0/1758976223 learned_addr learned my addr 172.21.15.119:0/1758976223 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1c9bfff700 1 --2- 172.21.15.119:0/1758976223 >> v2:172.21.15.145:3300/0 conn(0x7f1c9c0ade80 0x7f1c9c1652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1c9b7fe700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 0x7f1c9c0b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461016549 0 0) 0x7f1c9c071ac0 con 0x7f1c9c1612d0 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.328+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c880035d0 con 0x7f1c9c1612d0 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1913838059 0 0) 0x7f1c880035d0 con 0x7f1c9c1612d0 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 msgr2=0x7f1c9c0b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 0x7f1c9c0b28e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 >> v2:172.21.15.145:3300/0 conn(0x7f1c9c0ade80 msgr2=0x7f1c9c1652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 --2- 172.21.15.119:0/1758976223 >> v2:172.21.15.145:3300/0 conn(0x7f1c9c0ade80 0x7f1c9c1652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c070eb0 con 0x7f1c9c1612d0 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f1c940019f0 con 0x7f1c9c1612d0 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 >> v1:172.21.15.119:6789/0 conn(0x7f1c9c1612d0 legacy=0x7f1c9c161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88003a20 0x7f1c88003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 0x7f1c88004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88004db0 0x7f1c88005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c1657f0 con 0x7f1c88004db0 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0b2e20 con 0x7f1c88003a20 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8800ba30 con 0x7f1c9c0b24a0 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88004db0 0x7f1c88005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9bfff700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88003a20 0x7f1c88003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9b7fe700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 0x7f1c88004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 -- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 msgr2=0x7f1c88004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:13.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c0b24a0 0x7f1c88004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 -- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88003a20 msgr2=0x7f1c88003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88003a20 0x7f1c88003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 -- 172.21.15.119:0/1758976223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c880035d0 con 0x7f1c88004db0 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88004db0 0x7f1c88005230 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400a860 tx=0x7f1c880058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0c87d97523be8e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940072d0 con 0x7f1c88004db0 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c94007430 con 0x7f1c88004db0 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1c9affd700 1 -- 172.21.15.119:0/1758976223 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c940077e0 con 0x7f1c88004db0 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca1503700 1 -- 172.21.15.119:0/1758976223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88004db0 msgr2=0x7f1c88005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca1503700 1 --2- 172.21.15.119:0/1758976223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c88004db0 0x7f1c88005230 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f1c9400a860 tx=0x7f1c880058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca0d02700 1 -- 172.21.15.119:0/1758976223 reap_dead start 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca1503700 1 -- 172.21.15.119:0/1758976223 shutdown_connections 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca1503700 1 -- 172.21.15.119:0/1758976223 >> 172.21.15.119:0/1758976223 conn(0x7f1c9c0b0980 msgr2=0x7f1c9c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:13.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca1503700 1 -- 172.21.15.119:0/1758976223 shutdown_connections 2024-09-13T16:26:13.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.332+0000 7f1ca1503700 1 -- 172.21.15.119:0/1758976223 wait complete. 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 Processor -- start 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 -- start start 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88004db0 0x7f1c9c14ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c1526e0 0x7f1c9c14bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c9c0aade0 0x7f1c9c0ab290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0ab7d0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c14f3b0 con 0x7f1c88004db0 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca1503700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9c0ad190 con 0x7f1c9c1526e0 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c9c0aade0 0x7f1c9c0ab290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c9c0aade0 0x7f1c9c0ab290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41764/0 (socket says 172.21.15.119:41764) 2024-09-13T16:26:13.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 -- 172.21.15.119:0/910705652 learned_addr learned my addr 172.21.15.119:0/910705652 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:13.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1c9bfff700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88004db0 0x7f1c9c14ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1c9b7fe700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c1526e0 0x7f1c9c14bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 -- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c1526e0 msgr2=0x7f1c9c14bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c1526e0 0x7f1c9c14bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 -- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88004db0 msgr2=0x7f1c9c14ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88004db0 0x7f1c9c14ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 -- 172.21.15.119:0/910705652 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9c0cb9c0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1ca0d02700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c9c0aade0 0x7f1c9c0ab290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011000 tx=0x7f1c94001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1c98ff9700 1 -- 172.21.15.119:0/910705652 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94025070 con 0x7f1c9c0aade0 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.336+0000 7f1c98ff9700 1 -- 172.21.15.119:0/910705652 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c940019c0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9c0cbca0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1c98ff9700 1 -- 172.21.15.119:0/910705652 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c94007bc0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c9c14c050 con 0x7f1c9c0aade0 2024-09-13T16:26:13.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c9c0cc1c0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1c98ff9700 1 -- 172.21.15.119:0/910705652 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1c9401a070 con 0x7f1c9c0aade0 2024-09-13T16:26:13.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1c98ff9700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c7c066730 0x7f1c7c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1c98ff9700 1 -- 172.21.15.119:0/910705652 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1c940858f0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.340+0000 7f1c9bfff700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c7c066730 0x7f1c7c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.344+0000 7f1c9bfff700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c7c066730 0x7f1c7c068bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009c70 tx=0x7f1c84009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.348+0000 7f1c98ff9700 1 -- 172.21.15.119:0/910705652 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1c94052b10 con 0x7f1c9c0aade0 2024-09-13T16:26:13.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.516+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f1c9c0b11a0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.516+0000 7f1c98ff9700 1 -- 172.21.15.119:0/910705652 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f1c940180a0 con 0x7f1c9c0aade0 2024-09-13T16:26:13.522 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:13.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c7c066730 msgr2=0x7f1c7c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c7c066730 0x7f1c7c068bf0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1c84009c70 tx=0x7f1c84009240 comp rx=0 tx=0).stop 2024-09-13T16:26:13.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c9c0aade0 msgr2=0x7f1c9c0ab290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c9c0aade0 0x7f1c9c0ab290 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1c94011000 tx=0x7f1c94001d80 comp rx=0 tx=0).stop 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 shutdown_connections 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c7c066730 0x7f1c7c068bf0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c9c0aade0 0x7f1c9c0ab290 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c9c1526e0 0x7f1c9c14bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 --2- 172.21.15.119:0/910705652 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c88004db0 0x7f1c9c14ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 >> 172.21.15.119:0/910705652 conn(0x7f1c9c0b0980 msgr2=0x7f1c9c161810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 shutdown_connections 2024-09-13T16:26:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.520+0000 7f1ca1503700 1 -- 172.21.15.119:0/910705652 wait complete. 2024-09-13T16:26:13.542 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:13.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.648+0000 7f0f8f2a4700 1 Processor -- start 2024-09-13T16:26:13.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.648+0000 7f0f8f2a4700 1 -- start start 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f880cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0f880b2200 0x7f0f880b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f880705d0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f880b2bf0 con 0x7f0f880b2200 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8806fca0 con 0x7f0f880b17f0 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0f880b2200 0x7f0f880b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0f880b2200 0x7f0f880b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41774/0 (socket says 172.21.15.119:41774) 2024-09-13T16:26:13.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 -- 172.21.15.119:0/3634866279 learned_addr learned my addr 172.21.15.119:0/3634866279 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:13.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8daa1700 1 --2- 172.21.15.119:0/3634866279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f880cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 -- 172.21.15.119:0/3634866279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 msgr2=0x7f0f880cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:13.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 --2- 172.21.15.119:0/3634866279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f880cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 -- 172.21.15.119:0/3634866279 >> v1:172.21.15.145:6789/0 conn(0x7f0f880cd8d0 legacy=0x7f0f8806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 -- 172.21.15.119:0/3634866279 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f8806ff80 con 0x7f0f880b2200 2024-09-13T16:26:13.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8eaa3700 1 --2- 172.21.15.119:0/3634866279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880b2200 0x7f0f880b26b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0f80009910 tx=0x7f0f80009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906f8a7ad41e1d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8d2a0700 1 -- 172.21.15.119:0/3634866279 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80011830 con 0x7f0f880b2200 2024-09-13T16:26:13.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8d2a0700 1 -- 172.21.15.119:0/3634866279 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f80011990 con 0x7f0f880b2200 2024-09-13T16:26:13.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8d2a0700 1 -- 172.21.15.119:0/3634866279 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80011d40 con 0x7f0f880b2200 2024-09-13T16:26:13.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/3634866279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880b2200 msgr2=0x7f0f880b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/3634866279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880b2200 0x7f0f880b26b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f0f80009910 tx=0x7f0f80009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:13.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/3634866279 shutdown_connections 2024-09-13T16:26:13.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/3634866279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880b2200 0x7f0f880b26b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/3634866279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f880cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/3634866279 >> 172.21.15.119:0/3634866279 conn(0x7f0f8806a350 msgr2=0x7f0f8806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:13.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/3634866279 shutdown_connections 2024-09-13T16:26:13.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.652+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/3634866279 wait complete. 2024-09-13T16:26:13.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.656+0000 7f0f8f2a4700 1 Processor -- start 2024-09-13T16:26:13.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.656+0000 7f0f8f2a4700 1 -- start start 2024-09-13T16:26:13.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.656+0000 7f0f8f2a4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f88162850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.656+0000 7f0f8f2a4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f880b2200 0x7f0f881f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8f2a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880cd8d0 0x7f0f880aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8f2a4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f881f6be0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8f2a4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8815d530 con 0x7f0f880b2200 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8f2a4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f8815d7d0 con 0x7f0f880b17f0 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8e2a2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f88162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8eaa3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880cd8d0 0x7f0f880aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8daa1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f880b2200 0x7f0f881f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8e2a2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f88162850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:51962/0 (socket says 172.21.15.119:51962) 2024-09-13T16:26:13.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8e2a2700 1 -- 172.21.15.119:0/1316065116 learned_addr learned my addr 172.21.15.119:0/1316065116 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:13.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8eaa3700 1 -- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 msgr2=0x7f0f88162850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8eaa3700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f88162850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8eaa3700 1 -- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f880b2200 msgr2=0x7f0f881f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8eaa3700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f880b2200 0x7f0f881f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8eaa3700 1 -- 172.21.15.119:0/1316065116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f80009570 con 0x7f0f880cd8d0 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8eaa3700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880cd8d0 0x7f0f880aac90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0f80016000 tx=0x7f0f80004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f7effd700 1 -- 172.21.15.119:0/1316065116 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f80004300 con 0x7f0f880cd8d0 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f7effd700 1 -- 172.21.15.119:0/1316065116 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f8001f070 con 0x7f0f880cd8d0 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f8815dab0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f7effd700 1 -- 172.21.15.119:0/1316065116 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f800036c0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f88162e30 con 0x7f0f880cd8d0 2024-09-13T16:26:13.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.660+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f8815dfd0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.664+0000 7f0f7effd700 1 -- 172.21.15.119:0/1316065116 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0f80021030 con 0x7f0f880cd8d0 2024-09-13T16:26:13.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.664+0000 7f0f7effd700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f74066730 0x7f0f74068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:13.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.664+0000 7f0f7effd700 1 -- 172.21.15.119:0/1316065116 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f0f800856e0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.664+0000 7f0f8e2a2700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f74066730 0x7f0f74068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:13.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.664+0000 7f0f8e2a2700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f74066730 0x7f0f74068bf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0f780018a0 tx=0x7f0f78009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:13.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.672+0000 7f0f7effd700 1 -- 172.21.15.119:0/1316065116 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0f8815dfd0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.848+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0f8815dfd0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.848+0000 7f0f7effd700 1 -- 172.21.15.119:0/1316065116 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f0f8815dfd0 con 0x7f0f880cd8d0 2024-09-13T16:26:13.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f74066730 msgr2=0x7f0f74068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f74066730 0x7f0f74068bf0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0f780018a0 tx=0x7f0f78009040 comp rx=0 tx=0).stop 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880cd8d0 msgr2=0x7f0f880aac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880cd8d0 0x7f0f880aac90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f0f80016000 tx=0x7f0f80004770 comp rx=0 tx=0).stop 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 shutdown_connections 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f74066730 0x7f0f74068bf0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f880cd8d0 0x7f0f880aac90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f880b2200 0x7f0f881f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 --2- 172.21.15.119:0/1316065116 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f880b17f0 0x7f0f88162850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 >> 172.21.15.119:0/1316065116 conn(0x7f0f8806a350 msgr2=0x7f0f880caa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 shutdown_connections 2024-09-13T16:26:13.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:13.852+0000 7f0f8f2a4700 1 -- 172.21.15.119:0/1316065116 wait complete. 2024-09-13T16:26:13.859 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:13.870 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi145.front.sepia.ceph.com... 2024-09-13T16:26:13.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:13.870 DEBUG:teuthology.orchestra.run.smithi145:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-09-13T16:26:13.901 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:13.902 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:26:13.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:13.902 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:26:13.926 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:13.927 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:13.934 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:13.934 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:13.982 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:14.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375e2bc700 1 Processor -- start 2024-09-13T16:26:14.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375e2bc700 1 -- start start 2024-09-13T16:26:14.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375e2bc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 0x7f37580b3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375e2bc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580b2870 con 0x7f3758159540 2024-09-13T16:26:14.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375e2bc700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580b2ac0 con 0x7f3758159920 2024-09-13T16:26:14.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375e2bc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37580b2da0 con 0x7f37580b1c10 2024-09-13T16:26:14.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3758159540 0x7f375806d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59372/0 (socket says 172.21.15.119:59372) 2024-09-13T16:26:14.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 -- 172.21.15.119:0/2244229623 learned_addr learned my addr 172.21.15.119:0/2244229623 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:14.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375dabb700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 0x7f37580b3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080239974 0 0) 0x7f37580b2870 con 0x7f3758159540 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37440035d0 con 0x7f3758159540 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4181714551 0 0) 0x7f37440035d0 con 0x7f3758159540 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 msgr2=0x7f37580b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 0x7f37580b3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 >> v1:172.21.15.145:6789/0 conn(0x7f3758159920 legacy=0x7f37580b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37580b3080 con 0x7f3758159540 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f37500019f0 con 0x7f3758159540 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375dabb700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 0x7f37580b3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 >> v1:172.21.15.119:6789/0 conn(0x7f3758159540 legacy=0x7f375806d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3744003a50 0x7f3744003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744004870 0x7f3744004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 0x7f3744005280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3744003a50 0x7f3744003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37580b2da0 con 0x7f3744003a50 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f374400bbc0 con 0x7f3744004870 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f374400bd80 con 0x7f37580b1c10 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375dabb700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 0x7f3744005280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375cab9700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744004870 0x7f3744004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 -- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 msgr2=0x7f3744005280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37580b1c10 0x7f3744005280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 -- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744004870 msgr2=0x7f3744004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744004870 0x7f3744004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 -- 172.21.15.119:0/2244229623 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37440035d0 con 0x7f3744003a50 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.088+0000 7f375d2ba700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3744003a50 0x7f3744003e70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f375000a760 tx=0x7f37440058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27dda343709d6834 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375000a250 con 0x7f3744003a50 2024-09-13T16:26:14.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37500072f0 con 0x7f3744003a50 2024-09-13T16:26:14.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f3757fff700 1 -- 172.21.15.119:0/2244229623 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37500076a0 con 0x7f3744003a50 2024-09-13T16:26:14.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f375e2bc700 1 -- 172.21.15.119:0/2244229623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3744003a50 msgr2=0x7f3744003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f375e2bc700 1 --2- 172.21.15.119:0/2244229623 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3744003a50 0x7f3744003e70 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f375000a760 tx=0x7f37440058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f375dabb700 1 -- 172.21.15.119:0/2244229623 reap_dead start 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f375e2bc700 1 -- 172.21.15.119:0/2244229623 shutdown_connections 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f375e2bc700 1 -- 172.21.15.119:0/2244229623 >> 172.21.15.119:0/2244229623 conn(0x7f375806b460 msgr2=0x7f375806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f375e2bc700 1 -- 172.21.15.119:0/2244229623 shutdown_connections 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.092+0000 7f375e2bc700 1 -- 172.21.15.119:0/2244229623 wait complete. 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 Processor -- start 2024-09-13T16:26:14.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 -- start start 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744003a50 0x7f375814cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3758152c30 0x7f37581530b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f375814f790 0x7f375814fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37581535f0 con 0x7f375814f790 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3758150150 con 0x7f3744003a50 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37581528a0 con 0x7f3758152c30 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f375814f790 0x7f375814fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f375814f790 0x7f375814fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41800/0 (socket says 172.21.15.119:41800) 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 -- 172.21.15.119:0/4256105515 learned_addr learned my addr 172.21.15.119:0/4256105515 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375cab9700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3758152c30 0x7f37581530b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375d2ba700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744003a50 0x7f375814cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 -- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3758152c30 msgr2=0x7f37581530b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3758152c30 0x7f37581530b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 -- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744003a50 msgr2=0x7f375814cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744003a50 0x7f375814cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 -- 172.21.15.119:0/4256105515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3758150370 con 0x7f375814f790 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375cab9700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3758152c30 0x7f37581530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375dabb700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f375814f790 0x7f375814fc10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f37480046d0 tx=0x7f374800f570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f3755ffb700 1 -- 172.21.15.119:0/4256105515 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374800c070 con 0x7f375814f790 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f3755ffb700 1 -- 172.21.15.119:0/4256105515 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3748014680 con 0x7f375814f790 2024-09-13T16:26:14.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3758150650 con 0x7f375814f790 2024-09-13T16:26:14.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f3755ffb700 1 -- 172.21.15.119:0/4256105515 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3748014a30 con 0x7f375814f790 2024-09-13T16:26:14.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.096+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3758150c10 con 0x7f375814f790 2024-09-13T16:26:14.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.100+0000 7f3755ffb700 1 -- 172.21.15.119:0/4256105515 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3748014b90 con 0x7f375814f790 2024-09-13T16:26:14.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.100+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37580b3c60 con 0x7f375814f790 2024-09-13T16:26:14.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.100+0000 7f3755ffb700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3738066730 0x7f3738068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.100+0000 7f3755ffb700 1 -- 172.21.15.119:0/4256105515 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3748084080 con 0x7f375814f790 2024-09-13T16:26:14.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.100+0000 7f375d2ba700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3738066730 0x7f3738068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.100+0000 7f375d2ba700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3738066730 0x7f3738068bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3750016040 tx=0x7f3750009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.112+0000 7f3755ffb700 1 -- 172.21.15.119:0/4256105515 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3748051030 con 0x7f375814f790 2024-09-13T16:26:14.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.284+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3758151a00 con 0x7f375814f790 2024-09-13T16:26:14.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.288+0000 7f3755ffb700 1 -- 172.21.15.119:0/4256105515 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f3748054650 con 0x7f375814f790 2024-09-13T16:26:14.294 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:14.294 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:14.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3738066730 msgr2=0x7f3738068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3738066730 0x7f3738068bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3750016040 tx=0x7f3750009000 comp rx=0 tx=0).stop 2024-09-13T16:26:14.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f375814f790 msgr2=0x7f375814fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f375814f790 0x7f375814fc10 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f37480046d0 tx=0x7f374800f570 comp rx=0 tx=0).stop 2024-09-13T16:26:14.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 shutdown_connections 2024-09-13T16:26:14.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3738066730 0x7f3738068bf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f375814f790 0x7f375814fc10 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3758152c30 0x7f37581530b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 --2- 172.21.15.119:0/4256105515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3744003a50 0x7f375814cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 >> 172.21.15.119:0/4256105515 conn(0x7f375806b460 msgr2=0x7f37580ce980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:14.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 shutdown_connections 2024-09-13T16:26:14.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.292+0000 7f375e2bc700 1 -- 172.21.15.119:0/4256105515 wait complete. 2024-09-13T16:26:14.309 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:14.309 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:14.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b16504700 1 Processor -- start 2024-09-13T16:26:14.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b16504700 1 -- start start 2024-09-13T16:26:14.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b16504700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 0x7f8b100cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b16504700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100d0120 con 0x7f8b100d1830 2024-09-13T16:26:14.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b16504700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b100cfe40 con 0x7f8b100cf8c0 2024-09-13T16:26:14.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b16504700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100b5b30 con 0x7f8b1015cc30 2024-09-13T16:26:14.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b15d03700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8b100d1830 0x7f8b100b6610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59380/0 (socket says 172.21.15.119:59380) 2024-09-13T16:26:14.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b15d03700 1 -- 172.21.15.119:0/1008976753 learned_addr learned my addr 172.21.15.119:0/1008976753 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:14.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b15502700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 0x7f8b100cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56129960 0 0) 0x7f8b100d0120 con 0x7f8b100d1830 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8afc0035d0 con 0x7f8b100d1830 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692915741 0 0) 0x7f8b100cfe40 con 0x7f8b100cf8c0 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b100d0120 con 0x7f8b100cf8c0 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1910078537 0 0) 0x7f8afc0035d0 con 0x7f8b100d1830 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 msgr2=0x7f8b100cf380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 0x7f8b100cf380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 >> v1:172.21.15.145:6789/0 conn(0x7f8b100cf8c0 legacy=0x7f8b100d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b100b5e10 con 0x7f8b100d1830 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f8b080019f0 con 0x7f8b100d1830 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 >> v1:172.21.15.119:6789/0 conn(0x7f8b100d1830 legacy=0x7f8b100b6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8afc003be0 0x7f8afc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8afc004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 0x7f8afc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b100b5b30 con 0x7f8afc004870 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8afc00bae0 con 0x7f8afc003be0 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8afc00bc70 con 0x7f8b1015cc30 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.416+0000 7f8b14d01700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8afc004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b15502700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8afc003be0 0x7f8afc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b15d03700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 0x7f8afc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b14d01700 1 -- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 msgr2=0x7f8afc005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b14d01700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b1015cc30 0x7f8afc005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b14d01700 1 -- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8afc003be0 msgr2=0x7f8afc003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b14d01700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8afc003be0 0x7f8afc003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b14d01700 1 -- 172.21.15.119:0/1008976753 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8afc0035d0 con 0x7f8afc004870 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b14d01700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8afc004cf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b100d0610 tx=0x7f8afc0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=239d04f224ebebc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04001450 con 0x7f8afc004870 2024-09-13T16:26:14.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b040052f0 con 0x7f8afc004870 2024-09-13T16:26:14.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b03fff700 1 -- 172.21.15.119:0/1008976753 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b04001d60 con 0x7f8afc004870 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b16504700 1 -- 172.21.15.119:0/1008976753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 msgr2=0x7f8afc004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b16504700 1 --2- 172.21.15.119:0/1008976753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8afc004cf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8b100d0610 tx=0x7f8afc0058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b15d03700 1 -- 172.21.15.119:0/1008976753 reap_dead start 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b16504700 1 -- 172.21.15.119:0/1008976753 shutdown_connections 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b16504700 1 -- 172.21.15.119:0/1008976753 >> 172.21.15.119:0/1008976753 conn(0x7f8b1008c0f0 msgr2=0x7f8b100d2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b16504700 1 -- 172.21.15.119:0/1008976753 shutdown_connections 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.420+0000 7f8b16504700 1 -- 172.21.15.119:0/1008976753 wait complete. 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 Processor -- start 2024-09-13T16:26:14.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 -- start start 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8b1015f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b101571c0 0x7f8b101520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b10158bf0 0x7f8b10159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b10154560 con 0x7f8afc004870 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1015f860 con 0x7f8b10158bf0 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1015fb40 con 0x7f8b101571c0 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8b1015f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b14d01700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b101571c0 0x7f8b101520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15d03700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b10158bf0 0x7f8b10159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8b1015f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41806/0 (socket says 172.21.15.119:41806) 2024-09-13T16:26:14.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 -- 172.21.15.119:0/4255905086 learned_addr learned my addr 172.21.15.119:0/4255905086 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:14.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 -- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b101571c0 msgr2=0x7f8b101520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b101571c0 0x7f8b101520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 -- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b10158bf0 msgr2=0x7f8b10159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b10158bf0 0x7f8b10159070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 -- 172.21.15.119:0/4255905086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b1015fe20 con 0x7f8afc004870 2024-09-13T16:26:14.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b15502700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8b1015f180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c011a20 tx=0x7f8b0c011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b01ffb700 1 -- 172.21.15.119:0/4255905086 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c011ec0 con 0x7f8afc004870 2024-09-13T16:26:14.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b01ffb700 1 -- 172.21.15.119:0/4255905086 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c00f940 con 0x7f8afc004870 2024-09-13T16:26:14.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b10160100 con 0x7f8afc004870 2024-09-13T16:26:14.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b01ffb700 1 -- 172.21.15.119:0/4255905086 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c00fcd0 con 0x7f8afc004870 2024-09-13T16:26:14.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.424+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b10157640 con 0x7f8afc004870 2024-09-13T16:26:14.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.428+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b100cf380 con 0x7f8afc004870 2024-09-13T16:26:14.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.428+0000 7f8b01ffb700 1 -- 172.21.15.119:0/4255905086 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8b0c01d400 con 0x7f8afc004870 2024-09-13T16:26:14.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.428+0000 7f8b01ffb700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af4066730 0x7f8af4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.428+0000 7f8b01ffb700 1 -- 172.21.15.119:0/4255905086 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f8b0c084ac0 con 0x7f8afc004870 2024-09-13T16:26:14.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.428+0000 7f8b14d01700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af4066730 0x7f8af4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.432+0000 7f8b14d01700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af4066730 0x7f8af4068bf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8b04005040 tx=0x7f8b0400a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.436+0000 7f8b01ffb700 1 -- 172.21.15.119:0/4255905086 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8b0c051b20 con 0x7f8afc004870 2024-09-13T16:26:14.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.608+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b1006b450 con 0x7f8afc004870 2024-09-13T16:26:14.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.608+0000 7f8b01ffb700 1 -- 172.21.15.119:0/4255905086 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b0c055140 con 0x7f8afc004870 2024-09-13T16:26:14.613 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:14.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.608+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af4066730 msgr2=0x7f8af4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.608+0000 7f8b16504700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af4066730 0x7f8af4068bf0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8b04005040 tx=0x7f8b0400a000 comp rx=0 tx=0).stop 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.608+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 msgr2=0x7f8b1015f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.608+0000 7f8b16504700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8b1015f180 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c011a20 tx=0x7f8b0c011d30 comp rx=0 tx=0).stop 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 shutdown_connections 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af4066730 0x7f8af4068bf0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b10158bf0 0x7f8b10159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b101571c0 0x7f8b101520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 --2- 172.21.15.119:0/4255905086 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8afc004870 0x7f8b1015f180 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 >> 172.21.15.119:0/4255905086 conn(0x7f8b1008c0f0 msgr2=0x7f8b100ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 shutdown_connections 2024-09-13T16:26:14.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.612+0000 7f8b16504700 1 -- 172.21.15.119:0/4255905086 wait complete. 2024-09-13T16:26:14.628 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:14.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.736+0000 7f4499469700 1 Processor -- start 2024-09-13T16:26:14.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.736+0000 7f4499469700 1 -- start start 2024-09-13T16:26:14.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4494153f00 0x7f449415a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4494073ab0 0x7f449406b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f449415dfb0 con 0x7f449406b9a0 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f449406be30 con 0x7f4494073ab0 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4494073f20 con 0x7f4494153f00 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f448a7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4494073ab0 0x7f449406b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f44937fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f449406b9a0 0x7f44940b0e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59384/0 (socket says 172.21.15.119:59384) 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f44937fe700 1 -- 172.21.15.119:0/724901318 learned_addr learned my addr 172.21.15.119:0/724901318 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4492ffd700 1 --2- 172.21.15.119:0/724901318 >> v2:172.21.15.145:3300/0 conn(0x7f4494153f00 0x7f449415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f44927fc700 1 -- 172.21.15.119:0/724901318 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3010224483 0 0) 0x7f449415dfb0 con 0x7f449406b9a0 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f44927fc700 1 -- 172.21.15.119:0/724901318 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44800035d0 con 0x7f449406b9a0 2024-09-13T16:26:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f448a7fc700 1 -- 172.21.15.119:0/724901318 >> v2:172.21.15.145:3300/0 conn(0x7f4494153f00 msgr2=0x7f449415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f448a7fc700 1 --2- 172.21.15.119:0/724901318 >> v2:172.21.15.145:3300/0 conn(0x7f4494153f00 0x7f449415a830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f448a7fc700 1 -- 172.21.15.119:0/724901318 >> v1:172.21.15.119:6789/0 conn(0x7f449406b9a0 legacy=0x7f44940b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f448a7fc700 1 -- 172.21.15.119:0/724901318 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f449415d3a0 con 0x7f4494073ab0 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f448a7fc700 1 --2- 172.21.15.119:0/724901318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494073ab0 0x7f449406b460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f44940b1f20 tx=0x7f447c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aa323f7ce90bb13 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f44927fc700 1 -- 172.21.15.119:0/724901318 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c011830 con 0x7f4494073ab0 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f44927fc700 1 -- 172.21.15.119:0/724901318 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f447c011990 con 0x7f4494073ab0 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f44927fc700 1 -- 172.21.15.119:0/724901318 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f447c011d20 con 0x7f4494073ab0 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- 172.21.15.119:0/724901318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494073ab0 msgr2=0x7f449406b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 --2- 172.21.15.119:0/724901318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494073ab0 0x7f449406b460 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f44940b1f20 tx=0x7f447c009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- 172.21.15.119:0/724901318 shutdown_connections 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 --2- 172.21.15.119:0/724901318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494073ab0 0x7f449406b460 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 --2- 172.21.15.119:0/724901318 >> v2:172.21.15.145:3300/0 conn(0x7f4494153f00 0x7f449415a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- 172.21.15.119:0/724901318 >> 172.21.15.119:0/724901318 conn(0x7f44940b22e0 msgr2=0x7f44940b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- 172.21.15.119:0/724901318 shutdown_connections 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.740+0000 7f4499469700 1 -- 172.21.15.119:0/724901318 wait complete. 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 Processor -- start 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 -- start start 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f449406b9a0 0x7f4494098a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4494073ab0 0x7f44940b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494153f00 0x7f449414bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44940b02f0 con 0x7f4494153f00 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4494098f50 con 0x7f4494073ab0 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44940b0500 con 0x7f449406b9a0 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494153f00 0x7f449414bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4492ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f449406b9a0 0x7f4494098a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494153f00 0x7f449414bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41826/0 (socket says 172.21.15.119:41826) 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4492ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f449406b9a0 0x7f4494098a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:52008/0 (socket says 172.21.15.119:52008) 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 -- 172.21.15.119:0/2462189517 learned_addr learned my addr 172.21.15.119:0/2462189517 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f448a7fc700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4494073ab0 0x7f44940b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 -- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f449406b9a0 msgr2=0x7f4494098a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f449406b9a0 0x7f4494098a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 -- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4494073ab0 msgr2=0x7f44940b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4494073ab0 0x7f44940b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 -- 172.21.15.119:0/2462189517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447c0094b0 con 0x7f4494153f00 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f44937fe700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494153f00 0x7f449414bb50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f448c00cb40 tx=0x7f448c00cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f448bfff700 1 -- 172.21.15.119:0/2462189517 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c007500 con 0x7f4494153f00 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f448bfff700 1 -- 172.21.15.119:0/2462189517 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448c007660 con 0x7f4494153f00 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44940b07e0 con 0x7f4494153f00 2024-09-13T16:26:14.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f448bfff700 1 -- 172.21.15.119:0/2462189517 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f448c0079f0 con 0x7f4494153f00 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.744+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44940b0da0 con 0x7f4494153f00 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.748+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f449406cd30 con 0x7f4494153f00 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.748+0000 7f448bfff700 1 -- 172.21.15.119:0/2462189517 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f448c01b3b0 con 0x7f4494153f00 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.748+0000 7f448bfff700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4474066730 0x7f4474068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.748+0000 7f448bfff700 1 -- 172.21.15.119:0/2462189517 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f448c083f20 con 0x7f4494153f00 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.748+0000 7f4492ffd700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4474066730 0x7f4474068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.752+0000 7f4492ffd700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4474066730 0x7f4474068bf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4484009b50 tx=0x7f4484009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:14.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.756+0000 7f448bfff700 1 -- 172.21.15.119:0/2462189517 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f448c050ee0 con 0x7f4494153f00 2024-09-13T16:26:14.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.924+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f449406ca50 con 0x7f4494153f00 2024-09-13T16:26:14.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.924+0000 7f448bfff700 1 -- 172.21.15.119:0/2462189517 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f448c017020 con 0x7f4494153f00 2024-09-13T16:26:14.929 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4474066730 msgr2=0x7f4474068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4474066730 0x7f4474068bf0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f4484009b50 tx=0x7f4484009240 comp rx=0 tx=0).stop 2024-09-13T16:26:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494153f00 msgr2=0x7f449414bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494153f00 0x7f449414bb50 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f448c00cb40 tx=0x7f448c00cb70 comp rx=0 tx=0).stop 2024-09-13T16:26:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 shutdown_connections 2024-09-13T16:26:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4474066730 0x7f4474068bf0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4494153f00 0x7f449414bb50 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4494073ab0 0x7f44940b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 --2- 172.21.15.119:0/2462189517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f449406b9a0 0x7f4494098a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:14.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 >> 172.21.15.119:0/2462189517 conn(0x7f44940b22e0 msgr2=0x7f4494066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:14.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 shutdown_connections 2024-09-13T16:26:14.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:14.928+0000 7f4499469700 1 -- 172.21.15.119:0/2462189517 wait complete. 2024-09-13T16:26:14.943 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:15.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.064+0000 7f53c61fb700 1 Processor -- start 2024-09-13T16:26:15.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.064+0000 7f53c61fb700 1 -- start start 2024-09-13T16:26:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.064+0000 7f53c61fb700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53c015cc00 0x7f53c00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.064+0000 7f53c61fb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c00b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.064+0000 7f53c61fb700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c00cdb00 con 0x7f53c00d0c20 2024-09-13T16:26:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.064+0000 7f53c61fb700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c00cdde0 con 0x7f53c015cc00 2024-09-13T16:26:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.064+0000 7f53c61fb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c00ce0c0 con 0x7f53c00d17e0 2024-09-13T16:26:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c49f8700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f53c00d0c20 0x7f53c00d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59398/0 (socket says 172.21.15.119:59398) 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c49f8700 1 -- 172.21.15.119:0/1089400157 learned_addr learned my addr 172.21.15.119:0/1089400157 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c59fa700 1 --2- 172.21.15.119:0/1089400157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c00b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c51f9700 1 --2- 172.21.15.119:0/1089400157 >> v2:172.21.15.145:3300/0 conn(0x7f53c015cc00 0x7f53c00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53b3fff700 1 -- 172.21.15.119:0/1089400157 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920410831 0 0) 0x7f53c00cdb00 con 0x7f53c00d0c20 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53b3fff700 1 -- 172.21.15.119:0/1089400157 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53ac0035d0 con 0x7f53c00d0c20 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c59fa700 1 -- 172.21.15.119:0/1089400157 >> v2:172.21.15.145:3300/0 conn(0x7f53c015cc00 msgr2=0x7f53c00d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c59fa700 1 --2- 172.21.15.119:0/1089400157 >> v2:172.21.15.145:3300/0 conn(0x7f53c015cc00 0x7f53c00d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c59fa700 1 -- 172.21.15.119:0/1089400157 >> v1:172.21.15.119:6789/0 conn(0x7f53c00d0c20 legacy=0x7f53c00d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c59fa700 1 -- 172.21.15.119:0/1089400157 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c00ce3a0 con 0x7f53c00d17e0 2024-09-13T16:26:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c59fa700 1 --2- 172.21.15.119:0/1089400157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c00b5960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009d90 tx=0x7f53b800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b96d26e4748d466 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53b3fff700 1 -- 172.21.15.119:0/1089400157 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b800e070 con 0x7f53c00d17e0 2024-09-13T16:26:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53b3fff700 1 -- 172.21.15.119:0/1089400157 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b800bb20 con 0x7f53c00d17e0 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53b3fff700 1 -- 172.21.15.119:0/1089400157 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8012430 con 0x7f53c00d17e0 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 -- 172.21.15.119:0/1089400157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 msgr2=0x7f53c00b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 --2- 172.21.15.119:0/1089400157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c00b5960 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f53b8009d90 tx=0x7f53b800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 -- 172.21.15.119:0/1089400157 shutdown_connections 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 --2- 172.21.15.119:0/1089400157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c00b5960 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 --2- 172.21.15.119:0/1089400157 >> v2:172.21.15.145:3300/0 conn(0x7f53c015cc00 0x7f53c00d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 -- 172.21.15.119:0/1089400157 >> 172.21.15.119:0/1089400157 conn(0x7f53c008c080 msgr2=0x7f53c00d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 -- 172.21.15.119:0/1089400157 shutdown_connections 2024-09-13T16:26:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.068+0000 7f53c61fb700 1 -- 172.21.15.119:0/1089400157 wait complete. 2024-09-13T16:26:15.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 Processor -- start 2024-09-13T16:26:15.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 -- start start 2024-09-13T16:26:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53c00d0c20 0x7f53c0154fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c01577c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53c015cc00 0x7f53c0157d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c0155510 con 0x7f53c00d0c20 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c015d300 con 0x7f53c015cc00 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c61fb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c015d5e0 con 0x7f53c00d17e0 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53c00d0c20 0x7f53c0154fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53c00d0c20 0x7f53c0154fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41838/0 (socket says 172.21.15.119:41838) 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 -- 172.21.15.119:0/2340252223 learned_addr learned my addr 172.21.15.119:0/2340252223 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c49f8700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c01577c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c59fa700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53c015cc00 0x7f53c0157d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 -- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 msgr2=0x7f53c01577c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c01577c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 -- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53c015cc00 msgr2=0x7f53c0157d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53c015cc00 0x7f53c0157d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.072+0000 7f53c51f9700 1 -- 172.21.15.119:0/2340252223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b8009a70 con 0x7f53c00d0c20 2024-09-13T16:26:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53c51f9700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53c00d0c20 0x7f53c0154fd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f53b400e840 tx=0x7f53b400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53b1ffb700 1 -- 172.21.15.119:0/2340252223 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b40095a0 con 0x7f53c00d0c20 2024-09-13T16:26:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53b1ffb700 1 -- 172.21.15.119:0/2340252223 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b4013680 con 0x7f53c00d0c20 2024-09-13T16:26:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c015d8c0 con 0x7f53c00d0c20 2024-09-13T16:26:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53b1ffb700 1 -- 172.21.15.119:0/2340252223 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b40139f0 con 0x7f53c00d0c20 2024-09-13T16:26:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c015de80 con 0x7f53c00d0c20 2024-09-13T16:26:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c00ce8c0 con 0x7f53c00d0c20 2024-09-13T16:26:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53b1ffb700 1 -- 172.21.15.119:0/2340252223 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f53b4013b50 con 0x7f53c00d0c20 2024-09-13T16:26:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53b1ffb700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53a4066730 0x7f53a4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53b1ffb700 1 -- 172.21.15.119:0/2340252223 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f53b40842d0 con 0x7f53c00d0c20 2024-09-13T16:26:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.076+0000 7f53c49f8700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53a4066730 0x7f53a4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.080+0000 7f53c49f8700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53a4066730 0x7f53a4068bf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f53c01607c0 tx=0x7f53bc007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.088+0000 7f53b1ffb700 1 -- 172.21.15.119:0/2340252223 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f53b4051310 con 0x7f53c00d0c20 2024-09-13T16:26:15.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.252+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f53c008c560 con 0x7f53c00d0c20 2024-09-13T16:26:15.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.252+0000 7f53b1ffb700 1 -- 172.21.15.119:0/2340252223 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f53b4054930 con 0x7f53c00d0c20 2024-09-13T16:26:15.258 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:15.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53a4066730 msgr2=0x7f53a4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53a4066730 0x7f53a4068bf0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f53c01607c0 tx=0x7f53bc007de0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53c00d0c20 msgr2=0x7f53c0154fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53c00d0c20 0x7f53c0154fd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f53b400e840 tx=0x7f53b400eb50 comp rx=0 tx=0).stop 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 shutdown_connections 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53a4066730 0x7f53a4068bf0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53c015cc00 0x7f53c0157d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53c00d17e0 0x7f53c01577c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 --2- 172.21.15.119:0/2340252223 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53c00d0c20 0x7f53c0154fd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 >> 172.21.15.119:0/2340252223 conn(0x7f53c008c080 msgr2=0x7f53c00b8040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 shutdown_connections 2024-09-13T16:26:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.256+0000 7f53c61fb700 1 -- 172.21.15.119:0/2340252223 wait complete. 2024-09-13T16:26:15.273 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:15.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 Processor -- start 2024-09-13T16:26:15.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 -- start start 2024-09-13T16:26:15.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f68800add60 0x7f68800ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f68800b3260 0x7f68800b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6880071be0 con 0x7f68800b3260 2024-09-13T16:26:15.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6880159af0 con 0x7f68800add60 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f68849ee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68800b3c30 con 0x7f688015dbe0 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f68800b3260 0x7f68800b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f68800b3260 0x7f68800b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41846/0 (socket says 172.21.15.119:41846) 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 -- 172.21.15.119:0/109379864 learned_addr learned my addr 172.21.15.119:0/109379864 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687e59c700 1 --2- 172.21.15.119:0/109379864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687ed9d700 1 --2- 172.21.15.119:0/109379864 >> v2:172.21.15.145:3300/0 conn(0x7f68800add60 0x7f68800ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 -- 172.21.15.119:0/109379864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 msgr2=0x7f68800cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 --2- 172.21.15.119:0/109379864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 -- 172.21.15.119:0/109379864 >> v2:172.21.15.145:3300/0 conn(0x7f68800add60 msgr2=0x7f68800ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 --2- 172.21.15.119:0/109379864 >> v2:172.21.15.145:3300/0 conn(0x7f68800add60 0x7f68800ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 -- 172.21.15.119:0/109379864 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68800cd4d0 con 0x7f68800b3260 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687e59c700 1 --2- 172.21.15.119:0/109379864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687ed9d700 1 --2- 172.21.15.119:0/109379864 >> v2:172.21.15.145:3300/0 conn(0x7f68800add60 0x7f68800ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.388+0000 7f687f59e700 1 --2- 172.21.15.119:0/109379864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800b3260 0x7f68800b36f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f68800b1530 tx=0x7f6878009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d4dcb2f279c8f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f687dd9b700 1 -- 172.21.15.119:0/109379864 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f687800ba20 con 0x7f68800b3260 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f687dd9b700 1 -- 172.21.15.119:0/109379864 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f687800bb80 con 0x7f68800b3260 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f687dd9b700 1 -- 172.21.15.119:0/109379864 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6878009e60 con 0x7f68800b3260 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 -- 172.21.15.119:0/109379864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800b3260 msgr2=0x7f68800b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 --2- 172.21.15.119:0/109379864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800b3260 0x7f68800b36f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f68800b1530 tx=0x7f6878009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 -- 172.21.15.119:0/109379864 shutdown_connections 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 --2- 172.21.15.119:0/109379864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800b3260 0x7f68800b36f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 --2- 172.21.15.119:0/109379864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 --2- 172.21.15.119:0/109379864 >> v2:172.21.15.145:3300/0 conn(0x7f68800add60 0x7f68800ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 -- 172.21.15.119:0/109379864 >> 172.21.15.119:0/109379864 conn(0x7f68800b0980 msgr2=0x7f68800b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 -- 172.21.15.119:0/109379864 shutdown_connections 2024-09-13T16:26:15.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 -- 172.21.15.119:0/109379864 wait complete. 2024-09-13T16:26:15.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 Processor -- start 2024-09-13T16:26:15.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.392+0000 7f68849ee700 1 -- start start 2024-09-13T16:26:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800add60 0x7f688006cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68800b3260 0x7f68801492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800ae340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688006d220 con 0x7f68800add60 2024-09-13T16:26:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6880153d90 con 0x7f68800b3260 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6880158340 con 0x7f688015dbe0 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800add60 0x7f688006cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800add60 0x7f688006cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41854/0 (socket says 172.21.15.119:41854) 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 -- 172.21.15.119:0/3963689994 learned_addr learned my addr 172.21.15.119:0/3963689994 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687f59e700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800ae340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687e59c700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68800b3260 0x7f68801492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 -- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 msgr2=0x7f68800ae340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800ae340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 -- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68800b3260 msgr2=0x7f68801492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68800b3260 0x7f68801492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 -- 172.21.15.119:0/3963689994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6878009570 con 0x7f68800add60 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f687ed9d700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800add60 0x7f688006cce0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f686800c770 tx=0x7f686800ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68777fe700 1 -- 172.21.15.119:0/3963689994 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868011800 con 0x7f68800add60 2024-09-13T16:26:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68777fe700 1 -- 172.21.15.119:0/3963689994 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6868011960 con 0x7f68800add60 2024-09-13T16:26:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68801585e0 con 0x7f68800add60 2024-09-13T16:26:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68777fe700 1 -- 172.21.15.119:0/3963689994 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6868011d10 con 0x7f68800add60 2024-09-13T16:26:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.396+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6880158ba0 con 0x7f68800add60 2024-09-13T16:26:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.400+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68800cd160 con 0x7f68800add60 2024-09-13T16:26:15.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.400+0000 7f68777fe700 1 -- 172.21.15.119:0/3963689994 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6868018360 con 0x7f68800add60 2024-09-13T16:26:15.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.400+0000 7f68777fe700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6864066730 0x7f6864068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.400+0000 7f68777fe700 1 -- 172.21.15.119:0/3963689994 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f68680837c0 con 0x7f68800add60 2024-09-13T16:26:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.400+0000 7f687e59c700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6864066730 0x7f6864068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.400+0000 7f687e59c700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6864066730 0x7f6864068bf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f68801f5c80 tx=0x7f68700077c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.404+0000 7f68777fe700 1 -- 172.21.15.119:0/3963689994 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6868050800 con 0x7f68800add60 2024-09-13T16:26:15.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.580+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6880150fc0 con 0x7f68800add60 2024-09-13T16:26:15.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68777fe700 1 -- 172.21.15.119:0/3963689994 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6868053e20 con 0x7f68800add60 2024-09-13T16:26:15.587 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:15.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6864066730 msgr2=0x7f6864068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6864066730 0x7f6864068bf0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f68801f5c80 tx=0x7f68700077c0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800add60 msgr2=0x7f688006cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800add60 0x7f688006cce0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f686800c770 tx=0x7f686800ca80 comp rx=0 tx=0).stop 2024-09-13T16:26:15.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 shutdown_connections 2024-09-13T16:26:15.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6864066730 0x7f6864068bf0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f688015dbe0 0x7f68800ae340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68800b3260 0x7f68801492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 --2- 172.21.15.119:0/3963689994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68800add60 0x7f688006cce0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 >> 172.21.15.119:0/3963689994 conn(0x7f68800b0980 msgr2=0x7f68801531e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:15.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 shutdown_connections 2024-09-13T16:26:15.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.584+0000 7f68849ee700 1 -- 172.21.15.119:0/3963689994 wait complete. 2024-09-13T16:26:15.603 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:15.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6ade665700 1 Processor -- start 2024-09-13T16:26:15.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6ade665700 1 -- start start 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6ade665700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad80adb30 0x7f6ad80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6ade665700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6ad80ce040 0x7f6ad80b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6ade665700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ad806cf60 con 0x7f6ad806bb90 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6ade665700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad806c350 con 0x7f6ad80ce040 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6ade665700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad806c630 con 0x7f6ad80adb30 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6ad80ce040 0x7f6ad80b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6add663700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad80adb30 0x7f6ad80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6ad80ce040 0x7f6ad80b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41866/0 (socket says 172.21.15.119:41866) 2024-09-13T16:26:15.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 -- 172.21.15.119:0/3200705385 learned_addr learned my addr 172.21.15.119:0/3200705385 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6acffff700 1 -- 172.21.15.119:0/3200705385 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649412267 0 0) 0x7f6ad806cf60 con 0x7f6ad806bb90 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 -- 172.21.15.119:0/3200705385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad80adb30 msgr2=0x7f6ad80cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 --2- 172.21.15.119:0/3200705385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad80adb30 0x7f6ad80cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 -- 172.21.15.119:0/3200705385 >> v1:172.21.15.145:6789/0 conn(0x7f6ad806bb90 legacy=0x7f6ad80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 -- 172.21.15.119:0/3200705385 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad806c910 con 0x7f6ad80ce040 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6add663700 1 --2- 172.21.15.119:0/3200705385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad80adb30 0x7f6ad80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6adde64700 1 --2- 172.21.15.119:0/3200705385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b2380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009910 tx=0x7f6ad0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d55cfd4842f2da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6acffff700 1 -- 172.21.15.119:0/3200705385 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0011830 con 0x7f6ad80ce040 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6acffff700 1 -- 172.21.15.119:0/3200705385 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad0011990 con 0x7f6ad80ce040 2024-09-13T16:26:15.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.712+0000 7f6acffff700 1 -- 172.21.15.119:0/3200705385 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad0011d40 con 0x7f6ad80ce040 2024-09-13T16:26:15.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- 172.21.15.119:0/3200705385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 msgr2=0x7f6ad80b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 --2- 172.21.15.119:0/3200705385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b2380 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009910 tx=0x7f6ad0009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:15.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- 172.21.15.119:0/3200705385 shutdown_connections 2024-09-13T16:26:15.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 --2- 172.21.15.119:0/3200705385 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b2380 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 --2- 172.21.15.119:0/3200705385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad80adb30 0x7f6ad80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- 172.21.15.119:0/3200705385 >> 172.21.15.119:0/3200705385 conn(0x7f6ad806a350 msgr2=0x7f6ad806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:15.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- 172.21.15.119:0/3200705385 shutdown_connections 2024-09-13T16:26:15.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- 172.21.15.119:0/3200705385 wait complete. 2024-09-13T16:26:15.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 Processor -- start 2024-09-13T16:26:15.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- start start 2024-09-13T16:26:15.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad806bb90 0x7f6ad80a8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ad80adb30 0x7f6ad81f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b3330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad81f4170 con 0x7f6ad80ce040 2024-09-13T16:26:15.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad81f3bb0 con 0x7f6ad80adb30 2024-09-13T16:26:15.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6ade665700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad81f3e90 con 0x7f6ad806bb90 2024-09-13T16:26:15.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6add663700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad806bb90 0x7f6ad80a8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41874/0 (socket says 172.21.15.119:41874) 2024-09-13T16:26:15.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 -- 172.21.15.119:0/1276931823 learned_addr learned my addr 172.21.15.119:0/1276931823 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:15.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adce62700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ad80adb30 0x7f6ad81f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 -- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad806bb90 msgr2=0x7f6ad80a8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad806bb90 0x7f6ad80a8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 -- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ad80adb30 msgr2=0x7f6ad81f31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ad80adb30 0x7f6ad81f31f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.716+0000 7f6adde64700 1 -- 172.21.15.119:0/1276931823 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4009a70 con 0x7f6ad80ce040 2024-09-13T16:26:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6adde64700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b3330 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009f70 tx=0x7f6ad0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6acdffb700 1 -- 172.21.15.119:0/1276931823 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad001a900 con 0x7f6ad80ce040 2024-09-13T16:26:15.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6acdffb700 1 -- 172.21.15.119:0/1276931823 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ad001f070 con 0x7f6ad80ce040 2024-09-13T16:26:15.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad0009570 con 0x7f6ad80ce040 2024-09-13T16:26:15.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6acdffb700 1 -- 172.21.15.119:0/1276931823 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ad001acc0 con 0x7f6ad80ce040 2024-09-13T16:26:15.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad815cf60 con 0x7f6ad80ce040 2024-09-13T16:26:15.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad815cae0 con 0x7f6ad80ce040 2024-09-13T16:26:15.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6acdffb700 1 -- 172.21.15.119:0/1276931823 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6ad0026480 con 0x7f6ad80ce040 2024-09-13T16:26:15.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6acdffb700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6abc066730 0x7f6abc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:15.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6acdffb700 1 -- 172.21.15.119:0/1276931823 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6ad0021070 con 0x7f6ad80ce040 2024-09-13T16:26:15.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.720+0000 7f6add663700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6abc066730 0x7f6abc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:15.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.724+0000 7f6add663700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6abc066730 0x7f6abc068bf0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009fd0 tx=0x7f6ad400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.724+0000 7f6acdffb700 1 -- 172.21.15.119:0/1276931823 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6ad815cae0 con 0x7f6ad80ce040 2024-09-13T16:26:15.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6ad815cae0 con 0x7f6ad80ce040 2024-09-13T16:26:15.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6acdffb700 1 -- 172.21.15.119:0/1276931823 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6ad815cae0 con 0x7f6ad80ce040 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6abc066730 msgr2=0x7f6abc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6abc066730 0x7f6abc068bf0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009fd0 tx=0x7f6ad400b040 comp rx=0 tx=0).stop 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 msgr2=0x7f6ad80b3330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b3330 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0009f70 tx=0x7f6ad0004770 comp rx=0 tx=0).stop 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 shutdown_connections 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6abc066730 0x7f6abc068bf0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ad80ce040 0x7f6ad80b3330 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ad80adb30 0x7f6ad81f31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 --2- 172.21.15.119:0/1276931823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ad806bb90 0x7f6ad80a8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:15.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 >> 172.21.15.119:0/1276931823 conn(0x7f6ad806a350 msgr2=0x7f6ad80cb5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:15.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 shutdown_connections 2024-09-13T16:26:15.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:15.908+0000 7f6ade665700 1 -- 172.21.15.119:0/1276931823 wait complete. 2024-09-13T16:26:15.916 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:15.928 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:15.928 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:15.936 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:16.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.036+0000 7ffa6db53700 1 Processor -- start 2024-09-13T16:26:16.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.036+0000 7ffa6db53700 1 -- start start 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b3800 0x7ffa6806e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffa6806edd0 0x7ffa680b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffa680b4120 0x7ffa680ae6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa680ae230 con 0x7ffa680b4120 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6806f540 con 0x7ffa6806edd0 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6806f820 con 0x7ffa680b3800 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffa680b4120 0x7ffa680ae6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffa680b4120 0x7ffa680ae6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41882/0 (socket says 172.21.15.119:41882) 2024-09-13T16:26:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 -- 172.21.15.119:0/2131450825 learned_addr learned my addr 172.21.15.119:0/2131450825 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa67fff700 1 --2- 172.21.15.119:0/2131450825 >> v2:172.21.15.145:3300/0 conn(0x7ffa6806edd0 0x7ffa680b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6cb51700 1 --2- 172.21.15.119:0/2131450825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b3800 0x7ffa6806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 -- 172.21.15.119:0/2131450825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b3800 msgr2=0x7ffa6806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 --2- 172.21.15.119:0/2131450825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b3800 0x7ffa6806e890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 -- 172.21.15.119:0/2131450825 >> v2:172.21.15.145:3300/0 conn(0x7ffa6806edd0 msgr2=0x7ffa680b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:16.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 --2- 172.21.15.119:0/2131450825 >> v2:172.21.15.145:3300/0 conn(0x7ffa6806edd0 0x7ffa680b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 -- 172.21.15.119:0/2131450825 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6806fb00 con 0x7ffa680b4120 2024-09-13T16:26:16.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6d352700 1 --2- 172.21.15.119:0/2131450825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa680b4120 0x7ffa680ae6c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ffa680710a0 tx=0x7ffa6000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c4bea9e670348c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa677fe700 1 -- 172.21.15.119:0/2131450825 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6000e070 con 0x7ffa680b4120 2024-09-13T16:26:16.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa677fe700 1 -- 172.21.15.119:0/2131450825 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6000b910 con 0x7ffa680b4120 2024-09-13T16:26:16.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa677fe700 1 -- 172.21.15.119:0/2131450825 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6000bcc0 con 0x7ffa680b4120 2024-09-13T16:26:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- 172.21.15.119:0/2131450825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa680b4120 msgr2=0x7ffa680ae6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 --2- 172.21.15.119:0/2131450825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa680b4120 0x7ffa680ae6c0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ffa680710a0 tx=0x7ffa6000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- 172.21.15.119:0/2131450825 shutdown_connections 2024-09-13T16:26:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 --2- 172.21.15.119:0/2131450825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa680b4120 0x7ffa680ae6c0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 --2- 172.21.15.119:0/2131450825 >> v2:172.21.15.145:3300/0 conn(0x7ffa6806edd0 0x7ffa680b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 --2- 172.21.15.119:0/2131450825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b3800 0x7ffa6806e890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- 172.21.15.119:0/2131450825 >> 172.21.15.119:0/2131450825 conn(0x7ffa680b5d20 msgr2=0x7ffa680b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- 172.21.15.119:0/2131450825 shutdown_connections 2024-09-13T16:26:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.040+0000 7ffa6db53700 1 -- 172.21.15.119:0/2131450825 wait complete. 2024-09-13T16:26:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 Processor -- start 2024-09-13T16:26:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 -- start start 2024-09-13T16:26:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa6806edd0 0x7ffa6814d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa680b3800 0x7ffa680ced70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b4120 0x7ffa6815cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa680cd8d0 con 0x7ffa6806edd0 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6815d5c0 con 0x7ffa680b3800 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6db53700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6814d970 con 0x7ffa680b4120 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6cb51700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa6806edd0 0x7ffa6814d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6cb51700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa6806edd0 0x7ffa6814d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41884/0 (socket says 172.21.15.119:41884) 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6cb51700 1 -- 172.21.15.119:0/70683459 learned_addr learned my addr 172.21.15.119:0/70683459 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa67fff700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa680b3800 0x7ffa680ced70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.044+0000 7ffa6d352700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b4120 0x7ffa6815cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6cb51700 1 -- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b4120 msgr2=0x7ffa6815cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6cb51700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b4120 0x7ffa6815cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6cb51700 1 -- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa680b3800 msgr2=0x7ffa680ced70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6cb51700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa680b3800 0x7ffa680ced70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6cb51700 1 -- 172.21.15.119:0/70683459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa60009b30 con 0x7ffa6806edd0 2024-09-13T16:26:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6cb51700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa6806edd0 0x7ffa6814d430 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00ebe0 tx=0x7ffa5c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa657fa700 1 -- 172.21.15.119:0/70683459 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c009740 con 0x7ffa6806edd0 2024-09-13T16:26:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa657fa700 1 -- 172.21.15.119:0/70683459 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa5c00c850 con 0x7ffa6806edd0 2024-09-13T16:26:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6814ebc0 con 0x7ffa6806edd0 2024-09-13T16:26:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa657fa700 1 -- 172.21.15.119:0/70683459 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c00cbe0 con 0x7ffa6806edd0 2024-09-13T16:26:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6814f180 con 0x7ffa6806edd0 2024-09-13T16:26:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa68074560 con 0x7ffa6806edd0 2024-09-13T16:26:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa657fa700 1 -- 172.21.15.119:0/70683459 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ffa5c00cd40 con 0x7ffa6806edd0 2024-09-13T16:26:16.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.048+0000 7ffa657fa700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa54066760 0x7ffa54068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.052+0000 7ffa657fa700 1 -- 172.21.15.119:0/70683459 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ffa5c084440 con 0x7ffa6806edd0 2024-09-13T16:26:16.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.052+0000 7ffa67fff700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa54066760 0x7ffa54068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.052+0000 7ffa67fff700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa54066760 0x7ffa54068c20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffa6814ea40 tx=0x7ffa58009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.052+0000 7ffa657fa700 1 -- 172.21.15.119:0/70683459 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ffa5c0513e0 con 0x7ffa6806edd0 2024-09-13T16:26:16.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.220+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa6806e890 con 0x7ffa6806edd0 2024-09-13T16:26:16.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.224+0000 7ffa657fa700 1 -- 172.21.15.119:0/70683459 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7ffa5c054a00 con 0x7ffa6806edd0 2024-09-13T16:26:16.229 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:16.229 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:16.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.224+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa54066760 msgr2=0x7ffa54068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.224+0000 7ffa6db53700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa54066760 0x7ffa54068c20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ffa6814ea40 tx=0x7ffa58009240 comp rx=0 tx=0).stop 2024-09-13T16:26:16.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.224+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa6806edd0 msgr2=0x7ffa6814d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.224+0000 7ffa6db53700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa6806edd0 0x7ffa6814d430 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c00ebe0 tx=0x7ffa5c00eef0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.224+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 shutdown_connections 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.228+0000 7ffa6db53700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa54066760 0x7ffa54068c20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.228+0000 7ffa6db53700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa680b4120 0x7ffa6815cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.228+0000 7ffa6db53700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa680b3800 0x7ffa680ced70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.228+0000 7ffa6db53700 1 --2- 172.21.15.119:0/70683459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa6806edd0 0x7ffa6814d430 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.228+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 >> 172.21.15.119:0/70683459 conn(0x7ffa680b5d20 msgr2=0x7ffa680d10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.228+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 shutdown_connections 2024-09-13T16:26:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.228+0000 7ffa6db53700 1 -- 172.21.15.119:0/70683459 wait complete. 2024-09-13T16:26:16.244 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:16.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:16.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.360+0000 7fdd9d232700 1 Processor -- start 2024-09-13T16:26:16.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.360+0000 7fdd9d232700 1 -- start start 2024-09-13T16:26:16.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdd980ce5f0 0x7fdd980cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdd980d0fb0 0x7fdd980d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd980cd600 con 0x7fdd9815cae0 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd980ce1d0 con 0x7fdd980ce5f0 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd980cea60 con 0x7fdd980d0fb0 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdd980ce5f0 0x7fdd980cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdd980ce5f0 0x7fdd980cff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41894/0 (socket says 172.21.15.119:41894) 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 -- 172.21.15.119:0/3764243505 learned_addr learned my addr 172.21.15.119:0/3764243505 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9ca31700 1 --2- 172.21.15.119:0/3764243505 >> v2:172.21.15.145:3300/0 conn(0x7fdd980d0fb0 0x7fdd980d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 -- 172.21.15.119:0/3764243505 >> v2:172.21.15.145:3300/0 conn(0x7fdd980d0fb0 msgr2=0x7fdd980d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 --2- 172.21.15.119:0/3764243505 >> v2:172.21.15.145:3300/0 conn(0x7fdd980d0fb0 0x7fdd980d1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 -- 172.21.15.119:0/3764243505 >> v1:172.21.15.145:6789/0 conn(0x7fdd9815cae0 legacy=0x7fdd980d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 -- 172.21.15.119:0/3764243505 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd980d0dd0 con 0x7fdd980ce5f0 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd977fe700 1 --2- 172.21.15.119:0/3764243505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980ce5f0 0x7fdd980cff60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdd9808baf0 tx=0x7fdd88009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ac42472bb9382b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd96ffd700 1 -- 172.21.15.119:0/3764243505 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88011830 con 0x7fdd980ce5f0 2024-09-13T16:26:16.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd96ffd700 1 -- 172.21.15.119:0/3764243505 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd88011990 con 0x7fdd980ce5f0 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd96ffd700 1 -- 172.21.15.119:0/3764243505 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88011d40 con 0x7fdd980ce5f0 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- 172.21.15.119:0/3764243505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980ce5f0 msgr2=0x7fdd980cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 --2- 172.21.15.119:0/3764243505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980ce5f0 0x7fdd980cff60 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fdd9808baf0 tx=0x7fdd88009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- 172.21.15.119:0/3764243505 shutdown_connections 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 --2- 172.21.15.119:0/3764243505 >> v2:172.21.15.145:3300/0 conn(0x7fdd980d0fb0 0x7fdd980d1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 --2- 172.21.15.119:0/3764243505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980ce5f0 0x7fdd980cff60 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- 172.21.15.119:0/3764243505 >> 172.21.15.119:0/3764243505 conn(0x7fdd980d29f0 msgr2=0x7fdd980d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- 172.21.15.119:0/3764243505 shutdown_connections 2024-09-13T16:26:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.364+0000 7fdd9d232700 1 -- 172.21.15.119:0/3764243505 wait complete. 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 Processor -- start 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 -- start start 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd980ce5f0 0x7fdd981f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980d0fb0 0x7fdd981f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd9815cae0 0x7fdd980b29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd9815fd50 con 0x7fdd980d0fb0 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd981f6150 con 0x7fdd9815cae0 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd981f33e0 con 0x7fdd980ce5f0 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980d0fb0 0x7fdd981f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980d0fb0 0x7fdd981f2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41898/0 (socket says 172.21.15.119:41898) 2024-09-13T16:26:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 -- 172.21.15.119:0/320555323 learned_addr learned my addr 172.21.15.119:0/320555323 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9ca31700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd9815cae0 0x7fdd980b29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd97fff700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd980ce5f0 0x7fdd981f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 -- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd980ce5f0 msgr2=0x7fdd981f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd980ce5f0 0x7fdd981f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 -- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd9815cae0 msgr2=0x7fdd980b29c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd9815cae0 0x7fdd980b29c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 -- 172.21.15.119:0/320555323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009570 con 0x7fdd980d0fb0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd977fe700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980d0fb0 0x7fdd981f2ea0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdd88000c00 tx=0x7fdd88004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd94ff9700 1 -- 172.21.15.119:0/320555323 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88004590 con 0x7fdd980d0fb0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd94ff9700 1 -- 172.21.15.119:0/320555323 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8801e070 con 0x7fdd980d0fb0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd980b7460 con 0x7fdd980d0fb0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd94ff9700 1 -- 172.21.15.119:0/320555323 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd88003900 con 0x7fdd980d0fb0 2024-09-13T16:26:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd981f5b00 con 0x7fdd980d0fb0 2024-09-13T16:26:16.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.368+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd980744c0 con 0x7fdd980d0fb0 2024-09-13T16:26:16.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.372+0000 7fdd94ff9700 1 -- 172.21.15.119:0/320555323 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fdd88003a60 con 0x7fdd980d0fb0 2024-09-13T16:26:16.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.372+0000 7fdd94ff9700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdd84066730 0x7fdd84068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.372+0000 7fdd94ff9700 1 -- 172.21.15.119:0/320555323 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fdd88085380 con 0x7fdd980d0fb0 2024-09-13T16:26:16.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.372+0000 7fdd97fff700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdd84066730 0x7fdd84068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.372+0000 7fdd97fff700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdd84066730 0x7fdd84068bf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0018a0 tx=0x7fdd8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.380+0000 7fdd94ff9700 1 -- 172.21.15.119:0/320555323 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdd88052480 con 0x7fdd980d0fb0 2024-09-13T16:26:16.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.544+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdd9806bfc0 con 0x7fdd980d0fb0 2024-09-13T16:26:16.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.544+0000 7fdd94ff9700 1 -- 172.21.15.119:0/320555323 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdd88055aa0 con 0x7fdd980d0fb0 2024-09-13T16:26:16.549 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:16.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.544+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdd84066730 msgr2=0x7fdd84068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.544+0000 7fdd9d232700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdd84066730 0x7fdd84068bf0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c0018a0 tx=0x7fdd8c009040 comp rx=0 tx=0).stop 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980d0fb0 msgr2=0x7fdd981f2ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980d0fb0 0x7fdd981f2ea0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fdd88000c00 tx=0x7fdd88004770 comp rx=0 tx=0).stop 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 shutdown_connections 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdd84066730 0x7fdd84068bf0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdd9815cae0 0x7fdd980b29c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdd980d0fb0 0x7fdd981f2ea0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 --2- 172.21.15.119:0/320555323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdd980ce5f0 0x7fdd981f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 >> 172.21.15.119:0/320555323 conn(0x7fdd980d29f0 msgr2=0x7fdd9806c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:16.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 shutdown_connections 2024-09-13T16:26:16.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.548+0000 7fdd9d232700 1 -- 172.21.15.119:0/320555323 wait complete. 2024-09-13T16:26:16.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:16.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 Processor -- start 2024-09-13T16:26:16.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 -- start start 2024-09-13T16:26:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd6e815cad0 0x7fd6e80d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e80ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd6e80cf220 0x7fd6e80cf6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e80cfbf0 con 0x7fd6e80cf220 2024-09-13T16:26:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e80d05b0 con 0x7fd6e815cad0 2024-09-13T16:26:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.680+0000 7fd6eee48700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80d0870 con 0x7fd6e80ce0d0 2024-09-13T16:26:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd6e80cf220 0x7fd6e80cf6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd6e80cf220 0x7fd6e80cf6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41914/0 (socket says 172.21.15.119:41914) 2024-09-13T16:26:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 -- 172.21.15.119:0/637656881 learned_addr learned my addr 172.21.15.119:0/637656881 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ed645700 1 --2- 172.21.15.119:0/637656881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e80ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ede46700 1 --2- 172.21.15.119:0/637656881 >> v2:172.21.15.145:3300/0 conn(0x7fd6e815cad0 0x7fd6e80d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 -- 172.21.15.119:0/637656881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 msgr2=0x7fd6e80ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 --2- 172.21.15.119:0/637656881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e80ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 -- 172.21.15.119:0/637656881 >> v2:172.21.15.145:3300/0 conn(0x7fd6e815cad0 msgr2=0x7fd6e80d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 --2- 172.21.15.119:0/637656881 >> v2:172.21.15.145:3300/0 conn(0x7fd6e815cad0 0x7fd6e80d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 -- 172.21.15.119:0/637656881 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e80d0b50 con 0x7fd6e80cf220 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ee647700 1 --2- 172.21.15.119:0/637656881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80cf6b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80b55b0 tx=0x7fd6e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f4dbe54d4f63b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ece44700 1 -- 172.21.15.119:0/637656881 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000e070 con 0x7fd6e80cf220 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ece44700 1 -- 172.21.15.119:0/637656881 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e000b910 con 0x7fd6e80cf220 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6ece44700 1 -- 172.21.15.119:0/637656881 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e000bcc0 con 0x7fd6e80cf220 2024-09-13T16:26:16.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 -- 172.21.15.119:0/637656881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 msgr2=0x7fd6e80cf6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 --2- 172.21.15.119:0/637656881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80cf6b0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80b55b0 tx=0x7fd6e000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 -- 172.21.15.119:0/637656881 shutdown_connections 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 --2- 172.21.15.119:0/637656881 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80cf6b0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 --2- 172.21.15.119:0/637656881 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e80ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 --2- 172.21.15.119:0/637656881 >> v2:172.21.15.145:3300/0 conn(0x7fd6e815cad0 0x7fd6e80d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 -- 172.21.15.119:0/637656881 >> 172.21.15.119:0/637656881 conn(0x7fd6e808bf70 msgr2=0x7fd6e80d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 -- 172.21.15.119:0/637656881 shutdown_connections 2024-09-13T16:26:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.684+0000 7fd6eee48700 1 -- 172.21.15.119:0/637656881 wait complete. 2024-09-13T16:26:16.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 Processor -- start 2024-09-13T16:26:16.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 -- start start 2024-09-13T16:26:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e8156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6e815cad0 0x7fd6e80b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80b34a0 con 0x7fd6e80cf220 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80c7990 con 0x7fd6e815cad0 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e80c7c70 con 0x7fd6e80ce0d0 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80caca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41924/0 (socket says 172.21.15.119:41924) 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 -- 172.21.15.119:0/1807528105 learned_addr learned my addr 172.21.15.119:0/1807528105 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ee647700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6e815cad0 0x7fd6e80b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ede46700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e8156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 -- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 msgr2=0x7fd6e8156260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e8156260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 -- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6e815cad0 msgr2=0x7fd6e80b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6e815cad0 0x7fd6e80b29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 -- 172.21.15.119:0/1807528105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e0009b30 con 0x7fd6e80cf220 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6ed645700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80caca0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e960 tx=0x7fd6e400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6de7fc700 1 -- 172.21.15.119:0/1807528105 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e40096c0 con 0x7fd6e80cf220 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6de7fc700 1 -- 172.21.15.119:0/1807528105 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e400eeb0 con 0x7fd6e80cf220 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e80c7f50 con 0x7fd6e80cf220 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6de7fc700 1 -- 172.21.15.119:0/1807528105 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6e400ca50 con 0x7fd6e80cf220 2024-09-13T16:26:16.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.688+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e81567a0 con 0x7fd6e80cf220 2024-09-13T16:26:16.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.692+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e80c8470 con 0x7fd6e80cf220 2024-09-13T16:26:16.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.692+0000 7fd6de7fc700 1 -- 172.21.15.119:0/1807528105 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd6e400cc50 con 0x7fd6e80cf220 2024-09-13T16:26:16.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.692+0000 7fd6de7fc700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6cc066760 0x7fd6cc068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:16.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.692+0000 7fd6de7fc700 1 -- 172.21.15.119:0/1807528105 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fd6e40844a0 con 0x7fd6e80cf220 2024-09-13T16:26:16.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.692+0000 7fd6ede46700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6cc066760 0x7fd6cc068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:16.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.692+0000 7fd6ede46700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6cc066760 0x7fd6cc068c20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80b17b0 tx=0x7fd6d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:16.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.700+0000 7fd6de7fc700 1 -- 172.21.15.119:0/1807528105 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd6e80c8470 con 0x7fd6e80cf220 2024-09-13T16:26:16.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.868+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6e80c8470 con 0x7fd6e80cf220 2024-09-13T16:26:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.868+0000 7fd6de7fc700 1 -- 172.21.15.119:0/1807528105 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6e80c8470 con 0x7fd6e80cf220 2024-09-13T16:26:16.874 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:16.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6cc066760 msgr2=0x7fd6cc068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6cc066760 0x7fd6cc068c20 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6e80b17b0 tx=0x7fd6d8008040 comp rx=0 tx=0).stop 2024-09-13T16:26:16.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 msgr2=0x7fd6e80caca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:16.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80caca0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd6e400e960 tx=0x7fd6e400ec70 comp rx=0 tx=0).stop 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 shutdown_connections 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6cc066760 0x7fd6cc068c20 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6e815cad0 0x7fd6e80b29e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6e80cf220 0x7fd6e80caca0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 --2- 172.21.15.119:0/1807528105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6e80ce0d0 0x7fd6e8156260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 >> 172.21.15.119:0/1807528105 conn(0x7fd6e808bf70 msgr2=0x7fd6e815ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 shutdown_connections 2024-09-13T16:26:16.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:16.872+0000 7fd6eee48700 1 -- 172.21.15.119:0/1807528105 wait complete. 2024-09-13T16:26:16.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:17.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 Processor -- start 2024-09-13T16:26:17.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 -- start start 2024-09-13T16:26:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9a281594c0 0x7f9a28070bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9a280710f0 0x7f9a2806ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a2806b2a0 0x7f9a2815d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a28071580 con 0x7f9a281594c0 2024-09-13T16:26:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2806ba90 con 0x7f9a280710f0 2024-09-13T16:26:17.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a2d103700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a280b23f0 con 0x7f9a2806b2a0 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a27fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9a281594c0 0x7f9a28070bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a27fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9a281594c0 0x7f9a28070bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41928/0 (socket says 172.21.15.119:41928) 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.004+0000 7f9a27fff700 1 -- 172.21.15.119:0/2720615823 learned_addr learned my addr 172.21.15.119:0/2720615823 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a277fe700 1 --2- 172.21.15.119:0/2720615823 >> v2:172.21.15.145:3300/0 conn(0x7f9a280710f0 0x7f9a2806ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2c902700 1 --2- 172.21.15.119:0/2720615823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a2806b2a0 0x7f9a2815d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a27fff700 1 -- 172.21.15.119:0/2720615823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a2806b2a0 msgr2=0x7f9a2815d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a27fff700 1 --2- 172.21.15.119:0/2720615823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a2806b2a0 0x7f9a2815d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a27fff700 1 -- 172.21.15.119:0/2720615823 >> v2:172.21.15.145:3300/0 conn(0x7f9a280710f0 msgr2=0x7f9a2806ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a27fff700 1 --2- 172.21.15.119:0/2720615823 >> v2:172.21.15.145:3300/0 conn(0x7f9a280710f0 0x7f9a2806ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a27fff700 1 -- 172.21.15.119:0/2720615823 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a280b26d0 con 0x7f9a281594c0 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2c902700 1 --2- 172.21.15.119:0/2720615823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a2806b2a0 0x7f9a2815d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a27fff700 1 --2- 172.21.15.119:0/2720615823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a281594c0 0x7f9a28070bb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9a2806fec0 tx=0x7f9a18009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e090c9c701f3fb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a26ffd700 1 -- 172.21.15.119:0/2720615823 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18011830 con 0x7f9a281594c0 2024-09-13T16:26:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a26ffd700 1 -- 172.21.15.119:0/2720615823 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a18011990 con 0x7f9a281594c0 2024-09-13T16:26:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a26ffd700 1 -- 172.21.15.119:0/2720615823 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18011d20 con 0x7f9a281594c0 2024-09-13T16:26:17.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 -- 172.21.15.119:0/2720615823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a281594c0 msgr2=0x7f9a28070bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 --2- 172.21.15.119:0/2720615823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a281594c0 0x7f9a28070bb0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f9a2806fec0 tx=0x7f9a18009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 -- 172.21.15.119:0/2720615823 shutdown_connections 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 --2- 172.21.15.119:0/2720615823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a2806b2a0 0x7f9a2815d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 --2- 172.21.15.119:0/2720615823 >> v2:172.21.15.145:3300/0 conn(0x7f9a280710f0 0x7f9a2806ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 --2- 172.21.15.119:0/2720615823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a281594c0 0x7f9a28070bb0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 -- 172.21.15.119:0/2720615823 >> 172.21.15.119:0/2720615823 conn(0x7f9a280cef70 msgr2=0x7f9a280cf350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 -- 172.21.15.119:0/2720615823 shutdown_connections 2024-09-13T16:26:17.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.008+0000 7f9a2d103700 1 -- 172.21.15.119:0/2720615823 wait complete. 2024-09-13T16:26:17.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 Processor -- start 2024-09-13T16:26:17.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 -- start start 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a2806b2a0 0x7f9a2814db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a280710f0 0x7f9a2815a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a281594c0 0x7f9a2815d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2815cd60 con 0x7f9a2806b2a0 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2815c0e0 con 0x7f9a281594c0 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2815c3c0 con 0x7f9a280710f0 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a2806b2a0 0x7f9a2814db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a2806b2a0 0x7f9a2814db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41930/0 (socket says 172.21.15.119:41930) 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 -- 172.21.15.119:0/515405799 learned_addr learned my addr 172.21.15.119:0/515405799 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2c902700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a281594c0 0x7f9a2815d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a277fe700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a280710f0 0x7f9a2815a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 -- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a280710f0 msgr2=0x7f9a2815a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a280710f0 0x7f9a2815a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 -- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a281594c0 msgr2=0x7f9a2815d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a281594c0 0x7f9a2815d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 -- 172.21.15.119:0/515405799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a18009570 con 0x7f9a2806b2a0 2024-09-13T16:26:17.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a27fff700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a2806b2a0 0x7f9a2814db80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9a2804fc20 tx=0x7f9a18004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a24ff9700 1 -- 172.21.15.119:0/515405799 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a18004240 con 0x7f9a2806b2a0 2024-09-13T16:26:17.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a24ff9700 1 -- 172.21.15.119:0/515405799 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1801f070 con 0x7f9a2806b2a0 2024-09-13T16:26:17.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2815c6a0 con 0x7f9a2806b2a0 2024-09-13T16:26:17.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a24ff9700 1 -- 172.21.15.119:0/515405799 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a180045c0 con 0x7f9a2806b2a0 2024-09-13T16:26:17.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.012+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2814e0c0 con 0x7f9a2806b2a0 2024-09-13T16:26:17.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.016+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2806ad60 con 0x7f9a2806b2a0 2024-09-13T16:26:17.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.016+0000 7f9a24ff9700 1 -- 172.21.15.119:0/515405799 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9a18021030 con 0x7f9a2806b2a0 2024-09-13T16:26:17.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.016+0000 7f9a24ff9700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a14066730 0x7f9a14068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.016+0000 7f9a24ff9700 1 -- 172.21.15.119:0/515405799 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f9a180856c0 con 0x7f9a2806b2a0 2024-09-13T16:26:17.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.016+0000 7f9a277fe700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a14066730 0x7f9a14068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.016+0000 7f9a277fe700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a14066730 0x7f9a14068bf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a280b37e0 tx=0x7f9a1c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.024+0000 7f9a24ff9700 1 -- 172.21.15.119:0/515405799 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9a18052660 con 0x7f9a2806b2a0 2024-09-13T16:26:17.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.196+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9a280cc660 con 0x7f9a2806b2a0 2024-09-13T16:26:17.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a24ff9700 1 -- 172.21.15.119:0/515405799 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9a18055c80 con 0x7f9a2806b2a0 2024-09-13T16:26:17.204 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:17.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a14066730 msgr2=0x7f9a14068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a14066730 0x7f9a14068bf0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9a280b37e0 tx=0x7f9a1c009240 comp rx=0 tx=0).stop 2024-09-13T16:26:17.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a2806b2a0 msgr2=0x7f9a2814db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a2806b2a0 0x7f9a2814db80 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f9a2804fc20 tx=0x7f9a18004770 comp rx=0 tx=0).stop 2024-09-13T16:26:17.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 shutdown_connections 2024-09-13T16:26:17.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a14066730 0x7f9a14068bf0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a281594c0 0x7f9a2815d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a280710f0 0x7f9a2815a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 --2- 172.21.15.119:0/515405799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a2806b2a0 0x7f9a2814db80 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 >> 172.21.15.119:0/515405799 conn(0x7f9a280cef70 msgr2=0x7f9a280aff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:17.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 shutdown_connections 2024-09-13T16:26:17.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.200+0000 7f9a2d103700 1 -- 172.21.15.119:0/515405799 wait complete. 2024-09-13T16:26:17.220 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:17.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.332+0000 7f2938195700 1 Processor -- start 2024-09-13T16:26:17.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.332+0000 7f2938195700 1 -- start start 2024-09-13T16:26:17.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2930156ff0 0x7f29300aca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29301573d0 0x7f29300aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293015db70 0x7f293006a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293006e890 con 0x7f2930156ff0 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f293015d160 con 0x7f29301573d0 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29300af420 con 0x7f293015db70 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2930156ff0 0x7f29300aca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2930156ff0 0x7f29300aca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41946/0 (socket says 172.21.15.119:41946) 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 -- 172.21.15.119:0/3177937674 learned_addr learned my addr 172.21.15.119:0/3177937674 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2936732700 1 --2- 172.21.15.119:0/3177937674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293015db70 0x7f293006a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935730700 1 --2- 172.21.15.119:0/3177937674 >> v2:172.21.15.145:3300/0 conn(0x7f29301573d0 0x7f29300aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 -- 172.21.15.119:0/3177937674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293015db70 msgr2=0x7f293006a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 --2- 172.21.15.119:0/3177937674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293015db70 0x7f293006a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 -- 172.21.15.119:0/3177937674 >> v2:172.21.15.145:3300/0 conn(0x7f29301573d0 msgr2=0x7f29300aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:17.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 --2- 172.21.15.119:0/3177937674 >> v2:172.21.15.145:3300/0 conn(0x7f29301573d0 0x7f29300aeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 -- 172.21.15.119:0/3177937674 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29300ad740 con 0x7f2930156ff0 2024-09-13T16:26:17.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2935f31700 1 --2- 172.21.15.119:0/3177937674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2930156ff0 0x7f29300aca80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f29300ae6f0 tx=0x7f2920009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80884d38e727031 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2934f2f700 1 -- 172.21.15.119:0/3177937674 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920011830 con 0x7f2930156ff0 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2934f2f700 1 -- 172.21.15.119:0/3177937674 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2920011990 con 0x7f2930156ff0 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2934f2f700 1 -- 172.21.15.119:0/3177937674 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2920011d20 con 0x7f2930156ff0 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- 172.21.15.119:0/3177937674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2930156ff0 msgr2=0x7f29300aca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 --2- 172.21.15.119:0/3177937674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2930156ff0 0x7f29300aca80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f29300ae6f0 tx=0x7f2920009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- 172.21.15.119:0/3177937674 shutdown_connections 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 --2- 172.21.15.119:0/3177937674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f293015db70 0x7f293006a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 --2- 172.21.15.119:0/3177937674 >> v2:172.21.15.145:3300/0 conn(0x7f29301573d0 0x7f29300aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 --2- 172.21.15.119:0/3177937674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2930156ff0 0x7f29300aca80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- 172.21.15.119:0/3177937674 >> 172.21.15.119:0/3177937674 conn(0x7f293006b460 msgr2=0x7f293006b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- 172.21.15.119:0/3177937674 shutdown_connections 2024-09-13T16:26:17.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.336+0000 7f2938195700 1 -- 172.21.15.119:0/3177937674 wait complete. 2024-09-13T16:26:17.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 Processor -- start 2024-09-13T16:26:17.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 -- start start 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2930156ff0 0x7f29300afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29301573d0 0x7f293014d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f293015db70 0x7f293014de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293014e3b0 con 0x7f293015db70 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2930154660 con 0x7f2930156ff0 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2930154940 con 0x7f29301573d0 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f293015db70 0x7f293014de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f293015db70 0x7f293014de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41952/0 (socket says 172.21.15.119:41952) 2024-09-13T16:26:17.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 -- 172.21.15.119:0/3842422087 learned_addr learned my addr 172.21.15.119:0/3842422087 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2935730700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29301573d0 0x7f293014d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2935f31700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2930156ff0 0x7f29300afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 -- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29301573d0 msgr2=0x7f293014d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29301573d0 0x7f293014d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 -- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2930156ff0 msgr2=0x7f29300afef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2930156ff0 0x7f29300afef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 -- 172.21.15.119:0/3842422087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2920009570 con 0x7f293015db70 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2936732700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f293015db70 0x7f293014de70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f292800efc0 tx=0x7f292800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f29267fc700 1 -- 172.21.15.119:0/3842422087 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292800cad0 con 0x7f293015db70 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f29267fc700 1 -- 172.21.15.119:0/3842422087 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f292800f040 con 0x7f293015db70 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f29267fc700 1 -- 172.21.15.119:0/3842422087 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f292800ce70 con 0x7f293015db70 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2930154c20 con 0x7f293015db70 2024-09-13T16:26:17.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.340+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2930155420 con 0x7f293015db70 2024-09-13T16:26:17.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.344+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2930155140 con 0x7f293015db70 2024-09-13T16:26:17.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.344+0000 7f29267fc700 1 -- 172.21.15.119:0/3842422087 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2928014480 con 0x7f293015db70 2024-09-13T16:26:17.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.344+0000 7f29267fc700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2914066730 0x7f2914068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.344+0000 7f29267fc700 1 -- 172.21.15.119:0/3842422087 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f2928085260 con 0x7f293015db70 2024-09-13T16:26:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.344+0000 7f2935f31700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2914066730 0x7f2914068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.344+0000 7f2935f31700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2914066730 0x7f2914068bf0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2920000c00 tx=0x7f2920008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.352+0000 7f29267fc700 1 -- 172.21.15.119:0/3842422087 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f29280521e0 con 0x7f293015db70 2024-09-13T16:26:17.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2930155140 con 0x7f293015db70 2024-09-13T16:26:17.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f29267fc700 1 -- 172.21.15.119:0/3842422087 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2930155140 con 0x7f293015db70 2024-09-13T16:26:17.524 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2914066730 msgr2=0x7f2914068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2914066730 0x7f2914068bf0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2920000c00 tx=0x7f2920008030 comp rx=0 tx=0).stop 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f293015db70 msgr2=0x7f293014de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f293015db70 0x7f293014de70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f292800efc0 tx=0x7f292800c0c0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 shutdown_connections 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2914066730 0x7f2914068bf0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f293015db70 0x7f293014de70 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29301573d0 0x7f293014d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 --2- 172.21.15.119:0/3842422087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2930156ff0 0x7f29300afef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 >> 172.21.15.119:0/3842422087 conn(0x7f293006b460 msgr2=0x7f29300b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:17.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 shutdown_connections 2024-09-13T16:26:17.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.520+0000 7f2938195700 1 -- 172.21.15.119:0/3842422087 wait complete. 2024-09-13T16:26:17.544 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:17.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e28ff700 1 Processor -- start 2024-09-13T16:26:17.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e28ff700 1 -- start start 2024-09-13T16:26:17.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e28ff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f31dc06ef30 0x7f31dc15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e28ff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e28ff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31dc15ae30 con 0x7f31dc0cef90 2024-09-13T16:26:17.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e28ff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31dc06e490 con 0x7f31dc06ef30 2024-09-13T16:26:17.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e28ff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc159d20 con 0x7f31dc0cf370 2024-09-13T16:26:17.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f31dc06ef30 0x7f31dc15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e20fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f31dc06ef30 0x7f31dc15a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41964/0 (socket says 172.21.15.119:41964) 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e20fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:52130/0 (socket says 172.21.15.119:52130) 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 -- 172.21.15.119:0/2720929412 learned_addr learned my addr 172.21.15.119:0/2720929412 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 -- 172.21.15.119:0/2720929412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 msgr2=0x7f31dc0ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 --2- 172.21.15.119:0/2720929412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 -- 172.21.15.119:0/2720929412 >> v1:172.21.15.145:6789/0 conn(0x7f31dc0cef90 legacy=0x7f31dc06dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 -- 172.21.15.119:0/2720929412 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31dc06f410 con 0x7f31dc06ef30 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e08fb700 1 -- 172.21.15.119:0/2720929412 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177543644 0 0) 0x7f31d8001110 con 0x7f31dc0cef90 2024-09-13T16:26:17.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e10fc700 1 --2- 172.21.15.119:0/2720929412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc15a8f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009940 tx=0x7f31cc009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93034db82fbc0187 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.656+0000 7f31e08fb700 1 -- 172.21.15.119:0/2720929412 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc011830 con 0x7f31dc06ef30 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e08fb700 1 -- 172.21.15.119:0/2720929412 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31cc011990 con 0x7f31dc06ef30 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e08fb700 1 -- 172.21.15.119:0/2720929412 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31cc011d20 con 0x7f31dc06ef30 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- 172.21.15.119:0/2720929412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 msgr2=0x7f31dc15a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 --2- 172.21.15.119:0/2720929412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc15a8f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f31cc009940 tx=0x7f31cc009c50 comp rx=0 tx=0).stop 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- 172.21.15.119:0/2720929412 shutdown_connections 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 --2- 172.21.15.119:0/2720929412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 --2- 172.21.15.119:0/2720929412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc15a8f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- 172.21.15.119:0/2720929412 >> 172.21.15.119:0/2720929412 conn(0x7f31dc0b1d80 msgr2=0x7f31dc06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- 172.21.15.119:0/2720929412 shutdown_connections 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- 172.21.15.119:0/2720929412 wait complete. 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 Processor -- start 2024-09-13T16:26:17.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- start start 2024-09-13T16:26:17.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc0aad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31dc0cef90 0x7f31dc14c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ab440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc14ca40 con 0x7f31dc06ef30 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0ab980 con 0x7f31dc0cef90 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e28ff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31dc0abc20 con 0x7f31dc0cf370 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e18fd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc0aad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e18fd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc0aad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41974/0 (socket says 172.21.15.119:41974) 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e18fd700 1 -- 172.21.15.119:0/1101641224 learned_addr learned my addr 172.21.15.119:0/1101641224 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e10fc700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31dc0cef90 0x7f31dc14c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.660+0000 7f31e20fe700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ab440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e18fd700 1 -- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 msgr2=0x7f31dc0ab440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e18fd700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ab440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e18fd700 1 -- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31dc0cef90 msgr2=0x7f31dc14c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e18fd700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31dc0cef90 0x7f31dc14c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e18fd700 1 -- 172.21.15.119:0/1101641224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31cc009570 con 0x7f31dc06ef30 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e20fe700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e18fd700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc0aad50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f31d800c780 tx=0x7f31d800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31d27fc700 1 -- 172.21.15.119:0/1101641224 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d80052d0 con 0x7f31dc06ef30 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31d27fc700 1 -- 172.21.15.119:0/1101641224 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d8005450 con 0x7f31dc06ef30 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31dc0abf00 con 0x7f31dc06ef30 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31d27fc700 1 -- 172.21.15.119:0/1101641224 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8003900 con 0x7f31dc06ef30 2024-09-13T16:26:17.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31dc0ac4c0 con 0x7f31dc06ef30 2024-09-13T16:26:17.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31dc0ab290 con 0x7f31dc06ef30 2024-09-13T16:26:17.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31d27fc700 1 -- 172.21.15.119:0/1101641224 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f31d8003a60 con 0x7f31dc06ef30 2024-09-13T16:26:17.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31d27fc700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0066730 0x7f31c0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:17.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31d27fc700 1 -- 172.21.15.119:0/1101641224 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f31d80837e0 con 0x7f31dc06ef30 2024-09-13T16:26:17.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.664+0000 7f31e10fc700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0066730 0x7f31c0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:17.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.668+0000 7f31e10fc700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0066730 0x7f31c0068bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f31dc06f340 tx=0x7f31cc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:17.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.672+0000 7f31d27fc700 1 -- 172.21.15.119:0/1101641224 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f31d8050790 con 0x7f31dc06ef30 2024-09-13T16:26:17.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.852+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f31dc15ae30 con 0x7f31dc06ef30 2024-09-13T16:26:17.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.852+0000 7f31d27fc700 1 -- 172.21.15.119:0/1101641224 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f31dc15ae30 con 0x7f31dc06ef30 2024-09-13T16:26:17.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0066730 msgr2=0x7f31c0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0066730 0x7f31c0068bf0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f31dc06f340 tx=0x7f31cc008030 comp rx=0 tx=0).stop 2024-09-13T16:26:17.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 msgr2=0x7f31dc0aad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:17.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc0aad50 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f31d800c780 tx=0x7f31d800ca90 comp rx=0 tx=0).stop 2024-09-13T16:26:17.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 shutdown_connections 2024-09-13T16:26:17.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0066730 0x7f31c0068bf0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31dc0cf370 0x7f31dc0ab440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f31dc0cef90 0x7f31dc14c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 --2- 172.21.15.119:0/1101641224 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f31dc06ef30 0x7f31dc0aad50 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 >> 172.21.15.119:0/1101641224 conn(0x7f31dc0b1d80 msgr2=0x7f31dc15bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 shutdown_connections 2024-09-13T16:26:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:17.856+0000 7f31e28ff700 1 -- 172.21.15.119:0/1101641224 wait complete. 2024-09-13T16:26:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:17.872 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:26:17.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:17.872 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-09-13T16:26:17.887 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:17.887 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:26:17.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:17.888 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:26:17.892 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:26:17.893 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:17.900 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:26:17.900 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:17.949 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:18.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.060+0000 7f5ad2c1b700 1 Processor -- start 2024-09-13T16:26:18.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.060+0000 7f5ad2c1b700 1 -- start start 2024-09-13T16:26:18.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5acc1594f0 0x7f5acc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5acc06da70 0x7f5acc0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc0b3630 0x7f5acc15d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5acc06dee0 con 0x7f5acc06da70 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5acc0b2280 con 0x7f5acc1594f0 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0cc890 con 0x7f5acc0b3630 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5acc06da70 0x7f5acc0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5acc06da70 0x7f5acc0b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41984/0 (socket says 172.21.15.119:41984) 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 -- 172.21.15.119:0/942529384 learned_addr learned my addr 172.21.15.119:0/942529384 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad241a700 1 --2- 172.21.15.119:0/942529384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc0b3630 0x7f5acc15d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1c19700 1 --2- 172.21.15.119:0/942529384 >> v2:172.21.15.145:3300/0 conn(0x7f5acc1594f0 0x7f5acc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 -- 172.21.15.119:0/942529384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc0b3630 msgr2=0x7f5acc15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 --2- 172.21.15.119:0/942529384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc0b3630 0x7f5acc15d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 -- 172.21.15.119:0/942529384 >> v2:172.21.15.145:3300/0 conn(0x7f5acc1594f0 msgr2=0x7f5acc06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:18.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 --2- 172.21.15.119:0/942529384 >> v2:172.21.15.145:3300/0 conn(0x7f5acc1594f0 0x7f5acc06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 -- 172.21.15.119:0/942529384 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc0b1970 con 0x7f5acc06da70 2024-09-13T16:26:18.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1c19700 1 --2- 172.21.15.119:0/942529384 >> v2:172.21.15.145:3300/0 conn(0x7f5acc1594f0 0x7f5acc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:18.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad1418700 1 --2- 172.21.15.119:0/942529384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc06da70 0x7f5acc0b30f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0b1b60 tx=0x7f5abc009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a840f1025e1c6a5a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad0c17700 1 -- 172.21.15.119:0/942529384 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc004230 con 0x7f5acc06da70 2024-09-13T16:26:18.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad0c17700 1 -- 172.21.15.119:0/942529384 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc004390 con 0x7f5acc06da70 2024-09-13T16:26:18.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad0c17700 1 -- 172.21.15.119:0/942529384 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5abc0036c0 con 0x7f5acc06da70 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/942529384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc06da70 msgr2=0x7f5acc0b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/942529384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc06da70 0x7f5acc0b30f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f5acc0b1b60 tx=0x7f5abc009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/942529384 shutdown_connections 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/942529384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc0b3630 0x7f5acc15d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/942529384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc06da70 0x7f5acc0b30f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/942529384 >> v2:172.21.15.145:3300/0 conn(0x7f5acc1594f0 0x7f5acc06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/942529384 >> 172.21.15.119:0/942529384 conn(0x7f5acc06a350 msgr2=0x7f5acc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:18.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/942529384 shutdown_connections 2024-09-13T16:26:18.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.064+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/942529384 wait complete. 2024-09-13T16:26:18.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 Processor -- start 2024-09-13T16:26:18.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 -- start start 2024-09-13T16:26:18.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc06da70 0x7f5acc14fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5acc0b3630 0x7f5acc0a9680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc1594f0 0x7f5acc06ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc150af0 con 0x7f5acc1594f0 2024-09-13T16:26:18.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc14ee80 con 0x7f5acc0b3630 2024-09-13T16:26:18.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad2c1b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5acc0a84b0 con 0x7f5acc06da70 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc1594f0 0x7f5acc06ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc1594f0 0x7f5acc06ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41998/0 (socket says 172.21.15.119:41998) 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 -- 172.21.15.119:0/1162919021 learned_addr learned my addr 172.21.15.119:0/1162919021 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad1c19700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc06da70 0x7f5acc14fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad1418700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5acc0b3630 0x7f5acc0a9680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 -- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc06da70 msgr2=0x7f5acc14fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc06da70 0x7f5acc14fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 -- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5acc0b3630 msgr2=0x7f5acc0a9680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5acc0b3630 0x7f5acc0a9680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 -- 172.21.15.119:0/1162919021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc009570 con 0x7f5acc1594f0 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ad241a700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc1594f0 0x7f5acc06ce80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4007e60 tx=0x7f5ac400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.068+0000 7f5ac27fc700 1 -- 172.21.15.119:0/1162919021 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac400ed30 con 0x7f5acc1594f0 2024-09-13T16:26:18.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.072+0000 7f5ac27fc700 1 -- 172.21.15.119:0/1162919021 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac4014420 con 0x7f5acc1594f0 2024-09-13T16:26:18.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.072+0000 7f5ac27fc700 1 -- 172.21.15.119:0/1162919021 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac40147d0 con 0x7f5acc1594f0 2024-09-13T16:26:18.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.072+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc0a8790 con 0x7f5acc1594f0 2024-09-13T16:26:18.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.072+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc14f690 con 0x7f5acc1594f0 2024-09-13T16:26:18.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.072+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc14eff0 con 0x7f5acc1594f0 2024-09-13T16:26:18.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.072+0000 7f5ac27fc700 1 -- 172.21.15.119:0/1162919021 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5ac4014930 con 0x7f5acc1594f0 2024-09-13T16:26:18.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.076+0000 7f5ac27fc700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ab0066730 0x7f5ab0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.076+0000 7f5ac27fc700 1 -- 172.21.15.119:0/1162919021 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5ac4084a90 con 0x7f5acc1594f0 2024-09-13T16:26:18.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.076+0000 7f5ad1c19700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ab0066730 0x7f5ab0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.076+0000 7f5ad1c19700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ab0066730 0x7f5ab0068bf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800ab60 tx=0x7f5ac8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.080+0000 7f5ac27fc700 1 -- 172.21.15.119:0/1162919021 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5acc14eff0 con 0x7f5acc1594f0 2024-09-13T16:26:18.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.256+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5acc14eff0 con 0x7f5acc1594f0 2024-09-13T16:26:18.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ac27fc700 1 -- 172.21.15.119:0/1162919021 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f5acc14eff0 con 0x7f5acc1594f0 2024-09-13T16:26:18.264 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:18.264 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ab0066730 msgr2=0x7f5ab0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ab0066730 0x7f5ab0068bf0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5ac800ab60 tx=0x7f5ac8008040 comp rx=0 tx=0).stop 2024-09-13T16:26:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc1594f0 msgr2=0x7f5acc06ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc1594f0 0x7f5acc06ce80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5ac4007e60 tx=0x7f5ac400e3c0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 shutdown_connections 2024-09-13T16:26:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ab0066730 0x7f5ab0068bf0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5acc1594f0 0x7f5acc06ce80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5acc0b3630 0x7f5acc0a9680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 --2- 172.21.15.119:0/1162919021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5acc06da70 0x7f5acc14fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 >> 172.21.15.119:0/1162919021 conn(0x7f5acc06a350 msgr2=0x7f5acc06fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:18.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 shutdown_connections 2024-09-13T16:26:18.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.260+0000 7f5ad2c1b700 1 -- 172.21.15.119:0/1162919021 wait complete. 2024-09-13T16:26:18.280 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:18.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:18.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e508f700 1 Processor -- start 2024-09-13T16:26:18.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e508f700 1 -- start start 2024-09-13T16:26:18.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e508f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f49e00b29e0 0x7f49e006b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e508f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f49e006b8a0 0x7f49e006d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e508f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e00b2e50 con 0x7f49e015a760 2024-09-13T16:26:18.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e508f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e006bca0 con 0x7f49e00b29e0 2024-09-13T16:26:18.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e508f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e015ab40 con 0x7f49e006b8a0 2024-09-13T16:26:18.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49df7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f49e00b29e0 0x7f49e006b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49df7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f49e00b29e0 0x7f49e006b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42014/0 (socket says 172.21.15.119:42014) 2024-09-13T16:26:18.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49df7fe700 1 -- 172.21.15.119:0/3973178692 learned_addr learned my addr 172.21.15.119:0/3973178692 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:18.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e488e700 1 --2- 172.21.15.119:0/3973178692 >> v2:172.21.15.145:3300/0 conn(0x7f49e006b8a0 0x7f49e006d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e488e700 1 -- 172.21.15.119:0/3973178692 >> v2:172.21.15.119:3300/0 conn(0x7f49e00b29e0 msgr2=0x7f49e006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e488e700 1 --2- 172.21.15.119:0/3973178692 >> v2:172.21.15.119:3300/0 conn(0x7f49e00b29e0 0x7f49e006b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e488e700 1 -- 172.21.15.119:0/3973178692 >> v1:172.21.15.145:6789/0 conn(0x7f49e015a760 legacy=0x7f49e00acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e488e700 1 -- 172.21.15.119:0/3973178692 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e00ad760 con 0x7f49e006b8a0 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49df7fe700 1 --2- 172.21.15.119:0/3973178692 >> v2:172.21.15.119:3300/0 conn(0x7f49e00b29e0 0x7f49e006b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49e488e700 1 --2- 172.21.15.119:0/3973178692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e006b8a0 0x7f49e006d860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f49e00b42b0 tx=0x7f49d800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=987a6c72b8e55f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49deffd700 1 -- 172.21.15.119:0/3973178692 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d800e070 con 0x7f49e006b8a0 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49deffd700 1 -- 172.21.15.119:0/3973178692 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d800bb20 con 0x7f49e006b8a0 2024-09-13T16:26:18.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.392+0000 7f49deffd700 1 -- 172.21.15.119:0/3973178692 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8012430 con 0x7f49e006b8a0 2024-09-13T16:26:18.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 -- 172.21.15.119:0/3973178692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e006b8a0 msgr2=0x7f49e006d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 --2- 172.21.15.119:0/3973178692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e006b8a0 0x7f49e006d860 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f49e00b42b0 tx=0x7f49d800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:18.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 -- 172.21.15.119:0/3973178692 shutdown_connections 2024-09-13T16:26:18.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 --2- 172.21.15.119:0/3973178692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e006b8a0 0x7f49e006d860 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 --2- 172.21.15.119:0/3973178692 >> v2:172.21.15.119:3300/0 conn(0x7f49e00b29e0 0x7f49e006b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 -- 172.21.15.119:0/3973178692 >> 172.21.15.119:0/3973178692 conn(0x7f49e00cee60 msgr2=0x7f49e00cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:18.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 -- 172.21.15.119:0/3973178692 shutdown_connections 2024-09-13T16:26:18.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 -- 172.21.15.119:0/3973178692 wait complete. 2024-09-13T16:26:18.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 Processor -- start 2024-09-13T16:26:18.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 -- start start 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e006b8a0 0x7f49e009a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.396+0000 7f49e508f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e00b29e0 0x7f49e00cb290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e508f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e015a760 0x7f49e0140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e508f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e009ae90 con 0x7f49e015a760 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e508f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e009b0c0 con 0x7f49e00b29e0 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e508f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e009b3a0 con 0x7f49e006b8a0 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e015a760 0x7f49e0140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e015a760 0x7f49e0140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42030/0 (socket says 172.21.15.119:42030) 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 -- 172.21.15.119:0/945504249 learned_addr learned my addr 172.21.15.119:0/945504249 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49df7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e00b29e0 0x7f49e00cb290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49dffff700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e006b8a0 0x7f49e009a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 -- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e006b8a0 msgr2=0x7f49e009a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e006b8a0 0x7f49e009a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 -- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e00b29e0 msgr2=0x7f49e00cb290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e00b29e0 0x7f49e00cb290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 -- 172.21.15.119:0/945504249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49c8009570 con 0x7f49e015a760 2024-09-13T16:26:18.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e488e700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e015a760 0x7f49e0140780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f49d8014040 tx=0x7f49d8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49dcff9700 1 -- 172.21.15.119:0/945504249 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8012430 con 0x7f49e015a760 2024-09-13T16:26:18.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49dcff9700 1 -- 172.21.15.119:0/945504249 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d8020070 con 0x7f49e015a760 2024-09-13T16:26:18.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d8009a70 con 0x7f49e015a760 2024-09-13T16:26:18.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49dcff9700 1 -- 172.21.15.119:0/945504249 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8004280 con 0x7f49e015a760 2024-09-13T16:26:18.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e009b960 con 0x7f49e015a760 2024-09-13T16:26:18.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e006b360 con 0x7f49e015a760 2024-09-13T16:26:18.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.400+0000 7f49dcff9700 1 -- 172.21.15.119:0/945504249 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f49d80043e0 con 0x7f49e015a760 2024-09-13T16:26:18.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.404+0000 7f49dcff9700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49c0066730 0x7f49c0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.404+0000 7f49dcff9700 1 -- 172.21.15.119:0/945504249 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f49d8022070 con 0x7f49e015a760 2024-09-13T16:26:18.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.404+0000 7f49dffff700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49c0066730 0x7f49c0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.404+0000 7f49dffff700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49c0066730 0x7f49c0068bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f49d0000f60 tx=0x7f49d0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.408+0000 7f49dcff9700 1 -- 172.21.15.119:0/945504249 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f49d8053530 con 0x7f49e015a760 2024-09-13T16:26:18.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49e01588e0 con 0x7f49e015a760 2024-09-13T16:26:18.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49dcff9700 1 -- 172.21.15.119:0/945504249 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7f49d801ae60 con 0x7f49e015a760 2024-09-13T16:26:18.584 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49c0066730 msgr2=0x7f49c0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49c0066730 0x7f49c0068bf0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f49d0000f60 tx=0x7f49d0009040 comp rx=0 tx=0).stop 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e015a760 msgr2=0x7f49e0140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e015a760 0x7f49e0140780 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f49d8014040 tx=0x7f49d8004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 shutdown_connections 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49c0066730 0x7f49c0068bf0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e015a760 0x7f49e0140780 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e00b29e0 0x7f49e00cb290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 --2- 172.21.15.119:0/945504249 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e006b8a0 0x7f49e009a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 >> 172.21.15.119:0/945504249 conn(0x7f49e00cee60 msgr2=0x7f49e00b01f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:18.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 shutdown_connections 2024-09-13T16:26:18.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.580+0000 7f49e508f700 1 -- 172.21.15.119:0/945504249 wait complete. 2024-09-13T16:26:18.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:18.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 Processor -- start 2024-09-13T16:26:18.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- start start 2024-09-13T16:26:18.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98e406ba70 0x7f98e41652c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e40b2e50 con 0x7f98e415a760 2024-09-13T16:26:18.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98e415ab40 con 0x7f98e40b29e0 2024-09-13T16:26:18.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98e40ad760 con 0x7f98e406ba70 2024-09-13T16:26:18.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e8a51700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98e406ba70 0x7f98e41652c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f98e415a760 0x7f98e40acf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59410/0 (socket says 172.21.15.119:59410) 2024-09-13T16:26:18.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e3fff700 1 -- 172.21.15.119:0/3455508756 learned_addr learned my addr 172.21.15.119:0/3455508756 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e37fe700 1 -- 172.21.15.119:0/3455508756 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354048501 0 0) 0x7f98e40b2e50 con 0x7f98e415a760 2024-09-13T16:26:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e37fe700 1 -- 172.21.15.119:0/3455508756 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d00035d0 con 0x7f98e415a760 2024-09-13T16:26:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e8a51700 1 -- 172.21.15.119:0/3455508756 >> v1:172.21.15.145:6789/0 conn(0x7f98e40b29e0 legacy=0x7f98e406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e8a51700 1 -- 172.21.15.119:0/3455508756 >> v1:172.21.15.119:6789/0 conn(0x7f98e415a760 legacy=0x7f98e40acf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e8a51700 1 -- 172.21.15.119:0/3455508756 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e40ada40 con 0x7f98e406ba70 2024-09-13T16:26:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e8a51700 1 --2- 172.21.15.119:0/3455508756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e406ba70 0x7f98e41652c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009d90 tx=0x7f98dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d84c45e30dd2caf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e37fe700 1 -- 172.21.15.119:0/3455508756 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc00e070 con 0x7f98e406ba70 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e37fe700 1 -- 172.21.15.119:0/3455508756 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98dc00bb20 con 0x7f98e406ba70 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e37fe700 1 -- 172.21.15.119:0/3455508756 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98dc012430 con 0x7f98e406ba70 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- 172.21.15.119:0/3455508756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e406ba70 msgr2=0x7f98e41652c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 --2- 172.21.15.119:0/3455508756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e406ba70 0x7f98e41652c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f98dc009d90 tx=0x7f98dc00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- 172.21.15.119:0/3455508756 shutdown_connections 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 --2- 172.21.15.119:0/3455508756 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e406ba70 0x7f98e41652c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- 172.21.15.119:0/3455508756 >> 172.21.15.119:0/3455508756 conn(0x7f98e40cee60 msgr2=0x7f98e40cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- 172.21.15.119:0/3455508756 shutdown_connections 2024-09-13T16:26:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.720+0000 7f98e9252700 1 -- 172.21.15.119:0/3455508756 wait complete. 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 Processor -- start 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 -- start start 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e406ba70 0x7f98e40cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e40b29e0 0x7f98e41f3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e415a760 0x7f98e40cb020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40cdb40 con 0x7f98e40b29e0 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40cd210 con 0x7f98e406ba70 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e40cd4f0 con 0x7f98e415a760 2024-09-13T16:26:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98db7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e40b29e0 0x7f98e41f3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98db7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e40b29e0 0x7f98e41f3bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42042/0 (socket says 172.21.15.119:42042) 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98db7fe700 1 -- 172.21.15.119:0/3941605953 learned_addr learned my addr 172.21.15.119:0/3941605953 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e3fff700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e406ba70 0x7f98e40cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e8a51700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e415a760 0x7f98e40cb020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e3fff700 1 -- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e415a760 msgr2=0x7f98e40cb020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e3fff700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e415a760 0x7f98e40cb020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e3fff700 1 -- 172.21.15.119:0/3941605953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e40b29e0 msgr2=0x7f98e41f3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e3fff700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e40b29e0 0x7f98e41f3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e3fff700 1 -- 172.21.15.119:0/3941605953 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98dc009a70 con 0x7f98e406ba70 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98db7fe700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e40b29e0 0x7f98e41f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e3fff700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e406ba70 0x7f98e40cdfd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98d400c410 tx=0x7f98d400c720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e17fa700 1 -- 172.21.15.119:0/3941605953 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d4005d90 con 0x7f98e406ba70 2024-09-13T16:26:18.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e17fa700 1 -- 172.21.15.119:0/3941605953 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98d400ce20 con 0x7f98e406ba70 2024-09-13T16:26:18.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98e40cd7d0 con 0x7f98e406ba70 2024-09-13T16:26:18.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e17fa700 1 -- 172.21.15.119:0/3941605953 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98d40072d0 con 0x7f98e406ba70 2024-09-13T16:26:18.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.724+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98e41f3650 con 0x7f98e406ba70 2024-09-13T16:26:18.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.728+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98e41f3870 con 0x7f98e406ba70 2024-09-13T16:26:18.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.728+0000 7f98e17fa700 1 -- 172.21.15.119:0/3941605953 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f98d4007430 con 0x7f98e406ba70 2024-09-13T16:26:18.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.728+0000 7f98e17fa700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98c4066760 0x7f98c4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:18.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.728+0000 7f98e17fa700 1 -- 172.21.15.119:0/3941605953 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f98d4083800 con 0x7f98e406ba70 2024-09-13T16:26:18.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.728+0000 7f98db7fe700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98c4066760 0x7f98c4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:18.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.732+0000 7f98db7fe700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98c4066760 0x7f98c4068c20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f98e414e360 tx=0x7f98cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:18.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.736+0000 7f98e17fa700 1 -- 172.21.15.119:0/3941605953 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f98e41f3870 con 0x7f98e406ba70 2024-09-13T16:26:18.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.912+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98e41f3870 con 0x7f98e406ba70 2024-09-13T16:26:18.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.912+0000 7f98e17fa700 1 -- 172.21.15.119:0/3941605953 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f98e41f3870 con 0x7f98e406ba70 2024-09-13T16:26:18.917 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.912+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98c4066760 msgr2=0x7f98c4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.912+0000 7f98e9252700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98c4066760 0x7f98c4068c20 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f98e414e360 tx=0x7f98cc009040 comp rx=0 tx=0).stop 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.912+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e406ba70 msgr2=0x7f98e40cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e406ba70 0x7f98e40cdfd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f98d400c410 tx=0x7f98d400c720 comp rx=0 tx=0).stop 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 shutdown_connections 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98c4066760 0x7f98c4068c20 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98e415a760 0x7f98e40cb020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98e40b29e0 0x7f98e41f3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 --2- 172.21.15.119:0/3941605953 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98e406ba70 0x7f98e40cdfd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 >> 172.21.15.119:0/3941605953 conn(0x7f98e40cee60 msgr2=0x7f98e406d760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 shutdown_connections 2024-09-13T16:26:18.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:18.916+0000 7f98e9252700 1 -- 172.21.15.119:0/3941605953 wait complete. 2024-09-13T16:26:18.932 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:19.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.048+0000 7fe93d2ca700 1 Processor -- start 2024-09-13T16:26:19.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.048+0000 7fe93d2ca700 1 -- start start 2024-09-13T16:26:19.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe93d2ca700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe9381599c0 0x7fe93806b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe93d2ca700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93806b8a0 0x7fe93806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe93d2ca700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9380710b0 con 0x7fe9381595e0 2024-09-13T16:26:19.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe93d2ca700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe93806c3f0 con 0x7fe9381599c0 2024-09-13T16:26:19.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe93d2ca700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe93806bd50 con 0x7fe93806b8a0 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe937fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe9381595e0 0x7fe93806e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59412/0 (socket says 172.21.15.119:59412) 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe937fff700 1 -- 172.21.15.119:0/695621406 learned_addr learned my addr 172.21.15.119:0/695621406 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe92ffff700 1 --2- 172.21.15.119:0/695621406 >> v2:172.21.15.145:3300/0 conn(0x7fe9381599c0 0x7fe93806b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe93cac9700 1 --2- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93806b8a0 0x7fe93806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe9377fe700 1 -- 172.21.15.119:0/695621406 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118248463 0 0) 0x7fe9380710b0 con 0x7fe9381595e0 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe9377fe700 1 -- 172.21.15.119:0/695621406 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9240035d0 con 0x7fe9381595e0 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe92ffff700 1 -- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93806b8a0 msgr2=0x7fe93806c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe92ffff700 1 --2- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93806b8a0 0x7fe93806c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe92ffff700 1 -- 172.21.15.119:0/695621406 >> v1:172.21.15.119:6789/0 conn(0x7fe9381595e0 legacy=0x7fe93806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe92ffff700 1 -- 172.21.15.119:0/695621406 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe938070810 con 0x7fe9381599c0 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe92ffff700 1 --2- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9381599c0 0x7fe93806b360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe920009850 tx=0x7fe920009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=656915c2968481fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe9377fe700 1 -- 172.21.15.119:0/695621406 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920011830 con 0x7fe9381599c0 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe9377fe700 1 -- 172.21.15.119:0/695621406 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe920011990 con 0x7fe9381599c0 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.052+0000 7fe9377fe700 1 -- 172.21.15.119:0/695621406 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920011d40 con 0x7fe9381599c0 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 -- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9381599c0 msgr2=0x7fe93806b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9381599c0 0x7fe93806b360 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fe920009850 tx=0x7fe920009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 -- 172.21.15.119:0/695621406 shutdown_connections 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe93806b8a0 0x7fe93806c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/695621406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9381599c0 0x7fe93806b360 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 -- 172.21.15.119:0/695621406 >> 172.21.15.119:0/695621406 conn(0x7fe9380cb8f0 msgr2=0x7fe9380cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 -- 172.21.15.119:0/695621406 shutdown_connections 2024-09-13T16:26:19.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 -- 172.21.15.119:0/695621406 wait complete. 2024-09-13T16:26:19.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 Processor -- start 2024-09-13T16:26:19.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.056+0000 7fe93d2ca700 1 -- start start 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93806b8a0 0x7fe938147610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9381595e0 0x7fe938147b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9381599c0 0x7fe938157e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93806ee20 con 0x7fe9381595e0 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe938155ff0 con 0x7fe93806b8a0 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe938156290 con 0x7fe9381599c0 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9381595e0 0x7fe938147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9381595e0 0x7fe938147b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42050/0 (socket says 172.21.15.119:42050) 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 -- 172.21.15.119:0/729234027 learned_addr learned my addr 172.21.15.119:0/729234027 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:19.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe937fff700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93806b8a0 0x7fe938147610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93cac9700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9381599c0 0x7fe938157e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 -- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9381599c0 msgr2=0x7fe938157e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9381599c0 0x7fe938157e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 -- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93806b8a0 msgr2=0x7fe938147610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93806b8a0 0x7fe938147610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 -- 172.21.15.119:0/729234027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9200094b0 con 0x7fe9381595e0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe937fff700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93806b8a0 0x7fe938147610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe92ffff700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9381595e0 0x7fe938147b50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe920014040 tx=0x7fe920004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe9357fa700 1 -- 172.21.15.119:0/729234027 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9200036c0 con 0x7fe9381595e0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93815a9b0 con 0x7fe9381595e0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe9357fa700 1 -- 172.21.15.119:0/729234027 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe92001e070 con 0x7fe9381595e0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93815af70 con 0x7fe9381595e0 2024-09-13T16:26:19.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.060+0000 7fe9357fa700 1 -- 172.21.15.119:0/729234027 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe920003a40 con 0x7fe9381595e0 2024-09-13T16:26:19.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.064+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93815b180 con 0x7fe9381595e0 2024-09-13T16:26:19.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.064+0000 7fe9357fa700 1 -- 172.21.15.119:0/729234027 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe920003ba0 con 0x7fe9381595e0 2024-09-13T16:26:19.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.064+0000 7fe9357fa700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe918066760 0x7fe918068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.064+0000 7fe9357fa700 1 -- 172.21.15.119:0/729234027 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe920020070 con 0x7fe9381595e0 2024-09-13T16:26:19.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.064+0000 7fe937fff700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe918066760 0x7fe918068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.064+0000 7fe937fff700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe918066760 0x7fe918068c20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe9280011e0 tx=0x7fe928009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.072+0000 7fe9357fa700 1 -- 172.21.15.119:0/729234027 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe920052990 con 0x7fe9381595e0 2024-09-13T16:26:19.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.232+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe93806b360 con 0x7fe9381595e0 2024-09-13T16:26:19.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.232+0000 7fe9357fa700 1 -- 172.21.15.119:0/729234027 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe920015020 con 0x7fe9381595e0 2024-09-13T16:26:19.237 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe918066760 msgr2=0x7fe918068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe918066760 0x7fe918068c20 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fe9280011e0 tx=0x7fe928009040 comp rx=0 tx=0).stop 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9381595e0 msgr2=0x7fe938147b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9381595e0 0x7fe938147b50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe920014040 tx=0x7fe920004770 comp rx=0 tx=0).stop 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 shutdown_connections 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe918066760 0x7fe918068c20 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9381599c0 0x7fe938157e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9381595e0 0x7fe938147b50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 --2- 172.21.15.119:0/729234027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe93806b8a0 0x7fe938147610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 >> 172.21.15.119:0/729234027 conn(0x7fe9380cb8f0 msgr2=0x7fe9380b0110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:19.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 shutdown_connections 2024-09-13T16:26:19.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.236+0000 7fe93d2ca700 1 -- 172.21.15.119:0/729234027 wait complete. 2024-09-13T16:26:19.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:19.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.360+0000 7f1978f4e700 1 Processor -- start 2024-09-13T16:26:19.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.360+0000 7f1978f4e700 1 -- start start 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f19740ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f19740cff60 0x7f19740d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19740cd600 con 0x7f19740d0fb0 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19740ce1d0 con 0x7f19740cff60 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19740d03a0 con 0x7f197415cae0 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1973fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f19740d0fb0 0x7f19740b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59414/0 (socket says 172.21.15.119:59414) 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1973fff700 1 -- 172.21.15.119:0/842987877 learned_addr learned my addr 172.21.15.119:0/842987877 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1972ffd700 1 --2- 172.21.15.119:0/842987877 >> v2:172.21.15.145:3300/0 conn(0x7f19740cff60 0x7f19740d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f19737fe700 1 --2- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f19740ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f19727fc700 1 -- 172.21.15.119:0/842987877 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128892496 0 0) 0x7f19740cd600 con 0x7f19740d0fb0 2024-09-13T16:26:19.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f19727fc700 1 -- 172.21.15.119:0/842987877 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f195c0035d0 con 0x7f19740d0fb0 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1972ffd700 1 -- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 msgr2=0x7f19740ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1972ffd700 1 --2- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f19740ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1972ffd700 1 -- 172.21.15.119:0/842987877 >> v1:172.21.15.119:6789/0 conn(0x7f19740d0fb0 legacy=0x7f19740b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1972ffd700 1 -- 172.21.15.119:0/842987877 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19740d0dd0 con 0x7f19740cff60 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1972ffd700 1 --2- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740d05c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1964009d90 tx=0x7f196400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5be88a458ffaed76 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f19727fc700 1 -- 172.21.15.119:0/842987877 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196400e070 con 0x7f19740cff60 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f19727fc700 1 -- 172.21.15.119:0/842987877 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196400bb20 con 0x7f19740cff60 2024-09-13T16:26:19.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f19727fc700 1 -- 172.21.15.119:0/842987877 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f196400be90 con 0x7f19740cff60 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 msgr2=0x7f19740d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 --2- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740d05c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1964009d90 tx=0x7f196400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- 172.21.15.119:0/842987877 shutdown_connections 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 --2- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740d05c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 --2- 172.21.15.119:0/842987877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f19740ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- 172.21.15.119:0/842987877 >> 172.21.15.119:0/842987877 conn(0x7f19740d29f0 msgr2=0x7f19740d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- 172.21.15.119:0/842987877 shutdown_connections 2024-09-13T16:26:19.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.364+0000 7f1978f4e700 1 -- 172.21.15.119:0/842987877 wait complete. 2024-09-13T16:26:19.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 Processor -- start 2024-09-13T16:26:19.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 -- start start 2024-09-13T16:26:19.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740b3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19740d0fb0 0x7f19740b7c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f1974153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19740b86f0 con 0x7f19740d0fb0 2024-09-13T16:26:19.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19740c9bf0 con 0x7f19740cff60 2024-09-13T16:26:19.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1978f4e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19740c9e90 con 0x7f197415cae0 2024-09-13T16:26:19.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19740d0fb0 0x7f19740b7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19740d0fb0 0x7f19740b7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42064/0 (socket says 172.21.15.119:42064) 2024-09-13T16:26:19.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 -- 172.21.15.119:0/3168108365 learned_addr learned my addr 172.21.15.119:0/3168108365 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:19.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f19737fe700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740b3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1973fff700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f1974153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 -- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 msgr2=0x7f1974153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f1974153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 -- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 msgr2=0x7f19740b3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740b3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 -- 172.21.15.119:0/3168108365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1964009a70 con 0x7f19740d0fb0 2024-09-13T16:26:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f19737fe700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:19.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.368+0000 7f1972ffd700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19740d0fb0 0x7f19740b7c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f196400b370 tx=0x7f1964004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f195bfff700 1 -- 172.21.15.119:0/3168108365 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1964007000 con 0x7f19740d0fb0 2024-09-13T16:26:19.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f195bfff700 1 -- 172.21.15.119:0/3168108365 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1964020070 con 0x7f19740d0fb0 2024-09-13T16:26:19.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19740ca170 con 0x7f19740d0fb0 2024-09-13T16:26:19.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f195bfff700 1 -- 172.21.15.119:0/3168108365 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19640098f0 con 0x7f19740d0fb0 2024-09-13T16:26:19.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19741543e0 con 0x7f19740d0fb0 2024-09-13T16:26:19.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19740ca690 con 0x7f19740d0fb0 2024-09-13T16:26:19.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f195bfff700 1 -- 172.21.15.119:0/3168108365 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1964027480 con 0x7f19740d0fb0 2024-09-13T16:26:19.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f195bfff700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1960066730 0x7f1960068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.372+0000 7f195bfff700 1 -- 172.21.15.119:0/3168108365 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1964022070 con 0x7f19740d0fb0 2024-09-13T16:26:19.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.376+0000 7f19737fe700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1960066730 0x7f1960068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.376+0000 7f19737fe700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1960066730 0x7f1960068bf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f19740d08c0 tx=0x7f1968009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.384+0000 7f195bfff700 1 -- 172.21.15.119:0/3168108365 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f19740ca690 con 0x7f19740d0fb0 2024-09-13T16:26:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f19740ca690 con 0x7f19740d0fb0 2024-09-13T16:26:19.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f195bfff700 1 -- 172.21.15.119:0/3168108365 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f19740ca690 con 0x7f19740d0fb0 2024-09-13T16:26:19.557 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:19.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1960066730 msgr2=0x7f1960068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f1978f4e700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1960066730 0x7f1960068bf0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f19740d08c0 tx=0x7f1968009240 comp rx=0 tx=0).stop 2024-09-13T16:26:19.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19740d0fb0 msgr2=0x7f19740b7c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f1978f4e700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19740d0fb0 0x7f19740b7c00 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f196400b370 tx=0x7f1964004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 shutdown_connections 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.552+0000 7f1978f4e700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1960066730 0x7f1960068bf0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.556+0000 7f1978f4e700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f197415cae0 0x7f1974153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.556+0000 7f1978f4e700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f19740d0fb0 0x7f19740b7c00 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.556+0000 7f1978f4e700 1 --2- 172.21.15.119:0/3168108365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f19740cff60 0x7f19740b3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.556+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 >> 172.21.15.119:0/3168108365 conn(0x7f19740d29f0 msgr2=0x7f19740af5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.556+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 shutdown_connections 2024-09-13T16:26:19.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.556+0000 7f1978f4e700 1 -- 172.21.15.119:0/3168108365 wait complete. 2024-09-13T16:26:19.571 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82c74f700 1 Processor -- start 2024-09-13T16:26:19.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82c74f700 1 -- start start 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82c74f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa8241440d0 0x7fa82415b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82c74f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82406d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82c74f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82406bd60 con 0x7fa8240cbb10 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82c74f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82415bf10 con 0x7fa8241440d0 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82c74f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82406bfe0 con 0x7fa8241444b0 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa829cea700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82406d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82acec700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa8240cbb10 0x7fa8240b1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:58746/0 (socket says 172.21.15.119:58746) 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82a4eb700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa8241440d0 0x7fa82415b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.676+0000 7fa82acec700 1 -- 172.21.15.119:0/1493892952 learned_addr learned my addr 172.21.15.119:0/1493892952 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa8294e9700 1 -- 172.21.15.119:0/1493892952 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128853688 0 0) 0x7fa82406bd60 con 0x7fa8240cbb10 2024-09-13T16:26:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa829cea700 1 -- 172.21.15.119:0/1493892952 >> v2:172.21.15.145:3300/0 conn(0x7fa8241440d0 msgr2=0x7fa82415b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa829cea700 1 --2- 172.21.15.119:0/1493892952 >> v2:172.21.15.145:3300/0 conn(0x7fa8241440d0 0x7fa82415b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa829cea700 1 -- 172.21.15.119:0/1493892952 >> v1:172.21.15.145:6789/0 conn(0x7fa8240cbb10 legacy=0x7fa8240b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa829cea700 1 -- 172.21.15.119:0/1493892952 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82406c2c0 con 0x7fa8241444b0 2024-09-13T16:26:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa829cea700 1 --2- 172.21.15.119:0/1493892952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82406d080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa814009850 tx=0x7fa814009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c50c585846dce61 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa8294e9700 1 -- 172.21.15.119:0/1493892952 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814011830 con 0x7fa8241444b0 2024-09-13T16:26:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa8294e9700 1 -- 172.21.15.119:0/1493892952 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa814011990 con 0x7fa8241444b0 2024-09-13T16:26:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa8294e9700 1 -- 172.21.15.119:0/1493892952 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa814011d40 con 0x7fa8241444b0 2024-09-13T16:26:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 -- 172.21.15.119:0/1493892952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 msgr2=0x7fa82406d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 --2- 172.21.15.119:0/1493892952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82406d080 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa814009850 tx=0x7fa814009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 -- 172.21.15.119:0/1493892952 shutdown_connections 2024-09-13T16:26:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 --2- 172.21.15.119:0/1493892952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82406d080 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 --2- 172.21.15.119:0/1493892952 >> v2:172.21.15.145:3300/0 conn(0x7fa8241440d0 0x7fa82415b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 -- 172.21.15.119:0/1493892952 >> 172.21.15.119:0/1493892952 conn(0x7fa82406a350 msgr2=0x7fa82406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 -- 172.21.15.119:0/1493892952 shutdown_connections 2024-09-13T16:26:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.680+0000 7fa82c74f700 1 -- 172.21.15.119:0/1493892952 wait complete. 2024-09-13T16:26:19.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 Processor -- start 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 -- start start 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8240cbb10 0x7fa8241f3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8241440d0 0x7fa82413e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82413ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82413f2b0 con 0x7fa8240cbb10 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82413d360 con 0x7fa8241440d0 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82413ab30 con 0x7fa8241444b0 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8240cbb10 0x7fa8241f3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8240cbb10 0x7fa8241f3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42074/0 (socket says 172.21.15.119:42074) 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 -- 172.21.15.119:0/2119071313 learned_addr learned my addr 172.21.15.119:0/2119071313 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa829cea700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8241440d0 0x7fa82413e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82acec700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82413ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 -- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 msgr2=0x7fa82413ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82413ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 -- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8241440d0 msgr2=0x7fa82413e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8241440d0 0x7fa82413e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 -- 172.21.15.119:0/2119071313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8140094b0 con 0x7fa8240cbb10 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82a4eb700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8240cbb10 0x7fa8241f3910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e9d0 tx=0x7fa82000ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa81affd700 1 -- 172.21.15.119:0/2119071313 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820009720 con 0x7fa8240cbb10 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa81affd700 1 -- 172.21.15.119:0/2119071313 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82000ee70 con 0x7fa8240cbb10 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82413ae10 con 0x7fa8240cbb10 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa81affd700 1 -- 172.21.15.119:0/2119071313 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82000ca50 con 0x7fa8240cbb10 2024-09-13T16:26:19.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.684+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8241400d0 con 0x7fa8240cbb10 2024-09-13T16:26:19.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.688+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa824067760 con 0x7fa8240cbb10 2024-09-13T16:26:19.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.688+0000 7fa81affd700 1 -- 172.21.15.119:0/2119071313 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa82000cbb0 con 0x7fa8240cbb10 2024-09-13T16:26:19.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.688+0000 7fa81affd700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa808066730 0x7fa808068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:19.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.688+0000 7fa81affd700 1 -- 172.21.15.119:0/2119071313 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fa8200844d0 con 0x7fa8240cbb10 2024-09-13T16:26:19.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.688+0000 7fa829cea700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa808066730 0x7fa808068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:19.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.692+0000 7fa829cea700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa808066730 0x7fa808068bf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa8140090b0 tx=0x7fa81401c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:19.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.696+0000 7fa81affd700 1 -- 172.21.15.119:0/2119071313 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa820051480 con 0x7fa8240cbb10 2024-09-13T16:26:19.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.880+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa8240b2070 con 0x7fa8240cbb10 2024-09-13T16:26:19.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.880+0000 7fa81affd700 1 -- 172.21.15.119:0/2119071313 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa82000ce60 con 0x7fa8240cbb10 2024-09-13T16:26:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa808066730 msgr2=0x7fa808068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa808066730 0x7fa808068bf0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa8140090b0 tx=0x7fa81401c040 comp rx=0 tx=0).stop 2024-09-13T16:26:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8240cbb10 msgr2=0x7fa8241f3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8240cbb10 0x7fa8241f3910 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa82000e9d0 tx=0x7fa82000ece0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 shutdown_connections 2024-09-13T16:26:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa808066730 0x7fa808068bf0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8241444b0 0x7fa82413ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8241440d0 0x7fa82413e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 --2- 172.21.15.119:0/2119071313 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8240cbb10 0x7fa8241f3910 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 >> 172.21.15.119:0/2119071313 conn(0x7fa82406a350 msgr2=0x7fa82406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 shutdown_connections 2024-09-13T16:26:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:19.884+0000 7fa82c74f700 1 -- 172.21.15.119:0/2119071313 wait complete. 2024-09-13T16:26:19.893 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:19.900 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:26:19.901 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:19.909 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:20.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.016+0000 7f256497e700 1 Processor -- start 2024-09-13T16:26:20.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.016+0000 7f256497e700 1 -- start start 2024-09-13T16:26:20.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.016+0000 7f256497e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256008c6b0 0x7f25600ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.016+0000 7f256497e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f256015d020 con 0x7f256015cc40 2024-09-13T16:26:20.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.016+0000 7f256497e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25600d0720 con 0x7f25600cd860 2024-09-13T16:26:20.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.016+0000 7f256497e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25600b44c0 con 0x7f256008c6b0 2024-09-13T16:26:20.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255ed9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f256015cc40 0x7f256008bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59424/0 (socket says 172.21.15.119:59424) 2024-09-13T16:26:20.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255ed9d700 1 -- 172.21.15.119:0/3029080520 learned_addr learned my addr 172.21.15.119:0/3029080520 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:20.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255e59c700 1 --2- 172.21.15.119:0/3029080520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256008c6b0 0x7f25600ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255dd9b700 1 -- 172.21.15.119:0/3029080520 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633298110 0 0) 0x7f256015d020 con 0x7f256015cc40 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255dd9b700 1 -- 172.21.15.119:0/3029080520 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f25480035d0 con 0x7f256015cc40 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255e59c700 1 -- 172.21.15.119:0/3029080520 >> v1:172.21.15.145:6789/0 conn(0x7f25600cd860 legacy=0x7f2560166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255e59c700 1 -- 172.21.15.119:0/3029080520 >> v1:172.21.15.119:6789/0 conn(0x7f256015cc40 legacy=0x7f256008bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255e59c700 1 -- 172.21.15.119:0/3029080520 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25600d09c0 con 0x7f256008c6b0 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255e59c700 1 --2- 172.21.15.119:0/3029080520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256008c6b0 0x7f25600ce710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2554009880 tx=0x7f2554009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60afbe73e143d34e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255dd9b700 1 -- 172.21.15.119:0/3029080520 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554011830 con 0x7f256008c6b0 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255dd9b700 1 -- 172.21.15.119:0/3029080520 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2554011990 con 0x7f256008c6b0 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f255dd9b700 1 -- 172.21.15.119:0/3029080520 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2554011d20 con 0x7f256008c6b0 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f256497e700 1 -- 172.21.15.119:0/3029080520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256008c6b0 msgr2=0x7f25600ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f256497e700 1 --2- 172.21.15.119:0/3029080520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256008c6b0 0x7f25600ce710 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2554009880 tx=0x7f2554009b90 comp rx=0 tx=0).stop 2024-09-13T16:26:20.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f256497e700 1 -- 172.21.15.119:0/3029080520 shutdown_connections 2024-09-13T16:26:20.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f256497e700 1 --2- 172.21.15.119:0/3029080520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256008c6b0 0x7f25600ce710 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f256497e700 1 -- 172.21.15.119:0/3029080520 >> 172.21.15.119:0/3029080520 conn(0x7f25600b4640 msgr2=0x7f25600b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:20.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f256497e700 1 -- 172.21.15.119:0/3029080520 shutdown_connections 2024-09-13T16:26:20.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.020+0000 7f256497e700 1 -- 172.21.15.119:0/3029080520 wait complete. 2024-09-13T16:26:20.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 Processor -- start 2024-09-13T16:26:20.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 -- start start 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f256008c6b0 0x7f25601561b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25600cd860 0x7f25600c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256015cc40 0x7f25600b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25600ca470 con 0x7f256008c6b0 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25600ca750 con 0x7f25600cd860 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25600caa30 con 0x7f256015cc40 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f256008c6b0 0x7f25601561b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f256008c6b0 0x7f25601561b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42076/0 (socket says 172.21.15.119:42076) 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 -- 172.21.15.119:0/2152815483 learned_addr learned my addr 172.21.15.119:0/2152815483 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255e59c700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25600cd860 0x7f25600c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255f59e700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256015cc40 0x7f25600b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 -- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256015cc40 msgr2=0x7f25600b7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256015cc40 0x7f25600b7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 -- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25600cd860 msgr2=0x7f25600c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25600cd860 0x7f25600c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 -- 172.21.15.119:0/2152815483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25540094b0 con 0x7f256008c6b0 2024-09-13T16:26:20.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f255ed9d700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f256008c6b0 0x7f25601561b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f255800cbc0 tx=0x7f255800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f25477fe700 1 -- 172.21.15.119:0/2152815483 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800a770 con 0x7f256008c6b0 2024-09-13T16:26:20.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f25477fe700 1 -- 172.21.15.119:0/2152815483 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255800a8d0 con 0x7f256008c6b0 2024-09-13T16:26:20.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25600cad10 con 0x7f256008c6b0 2024-09-13T16:26:20.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f25477fe700 1 -- 172.21.15.119:0/2152815483 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f255800ac60 con 0x7f256008c6b0 2024-09-13T16:26:20.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.024+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25600ca160 con 0x7f256008c6b0 2024-09-13T16:26:20.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.028+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25600cb230 con 0x7f256008c6b0 2024-09-13T16:26:20.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.028+0000 7f25477fe700 1 -- 172.21.15.119:0/2152815483 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f255801b3b0 con 0x7f256008c6b0 2024-09-13T16:26:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.028+0000 7f25477fe700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f254c066730 0x7f254c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.028+0000 7f25477fe700 1 -- 172.21.15.119:0/2152815483 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f25580840b0 con 0x7f256008c6b0 2024-09-13T16:26:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.028+0000 7f255e59c700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f254c066730 0x7f254c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.028+0000 7f255e59c700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f254c066730 0x7f254c068bf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25540090b0 tx=0x7f2554010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.032+0000 7f25477fe700 1 -- 172.21.15.119:0/2152815483 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f25600cb230 con 0x7f256008c6b0 2024-09-13T16:26:20.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.208+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25600cb230 con 0x7f256008c6b0 2024-09-13T16:26:20.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.212+0000 7f25477fe700 1 -- 172.21.15.119:0/2152815483 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f25600cb230 con 0x7f256008c6b0 2024-09-13T16:26:20.221 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:20.221 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:20.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.212+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f254c066730 msgr2=0x7f254c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.212+0000 7f256497e700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f254c066730 0x7f254c068bf0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f25540090b0 tx=0x7f2554010000 comp rx=0 tx=0).stop 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.212+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f256008c6b0 msgr2=0x7f25601561b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.212+0000 7f256497e700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f256008c6b0 0x7f25601561b0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f255800cbc0 tx=0x7f255800cbf0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 shutdown_connections 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f254c066730 0x7f254c068bf0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f256015cc40 0x7f25600b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25600cd860 0x7f25600c9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 --2- 172.21.15.119:0/2152815483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f256008c6b0 0x7f25601561b0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 >> 172.21.15.119:0/2152815483 conn(0x7f25600b4640 msgr2=0x7f256006c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 shutdown_connections 2024-09-13T16:26:20.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.216+0000 7f256497e700 1 -- 172.21.15.119:0/2152815483 wait complete. 2024-09-13T16:26:20.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:20.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:20.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 Processor -- start 2024-09-13T16:26:20.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- start start 2024-09-13T16:26:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe51006c480 0x7fe5100cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe5100ce000 0x7fe5100b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe51006c8c0 con 0x7fe510144180 2024-09-13T16:26:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5100b1c50 con 0x7fe5100ce000 2024-09-13T16:26:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe51006cb40 con 0x7fe51006c480 2024-09-13T16:26:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515855700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe5100ce000 0x7fe5100b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515054700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe510144180 0x7fe5100cc010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59426/0 (socket says 172.21.15.119:59426) 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515054700 1 -- 172.21.15.119:0/4146762038 learned_addr learned my addr 172.21.15.119:0/4146762038 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe514853700 1 --2- 172.21.15.119:0/4146762038 >> v2:172.21.15.145:3300/0 conn(0x7fe51006c480 0x7fe5100cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515855700 1 -- 172.21.15.119:0/4146762038 >> v2:172.21.15.145:3300/0 conn(0x7fe51006c480 msgr2=0x7fe5100cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515855700 1 --2- 172.21.15.119:0/4146762038 >> v2:172.21.15.145:3300/0 conn(0x7fe51006c480 0x7fe5100cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515855700 1 -- 172.21.15.119:0/4146762038 >> v1:172.21.15.119:6789/0 conn(0x7fe510144180 legacy=0x7fe5100cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515855700 1 -- 172.21.15.119:0/4146762038 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51006ce20 con 0x7fe5100ce000 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe515855700 1 --2- 172.21.15.119:0/4146762038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100b20a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe5100ae2c0 tx=0x7fe508009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e94faa55353e7d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe50ffff700 1 -- 172.21.15.119:0/4146762038 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508011830 con 0x7fe5100ce000 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe50ffff700 1 -- 172.21.15.119:0/4146762038 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe508011990 con 0x7fe5100ce000 2024-09-13T16:26:20.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe50ffff700 1 -- 172.21.15.119:0/4146762038 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe508011d40 con 0x7fe5100ce000 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- 172.21.15.119:0/4146762038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 msgr2=0x7fe5100b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 --2- 172.21.15.119:0/4146762038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100b20a0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe5100ae2c0 tx=0x7fe508009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- 172.21.15.119:0/4146762038 shutdown_connections 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 --2- 172.21.15.119:0/4146762038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100b20a0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 --2- 172.21.15.119:0/4146762038 >> v2:172.21.15.145:3300/0 conn(0x7fe51006c480 0x7fe5100cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- 172.21.15.119:0/4146762038 >> 172.21.15.119:0/4146762038 conn(0x7fe5100b0980 msgr2=0x7fe5100b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- 172.21.15.119:0/4146762038 shutdown_connections 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.344+0000 7fe516056700 1 -- 172.21.15.119:0/4146762038 wait complete. 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 Processor -- start 2024-09-13T16:26:20.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 -- start start 2024-09-13T16:26:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe51006c480 0x7fe51013cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe510144180 0x7fe5100ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510070740 con 0x7fe5100ce000 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe510070b60 con 0x7fe51006c480 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51006fb00 con 0x7fe510144180 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42102/0 (socket says 172.21.15.119:42102) 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 -- 172.21.15.119:0/3793962230 learned_addr learned my addr 172.21.15.119:0/3793962230 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe515855700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe510144180 0x7fe5100ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe515054700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe51006c480 0x7fe51013cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 -- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe510144180 msgr2=0x7fe5100ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe510144180 0x7fe5100ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 -- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe51006c480 msgr2=0x7fe51013cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe51006c480 0x7fe51013cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 -- 172.21.15.119:0/3793962230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5080094b0 con 0x7fe5100ce000 2024-09-13T16:26:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe514853700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100ac8c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe50400d800 tx=0x7fe50400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe50dffb700 1 -- 172.21.15.119:0/3793962230 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5040125f0 con 0x7fe5100ce000 2024-09-13T16:26:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe50dffb700 1 -- 172.21.15.119:0/3793962230 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe504012750 con 0x7fe5100ce000 2024-09-13T16:26:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51006fda0 con 0x7fe5100ce000 2024-09-13T16:26:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe50dffb700 1 -- 172.21.15.119:0/3793962230 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe504012ac0 con 0x7fe5100ce000 2024-09-13T16:26:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.348+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe510070360 con 0x7fe5100ce000 2024-09-13T16:26:20.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.352+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5100cdac0 con 0x7fe5100ce000 2024-09-13T16:26:20.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.352+0000 7fe50dffb700 1 -- 172.21.15.119:0/3793962230 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe504012c20 con 0x7fe5100ce000 2024-09-13T16:26:20.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.352+0000 7fe50dffb700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4fc066730 0x7fe4fc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.352+0000 7fe50dffb700 1 -- 172.21.15.119:0/3793962230 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fe504084570 con 0x7fe5100ce000 2024-09-13T16:26:20.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.352+0000 7fe515054700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4fc066730 0x7fe4fc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.352+0000 7fe515054700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4fc066730 0x7fe4fc068bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5100ad5a0 tx=0x7fe500001a30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.360+0000 7fe50dffb700 1 -- 172.21.15.119:0/3793962230 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe504051520 con 0x7fe5100ce000 2024-09-13T16:26:20.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.524+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe51015d980 con 0x7fe5100ce000 2024-09-13T16:26:20.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe50dffb700 1 -- 172.21.15.119:0/3793962230 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe504054b40 con 0x7fe5100ce000 2024-09-13T16:26:20.532 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:20.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4fc066730 msgr2=0x7fe4fc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4fc066730 0x7fe4fc068bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fe5100ad5a0 tx=0x7fe500001a30 comp rx=0 tx=0).stop 2024-09-13T16:26:20.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 msgr2=0x7fe5100ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100ac8c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe50400d800 tx=0x7fe50400db10 comp rx=0 tx=0).stop 2024-09-13T16:26:20.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 shutdown_connections 2024-09-13T16:26:20.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4fc066730 0x7fe4fc068bf0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe510144180 0x7fe5100ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5100ce000 0x7fe5100ac8c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 --2- 172.21.15.119:0/3793962230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe51006c480 0x7fe51013cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 >> 172.21.15.119:0/3793962230 conn(0x7fe5100b0980 msgr2=0x7fe51015ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:20.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 shutdown_connections 2024-09-13T16:26:20.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.528+0000 7fe516056700 1 -- 172.21.15.119:0/3793962230 wait complete. 2024-09-13T16:26:20.554 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.668+0000 7f6b5949e700 1 Processor -- start 2024-09-13T16:26:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.668+0000 7f6b5949e700 1 -- start start 2024-09-13T16:26:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.668+0000 7f6b5949e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 0x7f6b4c01c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.668+0000 7f6b5949e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6b4c050b90 0x7f6b4c005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.668+0000 7f6b5949e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b4c0e3310 con 0x7f6b4c04fe30 2024-09-13T16:26:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.668+0000 7f6b5949e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4c0e35f0 con 0x7f6b4c050b90 2024-09-13T16:26:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.668+0000 7f6b5949e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0e38d0 con 0x7f6b4c0df8e0 2024-09-13T16:26:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6b4c04fe30 0x7f6b4c050480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59436/0 (socket says 172.21.15.119:59436) 2024-09-13T16:26:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 -- 172.21.15.119:0/1448747131 learned_addr learned my addr 172.21.15.119:0/1448747131 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b58c9d700 1 --2- 172.21.15.119:0/1448747131 >> v2:172.21.15.145:3300/0 conn(0x7f6b4c050b90 0x7f6b4c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b53fff700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 0x7f6b4c01c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133082908 0 0) 0x7f6b4c0e3310 con 0x7f6b4c04fe30 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b440035d0 con 0x7f6b4c04fe30 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2641527669 0 0) 0x7f6b440035d0 con 0x7f6b4c04fe30 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 msgr2=0x7f6b4c01c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 0x7f6b4c01c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 >> v2:172.21.15.145:3300/0 conn(0x7f6b4c050b90 msgr2=0x7f6b4c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 --2- 172.21.15.119:0/1448747131 >> v2:172.21.15.145:3300/0 conn(0x7f6b4c050b90 0x7f6b4c005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0e3bb0 con 0x7f6b4c04fe30 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f6b40001670 con 0x7f6b4c04fe30 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b58c9d700 1 --2- 172.21.15.119:0/1448747131 >> v2:172.21.15.145:3300/0 conn(0x7f6b4c050b90 0x7f6b4c005b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 >> v1:172.21.15.119:6789/0 conn(0x7f6b4c04fe30 legacy=0x7f6b4c050480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b44003a20 0x7f6b44003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b44004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 0x7f6b44005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0e35f0 con 0x7f6b44004870 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0e38d0 con 0x7f6b44003a20 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4400ba30 con 0x7f6b4c0df8e0 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b44004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b53fff700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b44003a20 0x7f6b44003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b58c9d700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 0x7f6b44005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 -- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 msgr2=0x7f6b44005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c0df8e0 0x7f6b44005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 -- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b44003a20 msgr2=0x7f6b44003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b44003a20 0x7f6b44003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 -- 172.21.15.119:0/1448747131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b440035d0 con 0x7f6b44004870 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b537fe700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b44004cf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000c2f0 tx=0x7f6b440058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3954958adafa1949 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4000ce30 con 0x7f6b44004870 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b40005980 con 0x7f6b44004870 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b52ffd700 1 -- 172.21.15.119:0/1448747131 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b40005d30 con 0x7f6b44004870 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b5949e700 1 -- 172.21.15.119:0/1448747131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 msgr2=0x7f6b44004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b5949e700 1 --2- 172.21.15.119:0/1448747131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b44004cf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f6b4000c2f0 tx=0x7f6b440058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b58c9d700 1 -- 172.21.15.119:0/1448747131 reap_dead start 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b5949e700 1 -- 172.21.15.119:0/1448747131 shutdown_connections 2024-09-13T16:26:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b5949e700 1 -- 172.21.15.119:0/1448747131 >> 172.21.15.119:0/1448747131 conn(0x7f6b4c036140 msgr2=0x7f6b4c036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b5949e700 1 -- 172.21.15.119:0/1448747131 shutdown_connections 2024-09-13T16:26:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.672+0000 7f6b5949e700 1 -- 172.21.15.119:0/1448747131 wait complete. 2024-09-13T16:26:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.676+0000 7f6b5949e700 1 Processor -- start 2024-09-13T16:26:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.676+0000 7f6b5949e700 1 -- start start 2024-09-13T16:26:20.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.676+0000 7f6b5949e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b4c0d3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.676+0000 7f6b5949e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b4c0d9d10 0x7f6b4c0da170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b5949e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c04d8a0 0x7f6b4c04dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b5949e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c04e260 con 0x7f6b44004870 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b5949e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0da7d0 con 0x7f6b4c0d9d10 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b5949e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b4c0daab0 con 0x7f6b4c04d8a0 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b4c0d3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b4c0d3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42126/0 (socket says 172.21.15.119:42126) 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 -- 172.21.15.119:0/4123938263 learned_addr learned my addr 172.21.15.119:0/4123938263 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b58c9d700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c04d8a0 0x7f6b4c04dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b537fe700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b4c0d9d10 0x7f6b4c0da170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 -- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c04d8a0 msgr2=0x7f6b4c04dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c04d8a0 0x7f6b4c04dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 -- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b4c0d9d10 msgr2=0x7f6b4c0da170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b4c0d9d10 0x7f6b4c0da170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 -- 172.21.15.119:0/4123938263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0d4a70 con 0x7f6b44004870 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b53fff700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b4c0d3d90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6b48004700 tx=0x7f6b48011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b50ff9700 1 -- 172.21.15.119:0/4123938263 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48005c50 con 0x7f6b44004870 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b50ff9700 1 -- 172.21.15.119:0/4123938263 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b48005db0 con 0x7f6b44004870 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b4c0d4d50 con 0x7f6b44004870 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b50ff9700 1 -- 172.21.15.119:0/4123938263 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b48006290 con 0x7f6b44004870 2024-09-13T16:26:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4c0334a0 con 0x7f6b44004870 2024-09-13T16:26:20.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.680+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b4c01e140 con 0x7f6b44004870 2024-09-13T16:26:20.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.684+0000 7f6b50ff9700 1 -- 172.21.15.119:0/4123938263 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6b480063f0 con 0x7f6b44004870 2024-09-13T16:26:20.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.684+0000 7f6b50ff9700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6b38066730 0x7f6b38068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:20.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.684+0000 7f6b50ff9700 1 -- 172.21.15.119:0/4123938263 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f6b480849e0 con 0x7f6b44004870 2024-09-13T16:26:20.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.684+0000 7f6b537fe700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6b38066730 0x7f6b38068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:20.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.684+0000 7f6b537fe700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6b38066730 0x7f6b38068bf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b40015040 tx=0x7f6b4000a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:20.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.692+0000 7f6b50ff9700 1 -- 172.21.15.119:0/4123938263 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6b48051990 con 0x7f6b44004870 2024-09-13T16:26:20.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b4c036e40 con 0x7f6b44004870 2024-09-13T16:26:20.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b50ff9700 1 -- 172.21.15.119:0/4123938263 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b48054fb0 con 0x7f6b44004870 2024-09-13T16:26:20.868 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6b38066730 msgr2=0x7f6b38068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6b38066730 0x7f6b38068bf0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6b40015040 tx=0x7f6b4000a000 comp rx=0 tx=0).stop 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 msgr2=0x7f6b4c0d3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b4c0d3d90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6b48004700 tx=0x7f6b48011ca0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 shutdown_connections 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6b38066730 0x7f6b38068bf0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6b4c04d8a0 0x7f6b4c04dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6b4c0d9d10 0x7f6b4c0da170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 --2- 172.21.15.119:0/4123938263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6b44004870 0x7f6b4c0d3d90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 >> 172.21.15.119:0/4123938263 conn(0x7f6b4c036140 msgr2=0x7f6b4c0dd9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 shutdown_connections 2024-09-13T16:26:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.864+0000 7f6b5949e700 1 -- 172.21.15.119:0/4123938263 wait complete. 2024-09-13T16:26:20.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:20.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.992+0000 7f596d5a6700 1 Processor -- start 2024-09-13T16:26:20.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.992+0000 7f596d5a6700 1 -- start start 2024-09-13T16:26:21.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f596806ad60 0x7f596806e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f596806f220 con 0x7f596806edd0 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59680710e0 con 0x7f596806ad60 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59680713c0 con 0x7f596815dcc0 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f596806ad60 0x7f596806e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f596806ad60 0x7f596806e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42142/0 (socket says 172.21.15.119:42142) 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 -- 172.21.15.119:0/1616330498 learned_addr learned my addr 172.21.15.119:0/1616330498 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f5967fff700 1 --2- 172.21.15.119:0/1616330498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 -- 172.21.15.119:0/1616330498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 msgr2=0x7f5968070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 --2- 172.21.15.119:0/1616330498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968070b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 -- 172.21.15.119:0/1616330498 >> v1:172.21.15.145:6789/0 conn(0x7f596806edd0 legacy=0x7f596806ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 -- 172.21.15.119:0/1616330498 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59680716a0 con 0x7f596806ad60 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f59677fe700 1 --2- 172.21.15.119:0/1616330498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f596806e890 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5950009910 tx=0x7f5950009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12d0876225c8d38 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f5966ffd700 1 -- 172.21.15.119:0/1616330498 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f595000ba60 con 0x7f596806ad60 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f5966ffd700 1 -- 172.21.15.119:0/1616330498 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f595000bbc0 con 0x7f596806ad60 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f5966ffd700 1 -- 172.21.15.119:0/1616330498 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5950009e60 con 0x7f596806ad60 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- 172.21.15.119:0/1616330498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 msgr2=0x7f596806e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 --2- 172.21.15.119:0/1616330498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f596806e890 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5950009910 tx=0x7f5950009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- 172.21.15.119:0/1616330498 shutdown_connections 2024-09-13T16:26:21.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 --2- 172.21.15.119:0/1616330498 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f596806e890 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 --2- 172.21.15.119:0/1616330498 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- 172.21.15.119:0/1616330498 >> 172.21.15.119:0/1616330498 conn(0x7f59680b1500 msgr2=0x7f59680b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:21.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- 172.21.15.119:0/1616330498 shutdown_connections 2024-09-13T16:26:21.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:20.996+0000 7f596d5a6700 1 -- 172.21.15.119:0/1616330498 wait complete. 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 Processor -- start 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 -- start start 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f5968153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f596806edd0 0x7f5968154180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5968155570 con 0x7f596806ad60 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59681546c0 con 0x7f596806edd0 2024-09-13T16:26:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596d5a6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5968155ed0 con 0x7f596815dcc0 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f5968153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f5968153940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42154/0 (socket says 172.21.15.119:42154) 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 -- 172.21.15.119:0/2536858035 learned_addr learned my addr 172.21.15.119:0/2536858035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f59677fe700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f596806edd0 0x7f5968154180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f596cda5700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 -- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 msgr2=0x7f5968155990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968155990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 -- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f596806edd0 msgr2=0x7f5968154180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f596806edd0 0x7f5968154180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 -- 172.21.15.119:0/2536858035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5950009570 con 0x7f596806ad60 2024-09-13T16:26:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5967fff700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f5968153940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f595800d800 tx=0x7f595800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5964ff9700 1 -- 172.21.15.119:0/2536858035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59580125f0 con 0x7f596806ad60 2024-09-13T16:26:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.000+0000 7f5964ff9700 1 -- 172.21.15.119:0/2536858035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5958012750 con 0x7f596806ad60 2024-09-13T16:26:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59681548f0 con 0x7f596806ad60 2024-09-13T16:26:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f5964ff9700 1 -- 172.21.15.119:0/2536858035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5958012ac0 con 0x7f596806ad60 2024-09-13T16:26:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5968154eb0 con 0x7f596806ad60 2024-09-13T16:26:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f596806e890 con 0x7f596806ad60 2024-09-13T16:26:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f5964ff9700 1 -- 172.21.15.119:0/2536858035 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5958012c20 con 0x7f596806ad60 2024-09-13T16:26:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f5964ff9700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5948066730 0x7f5948068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f5964ff9700 1 -- 172.21.15.119:0/2536858035 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5958084550 con 0x7f596806ad60 2024-09-13T16:26:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.004+0000 7f59677fe700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5948066730 0x7f5948068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.008+0000 7f59677fe700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5948066730 0x7f5948068bf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5950000c00 tx=0x7f5950008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.012+0000 7f5964ff9700 1 -- 172.21.15.119:0/2536858035 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5958051590 con 0x7f596806ad60 2024-09-13T16:26:21.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f596815bbb0 con 0x7f596806ad60 2024-09-13T16:26:21.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f5964ff9700 1 -- 172.21.15.119:0/2536858035 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5958054bb0 con 0x7f596806ad60 2024-09-13T16:26:21.185 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:21.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5948066730 msgr2=0x7f5948068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5948066730 0x7f5948068bf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5950000c00 tx=0x7f5950008030 comp rx=0 tx=0).stop 2024-09-13T16:26:21.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 msgr2=0x7f5968153940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f5968153940 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f595800d800 tx=0x7f595800db10 comp rx=0 tx=0).stop 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 shutdown_connections 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5948066730 0x7f5948068bf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f596815dcc0 0x7f5968155990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f596806edd0 0x7f5968154180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 --2- 172.21.15.119:0/2536858035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f596806ad60 0x7f5968153940 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.180+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 >> 172.21.15.119:0/2536858035 conn(0x7f59680b1500 msgr2=0x7f596815b070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.184+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 shutdown_connections 2024-09-13T16:26:21.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.184+0000 7f596d5a6700 1 -- 172.21.15.119:0/2536858035 wait complete. 2024-09-13T16:26:21.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:21.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.312+0000 7f469f59e700 1 Processor -- start 2024-09-13T16:26:21.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.312+0000 7f469f59e700 1 -- start start 2024-09-13T16:26:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3740 0x7f46a00b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469f59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a00711b0 con 0x7f46a0153540 2024-09-13T16:26:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a006b360 con 0x7f46a00b3070 2024-09-13T16:26:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0072210 con 0x7f46a00b3740 2024-09-13T16:26:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469ed9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f46a0153540 0x7f46a0159560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59452/0 (socket says 172.21.15.119:59452) 2024-09-13T16:26:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469ed9d700 1 -- 172.21.15.119:0/2447490107 learned_addr learned my addr 172.21.15.119:0/2447490107 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:21.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469e59c700 1 --2- 172.21.15.119:0/2447490107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3740 0x7f46a00b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469dd9b700 1 -- 172.21.15.119:0/2447490107 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504675767 0 0) 0x7f46a00711b0 con 0x7f46a0153540 2024-09-13T16:26:21.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469dd9b700 1 -- 172.21.15.119:0/2447490107 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f468c0034f0 con 0x7f46a0153540 2024-09-13T16:26:21.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469dd9b700 1 -- 172.21.15.119:0/2447490107 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339959275 0 0) 0x7f46a006b360 con 0x7f46a00b3070 2024-09-13T16:26:21.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469e59c700 1 -- 172.21.15.119:0/2447490107 >> v1:172.21.15.145:6789/0 conn(0x7f46a00b3070 legacy=0x7f46a0152e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469e59c700 1 -- 172.21.15.119:0/2447490107 >> v1:172.21.15.119:6789/0 conn(0x7f46a0153540 legacy=0x7f46a0159560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469e59c700 1 -- 172.21.15.119:0/2447490107 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a0071450 con 0x7f46a00b3740 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469e59c700 1 --2- 172.21.15.119:0/2447490107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3740 0x7f46a00b2b30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4690009880 tx=0x7f4690009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab2d4d1ca9909c83 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469dd9b700 1 -- 172.21.15.119:0/2447490107 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690011830 con 0x7f46a00b3740 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469dd9b700 1 -- 172.21.15.119:0/2447490107 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4690011990 con 0x7f46a00b3740 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.316+0000 7f469dd9b700 1 -- 172.21.15.119:0/2447490107 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690011d00 con 0x7f46a00b3740 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.320+0000 7f469f59e700 1 -- 172.21.15.119:0/2447490107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3740 msgr2=0x7f46a00b2b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.320+0000 7f469f59e700 1 --2- 172.21.15.119:0/2447490107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3740 0x7f46a00b2b30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4690009880 tx=0x7f4690009b90 comp rx=0 tx=0).stop 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.320+0000 7f469f59e700 1 -- 172.21.15.119:0/2447490107 shutdown_connections 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.320+0000 7f469f59e700 1 --2- 172.21.15.119:0/2447490107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3740 0x7f46a00b2b30 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.320+0000 7f469f59e700 1 -- 172.21.15.119:0/2447490107 >> 172.21.15.119:0/2447490107 conn(0x7f46a006c880 msgr2=0x7f46a006cc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.320+0000 7f469f59e700 1 -- 172.21.15.119:0/2447490107 shutdown_connections 2024-09-13T16:26:21.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.320+0000 7f469f59e700 1 -- 172.21.15.119:0/2447490107 wait complete. 2024-09-13T16:26:21.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 Processor -- start 2024-09-13T16:26:21.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 -- start start 2024-09-13T16:26:21.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3070 0x7f46a0142920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a00b3740 0x7f46a0155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0153540 0x7f46a015be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0156490 con 0x7f46a00b3740 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0154eb0 con 0x7f46a0153540 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a0155190 con 0x7f46a00b3070 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a00b3740 0x7f46a0155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a00b3740 0x7f46a0155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42156/0 (socket says 172.21.15.119:42156) 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 -- 172.21.15.119:0/3917114909 learned_addr learned my addr 172.21.15.119:0/3917114909 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469e59c700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3070 0x7f46a0142920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469ed9d700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0153540 0x7f46a015be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 -- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3070 msgr2=0x7f46a0142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3070 0x7f46a0142920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 -- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0153540 msgr2=0x7f46a015be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0153540 0x7f46a015be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 -- 172.21.15.119:0/3917114909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46900094b0 con 0x7f46a00b3740 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469ed9d700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0153540 0x7f46a015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4697fff700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a00b3740 0x7f46a0155f50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f468800c550 tx=0x7f468800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4696ffd700 1 -- 172.21.15.119:0/3917114909 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46880054e0 con 0x7f46a00b3740 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4696ffd700 1 -- 172.21.15.119:0/3917114909 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46880036a0 con 0x7f46a00b3740 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a0155470 con 0x7f46a00b3740 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f4696ffd700 1 -- 172.21.15.119:0/3917114909 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4688003a50 con 0x7f46a00b3740 2024-09-13T16:26:21.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.324+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a0155a30 con 0x7f46a00b3740 2024-09-13T16:26:21.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.328+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a00b2b30 con 0x7f46a00b3740 2024-09-13T16:26:21.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.328+0000 7f4696ffd700 1 -- 172.21.15.119:0/3917114909 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4688003bb0 con 0x7f46a00b3740 2024-09-13T16:26:21.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.328+0000 7f4696ffd700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4680066730 0x7f4680068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.328+0000 7f4696ffd700 1 -- 172.21.15.119:0/3917114909 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f46880838d0 con 0x7f46a00b3740 2024-09-13T16:26:21.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.328+0000 7f469e59c700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4680066730 0x7f4680068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.328+0000 7f469e59c700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4680066730 0x7f4680068bf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f46900097d0 tx=0x7f4690010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.332+0000 7f4696ffd700 1 -- 172.21.15.119:0/3917114909 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4688050910 con 0x7f46a00b3740 2024-09-13T16:26:21.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.504+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f46a0158450 con 0x7f46a00b3740 2024-09-13T16:26:21.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.504+0000 7f4696ffd700 1 -- 172.21.15.119:0/3917114909 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f46880170a0 con 0x7f46a00b3740 2024-09-13T16:26:21.508 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4680066730 msgr2=0x7f4680068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4680066730 0x7f4680068bf0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f46900097d0 tx=0x7f4690010000 comp rx=0 tx=0).stop 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a00b3740 msgr2=0x7f46a0155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a00b3740 0x7f46a0155f50 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f468800c550 tx=0x7f468800c860 comp rx=0 tx=0).stop 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 shutdown_connections 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4680066730 0x7f4680068bf0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0153540 0x7f46a015be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a00b3740 0x7f46a0155f50 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 --2- 172.21.15.119:0/3917114909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a00b3070 0x7f46a0142920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 >> 172.21.15.119:0/3917114909 conn(0x7f46a006c880 msgr2=0x7f46a00cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:21.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 shutdown_connections 2024-09-13T16:26:21.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.508+0000 7f469f59e700 1 -- 172.21.15.119:0/3917114909 wait complete. 2024-09-13T16:26:21.529 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:21.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f374c9c1700 1 Processor -- start 2024-09-13T16:26:21.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f374c9c1700 1 -- start start 2024-09-13T16:26:21.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f374c9c1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f374815d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f374c9c1700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f374806cbb0 0x7f374815d780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f374c9c1700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37480b3830 con 0x7f374815dcc0 2024-09-13T16:26:21.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f374c9c1700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37480b2c20 con 0x7f374806cbb0 2024-09-13T16:26:21.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f374c9c1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37480b2f00 con 0x7f3748159800 2024-09-13T16:26:21.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f3747fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f374815dcc0 0x7f374806daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59468/0 (socket says 172.21.15.119:59468) 2024-09-13T16:26:21.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f3747fff700 1 -- 172.21.15.119:0/3272492224 learned_addr learned my addr 172.21.15.119:0/3272492224 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:21.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f37477fe700 1 --2- 172.21.15.119:0/3272492224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f374815d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f3746ffd700 1 --2- 172.21.15.119:0/3272492224 >> v2:172.21.15.145:3300/0 conn(0x7f374806cbb0 0x7f374815d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f37477fe700 1 -- 172.21.15.119:0/3272492224 >> v2:172.21.15.145:3300/0 conn(0x7f374806cbb0 msgr2=0x7f374815d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:21.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f37477fe700 1 --2- 172.21.15.119:0/3272492224 >> v2:172.21.15.145:3300/0 conn(0x7f374806cbb0 0x7f374815d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f37477fe700 1 -- 172.21.15.119:0/3272492224 >> v1:172.21.15.119:6789/0 conn(0x7f374815dcc0 legacy=0x7f374806daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.640+0000 7f37477fe700 1 -- 172.21.15.119:0/3272492224 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37480b31e0 con 0x7f3748159800 2024-09-13T16:26:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.644+0000 7f37477fe700 1 --2- 172.21.15.119:0/3272492224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f374815d240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3740009d90 tx=0x7f374000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3053266784b0c549 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f37467fc700 1 -- 172.21.15.119:0/3272492224 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374000e070 con 0x7f3748159800 2024-09-13T16:26:21.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f37467fc700 1 -- 172.21.15.119:0/3272492224 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374000bb20 con 0x7f3748159800 2024-09-13T16:26:21.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f37467fc700 1 -- 172.21.15.119:0/3272492224 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3740012430 con 0x7f3748159800 2024-09-13T16:26:21.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 -- 172.21.15.119:0/3272492224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 msgr2=0x7f374815d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 --2- 172.21.15.119:0/3272492224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f374815d240 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3740009d90 tx=0x7f374000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:21.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 -- 172.21.15.119:0/3272492224 shutdown_connections 2024-09-13T16:26:21.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 --2- 172.21.15.119:0/3272492224 >> v2:172.21.15.145:3300/0 conn(0x7f374806cbb0 0x7f374815d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 --2- 172.21.15.119:0/3272492224 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f374815d240 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 -- 172.21.15.119:0/3272492224 >> 172.21.15.119:0/3272492224 conn(0x7f37480b17a0 msgr2=0x7f37480b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:21.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 -- 172.21.15.119:0/3272492224 shutdown_connections 2024-09-13T16:26:21.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 -- 172.21.15.119:0/3272492224 wait complete. 2024-09-13T16:26:21.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 Processor -- start 2024-09-13T16:26:21.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.648+0000 7f374c9c1700 1 -- start start 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f374806cbb0 0x7f3748142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f37480cd9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f374815dcc0 0x7f37480cdf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37481453a0 con 0x7f374815dcc0 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3748143570 con 0x7f374806cbb0 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3748143850 con 0x7f3748159800 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f374815dcc0 0x7f37480cdf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f374815dcc0 0x7f37480cdf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42166/0 (socket says 172.21.15.119:42166) 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 -- 172.21.15.119:0/1541434430 learned_addr learned my addr 172.21.15.119:0/1541434430 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f37477fe700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f374806cbb0 0x7f3748142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3746ffd700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f37480cd9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 -- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 msgr2=0x7f37480cd9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f37480cd9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 -- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f374806cbb0 msgr2=0x7f3748142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f374806cbb0 0x7f3748142d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 -- 172.21.15.119:0/1541434430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3740009a70 con 0x7f374815dcc0 2024-09-13T16:26:21.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f3747fff700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f374815dcc0 0x7f37480cdf00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f373800c4d0 tx=0x7f373800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f372ffff700 1 -- 172.21.15.119:0/1541434430 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3738005d90 con 0x7f374815dcc0 2024-09-13T16:26:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3748143b30 con 0x7f374815dcc0 2024-09-13T16:26:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3748143260 con 0x7f374815dcc0 2024-09-13T16:26:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f372ffff700 1 -- 172.21.15.119:0/1541434430 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373800ce80 con 0x7f374815dcc0 2024-09-13T16:26:21.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.652+0000 7f372ffff700 1 -- 172.21.15.119:0/1541434430 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37380011c0 con 0x7f374815dcc0 2024-09-13T16:26:21.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.656+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3728005310 con 0x7f374815dcc0 2024-09-13T16:26:21.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.656+0000 7f372ffff700 1 -- 172.21.15.119:0/1541434430 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f37380072d0 con 0x7f374815dcc0 2024-09-13T16:26:21.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.656+0000 7f372ffff700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3734066730 0x7f3734068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:21.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.656+0000 7f372ffff700 1 -- 172.21.15.119:0/1541434430 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f37380836e0 con 0x7f374815dcc0 2024-09-13T16:26:21.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.656+0000 7f37477fe700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3734066730 0x7f3734068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:21.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.656+0000 7f37477fe700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3734066730 0x7f3734068bf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3740014040 tx=0x7f3740011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:21.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.664+0000 7f372ffff700 1 -- 172.21.15.119:0/1541434430 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3738050750 con 0x7f374815dcc0 2024-09-13T16:26:21.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f37280055c0 con 0x7f374815dcc0 2024-09-13T16:26:21.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f372ffff700 1 -- 172.21.15.119:0/1541434430 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v27) v1 ==== 80+0+0 (secure 0 0 0) 0x7f3738053d70 con 0x7f374815dcc0 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3734066730 msgr2=0x7f3734068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3734066730 0x7f3734068bf0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3740014040 tx=0x7f3740011000 comp rx=0 tx=0).stop 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f374815dcc0 msgr2=0x7f37480cdf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f374815dcc0 0x7f37480cdf00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f373800c4d0 tx=0x7f373800c7e0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 shutdown_connections 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3734066730 0x7f3734068bf0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f374815dcc0 0x7f37480cdf00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3748159800 0x7f37480cd9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 --2- 172.21.15.119:0/1541434430 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f374806cbb0 0x7f3748142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 >> 172.21.15.119:0/1541434430 conn(0x7f37480b17a0 msgr2=0x7f3748070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 shutdown_connections 2024-09-13T16:26:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:21.848+0000 7f374c9c1700 1 -- 172.21.15.119:0/1541434430 wait complete. 2024-09-13T16:26:21.857 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:21.867 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi145.front.sepia.ceph.com... 2024-09-13T16:26:21.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:21.867 DEBUG:teuthology.orchestra.run.smithi145:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-09-13T16:26:21.885 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:21.885 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:26:21.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:21.885 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:26:21.933 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T16:26:21.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.048+0000 7f3e1da66700 1 Processor -- start 2024-09-13T16:26:22.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.048+0000 7f3e1da66700 1 -- start start 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3e100e18c0 0x7f3e1001ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e10006ce0 con 0x7f3e1000a5b0 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e100060d0 con 0x7f3e10005b10 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e100063b0 con 0x7f3e100e18c0 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1ca64700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3e1000a5b0 0x7f3e1000a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59482/0 (socket says 172.21.15.119:59482) 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1ca64700 1 -- 172.21.15.119:0/476179396 learned_addr learned my addr 172.21.15.119:0/476179396 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1d265700 1 --2- 172.21.15.119:0/476179396 >> v2:172.21.15.145:3300/0 conn(0x7f3e100e18c0 0x7f3e1001ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e177fe700 1 -- 172.21.15.119:0/476179396 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859096358 0 0) 0x7f3e10006ce0 con 0x7f3e1000a5b0 2024-09-13T16:26:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e177fe700 1 -- 172.21.15.119:0/476179396 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e000035d0 con 0x7f3e1000a5b0 2024-09-13T16:26:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1d265700 1 -- 172.21.15.119:0/476179396 >> v1:172.21.15.145:6789/0 conn(0x7f3e10005b10 legacy=0x7f3e1001c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1d265700 1 -- 172.21.15.119:0/476179396 >> v1:172.21.15.119:6789/0 conn(0x7f3e1000a5b0 legacy=0x7f3e1000a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e177fe700 1 -- 172.21.15.119:0/476179396 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3979186805 0 0) 0x7f3e000035d0 con 0x7f3e1000a5b0 2024-09-13T16:26:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1d265700 1 -- 172.21.15.119:0/476179396 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e10006690 con 0x7f3e100e18c0 2024-09-13T16:26:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1d265700 1 --2- 172.21.15.119:0/476179396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 0x7f3e1001ce50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3e1804ff70 tx=0x7f3e18068790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a10ccedc1ba7a31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e177fe700 1 -- 172.21.15.119:0/476179396 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e080017f0 con 0x7f3e100e18c0 2024-09-13T16:26:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e177fe700 1 -- 172.21.15.119:0/476179396 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e18067280 con 0x7f3e100e18c0 2024-09-13T16:26:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e177fe700 1 -- 172.21.15.119:0/476179396 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180675f0 con 0x7f3e100e18c0 2024-09-13T16:26:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- 172.21.15.119:0/476179396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 msgr2=0x7f3e1001ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 --2- 172.21.15.119:0/476179396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 0x7f3e1001ce50 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3e1804ff70 tx=0x7f3e18068790 comp rx=0 tx=0).stop 2024-09-13T16:26:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- 172.21.15.119:0/476179396 shutdown_connections 2024-09-13T16:26:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 --2- 172.21.15.119:0/476179396 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 0x7f3e1001ce50 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- 172.21.15.119:0/476179396 >> 172.21.15.119:0/476179396 conn(0x7f3e10036140 msgr2=0x7f3e10036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- 172.21.15.119:0/476179396 shutdown_connections 2024-09-13T16:26:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.052+0000 7f3e1da66700 1 -- 172.21.15.119:0/476179396 wait complete. 2024-09-13T16:26:22.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.056+0000 7f3e1da66700 1 Processor -- start 2024-09-13T16:26:22.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.056+0000 7f3e1da66700 1 -- start start 2024-09-13T16:26:22.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e10005b10 0x7f3e100480f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e1000a5b0 0x7f3e10175c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 0x7f3e10179a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e100e2a10 con 0x7f3e10005b10 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e100e2cb0 con 0x7f3e100e18c0 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e100e2f90 con 0x7f3e1000a5b0 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e10005b10 0x7f3e100480f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e10005b10 0x7f3e100480f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42168/0 (socket says 172.21.15.119:42168) 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 -- 172.21.15.119:0/2489340947 learned_addr learned my addr 172.21.15.119:0/2489340947 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e17fff700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e1000a5b0 0x7f3e10175c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1d265700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 0x7f3e10179a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 -- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e1000a5b0 msgr2=0x7f3e10175c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e1000a5b0 0x7f3e10175c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 -- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 msgr2=0x7f3e10179a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 0x7f3e10179a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 -- 172.21.15.119:0/2489340947 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e100e3270 con 0x7f3e10005b10 2024-09-13T16:26:22.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1ca64700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e10005b10 0x7f3e100480f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c530 tx=0x7f3e0800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e157fa700 1 -- 172.21.15.119:0/2489340947 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e080074e0 con 0x7f3e10005b10 2024-09-13T16:26:22.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e157fa700 1 -- 172.21.15.119:0/2489340947 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e08007640 con 0x7f3e10005b10 2024-09-13T16:26:22.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1004b1d0 con 0x7f3e10005b10 2024-09-13T16:26:22.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e157fa700 1 -- 172.21.15.119:0/2489340947 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08005a70 con 0x7f3e10005b10 2024-09-13T16:26:22.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e101761c0 con 0x7f3e10005b10 2024-09-13T16:26:22.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.060+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e100e34b0 con 0x7f3e10005b10 2024-09-13T16:26:22.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.064+0000 7f3e157fa700 1 -- 172.21.15.119:0/2489340947 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3e08005bd0 con 0x7f3e10005b10 2024-09-13T16:26:22.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.064+0000 7f3e157fa700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e04066730 0x7f3e04068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.064+0000 7f3e157fa700 1 -- 172.21.15.119:0/2489340947 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3e08083a40 con 0x7f3e10005b10 2024-09-13T16:26:22.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.064+0000 7f3e17fff700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e04066730 0x7f3e04068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.064+0000 7f3e17fff700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e04066730 0x7f3e04068bf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e100224f0 tx=0x7f3e0c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.076+0000 7f3e157fa700 1 -- 172.21.15.119:0/2489340947 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3e100e34b0 con 0x7f3e10005b10 2024-09-13T16:26:22.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.280+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e100e34b0 con 0x7f3e10005b10 2024-09-13T16:26:22.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e157fa700 1 -- 172.21.15.119:0/2489340947 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f3e100e34b0 con 0x7f3e10005b10 2024-09-13T16:26:22.288 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:22.288 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:26:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e04066730 msgr2=0x7f3e04068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e04066730 0x7f3e04068bf0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3e100224f0 tx=0x7f3e0c009040 comp rx=0 tx=0).stop 2024-09-13T16:26:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e10005b10 msgr2=0x7f3e100480f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e10005b10 0x7f3e100480f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f3e0800c530 tx=0x7f3e0800c840 comp rx=0 tx=0).stop 2024-09-13T16:26:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 shutdown_connections 2024-09-13T16:26:22.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e04066730 0x7f3e04068bf0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e100e18c0 0x7f3e10179a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e1000a5b0 0x7f3e10175c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 --2- 172.21.15.119:0/2489340947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e10005b10 0x7f3e100480f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 >> 172.21.15.119:0/2489340947 conn(0x7f3e10036140 msgr2=0x7f3e10035340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 shutdown_connections 2024-09-13T16:26:22.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.284+0000 7f3e1da66700 1 -- 172.21.15.119:0/2489340947 wait complete. 2024-09-13T16:26:22.293 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:26:22.303 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:22.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1467fff700 1 Processor -- start 2024-09-13T16:26:22.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1467fff700 1 -- start start 2024-09-13T16:26:22.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1467fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f146806d240 0x7f146806d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1467fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146806c230 con 0x7f1468159ab0 2024-09-13T16:26:22.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1467fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f146806c480 con 0x7f1468159e90 2024-09-13T16:26:22.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1467fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f146806ce20 con 0x7f146806d240 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1466ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1468159ab0 0x7f14680b3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51240/0 (socket says 172.21.15.119:51240) 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1466ffd700 1 -- 172.21.15.119:0/2674637521 learned_addr learned my addr 172.21.15.119:0/2674637521 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f14677fe700 1 --2- 172.21.15.119:0/2674637521 >> v2:172.21.15.145:3300/0 conn(0x7f146806d240 0x7f146806d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488306664 0 0) 0x7f146806c230 con 0x7f1468159ab0 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14500035d0 con 0x7f1468159ab0 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2911216126 0 0) 0x7f14500035d0 con 0x7f1468159ab0 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 >> v2:172.21.15.145:3300/0 conn(0x7f146806d240 msgr2=0x7f146806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 --2- 172.21.15.119:0/2674637521 >> v2:172.21.15.145:3300/0 conn(0x7f146806d240 0x7f146806d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 >> v1:172.21.15.145:6789/0 conn(0x7f1468159e90 legacy=0x7f146815a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146806c760 con 0x7f1468159ab0 2024-09-13T16:26:22.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f14600019f0 con 0x7f1468159ab0 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 >> v1:172.21.15.119:6789/0 conn(0x7f1468159ab0 legacy=0x7f14680b3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1450003a20 0x7f1450003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450004870 0x7f1450004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1450005350 0x7f145000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f146806ce20 con 0x7f1450005350 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f145000c220 con 0x7f1450004870 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f145000c3e0 con 0x7f1450003a20 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f14677fe700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1450005350 0x7f145000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f1466ffd700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1450003a20 0x7f1450003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.408+0000 7f14667fc700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450004870 0x7f1450004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f14677fe700 1 -- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1450003a20 msgr2=0x7f1450003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f14677fe700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1450003a20 0x7f1450003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f14677fe700 1 -- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450004870 msgr2=0x7f1450004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f14677fe700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450004870 0x7f1450004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f14677fe700 1 -- 172.21.15.119:0/2674637521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14500035d0 con 0x7f1450005350 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f14677fe700 1 -- 172.21.15.119:0/2674637521 reap_dead start 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f14677fe700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1450005350 0x7f145000ba30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1458009b40 tx=0x7f14500058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b62132d5a2b9a75 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458004020 con 0x7f1450005350 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1458004180 con 0x7f1450005350 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1465ffb700 1 -- 172.21.15.119:0/2674637521 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458004530 con 0x7f1450005350 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1467fff700 1 -- 172.21.15.119:0/2674637521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1450005350 msgr2=0x7f145000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1467fff700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1450005350 0x7f145000ba30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f1458009b40 tx=0x7f14500058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1467fff700 1 -- 172.21.15.119:0/2674637521 shutdown_connections 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1467fff700 1 --2- 172.21.15.119:0/2674637521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1450005350 0x7f145000ba30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1467fff700 1 -- 172.21.15.119:0/2674637521 >> 172.21.15.119:0/2674637521 conn(0x7f146806a350 msgr2=0x7f146806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1467fff700 1 -- 172.21.15.119:0/2674637521 shutdown_connections 2024-09-13T16:26:22.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.412+0000 7f1467fff700 1 -- 172.21.15.119:0/2674637521 wait complete. 2024-09-13T16:26:22.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 Processor -- start 2024-09-13T16:26:22.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 -- start start 2024-09-13T16:26:22.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450005350 0x7f1468141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14681559d0 0x7f1468070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1468155e50 0x7f14680714c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14681549b0 con 0x7f1468155e50 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14681546d0 con 0x7f1450005350 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1467fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1468154ef0 con 0x7f14681559d0 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1468155e50 0x7f14680714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14677fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1468155e50 0x7f14680714c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48766/0 (socket says 172.21.15.119:48766) 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14677fe700 1 -- 172.21.15.119:0/2789407974 learned_addr learned my addr 172.21.15.119:0/2789407974 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f1466ffd700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450005350 0x7f1468141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14667fc700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14681559d0 0x7f1468070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14677fe700 1 -- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14681559d0 msgr2=0x7f1468070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14677fe700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14681559d0 0x7f1468070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14677fe700 1 -- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450005350 msgr2=0x7f1468141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.416+0000 7f14677fe700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450005350 0x7f1468141dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f14677fe700 1 -- 172.21.15.119:0/2789407974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14681551d0 con 0x7f1468155e50 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f1466ffd700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450005350 0x7f1468141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f14677fe700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1468155e50 0x7f14680714c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1458009b40 tx=0x7f14580047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f144f7fe700 1 -- 172.21.15.119:0/2789407974 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1458004020 con 0x7f1468155e50 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f144f7fe700 1 -- 172.21.15.119:0/2789407974 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145800e8e0 con 0x7f1468155e50 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f144f7fe700 1 -- 172.21.15.119:0/2789407974 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145800ec90 con 0x7f1468155e50 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14681554b0 con 0x7f1468155e50 2024-09-13T16:26:22.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14681539a0 con 0x7f1468155e50 2024-09-13T16:26:22.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1468154d50 con 0x7f1468155e50 2024-09-13T16:26:22.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.420+0000 7f144f7fe700 1 -- 172.21.15.119:0/2789407974 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1458013470 con 0x7f1468155e50 2024-09-13T16:26:22.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.424+0000 7f144f7fe700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1454066760 0x7f1454068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.424+0000 7f144f7fe700 1 -- 172.21.15.119:0/2789407974 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f14580853e0 con 0x7f1468155e50 2024-09-13T16:26:22.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.424+0000 7f1466ffd700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1454066760 0x7f1454068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.424+0000 7f1466ffd700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1454066760 0x7f1454068c20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1460000f80 tx=0x7f1460006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.428+0000 7f144f7fe700 1 -- 172.21.15.119:0/2789407974 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f14580526e0 con 0x7f1468155e50 2024-09-13T16:26:22.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.600+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1468154d50 con 0x7f1468155e50 2024-09-13T16:26:22.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f144f7fe700 1 -- 172.21.15.119:0/2789407974 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f1468154d50 con 0x7f1468155e50 2024-09-13T16:26:22.609 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:22.609 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1454066760 msgr2=0x7f1454068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1454066760 0x7f1454068c20 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f1460000f80 tx=0x7f1460006040 comp rx=0 tx=0).stop 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1468155e50 msgr2=0x7f14680714c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1468155e50 0x7f14680714c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1458009b40 tx=0x7f14580047e0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 shutdown_connections 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1454066760 0x7f1454068c20 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1468155e50 0x7f14680714c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14681559d0 0x7f1468070f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 --2- 172.21.15.119:0/2789407974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1450005350 0x7f1468141dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 >> 172.21.15.119:0/2789407974 conn(0x7f146806a350 msgr2=0x7f14680cdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:22.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 shutdown_connections 2024-09-13T16:26:22.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.604+0000 7f1467fff700 1 -- 172.21.15.119:0/2789407974 wait complete. 2024-09-13T16:26:22.627 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:25:14.874497+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:25:15.904673+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-09-13T16:25:18.965554+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '25', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-09-13T16:26:22.627 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:22.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8fcd07700 1 Processor -- start 2024-09-13T16:26:22.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8fcd07700 1 -- start start 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8fcd07700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cd690 0x7ff8f80cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8fcd07700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80b2690 con 0x7ff8f815e430 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8fcd07700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f8071a10 con 0x7ff8f80cb210 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8fcd07700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f80b2970 con 0x7ff8f80cd690 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f659c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff8f815e430 0x7ff8f815e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51246/0 (socket says 172.21.15.119:51246) 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f659c700 1 -- 172.21.15.119:0/2472653562 learned_addr learned my addr 172.21.15.119:0/2472653562 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f6d9d700 1 --2- 172.21.15.119:0/2472653562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cd690 0x7ff8f80cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4154765323 0 0) 0x7ff8f80b2690 con 0x7ff8f815e430 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e00035d0 con 0x7ff8f815e430 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1147129332 0 0) 0x7ff8f8071a10 con 0x7ff8f80cb210 2024-09-13T16:26:22.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80b2690 con 0x7ff8f80cb210 2024-09-13T16:26:22.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1562581939 0 0) 0x7ff8e00035d0 con 0x7ff8f815e430 2024-09-13T16:26:22.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f6d9d700 1 -- 172.21.15.119:0/2472653562 >> v1:172.21.15.145:6789/0 conn(0x7ff8f80cb210 legacy=0x7ff8f80ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f6d9d700 1 -- 172.21.15.119:0/2472653562 >> v1:172.21.15.119:6789/0 conn(0x7ff8f815e430 legacy=0x7ff8f815e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.736+0000 7ff8f6d9d700 1 -- 172.21.15.119:0/2472653562 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f8071cf0 con 0x7ff8f80cd690 2024-09-13T16:26:22.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8f6d9d700 1 --2- 172.21.15.119:0/2472653562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cd690 0x7ff8f80cdb20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0009dc0 tx=0x7ff8f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=73ae657c2e5988bf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8f000e070 con 0x7ff8f80cd690 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f0003a40 con 0x7ff8f80cd690 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8f559a700 1 -- 172.21.15.119:0/2472653562 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8ec001670 con 0x7ff8f80cd690 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 -- 172.21.15.119:0/2472653562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cd690 msgr2=0x7ff8f80cdb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/2472653562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cd690 0x7ff8f80cdb20 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0009dc0 tx=0x7ff8f000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 -- 172.21.15.119:0/2472653562 shutdown_connections 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/2472653562 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cd690 0x7ff8f80cdb20 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 -- 172.21.15.119:0/2472653562 >> 172.21.15.119:0/2472653562 conn(0x7ff8f80b0560 msgr2=0x7ff8f80b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 -- 172.21.15.119:0/2472653562 shutdown_connections 2024-09-13T16:26:22.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 -- 172.21.15.119:0/2472653562 wait complete. 2024-09-13T16:26:22.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 Processor -- start 2024-09-13T16:26:22.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.740+0000 7ff8fcd07700 1 -- start start 2024-09-13T16:26:22.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cb210 0x7ff8f81564c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8f80cd690 0x7ff8f806c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8f815e430 0x7ff8f806cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f806d370 con 0x7ff8f80cd690 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f80c3970 con 0x7ff8f815e430 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f80c3c50 con 0x7ff8f80cb210 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8f80cd690 0x7ff8f806c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8f80cd690 0x7ff8f806c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48770/0 (socket says 172.21.15.119:48770) 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 -- 172.21.15.119:0/3942035545 learned_addr learned my addr 172.21.15.119:0/3942035545 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f659c700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cb210 0x7ff8f81564c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f6d9d700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8f815e430 0x7ff8f806cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 -- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cb210 msgr2=0x7ff8f81564c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cb210 0x7ff8f81564c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 -- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8f815e430 msgr2=0x7ff8f806cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8f815e430 0x7ff8f806cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 -- 172.21.15.119:0/3942035545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0009a70 con 0x7ff8f80cd690 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f6d9d700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8f815e430 0x7ff8f806cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8f5d9b700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8f80cd690 0x7ff8f806c850 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c730 tx=0x7ff8e800ca40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8e6ffd700 1 -- 172.21.15.119:0/3942035545 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80052d0 con 0x7ff8f80cd690 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8e6ffd700 1 -- 172.21.15.119:0/3942035545 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e8005450 con 0x7ff8f80cd690 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f80c3f30 con 0x7ff8f80cd690 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8e6ffd700 1 -- 172.21.15.119:0/3942035545 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80038e0 con 0x7ff8f80cd690 2024-09-13T16:26:22.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f80aed30 con 0x7ff8f80cd690 2024-09-13T16:26:22.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.744+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f8071a10 con 0x7ff8f80cd690 2024-09-13T16:26:22.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.748+0000 7ff8e6ffd700 1 -- 172.21.15.119:0/3942035545 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff8e8003a40 con 0x7ff8f80cd690 2024-09-13T16:26:22.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.748+0000 7ff8e6ffd700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff8d8066730 0x7ff8d8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:22.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.748+0000 7ff8e6ffd700 1 -- 172.21.15.119:0/3942035545 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7ff8e8083a90 con 0x7ff8f80cd690 2024-09-13T16:26:22.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.748+0000 7ff8f659c700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff8d8066730 0x7ff8d8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:22.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.748+0000 7ff8f659c700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff8d8066730 0x7ff8d8068bf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec004a80 tx=0x7ff8ec000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:22.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.752+0000 7ff8e6ffd700 1 -- 172.21.15.119:0/3942035545 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff8e8050a50 con 0x7ff8f80cd690 2024-09-13T16:26:22.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.920+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff8f80af6f0 con 0x7ff8f80cd690 2024-09-13T16:26:22.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.920+0000 7ff8e6ffd700 1 -- 172.21.15.119:0/3942035545 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v27) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff8e8054070 con 0x7ff8f80cd690 2024-09-13T16:26:22.928 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:22.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff8d8066730 msgr2=0x7ff8d8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff8d8066730 0x7ff8d8068bf0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff8ec004a80 tx=0x7ff8ec000bc0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8f80cd690 msgr2=0x7ff8f806c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:22.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8f80cd690 0x7ff8f806c850 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800c730 tx=0x7ff8e800ca40 comp rx=0 tx=0).stop 2024-09-13T16:26:22.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 shutdown_connections 2024-09-13T16:26:22.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff8d8066730 0x7ff8d8068bf0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8f815e430 0x7ff8f806cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8f80cd690 0x7ff8f806c850 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 --2- 172.21.15.119:0/3942035545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8f80cb210 0x7ff8f81564c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 >> 172.21.15.119:0/3942035545 conn(0x7ff8f80b0560 msgr2=0x7ff8f80ccf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 shutdown_connections 2024-09-13T16:26:22.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:22.924+0000 7ff8fcd07700 1 -- 172.21.15.119:0/3942035545 wait complete. 2024-09-13T16:26:22.941 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:23.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.048+0000 7fcceb59e700 1 Processor -- start 2024-09-13T16:26:23.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 -- start start 2024-09-13T16:26:23.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fccec0ce8c0 0x7fccec0d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0d1a20 con 0x7fccec15cc00 2024-09-13T16:26:23.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccec0d0df0 con 0x7fccec0d1610 2024-09-13T16:26:23.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccec0cdb00 con 0x7fccec0ce8c0 2024-09-13T16:26:23.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fccea59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fccec15cc00 0x7fccec0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51258/0 (socket says 172.21.15.119:51258) 2024-09-13T16:26:23.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce9d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fccec0ce8c0 0x7fccec0d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fccea59c700 1 -- 172.21.15.119:0/386215380 learned_addr learned my addr 172.21.15.119:0/386215380 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:23.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce9d9b700 1 -- 172.21.15.119:0/386215380 >> v1:172.21.15.145:6789/0 conn(0x7fccec0d1610 legacy=0x7fccec0b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce9d9b700 1 -- 172.21.15.119:0/386215380 >> v1:172.21.15.119:6789/0 conn(0x7fccec15cc00 legacy=0x7fccec0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce959a700 1 -- 172.21.15.119:0/386215380 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 16906903 0 0) 0x7fccec0d1a20 con 0x7fccec15cc00 2024-09-13T16:26:23.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce9d9b700 1 -- 172.21.15.119:0/386215380 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccec0cdde0 con 0x7fccec0ce8c0 2024-09-13T16:26:23.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce9d9b700 1 --2- 172.21.15.119:0/386215380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0ce8c0 0x7fccec0d10d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009850 tx=0x7fccdc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491333a372fae951 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce959a700 1 -- 172.21.15.119:0/386215380 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc011830 con 0x7fccec0ce8c0 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce959a700 1 -- 172.21.15.119:0/386215380 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc011990 con 0x7fccec0ce8c0 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcce959a700 1 -- 172.21.15.119:0/386215380 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc011d20 con 0x7fccec0ce8c0 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 -- 172.21.15.119:0/386215380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0ce8c0 msgr2=0x7fccec0d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 --2- 172.21.15.119:0/386215380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0ce8c0 0x7fccec0d10d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fccdc009850 tx=0x7fccdc009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 -- 172.21.15.119:0/386215380 shutdown_connections 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 --2- 172.21.15.119:0/386215380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0ce8c0 0x7fccec0d10d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.052+0000 7fcceb59e700 1 -- 172.21.15.119:0/386215380 >> 172.21.15.119:0/386215380 conn(0x7fccec08c080 msgr2=0x7fccec0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.056+0000 7fcceb59e700 1 -- 172.21.15.119:0/386215380 shutdown_connections 2024-09-13T16:26:23.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.056+0000 7fcceb59e700 1 -- 172.21.15.119:0/386215380 wait complete. 2024-09-13T16:26:23.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.056+0000 7fcceb59e700 1 Processor -- start 2024-09-13T16:26:23.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.056+0000 7fcceb59e700 1 -- start start 2024-09-13T16:26:23.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccec0ce8c0 0x7fccec157400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0d1610 0x7fccec1559f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccec15cc00 0x7fccec1546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec1587f0 con 0x7fccec0d1610 2024-09-13T16:26:23.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccec157940 con 0x7fccec15cc00 2024-09-13T16:26:23.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccec158c10 con 0x7fccec0ce8c0 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0d1610 0x7fccec1559f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0d1610 0x7fccec1559f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48782/0 (socket says 172.21.15.119:48782) 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 -- 172.21.15.119:0/231810688 learned_addr learned my addr 172.21.15.119:0/231810688 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fccea59c700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccec0ce8c0 0x7fccec157400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fccead9d700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccec15cc00 0x7fccec1546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 -- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccec0ce8c0 msgr2=0x7fccec157400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccec0ce8c0 0x7fccec157400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 -- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccec15cc00 msgr2=0x7fccec1546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccec15cc00 0x7fccec1546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 -- 172.21.15.119:0/231810688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccdc0094b0 con 0x7fccec0d1610 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fccead9d700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccec15cc00 0x7fccec1546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:23.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcce9d9b700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0d1610 0x7fccec1559f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fccec0d1bd0 tx=0x7fccdc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fccdaffd700 1 -- 172.21.15.119:0/231810688 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc004590 con 0x7fccec0d1610 2024-09-13T16:26:23.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fccdaffd700 1 -- 172.21.15.119:0/231810688 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc01f070 con 0x7fccec0d1610 2024-09-13T16:26:23.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccec157b70 con 0x7fccec0d1610 2024-09-13T16:26:23.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fccdaffd700 1 -- 172.21.15.119:0/231810688 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccdc003900 con 0x7fccec0d1610 2024-09-13T16:26:23.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccec158130 con 0x7fccec0d1610 2024-09-13T16:26:23.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccec158930 con 0x7fccec0d1610 2024-09-13T16:26:23.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.060+0000 7fccdaffd700 1 -- 172.21.15.119:0/231810688 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fccdc003a60 con 0x7fccec0d1610 2024-09-13T16:26:23.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.064+0000 7fccdaffd700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcccc066730 0x7fcccc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.064+0000 7fccdaffd700 1 -- 172.21.15.119:0/231810688 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fccdc021070 con 0x7fccec0d1610 2024-09-13T16:26:23.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.064+0000 7fccea59c700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcccc066730 0x7fcccc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.064+0000 7fccea59c700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcccc066730 0x7fcccc068bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcce0000f60 tx=0x7fcce0007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.072+0000 7fccdaffd700 1 -- 172.21.15.119:0/231810688 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fccec158930 con 0x7fccec0d1610 2024-09-13T16:26:23.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fccec158930 con 0x7fccec0d1610 2024-09-13T16:26:23.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fccdaffd700 1 -- 172.21.15.119:0/231810688 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v27) v1 ==== 104+0+11 (secure 0 0 0) 0x7fccdc015020 con 0x7fccec0d1610 2024-09-13T16:26:23.245 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:23.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcccc066730 msgr2=0x7fcccc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcccc066730 0x7fcccc068bf0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fcce0000f60 tx=0x7fcce0007e90 comp rx=0 tx=0).stop 2024-09-13T16:26:23.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0d1610 msgr2=0x7fccec1559f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0d1610 0x7fccec1559f0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fccec0d1bd0 tx=0x7fccdc004770 comp rx=0 tx=0).stop 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 shutdown_connections 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcccc066730 0x7fcccc068bf0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccec15cc00 0x7fccec1546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccec0d1610 0x7fccec1559f0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 --2- 172.21.15.119:0/231810688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccec0ce8c0 0x7fccec157400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.240+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 >> 172.21.15.119:0/231810688 conn(0x7fccec08c080 msgr2=0x7fccec0b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.244+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 shutdown_connections 2024-09-13T16:26:23.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.244+0000 7fcceb59e700 1 -- 172.21.15.119:0/231810688 wait complete. 2024-09-13T16:26:23.259 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:23.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.376+0000 7f0504894700 1 Processor -- start 2024-09-13T16:26:23.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.376+0000 7f0504894700 1 -- start start 2024-09-13T16:26:23.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.376+0000 7f0504894700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f050015cc00 0x7f05000d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.376+0000 7f0504894700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05000cdb00 con 0x7f05000d17e0 2024-09-13T16:26:23.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.376+0000 7f0504894700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05000cdde0 con 0x7f05000d0c20 2024-09-13T16:26:23.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.376+0000 7f0504894700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05000ce0c0 con 0x7f050015cc00 2024-09-13T16:26:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f05000d17e0 0x7f05000b5960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51272/0 (socket says 172.21.15.119:51272) 2024-09-13T16:26:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 -- 172.21.15.119:0/2328753109 learned_addr learned my addr 172.21.15.119:0/2328753109 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04ff7fe700 1 --2- 172.21.15.119:0/2328753109 >> v2:172.21.15.145:3300/0 conn(0x7f050015cc00 0x7f05000d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437720246 0 0) 0x7f05000cdb00 con 0x7f05000d17e0 2024-09-13T16:26:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec0035d0 con 0x7f05000d17e0 2024-09-13T16:26:23.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3938008544 0 0) 0x7f05000cdde0 con 0x7f05000d0c20 2024-09-13T16:26:23.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05000cdb00 con 0x7f05000d0c20 2024-09-13T16:26:23.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1084885016 0 0) 0x7f04ec0035d0 con 0x7f05000d17e0 2024-09-13T16:26:23.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 >> v2:172.21.15.145:3300/0 conn(0x7f050015cc00 msgr2=0x7f05000d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:23.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 --2- 172.21.15.119:0/2328753109 >> v2:172.21.15.145:3300/0 conn(0x7f050015cc00 0x7f05000d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 >> v1:172.21.15.145:6789/0 conn(0x7f05000d0c20 legacy=0x7f05000d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05000ce3a0 con 0x7f05000d17e0 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f04f80019f0 con 0x7f05000d17e0 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 >> v1:172.21.15.119:6789/0 conn(0x7f05000d17e0 legacy=0x7f05000b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec003c00 0x7f04ec004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04ec004870 0x7f04ec004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec005230 0x7f04ec00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05000ce0c0 con 0x7f04ec005230 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec00c100 con 0x7f04ec003c00 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04ec00c2c0 con 0x7f04ec004870 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec005230 0x7f04ec00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04ff7fe700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec003c00 0x7f04ec004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04feffd700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04ec004870 0x7f04ec004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 -- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04ec004870 msgr2=0x7f04ec004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04ec004870 0x7f04ec004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 -- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec003c00 msgr2=0x7f04ec004000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec003c00 0x7f04ec004000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 -- 172.21.15.119:0/2328753109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec0035d0 con 0x7f04ec005230 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 -- 172.21.15.119:0/2328753109 reap_dead start 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fffff700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec005230 0x7f04ec00ba30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f05001607e0 tx=0x7f04ec0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c45bfdc6d57748c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8001db0 con 0x7f04ec005230 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f80070d0 con 0x7f04ec005230 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f04fe7fc700 1 -- 172.21.15.119:0/2328753109 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8007480 con 0x7f04ec005230 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f0504894700 1 -- 172.21.15.119:0/2328753109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec005230 msgr2=0x7f04ec00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f0504894700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec005230 0x7f04ec00ba30 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f05001607e0 tx=0x7f04ec0058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f0504894700 1 -- 172.21.15.119:0/2328753109 shutdown_connections 2024-09-13T16:26:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f0504894700 1 --2- 172.21.15.119:0/2328753109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04ec005230 0x7f04ec00ba30 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f0504894700 1 -- 172.21.15.119:0/2328753109 >> 172.21.15.119:0/2328753109 conn(0x7f050008c080 msgr2=0x7f05000d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:23.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f0504894700 1 -- 172.21.15.119:0/2328753109 shutdown_connections 2024-09-13T16:26:23.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.380+0000 7f0504894700 1 -- 172.21.15.119:0/2328753109 wait complete. 2024-09-13T16:26:23.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 Processor -- start 2024-09-13T16:26:23.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 -- start start 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec005230 0x7f05000c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05000cbb30 0x7f05000b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0500154a80 0x7f0500154f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0500156c80 con 0x7f0500154a80 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0500155440 con 0x7f04ec005230 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.384+0000 7f0504894700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05000c7120 con 0x7f05000cbb30 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0500154a80 0x7f0500154f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0500154a80 0x7f0500154f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48800/0 (socket says 172.21.15.119:48800) 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 -- 172.21.15.119:0/1942507412 learned_addr learned my addr 172.21.15.119:0/1942507412 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04ff7fe700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec005230 0x7f05000c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04feffd700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05000cbb30 0x7f05000b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 -- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05000cbb30 msgr2=0x7f05000b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05000cbb30 0x7f05000b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 -- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec005230 msgr2=0x7f05000c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec005230 0x7f05000c6820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 -- 172.21.15.119:0/1942507412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05000c7400 con 0x7f0500154a80 2024-09-13T16:26:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04fffff700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0500154a80 0x7f0500154f00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f050008c530 tx=0x7f04f8001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04f7fff700 1 -- 172.21.15.119:0/1942507412 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8007830 con 0x7f0500154a80 2024-09-13T16:26:23.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04f7fff700 1 -- 172.21.15.119:0/1942507412 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f8007990 con 0x7f0500154a80 2024-09-13T16:26:23.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f04f7fff700 1 -- 172.21.15.119:0/1942507412 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f801d410 con 0x7f0500154a80 2024-09-13T16:26:23.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0500156ed0 con 0x7f0500154a80 2024-09-13T16:26:23.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05000c6d60 con 0x7f0500154a80 2024-09-13T16:26:23.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.388+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05000c6ef0 con 0x7f0500154a80 2024-09-13T16:26:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.392+0000 7f04f7fff700 1 -- 172.21.15.119:0/1942507412 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04f801a030 con 0x7f0500154a80 2024-09-13T16:26:23.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.392+0000 7f04f7fff700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066760 0x7f04e0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.392+0000 7f04f7fff700 1 -- 172.21.15.119:0/1942507412 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f04f80857c0 con 0x7f0500154a80 2024-09-13T16:26:23.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.392+0000 7f04ff7fe700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066760 0x7f04e0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.392+0000 7f04ff7fe700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066760 0x7f04e0068c20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04e8009bf0 tx=0x7f04e8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.396+0000 7f04f7fff700 1 -- 172.21.15.119:0/1942507412 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f04f80528c0 con 0x7f0500154a80 2024-09-13T16:26:23.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.564+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f05001573f0 con 0x7f0500154a80 2024-09-13T16:26:23.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f04f7fff700 1 -- 172.21.15.119:0/1942507412 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v27) v1 ==== 100+0+11 (secure 0 0 0) 0x7f05001573f0 con 0x7f0500154a80 2024-09-13T16:26:23.572 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066760 msgr2=0x7f04e0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066760 0x7f04e0068c20 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f04e8009bf0 tx=0x7f04e8009240 comp rx=0 tx=0).stop 2024-09-13T16:26:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0500154a80 msgr2=0x7f0500154f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0500154a80 0x7f0500154f00 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f050008c530 tx=0x7f04f8001cb0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 shutdown_connections 2024-09-13T16:26:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066760 0x7f04e0068c20 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0500154a80 0x7f0500154f00 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05000cbb30 0x7f05000b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 --2- 172.21.15.119:0/1942507412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04ec005230 0x7f05000c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 >> 172.21.15.119:0/1942507412 conn(0x7f050008c080 msgr2=0x7f050006bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 shutdown_connections 2024-09-13T16:26:23.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.568+0000 7f0504894700 1 -- 172.21.15.119:0/1942507412 wait complete. 2024-09-13T16:26:23.587 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-09-13T16:26:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8476202700 1 Processor -- start 2024-09-13T16:26:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8476202700 1 -- start start 2024-09-13T16:26:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8476202700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f84680eb420 0x7f846804ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8476202700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468007260 0x7f846804f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8476202700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f846801cfa0 con 0x7f8468033ac0 2024-09-13T16:26:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8476202700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f846801c670 con 0x7f84680eb420 2024-09-13T16:26:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8476202700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846801c950 con 0x7f8468007260 2024-09-13T16:26:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8475200700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8468033ac0 0x7f8468009ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51276/0 (socket says 172.21.15.119:51276) 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8475200700 1 -- 172.21.15.119:0/473880946 learned_addr learned my addr 172.21.15.119:0/473880946 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f84749ff700 1 --2- 172.21.15.119:0/473880946 >> v2:172.21.15.119:3300/0 conn(0x7f84680eb420 0x7f846804ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8475a01700 1 --2- 172.21.15.119:0/473880946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468007260 0x7f846804f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8467fff700 1 -- 172.21.15.119:0/473880946 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011624442 0 0) 0x7f846801cfa0 con 0x7f8468033ac0 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f8467fff700 1 -- 172.21.15.119:0/473880946 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c0035d0 con 0x7f8468033ac0 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f84749ff700 1 -- 172.21.15.119:0/473880946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468007260 msgr2=0x7f846804f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f84749ff700 1 --2- 172.21.15.119:0/473880946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468007260 0x7f846804f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f84749ff700 1 -- 172.21.15.119:0/473880946 >> v1:172.21.15.119:6789/0 conn(0x7f8468033ac0 legacy=0x7f8468009ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.696+0000 7f84749ff700 1 -- 172.21.15.119:0/473880946 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846801cc30 con 0x7f84680eb420 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f84749ff700 1 --2- 172.21.15.119:0/473880946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680eb420 0x7f846804ec10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8460009850 tx=0x7f8460009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1fc6bed9e347ee9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8467fff700 1 -- 172.21.15.119:0/473880946 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460011830 con 0x7f84680eb420 2024-09-13T16:26:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8467fff700 1 -- 172.21.15.119:0/473880946 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8460011990 con 0x7f84680eb420 2024-09-13T16:26:23.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8467fff700 1 -- 172.21.15.119:0/473880946 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8460011d40 con 0x7f84680eb420 2024-09-13T16:26:23.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- 172.21.15.119:0/473880946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680eb420 msgr2=0x7f846804ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 --2- 172.21.15.119:0/473880946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680eb420 0x7f846804ec10 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f8460009850 tx=0x7f8460009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- 172.21.15.119:0/473880946 shutdown_connections 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 --2- 172.21.15.119:0/473880946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468007260 0x7f846804f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 --2- 172.21.15.119:0/473880946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680eb420 0x7f846804ec10 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- 172.21.15.119:0/473880946 >> 172.21.15.119:0/473880946 conn(0x7f8468036140 msgr2=0x7f8468036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- 172.21.15.119:0/473880946 shutdown_connections 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- 172.21.15.119:0/473880946 wait complete. 2024-09-13T16:26:23.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 Processor -- start 2024-09-13T16:26:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- start start 2024-09-13T16:26:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8468007260 0x7f846817dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468033ac0 0x7f846804ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8468179b80 0x7f8468179fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846817a520 con 0x7f8468179b80 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846804c660 con 0x7f8468007260 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.700+0000 7f8476202700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8468048250 con 0x7f8468033ac0 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8468179b80 0x7f8468179fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8468179b80 0x7f8468179fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48820/0 (socket says 172.21.15.119:48820) 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 -- 172.21.15.119:0/705156158 learned_addr learned my addr 172.21.15.119:0/705156158 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f84749ff700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468033ac0 0x7f846804ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475200700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8468007260 0x7f846817dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 -- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468033ac0 msgr2=0x7f846804ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468033ac0 0x7f846804ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 -- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8468007260 msgr2=0x7f846817dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8468007260 0x7f846817dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 -- 172.21.15.119:0/705156158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84600094b0 con 0x7f8468179b80 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8475a01700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8468179b80 0x7f8468179fe0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f846c00e810 tx=0x7f846c00ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8465ffb700 1 -- 172.21.15.119:0/705156158 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c013610 con 0x7f8468179b80 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8468048530 con 0x7f8468179b80 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8465ffb700 1 -- 172.21.15.119:0/705156158 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c013790 con 0x7f8468179b80 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846804b1d0 con 0x7f8468179b80 2024-09-13T16:26:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8465ffb700 1 -- 172.21.15.119:0/705156158 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c013b00 con 0x7f8468179b80 2024-09-13T16:26:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8465ffb700 1 -- 172.21.15.119:0/705156158 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f846c013c60 con 0x7f8468179b80 2024-09-13T16:26:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8468052f80 con 0x7f8468179b80 2024-09-13T16:26:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.704+0000 7f8465ffb700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066730 0x7f8454068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.708+0000 7f8465ffb700 1 -- 172.21.15.119:0/705156158 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f846c0841b0 con 0x7f8468179b80 2024-09-13T16:26:23.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.708+0000 7f8475200700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066730 0x7f8454068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:23.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.708+0000 7f8475200700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066730 0x7f8454068bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f847004ff70 tx=0x7f8470065ea0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:23.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.712+0000 7f8465ffb700 1 -- 172.21.15.119:0/705156158 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f846c0510e0 con 0x7f8468179b80 2024-09-13T16:26:23.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.876+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8468009ba0 con 0x7f8468179b80 2024-09-13T16:26:23.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.876+0000 7f8465ffb700 1 -- 172.21.15.119:0/705156158 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v27) v1 ==== 103+0+11 (secure 0 0 0) 0x7f846c054700 con 0x7f8468179b80 2024-09-13T16:26:23.883 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:23.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066730 msgr2=0x7f8454068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066730 0x7f8454068bf0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f847004ff70 tx=0x7f8470065ea0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8468179b80 msgr2=0x7f8468179fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:23.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8468179b80 0x7f8468179fe0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f846c00e810 tx=0x7f846c00ebd0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 shutdown_connections 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066730 0x7f8454068bf0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8468179b80 0x7f8468179fe0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8468033ac0 0x7f846804ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 --2- 172.21.15.119:0/705156158 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8468007260 0x7f846817dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 >> 172.21.15.119:0/705156158 conn(0x7f8468036140 msgr2=0x7f8468052190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 shutdown_connections 2024-09-13T16:26:23.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:23.880+0000 7f8476202700 1 -- 172.21.15.119:0/705156158 wait complete. 2024-09-13T16:26:23.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:24.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3742af6700 1 Processor -- start 2024-09-13T16:26:24.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3742af6700 1 -- start start 2024-09-13T16:26:24.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3742af6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c06dc40 0x7f373c0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3742af6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f373c0b3630 0x7f373c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3742af6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f373c0b2280 con 0x7f373c1594f0 2024-09-13T16:26:24.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3742af6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f373c0cc890 con 0x7f373c0b3630 2024-09-13T16:26:24.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3742af6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373c0b3ac0 con 0x7f373c06dc40 2024-09-13T16:26:24.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f37412f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c06dc40 0x7f373c0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f37422f5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f373c0b3630 0x7f373c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f37412f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c06dc40 0x7f373c0b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:58586/0 (socket says 172.21.15.119:58586) 2024-09-13T16:26:24.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f3741af4700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f373c1594f0 0x7f373c06d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:37722/0 (socket says 172.21.15.119:37722) 2024-09-13T16:26:24.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.008+0000 7f37412f3700 1 -- 172.21.15.119:0/2817523194 learned_addr learned my addr 172.21.15.119:0/2817523194 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f37412f3700 1 -- 172.21.15.119:0/2817523194 >> v2:172.21.15.145:3300/0 conn(0x7f373c0b3630 msgr2=0x7f373c06fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f37412f3700 1 --2- 172.21.15.119:0/2817523194 >> v2:172.21.15.145:3300/0 conn(0x7f373c0b3630 0x7f373c06fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f37412f3700 1 -- 172.21.15.119:0/2817523194 >> v1:172.21.15.145:6789/0 conn(0x7f373c1594f0 legacy=0x7f373c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f37412f3700 1 -- 172.21.15.119:0/2817523194 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f373c0b1970 con 0x7f373c06dc40 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f37422f5700 1 --2- 172.21.15.119:0/2817523194 >> v2:172.21.15.145:3300/0 conn(0x7f373c0b3630 0x7f373c06fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f37412f3700 1 --2- 172.21.15.119:0/2817523194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c06dc40 0x7f373c0b30f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f373c0b1b60 tx=0x7f3730009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba5c94293eb3d72 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3740af2700 1 -- 172.21.15.119:0/2817523194 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730004230 con 0x7f373c06dc40 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3740af2700 1 -- 172.21.15.119:0/2817523194 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3730004390 con 0x7f373c06dc40 2024-09-13T16:26:24.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3740af2700 1 -- 172.21.15.119:0/2817523194 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37300036c0 con 0x7f373c06dc40 2024-09-13T16:26:24.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 -- 172.21.15.119:0/2817523194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c06dc40 msgr2=0x7f373c0b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 --2- 172.21.15.119:0/2817523194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c06dc40 0x7f373c0b30f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f373c0b1b60 tx=0x7f3730009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 -- 172.21.15.119:0/2817523194 shutdown_connections 2024-09-13T16:26:24.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 --2- 172.21.15.119:0/2817523194 >> v2:172.21.15.145:3300/0 conn(0x7f373c0b3630 0x7f373c06fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 --2- 172.21.15.119:0/2817523194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c06dc40 0x7f373c0b30f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 -- 172.21.15.119:0/2817523194 >> 172.21.15.119:0/2817523194 conn(0x7f373c06a350 msgr2=0x7f373c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:24.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 -- 172.21.15.119:0/2817523194 shutdown_connections 2024-09-13T16:26:24.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 -- 172.21.15.119:0/2817523194 wait complete. 2024-09-13T16:26:24.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 Processor -- start 2024-09-13T16:26:24.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.012+0000 7f3742af6700 1 -- start start 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c06dc40 0x7f373c14f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c0b3630 0x7f373c14fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f373c1594f0 0x7f373c156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373c1efd00 con 0x7f373c1594f0 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f373c154ea0 con 0x7f373c06dc40 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373c14d6d0 con 0x7f373c0b3630 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f373c1594f0 0x7f373c156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3741af4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c06dc40 0x7f373c14f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f373c1594f0 0x7f373c156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48824/0 (socket says 172.21.15.119:48824) 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3741af4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c06dc40 0x7f373c14f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:37124/0 (socket says 172.21.15.119:37124) 2024-09-13T16:26:24.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 -- 172.21.15.119:0/3232203103 learned_addr learned my addr 172.21.15.119:0/3232203103 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37412f3700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c0b3630 0x7f373c14fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 -- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c0b3630 msgr2=0x7f373c14fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c0b3630 0x7f373c14fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 -- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c06dc40 msgr2=0x7f373c14f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c06dc40 0x7f373c14f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 -- 172.21.15.119:0/3232203103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3734009a70 con 0x7f373c1594f0 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3741af4700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c06dc40 0x7f373c14f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:24.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f37422f5700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f373c1594f0 0x7f373c156ca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f373400efc0 tx=0x7f373400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f372e7fc700 1 -- 172.21.15.119:0/3232203103 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373400ca90 con 0x7f373c1594f0 2024-09-13T16:26:24.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f372e7fc700 1 -- 172.21.15.119:0/3232203103 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f373400f040 con 0x7f373c1594f0 2024-09-13T16:26:24.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3730009570 con 0x7f373c1594f0 2024-09-13T16:26:24.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f372e7fc700 1 -- 172.21.15.119:0/3232203103 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f373400ce10 con 0x7f373c1594f0 2024-09-13T16:26:24.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373c150300 con 0x7f373c1594f0 2024-09-13T16:26:24.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.016+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f373c155030 con 0x7f373c1594f0 2024-09-13T16:26:24.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.020+0000 7f372e7fc700 1 -- 172.21.15.119:0/3232203103 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3734014480 con 0x7f373c1594f0 2024-09-13T16:26:24.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.020+0000 7f372e7fc700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3720066730 0x7f3720068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.020+0000 7f372e7fc700 1 -- 172.21.15.119:0/3232203103 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3734084b30 con 0x7f373c1594f0 2024-09-13T16:26:24.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.020+0000 7f3741af4700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3720066730 0x7f3720068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.020+0000 7f3741af4700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3720066730 0x7f3720068bf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3738000f60 tx=0x7f3738007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.028+0000 7f372e7fc700 1 -- 172.21.15.119:0/3232203103 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3734051ae0 con 0x7f373c1594f0 2024-09-13T16:26:24.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.236+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f373c0b30f0 con 0x7f373c1594f0 2024-09-13T16:26:24.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.236+0000 7f372e7fc700 1 -- 172.21.15.119:0/3232203103 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f3734009d70 con 0x7f373c1594f0 2024-09-13T16:26:24.241 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:24.241 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:26:24.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3720066730 msgr2=0x7f3720068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3720066730 0x7f3720068bf0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f3738000f60 tx=0x7f3738007ef0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f373c1594f0 msgr2=0x7f373c156ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f373c1594f0 0x7f373c156ca0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f373400efc0 tx=0x7f373400c3a0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 shutdown_connections 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3720066730 0x7f3720068bf0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f373c1594f0 0x7f373c156ca0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f373c0b3630 0x7f373c14fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 --2- 172.21.15.119:0/3232203103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f373c06dc40 0x7f373c14f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 >> 172.21.15.119:0/3232203103 conn(0x7f373c06a350 msgr2=0x7f373c06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 shutdown_connections 2024-09-13T16:26:24.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.240+0000 7f3742af6700 1 -- 172.21.15.119:0/3232203103 wait complete. 2024-09-13T16:26:24.247 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:26:24.262 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:24.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1c72700 1 Processor -- start 2024-09-13T16:26:24.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1c72700 1 -- start start 2024-09-13T16:26:24.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1c72700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 0x7f39bc165450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1c72700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39bc06cf60 con 0x7f39bc0cc060 2024-09-13T16:26:24.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1c72700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39bc0cc4d0 con 0x7f39bc0adb30 2024-09-13T16:26:24.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1c72700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0b3390 con 0x7f39bc0ce040 2024-09-13T16:26:24.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bbfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f39bc0cc060 0x7f39bc0cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51288/0 (socket says 172.21.15.119:51288) 2024-09-13T16:26:24.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bbfff700 1 -- 172.21.15.119:0/2222383861 learned_addr learned my addr 172.21.15.119:0/2222383861 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:24.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1471700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 0x7f39bc165450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898549801 0 0) 0x7f39bc06cf60 con 0x7f39bc0cc060 2024-09-13T16:26:24.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39a40035d0 con 0x7f39bc0cc060 2024-09-13T16:26:24.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958585194 0 0) 0x7f39bc0cc4d0 con 0x7f39bc0adb30 2024-09-13T16:26:24.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39bc06cf60 con 0x7f39bc0adb30 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1932383552 0 0) 0x7f39a40035d0 con 0x7f39bc0cc060 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 msgr2=0x7f39bc165450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 0x7f39bc165450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 >> v1:172.21.15.145:6789/0 conn(0x7f39bc0adb30 legacy=0x7f39bc06bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc06c350 con 0x7f39bc0cc060 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f39b00019f0 con 0x7f39bc0cc060 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1471700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 0x7f39bc165450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 >> v1:172.21.15.119:6789/0 conn(0x7f39bc0cc060 legacy=0x7f39bc0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a4003be0 0x7f39a4003fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 0x7f39a4004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a4004db0 0x7f39a4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc0b3390 con 0x7f39a4003be0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a400bae0 con 0x7f39a4004db0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a4003be0 0x7f39a4003fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a400bc70 con 0x7f39bc0ce040 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39bbfff700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 0x7f39a4004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c1471700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a4004db0 0x7f39a4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c0c70700 1 -- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 msgr2=0x7f39a4004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39bc0ce040 0x7f39a4004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c0c70700 1 -- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a4004db0 msgr2=0x7f39a4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a4004db0 0x7f39a4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c0c70700 1 -- 172.21.15.119:0/2222383861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39b40094b0 con 0x7f39a4003be0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.372+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a4003be0 0x7f39a4003fc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f39ac001960 tx=0x7f39a40058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=555a40b0db32a450 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac001450 con 0x7f39a4003be0 2024-09-13T16:26:24.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39ac0052f0 con 0x7f39a4003be0 2024-09-13T16:26:24.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39bb7fe700 1 -- 172.21.15.119:0/2222383861 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac0036a0 con 0x7f39a4003be0 2024-09-13T16:26:24.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39c1c72700 1 -- 172.21.15.119:0/2222383861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a4003be0 msgr2=0x7f39a4003fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39c1c72700 1 --2- 172.21.15.119:0/2222383861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a4003be0 0x7f39a4003fc0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f39ac001960 tx=0x7f39a40058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39c1471700 1 -- 172.21.15.119:0/2222383861 reap_dead start 2024-09-13T16:26:24.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39c1c72700 1 -- 172.21.15.119:0/2222383861 shutdown_connections 2024-09-13T16:26:24.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39c1c72700 1 -- 172.21.15.119:0/2222383861 >> 172.21.15.119:0/2222383861 conn(0x7f39bc06a350 msgr2=0x7f39bc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:24.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39c1c72700 1 -- 172.21.15.119:0/2222383861 shutdown_connections 2024-09-13T16:26:24.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.376+0000 7f39c1c72700 1 -- 172.21.15.119:0/2222383861 wait complete. 2024-09-13T16:26:24.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 Processor -- start 2024-09-13T16:26:24.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 -- start start 2024-09-13T16:26:24.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a4003be0 0x7f39bc15d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39bc06ffb0 0x7f39bc13c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc0714e0 0x7f39bc0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc071050 con 0x7f39bc0714e0 2024-09-13T16:26:24.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc15dde0 con 0x7f39bc06ffb0 2024-09-13T16:26:24.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39bc070440 con 0x7f39a4003be0 2024-09-13T16:26:24.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc0714e0 0x7f39bc0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc0714e0 0x7f39bc0ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48834/0 (socket says 172.21.15.119:48834) 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 -- 172.21.15.119:0/2239202007 learned_addr learned my addr 172.21.15.119:0/2239202007 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39bbfff700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39bc06ffb0 0x7f39bc13c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a4003be0 0x7f39bc15d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 -- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a4003be0 msgr2=0x7f39bc15d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a4003be0 0x7f39bc15d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 -- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39bc06ffb0 msgr2=0x7f39bc13c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39bc06ffb0 0x7f39bc13c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 -- 172.21.15.119:0/2239202007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39bc070720 con 0x7f39bc0714e0 2024-09-13T16:26:24.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a4003be0 0x7f39bc15d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:24.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1471700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc0714e0 0x7f39bc0ac8c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0cb1e0 tx=0x7f39b4011eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39b97fa700 1 -- 172.21.15.119:0/2239202007 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b401b070 con 0x7f39bc0714e0 2024-09-13T16:26:24.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39b97fa700 1 -- 172.21.15.119:0/2239202007 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39b4004020 con 0x7f39bc0714e0 2024-09-13T16:26:24.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39bc070a00 con 0x7f39bc0714e0 2024-09-13T16:26:24.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39b97fa700 1 -- 172.21.15.119:0/2239202007 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39b40041d0 con 0x7f39bc0714e0 2024-09-13T16:26:24.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.380+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39bc15a3c0 con 0x7f39bc0714e0 2024-09-13T16:26:24.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.384+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39bc0711c0 con 0x7f39bc0714e0 2024-09-13T16:26:24.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.384+0000 7f39b97fa700 1 -- 172.21.15.119:0/2239202007 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f39b4004330 con 0x7f39bc0714e0 2024-09-13T16:26:24.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.384+0000 7f39b97fa700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39a8066760 0x7f39a8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.384+0000 7f39b97fa700 1 -- 172.21.15.119:0/2239202007 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f39b4084030 con 0x7f39bc0714e0 2024-09-13T16:26:24.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.384+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39a8066760 0x7f39a8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.384+0000 7f39c0c70700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39a8066760 0x7f39a8068c20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f39ac015040 tx=0x7f39ac00a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.392+0000 7f39b97fa700 1 -- 172.21.15.119:0/2239202007 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f39b4050fe0 con 0x7f39bc0714e0 2024-09-13T16:26:24.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39bc0adf10 con 0x7f39bc0714e0 2024-09-13T16:26:24.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39b97fa700 1 -- 172.21.15.119:0/2239202007 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f39b4054600 con 0x7f39bc0714e0 2024-09-13T16:26:24.573 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:24.573 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39a8066760 msgr2=0x7f39a8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39a8066760 0x7f39a8068c20 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f39ac015040 tx=0x7f39ac00a000 comp rx=0 tx=0).stop 2024-09-13T16:26:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc0714e0 msgr2=0x7f39bc0ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc0714e0 0x7f39bc0ac8c0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f39bc0cb1e0 tx=0x7f39b4011eb0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 shutdown_connections 2024-09-13T16:26:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f39a8066760 0x7f39a8068c20 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39bc0714e0 0x7f39bc0ac8c0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39bc06ffb0 0x7f39bc13c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 --2- 172.21.15.119:0/2239202007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a4003be0 0x7f39bc15d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.568+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 >> 172.21.15.119:0/2239202007 conn(0x7f39bc06a350 msgr2=0x7f39bc15ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.572+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 shutdown_connections 2024-09-13T16:26:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.572+0000 7f39c1c72700 1 -- 172.21.15.119:0/2239202007 wait complete. 2024-09-13T16:26:24.589 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:26:24.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:24.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526ae44700 1 Processor -- start 2024-09-13T16:26:24.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526ae44700 1 -- start start 2024-09-13T16:26:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526ae44700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f526408c6b0 0x7f52640ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526ae44700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f52640cd860 0x7f5264166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526ae44700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f526415d020 con 0x7f526415cc40 2024-09-13T16:26:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526ae44700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52640b44c0 con 0x7f526408c6b0 2024-09-13T16:26:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526ae44700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52640d0720 con 0x7f52640cd860 2024-09-13T16:26:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f526408c6b0 0x7f52640ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f526408c6b0 0x7f52640ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48836/0 (socket says 172.21.15.119:48836) 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 -- 172.21.15.119:0/2207231246 learned_addr learned my addr 172.21.15.119:0/2207231246 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526a643700 1 --2- 172.21.15.119:0/2207231246 >> v2:172.21.15.145:3300/0 conn(0x7f52640cd860 0x7f5264166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 -- 172.21.15.119:0/2207231246 >> v2:172.21.15.145:3300/0 conn(0x7f52640cd860 msgr2=0x7f5264166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 --2- 172.21.15.119:0/2207231246 >> v2:172.21.15.145:3300/0 conn(0x7f52640cd860 0x7f5264166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 -- 172.21.15.119:0/2207231246 >> v1:172.21.15.145:6789/0 conn(0x7f526415cc40 legacy=0x7f526408bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 -- 172.21.15.119:0/2207231246 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52640d09c0 con 0x7f526408c6b0 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f526a643700 1 --2- 172.21.15.119:0/2207231246 >> v2:172.21.15.145:3300/0 conn(0x7f52640cd860 0x7f5264166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.688+0000 7f5269641700 1 --2- 172.21.15.119:0/2207231246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640ce710 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5258009910 tx=0x7f5258009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7af5dbef5a9ee81a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f5268e40700 1 -- 172.21.15.119:0/2207231246 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258004230 con 0x7f526408c6b0 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f5268e40700 1 -- 172.21.15.119:0/2207231246 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5258004390 con 0x7f526408c6b0 2024-09-13T16:26:24.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f5268e40700 1 -- 172.21.15.119:0/2207231246 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5258009e60 con 0x7f526408c6b0 2024-09-13T16:26:24.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 -- 172.21.15.119:0/2207231246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 msgr2=0x7f52640ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 --2- 172.21.15.119:0/2207231246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640ce710 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f5258009910 tx=0x7f5258009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:24.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 -- 172.21.15.119:0/2207231246 shutdown_connections 2024-09-13T16:26:24.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 --2- 172.21.15.119:0/2207231246 >> v2:172.21.15.145:3300/0 conn(0x7f52640cd860 0x7f5264166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 --2- 172.21.15.119:0/2207231246 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640ce710 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 -- 172.21.15.119:0/2207231246 >> 172.21.15.119:0/2207231246 conn(0x7f52640b4640 msgr2=0x7f52640b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:24.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 -- 172.21.15.119:0/2207231246 shutdown_connections 2024-09-13T16:26:24.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 -- 172.21.15.119:0/2207231246 wait complete. 2024-09-13T16:26:24.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 Processor -- start 2024-09-13T16:26:24.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.692+0000 7f526ae44700 1 -- start start 2024-09-13T16:26:24.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52640cd860 0x7f5264159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f526415cc40 0x7f52640b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52641f2290 con 0x7f526408c6b0 2024-09-13T16:26:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52640b7e10 con 0x7f52640cd860 2024-09-13T16:26:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52640b5ec0 con 0x7f526415cc40 2024-09-13T16:26:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48842/0 (socket says 172.21.15.119:48842) 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 -- 172.21.15.119:0/460206043 learned_addr learned my addr 172.21.15.119:0/460206043 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526a643700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f526415cc40 0x7f52640b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269641700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52640cd860 0x7f5264159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 -- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f526415cc40 msgr2=0x7f52640b8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f526415cc40 0x7f52640b8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 -- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52640cd860 msgr2=0x7f5264159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52640cd860 0x7f5264159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 -- 172.21.15.119:0/460206043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5258009570 con 0x7f526408c6b0 2024-09-13T16:26:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526a643700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f526415cc40 0x7f52640b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:24.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f5269e42700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640b5980 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f526000b4d0 tx=0x7f526000b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f52567fc700 1 -- 172.21.15.119:0/460206043 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5260011830 con 0x7f526408c6b0 2024-09-13T16:26:24.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f52567fc700 1 -- 172.21.15.119:0/460206043 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5260011990 con 0x7f526408c6b0 2024-09-13T16:26:24.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f526415a6f0 con 0x7f526408c6b0 2024-09-13T16:26:24.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f52567fc700 1 -- 172.21.15.119:0/460206043 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5260011d40 con 0x7f526408c6b0 2024-09-13T16:26:24.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52640b7750 con 0x7f526408c6b0 2024-09-13T16:26:24.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52640b7f50 con 0x7f526408c6b0 2024-09-13T16:26:24.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.696+0000 7f52567fc700 1 -- 172.21.15.119:0/460206043 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f526001a400 con 0x7f526408c6b0 2024-09-13T16:26:24.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.700+0000 7f52567fc700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5248066730 0x7f5248068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:24.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.700+0000 7f52567fc700 1 -- 172.21.15.119:0/460206043 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f5260084160 con 0x7f526408c6b0 2024-09-13T16:26:24.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.700+0000 7f5269641700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5248066730 0x7f5248068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:24.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.700+0000 7f5269641700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5248066730 0x7f5248068bf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5258009910 tx=0x7f5258008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:24.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.704+0000 7f52567fc700 1 -- 172.21.15.119:0/460206043 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f52600511a0 con 0x7f526408c6b0 2024-09-13T16:26:24.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.912+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52640d1a60 con 0x7f526408c6b0 2024-09-13T16:26:24.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.912+0000 7f52567fc700 1 -- 172.21.15.119:0/460206043 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f52600547c0 con 0x7f526408c6b0 2024-09-13T16:26:24.917 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:24.918 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:26:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5248066730 msgr2=0x7f5248068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5248066730 0x7f5248068bf0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f5258009910 tx=0x7f5258008030 comp rx=0 tx=0).stop 2024-09-13T16:26:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 msgr2=0x7f52640b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640b5980 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f526000b4d0 tx=0x7f526000b7e0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 shutdown_connections 2024-09-13T16:26:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5248066730 0x7f5248068bf0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f526415cc40 0x7f52640b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52640cd860 0x7f5264159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 --2- 172.21.15.119:0/460206043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f526408c6b0 0x7f52640b5980 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:24.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 >> 172.21.15.119:0/460206043 conn(0x7f52640b4640 msgr2=0x7f526408bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:24.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 shutdown_connections 2024-09-13T16:26:24.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:24.916+0000 7f526ae44700 1 -- 172.21.15.119:0/460206043 wait complete. 2024-09-13T16:26:24.922 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:26:24.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:25.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.040+0000 7f1c2d896700 1 Processor -- start 2024-09-13T16:26:25.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.040+0000 7f1c2d896700 1 -- start start 2024-09-13T16:26:25.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1c280cc060 0x7f1c280cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2806cf60 con 0x7f1c280cde70 2024-09-13T16:26:25.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c280cc4d0 con 0x7f1c280adb30 2024-09-13T16:26:25.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2806c350 con 0x7f1c280cc060 2024-09-13T16:26:25.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d095700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1c280cde70 0x7f1c280b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51294/0 (socket says 172.21.15.119:51294) 2024-09-13T16:26:25.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d095700 1 -- 172.21.15.119:0/1242849983 learned_addr learned my addr 172.21.15.119:0/1242849983 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:25.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c27fff700 1 --2- 172.21.15.119:0/1242849983 >> v2:172.21.15.145:3300/0 conn(0x7f1c280cc060 0x7f1c280cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c277fe700 1 -- 172.21.15.119:0/1242849983 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3812934978 0 0) 0x7f1c2806cf60 con 0x7f1c280cde70 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c277fe700 1 -- 172.21.15.119:0/1242849983 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c180035d0 con 0x7f1c280cde70 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c27fff700 1 -- 172.21.15.119:0/1242849983 >> v1:172.21.15.145:6789/0 conn(0x7f1c280adb30 legacy=0x7f1c2806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c27fff700 1 -- 172.21.15.119:0/1242849983 >> v1:172.21.15.119:6789/0 conn(0x7f1c280cde70 legacy=0x7f1c280b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c27fff700 1 -- 172.21.15.119:0/1242849983 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2806c630 con 0x7f1c280cc060 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c27fff700 1 --2- 172.21.15.119:0/1242849983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cc060 0x7f1c280cd930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009850 tx=0x7f1c14009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=263f9cb11b9288b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c277fe700 1 -- 172.21.15.119:0/1242849983 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14011830 con 0x7f1c280cc060 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c277fe700 1 -- 172.21.15.119:0/1242849983 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c14011990 con 0x7f1c280cc060 2024-09-13T16:26:25.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c277fe700 1 -- 172.21.15.119:0/1242849983 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14011d00 con 0x7f1c280cc060 2024-09-13T16:26:25.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- 172.21.15.119:0/1242849983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cc060 msgr2=0x7f1c280cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 --2- 172.21.15.119:0/1242849983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cc060 0x7f1c280cd930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009850 tx=0x7f1c14009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:25.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- 172.21.15.119:0/1242849983 shutdown_connections 2024-09-13T16:26:25.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 --2- 172.21.15.119:0/1242849983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cc060 0x7f1c280cd930 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- 172.21.15.119:0/1242849983 >> 172.21.15.119:0/1242849983 conn(0x7f1c2806a350 msgr2=0x7f1c2806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:25.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- 172.21.15.119:0/1242849983 shutdown_connections 2024-09-13T16:26:25.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.044+0000 7f1c2d896700 1 -- 172.21.15.119:0/1242849983 wait complete. 2024-09-13T16:26:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 Processor -- start 2024-09-13T16:26:25.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 -- start start 2024-09-13T16:26:25.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c280adb30 0x7f1c2815d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c280cc060 0x7f1c2815a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cde70 0x7f1c280ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281f3160 con 0x7f1c280cc060 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281f2550 con 0x7f1c280cde70 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d896700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c281f2830 con 0x7f1c280adb30 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c280cc060 0x7f1c2815a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c280cc060 0x7f1c2815a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48844/0 (socket says 172.21.15.119:48844) 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 -- 172.21.15.119:0/2719427394 learned_addr learned my addr 172.21.15.119:0/2719427394 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d095700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cde70 0x7f1c280ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2c894700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c280adb30 0x7f1c2815d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 -- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c280adb30 msgr2=0x7f1c2815d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c280adb30 0x7f1c2815d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 -- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cde70 msgr2=0x7f1c280ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cde70 0x7f1c280ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c27fff700 1 -- 172.21.15.119:0/2719427394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c140094b0 con 0x7f1c280cc060 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.048+0000 7f1c2d095700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cde70 0x7f1c280ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c27fff700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c280cc060 0x7f1c2815a170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009ba0 tx=0x7f1c14004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c257fa700 1 -- 172.21.15.119:0/2719427394 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c140036c0 con 0x7f1c280cc060 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c257fa700 1 -- 172.21.15.119:0/2719427394 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1401f070 con 0x7f1c280cc060 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c257fa700 1 -- 172.21.15.119:0/2719427394 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c14003a60 con 0x7f1c280cc060 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c281f2b10 con 0x7f1c280cc060 2024-09-13T16:26:25.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c281f20d0 con 0x7f1c280cc060 2024-09-13T16:26:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c257fa700 1 -- 172.21.15.119:0/2719427394 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1c14003bc0 con 0x7f1c280cc060 2024-09-13T16:26:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c280ce300 con 0x7f1c280cc060 2024-09-13T16:26:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c257fa700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c0c066730 0x7f1c0c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c257fa700 1 -- 172.21.15.119:0/2719427394 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f1c14021070 con 0x7f1c280cc060 2024-09-13T16:26:25.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.052+0000 7f1c2c894700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c0c066730 0x7f1c0c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.056+0000 7f1c2c894700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c0c066730 0x7f1c0c068bf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1c2815af00 tx=0x7f1c1c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.060+0000 7f1c257fa700 1 -- 172.21.15.119:0/2719427394 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1c14052a50 con 0x7f1c280cc060 2024-09-13T16:26:25.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c280adf10 con 0x7f1c280cc060 2024-09-13T16:26:25.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c257fa700 1 -- 172.21.15.119:0/2719427394 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+4742 (secure 0 0 0) 0x7f1c1402a090 con 0x7f1c280cc060 2024-09-13T16:26:25.276 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:25.276 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4292578996","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4292578996},{"type":"v1","addr":"172.21.15.119:6833","nonce":4292578996}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4217,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/3331037136","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3331037136},{"type":"v1","addr":"172.21.15.145:6836","nonce":3331037136}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4282,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3997039143","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":3997039143},{"type":"v1","addr":"172.21.15.145:6837","nonce":3997039143}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:25:16.907443+0000","modified":"2024-09-13T16:25:33.005194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4285":{"gid":4285,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3556289056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3556289056},{"type":"v1","addr":"172.21.15.119:6835","nonce":3556289056}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-09-13T16:26:25.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c0c066730 msgr2=0x7f1c0c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c0c066730 0x7f1c0c068bf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f1c2815af00 tx=0x7f1c1c009040 comp rx=0 tx=0).stop 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c280cc060 msgr2=0x7f1c2815a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c280cc060 0x7f1c2815a170 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1c14009ba0 tx=0x7f1c14004770 comp rx=0 tx=0).stop 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 shutdown_connections 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c0c066730 0x7f1c0c068bf0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c280cde70 0x7f1c280ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c280cc060 0x7f1c2815a170 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 --2- 172.21.15.119:0/2719427394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c280adb30 0x7f1c2815d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 >> 172.21.15.119:0/2719427394 conn(0x7f1c2806a350 msgr2=0x7f1c280cb660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 shutdown_connections 2024-09-13T16:26:25.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.272+0000 7f1c2d896700 1 -- 172.21.15.119:0/2719427394 wait complete. 2024-09-13T16:26:25.280 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 9 2024-09-13T16:26:25.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:25.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.392+0000 7f3623fff700 1 Processor -- start 2024-09-13T16:26:25.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.392+0000 7f3623fff700 1 -- start start 2024-09-13T16:26:25.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.392+0000 7f3623fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3624156ed0 0x7f362406e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.392+0000 7f3623fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f36240b42d0 0x7f3624071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.392+0000 7f3623fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36240aec60 con 0x7f36241572b0 2024-09-13T16:26:25.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.392+0000 7f3623fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f362415e0a0 con 0x7f36240b42d0 2024-09-13T16:26:25.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.392+0000 7f3623fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36240b47b0 con 0x7f3624156ed0 2024-09-13T16:26:25.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f36240b42d0 0x7f3624071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3622ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3624156ed0 0x7f362406e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f36240b42d0 0x7f3624071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:37184/0 (socket says 172.21.15.119:37184) 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 -- 172.21.15.119:0/3399076143 learned_addr learned my addr 172.21.15.119:0/3399076143 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36237fe700 1 --1- 172.21.15.119:0/3399076143 >> v1:172.21.15.145:6789/0 conn(0x7f36241572b0 0x7f362406a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:37758/0 (socket says 172.21.15.119:37758) 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 -- 172.21.15.119:0/3399076143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3624156ed0 msgr2=0x7f362406e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 --2- 172.21.15.119:0/3399076143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3624156ed0 0x7f362406e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 -- 172.21.15.119:0/3399076143 >> v1:172.21.15.145:6789/0 conn(0x7f36241572b0 legacy=0x7f362406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 -- 172.21.15.119:0/3399076143 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36240b4a90 con 0x7f36240b42d0 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3621ffb700 1 -- 172.21.15.119:0/3399076143 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335230418 0 0) 0x7f36240aec60 con 0x7f36241572b0 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f36227fc700 1 --2- 172.21.15.119:0/3399076143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36240b42d0 0x7f3624071620 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3614009910 tx=0x7f3614009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26784defaea8c07d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3621ffb700 1 -- 172.21.15.119:0/3399076143 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614011830 con 0x7f36240b42d0 2024-09-13T16:26:25.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3621ffb700 1 -- 172.21.15.119:0/3399076143 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3614011990 con 0x7f36240b42d0 2024-09-13T16:26:25.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3621ffb700 1 -- 172.21.15.119:0/3399076143 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614011d40 con 0x7f36240b42d0 2024-09-13T16:26:25.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 -- 172.21.15.119:0/3399076143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36240b42d0 msgr2=0x7f3624071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 --2- 172.21.15.119:0/3399076143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36240b42d0 0x7f3624071620 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f3614009910 tx=0x7f3614009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:25.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 -- 172.21.15.119:0/3399076143 shutdown_connections 2024-09-13T16:26:25.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 --2- 172.21.15.119:0/3399076143 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36240b42d0 0x7f3624071620 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 --2- 172.21.15.119:0/3399076143 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3624156ed0 0x7f362406e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 -- 172.21.15.119:0/3399076143 >> 172.21.15.119:0/3399076143 conn(0x7f362406b460 msgr2=0x7f362406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:25.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 -- 172.21.15.119:0/3399076143 shutdown_connections 2024-09-13T16:26:25.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.396+0000 7f3623fff700 1 -- 172.21.15.119:0/3399076143 wait complete. 2024-09-13T16:26:25.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 Processor -- start 2024-09-13T16:26:25.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 -- start start 2024-09-13T16:26:25.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36240b42d0 0x7f3624155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3624156ed0 0x7f36240ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36241572b0 0x7f36240ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36241f2ec0 con 0x7f3624156ed0 2024-09-13T16:26:25.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3624155bd0 con 0x7f36241572b0 2024-09-13T16:26:25.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36241f22f0 con 0x7f36240b42d0 2024-09-13T16:26:25.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3624156ed0 0x7f36240ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3624156ed0 0x7f36240ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48848/0 (socket says 172.21.15.119:48848) 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 -- 172.21.15.119:0/2712525330 learned_addr learned my addr 172.21.15.119:0/2712525330 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36237fe700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36241572b0 0x7f36240ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3622ffd700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36240b42d0 0x7f3624155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 -- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36240b42d0 msgr2=0x7f3624155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36240b42d0 0x7f3624155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 -- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36241572b0 msgr2=0x7f36240ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36241572b0 0x7f36240ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 -- 172.21.15.119:0/2712525330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3614009570 con 0x7f3624156ed0 2024-09-13T16:26:25.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f36227fc700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3624156ed0 0x7f36240ac8c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f362406ed10 tx=0x7f3614004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f360bfff700 1 -- 172.21.15.119:0/2712525330 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3614004300 con 0x7f3624156ed0 2024-09-13T16:26:25.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f360bfff700 1 -- 172.21.15.119:0/2712525330 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f361401f070 con 0x7f3624156ed0 2024-09-13T16:26:25.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f360bfff700 1 -- 172.21.15.119:0/2712525330 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36140036c0 con 0x7f3624156ed0 2024-09-13T16:26:25.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36241f2590 con 0x7f3624156ed0 2024-09-13T16:26:25.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.400+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36241f2b50 con 0x7f3624156ed0 2024-09-13T16:26:25.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.404+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f362406b9e0 con 0x7f3624156ed0 2024-09-13T16:26:25.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.404+0000 7f360bfff700 1 -- 172.21.15.119:0/2712525330 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3614021030 con 0x7f3624156ed0 2024-09-13T16:26:25.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.404+0000 7f360bfff700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3610066730 0x7f3610068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.404+0000 7f360bfff700 1 -- 172.21.15.119:0/2712525330 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f36140859c0 con 0x7f3624156ed0 2024-09-13T16:26:25.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.404+0000 7f3622ffd700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3610066730 0x7f3610068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.404+0000 7f3622ffd700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3610066730 0x7f3610068bf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36240b16e0 tx=0x7f3618009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.412+0000 7f360bfff700 1 -- 172.21.15.119:0/2712525330 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3614052960 con 0x7f3624156ed0 2024-09-13T16:26:25.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.588+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36240cfef0 con 0x7f3624156ed0 2024-09-13T16:26:25.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f360bfff700 1 -- 172.21.15.119:0/2712525330 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v27) v1 ==== 74+0+18794 (secure 0 0 0) 0x7f3614055f80 con 0x7f3624156ed0 2024-09-13T16:26:25.595 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:25.596 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":27,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:06.558897+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:25:14.874497+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-09-13T16:25:15.904673+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-09-13T16:25:18.965554+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.d","mappings":[{"from":7,"to":2}]},{"pgid":"3.21","mappings":[{"from":5,"to":1}]},{"pgid":"3.38","mappings":[{"from":3,"to":2}]},{"pgid":"3.3c","mappings":[{"from":5,"to":2}]},{"pgid":"4.0","mappings":[{"from":1,"to":2}]},{"pgid":"4.6","mappings":[{"from":1,"to":5}]},{"pgid":"4.1e","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:25.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3610066730 msgr2=0x7f3610068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3610066730 0x7f3610068bf0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f36240b16e0 tx=0x7f3618009240 comp rx=0 tx=0).stop 2024-09-13T16:26:25.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3624156ed0 msgr2=0x7f36240ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3624156ed0 0x7f36240ac8c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f362406ed10 tx=0x7f3614004770 comp rx=0 tx=0).stop 2024-09-13T16:26:25.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 shutdown_connections 2024-09-13T16:26:25.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3610066730 0x7f3610068bf0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36241572b0 0x7f36240ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3624156ed0 0x7f36240ac8c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 --2- 172.21.15.119:0/2712525330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36240b42d0 0x7f3624155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 >> 172.21.15.119:0/2712525330 conn(0x7f362406b460 msgr2=0x7f362406cce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:25.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 shutdown_connections 2024-09-13T16:26:25.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.592+0000 7f3623fff700 1 -- 172.21.15.119:0/2712525330 wait complete. 2024-09-13T16:26:25.611 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:26:25.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.720+0000 7f3acc972700 1 Processor -- start 2024-09-13T16:26:25.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.720+0000 7f3acc972700 1 -- start start 2024-09-13T16:26:25.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3ac80ce040 0x7f3ac8165450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac806cf60 con 0x7f3ac80cc060 2024-09-13T16:26:25.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ac80cc4d0 con 0x7f3ac80adb30 2024-09-13T16:26:25.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ac80b3390 con 0x7f3ac80ce040 2024-09-13T16:26:25.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3ac80ce040 0x7f3ac8165450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac659c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3ac80cc060 0x7f3ac80cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51296/0 (socket says 172.21.15.119:51296) 2024-09-13T16:26:25.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac659c700 1 -- 172.21.15.119:0/2352324485 learned_addr learned my addr 172.21.15.119:0/2352324485 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:25.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac759e700 1 -- 172.21.15.119:0/2352324485 >> v1:172.21.15.145:6789/0 conn(0x7f3ac80adb30 legacy=0x7f3ac806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac759e700 1 -- 172.21.15.119:0/2352324485 >> v1:172.21.15.119:6789/0 conn(0x7f3ac80cc060 legacy=0x7f3ac80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac5d9b700 1 -- 172.21.15.119:0/2352324485 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292815016 0 0) 0x7f3ac806cf60 con 0x7f3ac80cc060 2024-09-13T16:26:25.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac759e700 1 -- 172.21.15.119:0/2352324485 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac806c350 con 0x7f3ac80ce040 2024-09-13T16:26:25.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac759e700 1 --2- 172.21.15.119:0/2352324485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80ce040 0x7f3ac8165450 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009850 tx=0x7f3ac0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693dbf4104b93ffc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac5d9b700 1 -- 172.21.15.119:0/2352324485 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0011830 con 0x7f3ac80ce040 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac5d9b700 1 -- 172.21.15.119:0/2352324485 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac0011990 con 0x7f3ac80ce040 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3ac5d9b700 1 -- 172.21.15.119:0/2352324485 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac0011d20 con 0x7f3ac80ce040 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- 172.21.15.119:0/2352324485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80ce040 msgr2=0x7f3ac8165450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 --2- 172.21.15.119:0/2352324485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80ce040 0x7f3ac8165450 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3ac0009850 tx=0x7f3ac0009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- 172.21.15.119:0/2352324485 shutdown_connections 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 --2- 172.21.15.119:0/2352324485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80ce040 0x7f3ac8165450 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- 172.21.15.119:0/2352324485 >> 172.21.15.119:0/2352324485 conn(0x7f3ac806a350 msgr2=0x7f3ac806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- 172.21.15.119:0/2352324485 shutdown_connections 2024-09-13T16:26:25.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- 172.21.15.119:0/2352324485 wait complete. 2024-09-13T16:26:25.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 Processor -- start 2024-09-13T16:26:25.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.724+0000 7f3acc972700 1 -- start start 2024-09-13T16:26:25.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ac80adb30 0x7f3ac813e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80cc060 0x7f3ac80a9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ac80ce040 0x7f3ac80a98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac80a9e00 con 0x7f3ac80cc060 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac815b7b0 con 0x7f3ac80ce040 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac81f11e0 con 0x7f3ac80adb30 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80cc060 0x7f3ac80a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80cc060 0x7f3ac80a9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48866/0 (socket says 172.21.15.119:48866) 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 -- 172.21.15.119:0/2376090480 learned_addr learned my addr 172.21.15.119:0/2376090480 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac759e700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ac80ce040 0x7f3ac80a98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac6d9d700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ac80adb30 0x7f3ac813e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 -- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ac80adb30 msgr2=0x7f3ac813e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ac80adb30 0x7f3ac813e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 -- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ac80ce040 msgr2=0x7f3ac80a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ac80ce040 0x7f3ac80a98c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 -- 172.21.15.119:0/2376090480 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac00094b0 con 0x7f3ac80cc060 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3ac659c700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80cc060 0x7f3ac80a9380 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800cc80 tx=0x7f3ab800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3aaf7fe700 1 -- 172.21.15.119:0/2376090480 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab800ade0 con 0x7f3ac80cc060 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3aaf7fe700 1 -- 172.21.15.119:0/2376090480 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ab800ce80 con 0x7f3ac80cc060 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac815ae80 con 0x7f3ac80cc060 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3aaf7fe700 1 -- 172.21.15.119:0/2376090480 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ab8007310 con 0x7f3ac80cc060 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac815b440 con 0x7f3ac80cc060 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.728+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac80adf10 con 0x7f3ac80cc060 2024-09-13T16:26:25.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.732+0000 7f3aaf7fe700 1 -- 172.21.15.119:0/2376090480 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3ab8007470 con 0x7f3ac80cc060 2024-09-13T16:26:25.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.732+0000 7f3aaf7fe700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ab4066730 0x7f3ab4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:25.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.732+0000 7f3aaf7fe700 1 -- 172.21.15.119:0/2376090480 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f3ab80843f0 con 0x7f3ac80cc060 2024-09-13T16:26:25.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.732+0000 7f3ac6d9d700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ab4066730 0x7f3ab4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:25.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.732+0000 7f3ac6d9d700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ab4066730 0x7f3ab4068bf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3ac815aa90 tx=0x7f3abc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:25.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.736+0000 7f3aaf7fe700 1 -- 172.21.15.119:0/2376090480 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3ab8051430 con 0x7f3ac80cc060 2024-09-13T16:26:25.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.940+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3ac815b8f0 con 0x7f3ac80cc060 2024-09-13T16:26:25.946 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:26:25.940+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:26:25.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.952+0000 7f3aaf7fe700 1 -- 172.21.15.119:0/2376090480 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v10) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3ab8054a50 con 0x7f3ac80cc060 2024-09-13T16:26:25.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ab4066730 msgr2=0x7f3ab4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ab4066730 0x7f3ab4068bf0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3ac815aa90 tx=0x7f3abc009040 comp rx=0 tx=0).stop 2024-09-13T16:26:25.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80cc060 msgr2=0x7f3ac80a9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:25.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80cc060 0x7f3ac80a9380 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f3ab800cc80 tx=0x7f3ab800a340 comp rx=0 tx=0).stop 2024-09-13T16:26:25.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 shutdown_connections 2024-09-13T16:26:25.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ab4066730 0x7f3ab4068bf0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3ac80ce040 0x7f3ac80a98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3ac80cc060 0x7f3ac80a9380 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 --2- 172.21.15.119:0/2376090480 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3ac80adb30 0x7f3ac813e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:25.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 >> 172.21.15.119:0/2376090480 conn(0x7f3ac806a350 msgr2=0x7f3ac80b0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:25.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 shutdown_connections 2024-09-13T16:26:25.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:25.956+0000 7f3acc972700 1 -- 172.21.15.119:0/2376090480 wait complete. 2024-09-13T16:26:25.962 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:26:25.966 INFO:tasks.ceph.mds.c.smithi119.stderr: -4332> 2024-09-13T16:25:22.634+0000 7f1b3432e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:26:25.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:26:26.005 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:26:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.080+0000 7f9009b3c700 1 Processor -- start 2024-09-13T16:26:26.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.080+0000 7f9009b3c700 1 -- start start 2024-09-13T16:26:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90040b1ad0 0x7f9004071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9004168a10 0x7f900416ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9004162270 con 0x7f900406db40 2024-09-13T16:26:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f900406ce40 con 0x7f9004168a10 2024-09-13T16:26:26.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90040702f0 con 0x7f90040b1ad0 2024-09-13T16:26:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9003fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9004168a10 0x7f900416ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9002ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f900406db40 0x7f90040b1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51314/0 (socket says 172.21.15.119:51314) 2024-09-13T16:26:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9002ffd700 1 -- 172.21.15.119:0/190697101 learned_addr learned my addr 172.21.15.119:0/190697101 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:26.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90037fe700 1 --2- 172.21.15.119:0/190697101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90040b1ad0 0x7f9004071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90037fe700 1 -- 172.21.15.119:0/190697101 >> v2:172.21.15.119:3300/0 conn(0x7f9004168a10 msgr2=0x7f900416ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90037fe700 1 --2- 172.21.15.119:0/190697101 >> v2:172.21.15.119:3300/0 conn(0x7f9004168a10 0x7f900416ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90037fe700 1 -- 172.21.15.119:0/190697101 >> v1:172.21.15.119:6789/0 conn(0x7f900406db40 legacy=0x7f90040b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:26.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90037fe700 1 -- 172.21.15.119:0/190697101 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9004071730 con 0x7f90040b1ad0 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90037fe700 1 --2- 172.21.15.119:0/190697101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90040b1ad0 0x7f9004071c10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009d90 tx=0x7f8ff400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8a544dfd20991d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90027fc700 1 -- 172.21.15.119:0/190697101 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff400e070 con 0x7f90040b1ad0 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90027fc700 1 -- 172.21.15.119:0/190697101 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff400bb20 con 0x7f90040b1ad0 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f90027fc700 1 -- 172.21.15.119:0/190697101 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4012430 con 0x7f90040b1ad0 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- 172.21.15.119:0/190697101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90040b1ad0 msgr2=0x7f9004071c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 --2- 172.21.15.119:0/190697101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90040b1ad0 0x7f9004071c10 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4009d90 tx=0x7f8ff400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- 172.21.15.119:0/190697101 shutdown_connections 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 --2- 172.21.15.119:0/190697101 >> v2:172.21.15.119:3300/0 conn(0x7f9004168a10 0x7f900416ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 --2- 172.21.15.119:0/190697101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f90040b1ad0 0x7f9004071c10 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- 172.21.15.119:0/190697101 >> 172.21.15.119:0/190697101 conn(0x7f90040b0560 msgr2=0x7f90040b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- 172.21.15.119:0/190697101 shutdown_connections 2024-09-13T16:26:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.084+0000 7f9009b3c700 1 -- 172.21.15.119:0/190697101 wait complete. 2024-09-13T16:26:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 Processor -- start 2024-09-13T16:26:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 -- start start 2024-09-13T16:26:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f900406db40 0x7f900414e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90040b1ad0 0x7f9004161030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9004168a10 0x7f9004161570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9004152640 con 0x7f900406db40 2024-09-13T16:26:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90041515a0 con 0x7f90040b1ad0 2024-09-13T16:26:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9004151880 con 0x7f9004168a10 2024-09-13T16:26:26.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f900406db40 0x7f900414e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f900406db40 0x7f900414e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48896/0 (socket says 172.21.15.119:48896) 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 -- 172.21.15.119:0/1904196462 learned_addr learned my addr 172.21.15.119:0/1904196462 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9002ffd700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90040b1ad0 0x7f9004161030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9003fff700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9004168a10 0x7f9004161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 -- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9004168a10 msgr2=0x7f9004161570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9004168a10 0x7f9004161570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 -- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90040b1ad0 msgr2=0x7f9004161030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90040b1ad0 0x7f9004161030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 -- 172.21.15.119:0/1904196462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4009a70 con 0x7f900406db40 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f90037fe700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f900406db40 0x7f900414e560 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4014040 tx=0x7f8ff4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f8ffbfff700 1 -- 172.21.15.119:0/1904196462 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4012430 con 0x7f900406db40 2024-09-13T16:26:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9004151b60 con 0x7f900406db40 2024-09-13T16:26:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f8ffbfff700 1 -- 172.21.15.119:0/1904196462 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ff4020070 con 0x7f900406db40 2024-09-13T16:26:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f8ffbfff700 1 -- 172.21.15.119:0/1904196462 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ff4004280 con 0x7f900406db40 2024-09-13T16:26:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.088+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9004152120 con 0x7f900406db40 2024-09-13T16:26:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.092+0000 7f8ffbfff700 1 -- 172.21.15.119:0/1904196462 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8ff40043e0 con 0x7f900406db40 2024-09-13T16:26:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.092+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9004073590 con 0x7f900406db40 2024-09-13T16:26:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.092+0000 7f8ffbfff700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fe4066730 0x7f8fe4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.092+0000 7f8ffbfff700 1 -- 172.21.15.119:0/1904196462 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f8ff4022070 con 0x7f900406db40 2024-09-13T16:26:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.092+0000 7f9002ffd700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fe4066730 0x7f8fe4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:26.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.092+0000 7f9002ffd700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fe4066730 0x7f8fe4068bf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f900414f4d0 tx=0x7f8fec007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:26.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.100+0000 7f8ffbfff700 1 -- 172.21.15.119:0/1904196462 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8ff40534d0 con 0x7f900406db40 2024-09-13T16:26:26.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.300+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f90040b2650 con 0x7f900406db40 2024-09-13T16:26:26.878 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:26:26.880 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:26:26.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.956+0000 7f8ffbfff700 1 -- 172.21.15.119:0/1904196462 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8ff4056af0 con 0x7f900406db40 2024-09-13T16:26:26.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.960+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fe4066730 msgr2=0x7f8fe4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.960+0000 7f9009b3c700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fe4066730 0x7f8fe4068bf0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f900414f4d0 tx=0x7f8fec007f30 comp rx=0 tx=0).stop 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f900406db40 msgr2=0x7f900414e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f900406db40 0x7f900414e560 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f8ff4014040 tx=0x7f8ff4004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 shutdown_connections 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fe4066730 0x7f8fe4068bf0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9004168a10 0x7f9004161570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f90040b1ad0 0x7f9004161030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 --2- 172.21.15.119:0/1904196462 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f900406db40 0x7f900414e560 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 >> 172.21.15.119:0/1904196462 conn(0x7f90040b0560 msgr2=0x7f9004159370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 shutdown_connections 2024-09-13T16:26:26.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:26.964+0000 7f9009b3c700 1 -- 172.21.15.119:0/1904196462 wait complete. 2024-09-13T16:26:26.982 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:26:27.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef960e700 1 Processor -- start 2024-09-13T16:26:27.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef960e700 1 -- start start 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef960e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef415ca30 0x7f9ef406d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef960e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9ef406d8d0 0x7f9ef40ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef960e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef40ad380 con 0x7f9ef40ace00 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef960e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef40ad940 con 0x7f9ef406d8d0 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef960e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef40ad660 con 0x7f9ef415ca30 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef8e0d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9ef40ace00 0x7f9ef40b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51316/0 (socket says 172.21.15.119:51316) 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef8e0d700 1 -- 172.21.15.119:0/3670449587 learned_addr learned my addr 172.21.15.119:0/3670449587 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef37fe700 1 --2- 172.21.15.119:0/3670449587 >> v2:172.21.15.145:3300/0 conn(0x7f9ef406d8d0 0x7f9ef40ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:27.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef3fff700 1 --2- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef415ca30 0x7f9ef406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:27.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef2ffd700 1 -- 172.21.15.119:0/3670449587 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998587134 0 0) 0x7f9ef40ad380 con 0x7f9ef40ace00 2024-09-13T16:26:27.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef2ffd700 1 -- 172.21.15.119:0/3670449587 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee00035d0 con 0x7f9ef40ace00 2024-09-13T16:26:27.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef37fe700 1 -- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef415ca30 msgr2=0x7f9ef406d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:27.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef37fe700 1 --2- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef415ca30 0x7f9ef406d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef37fe700 1 -- 172.21.15.119:0/3670449587 >> v1:172.21.15.119:6789/0 conn(0x7f9ef40ace00 legacy=0x7f9ef40b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:27.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef37fe700 1 -- 172.21.15.119:0/3670449587 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4070140 con 0x7f9ef406d8d0 2024-09-13T16:26:27.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.096+0000 7f9ef37fe700 1 --2- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef406d8d0 0x7f9ef40ac8c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009d90 tx=0x7f9eec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3a0d1453796daa1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:27.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef2ffd700 1 -- 172.21.15.119:0/3670449587 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec00e070 con 0x7f9ef406d8d0 2024-09-13T16:26:27.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef2ffd700 1 -- 172.21.15.119:0/3670449587 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9eec00bb20 con 0x7f9ef406d8d0 2024-09-13T16:26:27.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef2ffd700 1 -- 172.21.15.119:0/3670449587 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9eec00be90 con 0x7f9ef406d8d0 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 -- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef406d8d0 msgr2=0x7f9ef40ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 --2- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef406d8d0 0x7f9ef40ac8c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009d90 tx=0x7f9eec00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 -- 172.21.15.119:0/3670449587 shutdown_connections 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 --2- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef406d8d0 0x7f9ef40ac8c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 --2- 172.21.15.119:0/3670449587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef415ca30 0x7f9ef406d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 -- 172.21.15.119:0/3670449587 >> 172.21.15.119:0/3670449587 conn(0x7f9ef406a350 msgr2=0x7f9ef406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 -- 172.21.15.119:0/3670449587 shutdown_connections 2024-09-13T16:26:27.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 -- 172.21.15.119:0/3670449587 wait complete. 2024-09-13T16:26:27.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 Processor -- start 2024-09-13T16:26:27.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.100+0000 7f9ef960e700 1 -- start start 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef406d8d0 0x7f9ef4155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef40ace00 0x7f9ef40c3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef415ca30 0x7f9ef40cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef4143170 con 0x7f9ef415ca30 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef4143450 con 0x7f9ef40ace00 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef4143730 con 0x7f9ef406d8d0 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef8e0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef415ca30 0x7f9ef40cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef406d8d0 0x7f9ef4155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef8e0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef415ca30 0x7f9ef40cde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48908/0 (socket says 172.21.15.119:48908) 2024-09-13T16:26:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef37fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef40ace00 0x7f9ef40c3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef406d8d0 0x7f9ef4155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:58716/0 (socket says 172.21.15.119:58716) 2024-09-13T16:26:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef8e0d700 1 -- 172.21.15.119:0/493295082 learned_addr learned my addr 172.21.15.119:0/493295082 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 -- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef40ace00 msgr2=0x7f9ef40c3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef40ace00 0x7f9ef40c3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 -- 172.21.15.119:0/493295082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef415ca30 msgr2=0x7f9ef40cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:27.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef415ca30 0x7f9ef40cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 -- 172.21.15.119:0/493295082 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eec009a70 con 0x7f9ef406d8d0 2024-09-13T16:26:27.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef8e0d700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef415ca30 0x7f9ef40cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef3fff700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef406d8d0 0x7f9ef4155620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00e900 tx=0x7f9edc00ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef0ff9700 1 -- 172.21.15.119:0/493295082 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc013680 con 0x7f9ef406d8d0 2024-09-13T16:26:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef0ff9700 1 -- 172.21.15.119:0/493295082 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9edc0137e0 con 0x7f9ef406d8d0 2024-09-13T16:26:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef4143a10 con 0x7f9ef406d8d0 2024-09-13T16:26:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef0ff9700 1 -- 172.21.15.119:0/493295082 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9edc013b50 con 0x7f9ef406d8d0 2024-09-13T16:26:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.104+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef40ce3a0 con 0x7f9ef406d8d0 2024-09-13T16:26:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.108+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef40ac8c0 con 0x7f9ef406d8d0 2024-09-13T16:26:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.108+0000 7f9ef0ff9700 1 -- 172.21.15.119:0/493295082 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9edc013cb0 con 0x7f9ef406d8d0 2024-09-13T16:26:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.108+0000 7f9ef0ff9700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ed4066760 0x7f9ed4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.108+0000 7f9ef0ff9700 1 -- 172.21.15.119:0/493295082 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6749+0+0 (secure 0 0 0) 0x7f9edc0848b0 con 0x7f9ef406d8d0 2024-09-13T16:26:27.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.108+0000 7f9ef37fe700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ed4066760 0x7f9ed4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:27.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.108+0000 7f9ef37fe700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ed4066760 0x7f9ed4068c20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009d90 tx=0x7f9eec011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:27.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.120+0000 7f9ef0ff9700 1 -- 172.21.15.119:0/493295082 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9edc0516f0 con 0x7f9ef406d8d0 2024-09-13T16:26:27.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.288+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9ef415cfa0 con 0x7f9ef406d8d0 2024-09-13T16:26:27.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.972+0000 7f9ef0ff9700 1 -- 172.21.15.119:0/493295082 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v29) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9edc054d10 con 0x7f9ef406d8d0 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ed4066760 msgr2=0x7f9ed4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ed4066760 0x7f9ed4068c20 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9eec009d90 tx=0x7f9eec011000 comp rx=0 tx=0).stop 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef406d8d0 msgr2=0x7f9ef4155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef406d8d0 0x7f9ef4155620 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9edc00e900 tx=0x7f9edc00ec10 comp rx=0 tx=0).stop 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 shutdown_connections 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ed4066760 0x7f9ed4068c20 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ef415ca30 0x7f9ef40cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ef40ace00 0x7f9ef40c3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 --2- 172.21.15.119:0/493295082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ef406d8d0 0x7f9ef4155620 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 >> 172.21.15.119:0/493295082 conn(0x7f9ef406a350 msgr2=0x7f9ef40afe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 shutdown_connections 2024-09-13T16:26:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:27.980+0000 7f9ef960e700 1 -- 172.21.15.119:0/493295082 wait complete. 2024-09-13T16:26:27.986 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:26:28.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:26:28.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.112+0000 7f7c76c35700 1 Processor -- start 2024-09-13T16:26:28.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.112+0000 7f7c76c35700 1 -- start start 2024-09-13T16:26:28.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c7015c900 0x7f7c7015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:28.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7c700d35a0 0x7f7c700b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:28.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700d0650 con 0x7f7c700b26e0 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c700d39e0 con 0x7f7c700d35a0 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c700ce5a0 con 0x7f7c7015c900 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7c700d35a0 0x7f7c700b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7c700d35a0 0x7f7c700b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48920/0 (socket says 172.21.15.119:48920) 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 -- 172.21.15.119:0/1564340609 learned_addr learned my addr 172.21.15.119:0/1564340609 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75c33700 1 --2- 172.21.15.119:0/1564340609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c7015c900 0x7f7c7015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 -- 172.21.15.119:0/1564340609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c7015c900 msgr2=0x7f7c7015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 --2- 172.21.15.119:0/1564340609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c7015c900 0x7f7c7015cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 -- 172.21.15.119:0/1564340609 >> v1:172.21.15.145:6789/0 conn(0x7f7c700b26e0 legacy=0x7f7c700d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 -- 172.21.15.119:0/1564340609 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700ce880 con 0x7f7c700d35a0 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75c33700 1 --2- 172.21.15.119:0/1564340609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c7015c900 0x7f7c7015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:28.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c75432700 1 --2- 172.21.15.119:0/1564340609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c700d35a0 0x7f7c700b21a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009910 tx=0x7f7c64009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a52ad3b728ba8e82 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:28.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c74c31700 1 -- 172.21.15.119:0/1564340609 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6400ba60 con 0x7f7c700d35a0 2024-09-13T16:26:28.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c74c31700 1 -- 172.21.15.119:0/1564340609 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6400bbc0 con 0x7f7c700d35a0 2024-09-13T16:26:28.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c74c31700 1 -- 172.21.15.119:0/1564340609 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c64009e60 con 0x7f7c700d35a0 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- 172.21.15.119:0/1564340609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c700d35a0 msgr2=0x7f7c700b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 --2- 172.21.15.119:0/1564340609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c700d35a0 0x7f7c700b21a0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7c64009910 tx=0x7f7c64009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- 172.21.15.119:0/1564340609 shutdown_connections 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 --2- 172.21.15.119:0/1564340609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c700d35a0 0x7f7c700b21a0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 --2- 172.21.15.119:0/1564340609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c7015c900 0x7f7c7015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- 172.21.15.119:0/1564340609 >> 172.21.15.119:0/1564340609 conn(0x7f7c700d2a80 msgr2=0x7f7c700d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- 172.21.15.119:0/1564340609 shutdown_connections 2024-09-13T16:26:28.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.116+0000 7f7c76c35700 1 -- 172.21.15.119:0/1564340609 wait complete. 2024-09-13T16:26:28.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.120+0000 7f7c76c35700 1 Processor -- start 2024-09-13T16:26:28.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.120+0000 7f7c76c35700 1 -- start start 2024-09-13T16:26:28.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c700b26e0 0x7f7c7015bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:28.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c700d35a0 0x7f7c700c9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:28.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c7015c900 0x7f7c700ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:28.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c701f62b0 con 0x7f7c7015c900 2024-09-13T16:26:28.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c701f6590 con 0x7f7c700b26e0 2024-09-13T16:26:28.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c701f6870 con 0x7f7c700d35a0 2024-09-13T16:26:28.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c7015c900 0x7f7c700ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:28.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c75c33700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c700b26e0 0x7f7c7015bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:28.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c7015c900 0x7f7c700ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48928/0 (socket says 172.21.15.119:48928) 2024-09-13T16:26:28.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 -- 172.21.15.119:0/65065913 learned_addr learned my addr 172.21.15.119:0/65065913 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:28.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c75432700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c700d35a0 0x7f7c700c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:28.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 -- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c700d35a0 msgr2=0x7f7c700c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:28.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c700d35a0 0x7f7c700c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 -- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c700b26e0 msgr2=0x7f7c7015bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:28.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c700b26e0 0x7f7c7015bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 -- 172.21.15.119:0/65065913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c64009570 con 0x7f7c7015c900 2024-09-13T16:26:28.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76434700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c7015c900 0x7f7c700ca4a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800caf0 tx=0x7f7c6800ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:28.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c627fc700 1 -- 172.21.15.119:0/65065913 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c68003c20 con 0x7f7c7015c900 2024-09-13T16:26:28.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c627fc700 1 -- 172.21.15.119:0/65065913 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c68003d80 con 0x7f7c7015c900 2024-09-13T16:26:28.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c627fc700 1 -- 172.21.15.119:0/65065913 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800aa20 con 0x7f7c7015c900 2024-09-13T16:26:28.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c701f6b50 con 0x7f7c7015c900 2024-09-13T16:26:28.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7015c4d0 con 0x7f7c7015c900 2024-09-13T16:26:28.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.124+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c70074560 con 0x7f7c7015c900 2024-09-13T16:26:28.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.128+0000 7f7c627fc700 1 -- 172.21.15.119:0/65065913 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7c6800ab80 con 0x7f7c7015c900 2024-09-13T16:26:28.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.128+0000 7f7c627fc700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c54066730 0x7f7c54068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:28.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.128+0000 7f7c627fc700 1 -- 172.21.15.119:0/65065913 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 6311+0+0 (secure 0 0 0) 0x7f7c68084240 con 0x7f7c7015c900 2024-09-13T16:26:28.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.128+0000 7f7c75c33700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c54066730 0x7f7c54068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:28.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.128+0000 7f7c75c33700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c54066730 0x7f7c54068bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c006fd0 tx=0x7f7c6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:28.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.136+0000 7f7c627fc700 1 -- 172.21.15.119:0/65065913 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7c68051350 con 0x7f7c7015c900 2024-09-13T16:26:28.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.296+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7c701f63f0 con 0x7f7c7015c900 2024-09-13T16:26:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.984+0000 7f7c627fc700 1 -- 172.21.15.119:0/65065913 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v30) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7c701f63f0 con 0x7f7c7015c900 2024-09-13T16:26:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c54066730 msgr2=0x7f7c54068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c54066730 0x7f7c54068bf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c006fd0 tx=0x7f7c6c008040 comp rx=0 tx=0).stop 2024-09-13T16:26:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c7015c900 msgr2=0x7f7c700ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c7015c900 0x7f7c700ca4a0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800caf0 tx=0x7f7c6800ce00 comp rx=0 tx=0).stop 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 shutdown_connections 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c54066730 0x7f7c54068bf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c7015c900 0x7f7c700ca4a0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c700d35a0 0x7f7c700c9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 --2- 172.21.15.119:0/65065913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c700b26e0 0x7f7c7015bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 >> 172.21.15.119:0/65065913 conn(0x7f7c700d2a80 msgr2=0x7f7c7006b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 shutdown_connections 2024-09-13T16:26:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:28.992+0000 7f7c76c35700 1 -- 172.21.15.119:0/65065913 wait complete. 2024-09-13T16:26:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:26:29.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-09-13T16:26:29.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.120+0000 7f8b69d2c700 1 Processor -- start 2024-09-13T16:26:29.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.120+0000 7f8b69d2c700 1 -- start start 2024-09-13T16:26:29.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b69d2c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 0x7f8b6415cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b69d2c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640b3aa0 con 0x7f8b6415b9d0 2024-09-13T16:26:29.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b69d2c700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6415e280 con 0x7f8b640b4620 2024-09-13T16:26:29.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b69d2c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6415d6d0 con 0x7f8b640b4a00 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b6952b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8b6415b9d0 0x7f8b6409b240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51330/0 (socket says 172.21.15.119:51330) 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b6952b700 1 -- 172.21.15.119:0/151911100 learned_addr learned my addr 172.21.15.119:0/151911100 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b63fff700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 0x7f8b6415cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133920905 0 0) 0x7f8b640b3aa0 con 0x7f8b6415b9d0 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b4c0035d0 con 0x7f8b6415b9d0 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 864014584 0 0) 0x7f8b6415e280 con 0x7f8b640b4620 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b640b3aa0 con 0x7f8b640b4620 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1172624144 0 0) 0x7f8b4c0035d0 con 0x7f8b6415b9d0 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 msgr2=0x7f8b6415cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 0x7f8b6415cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 >> v1:172.21.15.145:6789/0 conn(0x7f8b640b4620 legacy=0x7f8b6409ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:29.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6415d970 con 0x7f8b6415b9d0 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f8b5c001970 con 0x7f8b6415b9d0 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 >> v1:172.21.15.119:6789/0 conn(0x7f8b6415b9d0 legacy=0x7f8b6409b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b4c003be0 0x7f8b4c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c004870 0x7f8b4c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 0x7f8b4c005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6415d6d0 con 0x7f8b4c003be0 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c00bbc0 con 0x7f8b4c004870 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4c00bd80 con 0x7f8b640b4a00 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b4c003be0 0x7f8b4c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b63fff700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c004870 0x7f8b4c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b6952b700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 0x7f8b4c005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b68d2a700 1 -- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 msgr2=0x7f8b4c005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640b4a00 0x7f8b4c005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b68d2a700 1 -- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c004870 msgr2=0x7f8b4c004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b4c004870 0x7f8b4c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.124+0000 7f8b68d2a700 1 -- 172.21.15.119:0/151911100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c0035d0 con 0x7f8b4c003be0 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b4c003be0 0x7f8b4c003fe0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8b54001960 tx=0x7f8b4c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84836d858f3c05d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54001450 con 0x7f8b4c003be0 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b540052f0 con 0x7f8b4c003be0 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b637fe700 1 -- 172.21.15.119:0/151911100 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540036a0 con 0x7f8b4c003be0 2024-09-13T16:26:29.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- 172.21.15.119:0/151911100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b4c003be0 msgr2=0x7f8b4c003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:29.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 --2- 172.21.15.119:0/151911100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b4c003be0 0x7f8b4c003fe0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f8b54001960 tx=0x7f8b4c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:29.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b6952b700 1 -- 172.21.15.119:0/151911100 reap_dead start 2024-09-13T16:26:29.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- 172.21.15.119:0/151911100 shutdown_connections 2024-09-13T16:26:29.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- 172.21.15.119:0/151911100 >> 172.21.15.119:0/151911100 conn(0x7f8b640722c0 msgr2=0x7f8b64099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:29.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- 172.21.15.119:0/151911100 shutdown_connections 2024-09-13T16:26:29.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- 172.21.15.119:0/151911100 wait complete. 2024-09-13T16:26:29.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 Processor -- start 2024-09-13T16:26:29.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- start start 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b64145200 0x7f8b641513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b6414f2a0 0x7f8b6406ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b6406f4a0 0x7f8b641f09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64150f80 con 0x7f8b6406f4a0 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6406f920 con 0x7f8b64145200 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b69d2c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64150300 con 0x7f8b6414f2a0 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b6952b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b6406f4a0 0x7f8b641f09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b6952b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b6406f4a0 0x7f8b641f09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48936/0 (socket says 172.21.15.119:48936) 2024-09-13T16:26:29.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.128+0000 7f8b6952b700 1 -- 172.21.15.119:0/2371785505 learned_addr learned my addr 172.21.15.119:0/2371785505 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b63fff700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b6414f2a0 0x7f8b6406ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b64145200 0x7f8b641513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b6952b700 1 -- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b6414f2a0 msgr2=0x7f8b6406ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b6952b700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b6414f2a0 0x7f8b6406ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b6952b700 1 -- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b64145200 msgr2=0x7f8b641513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b6952b700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b64145200 0x7f8b641513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b6952b700 1 -- 172.21.15.119:0/2371785505 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b641505e0 con 0x7f8b6406f4a0 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b64145200 0x7f8b641513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b6952b700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b6406f4a0 0x7f8b641f09e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8b6415bdb0 tx=0x7f8b5c00f280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b617fa700 1 -- 172.21.15.119:0/2371785505 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c00f910 con 0x7f8b6406f4a0 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b617fa700 1 -- 172.21.15.119:0/2371785505 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c00fad0 con 0x7f8b6406f4a0 2024-09-13T16:26:29.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b641508c0 con 0x7f8b6406f4a0 2024-09-13T16:26:29.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b617fa700 1 -- 172.21.15.119:0/2371785505 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c00fe80 con 0x7f8b6406f4a0 2024-09-13T16:26:29.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6414f720 con 0x7f8b6406f4a0 2024-09-13T16:26:29.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b641510c0 con 0x7f8b6406f4a0 2024-09-13T16:26:29.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b617fa700 1 -- 172.21.15.119:0/2371785505 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8b5c01d460 con 0x7f8b6406f4a0 2024-09-13T16:26:29.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.132+0000 7f8b617fa700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b50066760 0x7f8b50068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:29.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.136+0000 7f8b617fa700 1 -- 172.21.15.119:0/2371785505 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f8b5c084360 con 0x7f8b6406f4a0 2024-09-13T16:26:29.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.136+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b50066760 0x7f8b50068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:29.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.136+0000 7f8b68d2a700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b50066760 0x7f8b50068c20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8b6404ffd0 tx=0x7f8b5400a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:29.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.140+0000 7f8b617fa700 1 -- 172.21.15.119:0/2371785505 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8b641510c0 con 0x7f8b6406f4a0 2024-09-13T16:26:29.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.308+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8b641510c0 con 0x7f8b6406f4a0 2024-09-13T16:26:29.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:29.992+0000 7f8b617fa700 1 -- 172.21.15.119:0/2371785505 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v31) v1 ==== 178+0+0 (secure 0 0 0) 0x7f8b641510c0 con 0x7f8b6406f4a0 2024-09-13T16:26:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b50066760 msgr2=0x7f8b50068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b50066760 0x7f8b50068c20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f8b6404ffd0 tx=0x7f8b5400a000 comp rx=0 tx=0).stop 2024-09-13T16:26:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b6406f4a0 msgr2=0x7f8b641f09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b6406f4a0 0x7f8b641f09e0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f8b6415bdb0 tx=0x7f8b5c00f280 comp rx=0 tx=0).stop 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 shutdown_connections 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b50066760 0x7f8b50068c20 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b6406f4a0 0x7f8b641f09e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b6414f2a0 0x7f8b6406ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 --2- 172.21.15.119:0/2371785505 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b64145200 0x7f8b641513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 >> 172.21.15.119:0/2371785505 conn(0x7f8b640722c0 msgr2=0x7f8b6415a790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 shutdown_connections 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.000+0000 7f8b69d2c700 1 -- 172.21.15.119:0/2371785505 wait complete. 2024-09-13T16:26:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data_ec' removed 2024-09-13T16:26:30.021 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:26:30.021 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T16:26:30.021 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:26:30.021 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:30.021 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:26:30.021 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T16:26:30.022 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:26:30.022 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:30.022 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:26:30.022 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T16:26:30.022 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:26:30.022 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:30.023 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:26:30.023 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T16:26:30.023 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:26:30.023 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:30.034 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:26:30.034 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:26:30.035 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:26:30.036 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:26:30.037 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:26:30.037 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:26:30.038 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:26:30.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:26:30.038 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:26:30.039 DEBUG:teuthology.parallel:result is None 2024-09-13T16:26:30.040 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:26:30.040 DEBUG:teuthology.parallel:result is None 2024-09-13T16:26:30.041 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:26:30.041 DEBUG:teuthology.parallel:result is None 2024-09-13T16:26:30.041 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:26:30.041 DEBUG:teuthology.parallel:result is None 2024-09-13T16:26:30.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T16:26:30.106 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:26:30.113 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:26:30.113 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:26:30.127 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:26:30.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76f2cf700 1 Processor -- start 2024-09-13T16:26:30.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76f2cf700 1 -- start start 2024-09-13T16:26:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76f2cf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe76815fee0 0x7fe76806b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76f2cf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe7680714c0 0x7fe768071900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76f2cf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7680ad110 con 0x7fe768070360 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76f2cf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe768071e40 con 0x7fe7680714c0 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76f2cf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7680acb50 con 0x7fe76815fee0 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76eace700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe768070360 0x7fe7680cc320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51342/0 (socket says 172.21.15.119:51342) 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76eace700 1 -- 172.21.15.119:0/2167168788 learned_addr learned my addr 172.21.15.119:0/2167168788 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76e2cd700 1 --2- 172.21.15.119:0/2167168788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe76815fee0 0x7fe76806b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76dacc700 1 --2- 172.21.15.119:0/2167168788 >> v2:172.21.15.145:3300/0 conn(0x7fe7680714c0 0x7fe768071900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76e2cd700 1 -- 172.21.15.119:0/2167168788 >> v2:172.21.15.145:3300/0 conn(0x7fe7680714c0 msgr2=0x7fe768071900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76e2cd700 1 --2- 172.21.15.119:0/2167168788 >> v2:172.21.15.145:3300/0 conn(0x7fe7680714c0 0x7fe768071900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76e2cd700 1 -- 172.21.15.119:0/2167168788 >> v1:172.21.15.119:6789/0 conn(0x7fe768070360 legacy=0x7fe7680cc320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76e2cd700 1 -- 172.21.15.119:0/2167168788 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7680ac2b0 con 0x7fe76815fee0 2024-09-13T16:26:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76dacc700 1 --2- 172.21.15.119:0/2167168788 >> v2:172.21.15.145:3300/0 conn(0x7fe7680714c0 0x7fe768071900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:30.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.148+0000 7fe76e2cd700 1 --2- 172.21.15.119:0/2167168788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe76815fee0 0x7fe76806b4f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe764009850 tx=0x7fe764009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7628d9f28f57d90b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76d2cb700 1 -- 172.21.15.119:0/2167168788 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764011830 con 0x7fe76815fee0 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76d2cb700 1 -- 172.21.15.119:0/2167168788 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe764011990 con 0x7fe76815fee0 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76d2cb700 1 -- 172.21.15.119:0/2167168788 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe764011d40 con 0x7fe76815fee0 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- 172.21.15.119:0/2167168788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe76815fee0 msgr2=0x7fe76806b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/2167168788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe76815fee0 0x7fe76806b4f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe764009850 tx=0x7fe764009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- 172.21.15.119:0/2167168788 shutdown_connections 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/2167168788 >> v2:172.21.15.145:3300/0 conn(0x7fe7680714c0 0x7fe768071900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/2167168788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe76815fee0 0x7fe76806b4f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- 172.21.15.119:0/2167168788 >> 172.21.15.119:0/2167168788 conn(0x7fe7680ae4b0 msgr2=0x7fe7680b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- 172.21.15.119:0/2167168788 shutdown_connections 2024-09-13T16:26:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- 172.21.15.119:0/2167168788 wait complete. 2024-09-13T16:26:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 Processor -- start 2024-09-13T16:26:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- start start 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe768070360 0x7fe7681ef940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7680714c0 0x7fe7681f5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe76815fee0 0x7fe7681f35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681f4c50 con 0x7fe76815fee0 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681efe80 con 0x7fe7680714c0 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76f2cf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7681f3fd0 con 0x7fe768070360 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76eace700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe76815fee0 0x7fe7681f35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76eace700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe76815fee0 0x7fe7681f35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48976/0 (socket says 172.21.15.119:48976) 2024-09-13T16:26:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.152+0000 7fe76eace700 1 -- 172.21.15.119:0/3809910286 learned_addr learned my addr 172.21.15.119:0/3809910286 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76dacc700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7680714c0 0x7fe7681f5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76e2cd700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe768070360 0x7fe7681ef940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76eace700 1 -- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe768070360 msgr2=0x7fe7681ef940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76eace700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe768070360 0x7fe7681ef940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76eace700 1 -- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7680714c0 msgr2=0x7fe7681f5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76eace700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7680714c0 0x7fe7681f5070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76eace700 1 -- 172.21.15.119:0/3809910286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7580094b0 con 0x7fe76815fee0 2024-09-13T16:26:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76eace700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe76815fee0 0x7fe7681f35f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe76000cbf0 tx=0x7fe76000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe75effd700 1 -- 172.21.15.119:0/3809910286 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76000a9a0 con 0x7fe76815fee0 2024-09-13T16:26:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe75effd700 1 -- 172.21.15.119:0/3809910286 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76000ab00 con 0x7fe76815fee0 2024-09-13T16:26:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7640094b0 con 0x7fe76815fee0 2024-09-13T16:26:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe75effd700 1 -- 172.21.15.119:0/3809910286 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76000ae70 con 0x7fe76815fee0 2024-09-13T16:26:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7681f4590 con 0x7fe76815fee0 2024-09-13T16:26:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7680742c0 con 0x7fe76815fee0 2024-09-13T16:26:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.156+0000 7fe75effd700 1 -- 172.21.15.119:0/3809910286 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe76001b3b0 con 0x7fe76815fee0 2024-09-13T16:26:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.160+0000 7fe75effd700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe74c066730 0x7fe74c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.160+0000 7fe75effd700 1 -- 172.21.15.119:0/3809910286 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fe7600837d0 con 0x7fe76815fee0 2024-09-13T16:26:30.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.160+0000 7fe76e2cd700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe74c066730 0x7fe74c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:30.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.160+0000 7fe76e2cd700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe74c066730 0x7fe74c068bf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe7681f35a0 tx=0x7fe764008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:30.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.164+0000 7fe75effd700 1 -- 172.21.15.119:0/3809910286 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe760050e40 con 0x7fe76815fee0 2024-09-13T16:26:30.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:30.344+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe7681f4d90 con 0x7fe76815fee0 2024-09-13T16:26:31.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.164+0000 7fe75effd700 1 -- 172.21.15.119:0/3809910286 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v36) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe7681f4d90 con 0x7fe76815fee0 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe74c066730 msgr2=0x7fe74c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe74c066730 0x7fe74c068bf0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe7681f35a0 tx=0x7fe764008030 comp rx=0 tx=0).stop 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe76815fee0 msgr2=0x7fe7681f35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe76815fee0 0x7fe7681f35f0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fe76000cbf0 tx=0x7fe76000ccd0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 shutdown_connections 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe74c066730 0x7fe74c068bf0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe76815fee0 0x7fe7681f35f0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7680714c0 0x7fe7681f5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 --2- 172.21.15.119:0/3809910286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe768070360 0x7fe7681ef940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 >> 172.21.15.119:0/3809910286 conn(0x7fe7680ae4b0 msgr2=0x7fe76806ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:31.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 shutdown_connections 2024-09-13T16:26:31.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.168+0000 7fe76f2cf700 1 -- 172.21.15.119:0/3809910286 wait complete. 2024-09-13T16:26:31.175 INFO:teuthology.orchestra.run.smithi119.stderr: removed all blocklist entries 2024-09-13T16:26:31.190 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T16:26:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.304+0000 7f8143053700 1 Processor -- start 2024-09-13T16:26:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- start start 2024-09-13T16:26:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f813c0b3ad0 0x7f813c159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f813c159880 0x7f813c0cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c159d30 con 0x7f813c0b36f0 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c0b2050 con 0x7f813c0b3ad0 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f813c0b2320 con 0x7f813c159880 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f813c0b3ad0 0x7f813c159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f813c0b3ad0 0x7f813c159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48978/0 (socket says 172.21.15.119:48978) 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 -- 172.21.15.119:0/2150889943 learned_addr learned my addr 172.21.15.119:0/2150889943 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f81415f0700 1 --2- 172.21.15.119:0/2150889943 >> v2:172.21.15.145:3300/0 conn(0x7f813c159880 0x7f813c0cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 -- 172.21.15.119:0/2150889943 >> v2:172.21.15.145:3300/0 conn(0x7f813c159880 msgr2=0x7f813c0cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 --2- 172.21.15.119:0/2150889943 >> v2:172.21.15.145:3300/0 conn(0x7f813c159880 0x7f813c0cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 -- 172.21.15.119:0/2150889943 >> v1:172.21.15.145:6789/0 conn(0x7f813c0b36f0 legacy=0x7f813c0b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 -- 172.21.15.119:0/2150889943 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c0b2600 con 0x7f813c0b3ad0 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f81415f0700 1 --2- 172.21.15.119:0/2150889943 >> v2:172.21.15.145:3300/0 conn(0x7f813c159880 0x7f813c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:31.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813bfff700 1 --2- 172.21.15.119:0/2150889943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c159340 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f813c0cb9d0 tx=0x7f812c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a8447555ac4ea4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813b7fe700 1 -- 172.21.15.119:0/2150889943 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c004230 con 0x7f813c0b3ad0 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813b7fe700 1 -- 172.21.15.119:0/2150889943 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c004390 con 0x7f813c0b3ad0 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f813b7fe700 1 -- 172.21.15.119:0/2150889943 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c0036c0 con 0x7f813c0b3ad0 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- 172.21.15.119:0/2150889943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 msgr2=0x7f813c159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 --2- 172.21.15.119:0/2150889943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c159340 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f813c0cb9d0 tx=0x7f812c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- 172.21.15.119:0/2150889943 shutdown_connections 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 --2- 172.21.15.119:0/2150889943 >> v2:172.21.15.145:3300/0 conn(0x7f813c159880 0x7f813c0cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 --2- 172.21.15.119:0/2150889943 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c159340 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- 172.21.15.119:0/2150889943 >> 172.21.15.119:0/2150889943 conn(0x7f813c06a350 msgr2=0x7f813c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- 172.21.15.119:0/2150889943 shutdown_connections 2024-09-13T16:26:31.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.308+0000 7f8143053700 1 -- 172.21.15.119:0/2150889943 wait complete. 2024-09-13T16:26:31.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 Processor -- start 2024-09-13T16:26:31.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 -- start start 2024-09-13T16:26:31.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f813c0b36f0 0x7f813c14e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c147270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f813c159880 0x7f813c14ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c1477b0 con 0x7f813c0b3ad0 2024-09-13T16:26:31.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f813c14d940 con 0x7f813c0b36f0 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8143053700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f813c14dc20 con 0x7f813c159880 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c147270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c147270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48982/0 (socket says 172.21.15.119:48982) 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 -- 172.21.15.119:0/3960375703 learned_addr learned my addr 172.21.15.119:0/3960375703 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f81415f0700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f813c159880 0x7f813c14ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f8140def700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f813c0b36f0 0x7f813c14e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 -- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f813c159880 msgr2=0x7f813c14ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f813c159880 0x7f813c14ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 -- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f813c0b36f0 msgr2=0x7f813c14e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f813c0b36f0 0x7f813c14e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.312+0000 7f813bfff700 1 -- 172.21.15.119:0/3960375703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812c009570 con 0x7f813c0b3ad0 2024-09-13T16:26:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f813bfff700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c147270 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f812c004000 tx=0x7f812c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f81397fa700 1 -- 172.21.15.119:0/3960375703 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00b860 con 0x7f813c0b3ad0 2024-09-13T16:26:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f81397fa700 1 -- 172.21.15.119:0/3960375703 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f812c00b9c0 con 0x7f813c0b3ad0 2024-09-13T16:26:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c14df00 con 0x7f813c0b3ad0 2024-09-13T16:26:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f81397fa700 1 -- 172.21.15.119:0/3960375703 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f812c00bd50 con 0x7f813c0b3ad0 2024-09-13T16:26:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c15cb20 con 0x7f813c0b3ad0 2024-09-13T16:26:31.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c14e420 con 0x7f813c0b3ad0 2024-09-13T16:26:31.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f81397fa700 1 -- 172.21.15.119:0/3960375703 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f812c021070 con 0x7f813c0b3ad0 2024-09-13T16:26:31.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.316+0000 7f81397fa700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8128066730 0x7f8128068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.320+0000 7f81397fa700 1 -- 172.21.15.119:0/3960375703 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f812c0857d0 con 0x7f813c0b3ad0 2024-09-13T16:26:31.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.320+0000 7f8140def700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8128066730 0x7f8128068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.320+0000 7f8140def700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8128066730 0x7f8128068bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f813c0cdf20 tx=0x7f8130007ee0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:31.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.324+0000 7f81397fa700 1 -- 172.21.15.119:0/3960375703 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f813c14e420 con 0x7f813c0b3ad0 2024-09-13T16:26:31.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.552+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f813c14e420 con 0x7f813c0b3ad0 2024-09-13T16:26:31.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.552+0000 7f81397fa700 1 -- 172.21.15.119:0/3960375703 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) v1 ==== 80+0+6645 (secure 0 0 0) 0x7f813c14e420 con 0x7f813c0b3ad0 2024-09-13T16:26:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:31.558 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:26:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_dump": [ 2024-09-13T16:26:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.a", 2024-09-13T16:26:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA==", 2024-09-13T16:26:31.558 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.b", 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g==", 2024-09-13T16:26:31.559 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.c", 2024-09-13T16:26:31.560 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ==", 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.d", 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw==", 2024-09-13T16:26:31.561 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.562 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.0", 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA==", 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.1", 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw==", 2024-09-13T16:26:31.563 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.2", 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ==", 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.564 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.3", 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww==", 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.565 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.4", 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA==", 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.5", 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q==", 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.566 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.6", 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg==", 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.567 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.7", 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ==", 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.0", 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ==", 2024-09-13T16:26:31.568 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.1", 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ==", 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:26:31.569 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.2", 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw==", 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:26:31.570 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.3", 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow==", 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.571 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.admin", 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmpEcAJRAAYorOK9AU5Q/2uOTIUrZhJw==", 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:26:31.572 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.bootstrap-mds", 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBcZ+Rmp51FAxAA3pjIekDS2Og75d3at78vkw==", 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile bootstrap-mds" 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.bootstrap-mgr", 2024-09-13T16:26:31.573 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBcZ+RmMLNFAxAAxXE8FqxKinvZRoNEOk0EQA==", 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile bootstrap-mgr" 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.bootstrap-osd", 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBcZ+RmKsRFAxAAx3/J22t271DSVVZJZLYaAw==", 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile bootstrap-osd" 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.574 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.bootstrap-rbd", 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBcZ+RmctRFAxAAOhqchjen3Z4Oyx2Z5cgI5Q==", 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile bootstrap-rbd" 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBcZ+RmQ+VFAxAAKsFv+TdnYZFtFcyEeSMtIQ==", 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.575 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.bootstrap-rgw", 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBcZ+Rma/lFAxAAywUuSIjXMVxjf2IH8Nr7bA==", 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile bootstrap-rgw" 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.x", 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ==", 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.576 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.y", 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw==", 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:26:31.577 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:26:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8128066730 msgr2=0x7f8128068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8128066730 0x7f8128068bf0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f813c0cdf20 tx=0x7f8130007ee0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 msgr2=0x7f813c147270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c147270 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f812c004000 tx=0x7f812c004770 comp rx=0 tx=0).stop 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 shutdown_connections 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8128066730 0x7f8128068bf0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f813c159880 0x7f813c14ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f813c0b3ad0 0x7f813c147270 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 --2- 172.21.15.119:0/3960375703 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f813c0b36f0 0x7f813c14e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 >> 172.21.15.119:0/3960375703 conn(0x7f813c06a350 msgr2=0x7f813c06fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 shutdown_connections 2024-09-13T16:26:31.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.556+0000 7f8143053700 1 -- 172.21.15.119:0/3960375703 wait complete. 2024-09-13T16:26:31.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-09-13T16:26:31.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.688+0000 7f4574a05700 1 Processor -- start 2024-09-13T16:26:31.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.688+0000 7f4574a05700 1 -- start start 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457008c6b0 0x7f45700ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457015d020 con 0x7f457015cc40 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45700d0720 con 0x7f45700cd860 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45700b44c0 con 0x7f457008c6b0 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f457015cc40 0x7f457008bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51354/0 (socket says 172.21.15.119:51354) 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456f7fe700 1 -- 172.21.15.119:0/3316742094 learned_addr learned my addr 172.21.15.119:0/3316742094 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4566ffd700 1 --2- 172.21.15.119:0/3316742094 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457008c6b0 0x7f45700ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456effd700 1 -- 172.21.15.119:0/3316742094 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2677727907 0 0) 0x7f457015d020 con 0x7f457015cc40 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456effd700 1 -- 172.21.15.119:0/3316742094 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f455c0035d0 con 0x7f457015cc40 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4566ffd700 1 -- 172.21.15.119:0/3316742094 >> v1:172.21.15.145:6789/0 conn(0x7f45700cd860 legacy=0x7f4570166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456effd700 1 -- 172.21.15.119:0/3316742094 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626683009 0 0) 0x7f45700d0720 con 0x7f45700cd860 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4566ffd700 1 -- 172.21.15.119:0/3316742094 >> v1:172.21.15.119:6789/0 conn(0x7f457015cc40 legacy=0x7f457008bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4566ffd700 1 -- 172.21.15.119:0/3316742094 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45700d09c0 con 0x7f457008c6b0 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4566ffd700 1 --2- 172.21.15.119:0/3316742094 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457008c6b0 0x7f45700ce710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4558009880 tx=0x7f4558009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b8191eb41211509 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456effd700 1 -- 172.21.15.119:0/3316742094 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560001670 con 0x7f457008c6b0 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456effd700 1 -- 172.21.15.119:0/3316742094 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4558010590 con 0x7f457008c6b0 2024-09-13T16:26:31.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f456effd700 1 -- 172.21.15.119:0/3316742094 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558010900 con 0x7f457008c6b0 2024-09-13T16:26:31.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- 172.21.15.119:0/3316742094 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457008c6b0 msgr2=0x7f45700ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 --2- 172.21.15.119:0/3316742094 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457008c6b0 0x7f45700ce710 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4558009880 tx=0x7f4558009b90 comp rx=0 tx=0).stop 2024-09-13T16:26:31.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- 172.21.15.119:0/3316742094 shutdown_connections 2024-09-13T16:26:31.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 --2- 172.21.15.119:0/3316742094 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457008c6b0 0x7f45700ce710 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- 172.21.15.119:0/3316742094 >> 172.21.15.119:0/3316742094 conn(0x7f45700b4640 msgr2=0x7f45700b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:31.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- 172.21.15.119:0/3316742094 shutdown_connections 2024-09-13T16:26:31.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.692+0000 7f4574a05700 1 -- 172.21.15.119:0/3316742094 wait complete. 2024-09-13T16:26:31.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 Processor -- start 2024-09-13T16:26:31.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 -- start start 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f457008c6b0 0x7f45701f1e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45700cd860 0x7f45701f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457015cc40 0x7f45701f4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45701f28b0 con 0x7f457008c6b0 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457009da10 con 0x7f45700cd860 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4574a05700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457009ce00 con 0x7f457015cc40 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f457008c6b0 0x7f45701f1e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f457008c6b0 0x7f45701f1e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48996/0 (socket says 172.21.15.119:48996) 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 -- 172.21.15.119:0/2664209780 learned_addr learned my addr 172.21.15.119:0/2664209780 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:31.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456ffff700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457015cc40 0x7f45701f4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4566ffd700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45700cd860 0x7f45701f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 -- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457015cc40 msgr2=0x7f45701f4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457015cc40 0x7f45701f4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 -- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45700cd860 msgr2=0x7f45701f2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45700cd860 0x7f45701f2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 -- 172.21.15.119:0/2664209780 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580094b0 con 0x7f457008c6b0 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f4566ffd700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45700cd860 0x7f45701f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456f7fe700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f457008c6b0 0x7f45701f1e30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f456000c6d0 tx=0x7f456000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:31.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.696+0000 7f456cff9700 1 -- 172.21.15.119:0/2664209780 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560004ca0 con 0x7f457008c6b0 2024-09-13T16:26:31.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f456cff9700 1 -- 172.21.15.119:0/2664209780 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4560004e20 con 0x7f457008c6b0 2024-09-13T16:26:31.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f456cff9700 1 -- 172.21.15.119:0/2664209780 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4560007310 con 0x7f457008c6b0 2024-09-13T16:26:31.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f457009d0e0 con 0x7f457008c6b0 2024-09-13T16:26:31.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f457009d6a0 con 0x7f457008c6b0 2024-09-13T16:26:31.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457009d830 con 0x7f457008c6b0 2024-09-13T16:26:31.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f456cff9700 1 -- 172.21.15.119:0/2664209780 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4560007470 con 0x7f457008c6b0 2024-09-13T16:26:31.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f456cff9700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4550066730 0x7f4550068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:31.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f456cff9700 1 -- 172.21.15.119:0/2664209780 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4560083130 con 0x7f457008c6b0 2024-09-13T16:26:31.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.700+0000 7f4566ffd700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4550066730 0x7f4550068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:31.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.704+0000 7f4566ffd700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4550066730 0x7f4550068bf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f45580097d0 tx=0x7f4558010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:31.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.708+0000 7f456cff9700 1 -- 172.21.15.119:0/2664209780 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4560054140 con 0x7f457008c6b0 2024-09-13T16:26:31.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.940+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f45700d1a60 con 0x7f457008c6b0 2024-09-13T16:26:31.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f456cff9700 1 -- 172.21.15.119:0/2664209780 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f4560007720 con 0x7f457008c6b0 2024-09-13T16:26:31.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4550066730 msgr2=0x7f4550068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f4574a05700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4550066730 0x7f4550068bf0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f45580097d0 tx=0x7f4558010000 comp rx=0 tx=0).stop 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f457008c6b0 msgr2=0x7f45701f1e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f4574a05700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f457008c6b0 0x7f45701f1e30 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f456000c6d0 tx=0x7f456000c9e0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 shutdown_connections 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f4574a05700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4550066730 0x7f4550068bf0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.948+0000 7f4574a05700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f457015cc40 0x7f45701f4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.952+0000 7f4574a05700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45700cd860 0x7f45701f2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.952+0000 7f4574a05700 1 --2- 172.21.15.119:0/2664209780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f457008c6b0 0x7f45701f1e30 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.952+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 >> 172.21.15.119:0/2664209780 conn(0x7f45700b4640 msgr2=0x7f457008bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.952+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 shutdown_connections 2024-09-13T16:26:31.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:31.952+0000 7f4574a05700 1 -- 172.21.15.119:0/2664209780 wait complete. 2024-09-13T16:26:31.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-09-13T16:26:32.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff588865700 1 Processor -- start 2024-09-13T16:26:32.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff588865700 1 -- start start 2024-09-13T16:26:32.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff588865700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5840adb30 0x7ff5840cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff588865700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58406cf60 con 0x7ff5840ce040 2024-09-13T16:26:32.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff588865700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff58406c350 con 0x7ff58406bb90 2024-09-13T16:26:32.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff588865700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58406c630 con 0x7ff5840adb30 2024-09-13T16:26:32.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff583fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff5840ce040 0x7ff5840b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:51366/0 (socket says 172.21.15.119:51366) 2024-09-13T16:26:32.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff583fff700 1 -- 172.21.15.119:0/1519640458 learned_addr learned my addr 172.21.15.119:0/1519640458 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:32.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.080+0000 7ff5837fe700 1 --2- 172.21.15.119:0/1519640458 >> v2:172.21.15.145:3300/0 conn(0x7ff5840adb30 0x7ff5840cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5827fc700 1 -- 172.21.15.119:0/1519640458 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1611664396 0 0) 0x7ff58406cf60 con 0x7ff5840ce040 2024-09-13T16:26:32.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5827fc700 1 -- 172.21.15.119:0/1519640458 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5700035d0 con 0x7ff5840ce040 2024-09-13T16:26:32.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5837fe700 1 -- 172.21.15.119:0/1519640458 >> v1:172.21.15.145:6789/0 conn(0x7ff58406bb90 legacy=0x7ff5840cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5837fe700 1 -- 172.21.15.119:0/1519640458 >> v1:172.21.15.119:6789/0 conn(0x7ff5840ce040 legacy=0x7ff5840b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5837fe700 1 -- 172.21.15.119:0/1519640458 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58406c910 con 0x7ff5840adb30 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5837fe700 1 --2- 172.21.15.119:0/1519640458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5840adb30 0x7ff5840cc060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff574009850 tx=0x7ff574009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5289e8756dd2adba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5827fc700 1 -- 172.21.15.119:0/1519640458 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574011830 con 0x7ff5840adb30 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5827fc700 1 -- 172.21.15.119:0/1519640458 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff574011990 con 0x7ff5840adb30 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff5827fc700 1 -- 172.21.15.119:0/1519640458 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff574011d40 con 0x7ff5840adb30 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 -- 172.21.15.119:0/1519640458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5840adb30 msgr2=0x7ff5840cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 --2- 172.21.15.119:0/1519640458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5840adb30 0x7ff5840cc060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff574009850 tx=0x7ff574009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 -- 172.21.15.119:0/1519640458 shutdown_connections 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 --2- 172.21.15.119:0/1519640458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5840adb30 0x7ff5840cc060 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 -- 172.21.15.119:0/1519640458 >> 172.21.15.119:0/1519640458 conn(0x7ff58406a350 msgr2=0x7ff58406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 -- 172.21.15.119:0/1519640458 shutdown_connections 2024-09-13T16:26:32.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 -- 172.21.15.119:0/1519640458 wait complete. 2024-09-13T16:26:32.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 Processor -- start 2024-09-13T16:26:32.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.084+0000 7ff588865700 1 -- start start 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff58406bb90 0x7ff58415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5840adb30 0x7ff58413b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5840ce040 0x7ff58413c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58413c310 con 0x7ff5840adb30 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff584070470 con 0x7ff58406bb90 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff584070750 con 0x7ff5840ce040 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5840adb30 0x7ff58413b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5840adb30 0x7ff58413b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49006/0 (socket says 172.21.15.119:49006) 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 -- 172.21.15.119:0/187367875 learned_addr learned my addr 172.21.15.119:0/187367875 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff5837fe700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff58406bb90 0x7ff58415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff583fff700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5840ce040 0x7ff58413c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 -- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5840ce040 msgr2=0x7ff58413c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5840ce040 0x7ff58413c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 -- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff58406bb90 msgr2=0x7ff58415a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff58406bb90 0x7ff58415a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 -- 172.21.15.119:0/187367875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5740094b0 con 0x7ff5840adb30 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff582ffd700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5840adb30 0x7ff58413b9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff56c001ed0 tx=0x7ff56c0021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff57bfff700 1 -- 172.21.15.119:0/187367875 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c003c20 con 0x7ff5840adb30 2024-09-13T16:26:32.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff57bfff700 1 -- 172.21.15.119:0/187367875 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff56c003d80 con 0x7ff5840adb30 2024-09-13T16:26:32.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff584070a30 con 0x7ff5840adb30 2024-09-13T16:26:32.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff57bfff700 1 -- 172.21.15.119:0/187367875 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff56c00ab40 con 0x7ff5840adb30 2024-09-13T16:26:32.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58413bf00 con 0x7ff5840adb30 2024-09-13T16:26:32.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.088+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5840705b0 con 0x7ff5840adb30 2024-09-13T16:26:32.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.092+0000 7ff57bfff700 1 -- 172.21.15.119:0/187367875 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff56c00aca0 con 0x7ff5840adb30 2024-09-13T16:26:32.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.092+0000 7ff57bfff700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.092+0000 7ff57bfff700 1 -- 172.21.15.119:0/187367875 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff56c083f30 con 0x7ff5840adb30 2024-09-13T16:26:32.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.092+0000 7ff5837fe700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.092+0000 7ff5837fe700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff5740090b0 tx=0x7ff574008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.100+0000 7ff57bfff700 1 -- 172.21.15.119:0/187367875 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff5840705b0 con 0x7ff5840adb30 2024-09-13T16:26:32.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.328+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7ff58406d3d0 con 0x7ff5840adb30 2024-09-13T16:26:32.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.332+0000 7ff57bfff700 1 -- 172.21.15.119:0/187367875 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff58406d3d0 con 0x7ff5840adb30 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 msgr2=0x7ff564068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff5740090b0 tx=0x7ff574008030 comp rx=0 tx=0).stop 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5840adb30 msgr2=0x7ff58413b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5840adb30 0x7ff58413b9c0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7ff56c001ed0 tx=0x7ff56c0021e0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 shutdown_connections 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5840ce040 0x7ff58413c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5840adb30 0x7ff58413b9c0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 --2- 172.21.15.119:0/187367875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff58406bb90 0x7ff58415a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 >> 172.21.15.119:0/187367875 conn(0x7ff58406a350 msgr2=0x7ff5840cb4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:32.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 shutdown_connections 2024-09-13T16:26:32.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.336+0000 7ff588865700 1 -- 172.21.15.119:0/187367875 wait complete. 2024-09-13T16:26:32.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-09-13T16:26:32.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.460+0000 7f2649cde700 1 Processor -- start 2024-09-13T16:26:32.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.460+0000 7f2649cde700 1 -- start start 2024-09-13T16:26:32.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f26440b36f0 0x7f26440b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 0x7f2644159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26440b2320 con 0x7f26440b3ad0 2024-09-13T16:26:32.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26440b2600 con 0x7f26440b36f0 2024-09-13T16:26:32.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26440b28e0 con 0x7f26440b1e80 2024-09-13T16:26:32.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2643fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f26440b3ad0 0x7f264415d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45958/0 (socket says 172.21.15.119:45958) 2024-09-13T16:26:32.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2643fff700 1 -- 172.21.15.119:0/890577024 learned_addr learned my addr 172.21.15.119:0/890577024 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 0x7f2644159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f26437fe700 1 --2- 172.21.15.119:0/890577024 >> v2:172.21.15.145:3300/0 conn(0x7f26440b36f0 0x7f26440b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344460853 0 0) 0x7f26440b2320 con 0x7f26440b3ad0 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26300035d0 con 0x7f26440b3ad0 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 905980938 0 0) 0x7f26300035d0 con 0x7f26440b3ad0 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 msgr2=0x7f2644159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 0x7f2644159340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 >> v2:172.21.15.145:3300/0 conn(0x7f26440b36f0 msgr2=0x7f26440b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:32.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 --2- 172.21.15.119:0/890577024 >> v2:172.21.15.145:3300/0 conn(0x7f26440b36f0 0x7f26440b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440b2bc0 con 0x7f26440b3ad0 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f263c0019f0 con 0x7f26440b3ad0 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 >> v1:172.21.15.119:6789/0 conn(0x7f26440b3ad0 legacy=0x7f264415d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 0x7f2630003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2630003fa0 0x7f2630004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630004db0 0x7f2630005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26440b2600 con 0x7f2630003fa0 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26440b28e0 con 0x7f2630004db0 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263000ba30 con 0x7f26440b1e80 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2630003fa0 0x7f2630004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f26437fe700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 0x7f2630003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2643fff700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630004db0 0x7f2630005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 -- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 msgr2=0x7f2630003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26440b1e80 0x7f2630003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 -- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630004db0 msgr2=0x7f2630005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630004db0 0x7f2630005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 -- 172.21.15.119:0/890577024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26300035d0 con 0x7f2630003fa0 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2643fff700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630004db0 0x7f2630005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f263bfff700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2630003fa0 0x7f2630004870 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f262c009b70 tx=0x7f26300058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf327333958bba58 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00e660 con 0x7f2630003fa0 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c00e7c0 con 0x7f2630003fa0 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2642ffd700 1 -- 172.21.15.119:0/890577024 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00eb30 con 0x7f2630003fa0 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- 172.21.15.119:0/890577024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2630003fa0 msgr2=0x7f2630004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 --2- 172.21.15.119:0/890577024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2630003fa0 0x7f2630004870 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f262c009b70 tx=0x7f26300058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2643fff700 1 -- 172.21.15.119:0/890577024 reap_dead start 2024-09-13T16:26:32.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- 172.21.15.119:0/890577024 shutdown_connections 2024-09-13T16:26:32.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- 172.21.15.119:0/890577024 >> 172.21.15.119:0/890577024 conn(0x7f264406a350 msgr2=0x7f264406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:32.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- 172.21.15.119:0/890577024 shutdown_connections 2024-09-13T16:26:32.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.464+0000 7f2649cde700 1 -- 172.21.15.119:0/890577024 wait complete. 2024-09-13T16:26:32.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 Processor -- start 2024-09-13T16:26:32.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 -- start start 2024-09-13T16:26:32.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630003fa0 0x7f264413cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f264414b350 0x7f264414cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2644152f10 0x7f2644153390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26441538d0 con 0x7f264414b350 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264414e720 con 0x7f2630003fa0 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2649cde700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264414e290 con 0x7f2644152f10 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f263bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f264414b350 0x7f264414cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f263bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f264414b350 0x7f264414cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50206/0 (socket says 172.21.15.119:50206) 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f263bfff700 1 -- 172.21.15.119:0/688246620 learned_addr learned my addr 172.21.15.119:0/688246620 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f2643fff700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2644152f10 0x7f2644153390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.468+0000 7f26437fe700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630003fa0 0x7f264413cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f263bfff700 1 -- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2644152f10 msgr2=0x7f2644153390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f263bfff700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2644152f10 0x7f2644153390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f263bfff700 1 -- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630003fa0 msgr2=0x7f264413cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f263bfff700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630003fa0 0x7f264413cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f263bfff700 1 -- 172.21.15.119:0/688246620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f264413d4e0 con 0x7f264414b350 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f263bfff700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f264414b350 0x7f264414cea0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2630004710 tx=0x7f262c0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2640ff9700 1 -- 172.21.15.119:0/688246620 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c00e660 con 0x7f264414b350 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2640ff9700 1 -- 172.21.15.119:0/688246620 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f262c013d50 con 0x7f264414b350 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f264414d3e0 con 0x7f264414b350 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2640ff9700 1 -- 172.21.15.119:0/688246620 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f262c004290 con 0x7f264414b350 2024-09-13T16:26:32.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264414d960 con 0x7f264414b350 2024-09-13T16:26:32.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2644067760 con 0x7f264414b350 2024-09-13T16:26:32.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2640ff9700 1 -- 172.21.15.119:0/688246620 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f262c0043f0 con 0x7f264414b350 2024-09-13T16:26:32.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2640ff9700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2624066760 0x7f2624068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f2640ff9700 1 -- 172.21.15.119:0/688246620 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f262c0853e0 con 0x7f264414b350 2024-09-13T16:26:32.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.472+0000 7f26437fe700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2624066760 0x7f2624068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.476+0000 7f26437fe700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2624066760 0x7f2624068c20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2634009bf0 tx=0x7f2634008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.480+0000 7f2640ff9700 1 -- 172.21.15.119:0/688246620 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f262c052a90 con 0x7f264414b350 2024-09-13T16:26:32.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.712+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f26440b1940 con 0x7f264414b350 2024-09-13T16:26:32.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.720+0000 7f2640ff9700 1 -- 172.21.15.119:0/688246620 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) v1 ==== 90+0+0 (secure 0 0 0) 0x7f262c0560b0 con 0x7f264414b350 2024-09-13T16:26:32.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2624066760 msgr2=0x7f2624068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2624066760 0x7f2624068c20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2634009bf0 tx=0x7f2634008040 comp rx=0 tx=0).stop 2024-09-13T16:26:32.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f264414b350 msgr2=0x7f264414cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f264414b350 0x7f264414cea0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f2630004710 tx=0x7f262c0047e0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 shutdown_connections 2024-09-13T16:26:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2624066760 0x7f2624068c20 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2644152f10 0x7f2644153390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f264414b350 0x7f264414cea0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 --2- 172.21.15.119:0/688246620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2630003fa0 0x7f264413cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 >> 172.21.15.119:0/688246620 conn(0x7f264406a350 msgr2=0x7f2644155920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 shutdown_connections 2024-09-13T16:26:32.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.724+0000 7f2649cde700 1 -- 172.21.15.119:0/688246620 wait complete. 2024-09-13T16:26:32.743 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-09-13T16:26:32.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9f292700 1 Processor -- start 2024-09-13T16:26:32.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9f292700 1 -- start start 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9f292700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0e981594f0 0x7f0e9806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9f292700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0e9806da70 0x7f0e980b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9f292700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e980b2280 con 0x7f0e980b3630 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9f292700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e98070c10 con 0x7f0e9806da70 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9f292700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e9806dee0 con 0x7f0e981594f0 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9da8f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0e9806da70 0x7f0e980b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9ea91700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0e980b3630 0x7f0e9815d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45962/0 (socket says 172.21.15.119:45962) 2024-09-13T16:26:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9da8f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0e9806da70 0x7f0e980b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50208/0 (socket says 172.21.15.119:50208) 2024-09-13T16:26:32.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9ea91700 1 -- 172.21.15.119:0/1260593698 learned_addr learned my addr 172.21.15.119:0/1260593698 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:32.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.856+0000 7f0e9e290700 1 --2- 172.21.15.119:0/1260593698 >> v2:172.21.15.145:3300/0 conn(0x7f0e981594f0 0x7f0e9806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9da8f700 1 -- 172.21.15.119:0/1260593698 >> v2:172.21.15.145:3300/0 conn(0x7f0e981594f0 msgr2=0x7f0e9806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:32.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9da8f700 1 --2- 172.21.15.119:0/1260593698 >> v2:172.21.15.145:3300/0 conn(0x7f0e981594f0 0x7f0e9806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9d28e700 1 -- 172.21.15.119:0/1260593698 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136449825 0 0) 0x7f0e980b2280 con 0x7f0e980b3630 2024-09-13T16:26:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9da8f700 1 -- 172.21.15.119:0/1260593698 >> v1:172.21.15.119:6789/0 conn(0x7f0e980b3630 legacy=0x7f0e9815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9da8f700 1 -- 172.21.15.119:0/1260593698 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e980cc890 con 0x7f0e9806da70 2024-09-13T16:26:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9da8f700 1 --2- 172.21.15.119:0/1260593698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e9806da70 0x7f0e980b30f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e9806e0f0 tx=0x7f0e94009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=495c66777d2616e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9d28e700 1 -- 172.21.15.119:0/1260593698 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94011830 con 0x7f0e9806da70 2024-09-13T16:26:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9d28e700 1 -- 172.21.15.119:0/1260593698 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e94011990 con 0x7f0e9806da70 2024-09-13T16:26:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9d28e700 1 -- 172.21.15.119:0/1260593698 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94011d20 con 0x7f0e9806da70 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 -- 172.21.15.119:0/1260593698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e9806da70 msgr2=0x7f0e980b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 --2- 172.21.15.119:0/1260593698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e9806da70 0x7f0e980b30f0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e9806e0f0 tx=0x7f0e94009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 -- 172.21.15.119:0/1260593698 shutdown_connections 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 --2- 172.21.15.119:0/1260593698 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e9806da70 0x7f0e980b30f0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 --2- 172.21.15.119:0/1260593698 >> v2:172.21.15.145:3300/0 conn(0x7f0e981594f0 0x7f0e9806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 -- 172.21.15.119:0/1260593698 >> 172.21.15.119:0/1260593698 conn(0x7f0e9806a350 msgr2=0x7f0e9806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 -- 172.21.15.119:0/1260593698 shutdown_connections 2024-09-13T16:26:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.860+0000 7f0e9f292700 1 -- 172.21.15.119:0/1260593698 wait complete. 2024-09-13T16:26:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 Processor -- start 2024-09-13T16:26:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 -- start start 2024-09-13T16:26:32.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e9806da70 0x7f0e98146c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e980b3630 0x7f0e98166730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e981594f0 0x7f0e98166c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e98147180 con 0x7f0e981594f0 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e981661e0 con 0x7f0e980b3630 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e981f25c0 con 0x7f0e9806da70 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9ea91700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e981594f0 0x7f0e98166c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9ea91700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e981594f0 0x7f0e98166c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50224/0 (socket says 172.21.15.119:50224) 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9ea91700 1 -- 172.21.15.119:0/3866944781 learned_addr learned my addr 172.21.15.119:0/3866944781 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9da8f700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e980b3630 0x7f0e98166730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9e290700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e9806da70 0x7f0e98146c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9da8f700 1 -- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e9806da70 msgr2=0x7f0e98146c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9da8f700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e9806da70 0x7f0e98146c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9da8f700 1 -- 172.21.15.119:0/3866944781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e981594f0 msgr2=0x7f0e98166c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9da8f700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e981594f0 0x7f0e98166c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9da8f700 1 -- 172.21.15.119:0/3866944781 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e940094b0 con 0x7f0e980b3630 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9ea91700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e981594f0 0x7f0e98166c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9da8f700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e980b3630 0x7f0e98166730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e94016000 tx=0x7f0e94004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e8effd700 1 -- 172.21.15.119:0/3866944781 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94004550 con 0x7f0e980b3630 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e8effd700 1 -- 172.21.15.119:0/3866944781 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e9401f070 con 0x7f0e980b3630 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.864+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e981f28a0 con 0x7f0e980b3630 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.868+0000 7f0e8effd700 1 -- 172.21.15.119:0/3866944781 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e94003900 con 0x7f0e980b3630 2024-09-13T16:26:32.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.868+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e981f22b0 con 0x7f0e980b3630 2024-09-13T16:26:32.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.868+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e98066aa0 con 0x7f0e980b3630 2024-09-13T16:26:32.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.868+0000 7f0e8effd700 1 -- 172.21.15.119:0/3866944781 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0e94003a60 con 0x7f0e980b3630 2024-09-13T16:26:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.868+0000 7f0e8effd700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e84066730 0x7f0e84068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.868+0000 7f0e8effd700 1 -- 172.21.15.119:0/3866944781 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0e94021070 con 0x7f0e980b3630 2024-09-13T16:26:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.868+0000 7f0e9e290700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e84066730 0x7f0e84068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.872+0000 7f0e9e290700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e84066730 0x7f0e84068bf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009b50 tx=0x7f0e88009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:32.880+0000 7f0e8effd700 1 -- 172.21.15.119:0/3866944781 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0e94052c10 con 0x7f0e980b3630 2024-09-13T16:26:33.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.116+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f0e981f2dc0 con 0x7f0e980b3630 2024-09-13T16:26:33.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.124+0000 7f0e8effd700 1 -- 172.21.15.119:0/3866944781 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0e981f2dc0 con 0x7f0e980b3630 2024-09-13T16:26:33.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e84066730 msgr2=0x7f0e84068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e84066730 0x7f0e84068bf0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0e88009b50 tx=0x7f0e88009240 comp rx=0 tx=0).stop 2024-09-13T16:26:33.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e980b3630 msgr2=0x7f0e98166730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e980b3630 0x7f0e98166730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0e94016000 tx=0x7f0e94004770 comp rx=0 tx=0).stop 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 shutdown_connections 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e84066730 0x7f0e84068bf0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e981594f0 0x7f0e98166c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e980b3630 0x7f0e98166730 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 --2- 172.21.15.119:0/3866944781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e9806da70 0x7f0e98146c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 >> 172.21.15.119:0/3866944781 conn(0x7f0e9806a350 msgr2=0x7f0e980ade70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 shutdown_connections 2024-09-13T16:26:33.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.128+0000 7f0e9f292700 1 -- 172.21.15.119:0/3866944781 wait complete. 2024-09-13T16:26:33.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-09-13T16:26:33.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 Processor -- start 2024-09-13T16:26:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 -- start start 2024-09-13T16:26:33.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f612408c6b0 0x7f61240ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f612415d020 con 0x7f612415cc40 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61240d0720 con 0x7f61240cd860 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61240b44c0 con 0x7f612408c6b0 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f61290f2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f612415cc40 0x7f612408bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45972/0 (socket says 172.21.15.119:45972) 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f61290f2700 1 -- 172.21.15.119:0/3042056010 learned_addr learned my addr 172.21.15.119:0/3042056010 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f61288f1700 1 --2- 172.21.15.119:0/3042056010 >> v2:172.21.15.119:3300/0 conn(0x7f612408c6b0 0x7f61240ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f6117fff700 1 -- 172.21.15.119:0/3042056010 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 931646484 0 0) 0x7f612415d020 con 0x7f612415cc40 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f6117fff700 1 -- 172.21.15.119:0/3042056010 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61100035d0 con 0x7f612415cc40 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f61288f1700 1 -- 172.21.15.119:0/3042056010 >> v1:172.21.15.145:6789/0 conn(0x7f61240cd860 legacy=0x7f6124166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f61288f1700 1 -- 172.21.15.119:0/3042056010 >> v1:172.21.15.119:6789/0 conn(0x7f612415cc40 legacy=0x7f612408bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f6117fff700 1 -- 172.21.15.119:0/3042056010 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711107324 0 0) 0x7f61100035d0 con 0x7f612415cc40 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f61288f1700 1 -- 172.21.15.119:0/3042056010 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61240d09c0 con 0x7f612408c6b0 2024-09-13T16:26:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f61288f1700 1 --2- 172.21.15.119:0/3042056010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612408c6b0 0x7f61240ce710 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f611c009dc0 tx=0x7f611c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4062a50b16f99ed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:33.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f6117fff700 1 -- 172.21.15.119:0/3042056010 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611c00e070 con 0x7f612408c6b0 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f6117fff700 1 -- 172.21.15.119:0/3042056010 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f611c003a40 con 0x7f612408c6b0 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f6117fff700 1 -- 172.21.15.119:0/3042056010 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6120001670 con 0x7f612408c6b0 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 -- 172.21.15.119:0/3042056010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612408c6b0 msgr2=0x7f61240ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.260+0000 7f612a0f4700 1 --2- 172.21.15.119:0/3042056010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612408c6b0 0x7f61240ce710 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f611c009dc0 tx=0x7f611c00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- 172.21.15.119:0/3042056010 shutdown_connections 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 --2- 172.21.15.119:0/3042056010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612408c6b0 0x7f61240ce710 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- 172.21.15.119:0/3042056010 >> 172.21.15.119:0/3042056010 conn(0x7f61240b4640 msgr2=0x7f61240b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- 172.21.15.119:0/3042056010 shutdown_connections 2024-09-13T16:26:33.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- 172.21.15.119:0/3042056010 wait complete. 2024-09-13T16:26:33.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 Processor -- start 2024-09-13T16:26:33.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- start start 2024-09-13T16:26:33.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f612408c6b0 0x7f6124153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61240cd860 0x7f61240b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612415cc40 0x7f61240ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61240b7970 con 0x7f612415cc40 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61240b7620 con 0x7f61240cd860 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f612a0f4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61240b6360 con 0x7f612408c6b0 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f61298f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612415cc40 0x7f61240ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f61298f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612415cc40 0x7f61240ca320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50250/0 (socket says 172.21.15.119:50250) 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f61298f3700 1 -- 172.21.15.119:0/509613741 learned_addr learned my addr 172.21.15.119:0/509613741 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.264+0000 7f61288f1700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61240cd860 0x7f61240b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61290f2700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f612408c6b0 0x7f6124153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61298f3700 1 -- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f612408c6b0 msgr2=0x7f6124153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61298f3700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f612408c6b0 0x7f6124153980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61298f3700 1 -- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61240cd860 msgr2=0x7f61240b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61298f3700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61240cd860 0x7f61240b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61298f3700 1 -- 172.21.15.119:0/509613741 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c009a70 con 0x7f612415cc40 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61290f2700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f612408c6b0 0x7f6124153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f61298f3700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612415cc40 0x7f61240ca320 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f611800c3c0 tx=0x7f611800c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f6115ffb700 1 -- 172.21.15.119:0/509613741 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61180052d0 con 0x7f612415cc40 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f6115ffb700 1 -- 172.21.15.119:0/509613741 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6118005430 con 0x7f612415cc40 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61240b6640 con 0x7f612415cc40 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f6115ffb700 1 -- 172.21.15.119:0/509613741 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61180055a0 con 0x7f612415cc40 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6124153ec0 con 0x7f612415cc40 2024-09-13T16:26:33.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61240b7ab0 con 0x7f612415cc40 2024-09-13T16:26:33.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.268+0000 7f6115ffb700 1 -- 172.21.15.119:0/509613741 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f61180038e0 con 0x7f612415cc40 2024-09-13T16:26:33.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.272+0000 7f6115ffb700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6108066730 0x7f6108068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.272+0000 7f6115ffb700 1 -- 172.21.15.119:0/509613741 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6118083270 con 0x7f612415cc40 2024-09-13T16:26:33.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.272+0000 7f61290f2700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6108066730 0x7f6108068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.272+0000 7f61290f2700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6108066730 0x7f6108068bf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f612406c9d0 tx=0x7f6120009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:33.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.276+0000 7f6115ffb700 1 -- 172.21.15.119:0/509613741 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f61240b7ab0 con 0x7f612415cc40 2024-09-13T16:26:33.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.508+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f61240b7ab0 con 0x7f612415cc40 2024-09-13T16:26:33.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.516+0000 7f6115ffb700 1 -- 172.21.15.119:0/509613741 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7f61240b7ab0 con 0x7f612415cc40 2024-09-13T16:26:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6108066730 msgr2=0x7f6108068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6108066730 0x7f6108068bf0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f612406c9d0 tx=0x7f6120009040 comp rx=0 tx=0).stop 2024-09-13T16:26:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612415cc40 msgr2=0x7f61240ca320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612415cc40 0x7f61240ca320 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f611800c3c0 tx=0x7f611800c780 comp rx=0 tx=0).stop 2024-09-13T16:26:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 shutdown_connections 2024-09-13T16:26:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6108066730 0x7f6108068bf0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f612415cc40 0x7f61240ca320 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f61240cd860 0x7f61240b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 --2- 172.21.15.119:0/509613741 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f612408c6b0 0x7f6124153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 >> 172.21.15.119:0/509613741 conn(0x7f61240b4640 msgr2=0x7f61240af8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:33.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 shutdown_connections 2024-09-13T16:26:33.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.520+0000 7f612a0f4700 1 -- 172.21.15.119:0/509613741 wait complete. 2024-09-13T16:26:33.538 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-09-13T16:26:33.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 Processor -- start 2024-09-13T16:26:33.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- start start 2024-09-13T16:26:33.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f70080af6d0 0x7f70080b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f700806efa0 0x7f70080b4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70080ae230 con 0x7f70080b3800 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70080b4660 con 0x7f70080af6d0 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700806f540 con 0x7f700806efa0 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f70080af6d0 0x7f70080b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f70080af6d0 0x7f70080b3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50252/0 (socket says 172.21.15.119:50252) 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 -- 172.21.15.119:0/921456466 learned_addr learned my addr 172.21.15.119:0/921456466 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700ec30700 1 --2- 172.21.15.119:0/921456466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f700806efa0 0x7f70080b4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 -- 172.21.15.119:0/921456466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f700806efa0 msgr2=0x7f70080b4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 --2- 172.21.15.119:0/921456466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f700806efa0 0x7f70080b4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 -- 172.21.15.119:0/921456466 >> v1:172.21.15.145:6789/0 conn(0x7f70080b3800 legacy=0x7f700806e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 -- 172.21.15.119:0/921456466 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700806f820 con 0x7f70080af6d0 2024-09-13T16:26:33.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700ec30700 1 --2- 172.21.15.119:0/921456466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f700806efa0 0x7f70080b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:33.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700dc2e700 1 --2- 172.21.15.119:0/921456466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f70080b3be0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7000009e50 tx=0x7f700000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3e231b4bf0ba3d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:33.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700d42d700 1 -- 172.21.15.119:0/921456466 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700000e070 con 0x7f70080af6d0 2024-09-13T16:26:33.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700d42d700 1 -- 172.21.15.119:0/921456466 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000003a40 con 0x7f70080af6d0 2024-09-13T16:26:33.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700d42d700 1 -- 172.21.15.119:0/921456466 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000003df0 con 0x7f70080af6d0 2024-09-13T16:26:33.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- 172.21.15.119:0/921456466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 msgr2=0x7f70080b3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 --2- 172.21.15.119:0/921456466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f70080b3be0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f7000009e50 tx=0x7f700000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:33.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- 172.21.15.119:0/921456466 shutdown_connections 2024-09-13T16:26:33.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 --2- 172.21.15.119:0/921456466 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f700806efa0 0x7f70080b4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 --2- 172.21.15.119:0/921456466 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f70080b3be0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- 172.21.15.119:0/921456466 >> 172.21.15.119:0/921456466 conn(0x7f70080b5d20 msgr2=0x7f70080b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:33.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- 172.21.15.119:0/921456466 shutdown_connections 2024-09-13T16:26:33.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.648+0000 7f700f431700 1 -- 172.21.15.119:0/921456466 wait complete. 2024-09-13T16:26:33.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.652+0000 7f700f431700 1 Processor -- start 2024-09-13T16:26:33.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.652+0000 7f700f431700 1 -- start start 2024-09-13T16:26:33.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f700806efa0 0x7f7008146850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f700814f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70080b3800 0x7f7008145f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70080b1ff0 con 0x7f70080af6d0 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700814ae70 con 0x7f700806efa0 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f700814ed90 con 0x7f70080b3800 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f700814f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f700814f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50254/0 (socket says 172.21.15.119:50254) 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 -- 172.21.15.119:0/2410688369 learned_addr learned my addr 172.21.15.119:0/2410688369 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700e42f700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f700806efa0 0x7f7008146850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700ec30700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70080b3800 0x7f7008145f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 -- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70080b3800 msgr2=0x7f7008145f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70080b3800 0x7f7008145f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 -- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f700806efa0 msgr2=0x7f7008146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f700806efa0 0x7f7008146850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 -- 172.21.15.119:0/2410688369 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7000009b30 con 0x7f70080af6d0 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700dc2e700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f700814f220 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f70080b5cd0 tx=0x7f7000004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f7006ffd700 1 -- 172.21.15.119:0/2410688369 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70000044c0 con 0x7f70080af6d0 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f7006ffd700 1 -- 172.21.15.119:0/2410688369 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7000004640 con 0x7f70080af6d0 2024-09-13T16:26:33.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700814ab90 con 0x7f70080af6d0 2024-09-13T16:26:33.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f7006ffd700 1 -- 172.21.15.119:0/2410688369 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700001c440 con 0x7f70080af6d0 2024-09-13T16:26:33.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f700814ea20 con 0x7f70080af6d0 2024-09-13T16:26:33.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.656+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7008074560 con 0x7f70080af6d0 2024-09-13T16:26:33.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.660+0000 7f7006ffd700 1 -- 172.21.15.119:0/2410688369 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f700001c5a0 con 0x7f70080af6d0 2024-09-13T16:26:33.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.660+0000 7f7006ffd700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6ff4066730 0x7f6ff4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:33.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.660+0000 7f7006ffd700 1 -- 172.21.15.119:0/2410688369 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f7000085400 con 0x7f70080af6d0 2024-09-13T16:26:33.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.660+0000 7f700e42f700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6ff4066730 0x7f6ff4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:33.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.660+0000 7f700e42f700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6ff4066730 0x7f6ff4068bf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f70081509c0 tx=0x7f6ffc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:33.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.664+0000 7f7006ffd700 1 -- 172.21.15.119:0/2410688369 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f70000528f0 con 0x7f70080af6d0 2024-09-13T16:26:33.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.896+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f700806e890 con 0x7f70080af6d0 2024-09-13T16:26:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.900+0000 7f7006ffd700 1 -- 172.21.15.119:0/2410688369 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) v1 ==== 90+0+0 (secure 0 0 0) 0x7f70000740a0 con 0x7f70080af6d0 2024-09-13T16:26:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6ff4066730 msgr2=0x7f6ff4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6ff4066730 0x7f6ff4068bf0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f70081509c0 tx=0x7f6ffc009040 comp rx=0 tx=0).stop 2024-09-13T16:26:33.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 msgr2=0x7f700814f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f700814f220 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f70080b5cd0 tx=0x7f7000004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 shutdown_connections 2024-09-13T16:26:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6ff4066730 0x7f6ff4068bf0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f70080b3800 0x7f7008145f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f70080af6d0 0x7f700814f220 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 --2- 172.21.15.119:0/2410688369 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f700806efa0 0x7f7008146850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 >> 172.21.15.119:0/2410688369 conn(0x7f70080b5d20 msgr2=0x7f7008151390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 shutdown_connections 2024-09-13T16:26:33.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:33.904+0000 7f700f431700 1 -- 172.21.15.119:0/2410688369 wait complete. 2024-09-13T16:26:33.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:34.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.032+0000 7fd925657700 1 Processor -- start 2024-09-13T16:26:34.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.032+0000 7fd925657700 1 -- start start 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9200adb30 0x7fd9200cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92006cf60 con 0x7fd9200ce040 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd92006c350 con 0x7fd92006bb90 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd92006c630 con 0x7fd9200adb30 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9200adb30 0x7fd9200cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd924e56700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd9200ce040 0x7fd9200b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45976/0 (socket says 172.21.15.119:45976) 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd924e56700 1 -- 172.21.15.119:0/1883644633 learned_addr learned my addr 172.21.15.119:0/1883644633 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91effd700 1 -- 172.21.15.119:0/1883644633 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1963758710 0 0) 0x7fd92006cf60 con 0x7fd9200ce040 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91effd700 1 -- 172.21.15.119:0/1883644633 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9080035d0 con 0x7fd9200ce040 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91ffff700 1 -- 172.21.15.119:0/1883644633 >> v1:172.21.15.145:6789/0 conn(0x7fd92006bb90 legacy=0x7fd9200cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91ffff700 1 -- 172.21.15.119:0/1883644633 >> v1:172.21.15.119:6789/0 conn(0x7fd9200ce040 legacy=0x7fd9200b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91ffff700 1 -- 172.21.15.119:0/1883644633 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92006c910 con 0x7fd9200adb30 2024-09-13T16:26:34.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91ffff700 1 --2- 172.21.15.119:0/1883644633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9200adb30 0x7fd9200cc060 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd910009850 tx=0x7fd910009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31e6129728055b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91effd700 1 -- 172.21.15.119:0/1883644633 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910011830 con 0x7fd9200adb30 2024-09-13T16:26:34.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91effd700 1 -- 172.21.15.119:0/1883644633 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910011990 con 0x7fd9200adb30 2024-09-13T16:26:34.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd91effd700 1 -- 172.21.15.119:0/1883644633 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd910011d40 con 0x7fd9200adb30 2024-09-13T16:26:34.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- 172.21.15.119:0/1883644633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9200adb30 msgr2=0x7fd9200cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 --2- 172.21.15.119:0/1883644633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9200adb30 0x7fd9200cc060 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fd910009850 tx=0x7fd910009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:34.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- 172.21.15.119:0/1883644633 shutdown_connections 2024-09-13T16:26:34.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 --2- 172.21.15.119:0/1883644633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9200adb30 0x7fd9200cc060 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- 172.21.15.119:0/1883644633 >> 172.21.15.119:0/1883644633 conn(0x7fd92006a350 msgr2=0x7fd92006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- 172.21.15.119:0/1883644633 shutdown_connections 2024-09-13T16:26:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.036+0000 7fd925657700 1 -- 172.21.15.119:0/1883644633 wait complete. 2024-09-13T16:26:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 Processor -- start 2024-09-13T16:26:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 -- start start 2024-09-13T16:26:34.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd92006bb90 0x7fd92013be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9200adb30 0x7fd920148f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9200ce040 0x7fd92013ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92013d810 con 0x7fd92006bb90 2024-09-13T16:26:34.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd92015bd70 con 0x7fd9200ce040 2024-09-13T16:26:34.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9201592e0 con 0x7fd9200adb30 2024-09-13T16:26:34.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd92006bb90 0x7fd92013be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd92006bb90 0x7fd92013be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50268/0 (socket says 172.21.15.119:50268) 2024-09-13T16:26:34.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 -- 172.21.15.119:0/3565464314 learned_addr learned my addr 172.21.15.119:0/3565464314 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:34.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd924e56700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9200ce040 0x7fd92013ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91f7fe700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9200adb30 0x7fd920148f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 -- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9200adb30 msgr2=0x7fd920148f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9200adb30 0x7fd920148f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 -- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9200ce040 msgr2=0x7fd92013ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9200ce040 0x7fd92013ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 -- 172.21.15.119:0/3565464314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9100094b0 con 0x7fd92006bb90 2024-09-13T16:26:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91f7fe700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9200adb30 0x7fd920148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91ffff700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd92006bb90 0x7fd92013be70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd910000c00 tx=0x7fd910004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91cff9700 1 -- 172.21.15.119:0/3565464314 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91001aac0 con 0x7fd92006bb90 2024-09-13T16:26:34.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91cff9700 1 -- 172.21.15.119:0/3565464314 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd91001f070 con 0x7fd92006bb90 2024-09-13T16:26:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd92015bfc0 con 0x7fd92006bb90 2024-09-13T16:26:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd91cff9700 1 -- 172.21.15.119:0/3565464314 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91001ae60 con 0x7fd92006bb90 2024-09-13T16:26:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.040+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd92015c580 con 0x7fd92006bb90 2024-09-13T16:26:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.044+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9200cc060 con 0x7fd92006bb90 2024-09-13T16:26:34.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.044+0000 7fd91cff9700 1 -- 172.21.15.119:0/3565464314 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd91001b430 con 0x7fd92006bb90 2024-09-13T16:26:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.044+0000 7fd91cff9700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd90c066730 0x7fd90c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.044+0000 7fd91cff9700 1 -- 172.21.15.119:0/3565464314 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd910085100 con 0x7fd92006bb90 2024-09-13T16:26:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.044+0000 7fd91f7fe700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd90c066730 0x7fd90c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.044+0000 7fd91f7fe700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd90c066730 0x7fd90c068bf0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd920159ac0 tx=0x7fd914009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.052+0000 7fd91cff9700 1 -- 172.21.15.119:0/3565464314 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd910052690 con 0x7fd92006bb90 2024-09-13T16:26:34.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.224+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9201426b0 con 0x7fd92006bb90 2024-09-13T16:26:34.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.224+0000 7fd91cff9700 1 -- 172.21.15.119:0/3565464314 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) v1 ==== 74+0+13122 (secure 0 0 0) 0x7fd910055cb0 con 0x7fd92006bb90 2024-09-13T16:26:34.231 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:34.231 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":36,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:31.159082+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":26,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd90c066730 msgr2=0x7fd90c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd90c066730 0x7fd90c068bf0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fd920159ac0 tx=0x7fd914009040 comp rx=0 tx=0).stop 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd92006bb90 msgr2=0x7fd92013be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd92006bb90 0x7fd92013be70 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fd910000c00 tx=0x7fd910004770 comp rx=0 tx=0).stop 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 shutdown_connections 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd90c066730 0x7fd90c068bf0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9200ce040 0x7fd92013ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9200adb30 0x7fd920148f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 --2- 172.21.15.119:0/3565464314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd92006bb90 0x7fd92013be70 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 >> 172.21.15.119:0/3565464314 conn(0x7fd92006a350 msgr2=0x7fd920066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 shutdown_connections 2024-09-13T16:26:34.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.228+0000 7fd925657700 1 -- 172.21.15.119:0/3565464314 wait complete. 2024-09-13T16:26:34.245 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:26:34.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:34.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b24fbf700 1 Processor -- start 2024-09-13T16:26:34.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b24fbf700 1 -- start start 2024-09-13T16:26:34.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b24fbf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2006db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b24fbf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8b20168a10 0x7f8b2016ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b24fbf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b20162270 con 0x7f8b200b1ad0 2024-09-13T16:26:34.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b24fbf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b2006e080 con 0x7f8b20168a10 2024-09-13T16:26:34.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b24fbf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2006ce40 con 0x7f8b20071c10 2024-09-13T16:26:34.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1e59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8b200b1ad0 0x7f8b200b1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45982/0 (socket says 172.21.15.119:45982) 2024-09-13T16:26:34.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1e59c700 1 -- 172.21.15.119:0/2718412951 learned_addr learned my addr 172.21.15.119:0/2718412951 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:34.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1ed9d700 1 --2- 172.21.15.119:0/2718412951 >> v2:172.21.15.145:3300/0 conn(0x7f8b20168a10 0x7f8b2016ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1dd9b700 1 --2- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2006db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1d59a700 1 -- 172.21.15.119:0/2718412951 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894671893 0 0) 0x7f8b20162270 con 0x7f8b200b1ad0 2024-09-13T16:26:34.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1d59a700 1 -- 172.21.15.119:0/2718412951 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b080035d0 con 0x7f8b200b1ad0 2024-09-13T16:26:34.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1ed9d700 1 -- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 msgr2=0x7f8b2006db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1ed9d700 1 --2- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2006db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1ed9d700 1 -- 172.21.15.119:0/2718412951 >> v1:172.21.15.119:6789/0 conn(0x7f8b200b1ad0 legacy=0x7f8b200b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1ed9d700 1 -- 172.21.15.119:0/2718412951 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b200702f0 con 0x7f8b20168a10 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1ed9d700 1 --2- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b2016ae00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009d90 tx=0x7f8b1800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1a99a6fa51722 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1d59a700 1 -- 172.21.15.119:0/2718412951 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800e070 con 0x7f8b20168a10 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1d59a700 1 -- 172.21.15.119:0/2718412951 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1800bb20 con 0x7f8b20168a10 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.360+0000 7f8b1d59a700 1 -- 172.21.15.119:0/2718412951 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1800be90 con 0x7f8b20168a10 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 msgr2=0x7f8b2016ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b2016ae00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8b18009d90 tx=0x7f8b1800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2718412951 shutdown_connections 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b2016ae00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2718412951 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2006db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2718412951 >> 172.21.15.119:0/2718412951 conn(0x7f8b200b0560 msgr2=0x7f8b200b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:34.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2718412951 shutdown_connections 2024-09-13T16:26:34.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2718412951 wait complete. 2024-09-13T16:26:34.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 Processor -- start 2024-09-13T16:26:34.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- start start 2024-09-13T16:26:34.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2015cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b200b1ad0 0x7f8b200a92a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b200ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b20155580 con 0x7f8b200b1ad0 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b200ab2f0 con 0x7f8b20168a10 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b24fbf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b200a8550 con 0x7f8b20071c10 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b200b1ad0 0x7f8b200a92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b200b1ad0 0x7f8b200a92a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50276/0 (socket says 172.21.15.119:50276) 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 -- 172.21.15.119:0/2113549244 learned_addr learned my addr 172.21.15.119:0/2113549244 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1ed9d700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b200ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1e59c700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2015cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 -- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 msgr2=0x7f8b2015cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2015cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 -- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 msgr2=0x7f8b200ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b200ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 -- 172.21.15.119:0/2113549244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b18009a70 con 0x7f8b200b1ad0 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1e59c700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1ed9d700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b200ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.364+0000 7f8b1dd9b700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b200b1ad0 0x7f8b200a92a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000e9d0 tx=0x7f8b1000ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b0effd700 1 -- 172.21.15.119:0/2113549244 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b10009720 con 0x7f8b200b1ad0 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b0effd700 1 -- 172.21.15.119:0/2113549244 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1000c850 con 0x7f8b200b1ad0 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b200a97e0 con 0x7f8b200b1ad0 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b0effd700 1 -- 172.21.15.119:0/2113549244 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1000cbc0 con 0x7f8b200b1ad0 2024-09-13T16:26:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b200a8aa0 con 0x7f8b200b1ad0 2024-09-13T16:26:34.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b20073590 con 0x7f8b200b1ad0 2024-09-13T16:26:34.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b0effd700 1 -- 172.21.15.119:0/2113549244 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8b1000cd20 con 0x7f8b200b1ad0 2024-09-13T16:26:34.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b0effd700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b00066730 0x7f8b00068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b0effd700 1 -- 172.21.15.119:0/2113549244 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8b10084070 con 0x7f8b200b1ad0 2024-09-13T16:26:34.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b1e59c700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b00066730 0x7f8b00068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.368+0000 7f8b1e59c700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b00066730 0x7f8b00068bf0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005930 tx=0x7f8b14009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.376+0000 7f8b0effd700 1 -- 172.21.15.119:0/2113549244 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8b10051600 con 0x7f8b200b1ad0 2024-09-13T16:26:34.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.548+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8b200b2650 con 0x7f8b200b1ad0 2024-09-13T16:26:34.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.548+0000 7f8b0effd700 1 -- 172.21.15.119:0/2113549244 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8b10054c20 con 0x7f8b200b1ad0 2024-09-13T16:26:34.554 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b00066730 msgr2=0x7f8b00068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b00066730 0x7f8b00068bf0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f8b14005930 tx=0x7f8b14009040 comp rx=0 tx=0).stop 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b200b1ad0 msgr2=0x7f8b200a92a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b200b1ad0 0x7f8b200a92a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f8b1000e9d0 tx=0x7f8b1000ed90 comp rx=0 tx=0).stop 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 shutdown_connections 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b00066730 0x7f8b00068bf0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b20168a10 0x7f8b200ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b200b1ad0 0x7f8b200a92a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 --2- 172.21.15.119:0/2113549244 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b20071c10 0x7f8b2015cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 >> 172.21.15.119:0/2113549244 conn(0x7f8b200b0560 msgr2=0x7f8b20160680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:34.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 shutdown_connections 2024-09-13T16:26:34.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.552+0000 7f8b24fbf700 1 -- 172.21.15.119:0/2113549244 wait complete. 2024-09-13T16:26:34.569 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:26:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 Processor -- start 2024-09-13T16:26:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- start start 2024-09-13T16:26:34.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00d37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0b00d1620 0x7fd0b00ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b00d1190 con 0x7fd0b01607c0 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00ceaf0 con 0x7fd0b00d1620 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00d0860 con 0x7fd0b015ce70 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0b00d1620 0x7fd0b00ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0b00d1620 0x7fd0b00ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50282/0 (socket says 172.21.15.119:50282) 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 -- 172.21.15.119:0/1632575433 learned_addr learned my addr 172.21.15.119:0/1632575433 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b54ac700 1 --2- 172.21.15.119:0/1632575433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00d37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 -- 172.21.15.119:0/1632575433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 msgr2=0x7fd0b00d37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 --2- 172.21.15.119:0/1632575433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00d37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 -- 172.21.15.119:0/1632575433 >> v1:172.21.15.145:6789/0 conn(0x7fd0b01607c0 legacy=0x7fd0b00b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 -- 172.21.15.119:0/1632575433 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00d0b40 con 0x7fd0b00d1620 2024-09-13T16:26:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b54ac700 1 --2- 172.21.15.119:0/1632575433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00d37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b4cab700 1 --2- 172.21.15.119:0/1632575433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00ce2a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e80 tx=0x7fd0a800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbec9b4d65e16173 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0a7fff700 1 -- 172.21.15.119:0/1632575433 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a800e070 con 0x7fd0b00d1620 2024-09-13T16:26:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0a7fff700 1 -- 172.21.15.119:0/1632575433 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a8003a40 con 0x7fd0b00d1620 2024-09-13T16:26:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0a7fff700 1 -- 172.21.15.119:0/1632575433 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8003df0 con 0x7fd0b00d1620 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- 172.21.15.119:0/1632575433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 msgr2=0x7fd0b00ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1632575433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00ce2a0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e80 tx=0x7fd0a800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- 172.21.15.119:0/1632575433 shutdown_connections 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1632575433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00ce2a0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1632575433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00d37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- 172.21.15.119:0/1632575433 >> 172.21.15.119:0/1632575433 conn(0x7fd0b00d2bf0 msgr2=0x7fd0b00d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- 172.21.15.119:0/1632575433 shutdown_connections 2024-09-13T16:26:34.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.684+0000 7fd0b7710700 1 -- 172.21.15.119:0/1632575433 wait complete. 2024-09-13T16:26:34.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 Processor -- start 2024-09-13T16:26:34.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 -- start start 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00b79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0b01607c0 0x7fd0b00b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b015e600 con 0x7fd0b00d1620 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b015e8e0 con 0x7fd0b01607c0 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b7710700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b015ebc0 con 0x7fd0b015ce70 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00b79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00b79f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50296/0 (socket says 172.21.15.119:50296) 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 -- 172.21.15.119:0/1177787917 learned_addr learned my addr 172.21.15.119:0/1177787917 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b4cab700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00c87b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b5cad700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0b01607c0 0x7fd0b00b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 -- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 msgr2=0x7fd0b00c87b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00c87b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 -- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0b01607c0 msgr2=0x7fd0b00b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0b01607c0 0x7fd0b00b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 -- 172.21.15.119:0/1177787917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a8009b30 con 0x7fd0b00d1620 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0b54ac700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00b79f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000c770 tx=0x7fd0a000ca80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.688+0000 7fd0a5ffb700 1 -- 172.21.15.119:0/1177787917 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0011800 con 0x7fd0b00d1620 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0a5ffb700 1 -- 172.21.15.119:0/1177787917 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0011960 con 0x7fd0b00d1620 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b015eea0 con 0x7fd0b00d1620 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0a5ffb700 1 -- 172.21.15.119:0/1177787917 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a0011d10 con 0x7fd0b00d1620 2024-09-13T16:26:34.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b00b7f30 con 0x7fd0b00d1620 2024-09-13T16:26:34.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0a5ffb700 1 -- 172.21.15.119:0/1177787917 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd0a0018360 con 0x7fd0b00d1620 2024-09-13T16:26:34.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b0161030 con 0x7fd0b00d1620 2024-09-13T16:26:34.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0a5ffb700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd094066730 0x7fd094068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:34.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0a5ffb700 1 -- 172.21.15.119:0/1177787917 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd0a0083ea0 con 0x7fd0b00d1620 2024-09-13T16:26:34.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.692+0000 7fd0b4cab700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd094066730 0x7fd094068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:34.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.696+0000 7fd0b4cab700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd094066730 0x7fd094068bf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e80 tx=0x7fd0a8009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:34.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.700+0000 7fd0a5ffb700 1 -- 172.21.15.119:0/1177787917 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd0a0051410 con 0x7fd0b00d1620 2024-09-13T16:26:34.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.876+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd0b0072730 con 0x7fd0b00d1620 2024-09-13T16:26:34.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.876+0000 7fd0a5ffb700 1 -- 172.21.15.119:0/1177787917 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v36) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd0a0054a30 con 0x7fd0b00d1620 2024-09-13T16:26:34.883 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:34.883 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:26:34.883 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:26:34.883 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:26:34.883 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:26:34.883 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:34.883 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:26:34.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd094066730 msgr2=0x7fd094068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd094066730 0x7fd094068bf0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8009e80 tx=0x7fd0a8009000 comp rx=0 tx=0).stop 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 msgr2=0x7fd0b00b79f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00b79f0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd0a000c770 tx=0x7fd0a000ca80 comp rx=0 tx=0).stop 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 shutdown_connections 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd094066730 0x7fd094068bf0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0b01607c0 0x7fd0b00b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0b015ce70 0x7fd0b00c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 --2- 172.21.15.119:0/1177787917 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0b00d1620 0x7fd0b00b79f0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:34.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 >> 172.21.15.119:0/1177787917 conn(0x7fd0b00d2bf0 msgr2=0x7fd0b00b1aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 shutdown_connections 2024-09-13T16:26:34.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:34.880+0000 7fd0b7710700 1 -- 172.21.15.119:0/1177787917 wait complete. 2024-09-13T16:26:34.898 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:26:34.898 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:26:35.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.016+0000 7fa532315700 1 Processor -- start 2024-09-13T16:26:35.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.016+0000 7fa532315700 1 -- start start 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c15c900 0x7fa52c15cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa52c0d35a0 0x7fa52c0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa52c0d0650 con 0x7fa52c0b26e0 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa52c0d39e0 con 0x7fa52c0d35a0 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0ce5a0 con 0x7fa52c15c900 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa530b12700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa52c0d35a0 0x7fa52c0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa531b14700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa52c0b26e0 0x7fa52c0d10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45990/0 (socket says 172.21.15.119:45990) 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa531b14700 1 -- 172.21.15.119:0/3820487914 learned_addr learned my addr 172.21.15.119:0/3820487914 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:35.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa531313700 1 --2- 172.21.15.119:0/3820487914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c15c900 0x7fa52c15cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa530b12700 1 -- 172.21.15.119:0/3820487914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c15c900 msgr2=0x7fa52c15cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa530b12700 1 --2- 172.21.15.119:0/3820487914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c15c900 0x7fa52c15cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa530b12700 1 -- 172.21.15.119:0/3820487914 >> v1:172.21.15.119:6789/0 conn(0x7fa52c0b26e0 legacy=0x7fa52c0d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa530b12700 1 -- 172.21.15.119:0/3820487914 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa52c0ce880 con 0x7fa52c0d35a0 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa52bfff700 1 -- 172.21.15.119:0/3820487914 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235668188 0 0) 0x7fa52c0d0650 con 0x7fa52c0b26e0 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa531313700 1 --2- 172.21.15.119:0/3820487914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c15c900 0x7fa52c15cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa530b12700 1 --2- 172.21.15.119:0/3820487914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0d35a0 0x7fa52c0b21a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa520009850 tx=0x7fa520009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea0198aef59b71c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa52bfff700 1 -- 172.21.15.119:0/3820487914 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520011830 con 0x7fa52c0d35a0 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa52bfff700 1 -- 172.21.15.119:0/3820487914 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa520011990 con 0x7fa52c0d35a0 2024-09-13T16:26:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa52bfff700 1 -- 172.21.15.119:0/3820487914 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa520011d20 con 0x7fa52c0d35a0 2024-09-13T16:26:35.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- 172.21.15.119:0/3820487914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0d35a0 msgr2=0x7fa52c0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:35.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 --2- 172.21.15.119:0/3820487914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0d35a0 0x7fa52c0b21a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa520009850 tx=0x7fa520009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- 172.21.15.119:0/3820487914 shutdown_connections 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 --2- 172.21.15.119:0/3820487914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0d35a0 0x7fa52c0b21a0 secure :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fa520009850 tx=0x7fa520009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 --2- 172.21.15.119:0/3820487914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c15c900 0x7fa52c15cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- 172.21.15.119:0/3820487914 >> 172.21.15.119:0/3820487914 conn(0x7fa52c0d2a80 msgr2=0x7fa52c0d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- 172.21.15.119:0/3820487914 shutdown_connections 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.020+0000 7fa532315700 1 -- 172.21.15.119:0/3820487914 wait complete. 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 Processor -- start 2024-09-13T16:26:35.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 -- start start 2024-09-13T16:26:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c0b26e0 0x7fa52c157f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c15c900 0x7fa52c0ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0b7bb0 0x7fa52c0c72b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0c6580 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c1584b0 con 0x7fa52c15c900 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa532315700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa52c0cd020 con 0x7fa52c0b26e0 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0b7bb0 0x7fa52c0c72b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0b7bb0 0x7fa52c0c72b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50318/0 (socket says 172.21.15.119:50318) 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 -- 172.21.15.119:0/237329684 learned_addr learned my addr 172.21.15.119:0/237329684 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa530b12700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c15c900 0x7fa52c0ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531313700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c0b26e0 0x7fa52c157f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 -- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c0b26e0 msgr2=0x7fa52c157f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c0b26e0 0x7fa52c157f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 -- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c15c900 msgr2=0x7fa52c0ccae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c15c900 0x7fa52c0ccae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531b14700 1 -- 172.21.15.119:0/237329684 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5200094b0 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa530b12700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c15c900 0x7fa52c0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.024+0000 7fa531313700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c0b26e0 0x7fa52c157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa531b14700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0b7bb0 0x7fa52c0c72b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa52400c4b0 tx=0x7fa52400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa529ffb700 1 -- 172.21.15.119:0/237329684 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa524004ca0 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa529ffb700 1 -- 172.21.15.119:0/237329684 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa524004e20 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa52c0b8150 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa529ffb700 1 -- 172.21.15.119:0/237329684 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa524007310 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa52c0c6ab0 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa52c074560 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.028+0000 7fa529ffb700 1 -- 172.21.15.119:0/237329684 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa524007470 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.032+0000 7fa529ffb700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa518066730 0x7fa518068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:35.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.032+0000 7fa529ffb700 1 -- 172.21.15.119:0/237329684 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa524083a80 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.032+0000 7fa531313700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa518066730 0x7fa518068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:35.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.032+0000 7fa531313700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa518066730 0x7fa518068bf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa51c003fb0 tx=0x7fa51c009910 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:35.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.036+0000 7fa529ffb700 1 -- 172.21.15.119:0/237329684 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa524051150 con 0x7fa52c0b7bb0 2024-09-13T16:26:35.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:35.204+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa52c0b21a0 con 0x7fa52c0b7bb0 2024-09-13T16:26:36.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.028+0000 7fa529ffb700 1 -- 172.21.15.119:0/237329684 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v37) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa524007720 con 0x7fa52c0b7bb0 2024-09-13T16:26:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa518066730 msgr2=0x7fa518068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa518066730 0x7fa518068bf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa51c003fb0 tx=0x7fa51c009910 comp rx=0 tx=0).stop 2024-09-13T16:26:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0b7bb0 msgr2=0x7fa52c0c72b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0b7bb0 0x7fa52c0c72b0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fa52400c4b0 tx=0x7fa52400c7c0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 shutdown_connections 2024-09-13T16:26:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa518066730 0x7fa518068bf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa52c0b7bb0 0x7fa52c0c72b0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa52c15c900 0x7fa52c0ccae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 --2- 172.21.15.119:0/237329684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa52c0b26e0 0x7fa52c157f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 >> 172.21.15.119:0/237329684 conn(0x7fa52c0d2a80 msgr2=0x7fa52c0d2e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 shutdown_connections 2024-09-13T16:26:36.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.036+0000 7fa532315700 1 -- 172.21.15.119:0/237329684 wait complete. 2024-09-13T16:26:36.045 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:26:36.058 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:26:36.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.180+0000 7fa836454700 1 Processor -- start 2024-09-13T16:26:36.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.180+0000 7fa836454700 1 -- start start 2024-09-13T16:26:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa8300cff80 0x7fa830167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8300cef70 con 0x7fa8300b3100 2024-09-13T16:26:36.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa83015d1d0 con 0x7fa83015cdf0 2024-09-13T16:26:36.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8300cf1c0 con 0x7fa8300cff80 2024-09-13T16:26:36.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa834c51700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa8300b3100 0x7fa8300b3ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46004/0 (socket says 172.21.15.119:46004) 2024-09-13T16:26:36.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa834c51700 1 -- 172.21.15.119:0/1171563055 learned_addr learned my addr 172.21.15.119:0/1171563055 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:36.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa835c53700 1 --2- 172.21.15.119:0/1171563055 >> v2:172.21.15.145:3300/0 conn(0x7fa8300cff80 0x7fa830167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa827fff700 1 -- 172.21.15.119:0/1171563055 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 586829930 0 0) 0x7fa8300cef70 con 0x7fa8300b3100 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa827fff700 1 -- 172.21.15.119:0/1171563055 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa81c0035d0 con 0x7fa8300b3100 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa835c53700 1 -- 172.21.15.119:0/1171563055 >> v1:172.21.15.145:6789/0 conn(0x7fa83015cdf0 legacy=0x7fa8300b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa835c53700 1 -- 172.21.15.119:0/1171563055 >> v1:172.21.15.119:6789/0 conn(0x7fa8300b3100 legacy=0x7fa8300b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa835c53700 1 -- 172.21.15.119:0/1171563055 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8300cf4a0 con 0x7fa8300cff80 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa835c53700 1 --2- 172.21.15.119:0/1171563055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 0x7fa830167910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa828009d90 tx=0x7fa82800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=762109068d2cb0cb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa827fff700 1 -- 172.21.15.119:0/1171563055 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82800e070 con 0x7fa8300cff80 2024-09-13T16:26:36.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa827fff700 1 -- 172.21.15.119:0/1171563055 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82800bb20 con 0x7fa8300cff80 2024-09-13T16:26:36.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa827fff700 1 -- 172.21.15.119:0/1171563055 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828012430 con 0x7fa8300cff80 2024-09-13T16:26:36.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- 172.21.15.119:0/1171563055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 msgr2=0x7fa830167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:36.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 --2- 172.21.15.119:0/1171563055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 0x7fa830167910 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fa828009d90 tx=0x7fa82800b330 comp rx=0 tx=0).stop 2024-09-13T16:26:36.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- 172.21.15.119:0/1171563055 shutdown_connections 2024-09-13T16:26:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 --2- 172.21.15.119:0/1171563055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 0x7fa830167910 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- 172.21.15.119:0/1171563055 >> 172.21.15.119:0/1171563055 conn(0x7fa8300b25e0 msgr2=0x7fa8300b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- 172.21.15.119:0/1171563055 shutdown_connections 2024-09-13T16:26:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.184+0000 7fa836454700 1 -- 172.21.15.119:0/1171563055 wait complete. 2024-09-13T16:26:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 Processor -- start 2024-09-13T16:26:36.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 -- start start 2024-09-13T16:26:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8300b3100 0x7fa830151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:36.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 0x7fa8301f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83015cdf0 0x7fa830154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8301551e0 con 0x7fa83015cdf0 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8301f68c0 con 0x7fa8300cff80 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa830152550 con 0x7fa8300b3100 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835c53700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83015cdf0 0x7fa830154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835c53700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83015cdf0 0x7fa830154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50332/0 (socket says 172.21.15.119:50332) 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835c53700 1 -- 172.21.15.119:0/2338650959 learned_addr learned my addr 172.21.15.119:0/2338650959 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835452700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8300b3100 0x7fa830151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:36.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa834c51700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 0x7fa8301f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835452700 1 -- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 msgr2=0x7fa8301f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835452700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 0x7fa8301f5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835452700 1 -- 172.21.15.119:0/2338650959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83015cdf0 msgr2=0x7fa830154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835452700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83015cdf0 0x7fa830154ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835452700 1 -- 172.21.15.119:0/2338650959 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa828009a70 con 0x7fa8300b3100 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835c53700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83015cdf0 0x7fa830154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa835452700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8300b3100 0x7fa830151ba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c480 tx=0x7fa82c00c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa825ffb700 1 -- 172.21.15.119:0/2338650959 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0052d0 con 0x7fa8300b3100 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa825ffb700 1 -- 172.21.15.119:0/2338650959 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c005430 con 0x7fa8300b3100 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8301f6020 con 0x7fa8300b3100 2024-09-13T16:26:36.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa825ffb700 1 -- 172.21.15.119:0/2338650959 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c0055a0 con 0x7fa8300b3100 2024-09-13T16:26:36.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.188+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8301f65e0 con 0x7fa8300b3100 2024-09-13T16:26:36.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.192+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8300b38e0 con 0x7fa8300b3100 2024-09-13T16:26:36.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.192+0000 7fa825ffb700 1 -- 172.21.15.119:0/2338650959 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa82c0038e0 con 0x7fa8300b3100 2024-09-13T16:26:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.192+0000 7fa825ffb700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.192+0000 7fa825ffb700 1 -- 172.21.15.119:0/2338650959 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5547+0+0 (secure 0 0 0) 0x7fa82c0833c0 con 0x7fa8300b3100 2024-09-13T16:26:36.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.192+0000 7fa834c51700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:36.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.196+0000 7fa834c51700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa8301f58c0 tx=0x7fa820007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:36.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.204+0000 7fa825ffb700 1 -- 172.21.15.119:0/2338650959 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa82c0507c0 con 0x7fa8300b3100 2024-09-13T16:26:36.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:36.376+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa8300c92b0 con 0x7fa8300b3100 2024-09-13T16:26:37.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.040+0000 7fa825ffb700 1 -- 172.21.15.119:0/2338650959 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v38) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa82c053de0 con 0x7fa8300b3100 2024-09-13T16:26:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 msgr2=0x7fa814068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa8301f58c0 tx=0x7fa820007de0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8300b3100 msgr2=0x7fa830151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8300b3100 0x7fa830151ba0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c480 tx=0x7fa82c00c790 comp rx=0 tx=0).stop 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 shutdown_connections 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83015cdf0 0x7fa830154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8300cff80 0x7fa8301f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 --2- 172.21.15.119:0/2338650959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8300b3100 0x7fa830151ba0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 >> 172.21.15.119:0/2338650959 conn(0x7fa8300b25e0 msgr2=0x7fa83006ea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 shutdown_connections 2024-09-13T16:26:37.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.060+0000 7fa836454700 1 -- 172.21.15.119:0/2338650959 wait complete. 2024-09-13T16:26:37.067 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:26:37.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:26:37.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.184+0000 7f42bf4c4700 1 Processor -- start 2024-09-13T16:26:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.184+0000 7f42bf4c4700 1 -- start start 2024-09-13T16:26:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.184+0000 7f42bf4c4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.184+0000 7f42bf4c4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f42b80cd750 0x7f42b80cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.184+0000 7f42bf4c4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42b815af90 con 0x7f42b815abb0 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.184+0000 7f42bf4c4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42b80cdc00 con 0x7f42b80cd750 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.184+0000 7f42bf4c4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80cb6d0 con 0x7f42b80b31e0 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42be4c2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f42b815abb0 0x7f42b80cf530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46016/0 (socket says 172.21.15.119:46016) 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42be4c2700 1 -- 172.21.15.119:0/3426685298 learned_addr learned my addr 172.21.15.119:0/3426685298 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42becc3700 1 --2- 172.21.15.119:0/3426685298 >> v2:172.21.15.145:3300/0 conn(0x7f42b80cd750 0x7f42b80cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bdcc1700 1 --2- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bd4c0700 1 -- 172.21.15.119:0/3426685298 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1582005520 0 0) 0x7f42a8001110 con 0x7f42b815abb0 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bd4c0700 1 -- 172.21.15.119:0/3426685298 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42a40035d0 con 0x7f42b815abb0 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42becc3700 1 -- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 msgr2=0x7f42b80cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42becc3700 1 --2- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bd4c0700 1 -- 172.21.15.119:0/3426685298 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2103512170 0 0) 0x7f42a40035d0 con 0x7f42b815abb0 2024-09-13T16:26:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42becc3700 1 -- 172.21.15.119:0/3426685298 >> v1:172.21.15.119:6789/0 conn(0x7f42b815abb0 legacy=0x7f42b80cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42becc3700 1 -- 172.21.15.119:0/3426685298 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b80d0060 con 0x7f42b80cd750 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bdcc1700 1 --2- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42becc3700 1 --2- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b80cd750 0x7f42b80cbb60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009d90 tx=0x7f42b000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2104cccd80310112 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bd4c0700 1 -- 172.21.15.119:0/3426685298 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b000e070 con 0x7f42b80cd750 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bd4c0700 1 -- 172.21.15.119:0/3426685298 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b0003a40 con 0x7f42b80cd750 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bd4c0700 1 -- 172.21.15.119:0/3426685298 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42a8001800 con 0x7f42b80cd750 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 -- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b80cd750 msgr2=0x7f42b80cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b80cd750 0x7f42b80cbb60 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f42b0009d90 tx=0x7f42b000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 -- 172.21.15.119:0/3426685298 shutdown_connections 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b80cd750 0x7f42b80cbb60 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/3426685298 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 -- 172.21.15.119:0/3426685298 >> 172.21.15.119:0/3426685298 conn(0x7f42b80b25e0 msgr2=0x7f42b80b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:37.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 -- 172.21.15.119:0/3426685298 shutdown_connections 2024-09-13T16:26:37.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.188+0000 7f42bf4c4700 1 -- 172.21.15.119:0/3426685298 wait complete. 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 Processor -- start 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 -- start start 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42b80cd750 0x7f42b81f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b815abb0 0x7f42b80b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80caa10 con 0x7f42b80cd750 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80b6690 con 0x7f42b815abb0 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42b80b6080 con 0x7f42b80b31e0 2024-09-13T16:26:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42b80cd750 0x7f42b81f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42b80cd750 0x7f42b81f1a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50348/0 (socket says 172.21.15.119:50348) 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 -- 172.21.15.119:0/1313037806 learned_addr learned my addr 172.21.15.119:0/1313037806 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42be4c2700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42becc3700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b815abb0 0x7f42b80b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 -- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 msgr2=0x7f42b80b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 -- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b815abb0 msgr2=0x7f42b80b52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b815abb0 0x7f42b80b52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 -- 172.21.15.119:0/1313037806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b40094b0 con 0x7f42b80cd750 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bdcc1700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42b80cd750 0x7f42b81f1a70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f42b400ebb0 tx=0x7f42b400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42aeffd700 1 -- 172.21.15.119:0/1313037806 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400c910 con 0x7f42b80cd750 2024-09-13T16:26:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42aeffd700 1 -- 172.21.15.119:0/1313037806 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42b400ca70 con 0x7f42b80cd750 2024-09-13T16:26:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b0009a70 con 0x7f42b80cd750 2024-09-13T16:26:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42aeffd700 1 -- 172.21.15.119:0/1313037806 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42b400ce00 con 0x7f42b80cd750 2024-09-13T16:26:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.192+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b80b6320 con 0x7f42b80cd750 2024-09-13T16:26:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.196+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b80cd210 con 0x7f42b80cd750 2024-09-13T16:26:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.196+0000 7f42aeffd700 1 -- 172.21.15.119:0/1313037806 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f42b4013470 con 0x7f42b80cd750 2024-09-13T16:26:37.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.196+0000 7f42aeffd700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f429c066730 0x7f429c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.196+0000 7f42aeffd700 1 -- 172.21.15.119:0/1313037806 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f42b4084c00 con 0x7f42b80cd750 2024-09-13T16:26:37.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.196+0000 7f42be4c2700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f429c066730 0x7f429c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.196+0000 7f42be4c2700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f429c066730 0x7f429c068bf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f42b806eb80 tx=0x7f42a8001070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.204+0000 7f42aeffd700 1 -- 172.21.15.119:0/1313037806 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f42b4051e70 con 0x7f42b80cd750 2024-09-13T16:26:37.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.412+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f42b80b1eb0 con 0x7f42b80cd750 2024-09-13T16:26:37.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.420+0000 7f42aeffd700 1 -- 172.21.15.119:0/1313037806 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 5 and data pool 6 v13) v1 ==== 172+0+0 (secure 0 0 0) 0x7f42b40210c0 con 0x7f42b80cd750 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f429c066730 msgr2=0x7f429c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f429c066730 0x7f429c068bf0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f42b806eb80 tx=0x7f42a8001070 comp rx=0 tx=0).stop 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42b80cd750 msgr2=0x7f42b81f1a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42b80cd750 0x7f42b81f1a70 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f42b400ebb0 tx=0x7f42b400ef70 comp rx=0 tx=0).stop 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 shutdown_connections 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f429c066730 0x7f429c068bf0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42b815abb0 0x7f42b80b52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42b80cd750 0x7f42b81f1a70 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 --2- 172.21.15.119:0/1313037806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42b80b31e0 0x7f42b80b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 >> 172.21.15.119:0/1313037806 conn(0x7f42b80b25e0 msgr2=0x7f42b80c6410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:37.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.428+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 shutdown_connections 2024-09-13T16:26:37.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.432+0000 7f42bf4c4700 1 -- 172.21.15.119:0/1313037806 wait complete. 2024-09-13T16:26:37.445 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 5 and data pool 6 2024-09-13T16:26:37.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:37.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae8775700 1 Processor -- start 2024-09-13T16:26:37.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae8775700 1 -- start start 2024-09-13T16:26:37.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae8775700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9ae015aeb0 0x7f9ae00c6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae8775700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae00b5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae8775700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ae015b290 con 0x7f9ae00c67a0 2024-09-13T16:26:37.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae8775700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ae00b21b0 con 0x7f9ae015aeb0 2024-09-13T16:26:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae8775700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00c6c10 con 0x7f9ae00cfd10 2024-09-13T16:26:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae5d10700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9ae00c67a0 0x7f9ae00b5770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46026/0 (socket says 172.21.15.119:46026) 2024-09-13T16:26:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae5d10700 1 -- 172.21.15.119:0/3773665604 learned_addr learned my addr 172.21.15.119:0/3773665604 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae6511700 1 --2- 172.21.15.119:0/3773665604 >> v2:172.21.15.145:3300/0 conn(0x7f9ae015aeb0 0x7f9ae00c6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae6d12700 1 --2- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae00b5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae550f700 1 -- 172.21.15.119:0/3773665604 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508797942 0 0) 0x7f9ae015b290 con 0x7f9ae00c67a0 2024-09-13T16:26:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae550f700 1 -- 172.21.15.119:0/3773665604 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9acc0035d0 con 0x7f9ae00c67a0 2024-09-13T16:26:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae550f700 1 -- 172.21.15.119:0/3773665604 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 522992144 0 0) 0x7f9acc0035d0 con 0x7f9ae00c67a0 2024-09-13T16:26:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae6511700 1 -- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 msgr2=0x7f9ae00b5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae6511700 1 --2- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae00b5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae6511700 1 -- 172.21.15.119:0/3773665604 >> v1:172.21.15.119:6789/0 conn(0x7f9ae00c67a0 legacy=0x7f9ae00b5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae6511700 1 -- 172.21.15.119:0/3773665604 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ae00b46d0 con 0x7f9ae015aeb0 2024-09-13T16:26:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae6511700 1 --2- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 0x7f9ae00c6260 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009850 tx=0x7f9adc009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb3424e0cbf49941 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae550f700 1 -- 172.21.15.119:0/3773665604 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ad80019f0 con 0x7f9ae015aeb0 2024-09-13T16:26:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae550f700 1 -- 172.21.15.119:0/3773665604 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc004030 con 0x7f9ae015aeb0 2024-09-13T16:26:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.556+0000 7f9ae550f700 1 -- 172.21.15.119:0/3773665604 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc0043a0 con 0x7f9ae015aeb0 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 -- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 msgr2=0x7f9ae00c6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 0x7f9ae00c6260 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f9adc009850 tx=0x7f9adc009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 -- 172.21.15.119:0/3773665604 shutdown_connections 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae00b5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3773665604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 0x7f9ae00c6260 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 -- 172.21.15.119:0/3773665604 >> 172.21.15.119:0/3773665604 conn(0x7f9ae00b25e0 msgr2=0x7f9ae00b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 -- 172.21.15.119:0/3773665604 shutdown_connections 2024-09-13T16:26:37.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.560+0000 7f9ae8775700 1 -- 172.21.15.119:0/3773665604 wait complete. 2024-09-13T16:26:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 Processor -- start 2024-09-13T16:26:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 -- start start 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ae00c67a0 0x7f9ae0152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae01525d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 0x7f9ae01548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00cd200 con 0x7f9ae00c67a0 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00cc580 con 0x7f9ae015aeb0 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ae00cc860 con 0x7f9ae00cfd10 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ae00c67a0 0x7f9ae0152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ae00c67a0 0x7f9ae0152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50364/0 (socket says 172.21.15.119:50364) 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 -- 172.21.15.119:0/3220089035 learned_addr learned my addr 172.21.15.119:0/3220089035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae5d10700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae01525d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6d12700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 0x7f9ae01548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 -- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 msgr2=0x7f9ae01525d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae01525d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 -- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 msgr2=0x7f9ae01548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 0x7f9ae01548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 -- 172.21.15.119:0/3220089035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9adc0094b0 con 0x7f9ae00c67a0 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae5d10700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae01525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae6511700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ae00c67a0 0x7f9ae0152090 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0090b0 tx=0x7f9adc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ad2ffd700 1 -- 172.21.15.119:0/3220089035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc004590 con 0x7f9ae00c67a0 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ad2ffd700 1 -- 172.21.15.119:0/3220089035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9adc01f070 con 0x7f9ae00c67a0 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ae00ccb40 con 0x7f9ae00c67a0 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ad2ffd700 1 -- 172.21.15.119:0/3220089035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9adc011850 con 0x7f9ae00c67a0 2024-09-13T16:26:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.564+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ae00cc010 con 0x7f9ae00c67a0 2024-09-13T16:26:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.568+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ae00b23e0 con 0x7f9ae00c67a0 2024-09-13T16:26:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.568+0000 7f9ad2ffd700 1 -- 172.21.15.119:0/3220089035 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9adc0119b0 con 0x7f9ae00c67a0 2024-09-13T16:26:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.568+0000 7f9ad2ffd700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ac4066730 0x7f9ac4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.568+0000 7f9ad2ffd700 1 -- 172.21.15.119:0/3220089035 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9adc021070 con 0x7f9ae00c67a0 2024-09-13T16:26:37.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.568+0000 7f9ae5d10700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ac4066730 0x7f9ac4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.568+0000 7f9ae5d10700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ac4066730 0x7f9ac4068bf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00c5960 tx=0x7f9ad8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.576+0000 7f9ad2ffd700 1 -- 172.21.15.119:0/3220089035 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9adc052ee0 con 0x7f9ae00c67a0 2024-09-13T16:26:37.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.748+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ae0067760 con 0x7f9ae00c67a0 2024-09-13T16:26:37.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.752+0000 7f9ad2ffd700 1 -- 172.21.15.119:0/3220089035 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f9adc056500 con 0x7f9ae00c67a0 2024-09-13T16:26:37.758 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:37.758 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":39,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:37.415967+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:37.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ac4066730 msgr2=0x7f9ac4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ac4066730 0x7f9ac4068bf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ae00c5960 tx=0x7f9ad8009040 comp rx=0 tx=0).stop 2024-09-13T16:26:37.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ae00c67a0 msgr2=0x7f9ae0152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ae00c67a0 0x7f9ae0152090 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f9adc0090b0 tx=0x7f9adc004770 comp rx=0 tx=0).stop 2024-09-13T16:26:37.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 shutdown_connections 2024-09-13T16:26:37.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9ac4066730 0x7f9ac4068bf0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ae015aeb0 0x7f9ae01548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ae00cfd10 0x7f9ae01525d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 --2- 172.21.15.119:0/3220089035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ae00c67a0 0x7f9ae0152090 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 >> 172.21.15.119:0/3220089035 conn(0x7f9ae00b25e0 msgr2=0x7f9ae00b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:37.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 shutdown_connections 2024-09-13T16:26:37.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.756+0000 7f9ae8775700 1 -- 172.21.15.119:0/3220089035 wait complete. 2024-09-13T16:26:37.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:37.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f3153b2f700 1 Processor -- start 2024-09-13T16:26:37.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f3153b2f700 1 -- start start 2024-09-13T16:26:37.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f3153b2f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c1444b0 0x7f314c06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f3153b2f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f314c0cbb10 0x7f314c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f3153b2f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f314c06bd60 con 0x7f314c1440d0 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f3153b2f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f314c0cc4c0 con 0x7f314c0cbb10 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f3153b2f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c06bfe0 con 0x7f314c1444b0 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31518cb700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f314c1440d0 0x7f314c15b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46038/0 (socket says 172.21.15.119:46038) 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31518cb700 1 -- 172.21.15.119:0/134410654 learned_addr learned my addr 172.21.15.119:0/134410654 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31520cc700 1 --2- 172.21.15.119:0/134410654 >> v2:172.21.15.119:3300/0 conn(0x7f314c0cbb10 0x7f314c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31510ca700 1 --2- 172.21.15.119:0/134410654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c1444b0 0x7f314c06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31508c9700 1 -- 172.21.15.119:0/134410654 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4293648680 0 0) 0x7f314c06bd60 con 0x7f314c1440d0 2024-09-13T16:26:37.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31508c9700 1 -- 172.21.15.119:0/134410654 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31380035d0 con 0x7f314c1440d0 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31520cc700 1 -- 172.21.15.119:0/134410654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c1444b0 msgr2=0x7f314c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.876+0000 7f31520cc700 1 --2- 172.21.15.119:0/134410654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c1444b0 0x7f314c06d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f31520cc700 1 -- 172.21.15.119:0/134410654 >> v1:172.21.15.119:6789/0 conn(0x7f314c1440d0 legacy=0x7f314c15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f31520cc700 1 -- 172.21.15.119:0/134410654 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f314c06c2c0 con 0x7f314c0cbb10 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f31520cc700 1 --2- 172.21.15.119:0/134410654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c0cbb10 0x7f314c0cbf80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3148009850 tx=0x7f3148009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdde1b2a0a1d69bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f31508c9700 1 -- 172.21.15.119:0/134410654 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148011830 con 0x7f314c0cbb10 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f31508c9700 1 -- 172.21.15.119:0/134410654 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3148011990 con 0x7f314c0cbb10 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f31508c9700 1 -- 172.21.15.119:0/134410654 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3148011d00 con 0x7f314c0cbb10 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 -- 172.21.15.119:0/134410654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c0cbb10 msgr2=0x7f314c0cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 --2- 172.21.15.119:0/134410654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c0cbb10 0x7f314c0cbf80 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f3148009850 tx=0x7f3148009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 -- 172.21.15.119:0/134410654 shutdown_connections 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 --2- 172.21.15.119:0/134410654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c0cbb10 0x7f314c0cbf80 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 --2- 172.21.15.119:0/134410654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c1444b0 0x7f314c06d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 -- 172.21.15.119:0/134410654 >> 172.21.15.119:0/134410654 conn(0x7f314c06a350 msgr2=0x7f314c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 -- 172.21.15.119:0/134410654 shutdown_connections 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 -- 172.21.15.119:0/134410654 wait complete. 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 Processor -- start 2024-09-13T16:26:37.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.880+0000 7f3153b2f700 1 -- start start 2024-09-13T16:26:37.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c0cbb10 0x7f314c1f2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c1440d0 0x7f314c15a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f314c1444b0 0x7f314c0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c06fe60 con 0x7f314c1440d0 2024-09-13T16:26:37.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f314c0b11d0 con 0x7f314c1444b0 2024-09-13T16:26:37.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f314c1f2cd0 con 0x7f314c0cbb10 2024-09-13T16:26:37.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c1440d0 0x7f314c15a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c1440d0 0x7f314c15a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50384/0 (socket says 172.21.15.119:50384) 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 -- 172.21.15.119:0/1646067295 learned_addr learned my addr 172.21.15.119:0/1646067295 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31518cb700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c0cbb10 0x7f314c1f2790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31520cc700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f314c1444b0 0x7f314c0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 -- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c0cbb10 msgr2=0x7f314c1f2790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c0cbb10 0x7f314c1f2790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 -- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f314c1444b0 msgr2=0x7f314c0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f314c1444b0 0x7f314c0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 -- 172.21.15.119:0/1646067295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31480094b0 con 0x7f314c1440d0 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31510ca700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c1440d0 0x7f314c15a320 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f313c00e8a0 tx=0x7f313c00ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31427fc700 1 -- 172.21.15.119:0/1646067295 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c0095c0 con 0x7f314c1440d0 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31427fc700 1 -- 172.21.15.119:0/1646067295 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f313c013680 con 0x7f314c1440d0 2024-09-13T16:26:37.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f314c0aff10 con 0x7f314c1440d0 2024-09-13T16:26:37.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31427fc700 1 -- 172.21.15.119:0/1646067295 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f313c013a10 con 0x7f314c1440d0 2024-09-13T16:26:37.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f314c0700b0 con 0x7f314c1440d0 2024-09-13T16:26:37.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f314c067760 con 0x7f314c1440d0 2024-09-13T16:26:37.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.884+0000 7f31427fc700 1 -- 172.21.15.119:0/1646067295 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f313c013bd0 con 0x7f314c1440d0 2024-09-13T16:26:37.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.888+0000 7f31427fc700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3130066730 0x7f3130068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:37.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.888+0000 7f31427fc700 1 -- 172.21.15.119:0/1646067295 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f313c084260 con 0x7f314c1440d0 2024-09-13T16:26:37.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.888+0000 7f31518cb700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3130066730 0x7f3130068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:37.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.888+0000 7f31518cb700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3130066730 0x7f3130068bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3144005cb0 tx=0x7f3144009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:37.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:37.892+0000 7f31427fc700 1 -- 172.21.15.119:0/1646067295 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f313c051480 con 0x7f314c1440d0 2024-09-13T16:26:38.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.060+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f314c0cbf80 con 0x7f314c1440d0 2024-09-13T16:26:38.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.064+0000 7f31427fc700 1 -- 172.21.15.119:0/1646067295 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v39) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f313c054aa0 con 0x7f314c1440d0 2024-09-13T16:26:38.069 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:38.070 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":39,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:37.415967+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:38.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3130066730 msgr2=0x7f3130068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3130066730 0x7f3130068bf0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3144005cb0 tx=0x7f3144009040 comp rx=0 tx=0).stop 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c1440d0 msgr2=0x7f314c15a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c1440d0 0x7f314c15a320 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f313c00e8a0 tx=0x7f313c00ebb0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 shutdown_connections 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3130066730 0x7f3130068bf0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f314c1444b0 0x7f314c0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f314c1440d0 0x7f314c15a320 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 --2- 172.21.15.119:0/1646067295 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f314c0cbb10 0x7f314c1f2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 >> 172.21.15.119:0/1646067295 conn(0x7f314c06a350 msgr2=0x7f314c142240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 shutdown_connections 2024-09-13T16:26:38.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.068+0000 7f3153b2f700 1 -- 172.21.15.119:0/1646067295 wait complete. 2024-09-13T16:26:38.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:26:38.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.192+0000 7f341759e700 1 Processor -- start 2024-09-13T16:26:38.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.192+0000 7f341759e700 1 -- start start 2024-09-13T16:26:38.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f341815b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f34180b1d50 0x7f34180b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f341815cb70 con 0x7f34180b4c80 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f341815bf60 con 0x7f34180b1d50 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f341815c240 con 0x7f34180b4360 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f34180b1d50 0x7f34180b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f34180b1d50 0x7f34180b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50390/0 (socket says 172.21.15.119:50390) 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 -- 172.21.15.119:0/4061604110 learned_addr learned my addr 172.21.15.119:0/4061604110 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341659c700 1 --2- 172.21.15.119:0/4061604110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f341815b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 -- 172.21.15.119:0/4061604110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 msgr2=0x7f341815b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 --2- 172.21.15.119:0/4061604110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f341815b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 -- 172.21.15.119:0/4061604110 >> v1:172.21.15.145:6789/0 conn(0x7f34180b4c80 legacy=0x7f341815d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:38.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 -- 172.21.15.119:0/4061604110 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f341815c520 con 0x7f34180b1d50 2024-09-13T16:26:38.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341659c700 1 --2- 172.21.15.119:0/4061604110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f341815b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:38.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f3415d9b700 1 --2- 172.21.15.119:0/4061604110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b1d50 0x7f34180b4740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3408009910 tx=0x7f3408009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa6b8e4dcceb86e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:38.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341559a700 1 -- 172.21.15.119:0/4061604110 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408004230 con 0x7f34180b1d50 2024-09-13T16:26:38.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341559a700 1 -- 172.21.15.119:0/4061604110 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3408004390 con 0x7f34180b1d50 2024-09-13T16:26:38.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341559a700 1 -- 172.21.15.119:0/4061604110 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3408009e60 con 0x7f34180b1d50 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- 172.21.15.119:0/4061604110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b1d50 msgr2=0x7f34180b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 --2- 172.21.15.119:0/4061604110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b1d50 0x7f34180b4740 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f3408009910 tx=0x7f3408009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- 172.21.15.119:0/4061604110 shutdown_connections 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 --2- 172.21.15.119:0/4061604110 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b1d50 0x7f34180b4740 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 --2- 172.21.15.119:0/4061604110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f341815b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- 172.21.15.119:0/4061604110 >> 172.21.15.119:0/4061604110 conn(0x7f3418072240 msgr2=0x7f3418099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- 172.21.15.119:0/4061604110 shutdown_connections 2024-09-13T16:26:38.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- 172.21.15.119:0/4061604110 wait complete. 2024-09-13T16:26:38.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 Processor -- start 2024-09-13T16:26:38.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.196+0000 7f341759e700 1 -- start start 2024-09-13T16:26:38.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34180b1d50 0x7f3418147ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f3418144300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b4c80 0x7f3418144840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f341806b6e0 con 0x7f34180b4c80 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34181554c0 con 0x7f34180b1d50 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34181557a0 con 0x7f34180b4360 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b4c80 0x7f3418144840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3415d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f3418144300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b4c80 0x7f3418144840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50404/0 (socket says 172.21.15.119:50404) 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 -- 172.21.15.119:0/1929310501 learned_addr learned my addr 172.21.15.119:0/1929310501 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341659c700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34180b1d50 0x7f3418147ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 -- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 msgr2=0x7f3418144300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:38.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f3418144300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 -- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34180b1d50 msgr2=0x7f3418147ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34180b1d50 0x7f3418147ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 -- 172.21.15.119:0/1929310501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3408009570 con 0x7f34180b4c80 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3416d9d700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b4c80 0x7f3418144840 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f341000b970 tx=0x7f341000bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3406ffd700 1 -- 172.21.15.119:0/1929310501 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341000d860 con 0x7f34180b4c80 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3406ffd700 1 -- 172.21.15.119:0/1929310501 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f341000d9c0 con 0x7f34180b4c80 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f3406ffd700 1 -- 172.21.15.119:0/1929310501 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f341000dd70 con 0x7f34180b4c80 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3418155a80 con 0x7f34180b4c80 2024-09-13T16:26:38.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34181481d0 con 0x7f34180b4c80 2024-09-13T16:26:38.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.200+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34180d18a0 con 0x7f34180b4c80 2024-09-13T16:26:38.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.204+0000 7f3406ffd700 1 -- 172.21.15.119:0/1929310501 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f341000e430 con 0x7f34180b4c80 2024-09-13T16:26:38.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.204+0000 7f3406ffd700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33f8066730 0x7f33f8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:38.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.204+0000 7f3406ffd700 1 -- 172.21.15.119:0/1929310501 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f34100842b0 con 0x7f34180b4c80 2024-09-13T16:26:38.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.204+0000 7f341659c700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33f8066730 0x7f33f8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:38.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.204+0000 7f341659c700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33f8066730 0x7f33f8068bf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f340c006fd0 tx=0x7f340c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:38.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.212+0000 7f3406ffd700 1 -- 172.21.15.119:0/1929310501 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3410051570 con 0x7f34180b4c80 2024-09-13T16:26:38.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:38.424+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f341806b820 con 0x7f34180b4c80 2024-09-13T16:26:39.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.432+0000 7f3406ffd700 1 -- 172.21.15.119:0/1929310501 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v15) v1 ==== 118+0+0 (secure 0 0 0) 0x7f341806b820 con 0x7f34180b4c80 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33f8066730 msgr2=0x7f33f8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33f8066730 0x7f33f8068bf0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f340c006fd0 tx=0x7f340c008040 comp rx=0 tx=0).stop 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b4c80 msgr2=0x7f3418144840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b4c80 0x7f3418144840 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f341000b970 tx=0x7f341000bc80 comp rx=0 tx=0).stop 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 shutdown_connections 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33f8066730 0x7f33f8068bf0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34180b4c80 0x7f3418144840 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34180b4360 0x7f3418144300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 --2- 172.21.15.119:0/1929310501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34180b1d50 0x7f3418147ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 >> 172.21.15.119:0/1929310501 conn(0x7f3418072240 msgr2=0x7f3418099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:39.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 shutdown_connections 2024-09-13T16:26:39.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.436+0000 7f341759e700 1 -- 172.21.15.119:0/1929310501 wait complete. 2024-09-13T16:26:39.456 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T16:26:39.456 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:26:39.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e6cd97700 1 Processor -- start 2024-09-13T16:26:39.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e6cd97700 1 -- start start 2024-09-13T16:26:39.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e6cd97700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681599c0 0x7f9e6806b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:39.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e6cd97700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9e6806b8a0 0x7f9e6806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:39.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e6cd97700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e6806bd00 con 0x7f9e681595e0 2024-09-13T16:26:39.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e6cd97700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e6806c3f0 con 0x7f9e6806b8a0 2024-09-13T16:26:39.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e6cd97700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6806ee20 con 0x7f9e681599c0 2024-09-13T16:26:39.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e677fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9e681595e0 0x7f9e6806e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46052/0 (socket says 172.21.15.119:46052) 2024-09-13T16:26:39.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.580+0000 7f9e677fe700 1 -- 172.21.15.119:0/2923667343 learned_addr learned my addr 172.21.15.119:0/2923667343 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:39.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e67fff700 1 --2- 172.21.15.119:0/2923667343 >> v2:172.21.15.145:3300/0 conn(0x7f9e6806b8a0 0x7f9e6806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:39.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e5ffff700 1 --2- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681599c0 0x7f9e6806b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:39.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e66ffd700 1 -- 172.21.15.119:0/2923667343 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640984629 0 0) 0x7f9e6806bd00 con 0x7f9e681595e0 2024-09-13T16:26:39.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e66ffd700 1 -- 172.21.15.119:0/2923667343 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e540035d0 con 0x7f9e681595e0 2024-09-13T16:26:39.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e66ffd700 1 -- 172.21.15.119:0/2923667343 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 161226250 0 0) 0x7f9e540035d0 con 0x7f9e681595e0 2024-09-13T16:26:39.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e67fff700 1 -- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681599c0 msgr2=0x7f9e6806b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e67fff700 1 --2- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681599c0 0x7f9e6806b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e67fff700 1 -- 172.21.15.119:0/2923667343 >> v1:172.21.15.119:6789/0 conn(0x7f9e681595e0 legacy=0x7f9e6806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e67fff700 1 -- 172.21.15.119:0/2923667343 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e680710b0 con 0x7f9e6806b8a0 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e67fff700 1 --2- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 0x7f9e6806c810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e6000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2036b82951ae4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e66ffd700 1 -- 172.21.15.119:0/2923667343 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e6000e070 con 0x7f9e6806b8a0 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e66ffd700 1 -- 172.21.15.119:0/2923667343 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60003a40 con 0x7f9e6806b8a0 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e66ffd700 1 -- 172.21.15.119:0/2923667343 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e58001670 con 0x7f9e6806b8a0 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 -- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 msgr2=0x7f9e6806c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 0x7f9e6806c810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f9e60009d90 tx=0x7f9e6000b330 comp rx=0 tx=0).stop 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 -- 172.21.15.119:0/2923667343 shutdown_connections 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 0x7f9e6806c810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/2923667343 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681599c0 0x7f9e6806b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 -- 172.21.15.119:0/2923667343 >> 172.21.15.119:0/2923667343 conn(0x7f9e680cb8f0 msgr2=0x7f9e680cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:39.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 -- 172.21.15.119:0/2923667343 shutdown_connections 2024-09-13T16:26:39.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.584+0000 7f9e6cd97700 1 -- 172.21.15.119:0/2923667343 wait complete. 2024-09-13T16:26:39.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 Processor -- start 2024-09-13T16:26:39.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 -- start start 2024-09-13T16:26:39.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 0x7f9e681f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:39.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681595e0 0x7f9e68150870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:39.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e681599c0 0x7f9e6806e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:39.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e681f6920 con 0x7f9e681599c0 2024-09-13T16:26:39.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6815ad50 con 0x7f9e6806b8a0 2024-09-13T16:26:39.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e68150db0 con 0x7f9e681595e0 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e681599c0 0x7f9e6806e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e681599c0 0x7f9e6806e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50410/0 (socket says 172.21.15.119:50410) 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 -- 172.21.15.119:0/306142064 learned_addr learned my addr 172.21.15.119:0/306142064 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e5ffff700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681595e0 0x7f9e68150870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e677fe700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 0x7f9e681f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 -- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681595e0 msgr2=0x7f9e68150870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681595e0 0x7f9e68150870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 -- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 msgr2=0x7f9e681f63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 0x7f9e681f63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 -- 172.21.15.119:0/306142064 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e60009a70 con 0x7f9e681599c0 2024-09-13T16:26:39.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e67fff700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e681599c0 0x7f9e6806e1e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9e680cfe60 tx=0x7f9e60004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:39.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e64ff9700 1 -- 172.21.15.119:0/306142064 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e600044c0 con 0x7f9e681599c0 2024-09-13T16:26:39.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e64ff9700 1 -- 172.21.15.119:0/306142064 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e60004640 con 0x7f9e681599c0 2024-09-13T16:26:39.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e6815a140 con 0x7f9e681599c0 2024-09-13T16:26:39.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e64ff9700 1 -- 172.21.15.119:0/306142064 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e60012430 con 0x7f9e681599c0 2024-09-13T16:26:39.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.588+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e6815a700 con 0x7f9e681599c0 2024-09-13T16:26:39.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.592+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e6806b360 con 0x7f9e681599c0 2024-09-13T16:26:39.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.592+0000 7f9e64ff9700 1 -- 172.21.15.119:0/306142064 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9e60012590 con 0x7f9e681599c0 2024-09-13T16:26:39.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.592+0000 7f9e64ff9700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e48066730 0x7f9e48068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:39.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.592+0000 7f9e64ff9700 1 -- 172.21.15.119:0/306142064 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9e600864f0 con 0x7f9e681599c0 2024-09-13T16:26:39.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.592+0000 7f9e677fe700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e48066730 0x7f9e48068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:39.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.592+0000 7f9e677fe700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e48066730 0x7f9e48068bf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9e58000f60 tx=0x7f9e58001230 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:39.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.600+0000 7f9e64ff9700 1 -- 172.21.15.119:0/306142064 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9e600537b0 con 0x7f9e681599c0 2024-09-13T16:26:39.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:39.808+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9e680cb0c0 con 0x7f9e681599c0 2024-09-13T16:26:40.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.440+0000 7f9e64ff9700 1 -- 172.21.15.119:0/306142064 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9e60056dd0 con 0x7f9e681599c0 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e48066730 msgr2=0x7f9e48068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e48066730 0x7f9e48068bf0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f9e58000f60 tx=0x7f9e58001230 comp rx=0 tx=0).stop 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e681599c0 msgr2=0x7f9e6806e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e681599c0 0x7f9e6806e1e0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f9e680cfe60 tx=0x7f9e60004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 shutdown_connections 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e48066730 0x7f9e48068bf0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e681599c0 0x7f9e6806e1e0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e681595e0 0x7f9e68150870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 --2- 172.21.15.119:0/306142064 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e6806b8a0 0x7f9e681f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 >> 172.21.15.119:0/306142064 conn(0x7f9e680cb8f0 msgr2=0x7f9e68158890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:40.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 shutdown_connections 2024-09-13T16:26:40.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.444+0000 7f9e6cd97700 1 -- 172.21.15.119:0/306142064 wait complete. 2024-09-13T16:26:40.463 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:26:40.464 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:40.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f970e700 1 Processor -- start 2024-09-13T16:26:40.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f970e700 1 -- start start 2024-09-13T16:26:40.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f970e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc3f41594f0 0x7fc3f406d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f970e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f970e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f40b2280 con 0x7fc3f406da70 2024-09-13T16:26:40.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f970e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f406dee0 con 0x7fc3f41594f0 2024-09-13T16:26:40.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f970e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f40cc890 con 0x7fc3f40b3800 2024-09-13T16:26:40.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc3f41594f0 0x7fc3f406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc3f41594f0 0x7fc3f406d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50424/0 (socket says 172.21.15.119:50424) 2024-09-13T16:26:40.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3025816853 learned_addr learned my addr 172.21.15.119:0/3025816853 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:40.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f8f0d700 1 --2- 172.21.15.119:0/3025816853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3025816853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 msgr2=0x7fc3f406fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:40.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 --2- 172.21.15.119:0/3025816853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f406fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3025816853 >> v1:172.21.15.145:6789/0 conn(0x7fc3f406da70 legacy=0x7fc3f40b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3025816853 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f40b1970 con 0x7fc3f41594f0 2024-09-13T16:26:40.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.576+0000 7fc3f3fff700 1 --2- 172.21.15.119:0/3025816853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f41594f0 0x7fc3f406d530 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40b1b60 tx=0x7fc3e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ce57b92e8bae5b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:40.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f2ffd700 1 -- 172.21.15.119:0/3025816853 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e400e070 con 0x7fc3f41594f0 2024-09-13T16:26:40.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f2ffd700 1 -- 172.21.15.119:0/3025816853 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4003a40 con 0x7fc3f41594f0 2024-09-13T16:26:40.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f2ffd700 1 -- 172.21.15.119:0/3025816853 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4003df0 con 0x7fc3f41594f0 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 -- 172.21.15.119:0/3025816853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f41594f0 msgr2=0x7fc3f406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3025816853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f41594f0 0x7fc3f406d530 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40b1b60 tx=0x7fc3e400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 -- 172.21.15.119:0/3025816853 shutdown_connections 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3025816853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f406fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3025816853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f41594f0 0x7fc3f406d530 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 -- 172.21.15.119:0/3025816853 >> 172.21.15.119:0/3025816853 conn(0x7fc3f406a350 msgr2=0x7fc3f406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 -- 172.21.15.119:0/3025816853 shutdown_connections 2024-09-13T16:26:40.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 -- 172.21.15.119:0/3025816853 wait complete. 2024-09-13T16:26:40.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 Processor -- start 2024-09-13T16:26:40.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.580+0000 7fc3f970e700 1 -- start start 2024-09-13T16:26:40.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f406da70 0x7fc3f41f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f4140fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3f41594f0 0x7fc3f4150620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4153210 con 0x7fc3f406da70 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f4141500 con 0x7fc3f41594f0 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f414e990 con 0x7fc3f40b3800 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f406da70 0x7fc3f41f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f406da70 0x7fc3f41f04b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50426/0 (socket says 172.21.15.119:50426) 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3870985197 learned_addr learned my addr 172.21.15.119:0/3870985197 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f37fe700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f4140fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f8f0d700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3f41594f0 0x7fc3f4150620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 msgr2=0x7fc3f4140fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f4140fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3f41594f0 msgr2=0x7fc3f4150620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3f41594f0 0x7fc3f4150620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 -- 172.21.15.119:0/3870985197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e4009b30 con 0x7fc3f406da70 2024-09-13T16:26:40.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f3fff700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f406da70 0x7fc3f41f04b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4015040 tx=0x7fc3e4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:40.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f0ff9700 1 -- 172.21.15.119:0/3870985197 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4004440 con 0x7fc3f406da70 2024-09-13T16:26:40.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f0ff9700 1 -- 172.21.15.119:0/3870985197 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e40092e0 con 0x7fc3f406da70 2024-09-13T16:26:40.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f414db00 con 0x7fc3f406da70 2024-09-13T16:26:40.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f0ff9700 1 -- 172.21.15.119:0/3870985197 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e40047a0 con 0x7fc3f406da70 2024-09-13T16:26:40.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.584+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f414e060 con 0x7fc3f406da70 2024-09-13T16:26:40.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.588+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f406d530 con 0x7fc3f406da70 2024-09-13T16:26:40.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.588+0000 7fc3f0ff9700 1 -- 172.21.15.119:0/3870985197 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc3e4012430 con 0x7fc3f406da70 2024-09-13T16:26:40.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.588+0000 7fc3f0ff9700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc3e0066760 0x7fc3e0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.588+0000 7fc3f0ff9700 1 -- 172.21.15.119:0/3870985197 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc3e4085640 con 0x7fc3f406da70 2024-09-13T16:26:40.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.588+0000 7fc3f37fe700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc3e0066760 0x7fc3e0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.588+0000 7fc3f37fe700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc3e0066760 0x7fc3e0068c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4158e00 tx=0x7fc3e8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:40.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.596+0000 7fc3f0ff9700 1 -- 172.21.15.119:0/3870985197 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc3e40562f0 con 0x7fc3f406da70 2024-09-13T16:26:40.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.800+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3f415ca00 con 0x7fc3f406da70 2024-09-13T16:26:40.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f0ff9700 1 -- 172.21.15.119:0/3870985197 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7fc3e4055e80 con 0x7fc3f406da70 2024-09-13T16:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:40.807 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:26:40.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc3e0066760 msgr2=0x7fc3e0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc3e0066760 0x7fc3e0068c20 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fc3f4158e00 tx=0x7fc3e8009040 comp rx=0 tx=0).stop 2024-09-13T16:26:40.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f406da70 msgr2=0x7fc3f41f04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f406da70 0x7fc3f41f04b0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4015040 tx=0x7fc3e4004af0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 shutdown_connections 2024-09-13T16:26:40.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc3e0066760 0x7fc3e0068c20 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3f41594f0 0x7fc3f4150620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3f40b3800 0x7fc3f4140fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 --2- 172.21.15.119:0/3870985197 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3f406da70 0x7fc3f41f04b0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 >> 172.21.15.119:0/3870985197 conn(0x7fc3f406a350 msgr2=0x7fc3f415bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 shutdown_connections 2024-09-13T16:26:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.804+0000 7fc3f970e700 1 -- 172.21.15.119:0/3870985197 wait complete. 2024-09-13T16:26:40.811 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:26:40.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:40.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51ccead700 1 Processor -- start 2024-09-13T16:26:40.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51ccead700 1 -- start start 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51ccead700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f51c8146230 0x7f51c806bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51ccead700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c80ac8c0 0x7f51c80accf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51ccead700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51c806cd80 con 0x7f51c80b2a30 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51ccead700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51c80b2ea0 con 0x7f51c8146230 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51ccead700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80ad230 con 0x7f51c80ac8c0 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c6ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f51c80b2a30 0x7f51c815cec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46058/0 (socket says 172.21.15.119:46058) 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c6ffd700 1 -- 172.21.15.119:0/2323777751 learned_addr learned my addr 172.21.15.119:0/2323777751 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c7fff700 1 --2- 172.21.15.119:0/2323777751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c80ac8c0 0x7f51c80accf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c77fe700 1 --2- 172.21.15.119:0/2323777751 >> v2:172.21.15.145:3300/0 conn(0x7f51c8146230 0x7f51c806bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c67fc700 1 -- 172.21.15.119:0/2323777751 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863202488 0 0) 0x7f51c806cd80 con 0x7f51c80b2a30 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c67fc700 1 -- 172.21.15.119:0/2323777751 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51b40035d0 con 0x7f51c80b2a30 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c7fff700 1 -- 172.21.15.119:0/2323777751 >> v2:172.21.15.145:3300/0 conn(0x7f51c8146230 msgr2=0x7f51c806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c7fff700 1 --2- 172.21.15.119:0/2323777751 >> v2:172.21.15.145:3300/0 conn(0x7f51c8146230 0x7f51c806bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c7fff700 1 -- 172.21.15.119:0/2323777751 >> v1:172.21.15.119:6789/0 conn(0x7f51c80b2a30 legacy=0x7f51c815cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.928+0000 7f51c7fff700 1 -- 172.21.15.119:0/2323777751 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c806c170 con 0x7f51c80ac8c0 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51c7fff700 1 --2- 172.21.15.119:0/2323777751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c80ac8c0 0x7f51c80accf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009850 tx=0x7f51c0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcc51fd2f0480168 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51c67fc700 1 -- 172.21.15.119:0/2323777751 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0011830 con 0x7f51c80ac8c0 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51c67fc700 1 -- 172.21.15.119:0/2323777751 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51c0011990 con 0x7f51c80ac8c0 2024-09-13T16:26:40.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51c67fc700 1 -- 172.21.15.119:0/2323777751 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51c0011d00 con 0x7f51c80ac8c0 2024-09-13T16:26:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 -- 172.21.15.119:0/2323777751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c80ac8c0 msgr2=0x7f51c80accf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 --2- 172.21.15.119:0/2323777751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c80ac8c0 0x7f51c80accf0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f51c0009850 tx=0x7f51c0009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 -- 172.21.15.119:0/2323777751 shutdown_connections 2024-09-13T16:26:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 --2- 172.21.15.119:0/2323777751 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c80ac8c0 0x7f51c80accf0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 --2- 172.21.15.119:0/2323777751 >> v2:172.21.15.145:3300/0 conn(0x7f51c8146230 0x7f51c806bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 -- 172.21.15.119:0/2323777751 >> 172.21.15.119:0/2323777751 conn(0x7f51c806a350 msgr2=0x7f51c806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 -- 172.21.15.119:0/2323777751 shutdown_connections 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 -- 172.21.15.119:0/2323777751 wait complete. 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 Processor -- start 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.932+0000 7f51ccead700 1 -- start start 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51c80ac8c0 0x7f51c81f2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c80b2a30 0x7f51c8070530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c8146230 0x7f51c813cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c81f3480 con 0x7f51c80b2a30 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51c81f3240 con 0x7f51c80ac8c0 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51c80c3ce0 con 0x7f51c8146230 2024-09-13T16:26:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c80b2a30 0x7f51c8070530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c80b2a30 0x7f51c8070530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50432/0 (socket says 172.21.15.119:50432) 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 -- 172.21.15.119:0/2119971660 learned_addr learned my addr 172.21.15.119:0/2119971660 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c77fe700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51c80ac8c0 0x7f51c81f2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c7fff700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c8146230 0x7f51c813cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 -- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c8146230 msgr2=0x7f51c813cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c8146230 0x7f51c813cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 -- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51c80ac8c0 msgr2=0x7f51c81f2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51c80ac8c0 0x7f51c81f2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 -- 172.21.15.119:0/2119971660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51c00094b0 con 0x7f51c80b2a30 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c77fe700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51c80ac8c0 0x7f51c81f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51c6ffd700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c80b2a30 0x7f51c8070530 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f51b000c4f0 tx=0x7f51b000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51bffff700 1 -- 172.21.15.119:0/2119971660 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b0005d90 con 0x7f51c80b2a30 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51bffff700 1 -- 172.21.15.119:0/2119971660 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51b00070d0 con 0x7f51c80b2a30 2024-09-13T16:26:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51c81f3740 con 0x7f51c80b2a30 2024-09-13T16:26:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51bffff700 1 -- 172.21.15.119:0/2119971660 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51b0007440 con 0x7f51c80b2a30 2024-09-13T16:26:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51c81f3d00 con 0x7f51c80b2a30 2024-09-13T16:26:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51c81f3e90 con 0x7f51c80b2a30 2024-09-13T16:26:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.936+0000 7f51bffff700 1 -- 172.21.15.119:0/2119971660 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f51b00075a0 con 0x7f51c80b2a30 2024-09-13T16:26:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.940+0000 7f51bffff700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f51a8066760 0x7f51a8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.940+0000 7f51bffff700 1 -- 172.21.15.119:0/2119971660 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f51b0083600 con 0x7f51c80b2a30 2024-09-13T16:26:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.940+0000 7f51c77fe700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f51a8066760 0x7f51a8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:40.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.940+0000 7f51c77fe700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f51a8066760 0x7f51a8068c20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009fd0 tx=0x7f51b800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:40.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:40.944+0000 7f51bffff700 1 -- 172.21.15.119:0/2119971660 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f51b0050970 con 0x7f51c80b2a30 2024-09-13T16:26:41.007 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-09-13T16:26:41.003+0000 7fcc3db96700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:26:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.116+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51c80accf0 con 0x7f51c80b2a30 2024-09-13T16:26:41.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.116+0000 7f51bffff700 1 -- 172.21.15.119:0/2119971660 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16834 (secure 0 0 0) 0x7f51b0053f90 con 0x7f51c80b2a30 2024-09-13T16:26:41.123 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:41.124 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":40,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:38.420314+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:41.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f51a8066760 msgr2=0x7f51a8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f51a8066760 0x7f51a8068c20 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009fd0 tx=0x7f51b800b040 comp rx=0 tx=0).stop 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c80b2a30 msgr2=0x7f51c8070530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c80b2a30 0x7f51c8070530 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f51b000c4f0 tx=0x7f51b000c8b0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 shutdown_connections 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f51a8066760 0x7f51a8068c20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51c8146230 0x7f51c813cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f51c80b2a30 0x7f51c8070530 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 --2- 172.21.15.119:0/2119971660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f51c80ac8c0 0x7f51c81f2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 >> 172.21.15.119:0/2119971660 conn(0x7f51c806a350 msgr2=0x7f51c815f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 shutdown_connections 2024-09-13T16:26:41.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.120+0000 7f51ccead700 1 -- 172.21.15.119:0/2119971660 wait complete. 2024-09-13T16:26:41.139 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:26:41.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:26:41.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff354c7a700 1 Processor -- start 2024-09-13T16:26:41.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff354c7a700 1 -- start start 2024-09-13T16:26:41.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff354c7a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff35008c6b0 0x7ff3500ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff354c7a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3500cd860 0x7ff350166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff354c7a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff35015d020 con 0x7ff35015cc40 2024-09-13T16:26:41.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff354c7a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3500b44c0 con 0x7ff35008c6b0 2024-09-13T16:26:41.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff354c7a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500d0720 con 0x7ff3500cd860 2024-09-13T16:26:41.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff35008c6b0 0x7ff3500ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff35015cc40 0x7ff35008bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46072/0 (socket says 172.21.15.119:46072) 2024-09-13T16:26:41.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34f7fe700 1 -- 172.21.15.119:0/1893175067 learned_addr learned my addr 172.21.15.119:0/1893175067 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:41.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34ffff700 1 --2- 172.21.15.119:0/1893175067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3500cd860 0x7ff350166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34effd700 1 -- 172.21.15.119:0/1893175067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3500cd860 msgr2=0x7ff350166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34e7fc700 1 -- 172.21.15.119:0/1893175067 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652243137 0 0) 0x7ff35015d020 con 0x7ff35015cc40 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34effd700 1 --2- 172.21.15.119:0/1893175067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3500cd860 0x7ff350166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34effd700 1 -- 172.21.15.119:0/1893175067 >> v1:172.21.15.119:6789/0 conn(0x7ff35015cc40 legacy=0x7ff35008bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34effd700 1 -- 172.21.15.119:0/1893175067 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3500d09c0 con 0x7ff35008c6b0 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.244+0000 7ff34effd700 1 --2- 172.21.15.119:0/1893175067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff35008c6b0 0x7ff3500ce710 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff338009880 tx=0x7ff338009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e235268a962a274 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff34e7fc700 1 -- 172.21.15.119:0/1893175067 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338011830 con 0x7ff35008c6b0 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff34e7fc700 1 -- 172.21.15.119:0/1893175067 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff338011990 con 0x7ff35008c6b0 2024-09-13T16:26:41.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff34e7fc700 1 -- 172.21.15.119:0/1893175067 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338011d20 con 0x7ff35008c6b0 2024-09-13T16:26:41.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 -- 172.21.15.119:0/1893175067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff35008c6b0 msgr2=0x7ff3500ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 --2- 172.21.15.119:0/1893175067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff35008c6b0 0x7ff3500ce710 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7ff338009880 tx=0x7ff338009b90 comp rx=0 tx=0).stop 2024-09-13T16:26:41.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 -- 172.21.15.119:0/1893175067 shutdown_connections 2024-09-13T16:26:41.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 --2- 172.21.15.119:0/1893175067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3500cd860 0x7ff350166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 --2- 172.21.15.119:0/1893175067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff35008c6b0 0x7ff3500ce710 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 -- 172.21.15.119:0/1893175067 >> 172.21.15.119:0/1893175067 conn(0x7ff3500b4640 msgr2=0x7ff3500b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:41.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 -- 172.21.15.119:0/1893175067 shutdown_connections 2024-09-13T16:26:41.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.248+0000 7ff354c7a700 1 -- 172.21.15.119:0/1893175067 wait complete. 2024-09-13T16:26:41.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 Processor -- start 2024-09-13T16:26:41.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 -- start start 2024-09-13T16:26:41.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff35008c6b0 0x7ff3501f5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3500cd860 0x7ff3500b5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff35015cc40 0x7ff3500b64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500b6d80 con 0x7ff3500cd860 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500b7060 con 0x7ff35015cc40 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500b7340 con 0x7ff35008c6b0 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3500cd860 0x7ff3500b5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3500cd860 0x7ff3500b5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50444/0 (socket says 172.21.15.119:50444) 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 -- 172.21.15.119:0/4114762056 learned_addr learned my addr 172.21.15.119:0/4114762056 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34f7fe700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff35008c6b0 0x7ff3501f5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34ffff700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff35015cc40 0x7ff3500b64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 -- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff35008c6b0 msgr2=0x7ff3501f5ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff35008c6b0 0x7ff3501f5ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 -- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff35015cc40 msgr2=0x7ff3500b64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff35015cc40 0x7ff3500b64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 -- 172.21.15.119:0/4114762056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3380094b0 con 0x7ff3500cd860 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff34effd700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3500cd860 0x7ff3500b5fb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff338009850 tx=0x7ff338004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff347fff700 1 -- 172.21.15.119:0/4114762056 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3380036c0 con 0x7ff3500cd860 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3500b7620 con 0x7ff3500cd860 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff347fff700 1 -- 172.21.15.119:0/4114762056 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33801e070 con 0x7ff3500cd860 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff347fff700 1 -- 172.21.15.119:0/4114762056 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338003a60 con 0x7ff3500cd860 2024-09-13T16:26:41.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.252+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3501f60e0 con 0x7ff3500cd860 2024-09-13T16:26:41.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.256+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3500d1a60 con 0x7ff3500cd860 2024-09-13T16:26:41.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.256+0000 7ff347fff700 1 -- 172.21.15.119:0/4114762056 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff338003bc0 con 0x7ff3500cd860 2024-09-13T16:26:41.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.256+0000 7ff347fff700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff334066730 0x7ff334068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.256+0000 7ff347fff700 1 -- 172.21.15.119:0/4114762056 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff3380851c0 con 0x7ff3500cd860 2024-09-13T16:26:41.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.256+0000 7ff34f7fe700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff334066730 0x7ff334068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.256+0000 7ff34f7fe700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff334066730 0x7ff334068bf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff340000f60 tx=0x7ff340007e00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.264+0000 7ff347fff700 1 -- 172.21.15.119:0/4114762056 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff338052360 con 0x7ff3500cd860 2024-09-13T16:26:41.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.428+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3500d1260 con 0x7ff334066730 2024-09-13T16:26:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.444+0000 7ff347fff700 1 -- 172.21.15.119:0/4114762056 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+273127 (secure 0 0 0) 0x7ff3400013f0 con 0x7ff334066730 2024-09-13T16:26:41.455 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:41.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff334066730 msgr2=0x7ff334068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff334066730 0x7ff334068bf0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7ff340000f60 tx=0x7ff340007e00 comp rx=0 tx=0).stop 2024-09-13T16:26:41.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3500cd860 msgr2=0x7ff3500b5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3500cd860 0x7ff3500b5fb0 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7ff338009850 tx=0x7ff338004770 comp rx=0 tx=0).stop 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 shutdown_connections 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff334066730 0x7ff334068bf0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff35015cc40 0x7ff3500b64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3500cd860 0x7ff3500b5fb0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 --2- 172.21.15.119:0/4114762056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff35008c6b0 0x7ff3501f5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 >> 172.21.15.119:0/4114762056 conn(0x7ff3500b4640 msgr2=0x7ff35006ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 shutdown_connections 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.452+0000 7ff354c7a700 1 -- 172.21.15.119:0/4114762056 wait complete. 2024-09-13T16:26:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:26:41.475 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":78,"stamp":"2024-09-13T16:26:39.449094+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":220572,"kb_used_data":4360,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754148,"statfs":{"total":773094113280,"available":772868247552,"internally_reserved":0,"allocated":4464640,"data_stored":2547720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000994"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:37.027026+0000","last_change":"2024-09-13T16:26:37.027026+0000","last_active":"2024-09-13T16:26:37.027026+0000","last_peered":"2024-09-13T16:26:37.027026+0000","last_clean":"2024-09-13T16:26:37.027026+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:37.027026+0000","last_undegraded":"2024-09-13T16:26:37.027026+0000","last_fullsized":"2024-09-13T16:26:37.027026+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:26:36.013488+0000","last_change":"2024-09-13T16:26:36.013488+0000","last_active":"2024-09-13T16:26:36.013488+0000","last_peered":"2024-09-13T16:26:36.013488+0000","last_clean":"2024-09-13T16:26:36.013488+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:26:36.013488+0000","last_undegraded":"2024-09-13T16:26:36.013488+0000","last_fullsized":"2024-09-13T16:26:36.013488+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":174,"reported_epoch":35,"state":"active+clean","last_fresh":"2024-09-13T16:26:30.162307+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:26:30.162307+0000","last_peered":"2024-09-13T16:26:30.162307+0000","last_clean":"2024-09-13T16:26:30.162307+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:26:30.162307+0000","last_undegraded":"2024-09-13T16:26:30.162307+0000","last_fullsized":"2024-09-13T16:26:30.162307+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343840,"statfs":{"total":96636764160,"available":96608092160,"internally_reserved":0,"allocated":1003520,"data_stored":761241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.97199999999999998},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.65100000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":0.90200000000000002},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":1.226}]},{"osd":2,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.70699999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":1.0489999999999999},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.83799999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.80400000000000005},{"interface":"front","average":{"1min":1.103,"5min":1.103,"15min":1.103},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.589,"5min":1.589,"15min":1.589},"last":1.123}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.1439999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.4199999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27424,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":409600,"data_stored":170873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.88}]},{"osd":1,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":1.036}]},{"osd":2,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.008},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":0.76300000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":1.1819999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":1.121}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":1.083},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1559999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":1.2010000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27420,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344420,"statfs":{"total":96636764160,"available":96608686080,"internally_reserved":0,"allocated":409600,"data_stored":170873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":1.0580000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":1.0369999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.84799999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":0.999},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.88600000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":0.93700000000000006},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":0.754}]},{"osd":4,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":3.6869999999999998,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":1.1619999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":1.083},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.133}]},{"osd":7,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":3.9830000000000001,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.194}]}]},{"osd":4,"up_from":14,"seq":60129542163,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27440,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344400,"statfs":{"total":96636764160,"available":96608665600,"internally_reserved":0,"allocated":409600,"data_stored":170873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.746}]},{"osd":2,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.80500000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.91300000000000003},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.70499999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.8860000000000001,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.97099999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.77100000000000002},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":0.98499999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":0.85199999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27428,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344412,"statfs":{"total":96636764160,"available":96608677888,"internally_reserved":0,"allocated":409600,"data_stored":170873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.81599999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.86299999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.782,"5min":1.782,"15min":1.782},"last":1.006}]},{"osd":5,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.2749999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.179},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":1.026}]},{"osd":7,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":1.889},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":1.0469999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542165,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27424,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344416,"statfs":{"total":96636764160,"available":96608681984,"internally_reserved":0,"allocated":409600,"data_stored":170873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.53300000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.69699999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.502},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.83999999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.93799999999999994}]},{"osd":6,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":0.82399999999999995},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.97699999999999998}]},{"osd":7,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":0.96099999999999997}]}]},{"osd":1,"up_from":14,"seq":60129542164,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27432,"kb_used_data":400,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344408,"statfs":{"total":96636764160,"available":96608673792,"internally_reserved":0,"allocated":409600,"data_stored":170873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.59199999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.63900000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.78700000000000003}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":0.754},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.93300000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.97799999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.008,"15min":1.008},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":0.85699999999999998}]},{"osd":7,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.145},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.84699999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542164,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28004,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343836,"statfs":{"total":96636764160,"available":96608088064,"internally_reserved":0,"allocated":1003520,"data_stored":761241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.61299999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.64300000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.75},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":0.94099999999999995}]},{"osd":5,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.95899999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":0.82199999999999995}]},{"osd":7,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":0.92600000000000005},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.079}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:26:41.481 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:26:41.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 Processor -- start 2024-09-13T16:26:41.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- start start 2024-09-13T16:26:41.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f95bc1444b0 0x7f95bc06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0cbb10 0x7f95bc0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95bc06bd60 con 0x7f95bc1440d0 2024-09-13T16:26:41.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95bc0cc4c0 con 0x7f95bc1444b0 2024-09-13T16:26:41.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc06bfe0 con 0x7f95bc0cbb10 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f95bc1444b0 0x7f95bc06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f95bc1444b0 0x7f95bc06d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50456/0 (socket says 172.21.15.119:50456) 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 -- 172.21.15.119:0/2040847138 learned_addr learned my addr 172.21.15.119:0/2040847138 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c1bad700 1 --2- 172.21.15.119:0/2040847138 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0cbb10 0x7f95bc0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 -- 172.21.15.119:0/2040847138 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0cbb10 msgr2=0x7f95bc0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 --2- 172.21.15.119:0/2040847138 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0cbb10 0x7f95bc0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 -- 172.21.15.119:0/2040847138 >> v1:172.21.15.145:6789/0 conn(0x7f95bc1440d0 legacy=0x7f95bc15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 -- 172.21.15.119:0/2040847138 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc06c2c0 con 0x7f95bc1444b0 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c0bab700 1 --2- 172.21.15.119:0/2040847138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc06d080 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009910 tx=0x7f95b0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9d1232abb75f739 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95affff700 1 -- 172.21.15.119:0/2040847138 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b000ba60 con 0x7f95bc1444b0 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95affff700 1 -- 172.21.15.119:0/2040847138 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b000bbc0 con 0x7f95bc1444b0 2024-09-13T16:26:41.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95affff700 1 -- 172.21.15.119:0/2040847138 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b0009e60 con 0x7f95bc1444b0 2024-09-13T16:26:41.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- 172.21.15.119:0/2040847138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 msgr2=0x7f95bc06d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 --2- 172.21.15.119:0/2040847138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc06d080 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f95b0009910 tx=0x7f95b0009c20 comp rx=0 tx=0).stop 2024-09-13T16:26:41.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- 172.21.15.119:0/2040847138 shutdown_connections 2024-09-13T16:26:41.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 --2- 172.21.15.119:0/2040847138 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc0cbb10 0x7f95bc0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 --2- 172.21.15.119:0/2040847138 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc06d080 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- 172.21.15.119:0/2040847138 >> 172.21.15.119:0/2040847138 conn(0x7f95bc06a350 msgr2=0x7f95bc06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:41.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- 172.21.15.119:0/2040847138 shutdown_connections 2024-09-13T16:26:41.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.592+0000 7f95c3610700 1 -- 172.21.15.119:0/2040847138 wait complete. 2024-09-13T16:26:41.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 Processor -- start 2024-09-13T16:26:41.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 -- start start 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0cbb10 0x7f95bc1efc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc1440d0 0x7f95bc0af8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc0afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc13fce0 con 0x7f95bc1444b0 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc1f0140 con 0x7f95bc0cbb10 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c3610700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95bc13fa00 con 0x7f95bc1440d0 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c1bad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc0afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c1bad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc0afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50470/0 (socket says 172.21.15.119:50470) 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c1bad700 1 -- 172.21.15.119:0/2083196983 learned_addr learned my addr 172.21.15.119:0/2083196983 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c13ac700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0cbb10 0x7f95bc1efc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.596+0000 7f95c0bab700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc1440d0 0x7f95bc0af8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c1bad700 1 -- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc1440d0 msgr2=0x7f95bc0af8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c1bad700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc1440d0 0x7f95bc0af8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c1bad700 1 -- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0cbb10 msgr2=0x7f95bc1efc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c1bad700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0cbb10 0x7f95bc1efc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c1bad700 1 -- 172.21.15.119:0/2083196983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95b0009570 con 0x7f95bc1444b0 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c0bab700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc1440d0 0x7f95bc0af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c13ac700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0cbb10 0x7f95bc1efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c1bad700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc0afde0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f95b4007e70 tx=0x7f95b400e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95adffb700 1 -- 172.21.15.119:0/2083196983 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b400ee50 con 0x7f95bc1444b0 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95adffb700 1 -- 172.21.15.119:0/2083196983 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95b4014420 con 0x7f95bc1444b0 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95bc139af0 con 0x7f95bc1444b0 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95adffb700 1 -- 172.21.15.119:0/2083196983 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b40147d0 con 0x7f95bc1444b0 2024-09-13T16:26:41.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95bc13a0b0 con 0x7f95bc1444b0 2024-09-13T16:26:41.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95bc067760 con 0x7f95bc1444b0 2024-09-13T16:26:41.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95adffb700 1 -- 172.21.15.119:0/2083196983 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f95b4014930 con 0x7f95bc1444b0 2024-09-13T16:26:41.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95adffb700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f95a0066730 0x7f95a0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95adffb700 1 -- 172.21.15.119:0/2083196983 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f95b4084a50 con 0x7f95bc1444b0 2024-09-13T16:26:41.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.600+0000 7f95c13ac700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f95a0066730 0x7f95a0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.604+0000 7f95c13ac700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f95a0066730 0x7f95a0068bf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f95b8000f60 tx=0x7f95b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.608+0000 7f95adffb700 1 -- 172.21.15.119:0/2083196983 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f95b4051c90 con 0x7f95bc1444b0 2024-09-13T16:26:41.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.772+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f95bc06cb60 con 0x7f95a0066730 2024-09-13T16:26:41.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.788+0000 7f95adffb700 1 -- 172.21.15.119:0/2083196983 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287880 (secure 0 0 0) 0x7f95b8001d70 con 0x7f95a0066730 2024-09-13T16:26:41.798 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:41.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f95a0066730 msgr2=0x7f95a0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f95a0066730 0x7f95a0068bf0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f95b8000f60 tx=0x7f95b8009040 comp rx=0 tx=0).stop 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 msgr2=0x7f95bc0afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc0afde0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f95b4007e70 tx=0x7f95b400e3c0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 shutdown_connections 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f95a0066730 0x7f95a0068bf0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f95bc1444b0 0x7f95bc0afde0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f95bc1440d0 0x7f95bc0af8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 --2- 172.21.15.119:0/2083196983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f95bc0cbb10 0x7f95bc1efc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 >> 172.21.15.119:0/2083196983 conn(0x7f95bc06a350 msgr2=0x7f95bc15d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 shutdown_connections 2024-09-13T16:26:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.796+0000 7f95c3610700 1 -- 172.21.15.119:0/2083196983 wait complete. 2024-09-13T16:26:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:26:41.814 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":79,"stamp":"2024-09-13T16:26:41.449638+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":168,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221024,"kb_used_data":4808,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753696,"statfs":{"total":773094113280,"available":772867784704,"internally_reserved":0,"allocated":4923392,"data_stored":2764020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":221236793},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.420610"},"pg_stats":[{"pgid":"6.3f","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.433188+0000","last_change":"2024-09-13T16:26:37.441550+0000","last_active":"2024-09-13T16:26:38.433188+0000","last_peered":"2024-09-13T16:26:38.433188+0000","last_clean":"2024-09-13T16:26:38.433188+0000","last_became_active":"2024-09-13T16:26:37.441341+0000","last_became_peered":"2024-09-13T16:26:37.441341+0000","last_unstale":"2024-09-13T16:26:38.433188+0000","last_undegraded":"2024-09-13T16:26:38.433188+0000","last_fullsized":"2024-09-13T16:26:38.433188+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:42.081816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3e","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432660+0000","last_change":"2024-09-13T16:26:37.441822+0000","last_active":"2024-09-13T16:26:38.432660+0000","last_peered":"2024-09-13T16:26:38.432660+0000","last_clean":"2024-09-13T16:26:38.432660+0000","last_became_active":"2024-09-13T16:26:37.441561+0000","last_became_peered":"2024-09-13T16:26:37.441561+0000","last_unstale":"2024-09-13T16:26:38.432660+0000","last_undegraded":"2024-09-13T16:26:38.432660+0000","last_fullsized":"2024-09-13T16:26:38.432660+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:17.604909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3d","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293485+0000","last_change":"2024-09-13T16:26:37.452531+0000","last_active":"2024-09-13T16:26:39.293485+0000","last_peered":"2024-09-13T16:26:39.293485+0000","last_clean":"2024-09-13T16:26:39.293485+0000","last_became_active":"2024-09-13T16:26:37.452239+0000","last_became_peered":"2024-09-13T16:26:37.452239+0000","last_unstale":"2024-09-13T16:26:39.293485+0000","last_undegraded":"2024-09-13T16:26:39.293485+0000","last_fullsized":"2024-09-13T16:26:39.293485+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:10:32.085156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3c","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292240+0000","last_change":"2024-09-13T16:26:37.453633+0000","last_active":"2024-09-13T16:26:39.292240+0000","last_peered":"2024-09-13T16:26:39.292240+0000","last_clean":"2024-09-13T16:26:39.292240+0000","last_became_active":"2024-09-13T16:26:37.453394+0000","last_became_peered":"2024-09-13T16:26:37.453394+0000","last_unstale":"2024-09-13T16:26:39.292240+0000","last_undegraded":"2024-09-13T16:26:39.292240+0000","last_fullsized":"2024-09-13T16:26:39.292240+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:48.074413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3b","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.291999+0000","last_change":"2024-09-13T16:26:37.452538+0000","last_active":"2024-09-13T16:26:39.291999+0000","last_peered":"2024-09-13T16:26:39.291999+0000","last_clean":"2024-09-13T16:26:39.291999+0000","last_became_active":"2024-09-13T16:26:37.452271+0000","last_became_peered":"2024-09-13T16:26:37.452271+0000","last_unstale":"2024-09-13T16:26:39.291999+0000","last_undegraded":"2024-09-13T16:26:39.291999+0000","last_fullsized":"2024-09-13T16:26:39.291999+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:02.793431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.3a","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253631+0000","last_change":"2024-09-13T16:26:37.437239+0000","last_active":"2024-09-13T16:26:39.253631+0000","last_peered":"2024-09-13T16:26:39.253631+0000","last_clean":"2024-09-13T16:26:39.253631+0000","last_became_active":"2024-09-13T16:26:37.436979+0000","last_became_peered":"2024-09-13T16:26:37.436979+0000","last_unstale":"2024-09-13T16:26:39.253631+0000","last_undegraded":"2024-09-13T16:26:39.253631+0000","last_fullsized":"2024-09-13T16:26:39.253631+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:58.246218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.39","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.436767+0000","last_change":"2024-09-13T16:26:37.442822+0000","last_active":"2024-09-13T16:26:38.436767+0000","last_peered":"2024-09-13T16:26:38.436767+0000","last_clean":"2024-09-13T16:26:38.436767+0000","last_became_active":"2024-09-13T16:26:37.442516+0000","last_became_peered":"2024-09-13T16:26:37.442516+0000","last_unstale":"2024-09-13T16:26:38.436767+0000","last_undegraded":"2024-09-13T16:26:38.436767+0000","last_fullsized":"2024-09-13T16:26:38.436767+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:49:00.674469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.38","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.431823+0000","last_change":"2024-09-13T16:26:37.443422+0000","last_active":"2024-09-13T16:26:38.431823+0000","last_peered":"2024-09-13T16:26:38.431823+0000","last_clean":"2024-09-13T16:26:38.431823+0000","last_became_active":"2024-09-13T16:26:37.443160+0000","last_became_peered":"2024-09-13T16:26:37.443160+0000","last_unstale":"2024-09-13T16:26:38.431823+0000","last_undegraded":"2024-09-13T16:26:38.431823+0000","last_fullsized":"2024-09-13T16:26:38.431823+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:30.436331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.37","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293766+0000","last_change":"2024-09-13T16:26:37.449454+0000","last_active":"2024-09-13T16:26:39.293766+0000","last_peered":"2024-09-13T16:26:39.293766+0000","last_clean":"2024-09-13T16:26:39.293766+0000","last_became_active":"2024-09-13T16:26:37.449047+0000","last_became_peered":"2024-09-13T16:26:37.449047+0000","last_unstale":"2024-09-13T16:26:39.293766+0000","last_undegraded":"2024-09-13T16:26:39.293766+0000","last_fullsized":"2024-09-13T16:26:39.293766+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:17.604909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.36","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437986+0000","last_change":"2024-09-13T16:26:37.451524+0000","last_active":"2024-09-13T16:26:38.437986+0000","last_peered":"2024-09-13T16:26:38.437986+0000","last_clean":"2024-09-13T16:26:38.437986+0000","last_became_active":"2024-09-13T16:26:37.450923+0000","last_became_peered":"2024-09-13T16:26:37.450923+0000","last_unstale":"2024-09-13T16:26:38.437986+0000","last_undegraded":"2024-09-13T16:26:38.437986+0000","last_fullsized":"2024-09-13T16:26:38.437986+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:45.022985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.35","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438001+0000","last_change":"2024-09-13T16:26:37.453124+0000","last_active":"2024-09-13T16:26:38.438001+0000","last_peered":"2024-09-13T16:26:38.438001+0000","last_clean":"2024-09-13T16:26:38.438001+0000","last_became_active":"2024-09-13T16:26:37.452846+0000","last_became_peered":"2024-09-13T16:26:37.452846+0000","last_unstale":"2024-09-13T16:26:38.438001+0000","last_undegraded":"2024-09-13T16:26:38.438001+0000","last_fullsized":"2024-09-13T16:26:38.438001+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:27:52.130855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.34","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293440+0000","last_change":"2024-09-13T16:26:37.444981+0000","last_active":"2024-09-13T16:26:39.293440+0000","last_peered":"2024-09-13T16:26:39.293440+0000","last_clean":"2024-09-13T16:26:39.293440+0000","last_became_active":"2024-09-13T16:26:37.443804+0000","last_became_peered":"2024-09-13T16:26:37.443804+0000","last_unstale":"2024-09-13T16:26:39.293440+0000","last_undegraded":"2024-09-13T16:26:39.293440+0000","last_fullsized":"2024-09-13T16:26:39.293440+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:29.667578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.33","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437195+0000","last_change":"2024-09-13T16:26:37.451491+0000","last_active":"2024-09-13T16:26:38.437195+0000","last_peered":"2024-09-13T16:26:38.437195+0000","last_clean":"2024-09-13T16:26:38.437195+0000","last_became_active":"2024-09-13T16:26:37.450507+0000","last_became_peered":"2024-09-13T16:26:37.450507+0000","last_unstale":"2024-09-13T16:26:38.437195+0000","last_undegraded":"2024-09-13T16:26:38.437195+0000","last_fullsized":"2024-09-13T16:26:38.437195+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:01:07.431910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.32","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293349+0000","last_change":"2024-09-13T16:26:37.459947+0000","last_active":"2024-09-13T16:26:39.293349+0000","last_peered":"2024-09-13T16:26:39.293349+0000","last_clean":"2024-09-13T16:26:39.293349+0000","last_became_active":"2024-09-13T16:26:37.459687+0000","last_became_peered":"2024-09-13T16:26:37.459687+0000","last_unstale":"2024-09-13T16:26:39.293349+0000","last_undegraded":"2024-09-13T16:26:39.293349+0000","last_fullsized":"2024-09-13T16:26:39.293349+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:51.842712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.31","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437007+0000","last_change":"2024-09-13T16:26:37.445585+0000","last_active":"2024-09-13T16:26:38.437007+0000","last_peered":"2024-09-13T16:26:38.437007+0000","last_clean":"2024-09-13T16:26:38.437007+0000","last_became_active":"2024-09-13T16:26:37.444916+0000","last_became_peered":"2024-09-13T16:26:37.444916+0000","last_unstale":"2024-09-13T16:26:38.437007+0000","last_undegraded":"2024-09-13T16:26:38.437007+0000","last_fullsized":"2024-09-13T16:26:38.437007+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:49.288794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.30","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292425+0000","last_change":"2024-09-13T16:26:37.444359+0000","last_active":"2024-09-13T16:26:39.292425+0000","last_peered":"2024-09-13T16:26:39.292425+0000","last_clean":"2024-09-13T16:26:39.292425+0000","last_became_active":"2024-09-13T16:26:37.444106+0000","last_became_peered":"2024-09-13T16:26:37.444106+0000","last_unstale":"2024-09-13T16:26:39.292425+0000","last_undegraded":"2024-09-13T16:26:39.292425+0000","last_fullsized":"2024-09-13T16:26:39.292425+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:21.340977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2f","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.294169+0000","last_change":"2024-09-13T16:26:37.451334+0000","last_active":"2024-09-13T16:26:39.294169+0000","last_peered":"2024-09-13T16:26:39.294169+0000","last_clean":"2024-09-13T16:26:39.294169+0000","last_became_active":"2024-09-13T16:26:37.450962+0000","last_became_peered":"2024-09-13T16:26:37.450962+0000","last_unstale":"2024-09-13T16:26:39.294169+0000","last_undegraded":"2024-09-13T16:26:39.294169+0000","last_fullsized":"2024-09-13T16:26:39.294169+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:01:32.295711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2e","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437727+0000","last_change":"2024-09-13T16:26:37.453936+0000","last_active":"2024-09-13T16:26:38.437727+0000","last_peered":"2024-09-13T16:26:38.437727+0000","last_clean":"2024-09-13T16:26:38.437727+0000","last_became_active":"2024-09-13T16:26:37.453687+0000","last_became_peered":"2024-09-13T16:26:37.453687+0000","last_unstale":"2024-09-13T16:26:38.437727+0000","last_undegraded":"2024-09-13T16:26:38.437727+0000","last_fullsized":"2024-09-13T16:26:38.437727+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:25.314603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2d","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292476+0000","last_change":"2024-09-13T16:26:37.445265+0000","last_active":"2024-09-13T16:26:39.292476+0000","last_peered":"2024-09-13T16:26:39.292476+0000","last_clean":"2024-09-13T16:26:39.292476+0000","last_became_active":"2024-09-13T16:26:37.444993+0000","last_became_peered":"2024-09-13T16:26:37.444993+0000","last_unstale":"2024-09-13T16:26:39.292476+0000","last_undegraded":"2024-09-13T16:26:39.292476+0000","last_fullsized":"2024-09-13T16:26:39.292476+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:26.118626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.2c","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437135+0000","last_change":"2024-09-13T16:26:37.452872+0000","last_active":"2024-09-13T16:26:38.437135+0000","last_peered":"2024-09-13T16:26:38.437135+0000","last_clean":"2024-09-13T16:26:38.437135+0000","last_became_active":"2024-09-13T16:26:37.452598+0000","last_became_peered":"2024-09-13T16:26:37.452598+0000","last_unstale":"2024-09-13T16:26:38.437135+0000","last_undegraded":"2024-09-13T16:26:38.437135+0000","last_fullsized":"2024-09-13T16:26:38.437135+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:24.964357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2b","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437342+0000","last_change":"2024-09-13T16:26:37.446041+0000","last_active":"2024-09-13T16:26:38.437342+0000","last_peered":"2024-09-13T16:26:38.437342+0000","last_clean":"2024-09-13T16:26:38.437342+0000","last_became_active":"2024-09-13T16:26:37.445755+0000","last_became_peered":"2024-09-13T16:26:37.445755+0000","last_unstale":"2024-09-13T16:26:38.437342+0000","last_undegraded":"2024-09-13T16:26:38.437342+0000","last_fullsized":"2024-09-13T16:26:38.437342+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:06.200531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2a","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.294007+0000","last_change":"2024-09-13T16:26:37.457929+0000","last_active":"2024-09-13T16:26:39.294007+0000","last_peered":"2024-09-13T16:26:39.294007+0000","last_clean":"2024-09-13T16:26:39.294007+0000","last_became_active":"2024-09-13T16:26:37.457312+0000","last_became_peered":"2024-09-13T16:26:37.457312+0000","last_unstale":"2024-09-13T16:26:39.294007+0000","last_undegraded":"2024-09-13T16:26:39.294007+0000","last_fullsized":"2024-09-13T16:26:39.294007+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:58.246218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.29","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292560+0000","last_change":"2024-09-13T16:26:37.448057+0000","last_active":"2024-09-13T16:26:39.292560+0000","last_peered":"2024-09-13T16:26:39.292560+0000","last_clean":"2024-09-13T16:26:39.292560+0000","last_became_active":"2024-09-13T16:26:37.447390+0000","last_became_peered":"2024-09-13T16:26:37.447390+0000","last_unstale":"2024-09-13T16:26:39.292560+0000","last_undegraded":"2024-09-13T16:26:39.292560+0000","last_fullsized":"2024-09-13T16:26:39.292560+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:51.329455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.291684+0000","last_change":"2024-09-13T16:26:37.450472+0000","last_active":"2024-09-13T16:26:39.291684+0000","last_peered":"2024-09-13T16:26:39.291684+0000","last_clean":"2024-09-13T16:26:39.291684+0000","last_became_active":"2024-09-13T16:26:37.450125+0000","last_became_peered":"2024-09-13T16:26:37.450125+0000","last_unstale":"2024-09-13T16:26:39.291684+0000","last_undegraded":"2024-09-13T16:26:39.291684+0000","last_fullsized":"2024-09-13T16:26:39.291684+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:27.196467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437968+0000","last_change":"2024-09-13T16:26:37.442657+0000","last_active":"2024-09-13T16:26:38.437968+0000","last_peered":"2024-09-13T16:26:38.437968+0000","last_clean":"2024-09-13T16:26:38.437968+0000","last_became_active":"2024-09-13T16:26:37.442415+0000","last_became_peered":"2024-09-13T16:26:37.442415+0000","last_unstale":"2024-09-13T16:26:38.437968+0000","last_undegraded":"2024-09-13T16:26:38.437968+0000","last_fullsized":"2024-09-13T16:26:38.437968+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:22.300657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253957+0000","last_change":"2024-09-13T16:26:37.444099+0000","last_active":"2024-09-13T16:26:39.253957+0000","last_peered":"2024-09-13T16:26:39.253957+0000","last_clean":"2024-09-13T16:26:39.253957+0000","last_became_active":"2024-09-13T16:26:37.443859+0000","last_became_peered":"2024-09-13T16:26:37.443859+0000","last_unstale":"2024-09-13T16:26:39.253957+0000","last_undegraded":"2024-09-13T16:26:39.253957+0000","last_fullsized":"2024-09-13T16:26:39.253957+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.288450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437477+0000","last_change":"2024-09-13T16:26:37.443346+0000","last_active":"2024-09-13T16:26:38.437477+0000","last_peered":"2024-09-13T16:26:38.437477+0000","last_clean":"2024-09-13T16:26:38.437477+0000","last_became_active":"2024-09-13T16:26:37.443073+0000","last_became_peered":"2024-09-13T16:26:37.443073+0000","last_unstale":"2024-09-13T16:26:38.437477+0000","last_undegraded":"2024-09-13T16:26:38.437477+0000","last_fullsized":"2024-09-13T16:26:38.437477+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:28.507015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437873+0000","last_change":"2024-09-13T16:26:37.442747+0000","last_active":"2024-09-13T16:26:38.437873+0000","last_peered":"2024-09-13T16:26:38.437873+0000","last_clean":"2024-09-13T16:26:38.437873+0000","last_became_active":"2024-09-13T16:26:37.442466+0000","last_became_peered":"2024-09-13T16:26:37.442466+0000","last_unstale":"2024-09-13T16:26:38.437873+0000","last_undegraded":"2024-09-13T16:26:38.437873+0000","last_fullsized":"2024-09-13T16:26:38.437873+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:43:13.523576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437042+0000","last_change":"2024-09-13T16:26:37.434459+0000","last_active":"2024-09-13T16:26:38.437042+0000","last_peered":"2024-09-13T16:26:38.437042+0000","last_clean":"2024-09-13T16:26:38.437042+0000","last_became_active":"2024-09-13T16:26:37.434203+0000","last_became_peered":"2024-09-13T16:26:37.434203+0000","last_unstale":"2024-09-13T16:26:38.437042+0000","last_undegraded":"2024-09-13T16:26:38.437042+0000","last_fullsized":"2024-09-13T16:26:38.437042+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:37:11.532820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.436732+0000","last_change":"2024-09-13T16:26:37.440167+0000","last_active":"2024-09-13T16:26:38.436732+0000","last_peered":"2024-09-13T16:26:38.436732+0000","last_clean":"2024-09-13T16:26:38.436732+0000","last_became_active":"2024-09-13T16:26:37.439137+0000","last_became_peered":"2024-09-13T16:26:37.439137+0000","last_unstale":"2024-09-13T16:26:38.436732+0000","last_undegraded":"2024-09-13T16:26:38.436732+0000","last_fullsized":"2024-09-13T16:26:38.436732+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:04:00.165353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292732+0000","last_change":"2024-09-13T16:26:37.445486+0000","last_active":"2024-09-13T16:26:39.292732+0000","last_peered":"2024-09-13T16:26:39.292732+0000","last_clean":"2024-09-13T16:26:39.292732+0000","last_became_active":"2024-09-13T16:26:37.445169+0000","last_became_peered":"2024-09-13T16:26:37.445169+0000","last_unstale":"2024-09-13T16:26:39.292732+0000","last_undegraded":"2024-09-13T16:26:39.292732+0000","last_fullsized":"2024-09-13T16:26:39.292732+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:49.093218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253384+0000","last_change":"2024-09-13T16:26:37.442463+0000","last_active":"2024-09-13T16:26:39.253384+0000","last_peered":"2024-09-13T16:26:39.253384+0000","last_clean":"2024-09-13T16:26:39.253384+0000","last_became_active":"2024-09-13T16:26:37.442236+0000","last_became_peered":"2024-09-13T16:26:37.442236+0000","last_unstale":"2024-09-13T16:26:39.253384+0000","last_undegraded":"2024-09-13T16:26:39.253384+0000","last_fullsized":"2024-09-13T16:26:39.253384+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:47:15.354268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292960+0000","last_change":"2024-09-13T16:26:37.446218+0000","last_active":"2024-09-13T16:26:39.292960+0000","last_peered":"2024-09-13T16:26:39.292960+0000","last_clean":"2024-09-13T16:26:39.292960+0000","last_became_active":"2024-09-13T16:26:37.445965+0000","last_became_peered":"2024-09-13T16:26:37.445965+0000","last_unstale":"2024-09-13T16:26:39.292960+0000","last_undegraded":"2024-09-13T16:26:39.292960+0000","last_fullsized":"2024-09-13T16:26:39.292960+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:47:15.354268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"39'4","reported_seq":15,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438097+0000","last_change":"2024-09-13T16:26:37.052205+0000","last_active":"2024-09-13T16:26:38.438097+0000","last_peered":"2024-09-13T16:26:38.438097+0000","last_clean":"2024-09-13T16:26:38.438097+0000","last_became_active":"2024-09-13T16:26:37.051925+0000","last_became_peered":"2024-09-13T16:26:37.051925+0000","last_unstale":"2024-09-13T16:26:38.438097+0000","last_undegraded":"2024-09-13T16:26:38.438097+0000","last_fullsized":"2024-09-13T16:26:38.438097+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:54:29.609982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.254196+0000","last_change":"2024-09-13T16:26:37.453178+0000","last_active":"2024-09-13T16:26:39.254196+0000","last_peered":"2024-09-13T16:26:39.254196+0000","last_clean":"2024-09-13T16:26:39.254196+0000","last_became_active":"2024-09-13T16:26:37.452968+0000","last_became_peered":"2024-09-13T16:26:37.452968+0000","last_unstale":"2024-09-13T16:26:39.254196+0000","last_undegraded":"2024-09-13T16:26:39.254196+0000","last_fullsized":"2024-09-13T16:26:39.254196+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:27.543765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"39'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293696+0000","last_change":"2024-09-13T16:26:37.056618+0000","last_active":"2024-09-13T16:26:39.293696+0000","last_peered":"2024-09-13T16:26:39.293696+0000","last_clean":"2024-09-13T16:26:39.293696+0000","last_became_active":"2024-09-13T16:26:37.056298+0000","last_became_peered":"2024-09-13T16:26:37.056298+0000","last_unstale":"2024-09-13T16:26:39.293696+0000","last_undegraded":"2024-09-13T16:26:39.293696+0000","last_fullsized":"2024-09-13T16:26:39.293696+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:52.274912+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.294177+0000","last_change":"2024-09-13T16:26:37.459963+0000","last_active":"2024-09-13T16:26:39.294177+0000","last_peered":"2024-09-13T16:26:39.294177+0000","last_clean":"2024-09-13T16:26:39.294177+0000","last_became_active":"2024-09-13T16:26:37.459692+0000","last_became_peered":"2024-09-13T16:26:37.459692+0000","last_unstale":"2024-09-13T16:26:39.294177+0000","last_undegraded":"2024-09-13T16:26:39.294177+0000","last_fullsized":"2024-09-13T16:26:39.294177+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:16:24.244662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"39'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.254309+0000","last_change":"2024-09-13T16:26:37.054953+0000","last_active":"2024-09-13T16:26:39.254309+0000","last_peered":"2024-09-13T16:26:39.254309+0000","last_clean":"2024-09-13T16:26:39.254309+0000","last_became_active":"2024-09-13T16:26:37.054609+0000","last_became_peered":"2024-09-13T16:26:37.054609+0000","last_unstale":"2024-09-13T16:26:39.254309+0000","last_undegraded":"2024-09-13T16:26:39.254309+0000","last_fullsized":"2024-09-13T16:26:39.254309+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:59.151815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432977+0000","last_change":"2024-09-13T16:26:37.433580+0000","last_active":"2024-09-13T16:26:38.432977+0000","last_peered":"2024-09-13T16:26:38.432977+0000","last_clean":"2024-09-13T16:26:38.432977+0000","last_became_active":"2024-09-13T16:26:37.433316+0000","last_became_peered":"2024-09-13T16:26:37.433316+0000","last_unstale":"2024-09-13T16:26:38.432977+0000","last_undegraded":"2024-09-13T16:26:38.432977+0000","last_fullsized":"2024-09-13T16:26:38.432977+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:39.782793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"39'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.294171+0000","last_change":"2024-09-13T16:26:37.059307+0000","last_active":"2024-09-13T16:26:39.294171+0000","last_peered":"2024-09-13T16:26:39.294171+0000","last_clean":"2024-09-13T16:26:39.294171+0000","last_became_active":"2024-09-13T16:26:37.056029+0000","last_became_peered":"2024-09-13T16:26:37.056029+0000","last_unstale":"2024-09-13T16:26:39.294171+0000","last_undegraded":"2024-09-13T16:26:39.294171+0000","last_fullsized":"2024-09-13T16:26:39.294171+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:21.287119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292012+0000","last_change":"2024-09-13T16:26:37.440929+0000","last_active":"2024-09-13T16:26:39.292012+0000","last_peered":"2024-09-13T16:26:39.292012+0000","last_clean":"2024-09-13T16:26:39.292012+0000","last_became_active":"2024-09-13T16:26:37.440682+0000","last_became_peered":"2024-09-13T16:26:37.440682+0000","last_unstale":"2024-09-13T16:26:39.292012+0000","last_undegraded":"2024-09-13T16:26:39.292012+0000","last_fullsized":"2024-09-13T16:26:39.292012+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:25:46.811545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"39'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437469+0000","last_change":"2024-09-13T16:26:37.053540+0000","last_active":"2024-09-13T16:26:38.437469+0000","last_peered":"2024-09-13T16:26:38.437469+0000","last_clean":"2024-09-13T16:26:38.437469+0000","last_became_active":"2024-09-13T16:26:37.053191+0000","last_became_peered":"2024-09-13T16:26:37.053191+0000","last_unstale":"2024-09-13T16:26:38.437469+0000","last_undegraded":"2024-09-13T16:26:38.437469+0000","last_fullsized":"2024-09-13T16:26:38.437469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:47.060876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.437240+0000","last_change":"2024-09-13T16:26:37.445783+0000","last_active":"2024-09-13T16:26:38.437240+0000","last_peered":"2024-09-13T16:26:38.437240+0000","last_clean":"2024-09-13T16:26:38.437240+0000","last_became_active":"2024-09-13T16:26:37.445287+0000","last_became_peered":"2024-09-13T16:26:37.445287+0000","last_unstale":"2024-09-13T16:26:38.437240+0000","last_undegraded":"2024-09-13T16:26:38.437240+0000","last_fullsized":"2024-09-13T16:26:38.437240+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:51:33.441005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"39'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292996+0000","last_change":"2024-09-13T16:26:37.061716+0000","last_active":"2024-09-13T16:26:39.292996+0000","last_peered":"2024-09-13T16:26:39.292996+0000","last_clean":"2024-09-13T16:26:39.292996+0000","last_became_active":"2024-09-13T16:26:37.061462+0000","last_became_peered":"2024-09-13T16:26:37.061462+0000","last_unstale":"2024-09-13T16:26:39.292996+0000","last_undegraded":"2024-09-13T16:26:39.292996+0000","last_fullsized":"2024-09-13T16:26:39.292996+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:27.191544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293604+0000","last_change":"2024-09-13T16:26:37.451478+0000","last_active":"2024-09-13T16:26:39.293604+0000","last_peered":"2024-09-13T16:26:39.293604+0000","last_clean":"2024-09-13T16:26:39.293604+0000","last_became_active":"2024-09-13T16:26:37.451139+0000","last_became_peered":"2024-09-13T16:26:37.451139+0000","last_unstale":"2024-09-13T16:26:39.293604+0000","last_undegraded":"2024-09-13T16:26:39.293604+0000","last_fullsized":"2024-09-13T16:26:39.293604+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:33:53.288450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253619+0000","last_change":"2024-09-13T16:26:37.051421+0000","last_active":"2024-09-13T16:26:39.253619+0000","last_peered":"2024-09-13T16:26:39.253619+0000","last_clean":"2024-09-13T16:26:39.253619+0000","last_became_active":"2024-09-13T16:26:37.051141+0000","last_became_peered":"2024-09-13T16:26:37.051141+0000","last_unstale":"2024-09-13T16:26:39.253619+0000","last_undegraded":"2024-09-13T16:26:39.253619+0000","last_fullsized":"2024-09-13T16:26:39.253619+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:21.287119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432212+0000","last_change":"2024-09-13T16:26:37.435536+0000","last_active":"2024-09-13T16:26:38.432212+0000","last_peered":"2024-09-13T16:26:38.432212+0000","last_clean":"2024-09-13T16:26:38.432212+0000","last_became_active":"2024-09-13T16:26:37.435068+0000","last_became_peered":"2024-09-13T16:26:37.435068+0000","last_unstale":"2024-09-13T16:26:38.432212+0000","last_undegraded":"2024-09-13T16:26:38.432212+0000","last_fullsized":"2024-09-13T16:26:38.432212+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:20.409058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293148+0000","last_change":"2024-09-13T16:26:37.055656+0000","last_active":"2024-09-13T16:26:39.293148+0000","last_peered":"2024-09-13T16:26:39.293148+0000","last_clean":"2024-09-13T16:26:39.293148+0000","last_became_active":"2024-09-13T16:26:37.055411+0000","last_became_peered":"2024-09-13T16:26:37.055411+0000","last_unstale":"2024-09-13T16:26:39.293148+0000","last_undegraded":"2024-09-13T16:26:39.293148+0000","last_fullsized":"2024-09-13T16:26:39.293148+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:29:03.961839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292417+0000","last_change":"2024-09-13T16:26:37.453051+0000","last_active":"2024-09-13T16:26:39.292417+0000","last_peered":"2024-09-13T16:26:39.292417+0000","last_clean":"2024-09-13T16:26:39.292417+0000","last_became_active":"2024-09-13T16:26:37.452270+0000","last_became_peered":"2024-09-13T16:26:37.452270+0000","last_unstale":"2024-09-13T16:26:39.292417+0000","last_undegraded":"2024-09-13T16:26:39.292417+0000","last_fullsized":"2024-09-13T16:26:39.292417+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:55.782606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.433323+0000","last_change":"2024-09-13T16:26:37.052057+0000","last_active":"2024-09-13T16:26:38.433323+0000","last_peered":"2024-09-13T16:26:38.433323+0000","last_clean":"2024-09-13T16:26:38.433323+0000","last_became_active":"2024-09-13T16:26:37.051823+0000","last_became_peered":"2024-09-13T16:26:37.051823+0000","last_unstale":"2024-09-13T16:26:38.433323+0000","last_undegraded":"2024-09-13T16:26:38.433323+0000","last_fullsized":"2024-09-13T16:26:38.433323+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:14.409181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293570+0000","last_change":"2024-09-13T16:26:37.448066+0000","last_active":"2024-09-13T16:26:39.293570+0000","last_peered":"2024-09-13T16:26:39.293570+0000","last_clean":"2024-09-13T16:26:39.293570+0000","last_became_active":"2024-09-13T16:26:37.447820+0000","last_became_peered":"2024-09-13T16:26:37.447820+0000","last_unstale":"2024-09-13T16:26:39.293570+0000","last_undegraded":"2024-09-13T16:26:39.293570+0000","last_fullsized":"2024-09-13T16:26:39.293570+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:27.543765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438302+0000","last_change":"2024-09-13T16:26:37.061088+0000","last_active":"2024-09-13T16:26:38.438302+0000","last_peered":"2024-09-13T16:26:38.438302+0000","last_clean":"2024-09-13T16:26:38.438302+0000","last_became_active":"2024-09-13T16:26:37.060790+0000","last_became_peered":"2024-09-13T16:26:37.060790+0000","last_unstale":"2024-09-13T16:26:38.438302+0000","last_undegraded":"2024-09-13T16:26:38.438302+0000","last_fullsized":"2024-09-13T16:26:38.438302+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:47:01.779894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293582+0000","last_change":"2024-09-13T16:26:37.439810+0000","last_active":"2024-09-13T16:26:39.293582+0000","last_peered":"2024-09-13T16:26:39.293582+0000","last_clean":"2024-09-13T16:26:39.293582+0000","last_became_active":"2024-09-13T16:26:37.439554+0000","last_became_peered":"2024-09-13T16:26:37.439554+0000","last_unstale":"2024-09-13T16:26:39.293582+0000","last_undegraded":"2024-09-13T16:26:39.293582+0000","last_fullsized":"2024-09-13T16:26:39.293582+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:15:12.305751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"39'3","reported_seq":14,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438146+0000","last_change":"2024-09-13T16:26:37.054839+0000","last_active":"2024-09-13T16:26:38.438146+0000","last_peered":"2024-09-13T16:26:38.438146+0000","last_clean":"2024-09-13T16:26:38.438146+0000","last_became_active":"2024-09-13T16:26:37.054563+0000","last_became_peered":"2024-09-13T16:26:37.054563+0000","last_unstale":"2024-09-13T16:26:38.438146+0000","last_undegraded":"2024-09-13T16:26:38.438146+0000","last_fullsized":"2024-09-13T16:26:38.438146+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:17:26.182929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.294429+0000","last_change":"2024-09-13T16:26:37.458067+0000","last_active":"2024-09-13T16:26:39.294429+0000","last_peered":"2024-09-13T16:26:39.294429+0000","last_clean":"2024-09-13T16:26:39.294429+0000","last_became_active":"2024-09-13T16:26:37.457653+0000","last_became_peered":"2024-09-13T16:26:37.457653+0000","last_unstale":"2024-09-13T16:26:39.294429+0000","last_undegraded":"2024-09-13T16:26:39.294429+0000","last_fullsized":"2024-09-13T16:26:39.294429+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:55.822650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"39'6","reported_seq":17,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432705+0000","last_change":"2024-09-13T16:26:37.056418+0000","last_active":"2024-09-13T16:26:38.432705+0000","last_peered":"2024-09-13T16:26:38.432705+0000","last_clean":"2024-09-13T16:26:38.432705+0000","last_became_active":"2024-09-13T16:26:37.055734+0000","last_became_peered":"2024-09-13T16:26:37.055734+0000","last_unstale":"2024-09-13T16:26:38.432705+0000","last_undegraded":"2024-09-13T16:26:38.432705+0000","last_fullsized":"2024-09-13T16:26:38.432705+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:16:23.231125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292812+0000","last_change":"2024-09-13T16:26:37.443421+0000","last_active":"2024-09-13T16:26:39.292812+0000","last_peered":"2024-09-13T16:26:39.292812+0000","last_clean":"2024-09-13T16:26:39.292812+0000","last_became_active":"2024-09-13T16:26:37.443141+0000","last_became_peered":"2024-09-13T16:26:37.443141+0000","last_unstale":"2024-09-13T16:26:39.292812+0000","last_undegraded":"2024-09-13T16:26:39.292812+0000","last_fullsized":"2024-09-13T16:26:39.292812+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:11:51.495216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"39'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432381+0000","last_change":"2024-09-13T16:26:37.056457+0000","last_active":"2024-09-13T16:26:38.432381+0000","last_peered":"2024-09-13T16:26:38.432381+0000","last_clean":"2024-09-13T16:26:38.432381+0000","last_became_active":"2024-09-13T16:26:37.055463+0000","last_became_peered":"2024-09-13T16:26:37.055463+0000","last_unstale":"2024-09-13T16:26:38.432381+0000","last_undegraded":"2024-09-13T16:26:38.432381+0000","last_fullsized":"2024-09-13T16:26:38.432381+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:52.834682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432149+0000","last_change":"2024-09-13T16:26:37.436802+0000","last_active":"2024-09-13T16:26:38.432149+0000","last_peered":"2024-09-13T16:26:38.432149+0000","last_clean":"2024-09-13T16:26:38.432149+0000","last_became_active":"2024-09-13T16:26:37.436575+0000","last_became_peered":"2024-09-13T16:26:37.436575+0000","last_unstale":"2024-09-13T16:26:38.432149+0000","last_undegraded":"2024-09-13T16:26:38.432149+0000","last_fullsized":"2024-09-13T16:26:38.432149+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:35.167646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253614+0000","last_change":"2024-09-13T16:26:37.051917+0000","last_active":"2024-09-13T16:26:39.253614+0000","last_peered":"2024-09-13T16:26:39.253614+0000","last_clean":"2024-09-13T16:26:39.253614+0000","last_became_active":"2024-09-13T16:26:37.051661+0000","last_became_peered":"2024-09-13T16:26:37.051661+0000","last_unstale":"2024-09-13T16:26:39.253614+0000","last_undegraded":"2024-09-13T16:26:39.253614+0000","last_fullsized":"2024-09-13T16:26:39.253614+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:53:23.950820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292757+0000","last_change":"2024-09-13T16:26:37.445399+0000","last_active":"2024-09-13T16:26:39.292757+0000","last_peered":"2024-09-13T16:26:39.292757+0000","last_clean":"2024-09-13T16:26:39.292757+0000","last_became_active":"2024-09-13T16:26:37.445140+0000","last_became_peered":"2024-09-13T16:26:37.445140+0000","last_unstale":"2024-09-13T16:26:39.292757+0000","last_undegraded":"2024-09-13T16:26:39.292757+0000","last_fullsized":"2024-09-13T16:26:39.292757+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:25.314603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.436556+0000","last_change":"2024-09-13T16:26:37.055815+0000","last_active":"2024-09-13T16:26:38.436556+0000","last_peered":"2024-09-13T16:26:38.436556+0000","last_clean":"2024-09-13T16:26:38.436556+0000","last_became_active":"2024-09-13T16:26:37.055531+0000","last_became_peered":"2024-09-13T16:26:37.055531+0000","last_unstale":"2024-09-13T16:26:38.436556+0000","last_undegraded":"2024-09-13T16:26:38.436556+0000","last_fullsized":"2024-09-13T16:26:38.436556+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:34.039058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432304+0000","last_change":"2024-09-13T16:26:37.440994+0000","last_active":"2024-09-13T16:26:38.432304+0000","last_peered":"2024-09-13T16:26:38.432304+0000","last_clean":"2024-09-13T16:26:38.432304+0000","last_became_active":"2024-09-13T16:26:37.440739+0000","last_became_peered":"2024-09-13T16:26:37.440739+0000","last_unstale":"2024-09-13T16:26:38.432304+0000","last_undegraded":"2024-09-13T16:26:38.432304+0000","last_fullsized":"2024-09-13T16:26:38.432304+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:46:11.921862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"39'8","reported_seq":19,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292601+0000","last_change":"2024-09-13T16:26:37.061966+0000","last_active":"2024-09-13T16:26:39.292601+0000","last_peered":"2024-09-13T16:26:39.292601+0000","last_clean":"2024-09-13T16:26:39.292601+0000","last_became_active":"2024-09-13T16:26:37.061726+0000","last_became_peered":"2024-09-13T16:26:37.061726+0000","last_unstale":"2024-09-13T16:26:39.292601+0000","last_undegraded":"2024-09-13T16:26:39.292601+0000","last_fullsized":"2024-09-13T16:26:39.292601+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:01:59.222571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432488+0000","last_change":"2024-09-13T16:26:37.457390+0000","last_active":"2024-09-13T16:26:38.432488+0000","last_peered":"2024-09-13T16:26:38.432488+0000","last_clean":"2024-09-13T16:26:38.432488+0000","last_became_active":"2024-09-13T16:26:37.457153+0000","last_became_peered":"2024-09-13T16:26:37.457153+0000","last_unstale":"2024-09-13T16:26:38.432488+0000","last_undegraded":"2024-09-13T16:26:38.432488+0000","last_fullsized":"2024-09-13T16:26:38.432488+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:45.553240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"39'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438471+0000","last_change":"2024-09-13T16:26:37.061532+0000","last_active":"2024-09-13T16:26:38.438471+0000","last_peered":"2024-09-13T16:26:38.438471+0000","last_clean":"2024-09-13T16:26:38.438471+0000","last_became_active":"2024-09-13T16:26:37.060950+0000","last_became_peered":"2024-09-13T16:26:37.060950+0000","last_unstale":"2024-09-13T16:26:38.438471+0000","last_undegraded":"2024-09-13T16:26:38.438471+0000","last_fullsized":"2024-09-13T16:26:38.438471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:43.305841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438035+0000","last_change":"2024-09-13T16:26:37.448485+0000","last_active":"2024-09-13T16:26:38.438035+0000","last_peered":"2024-09-13T16:26:38.438035+0000","last_clean":"2024-09-13T16:26:38.438035+0000","last_became_active":"2024-09-13T16:26:37.448196+0000","last_became_peered":"2024-09-13T16:26:37.448196+0000","last_unstale":"2024-09-13T16:26:38.438035+0000","last_undegraded":"2024-09-13T16:26:38.438035+0000","last_fullsized":"2024-09-13T16:26:38.438035+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:03:58.227648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"39'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.294411+0000","last_change":"2024-09-13T16:26:37.059556+0000","last_active":"2024-09-13T16:26:39.294411+0000","last_peered":"2024-09-13T16:26:39.294411+0000","last_clean":"2024-09-13T16:26:39.294411+0000","last_became_active":"2024-09-13T16:26:37.056216+0000","last_became_peered":"2024-09-13T16:26:37.056216+0000","last_unstale":"2024-09-13T16:26:39.294411+0000","last_undegraded":"2024-09-13T16:26:39.294411+0000","last_fullsized":"2024-09-13T16:26:39.294411+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:43:01.696217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.294177+0000","last_change":"2024-09-13T16:26:37.450827+0000","last_active":"2024-09-13T16:26:39.294177+0000","last_peered":"2024-09-13T16:26:39.294177+0000","last_clean":"2024-09-13T16:26:39.294177+0000","last_became_active":"2024-09-13T16:26:37.448735+0000","last_became_peered":"2024-09-13T16:26:37.448735+0000","last_unstale":"2024-09-13T16:26:39.294177+0000","last_undegraded":"2024-09-13T16:26:39.294177+0000","last_fullsized":"2024-09-13T16:26:39.294177+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:03.704173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"39'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292469+0000","last_change":"2024-09-13T16:26:37.049882+0000","last_active":"2024-09-13T16:26:39.292469+0000","last_peered":"2024-09-13T16:26:39.292469+0000","last_clean":"2024-09-13T16:26:39.292469+0000","last_became_active":"2024-09-13T16:26:37.049602+0000","last_became_peered":"2024-09-13T16:26:37.049602+0000","last_unstale":"2024-09-13T16:26:39.292469+0000","last_undegraded":"2024-09-13T16:26:39.292469+0000","last_fullsized":"2024-09-13T16:26:39.292469+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:09:38.855528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292415+0000","last_change":"2024-09-13T16:26:37.443375+0000","last_active":"2024-09-13T16:26:39.292415+0000","last_peered":"2024-09-13T16:26:39.292415+0000","last_clean":"2024-09-13T16:26:39.292415+0000","last_became_active":"2024-09-13T16:26:37.442686+0000","last_became_peered":"2024-09-13T16:26:37.442686+0000","last_unstale":"2024-09-13T16:26:39.292415+0000","last_undegraded":"2024-09-13T16:26:39.292415+0000","last_fullsized":"2024-09-13T16:26:39.292415+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:44.319379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253842+0000","last_change":"2024-09-13T16:26:37.055533+0000","last_active":"2024-09-13T16:26:39.253842+0000","last_peered":"2024-09-13T16:26:39.253842+0000","last_clean":"2024-09-13T16:26:39.253842+0000","last_became_active":"2024-09-13T16:26:37.054655+0000","last_became_peered":"2024-09-13T16:26:37.054655+0000","last_unstale":"2024-09-13T16:26:39.253842+0000","last_undegraded":"2024-09-13T16:26:39.253842+0000","last_fullsized":"2024-09-13T16:26:39.253842+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:47:27.493477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292718+0000","last_change":"2024-09-13T16:26:37.444292+0000","last_active":"2024-09-13T16:26:39.292718+0000","last_peered":"2024-09-13T16:26:39.292718+0000","last_clean":"2024-09-13T16:26:39.292718+0000","last_became_active":"2024-09-13T16:26:37.443780+0000","last_became_peered":"2024-09-13T16:26:37.443780+0000","last_unstale":"2024-09-13T16:26:39.292718+0000","last_undegraded":"2024-09-13T16:26:39.292718+0000","last_fullsized":"2024-09-13T16:26:39.292718+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:22.300657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"39'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292136+0000","last_change":"2024-09-13T16:26:37.048313+0000","last_active":"2024-09-13T16:26:39.292136+0000","last_peered":"2024-09-13T16:26:39.292136+0000","last_clean":"2024-09-13T16:26:39.292136+0000","last_became_active":"2024-09-13T16:26:37.048012+0000","last_became_peered":"2024-09-13T16:26:37.048012+0000","last_unstale":"2024-09-13T16:26:39.292136+0000","last_undegraded":"2024-09-13T16:26:39.292136+0000","last_fullsized":"2024-09-13T16:26:39.292136+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:16.857680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293033+0000","last_change":"2024-09-13T16:26:37.440225+0000","last_active":"2024-09-13T16:26:39.293033+0000","last_peered":"2024-09-13T16:26:39.293033+0000","last_clean":"2024-09-13T16:26:39.293033+0000","last_became_active":"2024-09-13T16:26:37.439850+0000","last_became_peered":"2024-09-13T16:26:37.439850+0000","last_unstale":"2024-09-13T16:26:39.293033+0000","last_undegraded":"2024-09-13T16:26:39.293033+0000","last_fullsized":"2024-09-13T16:26:39.293033+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:00.015227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292246+0000","last_change":"2024-09-13T16:26:37.051440+0000","last_active":"2024-09-13T16:26:39.292246+0000","last_peered":"2024-09-13T16:26:39.292246+0000","last_clean":"2024-09-13T16:26:39.292246+0000","last_became_active":"2024-09-13T16:26:37.051164+0000","last_became_peered":"2024-09-13T16:26:37.051164+0000","last_unstale":"2024-09-13T16:26:39.292246+0000","last_undegraded":"2024-09-13T16:26:39.292246+0000","last_fullsized":"2024-09-13T16:26:39.292246+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:05:13.095822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.436950+0000","last_change":"2024-09-13T16:26:37.458618+0000","last_active":"2024-09-13T16:26:38.436950+0000","last_peered":"2024-09-13T16:26:38.436950+0000","last_clean":"2024-09-13T16:26:38.436950+0000","last_became_active":"2024-09-13T16:26:37.458303+0000","last_became_peered":"2024-09-13T16:26:37.458303+0000","last_unstale":"2024-09-13T16:26:38.436950+0000","last_undegraded":"2024-09-13T16:26:38.436950+0000","last_fullsized":"2024-09-13T16:26:38.436950+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:26.587739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292788+0000","last_change":"2024-09-13T16:26:37.056538+0000","last_active":"2024-09-13T16:26:39.292788+0000","last_peered":"2024-09-13T16:26:39.292788+0000","last_clean":"2024-09-13T16:26:39.292788+0000","last_became_active":"2024-09-13T16:26:37.056175+0000","last_became_peered":"2024-09-13T16:26:37.056175+0000","last_unstale":"2024-09-13T16:26:39.292788+0000","last_undegraded":"2024-09-13T16:26:39.292788+0000","last_fullsized":"2024-09-13T16:26:39.292788+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:13:34.085289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432322+0000","last_change":"2024-09-13T16:26:37.438784+0000","last_active":"2024-09-13T16:26:38.432322+0000","last_peered":"2024-09-13T16:26:38.432322+0000","last_clean":"2024-09-13T16:26:38.432322+0000","last_became_active":"2024-09-13T16:26:37.438342+0000","last_became_peered":"2024-09-13T16:26:37.438342+0000","last_unstale":"2024-09-13T16:26:38.432322+0000","last_undegraded":"2024-09-13T16:26:38.432322+0000","last_fullsized":"2024-09-13T16:26:38.432322+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:57.619453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.291758+0000","last_change":"2024-09-13T16:26:37.046849+0000","last_active":"2024-09-13T16:26:39.291758+0000","last_peered":"2024-09-13T16:26:39.291758+0000","last_clean":"2024-09-13T16:26:39.291758+0000","last_became_active":"2024-09-13T16:26:37.046593+0000","last_became_peered":"2024-09-13T16:26:37.046593+0000","last_unstale":"2024-09-13T16:26:39.291758+0000","last_undegraded":"2024-09-13T16:26:39.291758+0000","last_fullsized":"2024-09-13T16:26:39.291758+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:17:12.203526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253980+0000","last_change":"2024-09-13T16:26:37.444153+0000","last_active":"2024-09-13T16:26:39.253980+0000","last_peered":"2024-09-13T16:26:39.253980+0000","last_clean":"2024-09-13T16:26:39.253980+0000","last_became_active":"2024-09-13T16:26:37.443875+0000","last_became_peered":"2024-09-13T16:26:37.443875+0000","last_unstale":"2024-09-13T16:26:39.253980+0000","last_undegraded":"2024-09-13T16:26:39.253980+0000","last_fullsized":"2024-09-13T16:26:39.253980+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:03.704173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"39'2","reported_seq":13,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438536+0000","last_change":"2024-09-13T16:26:37.054812+0000","last_active":"2024-09-13T16:26:38.438536+0000","last_peered":"2024-09-13T16:26:38.438536+0000","last_clean":"2024-09-13T16:26:38.438536+0000","last_became_active":"2024-09-13T16:26:37.054532+0000","last_became_peered":"2024-09-13T16:26:37.054532+0000","last_unstale":"2024-09-13T16:26:38.438536+0000","last_undegraded":"2024-09-13T16:26:38.438536+0000","last_fullsized":"2024-09-13T16:26:38.438536+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:54.769069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293791+0000","last_change":"2024-09-13T16:26:37.457298+0000","last_active":"2024-09-13T16:26:39.293791+0000","last_peered":"2024-09-13T16:26:39.293791+0000","last_clean":"2024-09-13T16:26:39.293791+0000","last_became_active":"2024-09-13T16:26:37.457007+0000","last_became_peered":"2024-09-13T16:26:37.457007+0000","last_unstale":"2024-09-13T16:26:39.293791+0000","last_undegraded":"2024-09-13T16:26:39.293791+0000","last_fullsized":"2024-09-13T16:26:39.293791+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:03.704173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.6","version":"40'1","reported_seq":12,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:40.121343+0000","last_change":"2024-09-13T16:26:37.049289+0000","last_active":"2024-09-13T16:26:40.121343+0000","last_peered":"2024-09-13T16:26:40.121343+0000","last_clean":"2024-09-13T16:26:40.121343+0000","last_became_active":"2024-09-13T16:26:37.049069+0000","last_became_peered":"2024-09-13T16:26:37.049069+0000","last_unstale":"2024-09-13T16:26:40.121343+0000","last_undegraded":"2024-09-13T16:26:40.121343+0000","last_fullsized":"2024-09-13T16:26:40.121343+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:11.315691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.432808+0000","last_change":"2024-09-13T16:26:37.452579+0000","last_active":"2024-09-13T16:26:38.432808+0000","last_peered":"2024-09-13T16:26:38.432808+0000","last_clean":"2024-09-13T16:26:38.432808+0000","last_became_active":"2024-09-13T16:26:37.452351+0000","last_became_peered":"2024-09-13T16:26:37.452351+0000","last_unstale":"2024-09-13T16:26:38.432808+0000","last_undegraded":"2024-09-13T16:26:38.432808+0000","last_fullsized":"2024-09-13T16:26:38.432808+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:26:57.767864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293537+0000","last_change":"2024-09-13T16:26:37.052277+0000","last_active":"2024-09-13T16:26:39.293537+0000","last_peered":"2024-09-13T16:26:39.293537+0000","last_clean":"2024-09-13T16:26:39.293537+0000","last_became_active":"2024-09-13T16:26:37.051993+0000","last_became_peered":"2024-09-13T16:26:37.051993+0000","last_unstale":"2024-09-13T16:26:39.293537+0000","last_undegraded":"2024-09-13T16:26:39.293537+0000","last_fullsized":"2024-09-13T16:26:39.293537+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:28:54.809113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.292860+0000","last_change":"2024-09-13T16:26:37.439659+0000","last_active":"2024-09-13T16:26:39.292860+0000","last_peered":"2024-09-13T16:26:39.292860+0000","last_clean":"2024-09-13T16:26:39.292860+0000","last_became_active":"2024-09-13T16:26:37.439401+0000","last_became_peered":"2024-09-13T16:26:37.439401+0000","last_unstale":"2024-09-13T16:26:39.292860+0000","last_undegraded":"2024-09-13T16:26:39.292860+0000","last_fullsized":"2024-09-13T16:26:39.292860+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:39.782793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293471+0000","last_change":"2024-09-13T16:26:37.054179+0000","last_active":"2024-09-13T16:26:39.293471+0000","last_peered":"2024-09-13T16:26:39.293471+0000","last_clean":"2024-09-13T16:26:39.293471+0000","last_became_active":"2024-09-13T16:26:37.053768+0000","last_became_peered":"2024-09-13T16:26:37.053768+0000","last_unstale":"2024-09-13T16:26:39.293471+0000","last_undegraded":"2024-09-13T16:26:39.293471+0000","last_fullsized":"2024-09-13T16:26:39.293471+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:57.232681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.253511+0000","last_change":"2024-09-13T16:26:37.457279+0000","last_active":"2024-09-13T16:26:39.253511+0000","last_peered":"2024-09-13T16:26:39.253511+0000","last_clean":"2024-09-13T16:26:39.253511+0000","last_became_active":"2024-09-13T16:26:37.456977+0000","last_became_peered":"2024-09-13T16:26:37.456977+0000","last_unstale":"2024-09-13T16:26:39.253511+0000","last_undegraded":"2024-09-13T16:26:39.253511+0000","last_fullsized":"2024-09-13T16:26:39.253511+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:44.930362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.436793+0000","last_change":"2024-09-13T16:26:37.050861+0000","last_active":"2024-09-13T16:26:38.436793+0000","last_peered":"2024-09-13T16:26:38.436793+0000","last_clean":"2024-09-13T16:26:38.436793+0000","last_became_active":"2024-09-13T16:26:37.050527+0000","last_became_peered":"2024-09-13T16:26:37.050527+0000","last_unstale":"2024-09-13T16:26:38.436793+0000","last_undegraded":"2024-09-13T16:26:38.436793+0000","last_fullsized":"2024-09-13T16:26:38.436793+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:57:27.191544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293556+0000","last_change":"2024-09-13T16:26:37.458200+0000","last_active":"2024-09-13T16:26:39.293556+0000","last_peered":"2024-09-13T16:26:39.293556+0000","last_clean":"2024-09-13T16:26:39.293556+0000","last_became_active":"2024-09-13T16:26:37.457946+0000","last_became_peered":"2024-09-13T16:26:37.457946+0000","last_unstale":"2024-09-13T16:26:39.293556+0000","last_undegraded":"2024-09-13T16:26:39.293556+0000","last_fullsized":"2024-09-13T16:26:39.293556+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:42:53.848220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293175+0000","last_change":"2024-09-13T16:26:37.061130+0000","last_active":"2024-09-13T16:26:39.293175+0000","last_peered":"2024-09-13T16:26:39.293175+0000","last_clean":"2024-09-13T16:26:39.293175+0000","last_became_active":"2024-09-13T16:26:37.060841+0000","last_became_peered":"2024-09-13T16:26:37.060841+0000","last_unstale":"2024-09-13T16:26:39.293175+0000","last_undegraded":"2024-09-13T16:26:39.293175+0000","last_fullsized":"2024-09-13T16:26:39.293175+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:24.301065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293404+0000","last_change":"2024-09-13T16:26:37.434642+0000","last_active":"2024-09-13T16:26:39.293404+0000","last_peered":"2024-09-13T16:26:39.293404+0000","last_clean":"2024-09-13T16:26:39.293404+0000","last_became_active":"2024-09-13T16:26:37.433972+0000","last_became_peered":"2024-09-13T16:26:37.433972+0000","last_unstale":"2024-09-13T16:26:39.293404+0000","last_undegraded":"2024-09-13T16:26:39.293404+0000","last_fullsized":"2024-09-13T16:26:39.293404+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:44.930362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.431895+0000","last_change":"2024-09-13T16:26:37.051037+0000","last_active":"2024-09-13T16:26:38.431895+0000","last_peered":"2024-09-13T16:26:38.431895+0000","last_clean":"2024-09-13T16:26:38.431895+0000","last_became_active":"2024-09-13T16:26:37.050786+0000","last_became_peered":"2024-09-13T16:26:37.050786+0000","last_unstale":"2024-09-13T16:26:38.431895+0000","last_undegraded":"2024-09-13T16:26:38.431895+0000","last_fullsized":"2024-09-13T16:26:38.431895+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:50.829175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":10,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:38.438368+0000","last_change":"2024-09-13T16:26:37.440233+0000","last_active":"2024-09-13T16:26:38.438368+0000","last_peered":"2024-09-13T16:26:38.438368+0000","last_clean":"2024-09-13T16:26:38.438368+0000","last_became_active":"2024-09-13T16:26:37.439980+0000","last_became_peered":"2024-09-13T16:26:37.439980+0000","last_unstale":"2024-09-13T16:26:38.438368+0000","last_undegraded":"2024-09-13T16:26:38.438368+0000","last_fullsized":"2024-09-13T16:26:38.438368+0000","mapping_epoch":38,"log_start":"0'0","ondisk_log_start":"0'0","created":38,"last_epoch_clean":39,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:37.027026+0000","last_clean_scrub_stamp":"2024-09-13T16:26:37.027026+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:16.949221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-09-13T16:26:39.293321+0000","last_change":"2024-09-13T16:26:37.060058+0000","last_active":"2024-09-13T16:26:39.293321+0000","last_peered":"2024-09-13T16:26:39.293321+0000","last_clean":"2024-09-13T16:26:39.293321+0000","last_became_active":"2024-09-13T16:26:37.059745+0000","last_became_peered":"2024-09-13T16:26:37.059745+0000","last_unstale":"2024-09-13T16:26:39.293321+0000","last_undegraded":"2024-09-13T16:26:39.293321+0000","last_fullsized":"2024-09-13T16:26:39.293321+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":37,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:26:36.013488+0000","last_clean_scrub_stamp":"2024-09-13T16:26:36.013488+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:46:27.084823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":178,"reported_epoch":39,"state":"active+clean","last_fresh":"2024-09-13T16:26:37.431928+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:26:37.431928+0000","last_peered":"2024-09-13T16:26:37.431928+0000","last_clean":"2024-09-13T16:26:37.431928+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:26:37.431928+0000","last_undegraded":"2024-09-13T16:26:37.431928+0000","last_fullsized":"2024-09-13T16:26:37.431928+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542165,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1069056,"data_stored":789322,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.98699999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.121,"5min":1.121,"15min":1.121},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.228,"5min":1.228,"15min":1.228},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.83099999999999996,"15min":0.83099999999999996},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.88800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.02,"5min":1.02,"15min":1.02},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.238,"5min":1.238,"15min":1.238},"last":1.123},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":1.2190000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.103,"5min":1.103,"15min":1.103},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":1.589,"5min":1.589,"15min":1.589},"last":1.0640000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":1.0469999999999999},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.0940000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"min":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.2649999999999999,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.194,"5min":1.194,"15min":1.194},"min":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.169}]}]},{"osd":6,"up_from":14,"seq":60129542164,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":462848,"data_stored":197902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.153,"5min":1.153,"15min":1.153},"last":1.0660000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.143,"5min":1.143,"15min":1.143},"last":0.91600000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":1.2030000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.91500000000000004},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.296,"5min":1.296,"15min":1.296},"last":0.86799999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.2609999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"last":1.1870000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.96899999999999997,"15min":0.96899999999999997},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.536,"5min":1.536,"15min":1.536},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.99099999999999999,"5min":0.99099999999999999,"15min":0.99099999999999999},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.409,"5min":1.409,"15min":1.409},"last":1.036}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.1639999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.224,"5min":1.224,"15min":1.224},"last":1.107}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91400000000000003,"15min":0.91400000000000003},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":1.6439999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.234}]},{"osd":7,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.125,"5min":1.125,"15min":1.125},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.5029999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.2869999999999999},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0589999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":1.337}]}]},{"osd":5,"up_from":14,"seq":60129542166,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27484,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344356,"statfs":{"total":96636764160,"available":96608620544,"internally_reserved":0,"allocated":466944,"data_stored":197432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.675,"5min":1.675,"15min":1.675},"last":0.874}]},{"osd":1,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.369,"5min":1.369,"15min":1.369},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1830000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.79600000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.625,"5min":1.625,"15min":1.625},"last":1.034},{"interface":"front","average":{"1min":0.995,"5min":0.995,"15min":0.995},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.552,"5min":1.552,"15min":1.552},"last":0.70099999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.0409999999999999,"5min":1.0409999999999999,"15min":1.0409999999999999},"last":1.0089999999999999},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0389999999999999,"15min":1.0389999999999999},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":3.8460000000000001,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":0.89400000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":3.6869999999999998,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.601,"5min":1.601,"15min":1.601},"last":1.0580000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.97999999999999998,"15min":0.97999999999999998},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.421,"5min":1.421,"15min":1.421},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0800000000000001,"15min":1.0800000000000001},"min":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":1.5469999999999999,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":1.0780000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:26:10 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"max":{"1min":3.9830000000000001,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":0.745}]}]},{"osd":4,"up_from":14,"seq":60129542164,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27476,"kb_used_data":456,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344364,"statfs":{"total":96636764160,"available":96608628736,"internally_reserved":0,"allocated":466944,"data_stored":197924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1339999999999999,"5min":1.1339999999999999,"15min":1.1339999999999999},"last":1.04}]},{"osd":1,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"min":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.93799999999999994},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.353,"5min":1.353,"15min":1.353},"last":0.76000000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.181,"5min":1.181,"15min":1.181},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.498,"5min":0.498,"15min":0.498},"max":{"1min":1.603,"5min":1.603,"15min":1.603},"last":0.61299999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"min":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.472,"5min":1.472,"15min":1.472},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"min":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.425,"5min":1.425,"15min":1.425},"last":0.79200000000000004}]},{"osd":5,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.048,"5min":2.048,"15min":2.048},"last":0.86499999999999999},{"interface":"front","average":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.1539999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.8860000000000001,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.83699999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.343,"5min":1.343,"15min":1.343},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1020000000000001,"15min":1.1020000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":2.1890000000000001,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":0.99099999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.661,"5min":1.661,"15min":1.661},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.119,"5min":1.119,"15min":1.119},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.3779999999999999,"5min":1.3779999999999999,"15min":1.3779999999999999},"last":1.3200000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542165,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27496,"kb_used_data":464,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344344,"statfs":{"total":96636764160,"available":96608608256,"internally_reserved":0,"allocated":475136,"data_stored":198918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.53500000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.88400000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.751,"5min":0.751,"15min":0.751},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.95199999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.625,"5min":0.625,"15min":0.625},"max":{"1min":1.782,"5min":1.782,"15min":1.782},"last":0.90700000000000003}]},{"osd":5,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"min":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.8140000000000001,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.1419999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.1459999999999999,"15min":1.1459999999999999},"min":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.885,"5min":1.885,"15min":1.885},"last":1.194},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.95,"5min":1.95,"15min":1.95},"last":0.81200000000000006}]},{"osd":7,"last update":"Fri Sep 13 16:26:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":1.01}]}]},{"osd":2,"up_from":14,"seq":60129542166,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27480,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":462848,"data_stored":197390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.63}]},{"osd":1,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.123,"5min":1.123,"15min":1.123},"last":0.80700000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.073,"5min":1.073,"15min":1.073},"last":0.83699999999999997},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":1.2070000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.90500000000000003},{"interface":"front","average":{"1min":0.998,"5min":0.998,"15min":0.998},"min":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1020000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"min":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.4470000000000001,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.96999999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.0569999999999999,"15min":1.0569999999999999},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.024,"5min":2.024,"15min":2.024},"last":1},{"interface":"front","average":{"1min":1.071,"5min":1.071,"15min":1.071},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.6359999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.129}]},{"osd":7,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.089,"5min":1.089,"15min":1.089},"min":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.55,"5min":1.55,"15min":1.55},"last":1.0529999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542165,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27472,"kb_used_data":452,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344368,"statfs":{"total":96636764160,"available":96608632832,"internally_reserved":0,"allocated":462848,"data_stored":197410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.628},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.77100000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1,"5min":1,"15min":1},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.458,"5min":1.458,"15min":1.458},"last":0.85899999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":1.0249999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":1.403,"5min":1.403,"15min":1.403},"last":0.92600000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.546,"5min":1.546,"15min":1.546},"last":1.1160000000000001},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":2.2040000000000002,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":1.0609999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3440000000000001,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.3089999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.98799999999999999,"15min":0.98799999999999999},"min":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.2250000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.008,"15min":1.008},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.613,"5min":1.613,"15min":1.613},"last":1.258},{"interface":"front","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.698,"5min":1.698,"15min":1.698},"last":1.2}]},{"osd":7,"last update":"Fri Sep 13 16:26:11 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.012,"15min":1.012},"min":{"1min":0.62,"5min":0.62,"15min":0.62},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.151},{"interface":"front","average":{"1min":1.036,"5min":1.036,"15min":1.036},"min":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.3640000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542165,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28060,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343780,"statfs":{"total":96636764160,"available":96608030720,"internally_reserved":0,"allocated":1056768,"data_stored":787722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.94799999999999995,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.64400000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.877}]},{"osd":3,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"last":0.502},{"interface":"front","average":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.0980000000000001},"last":0.79700000000000004}]},{"osd":4,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.3109999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.77100000000000002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95299999999999996},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.478,"5min":1.478,"15min":1.478},"last":1.0680000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"max":{"1min":1.24,"5min":1.24,"15min":1.24},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.002,"5min":1.002,"15min":1.002},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.93300000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"max":{"1min":1.3009999999999999,"5min":1.3009999999999999,"15min":1.3009999999999999},"last":1.0289999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:26:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.0429999999999999},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.026,"15min":1.026},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":1.0069999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:26:41.819 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:26:41.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T16:26:41.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac21b2700 1 Processor -- start 2024-09-13T16:26:41.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac21b2700 1 -- start start 2024-09-13T16:26:41.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac21b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc15cad0 0x7feabc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac21b2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feabc0ce0d0 0x7feabc0ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac21b2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0d05b0 con 0x7feabc0cf220 2024-09-13T16:26:41.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac21b2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feabc0d0870 con 0x7feabc0ce0d0 2024-09-13T16:26:41.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac21b2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0d0b50 con 0x7feabc15cad0 2024-09-13T16:26:41.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac19b1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7feabc0cf220 0x7feabc0b5810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46078/0 (socket says 172.21.15.119:46078) 2024-09-13T16:26:41.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac09af700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feabc0ce0d0 0x7feabc0ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac19b1700 1 -- 172.21.15.119:0/978669977 learned_addr learned my addr 172.21.15.119:0/978669977 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:41.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac11b0700 1 --2- 172.21.15.119:0/978669977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc15cad0 0x7feabc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac09af700 1 -- 172.21.15.119:0/978669977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc15cad0 msgr2=0x7feabc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac09af700 1 --2- 172.21.15.119:0/978669977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc15cad0 0x7feabc0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac09af700 1 -- 172.21.15.119:0/978669977 >> v1:172.21.15.119:6789/0 conn(0x7feabc0cf220 legacy=0x7feabc0b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feac09af700 1 -- 172.21.15.119:0/978669977 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc0d0e30 con 0x7feabc0ce0d0 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.944+0000 7feab3fff700 1 -- 172.21.15.119:0/978669977 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496034276 0 0) 0x7feabc0d05b0 con 0x7feabc0cf220 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac09af700 1 --2- 172.21.15.119:0/978669977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0ce0d0 0x7feabc0ce510 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7feab8009850 tx=0x7feab8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31c1484c49e00a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feab3fff700 1 -- 172.21.15.119:0/978669977 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8011830 con 0x7feabc0ce0d0 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feab3fff700 1 -- 172.21.15.119:0/978669977 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab8011990 con 0x7feabc0ce0d0 2024-09-13T16:26:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feab3fff700 1 -- 172.21.15.119:0/978669977 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab8011d40 con 0x7feabc0ce0d0 2024-09-13T16:26:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 -- 172.21.15.119:0/978669977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0ce0d0 msgr2=0x7feabc0ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 --2- 172.21.15.119:0/978669977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0ce0d0 0x7feabc0ce510 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7feab8009850 tx=0x7feab8009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 -- 172.21.15.119:0/978669977 shutdown_connections 2024-09-13T16:26:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 --2- 172.21.15.119:0/978669977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0ce0d0 0x7feabc0ce510 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 --2- 172.21.15.119:0/978669977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc15cad0 0x7feabc0d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 -- 172.21.15.119:0/978669977 >> 172.21.15.119:0/978669977 conn(0x7feabc08bf70 msgr2=0x7feabc0d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 -- 172.21.15.119:0/978669977 shutdown_connections 2024-09-13T16:26:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 -- 172.21.15.119:0/978669977 wait complete. 2024-09-13T16:26:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 Processor -- start 2024-09-13T16:26:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.948+0000 7feac21b2700 1 -- start start 2024-09-13T16:26:41.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc0ce0d0 0x7feabc0cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0cf220 0x7feabc1586c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feabc15cad0 0x7feabc1f24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c9ab0 con 0x7feabc0cf220 2024-09-13T16:26:41.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc158c00 con 0x7feabc15cad0 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc1f2d40 con 0x7feabc0ce0d0 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0cf220 0x7feabc1586c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0cf220 0x7feabc1586c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50490/0 (socket says 172.21.15.119:50490) 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 -- 172.21.15.119:0/3254863976 learned_addr learned my addr 172.21.15.119:0/3254863976 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac11b0700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc0ce0d0 0x7feabc0cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac19b1700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feabc15cad0 0x7feabc1f24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 -- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc0ce0d0 msgr2=0x7feabc0cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc0ce0d0 0x7feabc0cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 -- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feabc15cad0 msgr2=0x7feabc1f24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feabc15cad0 0x7feabc1f24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:41.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 -- 172.21.15.119:0/3254863976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab80094b0 con 0x7feabc0cf220 2024-09-13T16:26:41.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac09af700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0cf220 0x7feabc1586c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7feab8000c00 tx=0x7feab8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feab1ffb700 1 -- 172.21.15.119:0/3254863976 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab801aac0 con 0x7feabc0cf220 2024-09-13T16:26:41.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feab1ffb700 1 -- 172.21.15.119:0/3254863976 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab801f070 con 0x7feabc0cf220 2024-09-13T16:26:41.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc1f3020 con 0x7feabc0cf220 2024-09-13T16:26:41.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feab1ffb700 1 -- 172.21.15.119:0/3254863976 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab801ae60 con 0x7feabc0cf220 2024-09-13T16:26:41.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.952+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc0c9cc0 con 0x7feabc0cf220 2024-09-13T16:26:41.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.956+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc0c9eb0 con 0x7feabc0cf220 2024-09-13T16:26:41.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.956+0000 7feab1ffb700 1 -- 172.21.15.119:0/3254863976 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7feab802e450 con 0x7feabc0cf220 2024-09-13T16:26:41.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.956+0000 7feab1ffb700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feaa0066730 0x7feaa0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:41.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.956+0000 7feab1ffb700 1 -- 172.21.15.119:0/3254863976 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feab8085400 con 0x7feabc0cf220 2024-09-13T16:26:41.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.956+0000 7feac11b0700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feaa0066730 0x7feaa0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:41.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.956+0000 7feac11b0700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feaa0066730 0x7feaa0068bf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feaac009ae0 tx=0x7feaac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:41.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:41.960+0000 7feab1ffb700 1 -- 172.21.15.119:0/3254863976 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7feab8052620 con 0x7feabc0cf220 2024-09-13T16:26:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.188+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7feabc0cfae0 con 0x7feabc0cf220 2024-09-13T16:26:42.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.196+0000 7feab1ffb700 1 -- 172.21.15.119:0/3254863976 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) v1 ==== 190+0+0 (secure 0 0 0) 0x7feab80210e0 con 0x7feabc0cf220 2024-09-13T16:26:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feaa0066730 msgr2=0x7feaa0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feaa0066730 0x7feaa0068bf0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feaac009ae0 tx=0x7feaac008040 comp rx=0 tx=0).stop 2024-09-13T16:26:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0cf220 msgr2=0x7feabc1586c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0cf220 0x7feabc1586c0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7feab8000c00 tx=0x7feab8004770 comp rx=0 tx=0).stop 2024-09-13T16:26:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 shutdown_connections 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feaa0066730 0x7feaa0068bf0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feabc15cad0 0x7feabc1f24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feabc0cf220 0x7feabc1586c0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 --2- 172.21.15.119:0/3254863976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feabc0ce0d0 0x7feabc0cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 >> 172.21.15.119:0/3254863976 conn(0x7feabc08bf70 msgr2=0x7feabc0b8030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 shutdown_connections 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.200+0000 7feac21b2700 1 -- 172.21.15.119:0/3254863976 wait complete. 2024-09-13T16:26:42.206 INFO:teuthology.orchestra.run.smithi119.stderr:updated caps for client.0 2024-09-13T16:26:42.218 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625e464700 1 Processor -- start 2024-09-13T16:26:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625e464700 1 -- start start 2024-09-13T16:26:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625e464700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f625806fd40 0x7f6258070190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625e464700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580ac960 con 0x7f62581612f0 2024-09-13T16:26:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625e464700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580acc40 con 0x7f625815a620 2024-09-13T16:26:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625e464700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62580706d0 con 0x7f625806fd40 2024-09-13T16:26:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625ca01700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f625806fd40 0x7f6258070190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f62577fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f62581612f0 0x7f62580ada00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46092/0 (socket says 172.21.15.119:46092) 2024-09-13T16:26:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f62577fe700 1 -- 172.21.15.119:0/3519665100 learned_addr learned my addr 172.21.15.119:0/3519665100 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625ca01700 1 -- 172.21.15.119:0/3519665100 >> v1:172.21.15.145:6789/0 conn(0x7f625815a620 legacy=0x7f62580b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625ca01700 1 -- 172.21.15.119:0/3519665100 >> v1:172.21.15.119:6789/0 conn(0x7f62581612f0 legacy=0x7f62580ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f6256ffd700 1 -- 172.21.15.119:0/3519665100 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526924676 0 0) 0x7f62580ac960 con 0x7f62581612f0 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.324+0000 7f625ca01700 1 -- 172.21.15.119:0/3519665100 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580acf20 con 0x7f625806fd40 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625ca01700 1 --2- 172.21.15.119:0/3519665100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f6258070190 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6250009850 tx=0x7f6250009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2511b90fc8d3164d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f6256ffd700 1 -- 172.21.15.119:0/3519665100 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250011830 con 0x7f625806fd40 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f6256ffd700 1 -- 172.21.15.119:0/3519665100 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6250011990 con 0x7f625806fd40 2024-09-13T16:26:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f6256ffd700 1 -- 172.21.15.119:0/3519665100 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6250011d20 con 0x7f625806fd40 2024-09-13T16:26:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 -- 172.21.15.119:0/3519665100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 msgr2=0x7f6258070190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 --2- 172.21.15.119:0/3519665100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f6258070190 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6250009850 tx=0x7f6250009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 -- 172.21.15.119:0/3519665100 shutdown_connections 2024-09-13T16:26:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 --2- 172.21.15.119:0/3519665100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f6258070190 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 -- 172.21.15.119:0/3519665100 >> 172.21.15.119:0/3519665100 conn(0x7f625806a350 msgr2=0x7f625806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 -- 172.21.15.119:0/3519665100 shutdown_connections 2024-09-13T16:26:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 -- 172.21.15.119:0/3519665100 wait complete. 2024-09-13T16:26:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 Processor -- start 2024-09-13T16:26:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.328+0000 7f625e464700 1 -- start start 2024-09-13T16:26:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f625813b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625815a620 0x7f625813acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62581612f0 0x7f62580cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62580cc260 con 0x7f625806fd40 2024-09-13T16:26:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62580cc540 con 0x7f625815a620 2024-09-13T16:26:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62580cc820 con 0x7f62581612f0 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f625813b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f625813b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50512/0 (socket says 172.21.15.119:50512) 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 -- 172.21.15.119:0/1893420348 learned_addr learned my addr 172.21.15.119:0/1893420348 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f62577fe700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625815a620 0x7f625813acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625ca01700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62581612f0 0x7f62580cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 -- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62581612f0 msgr2=0x7f62580cd300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62581612f0 0x7f62580cd300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 -- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625815a620 msgr2=0x7f625813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625815a620 0x7f625813acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 -- 172.21.15.119:0/1893420348 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62500094b0 con 0x7f625806fd40 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6257fff700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f625813b2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f624800c500 tx=0x7f624800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6254ff9700 1 -- 172.21.15.119:0/1893420348 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62480052d0 con 0x7f625806fd40 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6254ff9700 1 -- 172.21.15.119:0/1893420348 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6248005430 con 0x7f625806fd40 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62580ccb00 con 0x7f625806fd40 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f6254ff9700 1 -- 172.21.15.119:0/1893420348 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62480055a0 con 0x7f625806fd40 2024-09-13T16:26:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.332+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62580cd840 con 0x7f625806fd40 2024-09-13T16:26:42.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.336+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62580cd020 con 0x7f625806fd40 2024-09-13T16:26:42.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.336+0000 7f6254ff9700 1 -- 172.21.15.119:0/1893420348 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f62480038e0 con 0x7f625806fd40 2024-09-13T16:26:42.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.336+0000 7f6254ff9700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6238066730 0x7f6238068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:42.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.336+0000 7f6254ff9700 1 -- 172.21.15.119:0/1893420348 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f62480835d0 con 0x7f625806fd40 2024-09-13T16:26:42.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.336+0000 7f62577fe700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6238066730 0x7f6238068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:42.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.336+0000 7f62577fe700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6238066730 0x7f6238068bf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6240001cf0 tx=0x7f6240001c80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:42.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.348+0000 7f6254ff9700 1 -- 172.21.15.119:0/1893420348 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6248050890 con 0x7f625806fd40 2024-09-13T16:26:42.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.556+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62580cda00 con 0x7f625806fd40 2024-09-13T16:26:42.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.556+0000 7f6254ff9700 1 -- 172.21.15.119:0/1893420348 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f6248053eb0 con 0x7f625806fd40 2024-09-13T16:26:42.562 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:42.562 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:26:42.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6238066730 msgr2=0x7f6238068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6238066730 0x7f6238068bf0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f6240001cf0 tx=0x7f6240001c80 comp rx=0 tx=0).stop 2024-09-13T16:26:42.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 msgr2=0x7f625813b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:42.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f625813b2f0 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f624800c500 tx=0x7f624800c810 comp rx=0 tx=0).stop 2024-09-13T16:26:42.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 shutdown_connections 2024-09-13T16:26:42.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6238066730 0x7f6238068bf0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62581612f0 0x7f62580cd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f625815a620 0x7f625813acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 --2- 172.21.15.119:0/1893420348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f625806fd40 0x7f625813b2f0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:42.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 >> 172.21.15.119:0/1893420348 conn(0x7f625806a350 msgr2=0x7f6258066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:42.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 shutdown_connections 2024-09-13T16:26:42.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:42.560+0000 7f625e464700 1 -- 172.21.15.119:0/1893420348 wait complete. 2024-09-13T16:26:42.566 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:26:42.979 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:26:37.416714+0000', 'modified': '2024-09-13T16:26:40.432024+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5586}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5586': {'gid': 5586, 'name': 'b', 'rank': 0, 'incarnation': 13, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/2305196370', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6835', 'nonce': 2305196370}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 2305196370}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:26:42.980 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:26:42.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-13T16:26:43.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.100+0000 7fabc495f700 1 Processor -- start 2024-09-13T16:26:43.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.100+0000 7fabc495f700 1 -- start start 2024-09-13T16:26:43.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fabc015cc40 0x7fabc00ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fabc00cd860 0x7fabc008bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabc015d020 con 0x7fabc008c4e0 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc008c990 con 0x7fabc015cc40 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabc00b44c0 con 0x7fabc00cd860 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fabc008c4e0 0x7fabc0160720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54072/0 (socket says 172.21.15.119:54072) 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbf7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fabc015cc40 0x7fabc00ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbffff700 1 -- 172.21.15.119:0/72531907 learned_addr learned my addr 172.21.15.119:0/72531907 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:43.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbeffd700 1 --2- 172.21.15.119:0/72531907 >> v2:172.21.15.145:3300/0 conn(0x7fabc00cd860 0x7fabc008bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbf7fe700 1 -- 172.21.15.119:0/72531907 >> v2:172.21.15.145:3300/0 conn(0x7fabc00cd860 msgr2=0x7fabc008bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/72531907 >> v2:172.21.15.145:3300/0 conn(0x7fabc00cd860 0x7fabc008bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbf7fe700 1 -- 172.21.15.119:0/72531907 >> v1:172.21.15.119:6789/0 conn(0x7fabc008c4e0 legacy=0x7fabc0160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbe7fc700 1 -- 172.21.15.119:0/72531907 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339699903 0 0) 0x7fabc015d020 con 0x7fabc008c4e0 2024-09-13T16:26:43.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbf7fe700 1 -- 172.21.15.119:0/72531907 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc00d0720 con 0x7fabc015cc40 2024-09-13T16:26:43.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/72531907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc015cc40 0x7fabc00ce710 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009880 tx=0x7fabb4009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f61d42e01db1c356 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbe7fc700 1 -- 172.21.15.119:0/72531907 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4011830 con 0x7fabc015cc40 2024-09-13T16:26:43.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbe7fc700 1 -- 172.21.15.119:0/72531907 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4011990 con 0x7fabc015cc40 2024-09-13T16:26:43.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabbe7fc700 1 -- 172.21.15.119:0/72531907 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4011d20 con 0x7fabc015cc40 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- 172.21.15.119:0/72531907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc015cc40 msgr2=0x7fabc00ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 --2- 172.21.15.119:0/72531907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc015cc40 0x7fabc00ce710 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009880 tx=0x7fabb4009b90 comp rx=0 tx=0).stop 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- 172.21.15.119:0/72531907 shutdown_connections 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 --2- 172.21.15.119:0/72531907 >> v2:172.21.15.145:3300/0 conn(0x7fabc00cd860 0x7fabc008bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 --2- 172.21.15.119:0/72531907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc015cc40 0x7fabc00ce710 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- 172.21.15.119:0/72531907 >> 172.21.15.119:0/72531907 conn(0x7fabc00b4640 msgr2=0x7fabc00b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- 172.21.15.119:0/72531907 shutdown_connections 2024-09-13T16:26:43.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.104+0000 7fabc495f700 1 -- 172.21.15.119:0/72531907 wait complete. 2024-09-13T16:26:43.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 Processor -- start 2024-09-13T16:26:43.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 -- start start 2024-09-13T16:26:43.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc008c4e0 0x7fabc00b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabc00cd860 0x7fabc00b8500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabc015cc40 0x7fabc0159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc01567b0 con 0x7fabc008c4e0 2024-09-13T16:26:43.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabc00b8070 con 0x7fabc00cd860 2024-09-13T16:26:43.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabc00b8a40 con 0x7fabc015cc40 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc008c4e0 0x7fabc00b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc008c4e0 0x7fabc00b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34912/0 (socket says 172.21.15.119:34912) 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3010991054 learned_addr learned my addr 172.21.15.119:0/3010991054 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbffff700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabc015cc40 0x7fabc0159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbeffd700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabc00cd860 0x7fabc00b8500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabc015cc40 msgr2=0x7fabc0159570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabc015cc40 0x7fabc0159570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabc00cd860 msgr2=0x7fabc00b8500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabc00cd860 0x7fabc00b8500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3010991054 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb40094b0 con 0x7fabc008c4e0 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbffff700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabc015cc40 0x7fabc0159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbeffd700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabc00cd860 0x7fabc00b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:43.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc008c4e0 0x7fabc00b5980 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009bd0 tx=0x7fabb4004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7faba7fff700 1 -- 172.21.15.119:0/3010991054 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40036c0 con 0x7fabc008c4e0 2024-09-13T16:26:43.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7faba7fff700 1 -- 172.21.15.119:0/3010991054 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4003820 con 0x7fabc008c4e0 2024-09-13T16:26:43.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabc00b7d40 con 0x7fabc008c4e0 2024-09-13T16:26:43.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.108+0000 7faba7fff700 1 -- 172.21.15.119:0/3010991054 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4003b90 con 0x7fabc008c4e0 2024-09-13T16:26:43.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.112+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabc00b6340 con 0x7fabc008c4e0 2024-09-13T16:26:43.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.112+0000 7faba7fff700 1 -- 172.21.15.119:0/3010991054 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fabb4003cf0 con 0x7fabc008c4e0 2024-09-13T16:26:43.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.112+0000 7faba7fff700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fabac066730 0x7fabac068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.112+0000 7fabc8e72740 1 Processor -- start 2024-09-13T16:26:43.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.112+0000 7faba7fff700 1 -- 172.21.15.119:0/3010991054 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fabb4085920 con 0x7fabc008c4e0 2024-09-13T16:26:43.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- start start 2024-09-13T16:26:43.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbeffd700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fabac066730 0x7fabac068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbeffd700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fabac066730 0x7fabac068bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fabc01e8e90 tx=0x7fabb0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4d230 0x1f4d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f15890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4db50 0x1f15dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e74d80 con 0x1f4db50 2024-09-13T16:26:43.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e72510 con 0x1f2be90 2024-09-13T16:26:43.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e6de00 con 0x1f4d230 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4db50 0x1f15dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4db50 0x1f15dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34916/0 (socket says 172.21.15.119:34916) 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 -- 172.21.15.119:0/727614087 learned_addr learned my addr 172.21.15.119:0/727614087 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbeffd700 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f15890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4d230 0x1f4d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 -- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4d230 msgr2=0x1f4d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4d230 0x1f4d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 -- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 msgr2=0x1f15890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f15890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 -- 172.21.15.119:0/727614087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1ce0960 con 0x1f4db50 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4d230 0x1f4d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabbffff700 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4db50 0x1f15dd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fabb80021c0 tx=0x7fabb80021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=600a1180e9e20bf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7faba57fa700 1 -- 172.21.15.119:0/727614087 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8014480 con 0x1f4db50 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7faba57fa700 1 -- 172.21.15.119:0/727614087 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb80145e0 con 0x1f4db50 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7faba57fa700 1 -- 172.21.15.119:0/727614087 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb8014950 con 0x1f4db50 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- 172.21.15.119:0/727614087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4db50 msgr2=0x1f15dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4db50 0x1f15dd0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7fabb80021c0 tx=0x7fabb80021f0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- 172.21.15.119:0/727614087 shutdown_connections 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4db50 0x1f15dd0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f15890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 --2- 172.21.15.119:0/727614087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4d230 0x1f4d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- 172.21.15.119:0/727614087 >> 172.21.15.119:0/727614087 conn(0x1c954d0 msgr2=0x1f2ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- 172.21.15.119:0/727614087 shutdown_connections 2024-09-13T16:26:43.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.116+0000 7fabc8e72740 1 -- 172.21.15.119:0/727614087 wait complete. 2024-09-13T16:26:43.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 -- start start 2024-09-13T16:26:43.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 Processor -- start 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f18030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1f05e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f456f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e2b730 con 0x1f2be90 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e315b0 con 0x1f4d230 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d19aa0 con 0x1f4db50 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f18030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 10 client.0 _ll_register_callbacks cb 0x1e67610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f18030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34924/0 (socket says 172.21.15.119:34924) 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 -- 172.21.15.119:0/652399285 learned_addr learned my addr 172.21.15.119:0/652399285 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbeffd700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1f05e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbffff700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f456f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 -- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 msgr2=0x1f456f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f456f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 -- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 msgr2=0x1f05e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1f05e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 -- 172.21.15.119:0/652399285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabb800f040 con 0x1f2be90 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbffff700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbeffd700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1f05e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f18030 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fabb401e040 tx=0x7fabb40568b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=879768d12c69960a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fab9e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fab9e7fc700 1 -- 172.21.15.119:0/652399285 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4055cf0 con 0x1f2be90 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fab9e7fc700 1 -- 172.21.15.119:0/652399285 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fabb406be30 con 0x1f2be90 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1ca21e0 con 0x1f2be90 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fab9e7fc700 1 -- 172.21.15.119:0/652399285 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabb4055e50 con 0x1f2be90 2024-09-13T16:26:43.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 10 client.5775 fetch_fsmap learned FSMap version 16 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x1e34220 con 0x1f2be90 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fab9e7fc700 1 -- 172.21.15.119:0/652399285 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1ca21e0 con 0x1f2be90 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fab9e7fc700 1 -- 172.21.15.119:0/652399285 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fabb405e760 con 0x1f2be90 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 10 client.5775 fetch_fsmap finished waiting for FSMap version 16 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 10 client.5775 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 20 client.5775 populate_metadata read hostname 'smithi119' 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1f04ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 4 client.5775 mds_command: new command op to 5586 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.120+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 --> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1cec460 con 0x1f537a0 2024-09-13T16:26:43.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabbeffd700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1f04ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fab9e7fc700 10 client.5775 ms_handle_connect on v2:172.21.15.145:6835/2305196370 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabbeffd700 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1f04ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af2071263949e7cf server_cookie=a86f55b4290c9bfe in_seq=0 out_seq=0 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fab9e7fc700 1 -- 172.21.15.119:0/652399285 <== mds.0 v2:172.21.15.145:6835/2305196370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1cec460 con 0x1f537a0 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fab9e7fc700 10 client.5775 handle_command_reply: tid=0 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 client.5775 shutdown 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 msgr2=0x1f18030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f18030 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fabb401e040 tx=0x7fabb40568b0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 shutdown_connections 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1f04ab0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f456f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fab9e7fc700 0 client.5775 ms_handle_reset on v2:172.21.15.145:6835/2305196370 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1f05e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 --2- 172.21.15.119:0/652399285 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f18030 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 >> 172.21.15.119:0/652399285 conn(0x1c954d0 msgr2=0x1f46230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 shutdown_connections 2024-09-13T16:26:43.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 1 -- 172.21.15.119:0/652399285 wait complete. 2024-09-13T16:26:43.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.124+0000 7fabc8e72740 20 client.5775 trim_cache size 0 max 16384 2024-09-13T16:26:43.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 Processor -- start 2024-09-13T16:26:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 -- start start 2024-09-13T16:26:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f053d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4d230 0x1f4ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f50480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e9cec0 con 0x1f4d230 2024-09-13T16:26:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e27c30 con 0x1f2be90 2024-09-13T16:26:43.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e35530 con 0x1f4db50 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabbeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4d230 0x1f4ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabbf7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f053d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabbeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4d230 0x1f4ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34928/0 (socket says 172.21.15.119:34928) 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabbeffd700 1 -- 172.21.15.119:0/3355488366 learned_addr learned my addr 172.21.15.119:0/3355488366 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.148+0000 7fabbffff700 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f50480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 msgr2=0x1f50480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f50480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3355488366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4d230 msgr2=0x1f4ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4d230 0x1f4ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3355488366 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1e1a450 con 0x1f2be90 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f053d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabb4052d00 tx=0x7fabb4027000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae3a3cc829f2a9b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7faba5ffb700 1 -- 172.21.15.119:0/3355488366 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4052b40 con 0x1f2be90 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7faba5ffb700 1 -- 172.21.15.119:0/3355488366 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb405e760 con 0x1f2be90 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7faba5ffb700 1 -- 172.21.15.119:0/3355488366 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4055ef0 con 0x1f2be90 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 msgr2=0x1f053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f053d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fabb4052d00 tx=0x7fabb4027000 comp rx=0 tx=0).stop 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- 172.21.15.119:0/3355488366 shutdown_connections 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1f50480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f4d230 0x1f4ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3355488366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f2be90 0x1f053d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- 172.21.15.119:0/3355488366 >> 172.21.15.119:0/3355488366 conn(0x1c954d0 msgr2=0x1f04ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- 172.21.15.119:0/3355488366 shutdown_connections 2024-09-13T16:26:43.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- 172.21.15.119:0/3355488366 wait complete. 2024-09-13T16:26:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- start start 2024-09-13T16:26:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 Processor -- start 2024-09-13T16:26:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f6aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1fad860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1faeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e3f240 con 0x1f2be90 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f03be0 con 0x1f4d230 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f060d0 con 0x1f4db50 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabc8e72740 10 client.0 _ll_register_callbacks cb 0x1e1fce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f6aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f6aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34938/0 (socket says 172.21.15.119:34938) 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3915787024 learned_addr learned my addr 172.21.15.119:0/3915787024 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbffff700 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1faeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbeffd700 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1fad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 msgr2=0x1faeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1faeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 msgr2=0x1fad860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1fad860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.152+0000 7fabbf7fe700 1 -- 172.21.15.119:0/3915787024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1e34220 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabbf7fe700 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f6aa80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fabb4056790 tx=0x7fabb4083bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183349323d485a93 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 1 -- 172.21.15.119:0/3915787024 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb405e8d0 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fabb80145e0 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 1 -- 172.21.15.119:0/3915787024 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fabb405ee40 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 1 -- 172.21.15.119:0/3915787024 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7fabb4052b40 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 10 client.5781 fetch_fsmap learned FSMap version 16 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) v3 -- 0x1e0e720 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 1 -- 172.21.15.119:0/3915787024 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb80145e0 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 1 -- 172.21.15.119:0/3915787024 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 16) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fabb4052800 con 0x1f2be90 2024-09-13T16:26:43.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 10 client.5781 fetch_fsmap finished waiting for FSMap version 16 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 10 client.5781 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 20 client.5781 populate_metadata read hostname 'smithi119' 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1faf090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 4 client.5781 mds_command: new command op to 5586 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 --> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1d111e0 con 0x1f537a0 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabbeffd700 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1faf090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 10 client.5781 ms_handle_connect on v2:172.21.15.145:6835/2305196370 2024-09-13T16:26:43.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabbeffd700 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1faf090 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=570aa6691c3d353b server_cookie=19c47be1fe30d169 in_seq=0 out_seq=0 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 1 -- 172.21.15.119:0/3915787024 <== mds.0 v2:172.21.15.145:6835/2305196370 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+317 (crc 0 0 0) 0x1d111e0 con 0x1f537a0 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 10 client.5781 handle_command_reply: tid=0 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 client.5781 shutdown 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 msgr2=0x1f6aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f6aa80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fabb4056790 tx=0x7fabb4083bd0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 shutdown_connections 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:6835/2305196370,v1:172.21.15.145:6837/2305196370] conn(0x1f537a0 0x1faf090 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f4db50 0x1faeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f4d230 0x1fad860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fab9e7fc700 0 client.5781 ms_handle_reset on v2:172.21.15.145:6835/2305196370 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 --2- 172.21.15.119:0/3915787024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f2be90 0x1f6aa80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 >> 172.21.15.119:0/3915787024 conn(0x1c954d0 msgr2=0x1f052a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 shutdown_connections 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 1 -- 172.21.15.119:0/3915787024 wait complete. 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.156+0000 7fabc8e72740 20 client.5781 trim_cache size 0 max 16384 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 5586, 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:26:43.167 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 5.741500984, 2024-09-13T16:26:43.168 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 16, 2024-09-13T16:26:43.168 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 40, 2024-09-13T16:26:43.168 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 40, 2024-09-13T16:26:43.168 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 13.04677414 2024-09-13T16:26:43.168 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fabac066730 msgr2=0x7fabac068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fabac066730 0x7fabac068bf0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fabc01e8e90 tx=0x7fabb0008040 comp rx=0 tx=0).stop 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc008c4e0 msgr2=0x7fabc00b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc008c4e0 0x7fabc00b5980 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fabb4009bd0 tx=0x7fabb4004770 comp rx=0 tx=0).stop 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 shutdown_connections 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fabac066730 0x7fabac068bf0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fabc015cc40 0x7fabc0159570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fabc00cd860 0x7fabc00b8500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 --2- 172.21.15.119:0/3010991054 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fabc008c4e0 0x7fabc00b5980 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 >> 172.21.15.119:0/3010991054 conn(0x7fabc00b4640 msgr2=0x7fabc00afdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 shutdown_connections 2024-09-13T16:26:43.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.160+0000 7fabc495f700 1 -- 172.21.15.119:0/3010991054 wait complete. 2024-09-13T16:26:43.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:26:43.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f188ada6700 1 Processor -- start 2024-09-13T16:26:43.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f188ada6700 1 -- start start 2024-09-13T16:26:43.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f188ada6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f188415ce70 0x7f18840d37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f188ada6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f18841607c0 0x7f1884160c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f188ada6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18840d1190 con 0x7f18840d1620 2024-09-13T16:26:43.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f188ada6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1884161190 con 0x7f18841607c0 2024-09-13T16:26:43.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f188ada6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18840ceaf0 con 0x7f188415ce70 2024-09-13T16:26:43.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1883fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f18840d1620 0x7f18840ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54078/0 (socket says 172.21.15.119:54078) 2024-09-13T16:26:43.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1883fff700 1 -- 172.21.15.119:0/1587033642 learned_addr learned my addr 172.21.15.119:0/1587033642 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:43.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1889343700 1 --2- 172.21.15.119:0/1587033642 >> v2:172.21.15.119:3300/0 conn(0x7f18841607c0 0x7f1884160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1888b42700 1 --2- 172.21.15.119:0/1587033642 >> v2:172.21.15.145:3300/0 conn(0x7f188415ce70 0x7f18840d37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f18837fe700 1 -- 172.21.15.119:0/1587033642 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173806226 0 0) 0x7f18840d1190 con 0x7f18840d1620 2024-09-13T16:26:43.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f18837fe700 1 -- 172.21.15.119:0/1587033642 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18700035d0 con 0x7f18840d1620 2024-09-13T16:26:43.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1889343700 1 -- 172.21.15.119:0/1587033642 >> v2:172.21.15.145:3300/0 conn(0x7f188415ce70 msgr2=0x7f18840d37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:43.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1889343700 1 --2- 172.21.15.119:0/1587033642 >> v2:172.21.15.145:3300/0 conn(0x7f188415ce70 0x7f18840d37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f18837fe700 1 -- 172.21.15.119:0/1587033642 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1738401600 0 0) 0x7f18700035d0 con 0x7f18840d1620 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1889343700 1 -- 172.21.15.119:0/1587033642 >> v1:172.21.15.119:6789/0 conn(0x7f18840d1620 legacy=0x7f18840ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1889343700 1 -- 172.21.15.119:0/1587033642 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18840d0860 con 0x7f18841607c0 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1888b42700 1 --2- 172.21.15.119:0/1587033642 >> v2:172.21.15.145:3300/0 conn(0x7f188415ce70 0x7f18840d37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f1889343700 1 --2- 172.21.15.119:0/1587033642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18841607c0 0x7f1884160c50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f18840b56f0 tx=0x7f1874009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=758bbfd1fd8fa985 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f18837fe700 1 -- 172.21.15.119:0/1587033642 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874011830 con 0x7f18841607c0 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f18837fe700 1 -- 172.21.15.119:0/1587033642 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1874011990 con 0x7f18841607c0 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.292+0000 7f18837fe700 1 -- 172.21.15.119:0/1587033642 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1874011d00 con 0x7f18841607c0 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- 172.21.15.119:0/1587033642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18841607c0 msgr2=0x7f1884160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 --2- 172.21.15.119:0/1587033642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18841607c0 0x7f1884160c50 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f18840b56f0 tx=0x7f1874009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- 172.21.15.119:0/1587033642 shutdown_connections 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 --2- 172.21.15.119:0/1587033642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18841607c0 0x7f1884160c50 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 --2- 172.21.15.119:0/1587033642 >> v2:172.21.15.145:3300/0 conn(0x7f188415ce70 0x7f18840d37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- 172.21.15.119:0/1587033642 >> 172.21.15.119:0/1587033642 conn(0x7f18840d2bf0 msgr2=0x7f18840d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- 172.21.15.119:0/1587033642 shutdown_connections 2024-09-13T16:26:43.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- 172.21.15.119:0/1587033642 wait complete. 2024-09-13T16:26:43.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 Processor -- start 2024-09-13T16:26:43.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- start start 2024-09-13T16:26:43.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18840d1620 0x7f18840ca740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f188415ce70 0x7f188414e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18841607c0 0x7f18840b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18840cb5d0 con 0x7f18840d1620 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18840b7390 con 0x7f18841607c0 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.296+0000 7f188ada6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18840c6720 con 0x7f188415ce70 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18840d1620 0x7f18840ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18840d1620 0x7f18840ca740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34970/0 (socket says 172.21.15.119:34970) 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 -- 172.21.15.119:0/2332400198 learned_addr learned my addr 172.21.15.119:0/2332400198 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1883fff700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f188415ce70 0x7f188414e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1889343700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18841607c0 0x7f18840b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 -- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f188415ce70 msgr2=0x7f188414e3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f188415ce70 0x7f188414e3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 -- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18841607c0 msgr2=0x7f18840b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18841607c0 0x7f18840b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 -- 172.21.15.119:0/2332400198 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f186c0094b0 con 0x7f18840d1620 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f1888b42700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18840d1620 0x7f18840ca740 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f186c00ebe0 tx=0x7f186c00eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f18817fa700 1 -- 172.21.15.119:0/2332400198 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c00c850 con 0x7f18840d1620 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f18817fa700 1 -- 172.21.15.119:0/2332400198 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f186c00c9b0 con 0x7f18840d1620 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18740094b0 con 0x7f18840d1620 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f18817fa700 1 -- 172.21.15.119:0/2332400198 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f186c00cd40 con 0x7f18840d1620 2024-09-13T16:26:43.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18840b70b0 con 0x7f18840d1620 2024-09-13T16:26:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.300+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18840d37a0 con 0x7f18840d1620 2024-09-13T16:26:43.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.304+0000 7f18817fa700 1 -- 172.21.15.119:0/2332400198 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f186c013470 con 0x7f18840d1620 2024-09-13T16:26:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.304+0000 7f18817fa700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1864066730 0x7f1864068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.304+0000 7f18817fa700 1 -- 172.21.15.119:0/2332400198 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f186c084390 con 0x7f18840d1620 2024-09-13T16:26:43.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.304+0000 7f1883fff700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1864066730 0x7f1864068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:43.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.304+0000 7f1883fff700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1864066730 0x7f1864068bf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f18841f5ba0 tx=0x7f187c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:43.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.312+0000 7f18817fa700 1 -- 172.21.15.119:0/2332400198 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f186c051540 con 0x7f18840d1620 2024-09-13T16:26:43.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.512+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f1884072730 con 0x7f18840d1620 2024-09-13T16:26:43.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.512+0000 7f18817fa700 1 -- 172.21.15.119:0/2332400198 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+73 (secure 0 0 0) 0x7f186c00cea0 con 0x7f18840d1620 2024-09-13T16:26:43.518 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:26:43.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1864066730 msgr2=0x7f1864068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1864066730 0x7f1864068bf0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f18841f5ba0 tx=0x7f187c009040 comp rx=0 tx=0).stop 2024-09-13T16:26:43.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18840d1620 msgr2=0x7f18840ca740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:43.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18840d1620 0x7f18840ca740 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f186c00ebe0 tx=0x7f186c00eef0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 shutdown_connections 2024-09-13T16:26:43.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1864066730 0x7f1864068bf0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18841607c0 0x7f18840b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f188415ce70 0x7f188414e3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 --2- 172.21.15.119:0/2332400198 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18840d1620 0x7f18840ca740 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:43.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 >> 172.21.15.119:0/2332400198 conn(0x7f18840d2bf0 msgr2=0x7f188415bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:43.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 shutdown_connections 2024-09-13T16:26:43.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:43.516+0000 7f188ada6700 1 -- 172.21.15.119:0/2332400198 wait complete. 2024-09-13T16:26:43.537 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:26:43.537 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:26:43.537 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:26:43.538 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:26:43.538 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:43.538 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:26:43.538 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:26:43.538 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:26:43.538 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:26:43.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.538 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:26:43.545 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42537sec preferred_lft 42537sec 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:26:43.546 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:26:43.547 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:26:43.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.548 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:26:43.604 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:26:43.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.606 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:26:43.614 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:26:43.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.615 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:26:43.616 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:26:43.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:26:43.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:26:43.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:26:43.616 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:26:43.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.656 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:26:43.706 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:26:43.707 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:26:43.707 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:26:43.707 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.707 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:26:43.724 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:26:43.725 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:26:43.725 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:43.731 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:26:43.731 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:26:43.732 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:43.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:26:43.753 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:26:43.753 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:26:43.765 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:26:43.766 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:26:43.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.767 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:26:43.773 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:26:43.773 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:26:43.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:26:43.834 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:26:43.834 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:26:43.843 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:26:43.844 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:26:43.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:43.845 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:26:43.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:26:43.908+0000 7fdb1a826640 -1 init, newargv = 0x55a76b82e8c0 newargc=15 2024-09-13T16:26:43.913 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[38522]: starting ceph client 2024-09-13T16:26:43.922 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[38522]: starting fuse 2024-09-13T16:26:44.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:26:44.970 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:26:44.971 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:26:44.980 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:26:44.981 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:26:44.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:44.982 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:26:45.031 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:26:45.032 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:26:45.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:26:45.032 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:26:45.032 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:26:45.032 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:26:45.032 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:26:45.032 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:45.032 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:26:45.033 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:26:45.034 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:26:45.034 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:26:45.034 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:26:45.034 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:45.034 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:26:45.034 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:26:45.118 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.38522.asok 2024-09-13T16:26:45.302 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.38522.asok 2024-09-13T16:26:45.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.302 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.38522.asok status 2024-09-13T16:26:45.408 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:26:45.408 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:26:45.408 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:26:45.408 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:26:45.408 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "38522", 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 5796, 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:26:45.409 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 5796 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 48266912 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:26:45.410 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 48266912 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.5796 192.168.0.1:0/48266912", 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/48266912", 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 16, 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 40, 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:26:45.411 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:26:45.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.419 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:26:45.419 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:45.425 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:26:45.425 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:45.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.425 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:45.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.441 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir) 2024-09-13T16:26:45.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.458 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parentdir/childdir) 2024-09-13T16:26:45.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.470 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch rfile) 2024-09-13T16:26:45.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.484 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/pfile) 2024-09-13T16:26:45.503 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.503 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parentdir/childdir/cfile) 2024-09-13T16:26:45.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:45.561 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-09-13T16:26:45.594 INFO:teuthology.orchestra.run.smithi119.stdout:./ 2024-09-13T16:26:45.594 INFO:teuthology.orchestra.run.smithi119.stdout:./rfile 2024-09-13T16:26:45.594 INFO:teuthology.orchestra.run.smithi119.stdout:./parentdir 2024-09-13T16:26:45.594 INFO:teuthology.orchestra.run.smithi119.stdout:./parentdir/pfile 2024-09-13T16:26:45.594 INFO:teuthology.orchestra.run.smithi119.stdout:./parentdir/childdir 2024-09-13T16:26:45.594 INFO:teuthology.orchestra.run.smithi119.stdout:./parentdir/childdir/cfile 2024-09-13T16:26:45.595 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:26:45.596 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:26:45.596 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:26:45.596 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:45.596 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-09-13T16:26:45.596 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:26:45.664 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:26:45.855 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:26:45.855 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:26:45.855 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:26:45.855 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:45.855 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./rfile").st_ino) 2024-09-13T16:26:45.855 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:26:45.925 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627778 2024-09-13T16:26:46.115 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:26:46.115 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:26:46.115 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:26:46.115 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:46.115 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir").st_ino) 2024-09-13T16:26:46.115 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:26:46.230 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627776 2024-09-13T16:26:46.416 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:26:46.417 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:26:46.417 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:26:46.417 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:46.417 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/pfile").st_ino) 2024-09-13T16:26:46.417 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:26:46.482 INFO:teuthology.orchestra.run.smithi119.stdout:1099511628279 2024-09-13T16:26:46.674 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:26:46.674 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:26:46.674 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:26:46.674 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:46.674 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir").st_ino) 2024-09-13T16:26:46.674 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:26:46.740 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627777 2024-09-13T16:26:46.929 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:26:46.930 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:26:46.930 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:26:46.930 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:26:46.930 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parentdir/childdir/cfile").st_ino) 2024-09-13T16:26:46.930 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:26:46.996 INFO:teuthology.orchestra.run.smithi119.stdout:1099511628280 2024-09-13T16:26:47.188 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:47.188 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:47.194 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:47.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.352+0000 7ff84f59e700 1 Processor -- start 2024-09-13T16:26:47.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.352+0000 7ff84f59e700 1 -- start start 2024-09-13T16:26:47.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.352+0000 7ff84f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8500abf40 0x7ff85006cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.352+0000 7ff84f59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff8500721f0 0x7ff85006dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.352+0000 7ff84f59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff850072680 con 0x7ff850158fd0 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff850071d50 con 0x7ff8500721f0 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85006d350 con 0x7ff8500abf40 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84dd9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff850158fd0 0x7ff8500ac320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54088/0 (socket says 172.21.15.119:54088) 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84dd9b700 1 -- 172.21.15.119:0/905107192 learned_addr learned my addr 172.21.15.119:0/905107192 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84e59c700 1 --2- 172.21.15.119:0/905107192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8500abf40 0x7ff85006cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84ed9d700 1 --2- 172.21.15.119:0/905107192 >> v2:172.21.15.145:3300/0 conn(0x7ff8500721f0 0x7ff85006dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84d59a700 1 -- 172.21.15.119:0/905107192 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667628444 0 0) 0x7ff850072680 con 0x7ff850158fd0 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84d59a700 1 -- 172.21.15.119:0/905107192 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8380035d0 con 0x7ff850158fd0 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84e59c700 1 -- 172.21.15.119:0/905107192 >> v2:172.21.15.145:3300/0 conn(0x7ff8500721f0 msgr2=0x7ff85006dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84e59c700 1 --2- 172.21.15.119:0/905107192 >> v2:172.21.15.145:3300/0 conn(0x7ff8500721f0 0x7ff85006dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84e59c700 1 -- 172.21.15.119:0/905107192 >> v1:172.21.15.119:6789/0 conn(0x7ff850158fd0 legacy=0x7ff8500ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84d59a700 1 -- 172.21.15.119:0/905107192 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1824996695 0 0) 0x7ff8380035d0 con 0x7ff850158fd0 2024-09-13T16:26:47.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84e59c700 1 -- 172.21.15.119:0/905107192 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8500acec0 con 0x7ff8500abf40 2024-09-13T16:26:47.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84ed9d700 1 --2- 172.21.15.119:0/905107192 >> v2:172.21.15.145:3300/0 conn(0x7ff8500721f0 0x7ff85006dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:47.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84e59c700 1 --2- 172.21.15.119:0/905107192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8500abf40 0x7ff85006cd40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff840009850 tx=0x7ff840009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64647c54f0ad1ecb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84d59a700 1 -- 172.21.15.119:0/905107192 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840011830 con 0x7ff8500abf40 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84d59a700 1 -- 172.21.15.119:0/905107192 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff840011990 con 0x7ff8500abf40 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84d59a700 1 -- 172.21.15.119:0/905107192 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840011d00 con 0x7ff8500abf40 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 -- 172.21.15.119:0/905107192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8500abf40 msgr2=0x7ff85006cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 --2- 172.21.15.119:0/905107192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8500abf40 0x7ff85006cd40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ff840009850 tx=0x7ff840009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 -- 172.21.15.119:0/905107192 shutdown_connections 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 --2- 172.21.15.119:0/905107192 >> v2:172.21.15.145:3300/0 conn(0x7ff8500721f0 0x7ff85006dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 --2- 172.21.15.119:0/905107192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff8500abf40 0x7ff85006cd40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 -- 172.21.15.119:0/905107192 >> 172.21.15.119:0/905107192 conn(0x7ff8500ae750 msgr2=0x7ff8500b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:47.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 -- 172.21.15.119:0/905107192 shutdown_connections 2024-09-13T16:26:47.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.356+0000 7ff84f59e700 1 -- 172.21.15.119:0/905107192 wait complete. 2024-09-13T16:26:47.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 Processor -- start 2024-09-13T16:26:47.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 -- start start 2024-09-13T16:26:47.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8500abf40 0x7ff850145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff850158fd0 0x7ff85015b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff850147d90 0x7ff85015bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff850099bd0 con 0x7ff8500abf40 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8500b40b0 con 0x7ff850158fd0 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff850099e90 con 0x7ff850147d90 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8500abf40 0x7ff850145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8500abf40 0x7ff850145090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34986/0 (socket says 172.21.15.119:34986) 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 -- 172.21.15.119:0/1929664099 learned_addr learned my addr 172.21.15.119:0/1929664099 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84dd9b700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff850158fd0 0x7ff85015b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84ed9d700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff850147d90 0x7ff85015bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 -- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff850147d90 msgr2=0x7ff85015bb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff850147d90 0x7ff85015bb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 -- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff850158fd0 msgr2=0x7ff85015b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff850158fd0 0x7ff85015b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 -- 172.21.15.119:0/1929664099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff848009a70 con 0x7ff8500abf40 2024-09-13T16:26:47.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff84e59c700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8500abf40 0x7ff850145090 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff8400090b0 tx=0x7ff840004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:47.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff83effd700 1 -- 172.21.15.119:0/1929664099 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8400042c0 con 0x7ff8500abf40 2024-09-13T16:26:47.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.360+0000 7ff83effd700 1 -- 172.21.15.119:0/1929664099 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84001f070 con 0x7ff8500abf40 2024-09-13T16:26:47.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff83effd700 1 -- 172.21.15.119:0/1929664099 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8400036c0 con 0x7ff8500abf40 2024-09-13T16:26:47.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8400094b0 con 0x7ff8500abf40 2024-09-13T16:26:47.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8501455d0 con 0x7ff8500abf40 2024-09-13T16:26:47.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85009a3b0 con 0x7ff8500abf40 2024-09-13T16:26:47.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff83effd700 1 -- 172.21.15.119:0/1929664099 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff840021030 con 0x7ff8500abf40 2024-09-13T16:26:47.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff83effd700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff830066730 0x7ff830068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff83effd700 1 -- 172.21.15.119:0/1929664099 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff8400855f0 con 0x7ff8500abf40 2024-09-13T16:26:47.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.364+0000 7ff84dd9b700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff830066730 0x7ff830068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.368+0000 7ff84dd9b700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff830066730 0x7ff830068bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8500b3a80 tx=0x7ff844009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.372+0000 7ff83effd700 1 -- 172.21.15.119:0/1929664099 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff85009a3b0 con 0x7ff8500abf40 2024-09-13T16:26:47.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.548+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff85009a3b0 con 0x7ff8500abf40 2024-09-13T16:26:47.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff83effd700 1 -- 172.21.15.119:0/1929664099 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16834 (secure 0 0 0) 0x7ff85009a3b0 con 0x7ff8500abf40 2024-09-13T16:26:47.557 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:47.557 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":40,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:38.420314+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:47.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff830066730 msgr2=0x7ff830068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff830066730 0x7ff830068bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7ff8500b3a80 tx=0x7ff844009040 comp rx=0 tx=0).stop 2024-09-13T16:26:47.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8500abf40 msgr2=0x7ff850145090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8500abf40 0x7ff850145090 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7ff8400090b0 tx=0x7ff840004770 comp rx=0 tx=0).stop 2024-09-13T16:26:47.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 shutdown_connections 2024-09-13T16:26:47.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff830066730 0x7ff830068bf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff850147d90 0x7ff85015bb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff850158fd0 0x7ff85015b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 --2- 172.21.15.119:0/1929664099 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff8500abf40 0x7ff850145090 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 >> 172.21.15.119:0/1929664099 conn(0x7ff8500721f0 msgr2=0x7ff850066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:47.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.552+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 shutdown_connections 2024-09-13T16:26:47.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.556+0000 7ff84f59e700 1 -- 172.21.15.119:0/1929664099 wait complete. 2024-09-13T16:26:47.575 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:26:35.251936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:26:36.423580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:26:47.575 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:47.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91ebfff700 1 Processor -- start 2024-09-13T16:26:47.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91ebfff700 1 -- start start 2024-09-13T16:26:47.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91ebfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f91ec0d16c0 0x7f91ec0cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91ebfff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec0cd600 con 0x7f91ec0ce5f0 2024-09-13T16:26:47.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91ebfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec0d0dd0 con 0x7f91ec15cae0 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91ebfff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec0ce1d0 con 0x7f91ec0d16c0 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91eb7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f91ec0d16c0 0x7f91ec0cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91e27fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f91ec0ce5f0 0x7f91ec0d0fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54104/0 (socket says 172.21.15.119:54104) 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.688+0000 7f91e27fc700 1 -- 172.21.15.119:0/2363438522 learned_addr learned my addr 172.21.15.119:0/2363438522 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ea7fc700 1 -- 172.21.15.119:0/2363438522 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17759794 0 0) 0x7f91ec0cd600 con 0x7f91ec0ce5f0 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91eb7fe700 1 -- 172.21.15.119:0/2363438522 >> v1:172.21.15.145:6789/0 conn(0x7f91ec15cae0 legacy=0x7f91ec0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91eb7fe700 1 -- 172.21.15.119:0/2363438522 >> v1:172.21.15.119:6789/0 conn(0x7f91ec0ce5f0 legacy=0x7f91ec0d0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91eb7fe700 1 -- 172.21.15.119:0/2363438522 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec0cd830 con 0x7f91ec0d16c0 2024-09-13T16:26:47.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91eb7fe700 1 --2- 172.21.15.119:0/2363438522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec0cff60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009d90 tx=0x7f91e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c18ebd8a00f48b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:47.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ea7fc700 1 -- 172.21.15.119:0/2363438522 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e400e070 con 0x7f91ec0d16c0 2024-09-13T16:26:47.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ea7fc700 1 -- 172.21.15.119:0/2363438522 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91e400bbd0 con 0x7f91ec0d16c0 2024-09-13T16:26:47.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ea7fc700 1 -- 172.21.15.119:0/2363438522 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91e4012430 con 0x7f91ec0d16c0 2024-09-13T16:26:47.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 -- 172.21.15.119:0/2363438522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 msgr2=0x7f91ec0cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 --2- 172.21.15.119:0/2363438522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec0cff60 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f91e4009d90 tx=0x7f91e400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:47.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 -- 172.21.15.119:0/2363438522 shutdown_connections 2024-09-13T16:26:47.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 --2- 172.21.15.119:0/2363438522 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec0cff60 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 -- 172.21.15.119:0/2363438522 >> 172.21.15.119:0/2363438522 conn(0x7f91ec0d29f0 msgr2=0x7f91ec0d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:47.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 -- 172.21.15.119:0/2363438522 shutdown_connections 2024-09-13T16:26:47.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 -- 172.21.15.119:0/2363438522 wait complete. 2024-09-13T16:26:47.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 Processor -- start 2024-09-13T16:26:47.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.692+0000 7f91ebfff700 1 -- start start 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91ec0ce5f0 0x7f91ec153800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec154c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91ec15cae0 0x7f91ec15f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0cea60 con 0x7f91ec0d16c0 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec153d40 con 0x7f91ec15cae0 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec15dad0 con 0x7f91ec0ce5f0 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec154c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec154c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34992/0 (socket says 172.21.15.119:34992) 2024-09-13T16:26:47.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 -- 172.21.15.119:0/1223815068 learned_addr learned my addr 172.21.15.119:0/1223815068 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91eaffd700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91ec0ce5f0 0x7f91ec153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 -- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91ec0ce5f0 msgr2=0x7f91ec153800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91ec0ce5f0 0x7f91ec153800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91eb7fe700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91ec15cae0 0x7f91ec15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 -- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91ec15cae0 msgr2=0x7f91ec15f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91ec15cae0 0x7f91ec15f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 -- 172.21.15.119:0/1223815068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e4009a70 con 0x7f91ec0d16c0 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91eaffd700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91ec0ce5f0 0x7f91ec153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91eb7fe700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91ec15cae0 0x7f91ec15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e27fc700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec154c30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c4a0 tx=0x7f91d400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e3fff700 1 -- 172.21.15.119:0/1223815068 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4004ca0 con 0x7f91ec0d16c0 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e3fff700 1 -- 172.21.15.119:0/1223815068 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4004e00 con 0x7f91ec0d16c0 2024-09-13T16:26:47.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec155170 con 0x7f91ec0d16c0 2024-09-13T16:26:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91e3fff700 1 -- 172.21.15.119:0/1223815068 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4007310 con 0x7f91ec0d16c0 2024-09-13T16:26:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec0c97a0 con 0x7f91ec0d16c0 2024-09-13T16:26:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.696+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ec073590 con 0x7f91ec0d16c0 2024-09-13T16:26:47.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.700+0000 7f91e3fff700 1 -- 172.21.15.119:0/1223815068 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f91d4007470 con 0x7f91ec0d16c0 2024-09-13T16:26:47.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.700+0000 7f91e3fff700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f91d0066730 0x7f91d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.700+0000 7f91e3fff700 1 -- 172.21.15.119:0/1223815068 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f91d4083f00 con 0x7f91ec0d16c0 2024-09-13T16:26:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.700+0000 7f91eaffd700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f91d0066730 0x7f91d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:47.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.700+0000 7f91eaffd700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f91d0066730 0x7f91d0068bf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f91dc000f60 tx=0x7f91dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:47.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.708+0000 7f91e3fff700 1 -- 172.21.15.119:0/1223815068 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f91d4051120 con 0x7f91ec0d16c0 2024-09-13T16:26:47.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.876+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f91ec0cff60 con 0x7f91ec0d16c0 2024-09-13T16:26:47.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.876+0000 7f91e3fff700 1 -- 172.21.15.119:0/1223815068 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91d4007720 con 0x7f91ec0d16c0 2024-09-13T16:26:47.882 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f91d0066730 msgr2=0x7f91d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f91d0066730 0x7f91d0068bf0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f91dc000f60 tx=0x7f91dc009040 comp rx=0 tx=0).stop 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 msgr2=0x7f91ec154c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec154c30 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f91d400c4a0 tx=0x7f91d400c860 comp rx=0 tx=0).stop 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 shutdown_connections 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f91d0066730 0x7f91d0068bf0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f91ec15cae0 0x7f91ec15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f91ec0d16c0 0x7f91ec154c30 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 --2- 172.21.15.119:0/1223815068 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f91ec0ce5f0 0x7f91ec153800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:47.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 >> 172.21.15.119:0/1223815068 conn(0x7f91ec0d29f0 msgr2=0x7f91ec0d0f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:47.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 shutdown_connections 2024-09-13T16:26:47.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:47.880+0000 7f91ebfff700 1 -- 172.21.15.119:0/1223815068 wait complete. 2024-09-13T16:26:47.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:48.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.016+0000 7faa82979700 1 Processor -- start 2024-09-13T16:26:48.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.016+0000 7faa82979700 1 -- start start 2024-09-13T16:26:48.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0cdd50 0x7faa7c15d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faa7c141fb0 0x7faa7c0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c0712a0 con 0x7faa7c0ace00 2024-09-13T16:26:48.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c0ad2b0 con 0x7faa7c141fb0 2024-09-13T16:26:48.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c06fb00 con 0x7faa7c0cdd50 2024-09-13T16:26:48.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81176700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faa7c141fb0 0x7faa7c0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81977700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0cdd50 0x7faa7c15d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82178700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7faa7c0ace00 0x7faa7c06ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:43654/0 (socket says 172.21.15.119:43654) 2024-09-13T16:26:48.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82178700 1 -- 172.21.15.119:0/2739439560 learned_addr learned my addr 172.21.15.119:0/2739439560 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:48.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81977700 1 -- 172.21.15.119:0/2739439560 >> v2:172.21.15.145:3300/0 conn(0x7faa7c141fb0 msgr2=0x7faa7c0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:48.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81977700 1 --2- 172.21.15.119:0/2739439560 >> v2:172.21.15.145:3300/0 conn(0x7faa7c141fb0 0x7faa7c0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa80975700 1 -- 172.21.15.119:0/2739439560 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1826474793 0 0) 0x7faa7c0712a0 con 0x7faa7c0ace00 2024-09-13T16:26:48.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81977700 1 -- 172.21.15.119:0/2739439560 >> v1:172.21.15.145:6789/0 conn(0x7faa7c0ace00 legacy=0x7faa7c06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81977700 1 -- 172.21.15.119:0/2739439560 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c142490 con 0x7faa7c0cdd50 2024-09-13T16:26:48.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81176700 1 --2- 172.21.15.119:0/2739439560 >> v2:172.21.15.145:3300/0 conn(0x7faa7c141fb0 0x7faa7c0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:48.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa81977700 1 --2- 172.21.15.119:0/2739439560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0cdd50 0x7faa7c15d950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa70009850 tx=0x7faa70009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3afe9926d5f7fa16 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa80975700 1 -- 172.21.15.119:0/2739439560 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70011830 con 0x7faa7c0cdd50 2024-09-13T16:26:48.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa80975700 1 -- 172.21.15.119:0/2739439560 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa70011990 con 0x7faa7c0cdd50 2024-09-13T16:26:48.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa80975700 1 -- 172.21.15.119:0/2739439560 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa70011d40 con 0x7faa7c0cdd50 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- 172.21.15.119:0/2739439560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0cdd50 msgr2=0x7faa7c15d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 --2- 172.21.15.119:0/2739439560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0cdd50 0x7faa7c15d950 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7faa70009850 tx=0x7faa70009b60 comp rx=0 tx=0).stop 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- 172.21.15.119:0/2739439560 shutdown_connections 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 --2- 172.21.15.119:0/2739439560 >> v2:172.21.15.145:3300/0 conn(0x7faa7c141fb0 0x7faa7c0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 --2- 172.21.15.119:0/2739439560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0cdd50 0x7faa7c15d950 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- 172.21.15.119:0/2739439560 >> 172.21.15.119:0/2739439560 conn(0x7faa7c0b0980 msgr2=0x7faa7c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- 172.21.15.119:0/2739439560 shutdown_connections 2024-09-13T16:26:48.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.020+0000 7faa82979700 1 -- 172.21.15.119:0/2739439560 wait complete. 2024-09-13T16:26:48.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 Processor -- start 2024-09-13T16:26:48.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 -- start start 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0ace00 0x7faa7c12abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa7c0cdd50 0x7faa7c12b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa7c141fb0 0x7faa7c12b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c13d450 con 0x7faa7c141fb0 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c13c840 con 0x7faa7c0cdd50 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82979700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c13cb20 con 0x7faa7c0ace00 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82178700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa7c141fb0 0x7faa7c12b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82178700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa7c141fb0 0x7faa7c12b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34994/0 (socket says 172.21.15.119:34994) 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa82178700 1 -- 172.21.15.119:0/2219510735 learned_addr learned my addr 172.21.15.119:0/2219510735 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa81176700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa7c0cdd50 0x7faa7c12b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.024+0000 7faa81977700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0ace00 0x7faa7c12abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82178700 1 -- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0ace00 msgr2=0x7faa7c12abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82178700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0ace00 0x7faa7c12abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82178700 1 -- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa7c0cdd50 msgr2=0x7faa7c12b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82178700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa7c0cdd50 0x7faa7c12b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82178700 1 -- 172.21.15.119:0/2219510735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa780094b0 con 0x7faa7c141fb0 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82178700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa7c141fb0 0x7faa7c12b660 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faa7400cba0 tx=0x7faa7400ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa6e7fc700 1 -- 172.21.15.119:0/2219510735 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7400aaa0 con 0x7faa7c141fb0 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa6e7fc700 1 -- 172.21.15.119:0/2219510735 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7400ac20 con 0x7faa7c141fb0 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa6e7fc700 1 -- 172.21.15.119:0/2219510735 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa740070d0 con 0x7faa7c141fb0 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa700094b0 con 0x7faa7c141fb0 2024-09-13T16:26:48.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c13d0e0 con 0x7faa7c141fb0 2024-09-13T16:26:48.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.028+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c13d5c0 con 0x7faa7c141fb0 2024-09-13T16:26:48.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.032+0000 7faa6e7fc700 1 -- 172.21.15.119:0/2219510735 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7faa74007230 con 0x7faa7c141fb0 2024-09-13T16:26:48.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.032+0000 7faa6e7fc700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa60066730 0x7faa60068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.032+0000 7faa6e7fc700 1 -- 172.21.15.119:0/2219510735 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7faa74083cf0 con 0x7faa7c141fb0 2024-09-13T16:26:48.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.032+0000 7faa81977700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa60066730 0x7faa60068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.032+0000 7faa81977700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa60066730 0x7faa60068bf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faa700090b0 tx=0x7faa70008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.040+0000 7faa6e7fc700 1 -- 172.21.15.119:0/2219510735 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7faa74088070 con 0x7faa7c141fb0 2024-09-13T16:26:48.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.208+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa7c0ac8c0 con 0x7faa7c141fb0 2024-09-13T16:26:48.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.208+0000 7faa6e7fc700 1 -- 172.21.15.119:0/2219510735 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa74018020 con 0x7faa7c141fb0 2024-09-13T16:26:48.214 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:48.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa60066730 msgr2=0x7faa60068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa60066730 0x7faa60068bf0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7faa700090b0 tx=0x7faa70008030 comp rx=0 tx=0).stop 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa7c141fb0 msgr2=0x7faa7c12b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa7c141fb0 0x7faa7c12b660 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faa7400cba0 tx=0x7faa7400ceb0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 shutdown_connections 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa60066730 0x7faa60068bf0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa7c141fb0 0x7faa7c12b660 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa7c0cdd50 0x7faa7c12b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 --2- 172.21.15.119:0/2219510735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa7c0ace00 0x7faa7c12abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 >> 172.21.15.119:0/2219510735 conn(0x7faa7c0b0980 msgr2=0x7faa7c06c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 shutdown_connections 2024-09-13T16:26:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.212+0000 7faa82979700 1 -- 172.21.15.119:0/2219510735 wait complete. 2024-09-13T16:26:48.229 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:48.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f45491ee700 1 Processor -- start 2024-09-13T16:26:48.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f45491ee700 1 -- start start 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f45491ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45440d1560 0x7f45440b30c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f45491ee700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45440b2240 con 0x7f4544160210 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f45491ee700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45440b2520 con 0x7f45440d10d0 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f45491ee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45441605f0 con 0x7f45440d1560 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f4543fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4544160210 0x7f45440d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54108/0 (socket says 172.21.15.119:54108) 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f4543fff700 1 -- 172.21.15.119:0/895406513 learned_addr learned my addr 172.21.15.119:0/895406513 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f45489ed700 1 --2- 172.21.15.119:0/895406513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45440d1560 0x7f45440b30c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f4542ffd700 1 -- 172.21.15.119:0/895406513 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170793384 0 0) 0x7f45440b2240 con 0x7f4544160210 2024-09-13T16:26:48.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.340+0000 7f4542ffd700 1 -- 172.21.15.119:0/895406513 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f452c0035d0 con 0x7f4544160210 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45489ed700 1 -- 172.21.15.119:0/895406513 >> v1:172.21.15.145:6789/0 conn(0x7f45440d10d0 legacy=0x7f45440b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45489ed700 1 -- 172.21.15.119:0/895406513 >> v1:172.21.15.119:6789/0 conn(0x7f4544160210 legacy=0x7f45440d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45489ed700 1 -- 172.21.15.119:0/895406513 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45440d19f0 con 0x7f45440d1560 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45489ed700 1 --2- 172.21.15.119:0/895406513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45440d1560 0x7f45440b30c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f453c009dc0 tx=0x7f453c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db63d8f482e545b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f4542ffd700 1 -- 172.21.15.119:0/895406513 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c00e070 con 0x7f45440d1560 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f4542ffd700 1 -- 172.21.15.119:0/895406513 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c00bb20 con 0x7f45440d1560 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f4542ffd700 1 -- 172.21.15.119:0/895406513 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c012430 con 0x7f45440d1560 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 -- 172.21.15.119:0/895406513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45440d1560 msgr2=0x7f45440b30c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 --2- 172.21.15.119:0/895406513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45440d1560 0x7f45440b30c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f453c009dc0 tx=0x7f453c00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:48.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 -- 172.21.15.119:0/895406513 shutdown_connections 2024-09-13T16:26:48.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 --2- 172.21.15.119:0/895406513 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45440d1560 0x7f45440b30c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 -- 172.21.15.119:0/895406513 >> 172.21.15.119:0/895406513 conn(0x7f454408c080 msgr2=0x7f45440d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:48.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 -- 172.21.15.119:0/895406513 shutdown_connections 2024-09-13T16:26:48.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 -- 172.21.15.119:0/895406513 wait complete. 2024-09-13T16:26:48.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 Processor -- start 2024-09-13T16:26:48.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.344+0000 7f45491ee700 1 -- start start 2024-09-13T16:26:48.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45440d10d0 0x7f45440b8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45440d1560 0x7f45440cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4544160210 0x7f45440c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45441ecea0 con 0x7f45440d10d0 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45440caf00 con 0x7f45440d1560 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45441f3930 con 0x7f4544160210 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45440d10d0 0x7f45440b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45440d10d0 0x7f45440b8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35006/0 (socket says 172.21.15.119:35006) 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 -- 172.21.15.119:0/4236705212 learned_addr learned my addr 172.21.15.119:0/4236705212 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45489ed700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4544160210 0x7f45440c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45437fe700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45440d1560 0x7f45440cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 -- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4544160210 msgr2=0x7f45440c6d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4544160210 0x7f45440c6d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 -- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45440d1560 msgr2=0x7f45440cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45440d1560 0x7f45440cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 -- 172.21.15.119:0/4236705212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f453c009a70 con 0x7f45440d10d0 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4543fff700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45440d10d0 0x7f45440b8360 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f453800c4d0 tx=0x7f453800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4540ff9700 1 -- 172.21.15.119:0/4236705212 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538005d90 con 0x7f45440d10d0 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4540ff9700 1 -- 172.21.15.119:0/4236705212 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45380070d0 con 0x7f45440d10d0 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45440cba60 con 0x7f45440d10d0 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f4540ff9700 1 -- 172.21.15.119:0/4236705212 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4538007480 con 0x7f45440d10d0 2024-09-13T16:26:48.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45440c79c0 con 0x7f45440d10d0 2024-09-13T16:26:48.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.348+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45440cfef0 con 0x7f45440d10d0 2024-09-13T16:26:48.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.352+0000 7f4540ff9700 1 -- 172.21.15.119:0/4236705212 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f45380075e0 con 0x7f45440d10d0 2024-09-13T16:26:48.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.352+0000 7f4540ff9700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4530066730 0x7f4530068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.352+0000 7f4540ff9700 1 -- 172.21.15.119:0/4236705212 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4538083510 con 0x7f45440d10d0 2024-09-13T16:26:48.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.352+0000 7f45437fe700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4530066730 0x7f4530068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.352+0000 7f45437fe700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4530066730 0x7f4530068bf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f45441f5e90 tx=0x7f4534009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.356+0000 7f4540ff9700 1 -- 172.21.15.119:0/4236705212 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4538050740 con 0x7f45440d10d0 2024-09-13T16:26:48.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.524+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45440c7990 con 0x7f45440d10d0 2024-09-13T16:26:48.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.524+0000 7f4540ff9700 1 -- 172.21.15.119:0/4236705212 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4538053d60 con 0x7f45440d10d0 2024-09-13T16:26:48.531 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:48.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4530066730 msgr2=0x7f4530068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4530066730 0x7f4530068bf0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f45441f5e90 tx=0x7f4534009040 comp rx=0 tx=0).stop 2024-09-13T16:26:48.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45440d10d0 msgr2=0x7f45440b8360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45440d10d0 0x7f45440b8360 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f453800c4d0 tx=0x7f453800c7e0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 shutdown_connections 2024-09-13T16:26:48.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4530066730 0x7f4530068bf0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4544160210 0x7f45440c6d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45440d1560 0x7f45440cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 --2- 172.21.15.119:0/4236705212 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45440d10d0 0x7f45440b8360 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 >> 172.21.15.119:0/4236705212 conn(0x7f454408c080 msgr2=0x7f45440b33e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:48.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 shutdown_connections 2024-09-13T16:26:48.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.528+0000 7f45491ee700 1 -- 172.21.15.119:0/4236705212 wait complete. 2024-09-13T16:26:48.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:48.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679e1ac700 1 Processor -- start 2024-09-13T16:26:48.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679e1ac700 1 -- start start 2024-09-13T16:26:48.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679e1ac700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158b30 0x7f6798071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679e1ac700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6798158f10 0x7f6798071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679e1ac700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679806de00 con 0x7f679806ce30 2024-09-13T16:26:48.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679e1ac700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679806db20 con 0x7f6798158f10 2024-09-13T16:26:48.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679e1ac700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67980b1920 con 0x7f6798158b30 2024-09-13T16:26:48.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679d1aa700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158b30 0x7f6798071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679c9a9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6798158f10 0x7f6798071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679d1aa700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158b30 0x7f6798071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:45852/0 (socket says 172.21.15.119:45852) 2024-09-13T16:26:48.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679c9a9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6798158f10 0x7f6798071570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:47866/0 (socket says 172.21.15.119:47866) 2024-09-13T16:26:48.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679d1aa700 1 -- 172.21.15.119:0/522304495 learned_addr learned my addr 172.21.15.119:0/522304495 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:48.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679c9a9700 1 -- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158b30 msgr2=0x7f6798071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:48.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679c9a9700 1 --2- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158b30 0x7f6798071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679c9a9700 1 -- 172.21.15.119:0/522304495 >> v1:172.21.15.145:6789/0 conn(0x7f679806ce30 legacy=0x7f679806d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679c9a9700 1 -- 172.21.15.119:0/522304495 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67980b1c00 con 0x7f6798158f10 2024-09-13T16:26:48.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f678ffff700 1 -- 172.21.15.119:0/522304495 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444741602 0 0) 0x7f679806de00 con 0x7f679806ce30 2024-09-13T16:26:48.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.660+0000 7f679d1aa700 1 --2- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158b30 0x7f6798071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:26:48.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679c9a9700 1 --2- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158f10 0x7f6798071570 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f679400f370 tx=0x7f679400f680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90951ed719e54eff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f678ffff700 1 -- 172.21.15.119:0/522304495 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6794012070 con 0x7f6798158f10 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f678ffff700 1 -- 172.21.15.119:0/522304495 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679400fe70 con 0x7f6798158f10 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f678ffff700 1 -- 172.21.15.119:0/522304495 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679400d2f0 con 0x7f6798158f10 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 -- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158f10 msgr2=0x7f6798071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 --2- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158f10 0x7f6798071570 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f679400f370 tx=0x7f679400f680 comp rx=0 tx=0).stop 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 -- 172.21.15.119:0/522304495 shutdown_connections 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 --2- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158f10 0x7f6798071570 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 --2- 172.21.15.119:0/522304495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158b30 0x7f6798071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 -- 172.21.15.119:0/522304495 >> 172.21.15.119:0/522304495 conn(0x7f679806a350 msgr2=0x7f679806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 -- 172.21.15.119:0/522304495 shutdown_connections 2024-09-13T16:26:48.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 -- 172.21.15.119:0/522304495 wait complete. 2024-09-13T16:26:48.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 Processor -- start 2024-09-13T16:26:48.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.664+0000 7f679e1ac700 1 -- start start 2024-09-13T16:26:48.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f679806ce30 0x7f679815a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158b30 0x7f67981509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158f10 0x7f67980ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679815a880 con 0x7f679806ce30 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67981537e0 con 0x7f6798158b30 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679815aab0 con 0x7f6798158f10 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f679806ce30 0x7f679815a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f679806ce30 0x7f679815a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35014/0 (socket says 172.21.15.119:35014) 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 -- 172.21.15.119:0/3114629783 learned_addr learned my addr 172.21.15.119:0/3114629783 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d9ab700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158f10 0x7f67980ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679c9a9700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158b30 0x7f67981509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 -- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158f10 msgr2=0x7f67980ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158f10 0x7f67980ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 -- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158b30 msgr2=0x7f67981509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158b30 0x7f67981509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 -- 172.21.15.119:0/3114629783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788009570 con 0x7f679806ce30 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d9ab700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158f10 0x7f67980ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679d1aa700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f679806ce30 0x7f679815a340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f678800ebe0 tx=0x7f6788009e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f678dffb700 1 -- 172.21.15.119:0/3114629783 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678800c910 con 0x7f679806ce30 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f678dffb700 1 -- 172.21.15.119:0/3114629783 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678800ca70 con 0x7f679806ce30 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679400f050 con 0x7f679806ce30 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f678dffb700 1 -- 172.21.15.119:0/3114629783 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678800cde0 con 0x7f679806ce30 2024-09-13T16:26:48.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679815b350 con 0x7f679806ce30 2024-09-13T16:26:48.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67980adb60 con 0x7f679806ce30 2024-09-13T16:26:48.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.668+0000 7f678dffb700 1 -- 172.21.15.119:0/3114629783 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6788004020 con 0x7f679806ce30 2024-09-13T16:26:48.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.672+0000 7f678dffb700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6784066730 0x7f6784068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:48.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.672+0000 7f678dffb700 1 -- 172.21.15.119:0/3114629783 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f67880843c0 con 0x7f679806ce30 2024-09-13T16:26:48.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.672+0000 7f679c9a9700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6784066730 0x7f6784068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:48.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.672+0000 7f679c9a9700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6784066730 0x7f6784068bf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f67980ce710 tx=0x7f679400d000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:48.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.676+0000 7f678dffb700 1 -- 172.21.15.119:0/3114629783 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6788051680 con 0x7f679806ce30 2024-09-13T16:26:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.860+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f679806d280 con 0x7f679806ce30 2024-09-13T16:26:48.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.860+0000 7f678dffb700 1 -- 172.21.15.119:0/3114629783 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v40) v1 ==== 80+0+0 (secure 0 0 0) 0x7f6788054ca0 con 0x7f679806ce30 2024-09-13T16:26:48.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6784066730 msgr2=0x7f6784068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6784066730 0x7f6784068bf0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f67980ce710 tx=0x7f679400d000 comp rx=0 tx=0).stop 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f679806ce30 msgr2=0x7f679815a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f679806ce30 0x7f679815a340 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f678800ebe0 tx=0x7f6788009e90 comp rx=0 tx=0).stop 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 shutdown_connections 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6784066730 0x7f6784068bf0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6798158f10 0x7f67980ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6798158b30 0x7f67981509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 --2- 172.21.15.119:0/3114629783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f679806ce30 0x7f679815a340 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 >> 172.21.15.119:0/3114629783 conn(0x7f679806a350 msgr2=0x7f67980aee60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 shutdown_connections 2024-09-13T16:26:48.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.864+0000 7f679e1ac700 1 -- 172.21.15.119:0/3114629783 wait complete. 2024-09-13T16:26:48.870 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:48.880 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:26:48.881 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:26:48.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:26:49.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bdc84700 1 Processor -- start 2024-09-13T16:26:49.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bdc84700 1 -- start start 2024-09-13T16:26:49.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bdc84700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff0b815cc40 0x7ff0b80ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bdc84700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80cd860 0x7ff0b808bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bdc84700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b815d020 con 0x7ff0b808c4e0 2024-09-13T16:26:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bdc84700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b808c970 con 0x7ff0b815cc40 2024-09-13T16:26:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bdc84700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b80b44c0 con 0x7ff0b80cd860 2024-09-13T16:26:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bd483700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff0b808c4e0 0x7ff0b8160720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54118/0 (socket says 172.21.15.119:54118) 2024-09-13T16:26:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bd483700 1 -- 172.21.15.119:0/204433315 learned_addr learned my addr 172.21.15.119:0/204433315 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bcc82700 1 --2- 172.21.15.119:0/204433315 >> v2:172.21.15.145:3300/0 conn(0x7ff0b815cc40 0x7ff0b80ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0b7fff700 1 --2- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80cd860 0x7ff0b808bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0b77fe700 1 -- 172.21.15.119:0/204433315 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314250987 0 0) 0x7ff0b815d020 con 0x7ff0b808c4e0 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0b77fe700 1 -- 172.21.15.119:0/204433315 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a40035d0 con 0x7ff0b808c4e0 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bcc82700 1 -- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80cd860 msgr2=0x7ff0b808bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bcc82700 1 --2- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80cd860 0x7ff0b808bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bcc82700 1 -- 172.21.15.119:0/204433315 >> v1:172.21.15.119:6789/0 conn(0x7ff0b808c4e0 legacy=0x7ff0b8160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bcc82700 1 -- 172.21.15.119:0/204433315 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b80d0720 con 0x7ff0b815cc40 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0bcc82700 1 --2- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 0x7ff0b80ce710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009880 tx=0x7ff0a8009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=468eaa4ce663f03b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0b77fe700 1 -- 172.21.15.119:0/204433315 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8011830 con 0x7ff0b815cc40 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0b77fe700 1 -- 172.21.15.119:0/204433315 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8011990 con 0x7ff0b815cc40 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:48.996+0000 7ff0b77fe700 1 -- 172.21.15.119:0/204433315 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8011d20 con 0x7ff0b815cc40 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 msgr2=0x7ff0b80ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 0x7ff0b80ce710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009880 tx=0x7ff0a8009b90 comp rx=0 tx=0).stop 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- 172.21.15.119:0/204433315 shutdown_connections 2024-09-13T16:26:49.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80cd860 0x7ff0b808bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/204433315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 0x7ff0b80ce710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- 172.21.15.119:0/204433315 >> 172.21.15.119:0/204433315 conn(0x7ff0b80b4640 msgr2=0x7ff0b80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- 172.21.15.119:0/204433315 shutdown_connections 2024-09-13T16:26:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- 172.21.15.119:0/204433315 wait complete. 2024-09-13T16:26:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 Processor -- start 2024-09-13T16:26:49.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- start start 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b808c4e0 0x7ff0b81f1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80cd860 0x7ff0b80b6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 0x7ff0b80b6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81e2830 con 0x7ff0b80cd860 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b80b88c0 con 0x7ff0b815cc40 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0bdc84700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81e2280 con 0x7ff0b808c4e0 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.000+0000 7ff0b7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80cd860 0x7ff0b80b6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80cd860 0x7ff0b80b6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35018/0 (socket says 172.21.15.119:35018) 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 -- 172.21.15.119:0/675043847 learned_addr learned my addr 172.21.15.119:0/675043847 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0bd483700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 0x7ff0b80b6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0bcc82700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b808c4e0 0x7ff0b81f1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 -- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b808c4e0 msgr2=0x7ff0b81f1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b808c4e0 0x7ff0b81f1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 -- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 msgr2=0x7ff0b80b6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 0x7ff0b80b6b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 -- 172.21.15.119:0/675043847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a80094b0 con 0x7ff0b80cd860 2024-09-13T16:26:49.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b7fff700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80cd860 0x7ff0b80b6630 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000efc0 tx=0x7ff0a000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b57fa700 1 -- 172.21.15.119:0/675043847 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a000caf0 con 0x7ff0b80cd860 2024-09-13T16:26:49.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b57fa700 1 -- 172.21.15.119:0/675043847 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a000f040 con 0x7ff0b80cd860 2024-09-13T16:26:49.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b81f1f40 con 0x7ff0b80cd860 2024-09-13T16:26:49.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b57fa700 1 -- 172.21.15.119:0/675043847 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a000ce70 con 0x7ff0b80cd860 2024-09-13T16:26:49.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b80b8300 con 0x7ff0b80cd860 2024-09-13T16:26:49.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b81e2970 con 0x7ff0b80cd860 2024-09-13T16:26:49.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b57fa700 1 -- 172.21.15.119:0/675043847 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff0a0014480 con 0x7ff0b80cd860 2024-09-13T16:26:49.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.004+0000 7ff0b57fa700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff098066730 0x7ff098068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.008+0000 7ff0b57fa700 1 -- 172.21.15.119:0/675043847 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff0a0084a00 con 0x7ff0b80cd860 2024-09-13T16:26:49.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.008+0000 7ff0bcc82700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff098066730 0x7ff098068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.008+0000 7ff0bcc82700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff098066730 0x7ff098068bf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8000c00 tx=0x7ff0a8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.016+0000 7ff0b57fa700 1 -- 172.21.15.119:0/675043847 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff0a0051cc0 con 0x7ff0b80cd860 2024-09-13T16:26:49.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.184+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0b80d1540 con 0x7ff0b80cd860 2024-09-13T16:26:49.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.184+0000 7ff0b57fa700 1 -- 172.21.15.119:0/675043847 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) v1 ==== 74+0+16834 (secure 0 0 0) 0x7ff0a00552e0 con 0x7ff0b80cd860 2024-09-13T16:26:49.190 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:49.190 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":40,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:26:38.420314+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff098066730 msgr2=0x7ff098068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff098066730 0x7ff098068bf0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8000c00 tx=0x7ff0a8008030 comp rx=0 tx=0).stop 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80cd860 msgr2=0x7ff0b80b6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80cd860 0x7ff0b80b6630 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000efc0 tx=0x7ff0a000c0c0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 shutdown_connections 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff098066730 0x7ff098068bf0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b815cc40 0x7ff0b80b6b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80cd860 0x7ff0b80b6630 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 --2- 172.21.15.119:0/675043847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b808c4e0 0x7ff0b81f1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 >> 172.21.15.119:0/675043847 conn(0x7ff0b80b4640 msgr2=0x7ff0b81e56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 shutdown_connections 2024-09-13T16:26:49.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.188+0000 7ff0bdc84700 1 -- 172.21.15.119:0/675043847 wait complete. 2024-09-13T16:26:49.205 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:26:35.251936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:26:36.423580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:26:49.205 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:26:49.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.316+0000 7f3692ac7700 1 Processor -- start 2024-09-13T16:26:49.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.316+0000 7f3692ac7700 1 -- start start 2024-09-13T16:26:49.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f368c0cdd50 0x7f368c15d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f368c141fb0 0x7f368c0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f368c0712a0 con 0x7f368c0ace00 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368c06fb00 con 0x7f368c141fb0 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368c15de90 con 0x7f368c0cdd50 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36922c6700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f368c0ace00 0x7f368c06ff50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54128/0 (socket says 172.21.15.119:54128) 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36922c6700 1 -- 172.21.15.119:0/1730462764 learned_addr learned my addr 172.21.15.119:0/1730462764 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36912c4700 1 --2- 172.21.15.119:0/1730462764 >> v2:172.21.15.119:3300/0 conn(0x7f368c141fb0 0x7f368c0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3691ac5700 1 --2- 172.21.15.119:0/1730462764 >> v2:172.21.15.145:3300/0 conn(0x7f368c0cdd50 0x7f368c15d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3690ac3700 1 -- 172.21.15.119:0/1730462764 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495131669 0 0) 0x7f368c0712a0 con 0x7f368c0ace00 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3690ac3700 1 -- 172.21.15.119:0/1730462764 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36780035d0 con 0x7f368c0ace00 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36912c4700 1 -- 172.21.15.119:0/1730462764 >> v2:172.21.15.145:3300/0 conn(0x7f368c0cdd50 msgr2=0x7f368c15d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36912c4700 1 --2- 172.21.15.119:0/1730462764 >> v2:172.21.15.145:3300/0 conn(0x7f368c0cdd50 0x7f368c15d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36912c4700 1 -- 172.21.15.119:0/1730462764 >> v1:172.21.15.119:6789/0 conn(0x7f368c0ace00 legacy=0x7f368c06ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36912c4700 1 -- 172.21.15.119:0/1730462764 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368c142490 con 0x7f368c141fb0 2024-09-13T16:26:49.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f36912c4700 1 --2- 172.21.15.119:0/1730462764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c141fb0 0x7f368c0ac8c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f368c0b1560 tx=0x7f3674009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce7479a547f7385 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3690ac3700 1 -- 172.21.15.119:0/1730462764 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674011830 con 0x7f368c141fb0 2024-09-13T16:26:49.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3690ac3700 1 -- 172.21.15.119:0/1730462764 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3674011990 con 0x7f368c141fb0 2024-09-13T16:26:49.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3690ac3700 1 -- 172.21.15.119:0/1730462764 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674011d20 con 0x7f368c141fb0 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- 172.21.15.119:0/1730462764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c141fb0 msgr2=0x7f368c0ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 --2- 172.21.15.119:0/1730462764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c141fb0 0x7f368c0ac8c0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f368c0b1560 tx=0x7f3674009b30 comp rx=0 tx=0).stop 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- 172.21.15.119:0/1730462764 shutdown_connections 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 --2- 172.21.15.119:0/1730462764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c141fb0 0x7f368c0ac8c0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 --2- 172.21.15.119:0/1730462764 >> v2:172.21.15.145:3300/0 conn(0x7f368c0cdd50 0x7f368c15d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- 172.21.15.119:0/1730462764 >> 172.21.15.119:0/1730462764 conn(0x7f368c0b0980 msgr2=0x7f368c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- 172.21.15.119:0/1730462764 shutdown_connections 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 -- 172.21.15.119:0/1730462764 wait complete. 2024-09-13T16:26:49.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.320+0000 7f3692ac7700 1 Processor -- start 2024-09-13T16:26:49.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 -- start start 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f368c0ace00 0x7f368c153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c0cdd50 0x7f368c13b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f368c141fb0 0x7f368c1e62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c1544a0 con 0x7f368c0cdd50 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f368c06bb00 con 0x7f368c141fb0 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f368c06bde0 con 0x7f368c0ace00 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c0cdd50 0x7f368c13b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c0cdd50 0x7f368c13b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35050/0 (socket says 172.21.15.119:35050) 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 -- 172.21.15.119:0/290753646 learned_addr learned my addr 172.21.15.119:0/290753646 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36922c6700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f368c141fb0 0x7f368c1e62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3691ac5700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f368c0ace00 0x7f368c153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 -- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f368c0ace00 msgr2=0x7f368c153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f368c0ace00 0x7f368c153f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 -- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f368c141fb0 msgr2=0x7f368c1e62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f368c141fb0 0x7f368c1e62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 -- 172.21.15.119:0/290753646 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36740094b0 con 0x7f368c0cdd50 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36922c6700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f368c141fb0 0x7f368c1e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f36912c4700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c0cdd50 0x7f368c13b340 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f36740090b0 tx=0x7f3674004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f368a7fc700 1 -- 172.21.15.119:0/290753646 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3674004520 con 0x7f368c0cdd50 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f368a7fc700 1 -- 172.21.15.119:0/290753646 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367401f070 con 0x7f368c0cdd50 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368c06c0c0 con 0x7f368c0cdd50 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f368a7fc700 1 -- 172.21.15.119:0/290753646 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36740038c0 con 0x7f368c0cdd50 2024-09-13T16:26:49.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368c13dce0 con 0x7f368c0cdd50 2024-09-13T16:26:49.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.324+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368c06c5e0 con 0x7f368c0cdd50 2024-09-13T16:26:49.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.328+0000 7f368a7fc700 1 -- 172.21.15.119:0/290753646 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3674003a20 con 0x7f368c0cdd50 2024-09-13T16:26:49.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.328+0000 7f368a7fc700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f366c066730 0x7f366c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.328+0000 7f368a7fc700 1 -- 172.21.15.119:0/290753646 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3674021070 con 0x7f368c0cdd50 2024-09-13T16:26:49.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.328+0000 7f3691ac5700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f366c066730 0x7f366c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.328+0000 7f3691ac5700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f366c066730 0x7f366c068bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3684009f30 tx=0x7f368400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.332+0000 7f368a7fc700 1 -- 172.21.15.119:0/290753646 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3674052d60 con 0x7f368c0cdd50 2024-09-13T16:26:49.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.500+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f368c06c5e0 con 0x7f368c0cdd50 2024-09-13T16:26:49.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.500+0000 7f368a7fc700 1 -- 172.21.15.119:0/290753646 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) v1 ==== 93+0+10 (secure 0 0 0) 0x7f368c06c5e0 con 0x7f368c0cdd50 2024-09-13T16:26:49.506 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:26:49.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f366c066730 msgr2=0x7f366c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f366c066730 0x7f366c068bf0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3684009f30 tx=0x7f368400b040 comp rx=0 tx=0).stop 2024-09-13T16:26:49.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c0cdd50 msgr2=0x7f368c13b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c0cdd50 0x7f368c13b340 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f36740090b0 tx=0x7f3674004770 comp rx=0 tx=0).stop 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 shutdown_connections 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f366c066730 0x7f366c068bf0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f368c141fb0 0x7f368c1e62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f368c0cdd50 0x7f368c13b340 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 --2- 172.21.15.119:0/290753646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f368c0ace00 0x7f368c153f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 >> 172.21.15.119:0/290753646 conn(0x7f368c0b0980 msgr2=0x7f368c15e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 shutdown_connections 2024-09-13T16:26:49.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.504+0000 7f3692ac7700 1 -- 172.21.15.119:0/290753646 wait complete. 2024-09-13T16:26:49.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:26:49.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.628+0000 7f28ebfff700 1 Processor -- start 2024-09-13T16:26:49.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.628+0000 7f28ebfff700 1 -- start start 2024-09-13T16:26:49.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f28ec06c480 0x7f28ec0cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f28ec0ce000 0x7f28ec0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28ec06c8c0 con 0x7f28ec144180 2024-09-13T16:26:49.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28ec0b1c50 con 0x7f28ec06c480 2024-09-13T16:26:49.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28ec06cb40 con 0x7f28ec0ce000 2024-09-13T16:26:49.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f28ec06c480 0x7f28ec0cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f28ec06c480 0x7f28ec0cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35056/0 (socket says 172.21.15.119:35056) 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 -- 172.21.15.119:0/4012022341 learned_addr learned my addr 172.21.15.119:0/4012022341 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28eb7fe700 1 --2- 172.21.15.119:0/4012022341 >> v2:172.21.15.145:3300/0 conn(0x7f28ec0ce000 0x7f28ec0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 -- 172.21.15.119:0/4012022341 >> v2:172.21.15.145:3300/0 conn(0x7f28ec0ce000 msgr2=0x7f28ec0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 --2- 172.21.15.119:0/4012022341 >> v2:172.21.15.145:3300/0 conn(0x7f28ec0ce000 0x7f28ec0b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 -- 172.21.15.119:0/4012022341 >> v1:172.21.15.145:6789/0 conn(0x7f28ec144180 legacy=0x7f28ec0cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 -- 172.21.15.119:0/4012022341 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ec06ce20 con 0x7f28ec06c480 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28eb7fe700 1 --2- 172.21.15.119:0/4012022341 >> v2:172.21.15.145:3300/0 conn(0x7f28ec0ce000 0x7f28ec0b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28e27fc700 1 --2- 172.21.15.119:0/4012022341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec06c480 0x7f28ec0cdac0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0ce760 tx=0x7f28d4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fa1dd7f39241085 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ea7fc700 1 -- 172.21.15.119:0/4012022341 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4004230 con 0x7f28ec06c480 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ea7fc700 1 -- 172.21.15.119:0/4012022341 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d4004390 con 0x7f28ec06c480 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ea7fc700 1 -- 172.21.15.119:0/4012022341 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d40036c0 con 0x7f28ec06c480 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- 172.21.15.119:0/4012022341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec06c480 msgr2=0x7f28ec0cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 --2- 172.21.15.119:0/4012022341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec06c480 0x7f28ec0cdac0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f28ec0ce760 tx=0x7f28d4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- 172.21.15.119:0/4012022341 shutdown_connections 2024-09-13T16:26:49.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 --2- 172.21.15.119:0/4012022341 >> v2:172.21.15.145:3300/0 conn(0x7f28ec0ce000 0x7f28ec0b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 --2- 172.21.15.119:0/4012022341 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec06c480 0x7f28ec0cdac0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- 172.21.15.119:0/4012022341 >> 172.21.15.119:0/4012022341 conn(0x7f28ec0b0980 msgr2=0x7f28ec0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:49.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- 172.21.15.119:0/4012022341 shutdown_connections 2024-09-13T16:26:49.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.632+0000 7f28ebfff700 1 -- 172.21.15.119:0/4012022341 wait complete. 2024-09-13T16:26:49.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 Processor -- start 2024-09-13T16:26:49.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 -- start start 2024-09-13T16:26:49.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28ec06c480 0x7f28ec133890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec0ce000 0x7f28ec137820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28ec144180 0x7f28ec15a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec139090 con 0x7f28ec0ce000 2024-09-13T16:26:49.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec137d60 con 0x7f28ec144180 2024-09-13T16:26:49.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ec138880 con 0x7f28ec06c480 2024-09-13T16:26:49.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec0ce000 0x7f28ec137820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec0ce000 0x7f28ec137820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35072/0 (socket says 172.21.15.119:35072) 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 -- 172.21.15.119:0/2820328772 learned_addr learned my addr 172.21.15.119:0/2820328772 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28eb7fe700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28ec144180 0x7f28ec15a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28eaffd700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28ec06c480 0x7f28ec133890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 -- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28ec06c480 msgr2=0x7f28ec133890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28ec06c480 0x7f28ec133890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 -- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28ec144180 msgr2=0x7f28ec15a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28ec144180 0x7f28ec15a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 -- 172.21.15.119:0/2820328772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d4009570 con 0x7f28ec0ce000 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e27fc700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec0ce000 0x7f28ec137820 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f28d4004000 tx=0x7f28d4004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e8ff9700 1 -- 172.21.15.119:0/2820328772 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4003e60 con 0x7f28ec0ce000 2024-09-13T16:26:49.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e8ff9700 1 -- 172.21.15.119:0/2820328772 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d400b870 con 0x7f28ec0ce000 2024-09-13T16:26:49.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ec135500 con 0x7f28ec0ce000 2024-09-13T16:26:49.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28e8ff9700 1 -- 172.21.15.119:0/2820328772 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d400bc20 con 0x7f28ec0ce000 2024-09-13T16:26:49.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.636+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ec138ad0 con 0x7f28ec0ce000 2024-09-13T16:26:49.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.640+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ec135690 con 0x7f28ec0ce000 2024-09-13T16:26:49.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.640+0000 7f28e8ff9700 1 -- 172.21.15.119:0/2820328772 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f28d400bd80 con 0x7f28ec0ce000 2024-09-13T16:26:49.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.640+0000 7f28e8ff9700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28cc066730 0x7f28cc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.640+0000 7f28e8ff9700 1 -- 172.21.15.119:0/2820328772 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f28d4085a10 con 0x7f28ec0ce000 2024-09-13T16:26:49.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.640+0000 7f28eaffd700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28cc066730 0x7f28cc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.640+0000 7f28eaffd700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28cc066730 0x7f28cc068bf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f28ec138810 tx=0x7f28dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.648+0000 7f28e8ff9700 1 -- 172.21.15.119:0/2820328772 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f28d4052cd0 con 0x7f28ec0ce000 2024-09-13T16:26:49.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.816+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f28ec0cdac0 con 0x7f28ec0ce000 2024-09-13T16:26:49.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28e8ff9700 1 -- 172.21.15.119:0/2820328772 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v40) v1 ==== 104+0+11 (secure 0 0 0) 0x7f28d40562f0 con 0x7f28ec0ce000 2024-09-13T16:26:49.824 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:26:49.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28cc066730 msgr2=0x7f28cc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28cc066730 0x7f28cc068bf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f28ec138810 tx=0x7f28dc009040 comp rx=0 tx=0).stop 2024-09-13T16:26:49.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec0ce000 msgr2=0x7f28ec137820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec0ce000 0x7f28ec137820 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f28d4004000 tx=0x7f28d4004770 comp rx=0 tx=0).stop 2024-09-13T16:26:49.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 shutdown_connections 2024-09-13T16:26:49.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28cc066730 0x7f28cc068bf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28ec144180 0x7f28ec15a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28ec0ce000 0x7f28ec137820 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 --2- 172.21.15.119:0/2820328772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28ec06c480 0x7f28ec133890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 >> 172.21.15.119:0/2820328772 conn(0x7f28ec0b0980 msgr2=0x7f28ec13d080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:49.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 shutdown_connections 2024-09-13T16:26:49.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.820+0000 7f28ebfff700 1 -- 172.21.15.119:0/2820328772 wait complete. 2024-09-13T16:26:49.838 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:26:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68ab09700 1 Processor -- start 2024-09-13T16:26:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68ab09700 1 -- start start 2024-09-13T16:26:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68ab09700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 0x7fb6840d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68ab09700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6840cdb00 con 0x7fb6840d17e0 2024-09-13T16:26:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68ab09700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6840cdde0 con 0x7fb6840d0c20 2024-09-13T16:26:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68ab09700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840ce0c0 con 0x7fb68415cc00 2024-09-13T16:26:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68a308700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb6840d17e0 0x7fb6840b5960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54130/0 (socket says 172.21.15.119:54130) 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68a308700 1 -- 172.21.15.119:0/450994993 learned_addr learned my addr 172.21.15.119:0/450994993 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689b07700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 0x7fb6840d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166291659 0 0) 0x7fb6840cdb00 con 0x7fb6840d17e0 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6700035d0 con 0x7fb6840d17e0 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429057480 0 0) 0x7fb6840cdde0 con 0x7fb6840d0c20 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6840cdb00 con 0x7fb6840d0c20 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3743112375 0 0) 0x7fb6700035d0 con 0x7fb6840d17e0 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 msgr2=0x7fb6840d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 0x7fb6840d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 >> v1:172.21.15.145:6789/0 conn(0x7fb6840d0c20 legacy=0x7fb6840d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6840ce3a0 con 0x7fb6840d17e0 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fb67c0019f0 con 0x7fb6840d17e0 2024-09-13T16:26:49.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 >> v1:172.21.15.119:6789/0 conn(0x7fb6840d17e0 legacy=0x7fb6840b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670003be0 0x7fb670003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb670004870 0x7fb670004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 0x7fb670005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840ce0c0 con 0x7fb670004870 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000bae0 con 0x7fb670003be0 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67000bc70 con 0x7fb68415cc00 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689306700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb670004870 0x7fb670004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689b07700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670003be0 0x7fb670003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb68a308700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 0x7fb670005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689306700 1 -- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 msgr2=0x7fb670005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689306700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb68415cc00 0x7fb670005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689306700 1 -- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670003be0 msgr2=0x7fb670003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:49.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689306700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670003be0 0x7fb670003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.952+0000 7fb689306700 1 -- 172.21.15.119:0/450994993 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6700035d0 con 0x7fb670004870 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb670004870 0x7fb670004cf0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb6840c6610 tx=0x7fb6700058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f913ded22ac7f1fa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678001450 con 0x7fb670004870 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6780052f0 con 0x7fb670004870 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb688b05700 1 -- 172.21.15.119:0/450994993 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678001d60 con 0x7fb670004870 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- 172.21.15.119:0/450994993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb670004870 msgr2=0x7fb670004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 --2- 172.21.15.119:0/450994993 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb670004870 0x7fb670004cf0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fb6840c6610 tx=0x7fb6700058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68a308700 1 -- 172.21.15.119:0/450994993 reap_dead start 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- 172.21.15.119:0/450994993 shutdown_connections 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- 172.21.15.119:0/450994993 >> 172.21.15.119:0/450994993 conn(0x7fb68408c080 msgr2=0x7fb6840d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- 172.21.15.119:0/450994993 shutdown_connections 2024-09-13T16:26:49.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- 172.21.15.119:0/450994993 wait complete. 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 Processor -- start 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- start start 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670004870 0x7fb6841f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68415fd40 0x7fb6840cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb684156a20 0x7fb684157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb684156890 con 0x7fb68415fd40 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840cad90 con 0x7fb670004870 2024-09-13T16:26:49.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68ab09700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6840cb910 con 0x7fb684156a20 2024-09-13T16:26:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68415fd40 0x7fb6840cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68415fd40 0x7fb6840cbda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35088/0 (socket says 172.21.15.119:35088) 2024-09-13T16:26:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 -- 172.21.15.119:0/3040057842 learned_addr learned my addr 172.21.15.119:0/3040057842 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689b07700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670004870 0x7fb6841f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb68a308700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb684156a20 0x7fb684157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 -- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb684156a20 msgr2=0x7fb684157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb684156a20 0x7fb684157420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 -- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670004870 msgr2=0x7fb6841f3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670004870 0x7fb6841f3270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 -- 172.21.15.119:0/3040057842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6841601c0 con 0x7fb68415fd40 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.956+0000 7fb689306700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68415fd40 0x7fb6840cbda0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb68408cab0 tx=0x7fb6780012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb6767fc700 1 -- 172.21.15.119:0/3040057842 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678006c90 con 0x7fb68415fd40 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb6767fc700 1 -- 172.21.15.119:0/3040057842 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb678006df0 con 0x7fb68415fd40 2024-09-13T16:26:49.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6840cc2e0 con 0x7fb68415fd40 2024-09-13T16:26:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb6767fc700 1 -- 172.21.15.119:0/3040057842 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb678007240 con 0x7fb68415fd40 2024-09-13T16:26:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6840cb2c0 con 0x7fb68415fd40 2024-09-13T16:26:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6840cdde0 con 0x7fb68415fd40 2024-09-13T16:26:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb6767fc700 1 -- 172.21.15.119:0/3040057842 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb6780073a0 con 0x7fb68415fd40 2024-09-13T16:26:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb6767fc700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb668066730 0x7fb668068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:49.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb6767fc700 1 -- 172.21.15.119:0/3040057842 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb678085140 con 0x7fb68415fd40 2024-09-13T16:26:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.960+0000 7fb689b07700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb668066730 0x7fb668068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:49.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.964+0000 7fb689b07700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb668066730 0x7fb668068bf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb680009ad0 tx=0x7fb680008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:49.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:49.968+0000 7fb6767fc700 1 -- 172.21.15.119:0/3040057842 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb678055e10 con 0x7fb68415fd40 2024-09-13T16:26:50.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.132+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb6840cba50 con 0x7fb68415fd40 2024-09-13T16:26:50.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.132+0000 7fb6767fc700 1 -- 172.21.15.119:0/3040057842 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v40) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb6780559a0 con 0x7fb68415fd40 2024-09-13T16:26:50.138 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:26:50.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb668066730 msgr2=0x7fb668068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb668066730 0x7fb668068bf0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fb680009ad0 tx=0x7fb680008040 comp rx=0 tx=0).stop 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68415fd40 msgr2=0x7fb6840cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68415fd40 0x7fb6840cbda0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fb68408cab0 tx=0x7fb6780012d0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 shutdown_connections 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb668066730 0x7fb668068bf0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb684156a20 0x7fb684157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb68415fd40 0x7fb6840cbda0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 --2- 172.21.15.119:0/3040057842 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb670004870 0x7fb6841f3270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 >> 172.21.15.119:0/3040057842 conn(0x7fb68408c080 msgr2=0x7fb6840b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:50.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 shutdown_connections 2024-09-13T16:26:50.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.136+0000 7fb68ab09700 1 -- 172.21.15.119:0/3040057842 wait complete. 2024-09-13T16:26:50.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:26:50.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa833316700 1 Processor -- start 2024-09-13T16:26:50.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa833316700 1 -- start start 2024-09-13T16:26:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa833316700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa833316700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa82c0b3800 0x7fa82c06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa833316700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0b2280 con 0x7fa82c06da70 2024-09-13T16:26:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa833316700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa82c06deb0 con 0x7fa82c0b3800 2024-09-13T16:26:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa833316700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0cc890 con 0x7fa82c1594f0 2024-09-13T16:26:50.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa831b13700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa82c06da70 0x7fa82c0b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54134/0 (socket says 172.21.15.119:54134) 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa831b13700 1 -- 172.21.15.119:0/1554736019 learned_addr learned my addr 172.21.15.119:0/1554736019 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa832b15700 1 --2- 172.21.15.119:0/1554736019 >> v2:172.21.15.119:3300/0 conn(0x7fa82c0b3800 0x7fa82c06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa832314700 1 --2- 172.21.15.119:0/1554736019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.260+0000 7fa831312700 1 -- 172.21.15.119:0/1554736019 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116884945 0 0) 0x7fa82c0b2280 con 0x7fa82c06da70 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa831312700 1 -- 172.21.15.119:0/1554736019 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8180057e0 con 0x7fa82c06da70 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa832b15700 1 -- 172.21.15.119:0/1554736019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 msgr2=0x7fa82c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa832b15700 1 --2- 172.21.15.119:0/1554736019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa832b15700 1 -- 172.21.15.119:0/1554736019 >> v1:172.21.15.119:6789/0 conn(0x7fa82c06da70 legacy=0x7fa82c0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa832b15700 1 -- 172.21.15.119:0/1554736019 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0b1970 con 0x7fa82c0b3800 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa832b15700 1 --2- 172.21.15.119:0/1554736019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c0b3800 0x7fa82c06fc00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0b1b60 tx=0x7fa82400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ec08a85ea8d0b1e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa831312700 1 -- 172.21.15.119:0/1554736019 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa824016070 con 0x7fa82c0b3800 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa831312700 1 -- 172.21.15.119:0/1554736019 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82400bb20 con 0x7fa82c0b3800 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa831312700 1 -- 172.21.15.119:0/1554736019 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82400e070 con 0x7fa82c0b3800 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 -- 172.21.15.119:0/1554736019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c0b3800 msgr2=0x7fa82c06fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 --2- 172.21.15.119:0/1554736019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c0b3800 0x7fa82c06fc00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0b1b60 tx=0x7fa82400b330 comp rx=0 tx=0).stop 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 -- 172.21.15.119:0/1554736019 shutdown_connections 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 --2- 172.21.15.119:0/1554736019 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c0b3800 0x7fa82c06fc00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 --2- 172.21.15.119:0/1554736019 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 -- 172.21.15.119:0/1554736019 >> 172.21.15.119:0/1554736019 conn(0x7fa82c06a350 msgr2=0x7fa82c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:50.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 -- 172.21.15.119:0/1554736019 shutdown_connections 2024-09-13T16:26:50.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.264+0000 7fa833316700 1 -- 172.21.15.119:0/1554736019 wait complete. 2024-09-13T16:26:50.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 Processor -- start 2024-09-13T16:26:50.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 -- start start 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c06da70 0x7fa82c1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82c0b3800 0x7fa82c147d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c1584e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c157440 con 0x7fa82c06da70 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c157720 con 0x7fa82c0b3800 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa833316700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c157a00 con 0x7fa82c1594f0 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c06da70 0x7fa82c1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c06da70 0x7fa82c1553c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35106/0 (socket says 172.21.15.119:35106) 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 -- 172.21.15.119:0/4265426067 learned_addr learned my addr 172.21.15.119:0/4265426067 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832b15700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c1584e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa831b13700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82c0b3800 0x7fa82c147d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:50.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 -- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 msgr2=0x7fa82c1584e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c1584e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 -- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82c0b3800 msgr2=0x7fa82c147d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82c0b3800 0x7fa82c147d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 -- 172.21.15.119:0/4265426067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa824009a70 con 0x7fa82c06da70 2024-09-13T16:26:50.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.268+0000 7fa832314700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c06da70 0x7fa82c1553c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00e760 tx=0x7fa81c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:50.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa822ffd700 1 -- 172.21.15.119:0/4265426067 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c013610 con 0x7fa82c06da70 2024-09-13T16:26:50.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa822ffd700 1 -- 172.21.15.119:0/4265426067 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c013770 con 0x7fa82c06da70 2024-09-13T16:26:50.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c157ce0 con 0x7fa82c06da70 2024-09-13T16:26:50.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa822ffd700 1 -- 172.21.15.119:0/4265426067 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa81c013ae0 con 0x7fa82c06da70 2024-09-13T16:26:50.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c1570a0 con 0x7fa82c06da70 2024-09-13T16:26:50.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c0a9b80 con 0x7fa82c06da70 2024-09-13T16:26:50.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa822ffd700 1 -- 172.21.15.119:0/4265426067 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa81c013c40 con 0x7fa82c06da70 2024-09-13T16:26:50.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa822ffd700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa810066730 0x7fa810068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:50.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa822ffd700 1 -- 172.21.15.119:0/4265426067 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa81c083ed0 con 0x7fa82c06da70 2024-09-13T16:26:50.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa831b13700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa810066730 0x7fa810068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:50.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.272+0000 7fa831b13700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa810066730 0x7fa810068bf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0cad10 tx=0x7fa828009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:50.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.280+0000 7fa822ffd700 1 -- 172.21.15.119:0/4265426067 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa81c051190 con 0x7fa82c06da70 2024-09-13T16:26:50.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.460+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa82c0b30f0 con 0x7fa82c06da70 2024-09-13T16:26:50.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa822ffd700 1 -- 172.21.15.119:0/4265426067 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v40) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa81c0547b0 con 0x7fa82c06da70 2024-09-13T16:26:50.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa810066730 msgr2=0x7fa810068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa810066730 0x7fa810068bf0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa82c0cad10 tx=0x7fa828009040 comp rx=0 tx=0).stop 2024-09-13T16:26:50.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c06da70 msgr2=0x7fa82c1553c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:50.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c06da70 0x7fa82c1553c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fa81c00e760 tx=0x7fa81c00eb20 comp rx=0 tx=0).stop 2024-09-13T16:26:50.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 shutdown_connections 2024-09-13T16:26:50.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa810066730 0x7fa810068bf0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa82c1594f0 0x7fa82c1584e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa82c0b3800 0x7fa82c147d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 --2- 172.21.15.119:0/4265426067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa82c06da70 0x7fa82c1553c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:50.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 >> 172.21.15.119:0/4265426067 conn(0x7fa82c06a350 msgr2=0x7fa82c159b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:50.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 shutdown_connections 2024-09-13T16:26:50.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:50.464+0000 7fa833316700 1 -- 172.21.15.119:0/4265426067 wait complete. 2024-09-13T16:26:50.471 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:26:50.482 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:26:50.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:50.482 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:26:50.490 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[38522]: fuse finished with error 0 and tester_r 0 2024-09-13T16:26:50.492 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:26:56.594 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:26:56.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:26:56.594 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:26:56.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:56.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 Processor -- start 2024-09-13T16:26:56.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- start start 2024-09-13T16:26:56.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b36f0 0x7f5fb00b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:56.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5fb00b1e80 0x7f5fb0159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:56.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb00b2320 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb00b2600 con 0x7f5fb00b1e80 2024-09-13T16:26:56.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00b28e0 con 0x7f5fb00b36f0 2024-09-13T16:26:56.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb5e14700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5fb00b3ad0 0x7f5fb015d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36164/0 (socket says 172.21.15.119:36164) 2024-09-13T16:26:56.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb4e12700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5fb00b1e80 0x7f5fb0159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:56.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb5e14700 1 -- 172.21.15.119:0/3617232363 learned_addr learned my addr 172.21.15.119:0/3617232363 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb5613700 1 --2- 172.21.15.119:0/3617232363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b36f0 0x7f5fb00b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fa7fff700 1 -- 172.21.15.119:0/3617232363 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913200357 0 0) 0x7f5fb00b2320 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fa7fff700 1 -- 172.21.15.119:0/3617232363 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f9c0035d0 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb4e12700 1 -- 172.21.15.119:0/3617232363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b36f0 msgr2=0x7f5fb00b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb4e12700 1 --2- 172.21.15.119:0/3617232363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b36f0 0x7f5fb00b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb4e12700 1 -- 172.21.15.119:0/3617232363 >> v1:172.21.15.119:6789/0 conn(0x7f5fb00b3ad0 legacy=0x7f5fb015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb4e12700 1 -- 172.21.15.119:0/3617232363 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb00b2bc0 con 0x7f5fb00b1e80 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb5613700 1 --2- 172.21.15.119:0/3617232363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b36f0 0x7f5fb00b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb4e12700 1 --2- 172.21.15.119:0/3617232363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b1e80 0x7f5fb0159340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009d90 tx=0x7f5fac00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f833cf76422fa14 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fa7fff700 1 -- 172.21.15.119:0/3617232363 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac00e070 con 0x7f5fb00b1e80 2024-09-13T16:26:56.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fa7fff700 1 -- 172.21.15.119:0/3617232363 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fac00bb20 con 0x7f5fb00b1e80 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fa7fff700 1 -- 172.21.15.119:0/3617232363 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fac00be90 con 0x7f5fb00b1e80 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- 172.21.15.119:0/3617232363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b1e80 msgr2=0x7f5fb0159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 --2- 172.21.15.119:0/3617232363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b1e80 0x7f5fb0159340 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f5fac009d90 tx=0x7f5fac00b330 comp rx=0 tx=0).stop 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- 172.21.15.119:0/3617232363 shutdown_connections 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 --2- 172.21.15.119:0/3617232363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b1e80 0x7f5fb0159340 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 --2- 172.21.15.119:0/3617232363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b36f0 0x7f5fb00b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- 172.21.15.119:0/3617232363 >> 172.21.15.119:0/3617232363 conn(0x7f5fb006a350 msgr2=0x7f5fb006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- 172.21.15.119:0/3617232363 shutdown_connections 2024-09-13T16:26:56.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.763+0000 7f5fb7877700 1 -- 172.21.15.119:0/3617232363 wait complete. 2024-09-13T16:26:56.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 Processor -- start 2024-09-13T16:26:56.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 -- start start 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b1e80 0x7f5fb014a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fb00b36f0 0x7f5fb01f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b3ad0 0x7f5fb006c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c9040 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c92e0 con 0x7f5fb00b36f0 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb7877700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb00c95c0 con 0x7f5fb00b1e80 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb5e14700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b3ad0 0x7f5fb006c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb5e14700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b3ad0 0x7f5fb006c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34350/0 (socket says 172.21.15.119:34350) 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb5e14700 1 -- 172.21.15.119:0/164271975 learned_addr learned my addr 172.21.15.119:0/164271975 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb5613700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b1e80 0x7f5fb014a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.767+0000 7f5fb4e12700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fb00b36f0 0x7f5fb01f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:56.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb5e14700 1 -- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b1e80 msgr2=0x7f5fb014a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb5e14700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b1e80 0x7f5fb014a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb5e14700 1 -- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fb00b36f0 msgr2=0x7f5fb01f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb5e14700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fb00b36f0 0x7f5fb01f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb5e14700 1 -- 172.21.15.119:0/164271975 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fac009a70 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb5613700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b1e80 0x7f5fb014a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb4e12700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fb00b36f0 0x7f5fb01f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb5e14700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b3ad0 0x7f5fb006c5f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800c430 tx=0x7f5fa800c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fa5ffb700 1 -- 172.21.15.119:0/164271975 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8005d90 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb00c98a0 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fa5ffb700 1 -- 172.21.15.119:0/164271975 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa80070d0 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fa5ffb700 1 -- 172.21.15.119:0/164271975 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8007460 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb014f6a0 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb0067760 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fa5ffb700 1 -- 172.21.15.119:0/164271975 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5fa80075c0 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.771+0000 7f5fa5ffb700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f94066730 0x7f5f94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:56.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.775+0000 7f5fa5ffb700 1 -- 172.21.15.119:0/164271975 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5fa80837c0 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.775+0000 7f5fb5613700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f94066730 0x7f5f94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:56.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.775+0000 7f5fb5613700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f94066730 0x7f5f94068bf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00037d0 tx=0x7f5fa0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:56.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.783+0000 7f5fa5ffb700 1 -- 172.21.15.119:0/164271975 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5fa8050a30 con 0x7f5fb00b3ad0 2024-09-13T16:26:56.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:26:56.954 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:26:56.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.991+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fb00b1940 con 0x7f5fb00b3ad0 2024-09-13T16:26:57.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fa5ffb700 1 -- 172.21.15.119:0/164271975 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f5fa8054050 con 0x7f5fb00b3ad0 2024-09-13T16:26:57.000 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:57.000 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:26:57.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f94066730 msgr2=0x7f5f94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f94066730 0x7f5f94068bf0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f5fa00037d0 tx=0x7f5fa0008040 comp rx=0 tx=0).stop 2024-09-13T16:26:57.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b3ad0 msgr2=0x7f5fb006c5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b3ad0 0x7f5fb006c5f0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f5fa800c430 tx=0x7f5fa800c7f0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 shutdown_connections 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f94066730 0x7f5f94068bf0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5fb00b3ad0 0x7f5fb006c5f0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5fb00b36f0 0x7f5fb01f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 --2- 172.21.15.119:0/164271975 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5fb00b1e80 0x7f5fb014a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 >> 172.21.15.119:0/164271975 conn(0x7f5fb006a350 msgr2=0x7f5fb01577e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 shutdown_connections 2024-09-13T16:26:57.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:56.995+0000 7f5fb7877700 1 -- 172.21.15.119:0/164271975 wait complete. 2024-09-13T16:26:57.004 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:26:57.020 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-09-13T16:26:57.165 INFO:teuthology.orchestra.run.smithi145.stdout:{"message":"","return_code":0} 2024-09-13T16:26:57.165 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:26:57.166 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:26:57.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041f59e700 1 Processor -- start 2024-09-13T16:26:57.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041f59e700 1 -- start start 2024-09-13T16:26:57.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041f59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f042006baa0 0x7f042006d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041f59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f042006d180 con 0x7f04200703d0 2024-09-13T16:26:57.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04200b2410 con 0x7f04200b1fd0 2024-09-13T16:26:57.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041f59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042006c360 con 0x7f042006baa0 2024-09-13T16:26:57.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041e59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f04200703d0 0x7f04200707b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36168/0 (socket says 172.21.15.119:36168) 2024-09-13T16:26:57.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041e59c700 1 -- 172.21.15.119:0/4038547473 learned_addr learned my addr 172.21.15.119:0/4038547473 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:26:57.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041ed9d700 1 --2- 172.21.15.119:0/4038547473 >> v2:172.21.15.145:3300/0 conn(0x7f042006baa0 0x7f042006d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914783059 0 0) 0x7f042006d180 con 0x7f04200703d0 2024-09-13T16:26:57.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.283+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04080035d0 con 0x7f04200703d0 2024-09-13T16:26:57.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2209586357 0 0) 0x7f04080035d0 con 0x7f04200703d0 2024-09-13T16:26:57.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 >> v2:172.21.15.145:3300/0 conn(0x7f042006baa0 msgr2=0x7f042006d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 --2- 172.21.15.119:0/4038547473 >> v2:172.21.15.145:3300/0 conn(0x7f042006baa0 0x7f042006d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 >> v1:172.21.15.145:6789/0 conn(0x7f04200b1fd0 legacy=0x7f042006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042006c570 con 0x7f04200703d0 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f0410001670 con 0x7f04200703d0 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 >> v1:172.21.15.119:6789/0 conn(0x7f04200703d0 legacy=0x7f04200707b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0408003a20 0x7f0408003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0408004870 0x7f0408004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0408005230 0x7f040800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042006c360 con 0x7f0408005230 2024-09-13T16:26:57.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040800c0d0 con 0x7f0408004870 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040800c290 con 0x7f0408003a20 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0408005230 0x7f040800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041e59c700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0408003a20 0x7f0408003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041dd9b700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0408004870 0x7f0408004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 -- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0408003a20 msgr2=0x7f0408003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0408003a20 0x7f0408003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 -- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0408004870 msgr2=0x7f0408004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0408004870 0x7f0408004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 -- 172.21.15.119:0/4038547473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04080035d0 con 0x7f0408005230 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041dd9b700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0408004870 0x7f0408004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 -- 172.21.15.119:0/4038547473 reap_dead start 2024-09-13T16:26:57.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041ed9d700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0408005230 0x7f040800ba30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0418009f90 tx=0x7f04080058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d815073538c7e41a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418003a40 con 0x7f0408005230 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041800f040 con 0x7f0408005230 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041d59a700 1 -- 172.21.15.119:0/4038547473 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418003e00 con 0x7f0408005230 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 -- 172.21.15.119:0/4038547473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0408005230 msgr2=0x7f040800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0408005230 0x7f040800ba30 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f0418009f90 tx=0x7f04080058a0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 -- 172.21.15.119:0/4038547473 shutdown_connections 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 --2- 172.21.15.119:0/4038547473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0408005230 0x7f040800ba30 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 -- 172.21.15.119:0/4038547473 >> 172.21.15.119:0/4038547473 conn(0x7f04200ae750 msgr2=0x7f04200b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 -- 172.21.15.119:0/4038547473 shutdown_connections 2024-09-13T16:26:57.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 -- 172.21.15.119:0/4038547473 wait complete. 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 Processor -- start 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.287+0000 7f041f59e700 1 -- start start 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04200cd5e0 0x7f04201f1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04200cd9c0 0x7f04201eebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0420159b30 0x7f0420159fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042015a520 con 0x7f0420159b30 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04201ef110 con 0x7f04200cd9c0 2024-09-13T16:26:57.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04201f2670 con 0x7f04200cd5e0 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0420159b30 0x7f0420159fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0420159b30 0x7f0420159fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34360/0 (socket says 172.21.15.119:34360) 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 -- 172.21.15.119:0/433839856 learned_addr learned my addr 172.21.15.119:0/433839856 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041e59c700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04200cd5e0 0x7f04201f1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041dd9b700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04200cd9c0 0x7f04201eebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 -- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04200cd5e0 msgr2=0x7f04201f1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04200cd5e0 0x7f04201f1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 -- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04200cd9c0 msgr2=0x7f04201eebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04200cd9c0 0x7f04201eebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 -- 172.21.15.119:0/433839856 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04201f1dd0 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041dd9b700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04200cd9c0 0x7f04201eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041ed9d700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0420159b30 0x7f0420159fe0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0420074020 tx=0x7f0418004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f040effd700 1 -- 172.21.15.119:0/433839856 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0418003e30 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f040effd700 1 -- 172.21.15.119:0/433839856 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0418004230 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04201f20b0 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f040effd700 1 -- 172.21.15.119:0/433839856 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04180045e0 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0420140f50 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042006b390 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.291+0000 7f040effd700 1 -- 172.21.15.119:0/433839856 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0418004740 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.295+0000 7f040effd700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0400066730 0x7f0400068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.295+0000 7f040effd700 1 -- 172.21.15.119:0/433839856 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f04180853a0 con 0x7f0420159b30 2024-09-13T16:26:57.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.295+0000 7f041e59c700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0400066730 0x7f0400068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:26:57.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.295+0000 7f041e59c700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0400066730 0x7f0400068bf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0410000c00 tx=0x7f0410000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:26:57.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.299+0000 7f040effd700 1 -- 172.21.15.119:0/433839856 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0418052700 con 0x7f0420159b30 2024-09-13T16:26:57.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.511+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0420071050 con 0x7f0420159b30 2024-09-13T16:26:57.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.511+0000 7f040effd700 1 -- 172.21.15.119:0/433839856 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f0418055d20 con 0x7f0420159b30 2024-09-13T16:26:57.518 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:26:57.518 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:26:57.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0400066730 msgr2=0x7f0400068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0400066730 0x7f0400068bf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f0410000c00 tx=0x7f0410000ed0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0420159b30 msgr2=0x7f0420159fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0420159b30 0x7f0420159fe0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f0420074020 tx=0x7f0418004b60 comp rx=0 tx=0).stop 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 shutdown_connections 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0400066730 0x7f0400068bf0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0420159b30 0x7f0420159fe0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04200cd9c0 0x7f04201eebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 --2- 172.21.15.119:0/433839856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04200cd5e0 0x7f04201f1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 >> 172.21.15.119:0/433839856 conn(0x7f0408005230 msgr2=0x7f04200b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 shutdown_connections 2024-09-13T16:26:57.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:26:57.515+0000 7f041f59e700 1 -- 172.21.15.119:0/433839856 wait complete. 2024-09-13T16:26:57.522 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:26:57.533 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json tag path /parentdir mytag 2024-09-13T16:26:57.663 INFO:teuthology.orchestra.run.smithi145.stdout:{"return_code":0,"scrub_tag":"mytag","mode":"asynchronous"} 2024-09-13T16:26:57.664 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag" } 2024-09-13T16:27:07.664 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000000.00000000 scrub_tag 2024-09-13T16:27:07.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 Processor -- start 2024-09-13T16:27:07.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 -- start start 2024-09-13T16:27:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed54f0 0x55c04efafb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed58d0 0x55c04ef212b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04efabd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c04ee692f0 con 0x55c04ef92c20 2024-09-13T16:27:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c04ee258e0 con 0x55c04eed54f0 2024-09-13T16:27:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644efef140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c04ee25580 con 0x55c04eed58d0 2024-09-13T16:27:07.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04efabd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04efabd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34982/0 (socket says 172.21.15.119:34982) 2024-09-13T16:27:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644cb24700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed54f0 0x55c04efafb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 -- 172.21.15.119:0/3720863035 learned_addr learned my addr 172.21.15.119:0/3720863035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644c323700 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed58d0 0x55c04ef212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 -- 172.21.15.119:0/3720863035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed58d0 msgr2=0x55c04ef212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed58d0 0x55c04ef212b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 -- 172.21.15.119:0/3720863035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed54f0 msgr2=0x55c04efafb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed54f0 0x55c04efafb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.703+0000 7f644d325700 1 -- 172.21.15.119:0/3720863035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f643c00a100 con 0x55c04ef92c20 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644d325700 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04efabd00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6444001ba0 tx=0x7f644400b2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28e3dc7a6be5aa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644b321700 1 -- 172.21.15.119:0/3720863035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644400bb40 con 0x55c04ef92c20 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644b321700 1 -- 172.21.15.119:0/3720863035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644400bca0 con 0x55c04ef92c20 2024-09-13T16:27:07.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644b321700 1 -- 172.21.15.119:0/3720863035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644400e690 con 0x55c04ef92c20 2024-09-13T16:27:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 -- 172.21.15.119:0/3720863035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 msgr2=0x55c04efabd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04efabd00 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f6444001ba0 tx=0x7f644400b2f0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 -- 172.21.15.119:0/3720863035 shutdown_connections 2024-09-13T16:27:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04efabd00 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed58d0 0x55c04ef212b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 --2- 172.21.15.119:0/3720863035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed54f0 0x55c04efafb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 -- 172.21.15.119:0/3720863035 >> 172.21.15.119:0/3720863035 conn(0x55c04eed6b10 msgr2=0x55c04efacdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 -- 172.21.15.119:0/3720863035 shutdown_connections 2024-09-13T16:27:07.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 -- 172.21.15.119:0/3720863035 wait complete. 2024-09-13T16:27:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 Processor -- start 2024-09-13T16:27:07.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.707+0000 7f644efef140 1 -- start start 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed54f0 0x55c04eface30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed58d0 0x55c04ef093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04ef09930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c04edfb8d0 con 0x55c04ef92c20 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c04ef1a1b0 con 0x55c04eed58d0 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c04efad370 con 0x55c04eed54f0 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04ef09930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04ef09930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34988/0 (socket says 172.21.15.119:34988) 2024-09-13T16:27:07.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 -- 172.21.15.119:0/1468603107 learned_addr learned my addr 172.21.15.119:0/1468603107 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644c323700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed58d0 0x55c04ef093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644cb24700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed54f0 0x55c04eface30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 -- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed54f0 msgr2=0x55c04eface30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c04eed54f0 0x55c04eface30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 -- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed58d0 msgr2=0x55c04ef093f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c04eed58d0 0x55c04ef093f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 -- 172.21.15.119:0/1468603107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c04ee06bb0 con 0x55c04ef92c20 2024-09-13T16:27:07.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644d325700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04ef09930 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6444011040 tx=0x7f6444000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644a31f700 1 -- 172.21.15.119:0/1468603107 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644401e070 con 0x55c04ef92c20 2024-09-13T16:27:07.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644a31f700 1 -- 172.21.15.119:0/1468603107 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644400b940 con 0x55c04ef92c20 2024-09-13T16:27:07.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c04ee13b00 con 0x55c04ef92c20 2024-09-13T16:27:07.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644a31f700 1 -- 172.21.15.119:0/1468603107 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6444023640 con 0x55c04ef92c20 2024-09-13T16:27:07.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.711+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c04ee06f40 con 0x55c04ef92c20 2024-09-13T16:27:07.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644a31f700 1 -- 172.21.15.119:0/1468603107 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f644402c440 con 0x55c04ef92c20 2024-09-13T16:27:07.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644a31f700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f642c065060 0x7f642c067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644a31f700 1 -- 172.21.15.119:0/1468603107 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6444084510 con 0x55c04ef92c20 2024-09-13T16:27:07.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644cb24700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f642c065060 0x7f642c067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c04ef80940 0x55c04ef80d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:1 5.10 5:0cf6ad0f:::10000000000.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55c04ef81290 con 0x55c04ef80940 2024-09-13T16:27:07.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644c323700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c04ef80940 0x55c04ef80d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644cb24700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f642c065060 0x7f642c067520 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f643c013040 tx=0x7f643c007030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644c323700 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c04ef80940 0x55c04ef80d50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644c323700 1 -- 172.21.15.119:0/1468603107 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(1 10000000000.00000000 [getxattr (9) out=9b] v0'0 uv11 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f6434012040 con 0x55c04ef80940 2024-09-13T16:27:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c04ef80940 msgr2=0x55c04ef80d50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c04ef80940 0x55c04ef80d50 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f642c065060 msgr2=0x7f642c067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f642c065060 0x7f642c067520 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f643c013040 tx=0x7f643c007030 comp rx=0 tx=0).stop 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 msgr2=0x55c04ef09930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 --2- 172.21.15.119:0/1468603107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c04ef92c20 0x55c04ef09930 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f6444011040 tx=0x7f6444000bc0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644d325700 1 -- 172.21.15.119:0/1468603107 reap_dead start 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 shutdown_connections 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 >> 172.21.15.119:0/1468603107 conn(0x55c04eed6b10 msgr2=0x55c04ef212b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.715+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 shutdown_connections 2024-09-13T16:27:07.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.719+0000 7f644efef140 1 -- 172.21.15.119:0/1468603107 wait complete. 2024-09-13T16:27:07.726 INFO:teuthology.orchestra.run.smithi119.stdout:mytag 2024-09-13T16:27:07.727 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata getxattr 10000000001.00000000 scrub_tag 2024-09-13T16:27:07.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 Processor -- start 2024-09-13T16:27:07.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 -- start start 2024-09-13T16:27:07.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789414f0 0x555678a1bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x555678aaa9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789ff330 0x555678aaaf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556788d9a00 con 0x5556789414f0 2024-09-13T16:27:07.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556788d52f0 con 0x55567898c600 2024-09-13T16:27:07.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.803+0000 7f0504c09140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556788918e0 con 0x5556789ff330 2024-09-13T16:27:07.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f050273e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789414f0 0x555678a1bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x555678aaa9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0502f3f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789ff330 0x555678aaaf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x555678aaa9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:37226/0 (socket says 172.21.15.119:37226) 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f050273e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789414f0 0x555678a1bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34996/0 (socket says 172.21.15.119:34996) 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 -- 172.21.15.119:0/1096959718 learned_addr learned my addr 172.21.15.119:0/1096959718 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 -- 172.21.15.119:0/1096959718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789ff330 msgr2=0x555678aaaf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789ff330 0x555678aaaf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 -- 172.21.15.119:0/1096959718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789414f0 msgr2=0x555678a1bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789414f0 0x555678a1bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 -- 172.21.15.119:0/1096959718 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04fc00a100 con 0x55567898c600 2024-09-13T16:27:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f050273e700 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789414f0 0x555678a1bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0501f3d700 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x555678aaa9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f04ec001cb0 tx=0x7f04ec00b5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f353d9583e95a17 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0500f3b700 1 -- 172.21.15.119:0/1096959718 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec00bd70 con 0x55567898c600 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0500f3b700 1 -- 172.21.15.119:0/1096959718 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec00e470 con 0x55567898c600 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0500f3b700 1 -- 172.21.15.119:0/1096959718 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec00e820 con 0x55567898c600 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 -- 172.21.15.119:0/1096959718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 msgr2=0x555678aaa9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x555678aaa9f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f04ec001cb0 tx=0x7f04ec00b5d0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 -- 172.21.15.119:0/1096959718 shutdown_connections 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789ff330 0x555678aaaf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x555678aaa9f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 --2- 172.21.15.119:0/1096959718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789414f0 0x555678a1bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 -- 172.21.15.119:0/1096959718 >> 172.21.15.119:0/1096959718 conn(0x555678942b10 msgr2=0x555678a18ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 -- 172.21.15.119:0/1096959718 shutdown_connections 2024-09-13T16:27:07.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.807+0000 7f0504c09140 1 -- 172.21.15.119:0/1096959718 wait complete. 2024-09-13T16:27:07.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 Processor -- start 2024-09-13T16:27:07.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 -- start start 2024-09-13T16:27:07.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789414f0 0x555678a17110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x55567898b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789ff330 0x55567898bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555678891580 con 0x5556789ff330 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556788678d0 con 0x55567898c600 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5556789f0160 con 0x5556789414f0 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789ff330 0x55567898bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789ff330 0x55567898bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35004/0 (socket says 172.21.15.119:35004) 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 -- 172.21.15.119:0/2955659453 learned_addr learned my addr 172.21.15.119:0/2955659453 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f050273e700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789414f0 0x555678a17110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0501f3d700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x55567898b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 -- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789414f0 msgr2=0x555678a17110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789414f0 0x555678a17110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 -- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 msgr2=0x55567898b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x55567898b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 -- 172.21.15.119:0/2955659453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f400a040 con 0x5556789ff330 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0501f3d700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55567898c600 0x55567898b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f050273e700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5556789414f0 0x555678a17110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0502f3f700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789ff330 0x55567898bc10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00ab30 tx=0x7f04fc003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f04fb7fe700 1 -- 172.21.15.119:0/2955659453 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc005840 con 0x5556789ff330 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f04fb7fe700 1 -- 172.21.15.119:0/2955659453 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04fc009030 con 0x5556789ff330 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0001e10 con 0x5556789ff330 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f04fb7fe700 1 -- 172.21.15.119:0/2955659453 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04fc010830 con 0x5556789ff330 2024-09-13T16:27:07.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.811+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555678872f40 con 0x5556789ff330 2024-09-13T16:27:07.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.815+0000 7f04fb7fe700 1 -- 172.21.15.119:0/2955659453 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04fc017030 con 0x5556789ff330 2024-09-13T16:27:07.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.815+0000 7f04fb7fe700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e4065060 0x7f04e4067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.819+0000 7f04fb7fe700 1 -- 172.21.15.119:0/2955659453 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f04fc083fe0 con 0x5556789ff330 2024-09-13T16:27:07.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.819+0000 7f050273e700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e4065060 0x7f04e4067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.819+0000 7f0504c09140 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5556789f1080 0x5556789f1490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.819+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:1 5.7 5:e2e07930:::10000000001.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x5556789f19d0 con 0x5556789f1080 2024-09-13T16:27:07.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.819+0000 7f0501f3d700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5556789f1080 0x5556789f1490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.819+0000 7f050273e700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e4065060 0x7f04e4067520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f04f40011e0 tx=0x7f04f4001810 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.819+0000 7f0501f3d700 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5556789f1080 0x5556789f1490 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.823+0000 7f0501f3d700 1 -- 172.21.15.119:0/2955659453 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (9) out=9b] v0'0 uv5 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f04ec021040 con 0x5556789f1080 2024-09-13T16:27:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.823+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5556789f1080 msgr2=0x5556789f1490 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.823+0000 7f0504c09140 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5556789f1080 0x5556789f1490 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e4065060 msgr2=0x7f04e4067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e4065060 0x7f04e4067520 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f04f40011e0 tx=0x7f04f4001810 comp rx=0 tx=0).stop 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789ff330 msgr2=0x55567898bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 --2- 172.21.15.119:0/2955659453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5556789ff330 0x55567898bc10 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f04fc00ab30 tx=0x7f04fc003330 comp rx=0 tx=0).stop 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0502f3f700 1 -- 172.21.15.119:0/2955659453 reap_dead start 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 shutdown_connections 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 >> 172.21.15.119:0/2955659453 conn(0x555678942b10 msgr2=0x555678a18c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 shutdown_connections 2024-09-13T16:27:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.827+0000 7f0504c09140 1 -- 172.21.15.119:0/2955659453 wait complete. 2024-09-13T16:27:07.835 INFO:teuthology.orchestra.run.smithi119.stdout:mytag 2024-09-13T16:27:07.836 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-09-13T16:27:07.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 Processor -- start 2024-09-13T16:27:07.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 -- start start 2024-09-13T16:27:07.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953104f0 0x55f09535c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f09535b600 0x55f0954799a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f0953ce330 0x55f095479ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0952a8a00 con 0x55f0953104f0 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0952a42f0 con 0x55f0953ce330 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676c68b140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f0952608e0 con 0x55f09535b600 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953104f0 0x55f09535c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953104f0 0x55f09535c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35016/0 (socket says 172.21.15.119:35016) 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 -- 172.21.15.119:0/3741343676 learned_addr learned my addr 172.21.15.119:0/3741343676 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a9c1700 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f0953ce330 0x55f095479ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f67699bf700 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f09535b600 0x55f0954799a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 -- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f09535b600 msgr2=0x55f0954799a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f09535b600 0x55f0954799a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 -- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f0953ce330 msgr2=0x55f095479ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f0953ce330 0x55f095479ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.875+0000 7f676a1c0700 1 -- 172.21.15.119:0/3741343676 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c00a100 con 0x55f0953104f0 2024-09-13T16:27:07.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676a1c0700 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953104f0 0x55f09535c480 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6764001d00 tx=0x7f676400b5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b508e78be464da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f67689bd700 1 -- 172.21.15.119:0/3741343676 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400bd30 con 0x55f0953104f0 2024-09-13T16:27:07.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f67689bd700 1 -- 172.21.15.119:0/3741343676 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f676400be90 con 0x55f0953104f0 2024-09-13T16:27:07.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f67689bd700 1 -- 172.21.15.119:0/3741343676 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f676400e6b0 con 0x55f0953104f0 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 -- 172.21.15.119:0/3741343676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953104f0 msgr2=0x55f09535c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953104f0 0x55f09535c480 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f6764001d00 tx=0x7f676400b5d0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 -- 172.21.15.119:0/3741343676 shutdown_connections 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f0953ce330 0x55f095479ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f09535b600 0x55f0954799a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 --2- 172.21.15.119:0/3741343676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953104f0 0x55f09535c480 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 -- 172.21.15.119:0/3741343676 >> 172.21.15.119:0/3741343676 conn(0x55f095311b10 msgr2=0x55f0953ea810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 -- 172.21.15.119:0/3741343676 shutdown_connections 2024-09-13T16:27:07.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 -- 172.21.15.119:0/3741343676 wait complete. 2024-09-13T16:27:07.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 Processor -- start 2024-09-13T16:27:07.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.879+0000 7f676c68b140 1 -- start start 2024-09-13T16:27:07.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f0953104f0 0x55f095354e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f09535b600 0x55f095355370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953ce330 0x55f0953bee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f095260580 con 0x55f0953ce330 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0952368d0 con 0x55f09535b600 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f09547d480 con 0x55f0953104f0 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953ce330 0x55f0953bee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f67699bf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f09535b600 0x55f095355370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a1c0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f0953104f0 0x55f095354e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953ce330 0x55f0953bee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35024/0 (socket says 172.21.15.119:35024) 2024-09-13T16:27:07.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 -- 172.21.15.119:0/981599210 learned_addr learned my addr 172.21.15.119:0/981599210 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 -- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f0953104f0 msgr2=0x55f095354e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f0953104f0 0x55f095354e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 -- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f09535b600 msgr2=0x55f095355370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f09535b600 0x55f095355370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 -- 172.21.15.119:0/981599210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6758001e10 con 0x55f0953ce330 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a1c0700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f0953104f0 0x55f095354e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f67699bf700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f09535b600 0x55f095355370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676a9c1700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953ce330 0x55f0953bee90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f675c00a7f0 tx=0x7f675c003330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f67637fe700 1 -- 172.21.15.119:0/981599210 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c026680 con 0x55f0953ce330 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f67637fe700 1 -- 172.21.15.119:0/981599210 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c01f070 con 0x55f0953ce330 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f095241f40 con 0x55f0953ce330 2024-09-13T16:27:07.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f67637fe700 1 -- 172.21.15.119:0/981599210 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c026a20 con 0x55f0953ce330 2024-09-13T16:27:07.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.883+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0952419a0 con 0x55f0953ce330 2024-09-13T16:27:07.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f67637fe700 1 -- 172.21.15.119:0/981599210 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f675c017040 con 0x55f0953ce330 2024-09-13T16:27:07.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f67637fe700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f674c065060 0x7f674c067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f67637fe700 1 -- 172.21.15.119:0/981599210 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f675c021070 con 0x55f0953ce330 2024-09-13T16:27:07.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f676c68b140 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55f0953e2a00 0x55f09547e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:1 6.36 6:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55f09547edb0 con 0x55f0953e2a00 2024-09-13T16:27:07.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f676a1c0700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f674c065060 0x7f674c067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f67699bf700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55f0953e2a00 0x55f09547e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f676a1c0700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f674c065060 0x7f674c067520 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6764001bf0 tx=0x7f676401b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.887+0000 7f67699bf700 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55f0953e2a00 0x55f09547e870 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f67699bf700 1 -- 172.21.15.119:0/981599210 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr (9) out=9b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f6754001890 con 0x55f0953e2a00 2024-09-13T16:27:07.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55f0953e2a00 msgr2=0x55f09547e870 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55f0953e2a00 0x55f09547e870 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f674c065060 msgr2=0x7f674c067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f674c065060 0x7f674c067520 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6764001bf0 tx=0x7f676401b040 comp rx=0 tx=0).stop 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953ce330 msgr2=0x55f0953bee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 --2- 172.21.15.119:0/981599210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f0953ce330 0x55f0953bee90 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f675c00a7f0 tx=0x7f675c003330 comp rx=0 tx=0).stop 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676a9c1700 1 -- 172.21.15.119:0/981599210 reap_dead start 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 shutdown_connections 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 >> 172.21.15.119:0/981599210 conn(0x55f095311b10 msgr2=0x55f09535c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 shutdown_connections 2024-09-13T16:27:07.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.891+0000 7f676c68b140 1 -- 172.21.15.119:0/981599210 wait complete. 2024-09-13T16:27:07.900 INFO:teuthology.orchestra.run.smithi119.stdout:mytag 2024-09-13T16:27:07.901 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-09-13T16:27:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.935+0000 7f31e3093140 1 Processor -- start 2024-09-13T16:27:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.935+0000 7f31e3093140 1 -- start start 2024-09-13T16:27:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db226a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 0x558db2326fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db23d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558db2200a00 con 0x558db22684f0 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x558db21fc2f0 con 0x558db2325c20 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x558db21b88e0 con 0x558db22688d0 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db226a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db226a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35032/0 (socket says 172.21.15.119:35032) 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 -- 172.21.15.119:0/3631329896 learned_addr learned my addr 172.21.15.119:0/3631329896 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e03c7700 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 0x558db2326fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e13c9700 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db23d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 -- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 msgr2=0x558db2326fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 0x558db2326fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 -- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 msgr2=0x558db23d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db23d1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 -- 172.21.15.119:0/3631329896 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d8005d20 con 0x558db22684f0 2024-09-13T16:27:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e0bc8700 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db226a010 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f31d8015040 tx=0x7f31d800d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd176b94e4ea4f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31df3c5700 1 -- 172.21.15.119:0/3631329896 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8008ac0 con 0x558db22684f0 2024-09-13T16:27:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31df3c5700 1 -- 172.21.15.119:0/3631329896 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d8008c20 con 0x558db22684f0 2024-09-13T16:27:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31df3c5700 1 -- 172.21.15.119:0/3631329896 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d8012410 con 0x558db22684f0 2024-09-13T16:27:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- 172.21.15.119:0/3631329896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 msgr2=0x558db226a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db226a010 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f31d8015040 tx=0x7f31d800d000 comp rx=0 tx=0).stop 2024-09-13T16:27:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- 172.21.15.119:0/3631329896 shutdown_connections 2024-09-13T16:27:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db23d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 0x558db2326fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 --2- 172.21.15.119:0/3631329896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db226a010 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- 172.21.15.119:0/3631329896 >> 172.21.15.119:0/3631329896 conn(0x558db2269b10 msgr2=0x558db233fdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- 172.21.15.119:0/3631329896 shutdown_connections 2024-09-13T16:27:07.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.939+0000 7f31e3093140 1 -- 172.21.15.119:0/3631329896 wait complete. 2024-09-13T16:27:07.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 Processor -- start 2024-09-13T16:27:07.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 -- start start 2024-09-13T16:27:07.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db22ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 0x558db2326fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db2327500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x558db21b8580 con 0x558db22684f0 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x558db218e8d0 con 0x558db2325c20 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e3093140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x558db23d6f30 con 0x558db22688d0 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e0bc8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db22ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e13c9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db2327500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e0bc8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db22ace30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35046/0 (socket says 172.21.15.119:35046) 2024-09-13T16:27:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e0bc8700 1 -- 172.21.15.119:0/2139479201 learned_addr learned my addr 172.21.15.119:0/2139479201 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.943+0000 7f31e03c7700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 0x558db2326fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e13c9700 1 -- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 msgr2=0x558db2326fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e13c9700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x558db22688d0 0x558db2326fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e13c9700 1 -- 172.21.15.119:0/2139479201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 msgr2=0x558db22ace30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e13c9700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x558db22684f0 0x558db22ace30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e13c9700 1 -- 172.21.15.119:0/2139479201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d800ae70 con 0x558db2325c20 2024-09-13T16:27:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e13c9700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db2327500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f31d0001560 tx=0x7f31d000d440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31de3c3700 1 -- 172.21.15.119:0/2139479201 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d00062e0 con 0x558db2325c20 2024-09-13T16:27:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31de3c3700 1 -- 172.21.15.119:0/2139479201 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31d000d970 con 0x558db2325c20 2024-09-13T16:27:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x558db2199f40 con 0x558db2325c20 2024-09-13T16:27:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31de3c3700 1 -- 172.21.15.119:0/2139479201 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31d000dd00 con 0x558db2325c20 2024-09-13T16:27:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558db21999a0 con 0x558db2325c20 2024-09-13T16:27:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31de3c3700 1 -- 172.21.15.119:0/2139479201 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f31d0004500 con 0x558db2325c20 2024-09-13T16:27:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.947+0000 7f31de3c3700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0065060 0x7f31c0067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31de3c3700 1 -- 172.21.15.119:0/2139479201 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f31d0082400 con 0x558db2325c20 2024-09-13T16:27:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e0bc8700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0065060 0x7f31c0067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x558db23d7e50 0x558db23d8260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:1 6.19 6:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x558db23e1af0 con 0x558db23d7e50 2024-09-13T16:27:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e03c7700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x558db23d7e50 0x558db23d8260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e0bc8700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0065060 0x7f31c0067520 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f31d800a520 tx=0x7f31d8026040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e03c7700 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x558db23d7e50 0x558db23d8260 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e03c7700 1 -- 172.21.15.119:0/2139479201 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (9) out=9b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+9 (crc 0 0 0) 0x7f31c8001950 con 0x558db23d7e50 2024-09-13T16:27:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x558db23d7e50 msgr2=0x558db23d8260 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x558db23d7e50 0x558db23d8260 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0065060 msgr2=0x7f31c0067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f31c0065060 0x7f31c0067520 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f31d800a520 tx=0x7f31d8026040 comp rx=0 tx=0).stop 2024-09-13T16:27:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 msgr2=0x558db2327500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 --2- 172.21.15.119:0/2139479201 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x558db2325c20 0x558db2327500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f31d0001560 tx=0x7f31d000d440 comp rx=0 tx=0).stop 2024-09-13T16:27:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e13c9700 1 -- 172.21.15.119:0/2139479201 reap_dead start 2024-09-13T16:27:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 shutdown_connections 2024-09-13T16:27:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 >> 172.21.15.119:0/2139479201 conn(0x558db2269b10 msgr2=0x558db23299e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 shutdown_connections 2024-09-13T16:27:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.951+0000 7f31e3093140 1 -- 172.21.15.119:0/2139479201 wait complete. 2024-09-13T16:27:07.961 INFO:teuthology.orchestra.run.smithi119.stdout:mytag 2024-09-13T16:27:07.962 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-09-13T16:27:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.995+0000 7f81b4906140 1 Processor -- start 2024-09-13T16:27:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.995+0000 7f81b4906140 1 -- start start 2024-09-13T16:27:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d18f4f0 0x55d11d266a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 0x55d11d1db640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d24d330 0x55d11d2f8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11d1232f0 con 0x55d11d24d330 2024-09-13T16:27:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11d0df8e0 con 0x55d11d18f8d0 2024-09-13T16:27:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d11d0df580 con 0x55d11d18f4f0 2024-09-13T16:27:08.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d24d330 0x55d11d2f8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d24d330 0x55d11d2f8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35068/0 (socket says 172.21.15.119:35068) 2024-09-13T16:27:08.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 -- 172.21.15.119:0/1530162805 learned_addr learned my addr 172.21.15.119:0/1530162805 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:08.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b1c3a700 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 0x55d11d1db640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b243b700 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d18f4f0 0x55d11d266a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 -- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d18f4f0 msgr2=0x55d11d266a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d18f4f0 0x55d11d266a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 -- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 msgr2=0x55d11d1db640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 0x55d11d1db640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 -- 172.21.15.119:0/1530162805 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a4005d20 con 0x55d11d24d330 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b243b700 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d18f4f0 0x55d11d266a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b2c3c700 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d24d330 0x55d11d2f8770 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f81a400ac00 tx=0x7f81a400d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eba7c104f6471e4e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b0c38700 1 -- 172.21.15.119:0/1530162805 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4008b60 con 0x55d11d24d330 2024-09-13T16:27:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b0c38700 1 -- 172.21.15.119:0/1530162805 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4008cc0 con 0x55d11d24d330 2024-09-13T16:27:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b0c38700 1 -- 172.21.15.119:0/1530162805 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81a4012610 con 0x55d11d24d330 2024-09-13T16:27:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- 172.21.15.119:0/1530162805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d24d330 msgr2=0x55d11d2f8770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d24d330 0x55d11d2f8770 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f81a400ac00 tx=0x7f81a400d000 comp rx=0 tx=0).stop 2024-09-13T16:27:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- 172.21.15.119:0/1530162805 shutdown_connections 2024-09-13T16:27:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d24d330 0x55d11d2f8770 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 0x55d11d1db640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 --2- 172.21.15.119:0/1530162805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d18f4f0 0x55d11d266a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- 172.21.15.119:0/1530162805 >> 172.21.15.119:0/1530162805 conn(0x55d11d190b10 msgr2=0x55d11d1c38e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- 172.21.15.119:0/1530162805 shutdown_connections 2024-09-13T16:27:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:07.999+0000 7f81b4906140 1 -- 172.21.15.119:0/1530162805 wait complete. 2024-09-13T16:27:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 Processor -- start 2024-09-13T16:27:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 -- start start 2024-09-13T16:27:08.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d18f4f0 0x55d11d1d9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 0x55d11d266a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d24d330 0x55d11d2ffc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11d0b58d0 con 0x55d11d18f4f0 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d11d266240 con 0x55d11d18f8d0 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d11d2664e0 con 0x55d11d24d330 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d18f4f0 0x55d11d1d9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d18f4f0 0x55d11d1d9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35084/0 (socket says 172.21.15.119:35084) 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 -- 172.21.15.119:0/1084166124 learned_addr learned my addr 172.21.15.119:0/1084166124 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b2c3c700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d24d330 0x55d11d2ffc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b1c3a700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 0x55d11d266a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 -- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d24d330 msgr2=0x55d11d2ffc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d11d24d330 0x55d11d2ffc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 -- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 msgr2=0x55d11d266a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d11d18f8d0 0x55d11d266a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 -- 172.21.15.119:0/1084166124 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d11d0e7ca0 con 0x55d11d18f4f0 2024-09-13T16:27:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b243b700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d18f4f0 0x55d11d1d9820 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00df80 tx=0x7f81ac00b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81ab7fe700 1 -- 172.21.15.119:0/1084166124 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac010410 con 0x55d11d18f4f0 2024-09-13T16:27:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d11d0cdb00 con 0x55d11d18f4f0 2024-09-13T16:27:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81ab7fe700 1 -- 172.21.15.119:0/1084166124 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81ac010570 con 0x55d11d18f4f0 2024-09-13T16:27:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81ab7fe700 1 -- 172.21.15.119:0/1084166124 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81ac0108e0 con 0x55d11d18f4f0 2024-09-13T16:27:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.003+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d11d0c0f40 con 0x55d11d18f4f0 2024-09-13T16:27:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.007+0000 7f81ab7fe700 1 -- 172.21.15.119:0/1084166124 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f81ac010a40 con 0x55d11d18f4f0 2024-09-13T16:27:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.007+0000 7f81ab7fe700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8198065090 0x7f8198067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.007+0000 7f81ab7fe700 1 -- 172.21.15.119:0/1084166124 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f81ac00a040 con 0x55d11d18f4f0 2024-09-13T16:27:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.007+0000 7f81b4906140 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55d11d266fb0 0x55d11d300690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.007+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:1 6.12 6:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e42) v8 -- 0x55d11d1d5ad0 con 0x55d11d266fb0 2024-09-13T16:27:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.007+0000 7f81b1c3a700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8198065090 0x7f8198067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.007+0000 7f81b2c3c700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55d11d266fb0 0x55d11d300690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b1c3a700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8198065090 0x7f8198067550 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f819c005fd0 tx=0x7f819c007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b2c3c700 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55d11d266fb0 0x55d11d300690 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b2c3c700 1 -- 172.21.15.119:0/1084166124 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7f81a4019040 con 0x55d11d266fb0 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:error getting xattr cephfs_data/10000000002.00000000/scrub_tag: (61) No data available 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55d11d266fb0 msgr2=0x55d11d300690 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55d11d266fb0 0x55d11d300690 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8198065090 msgr2=0x7f8198067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8198065090 0x7f8198067550 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f819c005fd0 tx=0x7f819c007040 comp rx=0 tx=0).stop 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d18f4f0 msgr2=0x55d11d1d9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 --2- 172.21.15.119:0/1084166124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d11d18f4f0 0x55d11d1d9820 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f81ac00df80 tx=0x7f81ac00b5a0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b2c3c700 1 -- 172.21.15.119:0/1084166124 reap_dead start 2024-09-13T16:27:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 shutdown_connections 2024-09-13T16:27:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 >> 172.21.15.119:0/1084166124 conn(0x55d11d190b10 msgr2=0x55d11d1d3e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 shutdown_connections 2024-09-13T16:27:08.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.011+0000 7f81b4906140 1 -- 172.21.15.119:0/1084166124 wait complete. 2024-09-13T16:27:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:27:08.020 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:27:08.020 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:27:08.038 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:27:08.057 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:27:08.058 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:27:08.058 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:08.086 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:27:08.086 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:27:08.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:08.086 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:27:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T16:27:08.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:27:08.093 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:27:08.093 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:08.101 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:27:08.101 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:27:08.101 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:08.101 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:27:08.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:27:08.108 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-13T16:27:08.109 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:27:08.109 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:08.117 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:27:08.117 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:27:08.117 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:08.117 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:27:08.165 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-09-13T16:27:08.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:27:08.166 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:27:08.166 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:08.173 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:27:08.173 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:27:08.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:08.174 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:27:08.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:27:08.181 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-13T16:27:08.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:08.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7b59e700 1 Processor -- start 2024-09-13T16:27:08.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7b59e700 1 -- start start 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c06cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7b59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1e7c06d520 0x7f1e7c0b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7b59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e7c06d940 con 0x7f1e7c160cc0 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7b59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e7c0b2aa0 con 0x7f1e7c06d520 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c0b4e30 con 0x7f1e7c0b49f0 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7ad9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1e7c06d520 0x7f1e7c0b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e79d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c06cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7a59c700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f1e7c160cc0 0x7f1e7c164960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:39952/0 (socket says 172.21.15.119:39952) 2024-09-13T16:27:08.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7a59c700 1 -- 172.21.15.119:0/424082270 learned_addr learned my addr 172.21.15.119:0/424082270 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:08.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e79d9b700 1 -- 172.21.15.119:0/424082270 >> v2:172.21.15.145:3300/0 conn(0x7f1e7c06d520 msgr2=0x7f1e7c0b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e79d9b700 1 --2- 172.21.15.119:0/424082270 >> v2:172.21.15.145:3300/0 conn(0x7f1e7c06d520 0x7f1e7c0b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e79d9b700 1 -- 172.21.15.119:0/424082270 >> v1:172.21.15.145:6789/0 conn(0x7f1e7c160cc0 legacy=0x7f1e7c164960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e79d9b700 1 -- 172.21.15.119:0/424082270 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e7c165120 con 0x7f1e7c0b49f0 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e79d9b700 1 --2- 172.21.15.119:0/424082270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c06cfe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009d90 tx=0x7f1e7400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e5bf070d29d76e8e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7959a700 1 -- 172.21.15.119:0/424082270 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7400e070 con 0x7f1e7c0b49f0 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7959a700 1 -- 172.21.15.119:0/424082270 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e7400bb20 con 0x7f1e7c0b49f0 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.299+0000 7f1e7959a700 1 -- 172.21.15.119:0/424082270 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74012430 con 0x7f1e7c0b49f0 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 -- 172.21.15.119:0/424082270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 msgr2=0x7f1e7c06cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/424082270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c06cfe0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1e74009d90 tx=0x7f1e7400b330 comp rx=0 tx=0).stop 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 -- 172.21.15.119:0/424082270 shutdown_connections 2024-09-13T16:27:08.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/424082270 >> v2:172.21.15.145:3300/0 conn(0x7f1e7c06d520 0x7f1e7c0b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/424082270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c06cfe0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 -- 172.21.15.119:0/424082270 >> 172.21.15.119:0/424082270 conn(0x7f1e7c0b17a0 msgr2=0x7f1e7c0b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:08.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 -- 172.21.15.119:0/424082270 shutdown_connections 2024-09-13T16:27:08.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 -- 172.21.15.119:0/424082270 wait complete. 2024-09-13T16:27:08.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 Processor -- start 2024-09-13T16:27:08.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 -- start start 2024-09-13T16:27:08.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e7c06d520 0x7f1e7c0cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c159a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.303+0000 7f1e7b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e7c160cc0 0x7f1e7c0cc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7b59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c159fc0 con 0x7f1e7c160cc0 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7b59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c15a1b0 con 0x7f1e7c06d520 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e7c15a490 con 0x7f1e7c0b49f0 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e7c160cc0 0x7f1e7c0cc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e7c160cc0 0x7f1e7c0cc0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35094/0 (socket says 172.21.15.119:35094) 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 -- 172.21.15.119:0/2634486190 learned_addr learned my addr 172.21.15.119:0/2634486190 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7a59c700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e7c06d520 0x7f1e7c0cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e79d9b700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c159a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 -- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 msgr2=0x7f1e7c159a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c159a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 -- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e7c06d520 msgr2=0x7f1e7c0cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e7c06d520 0x7f1e7c0cf2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 -- 172.21.15.119:0/2634486190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e74009a70 con 0x7f1e7c160cc0 2024-09-13T16:27:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7a59c700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e7c06d520 0x7f1e7c0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:08.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7ad9d700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e7c160cc0 0x7f1e7c0cc0f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400eb60 tx=0x7f1e6400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e72ffd700 1 -- 172.21.15.119:0/2634486190 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e64009740 con 0x7f1e7c160cc0 2024-09-13T16:27:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e72ffd700 1 -- 172.21.15.119:0/2634486190 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e6400c850 con 0x7f1e7c160cc0 2024-09-13T16:27:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e7c15a770 con 0x7f1e7c160cc0 2024-09-13T16:27:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e72ffd700 1 -- 172.21.15.119:0/2634486190 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e6400cbe0 con 0x7f1e7c160cc0 2024-09-13T16:27:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e7c162660 con 0x7f1e7c160cc0 2024-09-13T16:27:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.307+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e7c0cf7f0 con 0x7f1e7c160cc0 2024-09-13T16:27:08.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.311+0000 7f1e72ffd700 1 -- 172.21.15.119:0/2634486190 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1e6400cd40 con 0x7f1e7c160cc0 2024-09-13T16:27:08.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.311+0000 7f1e72ffd700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e5c066730 0x7f1e5c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.311+0000 7f1e72ffd700 1 -- 172.21.15.119:0/2634486190 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f1e64084400 con 0x7f1e7c160cc0 2024-09-13T16:27:08.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.311+0000 7f1e7a59c700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e5c066730 0x7f1e5c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.311+0000 7f1e7a59c700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e5c066730 0x7f1e5c068bf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c007fd0 tx=0x7f1e6c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.315+0000 7f1e72ffd700 1 -- 172.21.15.119:0/2634486190 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1e640514f0 con 0x7f1e7c160cc0 2024-09-13T16:27:08.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.523+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e7c06cfe0 con 0x7f1e7c160cc0 2024-09-13T16:27:08.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.523+0000 7f1e72ffd700 1 -- 172.21.15.119:0/2634486190 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f1e64054b10 con 0x7f1e7c160cc0 2024-09-13T16:27:08.530 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:08.530 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:27:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e5c066730 msgr2=0x7f1e5c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e5c066730 0x7f1e5c068bf0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f1e6c007fd0 tx=0x7f1e6c009040 comp rx=0 tx=0).stop 2024-09-13T16:27:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e7c160cc0 msgr2=0x7f1e7c0cc0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e7c160cc0 0x7f1e7c0cc0f0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400eb60 tx=0x7f1e6400ee70 comp rx=0 tx=0).stop 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 shutdown_connections 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e5c066730 0x7f1e5c068bf0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e7c160cc0 0x7f1e7c0cc0f0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e7c0b49f0 0x7f1e7c159a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 --2- 172.21.15.119:0/2634486190 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e7c06d520 0x7f1e7c0cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 >> 172.21.15.119:0/2634486190 conn(0x7f1e7c0b17a0 msgr2=0x7f1e7c070e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 shutdown_connections 2024-09-13T16:27:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.527+0000 7f1e7b59e700 1 -- 172.21.15.119:0/2634486190 wait complete. 2024-09-13T16:27:08.536 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:27:08.551 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:08.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d5bb48700 1 Processor -- start 2024-09-13T16:27:08.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d5bb48700 1 -- start start 2024-09-13T16:27:08.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d5bb48700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d541586b0 0x7f4d54070a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d5bb48700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d540ac1c0 con 0x7f4d5406bac0 2024-09-13T16:27:08.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d5bb48700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d540ac6f0 con 0x7f4d540ac8c0 2024-09-13T16:27:08.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d5bb48700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d5406aa60 con 0x7f4d541586b0 2024-09-13T16:27:08.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d590e3700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4d5406bac0 0x7f4d5406a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56910/0 (socket says 172.21.15.119:56910) 2024-09-13T16:27:08.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d590e3700 1 -- 172.21.15.119:0/375943962 learned_addr learned my addr 172.21.15.119:0/375943962 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:08.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d598e4700 1 --2- 172.21.15.119:0/375943962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d541586b0 0x7f4d54070a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d588e2700 1 -- 172.21.15.119:0/375943962 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2841537642 0 0) 0x7f4d540ac1c0 con 0x7f4d5406bac0 2024-09-13T16:27:08.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d588e2700 1 -- 172.21.15.119:0/375943962 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d400035d0 con 0x7f4d5406bac0 2024-09-13T16:27:08.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d588e2700 1 -- 172.21.15.119:0/375943962 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956043304 0 0) 0x7f4d540ac6f0 con 0x7f4d540ac8c0 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d588e2700 1 -- 172.21.15.119:0/375943962 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d540ac1c0 con 0x7f4d540ac8c0 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d598e4700 1 -- 172.21.15.119:0/375943962 >> v1:172.21.15.145:6789/0 conn(0x7f4d540ac8c0 legacy=0x7f4d540b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d598e4700 1 -- 172.21.15.119:0/375943962 >> v1:172.21.15.119:6789/0 conn(0x7f4d5406bac0 legacy=0x7f4d5406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d598e4700 1 -- 172.21.15.119:0/375943962 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d540b0f00 con 0x7f4d541586b0 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d598e4700 1 --2- 172.21.15.119:0/375943962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d541586b0 0x7f4d54070a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009880 tx=0x7f4d50009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a74f4d5271804a28 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d588e2700 1 -- 172.21.15.119:0/375943962 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50011830 con 0x7f4d541586b0 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d588e2700 1 -- 172.21.15.119:0/375943962 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d50011990 con 0x7f4d541586b0 2024-09-13T16:27:08.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.659+0000 7f4d588e2700 1 -- 172.21.15.119:0/375943962 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d50011d00 con 0x7f4d541586b0 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- 172.21.15.119:0/375943962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d541586b0 msgr2=0x7f4d54070a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/375943962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d541586b0 0x7f4d54070a60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4d50009880 tx=0x7f4d50009b90 comp rx=0 tx=0).stop 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- 172.21.15.119:0/375943962 shutdown_connections 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/375943962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d541586b0 0x7f4d54070a60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- 172.21.15.119:0/375943962 >> 172.21.15.119:0/375943962 conn(0x7f4d540b2de0 msgr2=0x7f4d540b4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- 172.21.15.119:0/375943962 shutdown_connections 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- 172.21.15.119:0/375943962 wait complete. 2024-09-13T16:27:08.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 Processor -- start 2024-09-13T16:27:08.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- start start 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d5406bac0 0x7f4d54151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d540ac8c0 0x7f4d5415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d541586b0 0x7f4d5406d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54155010 con 0x7f4d541586b0 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d541554a0 con 0x7f4d5406bac0 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.663+0000 7f4d5bb48700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d54154440 con 0x7f4d540ac8c0 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d541586b0 0x7f4d5406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d590e3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d540ac8c0 0x7f4d5415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d541586b0 0x7f4d5406d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35110/0 (socket says 172.21.15.119:35110) 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 -- 172.21.15.119:0/1407452 learned_addr learned my addr 172.21.15.119:0/1407452 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:08.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d598e4700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d5406bac0 0x7f4d54151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 -- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d540ac8c0 msgr2=0x7f4d5415a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d540ac8c0 0x7f4d5415a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 -- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d5406bac0 msgr2=0x7f4d54151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d5406bac0 0x7f4d54151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 -- 172.21.15.119:0/1407452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d500094b0 con 0x7f4d541586b0 2024-09-13T16:27:08.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5a0e5700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d541586b0 0x7f4d5406d610 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00cc50 tx=0x7f4d4c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d4a7fc700 1 -- 172.21.15.119:0/1407452 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c003c20 con 0x7f4d541586b0 2024-09-13T16:27:08.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d4a7fc700 1 -- 172.21.15.119:0/1407452 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d4c003da0 con 0x7f4d541586b0 2024-09-13T16:27:08.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d4a7fc700 1 -- 172.21.15.119:0/1407452 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c00aad0 con 0x7f4d541586b0 2024-09-13T16:27:08.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d541546e0 con 0x7f4d541586b0 2024-09-13T16:27:08.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d54154ca0 con 0x7f4d541586b0 2024-09-13T16:27:08.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.667+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d54155180 con 0x7f4d541586b0 2024-09-13T16:27:08.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.671+0000 7f4d4a7fc700 1 -- 172.21.15.119:0/1407452 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4d4c00ac30 con 0x7f4d541586b0 2024-09-13T16:27:08.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.671+0000 7f4d4a7fc700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d38066760 0x7f4d38068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.671+0000 7f4d4a7fc700 1 -- 172.21.15.119:0/1407452 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4d4c0842b0 con 0x7f4d541586b0 2024-09-13T16:27:08.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.671+0000 7f4d598e4700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d38066760 0x7f4d38068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.671+0000 7f4d598e4700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d38066760 0x7f4d38068c20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4d500090b0 tx=0x7f4d50010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.675+0000 7f4d4a7fc700 1 -- 172.21.15.119:0/1407452 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4d4c0514b0 con 0x7f4d541586b0 2024-09-13T16:27:08.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.851+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d54070a60 con 0x7f4d541586b0 2024-09-13T16:27:08.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.851+0000 7f4d4a7fc700 1 -- 172.21.15.119:0/1407452 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f4d4c054ad0 con 0x7f4d541586b0 2024-09-13T16:27:08.858 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:08.858 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":42,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:07.530304+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.7","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:08.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d38066760 msgr2=0x7f4d38068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d38066760 0x7f4d38068c20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f4d500090b0 tx=0x7f4d50010000 comp rx=0 tx=0).stop 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d541586b0 msgr2=0x7f4d5406d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d541586b0 0x7f4d5406d610 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c00cc50 tx=0x7f4d4c00cf60 comp rx=0 tx=0).stop 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 shutdown_connections 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d38066760 0x7f4d38068c20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d541586b0 0x7f4d5406d610 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d540ac8c0 0x7f4d5415a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 --2- 172.21.15.119:0/1407452 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d5406bac0 0x7f4d54151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 >> 172.21.15.119:0/1407452 conn(0x7f4d540b2de0 msgr2=0x7f4d5406aba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 shutdown_connections 2024-09-13T16:27:08.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.855+0000 7f4d5bb48700 1 -- 172.21.15.119:0/1407452 wait complete. 2024-09-13T16:27:08.873 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:26:35.251936+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '39', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:26:36.423580+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '40', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:27:08.873 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:27:08.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99359e700 1 Processor -- start 2024-09-13T16:27:08.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99359e700 1 -- start start 2024-09-13T16:27:08.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd9940ce5f0 0x7fd9940cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9940d0fb0 0x7fd9940d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:08.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99359e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9940cd600 con 0x7fd99415cae0 2024-09-13T16:27:08.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9940ce1d0 con 0x7fd9940ce5f0 2024-09-13T16:27:08.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9940cea60 con 0x7fd9940d0fb0 2024-09-13T16:27:08.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99259c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd99415cae0 0x7fd9940d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56920/0 (socket says 172.21.15.119:56920) 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99259c700 1 -- 172.21.15.119:0/2726416235 learned_addr learned my addr 172.21.15.119:0/2726416235 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd992d9d700 1 --2- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9940d0fb0 0x7fd9940d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd991d9b700 1 --2- 172.21.15.119:0/2726416235 >> v2:172.21.15.145:3300/0 conn(0x7fd9940ce5f0 0x7fd9940cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99159a700 1 -- 172.21.15.119:0/2726416235 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943019124 0 0) 0x7fd9940cd600 con 0x7fd99415cae0 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99159a700 1 -- 172.21.15.119:0/2726416235 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9800035d0 con 0x7fd99415cae0 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd991d9b700 1 -- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9940d0fb0 msgr2=0x7fd9940d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd991d9b700 1 --2- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9940d0fb0 0x7fd9940d1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd991d9b700 1 -- 172.21.15.119:0/2726416235 >> v1:172.21.15.119:6789/0 conn(0x7fd99415cae0 legacy=0x7fd9940d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd991d9b700 1 -- 172.21.15.119:0/2726416235 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9940d0dd0 con 0x7fd9940ce5f0 2024-09-13T16:27:08.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd991d9b700 1 --2- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 0x7fd9940cff60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009880 tx=0x7fd97c009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c37410dbe4308aec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99159a700 1 -- 172.21.15.119:0/2726416235 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c011830 con 0x7fd9940ce5f0 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99159a700 1 -- 172.21.15.119:0/2726416235 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c011990 con 0x7fd9940ce5f0 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.987+0000 7fd99159a700 1 -- 172.21.15.119:0/2726416235 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c011d00 con 0x7fd9940ce5f0 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 -- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 msgr2=0x7fd9940cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 --2- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 0x7fd9940cff60 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd97c009880 tx=0x7fd97c009b90 comp rx=0 tx=0).stop 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 -- 172.21.15.119:0/2726416235 shutdown_connections 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 --2- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9940d0fb0 0x7fd9940d1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 --2- 172.21.15.119:0/2726416235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 0x7fd9940cff60 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 -- 172.21.15.119:0/2726416235 >> 172.21.15.119:0/2726416235 conn(0x7fd9940d29f0 msgr2=0x7fd9940d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 -- 172.21.15.119:0/2726416235 shutdown_connections 2024-09-13T16:27:08.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.991+0000 7fd99359e700 1 -- 172.21.15.119:0/2726416235 wait complete. 2024-09-13T16:27:08.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 Processor -- start 2024-09-13T16:27:08.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 -- start start 2024-09-13T16:27:09.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 0x7fd9940b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9940d0fb0 0x7fd9941497b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99415cae0 0x7fd994149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99415eaa0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd99414a230 con 0x7fd9940ce5f0 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd99415de20 con 0x7fd99415cae0 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9940d0fb0 0x7fd9941497b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9940d0fb0 0x7fd9941497b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35116/0 (socket says 172.21.15.119:35116) 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 -- 172.21.15.119:0/2472424763 learned_addr learned my addr 172.21.15.119:0/2472424763 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd992d9d700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99415cae0 0x7fd994149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99259c700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 0x7fd9940b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 -- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99415cae0 msgr2=0x7fd994149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99415cae0 0x7fd994149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 -- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 msgr2=0x7fd9940b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 0x7fd9940b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 -- 172.21.15.119:0/2472424763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97c0094b0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd991d9b700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9940d0fb0 0x7fd9941497b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd97c014040 tx=0x7fd97c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd98affd700 1 -- 172.21.15.119:0/2472424763 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c0044d0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd98affd700 1 -- 172.21.15.119:0/2472424763 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd97c01e070 con 0x7fd9940d0fb0 2024-09-13T16:27:09.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99415e100 con 0x7fd9940d0fb0 2024-09-13T16:27:09.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd98affd700 1 -- 172.21.15.119:0/2472424763 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd97c0038c0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.995+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99415e6c0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.999+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99415ebe0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.999+0000 7fd98affd700 1 -- 172.21.15.119:0/2472424763 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd97c003a20 con 0x7fd9940d0fb0 2024-09-13T16:27:09.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.999+0000 7fd98affd700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd974066730 0x7fd974068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.999+0000 7fd98affd700 1 -- 172.21.15.119:0/2472424763 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd97c085150 con 0x7fd9940d0fb0 2024-09-13T16:27:09.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:08.999+0000 7fd99259c700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd974066730 0x7fd974068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.003+0000 7fd99259c700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd974066730 0x7fd974068bf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd984007fd0 tx=0x7fd984000bc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.007+0000 7fd98affd700 1 -- 172.21.15.119:0/2472424763 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd99415ebe0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd99415ebe0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd98affd700 1 -- 172.21.15.119:0/2472424763 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v42) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd99415ebe0 con 0x7fd9940d0fb0 2024-09-13T16:27:09.184 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd974066730 msgr2=0x7fd974068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd974066730 0x7fd974068bf0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd984007fd0 tx=0x7fd984000bc0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9940d0fb0 msgr2=0x7fd9941497b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9940d0fb0 0x7fd9941497b0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd97c014040 tx=0x7fd97c004770 comp rx=0 tx=0).stop 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 shutdown_connections 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd974066730 0x7fd974068bf0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd99415cae0 0x7fd994149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9940d0fb0 0x7fd9941497b0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 --2- 172.21.15.119:0/2472424763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9940ce5f0 0x7fd9940b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 >> 172.21.15.119:0/2472424763 conn(0x7fd9940d29f0 msgr2=0x7fd9940af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 shutdown_connections 2024-09-13T16:27:09.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.179+0000 7fd99359e700 1 -- 172.21.15.119:0/2472424763 wait complete. 2024-09-13T16:27:09.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:27:09.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.311+0000 7fd6aa790700 1 Processor -- start 2024-09-13T16:27:09.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.311+0000 7fd6aa790700 1 -- start start 2024-09-13T16:27:09.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd6a415cae0 0x7fd6a40ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6a40cd600 con 0x7fd6a40cff60 2024-09-13T16:27:09.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6a40ce1d0 con 0x7fd6a415cae0 2024-09-13T16:27:09.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40d03d0 con 0x7fd6a40d0fb0 2024-09-13T16:27:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a9f8f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a978e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd6a415cae0 0x7fd6a40ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a8f8d700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd6a40cff60 0x7fd6a40d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:39958/0 (socket says 172.21.15.119:39958) 2024-09-13T16:27:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a8f8d700 1 -- 172.21.15.119:0/3878214712 learned_addr learned my addr 172.21.15.119:0/3878214712 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd697fff700 1 -- 172.21.15.119:0/3878214712 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794812013 0 0) 0x7fd6a40cd600 con 0x7fd6a40cff60 2024-09-13T16:27:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd697fff700 1 -- 172.21.15.119:0/3878214712 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6900035d0 con 0x7fd6a40cff60 2024-09-13T16:27:09.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a9f8f700 1 -- 172.21.15.119:0/3878214712 >> v2:172.21.15.145:3300/0 conn(0x7fd6a415cae0 msgr2=0x7fd6a40ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:09.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a9f8f700 1 --2- 172.21.15.119:0/3878214712 >> v2:172.21.15.145:3300/0 conn(0x7fd6a415cae0 0x7fd6a40ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a9f8f700 1 -- 172.21.15.119:0/3878214712 >> v1:172.21.15.145:6789/0 conn(0x7fd6a40cff60 legacy=0x7fd6a40d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a9f8f700 1 -- 172.21.15.119:0/3878214712 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a40d0dd0 con 0x7fd6a40d0fb0 2024-09-13T16:27:09.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a978e700 1 --2- 172.21.15.119:0/3878214712 >> v2:172.21.15.145:3300/0 conn(0x7fd6a415cae0 0x7fd6a40ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:09.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6a9f8f700 1 --2- 172.21.15.119:0/3878214712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40d1420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009d90 tx=0x7fd69c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f51337d7ad3b47 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd697fff700 1 -- 172.21.15.119:0/3878214712 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c00e070 con 0x7fd6a40d0fb0 2024-09-13T16:27:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd697fff700 1 -- 172.21.15.119:0/3878214712 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69c00bb20 con 0x7fd6a40d0fb0 2024-09-13T16:27:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd697fff700 1 -- 172.21.15.119:0/3878214712 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69c012430 con 0x7fd6a40d0fb0 2024-09-13T16:27:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- 172.21.15.119:0/3878214712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 msgr2=0x7fd6a40d1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 --2- 172.21.15.119:0/3878214712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40d1420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd69c009d90 tx=0x7fd69c00b330 comp rx=0 tx=0).stop 2024-09-13T16:27:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- 172.21.15.119:0/3878214712 shutdown_connections 2024-09-13T16:27:09.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 --2- 172.21.15.119:0/3878214712 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40d1420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 --2- 172.21.15.119:0/3878214712 >> v2:172.21.15.145:3300/0 conn(0x7fd6a415cae0 0x7fd6a40ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- 172.21.15.119:0/3878214712 >> 172.21.15.119:0/3878214712 conn(0x7fd6a40d29f0 msgr2=0x7fd6a40d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:09.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- 172.21.15.119:0/3878214712 shutdown_connections 2024-09-13T16:27:09.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.315+0000 7fd6aa790700 1 -- 172.21.15.119:0/3878214712 wait complete. 2024-09-13T16:27:09.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.319+0000 7fd6aa790700 1 Processor -- start 2024-09-13T16:27:09.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.319+0000 7fd6aa790700 1 -- start start 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.319+0000 7fd6aa790700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a40cff60 0x7fd6a40c6080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6a415cae0 0x7fd6a415b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40c6660 con 0x7fd6a40cff60 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40b81e0 con 0x7fd6a415cae0 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a40c9810 con 0x7fd6a40d0fb0 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a40cff60 0x7fd6a40c6080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a40cff60 0x7fd6a40c6080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35128/0 (socket says 172.21.15.119:35128) 2024-09-13T16:27:09.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 -- 172.21.15.119:0/4203560557 learned_addr learned my addr 172.21.15.119:0/4203560557 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a9f8f700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6a415cae0 0x7fd6a415b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a8f8d700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 -- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 msgr2=0x7fd6a40c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 -- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6a415cae0 msgr2=0x7fd6a415b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6a415cae0 0x7fd6a415b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 -- 172.21.15.119:0/4203560557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6980094b0 con 0x7fd6a40cff60 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a9f8f700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6a415cae0 0x7fd6a415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:09.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6a978e700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a40cff60 0x7fd6a40c6080 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd69800ec10 tx=0x7fd698009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd695ffb700 1 -- 172.21.15.119:0/4203560557 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800c940 con 0x7fd6a40cff60 2024-09-13T16:27:09.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd695ffb700 1 -- 172.21.15.119:0/4203560557 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69800caa0 con 0x7fd6a40cff60 2024-09-13T16:27:09.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd69c009a70 con 0x7fd6a40cff60 2024-09-13T16:27:09.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd695ffb700 1 -- 172.21.15.119:0/4203560557 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69800ce30 con 0x7fd6a40cff60 2024-09-13T16:27:09.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a40c8e40 con 0x7fd6a40cff60 2024-09-13T16:27:09.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.323+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6a4073590 con 0x7fd6a40cff60 2024-09-13T16:27:09.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.327+0000 7fd695ffb700 1 -- 172.21.15.119:0/4203560557 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd698013470 con 0x7fd6a40cff60 2024-09-13T16:27:09.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.327+0000 7fd695ffb700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6880667b0 0x7fd688068c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.327+0000 7fd695ffb700 1 -- 172.21.15.119:0/4203560557 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd698084bc0 con 0x7fd6a40cff60 2024-09-13T16:27:09.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.327+0000 7fd6a8f8d700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6880667b0 0x7fd688068c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.327+0000 7fd6a8f8d700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6880667b0 0x7fd688068c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4066890 tx=0x7fd6a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.327+0000 7fd695ffb700 1 -- 172.21.15.119:0/4203560557 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd698051eb0 con 0x7fd6a40cff60 2024-09-13T16:27:09.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.499+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd6a415cf50 con 0x7fd6a40cff60 2024-09-13T16:27:09.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.499+0000 7fd695ffb700 1 -- 172.21.15.119:0/4203560557 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd6980210a0 con 0x7fd6a40cff60 2024-09-13T16:27:09.507 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6880667b0 msgr2=0x7fd688068c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6880667b0 0x7fd688068c70 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd6a4066890 tx=0x7fd6a0009040 comp rx=0 tx=0).stop 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a40cff60 msgr2=0x7fd6a40c6080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a40cff60 0x7fd6a40c6080 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd69800ec10 tx=0x7fd698009ed0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 shutdown_connections 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd6880667b0 0x7fd688068c70 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd6a415cae0 0x7fd6a415b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd6a40d0fb0 0x7fd6a40c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 --2- 172.21.15.119:0/4203560557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd6a40cff60 0x7fd6a40c6080 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 >> 172.21.15.119:0/4203560557 conn(0x7fd6a40d29f0 msgr2=0x7fd6a40d05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:09.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 shutdown_connections 2024-09-13T16:27:09.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.503+0000 7fd6aa790700 1 -- 172.21.15.119:0/4203560557 wait complete. 2024-09-13T16:27:09.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:27:09.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39acb32700 1 Processor -- start 2024-09-13T16:27:09.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39acb32700 1 -- start start 2024-09-13T16:27:09.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39acb32700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f39a80ade80 0x7f39a81652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39acb32700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39acb32700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39a8071ac0 con 0x7f39a81612d0 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39acb32700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39a81657f0 con 0x7f39a80ade80 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39acb32700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80b2e20 con 0x7f39a80b24a0 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a7fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f39a81612d0 0x7f39a8161780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56922/0 (socket says 172.21.15.119:56922) 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a7fff700 1 -- 172.21.15.119:0/4290691433 learned_addr learned my addr 172.21.15.119:0/4290691433 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a6ffd700 1 --2- 172.21.15.119:0/4290691433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a77fe700 1 --2- 172.21.15.119:0/4290691433 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ade80 0x7f39a81652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a67fc700 1 -- 172.21.15.119:0/4290691433 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831139984 0 0) 0x7f39a8071ac0 con 0x7f39a81612d0 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a67fc700 1 -- 172.21.15.119:0/4290691433 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39900035d0 con 0x7f39a81612d0 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a6ffd700 1 -- 172.21.15.119:0/4290691433 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ade80 msgr2=0x7f39a81652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a6ffd700 1 --2- 172.21.15.119:0/4290691433 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ade80 0x7f39a81652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a6ffd700 1 -- 172.21.15.119:0/4290691433 >> v1:172.21.15.119:6789/0 conn(0x7f39a81612d0 legacy=0x7f39a8161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a6ffd700 1 -- 172.21.15.119:0/4290691433 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a8070eb0 con 0x7f39a80b24a0 2024-09-13T16:27:09.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a6ffd700 1 --2- 172.21.15.119:0/4290691433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80b28e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f399c009850 tx=0x7f399c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6bc9627135cc907 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a67fc700 1 -- 172.21.15.119:0/4290691433 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c011830 con 0x7f39a80b24a0 2024-09-13T16:27:09.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a67fc700 1 -- 172.21.15.119:0/4290691433 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f399c011990 con 0x7f39a80b24a0 2024-09-13T16:27:09.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.643+0000 7f39a67fc700 1 -- 172.21.15.119:0/4290691433 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f399c011d00 con 0x7f39a80b24a0 2024-09-13T16:27:09.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- 172.21.15.119:0/4290691433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 msgr2=0x7f39a80b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 --2- 172.21.15.119:0/4290691433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80b28e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f399c009850 tx=0x7f399c009b60 comp rx=0 tx=0).stop 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- 172.21.15.119:0/4290691433 shutdown_connections 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 --2- 172.21.15.119:0/4290691433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80b28e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 --2- 172.21.15.119:0/4290691433 >> v2:172.21.15.145:3300/0 conn(0x7f39a80ade80 0x7f39a81652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- 172.21.15.119:0/4290691433 >> 172.21.15.119:0/4290691433 conn(0x7f39a80b0980 msgr2=0x7f39a80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- 172.21.15.119:0/4290691433 shutdown_connections 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- 172.21.15.119:0/4290691433 wait complete. 2024-09-13T16:27:09.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 Processor -- start 2024-09-13T16:27:09.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- start start 2024-09-13T16:27:09.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80ade80 0x7f39a815ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80aafa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a81612d0 0x7f39a8164b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80aab80 con 0x7f39a81612d0 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39a80acf20 con 0x7f39a80ade80 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.647+0000 7f39acb32700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39a815cf40 con 0x7f39a80b24a0 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a81612d0 0x7f39a8164b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a81612d0 0x7f39a8164b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35134/0 (socket says 172.21.15.119:35134) 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 -- 172.21.15.119:0/3396779404 learned_addr learned my addr 172.21.15.119:0/3396779404 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a6ffd700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80aafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a77fe700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80ade80 0x7f39a815ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 -- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 msgr2=0x7f39a80aafa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80aafa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 -- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80ade80 msgr2=0x7f39a815ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80ade80 0x7f39a815ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 -- 172.21.15.119:0/3396779404 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399c0094b0 con 0x7f39a81612d0 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39a7fff700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a81612d0 0x7f39a8164b70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f39a000cb40 tx=0x7f39a000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f398ffff700 1 -- 172.21.15.119:0/3396779404 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a0007500 con 0x7f39a81612d0 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f398ffff700 1 -- 172.21.15.119:0/3396779404 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39a0007660 con 0x7f39a81612d0 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f398ffff700 1 -- 172.21.15.119:0/3396779404 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39a00079f0 con 0x7f39a81612d0 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39a81f7140 con 0x7f39a81612d0 2024-09-13T16:27:09.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39a80aa4c0 con 0x7f39a81612d0 2024-09-13T16:27:09.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39a80afc10 con 0x7f39a81612d0 2024-09-13T16:27:09.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f398ffff700 1 -- 172.21.15.119:0/3396779404 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f39a001b3b0 con 0x7f39a81612d0 2024-09-13T16:27:09.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.651+0000 7f398ffff700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3994066760 0x7f3994068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.655+0000 7f398ffff700 1 -- 172.21.15.119:0/3396779404 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f39a0083c30 con 0x7f39a81612d0 2024-09-13T16:27:09.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.655+0000 7f39a77fe700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3994066760 0x7f3994068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.655+0000 7f39a77fe700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3994066760 0x7f3994068c20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f39a80cab60 tx=0x7f3998009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.659+0000 7f398ffff700 1 -- 172.21.15.119:0/3396779404 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f39a0050ed0 con 0x7f39a81612d0 2024-09-13T16:27:09.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.823+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f39a80cad30 con 0x7f39a81612d0 2024-09-13T16:27:09.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.823+0000 7f398ffff700 1 -- 172.21.15.119:0/3396779404 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+11 (secure 0 0 0) 0x7f39a0017020 con 0x7f39a81612d0 2024-09-13T16:27:09.829 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:27:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.823+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3994066760 msgr2=0x7f3994068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.823+0000 7f39acb32700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3994066760 0x7f3994068c20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f39a80cab60 tx=0x7f3998009240 comp rx=0 tx=0).stop 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a81612d0 msgr2=0x7f39a8164b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a81612d0 0x7f39a8164b70 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f39a000cb40 tx=0x7f39a000cb70 comp rx=0 tx=0).stop 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 shutdown_connections 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3994066760 0x7f3994068c20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f39a81612d0 0x7f39a8164b70 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f39a80b24a0 0x7f39a80aafa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 --2- 172.21.15.119:0/3396779404 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f39a80ade80 0x7f39a815ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 >> 172.21.15.119:0/3396779404 conn(0x7f39a80b0980 msgr2=0x7f39a80ae960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 shutdown_connections 2024-09-13T16:27:09.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.827+0000 7f39acb32700 1 -- 172.21.15.119:0/3396779404 wait complete. 2024-09-13T16:27:09.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.951+0000 7fcf1d359700 1 Processor -- start 2024-09-13T16:27:09.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.951+0000 7fcf1d359700 1 -- start start 2024-09-13T16:27:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcf1806e6a0 0x7fcf1806ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 0x7fcf180adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf1806cff0 con 0x7fcf1806be70 2024-09-13T16:27:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf180b5b40 con 0x7fcf1806e6a0 2024-09-13T16:27:09.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1806d480 con 0x7fcf180ad7e0 2024-09-13T16:27:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcf1806be70 0x7fcf180b5430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56926/0 (socket says 172.21.15.119:56926) 2024-09-13T16:27:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 -- 172.21.15.119:0/3366656516 learned_addr learned my addr 172.21.15.119:0/3366656516 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1cb58700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 0x7fcf180adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf17fff700 1 --2- 172.21.15.119:0/3366656516 >> v2:172.21.15.145:3300/0 conn(0x7fcf1806e6a0 0x7fcf1806ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825400529 0 0) 0x7fcf1806cff0 con 0x7fcf1806be70 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf000035d0 con 0x7fcf1806be70 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 950301736 0 0) 0x7fcf000035d0 con 0x7fcf1806be70 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 msgr2=0x7fcf180adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 0x7fcf180adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 >> v2:172.21.15.145:3300/0 conn(0x7fcf1806e6a0 msgr2=0x7fcf1806ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 --2- 172.21.15.119:0/3366656516 >> v2:172.21.15.145:3300/0 conn(0x7fcf1806e6a0 0x7fcf1806ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf1806efc0 con 0x7fcf1806be70 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fcf0c001670 con 0x7fcf1806be70 2024-09-13T16:27:09.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 >> v1:172.21.15.119:6789/0 conn(0x7fcf1806be70 legacy=0x7fcf180b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00003a20 0x7fcf00003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf00004870 0x7fcf00004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 0x7fcf00005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180b5b40 con 0x7fcf00004870 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1806d480 con 0x7fcf00003a20 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf00004870 0x7fcf00004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0000ba30 con 0x7fcf180ad7e0 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf17fff700 1 --2- 172.21.15.119:0/3366656516 >> v2:172.21.15.145:3300/0 conn(0x7fcf1806e6a0 0x7fcf1806ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1cb58700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 0x7fcf00005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 -- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 msgr2=0x7fcf00005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 0x7fcf00005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 -- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00003a20 msgr2=0x7fcf00003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00003a20 0x7fcf00003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 -- 172.21.15.119:0/3366656516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf000035d0 con 0x7fcf00004870 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1cb58700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf180ad7e0 0x7fcf00005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf177fe700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf00004870 0x7fcf00004cf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00c2a0 tx=0x7fcf000058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9249b23b7f84594c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c007c80 con 0x7fcf00004870 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c00ce60 con 0x7fcf00004870 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf16ffd700 1 -- 172.21.15.119:0/3366656516 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c005bc0 con 0x7fcf00004870 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- 172.21.15.119:0/3366656516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf00004870 msgr2=0x7fcf00004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 --2- 172.21.15.119:0/3366656516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf00004870 0x7fcf00004cf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c00c2a0 tx=0x7fcf000058a0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1cb58700 1 -- 172.21.15.119:0/3366656516 reap_dead start 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- 172.21.15.119:0/3366656516 shutdown_connections 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- 172.21.15.119:0/3366656516 >> 172.21.15.119:0/3366656516 conn(0x7fcf180b5d20 msgr2=0x7fcf180b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- 172.21.15.119:0/3366656516 shutdown_connections 2024-09-13T16:27:09.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.955+0000 7fcf1d359700 1 -- 172.21.15.119:0/3366656516 wait complete. 2024-09-13T16:27:09.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 Processor -- start 2024-09-13T16:27:09.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 -- start start 2024-09-13T16:27:09.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00004870 0x7fcf180cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18150410 0x7fcf181567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf181547a0 0x7fcf18156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180cbff0 con 0x7fcf18150410 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180cc4e0 con 0x7fcf00004870 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf18151750 con 0x7fcf181547a0 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18150410 0x7fcf181567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18150410 0x7fcf181567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35158/0 (socket says 172.21.15.119:35158) 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 -- 172.21.15.119:0/2487018441 learned_addr learned my addr 172.21.15.119:0/2487018441 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf17fff700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00004870 0x7fcf180cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1cb58700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf181547a0 0x7fcf18156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 -- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf181547a0 msgr2=0x7fcf18156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf181547a0 0x7fcf18156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 -- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00004870 msgr2=0x7fcf180cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00004870 0x7fcf180cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:09.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 -- 172.21.15.119:0/2487018441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf180cc200 con 0x7fcf18150410 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1cb58700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf181547a0 0x7fcf18156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf177fe700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18150410 0x7fcf181567c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0018d0 tx=0x7fcf0c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf14ff9700 1 -- 172.21.15.119:0/2487018441 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c005db0 con 0x7fcf18150410 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf14ff9700 1 -- 172.21.15.119:0/2487018441 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c0070d0 con 0x7fcf18150410 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf18151100 con 0x7fcf18150410 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf14ff9700 1 -- 172.21.15.119:0/2487018441 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf0c007480 con 0x7fcf18150410 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.959+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf18153ec0 con 0x7fcf18150410 2024-09-13T16:27:09.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.963+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf18074560 con 0x7fcf18150410 2024-09-13T16:27:09.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.963+0000 7fcf14ff9700 1 -- 172.21.15.119:0/2487018441 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcf0c019030 con 0x7fcf18150410 2024-09-13T16:27:09.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.963+0000 7fcf14ff9700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf04066760 0x7fcf04068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:09.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.963+0000 7fcf14ff9700 1 -- 172.21.15.119:0/2487018441 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fcf0c085c70 con 0x7fcf18150410 2024-09-13T16:27:09.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.963+0000 7fcf17fff700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf04066760 0x7fcf04068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:09.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.963+0000 7fcf17fff700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf04066760 0x7fcf04068c20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009c90 tx=0x7fcf08008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:09.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:09.967+0000 7fcf14ff9700 1 -- 172.21.15.119:0/2487018441 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcf0c052ef0 con 0x7fcf18150410 2024-09-13T16:27:10.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.179+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf181518e0 con 0x7fcf18150410 2024-09-13T16:27:10.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.179+0000 7fcf14ff9700 1 -- 172.21.15.119:0/2487018441 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7fcf0c007de0 con 0x7fcf18150410 2024-09-13T16:27:10.185 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:10.185 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf04066760 msgr2=0x7fcf04068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf04066760 0x7fcf04068c20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fcf08009c90 tx=0x7fcf08008040 comp rx=0 tx=0).stop 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18150410 msgr2=0x7fcf181567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18150410 0x7fcf181567c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fcf0c0018d0 tx=0x7fcf0c001a00 comp rx=0 tx=0).stop 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 shutdown_connections 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf04066760 0x7fcf04068c20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf181547a0 0x7fcf18156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf18150410 0x7fcf181567c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 --2- 172.21.15.119:0/2487018441 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf00004870 0x7fcf180cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 >> 172.21.15.119:0/2487018441 conn(0x7fcf180b5d20 msgr2=0x7fcf1815c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 shutdown_connections 2024-09-13T16:27:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.183+0000 7fcf1d359700 1 -- 172.21.15.119:0/2487018441 wait complete. 2024-09-13T16:27:10.189 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:27:10.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:10.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45bfff700 1 Processor -- start 2024-09-13T16:27:10.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45bfff700 1 -- start start 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb45c158b30 0x7fb45c071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c06d540 0x7fb45c071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45bfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45c06de00 con 0x7fb45c158f10 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45bfff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45c06db20 con 0x7fb45c158b30 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45bfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c0b1920 con 0x7fb45c06d540 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb45c158b30 0x7fb45c071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb45c158b30 0x7fb45c071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35174/0 (socket says 172.21.15.119:35174) 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45a7fc700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fb45c158f10 0x7fb45c06ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:39972/0 (socket says 172.21.15.119:39972) 2024-09-13T16:27:10.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45b7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c06d540 0x7fb45c071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 -- 172.21.15.119:0/1841453495 learned_addr learned my addr 172.21.15.119:0/1841453495 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:10.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 -- 172.21.15.119:0/1841453495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c06d540 msgr2=0x7fb45c071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:10.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 --2- 172.21.15.119:0/1841453495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c06d540 0x7fb45c071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 -- 172.21.15.119:0/1841453495 >> v1:172.21.15.145:6789/0 conn(0x7fb45c158f10 legacy=0x7fb45c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb459ffb700 1 -- 172.21.15.119:0/1841453495 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588600766 0 0) 0x7fb45c06de00 con 0x7fb45c158f10 2024-09-13T16:27:10.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 -- 172.21.15.119:0/1841453495 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb45c0b1c00 con 0x7fb45c158b30 2024-09-13T16:27:10.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.307+0000 7fb45affd700 1 --2- 172.21.15.119:0/1841453495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158b30 0x7fb45c071030 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb44400bbb0 tx=0x7fb44400bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8752dda5856218b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb459ffb700 1 -- 172.21.15.119:0/1841453495 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44400dd30 con 0x7fb45c158b30 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb459ffb700 1 -- 172.21.15.119:0/1841453495 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb44400de90 con 0x7fb45c158b30 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb459ffb700 1 -- 172.21.15.119:0/1841453495 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb444010690 con 0x7fb45c158b30 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- 172.21.15.119:0/1841453495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158b30 msgr2=0x7fb45c071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 --2- 172.21.15.119:0/1841453495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158b30 0x7fb45c071030 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7fb44400bbb0 tx=0x7fb44400bec0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- 172.21.15.119:0/1841453495 shutdown_connections 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 --2- 172.21.15.119:0/1841453495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c06d540 0x7fb45c071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 --2- 172.21.15.119:0/1841453495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158b30 0x7fb45c071030 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- 172.21.15.119:0/1841453495 >> 172.21.15.119:0/1841453495 conn(0x7fb45c06a350 msgr2=0x7fb45c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- 172.21.15.119:0/1841453495 shutdown_connections 2024-09-13T16:27:10.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- 172.21.15.119:0/1841453495 wait complete. 2024-09-13T16:27:10.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 Processor -- start 2024-09-13T16:27:10.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- start start 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb45c06d540 0x7fb45c1ee100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c158b30 0x7fb45c15b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158f10 0x7fb45c150460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c15ba40 con 0x7fb45c158f10 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c1509a0 con 0x7fb45c06d540 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45bfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45c1592f0 con 0x7fb45c158b30 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158f10 0x7fb45c150460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45affd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb45c06d540 0x7fb45c1ee100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158f10 0x7fb45c150460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35184/0 (socket says 172.21.15.119:35184) 2024-09-13T16:27:10.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45b7fe700 1 -- 172.21.15.119:0/2204309057 learned_addr learned my addr 172.21.15.119:0/2204309057 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:10.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.311+0000 7fb45a7fc700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c158b30 0x7fb45c15b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45b7fe700 1 -- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c158b30 msgr2=0x7fb45c15b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45b7fe700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c158b30 0x7fb45c15b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45b7fe700 1 -- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb45c06d540 msgr2=0x7fb45c1ee100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45b7fe700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb45c06d540 0x7fb45c1ee100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45b7fe700 1 -- 172.21.15.119:0/2204309057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44400b810 con 0x7fb45c158f10 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45b7fe700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158f10 0x7fb45c150460 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb45400eb60 tx=0x7fb45400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb453fff700 1 -- 172.21.15.119:0/2204309057 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb454009740 con 0x7fb45c158f10 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb453fff700 1 -- 172.21.15.119:0/2204309057 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb454009e00 con 0x7fb45c158f10 2024-09-13T16:27:10.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb45c15c1c0 con 0x7fb45c158f10 2024-09-13T16:27:10.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb453fff700 1 -- 172.21.15.119:0/2204309057 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45400ca70 con 0x7fb45c158f10 2024-09-13T16:27:10.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb45c159540 con 0x7fb45c158f10 2024-09-13T16:27:10.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45c15c5f0 con 0x7fb45c158f10 2024-09-13T16:27:10.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.315+0000 7fb453fff700 1 -- 172.21.15.119:0/2204309057 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb45400cbd0 con 0x7fb45c158f10 2024-09-13T16:27:10.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.319+0000 7fb453fff700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb43c066760 0x7fb43c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.319+0000 7fb453fff700 1 -- 172.21.15.119:0/2204309057 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fb454084200 con 0x7fb45c158f10 2024-09-13T16:27:10.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.319+0000 7fb45affd700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb43c066760 0x7fb43c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.319+0000 7fb45affd700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb43c066760 0x7fb43c068c20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb44400bfd0 tx=0x7fb444013000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.327+0000 7fb453fff700 1 -- 172.21.15.119:0/2204309057 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb454051410 con 0x7fb45c158f10 2024-09-13T16:27:10.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.499+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb45c06d160 con 0x7fb45c158f10 2024-09-13T16:27:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.499+0000 7fb453fff700 1 -- 172.21.15.119:0/2204309057 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fb454054a30 con 0x7fb45c158f10 2024-09-13T16:27:10.507 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:10.507 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":42,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:07.530304+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.7","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb43c066760 msgr2=0x7fb43c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb43c066760 0x7fb43c068c20 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb44400bfd0 tx=0x7fb444013000 comp rx=0 tx=0).stop 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158f10 msgr2=0x7fb45c150460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158f10 0x7fb45c150460 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7fb45400eb60 tx=0x7fb45400ee70 comp rx=0 tx=0).stop 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 shutdown_connections 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb43c066760 0x7fb43c068c20 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb45c158f10 0x7fb45c150460 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb45c158b30 0x7fb45c15b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 --2- 172.21.15.119:0/2204309057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb45c06d540 0x7fb45c1ee100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 >> 172.21.15.119:0/2204309057 conn(0x7fb45c06a350 msgr2=0x7fb45c0cde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:10.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 shutdown_connections 2024-09-13T16:27:10.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.503+0000 7fb45bfff700 1 -- 172.21.15.119:0/2204309057 wait complete. 2024-09-13T16:27:10.522 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:27:10.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:10.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.623+0000 7f866edb0700 1 Processor -- start 2024-09-13T16:27:10.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.623+0000 7f866edb0700 1 -- start start 2024-09-13T16:27:10.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866edb0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f8668160f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866edb0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86680b4880 con 0x7f866806a5c0 2024-09-13T16:27:10.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866edb0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86680b3c70 con 0x7f866806a9a0 2024-09-13T16:27:10.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866edb0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866815dec0 con 0x7f86680717b0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866ddae700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f866806a5c0 0x7f866806cdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56930/0 (socket says 172.21.15.119:56930) 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866ddae700 1 -- 172.21.15.119:0/2763725753 learned_addr learned my addr 172.21.15.119:0/2763725753 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866d5ad700 1 --2- 172.21.15.119:0/2763725753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f8668160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828193655 0 0) 0x7f86680b4880 con 0x7f866806a5c0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86540035d0 con 0x7f866806a5c0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779083653 0 0) 0x7f86680b3c70 con 0x7f866806a9a0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86680b4880 con 0x7f866806a9a0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866d5ad700 1 -- 172.21.15.119:0/2763725753 >> v1:172.21.15.145:6789/0 conn(0x7f866806a9a0 legacy=0x7f866815d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866d5ad700 1 -- 172.21.15.119:0/2763725753 >> v1:172.21.15.119:6789/0 conn(0x7f866806a5c0 legacy=0x7f866806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3466662262 0 0) 0x7f86540035d0 con 0x7f866806a5c0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866d5ad700 1 -- 172.21.15.119:0/2763725753 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86680b3f50 con 0x7f86680717b0 2024-09-13T16:27:10.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866d5ad700 1 --2- 172.21.15.119:0/2763725753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f8668160f60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8658009880 tx=0x7f8658009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ae450afbb75ec1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664001670 con 0x7f86680717b0 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8658004030 con 0x7f86680717b0 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.627+0000 7f866cdac700 1 -- 172.21.15.119:0/2763725753 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86580043c0 con 0x7f86680717b0 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 -- 172.21.15.119:0/2763725753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 msgr2=0x7f8668160f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 --2- 172.21.15.119:0/2763725753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f8668160f60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8658009880 tx=0x7f8658009b90 comp rx=0 tx=0).stop 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 -- 172.21.15.119:0/2763725753 shutdown_connections 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 --2- 172.21.15.119:0/2763725753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f8668160f60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 -- 172.21.15.119:0/2763725753 >> 172.21.15.119:0/2763725753 conn(0x7f8668070cc0 msgr2=0x7f86680710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 -- 172.21.15.119:0/2763725753 shutdown_connections 2024-09-13T16:27:10.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 -- 172.21.15.119:0/2763725753 wait complete. 2024-09-13T16:27:10.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 Processor -- start 2024-09-13T16:27:10.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.631+0000 7f866edb0700 1 -- start start 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f866806a5c0 0x7f86681594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f866806a9a0 0x7f86681561a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f866815ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8668156d90 con 0x7f866806a5c0 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86681f25b0 con 0x7f866806a9a0 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8668159b80 con 0x7f86680717b0 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f866806a5c0 0x7f86681594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f866806a5c0 0x7f86681594a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35200/0 (socket says 172.21.15.119:35200) 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 -- 172.21.15.119:0/3286160106 learned_addr learned my addr 172.21.15.119:0/3286160106 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866d5ad700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f866806a9a0 0x7f86681561a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866e5af700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f866815ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 -- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 msgr2=0x7f866815ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f866815ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 -- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f866806a9a0 msgr2=0x7f86681561a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f866806a9a0 0x7f86681561a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 -- 172.21.15.119:0/3286160106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86580094b0 con 0x7f866806a5c0 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866d5ad700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f866806a9a0 0x7f86681561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866ddae700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f866806a5c0 0x7f86681594a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f866400c750 tx=0x7f866400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f865e7fc700 1 -- 172.21.15.119:0/3286160106 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8664004ca0 con 0x7f866806a5c0 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f865e7fc700 1 -- 172.21.15.119:0/3286160106 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8664004e00 con 0x7f866806a5c0 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8668159e60 con 0x7f866806a5c0 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f865e7fc700 1 -- 172.21.15.119:0/3286160106 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86640072d0 con 0x7f866806a5c0 2024-09-13T16:27:10.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86680cd390 con 0x7f866806a5c0 2024-09-13T16:27:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.635+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86680b3c70 con 0x7f866806a5c0 2024-09-13T16:27:10.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.639+0000 7f865e7fc700 1 -- 172.21.15.119:0/3286160106 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8664007430 con 0x7f866806a5c0 2024-09-13T16:27:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.639+0000 7f865e7fc700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f864c066730 0x7f864c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.639+0000 7f865e7fc700 1 -- 172.21.15.119:0/3286160106 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8664083c10 con 0x7f866806a5c0 2024-09-13T16:27:10.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.639+0000 7f866d5ad700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f864c066730 0x7f864c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.639+0000 7f866d5ad700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f864c066730 0x7f864c068bf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8658009880 tx=0x7f8658010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.643+0000 7f865e7fc700 1 -- 172.21.15.119:0/3286160106 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8664050d80 con 0x7f866806a5c0 2024-09-13T16:27:10.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.847+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f866806d000 con 0x7f866806a5c0 2024-09-13T16:27:10.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.847+0000 7f865e7fc700 1 -- 172.21.15.119:0/3286160106 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f86640543a0 con 0x7f866806a5c0 2024-09-13T16:27:10.854 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:10.854 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f864c066730 msgr2=0x7f864c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f864c066730 0x7f864c068bf0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8658009880 tx=0x7f8658010000 comp rx=0 tx=0).stop 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f866806a5c0 msgr2=0x7f86681594a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f866806a5c0 0x7f86681594a0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f866400c750 tx=0x7f866400ca60 comp rx=0 tx=0).stop 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 shutdown_connections 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f864c066730 0x7f864c068bf0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f86680717b0 0x7f866815ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f866806a9a0 0x7f86681561a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 --2- 172.21.15.119:0/3286160106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f866806a5c0 0x7f86681594a0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 >> 172.21.15.119:0/3286160106 conn(0x7f8668070cc0 msgr2=0x7f8668157620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:10.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 shutdown_connections 2024-09-13T16:27:10.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.851+0000 7f866edb0700 1 -- 172.21.15.119:0/3286160106 wait complete. 2024-09-13T16:27:10.858 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:27:10.869 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63c0d99700 1 Processor -- start 2024-09-13T16:27:10.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63c0d99700 1 -- start start 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63c0d99700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc1598e0 0x7f63bc0b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63c0d99700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0b2ac0 con 0x7f63bc0ad710 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63c0d99700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc0b2da0 con 0x7f63bc0b2350 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63c0d99700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0ccf00 con 0x7f63bc1598e0 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63bad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f63bc0ad710 0x7f63bc0cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56942/0 (socket says 172.21.15.119:56942) 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63bad9d700 1 -- 172.21.15.119:0/139711433 learned_addr learned my addr 172.21.15.119:0/139711433 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63ba59c700 1 --2- 172.21.15.119:0/139711433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc1598e0 0x7f63bc0b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 693826622 0 0) 0x7f63bc0b2ac0 con 0x7f63bc0ad710 2024-09-13T16:27:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63a40035d0 con 0x7f63bc0ad710 2024-09-13T16:27:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972518420 0 0) 0x7f63bc0b2da0 con 0x7f63bc0b2350 2024-09-13T16:27:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc0b2ac0 con 0x7f63bc0b2350 2024-09-13T16:27:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63ba59c700 1 -- 172.21.15.119:0/139711433 >> v1:172.21.15.145:6789/0 conn(0x7f63bc0b2350 legacy=0x7f63bc06fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.975+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 926041911 0 0) 0x7f63a40035d0 con 0x7f63bc0ad710 2024-09-13T16:27:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63ba59c700 1 -- 172.21.15.119:0/139711433 >> v1:172.21.15.119:6789/0 conn(0x7f63bc0ad710 legacy=0x7f63bc0cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63ba59c700 1 -- 172.21.15.119:0/139711433 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0cd1e0 con 0x7f63bc1598e0 2024-09-13T16:27:10.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63ba59c700 1 --2- 172.21.15.119:0/139711433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc1598e0 0x7f63bc0b3a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009880 tx=0x7f63ac009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6acaa5309146d26 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac011830 con 0x7f63bc1598e0 2024-09-13T16:27:10.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63ac011990 con 0x7f63bc1598e0 2024-09-13T16:27:10.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63b959a700 1 -- 172.21.15.119:0/139711433 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63ac011d20 con 0x7f63bc1598e0 2024-09-13T16:27:10.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 -- 172.21.15.119:0/139711433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc1598e0 msgr2=0x7f63bc0b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 --2- 172.21.15.119:0/139711433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc1598e0 0x7f63bc0b3a60 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f63ac009880 tx=0x7f63ac009b90 comp rx=0 tx=0).stop 2024-09-13T16:27:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 -- 172.21.15.119:0/139711433 shutdown_connections 2024-09-13T16:27:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 --2- 172.21.15.119:0/139711433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc1598e0 0x7f63bc0b3a60 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 -- 172.21.15.119:0/139711433 >> 172.21.15.119:0/139711433 conn(0x7f63bc06a350 msgr2=0x7f63bc06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 -- 172.21.15.119:0/139711433 shutdown_connections 2024-09-13T16:27:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 -- 172.21.15.119:0/139711433 wait complete. 2024-09-13T16:27:10.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 Processor -- start 2024-09-13T16:27:10.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.979+0000 7f63c0d99700 1 -- start start 2024-09-13T16:27:10.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63bc0ad710 0x7f63bc14f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc0b2350 0x7f63bc1f2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63bc1598e0 0x7f63bc0cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc15b3e0 con 0x7f63bc1598e0 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc14f850 con 0x7f63bc0ad710 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc1f33f0 con 0x7f63bc0b2350 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63bc1598e0 0x7f63bc0cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63ba59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63bc0ad710 0x7f63bc14f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63bc1598e0 0x7f63bc0cdaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35204/0 (socket says 172.21.15.119:35204) 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 -- 172.21.15.119:0/1139173914 learned_addr learned my addr 172.21.15.119:0/1139173914 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:10.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63b9d9b700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc0b2350 0x7f63bc1f2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 -- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc0b2350 msgr2=0x7f63bc1f2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc0b2350 0x7f63bc1f2eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 -- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63bc0ad710 msgr2=0x7f63bc14f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:10.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63bc0ad710 0x7f63bc14f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 -- 172.21.15.119:0/1139173914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63ac0094b0 con 0x7f63bc1598e0 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63bad9d700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63bc1598e0 0x7f63bc0cdaf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f63b400cc10 tx=0x7f63b400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63aaffd700 1 -- 172.21.15.119:0/1139173914 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4007480 con 0x7f63bc1598e0 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63aaffd700 1 -- 172.21.15.119:0/1139173914 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b40075e0 con 0x7f63bc1598e0 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc15b100 con 0x7f63bc1598e0 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63aaffd700 1 -- 172.21.15.119:0/1139173914 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4007990 con 0x7f63bc1598e0 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.983+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc0c8140 con 0x7f63bc1598e0 2024-09-13T16:27:10.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.987+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc0c7cc0 con 0x7f63bc1598e0 2024-09-13T16:27:10.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.987+0000 7f63aaffd700 1 -- 172.21.15.119:0/1139173914 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f63b401b400 con 0x7f63bc1598e0 2024-09-13T16:27:10.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.987+0000 7f63aaffd700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f639c066730 0x7f639c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:10.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.987+0000 7f63aaffd700 1 -- 172.21.15.119:0/1139173914 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f63b4084360 con 0x7f63bc1598e0 2024-09-13T16:27:10.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.987+0000 7f63ba59c700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f639c066730 0x7f639c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:10.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.987+0000 7f63ba59c700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f639c066730 0x7f639c068bf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0090b0 tx=0x7f63ac01c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:10.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:10.991+0000 7f63aaffd700 1 -- 172.21.15.119:0/1139173914 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f63bc0c7cc0 con 0x7f63bc1598e0 2024-09-13T16:27:11.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.199+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63bc0c7cc0 con 0x7f63bc1598e0 2024-09-13T16:27:11.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.199+0000 7f63aaffd700 1 -- 172.21.15.119:0/1139173914 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f63bc0c7cc0 con 0x7f63bc1598e0 2024-09-13T16:27:11.206 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:11.206 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:26:37.416714+0000","modified":"2024-09-13T16:26:40.432024+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5586},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5586":{"gid":5586,"name":"b","rank":0,"incarnation":13,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/2305196370","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":2305196370},{"type":"v1","addr":"172.21.15.145:6837","nonce":2305196370}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f639c066730 msgr2=0x7f639c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f639c066730 0x7f639c068bf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f63ac0090b0 tx=0x7f63ac01c040 comp rx=0 tx=0).stop 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63bc1598e0 msgr2=0x7f63bc0cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63bc1598e0 0x7f63bc0cdaf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f63b400cc10 tx=0x7f63b400cc40 comp rx=0 tx=0).stop 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 shutdown_connections 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f639c066730 0x7f639c068bf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63bc1598e0 0x7f63bc0cdaf0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63bc0b2350 0x7f63bc1f2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 --2- 172.21.15.119:0/1139173914 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63bc0ad710 0x7f63bc14f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 >> 172.21.15.119:0/1139173914 conn(0x7f63bc06a350 msgr2=0x7f63bc0ac840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:11.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 shutdown_connections 2024-09-13T16:27:11.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.203+0000 7f63c0d99700 1 -- 172.21.15.119:0/1139173914 wait complete. 2024-09-13T16:27:11.210 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 16 2024-09-13T16:27:11.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:11.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.323+0000 7f4dd2128700 1 Processor -- start 2024-09-13T16:27:11.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd2128700 1 -- start start 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd2128700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4dcc0adb30 0x7f4dcc0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd2128700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc06bb90 0x7f4dcc0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd2128700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dcc06cf60 con 0x7f4dcc0cde70 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd2128700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dcc0b3390 con 0x7f4dcc0adb30 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd2128700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc06c350 con 0x7f4dcc06bb90 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4dcc0adb30 0x7f4dcc0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4dcc0adb30 0x7f4dcc0cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35220/0 (socket says 172.21.15.119:35220) 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 -- 172.21.15.119:0/1550806643 learned_addr learned my addr 172.21.15.119:0/1550806643 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd0925700 1 --2- 172.21.15.119:0/1550806643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc06bb90 0x7f4dcc0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 -- 172.21.15.119:0/1550806643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc06bb90 msgr2=0x7f4dcc0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 --2- 172.21.15.119:0/1550806643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc06bb90 0x7f4dcc0cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 -- 172.21.15.119:0/1550806643 >> v1:172.21.15.145:6789/0 conn(0x7f4dcc0cde70 legacy=0x7f4dcc15e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dc3fff700 1 -- 172.21.15.119:0/1550806643 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197690059 0 0) 0x7f4dcc06cf60 con 0x7f4dcc0cde70 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 -- 172.21.15.119:0/1550806643 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc06c630 con 0x7f4dcc0adb30 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dd1126700 1 --2- 172.21.15.119:0/1550806643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0adb30 0x7f4dcc0cc060 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009910 tx=0x7f4dbc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62cff68efb4a1ff0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dc3fff700 1 -- 172.21.15.119:0/1550806643 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc011830 con 0x7f4dcc0adb30 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dc3fff700 1 -- 172.21.15.119:0/1550806643 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dbc011990 con 0x7f4dcc0adb30 2024-09-13T16:27:11.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.327+0000 7f4dc3fff700 1 -- 172.21.15.119:0/1550806643 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc011d40 con 0x7f4dcc0adb30 2024-09-13T16:27:11.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- 172.21.15.119:0/1550806643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0adb30 msgr2=0x7f4dcc0cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 --2- 172.21.15.119:0/1550806643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0adb30 0x7f4dcc0cc060 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009910 tx=0x7f4dbc009c20 comp rx=0 tx=0).stop 2024-09-13T16:27:11.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- 172.21.15.119:0/1550806643 shutdown_connections 2024-09-13T16:27:11.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 --2- 172.21.15.119:0/1550806643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc06bb90 0x7f4dcc0cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 --2- 172.21.15.119:0/1550806643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0adb30 0x7f4dcc0cc060 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- 172.21.15.119:0/1550806643 >> 172.21.15.119:0/1550806643 conn(0x7f4dcc06a350 msgr2=0x7f4dcc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:11.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- 172.21.15.119:0/1550806643 shutdown_connections 2024-09-13T16:27:11.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- 172.21.15.119:0/1550806643 wait complete. 2024-09-13T16:27:11.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 Processor -- start 2024-09-13T16:27:11.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- start start 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4dcc06bb90 0x7f4dcc1597b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc0adb30 0x7f4dcc0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0cde70 0x7f4dcc0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc132b10 con 0x7f4dcc0cde70 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc132d90 con 0x7f4dcc06bb90 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.331+0000 7f4dd2128700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dcc133070 con 0x7f4dcc0adb30 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0cde70 0x7f4dcc0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0cde70 0x7f4dcc0ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35232/0 (socket says 172.21.15.119:35232) 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 -- 172.21.15.119:0/3462991204 learned_addr learned my addr 172.21.15.119:0/3462991204 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd0925700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc0adb30 0x7f4dcc0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1126700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4dcc06bb90 0x7f4dcc1597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 -- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc0adb30 msgr2=0x7f4dcc0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc0adb30 0x7f4dcc0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 -- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4dcc06bb90 msgr2=0x7f4dcc1597b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4dcc06bb90 0x7f4dcc1597b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 -- 172.21.15.119:0/3462991204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dbc009570 con 0x7f4dcc0cde70 2024-09-13T16:27:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd0925700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc0adb30 0x7f4dcc0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:11.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd1927700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0cde70 0x7f4dcc0ace00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400cc10 tx=0x7f4dc400cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:11.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dc1ffb700 1 -- 172.21.15.119:0/3462991204 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4003e50 con 0x7f4dcc0cde70 2024-09-13T16:27:11.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dc1ffb700 1 -- 172.21.15.119:0/3462991204 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc400a700 con 0x7f4dcc0cde70 2024-09-13T16:27:11.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dcc133350 con 0x7f4dcc0cde70 2024-09-13T16:27:11.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dc1ffb700 1 -- 172.21.15.119:0/3462991204 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc400aab0 con 0x7f4dcc0cde70 2024-09-13T16:27:11.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dcc159cf0 con 0x7f4dcc0cde70 2024-09-13T16:27:11.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.335+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dcc06d3d0 con 0x7f4dcc0cde70 2024-09-13T16:27:11.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.339+0000 7f4dc1ffb700 1 -- 172.21.15.119:0/3462991204 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4dc400ac10 con 0x7f4dcc0cde70 2024-09-13T16:27:11.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.339+0000 7f4dc1ffb700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4db0066730 0x7f4db0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.339+0000 7f4dc1ffb700 1 -- 172.21.15.119:0/3462991204 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4dc4083cd0 con 0x7f4dcc0cde70 2024-09-13T16:27:11.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.339+0000 7f4dd1126700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4db0066730 0x7f4db0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.339+0000 7f4dd1126700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4db0066730 0x7f4db0068bf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc000c00 tx=0x7f4dbc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:11.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.347+0000 7f4dc1ffb700 1 -- 172.21.15.119:0/3462991204 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4dcc06d3d0 con 0x7f4dcc0cde70 2024-09-13T16:27:11.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.523+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4dcc06d3d0 con 0x7f4dcc0cde70 2024-09-13T16:27:11.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dc1ffb700 1 -- 172.21.15.119:0/3462991204 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f4dcc06d3d0 con 0x7f4dcc0cde70 2024-09-13T16:27:11.532 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:11.532 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":42,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:07.530304+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:26:35.251936+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"39","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_data","create_time":"2024-09-13T16:26:36.423580+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"6.7","mappings":[{"from":5,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:11.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4db0066730 msgr2=0x7f4db0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4db0066730 0x7f4db0068bf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc000c00 tx=0x7f4dbc008030 comp rx=0 tx=0).stop 2024-09-13T16:27:11.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0cde70 msgr2=0x7f4dcc0ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0cde70 0x7f4dcc0ace00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400cc10 tx=0x7f4dc400cf20 comp rx=0 tx=0).stop 2024-09-13T16:27:11.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 shutdown_connections 2024-09-13T16:27:11.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4db0066730 0x7f4db0068bf0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4dcc0cde70 0x7f4dcc0ace00 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4dcc0adb30 0x7f4dcc0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 --2- 172.21.15.119:0/3462991204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4dcc06bb90 0x7f4dcc1597b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 >> 172.21.15.119:0/3462991204 conn(0x7f4dcc06a350 msgr2=0x7f4dcc0ca9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:11.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 shutdown_connections 2024-09-13T16:27:11.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.527+0000 7f4dd2128700 1 -- 172.21.15.119:0/3462991204 wait complete. 2024-09-13T16:27:11.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:27:11.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819bfff700 1 Processor -- start 2024-09-13T16:27:11.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819bfff700 1 -- start start 2024-09-13T16:27:11.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819bfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f819c1509e0 0x7f819c159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f819c0cc220 0x7f819c15a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819bfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f819c0b44e0 con 0x7f819c15a740 2024-09-13T16:27:11.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819bfff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819c15ab90 con 0x7f819c0cc220 2024-09-13T16:27:11.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819bfff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f819c0cddf0 con 0x7f819c1509e0 2024-09-13T16:27:11.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f819c0cc220 0x7f819c15a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f819c0cc220 0x7f819c15a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35238/0 (socket says 172.21.15.119:35238) 2024-09-13T16:27:11.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 -- 172.21.15.119:0/947161870 learned_addr learned my addr 172.21.15.119:0/947161870 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:11.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819affd700 1 --2- 172.21.15.119:0/947161870 >> v2:172.21.15.145:3300/0 conn(0x7f819c1509e0 0x7f819c159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 -- 172.21.15.119:0/947161870 >> v2:172.21.15.145:3300/0 conn(0x7f819c1509e0 msgr2=0x7f819c159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 --2- 172.21.15.119:0/947161870 >> v2:172.21.15.145:3300/0 conn(0x7f819c1509e0 0x7f819c159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 -- 172.21.15.119:0/947161870 >> v1:172.21.15.145:6789/0 conn(0x7f819c15a740 legacy=0x7f819c15d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 -- 172.21.15.119:0/947161870 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f819c071a50 con 0x7f819c0cc220 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.659+0000 7f819a7fc700 1 --2- 172.21.15.119:0/947161870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c0cc220 0x7f819c15a200 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f819c072540 tx=0x7f8190009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5fc0943d3bf4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f8199ffb700 1 -- 172.21.15.119:0/947161870 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190011830 con 0x7f819c0cc220 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f8199ffb700 1 -- 172.21.15.119:0/947161870 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8190011990 con 0x7f819c0cc220 2024-09-13T16:27:11.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f8199ffb700 1 -- 172.21.15.119:0/947161870 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8190011d20 con 0x7f819c0cc220 2024-09-13T16:27:11.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 -- 172.21.15.119:0/947161870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c0cc220 msgr2=0x7f819c15a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 --2- 172.21.15.119:0/947161870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c0cc220 0x7f819c15a200 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7f819c072540 tx=0x7f8190009bf0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 -- 172.21.15.119:0/947161870 shutdown_connections 2024-09-13T16:27:11.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 --2- 172.21.15.119:0/947161870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c0cc220 0x7f819c15a200 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 --2- 172.21.15.119:0/947161870 >> v2:172.21.15.145:3300/0 conn(0x7f819c1509e0 0x7f819c159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 -- 172.21.15.119:0/947161870 >> 172.21.15.119:0/947161870 conn(0x7f819c070cc0 msgr2=0x7f819c0710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:11.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 -- 172.21.15.119:0/947161870 shutdown_connections 2024-09-13T16:27:11.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.663+0000 7f819bfff700 1 -- 172.21.15.119:0/947161870 wait complete. 2024-09-13T16:27:11.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 Processor -- start 2024-09-13T16:27:11.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 -- start start 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f819c0cc220 0x7f819c140200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f819c1509e0 0x7f819c155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c15a740 0x7f819c06a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c147b00 con 0x7f819c15a740 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819c140740 con 0x7f819c0cc220 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819c146c10 con 0x7f819c1509e0 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c15a740 0x7f819c06a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c15a740 0x7f819c06a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35246/0 (socket says 172.21.15.119:35246) 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 -- 172.21.15.119:0/2998202768 learned_addr learned my addr 172.21.15.119:0/2998202768 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819affd700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f819c0cc220 0x7f819c140200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819a7fc700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f819c1509e0 0x7f819c155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 -- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f819c1509e0 msgr2=0x7f819c155c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f819c1509e0 0x7f819c155c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 -- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f819c0cc220 msgr2=0x7f819c140200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f819c0cc220 0x7f819c140200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 -- 172.21.15.119:0/2998202768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8190009570 con 0x7f819c15a740 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819b7fe700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c15a740 0x7f819c06a350 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f819400cb80 tx=0x7f819400cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f81837fe700 1 -- 172.21.15.119:0/2998202768 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8194003c20 con 0x7f819c15a740 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f81837fe700 1 -- 172.21.15.119:0/2998202768 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8194003d80 con 0x7f819c15a740 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819c146ef0 con 0x7f819c15a740 2024-09-13T16:27:11.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f81837fe700 1 -- 172.21.15.119:0/2998202768 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f819400abb0 con 0x7f819c15a740 2024-09-13T16:27:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.667+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f819c1474b0 con 0x7f819c15a740 2024-09-13T16:27:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.671+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f819c147c70 con 0x7f819c15a740 2024-09-13T16:27:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.671+0000 7f81837fe700 1 -- 172.21.15.119:0/2998202768 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f819400ad10 con 0x7f819c15a740 2024-09-13T16:27:11.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.671+0000 7f81837fe700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8188066760 0x7f8188068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:11.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.671+0000 7f81837fe700 1 -- 172.21.15.119:0/2998202768 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f8194084270 con 0x7f819c15a740 2024-09-13T16:27:11.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.671+0000 7f819affd700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8188066760 0x7f8188068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:11.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.671+0000 7f819affd700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8188066760 0x7f8188068c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f818c009b80 tx=0x7f818c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:11.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.679+0000 7f81837fe700 1 -- 172.21.15.119:0/2998202768 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8194051510 con 0x7f819c15a740 2024-09-13T16:27:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.887+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f819c159cc0 con 0x7f819c15a740 2024-09-13T16:27:11.895 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:27:11.887+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:27:11.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.899+0000 7f81837fe700 1 -- 172.21.15.119:0/2998202768 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) v1 ==== 161+0+0 (secure 0 0 0) 0x7f8194054b30 con 0x7f819c15a740 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8188066760 msgr2=0x7f8188068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8188066760 0x7f8188068c20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f818c009b80 tx=0x7f818c008040 comp rx=0 tx=0).stop 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c15a740 msgr2=0x7f819c06a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c15a740 0x7f819c06a350 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f819400cb80 tx=0x7f819400cf40 comp rx=0 tx=0).stop 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 shutdown_connections 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8188066760 0x7f8188068c20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f819c15a740 0x7f819c06a350 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f819c1509e0 0x7f819c155c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 --2- 172.21.15.119:0/2998202768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f819c0cc220 0x7f819c140200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 >> 172.21.15.119:0/2998202768 conn(0x7f819c070cc0 msgr2=0x7f819c159340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 shutdown_connections 2024-09-13T16:27:11.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:11.903+0000 7f819bfff700 1 -- 172.21.15.119:0/2998202768 wait complete. 2024-09-13T16:27:11.911 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:27:11.912 INFO:tasks.ceph.mds.b.smithi145.stderr: -5752> 2024-09-13T16:26:41.003+0000 7fcc3db96700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:27:11.923 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:27:11.946 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:27:12.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.039+0000 7fc33e228700 1 Processor -- start 2024-09-13T16:27:12.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.039+0000 7fc33e228700 1 -- start start 2024-09-13T16:27:12.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33e228700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33806d0c0 0x7fc33806d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:12.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33e228700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33806da40 con 0x7fc33815a930 2024-09-13T16:27:12.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33e228700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33815ad10 con 0x7fc3380ac500 2024-09-13T16:27:12.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33e228700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc338071cb0 con 0x7fc33806d0c0 2024-09-13T16:27:12.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33d226700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc33815a930 0x7fc338071130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56956/0 (socket says 172.21.15.119:56956) 2024-09-13T16:27:12.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33d226700 1 -- 172.21.15.119:0/1936142684 learned_addr learned my addr 172.21.15.119:0/1936142684 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:12.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33ca25700 1 --2- 172.21.15.119:0/1936142684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33806d0c0 0x7fc33806d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:12.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc32ffff700 1 -- 172.21.15.119:0/1936142684 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800885164 0 0) 0x7fc33806da40 con 0x7fc33815a930 2024-09-13T16:27:12.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc32ffff700 1 -- 172.21.15.119:0/1936142684 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3240035d0 con 0x7fc33815a930 2024-09-13T16:27:12.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc32ffff700 1 -- 172.21.15.119:0/1936142684 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900880585 0 0) 0x7fc33815ad10 con 0x7fc3380ac500 2024-09-13T16:27:12.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33ca25700 1 -- 172.21.15.119:0/1936142684 >> v1:172.21.15.145:6789/0 conn(0x7fc3380ac500 legacy=0x7fc3380ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:12.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33ca25700 1 -- 172.21.15.119:0/1936142684 >> v1:172.21.15.119:6789/0 conn(0x7fc33815a930 legacy=0x7fc338071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:12.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33ca25700 1 -- 172.21.15.119:0/1936142684 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338071f90 con 0x7fc33806d0c0 2024-09-13T16:27:12.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc33ca25700 1 --2- 172.21.15.119:0/1936142684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33806d0c0 0x7fc33806d500 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc328009880 tx=0x7fc328009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3b193ff9013effd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:12.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc32ffff700 1 -- 172.21.15.119:0/1936142684 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc328011830 con 0x7fc33806d0c0 2024-09-13T16:27:12.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc32ffff700 1 -- 172.21.15.119:0/1936142684 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc328011990 con 0x7fc33806d0c0 2024-09-13T16:27:12.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.043+0000 7fc32ffff700 1 -- 172.21.15.119:0/1936142684 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc328011d00 con 0x7fc33806d0c0 2024-09-13T16:27:12.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- 172.21.15.119:0/1936142684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33806d0c0 msgr2=0x7fc33806d500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:12.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 --2- 172.21.15.119:0/1936142684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33806d0c0 0x7fc33806d500 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc328009880 tx=0x7fc328009b90 comp rx=0 tx=0).stop 2024-09-13T16:27:12.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- 172.21.15.119:0/1936142684 shutdown_connections 2024-09-13T16:27:12.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 --2- 172.21.15.119:0/1936142684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33806d0c0 0x7fc33806d500 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- 172.21.15.119:0/1936142684 >> 172.21.15.119:0/1936142684 conn(0x7fc3380b0cd0 msgr2=0x7fc3380b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:12.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- 172.21.15.119:0/1936142684 shutdown_connections 2024-09-13T16:27:12.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- 172.21.15.119:0/1936142684 wait complete. 2024-09-13T16:27:12.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 Processor -- start 2024-09-13T16:27:12.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- start start 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33806d0c0 0x7fc3381f5140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3380ac500 0x7fc3381f2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33815a930 0x7fc338159070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33815daa0 con 0x7fc33806d0c0 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381f3640 con 0x7fc3380ac500 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33e228700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380b2850 con 0x7fc33815a930 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33d226700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33806d0c0 0x7fc3381f5140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33d226700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33806d0c0 0x7fc3381f5140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35258/0 (socket says 172.21.15.119:35258) 2024-09-13T16:27:12.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33d226700 1 -- 172.21.15.119:0/4253954433 learned_addr learned my addr 172.21.15.119:0/4253954433 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.047+0000 7fc33d226700 1 -- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33815a930 msgr2=0x7fc338159070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33d226700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33815a930 0x7fc338159070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33d226700 1 -- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3380ac500 msgr2=0x7fc3381f2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33d226700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3380ac500 0x7fc3381f2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33d226700 1 -- 172.21.15.119:0/4253954433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280094b0 con 0x7fc33806d0c0 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33d226700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33806d0c0 0x7fc3381f5140 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c4f0 tx=0x7fc33400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc32dffb700 1 -- 172.21.15.119:0/4253954433 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334005d90 con 0x7fc33806d0c0 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc32dffb700 1 -- 172.21.15.119:0/4253954433 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3340070d0 con 0x7fc33806d0c0 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3381f2a10 con 0x7fc33806d0c0 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc32dffb700 1 -- 172.21.15.119:0/4253954433 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc334007440 con 0x7fc33806d0c0 2024-09-13T16:27:12.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3381f2fd0 con 0x7fc33806d0c0 2024-09-13T16:27:12.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc338073590 con 0x7fc33806d0c0 2024-09-13T16:27:12.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc32dffb700 1 -- 172.21.15.119:0/4253954433 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc3340075a0 con 0x7fc33806d0c0 2024-09-13T16:27:12.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc32dffb700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc31c066730 0x7fc31c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:12.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.051+0000 7fc32dffb700 1 -- 172.21.15.119:0/4253954433 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fc334083740 con 0x7fc33806d0c0 2024-09-13T16:27:12.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.055+0000 7fc33ca25700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc31c066730 0x7fc31c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:12.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.055+0000 7fc33ca25700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc31c066730 0x7fc31c068bf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc328009880 tx=0x7fc328009f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:12.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.059+0000 7fc32dffb700 1 -- 172.21.15.119:0/4253954433 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc3340509b0 con 0x7fc33806d0c0 2024-09-13T16:27:12.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.263+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc3381f3160 con 0x7fc33806d0c0 2024-09-13T16:27:12.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.903+0000 7fc32dffb700 1 -- 172.21.15.119:0/4253954433 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc334053fd0 con 0x7fc33806d0c0 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc31c066730 msgr2=0x7fc31c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc31c066730 0x7fc31c068bf0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc328009880 tx=0x7fc328009f90 comp rx=0 tx=0).stop 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33806d0c0 msgr2=0x7fc3381f5140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33806d0c0 0x7fc3381f5140 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc33400c4f0 tx=0x7fc33400c8b0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 shutdown_connections 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc31c066730 0x7fc31c068bf0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc33815a930 0x7fc338159070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3380ac500 0x7fc3381f2150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 --2- 172.21.15.119:0/4253954433 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc33806d0c0 0x7fc3381f5140 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:12.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 >> 172.21.15.119:0/4253954433 conn(0x7fc3380b0cd0 msgr2=0x7fc3380aeb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:12.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 shutdown_connections 2024-09-13T16:27:12.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:12.911+0000 7fc33e228700 1 -- 172.21.15.119:0/4253954433 wait complete. 2024-09-13T16:27:12.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:27:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.059+0000 7f76b17a3700 1 Processor -- start 2024-09-13T16:27:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.059+0000 7f76b17a3700 1 -- start start 2024-09-13T16:27:13.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f76ac0d1010 0x7f76ac15d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f76ac15d730 0x7f76ac06bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76ac1606e0 con 0x7f76ac15ccf0 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ac15df70 con 0x7f76ac15d730 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76ac160980 con 0x7f76ac0d1010 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f76ac15d730 0x7f76ac06bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f76ac15d730 0x7f76ac06bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35900/0 (socket says 172.21.15.119:35900) 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 -- 172.21.15.119:0/1034477669 learned_addr learned my addr 172.21.15.119:0/1034477669 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76a27fc700 1 --2- 172.21.15.119:0/1034477669 >> v2:172.21.15.145:3300/0 conn(0x7f76ac0d1010 0x7f76ac15d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 -- 172.21.15.119:0/1034477669 >> v2:172.21.15.145:3300/0 conn(0x7f76ac0d1010 msgr2=0x7f76ac15d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 --2- 172.21.15.119:0/1034477669 >> v2:172.21.15.145:3300/0 conn(0x7f76ac0d1010 0x7f76ac15d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 -- 172.21.15.119:0/1034477669 >> v1:172.21.15.145:6789/0 conn(0x7f76ac15ccf0 legacy=0x7f76ac0d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 -- 172.21.15.119:0/1034477669 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76ac160c60 con 0x7f76ac15d730 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76ab7fe700 1 --2- 172.21.15.119:0/1034477669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac15d730 0x7f76ac06bad0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009e50 tx=0x7f76a400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcbce33f371fb09 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76aa7fc700 1 -- 172.21.15.119:0/1034477669 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a400e070 con 0x7f76ac15d730 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76aa7fc700 1 -- 172.21.15.119:0/1034477669 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76a400b8d0 con 0x7f76ac15d730 2024-09-13T16:27:13.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76aa7fc700 1 -- 172.21.15.119:0/1034477669 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76a400bc80 con 0x7f76ac15d730 2024-09-13T16:27:13.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- 172.21.15.119:0/1034477669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac15d730 msgr2=0x7f76ac06bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 --2- 172.21.15.119:0/1034477669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac15d730 0x7f76ac06bad0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f76a4009e50 tx=0x7f76a400b330 comp rx=0 tx=0).stop 2024-09-13T16:27:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- 172.21.15.119:0/1034477669 shutdown_connections 2024-09-13T16:27:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 --2- 172.21.15.119:0/1034477669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac15d730 0x7f76ac06bad0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 --2- 172.21.15.119:0/1034477669 >> v2:172.21.15.145:3300/0 conn(0x7f76ac0d1010 0x7f76ac15d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- 172.21.15.119:0/1034477669 >> 172.21.15.119:0/1034477669 conn(0x7f76ac099a50 msgr2=0x7f76ac099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- 172.21.15.119:0/1034477669 shutdown_connections 2024-09-13T16:27:13.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.063+0000 7f76b17a3700 1 -- 172.21.15.119:0/1034477669 wait complete. 2024-09-13T16:27:13.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.067+0000 7f76b17a3700 1 Processor -- start 2024-09-13T16:27:13.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.067+0000 7f76b17a3700 1 -- start start 2024-09-13T16:27:13.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac0d1010 0x7f76ac09b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:13.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76ac15ccf0 0x7f76ac06e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76ac15d730 0x7f76ac06e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac14b0b0 con 0x7f76ac0d1010 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac09ce60 con 0x7f76ac15d730 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76ac09d140 con 0x7f76ac15ccf0 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac0d1010 0x7f76ac09b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac0d1010 0x7f76ac09b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35902/0 (socket says 172.21.15.119:35902) 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 -- 172.21.15.119:0/2466040827 learned_addr learned my addr 172.21.15.119:0/2466040827 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76ab7fe700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76ac15d730 0x7f76ac06e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:13.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76a27fc700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76ac15ccf0 0x7f76ac06e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 -- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76ac15ccf0 msgr2=0x7f76ac06e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76ac15ccf0 0x7f76ac06e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 -- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76ac15d730 msgr2=0x7f76ac06e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76ac15d730 0x7f76ac06e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 -- 172.21.15.119:0/2466040827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a4009b30 con 0x7f76ac0d1010 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76a27fc700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76ac15ccf0 0x7f76ac06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76aaffd700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac0d1010 0x7f76ac09b050 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f769c00c500 tx=0x7f769c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76a3fff700 1 -- 172.21.15.119:0/2466040827 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c0052d0 con 0x7f76ac0d1010 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76a3fff700 1 -- 172.21.15.119:0/2466040827 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769c005430 con 0x7f76ac0d1010 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76ac09d420 con 0x7f76ac0d1010 2024-09-13T16:27:13.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76a3fff700 1 -- 172.21.15.119:0/2466040827 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769c0055a0 con 0x7f76ac0d1010 2024-09-13T16:27:13.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76ac09b590 con 0x7f76ac0d1010 2024-09-13T16:27:13.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.071+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76ac067990 con 0x7f76ac0d1010 2024-09-13T16:27:13.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.075+0000 7f76a3fff700 1 -- 172.21.15.119:0/2466040827 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f769c0036a0 con 0x7f76ac0d1010 2024-09-13T16:27:13.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.075+0000 7f76a3fff700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f768c066730 0x7f768c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:13.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.075+0000 7f76a3fff700 1 -- 172.21.15.119:0/2466040827 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f769c083d50 con 0x7f76ac0d1010 2024-09-13T16:27:13.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.075+0000 7f76a27fc700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f768c066730 0x7f768c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:13.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.075+0000 7f76a27fc700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f768c066730 0x7f768c068bf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0b4200 tx=0x7f7694008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:13.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.079+0000 7f76a3fff700 1 -- 172.21.15.119:0/2466040827 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f769c051010 con 0x7f76ac0d1010 2024-09-13T16:27:13.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.251+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f76ac15d1f0 con 0x7f76ac0d1010 2024-09-13T16:27:13.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.919+0000 7f76a3fff700 1 -- 172.21.15.119:0/2466040827 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) v1 ==== 181+0+0 (secure 0 0 0) 0x7f769c0200d0 con 0x7f76ac0d1010 2024-09-13T16:27:13.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f768c066730 msgr2=0x7f768c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:13.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f768c066730 0x7f768c068bf0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f76ac0b4200 tx=0x7f7694008040 comp rx=0 tx=0).stop 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac0d1010 msgr2=0x7f76ac09b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac0d1010 0x7f76ac09b050 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f769c00c500 tx=0x7f769c00c810 comp rx=0 tx=0).stop 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 shutdown_connections 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f768c066730 0x7f768c068bf0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76ac15d730 0x7f76ac06e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76ac15ccf0 0x7f76ac06e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 --2- 172.21.15.119:0/2466040827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76ac0d1010 0x7f76ac09b050 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:13.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 >> 172.21.15.119:0/2466040827 conn(0x7f76ac099a50 msgr2=0x7f76ac099e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:13.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 shutdown_connections 2024-09-13T16:27:13.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:13.923+0000 7f76b17a3700 1 -- 172.21.15.119:0/2466040827 wait complete. 2024-09-13T16:27:13.932 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:27:13.944 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:27:14.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.063+0000 7f20a0b33700 1 Processor -- start 2024-09-13T16:27:14.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.063+0000 7f20a0b33700 1 -- start start 2024-09-13T16:27:14.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f209c0d1840 0x7f209c167b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:14.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c0d3260 con 0x7f209c0d0720 2024-09-13T16:27:14.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f209c0cd640 con 0x7f209c15cc40 2024-09-13T16:27:14.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f209c0d0c70 con 0x7f209c0d1840 2024-09-13T16:27:14.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209affd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f209c0d0720 0x7f209c0d1130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46070/0 (socket says 172.21.15.119:46070) 2024-09-13T16:27:14.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209affd700 1 -- 172.21.15.119:0/2198862983 learned_addr learned my addr 172.21.15.119:0/2198862983 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:14.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209bfff700 1 --2- 172.21.15.119:0/2198862983 >> v2:172.21.15.145:3300/0 conn(0x7f209c0d1840 0x7f209c167b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:14.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209a7fc700 1 -- 172.21.15.119:0/2198862983 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2334154064 0 0) 0x7f209c0d3260 con 0x7f209c0d0720 2024-09-13T16:27:14.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209a7fc700 1 -- 172.21.15.119:0/2198862983 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20840035d0 con 0x7f209c0d0720 2024-09-13T16:27:14.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209bfff700 1 -- 172.21.15.119:0/2198862983 >> v1:172.21.15.145:6789/0 conn(0x7f209c15cc40 legacy=0x7f209c0cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:14.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209bfff700 1 -- 172.21.15.119:0/2198862983 >> v1:172.21.15.119:6789/0 conn(0x7f209c0d0720 legacy=0x7f209c0d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:14.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209bfff700 1 -- 172.21.15.119:0/2198862983 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f209c0d3490 con 0x7f209c0d1840 2024-09-13T16:27:14.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209bfff700 1 --2- 172.21.15.119:0/2198862983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d1840 0x7f209c167b50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2094009d90 tx=0x7f209400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adb1a0089a3f5b50 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:14.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209a7fc700 1 -- 172.21.15.119:0/2198862983 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209400e070 con 0x7f209c0d1840 2024-09-13T16:27:14.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209a7fc700 1 -- 172.21.15.119:0/2198862983 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f209400bb20 con 0x7f209c0d1840 2024-09-13T16:27:14.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f209a7fc700 1 -- 172.21.15.119:0/2198862983 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f209400be90 con 0x7f209c0d1840 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- 172.21.15.119:0/2198862983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d1840 msgr2=0x7f209c167b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 --2- 172.21.15.119:0/2198862983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d1840 0x7f209c167b50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2094009d90 tx=0x7f209400b330 comp rx=0 tx=0).stop 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- 172.21.15.119:0/2198862983 shutdown_connections 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 --2- 172.21.15.119:0/2198862983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d1840 0x7f209c167b50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- 172.21.15.119:0/2198862983 >> 172.21.15.119:0/2198862983 conn(0x7f209c09be40 msgr2=0x7f209c0d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- 172.21.15.119:0/2198862983 shutdown_connections 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.067+0000 7f20a0b33700 1 -- 172.21.15.119:0/2198862983 wait complete. 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.071+0000 7f20a0b33700 1 Processor -- start 2024-09-13T16:27:14.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.071+0000 7f20a0b33700 1 -- start start 2024-09-13T16:27:14.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d0720 0x7f209c0b4010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:14.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f209c0d1840 0x7f209c0c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209c15cc40 0x7f209c0ca160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c0b52e0 con 0x7f209c0d1840 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f209c0c6160 con 0x7f209c0d0720 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f209c0b4550 con 0x7f209c15cc40 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f209c0d1840 0x7f209c0c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f209c0d1840 0x7f209c0c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35912/0 (socket says 172.21.15.119:35912) 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 -- 172.21.15.119:0/1512232624 learned_addr learned my addr 172.21.15.119:0/1512232624 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209b7fe700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d0720 0x7f209c0b4010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:14.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209bfff700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209c15cc40 0x7f209c0ca160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:14.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 -- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209c15cc40 msgr2=0x7f209c0ca160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:14.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209c15cc40 0x7f209c0ca160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 -- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d0720 msgr2=0x7f209c0b4010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d0720 0x7f209c0b4010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 -- 172.21.15.119:0/1512232624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2094009a70 con 0x7f209c0d1840 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f209affd700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f209c0d1840 0x7f209c0c9c20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f209000c4d0 tx=0x7f209000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f2083fff700 1 -- 172.21.15.119:0/1512232624 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090005d90 con 0x7f209c0d1840 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f2083fff700 1 -- 172.21.15.119:0/1512232624 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20900070d0 con 0x7f209c0d1840 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209c0b4760 con 0x7f209c0d1840 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f2083fff700 1 -- 172.21.15.119:0/1512232624 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2090007480 con 0x7f209c0d1840 2024-09-13T16:27:14.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.075+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209c0b4d20 con 0x7f209c0d1840 2024-09-13T16:27:14.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.079+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f209c0d1d00 con 0x7f209c0d1840 2024-09-13T16:27:14.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.079+0000 7f2083fff700 1 -- 172.21.15.119:0/1512232624 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f20900075e0 con 0x7f209c0d1840 2024-09-13T16:27:14.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.079+0000 7f2083fff700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2088066730 0x7f2088068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:14.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.079+0000 7f2083fff700 1 -- 172.21.15.119:0/1512232624 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5706+0+0 (secure 0 0 0) 0x7f2090083620 con 0x7f209c0d1840 2024-09-13T16:27:14.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.079+0000 7f209b7fe700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2088066730 0x7f2088068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:14.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.079+0000 7f209b7fe700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2088066730 0x7f2088068bf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f208c000f60 tx=0x7f208c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:14.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.087+0000 7f2083fff700 1 -- 172.21.15.119:0/1512232624 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f20900508f0 con 0x7f209c0d1840 2024-09-13T16:27:14.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.259+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f209c0d0320 con 0x7f209c0d1840 2024-09-13T16:27:14.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.927+0000 7f2083fff700 1 -- 172.21.15.119:0/1512232624 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2090053f10 con 0x7f209c0d1840 2024-09-13T16:27:14.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2088066730 msgr2=0x7f2088068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:14.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2088066730 0x7f2088068bf0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f208c000f60 tx=0x7f208c009040 comp rx=0 tx=0).stop 2024-09-13T16:27:14.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f209c0d1840 msgr2=0x7f209c0c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:14.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f209c0d1840 0x7f209c0c9c20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f209000c4d0 tx=0x7f209000c7e0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 shutdown_connections 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2088066730 0x7f2088068bf0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f209c15cc40 0x7f209c0ca160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f209c0d1840 0x7f209c0c9c20 secure :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f209000c4d0 tx=0x7f209000c7e0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 --2- 172.21.15.119:0/1512232624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f209c0d0720 0x7f209c0b4010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 >> 172.21.15.119:0/1512232624 conn(0x7f209c09be40 msgr2=0x7f209c0b5c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 shutdown_connections 2024-09-13T16:27:14.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:14.935+0000 7f20a0b33700 1 -- 172.21.15.119:0/1512232624 wait complete. 2024-09-13T16:27:14.942 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:27:14.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T16:27:15.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 Processor -- start 2024-09-13T16:27:15.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 -- start start 2024-09-13T16:27:15.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f629415cc00 0x7f62940d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f62940d0c20 0x7f62940d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62940ce8c0 0x7f62940d1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62940cdb00 con 0x7f62940ce8c0 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62940cdde0 con 0x7f62940d0c20 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299993700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62940ce0c0 con 0x7f629415cc00 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62940ce8c0 0x7f62940d1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62940ce8c0 0x7f62940d1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35928/0 (socket says 172.21.15.119:35928) 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 -- 172.21.15.119:0/1578984023 learned_addr learned my addr 172.21.15.119:0/1578984023 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6298991700 1 --2- 172.21.15.119:0/1578984023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f629415cc00 0x7f62940d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6293fff700 1 --2- 172.21.15.119:0/1578984023 >> v2:172.21.15.145:3300/0 conn(0x7f62940d0c20 0x7f62940d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 -- 172.21.15.119:0/1578984023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f629415cc00 msgr2=0x7f62940d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 --2- 172.21.15.119:0/1578984023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f629415cc00 0x7f62940d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 -- 172.21.15.119:0/1578984023 >> v2:172.21.15.145:3300/0 conn(0x7f62940d0c20 msgr2=0x7f62940d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 --2- 172.21.15.119:0/1578984023 >> v2:172.21.15.145:3300/0 conn(0x7f62940d0c20 0x7f62940d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 -- 172.21.15.119:0/1578984023 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62940ce3a0 con 0x7f62940ce8c0 2024-09-13T16:27:15.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6293fff700 1 --2- 172.21.15.119:0/1578984023 >> v2:172.21.15.145:3300/0 conn(0x7f62940d0c20 0x7f62940d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f6299192700 1 --2- 172.21.15.119:0/1578984023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940ce8c0 0x7f62940d1610 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f629408b6d0 tx=0x7f628c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ded5e9ac606c466d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f62937fe700 1 -- 172.21.15.119:0/1578984023 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c011830 con 0x7f62940ce8c0 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f62937fe700 1 -- 172.21.15.119:0/1578984023 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c011990 con 0x7f62940ce8c0 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.063+0000 7f62937fe700 1 -- 172.21.15.119:0/1578984023 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c011d20 con 0x7f62940ce8c0 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 -- 172.21.15.119:0/1578984023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940ce8c0 msgr2=0x7f62940d1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 --2- 172.21.15.119:0/1578984023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940ce8c0 0x7f62940d1610 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f629408b6d0 tx=0x7f628c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 -- 172.21.15.119:0/1578984023 shutdown_connections 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 --2- 172.21.15.119:0/1578984023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940ce8c0 0x7f62940d1610 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 --2- 172.21.15.119:0/1578984023 >> v2:172.21.15.145:3300/0 conn(0x7f62940d0c20 0x7f62940d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 --2- 172.21.15.119:0/1578984023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f629415cc00 0x7f62940d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 -- 172.21.15.119:0/1578984023 >> 172.21.15.119:0/1578984023 conn(0x7f629408c080 msgr2=0x7f62940d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 -- 172.21.15.119:0/1578984023 shutdown_connections 2024-09-13T16:27:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.067+0000 7f6299993700 1 -- 172.21.15.119:0/1578984023 wait complete. 2024-09-13T16:27:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 Processor -- start 2024-09-13T16:27:15.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 -- start start 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940ce8c0 0x7f62940cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d0c20 0x7f6294155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f629415cc00 0x7f62940c8f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294155b60 con 0x7f62940d0c20 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62940c6640 con 0x7f629415cc00 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6294156ea0 con 0x7f62940ce8c0 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d0c20 0x7f6294155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d0c20 0x7f6294155620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35934/0 (socket says 172.21.15.119:35934) 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 -- 172.21.15.119:0/2410104593 learned_addr learned my addr 172.21.15.119:0/2410104593 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:15.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6298991700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940ce8c0 0x7f62940cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299192700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f629415cc00 0x7f62940c8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 -- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940ce8c0 msgr2=0x7f62940cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940ce8c0 0x7f62940cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 -- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f629415cc00 msgr2=0x7f62940c8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f629415cc00 0x7f62940c8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 -- 172.21.15.119:0/2410104593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c009570 con 0x7f62940d0c20 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6293fff700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d0c20 0x7f6294155620 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f627c00d830 tx=0x7f627c00db40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f62917fa700 1 -- 172.21.15.119:0/2410104593 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c0125f0 con 0x7f62940d0c20 2024-09-13T16:27:15.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f62917fa700 1 -- 172.21.15.119:0/2410104593 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c012750 con 0x7f62940d0c20 2024-09-13T16:27:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.071+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62940cc680 con 0x7f62940d0c20 2024-09-13T16:27:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f62917fa700 1 -- 172.21.15.119:0/2410104593 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c012ae0 con 0x7f62940d0c20 2024-09-13T16:27:15.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62941573f0 con 0x7f62940d0c20 2024-09-13T16:27:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6294157580 con 0x7f62940d0c20 2024-09-13T16:27:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f62917fa700 1 -- 172.21.15.119:0/2410104593 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f627c012c40 con 0x7f62940d0c20 2024-09-13T16:27:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f62917fa700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6274066730 0x7f6274068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f62917fa700 1 -- 172.21.15.119:0/2410104593 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f627c084200 con 0x7f62940d0c20 2024-09-13T16:27:15.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f6298991700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6274066730 0x7f6274068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.075+0000 7f6298991700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6274066730 0x7f6274068bf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6284011340 tx=0x7f628400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:15.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.083+0000 7f62917fa700 1 -- 172.21.15.119:0/2410104593 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f627c051720 con 0x7f62940d0c20 2024-09-13T16:27:15.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.255+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f62940d06e0 con 0x7f62940d0c20 2024-09-13T16:27:15.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.263+0000 7f62917fa700 1 -- 172.21.15.119:0/2410104593 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f627c054d40 con 0x7f62940d0c20 2024-09-13T16:27:15.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.263+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6274066730 msgr2=0x7f6274068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.263+0000 7f6299993700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6274066730 0x7f6274068bf0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6284011340 tx=0x7f628400b040 comp rx=0 tx=0).stop 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d0c20 msgr2=0x7f6294155620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d0c20 0x7f6294155620 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f627c00d830 tx=0x7f627c00db40 comp rx=0 tx=0).stop 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 shutdown_connections 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6274066730 0x7f6274068bf0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f629415cc00 0x7f62940c8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d0c20 0x7f6294155620 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 --2- 172.21.15.119:0/2410104593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940ce8c0 0x7f62940cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 >> 172.21.15.119:0/2410104593 conn(0x7f629408c080 msgr2=0x7f62940b71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 shutdown_connections 2024-09-13T16:27:15.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.267+0000 7f6299993700 1 -- 172.21.15.119:0/2410104593 wait complete. 2024-09-13T16:27:15.286 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag' 2024-09-13T16:27:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.395+0000 7f9c1359e700 1 Processor -- start 2024-09-13T16:27:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.395+0000 7f9c1359e700 1 -- start start 2024-09-13T16:27:15.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.395+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c0e18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.395+0000 7f9c1359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c0c01c740 0x7f9c0c01cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c0c0e1e00 con 0x7f9c0c005b10 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c0c006ce0 con 0x7f9c0c01c740 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0060d0 con 0x7f9c0c00a5b0 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c0c01c740 0x7f9c0c01cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c0c01c740 0x7f9c0c01cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35950/0 (socket says 172.21.15.119:35950) 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 -- 172.21.15.119:0/4092352914 learned_addr learned my addr 172.21.15.119:0/4092352914 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 -- 172.21.15.119:0/4092352914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 msgr2=0x7f9c0c0e18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1259c700 1 --2- 172.21.15.119:0/4092352914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c0e18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 --2- 172.21.15.119:0/4092352914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c0e18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 -- 172.21.15.119:0/4092352914 >> v1:172.21.15.145:6789/0 conn(0x7f9c0c005b10 legacy=0x7f9c0c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 -- 172.21.15.119:0/4092352914 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c0063b0 con 0x7f9c0c01c740 2024-09-13T16:27:15.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1259c700 1 --2- 172.21.15.119:0/4092352914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c0e18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:15.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c12d9d700 1 --2- 172.21.15.119:0/4092352914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c01cbf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9c140698b0 tx=0x7f9c1404f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b75f495cc43e05e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:15.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1159a700 1 -- 172.21.15.119:0/4092352914 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14067690 con 0x7f9c0c01c740 2024-09-13T16:27:15.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1159a700 1 -- 172.21.15.119:0/4092352914 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c1404f010 con 0x7f9c0c01c740 2024-09-13T16:27:15.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1159a700 1 -- 172.21.15.119:0/4092352914 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c14067b90 con 0x7f9c0c01c740 2024-09-13T16:27:15.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- 172.21.15.119:0/4092352914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 msgr2=0x7f9c0c01cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 --2- 172.21.15.119:0/4092352914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c01cbf0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f9c140698b0 tx=0x7f9c1404f710 comp rx=0 tx=0).stop 2024-09-13T16:27:15.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- 172.21.15.119:0/4092352914 shutdown_connections 2024-09-13T16:27:15.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 --2- 172.21.15.119:0/4092352914 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c01cbf0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 --2- 172.21.15.119:0/4092352914 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c0e18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- 172.21.15.119:0/4092352914 >> 172.21.15.119:0/4092352914 conn(0x7f9c0c036140 msgr2=0x7f9c0c036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:15.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- 172.21.15.119:0/4092352914 shutdown_connections 2024-09-13T16:27:15.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 -- 172.21.15.119:0/4092352914 wait complete. 2024-09-13T16:27:15.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.399+0000 7f9c1359e700 1 Processor -- start 2024-09-13T16:27:15.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 -- start start 2024-09-13T16:27:15.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c005b10 0x7f9c0c0dcb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c037710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0de530 con 0x7f9c0c01c740 2024-09-13T16:27:15.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0dd490 con 0x7f9c0c005b10 2024-09-13T16:27:15.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c0c0dd770 con 0x7f9c0c00a5b0 2024-09-13T16:27:15.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c12d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c005b10 0x7f9c0c0dcb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c005b10 0x7f9c0c0dcb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:41334/0 (socket says 172.21.15.119:41334) 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1259c700 1 -- 172.21.15.119:0/526740445 learned_addr learned my addr 172.21.15.119:0/526740445 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c11d9b700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c037710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c12d9d700 1 -- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 msgr2=0x7f9c0c037710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c12d9d700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c037710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c12d9d700 1 -- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c005b10 msgr2=0x7f9c0c0dcb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c12d9d700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c005b10 0x7f9c0c0dcb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c12d9d700 1 -- 172.21.15.119:0/526740445 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c0dda50 con 0x7f9c0c01c740 2024-09-13T16:27:15.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c11d9b700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c037710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c12d9d700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c0ca880 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9c1404f480 tx=0x7f9c1406a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c0affd700 1 -- 172.21.15.119:0/526740445 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c1409a040 con 0x7f9c0c01c740 2024-09-13T16:27:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c0affd700 1 -- 172.21.15.119:0/526740445 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c14072350 con 0x7f9c0c01c740 2024-09-13T16:27:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c0affd700 1 -- 172.21.15.119:0/526740445 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c140726c0 con 0x7f9c0c01c740 2024-09-13T16:27:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c0c0ddd30 con 0x7f9c0c01c740 2024-09-13T16:27:15.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c0c0dd0d0 con 0x7f9c0c01c740 2024-09-13T16:27:15.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.403+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c0c0e18c0 con 0x7f9c0c01c740 2024-09-13T16:27:15.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.407+0000 7f9c0affd700 1 -- 172.21.15.119:0/526740445 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9c14072820 con 0x7f9c0c01c740 2024-09-13T16:27:15.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.407+0000 7f9c0affd700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:15.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.407+0000 7f9c0affd700 1 -- 172.21.15.119:0/526740445 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f9c140ff010 con 0x7f9c0c01c740 2024-09-13T16:27:15.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.407+0000 7f9c1259c700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:15.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.407+0000 7f9c1259c700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0e3cb0 tx=0x7f9c04008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:15.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.415+0000 7f9c0affd700 1 -- 172.21.15.119:0/526740445 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9c140cc620 con 0x7f9c0c01c740 2024-09-13T16:27:15.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.655+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]} v 0) v1 -- 0x7f9c0c037050 con 0x7f9c0c01c740 2024-09-13T16:27:15.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.931+0000 7f9c0affd700 1 -- 172.21.15.119:0/526740445 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_apply_tag"]}]=0 v122) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9c14067790 con 0x7f9c0c01c740 2024-09-13T16:27:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 msgr2=0x7f9bf4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0e3cb0 tx=0x7f9c04008040 comp rx=0 tx=0).stop 2024-09-13T16:27:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 msgr2=0x7f9c0c0ca880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:15.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c0ca880 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f9c1404f480 tx=0x7f9c1406a080 comp rx=0 tx=0).stop 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 shutdown_connections 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c0c01c740 0x7f9c0c0ca880 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c0c00a5b0 0x7f9c0c037710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 --2- 172.21.15.119:0/526740445 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c0c005b10 0x7f9c0c0dcb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 >> 172.21.15.119:0/526740445 conn(0x7f9c0c036140 msgr2=0x7f9c0c034ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 shutdown_connections 2024-09-13T16:27:15.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:15.935+0000 7f9c1359e700 1 -- 172.21.15.119:0/526740445 wait complete. 2024-09-13T16:27:15.959 INFO:tasks.cephfs_test_runner:test_apply_tag (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-09-13T16:27:15.959 INFO:tasks.cephfs_test_runner:Starting test: test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) That the MDS can repair an inodes backtrace in the data pool 2024-09-13T16:27:15.959 INFO:tasks.cephfs_test_runner:test_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-09-13T16:27:15.960 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:16.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.071+0000 7fda054fc700 1 Processor -- start 2024-09-13T16:27:16.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.071+0000 7fda054fc700 1 -- start start 2024-09-13T16:27:16.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fda0006b730 0x7fda000cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda000d2c90 0x7fda000d3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda000ceb60 con 0x7fda0015cd60 2024-09-13T16:27:16.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0015d140 con 0x7fda0006b730 2024-09-13T16:27:16.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda000ce230 con 0x7fda000d2c90 2024-09-13T16:27:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fda0006b730 0x7fda000cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fda0006b730 0x7fda000cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35974/0 (socket says 172.21.15.119:35974) 2024-09-13T16:27:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/1763784325 learned_addr learned my addr 172.21.15.119:0/1763784325 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda04cfb700 1 --2- 172.21.15.119:0/1763784325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda000d2c90 0x7fda000d3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/1763784325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda000d2c90 msgr2=0x7fda000d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 --2- 172.21.15.119:0/1763784325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda000d2c90 0x7fda000d3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/1763784325 >> v1:172.21.15.145:6789/0 conn(0x7fda0015cd60 legacy=0x7fda000cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/1763784325 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda000ce510 con 0x7fda0006b730 2024-09-13T16:27:16.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9ff7fe700 1 --2- 172.21.15.119:0/1763784325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0006b730 0x7fda000cfad0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009910 tx=0x7fd9e8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a12335c098b1bd9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9feffd700 1 -- 172.21.15.119:0/1763784325 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e800ba60 con 0x7fda0006b730 2024-09-13T16:27:16.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9feffd700 1 -- 172.21.15.119:0/1763784325 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e800bbc0 con 0x7fda0006b730 2024-09-13T16:27:16.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fd9feffd700 1 -- 172.21.15.119:0/1763784325 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8009e60 con 0x7fda0006b730 2024-09-13T16:27:16.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- 172.21.15.119:0/1763784325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0006b730 msgr2=0x7fda000cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 --2- 172.21.15.119:0/1763784325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0006b730 0x7fda000cfad0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8009910 tx=0x7fd9e8009c20 comp rx=0 tx=0).stop 2024-09-13T16:27:16.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- 172.21.15.119:0/1763784325 shutdown_connections 2024-09-13T16:27:16.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 --2- 172.21.15.119:0/1763784325 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda000d2c90 0x7fda000d3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 --2- 172.21.15.119:0/1763784325 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda0006b730 0x7fda000cfad0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- 172.21.15.119:0/1763784325 >> 172.21.15.119:0/1763784325 conn(0x7fda0008c240 msgr2=0x7fda0009c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:16.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- 172.21.15.119:0/1763784325 shutdown_connections 2024-09-13T16:27:16.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.075+0000 7fda054fc700 1 -- 172.21.15.119:0/1763784325 wait complete. 2024-09-13T16:27:16.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 Processor -- start 2024-09-13T16:27:16.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 -- start start 2024-09-13T16:27:16.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda0006b730 0x7fda000b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda000d2c90 0x7fda00150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda0015cd60 0x7fda000c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda000cd2e0 con 0x7fda000d2c90 2024-09-13T16:27:16.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0015e2f0 con 0x7fda0015cd60 2024-09-13T16:27:16.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda000b7fb0 con 0x7fda0006b730 2024-09-13T16:27:16.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda000d2c90 0x7fda00150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda000d2c90 0x7fda00150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35978/0 (socket says 172.21.15.119:35978) 2024-09-13T16:27:16.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/774593117 learned_addr learned my addr 172.21.15.119:0/774593117 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:16.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda04cfb700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda0015cd60 0x7fda000c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9fffff700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda0006b730 0x7fda000b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda0006b730 msgr2=0x7fda000b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda0006b730 0x7fda000b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda0015cd60 msgr2=0x7fda000c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda0015cd60 0x7fda000c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 -- 172.21.15.119:0/774593117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e8009570 con 0x7fda000d2c90 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9ff7fe700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda000d2c90 0x7fda00150200 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fda000b58c0 tx=0x7fd9e8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9fcff9700 1 -- 172.21.15.119:0/774593117 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8004510 con 0x7fda000d2c90 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9fcff9700 1 -- 172.21.15.119:0/774593117 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e801f070 con 0x7fda000d2c90 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda00150740 con 0x7fda000d2c90 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fd9fcff9700 1 -- 172.21.15.119:0/774593117 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e8003900 con 0x7fda000d2c90 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.079+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda000b8210 con 0x7fda000d2c90 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.083+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda000d3120 con 0x7fda000d2c90 2024-09-13T16:27:16.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.083+0000 7fd9fcff9700 1 -- 172.21.15.119:0/774593117 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd9e8003a60 con 0x7fda000d2c90 2024-09-13T16:27:16.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.083+0000 7fd9fcff9700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd9e0066730 0x7fd9e0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.083+0000 7fd9fcff9700 1 -- 172.21.15.119:0/774593117 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd9e8021070 con 0x7fda000d2c90 2024-09-13T16:27:16.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.083+0000 7fd9fffff700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd9e0066730 0x7fd9e0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.087+0000 7fd9fffff700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd9e0066730 0x7fd9e0068bf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fda0015c670 tx=0x7fd9f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.091+0000 7fd9fcff9700 1 -- 172.21.15.119:0/774593117 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd9e8052c40 con 0x7fda000d2c90 2024-09-13T16:27:16.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.263+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda0006b510 con 0x7fda000d2c90 2024-09-13T16:27:16.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.267+0000 7fd9fcff9700 1 -- 172.21.15.119:0/774593117 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v45) v1 ==== 74+0+13253 (secure 0 0 0) 0x7fd9e8056260 con 0x7fda000d2c90 2024-09-13T16:27:16.273 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:16.274 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":45,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:14.916234+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/2305196370":"2024-09-14T16:27:11.894183+0000","172.21.15.145:6835/2305196370":"2024-09-14T16:27:11.894183+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:16.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd9e0066730 msgr2=0x7fd9e0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd9e0066730 0x7fd9e0068bf0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fda0015c670 tx=0x7fd9f8009040 comp rx=0 tx=0).stop 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda000d2c90 msgr2=0x7fda00150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda000d2c90 0x7fda00150200 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7fda000b58c0 tx=0x7fd9e8004770 comp rx=0 tx=0).stop 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 shutdown_connections 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd9e0066730 0x7fd9e0068bf0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda0015cd60 0x7fda000c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda000d2c90 0x7fda00150200 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 --2- 172.21.15.119:0/774593117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda0006b730 0x7fda000b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 >> 172.21.15.119:0/774593117 conn(0x7fda0008c240 msgr2=0x7fda001e7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 shutdown_connections 2024-09-13T16:27:16.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.271+0000 7fda054fc700 1 -- 172.21.15.119:0/774593117 wait complete. 2024-09-13T16:27:16.290 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:27:16.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:27:16.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.399+0000 7fb08df40700 1 Processor -- start 2024-09-13T16:27:16.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.399+0000 7fb08df40700 1 -- start start 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb08806da10 0x7fb08815a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08815aaf0 con 0x7fb088157310 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb08806cf30 con 0x7fb0880714d0 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb08806cd50 con 0x7fb08806da10 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb087fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb08806da10 0x7fb08815a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb0877fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb088157310 0x7fb088070dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46080/0 (socket says 172.21.15.119:46080) 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb0877fe700 1 -- 172.21.15.119:0/2786317850 learned_addr learned my addr 172.21.15.119:0/2786317850 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb087fff700 1 -- 172.21.15.119:0/2786317850 >> v1:172.21.15.145:6789/0 conn(0x7fb0880714d0 legacy=0x7fb088159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb087fff700 1 -- 172.21.15.119:0/2786317850 >> v1:172.21.15.119:6789/0 conn(0x7fb088157310 legacy=0x7fb088070dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb087fff700 1 -- 172.21.15.119:0/2786317850 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08806d210 con 0x7fb08806da10 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb0867fc700 1 -- 172.21.15.119:0/2786317850 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244747948 0 0) 0x7fb08815aaf0 con 0x7fb088157310 2024-09-13T16:27:16.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb087fff700 1 --2- 172.21.15.119:0/2786317850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08806da10 0x7fb08815a440 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb080009d90 tx=0x7fb08000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82a0f2539e89182c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb0867fc700 1 -- 172.21.15.119:0/2786317850 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08000e070 con 0x7fb08806da10 2024-09-13T16:27:16.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb0867fc700 1 -- 172.21.15.119:0/2786317850 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08000bbd0 con 0x7fb08806da10 2024-09-13T16:27:16.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb0867fc700 1 -- 172.21.15.119:0/2786317850 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb080012430 con 0x7fb08806da10 2024-09-13T16:27:16.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- 172.21.15.119:0/2786317850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08806da10 msgr2=0x7fb08815a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 --2- 172.21.15.119:0/2786317850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08806da10 0x7fb08815a440 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fb080009d90 tx=0x7fb08000b330 comp rx=0 tx=0).stop 2024-09-13T16:27:16.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- 172.21.15.119:0/2786317850 shutdown_connections 2024-09-13T16:27:16.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 --2- 172.21.15.119:0/2786317850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb08806da10 0x7fb08815a440 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- 172.21.15.119:0/2786317850 >> 172.21.15.119:0/2786317850 conn(0x7fb08806b460 msgr2=0x7fb08806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:16.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- 172.21.15.119:0/2786317850 shutdown_connections 2024-09-13T16:27:16.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.403+0000 7fb08df40700 1 -- 172.21.15.119:0/2786317850 wait complete. 2024-09-13T16:27:16.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 Processor -- start 2024-09-13T16:27:16.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 -- start start 2024-09-13T16:27:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb08806da10 0x7fb0880ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0880714d0 0x7fb0880c8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb088157310 0x7fb0880c6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880cae90 con 0x7fb088157310 2024-09-13T16:27:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880cb170 con 0x7fb08806da10 2024-09-13T16:27:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0880cb450 con 0x7fb0880714d0 2024-09-13T16:27:16.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb088157310 0x7fb0880c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb088157310 0x7fb0880c6610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36000/0 (socket says 172.21.15.119:36000) 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 -- 172.21.15.119:0/1929327658 learned_addr learned my addr 172.21.15.119:0/1929327658 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb0877fe700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb08806da10 0x7fb0880ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb086ffd700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0880714d0 0x7fb0880c8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 -- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0880714d0 msgr2=0x7fb0880c8cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0880714d0 0x7fb0880c8cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 -- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb08806da10 msgr2=0x7fb0880ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb08806da10 0x7fb0880ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 -- 172.21.15.119:0/1929327658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb080009a70 con 0x7fb088157310 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb087fff700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb088157310 0x7fb0880c6610 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb080014040 tx=0x7fb080004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb07ffff700 1 -- 172.21.15.119:0/1929327658 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08001b8a0 con 0x7fb088157310 2024-09-13T16:27:16.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb07ffff700 1 -- 172.21.15.119:0/1929327658 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb080020070 con 0x7fb088157310 2024-09-13T16:27:16.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0880cb730 con 0x7fb088157310 2024-09-13T16:27:16.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb07ffff700 1 -- 172.21.15.119:0/1929327658 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb08001bc40 con 0x7fb088157310 2024-09-13T16:27:16.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.407+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0880c91f0 con 0x7fb088157310 2024-09-13T16:27:16.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.411+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb088070dc0 con 0x7fb088157310 2024-09-13T16:27:16.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.411+0000 7fb07ffff700 1 -- 172.21.15.119:0/1929327658 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb080026400 con 0x7fb088157310 2024-09-13T16:27:16.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.411+0000 7fb07ffff700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb068066730 0x7fb068068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.411+0000 7fb07ffff700 1 -- 172.21.15.119:0/1929327658 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fb080085ed0 con 0x7fb088157310 2024-09-13T16:27:16.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.411+0000 7fb0877fe700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb068066730 0x7fb068068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.411+0000 7fb0877fe700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb068066730 0x7fb068068bf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb0880c5ca0 tx=0x7fb070007e10 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.415+0000 7fb07ffff700 1 -- 172.21.15.119:0/1929327658 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb080056ea0 con 0x7fb088157310 2024-09-13T16:27:16.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.579+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb0880c4ba0 con 0x7fb088157310 2024-09-13T16:27:16.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.579+0000 7fb07ffff700 1 -- 172.21.15.119:0/1929327658 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v45) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb080056a30 con 0x7fb088157310 2024-09-13T16:27:16.585 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:27:16.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb068066730 msgr2=0x7fb068068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb068066730 0x7fb068068bf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb0880c5ca0 tx=0x7fb070007e10 comp rx=0 tx=0).stop 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb088157310 msgr2=0x7fb0880c6610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb088157310 0x7fb0880c6610 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb080014040 tx=0x7fb080004af0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 shutdown_connections 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb068066730 0x7fb068068bf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb088157310 0x7fb0880c6610 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0880714d0 0x7fb0880c8cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 --2- 172.21.15.119:0/1929327658 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb08806da10 0x7fb0880ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 >> 172.21.15.119:0/1929327658 conn(0x7fb08806b460 msgr2=0x7fb0880b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 shutdown_connections 2024-09-13T16:27:16.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.583+0000 7fb08df40700 1 -- 172.21.15.119:0/1929327658 wait complete. 2024-09-13T16:27:16.601 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-09-13T16:27:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca3a474700 1 Processor -- start 2024-09-13T16:27:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca3a474700 1 -- start start 2024-09-13T16:27:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca3a474700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cc060 0x7fca340cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca3a474700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fca340cde70 0x7fca340b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca3a474700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca3406cf60 con 0x7fca340adb30 2024-09-13T16:27:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca3a474700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca340cc4a0 con 0x7fca340cde70 2024-09-13T16:27:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca3a474700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3406c350 con 0x7fca340cc060 2024-09-13T16:27:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca38c71700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cc060 0x7fca340cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.711+0000 7fca39c73700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fca340cde70 0x7fca340b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca38c71700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cc060 0x7fca340cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:51626/0 (socket says 172.21.15.119:51626) 2024-09-13T16:27:16.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca38c71700 1 -- 172.21.15.119:0/1291165386 learned_addr learned my addr 172.21.15.119:0/1291165386 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:16.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca39c73700 1 -- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cc060 msgr2=0x7fca340cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:16.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca39c73700 1 --2- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cc060 0x7fca340cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca39c73700 1 -- 172.21.15.119:0/1291165386 >> v1:172.21.15.145:6789/0 conn(0x7fca340adb30 legacy=0x7fca3406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca39c73700 1 -- 172.21.15.119:0/1291165386 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3406c630 con 0x7fca340cde70 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca38c71700 1 --2- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cc060 0x7fca340cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca39c73700 1 --2- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340cde70 0x7fca340b2380 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009850 tx=0x7fca2c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=575d0bac5de00593 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca2bfff700 1 -- 172.21.15.119:0/1291165386 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c011830 con 0x7fca340cde70 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca2bfff700 1 -- 172.21.15.119:0/1291165386 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2c011990 con 0x7fca340cde70 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca2bfff700 1 -- 172.21.15.119:0/1291165386 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2c011d40 con 0x7fca340cde70 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 -- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340cde70 msgr2=0x7fca340b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 --2- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340cde70 0x7fca340b2380 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fca2c009850 tx=0x7fca2c009b60 comp rx=0 tx=0).stop 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 -- 172.21.15.119:0/1291165386 shutdown_connections 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 --2- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340cde70 0x7fca340b2380 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 --2- 172.21.15.119:0/1291165386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cc060 0x7fca340cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 -- 172.21.15.119:0/1291165386 >> 172.21.15.119:0/1291165386 conn(0x7fca3406a350 msgr2=0x7fca3406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:16.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 -- 172.21.15.119:0/1291165386 shutdown_connections 2024-09-13T16:27:16.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 -- 172.21.15.119:0/1291165386 wait complete. 2024-09-13T16:27:16.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 Processor -- start 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.715+0000 7fca3a474700 1 -- start start 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340adb30 0x7fca3413cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca340cc060 0x7fca341f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cde70 0x7fca341efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca3413a4e0 con 0x7fca340cc060 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca341f0740 con 0x7fca340adb30 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca3413d810 con 0x7fca340cde70 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca340cc060 0x7fca341f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca340cc060 0x7fca341f0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36008/0 (socket says 172.21.15.119:36008) 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 -- 172.21.15.119:0/4117005184 learned_addr learned my addr 172.21.15.119:0/4117005184 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca39c73700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cde70 0x7fca341efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca39472700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340adb30 0x7fca3413cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 -- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cde70 msgr2=0x7fca341efa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cde70 0x7fca341efa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 -- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340adb30 msgr2=0x7fca3413cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340adb30 0x7fca3413cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 -- 172.21.15.119:0/4117005184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30009b30 con 0x7fca340cc060 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca38c71700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca340cc060 0x7fca341f0bd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fca3000efc0 tx=0x7fca3000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca29ffb700 1 -- 172.21.15.119:0/4117005184 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3000cb00 con 0x7fca340cc060 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca29ffb700 1 -- 172.21.15.119:0/4117005184 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3000f040 con 0x7fca340cc060 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca2c0094b0 con 0x7fca340cc060 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca29ffb700 1 -- 172.21.15.119:0/4117005184 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3000ce80 con 0x7fca340cc060 2024-09-13T16:27:16.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca3413d530 con 0x7fca340cc060 2024-09-13T16:27:16.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca340adf10 con 0x7fca340cc060 2024-09-13T16:27:16.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.719+0000 7fca29ffb700 1 -- 172.21.15.119:0/4117005184 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fca30014480 con 0x7fca340cc060 2024-09-13T16:27:16.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.723+0000 7fca29ffb700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca18066730 0x7fca18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:16.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.723+0000 7fca29ffb700 1 -- 172.21.15.119:0/4117005184 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fca300845e0 con 0x7fca340cc060 2024-09-13T16:27:16.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.723+0000 7fca39472700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca18066730 0x7fca18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:16.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.723+0000 7fca39472700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca18066730 0x7fca18068bf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fca340acd50 tx=0x7fca24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:16.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.731+0000 7fca29ffb700 1 -- 172.21.15.119:0/4117005184 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fca30051b10 con 0x7fca340cc060 2024-09-13T16:27:16.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:16.959+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7fca341f0880 con 0x7fca340cc060 2024-09-13T16:27:17.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.939+0000 7fca29ffb700 1 -- 172.21.15.119:0/4117005184 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v124) v1 ==== 152+0+0 (secure 0 0 0) 0x7fca30055130 con 0x7fca340cc060 2024-09-13T16:27:17.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca18066730 msgr2=0x7fca18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca18066730 0x7fca18068bf0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fca340acd50 tx=0x7fca24009040 comp rx=0 tx=0).stop 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca340cc060 msgr2=0x7fca341f0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca340cc060 0x7fca341f0bd0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fca3000efc0 tx=0x7fca3000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 shutdown_connections 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca18066730 0x7fca18068bf0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca340cde70 0x7fca341efa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca340cc060 0x7fca341f0bd0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 --2- 172.21.15.119:0/4117005184 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca340adb30 0x7fca3413cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 >> 172.21.15.119:0/4117005184 conn(0x7fca3406a350 msgr2=0x7fca34066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 shutdown_connections 2024-09-13T16:27:17.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:17.943+0000 7fca3a474700 1 -- 172.21.15.119:0/4117005184 wait complete. 2024-09-13T16:27:17.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T16:27:18.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef74dc700 1 Processor -- start 2024-09-13T16:27:18.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef74dc700 1 -- start start 2024-09-13T16:27:18.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef74dc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 0x7fcef00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef74dc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcef015dbe0 0x7fcef00cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef74dc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcef00cd4d0 con 0x7fcef00b3260 2024-09-13T16:27:18.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef74dc700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcef0071be0 con 0x7fcef015dbe0 2024-09-13T16:27:18.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef74dc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef0159af0 con 0x7fcef00add60 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef6cdb700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcef00b3260 0x7fcef006fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46090/0 (socket says 172.21.15.119:46090) 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef6cdb700 1 -- 172.21.15.119:0/1577155696 learned_addr learned my addr 172.21.15.119:0/1577155696 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/1577155696 >> v2:172.21.15.145:3300/0 conn(0x7fcef015dbe0 0x7fcef00cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef64da700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 0x7fcef00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132148176 0 0) 0x7fcef00cd4d0 con 0x7fcef00b3260 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.091+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcedc0035d0 con 0x7fcef00b3260 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385555398 0 0) 0x7fcedc0035d0 con 0x7fcef00b3260 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 msgr2=0x7fcef00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 0x7fcef00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 >> v2:172.21.15.145:3300/0 conn(0x7fcef015dbe0 msgr2=0x7fcef00cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 --2- 172.21.15.119:0/1577155696 >> v2:172.21.15.145:3300/0 conn(0x7fcef015dbe0 0x7fcef00cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef015e130 con 0x7fcef00b3260 2024-09-13T16:27:18.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fcee80019f0 con 0x7fcef00b3260 2024-09-13T16:27:18.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/1577155696 >> v2:172.21.15.145:3300/0 conn(0x7fcef015dbe0 0x7fcef00cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:18.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 >> v1:172.21.15.119:6789/0 conn(0x7fcef00b3260 legacy=0x7fcef006fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc003a20 0x7fcedc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcedc004870 0x7fcedc004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 0x7fcedc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef0071be0 con 0x7fcedc004870 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef0159af0 con 0x7fcedc003a20 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcedc00dc40 con 0x7fcef00add60 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcedc004870 0x7fcedc004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef64da700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc003a20 0x7fcedc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef6cdb700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 0x7fcedc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 -- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 msgr2=0x7fcedc005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 0x7fcedc005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 -- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc003a20 msgr2=0x7fcedc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc003a20 0x7fcedc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 -- 172.21.15.119:0/1577155696 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcedc0035d0 con 0x7fcedc004870 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef6cdb700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef00add60 0x7fcedc005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcedc004870 0x7fcedc004cf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fceec009d10 tx=0x7fcedc0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=122aa1fa1136148d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec009e60 con 0x7fcedc004870 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec004030 con 0x7fcedc004870 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef54d8700 1 -- 172.21.15.119:0/1577155696 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec0043e0 con 0x7fcedc004870 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef74dc700 1 -- 172.21.15.119:0/1577155696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcedc004870 msgr2=0x7fcedc004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef74dc700 1 --2- 172.21.15.119:0/1577155696 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcedc004870 0x7fcedc004cf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fceec009d10 tx=0x7fcedc0058a0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef6cdb700 1 -- 172.21.15.119:0/1577155696 reap_dead start 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef74dc700 1 -- 172.21.15.119:0/1577155696 shutdown_connections 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef74dc700 1 -- 172.21.15.119:0/1577155696 >> 172.21.15.119:0/1577155696 conn(0x7fcef00b0980 msgr2=0x7fcef00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef74dc700 1 -- 172.21.15.119:0/1577155696 shutdown_connections 2024-09-13T16:27:18.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.095+0000 7fcef74dc700 1 -- 172.21.15.119:0/1577155696 wait complete. 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 Processor -- start 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 -- start start 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc004870 0x7fcef01ed770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcef014db90 0x7fcef0159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef015ac30 0x7fcef015b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef015b5f0 con 0x7fcef014db90 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcef01edcb0 con 0x7fcedc004870 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef74dc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcef0152d70 con 0x7fcef015ac30 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef5cd9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcef014db90 0x7fcef0159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef5cd9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcef014db90 0x7fcef0159c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36016/0 (socket says 172.21.15.119:36016) 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef5cd9700 1 -- 172.21.15.119:0/2779230273 learned_addr learned my addr 172.21.15.119:0/2779230273 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.099+0000 7fcef64da700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc004870 0x7fcef01ed770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef6cdb700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef015ac30 0x7fcef015b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef5cd9700 1 -- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef015ac30 msgr2=0x7fcef015b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef015ac30 0x7fcef015b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef5cd9700 1 -- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc004870 msgr2=0x7fcef01ed770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc004870 0x7fcef01ed770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef5cd9700 1 -- 172.21.15.119:0/2779230273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcef015a170 con 0x7fcef014db90 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef6cdb700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef015ac30 0x7fcef015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef5cd9700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcef014db90 0x7fcef0159c30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e790 tx=0x7fceec0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcee6ffd700 1 -- 172.21.15.119:0/2779230273 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec020070 con 0x7fcef014db90 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcee6ffd700 1 -- 172.21.15.119:0/2779230273 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fceec004570 con 0x7fcef014db90 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcef0152160 con 0x7fcef014db90 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcee6ffd700 1 -- 172.21.15.119:0/2779230273 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fceec006ea0 con 0x7fcef014db90 2024-09-13T16:27:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcef0152720 con 0x7fcef014db90 2024-09-13T16:27:18.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcef0152f00 con 0x7fcef014db90 2024-09-13T16:27:18.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcee6ffd700 1 -- 172.21.15.119:0/2779230273 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fceec024070 con 0x7fcef014db90 2024-09-13T16:27:18.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.103+0000 7fcee6ffd700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fced4066760 0x7fced4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.107+0000 7fcee6ffd700 1 -- 172.21.15.119:0/2779230273 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fceec085110 con 0x7fcef014db90 2024-09-13T16:27:18.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.107+0000 7fcef64da700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fced4066760 0x7fced4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.107+0000 7fcef64da700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fced4066760 0x7fced4068c20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcee00099e0 tx=0x7fcee0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.111+0000 7fcee6ffd700 1 -- 172.21.15.119:0/2779230273 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fceec0527f0 con 0x7fcef014db90 2024-09-13T16:27:18.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.287+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcef00ae140 con 0x7fcef014db90 2024-09-13T16:27:18.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.295+0000 7fcee6ffd700 1 -- 172.21.15.119:0/2779230273 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7fceec055e10 con 0x7fcef014db90 2024-09-13T16:27:18.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.295+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fced4066760 msgr2=0x7fced4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.295+0000 7fcef74dc700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fced4066760 0x7fced4068c20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fcee00099e0 tx=0x7fcee0008040 comp rx=0 tx=0).stop 2024-09-13T16:27:18.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.295+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcef014db90 msgr2=0x7fcef0159c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.295+0000 7fcef74dc700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcef014db90 0x7fcef0159c30 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fceec00e790 tx=0x7fceec0047e0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 shutdown_connections 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fced4066760 0x7fced4068c20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcef015ac30 0x7fcef015b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcef014db90 0x7fcef0159c30 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 --2- 172.21.15.119:0/2779230273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcedc004870 0x7fcef01ed770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 >> 172.21.15.119:0/2779230273 conn(0x7fcef00b0980 msgr2=0x7fcef0066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 shutdown_connections 2024-09-13T16:27:18.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.299+0000 7fcef74dc700 1 -- 172.21.15.119:0/2779230273 wait complete. 2024-09-13T16:27:18.317 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:27:18.318 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:27:18.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:27:18.355 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:27:18.355 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:27:18.355 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:18.386 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:27:18.386 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:27:18.386 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:18.405 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:27:18.405 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:27:18.405 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:18.412 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:27:18.412 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:27:18.413 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:27:18.420 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:27:18.420 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T16:27:18.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 Processor -- start 2024-09-13T16:27:18.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- start start 2024-09-13T16:27:18.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8ff015a620 0x7f8ff00b3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff00ac960 con 0x7f8ff006fd40 2024-09-13T16:27:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff00acc40 con 0x7f8ff01612f0 2024-09-13T16:27:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff00acf20 con 0x7f8ff015a620 2024-09-13T16:27:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff636a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8ff006fd40 0x7f8ff01606c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46094/0 (socket says 172.21.15.119:46094) 2024-09-13T16:27:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff636a700 1 -- 172.21.15.119:0/18249446 learned_addr learned my addr 172.21.15.119:0/18249446 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff5b69700 1 --2- 172.21.15.119:0/18249446 >> v2:172.21.15.145:3300/0 conn(0x7f8ff015a620 0x7f8ff00b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff4b67700 1 -- 172.21.15.119:0/18249446 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925492992 0 0) 0x7f8ff00ac960 con 0x7f8ff006fd40 2024-09-13T16:27:18.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff4b67700 1 -- 172.21.15.119:0/18249446 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fdc0035d0 con 0x7f8ff006fd40 2024-09-13T16:27:18.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff5b69700 1 -- 172.21.15.119:0/18249446 >> v1:172.21.15.145:6789/0 conn(0x7f8ff01612f0 legacy=0x7f8ff00ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff5b69700 1 -- 172.21.15.119:0/18249446 >> v1:172.21.15.119:6789/0 conn(0x7f8ff006fd40 legacy=0x7f8ff01606c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff5b69700 1 -- 172.21.15.119:0/18249446 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff00ad200 con 0x7f8ff015a620 2024-09-13T16:27:18.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff5b69700 1 --2- 172.21.15.119:0/18249446 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff015a620 0x7f8ff00b3720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009d90 tx=0x7f8fec00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9a893f08e72345c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff4b67700 1 -- 172.21.15.119:0/18249446 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00e070 con 0x7f8ff015a620 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff4b67700 1 -- 172.21.15.119:0/18249446 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec00bb20 con 0x7f8ff015a620 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff4b67700 1 -- 172.21.15.119:0/18249446 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fec00be90 con 0x7f8ff015a620 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/18249446 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff015a620 msgr2=0x7f8ff00b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/18249446 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff015a620 0x7f8ff00b3720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8fec009d90 tx=0x7f8fec00b330 comp rx=0 tx=0).stop 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/18249446 shutdown_connections 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/18249446 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff015a620 0x7f8ff00b3720 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/18249446 >> 172.21.15.119:0/18249446 conn(0x7f8ff006a350 msgr2=0x7f8ff006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/18249446 shutdown_connections 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.535+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/18249446 wait complete. 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 Processor -- start 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 -- start start 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff006fd40 0x7f8ff0157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff015a620 0x7f8ff0150ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff01612f0 0x7f8ff0151010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff015cb30 con 0x7f8ff015a620 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff015ce10 con 0x7f8ff01612f0 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff7dcd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff015d0f0 con 0x7f8ff006fd40 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff015a620 0x7f8ff0150ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff015a620 0x7f8ff0150ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36030/0 (socket says 172.21.15.119:36030) 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 -- 172.21.15.119:0/4227660093 learned_addr learned my addr 172.21.15.119:0/4227660093 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff636a700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff01612f0 0x7f8ff0151010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5b69700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff006fd40 0x7f8ff0157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 -- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff006fd40 msgr2=0x7f8ff0157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff006fd40 0x7f8ff0157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 -- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff01612f0 msgr2=0x7f8ff0151010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff01612f0 0x7f8ff0151010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 -- 172.21.15.119:0/4227660093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fec009a70 con 0x7f8ff015a620 2024-09-13T16:27:18.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8ff5368700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff015a620 0x7f8ff0150ad0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c410 tx=0x7f8fe000c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.539+0000 7f8fe67fc700 1 -- 172.21.15.119:0/4227660093 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe00052d0 con 0x7f8ff015a620 2024-09-13T16:27:18.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff015d3d0 con 0x7f8ff015a620 2024-09-13T16:27:18.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8fe67fc700 1 -- 172.21.15.119:0/4227660093 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe0005430 con 0x7f8ff015a620 2024-09-13T16:27:18.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8fe67fc700 1 -- 172.21.15.119:0/4227660093 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fe00055a0 con 0x7f8ff015a620 2024-09-13T16:27:18.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff015be20 con 0x7f8ff015a620 2024-09-13T16:27:18.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff0067760 con 0x7f8ff015a620 2024-09-13T16:27:18.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8fe67fc700 1 -- 172.21.15.119:0/4227660093 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8fe00036a0 con 0x7f8ff015a620 2024-09-13T16:27:18.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8fe67fc700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fd4066730 0x7f8fd4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.543+0000 7f8fe67fc700 1 -- 172.21.15.119:0/4227660093 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f8fe00832f0 con 0x7f8ff015a620 2024-09-13T16:27:18.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.547+0000 7f8ff5b69700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fd4066730 0x7f8fd4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.547+0000 7f8ff5b69700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fd4066730 0x7f8fd4068bf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00b370 tx=0x7f8fec011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.555+0000 7f8fe67fc700 1 -- 172.21.15.119:0/4227660093 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8fe00507a0 con 0x7f8ff015a620 2024-09-13T16:27:18.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.759+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ff00b3720 con 0x7f8ff015a620 2024-09-13T16:27:18.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8fe67fc700 1 -- 172.21.15.119:0/4227660093 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f8fe0053dc0 con 0x7f8ff015a620 2024-09-13T16:27:18.767 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:18.768 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5455,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/2438328857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2438328857},{"type":"v1","addr":"172.21.15.145:6836","nonce":2438328857}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5464,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2394723626","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2394723626},{"type":"v1","addr":"172.21.15.119:6835","nonce":2394723626}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5577,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3952304700","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3952304700},{"type":"v1","addr":"172.21.15.119:6833","nonce":3952304700}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":5803,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/15159786","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":15159786},{"type":"v1","addr":"172.21.15.145:6837","nonce":15159786}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":18}],"filesystems":[]} 2024-09-13T16:27:18.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fd4066730 msgr2=0x7f8fd4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fd4066730 0x7f8fd4068bf0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8fec00b370 tx=0x7f8fec011000 comp rx=0 tx=0).stop 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff015a620 msgr2=0x7f8ff0150ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff015a620 0x7f8ff0150ad0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f8fe000c410 tx=0x7f8fe000c7d0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 shutdown_connections 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8fd4066730 0x7f8fd4068bf0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ff01612f0 0x7f8ff0151010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ff015a620 0x7f8ff0150ad0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 --2- 172.21.15.119:0/4227660093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ff006fd40 0x7f8ff0157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 >> 172.21.15.119:0/4227660093 conn(0x7f8ff006a350 msgr2=0x7f8ff01603b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:18.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 shutdown_connections 2024-09-13T16:27:18.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.763+0000 7f8ff7dcd700 1 -- 172.21.15.119:0/4227660093 wait complete. 2024-09-13T16:27:18.772 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 18 2024-09-13T16:27:18.782 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:27:18.783 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T16:27:18.783 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:27:18.783 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:27:18.783 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:27:18.783 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T16:27:18.784 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:27:18.784 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:27:18.784 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:27:18.784 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T16:27:18.784 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:27:18.784 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:27:18.785 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:27:18.785 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T16:27:18.785 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:27:18.785 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:27:18.797 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:27:18.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:27:18.798 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:27:18.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:27:18.799 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:27:18.800 DEBUG:teuthology.parallel:result is None 2024-09-13T16:27:18.801 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:27:18.801 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:27:18.802 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:27:18.802 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:27:18.803 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:27:18.803 DEBUG:teuthology.parallel:result is None 2024-09-13T16:27:18.804 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:27:18.804 DEBUG:teuthology.parallel:result is None 2024-09-13T16:27:18.805 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:27:18.806 DEBUG:teuthology.parallel:result is None 2024-09-13T16:27:18.806 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T16:27:18.865 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:27:18.874 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:27:18.893 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:27:18.896 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:27:18.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f726a4f9700 1 Processor -- start 2024-09-13T16:27:18.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f726a4f9700 1 -- start start 2024-09-13T16:27:18.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f726a4f9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a630 0x7f726406e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f726a4f9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726406dd40 con 0x7f726415a250 2024-09-13T16:27:18.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f726a4f9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f726406d430 con 0x7f7264070c10 2024-09-13T16:27:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f726a4f9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f726406d6f0 con 0x7f726415a630 2024-09-13T16:27:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f72694f7700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f726415a250 0x7f72640b3210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46116/0 (socket says 172.21.15.119:46116) 2024-09-13T16:27:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f72694f7700 1 -- 172.21.15.119:0/4210560187 learned_addr learned my addr 172.21.15.119:0/4210560187 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f7268cf6700 1 --2- 172.21.15.119:0/4210560187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a630 0x7f726406e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f725bfff700 1 -- 172.21.15.119:0/4210560187 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3002077111 0 0) 0x7f726406dd40 con 0x7f726415a250 2024-09-13T16:27:18.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f725bfff700 1 -- 172.21.15.119:0/4210560187 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72500035d0 con 0x7f726415a250 2024-09-13T16:27:18.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f7268cf6700 1 -- 172.21.15.119:0/4210560187 >> v1:172.21.15.145:6789/0 conn(0x7f7264070c10 legacy=0x7f72640b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f725bfff700 1 -- 172.21.15.119:0/4210560187 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2959018501 0 0) 0x7f72500035d0 con 0x7f726415a250 2024-09-13T16:27:18.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f7268cf6700 1 -- 172.21.15.119:0/4210560187 >> v1:172.21.15.119:6789/0 conn(0x7f726415a250 legacy=0x7f72640b3210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.911+0000 7f7268cf6700 1 -- 172.21.15.119:0/4210560187 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f726406d9d0 con 0x7f726415a630 2024-09-13T16:27:18.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f7268cf6700 1 --2- 172.21.15.119:0/4210560187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a630 0x7f726406e1d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7254009880 tx=0x7f7254009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd779f4a3b1a9eb2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f725bfff700 1 -- 172.21.15.119:0/4210560187 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7260001670 con 0x7f726415a630 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f725bfff700 1 -- 172.21.15.119:0/4210560187 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7254004030 con 0x7f726415a630 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f725bfff700 1 -- 172.21.15.119:0/4210560187 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72540043c0 con 0x7f726415a630 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 -- 172.21.15.119:0/4210560187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a630 msgr2=0x7f726406e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 --2- 172.21.15.119:0/4210560187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a630 0x7f726406e1d0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7254009880 tx=0x7f7254009b90 comp rx=0 tx=0).stop 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 -- 172.21.15.119:0/4210560187 shutdown_connections 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 --2- 172.21.15.119:0/4210560187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a630 0x7f726406e1d0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 -- 172.21.15.119:0/4210560187 >> 172.21.15.119:0/4210560187 conn(0x7f72640b04f0 msgr2=0x7f72640b08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 -- 172.21.15.119:0/4210560187 shutdown_connections 2024-09-13T16:27:18.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 -- 172.21.15.119:0/4210560187 wait complete. 2024-09-13T16:27:18.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 Processor -- start 2024-09-13T16:27:18.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.915+0000 7f726a4f9700 1 -- start start 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7264070c10 0x7f7264142cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a250 0x7f7264158510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f726415a630 0x7f726413efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264158a50 con 0x7f726415a630 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264143360 con 0x7f7264070c10 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7264143640 con 0x7f726415a250 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f726415a630 0x7f726413efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f72694f7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7264070c10 0x7f7264142cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f726415a630 0x7f726413efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36080/0 (socket says 172.21.15.119:36080) 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f72694f7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7264070c10 0x7f7264142cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:41472/0 (socket says 172.21.15.119:41472) 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 -- 172.21.15.119:0/828143840 learned_addr learned my addr 172.21.15.119:0/828143840 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7268cf6700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a250 0x7f7264158510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 -- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a250 msgr2=0x7f7264158510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a250 0x7f7264158510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 -- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7264070c10 msgr2=0x7f7264142cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7264070c10 0x7f7264142cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 -- 172.21.15.119:0/828143840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72540094b0 con 0x7f726415a630 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7268cf6700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a250 0x7f7264158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7269cf8700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f726415a630 0x7f726413efc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f725c00cc50 tx=0x7f725c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7259ffb700 1 -- 172.21.15.119:0/828143840 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c003c20 con 0x7f726415a630 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7259ffb700 1 -- 172.21.15.119:0/828143840 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f725c003d80 con 0x7f726415a630 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7264143920 con 0x7f726415a630 2024-09-13T16:27:18.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f7259ffb700 1 -- 172.21.15.119:0/828143840 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c00aab0 con 0x7f726415a630 2024-09-13T16:27:18.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.919+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7264143ee0 con 0x7f726415a630 2024-09-13T16:27:18.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.923+0000 7f7259ffb700 1 -- 172.21.15.119:0/828143840 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f725c00ac10 con 0x7f726415a630 2024-09-13T16:27:18.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.923+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72640ce5b0 con 0x7f726415a630 2024-09-13T16:27:18.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.923+0000 7f7259ffb700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7248066730 0x7f7248068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:18.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.923+0000 7f7259ffb700 1 -- 172.21.15.119:0/828143840 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f725c083ef0 con 0x7f726415a630 2024-09-13T16:27:18.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.923+0000 7f72694f7700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7248066730 0x7f7248068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:18.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.923+0000 7f72694f7700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7248066730 0x7f7248068bf0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7260001eb0 tx=0x7f7260001730 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:18.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:18.927+0000 7f7259ffb700 1 -- 172.21.15.119:0/828143840 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f725c0514b0 con 0x7f726415a630 2024-09-13T16:27:19.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.107+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f7264071500 con 0x7f726415a630 2024-09-13T16:27:19.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.947+0000 7f7259ffb700 1 -- 172.21.15.119:0/828143840 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v50) v1 ==== 97+0+0 (secure 0 0 0) 0x7f725c054ad0 con 0x7f726415a630 2024-09-13T16:27:19.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7248066730 msgr2=0x7f7248068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7248066730 0x7f7248068bf0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f7260001eb0 tx=0x7f7260001730 comp rx=0 tx=0).stop 2024-09-13T16:27:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f726415a630 msgr2=0x7f726413efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f726415a630 0x7f726413efc0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7f725c00cc50 tx=0x7f725c00cf60 comp rx=0 tx=0).stop 2024-09-13T16:27:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 shutdown_connections 2024-09-13T16:27:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7248066730 0x7f7248068bf0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f726415a630 0x7f726413efc0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:19.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f726415a250 0x7f7264158510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:19.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 --2- 172.21.15.119:0/828143840 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7264070c10 0x7f7264142cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:19.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 >> 172.21.15.119:0/828143840 conn(0x7f72640b04f0 msgr2=0x7f72640ad3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:19.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 shutdown_connections 2024-09-13T16:27:19.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:19.955+0000 7f726a4f9700 1 -- 172.21.15.119:0/828143840 wait complete. 2024-09-13T16:27:19.963 INFO:teuthology.orchestra.run.smithi119.stderr: removed all blocklist entries 2024-09-13T16:27:19.976 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T16:27:20.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bb59e700 1 Processor -- start 2024-09-13T16:27:20.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bb59e700 1 -- start start 2024-09-13T16:27:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bb59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f25bc0cd8d0 0x7f25bc06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bb59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f25bc0b17f0 0x7f25bc0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bb59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25bc0705d0 con 0x7f25bc0b2200 2024-09-13T16:27:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bb59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25bc06d0d0 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bb59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25bc06fca0 con 0x7f25bc0b17f0 2024-09-13T16:27:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f25bc0b2200 0x7f25bc070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46118/0 (socket says 172.21.15.119:46118) 2024-09-13T16:27:20.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25bad9d700 1 -- 172.21.15.119:0/3018454570 learned_addr learned my addr 172.21.15.119:0/3018454570 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25ba59c700 1 --2- 172.21.15.119:0/3018454570 >> v2:172.21.15.119:3300/0 conn(0x7f25bc0cd8d0 0x7f25bc06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25b9d9b700 1 --2- 172.21.15.119:0/3018454570 >> v2:172.21.15.145:3300/0 conn(0x7f25bc0b17f0 0x7f25bc0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25ba59c700 1 -- 172.21.15.119:0/3018454570 >> v2:172.21.15.145:3300/0 conn(0x7f25bc0b17f0 msgr2=0x7f25bc0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25b959a700 1 -- 172.21.15.119:0/3018454570 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 38965322 0 0) 0x7f25bc0705d0 con 0x7f25bc0b2200 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25ba59c700 1 --2- 172.21.15.119:0/3018454570 >> v2:172.21.15.145:3300/0 conn(0x7f25bc0b17f0 0x7f25bc0cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25ba59c700 1 -- 172.21.15.119:0/3018454570 >> v1:172.21.15.119:6789/0 conn(0x7f25bc0b2200 legacy=0x7f25bc070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25ba59c700 1 -- 172.21.15.119:0/3018454570 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25bc06ff80 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.087+0000 7f25ba59c700 1 --2- 172.21.15.119:0/3018454570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc06cb90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0ae2c0 tx=0x7f25b0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99069395996e9239 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25b959a700 1 -- 172.21.15.119:0/3018454570 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0011830 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25b959a700 1 -- 172.21.15.119:0/3018454570 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b0011990 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25b959a700 1 -- 172.21.15.119:0/3018454570 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b0011d20 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- 172.21.15.119:0/3018454570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 msgr2=0x7f25bc06cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 --2- 172.21.15.119:0/3018454570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc06cb90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f25bc0ae2c0 tx=0x7f25b0009b30 comp rx=0 tx=0).stop 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- 172.21.15.119:0/3018454570 shutdown_connections 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 --2- 172.21.15.119:0/3018454570 >> v2:172.21.15.145:3300/0 conn(0x7f25bc0b17f0 0x7f25bc0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 --2- 172.21.15.119:0/3018454570 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc06cb90 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- 172.21.15.119:0/3018454570 >> 172.21.15.119:0/3018454570 conn(0x7f25bc06a350 msgr2=0x7f25bc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- 172.21.15.119:0/3018454570 shutdown_connections 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- 172.21.15.119:0/3018454570 wait complete. 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 Processor -- start 2024-09-13T16:27:20.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- start start 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25bc0b17f0 0x7f25bc14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25bc0b2200 0x7f25bc0b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc1491a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc160e30 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc0a8900 con 0x7f25bc0b17f0 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25bc149f70 con 0x7f25bc0b2200 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc1491a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc1491a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36094/0 (socket says 172.21.15.119:36094) 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 -- 172.21.15.119:0/1731377180 learned_addr learned my addr 172.21.15.119:0/1731377180 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25b9d9b700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25bc0b2200 0x7f25bc0b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25ba59c700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25bc0b17f0 0x7f25bc14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 -- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25bc0b2200 msgr2=0x7f25bc0b09f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25bc0b2200 0x7f25bc0b09f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 -- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25bc0b17f0 msgr2=0x7f25bc14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25bc0b17f0 0x7f25bc14c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.091+0000 7f25bad9d700 1 -- 172.21.15.119:0/1731377180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25b00094b0 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25bad9d700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc1491a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c550 tx=0x7f25b400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25aaffd700 1 -- 172.21.15.119:0/1731377180 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4004ca0 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25aaffd700 1 -- 172.21.15.119:0/1731377180 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25b4004e40 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25bc14a250 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25aaffd700 1 -- 172.21.15.119:0/1731377180 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25b4007310 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25bc14ce00 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25bc14a770 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25aaffd700 1 -- 172.21.15.119:0/1731377180 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f25b4007470 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25aaffd700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f259c066730 0x7f259c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25aaffd700 1 -- 172.21.15.119:0/1731377180 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f25b4083b10 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.095+0000 7f25ba59c700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f259c066730 0x7f259c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.099+0000 7f25ba59c700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f259c066730 0x7f259c068bf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f25b0000c00 tx=0x7f25b0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.103+0000 7f25aaffd700 1 -- 172.21.15.119:0/1731377180 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f25bc14a770 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.331+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f25bc14a770 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.331+0000 7f25aaffd700 1 -- 172.21.15.119:0/1731377180 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f25bc14a770 con 0x7f25bc0cd8d0 2024-09-13T16:27:20.338 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:20.338 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_dump": [ 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.a", 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA==", 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.339 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.b", 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g==", 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.c", 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ==", 2024-09-13T16:27:20.340 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.d", 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw==", 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:27:20.341 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.0", 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA==", 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.342 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.1", 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw==", 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.2", 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ==", 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.343 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.3", 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww==", 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.344 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.4", 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA==", 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.5", 2024-09-13T16:27:20.345 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q==", 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.6", 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg==", 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.346 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.7", 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ==", 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.0", 2024-09-13T16:27:20.347 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ==", 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow r", 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.1", 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ==", 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:27:20.348 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.2", 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw==", 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.349 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.3", 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow==", 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.admin", 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmpEcAJRAAYorOK9AU5Q/2uOTIUrZhJw==", 2024-09-13T16:27:20.350 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.x", 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ==", 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.351 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.y", 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw==", 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:27:20.352 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f259c066730 msgr2=0x7f259c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f259c066730 0x7f259c068bf0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f25b0000c00 tx=0x7f25b0008030 comp rx=0 tx=0).stop 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 msgr2=0x7f25bc1491a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc1491a0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f25b400c550 tx=0x7f25b400c860 comp rx=0 tx=0).stop 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 shutdown_connections 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f259c066730 0x7f259c068bf0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25bc0cd8d0 0x7f25bc1491a0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25bc0b2200 0x7f25bc0b09f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 --2- 172.21.15.119:0/1731377180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25bc0b17f0 0x7f25bc14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 >> 172.21.15.119:0/1731377180 conn(0x7f25bc06a350 msgr2=0x7f25bc0cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:20.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 shutdown_connections 2024-09-13T16:27:20.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.335+0000 7f25bb59e700 1 -- 172.21.15.119:0/1731377180 wait complete. 2024-09-13T16:27:20.355 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:20.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f75823a0700 1 Processor -- start 2024-09-13T16:27:20.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f75823a0700 1 -- start start 2024-09-13T16:27:20.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f75823a0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b17f0 0x7f757c0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f75823a0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f757c0b2200 0x7f757c0b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f75823a0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f757c0705d0 con 0x7f757c0cd8d0 2024-09-13T16:27:20.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f75823a0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f757c0b2bf0 con 0x7f757c0b2200 2024-09-13T16:27:20.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f75823a0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c06fca0 con 0x7f757c0b17f0 2024-09-13T16:27:20.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f7581b9f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f757c0b2200 0x7f757c0b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f7580b9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b17f0 0x7f757c0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f7581b9f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f757c0b2200 0x7f757c0b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:41506/0 (socket says 172.21.15.119:41506) 2024-09-13T16:27:20.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f758139e700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f757c0cd8d0 0x7f757c06cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:53266/0 (socket says 172.21.15.119:53266) 2024-09-13T16:27:20.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.455+0000 7f7581b9f700 1 -- 172.21.15.119:0/3028636264 learned_addr learned my addr 172.21.15.119:0/3028636264 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:20.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7581b9f700 1 -- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b17f0 msgr2=0x7f757c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:20.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7581b9f700 1 --2- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b17f0 0x7f757c0cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7581b9f700 1 -- 172.21.15.119:0/3028636264 >> v1:172.21.15.145:6789/0 conn(0x7f757c0cd8d0 legacy=0x7f757c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7581b9f700 1 -- 172.21.15.119:0/3028636264 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f757c06ff80 con 0x7f757c0b2200 2024-09-13T16:27:20.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7580b9d700 1 --2- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b17f0 0x7f757c0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:20.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7581b9f700 1 --2- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0b2200 0x7f757c0b26b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7574009e50 tx=0x7f757400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae0e26dd8dfd5441 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7573fff700 1 -- 172.21.15.119:0/3028636264 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757400e070 con 0x7f757c0b2200 2024-09-13T16:27:20.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7573fff700 1 -- 172.21.15.119:0/3028636264 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f757400b910 con 0x7f757c0b2200 2024-09-13T16:27:20.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f7573fff700 1 -- 172.21.15.119:0/3028636264 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f757400bcc0 con 0x7f757c0b2200 2024-09-13T16:27:20.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 -- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0b2200 msgr2=0x7f757c0b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 --2- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0b2200 0x7f757c0b26b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7574009e50 tx=0x7f757400b330 comp rx=0 tx=0).stop 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 -- 172.21.15.119:0/3028636264 shutdown_connections 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 --2- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0b2200 0x7f757c0b26b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 --2- 172.21.15.119:0/3028636264 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b17f0 0x7f757c0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 -- 172.21.15.119:0/3028636264 >> 172.21.15.119:0/3028636264 conn(0x7f757c06a350 msgr2=0x7f757c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 -- 172.21.15.119:0/3028636264 shutdown_connections 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 -- 172.21.15.119:0/3028636264 wait complete. 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 Processor -- start 2024-09-13T16:27:20.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.459+0000 7f75823a0700 1 -- start start 2024-09-13T16:27:20.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f757c0b17f0 0x7f757c162850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b2200 0x7f757c1f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0cd8d0 0x7f757c0aac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c1f6be0 con 0x7f757c0b17f0 2024-09-13T16:27:20.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f757c15d530 con 0x7f757c0cd8d0 2024-09-13T16:27:20.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f757c15d7d0 con 0x7f757c0b2200 2024-09-13T16:27:20.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f757c0b17f0 0x7f757c162850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f757c0b17f0 0x7f757c162850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36098/0 (socket says 172.21.15.119:36098) 2024-09-13T16:27:20.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 -- 172.21.15.119:0/646370663 learned_addr learned my addr 172.21.15.119:0/646370663 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:20.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f7580b9d700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b2200 0x7f757c1f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f7581b9f700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0cd8d0 0x7f757c0aac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 -- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b2200 msgr2=0x7f757c1f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b2200 0x7f757c1f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 -- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0cd8d0 msgr2=0x7f757c0aac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0cd8d0 0x7f757c0aac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 -- 172.21.15.119:0/646370663 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75780094b0 con 0x7f757c0b17f0 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f758139e700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f757c0b17f0 0x7f757c162850 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c4d0 tx=0x7f756c00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f7571ffb700 1 -- 172.21.15.119:0/646370663 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c00cea0 con 0x7f757c0b17f0 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f7571ffb700 1 -- 172.21.15.119:0/646370663 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f756c005cd0 con 0x7f757c0b17f0 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7574009b30 con 0x7f757c0b17f0 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f7571ffb700 1 -- 172.21.15.119:0/646370663 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f756c0072d0 con 0x7f757c0b17f0 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f757c15dd90 con 0x7f757c0b17f0 2024-09-13T16:27:20.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.463+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f757c0cdcb0 con 0x7f757c0b17f0 2024-09-13T16:27:20.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.467+0000 7f7571ffb700 1 -- 172.21.15.119:0/646370663 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f756c007430 con 0x7f757c0b17f0 2024-09-13T16:27:20.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.467+0000 7f7571ffb700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7560066730 0x7f7560068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.467+0000 7f7571ffb700 1 -- 172.21.15.119:0/646370663 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f756c0830a0 con 0x7f757c0b17f0 2024-09-13T16:27:20.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.467+0000 7f7580b9d700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7560066730 0x7f7560068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.467+0000 7f7580b9d700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7560066730 0x7f7560068bf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f757c15f360 tx=0x7f7578009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.475+0000 7f7571ffb700 1 -- 172.21.15.119:0/646370663 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f756c050760 con 0x7f757c0b17f0 2024-09-13T16:27:20.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.651+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f757c06d040 con 0x7f757c0b17f0 2024-09-13T16:27:20.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.651+0000 7f7571ffb700 1 -- 172.21.15.119:0/646370663 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+13122 (secure 0 0 0) 0x7f756c053d80 con 0x7f757c0b17f0 2024-09-13T16:27:20.658 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:20.658 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":50,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:19.942605+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:20.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7560066730 msgr2=0x7f7560068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7560066730 0x7f7560068bf0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f757c15f360 tx=0x7f7578009240 comp rx=0 tx=0).stop 2024-09-13T16:27:20.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f757c0b17f0 msgr2=0x7f757c162850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f757c0b17f0 0x7f757c162850 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7f756c00c4d0 tx=0x7f756c00c7e0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 shutdown_connections 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7560066730 0x7f7560068bf0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f757c0cd8d0 0x7f757c0aac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f757c0b2200 0x7f757c1f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 --2- 172.21.15.119:0/646370663 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f757c0b17f0 0x7f757c162850 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 >> 172.21.15.119:0/646370663 conn(0x7f757c06a350 msgr2=0x7f757c0caa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 shutdown_connections 2024-09-13T16:27:20.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.655+0000 7f75823a0700 1 -- 172.21.15.119:0/646370663 wait complete. 2024-09-13T16:27:20.673 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:27:20.674 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:27:20.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.787+0000 7f1f09625700 1 Processor -- start 2024-09-13T16:27:20.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.787+0000 7f1f09625700 1 -- start start 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1f0415d1c0 0x7f1f0415d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1f04159910 0x7f1f04159d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f0415db50 con 0x7f1f04159530 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040ce700 con 0x7f1f04159910 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f040ce9e0 con 0x7f1f0415d1c0 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1f04159910 0x7f1f04159d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1f04159910 0x7f1f04159d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36112/0 (socket says 172.21.15.119:36112) 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 -- 172.21.15.119:0/4211244876 learned_addr learned my addr 172.21.15.119:0/4211244876 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1efb7fe700 1 --2- 172.21.15.119:0/4211244876 >> v2:172.21.15.145:3300/0 conn(0x7f1f0415d1c0 0x7f1f0415d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 -- 172.21.15.119:0/4211244876 >> v2:172.21.15.145:3300/0 conn(0x7f1f0415d1c0 msgr2=0x7f1f0415d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 --2- 172.21.15.119:0/4211244876 >> v2:172.21.15.145:3300/0 conn(0x7f1f0415d1c0 0x7f1f0415d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 -- 172.21.15.119:0/4211244876 >> v1:172.21.15.145:6789/0 conn(0x7f1f04159530 legacy=0x7f1f040cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:20.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 -- 172.21.15.119:0/4211244876 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f040cecc0 con 0x7f1f04159910 2024-09-13T16:27:20.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f08e24700 1 --2- 172.21.15.119:0/4211244876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f04159d80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009e50 tx=0x7f1efc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdba644bd937457c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f037fe700 1 -- 172.21.15.119:0/4211244876 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc00e070 con 0x7f1f04159910 2024-09-13T16:27:20.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f037fe700 1 -- 172.21.15.119:0/4211244876 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1efc00b980 con 0x7f1f04159910 2024-09-13T16:27:20.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f037fe700 1 -- 172.21.15.119:0/4211244876 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1efc00bd30 con 0x7f1f04159910 2024-09-13T16:27:20.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- 172.21.15.119:0/4211244876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 msgr2=0x7f1f04159d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 --2- 172.21.15.119:0/4211244876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f04159d80 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1efc009e50 tx=0x7f1efc00b330 comp rx=0 tx=0).stop 2024-09-13T16:27:20.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- 172.21.15.119:0/4211244876 shutdown_connections 2024-09-13T16:27:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 --2- 172.21.15.119:0/4211244876 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f04159d80 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 --2- 172.21.15.119:0/4211244876 >> v2:172.21.15.145:3300/0 conn(0x7f1f0415d1c0 0x7f1f0415d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- 172.21.15.119:0/4211244876 >> 172.21.15.119:0/4211244876 conn(0x7f1f0406b460 msgr2=0x7f1f0406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- 172.21.15.119:0/4211244876 shutdown_connections 2024-09-13T16:27:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.791+0000 7f1f09625700 1 -- 172.21.15.119:0/4211244876 wait complete. 2024-09-13T16:27:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 Processor -- start 2024-09-13T16:27:20.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 -- start start 2024-09-13T16:27:20.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f04159530 0x7f1f0413e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f0415a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f0415d1c0 0x7f1f0414a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0415c670 con 0x7f1f04159910 2024-09-13T16:27:20.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0415ba80 con 0x7f1f0415d1c0 2024-09-13T16:27:20.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1efb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f0415a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f09625700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f0415bcd0 con 0x7f1f04159530 2024-09-13T16:27:20.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1efb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f0415a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36122/0 (socket says 172.21.15.119:36122) 2024-09-13T16:27:20.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1efb7fe700 1 -- 172.21.15.119:0/606616023 learned_addr learned my addr 172.21.15.119:0/606616023 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:20.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f03fff700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f04159530 0x7f1f0413e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.795+0000 7f1f08e24700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f0415d1c0 0x7f1f0414a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1efb7fe700 1 -- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f04159530 msgr2=0x7f1f0413e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1efb7fe700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f04159530 0x7f1f0413e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1efb7fe700 1 -- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f0415d1c0 msgr2=0x7f1f0414a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1efb7fe700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f0415d1c0 0x7f1f0414a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1efb7fe700 1 -- 172.21.15.119:0/606616023 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc009b30 con 0x7f1f04159910 2024-09-13T16:27:20.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f08e24700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f0415d1c0 0x7f1f0414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:20.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1efb7fe700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f0415a6e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00d800 tx=0x7f1eec00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f017fa700 1 -- 172.21.15.119:0/606616023 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec0125f0 con 0x7f1f04159910 2024-09-13T16:27:20.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f017fa700 1 -- 172.21.15.119:0/606616023 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eec012750 con 0x7f1f04159910 2024-09-13T16:27:20.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f0415bfb0 con 0x7f1f04159910 2024-09-13T16:27:20.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f017fa700 1 -- 172.21.15.119:0/606616023 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eec012ac0 con 0x7f1f04159910 2024-09-13T16:27:20.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f0413e920 con 0x7f1f04159910 2024-09-13T16:27:20.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f040b4d30 con 0x7f1f04159910 2024-09-13T16:27:20.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f017fa700 1 -- 172.21.15.119:0/606616023 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1eec012c20 con 0x7f1f04159910 2024-09-13T16:27:20.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.799+0000 7f1f017fa700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ee8066730 0x7f1ee8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:20.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.803+0000 7f1f017fa700 1 -- 172.21.15.119:0/606616023 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f1eec083fd0 con 0x7f1f04159910 2024-09-13T16:27:20.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.803+0000 7f1f03fff700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ee8066730 0x7f1ee8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:20.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.803+0000 7f1f03fff700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ee8066730 0x7f1ee8068bf0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40018a0 tx=0x7f1ef4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:20.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.807+0000 7f1f017fa700 1 -- 172.21.15.119:0/606616023 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1eec0515e0 con 0x7f1f04159910 2024-09-13T16:27:20.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f04155f40 con 0x7f1f04159910 2024-09-13T16:27:20.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f017fa700 1 -- 172.21.15.119:0/606616023 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v50) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1eec054c00 con 0x7f1f04159910 2024-09-13T16:27:20.985 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:27:20.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ee8066730 msgr2=0x7f1ee8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ee8066730 0x7f1ee8068bf0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f1ef40018a0 tx=0x7f1ef4009040 comp rx=0 tx=0).stop 2024-09-13T16:27:20.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 msgr2=0x7f1f0415a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:20.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f0415a6e0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f1eec00d800 tx=0x7f1eec00db10 comp rx=0 tx=0).stop 2024-09-13T16:27:20.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 shutdown_connections 2024-09-13T16:27:20.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ee8066730 0x7f1ee8068bf0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f0415d1c0 0x7f1f0414a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f04159910 0x7f1f0415a6e0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 --2- 172.21.15.119:0/606616023 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f04159530 0x7f1f0413e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:20.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.979+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 >> 172.21.15.119:0/606616023 conn(0x7f1f0406b460 msgr2=0x7f1f04156b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:20.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.983+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 shutdown_connections 2024-09-13T16:27:20.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:20.983+0000 7f1f09625700 1 -- 172.21.15.119:0/606616023 wait complete. 2024-09-13T16:27:21.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:27:21.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 Processor -- start 2024-09-13T16:27:21.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- start start 2024-09-13T16:27:21.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe29c15e840 0x7fe29c070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c15ec20 0x7fe29c161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c071210 con 0x7fe29c164a70 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe29c0714f0 con 0x7fe29c15e840 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c0717d0 con 0x7fe29c15ec20 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a22ce700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe29c164a70 0x7fe29c0ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46132/0 (socket says 172.21.15.119:46132) 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a22ce700 1 -- 172.21.15.119:0/2516880204 learned_addr learned my addr 172.21.15.119:0/2516880204 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a1acd700 1 --2- 172.21.15.119:0/2516880204 >> v2:172.21.15.145:3300/0 conn(0x7fe29c15e840 0x7fe29c070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a12cc700 1 --2- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c15ec20 0x7fe29c161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a0acb700 1 -- 172.21.15.119:0/2516880204 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957595830 0 0) 0x7fe29c071210 con 0x7fe29c164a70 2024-09-13T16:27:21.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a0acb700 1 -- 172.21.15.119:0/2516880204 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2840035d0 con 0x7fe29c164a70 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a1acd700 1 -- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c15ec20 msgr2=0x7fe29c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a1acd700 1 --2- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c15ec20 0x7fe29c161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a1acd700 1 -- 172.21.15.119:0/2516880204 >> v1:172.21.15.119:6789/0 conn(0x7fe29c164a70 legacy=0x7fe29c0ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a1acd700 1 -- 172.21.15.119:0/2516880204 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c071ab0 con 0x7fe29c15e840 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a12cc700 1 --2- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c15ec20 0x7fe29c161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a1acd700 1 --2- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c15e840 0x7fe29c070b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe290009850 tx=0x7fe290009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=741a0ecee51df42c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a0acb700 1 -- 172.21.15.119:0/2516880204 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290011830 con 0x7fe29c15e840 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a0acb700 1 -- 172.21.15.119:0/2516880204 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe290011990 con 0x7fe29c15e840 2024-09-13T16:27:21.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a0acb700 1 -- 172.21.15.119:0/2516880204 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290011d00 con 0x7fe29c15e840 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c15e840 msgr2=0x7fe29c070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c15e840 0x7fe29c070b80 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fe290009850 tx=0x7fe290009b60 comp rx=0 tx=0).stop 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- 172.21.15.119:0/2516880204 shutdown_connections 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c15ec20 0x7fe29c161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/2516880204 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c15e840 0x7fe29c070b80 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- 172.21.15.119:0/2516880204 >> 172.21.15.119:0/2516880204 conn(0x7fe29c06a350 msgr2=0x7fe29c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.107+0000 7fe2a3d31700 1 -- 172.21.15.119:0/2516880204 shutdown_connections 2024-09-13T16:27:21.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 -- 172.21.15.119:0/2516880204 wait complete. 2024-09-13T16:27:21.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 Processor -- start 2024-09-13T16:27:21.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 -- start start 2024-09-13T16:27:21.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe29c15e840 0x7fe29c1f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c164a70 0x7fe29c15fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c1f95b0 0x7fe29c0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c160120 con 0x7fe29c15e840 2024-09-13T16:27:21.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c160400 con 0x7fe29c1f95b0 2024-09-13T16:27:21.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe29c1606e0 con 0x7fe29c164a70 2024-09-13T16:27:21.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe29c15e840 0x7fe29c1f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe29c15e840 0x7fe29c1f72d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36132/0 (socket says 172.21.15.119:36132) 2024-09-13T16:27:21.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 -- 172.21.15.119:0/1835635798 learned_addr learned my addr 172.21.15.119:0/1835635798 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:21.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a22ce700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c1f95b0 0x7fe29c0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a12cc700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c164a70 0x7fe29c15fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 -- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c164a70 msgr2=0x7fe29c15fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c164a70 0x7fe29c15fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 -- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c1f95b0 msgr2=0x7fe29c0a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c1f95b0 0x7fe29c0a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 -- 172.21.15.119:0/1835635798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe294009a70 con 0x7fe29c15e840 2024-09-13T16:27:21.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a1acd700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe29c15e840 0x7fe29c1f72d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe290009ba0 tx=0x7fe290004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:21.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe29a7fc700 1 -- 172.21.15.119:0/1835635798 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290004610 con 0x7fe29c15e840 2024-09-13T16:27:21.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe29a7fc700 1 -- 172.21.15.119:0/1835635798 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe29001e070 con 0x7fe29c15e840 2024-09-13T16:27:21.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2900094b0 con 0x7fe29c15e840 2024-09-13T16:27:21.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe29a7fc700 1 -- 172.21.15.119:0/1835635798 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe290003900 con 0x7fe29c15e840 2024-09-13T16:27:21.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.111+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c160ca0 con 0x7fe29c15e840 2024-09-13T16:27:21.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.115+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c160ec0 con 0x7fe29c15e840 2024-09-13T16:27:21.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.115+0000 7fe29a7fc700 1 -- 172.21.15.119:0/1835635798 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe290003a60 con 0x7fe29c15e840 2024-09-13T16:27:21.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.115+0000 7fe29a7fc700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe288066730 0x7fe288068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.115+0000 7fe29a7fc700 1 -- 172.21.15.119:0/1835635798 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe290084d00 con 0x7fe29c15e840 2024-09-13T16:27:21.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.115+0000 7fe2a12cc700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe288066730 0x7fe288068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.115+0000 7fe2a12cc700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe288066730 0x7fe288068bf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe29c0b2880 tx=0x7fe29400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.119+0000 7fe29a7fc700 1 -- 172.21.15.119:0/1835635798 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe290052310 con 0x7fe29c15e840 2024-09-13T16:27:21.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.291+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe29c067760 con 0x7fe29c15e840 2024-09-13T16:27:21.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.291+0000 7fe29a7fc700 1 -- 172.21.15.119:0/1835635798 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v50) v1 ==== 84+0+65 (secure 0 0 0) 0x7fe290020090 con 0x7fe29c15e840 2024-09-13T16:27:21.299 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:21.299 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:27:21.299 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:21.299 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:27:21.299 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:27:21.299 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:21.300 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:27:21.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe288066730 msgr2=0x7fe288068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe288066730 0x7fe288068bf0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe29c0b2880 tx=0x7fe29400b040 comp rx=0 tx=0).stop 2024-09-13T16:27:21.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe29c15e840 msgr2=0x7fe29c1f72d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe29c15e840 0x7fe29c1f72d0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fe290009ba0 tx=0x7fe290004770 comp rx=0 tx=0).stop 2024-09-13T16:27:21.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 shutdown_connections 2024-09-13T16:27:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe288066730 0x7fe288068bf0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe29c1f95b0 0x7fe29c0a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe29c164a70 0x7fe29c15fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 --2- 172.21.15.119:0/1835635798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe29c15e840 0x7fe29c1f72d0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 >> 172.21.15.119:0/1835635798 conn(0x7fe29c06a350 msgr2=0x7fe29c066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 shutdown_connections 2024-09-13T16:27:21.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.295+0000 7fe2a3d31700 1 -- 172.21.15.119:0/1835635798 wait complete. 2024-09-13T16:27:21.313 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:27:21.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:27:21.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc06a1f0700 1 Processor -- start 2024-09-13T16:27:21.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc06a1f0700 1 -- start start 2024-09-13T16:27:21.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc06a1f0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc0640cd800 0x7fc0640d1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc06a1f0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0640ceaf0 con 0x7fc0640d11b0 2024-09-13T16:27:21.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc06a1f0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0640cdc70 con 0x7fc0640b26b0 2024-09-13T16:27:21.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc06a1f0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0640d1ad0 con 0x7fc0640cd800 2024-09-13T16:27:21.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0637fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc0640d11b0 0x7fc0640d0960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46146/0 (socket says 172.21.15.119:46146) 2024-09-13T16:27:21.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0637fe700 1 -- 172.21.15.119:0/3290924771 learned_addr learned my addr 172.21.15.119:0/3290924771 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:21.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc062ffd700 1 --2- 172.21.15.119:0/3290924771 >> v2:172.21.15.145:3300/0 conn(0x7fc0640cd800 0x7fc0640d1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015495946 0 0) 0x7fc0640ceaf0 con 0x7fc0640d11b0 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc04c0035d0 con 0x7fc0640d11b0 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3894469686 0 0) 0x7fc04c0035d0 con 0x7fc0640d11b0 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 >> v2:172.21.15.145:3300/0 conn(0x7fc0640cd800 msgr2=0x7fc0640d1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 --2- 172.21.15.119:0/3290924771 >> v2:172.21.15.145:3300/0 conn(0x7fc0640cd800 0x7fc0640d1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 >> v1:172.21.15.145:6789/0 conn(0x7fc0640b26b0 legacy=0x7fc0640b2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0640cdee0 con 0x7fc0640d11b0 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fc058001670 con 0x7fc0640d11b0 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 >> v1:172.21.15.119:6789/0 conn(0x7fc0640d11b0 legacy=0x7fc0640d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc04c003a20 0x7fc04c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c004870 0x7fc04c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc04c005230 0x7fc04c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640d1ad0 con 0x7fc04c003a20 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c00c0d0 con 0x7fc04c004870 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc04c00c290 con 0x7fc04c005230 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.423+0000 7fc0637fe700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc04c003a20 0x7fc04c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc062ffd700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c004870 0x7fc04c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc063fff700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc04c005230 0x7fc04c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0637fe700 1 -- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc04c005230 msgr2=0x7fc04c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0637fe700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc04c005230 0x7fc04c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0637fe700 1 -- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c004870 msgr2=0x7fc04c004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0637fe700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c004870 0x7fc04c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc063fff700 1 -- 172.21.15.119:0/3290924771 reap_dead start 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0637fe700 1 -- 172.21.15.119:0/3290924771 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc04c0035d0 con 0x7fc04c003a20 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc062ffd700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c004870 0x7fc04c004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0637fe700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc04c003a20 0x7fc04c003e40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc05800c320 tx=0x7fc04c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8be4cb200416724a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:21.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058007c30 con 0x7fc04c003a20 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05800ce60 con 0x7fc04c003a20 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0627fc700 1 -- 172.21.15.119:0/3290924771 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc058005bc0 con 0x7fc04c003a20 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- 172.21.15.119:0/3290924771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc04c003a20 msgr2=0x7fc04c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc04c003a20 0x7fc04c003e40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fc05800c320 tx=0x7fc04c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- 172.21.15.119:0/3290924771 shutdown_connections 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/3290924771 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc04c003a20 0x7fc04c003e40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- 172.21.15.119:0/3290924771 >> 172.21.15.119:0/3290924771 conn(0x7fc06408c160 msgr2=0x7fc0640d2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- 172.21.15.119:0/3290924771 shutdown_connections 2024-09-13T16:27:21.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- 172.21.15.119:0/3290924771 wait complete. 2024-09-13T16:27:21.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 Processor -- start 2024-09-13T16:27:21.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- start start 2024-09-13T16:27:21.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c003a20 0x7fc06415d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc06415dbd0 0x7fc06414cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc06415f670 0x7fc06415faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc06415d7c0 con 0x7fc06415f670 2024-09-13T16:27:21.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640b6730 con 0x7fc04c003a20 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc06a1f0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0640b6a10 con 0x7fc06415dbd0 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc063fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc06415f670 0x7fc06415faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc063fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc06415f670 0x7fc06415faf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36152/0 (socket says 172.21.15.119:36152) 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc063fff700 1 -- 172.21.15.119:0/1607072422 learned_addr learned my addr 172.21.15.119:0/1607072422 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc0637fe700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c003a20 0x7fc06415d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc062ffd700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc06415dbd0 0x7fc06414cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc063fff700 1 -- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc06415dbd0 msgr2=0x7fc06414cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.427+0000 7fc063fff700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc06415dbd0 0x7fc06414cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc063fff700 1 -- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c003a20 msgr2=0x7fc06415d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc063fff700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c003a20 0x7fc06415d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc063fff700 1 -- 172.21.15.119:0/1607072422 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0640b6cf0 con 0x7fc06415f670 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc062ffd700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc06415dbd0 0x7fc06414cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc063fff700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc06415f670 0x7fc06415faf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc06408bed0 tx=0x7fc05c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc04bfff700 1 -- 172.21.15.119:0/1607072422 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c003c30 con 0x7fc06415f670 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc04bfff700 1 -- 172.21.15.119:0/1607072422 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc05c003d90 con 0x7fc06415f670 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0640b6fd0 con 0x7fc06415f670 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc04bfff700 1 -- 172.21.15.119:0/1607072422 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc05c005310 con 0x7fc06415f670 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc06415e050 con 0x7fc06415f670 2024-09-13T16:27:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0640d1590 con 0x7fc06415f670 2024-09-13T16:27:21.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc04bfff700 1 -- 172.21.15.119:0/1607072422 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc05c005470 con 0x7fc06415f670 2024-09-13T16:27:21.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc04bfff700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc050066730 0x7fc050068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:21.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc04bfff700 1 -- 172.21.15.119:0/1607072422 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fc05c083f20 con 0x7fc06415f670 2024-09-13T16:27:21.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.431+0000 7fc0637fe700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc050066730 0x7fc050068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:21.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.435+0000 7fc0637fe700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc050066730 0x7fc050068bf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc05800d010 tx=0x7fc058007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.439+0000 7fc04bfff700 1 -- 172.21.15.119:0/1607072422 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc05c051490 con 0x7fc06415f670 2024-09-13T16:27:21.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.611+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc0640aff20 con 0x7fc06415f670 2024-09-13T16:27:21.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.963+0000 7fc04bfff700 1 -- 172.21.15.119:0/1607072422 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v51) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc05c01a310 con 0x7fc06415f670 2024-09-13T16:27:21.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc050066730 msgr2=0x7fc050068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc050066730 0x7fc050068bf0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fc05800d010 tx=0x7fc058007000 comp rx=0 tx=0).stop 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc06415f670 msgr2=0x7fc06415faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc06415f670 0x7fc06415faf0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fc06408bed0 tx=0x7fc05c00ee80 comp rx=0 tx=0).stop 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 shutdown_connections 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc050066730 0x7fc050068bf0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc06415f670 0x7fc06415faf0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc06415dbd0 0x7fc06414cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 --2- 172.21.15.119:0/1607072422 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc04c003a20 0x7fc06415d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 >> 172.21.15.119:0/1607072422 conn(0x7fc06408c160 msgr2=0x7fc0640b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 shutdown_connections 2024-09-13T16:27:21.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:21.971+0000 7fc06a1f0700 1 -- 172.21.15.119:0/1607072422 wait complete. 2024-09-13T16:27:21.980 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:27:21.991 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:27:22.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.107+0000 7f62cb358700 1 Processor -- start 2024-09-13T16:27:22.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.107+0000 7f62cb358700 1 -- start start 2024-09-13T16:27:22.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.107+0000 7f62cb358700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f62c4159870 0x7f62c406dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:22.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.107+0000 7f62cb358700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62c40713b0 0x7f62c40cda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:22.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.107+0000 7f62cb358700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c406e500 con 0x7f62c40ae9c0 2024-09-13T16:27:22.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.107+0000 7f62cb358700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c40af280 con 0x7f62c40713b0 2024-09-13T16:27:22.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.107+0000 7f62cb358700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c40b3840 con 0x7f62c4159870 2024-09-13T16:27:22.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cab57700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62c40713b0 0x7f62c40cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:22.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62c9b55700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f62c40ae9c0 0x7f62c4070ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:46156/0 (socket says 172.21.15.119:46156) 2024-09-13T16:27:22.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62c9b55700 1 -- 172.21.15.119:0/4003683972 learned_addr learned my addr 172.21.15.119:0/4003683972 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:22.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62ca356700 1 --2- 172.21.15.119:0/4003683972 >> v2:172.21.15.145:3300/0 conn(0x7f62c4159870 0x7f62c406dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cab57700 1 -- 172.21.15.119:0/4003683972 >> v2:172.21.15.145:3300/0 conn(0x7f62c4159870 msgr2=0x7f62c406dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cab57700 1 --2- 172.21.15.119:0/4003683972 >> v2:172.21.15.145:3300/0 conn(0x7f62c4159870 0x7f62c406dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cab57700 1 -- 172.21.15.119:0/4003683972 >> v1:172.21.15.119:6789/0 conn(0x7f62c40ae9c0 legacy=0x7f62c4070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62c9354700 1 -- 172.21.15.119:0/4003683972 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 421521455 0 0) 0x7f62c406e500 con 0x7f62c40ae9c0 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cab57700 1 -- 172.21.15.119:0/4003683972 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c40b2a80 con 0x7f62c40713b0 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cab57700 1 --2- 172.21.15.119:0/4003683972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c40cda30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009dc0 tx=0x7f62bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfd2817d6b201cda server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62c9354700 1 -- 172.21.15.119:0/4003683972 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc00e070 con 0x7f62c40713b0 2024-09-13T16:27:22.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62c9354700 1 -- 172.21.15.119:0/4003683972 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc00baa0 con 0x7f62c40713b0 2024-09-13T16:27:22.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62c9354700 1 -- 172.21.15.119:0/4003683972 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62bc00be50 con 0x7f62c40713b0 2024-09-13T16:27:22.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 -- 172.21.15.119:0/4003683972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 msgr2=0x7f62c40cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:22.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 --2- 172.21.15.119:0/4003683972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c40cda30 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f62bc009dc0 tx=0x7f62bc00b330 comp rx=0 tx=0).stop 2024-09-13T16:27:22.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 -- 172.21.15.119:0/4003683972 shutdown_connections 2024-09-13T16:27:22.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 --2- 172.21.15.119:0/4003683972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c40cda30 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:22.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 --2- 172.21.15.119:0/4003683972 >> v2:172.21.15.145:3300/0 conn(0x7f62c4159870 0x7f62c406dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:22.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 -- 172.21.15.119:0/4003683972 >> 172.21.15.119:0/4003683972 conn(0x7f62c40740e0 msgr2=0x7f62c406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:22.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 -- 172.21.15.119:0/4003683972 shutdown_connections 2024-09-13T16:27:22.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.111+0000 7f62cb358700 1 -- 172.21.15.119:0/4003683972 wait complete. 2024-09-13T16:27:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 Processor -- start 2024-09-13T16:27:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 -- start start 2024-09-13T16:27:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c41521e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:22.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62c40ae9c0 0x7f62c4156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62c4159870 0x7f62c413d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4152b30 con 0x7f62c40713b0 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4152e10 con 0x7f62c40ae9c0 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c41530f0 con 0x7f62c4159870 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c41521e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c41521e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36166/0 (socket says 172.21.15.119:36166) 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 -- 172.21.15.119:0/3512250219 learned_addr learned my addr 172.21.15.119:0/3512250219 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62c9b55700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62c40ae9c0 0x7f62c4156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 -- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62c4159870 msgr2=0x7f62c413d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cab57700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62c4159870 0x7f62c413d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:22.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62c4159870 0x7f62c413d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 -- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62c40ae9c0 msgr2=0x7f62c4156e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62c40ae9c0 0x7f62c4156e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 -- 172.21.15.119:0/3512250219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc009a70 con 0x7f62c40713b0 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62c9b55700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62c40ae9c0 0x7f62c4156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cab57700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62c4159870 0x7f62c413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62ca356700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c41521e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f62c000e840 tx=0x7f62c000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62b6ffd700 1 -- 172.21.15.119:0/3512250219 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c00095a0 con 0x7f62c40713b0 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62b6ffd700 1 -- 172.21.15.119:0/3512250219 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62c000c690 con 0x7f62c40713b0 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62b6ffd700 1 -- 172.21.15.119:0/3512250219 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c000ca00 con 0x7f62c40713b0 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c41533d0 con 0x7f62c40713b0 2024-09-13T16:27:22.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.115+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c4152720 con 0x7f62c40713b0 2024-09-13T16:27:22.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.119+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c41538f0 con 0x7f62c40713b0 2024-09-13T16:27:22.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.119+0000 7f62b6ffd700 1 -- 172.21.15.119:0/3512250219 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f62c000cb60 con 0x7f62c40713b0 2024-09-13T16:27:22.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.119+0000 7f62b6ffd700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f62a8066730 0x7f62a8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:22.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.119+0000 7f62b6ffd700 1 -- 172.21.15.119:0/3512250219 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f62c0084080 con 0x7f62c40713b0 2024-09-13T16:27:22.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.119+0000 7f62c9b55700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f62a8066730 0x7f62a8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:22.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.119+0000 7f62c9b55700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f62a8066730 0x7f62a8068bf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f62c41565f0 tx=0x7f62b8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:22.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.131+0000 7f62b6ffd700 1 -- 172.21.15.119:0/3512250219 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f62c0051520 con 0x7f62c40713b0 2024-09-13T16:27:22.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.295+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f62c40cda30 con 0x7f62c40713b0 2024-09-13T16:27:22.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.975+0000 7f62b6ffd700 1 -- 172.21.15.119:0/3512250219 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v52) v1 ==== 170+0+0 (secure 0 0 0) 0x7f62c0054b40 con 0x7f62c40713b0 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f62a8066730 msgr2=0x7f62a8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f62a8066730 0x7f62a8068bf0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f62c41565f0 tx=0x7f62b8009040 comp rx=0 tx=0).stop 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 msgr2=0x7f62c41521e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c41521e0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f62c000e840 tx=0x7f62c000ec00 comp rx=0 tx=0).stop 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 shutdown_connections 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f62a8066730 0x7f62a8068bf0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62c4159870 0x7f62c413d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62c40ae9c0 0x7f62c4156e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 --2- 172.21.15.119:0/3512250219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62c40713b0 0x7f62c41521e0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 >> 172.21.15.119:0/3512250219 conn(0x7f62c40740e0 msgr2=0x7f62c4066450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:23.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 shutdown_connections 2024-09-13T16:27:23.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:22.999+0000 7f62cb358700 1 -- 172.21.15.119:0/3512250219 wait complete. 2024-09-13T16:27:23.006 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:27:23.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:27:23.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 Processor -- start 2024-09-13T16:27:23.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 -- start start 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa90c15e430 0x7fa90c0cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c15e810 0x7fa90c06fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa90c0701f0 0x7fa90c0706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90c0b2970 con 0x7fa90c0701f0 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa90c0cb750 con 0x7fa90c15e430 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.119+0000 7fa912970700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0ccb30 con 0x7fa90c15e810 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa90c0701f0 0x7fa90c0706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa90c0701f0 0x7fa90c0706a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39566/0 (socket says 172.21.15.119:39566) 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 -- 172.21.15.119:0/377867661 learned_addr learned my addr 172.21.15.119:0/377867661 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa90bfff700 1 --2- 172.21.15.119:0/377867661 >> v2:172.21.15.145:3300/0 conn(0x7fa90c15e430 0x7fa90c0cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa90b7fe700 1 --2- 172.21.15.119:0/377867661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c15e810 0x7fa90c06fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 -- 172.21.15.119:0/377867661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c15e810 msgr2=0x7fa90c06fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 --2- 172.21.15.119:0/377867661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c15e810 0x7fa90c06fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 -- 172.21.15.119:0/377867661 >> v2:172.21.15.145:3300/0 conn(0x7fa90c15e430 msgr2=0x7fa90c0cb210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 --2- 172.21.15.119:0/377867661 >> v2:172.21.15.145:3300/0 conn(0x7fa90c15e430 0x7fa90c0cb210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 -- 172.21.15.119:0/377867661 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c0b2690 con 0x7fa90c0701f0 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa90b7fe700 1 --2- 172.21.15.119:0/377867661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c15e810 0x7fa90c06fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa910f0d700 1 --2- 172.21.15.119:0/377867661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c0701f0 0x7fa90c0706a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0cf770 tx=0x7fa8fc009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeefa9cbb52cf7f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa90affd700 1 -- 172.21.15.119:0/377867661 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc00b8a0 con 0x7fa90c0701f0 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa90affd700 1 -- 172.21.15.119:0/377867661 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8fc00ba00 con 0x7fa90c0701f0 2024-09-13T16:27:23.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa90affd700 1 -- 172.21.15.119:0/377867661 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc00bd90 con 0x7fa90c0701f0 2024-09-13T16:27:23.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- 172.21.15.119:0/377867661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c0701f0 msgr2=0x7fa90c0706a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 --2- 172.21.15.119:0/377867661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c0701f0 0x7fa90c0706a0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fa90c0cf770 tx=0x7fa8fc009bf0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- 172.21.15.119:0/377867661 shutdown_connections 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 --2- 172.21.15.119:0/377867661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c0701f0 0x7fa90c0706a0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 --2- 172.21.15.119:0/377867661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c15e810 0x7fa90c06fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 --2- 172.21.15.119:0/377867661 >> v2:172.21.15.145:3300/0 conn(0x7fa90c15e430 0x7fa90c0cb210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- 172.21.15.119:0/377867661 >> 172.21.15.119:0/377867661 conn(0x7fa90c0b0560 msgr2=0x7fa90c0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- 172.21.15.119:0/377867661 shutdown_connections 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- 172.21.15.119:0/377867661 wait complete. 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 Processor -- start 2024-09-13T16:27:23.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- start start 2024-09-13T16:27:23.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c0701f0 0x7fa90c156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa90c15e430 0x7fa90c1effa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c15e810 0x7fa90c1f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c1f35d0 con 0x7fa90c15e810 2024-09-13T16:27:23.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c0ad450 con 0x7fa90c15e430 2024-09-13T16:27:23.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.123+0000 7fa912970700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa90c1568c0 con 0x7fa90c0701f0 2024-09-13T16:27:23.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c15e810 0x7fa90c1f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c15e810 0x7fa90c1f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39578/0 (socket says 172.21.15.119:39578) 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 -- 172.21.15.119:0/913389332 learned_addr learned my addr 172.21.15.119:0/913389332 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa90b7fe700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa90c15e430 0x7fa90c1effa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa90bfff700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c0701f0 0x7fa90c156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 -- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c0701f0 msgr2=0x7fa90c156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c0701f0 0x7fa90c156380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 -- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa90c15e430 msgr2=0x7fa90c1effa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa90c15e430 0x7fa90c1effa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 -- 172.21.15.119:0/913389332 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc009570 con 0x7fa90c15e810 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa90bfff700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c0701f0 0x7fa90c156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa910f0d700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c15e810 0x7fa90c1f3d40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00b0e0 tx=0x7fa8fc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa908ff9700 1 -- 172.21.15.119:0/913389332 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc0036c0 con 0x7fa90c15e810 2024-09-13T16:27:23.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa912970700 1 -- 172.21.15.119:0/913389332 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa90c1f07c0 con 0x7fa90c15e810 2024-09-13T16:27:23.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa908ff9700 1 -- 172.21.15.119:0/913389332 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8fc01f070 con 0x7fa90c15e810 2024-09-13T16:27:23.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa908ff9700 1 -- 172.21.15.119:0/913389332 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc003a60 con 0x7fa90c15e810 2024-09-13T16:27:23.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa912970700 1 -- 172.21.15.119:0/913389332 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa90c1f0d80 con 0x7fa90c15e810 2024-09-13T16:27:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.127+0000 7fa912970700 1 -- 172.21.15.119:0/913389332 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa90c073590 con 0x7fa90c15e810 2024-09-13T16:27:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.131+0000 7fa908ff9700 1 -- 172.21.15.119:0/913389332 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa8fc021030 con 0x7fa90c15e810 2024-09-13T16:27:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.131+0000 7fa908ff9700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa8ec066730 0x7fa8ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.131+0000 7fa908ff9700 1 -- 172.21.15.119:0/913389332 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fa8fc085c40 con 0x7fa90c15e810 2024-09-13T16:27:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.131+0000 7fa90bfff700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa8ec066730 0x7fa8ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.131+0000 7fa90bfff700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa8ec066730 0x7fa8ec068bf0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa904009f70 tx=0x7fa90400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.135+0000 7fa908ff9700 1 -- 172.21.15.119:0/913389332 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa8fc052f60 con 0x7fa90c15e810 2024-09-13T16:27:23.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.339+0000 7fa912970700 1 -- 172.21.15.119:0/913389332 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fa90c15ec30 con 0x7fa90c15e810 2024-09-13T16:27:23.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.371+0000 7fa908ff9700 1 -- 172.21.15.119:0/913389332 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7fa8fc015020 con 0x7fa90c15e810 2024-09-13T16:27:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 -- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa8ec066730 msgr2=0x7fa8ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa8ec066730 0x7fa8ec068bf0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa904009f70 tx=0x7fa90400b040 comp rx=0 tx=0).stop 2024-09-13T16:27:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 -- 172.21.15.119:0/913389332 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c15e810 msgr2=0x7fa90c1f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c15e810 0x7fa90c1f3d40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fa8fc00b0e0 tx=0x7fa8fc004770 comp rx=0 tx=0).stop 2024-09-13T16:27:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 -- 172.21.15.119:0/913389332 shutdown_connections 2024-09-13T16:27:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa8ec066730 0x7fa8ec068bf0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa90c15e810 0x7fa90c1f3d40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa90c15e430 0x7fa90c1effa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 --2- 172.21.15.119:0/913389332 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa90c0701f0 0x7fa90c156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 -- 172.21.15.119:0/913389332 >> 172.21.15.119:0/913389332 conn(0x7fa90c0b0560 msgr2=0x7fa90c0ae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 -- 172.21.15.119:0/913389332 shutdown_connections 2024-09-13T16:27:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.383+0000 7fa9027fc700 1 -- 172.21.15.119:0/913389332 wait complete. 2024-09-13T16:27:23.393 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 7 and data pool 8 2024-09-13T16:27:23.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:23.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 Processor -- start 2024-09-13T16:27:23.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 -- start start 2024-09-13T16:27:23.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04f415b4f0 0x7f04f415c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4070e50 0x7f04f415b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04f406a350 0x7f04f406a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f406ad20 con 0x7f04f415b4f0 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f40cdbf0 con 0x7f04f406a350 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04fb343700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f4071e20 con 0x7f04f4070e50 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04f415b4f0 0x7f04f415c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04f415b4f0 0x7f04f415c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39594/0 (socket says 172.21.15.119:39594) 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 -- 172.21.15.119:0/4231614386 learned_addr learned my addr 172.21.15.119:0/4231614386 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f98e0700 1 --2- 172.21.15.119:0/4231614386 >> v2:172.21.15.145:3300/0 conn(0x7f04f406a350 0x7f04f406a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f88de700 1 --2- 172.21.15.119:0/4231614386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4070e50 0x7f04f415b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 -- 172.21.15.119:0/4231614386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4070e50 msgr2=0x7f04f415b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:23.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 --2- 172.21.15.119:0/4231614386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4070e50 0x7f04f415b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 -- 172.21.15.119:0/4231614386 >> v2:172.21.15.145:3300/0 conn(0x7f04f406a350 msgr2=0x7f04f406a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 --2- 172.21.15.119:0/4231614386 >> v2:172.21.15.145:3300/0 conn(0x7f04f406a350 0x7f04f406a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f90df700 1 -- 172.21.15.119:0/4231614386 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40712b0 con 0x7f04f415b4f0 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.499+0000 7f04f98e0700 1 --2- 172.21.15.119:0/4231614386 >> v2:172.21.15.145:3300/0 conn(0x7f04f406a350 0x7f04f406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04f90df700 1 --2- 172.21.15.119:0/4231614386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f415b4f0 0x7f04f415c180 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f04f40676a0 tx=0x7f04f0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8218cc3c4bb00e55 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04ebfff700 1 -- 172.21.15.119:0/4231614386 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f000b8a0 con 0x7f04f415b4f0 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04ebfff700 1 -- 172.21.15.119:0/4231614386 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f000ba00 con 0x7f04f415b4f0 2024-09-13T16:27:23.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04ebfff700 1 -- 172.21.15.119:0/4231614386 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f000bd90 con 0x7f04f415b4f0 2024-09-13T16:27:23.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 -- 172.21.15.119:0/4231614386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f415b4f0 msgr2=0x7f04f415c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 --2- 172.21.15.119:0/4231614386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f415b4f0 0x7f04f415c180 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f04f40676a0 tx=0x7f04f0009bf0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 -- 172.21.15.119:0/4231614386 shutdown_connections 2024-09-13T16:27:23.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 --2- 172.21.15.119:0/4231614386 >> v2:172.21.15.145:3300/0 conn(0x7f04f406a350 0x7f04f406a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 --2- 172.21.15.119:0/4231614386 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4070e50 0x7f04f415b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 --2- 172.21.15.119:0/4231614386 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f415b4f0 0x7f04f415c180 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 -- 172.21.15.119:0/4231614386 >> 172.21.15.119:0/4231614386 conn(0x7f04f40b1650 msgr2=0x7f04f40b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:23.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 -- 172.21.15.119:0/4231614386 shutdown_connections 2024-09-13T16:27:23.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 -- 172.21.15.119:0/4231614386 wait complete. 2024-09-13T16:27:23.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 Processor -- start 2024-09-13T16:27:23.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 -- start start 2024-09-13T16:27:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f406a350 0x7f04f414a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.503+0000 7f04fb343700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f4070e50 0x7f04f4152920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04fb343700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f415b4f0 0x7f04f4152e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04fb343700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f414a940 con 0x7f04f406a350 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04fb343700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f406cfb0 con 0x7f04f4070e50 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04fb343700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f406d290 con 0x7f04f415b4f0 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f406a350 0x7f04f414a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f406a350 0x7f04f414a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39610/0 (socket says 172.21.15.119:39610) 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 -- 172.21.15.119:0/1059658686 learned_addr learned my addr 172.21.15.119:0/1059658686 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f88de700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f4070e50 0x7f04f4152920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f98e0700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f415b4f0 0x7f04f4152e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 -- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f415b4f0 msgr2=0x7f04f4152e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f415b4f0 0x7f04f4152e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 -- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f4070e50 msgr2=0x7f04f4152920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f4070e50 0x7f04f4152920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 -- 172.21.15.119:0/1059658686 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f0009570 con 0x7f04f406a350 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f88de700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f4070e50 0x7f04f4152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f98e0700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f415b4f0 0x7f04f4152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04f90df700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f406a350 0x7f04f414a400 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f04f0000c00 tx=0x7f04f0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04e9ffb700 1 -- 172.21.15.119:0/1059658686 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f00038c0 con 0x7f04f406a350 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04e9ffb700 1 -- 172.21.15.119:0/1059658686 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f001e070 con 0x7f04f406a350 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f406d570 con 0x7f04f406a350 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04e9ffb700 1 -- 172.21.15.119:0/1059658686 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0003c40 con 0x7f04f406a350 2024-09-13T16:27:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f406db30 con 0x7f04f406a350 2024-09-13T16:27:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f4067a40 con 0x7f04f406a350 2024-09-13T16:27:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04e9ffb700 1 -- 172.21.15.119:0/1059658686 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04f0011420 con 0x7f04f406a350 2024-09-13T16:27:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04e9ffb700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04d8066730 0x7f04d8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.507+0000 7f04e9ffb700 1 -- 172.21.15.119:0/1059658686 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f04f0020070 con 0x7f04f406a350 2024-09-13T16:27:23.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.511+0000 7f04f88de700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04d8066730 0x7f04d8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.511+0000 7f04f88de700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04d8066730 0x7f04d8068bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f04f4155130 tx=0x7f04e4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.515+0000 7f04e9ffb700 1 -- 172.21.15.119:0/1059658686 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f04f0052d60 con 0x7f04f406a350 2024-09-13T16:27:23.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.691+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04f406a7e0 con 0x7f04f406a350 2024-09-13T16:27:23.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.691+0000 7f04e9ffb700 1 -- 172.21.15.119:0/1059658686 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f04f0056380 con 0x7f04f406a350 2024-09-13T16:27:23.699 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:23.699 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":53,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:23.347929+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:27:21.660330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-09-13T16:27:22.341239+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04d8066730 msgr2=0x7f04d8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04d8066730 0x7f04d8068bf0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f04f4155130 tx=0x7f04e4008040 comp rx=0 tx=0).stop 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f406a350 msgr2=0x7f04f414a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f406a350 0x7f04f414a400 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f04f0000c00 tx=0x7f04f0004770 comp rx=0 tx=0).stop 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 shutdown_connections 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04d8066730 0x7f04d8068bf0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f415b4f0 0x7f04f4152e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f4070e50 0x7f04f4152920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 --2- 172.21.15.119:0/1059658686 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f406a350 0x7f04f414a400 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 >> 172.21.15.119:0/1059658686 conn(0x7f04f40b1650 msgr2=0x7f04f4154020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 shutdown_connections 2024-09-13T16:27:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.695+0000 7f04fb343700 1 -- 172.21.15.119:0/1059658686 wait complete. 2024-09-13T16:27:23.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:23.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 Processor -- start 2024-09-13T16:27:23.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- start start 2024-09-13T16:27:23.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f128815a730 0x7f128815b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f12880afa80 0x7f12880b4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f128815bec0 con 0x7f128815ab10 2024-09-13T16:27:23.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12880adf10 con 0x7f128815a730 2024-09-13T16:27:23.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12880ae1b0 con 0x7f12880afa80 2024-09-13T16:27:23.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128ddbe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f128815a730 0x7f128815b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128d5bd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f128815ab10 0x7f12880af250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54036/0 (socket says 172.21.15.119:54036) 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128d5bd700 1 -- 172.21.15.119:0/1202866924 learned_addr learned my addr 172.21.15.119:0/1202866924 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128e5bf700 1 --2- 172.21.15.119:0/1202866924 >> v2:172.21.15.145:3300/0 conn(0x7f12880afa80 0x7f12880b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128ddbe700 1 -- 172.21.15.119:0/1202866924 >> v2:172.21.15.145:3300/0 conn(0x7f12880afa80 msgr2=0x7f12880b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128ddbe700 1 --2- 172.21.15.119:0/1202866924 >> v2:172.21.15.145:3300/0 conn(0x7f12880afa80 0x7f12880b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128ddbe700 1 -- 172.21.15.119:0/1202866924 >> v1:172.21.15.119:6789/0 conn(0x7f128815ab10 legacy=0x7f12880af250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128ddbe700 1 -- 172.21.15.119:0/1202866924 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12880ae490 con 0x7f128815a730 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128cdbc700 1 -- 172.21.15.119:0/1202866924 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3451368771 0 0) 0x7f128815bec0 con 0x7f128815ab10 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128e5bf700 1 --2- 172.21.15.119:0/1202866924 >> v2:172.21.15.145:3300/0 conn(0x7f12880afa80 0x7f12880b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128ddbe700 1 --2- 172.21.15.119:0/1202866924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f128815a730 0x7f128815b980 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1288071cb0 tx=0x7f1284009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0b0689f588a9f09 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128cdbc700 1 -- 172.21.15.119:0/1202866924 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284011830 con 0x7f128815a730 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128cdbc700 1 -- 172.21.15.119:0/1202866924 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1284011990 con 0x7f128815a730 2024-09-13T16:27:23.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128cdbc700 1 -- 172.21.15.119:0/1202866924 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1284011d40 con 0x7f128815a730 2024-09-13T16:27:23.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- 172.21.15.119:0/1202866924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f128815a730 msgr2=0x7f128815b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 --2- 172.21.15.119:0/1202866924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f128815a730 0x7f128815b980 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f1288071cb0 tx=0x7f1284009b30 comp rx=0 tx=0).stop 2024-09-13T16:27:23.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- 172.21.15.119:0/1202866924 shutdown_connections 2024-09-13T16:27:23.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 --2- 172.21.15.119:0/1202866924 >> v2:172.21.15.145:3300/0 conn(0x7f12880afa80 0x7f12880b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 --2- 172.21.15.119:0/1202866924 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f128815a730 0x7f128815b980 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- 172.21.15.119:0/1202866924 >> 172.21.15.119:0/1202866924 conn(0x7f1288072260 msgr2=0x7f1288099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:23.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- 172.21.15.119:0/1202866924 shutdown_connections 2024-09-13T16:27:23.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.827+0000 7f128edc0700 1 -- 172.21.15.119:0/1202866924 wait complete. 2024-09-13T16:27:23.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 Processor -- start 2024-09-13T16:27:23.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 -- start start 2024-09-13T16:27:23.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12880afa80 0x7f1288147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f128815a730 0x7f128814eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f128815ab10 0x7f12881f3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12881475c0 con 0x7f12880afa80 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128815d740 con 0x7f128815ab10 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128814f400 con 0x7f128815a730 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12880afa80 0x7f1288147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12880afa80 0x7f1288147080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39628/0 (socket says 172.21.15.119:39628) 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 -- 172.21.15.119:0/2958225927 learned_addr learned my addr 172.21.15.119:0/2958225927 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128d5bd700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f128815a730 0x7f128814eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128e5bf700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f128815ab10 0x7f12881f3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 -- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f128815a730 msgr2=0x7f128814eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f128815a730 0x7f128814eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 -- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f128815ab10 msgr2=0x7f12881f3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f128815ab10 0x7f12881f3490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 -- 172.21.15.119:0/2958225927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12840094b0 con 0x7f12880afa80 2024-09-13T16:27:23.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128ddbe700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12880afa80 0x7f1288147080 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1284000c00 tx=0x7f1284004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f127e7fc700 1 -- 172.21.15.119:0/2958225927 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128401aac0 con 0x7f12880afa80 2024-09-13T16:27:23.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f127e7fc700 1 -- 172.21.15.119:0/2958225927 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128401f070 con 0x7f12880afa80 2024-09-13T16:27:23.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f128815cb30 con 0x7f12880afa80 2024-09-13T16:27:23.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f127e7fc700 1 -- 172.21.15.119:0/2958225927 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128401ae80 con 0x7f12880afa80 2024-09-13T16:27:23.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.831+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f128815d0f0 con 0x7f12880afa80 2024-09-13T16:27:23.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.835+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12880b4d00 con 0x7f12880afa80 2024-09-13T16:27:23.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.835+0000 7f127e7fc700 1 -- 172.21.15.119:0/2958225927 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f128401b430 con 0x7f12880afa80 2024-09-13T16:27:23.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.835+0000 7f127e7fc700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f126c066730 0x7f126c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:23.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.835+0000 7f127e7fc700 1 -- 172.21.15.119:0/2958225927 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1284085460 con 0x7f12880afa80 2024-09-13T16:27:23.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.835+0000 7f128d5bd700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f126c066730 0x7f126c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:23.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.839+0000 7f128d5bd700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f126c066730 0x7f126c068bf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1288156600 tx=0x7f1278007cb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:23.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:23.847+0000 7f127e7fc700 1 -- 172.21.15.119:0/2958225927 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1284052720 con 0x7f12880afa80 2024-09-13T16:27:24.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.027+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f128806fa50 con 0x7f12880afa80 2024-09-13T16:27:24.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f127e7fc700 1 -- 172.21.15.119:0/2958225927 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v53) v1 ==== 74+0+16847 (secure 0 0 0) 0x7f1284021090 con 0x7f12880afa80 2024-09-13T16:27:24.036 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:24.036 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":53,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:23.347929+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:27:21.660330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-09-13T16:27:22.341239+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f126c066730 msgr2=0x7f126c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f126c066730 0x7f126c068bf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f1288156600 tx=0x7f1278007cb0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12880afa80 msgr2=0x7f1288147080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12880afa80 0x7f1288147080 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f1284000c00 tx=0x7f1284004770 comp rx=0 tx=0).stop 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 shutdown_connections 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f126c066730 0x7f126c068bf0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f128815ab10 0x7f12881f3490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f128815a730 0x7f128814eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 --2- 172.21.15.119:0/2958225927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f12880afa80 0x7f1288147080 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 >> 172.21.15.119:0/2958225927 conn(0x7f1288072260 msgr2=0x7f12881598c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:24.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 shutdown_connections 2024-09-13T16:27:24.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.031+0000 7f128edc0700 1 -- 172.21.15.119:0/2958225927 wait complete. 2024-09-13T16:27:24.051 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:27:24.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 Processor -- start 2024-09-13T16:27:24.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- start start 2024-09-13T16:27:24.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0458144180 0x7f045806c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:24.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04580ce1d0 0x7f04580b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:24.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045806cb40 con 0x7f04580cc010 2024-09-13T16:27:24.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04580b1c50 con 0x7f0458144180 2024-09-13T16:27:24.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f045806ce20 con 0x7f04580ce1d0 2024-09-13T16:27:24.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f0455d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f04580cc010 0x7f04580cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54044/0 (socket says 172.21.15.119:54044) 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f0455d9b700 1 -- 172.21.15.119:0/1474808361 learned_addr learned my addr 172.21.15.119:0/1474808361 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045659c700 1 --2- 172.21.15.119:0/1474808361 >> v2:172.21.15.119:3300/0 conn(0x7f0458144180 0x7f045806c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f0456d9d700 1 --2- 172.21.15.119:0/1474808361 >> v2:172.21.15.145:3300/0 conn(0x7f04580ce1d0 0x7f04580b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045559a700 1 -- 172.21.15.119:0/1474808361 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901731806 0 0) 0x7f045806cb40 con 0x7f04580cc010 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045559a700 1 -- 172.21.15.119:0/1474808361 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04400035d0 con 0x7f04580cc010 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045559a700 1 -- 172.21.15.119:0/1474808361 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 772862785 0 0) 0x7f04400035d0 con 0x7f04580cc010 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045659c700 1 -- 172.21.15.119:0/1474808361 >> v2:172.21.15.145:3300/0 conn(0x7f04580ce1d0 msgr2=0x7f04580b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045659c700 1 --2- 172.21.15.119:0/1474808361 >> v2:172.21.15.145:3300/0 conn(0x7f04580ce1d0 0x7f04580b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045659c700 1 -- 172.21.15.119:0/1474808361 >> v1:172.21.15.119:6789/0 conn(0x7f04580cc010 legacy=0x7f04580cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045659c700 1 -- 172.21.15.119:0/1474808361 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f045806d100 con 0x7f0458144180 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f0456d9d700 1 --2- 172.21.15.119:0/1474808361 >> v2:172.21.15.145:3300/0 conn(0x7f04580ce1d0 0x7f04580b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:24.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045659c700 1 --2- 172.21.15.119:0/1474808361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0458144180 0x7f045806c480 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f04580b1b80 tx=0x7f044c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=679b2ad259c754d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:24.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045559a700 1 -- 172.21.15.119:0/1474808361 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448001670 con 0x7f0458144180 2024-09-13T16:27:24.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045559a700 1 -- 172.21.15.119:0/1474808361 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f044c004030 con 0x7f0458144180 2024-09-13T16:27:24.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045559a700 1 -- 172.21.15.119:0/1474808361 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f044c0043a0 con 0x7f0458144180 2024-09-13T16:27:24.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- 172.21.15.119:0/1474808361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0458144180 msgr2=0x7f045806c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:24.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 --2- 172.21.15.119:0/1474808361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0458144180 0x7f045806c480 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f04580b1b80 tx=0x7f044c009b30 comp rx=0 tx=0).stop 2024-09-13T16:27:24.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- 172.21.15.119:0/1474808361 shutdown_connections 2024-09-13T16:27:24.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 --2- 172.21.15.119:0/1474808361 >> v2:172.21.15.145:3300/0 conn(0x7f04580ce1d0 0x7f04580b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 --2- 172.21.15.119:0/1474808361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0458144180 0x7f045806c480 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- 172.21.15.119:0/1474808361 >> 172.21.15.119:0/1474808361 conn(0x7f04580b0980 msgr2=0x7f04580b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:24.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- 172.21.15.119:0/1474808361 shutdown_connections 2024-09-13T16:27:24.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.167+0000 7f045759e700 1 -- 172.21.15.119:0/1474808361 wait complete. 2024-09-13T16:27:24.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.175+0000 7f045759e700 1 Processor -- start 2024-09-13T16:27:24.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.175+0000 7f045759e700 1 -- start start 2024-09-13T16:27:24.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.175+0000 7f045759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04580cc010 0x7f045813f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04580ce1d0 0x7f04580abe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0458144180 0x7f045812fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045813fb60 con 0x7f04580ce1d0 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f045813fe40 con 0x7f04580cc010 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0458140120 con 0x7f0458144180 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04580ce1d0 0x7f04580abe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04580ce1d0 0x7f04580abe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39636/0 (socket says 172.21.15.119:39636) 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 -- 172.21.15.119:0/1216532368 learned_addr learned my addr 172.21.15.119:0/1216532368 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045659c700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04580cc010 0x7f045813f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0456d9d700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0458144180 0x7f045812fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 -- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0458144180 msgr2=0x7f045812fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0458144180 0x7f045812fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 -- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04580cc010 msgr2=0x7f045813f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04580cc010 0x7f045813f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 -- 172.21.15.119:0/1216532368 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0450009a70 con 0x7f04580ce1d0 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0455d9b700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04580ce1d0 0x7f04580abe10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f04480011e0 tx=0x7f044800ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0446ffd700 1 -- 172.21.15.119:0/1216532368 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0448004ca0 con 0x7f04580ce1d0 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0446ffd700 1 -- 172.21.15.119:0/1216532368 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0448004e00 con 0x7f04580ce1d0 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f044c0094b0 con 0x7f04580ce1d0 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0446ffd700 1 -- 172.21.15.119:0/1216532368 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04480072f0 con 0x7f04580ce1d0 2024-09-13T16:27:24.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04581406e0 con 0x7f04580ce1d0 2024-09-13T16:27:24.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0446ffd700 1 -- 172.21.15.119:0/1216532368 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0448007450 con 0x7f04580ce1d0 2024-09-13T16:27:24.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045806c480 con 0x7f04580ce1d0 2024-09-13T16:27:24.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0446ffd700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0438066730 0x7f0438068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:24.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f0446ffd700 1 -- 172.21.15.119:0/1216532368 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0448083ff0 con 0x7f04580ce1d0 2024-09-13T16:27:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045659c700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0438066730 0x7f0438068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.179+0000 7f045659c700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0438066730 0x7f0438068bf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f044c0090b0 tx=0x7f044c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:24.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.191+0000 7f0446ffd700 1 -- 172.21.15.119:0/1216532368 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0448051220 con 0x7f04580ce1d0 2024-09-13T16:27:24.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:24.391+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f04580b2eb0 con 0x7f04580ce1d0 2024-09-13T16:27:25.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.375+0000 7f0446ffd700 1 -- 172.21.15.119:0/1216532368 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7f0448054840 con 0x7f04580ce1d0 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0438066730 msgr2=0x7f0438068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0438066730 0x7f0438068bf0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f044c0090b0 tx=0x7f044c008030 comp rx=0 tx=0).stop 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04580ce1d0 msgr2=0x7f04580abe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04580ce1d0 0x7f04580abe10 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f04480011e0 tx=0x7f044800ca50 comp rx=0 tx=0).stop 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 shutdown_connections 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0438066730 0x7f0438068bf0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0458144180 0x7f045812fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:25.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04580ce1d0 0x7f04580abe10 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:25.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 --2- 172.21.15.119:0/1216532368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04580cc010 0x7f045813f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:25.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 >> 172.21.15.119:0/1216532368 conn(0x7f04580b0980 msgr2=0x7f04580b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:25.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 shutdown_connections 2024-09-13T16:27:25.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.383+0000 7f045759e700 1 -- 172.21.15.119:0/1216532368 wait complete. 2024-09-13T16:27:25.401 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T16:27:25.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:27:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe535331700 1 Processor -- start 2024-09-13T16:27:25.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe535331700 1 -- start start 2024-09-13T16:27:25.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe535331700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe53015cc00 0x7fe5300d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:25.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe535331700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300cdb00 con 0x7fe5300d17e0 2024-09-13T16:27:25.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe535331700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300cdde0 con 0x7fe5300d0c20 2024-09-13T16:27:25.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe535331700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5300ce0c0 con 0x7fe53015cc00 2024-09-13T16:27:25.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe534b30700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe5300d17e0 0x7fe5300b5960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54052/0 (socket says 172.21.15.119:54052) 2024-09-13T16:27:25.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.523+0000 7fe534b30700 1 -- 172.21.15.119:0/3596504199 learned_addr learned my addr 172.21.15.119:0/3596504199 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:25.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52ffff700 1 --2- 172.21.15.119:0/3596504199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe53015cc00 0x7fe5300d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471084737 0 0) 0x7fe5300cdb00 con 0x7fe5300d17e0 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5180035d0 con 0x7fe5300d17e0 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360666859 0 0) 0x7fe5300cdde0 con 0x7fe5300d0c20 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5300cdb00 con 0x7fe5300d0c20 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 921447240 0 0) 0x7fe5180035d0 con 0x7fe5300d17e0 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52ffff700 1 -- 172.21.15.119:0/3596504199 >> v1:172.21.15.145:6789/0 conn(0x7fe5300d0c20 legacy=0x7fe5300d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52ffff700 1 -- 172.21.15.119:0/3596504199 >> v1:172.21.15.119:6789/0 conn(0x7fe5300d17e0 legacy=0x7fe5300b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:25.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52ffff700 1 -- 172.21.15.119:0/3596504199 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300ce3a0 con 0x7fe53015cc00 2024-09-13T16:27:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52ffff700 1 --2- 172.21.15.119:0/3596504199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe53015cc00 0x7fe5300d06e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe524009880 tx=0x7fe524009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f44e25df1bfbdc31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5280019f0 con 0x7fe53015cc00 2024-09-13T16:27:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe524004030 con 0x7fe53015cc00 2024-09-13T16:27:25.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe52effd700 1 -- 172.21.15.119:0/3596504199 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5240043c0 con 0x7fe53015cc00 2024-09-13T16:27:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe535331700 1 -- 172.21.15.119:0/3596504199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe53015cc00 msgr2=0x7fe5300d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe535331700 1 --2- 172.21.15.119:0/3596504199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe53015cc00 0x7fe5300d06e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fe524009880 tx=0x7fe524009b90 comp rx=0 tx=0).stop 2024-09-13T16:27:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe535331700 1 -- 172.21.15.119:0/3596504199 shutdown_connections 2024-09-13T16:27:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe535331700 1 --2- 172.21.15.119:0/3596504199 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe53015cc00 0x7fe5300d06e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe535331700 1 -- 172.21.15.119:0/3596504199 >> 172.21.15.119:0/3596504199 conn(0x7fe53008c080 msgr2=0x7fe5300d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe535331700 1 -- 172.21.15.119:0/3596504199 shutdown_connections 2024-09-13T16:27:25.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.527+0000 7fe535331700 1 -- 172.21.15.119:0/3596504199 wait complete. 2024-09-13T16:27:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 Processor -- start 2024-09-13T16:27:25.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 -- start start 2024-09-13T16:27:25.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5300d0c20 0x7fe53015bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:25.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5300d17e0 0x7fe5300c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:25.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe53015cc00 0x7fe5300c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:25.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5300c7bd0 con 0x7fe53015cc00 2024-09-13T16:27:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53015ea30 con 0x7fe5300d0c20 2024-09-13T16:27:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe535331700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5300c78f0 con 0x7fe5300d17e0 2024-09-13T16:27:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe534b30700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe53015cc00 0x7fe5300c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe534b30700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe53015cc00 0x7fe5300c72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39638/0 (socket says 172.21.15.119:39638) 2024-09-13T16:27:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe534b30700 1 -- 172.21.15.119:0/1657744942 learned_addr learned my addr 172.21.15.119:0/1657744942 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe52f7fe700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5300d17e0 0x7fe5300c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:25.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe52ffff700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5300d0c20 0x7fe53015bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe52f7fe700 1 -- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5300d0c20 msgr2=0x7fe53015bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe52f7fe700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5300d0c20 0x7fe53015bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe52f7fe700 1 -- 172.21.15.119:0/1657744942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe53015cc00 msgr2=0x7fe5300c72c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:25.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe52f7fe700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe53015cc00 0x7fe5300c72c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.531+0000 7fe52f7fe700 1 -- 172.21.15.119:0/1657744942 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5240094b0 con 0x7fe5300d17e0 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe52f7fe700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5300d17e0 0x7fe5300c6d80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe53008cab0 tx=0x7fe52000c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe52cff9700 1 -- 172.21.15.119:0/1657744942 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5200052d0 con 0x7fe5300d17e0 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe52cff9700 1 -- 172.21.15.119:0/1657744942 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe520005430 con 0x7fe5300d17e0 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe53015d520 con 0x7fe5300d17e0 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe52cff9700 1 -- 172.21.15.119:0/1657744942 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5200055a0 con 0x7fe5300d17e0 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53015dae0 con 0x7fe5300d17e0 2024-09-13T16:27:25.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe53015d380 con 0x7fe5300d17e0 2024-09-13T16:27:25.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.535+0000 7fe52cff9700 1 -- 172.21.15.119:0/1657744942 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe5200038e0 con 0x7fe5300d17e0 2024-09-13T16:27:25.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.539+0000 7fe52cff9700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe51c066780 0x7fe51c068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:25.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.539+0000 7fe52cff9700 1 -- 172.21.15.119:0/1657744942 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe520083790 con 0x7fe5300d17e0 2024-09-13T16:27:25.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.539+0000 7fe52ffff700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe51c066780 0x7fe51c068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:25.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.539+0000 7fe52ffff700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe51c066780 0x7fe51c068c40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe5240090b0 tx=0x7fe524010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:25.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.547+0000 7fe52cff9700 1 -- 172.21.15.119:0/1657744942 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe520050aa0 con 0x7fe5300d17e0 2024-09-13T16:27:25.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:25.755+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fe53015d380 con 0x7fe5300d17e0 2024-09-13T16:27:26.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.387+0000 7fe52cff9700 1 -- 172.21.15.119:0/1657744942 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe53015d380 con 0x7fe5300d17e0 2024-09-13T16:27:26.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe51c066780 msgr2=0x7fe51c068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe51c066780 0x7fe51c068c40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe5240090b0 tx=0x7fe524010000 comp rx=0 tx=0).stop 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5300d17e0 msgr2=0x7fe5300c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5300d17e0 0x7fe5300c6d80 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe53008cab0 tx=0x7fe52000c7b0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 shutdown_connections 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe51c066780 0x7fe51c068c40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe53015cc00 0x7fe5300c72c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe5300d17e0 0x7fe5300c6d80 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 --2- 172.21.15.119:0/1657744942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5300d0c20 0x7fe53015bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 >> 172.21.15.119:0/1657744942 conn(0x7fe53008c080 msgr2=0x7fe5301e7530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 shutdown_connections 2024-09-13T16:27:26.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.395+0000 7fe535331700 1 -- 172.21.15.119:0/1657744942 wait complete. 2024-09-13T16:27:26.410 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:27:26.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:26.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4abff700 1 Processor -- start 2024-09-13T16:27:26.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4abff700 1 -- start start 2024-09-13T16:27:26.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4abff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4abff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4d440ce040 0x7f4d440b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4abff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4406cf60 con 0x7f4d4406bb90 2024-09-13T16:27:26.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4abff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4406c350 con 0x7f4d440ce040 2024-09-13T16:27:26.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4abff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4406c630 con 0x7f4d440adb30 2024-09-13T16:27:26.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d493fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4d4406bb90 0x7f4d440cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54066/0 (socket says 172.21.15.119:54066) 2024-09-13T16:27:26.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d493fc700 1 -- 172.21.15.119:0/88849043 learned_addr learned my addr 172.21.15.119:0/88849043 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:26.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d4a3fe700 1 --2- 172.21.15.119:0/88849043 >> v2:172.21.15.119:3300/0 conn(0x7f4d440ce040 0x7f4d440b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d49bfd700 1 --2- 172.21.15.119:0/88849043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d48bfb700 1 -- 172.21.15.119:0/88849043 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536792972 0 0) 0x7f4d4406cf60 con 0x7f4d4406bb90 2024-09-13T16:27:26.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.523+0000 7f4d48bfb700 1 -- 172.21.15.119:0/88849043 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d300035d0 con 0x7f4d4406bb90 2024-09-13T16:27:26.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d49bfd700 1 -- 172.21.15.119:0/88849043 >> v2:172.21.15.119:3300/0 conn(0x7f4d440ce040 msgr2=0x7f4d440b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d49bfd700 1 --2- 172.21.15.119:0/88849043 >> v2:172.21.15.119:3300/0 conn(0x7f4d440ce040 0x7f4d440b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d49bfd700 1 -- 172.21.15.119:0/88849043 >> v1:172.21.15.119:6789/0 conn(0x7f4d4406bb90 legacy=0x7f4d440cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d49bfd700 1 -- 172.21.15.119:0/88849043 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4406c910 con 0x7f4d440adb30 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d49bfd700 1 --2- 172.21.15.119:0/88849043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440cc060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009850 tx=0x7f4d38009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8d26f535bcf5788 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d48bfb700 1 -- 172.21.15.119:0/88849043 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38011830 con 0x7f4d440adb30 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d48bfb700 1 -- 172.21.15.119:0/88849043 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d38011990 con 0x7f4d440adb30 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d48bfb700 1 -- 172.21.15.119:0/88849043 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d38011d40 con 0x7f4d440adb30 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 -- 172.21.15.119:0/88849043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 msgr2=0x7f4d440cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 --2- 172.21.15.119:0/88849043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440cc060 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4d38009850 tx=0x7f4d38009b60 comp rx=0 tx=0).stop 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 -- 172.21.15.119:0/88849043 shutdown_connections 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 --2- 172.21.15.119:0/88849043 >> v2:172.21.15.119:3300/0 conn(0x7f4d440ce040 0x7f4d440b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 --2- 172.21.15.119:0/88849043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440cc060 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 -- 172.21.15.119:0/88849043 >> 172.21.15.119:0/88849043 conn(0x7f4d4406a350 msgr2=0x7f4d4406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:26.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 -- 172.21.15.119:0/88849043 shutdown_connections 2024-09-13T16:27:26.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 -- 172.21.15.119:0/88849043 wait complete. 2024-09-13T16:27:26.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 Processor -- start 2024-09-13T16:27:26.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.527+0000 7f4d4abff700 1 -- start start 2024-09-13T16:27:26.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d4406bb90 0x7f4d441ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440afdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d440ce040 0x7f4d440b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440ad1f0 con 0x7f4d440ce040 2024-09-13T16:27:26.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4412e4c0 con 0x7f4d4406bb90 2024-09-13T16:27:26.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d440b0980 con 0x7f4d440adb30 2024-09-13T16:27:26.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d440ce040 0x7f4d440b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d493fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440afdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d440ce040 0x7f4d440b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39658/0 (socket says 172.21.15.119:39658) 2024-09-13T16:27:26.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d493fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440afdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:39370/0 (socket says 172.21.15.119:39370) 2024-09-13T16:27:26.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d49bfd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d4406bb90 0x7f4d441ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 -- 172.21.15.119:0/3708884024 learned_addr learned my addr 172.21.15.119:0/3708884024 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 -- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 msgr2=0x7f4d440afdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440afdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 -- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d4406bb90 msgr2=0x7f4d441ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d4406bb90 0x7f4d441ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 -- 172.21.15.119:0/3708884024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d380094b0 con 0x7f4d440ce040 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4a3fe700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d440ce040 0x7f4d440b0300 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00eb60 tx=0x7f4d3c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d367fc700 1 -- 172.21.15.119:0/3708884024 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c009740 con 0x7f4d440ce040 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d367fc700 1 -- 172.21.15.119:0/3708884024 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c009e00 con 0x7f4d440ce040 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d440b0c60 con 0x7f4d440ce040 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d367fc700 1 -- 172.21.15.119:0/3708884024 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00ca10 con 0x7f4d440ce040 2024-09-13T16:27:26.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4412e150 con 0x7f4d440ce040 2024-09-13T16:27:26.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.531+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c005310 con 0x7f4d440ce040 2024-09-13T16:27:26.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.535+0000 7f4d367fc700 1 -- 172.21.15.119:0/3708884024 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4d3c00cb70 con 0x7f4d440ce040 2024-09-13T16:27:26.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.535+0000 7f4d367fc700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d28066760 0x7f4d28068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.535+0000 7f4d367fc700 1 -- 172.21.15.119:0/3708884024 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4d3c0844f0 con 0x7f4d440ce040 2024-09-13T16:27:26.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.535+0000 7f4d49bfd700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d28066760 0x7f4d28068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.535+0000 7f4d49bfd700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d28066760 0x7f4d28068c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4d380090b0 tx=0x7f4d38008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:26.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.543+0000 7f4d367fc700 1 -- 172.21.15.119:0/3708884024 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4d3c051710 con 0x7f4d440ce040 2024-09-13T16:27:26.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.747+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d2c005b40 con 0x7f4d440ce040 2024-09-13T16:27:26.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.747+0000 7f4d367fc700 1 -- 172.21.15.119:0/3708884024 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f4d3c054d30 con 0x7f4d440ce040 2024-09-13T16:27:26.750 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:26.750 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d28066760 msgr2=0x7f4d28068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d28066760 0x7f4d28068c20 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4d380090b0 tx=0x7f4d38008030 comp rx=0 tx=0).stop 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d440ce040 msgr2=0x7f4d440b0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d440ce040 0x7f4d440b0300 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c00eb60 tx=0x7f4d3c00ee70 comp rx=0 tx=0).stop 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 shutdown_connections 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d28066760 0x7f4d28068c20 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d440ce040 0x7f4d440b0300 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d440adb30 0x7f4d440afdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 --2- 172.21.15.119:0/3708884024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d4406bb90 0x7f4d441ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 >> 172.21.15.119:0/3708884024 conn(0x7f4d4406a350 msgr2=0x7f4d44066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 shutdown_connections 2024-09-13T16:27:26.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.751+0000 7f4d4abff700 1 -- 172.21.15.119:0/3708884024 wait complete. 2024-09-13T16:27:26.754 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:27:26.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:27:26.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3b0f6700 1 Processor -- start 2024-09-13T16:27:26.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3b0f6700 1 -- start start 2024-09-13T16:27:26.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3b0f6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fda340adb30 0x7fda340cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3b0f6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fda340ce040 0x7fda340b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3b0f6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda3406cf60 con 0x7fda3406bb90 2024-09-13T16:27:26.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3b0f6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3406c350 con 0x7fda340adb30 2024-09-13T16:27:26.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3b0f6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda3406c630 con 0x7fda340ce040 2024-09-13T16:27:26.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3a0f4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fda340adb30 0x7fda340cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda398f3700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fda3406bb90 0x7fda340cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54074/0 (socket says 172.21.15.119:54074) 2024-09-13T16:27:26.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda398f3700 1 -- 172.21.15.119:0/119112258 learned_addr learned my addr 172.21.15.119:0/119112258 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:26.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3a8f5700 1 --2- 172.21.15.119:0/119112258 >> v2:172.21.15.145:3300/0 conn(0x7fda340ce040 0x7fda340b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3a0f4700 1 -- 172.21.15.119:0/119112258 >> v2:172.21.15.145:3300/0 conn(0x7fda340ce040 msgr2=0x7fda340b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3a0f4700 1 --2- 172.21.15.119:0/119112258 >> v2:172.21.15.145:3300/0 conn(0x7fda340ce040 0x7fda340b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3a0f4700 1 -- 172.21.15.119:0/119112258 >> v1:172.21.15.119:6789/0 conn(0x7fda3406bb90 legacy=0x7fda340cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda3a0f4700 1 -- 172.21.15.119:0/119112258 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda3406c910 con 0x7fda340adb30 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.871+0000 7fda390f2700 1 -- 172.21.15.119:0/119112258 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849938970 0 0) 0x7fda3406cf60 con 0x7fda3406bb90 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3a0f4700 1 --2- 172.21.15.119:0/119112258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda340cc060 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fda340cd1c0 tx=0x7fda2c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76718f36d91e4ca3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda390f2700 1 -- 172.21.15.119:0/119112258 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c00e070 con 0x7fda340adb30 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda390f2700 1 -- 172.21.15.119:0/119112258 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda2c00bbd0 con 0x7fda340adb30 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda390f2700 1 -- 172.21.15.119:0/119112258 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda2c012430 con 0x7fda340adb30 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 -- 172.21.15.119:0/119112258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 msgr2=0x7fda340cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/119112258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda340cc060 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7fda340cd1c0 tx=0x7fda2c00b330 comp rx=0 tx=0).stop 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 -- 172.21.15.119:0/119112258 shutdown_connections 2024-09-13T16:27:26.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/119112258 >> v2:172.21.15.145:3300/0 conn(0x7fda340ce040 0x7fda340b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/119112258 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda340cc060 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 -- 172.21.15.119:0/119112258 >> 172.21.15.119:0/119112258 conn(0x7fda3406a350 msgr2=0x7fda3406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:26.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 -- 172.21.15.119:0/119112258 shutdown_connections 2024-09-13T16:27:26.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.875+0000 7fda3b0f6700 1 -- 172.21.15.119:0/119112258 wait complete. 2024-09-13T16:27:26.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 Processor -- start 2024-09-13T16:27:26.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 -- start start 2024-09-13T16:27:26.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda3406bb90 0x7fda3413f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda3413a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda340ce040 0x7fda34159e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3413b9d0 con 0x7fda340adb30 2024-09-13T16:27:26.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda3413adc0 con 0x7fda3406bb90 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3b0f6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda3413b0a0 con 0x7fda340ce040 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda3413a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda3413a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39670/0 (socket says 172.21.15.119:39670) 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 -- 172.21.15.119:0/1132614451 learned_addr learned my addr 172.21.15.119:0/1132614451 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3a0f4700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda3406bb90 0x7fda3413f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda3a8f5700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda340ce040 0x7fda34159e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 -- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda340ce040 msgr2=0x7fda34159e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda340ce040 0x7fda34159e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 -- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda3406bb90 msgr2=0x7fda3413f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda3406bb90 0x7fda3413f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 -- 172.21.15.119:0/1132614451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda2c009a70 con 0x7fda340adb30 2024-09-13T16:27:26.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda398f3700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda3413a730 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c580 tx=0x7fda2400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:26.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.879+0000 7fda2affd700 1 -- 172.21.15.119:0/1132614451 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24004ca0 con 0x7fda340adb30 2024-09-13T16:27:26.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.883+0000 7fda2affd700 1 -- 172.21.15.119:0/1132614451 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda24004e20 con 0x7fda340adb30 2024-09-13T16:27:26.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.883+0000 7fda2affd700 1 -- 172.21.15.119:0/1132614451 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda24007310 con 0x7fda340adb30 2024-09-13T16:27:26.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.883+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda3413b380 con 0x7fda340adb30 2024-09-13T16:27:26.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.883+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda3413dbe0 con 0x7fda340adb30 2024-09-13T16:27:26.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.883+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda3406d3d0 con 0x7fda340adb30 2024-09-13T16:27:26.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.883+0000 7fda2affd700 1 -- 172.21.15.119:0/1132614451 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fda24007470 con 0x7fda340adb30 2024-09-13T16:27:26.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.887+0000 7fda2affd700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fda18066730 0x7fda18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:26.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.887+0000 7fda2affd700 1 -- 172.21.15.119:0/1132614451 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fda24084090 con 0x7fda340adb30 2024-09-13T16:27:26.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.887+0000 7fda3a0f4700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fda18066730 0x7fda18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:26.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.887+0000 7fda3a0f4700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fda18066730 0x7fda18068bf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fda2c014040 tx=0x7fda2c011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:26.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:26.891+0000 7fda2affd700 1 -- 172.21.15.119:0/1132614451 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fda3406d3d0 con 0x7fda340adb30 2024-09-13T16:27:27.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:27:27.016 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:27:27.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.067+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda3406d3d0 con 0x7fda340adb30 2024-09-13T16:27:27.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.067+0000 7fda2affd700 1 -- 172.21.15.119:0/1132614451 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16834 (secure 0 0 0) 0x7fda3406d3d0 con 0x7fda340adb30 2024-09-13T16:27:27.071 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:27.071 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":54,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:24.352264+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:27:21.660330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-09-13T16:27:22.341239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:27:27.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fda18066730 msgr2=0x7fda18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fda18066730 0x7fda18068bf0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fda2c014040 tx=0x7fda2c011000 comp rx=0 tx=0).stop 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 msgr2=0x7fda3413a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda3413a730 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fda2400c580 tx=0x7fda2400c940 comp rx=0 tx=0).stop 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 shutdown_connections 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fda18066730 0x7fda18068bf0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fda340ce040 0x7fda34159e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fda340adb30 0x7fda3413a730 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 --2- 172.21.15.119:0/1132614451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fda3406bb90 0x7fda3413f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 >> 172.21.15.119:0/1132614451 conn(0x7fda3406a350 msgr2=0x7fda3415d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 shutdown_connections 2024-09-13T16:27:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.071+0000 7fda3b0f6700 1 -- 172.21.15.119:0/1132614451 wait complete. 2024-09-13T16:27:27.086 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:27:27.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:27:27.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.195+0000 7f1057026700 1 Processor -- start 2024-09-13T16:27:27.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.195+0000 7f1057026700 1 -- start start 2024-09-13T16:27:27.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10500adb30 0x7f10500cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500ce040 0x7f10500b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f105006cf60 con 0x7f105006bb90 2024-09-13T16:27:27.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f105006c350 con 0x7f10500adb30 2024-09-13T16:27:27.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105006c630 con 0x7f10500ce040 2024-09-13T16:27:27.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056024700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f10500adb30 0x7f10500cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1055823700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f105006bb90 0x7f10500cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:57060/0 (socket says 172.21.15.119:57060) 2024-09-13T16:27:27.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1055823700 1 -- 172.21.15.119:0/1908104470 learned_addr learned my addr 172.21.15.119:0/1908104470 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:27.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056825700 1 --2- 172.21.15.119:0/1908104470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500ce040 0x7f10500b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1055022700 1 -- 172.21.15.119:0/1908104470 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551207373 0 0) 0x7f105006cf60 con 0x7f105006bb90 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1055022700 1 -- 172.21.15.119:0/1908104470 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103c0035d0 con 0x7f105006bb90 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056024700 1 -- 172.21.15.119:0/1908104470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500ce040 msgr2=0x7f10500b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056024700 1 --2- 172.21.15.119:0/1908104470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500ce040 0x7f10500b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056024700 1 -- 172.21.15.119:0/1908104470 >> v1:172.21.15.145:6789/0 conn(0x7f105006bb90 legacy=0x7f10500cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056024700 1 -- 172.21.15.119:0/1908104470 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105006c910 con 0x7f10500adb30 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056825700 1 --2- 172.21.15.119:0/1908104470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500ce040 0x7f10500b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1056024700 1 --2- 172.21.15.119:0/1908104470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10500adb30 0x7f10500cc060 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1040009850 tx=0x7f1040009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ff1a2bb4220554d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1055022700 1 -- 172.21.15.119:0/1908104470 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1040011830 con 0x7f10500adb30 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1055022700 1 -- 172.21.15.119:0/1908104470 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1040011990 con 0x7f10500adb30 2024-09-13T16:27:27.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1055022700 1 -- 172.21.15.119:0/1908104470 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1040011d00 con 0x7f10500adb30 2024-09-13T16:27:27.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- 172.21.15.119:0/1908104470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10500adb30 msgr2=0x7f10500cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 --2- 172.21.15.119:0/1908104470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10500adb30 0x7f10500cc060 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f1040009850 tx=0x7f1040009b60 comp rx=0 tx=0).stop 2024-09-13T16:27:27.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- 172.21.15.119:0/1908104470 shutdown_connections 2024-09-13T16:27:27.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 --2- 172.21.15.119:0/1908104470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500ce040 0x7f10500b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 --2- 172.21.15.119:0/1908104470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10500adb30 0x7f10500cc060 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- 172.21.15.119:0/1908104470 >> 172.21.15.119:0/1908104470 conn(0x7f105006a350 msgr2=0x7f105006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:27.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- 172.21.15.119:0/1908104470 shutdown_connections 2024-09-13T16:27:27.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.199+0000 7f1057026700 1 -- 172.21.15.119:0/1908104470 wait complete. 2024-09-13T16:27:27.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 Processor -- start 2024-09-13T16:27:27.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 -- start start 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f105006bb90 0x7f105014b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500adb30 0x7f105015bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10500ce040 0x7f10501386e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105014f2a0 con 0x7f105006bb90 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105014ed50 con 0x7f10500ce040 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105014be30 con 0x7f10500adb30 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056024700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f105006bb90 0x7f105014b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056024700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f105006bb90 0x7f105014b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39690/0 (socket says 172.21.15.119:39690) 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056024700 1 -- 172.21.15.119:0/1174449913 learned_addr learned my addr 172.21.15.119:0/1174449913 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:27.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056825700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10500ce040 0x7f10501386e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1055823700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500adb30 0x7f105015bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056825700 1 -- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500adb30 msgr2=0x7f105015bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056825700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500adb30 0x7f105015bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056825700 1 -- 172.21.15.119:0/1174449913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f105006bb90 msgr2=0x7f105014b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056825700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f105006bb90 0x7f105014b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056825700 1 -- 172.21.15.119:0/1174449913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10480094b0 con 0x7f10500ce040 2024-09-13T16:27:27.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056024700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f105006bb90 0x7f105014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1056825700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10500ce040 0x7f10501386e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f104800e930 tx=0x7f104800ec40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1046ffd700 1 -- 172.21.15.119:0/1174449913 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048004020 con 0x7f10500ce040 2024-09-13T16:27:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1046ffd700 1 -- 172.21.15.119:0/1174449913 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10480041a0 con 0x7f10500ce040 2024-09-13T16:27:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10400094b0 con 0x7f10500ce040 2024-09-13T16:27:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1046ffd700 1 -- 172.21.15.119:0/1174449913 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1048004550 con 0x7f10500ce040 2024-09-13T16:27:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.203+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10501f3e90 con 0x7f10500ce040 2024-09-13T16:27:27.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.207+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105006d3d0 con 0x7f10500ce040 2024-09-13T16:27:27.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.207+0000 7f1046ffd700 1 -- 172.21.15.119:0/1174449913 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1048005a40 con 0x7f10500ce040 2024-09-13T16:27:27.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.207+0000 7f1046ffd700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1034066730 0x7f1034068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.207+0000 7f1046ffd700 1 -- 172.21.15.119:0/1174449913 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1048084170 con 0x7f10500ce040 2024-09-13T16:27:27.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.207+0000 7f1056024700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1034066730 0x7f1034068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.207+0000 7f1056024700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1034066730 0x7f1034068bf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1040009ba0 tx=0x7f1040008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.219+0000 7f1046ffd700 1 -- 172.21.15.119:0/1174449913 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f10480513a0 con 0x7f10500ce040 2024-09-13T16:27:27.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.383+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f10500adf10 con 0x7f1034066730 2024-09-13T16:27:27.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.399+0000 7f1046ffd700 1 -- 172.21.15.119:0/1174449913 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+273212 (secure 0 0 0) 0x7f1040011630 con 0x7f1034066730 2024-09-13T16:27:27.407 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:27.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1034066730 msgr2=0x7f1034068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1034066730 0x7f1034068bf0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1040009ba0 tx=0x7f1040008030 comp rx=0 tx=0).stop 2024-09-13T16:27:27.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10500ce040 msgr2=0x7f10501386e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10500ce040 0x7f10501386e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f104800e930 tx=0x7f104800ec40 comp rx=0 tx=0).stop 2024-09-13T16:27:27.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 shutdown_connections 2024-09-13T16:27:27.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1034066730 0x7f1034068bf0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f10500ce040 0x7f10501386e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10500adb30 0x7f105015bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 --2- 172.21.15.119:0/1174449913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f105006bb90 0x7f105014b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 >> 172.21.15.119:0/1174449913 conn(0x7f105006a350 msgr2=0x7f10500cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:27.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 shutdown_connections 2024-09-13T16:27:27.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.407+0000 7f1057026700 1 -- 172.21.15.119:0/1174449913 wait complete. 2024-09-13T16:27:27.411 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:27:27.423 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":115,"stamp":"2024-09-13T16:27:25.464500+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":226016,"kb_used_data":5720,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754748704,"statfs":{"total":773094113280,"available":772862672896,"internally_reserved":0,"allocated":5857280,"data_stored":3205796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001280"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:22.963736+0000","last_change":"2024-09-13T16:27:22.963736+0000","last_active":"2024-09-13T16:27:22.963736+0000","last_peered":"2024-09-13T16:27:22.963736+0000","last_clean":"2024-09-13T16:27:22.963736+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:22.963736+0000","last_undegraded":"2024-09-13T16:27:22.963736+0000","last_fullsized":"2024-09-13T16:27:22.963736+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:27:21.954146+0000","last_change":"2024-09-13T16:27:21.954146+0000","last_active":"2024-09-13T16:27:21.954146+0000","last_peered":"2024-09-13T16:27:21.954146+0000","last_clean":"2024-09-13T16:27:21.954146+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:27:21.954146+0000","last_undegraded":"2024-09-13T16:27:21.954146+0000","last_fullsized":"2024-09-13T16:27:21.954146+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":186,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-09-13T16:27:18.914720+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:27:18.914720+0000","last_peered":"2024-09-13T16:27:18.914720+0000","last_clean":"2024-09-13T16:27:18.914720+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:27:18.914720+0000","last_undegraded":"2024-09-13T16:27:18.914720+0000","last_fullsized":"2024-09-13T16:27:18.914720+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542173,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28168,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343672,"statfs":{"total":96636764160,"available":96607920128,"internally_reserved":0,"allocated":1171456,"data_stored":840777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.85999999999999999,"15min":0.84799999999999998},"min":{"1min":0.65100000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.85499999999999998},"min":{"1min":0.67300000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0760000000000001,"5min":1.121,"15min":1.121},"last":1.0589999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.85399999999999998,"15min":0.84599999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.76000000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.89000000000000001,"15min":0.88700000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0880000000000001,"5min":1.228,"15min":1.228},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83299999999999996,"15min":0.83099999999999996},"min":{"1min":0.64500000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0640000000000001,"5min":1.091,"15min":1.091},"last":0.79500000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.02,"15min":1.02},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.228,"5min":1.238,"15min":1.238},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.0049999999999999,"15min":1.0089999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.2190000000000001,"5min":1.226,"15min":1.226},"last":0.98799999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.88300000000000001,"15min":0.89200000000000002},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.05,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0880000000000001,"15min":1.0980000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1830000000000001,"5min":1.589,"15min":1.589},"last":0.84899999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.024,"15min":1.0389999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.1799999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.119,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":1.0249999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.2170000000000001,"5min":1.337,"15min":1.337},"last":0.93500000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0880000000000001,"15min":1.0900000000000001},"min":{"1min":0.84299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.22,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":1.018},{"interface":"front","average":{"1min":1.133,"5min":1.181,"15min":1.1890000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.4199999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.044}]}]},{"osd":6,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27588,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":577536,"data_stored":250409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83699999999999997,"15min":0.84899999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1220000000000001,"5min":1.153,"15min":1.153},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.878,"5min":0.89300000000000002,"15min":0.89500000000000002},"min":{"1min":0.56799999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0660000000000001,"5min":1.143,"15min":1.143},"last":1.1200000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83399999999999996,"15min":0.83799999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0229999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90400000000000003,"15min":0.91100000000000003},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.0669999999999999,"5min":1.296,"15min":1.296},"last":1.0900000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.88100000000000001,"15min":0.88700000000000001},"min":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.0449999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84399999999999997,"15min":0.84299999999999997},"min":{"1min":0.60299999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.747}]},{"osd":3,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.264,"5min":1.536,"15min":1.536},"last":1.004},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98299999999999998,"15min":0.98799999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1819999999999999,"5min":1.409,"15min":1.409},"last":0.95999999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.97999999999999998,"15min":0.98299999999999998},"min":{"1min":0.66300000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.216,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.028,"5min":0.94799999999999995,"15min":0.93500000000000005},"min":{"1min":0.81200000000000006,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1719999999999999,"5min":1.224,"15min":1.224},"last":1.0449999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.91500000000000004,"15min":0.91400000000000003},"min":{"1min":0.73099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1379999999999999,"15min":1.143},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.329,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.68400000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.129,"15min":1.1259999999999999},"min":{"1min":0.90300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.4319999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":0.874},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.879,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":0.90400000000000003}]}]},{"osd":5,"up_from":14,"seq":60129542174,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31696,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340144,"statfs":{"total":96636764160,"available":96604307456,"internally_reserved":0,"allocated":589824,"data_stored":255856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.54000000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.96999999999999997},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.87,"15min":0.86199999999999999},"min":{"1min":0.60899999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3049999999999999,"5min":1.675,"15min":1.675},"last":0.79400000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.875,"15min":0.88300000000000001},"min":{"1min":0.625,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.0569999999999999,"5min":1.369,"15min":1.369},"last":0.90100000000000002},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.82999999999999996,"15min":0.82399999999999995},"min":{"1min":0.65400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0740000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.85199999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.81000000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.276,"5min":1.625,"15min":1.625},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.97399999999999998,"15min":0.98799999999999999},"min":{"1min":0.70099999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.286,"5min":1.552,"15min":1.552},"last":1.05}]},{"osd":3,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.84999999999999998,"15min":0.83599999999999997},"min":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.004,"15min":1.0269999999999999},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1419999999999999,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":0.76300000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.0669999999999999,"15min":1.081},"min":{"1min":0.61599999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.363,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0660000000000001,"15min":1.0620000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.47,"5min":1.601,"15min":1.601},"last":0.98699999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.97499999999999998,"15min":0.97799999999999998},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.2529999999999999,"5min":1.421,"15min":1.421},"last":0.63800000000000001},{"interface":"front","average":{"1min":1.054,"5min":1.0740000000000001,"15min":1.0780000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.397,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.81000000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.044,"15min":1.038},"min":{"1min":0.76300000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3819999999999999,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.071,"5min":1.2190000000000001,"15min":1.244},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.417,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":0.86799999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542172,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27608,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344232,"statfs":{"total":96636764160,"available":96608493568,"internally_reserved":0,"allocated":589824,"data_stored":255856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84199999999999997,"15min":0.83799999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84699999999999998,"15min":0.84899999999999998},"min":{"1min":0.59899999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.78100000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.91000000000000003,"15min":0.91400000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3049999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.1339999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.72799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.353,"15min":1.353},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.85199999999999998,"15min":0.84099999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.61099999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.2749999999999999,"5min":1.603,"15min":1.603},"last":0.68600000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.95599999999999996,"15min":0.95799999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.298,"5min":1.472,"15min":1.472},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96199999999999997,"15min":0.96499999999999997},"min":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.98399999999999999,"15min":0.97999999999999998},"min":{"1min":0.70099999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":2.048,"15min":2.048},"last":0.86099999999999999},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1519999999999999},"min":{"1min":0.83699999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4179999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.71299999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.749,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.1220000000000001,"15min":1.1080000000000001},"min":{"1min":0.876,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.74,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.016}]},{"osd":7,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.165,"15min":1.153},"min":{"1min":0.95999999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.0700000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27592,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":577536,"data_stored":250409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.95299999999999996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.624,"15min":0.62},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79000000000000004,"15min":0.78900000000000003},"min":{"1min":0.52000000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.1339999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.83699999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.69099999999999995,"15min":0.69799999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90100000000000002,"5min":1.27,"15min":1.27},"last":0.69399999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.755,"15min":0.752},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.745}]},{"osd":4,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.131,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.96299999999999997},"min":{"1min":0.78500000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.222,"5min":1.782,"15min":1.782},"last":1.0609999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.97999999999999998,"15min":0.98299999999999998},"min":{"1min":0.68300000000000005,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.3360000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1040000000000001,"15min":1.1060000000000001},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.2749999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.90200000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1339999999999999,"15min":1.1419999999999999},"min":{"1min":0.81100000000000005,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6950000000000001,"5min":1.885,"15min":1.885},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.052,"15min":1.0640000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.137,"5min":1.95,"15min":1.95},"last":0.871}]},{"osd":7,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0780000000000001,"15min":1.0760000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.999},{"interface":"front","average":{"1min":1.018,"5min":1.054,"15min":1.0600000000000001},"min":{"1min":0.83399999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3029999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":1.1120000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542174,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27596,"kb_used_data":576,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":589824,"data_stored":255856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.47899999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.84399999999999997,"5min":0.995,"15min":0.995},"last":0.67500000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.69199999999999995},"min":{"1min":0.441,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.95899999999999996,"5min":1.123,"15min":1.123},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.65900000000000003,"15min":0.66500000000000004},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":1.073,"15min":1.073},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79000000000000004,"15min":0.78700000000000003},"min":{"1min":0.63400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.98599999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.77800000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.94599999999999995,"15min":0.95399999999999996},"min":{"1min":0.69999999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.127,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.96999999999999997,"15min":0.98799999999999999},"min":{"1min":0.68500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1020000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.89000000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.98999999999999999,"15min":0.98899999999999999},"min":{"1min":0.76100000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.3759999999999999,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.94399999999999995},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3300000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.751}]},{"osd":6,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.0229999999999999,"15min":1.0449999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.046,"5min":2.024,"15min":2.024},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.069,"15min":1.0700000000000001},"min":{"1min":0.84899999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.431,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.92400000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.083,"15min":1.087},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.351,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.004,"5min":1.0629999999999999,"15min":1.073},"min":{"1min":0.76700000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.204,"5min":1.55,"15min":1.55},"last":0.95899999999999996}]}]},{"osd":1,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27588,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":577536,"data_stored":250409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41499999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79600000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71999999999999997,"15min":0.72099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.89400000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.378,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.98199999999999998,"5min":1,"15min":1},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.48099999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0629999999999999,"5min":1.458,"15min":1.458},"last":0.64100000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.70899999999999996,"15min":0.68500000000000005},"min":{"1min":0.56100000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97399999999999998,"5min":1.403,"15min":1.403},"last":0.55200000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.90500000000000003,"15min":0.90000000000000002},"min":{"1min":0.67300000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3160000000000001,"5min":1.546,"15min":1.546},"last":0.97099999999999997},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93300000000000005,"15min":0.93500000000000005},"min":{"1min":0.61299999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1240000000000001,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.85299999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.67900000000000005,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3089999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.96999999999999997,"15min":0.98199999999999998},"min":{"1min":0.75900000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2250000000000001,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.92800000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.999,"15min":1.0049999999999999},"min":{"1min":0.77200000000000002,"5min":0.626,"15min":0.626},"max":{"1min":1.3460000000000001,"5min":1.613,"15min":1.613},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.70899999999999996,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3839999999999999,"5min":1.698,"15min":1.698},"last":1.0629999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0029999999999999,"15min":1.0089999999999999},"min":{"1min":0.72999999999999998,"5min":0.62,"15min":0.62},"max":{"1min":1.446,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.0329999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0329999999999999,"15min":1.0349999999999999},"min":{"1min":0.69099999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.4179999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.145}]}]},{"osd":0,"up_from":14,"seq":60129542173,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28180,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343660,"statfs":{"total":96636764160,"available":96607907840,"internally_reserved":0,"allocated":1183744,"data_stored":846224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.75600000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.93799999999999994}]},{"osd":2,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.752}]},{"osd":3,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81000000000000005,"15min":0.81399999999999995},"min":{"1min":0.45300000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.71899999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":0.95999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2949999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":1.052},{"interface":"front","average":{"1min":1.04,"5min":0.96999999999999997,"15min":0.95799999999999996},"min":{"1min":0.68700000000000006,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.96299999999999997,"15min":0.96399999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2,"5min":1.24,"15min":1.24},"last":0.88},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.99299999999999999,"15min":0.999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.198,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.1060000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.995,"15min":0.97799999999999998},"min":{"1min":0.80900000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.036,"15min":1.04},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.0169999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.054,"15min":1.046},"min":{"1min":0.90000000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.149},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0620000000000001,"15min":1.038},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.92000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:27:27.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:27:27.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.539+0000 7f29ab1c5700 1 Processor -- start 2024-09-13T16:27:27.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.539+0000 7f29ab1c5700 1 -- start start 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f29a4071740 0x7f29a406ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f29a4158ea0 0x7f29a406d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29a406d8b0 con 0x7f29a4158ac0 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a406de40 con 0x7f29a4071740 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a406db60 con 0x7f29a4158ea0 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29aa1c3700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f29a4158ac0 0x7f29a4071030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54086/0 (socket says 172.21.15.119:54086) 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a99c2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f29a4071740 0x7f29a406ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29aa1c3700 1 -- 172.21.15.119:0/3169364725 learned_addr learned my addr 172.21.15.119:0/3169364725 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:27.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29aa9c4700 1 --2- 172.21.15.119:0/3169364725 >> v2:172.21.15.145:3300/0 conn(0x7f29a4158ea0 0x7f29a406d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a99c2700 1 -- 172.21.15.119:0/3169364725 >> v2:172.21.15.145:3300/0 conn(0x7f29a4158ea0 msgr2=0x7f29a406d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a99c2700 1 --2- 172.21.15.119:0/3169364725 >> v2:172.21.15.145:3300/0 conn(0x7f29a4158ea0 0x7f29a406d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a99c2700 1 -- 172.21.15.119:0/3169364725 >> v1:172.21.15.119:6789/0 conn(0x7f29a4158ac0 legacy=0x7f29a4071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a91c1700 1 -- 172.21.15.119:0/3169364725 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391526835 0 0) 0x7f29a406d8b0 con 0x7f29a4158ac0 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a99c2700 1 -- 172.21.15.119:0/3169364725 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a40b1990 con 0x7f29a4071740 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a99c2700 1 --2- 172.21.15.119:0/3169364725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4071740 0x7f29a406ce30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f29a415c810 tx=0x7f2994009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=249a2955f7b79db4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a91c1700 1 -- 172.21.15.119:0/3169364725 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994010800 con 0x7f29a4071740 2024-09-13T16:27:27.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a91c1700 1 -- 172.21.15.119:0/3169364725 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994010960 con 0x7f29a4071740 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29a91c1700 1 -- 172.21.15.119:0/3169364725 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994010cf0 con 0x7f29a4071740 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- 172.21.15.119:0/3169364725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4071740 msgr2=0x7f29a406ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/3169364725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4071740 0x7f29a406ce30 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f29a415c810 tx=0x7f2994009b30 comp rx=0 tx=0).stop 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- 172.21.15.119:0/3169364725 shutdown_connections 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/3169364725 >> v2:172.21.15.145:3300/0 conn(0x7f29a4158ea0 0x7f29a406d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/3169364725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4071740 0x7f29a406ce30 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- 172.21.15.119:0/3169364725 >> 172.21.15.119:0/3169364725 conn(0x7f29a406a350 msgr2=0x7f29a406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:27.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- 172.21.15.119:0/3169364725 shutdown_connections 2024-09-13T16:27:27.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.543+0000 7f29ab1c5700 1 -- 172.21.15.119:0/3169364725 wait complete. 2024-09-13T16:27:27.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 Processor -- start 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 -- start start 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a4071740 0x7f29a415bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4158ac0 0x7f29a4152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a4158ea0 0x7f29a41516a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a415c400 con 0x7f29a4158ac0 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a4155b90 con 0x7f29a4071740 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a4155260 con 0x7f29a4158ea0 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4158ac0 0x7f29a4152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4158ac0 0x7f29a4152ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39708/0 (socket says 172.21.15.119:39708) 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 -- 172.21.15.119:0/1832057062 learned_addr learned my addr 172.21.15.119:0/1832057062 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:27.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29aa1c3700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a4071740 0x7f29a415bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29aa9c4700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a4158ea0 0x7f29a41516a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 -- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a4158ea0 msgr2=0x7f29a41516a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a4158ea0 0x7f29a41516a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 -- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a4071740 msgr2=0x7f29a415bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a4071740 0x7f29a415bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 -- 172.21.15.119:0/1832057062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29940094b0 con 0x7f29a4158ac0 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29a99c2700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4158ac0 0x7f29a4152ba0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2994009fd0 tx=0x7f2994004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f299affd700 1 -- 172.21.15.119:0/1832057062 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994003ad0 con 0x7f29a4158ac0 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f299affd700 1 -- 172.21.15.119:0/1832057062 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994003c30 con 0x7f29a4158ac0 2024-09-13T16:27:27.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f299affd700 1 -- 172.21.15.119:0/1832057062 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29940045a0 con 0x7f29a4158ac0 2024-09-13T16:27:27.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a4155540 con 0x7f29a4158ac0 2024-09-13T16:27:27.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.547+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a41530e0 con 0x7f29a4158ac0 2024-09-13T16:27:27.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.551+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a406c0a0 con 0x7f29a4158ac0 2024-09-13T16:27:27.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.551+0000 7f299affd700 1 -- 172.21.15.119:0/1832057062 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2994018400 con 0x7f29a4158ac0 2024-09-13T16:27:27.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.551+0000 7f299affd700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2988066760 0x7f2988068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.551+0000 7f299affd700 1 -- 172.21.15.119:0/1832057062 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2994085180 con 0x7f29a4158ac0 2024-09-13T16:27:27.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.551+0000 7f29aa1c3700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2988066760 0x7f2988068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.555+0000 7f29aa1c3700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2988066760 0x7f2988068c20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f29a0000f60 tx=0x7f29a0007e00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.559+0000 7f299affd700 1 -- 172.21.15.119:0/1832057062 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f29940523a0 con 0x7f29a4158ac0 2024-09-13T16:27:27.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.727+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f29a40b1f50 con 0x7f2988066760 2024-09-13T16:27:27.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.739+0000 7f299affd700 1 -- 172.21.15.119:0/1832057062 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287701 (secure 0 0 0) 0x7f29a0001370 con 0x7f2988066760 2024-09-13T16:27:27.747 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:27.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2988066760 msgr2=0x7f2988068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2988066760 0x7f2988068c20 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f29a0000f60 tx=0x7f29a0007e00 comp rx=0 tx=0).stop 2024-09-13T16:27:27.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4158ac0 msgr2=0x7f29a4152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4158ac0 0x7f29a4152ba0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f2994009fd0 tx=0x7f2994004770 comp rx=0 tx=0).stop 2024-09-13T16:27:27.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 shutdown_connections 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2988066760 0x7f2988068c20 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29a4158ea0 0x7f29a41516a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29a4158ac0 0x7f29a4152ba0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 --2- 172.21.15.119:0/1832057062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f29a4071740 0x7f29a415bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 >> 172.21.15.119:0/1832057062 conn(0x7f29a406a350 msgr2=0x7f29a40ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 shutdown_connections 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.747+0000 7f29ab1c5700 1 -- 172.21.15.119:0/1832057062 wait complete. 2024-09-13T16:27:27.751 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:27:27.764 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-13T16:27:27.465122+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":116,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":226516,"kb_used_data":6216,"kb_used_omap":12,"kb_used_meta":220147,"kb_avail":754748204,"statfs":{"total":773094113280,"available":772862160896,"internally_reserved":0,"allocated":6365184,"data_stored":3443884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":225431097},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.499992"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.366206+0000","last_change":"2024-09-13T16:27:23.368566+0000","last_active":"2024-09-13T16:27:24.366206+0000","last_peered":"2024-09-13T16:27:24.366206+0000","last_clean":"2024-09-13T16:27:24.366206+0000","last_became_active":"2024-09-13T16:27:23.368292+0000","last_became_peered":"2024-09-13T16:27:23.368292+0000","last_unstale":"2024-09-13T16:27:24.366206+0000","last_undegraded":"2024-09-13T16:27:24.366206+0000","last_fullsized":"2024-09-13T16:27:24.366206+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:58:04.928965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477479+0000","last_change":"2024-09-13T16:27:23.373247+0000","last_active":"2024-09-13T16:27:24.477479+0000","last_peered":"2024-09-13T16:27:24.477479+0000","last_clean":"2024-09-13T16:27:24.477479+0000","last_became_active":"2024-09-13T16:27:23.372972+0000","last_became_peered":"2024-09-13T16:27:23.372972+0000","last_unstale":"2024-09-13T16:27:24.477479+0000","last_undegraded":"2024-09-13T16:27:24.477479+0000","last_fullsized":"2024-09-13T16:27:24.477479+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:05.896471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478725+0000","last_change":"2024-09-13T16:27:23.379730+0000","last_active":"2024-09-13T16:27:24.478725+0000","last_peered":"2024-09-13T16:27:24.478725+0000","last_clean":"2024-09-13T16:27:24.478725+0000","last_became_active":"2024-09-13T16:27:23.379497+0000","last_became_peered":"2024-09-13T16:27:23.379497+0000","last_unstale":"2024-09-13T16:27:24.478725+0000","last_undegraded":"2024-09-13T16:27:24.478725+0000","last_fullsized":"2024-09-13T16:27:24.478725+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:14:32.726748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477044+0000","last_change":"2024-09-13T16:27:23.374309+0000","last_active":"2024-09-13T16:27:24.477044+0000","last_peered":"2024-09-13T16:27:24.477044+0000","last_clean":"2024-09-13T16:27:24.477044+0000","last_became_active":"2024-09-13T16:27:23.373557+0000","last_became_peered":"2024-09-13T16:27:23.373557+0000","last_unstale":"2024-09-13T16:27:24.477044+0000","last_undegraded":"2024-09-13T16:27:24.477044+0000","last_fullsized":"2024-09-13T16:27:24.477044+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:19.345148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370974+0000","last_change":"2024-09-13T16:27:23.377249+0000","last_active":"2024-09-13T16:27:24.370974+0000","last_peered":"2024-09-13T16:27:24.370974+0000","last_clean":"2024-09-13T16:27:24.370974+0000","last_became_active":"2024-09-13T16:27:23.376969+0000","last_became_peered":"2024-09-13T16:27:23.376969+0000","last_unstale":"2024-09-13T16:27:24.370974+0000","last_undegraded":"2024-09-13T16:27:24.370974+0000","last_fullsized":"2024-09-13T16:27:24.370974+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:33.025697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365419+0000","last_change":"2024-09-13T16:27:23.376341+0000","last_active":"2024-09-13T16:27:24.365419+0000","last_peered":"2024-09-13T16:27:24.365419+0000","last_clean":"2024-09-13T16:27:24.365419+0000","last_became_active":"2024-09-13T16:27:23.376058+0000","last_became_peered":"2024-09-13T16:27:23.376058+0000","last_unstale":"2024-09-13T16:27:24.365419+0000","last_undegraded":"2024-09-13T16:27:24.365419+0000","last_fullsized":"2024-09-13T16:27:24.365419+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:33.025697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478708+0000","last_change":"2024-09-13T16:27:23.382944+0000","last_active":"2024-09-13T16:27:24.478708+0000","last_peered":"2024-09-13T16:27:24.478708+0000","last_clean":"2024-09-13T16:27:24.478708+0000","last_became_active":"2024-09-13T16:27:23.382704+0000","last_became_peered":"2024-09-13T16:27:23.382704+0000","last_unstale":"2024-09-13T16:27:24.478708+0000","last_undegraded":"2024-09-13T16:27:24.478708+0000","last_fullsized":"2024-09-13T16:27:24.478708+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:37.969508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476306+0000","last_change":"2024-09-13T16:27:23.374042+0000","last_active":"2024-09-13T16:27:24.476306+0000","last_peered":"2024-09-13T16:27:24.476306+0000","last_clean":"2024-09-13T16:27:24.476306+0000","last_became_active":"2024-09-13T16:27:23.373714+0000","last_became_peered":"2024-09-13T16:27:23.373714+0000","last_unstale":"2024-09-13T16:27:24.476306+0000","last_undegraded":"2024-09-13T16:27:24.476306+0000","last_fullsized":"2024-09-13T16:27:24.476306+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:08.223466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477289+0000","last_change":"2024-09-13T16:27:23.384113+0000","last_active":"2024-09-13T16:27:24.477289+0000","last_peered":"2024-09-13T16:27:24.477289+0000","last_clean":"2024-09-13T16:27:24.477289+0000","last_became_active":"2024-09-13T16:27:23.383879+0000","last_became_peered":"2024-09-13T16:27:23.383879+0000","last_unstale":"2024-09-13T16:27:24.477289+0000","last_undegraded":"2024-09-13T16:27:24.477289+0000","last_fullsized":"2024-09-13T16:27:24.477289+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:58.232141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365491+0000","last_change":"2024-09-13T16:27:23.379265+0000","last_active":"2024-09-13T16:27:24.365491+0000","last_peered":"2024-09-13T16:27:24.365491+0000","last_clean":"2024-09-13T16:27:24.365491+0000","last_became_active":"2024-09-13T16:27:23.378890+0000","last_became_peered":"2024-09-13T16:27:23.378890+0000","last_unstale":"2024-09-13T16:27:24.365491+0000","last_undegraded":"2024-09-13T16:27:24.365491+0000","last_fullsized":"2024-09-13T16:27:24.365491+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:52:00.115221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478348+0000","last_change":"2024-09-13T16:27:23.374919+0000","last_active":"2024-09-13T16:27:24.478348+0000","last_peered":"2024-09-13T16:27:24.478348+0000","last_clean":"2024-09-13T16:27:24.478348+0000","last_became_active":"2024-09-13T16:27:23.374672+0000","last_became_peered":"2024-09-13T16:27:23.374672+0000","last_unstale":"2024-09-13T16:27:24.478348+0000","last_undegraded":"2024-09-13T16:27:24.478348+0000","last_fullsized":"2024-09-13T16:27:24.478348+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:53.425731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477270+0000","last_change":"2024-09-13T16:27:23.385522+0000","last_active":"2024-09-13T16:27:24.477270+0000","last_peered":"2024-09-13T16:27:24.477270+0000","last_clean":"2024-09-13T16:27:24.477270+0000","last_became_active":"2024-09-13T16:27:23.384442+0000","last_became_peered":"2024-09-13T16:27:23.384442+0000","last_unstale":"2024-09-13T16:27:24.477270+0000","last_undegraded":"2024-09-13T16:27:24.477270+0000","last_fullsized":"2024-09-13T16:27:24.477270+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:20.448150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365265+0000","last_change":"2024-09-13T16:27:23.379401+0000","last_active":"2024-09-13T16:27:24.365265+0000","last_peered":"2024-09-13T16:27:24.365265+0000","last_clean":"2024-09-13T16:27:24.365265+0000","last_became_active":"2024-09-13T16:27:23.379054+0000","last_became_peered":"2024-09-13T16:27:23.379054+0000","last_unstale":"2024-09-13T16:27:24.365265+0000","last_undegraded":"2024-09-13T16:27:24.365265+0000","last_fullsized":"2024-09-13T16:27:24.365265+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:23:08.681118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479810+0000","last_change":"2024-09-13T16:27:23.380441+0000","last_active":"2024-09-13T16:27:24.479810+0000","last_peered":"2024-09-13T16:27:24.479810+0000","last_clean":"2024-09-13T16:27:24.479810+0000","last_became_active":"2024-09-13T16:27:23.378324+0000","last_became_peered":"2024-09-13T16:27:23.378324+0000","last_unstale":"2024-09-13T16:27:24.479810+0000","last_undegraded":"2024-09-13T16:27:24.479810+0000","last_fullsized":"2024-09-13T16:27:24.479810+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:17.799047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476507+0000","last_change":"2024-09-13T16:27:23.383283+0000","last_active":"2024-09-13T16:27:24.476507+0000","last_peered":"2024-09-13T16:27:24.476507+0000","last_clean":"2024-09-13T16:27:24.476507+0000","last_became_active":"2024-09-13T16:27:23.383049+0000","last_became_peered":"2024-09-13T16:27:23.383049+0000","last_unstale":"2024-09-13T16:27:24.476507+0000","last_undegraded":"2024-09-13T16:27:24.476507+0000","last_fullsized":"2024-09-13T16:27:24.476507+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:09:38.166507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365169+0000","last_change":"2024-09-13T16:27:23.379103+0000","last_active":"2024-09-13T16:27:24.365169+0000","last_peered":"2024-09-13T16:27:24.365169+0000","last_clean":"2024-09-13T16:27:24.365169+0000","last_became_active":"2024-09-13T16:27:23.376237+0000","last_became_peered":"2024-09-13T16:27:23.376237+0000","last_unstale":"2024-09-13T16:27:24.365169+0000","last_undegraded":"2024-09-13T16:27:24.365169+0000","last_fullsized":"2024-09-13T16:27:24.365169+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:16.314918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479328+0000","last_change":"2024-09-13T16:27:23.382740+0000","last_active":"2024-09-13T16:27:24.479328+0000","last_peered":"2024-09-13T16:27:24.479328+0000","last_clean":"2024-09-13T16:27:24.479328+0000","last_became_active":"2024-09-13T16:27:23.373909+0000","last_became_peered":"2024-09-13T16:27:23.373909+0000","last_unstale":"2024-09-13T16:27:24.479328+0000","last_undegraded":"2024-09-13T16:27:24.479328+0000","last_fullsized":"2024-09-13T16:27:24.479328+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:20.448150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477605+0000","last_change":"2024-09-13T16:27:23.374060+0000","last_active":"2024-09-13T16:27:24.477605+0000","last_peered":"2024-09-13T16:27:24.477605+0000","last_clean":"2024-09-13T16:27:24.477605+0000","last_became_active":"2024-09-13T16:27:23.373821+0000","last_became_peered":"2024-09-13T16:27:23.373821+0000","last_unstale":"2024-09-13T16:27:24.477605+0000","last_undegraded":"2024-09-13T16:27:24.477605+0000","last_fullsized":"2024-09-13T16:27:24.477605+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:46.671292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478936+0000","last_change":"2024-09-13T16:27:23.383011+0000","last_active":"2024-09-13T16:27:24.478936+0000","last_peered":"2024-09-13T16:27:24.478936+0000","last_clean":"2024-09-13T16:27:24.478936+0000","last_became_active":"2024-09-13T16:27:23.382766+0000","last_became_peered":"2024-09-13T16:27:23.382766+0000","last_unstale":"2024-09-13T16:27:24.478936+0000","last_undegraded":"2024-09-13T16:27:24.478936+0000","last_fullsized":"2024-09-13T16:27:24.478936+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:52.782406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477263+0000","last_change":"2024-09-13T16:27:23.384016+0000","last_active":"2024-09-13T16:27:24.477263+0000","last_peered":"2024-09-13T16:27:24.477263+0000","last_clean":"2024-09-13T16:27:24.477263+0000","last_became_active":"2024-09-13T16:27:23.383804+0000","last_became_peered":"2024-09-13T16:27:23.383804+0000","last_unstale":"2024-09-13T16:27:24.477263+0000","last_undegraded":"2024-09-13T16:27:24.477263+0000","last_fullsized":"2024-09-13T16:27:24.477263+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:19:25.434377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477258+0000","last_change":"2024-09-13T16:27:23.384179+0000","last_active":"2024-09-13T16:27:24.477258+0000","last_peered":"2024-09-13T16:27:24.477258+0000","last_clean":"2024-09-13T16:27:24.477258+0000","last_became_active":"2024-09-13T16:27:23.382459+0000","last_became_peered":"2024-09-13T16:27:23.382459+0000","last_unstale":"2024-09-13T16:27:24.477258+0000","last_undegraded":"2024-09-13T16:27:24.477258+0000","last_fullsized":"2024-09-13T16:27:24.477258+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:37.579059+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477009+0000","last_change":"2024-09-13T16:27:23.374408+0000","last_active":"2024-09-13T16:27:24.477009+0000","last_peered":"2024-09-13T16:27:24.477009+0000","last_clean":"2024-09-13T16:27:24.477009+0000","last_became_active":"2024-09-13T16:27:23.374136+0000","last_became_peered":"2024-09-13T16:27:23.374136+0000","last_unstale":"2024-09-13T16:27:24.477009+0000","last_undegraded":"2024-09-13T16:27:24.477009+0000","last_fullsized":"2024-09-13T16:27:24.477009+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:04.691153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476397+0000","last_change":"2024-09-13T16:27:23.375744+0000","last_active":"2024-09-13T16:27:24.476397+0000","last_peered":"2024-09-13T16:27:24.476397+0000","last_clean":"2024-09-13T16:27:24.476397+0000","last_became_active":"2024-09-13T16:27:23.375476+0000","last_became_peered":"2024-09-13T16:27:23.375476+0000","last_unstale":"2024-09-13T16:27:24.476397+0000","last_undegraded":"2024-09-13T16:27:24.476397+0000","last_fullsized":"2024-09-13T16:27:24.476397+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:41.706820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478384+0000","last_change":"2024-09-13T16:27:23.379263+0000","last_active":"2024-09-13T16:27:24.478384+0000","last_peered":"2024-09-13T16:27:24.478384+0000","last_clean":"2024-09-13T16:27:24.478384+0000","last_became_active":"2024-09-13T16:27:23.379046+0000","last_became_peered":"2024-09-13T16:27:23.379046+0000","last_unstale":"2024-09-13T16:27:24.478384+0000","last_undegraded":"2024-09-13T16:27:24.478384+0000","last_fullsized":"2024-09-13T16:27:24.478384+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:37.008562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479476+0000","last_change":"2024-09-13T16:27:23.374210+0000","last_active":"2024-09-13T16:27:24.479476+0000","last_peered":"2024-09-13T16:27:24.479476+0000","last_clean":"2024-09-13T16:27:24.479476+0000","last_became_active":"2024-09-13T16:27:23.373630+0000","last_became_peered":"2024-09-13T16:27:23.373630+0000","last_unstale":"2024-09-13T16:27:24.479476+0000","last_undegraded":"2024-09-13T16:27:24.479476+0000","last_fullsized":"2024-09-13T16:27:24.479476+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:23:44.949955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476784+0000","last_change":"2024-09-13T16:27:23.377186+0000","last_active":"2024-09-13T16:27:24.476784+0000","last_peered":"2024-09-13T16:27:24.476784+0000","last_clean":"2024-09-13T16:27:24.476784+0000","last_became_active":"2024-09-13T16:27:23.376912+0000","last_became_peered":"2024-09-13T16:27:23.376912+0000","last_unstale":"2024-09-13T16:27:24.476784+0000","last_undegraded":"2024-09-13T16:27:24.476784+0000","last_fullsized":"2024-09-13T16:27:24.476784+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:52.782406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365816+0000","last_change":"2024-09-13T16:27:23.379591+0000","last_active":"2024-09-13T16:27:24.365816+0000","last_peered":"2024-09-13T16:27:24.365816+0000","last_clean":"2024-09-13T16:27:24.365816+0000","last_became_active":"2024-09-13T16:27:23.375905+0000","last_became_peered":"2024-09-13T16:27:23.375905+0000","last_unstale":"2024-09-13T16:27:24.365816+0000","last_undegraded":"2024-09-13T16:27:24.365816+0000","last_fullsized":"2024-09-13T16:27:24.365816+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:49:28.659828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477562+0000","last_change":"2024-09-13T16:27:23.384626+0000","last_active":"2024-09-13T16:27:24.477562+0000","last_peered":"2024-09-13T16:27:24.477562+0000","last_clean":"2024-09-13T16:27:24.477562+0000","last_became_active":"2024-09-13T16:27:23.383463+0000","last_became_peered":"2024-09-13T16:27:23.383463+0000","last_unstale":"2024-09-13T16:27:24.477562+0000","last_undegraded":"2024-09-13T16:27:24.477562+0000","last_fullsized":"2024-09-13T16:27:24.477562+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:11.707954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479609+0000","last_change":"2024-09-13T16:27:23.386052+0000","last_active":"2024-09-13T16:27:24.479609+0000","last_peered":"2024-09-13T16:27:24.479609+0000","last_clean":"2024-09-13T16:27:24.479609+0000","last_became_active":"2024-09-13T16:27:23.385689+0000","last_became_peered":"2024-09-13T16:27:23.385689+0000","last_unstale":"2024-09-13T16:27:24.479609+0000","last_undegraded":"2024-09-13T16:27:24.479609+0000","last_fullsized":"2024-09-13T16:27:24.479609+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:09:58.232141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476588+0000","last_change":"2024-09-13T16:27:23.383596+0000","last_active":"2024-09-13T16:27:24.476588+0000","last_peered":"2024-09-13T16:27:24.476588+0000","last_clean":"2024-09-13T16:27:24.476588+0000","last_became_active":"2024-09-13T16:27:23.383385+0000","last_became_peered":"2024-09-13T16:27:23.383385+0000","last_unstale":"2024-09-13T16:27:24.476588+0000","last_undegraded":"2024-09-13T16:27:24.476588+0000","last_fullsized":"2024-09-13T16:27:24.476588+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:35.884112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478930+0000","last_change":"2024-09-13T16:27:23.378806+0000","last_active":"2024-09-13T16:27:24.478930+0000","last_peered":"2024-09-13T16:27:24.478930+0000","last_clean":"2024-09-13T16:27:24.478930+0000","last_became_active":"2024-09-13T16:27:23.378573+0000","last_became_peered":"2024-09-13T16:27:23.378573+0000","last_unstale":"2024-09-13T16:27:24.478930+0000","last_undegraded":"2024-09-13T16:27:24.478930+0000","last_fullsized":"2024-09-13T16:27:24.478930+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:04.691153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.369983+0000","last_change":"2024-09-13T16:27:23.375183+0000","last_active":"2024-09-13T16:27:24.369983+0000","last_peered":"2024-09-13T16:27:24.369983+0000","last_clean":"2024-09-13T16:27:24.369983+0000","last_became_active":"2024-09-13T16:27:23.374577+0000","last_became_peered":"2024-09-13T16:27:23.374577+0000","last_unstale":"2024-09-13T16:27:24.369983+0000","last_undegraded":"2024-09-13T16:27:24.369983+0000","last_fullsized":"2024-09-13T16:27:24.369983+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:06.085384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476052+0000","last_change":"2024-09-13T16:27:23.372461+0000","last_active":"2024-09-13T16:27:24.476052+0000","last_peered":"2024-09-13T16:27:24.476052+0000","last_clean":"2024-09-13T16:27:24.476052+0000","last_became_active":"2024-09-13T16:27:23.372173+0000","last_became_peered":"2024-09-13T16:27:23.372173+0000","last_unstale":"2024-09-13T16:27:24.476052+0000","last_undegraded":"2024-09-13T16:27:24.476052+0000","last_fullsized":"2024-09-13T16:27:24.476052+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:58:47.695532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"53'4","reported_seq":15,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477344+0000","last_change":"2024-09-13T16:27:22.994441+0000","last_active":"2024-09-13T16:27:24.477344+0000","last_peered":"2024-09-13T16:27:24.477344+0000","last_clean":"2024-09-13T16:27:24.477344+0000","last_became_active":"2024-09-13T16:27:22.994119+0000","last_became_peered":"2024-09-13T16:27:22.994119+0000","last_unstale":"2024-09-13T16:27:24.477344+0000","last_undegraded":"2024-09-13T16:27:24.477344+0000","last_fullsized":"2024-09-13T16:27:24.477344+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:36.294462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370388+0000","last_change":"2024-09-13T16:27:23.372363+0000","last_active":"2024-09-13T16:27:24.370388+0000","last_peered":"2024-09-13T16:27:24.370388+0000","last_clean":"2024-09-13T16:27:24.370388+0000","last_became_active":"2024-09-13T16:27:23.372095+0000","last_became_peered":"2024-09-13T16:27:23.372095+0000","last_unstale":"2024-09-13T16:27:24.370388+0000","last_undegraded":"2024-09-13T16:27:24.370388+0000","last_fullsized":"2024-09-13T16:27:24.370388+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:05:53.024722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"53'1","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.366147+0000","last_change":"2024-09-13T16:27:22.997398+0000","last_active":"2024-09-13T16:27:24.366147+0000","last_peered":"2024-09-13T16:27:24.366147+0000","last_clean":"2024-09-13T16:27:24.366147+0000","last_became_active":"2024-09-13T16:27:22.997089+0000","last_became_peered":"2024-09-13T16:27:22.997089+0000","last_unstale":"2024-09-13T16:27:24.366147+0000","last_undegraded":"2024-09-13T16:27:24.366147+0000","last_fullsized":"2024-09-13T16:27:24.366147+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:44:17.459971+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477484+0000","last_change":"2024-09-13T16:27:23.382744+0000","last_active":"2024-09-13T16:27:24.477484+0000","last_peered":"2024-09-13T16:27:24.477484+0000","last_clean":"2024-09-13T16:27:24.477484+0000","last_became_active":"2024-09-13T16:27:23.382326+0000","last_became_peered":"2024-09-13T16:27:23.382326+0000","last_unstale":"2024-09-13T16:27:24.477484+0000","last_undegraded":"2024-09-13T16:27:24.477484+0000","last_fullsized":"2024-09-13T16:27:24.477484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:01.025811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"53'2","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365992+0000","last_change":"2024-09-13T16:27:23.003107+0000","last_active":"2024-09-13T16:27:24.365992+0000","last_peered":"2024-09-13T16:27:24.365992+0000","last_clean":"2024-09-13T16:27:24.365992+0000","last_became_active":"2024-09-13T16:27:23.002841+0000","last_became_peered":"2024-09-13T16:27:23.002841+0000","last_unstale":"2024-09-13T16:27:24.365992+0000","last_undegraded":"2024-09-13T16:27:24.365992+0000","last_fullsized":"2024-09-13T16:27:24.365992+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:57.806295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476989+0000","last_change":"2024-09-13T16:27:23.374643+0000","last_active":"2024-09-13T16:27:24.476989+0000","last_peered":"2024-09-13T16:27:24.476989+0000","last_clean":"2024-09-13T16:27:24.476989+0000","last_became_active":"2024-09-13T16:27:23.374378+0000","last_became_peered":"2024-09-13T16:27:23.374378+0000","last_unstale":"2024-09-13T16:27:24.476989+0000","last_undegraded":"2024-09-13T16:27:24.476989+0000","last_fullsized":"2024-09-13T16:27:24.476989+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:01.235504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"53'2","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370697+0000","last_change":"2024-09-13T16:27:22.997488+0000","last_active":"2024-09-13T16:27:24.370697+0000","last_peered":"2024-09-13T16:27:24.370697+0000","last_clean":"2024-09-13T16:27:24.370697+0000","last_became_active":"2024-09-13T16:27:22.997218+0000","last_became_peered":"2024-09-13T16:27:22.997218+0000","last_unstale":"2024-09-13T16:27:24.370697+0000","last_undegraded":"2024-09-13T16:27:24.370697+0000","last_fullsized":"2024-09-13T16:27:24.370697+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:21.846797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370665+0000","last_change":"2024-09-13T16:27:23.377004+0000","last_active":"2024-09-13T16:27:24.370665+0000","last_peered":"2024-09-13T16:27:24.370665+0000","last_clean":"2024-09-13T16:27:24.370665+0000","last_became_active":"2024-09-13T16:27:23.376476+0000","last_became_peered":"2024-09-13T16:27:23.376476+0000","last_unstale":"2024-09-13T16:27:24.370665+0000","last_undegraded":"2024-09-13T16:27:24.370665+0000","last_fullsized":"2024-09-13T16:27:24.370665+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:52:00.115221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"53'1","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479829+0000","last_change":"2024-09-13T16:27:22.991542+0000","last_active":"2024-09-13T16:27:24.479829+0000","last_peered":"2024-09-13T16:27:24.479829+0000","last_clean":"2024-09-13T16:27:24.479829+0000","last_became_active":"2024-09-13T16:27:22.991283+0000","last_became_peered":"2024-09-13T16:27:22.991283+0000","last_unstale":"2024-09-13T16:27:24.479829+0000","last_undegraded":"2024-09-13T16:27:24.479829+0000","last_fullsized":"2024-09-13T16:27:24.479829+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:13.093696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477362+0000","last_change":"2024-09-13T16:27:23.380432+0000","last_active":"2024-09-13T16:27:24.477362+0000","last_peered":"2024-09-13T16:27:24.477362+0000","last_clean":"2024-09-13T16:27:24.477362+0000","last_became_active":"2024-09-13T16:27:23.380053+0000","last_became_peered":"2024-09-13T16:27:23.380053+0000","last_unstale":"2024-09-13T16:27:24.477362+0000","last_undegraded":"2024-09-13T16:27:24.477362+0000","last_fullsized":"2024-09-13T16:27:24.477362+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:52:00.115221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"53'1","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370734+0000","last_change":"2024-09-13T16:27:22.992504+0000","last_active":"2024-09-13T16:27:24.370734+0000","last_peered":"2024-09-13T16:27:24.370734+0000","last_clean":"2024-09-13T16:27:24.370734+0000","last_became_active":"2024-09-13T16:27:22.992250+0000","last_became_peered":"2024-09-13T16:27:22.992250+0000","last_unstale":"2024-09-13T16:27:24.370734+0000","last_undegraded":"2024-09-13T16:27:24.370734+0000","last_fullsized":"2024-09-13T16:27:24.370734+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:00.016222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477025+0000","last_change":"2024-09-13T16:27:23.376850+0000","last_active":"2024-09-13T16:27:24.477025+0000","last_peered":"2024-09-13T16:27:24.477025+0000","last_clean":"2024-09-13T16:27:24.477025+0000","last_became_active":"2024-09-13T16:27:23.376608+0000","last_became_peered":"2024-09-13T16:27:23.376608+0000","last_unstale":"2024-09-13T16:27:24.477025+0000","last_undegraded":"2024-09-13T16:27:24.477025+0000","last_fullsized":"2024-09-13T16:27:24.477025+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:51.739786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476586+0000","last_change":"2024-09-13T16:27:22.992015+0000","last_active":"2024-09-13T16:27:24.476586+0000","last_peered":"2024-09-13T16:27:24.476586+0000","last_clean":"2024-09-13T16:27:24.476586+0000","last_became_active":"2024-09-13T16:27:22.991765+0000","last_became_peered":"2024-09-13T16:27:22.991765+0000","last_unstale":"2024-09-13T16:27:24.476586+0000","last_undegraded":"2024-09-13T16:27:24.476586+0000","last_fullsized":"2024-09-13T16:27:24.476586+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:29.703748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477740+0000","last_change":"2024-09-13T16:27:23.384216+0000","last_active":"2024-09-13T16:27:24.477740+0000","last_peered":"2024-09-13T16:27:24.477740+0000","last_clean":"2024-09-13T16:27:24.477740+0000","last_became_active":"2024-09-13T16:27:23.383943+0000","last_became_peered":"2024-09-13T16:27:23.383943+0000","last_unstale":"2024-09-13T16:27:24.477740+0000","last_undegraded":"2024-09-13T16:27:24.477740+0000","last_fullsized":"2024-09-13T16:27:24.477740+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:21:33.025697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478488+0000","last_change":"2024-09-13T16:27:23.002761+0000","last_active":"2024-09-13T16:27:24.478488+0000","last_peered":"2024-09-13T16:27:24.478488+0000","last_clean":"2024-09-13T16:27:24.478488+0000","last_became_active":"2024-09-13T16:27:23.002462+0000","last_became_peered":"2024-09-13T16:27:23.002462+0000","last_unstale":"2024-09-13T16:27:24.478488+0000","last_undegraded":"2024-09-13T16:27:24.478488+0000","last_fullsized":"2024-09-13T16:27:24.478488+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:08:46.457428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370350+0000","last_change":"2024-09-13T16:27:23.374415+0000","last_active":"2024-09-13T16:27:24.370350+0000","last_peered":"2024-09-13T16:27:24.370350+0000","last_clean":"2024-09-13T16:27:24.370350+0000","last_became_active":"2024-09-13T16:27:23.374149+0000","last_became_peered":"2024-09-13T16:27:23.374149+0000","last_unstale":"2024-09-13T16:27:24.370350+0000","last_undegraded":"2024-09-13T16:27:24.370350+0000","last_fullsized":"2024-09-13T16:27:24.370350+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:02:46.671292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477859+0000","last_change":"2024-09-13T16:27:22.993500+0000","last_active":"2024-09-13T16:27:24.477859+0000","last_peered":"2024-09-13T16:27:24.477859+0000","last_clean":"2024-09-13T16:27:24.477859+0000","last_became_active":"2024-09-13T16:27:22.993175+0000","last_became_peered":"2024-09-13T16:27:22.993175+0000","last_unstale":"2024-09-13T16:27:24.477859+0000","last_undegraded":"2024-09-13T16:27:24.477859+0000","last_fullsized":"2024-09-13T16:27:24.477859+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:31.016369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370326+0000","last_change":"2024-09-13T16:27:23.372331+0000","last_active":"2024-09-13T16:27:24.370326+0000","last_peered":"2024-09-13T16:27:24.370326+0000","last_clean":"2024-09-13T16:27:24.370326+0000","last_became_active":"2024-09-13T16:27:23.372027+0000","last_became_peered":"2024-09-13T16:27:23.372027+0000","last_unstale":"2024-09-13T16:27:24.370326+0000","last_undegraded":"2024-09-13T16:27:24.370326+0000","last_fullsized":"2024-09-13T16:27:24.370326+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:24.771944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.371112+0000","last_change":"2024-09-13T16:27:22.989123+0000","last_active":"2024-09-13T16:27:24.371112+0000","last_peered":"2024-09-13T16:27:24.371112+0000","last_clean":"2024-09-13T16:27:24.371112+0000","last_became_active":"2024-09-13T16:27:22.988687+0000","last_became_peered":"2024-09-13T16:27:22.988687+0000","last_unstale":"2024-09-13T16:27:24.371112+0000","last_undegraded":"2024-09-13T16:27:24.371112+0000","last_fullsized":"2024-09-13T16:27:24.371112+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:00:36.569470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479923+0000","last_change":"2024-09-13T16:27:23.385971+0000","last_active":"2024-09-13T16:27:24.479923+0000","last_peered":"2024-09-13T16:27:24.479923+0000","last_clean":"2024-09-13T16:27:24.479923+0000","last_became_active":"2024-09-13T16:27:23.384514+0000","last_became_peered":"2024-09-13T16:27:23.384514+0000","last_unstale":"2024-09-13T16:27:24.479923+0000","last_undegraded":"2024-09-13T16:27:24.479923+0000","last_fullsized":"2024-09-13T16:27:24.479923+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:50:23.866298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"53'3","reported_seq":14,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477677+0000","last_change":"2024-09-13T16:27:23.004177+0000","last_active":"2024-09-13T16:27:24.477677+0000","last_peered":"2024-09-13T16:27:24.477677+0000","last_clean":"2024-09-13T16:27:24.477677+0000","last_became_active":"2024-09-13T16:27:23.003887+0000","last_became_peered":"2024-09-13T16:27:23.003887+0000","last_unstale":"2024-09-13T16:27:24.477677+0000","last_undegraded":"2024-09-13T16:27:24.477677+0000","last_fullsized":"2024-09-13T16:27:24.477677+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:48.590434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.371035+0000","last_change":"2024-09-13T16:27:23.376151+0000","last_active":"2024-09-13T16:27:24.371035+0000","last_peered":"2024-09-13T16:27:24.371035+0000","last_clean":"2024-09-13T16:27:24.371035+0000","last_became_active":"2024-09-13T16:27:23.375850+0000","last_became_peered":"2024-09-13T16:27:23.375850+0000","last_unstale":"2024-09-13T16:27:24.371035+0000","last_undegraded":"2024-09-13T16:27:24.371035+0000","last_fullsized":"2024-09-13T16:27:24.371035+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:05.896471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"53'6","reported_seq":17,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477488+0000","last_change":"2024-09-13T16:27:22.988513+0000","last_active":"2024-09-13T16:27:24.477488+0000","last_peered":"2024-09-13T16:27:24.477488+0000","last_clean":"2024-09-13T16:27:24.477488+0000","last_became_active":"2024-09-13T16:27:22.988103+0000","last_became_peered":"2024-09-13T16:27:22.988103+0000","last_unstale":"2024-09-13T16:27:24.477488+0000","last_undegraded":"2024-09-13T16:27:24.477488+0000","last_fullsized":"2024-09-13T16:27:24.477488+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:53:06.090167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370295+0000","last_change":"2024-09-13T16:27:23.375670+0000","last_active":"2024-09-13T16:27:24.370295+0000","last_peered":"2024-09-13T16:27:24.370295+0000","last_clean":"2024-09-13T16:27:24.370295+0000","last_became_active":"2024-09-13T16:27:23.375440+0000","last_became_peered":"2024-09-13T16:27:23.375440+0000","last_unstale":"2024-09-13T16:27:24.370295+0000","last_undegraded":"2024-09-13T16:27:24.370295+0000","last_fullsized":"2024-09-13T16:27:24.370295+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:20:23.661732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"53'2","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.480039+0000","last_change":"2024-09-13T16:27:23.002597+0000","last_active":"2024-09-13T16:27:24.480039+0000","last_peered":"2024-09-13T16:27:24.480039+0000","last_clean":"2024-09-13T16:27:24.480039+0000","last_became_active":"2024-09-13T16:27:23.002350+0000","last_became_peered":"2024-09-13T16:27:23.002350+0000","last_unstale":"2024-09-13T16:27:24.480039+0000","last_undegraded":"2024-09-13T16:27:24.480039+0000","last_fullsized":"2024-09-13T16:27:24.480039+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:52:35.062040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479176+0000","last_change":"2024-09-13T16:27:23.371625+0000","last_active":"2024-09-13T16:27:24.479176+0000","last_peered":"2024-09-13T16:27:24.479176+0000","last_clean":"2024-09-13T16:27:24.479176+0000","last_became_active":"2024-09-13T16:27:23.371379+0000","last_became_peered":"2024-09-13T16:27:23.371379+0000","last_unstale":"2024-09-13T16:27:24.479176+0000","last_undegraded":"2024-09-13T16:27:24.479176+0000","last_fullsized":"2024-09-13T16:27:24.479176+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:08.223466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477109+0000","last_change":"2024-09-13T16:27:23.008788+0000","last_active":"2024-09-13T16:27:24.477109+0000","last_peered":"2024-09-13T16:27:24.477109+0000","last_clean":"2024-09-13T16:27:24.477109+0000","last_became_active":"2024-09-13T16:27:23.008485+0000","last_became_peered":"2024-09-13T16:27:23.008485+0000","last_unstale":"2024-09-13T16:27:24.477109+0000","last_undegraded":"2024-09-13T16:27:24.477109+0000","last_fullsized":"2024-09-13T16:27:24.477109+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:21:53.096830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476885+0000","last_change":"2024-09-13T16:27:23.372859+0000","last_active":"2024-09-13T16:27:24.476885+0000","last_peered":"2024-09-13T16:27:24.476885+0000","last_clean":"2024-09-13T16:27:24.476885+0000","last_became_active":"2024-09-13T16:27:23.372588+0000","last_became_peered":"2024-09-13T16:27:23.372588+0000","last_unstale":"2024-09-13T16:27:24.476885+0000","last_undegraded":"2024-09-13T16:27:24.476885+0000","last_fullsized":"2024-09-13T16:27:24.476885+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:22.725944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476610+0000","last_change":"2024-09-13T16:27:22.994399+0000","last_active":"2024-09-13T16:27:24.476610+0000","last_peered":"2024-09-13T16:27:24.476610+0000","last_clean":"2024-09-13T16:27:24.476610+0000","last_became_active":"2024-09-13T16:27:22.994139+0000","last_became_peered":"2024-09-13T16:27:22.994139+0000","last_unstale":"2024-09-13T16:27:24.476610+0000","last_undegraded":"2024-09-13T16:27:24.476610+0000","last_fullsized":"2024-09-13T16:27:24.476610+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:54.668814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479408+0000","last_change":"2024-09-13T16:27:23.372770+0000","last_active":"2024-09-13T16:27:24.479408+0000","last_peered":"2024-09-13T16:27:24.479408+0000","last_clean":"2024-09-13T16:27:24.479408+0000","last_became_active":"2024-09-13T16:27:23.372519+0000","last_became_peered":"2024-09-13T16:27:23.372519+0000","last_unstale":"2024-09-13T16:27:24.479408+0000","last_undegraded":"2024-09-13T16:27:24.479408+0000","last_fullsized":"2024-09-13T16:27:24.479408+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:45.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"53'8","reported_seq":19,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477025+0000","last_change":"2024-09-13T16:27:22.987812+0000","last_active":"2024-09-13T16:27:24.477025+0000","last_peered":"2024-09-13T16:27:24.477025+0000","last_clean":"2024-09-13T16:27:24.477025+0000","last_became_active":"2024-09-13T16:27:22.986679+0000","last_became_peered":"2024-09-13T16:27:22.986679+0000","last_unstale":"2024-09-13T16:27:24.477025+0000","last_undegraded":"2024-09-13T16:27:24.477025+0000","last_fullsized":"2024-09-13T16:27:24.477025+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:13.093696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476417+0000","last_change":"2024-09-13T16:27:23.377964+0000","last_active":"2024-09-13T16:27:24.476417+0000","last_peered":"2024-09-13T16:27:24.476417+0000","last_clean":"2024-09-13T16:27:24.476417+0000","last_became_active":"2024-09-13T16:27:23.377534+0000","last_became_peered":"2024-09-13T16:27:23.377534+0000","last_unstale":"2024-09-13T16:27:24.476417+0000","last_undegraded":"2024-09-13T16:27:24.476417+0000","last_fullsized":"2024-09-13T16:27:24.476417+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:43:45.090948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"53'1","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479620+0000","last_change":"2024-09-13T16:27:23.003854+0000","last_active":"2024-09-13T16:27:24.479620+0000","last_peered":"2024-09-13T16:27:24.479620+0000","last_clean":"2024-09-13T16:27:24.479620+0000","last_became_active":"2024-09-13T16:27:23.003605+0000","last_became_peered":"2024-09-13T16:27:23.003605+0000","last_unstale":"2024-09-13T16:27:24.479620+0000","last_undegraded":"2024-09-13T16:27:24.479620+0000","last_fullsized":"2024-09-13T16:27:24.479620+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365098+0000","last_change":"2024-09-13T16:27:23.376352+0000","last_active":"2024-09-13T16:27:24.365098+0000","last_peered":"2024-09-13T16:27:24.365098+0000","last_clean":"2024-09-13T16:27:24.365098+0000","last_became_active":"2024-09-13T16:27:23.375836+0000","last_became_peered":"2024-09-13T16:27:23.375836+0000","last_unstale":"2024-09-13T16:27:24.365098+0000","last_undegraded":"2024-09-13T16:27:24.365098+0000","last_fullsized":"2024-09-13T16:27:24.365098+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:35.884112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"53'2","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478513+0000","last_change":"2024-09-13T16:27:22.993390+0000","last_active":"2024-09-13T16:27:24.478513+0000","last_peered":"2024-09-13T16:27:24.478513+0000","last_clean":"2024-09-13T16:27:24.478513+0000","last_became_active":"2024-09-13T16:27:22.993042+0000","last_became_peered":"2024-09-13T16:27:22.993042+0000","last_unstale":"2024-09-13T16:27:24.478513+0000","last_undegraded":"2024-09-13T16:27:24.478513+0000","last_fullsized":"2024-09-13T16:27:24.478513+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:41:25.763836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370648+0000","last_change":"2024-09-13T16:27:23.376645+0000","last_active":"2024-09-13T16:27:24.370648+0000","last_peered":"2024-09-13T16:27:24.370648+0000","last_clean":"2024-09-13T16:27:24.370648+0000","last_became_active":"2024-09-13T16:27:23.376362+0000","last_became_peered":"2024-09-13T16:27:23.376362+0000","last_unstale":"2024-09-13T16:27:24.370648+0000","last_undegraded":"2024-09-13T16:27:24.370648+0000","last_fullsized":"2024-09-13T16:27:24.370648+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:35.884112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"53'1","reported_seq":12,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477734+0000","last_change":"2024-09-13T16:27:23.003494+0000","last_active":"2024-09-13T16:27:24.477734+0000","last_peered":"2024-09-13T16:27:24.477734+0000","last_clean":"2024-09-13T16:27:24.477734+0000","last_became_active":"2024-09-13T16:27:23.003258+0000","last_became_peered":"2024-09-13T16:27:23.003258+0000","last_unstale":"2024-09-13T16:27:24.477734+0000","last_undegraded":"2024-09-13T16:27:24.477734+0000","last_fullsized":"2024-09-13T16:27:24.477734+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:23.907543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365384+0000","last_change":"2024-09-13T16:27:23.379003+0000","last_active":"2024-09-13T16:27:24.365384+0000","last_peered":"2024-09-13T16:27:24.365384+0000","last_clean":"2024-09-13T16:27:24.365384+0000","last_became_active":"2024-09-13T16:27:23.375836+0000","last_became_peered":"2024-09-13T16:27:23.375836+0000","last_unstale":"2024-09-13T16:27:24.365384+0000","last_undegraded":"2024-09-13T16:27:24.365384+0000","last_fullsized":"2024-09-13T16:27:24.365384+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:30.655214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477551+0000","last_change":"2024-09-13T16:27:22.997657+0000","last_active":"2024-09-13T16:27:24.477551+0000","last_peered":"2024-09-13T16:27:24.477551+0000","last_clean":"2024-09-13T16:27:24.477551+0000","last_became_active":"2024-09-13T16:27:22.997330+0000","last_became_peered":"2024-09-13T16:27:22.997330+0000","last_unstale":"2024-09-13T16:27:24.477551+0000","last_undegraded":"2024-09-13T16:27:24.477551+0000","last_fullsized":"2024-09-13T16:27:24.477551+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:02:18.201994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370534+0000","last_change":"2024-09-13T16:27:23.377195+0000","last_active":"2024-09-13T16:27:24.370534+0000","last_peered":"2024-09-13T16:27:24.370534+0000","last_clean":"2024-09-13T16:27:24.370534+0000","last_became_active":"2024-09-13T16:27:23.376831+0000","last_became_peered":"2024-09-13T16:27:23.376831+0000","last_unstale":"2024-09-13T16:27:24.370534+0000","last_undegraded":"2024-09-13T16:27:24.370534+0000","last_fullsized":"2024-09-13T16:27:24.370534+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:43.515028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"53'2","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365357+0000","last_change":"2024-09-13T16:27:23.002572+0000","last_active":"2024-09-13T16:27:24.365357+0000","last_peered":"2024-09-13T16:27:24.365357+0000","last_clean":"2024-09-13T16:27:24.365357+0000","last_became_active":"2024-09-13T16:27:23.002316+0000","last_became_peered":"2024-09-13T16:27:23.002316+0000","last_unstale":"2024-09-13T16:27:24.365357+0000","last_undegraded":"2024-09-13T16:27:24.365357+0000","last_fullsized":"2024-09-13T16:27:24.365357+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:48.590434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477519+0000","last_change":"2024-09-13T16:27:23.384828+0000","last_active":"2024-09-13T16:27:24.477519+0000","last_peered":"2024-09-13T16:27:24.477519+0000","last_clean":"2024-09-13T16:27:24.477519+0000","last_became_active":"2024-09-13T16:27:23.382352+0000","last_became_peered":"2024-09-13T16:27:23.382352+0000","last_unstale":"2024-09-13T16:27:24.477519+0000","last_undegraded":"2024-09-13T16:27:24.477519+0000","last_fullsized":"2024-09-13T16:27:24.477519+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:40:50.233420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476731+0000","last_change":"2024-09-13T16:27:22.987769+0000","last_active":"2024-09-13T16:27:24.476731+0000","last_peered":"2024-09-13T16:27:24.476731+0000","last_clean":"2024-09-13T16:27:24.476731+0000","last_became_active":"2024-09-13T16:27:22.987385+0000","last_became_peered":"2024-09-13T16:27:22.987385+0000","last_unstale":"2024-09-13T16:27:24.476731+0000","last_undegraded":"2024-09-13T16:27:24.476731+0000","last_fullsized":"2024-09-13T16:27:24.476731+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:03.282533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478945+0000","last_change":"2024-09-13T16:27:23.386040+0000","last_active":"2024-09-13T16:27:24.478945+0000","last_peered":"2024-09-13T16:27:24.478945+0000","last_clean":"2024-09-13T16:27:24.478945+0000","last_became_active":"2024-09-13T16:27:23.385668+0000","last_became_peered":"2024-09-13T16:27:23.385668+0000","last_unstale":"2024-09-13T16:27:24.478945+0000","last_undegraded":"2024-09-13T16:27:24.478945+0000","last_fullsized":"2024-09-13T16:27:24.478945+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:30:03.621588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.479320+0000","last_change":"2024-09-13T16:27:23.003175+0000","last_active":"2024-09-13T16:27:24.479320+0000","last_peered":"2024-09-13T16:27:24.479320+0000","last_clean":"2024-09-13T16:27:24.479320+0000","last_became_active":"2024-09-13T16:27:23.002927+0000","last_became_peered":"2024-09-13T16:27:23.002927+0000","last_unstale":"2024-09-13T16:27:24.479320+0000","last_undegraded":"2024-09-13T16:27:24.479320+0000","last_fullsized":"2024-09-13T16:27:24.479320+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:03:37.395226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.366273+0000","last_change":"2024-09-13T16:27:23.380126+0000","last_active":"2024-09-13T16:27:24.366273+0000","last_peered":"2024-09-13T16:27:24.366273+0000","last_clean":"2024-09-13T16:27:24.366273+0000","last_became_active":"2024-09-13T16:27:23.379787+0000","last_became_peered":"2024-09-13T16:27:23.379787+0000","last_unstale":"2024-09-13T16:27:24.366273+0000","last_undegraded":"2024-09-13T16:27:24.366273+0000","last_fullsized":"2024-09-13T16:27:24.366273+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:01:42.176382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476044+0000","last_change":"2024-09-13T16:27:22.992395+0000","last_active":"2024-09-13T16:27:24.476044+0000","last_peered":"2024-09-13T16:27:24.476044+0000","last_clean":"2024-09-13T16:27:24.476044+0000","last_became_active":"2024-09-13T16:27:22.992051+0000","last_became_peered":"2024-09-13T16:27:22.992051+0000","last_unstale":"2024-09-13T16:27:24.476044+0000","last_undegraded":"2024-09-13T16:27:24.476044+0000","last_fullsized":"2024-09-13T16:27:24.476044+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:56:39.167909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370264+0000","last_change":"2024-09-13T16:27:23.374967+0000","last_active":"2024-09-13T16:27:24.370264+0000","last_peered":"2024-09-13T16:27:24.370264+0000","last_clean":"2024-09-13T16:27:24.370264+0000","last_became_active":"2024-09-13T16:27:23.374234+0000","last_became_peered":"2024-09-13T16:27:23.374234+0000","last_unstale":"2024-09-13T16:27:24.370264+0000","last_undegraded":"2024-09-13T16:27:24.370264+0000","last_fullsized":"2024-09-13T16:27:24.370264+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:45.853570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"53'2","reported_seq":13,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478064+0000","last_change":"2024-09-13T16:27:23.008369+0000","last_active":"2024-09-13T16:27:24.478064+0000","last_peered":"2024-09-13T16:27:24.478064+0000","last_clean":"2024-09-13T16:27:24.478064+0000","last_became_active":"2024-09-13T16:27:23.008089+0000","last_became_peered":"2024-09-13T16:27:23.008089+0000","last_unstale":"2024-09-13T16:27:24.478064+0000","last_undegraded":"2024-09-13T16:27:24.478064+0000","last_fullsized":"2024-09-13T16:27:24.478064+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:40.525510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.365677+0000","last_change":"2024-09-13T16:27:23.379051+0000","last_active":"2024-09-13T16:27:24.365677+0000","last_peered":"2024-09-13T16:27:24.365677+0000","last_clean":"2024-09-13T16:27:24.365677+0000","last_became_active":"2024-09-13T16:27:23.376340+0000","last_became_peered":"2024-09-13T16:27:23.376340+0000","last_unstale":"2024-09-13T16:27:24.365677+0000","last_undegraded":"2024-09-13T16:27:24.365677+0000","last_fullsized":"2024-09-13T16:27:24.365677+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:34:19.345148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477795+0000","last_change":"2024-09-13T16:27:22.995090+0000","last_active":"2024-09-13T16:27:24.477795+0000","last_peered":"2024-09-13T16:27:24.477795+0000","last_clean":"2024-09-13T16:27:24.477795+0000","last_became_active":"2024-09-13T16:27:22.994754+0000","last_became_peered":"2024-09-13T16:27:22.994754+0000","last_unstale":"2024-09-13T16:27:24.477795+0000","last_undegraded":"2024-09-13T16:27:24.477795+0000","last_fullsized":"2024-09-13T16:27:24.477795+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:55:57.806295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477170+0000","last_change":"2024-09-13T16:27:23.377684+0000","last_active":"2024-09-13T16:27:24.477170+0000","last_peered":"2024-09-13T16:27:24.477170+0000","last_clean":"2024-09-13T16:27:24.477170+0000","last_became_active":"2024-09-13T16:27:23.377367+0000","last_became_peered":"2024-09-13T16:27:23.377367+0000","last_unstale":"2024-09-13T16:27:24.477170+0000","last_undegraded":"2024-09-13T16:27:24.477170+0000","last_fullsized":"2024-09-13T16:27:24.477170+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:37.008562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.480226+0000","last_change":"2024-09-13T16:27:22.998293+0000","last_active":"2024-09-13T16:27:24.480226+0000","last_peered":"2024-09-13T16:27:24.480226+0000","last_clean":"2024-09-13T16:27:24.480226+0000","last_became_active":"2024-09-13T16:27:22.997620+0000","last_became_peered":"2024-09-13T16:27:22.997620+0000","last_unstale":"2024-09-13T16:27:24.480226+0000","last_undegraded":"2024-09-13T16:27:24.480226+0000","last_fullsized":"2024-09-13T16:27:24.480226+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:20:01.691513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.480133+0000","last_change":"2024-09-13T16:27:23.373392+0000","last_active":"2024-09-13T16:27:24.480133+0000","last_peered":"2024-09-13T16:27:24.480133+0000","last_clean":"2024-09-13T16:27:24.480133+0000","last_became_active":"2024-09-13T16:27:23.373129+0000","last_became_peered":"2024-09-13T16:27:23.373129+0000","last_unstale":"2024-09-13T16:27:24.480133+0000","last_undegraded":"2024-09-13T16:27:24.480133+0000","last_fullsized":"2024-09-13T16:27:24.480133+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:23:53.559004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477277+0000","last_change":"2024-09-13T16:27:22.988474+0000","last_active":"2024-09-13T16:27:24.477277+0000","last_peered":"2024-09-13T16:27:24.477277+0000","last_clean":"2024-09-13T16:27:24.477277+0000","last_became_active":"2024-09-13T16:27:22.987560+0000","last_became_peered":"2024-09-13T16:27:22.987560+0000","last_unstale":"2024-09-13T16:27:24.477277+0000","last_undegraded":"2024-09-13T16:27:24.477277+0000","last_fullsized":"2024-09-13T16:27:24.477277+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:37.543662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370267+0000","last_change":"2024-09-13T16:27:23.374375+0000","last_active":"2024-09-13T16:27:24.370267+0000","last_peered":"2024-09-13T16:27:24.370267+0000","last_clean":"2024-09-13T16:27:24.370267+0000","last_became_active":"2024-09-13T16:27:23.374082+0000","last_became_peered":"2024-09-13T16:27:23.374082+0000","last_unstale":"2024-09-13T16:27:24.370267+0000","last_undegraded":"2024-09-13T16:27:24.370267+0000","last_fullsized":"2024-09-13T16:27:24.370267+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:09:35.690674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476881+0000","last_change":"2024-09-13T16:27:22.999213+0000","last_active":"2024-09-13T16:27:24.476881+0000","last_peered":"2024-09-13T16:27:24.476881+0000","last_clean":"2024-09-13T16:27:24.476881+0000","last_became_active":"2024-09-13T16:27:22.998909+0000","last_became_peered":"2024-09-13T16:27:22.998909+0000","last_unstale":"2024-09-13T16:27:24.476881+0000","last_undegraded":"2024-09-13T16:27:24.476881+0000","last_fullsized":"2024-09-13T16:27:24.476881+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:04:07.776688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478662+0000","last_change":"2024-09-13T16:27:23.380813+0000","last_active":"2024-09-13T16:27:24.478662+0000","last_peered":"2024-09-13T16:27:24.478662+0000","last_clean":"2024-09-13T16:27:24.478662+0000","last_became_active":"2024-09-13T16:27:23.380578+0000","last_became_peered":"2024-09-13T16:27:23.380578+0000","last_unstale":"2024-09-13T16:27:24.478662+0000","last_undegraded":"2024-09-13T16:27:24.478662+0000","last_fullsized":"2024-09-13T16:27:24.478662+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:22:50.880765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370515+0000","last_change":"2024-09-13T16:27:22.988309+0000","last_active":"2024-09-13T16:27:24.370515+0000","last_peered":"2024-09-13T16:27:24.370515+0000","last_clean":"2024-09-13T16:27:24.370515+0000","last_became_active":"2024-09-13T16:27:22.987921+0000","last_became_peered":"2024-09-13T16:27:22.987921+0000","last_unstale":"2024-09-13T16:27:24.370515+0000","last_undegraded":"2024-09-13T16:27:24.370515+0000","last_fullsized":"2024-09-13T16:27:24.370515+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:55.066814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.477446+0000","last_change":"2024-09-13T16:27:23.380048+0000","last_active":"2024-09-13T16:27:24.477446+0000","last_peered":"2024-09-13T16:27:24.477446+0000","last_clean":"2024-09-13T16:27:24.477446+0000","last_became_active":"2024-09-13T16:27:23.379730+0000","last_became_peered":"2024-09-13T16:27:23.379730+0000","last_unstale":"2024-09-13T16:27:24.477446+0000","last_undegraded":"2024-09-13T16:27:24.477446+0000","last_fullsized":"2024-09-13T16:27:24.477446+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:15:55.539411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.370139+0000","last_change":"2024-09-13T16:27:23.004012+0000","last_active":"2024-09-13T16:27:24.370139+0000","last_peered":"2024-09-13T16:27:24.370139+0000","last_clean":"2024-09-13T16:27:24.370139+0000","last_became_active":"2024-09-13T16:27:23.003648+0000","last_became_peered":"2024-09-13T16:27:23.003648+0000","last_unstale":"2024-09-13T16:27:24.370139+0000","last_undegraded":"2024-09-13T16:27:24.370139+0000","last_fullsized":"2024-09-13T16:27:24.370139+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:57:01.285938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":10,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.476484+0000","last_change":"2024-09-13T16:27:23.383763+0000","last_active":"2024-09-13T16:27:24.476484+0000","last_peered":"2024-09-13T16:27:24.476484+0000","last_clean":"2024-09-13T16:27:24.476484+0000","last_became_active":"2024-09-13T16:27:23.383476+0000","last_became_peered":"2024-09-13T16:27:23.383476+0000","last_unstale":"2024-09-13T16:27:24.476484+0000","last_undegraded":"2024-09-13T16:27:24.476484+0000","last_fullsized":"2024-09-13T16:27:24.476484+0000","mapping_epoch":52,"log_start":"0'0","ondisk_log_start":"0'0","created":52,"last_epoch_clean":53,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:22.963736+0000","last_clean_scrub_stamp":"2024-09-13T16:27:22.963736+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:56.076403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":54,"state":"active+clean","last_fresh":"2024-09-13T16:27:24.478944+0000","last_change":"2024-09-13T16:27:23.002815+0000","last_active":"2024-09-13T16:27:24.478944+0000","last_peered":"2024-09-13T16:27:24.478944+0000","last_clean":"2024-09-13T16:27:24.478944+0000","last_became_active":"2024-09-13T16:27:23.002573+0000","last_became_peered":"2024-09-13T16:27:23.002573+0000","last_unstale":"2024-09-13T16:27:24.478944+0000","last_undegraded":"2024-09-13T16:27:24.478944+0000","last_fullsized":"2024-09-13T16:27:24.478944+0000","mapping_epoch":51,"log_start":"0'0","ondisk_log_start":"0'0","created":51,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:27:21.954146+0000","last_clean_scrub_stamp":"2024-09-13T16:27:21.954146+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:48:37.543662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":191,"reported_epoch":53,"state":"active+clean","last_fresh":"2024-09-13T16:27:23.366561+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:27:23.366561+0000","last_peered":"2024-09-13T16:27:23.366561+0000","last_clean":"2024-09-13T16:27:23.366561+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:27:23.366561+0000","last_undegraded":"2024-09-13T16:27:23.366561+0000","last_fullsized":"2024-09-13T16:27:23.366561+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542174,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1241088,"data_stored":872795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.87,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.2010000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":1.111},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.85999999999999999,"15min":0.84799999999999998},"min":{"1min":0.65100000000000002,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.3460000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.84999999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.85299999999999998,"15min":0.85499999999999998},"min":{"1min":0.67300000000000004,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0760000000000001,"5min":1.121,"15min":1.121},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.85399999999999998,"15min":0.84599999999999997},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.92000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.89000000000000001,"15min":0.88700000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.0880000000000001,"5min":1.228,"15min":1.228},"last":1.2170000000000001},{"interface":"front","average":{"1min":0.84099999999999997,"5min":0.83299999999999996,"15min":0.83099999999999996},"min":{"1min":0.64500000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.0640000000000001,"5min":1.091,"15min":1.091},"last":0.80800000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.02,"15min":1.02},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"max":{"1min":1.228,"5min":1.238,"15min":1.238},"last":1.014},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.0049999999999999,"15min":1.0089999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.2190000000000001,"5min":1.226,"15min":1.226},"last":0.89700000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.83099999999999996,"5min":0.88300000000000001,"15min":0.89200000000000002},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.05,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0880000000000001,"15min":1.0980000000000001},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1830000000000001,"5min":1.589,"15min":1.589},"last":1.171}]},{"osd":5,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":1.024,"15min":1.0389999999999999},"min":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.1799999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.119,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":1.0249999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.2170000000000001,"5min":1.337,"15min":1.337},"last":1.0569999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:14 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.0880000000000001,"15min":1.0900000000000001},"min":{"1min":0.84299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.22,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.98899999999999999},{"interface":"front","average":{"1min":1.133,"5min":1.181,"15min":1.1890000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.751},"max":{"1min":1.4199999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.2589999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542173,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27652,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344188,"statfs":{"total":96636764160,"available":96608448512,"internally_reserved":0,"allocated":647168,"data_stored":282955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.83699999999999997,"15min":0.84899999999999998},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.1220000000000001,"5min":1.153,"15min":1.153},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.878,"5min":0.89300000000000002,"15min":0.89500000000000002},"min":{"1min":0.56799999999999995,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.0660000000000001,"5min":1.143,"15min":1.143},"last":1.4410000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.83399999999999996,"15min":0.83799999999999997},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0229999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.90400000000000003,"15min":0.91100000000000003},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.0669999999999999,"5min":1.296,"15min":1.296},"last":0.80800000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.88100000000000001,"15min":0.88700000000000001},"min":{"1min":0.65400000000000003,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.0449999999999999,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.84899999999999998,"5min":0.84399999999999997,"15min":0.84299999999999997},"min":{"1min":0.60299999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1870000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.94299999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69699999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.264,"5min":1.536,"15min":1.536},"last":1.002},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98299999999999998,"15min":0.98799999999999999},"min":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1819999999999999,"5min":1.409,"15min":1.409},"last":1.264}]},{"osd":4,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.97999999999999998,"15min":0.98299999999999998},"min":{"1min":0.66300000000000003,"5min":0.628,"15min":0.628},"max":{"1min":1.216,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":1.028,"5min":0.94799999999999995,"15min":0.93500000000000005},"min":{"1min":0.81200000000000006,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1719999999999999,"5min":1.224,"15min":1.224},"last":1.6140000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.91500000000000004,"15min":0.91400000000000003},"min":{"1min":0.73099999999999998,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":1.21,"5min":1.21,"15min":1.21},"last":1.2110000000000001},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.1379999999999999,"15min":1.143},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.329,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.099}]},{"osd":7,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.1459999999999999,"5min":1.129,"15min":1.1259999999999999},"min":{"1min":0.90300000000000002,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.4319999999999999,"5min":1.5029999999999999,"15min":1.5029999999999999},"last":1.522},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.879,"5min":0.86799999999999999,"15min":0.86799999999999999},"max":{"1min":1.337,"5min":1.337,"15min":1.337},"last":1.1339999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542175,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31748,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94340092,"statfs":{"total":96636764160,"available":96604254208,"internally_reserved":0,"allocated":638976,"data_stored":282303,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"min":{"1min":0.54000000000000004,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.87,"15min":0.86199999999999999},"min":{"1min":0.60899999999999999,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3049999999999999,"5min":1.675,"15min":1.675},"last":1.0609999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.875,"15min":0.88300000000000001},"min":{"1min":0.625,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.0569999999999999,"5min":1.369,"15min":1.369},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.82999999999999996,"15min":0.82399999999999995},"min":{"1min":0.65400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0740000000000001,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":1.141}]},{"osd":2,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.81000000000000005,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.276,"5min":1.625,"15min":1.625},"last":1.032},{"interface":"front","average":{"1min":0.89200000000000002,"5min":0.97399999999999998,"15min":0.98799999999999999},"min":{"1min":0.70099999999999996,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.286,"5min":1.552,"15min":1.552},"last":0.66500000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.84999999999999998,"15min":0.83599999999999997},"min":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3919999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":1.004,"15min":1.0269999999999999},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1419999999999999,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":1.115}]},{"osd":4,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.0669999999999999,"15min":1.081},"min":{"1min":0.61599999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.363,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.879},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.0660000000000001,"15min":1.0620000000000001},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.47,"5min":1.601,"15min":1.601},"last":1.2110000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.97499999999999998,"15min":0.97799999999999998},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.2529999999999999,"5min":1.421,"15min":1.421},"last":0.95099999999999996},{"interface":"front","average":{"1min":1.054,"5min":1.0740000000000001,"15min":1.0780000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.397,"5min":1.5469999999999999,"15min":1.5469999999999999},"last":0.998}]},{"osd":7,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.044,"15min":1.038},"min":{"1min":0.76300000000000001,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3819999999999999,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.264},{"interface":"front","average":{"1min":1.071,"5min":1.2190000000000001,"15min":1.244},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.417,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.1879999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542173,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27648,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344192,"statfs":{"total":96636764160,"available":96608452608,"internally_reserved":0,"allocated":643072,"data_stored":282885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84199999999999997,"15min":0.83799999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":1.335},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84699999999999998,"15min":0.84899999999999998},"min":{"1min":0.59899999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":1.929}]},{"osd":1,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.91000000000000003,"15min":0.91400000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3049999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.72799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.353,"15min":1.353},"last":1.5720000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.85199999999999998,"15min":0.84099999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.61099999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.2749999999999999,"5min":1.603,"15min":1.603},"last":1.175}]},{"osd":3,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.95599999999999996,"15min":0.95799999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.298,"5min":1.472,"15min":1.472},"last":0.92900000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96199999999999997,"15min":0.96499999999999997},"min":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":1.306}]},{"osd":5,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.98399999999999999,"15min":0.97999999999999998},"min":{"1min":0.70099999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":2.048,"15min":2.048},"last":1.073},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1519999999999999},"min":{"1min":0.83699999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4179999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.492}]},{"osd":6,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.749,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.72699999999999998},{"interface":"front","average":{"1min":1.202,"5min":1.1220000000000001,"15min":1.1080000000000001},"min":{"1min":0.876,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.74,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":1.8420000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.165,"15min":1.153},"min":{"1min":0.95999999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.1180000000000001},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.21}]}]},{"osd":3,"up_from":14,"seq":60129542174,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27660,"kb_used_data":640,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344180,"statfs":{"total":96636764160,"available":96608440320,"internally_reserved":0,"allocated":655360,"data_stored":283889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.63,"15min":0.63600000000000001},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.65300000000000002,"15min":0.65500000000000003},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.95299999999999996,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.68100000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65100000000000002,"5min":0.624,"15min":0.62},"min":{"1min":0.40600000000000003,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.84099999999999997,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.495},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79000000000000004,"15min":0.78900000000000003},"min":{"1min":0.52000000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.1339999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.78800000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.65000000000000002,"5min":0.69099999999999995,"15min":0.69799999999999995},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.90100000000000002,"5min":1.27,"15min":1.27},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.77500000000000002,"5min":0.755,"15min":0.752},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.84199999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.131,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.997},{"interface":"front","average":{"1min":0.94999999999999996,"5min":0.96099999999999997,"15min":0.96299999999999997},"min":{"1min":0.78500000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.222,"5min":1.782,"15min":1.782},"last":0.94899999999999995}]},{"osd":5,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.96399999999999997,"5min":0.97999999999999998,"15min":0.98299999999999998},"min":{"1min":0.68300000000000005,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.3360000000000001,"5min":1.5289999999999999,"15min":1.5289999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.1040000000000001,"15min":1.1060000000000001},"min":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.2749999999999999,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.97299999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0860000000000001,"5min":1.1339999999999999,"15min":1.1419999999999999},"min":{"1min":0.81100000000000005,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.6950000000000001,"5min":1.885,"15min":1.885},"last":1.056},{"interface":"front","average":{"1min":0.97999999999999998,"5min":1.052,"15min":1.0640000000000001},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.137,"5min":1.95,"15min":1.95},"last":0.81799999999999995}]},{"osd":7,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.0780000000000001,"15min":1.0760000000000001},"min":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.889,"5min":1.889,"15min":1.889},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.018,"5min":1.054,"15min":1.0600000000000001},"min":{"1min":0.83399999999999996,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.3029999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":1.0249999999999999}]}]},{"osd":2,"up_from":14,"seq":60129542175,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344176,"statfs":{"total":96636764160,"available":96608436224,"internally_reserved":0,"allocated":651264,"data_stored":283867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.47899999999999998,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.68300000000000005,"15min":0.69099999999999995},"min":{"1min":0.49299999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":0.84399999999999997,"5min":0.995,"15min":0.995},"last":0.80800000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.67000000000000004,"5min":0.70399999999999996,"15min":0.70999999999999996},"min":{"1min":0.433,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.68899999999999995,"15min":0.69199999999999995},"min":{"1min":0.441,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.95899999999999996,"5min":1.123,"15min":1.123},"last":0.45200000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.65900000000000003,"15min":0.66500000000000004},"min":{"1min":0.40200000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.83699999999999997,"5min":1.073,"15min":1.073},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79000000000000004,"15min":0.78700000000000003},"min":{"1min":0.63400000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.98599999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.84099999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.94599999999999995,"15min":0.95399999999999996},"min":{"1min":0.69999999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.127,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.022},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.96999999999999997,"15min":0.98799999999999999},"min":{"1min":0.68500000000000005,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1020000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.90300000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.98999999999999999,"15min":0.98899999999999999},"min":{"1min":0.76100000000000001,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.3759999999999999,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":1.153},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.96299999999999997,"15min":0.96299999999999997},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3300000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.93500000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.0229999999999999,"15min":1.0449999999999999},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.046,"5min":2.024,"15min":2.024},"last":0.996},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.069,"15min":1.0700000000000001},"min":{"1min":0.84899999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.431,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":1.0489999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:27:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0620000000000001,"5min":1.083,"15min":1.087},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.351,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.004,"5min":1.0629999999999999,"15min":1.073},"min":{"1min":0.76700000000000002,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.204,"5min":1.55,"15min":1.55},"last":1.109}]}]},{"osd":1,"up_from":14,"seq":60129542174,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27656,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344184,"statfs":{"total":96636764160,"available":96608444416,"internally_reserved":0,"allocated":647168,"data_stored":282463,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.60299999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.41499999999999998,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.79600000000000004,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.90400000000000003},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71999999999999997,"15min":0.72099999999999997},"min":{"1min":0.41799999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.141,"5min":1.141,"15min":1.141},"last":0.94599999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.61399999999999999,"15min":0.60699999999999998},"min":{"1min":0.378,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.98199999999999998,"5min":1,"15min":1},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.70499999999999996,"15min":0.70299999999999996},"min":{"1min":0.48099999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0629999999999999,"5min":1.458,"15min":1.458},"last":0.86699999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.85099999999999998,"5min":0.70899999999999996,"15min":0.68500000000000005},"min":{"1min":0.56100000000000005,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.236,"5min":1.236,"15min":1.236},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.71899999999999997,"5min":0.68600000000000005,"15min":0.68000000000000005},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.97399999999999998,"5min":1.403,"15min":1.403},"last":0.73799999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.90500000000000003,"15min":0.90000000000000002},"min":{"1min":0.67300000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.3160000000000001,"5min":1.546,"15min":1.546},"last":1.0349999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93300000000000005,"15min":0.93500000000000005},"min":{"1min":0.61299999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.1240000000000001,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":1.3089999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"min":{"1min":0.67900000000000005,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3089999999999999,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.97399999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.96999999999999997,"15min":0.98199999999999998},"min":{"1min":0.75900000000000001,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.2250000000000001,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.3979999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.999,"15min":1.0049999999999999},"min":{"1min":0.77200000000000002,"5min":0.626,"15min":0.626},"max":{"1min":1.3460000000000001,"5min":1.613,"15min":1.613},"last":1.2569999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"min":{"1min":0.70899999999999996,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3839999999999999,"5min":1.698,"15min":1.698},"last":1.1200000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:27:11 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":1.0029999999999999,"15min":1.0089999999999999},"min":{"1min":0.72999999999999998,"5min":0.62,"15min":0.62},"max":{"1min":1.446,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.47},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0329999999999999,"15min":1.0349999999999999},"min":{"1min":0.69099999999999995,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.4179999999999999,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.202}]}]},{"osd":0,"up_from":14,"seq":60129542174,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28244,"kb_used_data":1212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343596,"statfs":{"total":96636764160,"available":96607842304,"internally_reserved":0,"allocated":1241088,"data_stored":872727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.628},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.75600000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.504}]},{"osd":2,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.66300000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81000000000000005,"15min":0.81399999999999995},"min":{"1min":0.45300000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":0.73999999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":0.95999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2949999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.04,"5min":0.96999999999999997,"15min":0.95799999999999996},"min":{"1min":0.68700000000000006,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":0.96999999999999997}]},{"osd":5,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.96299999999999997,"15min":0.96399999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2,"5min":1.24,"15min":1.24},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.99299999999999999,"15min":0.999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.198,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":0.878}]},{"osd":6,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.995,"15min":0.97799999999999998},"min":{"1min":0.80900000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.2},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.036,"15min":1.04},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":0.85099999999999998}]},{"osd":7,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.054,"15min":1.046},"min":{"1min":0.90000000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0620000000000001,"15min":1.038},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.1539999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:27:27.769 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:27:27.769 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T16:27:27.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8af020a700 1 Processor -- start 2024-09-13T16:27:27.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8af020a700 1 -- start start 2024-09-13T16:27:27.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8af020a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8ae815a980 0x7f8ae80cf0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8af020a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae8072030 con 0x7f8ae815b310 2024-09-13T16:27:27.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8af020a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae80ce530 con 0x7f8ae80b3b40 2024-09-13T16:27:27.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8af020a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae80cdcd0 con 0x7f8ae815a980 2024-09-13T16:27:27.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aed7a5700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8ae815b310 0x7f8ae80ce9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54088/0 (socket says 172.21.15.119:54088) 2024-09-13T16:27:27.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aed7a5700 1 -- 172.21.15.119:0/707874871 learned_addr learned my addr 172.21.15.119:0/707874871 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:27.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aee7a7700 1 --2- 172.21.15.119:0/707874871 >> v2:172.21.15.145:3300/0 conn(0x7f8ae815a980 0x7f8ae80cf0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402581037 0 0) 0x7f8ae8072030 con 0x7f8ae815b310 2024-09-13T16:27:27.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad40035d0 con 0x7f8ae815b310 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955868662 0 0) 0x7f8ad40035d0 con 0x7f8ae815b310 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 >> v2:172.21.15.145:3300/0 conn(0x7f8ae815a980 msgr2=0x7f8ae80cf0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 --2- 172.21.15.119:0/707874871 >> v2:172.21.15.145:3300/0 conn(0x7f8ae815a980 0x7f8ae80cf0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 >> v1:172.21.15.145:6789/0 conn(0x7f8ae80b3b40 legacy=0x7f8ae80b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae8071d50 con 0x7f8ae815b310 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f8adc001670 con 0x7f8ae815b310 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 >> v1:172.21.15.119:6789/0 conn(0x7f8ae815b310 legacy=0x7f8ae80ce9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad4003a20 0x7f8ad4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad4004870 0x7f8ad4004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4005230 0x7f8ad400ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae80cdcd0 con 0x7f8ad4004870 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad400c0d0 con 0x7f8ad4005230 2024-09-13T16:27:27.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad400c290 con 0x7f8ad4003a20 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad4004870 0x7f8ad4004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.899+0000 7f8aee7a7700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4005230 0x7f8ad400ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aedfa6700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad4003a20 0x7f8ad4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aed7a5700 1 -- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad4003a20 msgr2=0x7f8ad4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad4003a20 0x7f8ad4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aed7a5700 1 -- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4005230 msgr2=0x7f8ad400ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4005230 0x7f8ad400ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aee7a7700 1 -- 172.21.15.119:0/707874871 reap_dead start 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aed7a5700 1 -- 172.21.15.119:0/707874871 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad40035d0 con 0x7f8ad4004870 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad4004870 0x7f8ad4004cf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00c320 tx=0x7f8ad40058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6db83ff57ffed641 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc007c30 con 0x7f8ad4004870 2024-09-13T16:27:27.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc00ce00 con 0x7f8ad4004870 2024-09-13T16:27:27.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8aecfa4700 1 -- 172.21.15.119:0/707874871 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc005bc0 con 0x7f8ad4004870 2024-09-13T16:27:27.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 -- 172.21.15.119:0/707874871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad4004870 msgr2=0x7f8ad4004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad4004870 0x7f8ad4004cf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f8adc00c320 tx=0x7f8ad40058a0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 -- 172.21.15.119:0/707874871 shutdown_connections 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 --2- 172.21.15.119:0/707874871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad4004870 0x7f8ad4004cf0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 -- 172.21.15.119:0/707874871 >> 172.21.15.119:0/707874871 conn(0x7f8ae8070dc0 msgr2=0x7f8ae80711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 -- 172.21.15.119:0/707874871 shutdown_connections 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 -- 172.21.15.119:0/707874871 wait complete. 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 Processor -- start 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.903+0000 7f8af020a700 1 -- start start 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4004870 0x7f8ae80ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae81f3500 0x7f8ae806e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ae80b0990 0x7f8ae81420d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae81f3980 con 0x7f8ae80b0990 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae81f3070 con 0x7f8ad4004870 2024-09-13T16:27:27.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae806ec20 con 0x7f8ae81f3500 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aee7a7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ae80b0990 0x7f8ae81420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aedfa6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4004870 0x7f8ae80ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aee7a7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ae80b0990 0x7f8ae81420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39716/0 (socket says 172.21.15.119:39716) 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aee7a7700 1 -- 172.21.15.119:0/3000567397 learned_addr learned my addr 172.21.15.119:0/3000567397 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae81f3500 0x7f8ae806e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aed7a5700 1 -- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4004870 msgr2=0x7f8ae80ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4004870 0x7f8ae80ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aed7a5700 1 -- 172.21.15.119:0/3000567397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ae80b0990 msgr2=0x7f8ae81420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ae80b0990 0x7f8ae81420d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aed7a5700 1 -- 172.21.15.119:0/3000567397 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae80acf50 con 0x7f8ae81f3500 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aee7a7700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ae80b0990 0x7f8ae81420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8aed7a5700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae81f3500 0x7f8ae806e6e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8adc014000 tx=0x7f8adc001a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8ada7fc700 1 -- 172.21.15.119:0/3000567397 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc004ca0 con 0x7f8ae81f3500 2024-09-13T16:27:27.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8ada7fc700 1 -- 172.21.15.119:0/3000567397 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8adc004e00 con 0x7f8ae81f3500 2024-09-13T16:27:27.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8ada7fc700 1 -- 172.21.15.119:0/3000567397 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8adc007310 con 0x7f8ae81f3500 2024-09-13T16:27:27.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ae80ad230 con 0x7f8ae81f3500 2024-09-13T16:27:27.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ae81f29f0 con 0x7f8ae81f3500 2024-09-13T16:27:27.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.907+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ae80ad420 con 0x7f8ae81f3500 2024-09-13T16:27:27.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.911+0000 7f8ada7fc700 1 -- 172.21.15.119:0/3000567397 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8adc007470 con 0x7f8ae81f3500 2024-09-13T16:27:27.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.911+0000 7f8ada7fc700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8acc066760 0x7f8acc068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:27.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.911+0000 7f8ada7fc700 1 -- 172.21.15.119:0/3000567397 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8adc085020 con 0x7f8ae81f3500 2024-09-13T16:27:27.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.911+0000 7f8aedfa6700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8acc066760 0x7f8acc068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:27.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.911+0000 7f8aedfa6700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8acc066760 0x7f8acc068c20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8139800 tx=0x7f8ae4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:27.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:27.919+0000 7f8ada7fc700 1 -- 172.21.15.119:0/3000567397 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8adc0522e0 con 0x7f8ae81f3500 2024-09-13T16:27:28.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.151+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8ae81435c0 con 0x7f8ae81f3500 2024-09-13T16:27:28.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.159+0000 7f8ada7fc700 1 -- 172.21.15.119:0/3000567397 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+0 (secure 0 0 0) 0x7f8adc007d90 con 0x7f8ae81f3500 2024-09-13T16:27:28.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8acc066760 msgr2=0x7f8acc068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:28.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8acc066760 0x7f8acc068c20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8ae8139800 tx=0x7f8ae4009040 comp rx=0 tx=0).stop 2024-09-13T16:27:28.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae81f3500 msgr2=0x7f8ae806e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae81f3500 0x7f8ae806e6e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f8adc014000 tx=0x7f8adc001a00 comp rx=0 tx=0).stop 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 shutdown_connections 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8acc066760 0x7f8acc068c20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ae80b0990 0x7f8ae81420d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ae81f3500 0x7f8ae806e6e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 --2- 172.21.15.119:0/3000567397 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad4004870 0x7f8ae80ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 >> 172.21.15.119:0/3000567397 conn(0x7f8ae8070dc0 msgr2=0x7f8ae815bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 shutdown_connections 2024-09-13T16:27:28.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.163+0000 7f8af020a700 1 -- 172.21.15.119:0/3000567397 wait complete. 2024-09-13T16:27:28.167 INFO:teuthology.orchestra.run.smithi119.stderr:updated caps for client.0 2024-09-13T16:27:28.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:28.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.287+0000 7f846da6f700 1 Processor -- start 2024-09-13T16:27:28.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.287+0000 7f846da6f700 1 -- start start 2024-09-13T16:27:28.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f846806ec50 0x7f84680cd410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:28.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f846806f030 0x7f84680b4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:28.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f84680cec40 0x7f846815ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:28.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84680cf590 con 0x7f84680cec40 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84680b3a60 con 0x7f846806ec50 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84680b3d40 con 0x7f846806f030 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f84680cec40 0x7f846815ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f84680cec40 0x7f846815ad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39718/0 (socket says 172.21.15.119:39718) 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 -- 172.21.15.119:0/1063383930 learned_addr learned my addr 172.21.15.119:0/1063383930 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f845ffff700 1 --2- 172.21.15.119:0/1063383930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f846806f030 0x7f84680b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846ca6d700 1 --2- 172.21.15.119:0/1063383930 >> v2:172.21.15.145:3300/0 conn(0x7f846806ec50 0x7f84680cd410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 -- 172.21.15.119:0/1063383930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f846806f030 msgr2=0x7f84680b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 --2- 172.21.15.119:0/1063383930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f846806f030 0x7f84680b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 -- 172.21.15.119:0/1063383930 >> v2:172.21.15.145:3300/0 conn(0x7f846806ec50 msgr2=0x7f84680cd410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 --2- 172.21.15.119:0/1063383930 >> v2:172.21.15.145:3300/0 conn(0x7f846806ec50 0x7f84680cd410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 -- 172.21.15.119:0/1063383930 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84680b4020 con 0x7f84680cec40 2024-09-13T16:27:28.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846d26e700 1 --2- 172.21.15.119:0/1063383930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680cec40 0x7f846815ad80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f84680b5b30 tx=0x7f846000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7acb018d607ba19a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:28.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f845f7fe700 1 -- 172.21.15.119:0/1063383930 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846000e070 con 0x7f84680cec40 2024-09-13T16:27:28.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f845f7fe700 1 -- 172.21.15.119:0/1063383930 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846000b9c0 con 0x7f84680cec40 2024-09-13T16:27:28.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f845f7fe700 1 -- 172.21.15.119:0/1063383930 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846000bd70 con 0x7f84680cec40 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- 172.21.15.119:0/1063383930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680cec40 msgr2=0x7f846815ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 --2- 172.21.15.119:0/1063383930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680cec40 0x7f846815ad80 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f84680b5b30 tx=0x7f846000b330 comp rx=0 tx=0).stop 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- 172.21.15.119:0/1063383930 shutdown_connections 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 --2- 172.21.15.119:0/1063383930 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84680cec40 0x7f846815ad80 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 --2- 172.21.15.119:0/1063383930 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f846806f030 0x7f84680b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 --2- 172.21.15.119:0/1063383930 >> v2:172.21.15.145:3300/0 conn(0x7f846806ec50 0x7f84680cd410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- 172.21.15.119:0/1063383930 >> 172.21.15.119:0/1063383930 conn(0x7f8468070cc0 msgr2=0x7f84680710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- 172.21.15.119:0/1063383930 shutdown_connections 2024-09-13T16:27:28.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.291+0000 7f846da6f700 1 -- 172.21.15.119:0/1063383930 wait complete. 2024-09-13T16:27:28.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 Processor -- start 2024-09-13T16:27:28.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 -- start start 2024-09-13T16:27:28.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f846806ec50 0x7f846813b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846806f030 0x7f846815d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84680cec40 0x7f84680a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846815c4c0 con 0x7f846806f030 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846815c7a0 con 0x7f846806ec50 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f846da6f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846815ca80 con 0x7f84680cec40 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f845ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846806f030 0x7f846815d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f845ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846806f030 0x7f846815d560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39720/0 (socket says 172.21.15.119:39720) 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.295+0000 7f845ffff700 1 -- 172.21.15.119:0/3293080318 learned_addr learned my addr 172.21.15.119:0/3293080318 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f846ca6d700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f846806ec50 0x7f846813b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f846d26e700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84680cec40 0x7f84680a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845ffff700 1 -- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84680cec40 msgr2=0x7f84680a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845ffff700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84680cec40 0x7f84680a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845ffff700 1 -- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f846806ec50 msgr2=0x7f846813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845ffff700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f846806ec50 0x7f846813b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845ffff700 1 -- 172.21.15.119:0/3293080318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8460009b30 con 0x7f846806f030 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845ffff700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846806f030 0x7f846815d560 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f845800e810 tx=0x7f845800eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845d7fa700 1 -- 172.21.15.119:0/3293080318 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845800c070 con 0x7f846806f030 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845d7fa700 1 -- 172.21.15.119:0/3293080318 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8458013610 con 0x7f846806f030 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846806f5f0 con 0x7f846806f030 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f845d7fa700 1 -- 172.21.15.119:0/3293080318 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84580139c0 con 0x7f846806f030 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846815d040 con 0x7f846806f030 2024-09-13T16:27:28.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.299+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846815d260 con 0x7f846806f030 2024-09-13T16:27:28.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.303+0000 7f845d7fa700 1 -- 172.21.15.119:0/3293080318 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8458013b20 con 0x7f846806f030 2024-09-13T16:27:28.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.303+0000 7f845d7fa700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066760 0x7f8454068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:28.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.303+0000 7f845d7fa700 1 -- 172.21.15.119:0/3293080318 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8458084060 con 0x7f846806f030 2024-09-13T16:27:28.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.303+0000 7f846ca6d700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066760 0x7f8454068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:28.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.303+0000 7f846ca6d700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066760 0x7f8454068c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8464009970 tx=0x7f8464008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:28.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.311+0000 7f845d7fa700 1 -- 172.21.15.119:0/3293080318 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8458051210 con 0x7f846806f030 2024-09-13T16:27:28.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.515+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8468144580 con 0x7f846806f030 2024-09-13T16:27:28.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.515+0000 7f845d7fa700 1 -- 172.21.15.119:0/3293080318 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f8458054830 con 0x7f846806f030 2024-09-13T16:27:28.519 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:28.519 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:27:28.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066760 msgr2=0x7f8454068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:28.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066760 0x7f8454068c20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f8464009970 tx=0x7f8464008040 comp rx=0 tx=0).stop 2024-09-13T16:27:28.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846806f030 msgr2=0x7f846815d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:28.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846806f030 0x7f846815d560 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f845800e810 tx=0x7f845800eb20 comp rx=0 tx=0).stop 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 shutdown_connections 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8454066760 0x7f8454068c20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84680cec40 0x7f84680a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f846806f030 0x7f846815d560 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 --2- 172.21.15.119:0/3293080318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f846806ec50 0x7f846813b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 >> 172.21.15.119:0/3293080318 conn(0x7f8468070cc0 msgr2=0x7f84681433f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 shutdown_connections 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:28.519+0000 7f846da6f700 1 -- 172.21.15.119:0/3293080318 wait complete. 2024-09-13T16:27:28.522 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:27:28.937 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:27:23.348672+0000', 'modified': '2024-09-13T16:27:26.375537+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6069}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6069': {'gid': 6069, 'name': 'a', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6835/4101111376', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 4101111376}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 4101111376}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:27:28.938 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:27:28.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-09-13T16:27:29.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.055+0000 7f8b68934700 1 Processor -- start 2024-09-13T16:27:29.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.055+0000 7f8b68934700 1 -- start start 2024-09-13T16:27:29.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640adb30 0x7f8b640cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8b6406bb90 0x7f8b640cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6406cf60 con 0x7f8b640cde70 2024-09-13T16:27:29.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b640b3390 con 0x7f8b6406bb90 2024-09-13T16:27:29.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6406c350 con 0x7f8b640adb30 2024-09-13T16:27:29.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b6359e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8b640cde70 0x7f8b6415e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54106/0 (socket says 172.21.15.119:54106) 2024-09-13T16:27:29.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b6359e700 1 -- 172.21.15.119:0/514847646 learned_addr learned my addr 172.21.15.119:0/514847646 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:29.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b6259c700 1 --2- 172.21.15.119:0/514847646 >> v2:172.21.15.145:3300/0 conn(0x7f8b6406bb90 0x7f8b640cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/514847646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640adb30 0x7f8b640cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b61d9b700 1 -- 172.21.15.119:0/514847646 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229483420 0 0) 0x7f8b6406cf60 con 0x7f8b640cde70 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b61d9b700 1 -- 172.21.15.119:0/514847646 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b440035d0 con 0x7f8b640cde70 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b62d9d700 1 -- 172.21.15.119:0/514847646 >> v2:172.21.15.145:3300/0 conn(0x7f8b6406bb90 msgr2=0x7f8b640cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/514847646 >> v2:172.21.15.145:3300/0 conn(0x7f8b6406bb90 0x7f8b640cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b62d9d700 1 -- 172.21.15.119:0/514847646 >> v1:172.21.15.119:6789/0 conn(0x7f8b640cde70 legacy=0x7f8b6415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b62d9d700 1 -- 172.21.15.119:0/514847646 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6406c630 con 0x7f8b640adb30 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b6259c700 1 --2- 172.21.15.119:0/514847646 >> v2:172.21.15.145:3300/0 conn(0x7f8b6406bb90 0x7f8b640cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/514847646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640adb30 0x7f8b640cc060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009850 tx=0x7f8b4c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7de929ccb5cbc5a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b61d9b700 1 -- 172.21.15.119:0/514847646 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c011830 con 0x7f8b640adb30 2024-09-13T16:27:29.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b61d9b700 1 -- 172.21.15.119:0/514847646 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c011990 con 0x7f8b640adb30 2024-09-13T16:27:29.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b61d9b700 1 -- 172.21.15.119:0/514847646 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c011d00 con 0x7f8b640adb30 2024-09-13T16:27:29.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- 172.21.15.119:0/514847646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640adb30 msgr2=0x7f8b640cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 --2- 172.21.15.119:0/514847646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640adb30 0x7f8b640cc060 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009850 tx=0x7f8b4c009b60 comp rx=0 tx=0).stop 2024-09-13T16:27:29.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- 172.21.15.119:0/514847646 shutdown_connections 2024-09-13T16:27:29.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 --2- 172.21.15.119:0/514847646 >> v2:172.21.15.145:3300/0 conn(0x7f8b6406bb90 0x7f8b640cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 --2- 172.21.15.119:0/514847646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640adb30 0x7f8b640cc060 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- 172.21.15.119:0/514847646 >> 172.21.15.119:0/514847646 conn(0x7f8b6406a350 msgr2=0x7f8b6406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- 172.21.15.119:0/514847646 shutdown_connections 2024-09-13T16:27:29.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.059+0000 7f8b68934700 1 -- 172.21.15.119:0/514847646 wait complete. 2024-09-13T16:27:29.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 Processor -- start 2024-09-13T16:27:29.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 -- start start 2024-09-13T16:27:29.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b6406bb90 0x7f8b6412ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b640adb30 0x7f8b64134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640cde70 0x7f8b64159fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64134940 con 0x7f8b640adb30 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64130ac0 con 0x7f8b6406bb90 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b68934700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b64130d70 con 0x7f8b640cde70 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b640adb30 0x7f8b64134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b640adb30 0x7f8b64134400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39734/0 (socket says 172.21.15.119:39734) 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 -- 172.21.15.119:0/1844065477 learned_addr learned my addr 172.21.15.119:0/1844065477 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b6406bb90 0x7f8b6412ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6359e700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640cde70 0x7f8b64159fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 -- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640cde70 msgr2=0x7f8b64159fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640cde70 0x7f8b64159fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 -- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b6406bb90 msgr2=0x7f8b6412ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b6406bb90 0x7f8b6412ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 -- 172.21.15.119:0/1844065477 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b54009a70 con 0x7f8b640adb30 2024-09-13T16:27:29.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b6406bb90 0x7f8b6412ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:29.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6359e700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640cde70 0x7f8b64159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:29.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.063+0000 7f8b6259c700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b640adb30 0x7f8b64134400 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400efc0 tx=0x7f8b5400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.067+0000 7f8b5b7fe700 1 -- 172.21.15.119:0/1844065477 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5400cee0 con 0x7f8b640adb30 2024-09-13T16:27:29.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.067+0000 7f8b5b7fe700 1 -- 172.21.15.119:0/1844065477 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5400f040 con 0x7f8b640adb30 2024-09-13T16:27:29.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.067+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b4c0094b0 con 0x7f8b640adb30 2024-09-13T16:27:29.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.067+0000 7f8b5b7fe700 1 -- 172.21.15.119:0/1844065477 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54014680 con 0x7f8b640adb30 2024-09-13T16:27:29.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.067+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b64131330 con 0x7f8b640adb30 2024-09-13T16:27:29.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.067+0000 7f8b5b7fe700 1 -- 172.21.15.119:0/1844065477 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8b540147e0 con 0x7f8b640adb30 2024-09-13T16:27:29.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b5b7fe700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b48066760 0x7f8b48068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b5b7fe700 1 -- 172.21.15.119:0/1844065477 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8b54085060 con 0x7f8b640adb30 2024-09-13T16:27:29.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b48066760 0x7f8b48068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 Processor -- start 2024-09-13T16:27:29.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 -- start start 2024-09-13T16:27:29.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b48066760 0x7f8b48068c20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0090b0 tx=0x7f8b4c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ca530 0x29ca910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29cb290 0x29f2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x29f2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2937d80 con 0x29cb290 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2935510 con 0x29cb670 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2930e00 con 0x29ca530 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29cb290 0x29f2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29cb290 0x29f2540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39746/0 (socket says 172.21.15.119:39746) 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 -- 172.21.15.119:0/3842369093 learned_addr learned my addr 172.21.15.119:0/3842369093 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ca530 0x29ca910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6359e700 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x29f2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 -- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ca530 msgr2=0x29ca910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ca530 0x29ca910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 -- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 msgr2=0x29f2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x29f2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 -- 172.21.15.119:0/3842369093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x27a3960 con 0x29cb290 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b6259c700 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29cb290 0x29f2540 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8b54083450 tx=0x7f8b54055bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99ec0469447dac6d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b58ff9700 1 -- 172.21.15.119:0/3842369093 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54059d90 con 0x29cb290 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b58ff9700 1 -- 172.21.15.119:0/3842369093 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5405e350 con 0x29cb290 2024-09-13T16:27:29.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.071+0000 7f8b58ff9700 1 -- 172.21.15.119:0/3842369093 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54024300 con 0x29cb290 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/3842369093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29cb290 msgr2=0x29f2540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29cb290 0x29f2540 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8b54083450 tx=0x7f8b54055bd0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/3842369093 shutdown_connections 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x29f2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29cb290 0x29f2540 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/3842369093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ca530 0x29ca910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/3842369093 >> 172.21.15.119:0/3842369093 conn(0x27584d0 msgr2=0x29f1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/3842369093 shutdown_connections 2024-09-13T16:27:29.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/3842369093 wait complete. 2024-09-13T16:27:29.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- start start 2024-09-13T16:27:29.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 Processor -- start 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x29f6b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x2a132b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x2a137f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x28ee730 con 0x29ca530 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x28f45b0 con 0x29cb290 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x27dcaa0 con 0x29cb670 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6c3e6740 10 client.0 _ll_register_callbacks cb 0x292a610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b62d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x29f6b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b62d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x29f6b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39752/0 (socket says 172.21.15.119:39752) 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b62d9d700 1 -- 172.21.15.119:0/142038128 learned_addr learned my addr 172.21.15.119:0/142038128 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6259c700 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x2a132b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.075+0000 7f8b6359e700 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x2a137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b62d9d700 1 -- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 msgr2=0x2a137f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x2a137f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b62d9d700 1 -- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 msgr2=0x2a132b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x2a132b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b62d9d700 1 -- 172.21.15.119:0/142038128 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b64131330 con 0x29ca530 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x29f6b40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8b6406c490 tx=0x7f8b4c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21a7cee99963f1d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 1 -- 172.21.15.119:0/142038128 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c02d070 con 0x29ca530 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x27651e0 con 0x29ca530 2024-09-13T16:27:29.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 1 -- 172.21.15.119:0/142038128 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8b4c0038d0 con 0x29ca530 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 1 -- 172.21.15.119:0/142038128 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b4c01f070 con 0x29ca530 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 10 client.6219 fetch_fsmap learned FSMap version 24 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x28f7220 con 0x29ca530 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 1 -- 172.21.15.119:0/142038128 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x27651e0 con 0x29ca530 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 1 -- 172.21.15.119:0/142038128 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8b4c003a90 con 0x29ca530 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 10 client.6219 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 10 client.6219 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 20 client.6219 populate_metadata read hostname 'smithi119' 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a1ee70 0x29f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 4 client.6219 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:27:29.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x27af460 con 0x2a1ee70 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6259c700 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a1ee70 0x29f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6259c700 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a1ee70 0x29f4d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cc1b0764d4afb16 server_cookie=f89815f2af358260 in_seq=0 out_seq=0 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 10 client.6219 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 1 -- 172.21.15.119:0/142038128 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x27af460 con 0x2a1ee70 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b51ffb700 10 client.6219 handle_command_reply: tid=0 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 1 client.6219 shutdown 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 msgr2=0x29f6b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.079+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x29f6b40 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8b6406c490 tx=0x7f8b4c00e000 comp rx=0 tx=0).stop 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 shutdown_connections 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x2a137f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a1ee70 0x29f4d00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x2a132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/142038128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x29f6b40 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 >> 172.21.15.119:0/142038128 conn(0x27584d0 msgr2=0x29c4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b51ffb700 0 client.6219 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:29.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 shutdown_connections 2024-09-13T16:27:29.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/142038128 wait complete. 2024-09-13T16:27:29.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.083+0000 7f8b6c3e6740 20 client.6219 trim_cache size 0 max 16384 2024-09-13T16:27:29.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 Processor -- start 2024-09-13T16:27:29.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 -- start start 2024-09-13T16:27:29.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a250d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x29f1c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x29f2160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x295fec0 con 0x29ca530 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a13930 con 0x29cb290 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x29cba50 con 0x29cb670 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b62d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a250d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x29f1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b62d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a250d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39758/0 (socket says 172.21.15.119:39758) 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x29f1c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:33452/0 (socket says 172.21.15.119:33452) 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 -- 172.21.15.119:0/1440900275 learned_addr learned my addr 172.21.15.119:0/1440900275 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6359e700 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x29f2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 -- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 msgr2=0x29f2160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x29f2160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 -- 172.21.15.119:0/1440900275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 msgr2=0x2a250d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a250d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6259c700 1 -- 172.21.15.119:0/1440900275 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x28dd450 con 0x29cb290 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.103+0000 7f8b6359e700 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x29f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6259c700 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x29f1c20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8b54052460 tx=0x7f8b5405c040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7574630a02547f9d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b597fa700 1 -- 172.21.15.119:0/1440900275 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540522a0 con 0x29cb290 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b597fa700 1 -- 172.21.15.119:0/1440900275 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b54059d90 con 0x29cb290 2024-09-13T16:27:29.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b597fa700 1 -- 172.21.15.119:0/1440900275 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b54024300 con 0x29cb290 2024-09-13T16:27:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 msgr2=0x29f1c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x29f1c20 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f8b54052460 tx=0x7f8b5405c040 comp rx=0 tx=0).stop 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/1440900275 shutdown_connections 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb670 0x29f2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb290 0x29f1c20 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/1440900275 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a250d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/1440900275 >> 172.21.15.119:0/1440900275 conn(0x27584d0 msgr2=0x29ec870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/1440900275 shutdown_connections 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/1440900275 wait complete. 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- start start 2024-09-13T16:27:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 Processor -- start 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a24b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb290 0x29c52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x2a21ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x29f2da0 con 0x29ca530 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a23e20 con 0x29cb670 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a24050 con 0x29cb290 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6c3e6740 10 client.0 _ll_register_callbacks cb 0x29cbb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b62d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a24b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb290 0x29c52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x2a21ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6259c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb290 0x29c52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:39494/0 (socket says 172.21.15.119:39494) 2024-09-13T16:27:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6259c700 1 -- 172.21.15.119:0/4217255176 learned_addr learned my addr 172.21.15.119:0/4217255176 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b62d9d700 1 -- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb290 msgr2=0x29c52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb290 0x29c52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b62d9d700 1 -- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 msgr2=0x2a21ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x2a21ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b62d9d700 1 -- 172.21.15.119:0/4217255176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5406d040 con 0x29ca530 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6259c700 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb290 0x29c52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.107+0000 7f8b6359e700 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x2a21ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b62d9d700 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a24b40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009ba0 tx=0x7f8b4c0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245c64e44a567e80 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 1 -- 172.21.15.119:0/4217255176 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c003b60 con 0x29ca530 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 1 -- 172.21.15.119:0/4217255176 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8b4c01a470 con 0x29ca530 2024-09-13T16:27:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2a10590 con 0x29ca530 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 1 -- 172.21.15.119:0/4217255176 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b4c01a630 con 0x29ca530 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 10 client.6225 fetch_fsmap learned FSMap version 24 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x28d1720 con 0x29ca530 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 1 -- 172.21.15.119:0/4217255176 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2a10590 con 0x29ca530 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 1 -- 172.21.15.119:0/4217255176 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f8b4c01aa30 con 0x29ca530 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 10 client.6225 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 10 client.6225 resolve_mds: resolved name 'a' to daemon mds.a 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 20 client.6225 populate_metadata read hostname 'smithi119' 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a22650 0x2a74db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 4 client.6225 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f8b5405e350 con 0x2a22650 2024-09-13T16:27:29.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6259c700 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a22650 0x2a74db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 10 client.6225 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6259c700 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a22650 0x2a74db0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a948c6a02cb00515 server_cookie=827df4142c515ce in_seq=0 out_seq=0 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 1 -- 172.21.15.119:0/4217255176 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+324 (crc 0 0 0) 0x7f8b5405e350 con 0x2a22650 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 10 client.6225 handle_command_reply: tid=0 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 client.6225 shutdown 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 msgr2=0x2a24b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a24b40 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009ba0 tx=0x7f8b4c0047e0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 shutdown_connections 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a22650 0x2a74db0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29cb670 0x2a21ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29cb290 0x29c52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 --2- 172.21.15.119:0/4217255176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29ca530 0x2a24b40 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b51ffb700 0 client.6225 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 >> 172.21.15.119:0/4217255176 conn(0x27584d0 msgr2=0x29f1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 shutdown_connections 2024-09-13T16:27:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 1 -- 172.21.15.119:0/4217255176 wait complete. 2024-09-13T16:27:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.111+0000 7f8b6c3e6740 20 client.6225 trim_cache size 0 max 16384 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 6069, 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 5.7613446619999999, 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 24, 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 54, 2024-09-13T16:27:29.121 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 54, 2024-09-13T16:27:29.122 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 10.23815321 2024-09-13T16:27:29.122 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b48066760 msgr2=0x7f8b48068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b48066760 0x7f8b48068c20 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c0090b0 tx=0x7f8b4c008030 comp rx=0 tx=0).stop 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b640adb30 msgr2=0x7f8b64134400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b640adb30 0x7f8b64134400 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8b5400efc0 tx=0x7f8b5400c3a0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 shutdown_connections 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b48066760 0x7f8b48068c20 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b640cde70 0x7f8b64159fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b640adb30 0x7f8b64134400 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 --2- 172.21.15.119:0/1844065477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b6406bb90 0x7f8b6412ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 >> 172.21.15.119:0/1844065477 conn(0x7f8b6406a350 msgr2=0x7f8b6415ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 shutdown_connections 2024-09-13T16:27:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.115+0000 7f8b68934700 1 -- 172.21.15.119:0/1844065477 wait complete. 2024-09-13T16:27:29.133 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:27:29.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.251+0000 7fb7ed866700 1 Processor -- start 2024-09-13T16:27:29.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.251+0000 7fb7ed866700 1 -- start start 2024-09-13T16:27:29.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb7e80adb30 0x7fb7e80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb7e806bb90 0x7fb7e80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e806cf60 con 0x7fb7e80cde70 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e80b3390 con 0x7fb7e80adb30 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e806c350 con 0x7fb7e806bb90 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed065700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb7e80cde70 0x7fb7e815e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54110/0 (socket says 172.21.15.119:54110) 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed065700 1 -- 172.21.15.119:0/1306639634 learned_addr learned my addr 172.21.15.119:0/1306639634 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ec864700 1 --2- 172.21.15.119:0/1306639634 >> v2:172.21.15.119:3300/0 conn(0x7fb7e80adb30 0x7fb7e80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7e7fff700 1 --2- 172.21.15.119:0/1306639634 >> v2:172.21.15.145:3300/0 conn(0x7fb7e806bb90 0x7fb7e80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7e77fe700 1 -- 172.21.15.119:0/1306639634 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718484975 0 0) 0x7fb7e806cf60 con 0x7fb7e80cde70 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7e77fe700 1 -- 172.21.15.119:0/1306639634 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7d00035d0 con 0x7fb7e80cde70 2024-09-13T16:27:29.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ec864700 1 -- 172.21.15.119:0/1306639634 >> v2:172.21.15.145:3300/0 conn(0x7fb7e806bb90 msgr2=0x7fb7e80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:29.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ec864700 1 --2- 172.21.15.119:0/1306639634 >> v2:172.21.15.145:3300/0 conn(0x7fb7e806bb90 0x7fb7e80cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ec864700 1 -- 172.21.15.119:0/1306639634 >> v1:172.21.15.119:6789/0 conn(0x7fb7e80cde70 legacy=0x7fb7e815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ec864700 1 -- 172.21.15.119:0/1306639634 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e806c630 con 0x7fb7e80adb30 2024-09-13T16:27:29.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ec864700 1 --2- 172.21.15.119:0/1306639634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e80adb30 0x7fb7e80cc060 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80cb210 tx=0x7fb7dc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f31c607dcbaae25 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7e77fe700 1 -- 172.21.15.119:0/1306639634 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc011830 con 0x7fb7e80adb30 2024-09-13T16:27:29.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7e77fe700 1 -- 172.21.15.119:0/1306639634 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc011990 con 0x7fb7e80adb30 2024-09-13T16:27:29.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7e77fe700 1 -- 172.21.15.119:0/1306639634 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc011d20 con 0x7fb7e80adb30 2024-09-13T16:27:29.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- 172.21.15.119:0/1306639634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e80adb30 msgr2=0x7fb7e80cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 --2- 172.21.15.119:0/1306639634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e80adb30 0x7fb7e80cc060 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb7e80cb210 tx=0x7fb7dc009b30 comp rx=0 tx=0).stop 2024-09-13T16:27:29.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- 172.21.15.119:0/1306639634 shutdown_connections 2024-09-13T16:27:29.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 --2- 172.21.15.119:0/1306639634 >> v2:172.21.15.145:3300/0 conn(0x7fb7e806bb90 0x7fb7e80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 --2- 172.21.15.119:0/1306639634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e80adb30 0x7fb7e80cc060 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- 172.21.15.119:0/1306639634 >> 172.21.15.119:0/1306639634 conn(0x7fb7e806a350 msgr2=0x7fb7e806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- 172.21.15.119:0/1306639634 shutdown_connections 2024-09-13T16:27:29.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.255+0000 7fb7ed866700 1 -- 172.21.15.119:0/1306639634 wait complete. 2024-09-13T16:27:29.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 Processor -- start 2024-09-13T16:27:29.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 -- start start 2024-09-13T16:27:29.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e806bb90 0x7fb7e813da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7e80adb30 0x7fb7e813dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7e80cde70 0x7fb7e815c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e815db50 con 0x7fb7e806bb90 2024-09-13T16:27:29.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e815cfd0 con 0x7fb7e80adb30 2024-09-13T16:27:29.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed866700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e815d2b0 con 0x7fb7e80cde70 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ec864700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e806bb90 0x7fb7e813da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ec864700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e806bb90 0x7fb7e813da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39782/0 (socket says 172.21.15.119:39782) 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ec864700 1 -- 172.21.15.119:0/183217020 learned_addr learned my addr 172.21.15.119:0/183217020 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7e7fff700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7e80adb30 0x7fb7e813dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.259+0000 7fb7ed065700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7e80cde70 0x7fb7e815c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ec864700 1 -- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7e80cde70 msgr2=0x7fb7e815c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ec864700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7e80cde70 0x7fb7e815c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ec864700 1 -- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7e80adb30 msgr2=0x7fb7e813dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ec864700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7e80adb30 0x7fb7e813dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ec864700 1 -- 172.21.15.119:0/183217020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc0094b0 con 0x7fb7e806bb90 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ec864700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e806bb90 0x7fb7e813da90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009b70 tx=0x7fb7dc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7e57fa700 1 -- 172.21.15.119:0/183217020 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc004380 con 0x7fb7e806bb90 2024-09-13T16:27:29.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7e57fa700 1 -- 172.21.15.119:0/183217020 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc01f070 con 0x7fb7e806bb90 2024-09-13T16:27:29.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e815d590 con 0x7fb7e806bb90 2024-09-13T16:27:29.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7e57fa700 1 -- 172.21.15.119:0/183217020 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc0036c0 con 0x7fb7e806bb90 2024-09-13T16:27:29.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e813c550 con 0x7fb7e806bb90 2024-09-13T16:27:29.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e806d3f0 con 0x7fb7e806bb90 2024-09-13T16:27:29.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7e57fa700 1 -- 172.21.15.119:0/183217020 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb7dc021030 con 0x7fb7e806bb90 2024-09-13T16:27:29.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7e57fa700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb7d4066730 0x7fb7d4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:29.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.263+0000 7fb7e57fa700 1 -- 172.21.15.119:0/183217020 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb7dc085470 con 0x7fb7e806bb90 2024-09-13T16:27:29.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.267+0000 7fb7e7fff700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb7d4066730 0x7fb7d4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:29.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.267+0000 7fb7e7fff700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb7d4066730 0x7fb7d4068bf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb7e813ed50 tx=0x7fb7d8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:29.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.271+0000 7fb7e57fa700 1 -- 172.21.15.119:0/183217020 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb7dc0527d0 con 0x7fb7e806bb90 2024-09-13T16:27:29.311 INFO:tasks.ceph.mds.a.smithi119.stderr:2024-09-13T16:27:29.311+0000 7f68dc26b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:27:29.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.483+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb7e80ad6a0 con 0x7fb7e806bb90 2024-09-13T16:27:29.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.483+0000 7fb7e57fa700 1 -- 172.21.15.119:0/183217020 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb7dc055df0 con 0x7fb7e806bb90 2024-09-13T16:27:29.486 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:27:29.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb7d4066730 msgr2=0x7fb7d4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb7d4066730 0x7fb7d4068bf0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fb7e813ed50 tx=0x7fb7d8009240 comp rx=0 tx=0).stop 2024-09-13T16:27:29.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e806bb90 msgr2=0x7fb7e813da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:29.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e806bb90 0x7fb7e813da90 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc009b70 tx=0x7fb7dc004770 comp rx=0 tx=0).stop 2024-09-13T16:27:29.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 shutdown_connections 2024-09-13T16:27:29.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb7d4066730 0x7fb7d4068bf0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb7e80cde70 0x7fb7e815c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb7e80adb30 0x7fb7e813dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 --2- 172.21.15.119:0/183217020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb7e806bb90 0x7fb7e813da90 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:29.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 >> 172.21.15.119:0/183217020 conn(0x7fb7e806a350 msgr2=0x7fb7e80af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:29.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 shutdown_connections 2024-09-13T16:27:29.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:29.487+0000 7fb7ed866700 1 -- 172.21.15.119:0/183217020 wait complete. 2024-09-13T16:27:29.500 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:27:29.500 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:27:29.501 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:27:29.501 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:27:29.501 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:27:29.501 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:27:29.501 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:27:29.501 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:27:29.501 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:27:29.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.501 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:27:29.509 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:27:29.509 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:27:29.510 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42492sec preferred_lft 42492sec 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:27:29.511 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:27:29.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.512 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:27:29.527 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:27:29.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.529 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:27:29.579 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:27:29.579 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:27:29.579 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:27:29.579 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:27:29.580 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:27:29.580 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:27:29.580 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:27:29.580 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:27:29.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.580 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:27:29.580 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:27:29.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:27:29.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:27:29.581 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:27:29.581 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:27:29.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.620 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:27:29.625 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:27:29.625 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:27:29.625 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:27:29.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.625 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:27:29.684 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:27:29.684 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:27:29.684 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:27:29.691 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:27:29.692 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:27:29.692 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:27:29.698 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:27:29.711 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:27:29.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:27:29.770 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:27:29.771 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:27:29.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.772 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:27:29.779 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:27:29.779 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:27:29.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:27:29.800 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:27:29.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:27:29.858 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:27:29.859 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:27:29.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:29.860 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:27:29.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:27:29.863+0000 7efd224fd640 -1 init, newargv = 0x55698777c470 newargc=15 2024-09-13T16:27:29.866 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[40288]: starting ceph client 2024-09-13T16:27:29.872 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[40288]: starting fuse 2024-09-13T16:27:30.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:27:30.987 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:27:30.988 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:27:31.051 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:27:31.051 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:27:31.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:31.052 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:27:31.057 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:27:31.057 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:27:31.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:27:31.057 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:27:31.057 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:27:31.058 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:27:31.059 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:27:31.059 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:27:31.059 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:27:31.059 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:27:31.059 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:27:31.059 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:27:31.059 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:27:31.060 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:27:31.060 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:27:31.060 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:27:31.142 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.40288.asok 2024-09-13T16:27:31.325 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.40288.asok 2024-09-13T16:27:31.325 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:31.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.40288.asok status 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:27:31.465 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "40288", 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 6243, 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 6243 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:27:31.466 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4121529632 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4121529632 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.6243 192.168.0.1:0/4121529632", 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/4121529632", 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 24, 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 54, 2024-09-13T16:27:31.467 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:27:31.468 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:27:31.468 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:27:31.468 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:27:31.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:31.475 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:27:31.475 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:27:31.483 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:27:31.483 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:27:31.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:31.483 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:27:31.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:31.496 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent_a) 2024-09-13T16:27:31.562 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:27:31.563 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch parent_a/alpha) 2024-09-13T16:27:31.574 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:27:31.574 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:27:31.574 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:27:31.574 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:27:31.574 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/parent_a/alpha").st_ino) 2024-09-13T16:27:31.574 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:27:31.642 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627777 2024-09-13T16:27:31.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:27:31.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 Processor -- start 2024-09-13T16:27:31.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- start start 2024-09-13T16:27:32.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6c800cd860 0x7f6c800ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c8015d020 con 0x7f6c8015cc40 2024-09-13T16:27:32.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c800d0720 con 0x7f6c8008c6b0 2024-09-13T16:27:32.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c800b44c0 con 0x7f6c800cd860 2024-09-13T16:27:32.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7e59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6c8015cc40 0x7f6c8008bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54126/0 (socket says 172.21.15.119:54126) 2024-09-13T16:27:32.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7e59c700 1 -- 172.21.15.119:0/1636655646 learned_addr learned my addr 172.21.15.119:0/1636655646 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:32.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7ed9d700 1 --2- 172.21.15.119:0/1636655646 >> v2:172.21.15.145:3300/0 conn(0x7f6c800cd860 0x7f6c800ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7dd9b700 1 -- 172.21.15.119:0/1636655646 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674752458 0 0) 0x7f6c8015d020 con 0x7f6c8015cc40 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7dd9b700 1 -- 172.21.15.119:0/1636655646 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c6c0035d0 con 0x7f6c8015cc40 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7ed9d700 1 -- 172.21.15.119:0/1636655646 >> v1:172.21.15.145:6789/0 conn(0x7f6c8008c6b0 legacy=0x7f6c80164a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7ed9d700 1 -- 172.21.15.119:0/1636655646 >> v1:172.21.15.119:6789/0 conn(0x7f6c8015cc40 legacy=0x7f6c8008bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7ed9d700 1 -- 172.21.15.119:0/1636655646 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c800d09c0 con 0x7f6c800cd860 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7ed9d700 1 --2- 172.21.15.119:0/1636655646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c800cd860 0x7f6c800ce710 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009dc0 tx=0x7f6c7800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79622c2ed2fe9757 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7dd9b700 1 -- 172.21.15.119:0/1636655646 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7800e070 con 0x7f6c800cd860 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7dd9b700 1 -- 172.21.15.119:0/1636655646 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7800bae0 con 0x7f6c800cd860 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7dd9b700 1 -- 172.21.15.119:0/1636655646 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7800be50 con 0x7f6c800cd860 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- 172.21.15.119:0/1636655646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c800cd860 msgr2=0x7f6c800ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/1636655646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c800cd860 0x7f6c800ce710 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009dc0 tx=0x7f6c7800b330 comp rx=0 tx=0).stop 2024-09-13T16:27:32.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- 172.21.15.119:0/1636655646 shutdown_connections 2024-09-13T16:27:32.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/1636655646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c800cd860 0x7f6c800ce710 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- 172.21.15.119:0/1636655646 >> 172.21.15.119:0/1636655646 conn(0x7f6c800b4640 msgr2=0x7f6c800b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- 172.21.15.119:0/1636655646 shutdown_connections 2024-09-13T16:27:32.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:31.999+0000 7f6c7f59e700 1 -- 172.21.15.119:0/1636655646 wait complete. 2024-09-13T16:27:32.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 Processor -- start 2024-09-13T16:27:32.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 -- start start 2024-09-13T16:27:32.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c8008c6b0 0x7f6c8015b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c800cd860 0x7f6c8015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c8015cc40 0x7f6c800b7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c800c9c20 con 0x7f6c800cd860 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8015dae0 con 0x7f6c8008c6b0 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8015ddc0 con 0x7f6c8015cc40 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c800cd860 0x7f6c8015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c800cd860 0x7f6c8015d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39796/0 (socket says 172.21.15.119:39796) 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 -- 172.21.15.119:0/2495584363 learned_addr learned my addr 172.21.15.119:0/2495584363 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7e59c700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c8008c6b0 0x7f6c8015b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c7ed9d700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c8015cc40 0x7f6c800b7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 -- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c8015cc40 msgr2=0x7f6c800b7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c8015cc40 0x7f6c800b7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 -- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c8008c6b0 msgr2=0x7f6c8015b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c8008c6b0 0x7f6c8015b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.003+0000 7f6c77fff700 1 -- 172.21.15.119:0/2495584363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78009a70 con 0x7f6c800cd860 2024-09-13T16:27:32.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c77fff700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c800cd860 0x7f6c8015d020 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800c500 tx=0x7f6c6800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c76ffd700 1 -- 172.21.15.119:0/2495584363 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c680052d0 con 0x7f6c800cd860 2024-09-13T16:27:32.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c76ffd700 1 -- 172.21.15.119:0/2495584363 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c68005430 con 0x7f6c800cd860 2024-09-13T16:27:32.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c76ffd700 1 -- 172.21.15.119:0/2495584363 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c680055a0 con 0x7f6c800cd860 2024-09-13T16:27:32.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8015e830 con 0x7f6c800cd860 2024-09-13T16:27:32.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8015b940 con 0x7f6c800cd860 2024-09-13T16:27:32.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8015e690 con 0x7f6c800cd860 2024-09-13T16:27:32.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c76ffd700 1 -- 172.21.15.119:0/2495584363 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6c680038e0 con 0x7f6c800cd860 2024-09-13T16:27:32.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c76ffd700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c60066730 0x7f6c60068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.007+0000 7f6c76ffd700 1 -- 172.21.15.119:0/2495584363 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6c68083500 con 0x7f6c800cd860 2024-09-13T16:27:32.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.011+0000 7f6c7e59c700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c60066730 0x7f6c60068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.011+0000 7f6c7e59c700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c60066730 0x7f6c60068bf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6c70007fd0 tx=0x7f6c70007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.019+0000 7f6c76ffd700 1 -- 172.21.15.119:0/2495584363 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6c8015e690 con 0x7f6c800cd860 2024-09-13T16:27:32.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.219+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c8015ed50 con 0x7f6c800cd860 2024-09-13T16:27:32.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.219+0000 7f6c76ffd700 1 -- 172.21.15.119:0/2495584363 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f6c8015ed50 con 0x7f6c800cd860 2024-09-13T16:27:32.222 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:32.222 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c60066730 msgr2=0x7f6c60068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c60066730 0x7f6c60068bf0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f6c70007fd0 tx=0x7f6c70007de0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c800cd860 msgr2=0x7f6c8015d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c800cd860 0x7f6c8015d020 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f6c6800c500 tx=0x7f6c6800c810 comp rx=0 tx=0).stop 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 shutdown_connections 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c60066730 0x7f6c60068bf0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c8015cc40 0x7f6c800b7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c800cd860 0x7f6c8015d020 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 --2- 172.21.15.119:0/2495584363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c8008c6b0 0x7f6c8015b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 >> 172.21.15.119:0/2495584363 conn(0x7f6c800b4640 msgr2=0x7f6c800b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 shutdown_connections 2024-09-13T16:27:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.223+0000 7f6c7f59e700 1 -- 172.21.15.119:0/2495584363 wait complete. 2024-09-13T16:27:32.227 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:27:32.238 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-13T16:27:32.361 INFO:teuthology.orchestra.run.smithi119.stdout:{"message":"","return_code":0} 2024-09-13T16:27:32.362 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:27:32.362 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-13T16:27:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.395+0000 7fed6f3a2140 1 Processor -- start 2024-09-13T16:27:32.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.395+0000 7fed6f3a2140 1 -- start start 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9847f0f0 0x55bc9847abe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9850b100 0x55bc98434d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b4e0 0x55bc98506bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc983838e0 con 0x55bc9850b100 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc98383580 con 0x55bc9850b4e0 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc983598d0 con 0x55bc9847f0f0 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9850b100 0x55bc98434d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9850b100 0x55bc98434d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39822/0 (socket says 172.21.15.119:39822) 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 -- 172.21.15.119:0/3017937350 learned_addr learned my addr 172.21.15.119:0/3017937350 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6d6d8700 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b4e0 0x55bc98506bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6ced7700 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9847f0f0 0x55bc9847abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 -- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9847f0f0 msgr2=0x55bc9847abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9847f0f0 0x55bc9847abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 -- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b4e0 msgr2=0x55bc98506bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b4e0 0x55bc98506bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 -- 172.21.15.119:0/3017937350 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c005de0 con 0x55bc9850b100 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6c6d6700 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9850b100 0x55bc98434d50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fed5c000c00 tx=0x7fed5c00d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd1dd8eb70a405c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6b6d4700 1 -- 172.21.15.119:0/3017937350 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c005310 con 0x55bc9850b100 2024-09-13T16:27:32.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6b6d4700 1 -- 172.21.15.119:0/3017937350 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c008940 con 0x55bc9850b100 2024-09-13T16:27:32.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6b6d4700 1 -- 172.21.15.119:0/3017937350 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c008cf0 con 0x55bc9850b100 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- 172.21.15.119:0/3017937350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9850b100 msgr2=0x55bc98434d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9850b100 0x55bc98434d50 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fed5c000c00 tx=0x7fed5c00d000 comp rx=0 tx=0).stop 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- 172.21.15.119:0/3017937350 shutdown_connections 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b4e0 0x55bc98506bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9850b100 0x55bc98434d50 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/3017937350 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9847f0f0 0x55bc9847abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- 172.21.15.119:0/3017937350 >> 172.21.15.119:0/3017937350 conn(0x55bc98436110 msgr2=0x55bc98507c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- 172.21.15.119:0/3017937350 shutdown_connections 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- 172.21.15.119:0/3017937350 wait complete. 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 Processor -- start 2024-09-13T16:27:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.399+0000 7fed6f3a2140 1 -- start start 2024-09-13T16:27:32.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9847f0f0 0x55bc98503000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b100 0x55bc98437d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9850b4e0 0x55bc9859a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc98503540 con 0x55bc9847f0f0 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bc98502120 con 0x55bc9850b100 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bc98502380 con 0x55bc9850b4e0 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9847f0f0 0x55bc98503000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9847f0f0 0x55bc98503000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46438/0 (socket says 172.21.15.119:46438) 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 -- 172.21.15.119:0/394624855 learned_addr learned my addr 172.21.15.119:0/394624855 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6c6d6700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b100 0x55bc98437d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6d6d8700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9850b4e0 0x55bc9859a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 -- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9850b4e0 msgr2=0x55bc9859a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9850b4e0 0x55bc9859a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 -- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b100 msgr2=0x55bc98437d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b100 0x55bc98437d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 -- 172.21.15.119:0/394624855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bc98364bb0 con 0x55bc9847f0f0 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6c6d6700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bc9850b100 0x55bc98437d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6d6d8700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bc9850b4e0 0x55bc9859a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6ced7700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9847f0f0 0x55bc98503000 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fed5400e4e0 tx=0x7fed5400e8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6a6d2700 1 -- 172.21.15.119:0/394624855 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed54013870 con 0x55bc9847f0f0 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6a6d2700 1 -- 172.21.15.119:0/394624855 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed540139d0 con 0x55bc9847f0f0 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bc9838bca0 con 0x55bc9847f0f0 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6a6d2700 1 -- 172.21.15.119:0/394624855 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed54013d40 con 0x55bc9847f0f0 2024-09-13T16:27:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.403+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bc98371b00 con 0x55bc9847f0f0 2024-09-13T16:27:32.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.407+0000 7fed6a6d2700 1 -- 172.21.15.119:0/394624855 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fed54014410 con 0x55bc9847f0f0 2024-09-13T16:27:32.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.407+0000 7fed6a6d2700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed4c065090 0x7fed4c067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.407+0000 7fed6a6d2700 1 -- 172.21.15.119:0/394624855 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fed54083820 con 0x55bc9847f0f0 2024-09-13T16:27:32.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.407+0000 7fed6c6d6700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed4c065090 0x7fed4c067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.407+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bc984382b0 0x55bc985a3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.407+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54) v8 -- 0x55bc9859ac00 con 0x55bc984382b0 2024-09-13T16:27:32.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.407+0000 7fed6d6d8700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bc984382b0 0x55bc985a3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6c6d6700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed4c065090 0x7fed4c067550 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed5c01e040 tx=0x7fed5c010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6d6d8700 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bc984382b0 0x55bc985a3e10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6d6d8700 1 -- 172.21.15.119:0/394624855 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7fed64009740 con 0x55bc984382b0 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bc984382b0 msgr2=0x55bc985a3e10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bc984382b0 0x55bc985a3e10 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed4c065090 msgr2=0x7fed4c067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed4c065090 0x7fed4c067550 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fed5c01e040 tx=0x7fed5c010000 comp rx=0 tx=0).stop 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9847f0f0 msgr2=0x55bc98503000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 --2- 172.21.15.119:0/394624855 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bc9847f0f0 0x55bc98503000 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7fed5400e4e0 tx=0x7fed5400e8a0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6d6d8700 1 -- 172.21.15.119:0/394624855 reap_dead start 2024-09-13T16:27:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 shutdown_connections 2024-09-13T16:27:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 >> 172.21.15.119:0/394624855 conn(0x55bc98436110 msgr2=0x55bc98463cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 shutdown_connections 2024-09-13T16:27:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.411+0000 7fed6f3a2140 1 -- 172.21.15.119:0/394624855 wait complete. 2024-09-13T16:27:32.417 INFO:teuthology.orchestra.run.smithi119.stdout:Yalphaparent_a 2024-09-13T16:27:32.417 DEBUG:teuthology.orchestra.run.smithi119:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-13T16:27:32.474 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:27:32.474 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627777, 2024-09-13T16:27:32.474 INFO:teuthology.orchestra.run.smithi119.stdout: "ancestors": [ 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "dirino": 1099511627776, 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "dname": "alpha", 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "version": 2 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "dirino": 1, 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "dname": "parent_a", 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "version": 4 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "pool": 8, 2024-09-13T16:27:32.475 INFO:teuthology.orchestra.run.smithi119.stdout: "old_pools": [] 2024-09-13T16:27:32.476 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:27:32.476 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:32.477 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'oh i'"'"'m sorry did i overwrite your xattr?' 2024-09-13T16:27:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.511+0000 7fe63ad90140 1 Processor -- start 2024-09-13T16:27:32.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.511+0000 7fe63ad90140 1 -- start start 2024-09-13T16:27:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8530 0x5654aca931c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8910 0x5654ac9eaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654ac9fce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ac950a00 con 0x5654ac9b8530 2024-09-13T16:27:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ac94c2f0 con 0x5654ac9b8910 2024-09-13T16:27:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654ac9088e0 con 0x5654aca75c60 2024-09-13T16:27:32.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8530 0x5654aca931c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8530 0x5654aca931c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46456/0 (socket says 172.21.15.119:46456) 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 -- 172.21.15.119:0/377318474 learned_addr learned my addr 172.21.15.119:0/377318474 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6380c4700 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8910 0x5654ac9eaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6390c6700 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654ac9fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 -- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 msgr2=0x5654ac9fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654ac9fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 -- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8910 msgr2=0x5654ac9eaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8910 0x5654ac9eaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 -- 172.21.15.119:0/377318474 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe620005d20 con 0x5654ac9b8530 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6390c6700 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654ac9fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6388c5700 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8530 0x5654aca931c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x5654ac9e9c60 tx=0x7fe62000d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf2143ea8599bc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6370c2700 1 -- 172.21.15.119:0/377318474 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620008a80 con 0x5654ac9b8530 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6370c2700 1 -- 172.21.15.119:0/377318474 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe620008c00 con 0x5654ac9b8530 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe6370c2700 1 -- 172.21.15.119:0/377318474 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe620013480 con 0x5654ac9b8530 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- 172.21.15.119:0/377318474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8530 msgr2=0x5654aca931c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8530 0x5654aca931c0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x5654ac9e9c60 tx=0x7fe62000d000 comp rx=0 tx=0).stop 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- 172.21.15.119:0/377318474 shutdown_connections 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654ac9fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8910 0x5654ac9eaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 --2- 172.21.15.119:0/377318474 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8530 0x5654aca931c0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- 172.21.15.119:0/377318474 >> 172.21.15.119:0/377318474 conn(0x5654ac9b9b50 msgr2=0x5654aca04e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- 172.21.15.119:0/377318474 shutdown_connections 2024-09-13T16:27:32.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- 172.21.15.119:0/377318474 wait complete. 2024-09-13T16:27:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 Processor -- start 2024-09-13T16:27:32.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.515+0000 7fe63ad90140 1 -- start start 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8530 0x5654aca73a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8910 0x5654aca8ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654aca8d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ac908580 con 0x5654ac9b8910 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654ac8de8d0 con 0x5654ac9b8530 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5654ac9ff990 con 0x5654aca75c60 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8910 0x5654aca8ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8910 0x5654aca8ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46458/0 (socket says 172.21.15.119:46458) 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 -- 172.21.15.119:0/1762980955 learned_addr learned my addr 172.21.15.119:0/1762980955 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6390c6700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654aca8d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6388c5700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8530 0x5654aca73a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 -- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 msgr2=0x5654aca8d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5654aca75c60 0x5654aca8d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 -- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8530 msgr2=0x5654aca73a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5654ac9b8530 0x5654aca73a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 -- 172.21.15.119:0/1762980955 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654ac8f6b00 con 0x5654ac9b8910 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6380c4700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8910 0x5654aca8ea20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe63000e4b0 tx=0x7fe63000e7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6360c0700 1 -- 172.21.15.119:0/1762980955 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630001f10 con 0x5654ac9b8910 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6360c0700 1 -- 172.21.15.119:0/1762980955 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe630011350 con 0x5654ac9b8910 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5654ac8e9f40 con 0x5654ac9b8910 2024-09-13T16:27:32.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe6360c0700 1 -- 172.21.15.119:0/1762980955 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe630011700 con 0x5654ac9b8910 2024-09-13T16:27:32.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.519+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5654ac8e99a0 con 0x5654ac9b8910 2024-09-13T16:27:32.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe6360c0700 1 -- 172.21.15.119:0/1762980955 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe630011860 con 0x5654ac9b8910 2024-09-13T16:27:32.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe6360c0700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618065090 0x7fe618067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe6360c0700 1 -- 172.21.15.119:0/1762980955 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe630081ed0 con 0x5654ac9b8910 2024-09-13T16:27:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe63ad90140 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5654aca8f050 0x5654acb25570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [setxattr parent (40) in=46b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) v8 -- 0x5654acb25ab0 con 0x5654aca8f050 2024-09-13T16:27:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe6388c5700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618065090 0x7fe618067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe6390c6700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5654aca8f050 0x5654acb25570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe6390c6700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5654aca8f050 0x5654acb25570 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.523+0000 7fe6388c5700 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618065090 0x7fe618067550 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe620021040 tx=0x7fe620015040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe6390c6700 1 -- 172.21.15.119:0/1762980955 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (40)] v54'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fe6280105b0 con 0x5654aca8f050 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5654aca8f050 msgr2=0x5654acb25570 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5654aca8f050 0x5654acb25570 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618065090 msgr2=0x7fe618067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618065090 0x7fe618067550 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe620021040 tx=0x7fe620015040 comp rx=0 tx=0).stop 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8910 msgr2=0x5654aca8ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 --2- 172.21.15.119:0/1762980955 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5654ac9b8910 0x5654aca8ea20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fe63000e4b0 tx=0x7fe63000e7c0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe6390c6700 1 -- 172.21.15.119:0/1762980955 reap_dead start 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 shutdown_connections 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 >> 172.21.15.119:0/1762980955 conn(0x5654ac9b9b50 msgr2=0x5654ac9ea8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 shutdown_connections 2024-09-13T16:27:32.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.527+0000 7fe63ad90140 1 -- 172.21.15.119:0/1762980955 wait complete. 2024-09-13T16:27:32.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo daemon-helper kill ceph --cluster ceph -w 2024-09-13T16:27:32.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub start / repair,recursive 2024-09-13T16:27:32.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.647+0000 7fd5b8aba700 1 Processor -- start 2024-09-13T16:27:32.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.647+0000 7fd5b8aba700 1 -- start start 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 0x7fd5b4164310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b415a280 con 0x7fd5b41595f0 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b415a560 con 0x7fd5b406e490 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b415a840 con 0x7fd5b406e900 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd5b41595f0 0x7fd5b41599d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:44870/0 (socket says 172.21.15.119:44870) 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 -- 172.21.15.119:0/3732765036 learned_addr learned my addr 172.21.15.119:0/3732765036 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 0x7fd5b4164310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571850228 0 0) 0x7fd5b415a280 con 0x7fd5b41595f0 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd59c0035d0 con 0x7fd5b41595f0 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216180926 0 0) 0x7fd5b415a560 con 0x7fd5b406e490 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5b415a280 con 0x7fd5b406e490 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36969323 0 0) 0x7fd59c0035d0 con 0x7fd5b41595f0 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 msgr2=0x7fd5b4164310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 0x7fd5b4164310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 >> v1:172.21.15.145:6789/0 conn(0x7fd5b406e490 legacy=0x7fd5b4070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b415ab20 con 0x7fd5b41595f0 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fd5a4001670 con 0x7fd5b41595f0 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 >> v1:172.21.15.119:6789/0 conn(0x7fd5b41595f0 legacy=0x7fd5b41599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd59c003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 0x7fd59c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd59c004db0 0x7fd59c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b415a840 con 0x7fd59c003be0 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c00bae0 con 0x7fd59c004db0 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c00bca0 con 0x7fd5b406e900 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd59c003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 0x7fd59c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd59c004db0 0x7fd59c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 -- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 msgr2=0x7fd59c004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b406e900 0x7fd59c004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 -- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd59c004db0 msgr2=0x7fd59c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd59c004db0 0x7fd59c005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 -- 172.21.15.119:0/3732765036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c0035d0 con 0x7fd59c003be0 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd59c004db0 0x7fd59c005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b259c700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd59c003fe0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c160 tx=0x7fd59c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9e559cc19436d15 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a400ccc0 con 0x7fd59c003be0 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a400ce40 con 0x7fd59c003be0 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b159a700 1 -- 172.21.15.119:0/3732765036 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a4005bc0 con 0x7fd59c003be0 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- 172.21.15.119:0/3732765036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 msgr2=0x7fd59c003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 --2- 172.21.15.119:0/3732765036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd59c003fe0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd5a400c160 tx=0x7fd59c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/3732765036 reap_dead start 2024-09-13T16:27:32.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- 172.21.15.119:0/3732765036 shutdown_connections 2024-09-13T16:27:32.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- 172.21.15.119:0/3732765036 >> 172.21.15.119:0/3732765036 conn(0x7fd5b406b460 msgr2=0x7fd5b406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- 172.21.15.119:0/3732765036 shutdown_connections 2024-09-13T16:27:32.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.651+0000 7fd5b8aba700 1 -- 172.21.15.119:0/3732765036 wait complete. 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 Processor -- start 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 -- start start 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd5b41528b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5b4140490 0x7fd5b40c8d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b413b1e0 0x7fd5b406a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b40c9280 con 0x7fd59c003be0 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41f6cb0 con 0x7fd5b4140490 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b8aba700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b41544f0 con 0x7fd5b413b1e0 2024-09-13T16:27:32.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd5b41528b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd5b41528b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46468/0 (socket says 172.21.15.119:46468) 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b259c700 1 -- 172.21.15.119:0/1417966763 learned_addr learned my addr 172.21.15.119:0/1417966763 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5b4140490 0x7fd5b40c8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.655+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b413b1e0 0x7fd5b406a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b259c700 1 -- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b413b1e0 msgr2=0x7fd5b406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b259c700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b413b1e0 0x7fd5b406a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b259c700 1 -- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5b4140490 msgr2=0x7fd5b40c8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b259c700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5b4140490 0x7fd5b40c8d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b259c700 1 -- 172.21.15.119:0/1417966763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b41547d0 con 0x7fd59c003be0 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b413b1e0 0x7fd5b406a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b259c700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd5b41528b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd59c005850 tx=0x7fd5a4001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5a2ffd700 1 -- 172.21.15.119:0/1417966763 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a40070d0 con 0x7fd59c003be0 2024-09-13T16:27:32.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5a2ffd700 1 -- 172.21.15.119:0/1417966763 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a4007230 con 0x7fd59c003be0 2024-09-13T16:27:32.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b4154ab0 con 0x7fd59c003be0 2024-09-13T16:27:32.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5a2ffd700 1 -- 172.21.15.119:0/1417966763 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a40075c0 con 0x7fd59c003be0 2024-09-13T16:27:32.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.659+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b41541a0 con 0x7fd59c003be0 2024-09-13T16:27:32.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5a2ffd700 1 -- 172.21.15.119:0/1417966763 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd5a401e430 con 0x7fd59c003be0 2024-09-13T16:27:32.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5a2ffd700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd594066760 0x7fd594068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5a2ffd700 1 -- 172.21.15.119:0/1417966763 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd5a40855c0 con 0x7fd59c003be0 2024-09-13T16:27:32.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd594066760 0x7fd594068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 Processor -- start 2024-09-13T16:27:32.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 -- start start 2024-09-13T16:27:32.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd594066760 0x7fd594068c20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40cbd10 tx=0x7fd5a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d74a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d74f50 0x1d75330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d75870 0x1d75c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c9f500 con 0x1d74f50 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c9cc90 con 0x1d74630 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c98580 con 0x1d75870 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d74f50 0x1d75330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d74f50 0x1d75330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46482/0 (socket says 172.21.15.119:46482) 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 -- 172.21.15.119:0/1558266752 learned_addr learned my addr 172.21.15.119:0/1558266752 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b259c700 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d74a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d75870 0x1d75c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 -- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d75870 msgr2=0x1d75c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d75870 0x1d75c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 -- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 msgr2=0x1d74a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d74a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 -- 172.21.15.119:0/1558266752 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1b8aee0 con 0x1d74f50 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d75870 0x1d75c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d74f50 0x1d75330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8011840 tx=0x7fd5a8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8965fe7ee8b8e631 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd593fff700 1 -- 172.21.15.119:0/1558266752 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8015640 con 0x1d74f50 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd593fff700 1 -- 172.21.15.119:0/1558266752 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a80157a0 con 0x1d74f50 2024-09-13T16:27:32.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.663+0000 7fd593fff700 1 -- 172.21.15.119:0/1558266752 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8015b10 con 0x1d74f50 2024-09-13T16:27:32.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35db7f700 1 Processor -- start 2024-09-13T16:27:32.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/1558266752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d74f50 msgr2=0x1d75330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d74f50 0x1d75330 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd5a8011840 tx=0x7fd5a8014000 comp rx=0 tx=0).stop 2024-09-13T16:27:32.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35db7f700 1 -- start start 2024-09-13T16:27:32.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/1558266752 shutdown_connections 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d75870 0x1d75c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d74f50 0x1d75330 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/1558266752 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d74a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/1558266752 >> 172.21.15.119:0/1558266752 conn(0x1abf780 msgr2=0x1d59c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/1558266752 shutdown_connections 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/1558266752 wait complete. 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- start start 2024-09-13T16:27:32.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 Processor -- start 2024-09-13T16:27:32.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35db7f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3581570e0 0x7fc35815db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35db7f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc3580cebb0 0x7fc35806beb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35db7f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc358070890 con 0x7fc3580ac8c0 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35db7f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc35815e040 con 0x7fc3580cebb0 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35db7f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580ad8e0 con 0x7fc3581570e0 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3581570e0 0x7fc35815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3581570e0 0x7fc35815db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:54064/0 (socket says 172.21.15.119:54064) 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35d37e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc3580cebb0 0x7fc35806beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 -- 172.21.15.119:0/3910804765 learned_addr learned my addr 172.21.15.119:0/3910804765 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc3577fe700 1 -- 172.21.15.119:0/3910804765 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065954010 0 0) 0x7fc358070890 con 0x7fc3580ac8c0 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 -- 172.21.15.119:0/3910804765 >> v2:172.21.15.145:3300/0 conn(0x7fc3580cebb0 msgr2=0x7fc35806beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 --2- 172.21.15.119:0/3910804765 >> v2:172.21.15.145:3300/0 conn(0x7fc3580cebb0 0x7fc35806beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 -- 172.21.15.119:0/3910804765 >> v1:172.21.15.145:6789/0 conn(0x7fc3580ac8c0 legacy=0x7fc3580ce4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 -- 172.21.15.119:0/3910804765 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3580b1a90 con 0x7fc3581570e0 2024-09-13T16:27:32.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d56bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1d2f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d2fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c5b920 con 0x1d75870 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c5b5c0 con 0x1d74630 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c2de10 con 0x1d74f50 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5bbd6b740 10 client.0 _ll_register_callbacks cb 0x1c91bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d2fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d2fb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46484/0 (socket says 172.21.15.119:46484) 2024-09-13T16:27:32.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc35cb7d700 1 --2- 172.21.15.119:0/3910804765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3581570e0 0x7fc35815db00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc344009910 tx=0x7fc344009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3be9afa9e67185e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2980489877 learned_addr learned my addr 172.21.15.119:0/2980489877 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b259c700 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d56bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1d2f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 msgr2=0x1d2f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1d2f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 msgr2=0x1d56bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d56bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc3577fe700 1 -- 172.21.15.119:0/3910804765 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344011830 con 0x7fc3581570e0 2024-09-13T16:27:32.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2980489877 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5a800f040 con 0x1d75870 2024-09-13T16:27:32.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc3577fe700 1 -- 172.21.15.119:0/3910804765 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc344011990 con 0x7fc3581570e0 2024-09-13T16:27:32.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.667+0000 7fc3577fe700 1 -- 172.21.15.119:0/3910804765 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc344011d40 con 0x7fc3581570e0 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d2fb80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac004570 tx=0x7fd5ac004880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a43428fe2865ad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 1 -- 172.21.15.119:0/2980489877 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac003e90 con 0x1d75870 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1acc490 con 0x1d75870 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 1 -- 172.21.15.119:0/2980489877 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd5ac0048f0 con 0x1d75870 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 1 -- 172.21.15.119:0/2980489877 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5ac0079b0 con 0x1d75870 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 10 client.6282 fetch_fsmap learned FSMap version 24 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x1c5e600 con 0x1d75870 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- 172.21.15.119:0/3910804765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3581570e0 msgr2=0x7fc35815db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 --2- 172.21.15.119:0/3910804765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3581570e0 0x7fc35815db00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fc344009910 tx=0x7fc344009c20 comp rx=0 tx=0).stop 2024-09-13T16:27:32.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 1 -- 172.21.15.119:0/2980489877 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1acc490 con 0x1d75870 2024-09-13T16:27:32.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- 172.21.15.119:0/3910804765 shutdown_connections 2024-09-13T16:27:32.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 --2- 172.21.15.119:0/3910804765 >> v2:172.21.15.145:3300/0 conn(0x7fc3580cebb0 0x7fc35806beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 --2- 172.21.15.119:0/3910804765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3581570e0 0x7fc35815db00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- 172.21.15.119:0/3910804765 >> 172.21.15.119:0/3910804765 conn(0x7fc35806a350 msgr2=0x7fc35806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- 172.21.15.119:0/3910804765 shutdown_connections 2024-09-13T16:27:32.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 1 -- 172.21.15.119:0/2980489877 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd5ac016070 con 0x1d75870 2024-09-13T16:27:32.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- 172.21.15.119:0/3910804765 wait complete. 2024-09-13T16:27:32.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 10 client.6282 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:27:32.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 10 client.6282 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-09-13T16:27:32.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 20 client.6282 populate_metadata read hostname 'smithi119' 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d32710 0x1d2dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 4 client.6282 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1b16710 con 0x1d32710 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5b259c700 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d32710 0x1d2dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5b259c700 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d32710 0x1d2dbb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61b8d900d67ba820 server_cookie=ec700fb3b071290c in_seq=0 out_seq=0 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 10 client.6282 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 1 -- 172.21.15.119:0/2980489877 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1b16710 con 0x1d32710 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 10 client.6282 handle_command_reply: tid=0 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 client.6282 shutdown 2024-09-13T16:27:32.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 Processor -- start 2024-09-13T16:27:32.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 msgr2=0x1d2fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d2fb80 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac004570 tx=0x7fd5ac004880 comp rx=0 tx=0).stop 2024-09-13T16:27:32.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- start start 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 shutdown_connections 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d2fb80 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1d2f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d32710 0x1d2dbb0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2980489877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d56bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5917fa700 0 client.6282 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 >> 172.21.15.119:0/2980489877 conn(0x1abf780 msgr2=0x1d5e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 shutdown_connections 2024-09-13T16:27:32.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2980489877 wait complete. 2024-09-13T16:27:32.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fd5bbd6b740 20 client.6282 trim_cache size 0 max 16384 2024-09-13T16:27:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3580ac8c0 0x7fc3581f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3580cebb0 0x7fc3581f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3581570e0 0x7fc3581576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580b0540 con 0x7fc3581570e0 2024-09-13T16:27:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3581f2070 con 0x7fc3580cebb0 2024-09-13T16:27:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35db7f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35815a020 con 0x7fc3580ac8c0 2024-09-13T16:27:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35d37e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3581570e0 0x7fc3581576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35d37e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3581570e0 0x7fc3581576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46494/0 (socket says 172.21.15.119:46494) 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35d37e700 1 -- 172.21.15.119:0/2503142107 learned_addr learned my addr 172.21.15.119:0/2503142107 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.671+0000 7fc35cb7d700 1 --2- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3580ac8c0 0x7fc3581f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35d37e700 1 -- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3580ac8c0 msgr2=0x7fc3581f1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35d37e700 1 --2- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3580ac8c0 0x7fc3581f1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35d37e700 1 -- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3580cebb0 msgr2=0x7fc3581f6140 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35d37e700 1 --2- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3580cebb0 0x7fc3581f6140 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35d37e700 1 -- 172.21.15.119:0/2503142107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344009570 con 0x7fc3581570e0 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35d37e700 1 --2- 172.21.15.119:0/2503142107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3581570e0 0x7fc3581576d0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fc35000efc0 tx=0x7fc35000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc35000cd40 con 0x7fc3581570e0 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc35000f040 con 0x7fc3581570e0 2024-09-13T16:27:32.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35db7f700 1 -- 172.21.15.119:0/2503142107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3581f6680 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3500146c0 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35db7f700 1 -- 172.21.15.119:0/2503142107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35815a270 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35db7f700 1 -- 172.21.15.119:0/2503142107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc3581574c0 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc350014820 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc3557fa700 1 --2- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc33c066760 0x7fc33c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc3500858c0 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35cb7d700 1 --2- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc33c066760 0x7fc33c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+501 (secure 0 0 0) 0x7fc350085c50 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.675+0000 7fc35db7f700 1 -- 172.21.15.119:0/2503142107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7fc3580b29f0 con 0x7fc3581570e0 2024-09-13T16:27:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.679+0000 7fc35cb7d700 1 --2- 172.21.15.119:0/2503142107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc33c066760 0x7fc33c068c20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fc344014040 tx=0x7fc344008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 Processor -- start 2024-09-13T16:27:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 -- start start 2024-09-13T16:27:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74630 0x1d739d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74f50 0x1d55ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d563e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c365a0 con 0x1d75870 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c93d70 con 0x1d74f50 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d2e010 con 0x1d74630 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46498/0 (socket says 172.21.15.119:46498) 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2692014059 learned_addr learned my addr 172.21.15.119:0/2692014059 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74f50 0x1d55ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b259c700 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74630 0x1d739d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74630 msgr2=0x1d739d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74630 0x1d739d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74f50 msgr2=0x1d55ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74f50 0x1d55ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/2692014059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1c44830 con 0x1d75870 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.687+0000 7fd5b259c700 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74630 0x1d739d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d563e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac007610 tx=0x7fd5ac011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc321dc80bbdc8a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5a0ff9700 1 -- 172.21.15.119:0/2692014059 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac003de0 con 0x1d75870 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5a0ff9700 1 -- 172.21.15.119:0/2692014059 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5ac011070 con 0x1d75870 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5a0ff9700 1 -- 172.21.15.119:0/2692014059 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac004880 con 0x1d75870 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2692014059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 msgr2=0x1d563e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d563e0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac007610 tx=0x7fd5ac011000 comp rx=0 tx=0).stop 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2692014059 shutdown_connections 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1d563e0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74f50 0x1d55ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/2692014059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74630 0x1d739d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2692014059 >> 172.21.15.119:0/2692014059 conn(0x1abf780 msgr2=0x1d8fe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2692014059 shutdown_connections 2024-09-13T16:27:32.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/2692014059 wait complete. 2024-09-13T16:27:32.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 -- start start 2024-09-13T16:27:32.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.691+0000 7fd5bbd6b740 1 Processor -- start 2024-09-13T16:27:32.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d72790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1ddc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1ddca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c96c50 con 0x1d75870 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c9b4e0 con 0x1d74630 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ddcf50 con 0x1d74f50 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 10 client.0 _ll_register_callbacks cb 0x1c97150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1ddca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1ddca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46506/0 (socket says 172.21.15.119:46506) 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/295146334 learned_addr learned my addr 172.21.15.119:0/295146334 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b259c700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d72790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1ddc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 msgr2=0x1ddc4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1ddc4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 msgr2=0x1d72790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d72790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 -- 172.21.15.119:0/295146334 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1c5e600 con 0x1d75870 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b1d9b700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1ddc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b2d9d700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1ddca10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac02c040 tx=0x7fd5ac004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19ce3cc4feb6bc11 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5917fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5917fa700 1 -- 172.21.15.119:0/295146334 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5ac004880 con 0x1d75870 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5917fa700 1 -- 172.21.15.119:0/295146334 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd5ac007dc0 con 0x1d75870 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5a80157a0 con 0x1d75870 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5917fa700 1 -- 172.21.15.119:0/295146334 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd5ac0299a0 con 0x1d75870 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 10 client.6291 fetch_fsmap learned FSMap version 24 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x1b0ac10 con 0x1d75870 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5917fa700 1 -- 172.21.15.119:0/295146334 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a80157a0 con 0x1d75870 2024-09-13T16:27:32.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5917fa700 1 -- 172.21.15.119:0/295146334 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd5ac029b00 con 0x1d75870 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 10 client.6291 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 10 client.6291 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 20 client.6291 populate_metadata read hostname 'smithi119' 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d72cd0 0x1d8c560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 4 client.6291 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x1b3d5f0 con 0x1d72cd0 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b259c700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d72cd0 0x1d8c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5917fa700 10 client.6291 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.695+0000 7fd5b259c700 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d72cd0 0x1d8c560 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adc3821d464bed server_cookie=4573f318e8eb1c6b in_seq=0 out_seq=0 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5917fa700 1 -- 172.21.15.119:0/295146334 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x1b3d5f0 con 0x1d72cd0 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5917fa700 10 client.6291 handle_command_reply: tid=0 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 client.6291 shutdown 2024-09-13T16:27:32.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 msgr2=0x1ddca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1ddca10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fd5ac02c040 tx=0x7fd5ac004b60 comp rx=0 tx=0).stop 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 shutdown_connections 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d75870 0x1ddca10 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1d74f50 0x1ddc4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1d72cd0 0x1d8c560 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 --2- 172.21.15.119:0/295146334 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d74630 0x1d72790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5917fa700 0 client.6291 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 >> 172.21.15.119:0/295146334 conn(0x1abf780 msgr2=0x1d55ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 shutdown_connections 2024-09-13T16:27:32.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 1 -- 172.21.15.119:0/295146334 wait complete. 2024-09-13T16:27:32.705 INFO:tasks.ceph.mds.a.smithi119.stderr:2024-09-13T16:27:32.699+0000 7f68d8a64700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6243=0-4194304@1} caps={6243=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6243 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55a8de063700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:27:32.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5bbd6b740 20 client.6291 trim_cache size 0 max 16384 2024-09-13T16:27:32.705 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:27:32.705 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:27:32.705 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "25755bbe-7e4a-4e81-8981-bcd38c396899", 2024-09-13T16:27:32.705 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:27:32.706 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:27:32.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd594066760 msgr2=0x7fd594068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd594066760 0x7fd594068c20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd5b40cbd10 tx=0x7fd5a8009040 comp rx=0 tx=0).stop 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 msgr2=0x7fd5b41528b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd5b41528b0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd59c005850 tx=0x7fd5a4001930 comp rx=0 tx=0).stop 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 shutdown_connections 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd594066760 0x7fd594068c20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd5b413b1e0 0x7fd5b406a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd5b4140490 0x7fd5b40c8d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 --2- 172.21.15.119:0/1417966763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd59c003be0 0x7fd5b41528b0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 >> 172.21.15.119:0/1417966763 conn(0x7fd5b406b460 msgr2=0x7fd5b4066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 shutdown_connections 2024-09-13T16:27:32.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.699+0000 7fd5b8aba700 1 -- 172.21.15.119:0/1417966763 wait complete. 2024-09-13T16:27:32.715 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-09-13T16:27:32.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.827+0000 7f46dcd8b700 1 Processor -- start 2024-09-13T16:27:32.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.827+0000 7f46dcd8b700 1 -- start start 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.827+0000 7f46dcd8b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f46d806ca60 0x7f46d8159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.827+0000 7f46dcd8b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80b1860 0x7f46d815a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.827+0000 7f46dcd8b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46d80b1dd0 con 0x7f46d80ad9a0 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.827+0000 7f46dcd8b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46d80b2fd0 con 0x7f46d806ca60 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.827+0000 7f46dcd8b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d815a520 con 0x7f46d80b1860 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d77fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f46d80ad9a0 0x7f46d806c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:44876/0 (socket says 172.21.15.119:44876) 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d77fe700 1 -- 172.21.15.119:0/281102681 learned_addr learned my addr 172.21.15.119:0/281102681 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/281102681 >> v2:172.21.15.145:3300/0 conn(0x7f46d806ca60 0x7f46d8159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d7fff700 1 --2- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80b1860 0x7f46d815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d67fc700 1 -- 172.21.15.119:0/281102681 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1876116562 0 0) 0x7f46d80b1dd0 con 0x7f46d80ad9a0 2024-09-13T16:27:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d67fc700 1 -- 172.21.15.119:0/281102681 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c40035d0 con 0x7f46d80ad9a0 2024-09-13T16:27:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d6ffd700 1 -- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80b1860 msgr2=0x7f46d815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80b1860 0x7f46d815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d6ffd700 1 -- 172.21.15.119:0/281102681 >> v1:172.21.15.119:6789/0 conn(0x7f46d80ad9a0 legacy=0x7f46d806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d67fc700 1 -- 172.21.15.119:0/281102681 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3955791833 0 0) 0x7f46c40035d0 con 0x7f46d80ad9a0 2024-09-13T16:27:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d6ffd700 1 -- 172.21.15.119:0/281102681 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d80b20b0 con 0x7f46d806ca60 2024-09-13T16:27:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 0x7f46d8159de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f46c000baf0 tx=0x7f46c000be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d91cc6fcfc286b53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d67fc700 1 -- 172.21.15.119:0/281102681 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c8001690 con 0x7f46d806ca60 2024-09-13T16:27:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d67fc700 1 -- 172.21.15.119:0/281102681 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c000da70 con 0x7f46d806ca60 2024-09-13T16:27:32.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46d67fc700 1 -- 172.21.15.119:0/281102681 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c000dde0 con 0x7f46d806ca60 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 -- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 msgr2=0x7f46d8159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 0x7f46d8159de0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f46c000baf0 tx=0x7f46c000be00 comp rx=0 tx=0).stop 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 -- 172.21.15.119:0/281102681 shutdown_connections 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80b1860 0x7f46d815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/281102681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 0x7f46d8159de0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 -- 172.21.15.119:0/281102681 >> 172.21.15.119:0/281102681 conn(0x7f46d80b0980 msgr2=0x7f46d80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 -- 172.21.15.119:0/281102681 shutdown_connections 2024-09-13T16:27:32.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.831+0000 7f46dcd8b700 1 -- 172.21.15.119:0/281102681 wait complete. 2024-09-13T16:27:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 Processor -- start 2024-09-13T16:27:32.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 -- start start 2024-09-13T16:27:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 0x7f46d81325c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80ad9a0 0x7f46d80ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d80b1860 0x7f46d80ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d814d910 con 0x7f46d80b1860 2024-09-13T16:27:32.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d8155220 con 0x7f46d806ca60 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46d815d5b0 con 0x7f46d80ad9a0 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d80b1860 0x7f46d80ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d80b1860 0x7f46d80ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46508/0 (socket says 172.21.15.119:46508) 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 -- 172.21.15.119:0/2295372331 learned_addr learned my addr 172.21.15.119:0/2295372331 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80ad9a0 0x7f46d80ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d77fe700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 0x7f46d81325c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 -- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80ad9a0 msgr2=0x7f46d80ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80ad9a0 0x7f46d80ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 -- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 msgr2=0x7f46d81325c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 0x7f46d81325c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 -- 172.21.15.119:0/2295372331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c000b750 con 0x7f46d80b1860 2024-09-13T16:27:32.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46d7fff700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d80b1860 0x7f46d80ace00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f46d000efc0 tx=0x7f46d000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46cffff700 1 -- 172.21.15.119:0/2295372331 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0009ba0 con 0x7f46d80b1860 2024-09-13T16:27:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46cffff700 1 -- 172.21.15.119:0/2295372331 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d000f040 con 0x7f46d80b1860 2024-09-13T16:27:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d8154bd0 con 0x7f46d80b1860 2024-09-13T16:27:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46cffff700 1 -- 172.21.15.119:0/2295372331 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0014480 con 0x7f46d80b1860 2024-09-13T16:27:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.835+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d80cdc20 con 0x7f46d80b1860 2024-09-13T16:27:32.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.839+0000 7f46cffff700 1 -- 172.21.15.119:0/2295372331 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f46d00145e0 con 0x7f46d80b1860 2024-09-13T16:27:32.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.839+0000 7f46cffff700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46b8066760 0x7f46b8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.839+0000 7f46cffff700 1 -- 172.21.15.119:0/2295372331 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f46d0084ae0 con 0x7f46d80b1860 2024-09-13T16:27:32.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.839+0000 7f46d77fe700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46b8066760 0x7f46b8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.839+0000 7f46d77fe700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46b8066760 0x7f46b8068c20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f46d80b1d00 tx=0x7f46c8000ed0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 Processor -- start 2024-09-13T16:27:32.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- start start 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a49d90 0x2a00810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29ff310 0x2a28520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a4a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2991010 con 0x2a4a170 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x291ef40 con 0x29ff310 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2971e30 con 0x2a49d90 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a4a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a4a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46512/0 (socket says 172.21.15.119:46512) 2024-09-13T16:27:32.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 -- 172.21.15.119:0/3565737266 learned_addr learned my addr 172.21.15.119:0/3565737266 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29ff310 0x2a28520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d77fe700 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a49d90 0x2a00810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 -- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a49d90 msgr2=0x2a00810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a49d90 0x2a00810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 -- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29ff310 msgr2=0x2a28520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29ff310 0x2a28520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 -- 172.21.15.119:0/3565737266 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x27dda40 con 0x2a4a170 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46d7fff700 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a4a550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f46d0051e80 tx=0x7f46d0055400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70e3f531c85f327f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46cd7fa700 1 -- 172.21.15.119:0/3565737266 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0059800 con 0x2a4a170 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46cd7fa700 1 -- 172.21.15.119:0/3565737266 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d005ddc0 con 0x2a4a170 2024-09-13T16:27:32.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46cd7fa700 1 -- 172.21.15.119:0/3565737266 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d006d1c0 con 0x2a4a170 2024-09-13T16:27:32.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- 172.21.15.119:0/3565737266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 msgr2=0x2a4a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a4a550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f46d0051e80 tx=0x7f46d0055400 comp rx=0 tx=0).stop 2024-09-13T16:27:32.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- 172.21.15.119:0/3565737266 shutdown_connections 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a4a550 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29ff310 0x2a28520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 --2- 172.21.15.119:0/3565737266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a49d90 0x2a00810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- 172.21.15.119:0/3565737266 >> 172.21.15.119:0/3565737266 conn(0x27925b0 msgr2=0x2a03b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- 172.21.15.119:0/3565737266 shutdown_connections 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.843+0000 7f46e129e740 1 -- 172.21.15.119:0/3565737266 wait complete. 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 -- start start 2024-09-13T16:27:32.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 Processor -- start 2024-09-13T16:27:32.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a298c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2a489b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2a2b800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x296d150 con 0x2a49d90 2024-09-13T16:27:32.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x296aeb0 con 0x2a4a170 2024-09-13T16:27:32.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x29288e0 con 0x29ff310 2024-09-13T16:27:32.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2a489b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2a489b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46514/0 (socket says 172.21.15.119:46514) 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 10 client.0 _ll_register_callbacks cb 0x2907670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 -- 172.21.15.119:0/324773284 learned_addr learned my addr 172.21.15.119:0/324773284 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d77fe700 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a298c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d7fff700 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2a2b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 -- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 msgr2=0x2a298c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a298c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 -- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 msgr2=0x2a2b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2a2b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 -- 172.21.15.119:0/324773284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d80cdc20 con 0x2a49d90 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2a489b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f46d806dcb0 tx=0x7f46c00047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17346026828997f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46b67fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46b67fc700 1 -- 172.21.15.119:0/324773284 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c0004470 con 0x2a49d90 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46b67fc700 1 -- 172.21.15.119:0/324773284 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f46c00036c0 con 0x2a49d90 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x279f2c0 con 0x2a49d90 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46b67fc700 1 -- 172.21.15.119:0/324773284 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46c00045d0 con 0x2a49d90 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 10 client.6303 fetch_fsmap learned FSMap version 24 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x2931290 con 0x2a49d90 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.847+0000 7f46b67fc700 1 -- 172.21.15.119:0/324773284 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x279f2c0 con 0x2a49d90 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46b67fc700 1 -- 172.21.15.119:0/324773284 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f46c0003880 con 0x2a49d90 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 10 client.6303 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 10 client.6303 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 20 client.6303 populate_metadata read hostname 'smithi119' 2024-09-13T16:27:32.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a5a690 0x2a5ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 4 client.6303 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:27:32.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x27e9540 con 0x2a5a690 2024-09-13T16:27:32.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46d77fe700 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a5a690 0x2a5ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46b67fc700 10 client.6303 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46d77fe700 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a5a690 0x2a5ca80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=83ccc857e3b90287 server_cookie=730aff5661d8d0c1 in_seq=0 out_seq=0 2024-09-13T16:27:32.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46b67fc700 1 -- 172.21.15.119:0/324773284 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x27e9540 con 0x2a5a690 2024-09-13T16:27:32.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46b67fc700 10 client.6303 handle_command_reply: tid=0 2024-09-13T16:27:32.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 client.6303 shutdown 2024-09-13T16:27:32.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 msgr2=0x2a489b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2a489b0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f46d806dcb0 tx=0x7f46c00047e0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 shutdown_connections 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a5a690 0x2a5ca80 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2a2b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46b67fc700 0 client.6303 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2a489b0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 --2- 172.21.15.119:0/324773284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a298c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 >> 172.21.15.119:0/324773284 conn(0x27925b0 msgr2=0x2a284b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 shutdown_connections 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 1 -- 172.21.15.119:0/324773284 wait complete. 2024-09-13T16:27:32.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.851+0000 7f46e129e740 20 client.6303 trim_cache size 0 max 16384 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 Processor -- start 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 -- start start 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a5eea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a49d90 0x2a253f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a25930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2816b80 con 0x2a4a170 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x292b810 con 0x2a49d90 2024-09-13T16:27:32.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.871+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x296b760 con 0x29ff310 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a25930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a25930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46528/0 (socket says 172.21.15.119:46528) 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 -- 172.21.15.119:0/780668497 learned_addr learned my addr 172.21.15.119:0/780668497 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a49d90 0x2a253f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d77fe700 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a5eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 -- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 msgr2=0x2a5eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a5eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 -- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a49d90 msgr2=0x2a253f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a49d90 0x2a253f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 -- 172.21.15.119:0/780668497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x29174c0 con 0x2a4a170 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d77fe700 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a5eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:27:32.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46d7fff700 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a25930 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f46d0055960 tx=0x7f46d0057000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0c289c2a4d4724 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46cdffb700 1 -- 172.21.15.119:0/780668497 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0059800 con 0x2a4a170 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46cdffb700 1 -- 172.21.15.119:0/780668497 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d0053d70 con 0x2a4a170 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46cdffb700 1 -- 172.21.15.119:0/780668497 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d006d400 con 0x2a4a170 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 -- 172.21.15.119:0/780668497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 msgr2=0x2a25930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a25930 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f46d0055960 tx=0x7f46d0057000 comp rx=0 tx=0).stop 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 -- 172.21.15.119:0/780668497 shutdown_connections 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a4a170 0x2a25930 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a49d90 0x2a253f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 --2- 172.21.15.119:0/780668497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a5eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 -- 172.21.15.119:0/780668497 >> 172.21.15.119:0/780668497 conn(0x27925b0 msgr2=0x2a47860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 -- 172.21.15.119:0/780668497 shutdown_connections 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 -- 172.21.15.119:0/780668497 wait complete. 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.875+0000 7f46e129e740 1 -- start start 2024-09-13T16:27:32.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 Processor -- start 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a69080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2aaaf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2aab4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a03190 con 0x2a49d90 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a26610 con 0x2a4a170 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a268b0 con 0x29ff310 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 10 client.0 _ll_register_callbacks cb 0x2969610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2aaaf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2aaaf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46542/0 (socket says 172.21.15.119:46542) 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d6ffd700 1 -- 172.21.15.119:0/3633513747 learned_addr learned my addr 172.21.15.119:0/3633513747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d77fe700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a69080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d7fff700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2aab4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d7fff700 1 -- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 msgr2=0x2a69080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d7fff700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a69080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d7fff700 1 -- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 msgr2=0x2aaaf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d7fff700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2aaaf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d7fff700 1 -- 172.21.15.119:0/3633513747 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x27dda40 con 0x2a4a170 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d6ffd700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2aaaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46d7fff700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2aab4a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x2a25280 tx=0x7f46d0009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd3e1939badf6c32 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46b67fc700 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-09-13T16:27:32.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46b67fc700 1 -- 172.21.15.119:0/3633513747 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46d0051d10 con 0x2a4a170 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46b67fc700 1 -- 172.21.15.119:0/3633513747 <== mon.1 v2:172.21.15.145:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f46d0071070 con 0x2a4a170 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f46d005ddc0 con 0x2a4a170 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.879+0000 7f46b67fc700 1 -- 172.21.15.119:0/3633513747 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46d0059800 con 0x2a4a170 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 10 client.6100 fetch_fsmap learned FSMap version 24 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x290b790 con 0x2a4a170 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46b67fc700 1 -- 172.21.15.119:0/3633513747 <== mon.1 v2:172.21.15.145:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46d005ddc0 con 0x2a4a170 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46b67fc700 1 -- 172.21.15.119:0/3633513747 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f46d0066050 con 0x2a4a170 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 10 client.6100 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 10 client.6100 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 20 client.6100 populate_metadata read hostname 'smithi119' 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a25310 0x2a25720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 4 client.6100 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:27:32.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x280e2c0 con 0x2a25310 2024-09-13T16:27:32.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46d77fe700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a25310 0x2a25720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:27:32.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46b67fc700 10 client.6100 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46d77fe700 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a25310 0x2a25720 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa36f46b72f97c5e server_cookie=a76e9301fb333d83 in_seq=0 out_seq=0 2024-09-13T16:27:32.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46b67fc700 1 -- 172.21.15.119:0/3633513747 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x280e2c0 con 0x2a25310 2024-09-13T16:27:32.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46b67fc700 10 client.6100 handle_command_reply: tid=0 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 client.6100 shutdown 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 msgr2=0x2aab4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2aab4a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x2a25280 tx=0x7f46d0009f90 comp rx=0 tx=0).stop 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 shutdown_connections 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x2a25310 0x2a25720 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a4a170 0x2aab4a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46b67fc700 0 client.6100 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a49d90 0x2aaaf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 --2- 172.21.15.119:0/3633513747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29ff310 0x2a69080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 >> 172.21.15.119:0/3633513747 conn(0x27925b0 msgr2=0x2a48730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 shutdown_connections 2024-09-13T16:27:32.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 1 -- 172.21.15.119:0/3633513747 wait complete. 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46e129e740 20 client.6100 trim_cache size 0 max 16384 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout: "25755bbe-7e4a-4e81-8981-bcd38c396899": { 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "25755bbe-7e4a-4e81-8981-bcd38c396899", 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,repair" 2024-09-13T16:27:32.890 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:32.891 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:27:32.891 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46b8066760 msgr2=0x7f46b8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.883+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46b8066760 0x7f46b8068c20 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f46d80b1d00 tx=0x7f46c8000ed0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d80b1860 msgr2=0x7f46d80ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d80b1860 0x7f46d80ace00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f46d000efc0 tx=0x7f46d000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 shutdown_connections 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f46b8066760 0x7f46b8068c20 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46d80b1860 0x7f46d80ace00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46d80ad9a0 0x7f46d80ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 --2- 172.21.15.119:0/2295372331 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46d806ca60 0x7f46d81325c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 >> 172.21.15.119:0/2295372331 conn(0x7f46d80b0980 msgr2=0x7f46d813c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:27:32.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 shutdown_connections 2024-09-13T16:27:32.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:32.887+0000 7f46dcd8b700 1 -- 172.21.15.119:0/2295372331 wait complete. 2024-09-13T16:27:32.902 INFO:tasks.cephfs.filesystem:scrub status for tag:25755bbe-7e4a-4e81-8981-bcd38c396899 - {'path': '/', 'tag': '25755bbe-7e4a-4e81-8981-bcd38c396899', 'options': 'recursive,repair'} 2024-09-13T16:27:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:33.531+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 7 ==== log(6 entries from seq 1 at 2024-09-13T16:27:32.698862+0000) v1 ==== 1078+0+0 (secure 0 0 0) 0x7fc350056ea0 con 0x7fc3581570e0 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: cluster: 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: id: 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: health: HEALTH_OK 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: services: 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: mon: 3 daemons, quorum a,b,c (age 2m) 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: mgr: x(active, since 2m), standbys: y 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: mds: 1/1 daemons up, 3 standby 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: osd: 8 osds: 8 up (since 2m), 8 in (since 2m) 2024-09-13T16:27:33.534 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: data: 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: volumes: 1/1 healthy 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: pools: 3 pools, 97 pgs 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: objects: 24 objects, 579 KiB 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: usage: 221 MiB used, 720 GiB / 720 GiB avail 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: pgs: 97 active+clean 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: io: 2024-09-13T16:27:33.535 INFO:teuthology.orchestra.run.smithi119.stdout: client: 1.8 KiB/s wr, 0 op/s rd, 5 op/s wr 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:27:32.698862+0000 mds.a [INF] scrub queued for path: / 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:27:32.698865+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:27:32.698869+0000 mds.a [INF] scrub summary: active paths [/] 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:27:32.700139+0000 mds.a [WRN] bad backtrace on inode 0x10000000001(/parent_a/alpha), rewriting it 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:27:32.700151+0000 mds.a [INF] Scrub repaired inode 0x10000000001 (/parent_a/alpha) 2024-09-13T16:27:33.536 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:27:32.700969+0000 mds.a [INF] scrub summary: idle+waiting paths [/] 2024-09-13T16:27:35.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:27:35.534+0000 7fc3557fa700 1 -- 172.21.15.119:0/2503142107 <== mon.0 v2:172.21.15.119:3300/0 8 ==== log(1 entries from seq 7 at 2024-09-13T16:27:33.878458+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc3500567a0 con 0x7fc3581570e0 2024-09-13T16:27:35.537 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:27:33.878458+0000 mds.a [INF] scrub summary: idle 2024-09-13T16:27:57.179 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:27:57.182 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:28:02.903 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.3:0 scrub status 2024-09-13T16:28:03.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa03c8bb700 1 Processor -- start 2024-09-13T16:28:03.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa03c8bb700 1 -- start start 2024-09-13T16:28:03.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa03c8bb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa038160210 0x7fa0380d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa03c8bb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d3760 0x7fa0380d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa03c8bb700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0380b2240 con 0x7fa0380d1610 2024-09-13T16:28:03.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa03c8bb700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0380d1ac0 con 0x7fa038160210 2024-09-13T16:28:03.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa03c8bb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0381605f0 con 0x7fa0380d3760 2024-09-13T16:28:03.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa0377fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa038160210 0x7fa0380d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa037fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa0380d1610 0x7fa0380cd620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34326/0 (socket says 172.21.15.119:34326) 2024-09-13T16:28:03.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa037fff700 1 -- 172.21.15.119:0/2034000022 learned_addr learned my addr 172.21.15.119:0/2034000022 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:03.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa0377fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa038160210 0x7fa0380d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33054/0 (socket says 172.21.15.119:33054) 2024-09-13T16:28:03.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.026+0000 7fa036ffd700 1 --2- 172.21.15.119:0/2034000022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d3760 0x7fa0380d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0377fe700 1 -- 172.21.15.119:0/2034000022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d3760 msgr2=0x7fa0380d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0377fe700 1 --2- 172.21.15.119:0/2034000022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d3760 0x7fa0380d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0367fc700 1 -- 172.21.15.119:0/2034000022 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294878298 0 0) 0x7fa0380b2240 con 0x7fa0380d1610 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0377fe700 1 -- 172.21.15.119:0/2034000022 >> v1:172.21.15.119:6789/0 conn(0x7fa0380d1610 legacy=0x7fa0380cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0377fe700 1 -- 172.21.15.119:0/2034000022 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0380b2520 con 0x7fa038160210 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0377fe700 1 --2- 172.21.15.119:0/2034000022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380d3220 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009880 tx=0x7fa02c009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de08f92ac61c7bac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0367fc700 1 -- 172.21.15.119:0/2034000022 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c011830 con 0x7fa038160210 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0367fc700 1 -- 172.21.15.119:0/2034000022 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c011990 con 0x7fa038160210 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa0367fc700 1 -- 172.21.15.119:0/2034000022 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c011d20 con 0x7fa038160210 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 -- 172.21.15.119:0/2034000022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 msgr2=0x7fa0380d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/2034000022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380d3220 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fa02c009880 tx=0x7fa02c009b90 comp rx=0 tx=0).stop 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 -- 172.21.15.119:0/2034000022 shutdown_connections 2024-09-13T16:28:03.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/2034000022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d3760 0x7fa0380d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/2034000022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380d3220 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 -- 172.21.15.119:0/2034000022 >> 172.21.15.119:0/2034000022 conn(0x7fa03808c080 msgr2=0x7fa0380d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 -- 172.21.15.119:0/2034000022 shutdown_connections 2024-09-13T16:28:03.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 -- 172.21.15.119:0/2034000022 wait complete. 2024-09-13T16:28:03.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 Processor -- start 2024-09-13T16:28:03.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.030+0000 7fa03c8bb700 1 -- start start 2024-09-13T16:28:03.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d1610 0x7fa03814c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0380d3760 0x7fa0380c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa038149730 con 0x7fa038160210 2024-09-13T16:28:03.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa038149fd0 con 0x7fa0380d3760 2024-09-13T16:28:03.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0380cc820 con 0x7fa0380d1610 2024-09-13T16:28:03.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33058/0 (socket says 172.21.15.119:33058) 2024-09-13T16:28:03.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 -- 172.21.15.119:0/3605147065 learned_addr learned my addr 172.21.15.119:0/3605147065 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa0377fe700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d1610 0x7fa03814c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa036ffd700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0380d3760 0x7fa0380c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 -- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d1610 msgr2=0x7fa03814c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d1610 0x7fa03814c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 -- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0380d3760 msgr2=0x7fa0380c7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0380d3760 0x7fa0380c7690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 -- 172.21.15.119:0/3605147065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02c0094b0 con 0x7fa038160210 2024-09-13T16:28:03.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa036ffd700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0380d3760 0x7fa0380c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:03.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa037fff700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380c7bd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cb90 tx=0x7fa03000cea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa01ffff700 1 -- 172.21.15.119:0/3605147065 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03000aa80 con 0x7fa038160210 2024-09-13T16:28:03.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa01ffff700 1 -- 172.21.15.119:0/3605147065 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03000abe0 con 0x7fa038160210 2024-09-13T16:28:03.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa038149980 con 0x7fa038160210 2024-09-13T16:28:03.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa01ffff700 1 -- 172.21.15.119:0/3605147065 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0300070d0 con 0x7fa038160210 2024-09-13T16:28:03.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.034+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03814c6d0 con 0x7fa038160210 2024-09-13T16:28:03.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.038+0000 7fa01ffff700 1 -- 172.21.15.119:0/3605147065 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa030007230 con 0x7fa038160210 2024-09-13T16:28:03.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.038+0000 7fa01ffff700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa024066730 0x7fa024068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.038+0000 7fa01ffff700 1 -- 172.21.15.119:0/3605147065 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa030083f60 con 0x7fa038160210 2024-09-13T16:28:03.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.038+0000 7fa0377fe700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa024066730 0x7fa024068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.038+0000 7fa0377fe700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa024066730 0x7fa024068bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0097d0 tx=0x7fa02c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.038+0000 7fa040dce740 1 Processor -- start 2024-09-13T16:28:03.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.038+0000 7fa040dce740 1 -- start start 2024-09-13T16:28:03.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a57d90 0x1a0e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a36520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a58550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x199f010 con 0x1a58170 2024-09-13T16:28:03.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x192cf40 con 0x1a0d310 2024-09-13T16:28:03.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x197fe30 con 0x1a57d90 2024-09-13T16:28:03.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a58550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a58550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33068/0 (socket says 172.21.15.119:33068) 2024-09-13T16:28:03.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 -- 172.21.15.119:0/3070100108 learned_addr learned my addr 172.21.15.119:0/3070100108 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa036ffd700 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a36520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa0377fe700 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a57d90 0x1a0e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 -- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a57d90 msgr2=0x1a0e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a57d90 0x1a0e810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 -- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 msgr2=0x1a36520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a36520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 -- 172.21.15.119:0/3070100108 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x17eba40 con 0x1a58170 2024-09-13T16:28:03.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa037fff700 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a58550 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa030082350 tx=0x7fa030054910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6102de35028b0613 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa01d7fa700 1 -- 172.21.15.119:0/3070100108 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030054ee0 con 0x1a58170 2024-09-13T16:28:03.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa01d7fa700 1 -- 172.21.15.119:0/3070100108 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa030058d10 con 0x1a58170 2024-09-13T16:28:03.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa01d7fa700 1 -- 172.21.15.119:0/3070100108 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03005d2d0 con 0x1a58170 2024-09-13T16:28:03.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- 172.21.15.119:0/3070100108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 msgr2=0x1a58550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a58550 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7fa030082350 tx=0x7fa030054910 comp rx=0 tx=0).stop 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- 172.21.15.119:0/3070100108 shutdown_connections 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a58550 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a36520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 --2- 172.21.15.119:0/3070100108 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a57d90 0x1a0e810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- 172.21.15.119:0/3070100108 >> 172.21.15.119:0/3070100108 conn(0x17a05b0 msgr2=0x1a11b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- 172.21.15.119:0/3070100108 shutdown_connections 2024-09-13T16:28:03.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.042+0000 7fa040dce740 1 -- 172.21.15.119:0/3070100108 wait complete. 2024-09-13T16:28:03.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 -- start start 2024-09-13T16:28:03.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 Processor -- start 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a0d310 0x1a39af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a57d90 0x1a378c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a56670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x197b150 con 0x1a58170 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1978eb0 con 0x1a57d90 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x19368e0 con 0x1a0d310 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 10 client.0 _ll_register_callbacks cb 0x1915670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a56670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a56670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33082/0 (socket says 172.21.15.119:33082) 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 -- 172.21.15.119:0/2119800554 learned_addr learned my addr 172.21.15.119:0/2119800554 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0377fe700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a0d310 0x1a39af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa036ffd700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a57d90 0x1a378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 -- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a0d310 msgr2=0x1a39af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a0d310 0x1a39af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 -- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a57d90 msgr2=0x1a378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a57d90 0x1a378c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 -- 172.21.15.119:0/2119800554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03814c6d0 con 0x1a58170 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa036ffd700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a57d90 0x1a378c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa037fff700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a56670 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cb60 tx=0x7fa03000af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1803b0ea54997d92 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0167fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0167fc700 1 -- 172.21.15.119:0/2119800554 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa030082070 con 0x1a58170 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0167fc700 1 -- 172.21.15.119:0/2119800554 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa03005d2d0 con 0x1a58170 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x17ad2c0 con 0x1a58170 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0167fc700 1 -- 172.21.15.119:0/2119800554 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa030054ee0 con 0x1a58170 2024-09-13T16:28:03.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 10 client.6324 fetch_fsmap learned FSMap version 24 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x193f290 con 0x1a58170 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0167fc700 1 -- 172.21.15.119:0/2119800554 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x17ad2c0 con 0x1a58170 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0167fc700 1 -- 172.21.15.119:0/2119800554 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa030068d20 con 0x1a58170 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 10 client.6324 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 10 client.6324 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 20 client.6324 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a56bb0 0x1a34e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 4 client.6324 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x17f7540 con 0x1a56bb0 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.046+0000 7fa0377fe700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a56bb0 0x1a34e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa0167fc700 10 client.6324 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:28:03.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa0377fe700 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a56bb0 0x1a34e40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d88f161b51afe513 server_cookie=f4fd88948e3459b in_seq=0 out_seq=0 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa0167fc700 1 -- 172.21.15.119:0/2119800554 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x17f7540 con 0x1a56bb0 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa0167fc700 10 client.6324 handle_command_reply: tid=0 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 client.6324 shutdown 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 msgr2=0x1a56670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a56670 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cb60 tx=0x7fa03000af90 comp rx=0 tx=0).stop 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 shutdown_connections 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a58170 0x1a56670 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a56bb0 0x1a34e40 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a57d90 0x1a378c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 --2- 172.21.15.119:0/2119800554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a0d310 0x1a39af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 >> 172.21.15.119:0/2119800554 conn(0x17a05b0 msgr2=0x1a364b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa0167fc700 0 client.6324 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:28:03.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 shutdown_connections 2024-09-13T16:28:03.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 1 -- 172.21.15.119:0/2119800554 wait complete. 2024-09-13T16:28:03.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.050+0000 7fa040dce740 20 client.6324 trim_cache size 0 max 16384 2024-09-13T16:28:03.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 Processor -- start 2024-09-13T16:28:03.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 -- start start 2024-09-13T16:28:03.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a55c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a57d90 0x1a609c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a60f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1824b80 con 0x1a57d90 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1976240 con 0x1a0d310 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x191e230 con 0x1a58170 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa036ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a57d90 0x1a609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa036ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a57d90 0x1a609c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33096/0 (socket says 172.21.15.119:33096) 2024-09-13T16:28:03.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.070+0000 7fa036ffd700 1 -- 172.21.15.119:0/130455346 learned_addr learned my addr 172.21.15.119:0/130455346 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa0377fe700 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a55c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa037fff700 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a60f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa036ffd700 1 -- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 msgr2=0x1a60f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa036ffd700 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a60f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa036ffd700 1 -- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 msgr2=0x1a55c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa036ffd700 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a55c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa036ffd700 1 -- 172.21.15.119:0/130455346 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1a39ea0 con 0x1a57d90 2024-09-13T16:28:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa0377fe700 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a55c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:03.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa036ffd700 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a57d90 0x1a609c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa0380d3b60 tx=0x7fa028009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e9f84e45b4567c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa01dffb700 1 -- 172.21.15.119:0/130455346 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028016c10 con 0x1a57d90 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa01dffb700 1 -- 172.21.15.119:0/130455346 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02801b070 con 0x1a57d90 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa01dffb700 1 -- 172.21.15.119:0/130455346 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028017440 con 0x1a57d90 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- 172.21.15.119:0/130455346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a57d90 msgr2=0x1a609c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a57d90 0x1a609c0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7fa0380d3b60 tx=0x7fa028009240 comp rx=0 tx=0).stop 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- 172.21.15.119:0/130455346 shutdown_connections 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a60f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a57d90 0x1a609c0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 --2- 172.21.15.119:0/130455346 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1a55c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- 172.21.15.119:0/130455346 >> 172.21.15.119:0/130455346 conn(0x17a05b0 msgr2=0x1a35220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- 172.21.15.119:0/130455346 shutdown_connections 2024-09-13T16:28:03.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- 172.21.15.119:0/130455346 wait complete. 2024-09-13T16:28:03.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- start start 2024-09-13T16:28:03.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 Processor -- start 2024-09-13T16:28:03.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1ab8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a55f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a564a0 0x1a6c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a6f700 con 0x1a564a0 2024-09-13T16:28:03.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a373e0 con 0x1a0d310 2024-09-13T16:28:03.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a6da20 con 0x1a58170 2024-09-13T16:28:03.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa040dce740 10 client.0 _ll_register_callbacks cb 0x1977610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:03.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a564a0 0x1a6c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa0377fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1ab8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa036ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a55f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa037fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a564a0 0x1a6c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33112/0 (socket says 172.21.15.119:33112) 2024-09-13T16:28:03.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.074+0000 7fa037fff700 1 -- 172.21.15.119:0/1793851848 learned_addr learned my addr 172.21.15.119:0/1793851848 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0377fe700 1 -- 172.21.15.119:0/1793851848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 msgr2=0x1a55f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0377fe700 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a55f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0377fe700 1 -- 172.21.15.119:0/1793851848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a564a0 msgr2=0x1a6c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0377fe700 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a564a0 0x1a6c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0377fe700 1 -- 172.21.15.119:0/1793851848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa028009890 con 0x1a0d310 2024-09-13T16:28:03.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0377fe700 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1ab8300 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa02c016000 tx=0x7fa02c00e000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4acedc13fa6b2d1d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-09-13T16:28:03.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 1 -- 172.21.15.119:0/1793851848 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c02c040 con 0x1a0d310 2024-09-13T16:28:03.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 1 -- 172.21.15.119:0/1793851848 <== mon.1 v2:172.21.15.145:3300/0 2 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa02c026df0 con 0x1a0d310 2024-09-13T16:28:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa030058d10 con 0x1a0d310 2024-09-13T16:28:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 1 -- 172.21.15.119:0/1793851848 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa02c020070 con 0x1a0d310 2024-09-13T16:28:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 10 client.6115 fetch_fsmap learned FSMap version 24 2024-09-13T16:28:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x19254c0 con 0x1a0d310 2024-09-13T16:28:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 1 -- 172.21.15.119:0/1793851848 <== mon.1 v2:172.21.15.145:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa030058d10 con 0x1a0d310 2024-09-13T16:28:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 1 -- 172.21.15.119:0/1793851848 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 24) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa02c03c020 con 0x1a0d310 2024-09-13T16:28:03.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 10 client.6115 fetch_fsmap finished waiting for FSMap version 24 2024-09-13T16:28:03.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 10 client.6115 resolve_mds: resolved 3:0 to role '3:0' aka daemon mds.a 2024-09-13T16:28:03.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 20 client.6115 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:03.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a6ceb0 0x1a6d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 4 client.6115 mds_command: new command op to 6069 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:28:03.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 --> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x181c2c0 con 0x1a6ceb0 2024-09-13T16:28:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa036ffd700 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a6ceb0 0x1a6d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 10 client.6115 ms_handle_connect on v2:172.21.15.119:6834/4101111376 2024-09-13T16:28:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa036ffd700 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a6ceb0 0x1a6d290 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ff94e4af122bdc8 server_cookie=f663202ef2232974 in_seq=0 out_seq=0 2024-09-13T16:28:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 1 -- 172.21.15.119:0/1793851848 <== mds.0 v2:172.21.15.119:6834/4101111376 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x181c2c0 con 0x1a6ceb0 2024-09-13T16:28:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 10 client.6115 handle_command_reply: tid=0 2024-09-13T16:28:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 client.6115 shutdown 2024-09-13T16:28:03.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 msgr2=0x1ab8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1ab8300 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fa02c016000 tx=0x7fa02c00e000 comp rx=0 tx=0).stop 2024-09-13T16:28:03.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 shutdown_connections 2024-09-13T16:28:03.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a564a0 0x1a6c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a58170 0x1a55f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.119:6834/4101111376,v1:172.21.15.119:6835/4101111376] conn(0x1a6ceb0 0x1a6d290 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 --2- 172.21.15.119:0/1793851848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a0d310 0x1ab8300 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa0167fc700 0 client.6115 ms_handle_reset on v2:172.21.15.119:6834/4101111376 2024-09-13T16:28:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 >> 172.21.15.119:0/1793851848 conn(0x17a05b0 msgr2=0x1a382f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 shutdown_connections 2024-09-13T16:28:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 1 -- 172.21.15.119:0/1793851848 wait complete. 2024-09-13T16:28:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.078+0000 7fa040dce740 20 client.6115 trim_cache size 0 max 16384 2024-09-13T16:28:03.093 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:03.093 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:28:03.093 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:28:03.094 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa024066730 msgr2=0x7fa024068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa024066730 0x7fa024068bf0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa02c0097d0 tx=0x7fa02c008030 comp rx=0 tx=0).stop 2024-09-13T16:28:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 msgr2=0x7fa0380c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380c7bd0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fa03000cb90 tx=0x7fa03000cea0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 shutdown_connections 2024-09-13T16:28:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa024066730 0x7fa024068bf0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa038160210 0x7fa0380c7bd0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0380d3760 0x7fa0380c7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 --2- 172.21.15.119:0/3605147065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0380d1610 0x7fa03814c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 >> 172.21.15.119:0/3605147065 conn(0x7fa03808c080 msgr2=0x7fa0380b48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 shutdown_connections 2024-09-13T16:28:03.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.082+0000 7fa03c8bb700 1 -- 172.21.15.119:0/3605147065 wait complete. 2024-09-13T16:28:03.099 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:28:03.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04758c3700 1 Processor -- start 2024-09-13T16:28:03.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04758c3700 1 -- start start 2024-09-13T16:28:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04758c3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700b3a20 0x7f04700cd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04758c3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04700cdf90 con 0x7f04700d0a50 2024-09-13T16:28:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04758c3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04700ce270 con 0x7f04700d2a30 2024-09-13T16:28:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04758c3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04700ce550 con 0x7f04700b3a20 2024-09-13T16:28:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04750c2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f04700d0a50 0x7f04700d0ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34336/0 (socket says 172.21.15.119:34336) 2024-09-13T16:28:03.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f04750c2700 1 -- 172.21.15.119:0/3896369400 learned_addr learned my addr 172.21.15.119:0/3896369400 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046ffff700 1 --2- 172.21.15.119:0/3896369400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700b3a20 0x7f04700cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046f7fe700 1 -- 172.21.15.119:0/3896369400 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2009555362 0 0) 0x7f04700cdf90 con 0x7f04700d0a50 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046f7fe700 1 -- 172.21.15.119:0/3896369400 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04580035d0 con 0x7f04700d0a50 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046ffff700 1 -- 172.21.15.119:0/3896369400 >> v1:172.21.15.145:6789/0 conn(0x7f04700d2a30 legacy=0x7f04700d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046ffff700 1 -- 172.21.15.119:0/3896369400 >> v1:172.21.15.119:6789/0 conn(0x7f04700d0a50 legacy=0x7f04700d0ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046ffff700 1 -- 172.21.15.119:0/3896369400 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04700ce830 con 0x7f04700b3a20 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046ffff700 1 --2- 172.21.15.119:0/3896369400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700b3a20 0x7f04700cd640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0460009dc0 tx=0x7f046000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6176149398ac200 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046f7fe700 1 -- 172.21.15.119:0/3896369400 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046000e070 con 0x7f04700b3a20 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046f7fe700 1 -- 172.21.15.119:0/3896369400 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046000bb20 con 0x7f04700b3a20 2024-09-13T16:28:03.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.318+0000 7f046f7fe700 1 -- 172.21.15.119:0/3896369400 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046000be90 con 0x7f04700b3a20 2024-09-13T16:28:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.322+0000 7f04758c3700 1 -- 172.21.15.119:0/3896369400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700b3a20 msgr2=0x7f04700cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.322+0000 7f04758c3700 1 --2- 172.21.15.119:0/3896369400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700b3a20 0x7f04700cd640 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0460009dc0 tx=0x7f046000b330 comp rx=0 tx=0).stop 2024-09-13T16:28:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.322+0000 7f04758c3700 1 -- 172.21.15.119:0/3896369400 shutdown_connections 2024-09-13T16:28:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.322+0000 7f04758c3700 1 --2- 172.21.15.119:0/3896369400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700b3a20 0x7f04700cd640 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.322+0000 7f04758c3700 1 -- 172.21.15.119:0/3896369400 >> 172.21.15.119:0/3896369400 conn(0x7f047008c0f0 msgr2=0x7f047008c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.322+0000 7f04758c3700 1 -- 172.21.15.119:0/3896369400 shutdown_connections 2024-09-13T16:28:03.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.322+0000 7f04758c3700 1 -- 172.21.15.119:0/3896369400 wait complete. 2024-09-13T16:28:03.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 Processor -- start 2024-09-13T16:28:03.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 -- start start 2024-09-13T16:28:03.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04700b3a20 0x7f047015bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700d0a50 0x7f04700c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04700d2a30 0x7f04700cea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04700ca3c0 con 0x7f04700b3a20 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04700ca6a0 con 0x7f04700d2a30 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04700ca980 con 0x7f04700d0a50 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04700b3a20 0x7f047015bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04700b3a20 0x7f047015bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33114/0 (socket says 172.21.15.119:33114) 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 -- 172.21.15.119:0/1048685379 learned_addr learned my addr 172.21.15.119:0/1048685379 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04750c2700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04700d2a30 0x7f04700cea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f046ffff700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700d0a50 0x7f04700c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 -- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700d0a50 msgr2=0x7f04700c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700d0a50 0x7f04700c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 -- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04700d2a30 msgr2=0x7f04700cea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04700d2a30 0x7f04700cea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 -- 172.21.15.119:0/1048685379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460009a70 con 0x7f04700b3a20 2024-09-13T16:28:03.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04748c1700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04700b3a20 0x7f047015bf40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f046400c500 tx=0x7f046400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f046d7fa700 1 -- 172.21.15.119:0/1048685379 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04640052d0 con 0x7f04700b3a20 2024-09-13T16:28:03.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f046d7fa700 1 -- 172.21.15.119:0/1048685379 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0464005430 con 0x7f04700b3a20 2024-09-13T16:28:03.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04700cac60 con 0x7f04700b3a20 2024-09-13T16:28:03.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f046d7fa700 1 -- 172.21.15.119:0/1048685379 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04640055a0 con 0x7f04700b3a20 2024-09-13T16:28:03.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04700ca0b0 con 0x7f04700b3a20 2024-09-13T16:28:03.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.326+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04700cdf90 con 0x7f04700b3a20 2024-09-13T16:28:03.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.330+0000 7f046d7fa700 1 -- 172.21.15.119:0/1048685379 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04640036a0 con 0x7f04700b3a20 2024-09-13T16:28:03.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.330+0000 7f046d7fa700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f045c066730 0x7f045c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.330+0000 7f046d7fa700 1 -- 172.21.15.119:0/1048685379 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f04640834f0 con 0x7f04700b3a20 2024-09-13T16:28:03.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.330+0000 7f046ffff700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f045c066730 0x7f045c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.330+0000 7f046ffff700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f045c066730 0x7f045c068bf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f30 tx=0x7f0460011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.338+0000 7f046d7fa700 1 -- 172.21.15.119:0/1048685379 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0464050690 con 0x7f04700b3a20 2024-09-13T16:28:03.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04700cd850 con 0x7f04700b3a20 2024-09-13T16:28:03.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f046d7fa700 1 -- 172.21.15.119:0/1048685379 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f0464053cb0 con 0x7f04700b3a20 2024-09-13T16:28:03.552 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:03.553 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:28:03.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f045c066730 msgr2=0x7f045c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f045c066730 0x7f045c068bf0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f0460009f30 tx=0x7f0460011000 comp rx=0 tx=0).stop 2024-09-13T16:28:03.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04700b3a20 msgr2=0x7f047015bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04700b3a20 0x7f047015bf40 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f046400c500 tx=0x7f046400c810 comp rx=0 tx=0).stop 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 shutdown_connections 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f045c066730 0x7f045c068bf0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04700d2a30 0x7f04700cea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04700d0a50 0x7f04700c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 --2- 172.21.15.119:0/1048685379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04700b3a20 0x7f047015bf40 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 >> 172.21.15.119:0/1048685379 conn(0x7f047008c0f0 msgr2=0x7f047008c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.550+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 shutdown_connections 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.554+0000 7f04758c3700 1 -- 172.21.15.119:0/1048685379 wait complete. 2024-09-13T16:28:03.556 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:28:03.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json flush journal 2024-09-13T16:28:03.698 INFO:teuthology.orchestra.run.smithi119.stdout:{"message":"","return_code":0} 2024-09-13T16:28:03.698 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:28:03.698 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-13T16:28:03.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.734+0000 7fe98d0bf140 1 Processor -- start 2024-09-13T16:28:03.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.734+0000 7fe98d0bf140 1 -- start start 2024-09-13T16:28:03.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9740030b0 0x55d4c2cc8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d4c2cca130 0x55d4c2ce47a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2d5a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c2c35a00 con 0x55d4c2d59870 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c2c312f0 con 0x55d4c2cca130 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4c2bed8e0 con 0x7fe9740030b0 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2d5a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98abf4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9740030b0 0x55d4c2cc8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2d5a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33120/0 (socket says 172.21.15.119:33120) 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 -- 172.21.15.119:0/2073332435 learned_addr learned my addr 172.21.15.119:0/2073332435 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98a3f3700 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d4c2cca130 0x55d4c2ce47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 -- 172.21.15.119:0/2073332435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9740030b0 msgr2=0x55d4c2cc8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9740030b0 0x55d4c2cc8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 -- 172.21.15.119:0/2073332435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d4c2cca130 msgr2=0x55d4c2ce47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d4c2cca130 0x55d4c2ce47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 -- 172.21.15.119:0/2073332435 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d4c2bcef40 con 0x55d4c2d59870 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98b3f5700 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2d5a080 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x55d4c2ca24b0 tx=0x7fe98400e5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37b449b6dbd6b056 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe9893f1700 1 -- 172.21.15.119:0/2073332435 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98400ea30 con 0x55d4c2d59870 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe9893f1700 1 -- 172.21.15.119:0/2073332435 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe984008030 con 0x55d4c2d59870 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe9893f1700 1 -- 172.21.15.119:0/2073332435 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98400ed90 con 0x55d4c2d59870 2024-09-13T16:28:03.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2073332435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 msgr2=0x55d4c2d5a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2d5a080 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x55d4c2ca24b0 tx=0x7fe98400e5b0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2073332435 shutdown_connections 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2d5a080 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55d4c2cca130 0x55d4c2ce47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2073332435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9740030b0 0x55d4c2cc8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2073332435 >> 172.21.15.119:0/2073332435 conn(0x55d4c2c9f910 msgr2=0x55d4c2cca910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2073332435 shutdown_connections 2024-09-13T16:28:03.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.738+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2073332435 wait complete. 2024-09-13T16:28:03.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 Processor -- start 2024-09-13T16:28:03.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 -- start start 2024-09-13T16:28:03.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d4c2cca130 0x55d4c2d58f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2cc0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9740030b0 0x55d4c2d73ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c2bed580 con 0x55d4c2d59870 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d4c2bc38d0 con 0x7fe9740030b0 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d4c2e0e0d0 con 0x55d4c2cca130 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2cc0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2cc0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33130/0 (socket says 172.21.15.119:33130) 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 -- 172.21.15.119:0/2740941898 learned_addr learned my addr 172.21.15.119:0/2740941898 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98b3f5700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9740030b0 0x55d4c2d73ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98abf4700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d4c2cca130 0x55d4c2d58f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 -- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d4c2cca130 msgr2=0x55d4c2d58f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d4c2cca130 0x55d4c2d58f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 -- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9740030b0 msgr2=0x55d4c2d73ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9740030b0 0x55d4c2d73ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 -- 172.21.15.119:0/2740941898 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98400a040 con 0x55d4c2d59870 2024-09-13T16:28:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98abf4700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55d4c2cca130 0x55d4c2d58f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98a3f3700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2cc0590 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe980015040 tx=0x7fe980003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe973fff700 1 -- 172.21.15.119:0/2740941898 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe98000f070 con 0x55d4c2d59870 2024-09-13T16:28:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe973fff700 1 -- 172.21.15.119:0/2740941898 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe980013630 con 0x55d4c2d59870 2024-09-13T16:28:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55d4c2bce9a0 con 0x55d4c2d59870 2024-09-13T16:28:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe973fff700 1 -- 172.21.15.119:0/2740941898 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9800139e0 con 0x55d4c2d59870 2024-09-13T16:28:03.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.742+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d4c2ccdd60 con 0x55d4c2d59870 2024-09-13T16:28:03.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.746+0000 7fe973fff700 1 -- 172.21.15.119:0/2740941898 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe980005040 con 0x55d4c2d59870 2024-09-13T16:28:03.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.746+0000 7fe973fff700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe978065090 0x7fe978067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.746+0000 7fe973fff700 1 -- 172.21.15.119:0/2740941898 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe980084320 con 0x55d4c2d59870 2024-09-13T16:28:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.746+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55d4c2cc0ad0 0x55d4c2e0b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.746+0000 7fe98abf4700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe978065090 0x7fe978067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.746+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 8.7 8:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e54) v8 -- 0x55d4c2e0bd80 con 0x55d4c2cc0ad0 2024-09-13T16:28:03.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.746+0000 7fe98b3f5700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55d4c2cc0ad0 0x55d4c2e0b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98b3f5700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55d4c2cc0ad0 0x55d4c2e0b840 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98abf4700 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe978065090 0x7fe978067550 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe97c006320 tx=0x7fe97c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98b3f5700 1 -- 172.21.15.119:0/2740941898 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x7fe984017070 con 0x55d4c2cc0ad0 2024-09-13T16:28:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55d4c2cc0ad0 msgr2=0x55d4c2e0b840 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55d4c2cc0ad0 0x55d4c2e0b840 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe978065090 msgr2=0x7fe978067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe978065090 0x7fe978067550 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fe97c006320 tx=0x7fe97c008040 comp rx=0 tx=0).stop 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 msgr2=0x55d4c2cc0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 --2- 172.21.15.119:0/2740941898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55d4c2d59870 0x55d4c2cc0590 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fe980015040 tx=0x7fe980003020 comp rx=0 tx=0).stop 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98b3f5700 1 -- 172.21.15.119:0/2740941898 reap_dead start 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 shutdown_connections 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 >> 172.21.15.119:0/2740941898 conn(0x55d4c2c9f910 msgr2=0x55d4c2d78220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 shutdown_connections 2024-09-13T16:28:03.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:03.750+0000 7fe98d0bf140 1 -- 172.21.15.119:0/2740941898 wait complete. 2024-09-13T16:28:03.755 INFO:teuthology.orchestra.run.smithi119.stdout:Yalphaparent_a 2024-09-13T16:28:03.756 DEBUG:teuthology.orchestra.run.smithi119:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627777, 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout: "ancestors": [ 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout: "dirino": 1099511627776, 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout: "dname": "alpha", 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout: "version": 4 2024-09-13T16:28:03.813 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: "dirino": 1, 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: "dname": "parent_a", 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: "version": 7 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: "pool": 8, 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: "old_pools": [] 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:03.814 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:03.816 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:28:03.816 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:28:03.834 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:28:03.856 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:28:03.856 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:28:03.856 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:03.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:04.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.022+0000 7f7d15639700 1 Processor -- start 2024-09-13T16:28:04.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.022+0000 7f7d15639700 1 -- start start 2024-09-13T16:28:04.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7d100cc060 0x7f7d100cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100cde70 0x7f7d100b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d1006cf60 con 0x7f7d100adb30 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d100cc4d0 con 0x7f7d100cc060 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1006c350 con 0x7f7d100cde70 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7d100cc060 0x7f7d100cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7d100cc060 0x7f7d100cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33136/0 (socket says 172.21.15.119:33136) 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 -- 172.21.15.119:0/560963542 learned_addr learned my addr 172.21.15.119:0/560963542 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d14e38700 1 --2- 172.21.15.119:0/560963542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100cde70 0x7f7d100b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 -- 172.21.15.119:0/560963542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100cde70 msgr2=0x7f7d100b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 --2- 172.21.15.119:0/560963542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100cde70 0x7f7d100b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 -- 172.21.15.119:0/560963542 >> v1:172.21.15.119:6789/0 conn(0x7f7d100adb30 legacy=0x7f7d1006bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 -- 172.21.15.119:0/560963542 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1006c630 con 0x7f7d100cc060 2024-09-13T16:28:04.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0f7fe700 1 --2- 172.21.15.119:0/560963542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cc060 0x7f7d100cd930 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009e50 tx=0x7f7cf800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3af299532e98980 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0effd700 1 -- 172.21.15.119:0/560963542 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800e070 con 0x7f7d100cc060 2024-09-13T16:28:04.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0effd700 1 -- 172.21.15.119:0/560963542 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf800b9c0 con 0x7f7d100cc060 2024-09-13T16:28:04.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d0effd700 1 -- 172.21.15.119:0/560963542 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800bd70 con 0x7f7d100cc060 2024-09-13T16:28:04.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- 172.21.15.119:0/560963542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cc060 msgr2=0x7f7d100cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 --2- 172.21.15.119:0/560963542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cc060 0x7f7d100cd930 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009e50 tx=0x7f7cf800b330 comp rx=0 tx=0).stop 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- 172.21.15.119:0/560963542 shutdown_connections 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 --2- 172.21.15.119:0/560963542 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100cde70 0x7f7d100b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 --2- 172.21.15.119:0/560963542 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cc060 0x7f7d100cd930 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- 172.21.15.119:0/560963542 >> 172.21.15.119:0/560963542 conn(0x7f7d1006a350 msgr2=0x7f7d1006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- 172.21.15.119:0/560963542 shutdown_connections 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- 172.21.15.119:0/560963542 wait complete. 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 Processor -- start 2024-09-13T16:28:04.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.026+0000 7f7d15639700 1 -- start start 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100adb30 0x7f7d10151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d100cc060 0x7f7d100ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cde70 0x7f7d100ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d10071000 con 0x7f7d100cde70 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d100712e0 con 0x7f7d100cc060 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d100715c0 con 0x7f7d100adb30 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cde70 0x7f7d100ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cde70 0x7f7d100ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33150/0 (socket says 172.21.15.119:33150) 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d0ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100adb30 0x7f7d10151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 -- 172.21.15.119:0/294144563 learned_addr learned my addr 172.21.15.119:0/294144563 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d0f7fe700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d100cc060 0x7f7d100ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 -- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100adb30 msgr2=0x7f7d10151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100adb30 0x7f7d10151be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 -- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d100cc060 msgr2=0x7f7d100ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d100cc060 0x7f7d100ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 -- 172.21.15.119:0/294144563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf8009b30 con 0x7f7d100cde70 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d14e38700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cde70 0x7f7d100ace00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800d800 tx=0x7f7d0800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d0cff9700 1 -- 172.21.15.119:0/294144563 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d080125f0 con 0x7f7d100cde70 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d0cff9700 1 -- 172.21.15.119:0/294144563 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d08012790 con 0x7f7d100cde70 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d100718a0 con 0x7f7d100cde70 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d0cff9700 1 -- 172.21.15.119:0/294144563 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d08012b40 con 0x7f7d100cde70 2024-09-13T16:28:04.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.030+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d10070490 con 0x7f7d100cde70 2024-09-13T16:28:04.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.034+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d10139910 con 0x7f7d100cde70 2024-09-13T16:28:04.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.034+0000 7f7d0cff9700 1 -- 172.21.15.119:0/294144563 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7d08012ca0 con 0x7f7d100cde70 2024-09-13T16:28:04.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.034+0000 7f7d0cff9700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cf0066730 0x7f7cf0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.034+0000 7f7d0cff9700 1 -- 172.21.15.119:0/294144563 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7d080842e0 con 0x7f7d100cde70 2024-09-13T16:28:04.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.034+0000 7f7d0ffff700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cf0066730 0x7f7cf0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.034+0000 7f7d0ffff700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cf0066730 0x7f7cf0068bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7d00000f60 tx=0x7f7d00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.038+0000 7f7d0cff9700 1 -- 172.21.15.119:0/294144563 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7d10139910 con 0x7f7d100cde70 2024-09-13T16:28:04.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.210+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d10139910 con 0x7f7d100cde70 2024-09-13T16:28:04.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.214+0000 7f7d0cff9700 1 -- 172.21.15.119:0/294144563 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16834 (secure 0 0 0) 0x7f7d10139910 con 0x7f7d100cde70 2024-09-13T16:28:04.218 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:04.218 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":54,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:24.352264+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:27:21.660330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-09-13T16:27:22.341239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cf0066730 msgr2=0x7f7cf0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cf0066730 0x7f7cf0068bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7d00000f60 tx=0x7f7d00009040 comp rx=0 tx=0).stop 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cde70 msgr2=0x7f7d100ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cde70 0x7f7d100ace00 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f7d0800d800 tx=0x7f7d0800db10 comp rx=0 tx=0).stop 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 shutdown_connections 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7cf0066730 0x7f7cf0068bf0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d100cde70 0x7f7d100ace00 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d100cc060 0x7f7d100ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 --2- 172.21.15.119:0/294144563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d100adb30 0x7f7d10151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 >> 172.21.15.119:0/294144563 conn(0x7f7d1006a350 msgr2=0x7f7d100afc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:04.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 shutdown_connections 2024-09-13T16:28:04.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.218+0000 7f7d15639700 1 -- 172.21.15.119:0/294144563 wait complete. 2024-09-13T16:28:04.233 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:27:21.660330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:27:22.341239+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:28:04.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:28:04.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 Processor -- start 2024-09-13T16:28:04.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 -- start start 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f38d015a2c0 0x7f38d015e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015a6a0 0x7f38d0071930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f38d015ab00 0x7f38d0071e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d006c0b0 con 0x7f38d015a2c0 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d015d560 con 0x7f38d015ab00 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38d5985700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d006bdd0 con 0x7f38d015a6a0 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f38d015a2c0 0x7f38d015e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f38d015a2c0 0x7f38d015e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33156/0 (socket says 172.21.15.119:33156) 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 -- 172.21.15.119:0/4080124476 learned_addr learned my addr 172.21.15.119:0/4080124476 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38cf7fe700 1 --2- 172.21.15.119:0/4080124476 >> v2:172.21.15.145:3300/0 conn(0x7f38d015ab00 0x7f38d0071e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 -- 172.21.15.119:0/4080124476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015a6a0 msgr2=0x7f38d0071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ce7fc700 1 --2- 172.21.15.119:0/4080124476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015a6a0 0x7f38d0071930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 --2- 172.21.15.119:0/4080124476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015a6a0 0x7f38d0071930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 -- 172.21.15.119:0/4080124476 >> v2:172.21.15.145:3300/0 conn(0x7f38d015ab00 msgr2=0x7f38d0071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 --2- 172.21.15.119:0/4080124476 >> v2:172.21.15.145:3300/0 conn(0x7f38d015ab00 0x7f38d0071e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 -- 172.21.15.119:0/4080124476 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d015d800 con 0x7f38d015a2c0 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38cf7fe700 1 --2- 172.21.15.119:0/4080124476 >> v2:172.21.15.145:3300/0 conn(0x7f38d015ab00 0x7f38d0071e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ce7fc700 1 --2- 172.21.15.119:0/4080124476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015a6a0 0x7f38d0071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.354+0000 7f38ceffd700 1 --2- 172.21.15.119:0/4080124476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a2c0 0x7f38d015e000 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f38d0070a10 tx=0x7f38c000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=996875534c3e0d8a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38cdffb700 1 -- 172.21.15.119:0/4080124476 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0010070 con 0x7f38d015a2c0 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38cdffb700 1 -- 172.21.15.119:0/4080124476 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c000dd90 con 0x7f38d015a2c0 2024-09-13T16:28:04.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38cdffb700 1 -- 172.21.15.119:0/4080124476 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c000b310 con 0x7f38d015a2c0 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 -- 172.21.15.119:0/4080124476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a2c0 msgr2=0x7f38d015e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 --2- 172.21.15.119:0/4080124476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a2c0 0x7f38d015e000 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f38d0070a10 tx=0x7f38c000d430 comp rx=0 tx=0).stop 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 -- 172.21.15.119:0/4080124476 shutdown_connections 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 --2- 172.21.15.119:0/4080124476 >> v2:172.21.15.145:3300/0 conn(0x7f38d015ab00 0x7f38d0071e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 --2- 172.21.15.119:0/4080124476 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015a6a0 0x7f38d0071930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 --2- 172.21.15.119:0/4080124476 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a2c0 0x7f38d015e000 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 -- 172.21.15.119:0/4080124476 >> 172.21.15.119:0/4080124476 conn(0x7f38d00b3ec0 msgr2=0x7f38d00b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 -- 172.21.15.119:0/4080124476 shutdown_connections 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 -- 172.21.15.119:0/4080124476 wait complete. 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 Processor -- start 2024-09-13T16:28:04.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.358+0000 7f38d5985700 1 -- start start 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d015a2c0 0x7f38d01f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a6a0 0x7f38d01551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015ab00 0x7f38d0155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d01580b0 con 0x7f38d015a6a0 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0157430 con 0x7f38d015a2c0 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d0157710 con 0x7f38d015ab00 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a6a0 0x7f38d01551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a6a0 0x7f38d01551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33160/0 (socket says 172.21.15.119:33160) 2024-09-13T16:28:04.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 -- 172.21.15.119:0/416903806 learned_addr learned my addr 172.21.15.119:0/416903806 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ceffd700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d015a2c0 0x7f38d01f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38cf7fe700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015ab00 0x7f38d0155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 -- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015ab00 msgr2=0x7f38d0155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015ab00 0x7f38d0155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 -- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d015a2c0 msgr2=0x7f38d01f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d015a2c0 0x7f38d01f21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 -- 172.21.15.119:0/416903806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c000bd40 con 0x7f38d015a6a0 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ceffd700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d015a2c0 0x7f38d01f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38ce7fc700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a6a0 0x7f38d01551c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c770 tx=0x7f38b800cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38c7fff700 1 -- 172.21.15.119:0/416903806 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8011800 con 0x7f38d015a6a0 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38c7fff700 1 -- 172.21.15.119:0/416903806 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b8011960 con 0x7f38d015a6a0 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d01579f0 con 0x7f38d015a6a0 2024-09-13T16:28:04.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38c7fff700 1 -- 172.21.15.119:0/416903806 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8011d10 con 0x7f38d015a6a0 2024-09-13T16:28:04.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d01f2700 con 0x7f38d015a6a0 2024-09-13T16:28:04.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.362+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d01f2920 con 0x7f38d015a6a0 2024-09-13T16:28:04.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.366+0000 7f38c7fff700 1 -- 172.21.15.119:0/416903806 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f38b8018360 con 0x7f38d015a6a0 2024-09-13T16:28:04.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.366+0000 7f38c7fff700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b0066730 0x7f38b0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.366+0000 7f38c7fff700 1 -- 172.21.15.119:0/416903806 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f38b80842c0 con 0x7f38d015a6a0 2024-09-13T16:28:04.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.366+0000 7f38ceffd700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b0066730 0x7f38b0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.366+0000 7f38ceffd700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b0066730 0x7f38b0068bf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f38c0000f80 tx=0x7f38c000b000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.374+0000 7f38c7fff700 1 -- 172.21.15.119:0/416903806 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f38b8051500 con 0x7f38d015a6a0 2024-09-13T16:28:04.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.542+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38d00b4e10 con 0x7f38d015a6a0 2024-09-13T16:28:04.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.542+0000 7f38c7fff700 1 -- 172.21.15.119:0/416903806 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38b8054b20 con 0x7f38d015a6a0 2024-09-13T16:28:04.547 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:28:04.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b0066730 msgr2=0x7f38b0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b0066730 0x7f38b0068bf0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f38c0000f80 tx=0x7f38c000b000 comp rx=0 tx=0).stop 2024-09-13T16:28:04.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a6a0 msgr2=0x7f38d01551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a6a0 0x7f38d01551c0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c770 tx=0x7f38b800cb30 comp rx=0 tx=0).stop 2024-09-13T16:28:04.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 shutdown_connections 2024-09-13T16:28:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b0066730 0x7f38b0068bf0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d015ab00 0x7f38d0155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d015a6a0 0x7f38d01551c0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 --2- 172.21.15.119:0/416903806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d015a2c0 0x7f38d01f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 >> 172.21.15.119:0/416903806 conn(0x7f38d00b3ec0 msgr2=0x7f38d00662d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 shutdown_connections 2024-09-13T16:28:04.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.546+0000 7f38d5985700 1 -- 172.21.15.119:0/416903806 wait complete. 2024-09-13T16:28:04.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:28:04.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.670+0000 7f4568d2f700 1 Processor -- start 2024-09-13T16:28:04.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.670+0000 7f4568d2f700 1 -- start start 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f456415cc00 0x7f45640d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45640d0c20 0x7f45640d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45640cdb00 con 0x7f45640ce8c0 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45640cdde0 con 0x7f45640d0c20 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45640ce0c0 con 0x7f456415cc00 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4562ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45640d0c20 0x7f45640d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4563fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f45640ce8c0 0x7f4564160880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34380/0 (socket says 172.21.15.119:34380) 2024-09-13T16:28:04.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4563fff700 1 -- 172.21.15.119:0/2305108318 learned_addr learned my addr 172.21.15.119:0/2305108318 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f45637fe700 1 --2- 172.21.15.119:0/2305108318 >> v2:172.21.15.145:3300/0 conn(0x7f456415cc00 0x7f45640d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f45627fc700 1 -- 172.21.15.119:0/2305108318 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716014945 0 0) 0x7f45640cdb00 con 0x7f45640ce8c0 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4562ffd700 1 -- 172.21.15.119:0/2305108318 >> v2:172.21.15.145:3300/0 conn(0x7f456415cc00 msgr2=0x7f45640d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4562ffd700 1 --2- 172.21.15.119:0/2305108318 >> v2:172.21.15.145:3300/0 conn(0x7f456415cc00 0x7f45640d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4562ffd700 1 -- 172.21.15.119:0/2305108318 >> v1:172.21.15.119:6789/0 conn(0x7f45640ce8c0 legacy=0x7f4564160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4562ffd700 1 -- 172.21.15.119:0/2305108318 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45640ce3a0 con 0x7f45640d0c20 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f45637fe700 1 --2- 172.21.15.119:0/2305108318 >> v2:172.21.15.145:3300/0 conn(0x7f456415cc00 0x7f45640d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4562ffd700 1 --2- 172.21.15.119:0/2305108318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45640d0c20 0x7f45640d10d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f45640b5610 tx=0x7f454c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb464ca05bb3fc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f45627fc700 1 -- 172.21.15.119:0/2305108318 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c011830 con 0x7f45640d0c20 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f45627fc700 1 -- 172.21.15.119:0/2305108318 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454c011990 con 0x7f45640d0c20 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f45627fc700 1 -- 172.21.15.119:0/2305108318 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454c011d40 con 0x7f45640d0c20 2024-09-13T16:28:04.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- 172.21.15.119:0/2305108318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45640d0c20 msgr2=0x7f45640d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- 172.21.15.119:0/2305108318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45640d0c20 0x7f45640d10d0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f45640b5610 tx=0x7f454c009b30 comp rx=0 tx=0).stop 2024-09-13T16:28:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- 172.21.15.119:0/2305108318 shutdown_connections 2024-09-13T16:28:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- 172.21.15.119:0/2305108318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45640d0c20 0x7f45640d10d0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- 172.21.15.119:0/2305108318 >> v2:172.21.15.145:3300/0 conn(0x7f456415cc00 0x7f45640d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- 172.21.15.119:0/2305108318 >> 172.21.15.119:0/2305108318 conn(0x7f456408c080 msgr2=0x7f45640d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- 172.21.15.119:0/2305108318 shutdown_connections 2024-09-13T16:28:04.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- 172.21.15.119:0/2305108318 wait complete. 2024-09-13T16:28:04.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 Processor -- start 2024-09-13T16:28:04.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 -- start start 2024-09-13T16:28:04.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45640ce8c0 0x7f45641f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45640d0c20 0x7f45641f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.674+0000 7f4568d2f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f456415cc00 0x7f45640c77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4568d2f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45641f2bd0 con 0x7f456415cc00 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4568d2f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45640c8390 con 0x7f45640ce8c0 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4568d2f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45640c8670 con 0x7f45640d0c20 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f456415cc00 0x7f45640c77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f456415cc00 0x7f45640c77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33182/0 (socket says 172.21.15.119:33182) 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 -- 172.21.15.119:0/3829412554 learned_addr learned my addr 172.21.15.119:0/3829412554 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f45637fe700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45640ce8c0 0x7f45641f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4562ffd700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45640d0c20 0x7f45641f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 -- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45640d0c20 msgr2=0x7f45641f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45640d0c20 0x7f45641f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 -- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45640ce8c0 msgr2=0x7f45641f5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45640ce8c0 0x7f45641f5e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 -- 172.21.15.119:0/3829412554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c0094b0 con 0x7f456415cc00 2024-09-13T16:28:04.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4562ffd700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45640d0c20 0x7f45641f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:04.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4563fff700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f456415cc00 0x7f45640c77b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f455c002230 tx=0x7f455c00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f455bfff700 1 -- 172.21.15.119:0/3829412554 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c00ac20 con 0x7f456415cc00 2024-09-13T16:28:04.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f455bfff700 1 -- 172.21.15.119:0/3829412554 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c00ada0 con 0x7f456415cc00 2024-09-13T16:28:04.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45640c8950 con 0x7f456415cc00 2024-09-13T16:28:04.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f455bfff700 1 -- 172.21.15.119:0/3829412554 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c007310 con 0x7f456415cc00 2024-09-13T16:28:04.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45640c8f10 con 0x7f456415cc00 2024-09-13T16:28:04.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456408c8a0 con 0x7f456415cc00 2024-09-13T16:28:04.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f455bfff700 1 -- 172.21.15.119:0/3829412554 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f455c007470 con 0x7f456415cc00 2024-09-13T16:28:04.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.678+0000 7f455bfff700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4544066730 0x7f4544068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.682+0000 7f455bfff700 1 -- 172.21.15.119:0/3829412554 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f455c0841b0 con 0x7f456415cc00 2024-09-13T16:28:04.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.682+0000 7f45637fe700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4544066730 0x7f4544068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.682+0000 7f45637fe700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4544066730 0x7f4544068bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f456415c460 tx=0x7f4554008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:04.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.686+0000 7f455bfff700 1 -- 172.21.15.119:0/3829412554 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f455c0513e0 con 0x7f456415cc00 2024-09-13T16:28:04.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.850+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f45640cdb00 con 0x7f456415cc00 2024-09-13T16:28:04.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.854+0000 7f455bfff700 1 -- 172.21.15.119:0/3829412554 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v54) v1 ==== 104+0+11 (secure 0 0 0) 0x7f455c054a00 con 0x7f456415cc00 2024-09-13T16:28:04.856 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:28:04.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4544066730 msgr2=0x7f4544068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4544066730 0x7f4544068bf0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f456415c460 tx=0x7f4554008040 comp rx=0 tx=0).stop 2024-09-13T16:28:04.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f456415cc00 msgr2=0x7f45640c77b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:04.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f456415cc00 0x7f45640c77b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f455c002230 tx=0x7f455c00a060 comp rx=0 tx=0).stop 2024-09-13T16:28:04.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 shutdown_connections 2024-09-13T16:28:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4544066730 0x7f4544068bf0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f456415cc00 0x7f45640c77b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45640d0c20 0x7f45641f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 --2- 172.21.15.119:0/3829412554 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45640ce8c0 0x7f45641f5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 >> 172.21.15.119:0/3829412554 conn(0x7f456408c080 msgr2=0x7f45640b6aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 shutdown_connections 2024-09-13T16:28:04.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.858+0000 7f4568d2f700 1 -- 172.21.15.119:0/3829412554 wait complete. 2024-09-13T16:28:04.877 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:28:04.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01c882700 1 Processor -- start 2024-09-13T16:28:04.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01c882700 1 -- start start 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01c882700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc0180d17e0 0x7fc018167bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01c882700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0180d0df0 con 0x7fc0180ce8c0 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01c882700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0180cdb00 con 0x7fc01815cc00 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01c882700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0180cdde0 con 0x7fc0180d17e0 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01659c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc0180ce8c0 0x7fc0180d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34384/0 (socket says 172.21.15.119:34384) 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01659c700 1 -- 172.21.15.119:0/492374845 learned_addr learned my addr 172.21.15.119:0/492374845 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc01759e700 1 --2- 172.21.15.119:0/492374845 >> v2:172.21.15.145:3300/0 conn(0x7fc0180d17e0 0x7fc018167bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.994+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 253697956 0 0) 0x7fc0180d0df0 con 0x7fc0180ce8c0 2024-09-13T16:28:04.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0000035d0 con 0x7fc0180ce8c0 2024-09-13T16:28:05.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285766604 0 0) 0x7fc0000035d0 con 0x7fc0180ce8c0 2024-09-13T16:28:05.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 >> v2:172.21.15.145:3300/0 conn(0x7fc0180d17e0 msgr2=0x7fc018167bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:05.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 --2- 172.21.15.119:0/492374845 >> v2:172.21.15.145:3300/0 conn(0x7fc0180d17e0 0x7fc018167bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 >> v1:172.21.15.145:6789/0 conn(0x7fc01815cc00 legacy=0x7fc0180d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0180ce0c0 con 0x7fc0180ce8c0 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fc008001670 con 0x7fc0180ce8c0 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 >> v1:172.21.15.119:6789/0 conn(0x7fc0180ce8c0 legacy=0x7fc0180d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc000003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc000004870 0x7fc000004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc000005230 0x7fc00000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180cdde0 con 0x7fc000003a20 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc00000c0d0 con 0x7fc000004870 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00000c290 con 0x7fc000005230 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc016d9d700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc000003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01659c700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc000004870 0x7fc000004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01759e700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc000005230 0x7fc00000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc016d9d700 1 -- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc000005230 msgr2=0x7fc00000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc016d9d700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc000005230 0x7fc00000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc016d9d700 1 -- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc000004870 msgr2=0x7fc000004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc016d9d700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc000004870 0x7fc000004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc016d9d700 1 -- 172.21.15.119:0/492374845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0000035d0 con 0x7fc000003a20 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01759e700 1 -- 172.21.15.119:0/492374845 reap_dead start 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc016d9d700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc000003e40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0017e0 tx=0x7fc0000058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5978a03496da8f0c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c0052d0 con 0x7fc000003a20 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c005430 con 0x7fc000003a20 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc015d9b700 1 -- 172.21.15.119:0/492374845 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c0055a0 con 0x7fc000003a20 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01c882700 1 -- 172.21.15.119:0/492374845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 msgr2=0x7fc000003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01c882700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc000003e40 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fc00c0017e0 tx=0x7fc0000058a0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01c882700 1 -- 172.21.15.119:0/492374845 shutdown_connections 2024-09-13T16:28:05.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01c882700 1 --2- 172.21.15.119:0/492374845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc000003e40 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01c882700 1 -- 172.21.15.119:0/492374845 >> 172.21.15.119:0/492374845 conn(0x7fc01808c080 msgr2=0x7fc0180d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:05.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01c882700 1 -- 172.21.15.119:0/492374845 shutdown_connections 2024-09-13T16:28:05.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:04.998+0000 7fc01c882700 1 -- 172.21.15.119:0/492374845 wait complete. 2024-09-13T16:28:05.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 Processor -- start 2024-09-13T16:28:05.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 -- start start 2024-09-13T16:28:05.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc0180c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0181522e0 0x7fc018151920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0180b7b10 0x7fc0180b7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181f3170 con 0x7fc000003a20 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0180b84d0 con 0x7fc0180b7b10 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01c882700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0181538d0 con 0x7fc0181522e0 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc0180c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc0180c8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33200/0 (socket says 172.21.15.119:33200) 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 -- 172.21.15.119:0/4049604196 learned_addr learned my addr 172.21.15.119:0/4049604196 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01659c700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0181522e0 0x7fc018151920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01759e700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0180b7b10 0x7fc0180b7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 -- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0181522e0 msgr2=0x7fc018151920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0181522e0 0x7fc018151920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 -- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0180b7b10 msgr2=0x7fc0180b7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0180b7b10 0x7fc0180b7f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc016d9d700 1 -- 172.21.15.119:0/4049604196 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc018153440 con 0x7fc000003a20 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01659c700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0181522e0 0x7fc018151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.002+0000 7fc01759e700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0180b7b10 0x7fc0180b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fc016d9d700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc0180c8800 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc00c005040 tx=0x7fc00c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fbfff7fe700 1 -- 172.21.15.119:0/4049604196 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c0055a0 con 0x7fc000003a20 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fbfff7fe700 1 -- 172.21.15.119:0/4049604196 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00c001290 con 0x7fc000003a20 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc018151e60 con 0x7fc000003a20 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc018152b10 con 0x7fc000003a20 2024-09-13T16:28:05.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fbfff7fe700 1 -- 172.21.15.119:0/4049604196 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c007280 con 0x7fc000003a20 2024-09-13T16:28:05.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0180c63f0 con 0x7fc000003a20 2024-09-13T16:28:05.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fbfff7fe700 1 -- 172.21.15.119:0/4049604196 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc00c0073e0 con 0x7fc000003a20 2024-09-13T16:28:05.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fbfff7fe700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc004066730 0x7fc004068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fbfff7fe700 1 -- 172.21.15.119:0/4049604196 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc00c084f00 con 0x7fc000003a20 2024-09-13T16:28:05.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.006+0000 7fc01659c700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc004066730 0x7fc004068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.010+0000 7fc01659c700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc004066730 0x7fc004068bf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc018155580 tx=0x7fc008009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.014+0000 7fbfff7fe700 1 -- 172.21.15.119:0/4049604196 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc0180c63f0 con 0x7fc000003a20 2024-09-13T16:28:05.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc0180c63f0 con 0x7fc000003a20 2024-09-13T16:28:05.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fbfff7fe700 1 -- 172.21.15.119:0/4049604196 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v54) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc0180c63f0 con 0x7fc000003a20 2024-09-13T16:28:05.188 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:28:05.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc004066730 msgr2=0x7fc004068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc004066730 0x7fc004068bf0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc018155580 tx=0x7fc008009040 comp rx=0 tx=0).stop 2024-09-13T16:28:05.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 msgr2=0x7fc0180c8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc0180c8800 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fc00c005040 tx=0x7fc00c001120 comp rx=0 tx=0).stop 2024-09-13T16:28:05.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 shutdown_connections 2024-09-13T16:28:05.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc004066730 0x7fc004068bf0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0180b7b10 0x7fc0180b7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc0181522e0 0x7fc018151920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 --2- 172.21.15.119:0/4049604196 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc000003a20 0x7fc0180c8800 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 >> 172.21.15.119:0/4049604196 conn(0x7fc01808c080 msgr2=0x7fc018066450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:05.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 shutdown_connections 2024-09-13T16:28:05.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.186+0000 7fc01c882700 1 -- 172.21.15.119:0/4049604196 wait complete. 2024-09-13T16:28:05.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:28:05.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.310+0000 7f9d24d2b700 1 Processor -- start 2024-09-13T16:28:05.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.310+0000 7f9d24d2b700 1 -- start start 2024-09-13T16:28:05.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9d2015e840 0x7f9d20070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d20161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d20071210 con 0x7f9d2015ec20 2024-09-13T16:28:05.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d200714f0 con 0x7f9d2015e840 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200717d0 con 0x7f9d20165180 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9d2015e840 0x7f9d20070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1dd9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9d2015ec20 0x7f9d20164a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34386/0 (socket says 172.21.15.119:34386) 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1dd9b700 1 -- 172.21.15.119:0/3547247890 learned_addr learned my addr 172.21.15.119:0/3547247890 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1ed9d700 1 --2- 172.21.15.119:0/3547247890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d20161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1d59a700 1 -- 172.21.15.119:0/3547247890 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231881730 0 0) 0x7f9d20071210 con 0x7f9d2015ec20 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1e59c700 1 -- 172.21.15.119:0/3547247890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 msgr2=0x7f9d20161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1e59c700 1 --2- 172.21.15.119:0/3547247890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d20161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1e59c700 1 -- 172.21.15.119:0/3547247890 >> v1:172.21.15.119:6789/0 conn(0x7f9d2015ec20 legacy=0x7f9d20164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1e59c700 1 -- 172.21.15.119:0/3547247890 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d20071ab0 con 0x7f9d2015e840 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1ed9d700 1 --2- 172.21.15.119:0/3547247890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d20161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1e59c700 1 --2- 172.21.15.119:0/3547247890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d20070b80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009850 tx=0x7f9d18009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1840c6fe28cb23f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1d59a700 1 -- 172.21.15.119:0/3547247890 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18011830 con 0x7f9d2015e840 2024-09-13T16:28:05.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1d59a700 1 -- 172.21.15.119:0/3547247890 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d18011990 con 0x7f9d2015e840 2024-09-13T16:28:05.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d1d59a700 1 -- 172.21.15.119:0/3547247890 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18011d20 con 0x7f9d2015e840 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- 172.21.15.119:0/3547247890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 msgr2=0x7f9d20070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/3547247890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d20070b80 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009850 tx=0x7f9d18009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- 172.21.15.119:0/3547247890 shutdown_connections 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/3547247890 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d20161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/3547247890 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d20070b80 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- 172.21.15.119:0/3547247890 >> 172.21.15.119:0/3547247890 conn(0x7f9d2006a350 msgr2=0x7f9d2006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- 172.21.15.119:0/3547247890 shutdown_connections 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.314+0000 7f9d24d2b700 1 -- 172.21.15.119:0/3547247890 wait complete. 2024-09-13T16:28:05.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 Processor -- start 2024-09-13T16:28:05.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 -- start start 2024-09-13T16:28:05.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d2014cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d2014a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2014a9d0 0x7f9d20160150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2014d470 con 0x7f9d2015e840 2024-09-13T16:28:05.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2015f0b0 con 0x7f9d2014a9d0 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2015f390 con 0x7f9d20165180 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d2014cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d2014cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33220/0 (socket says 172.21.15.119:33220) 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 -- 172.21.15.119:0/2830149680 learned_addr learned my addr 172.21.15.119:0/2830149680 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1dd9b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d2014a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1ed9d700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2014a9d0 0x7f9d20160150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 -- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 msgr2=0x7f9d2014a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d2014a490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 -- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2014a9d0 msgr2=0x7f9d20160150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2014a9d0 0x7f9d20160150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 -- 172.21.15.119:0/2830149680 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d180094b0 con 0x7f9d2015e840 2024-09-13T16:28:05.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d1e59c700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d2014cf30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d18000c00 tx=0x7f9d18004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d16ffd700 1 -- 172.21.15.119:0/2830149680 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18004550 con 0x7f9d2015e840 2024-09-13T16:28:05.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d16ffd700 1 -- 172.21.15.119:0/2830149680 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1801f070 con 0x7f9d2015e840 2024-09-13T16:28:05.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2015f670 con 0x7f9d2015e840 2024-09-13T16:28:05.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d16ffd700 1 -- 172.21.15.119:0/2830149680 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18003900 con 0x7f9d2015e840 2024-09-13T16:28:05.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.318+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d2015fc30 con 0x7f9d2015e840 2024-09-13T16:28:05.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.322+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d20067760 con 0x7f9d2015e840 2024-09-13T16:28:05.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.322+0000 7f9d16ffd700 1 -- 172.21.15.119:0/2830149680 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9d18003a60 con 0x7f9d2015e840 2024-09-13T16:28:05.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.322+0000 7f9d16ffd700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d04066760 0x7f9d04068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.326+0000 7f9d16ffd700 1 -- 172.21.15.119:0/2830149680 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9d18021070 con 0x7f9d2015e840 2024-09-13T16:28:05.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.326+0000 7f9d1dd9b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d04066760 0x7f9d04068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.326+0000 7f9d1dd9b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d04066760 0x7f9d04068c20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9d2015b3b0 tx=0x7f9d08001b70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.330+0000 7f9d16ffd700 1 -- 172.21.15.119:0/2830149680 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9d18052d60 con 0x7f9d2015e840 2024-09-13T16:28:05.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.510+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9d20071210 con 0x7f9d2015e840 2024-09-13T16:28:05.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.510+0000 7f9d16ffd700 1 -- 172.21.15.119:0/2830149680 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v54) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9d18015080 con 0x7f9d2015e840 2024-09-13T16:28:05.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d04066760 msgr2=0x7f9d04068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d04066760 0x7f9d04068c20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9d2015b3b0 tx=0x7f9d08001b70 comp rx=0 tx=0).stop 2024-09-13T16:28:05.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 msgr2=0x7f9d2014cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d2014cf30 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f9d18000c00 tx=0x7f9d18004770 comp rx=0 tx=0).stop 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 shutdown_connections 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d04066760 0x7f9d04068c20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2014a9d0 0x7f9d20160150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d20165180 0x7f9d2014a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 --2- 172.21.15.119:0/2830149680 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2015e840 0x7f9d2014cf30 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 >> 172.21.15.119:0/2830149680 conn(0x7f9d2006a350 msgr2=0x7f9d2006d110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 shutdown_connections 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.514+0000 7f9d24d2b700 1 -- 172.21.15.119:0/2830149680 wait complete. 2024-09-13T16:28:05.517 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:28:05.528 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:28:05.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:05.528 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:28:05.536 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[40288]: fuse finished with error 0 and tester_r 0 2024-09-13T16:28:05.538 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:28:05.538 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:05.538 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:28:05.664 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:28:05.665 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:05.672 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:28:05.673 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:28:05.673 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:05.673 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:28:05.724 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-13T16:28:05.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:28:05.725 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:28:05.725 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:05.733 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:28:05.733 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:28:05.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:05.733 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:28:05.781 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-09-13T16:28:05.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:28:05.782 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:28:05.782 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:05.789 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:28:05.789 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:28:05.789 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:05.789 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:28:05.796 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-13T16:28:05.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:28:05.797 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:05.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 Processor -- start 2024-09-13T16:28:05.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 -- start start 2024-09-13T16:28:05.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b401594f0 0x7f8b4006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8b400b3800 0x7f8b4006fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b400b2280 con 0x7f8b4006da70 2024-09-13T16:28:05.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b4006deb0 con 0x7f8b400b3800 2024-09-13T16:28:05.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b400cc890 con 0x7f8b401594f0 2024-09-13T16:28:05.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b401594f0 0x7f8b4006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3ffff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8b400b3800 0x7f8b4006fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3effd700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f8b4006da70 0x7f8b400b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:48904/0 (socket says 172.21.15.119:48904) 2024-09-13T16:28:05.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b401594f0 0x7f8b4006d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:41726/0 (socket says 172.21.15.119:41726) 2024-09-13T16:28:05.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3effd700 1 -- 172.21.15.119:0/1725729259 learned_addr learned my addr 172.21.15.119:0/1725729259 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:05.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3ffff700 1 -- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b401594f0 msgr2=0x7f8b4006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:05.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3ffff700 1 --2- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b401594f0 0x7f8b4006d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3ffff700 1 -- 172.21.15.119:0/1725729259 >> v1:172.21.15.145:6789/0 conn(0x7f8b4006da70 legacy=0x7f8b400b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3e7fc700 1 -- 172.21.15.119:0/1725729259 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073097524 0 0) 0x7f8b400b2280 con 0x7f8b4006da70 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3ffff700 1 -- 172.21.15.119:0/1725729259 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b400b1970 con 0x7f8b400b3800 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3f7fe700 1 --2- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b401594f0 0x7f8b4006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3ffff700 1 --2- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b400b3800 0x7f8b4006fc00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8b400b1b60 tx=0x7f8b3800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d29d129341ec001 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3e7fc700 1 -- 172.21.15.119:0/1725729259 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b38016070 con 0x7f8b400b3800 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3e7fc700 1 -- 172.21.15.119:0/1725729259 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3800bb20 con 0x7f8b400b3800 2024-09-13T16:28:05.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b3e7fc700 1 -- 172.21.15.119:0/1725729259 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b3800e070 con 0x7f8b400b3800 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 -- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b400b3800 msgr2=0x7f8b4006fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.918+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b400b3800 0x7f8b4006fc00 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8b400b1b60 tx=0x7f8b3800b330 comp rx=0 tx=0).stop 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- 172.21.15.119:0/1725729259 shutdown_connections 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b400b3800 0x7f8b4006fc00 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/1725729259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b401594f0 0x7f8b4006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- 172.21.15.119:0/1725729259 >> 172.21.15.119:0/1725729259 conn(0x7f8b4006a350 msgr2=0x7f8b4006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- 172.21.15.119:0/1725729259 shutdown_connections 2024-09-13T16:28:05.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- 172.21.15.119:0/1725729259 wait complete. 2024-09-13T16:28:05.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 Processor -- start 2024-09-13T16:28:05.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- start start 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4006da70 0x7f8b401f0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b400b3800 0x7f8b400ab6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b401594f0 0x7f8b400abbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b401eeb10 con 0x7f8b400b3800 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b401583e0 con 0x7f8b401594f0 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b44aa3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b401577d0 con 0x7f8b4006da70 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b3effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b400b3800 0x7f8b400ab6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b3f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4006da70 0x7f8b401f0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b3effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b400b3800 0x7f8b400ab6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33222/0 (socket says 172.21.15.119:33222) 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b3effd700 1 -- 172.21.15.119:0/980181118 learned_addr learned my addr 172.21.15.119:0/980181118 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:05.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.922+0000 7f8b3ffff700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b401594f0 0x7f8b400abbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b3effd700 1 -- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4006da70 msgr2=0x7f8b401f0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b3effd700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4006da70 0x7f8b401f0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b3effd700 1 -- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b401594f0 msgr2=0x7f8b400abbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b3effd700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b401594f0 0x7f8b400abbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b3effd700 1 -- 172.21.15.119:0/980181118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b300094b0 con 0x7f8b400b3800 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b3effd700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b400b3800 0x7f8b400ab6b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c4e0 tx=0x7f8b3400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b27fff700 1 -- 172.21.15.119:0/980181118 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34004ca0 con 0x7f8b400b3800 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b27fff700 1 -- 172.21.15.119:0/980181118 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34004e20 con 0x7f8b400b3800 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b38009a70 con 0x7f8b400b3800 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b27fff700 1 -- 172.21.15.119:0/980181118 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b34007310 con 0x7f8b400b3800 2024-09-13T16:28:05.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b40157d90 con 0x7f8b400b3800 2024-09-13T16:28:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b27fff700 1 -- 172.21.15.119:0/980181118 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8b34007470 con 0x7f8b400b3800 2024-09-13T16:28:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b401eec90 con 0x7f8b400b3800 2024-09-13T16:28:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b27fff700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b2c066730 0x7f8b2c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b27fff700 1 -- 172.21.15.119:0/980181118 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8b34083e30 con 0x7f8b400b3800 2024-09-13T16:28:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.926+0000 7f8b3f7fe700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b2c066730 0x7f8b2c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.930+0000 7f8b3f7fe700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b2c066730 0x7f8b2c068bf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009d80 tx=0x7f8b30009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:05.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:05.934+0000 7f8b27fff700 1 -- 172.21.15.119:0/980181118 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8b34051060 con 0x7f8b400b3800 2024-09-13T16:28:06.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.142+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b400b30f0 con 0x7f8b400b3800 2024-09-13T16:28:06.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.142+0000 7f8b27fff700 1 -- 172.21.15.119:0/980181118 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f8b34007720 con 0x7f8b400b3800 2024-09-13T16:28:06.146 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:06.147 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b2c066730 msgr2=0x7f8b2c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b2c066730 0x7f8b2c068bf0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8b30009d80 tx=0x7f8b30009240 comp rx=0 tx=0).stop 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b400b3800 msgr2=0x7f8b400ab6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b400b3800 0x7f8b400ab6b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c4e0 tx=0x7f8b3400c7f0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 shutdown_connections 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b2c066730 0x7f8b2c068bf0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b401594f0 0x7f8b400abbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b400b3800 0x7f8b400ab6b0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 --2- 172.21.15.119:0/980181118 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b4006da70 0x7f8b401f0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 >> 172.21.15.119:0/980181118 conn(0x7f8b4006a350 msgr2=0x7f8b400ae4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:06.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 shutdown_connections 2024-09-13T16:28:06.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.146+0000 7f8b44aa3700 1 -- 172.21.15.119:0/980181118 wait complete. 2024-09-13T16:28:06.150 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:28:06.161 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:06.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04ffb01700 1 Processor -- start 2024-09-13T16:28:06.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04ffb01700 1 -- start start 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04ffb01700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f815ec20 0x7f04f8070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04ffb01700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04f8165180 0x7f04f8161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04ffb01700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f8071210 con 0x7f04f815e840 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04ffb01700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f80714f0 con 0x7f04f8165180 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04ffb01700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80717d0 con 0x7f04f815ec20 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fe09e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04f8165180 0x7f04f8161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd89d700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f04f815e840 0x7f04f8164a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:48910/0 (socket says 172.21.15.119:48910) 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd89d700 1 -- 172.21.15.119:0/3750762452 learned_addr learned my addr 172.21.15.119:0/3750762452 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:06.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd09c700 1 --2- 172.21.15.119:0/3750762452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f815ec20 0x7f04f8070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fc89b700 1 -- 172.21.15.119:0/3750762452 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801031017 0 0) 0x7f04f8071210 con 0x7f04f815e840 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fc89b700 1 -- 172.21.15.119:0/3750762452 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e40035d0 con 0x7f04f815e840 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd09c700 1 -- 172.21.15.119:0/3750762452 >> v2:172.21.15.119:3300/0 conn(0x7f04f8165180 msgr2=0x7f04f8161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd09c700 1 --2- 172.21.15.119:0/3750762452 >> v2:172.21.15.119:3300/0 conn(0x7f04f8165180 0x7f04f8161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd09c700 1 -- 172.21.15.119:0/3750762452 >> v1:172.21.15.145:6789/0 conn(0x7f04f815e840 legacy=0x7f04f8164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd09c700 1 -- 172.21.15.119:0/3750762452 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8071ab0 con 0x7f04f815ec20 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fe09e700 1 --2- 172.21.15.119:0/3750762452 >> v2:172.21.15.119:3300/0 conn(0x7f04f8165180 0x7f04f8161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fd09c700 1 --2- 172.21.15.119:0/3750762452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f815ec20 0x7f04f8070b80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009d90 tx=0x7f04f000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8874abeb25b60f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fc89b700 1 -- 172.21.15.119:0/3750762452 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f000e070 con 0x7f04f815ec20 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fc89b700 1 -- 172.21.15.119:0/3750762452 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f000bb20 con 0x7f04f815ec20 2024-09-13T16:28:06.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.270+0000 7f04fc89b700 1 -- 172.21.15.119:0/3750762452 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0012430 con 0x7f04f815ec20 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 -- 172.21.15.119:0/3750762452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f815ec20 msgr2=0x7f04f8070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 --2- 172.21.15.119:0/3750762452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f815ec20 0x7f04f8070b80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f04f0009d90 tx=0x7f04f000b330 comp rx=0 tx=0).stop 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 -- 172.21.15.119:0/3750762452 shutdown_connections 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 --2- 172.21.15.119:0/3750762452 >> v2:172.21.15.119:3300/0 conn(0x7f04f8165180 0x7f04f8161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 --2- 172.21.15.119:0/3750762452 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f815ec20 0x7f04f8070b80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 -- 172.21.15.119:0/3750762452 >> 172.21.15.119:0/3750762452 conn(0x7f04f806a350 msgr2=0x7f04f806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 -- 172.21.15.119:0/3750762452 shutdown_connections 2024-09-13T16:28:06.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 -- 172.21.15.119:0/3750762452 wait complete. 2024-09-13T16:28:06.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 Processor -- start 2024-09-13T16:28:06.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.274+0000 7f04ffb01700 1 -- start start 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f815e840 0x7f04f80adbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f8165180 0x7f04f8159bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f80af9f0 0x7f04f8066090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80aee10 con 0x7f04f8165180 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f815a0f0 con 0x7f04f815e840 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f80ae200 con 0x7f04f80af9f0 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f8165180 0x7f04f8159bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f8165180 0x7f04f8159bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33248/0 (socket says 172.21.15.119:33248) 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 -- 172.21.15.119:0/997376515 learned_addr learned my addr 172.21.15.119:0/997376515 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd89d700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f815e840 0x7f04f80adbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fe09e700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f80af9f0 0x7f04f8066090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 -- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f80af9f0 msgr2=0x7f04f8066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f80af9f0 0x7f04f8066090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 -- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f815e840 msgr2=0x7f04f80adbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f815e840 0x7f04f80adbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 -- 172.21.15.119:0/997376515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40094b0 con 0x7f04f8165180 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fe09e700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f80af9f0 0x7f04f8066090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04fd09c700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f8165180 0x7f04f8159bb0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f04f0014040 tx=0x7f04f0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ee7fc700 1 -- 172.21.15.119:0/997376515 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0012430 con 0x7f04f8165180 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ee7fc700 1 -- 172.21.15.119:0/997376515 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f0020070 con 0x7f04f8165180 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f0009a70 con 0x7f04f8165180 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ee7fc700 1 -- 172.21.15.119:0/997376515 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f0004280 con 0x7f04f8165180 2024-09-13T16:28:06.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f80ae7c0 con 0x7f04f8165180 2024-09-13T16:28:06.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.278+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f8067760 con 0x7f04f8165180 2024-09-13T16:28:06.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.282+0000 7f04ee7fc700 1 -- 172.21.15.119:0/997376515 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04f00043e0 con 0x7f04f8165180 2024-09-13T16:28:06.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.282+0000 7f04ee7fc700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04dc066760 0x7f04dc068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.282+0000 7f04ee7fc700 1 -- 172.21.15.119:0/997376515 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f04f0022070 con 0x7f04f8165180 2024-09-13T16:28:06.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.282+0000 7f04fd89d700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04dc066760 0x7f04dc068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.282+0000 7f04fd89d700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04dc066760 0x7f04dc068c20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f04e8007fd0 tx=0x7f04e8007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.290+0000 7f04ee7fc700 1 -- 172.21.15.119:0/997376515 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f04f00534e0 con 0x7f04f8165180 2024-09-13T16:28:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.462+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04f8071210 con 0x7f04f8165180 2024-09-13T16:28:06.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.466+0000 7f04ee7fc700 1 -- 172.21.15.119:0/997376515 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+16834 (secure 0 0 0) 0x7f04f0056b00 con 0x7f04f8165180 2024-09-13T16:28:06.468 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:06.468 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":54,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:27:24.352264+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:27:21.660330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-09-13T16:27:22.341239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04dc066760 msgr2=0x7f04dc068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04dc066760 0x7f04dc068c20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f04e8007fd0 tx=0x7f04e8007de0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f8165180 msgr2=0x7f04f8159bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f8165180 0x7f04f8159bb0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f04f0014040 tx=0x7f04f0004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 shutdown_connections 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04dc066760 0x7f04dc068c20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f80af9f0 0x7f04f8066090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f8165180 0x7f04f8159bb0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 --2- 172.21.15.119:0/997376515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f815e840 0x7f04f80adbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 >> 172.21.15.119:0/997376515 conn(0x7f04f806a350 msgr2=0x7f04f81609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 shutdown_connections 2024-09-13T16:28:06.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.470+0000 7f04ffb01700 1 -- 172.21.15.119:0/997376515 wait complete. 2024-09-13T16:28:06.487 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:27:21.660330+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '53', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:27:22.341239+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '54', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:28:06.488 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:28:06.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.598+0000 7f0e13fff700 1 Processor -- start 2024-09-13T16:28:06.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e13fff700 1 -- start start 2024-09-13T16:28:06.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e13fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e1415cad0 0x7f0e140d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e13fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0e140ce0d0 0x7f0e140ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e13fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e140d05b0 con 0x7f0e140cf220 2024-09-13T16:28:06.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e13fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e140d0870 con 0x7f0e140ce0d0 2024-09-13T16:28:06.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e13fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e140d0b50 con 0x7f0e1415cad0 2024-09-13T16:28:06.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e12ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e1415cad0 0x7f0e140d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0e140ce0d0 0x7f0e140ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0e140ce0d0 0x7f0e140ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:49194/0 (socket says 172.21.15.119:49194) 2024-09-13T16:28:06.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e12ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e1415cad0 0x7f0e140d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:41754/0 (socket says 172.21.15.119:41754) 2024-09-13T16:28:06.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 -- 172.21.15.119:0/2671214048 learned_addr learned my addr 172.21.15.119:0/2671214048 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 -- 172.21.15.119:0/2671214048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e1415cad0 msgr2=0x7f0e140d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 --2- 172.21.15.119:0/2671214048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e1415cad0 0x7f0e140d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 -- 172.21.15.119:0/2671214048 >> v1:172.21.15.145:6789/0 conn(0x7f0e140cf220 legacy=0x7f0e140b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 -- 172.21.15.119:0/2671214048 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e140d0e30 con 0x7f0e140ce0d0 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e127fc700 1 --2- 172.21.15.119:0/2671214048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 0x7f0e140ce510 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009910 tx=0x7f0e08009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4840ef011d13ea00 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e11ffb700 1 -- 172.21.15.119:0/2671214048 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08011830 con 0x7f0e140ce0d0 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e11ffb700 1 -- 172.21.15.119:0/2671214048 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e08011990 con 0x7f0e140ce0d0 2024-09-13T16:28:06.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.602+0000 7f0e11ffb700 1 -- 172.21.15.119:0/2671214048 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e08011d40 con 0x7f0e140ce0d0 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 -- 172.21.15.119:0/2671214048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 msgr2=0x7f0e140ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2671214048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 0x7f0e140ce510 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f0e08009910 tx=0x7f0e08009c20 comp rx=0 tx=0).stop 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 -- 172.21.15.119:0/2671214048 shutdown_connections 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2671214048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 0x7f0e140ce510 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2671214048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e1415cad0 0x7f0e140d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 -- 172.21.15.119:0/2671214048 >> 172.21.15.119:0/2671214048 conn(0x7f0e1408bf70 msgr2=0x7f0e140d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 -- 172.21.15.119:0/2671214048 shutdown_connections 2024-09-13T16:28:06.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 -- 172.21.15.119:0/2671214048 wait complete. 2024-09-13T16:28:06.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 Processor -- start 2024-09-13T16:28:06.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.606+0000 7f0e13fff700 1 -- start start 2024-09-13T16:28:06.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 0x7f0e141f2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e140cf220 0x7f0e14157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e1415cad0 0x7f0e14158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14159c80 con 0x7f0e1415cad0 2024-09-13T16:28:06.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14159f60 con 0x7f0e140ce0d0 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e14154200 con 0x7f0e140cf220 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e1415cad0 0x7f0e14158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e1415cad0 0x7f0e14158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33258/0 (socket says 172.21.15.119:33258) 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 -- 172.21.15.119:0/2167314976 learned_addr learned my addr 172.21.15.119:0/2167314976 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e12ffd700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 0x7f0e141f2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e127fc700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e140cf220 0x7f0e14157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 -- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e140cf220 msgr2=0x7f0e14157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e140cf220 0x7f0e14157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 -- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 msgr2=0x7f0e141f2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 0x7f0e141f2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 -- 172.21.15.119:0/2167314976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e08009570 con 0x7f0e1415cad0 2024-09-13T16:28:06.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e137fe700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e1415cad0 0x7f0e14158350 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00cc50 tx=0x7f0e0c00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0dfbfff700 1 -- 172.21.15.119:0/2167314976 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c003c20 con 0x7f0e1415cad0 2024-09-13T16:28:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0dfbfff700 1 -- 172.21.15.119:0/2167314976 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e0c003d80 con 0x7f0e1415cad0 2024-09-13T16:28:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e141544e0 con 0x7f0e1415cad0 2024-09-13T16:28:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0dfbfff700 1 -- 172.21.15.119:0/2167314976 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e0c00aab0 con 0x7f0e1415cad0 2024-09-13T16:28:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e141f2970 con 0x7f0e1415cad0 2024-09-13T16:28:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e14154a00 con 0x7f0e1415cad0 2024-09-13T16:28:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.610+0000 7f0dfbfff700 1 -- 172.21.15.119:0/2167314976 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0e0c00ac10 con 0x7f0e1415cad0 2024-09-13T16:28:06.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.614+0000 7f0dfbfff700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e00066730 0x7f0e00068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.614+0000 7f0dfbfff700 1 -- 172.21.15.119:0/2167314976 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0e0c0842f0 con 0x7f0e1415cad0 2024-09-13T16:28:06.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.614+0000 7f0e12ffd700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e00066730 0x7f0e00068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.614+0000 7f0e12ffd700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e00066730 0x7f0e00068bf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009b50 tx=0x7f0e04009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.622+0000 7f0dfbfff700 1 -- 172.21.15.119:0/2167314976 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0e14154a00 con 0x7f0e1415cad0 2024-09-13T16:28:06.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.802+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e14154a00 con 0x7f0e1415cad0 2024-09-13T16:28:06.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.802+0000 7f0dfbfff700 1 -- 172.21.15.119:0/2167314976 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v55) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e14154a00 con 0x7f0e1415cad0 2024-09-13T16:28:06.804 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:28:06.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e00066730 msgr2=0x7f0e00068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e00066730 0x7f0e00068bf0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f0e04009b50 tx=0x7f0e04009240 comp rx=0 tx=0).stop 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e1415cad0 msgr2=0x7f0e14158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e1415cad0 0x7f0e14158350 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f0e0c00cc50 tx=0x7f0e0c00cf60 comp rx=0 tx=0).stop 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 shutdown_connections 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0e00066730 0x7f0e00068bf0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0e1415cad0 0x7f0e14158350 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0e140cf220 0x7f0e14157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 --2- 172.21.15.119:0/2167314976 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0e140ce0d0 0x7f0e141f2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 >> 172.21.15.119:0/2167314976 conn(0x7f0e1408bf70 msgr2=0x7f0e140d1480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 shutdown_connections 2024-09-13T16:28:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.806+0000 7f0e13fff700 1 -- 172.21.15.119:0/2167314976 wait complete. 2024-09-13T16:28:06.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:28:06.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 Processor -- start 2024-09-13T16:28:06.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 -- start start 2024-09-13T16:28:06.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f785806b710 0x7f785806bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f78580d1640 0x7f78580d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f785806c0a0 con 0x7f785815cfb0 2024-09-13T16:28:06.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78580cee40 con 0x7f785806b710 2024-09-13T16:28:06.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78580d1220 con 0x7f78580d1640 2024-09-13T16:28:06.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f785806b710 0x7f785806bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f785806b710 0x7f785806bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33268/0 (socket says 172.21.15.119:33268) 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 -- 172.21.15.119:0/2131253589 learned_addr learned my addr 172.21.15.119:0/2131253589 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785e0ef700 1 --2- 172.21.15.119:0/2131253589 >> v2:172.21.15.145:3300/0 conn(0x7f78580d1640 0x7f78580d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 -- 172.21.15.119:0/2131253589 >> v2:172.21.15.145:3300/0 conn(0x7f78580d1640 msgr2=0x7f78580d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 --2- 172.21.15.119:0/2131253589 >> v2:172.21.15.145:3300/0 conn(0x7f78580d1640 0x7f78580d1ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 -- 172.21.15.119:0/2131253589 >> v1:172.21.15.145:6789/0 conn(0x7f785815cfb0 legacy=0x7f785806ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 -- 172.21.15.119:0/2131253589 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78580ce880 con 0x7f785806b710 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785e0ef700 1 --2- 172.21.15.119:0/2131253589 >> v2:172.21.15.145:3300/0 conn(0x7f78580d1640 0x7f78580d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785d0ed700 1 --2- 172.21.15.119:0/2131253589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f785806bb60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f78580d2d40 tx=0x7f7854009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=986092d0f70dcb42 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785c8ec700 1 -- 172.21.15.119:0/2131253589 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7854004230 con 0x7f785806b710 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785c8ec700 1 -- 172.21.15.119:0/2131253589 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7854004390 con 0x7f785806b710 2024-09-13T16:28:06.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785c8ec700 1 -- 172.21.15.119:0/2131253589 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78540036c0 con 0x7f785806b710 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 -- 172.21.15.119:0/2131253589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 msgr2=0x7f785806bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.942+0000 7f785fb52700 1 --2- 172.21.15.119:0/2131253589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f785806bb60 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f78580d2d40 tx=0x7f7854009bf0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- 172.21.15.119:0/2131253589 shutdown_connections 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 --2- 172.21.15.119:0/2131253589 >> v2:172.21.15.145:3300/0 conn(0x7f78580d1640 0x7f78580d1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 --2- 172.21.15.119:0/2131253589 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f785806bb60 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- 172.21.15.119:0/2131253589 >> 172.21.15.119:0/2131253589 conn(0x7f78580d2da0 msgr2=0x7f78580d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- 172.21.15.119:0/2131253589 shutdown_connections 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- 172.21.15.119:0/2131253589 wait complete. 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 Processor -- start 2024-09-13T16:28:06.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- start start 2024-09-13T16:28:06.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f78580b18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78580d1640 0x7f78580b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f785815cfb0 0x7f7858150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78580b6ff0 con 0x7f785806b710 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78580b72d0 con 0x7f785815cfb0 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785fb52700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78580b75b0 con 0x7f78580d1640 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f78580b18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f78580b18d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33284/0 (socket says 172.21.15.119:33284) 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 -- 172.21.15.119:0/562713719 learned_addr learned my addr 172.21.15.119:0/562713719 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785e0ef700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f785815cfb0 0x7f7858150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d0ed700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78580d1640 0x7f78580b65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 -- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78580d1640 msgr2=0x7f78580b65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78580d1640 0x7f78580b65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 -- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f785815cfb0 msgr2=0x7f7858150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f785815cfb0 0x7f7858150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 -- 172.21.15.119:0/562713719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7854009570 con 0x7f785806b710 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.946+0000 7f785d8ee700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f78580b18d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f784c00b4a0 tx=0x7f784c00b7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f784a7fc700 1 -- 172.21.15.119:0/562713719 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c011830 con 0x7f785806b710 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78580b7890 con 0x7f785806b710 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f784a7fc700 1 -- 172.21.15.119:0/562713719 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c011990 con 0x7f785806b710 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78580b8090 con 0x7f785806b710 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f784a7fc700 1 -- 172.21.15.119:0/562713719 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784c011d00 con 0x7f785806b710 2024-09-13T16:28:06.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78580b7db0 con 0x7f785806b710 2024-09-13T16:28:06.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f784a7fc700 1 -- 172.21.15.119:0/562713719 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f784c01a400 con 0x7f785806b710 2024-09-13T16:28:06.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f784a7fc700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f783c066730 0x7f783c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:06.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f784a7fc700 1 -- 172.21.15.119:0/562713719 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f784c084250 con 0x7f785806b710 2024-09-13T16:28:06.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.950+0000 7f785d0ed700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f783c066730 0x7f783c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:06.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.954+0000 7f785d0ed700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f783c066730 0x7f783c068bf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f78580d2d40 tx=0x7f7854008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:06.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:06.958+0000 7f784a7fc700 1 -- 172.21.15.119:0/562713719 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f78580b7db0 con 0x7f785806b710 2024-09-13T16:28:07.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.126+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f78580b7db0 con 0x7f785806b710 2024-09-13T16:28:07.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.126+0000 7f784a7fc700 1 -- 172.21.15.119:0/562713719 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v55) v1 ==== 104+0+11 (secure 0 0 0) 0x7f78580b7db0 con 0x7f785806b710 2024-09-13T16:28:07.131 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:28:07.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f783c066730 msgr2=0x7f783c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f783c066730 0x7f783c068bf0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f78580d2d40 tx=0x7f7854008030 comp rx=0 tx=0).stop 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 msgr2=0x7f78580b18d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f78580b18d0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f784c00b4a0 tx=0x7f784c00b7b0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 shutdown_connections 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f783c066730 0x7f783c068bf0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f785815cfb0 0x7f7858150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f78580d1640 0x7f78580b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 --2- 172.21.15.119:0/562713719 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f785806b710 0x7f78580b18d0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 >> 172.21.15.119:0/562713719 conn(0x7f78580d2da0 msgr2=0x7f78580b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 shutdown_connections 2024-09-13T16:28:07.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.130+0000 7f785fb52700 1 -- 172.21.15.119:0/562713719 wait complete. 2024-09-13T16:28:07.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:28:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f35a4926700 1 Processor -- start 2024-09-13T16:28:07.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f35a4926700 1 -- start start 2024-09-13T16:28:07.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f35a4926700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35a006ca60 0x7f35a0159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f35a4926700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00b1dd0 con 0x7f35a00b1860 2024-09-13T16:28:07.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f35a4926700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a00b20b0 con 0x7f35a00ad9a0 2024-09-13T16:28:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f35a4926700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a00b2fd0 con 0x7f35a006ca60 2024-09-13T16:28:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f359ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f35a00b1860 0x7f35a015a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34400/0 (socket says 172.21.15.119:34400) 2024-09-13T16:28:07.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f359ffff700 1 -- 172.21.15.119:0/3983868362 learned_addr learned my addr 172.21.15.119:0/3983868362 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f359effd700 1 --2- 172.21.15.119:0/3983868362 >> v2:172.21.15.145:3300/0 conn(0x7f35a006ca60 0x7f35a0159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2937903834 0 0) 0x7f35a00b1dd0 con 0x7f35a00b1860 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.258+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c0035d0 con 0x7f35a00b1860 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2508759440 0 0) 0x7f358c0035d0 con 0x7f35a00b1860 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 >> v2:172.21.15.145:3300/0 conn(0x7f35a006ca60 msgr2=0x7f35a0159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 --2- 172.21.15.119:0/3983868362 >> v2:172.21.15.145:3300/0 conn(0x7f35a006ca60 0x7f35a0159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 >> v1:172.21.15.145:6789/0 conn(0x7f35a00ad9a0 legacy=0x7f35a006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00b2390 con 0x7f35a00b1860 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f35980019f0 con 0x7f35a00b1860 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 >> v1:172.21.15.119:6789/0 conn(0x7f35a00b1860 legacy=0x7f35a015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c003a20 0x7f358c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f358c004420 0x7f358c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f358c005190 0x7f358c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c00c0a0 con 0x7f358c005190 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f358c00c210 con 0x7f358c003a20 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c00c3d0 con 0x7f358c004420 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359f7fe700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c003a20 0x7f358c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359effd700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f358c004420 0x7f358c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f358c005190 0x7f358c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 -- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f358c004420 msgr2=0x7f358c0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f358c004420 0x7f358c0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 -- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c003a20 msgr2=0x7f358c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c003a20 0x7f358c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 -- 172.21.15.119:0/3983868362 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c0035d0 con 0x7f358c005190 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359effd700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f358c004420 0x7f358c0048f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:07.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359f7fe700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c003a20 0x7f358c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 -- 172.21.15.119:0/3983868362 reap_dead start 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359ffff700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f358c005190 0x7f358c00ba30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f359800a840 tx=0x7f358c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bab35320bb62b73d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35980072d0 con 0x7f358c005190 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3598007470 con 0x7f358c005190 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f359e7fc700 1 -- 172.21.15.119:0/3983868362 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3598007820 con 0x7f358c005190 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f35a4926700 1 -- 172.21.15.119:0/3983868362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f358c005190 msgr2=0x7f358c00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f35a4926700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f358c005190 0x7f358c00ba30 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f359800a840 tx=0x7f358c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f35a4926700 1 -- 172.21.15.119:0/3983868362 shutdown_connections 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f35a4926700 1 --2- 172.21.15.119:0/3983868362 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f358c005190 0x7f358c00ba30 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f35a4926700 1 -- 172.21.15.119:0/3983868362 >> 172.21.15.119:0/3983868362 conn(0x7f35a00b0980 msgr2=0x7f35a00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f35a4926700 1 -- 172.21.15.119:0/3983868362 shutdown_connections 2024-09-13T16:28:07.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.262+0000 7f35a4926700 1 -- 172.21.15.119:0/3983868362 wait complete. 2024-09-13T16:28:07.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 Processor -- start 2024-09-13T16:28:07.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 -- start start 2024-09-13T16:28:07.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c005190 0x7f35a0158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a006a5c0 0x7f35a0133cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35a006aa40 0x7f35a01470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a006b0c0 con 0x7f35a006aa40 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a0133820 con 0x7f358c005190 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f35a4926700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a014c4b0 con 0x7f35a006a5c0 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35a006aa40 0x7f35a01470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35a006aa40 0x7f35a01470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33304/0 (socket says 172.21.15.119:33304) 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 -- 172.21.15.119:0/2066678169 learned_addr learned my addr 172.21.15.119:0/2066678169 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359effd700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a006a5c0 0x7f35a0133cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359f7fe700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c005190 0x7f35a0158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 -- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a006a5c0 msgr2=0x7f35a0133cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a006a5c0 0x7f35a0133cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 -- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c005190 msgr2=0x7f35a0158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c005190 0x7f35a0158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 -- 172.21.15.119:0/2066678169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a0132ef0 con 0x7f35a006aa40 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.266+0000 7f359ffff700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35a006aa40 0x7f35a01470f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f35a00b0510 tx=0x7f35a00cb6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f3597fff700 1 -- 172.21.15.119:0/2066678169 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35980079f0 con 0x7f35a006aa40 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f3597fff700 1 -- 172.21.15.119:0/2066678169 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3598007b50 con 0x7f35a006aa40 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f3597fff700 1 -- 172.21.15.119:0/2066678169 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359801d630 con 0x7f35a006aa40 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a01331d0 con 0x7f35a006aa40 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a014c140 con 0x7f35a006aa40 2024-09-13T16:28:07.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a0133990 con 0x7f35a006aa40 2024-09-13T16:28:07.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f3597fff700 1 -- 172.21.15.119:0/2066678169 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f359801a030 con 0x7f35a006aa40 2024-09-13T16:28:07.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f3597fff700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3580066760 0x7f3580068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f3597fff700 1 -- 172.21.15.119:0/2066678169 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f35980852a0 con 0x7f35a006aa40 2024-09-13T16:28:07.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.270+0000 7f359f7fe700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3580066760 0x7f3580068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.274+0000 7f359f7fe700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3580066760 0x7f3580068c20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35880014c0 tx=0x7f3588009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.278+0000 7f3597fff700 1 -- 172.21.15.119:0/2066678169 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3598052640 con 0x7f35a006aa40 2024-09-13T16:28:07.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.442+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f35a015a170 con 0x7f35a006aa40 2024-09-13T16:28:07.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.442+0000 7f3597fff700 1 -- 172.21.15.119:0/2066678169 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v55) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3598055c60 con 0x7f35a006aa40 2024-09-13T16:28:07.447 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:28:07.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3580066760 msgr2=0x7f3580068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3580066760 0x7f3580068c20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f35880014c0 tx=0x7f3588009040 comp rx=0 tx=0).stop 2024-09-13T16:28:07.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35a006aa40 msgr2=0x7f35a01470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35a006aa40 0x7f35a01470f0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f35a00b0510 tx=0x7f35a00cb6c0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 shutdown_connections 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3580066760 0x7f3580068c20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35a006aa40 0x7f35a01470f0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35a006a5c0 0x7f35a0133cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 --2- 172.21.15.119:0/2066678169 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f358c005190 0x7f35a0158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 >> 172.21.15.119:0/2066678169 conn(0x7f35a00b0980 msgr2=0x7f35a014d470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 shutdown_connections 2024-09-13T16:28:07.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.446+0000 7f35a4926700 1 -- 172.21.15.119:0/2066678169 wait complete. 2024-09-13T16:28:07.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:07.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09ab014700 1 Processor -- start 2024-09-13T16:28:07.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09ab014700 1 -- start start 2024-09-13T16:28:07.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09ab014700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f09a41439b0 0x7f09a406a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09ab014700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a406ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09ab014700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a4143d90 con 0x7f09a4143f50 2024-09-13T16:28:07.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09ab014700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a406f760 con 0x7f09a41439b0 2024-09-13T16:28:07.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09ab014700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a406afa0 con 0x7f09a41444a0 2024-09-13T16:28:07.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09a9811700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f09a4143f50 0x7f09a406a890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:48942/0 (socket says 172.21.15.119:48942) 2024-09-13T16:28:07.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09a9811700 1 -- 172.21.15.119:0/3556786304 learned_addr learned my addr 172.21.15.119:0/3556786304 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:07.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa012700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f09a41439b0 0x7f09a406a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa813700 1 --2- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a406ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09a9010700 1 -- 172.21.15.119:0/3556786304 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502879393 0 0) 0x7f0994001110 con 0x7f09a4143f50 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09a9010700 1 -- 172.21.15.119:0/3556786304 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09900035d0 con 0x7f09a4143f50 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa012700 1 -- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 msgr2=0x7f09a406ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa012700 1 --2- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a406ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa012700 1 -- 172.21.15.119:0/3556786304 >> v1:172.21.15.145:6789/0 conn(0x7f09a4143f50 legacy=0x7f09a406a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa012700 1 -- 172.21.15.119:0/3556786304 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a406b1f0 con 0x7f09a41439b0 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa813700 1 --2- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a406ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:07.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.574+0000 7f09aa012700 1 --2- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a41439b0 0x7f09a406a350 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009850 tx=0x7f09a0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17fa4c8403820467 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09a9010700 1 -- 172.21.15.119:0/3556786304 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0011830 con 0x7f09a41439b0 2024-09-13T16:28:07.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09a9010700 1 -- 172.21.15.119:0/3556786304 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a0011990 con 0x7f09a41439b0 2024-09-13T16:28:07.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09a9010700 1 -- 172.21.15.119:0/3556786304 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0011d40 con 0x7f09a41439b0 2024-09-13T16:28:07.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 -- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a41439b0 msgr2=0x7f09a406a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 --2- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a41439b0 0x7f09a406a350 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f09a0009850 tx=0x7f09a0009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:07.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 -- 172.21.15.119:0/3556786304 shutdown_connections 2024-09-13T16:28:07.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 --2- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a406ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 --2- 172.21.15.119:0/3556786304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a41439b0 0x7f09a406a350 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 -- 172.21.15.119:0/3556786304 >> 172.21.15.119:0/3556786304 conn(0x7f09a40b17a0 msgr2=0x7f09a40b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:07.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 -- 172.21.15.119:0/3556786304 shutdown_connections 2024-09-13T16:28:07.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 -- 172.21.15.119:0/3556786304 wait complete. 2024-09-13T16:28:07.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.578+0000 7f09ab014700 1 Processor -- start 2024-09-13T16:28:07.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 -- start start 2024-09-13T16:28:07.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a41439b0 0x7f09a4130c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a4143f50 0x7f09a41404c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a412e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4155f50 con 0x7f09a41439b0 2024-09-13T16:28:07.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4156a90 con 0x7f09a4143f50 2024-09-13T16:28:07.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a4140a00 con 0x7f09a41444a0 2024-09-13T16:28:07.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a41439b0 0x7f09a4130c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a41439b0 0x7f09a4130c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33316/0 (socket says 172.21.15.119:33316) 2024-09-13T16:28:07.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 -- 172.21.15.119:0/3509690283 learned_addr learned my addr 172.21.15.119:0/3509690283 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:07.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa813700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a412e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09a9811700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a4143f50 0x7f09a41404c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 -- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 msgr2=0x7f09a412e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a412e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 -- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a4143f50 msgr2=0x7f09a41404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a4143f50 0x7f09a41404c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 -- 172.21.15.119:0/3509690283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c009a70 con 0x7f09a41439b0 2024-09-13T16:28:07.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09aa012700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a41439b0 0x7f09a4130c30 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f09a00090b0 tx=0x7f09a0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f099affd700 1 -- 172.21.15.119:0/3509690283 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0004450 con 0x7f09a41439b0 2024-09-13T16:28:07.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f099affd700 1 -- 172.21.15.119:0/3509690283 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a001f070 con 0x7f09a41439b0 2024-09-13T16:28:07.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a00094b0 con 0x7f09a41439b0 2024-09-13T16:28:07.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f099affd700 1 -- 172.21.15.119:0/3509690283 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a0003900 con 0x7f09a41439b0 2024-09-13T16:28:07.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a4156440 con 0x7f09a41439b0 2024-09-13T16:28:07.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.582+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a40cccf0 con 0x7f09a41439b0 2024-09-13T16:28:07.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.586+0000 7f099affd700 1 -- 172.21.15.119:0/3509690283 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f09a0003a60 con 0x7f09a41439b0 2024-09-13T16:28:07.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.586+0000 7f099affd700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0988066730 0x7f0988068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.586+0000 7f099affd700 1 -- 172.21.15.119:0/3509690283 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f09a0021070 con 0x7f09a41439b0 2024-09-13T16:28:07.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.586+0000 7f09a9811700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0988066730 0x7f0988068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.586+0000 7f09a9811700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0988066730 0x7f0988068bf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0994007fd0 tx=0x7f0994007c60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.594+0000 7f099affd700 1 -- 172.21.15.119:0/3509690283 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f09a0052b80 con 0x7f09a41439b0 2024-09-13T16:28:07.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.806+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09a406a350 con 0x7f09a41439b0 2024-09-13T16:28:07.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.806+0000 7f099affd700 1 -- 172.21.15.119:0/3509690283 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f09a0015090 con 0x7f09a41439b0 2024-09-13T16:28:07.811 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:07.811 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:28:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0988066730 msgr2=0x7f0988068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0988066730 0x7f0988068bf0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f0994007fd0 tx=0x7f0994007c60 comp rx=0 tx=0).stop 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a41439b0 msgr2=0x7f09a4130c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a41439b0 0x7f09a4130c30 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f09a00090b0 tx=0x7f09a0004770 comp rx=0 tx=0).stop 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 shutdown_connections 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0988066730 0x7f0988068bf0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f09a41444a0 0x7f09a412e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f09a4143f50 0x7f09a41404c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 --2- 172.21.15.119:0/3509690283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f09a41439b0 0x7f09a4130c30 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 >> 172.21.15.119:0/3509690283 conn(0x7f09a40b17a0 msgr2=0x7f09a406e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 shutdown_connections 2024-09-13T16:28:07.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.810+0000 7f09ab014700 1 -- 172.21.15.119:0/3509690283 wait complete. 2024-09-13T16:28:07.815 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:28:07.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:07.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce36ba7700 1 Processor -- start 2024-09-13T16:28:07.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce36ba7700 1 -- start start 2024-09-13T16:28:07.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce36ba7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fce30146230 0x7fce3006bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce36ba7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce3015cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce36ba7700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300b2e70 con 0x7fce300ac8c0 2024-09-13T16:28:07.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce36ba7700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3006cd80 con 0x7fce30146230 2024-09-13T16:28:07.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce36ba7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3006c170 con 0x7fce300b2a30 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce35ba5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fce30146230 0x7fce3006bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce35ba5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fce30146230 0x7fce3006bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33318/0 (socket says 172.21.15.119:33318) 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce35ba5700 1 -- 172.21.15.119:0/3610311581 learned_addr learned my addr 172.21.15.119:0/3610311581 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce363a6700 1 --1- 172.21.15.119:0/3610311581 >> v1:172.21.15.145:6789/0 conn(0x7fce300ac8c0 0x7fce3006d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:48946/0 (socket says 172.21.15.119:48946) 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.926+0000 7fce353a4700 1 --2- 172.21.15.119:0/3610311581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce3015cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce35ba5700 1 -- 172.21.15.119:0/3610311581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 msgr2=0x7fce3015cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce35ba5700 1 --2- 172.21.15.119:0/3610311581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce3015cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce34ba3700 1 -- 172.21.15.119:0/3610311581 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380067582 0 0) 0x7fce28001110 con 0x7fce300ac8c0 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce35ba5700 1 -- 172.21.15.119:0/3610311581 >> v1:172.21.15.145:6789/0 conn(0x7fce300ac8c0 legacy=0x7fce3006d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce35ba5700 1 -- 172.21.15.119:0/3610311581 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3006c450 con 0x7fce30146230 2024-09-13T16:28:07.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce35ba5700 1 --2- 172.21.15.119:0/3610311581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce30146230 0x7fce3006bb90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fce20009910 tx=0x7fce20009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b316bd97e79a5ad4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce34ba3700 1 -- 172.21.15.119:0/3610311581 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20011830 con 0x7fce30146230 2024-09-13T16:28:07.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce34ba3700 1 -- 172.21.15.119:0/3610311581 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce20011990 con 0x7fce30146230 2024-09-13T16:28:07.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce34ba3700 1 -- 172.21.15.119:0/3610311581 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce20011d40 con 0x7fce30146230 2024-09-13T16:28:07.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 -- 172.21.15.119:0/3610311581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce30146230 msgr2=0x7fce3006bb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 --2- 172.21.15.119:0/3610311581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce30146230 0x7fce3006bb90 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fce20009910 tx=0x7fce20009c20 comp rx=0 tx=0).stop 2024-09-13T16:28:07.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 -- 172.21.15.119:0/3610311581 shutdown_connections 2024-09-13T16:28:07.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 --2- 172.21.15.119:0/3610311581 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce3015cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 --2- 172.21.15.119:0/3610311581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce30146230 0x7fce3006bb90 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 -- 172.21.15.119:0/3610311581 >> 172.21.15.119:0/3610311581 conn(0x7fce3006a350 msgr2=0x7fce3006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:07.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 -- 172.21.15.119:0/3610311581 shutdown_connections 2024-09-13T16:28:07.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 -- 172.21.15.119:0/3610311581 wait complete. 2024-09-13T16:28:07.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 Processor -- start 2024-09-13T16:28:07.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.930+0000 7fce36ba7700 1 -- start start 2024-09-13T16:28:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce300ac8c0 0x7fce300c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce301f2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce30146230 0x7fce301f2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce30145290 con 0x7fce300ac8c0 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300ca120 con 0x7fce30146230 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce300ca400 con 0x7fce300b2a30 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce300ac8c0 0x7fce300c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce300ac8c0 0x7fce300c9740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33326/0 (socket says 172.21.15.119:33326) 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 -- 172.21.15.119:0/31534679 learned_addr learned my addr 172.21.15.119:0/31534679 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce363a6700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce30146230 0x7fce301f2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce353a4700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce301f2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 -- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 msgr2=0x7fce301f2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce301f2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 -- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce30146230 msgr2=0x7fce301f2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce30146230 0x7fce301f2940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 -- 172.21.15.119:0/31534679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce20009570 con 0x7fce300ac8c0 2024-09-13T16:28:07.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce363a6700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce30146230 0x7fce301f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:07.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce35ba5700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce300ac8c0 0x7fce300c9740 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fce20014040 tx=0x7fce20004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce267fc700 1 -- 172.21.15.119:0/31534679 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2000ba70 con 0x7fce300ac8c0 2024-09-13T16:28:07.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce267fc700 1 -- 172.21.15.119:0/31534679 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2001e070 con 0x7fce300ac8c0 2024-09-13T16:28:07.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce300ca6e0 con 0x7fce300ac8c0 2024-09-13T16:28:07.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce267fc700 1 -- 172.21.15.119:0/31534679 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2000be50 con 0x7fce300ac8c0 2024-09-13T16:28:07.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce300caca0 con 0x7fce300ac8c0 2024-09-13T16:28:07.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.934+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3015cec0 con 0x7fce300ac8c0 2024-09-13T16:28:07.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.938+0000 7fce267fc700 1 -- 172.21.15.119:0/31534679 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fce2002d450 con 0x7fce300ac8c0 2024-09-13T16:28:07.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.938+0000 7fce267fc700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce14066730 0x7fce14068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:07.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.938+0000 7fce267fc700 1 -- 172.21.15.119:0/31534679 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6087+0+0 (secure 0 0 0) 0x7fce200852f0 con 0x7fce300ac8c0 2024-09-13T16:28:07.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.938+0000 7fce353a4700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce14066730 0x7fce14068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:07.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.938+0000 7fce353a4700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce14066730 0x7fce14068bf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0074a0 tx=0x7fce2c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:07.942+0000 7fce267fc700 1 -- 172.21.15.119:0/31534679 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fce20052570 con 0x7fce300ac8c0 2024-09-13T16:28:08.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.122+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce300cae90 con 0x7fce300ac8c0 2024-09-13T16:28:08.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.122+0000 7fce267fc700 1 -- 172.21.15.119:0/31534679 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16874 (secure 0 0 0) 0x7fce20055b90 con 0x7fce300ac8c0 2024-09-13T16:28:08.126 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:08.127 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":56,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:28:07.638635+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:27:21.660330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-09-13T16:27:22.341239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.17","mappings":[{"from":0,"to":6}]},{"pgid":"8.1f","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce14066730 msgr2=0x7fce14068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce14066730 0x7fce14068bf0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fce2c0074a0 tx=0x7fce2c00b040 comp rx=0 tx=0).stop 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce300ac8c0 msgr2=0x7fce300c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce300ac8c0 0x7fce300c9740 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fce20014040 tx=0x7fce20004770 comp rx=0 tx=0).stop 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 shutdown_connections 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce14066730 0x7fce14068bf0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce30146230 0x7fce301f2940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce300b2a30 0x7fce301f2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 --2- 172.21.15.119:0/31534679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce300ac8c0 0x7fce300c9740 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 >> 172.21.15.119:0/31534679 conn(0x7fce3006a350 msgr2=0x7fce300ae360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:08.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 shutdown_connections 2024-09-13T16:28:08.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.126+0000 7fce36ba7700 1 -- 172.21.15.119:0/31534679 wait complete. 2024-09-13T16:28:08.142 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:28:08.142 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:08.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32b5729700 1 Processor -- start 2024-09-13T16:28:08.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32b5729700 1 -- start start 2024-09-13T16:28:08.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32b5729700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f32b006dc40 0x7f32b00b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32b5729700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00b2280 con 0x7f32b01594f0 2024-09-13T16:28:08.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32b5729700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b00cc890 con 0x7f32b00b3630 2024-09-13T16:28:08.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32b5729700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b00b1970 con 0x7f32b006dc40 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32affff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f32b01594f0 0x7f32b006d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34402/0 (socket says 172.21.15.119:34402) 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32affff700 1 -- 172.21.15.119:0/186182516 learned_addr learned my addr 172.21.15.119:0/186182516 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32af7fe700 1 --2- 172.21.15.119:0/186182516 >> v2:172.21.15.145:3300/0 conn(0x7f32b006dc40 0x7f32b00b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32aeffd700 1 -- 172.21.15.119:0/186182516 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762788235 0 0) 0x7f32b00b2280 con 0x7f32b01594f0 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32aeffd700 1 -- 172.21.15.119:0/186182516 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32900035d0 con 0x7f32b01594f0 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32af7fe700 1 -- 172.21.15.119:0/186182516 >> v1:172.21.15.145:6789/0 conn(0x7f32b00b3630 legacy=0x7f32b006fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32af7fe700 1 -- 172.21.15.119:0/186182516 >> v1:172.21.15.119:6789/0 conn(0x7f32b01594f0 legacy=0x7f32b006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32af7fe700 1 -- 172.21.15.119:0/186182516 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b00b1c30 con 0x7f32b006dc40 2024-09-13T16:28:08.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32af7fe700 1 --2- 172.21.15.119:0/186182516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b006dc40 0x7f32b00b30f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3298009d90 tx=0x7f329800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f7aa2647a8d128d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32aeffd700 1 -- 172.21.15.119:0/186182516 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f329800e070 con 0x7f32b006dc40 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32aeffd700 1 -- 172.21.15.119:0/186182516 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f329800bb20 con 0x7f32b006dc40 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.254+0000 7f32aeffd700 1 -- 172.21.15.119:0/186182516 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3298012430 con 0x7f32b006dc40 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- 172.21.15.119:0/186182516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b006dc40 msgr2=0x7f32b00b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 --2- 172.21.15.119:0/186182516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b006dc40 0x7f32b00b30f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f3298009d90 tx=0x7f329800b330 comp rx=0 tx=0).stop 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- 172.21.15.119:0/186182516 shutdown_connections 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 --2- 172.21.15.119:0/186182516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b006dc40 0x7f32b00b30f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- 172.21.15.119:0/186182516 >> 172.21.15.119:0/186182516 conn(0x7f32b006a350 msgr2=0x7f32b006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- 172.21.15.119:0/186182516 shutdown_connections 2024-09-13T16:28:08.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- 172.21.15.119:0/186182516 wait complete. 2024-09-13T16:28:08.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 Processor -- start 2024-09-13T16:28:08.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- start start 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32b006dc40 0x7f32b0148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b00b3630 0x7f32b00ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32b01594f0 0x7f32b00ad930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b00ab350 con 0x7f32b006dc40 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0148790 con 0x7f32b00b3630 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b0148c30 con 0x7f32b01594f0 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32b006dc40 0x7f32b0148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32b006dc40 0x7f32b0148250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33336/0 (socket says 172.21.15.119:33336) 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 -- 172.21.15.119:0/3029642925 learned_addr learned my addr 172.21.15.119:0/3029642925 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:08.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32af7fe700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b00b3630 0x7f32b00ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b4f28700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32b01594f0 0x7f32b00ad930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 -- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32b01594f0 msgr2=0x7f32b00ad930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32b01594f0 0x7f32b00ad930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 -- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b00b3630 msgr2=0x7f32b00ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b00b3630 0x7f32b00ad3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 -- 172.21.15.119:0/3029642925 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3298009a70 con 0x7f32b006dc40 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b4f28700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32b01594f0 0x7f32b00ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32affff700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32b006dc40 0x7f32b0148250 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c500 tx=0x7f32a800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32acff9700 1 -- 172.21.15.119:0/3029642925 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8005d90 con 0x7f32b006dc40 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32acff9700 1 -- 172.21.15.119:0/3029642925 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32a80070d0 con 0x7f32b006dc40 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b00a9d80 con 0x7f32b006dc40 2024-09-13T16:28:08.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b0149160 con 0x7f32b006dc40 2024-09-13T16:28:08.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.258+0000 7f32acff9700 1 -- 172.21.15.119:0/3029642925 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32a8007480 con 0x7f32b006dc40 2024-09-13T16:28:08.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.262+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b006ac20 con 0x7f32b006dc40 2024-09-13T16:28:08.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.262+0000 7f32acff9700 1 -- 172.21.15.119:0/3029642925 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f32a80075e0 con 0x7f32b006dc40 2024-09-13T16:28:08.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.262+0000 7f32acff9700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3294066760 0x7f3294068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.266+0000 7f32acff9700 1 -- 172.21.15.119:0/3029642925 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f32a80837a0 con 0x7f32b006dc40 2024-09-13T16:28:08.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.266+0000 7f32af7fe700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3294066760 0x7f3294068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.266+0000 7f32af7fe700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3294066760 0x7f3294068c20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3298014040 tx=0x7f3298011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.270+0000 7f32acff9700 1 -- 172.21.15.119:0/3029642925 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f32a8050990 con 0x7f32b006dc40 2024-09-13T16:28:08.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.474+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32b00b3a70 con 0x7f32b006dc40 2024-09-13T16:28:08.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32acff9700 1 -- 172.21.15.119:0/3029642925 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f32a8053fb0 con 0x7f32b006dc40 2024-09-13T16:28:08.479 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:08.479 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:28:08.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3294066760 msgr2=0x7f3294068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3294066760 0x7f3294068c20 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f3298014040 tx=0x7f3298011000 comp rx=0 tx=0).stop 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32b006dc40 msgr2=0x7f32b0148250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32b006dc40 0x7f32b0148250 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f32a800c500 tx=0x7f32a800c810 comp rx=0 tx=0).stop 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 shutdown_connections 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3294066760 0x7f3294068c20 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f32b01594f0 0x7f32b00ad930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f32b00b3630 0x7f32b00ad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 --2- 172.21.15.119:0/3029642925 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f32b006dc40 0x7f32b0148250 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 >> 172.21.15.119:0/3029642925 conn(0x7f32b006a350 msgr2=0x7f32b006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 shutdown_connections 2024-09-13T16:28:08.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.478+0000 7f32b5729700 1 -- 172.21.15.119:0/3029642925 wait complete. 2024-09-13T16:28:08.483 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:28:08.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.606+0000 7faaa1547700 1 Processor -- start 2024-09-13T16:28:08.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.606+0000 7faaa1547700 1 -- start start 2024-09-13T16:28:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7faa9c06cd70 0x7faa9c070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c0b1fd0 0x7faa9c0708a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa9c06aa40 con 0x7faa9c159690 2024-09-13T16:28:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa9c06d2e0 con 0x7faa9c06cd70 2024-09-13T16:28:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c06d5c0 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa9bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7faa9c159690 0x7faa9c06dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34410/0 (socket says 172.21.15.119:34410) 2024-09-13T16:28:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa9bfff700 1 -- 172.21.15.119:0/861284355 learned_addr learned my addr 172.21.15.119:0/861284355 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa937fe700 1 --2- 172.21.15.119:0/861284355 >> v2:172.21.15.145:3300/0 conn(0x7faa9c06cd70 0x7faa9c070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa0d46700 1 --2- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c0b1fd0 0x7faa9c0708a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa9b7fe700 1 -- 172.21.15.119:0/861284355 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682383553 0 0) 0x7faa9c06aa40 con 0x7faa9c159690 2024-09-13T16:28:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa9b7fe700 1 -- 172.21.15.119:0/861284355 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa880035d0 con 0x7faa9c159690 2024-09-13T16:28:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa937fe700 1 -- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c0b1fd0 msgr2=0x7faa9c0708a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:08.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa937fe700 1 --2- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c0b1fd0 0x7faa9c0708a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa937fe700 1 -- 172.21.15.119:0/861284355 >> v1:172.21.15.119:6789/0 conn(0x7faa9c159690 legacy=0x7faa9c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa937fe700 1 -- 172.21.15.119:0/861284355 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa9c06a3f0 con 0x7faa9c06cd70 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa937fe700 1 --2- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c06cd70 0x7faa9c070360 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faa84009880 tx=0x7faa84009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=63de8cee4074357b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa9b7fe700 1 -- 172.21.15.119:0/861284355 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84011830 con 0x7faa9c06cd70 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa9b7fe700 1 -- 172.21.15.119:0/861284355 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa84011990 con 0x7faa9c06cd70 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faa9b7fe700 1 -- 172.21.15.119:0/861284355 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84011d00 con 0x7faa9c06cd70 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c06cd70 msgr2=0x7faa9c070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 --2- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c06cd70 0x7faa9c070360 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7faa84009880 tx=0x7faa84009b90 comp rx=0 tx=0).stop 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- 172.21.15.119:0/861284355 shutdown_connections 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 --2- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c0b1fd0 0x7faa9c0708a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 --2- 172.21.15.119:0/861284355 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c06cd70 0x7faa9c070360 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- 172.21.15.119:0/861284355 >> 172.21.15.119:0/861284355 conn(0x7faa9c0ae750 msgr2=0x7faa9c0b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:08.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- 172.21.15.119:0/861284355 shutdown_connections 2024-09-13T16:28:08.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.610+0000 7faaa1547700 1 -- 172.21.15.119:0/861284355 wait complete. 2024-09-13T16:28:08.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 Processor -- start 2024-09-13T16:28:08.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 -- start start 2024-09-13T16:28:08.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa9c0b1fd0 0x7faa9c159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c159690 0x7faa9c156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c15b320 0x7faa9c152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c156f70 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c15ae90 con 0x7faa9c159690 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9c153260 con 0x7faa9c15b320 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa9c0b1fd0 0x7faa9c159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa9c0b1fd0 0x7faa9c159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33346/0 (socket says 172.21.15.119:33346) 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 -- 172.21.15.119:0/59451905 learned_addr learned my addr 172.21.15.119:0/59451905 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa937fe700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c159690 0x7faa9c156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa0d46700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c15b320 0x7faa9c152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 -- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c15b320 msgr2=0x7faa9c152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c15b320 0x7faa9c152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 -- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c159690 msgr2=0x7faa9c156a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c159690 0x7faa9c156a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 -- 172.21.15.119:0/59451905 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa840094b0 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa9bfff700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa9c0b1fd0 0x7faa9c159bf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c500 tx=0x7faa8c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa997fa700 1 -- 172.21.15.119:0/59451905 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c005d90 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa997fa700 1 -- 172.21.15.119:0/59451905 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa8c00ce70 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa9c15a280 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faa997fa700 1 -- 172.21.15.119:0/59451905 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8c0072d0 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.614+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa9c15a840 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.618+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa9c06dc50 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.618+0000 7faa997fa700 1 -- 172.21.15.119:0/59451905 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7faa8c007430 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.618+0000 7faa997fa700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa7c066730 0x7faa7c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.618+0000 7faa997fa700 1 -- 172.21.15.119:0/59451905 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6087+0+0 (secure 0 0 0) 0x7faa8c0834a0 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.618+0000 7faa937fe700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa7c066730 0x7faa7c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.622+0000 7faa937fe700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa7c066730 0x7faa7c068bf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7faa84000c00 tx=0x7faa84008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.622+0000 7faa997fa700 1 -- 172.21.15.119:0/59451905 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7faa8c0507d0 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.830+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa9c0ac0e0 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.830+0000 7faa997fa700 1 -- 172.21.15.119:0/59451905 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4746 (secure 0 0 0) 0x7faa8c053df0 con 0x7faa9c0b1fd0 2024-09-13T16:28:08.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:08.833 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:27:23.348672+0000","modified":"2024-09-13T16:27:26.375537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6069},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6069":{"gid":6069,"name":"a","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/4101111376","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":4101111376},{"type":"v1","addr":"172.21.15.119:6835","nonce":4101111376}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2024-09-13T16:28:08.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa7c066730 msgr2=0x7faa7c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa7c066730 0x7faa7c068bf0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7faa84000c00 tx=0x7faa84008030 comp rx=0 tx=0).stop 2024-09-13T16:28:08.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa9c0b1fd0 msgr2=0x7faa9c159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa9c0b1fd0 0x7faa9c159bf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faa8c00c500 tx=0x7faa8c00c810 comp rx=0 tx=0).stop 2024-09-13T16:28:08.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 shutdown_connections 2024-09-13T16:28:08.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa7c066730 0x7faa7c068bf0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa9c15b320 0x7faa9c152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa9c159690 0x7faa9c156a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 --2- 172.21.15.119:0/59451905 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa9c0b1fd0 0x7faa9c159bf0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 >> 172.21.15.119:0/59451905 conn(0x7faa9c06cd70 msgr2=0x7faa9c0b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:08.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 shutdown_connections 2024-09-13T16:28:08.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.834+0000 7faaa1547700 1 -- 172.21.15.119:0/59451905 wait complete. 2024-09-13T16:28:08.839 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 24 2024-09-13T16:28:08.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:08.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3230efa700 1 Processor -- start 2024-09-13T16:28:08.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3230efa700 1 -- start start 2024-09-13T16:28:08.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3230efa700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f322c08c6b0 0x7f322c0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3230efa700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c15d020 con 0x7f322c0cd860 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3230efa700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f322c0d0720 con 0x7f322c15cc40 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3230efa700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f322c0b44c0 con 0x7f322c08c6b0 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f322c0cd860 0x7f322c166a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34416/0 (socket says 172.21.15.119:34416) 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 -- 172.21.15.119:0/499803689 learned_addr learned my addr 172.21.15.119:0/499803689 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3222ffd700 1 --2- 172.21.15.119:0/499803689 >> v2:172.21.15.145:3300/0 conn(0x7f322c08c6b0 0x7f322c0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2441749915 0 0) 0x7f322c15d020 con 0x7f322c0cd860 2024-09-13T16:28:08.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32180035d0 con 0x7f322c0cd860 2024-09-13T16:28:08.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1858522795 0 0) 0x7f32180035d0 con 0x7f322c0cd860 2024-09-13T16:28:08.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 >> v2:172.21.15.145:3300/0 conn(0x7f322c08c6b0 msgr2=0x7f322c0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:08.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 --2- 172.21.15.119:0/499803689 >> v2:172.21.15.145:3300/0 conn(0x7f322c08c6b0 0x7f322c0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 >> v1:172.21.15.145:6789/0 conn(0x7f322c15cc40 legacy=0x7f322c08bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0d09c0 con 0x7f322c0cd860 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f32240019f0 con 0x7f322c0cd860 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 >> v1:172.21.15.119:6789/0 conn(0x7f322c0cd860 legacy=0x7f322c166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3218003a50 0x7f3218003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218004870 0x7f3218004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3218005230 0x7f321800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0b44c0 con 0x7f3218005230 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f321800c100 con 0x7f3218003a50 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f321800c2c0 con 0x7f3218004870 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3218005230 0x7f321800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322b7fe700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3218003a50 0x7f3218003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f3222ffd700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218004870 0x7f3218004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 -- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218004870 msgr2=0x7f3218004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218004870 0x7f3218004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 -- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3218003a50 msgr2=0x7f3218003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3218003a50 0x7f3218003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 -- 172.21.15.119:0/499803689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32180035d0 con 0x7f3218005230 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 -- 172.21.15.119:0/499803689 reap_dead start 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322bfff700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3218005230 0x7f321800ba30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f322400a800 tx=0x7f32180058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7d90e59b71a0462 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f322400a2f0 con 0x7f3218005230 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32240072d0 con 0x7f3218005230 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.970+0000 7f322affd700 1 -- 172.21.15.119:0/499803689 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3224007680 con 0x7f3218005230 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.974+0000 7f3230efa700 1 -- 172.21.15.119:0/499803689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3218005230 msgr2=0x7f321800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.974+0000 7f3230efa700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3218005230 0x7f321800ba30 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f322400a800 tx=0x7f32180058a0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.974+0000 7f3230efa700 1 -- 172.21.15.119:0/499803689 shutdown_connections 2024-09-13T16:28:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.974+0000 7f3230efa700 1 --2- 172.21.15.119:0/499803689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3218005230 0x7f321800ba30 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.974+0000 7f3230efa700 1 -- 172.21.15.119:0/499803689 >> 172.21.15.119:0/499803689 conn(0x7f322c0b4640 msgr2=0x7f322c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.974+0000 7f3230efa700 1 -- 172.21.15.119:0/499803689 shutdown_connections 2024-09-13T16:28:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.974+0000 7f3230efa700 1 -- 172.21.15.119:0/499803689 wait complete. 2024-09-13T16:28:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 Processor -- start 2024-09-13T16:28:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 -- start start 2024-09-13T16:28:08.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218005230 0x7f322c15e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f322c14f020 0x7f322c0b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f322c0b71e0 0x7f322c15ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0b6d50 con 0x7f322c14f020 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f322c14f480 con 0x7f322c0b71e0 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f322c0b5ec0 con 0x7f3218005230 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f322c14f020 0x7f322c0b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f322c14f020 0x7f322c0b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33362/0 (socket says 172.21.15.119:33362) 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 -- 172.21.15.119:0/2039895891 learned_addr learned my addr 172.21.15.119:0/2039895891 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f322bfff700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f322c0b71e0 0x7f322c15ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f322b7fe700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218005230 0x7f322c15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 -- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218005230 msgr2=0x7f322c15e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218005230 0x7f322c15e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 -- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f322c0b71e0 msgr2=0x7f322c15ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:08.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f322c0b71e0 0x7f322c15ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 -- 172.21.15.119:0/2039895891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f322c0b6140 con 0x7f322c14f020 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f322bfff700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f322c0b71e0 0x7f322c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f322b7fe700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218005230 0x7f322c15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3222ffd700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f322c14f020 0x7f322c0b5980 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f32140119e0 tx=0x7f3214011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3228ff9700 1 -- 172.21.15.119:0/2039895891 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400f940 con 0x7f322c14f020 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3228ff9700 1 -- 172.21.15.119:0/2039895891 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f321400faa0 con 0x7f322c14f020 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f322c0b6420 con 0x7f322c14f020 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3228ff9700 1 -- 172.21.15.119:0/2039895891 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f321400fe30 con 0x7f322c14f020 2024-09-13T16:28:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.978+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f322c0b69e0 con 0x7f322c14f020 2024-09-13T16:28:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.982+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f322c0b6f00 con 0x7f322c14f020 2024-09-13T16:28:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.982+0000 7f3228ff9700 1 -- 172.21.15.119:0/2039895891 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3214014440 con 0x7f322c14f020 2024-09-13T16:28:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.982+0000 7f3228ff9700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f320c066760 0x7f320c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:08.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.982+0000 7f3228ff9700 1 -- 172.21.15.119:0/2039895891 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f3214084a90 con 0x7f322c14f020 2024-09-13T16:28:08.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.982+0000 7f322b7fe700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f320c066760 0x7f320c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:08.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.982+0000 7f322b7fe700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f320c066760 0x7f320c068c20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f321c007fd0 tx=0x7f321c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:08.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:08.986+0000 7f3228ff9700 1 -- 172.21.15.119:0/2039895891 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f322c0b6f00 con 0x7f322c14f020 2024-09-13T16:28:09.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.162+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f322c0b6f00 con 0x7f322c14f020 2024-09-13T16:28:09.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.162+0000 7f3228ff9700 1 -- 172.21.15.119:0/2039895891 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v56) v1 ==== 74+0+16874 (secure 0 0 0) 0x7f322c0b6f00 con 0x7f322c14f020 2024-09-13T16:28:09.167 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:09.167 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":56,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:28:07.638635+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:27:21.660330+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"53","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-09-13T16:27:22.341239+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"8.17","mappings":[{"from":0,"to":6}]},{"pgid":"8.1f","mappings":[{"from":0,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f320c066760 msgr2=0x7f320c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f320c066760 0x7f320c068c20 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f321c007fd0 tx=0x7f321c009040 comp rx=0 tx=0).stop 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f322c14f020 msgr2=0x7f322c0b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f322c14f020 0x7f322c0b5980 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f32140119e0 tx=0x7f3214011da0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 shutdown_connections 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f320c066760 0x7f320c068c20 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f322c0b71e0 0x7f322c15ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f322c14f020 0x7f322c0b5980 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 --2- 172.21.15.119:0/2039895891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3218005230 0x7f322c15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 >> 172.21.15.119:0/2039895891 conn(0x7f322c0b4640 msgr2=0x7f322c08bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:09.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 shutdown_connections 2024-09-13T16:28:09.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.166+0000 7f3230efa700 1 -- 172.21.15.119:0/2039895891 wait complete. 2024-09-13T16:28:09.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:28:09.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a8cb83700 1 Processor -- start 2024-09-13T16:28:09.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a8cb83700 1 -- start start 2024-09-13T16:28:09.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a8cb83700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0a8815e840 0x7f0a88070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a8cb83700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a88165180 0x7f0a88161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a8cb83700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a88071210 con 0x7f0a8815ec20 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a8cb83700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a880714f0 con 0x7f0a8815e840 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a8cb83700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a880717d0 con 0x7f0a88165180 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a85d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0a8815ec20 0x7f0a88164a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34426/0 (socket says 172.21.15.119:34426) 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.294+0000 7f0a85d9b700 1 -- 172.21.15.119:0/540075757 learned_addr learned my addr 172.21.15.119:0/540075757 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8659c700 1 --2- 172.21.15.119:0/540075757 >> v2:172.21.15.119:3300/0 conn(0x7f0a8815e840 0x7f0a88070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a86d9d700 1 --2- 172.21.15.119:0/540075757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a88165180 0x7f0a88161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8559a700 1 -- 172.21.15.119:0/540075757 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289238496 0 0) 0x7f0a88071210 con 0x7f0a8815ec20 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8559a700 1 -- 172.21.15.119:0/540075757 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a680035d0 con 0x7f0a8815ec20 2024-09-13T16:28:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8659c700 1 -- 172.21.15.119:0/540075757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a88165180 msgr2=0x7f0a88161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8659c700 1 --2- 172.21.15.119:0/540075757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a88165180 0x7f0a88161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8659c700 1 -- 172.21.15.119:0/540075757 >> v1:172.21.15.119:6789/0 conn(0x7f0a8815ec20 legacy=0x7f0a88164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8659c700 1 -- 172.21.15.119:0/540075757 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a88071ab0 con 0x7f0a8815e840 2024-09-13T16:28:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8659c700 1 --2- 172.21.15.119:0/540075757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a88070b80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009850 tx=0x7f0a78009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63e29e84d6a14e80 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8559a700 1 -- 172.21.15.119:0/540075757 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78011830 con 0x7f0a8815e840 2024-09-13T16:28:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8559a700 1 -- 172.21.15.119:0/540075757 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a78011990 con 0x7f0a8815e840 2024-09-13T16:28:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8559a700 1 -- 172.21.15.119:0/540075757 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78011d40 con 0x7f0a8815e840 2024-09-13T16:28:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 -- 172.21.15.119:0/540075757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 msgr2=0x7f0a88070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/540075757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a88070b80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f0a78009850 tx=0x7f0a78009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 -- 172.21.15.119:0/540075757 shutdown_connections 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/540075757 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a88165180 0x7f0a88161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/540075757 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a88070b80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 -- 172.21.15.119:0/540075757 >> 172.21.15.119:0/540075757 conn(0x7f0a8806a350 msgr2=0x7f0a8806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 -- 172.21.15.119:0/540075757 shutdown_connections 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.298+0000 7f0a8cb83700 1 -- 172.21.15.119:0/540075757 wait complete. 2024-09-13T16:28:09.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 Processor -- start 2024-09-13T16:28:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 -- start start 2024-09-13T16:28:09.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a881f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a88165180 0x7f0a8815f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a881f67e0 0x7f0a880a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a88160990 con 0x7f0a8815e840 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a881f7930 con 0x7f0a88165180 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8815f8f0 con 0x7f0a881f67e0 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a881f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a881f73f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33382/0 (socket says 172.21.15.119:33382) 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 -- 172.21.15.119:0/3387147734 learned_addr learned my addr 172.21.15.119:0/3387147734 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 -- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a881f67e0 msgr2=0x7f0a880a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a881f67e0 0x7f0a880a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 -- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a88165180 msgr2=0x7f0a8815f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a85d9b700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a88165180 0x7f0a8815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a88165180 0x7f0a8815f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 -- 172.21.15.119:0/3387147734 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a780094b0 con 0x7f0a8815e840 2024-09-13T16:28:09.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a85d9b700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a88165180 0x7f0a8815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:09.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8659c700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a881f73f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0a780090b0 tx=0x7f0a78004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:09.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a7effd700 1 -- 172.21.15.119:0/3387147734 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a78004400 con 0x7f0a8815e840 2024-09-13T16:28:09.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a7effd700 1 -- 172.21.15.119:0/3387147734 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a7801f070 con 0x7f0a8815e840 2024-09-13T16:28:09.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a7effd700 1 -- 172.21.15.119:0/3387147734 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a780036c0 con 0x7f0a8815e840 2024-09-13T16:28:09.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8815fbd0 con 0x7f0a8815e840 2024-09-13T16:28:09.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.302+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a88160190 con 0x7f0a8815e840 2024-09-13T16:28:09.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.306+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a88067760 con 0x7f0a8815e840 2024-09-13T16:28:09.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.306+0000 7f0a7effd700 1 -- 172.21.15.119:0/3387147734 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0a78021030 con 0x7f0a8815e840 2024-09-13T16:28:09.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.306+0000 7f0a7effd700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a6c066760 0x7f0a6c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.306+0000 7f0a7effd700 1 -- 172.21.15.119:0/3387147734 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f0a78085a50 con 0x7f0a8815e840 2024-09-13T16:28:09.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.306+0000 7f0a85d9b700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a6c066760 0x7f0a6c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.306+0000 7f0a85d9b700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a6c066760 0x7f0a6c068c20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a881f9c50 tx=0x7f0a70009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:09.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.310+0000 7f0a7effd700 1 -- 172.21.15.119:0/3387147734 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0a78052cb0 con 0x7f0a8815e840 2024-09-13T16:28:09.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.518+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0a88071210 con 0x7f0a8815e840 2024-09-13T16:28:09.522 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:28:09.518+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:28:09.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.530+0000 7f0a7effd700 1 -- 172.21.15.119:0/3387147734 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0a78015090 con 0x7f0a8815e840 2024-09-13T16:28:09.536 INFO:tasks.ceph.mds.a.smithi119.stderr: -4554> 2024-09-13T16:27:29.311+0000 7f68dc26b700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a6c066760 msgr2=0x7f0a6c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a6c066760 0x7f0a6c068c20 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f0a881f9c50 tx=0x7f0a70009040 comp rx=0 tx=0).stop 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 msgr2=0x7f0a881f73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a881f73f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f0a780090b0 tx=0x7f0a78004770 comp rx=0 tx=0).stop 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 shutdown_connections 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0a6c066760 0x7f0a6c068c20 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0a881f67e0 0x7f0a880a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0a88165180 0x7f0a8815f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 --2- 172.21.15.119:0/3387147734 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0a8815e840 0x7f0a881f73f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 >> 172.21.15.119:0/3387147734 conn(0x7f0a8806a350 msgr2=0x7f0a8806bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:09.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 shutdown_connections 2024-09-13T16:28:09.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.534+0000 7f0a8cb83700 1 -- 172.21.15.119:0/3387147734 wait complete. 2024-09-13T16:28:09.539 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:28:09.539 INFO:tasks.ceph.mds.a.smithi119.stderr: -3034> 2024-09-13T16:27:32.699+0000 7f68d8a64700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /parent_a/alpha auth v2 ap=1 DIRTYPARENT s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6243=0-4194304@1} caps={6243=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6243 | request=0 caps=1 dirtyparent=1 dirty=0 authpin=1 scrubqueue=0 0x55a8de063700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(8)0x10000000001:[<0x10000000000/alpha v2>,<0x1/parent_a v4>]//[]","error_str":"failed to decode on-disk backtrace (40 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:28:09.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:28:09.571 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:28:09.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f9554c3f700 1 Processor -- start 2024-09-13T16:28:09.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f9554c3f700 1 -- start start 2024-09-13T16:28:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f9554c3f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f955006c160 0x7f955006e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f9554c3f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95500cf590 con 0x7f955015cf30 2024-09-13T16:28:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f9554c3f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f955015d310 con 0x7f955006ec20 2024-09-13T16:28:09.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f9554c3f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95500c6660 con 0x7f955006c160 2024-09-13T16:28:09.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f954effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f955006c160 0x7f955006e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f954effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f955006c160 0x7f955006e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33412/0 (socket says 172.21.15.119:33412) 2024-09-13T16:28:09.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f954effd700 1 -- 172.21.15.119:0/2627731097 learned_addr learned my addr 172.21.15.119:0/2627731097 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:09.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f954effd700 1 -- 172.21.15.119:0/2627731097 >> v1:172.21.15.145:6789/0 conn(0x7f955006ec20 legacy=0x7f955006f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f954effd700 1 -- 172.21.15.119:0/2627731097 >> v1:172.21.15.119:6789/0 conn(0x7f955015cf30 legacy=0x7f955006ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f954effd700 1 -- 172.21.15.119:0/2627731097 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95500c6940 con 0x7f955006c160 2024-09-13T16:28:09.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.658+0000 7f954effd700 1 --2- 172.21.15.119:0/2627731097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006c160 0x7f955006e6e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9540009910 tx=0x7f9540009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c39e509292ad75 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f954e7fc700 1 -- 172.21.15.119:0/2627731097 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954000ba60 con 0x7f955006c160 2024-09-13T16:28:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f954e7fc700 1 -- 172.21.15.119:0/2627731097 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954000bbc0 con 0x7f955006c160 2024-09-13T16:28:09.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f954e7fc700 1 -- 172.21.15.119:0/2627731097 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540009e60 con 0x7f955006c160 2024-09-13T16:28:09.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f9554c3f700 1 -- 172.21.15.119:0/2627731097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006c160 msgr2=0x7f955006e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f9554c3f700 1 --2- 172.21.15.119:0/2627731097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006c160 0x7f955006e6e0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f9540009910 tx=0x7f9540009c20 comp rx=0 tx=0).stop 2024-09-13T16:28:09.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f9554c3f700 1 -- 172.21.15.119:0/2627731097 shutdown_connections 2024-09-13T16:28:09.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f9554c3f700 1 --2- 172.21.15.119:0/2627731097 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006c160 0x7f955006e6e0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f9554c3f700 1 -- 172.21.15.119:0/2627731097 >> 172.21.15.119:0/2627731097 conn(0x7f95500b2600 msgr2=0x7f95500b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:09.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f9554c3f700 1 -- 172.21.15.119:0/2627731097 shutdown_connections 2024-09-13T16:28:09.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.662+0000 7f9554c3f700 1 -- 172.21.15.119:0/2627731097 wait complete. 2024-09-13T16:28:09.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 Processor -- start 2024-09-13T16:28:09.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 -- start start 2024-09-13T16:28:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f955006c160 0x7f9550153a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006ec20 0x7f955015bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f955015cf30 0x7f955015c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95501f6a10 con 0x7f955006ec20 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95500b6830 con 0x7f955015cf30 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95501f6ea0 con 0x7f955006c160 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006ec20 0x7f955015bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006ec20 0x7f955015bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33428/0 (socket says 172.21.15.119:33428) 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 -- 172.21.15.119:0/3986831911 learned_addr learned my addr 172.21.15.119:0/3986831911 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954ffff700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f955015cf30 0x7f955015c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954f7fe700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f955006c160 0x7f9550153a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 -- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f955006c160 msgr2=0x7f9550153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f955006c160 0x7f9550153a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 -- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f955015cf30 msgr2=0x7f955015c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f955015cf30 0x7f955015c110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 -- 172.21.15.119:0/3986831911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9540009570 con 0x7f955006ec20 2024-09-13T16:28:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f954effd700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006ec20 0x7f955015bbd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9540014040 tx=0x7f9540004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:09.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9547fff700 1 -- 172.21.15.119:0/3986831911 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95400036c0 con 0x7f955006ec20 2024-09-13T16:28:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9547fff700 1 -- 172.21.15.119:0/3986831911 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954001e070 con 0x7f955006ec20 2024-09-13T16:28:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95501f60e0 con 0x7f955006ec20 2024-09-13T16:28:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9547fff700 1 -- 172.21.15.119:0/3986831911 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9540003a60 con 0x7f955006ec20 2024-09-13T16:28:09.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.666+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95501f66a0 con 0x7f955006ec20 2024-09-13T16:28:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.670+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95501f6830 con 0x7f955006ec20 2024-09-13T16:28:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.670+0000 7f9547fff700 1 -- 172.21.15.119:0/3986831911 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9540003bc0 con 0x7f955006ec20 2024-09-13T16:28:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.670+0000 7f9547fff700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9530066730 0x7f9530068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.670+0000 7f9547fff700 1 -- 172.21.15.119:0/3986831911 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f9540020070 con 0x7f955006ec20 2024-09-13T16:28:09.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.670+0000 7f954f7fe700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9530066730 0x7f9530068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:09.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.670+0000 7f954f7fe700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9530066730 0x7f9530068bf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9538000f60 tx=0x7f9538009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:09.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.678+0000 7f9547fff700 1 -- 172.21.15.119:0/3986831911 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9540052a30 con 0x7f955006ec20 2024-09-13T16:28:09.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:09.886+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f95500c7700 con 0x7f955006ec20 2024-09-13T16:28:10.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.534+0000 7f9547fff700 1 -- 172.21.15.119:0/3986831911 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9540056050 con 0x7f955006ec20 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9530066730 msgr2=0x7f9530068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9530066730 0x7f9530068bf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f9538000f60 tx=0x7f9538009040 comp rx=0 tx=0).stop 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006ec20 msgr2=0x7f955015bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006ec20 0x7f955015bbd0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f9540014040 tx=0x7f9540004770 comp rx=0 tx=0).stop 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 shutdown_connections 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9530066730 0x7f9530068bf0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f955015cf30 0x7f955015c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f955006ec20 0x7f955015bbd0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:10.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 --2- 172.21.15.119:0/3986831911 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f955006c160 0x7f9550153a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:10.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 >> 172.21.15.119:0/3986831911 conn(0x7f95500b2600 msgr2=0x7f9550066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:10.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 shutdown_connections 2024-09-13T16:28:10.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.542+0000 7f9554c3f700 1 -- 172.21.15.119:0/3986831911 wait complete. 2024-09-13T16:28:10.561 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:28:10.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6be40700 1 Processor -- start 2024-09-13T16:28:10.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6be40700 1 -- start start 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6be40700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fca640b7020 0x7fca64167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6be40700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640d1280 con 0x7fca640d3790 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6be40700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca640d1560 con 0x7fca6415cac0 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6be40700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca640d1840 con 0x7fca640b7020 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6a3dd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fca640b7020 0x7fca64167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca6a3dd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fca640b7020 0x7fca64167910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33440/0 (socket says 172.21.15.119:33440) 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.686+0000 7fca693db700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fca640d3790 0x7fca640cdeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34454/0 (socket says 172.21.15.119:34454) 2024-09-13T16:28:10.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6a3dd700 1 -- 172.21.15.119:0/3291294154 learned_addr learned my addr 172.21.15.119:0/3291294154 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:10.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca68bda700 1 -- 172.21.15.119:0/3291294154 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602836996 0 0) 0x7fca640d1280 con 0x7fca640d3790 2024-09-13T16:28:10.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca68bda700 1 -- 172.21.15.119:0/3291294154 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca500035d0 con 0x7fca640d3790 2024-09-13T16:28:10.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6a3dd700 1 -- 172.21.15.119:0/3291294154 >> v1:172.21.15.145:6789/0 conn(0x7fca6415cac0 legacy=0x7fca640cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:10.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6a3dd700 1 -- 172.21.15.119:0/3291294154 >> v1:172.21.15.119:6789/0 conn(0x7fca640d3790 legacy=0x7fca640cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:10.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6a3dd700 1 -- 172.21.15.119:0/3291294154 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca640d1b20 con 0x7fca640b7020 2024-09-13T16:28:10.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6a3dd700 1 --2- 172.21.15.119:0/3291294154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca64167910 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009e50 tx=0x7fca5c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d308dc5e897057b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:10.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca68bda700 1 -- 172.21.15.119:0/3291294154 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c00e070 con 0x7fca640b7020 2024-09-13T16:28:10.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca68bda700 1 -- 172.21.15.119:0/3291294154 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca5c00b910 con 0x7fca640b7020 2024-09-13T16:28:10.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca68bda700 1 -- 172.21.15.119:0/3291294154 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca5c00bcc0 con 0x7fca640b7020 2024-09-13T16:28:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 -- 172.21.15.119:0/3291294154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 msgr2=0x7fca64167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 --2- 172.21.15.119:0/3291294154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca64167910 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fca5c009e50 tx=0x7fca5c00b330 comp rx=0 tx=0).stop 2024-09-13T16:28:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 -- 172.21.15.119:0/3291294154 shutdown_connections 2024-09-13T16:28:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 --2- 172.21.15.119:0/3291294154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca64167910 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 -- 172.21.15.119:0/3291294154 >> 172.21.15.119:0/3291294154 conn(0x7fca640d2c00 msgr2=0x7fca640d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 -- 172.21.15.119:0/3291294154 shutdown_connections 2024-09-13T16:28:10.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 -- 172.21.15.119:0/3291294154 wait complete. 2024-09-13T16:28:10.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 Processor -- start 2024-09-13T16:28:10.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.690+0000 7fca6be40700 1 -- start start 2024-09-13T16:28:10.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca6415f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:10.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca640d3790 0x7fca641f5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca6415cac0 0x7fca641f62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6415eb10 con 0x7fca640b7020 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6415edf0 con 0x7fca6415cac0 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca640ca440 con 0x7fca640d3790 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca6415f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca6415f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33442/0 (socket says 172.21.15.119:33442) 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 -- 172.21.15.119:0/4178775725 learned_addr learned my addr 172.21.15.119:0/4178775725 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca693db700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca640d3790 0x7fca641f5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6a3dd700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca6415cac0 0x7fca641f62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 -- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca640d3790 msgr2=0x7fca641f5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca640d3790 0x7fca641f5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:10.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 -- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca6415cac0 msgr2=0x7fca641f62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca6415cac0 0x7fca641f62f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 -- 172.21.15.119:0/4178775725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca5c009b30 con 0x7fca640b7020 2024-09-13T16:28:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca69bdc700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca6415f280 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fca5400c500 tx=0x7fca5400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:10.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca5a7fc700 1 -- 172.21.15.119:0/4178775725 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca540052d0 con 0x7fca640b7020 2024-09-13T16:28:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca5a7fc700 1 -- 172.21.15.119:0/4178775725 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca54005430 con 0x7fca640b7020 2024-09-13T16:28:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca640ca720 con 0x7fca640b7020 2024-09-13T16:28:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca5a7fc700 1 -- 172.21.15.119:0/4178775725 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca540055a0 con 0x7fca640b7020 2024-09-13T16:28:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.694+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6415b620 con 0x7fca640b7020 2024-09-13T16:28:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.698+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca6415cea0 con 0x7fca640b7020 2024-09-13T16:28:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.698+0000 7fca5a7fc700 1 -- 172.21.15.119:0/4178775725 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fca540038e0 con 0x7fca640b7020 2024-09-13T16:28:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.698+0000 7fca5a7fc700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca48066760 0x7fca48068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:10.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.698+0000 7fca5a7fc700 1 -- 172.21.15.119:0/4178775725 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6173+0+0 (secure 0 0 0) 0x7fca54083620 con 0x7fca640b7020 2024-09-13T16:28:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.698+0000 7fca693db700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca48066760 0x7fca48068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:10.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.698+0000 7fca693db700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca48066760 0x7fca48068c20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fca640ce0a0 tx=0x7fca60007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:10.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.702+0000 7fca5a7fc700 1 -- 172.21.15.119:0/4178775725 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fca540507c0 con 0x7fca640b7020 2024-09-13T16:28:10.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:10.878+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fca6406be20 con 0x7fca640b7020 2024-09-13T16:28:11.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.678+0000 7fca5a7fc700 1 -- 172.21.15.119:0/4178775725 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v58) v1 ==== 181+0+0 (secure 0 0 0) 0x7fca54053de0 con 0x7fca640b7020 2024-09-13T16:28:11.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca48066760 msgr2=0x7fca48068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:11.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca48066760 0x7fca48068c20 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fca640ce0a0 tx=0x7fca60007de0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 msgr2=0x7fca6415f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:11.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca6415f280 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fca5400c500 tx=0x7fca5400c810 comp rx=0 tx=0).stop 2024-09-13T16:28:11.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 shutdown_connections 2024-09-13T16:28:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca48066760 0x7fca48068c20 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca6415cac0 0x7fca641f62f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca640d3790 0x7fca641f5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 --2- 172.21.15.119:0/4178775725 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca640b7020 0x7fca6415f280 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 >> 172.21.15.119:0/4178775725 conn(0x7fca640d2c00 msgr2=0x7fca6406c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 shutdown_connections 2024-09-13T16:28:11.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.682+0000 7fca6be40700 1 -- 172.21.15.119:0/4178775725 wait complete. 2024-09-13T16:28:11.686 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:28:11.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:28:11.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.810+0000 7f109c930700 1 Processor -- start 2024-09-13T16:28:11.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.810+0000 7f109c930700 1 -- start start 2024-09-13T16:28:11.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.810+0000 7f109c930700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f109815ca70 0x7f10980b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:11.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.810+0000 7f109c930700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10980cac00 con 0x7f109815ce50 2024-09-13T16:28:11.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.810+0000 7f109c930700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10980c9ff0 con 0x7f10980afe00 2024-09-13T16:28:11.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.810+0000 7f109c930700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10980ce490 con 0x7f109815ca70 2024-09-13T16:28:11.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.810+0000 7f10977fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f109815ca70 0x7f10980b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:11.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f1096ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f109815ce50 0x7f109806c9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:34464/0 (socket says 172.21.15.119:34464) 2024-09-13T16:28:11.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f1096ffd700 1 -- 172.21.15.119:0/3795876092 learned_addr learned my addr 172.21.15.119:0/3795876092 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:11.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10967fc700 1 -- 172.21.15.119:0/3795876092 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428938662 0 0) 0x7f10980cac00 con 0x7f109815ce50 2024-09-13T16:28:11.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10967fc700 1 -- 172.21.15.119:0/3795876092 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10840035d0 con 0x7f109815ce50 2024-09-13T16:28:11.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10977fe700 1 -- 172.21.15.119:0/3795876092 >> v1:172.21.15.145:6789/0 conn(0x7f10980afe00 legacy=0x7f109806b610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:11.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10977fe700 1 -- 172.21.15.119:0/3795876092 >> v1:172.21.15.119:6789/0 conn(0x7f109815ce50 legacy=0x7f109806c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:11.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10967fc700 1 -- 172.21.15.119:0/3795876092 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077849687 0 0) 0x7f10980c9ff0 con 0x7f10980afe00 2024-09-13T16:28:11.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10977fe700 1 -- 172.21.15.119:0/3795876092 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10980ca2d0 con 0x7f109815ca70 2024-09-13T16:28:11.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10977fe700 1 --2- 172.21.15.119:0/3795876092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109815ca70 0x7f10980b30b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1088009850 tx=0x7f1088009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b7ba5b0178c635 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:11.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10967fc700 1 -- 172.21.15.119:0/3795876092 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088011830 con 0x7f109815ca70 2024-09-13T16:28:11.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10967fc700 1 -- 172.21.15.119:0/3795876092 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1088011990 con 0x7f109815ca70 2024-09-13T16:28:11.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f10967fc700 1 -- 172.21.15.119:0/3795876092 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088011d00 con 0x7f109815ca70 2024-09-13T16:28:11.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f109c930700 1 -- 172.21.15.119:0/3795876092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109815ca70 msgr2=0x7f10980b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:11.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f109c930700 1 --2- 172.21.15.119:0/3795876092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109815ca70 0x7f10980b30b0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1088009850 tx=0x7f1088009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:11.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f109c930700 1 -- 172.21.15.119:0/3795876092 shutdown_connections 2024-09-13T16:28:11.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f109c930700 1 --2- 172.21.15.119:0/3795876092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f109815ca70 0x7f10980b30b0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f109c930700 1 -- 172.21.15.119:0/3795876092 >> 172.21.15.119:0/3795876092 conn(0x7f10980b2590 msgr2=0x7f10980b29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:11.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f109c930700 1 -- 172.21.15.119:0/3795876092 shutdown_connections 2024-09-13T16:28:11.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.814+0000 7f109c930700 1 -- 172.21.15.119:0/3795876092 wait complete. 2024-09-13T16:28:11.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 Processor -- start 2024-09-13T16:28:11.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 -- start start 2024-09-13T16:28:11.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10980afe00 0x7f109815ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:11.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f109815ca70 0x7f1098158b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:11.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1098159080 0x7f10980ae8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:11.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f109814dce0 con 0x7f10980afe00 2024-09-13T16:28:11.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10980b3890 con 0x7f1098159080 2024-09-13T16:28:11.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10980b3ae0 con 0x7f109815ca70 2024-09-13T16:28:11.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10980afe00 0x7f109815ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10980afe00 0x7f109815ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33456/0 (socket says 172.21.15.119:33456) 2024-09-13T16:28:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 -- 172.21.15.119:0/421819572 learned_addr learned my addr 172.21.15.119:0/421819572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f1097fff700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1098159080 0x7f10980ae8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f1096ffd700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f109815ca70 0x7f1098158b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 -- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f109815ca70 msgr2=0x7f1098158b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f109815ca70 0x7f1098158b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 -- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1098159080 msgr2=0x7f10980ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:11.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1098159080 0x7f10980ae8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:11.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 -- 172.21.15.119:0/421819572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880094b0 con 0x7f10980afe00 2024-09-13T16:28:11.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f10977fe700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10980afe00 0x7f109815ac40 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1098070af0 tx=0x7f1088004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:11.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f108ffff700 1 -- 172.21.15.119:0/421819572 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10880036c0 con 0x7f10980afe00 2024-09-13T16:28:11.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f108ffff700 1 -- 172.21.15.119:0/421819572 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108801f070 con 0x7f10980afe00 2024-09-13T16:28:11.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10980b3dc0 con 0x7f10980afe00 2024-09-13T16:28:11.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f108ffff700 1 -- 172.21.15.119:0/421819572 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1088003a40 con 0x7f10980afe00 2024-09-13T16:28:11.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.818+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10980b4380 con 0x7f10980afe00 2024-09-13T16:28:11.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.822+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1098074560 con 0x7f10980afe00 2024-09-13T16:28:11.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.822+0000 7f108ffff700 1 -- 172.21.15.119:0/421819572 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1088003ba0 con 0x7f10980afe00 2024-09-13T16:28:11.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.822+0000 7f108ffff700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1078066730 0x7f1078068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:11.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.822+0000 7f108ffff700 1 -- 172.21.15.119:0/421819572 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5735+0+0 (secure 0 0 0) 0x7f1088021070 con 0x7f10980afe00 2024-09-13T16:28:11.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.822+0000 7f1096ffd700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1078066730 0x7f1078068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:11.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.822+0000 7f1096ffd700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1078066730 0x7f1078068bf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f10980cd1d0 tx=0x7f1080009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:11.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:11.830+0000 7f108ffff700 1 -- 172.21.15.119:0/421819572 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1088052e50 con 0x7f10980afe00 2024-09-13T16:28:12.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.002+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f10980b30b0 con 0x7f10980afe00 2024-09-13T16:28:12.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.678+0000 7f108ffff700 1 -- 172.21.15.119:0/421819572 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v59) v1 ==== 169+0+0 (secure 0 0 0) 0x7f1088015020 con 0x7f10980afe00 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1078066730 msgr2=0x7f1078068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1078066730 0x7f1078068bf0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f10980cd1d0 tx=0x7f1080009040 comp rx=0 tx=0).stop 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10980afe00 msgr2=0x7f109815ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10980afe00 0x7f109815ac40 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1098070af0 tx=0x7f1088004770 comp rx=0 tx=0).stop 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 shutdown_connections 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1078066730 0x7f1078068bf0 secure :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f10980cd1d0 tx=0x7f1080009040 comp rx=0 tx=0).stop 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1098159080 0x7f10980ae8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:12.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f109815ca70 0x7f1098158b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:12.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 --2- 172.21.15.119:0/421819572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10980afe00 0x7f109815ac40 secure :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f1098070af0 tx=0x7f1088004770 comp rx=0 tx=0).stop 2024-09-13T16:28:12.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 >> 172.21.15.119:0/421819572 conn(0x7f10980b2590 msgr2=0x7f10980b2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:12.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.682+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 shutdown_connections 2024-09-13T16:28:12.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.686+0000 7f109c930700 1 -- 172.21.15.119:0/421819572 wait complete. 2024-09-13T16:28:12.688 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:28:12.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T16:28:12.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f8275371700 1 Processor -- start 2024-09-13T16:28:12.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f8275371700 1 -- start start 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f8275371700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f827015dbe0 0x7f82700ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f8275371700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82700cd4d0 con 0x7f82700add60 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f8275371700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f827015e130 con 0x7f82700b3260 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f8275371700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8270071be0 con 0x7f827015dbe0 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f82700add60 0x7f82700cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42352/0 (socket says 172.21.15.119:42352) 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826ffff700 1 -- 172.21.15.119:0/3589307161 learned_addr learned my addr 172.21.15.119:0/3589307161 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826f7fe700 1 --2- 172.21.15.119:0/3589307161 >> v2:172.21.15.145:3300/0 conn(0x7f827015dbe0 0x7f82700ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826effd700 1 -- 172.21.15.119:0/3589307161 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 859858084 0 0) 0x7f82700cd4d0 con 0x7f82700add60 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826effd700 1 -- 172.21.15.119:0/3589307161 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82580035d0 con 0x7f82700add60 2024-09-13T16:28:12.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826effd700 1 -- 172.21.15.119:0/3589307161 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 898051290 0 0) 0x7f827015e130 con 0x7f82700b3260 2024-09-13T16:28:12.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826effd700 1 -- 172.21.15.119:0/3589307161 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82700cd4d0 con 0x7f82700b3260 2024-09-13T16:28:12.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826f7fe700 1 -- 172.21.15.119:0/3589307161 >> v1:172.21.15.145:6789/0 conn(0x7f82700b3260 legacy=0x7f82700b3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:12.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826f7fe700 1 -- 172.21.15.119:0/3589307161 >> v1:172.21.15.119:6789/0 conn(0x7f82700add60 legacy=0x7f82700cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:12.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826f7fe700 1 -- 172.21.15.119:0/3589307161 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82700ccba0 con 0x7f827015dbe0 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826f7fe700 1 --2- 172.21.15.119:0/3589307161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 0x7f82700ce0f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8260009850 tx=0x7f8260009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bea191b5c8471157 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826effd700 1 -- 172.21.15.119:0/3589307161 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260011830 con 0x7f827015dbe0 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826effd700 1 -- 172.21.15.119:0/3589307161 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8260011990 con 0x7f827015dbe0 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.814+0000 7f826effd700 1 -- 172.21.15.119:0/3589307161 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8260011d40 con 0x7f827015dbe0 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- 172.21.15.119:0/3589307161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 msgr2=0x7f82700ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 --2- 172.21.15.119:0/3589307161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 0x7f82700ce0f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f8260009850 tx=0x7f8260009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- 172.21.15.119:0/3589307161 shutdown_connections 2024-09-13T16:28:12.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 --2- 172.21.15.119:0/3589307161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 0x7f82700ce0f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:12.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- 172.21.15.119:0/3589307161 >> 172.21.15.119:0/3589307161 conn(0x7f82700b0980 msgr2=0x7f82700b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:12.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- 172.21.15.119:0/3589307161 shutdown_connections 2024-09-13T16:28:12.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- 172.21.15.119:0/3589307161 wait complete. 2024-09-13T16:28:12.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 Processor -- start 2024-09-13T16:28:12.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- start start 2024-09-13T16:28:12.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82700add60 0x7f8270151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:12.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82700b3260 0x7f82700ae930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:12.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 0x7f82700aee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:12.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270153a30 con 0x7f82700add60 2024-09-13T16:28:12.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82701f30b0 con 0x7f827015dbe0 2024-09-13T16:28:12.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.818+0000 7f8275371700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8270153c50 con 0x7f82700b3260 2024-09-13T16:28:12.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82700add60 0x7f8270151530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:12.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82700add60 0x7f8270151530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50194/0 (socket says 172.21.15.119:50194) 2024-09-13T16:28:12.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 -- 172.21.15.119:0/2718436179 learned_addr learned my addr 172.21.15.119:0/2718436179 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:12.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826f7fe700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82700b3260 0x7f82700ae930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:12.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f8274b70700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 0x7f82700aee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:12.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 -- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82700b3260 msgr2=0x7f82700ae930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:12.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82700b3260 0x7f82700ae930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:12.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 -- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 msgr2=0x7f82700aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:12.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 0x7f82700aee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:12.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 -- 172.21.15.119:0/2718436179 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82600094b0 con 0x7f82700add60 2024-09-13T16:28:12.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826ffff700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82700add60 0x7f8270151530 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f826400c560 tx=0x7f826400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:12.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826cff9700 1 -- 172.21.15.119:0/2718436179 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264005d90 con 0x7f82700add60 2024-09-13T16:28:12.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826cff9700 1 -- 172.21.15.119:0/2718436179 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82640070d0 con 0x7f82700add60 2024-09-13T16:28:12.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826cff9700 1 -- 172.21.15.119:0/2718436179 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264007460 con 0x7f82700add60 2024-09-13T16:28:12.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8270153ef0 con 0x7f82700add60 2024-09-13T16:28:12.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82701544b0 con 0x7f82700add60 2024-09-13T16:28:12.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8270071050 con 0x7f82700add60 2024-09-13T16:28:12.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826cff9700 1 -- 172.21.15.119:0/2718436179 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f82640075c0 con 0x7f82700add60 2024-09-13T16:28:12.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826cff9700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f825c066730 0x7f825c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:12.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826cff9700 1 -- 172.21.15.119:0/2718436179 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f8264083550 con 0x7f82700add60 2024-09-13T16:28:12.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.822+0000 7f826f7fe700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f825c066730 0x7f825c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:12.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.826+0000 7f826f7fe700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f825c066730 0x7f825c068bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f82600090b0 tx=0x7f826001c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:12.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.826+0000 7f826cff9700 1 -- 172.21.15.119:0/2718436179 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8264050a80 con 0x7f82700add60 2024-09-13T16:28:13.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:12.998+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f8270070d70 con 0x7f82700add60 2024-09-13T16:28:13.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.006+0000 7f826cff9700 1 -- 172.21.15.119:0/2718436179 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f82640540a0 con 0x7f82700add60 2024-09-13T16:28:13.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.006+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f825c066730 msgr2=0x7f825c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.006+0000 7f8275371700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f825c066730 0x7f825c068bf0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f82600090b0 tx=0x7f826001c040 comp rx=0 tx=0).stop 2024-09-13T16:28:13.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.006+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82700add60 msgr2=0x7f8270151530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.006+0000 7f8275371700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82700add60 0x7f8270151530 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f826400c560 tx=0x7f826400c920 comp rx=0 tx=0).stop 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.006+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 shutdown_connections 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.010+0000 7f8275371700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f825c066730 0x7f825c068bf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.010+0000 7f8275371700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f827015dbe0 0x7f82700aee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.010+0000 7f8275371700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82700b3260 0x7f82700ae930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.010+0000 7f8275371700 1 --2- 172.21.15.119:0/2718436179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82700add60 0x7f8270151530 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.010+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 >> 172.21.15.119:0/2718436179 conn(0x7f82700b0980 msgr2=0x7f82700b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.010+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 shutdown_connections 2024-09-13T16:28:13.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.010+0000 7f8275371700 1 -- 172.21.15.119:0/2718436179 wait complete. 2024-09-13T16:28:13.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair' 2024-09-13T16:28:13.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32f08a700 1 Processor -- start 2024-09-13T16:28:13.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32f08a700 1 -- start start 2024-09-13T16:28:13.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32f08a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3280b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32f08a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd328071770 con 0x7fd328159360 2024-09-13T16:28:13.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32f08a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32815e0b0 con 0x7fd32806dc40 2024-09-13T16:28:13.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32f08a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3280cc890 con 0x7fd3280adf00 2024-09-13T16:28:13.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32e088700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd328159360 0x7fd32806d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42358/0 (socket says 172.21.15.119:42358) 2024-09-13T16:28:13.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32e088700 1 -- 172.21.15.119:0/2054342639 learned_addr learned my addr 172.21.15.119:0/2054342639 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:13.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32e889700 1 --2- 172.21.15.119:0/2054342639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3280b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.134+0000 7fd32d086700 1 -- 172.21.15.119:0/2054342639 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392584741 0 0) 0x7fd328071770 con 0x7fd328159360 2024-09-13T16:28:13.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32d086700 1 -- 172.21.15.119:0/2054342639 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3140035d0 con 0x7fd328159360 2024-09-13T16:28:13.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32e889700 1 -- 172.21.15.119:0/2054342639 >> v1:172.21.15.145:6789/0 conn(0x7fd32806dc40 legacy=0x7fd3280b17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32d086700 1 -- 172.21.15.119:0/2054342639 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351790439 0 0) 0x7fd32815e0b0 con 0x7fd32806dc40 2024-09-13T16:28:13.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32e889700 1 -- 172.21.15.119:0/2054342639 >> v1:172.21.15.119:6789/0 conn(0x7fd328159360 legacy=0x7fd32806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32e889700 1 -- 172.21.15.119:0/2054342639 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32815d2f0 con 0x7fd3280adf00 2024-09-13T16:28:13.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32e889700 1 --2- 172.21.15.119:0/2054342639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3280b1f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd320009dc0 tx=0x7fd32000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33247591f0b469f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:13.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32d086700 1 -- 172.21.15.119:0/2054342639 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32000bd20 con 0x7fd3280adf00 2024-09-13T16:28:13.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32d086700 1 -- 172.21.15.119:0/2054342639 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd32000be80 con 0x7fd3280adf00 2024-09-13T16:28:13.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32d086700 1 -- 172.21.15.119:0/2054342639 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320011660 con 0x7fd3280adf00 2024-09-13T16:28:13.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 -- 172.21.15.119:0/2054342639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 msgr2=0x7fd3280b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2054342639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3280b1f00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fd320009dc0 tx=0x7fd32000b330 comp rx=0 tx=0).stop 2024-09-13T16:28:13.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 -- 172.21.15.119:0/2054342639 shutdown_connections 2024-09-13T16:28:13.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2054342639 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3280b1f00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 -- 172.21.15.119:0/2054342639 >> 172.21.15.119:0/2054342639 conn(0x7fd32806a350 msgr2=0x7fd32806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:13.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 -- 172.21.15.119:0/2054342639 shutdown_connections 2024-09-13T16:28:13.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 -- 172.21.15.119:0/2054342639 wait complete. 2024-09-13T16:28:13.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 Processor -- start 2024-09-13T16:28:13.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.138+0000 7fd32f08a700 1 -- start start 2024-09-13T16:28:13.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd32806dc40 0x7fd3281ee960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3281ed780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd328159360 0x7fd328156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3280a8650 con 0x7fd328159360 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3281eeea0 con 0x7fd32806dc40 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3281507d0 con 0x7fd3280adf00 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd328159360 0x7fd328156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd328159360 0x7fd328156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50210/0 (socket says 172.21.15.119:50210) 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 -- 172.21.15.119:0/2377173994 learned_addr learned my addr 172.21.15.119:0/2377173994 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:13.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e088700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd32806dc40 0x7fd3281ee960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32d887700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3281ed780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 -- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 msgr2=0x7fd3281ed780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3281ed780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 -- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd32806dc40 msgr2=0x7fd3281ee960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd32806dc40 0x7fd3281ee960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 -- 172.21.15.119:0/2377173994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd320009a70 con 0x7fd328159360 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32e889700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd328159360 0x7fd328156a30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd320015000 tx=0x7fd320004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd31effd700 1 -- 172.21.15.119:0/2377173994 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd32001d040 con 0x7fd328159360 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd31effd700 1 -- 172.21.15.119:0/2377173994 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd320003c40 con 0x7fd328159360 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd328150ab0 con 0x7fd328159360 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd31effd700 1 -- 172.21.15.119:0/2377173994 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd320007000 con 0x7fd328159360 2024-09-13T16:28:13.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd328151070 con 0x7fd328159360 2024-09-13T16:28:13.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.142+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd328159740 con 0x7fd328159360 2024-09-13T16:28:13.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.146+0000 7fd31effd700 1 -- 172.21.15.119:0/2377173994 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd320020070 con 0x7fd328159360 2024-09-13T16:28:13.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.146+0000 7fd31effd700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd30c066730 0x7fd30c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.146+0000 7fd31effd700 1 -- 172.21.15.119:0/2377173994 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd320088790 con 0x7fd328159360 2024-09-13T16:28:13.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.146+0000 7fd32e088700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd30c066730 0x7fd30c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.146+0000 7fd32e088700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd30c066730 0x7fd30c068bf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd318007fd0 tx=0x7fd318009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:13.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.150+0000 7fd31effd700 1 -- 172.21.15.119:0/2377173994 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd320055d50 con 0x7fd328159360 2024-09-13T16:28:13.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.382+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]} v 0) v1 -- 0x7fd328070970 con 0x7fd328159360 2024-09-13T16:28:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.694+0000 7fd31effd700 1 -- 172.21.15.119:0/2377173994 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_backtrace_repair"]}]=0 v165) v1 ==== 149+0+0 (secure 0 0 0) 0x7fd320059370 con 0x7fd328159360 2024-09-13T16:28:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd30c066730 msgr2=0x7fd30c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd30c066730 0x7fd30c068bf0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd318007fd0 tx=0x7fd318009040 comp rx=0 tx=0).stop 2024-09-13T16:28:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd328159360 msgr2=0x7fd328156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd328159360 0x7fd328156a30 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd320015000 tx=0x7fd320004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 shutdown_connections 2024-09-13T16:28:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd30c066730 0x7fd30c068bf0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd328159360 0x7fd328156a30 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3280adf00 0x7fd3281ed780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 --2- 172.21.15.119:0/2377173994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd32806dc40 0x7fd3281ee960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 >> 172.21.15.119:0/2377173994 conn(0x7fd32806a350 msgr2=0x7fd328155d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 shutdown_connections 2024-09-13T16:28:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.698+0000 7fd32f08a700 1 -- 172.21.15.119:0/2377173994 wait complete. 2024-09-13T16:28:13.719 INFO:tasks.cephfs_test_runner:That the MDS can repair an inodes backtrace in the data pool ... ok 2024-09-13T16:28:13.719 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-09-13T16:28:13.719 INFO:tasks.cephfs_test_runner:test_health_status_after_backtrace_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-09-13T16:28:13.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:13.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 Processor -- start 2024-09-13T16:28:13.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- start start 2024-09-13T16:28:13.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b36f0 0x7fd9b00b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9b00b1e80 0x7fd9b0159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9b00b2320 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9b00b2600 con 0x7fd9b00b1e80 2024-09-13T16:28:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b00b28e0 con 0x7fd9b00b36f0 2024-09-13T16:28:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9b00b1e80 0x7fd9b0159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9b00b1e80 0x7fd9b0159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50218/0 (socket says 172.21.15.119:50218) 2024-09-13T16:28:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 -- 172.21.15.119:0/437311821 learned_addr learned my addr 172.21.15.119:0/437311821 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9ae59c700 1 --2- 172.21.15.119:0/437311821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b36f0 0x7fd9b00b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 -- 172.21.15.119:0/437311821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b36f0 msgr2=0x7fd9b00b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 --2- 172.21.15.119:0/437311821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b36f0 0x7fd9b00b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 -- 172.21.15.119:0/437311821 >> v1:172.21.15.145:6789/0 conn(0x7fd9b00b3ad0 legacy=0x7fd9b015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 -- 172.21.15.119:0/437311821 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9b00b2bc0 con 0x7fd9b00b1e80 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9a5d9b700 1 --2- 172.21.15.119:0/437311821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b1e80 0x7fd9b0159340 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd998009910 tx=0x7fd998009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d8ae41a7ce9cf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9add9b700 1 -- 172.21.15.119:0/437311821 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd998011830 con 0x7fd9b00b1e80 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9add9b700 1 -- 172.21.15.119:0/437311821 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd998011990 con 0x7fd9b00b1e80 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9add9b700 1 -- 172.21.15.119:0/437311821 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd998011d40 con 0x7fd9b00b1e80 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- 172.21.15.119:0/437311821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b1e80 msgr2=0x7fd9b0159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/437311821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b1e80 0x7fd9b0159340 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd998009910 tx=0x7fd998009c20 comp rx=0 tx=0).stop 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- 172.21.15.119:0/437311821 shutdown_connections 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/437311821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b1e80 0x7fd9b0159340 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/437311821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b36f0 0x7fd9b00b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- 172.21.15.119:0/437311821 >> 172.21.15.119:0/437311821 conn(0x7fd9b006a350 msgr2=0x7fd9b006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:13.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- 172.21.15.119:0/437311821 shutdown_connections 2024-09-13T16:28:13.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.834+0000 7fd9b4cec700 1 -- 172.21.15.119:0/437311821 wait complete. 2024-09-13T16:28:13.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 Processor -- start 2024-09-13T16:28:13.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 -- start start 2024-09-13T16:28:13.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b1e80 0x7fd9b0139ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b00b36f0 0x7fd9b01f6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b3ad0 0x7fd9b01f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b015b760 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b01f64e0 con 0x7fd9b00b36f0 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9b4cec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9b01f6eb0 con 0x7fd9b00b1e80 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9aed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b3ad0 0x7fd9b01f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9ae59c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b1e80 0x7fd9b0139ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9aed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b3ad0 0x7fd9b01f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50224/0 (socket says 172.21.15.119:50224) 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9a5d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b00b36f0 0x7fd9b01f6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.838+0000 7fd9aed9d700 1 -- 172.21.15.119:0/3631209796 learned_addr learned my addr 172.21.15.119:0/3631209796 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:13.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9aed9d700 1 -- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b1e80 msgr2=0x7fd9b0139ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9aed9d700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b1e80 0x7fd9b0139ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9aed9d700 1 -- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b00b36f0 msgr2=0x7fd9b01f6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9aed9d700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b00b36f0 0x7fd9b01f6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9aed9d700 1 -- 172.21.15.119:0/3631209796 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd998009570 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9a5d9b700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b00b36f0 0x7fd9b01f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9aed9d700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b3ad0 0x7fd9b01f2a10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800cb50 tx=0x7fd9a800ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9a77fe700 1 -- 172.21.15.119:0/3631209796 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a8003c20 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9a77fe700 1 -- 172.21.15.119:0/3631209796 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9a8003d80 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9b01f3430 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9a77fe700 1 -- 172.21.15.119:0/3631209796 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9a800aa20 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9b01f6170 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9b01f5fd0 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9a77fe700 1 -- 172.21.15.119:0/3631209796 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd9a800ab80 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.842+0000 7fd9a77fe700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd990066730 0x7fd990068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:13.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.846+0000 7fd9a77fe700 1 -- 172.21.15.119:0/3631209796 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fd9a8083ea0 con 0x7fd9b00b3ad0 2024-09-13T16:28:13.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.846+0000 7fd9ae59c700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd990066730 0x7fd990068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:13.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.846+0000 7fd9ae59c700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd990066730 0x7fd990068bf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd9b01574c0 tx=0x7fd9a0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:13.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:13.850+0000 7fd9a77fe700 1 -- 172.21.15.119:0/3631209796 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd9b01f5fd0 con 0x7fd9b00b3ad0 2024-09-13T16:28:14.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.022+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9b01f5fd0 con 0x7fd9b00b3ad0 2024-09-13T16:28:14.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9a77fe700 1 -- 172.21.15.119:0/3631209796 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+13253 (secure 0 0 0) 0x7fd9b01f5fd0 con 0x7fd9b00b3ad0 2024-09-13T16:28:14.028 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:14.028 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":59,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:28:12.672633+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/4101111376":"2024-09-14T16:28:09.521069+0000","172.21.15.119:6834/4101111376":"2024-09-14T16:28:09.521069+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:14.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd990066730 msgr2=0x7fd990068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd990066730 0x7fd990068bf0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd9b01574c0 tx=0x7fd9a0009240 comp rx=0 tx=0).stop 2024-09-13T16:28:14.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b3ad0 msgr2=0x7fd9b01f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b3ad0 0x7fd9b01f2a10 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd9a800cb50 tx=0x7fd9a800ce60 comp rx=0 tx=0).stop 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 shutdown_connections 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd990066730 0x7fd990068bf0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9b00b3ad0 0x7fd9b01f2a10 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9b00b36f0 0x7fd9b01f6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 --2- 172.21.15.119:0/3631209796 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9b00b1e80 0x7fd9b0139ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 >> 172.21.15.119:0/3631209796 conn(0x7fd9b006a350 msgr2=0x7fd9b0156560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 shutdown_connections 2024-09-13T16:28:14.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.026+0000 7fd9b4cec700 1 -- 172.21.15.119:0/3631209796 wait complete. 2024-09-13T16:28:14.043 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:28:14.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:28:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 Processor -- start 2024-09-13T16:28:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- start start 2024-09-13T16:28:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8a100ade80 0x7f8a101652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a10071ac0 con 0x7f8a100b24a0 2024-09-13T16:28:14.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a10070eb0 con 0x7f8a1006cdb0 2024-09-13T16:28:14.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a101657f0 con 0x7f8a100ade80 2024-09-13T16:28:14.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0f7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8a100ade80 0x7f8a101652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0effd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8a100b24a0 0x7f8a101612d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42368/0 (socket says 172.21.15.119:42368) 2024-09-13T16:28:14.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0effd700 1 -- 172.21.15.119:0/1021791269 learned_addr learned my addr 172.21.15.119:0/1021791269 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:14.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0f7fe700 1 -- 172.21.15.119:0/1021791269 >> v1:172.21.15.145:6789/0 conn(0x7f8a1006cdb0 legacy=0x7f8a101619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0f7fe700 1 -- 172.21.15.119:0/1021791269 >> v1:172.21.15.119:6789/0 conn(0x7f8a100b24a0 legacy=0x7f8a101612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0e7fc700 1 -- 172.21.15.119:0/1021791269 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161049182 0 0) 0x7f8a10071ac0 con 0x7f8a100b24a0 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0f7fe700 1 -- 172.21.15.119:0/1021791269 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10071190 con 0x7f8a100ade80 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0f7fe700 1 --2- 172.21.15.119:0/1021791269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100ade80 0x7f8a101652b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009850 tx=0x7f8a04009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dca258b8de67f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0e7fc700 1 -- 172.21.15.119:0/1021791269 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a04004230 con 0x7f8a100ade80 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0e7fc700 1 -- 172.21.15.119:0/1021791269 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a04004390 con 0x7f8a100ade80 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a0e7fc700 1 -- 172.21.15.119:0/1021791269 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a040036c0 con 0x7f8a100ade80 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- 172.21.15.119:0/1021791269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100ade80 msgr2=0x7f8a101652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 --2- 172.21.15.119:0/1021791269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100ade80 0x7f8a101652b0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009850 tx=0x7f8a04009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- 172.21.15.119:0/1021791269 shutdown_connections 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 --2- 172.21.15.119:0/1021791269 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100ade80 0x7f8a101652b0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- 172.21.15.119:0/1021791269 >> 172.21.15.119:0/1021791269 conn(0x7f8a100b0980 msgr2=0x7f8a100b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- 172.21.15.119:0/1021791269 shutdown_connections 2024-09-13T16:28:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.158+0000 7f8a14861700 1 -- 172.21.15.119:0/1021791269 wait complete. 2024-09-13T16:28:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 Processor -- start 2024-09-13T16:28:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 -- start start 2024-09-13T16:28:14.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a1006cdb0 0x7f8a10150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a100ade80 0x7f8a1014d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100b24a0 0x7f8a100a9a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a10151030 con 0x7f8a100b24a0 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1014dd20 con 0x7f8a1006cdb0 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a14861700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a1014e380 con 0x7f8a100ade80 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a0ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100b24a0 0x7f8a100a9a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a0f7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a1006cdb0 0x7f8a10150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a0ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100b24a0 0x7f8a100a9a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50254/0 (socket says 172.21.15.119:50254) 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a0ffff700 1 -- 172.21.15.119:0/3162735850 learned_addr learned my addr 172.21.15.119:0/3162735850 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:14.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a0effd700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a100ade80 0x7f8a1014d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.162+0000 7f8a0ffff700 1 -- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a100ade80 msgr2=0x7f8a1014d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a0ffff700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a100ade80 0x7f8a1014d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a0ffff700 1 -- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a1006cdb0 msgr2=0x7f8a10150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a0ffff700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a1006cdb0 0x7f8a10150af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a0ffff700 1 -- 172.21.15.119:0/3162735850 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a040094b0 con 0x7f8a100b24a0 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a0f7fe700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a1006cdb0 0x7f8a10150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a0ffff700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100b24a0 0x7f8a100a9a40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b9e0 tx=0x7f8a0800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f89f7fff700 1 -- 172.21.15.119:0/3162735850 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a08003c20 con 0x7f8a100b24a0 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f89f7fff700 1 -- 172.21.15.119:0/3162735850 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a08003dc0 con 0x7f8a100b24a0 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a1014e660 con 0x7f8a100b24a0 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f89f7fff700 1 -- 172.21.15.119:0/3162735850 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a0800d920 con 0x7f8a100b24a0 2024-09-13T16:28:14.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a1014ec20 con 0x7f8a100b24a0 2024-09-13T16:28:14.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a10072c50 con 0x7f8a100b24a0 2024-09-13T16:28:14.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f89f7fff700 1 -- 172.21.15.119:0/3162735850 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8a0800da80 con 0x7f8a100b24a0 2024-09-13T16:28:14.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f89f7fff700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f89fc066730 0x7f89fc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.166+0000 7f89f7fff700 1 -- 172.21.15.119:0/3162735850 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f8a08011070 con 0x7f8a100b24a0 2024-09-13T16:28:14.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.170+0000 7f8a0f7fe700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f89fc066730 0x7f89fc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.170+0000 7f8a0f7fe700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f89fc066730 0x7f89fc068bf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009850 tx=0x7f8a04008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:14.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.174+0000 7f89f7fff700 1 -- 172.21.15.119:0/3162735850 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8a080513f0 con 0x7f8a100b24a0 2024-09-13T16:28:14.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.338+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8a10070a50 con 0x7f8a100b24a0 2024-09-13T16:28:14.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f89f7fff700 1 -- 172.21.15.119:0/3162735850 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8a08054a10 con 0x7f8a100b24a0 2024-09-13T16:28:14.344 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f89fc066730 msgr2=0x7f89fc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f89fc066730 0x7f89fc068bf0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009850 tx=0x7f8a04008030 comp rx=0 tx=0).stop 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100b24a0 msgr2=0x7f8a100a9a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100b24a0 0x7f8a100a9a40 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f8a0800b9e0 tx=0x7f8a0800bcf0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 shutdown_connections 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f89fc066730 0x7f89fc068bf0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8a100b24a0 0x7f8a100a9a40 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8a100ade80 0x7f8a1014d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 --2- 172.21.15.119:0/3162735850 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8a1006cdb0 0x7f8a10150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 >> 172.21.15.119:0/3162735850 conn(0x7f8a100b0980 msgr2=0x7f8a1006c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 shutdown_connections 2024-09-13T16:28:14.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.342+0000 7f8a14861700 1 -- 172.21.15.119:0/3162735850 wait complete. 2024-09-13T16:28:14.358 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-09-13T16:28:14.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.470+0000 7f049f59e700 1 Processor -- start 2024-09-13T16:28:14.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.470+0000 7f049f59e700 1 -- start start 2024-09-13T16:28:14.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.470+0000 7f049f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04a00adb30 0x7f04a00cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.470+0000 7f049f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a00b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.470+0000 7f049f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04a006cf60 con 0x7f04a006bb90 2024-09-13T16:28:14.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.470+0000 7f049f59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04a006c350 con 0x7f04a00adb30 2024-09-13T16:28:14.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.470+0000 7f049f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a006c630 con 0x7f04a00ce040 2024-09-13T16:28:14.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04a00adb30 0x7f04a00cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f04a00adb30 0x7f04a00cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50266/0 (socket says 172.21.15.119:50266) 2024-09-13T16:28:14.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049ed9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a00b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049dd9b700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f04a006bb90 0x7f04a00cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:59794/0 (socket says 172.21.15.119:59794) 2024-09-13T16:28:14.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 -- 172.21.15.119:0/3957279804 learned_addr learned my addr 172.21.15.119:0/3957279804 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:14.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 -- 172.21.15.119:0/3957279804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 msgr2=0x7f04a00b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:14.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 --2- 172.21.15.119:0/3957279804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a00b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 -- 172.21.15.119:0/3957279804 >> v1:172.21.15.145:6789/0 conn(0x7f04a006bb90 legacy=0x7f04a00cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 -- 172.21.15.119:0/3957279804 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04a006c910 con 0x7f04a00adb30 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049e59c700 1 --2- 172.21.15.119:0/3957279804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a00cc060 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0494009910 tx=0x7f0494009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6014cb2e69fcae8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049d59a700 1 -- 172.21.15.119:0/3957279804 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494011830 con 0x7f04a00adb30 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049d59a700 1 -- 172.21.15.119:0/3957279804 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0494011990 con 0x7f04a00adb30 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049d59a700 1 -- 172.21.15.119:0/3957279804 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0494011d40 con 0x7f04a00adb30 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 -- 172.21.15.119:0/3957279804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 msgr2=0x7f04a00cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 --2- 172.21.15.119:0/3957279804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a00cc060 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f0494009910 tx=0x7f0494009c20 comp rx=0 tx=0).stop 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 -- 172.21.15.119:0/3957279804 shutdown_connections 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 --2- 172.21.15.119:0/3957279804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a00b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 --2- 172.21.15.119:0/3957279804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a00cc060 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 -- 172.21.15.119:0/3957279804 >> 172.21.15.119:0/3957279804 conn(0x7f04a006a350 msgr2=0x7f04a006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 -- 172.21.15.119:0/3957279804 shutdown_connections 2024-09-13T16:28:14.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 -- 172.21.15.119:0/3957279804 wait complete. 2024-09-13T16:28:14.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 Processor -- start 2024-09-13T16:28:14.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.474+0000 7f049f59e700 1 -- start start 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a006bb90 0x7f04a015a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a01f2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a013f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a01f3370 con 0x7f04a00adb30 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04a015c170 con 0x7f04a006bb90 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04a01f2870 con 0x7f04a00ce040 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a01f2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a01f2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50268/0 (socket says 172.21.15.119:50268) 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 -- 172.21.15.119:0/60302811 learned_addr learned my addr 172.21.15.119:0/60302811 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049e59c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a006bb90 0x7f04a015a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049ed9d700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a013f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 -- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 msgr2=0x7f04a013f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a013f6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 -- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a006bb90 msgr2=0x7f04a015a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:14.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a006bb90 0x7f04a015a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:14.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 -- 172.21.15.119:0/60302811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0494009570 con 0x7f04a00adb30 2024-09-13T16:28:14.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049dd9b700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a01f2330 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f049000cc50 tx=0x7f049000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:14.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f048effd700 1 -- 172.21.15.119:0/60302811 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049000aad0 con 0x7f04a00adb30 2024-09-13T16:28:14.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f048effd700 1 -- 172.21.15.119:0/60302811 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f049000ac30 con 0x7f04a00adb30 2024-09-13T16:28:14.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04a013fc30 con 0x7f04a00adb30 2024-09-13T16:28:14.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f048effd700 1 -- 172.21.15.119:0/60302811 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0490007070 con 0x7f04a00adb30 2024-09-13T16:28:14.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.478+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04a01f2db0 con 0x7f04a00adb30 2024-09-13T16:28:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.482+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04a006d3d0 con 0x7f04a00adb30 2024-09-13T16:28:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.482+0000 7f048effd700 1 -- 172.21.15.119:0/60302811 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04900071d0 con 0x7f04a00adb30 2024-09-13T16:28:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.482+0000 7f048effd700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0480066730 0x7f0480068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.482+0000 7f048effd700 1 -- 172.21.15.119:0/60302811 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f0490083ba0 con 0x7f04a00adb30 2024-09-13T16:28:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.482+0000 7f049e59c700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0480066730 0x7f0480068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.486+0000 7f049e59c700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0480066730 0x7f0480068bf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0494000c00 tx=0x7f0494008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:14.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.486+0000 7f048effd700 1 -- 172.21.15.119:0/60302811 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f04a006d3d0 con 0x7f04a00adb30 2024-09-13T16:28:14.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:14.722+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7f04a006d3d0 con 0x7f04a00adb30 2024-09-13T16:28:15.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.702+0000 7f048effd700 1 -- 172.21.15.119:0/60302811 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v167) v1 ==== 172+0+0 (secure 0 0 0) 0x7f04a006d3d0 con 0x7f04a00adb30 2024-09-13T16:28:15.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0480066730 msgr2=0x7f0480068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:15.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0480066730 0x7f0480068bf0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f0494000c00 tx=0x7f0494008030 comp rx=0 tx=0).stop 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 msgr2=0x7f04a01f2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a01f2330 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f049000cc50 tx=0x7f049000cf60 comp rx=0 tx=0).stop 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 shutdown_connections 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0480066730 0x7f0480068bf0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04a00ce040 0x7f04a013f6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04a00adb30 0x7f04a01f2330 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 --2- 172.21.15.119:0/60302811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04a006bb90 0x7f04a015a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 >> 172.21.15.119:0/60302811 conn(0x7f04a006a350 msgr2=0x7f04a0066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 shutdown_connections 2024-09-13T16:28:15.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.706+0000 7f049f59e700 1 -- 172.21.15.119:0/60302811 wait complete. 2024-09-13T16:28:15.727 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T16:28:15.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f540372a700 1 Processor -- start 2024-09-13T16:28:15.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f540372a700 1 -- start start 2024-09-13T16:28:15.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f540372a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc0b1ad0 0x7f53fc071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:15.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f540372a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53fc168a10 0x7f53fc16ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:15.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f540372a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53fc162270 con 0x7f53fc06db40 2024-09-13T16:28:15.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f540372a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53fc06ce40 con 0x7f53fc168a10 2024-09-13T16:28:15.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f540372a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc0702f0 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5400cc5700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f53fc06db40 0x7f53fc0b1eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42378/0 (socket says 172.21.15.119:42378) 2024-09-13T16:28:15.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5400cc5700 1 -- 172.21.15.119:0/53031393 learned_addr learned my addr 172.21.15.119:0/53031393 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:15.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5401cc7700 1 --2- 172.21.15.119:0/53031393 >> v2:172.21.15.145:3300/0 conn(0x7f53fc168a10 0x7f53fc16ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:15.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f54014c6700 1 --2- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc0b1ad0 0x7f53fc071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:15.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f53f3fff700 1 -- 172.21.15.119:0/53031393 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5340642 0 0) 0x7f53fc162270 con 0x7f53fc06db40 2024-09-13T16:28:15.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f53f3fff700 1 -- 172.21.15.119:0/53031393 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53e80035d0 con 0x7f53fc06db40 2024-09-13T16:28:15.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5401cc7700 1 -- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc0b1ad0 msgr2=0x7f53fc071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:15.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5401cc7700 1 --2- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc0b1ad0 0x7f53fc071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5401cc7700 1 -- 172.21.15.119:0/53031393 >> v1:172.21.15.119:6789/0 conn(0x7f53fc06db40 legacy=0x7f53fc0b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:15.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5401cc7700 1 -- 172.21.15.119:0/53031393 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53fc071730 con 0x7f53fc168a10 2024-09-13T16:28:15.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f5401cc7700 1 --2- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 0x7f53fc16ae00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009d90 tx=0x7f53f400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd9ad53649e247f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:15.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f53f3fff700 1 -- 172.21.15.119:0/53031393 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f400e070 con 0x7f53fc168a10 2024-09-13T16:28:15.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f53f3fff700 1 -- 172.21.15.119:0/53031393 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f400bb20 con 0x7f53fc168a10 2024-09-13T16:28:15.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.846+0000 7f53f3fff700 1 -- 172.21.15.119:0/53031393 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f4012430 con 0x7f53fc168a10 2024-09-13T16:28:15.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 msgr2=0x7f53fc16ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:15.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 --2- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 0x7f53fc16ae00 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f53f4009d90 tx=0x7f53f400b330 comp rx=0 tx=0).stop 2024-09-13T16:28:15.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- 172.21.15.119:0/53031393 shutdown_connections 2024-09-13T16:28:15.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 --2- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 0x7f53fc16ae00 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 --2- 172.21.15.119:0/53031393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc0b1ad0 0x7f53fc071c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- 172.21.15.119:0/53031393 >> 172.21.15.119:0/53031393 conn(0x7f53fc0b0560 msgr2=0x7f53fc0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:15.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- 172.21.15.119:0/53031393 shutdown_connections 2024-09-13T16:28:15.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- 172.21.15.119:0/53031393 wait complete. 2024-09-13T16:28:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 Processor -- start 2024-09-13T16:28:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- start start 2024-09-13T16:28:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc06db40 0x7f53fc158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53fc0b1ad0 0x7f53fc15e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 0x7f53fc15ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc166920 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc1605d0 con 0x7f53fc168a10 2024-09-13T16:28:15.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f540372a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53fc158540 con 0x7f53fc06db40 2024-09-13T16:28:15.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f5400cc5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53fc0b1ad0 0x7f53fc15e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:15.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f5400cc5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53fc0b1ad0 0x7f53fc15e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50272/0 (socket says 172.21.15.119:50272) 2024-09-13T16:28:15.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f5400cc5700 1 -- 172.21.15.119:0/2707084607 learned_addr learned my addr 172.21.15.119:0/2707084607 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:15.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f5401cc7700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 0x7f53fc15ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:15.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.850+0000 7f54014c6700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc06db40 0x7f53fc158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:15.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f5400cc5700 1 -- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc06db40 msgr2=0x7f53fc158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:15.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f5400cc5700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc06db40 0x7f53fc158790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f5400cc5700 1 -- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 msgr2=0x7f53fc15ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f5400cc5700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 0x7f53fc15ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f5400cc5700 1 -- 172.21.15.119:0/2707084607 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f4009a70 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f5400cc5700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53fc0b1ad0 0x7f53fc15e6d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f53f800c700 tx=0x7f53f800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f53f1ffb700 1 -- 172.21.15.119:0/2707084607 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8005d90 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f53f1ffb700 1 -- 172.21.15.119:0/2707084607 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53f80070d0 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53fc158260 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f53f1ffb700 1 -- 172.21.15.119:0/2707084607 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f8007480 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53fc1602f0 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53fc073590 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f53f1ffb700 1 -- 172.21.15.119:0/2707084607 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f53f80075e0 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f53f1ffb700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53e0066760 0x7f53e0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:15.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.854+0000 7f53f1ffb700 1 -- 172.21.15.119:0/2707084607 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f53f80833c0 con 0x7f53fc0b1ad0 2024-09-13T16:28:15.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.858+0000 7f54014c6700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53e0066760 0x7f53e0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:15.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.858+0000 7f54014c6700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53e0066760 0x7f53e0068c20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0af230 tx=0x7f53ec009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:15.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:15.866+0000 7f53f1ffb700 1 -- 172.21.15.119:0/2707084607 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f53f8050900 con 0x7f53fc0b1ad0 2024-09-13T16:28:16.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.038+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f53fc0b0e30 con 0x7f53fc0b1ad0 2024-09-13T16:28:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f53f1ffb700 1 -- 172.21.15.119:0/2707084607 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7f53f8053f20 con 0x7f53fc0b1ad0 2024-09-13T16:28:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53e0066760 msgr2=0x7f53e0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53e0066760 0x7f53e0068c20 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f53fc0af230 tx=0x7f53ec009240 comp rx=0 tx=0).stop 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53fc0b1ad0 msgr2=0x7f53fc15e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53fc0b1ad0 0x7f53fc15e6d0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f53f800c700 tx=0x7f53f800ca10 comp rx=0 tx=0).stop 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 shutdown_connections 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f53e0066760 0x7f53e0068c20 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53fc168a10 0x7f53fc15ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53fc0b1ad0 0x7f53fc15e6d0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 --2- 172.21.15.119:0/2707084607 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53fc06db40 0x7f53fc158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 >> 172.21.15.119:0/2707084607 conn(0x7f53fc0b0560 msgr2=0x7f53fc0ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 shutdown_connections 2024-09-13T16:28:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.046+0000 7f540372a700 1 -- 172.21.15.119:0/2707084607 wait complete. 2024-09-13T16:28:16.065 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:28:16.065 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:28:16.078 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:28:16.099 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:28:16.099 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:28:16.099 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:16.106 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:28:16.107 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:28:16.107 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:16.114 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:28:16.114 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:28:16.114 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:16.121 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:28:16.121 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:28:16.122 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:28:16.129 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:28:16.129 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T16:28:16.129 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:16.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f8649def700 1 Processor -- start 2024-09-13T16:28:16.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f8649def700 1 -- start start 2024-09-13T16:28:16.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f8649def700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f86441594f0 0x7f864406d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f8649def700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86440b2280 con 0x7f864406da70 2024-09-13T16:28:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f8649def700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86440cc890 con 0x7f86440b3800 2024-09-13T16:28:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f8649def700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f864406dee0 con 0x7f86441594f0 2024-09-13T16:28:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f863bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f864406da70 0x7f86440b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42392/0 (socket says 172.21.15.119:42392) 2024-09-13T16:28:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.245+0000 7f863bfff700 1 -- 172.21.15.119:0/1274229254 learned_addr learned my addr 172.21.15.119:0/1274229254 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:16.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8648ded700 1 --2- 172.21.15.119:0/1274229254 >> v2:172.21.15.145:3300/0 conn(0x7f86441594f0 0x7f864406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182463813 0 0) 0x7f86440b2280 con 0x7f864406da70 2024-09-13T16:28:16.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f862c0057e0 con 0x7f864406da70 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2489745829 0 0) 0x7f862c0057e0 con 0x7f864406da70 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 >> v2:172.21.15.145:3300/0 conn(0x7f86441594f0 msgr2=0x7f864406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 --2- 172.21.15.119:0/1274229254 >> v2:172.21.15.145:3300/0 conn(0x7f86441594f0 0x7f864406d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 >> v1:172.21.15.145:6789/0 conn(0x7f86440b3800 legacy=0x7f864406fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86440b1970 con 0x7f864406da70 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f8634001670 con 0x7f864406da70 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 >> v1:172.21.15.119:6789/0 conn(0x7f864406da70 legacy=0x7f86440b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c005c30 0x7f862c006010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c006630 0x7f862c006b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c007300 0x7f862c00dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864406dee0 con 0x7f862c007300 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c00e260 con 0x7f862c006630 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c00e420 con 0x7f862c005c30 2024-09-13T16:28:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8648ded700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c005c30 0x7f862c006010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863bfff700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c006630 0x7f862c006b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f86495ee700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c007300 0x7f862c00dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863bfff700 1 -- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c005c30 msgr2=0x7f862c006010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863bfff700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c005c30 0x7f862c006010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863bfff700 1 -- 172.21.15.119:0/1274229254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c007300 msgr2=0x7f862c00dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863bfff700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c007300 0x7f862c00dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f86495ee700 1 -- 172.21.15.119:0/1274229254 reap_dead start 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863bfff700 1 -- 172.21.15.119:0/1274229254 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c0057e0 con 0x7f862c006630 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863bfff700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c006630 0x7f862c006b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f863400c2f0 tx=0x7f862c007ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbde1a16b590c1ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f863400ce30 con 0x7f862c006630 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8634005980 con 0x7f862c006630 2024-09-13T16:28:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f863b7fe700 1 -- 172.21.15.119:0/1274229254 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634005d30 con 0x7f862c006630 2024-09-13T16:28:16.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8649def700 1 -- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c006630 msgr2=0x7f862c006b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8649def700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c006630 0x7f862c006b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f863400c2f0 tx=0x7f862c007ab0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8649def700 1 -- 172.21.15.119:0/1274229254 shutdown_connections 2024-09-13T16:28:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8649def700 1 --2- 172.21.15.119:0/1274229254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c006630 0x7f862c006b00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8649def700 1 -- 172.21.15.119:0/1274229254 >> 172.21.15.119:0/1274229254 conn(0x7f864406a350 msgr2=0x7f864406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8649def700 1 -- 172.21.15.119:0/1274229254 shutdown_connections 2024-09-13T16:28:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.249+0000 7f8649def700 1 -- 172.21.15.119:0/1274229254 wait complete. 2024-09-13T16:28:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 Processor -- start 2024-09-13T16:28:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 -- start start 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c006630 0x7f8644148ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8644159bd0 0x7f864406c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f864406cb70 0x7f864415b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864406cff0 con 0x7f8644159bd0 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864415ad80 con 0x7f864406cb70 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8649def700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f864415a170 con 0x7f862c006630 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8644159bd0 0x7f864406c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8644159bd0 0x7f864406c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50288/0 (socket says 172.21.15.119:50288) 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 -- 172.21.15.119:0/1090484011 learned_addr learned my addr 172.21.15.119:0/1090484011 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8648ded700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c006630 0x7f8644148ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f86495ee700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f864406cb70 0x7f864415b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 -- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c006630 msgr2=0x7f8644148ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c006630 0x7f8644148ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 -- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f864406cb70 msgr2=0x7f864415b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f864406cb70 0x7f864415b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 -- 172.21.15.119:0/1090484011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f864415a450 con 0x7f8644159bd0 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f8648ded700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c006630 0x7f8644148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.253+0000 7f863bfff700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8644159bd0 0x7f864406c630 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f863400c8e0 tx=0x7f8634001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.257+0000 7f86397fa700 1 -- 172.21.15.119:0/1090484011 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634004a90 con 0x7f8644159bd0 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.257+0000 7f86397fa700 1 -- 172.21.15.119:0/1090484011 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8634004bf0 con 0x7f8644159bd0 2024-09-13T16:28:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.257+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864415a730 con 0x7f8644159bd0 2024-09-13T16:28:16.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.257+0000 7f86397fa700 1 -- 172.21.15.119:0/1090484011 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8634006e10 con 0x7f8644159bd0 2024-09-13T16:28:16.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.257+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86441505c0 con 0x7f8644159bd0 2024-09-13T16:28:16.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.257+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864415af10 con 0x7f8644159bd0 2024-09-13T16:28:16.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.257+0000 7f86397fa700 1 -- 172.21.15.119:0/1090484011 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8634005070 con 0x7f8644159bd0 2024-09-13T16:28:16.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.261+0000 7f86397fa700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8630066730 0x7f8630068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.261+0000 7f86397fa700 1 -- 172.21.15.119:0/1090484011 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f86340845c0 con 0x7f8644159bd0 2024-09-13T16:28:16.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.261+0000 7f8648ded700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8630066730 0x7f8630068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.261+0000 7f8648ded700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8630066730 0x7f8630068bf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f86400144c0 tx=0x7f8640009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:16.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.265+0000 7f86397fa700 1 -- 172.21.15.119:0/1090484011 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8634051c20 con 0x7f8644159bd0 2024-09-13T16:28:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.473+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8644071b70 con 0x7f8644159bd0 2024-09-13T16:28:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.473+0000 7f86397fa700 1 -- 172.21.15.119:0/1090484011 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3538 (secure 0 0 0) 0x7f8634024df0 con 0x7f8644159bd0 2024-09-13T16:28:16.478 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:16.478 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5875,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4188418310","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4188418310},{"type":"v1","addr":"172.21.15.119:6833","nonce":4188418310}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5887,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2047836618","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2047836618},{"type":"v1","addr":"172.21.15.145:6835","nonce":2047836618}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":5890,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1677559576","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1677559576},{"type":"v1","addr":"172.21.15.145:6837","nonce":1677559576}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":19},{"gid":6453,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/571001128","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":571001128},{"type":"v1","addr":"172.21.15.119:6835","nonce":571001128}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":26}],"filesystems":[]} 2024-09-13T16:28:16.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8630066730 msgr2=0x7f8630068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8630066730 0x7f8630068bf0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f86400144c0 tx=0x7f8640009240 comp rx=0 tx=0).stop 2024-09-13T16:28:16.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8644159bd0 msgr2=0x7f864406c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8644159bd0 0x7f864406c630 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f863400c8e0 tx=0x7f8634001a00 comp rx=0 tx=0).stop 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 shutdown_connections 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8630066730 0x7f8630068bf0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f864406cb70 0x7f864415b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8644159bd0 0x7f864406c630 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 --2- 172.21.15.119:0/1090484011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c006630 0x7f8644148ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 >> 172.21.15.119:0/1090484011 conn(0x7f864406a350 msgr2=0x7f86440cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 shutdown_connections 2024-09-13T16:28:16.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.477+0000 7f8649def700 1 -- 172.21.15.119:0/1090484011 wait complete. 2024-09-13T16:28:16.482 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 26 2024-09-13T16:28:16.493 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:28:16.493 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T16:28:16.493 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:28:16.493 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:28:16.494 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:28:16.494 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T16:28:16.494 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:28:16.494 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:28:16.494 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:28:16.494 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T16:28:16.494 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:28:16.494 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:28:16.495 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:28:16.495 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T16:28:16.495 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:28:16.495 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:28:16.506 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:28:16.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:28:16.509 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:28:16.509 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:28:16.509 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:28:16.510 DEBUG:teuthology.parallel:result is None 2024-09-13T16:28:16.511 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:28:16.511 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:28:16.512 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:28:16.512 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:28:16.513 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:28:16.513 DEBUG:teuthology.parallel:result is None 2024-09-13T16:28:16.514 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:28:16.514 DEBUG:teuthology.parallel:result is None 2024-09-13T16:28:16.515 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:28:16.515 DEBUG:teuthology.parallel:result is None 2024-09-13T16:28:16.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T16:28:16.580 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:28:16.586 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:28:16.594 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:28:16.599 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:28:16.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.617+0000 7f5f7a0ba700 1 Processor -- start 2024-09-13T16:28:16.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.617+0000 7f5f7a0ba700 1 -- start start 2024-09-13T16:28:16.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f7a0ba700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5f740abf40 0x7f5f7406cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f7a0ba700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f74072640 con 0x7f5f740721f0 2024-09-13T16:28:16.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f7a0ba700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f74071d50 con 0x7f5f74158fd0 2024-09-13T16:28:16.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f7a0ba700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f7406d350 con 0x7f5f740abf40 2024-09-13T16:28:16.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f798b9700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5f740721f0 0x7f5f7406dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42410/0 (socket says 172.21.15.119:42410) 2024-09-13T16:28:16.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f790b8700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5f740abf40 0x7f5f7406cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f798b9700 1 -- 172.21.15.119:0/39083453 learned_addr learned my addr 172.21.15.119:0/39083453 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:16.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f790b8700 1 -- 172.21.15.119:0/39083453 >> v1:172.21.15.145:6789/0 conn(0x7f5f74158fd0 legacy=0x7f5f740ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f790b8700 1 -- 172.21.15.119:0/39083453 >> v1:172.21.15.119:6789/0 conn(0x7f5f740721f0 legacy=0x7f5f7406dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f790b8700 1 -- 172.21.15.119:0/39083453 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f740acec0 con 0x7f5f740abf40 2024-09-13T16:28:16.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.621+0000 7f5f790b8700 1 --2- 172.21.15.119:0/39083453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740abf40 0x7f5f7406cd40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000a970 tx=0x7f5f7000ac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6de390f1372fab2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:16.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.625+0000 7f5f6bfff700 1 -- 172.21.15.119:0/39083453 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000ca60 con 0x7f5f740abf40 2024-09-13T16:28:16.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.625+0000 7f5f6bfff700 1 -- 172.21.15.119:0/39083453 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7000cbc0 con 0x7f5f740abf40 2024-09-13T16:28:16.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.625+0000 7f5f6bfff700 1 -- 172.21.15.119:0/39083453 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70012420 con 0x7f5f740abf40 2024-09-13T16:28:16.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.625+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/39083453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740abf40 msgr2=0x7f5f7406cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.625+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/39083453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740abf40 0x7f5f7406cd40 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000a970 tx=0x7f5f7000ac80 comp rx=0 tx=0).stop 2024-09-13T16:28:16.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/39083453 shutdown_connections 2024-09-13T16:28:16.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/39083453 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f740abf40 0x7f5f7406cd40 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/39083453 >> 172.21.15.119:0/39083453 conn(0x7f5f740ae750 msgr2=0x7f5f740b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:16.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/39083453 shutdown_connections 2024-09-13T16:28:16.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/39083453 wait complete. 2024-09-13T16:28:16.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 Processor -- start 2024-09-13T16:28:16.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 -- start start 2024-09-13T16:28:16.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740abf40 0x7f5f741f17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.629+0000 7f5f7a0ba700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f74158fd0 0x7f5f740b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f7a0ba700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f7415b210 0x7f5f741f3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f7a0ba700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f741f1d00 con 0x7f5f740abf40 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f7a0ba700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7414c520 con 0x7f5f74158fd0 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f7a0ba700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f7414c770 con 0x7f5f7415b210 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740abf40 0x7f5f741f17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740abf40 0x7f5f741f17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50320/0 (socket says 172.21.15.119:50320) 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 -- 172.21.15.119:0/3404510624 learned_addr learned my addr 172.21.15.119:0/3404510624 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f788b7700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f74158fd0 0x7f5f740b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f798b9700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f7415b210 0x7f5f741f3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 -- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f7415b210 msgr2=0x7f5f741f3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f7415b210 0x7f5f741f3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 -- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f74158fd0 msgr2=0x7f5f740b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f74158fd0 0x7f5f740b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:16.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 -- 172.21.15.119:0/3404510624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7000a5d0 con 0x7f5f740abf40 2024-09-13T16:28:16.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f798b9700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f7415b210 0x7f5f741f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:16.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f790b8700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740abf40 0x7f5f741f17c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5f70000c00 tx=0x7f5f70004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:16.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f69ffb700 1 -- 172.21.15.119:0/3404510624 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70005d80 con 0x7f5f740abf40 2024-09-13T16:28:16.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f69ffb700 1 -- 172.21.15.119:0/3404510624 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f7001f070 con 0x7f5f740abf40 2024-09-13T16:28:16.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f7414ca50 con 0x7f5f740abf40 2024-09-13T16:28:16.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f69ffb700 1 -- 172.21.15.119:0/3404510624 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f70004280 con 0x7f5f740abf40 2024-09-13T16:28:16.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f7414d010 con 0x7f5f740abf40 2024-09-13T16:28:16.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f74073fd0 con 0x7f5f740abf40 2024-09-13T16:28:16.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f69ffb700 1 -- 172.21.15.119:0/3404510624 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5f70021030 con 0x7f5f740abf40 2024-09-13T16:28:16.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f69ffb700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f58066760 0x7f5f58068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:16.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.633+0000 7f5f69ffb700 1 -- 172.21.15.119:0/3404510624 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f5f70084ed0 con 0x7f5f740abf40 2024-09-13T16:28:16.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.637+0000 7f5f788b7700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f58066760 0x7f5f58068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:16.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.637+0000 7f5f788b7700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f58066760 0x7f5f58068c20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5f74070fa0 tx=0x7f5f64009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:16.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.641+0000 7f5f69ffb700 1 -- 172.21.15.119:0/3404510624 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5f70052490 con 0x7f5f740abf40 2024-09-13T16:28:16.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:16.821+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f5f740cfd70 con 0x7f5f740abf40 2024-09-13T16:28:17.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.713+0000 7f5f69ffb700 1 -- 172.21.15.119:0/3404510624 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v64) v1 ==== 97+0+0 (secure 0 0 0) 0x7f5f70007df0 con 0x7f5f740abf40 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f58066760 msgr2=0x7f5f58068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f58066760 0x7f5f58068c20 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f5f74070fa0 tx=0x7f5f64009040 comp rx=0 tx=0).stop 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740abf40 msgr2=0x7f5f741f17c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740abf40 0x7f5f741f17c0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f5f70000c00 tx=0x7f5f70004770 comp rx=0 tx=0).stop 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 shutdown_connections 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5f58066760 0x7f5f58068c20 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f7415b210 0x7f5f741f3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f74158fd0 0x7f5f740b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 --2- 172.21.15.119:0/3404510624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f740abf40 0x7f5f741f17c0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 >> 172.21.15.119:0/3404510624 conn(0x7f5f740721f0 msgr2=0x7f5f7406c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:17.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 shutdown_connections 2024-09-13T16:28:17.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.717+0000 7f5f7a0ba700 1 -- 172.21.15.119:0/3404510624 wait complete. 2024-09-13T16:28:17.724 INFO:teuthology.orchestra.run.smithi119.stderr: removed all blocklist entries 2024-09-13T16:28:17.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T16:28:17.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 Processor -- start 2024-09-13T16:28:17.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- start start 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f381006c5f0 0x7f38100cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f381015a010 0x7f381006c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f381006cf10 con 0x7f38100cd200 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381006c470 con 0x7f381015a010 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38100ce540 con 0x7f381006c5f0 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f381015a010 0x7f381006c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f38100cd200 0x7f381006bd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42424/0 (socket says 172.21.15.119:42424) 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380ffff700 1 -- 172.21.15.119:0/2770459835 learned_addr learned my addr 172.21.15.119:0/2770459835 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380f7fe700 1 --2- 172.21.15.119:0/2770459835 >> v2:172.21.15.145:3300/0 conn(0x7f381006c5f0 0x7f38100cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380effd700 1 -- 172.21.15.119:0/2770459835 >> v2:172.21.15.145:3300/0 conn(0x7f381006c5f0 msgr2=0x7f38100cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:17.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380effd700 1 --2- 172.21.15.119:0/2770459835 >> v2:172.21.15.145:3300/0 conn(0x7f381006c5f0 0x7f38100cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380effd700 1 -- 172.21.15.119:0/2770459835 >> v1:172.21.15.119:6789/0 conn(0x7f38100cd200 legacy=0x7f381006bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:17.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380effd700 1 -- 172.21.15.119:0/2770459835 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38100cc720 con 0x7f381015a010 2024-09-13T16:28:17.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380effd700 1 --2- 172.21.15.119:0/2770459835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f381015a010 0x7f381006c9d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f381006dfe0 tx=0x7f3800009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98898fd2fc5ef400 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380e7fc700 1 -- 172.21.15.119:0/2770459835 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800011830 con 0x7f381015a010 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380e7fc700 1 -- 172.21.15.119:0/2770459835 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3800011990 con 0x7f381015a010 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f380e7fc700 1 -- 172.21.15.119:0/2770459835 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800011d40 con 0x7f381015a010 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- 172.21.15.119:0/2770459835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f381015a010 msgr2=0x7f381006c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 --2- 172.21.15.119:0/2770459835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f381015a010 0x7f381006c9d0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f381006dfe0 tx=0x7f3800009b30 comp rx=0 tx=0).stop 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- 172.21.15.119:0/2770459835 shutdown_connections 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 --2- 172.21.15.119:0/2770459835 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f381015a010 0x7f381006c9d0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 --2- 172.21.15.119:0/2770459835 >> v2:172.21.15.145:3300/0 conn(0x7f381006c5f0 0x7f38100cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- 172.21.15.119:0/2770459835 >> 172.21.15.119:0/2770459835 conn(0x7f381006a350 msgr2=0x7f381006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:17.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- 172.21.15.119:0/2770459835 shutdown_connections 2024-09-13T16:28:17.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.857+0000 7f381611a700 1 -- 172.21.15.119:0/2770459835 wait complete. 2024-09-13T16:28:17.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 Processor -- start 2024-09-13T16:28:17.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 -- start start 2024-09-13T16:28:17.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f381006c5f0 0x7f3810144d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:17.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38100cd200 0x7f38100ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:17.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f381015a010 0x7f38100ac980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:17.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38101ee300 con 0x7f38100cd200 2024-09-13T16:28:17.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38101454b0 con 0x7f381006c5f0 2024-09-13T16:28:17.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38100aa420 con 0x7f381015a010 2024-09-13T16:28:17.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38100cd200 0x7f38100ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:17.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38100cd200 0x7f38100ac440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50346/0 (socket says 172.21.15.119:50346) 2024-09-13T16:28:17.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 -- 172.21.15.119:0/3693546590 learned_addr learned my addr 172.21.15.119:0/3693546590 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:17.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380ffff700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f381015a010 0x7f38100ac980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:17.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380f7fe700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f381006c5f0 0x7f3810144d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:17.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 -- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f381015a010 msgr2=0x7f38100ac980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:17.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f381015a010 0x7f38100ac980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 -- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f381006c5f0 msgr2=0x7f3810144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:17.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f381006c5f0 0x7f3810144d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:17.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 -- 172.21.15.119:0/3693546590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38000094b0 con 0x7f38100cd200 2024-09-13T16:28:17.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f380effd700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38100cd200 0x7f38100ac440 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f381006dda0 tx=0x7f3800004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:17.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f37f7fff700 1 -- 172.21.15.119:0/3693546590 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38000044d0 con 0x7f38100cd200 2024-09-13T16:28:17.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38100a9290 con 0x7f38100cd200 2024-09-13T16:28:17.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f37f7fff700 1 -- 172.21.15.119:0/3693546590 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38000036c0 con 0x7f38100cd200 2024-09-13T16:28:17.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f37f7fff700 1 -- 172.21.15.119:0/3693546590 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3800003a70 con 0x7f38100cd200 2024-09-13T16:28:17.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.861+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38101459e0 con 0x7f38100cd200 2024-09-13T16:28:17.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.865+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3810067760 con 0x7f38100cd200 2024-09-13T16:28:17.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.865+0000 7f37f7fff700 1 -- 172.21.15.119:0/3693546590 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3800003bd0 con 0x7f38100cd200 2024-09-13T16:28:17.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.865+0000 7f37f7fff700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37fc066730 0x7f37fc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:17.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.865+0000 7f37f7fff700 1 -- 172.21.15.119:0/3693546590 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f3800085360 con 0x7f38100cd200 2024-09-13T16:28:17.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.865+0000 7f380f7fe700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37fc066730 0x7f37fc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:17.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.865+0000 7f380f7fe700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37fc066730 0x7f37fc068bf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3804009a00 tx=0x7f3804008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:17.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:17.873+0000 7f37f7fff700 1 -- 172.21.15.119:0/3693546590 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f38000528f0 con 0x7f38100cd200 2024-09-13T16:28:18.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.105+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f38100cd670 con 0x7f38100cd200 2024-09-13T16:28:18.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.105+0000 7f37f7fff700 1 -- 172.21.15.119:0/3693546590 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v12) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f3800074090 con 0x7f38100cd200 2024-09-13T16:28:18.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:18.110 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:18.110 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_dump": [ 2024-09-13T16:28:18.110 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.110 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.a", 2024-09-13T16:28:18.110 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA==", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.b", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g==", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:28:18.111 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.c", 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ==", 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.d", 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw==", 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.0", 2024-09-13T16:28:18.113 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA==", 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.1", 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw==", 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.114 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.2", 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ==", 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.3", 2024-09-13T16:28:18.115 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww==", 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.4", 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA==", 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.116 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.5", 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q==", 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.6", 2024-09-13T16:28:18.117 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg==", 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.7", 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ==", 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:28:18.118 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.0", 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ==", 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow r", 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.119 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.1", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ==", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.2", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw==", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.120 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.3", 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow==", 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:28:18.121 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.admin", 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmpEcAJRAAYorOK9AU5Q/2uOTIUrZhJw==", 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.x", 2024-09-13T16:28:18.122 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ==", 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.y", 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw==", 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:28:18.123 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:28:18.124 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.124 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:18.124 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:28:18.124 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:18.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37fc066730 msgr2=0x7f37fc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37fc066730 0x7f37fc068bf0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f3804009a00 tx=0x7f3804008040 comp rx=0 tx=0).stop 2024-09-13T16:28:18.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38100cd200 msgr2=0x7f38100ac440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38100cd200 0x7f38100ac440 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f381006dda0 tx=0x7f3800004770 comp rx=0 tx=0).stop 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 shutdown_connections 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37fc066730 0x7f37fc068bf0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f381015a010 0x7f38100ac980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38100cd200 0x7f38100ac440 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 --2- 172.21.15.119:0/3693546590 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f381006c5f0 0x7f3810144d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 >> 172.21.15.119:0/3693546590 conn(0x7f381006a350 msgr2=0x7f381006bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 shutdown_connections 2024-09-13T16:28:18.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.109+0000 7f381611a700 1 -- 172.21.15.119:0/3693546590 wait complete. 2024-09-13T16:28:18.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:18.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.237+0000 7fa896957700 1 Processor -- start 2024-09-13T16:28:18.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.237+0000 7fa896957700 1 -- start start 2024-09-13T16:28:18.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.237+0000 7fa896957700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa89006dc40 0x7fa8900b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.237+0000 7fa896957700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900b2280 con 0x7fa8900b3630 2024-09-13T16:28:18.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.237+0000 7fa896957700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8900cc890 con 0x7fa8901594f0 2024-09-13T16:28:18.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.237+0000 7fa896957700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8900b1970 con 0x7fa89006dc40 2024-09-13T16:28:18.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896156700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa8900b3630 0x7fa89006fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42430/0 (socket says 172.21.15.119:42430) 2024-09-13T16:28:18.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896156700 1 -- 172.21.15.119:0/2503503110 learned_addr learned my addr 172.21.15.119:0/2503503110 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:18.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa895154700 1 --2- 172.21.15.119:0/2503503110 >> v2:172.21.15.145:3300/0 conn(0x7fa89006dc40 0x7fa8900b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa894953700 1 -- 172.21.15.119:0/2503503110 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195180719 0 0) 0x7fa8900b2280 con 0x7fa8900b3630 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa894953700 1 -- 172.21.15.119:0/2503503110 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa87c0035d0 con 0x7fa8900b3630 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa895154700 1 -- 172.21.15.119:0/2503503110 >> v1:172.21.15.145:6789/0 conn(0x7fa8901594f0 legacy=0x7fa89006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa894953700 1 -- 172.21.15.119:0/2503503110 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2296006435 0 0) 0x7fa8900cc890 con 0x7fa8901594f0 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa895154700 1 -- 172.21.15.119:0/2503503110 >> v1:172.21.15.119:6789/0 conn(0x7fa8900b3630 legacy=0x7fa89006fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa895154700 1 -- 172.21.15.119:0/2503503110 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8900b1c30 con 0x7fa89006dc40 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa895154700 1 --2- 172.21.15.119:0/2503503110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa89006dc40 0x7fa8900b30f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa884009850 tx=0x7fa884009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e1e38242b784f31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa894953700 1 -- 172.21.15.119:0/2503503110 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884011830 con 0x7fa89006dc40 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa894953700 1 -- 172.21.15.119:0/2503503110 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa884011990 con 0x7fa89006dc40 2024-09-13T16:28:18.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa894953700 1 -- 172.21.15.119:0/2503503110 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa884011d00 con 0x7fa89006dc40 2024-09-13T16:28:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896957700 1 -- 172.21.15.119:0/2503503110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa89006dc40 msgr2=0x7fa8900b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896957700 1 --2- 172.21.15.119:0/2503503110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa89006dc40 0x7fa8900b30f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fa884009850 tx=0x7fa884009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896957700 1 -- 172.21.15.119:0/2503503110 shutdown_connections 2024-09-13T16:28:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896957700 1 --2- 172.21.15.119:0/2503503110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa89006dc40 0x7fa8900b30f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896957700 1 -- 172.21.15.119:0/2503503110 >> 172.21.15.119:0/2503503110 conn(0x7fa89006a350 msgr2=0x7fa89006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896957700 1 -- 172.21.15.119:0/2503503110 shutdown_connections 2024-09-13T16:28:18.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.241+0000 7fa896957700 1 -- 172.21.15.119:0/2503503110 wait complete. 2024-09-13T16:28:18.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 Processor -- start 2024-09-13T16:28:18.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 -- start start 2024-09-13T16:28:18.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89006dc40 0x7fa8901481a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8900b3630 0x7fa8900af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8901594f0 0x7fa8900afb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa890157fe0 con 0x7fa89006dc40 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8901582c0 con 0x7fa8900b3630 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896957700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8901585a0 con 0x7fa8901594f0 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89006dc40 0x7fa8901481a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89006dc40 0x7fa8901481a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50360/0 (socket says 172.21.15.119:50360) 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 -- 172.21.15.119:0/1104205563 learned_addr learned my addr 172.21.15.119:0/1104205563 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa896156700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8901594f0 0x7fa8900afb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895154700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8900b3630 0x7fa8900af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 -- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8901594f0 msgr2=0x7fa8900afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8901594f0 0x7fa8900afb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 -- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8900b3630 msgr2=0x7fa8900af610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8900b3630 0x7fa8900af610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 -- 172.21.15.119:0/1104205563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8840094b0 con 0x7fa89006dc40 2024-09-13T16:28:18.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.245+0000 7fa895955700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89006dc40 0x7fa8901481a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c520 tx=0x7fa88c00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa8827fc700 1 -- 172.21.15.119:0/1104205563 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c0054e0 con 0x7fa89006dc40 2024-09-13T16:28:18.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa8827fc700 1 -- 172.21.15.119:0/1104205563 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c0036a0 con 0x7fa89006dc40 2024-09-13T16:28:18.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa8827fc700 1 -- 172.21.15.119:0/1104205563 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c003a50 con 0x7fa89006dc40 2024-09-13T16:28:18.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa890158880 con 0x7fa89006dc40 2024-09-13T16:28:18.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8900a8f30 con 0x7fa89006dc40 2024-09-13T16:28:18.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8900b30f0 con 0x7fa89006dc40 2024-09-13T16:28:18.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa8827fc700 1 -- 172.21.15.119:0/1104205563 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa88c003bb0 con 0x7fa89006dc40 2024-09-13T16:28:18.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa8827fc700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa874066760 0x7fa874068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa8827fc700 1 -- 172.21.15.119:0/1104205563 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa88c0832f0 con 0x7fa89006dc40 2024-09-13T16:28:18.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.249+0000 7fa895154700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa874066760 0x7fa874068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.253+0000 7fa895154700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa874066760 0x7fa874068c20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa8840090b0 tx=0x7fa884008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.253+0000 7fa8827fc700 1 -- 172.21.15.119:0/1104205563 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa88c050870 con 0x7fa89006dc40 2024-09-13T16:28:18.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.429+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa890155700 con 0x7fa89006dc40 2024-09-13T16:28:18.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa8827fc700 1 -- 172.21.15.119:0/1104205563 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v64) v1 ==== 74+0+13122 (secure 0 0 0) 0x7fa88c003e60 con 0x7fa89006dc40 2024-09-13T16:28:18.436 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:18.436 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":64,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:28:17.704169+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":55,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa874066760 msgr2=0x7fa874068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa874066760 0x7fa874068c20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa8840090b0 tx=0x7fa884008030 comp rx=0 tx=0).stop 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89006dc40 msgr2=0x7fa8901481a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89006dc40 0x7fa8901481a0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7fa88c00c520 tx=0x7fa88c00c830 comp rx=0 tx=0).stop 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 shutdown_connections 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa874066760 0x7fa874068c20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8901594f0 0x7fa8900afb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8900b3630 0x7fa8900af610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 --2- 172.21.15.119:0/1104205563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa89006dc40 0x7fa8901481a0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 >> 172.21.15.119:0/1104205563 conn(0x7fa89006a350 msgr2=0x7fa89006d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:18.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 shutdown_connections 2024-09-13T16:28:18.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.433+0000 7fa896957700 1 -- 172.21.15.119:0/1104205563 wait complete. 2024-09-13T16:28:18.452 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:28:18.452 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:28:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.561+0000 7f64f6689700 1 Processor -- start 2024-09-13T16:28:18.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.561+0000 7f64f6689700 1 -- start start 2024-09-13T16:28:18.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f64f00add60 0x7f64f00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64f00b3260 0x7f64f00b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f00cd4d0 con 0x7f64f015dbe0 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f0071be0 con 0x7f64f00b3260 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f0159af0 con 0x7f64f00add60 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64f00b3260 0x7f64f00b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64f00b3260 0x7f64f00b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50364/0 (socket says 172.21.15.119:50364) 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 -- 172.21.15.119:0/3167113450 learned_addr learned my addr 172.21.15.119:0/3167113450 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f4e86700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f64f015dbe0 0x7f64f00cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42436/0 (socket says 172.21.15.119:42436) 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5687700 1 --2- 172.21.15.119:0/3167113450 >> v2:172.21.15.145:3300/0 conn(0x7f64f00add60 0x7f64f00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 -- 172.21.15.119:0/3167113450 >> v2:172.21.15.145:3300/0 conn(0x7f64f00add60 msgr2=0x7f64f00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 --2- 172.21.15.119:0/3167113450 >> v2:172.21.15.145:3300/0 conn(0x7f64f00add60 0x7f64f00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 -- 172.21.15.119:0/3167113450 >> v1:172.21.15.119:6789/0 conn(0x7f64f015dbe0 legacy=0x7f64f00cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64effff700 1 -- 172.21.15.119:0/3167113450 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172760948 0 0) 0x7f64f00cd4d0 con 0x7f64f015dbe0 2024-09-13T16:28:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 -- 172.21.15.119:0/3167113450 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f015e130 con 0x7f64f00b3260 2024-09-13T16:28:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f5e88700 1 --2- 172.21.15.119:0/3167113450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f00b3670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f64f00b1530 tx=0x7f64e8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de41c8a9e8b6739c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64effff700 1 -- 172.21.15.119:0/3167113450 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8011830 con 0x7f64f00b3260 2024-09-13T16:28:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64effff700 1 -- 172.21.15.119:0/3167113450 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e8011990 con 0x7f64f00b3260 2024-09-13T16:28:18.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64effff700 1 -- 172.21.15.119:0/3167113450 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e8011d20 con 0x7f64f00b3260 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- 172.21.15.119:0/3167113450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 msgr2=0x7f64f00b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 --2- 172.21.15.119:0/3167113450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f00b3670 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f64f00b1530 tx=0x7f64e8009bf0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- 172.21.15.119:0/3167113450 shutdown_connections 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 --2- 172.21.15.119:0/3167113450 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f00b3670 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 --2- 172.21.15.119:0/3167113450 >> v2:172.21.15.145:3300/0 conn(0x7f64f00add60 0x7f64f00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- 172.21.15.119:0/3167113450 >> 172.21.15.119:0/3167113450 conn(0x7f64f00b0980 msgr2=0x7f64f00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- 172.21.15.119:0/3167113450 shutdown_connections 2024-09-13T16:28:18.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- 172.21.15.119:0/3167113450 wait complete. 2024-09-13T16:28:18.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 Processor -- start 2024-09-13T16:28:18.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.565+0000 7f64f6689700 1 -- start start 2024-09-13T16:28:18.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f00add60 0x7f64f014f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f01558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f015dbe0 0x7f64f015bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0150e70 con 0x7f64f00b3260 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0151150 con 0x7f64f00add60 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f0151430 con 0x7f64f015dbe0 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f01558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f01558e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50366/0 (socket says 172.21.15.119:50366) 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 -- 172.21.15.119:0/280935765 learned_addr learned my addr 172.21.15.119:0/280935765 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f5687700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f00add60 0x7f64f014f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f5e88700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f015dbe0 0x7f64f015bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 -- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f015dbe0 msgr2=0x7f64f015bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f015dbe0 0x7f64f015bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 -- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f00add60 msgr2=0x7f64f014f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f00add60 0x7f64f014f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 -- 172.21.15.119:0/280935765 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64e8009570 con 0x7f64f00b3260 2024-09-13T16:28:18.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f4e86700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f01558e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c4e0 tx=0x7f64e000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64edffb700 1 -- 172.21.15.119:0/280935765 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0004ca0 con 0x7f64f00b3260 2024-09-13T16:28:18.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64edffb700 1 -- 172.21.15.119:0/280935765 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e0004e20 con 0x7f64f00b3260 2024-09-13T16:28:18.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64edffb700 1 -- 172.21.15.119:0/280935765 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e0007310 con 0x7f64f00b3260 2024-09-13T16:28:18.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f0151710 con 0x7f64f00b3260 2024-09-13T16:28:18.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f0150ab0 con 0x7f64f00b3260 2024-09-13T16:28:18.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.569+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f0151290 con 0x7f64f00b3260 2024-09-13T16:28:18.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.573+0000 7f64edffb700 1 -- 172.21.15.119:0/280935765 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f64e0007470 con 0x7f64f00b3260 2024-09-13T16:28:18.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.573+0000 7f64edffb700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d0066730 0x7f64d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.573+0000 7f64edffb700 1 -- 172.21.15.119:0/280935765 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f64e0083b90 con 0x7f64f00b3260 2024-09-13T16:28:18.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.573+0000 7f64f5687700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d0066730 0x7f64d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.573+0000 7f64f5687700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d0066730 0x7f64d0068bf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f64d80113e0 tx=0x7f64d800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.577+0000 7f64edffb700 1 -- 172.21.15.119:0/280935765 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f64f0151290 con 0x7f64f00b3260 2024-09-13T16:28:18.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.745+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f64f0151c30 con 0x7f64f00b3260 2024-09-13T16:28:18.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.745+0000 7f64edffb700 1 -- 172.21.15.119:0/280935765 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v64) v1 ==== 93+0+10 (secure 0 0 0) 0x7f64f0151c30 con 0x7f64f00b3260 2024-09-13T16:28:18.749 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:28:18.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d0066730 msgr2=0x7f64d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d0066730 0x7f64d0068bf0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f64d80113e0 tx=0x7f64d800b040 comp rx=0 tx=0).stop 2024-09-13T16:28:18.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 msgr2=0x7f64f01558e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f01558e0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f64e000c4e0 tx=0x7f64e000c7f0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 shutdown_connections 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d0066730 0x7f64d0068bf0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f015dbe0 0x7f64f015bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f00b3260 0x7f64f01558e0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 --2- 172.21.15.119:0/280935765 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f00add60 0x7f64f014f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 >> 172.21.15.119:0/280935765 conn(0x7f64f00b0980 msgr2=0x7f64f00b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 shutdown_connections 2024-09-13T16:28:18.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.749+0000 7f64f6689700 1 -- 172.21.15.119:0/280935765 wait complete. 2024-09-13T16:28:18.764 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:28:18.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.873+0000 7fa4d64f3700 1 Processor -- start 2024-09-13T16:28:18.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.873+0000 7fa4d64f3700 1 -- start start 2024-09-13T16:28:18.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d015cf30 0x7fa4d00715e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa4d0071b20 0x7fa4d006d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d006c3b0 con 0x7fa4d00ac1b0 2024-09-13T16:28:18.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d015d430 con 0x7fa4d0071b20 2024-09-13T16:28:18.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d015d710 con 0x7fa4d015cf30 2024-09-13T16:28:18.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa4d0071b20 0x7fa4d006d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa4d0071b20 0x7fa4d006d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50372/0 (socket says 172.21.15.119:50372) 2024-09-13T16:28:18.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 -- 172.21.15.119:0/3080759273 learned_addr learned my addr 172.21.15.119:0/3080759273 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:18.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d4cf0700 1 --2- 172.21.15.119:0/3080759273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d015cf30 0x7fa4d00715e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 -- 172.21.15.119:0/3080759273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d015cf30 msgr2=0x7fa4d00715e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:18.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 --2- 172.21.15.119:0/3080759273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d015cf30 0x7fa4d00715e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 -- 172.21.15.119:0/3080759273 >> v1:172.21.15.145:6789/0 conn(0x7fa4d00ac1b0 legacy=0x7fa4d006b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 -- 172.21.15.119:0/3080759273 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d015d9f0 con 0x7fa4d0071b20 2024-09-13T16:28:18.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d5cf2700 1 --2- 172.21.15.119:0/3080759273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0071b20 0x7fa4d006d900 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009910 tx=0x7fa4c8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc0d2b3038eb6dab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4c7fff700 1 -- 172.21.15.119:0/3080759273 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8011830 con 0x7fa4d0071b20 2024-09-13T16:28:18.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4c7fff700 1 -- 172.21.15.119:0/3080759273 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c8011990 con 0x7fa4d0071b20 2024-09-13T16:28:18.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4c7fff700 1 -- 172.21.15.119:0/3080759273 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4c8011d40 con 0x7fa4d0071b20 2024-09-13T16:28:18.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- 172.21.15.119:0/3080759273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0071b20 msgr2=0x7fa4d006d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/3080759273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0071b20 0x7fa4d006d900 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa4c8009910 tx=0x7fa4c8009c20 comp rx=0 tx=0).stop 2024-09-13T16:28:18.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- 172.21.15.119:0/3080759273 shutdown_connections 2024-09-13T16:28:18.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/3080759273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0071b20 0x7fa4d006d900 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/3080759273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d015cf30 0x7fa4d00715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- 172.21.15.119:0/3080759273 >> 172.21.15.119:0/3080759273 conn(0x7fa4d00ae6e0 msgr2=0x7fa4d00b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:18.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- 172.21.15.119:0/3080759273 shutdown_connections 2024-09-13T16:28:18.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.877+0000 7fa4d64f3700 1 -- 172.21.15.119:0/3080759273 wait complete. 2024-09-13T16:28:18.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.881+0000 7fa4d64f3700 1 Processor -- start 2024-09-13T16:28:18.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.881+0000 7fa4d64f3700 1 -- start start 2024-09-13T16:28:18.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4d015cf30 0x7fa4d01f2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0158580 0x7fa4d0158a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d009b7d0 0x7fa4d009bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01545f0 con 0x7fa4d0158580 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01e2dc0 con 0x7fa4d015cf30 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d01f2cb0 con 0x7fa4d009b7d0 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0158580 0x7fa4d0158a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0158580 0x7fa4d0158a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50378/0 (socket says 172.21.15.119:50378) 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 -- 172.21.15.119:0/2772708189 learned_addr learned my addr 172.21.15.119:0/2772708189 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d5cf2700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d009b7d0 0x7fa4d009bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d54f1700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4d015cf30 0x7fa4d01f2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 -- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d009b7d0 msgr2=0x7fa4d009bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d009b7d0 0x7fa4d009bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 -- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4d015cf30 msgr2=0x7fa4d01f2770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4d015cf30 0x7fa4d01f2770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 -- 172.21.15.119:0/2772708189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c8009570 con 0x7fa4d0158580 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d54f1700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4d015cf30 0x7fa4d01f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d4cf0700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0158580 0x7fa4d0158a00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009d10 tx=0x7fa4cc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4c5ffb700 1 -- 172.21.15.119:0/2772708189 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc00cd10 con 0x7fa4d0158580 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4c5ffb700 1 -- 172.21.15.119:0/2772708189 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc00f040 con 0x7fa4d0158580 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4d009aaa0 con 0x7fa4d0158580 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4c5ffb700 1 -- 172.21.15.119:0/2772708189 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4cc0146a0 con 0x7fa4d0158580 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d009b340 con 0x7fa4d0158580 2024-09-13T16:28:18.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.885+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d009b4d0 con 0x7fa4d0158580 2024-09-13T16:28:18.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.889+0000 7fa4c5ffb700 1 -- 172.21.15.119:0/2772708189 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa4cc014800 con 0x7fa4d0158580 2024-09-13T16:28:18.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.889+0000 7fa4c5ffb700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4bc066730 0x7fa4bc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:18.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.889+0000 7fa4c5ffb700 1 -- 172.21.15.119:0/2772708189 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa4cc084a70 con 0x7fa4d0158580 2024-09-13T16:28:18.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.889+0000 7fa4d54f1700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4bc066730 0x7fa4bc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:18.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.889+0000 7fa4d54f1700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4bc066730 0x7fa4bc068bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0000f60 tx=0x7fa4c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:18.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:18.893+0000 7fa4c5ffb700 1 -- 172.21.15.119:0/2772708189 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa4cc051ed0 con 0x7fa4d0158580 2024-09-13T16:28:19.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.065+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa4d00715e0 con 0x7fa4d0158580 2024-09-13T16:28:19.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4c5ffb700 1 -- 172.21.15.119:0/2772708189 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v64) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa4cc009e60 con 0x7fa4d0158580 2024-09-13T16:28:19.071 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:19.071 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:28:19.071 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:19.071 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:28:19.071 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:28:19.072 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:19.072 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4bc066730 msgr2=0x7fa4bc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4bc066730 0x7fa4bc068bf0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fa4c0000f60 tx=0x7fa4c0009040 comp rx=0 tx=0).stop 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0158580 msgr2=0x7fa4d0158a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0158580 0x7fa4d0158a00 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa4cc009d10 tx=0x7fa4cc00c3a0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 shutdown_connections 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4bc066730 0x7fa4bc068bf0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa4d009b7d0 0x7fa4d009bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa4d0158580 0x7fa4d0158a00 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 --2- 172.21.15.119:0/2772708189 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa4d015cf30 0x7fa4d01f2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 >> 172.21.15.119:0/2772708189 conn(0x7fa4d00ac1b0 msgr2=0x7fa4d00b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 shutdown_connections 2024-09-13T16:28:19.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.069+0000 7fa4d64f3700 1 -- 172.21.15.119:0/2772708189 wait complete. 2024-09-13T16:28:19.092 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:28:19.092 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:28:19.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.209+0000 7f9a10f4e700 1 Processor -- start 2024-09-13T16:28:19.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.209+0000 7f9a10f4e700 1 -- start start 2024-09-13T16:28:19.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a10f4e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 0x7f9a0c0cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a10f4e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0d3260 con 0x7f9a0c0d0720 2024-09-13T16:28:19.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a10f4e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a0c0cd640 con 0x7f9a0c0d1840 2024-09-13T16:28:19.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a10f4e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0d0130 con 0x7f9a0c15cc40 2024-09-13T16:28:19.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0affd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9a0c0d0720 0x7f9a0c0d1130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42448/0 (socket says 172.21.15.119:42448) 2024-09-13T16:28:19.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0affd700 1 -- 172.21.15.119:0/1227698090 learned_addr learned my addr 172.21.15.119:0/1227698090 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:19.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0b7fe700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 0x7f9a0c0cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787515078 0 0) 0x7f9a0c0d3260 con 0x7f9a0c0d0720 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f80035d0 con 0x7f9a0c0d0720 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 588498943 0 0) 0x7f99f80035d0 con 0x7f9a0c0d0720 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 msgr2=0x7f9a0c0cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 0x7f9a0c0cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 >> v1:172.21.15.145:6789/0 conn(0x7f9a0c0d1840 legacy=0x7f9a0c0cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c0d0c70 con 0x7f9a0c0d0720 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f99fc001670 con 0x7f9a0c0d0720 2024-09-13T16:28:19.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 >> v1:172.21.15.119:6789/0 conn(0x7f9a0c0d0720 legacy=0x7f9a0c0d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 0x7f99f8003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8003f90 0x7f99f8004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99f8004db0 0x7f99f8005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0d0130 con 0x7f99f8004db0 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f800bb40 con 0x7f99f8003f90 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f800bd00 con 0x7f9a0c15cc40 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0bfff700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99f8004db0 0x7f99f8005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0b7fe700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 0x7f99f8003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0affd700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8003f90 0x7f99f8004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0bfff700 1 -- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 msgr2=0x7f99f8003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0bfff700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 0x7f99f8003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0bfff700 1 -- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8003f90 msgr2=0x7f99f8004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:19.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0bfff700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8003f90 0x7f99f8004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0bfff700 1 -- 172.21.15.119:0/1227698090 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80035d0 con 0x7f99f8004db0 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0b7fe700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15cc40 0x7f99f8003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0bfff700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99f8004db0 0x7f99f8005230 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c260 tx=0x7f99f80058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52210b9026796277 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04001e70 con 0x7f99f8004db0 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a0400ce20 con 0x7f99f8004db0 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.213+0000 7f9a0a7fc700 1 -- 172.21.15.119:0/1227698090 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a04012660 con 0x7f99f8004db0 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.217+0000 7f9a10f4e700 1 -- 172.21.15.119:0/1227698090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99f8004db0 msgr2=0x7f99f8005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.217+0000 7f9a10f4e700 1 --2- 172.21.15.119:0/1227698090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99f8004db0 0x7f99f8005230 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f9a0400c260 tx=0x7f99f80058a0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.217+0000 7f9a0bfff700 1 -- 172.21.15.119:0/1227698090 reap_dead start 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.217+0000 7f9a10f4e700 1 -- 172.21.15.119:0/1227698090 shutdown_connections 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.217+0000 7f9a10f4e700 1 -- 172.21.15.119:0/1227698090 >> 172.21.15.119:0/1227698090 conn(0x7f9a0c09be40 msgr2=0x7f9a0c0d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:19.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.217+0000 7f9a10f4e700 1 -- 172.21.15.119:0/1227698090 shutdown_connections 2024-09-13T16:28:19.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.217+0000 7f9a10f4e700 1 -- 172.21.15.119:0/1227698090 wait complete. 2024-09-13T16:28:19.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 Processor -- start 2024-09-13T16:28:19.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 -- start start 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8004db0 0x7f9a0c154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15d530 0x7f9a0c15d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a0c0b08d0 0x7f9a0c0b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c0b1290 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c15ec00 con 0x7f99f8004db0 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0c15dff0 con 0x7f9a0c15d530 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a0c0b08d0 0x7f9a0c0b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a0c0b08d0 0x7f9a0c0b0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50398/0 (socket says 172.21.15.119:50398) 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 -- 172.21.15.119:0/2784478161 learned_addr learned my addr 172.21.15.119:0/2784478161 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:19.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0b7fe700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8004db0 0x7f9a0c154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0affd700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15d530 0x7f9a0c15d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 -- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15d530 msgr2=0x7f9a0c15d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15d530 0x7f9a0c15d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 -- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8004db0 msgr2=0x7f9a0c154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8004db0 0x7f9a0c154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 -- 172.21.15.119:0/2784478161 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0c15e2d0 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0affd700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15d530 0x7f9a0c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a0bfff700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a0c0b08d0 0x7f9a0c0b0d50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f9a04013000 tx=0x7f9a040015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a03fff700 1 -- 172.21.15.119:0/2784478161 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040050c0 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a03fff700 1 -- 172.21.15.119:0/2784478161 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a04005240 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a03fff700 1 -- 172.21.15.119:0/2784478161 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a040055f0 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a0c15e5b0 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.221+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a0c154760 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.225+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0c06ab20 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.225+0000 7f9a03fff700 1 -- 172.21.15.119:0/2784478161 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9a04023070 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.225+0000 7f9a03fff700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99ec066730 0x7f99ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.225+0000 7f9a03fff700 1 -- 172.21.15.119:0/2784478161 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f9a040858d0 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.225+0000 7f9a0b7fe700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99ec066730 0x7f99ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.229+0000 7f9a0b7fe700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99ec066730 0x7f99ec068bf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009bf0 tx=0x7f99f4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:19.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.237+0000 7f9a03fff700 1 -- 172.21.15.119:0/2784478161 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9a04052e40 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.401+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f9a0c0d0320 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.725+0000 7f9a03fff700 1 -- 172.21.15.119:0/2784478161 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9a04056460 con 0x7f9a0c0b08d0 2024-09-13T16:28:19.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99ec066730 msgr2=0x7f99ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99ec066730 0x7f99ec068bf0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009bf0 tx=0x7f99f4008040 comp rx=0 tx=0).stop 2024-09-13T16:28:19.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a0c0b08d0 msgr2=0x7f9a0c0b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a0c0b08d0 0x7f9a0c0b0d50 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f9a04013000 tx=0x7f9a040015b0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 shutdown_connections 2024-09-13T16:28:19.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99ec066730 0x7f99ec068bf0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a0c0b08d0 0x7f9a0c0b0d50 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9a0c15d530 0x7f9a0c15d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 --2- 172.21.15.119:0/2784478161 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99f8004db0 0x7f9a0c154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 >> 172.21.15.119:0/2784478161 conn(0x7f9a0c09be40 msgr2=0x7f9a0c0cf940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:19.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 shutdown_connections 2024-09-13T16:28:19.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.737+0000 7f9a10f4e700 1 -- 172.21.15.119:0/2784478161 wait complete. 2024-09-13T16:28:19.743 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:28:19.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:28:19.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5242ad0700 1 Processor -- start 2024-09-13T16:28:19.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5242ad0700 1 -- start start 2024-09-13T16:28:19.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5242ad0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f523c0b2bb0 0x7f523c1652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5242ad0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c06bda0 0x7f523c06c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5242ad0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f523c071ac0 con 0x7f523c0ade80 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5242ad0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523c1657f0 con 0x7f523c0b2bb0 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5242ad0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c070eb0 con 0x7f523c06bda0 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52412cd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f523c0b2bb0 0x7f523c1652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c06bda0 0x7f523c06c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c06bda0 0x7f523c06c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:57650/0 (socket says 172.21.15.119:57650) 2024-09-13T16:28:19.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 -- 172.21.15.119:0/3254287503 learned_addr learned my addr 172.21.15.119:0/3254287503 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:19.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 -- 172.21.15.119:0/3254287503 >> v2:172.21.15.145:3300/0 conn(0x7f523c0b2bb0 msgr2=0x7f523c1652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:19.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 --2- 172.21.15.119:0/3254287503 >> v2:172.21.15.145:3300/0 conn(0x7f523c0b2bb0 0x7f523c1652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 -- 172.21.15.119:0/3254287503 >> v1:172.21.15.145:6789/0 conn(0x7f523c0ade80 legacy=0x7f523c0b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 -- 172.21.15.119:0/3254287503 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c071190 con 0x7f523c06bda0 2024-09-13T16:28:19.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52412cd700 1 --2- 172.21.15.119:0/3254287503 >> v2:172.21.15.145:3300/0 conn(0x7f523c0b2bb0 0x7f523c1652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:19.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f52422cf700 1 --2- 172.21.15.119:0/3254287503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c06bda0 0x7f523c06c250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5234009e50 tx=0x7f523400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f53cc21cfb0c664 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:19.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.873+0000 7f5240acc700 1 -- 172.21.15.119:0/3254287503 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523400e070 con 0x7f523c06bda0 2024-09-13T16:28:19.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5240acc700 1 -- 172.21.15.119:0/3254287503 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523400b910 con 0x7f523c06bda0 2024-09-13T16:28:19.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5240acc700 1 -- 172.21.15.119:0/3254287503 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523400bcc0 con 0x7f523c06bda0 2024-09-13T16:28:19.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 -- 172.21.15.119:0/3254287503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c06bda0 msgr2=0x7f523c06c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 --2- 172.21.15.119:0/3254287503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c06bda0 0x7f523c06c250 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5234009e50 tx=0x7f523400b330 comp rx=0 tx=0).stop 2024-09-13T16:28:19.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 -- 172.21.15.119:0/3254287503 shutdown_connections 2024-09-13T16:28:19.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 --2- 172.21.15.119:0/3254287503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c06bda0 0x7f523c06c250 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 --2- 172.21.15.119:0/3254287503 >> v2:172.21.15.145:3300/0 conn(0x7f523c0b2bb0 0x7f523c1652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 -- 172.21.15.119:0/3254287503 >> 172.21.15.119:0/3254287503 conn(0x7f523c0b0980 msgr2=0x7f523c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:19.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 -- 172.21.15.119:0/3254287503 shutdown_connections 2024-09-13T16:28:19.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 -- 172.21.15.119:0/3254287503 wait complete. 2024-09-13T16:28:19.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 Processor -- start 2024-09-13T16:28:19.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.877+0000 7f5242ad0700 1 -- start start 2024-09-13T16:28:19.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f523c06bda0 0x7f523c161740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c0ade80 0x7f523c0a8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c0b2bb0 0x7f523c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c1f6b00 con 0x7f523c0ade80 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c1f6de0 con 0x7f523c06bda0 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523c1f70c0 con 0x7f523c0b2bb0 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c0ade80 0x7f523c0a8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c0ade80 0x7f523c0a8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50402/0 (socket says 172.21.15.119:50402) 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 -- 172.21.15.119:0/2046056268 learned_addr learned my addr 172.21.15.119:0/2046056268 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52422cf700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c0b2bb0 0x7f523c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5241ace700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f523c06bda0 0x7f523c161740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 -- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c0b2bb0 msgr2=0x7f523c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c0b2bb0 0x7f523c15c920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 -- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f523c06bda0 msgr2=0x7f523c161740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:19.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f523c06bda0 0x7f523c161740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 -- 172.21.15.119:0/2046056268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52380094b0 con 0x7f523c0ade80 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5241ace700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f523c06bda0 0x7f523c161740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52412cd700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c0ade80 0x7f523c0a8ba0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f523800ebe0 tx=0x7f5238009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52327fc700 1 -- 172.21.15.119:0/2046056268 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800c910 con 0x7f523c0ade80 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52327fc700 1 -- 172.21.15.119:0/2046056268 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523800ca70 con 0x7f523c0ade80 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5234009b30 con 0x7f523c0ade80 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f52327fc700 1 -- 172.21.15.119:0/2046056268 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523800ce00 con 0x7f523c0ade80 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c1f7680 con 0x7f523c0ade80 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.881+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c1652b0 con 0x7f523c0ade80 2024-09-13T16:28:19.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.885+0000 7f52327fc700 1 -- 172.21.15.119:0/2046056268 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5238013470 con 0x7f523c0ade80 2024-09-13T16:28:19.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.885+0000 7f52327fc700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5220066730 0x7f5220068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:19.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.885+0000 7f52327fc700 1 -- 172.21.15.119:0/2046056268 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f5238084270 con 0x7f523c0ade80 2024-09-13T16:28:19.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.885+0000 7f5241ace700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5220066730 0x7f5220068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:19.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.885+0000 7f5241ace700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5220066730 0x7f5220068bf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f522c000f60 tx=0x7f522c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:19.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:19.889+0000 7f52327fc700 1 -- 172.21.15.119:0/2046056268 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5238051690 con 0x7f523c0ade80 2024-09-13T16:28:20.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.061+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f523c0705b0 con 0x7f523c0ade80 2024-09-13T16:28:20.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.741+0000 7f52327fc700 1 -- 172.21.15.119:0/2046056268 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v66) v1 ==== 170+0+0 (secure 0 0 0) 0x7f5238054cb0 con 0x7f523c0ade80 2024-09-13T16:28:20.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5220066730 msgr2=0x7f5220068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:20.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5220066730 0x7f5220068bf0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f522c000f60 tx=0x7f522c009040 comp rx=0 tx=0).stop 2024-09-13T16:28:20.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c0ade80 msgr2=0x7f523c0a8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:20.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c0ade80 0x7f523c0a8ba0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f523800ebe0 tx=0x7f5238009ed0 comp rx=0 tx=0).stop 2024-09-13T16:28:20.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 shutdown_connections 2024-09-13T16:28:20.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5220066730 0x7f5220068bf0 secure :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f522c000f60 tx=0x7f522c009040 comp rx=0 tx=0).stop 2024-09-13T16:28:20.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f523c0b2bb0 0x7f523c15c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:20.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f523c0ade80 0x7f523c0a8ba0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:20.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 --2- 172.21.15.119:0/2046056268 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f523c06bda0 0x7f523c161740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:20.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.753+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 >> 172.21.15.119:0/2046056268 conn(0x7f523c0b0980 msgr2=0x7f523c0ae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:20.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.757+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 shutdown_connections 2024-09-13T16:28:20.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.757+0000 7f5242ad0700 1 -- 172.21.15.119:0/2046056268 wait complete. 2024-09-13T16:28:20.763 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:28:20.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:28:20.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0caa5b9700 1 Processor -- start 2024-09-13T16:28:20.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0caa5b9700 1 -- start start 2024-09-13T16:28:20.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0caa5b9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0ca406c480 0x7f0ca40cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:20.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0caa5b9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca406c8c0 con 0x7f0ca40ce000 2024-09-13T16:28:20.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0caa5b9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ca406cb40 con 0x7f0ca4144180 2024-09-13T16:28:20.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0caa5b9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ca40b1c50 con 0x7f0ca406c480 2024-09-13T16:28:20.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0ca8db6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0ca406c480 0x7f0ca40cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:20.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0ca8db6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0ca406c480 0x7f0ca40cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50408/0 (socket says 172.21.15.119:50408) 2024-09-13T16:28:20.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0ca8db6700 1 -- 172.21.15.119:0/4002222949 learned_addr learned my addr 172.21.15.119:0/4002222949 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:20.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0ca9db8700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0ca40ce000 0x7f0ca40b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42454/0 (socket says 172.21.15.119:42454) 2024-09-13T16:28:20.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0ca8db6700 1 -- 172.21.15.119:0/4002222949 >> v1:172.21.15.145:6789/0 conn(0x7f0ca4144180 legacy=0x7f0ca40cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:20.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0ca8db6700 1 -- 172.21.15.119:0/4002222949 >> v1:172.21.15.119:6789/0 conn(0x7f0ca40ce000 legacy=0x7f0ca40b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:20.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0ca8db6700 1 -- 172.21.15.119:0/4002222949 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca406ce20 con 0x7f0ca406c480 2024-09-13T16:28:20.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.885+0000 7f0c9bfff700 1 -- 172.21.15.119:0/4002222949 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138132611 0 0) 0x7f0ca406c8c0 con 0x7f0ca40ce000 2024-09-13T16:28:20.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0ca8db6700 1 --2- 172.21.15.119:0/4002222949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca406c480 0x7f0ca40cdac0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40b1d90 tx=0x7f0ca000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf51631df376227 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0c9bfff700 1 -- 172.21.15.119:0/4002222949 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca000e070 con 0x7f0ca406c480 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0c9bfff700 1 -- 172.21.15.119:0/4002222949 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca0003a40 con 0x7f0ca406c480 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0c9bfff700 1 -- 172.21.15.119:0/4002222949 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0003df0 con 0x7f0ca406c480 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 -- 172.21.15.119:0/4002222949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca406c480 msgr2=0x7f0ca40cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/4002222949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca406c480 0x7f0ca40cdac0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f0ca40b1d90 tx=0x7f0ca000b330 comp rx=0 tx=0).stop 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 -- 172.21.15.119:0/4002222949 shutdown_connections 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/4002222949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca406c480 0x7f0ca40cdac0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 -- 172.21.15.119:0/4002222949 >> 172.21.15.119:0/4002222949 conn(0x7f0ca40b0980 msgr2=0x7f0ca40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 -- 172.21.15.119:0/4002222949 shutdown_connections 2024-09-13T16:28:20.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 -- 172.21.15.119:0/4002222949 wait complete. 2024-09-13T16:28:20.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 Processor -- start 2024-09-13T16:28:20.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.889+0000 7f0caa5b9700 1 -- start start 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ca406c480 0x7f0ca4140990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca40ce000 0x7f0ca40ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ca4144180 0x7f0ca413d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40ace00 con 0x7f0ca40ce000 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40ad050 con 0x7f0ca406c480 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ca40ad330 con 0x7f0ca4144180 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca40ce000 0x7f0ca40ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca40ce000 0x7f0ca40ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50424/0 (socket says 172.21.15.119:50424) 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 -- 172.21.15.119:0/1735676107 learned_addr learned my addr 172.21.15.119:0/1735676107 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca95b7700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ca406c480 0x7f0ca4140990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:20.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca9db8700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ca4144180 0x7f0ca413d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 -- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ca4144180 msgr2=0x7f0ca413d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ca4144180 0x7f0ca413d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 -- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ca406c480 msgr2=0x7f0ca4140990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ca406c480 0x7f0ca4140990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 -- 172.21.15.119:0/1735676107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca0009b30 con 0x7f0ca40ce000 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca95b7700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ca406c480 0x7f0ca4140990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca9db8700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ca4144180 0x7f0ca413d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0ca8db6700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca40ce000 0x7f0ca40ac8c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0015040 tx=0x7f0ca0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0c99ffb700 1 -- 172.21.15.119:0/1735676107 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0004540 con 0x7f0ca40ce000 2024-09-13T16:28:20.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0c99ffb700 1 -- 172.21.15.119:0/1735676107 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca00046c0 con 0x7f0ca40ce000 2024-09-13T16:28:20.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ca40ad610 con 0x7f0ca40ce000 2024-09-13T16:28:20.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0c99ffb700 1 -- 172.21.15.119:0/1735676107 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca0012430 con 0x7f0ca40ce000 2024-09-13T16:28:20.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ca40a8550 con 0x7f0ca40ce000 2024-09-13T16:28:20.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.893+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ca40cdac0 con 0x7f0ca40ce000 2024-09-13T16:28:20.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.897+0000 7f0c99ffb700 1 -- 172.21.15.119:0/1735676107 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0ca0012590 con 0x7f0ca40ce000 2024-09-13T16:28:20.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.897+0000 7f0c99ffb700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0c88066730 0x7f0c88068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:20.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.897+0000 7f0c99ffb700 1 -- 172.21.15.119:0/1735676107 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f0ca0085c20 con 0x7f0ca40ce000 2024-09-13T16:28:20.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.897+0000 7f0ca95b7700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0c88066730 0x7f0c88068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:20.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.897+0000 7f0ca95b7700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0c88066730 0x7f0c88068bf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0c94000f60 tx=0x7f0c94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:20.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:20.905+0000 7f0c99ffb700 1 -- 172.21.15.119:0/1735676107 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0ca0085e60 con 0x7f0ca40ce000 2024-09-13T16:28:21.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.109+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0ca415d200 con 0x7f0ca40ce000 2024-09-13T16:28:21.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.117+0000 7f0c99ffb700 1 -- 172.21.15.119:0/1735676107 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 9 and data pool 10 v28) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0ca0056930 con 0x7f0ca40ce000 2024-09-13T16:28:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0c88066730 msgr2=0x7f0c88068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0c88066730 0x7f0c88068bf0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f0c94000f60 tx=0x7f0c94009040 comp rx=0 tx=0).stop 2024-09-13T16:28:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca40ce000 msgr2=0x7f0ca40ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca40ce000 0x7f0ca40ac8c0 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f0ca0015040 tx=0x7f0ca0004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 shutdown_connections 2024-09-13T16:28:21.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0c88066730 0x7f0c88068bf0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0ca4144180 0x7f0ca413d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0ca40ce000 0x7f0ca40ac8c0 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 --2- 172.21.15.119:0/1735676107 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0ca406c480 0x7f0ca4140990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 >> 172.21.15.119:0/1735676107 conn(0x7f0ca40b0980 msgr2=0x7f0ca415d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:21.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 shutdown_connections 2024-09-13T16:28:21.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.121+0000 7f0caa5b9700 1 -- 172.21.15.119:0/1735676107 wait complete. 2024-09-13T16:28:21.130 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 9 and data pool 10 2024-09-13T16:28:21.146 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:21.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d251af700 1 Processor -- start 2024-09-13T16:28:21.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d251af700 1 -- start start 2024-09-13T16:28:21.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d251af700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 0x7f9d200cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d251af700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9d2006bb90 0x7f9d200cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d251af700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2006cf60 con 0x7f9d200cde70 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d251af700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d200b3390 con 0x7f9d2006bb90 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d251af700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2006c350 con 0x7f9d200adb30 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d249ae700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9d200cde70 0x7f9d2015e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42460/0 (socket says 172.21.15.119:42460) 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d249ae700 1 -- 172.21.15.119:0/3308887322 learned_addr learned my addr 172.21.15.119:0/3308887322 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1f7fe700 1 --2- 172.21.15.119:0/3308887322 >> v2:172.21.15.119:3300/0 conn(0x7f9d2006bb90 0x7f9d200cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 0x7f9d200cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856601153 0 0) 0x7f9d2006cf60 con 0x7f9d200cde70 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d080035d0 con 0x7f9d200cde70 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1966027296 0 0) 0x7f9d080035d0 con 0x7f9d200cde70 2024-09-13T16:28:21.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 msgr2=0x7f9d200cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 0x7f9d200cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 >> v2:172.21.15.119:3300/0 conn(0x7f9d2006bb90 msgr2=0x7f9d200cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 --2- 172.21.15.119:0/3308887322 >> v2:172.21.15.119:3300/0 conn(0x7f9d2006bb90 0x7f9d200cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2006c630 con 0x7f9d200cde70 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f9d10001670 con 0x7f9d200cde70 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 >> v1:172.21.15.119:6789/0 conn(0x7f9d200cde70 legacy=0x7f9d2015e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d08003a20 0x7f9d08003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d08004870 0x7f9d08004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 0x7f9d08005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200b3390 con 0x7f9d08003a20 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2006c350 con 0x7f9d08004870 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0800fe40 con 0x7f9d200adb30 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1f7fe700 1 --2- 172.21.15.119:0/3308887322 >> v2:172.21.15.119:3300/0 conn(0x7f9d2006bb90 0x7f9d200cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:21.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.241+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d08003a20 0x7f9d08003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d249ae700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 0x7f9d08005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1f7fe700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d08004870 0x7f9d08004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1ffff700 1 -- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 msgr2=0x7f9d08005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 0x7f9d08005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1ffff700 1 -- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d08004870 msgr2=0x7f9d08004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d08004870 0x7f9d08004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1ffff700 1 -- 172.21.15.119:0/3308887322 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d080035d0 con 0x7f9d08003a20 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d249ae700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200adb30 0x7f9d08005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1f7fe700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d08004870 0x7f9d08004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d08003a20 0x7f9d08003e40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009fc0 tx=0x7f9d080058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d0af20f1f0856d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800ec30 con 0x7f9d08003a20 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1800f040 con 0x7f9d08003a20 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d1effd700 1 -- 172.21.15.119:0/3308887322 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18014480 con 0x7f9d08003a20 2024-09-13T16:28:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 -- 172.21.15.119:0/3308887322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d08003a20 msgr2=0x7f9d08003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 --2- 172.21.15.119:0/3308887322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d08003a20 0x7f9d08003e40 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f9d18009fc0 tx=0x7f9d080058a0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d249ae700 1 -- 172.21.15.119:0/3308887322 reap_dead start 2024-09-13T16:28:21.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 -- 172.21.15.119:0/3308887322 shutdown_connections 2024-09-13T16:28:21.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 -- 172.21.15.119:0/3308887322 >> 172.21.15.119:0/3308887322 conn(0x7f9d2006a350 msgr2=0x7f9d2006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:21.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 -- 172.21.15.119:0/3308887322 shutdown_connections 2024-09-13T16:28:21.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 -- 172.21.15.119:0/3308887322 wait complete. 2024-09-13T16:28:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 Processor -- start 2024-09-13T16:28:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.245+0000 7f9d251af700 1 -- start start 2024-09-13T16:28:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2012ca30 0x7f9d2015dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2012ce10 0x7f9d200ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200a84b0 0x7f9d200ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200a8a20 con 0x7f9d2012ce10 2024-09-13T16:28:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2012d340 con 0x7f9d2012ca30 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d2012d620 con 0x7f9d200a84b0 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2012ce10 0x7f9d200ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2012ce10 0x7f9d200ae360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50456/0 (socket says 172.21.15.119:50456) 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 -- 172.21.15.119:0/1054022421 learned_addr learned my addr 172.21.15.119:0/1054022421 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2012ca30 0x7f9d2015dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d249ae700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200a84b0 0x7f9d200ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 -- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200a84b0 msgr2=0x7f9d200ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200a84b0 0x7f9d200ae8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 -- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2012ca30 msgr2=0x7f9d2015dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2012ca30 0x7f9d2015dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 -- 172.21.15.119:0/1054022421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2012a7a0 con 0x7f9d2012ce10 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1f7fe700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2012ce10 0x7f9d200ae360 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9d14010c00 tx=0x7f9d14010f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1cff9700 1 -- 172.21.15.119:0/1054022421 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1400eab0 con 0x7f9d2012ce10 2024-09-13T16:28:21.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1cff9700 1 -- 172.21.15.119:0/1054022421 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1400ec10 con 0x7f9d2012ce10 2024-09-13T16:28:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d1cff9700 1 -- 172.21.15.119:0/1054022421 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1401c400 con 0x7f9d2012ce10 2024-09-13T16:28:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2012aa80 con 0x7f9d2012ce10 2024-09-13T16:28:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.249+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d201ed1c0 con 0x7f9d2012ce10 2024-09-13T16:28:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.253+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d2006bfd0 con 0x7f9d2012ce10 2024-09-13T16:28:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.253+0000 7f9d1cff9700 1 -- 172.21.15.119:0/1054022421 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9d140143f0 con 0x7f9d2012ce10 2024-09-13T16:28:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.253+0000 7f9d1cff9700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d0c066760 0x7f9d0c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.253+0000 7f9d1cff9700 1 -- 172.21.15.119:0/1054022421 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9d14084490 con 0x7f9d2012ce10 2024-09-13T16:28:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.253+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d0c066760 0x7f9d0c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.253+0000 7f9d1ffff700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d0c066760 0x7f9d0c068c20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9d2004ffd0 tx=0x7f9d18013000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.261+0000 7f9d1cff9700 1 -- 172.21.15.119:0/1054022421 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9d140515a0 con 0x7f9d2012ce10 2024-09-13T16:28:21.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.433+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d200caff0 con 0x7f9d2012ce10 2024-09-13T16:28:21.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d1cff9700 1 -- 172.21.15.119:0/1054022421 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f9d14054bc0 con 0x7f9d2012ce10 2024-09-13T16:28:21.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:21.440 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":67,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:28:21.112804+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:28:19.445072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-09-13T16:28:20.106803+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:21.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d0c066760 msgr2=0x7f9d0c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d0c066760 0x7f9d0c068c20 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f9d2004ffd0 tx=0x7f9d18013000 comp rx=0 tx=0).stop 2024-09-13T16:28:21.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2012ce10 msgr2=0x7f9d200ae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2012ce10 0x7f9d200ae360 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f9d14010c00 tx=0x7f9d14010f10 comp rx=0 tx=0).stop 2024-09-13T16:28:21.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 shutdown_connections 2024-09-13T16:28:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9d0c066760 0x7f9d0c068c20 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9d200a84b0 0x7f9d200ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d2012ce10 0x7f9d200ae360 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 --2- 172.21.15.119:0/1054022421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9d2012ca30 0x7f9d2015dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 >> 172.21.15.119:0/1054022421 conn(0x7f9d2006a350 msgr2=0x7f9d20140660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 shutdown_connections 2024-09-13T16:28:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.437+0000 7f9d251af700 1 -- 172.21.15.119:0/1054022421 wait complete. 2024-09-13T16:28:21.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:21.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.565+0000 7f629ef87700 1 Processor -- start 2024-09-13T16:28:21.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.565+0000 7f629ef87700 1 -- start start 2024-09-13T16:28:21.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.565+0000 7f629ef87700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62980acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62980ac5e0 0x7f6298070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62980ad270 con 0x7f62980cd9b0 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f629806eb90 con 0x7f62980ac5e0 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62980cced0 con 0x7f6298158e00 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629d784700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f62980cd9b0 0x7f62980abed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42472/0 (socket says 172.21.15.119:42472) 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629d784700 1 -- 172.21.15.119:0/907184036 learned_addr learned my addr 172.21.15.119:0/907184036 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629e786700 1 --2- 172.21.15.119:0/907184036 >> v2:172.21.15.119:3300/0 conn(0x7f62980ac5e0 0x7f6298070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629df85700 1 --2- 172.21.15.119:0/907184036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62980acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629cf83700 1 -- 172.21.15.119:0/907184036 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955254551 0 0) 0x7f62980ad270 con 0x7f62980cd9b0 2024-09-13T16:28:21.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629cf83700 1 -- 172.21.15.119:0/907184036 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62840035d0 con 0x7f62980cd9b0 2024-09-13T16:28:21.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629e786700 1 -- 172.21.15.119:0/907184036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 msgr2=0x7f62980acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:21.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629e786700 1 --2- 172.21.15.119:0/907184036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62980acd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629e786700 1 -- 172.21.15.119:0/907184036 >> v1:172.21.15.119:6789/0 conn(0x7f62980cd9b0 legacy=0x7f62980abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629e786700 1 -- 172.21.15.119:0/907184036 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62980cd1b0 con 0x7f62980ac5e0 2024-09-13T16:28:21.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629e786700 1 --2- 172.21.15.119:0/907184036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f6298070360 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f629000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c265cbda48e2294 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629cf83700 1 -- 172.21.15.119:0/907184036 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629000bb20 con 0x7f62980ac5e0 2024-09-13T16:28:21.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629cf83700 1 -- 172.21.15.119:0/907184036 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629000bc80 con 0x7f62980ac5e0 2024-09-13T16:28:21.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629cf83700 1 -- 172.21.15.119:0/907184036 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290011400 con 0x7f62980ac5e0 2024-09-13T16:28:21.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- 172.21.15.119:0/907184036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 msgr2=0x7f6298070360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 --2- 172.21.15.119:0/907184036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f6298070360 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f629000b330 comp rx=0 tx=0).stop 2024-09-13T16:28:21.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- 172.21.15.119:0/907184036 shutdown_connections 2024-09-13T16:28:21.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 --2- 172.21.15.119:0/907184036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f6298070360 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 --2- 172.21.15.119:0/907184036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62980acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- 172.21.15.119:0/907184036 >> 172.21.15.119:0/907184036 conn(0x7f62980ae4b0 msgr2=0x7f62980b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:21.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- 172.21.15.119:0/907184036 shutdown_connections 2024-09-13T16:28:21.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.569+0000 7f629ef87700 1 -- 172.21.15.119:0/907184036 wait complete. 2024-09-13T16:28:21.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 Processor -- start 2024-09-13T16:28:21.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 -- start start 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f62980cae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62980cd9b0 0x7f62980afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62981ed900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6298099df0 con 0x7f62980ac5e0 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629809a0d0 con 0x7f62980cd9b0 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629ef87700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629809a4b0 con 0x7f6298158e00 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f62980cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f62980cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50462/0 (socket says 172.21.15.119:50462) 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 -- 172.21.15.119:0/501429087 learned_addr learned my addr 172.21.15.119:0/501429087 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629d784700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62980cd9b0 0x7f62980afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629e786700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62981ed900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 -- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 msgr2=0x7f62981ed900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62981ed900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 -- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62980cd9b0 msgr2=0x7f62980afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62980cd9b0 0x7f62980afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 -- 172.21.15.119:0/501429087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290009a70 con 0x7f62980ac5e0 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.573+0000 7f629df85700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f62980cae10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f629400e840 tx=0x7f629400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f628e7fc700 1 -- 172.21.15.119:0/501429087 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62940095a0 con 0x7f62980ac5e0 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629809a790 con 0x7f62980ac5e0 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f628e7fc700 1 -- 172.21.15.119:0/501429087 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6294013680 con 0x7f62980ac5e0 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f628e7fc700 1 -- 172.21.15.119:0/501429087 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62940139f0 con 0x7f62980ac5e0 2024-09-13T16:28:21.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6298099a50 con 0x7f62980ac5e0 2024-09-13T16:28:21.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62980b18a0 con 0x7f62980ac5e0 2024-09-13T16:28:21.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f628e7fc700 1 -- 172.21.15.119:0/501429087 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6294013b50 con 0x7f62980ac5e0 2024-09-13T16:28:21.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f628e7fc700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f627c066730 0x7f627c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f628e7fc700 1 -- 172.21.15.119:0/501429087 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6294084170 con 0x7f62980ac5e0 2024-09-13T16:28:21.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.577+0000 7f629d784700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f627c066730 0x7f627c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.581+0000 7f629d784700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f627c066730 0x7f627c068bf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f62980ac470 tx=0x7f6288009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.585+0000 7f628e7fc700 1 -- 172.21.15.119:0/501429087 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6294051310 con 0x7f62980ac5e0 2024-09-13T16:28:21.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.757+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62980cfa30 con 0x7f62980ac5e0 2024-09-13T16:28:21.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.757+0000 7f628e7fc700 1 -- 172.21.15.119:0/501429087 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f6294054930 con 0x7f62980ac5e0 2024-09-13T16:28:21.762 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:21.763 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":67,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:28:21.112804+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:28:19.445072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-09-13T16:28:20.106803+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f627c066730 msgr2=0x7f627c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f627c066730 0x7f627c068bf0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f62980ac470 tx=0x7f6288009040 comp rx=0 tx=0).stop 2024-09-13T16:28:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 msgr2=0x7f62980cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f62980cae10 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f629400e840 tx=0x7f629400eb50 comp rx=0 tx=0).stop 2024-09-13T16:28:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 shutdown_connections 2024-09-13T16:28:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f627c066730 0x7f627c068bf0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6298158e00 0x7f62981ed900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62980cd9b0 0x7f62980afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 --2- 172.21.15.119:0/501429087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62980ac5e0 0x7f62980cae10 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 >> 172.21.15.119:0/501429087 conn(0x7f62980ae4b0 msgr2=0x7f6298073ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:21.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 shutdown_connections 2024-09-13T16:28:21.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.761+0000 7f629ef87700 1 -- 172.21.15.119:0/501429087 wait complete. 2024-09-13T16:28:21.781 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:28:21.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3d427700 1 Processor -- start 2024-09-13T16:28:21.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3d427700 1 -- start start 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3d427700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7a38071010 0x7f7a380713f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3d427700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806cbc0 0x7f7a380cd100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3d427700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a380b2830 con 0x7f7a3806c780 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3d427700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a38071930 con 0x7f7a38071010 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3d427700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a380b1c20 con 0x7f7a3806cbc0 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a377fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7a3806c780 0x7f7a3806fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42488/0 (socket says 172.21.15.119:42488) 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a377fe700 1 -- 172.21.15.119:0/3294510596 learned_addr learned my addr 172.21.15.119:0/3294510596 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:21.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a37fff700 1 --2- 172.21.15.119:0/3294510596 >> v2:172.21.15.145:3300/0 conn(0x7f7a38071010 0x7f7a380713f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3cc26700 1 --2- 172.21.15.119:0/3294510596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806cbc0 0x7f7a380cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a36ffd700 1 -- 172.21.15.119:0/3294510596 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988234131 0 0) 0x7f7a380b2830 con 0x7f7a3806c780 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a36ffd700 1 -- 172.21.15.119:0/3294510596 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a240035d0 con 0x7f7a3806c780 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3cc26700 1 -- 172.21.15.119:0/3294510596 >> v2:172.21.15.145:3300/0 conn(0x7f7a38071010 msgr2=0x7f7a380713f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3cc26700 1 --2- 172.21.15.119:0/3294510596 >> v2:172.21.15.145:3300/0 conn(0x7f7a38071010 0x7f7a380713f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3cc26700 1 -- 172.21.15.119:0/3294510596 >> v1:172.21.15.119:6789/0 conn(0x7f7a3806c780 legacy=0x7f7a3806fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.901+0000 7f7a3cc26700 1 -- 172.21.15.119:0/3294510596 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a380b1f00 con 0x7f7a3806cbc0 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3cc26700 1 --2- 172.21.15.119:0/3294510596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806cbc0 0x7f7a380cd100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009850 tx=0x7f7a30009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3eb7a26e195b7f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a36ffd700 1 -- 172.21.15.119:0/3294510596 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30011830 con 0x7f7a3806cbc0 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a36ffd700 1 -- 172.21.15.119:0/3294510596 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30011990 con 0x7f7a3806cbc0 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a36ffd700 1 -- 172.21.15.119:0/3294510596 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a30011d40 con 0x7f7a3806cbc0 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 -- 172.21.15.119:0/3294510596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806cbc0 msgr2=0x7f7a380cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 --2- 172.21.15.119:0/3294510596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806cbc0 0x7f7a380cd100 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f7a30009850 tx=0x7f7a30009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 -- 172.21.15.119:0/3294510596 shutdown_connections 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 --2- 172.21.15.119:0/3294510596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806cbc0 0x7f7a380cd100 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 --2- 172.21.15.119:0/3294510596 >> v2:172.21.15.145:3300/0 conn(0x7f7a38071010 0x7f7a380713f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 -- 172.21.15.119:0/3294510596 >> 172.21.15.119:0/3294510596 conn(0x7f7a380b0980 msgr2=0x7f7a380b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:21.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 -- 172.21.15.119:0/3294510596 shutdown_connections 2024-09-13T16:28:21.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 -- 172.21.15.119:0/3294510596 wait complete. 2024-09-13T16:28:21.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 Processor -- start 2024-09-13T16:28:21.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.905+0000 7f7a3d427700 1 -- start start 2024-09-13T16:28:21.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806c780 0x7f7a380ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a3806cbc0 0x7f7a38144ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a38071010 0x7f7a3814a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a381424d0 con 0x7f7a38071010 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38148350 con 0x7f7a3806cbc0 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a38148630 con 0x7f7a3806c780 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3cc26700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a38071010 0x7f7a3814a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a377fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a3806cbc0 0x7f7a38144ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3cc26700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a38071010 0x7f7a3814a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50466/0 (socket says 172.21.15.119:50466) 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3cc26700 1 -- 172.21.15.119:0/67540763 learned_addr learned my addr 172.21.15.119:0/67540763 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a37fff700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806c780 0x7f7a380ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a37fff700 1 -- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a3806cbc0 msgr2=0x7f7a38144ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a37fff700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a3806cbc0 0x7f7a38144ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a37fff700 1 -- 172.21.15.119:0/67540763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a38071010 msgr2=0x7f7a3814a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a37fff700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a38071010 0x7f7a3814a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a37fff700 1 -- 172.21.15.119:0/67540763 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300094b0 con 0x7f7a3806c780 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3cc26700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a38071010 0x7f7a3814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a37fff700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806c780 0x7f7a380ad9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800e870 tx=0x7f7a2800eb80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a34ff9700 1 -- 172.21.15.119:0/67540763 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28013680 con 0x7f7a3806c780 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a34ff9700 1 -- 172.21.15.119:0/67540763 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a280137e0 con 0x7f7a3806c780 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a381446d0 con 0x7f7a3806c780 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a34ff9700 1 -- 172.21.15.119:0/67540763 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a28013b70 con 0x7f7a3806c780 2024-09-13T16:28:21.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.909+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3814a630 con 0x7f7a3806c780 2024-09-13T16:28:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.913+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a38148850 con 0x7f7a3806c780 2024-09-13T16:28:21.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.913+0000 7f7a34ff9700 1 -- 172.21.15.119:0/67540763 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7a28013cd0 con 0x7f7a3806c780 2024-09-13T16:28:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.913+0000 7f7a34ff9700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a18066730 0x7f7a18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.913+0000 7f7a34ff9700 1 -- 172.21.15.119:0/67540763 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7a28084820 con 0x7f7a3806c780 2024-09-13T16:28:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.913+0000 7f7a377fe700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a18066730 0x7f7a18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:21.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.913+0000 7f7a377fe700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a18066730 0x7f7a18068bf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a38165a60 tx=0x7f7a20001a90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:21.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:21.925+0000 7f7a34ff9700 1 -- 172.21.15.119:0/67540763 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7a28051ae0 con 0x7f7a3806c780 2024-09-13T16:28:22.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:22.133+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f7a380713f0 con 0x7f7a3806c780 2024-09-13T16:28:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.129+0000 7f7a34ff9700 1 -- 172.21.15.119:0/67540763 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7f7a280210b0 con 0x7f7a3806c780 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.133+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a18066730 msgr2=0x7f7a18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.133+0000 7f7a3d427700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a18066730 0x7f7a18068bf0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7a38165a60 tx=0x7f7a20001a90 comp rx=0 tx=0).stop 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.133+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806c780 msgr2=0x7f7a380ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.133+0000 7f7a3d427700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806c780 0x7f7a380ad9c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a2800e870 tx=0x7f7a2800eb80 comp rx=0 tx=0).stop 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 shutdown_connections 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a18066730 0x7f7a18068bf0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7a38071010 0x7f7a3814a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7a3806cbc0 0x7f7a38144ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 --2- 172.21.15.119:0/67540763 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7a3806c780 0x7f7a380ad9c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 >> 172.21.15.119:0/67540763 conn(0x7f7a380b0980 msgr2=0x7f7a38158120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:23.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 shutdown_connections 2024-09-13T16:28:23.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.137+0000 7f7a3d427700 1 -- 172.21.15.119:0/67540763 wait complete. 2024-09-13T16:28:23.155 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T16:28:23.155 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:28:23.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.277+0000 7f07263cf700 1 Processor -- start 2024-09-13T16:28:23.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.277+0000 7f07263cf700 1 -- start start 2024-09-13T16:28:23.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f072015cc00 0x7f07200d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:23.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f07200d17e0 0x7f07200b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:23.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07200cdb00 con 0x7f07200d0c20 2024-09-13T16:28:23.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07200cdde0 con 0x7f07200d17e0 2024-09-13T16:28:23.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07200ce0c0 con 0x7f072015cc00 2024-09-13T16:28:23.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0725bce700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f07200d17e0 0x7f07200b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:23.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0724bcc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f07200d0c20 0x7f07200d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48908/0 (socket says 172.21.15.119:48908) 2024-09-13T16:28:23.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0724bcc700 1 -- 172.21.15.119:0/3873732724 learned_addr learned my addr 172.21.15.119:0/3873732724 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:23.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0725bce700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f07200d17e0 0x7f07200b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33656/0 (socket says 172.21.15.119:33656) 2024-09-13T16:28:23.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07253cd700 1 --2- 172.21.15.119:0/3873732724 >> v2:172.21.15.145:3300/0 conn(0x7f072015cc00 0x7f07200d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:23.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0717fff700 1 -- 172.21.15.119:0/3873732724 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065625236 0 0) 0x7f07200cdb00 con 0x7f07200d0c20 2024-09-13T16:28:23.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0725bce700 1 -- 172.21.15.119:0/3873732724 >> v2:172.21.15.145:3300/0 conn(0x7f072015cc00 msgr2=0x7f07200d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:23.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0725bce700 1 --2- 172.21.15.119:0/3873732724 >> v2:172.21.15.145:3300/0 conn(0x7f072015cc00 0x7f07200d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0725bce700 1 -- 172.21.15.119:0/3873732724 >> v1:172.21.15.119:6789/0 conn(0x7f07200d0c20 legacy=0x7f07200d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:23.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0725bce700 1 -- 172.21.15.119:0/3873732724 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07200ce3a0 con 0x7f07200d17e0 2024-09-13T16:28:23.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0725bce700 1 --2- 172.21.15.119:0/3873732724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07200d17e0 0x7f07200b5960 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f07200b5610 tx=0x7f071800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13ba16daed7d0231 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:23.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0717fff700 1 -- 172.21.15.119:0/3873732724 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f071800e070 con 0x7f07200d17e0 2024-09-13T16:28:23.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0717fff700 1 -- 172.21.15.119:0/3873732724 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071800bb20 con 0x7f07200d17e0 2024-09-13T16:28:23.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f0717fff700 1 -- 172.21.15.119:0/3873732724 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718012430 con 0x7f07200d17e0 2024-09-13T16:28:23.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- 172.21.15.119:0/3873732724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07200d17e0 msgr2=0x7f07200b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:23.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 --2- 172.21.15.119:0/3873732724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07200d17e0 0x7f07200b5960 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f07200b5610 tx=0x7f071800b330 comp rx=0 tx=0).stop 2024-09-13T16:28:23.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- 172.21.15.119:0/3873732724 shutdown_connections 2024-09-13T16:28:23.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 --2- 172.21.15.119:0/3873732724 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07200d17e0 0x7f07200b5960 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 --2- 172.21.15.119:0/3873732724 >> v2:172.21.15.145:3300/0 conn(0x7f072015cc00 0x7f07200d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- 172.21.15.119:0/3873732724 >> 172.21.15.119:0/3873732724 conn(0x7f072008c080 msgr2=0x7f07200d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:23.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- 172.21.15.119:0/3873732724 shutdown_connections 2024-09-13T16:28:23.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.281+0000 7f07263cf700 1 -- 172.21.15.119:0/3873732724 wait complete. 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.285+0000 7f07263cf700 1 Processor -- start 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.285+0000 7f07263cf700 1 -- start start 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07200d0c20 0x7f0720154f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07200d17e0 0x7f0720159d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f072015cc00 0x7f0720156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072015a2a0 con 0x7f072015cc00 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072015dd50 con 0x7f07200d0c20 2024-09-13T16:28:23.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0720156c70 con 0x7f07200d17e0 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f072015cc00 0x7f0720156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f072015cc00 0x7f0720156290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33658/0 (socket says 172.21.15.119:33658) 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 -- 172.21.15.119:0/1082875241 learned_addr learned my addr 172.21.15.119:0/1082875241 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07253cd700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07200d0c20 0x7f0720154f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0724bcc700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07200d17e0 0x7f0720159d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 -- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07200d17e0 msgr2=0x7f0720159d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07200d17e0 0x7f0720159d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 -- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07200d0c20 msgr2=0x7f0720154f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:23.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07200d0c20 0x7f0720154f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:23.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 -- 172.21.15.119:0/1082875241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0718009a70 con 0x7f072015cc00 2024-09-13T16:28:23.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0725bce700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f072015cc00 0x7f0720156290 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f0718013000 tx=0x7f0718004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:23.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0715ffb700 1 -- 172.21.15.119:0/1082875241 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718012430 con 0x7f072015cc00 2024-09-13T16:28:23.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0715ffb700 1 -- 172.21.15.119:0/1082875241 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0718020070 con 0x7f072015cc00 2024-09-13T16:28:23.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072015d700 con 0x7f072015cc00 2024-09-13T16:28:23.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f0715ffb700 1 -- 172.21.15.119:0/1082875241 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0718004280 con 0x7f072015cc00 2024-09-13T16:28:23.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07201603f0 con 0x7f072015cc00 2024-09-13T16:28:23.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.289+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072015de90 con 0x7f072015cc00 2024-09-13T16:28:23.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.293+0000 7f0715ffb700 1 -- 172.21.15.119:0/1082875241 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f07180043e0 con 0x7f072015cc00 2024-09-13T16:28:23.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.293+0000 7f0715ffb700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0704066730 0x7f0704068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:23.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.293+0000 7f0715ffb700 1 -- 172.21.15.119:0/1082875241 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0718022070 con 0x7f072015cc00 2024-09-13T16:28:23.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.293+0000 7f07253cd700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0704066730 0x7f0704068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:23.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.293+0000 7f07253cd700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0704066730 0x7f0704068bf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0710009b50 tx=0x7f0710009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:23.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.301+0000 7f0715ffb700 1 -- 172.21.15.119:0/1082875241 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0718053040 con 0x7f072015cc00 2024-09-13T16:28:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:23.509+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f07200cf090 con 0x7f072015cc00 2024-09-13T16:28:24.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.137+0000 7f0715ffb700 1 -- 172.21.15.119:0/1082875241 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f071802c0c0 con 0x7f072015cc00 2024-09-13T16:28:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0704066730 msgr2=0x7f0704068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0704066730 0x7f0704068bf0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0710009b50 tx=0x7f0710009240 comp rx=0 tx=0).stop 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f072015cc00 msgr2=0x7f0720156290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f072015cc00 0x7f0720156290 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f0718013000 tx=0x7f0718004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 shutdown_connections 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0704066730 0x7f0704068bf0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f072015cc00 0x7f0720156290 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07200d17e0 0x7f0720159d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 --2- 172.21.15.119:0/1082875241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07200d0c20 0x7f0720154f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 >> 172.21.15.119:0/1082875241 conn(0x7f072008c080 msgr2=0x7f07200b6290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 shutdown_connections 2024-09-13T16:28:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.141+0000 7f07263cf700 1 -- 172.21.15.119:0/1082875241 wait complete. 2024-09-13T16:28:24.163 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:28:24.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550f59e700 1 Processor -- start 2024-09-13T16:28:24.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550f59e700 1 -- start start 2024-09-13T16:28:24.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550f59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f55100cd860 0x7f551008bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550f59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f551015d020 con 0x7f551008c4e0 2024-09-13T16:28:24.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550f59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551008c990 con 0x7f55100cd860 2024-09-13T16:28:24.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55100b44c0 con 0x7f551015cc40 2024-09-13T16:28:24.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550ed9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f551008c4e0 0x7f5510160720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48912/0 (socket says 172.21.15.119:48912) 2024-09-13T16:28:24.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550ed9d700 1 -- 172.21.15.119:0/1922058479 learned_addr learned my addr 172.21.15.119:0/1922058479 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:24.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550dd9b700 1 --2- 172.21.15.119:0/1922058479 >> v2:172.21.15.145:3300/0 conn(0x7f55100cd860 0x7f551008bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550e59c700 1 --2- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550d59a700 1 -- 172.21.15.119:0/1922058479 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189011005 0 0) 0x7f551015d020 con 0x7f551008c4e0 2024-09-13T16:28:24.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550d59a700 1 -- 172.21.15.119:0/1922058479 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54f80035d0 con 0x7f551008c4e0 2024-09-13T16:28:24.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550dd9b700 1 -- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 msgr2=0x7f55100ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:24.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550dd9b700 1 --2- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550dd9b700 1 -- 172.21.15.119:0/1922058479 >> v1:172.21.15.119:6789/0 conn(0x7f551008c4e0 legacy=0x7f5510160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.277+0000 7f550dd9b700 1 -- 172.21.15.119:0/1922058479 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100d0720 con 0x7f55100cd860 2024-09-13T16:28:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550dd9b700 1 --2- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 0x7f551008bfa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5500009880 tx=0x7f5500009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f62f7a477568df6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550d59a700 1 -- 172.21.15.119:0/1922058479 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500011830 con 0x7f55100cd860 2024-09-13T16:28:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550d59a700 1 -- 172.21.15.119:0/1922058479 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5500011990 con 0x7f55100cd860 2024-09-13T16:28:24.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550d59a700 1 -- 172.21.15.119:0/1922058479 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5500011d20 con 0x7f55100cd860 2024-09-13T16:28:24.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 -- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 msgr2=0x7f551008bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 --2- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 0x7f551008bfa0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5500009880 tx=0x7f5500009b90 comp rx=0 tx=0).stop 2024-09-13T16:28:24.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 -- 172.21.15.119:0/1922058479 shutdown_connections 2024-09-13T16:28:24.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 --2- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 0x7f551008bfa0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 --2- 172.21.15.119:0/1922058479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 -- 172.21.15.119:0/1922058479 >> 172.21.15.119:0/1922058479 conn(0x7f55100b4640 msgr2=0x7f55100b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:24.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 -- 172.21.15.119:0/1922058479 shutdown_connections 2024-09-13T16:28:24.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 -- 172.21.15.119:0/1922058479 wait complete. 2024-09-13T16:28:24.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.281+0000 7f550f59e700 1 Processor -- start 2024-09-13T16:28:24.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 -- start start 2024-09-13T16:28:24.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551008c4e0 0x7f55100b6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 0x7f551015d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100b6970 con 0x7f551008c4e0 2024-09-13T16:28:24.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55100b7b30 con 0x7f55100cd860 2024-09-13T16:28:24.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55101577d0 con 0x7f551015cc40 2024-09-13T16:28:24.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551008c4e0 0x7f55100b6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551008c4e0 0x7f55100b6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33662/0 (socket says 172.21.15.119:33662) 2024-09-13T16:28:24.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 -- 172.21.15.119:0/1554812519 learned_addr learned my addr 172.21.15.119:0/1554812519 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:24.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550ed9d700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550dd9b700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 0x7f551015d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 -- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 msgr2=0x7f55100c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 -- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 msgr2=0x7f551015d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 0x7f551015d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 -- 172.21.15.119:0/1554812519 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55000094b0 con 0x7f551008c4e0 2024-09-13T16:28:24.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550e59c700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551008c4e0 0x7f55100b6390 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f550400e9d0 tx=0x7f550400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f54feffd700 1 -- 172.21.15.119:0/1554812519 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5504009720 con 0x7f551008c4e0 2024-09-13T16:28:24.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f54feffd700 1 -- 172.21.15.119:0/1554812519 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550400ee70 con 0x7f551008c4e0 2024-09-13T16:28:24.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55100b7200 con 0x7f551008c4e0 2024-09-13T16:28:24.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f54feffd700 1 -- 172.21.15.119:0/1554812519 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550400ca50 con 0x7f551008c4e0 2024-09-13T16:28:24.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.285+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55100b77c0 con 0x7f551008c4e0 2024-09-13T16:28:24.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.289+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55100d1540 con 0x7f551008c4e0 2024-09-13T16:28:24.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.289+0000 7f54feffd700 1 -- 172.21.15.119:0/1554812519 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f550400cbb0 con 0x7f551008c4e0 2024-09-13T16:28:24.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.289+0000 7f54feffd700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f54f0066730 0x7f54f0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.289+0000 7f54feffd700 1 -- 172.21.15.119:0/1554812519 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5504084240 con 0x7f551008c4e0 2024-09-13T16:28:24.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.289+0000 7f550dd9b700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f54f0066730 0x7f54f0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.289+0000 7f550dd9b700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f54f0066730 0x7f54f0068bf0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5500009880 tx=0x7f5500008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.297+0000 7f54feffd700 1 -- 172.21.15.119:0/1554812519 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5504051470 con 0x7f551008c4e0 2024-09-13T16:28:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.501+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f55100cea30 con 0x7f551008c4e0 2024-09-13T16:28:24.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.501+0000 7f54feffd700 1 -- 172.21.15.119:0/1554812519 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f550400ce60 con 0x7f551008c4e0 2024-09-13T16:28:24.506 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:24.506 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:28:24.129975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:28:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f54f0066730 msgr2=0x7f54f0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f54f0066730 0x7f54f0068bf0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5500009880 tx=0x7f5500008030 comp rx=0 tx=0).stop 2024-09-13T16:28:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551008c4e0 msgr2=0x7f55100b6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551008c4e0 0x7f55100b6390 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f550400e9d0 tx=0x7f550400ece0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 shutdown_connections 2024-09-13T16:28:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f54f0066730 0x7f54f0068bf0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f551015cc40 0x7f55100c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f55100cd860 0x7f551015d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 --2- 172.21.15.119:0/1554812519 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f551008c4e0 0x7f55100b6390 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 >> 172.21.15.119:0/1554812519 conn(0x7f55100b4640 msgr2=0x7f55100b2bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 shutdown_connections 2024-09-13T16:28:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.505+0000 7f550f59e700 1 -- 172.21.15.119:0/1554812519 wait complete. 2024-09-13T16:28:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 32 2024-09-13T16:28:24.520 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:28:24.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa863fff700 1 Processor -- start 2024-09-13T16:28:24.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa863fff700 1 -- start start 2024-09-13T16:28:24.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa863fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa8641570d0 0x7fa86406b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa863fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa86406e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa863fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa86406ebf0 con 0x7fa8640d0f70 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa863fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8640ce730 con 0x7fa8641570d0 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa863fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640cf840 con 0x7fa86406e270 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa8641570d0 0x7fa86406b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa8641570d0 0x7fa86406b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33672/0 (socket says 172.21.15.119:33672) 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 -- 172.21.15.119:0/3391169878 learned_addr learned my addr 172.21.15.119:0/3391169878 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa8627fc700 1 --2- 172.21.15.119:0/3391169878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa86406e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 -- 172.21.15.119:0/3391169878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 msgr2=0x7fa86406e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 --2- 172.21.15.119:0/3391169878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa86406e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 -- 172.21.15.119:0/3391169878 >> v1:172.21.15.145:6789/0 conn(0x7fa8640d0f70 legacy=0x7fa8640d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 -- 172.21.15.119:0/3391169878 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8640cfb20 con 0x7fa8641570d0 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa862ffd700 1 --2- 172.21.15.119:0/3391169878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8641570d0 0x7fa86406b930 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa854009e50 tx=0x7fa85400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45205bec8a65e55e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.629+0000 7fa861ffb700 1 -- 172.21.15.119:0/3391169878 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85400e070 con 0x7fa8641570d0 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa861ffb700 1 -- 172.21.15.119:0/3391169878 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85400b9c0 con 0x7fa8641570d0 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa861ffb700 1 -- 172.21.15.119:0/3391169878 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85400bd70 con 0x7fa8641570d0 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 -- 172.21.15.119:0/3391169878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8641570d0 msgr2=0x7fa86406b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 --2- 172.21.15.119:0/3391169878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8641570d0 0x7fa86406b930 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa854009e50 tx=0x7fa85400b330 comp rx=0 tx=0).stop 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 -- 172.21.15.119:0/3391169878 shutdown_connections 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 --2- 172.21.15.119:0/3391169878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa86406e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 --2- 172.21.15.119:0/3391169878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8641570d0 0x7fa86406b930 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 -- 172.21.15.119:0/3391169878 >> 172.21.15.119:0/3391169878 conn(0x7fa864099a50 msgr2=0x7fa864099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 -- 172.21.15.119:0/3391169878 shutdown_connections 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 -- 172.21.15.119:0/3391169878 wait complete. 2024-09-13T16:28:24.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 Processor -- start 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.633+0000 7fa863fff700 1 -- start start 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa8640b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8640d0f70 0x7fa86414b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8641570d0 0x7fa86414b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640b2f10 con 0x7fa8640d0f70 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86414ab60 con 0x7fa8641570d0 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8640b1930 con 0x7fa86406e270 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8640d0f70 0x7fa86414b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8640d0f70 0x7fa86414b000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33688/0 (socket says 172.21.15.119:33688) 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 -- 172.21.15.119:0/2793488629 learned_addr learned my addr 172.21.15.119:0/2793488629 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa862ffd700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa8640b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8637fe700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8641570d0 0x7fa86414b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 -- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 msgr2=0x7fa8640b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa8640b29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 -- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8641570d0 msgr2=0x7fa86414b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8641570d0 0x7fa86414b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 -- 172.21.15.119:0/2793488629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa854009b30 con 0x7fa8640d0f70 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8637fe700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8641570d0 0x7fa86414b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa8627fc700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8640d0f70 0x7fa86414b000 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00d800 tx=0x7fa84c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa85b7fe700 1 -- 172.21.15.119:0/2793488629 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c0125f0 con 0x7fa8640d0f70 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa85b7fe700 1 -- 172.21.15.119:0/2793488629 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa84c012750 con 0x7fa8640d0f70 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa85b7fe700 1 -- 172.21.15.119:0/2793488629 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c012ac0 con 0x7fa8640d0f70 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8640b1c10 con 0x7fa8640d0f70 2024-09-13T16:28:24.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.637+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8640b21d0 con 0x7fa8640d0f70 2024-09-13T16:28:24.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.641+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86406e6b0 con 0x7fa8640d0f70 2024-09-13T16:28:24.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.641+0000 7fa85b7fe700 1 -- 172.21.15.119:0/2793488629 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa84c012c20 con 0x7fa8640d0f70 2024-09-13T16:28:24.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.641+0000 7fa85b7fe700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa844066730 0x7fa844068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.641+0000 7fa85b7fe700 1 -- 172.21.15.119:0/2793488629 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa84c0843b0 con 0x7fa8640d0f70 2024-09-13T16:28:24.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.641+0000 7fa862ffd700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa844066730 0x7fa844068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.641+0000 7fa862ffd700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa844066730 0x7fa844068bf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa85400b370 tx=0x7fa854009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.645+0000 7fa85b7fe700 1 -- 172.21.15.119:0/2793488629 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa84c0515e0 con 0x7fa8640d0f70 2024-09-13T16:28:24.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.821+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8640adcb0 con 0x7fa8640d0f70 2024-09-13T16:28:24.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.821+0000 7fa85b7fe700 1 -- 172.21.15.119:0/2793488629 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+16776 (secure 0 0 0) 0x7fa84c054c00 con 0x7fa8640d0f70 2024-09-13T16:28:24.827 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:24.828 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":68,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:28:22.116881+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:28:19.445072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-09-13T16:28:20.106803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:28:24.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa844066730 msgr2=0x7fa844068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa844066730 0x7fa844068bf0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fa85400b370 tx=0x7fa854009000 comp rx=0 tx=0).stop 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8640d0f70 msgr2=0x7fa86414b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8640d0f70 0x7fa86414b000 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00d800 tx=0x7fa84c00db10 comp rx=0 tx=0).stop 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 shutdown_connections 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa844066730 0x7fa844068bf0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8641570d0 0x7fa86414b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa8640d0f70 0x7fa86414b000 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 --2- 172.21.15.119:0/2793488629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa86406e270 0x7fa8640b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 >> 172.21.15.119:0/2793488629 conn(0x7fa864099a50 msgr2=0x7fa8640b06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 shutdown_connections 2024-09-13T16:28:24.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.825+0000 7fa863fff700 1 -- 172.21.15.119:0/2793488629 wait complete. 2024-09-13T16:28:24.842 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:28:24.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:28:24.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.949+0000 7f71ea62c700 1 Processor -- start 2024-09-13T16:28:24.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.949+0000 7f71ea62c700 1 -- start start 2024-09-13T16:28:24.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71ea62c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 0x7f71e40ada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71ea62c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f71e40adf40 0x7f71e406fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71ea62c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e40ac960 con 0x7f71e415a620 2024-09-13T16:28:24.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71ea62c700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e40acc40 con 0x7f71e40adf40 2024-09-13T16:28:24.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71ea62c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e40acf20 con 0x7f71e41612f0 2024-09-13T16:28:24.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f71e415a620 0x7f71e40b3720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48920/0 (socket says 172.21.15.119:48920) 2024-09-13T16:28:24.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 -- 172.21.15.119:0/896472111 learned_addr learned my addr 172.21.15.119:0/896472111 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:24.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e37fe700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 0x7f71e40ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e8bc9700 1 --2- 172.21.15.119:0/896472111 >> v2:172.21.15.145:3300/0 conn(0x7f71e40adf40 0x7f71e406fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1238416323 0 0) 0x7f71e40ac960 con 0x7f71e415a620 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71d00035d0 con 0x7f71e415a620 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1602937642 0 0) 0x7f71d00035d0 con 0x7f71e415a620 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 msgr2=0x7f71e40ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 0x7f71e40ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 >> v2:172.21.15.145:3300/0 conn(0x7f71e40adf40 msgr2=0x7f71e406fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 --2- 172.21.15.119:0/896472111 >> v2:172.21.15.145:3300/0 conn(0x7f71e40adf40 0x7f71e406fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e40ad200 con 0x7f71e415a620 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f71cc001670 con 0x7f71e415a620 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 >> v1:172.21.15.119:6789/0 conn(0x7f71e415a620 legacy=0x7f71e40b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d0003a20 0x7f71d0003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0004870 0x7f71d0004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 0x7f71d0005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e40acc40 con 0x7f71d0003a20 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e40acf20 con 0x7f71d0004870 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d000ba60 con 0x7f71e41612f0 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d0003a20 0x7f71d0003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e37fe700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0004870 0x7f71d0004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e8bc9700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 0x7f71d0005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 -- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 msgr2=0x7f71d0005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 0x7f71d0005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 -- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0004870 msgr2=0x7f71d0004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0004870 0x7f71d0004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 -- 172.21.15.119:0/896472111 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00035d0 con 0x7f71d0003a20 2024-09-13T16:28:24.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e37fe700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0004870 0x7f71d0004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e8bc9700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e41612f0 0x7f71d0005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e3fff700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d0003a20 0x7f71d0003e40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00c2f0 tx=0x7f71d00058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a905990bb8f1da63 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc007cb0 con 0x7f71d0003a20 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc00ce30 con 0x7f71d0003a20 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71e2ffd700 1 -- 172.21.15.119:0/896472111 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71cc005bc0 con 0x7f71d0003a20 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71ea62c700 1 -- 172.21.15.119:0/896472111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d0003a20 msgr2=0x7f71d0003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.953+0000 7f71ea62c700 1 --2- 172.21.15.119:0/896472111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71d0003a20 0x7f71d0003e40 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f71cc00c2f0 tx=0x7f71d00058a0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.957+0000 7f71e8bc9700 1 -- 172.21.15.119:0/896472111 reap_dead start 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.957+0000 7f71ea62c700 1 -- 172.21.15.119:0/896472111 shutdown_connections 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.957+0000 7f71ea62c700 1 -- 172.21.15.119:0/896472111 >> 172.21.15.119:0/896472111 conn(0x7f71e406a350 msgr2=0x7f71e406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.957+0000 7f71ea62c700 1 -- 172.21.15.119:0/896472111 shutdown_connections 2024-09-13T16:28:24.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.957+0000 7f71ea62c700 1 -- 172.21.15.119:0/896472111 wait complete. 2024-09-13T16:28:24.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.957+0000 7f71ea62c700 1 Processor -- start 2024-09-13T16:28:24.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.957+0000 7f71ea62c700 1 -- start start 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0003a20 0x7f71e41f1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e4135700 0x7f71e4135b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e415f000 0x7f71e415f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e415f9c0 con 0x7f71e4135700 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4136210 con 0x7f71d0003a20 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e41364f0 con 0x7f71e415f000 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e4135700 0x7f71e4135b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e4135700 0x7f71e4135b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33714/0 (socket says 172.21.15.119:33714) 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 -- 172.21.15.119:0/3682855798 learned_addr learned my addr 172.21.15.119:0/3682855798 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e3fff700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0003a20 0x7f71e41f1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e8bc9700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e415f000 0x7f71e415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 -- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e415f000 msgr2=0x7f71e415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e415f000 0x7f71e415f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 -- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0003a20 msgr2=0x7f71e41f1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0003a20 0x7f71e41f1bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 -- 172.21.15.119:0/3682855798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e41367d0 con 0x7f71e4135700 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e37fe700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e4135700 0x7f71e4135b80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f71d40108c0 tx=0x7f71d4010bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e0ff9700 1 -- 172.21.15.119:0/3682855798 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4004020 con 0x7f71e4135700 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e0ff9700 1 -- 172.21.15.119:0/3682855798 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d4004180 con 0x7f71e4135700 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71e0ff9700 1 -- 172.21.15.119:0/3682855798 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71d4004510 con 0x7f71e4135700 2024-09-13T16:28:24.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e4136ab0 con 0x7f71e4135700 2024-09-13T16:28:24.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.961+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e41f2100 con 0x7f71e4135700 2024-09-13T16:28:24.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.965+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e4067760 con 0x7f71e4135700 2024-09-13T16:28:24.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.965+0000 7f71e0ff9700 1 -- 172.21.15.119:0/3682855798 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f71d40036c0 con 0x7f71e4135700 2024-09-13T16:28:24.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.965+0000 7f71e0ff9700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71c4066730 0x7f71c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:24.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.965+0000 7f71e0ff9700 1 -- 172.21.15.119:0/3682855798 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f71d4084460 con 0x7f71e4135700 2024-09-13T16:28:24.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.965+0000 7f71e3fff700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71c4066730 0x7f71c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:24.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.965+0000 7f71e3fff700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71c4066730 0x7f71c4068bf0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f71cc015040 tx=0x7f71cc00a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:24.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:24.973+0000 7f71e0ff9700 1 -- 172.21.15.119:0/3682855798 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f71d4051680 con 0x7f71e4135700 2024-09-13T16:28:25.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.141+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f71e415aa00 con 0x7f71c4066730 2024-09-13T16:28:25.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.153+0000 7f71e0ff9700 1 -- 172.21.15.119:0/3682855798 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276877 (secure 0 0 0) 0x7f71cc001430 con 0x7f71c4066730 2024-09-13T16:28:25.164 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:25.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.157+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71c4066730 msgr2=0x7f71c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.157+0000 7f71ea62c700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71c4066730 0x7f71c4068bf0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f71cc015040 tx=0x7f71cc00a000 comp rx=0 tx=0).stop 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.157+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e4135700 msgr2=0x7f71e4135b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.157+0000 7f71ea62c700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e4135700 0x7f71e4135b80 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f71d40108c0 tx=0x7f71d4010bd0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 shutdown_connections 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71c4066730 0x7f71c4068bf0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f71e415f000 0x7f71e415f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71e4135700 0x7f71e4135b80 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 --2- 172.21.15.119:0/3682855798 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f71d0003a20 0x7f71e41f1bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 >> 172.21.15.119:0/3682855798 conn(0x7f71e406a350 msgr2=0x7f71e40cd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 shutdown_connections 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.161+0000 7f71ea62c700 1 -- 172.21.15.119:0/3682855798 wait complete. 2024-09-13T16:28:25.167 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:28:25.177 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-09-13T16:28:23.484430+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":235728,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":228339,"kb_avail":754738992,"statfs":{"total":773094113280,"available":772852727808,"internally_reserved":0,"allocated":7434240,"data_stored":3951767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":233819705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001059"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733380+0000","last_change":"2024-09-13T16:28:19.733380+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733380+0000","last_undegraded":"2024-09-13T16:28:19.733380+0000","last_fullsized":"2024-09-13T16:28:19.733380+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736612+0000","last_change":"2024-09-13T16:28:19.736612+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736612+0000","last_undegraded":"2024-09-13T16:28:19.736612+0000","last_fullsized":"2024-09-13T16:28:19.736612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:07.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.737602+0000","last_change":"2024-09-13T16:28:19.737602+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.737602+0000","last_undegraded":"2024-09-13T16:28:19.737602+0000","last_fullsized":"2024-09-13T16:28:19.737602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:59.677392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736454+0000","last_change":"2024-09-13T16:28:19.736454+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736454+0000","last_undegraded":"2024-09-13T16:28:19.736454+0000","last_fullsized":"2024-09-13T16:28:19.736454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734150+0000","last_change":"2024-09-13T16:28:19.734150+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734150+0000","last_undegraded":"2024-09-13T16:28:19.734150+0000","last_fullsized":"2024-09-13T16:28:19.734150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:43.928576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734089+0000","last_change":"2024-09-13T16:28:19.734089+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734089+0000","last_undegraded":"2024-09-13T16:28:19.734089+0000","last_fullsized":"2024-09-13T16:28:19.734089+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:19.660866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734751+0000","last_change":"2024-09-13T16:28:19.734751+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734751+0000","last_undegraded":"2024-09-13T16:28:19.734751+0000","last_fullsized":"2024-09-13T16:28:19.734751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.740412+0000","last_change":"2024-09-13T16:28:19.740412+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.740412+0000","last_undegraded":"2024-09-13T16:28:19.740412+0000","last_fullsized":"2024-09-13T16:28:19.740412+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:24.851511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734673+0000","last_change":"2024-09-13T16:28:19.734673+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734673+0000","last_undegraded":"2024-09-13T16:28:19.734673+0000","last_fullsized":"2024-09-13T16:28:19.734673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730285+0000","last_change":"2024-09-13T16:28:19.730285+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730285+0000","last_undegraded":"2024-09-13T16:28:19.730285+0000","last_fullsized":"2024-09-13T16:28:19.730285+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:28.409095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734827+0000","last_change":"2024-09-13T16:28:19.734827+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734827+0000","last_undegraded":"2024-09-13T16:28:19.734827+0000","last_fullsized":"2024-09-13T16:28:19.734827+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733583+0000","last_change":"2024-09-13T16:28:19.733583+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733583+0000","last_undegraded":"2024-09-13T16:28:19.733583+0000","last_fullsized":"2024-09-13T16:28:19.733583+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735789+0000","last_change":"2024-09-13T16:28:19.735789+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735789+0000","last_undegraded":"2024-09-13T16:28:19.735789+0000","last_fullsized":"2024-09-13T16:28:19.735789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730486+0000","last_change":"2024-09-13T16:28:19.730486+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730486+0000","last_undegraded":"2024-09-13T16:28:19.730486+0000","last_fullsized":"2024-09-13T16:28:19.730486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:09.620313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734680+0000","last_change":"2024-09-13T16:28:19.734680+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734680+0000","last_undegraded":"2024-09-13T16:28:19.734680+0000","last_fullsized":"2024-09-13T16:28:19.734680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729778+0000","last_change":"2024-09-13T16:28:19.729778+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729778+0000","last_undegraded":"2024-09-13T16:28:19.729778+0000","last_fullsized":"2024-09-13T16:28:19.729778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:58.987147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730913+0000","last_change":"2024-09-13T16:28:19.730913+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730913+0000","last_undegraded":"2024-09-13T16:28:19.730913+0000","last_fullsized":"2024-09-13T16:28:19.730913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:14.580434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735686+0000","last_change":"2024-09-13T16:28:19.735686+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735686+0000","last_undegraded":"2024-09-13T16:28:19.735686+0000","last_fullsized":"2024-09-13T16:28:19.735686+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:04.745546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735766+0000","last_change":"2024-09-13T16:28:19.735766+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735766+0000","last_undegraded":"2024-09-13T16:28:19.735766+0000","last_fullsized":"2024-09-13T16:28:19.735766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:26.842748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733365+0000","last_change":"2024-09-13T16:28:19.733365+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733365+0000","last_undegraded":"2024-09-13T16:28:19.733365+0000","last_fullsized":"2024-09-13T16:28:19.733365+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:21.436784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736501+0000","last_change":"2024-09-13T16:28:19.736501+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736501+0000","last_undegraded":"2024-09-13T16:28:19.736501+0000","last_fullsized":"2024-09-13T16:28:19.736501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:47.232880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:31.462343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736347+0000","last_change":"2024-09-13T16:28:19.736347+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736347+0000","last_undegraded":"2024-09-13T16:28:19.736347+0000","last_fullsized":"2024-09-13T16:28:19.736347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:40.488362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:30.820069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729527+0000","last_change":"2024-09-13T16:28:19.729527+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729527+0000","last_undegraded":"2024-09-13T16:28:19.729527+0000","last_fullsized":"2024-09-13T16:28:19.729527+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:34.445856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736343+0000","last_change":"2024-09-13T16:28:19.736343+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736343+0000","last_undegraded":"2024-09-13T16:28:19.736343+0000","last_fullsized":"2024-09-13T16:28:19.736343+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:03.708069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732876+0000","last_change":"2024-09-13T16:28:19.732876+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732876+0000","last_undegraded":"2024-09-13T16:28:19.732876+0000","last_fullsized":"2024-09-13T16:28:19.732876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734588+0000","last_change":"2024-09-13T16:28:19.734588+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734588+0000","last_undegraded":"2024-09-13T16:28:19.734588+0000","last_fullsized":"2024-09-13T16:28:19.734588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:02.929894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733282+0000","last_change":"2024-09-13T16:28:19.733282+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733282+0000","last_undegraded":"2024-09-13T16:28:19.733282+0000","last_fullsized":"2024-09-13T16:28:19.733282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:46.314613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733922+0000","last_change":"2024-09-13T16:28:19.733922+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733922+0000","last_undegraded":"2024-09-13T16:28:19.733922+0000","last_fullsized":"2024-09-13T16:28:19.733922+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:35.488631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732879+0000","last_change":"2024-09-13T16:28:19.732879+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732879+0000","last_undegraded":"2024-09-13T16:28:19.732879+0000","last_fullsized":"2024-09-13T16:28:19.732879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733012+0000","last_change":"2024-09-13T16:28:19.733012+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733012+0000","last_undegraded":"2024-09-13T16:28:19.733012+0000","last_fullsized":"2024-09-13T16:28:19.733012+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:21.792298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T16:28:17.717312+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:28:17.717312+0000","last_peered":"2024-09-13T16:28:17.717312+0000","last_clean":"2024-09-13T16:28:17.717312+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:28:17.717312+0000","last_undegraded":"2024-09-13T16:28:17.717312+0000","last_fullsized":"2024-09-13T16:28:17.717312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.86199999999999999,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.86099999999999999,"15min":0.84899999999999998},"min":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.085,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.71699999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.60299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0880000000000001,"5min":1.121,"15min":1.121},"last":1},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.84799999999999998},"min":{"1min":0.70599999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.131,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2170000000000001,"5min":1.228,"15min":1.228},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0389999999999999,"5min":1.091,"15min":1.091},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1220000000000001,"5min":1.238,"15min":1.238},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.997,"15min":1.0069999999999999},"min":{"1min":0.746,"5min":0.745,"15min":0.745},"max":{"1min":1.214,"5min":1.226,"15min":1.226},"last":1.101}]},{"osd":4,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88700000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.0580000000000001,"15min":1.0880000000000001},"min":{"1min":0.77800000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1870000000000001,"5min":1.589,"15min":1.589},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.996,"15min":1.03},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1439999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1120000000000001,"15min":1.1200000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.244,"5min":1.337,"15min":1.337},"last":0.90600000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.077,"15min":1.087},"min":{"1min":0.85599999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.196,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.109,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.99299999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.2609999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.95099999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.85199999999999998},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.628,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.0860000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0780000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.97499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2110000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.264},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0840000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.89200000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69499999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.248,"5min":1.536,"15min":1.536},"last":1.228},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.98699999999999999,"15min":0.98899999999999999},"min":{"1min":0.71799999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.264,"5min":1.409,"15min":1.409},"last":1.202}]},{"osd":4,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.96299999999999997,"15min":0.97699999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1910000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.365},{"interface":"front","average":{"1min":1.056,"5min":0.97399999999999998,"15min":0.94399999999999995},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91900000000000004,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1220000000000001,"15min":1.1379999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.75}]},{"osd":7,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0509999999999999,"15min":1.056},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.3149999999999999,"5min":1.337,"15min":1.337},"last":1.133}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":770048,"data_stored":341009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.5,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.131,"5min":1.675,"15min":1.675},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.876,"15min":0.88400000000000001},"min":{"1min":0.66500000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.321,"5min":1.369,"15min":1.369},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.84899999999999998,"15min":0.83099999999999996},"min":{"1min":0.66400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.169,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93000000000000005,"15min":0.93700000000000006},"min":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.032,"5min":1.625,"15min":1.625},"last":1.133},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.97299999999999998,"15min":0.98699999999999999},"min":{"1min":0.627,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5129999999999999,"5min":1.552,"15min":1.552},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85599999999999998,"15min":0.83799999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1639999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.97199999999999998,"15min":1.016},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1819999999999999,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":1.0489999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.0720000000000001},"min":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.579,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.85799999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2210000000000001,"5min":1.601,"15min":1.601},"last":0.93100000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96099999999999997,"15min":0.97299999999999998},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.179,"5min":1.421,"15min":1.421},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0740000000000001,"15min":1.0780000000000001},"min":{"1min":0.76000000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.101}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.036},"min":{"1min":0.68799999999999994,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3180000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.006},{"interface":"front","average":{"1min":1.034,"5min":1.175,"15min":1.2290000000000001},"min":{"1min":0.78900000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.266,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.51}]}]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84199999999999997,"15min":0.83799999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84699999999999998,"15min":0.84899999999999998},"min":{"1min":0.59899999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.91000000000000003,"15min":0.91400000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3049999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.72799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.353,"15min":1.353},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.85199999999999998,"15min":0.84099999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.091},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.61099999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.2749999999999999,"5min":1.603,"15min":1.603},"last":0.75800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.95599999999999996,"15min":0.95799999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.298,"5min":1.472,"15min":1.472},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96199999999999997,"15min":0.96499999999999997},"min":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":1.1419999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.98399999999999999,"15min":0.97999999999999998},"min":{"1min":0.70099999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":2.048,"15min":2.048},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1519999999999999},"min":{"1min":0.83699999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4179999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.749,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.014},{"interface":"front","average":{"1min":1.202,"5min":1.1220000000000001,"15min":1.1080000000000001},"min":{"1min":0.876,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.74,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":0.94399999999999995}]},{"osd":7,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.165,"15min":1.153},"min":{"1min":0.95999999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.054},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.1870000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.96099999999999997,"5min":1.04,"15min":1.04},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.498,"5min":0.433,"15min":0.433},"max":{"1min":0.96999999999999997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.378,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2310000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68500000000000005,"15min":0.69599999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":1.27,"15min":1.27},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.755},"min":{"1min":0.56399999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.139,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.70899999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.214,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95899999999999996,"15min":0.96199999999999997},"min":{"1min":0.66200000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.538,"5min":1.782,"15min":1.782},"last":1.0069999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97699999999999998,"15min":0.98199999999999998},"min":{"1min":0.71899999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.0920000000000001,"15min":1.1020000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.484,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.98099999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.119,"15min":1.137},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.571,"5min":1.885,"15min":1.885},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.032,"15min":1.0580000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.268,"5min":1.95,"15min":1.95},"last":1.0369999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.579,"5min":1.889,"15min":1.889},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0580000000000001,"15min":1.0609999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.89000000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31880,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":1.0960000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.42099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96699999999999997,"5min":0.997,"15min":0.997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.68799999999999994},"min":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88500000000000001,"5min":1.123,"15min":1.123},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.47699999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86899999999999999,"5min":1.073,"15min":1.073},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.78800000000000003},"min":{"1min":0.64300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.0740000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.94899999999999995,"15min":0.95499999999999996},"min":{"1min":0.70399999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2809999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.052},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95699999999999996,"15min":0.98399999999999999},"min":{"1min":0.78000000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.367,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.72599999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.153,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.72799999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.234,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.014,"15min":1.042},"min":{"1min":0.73899999999999999,"5min":0.623,"15min":0.623},"max":{"1min":1.2729999999999999,"5min":2.024,"15min":2.024},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0609999999999999,"15min":1.0669999999999999},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4930000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.95499999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0669999999999999,"15min":1.081},"min":{"1min":0.78900000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.252,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.0569999999999999,"15min":1.071},"min":{"1min":0.81499999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.4430000000000001,"5min":1.55,"15min":1.55},"last":1.01}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.51200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.371},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.67400000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0209999999999999,"5min":1.141,"15min":1.141},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.64500000000000002,"15min":0.61699999999999999},"min":{"1min":0.54700000000000004,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70299999999999996},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90600000000000003,"5min":1.458,"15min":1.458},"last":0.89000000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.72599999999999998,"15min":0.69099999999999995},"min":{"1min":0.52300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.026,"5min":1.236,"15min":1.236},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94799999999999995,"5min":1.403,"15min":1.403},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.92500000000000004,"15min":0.90700000000000003},"min":{"1min":0.81699999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2569999999999999,"5min":1.546,"15min":1.546},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3089999999999999,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95499999999999996,"15min":0.94699999999999995},"min":{"1min":0.80200000000000005,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.97099999999999997,"15min":0.98199999999999998},"min":{"1min":0.77700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3979999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.873}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.70899999999999996,"5min":0.626,"15min":0.626},"max":{"1min":1.4139999999999999,"5min":1.613,"15min":1.613},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.026,"5min":1.014,"15min":1.012},"min":{"1min":0.746,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3480000000000001,"5min":1.698,"15min":1.698},"last":0.99199999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.008,"15min":1.01},"min":{"1min":0.748,"5min":0.62,"15min":0.62},"max":{"1min":1.494,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.054,"15min":1.042},"min":{"1min":0.873,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.448,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.96099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.75600000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.90100000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":1.0249999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81000000000000005,"15min":0.81399999999999995},"min":{"1min":0.45300000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.6100000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":0.95999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2949999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.04,"5min":0.96999999999999997,"15min":0.95799999999999996},"min":{"1min":0.68700000000000006,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.5720000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.96299999999999997,"15min":0.96399999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2,"5min":1.24,"15min":1.24},"last":1.5409999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.99299999999999999,"15min":0.999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.198,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.6819999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.995,"15min":0.97799999999999998},"min":{"1min":0.80900000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.101},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.036,"15min":1.04},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.345}]},{"osd":7,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.054,"15min":1.046},"min":{"1min":0.90000000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.492},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0620000000000001,"15min":1.038},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:28:25.183 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:28:25.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 Processor -- start 2024-09-13T16:28:25.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- start start 2024-09-13T16:28:25.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e40eb420 0x7f06e404ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f06e4007260 0x7f06e404f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e401cfa0 con 0x7f06e4033ac0 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e401c670 con 0x7f06e4007260 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e401c950 con 0x7f06e40eb420 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f1ddc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f06e4007260 0x7f06e404f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f15db700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f06e4033ac0 0x7f06e4009ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48930/0 (socket says 172.21.15.119:48930) 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f15db700 1 -- 172.21.15.119:0/1240463840 learned_addr learned my addr 172.21.15.119:0/1240463840 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f0dda700 1 --2- 172.21.15.119:0/1240463840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e40eb420 0x7f06e404ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f1ddc700 1 -- 172.21.15.119:0/1240463840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e40eb420 msgr2=0x7f06e404ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f1ddc700 1 --2- 172.21.15.119:0/1240463840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e40eb420 0x7f06e404ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f1ddc700 1 -- 172.21.15.119:0/1240463840 >> v1:172.21.15.119:6789/0 conn(0x7f06e4033ac0 legacy=0x7f06e4009ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06e3fff700 1 -- 172.21.15.119:0/1240463840 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281545431 0 0) 0x7f06e401cfa0 con 0x7f06e4033ac0 2024-09-13T16:28:25.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f1ddc700 1 -- 172.21.15.119:0/1240463840 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e401cc30 con 0x7f06e4007260 2024-09-13T16:28:25.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f1ddc700 1 --2- 172.21.15.119:0/1240463840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4007260 0x7f06e404f150 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f06ec04ff70 tx=0x7f06ec068790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f4c4f96d8f6681 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:25.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06e3fff700 1 -- 172.21.15.119:0/1240463840 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec067690 con 0x7f06e4007260 2024-09-13T16:28:25.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06e3fff700 1 -- 172.21.15.119:0/1240463840 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06ec04f010 con 0x7f06e4007260 2024-09-13T16:28:25.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06e3fff700 1 -- 172.21.15.119:0/1240463840 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06ec067bb0 con 0x7f06e4007260 2024-09-13T16:28:25.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- 172.21.15.119:0/1240463840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4007260 msgr2=0x7f06e404f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 --2- 172.21.15.119:0/1240463840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4007260 0x7f06e404f150 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f06ec04ff70 tx=0x7f06ec068790 comp rx=0 tx=0).stop 2024-09-13T16:28:25.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- 172.21.15.119:0/1240463840 shutdown_connections 2024-09-13T16:28:25.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 --2- 172.21.15.119:0/1240463840 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4007260 0x7f06e404f150 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 --2- 172.21.15.119:0/1240463840 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e40eb420 0x7f06e404ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- 172.21.15.119:0/1240463840 >> 172.21.15.119:0/1240463840 conn(0x7f06e4036140 msgr2=0x7f06e4036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:25.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- 172.21.15.119:0/1240463840 shutdown_connections 2024-09-13T16:28:25.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.297+0000 7f06f25dd700 1 -- 172.21.15.119:0/1240463840 wait complete. 2024-09-13T16:28:25.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 Processor -- start 2024-09-13T16:28:25.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 -- start start 2024-09-13T16:28:25.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e4007260 0x7f06e40e6590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4033ac0 0x7f06e4180600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06e40391f0 0x7f06e40352b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e417c500 con 0x7f06e4033ac0 2024-09-13T16:28:25.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e4048810 con 0x7f06e40391f0 2024-09-13T16:28:25.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e40ea460 con 0x7f06e4007260 2024-09-13T16:28:25.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4033ac0 0x7f06e4180600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4033ac0 0x7f06e4180600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33736/0 (socket says 172.21.15.119:33736) 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 -- 172.21.15.119:0/3637923738 learned_addr learned my addr 172.21.15.119:0/3637923738 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f1ddc700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06e40391f0 0x7f06e40352b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f15db700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e4007260 0x7f06e40e6590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 -- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e4007260 msgr2=0x7f06e40e6590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e4007260 0x7f06e40e6590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 -- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06e40391f0 msgr2=0x7f06e40352b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06e40391f0 0x7f06e40352b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 -- 172.21.15.119:0/3637923738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e4180b40 con 0x7f06e4033ac0 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f0dda700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4033ac0 0x7f06e4180600 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f06e800e870 tx=0x7f06e800eb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06e1ffb700 1 -- 172.21.15.119:0/3637923738 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8009540 con 0x7f06e4033ac0 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06e1ffb700 1 -- 172.21.15.119:0/3637923738 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06e8013680 con 0x7f06e4033ac0 2024-09-13T16:28:25.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e40399d0 con 0x7f06e4033ac0 2024-09-13T16:28:25.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06e1ffb700 1 -- 172.21.15.119:0/3637923738 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e80139f0 con 0x7f06e4033ac0 2024-09-13T16:28:25.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.301+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e40396f0 con 0x7f06e4033ac0 2024-09-13T16:28:25.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.305+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e4009ba0 con 0x7f06e4033ac0 2024-09-13T16:28:25.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.305+0000 7f06e1ffb700 1 -- 172.21.15.119:0/3637923738 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f06e8013b50 con 0x7f06e4033ac0 2024-09-13T16:28:25.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.305+0000 7f06e1ffb700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f06d0066730 0x7f06d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.305+0000 7f06e1ffb700 1 -- 172.21.15.119:0/3637923738 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f06e8084410 con 0x7f06e4033ac0 2024-09-13T16:28:25.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.305+0000 7f06f15db700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f06d0066730 0x7f06d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.305+0000 7f06f15db700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f06d0066730 0x7f06d0068bf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0010e0 tx=0x7f06dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:25.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.313+0000 7f06e1ffb700 1 -- 172.21.15.119:0/3637923738 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f06e8051650 con 0x7f06e4033ac0 2024-09-13T16:28:25.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.485+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f06e400aef0 con 0x7f06d0066730 2024-09-13T16:28:25.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.501+0000 7f06e1ffb700 1 -- 172.21.15.119:0/3637923738 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276877 (secure 0 0 0) 0x7f06e400aef0 con 0x7f06d0066730 2024-09-13T16:28:25.509 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f06d0066730 msgr2=0x7f06d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f06d0066730 0x7f06d0068bf0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f06dc0010e0 tx=0x7f06dc009040 comp rx=0 tx=0).stop 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4033ac0 msgr2=0x7f06e4180600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4033ac0 0x7f06e4180600 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f06e800e870 tx=0x7f06e800eb80 comp rx=0 tx=0).stop 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 shutdown_connections 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f06d0066730 0x7f06d0068bf0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06e40391f0 0x7f06e40352b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06e4033ac0 0x7f06e4180600 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 --2- 172.21.15.119:0/3637923738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f06e4007260 0x7f06e40e6590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 >> 172.21.15.119:0/3637923738 conn(0x7f06e4036140 msgr2=0x7f06e400a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:25.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 shutdown_connections 2024-09-13T16:28:25.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.505+0000 7f06f25dd700 1 -- 172.21.15.119:0/3637923738 wait complete. 2024-09-13T16:28:25.513 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:28:25.525 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-13T16:28:25.485014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":235728,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":228339,"kb_avail":754738992,"statfs":{"total":773094113280,"available":772852727808,"internally_reserved":0,"allocated":7434240,"data_stored":3951767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":233819705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758017"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733380+0000","last_change":"2024-09-13T16:28:19.733380+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733380+0000","last_undegraded":"2024-09-13T16:28:19.733380+0000","last_fullsized":"2024-09-13T16:28:19.733380+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736612+0000","last_change":"2024-09-13T16:28:19.736612+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736612+0000","last_undegraded":"2024-09-13T16:28:19.736612+0000","last_fullsized":"2024-09-13T16:28:19.736612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:07.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.737602+0000","last_change":"2024-09-13T16:28:19.737602+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.737602+0000","last_undegraded":"2024-09-13T16:28:19.737602+0000","last_fullsized":"2024-09-13T16:28:19.737602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:59.677392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736454+0000","last_change":"2024-09-13T16:28:19.736454+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736454+0000","last_undegraded":"2024-09-13T16:28:19.736454+0000","last_fullsized":"2024-09-13T16:28:19.736454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734150+0000","last_change":"2024-09-13T16:28:19.734150+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734150+0000","last_undegraded":"2024-09-13T16:28:19.734150+0000","last_fullsized":"2024-09-13T16:28:19.734150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:43.928576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734089+0000","last_change":"2024-09-13T16:28:19.734089+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734089+0000","last_undegraded":"2024-09-13T16:28:19.734089+0000","last_fullsized":"2024-09-13T16:28:19.734089+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:19.660866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734751+0000","last_change":"2024-09-13T16:28:19.734751+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734751+0000","last_undegraded":"2024-09-13T16:28:19.734751+0000","last_fullsized":"2024-09-13T16:28:19.734751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.740412+0000","last_change":"2024-09-13T16:28:19.740412+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.740412+0000","last_undegraded":"2024-09-13T16:28:19.740412+0000","last_fullsized":"2024-09-13T16:28:19.740412+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:24.851511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734673+0000","last_change":"2024-09-13T16:28:19.734673+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734673+0000","last_undegraded":"2024-09-13T16:28:19.734673+0000","last_fullsized":"2024-09-13T16:28:19.734673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730285+0000","last_change":"2024-09-13T16:28:19.730285+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730285+0000","last_undegraded":"2024-09-13T16:28:19.730285+0000","last_fullsized":"2024-09-13T16:28:19.730285+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:28.409095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734827+0000","last_change":"2024-09-13T16:28:19.734827+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734827+0000","last_undegraded":"2024-09-13T16:28:19.734827+0000","last_fullsized":"2024-09-13T16:28:19.734827+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733583+0000","last_change":"2024-09-13T16:28:19.733583+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733583+0000","last_undegraded":"2024-09-13T16:28:19.733583+0000","last_fullsized":"2024-09-13T16:28:19.733583+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735789+0000","last_change":"2024-09-13T16:28:19.735789+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735789+0000","last_undegraded":"2024-09-13T16:28:19.735789+0000","last_fullsized":"2024-09-13T16:28:19.735789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730486+0000","last_change":"2024-09-13T16:28:19.730486+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730486+0000","last_undegraded":"2024-09-13T16:28:19.730486+0000","last_fullsized":"2024-09-13T16:28:19.730486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:09.620313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734680+0000","last_change":"2024-09-13T16:28:19.734680+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734680+0000","last_undegraded":"2024-09-13T16:28:19.734680+0000","last_fullsized":"2024-09-13T16:28:19.734680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729778+0000","last_change":"2024-09-13T16:28:19.729778+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729778+0000","last_undegraded":"2024-09-13T16:28:19.729778+0000","last_fullsized":"2024-09-13T16:28:19.729778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:58.987147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730913+0000","last_change":"2024-09-13T16:28:19.730913+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730913+0000","last_undegraded":"2024-09-13T16:28:19.730913+0000","last_fullsized":"2024-09-13T16:28:19.730913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:14.580434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735686+0000","last_change":"2024-09-13T16:28:19.735686+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735686+0000","last_undegraded":"2024-09-13T16:28:19.735686+0000","last_fullsized":"2024-09-13T16:28:19.735686+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:04.745546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735766+0000","last_change":"2024-09-13T16:28:19.735766+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735766+0000","last_undegraded":"2024-09-13T16:28:19.735766+0000","last_fullsized":"2024-09-13T16:28:19.735766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:26.842748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733365+0000","last_change":"2024-09-13T16:28:19.733365+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733365+0000","last_undegraded":"2024-09-13T16:28:19.733365+0000","last_fullsized":"2024-09-13T16:28:19.733365+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:21.436784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736501+0000","last_change":"2024-09-13T16:28:19.736501+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736501+0000","last_undegraded":"2024-09-13T16:28:19.736501+0000","last_fullsized":"2024-09-13T16:28:19.736501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:47.232880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:31.462343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736347+0000","last_change":"2024-09-13T16:28:19.736347+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736347+0000","last_undegraded":"2024-09-13T16:28:19.736347+0000","last_fullsized":"2024-09-13T16:28:19.736347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:40.488362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:30.820069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729527+0000","last_change":"2024-09-13T16:28:19.729527+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729527+0000","last_undegraded":"2024-09-13T16:28:19.729527+0000","last_fullsized":"2024-09-13T16:28:19.729527+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:34.445856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736343+0000","last_change":"2024-09-13T16:28:19.736343+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736343+0000","last_undegraded":"2024-09-13T16:28:19.736343+0000","last_fullsized":"2024-09-13T16:28:19.736343+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:03.708069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732876+0000","last_change":"2024-09-13T16:28:19.732876+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732876+0000","last_undegraded":"2024-09-13T16:28:19.732876+0000","last_fullsized":"2024-09-13T16:28:19.732876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734588+0000","last_change":"2024-09-13T16:28:19.734588+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734588+0000","last_undegraded":"2024-09-13T16:28:19.734588+0000","last_fullsized":"2024-09-13T16:28:19.734588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:02.929894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733282+0000","last_change":"2024-09-13T16:28:19.733282+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733282+0000","last_undegraded":"2024-09-13T16:28:19.733282+0000","last_fullsized":"2024-09-13T16:28:19.733282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:46.314613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733922+0000","last_change":"2024-09-13T16:28:19.733922+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733922+0000","last_undegraded":"2024-09-13T16:28:19.733922+0000","last_fullsized":"2024-09-13T16:28:19.733922+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:35.488631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732879+0000","last_change":"2024-09-13T16:28:19.732879+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732879+0000","last_undegraded":"2024-09-13T16:28:19.732879+0000","last_fullsized":"2024-09-13T16:28:19.732879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733012+0000","last_change":"2024-09-13T16:28:19.733012+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733012+0000","last_undegraded":"2024-09-13T16:28:19.733012+0000","last_fullsized":"2024-09-13T16:28:19.733012+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:21.792298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T16:28:17.717312+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:28:17.717312+0000","last_peered":"2024-09-13T16:28:17.717312+0000","last_clean":"2024-09-13T16:28:17.717312+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:28:17.717312+0000","last_undegraded":"2024-09-13T16:28:17.717312+0000","last_fullsized":"2024-09-13T16:28:17.717312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.86199999999999999,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.86099999999999999,"15min":0.84899999999999998},"min":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.085,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.71699999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.60299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0880000000000001,"5min":1.121,"15min":1.121},"last":1},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.84799999999999998},"min":{"1min":0.70599999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.131,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2170000000000001,"5min":1.228,"15min":1.228},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0389999999999999,"5min":1.091,"15min":1.091},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1220000000000001,"5min":1.238,"15min":1.238},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.997,"15min":1.0069999999999999},"min":{"1min":0.746,"5min":0.745,"15min":0.745},"max":{"1min":1.214,"5min":1.226,"15min":1.226},"last":1.101}]},{"osd":4,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88700000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.0580000000000001,"15min":1.0880000000000001},"min":{"1min":0.77800000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1870000000000001,"5min":1.589,"15min":1.589},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.996,"15min":1.03},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1439999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1120000000000001,"15min":1.1200000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.244,"5min":1.337,"15min":1.337},"last":0.90600000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.077,"15min":1.087},"min":{"1min":0.85599999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.196,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.109,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.99299999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.2609999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.95099999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.85199999999999998},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.628,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.0860000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0780000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.97499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2110000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.264},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0840000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.89200000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69499999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.248,"5min":1.536,"15min":1.536},"last":1.228},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.98699999999999999,"15min":0.98899999999999999},"min":{"1min":0.71799999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.264,"5min":1.409,"15min":1.409},"last":1.202}]},{"osd":4,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.96299999999999997,"15min":0.97699999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1910000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.365},{"interface":"front","average":{"1min":1.056,"5min":0.97399999999999998,"15min":0.94399999999999995},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91900000000000004,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1220000000000001,"15min":1.1379999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.75}]},{"osd":7,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0509999999999999,"15min":1.056},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.3149999999999999,"5min":1.337,"15min":1.337},"last":1.133}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":770048,"data_stored":341009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.5,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.131,"5min":1.675,"15min":1.675},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.876,"15min":0.88400000000000001},"min":{"1min":0.66500000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.321,"5min":1.369,"15min":1.369},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.84899999999999998,"15min":0.83099999999999996},"min":{"1min":0.66400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.169,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93000000000000005,"15min":0.93700000000000006},"min":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.032,"5min":1.625,"15min":1.625},"last":1.133},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.97299999999999998,"15min":0.98699999999999999},"min":{"1min":0.627,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5129999999999999,"5min":1.552,"15min":1.552},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85599999999999998,"15min":0.83799999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1639999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.97199999999999998,"15min":1.016},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1819999999999999,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":1.0489999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.0720000000000001},"min":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.579,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.85799999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2210000000000001,"5min":1.601,"15min":1.601},"last":0.93100000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96099999999999997,"15min":0.97299999999999998},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.179,"5min":1.421,"15min":1.421},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0740000000000001,"15min":1.0780000000000001},"min":{"1min":0.76000000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.101}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.036},"min":{"1min":0.68799999999999994,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3180000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.006},{"interface":"front","average":{"1min":1.034,"5min":1.175,"15min":1.2290000000000001},"min":{"1min":0.78900000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.266,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.51}]}]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84199999999999997,"15min":0.83799999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84699999999999998,"15min":0.84899999999999998},"min":{"1min":0.59899999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.91000000000000003,"15min":0.91400000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3049999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.72799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.353,"15min":1.353},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.85199999999999998,"15min":0.84099999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.091},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.61099999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.2749999999999999,"5min":1.603,"15min":1.603},"last":0.75800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.95599999999999996,"15min":0.95799999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.298,"5min":1.472,"15min":1.472},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96199999999999997,"15min":0.96499999999999997},"min":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":1.1419999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.98399999999999999,"15min":0.97999999999999998},"min":{"1min":0.70099999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":2.048,"15min":2.048},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1519999999999999},"min":{"1min":0.83699999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4179999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.749,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.014},{"interface":"front","average":{"1min":1.202,"5min":1.1220000000000001,"15min":1.1080000000000001},"min":{"1min":0.876,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.74,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":0.94399999999999995}]},{"osd":7,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.165,"15min":1.153},"min":{"1min":0.95999999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.054},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.1870000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.96099999999999997,"5min":1.04,"15min":1.04},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.498,"5min":0.433,"15min":0.433},"max":{"1min":0.96999999999999997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.378,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2310000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68500000000000005,"15min":0.69599999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":1.27,"15min":1.27},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.755},"min":{"1min":0.56399999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.139,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.70899999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.214,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95899999999999996,"15min":0.96199999999999997},"min":{"1min":0.66200000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.538,"5min":1.782,"15min":1.782},"last":1.0069999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97699999999999998,"15min":0.98199999999999998},"min":{"1min":0.71899999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.0920000000000001,"15min":1.1020000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.484,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.98099999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.119,"15min":1.137},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.571,"5min":1.885,"15min":1.885},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.032,"15min":1.0580000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.268,"5min":1.95,"15min":1.95},"last":1.0369999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.579,"5min":1.889,"15min":1.889},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0580000000000001,"15min":1.0609999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.89000000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31880,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":1.0960000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.42099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96699999999999997,"5min":0.997,"15min":0.997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.68799999999999994},"min":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88500000000000001,"5min":1.123,"15min":1.123},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.47699999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86899999999999999,"5min":1.073,"15min":1.073},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.78800000000000003},"min":{"1min":0.64300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.0740000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.94899999999999995,"15min":0.95499999999999996},"min":{"1min":0.70399999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2809999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.052},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95699999999999996,"15min":0.98399999999999999},"min":{"1min":0.78000000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.367,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.72599999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.153,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.72799999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.234,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.014,"15min":1.042},"min":{"1min":0.73899999999999999,"5min":0.623,"15min":0.623},"max":{"1min":1.2729999999999999,"5min":2.024,"15min":2.024},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0609999999999999,"15min":1.0669999999999999},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4930000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.95499999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0669999999999999,"15min":1.081},"min":{"1min":0.78900000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.252,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.0569999999999999,"15min":1.071},"min":{"1min":0.81499999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.4430000000000001,"5min":1.55,"15min":1.55},"last":1.01}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.51200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.371},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.67400000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0209999999999999,"5min":1.141,"15min":1.141},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.64500000000000002,"15min":0.61699999999999999},"min":{"1min":0.54700000000000004,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70299999999999996},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90600000000000003,"5min":1.458,"15min":1.458},"last":0.89000000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.72599999999999998,"15min":0.69099999999999995},"min":{"1min":0.52300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.026,"5min":1.236,"15min":1.236},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94799999999999995,"5min":1.403,"15min":1.403},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.92500000000000004,"15min":0.90700000000000003},"min":{"1min":0.81699999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2569999999999999,"5min":1.546,"15min":1.546},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3089999999999999,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95499999999999996,"15min":0.94699999999999995},"min":{"1min":0.80200000000000005,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.97099999999999997,"15min":0.98199999999999998},"min":{"1min":0.77700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3979999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.873}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.70899999999999996,"5min":0.626,"15min":0.626},"max":{"1min":1.4139999999999999,"5min":1.613,"15min":1.613},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.026,"5min":1.014,"15min":1.012},"min":{"1min":0.746,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3480000000000001,"5min":1.698,"15min":1.698},"last":0.99199999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.008,"15min":1.01},"min":{"1min":0.748,"5min":0.62,"15min":0.62},"max":{"1min":1.494,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.054,"15min":1.042},"min":{"1min":0.873,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.448,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.96099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.75600000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.90100000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":1.0249999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81000000000000005,"15min":0.81399999999999995},"min":{"1min":0.45300000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.6100000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":0.95999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2949999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.04,"5min":0.96999999999999997,"15min":0.95799999999999996},"min":{"1min":0.68700000000000006,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.5720000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.96299999999999997,"15min":0.96399999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2,"5min":1.24,"15min":1.24},"last":1.5409999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.99299999999999999,"15min":0.999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.198,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.6819999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.995,"15min":0.97799999999999998},"min":{"1min":0.80900000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.101},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.036,"15min":1.04},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.345}]},{"osd":7,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.054,"15min":1.046},"min":{"1min":0.90000000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.492},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0620000000000001,"15min":1.038},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:28:25.530 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:28:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.681+0000 7f8b589b5700 1 Processor -- start 2024-09-13T16:28:25.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.681+0000 7f8b589b5700 1 -- start start 2024-09-13T16:28:25.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.681+0000 7f8b589b5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8b540cc060 0x7f8b540cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.681+0000 7f8b589b5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b5406cf60 con 0x7f8b540cde70 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.681+0000 7f8b589b5700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b540cc4d0 con 0x7f8b540adb30 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.681+0000 7f8b589b5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b5406c350 con 0x7f8b540cc060 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8b540cc060 0x7f8b540cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8b540cc060 0x7f8b540cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33746/0 (socket says 172.21.15.119:33746) 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5259c700 1 -- 172.21.15.119:0/2945680958 learned_addr learned my addr 172.21.15.119:0/2945680958 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5359e700 1 --1- 172.21.15.119:0/2945680958 >> v1:172.21.15.119:6789/0 conn(0x7f8b540cde70 0x7f8b540b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48932/0 (socket says 172.21.15.119:48932) 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5259c700 1 -- 172.21.15.119:0/2945680958 >> v1:172.21.15.145:6789/0 conn(0x7f8b540adb30 legacy=0x7f8b5406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5259c700 1 -- 172.21.15.119:0/2945680958 >> v1:172.21.15.119:6789/0 conn(0x7f8b540cde70 legacy=0x7f8b540b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5259c700 1 -- 172.21.15.119:0/2945680958 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b5406c630 con 0x7f8b540cc060 2024-09-13T16:28:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b5259c700 1 --2- 172.21.15.119:0/2945680958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540cc060 0x7f8b540cd930 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009e50 tx=0x7f8b4400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe81e9f18c7468b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b51d9b700 1 -- 172.21.15.119:0/2945680958 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4400e070 con 0x7f8b540cc060 2024-09-13T16:28:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b51d9b700 1 -- 172.21.15.119:0/2945680958 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b44003a40 con 0x7f8b540cc060 2024-09-13T16:28:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b51d9b700 1 -- 172.21.15.119:0/2945680958 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b44003df0 con 0x7f8b540cc060 2024-09-13T16:28:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b589b5700 1 -- 172.21.15.119:0/2945680958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540cc060 msgr2=0x7f8b540cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2945680958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540cc060 0x7f8b540cd930 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f8b44009e50 tx=0x7f8b4400b330 comp rx=0 tx=0).stop 2024-09-13T16:28:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b589b5700 1 -- 172.21.15.119:0/2945680958 shutdown_connections 2024-09-13T16:28:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2945680958 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540cc060 0x7f8b540cd930 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b589b5700 1 -- 172.21.15.119:0/2945680958 >> 172.21.15.119:0/2945680958 conn(0x7f8b5406a350 msgr2=0x7f8b5406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b589b5700 1 -- 172.21.15.119:0/2945680958 shutdown_connections 2024-09-13T16:28:25.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.685+0000 7f8b589b5700 1 -- 172.21.15.119:0/2945680958 wait complete. 2024-09-13T16:28:25.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 Processor -- start 2024-09-13T16:28:25.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 -- start start 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540adb30 0x7f8b541f0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b540cc060 0x7f8b5415c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b540cde70 0x7f8b5413e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5415adc0 con 0x7f8b540adb30 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b541f1280 con 0x7f8b540cde70 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b589b5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b5413a9a0 con 0x7f8b540cc060 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540adb30 0x7f8b541f0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540adb30 0x7f8b541f0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33762/0 (socket says 172.21.15.119:33762) 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 -- 172.21.15.119:0/2333344448 learned_addr learned my addr 172.21.15.119:0/2333344448 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b5259c700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b540cc060 0x7f8b5415c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b5359e700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b540cde70 0x7f8b5413e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 -- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b540cc060 msgr2=0x7f8b5415c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b540cc060 0x7f8b5415c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 -- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b540cde70 msgr2=0x7f8b5413e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b540cde70 0x7f8b5413e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b52d9d700 1 -- 172.21.15.119:0/2333344448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44009b30 con 0x7f8b540adb30 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.689+0000 7f8b5359e700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b540cde70 0x7f8b5413e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:25.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b52d9d700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540adb30 0x7f8b541f0d40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800c4e0 tx=0x7f8b4800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:25.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b3b7fe700 1 -- 172.21.15.119:0/2333344448 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48011790 con 0x7f8b540adb30 2024-09-13T16:28:25.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b3b7fe700 1 -- 172.21.15.119:0/2333344448 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b48011910 con 0x7f8b540adb30 2024-09-13T16:28:25.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b5413ac80 con 0x7f8b540adb30 2024-09-13T16:28:25.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b3b7fe700 1 -- 172.21.15.119:0/2333344448 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b48011cc0 con 0x7f8b540adb30 2024-09-13T16:28:25.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5413b240 con 0x7f8b540adb30 2024-09-13T16:28:25.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b5406d3d0 con 0x7f8b540adb30 2024-09-13T16:28:25.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.693+0000 7f8b3b7fe700 1 -- 172.21.15.119:0/2333344448 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8b4801c460 con 0x7f8b540adb30 2024-09-13T16:28:25.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.697+0000 7f8b3b7fe700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b40066730 0x7f8b40068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:25.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.697+0000 7f8b3b7fe700 1 -- 172.21.15.119:0/2333344448 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8b480840b0 con 0x7f8b540adb30 2024-09-13T16:28:25.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.697+0000 7f8b5259c700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b40066730 0x7f8b40068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:25.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.697+0000 7f8b5259c700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b40066730 0x7f8b40068bf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b44015040 tx=0x7f8b44009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:25.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.705+0000 7f8b3b7fe700 1 -- 172.21.15.119:0/2333344448 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8b5406d3d0 con 0x7f8b540adb30 2024-09-13T16:28:25.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.869+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8b540ce300 con 0x7f8b40066730 2024-09-13T16:28:25.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.885+0000 7f8b3b7fe700 1 -- 172.21.15.119:0/2333344448 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276877 (secure 0 0 0) 0x7f8b4400b980 con 0x7f8b40066730 2024-09-13T16:28:25.895 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b40066730 msgr2=0x7f8b40068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b40066730 0x7f8b40068bf0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b44015040 tx=0x7f8b44009000 comp rx=0 tx=0).stop 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540adb30 msgr2=0x7f8b541f0d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540adb30 0x7f8b541f0d40 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f8b4800c4e0 tx=0x7f8b4800c8a0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 shutdown_connections 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8b40066730 0x7f8b40068bf0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b540cde70 0x7f8b5413e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b540cc060 0x7f8b5415c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 --2- 172.21.15.119:0/2333344448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b540adb30 0x7f8b541f0d40 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:25.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 >> 172.21.15.119:0/2333344448 conn(0x7f8b5406a350 msgr2=0x7f8b54066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:25.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 shutdown_connections 2024-09-13T16:28:25.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:25.889+0000 7f8b589b5700 1 -- 172.21.15.119:0/2333344448 wait complete. 2024-09-13T16:28:25.898 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:28:25.910 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-13T16:28:25.485014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":235728,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":228339,"kb_avail":754738992,"statfs":{"total":773094113280,"available":772852727808,"internally_reserved":0,"allocated":7434240,"data_stored":3951767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":233819705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758017"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733380+0000","last_change":"2024-09-13T16:28:19.733380+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733380+0000","last_undegraded":"2024-09-13T16:28:19.733380+0000","last_fullsized":"2024-09-13T16:28:19.733380+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736612+0000","last_change":"2024-09-13T16:28:19.736612+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736612+0000","last_undegraded":"2024-09-13T16:28:19.736612+0000","last_fullsized":"2024-09-13T16:28:19.736612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:07.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.737602+0000","last_change":"2024-09-13T16:28:19.737602+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.737602+0000","last_undegraded":"2024-09-13T16:28:19.737602+0000","last_fullsized":"2024-09-13T16:28:19.737602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:59.677392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736454+0000","last_change":"2024-09-13T16:28:19.736454+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736454+0000","last_undegraded":"2024-09-13T16:28:19.736454+0000","last_fullsized":"2024-09-13T16:28:19.736454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734150+0000","last_change":"2024-09-13T16:28:19.734150+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734150+0000","last_undegraded":"2024-09-13T16:28:19.734150+0000","last_fullsized":"2024-09-13T16:28:19.734150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:43.928576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734089+0000","last_change":"2024-09-13T16:28:19.734089+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734089+0000","last_undegraded":"2024-09-13T16:28:19.734089+0000","last_fullsized":"2024-09-13T16:28:19.734089+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:19.660866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734751+0000","last_change":"2024-09-13T16:28:19.734751+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734751+0000","last_undegraded":"2024-09-13T16:28:19.734751+0000","last_fullsized":"2024-09-13T16:28:19.734751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.740412+0000","last_change":"2024-09-13T16:28:19.740412+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.740412+0000","last_undegraded":"2024-09-13T16:28:19.740412+0000","last_fullsized":"2024-09-13T16:28:19.740412+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:24.851511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734673+0000","last_change":"2024-09-13T16:28:19.734673+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734673+0000","last_undegraded":"2024-09-13T16:28:19.734673+0000","last_fullsized":"2024-09-13T16:28:19.734673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730285+0000","last_change":"2024-09-13T16:28:19.730285+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730285+0000","last_undegraded":"2024-09-13T16:28:19.730285+0000","last_fullsized":"2024-09-13T16:28:19.730285+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:28.409095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734827+0000","last_change":"2024-09-13T16:28:19.734827+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734827+0000","last_undegraded":"2024-09-13T16:28:19.734827+0000","last_fullsized":"2024-09-13T16:28:19.734827+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733583+0000","last_change":"2024-09-13T16:28:19.733583+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733583+0000","last_undegraded":"2024-09-13T16:28:19.733583+0000","last_fullsized":"2024-09-13T16:28:19.733583+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735789+0000","last_change":"2024-09-13T16:28:19.735789+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735789+0000","last_undegraded":"2024-09-13T16:28:19.735789+0000","last_fullsized":"2024-09-13T16:28:19.735789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730486+0000","last_change":"2024-09-13T16:28:19.730486+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730486+0000","last_undegraded":"2024-09-13T16:28:19.730486+0000","last_fullsized":"2024-09-13T16:28:19.730486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:09.620313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734680+0000","last_change":"2024-09-13T16:28:19.734680+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734680+0000","last_undegraded":"2024-09-13T16:28:19.734680+0000","last_fullsized":"2024-09-13T16:28:19.734680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729778+0000","last_change":"2024-09-13T16:28:19.729778+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729778+0000","last_undegraded":"2024-09-13T16:28:19.729778+0000","last_fullsized":"2024-09-13T16:28:19.729778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:58.987147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730913+0000","last_change":"2024-09-13T16:28:19.730913+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730913+0000","last_undegraded":"2024-09-13T16:28:19.730913+0000","last_fullsized":"2024-09-13T16:28:19.730913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:14.580434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735686+0000","last_change":"2024-09-13T16:28:19.735686+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735686+0000","last_undegraded":"2024-09-13T16:28:19.735686+0000","last_fullsized":"2024-09-13T16:28:19.735686+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:04.745546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735766+0000","last_change":"2024-09-13T16:28:19.735766+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735766+0000","last_undegraded":"2024-09-13T16:28:19.735766+0000","last_fullsized":"2024-09-13T16:28:19.735766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:26.842748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733365+0000","last_change":"2024-09-13T16:28:19.733365+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733365+0000","last_undegraded":"2024-09-13T16:28:19.733365+0000","last_fullsized":"2024-09-13T16:28:19.733365+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:21.436784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736501+0000","last_change":"2024-09-13T16:28:19.736501+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736501+0000","last_undegraded":"2024-09-13T16:28:19.736501+0000","last_fullsized":"2024-09-13T16:28:19.736501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:47.232880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:31.462343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736347+0000","last_change":"2024-09-13T16:28:19.736347+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736347+0000","last_undegraded":"2024-09-13T16:28:19.736347+0000","last_fullsized":"2024-09-13T16:28:19.736347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:40.488362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:30.820069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729527+0000","last_change":"2024-09-13T16:28:19.729527+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729527+0000","last_undegraded":"2024-09-13T16:28:19.729527+0000","last_fullsized":"2024-09-13T16:28:19.729527+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:34.445856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736343+0000","last_change":"2024-09-13T16:28:19.736343+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736343+0000","last_undegraded":"2024-09-13T16:28:19.736343+0000","last_fullsized":"2024-09-13T16:28:19.736343+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:03.708069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732876+0000","last_change":"2024-09-13T16:28:19.732876+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732876+0000","last_undegraded":"2024-09-13T16:28:19.732876+0000","last_fullsized":"2024-09-13T16:28:19.732876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734588+0000","last_change":"2024-09-13T16:28:19.734588+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734588+0000","last_undegraded":"2024-09-13T16:28:19.734588+0000","last_fullsized":"2024-09-13T16:28:19.734588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:02.929894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733282+0000","last_change":"2024-09-13T16:28:19.733282+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733282+0000","last_undegraded":"2024-09-13T16:28:19.733282+0000","last_fullsized":"2024-09-13T16:28:19.733282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:46.314613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733922+0000","last_change":"2024-09-13T16:28:19.733922+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733922+0000","last_undegraded":"2024-09-13T16:28:19.733922+0000","last_fullsized":"2024-09-13T16:28:19.733922+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:35.488631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732879+0000","last_change":"2024-09-13T16:28:19.732879+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732879+0000","last_undegraded":"2024-09-13T16:28:19.732879+0000","last_fullsized":"2024-09-13T16:28:19.732879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733012+0000","last_change":"2024-09-13T16:28:19.733012+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733012+0000","last_undegraded":"2024-09-13T16:28:19.733012+0000","last_fullsized":"2024-09-13T16:28:19.733012+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:21.792298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T16:28:17.717312+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:28:17.717312+0000","last_peered":"2024-09-13T16:28:17.717312+0000","last_clean":"2024-09-13T16:28:17.717312+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:28:17.717312+0000","last_undegraded":"2024-09-13T16:28:17.717312+0000","last_fullsized":"2024-09-13T16:28:17.717312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.86199999999999999,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.86099999999999999,"15min":0.84899999999999998},"min":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.085,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.71699999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.60299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0880000000000001,"5min":1.121,"15min":1.121},"last":1},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.84799999999999998},"min":{"1min":0.70599999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.131,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2170000000000001,"5min":1.228,"15min":1.228},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0389999999999999,"5min":1.091,"15min":1.091},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1220000000000001,"5min":1.238,"15min":1.238},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.997,"15min":1.0069999999999999},"min":{"1min":0.746,"5min":0.745,"15min":0.745},"max":{"1min":1.214,"5min":1.226,"15min":1.226},"last":1.101}]},{"osd":4,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88700000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.0580000000000001,"15min":1.0880000000000001},"min":{"1min":0.77800000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1870000000000001,"5min":1.589,"15min":1.589},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.996,"15min":1.03},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1439999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1120000000000001,"15min":1.1200000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.244,"5min":1.337,"15min":1.337},"last":0.90600000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.077,"15min":1.087},"min":{"1min":0.85599999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.196,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.109,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.99299999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.2609999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.95099999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.85199999999999998},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.628,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.0860000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0780000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.97499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2110000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.264},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0840000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.89200000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69499999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.248,"5min":1.536,"15min":1.536},"last":1.228},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.98699999999999999,"15min":0.98899999999999999},"min":{"1min":0.71799999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.264,"5min":1.409,"15min":1.409},"last":1.202}]},{"osd":4,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.96299999999999997,"15min":0.97699999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1910000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.365},{"interface":"front","average":{"1min":1.056,"5min":0.97399999999999998,"15min":0.94399999999999995},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91900000000000004,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1220000000000001,"15min":1.1379999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.75}]},{"osd":7,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0509999999999999,"15min":1.056},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.3149999999999999,"5min":1.337,"15min":1.337},"last":1.133}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":770048,"data_stored":341009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.5,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.131,"5min":1.675,"15min":1.675},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.876,"15min":0.88400000000000001},"min":{"1min":0.66500000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.321,"5min":1.369,"15min":1.369},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.84899999999999998,"15min":0.83099999999999996},"min":{"1min":0.66400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.169,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93000000000000005,"15min":0.93700000000000006},"min":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.032,"5min":1.625,"15min":1.625},"last":1.133},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.97299999999999998,"15min":0.98699999999999999},"min":{"1min":0.627,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5129999999999999,"5min":1.552,"15min":1.552},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85599999999999998,"15min":0.83799999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1639999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.97199999999999998,"15min":1.016},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1819999999999999,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":1.0489999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.0720000000000001},"min":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.579,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.85799999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2210000000000001,"5min":1.601,"15min":1.601},"last":0.93100000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96099999999999997,"15min":0.97299999999999998},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.179,"5min":1.421,"15min":1.421},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0740000000000001,"15min":1.0780000000000001},"min":{"1min":0.76000000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.101}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.036},"min":{"1min":0.68799999999999994,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3180000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.006},{"interface":"front","average":{"1min":1.034,"5min":1.175,"15min":1.2290000000000001},"min":{"1min":0.78900000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.266,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.51}]}]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84199999999999997,"15min":0.83799999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84699999999999998,"15min":0.84899999999999998},"min":{"1min":0.59899999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.91000000000000003,"15min":0.91400000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3049999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.72799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.353,"15min":1.353},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.85199999999999998,"15min":0.84099999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.091},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.61099999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.2749999999999999,"5min":1.603,"15min":1.603},"last":0.75800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.95599999999999996,"15min":0.95799999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.298,"5min":1.472,"15min":1.472},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96199999999999997,"15min":0.96499999999999997},"min":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":1.1419999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.98399999999999999,"15min":0.97999999999999998},"min":{"1min":0.70099999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":2.048,"15min":2.048},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1519999999999999},"min":{"1min":0.83699999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4179999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.749,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.014},{"interface":"front","average":{"1min":1.202,"5min":1.1220000000000001,"15min":1.1080000000000001},"min":{"1min":0.876,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.74,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":0.94399999999999995}]},{"osd":7,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.165,"15min":1.153},"min":{"1min":0.95999999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.054},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.1870000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.96099999999999997,"5min":1.04,"15min":1.04},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.498,"5min":0.433,"15min":0.433},"max":{"1min":0.96999999999999997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.378,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2310000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68500000000000005,"15min":0.69599999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":1.27,"15min":1.27},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.755},"min":{"1min":0.56399999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.139,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.70899999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.214,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95899999999999996,"15min":0.96199999999999997},"min":{"1min":0.66200000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.538,"5min":1.782,"15min":1.782},"last":1.0069999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97699999999999998,"15min":0.98199999999999998},"min":{"1min":0.71899999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.0920000000000001,"15min":1.1020000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.484,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.98099999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.119,"15min":1.137},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.571,"5min":1.885,"15min":1.885},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.032,"15min":1.0580000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.268,"5min":1.95,"15min":1.95},"last":1.0369999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.579,"5min":1.889,"15min":1.889},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0580000000000001,"15min":1.0609999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.89000000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31880,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":1.0960000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.42099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96699999999999997,"5min":0.997,"15min":0.997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.68799999999999994},"min":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88500000000000001,"5min":1.123,"15min":1.123},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.47699999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86899999999999999,"5min":1.073,"15min":1.073},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.78800000000000003},"min":{"1min":0.64300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.0740000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.94899999999999995,"15min":0.95499999999999996},"min":{"1min":0.70399999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2809999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.052},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95699999999999996,"15min":0.98399999999999999},"min":{"1min":0.78000000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.367,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.72599999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.153,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.72799999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.234,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.014,"15min":1.042},"min":{"1min":0.73899999999999999,"5min":0.623,"15min":0.623},"max":{"1min":1.2729999999999999,"5min":2.024,"15min":2.024},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0609999999999999,"15min":1.0669999999999999},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4930000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.95499999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0669999999999999,"15min":1.081},"min":{"1min":0.78900000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.252,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.0569999999999999,"15min":1.071},"min":{"1min":0.81499999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.4430000000000001,"5min":1.55,"15min":1.55},"last":1.01}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.51200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.371},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.67400000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0209999999999999,"5min":1.141,"15min":1.141},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.64500000000000002,"15min":0.61699999999999999},"min":{"1min":0.54700000000000004,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70299999999999996},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90600000000000003,"5min":1.458,"15min":1.458},"last":0.89000000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.72599999999999998,"15min":0.69099999999999995},"min":{"1min":0.52300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.026,"5min":1.236,"15min":1.236},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94799999999999995,"5min":1.403,"15min":1.403},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.92500000000000004,"15min":0.90700000000000003},"min":{"1min":0.81699999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2569999999999999,"5min":1.546,"15min":1.546},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3089999999999999,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95499999999999996,"15min":0.94699999999999995},"min":{"1min":0.80200000000000005,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.97099999999999997,"15min":0.98199999999999998},"min":{"1min":0.77700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3979999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.873}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.70899999999999996,"5min":0.626,"15min":0.626},"max":{"1min":1.4139999999999999,"5min":1.613,"15min":1.613},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.026,"5min":1.014,"15min":1.012},"min":{"1min":0.746,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3480000000000001,"5min":1.698,"15min":1.698},"last":0.99199999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.008,"15min":1.01},"min":{"1min":0.748,"5min":0.62,"15min":0.62},"max":{"1min":1.494,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.054,"15min":1.042},"min":{"1min":0.873,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.448,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.96099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.75600000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.90100000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":1.0249999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81000000000000005,"15min":0.81399999999999995},"min":{"1min":0.45300000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.6100000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":0.95999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2949999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.04,"5min":0.96999999999999997,"15min":0.95799999999999996},"min":{"1min":0.68700000000000006,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.5720000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.96299999999999997,"15min":0.96399999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2,"5min":1.24,"15min":1.24},"last":1.5409999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.99299999999999999,"15min":0.999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.198,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.6819999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.995,"15min":0.97799999999999998},"min":{"1min":0.80900000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.101},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.036,"15min":1.04},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.345}]},{"osd":7,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.054,"15min":1.046},"min":{"1min":0.90000000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.492},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0620000000000001,"15min":1.038},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:28:25.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3f is not active+clean 2024-09-13T16:28:25.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.916 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3e is not active+clean 2024-09-13T16:28:25.916 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3d is not active+clean 2024-09-13T16:28:25.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3c is not active+clean 2024-09-13T16:28:25.917 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.917 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3b is not active+clean 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3a is not active+clean 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.39 is not active+clean 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.38 is not active+clean 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.37 is not active+clean 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.36 is not active+clean 2024-09-13T16:28:25.918 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.35 is not active+clean 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.34 is not active+clean 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.33 is not active+clean 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.32 is not active+clean 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.31 is not active+clean 2024-09-13T16:28:25.919 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.30 is not active+clean 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2f is not active+clean 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2e is not active+clean 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2d is not active+clean 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2c is not active+clean 2024-09-13T16:28:25.920 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2b is not active+clean 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2a is not active+clean 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.29 is not active+clean 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.28 is not active+clean 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.27 is not active+clean 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.921 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.26 is not active+clean 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.25 is not active+clean 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.24 is not active+clean 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.23 is not active+clean 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.22 is not active+clean 2024-09-13T16:28:25.922 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.21 is not active+clean 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.20 is not active+clean 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1c is not active+clean 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1f is not active+clean 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.733380+0000', 'last_change': '2024-09-13T16:28:19.733380+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.733380+0000', 'last_undegraded': '2024-09-13T16:28:19.733380+0000', 'last_fullsized': '2024-09-13T16:28:19.733380+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1d is not active+clean 2024-09-13T16:28:25.923 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1e is not active+clean 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.736612+0000', 'last_change': '2024-09-13T16:28:19.736612+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.736612+0000', 'last_undegraded': '2024-09-13T16:28:19.736612+0000', 'last_fullsized': '2024-09-13T16:28:19.736612+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T18:04:07.390953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1e is not active+clean 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1d is not active+clean 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.737602+0000', 'last_change': '2024-09-13T16:28:19.737602+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.737602+0000', 'last_undegraded': '2024-09-13T16:28:19.737602+0000', 'last_fullsized': '2024-09-13T16:28:19.737602+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T23:16:59.677392+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1f is not active+clean 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1c is not active+clean 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.736454+0000', 'last_change': '2024-09-13T16:28:19.736454+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.736454+0000', 'last_undegraded': '2024-09-13T16:28:19.736454+0000', 'last_fullsized': '2024-09-13T16:28:19.736454+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-13T16:28:25.924 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.18 is not active+clean 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1b is not active+clean 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.734150+0000', 'last_change': '2024-09-13T16:28:19.734150+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.734150+0000', 'last_undegraded': '2024-09-13T16:28:19.734150+0000', 'last_fullsized': '2024-09-13T16:28:19.734150+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T23:38:43.928576+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.19 is not active+clean 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1a is not active+clean 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.734089+0000', 'last_change': '2024-09-13T16:28:19.734089+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.734089+0000', 'last_undegraded': '2024-09-13T16:28:19.734089+0000', 'last_fullsized': '2024-09-13T16:28:19.734089+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T16:52:19.660866+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1a is not active+clean 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.925 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.19 is not active+clean 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.734751+0000', 'last_change': '2024-09-13T16:28:19.734751+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.734751+0000', 'last_undegraded': '2024-09-13T16:28:19.734751+0000', 'last_fullsized': '2024-09-13T16:28:19.734751+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1b is not active+clean 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.18 is not active+clean 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.740412+0000', 'last_change': '2024-09-13T16:28:19.740412+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.740412+0000', 'last_undegraded': '2024-09-13T16:28:19.740412+0000', 'last_fullsized': '2024-09-13T16:28:19.740412+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T02:34:24.851511+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 7], 'acting': [6, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.14 is not active+clean 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.17 is not active+clean 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.734673+0000', 'last_change': '2024-09-13T16:28:19.734673+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.734673+0000', 'last_undegraded': '2024-09-13T16:28:19.734673+0000', 'last_fullsized': '2024-09-13T16:28:19.734673+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.15 is not active+clean 2024-09-13T16:28:25.926 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.16 is not active+clean 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.730285+0000', 'last_change': '2024-09-13T16:28:19.730285+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.730285+0000', 'last_undegraded': '2024-09-13T16:28:19.730285+0000', 'last_fullsized': '2024-09-13T16:28:19.730285+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T00:38:28.409095+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.16 is not active+clean 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.15 is not active+clean 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.734827+0000', 'last_change': '2024-09-13T16:28:19.734827+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.734827+0000', 'last_undegraded': '2024-09-13T16:28:19.734827+0000', 'last_fullsized': '2024-09-13T16:28:19.734827+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.17 is not active+clean 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.14 is not active+clean 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.733583+0000', 'last_change': '2024-09-13T16:28:19.733583+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.733583+0000', 'last_undegraded': '2024-09-13T16:28:19.733583+0000', 'last_fullsized': '2024-09-13T16:28:19.733583+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.10 is not active+clean 2024-09-13T16:28:25.927 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.13 is not active+clean 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.735789+0000', 'last_change': '2024-09-13T16:28:19.735789+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.735789+0000', 'last_undegraded': '2024-09-13T16:28:19.735789+0000', 'last_fullsized': '2024-09-13T16:28:19.735789+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.11 is not active+clean 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.12 is not active+clean 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.730486+0000', 'last_change': '2024-09-13T16:28:19.730486+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.730486+0000', 'last_undegraded': '2024-09-13T16:28:19.730486+0000', 'last_fullsized': '2024-09-13T16:28:19.730486+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T21:39:09.620313+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.12 is not active+clean 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.11 is not active+clean 2024-09-13T16:28:25.928 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.734680+0000', 'last_change': '2024-09-13T16:28:19.734680+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.734680+0000', 'last_undegraded': '2024-09-13T16:28:19.734680+0000', 'last_fullsized': '2024-09-13T16:28:19.734680+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.13 is not active+clean 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.10 is not active+clean 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.729778+0000', 'last_change': '2024-09-13T16:28:19.729778+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.729778+0000', 'last_undegraded': '2024-09-13T16:28:19.729778+0000', 'last_fullsized': '2024-09-13T16:28:19.729778+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T20:16:58.987147+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 7], 'acting': [4, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.c is not active+clean 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.f is not active+clean 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.730913+0000', 'last_change': '2024-09-13T16:28:19.730913+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.730913+0000', 'last_undegraded': '2024-09-13T16:28:19.730913+0000', 'last_fullsized': '2024-09-13T16:28:19.730913+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T21:11:14.580434+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.d is not active+clean 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.929 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.e is not active+clean 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.735686+0000', 'last_change': '2024-09-13T16:28:19.735686+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.735686+0000', 'last_undegraded': '2024-09-13T16:28:19.735686+0000', 'last_fullsized': '2024-09-13T16:28:19.735686+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T18:43:04.745546+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.e is not active+clean 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.d is not active+clean 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.735766+0000', 'last_change': '2024-09-13T16:28:19.735766+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.735766+0000', 'last_undegraded': '2024-09-13T16:28:19.735766+0000', 'last_fullsized': '2024-09-13T16:28:19.735766+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T18:10:26.842748+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.f is not active+clean 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.c is not active+clean 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.733365+0000', 'last_change': '2024-09-13T16:28:19.733365+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.733365+0000', 'last_undegraded': '2024-09-13T16:28:19.733365+0000', 'last_fullsized': '2024-09-13T16:28:19.733365+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T21:45:21.436784+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 4], 'acting': [3, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.8 is not active+clean 2024-09-13T16:28:25.930 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.b is not active+clean 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.736501+0000', 'last_change': '2024-09-13T16:28:19.736501+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.736501+0000', 'last_undegraded': '2024-09-13T16:28:19.736501+0000', 'last_fullsized': '2024-09-13T16:28:19.736501+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T16:28:47.232880+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.9 is not active+clean 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.a is not active+clean 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.729438+0000', 'last_change': '2024-09-13T16:28:19.729438+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.729438+0000', 'last_undegraded': '2024-09-13T16:28:19.729438+0000', 'last_fullsized': '2024-09-13T16:28:19.729438+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T03:35:31.462343+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.a is not active+clean 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.9 is not active+clean 2024-09-13T16:28:25.931 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.736347+0000', 'last_change': '2024-09-13T16:28:19.736347+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.736347+0000', 'last_undegraded': '2024-09-13T16:28:19.736347+0000', 'last_fullsized': '2024-09-13T16:28:19.736347+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T20:08:40.488362+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.4 is not active+clean 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.7 is not active+clean 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.729438+0000', 'last_change': '2024-09-13T16:28:19.729438+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.729438+0000', 'last_undegraded': '2024-09-13T16:28:19.729438+0000', 'last_fullsized': '2024-09-13T16:28:19.729438+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T19:31:30.820069+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.5 is not active+clean 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.6 is not active+clean 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.729527+0000', 'last_change': '2024-09-13T16:28:19.729527+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.729527+0000', 'last_undegraded': '2024-09-13T16:28:19.729527+0000', 'last_fullsized': '2024-09-13T16:28:19.729527+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T18:18:34.445856+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.6 is not active+clean 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.932 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.5 is not active+clean 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.736343+0000', 'last_change': '2024-09-13T16:28:19.736343+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.736343+0000', 'last_undegraded': '2024-09-13T16:28:19.736343+0000', 'last_fullsized': '2024-09-13T16:28:19.736343+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T20:38:03.708069+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.7 is not active+clean 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.4 is not active+clean 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.732876+0000', 'last_change': '2024-09-13T16:28:19.732876+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.732876+0000', 'last_undegraded': '2024-09-13T16:28:19.732876+0000', 'last_fullsized': '2024-09-13T16:28:19.732876+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.0 is not active+clean 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.3 is not active+clean 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.734588+0000', 'last_change': '2024-09-13T16:28:19.734588+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.734588+0000', 'last_undegraded': '2024-09-13T16:28:19.734588+0000', 'last_fullsized': '2024-09-13T16:28:19.734588+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T21:08:02.929894+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.1 is not active+clean 2024-09-13T16:28:25.933 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.2 is not active+clean 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.733282+0000', 'last_change': '2024-09-13T16:28:19.733282+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.733282+0000', 'last_undegraded': '2024-09-13T16:28:19.733282+0000', 'last_fullsized': '2024-09-13T16:28:19.733282+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T22:44:46.314613+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.2 is not active+clean 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.1 is not active+clean 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.733922+0000', 'last_change': '2024-09-13T16:28:19.733922+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.733922+0000', 'last_undegraded': '2024-09-13T16:28:19.733922+0000', 'last_fullsized': '2024-09-13T16:28:19.733922+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-15T02:56:35.488631+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 6], 'acting': [3, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.3 is not active+clean 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.0 is not active+clean 2024-09-13T16:28:25.934 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.732879+0000', 'last_change': '2024-09-13T16:28:19.732879+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.732879+0000', 'last_undegraded': '2024-09-13T16:28:19.732879+0000', 'last_fullsized': '2024-09-13T16:28:19.732879+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-09-13T16:28:25.935 INFO:tasks.cephfs.filesystem.ceph_manager:PG 10.b is not active+clean 2024-09-13T16:28:25.935 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '10.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-09-13T16:28:20.725773+0000', 'last_change': '2024-09-13T16:28:20.725773+0000', 'last_active': '2024-09-13T16:28:20.725773+0000', 'last_peered': '2024-09-13T16:28:20.725773+0000', 'last_clean': '2024-09-13T16:28:20.725773+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:20.725773+0000', 'last_undegraded': '2024-09-13T16:28:20.725773+0000', 'last_fullsized': '2024-09-13T16:28:20.725773+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:20.725773+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-09-13T16:28:25.935 INFO:tasks.cephfs.filesystem.ceph_manager:PG 9.8 is not active+clean 2024-09-13T16:28:25.935 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '9.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 65, 'state': 'creating+peering', 'last_fresh': '2024-09-13T16:28:19.733012+0000', 'last_change': '2024-09-13T16:28:19.733012+0000', 'last_active': '2024-09-13T16:28:19.715810+0000', 'last_peered': '2024-09-13T16:28:19.715810+0000', 'last_clean': '2024-09-13T16:28:19.715810+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-09-13T16:28:19.733012+0000', 'last_undegraded': '2024-09-13T16:28:19.733012+0000', 'last_fullsized': '2024-09-13T16:28:19.733012+0000', 'mapping_epoch': 65, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 65, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'last_clean_scrub_stamp': '2024-09-13T16:28:19.715810+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-09-14T16:29:21.792298+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-09-13T16:28:25.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-13T16:28:26.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.045+0000 7f8f462fe700 1 Processor -- start 2024-09-13T16:28:26.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.045+0000 7f8f462fe700 1 -- start start 2024-09-13T16:28:26.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f400d1060 0x7f8f400d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8f400b3070 0x7f8f400b3520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f400d1a00 con 0x7f8f4015ccb0 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f400b2be0 con 0x7f8f400b3070 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400ce150 con 0x7f8f400d1060 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f45afd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8f400b3070 0x7f8f400b3520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f452fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8f4015ccb0 0x7f8f400cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48946/0 (socket says 172.21.15.119:48946) 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f452fc700 1 -- 172.21.15.119:0/2908390397 learned_addr learned my addr 172.21.15.119:0/2908390397 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f44afb700 1 --2- 172.21.15.119:0/2908390397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f400d1060 0x7f8f400d14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f45afd700 1 -- 172.21.15.119:0/2908390397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f400d1060 msgr2=0x7f8f400d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f45afd700 1 --2- 172.21.15.119:0/2908390397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f400d1060 0x7f8f400d14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f45afd700 1 -- 172.21.15.119:0/2908390397 >> v1:172.21.15.119:6789/0 conn(0x7f8f4015ccb0 legacy=0x7f8f400cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f45afd700 1 -- 172.21.15.119:0/2908390397 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f400ce430 con 0x7f8f400b3070 2024-09-13T16:28:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f45afd700 1 --2- 172.21.15.119:0/2908390397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400b3070 0x7f8f400b3520 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009d90 tx=0x7f8f3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc466c97f5effa40 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f37fff700 1 -- 172.21.15.119:0/2908390397 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f3800e070 con 0x7f8f400b3070 2024-09-13T16:28:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f37fff700 1 -- 172.21.15.119:0/2908390397 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f3800bbd0 con 0x7f8f400b3070 2024-09-13T16:28:26.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f37fff700 1 -- 172.21.15.119:0/2908390397 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f38012430 con 0x7f8f400b3070 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- 172.21.15.119:0/2908390397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400b3070 msgr2=0x7f8f400b3520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2908390397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400b3070 0x7f8f400b3520 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f8f38009d90 tx=0x7f8f3800b330 comp rx=0 tx=0).stop 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- 172.21.15.119:0/2908390397 shutdown_connections 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2908390397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400b3070 0x7f8f400b3520 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2908390397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f400d1060 0x7f8f400d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- 172.21.15.119:0/2908390397 >> 172.21.15.119:0/2908390397 conn(0x7f8f4008c0f0 msgr2=0x7f8f4008c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- 172.21.15.119:0/2908390397 shutdown_connections 2024-09-13T16:28:26.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.049+0000 7f8f462fe700 1 -- 172.21.15.119:0/2908390397 wait complete. 2024-09-13T16:28:26.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 Processor -- start 2024-09-13T16:28:26.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 -- start start 2024-09-13T16:28:26.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f400b3070 0x7f8f4015f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400d1060 0x7f8f400b82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f4015ccb0 0x7f8f401f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400c65d0 con 0x7f8f400d1060 2024-09-13T16:28:26.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f400c68b0 con 0x7f8f400b3070 2024-09-13T16:28:26.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f4015ee50 con 0x7f8f4015ccb0 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400d1060 0x7f8f400b82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400d1060 0x7f8f400b82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33782/0 (socket says 172.21.15.119:33782) 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 -- 172.21.15.119:0/2086719131 learned_addr learned my addr 172.21.15.119:0/2086719131 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f45afd700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f4015ccb0 0x7f8f401f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f452fc700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f400b3070 0x7f8f4015f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 -- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f4015ccb0 msgr2=0x7f8f401f5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f4015ccb0 0x7f8f401f5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 -- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f400b3070 msgr2=0x7f8f4015f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f400b3070 0x7f8f4015f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 -- 172.21.15.119:0/2086719131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f38009a70 con 0x7f8f400d1060 2024-09-13T16:28:26.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f44afb700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400d1060 0x7f8f400b82a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000e730 tx=0x7f8f3000eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f35ffb700 1 -- 172.21.15.119:0/2086719131 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f30004220 con 0x7f8f400d1060 2024-09-13T16:28:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f35ffb700 1 -- 172.21.15.119:0/2086719131 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f30004380 con 0x7f8f400d1060 2024-09-13T16:28:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f35ffb700 1 -- 172.21.15.119:0/2086719131 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f300036c0 con 0x7f8f400d1060 2024-09-13T16:28:26.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f4015f130 con 0x7f8f400d1060 2024-09-13T16:28:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.053+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f400c9490 con 0x7f8f400d1060 2024-09-13T16:28:26.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.057+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f400d0c90 con 0x7f8f400d1060 2024-09-13T16:28:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.057+0000 7f8f35ffb700 1 -- 172.21.15.119:0/2086719131 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8f30003820 con 0x7f8f400d1060 2024-09-13T16:28:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.057+0000 7f8f35ffb700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8f24066730 0x7f8f24068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.057+0000 7f8f35ffb700 1 -- 172.21.15.119:0/2086719131 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8f30083b20 con 0x7f8f400d1060 2024-09-13T16:28:26.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.057+0000 7f8f452fc700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8f24066730 0x7f8f24068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.057+0000 7f8f452fc700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8f24066730 0x7f8f24068bf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c005930 tx=0x7f8f3c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:26.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.069+0000 7f8f35ffb700 1 -- 172.21.15.119:0/2086719131 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8f30050de0 con 0x7f8f400d1060 2024-09-13T16:28:26.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f8f400b1c90 con 0x7f8f400d1060 2024-09-13T16:28:26.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f35ffb700 1 -- 172.21.15.119:0/2086719131 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1656 (secure 0 0 0) 0x7f8f30020690 con 0x7f8f400d1060 2024-09-13T16:28:26.309 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:26.309 INFO:teuthology.orchestra.run.smithi119.stdout:{"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":206,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":68,"num_osds":8,"num_up_osds":8,"osd_up_since":1726244710,"num_in_osds":8,"osd_in_since":1726244702,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":241385472,"bytes_avail":772852727808,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":32,"id":4,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":4,"rank":0,"name":"d","status":"up:active","gid":6546}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":9,"modified":"2024-09-13T16:27:39.469650+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-09-13T16:28:26.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8f24066730 msgr2=0x7f8f24068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8f24066730 0x7f8f24068bf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8f3c005930 tx=0x7f8f3c009040 comp rx=0 tx=0).stop 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400d1060 msgr2=0x7f8f400b82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400d1060 0x7f8f400b82a0 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f8f3000e730 tx=0x7f8f3000eaf0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 shutdown_connections 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8f24066730 0x7f8f24068bf0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8f4015ccb0 0x7f8f401f5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8f400d1060 0x7f8f400b82a0 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.305+0000 7f8f462fe700 1 --2- 172.21.15.119:0/2086719131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8f400b3070 0x7f8f4015f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.309+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 >> 172.21.15.119:0/2086719131 conn(0x7f8f4008c0f0 msgr2=0x7f8f400b4300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:26.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.309+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 shutdown_connections 2024-09-13T16:28:26.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.309+0000 7f8f462fe700 1 -- 172.21.15.119:0/2086719131 wait complete. 2024-09-13T16:28:26.324 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-09-13T16:28:26.324 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:28:26.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.429+0000 7fe6a74b4700 1 Processor -- start 2024-09-13T16:28:26.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.429+0000 7fe6a74b4700 1 -- start start 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a00b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe6a00b1e80 0x7fe6a0159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a00b2320 con 0x7fe6a00b3ad0 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6a00b2600 con 0x7fe6a00b1e80 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a00b28e0 con 0x7fe6a00b36f0 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe6a00b1e80 0x7fe6a0159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a5250700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a00b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe6a00b1e80 0x7fe6a0159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:42462/0 (socket says 172.21.15.119:42462) 2024-09-13T16:28:26.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 -- 172.21.15.119:0/1369861292 learned_addr learned my addr 172.21.15.119:0/1369861292 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:26.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 -- 172.21.15.119:0/1369861292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 msgr2=0x7fe6a00b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:26.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 --2- 172.21.15.119:0/1369861292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a00b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 -- 172.21.15.119:0/1369861292 >> v1:172.21.15.145:6789/0 conn(0x7fe6a00b3ad0 legacy=0x7fe6a015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 -- 172.21.15.119:0/1369861292 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a00b2bc0 con 0x7fe6a00b1e80 2024-09-13T16:28:26.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a4a4f700 1 --2- 172.21.15.119:0/1369861292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0159340 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009e50 tx=0x7fe69c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=70bfadd52d5dfa77 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:26.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe697fff700 1 -- 172.21.15.119:0/1369861292 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00e070 con 0x7fe6a00b1e80 2024-09-13T16:28:26.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe697fff700 1 -- 172.21.15.119:0/1369861292 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c00b910 con 0x7fe6a00b1e80 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe697fff700 1 -- 172.21.15.119:0/1369861292 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c00bcc0 con 0x7fe6a00b1e80 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- 172.21.15.119:0/1369861292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b1e80 msgr2=0x7fe6a0159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/1369861292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0159340 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe69c009e50 tx=0x7fe69c00b330 comp rx=0 tx=0).stop 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- 172.21.15.119:0/1369861292 shutdown_connections 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/1369861292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0159340 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/1369861292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a00b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- 172.21.15.119:0/1369861292 >> 172.21.15.119:0/1369861292 conn(0x7fe6a006a350 msgr2=0x7fe6a006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- 172.21.15.119:0/1369861292 shutdown_connections 2024-09-13T16:28:26.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.433+0000 7fe6a74b4700 1 -- 172.21.15.119:0/1369861292 wait complete. 2024-09-13T16:28:26.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 Processor -- start 2024-09-13T16:28:26.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 -- start start 2024-09-13T16:28:26.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0138e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a0070450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b3ad0 0x7fe6a0070990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a0159d00 con 0x7fe6a00b1e80 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a013bd40 con 0x7fe6a00b3ad0 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6a0159110 con 0x7fe6a00b36f0 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a5250700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0138e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a4a4f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a0070450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a5250700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0138e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33790/0 (socket says 172.21.15.119:33790) 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a5250700 1 -- 172.21.15.119:0/2846897572 learned_addr learned my addr 172.21.15.119:0/2846897572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:26.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a5a51700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b3ad0 0x7fe6a0070990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a4a4f700 1 -- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b3ad0 msgr2=0x7fe6a0070990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a4a4f700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b3ad0 0x7fe6a0070990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a4a4f700 1 -- 172.21.15.119:0/2846897572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6a00b1e80 msgr2=0x7fe6a0138e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a4a4f700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0138e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a4a4f700 1 -- 172.21.15.119:0/2846897572 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe69c009b30 con 0x7fe6a00b36f0 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a4a4f700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a0070450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe69c015000 tx=0x7fe69c004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe695ffb700 1 -- 172.21.15.119:0/2846897572 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c0048c0 con 0x7fe6a00b36f0 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe695ffb700 1 -- 172.21.15.119:0/2846897572 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe69c0092e0 con 0x7fe6a00b36f0 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.437+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a01593d0 con 0x7fe6a00b36f0 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.441+0000 7fe695ffb700 1 -- 172.21.15.119:0/2846897572 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe69c01c680 con 0x7fe6a00b36f0 2024-09-13T16:28:26.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.441+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a0159990 con 0x7fe6a00b36f0 2024-09-13T16:28:26.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.441+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a0159b20 con 0x7fe6a00b36f0 2024-09-13T16:28:26.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.441+0000 7fe695ffb700 1 -- 172.21.15.119:0/2846897572 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe69c01c7e0 con 0x7fe6a00b36f0 2024-09-13T16:28:26.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.441+0000 7fe695ffb700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe684066730 0x7fe684068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:26.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.441+0000 7fe695ffb700 1 -- 172.21.15.119:0/2846897572 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe69c085c60 con 0x7fe6a00b36f0 2024-09-13T16:28:26.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.441+0000 7fe6a5250700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe684066730 0x7fe684068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:26.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.445+0000 7fe6a5250700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe684066730 0x7fe684068bf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe690009cb0 tx=0x7fe690009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:26.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.449+0000 7fe695ffb700 1 -- 172.21.15.119:0/2846897572 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe69c053040 con 0x7fe6a00b36f0 2024-09-13T16:28:26.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.617+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe6a0159e70 con 0x7fe684066730 2024-09-13T16:28:26.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.637+0000 7fe695ffb700 1 -- 172.21.15.119:0/2846897572 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+276877 (secure 0 0 0) 0x7fe6a0159e70 con 0x7fe684066730 2024-09-13T16:28:26.645 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe684066730 msgr2=0x7fe684068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe684066730 0x7fe684068bf0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe690009cb0 tx=0x7fe690009240 comp rx=0 tx=0).stop 2024-09-13T16:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 msgr2=0x7fe6a0070450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a0070450 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe69c015000 tx=0x7fe69c004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 shutdown_connections 2024-09-13T16:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe684066730 0x7fe684068bf0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6a00b3ad0 0x7fe6a0070990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6a00b36f0 0x7fe6a0070450 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 --2- 172.21.15.119:0/2846897572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6a00b1e80 0x7fe6a0138e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:26.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 >> 172.21.15.119:0/2846897572 conn(0x7fe6a006a350 msgr2=0x7fe6a006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:26.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 shutdown_connections 2024-09-13T16:28:26.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:26.641+0000 7fe6a74b4700 1 -- 172.21.15.119:0/2846897572 wait complete. 2024-09-13T16:28:26.649 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:28:26.660 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-09-13T16:28:25.485014+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":9,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":235728,"kb_used_data":7260,"kb_used_omap":12,"kb_used_meta":228339,"kb_avail":754738992,"statfs":{"total":773094113280,"available":772852727808,"internally_reserved":0,"allocated":7434240,"data_stored":3951767,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":233819705},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.758017"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733380+0000","last_change":"2024-09-13T16:28:19.733380+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733380+0000","last_undegraded":"2024-09-13T16:28:19.733380+0000","last_fullsized":"2024-09-13T16:28:19.733380+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736612+0000","last_change":"2024-09-13T16:28:19.736612+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736612+0000","last_undegraded":"2024-09-13T16:28:19.736612+0000","last_fullsized":"2024-09-13T16:28:19.736612+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:07.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.737602+0000","last_change":"2024-09-13T16:28:19.737602+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.737602+0000","last_undegraded":"2024-09-13T16:28:19.737602+0000","last_fullsized":"2024-09-13T16:28:19.737602+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:59.677392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736454+0000","last_change":"2024-09-13T16:28:19.736454+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736454+0000","last_undegraded":"2024-09-13T16:28:19.736454+0000","last_fullsized":"2024-09-13T16:28:19.736454+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734150+0000","last_change":"2024-09-13T16:28:19.734150+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734150+0000","last_undegraded":"2024-09-13T16:28:19.734150+0000","last_fullsized":"2024-09-13T16:28:19.734150+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:43.928576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734089+0000","last_change":"2024-09-13T16:28:19.734089+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734089+0000","last_undegraded":"2024-09-13T16:28:19.734089+0000","last_fullsized":"2024-09-13T16:28:19.734089+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:19.660866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734751+0000","last_change":"2024-09-13T16:28:19.734751+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734751+0000","last_undegraded":"2024-09-13T16:28:19.734751+0000","last_fullsized":"2024-09-13T16:28:19.734751+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.740412+0000","last_change":"2024-09-13T16:28:19.740412+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.740412+0000","last_undegraded":"2024-09-13T16:28:19.740412+0000","last_fullsized":"2024-09-13T16:28:19.740412+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:24.851511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734673+0000","last_change":"2024-09-13T16:28:19.734673+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734673+0000","last_undegraded":"2024-09-13T16:28:19.734673+0000","last_fullsized":"2024-09-13T16:28:19.734673+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730285+0000","last_change":"2024-09-13T16:28:19.730285+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730285+0000","last_undegraded":"2024-09-13T16:28:19.730285+0000","last_fullsized":"2024-09-13T16:28:19.730285+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:28.409095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734827+0000","last_change":"2024-09-13T16:28:19.734827+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734827+0000","last_undegraded":"2024-09-13T16:28:19.734827+0000","last_fullsized":"2024-09-13T16:28:19.734827+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733583+0000","last_change":"2024-09-13T16:28:19.733583+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733583+0000","last_undegraded":"2024-09-13T16:28:19.733583+0000","last_fullsized":"2024-09-13T16:28:19.733583+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735789+0000","last_change":"2024-09-13T16:28:19.735789+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735789+0000","last_undegraded":"2024-09-13T16:28:19.735789+0000","last_fullsized":"2024-09-13T16:28:19.735789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730486+0000","last_change":"2024-09-13T16:28:19.730486+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730486+0000","last_undegraded":"2024-09-13T16:28:19.730486+0000","last_fullsized":"2024-09-13T16:28:19.730486+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:09.620313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734680+0000","last_change":"2024-09-13T16:28:19.734680+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734680+0000","last_undegraded":"2024-09-13T16:28:19.734680+0000","last_fullsized":"2024-09-13T16:28:19.734680+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729778+0000","last_change":"2024-09-13T16:28:19.729778+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729778+0000","last_undegraded":"2024-09-13T16:28:19.729778+0000","last_fullsized":"2024-09-13T16:28:19.729778+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:58.987147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.730913+0000","last_change":"2024-09-13T16:28:19.730913+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.730913+0000","last_undegraded":"2024-09-13T16:28:19.730913+0000","last_fullsized":"2024-09-13T16:28:19.730913+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:14.580434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735686+0000","last_change":"2024-09-13T16:28:19.735686+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735686+0000","last_undegraded":"2024-09-13T16:28:19.735686+0000","last_fullsized":"2024-09-13T16:28:19.735686+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:04.745546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.735766+0000","last_change":"2024-09-13T16:28:19.735766+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.735766+0000","last_undegraded":"2024-09-13T16:28:19.735766+0000","last_fullsized":"2024-09-13T16:28:19.735766+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:26.842748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733365+0000","last_change":"2024-09-13T16:28:19.733365+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733365+0000","last_undegraded":"2024-09-13T16:28:19.733365+0000","last_fullsized":"2024-09-13T16:28:19.733365+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:21.436784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736501+0000","last_change":"2024-09-13T16:28:19.736501+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736501+0000","last_undegraded":"2024-09-13T16:28:19.736501+0000","last_fullsized":"2024-09-13T16:28:19.736501+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:47.232880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:31.462343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736347+0000","last_change":"2024-09-13T16:28:19.736347+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736347+0000","last_undegraded":"2024-09-13T16:28:19.736347+0000","last_fullsized":"2024-09-13T16:28:19.736347+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:40.488362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729438+0000","last_change":"2024-09-13T16:28:19.729438+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729438+0000","last_undegraded":"2024-09-13T16:28:19.729438+0000","last_fullsized":"2024-09-13T16:28:19.729438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:30.820069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.729527+0000","last_change":"2024-09-13T16:28:19.729527+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.729527+0000","last_undegraded":"2024-09-13T16:28:19.729527+0000","last_fullsized":"2024-09-13T16:28:19.729527+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:34.445856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.736343+0000","last_change":"2024-09-13T16:28:19.736343+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.736343+0000","last_undegraded":"2024-09-13T16:28:19.736343+0000","last_fullsized":"2024-09-13T16:28:19.736343+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:03.708069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732876+0000","last_change":"2024-09-13T16:28:19.732876+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732876+0000","last_undegraded":"2024-09-13T16:28:19.732876+0000","last_fullsized":"2024-09-13T16:28:19.732876+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.734588+0000","last_change":"2024-09-13T16:28:19.734588+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.734588+0000","last_undegraded":"2024-09-13T16:28:19.734588+0000","last_fullsized":"2024-09-13T16:28:19.734588+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:02.929894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733282+0000","last_change":"2024-09-13T16:28:19.733282+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733282+0000","last_undegraded":"2024-09-13T16:28:19.733282+0000","last_fullsized":"2024-09-13T16:28:19.733282+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:46.314613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733922+0000","last_change":"2024-09-13T16:28:19.733922+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733922+0000","last_undegraded":"2024-09-13T16:28:19.733922+0000","last_fullsized":"2024-09-13T16:28:19.733922+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:35.488631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.732879+0000","last_change":"2024-09-13T16:28:19.732879+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.732879+0000","last_undegraded":"2024-09-13T16:28:19.732879+0000","last_fullsized":"2024-09-13T16:28:19.732879+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-09-13T16:28:20.725773+0000","last_change":"2024-09-13T16:28:20.725773+0000","last_active":"2024-09-13T16:28:20.725773+0000","last_peered":"2024-09-13T16:28:20.725773+0000","last_clean":"2024-09-13T16:28:20.725773+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:20.725773+0000","last_undegraded":"2024-09-13T16:28:20.725773+0000","last_fullsized":"2024-09-13T16:28:20.725773+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1,"reported_epoch":65,"state":"creating+peering","last_fresh":"2024-09-13T16:28:19.733012+0000","last_change":"2024-09-13T16:28:19.733012+0000","last_active":"2024-09-13T16:28:19.715810+0000","last_peered":"2024-09-13T16:28:19.715810+0000","last_clean":"2024-09-13T16:28:19.715810+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-13T16:28:19.733012+0000","last_undegraded":"2024-09-13T16:28:19.733012+0000","last_fullsized":"2024-09-13T16:28:19.733012+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:21.792298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":200,"reported_epoch":64,"state":"active+clean","last_fresh":"2024-09-13T16:28:17.717312+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:28:17.717312+0000","last_peered":"2024-09-13T16:28:17.717312+0000","last_clean":"2024-09-13T16:28:17.717312+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:28:17.717312+0000","last_undegraded":"2024-09-13T16:28:17.717312+0000","last_fullsized":"2024-09-13T16:28:17.717312+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542185,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32460,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339380,"statfs":{"total":96636764160,"available":96603525120,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.86199999999999999,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.86099999999999999,"15min":0.84899999999999998},"min":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.085,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.71699999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.60299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0880000000000001,"5min":1.121,"15min":1.121},"last":1},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.84799999999999998},"min":{"1min":0.70599999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.131,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.55000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2170000000000001,"5min":1.228,"15min":1.228},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0389999999999999,"5min":1.091,"15min":1.091},"last":0.85799999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1220000000000001,"5min":1.238,"15min":1.238},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.997,"15min":1.0069999999999999},"min":{"1min":0.746,"5min":0.745,"15min":0.745},"max":{"1min":1.214,"5min":1.226,"15min":1.226},"last":1.101}]},{"osd":4,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88700000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.0580000000000001,"15min":1.0880000000000001},"min":{"1min":0.77800000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1870000000000001,"5min":1.589,"15min":1.589},"last":0.82799999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.996,"15min":1.03},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1439999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0589999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1120000000000001,"15min":1.1200000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.244,"5min":1.337,"15min":1.337},"last":0.90600000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.077,"15min":1.087},"min":{"1min":0.85599999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.196,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":1.109,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.99299999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.2609999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":0.95099999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.85199999999999998},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.3069999999999999},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.628,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.0860000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0780000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.016,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.97499999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2110000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.264},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0840000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.89200000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69499999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.248,"5min":1.536,"15min":1.536},"last":1.228},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.98699999999999999,"15min":0.98899999999999999},"min":{"1min":0.71799999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.264,"5min":1.409,"15min":1.409},"last":1.202}]},{"osd":4,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.96299999999999997,"15min":0.97699999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1910000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.365},{"interface":"front","average":{"1min":1.056,"5min":0.97399999999999998,"15min":0.94399999999999995},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.80100000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91900000000000004,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1220000000000001,"15min":1.1379999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.75}]},{"osd":7,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":1.1579999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0509999999999999,"15min":1.056},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.3149999999999999,"5min":1.337,"15min":1.337},"last":1.133}]}]},{"osd":5,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31880,"kb_used_data":752,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":770048,"data_stored":341009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.5,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.131,"5min":1.675,"15min":1.675},"last":0.73999999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.876,"15min":0.88400000000000001},"min":{"1min":0.66500000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.321,"5min":1.369,"15min":1.369},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.84899999999999998,"15min":0.83099999999999996},"min":{"1min":0.66400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.169,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93000000000000005,"15min":0.93700000000000006},"min":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.032,"5min":1.625,"15min":1.625},"last":1.133},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.97299999999999998,"15min":0.98699999999999999},"min":{"1min":0.627,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5129999999999999,"5min":1.552,"15min":1.552},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85599999999999998,"15min":0.83799999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1639999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.86299999999999999},{"interface":"front","average":{"1min":0.879,"5min":0.97199999999999998,"15min":1.016},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1819999999999999,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":1.0489999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.0720000000000001},"min":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.579,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.81200000000000006},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.85799999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2210000000000001,"5min":1.601,"15min":1.601},"last":0.93100000000000005}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96099999999999997,"15min":0.97299999999999998},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.179,"5min":1.421,"15min":1.421},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0740000000000001,"15min":1.0780000000000001},"min":{"1min":0.76000000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":1.101}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.036},"min":{"1min":0.68799999999999994,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3180000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":1.006},{"interface":"front","average":{"1min":1.034,"5min":1.175,"15min":1.2290000000000001},"min":{"1min":0.78900000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.266,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.51}]}]},{"osd":4,"up_from":14,"seq":60129542184,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.84199999999999997,"15min":0.83799999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.98599999999999999},{"interface":"front","average":{"1min":0.83499999999999996,"5min":0.84699999999999998,"15min":0.84899999999999998},"min":{"1min":0.59899999999999998,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.508,"5min":1.508,"15min":1.508},"last":0.60599999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.91000000000000003,"15min":0.91400000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.3049999999999999,"5min":1.8160000000000001,"15min":1.8160000000000001},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.92600000000000005,"5min":0.91800000000000004,"15min":0.91600000000000004},"min":{"1min":0.72799999999999998,"5min":0.61899999999999999,"15min":0.61899999999999999},"max":{"1min":1.26,"5min":1.353,"15min":1.353},"last":0.71199999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.91700000000000004,"5min":0.85199999999999998,"15min":0.84099999999999997},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":1.091},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.95699999999999996,"15min":0.96299999999999997},"min":{"1min":0.61099999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.2749999999999999,"5min":1.603,"15min":1.603},"last":0.75800000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94299999999999995,"5min":0.95599999999999996,"15min":0.95799999999999996},"min":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.298,"5min":1.472,"15min":1.472},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.96199999999999997,"15min":0.96499999999999997},"min":{"1min":0.55100000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.577,"5min":1.577,"15min":1.577},"last":1.1419999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":0.98399999999999999,"15min":0.97999999999999998},"min":{"1min":0.70099999999999996,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":1.573,"5min":2.048,"15min":2.048},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.127,"5min":1.1479999999999999,"15min":1.1519999999999999},"min":{"1min":0.83699999999999997,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.4179999999999999,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.86199999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":0.95599999999999996,"15min":0.94599999999999995},"min":{"1min":0.749,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6499999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.014},{"interface":"front","average":{"1min":1.202,"5min":1.1220000000000001,"15min":1.1080000000000001},"min":{"1min":0.876,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.74,"5min":2.1890000000000001,"15min":2.1890000000000001},"last":0.94399999999999995}]},{"osd":7,"last update":"Fri Sep 13 16:27:19 2024","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.165,"15min":1.153},"min":{"1min":0.95999999999999996,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.1230000000000002,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":1.054},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.1319999999999999,"15min":1.123},"min":{"1min":0.85199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.8109999999999999,"5min":1.8109999999999999,"15min":1.8109999999999999},"last":1.1870000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27784,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344056,"statfs":{"total":96636764160,"available":96608313344,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.96099999999999997,"5min":1.04,"15min":1.04},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.498,"5min":0.433,"15min":0.433},"max":{"1min":0.96999999999999997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.378,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2310000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.64500000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68500000000000005,"15min":0.69599999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":1.27,"15min":1.27},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.755},"min":{"1min":0.56399999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.139,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.70899999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.214,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.85499999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95899999999999996,"15min":0.96199999999999997},"min":{"1min":0.66200000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.538,"5min":1.782,"15min":1.782},"last":1.0069999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97699999999999998,"15min":0.98199999999999998},"min":{"1min":0.71899999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.046,"5min":1.0920000000000001,"15min":1.1020000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.484,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.98099999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.119,"15min":1.137},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.571,"5min":1.885,"15min":1.885},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.032,"15min":1.0580000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.268,"5min":1.95,"15min":1.95},"last":1.0369999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.579,"5min":1.889,"15min":1.889},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0580000000000001,"15min":1.0609999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":0.89000000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542186,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":31880,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339960,"statfs":{"total":96636764160,"available":96604119040,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":1.0960000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.42099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96699999999999997,"5min":0.997,"15min":0.997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.68799999999999994},"min":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88500000000000001,"5min":1.123,"15min":1.123},"last":0.69299999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.47699999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86899999999999999,"5min":1.073,"15min":1.073},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.78800000000000003},"min":{"1min":0.64300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.0740000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.94899999999999995,"15min":0.95499999999999996},"min":{"1min":0.70399999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2809999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":1.052},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95699999999999996,"15min":0.98399999999999999},"min":{"1min":0.78000000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.367,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.1200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.72599999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.153,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.78400000000000003},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.72799999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.234,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.92400000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.014,"15min":1.042},"min":{"1min":0.73899999999999999,"5min":0.623,"15min":0.623},"max":{"1min":1.2729999999999999,"5min":2.024,"15min":2.024},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0609999999999999,"15min":1.0669999999999999},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4930000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.95499999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0669999999999999,"15min":1.081},"min":{"1min":0.78900000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.252,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.0569999999999999,"15min":1.071},"min":{"1min":0.81499999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.4430000000000001,"5min":1.55,"15min":1.55},"last":1.01}]}]},{"osd":1,"up_from":14,"seq":60129542185,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":764,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":782336,"data_stored":347146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.51200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.371},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.67400000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0209999999999999,"5min":1.141,"15min":1.141},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.64500000000000002,"15min":0.61699999999999999},"min":{"1min":0.54700000000000004,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70299999999999996},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90600000000000003,"5min":1.458,"15min":1.458},"last":0.89000000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.72599999999999998,"15min":0.69099999999999995},"min":{"1min":0.52300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.026,"5min":1.236,"15min":1.236},"last":0.68799999999999994},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94799999999999995,"5min":1.403,"15min":1.403},"last":0.44400000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.92500000000000004,"15min":0.90700000000000003},"min":{"1min":0.81699999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2569999999999999,"5min":1.546,"15min":1.546},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3089999999999999,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.76200000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95499999999999996,"15min":0.94699999999999995},"min":{"1min":0.80200000000000005,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.0289999999999999},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.97099999999999997,"15min":0.98199999999999998},"min":{"1min":0.77700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3979999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.873}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.70899999999999996,"5min":0.626,"15min":0.626},"max":{"1min":1.4139999999999999,"5min":1.613,"15min":1.613},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.026,"5min":1.014,"15min":1.012},"min":{"1min":0.746,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3480000000000001,"5min":1.698,"15min":1.698},"last":0.99199999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.008,"15min":1.01},"min":{"1min":0.748,"5min":0.62,"15min":0.62},"max":{"1min":1.494,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.054,"15min":1.042},"min":{"1min":0.873,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.448,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.96099999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542185,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28364,"kb_used_data":1344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343476,"statfs":{"total":96636764160,"available":96607719424,"internally_reserved":0,"allocated":1376256,"data_stored":937514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0549999999999999,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.64300000000000002,"15min":0.65000000000000002},"min":{"1min":0.42399999999999999,"5min":0.377,"15min":0.377},"max":{"1min":0.75600000000000001,"5min":0.94799999999999995,"15min":0.94799999999999995},"last":0.90100000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.71199999999999997,"15min":0.71999999999999997},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":0.93899999999999995,"5min":1.1930000000000001,"15min":1.1930000000000001},"last":1.202},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.92900000000000005,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":1.0249999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.77200000000000002,"5min":0.66000000000000003,"15min":0.64100000000000001},"min":{"1min":0.45500000000000002,"5min":0.312,"15min":0.312},"max":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.81000000000000005,"15min":0.81399999999999995},"min":{"1min":0.45300000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1639999999999999,"5min":1.1639999999999999,"15min":1.1639999999999999},"last":1.6100000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.92500000000000004,"5min":0.95499999999999996,"15min":0.95999999999999996},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.2949999999999999,"5min":1.3109999999999999,"15min":1.3109999999999999},"last":1.2370000000000001},{"interface":"front","average":{"1min":1.04,"5min":0.96999999999999997,"15min":0.95799999999999996},"min":{"1min":0.68700000000000006,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.704,"5min":1.704,"15min":1.704},"last":1.5720000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.96299999999999997,"15min":0.96399999999999997},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.2,"5min":1.24,"15min":1.24},"last":1.5409999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.99299999999999999,"15min":0.999},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.198,"5min":1.5489999999999999,"15min":1.5489999999999999},"last":1.6819999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.0980000000000001,"5min":0.995,"15min":0.97799999999999998},"min":{"1min":0.80900000000000005,"5min":0.628,"15min":0.628},"max":{"1min":1.944,"5min":1.944,"15min":1.944},"last":1.101},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.036,"15min":1.04},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"last":1.345}]},{"osd":7,"last update":"Fri Sep 13 16:27:18 2024","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.054,"15min":1.046},"min":{"1min":0.90000000000000002,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":1.492},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.0620000000000001,"15min":1.038},"min":{"1min":0.91400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":2.0139999999999998,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.371}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:28:27.059 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-09-13T16:28:27.056+0000 7f9e49498700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:28:27.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:28:27.235 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:28:29.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:28:29.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.785+0000 7f9bf3412700 1 Processor -- start 2024-09-13T16:28:29.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.785+0000 7f9bf3412700 1 -- start start 2024-09-13T16:28:29.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf3412700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec0b3800 0x7f9bec1642b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:29.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf3412700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec0b2280 con 0x7f9bec1594f0 2024-09-13T16:28:29.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf3412700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec0cc890 con 0x7f9bec06dc40 2024-09-13T16:28:29.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf3412700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec070c10 con 0x7f9bec0b3800 2024-09-13T16:28:29.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf2410700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9bec1594f0 0x7f9bec06d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48962/0 (socket says 172.21.15.119:48962) 2024-09-13T16:28:29.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf2410700 1 -- 172.21.15.119:0/3433055586 learned_addr learned my addr 172.21.15.119:0/3433055586 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:29.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf2c11700 1 --2- 172.21.15.119:0/3433055586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec0b3800 0x7f9bec1642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf140e700 1 -- 172.21.15.119:0/3433055586 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430573813 0 0) 0x7f9bec0b2280 con 0x7f9bec1594f0 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf140e700 1 -- 172.21.15.119:0/3433055586 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd80035d0 con 0x7f9bec1594f0 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf140e700 1 -- 172.21.15.119:0/3433055586 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694655126 0 0) 0x7f9bec0cc890 con 0x7f9bec06dc40 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf140e700 1 -- 172.21.15.119:0/3433055586 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bec0b2280 con 0x7f9bec06dc40 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf2c11700 1 -- 172.21.15.119:0/3433055586 >> v1:172.21.15.145:6789/0 conn(0x7f9bec06dc40 legacy=0x7f9bec0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf2c11700 1 -- 172.21.15.119:0/3433055586 >> v1:172.21.15.119:6789/0 conn(0x7f9bec1594f0 legacy=0x7f9bec06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf2c11700 1 -- 172.21.15.119:0/3433055586 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec0b1970 con 0x7f9bec0b3800 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf2c11700 1 --2- 172.21.15.119:0/3433055586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec0b3800 0x7f9bec1642b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009dc0 tx=0x7f9be400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ce56232bba83e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf140e700 1 -- 172.21.15.119:0/3433055586 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be400e070 con 0x7f9bec0b3800 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf140e700 1 -- 172.21.15.119:0/3433055586 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be400bb20 con 0x7f9bec0b3800 2024-09-13T16:28:29.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf140e700 1 -- 172.21.15.119:0/3433055586 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be400be90 con 0x7f9bec0b3800 2024-09-13T16:28:29.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf3412700 1 -- 172.21.15.119:0/3433055586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec0b3800 msgr2=0x7f9bec1642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:29.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.789+0000 7f9bf3412700 1 --2- 172.21.15.119:0/3433055586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec0b3800 0x7f9bec1642b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9be4009dc0 tx=0x7f9be400b330 comp rx=0 tx=0).stop 2024-09-13T16:28:29.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.793+0000 7f9bf3412700 1 -- 172.21.15.119:0/3433055586 shutdown_connections 2024-09-13T16:28:29.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.793+0000 7f9bf3412700 1 --2- 172.21.15.119:0/3433055586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec0b3800 0x7f9bec1642b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:29.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.793+0000 7f9bf3412700 1 -- 172.21.15.119:0/3433055586 >> 172.21.15.119:0/3433055586 conn(0x7f9bec06a350 msgr2=0x7f9bec06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:29.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.793+0000 7f9bf3412700 1 -- 172.21.15.119:0/3433055586 shutdown_connections 2024-09-13T16:28:29.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.793+0000 7f9bf3412700 1 -- 172.21.15.119:0/3433055586 wait complete. 2024-09-13T16:28:29.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.793+0000 7f9bf3412700 1 Processor -- start 2024-09-13T16:28:29.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.793+0000 7f9bf3412700 1 -- start start 2024-09-13T16:28:29.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec06dc40 0x7f9bec15c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bec0b3800 0x7f9bec0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bec1594f0 0x7f9bec0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec149ab0 con 0x7f9bec1594f0 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0ca2f0 con 0x7f9bec0b3800 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec15d790 con 0x7f9bec06dc40 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bec1594f0 0x7f9bec0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bec1594f0 0x7f9bec0ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33796/0 (socket says 172.21.15.119:33796) 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 -- 172.21.15.119:0/1801542873 learned_addr learned my addr 172.21.15.119:0/1801542873 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2410700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec06dc40 0x7f9bec15c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf1c0f700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bec0b3800 0x7f9bec0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 -- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec06dc40 msgr2=0x7f9bec15c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:29.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec06dc40 0x7f9bec15c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 -- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bec0b3800 msgr2=0x7f9bec0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bec0b3800 0x7f9bec0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 -- 172.21.15.119:0/1801542873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4009a70 con 0x7f9bec1594f0 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf1c0f700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bec0b3800 0x7f9bec0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf2c11700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bec1594f0 0x7f9bec0ace00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9be4013000 tx=0x7f9be4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bdeffd700 1 -- 172.21.15.119:0/1801542873 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4003c40 con 0x7f9bec1594f0 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bdeffd700 1 -- 172.21.15.119:0/1801542873 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be401f070 con 0x7f9bec1594f0 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec149c90 con 0x7f9bec1594f0 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bdeffd700 1 -- 172.21.15.119:0/1801542873 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be40098f0 con 0x7f9bec1594f0 2024-09-13T16:28:29.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.797+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec15d9a0 con 0x7f9bec1594f0 2024-09-13T16:28:29.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.801+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec0b30f0 con 0x7f9bec1594f0 2024-09-13T16:28:29.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.801+0000 7f9bdeffd700 1 -- 172.21.15.119:0/1801542873 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9be4004030 con 0x7f9bec1594f0 2024-09-13T16:28:29.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.801+0000 7f9bdeffd700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bd0066760 0x7f9bd0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:29.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.801+0000 7f9bdeffd700 1 -- 172.21.15.119:0/1801542873 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9be40859c0 con 0x7f9bec1594f0 2024-09-13T16:28:29.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.801+0000 7f9bf2410700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bd0066760 0x7f9bd0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:29.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.801+0000 7f9bf2410700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bd0066760 0x7f9bd0068c20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9be8000f60 tx=0x7f9be8007e20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:29.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.809+0000 7f9bdeffd700 1 -- 172.21.15.119:0/1801542873 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9be4056710 con 0x7f9bec1594f0 2024-09-13T16:28:29.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.973+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bec04e700 con 0x7f9bd0066760 2024-09-13T16:28:29.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.989+0000 7f9bdeffd700 1 -- 172.21.15.119:0/1801542873 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287513 (secure 0 0 0) 0x7f9bec04e700 con 0x7f9bd0066760 2024-09-13T16:28:30.000 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bd0066760 msgr2=0x7f9bd0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bd0066760 0x7f9bd0068c20 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9be8000f60 tx=0x7f9be8007e20 comp rx=0 tx=0).stop 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bec1594f0 msgr2=0x7f9bec0ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bec1594f0 0x7f9bec0ace00 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f9be4013000 tx=0x7f9be4004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 shutdown_connections 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bd0066760 0x7f9bd0068c20 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bec1594f0 0x7f9bec0ace00 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9bec0b3800 0x7f9bec0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 --2- 172.21.15.119:0/1801542873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9bec06dc40 0x7f9bec15c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 >> 172.21.15.119:0/1801542873 conn(0x7f9bec06a350 msgr2=0x7f9bec06ced0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 shutdown_connections 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:29.997+0000 7f9bf3412700 1 -- 172.21.15.119:0/1801542873 wait complete. 2024-09-13T16:28:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:28:30.014 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":161,"stamp":"2024-09-13T16:28:29.486531+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":240200,"kb_used_data":7624,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754734520,"statfs":{"total":773094113280,"available":772848148480,"internally_reserved":0,"allocated":7806976,"data_stored":4125044,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.363786"},"pg_stats":[{"pgid":"10.3f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133238+0000","last_change":"2024-09-13T16:28:21.140571+0000","last_active":"2024-09-13T16:28:22.133238+0000","last_peered":"2024-09-13T16:28:22.133238+0000","last_clean":"2024-09-13T16:28:22.133238+0000","last_became_active":"2024-09-13T16:28:21.140341+0000","last_became_peered":"2024-09-13T16:28:21.140341+0000","last_unstale":"2024-09-13T16:28:22.133238+0000","last_undegraded":"2024-09-13T16:28:22.133238+0000","last_fullsized":"2024-09-13T16:28:22.133238+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:16.792890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.132757+0000","last_change":"2024-09-13T16:28:21.133753+0000","last_active":"2024-09-13T16:28:22.132757+0000","last_peered":"2024-09-13T16:28:22.132757+0000","last_clean":"2024-09-13T16:28:22.132757+0000","last_became_active":"2024-09-13T16:28:21.133442+0000","last_became_peered":"2024-09-13T16:28:21.133442+0000","last_unstale":"2024-09-13T16:28:22.132757+0000","last_undegraded":"2024-09-13T16:28:22.132757+0000","last_fullsized":"2024-09-13T16:28:22.132757+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:58:57.506087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.3d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.593217+0000","last_change":"2024-09-13T16:28:21.148975+0000","last_active":"2024-09-13T16:28:22.593217+0000","last_peered":"2024-09-13T16:28:22.593217+0000","last_clean":"2024-09-13T16:28:22.593217+0000","last_became_active":"2024-09-13T16:28:21.148711+0000","last_became_peered":"2024-09-13T16:28:21.148711+0000","last_unstale":"2024-09-13T16:28:22.593217+0000","last_undegraded":"2024-09-13T16:28:22.593217+0000","last_fullsized":"2024-09-13T16:28:22.593217+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:55.164739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.3c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133837+0000","last_change":"2024-09-13T16:28:21.147151+0000","last_active":"2024-09-13T16:28:22.133837+0000","last_peered":"2024-09-13T16:28:22.133837+0000","last_clean":"2024-09-13T16:28:22.133837+0000","last_became_active":"2024-09-13T16:28:21.146860+0000","last_became_peered":"2024-09-13T16:28:21.146860+0000","last_unstale":"2024-09-13T16:28:22.133837+0000","last_undegraded":"2024-09-13T16:28:22.133837+0000","last_fullsized":"2024-09-13T16:28:22.133837+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.901645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.3b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.593182+0000","last_change":"2024-09-13T16:28:21.156890+0000","last_active":"2024-09-13T16:28:22.593182+0000","last_peered":"2024-09-13T16:28:22.593182+0000","last_clean":"2024-09-13T16:28:22.593182+0000","last_became_active":"2024-09-13T16:28:21.156561+0000","last_became_peered":"2024-09-13T16:28:21.156561+0000","last_unstale":"2024-09-13T16:28:22.593182+0000","last_undegraded":"2024-09-13T16:28:22.593182+0000","last_fullsized":"2024-09-13T16:28:22.593182+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:03.460928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.3a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.485136+0000","last_change":"2024-09-13T16:28:21.145665+0000","last_active":"2024-09-13T16:28:22.485136+0000","last_peered":"2024-09-13T16:28:22.485136+0000","last_clean":"2024-09-13T16:28:22.485136+0000","last_became_active":"2024-09-13T16:28:21.145387+0000","last_became_peered":"2024-09-13T16:28:21.145387+0000","last_unstale":"2024-09-13T16:28:22.485136+0000","last_undegraded":"2024-09-13T16:28:22.485136+0000","last_fullsized":"2024-09-13T16:28:22.485136+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:47:26.017625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.39","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.132236+0000","last_change":"2024-09-13T16:28:21.152070+0000","last_active":"2024-09-13T16:28:22.132236+0000","last_peered":"2024-09-13T16:28:22.132236+0000","last_clean":"2024-09-13T16:28:22.132236+0000","last_became_active":"2024-09-13T16:28:21.151802+0000","last_became_peered":"2024-09-13T16:28:21.151802+0000","last_unstale":"2024-09-13T16:28:22.132236+0000","last_undegraded":"2024-09-13T16:28:22.132236+0000","last_fullsized":"2024-09-13T16:28:22.132236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:30.714051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.38","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.589951+0000","last_change":"2024-09-13T16:28:21.144798+0000","last_active":"2024-09-13T16:28:22.589951+0000","last_peered":"2024-09-13T16:28:22.589951+0000","last_clean":"2024-09-13T16:28:22.589951+0000","last_became_active":"2024-09-13T16:28:21.144553+0000","last_became_peered":"2024-09-13T16:28:21.144553+0000","last_unstale":"2024-09-13T16:28:22.589951+0000","last_undegraded":"2024-09-13T16:28:22.589951+0000","last_fullsized":"2024-09-13T16:28:22.589951+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:24.313372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.37","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.592236+0000","last_change":"2024-09-13T16:28:21.157255+0000","last_active":"2024-09-13T16:28:22.592236+0000","last_peered":"2024-09-13T16:28:22.592236+0000","last_clean":"2024-09-13T16:28:22.592236+0000","last_became_active":"2024-09-13T16:28:21.156998+0000","last_became_peered":"2024-09-13T16:28:21.156998+0000","last_unstale":"2024-09-13T16:28:22.592236+0000","last_undegraded":"2024-09-13T16:28:22.592236+0000","last_fullsized":"2024-09-13T16:28:22.592236+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:00.991635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.36","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591796+0000","last_change":"2024-09-13T16:28:21.148572+0000","last_active":"2024-09-13T16:28:22.591796+0000","last_peered":"2024-09-13T16:28:22.591796+0000","last_clean":"2024-09-13T16:28:22.591796+0000","last_became_active":"2024-09-13T16:28:21.148254+0000","last_became_peered":"2024-09-13T16:28:21.148254+0000","last_unstale":"2024-09-13T16:28:22.591796+0000","last_undegraded":"2024-09-13T16:28:22.591796+0000","last_fullsized":"2024-09-13T16:28:22.591796+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:56.064248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.35","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.134081+0000","last_change":"2024-09-13T16:28:21.145048+0000","last_active":"2024-09-13T16:28:22.134081+0000","last_peered":"2024-09-13T16:28:22.134081+0000","last_clean":"2024-09-13T16:28:22.134081+0000","last_became_active":"2024-09-13T16:28:21.144745+0000","last_became_peered":"2024-09-13T16:28:21.144745+0000","last_unstale":"2024-09-13T16:28:22.134081+0000","last_undegraded":"2024-09-13T16:28:22.134081+0000","last_fullsized":"2024-09-13T16:28:22.134081+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:36.553538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.34","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590394+0000","last_change":"2024-09-13T16:28:21.140857+0000","last_active":"2024-09-13T16:28:22.590394+0000","last_peered":"2024-09-13T16:28:22.590394+0000","last_clean":"2024-09-13T16:28:22.590394+0000","last_became_active":"2024-09-13T16:28:21.139539+0000","last_became_peered":"2024-09-13T16:28:21.139539+0000","last_unstale":"2024-09-13T16:28:22.590394+0000","last_undegraded":"2024-09-13T16:28:22.590394+0000","last_fullsized":"2024-09-13T16:28:22.590394+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:45:38.542580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.33","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591034+0000","last_change":"2024-09-13T16:28:21.152115+0000","last_active":"2024-09-13T16:28:22.591034+0000","last_peered":"2024-09-13T16:28:22.591034+0000","last_clean":"2024-09-13T16:28:22.591034+0000","last_became_active":"2024-09-13T16:28:21.151800+0000","last_became_peered":"2024-09-13T16:28:21.151800+0000","last_unstale":"2024-09-13T16:28:22.591034+0000","last_undegraded":"2024-09-13T16:28:22.591034+0000","last_fullsized":"2024-09-13T16:28:22.591034+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:08:27.816817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.32","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590902+0000","last_change":"2024-09-13T16:28:21.148842+0000","last_active":"2024-09-13T16:28:22.590902+0000","last_peered":"2024-09-13T16:28:22.590902+0000","last_clean":"2024-09-13T16:28:22.590902+0000","last_became_active":"2024-09-13T16:28:21.148546+0000","last_became_peered":"2024-09-13T16:28:21.148546+0000","last_unstale":"2024-09-13T16:28:22.590902+0000","last_undegraded":"2024-09-13T16:28:22.590902+0000","last_fullsized":"2024-09-13T16:28:22.590902+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:02.243298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.31","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.592645+0000","last_change":"2024-09-13T16:28:21.147105+0000","last_active":"2024-09-13T16:28:22.592645+0000","last_peered":"2024-09-13T16:28:22.592645+0000","last_clean":"2024-09-13T16:28:22.592645+0000","last_became_active":"2024-09-13T16:28:21.145033+0000","last_became_peered":"2024-09-13T16:28:21.145033+0000","last_unstale":"2024-09-13T16:28:22.592645+0000","last_undegraded":"2024-09-13T16:28:22.592645+0000","last_fullsized":"2024-09-13T16:28:22.592645+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:02:10.881716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.30","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.131908+0000","last_change":"2024-09-13T16:28:21.141213+0000","last_active":"2024-09-13T16:28:22.131908+0000","last_peered":"2024-09-13T16:28:22.131908+0000","last_clean":"2024-09-13T16:28:22.131908+0000","last_became_active":"2024-09-13T16:28:21.140969+0000","last_became_peered":"2024-09-13T16:28:21.140969+0000","last_unstale":"2024-09-13T16:28:22.131908+0000","last_undegraded":"2024-09-13T16:28:22.131908+0000","last_fullsized":"2024-09-13T16:28:22.131908+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:36.965429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.2f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133322+0000","last_change":"2024-09-13T16:28:21.144651+0000","last_active":"2024-09-13T16:28:22.133322+0000","last_peered":"2024-09-13T16:28:22.133322+0000","last_clean":"2024-09-13T16:28:22.133322+0000","last_became_active":"2024-09-13T16:28:21.144388+0000","last_became_peered":"2024-09-13T16:28:21.144388+0000","last_unstale":"2024-09-13T16:28:22.133322+0000","last_undegraded":"2024-09-13T16:28:22.133322+0000","last_fullsized":"2024-09-13T16:28:22.133322+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:02.243298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.2e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.134288+0000","last_change":"2024-09-13T16:28:21.146838+0000","last_active":"2024-09-13T16:28:22.134288+0000","last_peered":"2024-09-13T16:28:22.134288+0000","last_clean":"2024-09-13T16:28:22.134288+0000","last_became_active":"2024-09-13T16:28:21.146478+0000","last_became_peered":"2024-09-13T16:28:21.146478+0000","last_unstale":"2024-09-13T16:28:22.134288+0000","last_undegraded":"2024-09-13T16:28:22.134288+0000","last_fullsized":"2024-09-13T16:28:22.134288+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:27.852711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.2d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590775+0000","last_change":"2024-09-13T16:28:21.153260+0000","last_active":"2024-09-13T16:28:22.590775+0000","last_peered":"2024-09-13T16:28:22.590775+0000","last_clean":"2024-09-13T16:28:22.590775+0000","last_became_active":"2024-09-13T16:28:21.152987+0000","last_became_peered":"2024-09-13T16:28:21.152987+0000","last_unstale":"2024-09-13T16:28:22.590775+0000","last_undegraded":"2024-09-13T16:28:22.590775+0000","last_fullsized":"2024-09-13T16:28:22.590775+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:30.594662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.589984+0000","last_change":"2024-09-13T16:28:21.149131+0000","last_active":"2024-09-13T16:28:22.589984+0000","last_peered":"2024-09-13T16:28:22.589984+0000","last_clean":"2024-09-13T16:28:22.589984+0000","last_became_active":"2024-09-13T16:28:21.148835+0000","last_became_peered":"2024-09-13T16:28:21.148835+0000","last_unstale":"2024-09-13T16:28:22.589984+0000","last_undegraded":"2024-09-13T16:28:22.589984+0000","last_fullsized":"2024-09-13T16:28:22.589984+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:29.760389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591190+0000","last_change":"2024-09-13T16:28:21.140094+0000","last_active":"2024-09-13T16:28:22.591190+0000","last_peered":"2024-09-13T16:28:22.591190+0000","last_clean":"2024-09-13T16:28:22.591190+0000","last_became_active":"2024-09-13T16:28:21.139809+0000","last_became_peered":"2024-09-13T16:28:21.139809+0000","last_unstale":"2024-09-13T16:28:22.591190+0000","last_undegraded":"2024-09-13T16:28:22.591190+0000","last_fullsized":"2024-09-13T16:28:22.591190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:40:28.653660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.2a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.589418+0000","last_change":"2024-09-13T16:28:21.137622+0000","last_active":"2024-09-13T16:28:22.589418+0000","last_peered":"2024-09-13T16:28:22.589418+0000","last_clean":"2024-09-13T16:28:22.589418+0000","last_became_active":"2024-09-13T16:28:21.137368+0000","last_became_peered":"2024-09-13T16:28:21.137368+0000","last_unstale":"2024-09-13T16:28:22.589418+0000","last_undegraded":"2024-09-13T16:28:22.589418+0000","last_fullsized":"2024-09-13T16:28:22.589418+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:47:44.733896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.29","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.589326+0000","last_change":"2024-09-13T16:28:21.137703+0000","last_active":"2024-09-13T16:28:22.589326+0000","last_peered":"2024-09-13T16:28:22.589326+0000","last_clean":"2024-09-13T16:28:22.589326+0000","last_became_active":"2024-09-13T16:28:21.137389+0000","last_became_peered":"2024-09-13T16:28:21.137389+0000","last_unstale":"2024-09-13T16:28:22.589326+0000","last_undegraded":"2024-09-13T16:28:22.589326+0000","last_fullsized":"2024-09-13T16:28:22.589326+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:10.021130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.28","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.592866+0000","last_change":"2024-09-13T16:28:21.153265+0000","last_active":"2024-09-13T16:28:22.592866+0000","last_peered":"2024-09-13T16:28:22.592866+0000","last_clean":"2024-09-13T16:28:22.592866+0000","last_became_active":"2024-09-13T16:28:21.152130+0000","last_became_peered":"2024-09-13T16:28:21.152130+0000","last_unstale":"2024-09-13T16:28:22.592866+0000","last_undegraded":"2024-09-13T16:28:22.592866+0000","last_fullsized":"2024-09-13T16:28:22.592866+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:16.792890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.27","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591249+0000","last_change":"2024-09-13T16:28:21.141725+0000","last_active":"2024-09-13T16:28:22.591249+0000","last_peered":"2024-09-13T16:28:22.591249+0000","last_clean":"2024-09-13T16:28:22.591249+0000","last_became_active":"2024-09-13T16:28:21.141298+0000","last_became_peered":"2024-09-13T16:28:21.141298+0000","last_unstale":"2024-09-13T16:28:22.591249+0000","last_undegraded":"2024-09-13T16:28:22.591249+0000","last_fullsized":"2024-09-13T16:28:22.591249+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:24:39.401620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.26","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591204+0000","last_change":"2024-09-13T16:28:21.140498+0000","last_active":"2024-09-13T16:28:22.591204+0000","last_peered":"2024-09-13T16:28:22.591204+0000","last_clean":"2024-09-13T16:28:22.591204+0000","last_became_active":"2024-09-13T16:28:21.140065+0000","last_became_peered":"2024-09-13T16:28:21.140065+0000","last_unstale":"2024-09-13T16:28:22.591204+0000","last_undegraded":"2024-09-13T16:28:22.591204+0000","last_fullsized":"2024-09-13T16:28:22.591204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:42.082177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.25","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.134244+0000","last_change":"2024-09-13T16:28:21.138977+0000","last_active":"2024-09-13T16:28:22.134244+0000","last_peered":"2024-09-13T16:28:22.134244+0000","last_clean":"2024-09-13T16:28:22.134244+0000","last_became_active":"2024-09-13T16:28:21.138712+0000","last_became_peered":"2024-09-13T16:28:21.138712+0000","last_unstale":"2024-09-13T16:28:22.134244+0000","last_undegraded":"2024-09-13T16:28:22.134244+0000","last_fullsized":"2024-09-13T16:28:22.134244+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:21.081308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.24","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133025+0000","last_change":"2024-09-13T16:28:21.148835+0000","last_active":"2024-09-13T16:28:22.133025+0000","last_peered":"2024-09-13T16:28:22.133025+0000","last_clean":"2024-09-13T16:28:22.133025+0000","last_became_active":"2024-09-13T16:28:21.148557+0000","last_became_peered":"2024-09-13T16:28:21.148557+0000","last_unstale":"2024-09-13T16:28:22.133025+0000","last_undegraded":"2024-09-13T16:28:22.133025+0000","last_fullsized":"2024-09-13T16:28:22.133025+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:08.281282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.23","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591357+0000","last_change":"2024-09-13T16:28:21.156095+0000","last_active":"2024-09-13T16:28:22.591357+0000","last_peered":"2024-09-13T16:28:22.591357+0000","last_clean":"2024-09-13T16:28:22.591357+0000","last_became_active":"2024-09-13T16:28:21.155832+0000","last_became_peered":"2024-09-13T16:28:21.155832+0000","last_unstale":"2024-09-13T16:28:22.591357+0000","last_undegraded":"2024-09-13T16:28:22.591357+0000","last_fullsized":"2024-09-13T16:28:22.591357+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:03:14.809631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.22","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590204+0000","last_change":"2024-09-13T16:28:21.146140+0000","last_active":"2024-09-13T16:28:22.590204+0000","last_peered":"2024-09-13T16:28:22.590204+0000","last_clean":"2024-09-13T16:28:22.590204+0000","last_became_active":"2024-09-13T16:28:21.145781+0000","last_became_peered":"2024-09-13T16:28:21.145781+0000","last_unstale":"2024-09-13T16:28:22.590204+0000","last_undegraded":"2024-09-13T16:28:22.590204+0000","last_fullsized":"2024-09-13T16:28:22.590204+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:33:44.466135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.21","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133419+0000","last_change":"2024-09-13T16:28:21.140643+0000","last_active":"2024-09-13T16:28:22.133419+0000","last_peered":"2024-09-13T16:28:22.133419+0000","last_clean":"2024-09-13T16:28:22.133419+0000","last_became_active":"2024-09-13T16:28:21.140392+0000","last_became_peered":"2024-09-13T16:28:21.140392+0000","last_unstale":"2024-09-13T16:28:22.133419+0000","last_undegraded":"2024-09-13T16:28:22.133419+0000","last_fullsized":"2024-09-13T16:28:22.133419+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:59:03.460928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.20","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.484823+0000","last_change":"2024-09-13T16:28:21.139323+0000","last_active":"2024-09-13T16:28:22.484823+0000","last_peered":"2024-09-13T16:28:22.484823+0000","last_clean":"2024-09-13T16:28:22.484823+0000","last_became_active":"2024-09-13T16:28:21.139076+0000","last_became_peered":"2024-09-13T16:28:21.139076+0000","last_unstale":"2024-09-13T16:28:22.484823+0000","last_undegraded":"2024-09-13T16:28:22.484823+0000","last_fullsized":"2024-09-13T16:28:22.484823+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:35.335068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590939+0000","last_change":"2024-09-13T16:28:21.147867+0000","last_active":"2024-09-13T16:28:22.590939+0000","last_peered":"2024-09-13T16:28:22.590939+0000","last_clean":"2024-09-13T16:28:22.590939+0000","last_became_active":"2024-09-13T16:28:21.147513+0000","last_became_peered":"2024-09-13T16:28:21.147513+0000","last_unstale":"2024-09-13T16:28:22.590939+0000","last_undegraded":"2024-09-13T16:28:22.590939+0000","last_fullsized":"2024-09-13T16:28:22.590939+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:05.755510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.1f","version":"67'4","reported_seq":15,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.139396+0000","last_change":"2024-09-13T16:28:20.765934+0000","last_active":"2024-09-13T16:28:21.139396+0000","last_peered":"2024-09-13T16:28:21.139396+0000","last_clean":"2024-09-13T16:28:21.139396+0000","last_became_active":"2024-09-13T16:28:20.765500+0000","last_became_peered":"2024-09-13T16:28:20.765500+0000","last_unstale":"2024-09-13T16:28:21.139396+0000","last_undegraded":"2024-09-13T16:28:21.139396+0000","last_fullsized":"2024-09-13T16:28:21.139396+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.134190+0000","last_change":"2024-09-13T16:28:21.148330+0000","last_active":"2024-09-13T16:28:22.134190+0000","last_peered":"2024-09-13T16:28:22.134190+0000","last_clean":"2024-09-13T16:28:22.134190+0000","last_became_active":"2024-09-13T16:28:21.148070+0000","last_became_peered":"2024-09-13T16:28:21.148070+0000","last_unstale":"2024-09-13T16:28:22.134190+0000","last_undegraded":"2024-09-13T16:28:22.134190+0000","last_fullsized":"2024-09-13T16:28:22.134190+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:48:44.914534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1e","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.145108+0000","last_change":"2024-09-13T16:28:20.757325+0000","last_active":"2024-09-13T16:28:21.145108+0000","last_peered":"2024-09-13T16:28:21.145108+0000","last_clean":"2024-09-13T16:28:21.145108+0000","last_became_active":"2024-09-13T16:28:20.757079+0000","last_became_peered":"2024-09-13T16:28:20.757079+0000","last_unstale":"2024-09-13T16:28:21.145108+0000","last_undegraded":"2024-09-13T16:28:21.145108+0000","last_fullsized":"2024-09-13T16:28:21.145108+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:04:07.390953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.485589+0000","last_change":"2024-09-13T16:28:21.138704+0000","last_active":"2024-09-13T16:28:22.485589+0000","last_peered":"2024-09-13T16:28:22.485589+0000","last_clean":"2024-09-13T16:28:22.485589+0000","last_became_active":"2024-09-13T16:28:21.138475+0000","last_became_peered":"2024-09-13T16:28:21.138475+0000","last_unstale":"2024-09-13T16:28:22.485589+0000","last_undegraded":"2024-09-13T16:28:22.485589+0000","last_fullsized":"2024-09-13T16:28:22.485589+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:19.971159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.144937+0000","last_change":"2024-09-13T16:28:20.761209+0000","last_active":"2024-09-13T16:28:21.144937+0000","last_peered":"2024-09-13T16:28:21.144937+0000","last_clean":"2024-09-13T16:28:21.144937+0000","last_became_active":"2024-09-13T16:28:20.760926+0000","last_became_peered":"2024-09-13T16:28:20.760926+0000","last_unstale":"2024-09-13T16:28:21.144937+0000","last_undegraded":"2024-09-13T16:28:21.144937+0000","last_fullsized":"2024-09-13T16:28:21.144937+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:59.677392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.1f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591916+0000","last_change":"2024-09-13T16:28:21.151793+0000","last_active":"2024-09-13T16:28:22.591916+0000","last_peered":"2024-09-13T16:28:22.591916+0000","last_clean":"2024-09-13T16:28:22.591916+0000","last_became_active":"2024-09-13T16:28:21.151499+0000","last_became_peered":"2024-09-13T16:28:21.151499+0000","last_unstale":"2024-09-13T16:28:22.591916+0000","last_undegraded":"2024-09-13T16:28:22.591916+0000","last_fullsized":"2024-09-13T16:28:22.591916+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:56.559195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.1c","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.144920+0000","last_change":"2024-09-13T16:28:20.758885+0000","last_active":"2024-09-13T16:28:21.144920+0000","last_peered":"2024-09-13T16:28:21.144920+0000","last_clean":"2024-09-13T16:28:21.144920+0000","last_became_active":"2024-09-13T16:28:20.758617+0000","last_became_peered":"2024-09-13T16:28:20.758617+0000","last_unstale":"2024-09-13T16:28:21.144920+0000","last_undegraded":"2024-09-13T16:28:21.144920+0000","last_fullsized":"2024-09-13T16:28:21.144920+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.132004+0000","last_change":"2024-09-13T16:28:21.138102+0000","last_active":"2024-09-13T16:28:22.132004+0000","last_peered":"2024-09-13T16:28:22.132004+0000","last_clean":"2024-09-13T16:28:22.132004+0000","last_became_active":"2024-09-13T16:28:21.136739+0000","last_became_peered":"2024-09-13T16:28:21.136739+0000","last_unstale":"2024-09-13T16:28:22.132004+0000","last_undegraded":"2024-09-13T16:28:22.132004+0000","last_fullsized":"2024-09-13T16:28:22.132004+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:50.463822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1b","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.145353+0000","last_change":"2024-09-13T16:28:20.760970+0000","last_active":"2024-09-13T16:28:21.145353+0000","last_peered":"2024-09-13T16:28:21.145353+0000","last_clean":"2024-09-13T16:28:21.145353+0000","last_became_active":"2024-09-13T16:28:20.760658+0000","last_became_peered":"2024-09-13T16:28:20.760658+0000","last_unstale":"2024-09-13T16:28:21.145353+0000","last_undegraded":"2024-09-13T16:28:21.145353+0000","last_fullsized":"2024-09-13T16:28:21.145353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:38:43.928576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.131369+0000","last_change":"2024-09-13T16:28:21.144773+0000","last_active":"2024-09-13T16:28:22.131369+0000","last_peered":"2024-09-13T16:28:22.131369+0000","last_clean":"2024-09-13T16:28:22.131369+0000","last_became_active":"2024-09-13T16:28:21.144513+0000","last_became_peered":"2024-09-13T16:28:21.144513+0000","last_unstale":"2024-09-13T16:28:22.131369+0000","last_undegraded":"2024-09-13T16:28:22.131369+0000","last_fullsized":"2024-09-13T16:28:22.131369+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:55.164739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1a","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.136787+0000","last_change":"2024-09-13T16:28:20.751953+0000","last_active":"2024-09-13T16:28:21.136787+0000","last_peered":"2024-09-13T16:28:21.136787+0000","last_clean":"2024-09-13T16:28:21.136787+0000","last_became_active":"2024-09-13T16:28:20.751644+0000","last_became_peered":"2024-09-13T16:28:20.751644+0000","last_unstale":"2024-09-13T16:28:21.136787+0000","last_undegraded":"2024-09-13T16:28:21.136787+0000","last_fullsized":"2024-09-13T16:28:21.136787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:52:19.660866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591762+0000","last_change":"2024-09-13T16:28:21.148295+0000","last_active":"2024-09-13T16:28:22.591762+0000","last_peered":"2024-09-13T16:28:22.591762+0000","last_clean":"2024-09-13T16:28:22.591762+0000","last_became_active":"2024-09-13T16:28:21.147906+0000","last_became_peered":"2024-09-13T16:28:21.147906+0000","last_unstale":"2024-09-13T16:28:22.591762+0000","last_undegraded":"2024-09-13T16:28:22.591762+0000","last_fullsized":"2024-09-13T16:28:22.591762+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:37.901645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.127357+0000","last_change":"2024-09-13T16:28:20.763774+0000","last_active":"2024-09-13T16:28:21.127357+0000","last_peered":"2024-09-13T16:28:21.127357+0000","last_clean":"2024-09-13T16:28:21.127357+0000","last_became_active":"2024-09-13T16:28:20.763535+0000","last_became_peered":"2024-09-13T16:28:20.763535+0000","last_unstale":"2024-09-13T16:28:21.127357+0000","last_undegraded":"2024-09-13T16:28:21.127357+0000","last_fullsized":"2024-09-13T16:28:21.127357+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:13:42.023785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.1b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.589676+0000","last_change":"2024-09-13T16:28:21.140695+0000","last_active":"2024-09-13T16:28:22.589676+0000","last_peered":"2024-09-13T16:28:22.589676+0000","last_clean":"2024-09-13T16:28:22.589676+0000","last_became_active":"2024-09-13T16:28:21.140437+0000","last_became_peered":"2024-09-13T16:28:21.140437+0000","last_unstale":"2024-09-13T16:28:22.589676+0000","last_undegraded":"2024-09-13T16:28:22.589676+0000","last_fullsized":"2024-09-13T16:28:22.589676+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:13:15.937292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.142763+0000","last_change":"2024-09-13T16:28:20.759034+0000","last_active":"2024-09-13T16:28:21.142763+0000","last_peered":"2024-09-13T16:28:21.142763+0000","last_clean":"2024-09-13T16:28:21.142763+0000","last_became_active":"2024-09-13T16:28:20.758749+0000","last_became_peered":"2024-09-13T16:28:20.758749+0000","last_unstale":"2024-09-13T16:28:21.142763+0000","last_undegraded":"2024-09-13T16:28:21.142763+0000","last_fullsized":"2024-09-13T16:28:21.142763+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:34:24.851511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.14","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.132765+0000","last_change":"2024-09-13T16:28:21.138872+0000","last_active":"2024-09-13T16:28:22.132765+0000","last_peered":"2024-09-13T16:28:22.132765+0000","last_clean":"2024-09-13T16:28:22.132765+0000","last_became_active":"2024-09-13T16:28:21.138634+0000","last_became_peered":"2024-09-13T16:28:21.138634+0000","last_unstale":"2024-09-13T16:28:22.132765+0000","last_undegraded":"2024-09-13T16:28:22.132765+0000","last_fullsized":"2024-09-13T16:28:22.132765+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:46.092961+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.138054+0000","last_change":"2024-09-13T16:28:20.763443+0000","last_active":"2024-09-13T16:28:21.138054+0000","last_peered":"2024-09-13T16:28:21.138054+0000","last_clean":"2024-09-13T16:28:21.138054+0000","last_became_active":"2024-09-13T16:28:20.762025+0000","last_became_peered":"2024-09-13T16:28:20.762025+0000","last_unstale":"2024-09-13T16:28:21.138054+0000","last_undegraded":"2024-09-13T16:28:21.138054+0000","last_fullsized":"2024-09-13T16:28:21.138054+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:02.083953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.15","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.589857+0000","last_change":"2024-09-13T16:28:21.137108+0000","last_active":"2024-09-13T16:28:22.589857+0000","last_peered":"2024-09-13T16:28:22.589857+0000","last_clean":"2024-09-13T16:28:22.589857+0000","last_became_active":"2024-09-13T16:28:21.136870+0000","last_became_peered":"2024-09-13T16:28:21.136870+0000","last_unstale":"2024-09-13T16:28:22.589857+0000","last_undegraded":"2024-09-13T16:28:22.589857+0000","last_fullsized":"2024-09-13T16:28:22.589857+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:19.971159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.130190+0000","last_change":"2024-09-13T16:28:20.758921+0000","last_active":"2024-09-13T16:28:21.130190+0000","last_peered":"2024-09-13T16:28:21.130190+0000","last_clean":"2024-09-13T16:28:21.130190+0000","last_became_active":"2024-09-13T16:28:20.758548+0000","last_became_peered":"2024-09-13T16:28:20.758548+0000","last_unstale":"2024-09-13T16:28:21.130190+0000","last_undegraded":"2024-09-13T16:28:21.130190+0000","last_fullsized":"2024-09-13T16:28:21.130190+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:38:28.409095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.592053+0000","last_change":"2024-09-13T16:28:21.152287+0000","last_active":"2024-09-13T16:28:22.592053+0000","last_peered":"2024-09-13T16:28:22.592053+0000","last_clean":"2024-09-13T16:28:22.592053+0000","last_became_active":"2024-09-13T16:28:21.151420+0000","last_became_peered":"2024-09-13T16:28:21.151420+0000","last_unstale":"2024-09-13T16:28:22.592053+0000","last_undegraded":"2024-09-13T16:28:22.592053+0000","last_fullsized":"2024-09-13T16:28:22.592053+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:12:21.081308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.15","version":"67'3","reported_seq":14,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.152045+0000","last_change":"2024-09-13T16:28:20.763233+0000","last_active":"2024-09-13T16:28:21.152045+0000","last_peered":"2024-09-13T16:28:21.152045+0000","last_clean":"2024-09-13T16:28:21.152045+0000","last_became_active":"2024-09-13T16:28:20.761968+0000","last_became_peered":"2024-09-13T16:28:20.761968+0000","last_unstale":"2024-09-13T16:28:21.152045+0000","last_undegraded":"2024-09-13T16:28:21.152045+0000","last_fullsized":"2024-09-13T16:28:21.152045+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.134456+0000","last_change":"2024-09-13T16:28:21.140313+0000","last_active":"2024-09-13T16:28:22.134456+0000","last_peered":"2024-09-13T16:28:22.134456+0000","last_clean":"2024-09-13T16:28:22.134456+0000","last_became_active":"2024-09-13T16:28:21.138347+0000","last_became_peered":"2024-09-13T16:28:21.138347+0000","last_unstale":"2024-09-13T16:28:22.134456+0000","last_undegraded":"2024-09-13T16:28:22.134456+0000","last_fullsized":"2024-09-13T16:28:22.134456+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:30.594662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.14","version":"67'6","reported_seq":17,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.156808+0000","last_change":"2024-09-13T16:28:20.758191+0000","last_active":"2024-09-13T16:28:21.156808+0000","last_peered":"2024-09-13T16:28:21.156808+0000","last_clean":"2024-09-13T16:28:21.156808+0000","last_became_active":"2024-09-13T16:28:20.757767+0000","last_became_peered":"2024-09-13T16:28:20.757767+0000","last_unstale":"2024-09-13T16:28:21.156808+0000","last_undegraded":"2024-09-13T16:28:21.156808+0000","last_fullsized":"2024-09-13T16:28:21.156808+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.10","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133311+0000","last_change":"2024-09-13T16:28:21.154608+0000","last_active":"2024-09-13T16:28:22.133311+0000","last_peered":"2024-09-13T16:28:22.133311+0000","last_clean":"2024-09-13T16:28:22.133311+0000","last_became_active":"2024-09-13T16:28:21.154367+0000","last_became_peered":"2024-09-13T16:28:21.154367+0000","last_unstale":"2024-09-13T16:28:22.133311+0000","last_undegraded":"2024-09-13T16:28:22.133311+0000","last_fullsized":"2024-09-13T16:28:22.133311+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:40.957260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.13","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.151701+0000","last_change":"2024-09-13T16:28:20.763633+0000","last_active":"2024-09-13T16:28:21.151701+0000","last_peered":"2024-09-13T16:28:21.151701+0000","last_clean":"2024-09-13T16:28:21.151701+0000","last_became_active":"2024-09-13T16:28:20.763379+0000","last_became_peered":"2024-09-13T16:28:20.763379+0000","last_unstale":"2024-09-13T16:28:21.151701+0000","last_undegraded":"2024-09-13T16:28:21.151701+0000","last_fullsized":"2024-09-13T16:28:21.151701+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:29.506319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590046+0000","last_change":"2024-09-13T16:28:21.138141+0000","last_active":"2024-09-13T16:28:22.590046+0000","last_peered":"2024-09-13T16:28:22.590046+0000","last_clean":"2024-09-13T16:28:22.590046+0000","last_became_active":"2024-09-13T16:28:21.137812+0000","last_became_peered":"2024-09-13T16:28:21.137812+0000","last_unstale":"2024-09-13T16:28:22.590046+0000","last_undegraded":"2024-09-13T16:28:22.590046+0000","last_fullsized":"2024-09-13T16:28:22.590046+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:38.459865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.130389+0000","last_change":"2024-09-13T16:28:20.765938+0000","last_active":"2024-09-13T16:28:21.130389+0000","last_peered":"2024-09-13T16:28:21.130389+0000","last_clean":"2024-09-13T16:28:21.130389+0000","last_became_active":"2024-09-13T16:28:20.765664+0000","last_became_peered":"2024-09-13T16:28:20.765664+0000","last_unstale":"2024-09-13T16:28:21.130389+0000","last_undegraded":"2024-09-13T16:28:21.130389+0000","last_fullsized":"2024-09-13T16:28:21.130389+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:39:09.620313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590633+0000","last_change":"2024-09-13T16:28:21.149643+0000","last_active":"2024-09-13T16:28:22.590633+0000","last_peered":"2024-09-13T16:28:22.590633+0000","last_clean":"2024-09-13T16:28:22.590633+0000","last_became_active":"2024-09-13T16:28:21.148687+0000","last_became_peered":"2024-09-13T16:28:21.148687+0000","last_unstale":"2024-09-13T16:28:22.590633+0000","last_undegraded":"2024-09-13T16:28:22.590633+0000","last_fullsized":"2024-09-13T16:28:22.590633+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:03.093917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.138195+0000","last_change":"2024-09-13T16:28:20.761821+0000","last_active":"2024-09-13T16:28:21.138195+0000","last_peered":"2024-09-13T16:28:21.138195+0000","last_clean":"2024-09-13T16:28:21.138195+0000","last_became_active":"2024-09-13T16:28:20.761493+0000","last_became_peered":"2024-09-13T16:28:20.761493+0000","last_unstale":"2024-09-13T16:28:21.138195+0000","last_undegraded":"2024-09-13T16:28:21.138195+0000","last_fullsized":"2024-09-13T16:28:21.138195+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.132425+0000","last_change":"2024-09-13T16:28:21.135753+0000","last_active":"2024-09-13T16:28:22.132425+0000","last_peered":"2024-09-13T16:28:22.132425+0000","last_clean":"2024-09-13T16:28:22.132425+0000","last_became_active":"2024-09-13T16:28:21.135591+0000","last_became_peered":"2024-09-13T16:28:21.135591+0000","last_unstale":"2024-09-13T16:28:22.132425+0000","last_undegraded":"2024-09-13T16:28:22.132425+0000","last_fullsized":"2024-09-13T16:28:22.132425+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:00.991635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.10","version":"67'8","reported_seq":19,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.155081+0000","last_change":"2024-09-13T16:28:20.755485+0000","last_active":"2024-09-13T16:28:21.155081+0000","last_peered":"2024-09-13T16:28:21.155081+0000","last_clean":"2024-09-13T16:28:21.155081+0000","last_became_active":"2024-09-13T16:28:20.755072+0000","last_became_peered":"2024-09-13T16:28:20.755072+0000","last_unstale":"2024-09-13T16:28:21.155081+0000","last_undegraded":"2024-09-13T16:28:21.155081+0000","last_fullsized":"2024-09-13T16:28:21.155081+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:58.987147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590455+0000","last_change":"2024-09-13T16:28:21.155896+0000","last_active":"2024-09-13T16:28:22.590455+0000","last_peered":"2024-09-13T16:28:22.590455+0000","last_clean":"2024-09-13T16:28:22.590455+0000","last_became_active":"2024-09-13T16:28:21.155567+0000","last_became_peered":"2024-09-13T16:28:21.155567+0000","last_unstale":"2024-09-13T16:28:22.590455+0000","last_undegraded":"2024-09-13T16:28:22.590455+0000","last_fullsized":"2024-09-13T16:28:22.590455+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:10.936563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.f","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.139745+0000","last_change":"2024-09-13T16:28:20.764696+0000","last_active":"2024-09-13T16:28:21.139745+0000","last_peered":"2024-09-13T16:28:21.139745+0000","last_clean":"2024-09-13T16:28:21.139745+0000","last_became_active":"2024-09-13T16:28:20.764428+0000","last_became_peered":"2024-09-13T16:28:20.764428+0000","last_unstale":"2024-09-13T16:28:21.139745+0000","last_undegraded":"2024-09-13T16:28:21.139745+0000","last_fullsized":"2024-09-13T16:28:21.139745+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:11:14.580434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.d","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591006+0000","last_change":"2024-09-13T16:28:21.156546+0000","last_active":"2024-09-13T16:28:22.591006+0000","last_peered":"2024-09-13T16:28:22.591006+0000","last_clean":"2024-09-13T16:28:22.591006+0000","last_became_active":"2024-09-13T16:28:21.156287+0000","last_became_peered":"2024-09-13T16:28:21.156287+0000","last_unstale":"2024-09-13T16:28:22.591006+0000","last_undegraded":"2024-09-13T16:28:22.591006+0000","last_fullsized":"2024-09-13T16:28:22.591006+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:03.939858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.e","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.155592+0000","last_change":"2024-09-13T16:28:20.758319+0000","last_active":"2024-09-13T16:28:21.155592+0000","last_peered":"2024-09-13T16:28:21.155592+0000","last_clean":"2024-09-13T16:28:21.155592+0000","last_became_active":"2024-09-13T16:28:20.757959+0000","last_became_peered":"2024-09-13T16:28:20.757959+0000","last_unstale":"2024-09-13T16:28:21.155592+0000","last_undegraded":"2024-09-13T16:28:21.155592+0000","last_fullsized":"2024-09-13T16:28:21.155592+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:43:04.745546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.134438+0000","last_change":"2024-09-13T16:28:21.148687+0000","last_active":"2024-09-13T16:28:22.134438+0000","last_peered":"2024-09-13T16:28:22.134438+0000","last_clean":"2024-09-13T16:28:22.134438+0000","last_became_active":"2024-09-13T16:28:21.148089+0000","last_became_peered":"2024-09-13T16:28:21.148089+0000","last_unstale":"2024-09-13T16:28:22.134438+0000","last_undegraded":"2024-09-13T16:28:22.134438+0000","last_fullsized":"2024-09-13T16:28:22.134438+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:40.505266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.d","version":"67'1","reported_seq":12,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.151820+0000","last_change":"2024-09-13T16:28:20.761797+0000","last_active":"2024-09-13T16:28:21.151820+0000","last_peered":"2024-09-13T16:28:21.151820+0000","last_clean":"2024-09-13T16:28:21.151820+0000","last_became_active":"2024-09-13T16:28:20.761518+0000","last_became_peered":"2024-09-13T16:28:20.761518+0000","last_unstale":"2024-09-13T16:28:21.151820+0000","last_undegraded":"2024-09-13T16:28:21.151820+0000","last_fullsized":"2024-09-13T16:28:21.151820+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:26.842748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.f","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.592008+0000","last_change":"2024-09-13T16:28:21.151089+0000","last_active":"2024-09-13T16:28:22.592008+0000","last_peered":"2024-09-13T16:28:22.592008+0000","last_clean":"2024-09-13T16:28:22.592008+0000","last_became_active":"2024-09-13T16:28:21.150773+0000","last_became_peered":"2024-09-13T16:28:21.150773+0000","last_unstale":"2024-09-13T16:28:22.592008+0000","last_undegraded":"2024-09-13T16:28:22.592008+0000","last_fullsized":"2024-09-13T16:28:22.592008+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:32:30.516283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.127702+0000","last_change":"2024-09-13T16:28:20.753693+0000","last_active":"2024-09-13T16:28:21.127702+0000","last_peered":"2024-09-13T16:28:21.127702+0000","last_clean":"2024-09-13T16:28:21.127702+0000","last_became_active":"2024-09-13T16:28:20.753428+0000","last_became_peered":"2024-09-13T16:28:20.753428+0000","last_unstale":"2024-09-13T16:28:21.127702+0000","last_undegraded":"2024-09-13T16:28:21.127702+0000","last_fullsized":"2024-09-13T16:28:21.127702+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:45:21.436784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.485008+0000","last_change":"2024-09-13T16:28:21.138767+0000","last_active":"2024-09-13T16:28:22.485008+0000","last_peered":"2024-09-13T16:28:22.485008+0000","last_clean":"2024-09-13T16:28:22.485008+0000","last_became_active":"2024-09-13T16:28:21.138488+0000","last_became_peered":"2024-09-13T16:28:21.138488+0000","last_unstale":"2024-09-13T16:28:22.485008+0000","last_undegraded":"2024-09-13T16:28:22.485008+0000","last_fullsized":"2024-09-13T16:28:22.485008+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:42:55.804560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.b","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.146637+0000","last_change":"2024-09-13T16:28:20.763075+0000","last_active":"2024-09-13T16:28:21.146637+0000","last_peered":"2024-09-13T16:28:21.146637+0000","last_clean":"2024-09-13T16:28:21.146637+0000","last_became_active":"2024-09-13T16:28:20.762781+0000","last_became_peered":"2024-09-13T16:28:20.762781+0000","last_unstale":"2024-09-13T16:28:21.146637+0000","last_undegraded":"2024-09-13T16:28:21.146637+0000","last_fullsized":"2024-09-13T16:28:21.146637+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:28:47.232880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590299+0000","last_change":"2024-09-13T16:28:21.141027+0000","last_active":"2024-09-13T16:28:22.590299+0000","last_peered":"2024-09-13T16:28:22.590299+0000","last_clean":"2024-09-13T16:28:22.590299+0000","last_became_active":"2024-09-13T16:28:21.140752+0000","last_became_peered":"2024-09-13T16:28:21.140752+0000","last_unstale":"2024-09-13T16:28:22.590299+0000","last_undegraded":"2024-09-13T16:28:22.590299+0000","last_fullsized":"2024-09-13T16:28:22.590299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:26:08.564667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.130520+0000","last_change":"2024-09-13T16:28:20.765933+0000","last_active":"2024-09-13T16:28:21.130520+0000","last_peered":"2024-09-13T16:28:21.130520+0000","last_clean":"2024-09-13T16:28:21.130520+0000","last_became_active":"2024-09-13T16:28:20.765650+0000","last_became_peered":"2024-09-13T16:28:20.765650+0000","last_unstale":"2024-09-13T16:28:21.130520+0000","last_undegraded":"2024-09-13T16:28:21.130520+0000","last_fullsized":"2024-09-13T16:28:21.130520+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:31.462343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.590882+0000","last_change":"2024-09-13T16:28:21.148142+0000","last_active":"2024-09-13T16:28:22.590882+0000","last_peered":"2024-09-13T16:28:22.590882+0000","last_clean":"2024-09-13T16:28:22.590882+0000","last_became_active":"2024-09-13T16:28:21.147535+0000","last_became_peered":"2024-09-13T16:28:21.147535+0000","last_unstale":"2024-09-13T16:28:22.590882+0000","last_undegraded":"2024-09-13T16:28:22.590882+0000","last_fullsized":"2024-09-13T16:28:22.590882+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:57.225740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.119900+0000","last_change":"2024-09-13T16:28:20.757050+0000","last_active":"2024-09-13T16:28:21.119900+0000","last_peered":"2024-09-13T16:28:21.119900+0000","last_clean":"2024-09-13T16:28:21.119900+0000","last_became_active":"2024-09-13T16:28:20.756793+0000","last_became_peered":"2024-09-13T16:28:20.756793+0000","last_unstale":"2024-09-13T16:28:21.119900+0000","last_undegraded":"2024-09-13T16:28:21.119900+0000","last_fullsized":"2024-09-13T16:28:21.119900+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:08:40.488362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.485343+0000","last_change":"2024-09-13T16:28:21.138024+0000","last_active":"2024-09-13T16:28:22.485343+0000","last_peered":"2024-09-13T16:28:22.485343+0000","last_clean":"2024-09-13T16:28:22.485343+0000","last_became_active":"2024-09-13T16:28:21.137795+0000","last_became_peered":"2024-09-13T16:28:21.137795+0000","last_unstale":"2024-09-13T16:28:22.485343+0000","last_undegraded":"2024-09-13T16:28:22.485343+0000","last_fullsized":"2024-09-13T16:28:22.485343+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:48:10.021130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.7","version":"67'2","reported_seq":13,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.143448+0000","last_change":"2024-09-13T16:28:20.757314+0000","last_active":"2024-09-13T16:28:21.143448+0000","last_peered":"2024-09-13T16:28:21.143448+0000","last_clean":"2024-09-13T16:28:21.143448+0000","last_became_active":"2024-09-13T16:28:20.756827+0000","last_became_peered":"2024-09-13T16:28:20.756827+0000","last_unstale":"2024-09-13T16:28:21.143448+0000","last_undegraded":"2024-09-13T16:28:21.143448+0000","last_fullsized":"2024-09-13T16:28:21.143448+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:31:30.820069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133712+0000","last_change":"2024-09-13T16:28:21.146346+0000","last_active":"2024-09-13T16:28:22.133712+0000","last_peered":"2024-09-13T16:28:22.133712+0000","last_clean":"2024-09-13T16:28:22.133712+0000","last_became_active":"2024-09-13T16:28:21.145637+0000","last_became_peered":"2024-09-13T16:28:21.145637+0000","last_unstale":"2024-09-13T16:28:22.133712+0000","last_undegraded":"2024-09-13T16:28:22.133712+0000","last_fullsized":"2024-09-13T16:28:22.133712+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:51:07.378968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.130175+0000","last_change":"2024-09-13T16:28:20.758029+0000","last_active":"2024-09-13T16:28:21.130175+0000","last_peered":"2024-09-13T16:28:21.130175+0000","last_clean":"2024-09-13T16:28:21.130175+0000","last_became_active":"2024-09-13T16:28:20.757810+0000","last_became_peered":"2024-09-13T16:28:20.757810+0000","last_unstale":"2024-09-13T16:28:21.130175+0000","last_undegraded":"2024-09-13T16:28:21.130175+0000","last_fullsized":"2024-09-13T16:28:21.130175+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:18:34.445856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.591341+0000","last_change":"2024-09-13T16:28:21.140345+0000","last_active":"2024-09-13T16:28:22.591341+0000","last_peered":"2024-09-13T16:28:22.591341+0000","last_clean":"2024-09-13T16:28:22.591341+0000","last_became_active":"2024-09-13T16:28:21.139738+0000","last_became_peered":"2024-09-13T16:28:21.139738+0000","last_unstale":"2024-09-13T16:28:22.591341+0000","last_undegraded":"2024-09-13T16:28:22.591341+0000","last_fullsized":"2024-09-13T16:28:22.591341+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:16:50.517527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.120780+0000","last_change":"2024-09-13T16:28:20.757681+0000","last_active":"2024-09-13T16:28:21.120780+0000","last_peered":"2024-09-13T16:28:21.120780+0000","last_clean":"2024-09-13T16:28:21.120780+0000","last_became_active":"2024-09-13T16:28:20.757377+0000","last_became_peered":"2024-09-13T16:28:20.757377+0000","last_unstale":"2024-09-13T16:28:21.120780+0000","last_undegraded":"2024-09-13T16:28:21.120780+0000","last_fullsized":"2024-09-13T16:28:21.120780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:03.708069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.592130+0000","last_change":"2024-09-13T16:28:21.150729+0000","last_active":"2024-09-13T16:28:22.592130+0000","last_peered":"2024-09-13T16:28:22.592130+0000","last_clean":"2024-09-13T16:28:22.592130+0000","last_became_active":"2024-09-13T16:28:21.150475+0000","last_became_peered":"2024-09-13T16:28:21.150475+0000","last_unstale":"2024-09-13T16:28:22.592130+0000","last_undegraded":"2024-09-13T16:28:22.592130+0000","last_fullsized":"2024-09-13T16:28:22.592130+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:19:56.410363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.125722+0000","last_change":"2024-09-13T16:28:20.757527+0000","last_active":"2024-09-13T16:28:21.125722+0000","last_peered":"2024-09-13T16:28:21.125722+0000","last_clean":"2024-09-13T16:28:21.125722+0000","last_became_active":"2024-09-13T16:28:20.756428+0000","last_became_peered":"2024-09-13T16:28:20.756428+0000","last_unstale":"2024-09-13T16:28:21.125722+0000","last_undegraded":"2024-09-13T16:28:21.125722+0000","last_fullsized":"2024-09-13T16:28:21.125722+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:42:29.584698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.589914+0000","last_change":"2024-09-13T16:28:21.146547+0000","last_active":"2024-09-13T16:28:22.589914+0000","last_peered":"2024-09-13T16:28:22.589914+0000","last_clean":"2024-09-13T16:28:22.589914+0000","last_became_active":"2024-09-13T16:28:21.146280+0000","last_became_peered":"2024-09-13T16:28:21.146280+0000","last_unstale":"2024-09-13T16:28:22.589914+0000","last_undegraded":"2024-09-13T16:28:22.589914+0000","last_fullsized":"2024-09-13T16:28:22.589914+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:10:27.852711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.136787+0000","last_change":"2024-09-13T16:28:20.761758+0000","last_active":"2024-09-13T16:28:21.136787+0000","last_peered":"2024-09-13T16:28:21.136787+0000","last_clean":"2024-09-13T16:28:21.136787+0000","last_became_active":"2024-09-13T16:28:20.761462+0000","last_became_peered":"2024-09-13T16:28:20.761462+0000","last_unstale":"2024-09-13T16:28:21.136787+0000","last_undegraded":"2024-09-13T16:28:21.136787+0000","last_fullsized":"2024-09-13T16:28:21.136787+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:02.929894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.133356+0000","last_change":"2024-09-13T16:28:21.149277+0000","last_active":"2024-09-13T16:28:22.133356+0000","last_peered":"2024-09-13T16:28:22.133356+0000","last_clean":"2024-09-13T16:28:22.133356+0000","last_became_active":"2024-09-13T16:28:21.149011+0000","last_became_peered":"2024-09-13T16:28:21.149011+0000","last_unstale":"2024-09-13T16:28:22.133356+0000","last_undegraded":"2024-09-13T16:28:22.133356+0000","last_fullsized":"2024-09-13T16:28:22.133356+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:22:18.339737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.127543+0000","last_change":"2024-09-13T16:28:20.759496+0000","last_active":"2024-09-13T16:28:21.127543+0000","last_peered":"2024-09-13T16:28:21.127543+0000","last_clean":"2024-09-13T16:28:21.127543+0000","last_became_active":"2024-09-13T16:28:20.759218+0000","last_became_peered":"2024-09-13T16:28:20.759218+0000","last_unstale":"2024-09-13T16:28:21.127543+0000","last_undegraded":"2024-09-13T16:28:21.127543+0000","last_fullsized":"2024-09-13T16:28:21.127543+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:46.314613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.131704+0000","last_change":"2024-09-13T16:28:21.145225+0000","last_active":"2024-09-13T16:28:22.131704+0000","last_peered":"2024-09-13T16:28:22.131704+0000","last_clean":"2024-09-13T16:28:22.131704+0000","last_became_active":"2024-09-13T16:28:21.144946+0000","last_became_peered":"2024-09-13T16:28:21.144946+0000","last_unstale":"2024-09-13T16:28:22.131704+0000","last_undegraded":"2024-09-13T16:28:22.131704+0000","last_fullsized":"2024-09-13T16:28:22.131704+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:27:40.505266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.127532+0000","last_change":"2024-09-13T16:28:20.764494+0000","last_active":"2024-09-13T16:28:21.127532+0000","last_peered":"2024-09-13T16:28:21.127532+0000","last_clean":"2024-09-13T16:28:21.127532+0000","last_became_active":"2024-09-13T16:28:20.764144+0000","last_became_peered":"2024-09-13T16:28:20.764144+0000","last_unstale":"2024-09-13T16:28:21.127532+0000","last_undegraded":"2024-09-13T16:28:21.127532+0000","last_fullsized":"2024-09-13T16:28:21.127532+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:56:35.488631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.132254+0000","last_change":"2024-09-13T16:28:21.144430+0000","last_active":"2024-09-13T16:28:22.132254+0000","last_peered":"2024-09-13T16:28:22.132254+0000","last_clean":"2024-09-13T16:28:22.132254+0000","last_became_active":"2024-09-13T16:28:21.144159+0000","last_became_peered":"2024-09-13T16:28:21.144159+0000","last_unstale":"2024-09-13T16:28:22.132254+0000","last_undegraded":"2024-09-13T16:28:22.132254+0000","last_fullsized":"2024-09-13T16:28:22.132254+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:11:23.709307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.124365+0000","last_change":"2024-09-13T16:28:20.762598+0000","last_active":"2024-09-13T16:28:21.124365+0000","last_peered":"2024-09-13T16:28:21.124365+0000","last_clean":"2024-09-13T16:28:21.124365+0000","last_became_active":"2024-09-13T16:28:20.762333+0000","last_became_peered":"2024-09-13T16:28:20.762333+0000","last_unstale":"2024-09-13T16:28:21.124365+0000","last_undegraded":"2024-09-13T16:28:21.124365+0000","last_fullsized":"2024-09-13T16:28:21.124365+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:42:56.215776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.b","version":"0'0","reported_seq":10,"reported_epoch":68,"state":"active+clean","last_fresh":"2024-09-13T16:28:22.593299+0000","last_change":"2024-09-13T16:28:21.147759+0000","last_active":"2024-09-13T16:28:22.593299+0000","last_peered":"2024-09-13T16:28:22.593299+0000","last_clean":"2024-09-13T16:28:22.593299+0000","last_became_active":"2024-09-13T16:28:21.147348+0000","last_became_peered":"2024-09-13T16:28:21.147348+0000","last_unstale":"2024-09-13T16:28:22.593299+0000","last_undegraded":"2024-09-13T16:28:22.593299+0000","last_fullsized":"2024-09-13T16:28:22.593299+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":66,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:20.725773+0000","last_clean_scrub_stamp":"2024-09-13T16:28:20.725773+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:41:31.024007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.127133+0000","last_change":"2024-09-13T16:28:20.762524+0000","last_active":"2024-09-13T16:28:21.127133+0000","last_peered":"2024-09-13T16:28:21.127133+0000","last_clean":"2024-09-13T16:28:21.127133+0000","last_became_active":"2024-09-13T16:28:20.762204+0000","last_became_peered":"2024-09-13T16:28:20.762204+0000","last_unstale":"2024-09-13T16:28:21.127133+0000","last_undegraded":"2024-09-13T16:28:21.127133+0000","last_fullsized":"2024-09-13T16:28:21.127133+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:28:19.715810+0000","last_clean_scrub_stamp":"2024-09-13T16:28:19.715810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:29:21.792298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":203,"reported_epoch":67,"state":"active+clean","last_fresh":"2024-09-13T16:28:21.135764+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:28:21.135764+0000","last_peered":"2024-09-13T16:28:21.135764+0000","last_clean":"2024-09-13T16:28:21.135764+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:28:21.135764+0000","last_undegraded":"2024-09-13T16:28:21.135764+0000","last_fullsized":"2024-09-13T16:28:21.135764+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":10,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542186,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1396,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1429504,"data_stored":958592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.86199999999999999,"15min":0.85399999999999998},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.121,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.86099999999999999,"15min":0.84899999999999998},"min":{"1min":0.60399999999999998,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.085,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.91100000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.60299999999999998,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.0880000000000001,"5min":1.121,"15min":1.121},"last":0.749},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.84799999999999998},"min":{"1min":0.70599999999999996,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.131,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.71599999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.88900000000000001},"min":{"1min":0.63400000000000001,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.2170000000000001,"5min":1.228,"15min":1.228},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.83699999999999997,"15min":0.83299999999999996},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.0389999999999999,"5min":1.091,"15min":1.091},"last":1.0649999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":1.0029999999999999,"15min":1.014},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.1220000000000001,"5min":1.238,"15min":1.238},"last":0.96199999999999997},{"interface":"front","average":{"1min":0.97399999999999998,"5min":0.997,"15min":1.0069999999999999},"min":{"1min":0.746,"5min":0.745,"15min":0.745},"max":{"1min":1.214,"5min":1.226,"15min":1.226},"last":0.873}]},{"osd":4,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.81599999999999995,"5min":0.86699999999999999,"15min":0.88700000000000001},"min":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":1.0580000000000001,"15min":1.0880000000000001},"min":{"1min":0.77800000000000002,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.1870000000000001,"5min":1.589,"15min":1.589},"last":1.091}]},{"osd":5,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.996,"15min":1.03},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.1439999999999999,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.996},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.1120000000000001,"15min":1.1200000000000001},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.244,"5min":1.337,"15min":1.337},"last":1.1779999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:28:14 2024","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.077,"15min":1.087},"min":{"1min":0.85599999999999998,"5min":0.78900000000000003,"15min":0.78900000000000003},"max":{"1min":1.196,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":1.123},{"interface":"front","average":{"1min":1.109,"5min":1.1639999999999999,"15min":1.1839999999999999},"min":{"1min":0.99299999999999999,"5min":0.751,"15min":0.751},"max":{"1min":1.2609999999999999,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.1559999999999999}]}]},{"osd":6,"up_from":14,"seq":60129542185,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27828,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344012,"statfs":{"total":96636764160,"available":96608268288,"internally_reserved":0,"allocated":823296,"data_stored":367482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.84399999999999997,"15min":0.85199999999999998},"min":{"1min":0.59899999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.179,"5min":1.179,"15min":1.179},"last":1.123},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.89500000000000002,"15min":0.89600000000000002},"min":{"1min":0.628,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.82699999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.82899999999999996,"15min":0.83599999999999997},"min":{"1min":0.65200000000000002,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.0780000000000001,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":1.016,"5min":0.92400000000000004,"15min":0.91800000000000004},"min":{"1min":0.61699999999999999,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0369999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.83299999999999996,"5min":0.86899999999999999,"15min":0.88300000000000001},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.2110000000000001,"5min":1.2609999999999999,"15min":1.2609999999999999},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.83399999999999996,"5min":0.84199999999999997,"15min":0.84199999999999997},"min":{"1min":0.56899999999999995,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.0840000000000001,"5min":1.1870000000000001,"15min":1.1870000000000001},"last":0.88400000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.96799999999999997,"5min":0.95199999999999996,"15min":0.96299999999999997},"min":{"1min":0.69499999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.248,"5min":1.536,"15min":1.536},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.98699999999999999,"15min":0.98899999999999999},"min":{"1min":0.71799999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.264,"5min":1.409,"15min":1.409},"last":1.0680000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.96299999999999997,"15min":0.97699999999999998},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.1910000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.206},{"interface":"front","average":{"1min":1.056,"5min":0.97399999999999998,"15min":0.94399999999999995},"min":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.92600000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.91900000000000004,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":1.0049999999999999},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.1220000000000001,"15min":1.1379999999999999},"min":{"1min":0.68400000000000005,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.4259999999999999,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":1.1699999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:28:15 2024","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.123,"15min":1.1240000000000001},"min":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.522,"5min":1.522,"15min":1.522},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.024,"5min":1.0509999999999999,"15min":1.056},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.3149999999999999,"5min":1.337,"15min":1.337},"last":1.093}]}]},{"osd":5,"up_from":14,"seq":60129542187,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":804,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":823296,"data_stored":367470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.86399999999999999,"15min":0.86199999999999999},"min":{"1min":0.5,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.97699999999999998},{"interface":"front","average":{"1min":0.80400000000000005,"5min":0.85899999999999999,"15min":0.85899999999999999},"min":{"1min":0.48299999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.131,"5min":1.675,"15min":1.675},"last":0.91300000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.876,"15min":0.88400000000000001},"min":{"1min":0.66500000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.321,"5min":1.369,"15min":1.369},"last":1.016},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.84899999999999998,"15min":0.83099999999999996},"min":{"1min":0.66400000000000003,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.169,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":0.81100000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93000000000000005,"15min":0.93700000000000006},"min":{"1min":0.65200000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.032,"5min":1.625,"15min":1.625},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.98999999999999999,"5min":0.97299999999999998,"15min":0.98699999999999999},"min":{"1min":0.627,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.5129999999999999,"5min":1.552,"15min":1.552},"last":0.70699999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.86199999999999999,"5min":0.85599999999999998,"15min":0.83799999999999997},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1639999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.624},{"interface":"front","average":{"1min":0.879,"5min":0.97199999999999998,"15min":1.016},"min":{"1min":0.66000000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1819999999999999,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":0.76300000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":1.0409999999999999,"15min":1.0720000000000001},"min":{"1min":0.61899999999999999,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.579,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0640000000000001,"15min":1.0620000000000001},"min":{"1min":0.85799999999999998,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2210000000000001,"5min":1.601,"15min":1.601},"last":1.0009999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.96099999999999997,"15min":0.97299999999999998},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.179,"5min":1.421,"15min":1.421},"last":0.66900000000000004},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0740000000000001,"15min":1.0780000000000001},"min":{"1min":0.76000000000000001,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":2.21,"5min":2.21,"15min":2.21},"last":0.85399999999999998}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0389999999999999,"15min":1.036},"min":{"1min":0.68799999999999994,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.3180000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.034,"5min":1.175,"15min":1.2290000000000001},"min":{"1min":0.78900000000000003,"5min":0.745,"15min":0.745},"max":{"1min":1.266,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":0.89700000000000002}]}]},{"osd":4,"up_from":14,"seq":60129542185,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":831488,"data_stored":369012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.83899999999999997},"min":{"1min":0.51100000000000001,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.335,"5min":1.411,"15min":1.411},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.97699999999999998,"5min":0.873,"15min":0.85799999999999998},"min":{"1min":0.58199999999999996,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":2.899,"5min":2.899,"15min":2.899},"last":0.95399999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:28:21 2024","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.93100000000000005,"15min":0.92100000000000004},"min":{"1min":0.59799999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":3.085,"5min":3.085,"15min":3.085},"last":0.754},{"interface":"front","average":{"1min":0.97599999999999998,"5min":0.93000000000000005,"15min":0.92000000000000004},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.5720000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.91700000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.85999999999999999,"15min":0.84399999999999997},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.2,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.48699999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":0.97699999999999998,"15min":0.96999999999999997},"min":{"1min":0.68600000000000005,"5min":0.498,"15min":0.498},"max":{"1min":3.2160000000000002,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":0.80800000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.95699999999999996,"15min":0.95899999999999996},"min":{"1min":0.68200000000000005,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.331,"5min":1.472,"15min":1.472},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.95999999999999996,"15min":0.96399999999999997},"min":{"1min":0.68100000000000005,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.306,"5min":1.577,"15min":1.577},"last":0.77200000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:28:21 2024","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":0.997,"15min":0.98499999999999999},"min":{"1min":0.65600000000000003,"5min":0.59699999999999998,"15min":0.59699999999999998},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.137,"15min":1.1479999999999999},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.79,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":1.012}]},{"osd":6,"last update":"Fri Sep 13 16:28:21 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95299999999999996,"15min":0.94499999999999995},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5940000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":1.2350000000000001,"5min":1.1479999999999999,"15min":1.117},"min":{"1min":0.83299999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":3.1469999999999998,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":0.88800000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:28:21 2024","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.165,"15min":1.153},"min":{"1min":0.877,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.472,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.214,"5min":1.151,"15min":1.129},"min":{"1min":0.94999999999999996,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":3.1070000000000002,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.97699999999999998}]}]},{"osd":3,"up_from":14,"seq":60129542186,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27824,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344016,"statfs":{"total":96636764160,"available":96608272384,"internally_reserved":0,"allocated":819200,"data_stored":367448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.625,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.96099999999999997,"5min":1.04,"15min":1.04},"last":0.872},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.65500000000000003,"15min":0.65600000000000003},"min":{"1min":0.498,"5min":0.433,"15min":0.433},"max":{"1min":0.96999999999999997,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.63200000000000001,"15min":0.622},"min":{"1min":0.378,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.76900000000000002,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.2310000000000001,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.68500000000000005,"15min":0.69599999999999995},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.98799999999999999,"5min":1.27,"15min":1.27},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.76400000000000001,"15min":0.755},"min":{"1min":0.56399999999999995,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.139,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.69999999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.70899999999999996,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.214,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95899999999999996,"15min":0.96199999999999997},"min":{"1min":0.66200000000000003,"5min":0.625,"15min":0.625},"max":{"1min":1.538,"5min":1.782,"15min":1.782},"last":1.153}]},{"osd":5,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.97699999999999998,"15min":0.98199999999999998},"min":{"1min":0.71899999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.6299999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.046,"5min":1.0920000000000001,"15min":1.1020000000000001},"min":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.484,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":1.097}]},{"osd":6,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.075,"5min":1.119,"15min":1.137},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.571,"5min":1.885,"15min":1.885},"last":1.23},{"interface":"front","average":{"1min":0.96999999999999997,"5min":1.032,"15min":1.0580000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.268,"5min":1.95,"15min":1.95},"last":1.0800000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:28:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0640000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"min":{"1min":0.73699999999999999,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.579,"5min":1.889,"15min":1.889},"last":1.21},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0580000000000001,"15min":1.0609999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.756,"5min":1.756,"15min":1.756},"last":1.252}]}]},{"osd":2,"up_from":14,"seq":60129542187,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31928,"kb_used_data":800,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339912,"statfs":{"total":96636764160,"available":96604069888,"internally_reserved":0,"allocated":819200,"data_stored":367448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.59399999999999997,"15min":0.58599999999999997},"min":{"1min":0.40000000000000002,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.86499999999999999,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.68100000000000005,"15min":0.68999999999999995},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.88100000000000001,"5min":0.995,"15min":0.995},"last":0.67100000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.70099999999999996,"15min":0.70899999999999996},"min":{"1min":0.42099999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.96699999999999997,"5min":0.997,"15min":0.997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.67800000000000005,"15min":0.68799999999999994},"min":{"1min":0.45200000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88500000000000001,"5min":1.123,"15min":1.123},"last":0.54700000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.66000000000000003,"15min":0.66500000000000004},"min":{"1min":0.47699999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.86899999999999999,"5min":1.073,"15min":1.073},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.80000000000000004,"5min":0.79300000000000004,"15min":0.78800000000000003},"min":{"1min":0.64300000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.0740000000000001,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.624}]},{"osd":4,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97299999999999998,"5min":0.94899999999999995,"15min":0.95499999999999996},"min":{"1min":0.70399999999999996,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.2809999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.95699999999999996,"15min":0.98399999999999999},"min":{"1min":0.78000000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.367,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.032}]},{"osd":5,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93300000000000005,"5min":0.97899999999999998,"15min":0.98499999999999999},"min":{"1min":0.72599999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"max":{"1min":1.153,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.95599999999999996,"15min":0.96099999999999997},"min":{"1min":0.72799999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.234,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.86499999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":1.014,"15min":1.042},"min":{"1min":0.73899999999999999,"5min":0.623,"15min":0.623},"max":{"1min":1.2729999999999999,"5min":2.024,"15min":2.024},"last":0.81899999999999995},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0609999999999999,"15min":1.0669999999999999},"min":{"1min":0.81899999999999995,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.4930000000000001,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.89900000000000002}]},{"osd":7,"last update":"Fri Sep 13 16:28:16 2024","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":1.0669999999999999,"15min":1.081},"min":{"1min":0.78900000000000003,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.252,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.0569999999999999,"15min":1.071},"min":{"1min":0.81499999999999995,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.4430000000000001,"5min":1.55,"15min":1.55},"last":0.94099999999999995}]}]},{"osd":1,"up_from":14,"seq":60129542186,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27856,"kb_used_data":828,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343984,"statfs":{"total":96636764160,"available":96608239616,"internally_reserved":0,"allocated":847872,"data_stored":369776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.61699999999999999,"15min":0.60499999999999998},"min":{"1min":0.51200000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.0880000000000001,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.746,"15min":0.72999999999999998},"min":{"1min":0.67400000000000004,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.0209999999999999,"5min":1.141,"15min":1.141},"last":0.66600000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.75800000000000001,"5min":0.64500000000000002,"15min":0.61699999999999999},"min":{"1min":0.54700000000000004,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.0629999999999999,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.70599999999999996,"15min":0.70299999999999996},"min":{"1min":0.41299999999999998,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90600000000000003,"5min":1.458,"15min":1.458},"last":0.78900000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.72599999999999998,"15min":0.69099999999999995},"min":{"1min":0.52300000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.026,"5min":1.236,"15min":1.236},"last":0.877},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.68400000000000005,"15min":0.68000000000000005},"min":{"1min":0.46899999999999997,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.94799999999999995,"5min":1.403,"15min":1.403},"last":0.59399999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.92500000000000004,"15min":0.90700000000000003},"min":{"1min":0.81699999999999995,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2569999999999999,"5min":1.546,"15min":1.546},"last":1.0209999999999999},{"interface":"front","average":{"1min":0.91600000000000004,"5min":0.92900000000000005,"15min":0.93300000000000005},"min":{"1min":0.61199999999999999,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3089999999999999,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":0.95899999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":0.95499999999999996,"15min":0.94699999999999995},"min":{"1min":0.80200000000000005,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.389,"5min":1.389,"15min":1.389},"last":1.054},{"interface":"front","average":{"1min":0.99199999999999999,"5min":0.97099999999999997,"15min":0.98199999999999998},"min":{"1min":0.77700000000000002,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.3979999999999999,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":1.0049999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.70899999999999996,"5min":0.626,"15min":0.626},"max":{"1min":1.4139999999999999,"5min":1.613,"15min":1.613},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.026,"5min":1.014,"15min":1.012},"min":{"1min":0.746,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.3480000000000001,"5min":1.698,"15min":1.698},"last":1.101}]},{"osd":7,"last update":"Fri Sep 13 16:28:11 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.008,"15min":1.01},"min":{"1min":0.748,"5min":0.62,"15min":0.62},"max":{"1min":1.494,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.054,"15min":1.042},"min":{"1min":0.873,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.448,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":0.97999999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542186,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32496,"kb_used_data":1380,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339344,"statfs":{"total":96636764160,"available":96603488256,"internally_reserved":0,"allocated":1413120,"data_stored":957816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68799999999999994,"5min":0.67100000000000004,"15min":0.67600000000000005},"min":{"1min":0.42599999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.0600000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.65900000000000003,"15min":0.65500000000000003},"min":{"1min":0.443,"5min":0.377,"15min":0.377},"max":{"1min":1.157,"5min":1.157,"15min":1.157},"last":0.85999999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.71799999999999997,"15min":0.72199999999999998},"min":{"1min":0.53600000000000003,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.202},"last":1.071},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.69599999999999995,"15min":0.68200000000000005},"min":{"1min":0.53500000000000003,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.2629999999999999,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.99199999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.68200000000000005,"15min":0.64800000000000002},"min":{"1min":0.53600000000000003,"5min":0.312,"15min":0.312},"max":{"1min":1.0069999999999999,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.96299999999999997,"5min":0.83899999999999997,"15min":0.82299999999999995},"min":{"1min":0.67800000000000005,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":1.196}]},{"osd":4,"last update":"Fri Sep 13 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.95799999999999996,"15min":0.96099999999999997},"min":{"1min":0.64400000000000002,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.486,"5min":1.486,"15min":1.486},"last":1.3759999999999999},{"interface":"front","average":{"1min":0.98799999999999999,"5min":0.97699999999999998,"15min":0.96099999999999997},"min":{"1min":0.70399999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5720000000000001,"5min":1.704,"15min":1.704},"last":1.3089999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":0.997,"15min":0.97499999999999998},"min":{"1min":0.71199999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.575,"5min":1.575,"15min":1.575},"last":1.343},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.995,"15min":0.999},"min":{"1min":0.72299999999999998,"5min":0.443,"15min":0.443},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":1.105}]},{"osd":6,"last update":"Fri Sep 13 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":0.999,"15min":0.97899999999999998},"min":{"1min":0.76900000000000002,"5min":0.628,"15min":0.628},"max":{"1min":1.24,"5min":1.944,"15min":1.944},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.1639999999999999,"5min":1.0600000000000001,"15min":1.048},"min":{"1min":0.82599999999999996,"5min":0.627,"15min":0.627},"max":{"1min":1.5780000000000001,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":1.417}]},{"osd":7,"last update":"Fri Sep 13 16:28:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.0629999999999999,"15min":1.0489999999999999},"min":{"1min":0.79100000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.659,"5min":1.659,"15min":1.659},"last":1.2190000000000001},{"interface":"front","average":{"1min":1.097,"5min":1.0760000000000001,"15min":1.042},"min":{"1min":0.83399999999999996,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.4590000000000001,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":1.129}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:28:30.019 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:28:30.019 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T16:28:30.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f40610c3700 1 Processor -- start 2024-09-13T16:28:30.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f40610c3700 1 -- start start 2024-09-13T16:28:30.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f40610c3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 0x7f405c06ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f40610c3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f405c06b2d0 0x7f405c06b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f40610c3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405c0ce880 con 0x7f405c06bc50 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f40610c3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f405c0cee40 con 0x7f405c06b2d0 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f40610c3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405c06c100 con 0x7f405c15cfb0 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f405b59e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f405c06bc50 0x7f405c0cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48964/0 (socket says 172.21.15.119:48964) 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f405b59e700 1 -- 172.21.15.119:0/3562473345 learned_addr learned my addr 172.21.15.119:0/3562473345 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f405ad9d700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 0x7f405c06ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f405a59c700 1 --2- 172.21.15.119:0/3562473345 >> v2:172.21.15.145:3300/0 conn(0x7f405c06b2d0 0x7f405c06b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181888101 0 0) 0x7f405c0ce880 con 0x7f405c06bc50 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40440035d0 con 0x7f405c06bc50 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1494946792 0 0) 0x7f40440035d0 con 0x7f405c06bc50 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 msgr2=0x7f405c06ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 0x7f405c06ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 >> v2:172.21.15.145:3300/0 conn(0x7f405c06b2d0 msgr2=0x7f405c06b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 --2- 172.21.15.119:0/3562473345 >> v2:172.21.15.145:3300/0 conn(0x7f405c06b2d0 0x7f405c06b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c0ceb60 con 0x7f405c06bc50 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f40540019f0 con 0x7f405c06bc50 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.145+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 >> v1:172.21.15.119:6789/0 conn(0x7f405c06bc50 legacy=0x7f405c0cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 0x7f4044003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f4044004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4044004db0 0x7f4044005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c0cee40 con 0x7f4044003f60 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405a59c700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f4044004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405ad9d700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 0x7f4044003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c06c100 con 0x7f4044004db0 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f404400ba30 con 0x7f405c15cfb0 2024-09-13T16:28:30.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405b59e700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4044004db0 0x7f4044005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405a59c700 1 -- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 msgr2=0x7f4044003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405a59c700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c15cfb0 0x7f4044003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405a59c700 1 -- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4044004db0 msgr2=0x7f4044005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405a59c700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4044004db0 0x7f4044005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405a59c700 1 -- 172.21.15.119:0/3562473345 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40440035d0 con 0x7f4044003f60 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405a59c700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f4044004870 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f404c009b40 tx=0x7f40440058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99b8fb8b9833820c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00e660 con 0x7f4044003f60 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f404c00e7c0 con 0x7f4044003f60 2024-09-13T16:28:30.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f4059d9b700 1 -- 172.21.15.119:0/3562473345 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f404c00eb30 con 0x7f4044003f60 2024-09-13T16:28:30.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f40610c3700 1 -- 172.21.15.119:0/3562473345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 msgr2=0x7f4044004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f40610c3700 1 --2- 172.21.15.119:0/3562473345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f4044004870 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f404c009b40 tx=0x7f40440058a0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f405b59e700 1 -- 172.21.15.119:0/3562473345 reap_dead start 2024-09-13T16:28:30.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f40610c3700 1 -- 172.21.15.119:0/3562473345 shutdown_connections 2024-09-13T16:28:30.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f40610c3700 1 -- 172.21.15.119:0/3562473345 >> 172.21.15.119:0/3562473345 conn(0x7f405c0d2da0 msgr2=0x7f405c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:30.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f40610c3700 1 -- 172.21.15.119:0/3562473345 shutdown_connections 2024-09-13T16:28:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.149+0000 7f40610c3700 1 -- 172.21.15.119:0/3562473345 wait complete. 2024-09-13T16:28:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 Processor -- start 2024-09-13T16:28:30.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 -- start start 2024-09-13T16:28:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f405c157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c0b8230 0x7f405c1457b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f405c0c9550 0x7f405c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c0d1220 con 0x7f4044003f60 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405c0c85a0 con 0x7f405c0c9550 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405c0c90c0 con 0x7f405c0b8230 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f405c157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f405c157580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33806/0 (socket says 172.21.15.119:33806) 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 -- 172.21.15.119:0/3468401252 learned_addr learned my addr 172.21.15.119:0/3468401252 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405a59c700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c0b8230 0x7f405c1457b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405b59e700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f405c0c9550 0x7f405c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 -- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c0b8230 msgr2=0x7f405c1457b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c0b8230 0x7f405c1457b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 -- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f405c0c9550 msgr2=0x7f405c0c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f405c0c9550 0x7f405c0c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 -- 172.21.15.119:0/3468401252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f405c145cf0 con 0x7f4044003f60 2024-09-13T16:28:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f405ad9d700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f405c157580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4050011a20 tx=0x7f4050011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:30.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40437fe700 1 -- 172.21.15.119:0/3468401252 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4050011ec0 con 0x7f4044003f60 2024-09-13T16:28:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40437fe700 1 -- 172.21.15.119:0/3468401252 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405000f940 con 0x7f4044003f60 2024-09-13T16:28:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405c157b60 con 0x7f4044003f60 2024-09-13T16:28:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40437fe700 1 -- 172.21.15.119:0/3468401252 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f405000fcd0 con 0x7f4044003f60 2024-09-13T16:28:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.153+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f405c0c8a70 con 0x7f4044003f60 2024-09-13T16:28:30.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.157+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405c06b710 con 0x7f4044003f60 2024-09-13T16:28:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.157+0000 7f40437fe700 1 -- 172.21.15.119:0/3468401252 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f405001d400 con 0x7f4044003f60 2024-09-13T16:28:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.157+0000 7f40437fe700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4048066730 0x7f4048068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.157+0000 7f40437fe700 1 -- 172.21.15.119:0/3468401252 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4050084800 con 0x7f4044003f60 2024-09-13T16:28:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.157+0000 7f405a59c700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4048066730 0x7f4048068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.157+0000 7f405a59c700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4048066730 0x7f4048068bf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f404c0090b0 tx=0x7f404c01e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:30.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.165+0000 7f40437fe700 1 -- 172.21.15.119:0/3468401252 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4050051af0 con 0x7f4044003f60 2024-09-13T16:28:30.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.393+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f405c0b37e0 con 0x7f4044003f60 2024-09-13T16:28:30.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.401+0000 7f40437fe700 1 -- 172.21.15.119:0/3468401252 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v13) v1 ==== 190+0+0 (secure 0 0 0) 0x7f4050055110 con 0x7f4044003f60 2024-09-13T16:28:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.401+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4048066730 msgr2=0x7f4048068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.401+0000 7f40610c3700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4048066730 0x7f4048068bf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f404c0090b0 tx=0x7f404c01e040 comp rx=0 tx=0).stop 2024-09-13T16:28:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.401+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 msgr2=0x7f405c157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.401+0000 7f40610c3700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f405c157580 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f4050011a20 tx=0x7f4050011d30 comp rx=0 tx=0).stop 2024-09-13T16:28:30.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.401+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 shutdown_connections 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.405+0000 7f40610c3700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4048066730 0x7f4048068bf0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.405+0000 7f40610c3700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f405c0c9550 0x7f405c0c6730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.405+0000 7f40610c3700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f405c0b8230 0x7f405c1457b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.405+0000 7f40610c3700 1 --2- 172.21.15.119:0/3468401252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4044003f60 0x7f405c157580 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.405+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 >> 172.21.15.119:0/3468401252 conn(0x7f405c0d2da0 msgr2=0x7f405c0d1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.405+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 shutdown_connections 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.405+0000 7f40610c3700 1 -- 172.21.15.119:0/3468401252 wait complete. 2024-09-13T16:28:30.409 INFO:teuthology.orchestra.run.smithi119.stderr:updated caps for client.0 2024-09-13T16:28:30.422 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:30.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.533+0000 7f7ab32d8700 1 Processor -- start 2024-09-13T16:28:30.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.533+0000 7f7ab32d8700 1 -- start start 2024-09-13T16:28:30.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.533+0000 7f7ab32d8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac1570d0 0x7f7aac06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.533+0000 7f7ab32d8700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7aac06e270 0x7f7aac06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.533+0000 7f7ab32d8700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7aac06ebf0 con 0x7f7aac0d0f70 2024-09-13T16:28:30.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.533+0000 7f7ab32d8700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7aac0ce730 con 0x7f7aac06e270 2024-09-13T16:28:30.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.533+0000 7f7ab32d8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0cf840 con 0x7f7aac1570d0 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7aac06e270 0x7f7aac06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab22d6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac1570d0 0x7f7aac06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7aac06e270 0x7f7aac06e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:42526/0 (socket says 172.21.15.119:42526) 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1535748999 learned_addr learned my addr 172.21.15.119:0/1535748999 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1535748999 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac1570d0 msgr2=0x7f7aac06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 --2- 172.21.15.119:0/1535748999 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac1570d0 0x7f7aac06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1535748999 >> v1:172.21.15.145:6789/0 conn(0x7f7aac0d0f70 legacy=0x7f7aac0d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1535748999 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aac0cfb20 con 0x7f7aac06e270 2024-09-13T16:28:30.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab1ad5700 1 --2- 172.21.15.119:0/1535748999 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac06e270 0x7f7aac06e6b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009910 tx=0x7f7a9c009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ad0cec59a2b7465 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab12d4700 1 -- 172.21.15.119:0/1535748999 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c011830 con 0x7f7aac06e270 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab12d4700 1 -- 172.21.15.119:0/1535748999 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c011990 con 0x7f7aac06e270 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab12d4700 1 -- 172.21.15.119:0/1535748999 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c011d40 con 0x7f7aac06e270 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1535748999 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac06e270 msgr2=0x7f7aac06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1535748999 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac06e270 0x7f7aac06e6b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009910 tx=0x7f7a9c009c20 comp rx=0 tx=0).stop 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1535748999 shutdown_connections 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1535748999 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac06e270 0x7f7aac06e6b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1535748999 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac1570d0 0x7f7aac06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1535748999 >> 172.21.15.119:0/1535748999 conn(0x7f7aac099a50 msgr2=0x7f7aac099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1535748999 shutdown_connections 2024-09-13T16:28:30.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.537+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1535748999 wait complete. 2024-09-13T16:28:30.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 Processor -- start 2024-09-13T16:28:30.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 -- start start 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac06e270 0x7f7aac0b65f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7aac0d0f70 0x7f7aac13c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac1570d0 0x7f7aac0b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0b6b30 con 0x7f7aac0d0f70 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0b3a80 con 0x7f7aac1570d0 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab32d8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac0b4320 con 0x7f7aac06e270 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7aac0d0f70 0x7f7aac13c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7aac0d0f70 0x7f7aac13c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33810/0 (socket says 172.21.15.119:33810) 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1052695969 learned_addr learned my addr 172.21.15.119:0/1052695969 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab22d6700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac06e270 0x7f7aac0b65f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab2ad7700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac1570d0 0x7f7aac0b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac06e270 msgr2=0x7f7aac0b65f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac06e270 0x7f7aac0b65f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac1570d0 msgr2=0x7f7aac0b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac1570d0 0x7f7aac0b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 -- 172.21.15.119:0/1052695969 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c009570 con 0x7f7aac0d0f70 2024-09-13T16:28:30.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.541+0000 7f7ab1ad5700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7aac0d0f70 0x7f7aac13c2f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009c60 tx=0x7f7a9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:30.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.545+0000 7f7aa2ffd700 1 -- 172.21.15.119:0/1052695969 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c004300 con 0x7f7aac0d0f70 2024-09-13T16:28:30.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.545+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac14ee30 con 0x7f7aac0d0f70 2024-09-13T16:28:30.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.545+0000 7f7aa2ffd700 1 -- 172.21.15.119:0/1052695969 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a9c01f070 con 0x7f7aac0d0f70 2024-09-13T16:28:30.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.545+0000 7f7aa2ffd700 1 -- 172.21.15.119:0/1052695969 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a9c0036c0 con 0x7f7aac0d0f70 2024-09-13T16:28:30.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.545+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac0b3fb0 con 0x7f7aac0d0f70 2024-09-13T16:28:30.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.545+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac06e6b0 con 0x7f7aac0d0f70 2024-09-13T16:28:30.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.545+0000 7f7aa2ffd700 1 -- 172.21.15.119:0/1052695969 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7a9c021030 con 0x7f7aac0d0f70 2024-09-13T16:28:30.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.549+0000 7f7aa2ffd700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a90066730 0x7f7a90068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:30.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.549+0000 7f7aa2ffd700 1 -- 172.21.15.119:0/1052695969 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7a9c085860 con 0x7f7aac0d0f70 2024-09-13T16:28:30.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.549+0000 7f7ab22d6700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a90066730 0x7f7a90068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:30.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.549+0000 7f7ab22d6700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a90066730 0x7f7a90068bf0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009b50 tx=0x7f7aa8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:30.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.557+0000 7f7aa2ffd700 1 -- 172.21.15.119:0/1052695969 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7a9c052bc0 con 0x7f7aac0d0f70 2024-09-13T16:28:30.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.757+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7aac06c610 con 0x7f7aac0d0f70 2024-09-13T16:28:30.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7aa2ffd700 1 -- 172.21.15.119:0/1052695969 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f7a9c0561e0 con 0x7f7aac0d0f70 2024-09-13T16:28:30.765 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:30.765 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:28:24.129975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:28:30.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a90066730 msgr2=0x7f7a90068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a90066730 0x7f7a90068bf0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009b50 tx=0x7f7aa8009240 comp rx=0 tx=0).stop 2024-09-13T16:28:30.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7aac0d0f70 msgr2=0x7f7aac13c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:30.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7aac0d0f70 0x7f7aac13c2f0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f7a9c009c60 tx=0x7f7a9c004770 comp rx=0 tx=0).stop 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 shutdown_connections 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7a90066730 0x7f7a90068bf0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7aac1570d0 0x7f7aac0b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7aac0d0f70 0x7f7aac13c2f0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 --2- 172.21.15.119:0/1052695969 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7aac06e270 0x7f7aac0b65f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 >> 172.21.15.119:0/1052695969 conn(0x7f7aac099a50 msgr2=0x7f7aac06cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 shutdown_connections 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:30.761+0000 7f7ab32d8700 1 -- 172.21.15.119:0/1052695969 wait complete. 2024-09-13T16:28:30.767 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 32 2024-09-13T16:28:31.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:28:21.113532+0000', 'modified': '2024-09-13T16:28:24.129975+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6546}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6546': {'gid': 6546, 'name': 'd', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6837/3224929185', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6836', 'nonce': 3224929185}, {'type': 'v1', 'addr': '172.21.15.145:6837', 'nonce': 3224929185}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:28:31.180 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:28:31.180 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-09-13T16:28:31.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ae2ca700 1 Processor -- start 2024-09-13T16:28:31.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ae2ca700 1 -- start start 2024-09-13T16:28:31.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ae2ca700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd3a80cc060 0x7fd3a80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ae2ca700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80cde70 0x7fd3a80b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ae2ca700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a806cf60 con 0x7fd3a80adb30 2024-09-13T16:28:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ae2ca700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80cc4d0 con 0x7fd3a80cc060 2024-09-13T16:28:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ae2ca700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a806c350 con 0x7fd3a80cde70 2024-09-13T16:28:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ad2c8700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd3a80adb30 0x7fd3a806bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48978/0 (socket says 172.21.15.119:48978) 2024-09-13T16:28:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3ad2c8700 1 -- 172.21.15.119:0/450951667 learned_addr learned my addr 172.21.15.119:0/450951667 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3acac7700 1 --2- 172.21.15.119:0/450951667 >> v2:172.21.15.145:3300/0 conn(0x7fd3a80cc060 0x7fd3a80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3adac9700 1 --2- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80cde70 0x7fd3a80b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd39ffff700 1 -- 172.21.15.119:0/450951667 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139872670 0 0) 0x7fd3a806cf60 con 0x7fd3a80adb30 2024-09-13T16:28:31.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd39ffff700 1 -- 172.21.15.119:0/450951667 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3900035d0 con 0x7fd3a80adb30 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3acac7700 1 -- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80cde70 msgr2=0x7fd3a80b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3acac7700 1 --2- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80cde70 0x7fd3a80b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3acac7700 1 -- 172.21.15.119:0/450951667 >> v1:172.21.15.119:6789/0 conn(0x7fd3a80adb30 legacy=0x7fd3a806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.305+0000 7fd3acac7700 1 -- 172.21.15.119:0/450951667 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a806c630 con 0x7fd3a80cc060 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3acac7700 1 --2- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cc060 0x7fd3a80cd930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009d90 tx=0x7fd3a400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69e69ac81ddecaaf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd39ffff700 1 -- 172.21.15.119:0/450951667 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400e070 con 0x7fd3a80cc060 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd39ffff700 1 -- 172.21.15.119:0/450951667 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a400bb20 con 0x7fd3a80cc060 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd39ffff700 1 -- 172.21.15.119:0/450951667 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4012430 con 0x7fd3a80cc060 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cc060 msgr2=0x7fd3a80cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cc060 0x7fd3a80cd930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4009d90 tx=0x7fd3a400b330 comp rx=0 tx=0).stop 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/450951667 shutdown_connections 2024-09-13T16:28:31.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80cde70 0x7fd3a80b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/450951667 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cc060 0x7fd3a80cd930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/450951667 >> 172.21.15.119:0/450951667 conn(0x7fd3a806a350 msgr2=0x7fd3a806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/450951667 shutdown_connections 2024-09-13T16:28:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/450951667 wait complete. 2024-09-13T16:28:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 Processor -- start 2024-09-13T16:28:31.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.309+0000 7fd3ae2ca700 1 -- start start 2024-09-13T16:28:31.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80adb30 0x7fd3a815a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3a80cc060 0x7fd3a81f3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cde70 0x7fd3a8141b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80a8780 con 0x7fd3a80cc060 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a81f4890 con 0x7fd3a80cde70 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a81f4b70 con 0x7fd3a80adb30 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3a80cc060 0x7fd3a81f3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3a80cc060 0x7fd3a81f3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33816/0 (socket says 172.21.15.119:33816) 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 -- 172.21.15.119:0/2953350845 learned_addr learned my addr 172.21.15.119:0/2953350845 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80adb30 0x7fd3a815a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3adac9700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cde70 0x7fd3a8141b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 -- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80adb30 msgr2=0x7fd3a815a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80adb30 0x7fd3a815a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 -- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cde70 msgr2=0x7fd3a8141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cde70 0x7fd3a8141b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 -- 172.21.15.119:0/2953350845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a4009a70 con 0x7fd3a80cc060 2024-09-13T16:28:31.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3a80cc060 0x7fd3a81f3e40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4014040 tx=0x7fd3a4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd39dffb700 1 -- 172.21.15.119:0/2953350845 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4012430 con 0x7fd3a80cc060 2024-09-13T16:28:31.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd39dffb700 1 -- 172.21.15.119:0/2953350845 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a4020070 con 0x7fd3a80cc060 2024-09-13T16:28:31.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a80a89d0 con 0x7fd3a80cc060 2024-09-13T16:28:31.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd39dffb700 1 -- 172.21.15.119:0/2953350845 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4004280 con 0x7fd3a80cc060 2024-09-13T16:28:31.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.313+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a815a5d0 con 0x7fd3a80cc060 2024-09-13T16:28:31.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.317+0000 7fd39dffb700 1 -- 172.21.15.119:0/2953350845 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd3a40043e0 con 0x7fd3a80cc060 2024-09-13T16:28:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.317+0000 7fd39dffb700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd394066730 0x7fd394068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.317+0000 7fd39dffb700 1 -- 172.21.15.119:0/2953350845 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd3a4022070 con 0x7fd3a80cc060 2024-09-13T16:28:31.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.317+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd394066730 0x7fd394068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.317+0000 7fd3b27dd740 1 Processor -- start 2024-09-13T16:28:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.317+0000 7fd3b27dd740 1 -- start start 2024-09-13T16:28:31.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.317+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd394066730 0x7fd394068bf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00011e0 tx=0x7fd3a0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1daf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd7540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d1cd80 con 0x1db0670 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d1a510 con 0x1daf530 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d15e00 con 0x1db0290 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd7a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33822/0 (socket says 172.21.15.119:33822) 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 -- 172.21.15.119:0/2961395871 learned_addr learned my addr 172.21.15.119:0/2961395871 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1daf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 -- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 msgr2=0x1dd7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 -- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 msgr2=0x1daf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1daf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 -- 172.21.15.119:0/2961395871 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1b88960 con 0x1db0670 2024-09-13T16:28:31.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:31.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3adac9700 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd7a80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd398009b10 tx=0x7fd398009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea348940956f7084 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd38effd700 1 -- 172.21.15.119:0/2961395871 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398011950 con 0x1db0670 2024-09-13T16:28:31.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd38effd700 1 -- 172.21.15.119:0/2961395871 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398011ab0 con 0x1db0670 2024-09-13T16:28:31.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd38effd700 1 -- 172.21.15.119:0/2961395871 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398011e40 con 0x1db0670 2024-09-13T16:28:31.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2961395871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 msgr2=0x1dd7a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd7a80 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd398009b10 tx=0x7fd398009240 comp rx=0 tx=0).stop 2024-09-13T16:28:31.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2961395871 shutdown_connections 2024-09-13T16:28:31.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd7a80 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2961395871 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1daf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2961395871 >> 172.21.15.119:0/2961395871 conn(0x1b3d4d0 msgr2=0x1dd6e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2961395871 shutdown_connections 2024-09-13T16:28:31.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2961395871 wait complete. 2024-09-13T16:28:31.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.321+0000 7fd3b27dd740 1 -- start start 2024-09-13T16:28:31.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 Processor -- start 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1daf530 0x1dd72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dd1a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0670 0x1dd1f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cd3730 con 0x1daf530 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cd95b0 con 0x1db0290 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1bc1aa0 con 0x1db0670 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 10 client.0 _ll_register_callbacks cb 0x1d0f610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1daf530 0x1dd72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1daf530 0x1dd72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33834/0 (socket says 172.21.15.119:33834) 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 -- 172.21.15.119:0/2023654406 learned_addr learned my addr 172.21.15.119:0/2023654406 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:31.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dd1a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3adac9700 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0670 0x1dd1f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 -- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0670 msgr2=0x1dd1f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0670 0x1dd1f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 -- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 msgr2=0x1dd1a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dd1a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 -- 172.21.15.119:0/2023654406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3980097c0 con 0x1daf530 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1daf530 0x1dd72a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0007170 tx=0x7fd3a0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab90c766ae0a801 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd387fff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd387fff700 1 -- 172.21.15.119:0/2023654406 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a000e750 con 0x1daf530 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd387fff700 1 -- 172.21.15.119:0/2023654406 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd3a0022bd0 con 0x1daf530 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1b4a1e0 con 0x1daf530 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd387fff700 1 -- 172.21.15.119:0/2023654406 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd3a0022d90 con 0x1daf530 2024-09-13T16:28:31.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 10 client.6705 fetch_fsmap learned FSMap version 32 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x1cdc220 con 0x1daf530 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd387fff700 1 -- 172.21.15.119:0/2023654406 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1b4a1e0 con 0x1daf530 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd387fff700 1 -- 172.21.15.119:0/2023654406 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 32) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd3a000e8b0 con 0x1daf530 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 10 client.6705 fetch_fsmap finished waiting for FSMap version 32 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 10 client.6705 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 20 client.6705 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1dd9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 4 client.6705 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.325+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1b94460 con 0x1df6fe0 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1dd9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd387fff700 10 client.6705 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:31.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3acac7700 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1dd9990 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85581d5ba280a345 server_cookie=ebd5a9c801a82fdb in_seq=0 out_seq=0 2024-09-13T16:28:31.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd387fff700 1 -- 172.21.15.119:0/2023654406 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1b94460 con 0x1df6fe0 2024-09-13T16:28:31.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd387fff700 10 client.6705 handle_command_reply: tid=0 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 client.6705 shutdown 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1daf530 msgr2=0x1dd72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1daf530 0x1dd72a0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd3a0007170 tx=0x7fd3a0000ed0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 shutdown_connections 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1dd9990 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0670 0x1dd1f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dd1a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd387fff700 0 client.6705 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/2023654406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1daf530 0x1dd72a0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 >> 172.21.15.119:0/2023654406 conn(0x1b3d4d0 msgr2=0x1dd8060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 shutdown_connections 2024-09-13T16:28:31.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 1 -- 172.21.15.119:0/2023654406 wait complete. 2024-09-13T16:28:31.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.329+0000 7fd3b27dd740 20 client.6705 trim_cache size 0 max 16384 2024-09-13T16:28:31.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.349+0000 7fd3b27dd740 1 Processor -- start 2024-09-13T16:28:31.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.349+0000 7fd3b27dd740 1 -- start start 2024-09-13T16:28:31.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1dd1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cbb1c0 con 0x1db0670 2024-09-13T16:28:31.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cc9e10 con 0x1daf530 2024-09-13T16:28:31.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ce66b0 con 0x1db0290 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3acac7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33850/0 (socket says 172.21.15.119:33850) 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3ad2c8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1dd1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 -- 172.21.15.119:0/1934829861 learned_addr learned my addr 172.21.15.119:0/1934829861 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 -- 172.21.15.119:0/1934829861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 msgr2=0x1dd1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 --2- 172.21.15.119:0/1934829861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 -- 172.21.15.119:0/1934829861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 msgr2=0x1dd1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 --2- 172.21.15.119:0/1934829861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1dd1f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 -- 172.21.15.119:0/1934829861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1cc2450 con 0x1db0670 2024-09-13T16:28:31.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3adac9700 1 --2- 172.21.15.119:0/1934829861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd1590 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd398018000 tx=0x7fd398016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e17e09bdad6e82b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd38f7fe700 1 -- 172.21.15.119:0/1934829861 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd39802d070 con 0x1db0670 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd38f7fe700 1 -- 172.21.15.119:0/1934829861 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398024040 con 0x1db0670 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd38f7fe700 1 -- 172.21.15.119:0/1934829861 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398023050 con 0x1db0670 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- 172.21.15.119:0/1934829861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 msgr2=0x1dd1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/1934829861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd1590 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fd398018000 tx=0x7fd398016000 comp rx=0 tx=0).stop 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- 172.21.15.119:0/1934829861 shutdown_connections 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/1934829861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1dd1590 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/1934829861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1db0290 0x1dd1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/1934829861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1daf530 0x1dd1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- 172.21.15.119:0/1934829861 >> 172.21.15.119:0/1934829861 conn(0x1b3d4d0 msgr2=0x1e09350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- 172.21.15.119:0/1934829861 shutdown_connections 2024-09-13T16:28:31.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- 172.21.15.119:0/1934829861 wait complete. 2024-09-13T16:28:31.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.353+0000 7fd3b27dd740 1 -- start start 2024-09-13T16:28:31.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 Processor -- start 2024-09-13T16:28:31.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1daf530 0x1dae0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dae5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1e5eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dfab10 con 0x1db0670 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1daec40 con 0x1db0290 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1daeee0 con 0x1daf530 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 10 client.0 _ll_register_callbacks cb 0x1db0b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1e5eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1e5eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33866/0 (socket says 172.21.15.119:33866) 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 -- 172.21.15.119:0/4203534178 learned_addr learned my addr 172.21.15.119:0/4203534178 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1daf530 0x1dae0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3acac7700 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dae5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 -- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1daf530 msgr2=0x1dae0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1daf530 0x1dae0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 -- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 msgr2=0x1dae5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dae5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 -- 172.21.15.119:0/4203534178 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1b88960 con 0x1db0670 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd387fff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3adac9700 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1e5eaf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd39802b040 tx=0x7fd3980047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c019cbe2badcd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd387fff700 1 -- 172.21.15.119:0/4203534178 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd398023070 con 0x1db0670 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd387fff700 1 -- 172.21.15.119:0/4203534178 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd398033c00 con 0x1db0670 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd398011ab0 con 0x1db0670 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd387fff700 1 -- 172.21.15.119:0/4203534178 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd39802d040 con 0x1db0670 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 10 client.6711 fetch_fsmap learned FSMap version 32 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x1cb6720 con 0x1db0670 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd387fff700 1 -- 172.21.15.119:0/4203534178 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd398011ab0 con 0x1db0670 2024-09-13T16:28:31.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd387fff700 1 -- 172.21.15.119:0/4203534178 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 32) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fd398004230 con 0x1db0670 2024-09-13T16:28:31.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 10 client.6711 fetch_fsmap finished waiting for FSMap version 32 2024-09-13T16:28:31.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 10 client.6711 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:28:31.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 20 client.6711 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:31.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1df9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 4 client.6711 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:28:31.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "status"}) v1 -- 0x1bb91e0 con 0x1df6fe0 2024-09-13T16:28:31.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.357+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1df9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd387fff700 10 client.6711 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ad2c8700 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1df9050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e519ff71873ea872 server_cookie=5a4c2a7a13c2187c in_seq=0 out_seq=0 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd387fff700 1 -- 172.21.15.119:0/4203534178 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x1bb91e0 con 0x1df6fe0 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd387fff700 10 client.6711 handle_command_reply: tid=0 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 client.6711 shutdown 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 msgr2=0x1e5eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1e5eaf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fd39802b040 tx=0x7fd3980047e0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 shutdown_connections 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1df6fe0 0x1df9050 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd387fff700 0 client.6711 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1db0670 0x1e5eaf0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1db0290 0x1dae5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 --2- 172.21.15.119:0/4203534178 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1daf530 0x1dae0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 >> 172.21.15.119:0/4203534178 conn(0x1b3d4d0 msgr2=0x1e09350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 shutdown_connections 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 1 -- 172.21.15.119:0/4203534178 wait complete. 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3b27dd740 20 client.6711 trim_cache size 0 max 16384 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 6546, 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:28:31.368 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:28:31.369 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:28:31.369 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 10.246606618, 2024-09-13T16:28:31.369 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 32, 2024-09-13T16:28:31.369 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 68, 2024-09-13T16:28:31.369 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 68, 2024-09-13T16:28:31.369 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 14.761918627 2024-09-13T16:28:31.369 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd394066730 msgr2=0x7fd394068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd394066730 0x7fd394068bf0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fd3a00011e0 tx=0x7fd3a0009040 comp rx=0 tx=0).stop 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3a80cc060 msgr2=0x7fd3a81f3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3a80cc060 0x7fd3a81f3e40 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7fd3a4014040 tx=0x7fd3a4004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 shutdown_connections 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd394066730 0x7fd394068bf0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3a80cde70 0x7fd3a8141b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3a80cc060 0x7fd3a81f3e40 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 --2- 172.21.15.119:0/2953350845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd3a80adb30 0x7fd3a815a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 >> 172.21.15.119:0/2953350845 conn(0x7fd3a806a350 msgr2=0x7fd3a80b2810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 shutdown_connections 2024-09-13T16:28:31.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.361+0000 7fd3ae2ca700 1 -- 172.21.15.119:0/2953350845 wait complete. 2024-09-13T16:28:31.385 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:28:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240d204700 1 Processor -- start 2024-09-13T16:28:31.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240d204700 1 -- start start 2024-09-13T16:28:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240d204700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f24080ce480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240d204700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f24080ce9c0 0x7f240806ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240d204700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24080cee20 con 0x7f240806f2a0 2024-09-13T16:28:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240d204700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24080ad8e0 con 0x7f24080ce9c0 2024-09-13T16:28:31.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240d204700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24080adea0 con 0x7f2408146420 2024-09-13T16:28:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f2407fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f24080ce480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f24077fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f24080ce9c0 0x7f240806ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f2407fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f24080ce480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:35106/0 (socket says 172.21.15.119:35106) 2024-09-13T16:28:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f240ca03700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f240806f2a0 0x7f24080d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:55360/0 (socket says 172.21.15.119:55360) 2024-09-13T16:28:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f2407fff700 1 -- 172.21.15.119:0/3480807100 learned_addr learned my addr 172.21.15.119:0/3480807100 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:31.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f2406ffd700 1 -- 172.21.15.119:0/3480807100 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894628910 0 0) 0x7f24080cee20 con 0x7f240806f2a0 2024-09-13T16:28:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f2406ffd700 1 -- 172.21.15.119:0/3480807100 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23f40035d0 con 0x7f240806f2a0 2024-09-13T16:28:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f24077fe700 1 -- 172.21.15.119:0/3480807100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 msgr2=0x7f24080ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f24077fe700 1 --2- 172.21.15.119:0/3480807100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f24080ce480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f24077fe700 1 -- 172.21.15.119:0/3480807100 >> v1:172.21.15.145:6789/0 conn(0x7f240806f2a0 legacy=0x7f24080d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f24077fe700 1 -- 172.21.15.119:0/3480807100 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f240806f750 con 0x7f24080ce9c0 2024-09-13T16:28:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.501+0000 7f24077fe700 1 --2- 172.21.15.119:0/3480807100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24080ce9c0 0x7f240806ed60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009d90 tx=0x7f23f800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=987229a8fc53edc4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f2406ffd700 1 -- 172.21.15.119:0/3480807100 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f800e070 con 0x7f24080ce9c0 2024-09-13T16:28:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f2406ffd700 1 -- 172.21.15.119:0/3480807100 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f800bb20 con 0x7f24080ce9c0 2024-09-13T16:28:31.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f2406ffd700 1 -- 172.21.15.119:0/3480807100 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8012430 con 0x7f24080ce9c0 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 -- 172.21.15.119:0/3480807100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24080ce9c0 msgr2=0x7f240806ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 --2- 172.21.15.119:0/3480807100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24080ce9c0 0x7f240806ed60 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f23f8009d90 tx=0x7f23f800b330 comp rx=0 tx=0).stop 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 -- 172.21.15.119:0/3480807100 shutdown_connections 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 --2- 172.21.15.119:0/3480807100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24080ce9c0 0x7f240806ed60 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 --2- 172.21.15.119:0/3480807100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f24080ce480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 -- 172.21.15.119:0/3480807100 >> 172.21.15.119:0/3480807100 conn(0x7f2408099a50 msgr2=0x7f2408099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 -- 172.21.15.119:0/3480807100 shutdown_connections 2024-09-13T16:28:31.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.505+0000 7f240d204700 1 -- 172.21.15.119:0/3480807100 wait complete. 2024-09-13T16:28:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 Processor -- start 2024-09-13T16:28:31.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 -- start start 2024-09-13T16:28:31.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f240806f2a0 0x7f240813d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24080ce9c0 0x7f24080b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f2408144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240813da60 con 0x7f24080ce9c0 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240813eac0 con 0x7f240806f2a0 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240d204700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f240813dbd0 con 0x7f2408146420 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24080ce9c0 0x7f24080b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24080ce9c0 0x7f24080b5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33874/0 (socket says 172.21.15.119:33874) 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 -- 172.21.15.119:0/3684619662 learned_addr learned my addr 172.21.15.119:0/3684619662 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f2407fff700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f240806f2a0 0x7f240813d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240ca03700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f2408144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 -- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 msgr2=0x7f2408144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f2408144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 -- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f240806f2a0 msgr2=0x7f240813d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f240806f2a0 0x7f240813d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f24077fe700 1 -- 172.21.15.119:0/3684619662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f8009a70 con 0x7f24080ce9c0 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.509+0000 7f240ca03700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f2408144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f24077fe700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24080ce9c0 0x7f24080b5b40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f23f8014040 tx=0x7f23f8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f2404ff9700 1 -- 172.21.15.119:0/3684619662 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8012430 con 0x7f24080ce9c0 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f2404ff9700 1 -- 172.21.15.119:0/3684619662 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23f8020070 con 0x7f24080ce9c0 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240813deb0 con 0x7f24080ce9c0 2024-09-13T16:28:31.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f2404ff9700 1 -- 172.21.15.119:0/3684619662 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23f8004280 con 0x7f24080ce9c0 2024-09-13T16:28:31.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240813e470 con 0x7f24080ce9c0 2024-09-13T16:28:31.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f240813ec70 con 0x7f24080ce9c0 2024-09-13T16:28:31.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f2404ff9700 1 -- 172.21.15.119:0/3684619662 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f23f80043e0 con 0x7f24080ce9c0 2024-09-13T16:28:31.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f2404ff9700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23e8066760 0x7f23e8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:31.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f2404ff9700 1 -- 172.21.15.119:0/3684619662 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f23f8022070 con 0x7f24080ce9c0 2024-09-13T16:28:31.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.513+0000 7f2407fff700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23e8066760 0x7f23e8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:31.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.517+0000 7f2407fff700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23e8066760 0x7f23e8068c20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f23f0006d70 tx=0x7f23f00092e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:31.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.517+0000 7f2404ff9700 1 -- 172.21.15.119:0/3684619662 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f240813ec70 con 0x7f24080ce9c0 2024-09-13T16:28:31.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.725+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f240809a1c0 con 0x7f24080ce9c0 2024-09-13T16:28:31.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.725+0000 7f2404ff9700 1 -- 172.21.15.119:0/3684619662 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7f23f8053070 con 0x7f24080ce9c0 2024-09-13T16:28:31.730 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23e8066760 msgr2=0x7f23e8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23e8066760 0x7f23e8068c20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f23f0006d70 tx=0x7f23f00092e0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24080ce9c0 msgr2=0x7f24080b5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24080ce9c0 0x7f24080b5b40 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f23f8014040 tx=0x7f23f8004af0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 shutdown_connections 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f23e8066760 0x7f23e8068c20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2408146420 0x7f2408144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24080ce9c0 0x7f24080b5b40 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 --2- 172.21.15.119:0/3684619662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f240806f2a0 0x7f240813d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:31.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 >> 172.21.15.119:0/3684619662 conn(0x7f2408099a50 msgr2=0x7f24080af6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:31.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 shutdown_connections 2024-09-13T16:28:31.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:31.729+0000 7f240d204700 1 -- 172.21.15.119:0/3684619662 wait complete. 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:28:31.745 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:28:31.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:31.746 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:28:31.754 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42429sec preferred_lft 42429sec 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:28:31.755 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:28:31.756 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:28:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:28:31.757 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:28:31.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:31.757 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:28:31.810 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:28:31.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:31.812 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:28:31.821 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:28:31.821 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:28:31.821 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:28:31.821 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:28:31.821 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:28:31.822 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:28:31.822 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:28:31.822 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:28:31.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:31.822 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:28:31.822 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:28:31.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:28:31.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:28:31.822 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:28:31.822 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:28:31.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:31.861 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:28:31.910 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:28:31.910 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:28:31.910 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:28:31.910 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:31.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:28:31.924 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:28:31.924 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:28:31.924 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:28:31.932 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:28:31.932 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:28:31.932 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:28:31.981 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:28:31.995 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:28:31.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:28:32.006 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:28:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:28:32.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:32.007 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:28:32.012 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:28:32.013 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:28:32.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:28:32.073 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:28:32.073 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:28:32.083 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:28:32.085 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:28:32.085 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:32.085 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:28:32.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:28:32.141+0000 7fa3865c3640 -1 init, newargv = 0x55ff6d5e8700 newargc=15 2024-09-13T16:28:32.144 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[42159]: starting ceph client 2024-09-13T16:28:32.153 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[42159]: starting fuse 2024-09-13T16:28:33.234 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:28:33.250 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:28:33.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:28:33.266 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:28:33.267 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:28:33.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:33.268 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:28:33.317 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:28:33.317 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:28:33.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:28:33.318 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:28:33.319 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:28:33.400 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.42159.asok 2024-09-13T16:28:33.582 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.42159.asok 2024-09-13T16:28:33.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:33.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.42159.asok status 2024-09-13T16:28:33.687 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:33.687 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:28:33.687 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "42159", 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 6723, 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:28:33.688 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 6723 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4149966126 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4149966126 2024-09-13T16:28:33.689 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.6723 192.168.0.1:0/4149966126", 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/4149966126", 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 32, 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 68, 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:28:33.690 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:33.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:33.698 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:28:33.698 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:28:33.704 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:28:33.704 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:28:33.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:33.704 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:28:33.721 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:33.721 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir_test) 2024-09-13T16:28:33.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:28:33.741 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir_test/file) 2024-09-13T16:28:33.801 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:28:33.801 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:28:33.802 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:28:33.802 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:28:33.802 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir_test/file").st_ino) 2024-09-13T16:28:33.802 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:28:33.871 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627777 2024-09-13T16:28:34.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:28:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3dbb9700 1 Processor -- start 2024-09-13T16:28:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3dbb9700 1 -- start start 2024-09-13T16:28:34.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3dbb9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b17f0 0x7f4a380cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3dbb9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4a380b2200 0x7f4a380b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3dbb9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a380705d0 con 0x7f4a380cd8d0 2024-09-13T16:28:34.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3dbb9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a380b2bf0 con 0x7f4a380b2200 2024-09-13T16:28:34.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3dbb9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3806fca0 con 0x7f4a380b17f0 2024-09-13T16:28:34.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3cbb7700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4a380cd8d0 0x7f4a3806cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33232/0 (socket says 172.21.15.119:33232) 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3cbb7700 1 -- 172.21.15.119:0/1926950190 learned_addr learned my addr 172.21.15.119:0/1926950190 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3d3b8700 1 --2- 172.21.15.119:0/1926950190 >> v2:172.21.15.119:3300/0 conn(0x7f4a380b2200 0x7f4a380b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a37fff700 1 --2- 172.21.15.119:0/1926950190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b17f0 0x7f4a380cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a377fe700 1 -- 172.21.15.119:0/1926950190 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307198792 0 0) 0x7f4a380705d0 con 0x7f4a380cd8d0 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a377fe700 1 -- 172.21.15.119:0/1926950190 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a240035d0 con 0x7f4a380cd8d0 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3d3b8700 1 -- 172.21.15.119:0/1926950190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b17f0 msgr2=0x7f4a380cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3d3b8700 1 --2- 172.21.15.119:0/1926950190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b17f0 0x7f4a380cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3d3b8700 1 -- 172.21.15.119:0/1926950190 >> v1:172.21.15.119:6789/0 conn(0x7f4a380cd8d0 legacy=0x7f4a3806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.177+0000 7f4a3d3b8700 1 -- 172.21.15.119:0/1926950190 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3806ff80 con 0x7f4a380b2200 2024-09-13T16:28:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3d3b8700 1 --2- 172.21.15.119:0/1926950190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b2200 0x7f4a380b26b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009850 tx=0x7f4a30009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b65657d0e4013f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a377fe700 1 -- 172.21.15.119:0/1926950190 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30011830 con 0x7f4a380b2200 2024-09-13T16:28:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a377fe700 1 -- 172.21.15.119:0/1926950190 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a30011990 con 0x7f4a380b2200 2024-09-13T16:28:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a377fe700 1 -- 172.21.15.119:0/1926950190 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a30011d40 con 0x7f4a380b2200 2024-09-13T16:28:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/1926950190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b2200 msgr2=0x7f4a380b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/1926950190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b2200 0x7f4a380b26b0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f4a30009850 tx=0x7f4a30009b60 comp rx=0 tx=0).stop 2024-09-13T16:28:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/1926950190 shutdown_connections 2024-09-13T16:28:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/1926950190 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b2200 0x7f4a380b26b0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/1926950190 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b17f0 0x7f4a380cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/1926950190 >> 172.21.15.119:0/1926950190 conn(0x7f4a3806a350 msgr2=0x7f4a3806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/1926950190 shutdown_connections 2024-09-13T16:28:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.181+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/1926950190 wait complete. 2024-09-13T16:28:34.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 Processor -- start 2024-09-13T16:28:34.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 -- start start 2024-09-13T16:28:34.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b17f0 0x7f4a380ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b2200 0x7f4a380a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a380cd8d0 0x7f4a380a9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a3815eb10 con 0x7f4a380b17f0 2024-09-13T16:28:34.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380aa040 con 0x7f4a380cd8d0 2024-09-13T16:28:34.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a380a9a80 con 0x7f4a380b2200 2024-09-13T16:28:34.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b17f0 0x7f4a380ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b17f0 0x7f4a380ca220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59222/0 (socket says 172.21.15.119:59222) 2024-09-13T16:28:34.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 -- 172.21.15.119:0/3463198084 learned_addr learned my addr 172.21.15.119:0/3463198084 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a37fff700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b2200 0x7f4a380a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3d3b8700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a380cd8d0 0x7f4a380a9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 -- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b2200 msgr2=0x7f4a380a8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b2200 0x7f4a380a8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 -- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a380cd8d0 msgr2=0x7f4a380a9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a380cd8d0 0x7f4a380a9400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 -- 172.21.15.119:0/3463198084 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a300094b0 con 0x7f4a380b17f0 2024-09-13T16:28:34.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a37fff700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b2200 0x7f4a380a8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:34.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3d3b8700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a380cd8d0 0x7f4a380a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:34.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3cbb7700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b17f0 0x7f4a380ca220 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c4d0 tx=0x7f4a2800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a357fa700 1 -- 172.21.15.119:0/3463198084 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28005d90 con 0x7f4a380b17f0 2024-09-13T16:28:34.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a357fa700 1 -- 172.21.15.119:0/3463198084 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a280070d0 con 0x7f4a380b17f0 2024-09-13T16:28:34.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a380a9d60 con 0x7f4a380b17f0 2024-09-13T16:28:34.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a357fa700 1 -- 172.21.15.119:0/3463198084 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a28007480 con 0x7f4a380b17f0 2024-09-13T16:28:34.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.185+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a3814eb00 con 0x7f4a380b17f0 2024-09-13T16:28:34.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.189+0000 7f4a357fa700 1 -- 172.21.15.119:0/3463198084 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4a280075e0 con 0x7f4a380b17f0 2024-09-13T16:28:34.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.189+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a380cdcb0 con 0x7f4a380b17f0 2024-09-13T16:28:34.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.189+0000 7f4a357fa700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a18066730 0x7f4a18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.189+0000 7f4a357fa700 1 -- 172.21.15.119:0/3463198084 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4a28083780 con 0x7f4a380b17f0 2024-09-13T16:28:34.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.189+0000 7f4a37fff700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a18066730 0x7f4a18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.189+0000 7f4a37fff700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a18066730 0x7f4a18068bf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4a380ae2c0 tx=0x7f4a2000b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.197+0000 7f4a357fa700 1 -- 172.21.15.119:0/3463198084 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4a28050a40 con 0x7f4a380b17f0 2024-09-13T16:28:34.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.409+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4a380b29b0 con 0x7f4a380b17f0 2024-09-13T16:28:34.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.409+0000 7f4a357fa700 1 -- 172.21.15.119:0/3463198084 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f4a28054060 con 0x7f4a380b17f0 2024-09-13T16:28:34.413 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:34.413 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:28:24.129975+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:28:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.409+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a18066730 msgr2=0x7f4a18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.409+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a18066730 0x7f4a18068bf0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f4a380ae2c0 tx=0x7f4a2000b040 comp rx=0 tx=0).stop 2024-09-13T16:28:34.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.409+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b17f0 msgr2=0x7f4a380ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b17f0 0x7f4a380ca220 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f4a2800c4d0 tx=0x7f4a2800c890 comp rx=0 tx=0).stop 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 shutdown_connections 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a18066730 0x7f4a18068bf0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a380cd8d0 0x7f4a380a9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a380b2200 0x7f4a380a8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 --2- 172.21.15.119:0/3463198084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a380b17f0 0x7f4a380ca220 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 >> 172.21.15.119:0/3463198084 conn(0x7f4a3806a350 msgr2=0x7f4a38162440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 shutdown_connections 2024-09-13T16:28:34.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.413+0000 7f4a3dbb9700 1 -- 172.21.15.119:0/3463198084 wait complete. 2024-09-13T16:28:34.417 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 32 2024-09-13T16:28:34.427 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.d.asok --format=json flush journal 2024-09-13T16:28:34.577 INFO:teuthology.orchestra.run.smithi145.stdout:{"message":"","return_code":0} 2024-09-13T16:28:34.578 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:28:34.578 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000001.00000000 parent 2024-09-13T16:28:34.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 Processor -- start 2024-09-13T16:28:34.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 -- start start 2024-09-13T16:28:34.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e77b34f0 0x5562e77e7250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b38d0 0x5562e77ff640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e7871330 0x5562e7889b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e77472f0 con 0x5562e77b34f0 2024-09-13T16:28:34.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e77038e0 con 0x5562e77b38d0 2024-09-13T16:28:34.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abd900140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562e7703580 con 0x5562e7871330 2024-09-13T16:28:34.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abb435700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e77b34f0 0x5562e77e7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abb435700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e77b34f0 0x5562e77e7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59234/0 (socket says 172.21.15.119:59234) 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.617+0000 7f4abb435700 1 -- 172.21.15.119:0/2028926455 learned_addr learned my addr 172.21.15.119:0/2028926455 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abbc36700 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e7871330 0x5562e7889b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abac34700 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b38d0 0x5562e77ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abb435700 1 -- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e7871330 msgr2=0x5562e7889b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abb435700 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e7871330 0x5562e7889b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abb435700 1 -- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b38d0 msgr2=0x5562e77ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abb435700 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b38d0 0x5562e77ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abb435700 1 -- 172.21.15.119:0/2028926455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac00a1c0 con 0x5562e77b34f0 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abac34700 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b38d0 0x5562e77ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:34.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abb435700 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e77b34f0 0x5562e77e7250 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b340 tx=0x7f4ab400b650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ced32ac2d7ef8a02 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4ab9c32700 1 -- 172.21.15.119:0/2028926455 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4014070 con 0x5562e77b34f0 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4ab9c32700 1 -- 172.21.15.119:0/2028926455 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ab400f3f0 con 0x5562e77b34f0 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4ab9c32700 1 -- 172.21.15.119:0/2028926455 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ab400f760 con 0x5562e77b34f0 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 -- 172.21.15.119:0/2028926455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e77b34f0 msgr2=0x5562e77e7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e77b34f0 0x5562e77e7250 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400b340 tx=0x7f4ab400b650 comp rx=0 tx=0).stop 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 -- 172.21.15.119:0/2028926455 shutdown_connections 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e7871330 0x5562e7889b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b38d0 0x5562e77ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 --2- 172.21.15.119:0/2028926455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e77b34f0 0x5562e77e7250 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 -- 172.21.15.119:0/2028926455 >> 172.21.15.119:0/2028926455 conn(0x5562e77b4b10 msgr2=0x5562e788ac50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 -- 172.21.15.119:0/2028926455 shutdown_connections 2024-09-13T16:28:34.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 -- 172.21.15.119:0/2028926455 wait complete. 2024-09-13T16:28:34.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 Processor -- start 2024-09-13T16:28:34.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.621+0000 7f4abd900140 1 -- start start 2024-09-13T16:28:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b34f0 0x5562e77f8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e77b38d0 0x5562e77fd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e7871330 0x5562e77e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e76d98d0 con 0x5562e7871330 2024-09-13T16:28:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5562e7862de0 con 0x5562e77b34f0 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5562e7863080 con 0x5562e77b38d0 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e7871330 0x5562e77e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e7871330 0x5562e77e6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59246/0 (socket says 172.21.15.119:59246) 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 -- 172.21.15.119:0/968653277 learned_addr learned my addr 172.21.15.119:0/968653277 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abb435700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b34f0 0x5562e77f8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abac34700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e77b38d0 0x5562e77fd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 -- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e77b38d0 msgr2=0x5562e77fd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e77b38d0 0x5562e77fd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 -- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b34f0 msgr2=0x5562e77f8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5562e77b34f0 0x5562e77f8180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 -- 172.21.15.119:0/968653277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5562e770bca0 con 0x5562e7871330 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abac34700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5562e77b38d0 0x5562e77fd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:34.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abbc36700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e7871330 0x5562e77e6d50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0108e0 tx=0x7f4aac010ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4ab8c30700 1 -- 172.21.15.119:0/968653277 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac010e80 con 0x5562e7871330 2024-09-13T16:28:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4ab8c30700 1 -- 172.21.15.119:0/968653277 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aac00e8c0 con 0x5562e7871330 2024-09-13T16:28:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab4001c10 con 0x5562e7871330 2024-09-13T16:28:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4ab8c30700 1 -- 172.21.15.119:0/968653277 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aac00ec70 con 0x5562e7871330 2024-09-13T16:28:34.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.625+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aa4005d20 con 0x5562e7871330 2024-09-13T16:28:34.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4ab8c30700 1 -- 172.21.15.119:0/968653277 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4aac003020 con 0x5562e7871330 2024-09-13T16:28:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4ab8c30700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4aa0065060 0x7f4aa0067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4ab8c30700 1 -- 172.21.15.119:0/968653277 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4aac0832b0 con 0x5562e7871330 2024-09-13T16:28:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4abb435700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4aa0065060 0x7f4aa0067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4abd900140 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5562e77fd720 0x5562e7864210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68) v8 -- 0x5562e7862860 con 0x5562e77fd720 2024-09-13T16:28:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4abac34700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5562e77fd720 0x5562e7864210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.629+0000 7f4abb435700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4aa0065060 0x7f4aa0067520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4000c00 tx=0x7f4ab4008000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abac34700 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5562e77fd720 0x5562e7864210 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abac34700 1 -- 172.21.15.119:0/968653277 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(1 10000000001.00000000 [getxattr (94) out=94b] v0'0 uv1 ondisk = 0) v8 ==== 164+0+94 (crc 0 0 0) 0x7f4aa4019040 con 0x5562e77fd720 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5562e77fd720 msgr2=0x5562e7864210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5562e77fd720 0x5562e7864210 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4aa0065060 msgr2=0x7f4aa0067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4aa0065060 0x7f4aa0067520 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4000c00 tx=0x7f4ab4008000 comp rx=0 tx=0).stop 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e7871330 msgr2=0x5562e77e6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 --2- 172.21.15.119:0/968653277 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5562e7871330 0x5562e77e6d50 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4aac0108e0 tx=0x7f4aac010ca0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abbc36700 1 -- 172.21.15.119:0/968653277 reap_dead start 2024-09-13T16:28:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 shutdown_connections 2024-09-13T16:28:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 >> 172.21.15.119:0/968653277 conn(0x5562e77b4b10 msgr2=0x5562e788ad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 shutdown_connections 2024-09-13T16:28:34.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.633+0000 7f4abd900140 1 -- 172.21.15.119:0/968653277 wait complete. 2024-09-13T16:28:34.639 INFO:teuthology.orchestra.run.smithi119.stdout:Xfiledir_test 2024-09-13T16:28:34.642 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:34.642 DEBUG:teuthology.orchestra.run.smithi119:> ceph-dencoder type inode_backtrace_t import - decode dump_json 2024-09-13T16:28:34.743 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627777, 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: "ancestors": [ 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: "dirino": 1099511627776, 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: "dname": "file", 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: "version": 2 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:28:34.744 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: "dirino": 1, 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: "dname": "dir_test", 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: "version": 4 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: "pool": 10, 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: "old_pools": [] 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:34.745 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:28:34.747 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data setxattr 10000000001.00000000 parent 'The backtrace is corrupted' 2024-09-13T16:28:34.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.785+0000 7fa262157140 1 Processor -- start 2024-09-13T16:28:34.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 -- start start 2024-09-13T16:28:34.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e92c530 0x55b82e9ecce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea95820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e9e9c60 0x55b82ea95d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e8c02f0 con 0x55b82e92c530 2024-09-13T16:28:34.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e87c8e0 con 0x55b82e92c910 2024-09-13T16:28:34.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b82e87c580 con 0x55b82e9e9c60 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e92c530 0x55b82e9ecce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e92c530 0x55b82e9ecce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59274/0 (socket says 172.21.15.119:59274) 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 -- 172.21.15.119:0/3580679252 learned_addr learned my addr 172.21.15.119:0/3580679252 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa26048d700 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e9e9c60 0x55b82ea95d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 -- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e9e9c60 msgr2=0x55b82ea95d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25f48b700 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea95820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e9e9c60 0x55b82ea95d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 -- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 msgr2=0x55b82ea95820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea95820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 -- 172.21.15.119:0/3580679252 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa248005d20 con 0x55b82e92c530 2024-09-13T16:28:34.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa26048d700 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e9e9c60 0x55b82ea95d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25f48b700 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea95820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25fc8c700 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e92c530 0x55b82e9ecce0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa25800a810 tx=0x7fa258005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb64593264aa63b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25e489700 1 -- 172.21.15.119:0/3580679252 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258012660 con 0x55b82e92c530 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25e489700 1 -- 172.21.15.119:0/3580679252 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2580127c0 con 0x55b82e92c530 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa25e489700 1 -- 172.21.15.119:0/3580679252 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa258012b70 con 0x55b82e92c530 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 -- 172.21.15.119:0/3580679252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e92c530 msgr2=0x55b82e9ecce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e92c530 0x55b82e9ecce0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fa25800a810 tx=0x7fa258005d80 comp rx=0 tx=0).stop 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 -- 172.21.15.119:0/3580679252 shutdown_connections 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e9e9c60 0x55b82ea95d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea95820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 --2- 172.21.15.119:0/3580679252 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e92c530 0x55b82e9ecce0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.789+0000 7fa262157140 1 -- 172.21.15.119:0/3580679252 >> 172.21.15.119:0/3580679252 conn(0x55b82e92db50 msgr2=0x55b82ea03c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 -- 172.21.15.119:0/3580679252 shutdown_connections 2024-09-13T16:28:34.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 -- 172.21.15.119:0/3580679252 wait complete. 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 Processor -- start 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 -- start start 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e92c530 0x55b82e9d8f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea9d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e9e9c60 0x55b82e9e6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e8528d0 con 0x55b82e9e9c60 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b82e9d95f0 con 0x55b82e92c910 2024-09-13T16:28:34.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa262157140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b82e9d9890 con 0x55b82e92c530 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa26048d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e9e9c60 0x55b82e9e6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa25f48b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea9d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa25fc8c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e92c530 0x55b82e9d8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa26048d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e9e9c60 0x55b82e9e6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59286/0 (socket says 172.21.15.119:59286) 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.793+0000 7fa26048d700 1 -- 172.21.15.119:0/1230233318 learned_addr learned my addr 172.21.15.119:0/1230233318 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa26048d700 1 -- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e92c530 msgr2=0x55b82e9d8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa26048d700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e92c530 0x55b82e9d8f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa26048d700 1 -- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 msgr2=0x55b82ea9d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa26048d700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea9d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa26048d700 1 -- 172.21.15.119:0/1230233318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa25800a410 con 0x55b82e9e9c60 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa25fc8c700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55b82e92c530 0x55b82e9d8f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa25f48b700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55b82e92c910 0x55b82ea9d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:34.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa26048d700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e9e9c60 0x55b82e9e6850 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa25000cb80 tx=0x7fa25000ce90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa25d487700 1 -- 172.21.15.119:0/1230233318 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000ace0 con 0x55b82e9e9c60 2024-09-13T16:28:34.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa25d487700 1 -- 172.21.15.119:0/1230233318 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa25000ae40 con 0x55b82e9e9c60 2024-09-13T16:28:34.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55b82e86ab00 con 0x55b82e9e9c60 2024-09-13T16:28:34.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa25d487700 1 -- 172.21.15.119:0/1230233318 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa25000e660 con 0x55b82e9e9c60 2024-09-13T16:28:34.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b82e85df40 con 0x55b82e9e9c60 2024-09-13T16:28:34.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.797+0000 7fa25d487700 1 -- 172.21.15.119:0/1230233318 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa25000e7c0 con 0x55b82e9e9c60 2024-09-13T16:28:34.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa25d487700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa240065060 0x7fa240067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa25d487700 1 -- 172.21.15.119:0/1230233318 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa250082f40 con 0x55b82e9e9c60 2024-09-13T16:28:34.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa25fc8c700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa240065060 0x7fa240067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa262157140 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55b82e9d83a0 0x55b82e978db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:1 10.7 10:e2e07930:::10000000001.00000000:head [setxattr parent (26) in=32b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68) v8 -- 0x55b82e976570 con 0x55b82e9d83a0 2024-09-13T16:28:34.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa25fc8c700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa240065060 0x7fa240067520 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa258016040 tx=0x7fa25801e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa25f48b700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55b82e9d83a0 0x55b82e978db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.801+0000 7fa25f48b700 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55b82e9d83a0 0x55b82e978db0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa25f48b700 1 -- 172.21.15.119:0/1230233318 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(1 10000000001.00000000 [setxattr (26)] v68'2 uv2 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fa248008a80 con 0x55b82e9d83a0 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55b82e9d83a0 msgr2=0x55b82e978db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55b82e9d83a0 0x55b82e978db0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa240065060 msgr2=0x7fa240067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa240065060 0x7fa240067520 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa258016040 tx=0x7fa25801e040 comp rx=0 tx=0).stop 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e9e9c60 msgr2=0x55b82e9e6850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 --2- 172.21.15.119:0/1230233318 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55b82e9e9c60 0x55b82e9e6850 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fa25000cb80 tx=0x7fa25000ce90 comp rx=0 tx=0).stop 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa26048d700 1 -- 172.21.15.119:0/1230233318 reap_dead start 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 shutdown_connections 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 >> 172.21.15.119:0/1230233318 conn(0x55b82e92db50 msgr2=0x55b82e92df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 shutdown_connections 2024-09-13T16:28:34.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.805+0000 7fa262157140 1 -- 172.21.15.119:0/1230233318 wait complete. 2024-09-13T16:28:34.814 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start / recursive 2024-09-13T16:28:34.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50eca6700 1 Processor -- start 2024-09-13T16:28:34.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50eca6700 1 -- start start 2024-09-13T16:28:34.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50eca6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d1380 0x7fa5080d3260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50eca6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa5080d0720 0x7fa5080ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50eca6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5080cd640 con 0x7fa5080d1760 2024-09-13T16:28:34.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50eca6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5080d0f00 con 0x7fa5080d0720 2024-09-13T16:28:34.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50eca6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080cd870 con 0x7fa5080d1380 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50e4a5700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa5080d1760 0x7fa50806ab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33262/0 (socket says 172.21.15.119:33262) 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50e4a5700 1 -- 172.21.15.119:0/863272936 learned_addr learned my addr 172.21.15.119:0/863272936 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50dca4700 1 --2- 172.21.15.119:0/863272936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d1380 0x7fa5080d3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/863272936 >> v2:172.21.15.145:3300/0 conn(0x7fa5080d0720 0x7fa5080ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50cca2700 1 -- 172.21.15.119:0/863272936 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112085338 0 0) 0x7fa5080cd640 con 0x7fa5080d1760 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50cca2700 1 -- 172.21.15.119:0/863272936 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4f40035d0 con 0x7fa5080d1760 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50dca4700 1 -- 172.21.15.119:0/863272936 >> v2:172.21.15.145:3300/0 conn(0x7fa5080d0720 msgr2=0x7fa5080ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50dca4700 1 --2- 172.21.15.119:0/863272936 >> v2:172.21.15.145:3300/0 conn(0x7fa5080d0720 0x7fa5080ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50dca4700 1 -- 172.21.15.119:0/863272936 >> v1:172.21.15.119:6789/0 conn(0x7fa5080d1760 legacy=0x7fa50806ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50dca4700 1 -- 172.21.15.119:0/863272936 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080d37a0 con 0x7fa5080d1380 2024-09-13T16:28:34.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50dca4700 1 --2- 172.21.15.119:0/863272936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d1380 0x7fa5080d3260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009880 tx=0x7fa4fc009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dc404cc191e0e87e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50cca2700 1 -- 172.21.15.119:0/863272936 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc011830 con 0x7fa5080d1380 2024-09-13T16:28:34.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50cca2700 1 -- 172.21.15.119:0/863272936 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4fc011990 con 0x7fa5080d1380 2024-09-13T16:28:34.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.933+0000 7fa50cca2700 1 -- 172.21.15.119:0/863272936 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4fc011d20 con 0x7fa5080d1380 2024-09-13T16:28:34.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 -- 172.21.15.119:0/863272936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d1380 msgr2=0x7fa5080d3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 --2- 172.21.15.119:0/863272936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d1380 0x7fa5080d3260 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc009880 tx=0x7fa4fc009b90 comp rx=0 tx=0).stop 2024-09-13T16:28:34.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 -- 172.21.15.119:0/863272936 shutdown_connections 2024-09-13T16:28:34.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 --2- 172.21.15.119:0/863272936 >> v2:172.21.15.145:3300/0 conn(0x7fa5080d0720 0x7fa5080ce310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 --2- 172.21.15.119:0/863272936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d1380 0x7fa5080d3260 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 -- 172.21.15.119:0/863272936 >> 172.21.15.119:0/863272936 conn(0x7fa50809be40 msgr2=0x7fa5080d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 -- 172.21.15.119:0/863272936 shutdown_connections 2024-09-13T16:28:34.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 -- 172.21.15.119:0/863272936 wait complete. 2024-09-13T16:28:34.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.937+0000 7fa50eca6700 1 Processor -- start 2024-09-13T16:28:34.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 -- start start 2024-09-13T16:28:34.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d0720 0x7fa5081601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5080d1380 0x7fa50815e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5080d1760 0x7fa50809e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa508160730 con 0x7fa5080d1380 2024-09-13T16:28:34.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080cafd0 con 0x7fa5080d1760 2024-09-13T16:28:34.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa50809cc70 con 0x7fa5080d0720 2024-09-13T16:28:34.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5080d1380 0x7fa50815e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5080d1380 0x7fa50815e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59292/0 (socket says 172.21.15.119:59292) 2024-09-13T16:28:34.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 -- 172.21.15.119:0/842842413 learned_addr learned my addr 172.21.15.119:0/842842413 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5080d1760 0x7fa50809e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50dca4700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d0720 0x7fa5081601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 -- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d0720 msgr2=0x7fa5081601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d0720 0x7fa5081601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 -- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5080d1760 msgr2=0x7fa50809e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5080d1760 0x7fa50809e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 -- 172.21.15.119:0/842842413 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc0094b0 con 0x7fa5080d1380 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5080d1380 0x7fa50815e0a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa50400efc0 tx=0x7fa504009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa4fa7fc700 1 -- 172.21.15.119:0/842842413 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50400cae0 con 0x7fa5080d1380 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa4fa7fc700 1 -- 172.21.15.119:0/842842413 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50400f040 con 0x7fa5080d1380 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5080ca6a0 con 0x7fa5080d1380 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa4fa7fc700 1 -- 172.21.15.119:0/842842413 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa50400ce60 con 0x7fa5080d1380 2024-09-13T16:28:34.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.941+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5080cac60 con 0x7fa5080d1380 2024-09-13T16:28:34.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa4fa7fc700 1 -- 172.21.15.119:0/842842413 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa504014480 con 0x7fa5080d1380 2024-09-13T16:28:34.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa4fa7fc700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4ec066730 0x7fa4ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa4fa7fc700 1 -- 172.21.15.119:0/842842413 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa504084de0 con 0x7fa5080d1380 2024-09-13T16:28:34.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa50dca4700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4ec066730 0x7fa4ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 Processor -- start 2024-09-13T16:28:34.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 -- start start 2024-09-13T16:28:34.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa50dca4700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4ec066730 0x7fa4ec068bf0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0090b0 tx=0x7fa4fc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f0f330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9fb0 0x1f11880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1f141b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e35eb0 con 0x1f11dc0 2024-09-13T16:28:34.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e33640 con 0x1ec9fb0 2024-09-13T16:28:34.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e2ef30 con 0x1ec9bd0 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa50e4a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1f141b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.945+0000 7fa50e4a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1f141b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59300/0 (socket says 172.21.15.119:59300) 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50e4a5700 1 -- 172.21.15.119:0/568941630 learned_addr learned my addr 172.21.15.119:0/568941630 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50dca4700 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f0f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9fb0 0x1f11880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50e4a5700 1 -- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 msgr2=0x1f0f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f0f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50e4a5700 1 -- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9fb0 msgr2=0x1f11880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9fb0 0x1f11880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50e4a5700 1 -- 172.21.15.119:0/568941630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1e55950 con 0x1f11dc0 2024-09-13T16:28:34.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1f141b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa500006e00 tx=0x7fa500007320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78da00ea7d7e077e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa4eb7fe700 1 -- 172.21.15.119:0/568941630 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500006e30 con 0x1f11dc0 2024-09-13T16:28:34.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa4eb7fe700 1 -- 172.21.15.119:0/568941630 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa500001520 con 0x1f11dc0 2024-09-13T16:28:34.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa4eb7fe700 1 -- 172.21.15.119:0/568941630 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500001890 con 0x1f11dc0 2024-09-13T16:28:34.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 -- 172.21.15.119:0/568941630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 msgr2=0x1f141b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1f141b0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fa500006e00 tx=0x7fa500007320 comp rx=0 tx=0).stop 2024-09-13T16:28:34.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 -- 172.21.15.119:0/568941630 shutdown_connections 2024-09-13T16:28:34.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1f141b0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9fb0 0x1f11880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 --2- 172.21.15.119:0/568941630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f0f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 -- 172.21.15.119:0/568941630 >> 172.21.15.119:0/568941630 conn(0x1c56700 msgr2=0x1ef0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 -- 172.21.15.119:0/568941630 shutdown_connections 2024-09-13T16:28:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 -- 172.21.15.119:0/568941630 wait complete. 2024-09-13T16:28:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 -- start start 2024-09-13T16:28:34.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.949+0000 7fa5131b9740 1 Processor -- start 2024-09-13T16:28:34.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1eed340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eef160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eef6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1df2740 con 0x1f11dc0 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1df23e0 con 0x1ec9bd0 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1cdacd0 con 0x1ec9fb0 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 10 client.0 _ll_register_callbacks cb 0x1e80ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eef6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eef6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59316/0 (socket says 172.21.15.119:59316) 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2437623164 learned_addr learned my addr 172.21.15.119:0/2437623164 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1eed340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eef160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 msgr2=0x1eef160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eef160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 msgr2=0x1eed340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1eed340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2437623164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f40035d0 con 0x1f11dc0 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eef6a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa500001200 tx=0x7fa500000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b1eb4218d0e9a3f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa4e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2437623164 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500001aa0 con 0x1f11dc0 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2437623164 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa500001c00 con 0x1f11dc0 2024-09-13T16:28:34.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1c63410 con 0x1f11dc0 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2437623164 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa500006040 con 0x1f11dc0 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 10 client.6762 fetch_fsmap learned FSMap version 32 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x1ca1b90 con 0x1f11dc0 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2437623164 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1c63410 con 0x1f11dc0 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2437623164 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 32) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa500012070 con 0x1f11dc0 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 10 client.6762 fetch_fsmap finished waiting for FSMap version 32 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 10 client.6762 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 20 client.6762 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f21b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 4 client.6762 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:28:34.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1cad690 con 0x1f1f710 2024-09-13T16:28:34.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.953+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f21b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f21b00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb59d2737eabb36f server_cookie=1c4e3b4b36f52503 in_seq=0 out_seq=0 2024-09-13T16:28:34.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa4e8ff9700 10 client.6762 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:34.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2437623164 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1cad690 con 0x1f1f710 2024-09-13T16:28:34.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa4e8ff9700 10 client.6762 handle_command_reply: tid=0 2024-09-13T16:28:34.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 client.6762 shutdown 2024-09-13T16:28:34.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 msgr2=0x1eef6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eef6a0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fa500001200 tx=0x7fa500000ed0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 shutdown_connections 2024-09-13T16:28:34.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f21b00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eef6a0 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa4e8ff9700 0 client.6762 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eef160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2437623164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1eed340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 >> 172.21.15.119:0/2437623164 conn(0x1c56700 msgr2=0x1f111e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 shutdown_connections 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 1 -- 172.21.15.119:0/2437623164 wait complete. 2024-09-13T16:28:34.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.957+0000 7fa5131b9740 20 client.6762 trim_cache size 0 max 16384 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 Processor -- start 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 -- start start 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1f23f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eecfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eed500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1df0030 con 0x1f11dc0 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ec94f0 con 0x1ec9bd0 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e2a720 con 0x1ec9fb0 2024-09-13T16:28:34.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eed500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eed500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59332/0 (socket says 172.21.15.119:59332) 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2591627187 learned_addr learned my addr 172.21.15.119:0/2591627187 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eecfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1f23f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 msgr2=0x1eecfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eecfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 msgr2=0x1f23f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1f23f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 -- 172.21.15.119:0/2591627187 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1df5350 con 0x1f11dc0 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1f23f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eecfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:34.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eed500 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa500006490 tx=0x7fa500020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442b8f507eeeeb4b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa4ebfff700 1 -- 172.21.15.119:0/2591627187 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa500000e40 con 0x1f11dc0 2024-09-13T16:28:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa4ebfff700 1 -- 172.21.15.119:0/2591627187 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa500022070 con 0x1f11dc0 2024-09-13T16:28:34.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.977+0000 7fa4ebfff700 1 -- 172.21.15.119:0/2591627187 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5000018a0 con 0x1f11dc0 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 -- 172.21.15.119:0/2591627187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 msgr2=0x1eed500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eed500 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fa500006490 tx=0x7fa500020070 comp rx=0 tx=0).stop 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 -- 172.21.15.119:0/2591627187 shutdown_connections 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f11dc0 0x1eed500 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9fb0 0x1eecfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2591627187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ec9bd0 0x1f23f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 -- 172.21.15.119:0/2591627187 >> 172.21.15.119:0/2591627187 conn(0x1c56700 msgr2=0x1eec8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 -- 172.21.15.119:0/2591627187 shutdown_connections 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 -- 172.21.15.119:0/2591627187 wait complete. 2024-09-13T16:28:34.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 -- start start 2024-09-13T16:28:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.981+0000 7fa5131b9740 1 Processor -- start 2024-09-13T16:28:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f2b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ec9fb0 0x1f71ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f11dc0 0x1f72020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dd6fb0 con 0x1ec9fb0 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dfd480 con 0x1f11dc0 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f23e10 con 0x1ec9bd0 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 10 client.0 _ll_register_callbacks cb 0x1de0e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ec9fb0 0x1f71ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ec9fb0 0x1f71ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59346/0 (socket says 172.21.15.119:59346) 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 -- 172.21.15.119:0/2809902470 learned_addr learned my addr 172.21.15.119:0/2809902470 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f2b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f11dc0 0x1f72020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 -- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 msgr2=0x1f2b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f2b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 -- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f11dc0 msgr2=0x1f72020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:34.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f11dc0 0x1f72020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 -- 172.21.15.119:0/2809902470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1ca1b90 con 0x1ec9fb0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50e4a5700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f11dc0 0x1f72020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50d4a3700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ec9fb0 0x1f71ae0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa5040558b0 tx=0x7fa504052020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e967bc1a0d416f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa4e8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2809902470 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa504056010 con 0x1ec9fb0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2809902470 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fa5040550a0 con 0x1ec9fb0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa500001520 con 0x1ec9fb0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2809902470 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa504055260 con 0x1ec9fb0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 10 client.6768 fetch_fsmap learned FSMap version 32 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x1ddb580 con 0x1ec9fb0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2809902470 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa500001520 con 0x1ec9fb0 2024-09-13T16:28:34.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2809902470 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 32) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa50405db10 con 0x1ec9fb0 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 10 client.6768 fetch_fsmap finished waiting for FSMap version 32 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 10 client.6768 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 20 client.6768 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f22b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 4 client.6768 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}] 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["recursive"]}) v1 -- 0x1cd2410 con 0x1f1f710 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f22b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa4e8ff9700 10 client.6768 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:34.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.985+0000 7fa50dca4700 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f22b50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ee0caa6f4ae6a99 server_cookie=24bdfe9ed8375e4d in_seq=0 out_seq=0 2024-09-13T16:28:34.993 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-09-13T16:28:34.988+0000 7f9e45c91700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6723=0-4194304@1} caps={6723=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6723 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55f65c319600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa4e8ff9700 1 -- 172.21.15.119:0/2809902470 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x1cd2410 con 0x1f1f710 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa4e8ff9700 10 client.6768 handle_command_reply: tid=0 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 client.6768 shutdown 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ec9fb0 msgr2=0x1f71ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ec9fb0 0x1f71ae0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fa5040558b0 tx=0x7fa504052020 comp rx=0 tx=0).stop 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 shutdown_connections 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1f1f710 0x1f22b50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f11dc0 0x1f72020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ec9fb0 0x1f71ae0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 --2- 172.21.15.119:0/2809902470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1ec9bd0 0x1f2b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 >> 172.21.15.119:0/2809902470 conn(0x1c56700 msgr2=0x1eed0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa4e8ff9700 0 client.6768 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 shutdown_connections 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 1 -- 172.21.15.119:0/2809902470 wait complete. 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa5131b9740 20 client.6768 trim_cache size 0 max 16384 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "0e647766-429e-4be5-8322-bdf487fce443", 2024-09-13T16:28:34.995 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:28:34.996 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:34.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4ec066730 msgr2=0x7fa4ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4ec066730 0x7fa4ec068bf0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa4fc0090b0 tx=0x7fa4fc008030 comp rx=0 tx=0).stop 2024-09-13T16:28:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5080d1380 msgr2=0x7fa50815e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5080d1380 0x7fa50815e0a0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fa50400efc0 tx=0x7fa504009f40 comp rx=0 tx=0).stop 2024-09-13T16:28:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 shutdown_connections 2024-09-13T16:28:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa4ec066730 0x7fa4ec068bf0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa5080d1760 0x7fa50809e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa5080d1380 0x7fa50815e0a0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 --2- 172.21.15.119:0/842842413 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa5080d0720 0x7fa5081601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:34.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 >> 172.21.15.119:0/842842413 conn(0x7fa50809be40 msgr2=0x7fa50806ab20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:34.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 shutdown_connections 2024-09-13T16:28:34.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:34.989+0000 7fa50eca6700 1 -- 172.21.15.119:0/842842413 wait complete. 2024-09-13T16:28:35.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2024-09-13T16:28:35.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.121+0000 7f5d33fff700 1 Processor -- start 2024-09-13T16:28:35.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.121+0000 7f5d33fff700 1 -- start start 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d33fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3415d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d33fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5d34159910 0x7f5d34159da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d33fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d3415db20 con 0x7f5d34159530 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d33fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d340ce700 con 0x7f5d34159910 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d33fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d340ce9e0 con 0x7f5d3415d1c0 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d32ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5d34159530 0x7f5d340cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33276/0 (socket says 172.21.15.119:33276) 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d32ffd700 1 -- 172.21.15.119:0/998893578 learned_addr learned my addr 172.21.15.119:0/998893578 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:28:35.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d337fe700 1 --2- 172.21.15.119:0/998893578 >> v2:172.21.15.145:3300/0 conn(0x7f5d34159910 0x7f5d34159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3415d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d327fc700 1 -- 172.21.15.119:0/998893578 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692178939 0 0) 0x7f5d3415db20 con 0x7f5d34159530 2024-09-13T16:28:35.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d327fc700 1 -- 172.21.15.119:0/998893578 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d200035d0 con 0x7f5d34159530 2024-09-13T16:28:35.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d337fe700 1 -- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 msgr2=0x7f5d3415d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d337fe700 1 --2- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3415d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d327fc700 1 -- 172.21.15.119:0/998893578 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4147234953 0 0) 0x7f5d200035d0 con 0x7f5d34159530 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d337fe700 1 -- 172.21.15.119:0/998893578 >> v1:172.21.15.119:6789/0 conn(0x7f5d34159530 legacy=0x7f5d340cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d337fe700 1 -- 172.21.15.119:0/998893578 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d340cecc0 con 0x7f5d34159910 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.125+0000 7f5d337fe700 1 --2- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159910 0x7f5d34159da0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009d90 tx=0x7f5d2c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616c8b09b487b96c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d327fc700 1 -- 172.21.15.119:0/998893578 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2c00e070 con 0x7f5d34159910 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d327fc700 1 -- 172.21.15.119:0/998893578 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d2c003a40 con 0x7f5d34159910 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d327fc700 1 -- 172.21.15.119:0/998893578 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24001670 con 0x7f5d34159910 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 -- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159910 msgr2=0x7f5d34159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 --2- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159910 0x7f5d34159da0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5d2c009d90 tx=0x7f5d2c00b330 comp rx=0 tx=0).stop 2024-09-13T16:28:35.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 -- 172.21.15.119:0/998893578 shutdown_connections 2024-09-13T16:28:35.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 --2- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159910 0x7f5d34159da0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 --2- 172.21.15.119:0/998893578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3415d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 -- 172.21.15.119:0/998893578 >> 172.21.15.119:0/998893578 conn(0x7f5d3406b460 msgr2=0x7f5d3406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:35.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 -- 172.21.15.119:0/998893578 shutdown_connections 2024-09-13T16:28:35.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 -- 172.21.15.119:0/998893578 wait complete. 2024-09-13T16:28:35.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 Processor -- start 2024-09-13T16:28:35.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.129+0000 7f5d33fff700 1 -- start start 2024-09-13T16:28:35.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159530 0x7f5d341425b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d34159910 0x7f5d3406ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3414c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3414c6f0 con 0x7f5d34159910 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3414d4f0 con 0x7f5d34159530 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d3414c8e0 con 0x7f5d3415d1c0 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d34159910 0x7f5d3406ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d34159910 0x7f5d3406ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59354/0 (socket says 172.21.15.119:59354) 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/1486369649 learned_addr learned my addr 172.21.15.119:0/1486369649 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159530 0x7f5d341425b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d337fe700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3414c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 msgr2=0x7f5d3414c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3414c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159530 msgr2=0x7f5d341425b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159530 0x7f5d341425b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/1486369649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d2c009a70 con 0x7f5d34159910 2024-09-13T16:28:35.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d34159910 0x7f5d3406ea70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00e840 tx=0x7f5d1c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:35.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2bfff700 1 -- 172.21.15.119:0/1486369649 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c0095a0 con 0x7f5d34159910 2024-09-13T16:28:35.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2bfff700 1 -- 172.21.15.119:0/1486369649 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c013680 con 0x7f5d34159910 2024-09-13T16:28:35.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d3414cbc0 con 0x7f5d34159910 2024-09-13T16:28:35.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d2bfff700 1 -- 172.21.15.119:0/1486369649 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c0139f0 con 0x7f5d34159910 2024-09-13T16:28:35.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.133+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3414d180 con 0x7f5d34159910 2024-09-13T16:28:35.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.137+0000 7f5d2bfff700 1 -- 172.21.15.119:0/1486369649 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5d1c013b50 con 0x7f5d34159910 2024-09-13T16:28:35.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.137+0000 7f5d2bfff700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d14066730 0x7f5d14068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.137+0000 7f5d2bfff700 1 -- 172.21.15.119:0/1486369649 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5d1c0840e0 con 0x7f5d34159910 2024-09-13T16:28:35.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.137+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d14066730 0x7f5d14068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.137+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d14066730 0x7f5d14068bf0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5d24001eb0 tx=0x7f5d24001e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:35.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.137+0000 7f5d3c80f740 1 Processor -- start 2024-09-13T16:28:35.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.137+0000 7f5d3c80f740 1 -- start start 2024-09-13T16:28:35.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19bcd90 0x1973810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x199b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bd550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1904010 con 0x1972310 2024-09-13T16:28:35.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1891f40 con 0x19bcd90 2024-09-13T16:28:35.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x18e4e30 con 0x19bd170 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x199b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x199b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59368/0 (socket says 172.21.15.119:59368) 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/2216761219 learned_addr learned my addr 172.21.15.119:0/2216761219 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19bcd90 0x1973810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d337fe700 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 msgr2=0x19bd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19bcd90 msgr2=0x1973810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19bcd90 0x1973810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/2216761219 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1750a40 con 0x1972310 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x199b520 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0514d0 tx=0x7f5d1c055050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82134bd979668a57 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d28ff9700 1 -- 172.21.15.119:0/2216761219 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c05b070 con 0x1972310 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d28ff9700 1 -- 172.21.15.119:0/2216761219 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c053110 con 0x1972310 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d28ff9700 1 -- 172.21.15.119:0/2216761219 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c055230 con 0x1972310 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- 172.21.15.119:0/2216761219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 msgr2=0x199b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x199b520 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0514d0 tx=0x7f5d1c055050 comp rx=0 tx=0).stop 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- 172.21.15.119:0/2216761219 shutdown_connections 2024-09-13T16:28:35.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bd550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x199b520 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/2216761219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19bcd90 0x1973810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- 172.21.15.119:0/2216761219 >> 172.21.15.119:0/2216761219 conn(0x17055b0 msgr2=0x1976b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:35.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- 172.21.15.119:0/2216761219 shutdown_connections 2024-09-13T16:28:35.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.141+0000 7f5d3c80f740 1 -- 172.21.15.119:0/2216761219 wait complete. 2024-09-13T16:28:35.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 -- start start 2024-09-13T16:28:35.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 Processor -- start 2024-09-13T16:28:35.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x1996970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19bcd90 0x1999d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x18e0150 con 0x19bcd90 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x18ddeb0 con 0x1972310 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x189b8e0 con 0x19bd170 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d3c80f740 10 client.0 _ll_register_callbacks cb 0x187a670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19bcd90 0x1999d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19bcd90 0x1999d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59372/0 (socket says 172.21.15.119:59372) 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/3341262058 learned_addr learned my addr 172.21.15.119:0/3341262058 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d337fe700 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x1996970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 msgr2=0x19bb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 msgr2=0x1996970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x1996970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/3341262058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d3414d180 con 0x19bcd90 2024-09-13T16:28:35.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19bcd90 0x1999d40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c000c00 tx=0x7f5d1c00efd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d56a679523db1f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:35.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.145+0000 7f5d11ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:28:35.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 1 -- 172.21.15.119:0/3341262058 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d1c051310 con 0x19bcd90 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 1 -- 172.21.15.119:0/3341262058 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5d1c082130 con 0x19bcd90 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x17122c0 con 0x19bcd90 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 1 -- 172.21.15.119:0/3341262058 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d1c055230 con 0x19bcd90 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 10 client.6780 fetch_fsmap learned FSMap version 32 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x18a4290 con 0x19bcd90 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 1 -- 172.21.15.119:0/3341262058 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x17122c0 con 0x19bcd90 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 1 -- 172.21.15.119:0/3341262058 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 32) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5d1c059d20 con 0x19bcd90 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 10 client.6780 fetch_fsmap finished waiting for FSMap version 32 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 10 client.6780 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:28:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 20 client.6780 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1977a60 0x1994f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 4 client.6780 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x175c540 con 0x1977a60 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1977a60 0x1994f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 10 client.6780 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1977a60 0x1994f50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acc867b37f0192cf server_cookie=e79a062543ae7565 in_seq=0 out_seq=0 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 1 -- 172.21.15.119:0/3341262058 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x175c540 con 0x1977a60 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 10 client.6780 handle_command_reply: tid=0 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 client.6780 shutdown 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19bcd90 msgr2=0x1999d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19bcd90 0x1999d40 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c000c00 tx=0x7f5d1c00efd0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 shutdown_connections 2024-09-13T16:28:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1977a60 0x1994f50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19bd170 0x19bb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d11ffb700 0 client.6780 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19bcd90 0x1999d40 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3341262058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x1996970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 >> 172.21.15.119:0/3341262058 conn(0x17055b0 msgr2=0x199b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 shutdown_connections 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3341262058 wait complete. 2024-09-13T16:28:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.149+0000 7f5d3c80f740 20 client.6780 trim_cache size 0 max 16384 2024-09-13T16:28:35.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 Processor -- start 2024-09-13T16:28:35.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 -- start start 2024-09-13T16:28:35.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x17055b0 0x19965d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x19ba7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19bad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1789b80 con 0x1972310 2024-09-13T16:28:35.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x18e2f30 con 0x17055b0 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x189e810 con 0x1977a60 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x19ba7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d32ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x17055b0 0x19965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d2a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x19ba7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59382/0 (socket says 172.21.15.119:59382) 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d2a7fc700 1 -- 172.21.15.119:0/3247977574 learned_addr learned my addr 172.21.15.119:0/3247977574 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d337fe700 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19bad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d32ffd700 1 -- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 msgr2=0x19bad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19bad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d32ffd700 1 -- 172.21.15.119:0/3247977574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 msgr2=0x19ba7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x19ba7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d32ffd700 1 -- 172.21.15.119:0/3247977574 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x19778d0 con 0x17055b0 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x19ba7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x17055b0 0x19965d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x199a3d0 tx=0x7f5d24012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bedad44b25fc9d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.173+0000 7f5d297fa700 1 -- 172.21.15.119:0/3247977574 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2402a030 con 0x17055b0 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d297fa700 1 -- 172.21.15.119:0/3247977574 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d240260b0 con 0x17055b0 2024-09-13T16:28:35.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d297fa700 1 -- 172.21.15.119:0/3247977574 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d24026440 con 0x17055b0 2024-09-13T16:28:35.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x17055b0 msgr2=0x19965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x17055b0 0x19965d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x199a3d0 tx=0x7f5d24012000 comp rx=0 tx=0).stop 2024-09-13T16:28:35.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3247977574 shutdown_connections 2024-09-13T16:28:35.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19bad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1972310 0x19ba7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/3247977574 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x17055b0 0x19965d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3247977574 >> 172.21.15.119:0/3247977574 conn(0x1976f10 msgr2=0x1999780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3247977574 shutdown_connections 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- 172.21.15.119:0/3247977574 wait complete. 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- start start 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 Processor -- start 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x17055b0 0x1a1d550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x19d5510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19d5a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x19dbe90 con 0x17055b0 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19d4960 con 0x1972310 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x19995e0 con 0x1977a60 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d3c80f740 10 client.0 _ll_register_callbacks cb 0x18dc610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x17055b0 0x1a1d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x17055b0 0x1a1d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:59384/0 (socket says 172.21.15.119:59384) 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 -- 172.21.15.119:0/4188521994 learned_addr learned my addr 172.21.15.119:0/4188521994 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d337fe700 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19d5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x19d5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 -- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 msgr2=0x19d5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19d5a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 -- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 msgr2=0x19d5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x19d5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 -- 172.21.15.119:0/4188521994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x18a4290 con 0x17055b0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d11ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d32ffd700 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x17055b0 0x1a1d550 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5d24025040 tx=0x7f5d1c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=359afd0ece4c8d9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.177+0000 7f5d11ffb700 1 -- 172.21.15.119:0/4188521994 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d2402a070 con 0x17055b0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 1 -- 172.21.15.119:0/4188521994 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5d2402b070 con 0x17055b0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5d1c053110 con 0x17055b0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 1 -- 172.21.15.119:0/4188521994 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5d24001210 con 0x17055b0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 10 client.6786 fetch_fsmap learned FSMap version 32 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x7f5d2400f850 con 0x17055b0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 1 -- 172.21.15.119:0/4188521994 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d1c053110 con 0x17055b0 2024-09-13T16:28:35.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 1 -- 172.21.15.119:0/4188521994 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 32) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f5d24036020 con 0x17055b0 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 10 client.6786 fetch_fsmap finished waiting for FSMap version 32 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 10 client.6786 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 20 client.6786 populate_metadata read hostname 'smithi119' 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x19bcd90 0x19d35a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 4 client.6786 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x17812c0 con 0x19bcd90 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x19bcd90 0x19d35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d2a7fc700 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x19bcd90 0x19d35a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4deb55d372f36029 server_cookie=95152397695534c2 in_seq=0 out_seq=0 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 10 client.6786 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:35.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 1 -- 172.21.15.119:0/4188521994 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+253 (crc 0 0 0) 0x17812c0 con 0x19bcd90 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 10 client.6786 handle_command_reply: tid=0 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 client.6786 shutdown 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x17055b0 msgr2=0x1a1d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x17055b0 0x1a1d550 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f5d24025040 tx=0x7f5d1c017000 comp rx=0 tx=0).stop 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 shutdown_connections 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x19bcd90 0x19d35a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1977a60 0x19d5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d11ffb700 0 client.6786 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1972310 0x19d5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 --2- 172.21.15.119:0/4188521994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x17055b0 0x1a1d550 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 >> 172.21.15.119:0/4188521994 conn(0x1976f10 msgr2=0x1995480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 shutdown_connections 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 1 -- 172.21.15.119:0/4188521994 wait complete. 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.181+0000 7f5d3c80f740 20 client.6786 trim_cache size 0 max 16384 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stdout: "0e647766-429e-4be5-8322-bdf487fce443": { 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "0e647766-429e-4be5-8322-bdf487fce443", 2024-09-13T16:28:35.192 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive" 2024-09-13T16:28:35.193 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:35.193 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:28:35.193 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:28:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d14066730 msgr2=0x7f5d14068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d14066730 0x7f5d14068bf0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f5d24001eb0 tx=0x7f5d24001e40 comp rx=0 tx=0).stop 2024-09-13T16:28:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d34159910 msgr2=0x7f5d3406ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:28:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d34159910 0x7f5d3406ea70 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c00e840 tx=0x7f5d1c00eb50 comp rx=0 tx=0).stop 2024-09-13T16:28:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 shutdown_connections 2024-09-13T16:28:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d14066730 0x7f5d14068bf0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d3415d1c0 0x7f5d3414c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d34159910 0x7f5d3406ea70 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 --2- 172.21.15.119:0/1486369649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d34159530 0x7f5d341425b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:28:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 >> 172.21.15.119:0/1486369649 conn(0x7f5d3406b460 msgr2=0x7f5d34155b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:28:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 shutdown_connections 2024-09-13T16:28:35.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:28:35.185+0000 7f5d33fff700 1 -- 172.21.15.119:0/1486369649 wait complete. 2024-09-13T16:28:35.203 INFO:tasks.cephfs.filesystem:scrub status for tag:0e647766-429e-4be5-8322-bdf487fce443 - {'path': '/', 'tag': '0e647766-429e-4be5-8322-bdf487fce443', 'options': 'recursive'} 2024-09-13T16:28:37.564 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:28:37.561+0000 7fed948e1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-09-13T16:28:57.280 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:28:57.283 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:29:05.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2024-09-13T16:29:05.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265acbf700 1 Processor -- start 2024-09-13T16:29:05.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265acbf700 1 -- start start 2024-09-13T16:29:05.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265acbf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f26540ce8c0 0x7f26540d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265acbf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26540d1a20 con 0x7f265415cc00 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265acbf700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26540d0df0 con 0x7f26540d1610 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265acbf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26540cdb00 con 0x7f26540ce8c0 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f265415cc00 0x7f26540d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:44150/0 (socket says 172.21.15.119:44150) 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 -- 172.21.15.119:0/431789432 learned_addr learned my addr 172.21.15.119:0/431789432 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f26594bc700 1 --2- 172.21.15.119:0/431789432 >> v2:172.21.15.145:3300/0 conn(0x7f26540ce8c0 0x7f26540d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114431075 0 0) 0x7f26540d1a20 con 0x7f265415cc00 2024-09-13T16:29:05.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f26400035d0 con 0x7f265415cc00 2024-09-13T16:29:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3879165926 0 0) 0x7f26400035d0 con 0x7f265415cc00 2024-09-13T16:29:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 >> v2:172.21.15.145:3300/0 conn(0x7f26540ce8c0 msgr2=0x7f26540d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 --2- 172.21.15.119:0/431789432 >> v2:172.21.15.145:3300/0 conn(0x7f26540ce8c0 0x7f26540d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 >> v1:172.21.15.145:6789/0 conn(0x7f26540d1610 legacy=0x7f26540b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540cdde0 con 0x7f265415cc00 2024-09-13T16:29:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f2650001670 con 0x7f265415cc00 2024-09-13T16:29:05.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 >> v1:172.21.15.119:6789/0 conn(0x7f265415cc00 legacy=0x7f26540d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2640003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2640004870 0x7f2640004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2640005230 0x7f264000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26540cdb00 con 0x7f2640003a20 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f264000c0d0 con 0x7f2640004870 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264000c290 con 0x7f2640005230 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2640003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f26594bc700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2640004870 0x7f2640004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265a4be700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2640005230 0x7f264000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 -- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2640005230 msgr2=0x7f264000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2640005230 0x7f264000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 -- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2640004870 msgr2=0x7f2640004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2640004870 0x7f2640004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 -- 172.21.15.119:0/431789432 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26400035d0 con 0x7f2640003a20 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265a4be700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2640005230 0x7f264000ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f265a4be700 1 -- 172.21.15.119:0/431789432 reap_dead start 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2659cbd700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2640003e40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f265000c320 tx=0x7f26400058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e538ebf6656b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650007c30 con 0x7f2640003a20 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f265000ce60 con 0x7f2640003a20 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.328+0000 7f2658cbb700 1 -- 172.21.15.119:0/431789432 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650005bc0 con 0x7f2640003a20 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- 172.21.15.119:0/431789432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 msgr2=0x7f2640003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2640003e40 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f265000c320 tx=0x7f26400058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- 172.21.15.119:0/431789432 shutdown_connections 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 --2- 172.21.15.119:0/431789432 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2640003e40 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- 172.21.15.119:0/431789432 >> 172.21.15.119:0/431789432 conn(0x7f265408c080 msgr2=0x7f26540d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- 172.21.15.119:0/431789432 shutdown_connections 2024-09-13T16:29:05.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- 172.21.15.119:0/431789432 wait complete. 2024-09-13T16:29:05.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 Processor -- start 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- start start 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2654151740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26540cc6e0 0x7f26541511b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26540cadf0 0x7f265415f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265415ed00 con 0x7f2640003a20 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265415efe0 con 0x7f26540cc6e0 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265acbf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26540ca5f0 con 0x7f26540cadf0 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2654151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2654151740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40760/0 (socket says 172.21.15.119:40760) 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 -- 172.21.15.119:0/185376142 learned_addr learned my addr 172.21.15.119:0/185376142 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f26594bc700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26540cc6e0 0x7f26541511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265a4be700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26540cadf0 0x7f265415f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 -- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26540cadf0 msgr2=0x7f265415f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26540cadf0 0x7f265415f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 -- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26540cc6e0 msgr2=0x7f26541511b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26540cc6e0 0x7f26541511b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 -- 172.21.15.119:0/185376142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26540ca8d0 con 0x7f2640003a20 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f265a4be700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26540cadf0 0x7f265415f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f2659cbd700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2654151740 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f265000c910 tx=0x7f2650001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.332+0000 7f264a7fc700 1 -- 172.21.15.119:0/185376142 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650001640 con 0x7f2640003a20 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f264a7fc700 1 -- 172.21.15.119:0/185376142 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26500070d0 con 0x7f2640003a20 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2654151e50 con 0x7f2640003a20 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f264a7fc700 1 -- 172.21.15.119:0/185376142 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2650007480 con 0x7f2640003a20 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26540cb450 con 0x7f2640003a20 2024-09-13T16:29:05.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f264a7fc700 1 -- 172.21.15.119:0/185376142 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f26500075e0 con 0x7f2640003a20 2024-09-13T16:29:05.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f264a7fc700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2638066760 0x7f2638068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f264a7fc700 1 -- 172.21.15.119:0/185376142 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2650084a80 con 0x7f2640003a20 2024-09-13T16:29:05.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.336+0000 7f26594bc700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2638066760 0x7f2638068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2638066760 0x7f2638068c20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2654160310 tx=0x7f2644009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 Processor -- start 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 -- start start 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152bd90 0x14e2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x14e1310 0x150a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1473010 con 0x14e1310 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1400f40 con 0x152c170 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1453e30 con 0x152bd90 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x14e1310 0x150a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x14e1310 0x150a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40762/0 (socket says 172.21.15.119:40762) 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 -- 172.21.15.119:0/318256516 learned_addr learned my addr 172.21.15.119:0/318256516 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265a4be700 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f2659cbd700 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152bd90 0x14e2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 -- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152bd90 msgr2=0x14e2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152bd90 0x14e2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 -- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 msgr2=0x152c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 -- 172.21.15.119:0/318256516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x12bfa40 con 0x14e1310 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265a4be700 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26594bc700 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x14e1310 0x150a520 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2644004390 tx=0x7f2644017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bee26375067cd9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26377fe700 1 -- 172.21.15.119:0/318256516 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26440036c0 con 0x14e1310 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26377fe700 1 -- 172.21.15.119:0/318256516 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2644019070 con 0x14e1310 2024-09-13T16:29:05.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f26377fe700 1 -- 172.21.15.119:0/318256516 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644003a20 con 0x14e1310 2024-09-13T16:29:05.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 -- 172.21.15.119:0/318256516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x14e1310 msgr2=0x150a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.340+0000 7f265f1d2740 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x14e1310 0x150a520 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f2644004390 tx=0x7f2644017000 comp rx=0 tx=0).stop 2024-09-13T16:29:05.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- 172.21.15.119:0/318256516 shutdown_connections 2024-09-13T16:29:05.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x14e1310 0x150a520 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 --2- 172.21.15.119:0/318256516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152bd90 0x14e2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- 172.21.15.119:0/318256516 >> 172.21.15.119:0/318256516 conn(0x12745b0 msgr2=0x14e5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- 172.21.15.119:0/318256516 shutdown_connections 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- 172.21.15.119:0/318256516 wait complete. 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- start start 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 Processor -- start 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1509dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1508fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x144f150 con 0x152bd90 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x144ceb0 con 0x152c170 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x140a8e0 con 0x14e1310 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 10 client.0 _ll_register_callbacks cb 0x13e9670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1508fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1508fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40764/0 (socket says 172.21.15.119:40764) 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 -- 172.21.15.119:0/3868988366 learned_addr learned my addr 172.21.15.119:0/3868988366 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f2659cbd700 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1509dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265a4be700 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 -- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 msgr2=0x1509dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1509dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 -- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 msgr2=0x152a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 -- 172.21.15.119:0/3868988366 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2644004040 con 0x152bd90 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f26594bc700 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1508fc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2644014c00 tx=0x7f2644004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56a1b2c8493ba6eb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f2634ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f2634ff9700 1 -- 172.21.15.119:0/3868988366 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264402f070 con 0x152bd90 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f2634ff9700 1 -- 172.21.15.119:0/3868988366 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2644009900 con 0x152bd90 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.344+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x12812c0 con 0x152bd90 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2634ff9700 1 -- 172.21.15.119:0/3868988366 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2644003ea0 con 0x152bd90 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 10 client.6798 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1413290 con 0x152bd90 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2634ff9700 1 -- 172.21.15.119:0/3868988366 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x12812c0 con 0x152bd90 2024-09-13T16:29:05.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2634ff9700 1 -- 172.21.15.119:0/3868988366 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f264401d030 con 0x152bd90 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 10 client.6798 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 10 client.6798 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 20 client.6798 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x153c790 0x153eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 4 client.6798 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x12cb540 con 0x153c790 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2659cbd700 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x153c790 0x153eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2634ff9700 10 client.6798 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2659cbd700 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x153c790 0x153eb80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee4ae913b6f020b3 server_cookie=d74a1a9b270cb0c6 in_seq=0 out_seq=0 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2634ff9700 1 -- 172.21.15.119:0/3868988366 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x12cb540 con 0x153c790 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2634ff9700 10 client.6798 handle_command_reply: tid=0 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 client.6798 shutdown 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 msgr2=0x1508fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1508fc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f2644014c00 tx=0x7f2644004770 comp rx=0 tx=0).stop 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 shutdown_connections 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x153c790 0x153eb80 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152c170 0x152a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1508fc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 --2- 172.21.15.119:0/3868988366 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1509dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f2634ff9700 0 client.6798 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 >> 172.21.15.119:0/3868988366 conn(0x12745b0 msgr2=0x150a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 shutdown_connections 2024-09-13T16:29:05.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.348+0000 7f265f1d2740 1 -- 172.21.15.119:0/3868988366 wait complete. 2024-09-13T16:29:05.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.352+0000 7f265f1d2740 20 client.6798 trim_cache size 0 max 16384 2024-09-13T16:29:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 Processor -- start 2024-09-13T16:29:05.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 -- start start 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1587d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152bd90 0x1588290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152c170 0x1503ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x12f8b80 con 0x152c170 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x13d6a50 con 0x152bd90 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x140da00 con 0x14e1310 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152c170 0x1503ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152c170 0x1503ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40772/0 (socket says 172.21.15.119:40772) 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 -- 172.21.15.119:0/2723241596 learned_addr learned my addr 172.21.15.119:0/2723241596 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f26594bc700 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152bd90 0x1588290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f2659cbd700 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1587d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 -- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 msgr2=0x1587d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1587d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 -- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152bd90 msgr2=0x1588290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152bd90 0x1588290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 -- 172.21.15.119:0/2723241596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x13f94c0 con 0x152c170 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.372+0000 7f265a4be700 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152c170 0x1503ef0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f265408c500 tx=0x7f264c002170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=572bcff19e141052 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f2637fff700 1 -- 172.21.15.119:0/2723241596 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c005530 con 0x152c170 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f2637fff700 1 -- 172.21.15.119:0/2723241596 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c0056b0 con 0x152c170 2024-09-13T16:29:05.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f2637fff700 1 -- 172.21.15.119:0/2723241596 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c017460 con 0x152c170 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- 172.21.15.119:0/2723241596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152c170 msgr2=0x1503ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152c170 0x1503ef0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f265408c500 tx=0x7f264c002170 comp rx=0 tx=0).stop 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- 172.21.15.119:0/2723241596 shutdown_connections 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152c170 0x1503ef0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x152bd90 0x1588290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 --2- 172.21.15.119:0/2723241596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x14e1310 0x1587d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- 172.21.15.119:0/2723241596 >> 172.21.15.119:0/2723241596 conn(0x12745b0 msgr2=0x153f3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- 172.21.15.119:0/2723241596 shutdown_connections 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- 172.21.15.119:0/2723241596 wait complete. 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- start start 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 Processor -- start 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x14e1310 0x1548e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1586f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152c170 0x15874c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1587a00 con 0x152bd90 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x15498e0 con 0x14e1310 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1549b80 con 0x152c170 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 10 client.0 _ll_register_callbacks cb 0x144b610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1586f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1586f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40774/0 (socket says 172.21.15.119:40774) 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 -- 172.21.15.119:0/630674173 learned_addr learned my addr 172.21.15.119:0/630674173 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f2659cbd700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x14e1310 0x1548e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265a4be700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152c170 0x15874c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 -- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152c170 msgr2=0x15874c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152c170 0x15874c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 -- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x14e1310 msgr2=0x1548e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x14e1310 0x1548e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 -- 172.21.15.119:0/630674173 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f264c00f040 con 0x152bd90 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265a4be700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152c170 0x15874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f2634ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f26594bc700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1586f80 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2644014f60 tx=0x7f264401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5435b393526bc01d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f2634ff9700 1 -- 172.21.15.119:0/630674173 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2644022e20 con 0x152bd90 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f2634ff9700 1 -- 172.21.15.119:0/630674173 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2644005a40 con 0x152bd90 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.376+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2644019070 con 0x152bd90 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2634ff9700 1 -- 172.21.15.119:0/630674173 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2644005c00 con 0x152bd90 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 10 client.6804 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x13ed790 con 0x152bd90 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2634ff9700 1 -- 172.21.15.119:0/630674173 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2644019070 con 0x152bd90 2024-09-13T16:29:05.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2634ff9700 1 -- 172.21.15.119:0/630674173 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2644014920 con 0x152bd90 2024-09-13T16:29:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 10 client.6804 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 10 client.6804 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:29:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 20 client.6804 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x154ace0 0x153f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 4 client.6804 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:29:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x12f02c0 con 0x154ace0 2024-09-13T16:29:05.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2659cbd700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x154ace0 0x153f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2634ff9700 10 client.6804 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2659cbd700 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x154ace0 0x153f450 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a900a756afd45d0c server_cookie=a31c6aabb0cf4c93 in_seq=0 out_seq=0 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2634ff9700 1 -- 172.21.15.119:0/630674173 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x12f02c0 con 0x154ace0 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2634ff9700 10 client.6804 handle_command_reply: tid=0 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 client.6804 shutdown 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 msgr2=0x1586f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1586f80 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f2644014f60 tx=0x7f264401b000 comp rx=0 tx=0).stop 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 shutdown_connections 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x154ace0 0x153f450 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x152c170 0x15874c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f2634ff9700 0 client.6804 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x152bd90 0x1586f80 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 --2- 172.21.15.119:0/630674173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x14e1310 0x1548e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 >> 172.21.15.119:0/630674173 conn(0x12745b0 msgr2=0x1507770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 shutdown_connections 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 1 -- 172.21.15.119:0/630674173 wait complete. 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265f1d2740 20 client.6804 trim_cache size 0 max 16384 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:29:05.391 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:29:05.392 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.380+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2638066760 msgr2=0x7f2638068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2638066760 0x7f2638068c20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f2654160310 tx=0x7f2644009240 comp rx=0 tx=0).stop 2024-09-13T16:29:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 msgr2=0x7f2654151740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2654151740 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f265000c910 tx=0x7f2650001a00 comp rx=0 tx=0).stop 2024-09-13T16:29:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 shutdown_connections 2024-09-13T16:29:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2638066760 0x7f2638068c20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f26540cadf0 0x7f265415f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f26540cc6e0 0x7f26541511b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 --2- 172.21.15.119:0/185376142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2640003a20 0x7f2654151740 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 >> 172.21.15.119:0/185376142 conn(0x7f265408c080 msgr2=0x7f26540b64c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 shutdown_connections 2024-09-13T16:29:05.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.384+0000 7f265acbf700 1 -- 172.21.15.119:0/185376142 wait complete. 2024-09-13T16:29:05.404 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:29:05.404 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:05.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459265f700 1 Processor -- start 2024-09-13T16:29:05.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459265f700 1 -- start start 2024-09-13T16:29:05.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459265f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f458c06a9a0 0x7f458c15c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459265f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c15cb20 0x7f458c06d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459265f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f458c0ae960 con 0x7f458c06a5c0 2024-09-13T16:29:05.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459265f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f458c0af4e0 con 0x7f458c06a9a0 2024-09-13T16:29:05.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459265f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c0aec40 con 0x7f458c15cb20 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459165d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f458c06a5c0 0x7f458c0adcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:44158/0 (socket says 172.21.15.119:44158) 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f459165d700 1 -- 172.21.15.119:0/4020071495 learned_addr learned my addr 172.21.15.119:0/4020071495 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f4590e5c700 1 --2- 172.21.15.119:0/4020071495 >> v2:172.21.15.119:3300/0 conn(0x7f458c06a9a0 0x7f458c15c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.516+0000 7f4591e5e700 1 --2- 172.21.15.119:0/4020071495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c15cb20 0x7f458c06d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4590e5c700 1 -- 172.21.15.119:0/4020071495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c15cb20 msgr2=0x7f458c06d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4590e5c700 1 --2- 172.21.15.119:0/4020071495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c15cb20 0x7f458c06d2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4590e5c700 1 -- 172.21.15.119:0/4020071495 >> v1:172.21.15.119:6789/0 conn(0x7f458c06a5c0 legacy=0x7f458c0adcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4583fff700 1 -- 172.21.15.119:0/4020071495 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665065334 0 0) 0x7f458c0ae960 con 0x7f458c06a5c0 2024-09-13T16:29:05.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4590e5c700 1 -- 172.21.15.119:0/4020071495 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f458c0aef20 con 0x7f458c06a9a0 2024-09-13T16:29:05.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4590e5c700 1 --2- 172.21.15.119:0/4020071495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a9a0 0x7f458c15c5e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f457c009850 tx=0x7f457c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61c5b402ee44fcc1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4583fff700 1 -- 172.21.15.119:0/4020071495 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c011830 con 0x7f458c06a9a0 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4583fff700 1 -- 172.21.15.119:0/4020071495 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f457c011990 con 0x7f458c06a9a0 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f4583fff700 1 -- 172.21.15.119:0/4020071495 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f457c011d40 con 0x7f458c06a9a0 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 -- 172.21.15.119:0/4020071495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a9a0 msgr2=0x7f458c15c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 --2- 172.21.15.119:0/4020071495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a9a0 0x7f458c15c5e0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f457c009850 tx=0x7f457c009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 -- 172.21.15.119:0/4020071495 shutdown_connections 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 --2- 172.21.15.119:0/4020071495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c15cb20 0x7f458c06d2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 --2- 172.21.15.119:0/4020071495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a9a0 0x7f458c15c5e0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 -- 172.21.15.119:0/4020071495 >> 172.21.15.119:0/4020071495 conn(0x7f458c070cc0 msgr2=0x7f458c0710d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 -- 172.21.15.119:0/4020071495 shutdown_connections 2024-09-13T16:29:05.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.520+0000 7f459265f700 1 -- 172.21.15.119:0/4020071495 wait complete. 2024-09-13T16:29:05.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 Processor -- start 2024-09-13T16:29:05.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 -- start start 2024-09-13T16:29:05.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a5c0 0x7f458c157030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c06a9a0 0x7f458c1f2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f458c15cb20 0x7f458c1f2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c1541c0 con 0x7f458c06a5c0 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f458c150c20 con 0x7f458c15cb20 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f458c154450 con 0x7f458c06a9a0 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a5c0 0x7f458c157030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a5c0 0x7f458c157030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40788/0 (socket says 172.21.15.119:40788) 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 -- 172.21.15.119:0/2489534945 learned_addr learned my addr 172.21.15.119:0/2489534945 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f4590e5c700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c06a9a0 0x7f458c1f2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f4591e5e700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f458c15cb20 0x7f458c1f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 -- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c06a9a0 msgr2=0x7f458c1f2130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c06a9a0 0x7f458c1f2130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 -- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f458c15cb20 msgr2=0x7f458c1f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f458c15cb20 0x7f458c1f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 -- 172.21.15.119:0/2489534945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f457c0094b0 con 0x7f458c06a5c0 2024-09-13T16:29:05.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459165d700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a5c0 0x7f458c157030 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f458800c550 tx=0x7f458800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f4581ffb700 1 -- 172.21.15.119:0/2489534945 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4588004ca0 con 0x7f458c06a5c0 2024-09-13T16:29:05.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f4581ffb700 1 -- 172.21.15.119:0/2489534945 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4588004e00 con 0x7f458c06a5c0 2024-09-13T16:29:05.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f458c15f220 con 0x7f458c06a5c0 2024-09-13T16:29:05.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f4581ffb700 1 -- 172.21.15.119:0/2489534945 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4588007310 con 0x7f458c06a5c0 2024-09-13T16:29:05.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.524+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f458c15f7e0 con 0x7f458c06a5c0 2024-09-13T16:29:05.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.528+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f458c0677a0 con 0x7f458c06a5c0 2024-09-13T16:29:05.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.528+0000 7f4581ffb700 1 -- 172.21.15.119:0/2489534945 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4588007470 con 0x7f458c06a5c0 2024-09-13T16:29:05.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.528+0000 7f4581ffb700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4570066760 0x7f4570068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.528+0000 7f4581ffb700 1 -- 172.21.15.119:0/2489534945 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4588083e30 con 0x7f458c06a5c0 2024-09-13T16:29:05.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.528+0000 7f4590e5c700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4570066760 0x7f4570068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.528+0000 7f4590e5c700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4570066760 0x7f4570068c20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f457c0090b0 tx=0x7f457c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.536+0000 7f4581ffb700 1 -- 172.21.15.119:0/2489534945 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f45880510f0 con 0x7f458c06a5c0 2024-09-13T16:29:05.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.744+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f458c15c5e0 con 0x7f458c06a5c0 2024-09-13T16:29:05.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f4581ffb700 1 -- 172.21.15.119:0/2489534945 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f4588007720 con 0x7f458c06a5c0 2024-09-13T16:29:05.752 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:05.752 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:28:37.563850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:29:05.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4570066760 msgr2=0x7f4570068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4570066760 0x7f4570068c20 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f457c0090b0 tx=0x7f457c008030 comp rx=0 tx=0).stop 2024-09-13T16:29:05.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a5c0 msgr2=0x7f458c157030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a5c0 0x7f458c157030 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f458800c550 tx=0x7f458800c860 comp rx=0 tx=0).stop 2024-09-13T16:29:05.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 shutdown_connections 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4570066760 0x7f4570068c20 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f458c15cb20 0x7f458c1f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f458c06a9a0 0x7f458c1f2130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 --2- 172.21.15.119:0/2489534945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f458c06a5c0 0x7f458c157030 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 >> 172.21.15.119:0/2489534945 conn(0x7f458c070cc0 msgr2=0x7f458c0ae330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 shutdown_connections 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.748+0000 7f459265f700 1 -- 172.21.15.119:0/2489534945 wait complete. 2024-09-13T16:29:05.755 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 33 2024-09-13T16:29:05.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d damage ls --format=json-pretty 2024-09-13T16:29:05.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f3401b96700 1 Processor -- start 2024-09-13T16:29:05.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f3401b96700 1 -- start start 2024-09-13T16:29:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f3401b96700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc0cb210 0x7f33fc0ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f3401b96700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc0b2690 con 0x7f33fc15e430 2024-09-13T16:29:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f3401b96700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc071a10 con 0x7f33fc0cd4c0 2024-09-13T16:29:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f3401b96700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc0b2970 con 0x7f33fc0cb210 2024-09-13T16:29:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33fb7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f33fc15e430 0x7f33fc15e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:44168/0 (socket says 172.21.15.119:44168) 2024-09-13T16:29:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33fb7fe700 1 -- 172.21.15.119:0/1432835090 learned_addr learned my addr 172.21.15.119:0/1432835090 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:05.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33faffd700 1 --2- 172.21.15.119:0/1432835090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc0cb210 0x7f33fc0ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33fa7fc700 1 -- 172.21.15.119:0/1432835090 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3128037459 0 0) 0x7f33fc0b2690 con 0x7f33fc15e430 2024-09-13T16:29:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33fa7fc700 1 -- 172.21.15.119:0/1432835090 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33e40035d0 con 0x7f33fc15e430 2024-09-13T16:29:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33faffd700 1 -- 172.21.15.119:0/1432835090 >> v1:172.21.15.145:6789/0 conn(0x7f33fc0cd4c0 legacy=0x7f33fc0cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33faffd700 1 -- 172.21.15.119:0/1432835090 >> v1:172.21.15.119:6789/0 conn(0x7f33fc15e430 legacy=0x7f33fc15e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33faffd700 1 -- 172.21.15.119:0/1432835090 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc071cf0 con 0x7f33fc0cb210 2024-09-13T16:29:05.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.880+0000 7f33faffd700 1 --2- 172.21.15.119:0/1432835090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc0cb210 0x7f33fc0ccf80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009880 tx=0x7f33f0009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c952cd2f90f3c8d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f33fa7fc700 1 -- 172.21.15.119:0/1432835090 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0011830 con 0x7f33fc0cb210 2024-09-13T16:29:05.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f33fa7fc700 1 -- 172.21.15.119:0/1432835090 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33f0011990 con 0x7f33fc0cb210 2024-09-13T16:29:05.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f33fa7fc700 1 -- 172.21.15.119:0/1432835090 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0011d20 con 0x7f33fc0cb210 2024-09-13T16:29:05.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 -- 172.21.15.119:0/1432835090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc0cb210 msgr2=0x7f33fc0ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 --2- 172.21.15.119:0/1432835090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc0cb210 0x7f33fc0ccf80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009880 tx=0x7f33f0009b90 comp rx=0 tx=0).stop 2024-09-13T16:29:05.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 -- 172.21.15.119:0/1432835090 shutdown_connections 2024-09-13T16:29:05.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 --2- 172.21.15.119:0/1432835090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc0cb210 0x7f33fc0ccf80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 -- 172.21.15.119:0/1432835090 >> 172.21.15.119:0/1432835090 conn(0x7f33fc0b0560 msgr2=0x7f33fc0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 -- 172.21.15.119:0/1432835090 shutdown_connections 2024-09-13T16:29:05.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 -- 172.21.15.119:0/1432835090 wait complete. 2024-09-13T16:29:05.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 Processor -- start 2024-09-13T16:29:05.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.884+0000 7f3401b96700 1 -- start start 2024-09-13T16:29:05.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33fc0cb210 0x7f33fc155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33fc0cd4c0 0x7f33fc1e44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc15e430 0x7f33fc0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc15cbd0 con 0x7f33fc0cb210 2024-09-13T16:29:05.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc15be10 con 0x7f33fc0cd4c0 2024-09-13T16:29:05.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc15c0f0 con 0x7f33fc15e430 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33fc0cb210 0x7f33fc155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33fc0cb210 0x7f33fc155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40804/0 (socket says 172.21.15.119:40804) 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 -- 172.21.15.119:0/3312627059 learned_addr learned my addr 172.21.15.119:0/3312627059 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33faffd700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33fc0cd4c0 0x7f33fc1e44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fbfff700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc15e430 0x7f33fc0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 -- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc15e430 msgr2=0x7f33fc0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc15e430 0x7f33fc0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 -- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33fc0cd4c0 msgr2=0x7f33fc1e44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33fc0cd4c0 0x7f33fc1e44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 -- 172.21.15.119:0/3312627059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33f00094b0 con 0x7f33fc0cb210 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fbfff700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc15e430 0x7f33fc0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33fc0cb210 0x7f33fc155ed0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c4d0 tx=0x7f33ec00c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33e3fff700 1 -- 172.21.15.119:0/3312627059 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec005d90 con 0x7f33fc0cb210 2024-09-13T16:29:05.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33e3fff700 1 -- 172.21.15.119:0/3312627059 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec0070d0 con 0x7f33fc0cb210 2024-09-13T16:29:05.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc15c3d0 con 0x7f33fc0cb210 2024-09-13T16:29:05.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f33e3fff700 1 -- 172.21.15.119:0/3312627059 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec007480 con 0x7f33fc0cb210 2024-09-13T16:29:05.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.888+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc1e49e0 con 0x7f33fc0cb210 2024-09-13T16:29:05.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.892+0000 7f33e3fff700 1 -- 172.21.15.119:0/3312627059 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f33ec0075e0 con 0x7f33fc0cb210 2024-09-13T16:29:05.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.892+0000 7f33e3fff700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33e8066730 0x7f33e8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.892+0000 7f33e3fff700 1 -- 172.21.15.119:0/3312627059 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f33ec0836d0 con 0x7f33fc0cb210 2024-09-13T16:29:05.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.892+0000 7f33faffd700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33e8066730 0x7f33e8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.892+0000 7f33faffd700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33e8066730 0x7f33e8068bf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009880 tx=0x7f33f0010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 Processor -- start 2024-09-13T16:29:05.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 -- start start 2024-09-13T16:29:05.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd2d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd2d7f0 0xd4a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd4b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xc76810 con 0xc97520 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xc73fa0 con 0xd4aec0 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xc6f890 con 0xd2d7f0 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd2d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd2d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40806/0 (socket says 172.21.15.119:40806) 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 -- 172.21.15.119:0/490850402 learned_addr learned my addr 172.21.15.119:0/490850402 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33faffd700 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd2d7f0 0xd4a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fbfff700 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd4b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 -- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd2d7f0 msgr2=0xd4a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd2d7f0 0xd4a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 -- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 msgr2=0xd4b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd4b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 -- 172.21.15.119:0/490850402 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xae1ba0 con 0xc97520 2024-09-13T16:29:05.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33faffd700 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd2d7f0 0xd4a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd2d2b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f33ec05c510 tx=0x7f33ec053fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d5b51739764ff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33e17fa700 1 -- 172.21.15.119:0/490850402 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec054650 con 0xc97520 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33e17fa700 1 -- 172.21.15.119:0/490850402 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec058480 con 0xc97520 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f33e17fa700 1 -- 172.21.15.119:0/490850402 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec05ca40 con 0xc97520 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 -- 172.21.15.119:0/490850402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 msgr2=0xd2d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd2d2b0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f33ec05c510 tx=0x7f33ec053fb0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 -- 172.21.15.119:0/490850402 shutdown_connections 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.896+0000 7f3404e47740 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd4b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd2d7f0 0xd4a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 --2- 172.21.15.119:0/490850402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd2d2b0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 -- 172.21.15.119:0/490850402 >> 172.21.15.119:0/490850402 conn(0xa96710 msgr2=0xd304c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 -- 172.21.15.119:0/490850402 shutdown_connections 2024-09-13T16:29:05.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 -- 172.21.15.119:0/490850402 wait complete. 2024-09-13T16:29:05.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 -- start start 2024-09-13T16:29:05.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 Processor -- start 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xd311e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd2d7f0 0xd2c150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd2c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xc32d80 con 0xd2d7f0 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xc32a20 con 0xd4aec0 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xb62770 con 0xc97520 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f3404e47740 10 client.0 _ll_register_callbacks cb 0xc68f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f33faffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd2d7f0 0xd2c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f33faffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd2d7f0 0xd2c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40808/0 (socket says 172.21.15.119:40808) 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f33faffd700 1 -- 172.21.15.119:0/2904092995 learned_addr learned my addr 172.21.15.119:0/2904092995 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xd311e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f33fbfff700 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd2c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f33faffd700 1 -- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 msgr2=0xd311e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.900+0000 7f33faffd700 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xd311e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33faffd700 1 -- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 msgr2=0xd2c690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33faffd700 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd2c690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33faffd700 1 -- 172.21.15.119:0/2904092995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc1e49e0 con 0xd2d7f0 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33faffd700 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd2d7f0 0xd2c150 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009ee0 tx=0x7f33f00047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d09431caa0546a79 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33da7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33da7fc700 1 -- 172.21.15.119:0/2904092995 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f0032440 con 0xd2d7f0 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33da7fc700 1 -- 172.21.15.119:0/2904092995 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f33f00325a0 con 0xd2d7f0 2024-09-13T16:29:05.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xaa3420 con 0xd2d7f0 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33da7fc700 1 -- 172.21.15.119:0/2904092995 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f00327a0 con 0xd2d7f0 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 10 client.6825 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0xc35910 con 0xd2d7f0 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33da7fc700 1 -- 172.21.15.119:0/2904092995 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xaa3420 con 0xd2d7f0 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33da7fc700 1 -- 172.21.15.119:0/2904092995 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f33f0032900 con 0xd2d7f0 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 10 client.6825 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 10 client.6825 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 20 client.6825 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd303e0 0xd081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 4 client.6825 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xaed6a0 con 0xd303e0 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd303e0 0xd081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33da7fc700 10 client.6825 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.904+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd303e0 0xd081e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d01a941c417ccbb1 server_cookie=452f9503d9032b13 in_seq=0 out_seq=0 2024-09-13T16:29:05.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f33da7fc700 1 -- 172.21.15.119:0/2904092995 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0xaed6a0 con 0xd303e0 2024-09-13T16:29:05.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f33da7fc700 10 client.6825 handle_command_reply: tid=0 2024-09-13T16:29:05.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 client.6825 shutdown 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd2d7f0 msgr2=0xd2c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd2d7f0 0xd2c150 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009ee0 tx=0x7f33f00047e0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 shutdown_connections 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd303e0 0xd081e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd4aec0 0xd2c690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f33da7fc700 0 client.6825 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd2d7f0 0xd2c150 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 --2- 172.21.15.119:0/2904092995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xd311e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 >> 172.21.15.119:0/2904092995 conn(0xa96710 msgr2=0xd4a280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 shutdown_connections 2024-09-13T16:29:05.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 1 -- 172.21.15.119:0/2904092995 wait complete. 2024-09-13T16:29:05.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.908+0000 7f3404e47740 20 client.6825 trim_cache size 0 max 16384 2024-09-13T16:29:05.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 Processor -- start 2024-09-13T16:29:05.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 -- start start 2024-09-13T16:29:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd081e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xd08720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd303e0 0xd02fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xd2cc90 con 0xc97520 2024-09-13T16:29:05.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xc30660 con 0xd2d7f0 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xc2d0e0 con 0xd303e0 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd081e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd081e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40818/0 (socket says 172.21.15.119:40818) 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 -- 172.21.15.119:0/1417561208 learned_addr learned my addr 172.21.15.119:0/1417561208 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33faffd700 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xd08720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fbfff700 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd303e0 0xd02fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 -- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd303e0 msgr2=0xd02fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd303e0 0xd02fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 -- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 msgr2=0xd08720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xd08720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 -- 172.21.15.119:0/1417561208 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xc1bb40 con 0xc97520 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33faffd700 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xd08720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd081e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f33ec054350 tx=0x7f33ec083070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d1e20e9d59c3f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33e1ffb700 1 -- 172.21.15.119:0/1417561208 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec05ca40 con 0xc97520 2024-09-13T16:29:05.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33e1ffb700 1 -- 172.21.15.119:0/1417561208 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec0529f0 con 0xc97520 2024-09-13T16:29:05.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.924+0000 7f33e1ffb700 1 -- 172.21.15.119:0/1417561208 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33ec053b40 con 0xc97520 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- 172.21.15.119:0/1417561208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 msgr2=0xd081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd081e0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7f33ec054350 tx=0x7f33ec083070 comp rx=0 tx=0).stop 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- 172.21.15.119:0/1417561208 shutdown_connections 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xd303e0 0xd02fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xd08720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 --2- 172.21.15.119:0/1417561208 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xc97520 0xd081e0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- 172.21.15.119:0/1417561208 >> 172.21.15.119:0/1417561208 conn(0xa96710 msgr2=0xd53b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- 172.21.15.119:0/1417561208 shutdown_connections 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- 172.21.15.119:0/1417561208 wait complete. 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- start start 2024-09-13T16:29:05.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 Processor -- start 2024-09-13T16:29:05.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xdb6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xdb7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd303e0 0xdb8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xc77aa0 con 0xd303e0 2024-09-13T16:29:05.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xc6d0a0 con 0xd2d7f0 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xc723f0 con 0xc97520 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f3404e47740 10 client.0 _ll_register_callbacks cb 0xc6e460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd303e0 0xdb8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd303e0 0xdb8150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40824/0 (socket says 172.21.15.119:40824) 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fbfff700 1 -- 172.21.15.119:0/1650101921 learned_addr learned my addr 172.21.15.119:0/1650101921 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xdb6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33faffd700 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xdb7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fbfff700 1 -- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 msgr2=0xdb6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fbfff700 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xdb6e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fbfff700 1 -- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 msgr2=0xdb7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:05.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.928+0000 7f33fbfff700 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xdb7c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33fbfff700 1 -- 172.21.15.119:0/1650101921 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0xae1ba0 con 0xd303e0 2024-09-13T16:29:05.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:05.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33fbfff700 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd303e0 0xdb8150 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f33f40055f0 tx=0x7f33f4005900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c55447b52b15cea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 1 -- 172.21.15.119:0/1650101921 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33f40169e0 con 0xd303e0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 1 -- 172.21.15.119:0/1650101921 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f33f4021c60 con 0xd303e0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f33ec058480 con 0xd303e0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 1 -- 172.21.15.119:0/1650101921 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33f4021e20 con 0xd303e0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 10 client.6831 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0xc0fc20 con 0xd303e0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 1 -- 172.21.15.119:0/1650101921 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33ec058480 con 0xd303e0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 1 -- 172.21.15.119:0/1650101921 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f33f4016b40 con 0xd303e0 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 10 client.6831 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 10 client.6831 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 20 client.6831 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:05.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd4aec0 0xdb88d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 4 client.6831 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0xb12420 con 0xd4aec0 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd4aec0 0xdb88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33fb7fe700 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd4aec0 0xdb88d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ec155c312c698af server_cookie=a8aa7adcfe295515 in_seq=0 out_seq=0 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 10 client.6831 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 1 -- 172.21.15.119:0/1650101921 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+141 (crc 0 0 0) 0xb12420 con 0xd4aec0 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 10 client.6831 handle_command_reply: tid=0 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 client.6831 shutdown 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd303e0 msgr2=0xdb8150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd303e0 0xdb8150 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f33f40055f0 tx=0x7f33f4005900 comp rx=0 tx=0).stop 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 shutdown_connections 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0xd4aec0 0xdb88d0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xd303e0 0xdb8150 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f33da7fc700 0 client.6831 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xd2d7f0 0xdb7c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 --2- 172.21.15.119:0/1650101921 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xc97520 0xdb6e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 >> 172.21.15.119:0/1650101921 conn(0xa96710 msgr2=0xd532e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 shutdown_connections 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 1 -- 172.21.15.119:0/1650101921 wait complete. 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.932+0000 7f3404e47740 20 client.6831 trim_cache size 0 max 16384 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:05.942 INFO:teuthology.orchestra.run.smithi119.stdout: "damage_type": "backtrace", 2024-09-13T16:29:05.943 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 3674348917, 2024-09-13T16:29:05.943 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627777, 2024-09-13T16:29:05.943 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/dir_test/file" 2024-09-13T16:29:05.943 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:05.943 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33e8066730 msgr2=0x7f33e8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33e8066730 0x7f33e8068bf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f33f0009880 tx=0x7f33f0010000 comp rx=0 tx=0).stop 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33fc0cb210 msgr2=0x7f33fc155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33fc0cb210 0x7f33fc155ed0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f33ec00c4d0 tx=0x7f33ec00c7e0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 shutdown_connections 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f33e8066730 0x7f33e8068bf0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33fc15e430 0x7f33fc0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f33fc0cd4c0 0x7f33fc1e44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 --2- 172.21.15.119:0/3312627059 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33fc0cb210 0x7f33fc155ed0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 >> 172.21.15.119:0/3312627059 conn(0x7f33fc0b0560 msgr2=0x7f33fc0ae4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 shutdown_connections 2024-09-13T16:29:05.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:05.936+0000 7f3401b96700 1 -- 172.21.15.119:0/3312627059 wait complete. 2024-09-13T16:29:05.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:29:06.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3ffff700 1 Processor -- start 2024-09-13T16:29:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3ffff700 1 -- start start 2024-09-13T16:29:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7c401649b0 0x7f7c40164d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3ffff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c4015b8a0 con 0x7f7c40070100 2024-09-13T16:29:06.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3ffff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c40071d40 con 0x7f7c4006fc90 2024-09-13T16:29:06.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3ffff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c40070530 con 0x7f7c401649b0 2024-09-13T16:29:06.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7c401649b0 0x7f7c40164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7c40070100 0x7f7c400b3510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:44170/0 (socket says 172.21.15.119:44170) 2024-09-13T16:29:06.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.060+0000 7f7c3f7fe700 1 -- 172.21.15.119:0/2330152352 learned_addr learned my addr 172.21.15.119:0/2330152352 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:06.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3effd700 1 -- 172.21.15.119:0/2330152352 >> v1:172.21.15.145:6789/0 conn(0x7f7c4006fc90 legacy=0x7f7c400b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3effd700 1 -- 172.21.15.119:0/2330152352 >> v1:172.21.15.119:6789/0 conn(0x7f7c40070100 legacy=0x7f7c400b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3dffb700 1 -- 172.21.15.119:0/2330152352 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1124596156 0 0) 0x7f7c4015b8a0 con 0x7f7c40070100 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3effd700 1 -- 172.21.15.119:0/2330152352 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c401652d0 con 0x7f7c401649b0 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3effd700 1 --2- 172.21.15.119:0/2330152352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164d90 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009850 tx=0x7f7c34009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf7b9380963677e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3dffb700 1 -- 172.21.15.119:0/2330152352 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34011830 con 0x7f7c401649b0 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3dffb700 1 -- 172.21.15.119:0/2330152352 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c34011990 con 0x7f7c401649b0 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3dffb700 1 -- 172.21.15.119:0/2330152352 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34011d20 con 0x7f7c401649b0 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 -- 172.21.15.119:0/2330152352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 msgr2=0x7f7c40164d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/2330152352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164d90 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009850 tx=0x7f7c34009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 -- 172.21.15.119:0/2330152352 shutdown_connections 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/2330152352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164d90 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 -- 172.21.15.119:0/2330152352 >> 172.21.15.119:0/2330152352 conn(0x7f7c400b0640 msgr2=0x7f7c400b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 -- 172.21.15.119:0/2330152352 shutdown_connections 2024-09-13T16:29:06.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 -- 172.21.15.119:0/2330152352 wait complete. 2024-09-13T16:29:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 Processor -- start 2024-09-13T16:29:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.064+0000 7f7c3ffff700 1 -- start start 2024-09-13T16:29:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4006fc90 0x7f7c4014da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c40070100 0x7f7c40163dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c400a8ca0 con 0x7f7c401649b0 2024-09-13T16:29:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c401fb2a0 con 0x7f7c4006fc90 2024-09-13T16:29:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4015ae90 con 0x7f7c40070100 2024-09-13T16:29:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40848/0 (socket says 172.21.15.119:40848) 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 -- 172.21.15.119:0/3707000858 learned_addr learned my addr 172.21.15.119:0/3707000858 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3e7fc700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c40070100 0x7f7c40163dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3effd700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4006fc90 0x7f7c4014da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 -- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c40070100 msgr2=0x7f7c40163dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c40070100 0x7f7c40163dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 -- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4006fc90 msgr2=0x7f7c4014da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4006fc90 0x7f7c4014da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 -- 172.21.15.119:0/3707000858 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c340094b0 con 0x7f7c401649b0 2024-09-13T16:29:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3e7fc700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c40070100 0x7f7c40163dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3effd700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4006fc90 0x7f7c4014da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3f7fe700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164300 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800cca0 tx=0x7f7c3800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c277fe700 1 -- 172.21.15.119:0/3707000858 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3800cea0 con 0x7f7c401649b0 2024-09-13T16:29:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c277fe700 1 -- 172.21.15.119:0/3707000858 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3800ade0 con 0x7f7c401649b0 2024-09-13T16:29:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4014dfc0 con 0x7f7c401649b0 2024-09-13T16:29:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c277fe700 1 -- 172.21.15.119:0/3707000858 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c38007310 con 0x7f7c401649b0 2024-09-13T16:29:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.068+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c400a85e0 con 0x7f7c401649b0 2024-09-13T16:29:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.072+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c400ce690 con 0x7f7c401649b0 2024-09-13T16:29:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.072+0000 7f7c277fe700 1 -- 172.21.15.119:0/3707000858 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7c38007470 con 0x7f7c401649b0 2024-09-13T16:29:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.072+0000 7f7c277fe700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c2c066730 0x7f7c2c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.072+0000 7f7c277fe700 1 -- 172.21.15.119:0/3707000858 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7c38084140 con 0x7f7c401649b0 2024-09-13T16:29:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.072+0000 7f7c3effd700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c2c066730 0x7f7c2c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.072+0000 7f7c3effd700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c2c066730 0x7f7c2c068bf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c340090b0 tx=0x7f7c34008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.080+0000 7f7c277fe700 1 -- 172.21.15.119:0/3707000858 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7c38051400 con 0x7f7c401649b0 2024-09-13T16:29:06.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7c400b15a0 con 0x7f7c401649b0 2024-09-13T16:29:06.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c277fe700 1 -- 172.21.15.119:0/3707000858 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f7c38054a20 con 0x7f7c401649b0 2024-09-13T16:29:06.300 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:06.300 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c2c066730 msgr2=0x7f7c2c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c2c066730 0x7f7c2c068bf0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7c340090b0 tx=0x7f7c34008030 comp rx=0 tx=0).stop 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 msgr2=0x7f7c40164300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164300 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800cca0 tx=0x7f7c3800a340 comp rx=0 tx=0).stop 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 shutdown_connections 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c2c066730 0x7f7c2c068bf0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c401649b0 0x7f7c40164300 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c40070100 0x7f7c40163dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 --2- 172.21.15.119:0/3707000858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4006fc90 0x7f7c4014da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 >> 172.21.15.119:0/3707000858 conn(0x7f7c400b0640 msgr2=0x7f7c400b2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 shutdown_connections 2024-09-13T16:29:06.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.296+0000 7f7c3ffff700 1 -- 172.21.15.119:0/3707000858 wait complete. 2024-09-13T16:29:06.317 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-13T16:29:06.317 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub start / repair,recursive,force 2024-09-13T16:29:06.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.436+0000 7f47ed287700 1 Processor -- start 2024-09-13T16:29:06.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.436+0000 7f47ed287700 1 -- start start 2024-09-13T16:29:06.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f47e815cc60 0x7f47e80cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f47e80cf9c0 0x7f47e80cfe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80b25d0 0x7f47e80b2a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e80d0370 con 0x7f47e815cc60 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e80b44d0 con 0x7f47e80cf9c0 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e80d12d0 con 0x7f47e80b25d0 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f47e815cc60 0x7f47e80cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f47e815cc60 0x7f47e80cf480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40858/0 (socket says 172.21.15.119:40858) 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 -- 172.21.15.119:0/3499901151 learned_addr learned my addr 172.21.15.119:0/3499901151 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e67fc700 1 --2- 172.21.15.119:0/3499901151 >> v2:172.21.15.145:3300/0 conn(0x7f47e80cf9c0 0x7f47e80cfe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e77fe700 1 --2- 172.21.15.119:0/3499901151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80b25d0 0x7f47e80b2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 -- 172.21.15.119:0/3499901151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80b25d0 msgr2=0x7f47e80b2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/3499901151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80b25d0 0x7f47e80b2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 -- 172.21.15.119:0/3499901151 >> v2:172.21.15.145:3300/0 conn(0x7f47e80cf9c0 msgr2=0x7f47e80cfe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/3499901151 >> v2:172.21.15.145:3300/0 conn(0x7f47e80cf9c0 0x7f47e80cfe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 -- 172.21.15.119:0/3499901151 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e80d15b0 con 0x7f47e815cc60 2024-09-13T16:29:06.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/3499901151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e815cc60 0x7f47e80cf480 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f47e80cd4a0 tx=0x7f47dc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f27d4c0e362a466 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e5ffb700 1 -- 172.21.15.119:0/3499901151 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc011830 con 0x7f47e815cc60 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e5ffb700 1 -- 172.21.15.119:0/3499901151 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc011990 con 0x7f47e815cc60 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47e5ffb700 1 -- 172.21.15.119:0/3499901151 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc011d40 con 0x7f47e815cc60 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- 172.21.15.119:0/3499901151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e815cc60 msgr2=0x7f47e80cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 --2- 172.21.15.119:0/3499901151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e815cc60 0x7f47e80cf480 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f47e80cd4a0 tx=0x7f47dc009c20 comp rx=0 tx=0).stop 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- 172.21.15.119:0/3499901151 shutdown_connections 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 --2- 172.21.15.119:0/3499901151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80b25d0 0x7f47e80b2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 --2- 172.21.15.119:0/3499901151 >> v2:172.21.15.145:3300/0 conn(0x7f47e80cf9c0 0x7f47e80cfe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 --2- 172.21.15.119:0/3499901151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e815cc60 0x7f47e80cf480 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- 172.21.15.119:0/3499901151 >> 172.21.15.119:0/3499901151 conn(0x7f47e80d2b70 msgr2=0x7f47e80d2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- 172.21.15.119:0/3499901151 shutdown_connections 2024-09-13T16:29:06.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.440+0000 7f47ed287700 1 -- 172.21.15.119:0/3499901151 wait complete. 2024-09-13T16:29:06.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 Processor -- start 2024-09-13T16:29:06.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 -- start start 2024-09-13T16:29:06.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e80b25d0 0x7f47e815dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80cf9c0 0x7f47e8155d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e815cc60 0x7f47e8158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e815e420 con 0x7f47e80b25d0 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e80cab50 con 0x7f47e815cc60 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47ed287700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e80cadf0 con 0x7f47e80cf9c0 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e80b25d0 0x7f47e815dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e80b25d0 0x7f47e815dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40862/0 (socket says 172.21.15.119:40862) 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 -- 172.21.15.119:0/1256936516 learned_addr learned my addr 172.21.15.119:0/1256936516 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e67fc700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80cf9c0 0x7f47e8155d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e77fe700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e815cc60 0x7f47e8158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 -- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80cf9c0 msgr2=0x7f47e8155d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80cf9c0 0x7f47e8155d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 -- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e815cc60 msgr2=0x7f47e8158f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e815cc60 0x7f47e8158f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.444+0000 7f47e6ffd700 1 -- 172.21.15.119:0/1256936516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc009570 con 0x7f47e80b25d0 2024-09-13T16:29:06.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e80b25d0 0x7f47e815dee0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f47dc000c00 tx=0x7f47dc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47cf7fe700 1 -- 172.21.15.119:0/1256936516 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc004510 con 0x7f47e80b25d0 2024-09-13T16:29:06.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47cf7fe700 1 -- 172.21.15.119:0/1256936516 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc01e070 con 0x7f47e80b25d0 2024-09-13T16:29:06.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47cf7fe700 1 -- 172.21.15.119:0/1256936516 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc003900 con 0x7f47e80b25d0 2024-09-13T16:29:06.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e80cb0d0 con 0x7f47e80b25d0 2024-09-13T16:29:06.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e81562c0 con 0x7f47e80b25d0 2024-09-13T16:29:06.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47cf7fe700 1 -- 172.21.15.119:0/1256936516 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f47dc003a60 con 0x7f47e80b25d0 2024-09-13T16:29:06.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47cf7fe700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f47d4066730 0x7f47d4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47cf7fe700 1 -- 172.21.15.119:0/1256936516 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f47dc0850f0 con 0x7f47e80b25d0 2024-09-13T16:29:06.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.448+0000 7f47e67fc700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f47d4066730 0x7f47d4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f47d4066730 0x7f47d4068bf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f47e80c96c0 tx=0x7f47d8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 Processor -- start 2024-09-13T16:29:06.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 -- start start 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e7ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e5e150 0x1e479c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7f3b0 0x1e47f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1da6100 con 0x1e5e150 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1da3890 con 0x1e7f3b0 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d9f180 con 0x1e7ea90 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e5e150 0x1e479c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e5e150 0x1e479c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40868/0 (socket says 172.21.15.119:40868) 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 -- 172.21.15.119:0/3111967238 learned_addr learned my addr 172.21.15.119:0/3111967238 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e7ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e77fe700 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7f3b0 0x1e47f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 -- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 msgr2=0x1e7ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e7ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 -- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7f3b0 msgr2=0x1e47f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.452+0000 7f47e67fc700 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7f3b0 0x1e47f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47e67fc700 1 -- 172.21.15.119:0/3111967238 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1c11d00 con 0x1e5e150 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47e67fc700 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e5e150 0x1e479c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f47d8011bb0 tx=0x7f47d8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baac352f95017ae6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47ccff9700 1 -- 172.21.15.119:0/3111967238 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d8011ec0 con 0x1e5e150 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47ccff9700 1 -- 172.21.15.119:0/3111967238 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d80158d0 con 0x1e5e150 2024-09-13T16:29:06.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47ccff9700 1 -- 172.21.15.119:0/3111967238 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d8015c40 con 0x1e5e150 2024-09-13T16:29:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 -- 172.21.15.119:0/3111967238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e5e150 msgr2=0x1e479c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e5e150 0x1e479c0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f47d8011bb0 tx=0x7f47d8014000 comp rx=0 tx=0).stop 2024-09-13T16:29:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 -- 172.21.15.119:0/3111967238 shutdown_connections 2024-09-13T16:29:06.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7f3b0 0x1e47f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e5e150 0x1e479c0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 --2- 172.21.15.119:0/3111967238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e7ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 -- 172.21.15.119:0/3111967238 >> 172.21.15.119:0/3111967238 conn(0x1bc6870 msgr2=0x1e60790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 -- 172.21.15.119:0/3111967238 shutdown_connections 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 -- 172.21.15.119:0/3111967238 wait complete. 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 -- start start 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.456+0000 7f47f0538740 1 Processor -- start 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e7d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e78140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7f3b0 0x1e78680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d62900 con 0x1e7f3b0 2024-09-13T16:29:06.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d625a0 con 0x1e5e150 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c4ae40 con 0x1e7ea90 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 10 client.0 _ll_register_callbacks cb 0x1d5f630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7f3b0 0x1e78680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e67fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e78140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e6ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e7d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7f3b0 0x1e78680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40874/0 (socket says 172.21.15.119:40874) 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e67fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e78140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:53062/0 (socket says 172.21.15.119:53062) 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e67fc700 1 -- 172.21.15.119:0/2566882529 learned_addr learned my addr 172.21.15.119:0/2566882529 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 -- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 msgr2=0x1e78140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e78140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 -- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 msgr2=0x1e7d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e7d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 -- 172.21.15.119:0/2566882529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d8011860 con 0x1e7f3b0 2024-09-13T16:29:06.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e7d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:06.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e77fe700 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7f3b0 0x1e78680 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007da0 tx=0x7f47e0013530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2498e4d4d17e109 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47c5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:06.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47c5ffb700 1 -- 172.21.15.119:0/2566882529 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e000e050 con 0x1e7f3b0 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47c5ffb700 1 -- 172.21.15.119:0/2566882529 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f47e0013bc0 con 0x1e7f3b0 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1bd3580 con 0x1e7f3b0 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47c5ffb700 1 -- 172.21.15.119:0/2566882529 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e0007e90 con 0x1e7f3b0 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 10 client.6852 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1d65540 con 0x1e7f3b0 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47c5ffb700 1 -- 172.21.15.119:0/2566882529 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1bd3580 con 0x1e7f3b0 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47c5ffb700 1 -- 172.21.15.119:0/2566882529 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f47e0019480 con 0x1e7f3b0 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 10 client.6852 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 10 client.6852 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 20 client.6852 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:06.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7e320 0x1e334b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 4 client.6852 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1c1d800 con 0x1e7e320 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7e320 0x1e334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47c5ffb700 10 client.6852 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.460+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7e320 0x1e334b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ce1191508e96b88 server_cookie=395a385fbeee29e1 in_seq=0 out_seq=0 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47c5ffb700 1 -- 172.21.15.119:0/2566882529 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1c1d800 con 0x1e7e320 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47c5ffb700 10 client.6852 handle_command_reply: tid=0 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 client.6852 shutdown 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7f3b0 msgr2=0x1e78680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7f3b0 0x1e78680 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007da0 tx=0x7f47e0013530 comp rx=0 tx=0).stop 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 shutdown_connections 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7e320 0x1e334b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7f3b0 0x1e78680 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47c5ffb700 0 client.6852 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7ea90 0x1e78140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 --2- 172.21.15.119:0/2566882529 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e7d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 >> 172.21.15.119:0/2566882529 conn(0x1bc6870 msgr2=0x1e491e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 shutdown_connections 2024-09-13T16:29:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 1 -- 172.21.15.119:0/2566882529 wait complete. 2024-09-13T16:29:06.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.464+0000 7f47f0538740 20 client.6852 trim_cache size 0 max 16384 2024-09-13T16:29:06.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 Processor -- start 2024-09-13T16:29:06.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 -- start start 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e84980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7e320 0x1e7f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e7fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d6b580 con 0x1e7ea90 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1d6ef10 con 0x1e5e150 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.484+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d9c990 con 0x1e7e320 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e7fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e7fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40888/0 (socket says 172.21.15.119:40888) 2024-09-13T16:29:06.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 -- 172.21.15.119:0/1600699488 learned_addr learned my addr 172.21.15.119:0/1600699488 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e67fc700 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7e320 0x1e7f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e84980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 -- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7e320 msgr2=0x1e7f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7e320 0x1e7f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 -- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 msgr2=0x1e84980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e84980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 -- 172.21.15.119:0/1600699488 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1d4b770 con 0x1e7ea90 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e67fc700 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7e320 0x1e7f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:06.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47e77fe700 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e7fcd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007080 tx=0x7f47e0019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e51f82b2db00dc41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47cd7fa700 1 -- 172.21.15.119:0/1600699488 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e000e070 con 0x1e7ea90 2024-09-13T16:29:06.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47cd7fa700 1 -- 172.21.15.119:0/1600699488 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47e000f040 con 0x1e7ea90 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47cd7fa700 1 -- 172.21.15.119:0/1600699488 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e0007e90 con 0x1e7ea90 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 -- 172.21.15.119:0/1600699488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 msgr2=0x1e7fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e7fcd0 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007080 tx=0x7f47e0019000 comp rx=0 tx=0).stop 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 -- 172.21.15.119:0/1600699488 shutdown_connections 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e7fcd0 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e7e320 0x1e7f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 --2- 172.21.15.119:0/1600699488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e5e150 0x1e84980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 -- 172.21.15.119:0/1600699488 >> 172.21.15.119:0/1600699488 conn(0x1bc6870 msgr2=0x1e5c9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 -- 172.21.15.119:0/1600699488 shutdown_connections 2024-09-13T16:29:06.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 -- 172.21.15.119:0/1600699488 wait complete. 2024-09-13T16:29:06.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.488+0000 7f47f0538740 1 -- start start 2024-09-13T16:29:06.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 Processor -- start 2024-09-13T16:29:06.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e5e150 0x1eea1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7e320 0x1e83810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e83d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1dce050 con 0x1e7ea90 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e7d800 con 0x1e7e320 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d6a7c0 con 0x1e5e150 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 10 client.0 _ll_register_callbacks cb 0x1d51000 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e83d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e83d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40892/0 (socket says 172.21.15.119:40892) 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 -- 172.21.15.119:0/216571056 learned_addr learned my addr 172.21.15.119:0/216571056 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e67fc700 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7e320 0x1e83810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e5e150 0x1eea1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 -- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e5e150 msgr2=0x1eea1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e5e150 0x1eea1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 -- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7e320 msgr2=0x1e83810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7e320 0x1e83810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 -- 172.21.15.119:0/216571056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1d65540 con 0x1e7ea90 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47e77fe700 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e83d50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f47e0000f80 tx=0x7f47e0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e14f285ca6bec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47c5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47c5ffb700 1 -- 172.21.15.119:0/216571056 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47e000e040 con 0x1e7ea90 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47c5ffb700 1 -- 172.21.15.119:0/216571056 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f47e003a070 con 0x1e7ea90 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1e35a50 con 0x1e7ea90 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47c5ffb700 1 -- 172.21.15.119:0/216571056 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47e0007e90 con 0x1e7ea90 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 10 client.6858 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:06.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1d3fa40 con 0x1e7ea90 2024-09-13T16:29:06.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.492+0000 7f47c5ffb700 1 -- 172.21.15.119:0/216571056 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1e35a50 con 0x1e7ea90 2024-09-13T16:29:06.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47c5ffb700 1 -- 172.21.15.119:0/216571056 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f47e0019b40 con 0x1e7ea90 2024-09-13T16:29:06.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 10 client.6858 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:06.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 10 client.6858 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 20 client.6858 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7f3b0 0x1ea4450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 4 client.6858 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}] 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive", "force"]}) v1 -- 0x7f47d80158d0 con 0x1e7f3b0 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7f3b0 0x1ea4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47c5ffb700 10 client.6858 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47e6ffd700 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7f3b0 0x1ea4450 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4206d7189636cda server_cookie=d407ecea5889e544 in_seq=0 out_seq=0 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47c5ffb700 1 -- 172.21.15.119:0/216571056 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f47d80158d0 con 0x1e7f3b0 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47c5ffb700 10 client.6858 handle_command_reply: tid=0 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 client.6858 shutdown 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 msgr2=0x1e83d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e83d50 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f47e0000f80 tx=0x7f47e0016000 comp rx=0 tx=0).stop 2024-09-13T16:29:06.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 shutdown_connections 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1e7f3b0 0x1ea4450 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e7ea90 0x1e83d50 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47c5ffb700 0 client.6858 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1e7e320 0x1e83810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 --2- 172.21.15.119:0/216571056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e5e150 0x1eea1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 >> 172.21.15.119:0/216571056 conn(0x1bc6870 msgr2=0x1e87040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 shutdown_connections 2024-09-13T16:29:06.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.496+0000 7f47f0538740 1 -- 172.21.15.119:0/216571056 wait complete. 2024-09-13T16:29:06.505 INFO:tasks.ceph.mds.d.smithi145.stderr:2024-09-13T16:29:06.499+0000 7f9e45c91700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-09-13T16:28:33.800048+0000 1=1+0) (iversion lock) caps={6723=pAsLsXsFs/-@2},l=6723(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x55f65c319600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:29:06.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47f0538740 20 client.6858 trim_cache size 0 max 16384 2024-09-13T16:29:06.505 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:06.505 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:29:06.505 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "625a345e-a31e-49a9-8673-c0157acceb57", 2024-09-13T16:29:06.505 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:29:06.506 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:06.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f47d4066730 msgr2=0x7f47d4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f47d4066730 0x7f47d4068bf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f47e80c96c0 tx=0x7f47d8008040 comp rx=0 tx=0).stop 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e80b25d0 msgr2=0x7f47e815dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e80b25d0 0x7f47e815dee0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f47dc000c00 tx=0x7f47dc004770 comp rx=0 tx=0).stop 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 shutdown_connections 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f47d4066730 0x7f47d4068bf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47e815cc60 0x7f47e8158f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47e80cf9c0 0x7f47e8155d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 --2- 172.21.15.119:0/1256936516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f47e80b25d0 0x7f47e815dee0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 >> 172.21.15.119:0/1256936516 conn(0x7f47e80d2b70 msgr2=0x7f47e80cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 shutdown_connections 2024-09-13T16:29:06.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.500+0000 7f47ed287700 1 -- 172.21.15.119:0/1256936516 wait complete. 2024-09-13T16:29:06.523 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.4:0 scrub status 2024-09-13T16:29:06.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 Processor -- start 2024-09-13T16:29:06.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- start start 2024-09-13T16:29:06.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e040aebd0 0x7f1e040b4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1e0415da10 0x7f1e041568c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0406d8b0 con 0x7f1e0406d140 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0406db90 con 0x7f1e040aebd0 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0406de70 con 0x7f1e0415da10 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e040aebd0 0x7f1e040b4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e040aebd0 0x7f1e040b4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40898/0 (socket says 172.21.15.119:40898) 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 -- 172.21.15.119:0/2626163281 learned_addr learned my addr 172.21.15.119:0/2626163281 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e037fe700 1 --2- 172.21.15.119:0/2626163281 >> v2:172.21.15.145:3300/0 conn(0x7f1e0415da10 0x7f1e041568c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 -- 172.21.15.119:0/2626163281 >> v2:172.21.15.145:3300/0 conn(0x7f1e0415da10 msgr2=0x7f1e041568c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/2626163281 >> v2:172.21.15.145:3300/0 conn(0x7f1e0415da10 0x7f1e041568c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 -- 172.21.15.119:0/2626163281 >> v1:172.21.15.119:6789/0 conn(0x7f1e0406d140 legacy=0x7f1e0415d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 -- 172.21.15.119:0/2626163281 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0406e150 con 0x7f1e040aebd0 2024-09-13T16:29:06.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/2626163281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e040b4670 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1e040af440 tx=0x7f1dec009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ee2f1c04f63e5f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e027fc700 1 -- 172.21.15.119:0/2626163281 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00b8a0 con 0x7f1e040aebd0 2024-09-13T16:29:06.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e027fc700 1 -- 172.21.15.119:0/2626163281 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dec00ba00 con 0x7f1e040aebd0 2024-09-13T16:29:06.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e027fc700 1 -- 172.21.15.119:0/2626163281 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dec00bd90 con 0x7f1e040aebd0 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- 172.21.15.119:0/2626163281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 msgr2=0x7f1e040b4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 --2- 172.21.15.119:0/2626163281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e040b4670 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f1e040af440 tx=0x7f1dec009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- 172.21.15.119:0/2626163281 shutdown_connections 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 --2- 172.21.15.119:0/2626163281 >> v2:172.21.15.145:3300/0 conn(0x7f1e0415da10 0x7f1e041568c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 --2- 172.21.15.119:0/2626163281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e040b4670 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- 172.21.15.119:0/2626163281 >> 172.21.15.119:0/2626163281 conn(0x7f1e040b1500 msgr2=0x7f1e040b18e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- 172.21.15.119:0/2626163281 shutdown_connections 2024-09-13T16:29:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.640+0000 7f1e03fff700 1 -- 172.21.15.119:0/2626163281 wait complete. 2024-09-13T16:29:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 Processor -- start 2024-09-13T16:29:06.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 -- start start 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e0406d140 0x7f1e041500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e0414aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e0415da10 0x7f1e041f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04156fc0 con 0x7f1e040aebd0 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0414afb0 con 0x7f1e0415da10 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e03fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04152180 con 0x7f1e0406d140 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e0414aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e0414aa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40906/0 (socket says 172.21.15.119:40906) 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1dfbfff700 1 -- 172.21.15.119:0/20415521 learned_addr learned my addr 172.21.15.119:0/20415521 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.644+0000 7f1e037fe700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e0415da10 0x7f1e041f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e0406d140 0x7f1e041500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfbfff700 1 -- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e0406d140 msgr2=0x7f1e041500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e0406d140 0x7f1e041500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfbfff700 1 -- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e0415da10 msgr2=0x7f1e041f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e0415da10 0x7f1e041f2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfbfff700 1 -- 172.21.15.119:0/20415521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec009570 con 0x7f1e040aebd0 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e0414aa70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1df400e940 tx=0x7f1df400ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfb7fe700 1 -- 172.21.15.119:0/20415521 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df40052d0 con 0x7f1e040aebd0 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfb7fe700 1 -- 172.21.15.119:0/20415521 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df4005430 con 0x7f1e040aebd0 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e04152460 con 0x7f1e040aebd0 2024-09-13T16:29:06.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfb7fe700 1 -- 172.21.15.119:0/20415521 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df40055a0 con 0x7f1e040aebd0 2024-09-13T16:29:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e04152a20 con 0x7f1e040aebd0 2024-09-13T16:29:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfb7fe700 1 -- 172.21.15.119:0/20415521 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1df400ca20 con 0x7f1e040aebd0 2024-09-13T16:29:06.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfb7fe700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1de8066730 0x7f1de8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.648+0000 7f1dfb7fe700 1 -- 172.21.15.119:0/20415521 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1df4086350 con 0x7f1e040aebd0 2024-09-13T16:29:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1de8066730 0x7f1de8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1de8066730 0x7f1de8068bf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1dec000c00 tx=0x7f1dec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 Processor -- start 2024-09-13T16:29:06.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 -- start start 2024-09-13T16:29:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1270180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12706c0 0x1251210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x12269e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x11b6010 con 0x124d800 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1143f40 con 0x12706c0 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1196e30 con 0x124d420 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x12269e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x12269e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40916/0 (socket says 172.21.15.119:40916) 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 -- 172.21.15.119:0/2797385891 learned_addr learned my addr 172.21.15.119:0/2797385891 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1270180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12706c0 0x1251210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 -- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 msgr2=0x1270180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1270180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 -- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12706c0 msgr2=0x1251210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12706c0 0x1251210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 -- 172.21.15.119:0/2797385891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1002a40 con 0x124d800 2024-09-13T16:29:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1e037fe700 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x12269e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009f10 tx=0x7f1dfc003d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9825c971d103f890 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1df8ff9700 1 -- 172.21.15.119:0/2797385891 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc003e50 con 0x124d800 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1df8ff9700 1 -- 172.21.15.119:0/2797385891 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc0046e0 con 0x124d800 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.652+0000 7f1df8ff9700 1 -- 172.21.15.119:0/2797385891 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc015420 con 0x124d800 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- 172.21.15.119:0/2797385891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 msgr2=0x12269e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x12269e0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc009f10 tx=0x7f1dfc003d20 comp rx=0 tx=0).stop 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- 172.21.15.119:0/2797385891 shutdown_connections 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x12269e0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12706c0 0x1251210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2797385891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1270180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- 172.21.15.119:0/2797385891 >> 172.21.15.119:0/2797385891 conn(0xfb75b0 msgr2=0x1228b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- 172.21.15.119:0/2797385891 shutdown_connections 2024-09-13T16:29:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- 172.21.15.119:0/2797385891 wait complete. 2024-09-13T16:29:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- start start 2024-09-13T16:29:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 Processor -- start 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d420 0x1248190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1249df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12706c0 0x124fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1192150 con 0x124d800 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x118feb0 con 0x124d420 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x114d8e0 con 0x12706c0 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e0c571740 10 client.0 _ll_register_callbacks cb 0x112c670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1249df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e02ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d420 0x1248190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1249df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40928/0 (socket says 172.21.15.119:40928) 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1dfbfff700 1 -- 172.21.15.119:0/4164535942 learned_addr learned my addr 172.21.15.119:0/4164535942 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.656+0000 7f1e037fe700 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12706c0 0x124fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1dfbfff700 1 -- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12706c0 msgr2=0x124fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12706c0 0x124fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1dfbfff700 1 -- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d420 msgr2=0x1248190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d420 0x1248190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1dfbfff700 1 -- 172.21.15.119:0/4164535942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dfc011040 con 0x124d800 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1249df0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f1df4001440 tx=0x7f1df40579b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ede0ec96dc2ba3e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 1 -- 172.21.15.119:0/4164535942 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df405b060 con 0x124d800 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 1 -- 172.21.15.119:0/4164535942 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1df407fe20 con 0x124d800 2024-09-13T16:29:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xfc42c0 con 0x124d800 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 1 -- 172.21.15.119:0/4164535942 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1df405f620 con 0x124d800 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 10 client.6873 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1156290 con 0x124d800 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 1 -- 172.21.15.119:0/4164535942 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xfc42c0 con 0x124d800 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 1 -- 172.21.15.119:0/4164535942 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1df405a040 con 0x124d800 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 10 client.6873 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 10 client.6873 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 20 client.6873 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12458c0 0x12795f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 4 client.6873 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x100e540 con 0x12458c0 2024-09-13T16:29:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12458c0 0x12795f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12458c0 0x12795f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2606f5c40dc700f6 server_cookie=1d53745e2eac5e8d in_seq=0 out_seq=0 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 10 client.6873 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 1 -- 172.21.15.119:0/4164535942 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x100e540 con 0x12458c0 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1df1ffb700 10 client.6873 handle_command_reply: tid=0 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.660+0000 7f1e0c571740 1 client.6873 shutdown 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 msgr2=0x1249df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1249df0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f1df4001440 tx=0x7f1df40579b0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 shutdown_connections 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12458c0 0x12795f0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12706c0 0x124fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1df1ffb700 0 client.6873 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1249df0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 --2- 172.21.15.119:0/4164535942 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d420 0x1248190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 >> 172.21.15.119:0/4164535942 conn(0xfb75b0 msgr2=0x126fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 shutdown_connections 2024-09-13T16:29:06.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 1 -- 172.21.15.119:0/4164535942 wait complete. 2024-09-13T16:29:06.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.664+0000 7f1e0c571740 20 client.6873 trim_cache size 0 max 16384 2024-09-13T16:29:06.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.680+0000 7f1e0c571740 1 Processor -- start 2024-09-13T16:29:06.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.680+0000 7f1e0c571740 1 -- start start 2024-09-13T16:29:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12458c0 0x12245e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1224b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1225060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x103bb80 con 0x124d800 2024-09-13T16:29:06.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1160c00 con 0x12458c0 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1137680 con 0x124d420 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1225060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1225060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40934/0 (socket says 172.21.15.119:40934) 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 -- 172.21.15.119:0/1419066571 learned_addr learned my addr 172.21.15.119:0/1419066571 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12458c0 0x12245e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1224b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 -- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 msgr2=0x1224b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1224b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 -- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12458c0 msgr2=0x12245e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12458c0 0x12245e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 -- 172.21.15.119:0/1419066571 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x113c4c0 con 0x124d800 2024-09-13T16:29:06.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12458c0 0x12245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e037fe700 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1225060 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc004320 tx=0x7f1dfc00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc33f8139f20c47 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1df97fa700 1 -- 172.21.15.119:0/1419066571 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc003e50 con 0x124d800 2024-09-13T16:29:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1df97fa700 1 -- 172.21.15.119:0/1419066571 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc004850 con 0x124d800 2024-09-13T16:29:06.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1df97fa700 1 -- 172.21.15.119:0/1419066571 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dfc015660 con 0x124d800 2024-09-13T16:29:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- 172.21.15.119:0/1419066571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 msgr2=0x1225060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1225060 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f1dfc004320 tx=0x7f1dfc00e000 comp rx=0 tx=0).stop 2024-09-13T16:29:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- 172.21.15.119:0/1419066571 shutdown_connections 2024-09-13T16:29:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d800 0x1225060 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x124d420 0x1224b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 --2- 172.21.15.119:0/1419066571 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12458c0 0x12245e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- 172.21.15.119:0/1419066571 >> 172.21.15.119:0/1419066571 conn(0xfb75b0 msgr2=0x12712c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- 172.21.15.119:0/1419066571 shutdown_connections 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- 172.21.15.119:0/1419066571 wait complete. 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 -- start start 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.684+0000 7f1e0c571740 1 Processor -- start 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12458c0 0x12d8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d420 0x1269ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d800 0x126a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x11b7610 con 0x124d420 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x118d6c0 con 0x124d800 2024-09-13T16:29:06.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1245f60 con 0x12458c0 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 10 client.0 _ll_register_callbacks cb 0x118e610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d420 0x1269ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d420 0x1269ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40948/0 (socket says 172.21.15.119:40948) 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 -- 172.21.15.119:0/2711361676 learned_addr learned my addr 172.21.15.119:0/2711361676 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12458c0 0x12d8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e037fe700 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d800 0x126a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 -- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12458c0 msgr2=0x12d8d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12458c0 0x12d8d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 -- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d800 msgr2=0x126a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d800 0x126a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 -- 172.21.15.119:0/2711361676 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e0406e150 con 0x124d420 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1dfbfff700 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d420 0x1269ff0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1df40571c0 tx=0x7f1df4084600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8692c45e00eb32c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1df1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1df1ffb700 1 -- 172.21.15.119:0/2711361676 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df4057930 con 0x124d420 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1dfc0046e0 con 0x124d420 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1df1ffb700 1 -- 172.21.15.119:0/2711361676 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1df406ee40 con 0x124d420 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1df1ffb700 1 -- 172.21.15.119:0/2711361676 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1df4057000 con 0x124d420 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 10 client.6879 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1002a40 con 0x124d420 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1df1ffb700 1 -- 172.21.15.119:0/2711361676 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dfc0046e0 con 0x124d420 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1df1ffb700 1 -- 172.21.15.119:0/2711361676 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1df4053940 con 0x124d420 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 10 client.6879 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 10 client.6879 resolve_mds: resolved 4:0 to role '4:0' aka daemon mds.d 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 20 client.6879 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:06.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12706c0 0x126aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 4 client.6879 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:29:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x10332c0 con 0x12706c0 2024-09-13T16:29:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12706c0 0x126aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1df1ffb700 10 client.6879 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.688+0000 7f1e02ffd700 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12706c0 0x126aa70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c340abce089f6c51 server_cookie=5e5fbb0b44645b3a in_seq=0 out_seq=0 2024-09-13T16:29:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1df1ffb700 1 -- 172.21.15.119:0/2711361676 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x10332c0 con 0x12706c0 2024-09-13T16:29:06.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1df1ffb700 10 client.6879 handle_command_reply: tid=0 2024-09-13T16:29:06.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 client.6879 shutdown 2024-09-13T16:29:06.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d420 msgr2=0x1269ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d420 0x1269ff0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f1df40571c0 tx=0x7f1df4084600 comp rx=0 tx=0).stop 2024-09-13T16:29:06.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 shutdown_connections 2024-09-13T16:29:06.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x12706c0 0x126aa70 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x124d800 0x126a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1df1ffb700 0 client.6879 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:06.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x124d420 0x1269ff0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 --2- 172.21.15.119:0/2711361676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12458c0 0x12d8d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 >> 172.21.15.119:0/2711361676 conn(0xfb75b0 msgr2=0x12279c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 shutdown_connections 2024-09-13T16:29:06.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 1 -- 172.21.15.119:0/2711361676 wait complete. 2024-09-13T16:29:06.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e0c571740 20 client.6879 trim_cache size 0 max 16384 2024-09-13T16:29:06.704 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:06.704 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:29:06.704 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:29:06.704 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:06.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1de8066730 msgr2=0x7f1de8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1de8066730 0x7f1de8068bf0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f1dec000c00 tx=0x7f1dec008030 comp rx=0 tx=0).stop 2024-09-13T16:29:06.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 msgr2=0x7f1e0414aa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e0414aa70 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f1df400e940 tx=0x7f1df400ec50 comp rx=0 tx=0).stop 2024-09-13T16:29:06.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 shutdown_connections 2024-09-13T16:29:06.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1de8066730 0x7f1de8068bf0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e0415da10 0x7f1e041f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e040aebd0 0x7f1e0414aa70 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 --2- 172.21.15.119:0/20415521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e0406d140 0x7f1e041500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 >> 172.21.15.119:0/20415521 conn(0x7f1e040b1500 msgr2=0x7f1e040ced10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 shutdown_connections 2024-09-13T16:29:06.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.692+0000 7f1e03fff700 1 -- 172.21.15.119:0/20415521 wait complete. 2024-09-13T16:29:06.713 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:29:06.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:06.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.828+0000 7feb19c0f700 1 Processor -- start 2024-09-13T16:29:06.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.828+0000 7feb19c0f700 1 -- start start 2024-09-13T16:29:06.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb140b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feb140b3630 0x7feb1406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb140b2280 con 0x7feb141594f0 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb140cc890 con 0x7feb140b3630 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb140b3ac0 con 0x7feb1406dc40 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feb140b3630 0x7feb1406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feb140b3630 0x7feb1406fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40954/0 (socket says 172.21.15.119:40954) 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 -- 172.21.15.119:0/2551040064 learned_addr learned my addr 172.21.15.119:0/2551040064 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb0bfff700 1 --2- 172.21.15.119:0/2551040064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb140b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 -- 172.21.15.119:0/2551040064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 msgr2=0x7feb140b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 --2- 172.21.15.119:0/2551040064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb140b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 -- 172.21.15.119:0/2551040064 >> v1:172.21.15.145:6789/0 conn(0x7feb141594f0 legacy=0x7feb1406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 -- 172.21.15.119:0/2551040064 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb140b1970 con 0x7feb140b3630 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb1940e700 1 --2- 172.21.15.119:0/2551040064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1406fc00 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7feb140b1b60 tx=0x7feb0c00d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81da9ae26674bc11 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb0b7fe700 1 -- 172.21.15.119:0/2551040064 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c010070 con 0x7feb140b3630 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb0b7fe700 1 -- 172.21.15.119:0/2551040064 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb0c00dd90 con 0x7feb140b3630 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb0b7fe700 1 -- 172.21.15.119:0/2551040064 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb0c00b310 con 0x7feb140b3630 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- 172.21.15.119:0/2551040064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 msgr2=0x7feb1406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 --2- 172.21.15.119:0/2551040064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1406fc00 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7feb140b1b60 tx=0x7feb0c00d430 comp rx=0 tx=0).stop 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- 172.21.15.119:0/2551040064 shutdown_connections 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 --2- 172.21.15.119:0/2551040064 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1406fc00 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 --2- 172.21.15.119:0/2551040064 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb140b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- 172.21.15.119:0/2551040064 >> 172.21.15.119:0/2551040064 conn(0x7feb1406a350 msgr2=0x7feb1406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:06.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- 172.21.15.119:0/2551040064 shutdown_connections 2024-09-13T16:29:06.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.832+0000 7feb19c0f700 1 -- 172.21.15.119:0/2551040064 wait complete. 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 Processor -- start 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 -- start start 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb14155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1415bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb141594f0 0x7feb1415c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb141557b0 con 0x7feb140b3630 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb141582d0 con 0x7feb141594f0 2024-09-13T16:29:06.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb140aba70 con 0x7feb1406dc40 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1415bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1415bc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40970/0 (socket says 172.21.15.119:40970) 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 -- 172.21.15.119:0/433621906 learned_addr learned my addr 172.21.15.119:0/433621906 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb18c0d700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb14155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb1940e700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb141594f0 0x7feb1415c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 -- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 msgr2=0x7feb14155270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb14155270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 -- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb141594f0 msgr2=0x7feb1415c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb141594f0 0x7feb1415c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 -- 172.21.15.119:0/433621906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb0c00bd40 con 0x7feb140b3630 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb18c0d700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb14155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:06.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb0bfff700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1415bc00 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7feb0400e790 tx=0x7feb0400eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb097fa700 1 -- 172.21.15.119:0/433621906 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04013610 con 0x7feb140b3630 2024-09-13T16:29:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb097fa700 1 -- 172.21.15.119:0/433621906 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb04013770 con 0x7feb140b3630 2024-09-13T16:29:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb14157750 con 0x7feb140b3630 2024-09-13T16:29:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb097fa700 1 -- 172.21.15.119:0/433621906 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb04013ae0 con 0x7feb140b3630 2024-09-13T16:29:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.836+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb14157d10 con 0x7feb140b3630 2024-09-13T16:29:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.840+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb140b30f0 con 0x7feb140b3630 2024-09-13T16:29:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.840+0000 7feb097fa700 1 -- 172.21.15.119:0/433621906 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7feb04013c40 con 0x7feb140b3630 2024-09-13T16:29:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.840+0000 7feb097fa700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feb00066730 0x7feb00068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.840+0000 7feb097fa700 1 -- 172.21.15.119:0/433621906 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7feb04083e20 con 0x7feb140b3630 2024-09-13T16:29:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.840+0000 7feb18c0d700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feb00066730 0x7feb00068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.840+0000 7feb18c0d700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feb00066730 0x7feb00068bf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb10000f60 tx=0x7feb10009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:06.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:06.848+0000 7feb097fa700 1 -- 172.21.15.119:0/433621906 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7feb040510e0 con 0x7feb140b3630 2024-09-13T16:29:07.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.052+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb14070a90 con 0x7feb140b3630 2024-09-13T16:29:07.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb097fa700 1 -- 172.21.15.119:0/433621906 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 33 v33) v1 ==== 94+0+4741 (secure 0 0 0) 0x7feb04021020 con 0x7feb140b3630 2024-09-13T16:29:07.059 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:07.059 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":33,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":33,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:28:37.563850+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":7,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:29:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feb00066730 msgr2=0x7feb00068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feb00066730 0x7feb00068bf0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7feb10000f60 tx=0x7feb10009040 comp rx=0 tx=0).stop 2024-09-13T16:29:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 msgr2=0x7feb1415bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1415bc00 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7feb0400e790 tx=0x7feb0400eb50 comp rx=0 tx=0).stop 2024-09-13T16:29:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 shutdown_connections 2024-09-13T16:29:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feb00066730 0x7feb00068bf0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feb141594f0 0x7feb1415c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feb140b3630 0x7feb1415bc00 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 --2- 172.21.15.119:0/433621906 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feb1406dc40 0x7feb14155270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 >> 172.21.15.119:0/433621906 conn(0x7feb1406a350 msgr2=0x7feb1406fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 shutdown_connections 2024-09-13T16:29:07.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.056+0000 7feb19c0f700 1 -- 172.21.15.119:0/433621906 wait complete. 2024-09-13T16:29:07.063 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 33 2024-09-13T16:29:07.075 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d damage ls --format=json-pretty 2024-09-13T16:29:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.184+0000 7f27caaee700 1 Processor -- start 2024-09-13T16:29:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.184+0000 7f27caaee700 1 -- start start 2024-09-13T16:29:07.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f27c406a5c0 0x7f27c40cec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27c4154f60 0x7f27c40cf1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27c40cf850 con 0x7f27c406a9a0 2024-09-13T16:29:07.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c415f340 con 0x7f27c4154f60 2024-09-13T16:29:07.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27c406cb60 con 0x7f27c406a5c0 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27c4154f60 0x7f27c40cf1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27c4154f60 0x7f27c40cf1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40974/0 (socket says 172.21.15.119:40974) 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 -- 172.21.15.119:0/1227549111 learned_addr learned my addr 172.21.15.119:0/1227549111 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27c9aec700 1 --2- 172.21.15.119:0/1227549111 >> v2:172.21.15.145:3300/0 conn(0x7f27c406a5c0 0x7f27c40cec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 -- 172.21.15.119:0/1227549111 >> v2:172.21.15.145:3300/0 conn(0x7f27c406a5c0 msgr2=0x7f27c40cec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/1227549111 >> v2:172.21.15.145:3300/0 conn(0x7f27c406a5c0 0x7f27c40cec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 -- 172.21.15.119:0/1227549111 >> v1:172.21.15.145:6789/0 conn(0x7f27c406a9a0 legacy=0x7f27c4154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 -- 172.21.15.119:0/1227549111 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c415aee0 con 0x7f27c4154f60 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27c9aec700 1 --2- 172.21.15.119:0/1227549111 >> v2:172.21.15.145:3300/0 conn(0x7f27c406a5c0 0x7f27c40cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/1227549111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c4154f60 0x7f27c40cf1d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f27c4072fc0 tx=0x7f27bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36a5d74aa91c5809 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27c8aea700 1 -- 172.21.15.119:0/1227549111 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00e070 con 0x7f27c4154f60 2024-09-13T16:29:07.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27c8aea700 1 -- 172.21.15.119:0/1227549111 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27bc00b9c0 con 0x7f27c4154f60 2024-09-13T16:29:07.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27c8aea700 1 -- 172.21.15.119:0/1227549111 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27bc00bd70 con 0x7f27c4154f60 2024-09-13T16:29:07.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- 172.21.15.119:0/1227549111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c4154f60 msgr2=0x7f27c40cf1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 --2- 172.21.15.119:0/1227549111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c4154f60 0x7f27c40cf1d0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f27c4072fc0 tx=0x7f27bc00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:07.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- 172.21.15.119:0/1227549111 shutdown_connections 2024-09-13T16:29:07.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 --2- 172.21.15.119:0/1227549111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c4154f60 0x7f27c40cf1d0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 --2- 172.21.15.119:0/1227549111 >> v2:172.21.15.145:3300/0 conn(0x7f27c406a5c0 0x7f27c40cec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- 172.21.15.119:0/1227549111 >> 172.21.15.119:0/1227549111 conn(0x7f27c406b460 msgr2=0x7f27c406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- 172.21.15.119:0/1227549111 shutdown_connections 2024-09-13T16:29:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.188+0000 7f27caaee700 1 -- 172.21.15.119:0/1227549111 wait complete. 2024-09-13T16:29:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 Processor -- start 2024-09-13T16:29:07.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 -- start start 2024-09-13T16:29:07.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27c406a5c0 0x7f27c4155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c406a9a0 0x7f27c413e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27c4154f60 0x7f27c41577b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c4156350 con 0x7f27c406a9a0 2024-09-13T16:29:07.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27c413e820 con 0x7f27c406a5c0 2024-09-13T16:29:07.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27caaee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27c4157cf0 con 0x7f27c4154f60 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c406a9a0 0x7f27c413e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c406a9a0 0x7f27c413e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40986/0 (socket says 172.21.15.119:40986) 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27c92eb700 1 -- 172.21.15.119:0/1706808541 learned_addr learned my addr 172.21.15.119:0/1706808541 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27c4154f60 0x7f27c41577b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27c9aec700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27c406a5c0 0x7f27c4155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27c92eb700 1 -- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27c4154f60 msgr2=0x7f27c41577b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.192+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27c4154f60 0x7f27c41577b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27c92eb700 1 -- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27c406a5c0 msgr2=0x7f27c4155e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27c406a5c0 0x7f27c4155e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27c92eb700 1 -- 172.21.15.119:0/1706808541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc009b30 con 0x7f27c406a9a0 2024-09-13T16:29:07.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c406a9a0 0x7f27c413e2e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f27b800c530 tx=0x7f27b800c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27b67fc700 1 -- 172.21.15.119:0/1706808541 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b80052d0 con 0x7f27c406a9a0 2024-09-13T16:29:07.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27b67fc700 1 -- 172.21.15.119:0/1706808541 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b8005430 con 0x7f27c406a9a0 2024-09-13T16:29:07.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27c414b990 con 0x7f27c406a9a0 2024-09-13T16:29:07.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27b67fc700 1 -- 172.21.15.119:0/1706808541 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b80055a0 con 0x7f27c406a9a0 2024-09-13T16:29:07.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.196+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c41528c0 con 0x7f27c406a9a0 2024-09-13T16:29:07.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27b67fc700 1 -- 172.21.15.119:0/1706808541 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f27b80038e0 con 0x7f27c406a9a0 2024-09-13T16:29:07.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27b67fc700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27a8066730 0x7f27a8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27b67fc700 1 -- 172.21.15.119:0/1706808541 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f27b8083580 con 0x7f27c406a9a0 2024-09-13T16:29:07.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c9aec700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27a8066730 0x7f27a8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 Processor -- start 2024-09-13T16:29:07.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 -- start start 2024-09-13T16:29:07.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c9aec700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27a8066730 0x7f27a8068bf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f27c00099f0 tx=0x7f27c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13a1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x139dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x139e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x12c9810 con 0x13a1950 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x12c6fa0 con 0x13a1d30 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x12c2890 con 0x12ea520 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x139dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x139dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40992/0 (socket says 172.21.15.119:40992) 2024-09-13T16:29:07.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27ca2ed700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x139e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c9aec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 -- 172.21.15.119:0/1360106730 learned_addr learned my addr 172.21.15.119:0/1360106730 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 -- 172.21.15.119:0/1360106730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 msgr2=0x13a1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1360106730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13a1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 -- 172.21.15.119:0/1360106730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 msgr2=0x139e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1360106730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x139e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 -- 172.21.15.119:0/1360106730 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1134ba0 con 0x13a1950 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.200+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1360106730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x139dcd0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f27b8050a00 tx=0x7f27b8053ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=319c6969e1286cd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27a77fe700 1 -- 172.21.15.119:0/1360106730 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8058160 con 0x13a1950 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27a77fe700 1 -- 172.21.15.119:0/1360106730 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b8054c00 con 0x13a1950 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27a77fe700 1 -- 172.21.15.119:0/1360106730 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8054500 con 0x13a1950 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- 172.21.15.119:0/1360106730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 msgr2=0x139dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 --2- 172.21.15.119:0/1360106730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x139dcd0 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f27b8050a00 tx=0x7f27b8053ee0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- 172.21.15.119:0/1360106730 shutdown_connections 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 --2- 172.21.15.119:0/1360106730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x139e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 --2- 172.21.15.119:0/1360106730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x139dcd0 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 --2- 172.21.15.119:0/1360106730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13a1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- 172.21.15.119:0/1360106730 >> 172.21.15.119:0/1360106730 conn(0x10e9710 msgr2=0x13836d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- 172.21.15.119:0/1360106730 shutdown_connections 2024-09-13T16:29:07.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- 172.21.15.119:0/1360106730 wait complete. 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- start start 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 Processor -- start 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13ab500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1285d80 con 0x13a1950 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1285a20 con 0x12ea520 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x11b5770 con 0x13a1d30 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27cf001740 10 client.0 _ll_register_callbacks cb 0x12bbf00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13ab500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13ab500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41008/0 (socket says 172.21.15.119:41008) 2024-09-13T16:29:07.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.204+0000 7f27c92eb700 1 -- 172.21.15.119:0/3499592020 learned_addr learned my addr 172.21.15.119:0/3499592020 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c9aec700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c92eb700 1 -- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 msgr2=0x13aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c92eb700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c92eb700 1 -- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 msgr2=0x137ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c92eb700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c92eb700 1 -- 172.21.15.119:0/3499592020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27c41528c0 con 0x13a1950 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c92eb700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13ab500 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f27b80013b0 tx=0x7f27b8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94a37ab49f44973 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 1 -- 172.21.15.119:0/3499592020 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8081070 con 0x13a1950 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 1 -- 172.21.15.119:0/3499592020 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f27b805c8f0 con 0x13a1950 2024-09-13T16:29:07.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x10f6420 con 0x13a1950 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 1 -- 172.21.15.119:0/3499592020 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27b8054500 con 0x13a1950 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 10 client.6897 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1288910 con 0x13a1950 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 1 -- 172.21.15.119:0/3499592020 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x10f6420 con 0x13a1950 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 1 -- 172.21.15.119:0/3499592020 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f27b805c460 con 0x13a1950 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 10 client.6897 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 10 client.6897 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 20 client.6897 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x13b27c0 0x13b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 4 client.6897 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x11406a0 con 0x13b27c0 2024-09-13T16:29:07.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c9aec700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x13b27c0 0x13b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 10 client.6897 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27c9aec700 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x13b27c0 0x13b4bb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2dfce5c4b4aeaac5 server_cookie=1ec859096b79fa79 in_seq=0 out_seq=0 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 1 -- 172.21.15.119:0/3499592020 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x11406a0 con 0x13b27c0 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27a4ff9700 10 client.6897 handle_command_reply: tid=0 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.208+0000 7f27cf001740 1 client.6897 shutdown 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 msgr2=0x13ab500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13ab500 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f27b80013b0 tx=0x7f27b8000ed0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 shutdown_connections 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x13b27c0 0x13b4bb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27a4ff9700 0 client.6897 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:07.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13ab500 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 --2- 172.21.15.119:0/3499592020 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 >> 172.21.15.119:0/3499592020 conn(0x10e9710 msgr2=0x139e310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 shutdown_connections 2024-09-13T16:29:07.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 1 -- 172.21.15.119:0/3499592020 wait complete. 2024-09-13T16:29:07.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.212+0000 7f27cf001740 20 client.6897 trim_cache size 0 max 16384 2024-09-13T16:29:07.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 Processor -- start 2024-09-13T16:29:07.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- start start 2024-09-13T16:29:07.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x137fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13798d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x12800e0 con 0x13a1950 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x135c180 con 0x12ea520 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x13a0df0 con 0x13a1d30 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x137fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x137fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41022/0 (socket says 172.21.15.119:41022) 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 -- 172.21.15.119:0/1098671162 learned_addr learned my addr 172.21.15.119:0/1098671162 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c9aec700 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13798d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 -- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 msgr2=0x13798d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13798d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 -- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 msgr2=0x137f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 -- 172.21.15.119:0/1098671162 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x126eb40 con 0x13a1950 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13798d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c9aec700 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27c92eb700 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x137fee0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f27b807f040 tx=0x7f27b8056000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=811eadef05e1e413 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27a7fff700 1 -- 172.21.15.119:0/1098671162 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8081050 con 0x13a1950 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27a7fff700 1 -- 172.21.15.119:0/1098671162 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b80542f0 con 0x13a1950 2024-09-13T16:29:07.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27a7fff700 1 -- 172.21.15.119:0/1098671162 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8054500 con 0x13a1950 2024-09-13T16:29:07.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- 172.21.15.119:0/1098671162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 msgr2=0x137fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x137fee0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f27b807f040 tx=0x7f27b8056000 comp rx=0 tx=0).stop 2024-09-13T16:29:07.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- 172.21.15.119:0/1098671162 shutdown_connections 2024-09-13T16:29:07.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x13a1d30 0x13798d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x137fee0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 --2- 172.21.15.119:0/1098671162 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea520 0x137f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- 172.21.15.119:0/1098671162 >> 172.21.15.119:0/1098671162 conn(0x10e9710 msgr2=0x1357a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- 172.21.15.119:0/1098671162 shutdown_connections 2024-09-13T16:29:07.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.228+0000 7f27cf001740 1 -- 172.21.15.119:0/1098671162 wait complete. 2024-09-13T16:29:07.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 -- start start 2024-09-13T16:29:07.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 Processor -- start 2024-09-13T16:29:07.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13c8030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13c7150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x13ab630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1283660 con 0x13a1950 2024-09-13T16:29:07.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1293980 con 0x13a1d30 2024-09-13T16:29:07.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1285d80 con 0x12ea520 2024-09-13T16:29:07.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27cf001740 10 client.0 _ll_register_callbacks cb 0x12c1460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:07.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13c7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13c7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41028/0 (socket says 172.21.15.119:41028) 2024-09-13T16:29:07.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 -- 172.21.15.119:0/2887112477 learned_addr learned my addr 172.21.15.119:0/2887112477 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27ca2ed700 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x13ab630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c9aec700 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 -- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 msgr2=0x13c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 -- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 msgr2=0x13ab630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x13ab630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.232+0000 7f27c92eb700 1 -- 172.21.15.119:0/2887112477 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1288910 con 0x13a1950 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27c92eb700 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13c7150 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f27b8081a50 tx=0x7f27b8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=592244ef4d2fddc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 1 -- 172.21.15.119:0/2887112477 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27b8081040 con 0x13a1950 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 1 -- 172.21.15.119:0/2887112477 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f27b805c8f0 con 0x13a1950 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1165420 con 0x13a1950 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 1 -- 172.21.15.119:0/2887112477 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27b8058160 con 0x13a1950 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 10 client.6903 fetch_fsmap learned FSMap version 33 2024-09-13T16:29:07.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=2+}) v3 -- 0x1262c20 con 0x13a1950 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 1 -- 172.21.15.119:0/2887112477 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1165420 con 0x13a1950 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 1 -- 172.21.15.119:0/2887112477 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 33) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f27b805a4c0 con 0x13a1950 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 10 client.6903 fetch_fsmap finished waiting for FSMap version 33 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 10 client.6903 resolve_mds: resolved name 'd' to daemon mds.d 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 20 client.6903 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1409da0 0x140a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 4 client.6903 mds_command: new command op to 6546 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 --> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x1262c20 con 0x1409da0 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27c9aec700 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1409da0 0x140a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 10 client.6903 ms_handle_connect on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27c9aec700 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1409da0 0x140a180 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cff079fe29f3a334 server_cookie=f10ea03b2a3c4b74 in_seq=0 out_seq=0 2024-09-13T16:29:07.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 1 -- 172.21.15.119:0/2887112477 <== mds.0 v2:172.21.15.145:6836/3224929185 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x1262c20 con 0x1409da0 2024-09-13T16:29:07.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 10 client.6903 handle_command_reply: tid=0 2024-09-13T16:29:07.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 client.6903 shutdown 2024-09-13T16:29:07.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 msgr2=0x13c7150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13c7150 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f27b8081a50 tx=0x7f27b8001120 comp rx=0 tx=0).stop 2024-09-13T16:29:07.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 shutdown_connections 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:6836/3224929185,v1:172.21.15.145:6837/3224929185] conn(0x1409da0 0x140a180 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x13a1d30 0x13ab630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x13a1950 0x13c7150 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 --2- 172.21.15.119:0/2887112477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea520 0x13c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27a4ff9700 0 client.6903 ms_handle_reset on v2:172.21.15.145:6836/3224929185 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 >> 172.21.15.119:0/2887112477 conn(0x10e9710 msgr2=0x139f5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 shutdown_connections 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 1 -- 172.21.15.119:0/2887112477 wait complete. 2024-09-13T16:29:07.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.236+0000 7f27cf001740 20 client.6903 trim_cache size 0 max 16384 2024-09-13T16:29:07.247 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:07.247 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27a8066730 msgr2=0x7f27a8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27a8066730 0x7f27a8068bf0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f27c00099f0 tx=0x7f27c0008040 comp rx=0 tx=0).stop 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c406a9a0 msgr2=0x7f27c413e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c406a9a0 0x7f27c413e2e0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f27b800c530 tx=0x7f27b800c840 comp rx=0 tx=0).stop 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 shutdown_connections 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27a8066730 0x7f27a8068bf0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27c4154f60 0x7f27c41577b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27c406a9a0 0x7f27c413e2e0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 --2- 172.21.15.119:0/1706808541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27c406a5c0 0x7f27c4155e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 >> 172.21.15.119:0/1706808541 conn(0x7f27c406b460 msgr2=0x7f27c40b0b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 shutdown_connections 2024-09-13T16:29:07.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.240+0000 7f27caaee700 1 -- 172.21.15.119:0/1706808541 wait complete. 2024-09-13T16:29:07.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:29:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec2359e700 1 Processor -- start 2024-09-13T16:29:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec2359e700 1 -- start start 2024-09-13T16:29:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec2359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2406dc80 0x7fec24070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec2359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fec2415b030 0x7fec2415b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec2359e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec240ce2c0 con 0x7fec240cdee0 2024-09-13T16:29:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec2359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec2415bda0 con 0x7fec2415b030 2024-09-13T16:29:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec2359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec24070760 con 0x7fec2406dc80 2024-09-13T16:29:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec22d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fec2415b030 0x7fec2415b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec22d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fec2415b030 0x7fec2415b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41044/0 (socket says 172.21.15.119:41044) 2024-09-13T16:29:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.372+0000 7fec22d9d700 1 -- 172.21.15.119:0/1732689160 learned_addr learned my addr 172.21.15.119:0/1732689160 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec21d9b700 1 --2- 172.21.15.119:0/1732689160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2406dc80 0x7fec24070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec22d9d700 1 -- 172.21.15.119:0/1732689160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2406dc80 msgr2=0x7fec24070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec22d9d700 1 --2- 172.21.15.119:0/1732689160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2406dc80 0x7fec24070bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec22d9d700 1 -- 172.21.15.119:0/1732689160 >> v1:172.21.15.145:6789/0 conn(0x7fec240cdee0 legacy=0x7fec2406d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec22d9d700 1 -- 172.21.15.119:0/1732689160 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2406fe30 con 0x7fec2415b030 2024-09-13T16:29:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec21d9b700 1 --2- 172.21.15.119:0/1732689160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2406dc80 0x7fec24070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec22d9d700 1 --2- 172.21.15.119:0/1732689160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec2415b030 0x7fec2415b4c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009e50 tx=0x7fec1c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab0d0aed1e85a37 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2159a700 1 -- 172.21.15.119:0/1732689160 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c00e070 con 0x7fec2415b030 2024-09-13T16:29:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2159a700 1 -- 172.21.15.119:0/1732689160 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1c003a40 con 0x7fec2415b030 2024-09-13T16:29:07.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2159a700 1 -- 172.21.15.119:0/1732689160 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c003df0 con 0x7fec2415b030 2024-09-13T16:29:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 -- 172.21.15.119:0/1732689160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec2415b030 msgr2=0x7fec2415b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 --2- 172.21.15.119:0/1732689160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec2415b030 0x7fec2415b4c0 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7fec1c009e50 tx=0x7fec1c00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 -- 172.21.15.119:0/1732689160 shutdown_connections 2024-09-13T16:29:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 --2- 172.21.15.119:0/1732689160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec2415b030 0x7fec2415b4c0 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 --2- 172.21.15.119:0/1732689160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2406dc80 0x7fec24070bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 -- 172.21.15.119:0/1732689160 >> 172.21.15.119:0/1732689160 conn(0x7fec240b0640 msgr2=0x7fec240b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 -- 172.21.15.119:0/1732689160 shutdown_connections 2024-09-13T16:29:07.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.376+0000 7fec2359e700 1 -- 172.21.15.119:0/1732689160 wait complete. 2024-09-13T16:29:07.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 Processor -- start 2024-09-13T16:29:07.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 -- start start 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec2406dc80 0x7fec2415a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec240cdee0 0x7fec24155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2415b030 0x7fec241f3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2415ac90 con 0x7fec240cdee0 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2414a3a0 con 0x7fec2406dc80 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec2414a680 con 0x7fec2415b030 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec21d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec240cdee0 0x7fec24155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec2259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec2406dc80 0x7fec2415a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec21d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec240cdee0 0x7fec24155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41056/0 (socket says 172.21.15.119:41056) 2024-09-13T16:29:07.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec21d9b700 1 -- 172.21.15.119:0/965483434 learned_addr learned my addr 172.21.15.119:0/965483434 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.380+0000 7fec22d9d700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2415b030 0x7fec241f3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec21d9b700 1 -- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2415b030 msgr2=0x7fec241f3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec21d9b700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2415b030 0x7fec241f3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec21d9b700 1 -- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec2406dc80 msgr2=0x7fec2415a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec21d9b700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec2406dc80 0x7fec2415a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec21d9b700 1 -- 172.21.15.119:0/965483434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1c009b30 con 0x7fec240cdee0 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec22d9d700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2415b030 0x7fec241f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec21d9b700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec240cdee0 0x7fec24155430 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c7f0 tx=0x7fec0c00cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec1affd700 1 -- 172.21.15.119:0/965483434 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c011800 con 0x7fec240cdee0 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec1affd700 1 -- 172.21.15.119:0/965483434 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0c011960 con 0x7fec240cdee0 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2414a960 con 0x7fec240cdee0 2024-09-13T16:29:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec1affd700 1 -- 172.21.15.119:0/965483434 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0c011d10 con 0x7fec240cdee0 2024-09-13T16:29:07.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec24155970 con 0x7fec240cdee0 2024-09-13T16:29:07.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2414b160 con 0x7fec240cdee0 2024-09-13T16:29:07.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.384+0000 7fec1affd700 1 -- 172.21.15.119:0/965483434 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fec0c018360 con 0x7fec240cdee0 2024-09-13T16:29:07.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.388+0000 7fec1affd700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec08066730 0x7fec08068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:07.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.388+0000 7fec1affd700 1 -- 172.21.15.119:0/965483434 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fec0c0841b0 con 0x7fec240cdee0 2024-09-13T16:29:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.388+0000 7fec2259c700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec08066730 0x7fec08068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:07.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.388+0000 7fec2259c700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec08066730 0x7fec08068bf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fec14005150 tx=0x7fec14007ee0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:07.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.392+0000 7fec1affd700 1 -- 172.21.15.119:0/965483434 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fec2414b160 con 0x7fec240cdee0 2024-09-13T16:29:07.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fec2414b160 con 0x7fec240cdee0 2024-09-13T16:29:07.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec1affd700 1 -- 172.21.15.119:0/965483434 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7fec2414b160 con 0x7fec240cdee0 2024-09-13T16:29:07.600 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:07.600 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec08066730 msgr2=0x7fec08068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec08066730 0x7fec08068bf0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fec14005150 tx=0x7fec14007ee0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec240cdee0 msgr2=0x7fec24155430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec240cdee0 0x7fec24155430 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fec0c00c7f0 tx=0x7fec0c00cb00 comp rx=0 tx=0).stop 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 shutdown_connections 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec08066730 0x7fec08068bf0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec2415b030 0x7fec241f3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec240cdee0 0x7fec24155430 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 --2- 172.21.15.119:0/965483434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec2406dc80 0x7fec2415a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:07.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 >> 172.21.15.119:0/965483434 conn(0x7fec240b0640 msgr2=0x7fec2415da10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:07.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 shutdown_connections 2024-09-13T16:29:07.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:07.596+0000 7fec2359e700 1 -- 172.21.15.119:0/965483434 wait complete. 2024-09-13T16:29:07.614 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=100 retry_count=0)... 2024-09-13T16:29:12.616 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:29:12.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 Processor -- start 2024-09-13T16:29:12.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- start start 2024-09-13T16:29:12.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fca5406fc90 0x7fca54070100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:12.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fca540b2e00 0x7fca540b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca5415b8a0 con 0x7fca541649b0 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca54071d40 con 0x7fca5406fc90 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca540cd240 con 0x7fca540b2e00 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fca5406fc90 0x7fca54070100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fca5406fc90 0x7fca54070100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58486/0 (socket says 172.21.15.119:58486) 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 -- 172.21.15.119:0/3853228581 learned_addr learned my addr 172.21.15.119:0/3853228581 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca52d9d700 1 --2- 172.21.15.119:0/3853228581 >> v2:172.21.15.145:3300/0 conn(0x7fca540b2e00 0x7fca540b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 -- 172.21.15.119:0/3853228581 >> v2:172.21.15.145:3300/0 conn(0x7fca540b2e00 msgr2=0x7fca540b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 --2- 172.21.15.119:0/3853228581 >> v2:172.21.15.145:3300/0 conn(0x7fca540b2e00 0x7fca540b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 -- 172.21.15.119:0/3853228581 >> v1:172.21.15.145:6789/0 conn(0x7fca541649b0 legacy=0x7fca54164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 -- 172.21.15.119:0/3853228581 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca540cd520 con 0x7fca5406fc90 2024-09-13T16:29:12.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca49d9b700 1 --2- 172.21.15.119:0/3853228581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5406fc90 0x7fca54070100 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009910 tx=0x7fca3c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54a4c79b49c7a2df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:12.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca51d9b700 1 -- 172.21.15.119:0/3853228581 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c004230 con 0x7fca5406fc90 2024-09-13T16:29:12.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca51d9b700 1 -- 172.21.15.119:0/3853228581 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c004390 con 0x7fca5406fc90 2024-09-13T16:29:12.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca51d9b700 1 -- 172.21.15.119:0/3853228581 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c009e60 con 0x7fca5406fc90 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- 172.21.15.119:0/3853228581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5406fc90 msgr2=0x7fca54070100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 --2- 172.21.15.119:0/3853228581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5406fc90 0x7fca54070100 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fca3c009910 tx=0x7fca3c009c20 comp rx=0 tx=0).stop 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- 172.21.15.119:0/3853228581 shutdown_connections 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 --2- 172.21.15.119:0/3853228581 >> v2:172.21.15.145:3300/0 conn(0x7fca540b2e00 0x7fca540b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 --2- 172.21.15.119:0/3853228581 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca5406fc90 0x7fca54070100 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- 172.21.15.119:0/3853228581 >> 172.21.15.119:0/3853228581 conn(0x7fca540b0640 msgr2=0x7fca540b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- 172.21.15.119:0/3853228581 shutdown_connections 2024-09-13T16:29:12.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.732+0000 7fca5359e700 1 -- 172.21.15.119:0/3853228581 wait complete. 2024-09-13T16:29:12.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 Processor -- start 2024-09-13T16:29:12.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 -- start start 2024-09-13T16:29:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca5406fc90 0x7fca541562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca540b2e00 0x7fca54157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca541649b0 0x7fca54151940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca54156e70 con 0x7fca540b2e00 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca54155230 con 0x7fca5406fc90 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca54155510 con 0x7fca541649b0 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca540b2e00 0x7fca54157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca540b2e00 0x7fca54157a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58496/0 (socket says 172.21.15.119:58496) 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 -- 172.21.15.119:0/366789609 learned_addr learned my addr 172.21.15.119:0/366789609 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5259c700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca5406fc90 0x7fca541562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca52d9d700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca541649b0 0x7fca54151940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 -- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca541649b0 msgr2=0x7fca54151940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:12.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca541649b0 0x7fca54151940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 -- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca5406fc90 msgr2=0x7fca541562d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca5406fc90 0x7fca541562d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca49d9b700 1 -- 172.21.15.119:0/366789609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca3c009570 con 0x7fca540b2e00 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.736+0000 7fca5259c700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca5406fc90 0x7fca541562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca49d9b700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca540b2e00 0x7fca54157a10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fca3c008000 tx=0x7fca3c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca4b7fe700 1 -- 172.21.15.119:0/366789609 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00bd70 con 0x7fca540b2e00 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca4b7fe700 1 -- 172.21.15.119:0/366789609 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c01f070 con 0x7fca540b2e00 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca541557f0 con 0x7fca540b2e00 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca4b7fe700 1 -- 172.21.15.119:0/366789609 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c003900 con 0x7fca540b2e00 2024-09-13T16:29:12.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca54155db0 con 0x7fca540b2e00 2024-09-13T16:29:12.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca540ce690 con 0x7fca540b2e00 2024-09-13T16:29:12.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca4b7fe700 1 -- 172.21.15.119:0/366789609 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fca3c021030 con 0x7fca540b2e00 2024-09-13T16:29:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.740+0000 7fca4b7fe700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca34066730 0x7fca34068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.744+0000 7fca4b7fe700 1 -- 172.21.15.119:0/366789609 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fca3c085fc0 con 0x7fca540b2e00 2024-09-13T16:29:12.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.744+0000 7fca5259c700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca34066730 0x7fca34068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:12.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.744+0000 7fca5259c700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca34066730 0x7fca34068bf0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fca540ac960 tx=0x7fca44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:12.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.744+0000 7fca4b7fe700 1 -- 172.21.15.119:0/366789609 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fca3c0531e0 con 0x7fca540b2e00 2024-09-13T16:29:12.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fca540ce690 con 0x7fca540b2e00 2024-09-13T16:29:12.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca4b7fe700 1 -- 172.21.15.119:0/366789609 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fca3c023e90 con 0x7fca540b2e00 2024-09-13T16:29:12.956 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:12.956 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T16:29:12.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca34066730 msgr2=0x7fca34068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca34066730 0x7fca34068bf0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fca540ac960 tx=0x7fca44008040 comp rx=0 tx=0).stop 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca540b2e00 msgr2=0x7fca54157a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca540b2e00 0x7fca54157a10 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7fca3c008000 tx=0x7fca3c004770 comp rx=0 tx=0).stop 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 shutdown_connections 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fca34066730 0x7fca34068bf0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fca541649b0 0x7fca54151940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fca540b2e00 0x7fca54157a10 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 --2- 172.21.15.119:0/366789609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fca5406fc90 0x7fca541562d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 >> 172.21.15.119:0/366789609 conn(0x7fca540b0640 msgr2=0x7fca540cb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 shutdown_connections 2024-09-13T16:29:12.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:12.952+0000 7fca5359e700 1 -- 172.21.15.119:0/366789609 wait complete. 2024-09-13T16:29:12.970 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s and 0 retries 2024-09-13T16:29:12.970 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:29:12.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:29:12.988 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:29:13.008 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:29:13.008 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:13.008 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:13.015 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:13.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb1cb3700 1 Processor -- start 2024-09-13T16:29:13.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb1cb3700 1 -- start start 2024-09-13T16:29:13.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb1cb3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac0cf960 0x7fdcac06e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb1cb3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdcac06e7b0 0x7fdcac06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb1cb3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcac0ceba0 con 0x7fdcac160ed0 2024-09-13T16:29:13.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb1cb3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcac06ec40 con 0x7fdcac06e7b0 2024-09-13T16:29:13.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb1cb3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac0cee80 con 0x7fdcac0cf960 2024-09-13T16:29:13.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb0cb1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fdcac160ed0 0x7fdcac0d1120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58888/0 (socket says 172.21.15.119:58888) 2024-09-13T16:29:13.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb0cb1700 1 -- 172.21.15.119:0/4151720090 learned_addr learned my addr 172.21.15.119:0/4151720090 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:13.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcabfff700 1 --2- 172.21.15.119:0/4151720090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac0cf960 0x7fdcac06e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcb14b2700 1 --2- 172.21.15.119:0/4151720090 >> v2:172.21.15.145:3300/0 conn(0x7fdcac06e7b0 0x7fdcac06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcab7fe700 1 -- 172.21.15.119:0/4151720090 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840611410 0 0) 0x7fdcac0ceba0 con 0x7fdcac160ed0 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcab7fe700 1 -- 172.21.15.119:0/4151720090 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc8c0035d0 con 0x7fdcac160ed0 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcabfff700 1 -- 172.21.15.119:0/4151720090 >> v2:172.21.15.145:3300/0 conn(0x7fdcac06e7b0 msgr2=0x7fdcac06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcabfff700 1 --2- 172.21.15.119:0/4151720090 >> v2:172.21.15.145:3300/0 conn(0x7fdcac06e7b0 0x7fdcac06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcabfff700 1 -- 172.21.15.119:0/4151720090 >> v1:172.21.15.119:6789/0 conn(0x7fdcac160ed0 legacy=0x7fdcac0d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcabfff700 1 -- 172.21.15.119:0/4151720090 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcac0cf160 con 0x7fdcac0cf960 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcabfff700 1 --2- 172.21.15.119:0/4151720090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac0cf960 0x7fdcac06e270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009d90 tx=0x7fdca400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1468fc5b9fb58cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcab7fe700 1 -- 172.21.15.119:0/4151720090 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca400e070 con 0x7fdcac0cf960 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcab7fe700 1 -- 172.21.15.119:0/4151720090 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca400bb20 con 0x7fdcac0cf960 2024-09-13T16:29:13.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.172+0000 7fdcab7fe700 1 -- 172.21.15.119:0/4151720090 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdca400be90 con 0x7fdcac0cf960 2024-09-13T16:29:13.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/4151720090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac0cf960 msgr2=0x7fdcac06e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/4151720090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac0cf960 0x7fdcac06e270 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fdca4009d90 tx=0x7fdca400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:13.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/4151720090 shutdown_connections 2024-09-13T16:29:13.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/4151720090 >> v2:172.21.15.145:3300/0 conn(0x7fdcac06e7b0 0x7fdcac06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/4151720090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac0cf960 0x7fdcac06e270 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/4151720090 >> 172.21.15.119:0/4151720090 conn(0x7fdcac099a50 msgr2=0x7fdcac099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:13.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/4151720090 shutdown_connections 2024-09-13T16:29:13.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/4151720090 wait complete. 2024-09-13T16:29:13.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 Processor -- start 2024-09-13T16:29:13.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.176+0000 7fdcb1cb3700 1 -- start start 2024-09-13T16:29:13.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac06e7b0 0x7fdcac1f6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcac0cf960 0x7fdcac1566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcac160ed0 0x7fdcac15b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac0b44c0 con 0x7fdcac160ed0 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac1f65d0 con 0x7fdcac0cf960 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcac15a480 con 0x7fdcac06e7b0 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcac160ed0 0x7fdcac15b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcac160ed0 0x7fdcac15b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58508/0 (socket says 172.21.15.119:58508) 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 -- 172.21.15.119:0/3286144323 learned_addr learned my addr 172.21.15.119:0/3286144323 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcabfff700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcac0cf960 0x7fdcac1566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb0cb1700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac06e7b0 0x7fdcac1f6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 -- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac06e7b0 msgr2=0x7fdcac1f6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac06e7b0 0x7fdcac1f6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 -- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcac0cf960 msgr2=0x7fdcac1566d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcac0cf960 0x7fdcac1566d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 -- 172.21.15.119:0/3286144323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca4009a70 con 0x7fdcac160ed0 2024-09-13T16:29:13.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb14b2700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcac160ed0 0x7fdcac15b0c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00eb60 tx=0x7fdc9c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdca97fa700 1 -- 172.21.15.119:0/3286144323 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c009740 con 0x7fdcac160ed0 2024-09-13T16:29:13.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdca97fa700 1 -- 172.21.15.119:0/3286144323 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc9c009e00 con 0x7fdcac160ed0 2024-09-13T16:29:13.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcac0b3b20 con 0x7fdcac160ed0 2024-09-13T16:29:13.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdca97fa700 1 -- 172.21.15.119:0/3286144323 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc9c00ca70 con 0x7fdcac160ed0 2024-09-13T16:29:13.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.180+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcac0b40e0 con 0x7fdcac160ed0 2024-09-13T16:29:13.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.184+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcac15bda0 con 0x7fdcac160ed0 2024-09-13T16:29:13.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.184+0000 7fdca97fa700 1 -- 172.21.15.119:0/3286144323 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fdc9c00cbd0 con 0x7fdcac160ed0 2024-09-13T16:29:13.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.184+0000 7fdca97fa700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdc90066730 0x7fdc90068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.184+0000 7fdca97fa700 1 -- 172.21.15.119:0/3286144323 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fdc9c084330 con 0x7fdcac160ed0 2024-09-13T16:29:13.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.184+0000 7fdcb0cb1700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdc90066730 0x7fdc90068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.188+0000 7fdcb0cb1700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdc90066730 0x7fdc90068bf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdc94007fd0 tx=0x7fdc94009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.196+0000 7fdca97fa700 1 -- 172.21.15.119:0/3286144323 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdc9c051440 con 0x7fdcac160ed0 2024-09-13T16:29:13.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.368+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcac0b4600 con 0x7fdcac160ed0 2024-09-13T16:29:13.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.368+0000 7fdca97fa700 1 -- 172.21.15.119:0/3286144323 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16969 (secure 0 0 0) 0x7fdcac0b4600 con 0x7fdcac160ed0 2024-09-13T16:29:13.374 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:13.374 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":70,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:08.646341+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:28:19.445072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-09-13T16:28:20.106803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.17","mappings":[{"from":7,"to":2}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:13.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdc90066730 msgr2=0x7fdc90068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdc90066730 0x7fdc90068bf0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fdc94007fd0 tx=0x7fdc94009040 comp rx=0 tx=0).stop 2024-09-13T16:29:13.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcac160ed0 msgr2=0x7fdcac15b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcac160ed0 0x7fdcac15b0c0 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7fdc9c00eb60 tx=0x7fdc9c00ee70 comp rx=0 tx=0).stop 2024-09-13T16:29:13.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 shutdown_connections 2024-09-13T16:29:13.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdc90066730 0x7fdc90068bf0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcac160ed0 0x7fdcac15b0c0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcac0cf960 0x7fdcac1566d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 --2- 172.21.15.119:0/3286144323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcac06e7b0 0x7fdcac1f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 >> 172.21.15.119:0/3286144323 conn(0x7fdcac099a50 msgr2=0x7fdcac0b0ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:13.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 shutdown_connections 2024-09-13T16:29:13.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.372+0000 7fdcb1cb3700 1 -- 172.21.15.119:0/3286144323 wait complete. 2024-09-13T16:29:13.390 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:28:19.445072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:28:20.106803+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:13.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:13.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.500+0000 7f0b7a7e0700 1 Processor -- start 2024-09-13T16:29:13.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.500+0000 7f0b7a7e0700 1 -- start start 2024-09-13T16:29:13.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0b74144180 0x7f0b7406c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0b740ce1d0 0x7f0b740b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7406cb40 con 0x7f0b740cc010 2024-09-13T16:29:13.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b740b1c50 con 0x7f0b740ce1d0 2024-09-13T16:29:13.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7406ce20 con 0x7f0b74144180 2024-09-13T16:29:13.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0b740ce1d0 0x7f0b740b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0b740ce1d0 0x7f0b740b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58514/0 (socket says 172.21.15.119:58514) 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 -- 172.21.15.119:0/4170089510 learned_addr learned my addr 172.21.15.119:0/4170089510 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b797de700 1 --2- 172.21.15.119:0/4170089510 >> v2:172.21.15.145:3300/0 conn(0x7f0b74144180 0x7f0b7406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 -- 172.21.15.119:0/4170089510 >> v2:172.21.15.145:3300/0 conn(0x7f0b74144180 msgr2=0x7f0b7406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 --2- 172.21.15.119:0/4170089510 >> v2:172.21.15.145:3300/0 conn(0x7f0b74144180 0x7f0b7406c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 -- 172.21.15.119:0/4170089510 >> v1:172.21.15.145:6789/0 conn(0x7f0b740cc010 legacy=0x7f0b740cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 -- 172.21.15.119:0/4170089510 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7406d100 con 0x7f0b740ce1d0 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b79fdf700 1 --2- 172.21.15.119:0/4170089510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b740ce1d0 0x7f0b740b20a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0b740b1560 tx=0x7f0b6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ebc34e8517af7f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b67fff700 1 -- 172.21.15.119:0/4170089510 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c00e070 con 0x7f0b740ce1d0 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b67fff700 1 -- 172.21.15.119:0/4170089510 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c00b9c0 con 0x7f0b740ce1d0 2024-09-13T16:29:13.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b67fff700 1 -- 172.21.15.119:0/4170089510 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c00bd70 con 0x7f0b740ce1d0 2024-09-13T16:29:13.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/4170089510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b740ce1d0 msgr2=0x7f0b740b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/4170089510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b740ce1d0 0x7f0b740b20a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0b740b1560 tx=0x7f0b6c00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:13.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/4170089510 shutdown_connections 2024-09-13T16:29:13.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/4170089510 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b740ce1d0 0x7f0b740b20a0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/4170089510 >> v2:172.21.15.145:3300/0 conn(0x7f0b74144180 0x7f0b7406c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/4170089510 >> 172.21.15.119:0/4170089510 conn(0x7f0b740b0980 msgr2=0x7f0b740b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:13.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/4170089510 shutdown_connections 2024-09-13T16:29:13.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.504+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/4170089510 wait complete. 2024-09-13T16:29:13.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 Processor -- start 2024-09-13T16:29:13.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 -- start start 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b740cc010 0x7f0b7413e460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b740ce1d0 0x7f0b7413da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b74144180 0x7f0b741f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740ca4e0 con 0x7f0b74144180 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740aa230 con 0x7f0b740cc010 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b7a7e0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740acc30 con 0x7f0b740ce1d0 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b79fdf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b74144180 0x7f0b741f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b79fdf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b74144180 0x7f0b741f0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58528/0 (socket says 172.21.15.119:58528) 2024-09-13T16:29:13.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.508+0000 7f0b79fdf700 1 -- 172.21.15.119:0/3261731893 learned_addr learned my addr 172.21.15.119:0/3261731893 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:13.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b78fdd700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b740ce1d0 0x7f0b7413da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b797de700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b740cc010 0x7f0b7413e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b79fdf700 1 -- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b740ce1d0 msgr2=0x7f0b7413da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b79fdf700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b740ce1d0 0x7f0b7413da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b79fdf700 1 -- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b740cc010 msgr2=0x7f0b7413e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b79fdf700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b740cc010 0x7f0b7413e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b79fdf700 1 -- 172.21.15.119:0/3261731893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6c009b30 con 0x7f0b74144180 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b79fdf700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b74144180 0x7f0b741f0540 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0b7406e010 tx=0x7f0b6c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b65ffb700 1 -- 172.21.15.119:0/3261731893 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c004880 con 0x7f0b74144180 2024-09-13T16:29:13.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b65ffb700 1 -- 172.21.15.119:0/3261731893 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6c0099b0 con 0x7f0b74144180 2024-09-13T16:29:13.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b740a9b70 con 0x7f0b74144180 2024-09-13T16:29:13.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b65ffb700 1 -- 172.21.15.119:0/3261731893 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6c01c680 con 0x7f0b74144180 2024-09-13T16:29:13.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b741f0a80 con 0x7f0b74144180 2024-09-13T16:29:13.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b740aa370 con 0x7f0b74144180 2024-09-13T16:29:13.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.512+0000 7f0b65ffb700 1 -- 172.21.15.119:0/3261731893 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0b6c01c7e0 con 0x7f0b74144180 2024-09-13T16:29:13.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.516+0000 7f0b65ffb700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b58066730 0x7f0b58068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.516+0000 7f0b65ffb700 1 -- 172.21.15.119:0/3261731893 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f0b6c085e40 con 0x7f0b74144180 2024-09-13T16:29:13.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.516+0000 7f0b797de700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b58066730 0x7f0b58068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.516+0000 7f0b797de700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b58066730 0x7f0b58068bf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0b68007a60 tx=0x7f0b68008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.520+0000 7f0b65ffb700 1 -- 172.21.15.119:0/3261731893 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0b740aa370 con 0x7f0b74144180 2024-09-13T16:29:13.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.692+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b740aa370 con 0x7f0b74144180 2024-09-13T16:29:13.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.692+0000 7f0b65ffb700 1 -- 172.21.15.119:0/3261731893 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b740aa370 con 0x7f0b74144180 2024-09-13T16:29:13.699 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b58066730 msgr2=0x7f0b58068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b58066730 0x7f0b58068bf0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0b68007a60 tx=0x7f0b68008040 comp rx=0 tx=0).stop 2024-09-13T16:29:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b74144180 msgr2=0x7f0b741f0540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b74144180 0x7f0b741f0540 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0b7406e010 tx=0x7f0b6c004af0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 shutdown_connections 2024-09-13T16:29:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0b58066730 0x7f0b58068bf0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0b74144180 0x7f0b741f0540 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0b740ce1d0 0x7f0b7413da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 --2- 172.21.15.119:0/3261731893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0b740cc010 0x7f0b7413e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 >> 172.21.15.119:0/3261731893 conn(0x7f0b740b0980 msgr2=0x7f0b74066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 shutdown_connections 2024-09-13T16:29:13.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.696+0000 7f0b7a7e0700 1 -- 172.21.15.119:0/3261731893 wait complete. 2024-09-13T16:29:13.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:29:13.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a14cd700 1 Processor -- start 2024-09-13T16:29:13.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a14cd700 1 -- start start 2024-09-13T16:29:13.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a14cd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe19c0ce8c0 0x7fe19c0d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a14cd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c0d1610 0x7fe19c0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a14cd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe19c0d1a60 con 0x7fe19c15cc00 2024-09-13T16:29:13.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a14cd700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe19c0d0df0 con 0x7fe19c0ce8c0 2024-09-13T16:29:13.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a14cd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0cdb00 con 0x7fe19c0d1610 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe19c15cc00 0x7fe19c0d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58892/0 (socket says 172.21.15.119:58892) 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19bfff700 1 -- 172.21.15.119:0/806523497 learned_addr learned my addr 172.21.15.119:0/806523497 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19b7fe700 1 --2- 172.21.15.119:0/806523497 >> v2:172.21.15.145:3300/0 conn(0x7fe19c0ce8c0 0x7fe19c0d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe1a0ccc700 1 --2- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c0d1610 0x7fe19c0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19affd700 1 -- 172.21.15.119:0/806523497 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080173552 0 0) 0x7fe18c001110 con 0x7fe19c15cc00 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19affd700 1 -- 172.21.15.119:0/806523497 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1880035d0 con 0x7fe19c15cc00 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19b7fe700 1 -- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c0d1610 msgr2=0x7fe19c0b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19b7fe700 1 --2- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c0d1610 0x7fe19c0b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19b7fe700 1 -- 172.21.15.119:0/806523497 >> v1:172.21.15.119:6789/0 conn(0x7fe19c15cc00 legacy=0x7fe19c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19b7fe700 1 -- 172.21.15.119:0/806523497 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c0cdde0 con 0x7fe19c0ce8c0 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19b7fe700 1 --2- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 0x7fe19c0d10d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe184009850 tx=0x7fe184009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f89866afd8c0b1e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19affd700 1 -- 172.21.15.119:0/806523497 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184011830 con 0x7fe19c0ce8c0 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19affd700 1 -- 172.21.15.119:0/806523497 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe184011990 con 0x7fe19c0ce8c0 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.824+0000 7fe19affd700 1 -- 172.21.15.119:0/806523497 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184011d40 con 0x7fe19c0ce8c0 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 -- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 msgr2=0x7fe19c0d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 0x7fe19c0d10d0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fe184009850 tx=0x7fe184009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 -- 172.21.15.119:0/806523497 shutdown_connections 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c0d1610 0x7fe19c0b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/806523497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 0x7fe19c0d10d0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 -- 172.21.15.119:0/806523497 >> 172.21.15.119:0/806523497 conn(0x7fe19c08c080 msgr2=0x7fe19c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 -- 172.21.15.119:0/806523497 shutdown_connections 2024-09-13T16:29:13.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 -- 172.21.15.119:0/806523497 wait complete. 2024-09-13T16:29:13.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 Processor -- start 2024-09-13T16:29:13.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 -- start start 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 0x7fe19c155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.828+0000 7fe1a14cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe19c0d1610 0x7fe19c157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a14cd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c15cc00 0x7fe19c1e8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a14cd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c1579a0 con 0x7fe19c0d1610 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a14cd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c157c80 con 0x7fe19c0ce8c0 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a14cd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe19c0cb090 con 0x7fe19c15cc00 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe19c0d1610 0x7fe19c157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe19c0d1610 0x7fe19c157310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58542/0 (socket says 172.21.15.119:58542) 2024-09-13T16:29:13.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 -- 172.21.15.119:0/2689716176 learned_addr learned my addr 172.21.15.119:0/2689716176 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19bfff700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 0x7fe19c155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a0ccc700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c15cc00 0x7fe19c1e8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 -- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c15cc00 msgr2=0x7fe19c1e8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c15cc00 0x7fe19c1e8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 -- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 msgr2=0x7fe19c155d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 0x7fe19c155d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 -- 172.21.15.119:0/2689716176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1840094b0 con 0x7fe19c0d1610 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a0ccc700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c15cc00 0x7fe19c1e8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe19b7fe700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe19c0d1610 0x7fe19c157310 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe1840090b0 tx=0x7fe184004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe198ff9700 1 -- 172.21.15.119:0/2689716176 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe184004380 con 0x7fe19c0d1610 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe198ff9700 1 -- 172.21.15.119:0/2689716176 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe18401f070 con 0x7fe19c0d1610 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe19c0cb370 con 0x7fe19c0d1610 2024-09-13T16:29:13.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe198ff9700 1 -- 172.21.15.119:0/2689716176 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1840036c0 con 0x7fe19c0d1610 2024-09-13T16:29:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe19c1570a0 con 0x7fe19c0d1610 2024-09-13T16:29:13.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.832+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe19c0cb890 con 0x7fe19c0d1610 2024-09-13T16:29:13.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.836+0000 7fe198ff9700 1 -- 172.21.15.119:0/2689716176 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe184021030 con 0x7fe19c0d1610 2024-09-13T16:29:13.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.836+0000 7fe198ff9700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe17c066730 0x7fe17c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:13.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.836+0000 7fe198ff9700 1 -- 172.21.15.119:0/2689716176 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fe184085790 con 0x7fe19c0d1610 2024-09-13T16:29:13.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.836+0000 7fe19bfff700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe17c066730 0x7fe17c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:13.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.836+0000 7fe19bfff700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe17c066730 0x7fe17c068bf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe18c007fd0 tx=0x7fe18c007c60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:13.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:13.840+0000 7fe198ff9700 1 -- 172.21.15.119:0/2689716176 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe19c0cb890 con 0x7fe19c0d1610 2024-09-13T16:29:14.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.012+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe19c0cb890 con 0x7fe19c0d1610 2024-09-13T16:29:14.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.012+0000 7fe198ff9700 1 -- 172.21.15.119:0/2689716176 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe19c0cb890 con 0x7fe19c0d1610 2024-09-13T16:29:14.018 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe17c066730 msgr2=0x7fe17c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe17c066730 0x7fe17c068bf0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fe18c007fd0 tx=0x7fe18c007c60 comp rx=0 tx=0).stop 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe19c0d1610 msgr2=0x7fe19c157310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe19c0d1610 0x7fe19c157310 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7fe1840090b0 tx=0x7fe184004770 comp rx=0 tx=0).stop 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 shutdown_connections 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe17c066730 0x7fe17c068bf0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe19c15cc00 0x7fe19c1e8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe19c0d1610 0x7fe19c157310 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 --2- 172.21.15.119:0/2689716176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe19c0ce8c0 0x7fe19c155d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 >> 172.21.15.119:0/2689716176 conn(0x7fe19c08c080 msgr2=0x7fe19c06c390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:14.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 shutdown_connections 2024-09-13T16:29:14.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.016+0000 7fe1a14cd700 1 -- 172.21.15.119:0/2689716176 wait complete. 2024-09-13T16:29:14.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:29:14.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.148+0000 7fe6317c8700 1 Processor -- start 2024-09-13T16:29:14.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.148+0000 7fe6317c8700 1 -- start start 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c15cc00 0x7fe62c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe62c0d17e0 0x7fe62c0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62c0cdb00 con 0x7fe62c0d0c20 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe62c0cdde0 con 0x7fe62c0d17e0 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c0ce0c0 con 0x7fe62c15cc00 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe62c0d17e0 0x7fe62c0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe62c0d17e0 0x7fe62c0b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58548/0 (socket says 172.21.15.119:58548) 2024-09-13T16:29:14.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 -- 172.21.15.119:0/81571909 learned_addr learned my addr 172.21.15.119:0/81571909 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 -- 172.21.15.119:0/81571909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c15cc00 msgr2=0x7fe62c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe62bfff700 1 --2- 172.21.15.119:0/81571909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c15cc00 0x7fe62c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 --2- 172.21.15.119:0/81571909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c15cc00 0x7fe62c0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 -- 172.21.15.119:0/81571909 >> v1:172.21.15.145:6789/0 conn(0x7fe62c0d0c20 legacy=0x7fe62c0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 -- 172.21.15.119:0/81571909 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62c0ce3a0 con 0x7fe62c0d17e0 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe62bfff700 1 --2- 172.21.15.119:0/81571909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c15cc00 0x7fe62c0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe630fc7700 1 --2- 172.21.15.119:0/81571909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c0d17e0 0x7fe62c0b5960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe624009e50 tx=0x7fe62400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80ca2e2ab050f31 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe62affd700 1 -- 172.21.15.119:0/81571909 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe62400e070 con 0x7fe62c0d17e0 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe62affd700 1 -- 172.21.15.119:0/81571909 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624003a40 con 0x7fe62c0d17e0 2024-09-13T16:29:14.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe62affd700 1 -- 172.21.15.119:0/81571909 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624003df0 con 0x7fe62c0d17e0 2024-09-13T16:29:14.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- 172.21.15.119:0/81571909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c0d17e0 msgr2=0x7fe62c0b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 --2- 172.21.15.119:0/81571909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c0d17e0 0x7fe62c0b5960 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7fe624009e50 tx=0x7fe62400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:14.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- 172.21.15.119:0/81571909 shutdown_connections 2024-09-13T16:29:14.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 --2- 172.21.15.119:0/81571909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c0d17e0 0x7fe62c0b5960 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 --2- 172.21.15.119:0/81571909 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c15cc00 0x7fe62c0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- 172.21.15.119:0/81571909 >> 172.21.15.119:0/81571909 conn(0x7fe62c08c080 msgr2=0x7fe62c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- 172.21.15.119:0/81571909 shutdown_connections 2024-09-13T16:29:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.152+0000 7fe6317c8700 1 -- 172.21.15.119:0/81571909 wait complete. 2024-09-13T16:29:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 Processor -- start 2024-09-13T16:29:14.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 -- start start 2024-09-13T16:29:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe62c0d0c20 0x7fe62c158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c0d17e0 0x7fe62c156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c15cc00 0x7fe62c0c67b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c1574c0 con 0x7fe62c15cc00 2024-09-13T16:29:14.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c1577a0 con 0x7fe62c0d0c20 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62c157a80 con 0x7fe62c0d17e0 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c15cc00 0x7fe62c0c67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c15cc00 0x7fe62c0c67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58550/0 (socket says 172.21.15.119:58550) 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 -- 172.21.15.119:0/103888813 learned_addr learned my addr 172.21.15.119:0/103888813 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe62bfff700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe62c0d0c20 0x7fe62c158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 -- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c0d17e0 msgr2=0x7fe62c156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c0d17e0 0x7fe62c156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 -- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe62c0d0c20 msgr2=0x7fe62c158560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe62c0d0c20 0x7fe62c158560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 -- 172.21.15.119:0/103888813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624009b30 con 0x7fe62c15cc00 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe62bfff700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe62c0d0c20 0x7fe62c158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:14.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe630fc7700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c15cc00 0x7fe62c0c67b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe62c08bdf0 tx=0x7fe624004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:14.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe628ff9700 1 -- 172.21.15.119:0/103888813 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624004440 con 0x7fe62c15cc00 2024-09-13T16:29:14.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62c157d60 con 0x7fe62c15cc00 2024-09-13T16:29:14.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe628ff9700 1 -- 172.21.15.119:0/103888813 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6240045a0 con 0x7fe62c15cc00 2024-09-13T16:29:14.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62c156750 con 0x7fe62c15cc00 2024-09-13T16:29:14.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.156+0000 7fe628ff9700 1 -- 172.21.15.119:0/103888813 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624004910 con 0x7fe62c15cc00 2024-09-13T16:29:14.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.160+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62c158220 con 0x7fe62c15cc00 2024-09-13T16:29:14.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.160+0000 7fe628ff9700 1 -- 172.21.15.119:0/103888813 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe624019030 con 0x7fe62c15cc00 2024-09-13T16:29:14.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.160+0000 7fe628ff9700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618066730 0x7fe618068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.160+0000 7fe628ff9700 1 -- 172.21.15.119:0/103888813 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fe6240852f0 con 0x7fe62c15cc00 2024-09-13T16:29:14.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.160+0000 7fe62bfff700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618066730 0x7fe618068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.160+0000 7fe62bfff700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618066730 0x7fe618068bf0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe61c008930 tx=0x7fe61c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:14.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.168+0000 7fe628ff9700 1 -- 172.21.15.119:0/103888813 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe6240524b0 con 0x7fe62c15cc00 2024-09-13T16:29:14.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.336+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe62c0ce8c0 con 0x7fe62c15cc00 2024-09-13T16:29:14.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.336+0000 7fe628ff9700 1 -- 172.21.15.119:0/103888813 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe624055ad0 con 0x7fe62c15cc00 2024-09-13T16:29:14.342 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:29:14.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618066730 msgr2=0x7fe618068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618066730 0x7fe618068bf0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fe61c008930 tx=0x7fe61c008040 comp rx=0 tx=0).stop 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c15cc00 msgr2=0x7fe62c0c67b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c15cc00 0x7fe62c0c67b0 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7fe62c08bdf0 tx=0x7fe624004af0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 shutdown_connections 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe618066730 0x7fe618068bf0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe62c15cc00 0x7fe62c0c67b0 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe62c0d17e0 0x7fe62c156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 --2- 172.21.15.119:0/103888813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe62c0d0c20 0x7fe62c158560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 >> 172.21.15.119:0/103888813 conn(0x7fe62c08c080 msgr2=0x7fe62c0b6f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 shutdown_connections 2024-09-13T16:29:14.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.340+0000 7fe6317c8700 1 -- 172.21.15.119:0/103888813 wait complete. 2024-09-13T16:29:14.357 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.468+0000 7f9f997f9700 1 Processor -- start 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.468+0000 7f9f997f9700 1 -- start start 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f997f9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9f9415ce70 0x7f9f940d37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f997f9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f941607c0 0x7f9f94160c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f997f9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f940d1190 con 0x7f9f940d1620 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f997f9700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f94161190 con 0x7f9f9415ce70 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f997f9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f940ceaf0 con 0x7f9f941607c0 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f937fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f941607c0 0x7f9f94160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f8a7fc700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f9f940d1620 0x7f9f940ce2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:55104/0 (socket says 172.21.15.119:55104) 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f8a7fc700 1 -- 172.21.15.119:0/1957346326 learned_addr learned my addr 172.21.15.119:0/1957346326 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f92ffd700 1 --2- 172.21.15.119:0/1957346326 >> v2:172.21.15.145:3300/0 conn(0x7f9f9415ce70 0x7f9f940d37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f937fe700 1 -- 172.21.15.119:0/1957346326 >> v2:172.21.15.145:3300/0 conn(0x7f9f9415ce70 msgr2=0x7f9f940d37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f927fc700 1 -- 172.21.15.119:0/1957346326 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 696914710 0 0) 0x7f9f940d1190 con 0x7f9f940d1620 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f937fe700 1 --2- 172.21.15.119:0/1957346326 >> v2:172.21.15.145:3300/0 conn(0x7f9f9415ce70 0x7f9f940d37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f937fe700 1 -- 172.21.15.119:0/1957346326 >> v1:172.21.15.145:6789/0 conn(0x7f9f940d1620 legacy=0x7f9f940ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f937fe700 1 -- 172.21.15.119:0/1957346326 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f940d0860 con 0x7f9f941607c0 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f937fe700 1 --2- 172.21.15.119:0/1957346326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f941607c0 0x7f9f94160c50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009d90 tx=0x7f9f8c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87c650c1baaa3d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f927fc700 1 -- 172.21.15.119:0/1957346326 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c00e070 con 0x7f9f941607c0 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f927fc700 1 -- 172.21.15.119:0/1957346326 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c00bb20 con 0x7f9f941607c0 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f927fc700 1 -- 172.21.15.119:0/1957346326 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c012430 con 0x7f9f941607c0 2024-09-13T16:29:14.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f997f9700 1 -- 172.21.15.119:0/1957346326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f941607c0 msgr2=0x7f9f94160c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.472+0000 7f9f997f9700 1 --2- 172.21.15.119:0/1957346326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f941607c0 0x7f9f94160c50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9f8c009d90 tx=0x7f9f8c00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 -- 172.21.15.119:0/1957346326 shutdown_connections 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 --2- 172.21.15.119:0/1957346326 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f941607c0 0x7f9f94160c50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 --2- 172.21.15.119:0/1957346326 >> v2:172.21.15.145:3300/0 conn(0x7f9f9415ce70 0x7f9f940d37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 -- 172.21.15.119:0/1957346326 >> 172.21.15.119:0/1957346326 conn(0x7f9f940d2bf0 msgr2=0x7f9f940d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 -- 172.21.15.119:0/1957346326 shutdown_connections 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 -- 172.21.15.119:0/1957346326 wait complete. 2024-09-13T16:29:14.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 Processor -- start 2024-09-13T16:29:14.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.476+0000 7f9f997f9700 1 -- start start 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f940d1620 0x7f9f9415fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f9415ce70 0x7f9f941580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f941607c0 0x7f9f940cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9415f790 con 0x7f9f941607c0 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9415e8c0 con 0x7f9f940d1620 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f94158600 con 0x7f9f9415ce70 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f941607c0 0x7f9f940cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f941607c0 0x7f9f940cc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58556/0 (socket says 172.21.15.119:58556) 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 -- 172.21.15.119:0/4279582730 learned_addr learned my addr 172.21.15.119:0/4279582730 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f92ffd700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f940d1620 0x7f9f9415fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f8a7fc700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f9415ce70 0x7f9f941580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 -- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f9415ce70 msgr2=0x7f9f941580c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f9415ce70 0x7f9f941580c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 -- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f940d1620 msgr2=0x7f9f9415fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f940d1620 0x7f9f9415fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 -- 172.21.15.119:0/4279582730 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c009a70 con 0x7f9f941607c0 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f8a7fc700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f9415ce70 0x7f9f941580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f92ffd700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f940d1620 0x7f9f9415fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f937fe700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f941607c0 0x7f9f940cc520 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9f940cec30 tx=0x7f9f8c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:14.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f8bfff700 1 -- 172.21.15.119:0/4279582730 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c027480 con 0x7f9f941607c0 2024-09-13T16:29:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9415eb10 con 0x7f9f941607c0 2024-09-13T16:29:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f8bfff700 1 -- 172.21.15.119:0/4279582730 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c020070 con 0x7f9f941607c0 2024-09-13T16:29:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9415f0d0 con 0x7f9f941607c0 2024-09-13T16:29:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f8bfff700 1 -- 172.21.15.119:0/4279582730 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f8c0277e0 con 0x7f9f941607c0 2024-09-13T16:29:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.480+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9415f8d0 con 0x7f9f941607c0 2024-09-13T16:29:14.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.484+0000 7f9f8bfff700 1 -- 172.21.15.119:0/4279582730 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9f8c027940 con 0x7f9f941607c0 2024-09-13T16:29:14.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.484+0000 7f9f8bfff700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9f74066760 0x7f9f74068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:14.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.484+0000 7f9f8bfff700 1 -- 172.21.15.119:0/4279582730 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f9f8c022070 con 0x7f9f941607c0 2024-09-13T16:29:14.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.484+0000 7f9f92ffd700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9f74066760 0x7f9f74068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:14.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.484+0000 7f9f92ffd700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9f74066760 0x7f9f74068c20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009c40 tx=0x7f9f84008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:14.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.488+0000 7f9f8bfff700 1 -- 172.21.15.119:0/4279582730 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9f8c053220 con 0x7f9f941607c0 2024-09-13T16:29:14.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f9f940d37a0 con 0x7f9f941607c0 2024-09-13T16:29:14.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f8bfff700 1 -- 172.21.15.119:0/4279582730 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v70) v1 ==== 80+0+0 (secure 0 0 0) 0x7f9f8c056840 con 0x7f9f941607c0 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9f74066760 msgr2=0x7f9f74068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9f74066760 0x7f9f74068c20 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f9f84009c40 tx=0x7f9f84008040 comp rx=0 tx=0).stop 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f941607c0 msgr2=0x7f9f940cc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f941607c0 0x7f9f940cc520 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f9f940cec30 tx=0x7f9f8c004af0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 shutdown_connections 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9f74066760 0x7f9f74068c20 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9f941607c0 0x7f9f940cc520 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9f9415ce70 0x7f9f941580c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 --2- 172.21.15.119:0/4279582730 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9f940d1620 0x7f9f9415fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 >> 172.21.15.119:0/4279582730 conn(0x7f9f940d2bf0 msgr2=0x7f9f940cf000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 shutdown_connections 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:14.672+0000 7f9f997f9700 1 -- 172.21.15.119:0/4279582730 wait complete. 2024-09-13T16:29:14.681 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:29:14.691 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:29:14.691 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:14.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:14.700 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[42159]: fuse finished with error 0 and tester_r 0 2024-09-13T16:29:14.701 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:29:14.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:14.701 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:29:14.790 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:29:14.790 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:14.796 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:29:14.797 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:29:14.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:14.797 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:29:14.802 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-13T16:29:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:29:14.803 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:14.803 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:14.811 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:29:14.811 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:29:14.811 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:14.811 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:29:14.861 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-09-13T16:29:14.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:29:14.862 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:29:14.862 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:14.870 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:29:14.870 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:29:14.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:14.870 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:29:14.920 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-13T16:29:14.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:29:14.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:15.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.032+0000 7fd048ee0700 1 Processor -- start 2024-09-13T16:29:15.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.032+0000 7fd048ee0700 1 -- start start 2024-09-13T16:29:15.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd0440cd8d0 0x7fd04406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0440b17f0 0x7fd0440cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b2200 0x7fd0440b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0440b2bb0 con 0x7fd0440b17f0 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0440705d0 con 0x7fd0440cd8d0 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd04406d0d0 con 0x7fd0440b2200 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0440b17f0 0x7fd0440cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0440b17f0 0x7fd0440cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58570/0 (socket says 172.21.15.119:58570) 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 -- 172.21.15.119:0/4216767812 learned_addr learned my addr 172.21.15.119:0/4216767812 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd043fff700 1 --2- 172.21.15.119:0/4216767812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b2200 0x7fd0440b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd0437fe700 1 --2- 172.21.15.119:0/4216767812 >> v2:172.21.15.145:3300/0 conn(0x7fd0440cd8d0 0x7fd04406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 -- 172.21.15.119:0/4216767812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b2200 msgr2=0x7fd0440b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 --2- 172.21.15.119:0/4216767812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b2200 0x7fd0440b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 -- 172.21.15.119:0/4216767812 >> v2:172.21.15.145:3300/0 conn(0x7fd0440cd8d0 msgr2=0x7fd04406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 --2- 172.21.15.119:0/4216767812 >> v2:172.21.15.145:3300/0 conn(0x7fd0440cd8d0 0x7fd04406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 -- 172.21.15.119:0/4216767812 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd04406fca0 con 0x7fd0440b17f0 2024-09-13T16:29:15.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd043fff700 1 --2- 172.21.15.119:0/4216767812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b2200 0x7fd0440b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd042ffd700 1 --2- 172.21.15.119:0/4216767812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440b17f0 0x7fd0440cb800 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd044071850 tx=0x7fd02c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5387a93c90d23d44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd0427fc700 1 -- 172.21.15.119:0/4216767812 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00b8a0 con 0x7fd0440b17f0 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd0427fc700 1 -- 172.21.15.119:0/4216767812 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c00ba00 con 0x7fd0440b17f0 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd0427fc700 1 -- 172.21.15.119:0/4216767812 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00bd90 con 0x7fd0440b17f0 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- 172.21.15.119:0/4216767812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440b17f0 msgr2=0x7fd0440cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 --2- 172.21.15.119:0/4216767812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440b17f0 0x7fd0440cb800 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7fd044071850 tx=0x7fd02c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- 172.21.15.119:0/4216767812 shutdown_connections 2024-09-13T16:29:15.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 --2- 172.21.15.119:0/4216767812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b2200 0x7fd0440b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 --2- 172.21.15.119:0/4216767812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440b17f0 0x7fd0440cb800 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 --2- 172.21.15.119:0/4216767812 >> v2:172.21.15.145:3300/0 conn(0x7fd0440cd8d0 0x7fd04406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- 172.21.15.119:0/4216767812 >> 172.21.15.119:0/4216767812 conn(0x7fd04406a350 msgr2=0x7fd04406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:15.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- 172.21.15.119:0/4216767812 shutdown_connections 2024-09-13T16:29:15.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.036+0000 7fd048ee0700 1 -- 172.21.15.119:0/4216767812 wait complete. 2024-09-13T16:29:15.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 Processor -- start 2024-09-13T16:29:15.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 -- start start 2024-09-13T16:29:15.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b17f0 0x7fd0440a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0440b2200 0x7fd0440acb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440cd8d0 0x7fd0441f9f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0440a89f0 con 0x7fd0440cd8d0 2024-09-13T16:29:15.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0440a8cc0 con 0x7fd0440b2200 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd048ee0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0440a8fa0 con 0x7fd0440b17f0 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd043fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440cd8d0 0x7fd0441f9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd043fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440cd8d0 0x7fd0441f9f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58580/0 (socket says 172.21.15.119:58580) 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd043fff700 1 -- 172.21.15.119:0/1876525913 learned_addr learned my addr 172.21.15.119:0/1876525913 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd042ffd700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0440b2200 0x7fd0440acb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.040+0000 7fd0437fe700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b17f0 0x7fd0440a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd043fff700 1 -- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b17f0 msgr2=0x7fd0440a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd043fff700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b17f0 0x7fd0440a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd043fff700 1 -- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0440b2200 msgr2=0x7fd0440acb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd043fff700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0440b2200 0x7fd0440acb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd043fff700 1 -- 172.21.15.119:0/1876525913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c009570 con 0x7fd0440cd8d0 2024-09-13T16:29:15.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd043fff700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440cd8d0 0x7fd0441f9f60 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009fa0 tx=0x7fd03c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd03bfff700 1 -- 172.21.15.119:0/1876525913 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c00ddb0 con 0x7fd0440cd8d0 2024-09-13T16:29:15.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd03bfff700 1 -- 172.21.15.119:0/1876525913 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd03c00f040 con 0x7fd0440cd8d0 2024-09-13T16:29:15.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0440a9280 con 0x7fd0440cd8d0 2024-09-13T16:29:15.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd03bfff700 1 -- 172.21.15.119:0/1876525913 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd03c003c80 con 0x7fd0440cd8d0 2024-09-13T16:29:15.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0440a9840 con 0x7fd0440cd8d0 2024-09-13T16:29:15.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0440cdcb0 con 0x7fd0440cd8d0 2024-09-13T16:29:15.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.044+0000 7fd03bfff700 1 -- 172.21.15.119:0/1876525913 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd03c004030 con 0x7fd0440cd8d0 2024-09-13T16:29:15.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.048+0000 7fd03bfff700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd024066730 0x7fd024068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.048+0000 7fd03bfff700 1 -- 172.21.15.119:0/1876525913 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fd03c0843b0 con 0x7fd0440cd8d0 2024-09-13T16:29:15.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.048+0000 7fd0437fe700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd024066730 0x7fd024068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.048+0000 7fd0437fe700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd024066730 0x7fd024068bf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd0340079e0 tx=0x7fd034008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.052+0000 7fd03bfff700 1 -- 172.21.15.119:0/1876525913 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd03c051560 con 0x7fd0440cd8d0 2024-09-13T16:29:15.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.256+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0440a9a30 con 0x7fd0440cd8d0 2024-09-13T16:29:15.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.256+0000 7fd03bfff700 1 -- 172.21.15.119:0/1876525913 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fd03c054b80 con 0x7fd0440cd8d0 2024-09-13T16:29:15.262 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:15.262 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:29:09.657258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":15,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:29:15.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd024066730 msgr2=0x7fd024068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd024066730 0x7fd024068bf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fd0340079e0 tx=0x7fd034008040 comp rx=0 tx=0).stop 2024-09-13T16:29:15.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440cd8d0 msgr2=0x7fd0441f9f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440cd8d0 0x7fd0441f9f60 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009fa0 tx=0x7fd03c00d3b0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 shutdown_connections 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd024066730 0x7fd024068bf0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0440cd8d0 0x7fd0441f9f60 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0440b2200 0x7fd0440acb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 --2- 172.21.15.119:0/1876525913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0440b17f0 0x7fd0440a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 >> 172.21.15.119:0/1876525913 conn(0x7fd04406a350 msgr2=0x7fd0440b0b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 shutdown_connections 2024-09-13T16:29:15.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.260+0000 7fd048ee0700 1 -- 172.21.15.119:0/1876525913 wait complete. 2024-09-13T16:29:15.268 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 34 2024-09-13T16:29:15.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:15.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 Processor -- start 2024-09-13T16:29:15.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- start start 2024-09-13T16:29:15.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f53280ceb90 0x7f532806efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f532806f810 0x7f53280d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f532815ae90 con 0x7f532815aab0 2024-09-13T16:29:15.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f532806fe80 con 0x7f53280ceb90 2024-09-13T16:29:15.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532806e210 con 0x7f532806f810 2024-09-13T16:29:15.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f53277fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f532815aab0 0x7f53280ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58908/0 (socket says 172.21.15.119:58908) 2024-09-13T16:29:15.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f53277fe700 1 -- 172.21.15.119:0/75299485 learned_addr learned my addr 172.21.15.119:0/75299485 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:15.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f531effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f53280ceb90 0x7f532806efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f5327fff700 1 --2- 172.21.15.119:0/75299485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f532806f810 0x7f53280d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f5326ffd700 1 -- 172.21.15.119:0/75299485 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848092966 0 0) 0x7f532815ae90 con 0x7f532815aab0 2024-09-13T16:29:15.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f5326ffd700 1 -- 172.21.15.119:0/75299485 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53140035d0 con 0x7f532815aab0 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f531effd700 1 -- 172.21.15.119:0/75299485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f532806f810 msgr2=0x7f53280d0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f531effd700 1 --2- 172.21.15.119:0/75299485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f532806f810 0x7f53280d0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f531effd700 1 -- 172.21.15.119:0/75299485 >> v1:172.21.15.119:6789/0 conn(0x7f532815aab0 legacy=0x7f53280ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f531effd700 1 -- 172.21.15.119:0/75299485 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532806e4f0 con 0x7f53280ceb90 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f531effd700 1 --2- 172.21.15.119:0/75299485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53280ceb90 0x7f532806efd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f5310009850 tx=0x7f5310009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b162d3c658f28e06 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f5326ffd700 1 -- 172.21.15.119:0/75299485 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310011830 con 0x7f53280ceb90 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f5326ffd700 1 -- 172.21.15.119:0/75299485 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5310011990 con 0x7f53280ceb90 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f5326ffd700 1 -- 172.21.15.119:0/75299485 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5310011d40 con 0x7f53280ceb90 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- 172.21.15.119:0/75299485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53280ceb90 msgr2=0x7f532806efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 --2- 172.21.15.119:0/75299485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53280ceb90 0x7f532806efd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f5310009850 tx=0x7f5310009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:15.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- 172.21.15.119:0/75299485 shutdown_connections 2024-09-13T16:29:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 --2- 172.21.15.119:0/75299485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f532806f810 0x7f53280d0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 --2- 172.21.15.119:0/75299485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53280ceb90 0x7f532806efd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- 172.21.15.119:0/75299485 >> 172.21.15.119:0/75299485 conn(0x7f5328099a50 msgr2=0x7f5328099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- 172.21.15.119:0/75299485 shutdown_connections 2024-09-13T16:29:15.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.388+0000 7f532ca09700 1 -- 172.21.15.119:0/75299485 wait complete. 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 Processor -- start 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 -- start start 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f532806f810 0x7f53280b0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53280ceb90 0x7f5328152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f532815aab0 0x7f53280cba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53280cb4f0 con 0x7f532806f810 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53280b1110 con 0x7f532815aab0 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53280b13f0 con 0x7f53280ceb90 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f532806f810 0x7f53280b0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f532806f810 0x7f53280b0ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58592/0 (socket says 172.21.15.119:58592) 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 -- 172.21.15.119:0/3917218851 learned_addr learned my addr 172.21.15.119:0/3917218851 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:15.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f5327fff700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f532815aab0 0x7f53280cba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f531effd700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53280ceb90 0x7f5328152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 -- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53280ceb90 msgr2=0x7f5328152ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53280ceb90 0x7f5328152ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 -- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f532815aab0 msgr2=0x7f53280cba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f532815aab0 0x7f53280cba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 -- 172.21.15.119:0/3917218851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53100094b0 con 0x7f532806f810 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f531effd700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53280ceb90 0x7f5328152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f53277fe700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f532806f810 0x7f53280b0ae0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f531800c4d0 tx=0x7f531800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f5324ff9700 1 -- 172.21.15.119:0/3917218851 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318005d90 con 0x7f532806f810 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f5324ff9700 1 -- 172.21.15.119:0/3917218851 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53180070d0 con 0x7f532806f810 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f5324ff9700 1 -- 172.21.15.119:0/3917218851 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5318007480 con 0x7f532806f810 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.392+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53280b16d0 con 0x7f532806f810 2024-09-13T16:29:15.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.396+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53280b1c90 con 0x7f532806f810 2024-09-13T16:29:15.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.396+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532806efd0 con 0x7f532806f810 2024-09-13T16:29:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.396+0000 7f5324ff9700 1 -- 172.21.15.119:0/3917218851 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f53180075e0 con 0x7f532806f810 2024-09-13T16:29:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.396+0000 7f5324ff9700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5308066730 0x7f5308068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.396+0000 7f5324ff9700 1 -- 172.21.15.119:0/3917218851 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f53180837e0 con 0x7f532806f810 2024-09-13T16:29:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.396+0000 7f531effd700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5308066730 0x7f5308068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.396+0000 7f531effd700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5308066730 0x7f5308068bf0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f53100090b0 tx=0x7f5310008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.404+0000 7f5324ff9700 1 -- 172.21.15.119:0/3917218851 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5318050990 con 0x7f532806f810 2024-09-13T16:29:15.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.584+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53280b3eb0 con 0x7f532806f810 2024-09-13T16:29:15.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f5324ff9700 1 -- 172.21.15.119:0/3917218851 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16969 (secure 0 0 0) 0x7f5318053fb0 con 0x7f532806f810 2024-09-13T16:29:15.593 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:15.593 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":70,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:08.646341+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:28:19.445072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-09-13T16:28:20.106803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.17","mappings":[{"from":7,"to":2}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5308066730 msgr2=0x7f5308068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5308066730 0x7f5308068bf0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f53100090b0 tx=0x7f5310008030 comp rx=0 tx=0).stop 2024-09-13T16:29:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f532806f810 msgr2=0x7f53280b0ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f532806f810 0x7f53280b0ae0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f531800c4d0 tx=0x7f531800c7e0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 shutdown_connections 2024-09-13T16:29:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5308066730 0x7f5308068bf0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f532815aab0 0x7f53280cba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53280ceb90 0x7f5328152ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 --2- 172.21.15.119:0/3917218851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f532806f810 0x7f53280b0ae0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.588+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 >> 172.21.15.119:0/3917218851 conn(0x7f5328099a50 msgr2=0x7f53281594a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:15.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.592+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 shutdown_connections 2024-09-13T16:29:15.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.592+0000 7f532ca09700 1 -- 172.21.15.119:0/3917218851 wait complete. 2024-09-13T16:29:15.608 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:28:19.445072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:28:20.106803+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:15.608 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:15.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa837932700 1 Processor -- start 2024-09-13T16:29:15.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa837932700 1 -- start start 2024-09-13T16:29:15.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa837932700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83015cb90 0x7fa8300b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa837932700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa83006c4e0 0x7fa8300b4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa837932700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8300b4390 con 0x7fa83006b990 2024-09-13T16:29:15.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa837932700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8300b1be0 con 0x7fa83006c4e0 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa837932700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8300b1060 con 0x7fa83015cb90 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa83006c4e0 0x7fa8300b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa83006c4e0 0x7fa8300b4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58598/0 (socket says 172.21.15.119:58598) 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 -- 172.21.15.119:0/581632998 learned_addr learned my addr 172.21.15.119:0/581632998 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa8356ce700 1 --2- 172.21.15.119:0/581632998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83015cb90 0x7fa8300b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 -- 172.21.15.119:0/581632998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83015cb90 msgr2=0x7fa8300b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 --2- 172.21.15.119:0/581632998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83015cb90 0x7fa8300b1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 -- 172.21.15.119:0/581632998 >> v1:172.21.15.145:6789/0 conn(0x7fa83006b990 legacy=0x7fa83006bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 -- 172.21.15.119:0/581632998 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8300b1340 con 0x7fa83006c4e0 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.720+0000 7fa835ecf700 1 --2- 172.21.15.119:0/581632998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300b4850 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa828009e50 tx=0x7fa82800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ba3c97a0d726bbd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa827fff700 1 -- 172.21.15.119:0/581632998 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82800e070 con 0x7fa83006c4e0 2024-09-13T16:29:15.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa827fff700 1 -- 172.21.15.119:0/581632998 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa828003a40 con 0x7fa83006c4e0 2024-09-13T16:29:15.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa827fff700 1 -- 172.21.15.119:0/581632998 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa828003df0 con 0x7fa83006c4e0 2024-09-13T16:29:15.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- 172.21.15.119:0/581632998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 msgr2=0x7fa8300b4850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 --2- 172.21.15.119:0/581632998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300b4850 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fa828009e50 tx=0x7fa82800b330 comp rx=0 tx=0).stop 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- 172.21.15.119:0/581632998 shutdown_connections 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 --2- 172.21.15.119:0/581632998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300b4850 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 --2- 172.21.15.119:0/581632998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83015cb90 0x7fa8300b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- 172.21.15.119:0/581632998 >> 172.21.15.119:0/581632998 conn(0x7fa830099a50 msgr2=0x7fa830099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- 172.21.15.119:0/581632998 shutdown_connections 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- 172.21.15.119:0/581632998 wait complete. 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 Processor -- start 2024-09-13T16:29:15.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- start start 2024-09-13T16:29:15.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83006b990 0x7fa8300cc690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300ae1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa83015cb90 0x7fa8301483c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8300cb1b0 con 0x7fa83006c4e0 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8300ae720 con 0x7fa83015cb90 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.724+0000 7fa837932700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8300ae950 con 0x7fa83006b990 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300ae1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300ae1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58602/0 (socket says 172.21.15.119:58602) 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 -- 172.21.15.119:0/64819493 learned_addr learned my addr 172.21.15.119:0/64819493 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa8356ce700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83006b990 0x7fa8300cc690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa835ecf700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa83015cb90 0x7fa8301483c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 -- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83006b990 msgr2=0x7fa8300cc690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83006b990 0x7fa8300cc690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 -- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa83015cb90 msgr2=0x7fa8301483c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa83015cb90 0x7fa8301483c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 -- 172.21.15.119:0/64819493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa828009b30 con 0x7fa83006c4e0 2024-09-13T16:29:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa834ecd700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300ae1e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c550 tx=0x7fa82c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa825ffb700 1 -- 172.21.15.119:0/64819493 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c011790 con 0x7fa83006c4e0 2024-09-13T16:29:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa825ffb700 1 -- 172.21.15.119:0/64819493 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82c0118f0 con 0x7fa83006c4e0 2024-09-13T16:29:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8300aec30 con 0x7fa83006c4e0 2024-09-13T16:29:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa825ffb700 1 -- 172.21.15.119:0/64819493 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa82c011ca0 con 0x7fa83006c4e0 2024-09-13T16:29:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8300af1f0 con 0x7fa83006c4e0 2024-09-13T16:29:15.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.728+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8300b1d50 con 0x7fa83006c4e0 2024-09-13T16:29:15.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.732+0000 7fa825ffb700 1 -- 172.21.15.119:0/64819493 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa82c01c460 con 0x7fa83006c4e0 2024-09-13T16:29:15.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.732+0000 7fa825ffb700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:15.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.732+0000 7fa825ffb700 1 -- 172.21.15.119:0/64819493 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fa82c084b80 con 0x7fa83006c4e0 2024-09-13T16:29:15.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.732+0000 7fa8356ce700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:15.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.732+0000 7fa8356ce700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa82000aae0 tx=0x7fa820008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:15.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.736+0000 7fa825ffb700 1 -- 172.21.15.119:0/64819493 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa82c051c80 con 0x7fa83006c4e0 2024-09-13T16:29:15.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.900+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa83006b560 con 0x7fa83006c4e0 2024-09-13T16:29:15.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.900+0000 7fa825ffb700 1 -- 172.21.15.119:0/64819493 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v70) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa82c0552a0 con 0x7fa83006c4e0 2024-09-13T16:29:15.908 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 msgr2=0x7fa814068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa82000aae0 tx=0x7fa820008040 comp rx=0 tx=0).stop 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 msgr2=0x7fa8300ae1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300ae1e0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00c550 tx=0x7fa82c00c860 comp rx=0 tx=0).stop 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 shutdown_connections 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa814066730 0x7fa814068bf0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa83015cb90 0x7fa8301483c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa83006c4e0 0x7fa8300ae1e0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 --2- 172.21.15.119:0/64819493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa83006b990 0x7fa8300cc690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 >> 172.21.15.119:0/64819493 conn(0x7fa830099a50 msgr2=0x7fa83006f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:15.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 shutdown_connections 2024-09-13T16:29:15.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:15.904+0000 7fa837932700 1 -- 172.21.15.119:0/64819493 wait complete. 2024-09-13T16:29:15.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:29:16.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d8b63700 1 Processor -- start 2024-09-13T16:29:16.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d8b63700 1 -- start start 2024-09-13T16:29:16.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d8b63700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 0x7f34d40d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d8b63700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40cdb00 con 0x7f34d40d17e0 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d8b63700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40cdde0 con 0x7f34d40d0c20 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d8b63700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40ce0c0 con 0x7f34d415cc00 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f34d40d17e0 0x7f34d40b5960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58924/0 (socket says 172.21.15.119:58924) 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d3fff700 1 -- 172.21.15.119:0/573862895 learned_addr learned my addr 172.21.15.119:0/573862895 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d37fe700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 0x7f34d40d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917121627 0 0) 0x7f34d40cdb00 con 0x7f34d40d17e0 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c00035d0 con 0x7f34d40d17e0 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3638911794 0 0) 0x7f34c00035d0 con 0x7f34d40d17e0 2024-09-13T16:29:16.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 msgr2=0x7f34d40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 0x7f34d40d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 >> v1:172.21.15.145:6789/0 conn(0x7f34d40d0c20 legacy=0x7f34d40d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d40ce3a0 con 0x7f34d40d17e0 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f34cc0019f0 con 0x7f34d40d17e0 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 >> v1:172.21.15.119:6789/0 conn(0x7f34d40d17e0 legacy=0x7f34d40b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 0x7f34c0003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c0003f90 0x7f34c0004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34c0004db0 0x7f34c0005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40ce0c0 con 0x7f34c0003f90 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c000bae0 con 0x7f34c0004db0 2024-09-13T16:29:16.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c000bca0 con 0x7f34d415cc00 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d2ffd700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c0003f90 0x7f34c0004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d37fe700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 0x7f34c0003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d3fff700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34c0004db0 0x7f34c0005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d2ffd700 1 -- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 msgr2=0x7f34c0003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.040+0000 7f34d2ffd700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34d415cc00 0x7f34c0003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d2ffd700 1 -- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34c0004db0 msgr2=0x7f34c0005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d2ffd700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34c0004db0 0x7f34c0005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d2ffd700 1 -- 172.21.15.119:0/573862895 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c00035d0 con 0x7f34c0003f90 2024-09-13T16:29:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d2ffd700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c0003f90 0x7f34c0004870 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f34d40c6610 tx=0x7f34c00058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97eee841abeb5078 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40052d0 con 0x7f34c0003f90 2024-09-13T16:29:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c4005430 con 0x7f34c0003f90 2024-09-13T16:29:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d27fc700 1 -- 172.21.15.119:0/573862895 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40055a0 con 0x7f34c0003f90 2024-09-13T16:29:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d8b63700 1 -- 172.21.15.119:0/573862895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c0003f90 msgr2=0x7f34c0004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d8b63700 1 --2- 172.21.15.119:0/573862895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34c0003f90 0x7f34c0004870 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f34d40c6610 tx=0x7f34c00058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d3fff700 1 -- 172.21.15.119:0/573862895 reap_dead start 2024-09-13T16:29:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d8b63700 1 -- 172.21.15.119:0/573862895 shutdown_connections 2024-09-13T16:29:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d8b63700 1 -- 172.21.15.119:0/573862895 >> 172.21.15.119:0/573862895 conn(0x7f34d408c080 msgr2=0x7f34d40d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d8b63700 1 -- 172.21.15.119:0/573862895 shutdown_connections 2024-09-13T16:29:16.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.044+0000 7f34d8b63700 1 -- 172.21.15.119:0/573862895 wait complete. 2024-09-13T16:29:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 Processor -- start 2024-09-13T16:29:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 -- start start 2024-09-13T16:29:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34c0003f90 0x7f34d41533d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34d4153fb0 0x7f34d415e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34d4153a60 0x7f34d40cb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d415eda0 con 0x7f34d4153a60 2024-09-13T16:29:16.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d415f930 con 0x7f34d4153fb0 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d415dd80 con 0x7f34c0003f90 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34d4153a60 0x7f34d40cb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d37fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34c0003f90 0x7f34d41533d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d2ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34d4153fb0 0x7f34d415e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d37fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34c0003f90 0x7f34d41533d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:54408/0 (socket says 172.21.15.119:54408) 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d37fe700 1 -- 172.21.15.119:0/2873697537 learned_addr learned my addr 172.21.15.119:0/2873697537 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34d4153a60 0x7f34d40cb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58624/0 (socket says 172.21.15.119:58624) 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 -- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34c0003f90 msgr2=0x7f34d41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34c0003f90 0x7f34d41533d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 -- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34d4153fb0 msgr2=0x7f34d415e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34d4153fb0 0x7f34d415e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 -- 172.21.15.119:0/2873697537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d415e060 con 0x7f34d4153a60 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d3fff700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34d4153a60 0x7f34d40cb5e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f34d408c530 tx=0x7f34cc00f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34cbfff700 1 -- 172.21.15.119:0/2873697537 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00f970 con 0x7f34d4153a60 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34cbfff700 1 -- 172.21.15.119:0/2873697537 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34cc00faf0 con 0x7f34d4153a60 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d415e340 con 0x7f34d4153a60 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34cbfff700 1 -- 172.21.15.119:0/2873697537 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34cc00fea0 con 0x7f34d4153a60 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.048+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d40cc3a0 con 0x7f34d4153a60 2024-09-13T16:29:16.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.052+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d415fdf0 con 0x7f34d4153a60 2024-09-13T16:29:16.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.052+0000 7f34cbfff700 1 -- 172.21.15.119:0/2873697537 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f34cc01e3b0 con 0x7f34d4153a60 2024-09-13T16:29:16.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.052+0000 7f34cbfff700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34b4066730 0x7f34b4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.052+0000 7f34cbfff700 1 -- 172.21.15.119:0/2873697537 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f34cc083e80 con 0x7f34d4153a60 2024-09-13T16:29:16.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.052+0000 7f34d37fe700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34b4066730 0x7f34b4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.052+0000 7f34d37fe700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34b4066730 0x7f34b4068bf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f34bc006cb0 tx=0x7f34bc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.060+0000 7f34cbfff700 1 -- 172.21.15.119:0/2873697537 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f34cc051030 con 0x7f34d4153a60 2024-09-13T16:29:16.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f34d40cf090 con 0x7f34d4153a60 2024-09-13T16:29:16.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34cbfff700 1 -- 172.21.15.119:0/2873697537 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v70) v1 ==== 104+0+11 (secure 0 0 0) 0x7f34cc01a020 con 0x7f34d4153a60 2024-09-13T16:29:16.232 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:29:16.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34b4066730 msgr2=0x7f34b4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34b4066730 0x7f34b4068bf0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f34bc006cb0 tx=0x7f34bc008040 comp rx=0 tx=0).stop 2024-09-13T16:29:16.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34d4153a60 msgr2=0x7f34d40cb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34d4153a60 0x7f34d40cb5e0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f34d408c530 tx=0x7f34cc00f440 comp rx=0 tx=0).stop 2024-09-13T16:29:16.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 shutdown_connections 2024-09-13T16:29:16.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f34b4066730 0x7f34b4068bf0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34d4153a60 0x7f34d40cb5e0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34d4153fb0 0x7f34d415e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 --2- 172.21.15.119:0/2873697537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34c0003f90 0x7f34d41533d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 >> 172.21.15.119:0/2873697537 conn(0x7f34d408c080 msgr2=0x7f34d40b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:16.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 shutdown_connections 2024-09-13T16:29:16.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.228+0000 7f34d8b63700 1 -- 172.21.15.119:0/2873697537 wait complete. 2024-09-13T16:29:16.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:29:16.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.364+0000 7fe2d717c700 1 Processor -- start 2024-09-13T16:29:16.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.364+0000 7fe2d717c700 1 -- start start 2024-09-13T16:29:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d015cf80 0x7fe2d00d1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe2d00d3890 0x7fe2d00cf780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00cd910 con 0x7fe2d00cfcc0 2024-09-13T16:29:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe2d00d0170 con 0x7fe2d00d3890 2024-09-13T16:29:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d015d360 con 0x7fe2d015cf80 2024-09-13T16:29:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe2d00d3890 0x7fe2d00cf780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d5719700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe2d00cfcc0 0x7fe2d01609d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58938/0 (socket says 172.21.15.119:58938) 2024-09-13T16:29:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d5719700 1 -- 172.21.15.119:0/55610856 learned_addr learned my addr 172.21.15.119:0/55610856 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:16.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d4f18700 1 --2- 172.21.15.119:0/55610856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d015cf80 0x7fe2d00d1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cffff700 1 -- 172.21.15.119:0/55610856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d015cf80 msgr2=0x7fe2d00d1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cffff700 1 --2- 172.21.15.119:0/55610856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d015cf80 0x7fe2d00d1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cffff700 1 -- 172.21.15.119:0/55610856 >> v1:172.21.15.119:6789/0 conn(0x7fe2d00cfcc0 legacy=0x7fe2d01609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cffff700 1 -- 172.21.15.119:0/55610856 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00ce750 con 0x7fe2d00d3890 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d4f18700 1 --2- 172.21.15.119:0/55610856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d015cf80 0x7fe2d00d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cffff700 1 --2- 172.21.15.119:0/55610856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00cf780 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009850 tx=0x7fe2bc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab113db6c592f6ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cf7fe700 1 -- 172.21.15.119:0/55610856 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc011830 con 0x7fe2d00d3890 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cf7fe700 1 -- 172.21.15.119:0/55610856 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc011990 con 0x7fe2d00d3890 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2cf7fe700 1 -- 172.21.15.119:0/55610856 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc011d20 con 0x7fe2d00d3890 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- 172.21.15.119:0/55610856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 msgr2=0x7fe2d00cf780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 --2- 172.21.15.119:0/55610856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00cf780 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc009850 tx=0x7fe2bc009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:16.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- 172.21.15.119:0/55610856 shutdown_connections 2024-09-13T16:29:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 --2- 172.21.15.119:0/55610856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00cf780 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 --2- 172.21.15.119:0/55610856 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d015cf80 0x7fe2d00d1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- 172.21.15.119:0/55610856 >> 172.21.15.119:0/55610856 conn(0x7fe2d00d2d70 msgr2=0x7fe2d00d3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- 172.21.15.119:0/55610856 shutdown_connections 2024-09-13T16:29:16.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.368+0000 7fe2d717c700 1 -- 172.21.15.119:0/55610856 wait complete. 2024-09-13T16:29:16.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 Processor -- start 2024-09-13T16:29:16.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 -- start start 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d00cfcc0 0x7fe2d015c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2d015cf80 0x7fe2d01f2900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d015c8e0 con 0x7fe2d00d3890 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d015e520 con 0x7fe2d015cf80 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.372+0000 7fe2d717c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2d015f0a0 con 0x7fe2d00cfcc0 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00b0810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58632/0 (socket says 172.21.15.119:58632) 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 -- 172.21.15.119:0/2072265248 learned_addr learned my addr 172.21.15.119:0/2072265248 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2d4f18700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d00cfcc0 0x7fe2d015c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2d5719700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2d015cf80 0x7fe2d01f2900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 -- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d00cfcc0 msgr2=0x7fe2d015c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d00cfcc0 0x7fe2d015c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 -- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2d015cf80 msgr2=0x7fe2d01f2900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2d015cf80 0x7fe2d01f2900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 -- 172.21.15.119:0/2072265248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2bc0094b0 con 0x7fe2d00d3890 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cffff700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00b0810 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc000c00 tx=0x7fe2bc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cd7fa700 1 -- 172.21.15.119:0/2072265248 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc0036c0 con 0x7fe2d00d3890 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cd7fa700 1 -- 172.21.15.119:0/2072265248 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2bc01f070 con 0x7fe2d00d3890 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d015e770 con 0x7fe2d00d3890 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cd7fa700 1 -- 172.21.15.119:0/2072265248 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2bc003a60 con 0x7fe2d00d3890 2024-09-13T16:29:16.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d015ed30 con 0x7fe2d00d3890 2024-09-13T16:29:16.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d00677a0 con 0x7fe2d00d3890 2024-09-13T16:29:16.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.376+0000 7fe2cd7fa700 1 -- 172.21.15.119:0/2072265248 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe2bc003bc0 con 0x7fe2d00d3890 2024-09-13T16:29:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.380+0000 7fe2cd7fa700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe2b4066730 0x7fe2b4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.380+0000 7fe2cd7fa700 1 -- 172.21.15.119:0/2072265248 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fe2bc021070 con 0x7fe2d00d3890 2024-09-13T16:29:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.380+0000 7fe2d4f18700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe2b4066730 0x7fe2b4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.380+0000 7fe2d4f18700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe2b4066730 0x7fe2b4068bf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00b1570 tx=0x7fe2c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.384+0000 7fe2cd7fa700 1 -- 172.21.15.119:0/2072265248 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe2bc052ec0 con 0x7fe2d00d3890 2024-09-13T16:29:16.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.556+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe2d015eec0 con 0x7fe2d00d3890 2024-09-13T16:29:16.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.556+0000 7fe2cd7fa700 1 -- 172.21.15.119:0/2072265248 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v70) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe2bc015020 con 0x7fe2d00d3890 2024-09-13T16:29:16.561 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:29:16.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe2b4066730 msgr2=0x7fe2b4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe2b4066730 0x7fe2b4068bf0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe2d00b1570 tx=0x7fe2c4008040 comp rx=0 tx=0).stop 2024-09-13T16:29:16.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 msgr2=0x7fe2d00b0810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00b0810 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7fe2bc000c00 tx=0x7fe2bc004770 comp rx=0 tx=0).stop 2024-09-13T16:29:16.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 shutdown_connections 2024-09-13T16:29:16.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe2b4066730 0x7fe2b4068bf0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe2d015cf80 0x7fe2d01f2900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe2d00d3890 0x7fe2d00b0810 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 --2- 172.21.15.119:0/2072265248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe2d00cfcc0 0x7fe2d015c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 >> 172.21.15.119:0/2072265248 conn(0x7fe2d00d2d70 msgr2=0x7fe2d00b4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:16.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 shutdown_connections 2024-09-13T16:29:16.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.560+0000 7fe2d717c700 1 -- 172.21.15.119:0/2072265248 wait complete. 2024-09-13T16:29:16.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.684+0000 7fa10e943700 1 Processor -- start 2024-09-13T16:29:16.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.684+0000 7fa10e943700 1 -- start start 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e943700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa10806be70 0x7fa1080b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e943700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa1080adc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e943700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10806cff0 con 0x7fa10806e6a0 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e943700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa10806d480 con 0x7fa10806be70 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e943700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10806c3e0 con 0x7fa1080ad7e0 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10d941700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa10806e6a0 0x7fa10806ea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58944/0 (socket says 172.21.15.119:58944) 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10d941700 1 -- 172.21.15.119:0/2330105070 learned_addr learned my addr 172.21.15.119:0/2330105070 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10d140700 1 --2- 172.21.15.119:0/2330105070 >> v2:172.21.15.145:3300/0 conn(0x7fa10806be70 0x7fa1080b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e142700 1 --2- 172.21.15.119:0/2330105070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa1080adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10c93f700 1 -- 172.21.15.119:0/2330105070 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4269193882 0 0) 0x7fa10806cff0 con 0x7fa10806e6a0 2024-09-13T16:29:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10c93f700 1 -- 172.21.15.119:0/2330105070 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f40035d0 con 0x7fa10806e6a0 2024-09-13T16:29:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e142700 1 -- 172.21.15.119:0/2330105070 >> v2:172.21.15.145:3300/0 conn(0x7fa10806be70 msgr2=0x7fa1080b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:16.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e142700 1 --2- 172.21.15.119:0/2330105070 >> v2:172.21.15.145:3300/0 conn(0x7fa10806be70 0x7fa1080b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e142700 1 -- 172.21.15.119:0/2330105070 >> v1:172.21.15.119:6789/0 conn(0x7fa10806e6a0 legacy=0x7fa10806ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e142700 1 -- 172.21.15.119:0/2330105070 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10806c6c0 con 0x7fa1080ad7e0 2024-09-13T16:29:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10d140700 1 --2- 172.21.15.119:0/2330105070 >> v2:172.21.15.145:3300/0 conn(0x7fa10806be70 0x7fa1080b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:16.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.688+0000 7fa10e142700 1 --2- 172.21.15.119:0/2330105070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa1080adc70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa100009d90 tx=0x7fa10000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eef9f5847ce78d11 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10c93f700 1 -- 172.21.15.119:0/2330105070 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10000e070 con 0x7fa1080ad7e0 2024-09-13T16:29:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10c93f700 1 -- 172.21.15.119:0/2330105070 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10000bb20 con 0x7fa1080ad7e0 2024-09-13T16:29:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10c93f700 1 -- 172.21.15.119:0/2330105070 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa100012430 con 0x7fa1080ad7e0 2024-09-13T16:29:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 -- 172.21.15.119:0/2330105070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 msgr2=0x7fa1080adc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 --2- 172.21.15.119:0/2330105070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa1080adc70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa100009d90 tx=0x7fa10000b330 comp rx=0 tx=0).stop 2024-09-13T16:29:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 -- 172.21.15.119:0/2330105070 shutdown_connections 2024-09-13T16:29:16.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 --2- 172.21.15.119:0/2330105070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa1080adc70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 --2- 172.21.15.119:0/2330105070 >> v2:172.21.15.145:3300/0 conn(0x7fa10806be70 0x7fa1080b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 -- 172.21.15.119:0/2330105070 >> 172.21.15.119:0/2330105070 conn(0x7fa1080b5d20 msgr2=0x7fa1080b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:16.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 -- 172.21.15.119:0/2330105070 shutdown_connections 2024-09-13T16:29:16.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.696+0000 7fa10e943700 1 -- 172.21.15.119:0/2330105070 wait complete. 2024-09-13T16:29:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 Processor -- start 2024-09-13T16:29:16.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 -- start start 2024-09-13T16:29:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa10806be70 0x7fa108154a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa10806e6a0 0x7fa10815f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa108150110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108154280 con 0x7fa10806e6a0 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108160050 con 0x7fa10806be70 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e943700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa108154fa0 con 0x7fa1080ad7e0 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa10806e6a0 0x7fa10815f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa10806e6a0 0x7fa10815f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58638/0 (socket says 172.21.15.119:58638) 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 -- 172.21.15.119:0/1776797392 learned_addr learned my addr 172.21.15.119:0/1776797392 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d941700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa10806be70 0x7fa108154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e142700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa108150110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 -- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 msgr2=0x7fa108150110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa108150110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 -- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa10806be70 msgr2=0x7fa108154a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa10806be70 0x7fa108154a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 -- 172.21.15.119:0/1776797392 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1040094b0 con 0x7fa10806e6a0 2024-09-13T16:29:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10e142700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa108150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:16.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.700+0000 7fa10d140700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa10806e6a0 0x7fa10815f750 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa10400ebe0 tx=0x7fa10400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa0fe7fc700 1 -- 172.21.15.119:0/1776797392 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa104009770 con 0x7fa10806e6a0 2024-09-13T16:29:16.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa0fe7fc700 1 -- 172.21.15.119:0/1776797392 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10400c850 con 0x7fa10806e6a0 2024-09-13T16:29:16.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa100009a70 con 0x7fa10806e6a0 2024-09-13T16:29:16.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa0fe7fc700 1 -- 172.21.15.119:0/1776797392 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10400cbe0 con 0x7fa10806e6a0 2024-09-13T16:29:16.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10815d700 con 0x7fa10806e6a0 2024-09-13T16:29:16.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa108074560 con 0x7fa10806e6a0 2024-09-13T16:29:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa0fe7fc700 1 -- 172.21.15.119:0/1776797392 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa10400cd40 con 0x7fa10806e6a0 2024-09-13T16:29:16.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.704+0000 7fa0fe7fc700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0ec066730 0x7fa0ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.708+0000 7fa0fe7fc700 1 -- 172.21.15.119:0/1776797392 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fa104084300 con 0x7fa10806e6a0 2024-09-13T16:29:16.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.708+0000 7fa10d941700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0ec066730 0x7fa0ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:16.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.708+0000 7fa10d941700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0ec066730 0x7fa0ec068bf0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8007fd0 tx=0x7fa0f8007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:16.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.712+0000 7fa0fe7fc700 1 -- 172.21.15.119:0/1776797392 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa1040514b0 con 0x7fa10806e6a0 2024-09-13T16:29:16.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.916+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa10815d250 con 0x7fa10806e6a0 2024-09-13T16:29:16.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.916+0000 7fa0fe7fc700 1 -- 172.21.15.119:0/1776797392 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4742 (secure 0 0 0) 0x7fa104054ad0 con 0x7fa10806e6a0 2024-09-13T16:29:16.922 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:16.923 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:29:09.657258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":15,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:29:16.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0ec066730 msgr2=0x7fa0ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0ec066730 0x7fa0ec068bf0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8007fd0 tx=0x7fa0f8007de0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa10806e6a0 msgr2=0x7fa10815f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:16.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa10806e6a0 0x7fa10815f750 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7fa10400ebe0 tx=0x7fa10400eef0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 shutdown_connections 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0ec066730 0x7fa0ec068bf0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1080ad7e0 0x7fa108150110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa10806e6a0 0x7fa10815f750 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 --2- 172.21.15.119:0/1776797392 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa10806be70 0x7fa108154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 >> 172.21.15.119:0/1776797392 conn(0x7fa1080b5d20 msgr2=0x7fa1080b5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 shutdown_connections 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:16.920+0000 7fa10e943700 1 -- 172.21.15.119:0/1776797392 wait complete. 2024-09-13T16:29:16.926 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 34 2024-09-13T16:29:16.937 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:17.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c63b8700 1 Processor -- start 2024-09-13T16:29:17.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c63b8700 1 -- start start 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c63b8700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa0c00d37a0 0x7fa0c00ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c63b8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c0160c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c63b8700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0c00d1190 con 0x7fa0c015ce70 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c63b8700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0c00d3bc0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c63b8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c0161190 con 0x7fa0c01607c0 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa0c015ce70 0x7fa0c00d1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58956/0 (socket says 172.21.15.119:58956) 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bffff700 1 -- 172.21.15.119:0/3363113618 learned_addr learned my addr 172.21.15.119:0/3363113618 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bf7fe700 1 --2- 172.21.15.119:0/3363113618 >> v2:172.21.15.145:3300/0 conn(0x7fa0c00d37a0 0x7fa0c00ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0c4955700 1 --2- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c0160c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0beffd700 1 -- 172.21.15.119:0/3363113618 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898606922 0 0) 0x7fa0c00d1190 con 0x7fa0c015ce70 2024-09-13T16:29:17.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0beffd700 1 -- 172.21.15.119:0/3363113618 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ac0035d0 con 0x7fa0c015ce70 2024-09-13T16:29:17.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bf7fe700 1 -- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 msgr2=0x7fa0c0160c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:17.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bf7fe700 1 --2- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c0160c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bf7fe700 1 -- 172.21.15.119:0/3363113618 >> v1:172.21.15.119:6789/0 conn(0x7fa0c015ce70 legacy=0x7fa0c00d1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bf7fe700 1 -- 172.21.15.119:0/3363113618 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0c00ceaf0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0bf7fe700 1 --2- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c00d37a0 0x7fa0c00ce2a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009d90 tx=0x7fa0b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0bc4f85f32457ce server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0beffd700 1 -- 172.21.15.119:0/3363113618 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800e070 con 0x7fa0c00d37a0 2024-09-13T16:29:17.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0beffd700 1 -- 172.21.15.119:0/3363113618 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0b800bb20 con 0x7fa0c00d37a0 2024-09-13T16:29:17.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.044+0000 7fa0beffd700 1 -- 172.21.15.119:0/3363113618 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b800be90 con 0x7fa0c00d37a0 2024-09-13T16:29:17.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c00d37a0 msgr2=0x7fa0c00ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c00d37a0 0x7fa0c00ce2a0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8009d90 tx=0x7fa0b800b330 comp rx=0 tx=0).stop 2024-09-13T16:29:17.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- 172.21.15.119:0/3363113618 shutdown_connections 2024-09-13T16:29:17.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c0160c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/3363113618 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c00d37a0 0x7fa0c00ce2a0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- 172.21.15.119:0/3363113618 >> 172.21.15.119:0/3363113618 conn(0x7fa0c00d2bf0 msgr2=0x7fa0c00d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:17.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- 172.21.15.119:0/3363113618 shutdown_connections 2024-09-13T16:29:17.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- 172.21.15.119:0/3363113618 wait complete. 2024-09-13T16:29:17.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 Processor -- start 2024-09-13T16:29:17.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- start start 2024-09-13T16:29:17.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c00d37a0 0x7fa0c0158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c015ce70 0x7fa0c00b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c00b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00b8890 con 0x7fa0c00d37a0 2024-09-13T16:29:17.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00b77f0 con 0x7fa0c015ce70 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0c63b8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0c00b7ad0 con 0x7fa0c01607c0 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0bffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c00d37a0 0x7fa0c0158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0bffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c00d37a0 0x7fa0c0158350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58640/0 (socket says 172.21.15.119:58640) 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.048+0000 7fa0bffff700 1 -- 172.21.15.119:0/1083761502 learned_addr learned my addr 172.21.15.119:0/1083761502 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bf7fe700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c015ce70 0x7fa0c00b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0c4955700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c00b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bffff700 1 -- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 msgr2=0x7fa0c00b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bffff700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c00b0d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bffff700 1 -- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c015ce70 msgr2=0x7fa0c00b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bffff700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c015ce70 0x7fa0c00b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bffff700 1 -- 172.21.15.119:0/1083761502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8009a70 con 0x7fa0c00d37a0 2024-09-13T16:29:17.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bffff700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c00d37a0 0x7fa0c0158350 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800c4f0 tx=0x7fa0a800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bcff9700 1 -- 172.21.15.119:0/1083761502 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8005d90 con 0x7fa0c00d37a0 2024-09-13T16:29:17.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bcff9700 1 -- 172.21.15.119:0/1083761502 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a80070d0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0c00b7db0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bcff9700 1 -- 172.21.15.119:0/1083761502 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8007440 con 0x7fa0c00d37a0 2024-09-13T16:29:17.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0c00b8370 con 0x7fa0c00d37a0 2024-09-13T16:29:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0c00ce2a0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.052+0000 7fa0bcff9700 1 -- 172.21.15.119:0/1083761502 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa0a80075a0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.056+0000 7fa0bcff9700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0a0066730 0x7fa0a0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.056+0000 7fa0bcff9700 1 -- 172.21.15.119:0/1083761502 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fa0a80836a0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.056+0000 7fa0bf7fe700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0a0066730 0x7fa0a0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.056+0000 7fa0bf7fe700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0a0066730 0x7fa0a0068bf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8000f80 tx=0x7fa0b8011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.060+0000 7fa0bcff9700 1 -- 172.21.15.119:0/1083761502 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa0a8050880 con 0x7fa0c00d37a0 2024-09-13T16:29:17.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.240+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0c0072730 con 0x7fa0c00d37a0 2024-09-13T16:29:17.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0bcff9700 1 -- 172.21.15.119:0/1083761502 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16969 (secure 0 0 0) 0x7fa0a8053ea0 con 0x7fa0c00d37a0 2024-09-13T16:29:17.248 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:17.248 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":70,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:08.646341+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:28:19.445072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-09-13T16:28:20.106803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.17","mappings":[{"from":7,"to":2}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:17.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0a0066730 msgr2=0x7fa0a0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0a0066730 0x7fa0a0068bf0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fa0b8000f80 tx=0x7fa0b8011000 comp rx=0 tx=0).stop 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c00d37a0 msgr2=0x7fa0c0158350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c00d37a0 0x7fa0c0158350 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800c4f0 tx=0x7fa0a800c800 comp rx=0 tx=0).stop 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 shutdown_connections 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0a0066730 0x7fa0a0068bf0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0c01607c0 0x7fa0c00b0d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0c015ce70 0x7fa0c00b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 --2- 172.21.15.119:0/1083761502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0c00d37a0 0x7fa0c0158350 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 >> 172.21.15.119:0/1083761502 conn(0x7fa0c00d2bf0 msgr2=0x7fa0c006b010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 shutdown_connections 2024-09-13T16:29:17.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.244+0000 7fa0c63b8700 1 -- 172.21.15.119:0/1083761502 wait complete. 2024-09-13T16:29:17.263 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:29:17.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b471f3700 1 Processor -- start 2024-09-13T16:29:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b471f3700 1 -- start start 2024-09-13T16:29:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b471f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 0x7f3b400b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b471f3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3b400713b0 0x7f3b400b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b471f3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b400cf520 con 0x7f3b401597f0 2024-09-13T16:29:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b471f3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b400b2980 con 0x7f3b400713b0 2024-09-13T16:29:17.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b471f3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400b2c60 con 0x7f3b40159bd0 2024-09-13T16:29:17.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b461f1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3b401597f0 0x7f3b400cf7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58970/0 (socket says 172.21.15.119:58970) 2024-09-13T16:29:17.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b461f1700 1 -- 172.21.15.119:0/406805230 learned_addr learned my addr 172.21.15.119:0/406805230 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:17.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b469f2700 1 --2- 172.21.15.119:0/406805230 >> v2:172.21.15.145:3300/0 conn(0x7f3b400713b0 0x7f3b400b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 0x7f3b400b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454590952 0 0) 0x7f3b400cf520 con 0x7f3b401597f0 2024-09-13T16:29:17.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b2c0035d0 con 0x7f3b401597f0 2024-09-13T16:29:17.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 527532546 0 0) 0x7f3b2c0035d0 con 0x7f3b401597f0 2024-09-13T16:29:17.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 msgr2=0x7f3b400b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 0x7f3b400b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 >> v2:172.21.15.145:3300/0 conn(0x7f3b400713b0 msgr2=0x7f3b400b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 --2- 172.21.15.119:0/406805230 >> v2:172.21.15.145:3300/0 conn(0x7f3b400713b0 0x7f3b400b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b400b2f40 con 0x7f3b401597f0 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f3b30001670 con 0x7f3b401597f0 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 >> v1:172.21.15.119:6789/0 conn(0x7f3b401597f0 legacy=0x7f3b400cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2c003a20 0x7f3b2c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b2c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 0x7f3b2c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400b2980 con 0x7f3b2c004870 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400b2c60 con 0x7f3b2c003a20 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b2c00ba30 con 0x7f3b40159bd0 2024-09-13T16:29:17.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b2c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b469f2700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 0x7f3b2c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b461f1700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2c003a20 0x7f3b2c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 -- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 msgr2=0x7f3b2c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b40159bd0 0x7f3b2c005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 -- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2c003a20 msgr2=0x7f3b2c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b2c003a20 0x7f3b2c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 -- 172.21.15.119:0/406805230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b2c0035d0 con 0x7f3b2c004870 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.376+0000 7f3b459f0700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b2c004cf0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009fc0 tx=0x7f3b2c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98bb598c04fceead server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c00ec30 con 0x7f3b2c004870 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3c00f040 con 0x7f3b2c004870 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b451ef700 1 -- 172.21.15.119:0/406805230 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3c014480 con 0x7f3b2c004870 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- 172.21.15.119:0/406805230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 msgr2=0x7f3b2c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 --2- 172.21.15.119:0/406805230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b2c004cf0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c009fc0 tx=0x7f3b2c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b469f2700 1 -- 172.21.15.119:0/406805230 reap_dead start 2024-09-13T16:29:17.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- 172.21.15.119:0/406805230 shutdown_connections 2024-09-13T16:29:17.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- 172.21.15.119:0/406805230 >> 172.21.15.119:0/406805230 conn(0x7f3b4006b460 msgr2=0x7f3b4006b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:17.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- 172.21.15.119:0/406805230 shutdown_connections 2024-09-13T16:29:17.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- 172.21.15.119:0/406805230 wait complete. 2024-09-13T16:29:17.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 Processor -- start 2024-09-13T16:29:17.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- start start 2024-09-13T16:29:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b401403f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b4015c340 0x7f3b400a8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b400a9390 0x7f3b400aad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b400a9810 con 0x7f3b2c004870 2024-09-13T16:29:17.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b40140930 con 0x7f3b400a9390 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b471f3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b40165780 con 0x7f3b4015c340 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b401403f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b401403f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58660/0 (socket says 172.21.15.119:58660) 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 -- 172.21.15.119:0/2805258075 learned_addr learned my addr 172.21.15.119:0/2805258075 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b459f0700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b4015c340 0x7f3b400a8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b469f2700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b400a9390 0x7f3b400aad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 -- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b4015c340 msgr2=0x7f3b400a8e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b4015c340 0x7f3b400a8e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 -- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b400a9390 msgr2=0x7f3b400aad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b400a9390 0x7f3b400aad30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 -- 172.21.15.119:0/2805258075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b40165a60 con 0x7f3b2c004870 2024-09-13T16:29:17.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.380+0000 7f3b461f1700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b401403f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3b30001200 tx=0x7f3b3000edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b36ffd700 1 -- 172.21.15.119:0/2805258075 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b30005e00 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b36ffd700 1 -- 172.21.15.119:0/2805258075 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b3000c5d0 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b4015bb40 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b36ffd700 1 -- 172.21.15.119:0/2805258075 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b3000c980 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b40150c30 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b400677a0 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b36ffd700 1 -- 172.21.15.119:0/2805258075 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3b3000cae0 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b36ffd700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3b24066760 0x7f3b24068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b36ffd700 1 -- 172.21.15.119:0/2805258075 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f3b30083f30 con 0x7f3b2c004870 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.384+0000 7f3b459f0700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3b24066760 0x7f3b24068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.388+0000 7f3b459f0700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3b24066760 0x7f3b24068c20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3b400ae550 tx=0x7f3b3c013000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.392+0000 7f3b36ffd700 1 -- 172.21.15.119:0/2805258075 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3b30051170 con 0x7f3b2c004870 2024-09-13T16:29:17.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.596+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b400cd730 con 0x7f3b2c004870 2024-09-13T16:29:17.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.596+0000 7f3b36ffd700 1 -- 172.21.15.119:0/2805258075 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f3b30054790 con 0x7f3b2c004870 2024-09-13T16:29:17.601 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:17.601 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:29:09.657258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":15,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:29:17.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3b24066760 msgr2=0x7f3b24068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3b24066760 0x7f3b24068c20 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3b400ae550 tx=0x7f3b3c013000 comp rx=0 tx=0).stop 2024-09-13T16:29:17.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 msgr2=0x7f3b401403f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b401403f0 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f3b30001200 tx=0x7f3b3000edd0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 shutdown_connections 2024-09-13T16:29:17.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3b24066760 0x7f3b24068c20 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3b400a9390 0x7f3b400aad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3b4015c340 0x7f3b400a8e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 --2- 172.21.15.119:0/2805258075 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3b2c004870 0x7f3b401403f0 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 >> 172.21.15.119:0/2805258075 conn(0x7f3b4006b460 msgr2=0x7f3b40066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:17.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 shutdown_connections 2024-09-13T16:29:17.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.600+0000 7f3b471f3700 1 -- 172.21.15.119:0/2805258075 wait complete. 2024-09-13T16:29:17.606 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 34 2024-09-13T16:29:17.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:17.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 Processor -- start 2024-09-13T16:29:17.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- start start 2024-09-13T16:29:17.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f35dc0adb30 0x7f35dc0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc06cf60 con 0x7f35dc0ce040 2024-09-13T16:29:17.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35dc06c350 con 0x7f35dc06bb90 2024-09-13T16:29:17.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35dc06c630 con 0x7f35dc0adb30 2024-09-13T16:29:17.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e11ac700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f35dc0adb30 0x7f35dc0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e19ad700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f35dc0ce040 0x7f35dc0b2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58978/0 (socket says 172.21.15.119:58978) 2024-09-13T16:29:17.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e19ad700 1 -- 172.21.15.119:0/2459876489 learned_addr learned my addr 172.21.15.119:0/2459876489 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:17.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e11ac700 1 -- 172.21.15.119:0/2459876489 >> v1:172.21.15.145:6789/0 conn(0x7f35dc06bb90 legacy=0x7f35dc0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e11ac700 1 -- 172.21.15.119:0/2459876489 >> v1:172.21.15.119:6789/0 conn(0x7f35dc0ce040 legacy=0x7f35dc0b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e11ac700 1 -- 172.21.15.119:0/2459876489 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc06c910 con 0x7f35dc0adb30 2024-09-13T16:29:17.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e11ac700 1 --2- 172.21.15.119:0/2459876489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0adb30 0x7f35dc0cc060 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009d90 tx=0x7f35d400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2096c16c915c7c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35dbfff700 1 -- 172.21.15.119:0/2459876489 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d400e070 con 0x7f35dc0adb30 2024-09-13T16:29:17.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35dbfff700 1 -- 172.21.15.119:0/2459876489 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d400bbd0 con 0x7f35dc0adb30 2024-09-13T16:29:17.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35dbfff700 1 -- 172.21.15.119:0/2459876489 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d4012430 con 0x7f35dc0adb30 2024-09-13T16:29:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- 172.21.15.119:0/2459876489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0adb30 msgr2=0x7f35dc0cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 --2- 172.21.15.119:0/2459876489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0adb30 0x7f35dc0cc060 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f35d4009d90 tx=0x7f35d400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- 172.21.15.119:0/2459876489 shutdown_connections 2024-09-13T16:29:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 --2- 172.21.15.119:0/2459876489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0adb30 0x7f35dc0cc060 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- 172.21.15.119:0/2459876489 >> 172.21.15.119:0/2459876489 conn(0x7f35dc06a350 msgr2=0x7f35dc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- 172.21.15.119:0/2459876489 shutdown_connections 2024-09-13T16:29:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.724+0000 7f35e21ae700 1 -- 172.21.15.119:0/2459876489 wait complete. 2024-09-13T16:29:17.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 Processor -- start 2024-09-13T16:29:17.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 -- start start 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35dc06bb90 0x7f35dc159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35dc0adb30 0x7f35dc13d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0ce040 0x7f35dc1f0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc13c690 con 0x7f35dc0ce040 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc13d4f0 con 0x7f35dc06bb90 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35dc13c4b0 con 0x7f35dc0adb30 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0ce040 0x7f35dc1f0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0ce040 0x7f35dc1f0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58670/0 (socket says 172.21.15.119:58670) 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 -- 172.21.15.119:0/3066966802 learned_addr learned my addr 172.21.15.119:0/3066966802 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e09ab700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35dc0adb30 0x7f35dc13d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e11ac700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35dc06bb90 0x7f35dc159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 -- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35dc0adb30 msgr2=0x7f35dc13d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35dc0adb30 0x7f35dc13d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 -- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35dc06bb90 msgr2=0x7f35dc159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35dc06bb90 0x7f35dc159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 -- 172.21.15.119:0/3066966802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d4009a70 con 0x7f35dc0ce040 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e09ab700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35dc0adb30 0x7f35dc13d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e19ad700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0ce040 0x7f35dc1f0620 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00c450 tx=0x7f35cc00c760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35d9ffb700 1 -- 172.21.15.119:0/3066966802 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc005d90 con 0x7f35dc0ce040 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35dc13c8e0 con 0x7f35dc0ce040 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35dc13cea0 con 0x7f35dc0ce040 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35d9ffb700 1 -- 172.21.15.119:0/3066966802 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35cc00ce20 con 0x7f35dc0ce040 2024-09-13T16:29:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.728+0000 7f35d9ffb700 1 -- 172.21.15.119:0/3066966802 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc0072d0 con 0x7f35dc0ce040 2024-09-13T16:29:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.732+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35dc0ad730 con 0x7f35dc0ce040 2024-09-13T16:29:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.732+0000 7f35d9ffb700 1 -- 172.21.15.119:0/3066966802 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f35cc007430 con 0x7f35dc0ce040 2024-09-13T16:29:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.732+0000 7f35d9ffb700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35c8066760 0x7f35c8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.732+0000 7f35d9ffb700 1 -- 172.21.15.119:0/3066966802 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f35cc0839f0 con 0x7f35dc0ce040 2024-09-13T16:29:17.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.732+0000 7f35e11ac700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35c8066760 0x7f35c8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:17.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.736+0000 7f35e11ac700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35c8066760 0x7f35c8068c20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f35d4014040 tx=0x7f35d4011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.740+0000 7f35d9ffb700 1 -- 172.21.15.119:0/3066966802 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f35cc050c40 con 0x7f35dc0ce040 2024-09-13T16:29:17.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35dc06d3d0 con 0x7f35dc0ce040 2024-09-13T16:29:17.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35d9ffb700 1 -- 172.21.15.119:0/3066966802 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+4742 (secure 0 0 0) 0x7f35dc06d3d0 con 0x7f35dc0ce040 2024-09-13T16:29:17.948 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:17.948 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":34,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:28:21.113532+0000","modified":"2024-09-13T16:29:09.657258+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6546":{"gid":6546,"name":"d","rank":0,"incarnation":28,"state":"up:active","state_seq":15,"addr":"172.21.15.145:6837/3224929185","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3224929185},{"type":"v1","addr":"172.21.15.145:6837","nonce":3224929185}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2024-09-13T16:29:17.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35c8066760 msgr2=0x7f35c8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35c8066760 0x7f35c8068c20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f35d4014040 tx=0x7f35d4011000 comp rx=0 tx=0).stop 2024-09-13T16:29:17.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0ce040 msgr2=0x7f35dc1f0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:17.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0ce040 0x7f35dc1f0620 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f35cc00c450 tx=0x7f35cc00c760 comp rx=0 tx=0).stop 2024-09-13T16:29:17.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 shutdown_connections 2024-09-13T16:29:17.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35c8066760 0x7f35c8068c20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35dc0ce040 0x7f35dc1f0620 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35dc0adb30 0x7f35dc13d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 --2- 172.21.15.119:0/3066966802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35dc06bb90 0x7f35dc159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:17.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 >> 172.21.15.119:0/3066966802 conn(0x7f35dc06a350 msgr2=0x7f35dc066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:17.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 shutdown_connections 2024-09-13T16:29:17.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:17.944+0000 7f35e21ae700 1 -- 172.21.15.119:0/3066966802 wait complete. 2024-09-13T16:29:17.952 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 34 2024-09-13T16:29:17.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:18.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.072+0000 7fd29885b700 1 Processor -- start 2024-09-13T16:29:18.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.072+0000 7fd29885b700 1 -- start start 2024-09-13T16:29:18.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd28c033c50 0x7fd28c005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c01ce60 0x7fd28c01d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd28c0df440 con 0x7fd28c04fc90 2024-09-13T16:29:18.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd28c01d7b0 con 0x7fd28c033c50 2024-09-13T16:29:18.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c006050 con 0x7fd28c01ce60 2024-09-13T16:29:18.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29259c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd28c04fc90 0x7fd28c0e32e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58980/0 (socket says 172.21.15.119:58980) 2024-09-13T16:29:18.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29259c700 1 -- 172.21.15.119:0/1732488263 learned_addr learned my addr 172.21.15.119:0/1732488263 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:18.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29359e700 1 --2- 172.21.15.119:0/1732488263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c01ce60 0x7fd28c01d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd292d9d700 1 --2- 172.21.15.119:0/1732488263 >> v2:172.21.15.145:3300/0 conn(0x7fd28c033c50 0x7fd28c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd291d9b700 1 -- 172.21.15.119:0/1732488263 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077219612 0 0) 0x7fd27c001110 con 0x7fd28c04fc90 2024-09-13T16:29:18.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd291d9b700 1 -- 172.21.15.119:0/1732488263 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2800035d0 con 0x7fd28c04fc90 2024-09-13T16:29:18.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd291d9b700 1 -- 172.21.15.119:0/1732488263 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2357167247 0 0) 0x7fd2800035d0 con 0x7fd28c04fc90 2024-09-13T16:29:18.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29359e700 1 -- 172.21.15.119:0/1732488263 >> v2:172.21.15.145:3300/0 conn(0x7fd28c033c50 msgr2=0x7fd28c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29359e700 1 --2- 172.21.15.119:0/1732488263 >> v2:172.21.15.145:3300/0 conn(0x7fd28c033c50 0x7fd28c005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29359e700 1 -- 172.21.15.119:0/1732488263 >> v1:172.21.15.119:6789/0 conn(0x7fd28c04fc90 legacy=0x7fd28c0e32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29359e700 1 -- 172.21.15.119:0/1732488263 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd28c0e39f0 con 0x7fd28c01ce60 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29359e700 1 --2- 172.21.15.119:0/1732488263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c01ce60 0x7fd28c01d270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd29404ff70 tx=0x7fd29404ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e1256283364121e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd291d9b700 1 -- 172.21.15.119:0/1732488263 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd27c001800 con 0x7fd28c01ce60 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd291d9b700 1 -- 172.21.15.119:0/1732488263 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd294067490 con 0x7fd28c01ce60 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd291d9b700 1 -- 172.21.15.119:0/1732488263 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294067600 con 0x7fd28c01ce60 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- 172.21.15.119:0/1732488263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c01ce60 msgr2=0x7fd28c01d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 --2- 172.21.15.119:0/1732488263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c01ce60 0x7fd28c01d270 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd29404ff70 tx=0x7fd29404ed70 comp rx=0 tx=0).stop 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- 172.21.15.119:0/1732488263 shutdown_connections 2024-09-13T16:29:18.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 --2- 172.21.15.119:0/1732488263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c01ce60 0x7fd28c01d270 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 --2- 172.21.15.119:0/1732488263 >> v2:172.21.15.145:3300/0 conn(0x7fd28c033c50 0x7fd28c005b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- 172.21.15.119:0/1732488263 >> 172.21.15.119:0/1732488263 conn(0x7fd28c036140 msgr2=0x7fd28c036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- 172.21.15.119:0/1732488263 shutdown_connections 2024-09-13T16:29:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.076+0000 7fd29885b700 1 -- 172.21.15.119:0/1732488263 wait complete. 2024-09-13T16:29:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 Processor -- start 2024-09-13T16:29:18.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 -- start start 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd28c01ce60 0x7fd28c0d5a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c033c50 0x7fd28c0e15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd28c04fc90 0x7fd28c0e1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0d8db0 con 0x7fd28c01ce60 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0d9090 con 0x7fd28c04fc90 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29885b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd28c0d9370 con 0x7fd28c033c50 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd292d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd28c01ce60 0x7fd28c0d5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd292d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd28c01ce60 0x7fd28c0d5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58676/0 (socket says 172.21.15.119:58676) 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd292d9d700 1 -- 172.21.15.119:0/3287613319 learned_addr learned my addr 172.21.15.119:0/3287613319 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29259c700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c033c50 0x7fd28c0e15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.080+0000 7fd29359e700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd28c04fc90 0x7fd28c0e1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd292d9d700 1 -- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c033c50 msgr2=0x7fd28c0e15f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd292d9d700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c033c50 0x7fd28c0e15f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd292d9d700 1 -- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd28c04fc90 msgr2=0x7fd28c0e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd292d9d700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd28c04fc90 0x7fd28c0e1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd292d9d700 1 -- 172.21.15.119:0/3287613319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29404f5c0 con 0x7fd28c01ce60 2024-09-13T16:29:18.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd292d9d700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd28c01ce60 0x7fd28c0d5a90 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd28400e9b0 tx=0x7fd28400ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd28b7fe700 1 -- 172.21.15.119:0/3287613319 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284009670 con 0x7fd28c01ce60 2024-09-13T16:29:18.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd28b7fe700 1 -- 172.21.15.119:0/3287613319 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28400c640 con 0x7fd28c01ce60 2024-09-13T16:29:18.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd28c0d9650 con 0x7fd28c01ce60 2024-09-13T16:29:18.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd28b7fe700 1 -- 172.21.15.119:0/3287613319 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28400c9b0 con 0x7fd28c01ce60 2024-09-13T16:29:18.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd28c0d5fd0 con 0x7fd28c01ce60 2024-09-13T16:29:18.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd28b7fe700 1 -- 172.21.15.119:0/3287613319 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd28400cb10 con 0x7fd28c01ce60 2024-09-13T16:29:18.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd28c0e2dc0 con 0x7fd28c01ce60 2024-09-13T16:29:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.084+0000 7fd28b7fe700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd274066760 0x7fd274068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.088+0000 7fd28b7fe700 1 -- 172.21.15.119:0/3287613319 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fd2840845c0 con 0x7fd28c01ce60 2024-09-13T16:29:18.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.088+0000 7fd29259c700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd274066760 0x7fd274068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.088+0000 7fd29259c700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd274066760 0x7fd274068c20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd28c04a8f0 tx=0x7fd27c0012f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.092+0000 7fd28b7fe700 1 -- 172.21.15.119:0/3287613319 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd284051770 con 0x7fd28c01ce60 2024-09-13T16:29:18.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.268+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd28c0d8ef0 con 0x7fd28c01ce60 2024-09-13T16:29:18.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.268+0000 7fd28b7fe700 1 -- 172.21.15.119:0/3287613319 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16969 (secure 0 0 0) 0x7fd28c0d8ef0 con 0x7fd28c01ce60 2024-09-13T16:29:18.274 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:18.275 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":70,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:08.646341+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:28:19.445072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_data","create_time":"2024-09-13T16:28:20.106803+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"9.17","mappings":[{"from":7,"to":2}]},{"pgid":"10.35","mappings":[{"from":4,"to":3}]},{"pgid":"10.36","mappings":[{"from":7,"to":3}]},{"pgid":"10.3c","mappings":[{"from":4,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd274066760 msgr2=0x7fd274068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd274066760 0x7fd274068c20 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fd28c04a8f0 tx=0x7fd27c0012f0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd28c01ce60 msgr2=0x7fd28c0d5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd28c01ce60 0x7fd28c0d5a90 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fd28400e9b0 tx=0x7fd28400ecc0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 shutdown_connections 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd274066760 0x7fd274068c20 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd28c04fc90 0x7fd28c0e1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd28c033c50 0x7fd28c0e15f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 --2- 172.21.15.119:0/3287613319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd28c01ce60 0x7fd28c0d5a90 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 >> 172.21.15.119:0/3287613319 conn(0x7fd28c036140 msgr2=0x7fd28c0357a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:18.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 shutdown_connections 2024-09-13T16:29:18.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.272+0000 7fd29885b700 1 -- 172.21.15.119:0/3287613319 wait complete. 2024-09-13T16:29:18.294 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:29:18.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.404+0000 7fc7dcecc700 1 Processor -- start 2024-09-13T16:29:18.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.404+0000 7fc7dcecc700 1 -- start start 2024-09-13T16:29:18.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc7d806dc40 0x7fc7d80b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d80b2280 con 0x7fc7d81594f0 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7d80cc890 con 0x7fc7d80b3630 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7d80b1970 con 0x7fc7d806dc40 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc7d806dc40 0x7fc7d80b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d77fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc7d81594f0 0x7fc7d806d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58986/0 (socket says 172.21.15.119:58986) 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc7d806dc40 0x7fc7d80b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58688/0 (socket says 172.21.15.119:58688) 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d77fe700 1 -- 172.21.15.119:0/2817452851 learned_addr learned my addr 172.21.15.119:0/2817452851 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d6ffd700 1 -- 172.21.15.119:0/2817452851 >> v1:172.21.15.145:6789/0 conn(0x7fc7d80b3630 legacy=0x7fc7d806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d6ffd700 1 -- 172.21.15.119:0/2817452851 >> v1:172.21.15.119:6789/0 conn(0x7fc7d81594f0 legacy=0x7fc7d806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d6ffd700 1 -- 172.21.15.119:0/2817452851 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d80b1c30 con 0x7fc7d806dc40 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d6ffd700 1 --2- 172.21.15.119:0/2817452851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d806dc40 0x7fc7d80b30f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4b9dc71d98bdc1e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d67fc700 1 -- 172.21.15.119:0/2817452851 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c800e070 con 0x7fc7d806dc40 2024-09-13T16:29:18.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d67fc700 1 -- 172.21.15.119:0/2817452851 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7c800bbd0 con 0x7fc7d806dc40 2024-09-13T16:29:18.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7d67fc700 1 -- 172.21.15.119:0/2817452851 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c8012430 con 0x7fc7d806dc40 2024-09-13T16:29:18.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2817452851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d806dc40 msgr2=0x7fc7d80b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2817452851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d806dc40 0x7fc7d80b30f0 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009d90 tx=0x7fc7c800b330 comp rx=0 tx=0).stop 2024-09-13T16:29:18.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2817452851 shutdown_connections 2024-09-13T16:29:18.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2817452851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d806dc40 0x7fc7d80b30f0 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2817452851 >> 172.21.15.119:0/2817452851 conn(0x7fc7d806a350 msgr2=0x7fc7d806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:18.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2817452851 shutdown_connections 2024-09-13T16:29:18.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.408+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2817452851 wait complete. 2024-09-13T16:29:18.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 Processor -- start 2024-09-13T16:29:18.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 -- start start 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7d806dc40 0x7fc7d81ef780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7d80b3630 0x7fc7d8147940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d81594f0 0x7fc7d80ae660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d81efcc0 con 0x7fc7d81594f0 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d81488c0 con 0x7fc7d806dc40 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d8145d50 con 0x7fc7d80b3630 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d81594f0 0x7fc7d80ae660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d81594f0 0x7fc7d80ae660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58690/0 (socket says 172.21.15.119:58690) 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 -- 172.21.15.119:0/2013290407 learned_addr learned my addr 172.21.15.119:0/2013290407 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d6ffd700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7d80b3630 0x7fc7d8147940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d77fe700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7d806dc40 0x7fc7d81ef780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 -- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7d80b3630 msgr2=0x7fc7d8147940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7d80b3630 0x7fc7d8147940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 -- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7d806dc40 msgr2=0x7fc7d81ef780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7d806dc40 0x7fc7d81ef780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 -- 172.21.15.119:0/2013290407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c8009a70 con 0x7fc7d81594f0 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d77fe700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7d806dc40 0x7fc7d81ef780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7d7fff700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d81594f0 0x7fc7d80ae660 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c500 tx=0x7fc7d000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7cffff700 1 -- 172.21.15.119:0/2013290407 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00052d0 con 0x7fc7d81594f0 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7cffff700 1 -- 172.21.15.119:0/2013290407 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7d0005430 con 0x7fc7d81594f0 2024-09-13T16:29:18.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d8148300 con 0x7fc7d81594f0 2024-09-13T16:29:18.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d81459e0 con 0x7fc7d81594f0 2024-09-13T16:29:18.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.412+0000 7fc7cffff700 1 -- 172.21.15.119:0/2013290407 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7d00055a0 con 0x7fc7d81594f0 2024-09-13T16:29:18.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.416+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d80b30f0 con 0x7fc7d81594f0 2024-09-13T16:29:18.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.416+0000 7fc7cffff700 1 -- 172.21.15.119:0/2013290407 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc7d00036a0 con 0x7fc7d81594f0 2024-09-13T16:29:18.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.416+0000 7fc7cffff700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7b8066760 0x7fc7b8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.416+0000 7fc7cffff700 1 -- 172.21.15.119:0/2013290407 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fc7d00836a0 con 0x7fc7d81594f0 2024-09-13T16:29:18.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.416+0000 7fc7d77fe700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7b8066760 0x7fc7b8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.416+0000 7fc7d77fe700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7b8066760 0x7fc7b8068c20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0007fd0 tx=0x7fc7c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.424+0000 7fc7cffff700 1 -- 172.21.15.119:0/2013290407 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc7d00507c0 con 0x7fc7d81594f0 2024-09-13T16:29:18.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.632+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc7d8145e90 con 0x7fc7d81594f0 2024-09-13T16:29:18.637 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:29:18.632+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:29:18.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7cffff700 1 -- 172.21.15.119:0/2013290407 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v35) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc7d0081e60 con 0x7fc7d81594f0 2024-09-13T16:29:18.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7b8066760 msgr2=0x7fc7b8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7b8066760 0x7fc7b8068c20 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc7c0007fd0 tx=0x7fc7c0009040 comp rx=0 tx=0).stop 2024-09-13T16:29:18.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d81594f0 msgr2=0x7fc7d80ae660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d81594f0 0x7fc7d80ae660 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7fc7d000c500 tx=0x7fc7d000c810 comp rx=0 tx=0).stop 2024-09-13T16:29:18.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 shutdown_connections 2024-09-13T16:29:18.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc7b8066760 0x7fc7b8068c20 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc7d81594f0 0x7fc7d80ae660 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7d80b3630 0x7fc7d8147940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 --2- 172.21.15.119:0/2013290407 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7d806dc40 0x7fc7d81ef780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 >> 172.21.15.119:0/2013290407 conn(0x7fc7d806a350 msgr2=0x7fc7d80afa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:18.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 shutdown_connections 2024-09-13T16:29:18.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.648+0000 7fc7dcecc700 1 -- 172.21.15.119:0/2013290407 wait complete. 2024-09-13T16:29:18.658 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:29:18.658 INFO:tasks.ceph.mds.d.smithi145.stderr: -5192> 2024-09-13T16:28:27.056+0000 7f9e49498700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:29:18.659 INFO:tasks.ceph.mds.d.smithi145.stderr: -3526> 2024-09-13T16:28:34.988+0000 7f9e45c91700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v2 ap=1 s=0 n(v0 1=1+0) (iauth excl) (ifile excl) (ixattr excl) (iversion lock) cr={6723=0-4194304@1} caps={6723=pAsxLsXsxFsxcrwb/pAsxXsxFxwb@2},l=6723 | request=0 caps=1 dirtyparent=0 dirty=0 authpin=1 scrubqueue=0 0x55f65c319600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v2>,<0x1/dir_test v4>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:29:18.660 INFO:tasks.ceph.mds.d.smithi145.stderr: -1724> 2024-09-13T16:29:06.499+0000 7f9e45c91700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [2,head] /dir_test/file auth v4 ap=1 DIRTYPARENT s=0 n(v0 rc2024-09-13T16:28:33.800048+0000 1=1+0) (iversion lock) caps={6723=pAsLsXsFs/-@2},l=6723(-1) | ptrwaiter=0 request=0 lock=0 caps=1 dirtyparent=1 dirty=1 authpin=1 scrubqueue=0 0x55f65c319600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(10)0x10000000001:[<0x10000000000/file v4>,<0x1/dir_test v7>]//[]","error_str":"failed to decode on-disk backtrace (26 bytes)!"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:29:18.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:29:18.690 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:29:18.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.780+0000 7f6c39a4d700 1 Processor -- start 2024-09-13T16:29:18.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.780+0000 7f6c39a4d700 1 -- start start 2024-09-13T16:29:18.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6c3415cfb0 0x7f6c3406ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3406b2d0 0x7f6c3406b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c340ce880 con 0x7f6c3406bc50 2024-09-13T16:29:18.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c340cee40 con 0x7f6c3415cfb0 2024-09-13T16:29:18.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3406c100 con 0x7f6c3406b2d0 2024-09-13T16:29:18.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c327fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3406b2d0 0x7f6c3406b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c32ffd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6c3415cfb0 0x7f6c3406ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c337fe700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6c3406bc50 0x7f6c340cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:55140/0 (socket says 172.21.15.119:55140) 2024-09-13T16:29:18.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c337fe700 1 -- 172.21.15.119:0/456693647 learned_addr learned my addr 172.21.15.119:0/456693647 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:18.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c31ffb700 1 -- 172.21.15.119:0/456693647 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207774527 0 0) 0x7f6c340ce880 con 0x7f6c3406bc50 2024-09-13T16:29:18.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c31ffb700 1 -- 172.21.15.119:0/456693647 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c140035d0 con 0x7f6c3406bc50 2024-09-13T16:29:18.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c327fc700 1 -- 172.21.15.119:0/456693647 >> v2:172.21.15.145:3300/0 conn(0x7f6c3415cfb0 msgr2=0x7f6c3406ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:18.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c327fc700 1 --2- 172.21.15.119:0/456693647 >> v2:172.21.15.145:3300/0 conn(0x7f6c3415cfb0 0x7f6c3406ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c327fc700 1 -- 172.21.15.119:0/456693647 >> v1:172.21.15.145:6789/0 conn(0x7f6c3406bc50 legacy=0x7f6c340cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c327fc700 1 -- 172.21.15.119:0/456693647 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c340ceb60 con 0x7f6c3406b2d0 2024-09-13T16:29:18.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c32ffd700 1 --2- 172.21.15.119:0/456693647 >> v2:172.21.15.145:3300/0 conn(0x7f6c3415cfb0 0x7f6c3406ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c327fc700 1 --2- 172.21.15.119:0/456693647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3406b2d0 0x7f6c3406b710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c009d90 tx=0x7f6c1c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=903ceafadddcbd0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c31ffb700 1 -- 172.21.15.119:0/456693647 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00e070 con 0x7f6c3406b2d0 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c31ffb700 1 -- 172.21.15.119:0/456693647 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c00bb20 con 0x7f6c3406b2d0 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c31ffb700 1 -- 172.21.15.119:0/456693647 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00be90 con 0x7f6c3406b2d0 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- 172.21.15.119:0/456693647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3406b2d0 msgr2=0x7f6c3406b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/456693647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3406b2d0 0x7f6c3406b710 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c009d90 tx=0x7f6c1c00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- 172.21.15.119:0/456693647 shutdown_connections 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/456693647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3406b2d0 0x7f6c3406b710 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/456693647 >> v2:172.21.15.145:3300/0 conn(0x7f6c3415cfb0 0x7f6c3406ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- 172.21.15.119:0/456693647 >> 172.21.15.119:0/456693647 conn(0x7f6c340d2da0 msgr2=0x7f6c340d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- 172.21.15.119:0/456693647 shutdown_connections 2024-09-13T16:29:18.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.784+0000 7f6c39a4d700 1 -- 172.21.15.119:0/456693647 wait complete. 2024-09-13T16:29:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 Processor -- start 2024-09-13T16:29:18.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 -- start start 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c3406b2d0 0x7f6c340b6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c3406bc50 0x7f6c340b66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3415cfb0 0x7f6c340b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340d1220 con 0x7f6c3406bc50 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c340b7320 con 0x7f6c3406b2d0 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c39a4d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c3415f010 con 0x7f6c3415cfb0 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c327fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c3406bc50 0x7f6c340b66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c327fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c3406bc50 0x7f6c340b66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58700/0 (socket says 172.21.15.119:58700) 2024-09-13T16:29:18.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c337fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3415cfb0 0x7f6c340b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c327fc700 1 -- 172.21.15.119:0/994264707 learned_addr learned my addr 172.21.15.119:0/994264707 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:18.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.788+0000 7f6c32ffd700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c3406b2d0 0x7f6c340b6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c327fc700 1 -- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3415cfb0 msgr2=0x7f6c340b0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c327fc700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3415cfb0 0x7f6c340b0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c327fc700 1 -- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c3406b2d0 msgr2=0x7f6c340b6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c327fc700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c3406b2d0 0x7f6c340b6190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c327fc700 1 -- 172.21.15.119:0/994264707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240094b0 con 0x7f6c3406bc50 2024-09-13T16:29:18.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c32ffd700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c3406b2d0 0x7f6c340b6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c327fc700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c3406bc50 0x7f6c340b66d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c014040 tx=0x7f6c1c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c2bfff700 1 -- 172.21.15.119:0/994264707 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c007000 con 0x7f6c3406bc50 2024-09-13T16:29:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c2bfff700 1 -- 172.21.15.119:0/994264707 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c1c020070 con 0x7f6c3406bc50 2024-09-13T16:29:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c1c009a70 con 0x7f6c3406bc50 2024-09-13T16:29:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c2bfff700 1 -- 172.21.15.119:0/994264707 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c0098f0 con 0x7f6c3406bc50 2024-09-13T16:29:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c341f5f20 con 0x7f6c3406bc50 2024-09-13T16:29:18.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.792+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c3406b710 con 0x7f6c3406bc50 2024-09-13T16:29:18.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.796+0000 7f6c2bfff700 1 -- 172.21.15.119:0/994264707 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6c1c027480 con 0x7f6c3406bc50 2024-09-13T16:29:18.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.796+0000 7f6c2bfff700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c18066730 0x7f6c18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:18.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.796+0000 7f6c2bfff700 1 -- 172.21.15.119:0/994264707 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f6c1c022070 con 0x7f6c3406bc50 2024-09-13T16:29:18.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.796+0000 7f6c32ffd700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c18066730 0x7f6c18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:18.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.796+0000 7f6c32ffd700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c18066730 0x7f6c18068bf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6c240113d0 tx=0x7f6c24009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:18.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:18.804+0000 7f6c2bfff700 1 -- 172.21.15.119:0/994264707 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6c1c0b7db0 con 0x7f6c3406bc50 2024-09-13T16:29:19.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.008+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6c340b4800 con 0x7f6c3406bc50 2024-09-13T16:29:19.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.648+0000 7f6c2bfff700 1 -- 172.21.15.119:0/994264707 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v36) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6c1c056e20 con 0x7f6c3406bc50 2024-09-13T16:29:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c18066730 msgr2=0x7f6c18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c18066730 0x7f6c18068bf0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f6c240113d0 tx=0x7f6c24009240 comp rx=0 tx=0).stop 2024-09-13T16:29:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c3406bc50 msgr2=0x7f6c340b66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:19.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c3406bc50 0x7f6c340b66d0 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c014040 tx=0x7f6c1c004af0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 shutdown_connections 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c18066730 0x7f6c18068bf0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c3415cfb0 0x7f6c340b0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c3406bc50 0x7f6c340b66d0 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 --2- 172.21.15.119:0/994264707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c3406b2d0 0x7f6c340b6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 >> 172.21.15.119:0/994264707 conn(0x7f6c340d2da0 msgr2=0x7f6c340b4d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 shutdown_connections 2024-09-13T16:29:19.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.656+0000 7f6c39a4d700 1 -- 172.21.15.119:0/994264707 wait complete. 2024-09-13T16:29:19.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:29:19.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.796+0000 7f42057ee700 1 Processor -- start 2024-09-13T16:29:19.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.796+0000 7f42057ee700 1 -- start start 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f42000d0720 0x7f42000d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f42000d1670 0x7f42000cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42000d3260 con 0x7f420015cc40 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42000d0c70 con 0x7f42000d1670 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42000d3490 con 0x7f42000d0720 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f42000d1670 0x7f42000cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f42000d1670 0x7f42000cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58706/0 (socket says 172.21.15.119:58706) 2024-09-13T16:29:19.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 -- 172.21.15.119:0/1436239715 learned_addr learned my addr 172.21.15.119:0/1436239715 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:19.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f41ff7fe700 1 --2- 172.21.15.119:0/1436239715 >> v2:172.21.15.145:3300/0 conn(0x7f42000d0720 0x7f42000d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:19.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 -- 172.21.15.119:0/1436239715 >> v2:172.21.15.145:3300/0 conn(0x7f42000d0720 msgr2=0x7f42000d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:19.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 --2- 172.21.15.119:0/1436239715 >> v2:172.21.15.145:3300/0 conn(0x7f42000d0720 0x7f42000d1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 -- 172.21.15.119:0/1436239715 >> v1:172.21.15.145:6789/0 conn(0x7f420015cc40 legacy=0x7f42000cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:19.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 -- 172.21.15.119:0/1436239715 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42000cd640 con 0x7f42000d1670 2024-09-13T16:29:19.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f4204fed700 1 --2- 172.21.15.119:0/1436239715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f42000cdab0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f420008c050 tx=0x7f41f800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8dca75c599b5af server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f41feffd700 1 -- 172.21.15.119:0/1436239715 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f800e070 con 0x7f42000d1670 2024-09-13T16:29:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f41feffd700 1 -- 172.21.15.119:0/1436239715 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f8003a40 con 0x7f42000d1670 2024-09-13T16:29:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f41feffd700 1 -- 172.21.15.119:0/1436239715 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f8003df0 con 0x7f42000d1670 2024-09-13T16:29:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- 172.21.15.119:0/1436239715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 msgr2=0x7f42000cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 --2- 172.21.15.119:0/1436239715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f42000cdab0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f420008c050 tx=0x7f41f800b330 comp rx=0 tx=0).stop 2024-09-13T16:29:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- 172.21.15.119:0/1436239715 shutdown_connections 2024-09-13T16:29:19.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 --2- 172.21.15.119:0/1436239715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f42000cdab0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 --2- 172.21.15.119:0/1436239715 >> v2:172.21.15.145:3300/0 conn(0x7f42000d0720 0x7f42000d1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- 172.21.15.119:0/1436239715 >> 172.21.15.119:0/1436239715 conn(0x7f420009be40 msgr2=0x7f42000d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- 172.21.15.119:0/1436239715 shutdown_connections 2024-09-13T16:29:19.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 -- 172.21.15.119:0/1436239715 wait complete. 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.800+0000 7f42057ee700 1 Processor -- start 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 -- start start 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42000d0720 0x7f42001f5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f4200149450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f420015cc40 0x7f420015d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4200146cb0 con 0x7f42000d1670 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42000b34a0 con 0x7f42000d0720 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42001f64b0 con 0x7f420015cc40 2024-09-13T16:29:19.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f4200149450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f4200149450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58710/0 (socket says 172.21.15.119:58710) 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 -- 172.21.15.119:0/146556671 learned_addr learned my addr 172.21.15.119:0/146556671 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f4204fed700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f420015cc40 0x7f420015d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41fffff700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42000d0720 0x7f42001f5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 -- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f420015cc40 msgr2=0x7f420015d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f420015cc40 0x7f420015d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 -- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42000d0720 msgr2=0x7f42001f5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42000d0720 0x7f42001f5bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 -- 172.21.15.119:0/146556671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41f8009b30 con 0x7f42000d1670 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f4204fed700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f420015cc40 0x7f420015d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41ff7fe700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f4200149450 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e810 tx=0x7f41f400ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41fcff9700 1 -- 172.21.15.119:0/146556671 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4013610 con 0x7f42000d1670 2024-09-13T16:29:19.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41fcff9700 1 -- 172.21.15.119:0/146556671 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41f4013770 con 0x7f42000d1670 2024-09-13T16:29:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42001f6790 con 0x7f42000d1670 2024-09-13T16:29:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f41fcff9700 1 -- 172.21.15.119:0/146556671 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41f4013b20 con 0x7f42000d1670 2024-09-13T16:29:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42000b3130 con 0x7f42000d1670 2024-09-13T16:29:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.804+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f420006ab20 con 0x7f42000d1670 2024-09-13T16:29:19.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.808+0000 7f41fcff9700 1 -- 172.21.15.119:0/146556671 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f41f4013c80 con 0x7f42000d1670 2024-09-13T16:29:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.808+0000 7f41fcff9700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f41ec066730 0x7f41ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.808+0000 7f41fcff9700 1 -- 172.21.15.119:0/146556671 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f41f4084130 con 0x7f42000d1670 2024-09-13T16:29:19.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.808+0000 7f41fffff700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f41ec066730 0x7f41ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:19.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.808+0000 7f41fffff700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f41ec066730 0x7f41ec068bf0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f42001f5b10 tx=0x7f41f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:19.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.812+0000 7f41fcff9700 1 -- 172.21.15.119:0/146556671 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f41f40512a0 con 0x7f42000d1670 2024-09-13T16:29:19.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:19.984+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f42000b35e0 con 0x7f42000d1670 2024-09-13T16:29:20.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.720+0000 7f41fcff9700 1 -- 172.21.15.119:0/146556671 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) v1 ==== 181+0+0 (secure 0 0 0) 0x7f41f40548c0 con 0x7f42000d1670 2024-09-13T16:29:20.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f41ec066730 msgr2=0x7f41ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f41ec066730 0x7f41ec068bf0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f42001f5b10 tx=0x7f41f0008040 comp rx=0 tx=0).stop 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 msgr2=0x7f4200149450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f4200149450 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f41f400e810 tx=0x7f41f400ebd0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 shutdown_connections 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f41ec066730 0x7f41ec068bf0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f420015cc40 0x7f420015d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42000d1670 0x7f4200149450 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 --2- 172.21.15.119:0/146556671 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42000d0720 0x7f42001f5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 >> 172.21.15.119:0/146556671 conn(0x7f420009be40 msgr2=0x7f42000cfac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 shutdown_connections 2024-09-13T16:29:20.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.728+0000 7f42057ee700 1 -- 172.21.15.119:0/146556671 wait complete. 2024-09-13T16:29:20.735 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:29:20.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:29:20.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847ffff700 1 Processor -- start 2024-09-13T16:29:20.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847ffff700 1 -- start start 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847ffff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f848015c900 0x7f848015cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f84800d35a0 0x7f84800b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847ffff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84800d3a10 con 0x7f84800b26e0 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847ffff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84800d0650 con 0x7f84800d35a0 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847ffff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84800ce5a0 con 0x7f848015c900 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.864+0000 7f847e7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f84800d35a0 0x7f84800b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847e7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f84800d35a0 0x7f84800b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58720/0 (socket says 172.21.15.119:58720) 2024-09-13T16:29:20.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847e7fc700 1 -- 172.21.15.119:0/3191221004 learned_addr learned my addr 172.21.15.119:0/3191221004 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:20.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847effd700 1 --2- 172.21.15.119:0/3191221004 >> v2:172.21.15.145:3300/0 conn(0x7f848015c900 0x7f848015cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:20.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847e7fc700 1 -- 172.21.15.119:0/3191221004 >> v2:172.21.15.145:3300/0 conn(0x7f848015c900 msgr2=0x7f848015cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847e7fc700 1 --2- 172.21.15.119:0/3191221004 >> v2:172.21.15.145:3300/0 conn(0x7f848015c900 0x7f848015cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847dffb700 1 -- 172.21.15.119:0/3191221004 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132248299 0 0) 0x7f8478001490 con 0x7f84800b26e0 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847e7fc700 1 -- 172.21.15.119:0/3191221004 >> v1:172.21.15.145:6789/0 conn(0x7f84800b26e0 legacy=0x7f84800d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847e7fc700 1 -- 172.21.15.119:0/3191221004 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84800ce880 con 0x7f84800d35a0 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847e7fc700 1 --2- 172.21.15.119:0/3191221004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800b21a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f848008b280 tx=0x7f8470009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bf6b27872a3f0ba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847dffb700 1 -- 172.21.15.119:0/3191221004 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470011830 con 0x7f84800d35a0 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847dffb700 1 -- 172.21.15.119:0/3191221004 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8470011990 con 0x7f84800d35a0 2024-09-13T16:29:20.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847dffb700 1 -- 172.21.15.119:0/3191221004 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470011d20 con 0x7f84800d35a0 2024-09-13T16:29:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 -- 172.21.15.119:0/3191221004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 msgr2=0x7f84800b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 --2- 172.21.15.119:0/3191221004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800b21a0 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f848008b280 tx=0x7f8470009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 -- 172.21.15.119:0/3191221004 shutdown_connections 2024-09-13T16:29:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 --2- 172.21.15.119:0/3191221004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800b21a0 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 --2- 172.21.15.119:0/3191221004 >> v2:172.21.15.145:3300/0 conn(0x7f848015c900 0x7f848015cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 -- 172.21.15.119:0/3191221004 >> 172.21.15.119:0/3191221004 conn(0x7f84800d2a80 msgr2=0x7f84800d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 -- 172.21.15.119:0/3191221004 shutdown_connections 2024-09-13T16:29:20.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.868+0000 7f847ffff700 1 -- 172.21.15.119:0/3191221004 wait complete. 2024-09-13T16:29:20.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 Processor -- start 2024-09-13T16:29:20.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 -- start start 2024-09-13T16:29:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84800b26e0 0x7f8480153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f848015c900 0x7f84800b06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8480159370 con 0x7f84800d35a0 2024-09-13T16:29:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84801587a0 con 0x7f848015c900 2024-09-13T16:29:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8480154020 con 0x7f84800b26e0 2024-09-13T16:29:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58730/0 (socket says 172.21.15.119:58730) 2024-09-13T16:29:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 -- 172.21.15.119:0/1837630875 learned_addr learned my addr 172.21.15.119:0/1837630875 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847effd700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84800b26e0 0x7f8480153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847f7fe700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f848015c900 0x7f84800b06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:20.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 -- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84800b26e0 msgr2=0x7f8480153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:20.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84800b26e0 0x7f8480153ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 -- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f848015c900 msgr2=0x7f84800b06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:20.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f848015c900 0x7f84800b06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:20.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 -- 172.21.15.119:0/1837630875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8470009570 con 0x7f84800d35a0 2024-09-13T16:29:20.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847e7fc700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800cb740 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8470000c00 tx=0x7f8470004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:20.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f84677fe700 1 -- 172.21.15.119:0/1837630875 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470004510 con 0x7f84800d35a0 2024-09-13T16:29:20.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f84677fe700 1 -- 172.21.15.119:0/1837630875 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847001e070 con 0x7f84800d35a0 2024-09-13T16:29:20.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8480158a40 con 0x7f84800d35a0 2024-09-13T16:29:20.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f84677fe700 1 -- 172.21.15.119:0/1837630875 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8470003900 con 0x7f84800d35a0 2024-09-13T16:29:20.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.872+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8480159000 con 0x7f84800d35a0 2024-09-13T16:29:20.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.876+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8480074560 con 0x7f84800d35a0 2024-09-13T16:29:20.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.876+0000 7f84677fe700 1 -- 172.21.15.119:0/1837630875 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8470003a60 con 0x7f84800d35a0 2024-09-13T16:29:20.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.876+0000 7f84677fe700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f846c066730 0x7f846c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:20.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.876+0000 7f84677fe700 1 -- 172.21.15.119:0/1837630875 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 5764+0+0 (secure 0 0 0) 0x7f8470084be0 con 0x7f84800d35a0 2024-09-13T16:29:20.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.876+0000 7f847effd700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f846c066730 0x7f846c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:20.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.876+0000 7f847effd700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f846c066730 0x7f846c068bf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8474009b50 tx=0x7f8474009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:20.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:20.888+0000 7f84677fe700 1 -- 172.21.15.119:0/1837630875 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8470051fa0 con 0x7f84800d35a0 2024-09-13T16:29:21.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.060+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f84800b21a0 con 0x7f84800d35a0 2024-09-13T16:29:21.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.736+0000 7f84677fe700 1 -- 172.21.15.119:0/1837630875 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) v1 ==== 169+0+0 (secure 0 0 0) 0x7f84700150d0 con 0x7f84800d35a0 2024-09-13T16:29:21.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.744+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f846c066730 msgr2=0x7f846c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:21.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.744+0000 7f847ffff700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f846c066730 0x7f846c068bf0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8474009b50 tx=0x7f8474009240 comp rx=0 tx=0).stop 2024-09-13T16:29:21.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.744+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 msgr2=0x7f84800cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:21.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.744+0000 7f847ffff700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800cb740 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f8470000c00 tx=0x7f8470004770 comp rx=0 tx=0).stop 2024-09-13T16:29:21.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 shutdown_connections 2024-09-13T16:29:21.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f846c066730 0x7f846c068bf0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f848015c900 0x7f84800b06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f84800d35a0 0x7f84800cb740 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 --2- 172.21.15.119:0/1837630875 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f84800b26e0 0x7f8480153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 >> 172.21.15.119:0/1837630875 conn(0x7f84800d2a80 msgr2=0x7f848006ac90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:21.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 shutdown_connections 2024-09-13T16:29:21.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.748+0000 7f847ffff700 1 -- 172.21.15.119:0/1837630875 wait complete. 2024-09-13T16:29:21.754 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:29:21.767 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T16:29:21.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.872+0000 7f032ffff700 1 Processor -- start 2024-09-13T16:29:21.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.872+0000 7f032ffff700 1 -- start start 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.872+0000 7f032ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300ad9a0 0x7f0330159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.872+0000 7f032ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f03300b1860 0x7f033006c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.872+0000 7f032ffff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03300b1dd0 con 0x7f033006c890 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.872+0000 7f032ffff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03300b2fd0 con 0x7f03300b1860 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f033015a520 con 0x7f03300ad9a0 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f0327fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f03300b1860 0x7f033006c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f033006c890 0x7f033006bac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58992/0 (socket says 172.21.15.119:58992) 2024-09-13T16:29:21.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032f7fe700 1 -- 172.21.15.119:0/3101058819 learned_addr learned my addr 172.21.15.119:0/3101058819 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:21.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032effd700 1 --2- 172.21.15.119:0/3101058819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300ad9a0 0x7f0330159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:21.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f0327fff700 1 -- 172.21.15.119:0/3101058819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300ad9a0 msgr2=0x7f0330159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:21.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f0327fff700 1 --2- 172.21.15.119:0/3101058819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300ad9a0 0x7f0330159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f0327fff700 1 -- 172.21.15.119:0/3101058819 >> v1:172.21.15.119:6789/0 conn(0x7f033006c890 legacy=0x7f033006bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:21.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f0327fff700 1 -- 172.21.15.119:0/3101058819 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03300b20b0 con 0x7f03300b1860 2024-09-13T16:29:21.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032e7fc700 1 -- 172.21.15.119:0/3101058819 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251456541 0 0) 0x7f03300b1dd0 con 0x7f033006c890 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032effd700 1 --2- 172.21.15.119:0/3101058819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300ad9a0 0x7f0330159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f0327fff700 1 --2- 172.21.15.119:0/3101058819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300b1860 0x7f033006c350 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0318009850 tx=0x7f0318009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2645e6ced5510dbf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032e7fc700 1 -- 172.21.15.119:0/3101058819 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318011830 con 0x7f03300b1860 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032e7fc700 1 -- 172.21.15.119:0/3101058819 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0318011990 con 0x7f03300b1860 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032e7fc700 1 -- 172.21.15.119:0/3101058819 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318011d20 con 0x7f03300b1860 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 -- 172.21.15.119:0/3101058819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300b1860 msgr2=0x7f033006c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 --2- 172.21.15.119:0/3101058819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300b1860 0x7f033006c350 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f0318009850 tx=0x7f0318009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 -- 172.21.15.119:0/3101058819 shutdown_connections 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 --2- 172.21.15.119:0/3101058819 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300b1860 0x7f033006c350 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 --2- 172.21.15.119:0/3101058819 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300ad9a0 0x7f0330159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 -- 172.21.15.119:0/3101058819 >> 172.21.15.119:0/3101058819 conn(0x7f03300b0980 msgr2=0x7f03300b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 -- 172.21.15.119:0/3101058819 shutdown_connections 2024-09-13T16:29:21.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.876+0000 7f032ffff700 1 -- 172.21.15.119:0/3101058819 wait complete. 2024-09-13T16:29:21.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 Processor -- start 2024-09-13T16:29:21.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 -- start start 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f033006c890 0x7f03301498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300ad9a0 0x7f03300ccae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300b1860 0x7f0330157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0330152b80 con 0x7f03300ad9a0 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0330149df0 con 0x7f033006c890 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03300ce050 con 0x7f03300b1860 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300ad9a0 0x7f03300ccae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300ad9a0 0x7f03300ccae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58742/0 (socket says 172.21.15.119:58742) 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 -- 172.21.15.119:0/2557297592 learned_addr learned my addr 172.21.15.119:0/2557297592 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:21.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032effd700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f033006c890 0x7f03301498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032f7fe700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300b1860 0x7f0330157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 -- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300b1860 msgr2=0x7f0330157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300b1860 0x7f0330157d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 -- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f033006c890 msgr2=0x7f03301498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f033006c890 0x7f03301498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 -- 172.21.15.119:0/2557297592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03180094b0 con 0x7f03300ad9a0 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032f7fe700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300b1860 0x7f0330157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f0327fff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300ad9a0 0x7f03300ccae0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0318014040 tx=0x7f0318004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032cff9700 1 -- 172.21.15.119:0/2557297592 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03180036c0 con 0x7f03300ad9a0 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f033015ac50 con 0x7f03300ad9a0 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032cff9700 1 -- 172.21.15.119:0/2557297592 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f031801e070 con 0x7f03300ad9a0 2024-09-13T16:29:21.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03301527a0 con 0x7f03300ad9a0 2024-09-13T16:29:21.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.880+0000 7f032cff9700 1 -- 172.21.15.119:0/2557297592 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0318003a20 con 0x7f03300ad9a0 2024-09-13T16:29:21.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.884+0000 7f032cff9700 1 -- 172.21.15.119:0/2557297592 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0318003b80 con 0x7f03300ad9a0 2024-09-13T16:29:21.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.884+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03300add80 con 0x7f03300ad9a0 2024-09-13T16:29:21.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.884+0000 7f032cff9700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0310066730 0x7f0310068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:21.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.884+0000 7f032cff9700 1 -- 172.21.15.119:0/2557297592 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f03180852b0 con 0x7f03300ad9a0 2024-09-13T16:29:21.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.884+0000 7f032effd700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0310066730 0x7f0310068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:21.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.884+0000 7f032effd700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0310066730 0x7f0310068bf0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0320006fd0 tx=0x7f0320008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:21.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:21.892+0000 7f032cff9700 1 -- 172.21.15.119:0/2557297592 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f03180527f0 con 0x7f03300ad9a0 2024-09-13T16:29:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.060+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f03301531c0 con 0x7f03300ad9a0 2024-09-13T16:29:22.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032cff9700 1 -- 172.21.15.119:0/2557297592 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0318055e10 con 0x7f03300ad9a0 2024-09-13T16:29:22.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0310066730 msgr2=0x7f0310068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0310066730 0x7f0310068bf0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f0320006fd0 tx=0x7f0320008040 comp rx=0 tx=0).stop 2024-09-13T16:29:22.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300ad9a0 msgr2=0x7f03300ccae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300ad9a0 0x7f03300ccae0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f0318014040 tx=0x7f0318004770 comp rx=0 tx=0).stop 2024-09-13T16:29:22.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 shutdown_connections 2024-09-13T16:29:22.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0310066730 0x7f0310068bf0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f03300b1860 0x7f0330157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f03300ad9a0 0x7f03300ccae0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 --2- 172.21.15.119:0/2557297592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f033006c890 0x7f03301498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.068+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 >> 172.21.15.119:0/2557297592 conn(0x7f03300b0980 msgr2=0x7f0330071380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:22.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.072+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 shutdown_connections 2024-09-13T16:29:22.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.072+0000 7f032ffff700 1 -- 172.21.15.119:0/2557297592 wait complete. 2024-09-13T16:29:22.089 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair' 2024-09-13T16:29:22.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.196+0000 7ff2eb59e700 1 Processor -- start 2024-09-13T16:29:22.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.196+0000 7ff2eb59e700 1 -- start start 2024-09-13T16:29:22.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.196+0000 7ff2eb59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff2ec0b3800 0x7ff2ec06e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.196+0000 7ff2eb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec0b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.196+0000 7ff2eb59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec0ae230 con 0x7ff2ec0b4120 2024-09-13T16:29:22.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.196+0000 7ff2eb59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2ec06f540 con 0x7ff2ec0b3800 2024-09-13T16:29:22.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.196+0000 7ff2eb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec06f820 con 0x7ff2ec06edd0 2024-09-13T16:29:22.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ead9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff2ec0b4120 0x7ff2ec0ae6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59002/0 (socket says 172.21.15.119:59002) 2024-09-13T16:29:22.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2e9d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec0b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ead9d700 1 -- 172.21.15.119:0/1815831212 learned_addr learned my addr 172.21.15.119:0/1815831212 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:22.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ea59c700 1 --2- 172.21.15.119:0/1815831212 >> v2:172.21.15.145:3300/0 conn(0x7ff2ec0b3800 0x7ff2ec06e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ea59c700 1 -- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 msgr2=0x7ff2ec0b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:22.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ea59c700 1 --2- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec0b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2e959a700 1 -- 172.21.15.119:0/1815831212 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3817261865 0 0) 0x7ff2ec0ae230 con 0x7ff2ec0b4120 2024-09-13T16:29:22.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ea59c700 1 -- 172.21.15.119:0/1815831212 >> v1:172.21.15.119:6789/0 conn(0x7ff2ec0b4120 legacy=0x7ff2ec0ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ea59c700 1 -- 172.21.15.119:0/1815831212 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec06fb00 con 0x7ff2ec0b3800 2024-09-13T16:29:22.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2e9d9b700 1 --2- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec0b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:22.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2ea59c700 1 --2- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec06e890 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009d90 tx=0x7ff2dc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38695eae39e9671b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:22.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2e959a700 1 -- 172.21.15.119:0/1815831212 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc00e070 con 0x7ff2ec0b3800 2024-09-13T16:29:22.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2e959a700 1 -- 172.21.15.119:0/1815831212 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2dc00bb20 con 0x7ff2ec0b3800 2024-09-13T16:29:22.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2e959a700 1 -- 172.21.15.119:0/1815831212 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2dc012430 con 0x7ff2ec0b3800 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 -- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b3800 msgr2=0x7ff2ec06e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec06e890 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc009d90 tx=0x7ff2dc00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 -- 172.21.15.119:0/1815831212 shutdown_connections 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec0b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/1815831212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec06e890 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 -- 172.21.15.119:0/1815831212 >> 172.21.15.119:0/1815831212 conn(0x7ff2ec0b5d20 msgr2=0x7ff2ec0b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 -- 172.21.15.119:0/1815831212 shutdown_connections 2024-09-13T16:29:22.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.200+0000 7ff2eb59e700 1 -- 172.21.15.119:0/1815831212 wait complete. 2024-09-13T16:29:22.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 Processor -- start 2024-09-13T16:29:22.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 -- start start 2024-09-13T16:29:22.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec09d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec14aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b4120 0x7ff2ec151980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec09d780 con 0x7ff2ec0b3800 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec09d9d0 con 0x7ff2ec0b4120 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2ec09dcb0 con 0x7ff2ec06edd0 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec14aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec14aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:58758/0 (socket says 172.21.15.119:58758) 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 -- 172.21.15.119:0/488396961 learned_addr learned my addr 172.21.15.119:0/488396961 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2ead9d700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b4120 0x7ff2ec151980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2ea59c700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec09d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 -- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 msgr2=0x7ff2ec09d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec09d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 -- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b4120 msgr2=0x7ff2ec151980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b4120 0x7ff2ec151980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 -- 172.21.15.119:0/488396961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d40094b0 con 0x7ff2ec0b3800 2024-09-13T16:29:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e9d9b700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec14aa40 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400ebb0 tx=0x7ff2d400eec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e2ffd700 1 -- 172.21.15.119:0/488396961 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400c850 con 0x7ff2ec0b3800 2024-09-13T16:29:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e2ffd700 1 -- 172.21.15.119:0/488396961 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2d400c9b0 con 0x7ff2ec0b3800 2024-09-13T16:29:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2dc009a70 con 0x7ff2ec0b3800 2024-09-13T16:29:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2e2ffd700 1 -- 172.21.15.119:0/488396961 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2d400cd40 con 0x7ff2ec0b3800 2024-09-13T16:29:22.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.204+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2ec151180 con 0x7ff2ec0b3800 2024-09-13T16:29:22.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.208+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2ec074560 con 0x7ff2ec0b3800 2024-09-13T16:29:22.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.208+0000 7ff2e2ffd700 1 -- 172.21.15.119:0/488396961 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff2d4013470 con 0x7ff2ec0b3800 2024-09-13T16:29:22.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.208+0000 7ff2e2ffd700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff2cc066730 0x7ff2cc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.208+0000 7ff2e2ffd700 1 -- 172.21.15.119:0/488396961 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5263+0+0 (secure 0 0 0) 0x7ff2d40840b0 con 0x7ff2ec0b3800 2024-09-13T16:29:22.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.208+0000 7ff2ea59c700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff2cc066730 0x7ff2cc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.212+0000 7ff2ea59c700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff2cc066730 0x7ff2cc068bf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc014040 tx=0x7ff2dc011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:22.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.216+0000 7ff2e2ffd700 1 -- 172.21.15.119:0/488396961 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff2d40514c0 con 0x7ff2ec0b3800 2024-09-13T16:29:22.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.448+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]} v 0) v1 -- 0x7ff2ec0b3be0 con 0x7ff2ec0b3800 2024-09-13T16:29:22.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.740+0000 7ff2e2ffd700 1 -- 172.21.15.119:0/488396961 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_backtrace_repair"]}]=0 v215) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff2d4054ae0 con 0x7ff2ec0b3800 2024-09-13T16:29:22.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff2cc066730 msgr2=0x7ff2cc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff2cc066730 0x7ff2cc068bf0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7ff2dc014040 tx=0x7ff2dc011000 comp rx=0 tx=0).stop 2024-09-13T16:29:22.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ec0b3800 msgr2=0x7ff2ec14aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec14aa40 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7ff2d400ebb0 tx=0x7ff2d400eec0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 shutdown_connections 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff2cc066730 0x7ff2cc068bf0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2ec0b4120 0x7ff2ec151980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2ec0b3800 0x7ff2ec14aa40 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 --2- 172.21.15.119:0/488396961 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2ec06edd0 0x7ff2ec09d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 >> 172.21.15.119:0/488396961 conn(0x7ff2ec0b5d20 msgr2=0x7ff2ec15d420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 shutdown_connections 2024-09-13T16:29:22.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.744+0000 7ff2eb59e700 1 -- 172.21.15.119:0/488396961 wait complete. 2024-09-13T16:29:22.767 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-09-13T16:29:22.767 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-09-13T16:29:22.767 INFO:tasks.cephfs_test_runner:test_health_status_after_dentry_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-09-13T16:29:22.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:22.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.876+0000 7f04fa886700 1 Processor -- start 2024-09-13T16:29:22.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.876+0000 7f04fa886700 1 -- start start 2024-09-13T16:29:22.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04f41576c0 0x7f04f406ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 0x7f04f406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04f415aa80 con 0x7f04f406ccd0 2024-09-13T16:29:22.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04f4157aa0 con 0x7f04f41576c0 2024-09-13T16:29:22.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40cf790 con 0x7f04f40cfc20 2024-09-13T16:29:22.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f04f406ccd0 0x7f04f406a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37790/0 (socket says 172.21.15.119:37790) 2024-09-13T16:29:22.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 -- 172.21.15.119:0/422617631 learned_addr learned my addr 172.21.15.119:0/422617631 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:22.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f3fff700 1 --2- 172.21.15.119:0/422617631 >> v2:172.21.15.145:3300/0 conn(0x7f04f41576c0 0x7f04f406ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f37fe700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 0x7f04f406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937045733 0 0) 0x7f04f415aa80 con 0x7f04f406ccd0 2024-09-13T16:29:22.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04dc0035d0 con 0x7f04f406ccd0 2024-09-13T16:29:22.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2312555491 0 0) 0x7f04dc0035d0 con 0x7f04f406ccd0 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 msgr2=0x7f04f406c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 0x7f04f406c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 >> v2:172.21.15.145:3300/0 conn(0x7f04f41576c0 msgr2=0x7f04f406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 --2- 172.21.15.119:0/422617631 >> v2:172.21.15.145:3300/0 conn(0x7f04f41576c0 0x7f04f406ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f415afb0 con 0x7f04f406ccd0 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f04ec0019f0 con 0x7f04f406ccd0 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 >> v1:172.21.15.119:6789/0 conn(0x7f04f406ccd0 legacy=0x7f04f406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc003a20 0x7f04dc003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04dc004870 0x7f04dc004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 0x7f04dc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f4157aa0 con 0x7f04dc004870 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40cf790 con 0x7f04dc003a20 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04dc00ba60 con 0x7f04f40cfc20 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f3fff700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc003a20 0x7f04dc003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 0x7f04dc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f37fe700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04dc004870 0x7f04dc004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 -- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc003a20 msgr2=0x7f04dc003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04dc003a20 0x7f04dc003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 -- 172.21.15.119:0/422617631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04dc004870 msgr2=0x7f04dc004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04dc004870 0x7f04dc004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 -- 172.21.15.119:0/422617631 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04dc0035d0 con 0x7f04f40cfc20 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 0x7f04dc005230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00a460 tx=0x7f04dc0058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9de73194f36c1d7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec011470 con 0x7f04f40cfc20 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec0115d0 con 0x7f04f40cfc20 2024-09-13T16:29:22.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f2ffd700 1 -- 172.21.15.119:0/422617631 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec011980 con 0x7f04f40cfc20 2024-09-13T16:29:22.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 -- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 msgr2=0x7f04dc005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 --2- 172.21.15.119:0/422617631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f40cfc20 0x7f04dc005230 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f04ec00a460 tx=0x7f04dc0058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04f8e23700 1 -- 172.21.15.119:0/422617631 reap_dead start 2024-09-13T16:29:22.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 -- 172.21.15.119:0/422617631 shutdown_connections 2024-09-13T16:29:22.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.880+0000 7f04fa886700 1 -- 172.21.15.119:0/422617631 >> 172.21.15.119:0/422617631 conn(0x7f04f40b3fd0 msgr2=0x7f04f40b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:22.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 -- 172.21.15.119:0/422617631 shutdown_connections 2024-09-13T16:29:22.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 -- 172.21.15.119:0/422617631 wait complete. 2024-09-13T16:29:22.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 Processor -- start 2024-09-13T16:29:22.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 -- start start 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4151640 0x7f04f40b1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f40aea10 0x7f04f40aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f4151a20 0x7f04f41e1740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40aeea0 con 0x7f04f4151a20 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40af180 con 0x7f04f40aea10 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04fa886700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04f40af460 con 0x7f04f4151640 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f4151a20 0x7f04f41e1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f4151a20 0x7f04f41e1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38834/0 (socket says 172.21.15.119:38834) 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 -- 172.21.15.119:0/937610192 learned_addr learned my addr 172.21.15.119:0/937610192 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f3fff700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4151640 0x7f04f40b1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f37fe700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f40aea10 0x7f04f40aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 -- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4151640 msgr2=0x7f04f40b1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4151640 0x7f04f40b1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 -- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f40aea10 msgr2=0x7f04f40aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f40aea10 0x7f04f40aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.884+0000 7f04f8e23700 1 -- 172.21.15.119:0/937610192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f40af740 con 0x7f04f4151a20 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f8e23700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f4151a20 0x7f04f41e1740 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00bbf0 tx=0x7f04ec001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f0ff9700 1 -- 172.21.15.119:0/937610192 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec011470 con 0x7f04f4151a20 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f0ff9700 1 -- 172.21.15.119:0/937610192 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04ec025670 con 0x7f04f4151a20 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04f40afa20 con 0x7f04f4151a20 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f0ff9700 1 -- 172.21.15.119:0/937610192 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04ec025a20 con 0x7f04f4151a20 2024-09-13T16:29:22.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04f41e24a0 con 0x7f04f4151a20 2024-09-13T16:29:22.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04f406ba40 con 0x7f04f4151a20 2024-09-13T16:29:22.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f0ff9700 1 -- 172.21.15.119:0/937610192 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04ec022070 con 0x7f04f4151a20 2024-09-13T16:29:22.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f0ff9700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066730 0x7f04e0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:22.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f0ff9700 1 -- 172.21.15.119:0/937610192 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f04ec0857a0 con 0x7f04f4151a20 2024-09-13T16:29:22.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.888+0000 7f04f3fff700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066730 0x7f04e0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:22.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.892+0000 7f04f3fff700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066730 0x7f04e0068bf0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f04f40b0fc0 tx=0x7f04e4009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:22.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:22.896+0000 7f04f0ff9700 1 -- 172.21.15.119:0/937610192 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f04ec052e20 con 0x7f04f4151a20 2024-09-13T16:29:23.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.064+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04f4156e70 con 0x7f04f4151a20 2024-09-13T16:29:23.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.064+0000 7f04f0ff9700 1 -- 172.21.15.119:0/937610192 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+13254 (secure 0 0 0) 0x7f04ec056440 con 0x7f04f4151a20 2024-09-13T16:29:23.070 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:23.070 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":73,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:21.727427+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6837/3224929185":"2024-09-14T16:29:18.636704+0000","172.21.15.145:6836/3224929185":"2024-09-14T16:29:18.636704+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:23.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066730 msgr2=0x7f04e0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066730 0x7f04e0068bf0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f04f40b0fc0 tx=0x7f04e4009240 comp rx=0 tx=0).stop 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f4151a20 msgr2=0x7f04f41e1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f4151a20 0x7f04f41e1740 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f04dc00bbf0 tx=0x7f04ec001360 comp rx=0 tx=0).stop 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 shutdown_connections 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f04e0066730 0x7f04e0068bf0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04f4151a20 0x7f04f41e1740 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04f40aea10 0x7f04f40aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 --2- 172.21.15.119:0/937610192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04f4151640 0x7f04f40b1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 >> 172.21.15.119:0/937610192 conn(0x7f04f40b3fd0 msgr2=0x7f04f40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 shutdown_connections 2024-09-13T16:29:23.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.068+0000 7f04fa886700 1 -- 172.21.15.119:0/937610192 wait complete. 2024-09-13T16:29:23.084 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:23.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5949202700 1 Processor -- start 2024-09-13T16:29:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5949202700 1 -- start start 2024-09-13T16:29:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5949202700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 0x7f594415a690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5949202700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594406da70 con 0x7f59440cd530 2024-09-13T16:29:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5949202700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f594406c9d0 con 0x7f594406def0 2024-09-13T16:29:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5949202700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594406be10 con 0x7f594415a2b0 2024-09-13T16:29:23.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5948a01700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f59440cd530 0x7f5944150020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37804/0 (socket says 172.21.15.119:37804) 2024-09-13T16:29:23.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5948a01700 1 -- 172.21.15.119:0/3015001624 learned_addr learned my addr 172.21.15.119:0/3015001624 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:23.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5943fff700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 0x7f594415a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226650949 0 0) 0x7f594406da70 con 0x7f59440cd530 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.196+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59300035d0 con 0x7f59440cd530 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285752902 0 0) 0x7f594406c9d0 con 0x7f594406def0 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f594406da70 con 0x7f594406def0 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2756087721 0 0) 0x7f59300035d0 con 0x7f59440cd530 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 msgr2=0x7f594415a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 0x7f594415a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 >> v1:172.21.15.145:6789/0 conn(0x7f594406def0 legacy=0x7f59440cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f594406ccb0 con 0x7f59440cd530 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5943fff700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 0x7f594415a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f5934001670 con 0x7f59440cd530 2024-09-13T16:29:23.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 >> v1:172.21.15.119:6789/0 conn(0x7f59440cd530 legacy=0x7f5944150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 0x7f5930003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004120 0x7f5930004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5930004db0 0x7f5930005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594406be10 con 0x7f5930004db0 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f593000ba90 con 0x7f5930004120 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f593000bc50 con 0x7f594415a2b0 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5930004db0 0x7f5930005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f59437fe700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004120 0x7f5930004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5943fff700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 0x7f5930003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 -- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 msgr2=0x7f5930003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:23.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415a2b0 0x7f5930003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 -- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004120 msgr2=0x7f5930004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004120 0x7f5930004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 -- 172.21.15.119:0/3015001624 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f593c009a70 con 0x7f5930004db0 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5930004db0 0x7f5930005230 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5934005150 tx=0x7f59300058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4eaae55230f4e75 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934004bc0 con 0x7f5930004db0 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5934005ba0 con 0x7f5930004db0 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5942ffd700 1 -- 172.21.15.119:0/3015001624 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59340080e0 con 0x7f5930004db0 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 -- 172.21.15.119:0/3015001624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5930004db0 msgr2=0x7f5930005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 --2- 172.21.15.119:0/3015001624 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5930004db0 0x7f5930005230 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f5934005150 tx=0x7f59300058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5948a01700 1 -- 172.21.15.119:0/3015001624 reap_dead start 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 -- 172.21.15.119:0/3015001624 shutdown_connections 2024-09-13T16:29:23.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 -- 172.21.15.119:0/3015001624 >> 172.21.15.119:0/3015001624 conn(0x7f59440b0cd0 msgr2=0x7f59440b10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:23.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 -- 172.21.15.119:0/3015001624 shutdown_connections 2024-09-13T16:29:23.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 -- 172.21.15.119:0/3015001624 wait complete. 2024-09-13T16:29:23.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 Processor -- start 2024-09-13T16:29:23.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.200+0000 7f5949202700 1 -- start start 2024-09-13T16:29:23.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004db0 0x7f5944152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415c590 0x7f5944072160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59440cacb0 0x7f59440cb130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f594409be50 con 0x7f59440cacb0 2024-09-13T16:29:23.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59440cb670 con 0x7f5930004db0 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5944152d40 con 0x7f594415c590 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59440cacb0 0x7f59440cb130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59440cacb0 0x7f59440cb130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38844/0 (socket says 172.21.15.119:38844) 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 -- 172.21.15.119:0/1022687222 learned_addr learned my addr 172.21.15.119:0/1022687222 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f59437fe700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415c590 0x7f5944072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5943fff700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004db0 0x7f5944152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 -- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415c590 msgr2=0x7f5944072160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415c590 0x7f5944072160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 -- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004db0 msgr2=0x7f5944152800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004db0 0x7f5944152800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 -- 172.21.15.119:0/1022687222 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59440726a0 con 0x7f59440cacb0 2024-09-13T16:29:23.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f59437fe700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415c590 0x7f5944072160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5948a01700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59440cacb0 0x7f59440cb130 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5934011000 tx=0x7f5934004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5940ff9700 1 -- 172.21.15.119:0/1022687222 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934001ae0 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5940ff9700 1 -- 172.21.15.119:0/1022687222 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5934001c40 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f594415cb50 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5940ff9700 1 -- 172.21.15.119:0/1022687222 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5934008300 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f594409c340 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59440cce20 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5940ff9700 1 -- 172.21.15.119:0/1022687222 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5934008460 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.204+0000 7f5940ff9700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5924066730 0x7f5924068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.208+0000 7f5940ff9700 1 -- 172.21.15.119:0/1022687222 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f5934084d90 con 0x7f59440cacb0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.208+0000 7f5943fff700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5924066730 0x7f5924068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.208+0000 7f5943fff700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5924066730 0x7f5924068bf0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f593c007600 tx=0x7f593c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:23.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.212+0000 7f5940ff9700 1 -- 172.21.15.119:0/1022687222 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5934052330 con 0x7f59440cacb0 2024-09-13T16:29:23.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.376+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f59440cdac0 con 0x7f59440cacb0 2024-09-13T16:29:23.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.376+0000 7f5940ff9700 1 -- 172.21.15.119:0/1022687222 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5934055950 con 0x7f59440cacb0 2024-09-13T16:29:23.381 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5924066730 msgr2=0x7f5924068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5924066730 0x7f5924068bf0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f593c007600 tx=0x7f593c00b040 comp rx=0 tx=0).stop 2024-09-13T16:29:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59440cacb0 msgr2=0x7f59440cb130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59440cacb0 0x7f59440cb130 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f5934011000 tx=0x7f5934004a70 comp rx=0 tx=0).stop 2024-09-13T16:29:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 shutdown_connections 2024-09-13T16:29:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5924066730 0x7f5924068bf0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59440cacb0 0x7f59440cb130 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f594415c590 0x7f5944072160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 --2- 172.21.15.119:0/1022687222 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5930004db0 0x7f5944152800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 >> 172.21.15.119:0/1022687222 conn(0x7f59440b0cd0 msgr2=0x7f5944158bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:23.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 shutdown_connections 2024-09-13T16:29:23.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.380+0000 7f5949202700 1 -- 172.21.15.119:0/1022687222 wait complete. 2024-09-13T16:29:23.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-09-13T16:29:23.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3220c700 1 Processor -- start 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3220c700 1 -- start start 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3220c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffb2c0b3800 0x7ffb2c1642b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3220c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0b2280 con 0x7ffb2c1594f0 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3220c700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb2c0cc890 con 0x7ffb2c06dc40 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3220c700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb2c070c10 con 0x7ffb2c0b3800 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3120a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ffb2c1594f0 0x7ffb2c06d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37814/0 (socket says 172.21.15.119:37814) 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3120a700 1 -- 172.21.15.119:0/4103592891 learned_addr learned my addr 172.21.15.119:0/4103592891 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb31a0b700 1 --2- 172.21.15.119:0/4103592891 >> v2:172.21.15.145:3300/0 conn(0x7ffb2c0b3800 0x7ffb2c1642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511193677 0 0) 0x7ffb2c0b2280 con 0x7ffb2c1594f0 2024-09-13T16:29:23.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb180035d0 con 0x7ffb2c1594f0 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221602274 0 0) 0x7ffb2c0cc890 con 0x7ffb2c06dc40 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb2c0b2280 con 0x7ffb2c06dc40 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2336262101 0 0) 0x7ffb180035d0 con 0x7ffb2c1594f0 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 >> v2:172.21.15.145:3300/0 conn(0x7ffb2c0b3800 msgr2=0x7ffb2c1642b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 --2- 172.21.15.119:0/4103592891 >> v2:172.21.15.145:3300/0 conn(0x7ffb2c0b3800 0x7ffb2c1642b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 >> v1:172.21.15.145:6789/0 conn(0x7ffb2c06dc40 legacy=0x7ffb2c0b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c0b1970 con 0x7ffb2c1594f0 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7ffb28001670 con 0x7ffb2c1594f0 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 >> v1:172.21.15.119:6789/0 conn(0x7ffb2c1594f0 legacy=0x7ffb2c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb18003bb0 0x7ffb18003fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb18004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb18005230 0x7ffb1800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c070c10 con 0x7ffb18004870 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1800c0d0 con 0x7ffb18005230 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb1800c290 con 0x7ffb18003bb0 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb30a09700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb18004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb3120a700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb18003bb0 0x7ffb18003fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb31a0b700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb18005230 0x7ffb1800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb30a09700 1 -- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb18003bb0 msgr2=0x7ffb18003fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb30a09700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb18003bb0 0x7ffb18003fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb30a09700 1 -- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb18005230 msgr2=0x7ffb1800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb30a09700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb18005230 0x7ffb1800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb31a0b700 1 -- 172.21.15.119:0/4103592891 reap_dead start 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb30a09700 1 -- 172.21.15.119:0/4103592891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb180035d0 con 0x7ffb18004870 2024-09-13T16:29:23.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.504+0000 7ffb30a09700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb18004cf0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c001960 tx=0x7ffb180058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daa7b923083bddf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c001450 con 0x7ffb18004870 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb1c006aa0 con 0x7ffb18004870 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb23fff700 1 -- 172.21.15.119:0/4103592891 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c006e50 con 0x7ffb18004870 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 -- 172.21.15.119:0/4103592891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 msgr2=0x7ffb18004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb18004cf0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c001960 tx=0x7ffb180058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 -- 172.21.15.119:0/4103592891 shutdown_connections 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 --2- 172.21.15.119:0/4103592891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb18004cf0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 -- 172.21.15.119:0/4103592891 >> 172.21.15.119:0/4103592891 conn(0x7ffb2c06a350 msgr2=0x7ffb2c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 -- 172.21.15.119:0/4103592891 shutdown_connections 2024-09-13T16:29:23.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 -- 172.21.15.119:0/4103592891 wait complete. 2024-09-13T16:29:23.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 Processor -- start 2024-09-13T16:29:23.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.508+0000 7ffb3220c700 1 -- start start 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb2c0acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb2c15c220 0x7ffb2c142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb2c15c680 0x7ffb2c1434d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c143a10 con 0x7ffb18004870 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c146aa0 con 0x7ffb2c15c680 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb2c146f30 con 0x7ffb2c15c220 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb2c0acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb2c0acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38874/0 (socket says 172.21.15.119:38874) 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 -- 172.21.15.119:0/1012791337 learned_addr learned my addr 172.21.15.119:0/1012791337 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb31a0b700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb2c15c680 0x7ffb2c1434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb30a09700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb2c15c220 0x7ffb2c142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 -- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb2c15c220 msgr2=0x7ffb2c142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb2c15c220 0x7ffb2c142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 -- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb2c15c680 msgr2=0x7ffb2c1434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb2c15c680 0x7ffb2c1434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 -- 172.21.15.119:0/1012791337 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb2c145eb0 con 0x7ffb18004870 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb30a09700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb2c15c220 0x7ffb2c142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3120a700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb2c0acb30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7ffb28007340 tx=0x7ffb28007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb21ffb700 1 -- 172.21.15.119:0/1012791337 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2800f6c0 con 0x7ffb18004870 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb21ffb700 1 -- 172.21.15.119:0/1012791337 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb2800f860 con 0x7ffb18004870 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb2c146170 con 0x7ffb18004870 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb21ffb700 1 -- 172.21.15.119:0/1012791337 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2800fc10 con 0x7ffb18004870 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb2c146730 con 0x7ffb18004870 2024-09-13T16:29:23.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.512+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb2c0b30f0 con 0x7ffb18004870 2024-09-13T16:29:23.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.516+0000 7ffb21ffb700 1 -- 172.21.15.119:0/1012791337 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ffb2800fd70 con 0x7ffb18004870 2024-09-13T16:29:23.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.516+0000 7ffb21ffb700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffb10066760 0x7ffb10068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:23.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.516+0000 7ffb21ffb700 1 -- 172.21.15.119:0/1012791337 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5263+0+0 (secure 0 0 0) 0x7ffb28083f30 con 0x7ffb18004870 2024-09-13T16:29:23.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.516+0000 7ffb30a09700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffb10066760 0x7ffb10068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:23.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.516+0000 7ffb30a09700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffb10066760 0x7ffb10068c20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c000c00 tx=0x7ffb1c005040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:23.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.524+0000 7ffb21ffb700 1 -- 172.21.15.119:0/1012791337 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ffb28051510 con 0x7ffb18004870 2024-09-13T16:29:23.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:23.748+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) v1 -- 0x7ffb2c150350 con 0x7ffb18004870 2024-09-13T16:29:24.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.752+0000 7ffb21ffb700 1 -- 172.21.15.119:0/1012791337 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v217) v1 ==== 169+0+0 (secure 0 0 0) 0x7ffb28054b30 con 0x7ffb18004870 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffb10066760 msgr2=0x7ffb10068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffb10066760 0x7ffb10068c20 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c000c00 tx=0x7ffb1c005040 comp rx=0 tx=0).stop 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 msgr2=0x7ffb2c0acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb2c0acb30 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7ffb28007340 tx=0x7ffb28007650 comp rx=0 tx=0).stop 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 shutdown_connections 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffb10066760 0x7ffb10068c20 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffb2c15c680 0x7ffb2c1434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:24.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffb2c15c220 0x7ffb2c142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:24.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 --2- 172.21.15.119:0/1012791337 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffb18004870 0x7ffb2c0acb30 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:24.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 >> 172.21.15.119:0/1012791337 conn(0x7ffb2c06a350 msgr2=0x7ffb2c0cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:24.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 shutdown_connections 2024-09-13T16:29:24.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.756+0000 7ffb3220c700 1 -- 172.21.15.119:0/1012791337 wait complete. 2024-09-13T16:29:24.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T16:29:24.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.900+0000 7f824692e700 1 Processor -- start 2024-09-13T16:29:24.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.900+0000 7f824692e700 1 -- start start 2024-09-13T16:29:24.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8240159530 0x7f82400cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:24.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82400ce700 con 0x7f824015d8d0 2024-09-13T16:29:24.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82400ce9e0 con 0x7f82400ce110 2024-09-13T16:29:24.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82400cecc0 con 0x7f8240159530 2024-09-13T16:29:24.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824612d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f824015d8d0 0x7f8240159910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37826/0 (socket says 172.21.15.119:37826) 2024-09-13T16:29:24.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824612d700 1 -- 172.21.15.119:0/2214862167 learned_addr learned my addr 172.21.15.119:0/2214862167 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:24.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824592c700 1 --2- 172.21.15.119:0/2214862167 >> v2:172.21.15.145:3300/0 conn(0x7f8240159530 0x7f82400cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824492a700 1 -- 172.21.15.119:0/2214862167 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438521350 0 0) 0x7f82400ce700 con 0x7f824015d8d0 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824492a700 1 -- 172.21.15.119:0/2214862167 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f822c0035d0 con 0x7f824015d8d0 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824492a700 1 -- 172.21.15.119:0/2214862167 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573956233 0 0) 0x7f82400ce9e0 con 0x7f82400ce110 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824492a700 1 -- 172.21.15.119:0/2214862167 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82400ce700 con 0x7f82400ce110 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824592c700 1 -- 172.21.15.119:0/2214862167 >> v1:172.21.15.145:6789/0 conn(0x7f82400ce110 legacy=0x7f824015d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824592c700 1 -- 172.21.15.119:0/2214862167 >> v1:172.21.15.119:6789/0 conn(0x7f824015d8d0 legacy=0x7f8240159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824592c700 1 -- 172.21.15.119:0/2214862167 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82400cefa0 con 0x7f8240159530 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824592c700 1 --2- 172.21.15.119:0/2214862167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8240159530 0x7f82400cdbd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f823c009880 tx=0x7f823c009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8d01167e512181 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824492a700 1 -- 172.21.15.119:0/2214862167 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c011830 con 0x7f8240159530 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824492a700 1 -- 172.21.15.119:0/2214862167 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c011990 con 0x7f8240159530 2024-09-13T16:29:24.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824492a700 1 -- 172.21.15.119:0/2214862167 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c011d00 con 0x7f8240159530 2024-09-13T16:29:24.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 -- 172.21.15.119:0/2214862167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8240159530 msgr2=0x7f82400cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:24.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 --2- 172.21.15.119:0/2214862167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8240159530 0x7f82400cdbd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f823c009880 tx=0x7f823c009b90 comp rx=0 tx=0).stop 2024-09-13T16:29:24.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 -- 172.21.15.119:0/2214862167 shutdown_connections 2024-09-13T16:29:24.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 --2- 172.21.15.119:0/2214862167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8240159530 0x7f82400cdbd0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:24.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.904+0000 7f824692e700 1 -- 172.21.15.119:0/2214862167 >> 172.21.15.119:0/2214862167 conn(0x7f824006b460 msgr2=0x7f824006b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:24.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.908+0000 7f824692e700 1 -- 172.21.15.119:0/2214862167 shutdown_connections 2024-09-13T16:29:24.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.908+0000 7f824692e700 1 -- 172.21.15.119:0/2214862167 wait complete. 2024-09-13T16:29:24.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.908+0000 7f824692e700 1 Processor -- start 2024-09-13T16:29:24.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.908+0000 7f824692e700 1 -- start start 2024-09-13T16:29:24.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82400ce110 0x7f8240140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:24.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8240159530 0x7f824014ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:24.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f824015d8d0 0x7f824014cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824014b450 con 0x7f82400ce110 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f824014b730 con 0x7f824015d8d0 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f824014ba10 con 0x7f8240159530 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82400ce110 0x7f8240140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82400ce110 0x7f8240140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38890/0 (socket says 172.21.15.119:38890) 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 -- 172.21.15.119:0/4221851 learned_addr learned my addr 172.21.15.119:0/4221851 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824612d700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f824015d8d0 0x7f824014cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824512b700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8240159530 0x7f824014ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 -- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8240159530 msgr2=0x7f824014ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8240159530 0x7f824014ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 -- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f824015d8d0 msgr2=0x7f824014cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:24.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f824015d8d0 0x7f824014cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:24.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 -- 172.21.15.119:0/4221851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f823c0094b0 con 0x7f82400ce110 2024-09-13T16:29:24.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824592c700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82400ce110 0x7f8240140300 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8240071d70 tx=0x7f823c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:24.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f82367fc700 1 -- 172.21.15.119:0/4221851 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c004450 con 0x7f82400ce110 2024-09-13T16:29:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f82367fc700 1 -- 172.21.15.119:0/4221851 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f823c01f070 con 0x7f82400ce110 2024-09-13T16:29:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f82367fc700 1 -- 172.21.15.119:0/4221851 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f823c003900 con 0x7f82400ce110 2024-09-13T16:29:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f824014bcf0 con 0x7f82400ce110 2024-09-13T16:29:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.912+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8240140840 con 0x7f82400ce110 2024-09-13T16:29:24.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.916+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f824014c210 con 0x7f82400ce110 2024-09-13T16:29:24.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.916+0000 7f82367fc700 1 -- 172.21.15.119:0/4221851 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f823c003a60 con 0x7f82400ce110 2024-09-13T16:29:24.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.916+0000 7f82367fc700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8224066730 0x7f8224068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:24.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.916+0000 7f82367fc700 1 -- 172.21.15.119:0/4221851 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f823c021070 con 0x7f82400ce110 2024-09-13T16:29:24.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.916+0000 7f824512b700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8224066730 0x7f8224068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:24.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.920+0000 7f824512b700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8224066730 0x7f8224068bf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f82401f2dc0 tx=0x7f8230009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:24.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:24.924+0000 7f82367fc700 1 -- 172.21.15.119:0/4221851 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f824014c210 con 0x7f82400ce110 2024-09-13T16:29:25.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.096+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f824006c100 con 0x7f82400ce110 2024-09-13T16:29:25.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.100+0000 7f82367fc700 1 -- 172.21.15.119:0/4221851 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f823c0150c0 con 0x7f82400ce110 2024-09-13T16:29:25.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8224066730 msgr2=0x7f8224068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8224066730 0x7f8224068bf0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f82401f2dc0 tx=0x7f8230009040 comp rx=0 tx=0).stop 2024-09-13T16:29:25.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82400ce110 msgr2=0x7f8240140300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82400ce110 0x7f8240140300 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f8240071d70 tx=0x7f823c004770 comp rx=0 tx=0).stop 2024-09-13T16:29:25.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 shutdown_connections 2024-09-13T16:29:25.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8224066730 0x7f8224068bf0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f824015d8d0 0x7f824014cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8240159530 0x7f824014ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 --2- 172.21.15.119:0/4221851 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82400ce110 0x7f8240140300 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 >> 172.21.15.119:0/4221851 conn(0x7f824006b460 msgr2=0x7f8240159910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:25.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 shutdown_connections 2024-09-13T16:29:25.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.104+0000 7f824692e700 1 -- 172.21.15.119:0/4221851 wait complete. 2024-09-13T16:29:25.122 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:29:25.122 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:29:25.143 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:29:25.162 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:29:25.162 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:25.162 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:25.169 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:29:25.169 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:29:25.169 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:25.217 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:29:25.218 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:25.218 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:25.225 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:29:25.225 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:29:25.225 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:25.232 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:29:25.232 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T16:29:25.232 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:25.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.380+0000 7f6346e15700 1 Processor -- start 2024-09-13T16:29:25.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.380+0000 7f6346e15700 1 -- start start 2024-09-13T16:29:25.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401440d0 0x7f634015b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f63401444b0 0x7f634006d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f63400cbb10 0x7f63400cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63400cc470 con 0x7f63401444b0 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634006bd60 con 0x7f63400cbb10 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634015bf10 con 0x7f63401440d0 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f63401444b0 0x7f634006d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f63401444b0 0x7f634006d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38894/0 (socket says 172.21.15.119:38894) 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 -- 172.21.15.119:0/1565585987 learned_addr learned my addr 172.21.15.119:0/1565585987 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f63453b2700 1 --2- 172.21.15.119:0/1565585987 >> v2:172.21.15.145:3300/0 conn(0x7f63400cbb10 0x7f63400cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6344bb1700 1 --2- 172.21.15.119:0/1565585987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401440d0 0x7f634015b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 -- 172.21.15.119:0/1565585987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401440d0 msgr2=0x7f634015b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 --2- 172.21.15.119:0/1565585987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401440d0 0x7f634015b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 -- 172.21.15.119:0/1565585987 >> v2:172.21.15.145:3300/0 conn(0x7f63400cbb10 msgr2=0x7f63400cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:25.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 --2- 172.21.15.119:0/1565585987 >> v2:172.21.15.145:3300/0 conn(0x7f63400cbb10 0x7f63400cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 -- 172.21.15.119:0/1565585987 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634006bfe0 con 0x7f63401444b0 2024-09-13T16:29:25.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6344bb1700 1 --2- 172.21.15.119:0/1565585987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401440d0 0x7f634015b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633ffff700 1 --2- 172.21.15.119:0/1565585987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401444b0 0x7f634006d080 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f63400b4000 tx=0x7f6328009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18f16684bd75b27 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633f7fe700 1 -- 172.21.15.119:0/1565585987 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328011830 con 0x7f63401444b0 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633f7fe700 1 -- 172.21.15.119:0/1565585987 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6328011990 con 0x7f63401444b0 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f633f7fe700 1 -- 172.21.15.119:0/1565585987 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328011d20 con 0x7f63401444b0 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- 172.21.15.119:0/1565585987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401444b0 msgr2=0x7f634006d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 --2- 172.21.15.119:0/1565585987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401444b0 0x7f634006d080 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f63400b4000 tx=0x7f6328009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- 172.21.15.119:0/1565585987 shutdown_connections 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 --2- 172.21.15.119:0/1565585987 >> v2:172.21.15.145:3300/0 conn(0x7f63400cbb10 0x7f63400cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 --2- 172.21.15.119:0/1565585987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401444b0 0x7f634006d080 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 --2- 172.21.15.119:0/1565585987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401440d0 0x7f634015b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- 172.21.15.119:0/1565585987 >> 172.21.15.119:0/1565585987 conn(0x7f634006a350 msgr2=0x7f634006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:25.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- 172.21.15.119:0/1565585987 shutdown_connections 2024-09-13T16:29:25.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.384+0000 7f6346e15700 1 -- 172.21.15.119:0/1565585987 wait complete. 2024-09-13T16:29:25.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 Processor -- start 2024-09-13T16:29:25.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 -- start start 2024-09-13T16:29:25.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63400cbb10 0x7f634013f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401440d0 0x7f63401384a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401444b0 0x7f634013c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634013e810 con 0x7f63401440d0 2024-09-13T16:29:25.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63401389e0 con 0x7f63400cbb10 2024-09-13T16:29:25.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634013ea40 con 0x7f63401444b0 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401440d0 0x7f63401384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401440d0 0x7f63401384a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38900/0 (socket says 172.21.15.119:38900) 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 -- 172.21.15.119:0/342866812 learned_addr learned my addr 172.21.15.119:0/342866812 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6344bb1700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63400cbb10 0x7f634013f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f63453b2700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401444b0 0x7f634013c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 -- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401444b0 msgr2=0x7f634013c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401444b0 0x7f634013c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 -- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63400cbb10 msgr2=0x7f634013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63400cbb10 0x7f634013f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 -- 172.21.15.119:0/342866812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6328009570 con 0x7f63401440d0 2024-09-13T16:29:25.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633ffff700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401440d0 0x7f63401384a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6328000c00 tx=0x7f6328004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:25.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633d7fa700 1 -- 172.21.15.119:0/342866812 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328004460 con 0x7f63401440d0 2024-09-13T16:29:25.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633d7fa700 1 -- 172.21.15.119:0/342866812 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f632801e070 con 0x7f63401440d0 2024-09-13T16:29:25.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634013ed20 con 0x7f63401440d0 2024-09-13T16:29:25.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f633d7fa700 1 -- 172.21.15.119:0/342866812 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6328003900 con 0x7f63401440d0 2024-09-13T16:29:25.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.388+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634013f2e0 con 0x7f63401440d0 2024-09-13T16:29:25.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.392+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6340067760 con 0x7f63401440d0 2024-09-13T16:29:25.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.392+0000 7f633d7fa700 1 -- 172.21.15.119:0/342866812 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6328003a60 con 0x7f63401440d0 2024-09-13T16:29:25.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.392+0000 7f633d7fa700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6320066760 0x7f6320068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.392+0000 7f633d7fa700 1 -- 172.21.15.119:0/342866812 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5263+0+0 (secure 0 0 0) 0x7f6328020070 con 0x7f63401440d0 2024-09-13T16:29:25.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.392+0000 7f6344bb1700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6320066760 0x7f6320068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.392+0000 7f6344bb1700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6320066760 0x7f6320068c20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f634015f230 tx=0x7f6330008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:25.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.400+0000 7f633d7fa700 1 -- 172.21.15.119:0/342866812 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6328052790 con 0x7f63401440d0 2024-09-13T16:29:25.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.604+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63400cdfe0 con 0x7f63401440d0 2024-09-13T16:29:25.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.604+0000 7f633d7fa700 1 -- 172.21.15.119:0/342866812 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f6328015de0 con 0x7f63401440d0 2024-09-13T16:29:25.610 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:25.610 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":36,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6313,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2050588099","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2050588099},{"type":"v1","addr":"172.21.15.145:6835","nonce":2050588099}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6525,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/826203391","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":826203391},{"type":"v1","addr":"172.21.15.119:6833","nonce":826203391}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6534,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/704571941","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":704571941},{"type":"v1","addr":"172.21.15.119:6835","nonce":704571941}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":27},{"gid":6742,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/4093724807","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":4093724807},{"type":"v1","addr":"172.21.15.145:6837","nonce":4093724807}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":36}],"filesystems":[]} 2024-09-13T16:29:25.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6320066760 msgr2=0x7f6320068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6320066760 0x7f6320068c20 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f634015f230 tx=0x7f6330008040 comp rx=0 tx=0).stop 2024-09-13T16:29:25.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401440d0 msgr2=0x7f63401384a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401440d0 0x7f63401384a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f6328000c00 tx=0x7f6328004770 comp rx=0 tx=0).stop 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 shutdown_connections 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6320066760 0x7f6320068c20 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f63401444b0 0x7f634013c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f63401440d0 0x7f63401384a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 --2- 172.21.15.119:0/342866812 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f63400cbb10 0x7f634013f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 >> 172.21.15.119:0/342866812 conn(0x7f634006a350 msgr2=0x7f634015c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 shutdown_connections 2024-09-13T16:29:25.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.608+0000 7f6346e15700 1 -- 172.21.15.119:0/342866812 wait complete. 2024-09-13T16:29:25.614 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 36 2024-09-13T16:29:25.625 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:29:25.625 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T16:29:25.625 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:29:25.625 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:29:25.625 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:29:25.625 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T16:29:25.626 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:29:25.626 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:29:25.626 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:29:25.626 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T16:29:25.626 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:29:25.626 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:29:25.626 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:29:25.627 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T16:29:25.627 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:29:25.627 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:29:25.641 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:29:25.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:29:25.642 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:29:25.642 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:29:25.643 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:29:25.644 DEBUG:teuthology.parallel:result is None 2024-09-13T16:29:25.644 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:29:25.645 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:29:25.645 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:29:25.646 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:29:25.646 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:29:25.646 DEBUG:teuthology.parallel:result is None 2024-09-13T16:29:25.688 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:29:25.688 DEBUG:teuthology.parallel:result is None 2024-09-13T16:29:25.689 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:29:25.689 DEBUG:teuthology.parallel:result is None 2024-09-13T16:29:25.689 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T16:29:25.725 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:29:25.733 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:29:25.770 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:29:25.773 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:29:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.788+0000 7fcdd3fff700 1 Processor -- start 2024-09-13T16:29:25.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.788+0000 7fcdd3fff700 1 -- start start 2024-09-13T16:29:25.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcdd40adb30 0x7fcdd40cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd3fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcdd406bb90 0x7fcdd40cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd3fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd406cf60 con 0x7fcdd40cde70 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd3fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd40b3390 con 0x7fcdd40adb30 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd3fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd406c350 con 0x7fcdd406bb90 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcdd40adb30 0x7fcdd40cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcdd40adb30 0x7fcdd40cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38948/0 (socket says 172.21.15.119:38948) 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 -- 172.21.15.119:0/2587168747 learned_addr learned my addr 172.21.15.119:0/2587168747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd27fc700 1 --2- 172.21.15.119:0/2587168747 >> v2:172.21.15.145:3300/0 conn(0x7fcdd406bb90 0x7fcdd40cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 -- 172.21.15.119:0/2587168747 >> v2:172.21.15.145:3300/0 conn(0x7fcdd406bb90 msgr2=0x7fcdd40cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 --2- 172.21.15.119:0/2587168747 >> v2:172.21.15.145:3300/0 conn(0x7fcdd406bb90 0x7fcdd40cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 -- 172.21.15.119:0/2587168747 >> v1:172.21.15.145:6789/0 conn(0x7fcdd40cde70 legacy=0x7fcdd415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 -- 172.21.15.119:0/2587168747 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd406c630 con 0x7fcdd40adb30 2024-09-13T16:29:25.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.792+0000 7fcdd2ffd700 1 --2- 172.21.15.119:0/2587168747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40adb30 0x7fcdd40cc060 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40cb210 tx=0x7fcdc4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a6aae3639fc9c01 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:25.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd1ffb700 1 -- 172.21.15.119:0/2587168747 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4011830 con 0x7fcdd40adb30 2024-09-13T16:29:25.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd1ffb700 1 -- 172.21.15.119:0/2587168747 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdc4011990 con 0x7fcdd40adb30 2024-09-13T16:29:25.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd1ffb700 1 -- 172.21.15.119:0/2587168747 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdc4011d20 con 0x7fcdd40adb30 2024-09-13T16:29:25.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2587168747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40adb30 msgr2=0x7fcdd40cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2587168747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40adb30 0x7fcdd40cc060 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7fcdd40cb210 tx=0x7fcdc4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2587168747 shutdown_connections 2024-09-13T16:29:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2587168747 >> v2:172.21.15.145:3300/0 conn(0x7fcdd406bb90 0x7fcdd40cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2587168747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40adb30 0x7fcdd40cc060 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2587168747 >> 172.21.15.119:0/2587168747 conn(0x7fcdd406a350 msgr2=0x7fcdd406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2587168747 shutdown_connections 2024-09-13T16:29:25.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2587168747 wait complete. 2024-09-13T16:29:25.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 Processor -- start 2024-09-13T16:29:25.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.796+0000 7fcdd3fff700 1 -- start start 2024-09-13T16:29:25.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdd406bb90 0x7fcdd41e4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdd40adb30 0x7fcdd413e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40cde70 0x7fcdd413e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd4139d70 con 0x7fcdd40cde70 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd41e50d0 con 0x7fcdd406bb90 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd41e4b10 con 0x7fcdd40adb30 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40cde70 0x7fcdd413e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40cde70 0x7fcdd413e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38952/0 (socket says 172.21.15.119:38952) 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 -- 172.21.15.119:0/2825606533 learned_addr learned my addr 172.21.15.119:0/2825606533 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd2ffd700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdd406bb90 0x7fcdd41e4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd27fc700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdd40adb30 0x7fcdd413e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 -- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdd40adb30 msgr2=0x7fcdd413e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdd40adb30 0x7fcdd413e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 -- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdd406bb90 msgr2=0x7fcdd41e4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdd406bb90 0x7fcdd41e4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:25.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 -- 172.21.15.119:0/2825606533 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4009570 con 0x7fcdd40cde70 2024-09-13T16:29:25.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd27fc700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdd40adb30 0x7fcdd413e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:25.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd37fe700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40cde70 0x7fcdd413e6e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00cba0 tx=0x7fcdcc00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:25.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdcbfff700 1 -- 172.21.15.119:0/2825606533 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc003c20 con 0x7fcdd40cde70 2024-09-13T16:29:25.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdcbfff700 1 -- 172.21.15.119:0/2825606533 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdcc003d80 con 0x7fcdd40cde70 2024-09-13T16:29:25.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd41e4df0 con 0x7fcdd40cde70 2024-09-13T16:29:25.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdcbfff700 1 -- 172.21.15.119:0/2825606533 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc00ab10 con 0x7fcdd40cde70 2024-09-13T16:29:25.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.800+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd415b070 con 0x7fcdd40cde70 2024-09-13T16:29:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.804+0000 7fcdcbfff700 1 -- 172.21.15.119:0/2825606533 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcdcc00ac70 con 0x7fcdd40cde70 2024-09-13T16:29:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.804+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd415b290 con 0x7fcdd40cde70 2024-09-13T16:29:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.804+0000 7fcdcbfff700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdb4066730 0x7fcdb4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.804+0000 7fcdcbfff700 1 -- 172.21.15.119:0/2825606533 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5263+0+0 (secure 0 0 0) 0x7fcdcc084070 con 0x7fcdd40cde70 2024-09-13T16:29:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.804+0000 7fcdd2ffd700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdb4066730 0x7fcdb4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:25.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.804+0000 7fcdd2ffd700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdb4066730 0x7fcdb4068bf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4000c00 tx=0x7fcdc4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:25.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.808+0000 7fcdcbfff700 1 -- 172.21.15.119:0/2825606533 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcdcc051510 con 0x7fcdd40cde70 2024-09-13T16:29:25.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:25.992+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fcdd40cafc0 con 0x7fcdd40cde70 2024-09-13T16:29:26.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.808+0000 7fcdcbfff700 1 -- 172.21.15.119:0/2825606533 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v78) v1 ==== 97+0+0 (secure 0 0 0) 0x7fcdcc054b30 con 0x7fcdd40cde70 2024-09-13T16:29:26.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdb4066730 msgr2=0x7fcdb4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdb4066730 0x7fcdb4068bf0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fcdc4000c00 tx=0x7fcdc4008030 comp rx=0 tx=0).stop 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40cde70 msgr2=0x7fcdd413e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40cde70 0x7fcdd413e6e0 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00cba0 tx=0x7fcdcc00ceb0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 shutdown_connections 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcdb4066730 0x7fcdb4068bf0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcdd40cde70 0x7fcdd413e6e0 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcdd40adb30 0x7fcdd413e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 --2- 172.21.15.119:0/2825606533 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcdd406bb90 0x7fcdd41e4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 >> 172.21.15.119:0/2825606533 conn(0x7fcdd406a350 msgr2=0x7fcdd4141ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 shutdown_connections 2024-09-13T16:29:26.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.812+0000 7fcdd3fff700 1 -- 172.21.15.119:0/2825606533 wait complete. 2024-09-13T16:29:26.820 INFO:teuthology.orchestra.run.smithi119.stderr: removed all blocklist entries 2024-09-13T16:29:26.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T16:29:26.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.940+0000 7fd0bb59e700 1 Processor -- start 2024-09-13T16:29:26.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.940+0000 7fd0bb59e700 1 -- start start 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0bc0cc060 0x7fd0bc0cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd0bc0cde70 0x7fd0bc0b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0bc06cf60 con 0x7fd0bc0adb30 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc0cc4d0 con 0x7fd0bc0cc060 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0bc06c350 con 0x7fd0bc0cde70 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0bc0cc060 0x7fd0bc0cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd0bc0cc060 0x7fd0bc0cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38968/0 (socket says 172.21.15.119:38968) 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/1628871960 learned_addr learned my addr 172.21.15.119:0/1628871960 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bad9d700 1 --2- 172.21.15.119:0/1628871960 >> v2:172.21.15.145:3300/0 conn(0x7fd0bc0cde70 0x7fd0bc0b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:26.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/1628871960 >> v2:172.21.15.145:3300/0 conn(0x7fd0bc0cde70 msgr2=0x7fd0bc0b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 --2- 172.21.15.119:0/1628871960 >> v2:172.21.15.145:3300/0 conn(0x7fd0bc0cde70 0x7fd0bc0b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/1628871960 >> v1:172.21.15.145:6789/0 conn(0x7fd0bc0adb30 legacy=0x7fd0bc06bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/1628871960 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0bc06c630 con 0x7fd0bc0cc060 2024-09-13T16:29:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b9d9b700 1 --2- 172.21.15.119:0/1628871960 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc0cd930 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0ccee0 tx=0x7fd0a4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51d728d8c24e015 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:26.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b959a700 1 -- 172.21.15.119:0/1628871960 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4004230 con 0x7fd0bc0cc060 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b959a700 1 -- 172.21.15.119:0/1628871960 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a4004390 con 0x7fd0bc0cc060 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0b959a700 1 -- 172.21.15.119:0/1628871960 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a40036c0 con 0x7fd0bc0cc060 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- 172.21.15.119:0/1628871960 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 msgr2=0x7fd0bc0cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/1628871960 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc0cd930 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc0ccee0 tx=0x7fd0a4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- 172.21.15.119:0/1628871960 shutdown_connections 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/1628871960 >> v2:172.21.15.145:3300/0 conn(0x7fd0bc0cde70 0x7fd0bc0b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/1628871960 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc0cd930 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- 172.21.15.119:0/1628871960 >> 172.21.15.119:0/1628871960 conn(0x7fd0bc06a350 msgr2=0x7fd0bc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- 172.21.15.119:0/1628871960 shutdown_connections 2024-09-13T16:29:26.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- 172.21.15.119:0/1628871960 wait complete. 2024-09-13T16:29:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 Processor -- start 2024-09-13T16:29:26.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.944+0000 7fd0bb59e700 1 -- start start 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0bc0adb30 0x7fd0bc1efcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc13c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0cde70 0x7fd0bc13e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc13ce10 con 0x7fd0bc0cc060 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc1f01f0 con 0x7fd0bc0cde70 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0bc15c280 con 0x7fd0bc0adb30 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc13c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc13c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38984/0 (socket says 172.21.15.119:38984) 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/3375373491 learned_addr learned my addr 172.21.15.119:0/3375373491 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0ba59c700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0bc0adb30 0x7fd0bc1efcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:26.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bad9d700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0cde70 0x7fd0bc13e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0bc0adb30 msgr2=0x7fd0bc1efcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0bc0adb30 0x7fd0bc1efcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0cde70 msgr2=0x7fd0bc13e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0cde70 0x7fd0bc13e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 -- 172.21.15.119:0/3375373491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4009570 con 0x7fd0bc0cc060 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0ba59c700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0bc0adb30 0x7fd0bc1efcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b9d9b700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc13c3a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4004000 tx=0x7fd0a4004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b2ffd700 1 -- 172.21.15.119:0/3375373491 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4003e60 con 0x7fd0bc0cc060 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b2ffd700 1 -- 172.21.15.119:0/3375373491 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a401f070 con 0x7fd0bc0cc060 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0bc15c560 con 0x7fd0bc0cc060 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b2ffd700 1 -- 172.21.15.119:0/3375373491 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a400b8a0 con 0x7fd0bc0cc060 2024-09-13T16:29:26.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0bc15cb20 con 0x7fd0bc0cc060 2024-09-13T16:29:26.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0bc06d3d0 con 0x7fd0bc0cc060 2024-09-13T16:29:26.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.948+0000 7fd0b2ffd700 1 -- 172.21.15.119:0/3375373491 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd0a4021030 con 0x7fd0bc0cc060 2024-09-13T16:29:26.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.952+0000 7fd0b2ffd700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd09c066730 0x7fd09c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:26.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.952+0000 7fd0b2ffd700 1 -- 172.21.15.119:0/3375373491 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fd0a40854b0 con 0x7fd0bc0cc060 2024-09-13T16:29:26.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.952+0000 7fd0ba59c700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd09c066730 0x7fd09c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:26.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.952+0000 7fd0ba59c700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd09c066730 0x7fd09c068bf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc1f31c0 tx=0x7fd0ac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:26.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:26.960+0000 7fd0b2ffd700 1 -- 172.21.15.119:0/3375373491 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd0bc06d3d0 con 0x7fd0bc0cc060 2024-09-13T16:29:27.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.188+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd0bc06d3d0 con 0x7fd0bc0cc060 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.188+0000 7fd0b2ffd700 1 -- 172.21.15.119:0/3375373491 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd0bc06d3d0 con 0x7fd0bc0cc060 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_dump": [ 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.a", 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA==", 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.193 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.b", 2024-09-13T16:29:27.194 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g==", 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.c", 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ==", 2024-09-13T16:29:27.195 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.d", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw==", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:29:27.196 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.197 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.0", 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA==", 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.198 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.1", 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw==", 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.199 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.200 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.2", 2024-09-13T16:29:27.200 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ==", 2024-09-13T16:29:27.200 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.200 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.200 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.200 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.3", 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww==", 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.202 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.4", 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA==", 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.203 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.5", 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q==", 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.204 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.6", 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg==", 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.7", 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ==", 2024-09-13T16:29:27.205 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.0", 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ==", 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.206 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow r", 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.1", 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ==", 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.207 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.2", 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw==", 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.208 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.3", 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow==", 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:29:27.209 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.admin", 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmpEcAJRAAYorOK9AU5Q/2uOTIUrZhJw==", 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.210 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.x", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ==", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.y", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw==", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:29:27.211 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.212 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:27.212 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:29:27.212 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.188+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd09c066730 msgr2=0x7fd09c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.188+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd09c066730 0x7fd09c068bf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fd0bc1f31c0 tx=0x7fd0ac009040 comp rx=0 tx=0).stop 2024-09-13T16:29:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.188+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 msgr2=0x7fd0bc13c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.188+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc13c3a0 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4004000 tx=0x7fd0a4004770 comp rx=0 tx=0).stop 2024-09-13T16:29:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 shutdown_connections 2024-09-13T16:29:27.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd09c066730 0x7fd09c068bf0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0bc0cde70 0x7fd0bc13e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0bc0cc060 0x7fd0bc13c3a0 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 --2- 172.21.15.119:0/3375373491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0bc0adb30 0x7fd0bc1efcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 >> 172.21.15.119:0/3375373491 conn(0x7fd0bc06a350 msgr2=0x7fd0bc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:27.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 shutdown_connections 2024-09-13T16:29:27.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.192+0000 7fd0bb59e700 1 -- 172.21.15.119:0/3375373491 wait complete. 2024-09-13T16:29:27.214 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae1ac63700 1 Processor -- start 2024-09-13T16:29:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae1ac63700 1 -- start start 2024-09-13T16:29:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae1ac63700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fae140adb30 0x7fae140cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae1ac63700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1406cf60 con 0x7fae1406bb90 2024-09-13T16:29:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae1ac63700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1406c350 con 0x7fae140ce040 2024-09-13T16:29:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae1ac63700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1406c630 con 0x7fae140adb30 2024-09-13T16:29:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae19460700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fae1406bb90 0x7fae140cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37838/0 (socket says 172.21.15.119:37838) 2024-09-13T16:29:27.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae19460700 1 -- 172.21.15.119:0/1803868045 learned_addr learned my addr 172.21.15.119:0/1803868045 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae19c61700 1 --2- 172.21.15.119:0/1803868045 >> v2:172.21.15.145:3300/0 conn(0x7fae140adb30 0x7fae140cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368211666 0 0) 0x7fae1406cf60 con 0x7fae1406bb90 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae000035d0 con 0x7fae1406bb90 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 63754345 0 0) 0x7fae000035d0 con 0x7fae1406bb90 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 >> v2:172.21.15.145:3300/0 conn(0x7fae140adb30 msgr2=0x7fae140cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 --2- 172.21.15.119:0/1803868045 >> v2:172.21.15.145:3300/0 conn(0x7fae140adb30 0x7fae140cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 >> v1:172.21.15.145:6789/0 conn(0x7fae140ce040 legacy=0x7fae140b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1406c910 con 0x7fae1406bb90 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fae08001670 con 0x7fae1406bb90 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 >> v1:172.21.15.119:6789/0 conn(0x7fae1406bb90 legacy=0x7fae140cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae00003a20 0x7fae00003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae00004870 0x7fae00004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00005230 0x7fae0000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1406c630 con 0x7fae00003a20 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000c0d0 con 0x7fae00004870 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae0000c290 con 0x7fae00005230 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.324+0000 7fae19c61700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae00003a20 0x7fae00003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19460700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae00004870 0x7fae00004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1a462700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00005230 0x7fae0000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19c61700 1 -- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00005230 msgr2=0x7fae0000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19c61700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00005230 0x7fae0000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19c61700 1 -- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae00004870 msgr2=0x7fae00004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19c61700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae00004870 0x7fae00004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19c61700 1 -- 172.21.15.119:0/1803868045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae000035d0 con 0x7fae00003a20 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1a462700 1 -- 172.21.15.119:0/1803868045 reap_dead start 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19460700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae00004870 0x7fae00004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae19c61700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae00003a20 0x7fae00003e40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fae10009700 tx=0x7fae000058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca939cfa180e7a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000e6c0 con 0x7fae00003a20 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1000e820 con 0x7fae00003a20 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae18c5f700 1 -- 172.21.15.119:0/1803868045 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1000ebb0 con 0x7fae00003a20 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 -- 172.21.15.119:0/1803868045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae00003a20 msgr2=0x7fae00003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae00003a20 0x7fae00003e40 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7fae10009700 tx=0x7fae000058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 -- 172.21.15.119:0/1803868045 shutdown_connections 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 --2- 172.21.15.119:0/1803868045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae00003a20 0x7fae00003e40 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 -- 172.21.15.119:0/1803868045 >> 172.21.15.119:0/1803868045 conn(0x7fae1406a350 msgr2=0x7fae1406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 -- 172.21.15.119:0/1803868045 shutdown_connections 2024-09-13T16:29:27.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 -- 172.21.15.119:0/1803868045 wait complete. 2024-09-13T16:29:27.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:29:27.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 Processor -- start 2024-09-13T16:29:27.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.328+0000 7fae1ac63700 1 -- start start 2024-09-13T16:29:27.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00003a20 0x7fae14142cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141388c0 0x7fae141ed4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae14070890 0x7fae14070d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae14071250 con 0x7fae141388c0 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1413a3b0 con 0x7fae14070890 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae141390a0 con 0x7fae00003a20 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141388c0 0x7fae141ed4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141388c0 0x7fae141ed4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38996/0 (socket says 172.21.15.119:38996) 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 -- 172.21.15.119:0/3353345301 learned_addr learned my addr 172.21.15.119:0/3353345301 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19c61700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00003a20 0x7fae14142cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1a462700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae14070890 0x7fae14070d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 -- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00003a20 msgr2=0x7fae14142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00003a20 0x7fae14142cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 -- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae14070890 msgr2=0x7fae14070d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae14070890 0x7fae14070d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 -- 172.21.15.119:0/3353345301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1413a600 con 0x7fae141388c0 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae19460700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141388c0 0x7fae141ed4f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fae08007290 tx=0x7fae08007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae067fc700 1 -- 172.21.15.119:0/3353345301 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae08007c30 con 0x7fae141388c0 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae067fc700 1 -- 172.21.15.119:0/3353345301 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0800f850 con 0x7fae141388c0 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1413a8e0 con 0x7fae141388c0 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae067fc700 1 -- 172.21.15.119:0/3353345301 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0800fbe0 con 0x7fae141388c0 2024-09-13T16:29:27.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae14138d40 con 0x7fae141388c0 2024-09-13T16:29:27.341 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:29:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.332+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae140ad730 con 0x7fae141388c0 2024-09-13T16:29:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.336+0000 7fae067fc700 1 -- 172.21.15.119:0/3353345301 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fae0800fd40 con 0x7fae141388c0 2024-09-13T16:29:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.336+0000 7fae067fc700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fadf8066730 0x7fadf8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.336+0000 7fae067fc700 1 -- 172.21.15.119:0/3353345301 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fae08083de0 con 0x7fae141388c0 2024-09-13T16:29:27.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.336+0000 7fae19c61700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fadf8066730 0x7fadf8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.336+0000 7fae19c61700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fadf8066730 0x7fadf8068bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fae1001e040 tx=0x7fae10017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.344+0000 7fae067fc700 1 -- 172.21.15.119:0/3353345301 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fae08051350 con 0x7fae141388c0 2024-09-13T16:29:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.520+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae1413ae00 con 0x7fae141388c0 2024-09-13T16:29:27.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.520+0000 7fae067fc700 1 -- 172.21.15.119:0/3353345301 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+13123 (secure 0 0 0) 0x7fae1413ae00 con 0x7fae141388c0 2024-09-13T16:29:27.526 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:27.527 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":78,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:26.803313+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fadf8066730 msgr2=0x7fadf8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fadf8066730 0x7fadf8068bf0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fae1001e040 tx=0x7fae10017000 comp rx=0 tx=0).stop 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141388c0 msgr2=0x7fae141ed4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141388c0 0x7fae141ed4f0 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7fae08007290 tx=0x7fae08007650 comp rx=0 tx=0).stop 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 shutdown_connections 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fadf8066730 0x7fadf8068bf0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae14070890 0x7fae14070d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae141388c0 0x7fae141ed4f0 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 --2- 172.21.15.119:0/3353345301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae00003a20 0x7fae14142cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 >> 172.21.15.119:0/3353345301 conn(0x7fae1406a350 msgr2=0x7fae140cd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:27.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 shutdown_connections 2024-09-13T16:29:27.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.524+0000 7fae1ac63700 1 -- 172.21.15.119:0/3353345301 wait complete. 2024-09-13T16:29:27.542 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:27.542 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a8e2a700 1 Processor -- start 2024-09-13T16:29:27.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a8e2a700 1 -- start start 2024-09-13T16:29:27.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a8e2a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a4159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a8e2a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff4a40b1860 0x7ff4a406c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a8e2a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4a40b1dd0 con 0x7ff4a406c890 2024-09-13T16:29:27.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a8e2a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a40b2fd0 con 0x7ff4a40b1860 2024-09-13T16:29:27.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a8e2a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a415a520 con 0x7ff4a40ad9a0 2024-09-13T16:29:27.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff4a406c890 0x7ff4a415c950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37842/0 (socket says 172.21.15.119:37842) 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a3fff700 1 -- 172.21.15.119:0/689516391 learned_addr learned my addr 172.21.15.119:0/689516391 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a2ffd700 1 --2- 172.21.15.119:0/689516391 >> v2:172.21.15.119:3300/0 conn(0x7ff4a40b1860 0x7ff4a406c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.656+0000 7ff4a37fe700 1 --2- 172.21.15.119:0/689516391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a4159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a27fc700 1 -- 172.21.15.119:0/689516391 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476488322 0 0) 0x7ff4a40b1dd0 con 0x7ff4a406c890 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a27fc700 1 -- 172.21.15.119:0/689516391 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff48c0046e0 con 0x7ff4a406c890 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a2ffd700 1 -- 172.21.15.119:0/689516391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 msgr2=0x7ff4a4159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a2ffd700 1 --2- 172.21.15.119:0/689516391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a4159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a27fc700 1 -- 172.21.15.119:0/689516391 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4282948745 0 0) 0x7ff48c0046e0 con 0x7ff4a406c890 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a2ffd700 1 -- 172.21.15.119:0/689516391 >> v1:172.21.15.119:6789/0 conn(0x7ff4a406c890 legacy=0x7ff4a415c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a2ffd700 1 -- 172.21.15.119:0/689516391 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a40b20b0 con 0x7ff4a40b1860 2024-09-13T16:29:27.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a2ffd700 1 --2- 172.21.15.119:0/689516391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a40b1860 0x7ff4a406c350 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff494009850 tx=0x7ff494009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79c8f0d187529639 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a27fc700 1 -- 172.21.15.119:0/689516391 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49c0019f0 con 0x7ff4a40b1860 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a27fc700 1 -- 172.21.15.119:0/689516391 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff494004030 con 0x7ff4a40b1860 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a27fc700 1 -- 172.21.15.119:0/689516391 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4940043e0 con 0x7ff4a40b1860 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/689516391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a40b1860 msgr2=0x7ff4a406c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/689516391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a40b1860 0x7ff4a406c350 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7ff494009850 tx=0x7ff494009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/689516391 shutdown_connections 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/689516391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a40b1860 0x7ff4a406c350 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/689516391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a4159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/689516391 >> 172.21.15.119:0/689516391 conn(0x7ff4a40b0980 msgr2=0x7ff4a40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/689516391 shutdown_connections 2024-09-13T16:29:27.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/689516391 wait complete. 2024-09-13T16:29:27.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 Processor -- start 2024-09-13T16:29:27.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.660+0000 7ff4a8e2a700 1 -- start start 2024-09-13T16:29:27.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a406c890 0x7ff4a414c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a415bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4a40b1860 0x7ff4a415cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a415b6b0 con 0x7ff4a406c890 2024-09-13T16:29:27.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a415aaa0 con 0x7ff4a40b1860 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a415ad80 con 0x7ff4a40ad9a0 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a406c890 0x7ff4a414c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a406c890 0x7ff4a414c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39026/0 (socket says 172.21.15.119:39026) 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 -- 172.21.15.119:0/2677224009 learned_addr learned my addr 172.21.15.119:0/2677224009 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a2ffd700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a415bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a3fff700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4a40b1860 0x7ff4a415cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 -- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 msgr2=0x7ff4a415bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a415bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 -- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4a40b1860 msgr2=0x7ff4a415cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4a40b1860 0x7ff4a415cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 -- 172.21.15.119:0/2677224009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4940094b0 con 0x7ff4a406c890 2024-09-13T16:29:27.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a37fe700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a406c890 0x7ff4a414c950 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff49800e9d0 tx=0x7ff49800ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff48bfff700 1 -- 172.21.15.119:0/2677224009 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff498009720 con 0x7ff4a406c890 2024-09-13T16:29:27.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff48bfff700 1 -- 172.21.15.119:0/2677224009 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49800ee70 con 0x7ff4a406c890 2024-09-13T16:29:27.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a415b060 con 0x7ff4a406c890 2024-09-13T16:29:27.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff48bfff700 1 -- 172.21.15.119:0/2677224009 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49800ca50 con 0x7ff4a406c890 2024-09-13T16:29:27.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a414ce90 con 0x7ff4a406c890 2024-09-13T16:29:27.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.664+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a4159de0 con 0x7ff4a406c890 2024-09-13T16:29:27.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.668+0000 7ff48bfff700 1 -- 172.21.15.119:0/2677224009 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff49800cbb0 con 0x7ff4a406c890 2024-09-13T16:29:27.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.668+0000 7ff48bfff700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff490066730 0x7ff490068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.668+0000 7ff48bfff700 1 -- 172.21.15.119:0/2677224009 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff498084790 con 0x7ff4a406c890 2024-09-13T16:29:27.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.668+0000 7ff4a2ffd700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff490066730 0x7ff490068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.668+0000 7ff4a2ffd700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff490066730 0x7ff490068bf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff494009850 tx=0x7ff494008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.672+0000 7ff48bfff700 1 -- 172.21.15.119:0/2677224009 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff498051cf0 con 0x7ff4a406c890 2024-09-13T16:29:27.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.840+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4a4070720 con 0x7ff4a406c890 2024-09-13T16:29:27.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.840+0000 7ff48bfff700 1 -- 172.21.15.119:0/2677224009 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff498017020 con 0x7ff4a406c890 2024-09-13T16:29:27.847 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:27.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff490066730 msgr2=0x7ff490068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff490066730 0x7ff490068bf0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7ff494009850 tx=0x7ff494008030 comp rx=0 tx=0).stop 2024-09-13T16:29:27.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a406c890 msgr2=0x7ff4a414c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a406c890 0x7ff4a414c950 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7ff49800e9d0 tx=0x7ff49800ece0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 shutdown_connections 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff490066730 0x7ff490068bf0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff4a40b1860 0x7ff4a415cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4a40ad9a0 0x7ff4a415bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 --2- 172.21.15.119:0/2677224009 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4a406c890 0x7ff4a414c950 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 >> 172.21.15.119:0/2677224009 conn(0x7ff4a40b0980 msgr2=0x7ff4a40b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 shutdown_connections 2024-09-13T16:29:27.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.844+0000 7ff4a8e2a700 1 -- 172.21.15.119:0/2677224009 wait complete. 2024-09-13T16:29:27.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:29:27.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafe0081700 1 Processor -- start 2024-09-13T16:29:27.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafe0081700 1 -- start start 2024-09-13T16:29:27.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafe0081700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 0x7fafd80ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafe0081700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80d1190 con 0x7fafd815ce70 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafe0081700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafd80ceaf0 con 0x7fafd81607c0 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafe0081700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80d3bc0 con 0x7fafd80d37a0 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fafd815ce70 0x7fafd80d1620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37856/0 (socket says 172.21.15.119:37856) 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 -- 172.21.15.119:0/1138551660 learned_addr learned my addr 172.21.15.119:0/1138551660 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdd61c700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 0x7fafd80ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649991961 0 0) 0x7fafd80d1190 con 0x7fafd815ce70 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafc40035d0 con 0x7fafd815ce70 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 282670410 0 0) 0x7fafc40035d0 con 0x7fafd815ce70 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 msgr2=0x7fafd80ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:27.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 0x7fafd80ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 >> v1:172.21.15.145:6789/0 conn(0x7fafd81607c0 legacy=0x7fafd80b2550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80d0860 con 0x7fafd815ce70 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fafd4001670 con 0x7fafd815ce70 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 >> v1:172.21.15.119:6789/0 conn(0x7fafd815ce70 legacy=0x7fafd80d1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafc4003a50 0x7fafc4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 0x7fafc4004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafc4004db0 0x7fafc4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80d3bc0 con 0x7fafc4003a50 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc400bae0 con 0x7fafc4004db0 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafc400bca0 con 0x7fafd80d37a0 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafc4003a50 0x7fafc4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdd61c700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 0x7fafc4004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafde61e700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafc4004db0 0x7fafc4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 -- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 msgr2=0x7fafc4004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafd80d37a0 0x7fafc4004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 -- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafc4004db0 msgr2=0x7fafc4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafc4004db0 0x7fafc4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 -- 172.21.15.119:0/1138551660 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc40035d0 con 0x7fafc4003a50 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafde61e700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafc4004db0 0x7fafc4005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdde1d700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafc4003a50 0x7fafc4003e70 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c270 tx=0x7fafc40058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c13c9fd7e16e34b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4007c30 con 0x7fafc4003a50 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafd400cdd0 con 0x7fafc4003a50 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.968+0000 7fafdce1b700 1 -- 172.21.15.119:0/1138551660 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafd4005bc0 con 0x7fafc4003a50 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 -- 172.21.15.119:0/1138551660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafc4003a50 msgr2=0x7fafc4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 --2- 172.21.15.119:0/1138551660 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafc4003a50 0x7fafc4003e70 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7fafd400c270 tx=0x7fafc40058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafde61e700 1 -- 172.21.15.119:0/1138551660 reap_dead start 2024-09-13T16:29:27.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 -- 172.21.15.119:0/1138551660 shutdown_connections 2024-09-13T16:29:27.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 -- 172.21.15.119:0/1138551660 >> 172.21.15.119:0/1138551660 conn(0x7fafd80d2bf0 msgr2=0x7fafd80d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:27.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 -- 172.21.15.119:0/1138551660 shutdown_connections 2024-09-13T16:29:27.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 -- 172.21.15.119:0/1138551660 wait complete. 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 Processor -- start 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.972+0000 7fafe0081700 1 -- start start 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafc4003a50 0x7fafd815d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafd8160250 0x7fafd8154110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafd80b0810 0x7fafd80c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd8154650 con 0x7fafd80b0810 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80b72f0 con 0x7fafd8160250 2024-09-13T16:29:27.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd80b1110 con 0x7fafc4003a50 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafd80b0810 0x7fafd80c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafd80b0810 0x7fafd80c8da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39052/0 (socket says 172.21.15.119:39052) 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 -- 172.21.15.119:0/765272968 learned_addr learned my addr 172.21.15.119:0/765272968 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafdde1d700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafc4003a50 0x7fafd815d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafdd61c700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafd8160250 0x7fafd8154110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 -- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafc4003a50 msgr2=0x7fafd815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafc4003a50 0x7fafd815d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 -- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafd8160250 msgr2=0x7fafd8154110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafd8160250 0x7fafd8154110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 -- 172.21.15.119:0/765272968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd80b6ca0 con 0x7fafd80b0810 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafde61e700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafd80b0810 0x7fafd80c8da0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fafc4003a20 tx=0x7fafcc00d790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafca7fc700 1 -- 172.21.15.119:0/765272968 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc00b070 con 0x7fafd80b0810 2024-09-13T16:29:27.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafca7fc700 1 -- 172.21.15.119:0/765272968 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafcc005310 con 0x7fafd80b0810 2024-09-13T16:29:27.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafd80b6f80 con 0x7fafd80b0810 2024-09-13T16:29:27.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafca7fc700 1 -- 172.21.15.119:0/765272968 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafcc006880 con 0x7fafd80b0810 2024-09-13T16:29:27.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafd80b0e70 con 0x7fafd80b0810 2024-09-13T16:29:27.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafca7fc700 1 -- 172.21.15.119:0/765272968 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fafcc0069e0 con 0x7fafd80b0810 2024-09-13T16:29:27.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafd80d1620 con 0x7fafd80b0810 2024-09-13T16:29:27.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.976+0000 7fafca7fc700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafbc066730 0x7fafbc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:27.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.980+0000 7fafca7fc700 1 -- 172.21.15.119:0/765272968 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fafcc0837d0 con 0x7fafd80b0810 2024-09-13T16:29:27.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.980+0000 7fafdde1d700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafbc066730 0x7fafbc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:27.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.980+0000 7fafdde1d700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafbc066730 0x7fafbc068bf0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fafd4000c00 tx=0x7fafd4007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:27.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:27.988+0000 7fafca7fc700 1 -- 172.21.15.119:0/765272968 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fafcc050de0 con 0x7fafd80b0810 2024-09-13T16:29:28.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fafd8072730 con 0x7fafd80b0810 2024-09-13T16:29:28.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafca7fc700 1 -- 172.21.15.119:0/765272968 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7fafcc054400 con 0x7fafd80b0810 2024-09-13T16:29:28.172 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:28.172 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:29:28.172 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:29:28.172 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:29:28.172 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:29:28.172 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:28.173 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:29:28.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafbc066730 msgr2=0x7fafbc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafbc066730 0x7fafbc068bf0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fafd4000c00 tx=0x7fafd4007000 comp rx=0 tx=0).stop 2024-09-13T16:29:28.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafd80b0810 msgr2=0x7fafd80c8da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafd80b0810 0x7fafd80c8da0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7fafc4003a20 tx=0x7fafcc00d790 comp rx=0 tx=0).stop 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 shutdown_connections 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafbc066730 0x7fafbc068bf0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafd80b0810 0x7fafd80c8da0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafd8160250 0x7fafd8154110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 --2- 172.21.15.119:0/765272968 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafc4003a50 0x7fafd815d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 >> 172.21.15.119:0/765272968 conn(0x7fafd80d2bf0 msgr2=0x7fafd815be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 shutdown_connections 2024-09-13T16:29:28.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.168+0000 7fafe0081700 1 -- 172.21.15.119:0/765272968 wait complete. 2024-09-13T16:29:28.187 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:29:28.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:29:28.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.308+0000 7f30be198700 1 Processor -- start 2024-09-13T16:29:28.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.308+0000 7f30be198700 1 -- start start 2024-09-13T16:29:28.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f30b0052560 0x7f30b0052fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b0054430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30b00520d0 con 0x7f30b00e1910 2024-09-13T16:29:28.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30b0053990 con 0x7f30b0052560 2024-09-13T16:29:28.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b0057680 con 0x7f30b00534e0 2024-09-13T16:29:28.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f30b0052560 0x7f30b0052fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bd997700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b0054430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f30b0052560 0x7f30b0052fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:34320/0 (socket says 172.21.15.119:34320) 2024-09-13T16:29:28.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 -- 172.21.15.119:0/243781110 learned_addr learned my addr 172.21.15.119:0/243781110 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 -- 172.21.15.119:0/243781110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 msgr2=0x7f30b0054430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 --2- 172.21.15.119:0/243781110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b0054430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 -- 172.21.15.119:0/243781110 >> v1:172.21.15.145:6789/0 conn(0x7f30b00e1910 legacy=0x7f30b0056f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 -- 172.21.15.119:0/243781110 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b00517a0 con 0x7f30b0052560 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30bc995700 1 --2- 172.21.15.119:0/243781110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 0x7f30b0052fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009910 tx=0x7f30b4009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86b0d7e0ed98dfbe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30affff700 1 -- 172.21.15.119:0/243781110 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4011830 con 0x7f30b0052560 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30affff700 1 -- 172.21.15.119:0/243781110 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b4011990 con 0x7f30b0052560 2024-09-13T16:29:28.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30affff700 1 -- 172.21.15.119:0/243781110 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b4011d40 con 0x7f30b0052560 2024-09-13T16:29:28.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- 172.21.15.119:0/243781110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 msgr2=0x7f30b0052fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 --2- 172.21.15.119:0/243781110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 0x7f30b0052fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f30b4009910 tx=0x7f30b4009c20 comp rx=0 tx=0).stop 2024-09-13T16:29:28.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- 172.21.15.119:0/243781110 shutdown_connections 2024-09-13T16:29:28.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 --2- 172.21.15.119:0/243781110 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b0054430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 --2- 172.21.15.119:0/243781110 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 0x7f30b0052fa0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- 172.21.15.119:0/243781110 >> 172.21.15.119:0/243781110 conn(0x7f30b001fbf0 msgr2=0x7f30b0056860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:28.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- 172.21.15.119:0/243781110 shutdown_connections 2024-09-13T16:29:28.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.312+0000 7f30be198700 1 -- 172.21.15.119:0/243781110 wait complete. 2024-09-13T16:29:28.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.316+0000 7f30be198700 1 Processor -- start 2024-09-13T16:29:28.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.316+0000 7f30be198700 1 -- start start 2024-09-13T16:29:28.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 0x7f30b017a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b004fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b00e1910 0x7f30b00e0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b004e980 con 0x7f30b00e1910 2024-09-13T16:29:28.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b004f7e0 con 0x7f30b0052560 2024-09-13T16:29:28.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b004ebd0 con 0x7f30b00534e0 2024-09-13T16:29:28.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b00e1910 0x7f30b00e0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b00e1910 0x7f30b00e0190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39062/0 (socket says 172.21.15.119:39062) 2024-09-13T16:29:28.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 -- 172.21.15.119:0/668659235 learned_addr learned my addr 172.21.15.119:0/668659235 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:28.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bc995700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b004fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd196700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 0x7f30b017a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 -- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 msgr2=0x7f30b004fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b004fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 -- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 msgr2=0x7f30b017a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 0x7f30b017a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 -- 172.21.15.119:0/668659235 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4009570 con 0x7f30b00e1910 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30bd997700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b00e1910 0x7f30b00e0190 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f30b80695d0 tx=0x7f30b8067770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30adffb700 1 -- 172.21.15.119:0/668659235 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b808f040 con 0x7f30b00e1910 2024-09-13T16:29:28.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30adffb700 1 -- 172.21.15.119:0/668659235 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b8073120 con 0x7f30b00e1910 2024-09-13T16:29:28.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b004eeb0 con 0x7f30b00e1910 2024-09-13T16:29:28.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30adffb700 1 -- 172.21.15.119:0/668659235 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8072350 con 0x7f30b00e1910 2024-09-13T16:29:28.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.320+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b004f470 con 0x7f30b00e1910 2024-09-13T16:29:28.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.324+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b004f970 con 0x7f30b00e1910 2024-09-13T16:29:28.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.324+0000 7f30adffb700 1 -- 172.21.15.119:0/668659235 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f30b80724b0 con 0x7f30b00e1910 2024-09-13T16:29:28.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.324+0000 7f30adffb700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f309c066730 0x7f309c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.324+0000 7f30adffb700 1 -- 172.21.15.119:0/668659235 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f30b80fd9e0 con 0x7f30b00e1910 2024-09-13T16:29:28.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.324+0000 7f30bd196700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f309c066730 0x7f309c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.324+0000 7f30bd196700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f309c066730 0x7f309c068bf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f30b00d6670 tx=0x7f30a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:28.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.332+0000 7f30adffb700 1 -- 172.21.15.119:0/668659235 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f30b80caf50 con 0x7f30b00e1910 2024-09-13T16:29:28.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.504+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f30b00e1cf0 con 0x7f30b00e1910 2024-09-13T16:29:28.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.832+0000 7f30adffb700 1 -- 172.21.15.119:0/668659235 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7f30b80ce570 con 0x7f30b00e1910 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f309c066730 msgr2=0x7f309c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f309c066730 0x7f309c068bf0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f30b00d6670 tx=0x7f30a8009040 comp rx=0 tx=0).stop 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b00e1910 msgr2=0x7f30b00e0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b00e1910 0x7f30b00e0190 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f30b80695d0 tx=0x7f30b8067770 comp rx=0 tx=0).stop 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 shutdown_connections 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f309c066730 0x7f309c068bf0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30b00e1910 0x7f30b00e0190 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30b00534e0 0x7f30b004fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 --2- 172.21.15.119:0/668659235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30b0052560 0x7f30b017a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 >> 172.21.15.119:0/668659235 conn(0x7f30b001fbf0 msgr2=0x7f30b00543b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:28.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 shutdown_connections 2024-09-13T16:29:28.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.840+0000 7f30be198700 1 -- 172.21.15.119:0/668659235 wait complete. 2024-09-13T16:29:28.848 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:29:28.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:29:28.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72e7b2700 1 Processor -- start 2024-09-13T16:29:28.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72e7b2700 1 -- start start 2024-09-13T16:29:28.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72e7b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 0x7ff72806dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72e7b2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7281403e0 con 0x7ff7280cf190 2024-09-13T16:29:28.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72e7b2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff72806cc00 con 0x7ff72813ff50 2024-09-13T16:29:28.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72e7b2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72806cee0 con 0x7ff72806aa60 2024-09-13T16:29:28.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff7280cf190 0x7ff72815d560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37872/0 (socket says 172.21.15.119:37872) 2024-09-13T16:29:28.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 -- 172.21.15.119:0/3219570634 learned_addr learned my addr 172.21.15.119:0/3219570634 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:28.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72dfb1700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 0x7ff72806dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606401659 0 0) 0x7ff7281403e0 con 0x7ff7280cf190 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7140035d0 con 0x7ff7280cf190 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236543282 0 0) 0x7ff72806cc00 con 0x7ff72813ff50 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7281403e0 con 0x7ff72813ff50 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2375694602 0 0) 0x7ff7140035d0 con 0x7ff7280cf190 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 msgr2=0x7ff72806dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 0x7ff72806dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 >> v1:172.21.15.145:6789/0 conn(0x7ff72813ff50 legacy=0x7ff72806a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72806d1c0 con 0x7ff7280cf190 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7ff724001670 con 0x7ff7280cf190 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 >> v1:172.21.15.119:6789/0 conn(0x7ff7280cf190 legacy=0x7ff72815d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff714003be0 0x7ff714003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 0x7ff714004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff714004db0 0x7ff714005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72806cee0 con 0x7ff714003be0 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff71400bbc0 con 0x7ff714004db0 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71400bd80 con 0x7ff72806aa60 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff714003be0 0x7ff714003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72cfaf700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 0x7ff714004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72dfb1700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff714004db0 0x7ff714005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 -- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 msgr2=0x7ff714004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff72806aa60 0x7ff714004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 -- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff714004db0 msgr2=0x7ff714005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff714004db0 0x7ff714005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 -- 172.21.15.119:0/3219570634 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7140035d0 con 0x7ff714003be0 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72dfb1700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff714004db0 0x7ff714005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff72d7b0700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff714003be0 0x7ff714003fe0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c160 tx=0x7ff7140058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82be6de43d49c18f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72400ccc0 con 0x7ff714003be0 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72400ce20 con 0x7ff714003be0 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.980+0000 7ff71ffff700 1 -- 172.21.15.119:0/3219570634 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff724005bc0 con 0x7ff714003be0 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- 172.21.15.119:0/3219570634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff714003be0 msgr2=0x7ff714003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 --2- 172.21.15.119:0/3219570634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff714003be0 0x7ff714003fe0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7ff72400c160 tx=0x7ff7140058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72dfb1700 1 -- 172.21.15.119:0/3219570634 reap_dead start 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- 172.21.15.119:0/3219570634 shutdown_connections 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- 172.21.15.119:0/3219570634 >> 172.21.15.119:0/3219570634 conn(0x7ff72806b460 msgr2=0x7ff72806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- 172.21.15.119:0/3219570634 shutdown_connections 2024-09-13T16:29:28.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- 172.21.15.119:0/3219570634 wait complete. 2024-09-13T16:29:28.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 Processor -- start 2024-09-13T16:29:28.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- start start 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff714003be0 0x7ff7280b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff728156a60 0x7ff7280b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7280b3000 0x7ff7280b3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72812faa0 con 0x7ff7280b3000 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff72812f1b0 con 0x7ff728156a60 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.984+0000 7ff72e7b2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff72812f450 con 0x7ff714003be0 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7280b3000 0x7ff7280b3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7280b3000 0x7ff7280b3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39084/0 (socket says 172.21.15.119:39084) 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 -- 172.21.15.119:0/4222857873 learned_addr learned my addr 172.21.15.119:0/4222857873 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72d7b0700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff714003be0 0x7ff7280b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72cfaf700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff728156a60 0x7ff7280b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 -- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff714003be0 msgr2=0x7ff7280b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff714003be0 0x7ff7280b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 -- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff728156a60 msgr2=0x7ff7280b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff728156a60 0x7ff7280b2ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 -- 172.21.15.119:0/4222857873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff72812f730 con 0x7ff7280b3000 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72cfaf700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff728156a60 0x7ff7280b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72dfb1700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7280b3000 0x7ff7280b3480 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff728071f90 tx=0x7ff7200104a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff71dffb700 1 -- 172.21.15.119:0/4222857873 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72001b070 con 0x7ff7280b3000 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff71dffb700 1 -- 172.21.15.119:0/4222857873 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff72000f040 con 0x7ff7280b3000 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff71dffb700 1 -- 172.21.15.119:0/4222857873 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72000e050 con 0x7ff7280b3000 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff728156fa0 con 0x7ff7280b3000 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7281540c0 con 0x7ff7280b3000 2024-09-13T16:29:28.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff72812fc10 con 0x7ff7280b3000 2024-09-13T16:29:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.988+0000 7ff71dffb700 1 -- 172.21.15.119:0/4222857873 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff720003a40 con 0x7ff7280b3000 2024-09-13T16:29:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.992+0000 7ff71dffb700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff70c066760 0x7ff70c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.992+0000 7ff71dffb700 1 -- 172.21.15.119:0/4222857873 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5547+0+0 (secure 0 0 0) 0x7ff7200842d0 con 0x7ff7280b3000 2024-09-13T16:29:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.992+0000 7ff72d7b0700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff70c066760 0x7ff70c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:28.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.992+0000 7ff72d7b0700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff70c066760 0x7ff70c068c20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff724008fd0 tx=0x7ff724007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:29.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:28.996+0000 7ff71dffb700 1 -- 172.21.15.119:0/4222857873 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff7200516e0 con 0x7ff7280b3000 2024-09-13T16:29:29.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.164+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff72812f8c0 con 0x7ff7280b3000 2024-09-13T16:29:29.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.852+0000 7ff71dffb700 1 -- 172.21.15.119:0/4222857873 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff720054d00 con 0x7ff7280b3000 2024-09-13T16:29:29.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff70c066760 msgr2=0x7ff70c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:29.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff70c066760 0x7ff70c068c20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff724008fd0 tx=0x7ff724007000 comp rx=0 tx=0).stop 2024-09-13T16:29:29.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7280b3000 msgr2=0x7ff7280b3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:29.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7280b3000 0x7ff7280b3480 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7ff728071f90 tx=0x7ff7200104a0 comp rx=0 tx=0).stop 2024-09-13T16:29:29.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 shutdown_connections 2024-09-13T16:29:29.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff70c066760 0x7ff70c068c20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:29.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff7280b3000 0x7ff7280b3480 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:29.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff728156a60 0x7ff7280b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:29.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 --2- 172.21.15.119:0/4222857873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff714003be0 0x7ff7280b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:29.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 >> 172.21.15.119:0/4222857873 conn(0x7ff72806b460 msgr2=0x7ff728160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:29.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 shutdown_connections 2024-09-13T16:29:29.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.860+0000 7ff72e7b2700 1 -- 172.21.15.119:0/4222857873 wait complete. 2024-09-13T16:29:29.869 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:29:29.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:29:29.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006ee70700 1 Processor -- start 2024-09-13T16:29:29.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006ee70700 1 -- start start 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006ee70700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006806b710 0x7f006806bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006ee70700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006806c070 con 0x7f006815cfb0 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006ee70700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00680ce880 con 0x7f00680d1640 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006ee70700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00680cee40 con 0x7f006806b710 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006cc0c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f006815cfb0 0x7f006806ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37886/0 (socket says 172.21.15.119:37886) 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f006cc0c700 1 -- 172.21.15.119:0/3194691506 learned_addr learned my addr 172.21.15.119:0/3194691506 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f0067fff700 1 --2- 172.21.15.119:0/3194691506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006806b710 0x7f006806bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810561965 0 0) 0x7f006806c070 con 0x7f006815cfb0 2024-09-13T16:29:29.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00540035d0 con 0x7f006815cfb0 2024-09-13T16:29:29.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092819836 0 0) 0x7f00680ce880 con 0x7f00680d1640 2024-09-13T16:29:29.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f006806c070 con 0x7f00680d1640 2024-09-13T16:29:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2471887314 0 0) 0x7f00540035d0 con 0x7f006815cfb0 2024-09-13T16:29:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f0067fff700 1 -- 172.21.15.119:0/3194691506 >> v1:172.21.15.145:6789/0 conn(0x7f00680d1640 legacy=0x7f00680d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f0067fff700 1 -- 172.21.15.119:0/3194691506 >> v1:172.21.15.119:6789/0 conn(0x7f006815cfb0 legacy=0x7f006806ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f0067fff700 1 -- 172.21.15.119:0/3194691506 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00680ceb60 con 0x7f006806b710 2024-09-13T16:29:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.992+0000 7f0067fff700 1 --2- 172.21.15.119:0/3194691506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006806b710 0x7f006806bb30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0050009880 tx=0x7f0050009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1892a574250ecc09 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0058001670 con 0x7f006806b710 2024-09-13T16:29:30.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050004030 con 0x7f006806b710 2024-09-13T16:29:30.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f00677fe700 1 -- 172.21.15.119:0/3194691506 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00500043a0 con 0x7f006806b710 2024-09-13T16:29:30.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f006ee70700 1 -- 172.21.15.119:0/3194691506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006806b710 msgr2=0x7f006806bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f006ee70700 1 --2- 172.21.15.119:0/3194691506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006806b710 0x7f006806bb30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f0050009880 tx=0x7f0050009b90 comp rx=0 tx=0).stop 2024-09-13T16:29:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f006ee70700 1 -- 172.21.15.119:0/3194691506 shutdown_connections 2024-09-13T16:29:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f006ee70700 1 --2- 172.21.15.119:0/3194691506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006806b710 0x7f006806bb30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f006ee70700 1 -- 172.21.15.119:0/3194691506 >> 172.21.15.119:0/3194691506 conn(0x7f00680d2da0 msgr2=0x7f00680d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f006ee70700 1 -- 172.21.15.119:0/3194691506 shutdown_connections 2024-09-13T16:29:30.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:29.996+0000 7f006ee70700 1 -- 172.21.15.119:0/3194691506 wait complete. 2024-09-13T16:29:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 Processor -- start 2024-09-13T16:29:30.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 -- start start 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f006806b710 0x7f0068154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00680d1640 0x7f00680c70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006815cfb0 0x7f00680cf6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00680d1220 con 0x7f00680d1640 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0068154ab0 con 0x7f006806b710 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00680b7300 con 0x7f006815cfb0 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00680d1640 0x7f00680c70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00680d1640 0x7f00680c70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39090/0 (socket says 172.21.15.119:39090) 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 -- 172.21.15.119:0/94991523 learned_addr learned my addr 172.21.15.119:0/94991523 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006cc0c700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f006806b710 0x7f0068154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006d40d700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006815cfb0 0x7f00680cf6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 -- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006815cfb0 msgr2=0x7f00680cf6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006815cfb0 0x7f00680cf6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 -- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f006806b710 msgr2=0x7f0068154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f006806b710 0x7f0068154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 -- 172.21.15.119:0/94991523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00500094b0 con 0x7f00680d1640 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f0067fff700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00680d1640 0x7f00680c70b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f0050000c00 tx=0x7f0050004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f00657fa700 1 -- 172.21.15.119:0/94991523 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050004570 con 0x7f00680d1640 2024-09-13T16:29:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f00657fa700 1 -- 172.21.15.119:0/94991523 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050003db0 con 0x7f00680d1640 2024-09-13T16:29:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00680c7b00 con 0x7f00680d1640 2024-09-13T16:29:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f00657fa700 1 -- 172.21.15.119:0/94991523 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050011870 con 0x7f00680d1640 2024-09-13T16:29:30.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.000+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00680b75a0 con 0x7f00680d1640 2024-09-13T16:29:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.004+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f006806bb30 con 0x7f00680d1640 2024-09-13T16:29:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.004+0000 7f00657fa700 1 -- 172.21.15.119:0/94991523 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f00500119d0 con 0x7f00680d1640 2024-09-13T16:29:30.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.004+0000 7f00657fa700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.004+0000 7f00657fa700 1 -- 172.21.15.119:0/94991523 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f0050084f20 con 0x7f00680d1640 2024-09-13T16:29:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.004+0000 7f006cc0c700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.008+0000 7f006cc0c700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0058001eb0 tx=0x7f0058001d80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.016+0000 7f00657fa700 1 -- 172.21.15.119:0/94991523 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0050052360 con 0x7f00680d1640 2024-09-13T16:29:30.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.220+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f00680b53e0 con 0x7f00680d1640 2024-09-13T16:29:30.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.232+0000 7f00657fa700 1 -- 172.21.15.119:0/94991523 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 11 and data pool 12 v38) v1 ==== 174+0+0 (secure 0 0 0) 0x7f00500200c0 con 0x7f00680d1640 2024-09-13T16:29:30.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 msgr2=0x7f0048068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f0058001eb0 tx=0x7f0058001d80 comp rx=0 tx=0).stop 2024-09-13T16:29:30.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00680d1640 msgr2=0x7f00680c70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00680d1640 0x7f00680c70b0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f0050000c00 tx=0x7f0050004770 comp rx=0 tx=0).stop 2024-09-13T16:29:30.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 shutdown_connections 2024-09-13T16:29:30.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f006815cfb0 0x7f00680cf6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00680d1640 0x7f00680c70b0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 --2- 172.21.15.119:0/94991523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f006806b710 0x7f0068154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 >> 172.21.15.119:0/94991523 conn(0x7f00680d2da0 msgr2=0x7f00680b28e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:30.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 shutdown_connections 2024-09-13T16:29:30.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.248+0000 7f006ee70700 1 -- 172.21.15.119:0/94991523 wait complete. 2024-09-13T16:29:30.260 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 11 and data pool 12 2024-09-13T16:29:30.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:30.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49b094d700 1 Processor -- start 2024-09-13T16:29:30.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49b094d700 1 -- start start 2024-09-13T16:29:30.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49b094d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f49ac1656a0 0x7f49ac167a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49b094d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0cdd50 con 0x7f49ac0d2a30 2024-09-13T16:29:30.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49b094d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0cdf90 con 0x7f49ac0b3a20 2024-09-13T16:29:30.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49b094d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac0ce270 con 0x7f49ac1656a0 2024-09-13T16:29:30.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49ab7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f49ac0d2a30 0x7f49ac0d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37896/0 (socket says 172.21.15.119:37896) 2024-09-13T16:29:30.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49ab7fe700 1 -- 172.21.15.119:0/2337973276 learned_addr learned my addr 172.21.15.119:0/2337973276 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:30.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49abfff700 1 --2- 172.21.15.119:0/2337973276 >> v2:172.21.15.145:3300/0 conn(0x7f49ac1656a0 0x7f49ac167a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49aa7fc700 1 -- 172.21.15.119:0/2337973276 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220280949 0 0) 0x7f499c001110 con 0x7f49ac0d2a30 2024-09-13T16:29:30.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49aa7fc700 1 -- 172.21.15.119:0/2337973276 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49980035d0 con 0x7f49ac0d2a30 2024-09-13T16:29:30.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49aa7fc700 1 -- 172.21.15.119:0/2337973276 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 598344253 0 0) 0x7f49980035d0 con 0x7f49ac0d2a30 2024-09-13T16:29:30.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49abfff700 1 -- 172.21.15.119:0/2337973276 >> v1:172.21.15.145:6789/0 conn(0x7f49ac0b3a20 legacy=0x7f49ac0cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49abfff700 1 -- 172.21.15.119:0/2337973276 >> v1:172.21.15.119:6789/0 conn(0x7f49ac0d2a30 legacy=0x7f49ac0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49abfff700 1 -- 172.21.15.119:0/2337973276 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0ce550 con 0x7f49ac1656a0 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49abfff700 1 --2- 172.21.15.119:0/2337973276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac1656a0 0x7f49ac167a90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009dc0 tx=0x7f49a400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa18873bb8b01 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49aa7fc700 1 -- 172.21.15.119:0/2337973276 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a400e070 con 0x7f49ac1656a0 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49aa7fc700 1 -- 172.21.15.119:0/2337973276 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a4003a40 con 0x7f49ac1656a0 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.372+0000 7f49aa7fc700 1 -- 172.21.15.119:0/2337973276 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c001800 con 0x7f49ac1656a0 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- 172.21.15.119:0/2337973276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac1656a0 msgr2=0x7f49ac167a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 --2- 172.21.15.119:0/2337973276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac1656a0 0x7f49ac167a90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f49a4009dc0 tx=0x7f49a400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- 172.21.15.119:0/2337973276 shutdown_connections 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 --2- 172.21.15.119:0/2337973276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac1656a0 0x7f49ac167a90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- 172.21.15.119:0/2337973276 >> 172.21.15.119:0/2337973276 conn(0x7f49ac08c0f0 msgr2=0x7f49ac08c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- 172.21.15.119:0/2337973276 shutdown_connections 2024-09-13T16:29:30.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- 172.21.15.119:0/2337973276 wait complete. 2024-09-13T16:29:30.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 Processor -- start 2024-09-13T16:29:30.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- start start 2024-09-13T16:29:30.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49ac0b3a20 0x7f49ac156f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac0d2a30 0x7f49ac15eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ac1656a0 0x7f49ac0cf1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac1453a0 con 0x7f49ac0b3a20 2024-09-13T16:29:30.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac145680 con 0x7f49ac0d2a30 2024-09-13T16:29:30.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49b094d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac145960 con 0x7f49ac1656a0 2024-09-13T16:29:30.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49ac0b3a20 0x7f49ac156f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49ac0b3a20 0x7f49ac156f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39092/0 (socket says 172.21.15.119:39092) 2024-09-13T16:29:30.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 -- 172.21.15.119:0/346340829 learned_addr learned my addr 172.21.15.119:0/346340829 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:30.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49aaffd700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac0d2a30 0x7f49ac15eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49abfff700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ac1656a0 0x7f49ac0cf1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 -- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ac1656a0 msgr2=0x7f49ac0cf1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ac1656a0 0x7f49ac0cf1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 -- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac0d2a30 msgr2=0x7f49ac15eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac0d2a30 0x7f49ac15eb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.376+0000 7f49ab7fe700 1 -- 172.21.15.119:0/346340829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a4009a70 con 0x7f49ac0b3a20 2024-09-13T16:29:30.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49ab7fe700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49ac0b3a20 0x7f49ac156f90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c780 tx=0x7f499c00ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49a3fff700 1 -- 172.21.15.119:0/346340829 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c004ca0 con 0x7f49ac0b3a20 2024-09-13T16:29:30.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49a3fff700 1 -- 172.21.15.119:0/346340829 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f499c004e20 con 0x7f49ac0b3a20 2024-09-13T16:29:30.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac145c40 con 0x7f49ac0b3a20 2024-09-13T16:29:30.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49a3fff700 1 -- 172.21.15.119:0/346340829 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f499c007310 con 0x7f49ac0b3a20 2024-09-13T16:29:30.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac0cf8c0 con 0x7f49ac0b3a20 2024-09-13T16:29:30.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac145e80 con 0x7f49ac0b3a20 2024-09-13T16:29:30.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.380+0000 7f49a3fff700 1 -- 172.21.15.119:0/346340829 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f499c007470 con 0x7f49ac0b3a20 2024-09-13T16:29:30.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.384+0000 7f49a3fff700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f498c066730 0x7f498c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.384+0000 7f49a3fff700 1 -- 172.21.15.119:0/346340829 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f499c083be0 con 0x7f49ac0b3a20 2024-09-13T16:29:30.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.384+0000 7f49aaffd700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f498c066730 0x7f498c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.384+0000 7f49aaffd700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f498c066730 0x7f498c068bf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f49ac157f10 tx=0x7f4994009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.388+0000 7f49a3fff700 1 -- 172.21.15.119:0/346340829 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f499c050ea0 con 0x7f49ac0b3a20 2024-09-13T16:29:30.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.568+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49ac0cdd50 con 0x7f49ac0b3a20 2024-09-13T16:29:30.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.568+0000 7f49a3fff700 1 -- 172.21.15.119:0/346340829 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16850 (secure 0 0 0) 0x7f49ac0cdd50 con 0x7f49ac0b3a20 2024-09-13T16:29:30.573 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:30.574 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":81,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:30.226615+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:30.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f498c066730 msgr2=0x7f498c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f498c066730 0x7f498c068bf0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f49ac157f10 tx=0x7f4994009040 comp rx=0 tx=0).stop 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49ac0b3a20 msgr2=0x7f49ac156f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49ac0b3a20 0x7f49ac156f90 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f499c00c780 tx=0x7f499c00ca90 comp rx=0 tx=0).stop 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 shutdown_connections 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f498c066730 0x7f498c068bf0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49ac1656a0 0x7f49ac0cf1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49ac0d2a30 0x7f49ac15eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 --2- 172.21.15.119:0/346340829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49ac0b3a20 0x7f49ac156f90 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 >> 172.21.15.119:0/346340829 conn(0x7f49ac08c0f0 msgr2=0x7f49ac08c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 shutdown_connections 2024-09-13T16:29:30.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.572+0000 7f49b094d700 1 -- 172.21.15.119:0/346340829 wait complete. 2024-09-13T16:29:30.589 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:30.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.696+0000 7f1cb6784700 1 Processor -- start 2024-09-13T16:29:30.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.696+0000 7f1cb6784700 1 -- start start 2024-09-13T16:29:30.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1cb015e840 0x7f1cb0070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1cb015ec20 0x7f1cb0161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cb0071210 con 0x7f1cb0164a70 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb00714f0 con 0x7f1cb015ec20 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cb00717d0 con 0x7f1cb015e840 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1cb015ec20 0x7f1cb0161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1cb015ec20 0x7f1cb0161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39096/0 (socket says 172.21.15.119:39096) 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 -- 172.21.15.119:0/1459945179 learned_addr learned my addr 172.21.15.119:0/1459945179 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 -- 172.21.15.119:0/1459945179 >> v2:172.21.15.145:3300/0 conn(0x7f1cb015e840 msgr2=0x7f1cb0070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caffff700 1 --2- 172.21.15.119:0/1459945179 >> v2:172.21.15.145:3300/0 conn(0x7f1cb015e840 0x7f1cb0070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 --2- 172.21.15.119:0/1459945179 >> v2:172.21.15.145:3300/0 conn(0x7f1cb015e840 0x7f1cb0070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 -- 172.21.15.119:0/1459945179 >> v1:172.21.15.145:6789/0 conn(0x7f1cb0164a70 legacy=0x7f1cb00ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 -- 172.21.15.119:0/1459945179 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb0071ab0 con 0x7f1cb015ec20 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caffff700 1 --2- 172.21.15.119:0/1459945179 >> v2:172.21.15.145:3300/0 conn(0x7f1cb015e840 0x7f1cb0070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:30.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caf7fe700 1 --2- 172.21.15.119:0/1459945179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015ec20 0x7f1cb0161f00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1cb006dec0 tx=0x7f1ca000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28f11d18b7969fe0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caeffd700 1 -- 172.21.15.119:0/1459945179 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca000e070 con 0x7f1cb015ec20 2024-09-13T16:29:30.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caeffd700 1 -- 172.21.15.119:0/1459945179 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca0003a40 con 0x7f1cb015ec20 2024-09-13T16:29:30.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1caeffd700 1 -- 172.21.15.119:0/1459945179 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca0003df0 con 0x7f1cb015ec20 2024-09-13T16:29:30.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- 172.21.15.119:0/1459945179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015ec20 msgr2=0x7f1cb0161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 --2- 172.21.15.119:0/1459945179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015ec20 0x7f1cb0161f00 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f1cb006dec0 tx=0x7f1ca000b330 comp rx=0 tx=0).stop 2024-09-13T16:29:30.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- 172.21.15.119:0/1459945179 shutdown_connections 2024-09-13T16:29:30.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 --2- 172.21.15.119:0/1459945179 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015ec20 0x7f1cb0161f00 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 --2- 172.21.15.119:0/1459945179 >> v2:172.21.15.145:3300/0 conn(0x7f1cb015e840 0x7f1cb0070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- 172.21.15.119:0/1459945179 >> 172.21.15.119:0/1459945179 conn(0x7f1cb006a350 msgr2=0x7f1cb006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:30.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- 172.21.15.119:0/1459945179 shutdown_connections 2024-09-13T16:29:30.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.700+0000 7f1cb6784700 1 -- 172.21.15.119:0/1459945179 wait complete. 2024-09-13T16:29:30.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 Processor -- start 2024-09-13T16:29:30.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 -- start start 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cb015e840 0x7f1cb0156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cb0164a70 0x7f1cb00cc500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015a1b0 0x7f1cb0157b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb015f2d0 con 0x7f1cb015a1b0 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb015a640 con 0x7f1cb0164a70 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb6784700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cb00ccc00 con 0x7f1cb015e840 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb4d21700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015a1b0 0x7f1cb0157b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb4d21700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015a1b0 0x7f1cb0157b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39112/0 (socket says 172.21.15.119:39112) 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb4d21700 1 -- 172.21.15.119:0/163516287 learned_addr learned my addr 172.21.15.119:0/163516287 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1caffff700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cb015e840 0x7f1cb0156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1caf7fe700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cb0164a70 0x7f1cb00cc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb4d21700 1 -- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cb015e840 msgr2=0x7f1cb0156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb4d21700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cb015e840 0x7f1cb0156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb4d21700 1 -- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cb0164a70 msgr2=0x7f1cb00cc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.704+0000 7f1cb4d21700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cb0164a70 0x7f1cb00cc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cb4d21700 1 -- 172.21.15.119:0/163516287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca0009b30 con 0x7f1cb015a1b0 2024-09-13T16:29:30.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cb4d21700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015a1b0 0x7f1cb0157b90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800c470 tx=0x7f1ca800c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cacff9700 1 -- 172.21.15.119:0/163516287 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8011790 con 0x7f1cb015a1b0 2024-09-13T16:29:30.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb00ccee0 con 0x7f1cb015a1b0 2024-09-13T16:29:30.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cb00cd4a0 con 0x7f1cb015a1b0 2024-09-13T16:29:30.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cacff9700 1 -- 172.21.15.119:0/163516287 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca80118f0 con 0x7f1cb015a1b0 2024-09-13T16:29:30.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cacff9700 1 -- 172.21.15.119:0/163516287 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ca8011ca0 con 0x7f1cb015a1b0 2024-09-13T16:29:30.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c90005310 con 0x7f1cb015a1b0 2024-09-13T16:29:30.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cacff9700 1 -- 172.21.15.119:0/163516287 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1ca8026430 con 0x7f1cb015a1b0 2024-09-13T16:29:30.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cacff9700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c9c066760 0x7f1c9c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:30.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.708+0000 7f1cacff9700 1 -- 172.21.15.119:0/163516287 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1ca8084aa0 con 0x7f1cb015a1b0 2024-09-13T16:29:30.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.712+0000 7f1caffff700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c9c066760 0x7f1c9c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:30.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.712+0000 7f1caffff700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c9c066760 0x7f1c9c068c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1cb01598c0 tx=0x7f1ca4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:30.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.716+0000 7f1cacff9700 1 -- 172.21.15.119:0/163516287 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1ca8051c30 con 0x7f1cb015a1b0 2024-09-13T16:29:30.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.888+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c90005890 con 0x7f1cb015a1b0 2024-09-13T16:29:30.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.888+0000 7f1cacff9700 1 -- 172.21.15.119:0/163516287 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16850 (secure 0 0 0) 0x7f1ca8055250 con 0x7f1cb015a1b0 2024-09-13T16:29:30.895 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:30.895 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":81,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:30.226615+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:30.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c9c066760 msgr2=0x7f1c9c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c9c066760 0x7f1c9c068c20 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f1cb01598c0 tx=0x7f1ca4008040 comp rx=0 tx=0).stop 2024-09-13T16:29:30.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015a1b0 msgr2=0x7f1cb0157b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:30.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015a1b0 0x7f1cb0157b90 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7f1ca800c470 tx=0x7f1ca800c780 comp rx=0 tx=0).stop 2024-09-13T16:29:30.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 shutdown_connections 2024-09-13T16:29:30.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c9c066760 0x7f1c9c068c20 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1cb015a1b0 0x7f1cb0157b90 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1cb0164a70 0x7f1cb00cc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 --2- 172.21.15.119:0/163516287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1cb015e840 0x7f1cb0156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:30.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 >> 172.21.15.119:0/163516287 conn(0x7f1cb006a350 msgr2=0x7f1cb0162790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:30.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 shutdown_connections 2024-09-13T16:29:30.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:30.892+0000 7f1cb6784700 1 -- 172.21.15.119:0/163516287 wait complete. 2024-09-13T16:29:30.910 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:29:31.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.016+0000 7f9aa647a700 1 Processor -- start 2024-09-13T16:29:31.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.016+0000 7f9aa647a700 1 -- start start 2024-09-13T16:29:31.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa647a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 0x7f9aa00b29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa647a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa015ab40 con 0x7f9aa006ba70 2024-09-13T16:29:31.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa647a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aa00ad760 con 0x7f9aa00acf50 2024-09-13T16:29:31.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa647a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00ada40 con 0x7f9aa015a760 2024-09-13T16:29:31.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa5c79700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9aa006ba70 0x7f9aa006d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37900/0 (socket says 172.21.15.119:37900) 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa5c79700 1 -- 172.21.15.119:0/2978281656 learned_addr learned my addr 172.21.15.119:0/2978281656 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa5478700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 0x7f9aa00b29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123510505 0 0) 0x7f9aa015ab40 con 0x7f9aa006ba70 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a8c0035d0 con 0x7f9aa006ba70 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 55005873 0 0) 0x7f9aa00ad760 con 0x7f9aa00acf50 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa015ab40 con 0x7f9aa00acf50 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4058486377 0 0) 0x7f9a8c0035d0 con 0x7f9aa006ba70 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 msgr2=0x7f9aa00b29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 0x7f9aa00b29e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 >> v1:172.21.15.145:6789/0 conn(0x7f9aa00acf50 legacy=0x7f9aa006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa00add20 con 0x7f9aa006ba70 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f9a980019f0 con 0x7f9aa006ba70 2024-09-13T16:29:31.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 >> v1:172.21.15.119:6789/0 conn(0x7f9aa006ba70 legacy=0x7f9aa006d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 0x7f9a8c003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a8c004140 0x7f9a8c004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004db0 0x7f9a8c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00ada40 con 0x7f9a8c004140 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c00bb10 con 0x7f9a8c004db0 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a8c00bcd0 con 0x7f9aa015a760 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa4c77700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a8c004140 0x7f9a8c004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa5478700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 0x7f9a8c003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa5c79700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004db0 0x7f9a8c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa4c77700 1 -- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 msgr2=0x7f9a8c003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa4c77700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa015a760 0x7f9a8c003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa4c77700 1 -- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004db0 msgr2=0x7f9a8c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:31.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa4c77700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004db0 0x7f9a8c005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa4c77700 1 -- 172.21.15.119:0/2978281656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a8c0035d0 con 0x7f9a8c004140 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.020+0000 7f9aa4c77700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a8c004140 0x7f9a8c004870 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00aea30 tx=0x7f9a8c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b3d9732a42d8dcf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90001450 con 0x7f9a8c004140 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a900052f0 con 0x7f9a8c004140 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9a97fff700 1 -- 172.21.15.119:0/2978281656 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90001d60 con 0x7f9a8c004140 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 -- 172.21.15.119:0/2978281656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a8c004140 msgr2=0x7f9a8c004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 --2- 172.21.15.119:0/2978281656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9a8c004140 0x7f9a8c004870 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00aea30 tx=0x7f9a8c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa5c79700 1 -- 172.21.15.119:0/2978281656 reap_dead start 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 -- 172.21.15.119:0/2978281656 shutdown_connections 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 -- 172.21.15.119:0/2978281656 >> 172.21.15.119:0/2978281656 conn(0x7f9aa00cee60 msgr2=0x7f9aa00cf270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 -- 172.21.15.119:0/2978281656 shutdown_connections 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 -- 172.21.15.119:0/2978281656 wait complete. 2024-09-13T16:29:31.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 Processor -- start 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.024+0000 7f9aa647a700 1 -- start start 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004140 0x7f9aa009b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9aa0151420 0x7f9aa00cd6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa00cdbe0 0x7f9aa0141610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00cec90 con 0x7f9aa0151420 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00ce110 con 0x7f9a8c004140 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa00ce3f0 con 0x7f9aa00cdbe0 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9aa0151420 0x7f9aa00cd6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9aa0151420 0x7f9aa00cd6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39124/0 (socket says 172.21.15.119:39124) 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 -- 172.21.15.119:0/3775762609 learned_addr learned my addr 172.21.15.119:0/3775762609 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa5478700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004140 0x7f9aa009b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa5c79700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa00cdbe0 0x7f9aa0141610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 -- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa00cdbe0 msgr2=0x7f9aa0141610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:31.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa00cdbe0 0x7f9aa0141610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 -- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004140 msgr2=0x7f9aa009b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004140 0x7f9aa009b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 -- 172.21.15.119:0/3775762609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa00ce6d0 con 0x7f9aa0151420 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa5c79700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa00cdbe0 0x7f9aa0141610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa4c77700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9aa0151420 0x7f9aa00cd6a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00bae0 tx=0x7f9a900012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9a95ffb700 1 -- 172.21.15.119:0/3775762609 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90006c90 con 0x7f9aa0151420 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9a95ffb700 1 -- 172.21.15.119:0/3775762609 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a90006df0 con 0x7f9aa0151420 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa00ce9b0 con 0x7f9aa0151420 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9a95ffb700 1 -- 172.21.15.119:0/3775762609 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a90007280 con 0x7f9aa0151420 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aa009b900 con 0x7f9aa0151420 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aa00cebd0 con 0x7f9aa0151420 2024-09-13T16:29:31.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.028+0000 7f9a95ffb700 1 -- 172.21.15.119:0/3775762609 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9a900073e0 con 0x7f9aa0151420 2024-09-13T16:29:31.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.032+0000 7f9a95ffb700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a84066760 0x7f9a84068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:31.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.032+0000 7f9a95ffb700 1 -- 172.21.15.119:0/3775762609 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9a90084fd0 con 0x7f9aa0151420 2024-09-13T16:29:31.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.032+0000 7f9aa5478700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a84066760 0x7f9a84068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:31.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.032+0000 7f9aa5478700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a84066760 0x7f9a84068c20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0152cb0 tx=0x7f9a9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:31.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.036+0000 7f9a95ffb700 1 -- 172.21.15.119:0/3775762609 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9a90055ca0 con 0x7f9aa0151420 2024-09-13T16:29:31.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:31.248+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f9aa009b900 con 0x7f9aa0151420 2024-09-13T16:29:32.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.240+0000 7f9a95ffb700 1 -- 172.21.15.119:0/3775762609 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v41) v1 ==== 118+0+0 (secure 0 0 0) 0x7f9aa009b900 con 0x7f9aa0151420 2024-09-13T16:29:32.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a84066760 msgr2=0x7f9a84068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:32.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a84066760 0x7f9a84068c20 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0152cb0 tx=0x7f9a9c008040 comp rx=0 tx=0).stop 2024-09-13T16:29:32.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9aa0151420 msgr2=0x7f9aa00cd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:32.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9aa0151420 0x7f9aa00cd6a0 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c00bae0 tx=0x7f9a900012d0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 shutdown_connections 2024-09-13T16:29:32.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a84066760 0x7f9a84068c20 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9aa00cdbe0 0x7f9aa0141610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9aa0151420 0x7f9aa00cd6a0 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 --2- 172.21.15.119:0/3775762609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9a8c004140 0x7f9aa009b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.244+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 >> 172.21.15.119:0/3775762609 conn(0x7f9aa00cee60 msgr2=0x7f9aa006a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:32.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.248+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 shutdown_connections 2024-09-13T16:29:32.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.248+0000 7f9aa647a700 1 -- 172.21.15.119:0/3775762609 wait complete. 2024-09-13T16:29:32.268 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T16:29:32.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:29:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7209a98700 1 Processor -- start 2024-09-13T16:29:32.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7209a98700 1 -- start start 2024-09-13T16:29:32.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7209a98700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f72040ce8c0 0x7f72040d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7209a98700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f72040b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7209a98700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72040d1a60 con 0x7f720415cc00 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7209a98700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72040d0df0 con 0x7f72040ce8c0 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7209a98700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72040cdb00 con 0x7f72040d1610 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.396+0000 7f7203fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f72040ce8c0 0x7f72040d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7203fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f72040ce8c0 0x7f72040d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:39138/0 (socket says 172.21.15.119:39138) 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7203fff700 1 -- 172.21.15.119:0/836638684 learned_addr learned my addr 172.21.15.119:0/836638684 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209297700 1 --2- 172.21.15.119:0/836638684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f72040b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7203fff700 1 -- 172.21.15.119:0/836638684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 msgr2=0x7f72040b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7203fff700 1 --2- 172.21.15.119:0/836638684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f72040b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7203fff700 1 -- 172.21.15.119:0/836638684 >> v1:172.21.15.145:6789/0 conn(0x7f720415cc00 legacy=0x7f72040d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:32.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7203fff700 1 -- 172.21.15.119:0/836638684 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72040cdde0 con 0x7f72040ce8c0 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7203fff700 1 --2- 172.21.15.119:0/836638684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72040ce8c0 0x7f72040d10d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009910 tx=0x7f71f4009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1658cd59e6c0c297 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f72037fe700 1 -- 172.21.15.119:0/836638684 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4011830 con 0x7f72040ce8c0 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f72037fe700 1 -- 172.21.15.119:0/836638684 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f4011990 con 0x7f72040ce8c0 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f72037fe700 1 -- 172.21.15.119:0/836638684 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4011d40 con 0x7f72040ce8c0 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 -- 172.21.15.119:0/836638684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72040ce8c0 msgr2=0x7f72040d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 --2- 172.21.15.119:0/836638684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72040ce8c0 0x7f72040d10d0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f71f4009910 tx=0x7f71f4009c20 comp rx=0 tx=0).stop 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 -- 172.21.15.119:0/836638684 shutdown_connections 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 --2- 172.21.15.119:0/836638684 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f72040b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 --2- 172.21.15.119:0/836638684 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f72040ce8c0 0x7f72040d10d0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 -- 172.21.15.119:0/836638684 >> 172.21.15.119:0/836638684 conn(0x7f720408c080 msgr2=0x7f72040d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 -- 172.21.15.119:0/836638684 shutdown_connections 2024-09-13T16:29:32.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.400+0000 7f7209a98700 1 -- 172.21.15.119:0/836638684 wait complete. 2024-09-13T16:29:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 Processor -- start 2024-09-13T16:29:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 -- start start 2024-09-13T16:29:32.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72040ce8c0 0x7f720415bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:32.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f7204148a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f720415cc00 0x7f72040c62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72041e6680 con 0x7f720415cc00 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72041487d0 con 0x7f72040ce8c0 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209a98700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72041e6890 con 0x7f72040d1610 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209297700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f720415cc00 0x7f72040c62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7203fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f7204148a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209297700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f720415cc00 0x7f72040c62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45946/0 (socket says 172.21.15.119:45946) 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7208a96700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72040ce8c0 0x7f720415bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:32.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.404+0000 7f7209297700 1 -- 172.21.15.119:0/1875693709 learned_addr learned my addr 172.21.15.119:0/1875693709 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7203fff700 1 -- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72040ce8c0 msgr2=0x7f720415bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7203fff700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72040ce8c0 0x7f720415bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7203fff700 1 -- 172.21.15.119:0/1875693709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f720415cc00 msgr2=0x7f72040c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7203fff700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f720415cc00 0x7f72040c62e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7203fff700 1 -- 172.21.15.119:0/1875693709 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71f4009570 con 0x7f72040d1610 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7209297700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f720415cc00 0x7f72040c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7203fff700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f7204148a70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71f4016000 tx=0x7f71f4004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f72017fa700 1 -- 172.21.15.119:0/1875693709 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4004510 con 0x7f72040d1610 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f72017fa700 1 -- 172.21.15.119:0/1875693709 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71f401f070 con 0x7f72040d1610 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72041e6b70 con 0x7f72040d1610 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f72017fa700 1 -- 172.21.15.119:0/1875693709 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71f4003900 con 0x7f72040d1610 2024-09-13T16:29:32.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72041e7130 con 0x7f72040d1610 2024-09-13T16:29:32.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.408+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72040d3670 con 0x7f72040d1610 2024-09-13T16:29:32.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.412+0000 7f72017fa700 1 -- 172.21.15.119:0/1875693709 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f71f4003a60 con 0x7f72040d1610 2024-09-13T16:29:32.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.412+0000 7f72017fa700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71f0066760 0x7f71f0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:32.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.412+0000 7f72017fa700 1 -- 172.21.15.119:0/1875693709 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f71f4021070 con 0x7f72040d1610 2024-09-13T16:29:32.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.412+0000 7f7208a96700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71f0066760 0x7f71f0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:32.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.412+0000 7f7208a96700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71f0066760 0x7f71f0068c20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f71f8000f60 tx=0x7f71f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:32.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.424+0000 7f72017fa700 1 -- 172.21.15.119:0/1875693709 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f71f4052900 con 0x7f72040d1610 2024-09-13T16:29:32.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:32.632+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f720408c560 con 0x7f72040d1610 2024-09-13T16:29:33.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.252+0000 7f72017fa700 1 -- 172.21.15.119:0/1875693709 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v42) v1 ==== 122+0+0 (secure 0 0 0) 0x7f71f4006da0 con 0x7f72040d1610 2024-09-13T16:29:33.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71f0066760 msgr2=0x7f71f0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71f0066760 0x7f71f0068c20 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f71f8000f60 tx=0x7f71f8009040 comp rx=0 tx=0).stop 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 msgr2=0x7f7204148a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f7204148a70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f71f4016000 tx=0x7f71f4004770 comp rx=0 tx=0).stop 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 shutdown_connections 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f71f0066760 0x7f71f0068c20 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f720415cc00 0x7f72040c62e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f72040d1610 0x7f7204148a70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 --2- 172.21.15.119:0/1875693709 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f72040ce8c0 0x7f720415bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 >> 172.21.15.119:0/1875693709 conn(0x7f720408c080 msgr2=0x7f72040b59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 shutdown_connections 2024-09-13T16:29:33.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.260+0000 7f7209a98700 1 -- 172.21.15.119:0/1875693709 wait complete. 2024-09-13T16:29:33.281 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:29:33.281 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:33.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.392+0000 7fec7d263700 1 Processor -- start 2024-09-13T16:29:33.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.392+0000 7fec7d263700 1 -- start start 2024-09-13T16:29:33.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec78071890 0x7fec780ce000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec780cc460 con 0x7fec78159320 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec7806cf10 con 0x7fec78071890 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec780cc700 con 0x7fec7806c5f0 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec78071890 0x7fec780ce000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec76ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fec78071890 0x7fec780ce000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:60840/0 (socket says 172.21.15.119:60840) 2024-09-13T16:29:33.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 -- 172.21.15.119:0/1730058983 learned_addr learned my addr 172.21.15.119:0/1730058983 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:33.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 -- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 msgr2=0x7fec7806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:33.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 --2- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 -- 172.21.15.119:0/1730058983 >> v1:172.21.15.145:6789/0 conn(0x7fec78159320 legacy=0x7fec7806fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 -- 172.21.15.119:0/1730058983 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec780cc9e0 con 0x7fec78071890 2024-09-13T16:29:33.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec76ffd700 1 --2- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:33.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec767fc700 1 --2- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 0x7fec780ce000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fec68009e50 tx=0x7fec6800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82bb344995e10b63 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:33.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec75ffb700 1 -- 172.21.15.119:0/1730058983 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6800e070 con 0x7fec78071890 2024-09-13T16:29:33.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec75ffb700 1 -- 172.21.15.119:0/1730058983 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec6800b910 con 0x7fec78071890 2024-09-13T16:29:33.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec75ffb700 1 -- 172.21.15.119:0/1730058983 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec6800bcc0 con 0x7fec78071890 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 msgr2=0x7fec780ce000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 --2- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 0x7fec780ce000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fec68009e50 tx=0x7fec6800b330 comp rx=0 tx=0).stop 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- 172.21.15.119:0/1730058983 shutdown_connections 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 --2- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 0x7fec780ce000 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 --2- 172.21.15.119:0/1730058983 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- 172.21.15.119:0/1730058983 >> 172.21.15.119:0/1730058983 conn(0x7fec7806a350 msgr2=0x7fec7806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- 172.21.15.119:0/1730058983 shutdown_connections 2024-09-13T16:29:33.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.396+0000 7fec7d263700 1 -- 172.21.15.119:0/1730058983 wait complete. 2024-09-13T16:29:33.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 Processor -- start 2024-09-13T16:29:33.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 -- start start 2024-09-13T16:29:33.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7814cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 0x7fec7814e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec78159320 0x7fec780ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec78143110 con 0x7fec78159320 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec7814f220 con 0x7fec78071890 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec7d263700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec7814f500 con 0x7fec7806c5f0 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec777fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec78159320 0x7fec780ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec777fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec78159320 0x7fec780ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45948/0 (socket says 172.21.15.119:45948) 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec777fe700 1 -- 172.21.15.119:0/3768653136 learned_addr learned my addr 172.21.15.119:0/3768653136 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec76ffd700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7814cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.400+0000 7fec767fc700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 0x7fec7814e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec777fe700 1 -- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 msgr2=0x7fec7814cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec777fe700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7814cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec777fe700 1 -- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 msgr2=0x7fec7814e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec777fe700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 0x7fec7814e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec777fe700 1 -- 172.21.15.119:0/3768653136 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec6c0094b0 con 0x7fec78159320 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec777fe700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec78159320 0x7fec780ac8c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fec7000c3c0 tx=0x7fec7000c6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec5f7fe700 1 -- 172.21.15.119:0/3768653136 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec700052d0 con 0x7fec78159320 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec5f7fe700 1 -- 172.21.15.119:0/3768653136 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec70005450 con 0x7fec78159320 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec68009b30 con 0x7fec78159320 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec5f7fe700 1 -- 172.21.15.119:0/3768653136 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec700038e0 con 0x7fec78159320 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec7814fac0 con 0x7fec78159320 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec78143250 con 0x7fec78159320 2024-09-13T16:29:33.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.404+0000 7fec5f7fe700 1 -- 172.21.15.119:0/3768653136 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fec70003a40 con 0x7fec78159320 2024-09-13T16:29:33.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.408+0000 7fec5f7fe700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec64066730 0x7fec64068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.408+0000 7fec5f7fe700 1 -- 172.21.15.119:0/3768653136 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fec70083400 con 0x7fec78159320 2024-09-13T16:29:33.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.408+0000 7fec76ffd700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec64066730 0x7fec64068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.408+0000 7fec76ffd700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec64066730 0x7fec64068bf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fec780cae50 tx=0x7fec6c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:33.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.412+0000 7fec5f7fe700 1 -- 172.21.15.119:0/3768653136 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fec78143250 con 0x7fec78159320 2024-09-13T16:29:33.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.620+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec78143250 con 0x7fec78159320 2024-09-13T16:29:33.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.620+0000 7fec5f7fe700 1 -- 172.21.15.119:0/3768653136 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fec78143250 con 0x7fec78159320 2024-09-13T16:29:33.626 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:33.626 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:29:33.249934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7101":{"gid":7101,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/491709455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":491709455},{"type":"v1","addr":"172.21.15.119:6835","nonce":491709455}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:29:33.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec64066730 msgr2=0x7fec64068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec64066730 0x7fec64068bf0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fec780cae50 tx=0x7fec6c009240 comp rx=0 tx=0).stop 2024-09-13T16:29:33.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec78159320 msgr2=0x7fec780ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec78159320 0x7fec780ac8c0 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7fec7000c3c0 tx=0x7fec7000c6d0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 shutdown_connections 2024-09-13T16:29:33.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fec64066730 0x7fec64068bf0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fec78159320 0x7fec780ac8c0 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fec78071890 0x7fec7814e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 --2- 172.21.15.119:0/3768653136 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fec7806c5f0 0x7fec7814cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 >> 172.21.15.119:0/3768653136 conn(0x7fec7806a350 msgr2=0x7fec78152a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 shutdown_connections 2024-09-13T16:29:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.624+0000 7fec7d263700 1 -- 172.21.15.119:0/3768653136 wait complete. 2024-09-13T16:29:33.629 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 42 2024-09-13T16:29:33.641 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:33.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.748+0000 7f0733fff700 1 Processor -- start 2024-09-13T16:29:33.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.748+0000 7f0733fff700 1 -- start start 2024-09-13T16:29:33.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07340b3800 0x7f07341642b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07340b2280 con 0x7f07341594f0 2024-09-13T16:29:33.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07340cc890 con 0x7f073406dc40 2024-09-13T16:29:33.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0734070c10 con 0x7f07340b3800 2024-09-13T16:29:33.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0732ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f07341594f0 0x7f073406d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50618/0 (socket says 172.21.15.119:50618) 2024-09-13T16:29:33.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0732ffd700 1 -- 172.21.15.119:0/827613867 learned_addr learned my addr 172.21.15.119:0/827613867 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:33.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07337fe700 1 --2- 172.21.15.119:0/827613867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07340b3800 0x7f07341642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959018581 0 0) 0x7f07340b2280 con 0x7f07341594f0 2024-09-13T16:29:33.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07200035d0 con 0x7f07341594f0 2024-09-13T16:29:33.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3086746399 0 0) 0x7f07340cc890 con 0x7f073406dc40 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07340b2280 con 0x7f073406dc40 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07337fe700 1 -- 172.21.15.119:0/827613867 >> v1:172.21.15.145:6789/0 conn(0x7f073406dc40 legacy=0x7f07340b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07337fe700 1 -- 172.21.15.119:0/827613867 >> v1:172.21.15.119:6789/0 conn(0x7f07341594f0 legacy=0x7f073406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 602655247 0 0) 0x7f07200035d0 con 0x7f07341594f0 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07337fe700 1 -- 172.21.15.119:0/827613867 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07340b1970 con 0x7f07340b3800 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07337fe700 1 --2- 172.21.15.119:0/827613867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07340b3800 0x7f07341642b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f072c009dc0 tx=0x7f072c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bd06649ce96e494 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f072c00e070 con 0x7f07340b3800 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c003a40 con 0x7f07340b3800 2024-09-13T16:29:33.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f07327fc700 1 -- 172.21.15.119:0/827613867 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724001670 con 0x7f07340b3800 2024-09-13T16:29:33.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- 172.21.15.119:0/827613867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07340b3800 msgr2=0x7f07341642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 --2- 172.21.15.119:0/827613867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07340b3800 0x7f07341642b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f072c009dc0 tx=0x7f072c00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:33.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- 172.21.15.119:0/827613867 shutdown_connections 2024-09-13T16:29:33.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 --2- 172.21.15.119:0/827613867 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07340b3800 0x7f07341642b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- 172.21.15.119:0/827613867 >> 172.21.15.119:0/827613867 conn(0x7f073406a350 msgr2=0x7f073406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:33.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- 172.21.15.119:0/827613867 shutdown_connections 2024-09-13T16:29:33.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.752+0000 7f0733fff700 1 -- 172.21.15.119:0/827613867 wait complete. 2024-09-13T16:29:33.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 Processor -- start 2024-09-13T16:29:33.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 -- start start 2024-09-13T16:29:33.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f073406dc40 0x7f0734148130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07340b3800 0x7f07341f00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07341594f0 0x7f07341f0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07341f12b0 con 0x7f073406dc40 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0734148670 con 0x7f07340b3800 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07341f0c20 con 0x7f07341594f0 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f073406dc40 0x7f0734148130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f073406dc40 0x7f0734148130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45954/0 (socket says 172.21.15.119:45954) 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 -- 172.21.15.119:0/1728172841 learned_addr learned my addr 172.21.15.119:0/1728172841 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f07337fe700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07341594f0 0x7f07341f0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f072a7fc700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07340b3800 0x7f07341f00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 -- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07341594f0 msgr2=0x7f07341f0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07341594f0 0x7f07341f0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 -- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07340b3800 msgr2=0x7f07341f00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07340b3800 0x7f07341f00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 -- 172.21.15.119:0/1728172841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c009a70 con 0x7f073406dc40 2024-09-13T16:29:33.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f072a7fc700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07340b3800 0x7f07341f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:33.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0732ffd700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f073406dc40 0x7f0734148130 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f072400c750 tx=0x7f072400ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:33.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f072bfff700 1 -- 172.21.15.119:0/1728172841 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724004d00 con 0x7f073406dc40 2024-09-13T16:29:33.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f072bfff700 1 -- 172.21.15.119:0/1728172841 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0724004e60 con 0x7f073406dc40 2024-09-13T16:29:33.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07341f0f00 con 0x7f073406dc40 2024-09-13T16:29:33.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f072bfff700 1 -- 172.21.15.119:0/1728172841 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07240072f0 con 0x7f073406dc40 2024-09-13T16:29:33.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07341f17a0 con 0x7f073406dc40 2024-09-13T16:29:33.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.756+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07340b26f0 con 0x7f073406dc40 2024-09-13T16:29:33.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.760+0000 7f072bfff700 1 -- 172.21.15.119:0/1728172841 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0724007450 con 0x7f073406dc40 2024-09-13T16:29:33.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.760+0000 7f072bfff700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0714066730 0x7f0714068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:33.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.760+0000 7f072bfff700 1 -- 172.21.15.119:0/1728172841 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0724083e10 con 0x7f073406dc40 2024-09-13T16:29:33.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.760+0000 7f072a7fc700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0714066730 0x7f0714068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:33.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.760+0000 7f072a7fc700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0714066730 0x7f0714068bf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f07341f3640 tx=0x7f071c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:33.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.768+0000 7f072bfff700 1 -- 172.21.15.119:0/1728172841 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0724051020 con 0x7f073406dc40 2024-09-13T16:29:33.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.940+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07340b26f0 con 0x7f073406dc40 2024-09-13T16:29:33.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.940+0000 7f072bfff700 1 -- 172.21.15.119:0/1728172841 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f07340b26f0 con 0x7f073406dc40 2024-09-13T16:29:33.946 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:33.947 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":82,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:31.230578+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:33.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0714066730 msgr2=0x7f0714068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0714066730 0x7f0714068bf0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f07341f3640 tx=0x7f071c009040 comp rx=0 tx=0).stop 2024-09-13T16:29:33.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f073406dc40 msgr2=0x7f0734148130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:33.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f073406dc40 0x7f0734148130 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f072400c750 tx=0x7f072400ca60 comp rx=0 tx=0).stop 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 shutdown_connections 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0714066730 0x7f0714068bf0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07341594f0 0x7f07341f0630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07340b3800 0x7f07341f00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 --2- 172.21.15.119:0/1728172841 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f073406dc40 0x7f0734148130 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 >> 172.21.15.119:0/1728172841 conn(0x7f073406a350 msgr2=0x7f073406d0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 shutdown_connections 2024-09-13T16:29:33.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:33.944+0000 7f0733fff700 1 -- 172.21.15.119:0/1728172841 wait complete. 2024-09-13T16:29:33.967 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:29:33.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:29:34.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 Processor -- start 2024-09-13T16:29:34.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 -- start start 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f07cc15e840 0x7f07cc070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f07cc15ec20 0x7f07cc161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07cc071210 con 0x7f07cc164a70 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc0714f0 con 0x7f07cc15ec20 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07cc0717d0 con 0x7f07cc15e840 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c9d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f07cc15ec20 0x7f07cc161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07cad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f07cc164a70 0x7f07cc0ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50628/0 (socket says 172.21.15.119:50628) 2024-09-13T16:29:34.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07cad9d700 1 -- 172.21.15.119:0/3430084911 learned_addr learned my addr 172.21.15.119:0/3430084911 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:34.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07ca59c700 1 --2- 172.21.15.119:0/3430084911 >> v2:172.21.15.145:3300/0 conn(0x7f07cc15e840 0x7f07cc070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c9d9b700 1 -- 172.21.15.119:0/3430084911 >> v2:172.21.15.145:3300/0 conn(0x7f07cc15e840 msgr2=0x7f07cc070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:34.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c9d9b700 1 --2- 172.21.15.119:0/3430084911 >> v2:172.21.15.145:3300/0 conn(0x7f07cc15e840 0x7f07cc070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c9d9b700 1 -- 172.21.15.119:0/3430084911 >> v1:172.21.15.119:6789/0 conn(0x7f07cc164a70 legacy=0x7f07cc0ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c9d9b700 1 -- 172.21.15.119:0/3430084911 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc071ab0 con 0x7f07cc15ec20 2024-09-13T16:29:34.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c959a700 1 -- 172.21.15.119:0/3430084911 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486867004 0 0) 0x7f07cc071210 con 0x7f07cc164a70 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c9d9b700 1 --2- 172.21.15.119:0/3430084911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15ec20 0x7f07cc161f00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f07cc06dec0 tx=0x7f07c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff58339ef1ed2104 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c959a700 1 -- 172.21.15.119:0/3430084911 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c400e070 con 0x7f07cc15ec20 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c959a700 1 -- 172.21.15.119:0/3430084911 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c400bb20 con 0x7f07cc15ec20 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07c959a700 1 -- 172.21.15.119:0/3430084911 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c4012430 con 0x7f07cc15ec20 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 -- 172.21.15.119:0/3430084911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15ec20 msgr2=0x7f07cc161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/3430084911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15ec20 0x7f07cc161f00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f07cc06dec0 tx=0x7f07c400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 -- 172.21.15.119:0/3430084911 shutdown_connections 2024-09-13T16:29:34.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/3430084911 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15ec20 0x7f07cc161f00 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/3430084911 >> v2:172.21.15.145:3300/0 conn(0x7f07cc15e840 0x7f07cc070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.072+0000 7f07d0a7e700 1 -- 172.21.15.119:0/3430084911 >> 172.21.15.119:0/3430084911 conn(0x7f07cc06a350 msgr2=0x7f07cc06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:34.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 -- 172.21.15.119:0/3430084911 shutdown_connections 2024-09-13T16:29:34.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 -- 172.21.15.119:0/3430084911 wait complete. 2024-09-13T16:29:34.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 Processor -- start 2024-09-13T16:29:34.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 -- start start 2024-09-13T16:29:34.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07cc15e840 0x7f07cc15ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07cc164a70 0x7f07cc159660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15f010 0x7f07cc0a88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc1f6700 con 0x7f07cc15f010 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc160440 con 0x7f07cc15e840 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07d0a7e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07cc159ba0 con 0x7f07cc164a70 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15f010 0x7f07cc0a88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15f010 0x7f07cc0a88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45964/0 (socket says 172.21.15.119:45964) 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 -- 172.21.15.119:0/2668874891 learned_addr learned my addr 172.21.15.119:0/2668874891 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07ca59c700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07cc15e840 0x7f07cc15ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07c9d9b700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07cc164a70 0x7f07cc159660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 -- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07cc164a70 msgr2=0x7f07cc159660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07cc164a70 0x7f07cc159660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 -- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07cc15e840 msgr2=0x7f07cc15ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07cc15e840 0x7f07cc15ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07cad9d700 1 -- 172.21.15.119:0/2668874891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c4009a70 con 0x7f07cc15f010 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.076+0000 7f07ca59c700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07cc15e840 0x7f07cc15ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07cad9d700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15f010 0x7f07cc0a88e0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00c520 tx=0x7f07bc00c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07baffd700 1 -- 172.21.15.119:0/2668874891 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc004ca0 con 0x7f07cc15f010 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07baffd700 1 -- 172.21.15.119:0/2668874891 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07bc004e00 con 0x7f07cc15f010 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07baffd700 1 -- 172.21.15.119:0/2668874891 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc007310 con 0x7f07cc15f010 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07cc159ff0 con 0x7f07cc15f010 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07cc15a640 con 0x7f07cc15f010 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07cc067760 con 0x7f07cc15f010 2024-09-13T16:29:34.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07baffd700 1 -- 172.21.15.119:0/2668874891 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f07bc007470 con 0x7f07cc15f010 2024-09-13T16:29:34.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07baffd700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07ac066760 0x7f07ac068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07baffd700 1 -- 172.21.15.119:0/2668874891 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f07bc083e40 con 0x7f07cc15f010 2024-09-13T16:29:34.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.080+0000 7f07ca59c700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07ac066760 0x7f07ac068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.084+0000 7f07ca59c700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07ac066760 0x7f07ac068c20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f07c0009ba0 tx=0x7f07c0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.088+0000 7f07baffd700 1 -- 172.21.15.119:0/2668874891 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f07bc051080 con 0x7f07cc15f010 2024-09-13T16:29:34.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.256+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07cc0ce160 con 0x7f07ac066760 2024-09-13T16:29:34.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.272+0000 7f07baffd700 1 -- 172.21.15.119:0/2668874891 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288054 (secure 0 0 0) 0x7f07cc0ce160 con 0x7f07ac066760 2024-09-13T16:29:34.283 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:34.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07ac066760 msgr2=0x7f07ac068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07ac066760 0x7f07ac068c20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f07c0009ba0 tx=0x7f07c0008040 comp rx=0 tx=0).stop 2024-09-13T16:29:34.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15f010 msgr2=0x7f07cc0a88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15f010 0x7f07cc0a88e0 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7f07bc00c520 tx=0x7f07bc00c830 comp rx=0 tx=0).stop 2024-09-13T16:29:34.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 shutdown_connections 2024-09-13T16:29:34.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07ac066760 0x7f07ac068c20 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07cc15f010 0x7f07cc0a88e0 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07cc164a70 0x7f07cc159660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 --2- 172.21.15.119:0/2668874891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07cc15e840 0x7f07cc15ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 >> 172.21.15.119:0/2668874891 conn(0x7f07cc06a350 msgr2=0x7f07cc163d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:34.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 shutdown_connections 2024-09-13T16:29:34.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.280+0000 7f07d0a7e700 1 -- 172.21.15.119:0/2668874891 wait complete. 2024-09-13T16:29:34.287 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:29:34.298 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-13T16:29:33.508894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241328,"kb_used_data":8756,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754733392,"statfs":{"total":773094113280,"available":772846993408,"internally_reserved":0,"allocated":8966144,"data_stored":4681528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.677601"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263121+0000","last_change":"2024-09-13T16:29:30.258109+0000","last_active":"2024-09-13T16:29:30.263121+0000","last_peered":"2024-09-13T16:29:30.263121+0000","last_clean":"2024-09-13T16:29:30.263121+0000","last_became_active":"2024-09-13T16:29:30.257846+0000","last_became_peered":"2024-09-13T16:29:30.257846+0000","last_unstale":"2024-09-13T16:29:30.263121+0000","last_undegraded":"2024-09-13T16:29:30.263121+0000","last_fullsized":"2024-09-13T16:29:30.263121+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:03.240866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.251391+0000","last_change":"2024-09-13T16:29:30.249889+0000","last_active":"2024-09-13T16:29:30.251391+0000","last_peered":"2024-09-13T16:29:30.251391+0000","last_clean":"2024-09-13T16:29:30.251391+0000","last_became_active":"2024-09-13T16:29:30.249652+0000","last_became_peered":"2024-09-13T16:29:30.249652+0000","last_unstale":"2024-09-13T16:29:30.251391+0000","last_undegraded":"2024-09-13T16:29:30.251391+0000","last_fullsized":"2024-09-13T16:29:30.251391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:02.799312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265326+0000","last_change":"2024-09-13T16:29:30.260697+0000","last_active":"2024-09-13T16:29:30.265326+0000","last_peered":"2024-09-13T16:29:30.265326+0000","last_clean":"2024-09-13T16:29:30.265326+0000","last_became_active":"2024-09-13T16:29:30.260459+0000","last_became_peered":"2024-09-13T16:29:30.260459+0000","last_unstale":"2024-09-13T16:29:30.265326+0000","last_undegraded":"2024-09-13T16:29:30.265326+0000","last_fullsized":"2024-09-13T16:29:30.265326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:24.720034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263271+0000","last_change":"2024-09-13T16:29:30.260166+0000","last_active":"2024-09-13T16:29:30.263271+0000","last_peered":"2024-09-13T16:29:30.263271+0000","last_clean":"2024-09-13T16:29:30.263271+0000","last_became_active":"2024-09-13T16:29:30.259481+0000","last_became_peered":"2024-09-13T16:29:30.259481+0000","last_unstale":"2024-09-13T16:29:30.263271+0000","last_undegraded":"2024-09-13T16:29:30.263271+0000","last_fullsized":"2024-09-13T16:29:30.263271+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:59.966990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.256468+0000","last_change":"2024-09-13T16:29:30.253113+0000","last_active":"2024-09-13T16:29:30.256468+0000","last_peered":"2024-09-13T16:29:30.256468+0000","last_clean":"2024-09-13T16:29:30.256468+0000","last_became_active":"2024-09-13T16:29:30.252862+0000","last_became_peered":"2024-09-13T16:29:30.252862+0000","last_unstale":"2024-09-13T16:29:30.256468+0000","last_undegraded":"2024-09-13T16:29:30.256468+0000","last_fullsized":"2024-09-13T16:29:30.256468+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:24:24.122924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264343+0000","last_change":"2024-09-13T16:29:30.253878+0000","last_active":"2024-09-13T16:29:30.264343+0000","last_peered":"2024-09-13T16:29:30.264343+0000","last_clean":"2024-09-13T16:29:30.264343+0000","last_became_active":"2024-09-13T16:29:30.253451+0000","last_became_peered":"2024-09-13T16:29:30.253451+0000","last_unstale":"2024-09-13T16:29:30.264343+0000","last_undegraded":"2024-09-13T16:29:30.264343+0000","last_fullsized":"2024-09-13T16:29:30.264343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:49.782649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259895+0000","last_change":"2024-09-13T16:29:30.252719+0000","last_active":"2024-09-13T16:29:30.259895+0000","last_peered":"2024-09-13T16:29:30.259895+0000","last_clean":"2024-09-13T16:29:30.259895+0000","last_became_active":"2024-09-13T16:29:30.252038+0000","last_became_peered":"2024-09-13T16:29:30.252038+0000","last_unstale":"2024-09-13T16:29:30.259895+0000","last_undegraded":"2024-09-13T16:29:30.259895+0000","last_fullsized":"2024-09-13T16:29:30.259895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:40.683901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.256729+0000","last_change":"2024-09-13T16:29:30.255029+0000","last_active":"2024-09-13T16:29:30.256729+0000","last_peered":"2024-09-13T16:29:30.256729+0000","last_clean":"2024-09-13T16:29:30.256729+0000","last_became_active":"2024-09-13T16:29:30.254775+0000","last_became_peered":"2024-09-13T16:29:30.254775+0000","last_unstale":"2024-09-13T16:29:30.256729+0000","last_undegraded":"2024-09-13T16:29:30.256729+0000","last_fullsized":"2024-09-13T16:29:30.256729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:34.014331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.260506+0000","last_change":"2024-09-13T16:29:30.254051+0000","last_active":"2024-09-13T16:29:30.260506+0000","last_peered":"2024-09-13T16:29:30.260506+0000","last_clean":"2024-09-13T16:29:30.260506+0000","last_became_active":"2024-09-13T16:29:30.253799+0000","last_became_peered":"2024-09-13T16:29:30.253799+0000","last_unstale":"2024-09-13T16:29:30.260506+0000","last_undegraded":"2024-09-13T16:29:30.260506+0000","last_fullsized":"2024-09-13T16:29:30.260506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:40.173333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263214+0000","last_change":"2024-09-13T16:29:30.261082+0000","last_active":"2024-09-13T16:29:30.263214+0000","last_peered":"2024-09-13T16:29:30.263214+0000","last_clean":"2024-09-13T16:29:30.263214+0000","last_became_active":"2024-09-13T16:29:30.260712+0000","last_became_peered":"2024-09-13T16:29:30.260712+0000","last_unstale":"2024-09-13T16:29:30.263214+0000","last_undegraded":"2024-09-13T16:29:30.263214+0000","last_fullsized":"2024-09-13T16:29:30.263214+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:32.156726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.253737+0000","last_change":"2024-09-13T16:29:30.251530+0000","last_active":"2024-09-13T16:29:30.253737+0000","last_peered":"2024-09-13T16:29:30.253737+0000","last_clean":"2024-09-13T16:29:30.253737+0000","last_became_active":"2024-09-13T16:29:30.251156+0000","last_became_peered":"2024-09-13T16:29:30.251156+0000","last_unstale":"2024-09-13T16:29:30.253737+0000","last_undegraded":"2024-09-13T16:29:30.253737+0000","last_fullsized":"2024-09-13T16:29:30.253737+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:37:00.412198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263251+0000","last_change":"2024-09-13T16:29:30.259989+0000","last_active":"2024-09-13T16:29:30.263251+0000","last_peered":"2024-09-13T16:29:30.263251+0000","last_clean":"2024-09-13T16:29:30.263251+0000","last_became_active":"2024-09-13T16:29:30.259203+0000","last_became_peered":"2024-09-13T16:29:30.259203+0000","last_unstale":"2024-09-13T16:29:30.263251+0000","last_undegraded":"2024-09-13T16:29:30.263251+0000","last_fullsized":"2024-09-13T16:29:30.263251+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:02.107647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.251527+0000","last_change":"2024-09-13T16:29:30.248356+0000","last_active":"2024-09-13T16:29:30.251527+0000","last_peered":"2024-09-13T16:29:30.251527+0000","last_clean":"2024-09-13T16:29:30.251527+0000","last_became_active":"2024-09-13T16:29:30.247978+0000","last_became_peered":"2024-09-13T16:29:30.247978+0000","last_unstale":"2024-09-13T16:29:30.251527+0000","last_undegraded":"2024-09-13T16:29:30.251527+0000","last_fullsized":"2024-09-13T16:29:30.251527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:42.259015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264387+0000","last_change":"2024-09-13T16:29:30.254454+0000","last_active":"2024-09-13T16:29:30.264387+0000","last_peered":"2024-09-13T16:29:30.264387+0000","last_clean":"2024-09-13T16:29:30.264387+0000","last_became_active":"2024-09-13T16:29:30.254225+0000","last_became_peered":"2024-09-13T16:29:30.254225+0000","last_unstale":"2024-09-13T16:29:30.264387+0000","last_undegraded":"2024-09-13T16:29:30.264387+0000","last_fullsized":"2024-09-13T16:29:30.264387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:16.956955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265196+0000","last_change":"2024-09-13T16:29:30.262367+0000","last_active":"2024-09-13T16:29:30.265196+0000","last_peered":"2024-09-13T16:29:30.265196+0000","last_clean":"2024-09-13T16:29:30.265196+0000","last_became_active":"2024-09-13T16:29:30.262113+0000","last_became_peered":"2024-09-13T16:29:30.262113+0000","last_unstale":"2024-09-13T16:29:30.265196+0000","last_undegraded":"2024-09-13T16:29:30.265196+0000","last_fullsized":"2024-09-13T16:29:30.265196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:09.708577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264314+0000","last_change":"2024-09-13T16:29:30.261364+0000","last_active":"2024-09-13T16:29:30.264314+0000","last_peered":"2024-09-13T16:29:30.264314+0000","last_clean":"2024-09-13T16:29:30.264314+0000","last_became_active":"2024-09-13T16:29:30.261085+0000","last_became_peered":"2024-09-13T16:29:30.261085+0000","last_unstale":"2024-09-13T16:29:30.264314+0000","last_undegraded":"2024-09-13T16:29:30.264314+0000","last_fullsized":"2024-09-13T16:29:30.264314+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:43.815907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267607+0000","last_change":"2024-09-13T16:29:30.257447+0000","last_active":"2024-09-13T16:29:30.267607+0000","last_peered":"2024-09-13T16:29:30.267607+0000","last_clean":"2024-09-13T16:29:30.267607+0000","last_became_active":"2024-09-13T16:29:30.257155+0000","last_became_peered":"2024-09-13T16:29:30.257155+0000","last_unstale":"2024-09-13T16:29:30.267607+0000","last_undegraded":"2024-09-13T16:29:30.267607+0000","last_fullsized":"2024-09-13T16:29:30.267607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:07.207976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265078+0000","last_change":"2024-09-13T16:29:30.259934+0000","last_active":"2024-09-13T16:29:30.265078+0000","last_peered":"2024-09-13T16:29:30.265078+0000","last_clean":"2024-09-13T16:29:30.265078+0000","last_became_active":"2024-09-13T16:29:30.256869+0000","last_became_peered":"2024-09-13T16:29:30.256869+0000","last_unstale":"2024-09-13T16:29:30.265078+0000","last_undegraded":"2024-09-13T16:29:30.265078+0000","last_fullsized":"2024-09-13T16:29:30.265078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:14.752545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258396+0000","last_change":"2024-09-13T16:29:30.250318+0000","last_active":"2024-09-13T16:29:30.258396+0000","last_peered":"2024-09-13T16:29:30.258396+0000","last_clean":"2024-09-13T16:29:30.258396+0000","last_became_active":"2024-09-13T16:29:30.249349+0000","last_became_peered":"2024-09-13T16:29:30.249349+0000","last_unstale":"2024-09-13T16:29:30.258396+0000","last_undegraded":"2024-09-13T16:29:30.258396+0000","last_fullsized":"2024-09-13T16:29:30.258396+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:52.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267934+0000","last_change":"2024-09-13T16:29:30.262427+0000","last_active":"2024-09-13T16:29:30.267934+0000","last_peered":"2024-09-13T16:29:30.267934+0000","last_clean":"2024-09-13T16:29:30.267934+0000","last_became_active":"2024-09-13T16:29:30.262130+0000","last_became_peered":"2024-09-13T16:29:30.262130+0000","last_unstale":"2024-09-13T16:29:30.267934+0000","last_undegraded":"2024-09-13T16:29:30.267934+0000","last_fullsized":"2024-09-13T16:29:30.267934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:52.631382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.269871+0000","last_change":"2024-09-13T16:29:30.262630+0000","last_active":"2024-09-13T16:29:30.269871+0000","last_peered":"2024-09-13T16:29:30.269871+0000","last_clean":"2024-09-13T16:29:30.269871+0000","last_became_active":"2024-09-13T16:29:30.262213+0000","last_became_peered":"2024-09-13T16:29:30.262213+0000","last_unstale":"2024-09-13T16:29:30.269871+0000","last_undegraded":"2024-09-13T16:29:30.269871+0000","last_fullsized":"2024-09-13T16:29:30.269871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:44.413492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.253040+0000","last_change":"2024-09-13T16:29:30.251591+0000","last_active":"2024-09-13T16:29:30.253040+0000","last_peered":"2024-09-13T16:29:30.253040+0000","last_clean":"2024-09-13T16:29:30.253040+0000","last_became_active":"2024-09-13T16:29:30.251353+0000","last_became_peered":"2024-09-13T16:29:30.251353+0000","last_unstale":"2024-09-13T16:29:30.253040+0000","last_undegraded":"2024-09-13T16:29:30.253040+0000","last_fullsized":"2024-09-13T16:29:30.253040+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:50.456926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265354+0000","last_change":"2024-09-13T16:29:30.260444+0000","last_active":"2024-09-13T16:29:30.265354+0000","last_peered":"2024-09-13T16:29:30.265354+0000","last_clean":"2024-09-13T16:29:30.265354+0000","last_became_active":"2024-09-13T16:29:30.260157+0000","last_became_peered":"2024-09-13T16:29:30.260157+0000","last_unstale":"2024-09-13T16:29:30.265354+0000","last_undegraded":"2024-09-13T16:29:30.265354+0000","last_fullsized":"2024-09-13T16:29:30.265354+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:55.100927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257097+0000","last_change":"2024-09-13T16:29:30.252581+0000","last_active":"2024-09-13T16:29:30.257097+0000","last_peered":"2024-09-13T16:29:30.257097+0000","last_clean":"2024-09-13T16:29:30.257097+0000","last_became_active":"2024-09-13T16:29:30.252309+0000","last_became_peered":"2024-09-13T16:29:30.252309+0000","last_unstale":"2024-09-13T16:29:30.257097+0000","last_undegraded":"2024-09-13T16:29:30.257097+0000","last_fullsized":"2024-09-13T16:29:30.257097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:20.283634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264268+0000","last_change":"2024-09-13T16:29:30.259070+0000","last_active":"2024-09-13T16:29:30.264268+0000","last_peered":"2024-09-13T16:29:30.264268+0000","last_clean":"2024-09-13T16:29:30.264268+0000","last_became_active":"2024-09-13T16:29:30.258797+0000","last_became_peered":"2024-09-13T16:29:30.258797+0000","last_unstale":"2024-09-13T16:29:30.264268+0000","last_undegraded":"2024-09-13T16:29:30.264268+0000","last_fullsized":"2024-09-13T16:29:30.264268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:52.350780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262088+0000","last_change":"2024-09-13T16:29:30.252808+0000","last_active":"2024-09-13T16:29:30.262088+0000","last_peered":"2024-09-13T16:29:30.262088+0000","last_clean":"2024-09-13T16:29:30.262088+0000","last_became_active":"2024-09-13T16:29:30.252543+0000","last_became_peered":"2024-09-13T16:29:30.252543+0000","last_unstale":"2024-09-13T16:29:30.262088+0000","last_undegraded":"2024-09-13T16:29:30.262088+0000","last_fullsized":"2024-09-13T16:29:30.262088+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:47.132567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264812+0000","last_change":"2024-09-13T16:29:30.262780+0000","last_active":"2024-09-13T16:29:30.264812+0000","last_peered":"2024-09-13T16:29:30.264812+0000","last_clean":"2024-09-13T16:29:30.264812+0000","last_became_active":"2024-09-13T16:29:30.262379+0000","last_became_peered":"2024-09-13T16:29:30.262379+0000","last_unstale":"2024-09-13T16:29:30.264812+0000","last_undegraded":"2024-09-13T16:29:30.264812+0000","last_fullsized":"2024-09-13T16:29:30.264812+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:25.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267955+0000","last_change":"2024-09-13T16:29:30.262940+0000","last_active":"2024-09-13T16:29:30.267955+0000","last_peered":"2024-09-13T16:29:30.267955+0000","last_clean":"2024-09-13T16:29:30.267955+0000","last_became_active":"2024-09-13T16:29:30.262606+0000","last_became_peered":"2024-09-13T16:29:30.262606+0000","last_unstale":"2024-09-13T16:29:30.267955+0000","last_undegraded":"2024-09-13T16:29:30.267955+0000","last_fullsized":"2024-09-13T16:29:30.267955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:54.494599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264536+0000","last_change":"2024-09-13T16:29:30.257620+0000","last_active":"2024-09-13T16:29:30.264536+0000","last_peered":"2024-09-13T16:29:30.264536+0000","last_clean":"2024-09-13T16:29:30.264536+0000","last_became_active":"2024-09-13T16:29:30.257420+0000","last_became_peered":"2024-09-13T16:29:30.257420+0000","last_unstale":"2024-09-13T16:29:30.264536+0000","last_undegraded":"2024-09-13T16:29:30.264536+0000","last_fullsized":"2024-09-13T16:29:30.264536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:41.788985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262360+0000","last_change":"2024-09-13T16:29:30.254900+0000","last_active":"2024-09-13T16:29:30.262360+0000","last_peered":"2024-09-13T16:29:30.262360+0000","last_clean":"2024-09-13T16:29:30.262360+0000","last_became_active":"2024-09-13T16:29:30.254549+0000","last_became_peered":"2024-09-13T16:29:30.254549+0000","last_unstale":"2024-09-13T16:29:30.262360+0000","last_undegraded":"2024-09-13T16:29:30.262360+0000","last_fullsized":"2024-09-13T16:29:30.262360+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:51.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265764+0000","last_change":"2024-09-13T16:29:30.261052+0000","last_active":"2024-09-13T16:29:30.265764+0000","last_peered":"2024-09-13T16:29:30.265764+0000","last_clean":"2024-09-13T16:29:30.265764+0000","last_became_active":"2024-09-13T16:29:30.260808+0000","last_became_peered":"2024-09-13T16:29:30.260808+0000","last_unstale":"2024-09-13T16:29:30.265764+0000","last_undegraded":"2024-09-13T16:29:30.265764+0000","last_fullsized":"2024-09-13T16:29:30.265764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:02.352140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263213+0000","last_change":"2024-09-13T16:29:30.258796+0000","last_active":"2024-09-13T16:29:30.263213+0000","last_peered":"2024-09-13T16:29:30.263213+0000","last_clean":"2024-09-13T16:29:30.263213+0000","last_became_active":"2024-09-13T16:29:30.258537+0000","last_became_peered":"2024-09-13T16:29:30.258537+0000","last_unstale":"2024-09-13T16:29:30.263213+0000","last_undegraded":"2024-09-13T16:29:30.263213+0000","last_fullsized":"2024-09-13T16:29:30.263213+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:05.208568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265233+0000","last_change":"2024-09-13T16:29:30.260060+0000","last_active":"2024-09-13T16:29:30.265233+0000","last_peered":"2024-09-13T16:29:30.265233+0000","last_clean":"2024-09-13T16:29:30.265233+0000","last_became_active":"2024-09-13T16:29:30.257054+0000","last_became_peered":"2024-09-13T16:29:30.257054+0000","last_unstale":"2024-09-13T16:29:30.265233+0000","last_undegraded":"2024-09-13T16:29:30.265233+0000","last_fullsized":"2024-09-13T16:29:30.265233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:19.378838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"81'4","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264839+0000","last_change":"2024-09-13T16:29:29.860118+0000","last_active":"2024-09-13T16:29:30.264839+0000","last_peered":"2024-09-13T16:29:30.264839+0000","last_clean":"2024-09-13T16:29:30.264839+0000","last_became_active":"2024-09-13T16:29:29.859826+0000","last_became_peered":"2024-09-13T16:29:29.859826+0000","last_unstale":"2024-09-13T16:29:30.264839+0000","last_undegraded":"2024-09-13T16:29:30.264839+0000","last_fullsized":"2024-09-13T16:29:30.264839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:56.602411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264403+0000","last_change":"2024-09-13T16:29:30.260101+0000","last_active":"2024-09-13T16:29:30.264403+0000","last_peered":"2024-09-13T16:29:30.264403+0000","last_clean":"2024-09-13T16:29:30.264403+0000","last_became_active":"2024-09-13T16:29:30.259847+0000","last_became_peered":"2024-09-13T16:29:30.259847+0000","last_unstale":"2024-09-13T16:29:30.264403+0000","last_undegraded":"2024-09-13T16:29:30.264403+0000","last_fullsized":"2024-09-13T16:29:30.264403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:52.150152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257840+0000","last_change":"2024-09-13T16:29:29.870139+0000","last_active":"2024-09-13T16:29:30.257840+0000","last_peered":"2024-09-13T16:29:30.257840+0000","last_clean":"2024-09-13T16:29:30.257840+0000","last_became_active":"2024-09-13T16:29:29.869871+0000","last_became_peered":"2024-09-13T16:29:29.869871+0000","last_unstale":"2024-09-13T16:29:30.257840+0000","last_undegraded":"2024-09-13T16:29:30.257840+0000","last_fullsized":"2024-09-13T16:29:30.257840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:23.706244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259530+0000","last_change":"2024-09-13T16:29:30.256762+0000","last_active":"2024-09-13T16:29:30.259530+0000","last_peered":"2024-09-13T16:29:30.259530+0000","last_clean":"2024-09-13T16:29:30.259530+0000","last_became_active":"2024-09-13T16:29:30.256429+0000","last_became_peered":"2024-09-13T16:29:30.256429+0000","last_unstale":"2024-09-13T16:29:30.259530+0000","last_undegraded":"2024-09-13T16:29:30.259530+0000","last_fullsized":"2024-09-13T16:29:30.259530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:14.752545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.255427+0000","last_change":"2024-09-13T16:29:29.869071+0000","last_active":"2024-09-13T16:29:30.255427+0000","last_peered":"2024-09-13T16:29:30.255427+0000","last_clean":"2024-09-13T16:29:30.255427+0000","last_became_active":"2024-09-13T16:29:29.868802+0000","last_became_peered":"2024-09-13T16:29:29.868802+0000","last_unstale":"2024-09-13T16:29:30.255427+0000","last_undegraded":"2024-09-13T16:29:30.255427+0000","last_fullsized":"2024-09-13T16:29:30.255427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:12.070475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.269841+0000","last_change":"2024-09-13T16:29:30.262440+0000","last_active":"2024-09-13T16:29:30.269841+0000","last_peered":"2024-09-13T16:29:30.269841+0000","last_clean":"2024-09-13T16:29:30.269841+0000","last_became_active":"2024-09-13T16:29:30.261923+0000","last_became_peered":"2024-09-13T16:29:30.261923+0000","last_unstale":"2024-09-13T16:29:30.269841+0000","last_undegraded":"2024-09-13T16:29:30.269841+0000","last_fullsized":"2024-09-13T16:29:30.269841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:19.378838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258543+0000","last_change":"2024-09-13T16:29:29.865971+0000","last_active":"2024-09-13T16:29:30.258543+0000","last_peered":"2024-09-13T16:29:30.258543+0000","last_clean":"2024-09-13T16:29:30.258543+0000","last_became_active":"2024-09-13T16:29:29.865679+0000","last_became_peered":"2024-09-13T16:29:29.865679+0000","last_unstale":"2024-09-13T16:29:30.258543+0000","last_undegraded":"2024-09-13T16:29:30.258543+0000","last_fullsized":"2024-09-13T16:29:30.258543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:11.294463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.251618+0000","last_change":"2024-09-13T16:29:30.248291+0000","last_active":"2024-09-13T16:29:30.251618+0000","last_peered":"2024-09-13T16:29:30.251618+0000","last_clean":"2024-09-13T16:29:30.251618+0000","last_became_active":"2024-09-13T16:29:30.248058+0000","last_became_peered":"2024-09-13T16:29:30.248058+0000","last_unstale":"2024-09-13T16:29:30.251618+0000","last_undegraded":"2024-09-13T16:29:30.251618+0000","last_fullsized":"2024-09-13T16:29:30.251618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:30.835106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257834+0000","last_change":"2024-09-13T16:29:29.859962+0000","last_active":"2024-09-13T16:29:30.257834+0000","last_peered":"2024-09-13T16:29:30.257834+0000","last_clean":"2024-09-13T16:29:30.257834+0000","last_became_active":"2024-09-13T16:29:29.859697+0000","last_became_peered":"2024-09-13T16:29:29.859697+0000","last_unstale":"2024-09-13T16:29:30.257834+0000","last_undegraded":"2024-09-13T16:29:30.257834+0000","last_fullsized":"2024-09-13T16:29:30.257834+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:56.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.253718+0000","last_change":"2024-09-13T16:29:30.251381+0000","last_active":"2024-09-13T16:29:30.253718+0000","last_peered":"2024-09-13T16:29:30.253718+0000","last_clean":"2024-09-13T16:29:30.253718+0000","last_became_active":"2024-09-13T16:29:30.250872+0000","last_became_peered":"2024-09-13T16:29:30.250872+0000","last_unstale":"2024-09-13T16:29:30.253718+0000","last_undegraded":"2024-09-13T16:29:30.253718+0000","last_fullsized":"2024-09-13T16:29:30.253718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:52.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264568+0000","last_change":"2024-09-13T16:29:29.859296+0000","last_active":"2024-09-13T16:29:30.264568+0000","last_peered":"2024-09-13T16:29:30.264568+0000","last_clean":"2024-09-13T16:29:30.264568+0000","last_became_active":"2024-09-13T16:29:29.858994+0000","last_became_peered":"2024-09-13T16:29:29.858994+0000","last_unstale":"2024-09-13T16:29:30.264568+0000","last_undegraded":"2024-09-13T16:29:30.264568+0000","last_fullsized":"2024-09-13T16:29:30.264568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:02.103386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265492+0000","last_change":"2024-09-13T16:29:30.261295+0000","last_active":"2024-09-13T16:29:30.265492+0000","last_peered":"2024-09-13T16:29:30.265492+0000","last_clean":"2024-09-13T16:29:30.265492+0000","last_became_active":"2024-09-13T16:29:30.258934+0000","last_became_peered":"2024-09-13T16:29:30.258934+0000","last_unstale":"2024-09-13T16:29:30.265492+0000","last_undegraded":"2024-09-13T16:29:30.265492+0000","last_fullsized":"2024-09-13T16:29:30.265492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:02.352140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.245320+0000","last_change":"2024-09-13T16:29:29.873185+0000","last_active":"2024-09-13T16:29:30.245320+0000","last_peered":"2024-09-13T16:29:30.245320+0000","last_clean":"2024-09-13T16:29:30.245320+0000","last_became_active":"2024-09-13T16:29:29.872925+0000","last_became_peered":"2024-09-13T16:29:29.872925+0000","last_unstale":"2024-09-13T16:29:30.245320+0000","last_undegraded":"2024-09-13T16:29:30.245320+0000","last_fullsized":"2024-09-13T16:29:30.245320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:14.452377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259764+0000","last_change":"2024-09-13T16:29:30.256577+0000","last_active":"2024-09-13T16:29:30.259764+0000","last_peered":"2024-09-13T16:29:30.259764+0000","last_clean":"2024-09-13T16:29:30.259764+0000","last_became_active":"2024-09-13T16:29:30.256230+0000","last_became_peered":"2024-09-13T16:29:30.256230+0000","last_unstale":"2024-09-13T16:29:30.259764+0000","last_undegraded":"2024-09-13T16:29:30.259764+0000","last_fullsized":"2024-09-13T16:29:30.259764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:49.795757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.239331+0000","last_change":"2024-09-13T16:29:29.855011+0000","last_active":"2024-09-13T16:29:30.239331+0000","last_peered":"2024-09-13T16:29:30.239331+0000","last_clean":"2024-09-13T16:29:30.239331+0000","last_became_active":"2024-09-13T16:29:29.854635+0000","last_became_peered":"2024-09-13T16:29:29.854635+0000","last_unstale":"2024-09-13T16:29:30.239331+0000","last_undegraded":"2024-09-13T16:29:30.239331+0000","last_fullsized":"2024-09-13T16:29:30.239331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:41.149648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263002+0000","last_change":"2024-09-13T16:29:30.255530+0000","last_active":"2024-09-13T16:29:30.263002+0000","last_peered":"2024-09-13T16:29:30.263002+0000","last_clean":"2024-09-13T16:29:30.263002+0000","last_became_active":"2024-09-13T16:29:30.255092+0000","last_became_peered":"2024-09-13T16:29:30.255092+0000","last_unstale":"2024-09-13T16:29:30.263002+0000","last_undegraded":"2024-09-13T16:29:30.263002+0000","last_fullsized":"2024-09-13T16:29:30.263002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:22.425269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.247279+0000","last_change":"2024-09-13T16:29:29.870344+0000","last_active":"2024-09-13T16:29:30.247279+0000","last_peered":"2024-09-13T16:29:30.247279+0000","last_clean":"2024-09-13T16:29:30.247279+0000","last_became_active":"2024-09-13T16:29:29.870057+0000","last_became_peered":"2024-09-13T16:29:29.870057+0000","last_unstale":"2024-09-13T16:29:30.247279+0000","last_undegraded":"2024-09-13T16:29:30.247279+0000","last_fullsized":"2024-09-13T16:29:30.247279+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:39.159543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262153+0000","last_change":"2024-09-13T16:29:30.253818+0000","last_active":"2024-09-13T16:29:30.262153+0000","last_peered":"2024-09-13T16:29:30.262153+0000","last_clean":"2024-09-13T16:29:30.262153+0000","last_became_active":"2024-09-13T16:29:30.253577+0000","last_became_peered":"2024-09-13T16:29:30.253577+0000","last_unstale":"2024-09-13T16:29:30.262153+0000","last_undegraded":"2024-09-13T16:29:30.262153+0000","last_fullsized":"2024-09-13T16:29:30.262153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:52.247246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.249333+0000","last_change":"2024-09-13T16:29:29.864978+0000","last_active":"2024-09-13T16:29:30.249333+0000","last_peered":"2024-09-13T16:29:30.249333+0000","last_clean":"2024-09-13T16:29:30.249333+0000","last_became_active":"2024-09-13T16:29:29.864680+0000","last_became_peered":"2024-09-13T16:29:29.864680+0000","last_unstale":"2024-09-13T16:29:30.249333+0000","last_undegraded":"2024-09-13T16:29:30.249333+0000","last_fullsized":"2024-09-13T16:29:30.249333+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:08.194445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264808+0000","last_change":"2024-09-13T16:29:30.263145+0000","last_active":"2024-09-13T16:29:30.264808+0000","last_peered":"2024-09-13T16:29:30.264808+0000","last_clean":"2024-09-13T16:29:30.264808+0000","last_became_active":"2024-09-13T16:29:30.262899+0000","last_became_peered":"2024-09-13T16:29:30.262899+0000","last_unstale":"2024-09-13T16:29:30.264808+0000","last_undegraded":"2024-09-13T16:29:30.264808+0000","last_fullsized":"2024-09-13T16:29:30.264808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:41.788985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"81'3","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265706+0000","last_change":"2024-09-13T16:29:29.862444+0000","last_active":"2024-09-13T16:29:30.265706+0000","last_peered":"2024-09-13T16:29:30.265706+0000","last_clean":"2024-09-13T16:29:30.265706+0000","last_became_active":"2024-09-13T16:29:29.861970+0000","last_became_peered":"2024-09-13T16:29:29.861970+0000","last_unstale":"2024-09-13T16:29:30.265706+0000","last_undegraded":"2024-09-13T16:29:30.265706+0000","last_fullsized":"2024-09-13T16:29:30.265706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:50.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265717+0000","last_change":"2024-09-13T16:29:30.261403+0000","last_active":"2024-09-13T16:29:30.265717+0000","last_peered":"2024-09-13T16:29:30.265717+0000","last_clean":"2024-09-13T16:29:30.265717+0000","last_became_active":"2024-09-13T16:29:30.256675+0000","last_became_peered":"2024-09-13T16:29:30.256675+0000","last_unstale":"2024-09-13T16:29:30.265717+0000","last_undegraded":"2024-09-13T16:29:30.265717+0000","last_fullsized":"2024-09-13T16:29:30.265717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:55.100927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"81'6","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.270251+0000","last_change":"2024-09-13T16:29:29.865189+0000","last_active":"2024-09-13T16:29:30.270251+0000","last_peered":"2024-09-13T16:29:30.270251+0000","last_clean":"2024-09-13T16:29:30.270251+0000","last_became_active":"2024-09-13T16:29:29.864888+0000","last_became_peered":"2024-09-13T16:29:29.864888+0000","last_unstale":"2024-09-13T16:29:30.270251+0000","last_undegraded":"2024-09-13T16:29:30.270251+0000","last_fullsized":"2024-09-13T16:29:30.270251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:05.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263285+0000","last_change":"2024-09-13T16:29:30.260331+0000","last_active":"2024-09-13T16:29:30.263285+0000","last_peered":"2024-09-13T16:29:30.263285+0000","last_clean":"2024-09-13T16:29:30.263285+0000","last_became_active":"2024-09-13T16:29:30.259759+0000","last_became_peered":"2024-09-13T16:29:30.259759+0000","last_unstale":"2024-09-13T16:29:30.263285+0000","last_undegraded":"2024-09-13T16:29:30.263285+0000","last_fullsized":"2024-09-13T16:29:30.263285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:15.622277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262860+0000","last_change":"2024-09-13T16:29:29.856532+0000","last_active":"2024-09-13T16:29:30.262860+0000","last_peered":"2024-09-13T16:29:30.262860+0000","last_clean":"2024-09-13T16:29:30.262860+0000","last_became_active":"2024-09-13T16:29:29.856284+0000","last_became_peered":"2024-09-13T16:29:29.856284+0000","last_unstale":"2024-09-13T16:29:30.262860+0000","last_undegraded":"2024-09-13T16:29:30.262860+0000","last_fullsized":"2024-09-13T16:29:30.262860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:48.216891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267304+0000","last_change":"2024-09-13T16:29:30.264265+0000","last_active":"2024-09-13T16:29:30.267304+0000","last_peered":"2024-09-13T16:29:30.267304+0000","last_clean":"2024-09-13T16:29:30.267304+0000","last_became_active":"2024-09-13T16:29:30.263914+0000","last_became_peered":"2024-09-13T16:29:30.263914+0000","last_unstale":"2024-09-13T16:29:30.267304+0000","last_undegraded":"2024-09-13T16:29:30.267304+0000","last_fullsized":"2024-09-13T16:29:30.267304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:44.413492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.247311+0000","last_change":"2024-09-13T16:29:29.864606+0000","last_active":"2024-09-13T16:29:30.247311+0000","last_peered":"2024-09-13T16:29:30.247311+0000","last_clean":"2024-09-13T16:29:30.247311+0000","last_became_active":"2024-09-13T16:29:29.864320+0000","last_became_peered":"2024-09-13T16:29:29.864320+0000","last_unstale":"2024-09-13T16:29:30.247311+0000","last_undegraded":"2024-09-13T16:29:30.247311+0000","last_fullsized":"2024-09-13T16:29:30.247311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:59.755778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259792+0000","last_change":"2024-09-13T16:29:30.255976+0000","last_active":"2024-09-13T16:29:30.259792+0000","last_peered":"2024-09-13T16:29:30.259792+0000","last_clean":"2024-09-13T16:29:30.259792+0000","last_became_active":"2024-09-13T16:29:30.253678+0000","last_became_peered":"2024-09-13T16:29:30.253678+0000","last_unstale":"2024-09-13T16:29:30.259792+0000","last_undegraded":"2024-09-13T16:29:30.259792+0000","last_fullsized":"2024-09-13T16:29:30.259792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:02.722209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.242993+0000","last_change":"2024-09-13T16:29:29.860045+0000","last_active":"2024-09-13T16:29:30.242993+0000","last_peered":"2024-09-13T16:29:30.242993+0000","last_clean":"2024-09-13T16:29:30.242993+0000","last_became_active":"2024-09-13T16:29:29.859742+0000","last_became_peered":"2024-09-13T16:29:29.859742+0000","last_unstale":"2024-09-13T16:29:30.242993+0000","last_undegraded":"2024-09-13T16:29:30.242993+0000","last_fullsized":"2024-09-13T16:29:30.242993+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:44.548677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267129+0000","last_change":"2024-09-13T16:29:30.259553+0000","last_active":"2024-09-13T16:29:30.267129+0000","last_peered":"2024-09-13T16:29:30.267129+0000","last_clean":"2024-09-13T16:29:30.267129+0000","last_became_active":"2024-09-13T16:29:30.259291+0000","last_became_peered":"2024-09-13T16:29:30.259291+0000","last_unstale":"2024-09-13T16:29:30.267129+0000","last_undegraded":"2024-09-13T16:29:30.267129+0000","last_fullsized":"2024-09-13T16:29:30.267129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:49.782420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"81'8","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267182+0000","last_change":"2024-09-13T16:29:29.871596+0000","last_active":"2024-09-13T16:29:30.267182+0000","last_peered":"2024-09-13T16:29:30.267182+0000","last_clean":"2024-09-13T16:29:30.267182+0000","last_became_active":"2024-09-13T16:29:29.871297+0000","last_became_peered":"2024-09-13T16:29:29.871297+0000","last_unstale":"2024-09-13T16:29:30.267182+0000","last_undegraded":"2024-09-13T16:29:30.267182+0000","last_fullsized":"2024-09-13T16:29:30.267182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:43.854027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258308+0000","last_change":"2024-09-13T16:29:30.249276+0000","last_active":"2024-09-13T16:29:30.258308+0000","last_peered":"2024-09-13T16:29:30.258308+0000","last_clean":"2024-09-13T16:29:30.258308+0000","last_became_active":"2024-09-13T16:29:30.246660+0000","last_became_peered":"2024-09-13T16:29:30.246660+0000","last_unstale":"2024-09-13T16:29:30.258308+0000","last_undegraded":"2024-09-13T16:29:30.258308+0000","last_fullsized":"2024-09-13T16:29:30.258308+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:52.350780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263482+0000","last_change":"2024-09-13T16:29:29.871723+0000","last_active":"2024-09-13T16:29:30.263482+0000","last_peered":"2024-09-13T16:29:30.263482+0000","last_clean":"2024-09-13T16:29:30.263482+0000","last_became_active":"2024-09-13T16:29:29.871410+0000","last_became_peered":"2024-09-13T16:29:29.871410+0000","last_unstale":"2024-09-13T16:29:30.263482+0000","last_undegraded":"2024-09-13T16:29:30.263482+0000","last_fullsized":"2024-09-13T16:29:30.263482+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:05.250770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267631+0000","last_change":"2024-09-13T16:29:30.257974+0000","last_active":"2024-09-13T16:29:30.267631+0000","last_peered":"2024-09-13T16:29:30.267631+0000","last_clean":"2024-09-13T16:29:30.267631+0000","last_became_active":"2024-09-13T16:29:30.257717+0000","last_became_peered":"2024-09-13T16:29:30.257717+0000","last_unstale":"2024-09-13T16:29:30.267631+0000","last_undegraded":"2024-09-13T16:29:30.267631+0000","last_fullsized":"2024-09-13T16:29:30.267631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:01.932370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263247+0000","last_change":"2024-09-13T16:29:29.856955+0000","last_active":"2024-09-13T16:29:30.263247+0000","last_peered":"2024-09-13T16:29:30.263247+0000","last_clean":"2024-09-13T16:29:30.263247+0000","last_became_active":"2024-09-13T16:29:29.856706+0000","last_became_peered":"2024-09-13T16:29:29.856706+0000","last_unstale":"2024-09-13T16:29:30.263247+0000","last_undegraded":"2024-09-13T16:29:30.263247+0000","last_fullsized":"2024-09-13T16:29:30.263247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:39.670111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.256491+0000","last_change":"2024-09-13T16:29:30.254371+0000","last_active":"2024-09-13T16:29:30.256491+0000","last_peered":"2024-09-13T16:29:30.256491+0000","last_clean":"2024-09-13T16:29:30.256491+0000","last_became_active":"2024-09-13T16:29:30.254148+0000","last_became_peered":"2024-09-13T16:29:30.254148+0000","last_unstale":"2024-09-13T16:29:30.256491+0000","last_undegraded":"2024-09-13T16:29:30.256491+0000","last_fullsized":"2024-09-13T16:29:30.256491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:44.190567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.260400+0000","last_change":"2024-09-13T16:29:29.854404+0000","last_active":"2024-09-13T16:29:30.260400+0000","last_peered":"2024-09-13T16:29:30.260400+0000","last_clean":"2024-09-13T16:29:30.260400+0000","last_became_active":"2024-09-13T16:29:29.854124+0000","last_became_peered":"2024-09-13T16:29:29.854124+0000","last_unstale":"2024-09-13T16:29:30.260400+0000","last_undegraded":"2024-09-13T16:29:30.260400+0000","last_fullsized":"2024-09-13T16:29:30.260400+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:09.097825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258261+0000","last_change":"2024-09-13T16:29:30.247291+0000","last_active":"2024-09-13T16:29:30.258261+0000","last_peered":"2024-09-13T16:29:30.258261+0000","last_clean":"2024-09-13T16:29:30.258261+0000","last_became_active":"2024-09-13T16:29:30.246985+0000","last_became_peered":"2024-09-13T16:29:30.246985+0000","last_unstale":"2024-09-13T16:29:30.258261+0000","last_undegraded":"2024-09-13T16:29:30.258261+0000","last_fullsized":"2024-09-13T16:29:30.258261+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:09.708577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.234412+0000","last_change":"2024-09-13T16:29:29.856038+0000","last_active":"2024-09-13T16:29:30.234412+0000","last_peered":"2024-09-13T16:29:30.234412+0000","last_clean":"2024-09-13T16:29:30.234412+0000","last_became_active":"2024-09-13T16:29:29.855743+0000","last_became_peered":"2024-09-13T16:29:29.855743+0000","last_unstale":"2024-09-13T16:29:30.234412+0000","last_undegraded":"2024-09-13T16:29:30.234412+0000","last_fullsized":"2024-09-13T16:29:30.234412+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:56.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264811+0000","last_change":"2024-09-13T16:29:30.261519+0000","last_active":"2024-09-13T16:29:30.264811+0000","last_peered":"2024-09-13T16:29:30.264811+0000","last_clean":"2024-09-13T16:29:30.264811+0000","last_became_active":"2024-09-13T16:29:30.261296+0000","last_became_peered":"2024-09-13T16:29:30.261296+0000","last_unstale":"2024-09-13T16:29:30.264811+0000","last_undegraded":"2024-09-13T16:29:30.264811+0000","last_fullsized":"2024-09-13T16:29:30.264811+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:12.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265652+0000","last_change":"2024-09-13T16:29:29.871640+0000","last_active":"2024-09-13T16:29:30.265652+0000","last_peered":"2024-09-13T16:29:30.265652+0000","last_clean":"2024-09-13T16:29:30.265652+0000","last_became_active":"2024-09-13T16:29:29.871354+0000","last_became_peered":"2024-09-13T16:29:29.871354+0000","last_unstale":"2024-09-13T16:29:30.265652+0000","last_undegraded":"2024-09-13T16:29:30.265652+0000","last_fullsized":"2024-09-13T16:29:30.265652+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:00:48.666775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.254406+0000","last_change":"2024-09-13T16:29:30.251974+0000","last_active":"2024-09-13T16:29:30.254406+0000","last_peered":"2024-09-13T16:29:30.254406+0000","last_clean":"2024-09-13T16:29:30.254406+0000","last_became_active":"2024-09-13T16:29:30.251723+0000","last_became_peered":"2024-09-13T16:29:30.251723+0000","last_unstale":"2024-09-13T16:29:30.254406+0000","last_undegraded":"2024-09-13T16:29:30.254406+0000","last_fullsized":"2024-09-13T16:29:30.254406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:16.956955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.239342+0000","last_change":"2024-09-13T16:29:29.870699+0000","last_active":"2024-09-13T16:29:30.239342+0000","last_peered":"2024-09-13T16:29:30.239342+0000","last_clean":"2024-09-13T16:29:30.239342+0000","last_became_active":"2024-09-13T16:29:29.870401+0000","last_became_peered":"2024-09-13T16:29:29.870401+0000","last_unstale":"2024-09-13T16:29:30.239342+0000","last_undegraded":"2024-09-13T16:29:30.239342+0000","last_fullsized":"2024-09-13T16:29:30.239342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:04.445993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259636+0000","last_change":"2024-09-13T16:29:30.255909+0000","last_active":"2024-09-13T16:29:30.259636+0000","last_peered":"2024-09-13T16:29:30.259636+0000","last_clean":"2024-09-13T16:29:30.259636+0000","last_became_active":"2024-09-13T16:29:30.253543+0000","last_became_peered":"2024-09-13T16:29:30.253543+0000","last_unstale":"2024-09-13T16:29:30.259636+0000","last_undegraded":"2024-09-13T16:29:30.259636+0000","last_fullsized":"2024-09-13T16:29:30.259636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:13.567625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.239702+0000","last_change":"2024-09-13T16:29:29.867061+0000","last_active":"2024-09-13T16:29:30.239702+0000","last_peered":"2024-09-13T16:29:30.239702+0000","last_clean":"2024-09-13T16:29:30.239702+0000","last_became_active":"2024-09-13T16:29:29.866214+0000","last_became_peered":"2024-09-13T16:29:29.866214+0000","last_unstale":"2024-09-13T16:29:30.239702+0000","last_undegraded":"2024-09-13T16:29:30.239702+0000","last_fullsized":"2024-09-13T16:29:30.239702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:08.978163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259295+0000","last_change":"2024-09-13T16:29:30.255305+0000","last_active":"2024-09-13T16:29:30.259295+0000","last_peered":"2024-09-13T16:29:30.259295+0000","last_clean":"2024-09-13T16:29:30.259295+0000","last_became_active":"2024-09-13T16:29:30.255032+0000","last_became_peered":"2024-09-13T16:29:30.255032+0000","last_unstale":"2024-09-13T16:29:30.259295+0000","last_undegraded":"2024-09-13T16:29:30.259295+0000","last_fullsized":"2024-09-13T16:29:30.259295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:39.022356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265456+0000","last_change":"2024-09-13T16:29:29.871682+0000","last_active":"2024-09-13T16:29:30.265456+0000","last_peered":"2024-09-13T16:29:30.265456+0000","last_clean":"2024-09-13T16:29:30.265456+0000","last_became_active":"2024-09-13T16:29:29.871313+0000","last_became_peered":"2024-09-13T16:29:29.871313+0000","last_unstale":"2024-09-13T16:29:30.265456+0000","last_undegraded":"2024-09-13T16:29:30.265456+0000","last_fullsized":"2024-09-13T16:29:30.265456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:52.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259878+0000","last_change":"2024-09-13T16:29:30.253135+0000","last_active":"2024-09-13T16:29:30.259878+0000","last_peered":"2024-09-13T16:29:30.259878+0000","last_clean":"2024-09-13T16:29:30.259878+0000","last_became_active":"2024-09-13T16:29:30.252391+0000","last_became_peered":"2024-09-13T16:29:30.252391+0000","last_unstale":"2024-09-13T16:29:30.259878+0000","last_undegraded":"2024-09-13T16:29:30.259878+0000","last_fullsized":"2024-09-13T16:29:30.259878+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:57.530692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.246191+0000","last_change":"2024-09-13T16:29:29.868661+0000","last_active":"2024-09-13T16:29:30.246191+0000","last_peered":"2024-09-13T16:29:30.246191+0000","last_clean":"2024-09-13T16:29:30.246191+0000","last_became_active":"2024-09-13T16:29:29.868338+0000","last_became_peered":"2024-09-13T16:29:29.868338+0000","last_unstale":"2024-09-13T16:29:30.246191+0000","last_undegraded":"2024-09-13T16:29:30.246191+0000","last_fullsized":"2024-09-13T16:29:30.246191+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:51.725685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257588+0000","last_change":"2024-09-13T16:29:30.246796+0000","last_active":"2024-09-13T16:29:30.257588+0000","last_peered":"2024-09-13T16:29:30.257588+0000","last_clean":"2024-09-13T16:29:30.257588+0000","last_became_active":"2024-09-13T16:29:30.246462+0000","last_became_peered":"2024-09-13T16:29:30.246462+0000","last_unstale":"2024-09-13T16:29:30.257588+0000","last_undegraded":"2024-09-13T16:29:30.257588+0000","last_fullsized":"2024-09-13T16:29:30.257588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:24.720034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.247409+0000","last_change":"2024-09-13T16:29:29.865040+0000","last_active":"2024-09-13T16:29:30.247409+0000","last_peered":"2024-09-13T16:29:30.247409+0000","last_clean":"2024-09-13T16:29:30.247409+0000","last_became_active":"2024-09-13T16:29:29.864721+0000","last_became_peered":"2024-09-13T16:29:29.864721+0000","last_unstale":"2024-09-13T16:29:30.247409+0000","last_undegraded":"2024-09-13T16:29:30.247409+0000","last_fullsized":"2024-09-13T16:29:30.247409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:11.242058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262024+0000","last_change":"2024-09-13T16:29:30.255661+0000","last_active":"2024-09-13T16:29:30.262024+0000","last_peered":"2024-09-13T16:29:30.262024+0000","last_clean":"2024-09-13T16:29:30.262024+0000","last_became_active":"2024-09-13T16:29:30.255212+0000","last_became_peered":"2024-09-13T16:29:30.255212+0000","last_unstale":"2024-09-13T16:29:30.262024+0000","last_undegraded":"2024-09-13T16:29:30.262024+0000","last_fullsized":"2024-09-13T16:29:30.262024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:49.782649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.246798+0000","last_change":"2024-09-13T16:29:29.866104+0000","last_active":"2024-09-13T16:29:30.246798+0000","last_peered":"2024-09-13T16:29:30.246798+0000","last_clean":"2024-09-13T16:29:30.246798+0000","last_became_active":"2024-09-13T16:29:29.865835+0000","last_became_peered":"2024-09-13T16:29:29.865835+0000","last_unstale":"2024-09-13T16:29:30.246798+0000","last_undegraded":"2024-09-13T16:29:30.246798+0000","last_fullsized":"2024-09-13T16:29:30.246798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:13.738755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265168+0000","last_change":"2024-09-13T16:29:30.258487+0000","last_active":"2024-09-13T16:29:30.265168+0000","last_peered":"2024-09-13T16:29:30.265168+0000","last_clean":"2024-09-13T16:29:30.265168+0000","last_became_active":"2024-09-13T16:29:30.258233+0000","last_became_peered":"2024-09-13T16:29:30.258233+0000","last_unstale":"2024-09-13T16:29:30.265168+0000","last_undegraded":"2024-09-13T16:29:30.265168+0000","last_fullsized":"2024-09-13T16:29:30.265168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:08.780755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.244642+0000","last_change":"2024-09-13T16:29:29.861511+0000","last_active":"2024-09-13T16:29:30.244642+0000","last_peered":"2024-09-13T16:29:30.244642+0000","last_clean":"2024-09-13T16:29:30.244642+0000","last_became_active":"2024-09-13T16:29:29.860942+0000","last_became_peered":"2024-09-13T16:29:29.860942+0000","last_unstale":"2024-09-13T16:29:30.244642+0000","last_undegraded":"2024-09-13T16:29:30.244642+0000","last_fullsized":"2024-09-13T16:29:30.244642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:58.492407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259376+0000","last_change":"2024-09-13T16:29:30.253238+0000","last_active":"2024-09-13T16:29:30.259376+0000","last_peered":"2024-09-13T16:29:30.259376+0000","last_clean":"2024-09-13T16:29:30.259376+0000","last_became_active":"2024-09-13T16:29:30.252949+0000","last_became_peered":"2024-09-13T16:29:30.252949+0000","last_unstale":"2024-09-13T16:29:30.259376+0000","last_undegraded":"2024-09-13T16:29:30.259376+0000","last_fullsized":"2024-09-13T16:29:30.259376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:55.100927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.243428+0000","last_change":"2024-09-13T16:29:29.860197+0000","last_active":"2024-09-13T16:29:30.243428+0000","last_peered":"2024-09-13T16:29:30.243428+0000","last_clean":"2024-09-13T16:29:30.243428+0000","last_became_active":"2024-09-13T16:29:29.859944+0000","last_became_peered":"2024-09-13T16:29:29.859944+0000","last_unstale":"2024-09-13T16:29:30.243428+0000","last_undegraded":"2024-09-13T16:29:30.243428+0000","last_fullsized":"2024-09-13T16:29:30.243428+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:23.356169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265306+0000","last_change":"2024-09-13T16:29:30.260302+0000","last_active":"2024-09-13T16:29:30.265306+0000","last_peered":"2024-09-13T16:29:30.265306+0000","last_clean":"2024-09-13T16:29:30.265306+0000","last_became_active":"2024-09-13T16:29:30.260087+0000","last_became_peered":"2024-09-13T16:29:30.260087+0000","last_unstale":"2024-09-13T16:29:30.265306+0000","last_undegraded":"2024-09-13T16:29:30.265306+0000","last_fullsized":"2024-09-13T16:29:30.265306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:44.413492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.244732+0000","last_change":"2024-09-13T16:29:29.862371+0000","last_active":"2024-09-13T16:29:30.244732+0000","last_peered":"2024-09-13T16:29:30.244732+0000","last_clean":"2024-09-13T16:29:30.244732+0000","last_became_active":"2024-09-13T16:29:29.861768+0000","last_became_peered":"2024-09-13T16:29:29.861768+0000","last_unstale":"2024-09-13T16:29:30.244732+0000","last_undegraded":"2024-09-13T16:29:30.244732+0000","last_fullsized":"2024-09-13T16:29:30.244732+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:34.311910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265298+0000","last_change":"2024-09-13T16:29:30.260437+0000","last_active":"2024-09-13T16:29:30.265298+0000","last_peered":"2024-09-13T16:29:30.265298+0000","last_clean":"2024-09-13T16:29:30.265298+0000","last_became_active":"2024-09-13T16:29:30.259728+0000","last_became_peered":"2024-09-13T16:29:30.259728+0000","last_unstale":"2024-09-13T16:29:30.265298+0000","last_undegraded":"2024-09-13T16:29:30.265298+0000","last_fullsized":"2024-09-13T16:29:30.265298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:39.022356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.234379+0000","last_change":"2024-09-13T16:29:29.862693+0000","last_active":"2024-09-13T16:29:30.234379+0000","last_peered":"2024-09-13T16:29:30.234379+0000","last_clean":"2024-09-13T16:29:30.234379+0000","last_became_active":"2024-09-13T16:29:29.862380+0000","last_became_peered":"2024-09-13T16:29:29.862380+0000","last_unstale":"2024-09-13T16:29:30.234379+0000","last_undegraded":"2024-09-13T16:29:30.234379+0000","last_fullsized":"2024-09-13T16:29:30.234379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.209560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.269906+0000","last_change":"2024-09-13T16:29:30.265587+0000","last_active":"2024-09-13T16:29:30.269906+0000","last_peered":"2024-09-13T16:29:30.269906+0000","last_clean":"2024-09-13T16:29:30.269906+0000","last_became_active":"2024-09-13T16:29:30.265324+0000","last_became_peered":"2024-09-13T16:29:30.265324+0000","last_unstale":"2024-09-13T16:29:30.269906+0000","last_undegraded":"2024-09-13T16:29:30.269906+0000","last_fullsized":"2024-09-13T16:29:30.269906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:02.630559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.242183+0000","last_change":"2024-09-13T16:29:29.866481+0000","last_active":"2024-09-13T16:29:30.242183+0000","last_peered":"2024-09-13T16:29:30.242183+0000","last_clean":"2024-09-13T16:29:30.242183+0000","last_became_active":"2024-09-13T16:29:29.866194+0000","last_became_peered":"2024-09-13T16:29:29.866194+0000","last_unstale":"2024-09-13T16:29:30.242183+0000","last_undegraded":"2024-09-13T16:29:30.242183+0000","last_fullsized":"2024-09-13T16:29:30.242183+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:11.053068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":215,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T16:29:29.855793+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:29:29.855793+0000","last_peered":"2024-09-13T16:29:29.855793+0000","last_clean":"2024-09-13T16:29:29.855793+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:29:29.855793+0000","last_undegraded":"2024-09-13T16:29:29.855793+0000","last_fullsized":"2024-09-13T16:29:29.855793+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542199,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1556480,"data_stored":1022954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1830000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.86899999999999999,"15min":0.85099999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1020000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.69899999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.84099999999999997,"15min":0.85099999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.85899999999999999,"15min":0.84699999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1140000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.81000000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.89000000000000001,"15min":0.88700000000000001},"min":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.85599999999999998,"15min":0.83899999999999997},"min":{"1min":0.64600000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.98299999999999998,"15min":1.0069999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.1579999999999999,"5min":1.238,"15min":1.238},"last":1.147},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.98899999999999999,"15min":1.004},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.2250000000000001,"5min":1.226,"15min":1.226},"last":0.85599999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.84499999999999997,"15min":0.879},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0309999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.022,"15min":1.0760000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.105,"5min":1.589,"15min":1.589},"last":1.0740000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.96799999999999997,"15min":1.02},"min":{"1min":0.67000000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.745},{"interface":"front","average":{"1min":1.026,"5min":1.0920000000000001,"15min":1.1140000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.1779999999999999,"5min":1.337,"15min":1.337},"last":1.1180000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.046,"15min":1.0760000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.123,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":1.04},{"interface":"front","average":{"1min":1.026,"5min":1.131,"15min":1.173},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.2270000000000001,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.2090000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.84799999999999998,"15min":0.85299999999999998},"min":{"1min":0.61899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.89800000000000002,"15min":0.89700000000000002},"min":{"1min":0.67700000000000005,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.202,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1799999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.92600000000000005,"15min":0.91800000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.165,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.871,"15min":0.88400000000000001},"min":{"1min":0.64400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.86199999999999999,"15min":0.84899999999999998},"min":{"1min":0.71099999999999997,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.95499999999999996,"15min":0.96399999999999997},"min":{"1min":0.72699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.536,"15min":1.536},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.75600000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.254,"5min":1.409,"15min":1.409},"last":0.88}]},{"osd":4,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95699999999999996,"15min":0.97499999999999998},"min":{"1min":0.62,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.365,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.97499999999999998,"15min":0.94399999999999995},"min":{"1min":0.71699999999999997,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1850000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.0549999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92100000000000004,"15min":0.91600000000000004},"min":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2110000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.038,"5min":1.1000000000000001,"15min":1.1299999999999999},"min":{"1min":0.73099999999999998,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.319,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.92200000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1160000000000001,"15min":1.1220000000000001},"min":{"1min":0.79000000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.3240000000000001,"5min":1.522,"15min":1.522},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.048,"15min":1.0549999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.27,"5min":1.337,"15min":1.337},"last":0.83899999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":991232,"data_stored":446310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.86299999999999999,"15min":0.86199999999999999},"min":{"1min":0.50800000000000001,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.01,"5min":1.411,"15min":1.411},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.53800000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2909999999999999,"5min":1.675,"15min":1.675},"last":0.626}]},{"osd":1,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.872,"15min":0.88200000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.117,"5min":1.369,"15min":1.369},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.84399999999999997,"15min":0.82899999999999996},"min":{"1min":0.61699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0169999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":1.054}]},{"osd":2,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.92400000000000004,"15min":0.93500000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.258,"5min":1.625,"15min":1.625},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.96099999999999997,"15min":0.98299999999999998},"min":{"1min":0.69299999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2090000000000001,"5min":1.552,"15min":1.552},"last":0.876}]},{"osd":3,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.874,"15min":0.84399999999999997},"min":{"1min":0.624,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1399999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.94999999999999996,"15min":1.0089999999999999},"min":{"1min":0.65800000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1140000000000001,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":0.71699999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.0009999999999999,"15min":1.0589999999999999},"min":{"1min":0.59399999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2549999999999999,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.028,"5min":1.0569999999999999,"15min":1.0589999999999999},"min":{"1min":0.76100000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.26,"5min":1.601,"15min":1.601},"last":0.93899999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":0.97199999999999998},"min":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.319,"5min":1.421,"15min":1.421},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0620000000000001,"15min":1.0740000000000001},"min":{"1min":0.73399999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3460000000000001,"5min":2.21,"15min":2.21},"last":0.96399999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.038,"15min":1.036},"min":{"1min":0.81599999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.2270000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.139,"15min":1.2170000000000001},"min":{"1min":0.89700000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.51,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.0009999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83499999999999996,"15min":0.83599999999999997},"min":{"1min":0.57099999999999995,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.157,"5min":1.411,"15min":1.411},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.88400000000000001,"15min":0.86199999999999999},"min":{"1min":0.68100000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1859999999999999,"5min":2.899,"15min":2.899},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.92400000000000004,"15min":0.91900000000000004},"min":{"1min":0.59999999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1879999999999999,"5min":3.085,"15min":3.085},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93100000000000005,"15min":0.92100000000000004},"min":{"1min":0.58299999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.155,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.129}]},{"osd":2,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.85599999999999998,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.137,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.747},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.97699999999999998,"15min":0.96999999999999997},"min":{"1min":0.73499999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.3400000000000001,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":0.91700000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95199999999999996,"15min":0.95699999999999996},"min":{"1min":0.65300000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1799999999999999,"5min":1.472,"15min":1.472},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.94899999999999995,"15min":0.96099999999999997},"min":{"1min":0.64400000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.25,"5min":1.577,"15min":1.577},"last":0.621}]},{"osd":5,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.264,"5min":2.996,"15min":2.996},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.294,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.68799999999999994}]},{"osd":6,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.93899999999999995,"15min":0.93999999999999995},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.1180000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.873},{"interface":"front","average":{"1min":1.028,"5min":1.133,"15min":1.1120000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4450000000000001,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.038}]},{"osd":7,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.135,"15min":1.143},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.476,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.1240000000000001,"15min":1.1200000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3680000000000001,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":1.0029999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542199,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":962560,"data_stored":432586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61899999999999999,"15min":0.63200000000000001},"min":{"1min":0.47499999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.872,"5min":1.04,"15min":1.04},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.47599999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.90100000000000002,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.77700000000000002,"15min":0.78500000000000003},"min":{"1min":0.501,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0249999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0649999999999999,"5min":1.27,"15min":1.27},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.76200000000000001,"15min":0.754},"min":{"1min":0.52000000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.016,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.73899999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.65900000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.204,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.73499999999999999,"5min":0.625,"15min":0.625},"max":{"1min":1.153,"5min":1.782,"15min":1.782},"last":1.0149999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.66100000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.194,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.073,"15min":1.0960000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.214,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.91500000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.093,"15min":1.1279999999999999},"min":{"1min":0.75,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.2390000000000001,"5min":1.885,"15min":1.885},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.032,"5min":1.024,"15min":1.0549999999999999},"min":{"1min":0.70499999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.3360000000000001,"5min":1.95,"15min":1.95},"last":0.83899999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0740000000000001,"15min":1.075},"min":{"1min":0.79900000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.2569999999999999,"5min":1.889,"15min":1.889},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0580000000000001,"15min":1.0609999999999999},"min":{"1min":0.76600000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5289999999999999,"5min":1.756,"15min":1.756},"last":0.93400000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542200,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.70399999999999996,"15min":0.69799999999999995},"min":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.69299999999999995,"15min":0.70599999999999996},"min":{"1min":0.47499999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94799999999999995,"5min":0.997,"15min":0.997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.69099999999999995},"min":{"1min":0.52600000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.47499999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.054,"5min":1.073,"15min":1.073},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.80700000000000005,"15min":0.79300000000000004},"min":{"1min":0.45500000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.1359999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.94999999999999996,"15min":0.95599999999999996},"min":{"1min":0.77300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.147,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.97999999999999998},"min":{"1min":0.78500000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1200000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.95999999999999996,"15min":0.97899999999999998},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.149,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95999999999999996},"min":{"1min":0.72299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.246,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.65200000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.97599999999999998,"15min":1.03},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1160000000000001,"5min":2.024,"15min":2.024},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.018,"5min":1.05,"15min":1.0640000000000001},"min":{"1min":0.78700000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3109999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0629999999999999,"15min":1.0800000000000001},"min":{"1min":0.93300000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.2310000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.999,"5min":1.0409999999999999,"15min":1.0649999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.2010000000000001,"5min":1.55,"15min":1.55},"last":0.752}]}]},{"osd":1,"up_from":14,"seq":60129542199,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.621,"15min":0.60599999999999998},"min":{"1min":0.371,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.86699999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.498},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75900000000000001,"15min":0.73399999999999999},"min":{"1min":0.55500000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.095,"5min":1.141,"15min":1.141},"last":0.89500000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65900000000000003,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.036,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70399999999999996,"15min":0.70199999999999996},"min":{"1min":0.45200000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90200000000000002,"5min":1.458,"15min":1.458},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.74199999999999999,"15min":0.69599999999999995},"min":{"1min":0.496,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.98499999999999999,"5min":1.236,"15min":1.236},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93300000000000005,"5min":1.403,"15min":1.403},"last":0.85699999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93200000000000005,"15min":0.90900000000000003},"min":{"1min":0.66500000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1379999999999999,"5min":1.546,"15min":1.546},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92300000000000004,"15min":0.93100000000000005},"min":{"1min":0.72499999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0680000000000001,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":1.046}]},{"osd":5,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.96999999999999997,"15min":0.95199999999999996},"min":{"1min":0.879,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3109999999999999,"5min":1.389,"15min":1.389},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.95699999999999996,"15min":0.97699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1140000000000001,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.98399999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0109999999999999,"15min":1.0089999999999999},"min":{"1min":0.73099999999999998,"5min":0.626,"15min":0.626},"max":{"1min":1.4810000000000001,"5min":1.613,"15min":1.613},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.054,"5min":1.0229999999999999,"15min":1.0149999999999999},"min":{"1min":0.83999999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.4199999999999999,"5min":1.698,"15min":1.698},"last":1.304}]},{"osd":7,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.004,"15min":1.0089999999999999},"min":{"1min":0.72499999999999998,"5min":0.62,"15min":0.62},"max":{"1min":1.4430000000000001,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.0640000000000001,"15min":1.0449999999999999},"min":{"1min":0.80000000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.546,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.0600000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542199,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1556480,"data_stored":1022954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.93500000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66500000000000004,"15min":0.65700000000000003},"min":{"1min":0.47499999999999998,"5min":0.377,"15min":0.377},"max":{"1min":1.0169999999999999,"5min":1.157,"15min":1.157},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71999999999999997,"15min":0.72299999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0609999999999999,"5min":1.202,"15min":1.202},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.70999999999999996,"15min":0.68700000000000006},"min":{"1min":0.58199999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1160000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.65500000000000003},"min":{"1min":0.437,"5min":0.312,"15min":0.312},"max":{"1min":1.0309999999999999,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.85599999999999998,"15min":0.82899999999999996},"min":{"1min":0.54900000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.284,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.80500000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.95799999999999996,"15min":0.96099999999999997},"min":{"1min":0.66100000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.28,"5min":1.486,"15min":1.486},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.97699999999999998,"15min":0.96099999999999997},"min":{"1min":0.69999999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3109999999999999,"5min":1.704,"15min":1.704},"last":0.82899999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0109999999999999,"15min":0.97999999999999998},"min":{"1min":0.748,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.446,"5min":1.575,"15min":1.575},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.99099999999999999,"15min":0.998},"min":{"1min":0.76700000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.3320000000000001,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.82499999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.2709999999999999,"5min":1.944,"15min":1.944},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.079,"15min":1.0549999999999999},"min":{"1min":0.82399999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.5,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0660000000000001,"15min":1.05},"min":{"1min":0.79900000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.343,"5min":1.659,"15min":1.659},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.093,"15min":1.048},"min":{"1min":0.90300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.51,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:29:34.304 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:29:34.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.412+0000 7f6fbb344700 1 Processor -- start 2024-09-13T16:29:34.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.412+0000 7f6fbb344700 1 -- start start 2024-09-13T16:29:34.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.412+0000 7f6fbb344700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6fb406dc40 0x7f6fb40b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.412+0000 7f6fbb344700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6fb40b3630 0x7f6fb406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.412+0000 7f6fbb344700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fb40b2280 con 0x7f6fb41594f0 2024-09-13T16:29:34.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.412+0000 7f6fbb344700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb40cc890 con 0x7f6fb40b3630 2024-09-13T16:29:34.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.412+0000 7f6fbb344700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fb40b1970 con 0x7f6fb406dc40 2024-09-13T16:29:34.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbab43700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6fb40b3630 0x7f6fb406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fba342700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6fb41594f0 0x7f6fb406d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50640/0 (socket says 172.21.15.119:50640) 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fba342700 1 -- 172.21.15.119:0/1519450502 learned_addr learned my addr 172.21.15.119:0/1519450502 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fb9b41700 1 --2- 172.21.15.119:0/1519450502 >> v2:172.21.15.145:3300/0 conn(0x7f6fb406dc40 0x7f6fb40b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbab43700 1 -- 172.21.15.119:0/1519450502 >> v2:172.21.15.145:3300/0 conn(0x7f6fb406dc40 msgr2=0x7f6fb40b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbab43700 1 --2- 172.21.15.119:0/1519450502 >> v2:172.21.15.145:3300/0 conn(0x7f6fb406dc40 0x7f6fb40b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbab43700 1 -- 172.21.15.119:0/1519450502 >> v1:172.21.15.119:6789/0 conn(0x7f6fb41594f0 legacy=0x7f6fb406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbab43700 1 -- 172.21.15.119:0/1519450502 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb40b1c30 con 0x7f6fb40b3630 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fb9340700 1 -- 172.21.15.119:0/1519450502 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407648023 0 0) 0x7f6fb40b2280 con 0x7f6fb41594f0 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbab43700 1 --2- 172.21.15.119:0/1519450502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb406fc00 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40cc010 tx=0x7f6fb0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6065deb87e6c75f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fb9340700 1 -- 172.21.15.119:0/1519450502 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0011830 con 0x7f6fb40b3630 2024-09-13T16:29:34.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fb9340700 1 -- 172.21.15.119:0/1519450502 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fb0011990 con 0x7f6fb40b3630 2024-09-13T16:29:34.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fb9340700 1 -- 172.21.15.119:0/1519450502 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fb0011d20 con 0x7f6fb40b3630 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 -- 172.21.15.119:0/1519450502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 msgr2=0x7f6fb406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1519450502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb406fc00 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7f6fb40cc010 tx=0x7f6fb0009b30 comp rx=0 tx=0).stop 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 -- 172.21.15.119:0/1519450502 shutdown_connections 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1519450502 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb406fc00 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1519450502 >> v2:172.21.15.145:3300/0 conn(0x7f6fb406dc40 0x7f6fb40b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 -- 172.21.15.119:0/1519450502 >> 172.21.15.119:0/1519450502 conn(0x7f6fb406a350 msgr2=0x7f6fb406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 -- 172.21.15.119:0/1519450502 shutdown_connections 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 -- 172.21.15.119:0/1519450502 wait complete. 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 Processor -- start 2024-09-13T16:29:34.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.416+0000 7f6fbb344700 1 -- start start 2024-09-13T16:29:34.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb406dc40 0x7f6fb414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb40a9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb41594f0 0x7f6fb4158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40aa3b0 con 0x7f6fb40b3630 2024-09-13T16:29:34.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40aa690 con 0x7f6fb406dc40 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fb40aa970 con 0x7f6fb41594f0 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb40a9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb40a9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45988/0 (socket says 172.21.15.119:45988) 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 -- 172.21.15.119:0/1755239074 learned_addr learned my addr 172.21.15.119:0/1755239074 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fba342700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb406dc40 0x7f6fb414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbab43700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb41594f0 0x7f6fb4158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 -- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb41594f0 msgr2=0x7f6fb4158250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb41594f0 0x7f6fb4158250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 -- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb406dc40 msgr2=0x7f6fb414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb406dc40 0x7f6fb414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 -- 172.21.15.119:0/1755239074 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fb00094b0 con 0x7f6fb40b3630 2024-09-13T16:29:34.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fb9b41700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb40a9960 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00efc0 tx=0x7f6fac00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6faaffd700 1 -- 172.21.15.119:0/1755239074 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac009ea0 con 0x7f6fb40b3630 2024-09-13T16:29:34.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6faaffd700 1 -- 172.21.15.119:0/1755239074 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fac00f040 con 0x7f6fb40b3630 2024-09-13T16:29:34.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fb40ac3c0 con 0x7f6fb40b3630 2024-09-13T16:29:34.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6faaffd700 1 -- 172.21.15.119:0/1755239074 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fac014480 con 0x7f6fb40b3630 2024-09-13T16:29:34.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.420+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fb41502c0 con 0x7f6fb40b3630 2024-09-13T16:29:34.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.424+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fb40b26f0 con 0x7f6fb40b3630 2024-09-13T16:29:34.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.424+0000 7f6faaffd700 1 -- 172.21.15.119:0/1755239074 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6fac0145e0 con 0x7f6fb40b3630 2024-09-13T16:29:34.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.424+0000 7f6faaffd700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6f98066730 0x7f6f98068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.424+0000 7f6faaffd700 1 -- 172.21.15.119:0/1755239074 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6fac084ad0 con 0x7f6fb40b3630 2024-09-13T16:29:34.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.424+0000 7f6fba342700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6f98066730 0x7f6f98068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.424+0000 7f6fba342700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6f98066730 0x7f6f98068bf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4000f60 tx=0x7f6fa4007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.432+0000 7f6faaffd700 1 -- 172.21.15.119:0/1755239074 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6fb40b26f0 con 0x7f6fb40b3630 2024-09-13T16:29:34.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.600+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6fb415aef0 con 0x7f6f98066730 2024-09-13T16:29:34.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.620+0000 7f6faaffd700 1 -- 172.21.15.119:0/1755239074 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288054 (secure 0 0 0) 0x7f6fb415aef0 con 0x7f6f98066730 2024-09-13T16:29:34.626 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:34.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6f98066730 msgr2=0x7f6f98068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6f98066730 0x7f6f98068bf0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4000f60 tx=0x7f6fa4007e90 comp rx=0 tx=0).stop 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 msgr2=0x7f6fb40a9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb40a9960 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7f6fac00efc0 tx=0x7f6fac00c3a0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 shutdown_connections 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6f98066730 0x7f6f98068bf0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6fb41594f0 0x7f6fb4158250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6fb40b3630 0x7f6fb40a9960 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 --2- 172.21.15.119:0/1755239074 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6fb406dc40 0x7f6fb414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 >> 172.21.15.119:0/1755239074 conn(0x7f6fb406a350 msgr2=0x7f6fb415b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 shutdown_connections 2024-09-13T16:29:34.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.624+0000 7f6fbb344700 1 -- 172.21.15.119:0/1755239074 wait complete. 2024-09-13T16:29:34.630 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:29:34.645 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":207,"stamp":"2024-09-13T16:29:33.508894+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":68,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":241328,"kb_used_data":8756,"kb_used_omap":12,"kb_used_meta":232435,"kb_avail":754733392,"statfs":{"total":773094113280,"available":772846993408,"internally_reserved":0,"allocated":8966144,"data_stored":4681528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":238014009},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.677601"},"pg_stats":[{"pgid":"12.3f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263121+0000","last_change":"2024-09-13T16:29:30.258109+0000","last_active":"2024-09-13T16:29:30.263121+0000","last_peered":"2024-09-13T16:29:30.263121+0000","last_clean":"2024-09-13T16:29:30.263121+0000","last_became_active":"2024-09-13T16:29:30.257846+0000","last_became_peered":"2024-09-13T16:29:30.257846+0000","last_unstale":"2024-09-13T16:29:30.263121+0000","last_undegraded":"2024-09-13T16:29:30.263121+0000","last_fullsized":"2024-09-13T16:29:30.263121+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:56:03.240866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.251391+0000","last_change":"2024-09-13T16:29:30.249889+0000","last_active":"2024-09-13T16:29:30.251391+0000","last_peered":"2024-09-13T16:29:30.251391+0000","last_clean":"2024-09-13T16:29:30.251391+0000","last_became_active":"2024-09-13T16:29:30.249652+0000","last_became_peered":"2024-09-13T16:29:30.249652+0000","last_unstale":"2024-09-13T16:29:30.251391+0000","last_undegraded":"2024-09-13T16:29:30.251391+0000","last_fullsized":"2024-09-13T16:29:30.251391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:02.799312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.3d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265326+0000","last_change":"2024-09-13T16:29:30.260697+0000","last_active":"2024-09-13T16:29:30.265326+0000","last_peered":"2024-09-13T16:29:30.265326+0000","last_clean":"2024-09-13T16:29:30.265326+0000","last_became_active":"2024-09-13T16:29:30.260459+0000","last_became_peered":"2024-09-13T16:29:30.260459+0000","last_unstale":"2024-09-13T16:29:30.265326+0000","last_undegraded":"2024-09-13T16:29:30.265326+0000","last_fullsized":"2024-09-13T16:29:30.265326+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:24.720034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.3c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263271+0000","last_change":"2024-09-13T16:29:30.260166+0000","last_active":"2024-09-13T16:29:30.263271+0000","last_peered":"2024-09-13T16:29:30.263271+0000","last_clean":"2024-09-13T16:29:30.263271+0000","last_became_active":"2024-09-13T16:29:30.259481+0000","last_became_peered":"2024-09-13T16:29:30.259481+0000","last_unstale":"2024-09-13T16:29:30.263271+0000","last_undegraded":"2024-09-13T16:29:30.263271+0000","last_fullsized":"2024-09-13T16:29:30.263271+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:59.966990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.256468+0000","last_change":"2024-09-13T16:29:30.253113+0000","last_active":"2024-09-13T16:29:30.256468+0000","last_peered":"2024-09-13T16:29:30.256468+0000","last_clean":"2024-09-13T16:29:30.256468+0000","last_became_active":"2024-09-13T16:29:30.252862+0000","last_became_peered":"2024-09-13T16:29:30.252862+0000","last_unstale":"2024-09-13T16:29:30.256468+0000","last_undegraded":"2024-09-13T16:29:30.256468+0000","last_fullsized":"2024-09-13T16:29:30.256468+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:24:24.122924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.3a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264343+0000","last_change":"2024-09-13T16:29:30.253878+0000","last_active":"2024-09-13T16:29:30.264343+0000","last_peered":"2024-09-13T16:29:30.264343+0000","last_clean":"2024-09-13T16:29:30.264343+0000","last_became_active":"2024-09-13T16:29:30.253451+0000","last_became_peered":"2024-09-13T16:29:30.253451+0000","last_unstale":"2024-09-13T16:29:30.264343+0000","last_undegraded":"2024-09-13T16:29:30.264343+0000","last_fullsized":"2024-09-13T16:29:30.264343+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:49.782649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.39","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259895+0000","last_change":"2024-09-13T16:29:30.252719+0000","last_active":"2024-09-13T16:29:30.259895+0000","last_peered":"2024-09-13T16:29:30.259895+0000","last_clean":"2024-09-13T16:29:30.259895+0000","last_became_active":"2024-09-13T16:29:30.252038+0000","last_became_peered":"2024-09-13T16:29:30.252038+0000","last_unstale":"2024-09-13T16:29:30.259895+0000","last_undegraded":"2024-09-13T16:29:30.259895+0000","last_fullsized":"2024-09-13T16:29:30.259895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:40.683901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.38","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.256729+0000","last_change":"2024-09-13T16:29:30.255029+0000","last_active":"2024-09-13T16:29:30.256729+0000","last_peered":"2024-09-13T16:29:30.256729+0000","last_clean":"2024-09-13T16:29:30.256729+0000","last_became_active":"2024-09-13T16:29:30.254775+0000","last_became_peered":"2024-09-13T16:29:30.254775+0000","last_unstale":"2024-09-13T16:29:30.256729+0000","last_undegraded":"2024-09-13T16:29:30.256729+0000","last_fullsized":"2024-09-13T16:29:30.256729+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:46:34.014331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.37","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.260506+0000","last_change":"2024-09-13T16:29:30.254051+0000","last_active":"2024-09-13T16:29:30.260506+0000","last_peered":"2024-09-13T16:29:30.260506+0000","last_clean":"2024-09-13T16:29:30.260506+0000","last_became_active":"2024-09-13T16:29:30.253799+0000","last_became_peered":"2024-09-13T16:29:30.253799+0000","last_unstale":"2024-09-13T16:29:30.260506+0000","last_undegraded":"2024-09-13T16:29:30.260506+0000","last_fullsized":"2024-09-13T16:29:30.260506+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:40.173333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.36","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263214+0000","last_change":"2024-09-13T16:29:30.261082+0000","last_active":"2024-09-13T16:29:30.263214+0000","last_peered":"2024-09-13T16:29:30.263214+0000","last_clean":"2024-09-13T16:29:30.263214+0000","last_became_active":"2024-09-13T16:29:30.260712+0000","last_became_peered":"2024-09-13T16:29:30.260712+0000","last_unstale":"2024-09-13T16:29:30.263214+0000","last_undegraded":"2024-09-13T16:29:30.263214+0000","last_fullsized":"2024-09-13T16:29:30.263214+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:32.156726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.35","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.253737+0000","last_change":"2024-09-13T16:29:30.251530+0000","last_active":"2024-09-13T16:29:30.253737+0000","last_peered":"2024-09-13T16:29:30.253737+0000","last_clean":"2024-09-13T16:29:30.253737+0000","last_became_active":"2024-09-13T16:29:30.251156+0000","last_became_peered":"2024-09-13T16:29:30.251156+0000","last_unstale":"2024-09-13T16:29:30.253737+0000","last_undegraded":"2024-09-13T16:29:30.253737+0000","last_fullsized":"2024-09-13T16:29:30.253737+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:37:00.412198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.34","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263251+0000","last_change":"2024-09-13T16:29:30.259989+0000","last_active":"2024-09-13T16:29:30.263251+0000","last_peered":"2024-09-13T16:29:30.263251+0000","last_clean":"2024-09-13T16:29:30.263251+0000","last_became_active":"2024-09-13T16:29:30.259203+0000","last_became_peered":"2024-09-13T16:29:30.259203+0000","last_unstale":"2024-09-13T16:29:30.263251+0000","last_undegraded":"2024-09-13T16:29:30.263251+0000","last_fullsized":"2024-09-13T16:29:30.263251+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:15:02.107647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.33","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.251527+0000","last_change":"2024-09-13T16:29:30.248356+0000","last_active":"2024-09-13T16:29:30.251527+0000","last_peered":"2024-09-13T16:29:30.251527+0000","last_clean":"2024-09-13T16:29:30.251527+0000","last_became_active":"2024-09-13T16:29:30.247978+0000","last_became_peered":"2024-09-13T16:29:30.247978+0000","last_unstale":"2024-09-13T16:29:30.251527+0000","last_undegraded":"2024-09-13T16:29:30.251527+0000","last_fullsized":"2024-09-13T16:29:30.251527+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:33:42.259015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.32","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264387+0000","last_change":"2024-09-13T16:29:30.254454+0000","last_active":"2024-09-13T16:29:30.264387+0000","last_peered":"2024-09-13T16:29:30.264387+0000","last_clean":"2024-09-13T16:29:30.264387+0000","last_became_active":"2024-09-13T16:29:30.254225+0000","last_became_peered":"2024-09-13T16:29:30.254225+0000","last_unstale":"2024-09-13T16:29:30.264387+0000","last_undegraded":"2024-09-13T16:29:30.264387+0000","last_fullsized":"2024-09-13T16:29:30.264387+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:16.956955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.31","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265196+0000","last_change":"2024-09-13T16:29:30.262367+0000","last_active":"2024-09-13T16:29:30.265196+0000","last_peered":"2024-09-13T16:29:30.265196+0000","last_clean":"2024-09-13T16:29:30.265196+0000","last_became_active":"2024-09-13T16:29:30.262113+0000","last_became_peered":"2024-09-13T16:29:30.262113+0000","last_unstale":"2024-09-13T16:29:30.265196+0000","last_undegraded":"2024-09-13T16:29:30.265196+0000","last_fullsized":"2024-09-13T16:29:30.265196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:09.708577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.30","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264314+0000","last_change":"2024-09-13T16:29:30.261364+0000","last_active":"2024-09-13T16:29:30.264314+0000","last_peered":"2024-09-13T16:29:30.264314+0000","last_clean":"2024-09-13T16:29:30.264314+0000","last_became_active":"2024-09-13T16:29:30.261085+0000","last_became_peered":"2024-09-13T16:29:30.261085+0000","last_unstale":"2024-09-13T16:29:30.264314+0000","last_undegraded":"2024-09-13T16:29:30.264314+0000","last_fullsized":"2024-09-13T16:29:30.264314+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:07:43.815907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.2f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267607+0000","last_change":"2024-09-13T16:29:30.257447+0000","last_active":"2024-09-13T16:29:30.267607+0000","last_peered":"2024-09-13T16:29:30.267607+0000","last_clean":"2024-09-13T16:29:30.267607+0000","last_became_active":"2024-09-13T16:29:30.257155+0000","last_became_peered":"2024-09-13T16:29:30.257155+0000","last_unstale":"2024-09-13T16:29:30.267607+0000","last_undegraded":"2024-09-13T16:29:30.267607+0000","last_fullsized":"2024-09-13T16:29:30.267607+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:43:07.207976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265078+0000","last_change":"2024-09-13T16:29:30.259934+0000","last_active":"2024-09-13T16:29:30.265078+0000","last_peered":"2024-09-13T16:29:30.265078+0000","last_clean":"2024-09-13T16:29:30.265078+0000","last_became_active":"2024-09-13T16:29:30.256869+0000","last_became_peered":"2024-09-13T16:29:30.256869+0000","last_unstale":"2024-09-13T16:29:30.265078+0000","last_undegraded":"2024-09-13T16:29:30.265078+0000","last_fullsized":"2024-09-13T16:29:30.265078+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:14.752545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.2d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258396+0000","last_change":"2024-09-13T16:29:30.250318+0000","last_active":"2024-09-13T16:29:30.258396+0000","last_peered":"2024-09-13T16:29:30.258396+0000","last_clean":"2024-09-13T16:29:30.258396+0000","last_became_active":"2024-09-13T16:29:30.249349+0000","last_became_peered":"2024-09-13T16:29:30.249349+0000","last_unstale":"2024-09-13T16:29:30.258396+0000","last_undegraded":"2024-09-13T16:29:30.258396+0000","last_fullsized":"2024-09-13T16:29:30.258396+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:52.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.2c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267934+0000","last_change":"2024-09-13T16:29:30.262427+0000","last_active":"2024-09-13T16:29:30.267934+0000","last_peered":"2024-09-13T16:29:30.267934+0000","last_clean":"2024-09-13T16:29:30.267934+0000","last_became_active":"2024-09-13T16:29:30.262130+0000","last_became_peered":"2024-09-13T16:29:30.262130+0000","last_unstale":"2024-09-13T16:29:30.267934+0000","last_undegraded":"2024-09-13T16:29:30.267934+0000","last_fullsized":"2024-09-13T16:29:30.267934+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:52.631382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.2b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.269871+0000","last_change":"2024-09-13T16:29:30.262630+0000","last_active":"2024-09-13T16:29:30.269871+0000","last_peered":"2024-09-13T16:29:30.269871+0000","last_clean":"2024-09-13T16:29:30.269871+0000","last_became_active":"2024-09-13T16:29:30.262213+0000","last_became_peered":"2024-09-13T16:29:30.262213+0000","last_unstale":"2024-09-13T16:29:30.269871+0000","last_undegraded":"2024-09-13T16:29:30.269871+0000","last_fullsized":"2024-09-13T16:29:30.269871+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:44.413492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.2a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.253040+0000","last_change":"2024-09-13T16:29:30.251591+0000","last_active":"2024-09-13T16:29:30.253040+0000","last_peered":"2024-09-13T16:29:30.253040+0000","last_clean":"2024-09-13T16:29:30.253040+0000","last_became_active":"2024-09-13T16:29:30.251353+0000","last_became_peered":"2024-09-13T16:29:30.251353+0000","last_unstale":"2024-09-13T16:29:30.253040+0000","last_undegraded":"2024-09-13T16:29:30.253040+0000","last_fullsized":"2024-09-13T16:29:30.253040+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:46:50.456926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.29","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265354+0000","last_change":"2024-09-13T16:29:30.260444+0000","last_active":"2024-09-13T16:29:30.265354+0000","last_peered":"2024-09-13T16:29:30.265354+0000","last_clean":"2024-09-13T16:29:30.265354+0000","last_became_active":"2024-09-13T16:29:30.260157+0000","last_became_peered":"2024-09-13T16:29:30.260157+0000","last_unstale":"2024-09-13T16:29:30.265354+0000","last_undegraded":"2024-09-13T16:29:30.265354+0000","last_fullsized":"2024-09-13T16:29:30.265354+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:55.100927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.28","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257097+0000","last_change":"2024-09-13T16:29:30.252581+0000","last_active":"2024-09-13T16:29:30.257097+0000","last_peered":"2024-09-13T16:29:30.257097+0000","last_clean":"2024-09-13T16:29:30.257097+0000","last_became_active":"2024-09-13T16:29:30.252309+0000","last_became_peered":"2024-09-13T16:29:30.252309+0000","last_unstale":"2024-09-13T16:29:30.257097+0000","last_undegraded":"2024-09-13T16:29:30.257097+0000","last_fullsized":"2024-09-13T16:29:30.257097+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:18:20.283634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.27","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264268+0000","last_change":"2024-09-13T16:29:30.259070+0000","last_active":"2024-09-13T16:29:30.264268+0000","last_peered":"2024-09-13T16:29:30.264268+0000","last_clean":"2024-09-13T16:29:30.264268+0000","last_became_active":"2024-09-13T16:29:30.258797+0000","last_became_peered":"2024-09-13T16:29:30.258797+0000","last_unstale":"2024-09-13T16:29:30.264268+0000","last_undegraded":"2024-09-13T16:29:30.264268+0000","last_fullsized":"2024-09-13T16:29:30.264268+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:52.350780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.26","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262088+0000","last_change":"2024-09-13T16:29:30.252808+0000","last_active":"2024-09-13T16:29:30.262088+0000","last_peered":"2024-09-13T16:29:30.262088+0000","last_clean":"2024-09-13T16:29:30.262088+0000","last_became_active":"2024-09-13T16:29:30.252543+0000","last_became_peered":"2024-09-13T16:29:30.252543+0000","last_unstale":"2024-09-13T16:29:30.262088+0000","last_undegraded":"2024-09-13T16:29:30.262088+0000","last_fullsized":"2024-09-13T16:29:30.262088+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:34:47.132567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.25","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264812+0000","last_change":"2024-09-13T16:29:30.262780+0000","last_active":"2024-09-13T16:29:30.264812+0000","last_peered":"2024-09-13T16:29:30.264812+0000","last_clean":"2024-09-13T16:29:30.264812+0000","last_became_active":"2024-09-13T16:29:30.262379+0000","last_became_peered":"2024-09-13T16:29:30.262379+0000","last_unstale":"2024-09-13T16:29:30.264812+0000","last_undegraded":"2024-09-13T16:29:30.264812+0000","last_fullsized":"2024-09-13T16:29:30.264812+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:25.834555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.24","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267955+0000","last_change":"2024-09-13T16:29:30.262940+0000","last_active":"2024-09-13T16:29:30.267955+0000","last_peered":"2024-09-13T16:29:30.267955+0000","last_clean":"2024-09-13T16:29:30.267955+0000","last_became_active":"2024-09-13T16:29:30.262606+0000","last_became_peered":"2024-09-13T16:29:30.262606+0000","last_unstale":"2024-09-13T16:29:30.267955+0000","last_undegraded":"2024-09-13T16:29:30.267955+0000","last_fullsized":"2024-09-13T16:29:30.267955+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:54.494599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.23","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264536+0000","last_change":"2024-09-13T16:29:30.257620+0000","last_active":"2024-09-13T16:29:30.264536+0000","last_peered":"2024-09-13T16:29:30.264536+0000","last_clean":"2024-09-13T16:29:30.264536+0000","last_became_active":"2024-09-13T16:29:30.257420+0000","last_became_peered":"2024-09-13T16:29:30.257420+0000","last_unstale":"2024-09-13T16:29:30.264536+0000","last_undegraded":"2024-09-13T16:29:30.264536+0000","last_fullsized":"2024-09-13T16:29:30.264536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:41.788985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.22","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262360+0000","last_change":"2024-09-13T16:29:30.254900+0000","last_active":"2024-09-13T16:29:30.262360+0000","last_peered":"2024-09-13T16:29:30.262360+0000","last_clean":"2024-09-13T16:29:30.262360+0000","last_became_active":"2024-09-13T16:29:30.254549+0000","last_became_peered":"2024-09-13T16:29:30.254549+0000","last_unstale":"2024-09-13T16:29:30.262360+0000","last_undegraded":"2024-09-13T16:29:30.262360+0000","last_fullsized":"2024-09-13T16:29:30.262360+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:04:51.113492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.21","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265764+0000","last_change":"2024-09-13T16:29:30.261052+0000","last_active":"2024-09-13T16:29:30.265764+0000","last_peered":"2024-09-13T16:29:30.265764+0000","last_clean":"2024-09-13T16:29:30.265764+0000","last_became_active":"2024-09-13T16:29:30.260808+0000","last_became_peered":"2024-09-13T16:29:30.260808+0000","last_unstale":"2024-09-13T16:29:30.265764+0000","last_undegraded":"2024-09-13T16:29:30.265764+0000","last_fullsized":"2024-09-13T16:29:30.265764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:02.352140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.20","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263213+0000","last_change":"2024-09-13T16:29:30.258796+0000","last_active":"2024-09-13T16:29:30.263213+0000","last_peered":"2024-09-13T16:29:30.263213+0000","last_clean":"2024-09-13T16:29:30.263213+0000","last_became_active":"2024-09-13T16:29:30.258537+0000","last_became_peered":"2024-09-13T16:29:30.258537+0000","last_unstale":"2024-09-13T16:29:30.263213+0000","last_undegraded":"2024-09-13T16:29:30.263213+0000","last_fullsized":"2024-09-13T16:29:30.263213+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:04:05.208568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265233+0000","last_change":"2024-09-13T16:29:30.260060+0000","last_active":"2024-09-13T16:29:30.265233+0000","last_peered":"2024-09-13T16:29:30.265233+0000","last_clean":"2024-09-13T16:29:30.265233+0000","last_became_active":"2024-09-13T16:29:30.257054+0000","last_became_peered":"2024-09-13T16:29:30.257054+0000","last_unstale":"2024-09-13T16:29:30.265233+0000","last_undegraded":"2024-09-13T16:29:30.265233+0000","last_fullsized":"2024-09-13T16:29:30.265233+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:19.378838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1f","version":"81'4","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264839+0000","last_change":"2024-09-13T16:29:29.860118+0000","last_active":"2024-09-13T16:29:30.264839+0000","last_peered":"2024-09-13T16:29:30.264839+0000","last_clean":"2024-09-13T16:29:30.264839+0000","last_became_active":"2024-09-13T16:29:29.859826+0000","last_became_peered":"2024-09-13T16:29:29.859826+0000","last_unstale":"2024-09-13T16:29:30.264839+0000","last_undegraded":"2024-09-13T16:29:30.264839+0000","last_fullsized":"2024-09-13T16:29:30.264839+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:03:56.602411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264403+0000","last_change":"2024-09-13T16:29:30.260101+0000","last_active":"2024-09-13T16:29:30.264403+0000","last_peered":"2024-09-13T16:29:30.264403+0000","last_clean":"2024-09-13T16:29:30.264403+0000","last_became_active":"2024-09-13T16:29:30.259847+0000","last_became_peered":"2024-09-13T16:29:30.259847+0000","last_unstale":"2024-09-13T16:29:30.264403+0000","last_undegraded":"2024-09-13T16:29:30.264403+0000","last_fullsized":"2024-09-13T16:29:30.264403+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:52.150152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1e","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257840+0000","last_change":"2024-09-13T16:29:29.870139+0000","last_active":"2024-09-13T16:29:30.257840+0000","last_peered":"2024-09-13T16:29:30.257840+0000","last_clean":"2024-09-13T16:29:30.257840+0000","last_became_active":"2024-09-13T16:29:29.869871+0000","last_became_peered":"2024-09-13T16:29:29.869871+0000","last_unstale":"2024-09-13T16:29:30.257840+0000","last_undegraded":"2024-09-13T16:29:30.257840+0000","last_fullsized":"2024-09-13T16:29:30.257840+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:23.706244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259530+0000","last_change":"2024-09-13T16:29:30.256762+0000","last_active":"2024-09-13T16:29:30.259530+0000","last_peered":"2024-09-13T16:29:30.259530+0000","last_clean":"2024-09-13T16:29:30.259530+0000","last_became_active":"2024-09-13T16:29:30.256429+0000","last_became_peered":"2024-09-13T16:29:30.256429+0000","last_unstale":"2024-09-13T16:29:30.259530+0000","last_undegraded":"2024-09-13T16:29:30.259530+0000","last_fullsized":"2024-09-13T16:29:30.259530+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:14.752545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.255427+0000","last_change":"2024-09-13T16:29:29.869071+0000","last_active":"2024-09-13T16:29:30.255427+0000","last_peered":"2024-09-13T16:29:30.255427+0000","last_clean":"2024-09-13T16:29:30.255427+0000","last_became_active":"2024-09-13T16:29:29.868802+0000","last_became_peered":"2024-09-13T16:29:29.868802+0000","last_unstale":"2024-09-13T16:29:30.255427+0000","last_undegraded":"2024-09-13T16:29:30.255427+0000","last_fullsized":"2024-09-13T16:29:30.255427+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:25:12.070475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.269841+0000","last_change":"2024-09-13T16:29:30.262440+0000","last_active":"2024-09-13T16:29:30.269841+0000","last_peered":"2024-09-13T16:29:30.269841+0000","last_clean":"2024-09-13T16:29:30.269841+0000","last_became_active":"2024-09-13T16:29:30.261923+0000","last_became_peered":"2024-09-13T16:29:30.261923+0000","last_unstale":"2024-09-13T16:29:30.269841+0000","last_undegraded":"2024-09-13T16:29:30.269841+0000","last_fullsized":"2024-09-13T16:29:30.269841+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:02:19.378838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1c","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258543+0000","last_change":"2024-09-13T16:29:29.865971+0000","last_active":"2024-09-13T16:29:30.258543+0000","last_peered":"2024-09-13T16:29:30.258543+0000","last_clean":"2024-09-13T16:29:30.258543+0000","last_became_active":"2024-09-13T16:29:29.865679+0000","last_became_peered":"2024-09-13T16:29:29.865679+0000","last_unstale":"2024-09-13T16:29:30.258543+0000","last_undegraded":"2024-09-13T16:29:30.258543+0000","last_fullsized":"2024-09-13T16:29:30.258543+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:20:11.294463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.251618+0000","last_change":"2024-09-13T16:29:30.248291+0000","last_active":"2024-09-13T16:29:30.251618+0000","last_peered":"2024-09-13T16:29:30.251618+0000","last_clean":"2024-09-13T16:29:30.251618+0000","last_became_active":"2024-09-13T16:29:30.248058+0000","last_became_peered":"2024-09-13T16:29:30.248058+0000","last_unstale":"2024-09-13T16:29:30.251618+0000","last_undegraded":"2024-09-13T16:29:30.251618+0000","last_fullsized":"2024-09-13T16:29:30.251618+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:30.835106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1b","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257834+0000","last_change":"2024-09-13T16:29:29.859962+0000","last_active":"2024-09-13T16:29:30.257834+0000","last_peered":"2024-09-13T16:29:30.257834+0000","last_clean":"2024-09-13T16:29:30.257834+0000","last_became_active":"2024-09-13T16:29:29.859697+0000","last_became_peered":"2024-09-13T16:29:29.859697+0000","last_unstale":"2024-09-13T16:29:30.257834+0000","last_undegraded":"2024-09-13T16:29:30.257834+0000","last_fullsized":"2024-09-13T16:29:30.257834+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:56.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.253718+0000","last_change":"2024-09-13T16:29:30.251381+0000","last_active":"2024-09-13T16:29:30.253718+0000","last_peered":"2024-09-13T16:29:30.253718+0000","last_clean":"2024-09-13T16:29:30.253718+0000","last_became_active":"2024-09-13T16:29:30.250872+0000","last_became_peered":"2024-09-13T16:29:30.250872+0000","last_unstale":"2024-09-13T16:29:30.253718+0000","last_undegraded":"2024-09-13T16:29:30.253718+0000","last_fullsized":"2024-09-13T16:29:30.253718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:08:52.117910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1a","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264568+0000","last_change":"2024-09-13T16:29:29.859296+0000","last_active":"2024-09-13T16:29:30.264568+0000","last_peered":"2024-09-13T16:29:30.264568+0000","last_clean":"2024-09-13T16:29:30.264568+0000","last_became_active":"2024-09-13T16:29:29.858994+0000","last_became_peered":"2024-09-13T16:29:29.858994+0000","last_unstale":"2024-09-13T16:29:30.264568+0000","last_undegraded":"2024-09-13T16:29:30.264568+0000","last_fullsized":"2024-09-13T16:29:30.264568+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:10:02.103386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265492+0000","last_change":"2024-09-13T16:29:30.261295+0000","last_active":"2024-09-13T16:29:30.265492+0000","last_peered":"2024-09-13T16:29:30.265492+0000","last_clean":"2024-09-13T16:29:30.265492+0000","last_became_active":"2024-09-13T16:29:30.258934+0000","last_became_peered":"2024-09-13T16:29:30.258934+0000","last_unstale":"2024-09-13T16:29:30.265492+0000","last_undegraded":"2024-09-13T16:29:30.265492+0000","last_fullsized":"2024-09-13T16:29:30.265492+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:08:02.352140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.245320+0000","last_change":"2024-09-13T16:29:29.873185+0000","last_active":"2024-09-13T16:29:30.245320+0000","last_peered":"2024-09-13T16:29:30.245320+0000","last_clean":"2024-09-13T16:29:30.245320+0000","last_became_active":"2024-09-13T16:29:29.872925+0000","last_became_peered":"2024-09-13T16:29:29.872925+0000","last_unstale":"2024-09-13T16:29:30.245320+0000","last_undegraded":"2024-09-13T16:29:30.245320+0000","last_fullsized":"2024-09-13T16:29:30.245320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:20:14.452377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.1f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259764+0000","last_change":"2024-09-13T16:29:30.256577+0000","last_active":"2024-09-13T16:29:30.259764+0000","last_peered":"2024-09-13T16:29:30.259764+0000","last_clean":"2024-09-13T16:29:30.259764+0000","last_became_active":"2024-09-13T16:29:30.256230+0000","last_became_peered":"2024-09-13T16:29:30.256230+0000","last_unstale":"2024-09-13T16:29:30.259764+0000","last_undegraded":"2024-09-13T16:29:30.259764+0000","last_fullsized":"2024-09-13T16:29:30.259764+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:42:49.795757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.239331+0000","last_change":"2024-09-13T16:29:29.855011+0000","last_active":"2024-09-13T16:29:30.239331+0000","last_peered":"2024-09-13T16:29:30.239331+0000","last_clean":"2024-09-13T16:29:30.239331+0000","last_became_active":"2024-09-13T16:29:29.854635+0000","last_became_peered":"2024-09-13T16:29:29.854635+0000","last_unstale":"2024-09-13T16:29:30.239331+0000","last_undegraded":"2024-09-13T16:29:30.239331+0000","last_fullsized":"2024-09-13T16:29:30.239331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:04:41.149648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.10","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263002+0000","last_change":"2024-09-13T16:29:30.255530+0000","last_active":"2024-09-13T16:29:30.263002+0000","last_peered":"2024-09-13T16:29:30.263002+0000","last_clean":"2024-09-13T16:29:30.263002+0000","last_became_active":"2024-09-13T16:29:30.255092+0000","last_became_peered":"2024-09-13T16:29:30.255092+0000","last_unstale":"2024-09-13T16:29:30.263002+0000","last_undegraded":"2024-09-13T16:29:30.263002+0000","last_fullsized":"2024-09-13T16:29:30.263002+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:20:22.425269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.247279+0000","last_change":"2024-09-13T16:29:29.870344+0000","last_active":"2024-09-13T16:29:30.247279+0000","last_peered":"2024-09-13T16:29:30.247279+0000","last_clean":"2024-09-13T16:29:30.247279+0000","last_became_active":"2024-09-13T16:29:29.870057+0000","last_became_peered":"2024-09-13T16:29:29.870057+0000","last_unstale":"2024-09-13T16:29:30.247279+0000","last_undegraded":"2024-09-13T16:29:30.247279+0000","last_fullsized":"2024-09-13T16:29:30.247279+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:39.159543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262153+0000","last_change":"2024-09-13T16:29:30.253818+0000","last_active":"2024-09-13T16:29:30.262153+0000","last_peered":"2024-09-13T16:29:30.262153+0000","last_clean":"2024-09-13T16:29:30.262153+0000","last_became_active":"2024-09-13T16:29:30.253577+0000","last_became_peered":"2024-09-13T16:29:30.253577+0000","last_unstale":"2024-09-13T16:29:30.262153+0000","last_undegraded":"2024-09-13T16:29:30.262153+0000","last_fullsized":"2024-09-13T16:29:30.262153+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:50:52.247246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.249333+0000","last_change":"2024-09-13T16:29:29.864978+0000","last_active":"2024-09-13T16:29:30.249333+0000","last_peered":"2024-09-13T16:29:30.249333+0000","last_clean":"2024-09-13T16:29:30.249333+0000","last_became_active":"2024-09-13T16:29:29.864680+0000","last_became_peered":"2024-09-13T16:29:29.864680+0000","last_unstale":"2024-09-13T16:29:30.249333+0000","last_undegraded":"2024-09-13T16:29:30.249333+0000","last_fullsized":"2024-09-13T16:29:30.249333+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:43:08.194445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264808+0000","last_change":"2024-09-13T16:29:30.263145+0000","last_active":"2024-09-13T16:29:30.264808+0000","last_peered":"2024-09-13T16:29:30.264808+0000","last_clean":"2024-09-13T16:29:30.264808+0000","last_became_active":"2024-09-13T16:29:30.262899+0000","last_became_peered":"2024-09-13T16:29:30.262899+0000","last_unstale":"2024-09-13T16:29:30.264808+0000","last_undegraded":"2024-09-13T16:29:30.264808+0000","last_fullsized":"2024-09-13T16:29:30.264808+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:41.788985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"81'3","reported_seq":13,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265706+0000","last_change":"2024-09-13T16:29:29.862444+0000","last_active":"2024-09-13T16:29:30.265706+0000","last_peered":"2024-09-13T16:29:30.265706+0000","last_clean":"2024-09-13T16:29:30.265706+0000","last_became_active":"2024-09-13T16:29:29.861970+0000","last_became_peered":"2024-09-13T16:29:29.861970+0000","last_unstale":"2024-09-13T16:29:30.265706+0000","last_undegraded":"2024-09-13T16:29:30.265706+0000","last_fullsized":"2024-09-13T16:29:30.265706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:44:50.196250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.13","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265717+0000","last_change":"2024-09-13T16:29:30.261403+0000","last_active":"2024-09-13T16:29:30.265717+0000","last_peered":"2024-09-13T16:29:30.265717+0000","last_clean":"2024-09-13T16:29:30.265717+0000","last_became_active":"2024-09-13T16:29:30.256675+0000","last_became_peered":"2024-09-13T16:29:30.256675+0000","last_unstale":"2024-09-13T16:29:30.265717+0000","last_undegraded":"2024-09-13T16:29:30.265717+0000","last_fullsized":"2024-09-13T16:29:30.265717+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:55.100927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.14","version":"81'6","reported_seq":16,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.270251+0000","last_change":"2024-09-13T16:29:29.865189+0000","last_active":"2024-09-13T16:29:30.270251+0000","last_peered":"2024-09-13T16:29:30.270251+0000","last_clean":"2024-09-13T16:29:30.270251+0000","last_became_active":"2024-09-13T16:29:29.864888+0000","last_became_peered":"2024-09-13T16:29:29.864888+0000","last_unstale":"2024-09-13T16:29:30.270251+0000","last_undegraded":"2024-09-13T16:29:30.270251+0000","last_fullsized":"2024-09-13T16:29:30.270251+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:47:05.602220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.14","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263285+0000","last_change":"2024-09-13T16:29:30.260331+0000","last_active":"2024-09-13T16:29:30.263285+0000","last_peered":"2024-09-13T16:29:30.263285+0000","last_clean":"2024-09-13T16:29:30.263285+0000","last_became_active":"2024-09-13T16:29:30.259759+0000","last_became_peered":"2024-09-13T16:29:30.259759+0000","last_unstale":"2024-09-13T16:29:30.263285+0000","last_undegraded":"2024-09-13T16:29:30.263285+0000","last_fullsized":"2024-09-13T16:29:30.263285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:15.622277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.13","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262860+0000","last_change":"2024-09-13T16:29:29.856532+0000","last_active":"2024-09-13T16:29:30.262860+0000","last_peered":"2024-09-13T16:29:30.262860+0000","last_clean":"2024-09-13T16:29:30.262860+0000","last_became_active":"2024-09-13T16:29:29.856284+0000","last_became_peered":"2024-09-13T16:29:29.856284+0000","last_unstale":"2024-09-13T16:29:30.262860+0000","last_undegraded":"2024-09-13T16:29:30.262860+0000","last_fullsized":"2024-09-13T16:29:30.262860+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:44:48.216891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.15","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267304+0000","last_change":"2024-09-13T16:29:30.264265+0000","last_active":"2024-09-13T16:29:30.267304+0000","last_peered":"2024-09-13T16:29:30.267304+0000","last_clean":"2024-09-13T16:29:30.267304+0000","last_became_active":"2024-09-13T16:29:30.263914+0000","last_became_peered":"2024-09-13T16:29:30.263914+0000","last_unstale":"2024-09-13T16:29:30.267304+0000","last_undegraded":"2024-09-13T16:29:30.267304+0000","last_fullsized":"2024-09-13T16:29:30.267304+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:44.413492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.247311+0000","last_change":"2024-09-13T16:29:29.864606+0000","last_active":"2024-09-13T16:29:30.247311+0000","last_peered":"2024-09-13T16:29:30.247311+0000","last_clean":"2024-09-13T16:29:30.247311+0000","last_became_active":"2024-09-13T16:29:29.864320+0000","last_became_peered":"2024-09-13T16:29:29.864320+0000","last_unstale":"2024-09-13T16:29:30.247311+0000","last_undegraded":"2024-09-13T16:29:30.247311+0000","last_fullsized":"2024-09-13T16:29:30.247311+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:56:59.755778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259792+0000","last_change":"2024-09-13T16:29:30.255976+0000","last_active":"2024-09-13T16:29:30.259792+0000","last_peered":"2024-09-13T16:29:30.259792+0000","last_clean":"2024-09-13T16:29:30.259792+0000","last_became_active":"2024-09-13T16:29:30.253678+0000","last_became_peered":"2024-09-13T16:29:30.253678+0000","last_unstale":"2024-09-13T16:29:30.259792+0000","last_undegraded":"2024-09-13T16:29:30.259792+0000","last_fullsized":"2024-09-13T16:29:30.259792+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:02.722209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.242993+0000","last_change":"2024-09-13T16:29:29.860045+0000","last_active":"2024-09-13T16:29:30.242993+0000","last_peered":"2024-09-13T16:29:30.242993+0000","last_clean":"2024-09-13T16:29:30.242993+0000","last_became_active":"2024-09-13T16:29:29.859742+0000","last_became_peered":"2024-09-13T16:29:29.859742+0000","last_unstale":"2024-09-13T16:29:30.242993+0000","last_undegraded":"2024-09-13T16:29:30.242993+0000","last_fullsized":"2024-09-13T16:29:30.242993+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:41:44.548677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267129+0000","last_change":"2024-09-13T16:29:30.259553+0000","last_active":"2024-09-13T16:29:30.267129+0000","last_peered":"2024-09-13T16:29:30.267129+0000","last_clean":"2024-09-13T16:29:30.267129+0000","last_became_active":"2024-09-13T16:29:30.259291+0000","last_became_peered":"2024-09-13T16:29:30.259291+0000","last_unstale":"2024-09-13T16:29:30.267129+0000","last_undegraded":"2024-09-13T16:29:30.267129+0000","last_fullsized":"2024-09-13T16:29:30.267129+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:34:49.782420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"81'8","reported_seq":18,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267182+0000","last_change":"2024-09-13T16:29:29.871596+0000","last_active":"2024-09-13T16:29:30.267182+0000","last_peered":"2024-09-13T16:29:30.267182+0000","last_clean":"2024-09-13T16:29:30.267182+0000","last_became_active":"2024-09-13T16:29:29.871297+0000","last_became_peered":"2024-09-13T16:29:29.871297+0000","last_unstale":"2024-09-13T16:29:30.267182+0000","last_undegraded":"2024-09-13T16:29:30.267182+0000","last_fullsized":"2024-09-13T16:29:30.267182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:30:43.854027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258308+0000","last_change":"2024-09-13T16:29:30.249276+0000","last_active":"2024-09-13T16:29:30.258308+0000","last_peered":"2024-09-13T16:29:30.258308+0000","last_clean":"2024-09-13T16:29:30.258308+0000","last_became_active":"2024-09-13T16:29:30.246660+0000","last_became_peered":"2024-09-13T16:29:30.246660+0000","last_unstale":"2024-09-13T16:29:30.258308+0000","last_undegraded":"2024-09-13T16:29:30.258308+0000","last_fullsized":"2024-09-13T16:29:30.258308+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:37:52.350780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.f","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263482+0000","last_change":"2024-09-13T16:29:29.871723+0000","last_active":"2024-09-13T16:29:30.263482+0000","last_peered":"2024-09-13T16:29:30.263482+0000","last_clean":"2024-09-13T16:29:30.263482+0000","last_became_active":"2024-09-13T16:29:29.871410+0000","last_became_peered":"2024-09-13T16:29:29.871410+0000","last_unstale":"2024-09-13T16:29:30.263482+0000","last_undegraded":"2024-09-13T16:29:30.263482+0000","last_fullsized":"2024-09-13T16:29:30.263482+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:55:05.250770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.267631+0000","last_change":"2024-09-13T16:29:30.257974+0000","last_active":"2024-09-13T16:29:30.267631+0000","last_peered":"2024-09-13T16:29:30.267631+0000","last_clean":"2024-09-13T16:29:30.267631+0000","last_became_active":"2024-09-13T16:29:30.257717+0000","last_became_peered":"2024-09-13T16:29:30.257717+0000","last_unstale":"2024-09-13T16:29:30.267631+0000","last_undegraded":"2024-09-13T16:29:30.267631+0000","last_fullsized":"2024-09-13T16:29:30.267631+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:01.932370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.e","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.263247+0000","last_change":"2024-09-13T16:29:29.856955+0000","last_active":"2024-09-13T16:29:30.263247+0000","last_peered":"2024-09-13T16:29:30.263247+0000","last_clean":"2024-09-13T16:29:30.263247+0000","last_became_active":"2024-09-13T16:29:29.856706+0000","last_became_peered":"2024-09-13T16:29:29.856706+0000","last_unstale":"2024-09-13T16:29:30.263247+0000","last_undegraded":"2024-09-13T16:29:30.263247+0000","last_fullsized":"2024-09-13T16:29:30.263247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:26:39.670111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.256491+0000","last_change":"2024-09-13T16:29:30.254371+0000","last_active":"2024-09-13T16:29:30.256491+0000","last_peered":"2024-09-13T16:29:30.256491+0000","last_clean":"2024-09-13T16:29:30.256491+0000","last_became_active":"2024-09-13T16:29:30.254148+0000","last_became_peered":"2024-09-13T16:29:30.254148+0000","last_unstale":"2024-09-13T16:29:30.256491+0000","last_undegraded":"2024-09-13T16:29:30.256491+0000","last_fullsized":"2024-09-13T16:29:30.256491+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:35:44.190567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.d","version":"81'1","reported_seq":11,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.260400+0000","last_change":"2024-09-13T16:29:29.854404+0000","last_active":"2024-09-13T16:29:30.260400+0000","last_peered":"2024-09-13T16:29:30.260400+0000","last_clean":"2024-09-13T16:29:30.260400+0000","last_became_active":"2024-09-13T16:29:29.854124+0000","last_became_peered":"2024-09-13T16:29:29.854124+0000","last_unstale":"2024-09-13T16:29:30.260400+0000","last_undegraded":"2024-09-13T16:29:30.260400+0000","last_fullsized":"2024-09-13T16:29:30.260400+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:22:09.097825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.258261+0000","last_change":"2024-09-13T16:29:30.247291+0000","last_active":"2024-09-13T16:29:30.258261+0000","last_peered":"2024-09-13T16:29:30.258261+0000","last_clean":"2024-09-13T16:29:30.258261+0000","last_became_active":"2024-09-13T16:29:30.246985+0000","last_became_peered":"2024-09-13T16:29:30.246985+0000","last_unstale":"2024-09-13T16:29:30.258261+0000","last_undegraded":"2024-09-13T16:29:30.258261+0000","last_fullsized":"2024-09-13T16:29:30.258261+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:44:09.708577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.234412+0000","last_change":"2024-09-13T16:29:29.856038+0000","last_active":"2024-09-13T16:29:30.234412+0000","last_peered":"2024-09-13T16:29:30.234412+0000","last_clean":"2024-09-13T16:29:30.234412+0000","last_became_active":"2024-09-13T16:29:29.855743+0000","last_became_peered":"2024-09-13T16:29:29.855743+0000","last_unstale":"2024-09-13T16:29:30.234412+0000","last_undegraded":"2024-09-13T16:29:30.234412+0000","last_fullsized":"2024-09-13T16:29:30.234412+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:25:56.939951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.264811+0000","last_change":"2024-09-13T16:29:30.261519+0000","last_active":"2024-09-13T16:29:30.264811+0000","last_peered":"2024-09-13T16:29:30.264811+0000","last_clean":"2024-09-13T16:29:30.264811+0000","last_became_active":"2024-09-13T16:29:30.261296+0000","last_became_peered":"2024-09-13T16:29:30.261296+0000","last_unstale":"2024-09-13T16:29:30.264811+0000","last_undegraded":"2024-09-13T16:29:30.264811+0000","last_fullsized":"2024-09-13T16:29:30.264811+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:12.066858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.b","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265652+0000","last_change":"2024-09-13T16:29:29.871640+0000","last_active":"2024-09-13T16:29:30.265652+0000","last_peered":"2024-09-13T16:29:30.265652+0000","last_clean":"2024-09-13T16:29:30.265652+0000","last_became_active":"2024-09-13T16:29:29.871354+0000","last_became_peered":"2024-09-13T16:29:29.871354+0000","last_unstale":"2024-09-13T16:29:30.265652+0000","last_undegraded":"2024-09-13T16:29:30.265652+0000","last_fullsized":"2024-09-13T16:29:30.265652+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:00:48.666775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.e","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.254406+0000","last_change":"2024-09-13T16:29:30.251974+0000","last_active":"2024-09-13T16:29:30.254406+0000","last_peered":"2024-09-13T16:29:30.254406+0000","last_clean":"2024-09-13T16:29:30.254406+0000","last_became_active":"2024-09-13T16:29:30.251723+0000","last_became_peered":"2024-09-13T16:29:30.251723+0000","last_unstale":"2024-09-13T16:29:30.254406+0000","last_undegraded":"2024-09-13T16:29:30.254406+0000","last_fullsized":"2024-09-13T16:29:30.254406+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:21:16.956955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.239342+0000","last_change":"2024-09-13T16:29:29.870699+0000","last_active":"2024-09-13T16:29:30.239342+0000","last_peered":"2024-09-13T16:29:30.239342+0000","last_clean":"2024-09-13T16:29:30.239342+0000","last_became_active":"2024-09-13T16:29:29.870401+0000","last_became_peered":"2024-09-13T16:29:29.870401+0000","last_unstale":"2024-09-13T16:29:30.239342+0000","last_undegraded":"2024-09-13T16:29:30.239342+0000","last_fullsized":"2024-09-13T16:29:30.239342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:07:04.445993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.f","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259636+0000","last_change":"2024-09-13T16:29:30.255909+0000","last_active":"2024-09-13T16:29:30.259636+0000","last_peered":"2024-09-13T16:29:30.259636+0000","last_clean":"2024-09-13T16:29:30.259636+0000","last_became_active":"2024-09-13T16:29:30.253543+0000","last_became_peered":"2024-09-13T16:29:30.253543+0000","last_unstale":"2024-09-13T16:29:30.259636+0000","last_undegraded":"2024-09-13T16:29:30.259636+0000","last_fullsized":"2024-09-13T16:29:30.259636+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:06:13.567625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.239702+0000","last_change":"2024-09-13T16:29:29.867061+0000","last_active":"2024-09-13T16:29:30.239702+0000","last_peered":"2024-09-13T16:29:30.239702+0000","last_clean":"2024-09-13T16:29:30.239702+0000","last_became_active":"2024-09-13T16:29:29.866214+0000","last_became_peered":"2024-09-13T16:29:29.866214+0000","last_unstale":"2024-09-13T16:29:30.239702+0000","last_undegraded":"2024-09-13T16:29:30.239702+0000","last_fullsized":"2024-09-13T16:29:30.239702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:08.978163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259295+0000","last_change":"2024-09-13T16:29:30.255305+0000","last_active":"2024-09-13T16:29:30.259295+0000","last_peered":"2024-09-13T16:29:30.259295+0000","last_clean":"2024-09-13T16:29:30.259295+0000","last_became_active":"2024-09-13T16:29:30.255032+0000","last_became_peered":"2024-09-13T16:29:30.255032+0000","last_unstale":"2024-09-13T16:29:30.259295+0000","last_undegraded":"2024-09-13T16:29:30.259295+0000","last_fullsized":"2024-09-13T16:29:30.259295+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:39.022356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.7","version":"81'2","reported_seq":12,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265456+0000","last_change":"2024-09-13T16:29:29.871682+0000","last_active":"2024-09-13T16:29:30.265456+0000","last_peered":"2024-09-13T16:29:30.265456+0000","last_clean":"2024-09-13T16:29:30.265456+0000","last_became_active":"2024-09-13T16:29:29.871313+0000","last_became_peered":"2024-09-13T16:29:29.871313+0000","last_unstale":"2024-09-13T16:29:30.265456+0000","last_undegraded":"2024-09-13T16:29:30.265456+0000","last_fullsized":"2024-09-13T16:29:30.265456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:01:52.400533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259878+0000","last_change":"2024-09-13T16:29:30.253135+0000","last_active":"2024-09-13T16:29:30.259878+0000","last_peered":"2024-09-13T16:29:30.259878+0000","last_clean":"2024-09-13T16:29:30.259878+0000","last_became_active":"2024-09-13T16:29:30.252391+0000","last_became_peered":"2024-09-13T16:29:30.252391+0000","last_unstale":"2024-09-13T16:29:30.259878+0000","last_undegraded":"2024-09-13T16:29:30.259878+0000","last_fullsized":"2024-09-13T16:29:30.259878+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:40:57.530692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.246191+0000","last_change":"2024-09-13T16:29:29.868661+0000","last_active":"2024-09-13T16:29:30.246191+0000","last_peered":"2024-09-13T16:29:30.246191+0000","last_clean":"2024-09-13T16:29:30.246191+0000","last_became_active":"2024-09-13T16:29:29.868338+0000","last_became_peered":"2024-09-13T16:29:29.868338+0000","last_unstale":"2024-09-13T16:29:30.246191+0000","last_undegraded":"2024-09-13T16:29:30.246191+0000","last_fullsized":"2024-09-13T16:29:30.246191+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:17:51.725685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.257588+0000","last_change":"2024-09-13T16:29:30.246796+0000","last_active":"2024-09-13T16:29:30.257588+0000","last_peered":"2024-09-13T16:29:30.257588+0000","last_clean":"2024-09-13T16:29:30.257588+0000","last_became_active":"2024-09-13T16:29:30.246462+0000","last_became_peered":"2024-09-13T16:29:30.246462+0000","last_unstale":"2024-09-13T16:29:30.257588+0000","last_undegraded":"2024-09-13T16:29:30.257588+0000","last_fullsized":"2024-09-13T16:29:30.257588+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:22:24.720034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.247409+0000","last_change":"2024-09-13T16:29:29.865040+0000","last_active":"2024-09-13T16:29:30.247409+0000","last_peered":"2024-09-13T16:29:30.247409+0000","last_clean":"2024-09-13T16:29:30.247409+0000","last_became_active":"2024-09-13T16:29:29.864721+0000","last_became_peered":"2024-09-13T16:29:29.864721+0000","last_unstale":"2024-09-13T16:29:30.247409+0000","last_undegraded":"2024-09-13T16:29:30.247409+0000","last_fullsized":"2024-09-13T16:29:30.247409+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:54:11.242058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.262024+0000","last_change":"2024-09-13T16:29:30.255661+0000","last_active":"2024-09-13T16:29:30.262024+0000","last_peered":"2024-09-13T16:29:30.262024+0000","last_clean":"2024-09-13T16:29:30.262024+0000","last_became_active":"2024-09-13T16:29:30.255212+0000","last_became_peered":"2024-09-13T16:29:30.255212+0000","last_unstale":"2024-09-13T16:29:30.262024+0000","last_undegraded":"2024-09-13T16:29:30.262024+0000","last_fullsized":"2024-09-13T16:29:30.262024+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:29:49.782649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.246798+0000","last_change":"2024-09-13T16:29:29.866104+0000","last_active":"2024-09-13T16:29:30.246798+0000","last_peered":"2024-09-13T16:29:30.246798+0000","last_clean":"2024-09-13T16:29:30.246798+0000","last_became_active":"2024-09-13T16:29:29.865835+0000","last_became_peered":"2024-09-13T16:29:29.865835+0000","last_unstale":"2024-09-13T16:29:30.246798+0000","last_undegraded":"2024-09-13T16:29:30.246798+0000","last_fullsized":"2024-09-13T16:29:30.246798+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:13.738755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265168+0000","last_change":"2024-09-13T16:29:30.258487+0000","last_active":"2024-09-13T16:29:30.265168+0000","last_peered":"2024-09-13T16:29:30.265168+0000","last_clean":"2024-09-13T16:29:30.265168+0000","last_became_active":"2024-09-13T16:29:30.258233+0000","last_became_peered":"2024-09-13T16:29:30.258233+0000","last_unstale":"2024-09-13T16:29:30.265168+0000","last_undegraded":"2024-09-13T16:29:30.265168+0000","last_fullsized":"2024-09-13T16:29:30.265168+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:08.780755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.244642+0000","last_change":"2024-09-13T16:29:29.861511+0000","last_active":"2024-09-13T16:29:30.244642+0000","last_peered":"2024-09-13T16:29:30.244642+0000","last_clean":"2024-09-13T16:29:30.244642+0000","last_became_active":"2024-09-13T16:29:29.860942+0000","last_became_peered":"2024-09-13T16:29:29.860942+0000","last_unstale":"2024-09-13T16:29:30.244642+0000","last_undegraded":"2024-09-13T16:29:30.244642+0000","last_fullsized":"2024-09-13T16:29:30.244642+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:58.492407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.259376+0000","last_change":"2024-09-13T16:29:30.253238+0000","last_active":"2024-09-13T16:29:30.259376+0000","last_peered":"2024-09-13T16:29:30.259376+0000","last_clean":"2024-09-13T16:29:30.259376+0000","last_became_active":"2024-09-13T16:29:30.252949+0000","last_became_peered":"2024-09-13T16:29:30.252949+0000","last_unstale":"2024-09-13T16:29:30.259376+0000","last_undegraded":"2024-09-13T16:29:30.259376+0000","last_fullsized":"2024-09-13T16:29:30.259376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:09:55.100927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.243428+0000","last_change":"2024-09-13T16:29:29.860197+0000","last_active":"2024-09-13T16:29:30.243428+0000","last_peered":"2024-09-13T16:29:30.243428+0000","last_clean":"2024-09-13T16:29:30.243428+0000","last_became_active":"2024-09-13T16:29:29.859944+0000","last_became_peered":"2024-09-13T16:29:29.859944+0000","last_unstale":"2024-09-13T16:29:30.243428+0000","last_undegraded":"2024-09-13T16:29:30.243428+0000","last_fullsized":"2024-09-13T16:29:30.243428+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:04:23.356169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265306+0000","last_change":"2024-09-13T16:29:30.260302+0000","last_active":"2024-09-13T16:29:30.265306+0000","last_peered":"2024-09-13T16:29:30.265306+0000","last_clean":"2024-09-13T16:29:30.265306+0000","last_became_active":"2024-09-13T16:29:30.260087+0000","last_became_peered":"2024-09-13T16:29:30.260087+0000","last_unstale":"2024-09-13T16:29:30.265306+0000","last_undegraded":"2024-09-13T16:29:30.265306+0000","last_fullsized":"2024-09-13T16:29:30.265306+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:44.413492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.244732+0000","last_change":"2024-09-13T16:29:29.862371+0000","last_active":"2024-09-13T16:29:30.244732+0000","last_peered":"2024-09-13T16:29:30.244732+0000","last_clean":"2024-09-13T16:29:30.244732+0000","last_became_active":"2024-09-13T16:29:29.861768+0000","last_became_peered":"2024-09-13T16:29:29.861768+0000","last_unstale":"2024-09-13T16:29:30.244732+0000","last_undegraded":"2024-09-13T16:29:30.244732+0000","last_fullsized":"2024-09-13T16:29:30.244732+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:27:34.311910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.7","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.265298+0000","last_change":"2024-09-13T16:29:30.260437+0000","last_active":"2024-09-13T16:29:30.265298+0000","last_peered":"2024-09-13T16:29:30.265298+0000","last_clean":"2024-09-13T16:29:30.265298+0000","last_became_active":"2024-09-13T16:29:30.259728+0000","last_became_peered":"2024-09-13T16:29:30.259728+0000","last_unstale":"2024-09-13T16:29:30.265298+0000","last_undegraded":"2024-09-13T16:29:30.265298+0000","last_fullsized":"2024-09-13T16:29:30.265298+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:39.022356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.234379+0000","last_change":"2024-09-13T16:29:29.862693+0000","last_active":"2024-09-13T16:29:30.234379+0000","last_peered":"2024-09-13T16:29:30.234379+0000","last_clean":"2024-09-13T16:29:30.234379+0000","last_became_active":"2024-09-13T16:29:29.862380+0000","last_became_peered":"2024-09-13T16:29:29.862380+0000","last_unstale":"2024-09-13T16:29:30.234379+0000","last_undegraded":"2024-09-13T16:29:30.234379+0000","last_fullsized":"2024-09-13T16:29:30.234379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:50.209560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"0'0","reported_seq":9,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.269906+0000","last_change":"2024-09-13T16:29:30.265587+0000","last_active":"2024-09-13T16:29:30.269906+0000","last_peered":"2024-09-13T16:29:30.269906+0000","last_clean":"2024-09-13T16:29:30.269906+0000","last_became_active":"2024-09-13T16:29:30.265324+0000","last_became_peered":"2024-09-13T16:29:30.265324+0000","last_unstale":"2024-09-13T16:29:30.269906+0000","last_undegraded":"2024-09-13T16:29:30.269906+0000","last_fullsized":"2024-09-13T16:29:30.269906+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:29.830688+0000","last_clean_scrub_stamp":"2024-09-13T16:29:29.830688+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:47:02.630559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":10,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-09-13T16:29:30.242183+0000","last_change":"2024-09-13T16:29:29.866481+0000","last_active":"2024-09-13T16:29:30.242183+0000","last_peered":"2024-09-13T16:29:30.242183+0000","last_clean":"2024-09-13T16:29:30.242183+0000","last_became_active":"2024-09-13T16:29:29.866194+0000","last_became_peered":"2024-09-13T16:29:29.866194+0000","last_unstale":"2024-09-13T16:29:30.242183+0000","last_undegraded":"2024-09-13T16:29:30.242183+0000","last_fullsized":"2024-09-13T16:29:30.242183+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:29:28.816898+0000","last_clean_scrub_stamp":"2024-09-13T16:29:28.816898+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:51:11.053068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":215,"reported_epoch":80,"state":"active+clean","last_fresh":"2024-09-13T16:29:29.855793+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:29:29.855793+0000","last_peered":"2024-09-13T16:29:29.855793+0000","last_clean":"2024-09-13T16:29:29.855793+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:29:29.855793+0000","last_undegraded":"2024-09-13T16:29:29.855793+0000","last_fullsized":"2024-09-13T16:29:29.855793+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":12,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":11,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542199,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1556480,"data_stored":1022954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.80700000000000005,"5min":0.85399999999999998,"15min":0.85099999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1830000000000001,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.90800000000000003},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.86899999999999999,"15min":0.85099999999999998},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.1020000000000001,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.69899999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.84099999999999997,"15min":0.85099999999999998},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.2,"5min":1.2,"15min":1.2},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.85899999999999999,"15min":0.84699999999999998},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1140000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.81000000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.89000000000000001,"15min":0.88700000000000001},"min":{"1min":0.59899999999999998,"5min":0.56699999999999995,"15min":0.56699999999999995},"max":{"1min":1.242,"5min":1.242,"15min":1.242},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.85599999999999998,"15min":0.83899999999999997},"min":{"1min":0.64600000000000002,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"last":0.64600000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.98299999999999998,"15min":1.0069999999999999},"min":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.1579999999999999,"5min":1.238,"15min":1.238},"last":1.147},{"interface":"front","average":{"1min":0.97299999999999998,"5min":0.98899999999999999,"15min":1.004},"min":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.2250000000000001,"5min":1.226,"15min":1.226},"last":0.85599999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.84499999999999997,"15min":0.879},"min":{"1min":0.59799999999999998,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.0309999999999999,"5min":1.1559999999999999,"15min":1.1559999999999999},"last":0.94899999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":1.022,"15min":1.0760000000000001},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.105,"5min":1.589,"15min":1.589},"last":1.0740000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.96799999999999997,"15min":1.02},"min":{"1min":0.67000000000000004,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.2709999999999999,"5min":1.2709999999999999,"15min":1.2709999999999999},"last":0.745},{"interface":"front","average":{"1min":1.026,"5min":1.0920000000000001,"15min":1.1140000000000001},"min":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.1779999999999999,"5min":1.337,"15min":1.337},"last":1.1180000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:29:18 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":1.046,"15min":1.0760000000000001},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.123,"5min":1.2649999999999999,"15min":1.2649999999999999},"last":1.04},{"interface":"front","average":{"1min":1.026,"5min":1.131,"15min":1.173},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.2270000000000001,"5min":1.6779999999999999,"15min":1.6779999999999999},"last":1.2090000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343864,"statfs":{"total":96636764160,"available":96608116736,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.84799999999999998,"15min":0.85299999999999998},"min":{"1min":0.61899999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":1.0920000000000001},{"interface":"front","average":{"1min":0.90800000000000003,"5min":0.89800000000000002,"15min":0.89700000000000002},"min":{"1min":0.67700000000000005,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":1.202,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90600000000000003,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.61399999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":1.1799999999999999,"5min":1.2030000000000001,"15min":1.2030000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.92600000000000005,"15min":0.91800000000000004},"min":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.165,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.80700000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.871,"15min":0.88400000000000001},"min":{"1min":0.64400000000000002,"5min":0.503,"15min":0.503},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.94499999999999995,"5min":0.86199999999999999,"15min":0.84899999999999998},"min":{"1min":0.71099999999999997,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.71299999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":0.95499999999999996,"15min":0.96399999999999997},"min":{"1min":0.72699999999999998,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.228,"5min":1.536,"15min":1.536},"last":1.0229999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.98199999999999998,"15min":0.98799999999999999},"min":{"1min":0.75600000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.254,"5min":1.409,"15min":1.409},"last":0.88}]},{"osd":4,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.95699999999999996,"15min":0.97499999999999998},"min":{"1min":0.62,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.365,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.97499999999999998,"15min":0.94399999999999995},"min":{"1min":0.71699999999999997,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.1850000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":1.0549999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92100000000000004,"15min":0.91600000000000004},"min":{"1min":0.66800000000000004,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.2110000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.038,"5min":1.1000000000000001,"15min":1.1299999999999999},"min":{"1min":0.73099999999999998,"5min":0.68400000000000005,"15min":0.68400000000000005},"max":{"1min":1.319,"5min":1.6439999999999999,"15min":1.6439999999999999},"last":0.92200000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:29:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1160000000000001,"15min":1.1220000000000001},"min":{"1min":0.79000000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.3240000000000001,"5min":1.522,"15min":1.522},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.048,"15min":1.0549999999999999},"min":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.27,"5min":1.337,"15min":1.337},"last":0.83899999999999997}]}]},{"osd":5,"up_from":14,"seq":60129542200,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32092,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339748,"statfs":{"total":96636764160,"available":96603901952,"internally_reserved":0,"allocated":991232,"data_stored":446310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.86299999999999999,"15min":0.86199999999999999},"min":{"1min":0.50800000000000001,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.01,"5min":1.411,"15min":1.411},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.749,"5min":0.83699999999999997,"15min":0.85099999999999998},"min":{"1min":0.53800000000000003,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.2909999999999999,"5min":1.675,"15min":1.675},"last":0.626}]},{"osd":1,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.872,"15min":0.88200000000000001},"min":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.117,"5min":1.369,"15min":1.369},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.79800000000000004,"5min":0.84399999999999997,"15min":0.82899999999999996},"min":{"1min":0.61699999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.0169999999999999,"5min":1.1830000000000001,"15min":1.1830000000000001},"last":1.054}]},{"osd":2,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.92400000000000004,"15min":0.93500000000000005},"min":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.258,"5min":1.625,"15min":1.625},"last":0.90900000000000003},{"interface":"front","average":{"1min":0.93400000000000005,"5min":0.96099999999999997,"15min":0.98299999999999998},"min":{"1min":0.69299999999999995,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.2090000000000001,"5min":1.552,"15min":1.552},"last":0.876}]},{"osd":3,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.874,"15min":0.84399999999999997},"min":{"1min":0.624,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1399999999999999,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.94999999999999996,"15min":1.0089999999999999},"min":{"1min":0.65800000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.1140000000000001,"5min":3.8460000000000001,"15min":3.8460000000000001},"last":0.71699999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":1.0009999999999999,"15min":1.0589999999999999},"min":{"1min":0.59399999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.2549999999999999,"5min":3.6869999999999998,"15min":3.6869999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":1.028,"5min":1.0569999999999999,"15min":1.0589999999999999},"min":{"1min":0.76100000000000001,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.26,"5min":1.601,"15min":1.601},"last":0.93899999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95599999999999996,"15min":0.97199999999999998},"min":{"1min":0.64900000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"max":{"1min":1.319,"5min":1.421,"15min":1.421},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0620000000000001,"15min":1.0740000000000001},"min":{"1min":0.73399999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3460000000000001,"5min":2.21,"15min":2.21},"last":0.96399999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:29:12 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.038,"15min":1.036},"min":{"1min":0.81599999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.2270000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.139,"15min":1.2170000000000001},"min":{"1min":0.89700000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.51,"5min":3.9830000000000001,"15min":3.9830000000000001},"last":1.0009999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542198,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27980,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343860,"statfs":{"total":96636764160,"available":96608112640,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.83499999999999996,"15min":0.83599999999999997},"min":{"1min":0.57099999999999995,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.157,"5min":1.411,"15min":1.411},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.91000000000000003,"5min":0.88400000000000001,"15min":0.86199999999999999},"min":{"1min":0.68100000000000005,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.1859999999999999,"5min":2.899,"15min":2.899},"last":0.65600000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.92400000000000004,"15min":0.91900000000000004},"min":{"1min":0.59999999999999998,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.1879999999999999,"5min":3.085,"15min":3.085},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.93100000000000005,"15min":0.92100000000000004},"min":{"1min":0.58299999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.155,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":1.129}]},{"osd":2,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.85599999999999998,"15min":0.84199999999999997},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.137,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.747},{"interface":"front","average":{"1min":0.96699999999999997,"5min":0.97699999999999998,"15min":0.96999999999999997},"min":{"1min":0.73499999999999999,"5min":0.498,"15min":0.498},"max":{"1min":1.3400000000000001,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":0.91700000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.95199999999999996,"15min":0.95699999999999996},"min":{"1min":0.65300000000000002,"5min":0.58799999999999997,"15min":0.58799999999999997},"max":{"1min":1.1799999999999999,"5min":1.472,"15min":1.472},"last":1.0680000000000001},{"interface":"front","average":{"1min":0.91400000000000003,"5min":0.94899999999999995,"15min":0.96099999999999997},"min":{"1min":0.64400000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"max":{"1min":1.25,"5min":1.577,"15min":1.577},"last":0.621}]},{"osd":5,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.97799999999999998,"15min":0.97799999999999998},"min":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.264,"5min":2.996,"15min":2.996},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.96099999999999997,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":1.294,"5min":2.8860000000000001,"15min":2.8860000000000001},"last":0.68799999999999994}]},{"osd":6,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.93899999999999995,"15min":0.93999999999999995},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.1180000000000001,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":0.873},{"interface":"front","average":{"1min":1.028,"5min":1.133,"15min":1.1120000000000001},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.4450000000000001,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.038}]},{"osd":7,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.135,"15min":1.143},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.476,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.1240000000000001,"15min":1.1200000000000001},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3680000000000001,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":1.0029999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542199,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27964,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343876,"statfs":{"total":96636764160,"available":96608129024,"internally_reserved":0,"allocated":962560,"data_stored":432586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.61899999999999999,"15min":0.63200000000000001},"min":{"1min":0.47499999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.872,"5min":1.04,"15min":1.04},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.65800000000000003,"15min":0.65700000000000003},"min":{"1min":0.47599999999999998,"5min":0.433,"15min":0.433},"max":{"1min":0.90100000000000002,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.55900000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.70699999999999996,"5min":0.65000000000000002,"15min":0.628},"min":{"1min":0.41999999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.77700000000000002,"15min":0.78500000000000003},"min":{"1min":0.501,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0249999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.45400000000000001,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.0649999999999999,"5min":1.27,"15min":1.27},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.76200000000000001,"15min":0.754},"min":{"1min":0.52000000000000002,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":1.016,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.73899999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.90000000000000002,"15min":0.89700000000000002},"min":{"1min":0.65900000000000003,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.204,"5min":1.8460000000000001,"15min":1.8460000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":0.94599999999999995,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.73499999999999999,"5min":0.625,"15min":0.625},"max":{"1min":1.153,"5min":1.782,"15min":1.782},"last":1.0149999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.96599999999999997,"15min":0.97799999999999998},"min":{"1min":0.66100000000000003,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.194,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.86699999999999999},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.073,"15min":1.0960000000000001},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.214,"5min":1.8140000000000001,"15min":1.8140000000000001},"last":0.91500000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.093,"15min":1.1279999999999999},"min":{"1min":0.75,"5min":0.74199999999999999,"15min":0.74199999999999999},"max":{"1min":1.2390000000000001,"5min":1.885,"15min":1.885},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.032,"5min":1.024,"15min":1.0549999999999999},"min":{"1min":0.70499999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.3360000000000001,"5min":1.95,"15min":1.95},"last":0.83899999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:29:16 2024","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.0740000000000001,"15min":1.075},"min":{"1min":0.79900000000000004,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.2569999999999999,"5min":1.889,"15min":1.889},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.0580000000000001,"15min":1.0609999999999999},"min":{"1min":0.76600000000000001,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.5289999999999999,"5min":1.756,"15min":1.756},"last":0.93400000000000005}]}]},{"osd":2,"up_from":14,"seq":60129542200,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32072,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339768,"statfs":{"total":96636764160,"available":96603922432,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.59099999999999997,"15min":0.58499999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78400000000000003,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.70399999999999996,"15min":0.69799999999999995},"min":{"1min":0.50900000000000001,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.40799999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.69299999999999995,"15min":0.70599999999999996},"min":{"1min":0.47499999999999998,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.94799999999999995,"5min":0.997,"15min":0.997},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.68600000000000005,"15min":0.69099999999999995},"min":{"1min":0.52600000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.35699999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.47499999999999998,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.054,"5min":1.073,"15min":1.073},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.80700000000000005,"15min":0.79300000000000004},"min":{"1min":0.45500000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.1359999999999999,"5min":1.2070000000000001,"15min":1.2070000000000001},"last":0.68799999999999994}]},{"osd":4,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.94999999999999996,"15min":0.95599999999999996},"min":{"1min":0.77300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"max":{"1min":1.147,"5min":1.5149999999999999,"15min":1.5149999999999999},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.93899999999999995,"5min":0.94499999999999995,"15min":0.97999999999999998},"min":{"1min":0.78500000000000003,"5min":0.67500000000000004,"15min":0.67500000000000004},"max":{"1min":1.1200000000000001,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.67100000000000004}]},{"osd":5,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.95999999999999996,"15min":0.97899999999999998},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.149,"5min":1.4470000000000001,"15min":1.4470000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95999999999999996},"min":{"1min":0.72299999999999998,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.246,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.65200000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.97599999999999998,"15min":1.03},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1160000000000001,"5min":2.024,"15min":2.024},"last":0.70999999999999996},{"interface":"front","average":{"1min":1.018,"5min":1.05,"15min":1.0640000000000001},"min":{"1min":0.78700000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.3109999999999999,"5min":1.6359999999999999,"15min":1.6359999999999999},"last":0.82599999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:29:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0629999999999999,"15min":1.0800000000000001},"min":{"1min":0.93300000000000005,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.2310000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.999,"5min":1.0409999999999999,"15min":1.0649999999999999},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.2010000000000001,"5min":1.55,"15min":1.55},"last":0.752}]}]},{"osd":1,"up_from":14,"seq":60129542199,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":974848,"data_stored":439181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.621,"15min":0.60599999999999998},"min":{"1min":0.371,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.86699999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.498},{"interface":"front","average":{"1min":0.78600000000000003,"5min":0.75900000000000001,"15min":0.73399999999999999},"min":{"1min":0.55500000000000005,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":1.095,"5min":1.141,"15min":1.141},"last":0.89500000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.65900000000000003,"15min":0.622},"min":{"1min":0.47299999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":1.036,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.70399999999999996,"15min":0.70199999999999996},"min":{"1min":0.45200000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.90200000000000002,"5min":1.458,"15min":1.458},"last":0.73899999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.754,"5min":0.74199999999999999,"15min":0.69599999999999995},"min":{"1min":0.496,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.98499999999999999,"5min":1.236,"15min":1.236},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.64900000000000002,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.93300000000000005,"5min":1.403,"15min":1.403},"last":0.85699999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.93200000000000005,"15min":0.90900000000000003},"min":{"1min":0.66500000000000004,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.1379999999999999,"5min":1.546,"15min":1.546},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.90900000000000003,"5min":0.92300000000000004,"15min":0.93100000000000005},"min":{"1min":0.72499999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.0680000000000001,"5min":2.2040000000000002,"15min":2.2040000000000002},"last":1.046}]},{"osd":5,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.0169999999999999,"5min":0.96999999999999997,"15min":0.95199999999999996},"min":{"1min":0.879,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.3109999999999999,"5min":1.389,"15min":1.389},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.95699999999999996,"15min":0.97699999999999998},"min":{"1min":0.69599999999999995,"5min":0.60599999999999998,"15min":0.60599999999999998},"max":{"1min":1.1140000000000001,"5min":2.3029999999999999,"15min":2.3029999999999999},"last":0.98399999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":1.0109999999999999,"15min":1.0089999999999999},"min":{"1min":0.73099999999999998,"5min":0.626,"15min":0.626},"max":{"1min":1.4810000000000001,"5min":1.613,"15min":1.613},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.054,"5min":1.0229999999999999,"15min":1.0149999999999999},"min":{"1min":0.83999999999999997,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.4199999999999999,"5min":1.698,"15min":1.698},"last":1.304}]},{"osd":7,"last update":"Fri Sep 13 16:29:13 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":1.004,"15min":1.0089999999999999},"min":{"1min":0.72499999999999998,"5min":0.62,"15min":0.62},"max":{"1min":1.4430000000000001,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.0640000000000001,"15min":1.0449999999999999},"min":{"1min":0.80000000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.546,"5min":1.7689999999999999,"15min":1.7689999999999999},"last":1.0600000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542199,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32636,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339204,"statfs":{"total":96636764160,"available":96603344896,"internally_reserved":0,"allocated":1556480,"data_stored":1022954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.67500000000000004},"min":{"1min":0.51800000000000002,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.93500000000000005,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.66500000000000004,"15min":0.65700000000000003},"min":{"1min":0.47499999999999998,"5min":0.377,"15min":0.377},"max":{"1min":1.0169999999999999,"5min":1.157,"15min":1.157},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.73399999999999999,"5min":0.71999999999999997,"15min":0.72299999999999998},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0609999999999999,"5min":1.202,"15min":1.202},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.749,"5min":0.70999999999999996,"15min":0.68700000000000006},"min":{"1min":0.58199999999999996,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.1160000000000001,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.70099999999999996,"15min":0.65500000000000003},"min":{"1min":0.437,"5min":0.312,"15min":0.312},"max":{"1min":1.0309999999999999,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.90100000000000002,"5min":0.85599999999999998,"15min":0.82899999999999996},"min":{"1min":0.54900000000000004,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.284,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.80500000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.95799999999999996,"15min":0.96099999999999997},"min":{"1min":0.66100000000000003,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.28,"5min":1.486,"15min":1.486},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.95099999999999996,"5min":0.97699999999999998,"15min":0.96099999999999997},"min":{"1min":0.69999999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.3109999999999999,"5min":1.704,"15min":1.704},"last":0.82899999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0109999999999999,"15min":0.97999999999999998},"min":{"1min":0.748,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.446,"5min":1.575,"15min":1.575},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.98299999999999998,"5min":0.99099999999999999,"15min":0.998},"min":{"1min":0.76700000000000002,"5min":0.443,"15min":0.443},"max":{"1min":1.3320000000000001,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.52700000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98799999999999999,"5min":1.002,"15min":0.97999999999999998},"min":{"1min":0.82499999999999996,"5min":0.628,"15min":0.628},"max":{"1min":1.2709999999999999,"5min":1.944,"15min":1.944},"last":0.69499999999999995},{"interface":"front","average":{"1min":1.135,"5min":1.079,"15min":1.0549999999999999},"min":{"1min":0.82399999999999995,"5min":0.627,"15min":0.627},"max":{"1min":1.5,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.92500000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:29:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0589999999999999,"5min":1.0660000000000001,"15min":1.05},"min":{"1min":0.79900000000000004,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.343,"5min":1.659,"15min":1.659},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.1120000000000001,"5min":1.093,"15min":1.048},"min":{"1min":0.90300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.51,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.76700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:29:34.650 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:29:34.650 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T16:29:34.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.764+0000 7f2a494a5700 1 Processor -- start 2024-09-13T16:29:34.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.764+0000 7f2a494a5700 1 -- start start 2024-09-13T16:29:34.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2a440add60 0x7f2a440ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a4415dbe0 0x7f2a440cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a440cd4d0 con 0x7f2a440b3260 2024-09-13T16:29:34.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a44071be0 con 0x7f2a440add60 2024-09-13T16:29:34.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a44159af0 con 0x7f2a4415dbe0 2024-09-13T16:29:34.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a4415dbe0 0x7f2a440cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a43fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2a440add60 0x7f2a440ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a4415dbe0 0x7f2a440cd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:43666/0 (socket says 172.21.15.119:43666) 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 -- 172.21.15.119:0/2973073836 learned_addr learned my addr 172.21.15.119:0/2973073836 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 -- 172.21.15.119:0/2973073836 >> v2:172.21.15.145:3300/0 conn(0x7f2a440add60 msgr2=0x7f2a440ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 --2- 172.21.15.119:0/2973073836 >> v2:172.21.15.145:3300/0 conn(0x7f2a440add60 0x7f2a440ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 -- 172.21.15.119:0/2973073836 >> v1:172.21.15.145:6789/0 conn(0x7f2a440b3260 legacy=0x7f2a4406fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 -- 172.21.15.119:0/2973073836 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4415e130 con 0x7f2a4415dbe0 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a43fff700 1 --2- 172.21.15.119:0/2973073836 >> v2:172.21.15.145:3300/0 conn(0x7f2a440add60 0x7f2a440ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a437fe700 1 --2- 172.21.15.119:0/2973073836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a4415dbe0 0x7f2a440cd960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400bbb0 tx=0x7f2a3400bec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=780830c263536317 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a42ffd700 1 -- 172.21.15.119:0/2973073836 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3400dd30 con 0x7f2a4415dbe0 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a42ffd700 1 -- 172.21.15.119:0/2973073836 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3400de90 con 0x7f2a4415dbe0 2024-09-13T16:29:34.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a42ffd700 1 -- 172.21.15.119:0/2973073836 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34010690 con 0x7f2a4415dbe0 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- 172.21.15.119:0/2973073836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a4415dbe0 msgr2=0x7f2a440cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 --2- 172.21.15.119:0/2973073836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a4415dbe0 0x7f2a440cd960 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400bbb0 tx=0x7f2a3400bec0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- 172.21.15.119:0/2973073836 shutdown_connections 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 --2- 172.21.15.119:0/2973073836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a4415dbe0 0x7f2a440cd960 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 --2- 172.21.15.119:0/2973073836 >> v2:172.21.15.145:3300/0 conn(0x7f2a440add60 0x7f2a440ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- 172.21.15.119:0/2973073836 >> 172.21.15.119:0/2973073836 conn(0x7f2a440b0980 msgr2=0x7f2a440b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- 172.21.15.119:0/2973073836 shutdown_connections 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.768+0000 7f2a494a5700 1 -- 172.21.15.119:0/2973073836 wait complete. 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 Processor -- start 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 -- start start 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a440add60 0x7f2a441f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a440b3260 0x7f2a440c41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a4415dbe0 0x7f2a440c8460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a440c52a0 con 0x7f2a440b3260 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a440c6010 con 0x7f2a4415dbe0 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a441f2be0 con 0x7f2a440add60 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a440b3260 0x7f2a440c41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a440b3260 0x7f2a440c41e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46000/0 (socket says 172.21.15.119:46000) 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 -- 172.21.15.119:0/915522853 learned_addr learned my addr 172.21.15.119:0/915522853 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a48ca4700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a4415dbe0 0x7f2a440c8460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a43fff700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a440add60 0x7f2a441f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 -- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a440add60 msgr2=0x7f2a441f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a440add60 0x7f2a441f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 -- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a4415dbe0 msgr2=0x7f2a440c8460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a4415dbe0 0x7f2a440c8460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 -- 172.21.15.119:0/915522853 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a380094b0 con 0x7f2a440b3260 2024-09-13T16:29:34.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a437fe700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a440b3260 0x7f2a440c41e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2a34015000 tx=0x7f2a34004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a40ff9700 1 -- 172.21.15.119:0/915522853 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3401d070 con 0x7f2a440b3260 2024-09-13T16:29:34.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3400b810 con 0x7f2a440b3260 2024-09-13T16:29:34.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a440c4fc0 con 0x7f2a440b3260 2024-09-13T16:29:34.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a40ff9700 1 -- 172.21.15.119:0/915522853 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a34010620 con 0x7f2a440b3260 2024-09-13T16:29:34.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.772+0000 7f2a40ff9700 1 -- 172.21.15.119:0/915522853 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a34010c70 con 0x7f2a440b3260 2024-09-13T16:29:34.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.776+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a440cd160 con 0x7f2a440b3260 2024-09-13T16:29:34.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.776+0000 7f2a40ff9700 1 -- 172.21.15.119:0/915522853 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2a340036c0 con 0x7f2a440b3260 2024-09-13T16:29:34.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.776+0000 7f2a40ff9700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a30066730 0x7f2a30068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:34.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.776+0000 7f2a40ff9700 1 -- 172.21.15.119:0/915522853 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2a34085670 con 0x7f2a440b3260 2024-09-13T16:29:34.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.776+0000 7f2a43fff700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a30066730 0x7f2a30068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:34.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.776+0000 7f2a43fff700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a30066730 0x7f2a30068bf0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009d10 tx=0x7f2a38008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:34.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:34.784+0000 7f2a40ff9700 1 -- 172.21.15.119:0/915522853 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2a34052820 con 0x7f2a440b3260 2024-09-13T16:29:35.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.016+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2a440b23e0 con 0x7f2a440b3260 2024-09-13T16:29:35.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a40ff9700 1 -- 172.21.15.119:0/915522853 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+0 (secure 0 0 0) 0x7f2a34055e40 con 0x7f2a440b3260 2024-09-13T16:29:35.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a30066730 msgr2=0x7f2a30068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a30066730 0x7f2a30068bf0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f2a38009d10 tx=0x7f2a38008040 comp rx=0 tx=0).stop 2024-09-13T16:29:35.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a440b3260 msgr2=0x7f2a440c41e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a440b3260 0x7f2a440c41e0 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f2a34015000 tx=0x7f2a34004770 comp rx=0 tx=0).stop 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 shutdown_connections 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a30066730 0x7f2a30068bf0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a4415dbe0 0x7f2a440c8460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a440b3260 0x7f2a440c41e0 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 --2- 172.21.15.119:0/915522853 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a440add60 0x7f2a441f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 >> 172.21.15.119:0/915522853 conn(0x7f2a440b0980 msgr2=0x7f2a4406fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 shutdown_connections 2024-09-13T16:29:35.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.024+0000 7f2a494a5700 1 -- 172.21.15.119:0/915522853 wait complete. 2024-09-13T16:29:35.032 INFO:teuthology.orchestra.run.smithi119.stderr:updated caps for client.0 2024-09-13T16:29:35.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 Processor -- start 2024-09-13T16:29:35.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 -- start start 2024-09-13T16:29:35.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbc840cd8d0 0x7fbc8406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbc840b17f0 0x7fbc840cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc840b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc840b2bb0 con 0x7fbc840cd8d0 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc840705d0 con 0x7fbc840b17f0 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc888ed700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8406d0d0 con 0x7fbc840b2200 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbc840cd8d0 0x7fbc8406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbc840cd8d0 0x7fbc8406cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46006/0 (socket says 172.21.15.119:46006) 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 -- 172.21.15.119:0/683642585 learned_addr learned my addr 172.21.15.119:0/683642585 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc8359e700 1 --2- 172.21.15.119:0/683642585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc840b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 -- 172.21.15.119:0/683642585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 msgr2=0x7fbc840b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc8259c700 1 --2- 172.21.15.119:0/683642585 >> v2:172.21.15.145:3300/0 conn(0x7fbc840b17f0 0x7fbc840cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 --2- 172.21.15.119:0/683642585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc840b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 -- 172.21.15.119:0/683642585 >> v2:172.21.15.145:3300/0 conn(0x7fbc840b17f0 msgr2=0x7fbc840cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 --2- 172.21.15.119:0/683642585 >> v2:172.21.15.145:3300/0 conn(0x7fbc840b17f0 0x7fbc840cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc82d9d700 1 -- 172.21.15.119:0/683642585 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc8406fca0 con 0x7fbc840cd8d0 2024-09-13T16:29:35.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc8359e700 1 --2- 172.21.15.119:0/683642585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc840b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:35.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.152+0000 7fbc8259c700 1 --2- 172.21.15.119:0/683642585 >> v2:172.21.15.145:3300/0 conn(0x7fbc840b17f0 0x7fbc840cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:35.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc82d9d700 1 --2- 172.21.15.119:0/683642585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc8406cb90 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fbc840ae2c0 tx=0x7fbc7c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a3cbee50bd02 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc81d9b700 1 -- 172.21.15.119:0/683642585 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00e070 con 0x7fbc840cd8d0 2024-09-13T16:29:35.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc81d9b700 1 -- 172.21.15.119:0/683642585 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7c00b9c0 con 0x7fbc840cd8d0 2024-09-13T16:29:35.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc81d9b700 1 -- 172.21.15.119:0/683642585 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc7c00bd70 con 0x7fbc840cd8d0 2024-09-13T16:29:35.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 -- 172.21.15.119:0/683642585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 msgr2=0x7fbc8406cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 --2- 172.21.15.119:0/683642585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc8406cb90 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7fbc840ae2c0 tx=0x7fbc7c00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:35.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 -- 172.21.15.119:0/683642585 shutdown_connections 2024-09-13T16:29:35.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 --2- 172.21.15.119:0/683642585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc840b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 --2- 172.21.15.119:0/683642585 >> v2:172.21.15.145:3300/0 conn(0x7fbc840b17f0 0x7fbc840cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 --2- 172.21.15.119:0/683642585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc8406cb90 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 -- 172.21.15.119:0/683642585 >> 172.21.15.119:0/683642585 conn(0x7fbc8406a350 msgr2=0x7fbc8406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 -- 172.21.15.119:0/683642585 shutdown_connections 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.156+0000 7fbc888ed700 1 -- 172.21.15.119:0/683642585 wait complete. 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 Processor -- start 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 -- start start 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc840b17f0 0x7fbc84163250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc84163790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc841f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc841f74f0 con 0x7fbc840cd8d0 2024-09-13T16:29:35.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8414e540 con 0x7fbc840b17f0 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc8414e820 con 0x7fbc840b2200 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc841f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc841f6770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46016/0 (socket says 172.21.15.119:46016) 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 -- 172.21.15.119:0/2467710336 learned_addr learned my addr 172.21.15.119:0/2467710336 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8259c700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc84163790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc82d9d700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc840b17f0 0x7fbc84163250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 -- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 msgr2=0x7fbc84163790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc84163790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 -- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc840b17f0 msgr2=0x7fbc84163250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc840b17f0 0x7fbc84163250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 -- 172.21.15.119:0/2467710336 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc7c009b30 con 0x7fbc840cd8d0 2024-09-13T16:29:35.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc82d9d700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc840b17f0 0x7fbc84163250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:35.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc8359e700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc841f6770 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400c7c0 tx=0x7fbc7400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc7b7fe700 1 -- 172.21.15.119:0/2467710336 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74011800 con 0x7fbc840cd8d0 2024-09-13T16:29:35.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc7b7fe700 1 -- 172.21.15.119:0/2467710336 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc74011960 con 0x7fbc840cd8d0 2024-09-13T16:29:35.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc8414eb00 con 0x7fbc840cd8d0 2024-09-13T16:29:35.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc7b7fe700 1 -- 172.21.15.119:0/2467710336 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc74011d10 con 0x7fbc840cd8d0 2024-09-13T16:29:35.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc8414f020 con 0x7fbc840cd8d0 2024-09-13T16:29:35.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.160+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc8414ed40 con 0x7fbc840cd8d0 2024-09-13T16:29:35.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.164+0000 7fbc7b7fe700 1 -- 172.21.15.119:0/2467710336 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbc74021350 con 0x7fbc840cd8d0 2024-09-13T16:29:35.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.164+0000 7fbc7b7fe700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc64066760 0x7fbc64068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.164+0000 7fbc7b7fe700 1 -- 172.21.15.119:0/2467710336 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbc74084230 con 0x7fbc840cd8d0 2024-09-13T16:29:35.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.164+0000 7fbc82d9d700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc64066760 0x7fbc64068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.164+0000 7fbc82d9d700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc64066760 0x7fbc64068c20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c000f80 tx=0x7fbc7c009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.172+0000 7fbc7b7fe700 1 -- 172.21.15.119:0/2467710336 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbc8414ed40 con 0x7fbc840cd8d0 2024-09-13T16:29:35.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.376+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc840cdcb0 con 0x7fbc840cd8d0 2024-09-13T16:29:35.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.376+0000 7fbc7b7fe700 1 -- 172.21.15.119:0/2467710336 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fbc74051460 con 0x7fbc840cd8d0 2024-09-13T16:29:35.383 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:35.383 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:29:33.249934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7101":{"gid":7101,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/491709455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":491709455},{"type":"v1","addr":"172.21.15.119:6835","nonce":491709455}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc64066760 msgr2=0x7fbc64068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc64066760 0x7fbc64068c20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fbc7c000f80 tx=0x7fbc7c009000 comp rx=0 tx=0).stop 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 msgr2=0x7fbc841f6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc841f6770 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7fbc7400c7c0 tx=0x7fbc7400cb80 comp rx=0 tx=0).stop 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 shutdown_connections 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbc64066760 0x7fbc64068c20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbc840cd8d0 0x7fbc841f6770 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbc840b2200 0x7fbc84163790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 --2- 172.21.15.119:0/2467710336 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbc840b17f0 0x7fbc84163250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 >> 172.21.15.119:0/2467710336 conn(0x7fbc8406a350 msgr2=0x7fbc84161b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 shutdown_connections 2024-09-13T16:29:35.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.380+0000 7fbc888ed700 1 -- 172.21.15.119:0/2467710336 wait complete. 2024-09-13T16:29:35.387 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 42 2024-09-13T16:29:35.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 42, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:29:30.227373+0000', 'modified': '2024-09-13T16:29:33.249934+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7101}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7101': {'gid': 7101, 'name': 'c', 'rank': 0, 'incarnation': 38, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6835/491709455', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 491709455}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 491709455}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:29:35.801 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:29:35.801 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:29:35.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f182cdd1700 1 Processor -- start 2024-09-13T16:29:35.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f182cdd1700 1 -- start start 2024-09-13T16:29:35.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f182cdd1700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f18280cd8d0 0x7f182806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f182cdd1700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18280705d0 con 0x7f18280b17f0 2024-09-13T16:29:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f182cdd1700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f182806d0d0 con 0x7f18280b2200 2024-09-13T16:29:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f182cdd1700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18280cbf10 con 0x7f18280cd8d0 2024-09-13T16:29:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f1826ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f18280b17f0 0x7f18280cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50650/0 (socket says 172.21.15.119:50650) 2024-09-13T16:29:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f1826ffd700 1 -- 172.21.15.119:0/1510236913 learned_addr learned my addr 172.21.15.119:0/1510236913 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:35.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18277fe700 1 --2- 172.21.15.119:0/1510236913 >> v2:172.21.15.145:3300/0 conn(0x7f18280cd8d0 0x7f182806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18267fc700 1 -- 172.21.15.119:0/1510236913 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 940073583 0 0) 0x7f18280705d0 con 0x7f18280b17f0 2024-09-13T16:29:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18267fc700 1 -- 172.21.15.119:0/1510236913 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18140035d0 con 0x7f18280b17f0 2024-09-13T16:29:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18277fe700 1 -- 172.21.15.119:0/1510236913 >> v1:172.21.15.145:6789/0 conn(0x7f18280b2200 legacy=0x7f1828070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18277fe700 1 -- 172.21.15.119:0/1510236913 >> v1:172.21.15.119:6789/0 conn(0x7f18280b17f0 legacy=0x7f18280cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18277fe700 1 -- 172.21.15.119:0/1510236913 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182806fca0 con 0x7f18280cd8d0 2024-09-13T16:29:35.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18277fe700 1 --2- 172.21.15.119:0/1510236913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280cd8d0 0x7f182806cb90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1810009850 tx=0x7f1810009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73e15223cc2be984 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18267fc700 1 -- 172.21.15.119:0/1510236913 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810011830 con 0x7f18280cd8d0 2024-09-13T16:29:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18267fc700 1 -- 172.21.15.119:0/1510236913 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1810011990 con 0x7f18280cd8d0 2024-09-13T16:29:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.924+0000 7f18267fc700 1 -- 172.21.15.119:0/1510236913 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810011d00 con 0x7f18280cd8d0 2024-09-13T16:29:35.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 -- 172.21.15.119:0/1510236913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280cd8d0 msgr2=0x7f182806cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 --2- 172.21.15.119:0/1510236913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280cd8d0 0x7f182806cb90 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f1810009850 tx=0x7f1810009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 -- 172.21.15.119:0/1510236913 shutdown_connections 2024-09-13T16:29:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 --2- 172.21.15.119:0/1510236913 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280cd8d0 0x7f182806cb90 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 -- 172.21.15.119:0/1510236913 >> 172.21.15.119:0/1510236913 conn(0x7f182806a350 msgr2=0x7f182806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 -- 172.21.15.119:0/1510236913 shutdown_connections 2024-09-13T16:29:35.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 -- 172.21.15.119:0/1510236913 wait complete. 2024-09-13T16:29:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 Processor -- start 2024-09-13T16:29:35.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.928+0000 7f182cdd1700 1 -- start start 2024-09-13T16:29:35.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280b17f0 0x7f182814cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b2200 0x7f18281f9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18280cd8d0 0x7f18281fa2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182814e410 con 0x7f18280cd8d0 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182814e6f0 con 0x7f18280b17f0 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182814e9d0 con 0x7f18280b2200 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f18277fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280b17f0 0x7f182814cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1827fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18280cd8d0 0x7f18281fa2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f18277fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280b17f0 0x7f182814cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:60918/0 (socket says 172.21.15.119:60918) 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f18277fe700 1 -- 172.21.15.119:0/2679032874 learned_addr learned my addr 172.21.15.119:0/2679032874 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:35.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1826ffd700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b2200 0x7f18281f9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1827fff700 1 -- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b2200 msgr2=0x7f18281f9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1827fff700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b2200 0x7f18281f9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1827fff700 1 -- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280b17f0 msgr2=0x7f182814cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1827fff700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280b17f0 0x7f182814cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1827fff700 1 -- 172.21.15.119:0/2679032874 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18100094b0 con 0x7f18280cd8d0 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f1827fff700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18280cd8d0 0x7f18281fa2f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f181800c370 tx=0x7f181800c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f181ffff700 1 -- 172.21.15.119:0/2679032874 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180052d0 con 0x7f18280cd8d0 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f182814ecb0 con 0x7f18280cd8d0 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f181ffff700 1 -- 172.21.15.119:0/2679032874 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1818005430 con 0x7f18280cd8d0 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f181ffff700 1 -- 172.21.15.119:0/2679032874 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18180055a0 con 0x7f18280cd8d0 2024-09-13T16:29:35.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.932+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1828153830 con 0x7f18280cd8d0 2024-09-13T16:29:35.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.936+0000 7f181ffff700 1 -- 172.21.15.119:0/2679032874 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f18180036a0 con 0x7f18280cd8d0 2024-09-13T16:29:35.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.936+0000 7f181ffff700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1808066730 0x7f1808068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.936+0000 7f181ffff700 1 -- 172.21.15.119:0/2679032874 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f18180832e0 con 0x7f18280cd8d0 2024-09-13T16:29:35.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.936+0000 7f18277fe700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1808066730 0x7f1808068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18277fe700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1808066730 0x7f1808068bf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1810014040 tx=0x7f181001c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 Processor -- start 2024-09-13T16:29:35.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 -- start start 2024-09-13T16:29:35.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x18bc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1884890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x1884dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x17e3d80 con 0x18bcb50 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x17e1510 con 0x189ae90 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x17dce00 con 0x18bc230 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x1884dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18277fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x18bc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18277fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x18bc610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:43732/0 (socket says 172.21.15.119:43732) 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x1884dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46036/0 (socket says 172.21.15.119:46036) 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f18277fe700 1 -- 172.21.15.119:0/2749521828 learned_addr learned my addr 172.21.15.119:0/2749521828 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1826ffd700 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1884890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 -- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 msgr2=0x18bc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x18bc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 -- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 msgr2=0x1884890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1884890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 -- 172.21.15.119:0/2749521828 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x164f960 con 0x18bcb50 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1826ffd700 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1884890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f1827fff700 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x1884dd0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f18180817b0 tx=0x7f1818000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65a822e8ec7902df server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f181d7fa700 1 -- 172.21.15.119:0/2749521828 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818058010 con 0x18bcb50 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f181d7fa700 1 -- 172.21.15.119:0/2749521828 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181805c5d0 con 0x18bcb50 2024-09-13T16:29:35.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.940+0000 7f181d7fa700 1 -- 172.21.15.119:0/2749521828 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818052580 con 0x18bcb50 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- 172.21.15.119:0/2749521828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 msgr2=0x1884dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x1884dd0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f18180817b0 tx=0x7f1818000ed0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- 172.21.15.119:0/2749521828 shutdown_connections 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x1884dd0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1884890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 --2- 172.21.15.119:0/2749521828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x18bc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- 172.21.15.119:0/2749521828 >> 172.21.15.119:0/2749521828 conn(0x16044d0 msgr2=0x189de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- 172.21.15.119:0/2749521828 shutdown_connections 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- 172.21.15.119:0/2749521828 wait complete. 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- start start 2024-09-13T16:29:35.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 Processor -- start 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1871750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc230 0x1876c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bcb50 0x18c4f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x179a730 con 0x18bc230 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x17a05b0 con 0x189ae90 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1688aa0 con 0x18bcb50 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18312e4740 10 client.0 _ll_register_callbacks cb 0x17d6610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc230 0x1876c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc230 0x1876c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46042/0 (socket says 172.21.15.119:46042) 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 -- 172.21.15.119:0/466017037 learned_addr learned my addr 172.21.15.119:0/466017037 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1827fff700 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bcb50 0x18c4f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18277fe700 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1871750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 -- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bcb50 msgr2=0x18c4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bcb50 0x18c4f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 -- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 msgr2=0x1871750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1871750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 -- 172.21.15.119:0/466017037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1828153830 con 0x18bc230 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f1826ffd700 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc230 0x1876c30 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1820007b10 tx=0x7f1820007b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb07540f50260e9c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18067fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.944+0000 7f18067fc700 1 -- 172.21.15.119:0/466017037 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18200116e0 con 0x18bc230 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 1 -- 172.21.15.119:0/466017037 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1820011840 con 0x18bc230 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x16111e0 con 0x18bc230 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 1 -- 172.21.15.119:0/466017037 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1820011a50 con 0x18bc230 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 10 client.7233 fetch_fsmap learned FSMap version 42 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x17a3220 con 0x18bc230 2024-09-13T16:29:35.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 1 -- 172.21.15.119:0/466017037 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x16111e0 con 0x18bc230 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 1 -- 172.21.15.119:0/466017037 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1820011bb0 con 0x18bc230 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 10 client.7233 fetch_fsmap finished waiting for FSMap version 42 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 10 client.7233 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 20 client.7233 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x18c66e0 0x18c37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 4 client.7233 mds_command: new command op to 7101 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 --> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x165b460 con 0x18c66e0 2024-09-13T16:29:35.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18277fe700 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x18c66e0 0x18c37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 10 client.7233 ms_handle_connect on v2:172.21.15.119:6834/491709455 2024-09-13T16:29:35.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18277fe700 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x18c66e0 0x18c37a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fce7f4034cee2911 server_cookie=2d8c63be81a00296 in_seq=0 out_seq=0 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 1 -- 172.21.15.119:0/466017037 <== mds.0 v2:172.21.15.119:6834/491709455 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x165b460 con 0x18c66e0 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 10 client.7233 handle_command_reply: tid=0 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 client.7233 shutdown 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc230 msgr2=0x1876c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc230 0x1876c30 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f1820007b10 tx=0x7f1820007b40 comp rx=0 tx=0).stop 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 shutdown_connections 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bcb50 0x18c4f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x18c66e0 0x18c37a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc230 0x1876c30 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 --2- 172.21.15.119:0/466017037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1871750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 >> 172.21.15.119:0/466017037 conn(0x16044d0 msgr2=0x1897470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18067fc700 0 client.7233 ms_handle_reset on v2:172.21.15.119:6834/491709455 2024-09-13T16:29:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 shutdown_connections 2024-09-13T16:29:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 1 -- 172.21.15.119:0/466017037 wait complete. 2024-09-13T16:29:35.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.948+0000 7f18312e4740 20 client.7233 trim_cache size 0 max 16384 2024-09-13T16:29:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 Processor -- start 2024-09-13T16:29:35.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 -- start start 2024-09-13T16:29:35.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1874000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x1874540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x18bb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x180bec0 con 0x18bcb50 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x18c61b0 con 0x189ae90 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1877280 con 0x18bc230 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1827fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x18bb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1826ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x1874540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18277fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1874000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18277fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1874000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:60952/0 (socket says 172.21.15.119:60952) 2024-09-13T16:29:35.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18277fe700 1 -- 172.21.15.119:0/3891852864 learned_addr learned my addr 172.21.15.119:0/3891852864 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1827fff700 1 -- 172.21.15.119:0/3891852864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 msgr2=0x1874540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1827fff700 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x1874540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1827fff700 1 -- 172.21.15.119:0/3891852864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 msgr2=0x1874000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1827fff700 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1874000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1827fff700 1 -- 172.21.15.119:0/3891852864 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1789450 con 0x18bcb50 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f18277fe700 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1874000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f1827fff700 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x18bb1f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1818081730 tx=0x7f181800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8102fa5050f59b00 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f181dffb700 1 -- 172.21.15.119:0/3891852864 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818050520 con 0x18bcb50 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f181dffb700 1 -- 172.21.15.119:0/3891852864 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1818058010 con 0x18bcb50 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.972+0000 7f181dffb700 1 -- 172.21.15.119:0/3891852864 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1818052580 con 0x18bcb50 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 -- 172.21.15.119:0/3891852864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 msgr2=0x18bb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x18bb1f0 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f1818081730 tx=0x7f181800cf90 comp rx=0 tx=0).stop 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 -- 172.21.15.119:0/3891852864 shutdown_connections 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bcb50 0x18bb1f0 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x1874540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 --2- 172.21.15.119:0/3891852864 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x189ae90 0x1874000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 -- 172.21.15.119:0/3891852864 >> 172.21.15.119:0/3891852864 conn(0x16044d0 msgr2=0x1870700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 -- 172.21.15.119:0/3891852864 shutdown_connections 2024-09-13T16:29:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 -- 172.21.15.119:0/3891852864 wait complete. 2024-09-13T16:29:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 -- start start 2024-09-13T16:29:35.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 Processor -- start 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x189ae90 0x18bb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x191c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bcb50 0x191d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.976+0000 7f18312e4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x18c2190 con 0x189ae90 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x18c1ef0 con 0x18bcb50 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x18c1c50 con 0x18bc230 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x189ae90 0x18bb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 10 client.0 _ll_register_callbacks cb 0x178ece0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:35.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18277fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x189ae90 0x18bb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46062/0 (socket says 172.21.15.119:46062) 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18277fe700 1 -- 172.21.15.119:0/367475186 learned_addr learned my addr 172.21.15.119:0/367475186 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1826ffd700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x191c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1827fff700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bcb50 0x191d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1826ffd700 1 -- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bcb50 msgr2=0x191d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1826ffd700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bcb50 0x191d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1826ffd700 1 -- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x189ae90 msgr2=0x18bb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1826ffd700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x189ae90 0x18bb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1826ffd700 1 -- 172.21.15.119:0/367475186 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f182806fca0 con 0x18bc230 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18277fe700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x189ae90 0x18bb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f1826ffd700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x191c4e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1820007cf0 tx=0x7f182000c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5598327da007ad6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:35.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 10 client.0 ms_handle_connect on v2:172.21.15.145:3301/0 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 1 -- 172.21.15.119:0/367475186 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182002c040 con 0x18bc230 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 1 -- 172.21.15.119:0/367475186 <== mon.2 v2:172.21.15.145:3301/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f18200159a0 con 0x18bc230 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f181805c5d0 con 0x18bc230 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 1 -- 172.21.15.119:0/367475186 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_get_version_reply(handle=1 version=42) v2 ==== 24+0+0 (secure 0 0 0) 0x7f18200076a0 con 0x18bc230 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 10 client.6305 fetch_fsmap learned FSMap version 42 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,fsmap=42,monmap=2+}) v3 -- 0x164f960 con 0x18bc230 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 1 -- 172.21.15.119:0/367475186 <== mon.2 v2:172.21.15.145:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181805c5d0 con 0x18bc230 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 1 -- 172.21.15.119:0/367475186 <== mon.2 v2:172.21.15.145:3301/0 5 ==== fsmap(e 42) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1820015ba0 con 0x18bc230 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 10 client.6305 fetch_fsmap finished waiting for FSMap version 42 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 10 client.6305 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 20 client.6305 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x191df60 0x18bff90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:35.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 4 client.6305 mds_command: new command op to 7101 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 --> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] -- command(tid 0: {"prefix": "status"}) v1 -- 0x16801e0 con 0x191df60 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18277fe700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x191df60 0x18bff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 10 client.6305 ms_handle_connect on v2:172.21.15.119:6834/491709455 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18277fe700 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x191df60 0x18bff90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cd9db0a52111660 server_cookie=6f545e585aac4986 in_seq=0 out_seq=0 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 1 -- 172.21.15.119:0/367475186 <== mds.0 v2:172.21.15.119:6834/491709455 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x16801e0 con 0x191df60 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18067fc700 10 client.6305 handle_command_reply: tid=0 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.980+0000 7f18312e4740 1 client.6305 shutdown 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 msgr2=0x191c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x191c4e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1820007cf0 tx=0x7f182000c000 comp rx=0 tx=0).stop 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 shutdown_connections 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bcb50 0x191d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bc230 0x191c4e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:6834/491709455,v1:172.21.15.119:6835/491709455] conn(0x191df60 0x18bff90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 --2- 172.21.15.119:0/367475186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x189ae90 0x18bb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18067fc700 0 client.6305 ms_handle_reset on v2:172.21.15.119:6834/491709455 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 >> 172.21.15.119:0/367475186 conn(0x16044d0 msgr2=0x1874d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 shutdown_connections 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 1 -- 172.21.15.119:0/367475186 wait complete. 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f18312e4740 20 client.6305 trim_cache size 0 max 16384 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:35.990 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 7101, 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 5.7563309309999999, 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 42, 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 82, 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 82, 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 10.251836319000001 2024-09-13T16:29:35.991 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1808066730 msgr2=0x7f1808068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1808066730 0x7f1808068bf0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f1810014040 tx=0x7f181001c040 comp rx=0 tx=0).stop 2024-09-13T16:29:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18280cd8d0 msgr2=0x7f18281fa2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18280cd8d0 0x7f18281fa2f0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f181800c370 tx=0x7f181800c680 comp rx=0 tx=0).stop 2024-09-13T16:29:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 shutdown_connections 2024-09-13T16:29:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1808066730 0x7f1808068bf0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f18280cd8d0 0x7f18281fa2f0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f18280b2200 0x7f18281f9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 --2- 172.21.15.119:0/2679032874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f18280b17f0 0x7f182814cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 >> 172.21.15.119:0/2679032874 conn(0x7f182806a350 msgr2=0x7f1828162690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 shutdown_connections 2024-09-13T16:29:35.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:35.984+0000 7f182cdd1700 1 -- 172.21.15.119:0/2679032874 wait complete. 2024-09-13T16:29:36.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:29:36.125 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f20f0bc6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:29:36.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc7fff700 1 Processor -- start 2024-09-13T16:29:36.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc7fff700 1 -- start start 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc7fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4bc80b17f0 0x7f4bc80cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4bc80b2200 0x7f4bc80b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc7fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc80705d0 con 0x7f4bc80cd8d0 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc7fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc80b2bb0 con 0x7f4bc80b2200 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc7fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc806fca0 con 0x7f4bc80b17f0 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc77fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4bc80b2200 0x7f4bc80b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.120+0000 7f4bc77fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4bc80b2200 0x7f4bc80b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46076/0 (socket says 172.21.15.119:46076) 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc77fe700 1 -- 172.21.15.119:0/3257760632 learned_addr learned my addr 172.21.15.119:0/3257760632 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:36.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc67fc700 1 --2- 172.21.15.119:0/3257760632 >> v2:172.21.15.145:3300/0 conn(0x7f4bc80b17f0 0x7f4bc80cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc77fe700 1 -- 172.21.15.119:0/3257760632 >> v2:172.21.15.145:3300/0 conn(0x7f4bc80b17f0 msgr2=0x7f4bc80cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc77fe700 1 --2- 172.21.15.119:0/3257760632 >> v2:172.21.15.145:3300/0 conn(0x7f4bc80b17f0 0x7f4bc80cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc77fe700 1 -- 172.21.15.119:0/3257760632 >> v1:172.21.15.145:6789/0 conn(0x7f4bc80cd8d0 legacy=0x7f4bc806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc77fe700 1 -- 172.21.15.119:0/3257760632 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc806ff80 con 0x7f4bc80b2200 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc77fe700 1 --2- 172.21.15.119:0/3257760632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b2200 0x7f4bc80b2670 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80ae2c0 tx=0x7f4bb8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d0ff5e21193ffa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc5ffb700 1 -- 172.21.15.119:0/3257760632 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011830 con 0x7f4bc80b2200 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc5ffb700 1 -- 172.21.15.119:0/3257760632 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb8011990 con 0x7f4bc80b2200 2024-09-13T16:29:36.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc5ffb700 1 -- 172.21.15.119:0/3257760632 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb8011d20 con 0x7f4bc80b2200 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 -- 172.21.15.119:0/3257760632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b2200 msgr2=0x7f4bc80b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/3257760632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b2200 0x7f4bc80b2670 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f4bc80ae2c0 tx=0x7f4bb8009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 -- 172.21.15.119:0/3257760632 shutdown_connections 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/3257760632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b2200 0x7f4bc80b2670 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/3257760632 >> v2:172.21.15.145:3300/0 conn(0x7f4bc80b17f0 0x7f4bc80cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 -- 172.21.15.119:0/3257760632 >> 172.21.15.119:0/3257760632 conn(0x7f4bc806a350 msgr2=0x7f4bc806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 -- 172.21.15.119:0/3257760632 shutdown_connections 2024-09-13T16:29:36.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 -- 172.21.15.119:0/3257760632 wait complete. 2024-09-13T16:29:36.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 Processor -- start 2024-09-13T16:29:36.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.124+0000 7f4bc7fff700 1 -- start start 2024-09-13T16:29:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b17f0 0x7f4bc8154e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bc80b2200 0x7f4bc815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bc80cd8d0 0x7f4bc815c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80af910 con 0x7f4bc80b17f0 2024-09-13T16:29:36.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80aefe0 con 0x7f4bc80cd8d0 2024-09-13T16:29:36.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc80af2c0 con 0x7f4bc80b2200 2024-09-13T16:29:36.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b17f0 0x7f4bc8154e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:36.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b17f0 0x7f4bc8154e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46082/0 (socket says 172.21.15.119:46082) 2024-09-13T16:29:36.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc6ffd700 1 -- 172.21.15.119:0/2001052111 learned_addr learned my addr 172.21.15.119:0/2001052111 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:36.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc67fc700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bc80b2200 0x7f4bc815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:36.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc77fe700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bc80cd8d0 0x7f4bc815c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:36.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc67fc700 1 -- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bc80cd8d0 msgr2=0x7f4bc815c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:36.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc67fc700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bc80cd8d0 0x7f4bc815c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc67fc700 1 -- 172.21.15.119:0/2001052111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b17f0 msgr2=0x7f4bc8154e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:36.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc67fc700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b17f0 0x7f4bc8154e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc67fc700 1 -- 172.21.15.119:0/2001052111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009570 con 0x7f4bc80b2200 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc6ffd700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b17f0 0x7f4bc8154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc67fc700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bc80b2200 0x7f4bc815bbc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e900 tx=0x7f4bb000ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bbf7fe700 1 -- 172.21.15.119:0/2001052111 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0013680 con 0x7f4bc80b2200 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bbf7fe700 1 -- 172.21.15.119:0/2001052111 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb00137e0 con 0x7f4bc80b2200 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc80af5a0 con 0x7f4bc80b2200 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bbf7fe700 1 -- 172.21.15.119:0/2001052111 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb0013b50 con 0x7f4bc80b2200 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.128+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc8158890 con 0x7f4bc80b2200 2024-09-13T16:29:36.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.132+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc80cdcb0 con 0x7f4bc80b2200 2024-09-13T16:29:36.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.132+0000 7f4bbf7fe700 1 -- 172.21.15.119:0/2001052111 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4bb0013cb0 con 0x7f4bc80b2200 2024-09-13T16:29:36.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.132+0000 7f4bbf7fe700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ba8066760 0x7f4ba8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:36.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.132+0000 7f4bbf7fe700 1 -- 172.21.15.119:0/2001052111 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4bb0084740 con 0x7f4bc80b2200 2024-09-13T16:29:36.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.132+0000 7f4bc6ffd700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ba8066760 0x7f4ba8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:36.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.132+0000 7f4bc6ffd700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ba8066760 0x7f4ba8068c20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000d670 tx=0x7f4bc0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:36.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.140+0000 7f4bbf7fe700 1 -- 172.21.15.119:0/2001052111 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4bb0051840 con 0x7f4bc80b2200 2024-09-13T16:29:36.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.352+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f4bc80af400 con 0x7f4bc80b2200 2024-09-13T16:29:36.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.352+0000 7f4bbf7fe700 1 -- 172.21.15.119:0/2001052111 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v42) v1 ==== 53+0+73 (secure 0 0 0) 0x7f4bc80af400 con 0x7f4bc80b2200 2024-09-13T16:29:36.358 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:29:36.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ba8066760 msgr2=0x7f4ba8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ba8066760 0x7f4ba8068c20 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4bc000d670 tx=0x7f4bc0009040 comp rx=0 tx=0).stop 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bc80b2200 msgr2=0x7f4bc815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bc80b2200 0x7f4bc815bbc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000e900 tx=0x7f4bb000ec10 comp rx=0 tx=0).stop 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 shutdown_connections 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ba8066760 0x7f4ba8068c20 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4bc80cd8d0 0x7f4bc815c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4bc80b2200 0x7f4bc815bbc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 --2- 172.21.15.119:0/2001052111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4bc80b17f0 0x7f4bc8154e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 >> 172.21.15.119:0/2001052111 conn(0x7f4bc806a350 msgr2=0x7f4bc80ccd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 shutdown_connections 2024-09-13T16:29:36.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:36.356+0000 7f4bc7fff700 1 -- 172.21.15.119:0/2001052111 wait complete. 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:29:36.373 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:29:36.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.374 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:29:36.378 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:29:36.378 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:29:36.378 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:29:36.379 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42365sec preferred_lft 42365sec 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:29:36.380 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:36.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.381 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:29:36.395 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:29:36.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.397 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:29:36.447 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:29:36.447 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:29:36.447 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:29:36.447 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:29:36.448 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:29:36.448 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:29:36.448 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:29:36.448 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:29:36.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.448 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:29:36.448 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:29:36.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:29:36.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:29:36.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:29:36.449 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:29:36.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.489 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:29:36.496 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:29:36.496 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:29:36.496 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:29:36.497 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.497 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:29:36.509 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:29:36.509 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:29:36.509 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:36.558 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:29:36.558 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:29:36.558 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:36.565 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:29:36.578 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:29:36.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:29:36.636 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:29:36.637 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:29:36.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.638 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:29:36.644 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:29:36.645 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:29:36.647 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:29:36.662 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:29:36.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:29:36.673 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:29:36.674 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:29:36.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:36.675 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:29:36.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:29:36.728+0000 7f60fb114640 -1 init, newargv = 0x55caffe1c8a0 newargc=15 2024-09-13T16:29:36.733 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[44173]: starting ceph client 2024-09-13T16:29:36.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[44173]: starting fuse 2024-09-13T16:29:37.826 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:29:37.845 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:29:37.846 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:29:37.860 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:29:37.861 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:29:37.861 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:37.862 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:29:37.866 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:29:37.866 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:29:37.866 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:29:37.866 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:29:37.866 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:37.867 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:29:37.868 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:29:37.990 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.44173.asok 2024-09-13T16:29:38.175 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44173.asok 2024-09-13T16:29:38.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:38.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44173.asok status 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "44173", 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:29:38.279 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 7254, 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 7254 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:29:38.280 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3570179052 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3570179052 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.7254 192.168.0.1:0/3570179052", 2024-09-13T16:29:38.281 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/3570179052", 2024-09-13T16:29:38.282 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:29:38.282 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 42, 2024-09-13T16:29:38.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 82, 2024-09-13T16:29:38.282 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:29:38.282 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:29:38.282 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:29:38.282 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:38.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:38.289 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:29:38.289 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:38.295 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:29:38.295 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:38.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:38.295 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:38.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:38.356 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir subdir/) 2024-09-13T16:29:38.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:38.374 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_undamaged) 2024-09-13T16:29:38.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:38.388 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch subdir/file_to_be_damaged) 2024-09-13T16:29:38.400 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:29:38.400 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:29:38.401 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:29:38.401 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:38.401 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/subdir").st_ino) 2024-09-13T16:29:38.401 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:29:38.511 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627776 2024-09-13T16:29:38.700 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:38.700 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:38.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:38.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.820+0000 7fb41a1b3700 1 Processor -- start 2024-09-13T16:29:38.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.820+0000 7fb41a1b3700 1 -- start start 2024-09-13T16:29:38.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b2200 0x7fb4140b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:38.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4140705d0 con 0x7fb4140cd8d0 2024-09-13T16:29:38.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41406fca0 con 0x7fb4140b17f0 2024-09-13T16:29:38.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4140cbf10 con 0x7fb4140b2200 2024-09-13T16:29:38.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb4191b1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb4140cd8d0 0x7fb41406cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50664/0 (socket says 172.21.15.119:50664) 2024-09-13T16:29:38.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb4191b1700 1 -- 172.21.15.119:0/1165399500 learned_addr learned my addr 172.21.15.119:0/1165399500 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:38.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb4199b2700 1 --2- 172.21.15.119:0/1165399500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b2200 0x7fb4140b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:38.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb407fff700 1 -- 172.21.15.119:0/1165399500 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688815365 0 0) 0x7fb4140705d0 con 0x7fb4140cd8d0 2024-09-13T16:29:38.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb407fff700 1 -- 172.21.15.119:0/1165399500 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4000035d0 con 0x7fb4140cd8d0 2024-09-13T16:29:38.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb407fff700 1 -- 172.21.15.119:0/1165399500 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892559673 0 0) 0x7fb41406fca0 con 0x7fb4140b17f0 2024-09-13T16:29:38.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb407fff700 1 -- 172.21.15.119:0/1165399500 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4140705d0 con 0x7fb4140b17f0 2024-09-13T16:29:38.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb4199b2700 1 -- 172.21.15.119:0/1165399500 >> v1:172.21.15.145:6789/0 conn(0x7fb4140b17f0 legacy=0x7fb4140cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:38.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb4199b2700 1 -- 172.21.15.119:0/1165399500 >> v1:172.21.15.119:6789/0 conn(0x7fb4140cd8d0 legacy=0x7fb41406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:38.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb4199b2700 1 -- 172.21.15.119:0/1165399500 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41406ff80 con 0x7fb4140b2200 2024-09-13T16:29:38.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb4199b2700 1 --2- 172.21.15.119:0/1165399500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b2200 0x7fb4140b2670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009880 tx=0x7fb40c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74912408776683c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:38.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb407fff700 1 -- 172.21.15.119:0/1165399500 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c011830 con 0x7fb4140b2200 2024-09-13T16:29:38.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb407fff700 1 -- 172.21.15.119:0/1165399500 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c011990 con 0x7fb4140b2200 2024-09-13T16:29:38.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb407fff700 1 -- 172.21.15.119:0/1165399500 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c011d00 con 0x7fb4140b2200 2024-09-13T16:29:38.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- 172.21.15.119:0/1165399500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b2200 msgr2=0x7fb4140b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:38.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/1165399500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b2200 0x7fb4140b2670 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fb40c009880 tx=0x7fb40c009b90 comp rx=0 tx=0).stop 2024-09-13T16:29:38.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- 172.21.15.119:0/1165399500 shutdown_connections 2024-09-13T16:29:38.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/1165399500 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b2200 0x7fb4140b2670 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:38.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- 172.21.15.119:0/1165399500 >> 172.21.15.119:0/1165399500 conn(0x7fb41406a350 msgr2=0x7fb41406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:38.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- 172.21.15.119:0/1165399500 shutdown_connections 2024-09-13T16:29:38.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.824+0000 7fb41a1b3700 1 -- 172.21.15.119:0/1165399500 wait complete. 2024-09-13T16:29:38.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 Processor -- start 2024-09-13T16:29:38.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 -- start start 2024-09-13T16:29:38.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b17f0 0x7fb4141f5e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:38.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4140b2200 0x7fb41415c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:38.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4140cd8d0 0x7fb41415ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:38.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb414157dc0 con 0x7fb4140b2200 2024-09-13T16:29:38.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41415a850 con 0x7fb4140cd8d0 2024-09-13T16:29:38.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb41a1b3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4140c3bb0 con 0x7fb4140b17f0 2024-09-13T16:29:38.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4140b2200 0x7fb41415c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:38.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4140b2200 0x7fb41415c4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46086/0 (socket says 172.21.15.119:46086) 2024-09-13T16:29:38.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 -- 172.21.15.119:0/321690248 learned_addr learned my addr 172.21.15.119:0/321690248 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:38.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4191b1700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b17f0 0x7fb4141f5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:38.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4199b2700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4140cd8d0 0x7fb41415ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:38.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 -- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b17f0 msgr2=0x7fb4141f5e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:38.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b17f0 0x7fb4141f5e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:38.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 -- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4140cd8d0 msgr2=0x7fb41415ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:38.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4140cd8d0 0x7fb41415ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:38.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 -- 172.21.15.119:0/321690248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c0094b0 con 0x7fb4140b2200 2024-09-13T16:29:38.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.828+0000 7fb4189b0700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4140b2200 0x7fb41415c4d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb408001ee0 tx=0x7fb4080021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:38.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb405ffb700 1 -- 172.21.15.119:0/321690248 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb408003c20 con 0x7fb4140b2200 2024-09-13T16:29:38.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb405ffb700 1 -- 172.21.15.119:0/321690248 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb408003da0 con 0x7fb4140b2200 2024-09-13T16:29:38.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb405ffb700 1 -- 172.21.15.119:0/321690248 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40800ab00 con 0x7fb4140b2200 2024-09-13T16:29:38.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41415aaf0 con 0x7fb4140b2200 2024-09-13T16:29:38.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41414cd80 con 0x7fb4140b2200 2024-09-13T16:29:38.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb414158250 con 0x7fb4140b2200 2024-09-13T16:29:38.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb405ffb700 1 -- 172.21.15.119:0/321690248 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb40800ac60 con 0x7fb4140b2200 2024-09-13T16:29:38.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb405ffb700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3f8066730 0x7fb3f8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:38.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.832+0000 7fb405ffb700 1 -- 172.21.15.119:0/321690248 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb4080842c0 con 0x7fb4140b2200 2024-09-13T16:29:38.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.836+0000 7fb4191b1700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3f8066730 0x7fb3f8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:38.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.836+0000 7fb4191b1700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3f8066730 0x7fb3f8068bf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb410007fd0 tx=0x7fb410007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:38.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:38.840+0000 7fb405ffb700 1 -- 172.21.15.119:0/321690248 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb414158250 con 0x7fb4140b2200 2024-09-13T16:29:39.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.016+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb41414cf10 con 0x7fb4140b2200 2024-09-13T16:29:39.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.016+0000 7fb405ffb700 1 -- 172.21.15.119:0/321690248 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16837 (secure 0 0 0) 0x7fb408051500 con 0x7fb4140b2200 2024-09-13T16:29:39.023 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:39.023 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":82,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:31.230578+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:39.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3f8066730 msgr2=0x7fb3f8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3f8066730 0x7fb3f8068bf0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fb410007fd0 tx=0x7fb410007de0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4140b2200 msgr2=0x7fb41415c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4140b2200 0x7fb41415c4d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb408001ee0 tx=0x7fb4080021f0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 shutdown_connections 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3f8066730 0x7fb3f8068bf0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4140cd8d0 0x7fb41415ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4140b2200 0x7fb41415c4d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 --2- 172.21.15.119:0/321690248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4140b17f0 0x7fb4141f5e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 >> 172.21.15.119:0/321690248 conn(0x7fb41406a350 msgr2=0x7fb41406d700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 shutdown_connections 2024-09-13T16:29:39.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.020+0000 7fb41a1b3700 1 -- 172.21.15.119:0/321690248 wait complete. 2024-09-13T16:29:39.038 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:29:28.552550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:29:29.210253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:39.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:39.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 Processor -- start 2024-09-13T16:29:39.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 -- start start 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f07e00add60 0x7f07e00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e00b3260 0x7f07e00b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e00b3bf0 con 0x7f07e015dbe0 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e0071be0 con 0x7f07e00add60 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0159af0 con 0x7f07e00b3260 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e6cbc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e00b3260 0x7f07e00b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e64bb700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f07e00add60 0x7f07e00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e6cbc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e00b3260 0x7f07e00b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:43788/0 (socket says 172.21.15.119:43788) 2024-09-13T16:29:39.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e6cbc700 1 -- 172.21.15.119:0/2152328368 learned_addr learned my addr 172.21.15.119:0/2152328368 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:39.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e64bb700 1 -- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e00b3260 msgr2=0x7f07e00b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:39.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e64bb700 1 --2- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e00b3260 0x7f07e00b36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e64bb700 1 -- 172.21.15.119:0/2152328368 >> v1:172.21.15.145:6789/0 conn(0x7f07e015dbe0 legacy=0x7f07e00cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e64bb700 1 -- 172.21.15.119:0/2152328368 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e00cd4d0 con 0x7f07e00add60 2024-09-13T16:29:39.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e6cbc700 1 --2- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e00b3260 0x7f07e00b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:39.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e64bb700 1 --2- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 0x7f07e00ce0f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009850 tx=0x7f07dc009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60ca9923a71d44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e54b9700 1 -- 172.21.15.119:0/2152328368 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc00f8a0 con 0x7f07e00add60 2024-09-13T16:29:39.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e54b9700 1 -- 172.21.15.119:0/2152328368 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07dc00fa00 con 0x7f07e00add60 2024-09-13T16:29:39.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e54b9700 1 -- 172.21.15.119:0/2152328368 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07dc00fdb0 con 0x7f07e00add60 2024-09-13T16:29:39.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.156+0000 7f07e74bd700 1 -- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 msgr2=0x7f07e00ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 --2- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 0x7f07e00ce0f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f07dc009850 tx=0x7f07dc009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:39.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- 172.21.15.119:0/2152328368 shutdown_connections 2024-09-13T16:29:39.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 --2- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e00b3260 0x7f07e00b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 --2- 172.21.15.119:0/2152328368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 0x7f07e00ce0f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- 172.21.15.119:0/2152328368 >> 172.21.15.119:0/2152328368 conn(0x7f07e00b0980 msgr2=0x7f07e00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:39.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- 172.21.15.119:0/2152328368 shutdown_connections 2024-09-13T16:29:39.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- 172.21.15.119:0/2152328368 wait complete. 2024-09-13T16:29:39.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 Processor -- start 2024-09-13T16:29:39.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- start start 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 0x7f07e01f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07e00b3260 0x7f07e0155270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e015dbe0 0x7f07e01571f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0156dd0 con 0x7f07e00b3260 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e01f2ff0 con 0x7f07e00add60 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e74bd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e0156150 con 0x7f07e015dbe0 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07e00b3260 0x7f07e0155270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e64bb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 0x7f07e01f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07e00b3260 0x7f07e0155270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46088/0 (socket says 172.21.15.119:46088) 2024-09-13T16:29:39.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 -- 172.21.15.119:0/3565878352 learned_addr learned my addr 172.21.15.119:0/3565878352 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e6cbc700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e015dbe0 0x7f07e01571f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 -- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e015dbe0 msgr2=0x7f07e01571f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e015dbe0 0x7f07e01571f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 -- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 msgr2=0x7f07e01f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 0x7f07e01f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.160+0000 7f07e5cba700 1 -- 172.21.15.119:0/3565878352 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d8009b30 con 0x7f07e00b3260 2024-09-13T16:29:39.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07e5cba700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07e00b3260 0x7f07e0155270 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f07d000c500 tx=0x7f07d000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07d6ffd700 1 -- 172.21.15.119:0/3565878352 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d00052d0 con 0x7f07e00b3260 2024-09-13T16:29:39.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07dc0094b0 con 0x7f07e00b3260 2024-09-13T16:29:39.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07d6ffd700 1 -- 172.21.15.119:0/3565878352 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d0005430 con 0x7f07e00b3260 2024-09-13T16:29:39.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07d6ffd700 1 -- 172.21.15.119:0/3565878352 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07d00055a0 con 0x7f07e00b3260 2024-09-13T16:29:39.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e0156710 con 0x7f07e00b3260 2024-09-13T16:29:39.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e0156f10 con 0x7f07e00b3260 2024-09-13T16:29:39.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07d6ffd700 1 -- 172.21.15.119:0/3565878352 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f07d00038e0 con 0x7f07e00b3260 2024-09-13T16:29:39.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.164+0000 7f07d6ffd700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07c4066730 0x7f07c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.168+0000 7f07d6ffd700 1 -- 172.21.15.119:0/3565878352 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f07d00834c0 con 0x7f07e00b3260 2024-09-13T16:29:39.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.168+0000 7f07e64bb700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07c4066730 0x7f07c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.168+0000 7f07e64bb700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07c4066730 0x7f07c4068bf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0090b0 tx=0x7f07dc01b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.176+0000 7f07d6ffd700 1 -- 172.21.15.119:0/3565878352 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f07e0156f10 con 0x7f07e00b3260 2024-09-13T16:29:39.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.344+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f07e00b3bf0 con 0x7f07e00b3260 2024-09-13T16:29:39.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.344+0000 7f07d6ffd700 1 -- 172.21.15.119:0/3565878352 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f07e00b3bf0 con 0x7f07e00b3260 2024-09-13T16:29:39.351 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07c4066730 msgr2=0x7f07c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07c4066730 0x7f07c4068bf0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f07dc0090b0 tx=0x7f07dc01b040 comp rx=0 tx=0).stop 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07e00b3260 msgr2=0x7f07e0155270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07e00b3260 0x7f07e0155270 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7f07d000c500 tx=0x7f07d000c810 comp rx=0 tx=0).stop 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 shutdown_connections 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07c4066730 0x7f07c4068bf0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07e015dbe0 0x7f07e01571f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07e00b3260 0x7f07e0155270 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 --2- 172.21.15.119:0/3565878352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f07e00add60 0x7f07e01f2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 >> 172.21.15.119:0/3565878352 conn(0x7f07e00b0980 msgr2=0x7f07e00b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:39.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 shutdown_connections 2024-09-13T16:29:39.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.348+0000 7f07e74bd700 1 -- 172.21.15.119:0/3565878352 wait complete. 2024-09-13T16:29:39.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:29:39.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.480+0000 7fb516aa4700 1 Processor -- start 2024-09-13T16:29:39.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.480+0000 7fb516aa4700 1 -- start start 2024-09-13T16:29:39.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb516aa4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb51015dbe0 0x7fb5100ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb516aa4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb5100b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb516aa4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5100b3bf0 con 0x7fb5100add60 2024-09-13T16:29:39.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb516aa4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb510071be0 con 0x7fb51015dbe0 2024-09-13T16:29:39.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb516aa4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb510159af0 con 0x7fb5100b3260 2024-09-13T16:29:39.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb515aa2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb5100add60 0x7fb5100cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50678/0 (socket says 172.21.15.119:50678) 2024-09-13T16:29:39.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb515aa2700 1 -- 172.21.15.119:0/987095745 learned_addr learned my addr 172.21.15.119:0/987095745 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:39.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5152a1700 1 --2- 172.21.15.119:0/987095745 >> v2:172.21.15.145:3300/0 conn(0x7fb51015dbe0 0x7fb5100ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5162a3700 1 --2- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb5100b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb514aa0700 1 -- 172.21.15.119:0/987095745 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933451660 0 0) 0x7fb4f8001110 con 0x7fb5100add60 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb514aa0700 1 -- 172.21.15.119:0/987095745 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4fc0035d0 con 0x7fb5100add60 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5152a1700 1 -- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 msgr2=0x7fb5100b36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5152a1700 1 --2- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb5100b36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5152a1700 1 -- 172.21.15.119:0/987095745 >> v1:172.21.15.119:6789/0 conn(0x7fb5100add60 legacy=0x7fb5100cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5152a1700 1 -- 172.21.15.119:0/987095745 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5100cd4d0 con 0x7fb51015dbe0 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5162a3700 1 --2- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb5100b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb5152a1700 1 --2- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 0x7fb5100ce0f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb500009d90 tx=0x7fb50000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13054c8235af3a63 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb514aa0700 1 -- 172.21.15.119:0/987095745 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50000e070 con 0x7fb51015dbe0 2024-09-13T16:29:39.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb514aa0700 1 -- 172.21.15.119:0/987095745 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb50000bb20 con 0x7fb51015dbe0 2024-09-13T16:29:39.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb514aa0700 1 -- 172.21.15.119:0/987095745 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb50000be90 con 0x7fb51015dbe0 2024-09-13T16:29:39.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb516aa4700 1 -- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 msgr2=0x7fb5100ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.484+0000 7fb516aa4700 1 --2- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 0x7fb5100ce0f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fb500009d90 tx=0x7fb50000b330 comp rx=0 tx=0).stop 2024-09-13T16:29:39.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 -- 172.21.15.119:0/987095745 shutdown_connections 2024-09-13T16:29:39.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 --2- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb5100b36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 --2- 172.21.15.119:0/987095745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 0x7fb5100ce0f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 -- 172.21.15.119:0/987095745 >> 172.21.15.119:0/987095745 conn(0x7fb5100b0980 msgr2=0x7fb5100b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:39.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 -- 172.21.15.119:0/987095745 shutdown_connections 2024-09-13T16:29:39.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 -- 172.21.15.119:0/987095745 wait complete. 2024-09-13T16:29:39.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 Processor -- start 2024-09-13T16:29:39.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.488+0000 7fb516aa4700 1 -- start start 2024-09-13T16:29:39.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5100add60 0x7fb51014b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb51014f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 0x7fb510154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51014b9c0 con 0x7fb5100add60 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb51015bae0 con 0x7fb51015dbe0 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb51014fa70 con 0x7fb5100b3260 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5100add60 0x7fb51014b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5100add60 0x7fb51014b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46096/0 (socket says 172.21.15.119:46096) 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 -- 172.21.15.119:0/1845970176 learned_addr learned my addr 172.21.15.119:0/1845970176 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb5152a1700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb51014f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb5162a3700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 0x7fb510154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 -- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 msgr2=0x7fb51014f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb51014f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 -- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 msgr2=0x7fb510154ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 0x7fb510154ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 -- 172.21.15.119:0/1845970176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5080094b0 con 0x7fb5100add60 2024-09-13T16:29:39.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb515aa2700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5100add60 0x7fb51014b480 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800c4f0 tx=0x7fb4f800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb50e7fc700 1 -- 172.21.15.119:0/1845970176 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8005d90 con 0x7fb5100add60 2024-09-13T16:29:39.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb50e7fc700 1 -- 172.21.15.119:0/1845970176 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4f80070d0 con 0x7fb5100add60 2024-09-13T16:29:39.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb500009a70 con 0x7fb5100add60 2024-09-13T16:29:39.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb50e7fc700 1 -- 172.21.15.119:0/1845970176 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4f8007480 con 0x7fb5100add60 2024-09-13T16:29:39.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.492+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb510150030 con 0x7fb5100add60 2024-09-13T16:29:39.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.496+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5100cd160 con 0x7fb5100add60 2024-09-13T16:29:39.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.496+0000 7fb50e7fc700 1 -- 172.21.15.119:0/1845970176 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb4f80075e0 con 0x7fb5100add60 2024-09-13T16:29:39.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.496+0000 7fb50e7fc700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb4f0066760 0x7fb4f0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.496+0000 7fb50e7fc700 1 -- 172.21.15.119:0/1845970176 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb4f8083450 con 0x7fb5100add60 2024-09-13T16:29:39.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.496+0000 7fb5152a1700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb4f0066760 0x7fb4f0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.496+0000 7fb5152a1700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb4f0066760 0x7fb4f0068c20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb500014040 tx=0x7fb500011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.508+0000 7fb50e7fc700 1 -- 172.21.15.119:0/1845970176 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb4f8050710 con 0x7fb5100add60 2024-09-13T16:29:39.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.676+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5100b3bf0 con 0x7fb5100add60 2024-09-13T16:29:39.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.676+0000 7fb50e7fc700 1 -- 172.21.15.119:0/1845970176 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb5100b3bf0 con 0x7fb5100add60 2024-09-13T16:29:39.683 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:29:39.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb4f0066760 msgr2=0x7fb4f0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb4f0066760 0x7fb4f0068c20 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fb500014040 tx=0x7fb500011000 comp rx=0 tx=0).stop 2024-09-13T16:29:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5100add60 msgr2=0x7fb51014b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5100add60 0x7fb51014b480 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7fb4f800c4f0 tx=0x7fb4f800c800 comp rx=0 tx=0).stop 2024-09-13T16:29:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 shutdown_connections 2024-09-13T16:29:39.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb4f0066760 0x7fb4f0068c20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb51015dbe0 0x7fb510154ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5100b3260 0x7fb51014f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 --2- 172.21.15.119:0/1845970176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5100add60 0x7fb51014b480 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 >> 172.21.15.119:0/1845970176 conn(0x7fb5100b0980 msgr2=0x7fb510070a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:39.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 shutdown_connections 2024-09-13T16:29:39.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.680+0000 7fb516aa4700 1 -- 172.21.15.119:0/1845970176 wait complete. 2024-09-13T16:29:39.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:29:39.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.812+0000 7fccae4d6700 1 Processor -- start 2024-09-13T16:29:39.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.812+0000 7fccae4d6700 1 -- start start 2024-09-13T16:29:39.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.812+0000 7fccae4d6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 0x7fcca8071c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.812+0000 7fccae4d6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcca806db40 0x7fcca80b1eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.812+0000 7fccae4d6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcca806ce40 con 0x7fcca80b23f0 2024-09-13T16:29:39.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.812+0000 7fccae4d6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcca80702f0 con 0x7fcca806db40 2024-09-13T16:29:39.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.812+0000 7fccae4d6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8071730 con 0x7fcca80b1ad0 2024-09-13T16:29:39.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccaca73700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcca80b23f0 0x7fcca8161260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50684/0 (socket says 172.21.15.119:50684) 2024-09-13T16:29:39.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccaca73700 1 -- 172.21.15.119:0/1871151342 learned_addr learned my addr 172.21.15.119:0/1871151342 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:39.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca7fff700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 0x7fcca8071c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 --2- 172.21.15.119:0/1871151342 >> v2:172.21.15.145:3300/0 conn(0x7fcca806db40 0x7fcca80b1eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717460992 0 0) 0x7fcca0001110 con 0x7fcca80b23f0 2024-09-13T16:29:39.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc940035d0 con 0x7fcca80b23f0 2024-09-13T16:29:39.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1946871542 0 0) 0x7fcc940035d0 con 0x7fcca80b23f0 2024-09-13T16:29:39.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 msgr2=0x7fcca8071c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:39.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 0x7fcca8071c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 >> v2:172.21.15.145:3300/0 conn(0x7fcca806db40 msgr2=0x7fcca80b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:39.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 --2- 172.21.15.119:0/1871151342 >> v2:172.21.15.145:3300/0 conn(0x7fcca806db40 0x7fcca80b1eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca806c8a0 con 0x7fcca80b23f0 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fcca0001800 con 0x7fcca80b23f0 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 >> v1:172.21.15.119:6789/0 conn(0x7fcca80b23f0 legacy=0x7fcca8161260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc94003a20 0x7fcc94003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc94004870 0x7fcc94004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 0x7fcc94005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca80702f0 con 0x7fcc94004870 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8071730 con 0x7fcc94003a20 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc9400ba30 con 0x7fcca80b1ad0 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc94004870 0x7fcc94004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca7fff700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc94003a20 0x7fcc94003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccaca73700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 0x7fcc94005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 -- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 msgr2=0x7fcc94005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:39.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 0x7fcc94005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 -- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc94003a20 msgr2=0x7fcc94003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc94003a20 0x7fcc94003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 -- 172.21.15.119:0/1871151342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc940035d0 con 0x7fcc94004870 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccaca73700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca80b1ad0 0x7fcc94005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca7fff700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcc94003a20 0x7fcc94003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca77fe700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc94004870 0x7fcc94004cf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009b70 tx=0x7fcc940058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73e53b8dcdbea02c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9000e6c0 con 0x7fcc94004870 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc9000e820 con 0x7fcc94004870 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fcca6ffd700 1 -- 172.21.15.119:0/1871151342 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc9000eb90 con 0x7fcc94004870 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccae4d6700 1 -- 172.21.15.119:0/1871151342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc94004870 msgr2=0x7fcc94004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccae4d6700 1 --2- 172.21.15.119:0/1871151342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcc94004870 0x7fcc94004cf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fcc90009b70 tx=0x7fcc940058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccaca73700 1 -- 172.21.15.119:0/1871151342 reap_dead start 2024-09-13T16:29:39.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccae4d6700 1 -- 172.21.15.119:0/1871151342 shutdown_connections 2024-09-13T16:29:39.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccae4d6700 1 -- 172.21.15.119:0/1871151342 >> 172.21.15.119:0/1871151342 conn(0x7fcca80b0560 msgr2=0x7fcca80b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:39.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccae4d6700 1 -- 172.21.15.119:0/1871151342 shutdown_connections 2024-09-13T16:29:39.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.816+0000 7fccae4d6700 1 -- 172.21.15.119:0/1871151342 wait complete. 2024-09-13T16:29:39.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 Processor -- start 2024-09-13T16:29:39.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 -- start start 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca8153a20 0x7fcca8159740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca80a96c0 0x7fcca8156bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcca8157130 0x7fcca8157fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca81575e0 con 0x7fcca8157130 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8153e00 con 0x7fcca80a96c0 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccae4d6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8159c80 con 0x7fcca8153a20 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcca8157130 0x7fcca8157fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcca8157130 0x7fcca8157fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46108/0 (socket says 172.21.15.119:46108) 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 -- 172.21.15.119:0/292848408 learned_addr learned my addr 172.21.15.119:0/292848408 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fcca7fff700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca8153a20 0x7fcca8159740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fcca77fe700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca80a96c0 0x7fcca8156bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 -- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca8153a20 msgr2=0x7fcca8159740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca8153a20 0x7fcca8159740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 -- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca80a96c0 msgr2=0x7fcca8156bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:39.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca80a96c0 0x7fcca8156bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:39.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 -- 172.21.15.119:0/292848408 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca8159f50 con 0x7fcca8157130 2024-09-13T16:29:39.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fcca77fe700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca80a96c0 0x7fcca8156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:29:39.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.820+0000 7fccaca73700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcca8157130 0x7fcca8157fa0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcca0000fb0 tx=0x7fcca00073e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fcca4ff9700 1 -- 172.21.15.119:0/292848408 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0010510 con 0x7fcca8157130 2024-09-13T16:29:39.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fcca4ff9700 1 -- 172.21.15.119:0/292848408 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca0010690 con 0x7fcca8157130 2024-09-13T16:29:39.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca815a230 con 0x7fcca8157130 2024-09-13T16:29:39.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fcca4ff9700 1 -- 172.21.15.119:0/292848408 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0010a40 con 0x7fcca8157130 2024-09-13T16:29:39.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca815a7f0 con 0x7fcca8157130 2024-09-13T16:29:39.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca80b3ac0 con 0x7fcca8157130 2024-09-13T16:29:39.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fcca4ff9700 1 -- 172.21.15.119:0/292848408 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcca0010ba0 con 0x7fcca8157130 2024-09-13T16:29:39.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fcca4ff9700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcc88066730 0x7fcc88068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:39.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.824+0000 7fcca4ff9700 1 -- 172.21.15.119:0/292848408 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcca0083a10 con 0x7fcca8157130 2024-09-13T16:29:39.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.828+0000 7fcca7fff700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcc88066730 0x7fcc88068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:39.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.828+0000 7fcca7fff700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcc88066730 0x7fcc88068bf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcc98007440 tx=0x7fcc9800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:39.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:39.832+0000 7fcca4ff9700 1 -- 172.21.15.119:0/292848408 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcca0050c30 con 0x7fcca8157130 2024-09-13T16:29:40.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcca806ce40 con 0x7fcca8157130 2024-09-13T16:29:40.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fcca4ff9700 1 -- 172.21.15.119:0/292848408 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcca806ce40 con 0x7fcca8157130 2024-09-13T16:29:40.012 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:29:40.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcc88066730 msgr2=0x7fcc88068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcc88066730 0x7fcc88068bf0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fcc98007440 tx=0x7fcc9800b040 comp rx=0 tx=0).stop 2024-09-13T16:29:40.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcca8157130 msgr2=0x7fcca8157fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcca8157130 0x7fcca8157fa0 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fcca0000fb0 tx=0x7fcca00073e0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 shutdown_connections 2024-09-13T16:29:40.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcc88066730 0x7fcc88068bf0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcca8157130 0x7fcca8157fa0 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcca80a96c0 0x7fcca8156bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 --2- 172.21.15.119:0/292848408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcca8153a20 0x7fcca8159740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 >> 172.21.15.119:0/292848408 conn(0x7fcca80b0560 msgr2=0x7fcca80aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:40.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 shutdown_connections 2024-09-13T16:29:40.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.008+0000 7fccae4d6700 1 -- 172.21.15.119:0/292848408 wait complete. 2024-09-13T16:29:40.028 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:29:40.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc71a849700 1 Processor -- start 2024-09-13T16:29:40.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc71a849700 1 -- start start 2024-09-13T16:29:40.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc71a849700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc71415cd60 0x7fc71406b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc71a849700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7140ceb60 con 0x7fc7140cf3c0 2024-09-13T16:29:40.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc71a849700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7140ce230 con 0x7fc7140cfed0 2024-09-13T16:29:40.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc71a849700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc71415d140 con 0x7fc71415cd60 2024-09-13T16:29:40.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc719046700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc7140cf3c0 0x7fc7140cf7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50700/0 (socket says 172.21.15.119:50700) 2024-09-13T16:29:40.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc719046700 1 -- 172.21.15.119:0/3680033614 learned_addr learned my addr 172.21.15.119:0/3680033614 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:40.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc719847700 1 --2- 172.21.15.119:0/3680033614 >> v2:172.21.15.145:3300/0 conn(0x7fc71415cd60 0x7fc71406b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382294233 0 0) 0x7fc7140ceb60 con 0x7fc7140cf3c0 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.140+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7000035d0 con 0x7fc7140cf3c0 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321612121 0 0) 0x7fc7140ce230 con 0x7fc7140cfed0 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7140ceb60 con 0x7fc7140cfed0 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 347538241 0 0) 0x7fc7000035d0 con 0x7fc7140cf3c0 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 >> v2:172.21.15.145:3300/0 conn(0x7fc71415cd60 msgr2=0x7fc71406b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 --2- 172.21.15.119:0/3680033614 >> v2:172.21.15.145:3300/0 conn(0x7fc71415cd60 0x7fc71406b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 >> v1:172.21.15.145:6789/0 conn(0x7fc7140cfed0 legacy=0x7fc7140d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140ce510 con 0x7fc7140cf3c0 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fc710001670 con 0x7fc7140cf3c0 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 >> v1:172.21.15.119:6789/0 conn(0x7fc7140cf3c0 legacy=0x7fc7140cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc700003bb0 0x7fc700003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700004870 0x7fc700004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc700005230 0x7fc70000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71415d140 con 0x7fc700003bb0 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc70000c0b0 con 0x7fc700005230 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc70000c270 con 0x7fc700004870 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719847700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc700003bb0 0x7fc700003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719046700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700004870 0x7fc700004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a048700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc700005230 0x7fc70000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719847700 1 -- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700004870 msgr2=0x7fc700004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719847700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700004870 0x7fc700004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719847700 1 -- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc700005230 msgr2=0x7fc70000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719847700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc700005230 0x7fc70000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719847700 1 -- 172.21.15.119:0/3680033614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7000035d0 con 0x7fc700003bb0 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a048700 1 -- 172.21.15.119:0/3680033614 reap_dead start 2024-09-13T16:29:40.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc719847700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc700003bb0 0x7fc700003fb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fc704009b70 tx=0x7fc7000058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41bb15c13c3756cf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70400e6c0 con 0x7fc700003bb0 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc70400e820 con 0x7fc700003bb0 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc718845700 1 -- 172.21.15.119:0/3680033614 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc70400eb90 con 0x7fc700003bb0 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 -- 172.21.15.119:0/3680033614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc700003bb0 msgr2=0x7fc700003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc700003bb0 0x7fc700003fb0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fc704009b70 tx=0x7fc7000058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 -- 172.21.15.119:0/3680033614 shutdown_connections 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 --2- 172.21.15.119:0/3680033614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc700003bb0 0x7fc700003fb0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 -- 172.21.15.119:0/3680033614 >> 172.21.15.119:0/3680033614 conn(0x7fc71408c240 msgr2=0x7fc71409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 -- 172.21.15.119:0/3680033614 shutdown_connections 2024-09-13T16:29:40.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 -- 172.21.15.119:0/3680033614 wait complete. 2024-09-13T16:29:40.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.144+0000 7fc71a849700 1 Processor -- start 2024-09-13T16:29:40.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 -- start start 2024-09-13T16:29:40.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700003bb0 0x7fc714150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc71415b420 0x7fc71415b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc71415d280 0x7fc71415d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71415bda0 con 0x7fc71415b420 2024-09-13T16:29:40.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71415e290 con 0x7fc71415d280 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71415dfb0 con 0x7fc700003bb0 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc71415b420 0x7fc71415b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719847700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700003bb0 0x7fc714150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc71415b420 0x7fc71415b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46126/0 (socket says 172.21.15.119:46126) 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 -- 172.21.15.119:0/560370207 learned_addr learned my addr 172.21.15.119:0/560370207 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a048700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc71415d280 0x7fc71415d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 -- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700003bb0 msgr2=0x7fc714150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700003bb0 0x7fc714150d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 -- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc71415d280 msgr2=0x7fc71415d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc71415d280 0x7fc71415d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 -- 172.21.15.119:0/560370207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7140b5c60 con 0x7fc71415b420 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a048700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc71415d280 0x7fc71415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:40.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc719046700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc71415b420 0x7fc71415b860 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc710007310 tx=0x7fc710007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc70a7fc700 1 -- 172.21.15.119:0/560370207 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000f660 con 0x7fc71415b420 2024-09-13T16:29:40.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc70a7fc700 1 -- 172.21.15.119:0/560370207 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc71000f800 con 0x7fc71415b420 2024-09-13T16:29:40.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7140b5f40 con 0x7fc71415b420 2024-09-13T16:29:40.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc70a7fc700 1 -- 172.21.15.119:0/560370207 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc71000fbb0 con 0x7fc71415b420 2024-09-13T16:29:40.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7140c9dd0 con 0x7fc71415b420 2024-09-13T16:29:40.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.148+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7140b6460 con 0x7fc71415b420 2024-09-13T16:29:40.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.152+0000 7fc70a7fc700 1 -- 172.21.15.119:0/560370207 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc71000fd10 con 0x7fc71415b420 2024-09-13T16:29:40.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.152+0000 7fc70a7fc700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6f8066730 0x7fc6f8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.152+0000 7fc70a7fc700 1 -- 172.21.15.119:0/560370207 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc7100840f0 con 0x7fc71415b420 2024-09-13T16:29:40.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.152+0000 7fc719847700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6f8066730 0x7fc6f8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.152+0000 7fc719847700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6f8066730 0x7fc6f8068bf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc7040090b0 tx=0x7fc70401f040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.156+0000 7fc70a7fc700 1 -- 172.21.15.119:0/560370207 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc7140b6460 con 0x7fc71415b420 2024-09-13T16:29:40.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.340+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc7140b6460 con 0x7fc71415b420 2024-09-13T16:29:40.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.340+0000 7fc70a7fc700 1 -- 172.21.15.119:0/560370207 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc7140b6460 con 0x7fc71415b420 2024-09-13T16:29:40.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6f8066730 msgr2=0x7fc6f8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6f8066730 0x7fc6f8068bf0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fc7040090b0 tx=0x7fc70401f040 comp rx=0 tx=0).stop 2024-09-13T16:29:40.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc71415b420 msgr2=0x7fc71415b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc71415b420 0x7fc71415b860 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc710007310 tx=0x7fc710007620 comp rx=0 tx=0).stop 2024-09-13T16:29:40.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 shutdown_connections 2024-09-13T16:29:40.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6f8066730 0x7fc6f8068bf0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc71415d280 0x7fc71415d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc71415b420 0x7fc71415b860 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 --2- 172.21.15.119:0/560370207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc700003bb0 0x7fc714150d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 >> 172.21.15.119:0/560370207 conn(0x7fc71408c240 msgr2=0x7fc7140c8840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:40.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 shutdown_connections 2024-09-13T16:29:40.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.344+0000 7fc71a849700 1 -- 172.21.15.119:0/560370207 wait complete. 2024-09-13T16:29:40.352 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:29:40.362 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:40.362 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:40.370 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 Processor -- start 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- start start 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f85780df8e0 0x7f857801c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f8578005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85780e3310 con 0x7f857804fe30 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85780e35f0 con 0x7f85780df8e0 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85780e38d0 con 0x7f8578050b90 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f8585ce9700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f857804fe30 0x7f8578050480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50704/0 (socket says 172.21.15.119:50704) 2024-09-13T16:29:40.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85864ea700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f85780df8e0 0x7f857801c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f8585ce9700 1 -- 172.21.15.119:0/3287934676 learned_addr learned my addr 172.21.15.119:0/3287934676 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f8586ceb700 1 --2- 172.21.15.119:0/3287934676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f8578005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85854e8700 1 -- 172.21.15.119:0/3287934676 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 7801814 0 0) 0x7f85780e3310 con 0x7f857804fe30 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85854e8700 1 -- 172.21.15.119:0/3287934676 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f856c0035d0 con 0x7f857804fe30 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85864ea700 1 -- 172.21.15.119:0/3287934676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 msgr2=0x7f8578005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85864ea700 1 --2- 172.21.15.119:0/3287934676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f8578005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85864ea700 1 -- 172.21.15.119:0/3287934676 >> v1:172.21.15.119:6789/0 conn(0x7f857804fe30 legacy=0x7f8578050480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85864ea700 1 -- 172.21.15.119:0/3287934676 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85780e3bb0 con 0x7f85780df8e0 2024-09-13T16:29:40.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85864ea700 1 --2- 172.21.15.119:0/3287934676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85780df8e0 0x7f857801c5d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f857c009840 tx=0x7f857c009b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44c2acbc7eabb11e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85854e8700 1 -- 172.21.15.119:0/3287934676 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c011830 con 0x7f85780df8e0 2024-09-13T16:29:40.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85854e8700 1 -- 172.21.15.119:0/3287934676 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857c011990 con 0x7f85780df8e0 2024-09-13T16:29:40.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85854e8700 1 -- 172.21.15.119:0/3287934676 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c011d20 con 0x7f85780df8e0 2024-09-13T16:29:40.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- 172.21.15.119:0/3287934676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85780df8e0 msgr2=0x7f857801c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 --2- 172.21.15.119:0/3287934676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85780df8e0 0x7f857801c5d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f857c009840 tx=0x7f857c009b50 comp rx=0 tx=0).stop 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- 172.21.15.119:0/3287934676 shutdown_connections 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 --2- 172.21.15.119:0/3287934676 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f8578005b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 --2- 172.21.15.119:0/3287934676 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85780df8e0 0x7f857801c5d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- 172.21.15.119:0/3287934676 >> 172.21.15.119:0/3287934676 conn(0x7f8578036140 msgr2=0x7f8578036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- 172.21.15.119:0/3287934676 shutdown_connections 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.475+0000 7f85874ec700 1 -- 172.21.15.119:0/3287934676 wait complete. 2024-09-13T16:29:40.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 Processor -- start 2024-09-13T16:29:40.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 -- start start 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f857804fe30 0x7f8578031b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f857817bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85780df8e0 0x7f85780dce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857817cd10 con 0x7f857804fe30 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8578178730 con 0x7f85780df8e0 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85874ec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f857817c260 con 0x7f8578050b90 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85864ea700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f857804fe30 0x7f8578031b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85864ea700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f857804fe30 0x7f8578031b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46144/0 (socket says 172.21.15.119:46144) 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f85864ea700 1 -- 172.21.15.119:0/2465178772 learned_addr learned my addr 172.21.15.119:0/2465178772 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f8585ce9700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f857817bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.479+0000 7f8586ceb700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85780df8e0 0x7f85780dce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85864ea700 1 -- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 msgr2=0x7f857817bd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85864ea700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f857817bd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85864ea700 1 -- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85780df8e0 msgr2=0x7f85780dce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85864ea700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85780df8e0 0x7f85780dce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85864ea700 1 -- 172.21.15.119:0/2465178772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857c009450 con 0x7f857804fe30 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f8586ceb700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85780df8e0 0x7f85780dce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85864ea700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f857804fe30 0x7f8578031b30 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f857c009b90 tx=0x7f857c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f8576ffd700 1 -- 172.21.15.119:0/2465178772 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c0044d0 con 0x7f857804fe30 2024-09-13T16:29:40.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f8576ffd700 1 -- 172.21.15.119:0/2465178772 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f857c01e070 con 0x7f857804fe30 2024-09-13T16:29:40.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85781789d0 con 0x7f857804fe30 2024-09-13T16:29:40.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f8576ffd700 1 -- 172.21.15.119:0/2465178772 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f857c0038c0 con 0x7f857804fe30 2024-09-13T16:29:40.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857817c750 con 0x7f857804fe30 2024-09-13T16:29:40.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f8576ffd700 1 -- 172.21.15.119:0/2465178772 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f857c003a20 con 0x7f857804fe30 2024-09-13T16:29:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85780e3310 con 0x7f857804fe30 2024-09-13T16:29:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f8576ffd700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8564066730 0x7f8564068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.483+0000 7f8576ffd700 1 -- 172.21.15.119:0/2465178772 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f857c085000 con 0x7f857804fe30 2024-09-13T16:29:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.487+0000 7f8585ce9700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8564066730 0x7f8564068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.487+0000 7f8585ce9700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8564066730 0x7f8564068bf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8578000c70 tx=0x7f8570007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.491+0000 7f8576ffd700 1 -- 172.21.15.119:0/2465178772 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f857c0522c0 con 0x7f857804fe30 2024-09-13T16:29:40.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.671+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8578036e40 con 0x7f857804fe30 2024-09-13T16:29:40.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.671+0000 7f8576ffd700 1 -- 172.21.15.119:0/2465178772 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16837 (secure 0 0 0) 0x7f857c0558e0 con 0x7f857804fe30 2024-09-13T16:29:40.679 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:40.679 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":82,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:31.230578+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:40.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8564066730 msgr2=0x7f8564068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8564066730 0x7f8564068bf0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f8578000c70 tx=0x7f8570007de0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f857804fe30 msgr2=0x7f8578031b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f857804fe30 0x7f8578031b30 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f857c009b90 tx=0x7f857c004770 comp rx=0 tx=0).stop 2024-09-13T16:29:40.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 shutdown_connections 2024-09-13T16:29:40.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8564066730 0x7f8564068bf0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85780df8e0 0x7f85780dce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8578050b90 0x7f857817bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 --2- 172.21.15.119:0/2465178772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f857804fe30 0x7f8578031b30 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 >> 172.21.15.119:0/2465178772 conn(0x7f8578036140 msgr2=0x7f85780340b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:40.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 shutdown_connections 2024-09-13T16:29:40.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.675+0000 7f85874ec700 1 -- 172.21.15.119:0/2465178772 wait complete. 2024-09-13T16:29:40.694 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:29:28.552550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:29:29.210253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:40.695 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:40.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.807+0000 7f7381c4a700 1 Processor -- start 2024-09-13T16:29:40.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.807+0000 7f7381c4a700 1 -- start start 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0b31e0 0x7f737c0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f737c0cd750 0x7f737c0cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c15af90 con 0x7f737c15abb0 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737c0cdc00 con 0x7f737c0cd750 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0cb6d0 con 0x7f737c0b31e0 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f737c0cd750 0x7f737c0cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f737c0cd750 0x7f737c0cbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46148/0 (socket says 172.21.15.119:46148) 2024-09-13T16:29:40.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 -- 172.21.15.119:0/1773932217 learned_addr learned my addr 172.21.15.119:0/1773932217 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:40.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f737bfff700 1 --2- 172.21.15.119:0/1773932217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0b31e0 0x7f737c0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 -- 172.21.15.119:0/1773932217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0b31e0 msgr2=0x7f737c0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:40.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 --2- 172.21.15.119:0/1773932217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0b31e0 0x7f737c0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 -- 172.21.15.119:0/1773932217 >> v1:172.21.15.145:6789/0 conn(0x7f737c15abb0 legacy=0x7f737c0cf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 -- 172.21.15.119:0/1773932217 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c0d0060 con 0x7f737c0cd750 2024-09-13T16:29:40.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381449700 1 --2- 172.21.15.119:0/1773932217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0cd750 0x7f737c0cbb60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7374009e50 tx=0x7f737400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a8f2ea9f245dea1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f737b7fe700 1 -- 172.21.15.119:0/1773932217 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737400e070 con 0x7f737c0cd750 2024-09-13T16:29:40.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f737b7fe700 1 -- 172.21.15.119:0/1773932217 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737400b910 con 0x7f737c0cd750 2024-09-13T16:29:40.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f737b7fe700 1 -- 172.21.15.119:0/1773932217 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737400bcc0 con 0x7f737c0cd750 2024-09-13T16:29:40.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- 172.21.15.119:0/1773932217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0cd750 msgr2=0x7f737c0cbb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 --2- 172.21.15.119:0/1773932217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0cd750 0x7f737c0cbb60 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f7374009e50 tx=0x7f737400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:40.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- 172.21.15.119:0/1773932217 shutdown_connections 2024-09-13T16:29:40.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 --2- 172.21.15.119:0/1773932217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0cd750 0x7f737c0cbb60 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 --2- 172.21.15.119:0/1773932217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0b31e0 0x7f737c0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- 172.21.15.119:0/1773932217 >> 172.21.15.119:0/1773932217 conn(0x7f737c0b25e0 msgr2=0x7f737c0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:40.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- 172.21.15.119:0/1773932217 shutdown_connections 2024-09-13T16:29:40.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.811+0000 7f7381c4a700 1 -- 172.21.15.119:0/1773932217 wait complete. 2024-09-13T16:29:40.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.815+0000 7f7381c4a700 1 Processor -- start 2024-09-13T16:29:40.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.815+0000 7f7381c4a700 1 -- start start 2024-09-13T16:29:40.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0b31e0 0x7f737c15dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0cd750 0x7f737c15bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f737c15abb0 0x7f737c1588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c15b920 con 0x7f737c0b31e0 2024-09-13T16:29:40.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c1f1250 con 0x7f737c15abb0 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0caaa0 con 0x7f737c0cd750 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0b31e0 0x7f737c15dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0b31e0 0x7f737c15dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46162/0 (socket says 172.21.15.119:46162) 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 -- 172.21.15.119:0/600999744 learned_addr learned my addr 172.21.15.119:0/600999744 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f737bfff700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0cd750 0x7f737c15bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381449700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f737c15abb0 0x7f737c1588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 -- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0cd750 msgr2=0x7f737c15bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0cd750 0x7f737c15bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 -- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f737c15abb0 msgr2=0x7f737c1588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f737c15abb0 0x7f737c1588d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 -- 172.21.15.119:0/600999744 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7374009b30 con 0x7f737c0b31e0 2024-09-13T16:29:40.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7380c48700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0b31e0 0x7f737c15dfb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f736c00c470 tx=0x7f736c00c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f73797fa700 1 -- 172.21.15.119:0/600999744 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c0050d0 con 0x7f737c0b31e0 2024-09-13T16:29:40.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f73797fa700 1 -- 172.21.15.119:0/600999744 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c005230 con 0x7f737c0b31e0 2024-09-13T16:29:40.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f73797fa700 1 -- 172.21.15.119:0/600999744 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c006880 con 0x7f737c0b31e0 2024-09-13T16:29:40.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737c1f0f70 con 0x7f737c0b31e0 2024-09-13T16:29:40.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737c15b540 con 0x7f737c0b31e0 2024-09-13T16:29:40.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.819+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737c15ba60 con 0x7f737c0b31e0 2024-09-13T16:29:40.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.823+0000 7f73797fa700 1 -- 172.21.15.119:0/600999744 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f736c0036a0 con 0x7f737c0b31e0 2024-09-13T16:29:40.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.823+0000 7f73797fa700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f735c066730 0x7f735c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:40.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.823+0000 7f73797fa700 1 -- 172.21.15.119:0/600999744 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f736c083200 con 0x7f737c0b31e0 2024-09-13T16:29:40.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.823+0000 7f737bfff700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f735c066730 0x7f735c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:40.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.823+0000 7f737bfff700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f735c066730 0x7f735c068bf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f737c159720 tx=0x7f7364009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:40.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.827+0000 7f73797fa700 1 -- 172.21.15.119:0/600999744 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f737c15ba60 con 0x7f737c0b31e0 2024-09-13T16:29:41.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.995+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f737c15af90 con 0x7f737c0b31e0 2024-09-13T16:29:41.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f73797fa700 1 -- 172.21.15.119:0/600999744 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f737c15af90 con 0x7f737c0b31e0 2024-09-13T16:29:41.003 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:41.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f735c066730 msgr2=0x7f735c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f735c066730 0x7f735c068bf0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f737c159720 tx=0x7f7364009240 comp rx=0 tx=0).stop 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0b31e0 msgr2=0x7f737c15dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0b31e0 0x7f737c15dfb0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f736c00c470 tx=0x7f736c00c780 comp rx=0 tx=0).stop 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 shutdown_connections 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f735c066730 0x7f735c068bf0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f737c15abb0 0x7f737c1588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f737c0cd750 0x7f737c15bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 --2- 172.21.15.119:0/600999744 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f737c0b31e0 0x7f737c15dfb0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 >> 172.21.15.119:0/600999744 conn(0x7f737c0b25e0 msgr2=0x7f737c0c64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 shutdown_connections 2024-09-13T16:29:41.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:40.999+0000 7f7381c4a700 1 -- 172.21.15.119:0/600999744 wait complete. 2024-09-13T16:29:41.017 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:29:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc595073700 1 Processor -- start 2024-09-13T16:29:41.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc595073700 1 -- start start 2024-09-13T16:29:41.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc595073700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc5900adf60 0x7fc5900b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc595073700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5900ae340 con 0x7fc590159ca0 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc595073700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc59015b1d0 con 0x7fc59015a110 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc595073700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc59015ae40 con 0x7fc5900adf60 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58e59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc590159ca0 0x7fc590070630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50720/0 (socket says 172.21.15.119:50720) 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58e59c700 1 -- 172.21.15.119:0/3431208768 learned_addr learned my addr 172.21.15.119:0/3431208768 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58ed9d700 1 --2- 172.21.15.119:0/3431208768 >> v2:172.21.15.119:3300/0 conn(0x7fc5900adf60 0x7fc5900b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58dd9b700 1 -- 172.21.15.119:0/3431208768 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3111223123 0 0) 0x7fc5900ae340 con 0x7fc590159ca0 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58dd9b700 1 -- 172.21.15.119:0/3431208768 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5780035d0 con 0x7fc590159ca0 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58ed9d700 1 -- 172.21.15.119:0/3431208768 >> v1:172.21.15.145:6789/0 conn(0x7fc59015a110 legacy=0x7fc59015b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58ed9d700 1 -- 172.21.15.119:0/3431208768 >> v1:172.21.15.119:6789/0 conn(0x7fc590159ca0 legacy=0x7fc590070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.127+0000 7fc58ed9d700 1 -- 172.21.15.119:0/3431208768 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc59015a5a0 con 0x7fc5900adf60 2024-09-13T16:29:41.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58ed9d700 1 --2- 172.21.15.119:0/3431208768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5900b3450 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc584009850 tx=0x7fc584009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d6a65cdf2dc448f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58dd9b700 1 -- 172.21.15.119:0/3431208768 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584011830 con 0x7fc5900adf60 2024-09-13T16:29:41.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58dd9b700 1 -- 172.21.15.119:0/3431208768 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc584011990 con 0x7fc5900adf60 2024-09-13T16:29:41.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58dd9b700 1 -- 172.21.15.119:0/3431208768 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc584011d40 con 0x7fc5900adf60 2024-09-13T16:29:41.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- 172.21.15.119:0/3431208768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 msgr2=0x7fc5900b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 --2- 172.21.15.119:0/3431208768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5900b3450 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fc584009850 tx=0x7fc584009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- 172.21.15.119:0/3431208768 shutdown_connections 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 --2- 172.21.15.119:0/3431208768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5900b3450 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- 172.21.15.119:0/3431208768 >> 172.21.15.119:0/3431208768 conn(0x7fc5900b0560 msgr2=0x7fc5900b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- 172.21.15.119:0/3431208768 shutdown_connections 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- 172.21.15.119:0/3431208768 wait complete. 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 Processor -- start 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- start start 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5901e3050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc590159ca0 0x7fc590156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc590156540 0x7fc5900ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5901569f0 con 0x7fc5900adf60 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5900ce940 con 0x7fc590156540 2024-09-13T16:29:41.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc595073700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5901e3590 con 0x7fc590159ca0 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5901e3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5901e3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46178/0 (socket says 172.21.15.119:46178) 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58ed9d700 1 -- 172.21.15.119:0/2950268361 learned_addr learned my addr 172.21.15.119:0/2950268361 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.131+0000 7fc58e59c700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc590159ca0 0x7fc590156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc58f59e700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc590156540 0x7fc5900ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc58ed9d700 1 -- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc590159ca0 msgr2=0x7fc590156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc58ed9d700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc590159ca0 0x7fc590156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc58ed9d700 1 -- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc590156540 msgr2=0x7fc5900ccf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc58ed9d700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc590156540 0x7fc5900ccf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc58ed9d700 1 -- 172.21.15.119:0/2950268361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5840094b0 con 0x7fc5900adf60 2024-09-13T16:29:41.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc58ed9d700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5901e3050 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc5840090b0 tx=0x7fc584004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc5777fe700 1 -- 172.21.15.119:0/2950268361 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58401a990 con 0x7fc5900adf60 2024-09-13T16:29:41.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc5777fe700 1 -- 172.21.15.119:0/2950268361 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc58401f070 con 0x7fc5900adf60 2024-09-13T16:29:41.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5900cdcc0 con 0x7fc5900adf60 2024-09-13T16:29:41.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc5777fe700 1 -- 172.21.15.119:0/2950268361 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc58401ad30 con 0x7fc5900adf60 2024-09-13T16:29:41.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5900ce280 con 0x7fc5900adf60 2024-09-13T16:29:41.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5900b3450 con 0x7fc5900adf60 2024-09-13T16:29:41.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.135+0000 7fc5777fe700 1 -- 172.21.15.119:0/2950268361 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc58401b430 con 0x7fc5900adf60 2024-09-13T16:29:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.139+0000 7fc5777fe700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc57c066730 0x7fc57c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.139+0000 7fc5777fe700 1 -- 172.21.15.119:0/2950268361 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc584085310 con 0x7fc5900adf60 2024-09-13T16:29:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.139+0000 7fc58e59c700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc57c066730 0x7fc57c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.139+0000 7fc58e59c700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc57c066730 0x7fc57c068bf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc590156c60 tx=0x7fc580007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.143+0000 7fc5777fe700 1 -- 172.21.15.119:0/2950268361 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc584052550 con 0x7fc5900adf60 2024-09-13T16:29:41.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc59015d140 con 0x7fc5900adf60 2024-09-13T16:29:41.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc5777fe700 1 -- 172.21.15.119:0/2950268361 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v82) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc584006ce0 con 0x7fc5900adf60 2024-09-13T16:29:41.317 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:29:41.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc57c066730 msgr2=0x7fc57c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc57c066730 0x7fc57c068bf0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc590156c60 tx=0x7fc580007de0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 msgr2=0x7fc5901e3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5901e3050 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fc5840090b0 tx=0x7fc584004770 comp rx=0 tx=0).stop 2024-09-13T16:29:41.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 shutdown_connections 2024-09-13T16:29:41.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc57c066730 0x7fc57c068bf0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc590156540 0x7fc5900ccf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc590159ca0 0x7fc590156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 --2- 172.21.15.119:0/2950268361 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5900adf60 0x7fc5901e3050 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 >> 172.21.15.119:0/2950268361 conn(0x7fc5900b0560 msgr2=0x7fc59015bde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:41.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.311+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 shutdown_connections 2024-09-13T16:29:41.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.315+0000 7fc595073700 1 -- 172.21.15.119:0/2950268361 wait complete. 2024-09-13T16:29:41.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:29:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 Processor -- start 2024-09-13T16:29:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 -- start start 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f59840d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f59840ce0d0 0x7f59840ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f59840cf220 0x7f59840cf6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59840cfc10 con 0x7f59840ce0d0 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59840d05b0 con 0x7f59840cf220 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f598963f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840d0870 con 0x7f598415cad0 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f59840ce0d0 0x7f59840ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f59840ce0d0 0x7f59840ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46184/0 (socket says 172.21.15.119:46184) 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 -- 172.21.15.119:0/3371737572 learned_addr learned my addr 172.21.15.119:0/3371737572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f5983fff700 1 --2- 172.21.15.119:0/3371737572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f59840d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f5988e3e700 1 --2- 172.21.15.119:0/3371737572 >> v2:172.21.15.145:3300/0 conn(0x7f59840cf220 0x7f59840cf6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 -- 172.21.15.119:0/3371737572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 msgr2=0x7f59840d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 --2- 172.21.15.119:0/3371737572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f59840d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 -- 172.21.15.119:0/3371737572 >> v2:172.21.15.145:3300/0 conn(0x7f59840cf220 msgr2=0x7f59840cf6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 --2- 172.21.15.119:0/3371737572 >> v2:172.21.15.145:3300/0 conn(0x7f59840cf220 0x7f59840cf6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 -- 172.21.15.119:0/3371737572 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59840d0b50 con 0x7f59840ce0d0 2024-09-13T16:29:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f59837fe700 1 --2- 172.21.15.119:0/3371737572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840ce0d0 0x7f59840ce510 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f59840d2790 tx=0x7f5974009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e35e1e0ce298d078 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f5982ffd700 1 -- 172.21.15.119:0/3371737572 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974011830 con 0x7f59840ce0d0 2024-09-13T16:29:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f5982ffd700 1 -- 172.21.15.119:0/3371737572 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5974011990 con 0x7f59840ce0d0 2024-09-13T16:29:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.439+0000 7f5982ffd700 1 -- 172.21.15.119:0/3371737572 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974011d20 con 0x7f59840ce0d0 2024-09-13T16:29:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- 172.21.15.119:0/3371737572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840ce0d0 msgr2=0x7f59840ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 --2- 172.21.15.119:0/3371737572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840ce0d0 0x7f59840ce510 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f59840d2790 tx=0x7f5974009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- 172.21.15.119:0/3371737572 shutdown_connections 2024-09-13T16:29:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 --2- 172.21.15.119:0/3371737572 >> v2:172.21.15.145:3300/0 conn(0x7f59840cf220 0x7f59840cf6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 --2- 172.21.15.119:0/3371737572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840ce0d0 0x7f59840ce510 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 --2- 172.21.15.119:0/3371737572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f59840d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- 172.21.15.119:0/3371737572 >> 172.21.15.119:0/3371737572 conn(0x7f598408bf70 msgr2=0x7f59840d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- 172.21.15.119:0/3371737572 shutdown_connections 2024-09-13T16:29:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- 172.21.15.119:0/3371737572 wait complete. 2024-09-13T16:29:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 Processor -- start 2024-09-13T16:29:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- start start 2024-09-13T16:29:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ce0d0 0x7f598415b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840cf220 0x7f598409c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f598409e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840cc750 con 0x7f59840cf220 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59840cb6b0 con 0x7f59840ce0d0 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f598963f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59840cb990 con 0x7f598415cad0 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f59837fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840cf220 0x7f598409c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f59837fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840cf220 0x7f598409c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46188/0 (socket says 172.21.15.119:46188) 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f59837fe700 1 -- 172.21.15.119:0/341281948 learned_addr learned my addr 172.21.15.119:0/341281948 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.443+0000 7f5988e3e700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f598409e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f5983fff700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ce0d0 0x7f598415b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f59837fe700 1 -- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 msgr2=0x7f598409e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f59837fe700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f598409e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f59837fe700 1 -- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ce0d0 msgr2=0x7f598415b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f59837fe700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ce0d0 0x7f598415b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f59837fe700 1 -- 172.21.15.119:0/341281948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5974009570 con 0x7f59840cf220 2024-09-13T16:29:41.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f5983fff700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ce0d0 0x7f598415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f59837fe700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840cf220 0x7f598409c660 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f5974013000 tx=0x7f5974004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f5980ff9700 1 -- 172.21.15.119:0/341281948 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5974004240 con 0x7f59840cf220 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f5980ff9700 1 -- 172.21.15.119:0/341281948 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f597401f070 con 0x7f59840cf220 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59840cbc70 con 0x7f59840cf220 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f5980ff9700 1 -- 172.21.15.119:0/341281948 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59740045e0 con 0x7f59840cf220 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59840cc230 con 0x7f59840cf220 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59840ce510 con 0x7f59840cf220 2024-09-13T16:29:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.447+0000 7f5980ff9700 1 -- 172.21.15.119:0/341281948 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5974021030 con 0x7f59840cf220 2024-09-13T16:29:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.451+0000 7f5980ff9700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5964066800 0x7f5964068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.451+0000 7f5980ff9700 1 -- 172.21.15.119:0/341281948 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f59740852b0 con 0x7f59840cf220 2024-09-13T16:29:41.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.451+0000 7f5983fff700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5964066800 0x7f5964068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.451+0000 7f5983fff700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5964066800 0x7f5964068cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f59840b8560 tx=0x7f596c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.459+0000 7f5980ff9700 1 -- 172.21.15.119:0/341281948 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5974052610 con 0x7f59840cf220 2024-09-13T16:29:41.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.627+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f59840d32f0 con 0x7f59840cf220 2024-09-13T16:29:41.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.627+0000 7f5980ff9700 1 -- 172.21.15.119:0/341281948 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v82) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5974015020 con 0x7f59840cf220 2024-09-13T16:29:41.633 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:29:41.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5964066800 msgr2=0x7f5964068cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5964066800 0x7f5964068cc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f59840b8560 tx=0x7f596c008040 comp rx=0 tx=0).stop 2024-09-13T16:29:41.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840cf220 msgr2=0x7f598409c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840cf220 0x7f598409c660 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f5974013000 tx=0x7f5974004770 comp rx=0 tx=0).stop 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 shutdown_connections 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5964066800 0x7f5964068cc0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f598415cad0 0x7f598409e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59840cf220 0x7f598409c660 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 --2- 172.21.15.119:0/341281948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59840ce0d0 0x7f598415b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 >> 172.21.15.119:0/341281948 conn(0x7f598408bf70 msgr2=0x7f59840d16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 shutdown_connections 2024-09-13T16:29:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.631+0000 7f598963f700 1 -- 172.21.15.119:0/341281948 wait complete. 2024-09-13T16:29:41.649 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:29:41.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ee960700 1 Processor -- start 2024-09-13T16:29:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ee960700 1 -- start start 2024-09-13T16:29:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ee960700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe7e81440d0 0x7fe7e815b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ee960700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe7e81444b0 0x7fe7e806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ee960700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e806bd60 con 0x7fe7e80cbb10 2024-09-13T16:29:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ee960700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e815bf10 con 0x7fe7e81440d0 2024-09-13T16:29:41.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ee960700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e806bfe0 con 0x7fe7e81444b0 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe7e81440d0 0x7fe7e815b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ecefd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe7e80cbb10 0x7fe7e80b1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:50732/0 (socket says 172.21.15.119:50732) 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7ecefd700 1 -- 172.21.15.119:0/2328793029 learned_addr learned my addr 172.21.15.119:0/2328793029 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7df7fe700 1 --2- 172.21.15.119:0/2328793029 >> v2:172.21.15.145:3300/0 conn(0x7fe7e81444b0 0x7fe7e806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e7fff700 1 -- 172.21.15.119:0/2328793029 >> v2:172.21.15.145:3300/0 conn(0x7fe7e81444b0 msgr2=0x7fe7e806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e7fff700 1 --2- 172.21.15.119:0/2328793029 >> v2:172.21.15.145:3300/0 conn(0x7fe7e81444b0 0x7fe7e806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e7fff700 1 -- 172.21.15.119:0/2328793029 >> v1:172.21.15.119:6789/0 conn(0x7fe7e80cbb10 legacy=0x7fe7e80b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e77fe700 1 -- 172.21.15.119:0/2328793029 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593012737 0 0) 0x7fe7e806bd60 con 0x7fe7e80cbb10 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e7fff700 1 -- 172.21.15.119:0/2328793029 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e806c2c0 con 0x7fe7e81440d0 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e7fff700 1 --2- 172.21.15.119:0/2328793029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e81440d0 0x7fe7e815b9d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80b4030 tx=0x7fe7d8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80784d51b2595d61 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e77fe700 1 -- 172.21.15.119:0/2328793029 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8011830 con 0x7fe7e81440d0 2024-09-13T16:29:41.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e77fe700 1 -- 172.21.15.119:0/2328793029 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d8011990 con 0x7fe7e81440d0 2024-09-13T16:29:41.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.755+0000 7fe7e77fe700 1 -- 172.21.15.119:0/2328793029 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8011d20 con 0x7fe7e81440d0 2024-09-13T16:29:41.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- 172.21.15.119:0/2328793029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e81440d0 msgr2=0x7fe7e815b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 --2- 172.21.15.119:0/2328793029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e81440d0 0x7fe7e815b9d0 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80b4030 tx=0x7fe7d8009b30 comp rx=0 tx=0).stop 2024-09-13T16:29:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- 172.21.15.119:0/2328793029 shutdown_connections 2024-09-13T16:29:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 --2- 172.21.15.119:0/2328793029 >> v2:172.21.15.145:3300/0 conn(0x7fe7e81444b0 0x7fe7e806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 --2- 172.21.15.119:0/2328793029 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e81440d0 0x7fe7e815b9d0 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- 172.21.15.119:0/2328793029 >> 172.21.15.119:0/2328793029 conn(0x7fe7e806a350 msgr2=0x7fe7e806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- 172.21.15.119:0/2328793029 shutdown_connections 2024-09-13T16:29:41.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- 172.21.15.119:0/2328793029 wait complete. 2024-09-13T16:29:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 Processor -- start 2024-09-13T16:29:41.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- start start 2024-09-13T16:29:41.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e80cbb10 0x7fe7e815a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7e81440d0 0x7fe7e81f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e81444b0 0x7fe7e813f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e815a970 con 0x7fe7e80cbb10 2024-09-13T16:29:41.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e8070790 con 0x7fe7e81444b0 2024-09-13T16:29:41.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7ee960700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f2f70 con 0x7fe7e81440d0 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7e7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e80cbb10 0x7fe7e815a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.759+0000 7fe7e7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e80cbb10 0x7fe7e815a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46220/0 (socket says 172.21.15.119:46220) 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e7fff700 1 -- 172.21.15.119:0/3107492893 learned_addr learned my addr 172.21.15.119:0/3107492893 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7ecefd700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e81444b0 0x7fe7e813f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7df7fe700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7e81440d0 0x7fe7e81f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e7fff700 1 -- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7e81440d0 msgr2=0x7fe7e81f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e7fff700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7e81440d0 0x7fe7e81f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e7fff700 1 -- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e81444b0 msgr2=0x7fe7e813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e7fff700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e81444b0 0x7fe7e813f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e7fff700 1 -- 172.21.15.119:0/3107492893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7d80094b0 con 0x7fe7e80cbb10 2024-09-13T16:29:41.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e7fff700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e80cbb10 0x7fe7e815a430 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009fd0 tx=0x7fe7d8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e57fa700 1 -- 172.21.15.119:0/3107492893 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80045b0 con 0x7fe7e80cbb10 2024-09-13T16:29:41.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e57fa700 1 -- 172.21.15.119:0/3107492893 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d801f070 con 0x7fe7e80cbb10 2024-09-13T16:29:41.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e813fce0 con 0x7fe7e80cbb10 2024-09-13T16:29:41.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e57fa700 1 -- 172.21.15.119:0/3107492893 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80038e0 con 0x7fe7e80cbb10 2024-09-13T16:29:41.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e8070c60 con 0x7fe7e80cbb10 2024-09-13T16:29:41.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e8070900 con 0x7fe7e80cbb10 2024-09-13T16:29:41.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e57fa700 1 -- 172.21.15.119:0/3107492893 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe7d8021030 con 0x7fe7e80cbb10 2024-09-13T16:29:41.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e57fa700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7c8066730 0x7fe7c8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:41.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7e57fa700 1 -- 172.21.15.119:0/3107492893 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe7d8085630 con 0x7fe7e80cbb10 2024-09-13T16:29:41.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.763+0000 7fe7df7fe700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7c8066730 0x7fe7c8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:41.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.767+0000 7fe7df7fe700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7c8066730 0x7fe7c8068bf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe7e815dd40 tx=0x7fe7d0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:41.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.767+0000 7fe7e57fa700 1 -- 172.21.15.119:0/3107492893 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe7d8052a10 con 0x7fe7e80cbb10 2024-09-13T16:29:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.947+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe7e813fa90 con 0x7fe7e80cbb10 2024-09-13T16:29:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.947+0000 7fe7e57fa700 1 -- 172.21.15.119:0/3107492893 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v82) v1 ==== 80+0+0 (secure 0 0 0) 0x7fe7d8056030 con 0x7fe7e80cbb10 2024-09-13T16:29:41.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7c8066730 msgr2=0x7fe7c8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7c8066730 0x7fe7c8068bf0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fe7e815dd40 tx=0x7fe7d0008040 comp rx=0 tx=0).stop 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e80cbb10 msgr2=0x7fe7e815a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e80cbb10 0x7fe7e815a430 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009fd0 tx=0x7fe7d8004770 comp rx=0 tx=0).stop 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 shutdown_connections 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe7c8066730 0x7fe7c8068bf0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7e81444b0 0x7fe7e813f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe7e81440d0 0x7fe7e81f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 --2- 172.21.15.119:0/3107492893 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe7e80cbb10 0x7fe7e815a430 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 >> 172.21.15.119:0/3107492893 conn(0x7fe7e806a350 msgr2=0x7fe7e80b35b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 shutdown_connections 2024-09-13T16:29:41.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:41.951+0000 7fe7ee960700 1 -- 172.21.15.119:0/3107492893 wait complete. 2024-09-13T16:29:41.959 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:29:41.970 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:29:41.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:41.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:41.983 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[44173]: fuse finished with error 0 and tester_r 0 2024-09-13T16:29:41.984 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:29:48.086 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:29:48.086 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:48.087 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:29:48.095 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:48.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714ffff700 1 Processor -- start 2024-09-13T16:29:48.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714ffff700 1 -- start start 2024-09-13T16:29:48.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714ffff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f71501604a0 0x7f71500b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714ffff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71500d0ee0 con 0x7f71500d1710 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714ffff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71500cf490 con 0x7f71500d1330 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714ffff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71500cf770 con 0x7f71501604a0 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714e7fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f71500d1710 0x7f715015fd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:36992/0 (socket says 172.21.15.119:36992) 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714e7fc700 1 -- 172.21.15.119:0/1903669978 learned_addr learned my addr 172.21.15.119:0/1903669978 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714f7fe700 1 --2- 172.21.15.119:0/1903669978 >> v2:172.21.15.145:3300/0 conn(0x7f71501604a0 0x7f71500b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3378045193 0 0) 0x7f71500d0ee0 con 0x7f71500d1710 2024-09-13T16:29:48.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f713c0035d0 con 0x7f71500d1710 2024-09-13T16:29:48.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 907436810 0 0) 0x7f713c0035d0 con 0x7f71500d1710 2024-09-13T16:29:48.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 >> v2:172.21.15.145:3300/0 conn(0x7f71501604a0 msgr2=0x7f71500b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:48.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 --2- 172.21.15.119:0/1903669978 >> v2:172.21.15.145:3300/0 conn(0x7f71501604a0 0x7f71500b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 >> v1:172.21.15.145:6789/0 conn(0x7f71500d1330 legacy=0x7f71500d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71500cfa50 con 0x7f71500d1710 2024-09-13T16:29:48.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f71400019f0 con 0x7f71500d1710 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 >> v1:172.21.15.119:6789/0 conn(0x7f71500d1710 legacy=0x7f715015fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c003a20 0x7f713c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c004870 0x7f713c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f713c005230 0x7f713c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71500cf770 con 0x7f713c003a20 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713c00c1d0 con 0x7f713c005230 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f713c00c390 con 0x7f713c004870 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714effd700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c003a20 0x7f713c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714e7fc700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c004870 0x7f713c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.211+0000 7f714f7fe700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f713c005230 0x7f713c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714effd700 1 -- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c004870 msgr2=0x7f713c004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:48.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714effd700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c004870 0x7f713c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714effd700 1 -- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f713c005230 msgr2=0x7f713c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714effd700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f713c005230 0x7f713c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714f7fe700 1 -- 172.21.15.119:0/1903669978 reap_dead start 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714effd700 1 -- 172.21.15.119:0/1903669978 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f713c0035d0 con 0x7f713c003a20 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714e7fc700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c004870 0x7f713c004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714effd700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c003a20 0x7f713c003e40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f71380017e0 tx=0x7f713c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d491d0b40ff5575 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71380052d0 con 0x7f713c003a20 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7138005450 con 0x7f713c003a20 2024-09-13T16:29:48.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714dffb700 1 -- 172.21.15.119:0/1903669978 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71380038e0 con 0x7f713c003a20 2024-09-13T16:29:48.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714ffff700 1 -- 172.21.15.119:0/1903669978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c003a20 msgr2=0x7f713c003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714ffff700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c003a20 0x7f713c003e40 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f71380017e0 tx=0x7f713c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714ffff700 1 -- 172.21.15.119:0/1903669978 shutdown_connections 2024-09-13T16:29:48.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714ffff700 1 --2- 172.21.15.119:0/1903669978 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f713c003a20 0x7f713c003e40 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714ffff700 1 -- 172.21.15.119:0/1903669978 >> 172.21.15.119:0/1903669978 conn(0x7f715009bd70 msgr2=0x7f71500d2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:48.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714ffff700 1 -- 172.21.15.119:0/1903669978 shutdown_connections 2024-09-13T16:29:48.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.215+0000 7f714ffff700 1 -- 172.21.15.119:0/1903669978 wait complete. 2024-09-13T16:29:48.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 Processor -- start 2024-09-13T16:29:48.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 -- start start 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c003a20 0x7f71500b6310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7150153b20 0x7f7150153fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71501544e0 0x7f7150154960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150154ea0 con 0x7f71501544e0 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7150153360 con 0x7f7150153b20 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71500b82f0 con 0x7f713c003a20 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71501544e0 0x7f7150154960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71501544e0 0x7f7150154960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45480/0 (socket says 172.21.15.119:45480) 2024-09-13T16:29:48.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714f7fe700 1 -- 172.21.15.119:0/1220825499 learned_addr learned my addr 172.21.15.119:0/1220825499 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714effd700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c003a20 0x7f71500b6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.219+0000 7f714e7fc700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7150153b20 0x7f7150153fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714f7fe700 1 -- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c003a20 msgr2=0x7f71500b6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714f7fe700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c003a20 0x7f71500b6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714f7fe700 1 -- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7150153b20 msgr2=0x7f7150153fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714f7fe700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7150153b20 0x7f7150153fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714f7fe700 1 -- 172.21.15.119:0/1220825499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71500b6b90 con 0x7f71501544e0 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714f7fe700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71501544e0 0x7f7150154960 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7150067330 tx=0x7f7148011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f71477fe700 1 -- 172.21.15.119:0/1220825499 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148004240 con 0x7f71501544e0 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f71477fe700 1 -- 172.21.15.119:0/1220825499 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71480043a0 con 0x7f71501544e0 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71500b6e70 con 0x7f71501544e0 2024-09-13T16:29:48.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f71477fe700 1 -- 172.21.15.119:0/1220825499 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71480036c0 con 0x7f71501544e0 2024-09-13T16:29:48.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7150153600 con 0x7f71501544e0 2024-09-13T16:29:48.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7150153820 con 0x7f71501544e0 2024-09-13T16:29:48.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f71477fe700 1 -- 172.21.15.119:0/1220825499 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7148003820 con 0x7f71501544e0 2024-09-13T16:29:48.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f71477fe700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7130066760 0x7f7130068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f71477fe700 1 -- 172.21.15.119:0/1220825499 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f71480842b0 con 0x7f71501544e0 2024-09-13T16:29:48.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.223+0000 7f714effd700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7130066760 0x7f7130068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.227+0000 7f714effd700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7130066760 0x7f7130068c20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7138005040 tx=0x7f7138009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:48.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.235+0000 7f71477fe700 1 -- 172.21.15.119:0/1220825499 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7148051460 con 0x7f71501544e0 2024-09-13T16:29:48.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.439+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71500d06d0 con 0x7f71501544e0 2024-09-13T16:29:48.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f71477fe700 1 -- 172.21.15.119:0/1220825499 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f7148054a80 con 0x7f71501544e0 2024-09-13T16:29:48.447 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:48.447 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":42,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":42,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:29:33.249934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7101},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7101":{"gid":7101,"name":"c","rank":0,"incarnation":38,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/491709455","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":491709455},{"type":"v1","addr":"172.21.15.119:6835","nonce":491709455}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7130066760 msgr2=0x7f7130068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7130066760 0x7f7130068c20 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f7138005040 tx=0x7f7138009030 comp rx=0 tx=0).stop 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71501544e0 msgr2=0x7f7150154960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71501544e0 0x7f7150154960 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f7150067330 tx=0x7f7148011bb0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 shutdown_connections 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7130066760 0x7f7130068c20 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f71501544e0 0x7f7150154960 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7150153b20 0x7f7150153fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 --2- 172.21.15.119:0/1220825499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f713c003a20 0x7f71500b6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 >> 172.21.15.119:0/1220825499 conn(0x7f715009bd70 msgr2=0x7f71500ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:48.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 shutdown_connections 2024-09-13T16:29:48.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.443+0000 7f714ffff700 1 -- 172.21.15.119:0/1220825499 wait complete. 2024-09-13T16:29:48.451 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 42 2024-09-13T16:29:48.466 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-13T16:29:48.599 INFO:teuthology.orchestra.run.smithi119.stdout:{"message":"","return_code":0} 2024-09-13T16:29:48.599 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:29:48.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:29:48.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d1b6d700 1 Processor -- start 2024-09-13T16:29:48.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d1b6d700 1 -- start start 2024-09-13T16:29:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d1b6d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f98cc0cd8d0 0x7f98cc06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d1b6d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98cc0b17f0 0x7f98cc0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d1b6d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0705d0 con 0x7f98cc0b2200 2024-09-13T16:29:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d1b6d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc06d0d0 con 0x7f98cc0b17f0 2024-09-13T16:29:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d1b6d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc06fca0 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98cbfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98cc0b17f0 0x7f98cc0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98cbfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98cc0b17f0 0x7f98cc0cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45482/0 (socket says 172.21.15.119:45482) 2024-09-13T16:29:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98cbfff700 1 -- 172.21.15.119:0/2899193629 learned_addr learned my addr 172.21.15.119:0/2899193629 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.707+0000 7f98d0b6b700 1 --2- 172.21.15.119:0/2899193629 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0cd8d0 0x7f98cc06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cbfff700 1 -- 172.21.15.119:0/2899193629 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0cd8d0 msgr2=0x7f98cc06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cbfff700 1 --2- 172.21.15.119:0/2899193629 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0cd8d0 0x7f98cc06cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cbfff700 1 -- 172.21.15.119:0/2899193629 >> v1:172.21.15.145:6789/0 conn(0x7f98cc0b2200 legacy=0x7f98cc070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cbfff700 1 -- 172.21.15.119:0/2899193629 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc06ff80 con 0x7f98cc0b17f0 2024-09-13T16:29:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cbfff700 1 --2- 172.21.15.119:0/2899193629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0b17f0 0x7f98cc0cb800 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f98cc071850 tx=0x7f98bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2078644efe534d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:48.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cb7fe700 1 -- 172.21.15.119:0/2899193629 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00e070 con 0x7f98cc0b17f0 2024-09-13T16:29:48.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cb7fe700 1 -- 172.21.15.119:0/2899193629 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc00b910 con 0x7f98cc0b17f0 2024-09-13T16:29:48.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98cb7fe700 1 -- 172.21.15.119:0/2899193629 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00bcc0 con 0x7f98cc0b17f0 2024-09-13T16:29:48.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 -- 172.21.15.119:0/2899193629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0b17f0 msgr2=0x7f98cc0cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/2899193629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0b17f0 0x7f98cc0cb800 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7f98cc071850 tx=0x7f98bc00b330 comp rx=0 tx=0).stop 2024-09-13T16:29:48.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 -- 172.21.15.119:0/2899193629 shutdown_connections 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/2899193629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0b17f0 0x7f98cc0cb800 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/2899193629 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0cd8d0 0x7f98cc06cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 -- 172.21.15.119:0/2899193629 >> 172.21.15.119:0/2899193629 conn(0x7f98cc06a350 msgr2=0x7f98cc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 -- 172.21.15.119:0/2899193629 shutdown_connections 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 -- 172.21.15.119:0/2899193629 wait complete. 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 Processor -- start 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.711+0000 7f98d1b6d700 1 -- start start 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc0b17f0 0x7f98cc0acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b2200 0x7f98cc156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0cd8d0 0x7f98cc160500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0c3c60 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0c3f40 con 0x7f98cc0b2200 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0c4220 con 0x7f98cc0b17f0 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0cd8d0 0x7f98cc160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0cd8d0 0x7f98cc160500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45492/0 (socket says 172.21.15.119:45492) 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 -- 172.21.15.119:0/217948237 learned_addr learned my addr 172.21.15.119:0/217948237 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d0b6b700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc0b17f0 0x7f98cc0acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98cbfff700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b2200 0x7f98cc156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 -- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc0b17f0 msgr2=0x7f98cc0acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc0b17f0 0x7f98cc0acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 -- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b2200 msgr2=0x7f98cc156800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b2200 0x7f98cc156800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 -- 172.21.15.119:0/217948237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc009b30 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d0b6b700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc0b17f0 0x7f98cc0acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:48.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d136c700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0cd8d0 0x7f98cc160500 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c480 tx=0x7f98c400c790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:48.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98c97fa700 1 -- 172.21.15.119:0/217948237 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40052d0 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98c97fa700 1 -- 172.21.15.119:0/217948237 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c400ce90 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98c97fa700 1 -- 172.21.15.119:0/217948237 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c40036a0 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc0c4500 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.715+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc0aca90 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.719+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc0c4a20 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.719+0000 7f98c97fa700 1 -- 172.21.15.119:0/217948237 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f98c4003800 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.719+0000 7f98c97fa700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98ac066730 0x7f98ac068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:48.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.719+0000 7f98c97fa700 1 -- 172.21.15.119:0/217948237 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f98c4083560 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.719+0000 7f98d0b6b700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98ac066730 0x7f98ac068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:48.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.719+0000 7f98d0b6b700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98ac066730 0x7f98ac068bf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009f40 tx=0x7f98b4009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:48.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.723+0000 7f98c97fa700 1 -- 172.21.15.119:0/217948237 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f98cc0c4a20 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.935+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f98cc0c4a20 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.944 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:29:48.939+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:29:48.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.955+0000 7f98c97fa700 1 -- 172.21.15.119:0/217948237 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v43) v1 ==== 161+0+0 (secure 0 0 0) 0x7f98cc0c4a20 con 0x7f98cc0cd8d0 2024-09-13T16:29:48.962 INFO:tasks.ceph.mds.c.smithi119.stderr: -3307> 2024-09-13T16:29:36.120+0000 7f20f0bc6700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:29:48.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98ac066730 msgr2=0x7f98ac068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98ac066730 0x7f98ac068bf0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f98b4009f40 tx=0x7f98b4009240 comp rx=0 tx=0).stop 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0cd8d0 msgr2=0x7f98cc160500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0cd8d0 0x7f98cc160500 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7f98c400c480 tx=0x7f98c400c790 comp rx=0 tx=0).stop 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 shutdown_connections 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98ac066730 0x7f98ac068bf0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc0cd8d0 0x7f98cc160500 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b2200 0x7f98cc156800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 --2- 172.21.15.119:0/217948237 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc0b17f0 0x7f98cc0acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 >> 172.21.15.119:0/217948237 conn(0x7f98cc06a350 msgr2=0x7f98cc0adc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 shutdown_connections 2024-09-13T16:29:48.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:48.959+0000 7f98d1b6d700 1 -- 172.21.15.119:0/217948237 wait complete. 2024-09-13T16:29:48.966 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:29:48.980 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-13T16:29:48.993 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:29:49.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.015+0000 7f05b9493140 1 Processor -- start 2024-09-13T16:29:49.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.015+0000 7f05b9493140 1 -- start start 2024-09-13T16:29:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 0x564bfc99f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca33000 0x564bfca2fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca34410 0x564bfca2d200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bfc8ac8e0 con 0x564bfca33000 2024-09-13T16:29:49.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bfc8ac580 con 0x564bfca30c10 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x564bfc8828d0 con 0x564bfca34410 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca33000 0x564bfca2fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca33000 0x564bfca2fc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45530/0 (socket says 172.21.15.119:45530) 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 -- 172.21.15.119:0/10708391 learned_addr learned my addr 172.21.15.119:0/10708391 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b6fc8700 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 0x564bfc99f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b77c9700 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca34410 0x564bfca2d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 -- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca34410 msgr2=0x564bfca2d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca34410 0x564bfca2d200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 -- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 msgr2=0x564bfc99f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 0x564bfc99f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 -- 172.21.15.119:0/10708391 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a0005de0 con 0x564bfca33000 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b67c7700 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca33000 0x564bfca2fc60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f05a800a770 tx=0x7f05a800aa80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebf0bb42e36d2e85 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:49.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b57c5700 1 -- 172.21.15.119:0/10708391 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a800f070 con 0x564bfca33000 2024-09-13T16:29:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b57c5700 1 -- 172.21.15.119:0/10708391 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a80084c0 con 0x564bfca33000 2024-09-13T16:29:49.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b57c5700 1 -- 172.21.15.119:0/10708391 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05a8008830 con 0x564bfca33000 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 -- 172.21.15.119:0/10708391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca33000 msgr2=0x564bfca2fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca33000 0x564bfca2fc60 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f05a800a770 tx=0x7f05a800aa80 comp rx=0 tx=0).stop 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 -- 172.21.15.119:0/10708391 shutdown_connections 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca34410 0x564bfca2d200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca33000 0x564bfca2fc60 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 --2- 172.21.15.119:0/10708391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 0x564bfc99f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.019+0000 7f05b9493140 1 -- 172.21.15.119:0/10708391 >> 172.21.15.119:0/10708391 conn(0x564bfc98cfb0 msgr2=0x564bfc95ee80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 -- 172.21.15.119:0/10708391 shutdown_connections 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 -- 172.21.15.119:0/10708391 wait complete. 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 Processor -- start 2024-09-13T16:29:49.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 -- start start 2024-09-13T16:29:49.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 0x564bfcac5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca33000 0x564bfc9a2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca34410 0x564bfca2e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bfcac60d0 con 0x564bfca34410 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x564bfcac6610 con 0x564bfca30c10 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b9493140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x564bfcac68b0 con 0x564bfca33000 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca34410 0x564bfca2e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca34410 0x564bfca2e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45546/0 (socket says 172.21.15.119:45546) 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 -- 172.21.15.119:0/1547336606 learned_addr learned my addr 172.21.15.119:0/1547336606 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b67c7700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca33000 0x564bfc9a2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b6fc8700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 0x564bfcac5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 -- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca33000 msgr2=0x564bfc9a2150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x564bfca33000 0x564bfc9a2150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 -- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 msgr2=0x564bfcac5b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x564bfca30c10 0x564bfcac5b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 -- 172.21.15.119:0/1547336606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a800a420 con 0x564bfca34410 2024-09-13T16:29:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05b77c9700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca34410 0x564bfca2e1b0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f05b000f520 tx=0x7f05b000f830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.023+0000 7f05affff700 1 -- 172.21.15.119:0/1547336606 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b000d280 con 0x564bfca34410 2024-09-13T16:29:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05affff700 1 -- 172.21.15.119:0/1547336606 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05b000feb0 con 0x564bfca34410 2024-09-13T16:29:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x564bfc8b4ca0 con 0x564bfca34410 2024-09-13T16:29:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05affff700 1 -- 172.21.15.119:0/1547336606 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05b0013840 con 0x564bfca34410 2024-09-13T16:29:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564bfc89ab00 con 0x564bfca34410 2024-09-13T16:29:49.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05affff700 1 -- 172.21.15.119:0/1547336606 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f05b0013a00 con 0x564bfca34410 2024-09-13T16:29:49.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05affff700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0598067270 0x7f0598069730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05affff700 1 -- 172.21.15.119:0/1547336606 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f05b00832e0 con 0x564bfca34410 2024-09-13T16:29:49.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05b9493140 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x564bfcac7a40 0x564bfcac7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:1 11.10 11:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) v8 -- 0x564bfcad1870 con 0x564bfcac7a40 2024-09-13T16:29:49.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05b6fc8700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0598067270 0x7f0598069730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05b67c7700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x564bfcac7a40 0x564bfcac7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.027+0000 7f05b67c7700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x564bfcac7a40 0x564bfcac7e50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:49.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b6fc8700 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0598067270 0x7f0598069730 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f05a000a4f0 tx=0x7f05a000f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:49.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b67c7700 1 -- 172.21.15.119:0/1547336606 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v83'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f05a80089a0 con 0x564bfcac7a40 2024-09-13T16:29:49.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x564bfcac7a40 msgr2=0x564bfcac7e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b9493140 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x564bfcac7a40 0x564bfcac7e50 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0598067270 msgr2=0x7f0598069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b9493140 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0598067270 0x7f0598069730 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f05a000a4f0 tx=0x7f05a000f000 comp rx=0 tx=0).stop 2024-09-13T16:29:49.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca34410 msgr2=0x564bfca2e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.031+0000 7f05b9493140 1 --2- 172.21.15.119:0/1547336606 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x564bfca34410 0x564bfca2e1b0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f05b000f520 tx=0x7f05b000f830 comp rx=0 tx=0).stop 2024-09-13T16:29:49.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.035+0000 7f05b77c9700 1 -- 172.21.15.119:0/1547336606 reap_dead start 2024-09-13T16:29:49.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.035+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 shutdown_connections 2024-09-13T16:29:49.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.035+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 >> 172.21.15.119:0/1547336606 conn(0x564bfc98cfb0 msgr2=0x564bfca2d740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:49.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.035+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 shutdown_connections 2024-09-13T16:29:49.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.035+0000 7f05b9493140 1 -- 172.21.15.119:0/1547336606 wait complete. 2024-09-13T16:29:49.042 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-13T16:29:49.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.147+0000 7fedfe36b700 1 Processor -- start 2024-09-13T16:29:49.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.147+0000 7fedfe36b700 1 -- start start 2024-09-13T16:29:49.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfe36b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fedf81595e0 0x7fedf806e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfe36b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fedf81599c0 0x7fedf806b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfe36b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedf80710b0 con 0x7fedf806b8a0 2024-09-13T16:29:49.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfe36b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf806c3f0 con 0x7fedf81599c0 2024-09-13T16:29:49.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfe36b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedf806bd30 con 0x7fedf81595e0 2024-09-13T16:29:49.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfcb68700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fedf81599c0 0x7fedf806b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfdb6a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fedf806b8a0 0x7fedf815d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37024/0 (socket says 172.21.15.119:37024) 2024-09-13T16:29:49.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfdb6a700 1 -- 172.21.15.119:0/1990905894 learned_addr learned my addr 172.21.15.119:0/1990905894 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:49.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfd369700 1 --2- 172.21.15.119:0/1990905894 >> v2:172.21.15.145:3300/0 conn(0x7fedf81595e0 0x7fedf806e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedeffff700 1 -- 172.21.15.119:0/1990905894 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360021107 0 0) 0x7fedf80710b0 con 0x7fedf806b8a0 2024-09-13T16:29:49.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedeffff700 1 -- 172.21.15.119:0/1990905894 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fede40057e0 con 0x7fedf806b8a0 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfcb68700 1 -- 172.21.15.119:0/1990905894 >> v2:172.21.15.145:3300/0 conn(0x7fedf81595e0 msgr2=0x7fedf806e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfcb68700 1 --2- 172.21.15.119:0/1990905894 >> v2:172.21.15.145:3300/0 conn(0x7fedf81595e0 0x7fedf806e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfcb68700 1 -- 172.21.15.119:0/1990905894 >> v1:172.21.15.119:6789/0 conn(0x7fedf806b8a0 legacy=0x7fedf815d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfcb68700 1 -- 172.21.15.119:0/1990905894 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf8070810 con 0x7fedf81599c0 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedfcb68700 1 --2- 172.21.15.119:0/1990905894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf81599c0 0x7fedf806b360 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fedf8072ce0 tx=0x7fede8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a56a84495662d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedeffff700 1 -- 172.21.15.119:0/1990905894 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8011830 con 0x7fedf81599c0 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedeffff700 1 -- 172.21.15.119:0/1990905894 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fede8011990 con 0x7fedf81599c0 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.151+0000 7fedeffff700 1 -- 172.21.15.119:0/1990905894 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fede8011d20 con 0x7fedf81599c0 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- 172.21.15.119:0/1990905894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf81599c0 msgr2=0x7fedf806b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1990905894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf81599c0 0x7fedf806b360 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7fedf8072ce0 tx=0x7fede8009b30 comp rx=0 tx=0).stop 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- 172.21.15.119:0/1990905894 shutdown_connections 2024-09-13T16:29:49.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1990905894 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf81599c0 0x7fedf806b360 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1990905894 >> v2:172.21.15.145:3300/0 conn(0x7fedf81595e0 0x7fedf806e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- 172.21.15.119:0/1990905894 >> 172.21.15.119:0/1990905894 conn(0x7fedf80cb8f0 msgr2=0x7fedf80cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:49.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- 172.21.15.119:0/1990905894 shutdown_connections 2024-09-13T16:29:49.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- 172.21.15.119:0/1990905894 wait complete. 2024-09-13T16:29:49.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 Processor -- start 2024-09-13T16:29:49.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- start start 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf806b8a0 0x7fedf809b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedf81595e0 0x7fedf809c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedf81599c0 0x7fedf809c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf809beb0 con 0x7fedf806b8a0 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf81ee510 con 0x7fedf81595e0 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfe36b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf815add0 con 0x7fedf81599c0 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfd369700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf806b8a0 0x7fedf809b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfd369700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf806b8a0 0x7fedf809b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45560/0 (socket says 172.21.15.119:45560) 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.155+0000 7fedfd369700 1 -- 172.21.15.119:0/1064471783 learned_addr learned my addr 172.21.15.119:0/1064471783 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:49.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfcb68700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedf81595e0 0x7fedf809c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfdb6a700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedf81599c0 0x7fedf809c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfd369700 1 -- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedf81599c0 msgr2=0x7fedf809c930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfd369700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedf81599c0 0x7fedf809c930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfd369700 1 -- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedf81595e0 msgr2=0x7fedf809c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfd369700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedf81595e0 0x7fedf809c3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfd369700 1 -- 172.21.15.119:0/1064471783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fede80094b0 con 0x7fedf806b8a0 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfd369700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf806b8a0 0x7fedf809b970 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fedf400e9d0 tx=0x7fedf400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fededffb700 1 -- 172.21.15.119:0/1064471783 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf4009720 con 0x7fedf806b8a0 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fededffb700 1 -- 172.21.15.119:0/1064471783 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedf400ee70 con 0x7fedf806b8a0 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedf806edd0 con 0x7fedf806b8a0 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fededffb700 1 -- 172.21.15.119:0/1064471783 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf400ca50 con 0x7fedf806b8a0 2024-09-13T16:29:49.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedf815b390 con 0x7fedf806b8a0 2024-09-13T16:29:49.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedf8070430 con 0x7fedf806b8a0 2024-09-13T16:29:49.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fededffb700 1 -- 172.21.15.119:0/1064471783 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fedf400cbb0 con 0x7fedf806b8a0 2024-09-13T16:29:49.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.159+0000 7fededffb700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feddc066730 0x7feddc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:49.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.163+0000 7fededffb700 1 -- 172.21.15.119:0/1064471783 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fedf4084330 con 0x7fedf806b8a0 2024-09-13T16:29:49.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.163+0000 7fedfcb68700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feddc066730 0x7feddc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:49.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.163+0000 7fedfcb68700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feddc066730 0x7feddc068bf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fede8000c00 tx=0x7fede8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:49.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.171+0000 7fededffb700 1 -- 172.21.15.119:0/1064471783 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fedf4051500 con 0x7fedf806b8a0 2024-09-13T16:29:49.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.379+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7fedf80b25c0 con 0x7fedf806b8a0 2024-09-13T16:29:49.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.883+0000 7fededffb700 1 -- 172.21.15.119:0/1064471783 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v44) v1 ==== 162+0+0 (secure 0 0 0) 0x7fedf400ce60 con 0x7fedf806b8a0 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feddc066730 msgr2=0x7feddc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feddc066730 0x7feddc068bf0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fede8000c00 tx=0x7fede8008030 comp rx=0 tx=0).stop 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf806b8a0 msgr2=0x7fedf809b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf806b8a0 0x7fedf809b970 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7fedf400e9d0 tx=0x7fedf400ece0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 shutdown_connections 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feddc066730 0x7feddc068bf0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fedf81599c0 0x7fedf809c930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fedf81595e0 0x7fedf809c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 --2- 172.21.15.119:0/1064471783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fedf806b8a0 0x7fedf809b970 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 >> 172.21.15.119:0/1064471783 conn(0x7fedf80cb8f0 msgr2=0x7fedf80afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:49.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 shutdown_connections 2024-09-13T16:29:49.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:49.887+0000 7fedfe36b700 1 -- 172.21.15.119:0/1064471783 wait complete. 2024-09-13T16:29:49.896 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-13T16:29:49.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:50.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 Processor -- start 2024-09-13T16:29:50.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- start start 2024-09-13T16:29:50.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc03006b730 0x7fc0300cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc0300d2c90 0x7fc0300d30e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0300ceb60 con 0x7fc03015cd60 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc03015d140 con 0x7fc0300d2c90 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0300ce230 con 0x7fc03006b730 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc0300d2c90 0x7fc0300d30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc0300d2c90 0x7fc0300d30e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45574/0 (socket says 172.21.15.119:45574) 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 -- 172.21.15.119:0/2791286374 learned_addr learned my addr 172.21.15.119:0/2791286374 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc035ae5700 1 --2- 172.21.15.119:0/2791286374 >> v2:172.21.15.145:3300/0 conn(0x7fc03006b730 0x7fc0300cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 -- 172.21.15.119:0/2791286374 >> v2:172.21.15.145:3300/0 conn(0x7fc03006b730 msgr2=0x7fc0300cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 --2- 172.21.15.119:0/2791286374 >> v2:172.21.15.145:3300/0 conn(0x7fc03006b730 0x7fc0300cfad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 -- 172.21.15.119:0/2791286374 >> v1:172.21.15.145:6789/0 conn(0x7fc03015cd60 legacy=0x7fc0300cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 -- 172.21.15.119:0/2791286374 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0300ce510 con 0x7fc0300d2c90 2024-09-13T16:29:50.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc036ae7700 1 --2- 172.21.15.119:0/2791286374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0300d2c90 0x7fc0300d30e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fc03008c210 tx=0x7fc02c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57dd4ff425e49ae7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:50.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0352e4700 1 -- 172.21.15.119:0/2791286374 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c00b8a0 con 0x7fc0300d2c90 2024-09-13T16:29:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0352e4700 1 -- 172.21.15.119:0/2791286374 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c00ba00 con 0x7fc0300d2c90 2024-09-13T16:29:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0352e4700 1 -- 172.21.15.119:0/2791286374 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c00bd90 con 0x7fc0300d2c90 2024-09-13T16:29:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- 172.21.15.119:0/2791286374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0300d2c90 msgr2=0x7fc0300d30e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2791286374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0300d2c90 0x7fc0300d30e0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7fc03008c210 tx=0x7fc02c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- 172.21.15.119:0/2791286374 shutdown_connections 2024-09-13T16:29:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2791286374 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0300d2c90 0x7fc0300d30e0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2791286374 >> v2:172.21.15.145:3300/0 conn(0x7fc03006b730 0x7fc0300cfad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- 172.21.15.119:0/2791286374 >> 172.21.15.119:0/2791286374 conn(0x7fc03008c240 msgr2=0x7fc03009c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- 172.21.15.119:0/2791286374 shutdown_connections 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.015+0000 7fc0372e8700 1 -- 172.21.15.119:0/2791286374 wait complete. 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 Processor -- start 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 -- start start 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc03006b730 0x7fc0300b8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0300d2c90 0x7fc0300b7120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03015cd60 0x7fc0300cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300b8c00 con 0x7fc03015cd60 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300b7660 con 0x7fc0300d2c90 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0300b8920 con 0x7fc03006b730 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03015cd60 0x7fc0300cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:50.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03015cd60 0x7fc0300cb5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45590/0 (socket says 172.21.15.119:45590) 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 -- 172.21.15.119:0/2977274576 learned_addr learned my addr 172.21.15.119:0/2977274576 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc035ae5700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0300d2c90 0x7fc0300b7120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0362e6700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc03006b730 0x7fc0300b8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 -- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc03006b730 msgr2=0x7fc0300b8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc03006b730 0x7fc0300b8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 -- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0300d2c90 msgr2=0x7fc0300b7120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0300d2c90 0x7fc0300b7120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 -- 172.21.15.119:0/2977274576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02c009570 con 0x7fc03015cd60 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0362e6700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc03006b730 0x7fc0300b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc036ae7700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03015cd60 0x7fc0300cb5f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fc0300b58c0 tx=0x7fc02c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc026ffd700 1 -- 172.21.15.119:0/2977274576 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c0038c0 con 0x7fc03015cd60 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc026ffd700 1 -- 172.21.15.119:0/2977274576 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc02c01f070 con 0x7fc03015cd60 2024-09-13T16:29:50.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc03015d630 con 0x7fc03015cd60 2024-09-13T16:29:50.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc026ffd700 1 -- 172.21.15.119:0/2977274576 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc02c003c60 con 0x7fc03015cd60 2024-09-13T16:29:50.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.019+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc03015dbf0 con 0x7fc03015cd60 2024-09-13T16:29:50.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.023+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc03006c980 con 0x7fc03015cd60 2024-09-13T16:29:50.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.023+0000 7fc026ffd700 1 -- 172.21.15.119:0/2977274576 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc02c01a430 con 0x7fc03015cd60 2024-09-13T16:29:50.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.023+0000 7fc026ffd700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc014066730 0x7fc014068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:50.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.023+0000 7fc026ffd700 1 -- 172.21.15.119:0/2977274576 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fc02c021070 con 0x7fc03015cd60 2024-09-13T16:29:50.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.023+0000 7fc0362e6700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc014066730 0x7fc014068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:50.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.023+0000 7fc0362e6700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc014066730 0x7fc014068bf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc020001700 tx=0x7fc020009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:50.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.031+0000 7fc026ffd700 1 -- 172.21.15.119:0/2977274576 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc02c053030 con 0x7fc03015cd60 2024-09-13T16:29:50.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.235+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0300d30e0 con 0x7fc03015cd60 2024-09-13T16:29:50.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.235+0000 7fc026ffd700 1 -- 172.21.15.119:0/2977274576 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fc02c056650 con 0x7fc03015cd60 2024-09-13T16:29:50.243 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:50.243 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":44,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:29:49.880678+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:replay","state_seq":1,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:29:50.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc014066730 msgr2=0x7fc014068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:50.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc014066730 0x7fc014068bf0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fc020001700 tx=0x7fc020009040 comp rx=0 tx=0).stop 2024-09-13T16:29:50.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03015cd60 msgr2=0x7fc0300cb5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:50.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03015cd60 0x7fc0300cb5f0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fc0300b58c0 tx=0x7fc02c004770 comp rx=0 tx=0).stop 2024-09-13T16:29:50.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 shutdown_connections 2024-09-13T16:29:50.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc014066730 0x7fc014068bf0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc03015cd60 0x7fc0300cb5f0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0300d2c90 0x7fc0300b7120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 --2- 172.21.15.119:0/2977274576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc03006b730 0x7fc0300b8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:50.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 >> 172.21.15.119:0/2977274576 conn(0x7fc03008c240 msgr2=0x7fc0300afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:50.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 shutdown_connections 2024-09-13T16:29:50.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:50.239+0000 7fc0372e8700 1 -- 172.21.15.119:0/2977274576 wait complete. 2024-09-13T16:29:50.249 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 44 2024-09-13T16:29:50.661 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 44, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:29:30.227373+0000', 'modified': '2024-09-13T16:29:49.880678+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7341': {'gid': 7341, 'name': 'c', 'rank': 0, 'incarnation': 44, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/2534803404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2534803404}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2534803404}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:29:50.661 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7341:up:replay 2024-09-13T16:29:51.662 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:51.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 Processor -- start 2024-09-13T16:29:51.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- start start 2024-09-13T16:29:51.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300cd9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:51.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f14300d1190 0x7f14300cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:51.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14300b23b0 con 0x7f143015cea0 2024-09-13T16:29:51.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14300b2690 con 0x7f14300d1190 2024-09-13T16:29:51.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143015d280 con 0x7f14300cfca0 2024-09-13T16:29:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f1434cea700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f14300d1190 0x7f14300cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f142ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f143015cea0 0x7f14300cf590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:37032/0 (socket says 172.21.15.119:37032) 2024-09-13T16:29:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f142ffff700 1 -- 172.21.15.119:0/1170873609 learned_addr learned my addr 172.21.15.119:0/1170873609 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f142f7fe700 1 --2- 172.21.15.119:0/1170873609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300cd9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:51.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f1434cea700 1 -- 172.21.15.119:0/1170873609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 msgr2=0x7f14300cd9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f1434cea700 1 --2- 172.21.15.119:0/1170873609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300cd9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f1434cea700 1 -- 172.21.15.119:0/1170873609 >> v1:172.21.15.119:6789/0 conn(0x7f143015cea0 legacy=0x7f14300cf590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f1434cea700 1 -- 172.21.15.119:0/1170873609 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14300d03a0 con 0x7f14300d1190 2024-09-13T16:29:51.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f1434cea700 1 --2- 172.21.15.119:0/1170873609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f14300cdf30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1420009850 tx=0x7f1420009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d4dc313634f1680 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f142effd700 1 -- 172.21.15.119:0/1170873609 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420011830 con 0x7f14300d1190 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f142effd700 1 -- 172.21.15.119:0/1170873609 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1420011990 con 0x7f14300d1190 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f142effd700 1 -- 172.21.15.119:0/1170873609 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1420011d40 con 0x7f14300d1190 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- 172.21.15.119:0/1170873609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 msgr2=0x7f14300cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 --2- 172.21.15.119:0/1170873609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f14300cdf30 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f1420009850 tx=0x7f1420009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- 172.21.15.119:0/1170873609 shutdown_connections 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 --2- 172.21.15.119:0/1170873609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f14300cdf30 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 --2- 172.21.15.119:0/1170873609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300cd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:51.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- 172.21.15.119:0/1170873609 >> 172.21.15.119:0/1170873609 conn(0x7f14300d2c00 msgr2=0x7f14300d3010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:51.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- 172.21.15.119:0/1170873609 shutdown_connections 2024-09-13T16:29:51.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.779+0000 7f143674d700 1 -- 172.21.15.119:0/1170873609 wait complete. 2024-09-13T16:29:51.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 Processor -- start 2024-09-13T16:29:51.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 -- start start 2024-09-13T16:29:51.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300c89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:51.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f1430152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:51.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f143015cea0 0x7f14300b6020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:51.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14300c7100 con 0x7f14300d1190 2024-09-13T16:29:51.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f143015d780 con 0x7f143015cea0 2024-09-13T16:29:51.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f143674d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14300c84b0 con 0x7f14300cfca0 2024-09-13T16:29:51.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f1430152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:51.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f1430152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:45618/0 (socket says 172.21.15.119:45618) 2024-09-13T16:29:51.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 -- 172.21.15.119:0/143657458 learned_addr learned my addr 172.21.15.119:0/143657458 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:51.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142ffff700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300c89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:51.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f1434cea700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f143015cea0 0x7f14300b6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 -- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 msgr2=0x7f14300c89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300c89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 -- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f143015cea0 msgr2=0x7f14300b6020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f143015cea0 0x7f14300b6020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.783+0000 7f142f7fe700 1 -- 172.21.15.119:0/143657458 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14200094b0 con 0x7f14300d1190 2024-09-13T16:29:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f142f7fe700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f1430152440 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f142800efc0 tx=0x7f142800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:51.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f142cff9700 1 -- 172.21.15.119:0/143657458 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142800cd10 con 0x7f14300d1190 2024-09-13T16:29:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f142cff9700 1 -- 172.21.15.119:0/143657458 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142800f040 con 0x7f14300d1190 2024-09-13T16:29:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1430152980 con 0x7f14300d1190 2024-09-13T16:29:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f142cff9700 1 -- 172.21.15.119:0/143657458 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1428014680 con 0x7f14300d1190 2024-09-13T16:29:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14300c6a40 con 0x7f14300d1190 2024-09-13T16:29:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1430067760 con 0x7f14300d1190 2024-09-13T16:29:51.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.787+0000 7f142cff9700 1 -- 172.21.15.119:0/143657458 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f14280147e0 con 0x7f14300d1190 2024-09-13T16:29:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.791+0000 7f142cff9700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1410066730 0x7f1410068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.791+0000 7f142cff9700 1 -- 172.21.15.119:0/143657458 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1428084e80 con 0x7f14300d1190 2024-09-13T16:29:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.791+0000 7f142ffff700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1410066730 0x7f1410068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:51.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.791+0000 7f142ffff700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1410066730 0x7f1410068bf0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1418005930 tx=0x7f1418009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:51.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:51.799+0000 7f142cff9700 1 -- 172.21.15.119:0/143657458 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1428052040 con 0x7f14300d1190 2024-09-13T16:29:52.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f143006a450 con 0x7f14300d1190 2024-09-13T16:29:52.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f142cff9700 1 -- 172.21.15.119:0/143657458 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+4743 (secure 0 0 0) 0x7f1428009c80 con 0x7f14300d1190 2024-09-13T16:29:52.020 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:52.021 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":46,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:29:51.888262+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1410066730 msgr2=0x7f1410068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1410066730 0x7f1410068bf0 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1418005930 tx=0x7f1418009040 comp rx=0 tx=0).stop 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 msgr2=0x7f1430152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f1430152440 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f142800efc0 tx=0x7f142800c3a0 comp rx=0 tx=0).stop 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 shutdown_connections 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1410066730 0x7f1410068bf0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f143015cea0 0x7f14300b6020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14300d1190 0x7f1430152440 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 --2- 172.21.15.119:0/143657458 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f14300cfca0 0x7f14300c89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:52.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 >> 172.21.15.119:0/143657458 conn(0x7f14300d2c00 msgr2=0x7f14300cd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:52.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.015+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 shutdown_connections 2024-09-13T16:29:52.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:52.019+0000 7f143674d700 1 -- 172.21.15.119:0/143657458 wait complete. 2024-09-13T16:29:52.024 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 46 2024-09-13T16:29:52.438 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 46, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:29:30.227373+0000', 'modified': '2024-09-13T16:29:51.888262+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7341': {'gid': 7341, 'name': 'c', 'rank': 0, 'incarnation': 44, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.119:6835/2534803404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2534803404}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2534803404}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:29:52.438 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7341:up:rejoin 2024-09-13T16:29:53.439 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:29:53.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 Processor -- start 2024-09-13T16:29:53.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- start start 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe63c15e840 0x7fe63c070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe63c15ec20 0x7fe63c161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe63c071210 con 0x7fe63c164a70 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63c0714f0 con 0x7fe63c15ec20 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe63c0717d0 con 0x7fe63c15e840 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe63c15ec20 0x7fe63c161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe63c15ec20 0x7fe63c161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51426/0 (socket says 172.21.15.119:51426) 2024-09-13T16:29:53.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 -- 172.21.15.119:0/1040429435 learned_addr learned my addr 172.21.15.119:0/1040429435 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:53.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe63affd700 1 --2- 172.21.15.119:0/1040429435 >> v2:172.21.15.145:3300/0 conn(0x7fe63c15e840 0x7fe63c070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:53.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 -- 172.21.15.119:0/1040429435 >> v2:172.21.15.145:3300/0 conn(0x7fe63c15e840 msgr2=0x7fe63c070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:53.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 --2- 172.21.15.119:0/1040429435 >> v2:172.21.15.145:3300/0 conn(0x7fe63c15e840 0x7fe63c070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe63a7fc700 1 -- 172.21.15.119:0/1040429435 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220352072 0 0) 0x7fe63c071210 con 0x7fe63c164a70 2024-09-13T16:29:53.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 -- 172.21.15.119:0/1040429435 >> v1:172.21.15.119:6789/0 conn(0x7fe63c164a70 legacy=0x7fe63c0ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:53.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 -- 172.21.15.119:0/1040429435 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe63c071ab0 con 0x7fe63c15ec20 2024-09-13T16:29:53.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe6327fc700 1 --2- 172.21.15.119:0/1040429435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c15ec20 0x7fe63c161f00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fe63c06dec0 tx=0x7fe624009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e3ff191d759b4f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe63a7fc700 1 -- 172.21.15.119:0/1040429435 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624011830 con 0x7fe63c15ec20 2024-09-13T16:29:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe63a7fc700 1 -- 172.21.15.119:0/1040429435 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624011990 con 0x7fe63c15ec20 2024-09-13T16:29:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe63a7fc700 1 -- 172.21.15.119:0/1040429435 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624011d20 con 0x7fe63c15ec20 2024-09-13T16:29:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- 172.21.15.119:0/1040429435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c15ec20 msgr2=0x7fe63c161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 --2- 172.21.15.119:0/1040429435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c15ec20 0x7fe63c161f00 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fe63c06dec0 tx=0x7fe624009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- 172.21.15.119:0/1040429435 shutdown_connections 2024-09-13T16:29:53.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 --2- 172.21.15.119:0/1040429435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c15ec20 0x7fe63c161f00 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 --2- 172.21.15.119:0/1040429435 >> v2:172.21.15.145:3300/0 conn(0x7fe63c15e840 0x7fe63c070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- 172.21.15.119:0/1040429435 >> 172.21.15.119:0/1040429435 conn(0x7fe63c06a350 msgr2=0x7fe63c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:53.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- 172.21.15.119:0/1040429435 shutdown_connections 2024-09-13T16:29:53.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.559+0000 7fe64182d700 1 -- 172.21.15.119:0/1040429435 wait complete. 2024-09-13T16:29:53.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 Processor -- start 2024-09-13T16:29:53.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 -- start start 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe63c15e840 0x7fe63c1546e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c164a70 0x7fe63c1fa820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe63c1f6070 0x7fe63c0af710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c154c20 con 0x7fe63c164a70 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c1fad60 con 0x7fe63c1f6070 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe64182d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe63c157f50 con 0x7fe63c15e840 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe6327fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c164a70 0x7fe63c1fa820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe6327fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c164a70 0x7fe63c1fa820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51438/0 (socket says 172.21.15.119:51438) 2024-09-13T16:29:53.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe6327fc700 1 -- 172.21.15.119:0/2405807789 learned_addr learned my addr 172.21.15.119:0/2405807789 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe63affd700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe63c15e840 0x7fe63c1546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.563+0000 7fe63b7fe700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe63c1f6070 0x7fe63c0af710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe6327fc700 1 -- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe63c15e840 msgr2=0x7fe63c1546e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe6327fc700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe63c15e840 0x7fe63c1546e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe6327fc700 1 -- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe63c1f6070 msgr2=0x7fe63c0af710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe6327fc700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe63c1f6070 0x7fe63c0af710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe6327fc700 1 -- 172.21.15.119:0/2405807789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe624009570 con 0x7fe63c164a70 2024-09-13T16:29:53.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe6327fc700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c164a70 0x7fe63c1fa820 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe63c069530 tx=0x7fe624004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:53.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe638ff9700 1 -- 172.21.15.119:0/2405807789 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6240044d0 con 0x7fe63c164a70 2024-09-13T16:29:53.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe638ff9700 1 -- 172.21.15.119:0/2405807789 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe62401f070 con 0x7fe63c164a70 2024-09-13T16:29:53.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe63c158230 con 0x7fe63c164a70 2024-09-13T16:29:53.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe638ff9700 1 -- 172.21.15.119:0/2405807789 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624003900 con 0x7fe63c164a70 2024-09-13T16:29:53.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe63c1587f0 con 0x7fe63c164a70 2024-09-13T16:29:53.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe638ff9700 1 -- 172.21.15.119:0/2405807789 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe624003a60 con 0x7fe63c164a70 2024-09-13T16:29:53.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe63c067760 con 0x7fe63c164a70 2024-09-13T16:29:53.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe638ff9700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe61c066730 0x7fe61c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:53.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe638ff9700 1 -- 172.21.15.119:0/2405807789 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe624021070 con 0x7fe63c164a70 2024-09-13T16:29:53.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.567+0000 7fe63affd700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe61c066730 0x7fe61c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:53.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.571+0000 7fe63affd700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe61c066730 0x7fe61c068bf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009b50 tx=0x7fe62c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:53.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.575+0000 7fe638ff9700 1 -- 172.21.15.119:0/2405807789 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe6240529d0 con 0x7fe63c164a70 2024-09-13T16:29:53.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.775+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe63c070b80 con 0x7fe63c164a70 2024-09-13T16:29:53.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.775+0000 7fe638ff9700 1 -- 172.21.15.119:0/2405807789 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fe624083e60 con 0x7fe63c164a70 2024-09-13T16:29:53.782 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:53.782 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":47,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":47,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:29:52.899307+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe61c066730 msgr2=0x7fe61c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe61c066730 0x7fe61c068bf0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe62c009b50 tx=0x7fe62c009240 comp rx=0 tx=0).stop 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c164a70 msgr2=0x7fe63c1fa820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c164a70 0x7fe63c1fa820 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fe63c069530 tx=0x7fe624004770 comp rx=0 tx=0).stop 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 shutdown_connections 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe61c066730 0x7fe61c068bf0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe63c1f6070 0x7fe63c0af710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe63c164a70 0x7fe63c1fa820 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 --2- 172.21.15.119:0/2405807789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe63c15e840 0x7fe63c1546e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:53.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 >> 172.21.15.119:0/2405807789 conn(0x7fe63c06a350 msgr2=0x7fe63c1625b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:53.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 shutdown_connections 2024-09-13T16:29:53.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:53.779+0000 7fe64182d700 1 -- 172.21.15.119:0/2405807789 wait complete. 2024-09-13T16:29:53.786 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 47 2024-09-13T16:29:54.199 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 47, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:29:30.227373+0000', 'modified': '2024-09-13T16:29:52.899307+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 83, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7341}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7341': {'gid': 7341, 'name': 'c', 'rank': 0, 'incarnation': 44, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.119:6835/2534803404', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 2534803404}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 2534803404}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:29:54.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:29:54.200 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:29:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 Processor -- start 2024-09-13T16:29:54.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 -- start start 2024-09-13T16:29:54.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff230160ed0 0x7ff2300d1120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2300d1660 0x7ff23006e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff2300cf960 0x7ff23006e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2300ceba0 con 0x7ff2300cf960 2024-09-13T16:29:54.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2300cee80 con 0x7ff230160ed0 2024-09-13T16:29:54.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2300cf160 con 0x7ff2300d1660 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff2300cf960 0x7ff23006e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff2300cf960 0x7ff23006e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51446/0 (socket says 172.21.15.119:51446) 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 -- 172.21.15.119:0/1006232552 learned_addr learned my addr 172.21.15.119:0/1006232552 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff22ffff700 1 --2- 172.21.15.119:0/1006232552 >> v2:172.21.15.145:3300/0 conn(0x7ff230160ed0 0x7ff2300d1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/1006232552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2300d1660 0x7ff23006e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 -- 172.21.15.119:0/1006232552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2300d1660 msgr2=0x7ff23006e270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 --2- 172.21.15.119:0/1006232552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2300d1660 0x7ff23006e270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 -- 172.21.15.119:0/1006232552 >> v2:172.21.15.145:3300/0 conn(0x7ff230160ed0 msgr2=0x7ff2300d1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 --2- 172.21.15.119:0/1006232552 >> v2:172.21.15.145:3300/0 conn(0x7ff230160ed0 0x7ff2300d1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 -- 172.21.15.119:0/1006232552 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2300cf440 con 0x7ff2300cf960 2024-09-13T16:29:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff234aba700 1 --2- 172.21.15.119:0/1006232552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff23006e7b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff230067520 tx=0x7ff228009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=461929b0a39cad05 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff22effd700 1 -- 172.21.15.119:0/1006232552 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228011830 con 0x7ff2300cf960 2024-09-13T16:29:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff22effd700 1 -- 172.21.15.119:0/1006232552 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228011990 con 0x7ff2300cf960 2024-09-13T16:29:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff22effd700 1 -- 172.21.15.119:0/1006232552 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228011d20 con 0x7ff2300cf960 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 -- 172.21.15.119:0/1006232552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 msgr2=0x7ff23006e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.319+0000 7ff2352bb700 1 --2- 172.21.15.119:0/1006232552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff23006e7b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff230067520 tx=0x7ff228009bf0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- 172.21.15.119:0/1006232552 shutdown_connections 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 --2- 172.21.15.119:0/1006232552 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff23006e7b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 --2- 172.21.15.119:0/1006232552 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2300d1660 0x7ff23006e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 --2- 172.21.15.119:0/1006232552 >> v2:172.21.15.145:3300/0 conn(0x7ff230160ed0 0x7ff2300d1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- 172.21.15.119:0/1006232552 >> 172.21.15.119:0/1006232552 conn(0x7ff230099a50 msgr2=0x7ff230099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- 172.21.15.119:0/1006232552 shutdown_connections 2024-09-13T16:29:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- 172.21.15.119:0/1006232552 wait complete. 2024-09-13T16:29:54.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 Processor -- start 2024-09-13T16:29:54.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- start start 2024-09-13T16:29:54.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff230153c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2300d1660 0x7ff230163480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff230160ed0 0x7ff2301639c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23009ece0 con 0x7ff2300cf960 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff23009e0d0 con 0x7ff2300d1660 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff2352bb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23009e3b0 con 0x7ff230160ed0 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff230153c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff230153c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51462/0 (socket says 172.21.15.119:51462) 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff22ffff700 1 -- 172.21.15.119:0/2282339822 learned_addr learned my addr 172.21.15.119:0/2282339822 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.323+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2300d1660 0x7ff230163480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff234aba700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff230160ed0 0x7ff2301639c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22ffff700 1 -- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff230160ed0 msgr2=0x7ff2301639c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22ffff700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff230160ed0 0x7ff2301639c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22ffff700 1 -- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2300d1660 msgr2=0x7ff230163480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22ffff700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2300d1660 0x7ff230163480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22ffff700 1 -- 172.21.15.119:0/2282339822 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff228009570 con 0x7ff2300cf960 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff234aba700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff230160ed0 0x7ff2301639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22ffff700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff230153c90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff21800e840 tx=0x7ff21800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22cff9700 1 -- 172.21.15.119:0/2282339822 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2180095a0 con 0x7ff2300cf960 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22cff9700 1 -- 172.21.15.119:0/2282339822 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff218013680 con 0x7ff2300cf960 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff23009e690 con 0x7ff2300cf960 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22cff9700 1 -- 172.21.15.119:0/2282339822 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2180139f0 con 0x7ff2300cf960 2024-09-13T16:29:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff23009d310 con 0x7ff2300cf960 2024-09-13T16:29:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.327+0000 7ff22cff9700 1 -- 172.21.15.119:0/2282339822 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff218013b50 con 0x7ff2300cf960 2024-09-13T16:29:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff22cff9700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff210066730 0x7ff210068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff22cff9700 1 -- 172.21.15.119:0/2282339822 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff218084620 con 0x7ff2300cf960 2024-09-13T16:29:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff210066730 0x7ff210068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 Processor -- start 2024-09-13T16:29:54.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff210066730 0x7ff210068bf0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff23015e1f0 tx=0x7ff22000d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 -- start start 2024-09-13T16:29:54.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2416610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23f4e90 0x23de890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2416b50 0x23dedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x233dd80 con 0x2416b50 2024-09-13T16:29:54.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x233b510 con 0x23f4e90 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2336e00 con 0x2416230 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2416b50 0x23dedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2416b50 0x23dedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51464/0 (socket says 172.21.15.119:51464) 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2416610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 -- 172.21.15.119:0/2220491305 learned_addr learned my addr 172.21.15.119:0/2220491305 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23f4e90 0x23de890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 -- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 msgr2=0x2416610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2416610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 -- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23f4e90 msgr2=0x23de890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23f4e90 0x23de890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 -- 172.21.15.119:0/2220491305 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x21a9960 con 0x2416b50 2024-09-13T16:29:54.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23f4e90 0x23de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.331+0000 7ff234aba700 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2416b50 0x23dedd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff228013000 tx=0x7ff228010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb978c0626794f87 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff225ffb700 1 -- 172.21.15.119:0/2220491305 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228011b00 con 0x2416b50 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff225ffb700 1 -- 172.21.15.119:0/2220491305 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228011c60 con 0x2416b50 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff225ffb700 1 -- 172.21.15.119:0/2220491305 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff228004040 con 0x2416b50 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- 172.21.15.119:0/2220491305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2416b50 msgr2=0x23dedd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2416b50 0x23dedd0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7ff228013000 tx=0x7ff228010000 comp rx=0 tx=0).stop 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- 172.21.15.119:0/2220491305 shutdown_connections 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2416b50 0x23dedd0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23f4e90 0x23de890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2220491305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2416610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- 172.21.15.119:0/2220491305 >> 172.21.15.119:0/2220491305 conn(0x215e4d0 msgr2=0x23f7e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- 172.21.15.119:0/2220491305 shutdown_connections 2024-09-13T16:29:54.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- 172.21.15.119:0/2220491305 wait complete. 2024-09-13T16:29:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- start start 2024-09-13T16:29:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 Processor -- start 2024-09-13T16:29:54.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x23d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x240e610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x22f4730 con 0x23f4e90 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x22fa5b0 con 0x2416b50 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x21e2aa0 con 0x2416230 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff2397ce740 10 client.0 _ll_register_callbacks cb 0x2330610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51478/0 (socket says 172.21.15.119:51478) 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff22f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x23d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff22ffff700 1 -- 172.21.15.119:0/4127498254 learned_addr learned my addr 172.21.15.119:0/4127498254 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:54.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.335+0000 7ff234aba700 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x240e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22ffff700 1 -- 172.21.15.119:0/4127498254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 msgr2=0x23d0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22ffff700 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x23d0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22ffff700 1 -- 172.21.15.119:0/4127498254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 msgr2=0x240e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22ffff700 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x240e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22ffff700 1 -- 172.21.15.119:0/4127498254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2300cf440 con 0x23f4e90 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22ffff700 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7ff2180553b0 tx=0x7ff218059320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fed817a93e966b0a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 1 -- 172.21.15.119:0/4127498254 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218054910 con 0x23f4e90 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 1 -- 172.21.15.119:0/4127498254 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff218054a70 con 0x23f4e90 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x216b1e0 con 0x23f4e90 2024-09-13T16:29:54.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 1 -- 172.21.15.119:0/4127498254 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff21805d410 con 0x23f4e90 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 10 client.7395 fetch_fsmap learned FSMap version 47 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x22fd220 con 0x23f4e90 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 1 -- 172.21.15.119:0/4127498254 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x216b1e0 con 0x23f4e90 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 1 -- 172.21.15.119:0/4127498254 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 47) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff2180731f0 con 0x23f4e90 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 10 client.7395 fetch_fsmap finished waiting for FSMap version 47 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 10 client.7395 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 20 client.7395 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23cc080 0x2429da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 4 client.7395 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x21b5460 con 0x23cc080 2024-09-13T16:29:54.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23cc080 0x2429da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 10 client.7395 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:29:54.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23cc080 0x2429da0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b134e7bacbe27004 server_cookie=17b062749b5e6d20 in_seq=0 out_seq=0 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 1 -- 172.21.15.119:0/4127498254 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x21b5460 con 0x23cc080 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff20effd700 10 client.7395 handle_command_reply: tid=0 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.339+0000 7ff2397ce740 1 client.7395 shutdown 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 msgr2=0x2415a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7ff2180553b0 tx=0x7ff218059320 comp rx=0 tx=0).stop 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 shutdown_connections 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23cc080 0x2429da0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x240e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x23d0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 --2- 172.21.15.119:0/4127498254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a00 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 >> 172.21.15.119:0/4127498254 conn(0x215e4d0 msgr2=0x240f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff20effd700 0 client.7395 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:29:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 shutdown_connections 2024-09-13T16:29:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 1 -- 172.21.15.119:0/4127498254 wait complete. 2024-09-13T16:29:54.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.343+0000 7ff2397ce740 20 client.7395 trim_cache size 0 max 16384 2024-09-13T16:29:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 Processor -- start 2024-09-13T16:29:54.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 -- start start 2024-09-13T16:29:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416230 0x23f4120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416b50 0x241af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x23ccc40 con 0x23f4e90 2024-09-13T16:29:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x22dd950 con 0x2416230 2024-09-13T16:29:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x23cbf00 con 0x2416b50 2024-09-13T16:29:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51488/0 (socket says 172.21.15.119:51488) 2024-09-13T16:29:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 -- 172.21.15.119:0/3734532604 learned_addr learned my addr 172.21.15.119:0/3734532604 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff234aba700 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416b50 0x241af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416230 0x23f4120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 -- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416b50 msgr2=0x241af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416b50 0x241af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 -- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416230 msgr2=0x23f4120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416230 0x23f4120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22ffff700 1 -- 172.21.15.119:0/3734532604 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x22e3450 con 0x23f4e90 2024-09-13T16:29:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.359+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416230 0x23f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:29:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff22ffff700 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7ff218084900 tx=0x7ff218021000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7aad4f1425921f82 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2267fc700 1 -- 172.21.15.119:0/3734532604 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218054f60 con 0x23f4e90 2024-09-13T16:29:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2267fc700 1 -- 172.21.15.119:0/3734532604 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff218055b20 con 0x23f4e90 2024-09-13T16:29:54.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2267fc700 1 -- 172.21.15.119:0/3734532604 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff218051760 con 0x23f4e90 2024-09-13T16:29:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 -- 172.21.15.119:0/3734532604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 msgr2=0x2415a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7ff218084900 tx=0x7ff218021000 comp rx=0 tx=0).stop 2024-09-13T16:29:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 -- 172.21.15.119:0/3734532604 shutdown_connections 2024-09-13T16:29:54.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416b50 0x241af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416230 0x23f4120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 --2- 172.21.15.119:0/3734532604 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2415a80 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 -- 172.21.15.119:0/3734532604 >> 172.21.15.119:0/3734532604 conn(0x215e4d0 msgr2=0x24264f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 -- 172.21.15.119:0/3734532604 shutdown_connections 2024-09-13T16:29:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 -- 172.21.15.119:0/3734532604 wait complete. 2024-09-13T16:29:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 -- start start 2024-09-13T16:29:54.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.363+0000 7ff2397ce740 1 Processor -- start 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2412330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2411330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x247a540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x242e700 con 0x23f4e90 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2415aa0 con 0x2416b50 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2414db0 con 0x2416230 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 10 client.0 _ll_register_callbacks cb 0x22e8ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2412330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2411330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2412330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51498/0 (socket says 172.21.15.119:51498) 2024-09-13T16:29:54.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff234aba700 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x247a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22ffff700 1 -- 172.21.15.119:0/2108970238 learned_addr learned my addr 172.21.15.119:0/2108970238 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:54.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22f7fe700 1 -- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 msgr2=0x247a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x247a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22f7fe700 1 -- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 msgr2=0x2412330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2412330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22f7fe700 1 -- 172.21.15.119:0/2108970238 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x22fd220 con 0x2416230 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22ffff700 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2412330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff22f7fe700 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2411330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x240f100 tx=0x7ff22000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37f4c3466cae4293 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff20effd700 10 client.0 ms_handle_connect on v2:172.21.15.145:3301/0 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff20effd700 1 -- 172.21.15.119:0/2108970238 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff220010040 con 0x2416230 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff228011c60 con 0x2416230 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff20effd700 1 -- 172.21.15.119:0/2108970238 <== mon.2 v2:172.21.15.145:3301/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff220028070 con 0x2416230 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff20effd700 1 -- 172.21.15.119:0/2108970238 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_get_version_reply(handle=1 version=47) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff220028230 con 0x2416230 2024-09-13T16:29:54.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 10 client.6413 fetch_fsmap learned FSMap version 47 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,fsmap=47,monmap=2+}) v3 -- 0x22d7720 con 0x2416230 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.367+0000 7ff20effd700 1 -- 172.21.15.119:0/2108970238 <== mon.2 v2:172.21.15.145:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228011c60 con 0x2416230 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff20effd700 1 -- 172.21.15.119:0/2108970238 <== mon.2 v2:172.21.15.145:3301/0 5 ==== fsmap(e 47) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff220028390 con 0x2416230 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 10 client.6413 fetch_fsmap finished waiting for FSMap version 47 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 10 client.6413 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 20 client.6413 populate_metadata read hostname 'smithi119' 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23ce1a0 0x23ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 4 client.6413 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "status"}) v1 -- 0x21da1e0 con 0x23ce1a0 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff22ffff700 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23ce1a0 0x23ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff20effd700 10 client.6413 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff22ffff700 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23ce1a0 0x23ce580 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e625db0ce60c7552 server_cookie=aa335929797abe45 in_seq=0 out_seq=0 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff20effd700 1 -- 172.21.15.119:0/2108970238 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x21da1e0 con 0x23ce1a0 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff20effd700 10 client.6413 handle_command_reply: tid=0 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 client.6413 shutdown 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 msgr2=0x2411330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2411330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x240f100 tx=0x7ff22000b040 comp rx=0 tx=0).stop 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 shutdown_connections 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x23ce1a0 0x23ce580 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2416b50 0x247a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff20effd700 0 client.6413 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:29:54.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2416230 0x2411330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 --2- 172.21.15.119:0/2108970238 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x23f4e90 0x2412330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 >> 172.21.15.119:0/2108970238 conn(0x215e4d0 msgr2=0x23caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 shutdown_connections 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 1 -- 172.21.15.119:0/2108970238 wait complete. 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2397ce740 20 client.6413 trim_cache size 0 max 16384 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:29:54.381 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 7341, 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 4.4911536080000003, 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 47, 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 83, 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 83, 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 5.3817575929999997 2024-09-13T16:29:54.382 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:54.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff210066730 msgr2=0x7ff210068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff210066730 0x7ff210068bf0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7ff23015e1f0 tx=0x7ff22000d040 comp rx=0 tx=0).stop 2024-09-13T16:29:54.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 msgr2=0x7ff230153c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff230153c90 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7ff21800e840 tx=0x7ff21800eb50 comp rx=0 tx=0).stop 2024-09-13T16:29:54.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 shutdown_connections 2024-09-13T16:29:54.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff210066730 0x7ff210068bf0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff230160ed0 0x7ff2301639c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2300d1660 0x7ff230163480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 --2- 172.21.15.119:0/2282339822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2300cf960 0x7ff230153c90 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 >> 172.21.15.119:0/2282339822 conn(0x7ff230099a50 msgr2=0x7ff230099e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 shutdown_connections 2024-09-13T16:29:54.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.371+0000 7ff2352bb700 1 -- 172.21.15.119:0/2282339822 wait complete. 2024-09-13T16:29:54.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:29:54.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.503+0000 7f38dd7fc700 1 Processor -- start 2024-09-13T16:29:54.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.503+0000 7f38dd7fc700 1 -- start start 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f38d815ddc0 0x7f38d80abed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d80b3ae0 0x7f38d80ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d815e1a0 con 0x7f38d809cdb0 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d80cc130 con 0x7f38d815ddc0 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d80cc410 con 0x7f38d80b3ae0 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38cf7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f38d809cdb0 0x7f38d806a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56452/0 (socket says 172.21.15.119:56452) 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38cf7fe700 1 -- 172.21.15.119:0/2275079235 learned_addr learned my addr 172.21.15.119:0/2275079235 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38d7fff700 1 --2- 172.21.15.119:0/2275079235 >> v2:172.21.15.145:3300/0 conn(0x7f38d815ddc0 0x7f38d80abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dcffb700 1 --2- 172.21.15.119:0/2275079235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d80b3ae0 0x7f38d80ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38d77fe700 1 -- 172.21.15.119:0/2275079235 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3152266452 0 0) 0x7f38d815e1a0 con 0x7f38d809cdb0 2024-09-13T16:29:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38d77fe700 1 -- 172.21.15.119:0/2275079235 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c40035d0 con 0x7f38d809cdb0 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dcffb700 1 -- 172.21.15.119:0/2275079235 >> v2:172.21.15.145:3300/0 conn(0x7f38d815ddc0 msgr2=0x7f38d80abed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dcffb700 1 --2- 172.21.15.119:0/2275079235 >> v2:172.21.15.145:3300/0 conn(0x7f38d815ddc0 0x7f38d80abed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dcffb700 1 -- 172.21.15.119:0/2275079235 >> v1:172.21.15.119:6789/0 conn(0x7f38d809cdb0 legacy=0x7f38d806a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dcffb700 1 -- 172.21.15.119:0/2275079235 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d80cc6f0 con 0x7f38d80b3ae0 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dcffb700 1 --2- 172.21.15.119:0/2275079235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d80b3ae0 0x7f38d80ac410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38d0009d90 tx=0x7f38d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b14dbfee09b307db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38d77fe700 1 -- 172.21.15.119:0/2275079235 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d000e070 con 0x7f38d80b3ae0 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38d77fe700 1 -- 172.21.15.119:0/2275079235 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38d000bb20 con 0x7f38d80b3ae0 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38d77fe700 1 -- 172.21.15.119:0/2275079235 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38d0012430 con 0x7f38d80b3ae0 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- 172.21.15.119:0/2275079235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d80b3ae0 msgr2=0x7f38d80ac410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/2275079235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d80b3ae0 0x7f38d80ac410 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f38d0009d90 tx=0x7f38d000b330 comp rx=0 tx=0).stop 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- 172.21.15.119:0/2275079235 shutdown_connections 2024-09-13T16:29:54.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/2275079235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d80b3ae0 0x7f38d80ac410 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/2275079235 >> v2:172.21.15.145:3300/0 conn(0x7f38d815ddc0 0x7f38d80abed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- 172.21.15.119:0/2275079235 >> 172.21.15.119:0/2275079235 conn(0x7f38d80ae4b0 msgr2=0x7f38d80b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- 172.21.15.119:0/2275079235 shutdown_connections 2024-09-13T16:29:54.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.507+0000 7f38dd7fc700 1 -- 172.21.15.119:0/2275079235 wait complete. 2024-09-13T16:29:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 Processor -- start 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 -- start start 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d809cdb0 0x7f38d81556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d80b3ae0 0x7f38d8156a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d815ddc0 0x7f38d8099d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d8155c30 con 0x7f38d80b3ae0 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d8157400 con 0x7f38d809cdb0 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dd7fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38d809c2e0 con 0x7f38d815ddc0 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d80b3ae0 0x7f38d8156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dcffb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d815ddc0 0x7f38d8099d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38d7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d809cdb0 0x7f38d81556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d80b3ae0 0x7f38d8156a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51502/0 (socket says 172.21.15.119:51502) 2024-09-13T16:29:54.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dcffb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d815ddc0 0x7f38d8099d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:48134/0 (socket says 172.21.15.119:48134) 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 -- 172.21.15.119:0/1868727544 learned_addr learned my addr 172.21.15.119:0/1868727544 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 -- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d815ddc0 msgr2=0x7f38d8099d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d815ddc0 0x7f38d8099d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 -- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d809cdb0 msgr2=0x7f38d81556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d809cdb0 0x7f38d81556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 -- 172.21.15.119:0/1868727544 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d0009a70 con 0x7f38d80b3ae0 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38dcffb700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d815ddc0 0x7f38d8099d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:54.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.511+0000 7f38cf7fe700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d80b3ae0 0x7f38d8156a90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005930 tx=0x7f38c000c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38d57fa700 1 -- 172.21.15.119:0/1868727544 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0005d90 con 0x7f38d80b3ae0 2024-09-13T16:29:54.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38d57fa700 1 -- 172.21.15.119:0/1868727544 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c00070d0 con 0x7f38d80b3ae0 2024-09-13T16:29:54.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d809c5c0 con 0x7f38d80b3ae0 2024-09-13T16:29:54.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38d57fa700 1 -- 172.21.15.119:0/1868727544 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0007480 con 0x7f38d80b3ae0 2024-09-13T16:29:54.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d81576c0 con 0x7f38d80b3ae0 2024-09-13T16:29:54.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d8073590 con 0x7f38d80b3ae0 2024-09-13T16:29:54.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38d57fa700 1 -- 172.21.15.119:0/1868727544 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f38c00075e0 con 0x7f38d80b3ae0 2024-09-13T16:29:54.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38d57fa700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b8066730 0x7f38b8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:54.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.515+0000 7f38d57fa700 1 -- 172.21.15.119:0/1868727544 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f38c00837b0 con 0x7f38d80b3ae0 2024-09-13T16:29:54.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.519+0000 7f38d7fff700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b8066730 0x7f38b8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:54.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.519+0000 7f38d7fff700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b8066730 0x7f38b8068bf0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f38c8015230 tx=0x7f38c800f040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:54.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.523+0000 7f38d57fa700 1 -- 172.21.15.119:0/1868727544 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f38c0050900 con 0x7f38d80b3ae0 2024-09-13T16:29:54.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.727+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f38d80cd870 con 0x7f38d80b3ae0 2024-09-13T16:29:54.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.727+0000 7f38d57fa700 1 -- 172.21.15.119:0/1868727544 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v47) v1 ==== 53+0+73 (secure 0 0 0) 0x7f38c0053f20 con 0x7f38d80b3ae0 2024-09-13T16:29:54.734 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:29:54.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b8066730 msgr2=0x7f38b8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b8066730 0x7f38b8068bf0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f38c8015230 tx=0x7f38c800f040 comp rx=0 tx=0).stop 2024-09-13T16:29:54.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d80b3ae0 msgr2=0x7f38d8156a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:54.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d80b3ae0 0x7f38d8156a90 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f38c0005930 tx=0x7f38c000c9e0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 shutdown_connections 2024-09-13T16:29:54.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f38b8066730 0x7f38b8068bf0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f38d815ddc0 0x7f38d8099d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f38d80b3ae0 0x7f38d8156a90 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 --2- 172.21.15.119:0/1868727544 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f38d809cdb0 0x7f38d81556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:54.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 >> 172.21.15.119:0/1868727544 conn(0x7f38d80ae4b0 msgr2=0x7f38d815be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:54.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 shutdown_connections 2024-09-13T16:29:54.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:54.731+0000 7f38dd7fc700 1 -- 172.21.15.119:0/1868727544 wait complete. 2024-09-13T16:29:54.748 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:29:54.749 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:29:54.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:54.749 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:29:54.754 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:29:54.754 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:29:54.754 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:29:54.754 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.754 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:29:54.754 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42346sec preferred_lft 42346sec 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.755 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:29:54.756 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.757 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:29:54.757 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:29:54.757 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:29:54.757 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:29:54.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:54.757 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:29:54.771 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:29:54.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:54.772 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:29:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:29:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:29:54.825 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:29:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:29:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:29:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:29:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:29:54.826 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:29:54.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:54.827 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:29:54.827 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:29:54.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:29:54.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:29:54.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:29:54.828 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:29:54.869 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:54.869 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:29:54.898 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:29:54.891+0000 7f1b7691f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:29:54.905 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:29:54.905 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:29:54.905 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:29:54.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:54.906 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:29:54.963 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:29:54.963 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:29:54.963 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:54.969 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:29:54.969 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:29:54.969 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:54.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:29:54.987 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:29:54.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:29:55.003 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:29:55.004 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:29:55.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:55.004 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:29:55.011 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:29:55.011 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:29:55.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:29:55.074 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:29:55.076 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:29:55.086 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:29:55.087 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:29:55.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:55.088 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:29:55.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:29:55.147+0000 7ff8773ee640 -1 init, newargv = 0x559958548250 newargc=15 2024-09-13T16:29:55.152 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[44840]: starting ceph client 2024-09-13T16:29:55.161 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[44840]: starting fuse 2024-09-13T16:29:56.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:29:56.257 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:29:56.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:29:56.273 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:29:56.274 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:29:56.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:56.275 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:29:56.325 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:29:56.326 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:29:56.326 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:29:56.327 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:56.327 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:29:56.327 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:29:56.327 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:29:56.327 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:29:56.327 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:29:56.328 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:29:56.414 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.44840.asok 2024-09-13T16:29:56.597 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.44840.asok 2024-09-13T16:29:56.598 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:56.598 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.44840.asok status 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "44840", 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:29:56.698 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 7416, 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 7416 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3257231230 2024-09-13T16:29:56.699 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3257231230 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.7416 192.168.0.1:0/3257231230", 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/3257231230", 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 47, 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 83, 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:29:56.700 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:29:56.701 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:29:56.701 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:29:56.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:56.708 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:29:56.709 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:56.714 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:29:56.714 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:56.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:56.714 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:29:56.770 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:56.770 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls subdir/) 2024-09-13T16:29:56.786 INFO:teuthology.orchestra.run.smithi119.stdout:file_undamaged 2024-09-13T16:29:56.787 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:29:56.787 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:29:56.787 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> import json 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> import sys 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> try: 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> s = os.stat('"'"'/home/ubuntu/cephtest/mnt.0/subdir/file_to_be_damaged'"'"') 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> except OSError as e: 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> sys.exit(e.errno) 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> attrs = ["st_mode", "st_ino", "st_dev", "st_nlink", "st_uid", "st_gid", "st_size", "st_atime", "st_mtime", "st_ctime"] 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> print(json.dumps( 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> dict([(a, getattr(s, a)) for a in attrs]), 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> indent=2)) 2024-09-13T16:29:56.788 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:29:57.033 INFO:teuthology.orchestra.run.smithi119.stderr:daemon-helper: command failed with exit status 2 2024-09-13T16:29:57.044 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-09-13T16:29:57.045 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:29:57.045 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-09-13T16:29:57.099 INFO:teuthology.orchestra.run.smithi119.stdout:2 2024-09-13T16:29:57.099 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:57.099 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:57.107 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:57.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.219+0000 7fad49c77700 1 Processor -- start 2024-09-13T16:29:57.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.219+0000 7fad49c77700 1 -- start start 2024-09-13T16:29:57.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.219+0000 7fad49c77700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 0x7fad440ce380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.219+0000 7fad49c77700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad440d3ba0 con 0x7fad440d3780 2024-09-13T16:29:57.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.219+0000 7fad49c77700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad440cec40 con 0x7fad440d1770 2024-09-13T16:29:57.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.219+0000 7fad49c77700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad440ce960 con 0x7fad4406b160 2024-09-13T16:29:57.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fad440d3780 0x7fad4406b540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56468/0 (socket says 172.21.15.119:56468) 2024-09-13T16:29:57.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 -- 172.21.15.119:0/2630996896 learned_addr learned my addr 172.21.15.119:0/2630996896 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:57.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad437fe700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 0x7fad440ce380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376922907 0 0) 0x7fad440d3ba0 con 0x7fad440d3780 2024-09-13T16:29:57.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad300035d0 con 0x7fad440d3780 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2472722971 0 0) 0x7fad300035d0 con 0x7fad440d3780 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 msgr2=0x7fad440ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 0x7fad440ce380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 >> v1:172.21.15.145:6789/0 conn(0x7fad440d1770 legacy=0x7fad440d1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440d0c90 con 0x7fad440d3780 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fad2c001670 con 0x7fad440d3780 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 >> v1:172.21.15.119:6789/0 conn(0x7fad440d3780 legacy=0x7fad4406b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 0x7fad30003a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad30003f90 0x7fad30004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad30004db0 0x7fad30005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad440ce960 con 0x7fad30003f90 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3000ba90 con 0x7fad30004db0 2024-09-13T16:29:57.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3000bc50 con 0x7fad4406b160 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad30003f90 0x7fad30004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad437fe700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 0x7fad30003a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad43fff700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad30004db0 0x7fad30005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 -- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 msgr2=0x7fad30003a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 0x7fad30003a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 -- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad30004db0 msgr2=0x7fad30005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad30004db0 0x7fad30005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 -- 172.21.15.119:0/2630996896 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad300035d0 con 0x7fad30003f90 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad437fe700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad4406b160 0x7fad30003a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad3bfff700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad30003f90 0x7fad30004870 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fad44160930 tx=0x7fad300058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=631ccdc563a0f7da server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c00cd40 con 0x7fad30003f90 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2c005980 con 0x7fad30003f90 2024-09-13T16:29:57.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad42ffd700 1 -- 172.21.15.119:0/2630996896 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2c005d30 con 0x7fad30003f90 2024-09-13T16:29:57.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad49c77700 1 -- 172.21.15.119:0/2630996896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad30003f90 msgr2=0x7fad30004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad49c77700 1 --2- 172.21.15.119:0/2630996896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad30003f90 0x7fad30004870 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fad44160930 tx=0x7fad300058a0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad43fff700 1 -- 172.21.15.119:0/2630996896 reap_dead start 2024-09-13T16:29:57.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad49c77700 1 -- 172.21.15.119:0/2630996896 shutdown_connections 2024-09-13T16:29:57.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad49c77700 1 -- 172.21.15.119:0/2630996896 >> 172.21.15.119:0/2630996896 conn(0x7fad440d2c60 msgr2=0x7fad440d3070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:57.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad49c77700 1 -- 172.21.15.119:0/2630996896 shutdown_connections 2024-09-13T16:29:57.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.223+0000 7fad49c77700 1 -- 172.21.15.119:0/2630996896 wait complete. 2024-09-13T16:29:57.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 Processor -- start 2024-09-13T16:29:57.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 -- start start 2024-09-13T16:29:57.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad30003f90 0x7fad4415d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad44151eb0 0x7fad4414bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4415fba0 0x7fad44160020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad44160560 con 0x7fad4415fba0 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad440cb8f0 con 0x7fad44151eb0 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad49c77700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad440cace0 con 0x7fad30003f90 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad43fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4415fba0 0x7fad44160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad43fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4415fba0 0x7fad44160020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51514/0 (socket says 172.21.15.119:51514) 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad43fff700 1 -- 172.21.15.119:0/4261796808 learned_addr learned my addr 172.21.15.119:0/4261796808 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad437fe700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad30003f90 0x7fad4415d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.227+0000 7fad3bfff700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad44151eb0 0x7fad4414bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad43fff700 1 -- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad30003f90 msgr2=0x7fad4415d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad43fff700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad30003f90 0x7fad4415d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad43fff700 1 -- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad44151eb0 msgr2=0x7fad4414bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad43fff700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad44151eb0 0x7fad4414bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad43fff700 1 -- 172.21.15.119:0/4261796808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440cafc0 con 0x7fad4415fba0 2024-09-13T16:29:57.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad43fff700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4415fba0 0x7fad44160020 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fad440b5810 tx=0x7fad3c0021f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad40ff9700 1 -- 172.21.15.119:0/4261796808 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c003c30 con 0x7fad4415fba0 2024-09-13T16:29:57.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad440cb2a0 con 0x7fad4415fba0 2024-09-13T16:29:57.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad40ff9700 1 -- 172.21.15.119:0/4261796808 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c003d90 con 0x7fad4415fba0 2024-09-13T16:29:57.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad40ff9700 1 -- 172.21.15.119:0/4261796808 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c00db00 con 0x7fad4415fba0 2024-09-13T16:29:57.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad441f1730 con 0x7fad4415fba0 2024-09-13T16:29:57.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad4415d970 con 0x7fad4415fba0 2024-09-13T16:29:57.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad40ff9700 1 -- 172.21.15.119:0/4261796808 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fad3c00dc60 con 0x7fad4415fba0 2024-09-13T16:29:57.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad40ff9700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad24066760 0x7fad24068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.231+0000 7fad40ff9700 1 -- 172.21.15.119:0/4261796808 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fad3c084350 con 0x7fad4415fba0 2024-09-13T16:29:57.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.235+0000 7fad437fe700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad24066760 0x7fad24068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.235+0000 7fad437fe700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad24066760 0x7fad24068c20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fad34009c70 tx=0x7fad34009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.239+0000 7fad40ff9700 1 -- 172.21.15.119:0/4261796808 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fad3c0514a0 con 0x7fad4415fba0 2024-09-13T16:29:57.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:29:57.402 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:29:57.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.407+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad440ce380 con 0x7fad4415fba0 2024-09-13T16:29:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad40ff9700 1 -- 172.21.15.119:0/4261796808 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16966 (secure 0 0 0) 0x7fad3c054ac0 con 0x7fad4415fba0 2024-09-13T16:29:57.416 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:57.417 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":83,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:48.943212+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/491709455":"2024-09-14T16:29:48.943151+0000","172.21.15.119:6834/491709455":"2024-09-14T16:29:48.943151+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:57.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad24066760 msgr2=0x7fad24068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad24066760 0x7fad24068c20 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fad34009c70 tx=0x7fad34009240 comp rx=0 tx=0).stop 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4415fba0 msgr2=0x7fad44160020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4415fba0 0x7fad44160020 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fad440b5810 tx=0x7fad3c0021f0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 shutdown_connections 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad24066760 0x7fad24068c20 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4415fba0 0x7fad44160020 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad44151eb0 0x7fad4414bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 --2- 172.21.15.119:0/4261796808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad30003f90 0x7fad4415d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 >> 172.21.15.119:0/4261796808 conn(0x7fad440d2c60 msgr2=0x7fad440b5bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 shutdown_connections 2024-09-13T16:29:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.411+0000 7fad49c77700 1 -- 172.21.15.119:0/4261796808 wait complete. 2024-09-13T16:29:57.430 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:29:28.552550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:29:29.210253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:57.431 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:57.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cb59e700 1 Processor -- start 2024-09-13T16:29:57.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cb59e700 1 -- start start 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cb59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe3cc15ccb0 0x7fe3cc0d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0b5670 0x7fe3cc15e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cb59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3cc0b5bb0 con 0x7fe3cc0d0da0 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cb59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3cc0d16c0 con 0x7fe3cc15ccb0 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc0d1210 con 0x7fe3cc0b5670 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3ca59c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe3cc15ccb0 0x7fe3cc0d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cad9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0b5670 0x7fe3cc15e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3ca59c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe3cc15ccb0 0x7fe3cc0d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:53684/0 (socket says 172.21.15.119:53684) 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3cad9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0b5670 0x7fe3cc15e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:48168/0 (socket says 172.21.15.119:48168) 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.535+0000 7fe3ca59c700 1 -- 172.21.15.119:0/2991065668 learned_addr learned my addr 172.21.15.119:0/2991065668 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:57.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3c959a700 1 -- 172.21.15.119:0/2991065668 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399294571 0 0) 0x7fe3cc0b5bb0 con 0x7fe3cc0d0da0 2024-09-13T16:29:57.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3c959a700 1 -- 172.21.15.119:0/2991065668 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3b40035d0 con 0x7fe3cc0d0da0 2024-09-13T16:29:57.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cad9d700 1 -- 172.21.15.119:0/2991065668 >> v2:172.21.15.145:3300/0 conn(0x7fe3cc15ccb0 msgr2=0x7fe3cc0d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:57.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cad9d700 1 --2- 172.21.15.119:0/2991065668 >> v2:172.21.15.145:3300/0 conn(0x7fe3cc15ccb0 0x7fe3cc0d0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cad9d700 1 -- 172.21.15.119:0/2991065668 >> v1:172.21.15.145:6789/0 conn(0x7fe3cc0d0da0 legacy=0x7fe3cc0b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cad9d700 1 -- 172.21.15.119:0/2991065668 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3cc15d3e0 con 0x7fe3cc0b5670 2024-09-13T16:29:57.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3ca59c700 1 --2- 172.21.15.119:0/2991065668 >> v2:172.21.15.145:3300/0 conn(0x7fe3cc15ccb0 0x7fe3cc0d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:57.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cad9d700 1 --2- 172.21.15.119:0/2991065668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0b5670 0x7fe3cc15e1a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009d90 tx=0x7fe3c400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ba138537c6f6de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3c959a700 1 -- 172.21.15.119:0/2991065668 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c400e070 con 0x7fe3cc0b5670 2024-09-13T16:29:57.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3c959a700 1 -- 172.21.15.119:0/2991065668 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3c400bb20 con 0x7fe3cc0b5670 2024-09-13T16:29:57.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3c959a700 1 -- 172.21.15.119:0/2991065668 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3c4012430 con 0x7fe3cc0b5670 2024-09-13T16:29:57.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 -- 172.21.15.119:0/2991065668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0b5670 msgr2=0x7fe3cc15e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/2991065668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0b5670 0x7fe3cc15e1a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fe3c4009d90 tx=0x7fe3c400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:57.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 -- 172.21.15.119:0/2991065668 shutdown_connections 2024-09-13T16:29:57.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/2991065668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0b5670 0x7fe3cc15e1a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/2991065668 >> v2:172.21.15.145:3300/0 conn(0x7fe3cc15ccb0 0x7fe3cc0d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 -- 172.21.15.119:0/2991065668 >> 172.21.15.119:0/2991065668 conn(0x7fe3cc072240 msgr2=0x7fe3cc099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:57.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 -- 172.21.15.119:0/2991065668 shutdown_connections 2024-09-13T16:29:57.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.539+0000 7fe3cb59e700 1 -- 172.21.15.119:0/2991065668 wait complete. 2024-09-13T16:29:57.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 Processor -- start 2024-09-13T16:29:57.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 -- start start 2024-09-13T16:29:57.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3cc0b5670 0x7fe3cc14f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0d0da0 0x7fe3cc153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3cc15ccb0 0x7fe3cc1567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc1557d0 con 0x7fe3cc0b5670 2024-09-13T16:29:57.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc14fbd0 con 0x7fe3cc15ccb0 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3cc155bf0 con 0x7fe3cc0d0da0 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3cc0b5670 0x7fe3cc14f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3cc0b5670 0x7fe3cc14f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51516/0 (socket says 172.21.15.119:51516) 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 -- 172.21.15.119:0/3143881157 learned_addr learned my addr 172.21.15.119:0/3143881157 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cad9d700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3cc15ccb0 0x7fe3cc1567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3c9d9b700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0d0da0 0x7fe3cc153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 -- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0d0da0 msgr2=0x7fe3cc153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0d0da0 0x7fe3cc153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 -- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3cc15ccb0 msgr2=0x7fe3cc1567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3cc15ccb0 0x7fe3cc1567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 -- 172.21.15.119:0/3143881157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3bc009570 con 0x7fe3cc0b5670 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cad9d700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3cc15ccb0 0x7fe3cc1567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:57.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3ca59c700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3cc0b5670 0x7fe3cc14f690 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00ea00 tx=0x7fe3bc00ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3baffd700 1 -- 172.21.15.119:0/3143881157 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00eea0 con 0x7fe3cc0b5670 2024-09-13T16:29:57.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3baffd700 1 -- 172.21.15.119:0/3143881157 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3bc00c850 con 0x7fe3cc0b5670 2024-09-13T16:29:57.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3baffd700 1 -- 172.21.15.119:0/3143881157 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3bc00cbc0 con 0x7fe3cc0b5670 2024-09-13T16:29:57.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c4009a70 con 0x7fe3cc0b5670 2024-09-13T16:29:57.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.543+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3cc154e30 con 0x7fe3cc0b5670 2024-09-13T16:29:57.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.547+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3cc155a40 con 0x7fe3cc0b5670 2024-09-13T16:29:57.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.547+0000 7fe3baffd700 1 -- 172.21.15.119:0/3143881157 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe3bc00cd20 con 0x7fe3cc0b5670 2024-09-13T16:29:57.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.547+0000 7fe3baffd700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe3ac066730 0x7fe3ac068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.547+0000 7fe3baffd700 1 -- 172.21.15.119:0/3143881157 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe3bc083f10 con 0x7fe3cc0b5670 2024-09-13T16:29:57.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.547+0000 7fe3c9d9b700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe3ac066730 0x7fe3ac068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.551+0000 7fe3c9d9b700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe3ac066730 0x7fe3ac068bf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc06c7c0 tx=0x7fe3c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.555+0000 7fe3baffd700 1 -- 172.21.15.119:0/3143881157 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe3bc0510e0 con 0x7fe3cc0b5670 2024-09-13T16:29:57.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.723+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe3cc0d0860 con 0x7fe3cc0b5670 2024-09-13T16:29:57.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.723+0000 7fe3baffd700 1 -- 172.21.15.119:0/3143881157 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe3bc054700 con 0x7fe3cc0b5670 2024-09-13T16:29:57.730 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe3ac066730 msgr2=0x7fe3ac068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe3ac066730 0x7fe3ac068bf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fe3cc06c7c0 tx=0x7fe3c0009040 comp rx=0 tx=0).stop 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3cc0b5670 msgr2=0x7fe3cc14f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3cc0b5670 0x7fe3cc14f690 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fe3bc00ea00 tx=0x7fe3bc00ed10 comp rx=0 tx=0).stop 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 shutdown_connections 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe3ac066730 0x7fe3ac068bf0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe3cc15ccb0 0x7fe3cc1567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe3cc0d0da0 0x7fe3cc153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 --2- 172.21.15.119:0/3143881157 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3cc0b5670 0x7fe3cc14f690 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 >> 172.21.15.119:0/3143881157 conn(0x7fe3cc072240 msgr2=0x7fe3cc06c840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:57.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 shutdown_connections 2024-09-13T16:29:57.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.727+0000 7fe3cb59e700 1 -- 172.21.15.119:0/3143881157 wait complete. 2024-09-13T16:29:57.745 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:29:57.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6b59e700 1 Processor -- start 2024-09-13T16:29:57.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6b59e700 1 -- start start 2024-09-13T16:29:57.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6b59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7d6c0d17e0 0x7f7d6c167bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6b59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0d0df0 con 0x7f7d6c0ce8c0 2024-09-13T16:29:57.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6b59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0cdb00 con 0x7f7d6c15cc00 2024-09-13T16:29:57.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6b59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d6c0cdde0 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d69d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7d6c0ce8c0 0x7f7d6c0d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56484/0 (socket says 172.21.15.119:56484) 2024-09-13T16:29:57.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d69d9b700 1 -- 172.21.15.119:0/1919487119 learned_addr learned my addr 172.21.15.119:0/1919487119 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:57.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6ad9d700 1 --2- 172.21.15.119:0/1919487119 >> v2:172.21.15.145:3300/0 conn(0x7f7d6c0d17e0 0x7f7d6c167bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6959a700 1 -- 172.21.15.119:0/1919487119 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982813523 0 0) 0x7f7d6c0d0df0 con 0x7f7d6c0ce8c0 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6959a700 1 -- 172.21.15.119:0/1919487119 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d580035d0 con 0x7f7d6c0ce8c0 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6ad9d700 1 -- 172.21.15.119:0/1919487119 >> v1:172.21.15.145:6789/0 conn(0x7f7d6c15cc00 legacy=0x7f7d6c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6ad9d700 1 -- 172.21.15.119:0/1919487119 >> v1:172.21.15.119:6789/0 conn(0x7f7d6c0ce8c0 legacy=0x7f7d6c0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6ad9d700 1 -- 172.21.15.119:0/1919487119 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c0ce0c0 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.851+0000 7f7d6ad9d700 1 --2- 172.21.15.119:0/1919487119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c167bb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009d90 tx=0x7f7d6400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=915e6f73d856f430 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6959a700 1 -- 172.21.15.119:0/1919487119 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400e070 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6959a700 1 -- 172.21.15.119:0/1919487119 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d6400bb20 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6959a700 1 -- 172.21.15.119:0/1919487119 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d6400be90 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 -- 172.21.15.119:0/1919487119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0d17e0 msgr2=0x7f7d6c167bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/1919487119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c167bb0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f7d64009d90 tx=0x7f7d6400b330 comp rx=0 tx=0).stop 2024-09-13T16:29:57.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 -- 172.21.15.119:0/1919487119 shutdown_connections 2024-09-13T16:29:57.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/1919487119 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c167bb0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 -- 172.21.15.119:0/1919487119 >> 172.21.15.119:0/1919487119 conn(0x7f7d6c08c080 msgr2=0x7f7d6c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:57.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 -- 172.21.15.119:0/1919487119 shutdown_connections 2024-09-13T16:29:57.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 -- 172.21.15.119:0/1919487119 wait complete. 2024-09-13T16:29:57.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 Processor -- start 2024-09-13T16:29:57.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.855+0000 7f7d6b59e700 1 -- start start 2024-09-13T16:29:57.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0ce8c0 0x7f7d6c1542d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c1600c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d6c15cc00 0x7f7d6c15e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0cc600 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c15eb40 con 0x7f7d6c0ce8c0 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c15d560 con 0x7f7d6c15cc00 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c1600c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c1600c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51526/0 (socket says 172.21.15.119:51526) 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 -- 172.21.15.119:0/3628286464 learned_addr learned my addr 172.21.15.119:0/3628286464 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6a59c700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0ce8c0 0x7f7d6c1542d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6ad9d700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d6c15cc00 0x7f7d6c15e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 -- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d6c15cc00 msgr2=0x7f7d6c15e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d6c15cc00 0x7f7d6c15e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 -- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0ce8c0 msgr2=0x7f7d6c1542d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0ce8c0 0x7f7d6c1542d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 -- 172.21.15.119:0/3628286464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d64009a70 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d69d9b700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c1600c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005930 tx=0x7f7d5400c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d62ffd700 1 -- 172.21.15.119:0/3628286464 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54005d90 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d62ffd700 1 -- 172.21.15.119:0/3628286464 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d540070d0 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c15d840 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d62ffd700 1 -- 172.21.15.119:0/3628286464 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d54007480 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.859+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6c15de00 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.863+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6c08c560 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.863+0000 7f7d62ffd700 1 -- 172.21.15.119:0/3628286464 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7d540075e0 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.863+0000 7f7d62ffd700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d4c066730 0x7f7d4c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.863+0000 7f7d62ffd700 1 -- 172.21.15.119:0/3628286464 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f7d54083710 con 0x7f7d6c0d17e0 2024-09-13T16:29:57.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.863+0000 7f7d6a59c700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d4c066730 0x7f7d4c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:57.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.863+0000 7f7d6a59c700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d4c066730 0x7f7d4c068bf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c000f60 tx=0x7f7d5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:57.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:57.871+0000 7f7d62ffd700 1 -- 172.21.15.119:0/3628286464 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7d540508e0 con 0x7f7d6c0d17e0 2024-09-13T16:29:58.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7d6c06b990 con 0x7f7d6c0d17e0 2024-09-13T16:29:58.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d62ffd700 1 -- 172.21.15.119:0/3628286464 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d54053f00 con 0x7f7d6c0d17e0 2024-09-13T16:29:58.048 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:29:58.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d4c066730 msgr2=0x7f7d4c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d4c066730 0x7f7d4c068bf0 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c000f60 tx=0x7f7d5c009040 comp rx=0 tx=0).stop 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d6c0d17e0 msgr2=0x7f7d6c1600c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c1600c0 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7f7d54005930 tx=0x7f7d5400c7b0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 shutdown_connections 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d4c066730 0x7f7d4c068bf0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d6c15cc00 0x7f7d6c15e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d6c0d17e0 0x7f7d6c1600c0 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 --2- 172.21.15.119:0/3628286464 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d6c0ce8c0 0x7f7d6c1542d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 >> 172.21.15.119:0/3628286464 conn(0x7f7d6c08c080 msgr2=0x7f7d6c06c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 shutdown_connections 2024-09-13T16:29:58.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.043+0000 7f7d6b59e700 1 -- 172.21.15.119:0/3628286464 wait complete. 2024-09-13T16:29:58.063 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:29:58.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40c4f13700 1 Processor -- start 2024-09-13T16:29:58.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40c4f13700 1 -- start start 2024-09-13T16:29:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40c4f13700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f40c00b30b0 0x7f40c01654d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40c4f13700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c006c8c0 con 0x7f40c0144180 2024-09-13T16:29:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40c4f13700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c006cb40 con 0x7f40c006c480 2024-09-13T16:29:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40c4f13700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c00b1c50 con 0x7f40c00b30b0 2024-09-13T16:29:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40bf7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f40c0144180 0x7f40c00cc010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56488/0 (socket says 172.21.15.119:56488) 2024-09-13T16:29:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40bf7fe700 1 -- 172.21.15.119:0/850680848 learned_addr learned my addr 172.21.15.119:0/850680848 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:58.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.171+0000 7f40bffff700 1 --2- 172.21.15.119:0/850680848 >> v2:172.21.15.145:3300/0 conn(0x7f40c00b30b0 0x7f40c01654d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696946514 0 0) 0x7f40c006c8c0 con 0x7f40c0144180 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40ac0035d0 con 0x7f40c0144180 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2834143563 0 0) 0x7f40c006cb40 con 0x7f40c006c480 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40c006c8c0 con 0x7f40c006c480 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40bffff700 1 -- 172.21.15.119:0/850680848 >> v1:172.21.15.145:6789/0 conn(0x7f40c006c480 legacy=0x7f40c00cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1231051332 0 0) 0x7f40ac0035d0 con 0x7f40c0144180 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40bffff700 1 -- 172.21.15.119:0/850680848 >> v1:172.21.15.119:6789/0 conn(0x7f40c0144180 legacy=0x7f40c00cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40bffff700 1 -- 172.21.15.119:0/850680848 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c006ce20 con 0x7f40c00b30b0 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40bffff700 1 --2- 172.21.15.119:0/850680848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c00b30b0 0x7f40c01654d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009d90 tx=0x7f40b800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0e7bed8b9a8b6bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b800e070 con 0x7f40c00b30b0 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b8003a40 con 0x7f40c00b30b0 2024-09-13T16:29:58.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40be7fc700 1 -- 172.21.15.119:0/850680848 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0001670 con 0x7f40c00b30b0 2024-09-13T16:29:58.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40c4f13700 1 -- 172.21.15.119:0/850680848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c00b30b0 msgr2=0x7f40c01654d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40c4f13700 1 --2- 172.21.15.119:0/850680848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c00b30b0 0x7f40c01654d0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f40b8009d90 tx=0x7f40b800b330 comp rx=0 tx=0).stop 2024-09-13T16:29:58.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40c4f13700 1 -- 172.21.15.119:0/850680848 shutdown_connections 2024-09-13T16:29:58.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40c4f13700 1 --2- 172.21.15.119:0/850680848 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c00b30b0 0x7f40c01654d0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40c4f13700 1 -- 172.21.15.119:0/850680848 >> 172.21.15.119:0/850680848 conn(0x7f40c00b0980 msgr2=0x7f40c00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:58.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40c4f13700 1 -- 172.21.15.119:0/850680848 shutdown_connections 2024-09-13T16:29:58.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.175+0000 7f40c4f13700 1 -- 172.21.15.119:0/850680848 wait complete. 2024-09-13T16:29:58.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 Processor -- start 2024-09-13T16:29:58.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 -- start start 2024-09-13T16:29:58.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006c480 0x7f40c013ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c00b30b0 0x7f40c0139360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c0144180 0x7f40c015a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c013ea00 con 0x7f40c006c480 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c013dd80 con 0x7f40c0144180 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40c4f13700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c013e060 con 0x7f40c00b30b0 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006c480 0x7f40c013ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006c480 0x7f40c013ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51528/0 (socket says 172.21.15.119:51528) 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 -- 172.21.15.119:0/3087514954 learned_addr learned my addr 172.21.15.119:0/3087514954 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40beffd700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c00b30b0 0x7f40c0139360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bffff700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c0144180 0x7f40c015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 -- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c00b30b0 msgr2=0x7f40c0139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c00b30b0 0x7f40c0139360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 -- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c0144180 msgr2=0x7f40c015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c0144180 0x7f40c015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 -- 172.21.15.119:0/3087514954 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b8009a70 con 0x7f40c006c480 2024-09-13T16:29:58.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40beffd700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c00b30b0 0x7f40c0139360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:58.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.179+0000 7f40bf7fe700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006c480 0x7f40c013ee20 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f40b000c6b0 tx=0x7f40b000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40b7fff700 1 -- 172.21.15.119:0/3087514954 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0004ca0 con 0x7f40c006c480 2024-09-13T16:29:58.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40b7fff700 1 -- 172.21.15.119:0/3087514954 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b0004e00 con 0x7f40c006c480 2024-09-13T16:29:58.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c013e340 con 0x7f40c006c480 2024-09-13T16:29:58.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40b7fff700 1 -- 172.21.15.119:0/3087514954 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0007310 con 0x7f40c006c480 2024-09-13T16:29:58.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c013d910 con 0x7f40c006c480 2024-09-13T16:29:58.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c013ec70 con 0x7f40c006c480 2024-09-13T16:29:58.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40b7fff700 1 -- 172.21.15.119:0/3087514954 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f40b0007470 con 0x7f40c006c480 2024-09-13T16:29:58.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40b7fff700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a0066730 0x7f40a0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40b7fff700 1 -- 172.21.15.119:0/3087514954 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f40b0084110 con 0x7f40c006c480 2024-09-13T16:29:58.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.183+0000 7f40beffd700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a0066730 0x7f40a0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.187+0000 7f40beffd700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a0066730 0x7f40a0068bf0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f40c00a9100 tx=0x7f40a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.191+0000 7f40b7fff700 1 -- 172.21.15.119:0/3087514954 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f40b00511c0 con 0x7f40c006c480 2024-09-13T16:29:58.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.363+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f40c006cb40 con 0x7f40c006c480 2024-09-13T16:29:58.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.363+0000 7f40b7fff700 1 -- 172.21.15.119:0/3087514954 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f40b0007720 con 0x7f40c006c480 2024-09-13T16:29:58.369 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:29:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a0066730 msgr2=0x7f40a0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a0066730 0x7f40a0068bf0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f40c00a9100 tx=0x7f40a8009040 comp rx=0 tx=0).stop 2024-09-13T16:29:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006c480 msgr2=0x7f40c013ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006c480 0x7f40c013ee20 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7f40b000c6b0 tx=0x7f40b000c9c0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 shutdown_connections 2024-09-13T16:29:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f40a0066730 0x7f40a0068bf0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f40c0144180 0x7f40c015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f40c00b30b0 0x7f40c0139360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 --2- 172.21.15.119:0/3087514954 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f40c006c480 0x7f40c013ee20 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 >> 172.21.15.119:0/3087514954 conn(0x7f40c00b0980 msgr2=0x7f40c015d340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 shutdown_connections 2024-09-13T16:29:58.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.367+0000 7f40c4f13700 1 -- 172.21.15.119:0/3087514954 wait complete. 2024-09-13T16:29:58.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:29:58.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.491+0000 7f2abbfff700 1 Processor -- start 2024-09-13T16:29:58.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.491+0000 7f2abbfff700 1 -- start start 2024-09-13T16:29:58.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4049cd0 0x7f2ab4007fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab404a140 con 0x7f2ab4056150 2024-09-13T16:29:58.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ab40ea9f0 con 0x7f2ab400be50 2024-09-13T16:29:58.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab4055120 con 0x7f2ab4049cd0 2024-09-13T16:29:58.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2aba7fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2ab4056150 0x7f2ab40078c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56496/0 (socket says 172.21.15.119:56496) 2024-09-13T16:29:58.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2aba7fc700 1 -- 172.21.15.119:0/1298065248 learned_addr learned my addr 172.21.15.119:0/1298065248 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:58.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abb7fe700 1 --2- 172.21.15.119:0/1298065248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4049cd0 0x7f2ab4007fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2ab9ffb700 1 -- 172.21.15.119:0/1298065248 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623539450 0 0) 0x7f2ab404a140 con 0x7f2ab4056150 2024-09-13T16:29:58.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2ab9ffb700 1 -- 172.21.15.119:0/1298065248 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2aa40035d0 con 0x7f2ab4056150 2024-09-13T16:29:58.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abb7fe700 1 -- 172.21.15.119:0/1298065248 >> v1:172.21.15.145:6789/0 conn(0x7f2ab400be50 legacy=0x7f2ab400c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abb7fe700 1 -- 172.21.15.119:0/1298065248 >> v1:172.21.15.119:6789/0 conn(0x7f2ab4056150 legacy=0x7f2ab40078c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abb7fe700 1 -- 172.21.15.119:0/1298065248 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4055400 con 0x7f2ab4049cd0 2024-09-13T16:29:58.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abb7fe700 1 --2- 172.21.15.119:0/1298065248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4049cd0 0x7f2ab4007fd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2abc04ff70 tx=0x7f2abc068790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c8716955f9b66a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2ab9ffb700 1 -- 172.21.15.119:0/1298065248 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc067690 con 0x7f2ab4049cd0 2024-09-13T16:29:58.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2ab9ffb700 1 -- 172.21.15.119:0/1298065248 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc04f010 con 0x7f2ab4049cd0 2024-09-13T16:29:58.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2ab9ffb700 1 -- 172.21.15.119:0/1298065248 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc067b90 con 0x7f2ab4049cd0 2024-09-13T16:29:58.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- 172.21.15.119:0/1298065248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4049cd0 msgr2=0x7f2ab4007fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 --2- 172.21.15.119:0/1298065248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4049cd0 0x7f2ab4007fd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f2abc04ff70 tx=0x7f2abc068790 comp rx=0 tx=0).stop 2024-09-13T16:29:58.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- 172.21.15.119:0/1298065248 shutdown_connections 2024-09-13T16:29:58.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 --2- 172.21.15.119:0/1298065248 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4049cd0 0x7f2ab4007fd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- 172.21.15.119:0/1298065248 >> 172.21.15.119:0/1298065248 conn(0x7f2ab4056860 msgr2=0x7f2ab4056c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:58.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- 172.21.15.119:0/1298065248 shutdown_connections 2024-09-13T16:29:58.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.495+0000 7f2abbfff700 1 -- 172.21.15.119:0/1298065248 wait complete. 2024-09-13T16:29:58.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 Processor -- start 2024-09-13T16:29:58.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 -- start start 2024-09-13T16:29:58.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab400be50 0x7f2ab4036bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab4049cd0 0x7f2ab40e09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4056150 0x7f2ab40e8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab4037420 con 0x7f2ab4049cd0 2024-09-13T16:29:58.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab4037700 con 0x7f2ab400be50 2024-09-13T16:29:58.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ab40379e0 con 0x7f2ab4056150 2024-09-13T16:29:58.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2aba7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab4049cd0 0x7f2ab40e09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abb7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4056150 0x7f2ab40e8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2aba7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab4049cd0 0x7f2ab40e09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51544/0 (socket says 172.21.15.119:51544) 2024-09-13T16:29:58.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2aba7fc700 1 -- 172.21.15.119:0/3929959384 learned_addr learned my addr 172.21.15.119:0/3929959384 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:58.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abaffd700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab400be50 0x7f2ab4036bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abb7fe700 1 -- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab400be50 msgr2=0x7f2ab4036bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abb7fe700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab400be50 0x7f2ab4036bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abb7fe700 1 -- 172.21.15.119:0/3929959384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab4049cd0 msgr2=0x7f2ab40e09e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abb7fe700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab4049cd0 0x7f2ab40e09e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abb7fe700 1 -- 172.21.15.119:0/3929959384 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ab4037cc0 con 0x7f2ab4056150 2024-09-13T16:29:58.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abb7fe700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4056150 0x7f2ab40e8dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0641b0 tx=0x7f2abc068560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2aa37fe700 1 -- 172.21.15.119:0/3929959384 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc09a070 con 0x7f2ab4056150 2024-09-13T16:29:58.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2aa37fe700 1 -- 172.21.15.119:0/3929959384 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2abc073190 con 0x7f2ab4056150 2024-09-13T16:29:58.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ab4037110 con 0x7f2ab4056150 2024-09-13T16:29:58.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2aa37fe700 1 -- 172.21.15.119:0/3929959384 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2abc072350 con 0x7f2ab4056150 2024-09-13T16:29:58.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.499+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ab40e0f20 con 0x7f2ab4056150 2024-09-13T16:29:58.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.503+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ab40e1520 con 0x7f2ab4056150 2024-09-13T16:29:58.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.503+0000 7f2aa37fe700 1 -- 172.21.15.119:0/3929959384 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2abc0724b0 con 0x7f2ab4056150 2024-09-13T16:29:58.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.503+0000 7f2aa37fe700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa8066760 0x7f2aa8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.503+0000 7f2aa37fe700 1 -- 172.21.15.119:0/3929959384 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2abc0ff1a0 con 0x7f2ab4056150 2024-09-13T16:29:58.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.503+0000 7f2abaffd700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa8066760 0x7f2aa8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.503+0000 7f2abaffd700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa8066760 0x7f2aa8068c20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00017d0 tx=0x7f2ab0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.515+0000 7f2aa37fe700 1 -- 172.21.15.119:0/3929959384 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2abc0cc480 con 0x7f2ab4056150 2024-09-13T16:29:58.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.699+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2ab40e1520 con 0x7f2ab4056150 2024-09-13T16:29:58.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.699+0000 7f2aa37fe700 1 -- 172.21.15.119:0/3929959384 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v83) v1 ==== 80+0+122 (secure 0 0 0) 0x7f2ab40e1520 con 0x7f2ab4056150 2024-09-13T16:29:58.704 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6835/491709455 2024-09-14T16:29:48.943151+0000 2024-09-13T16:29:58.704 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6834/491709455 2024-09-14T16:29:48.943151+0000 2024-09-13T16:29:58.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa8066760 msgr2=0x7f2aa8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa8066760 0x7f2aa8068c20 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f2ab00017d0 tx=0x7f2ab0009040 comp rx=0 tx=0).stop 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4056150 msgr2=0x7f2ab40e8dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4056150 0x7f2ab40e8dc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2abc0641b0 tx=0x7f2abc068560 comp rx=0 tx=0).stop 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 shutdown_connections 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2aa8066760 0x7f2aa8068c20 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ab4056150 0x7f2ab40e8dc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ab4049cd0 0x7f2ab40e09e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 --2- 172.21.15.119:0/3929959384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ab400be50 0x7f2ab4036bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 >> 172.21.15.119:0/3929959384 conn(0x7f2ab4056860 msgr2=0x7f2ab4033450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 shutdown_connections 2024-09-13T16:29:58.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.703+0000 7f2abbfff700 1 -- 172.21.15.119:0/3929959384 wait complete. 2024-09-13T16:29:58.711 INFO:teuthology.orchestra.run.smithi119.stderr:listed 2 entries 2024-09-13T16:29:58.726 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:29:58.726 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:29:58.734 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:29:58.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.835+0000 7fc6effff700 1 Processor -- start 2024-09-13T16:29:58.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.835+0000 7fc6effff700 1 -- start start 2024-09-13T16:29:58.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6effff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc6f006b730 0x7fc6f00cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6effff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f00ceb60 con 0x7fc6f00d2c90 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6effff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f00ce230 con 0x7fc6f015cd60 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6effff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f015d140 con 0x7fc6f006b730 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6ef7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc6f00d2c90 0x7fc6f00d30e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56506/0 (socket says 172.21.15.119:56506) 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6ef7fe700 1 -- 172.21.15.119:0/694282010 learned_addr learned my addr 172.21.15.119:0/694282010 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6e67fc700 1 --2- 172.21.15.119:0/694282010 >> v2:172.21.15.119:3300/0 conn(0x7fc6f006b730 0x7fc6f00cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6ee7fc700 1 -- 172.21.15.119:0/694282010 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237785198 0 0) 0x7fc6f00ceb60 con 0x7fc6f00d2c90 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6e67fc700 1 -- 172.21.15.119:0/694282010 >> v1:172.21.15.145:6789/0 conn(0x7fc6f015cd60 legacy=0x7fc6f00cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6e67fc700 1 -- 172.21.15.119:0/694282010 >> v1:172.21.15.119:6789/0 conn(0x7fc6f00d2c90 legacy=0x7fc6f00d30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6e67fc700 1 -- 172.21.15.119:0/694282010 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f00ce510 con 0x7fc6f006b730 2024-09-13T16:29:58.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6e67fc700 1 --2- 172.21.15.119:0/694282010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f006b730 0x7fc6f00cfad0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009850 tx=0x7fc6d8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d0a2f807b5c3c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6ee7fc700 1 -- 172.21.15.119:0/694282010 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011830 con 0x7fc6f006b730 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6ee7fc700 1 -- 172.21.15.119:0/694282010 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d8011990 con 0x7fc6f006b730 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.839+0000 7fc6ee7fc700 1 -- 172.21.15.119:0/694282010 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011d20 con 0x7fc6f006b730 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 -- 172.21.15.119:0/694282010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f006b730 msgr2=0x7fc6f00cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 --2- 172.21.15.119:0/694282010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f006b730 0x7fc6f00cfad0 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009850 tx=0x7fc6d8009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 -- 172.21.15.119:0/694282010 shutdown_connections 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 --2- 172.21.15.119:0/694282010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f006b730 0x7fc6f00cfad0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 -- 172.21.15.119:0/694282010 >> 172.21.15.119:0/694282010 conn(0x7fc6f008c240 msgr2=0x7fc6f009c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 -- 172.21.15.119:0/694282010 shutdown_connections 2024-09-13T16:29:58.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 -- 172.21.15.119:0/694282010 wait complete. 2024-09-13T16:29:58.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.843+0000 7fc6effff700 1 Processor -- start 2024-09-13T16:29:58.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 -- start start 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b730 0x7fc6f00cc700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f00d2c90 0x7fc6f015f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f015cd60 0x7fc6f015fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0156a10 con 0x7fc6f015cd60 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0160150 con 0x7fc6f00d2c90 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f00cb3c0 con 0x7fc6f006b730 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f015cd60 0x7fc6f015fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f015cd60 0x7fc6f015fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51564/0 (socket says 172.21.15.119:51564) 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 -- 172.21.15.119:0/605141080 learned_addr learned my addr 172.21.15.119:0/605141080 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6eeffd700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b730 0x7fc6f00cc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6e67fc700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f00d2c90 0x7fc6f015f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 -- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b730 msgr2=0x7fc6f00cc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b730 0x7fc6f00cc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 -- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f00d2c90 msgr2=0x7fc6f015f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f00d2c90 0x7fc6f015f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 -- 172.21.15.119:0/605141080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d80094b0 con 0x7fc6f015cd60 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6ef7fe700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f015cd60 0x7fc6f015fc10 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800cca0 tx=0x7fc6e800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6e7fff700 1 -- 172.21.15.119:0/605141080 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e800ab70 con 0x7fc6f015cd60 2024-09-13T16:29:58.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6e7fff700 1 -- 172.21.15.119:0/605141080 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e800acd0 con 0x7fc6f015cd60 2024-09-13T16:29:58.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f00cb660 con 0x7fc6f015cd60 2024-09-13T16:29:58.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6e7fff700 1 -- 172.21.15.119:0/605141080 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8007310 con 0x7fc6f015cd60 2024-09-13T16:29:58.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.847+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f00cbc20 con 0x7fc6f015cd60 2024-09-13T16:29:58.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.851+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f00d30e0 con 0x7fc6f015cd60 2024-09-13T16:29:58.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.851+0000 7fc6e7fff700 1 -- 172.21.15.119:0/605141080 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc6e8007470 con 0x7fc6f015cd60 2024-09-13T16:29:58.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.851+0000 7fc6e7fff700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:58.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.851+0000 7fc6e7fff700 1 -- 172.21.15.119:0/605141080 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fc6e8084100 con 0x7fc6f015cd60 2024-09-13T16:29:58.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.851+0000 7fc6eeffd700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:58.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.851+0000 7fc6eeffd700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0000f60 tx=0x7fc6e0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:58.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:58.859+0000 7fc6e7fff700 1 -- 172.21.15.119:0/605141080 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc6e80512d0 con 0x7fc6f015cd60 2024-09-13T16:29:59.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.031+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6f006b510 con 0x7fc6f015cd60 2024-09-13T16:29:59.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.031+0000 7fc6e7fff700 1 -- 172.21.15.119:0/605141080 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16966 (secure 0 0 0) 0x7fc6e80548f0 con 0x7fc6f015cd60 2024-09-13T16:29:59.039 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:29:59.039 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":83,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:48.943212+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/491709455":"2024-09-14T16:29:48.943151+0000","172.21.15.119:6834/491709455":"2024-09-14T16:29:48.943151+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 msgr2=0x7fc6d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0000f60 tx=0x7fc6e0009040 comp rx=0 tx=0).stop 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f015cd60 msgr2=0x7fc6f015fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f015cd60 0x7fc6f015fc10 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fc6e800cca0 tx=0x7fc6e800a060 comp rx=0 tx=0).stop 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 shutdown_connections 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f015cd60 0x7fc6f015fc10 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f00d2c90 0x7fc6f015f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 --2- 172.21.15.119:0/605141080 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b730 0x7fc6f00cc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 >> 172.21.15.119:0/605141080 conn(0x7fc6f008c240 msgr2=0x7fc6f00b3980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:59.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 shutdown_connections 2024-09-13T16:29:59.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.035+0000 7fc6effff700 1 -- 172.21.15.119:0/605141080 wait complete. 2024-09-13T16:29:59.053 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:29:28.552550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:29:29.210253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:29:59.054 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:29:59.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f2864a00700 1 Processor -- start 2024-09-13T16:29:59.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f2864a00700 1 -- start start 2024-09-13T16:29:59.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f2864a00700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f28600ce8c0 0x7f28600d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f2864a00700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600d1610 0x7f28600b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f2864a00700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600d1a60 con 0x7f286015cc00 2024-09-13T16:29:59.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f2864a00700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28600d0df0 con 0x7f28600ce8c0 2024-09-13T16:29:59.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f2864a00700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28600cdb00 con 0x7f28600d1610 2024-09-13T16:29:59.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f28600ce8c0 0x7f28600d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285ed9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f286015cc00 0x7f28600d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56510/0 (socket says 172.21.15.119:56510) 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285ed9d700 1 -- 172.21.15.119:0/752616611 learned_addr learned my addr 172.21.15.119:0/752616611 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285f59e700 1 --2- 172.21.15.119:0/752616611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600d1610 0x7f28600b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285e59c700 1 -- 172.21.15.119:0/752616611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600d1610 msgr2=0x7f28600b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285e59c700 1 --2- 172.21.15.119:0/752616611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600d1610 0x7f28600b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285e59c700 1 -- 172.21.15.119:0/752616611 >> v1:172.21.15.119:6789/0 conn(0x7f286015cc00 legacy=0x7f28600d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285e59c700 1 -- 172.21.15.119:0/752616611 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600cdde0 con 0x7f28600ce8c0 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.175+0000 7f285e59c700 1 --2- 172.21.15.119:0/752616611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f28600d10d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2850009850 tx=0x7f2850009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f1237288d7fb771 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285dd9b700 1 -- 172.21.15.119:0/752616611 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850011830 con 0x7f28600ce8c0 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285dd9b700 1 -- 172.21.15.119:0/752616611 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2850011990 con 0x7f28600ce8c0 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285dd9b700 1 -- 172.21.15.119:0/752616611 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2850011d20 con 0x7f28600ce8c0 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- 172.21.15.119:0/752616611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 msgr2=0x7f28600d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 --2- 172.21.15.119:0/752616611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f28600d10d0 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f2850009850 tx=0x7f2850009b60 comp rx=0 tx=0).stop 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- 172.21.15.119:0/752616611 shutdown_connections 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 --2- 172.21.15.119:0/752616611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28600d1610 0x7f28600b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 --2- 172.21.15.119:0/752616611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f28600d10d0 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- 172.21.15.119:0/752616611 >> 172.21.15.119:0/752616611 conn(0x7f286008c080 msgr2=0x7f28600d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- 172.21.15.119:0/752616611 shutdown_connections 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- 172.21.15.119:0/752616611 wait complete. 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 Processor -- start 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- start start 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f2860155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28600d1610 0x7f28600c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f286015cc00 0x7f28600c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f286015d630 con 0x7f28600ce8c0 2024-09-13T16:29:59.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28600c8790 con 0x7f28600d1610 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28600c8a70 con 0x7f286015cc00 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f2860155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f2860155520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51592/0 (socket says 172.21.15.119:51592) 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 -- 172.21.15.119:0/2862119702 learned_addr learned my addr 172.21.15.119:0/2862119702 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285f59e700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f286015cc00 0x7f28600c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285e59c700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28600d1610 0x7f28600c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 -- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f286015cc00 msgr2=0x7f28600c8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f286015cc00 0x7f28600c8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 -- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28600d1610 msgr2=0x7f28600c7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28600d1610 0x7f28600c7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 -- 172.21.15.119:0/2862119702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28500094b0 con 0x7f28600ce8c0 2024-09-13T16:29:59.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f285ed9d700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f2860155520 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f285400c410 tx=0x7f285400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f28477fe700 1 -- 172.21.15.119:0/2862119702 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854005d90 con 0x7f28600ce8c0 2024-09-13T16:29:59.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f28477fe700 1 -- 172.21.15.119:0/2862119702 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28540070d0 con 0x7f28600ce8c0 2024-09-13T16:29:59.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286015cfe0 con 0x7f28600ce8c0 2024-09-13T16:29:59.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f28477fe700 1 -- 172.21.15.119:0/2862119702 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2854007460 con 0x7f28600ce8c0 2024-09-13T16:29:59.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.179+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28600cc3a0 con 0x7f28600ce8c0 2024-09-13T16:29:59.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.183+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28600c8be0 con 0x7f28600ce8c0 2024-09-13T16:29:59.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.183+0000 7f28477fe700 1 -- 172.21.15.119:0/2862119702 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f28540075c0 con 0x7f28600ce8c0 2024-09-13T16:29:59.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.183+0000 7f28477fe700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f284c066730 0x7f284c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.183+0000 7f28477fe700 1 -- 172.21.15.119:0/2862119702 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2854083660 con 0x7f28600ce8c0 2024-09-13T16:29:59.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.183+0000 7f285e59c700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f284c066730 0x7f284c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.187+0000 7f285e59c700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f284c066730 0x7f284c068bf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f28600cecc0 tx=0x7f2850008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.191+0000 7f28477fe700 1 -- 172.21.15.119:0/2862119702 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f28540507a0 con 0x7f28600ce8c0 2024-09-13T16:29:59.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.359+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f28600d1a60 con 0x7f28600ce8c0 2024-09-13T16:29:59.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.359+0000 7f28477fe700 1 -- 172.21.15.119:0/2862119702 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f28600d1a60 con 0x7f28600ce8c0 2024-09-13T16:29:59.366 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f284c066730 msgr2=0x7f284c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f284c066730 0x7f284c068bf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f28600cecc0 tx=0x7f2850008030 comp rx=0 tx=0).stop 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 msgr2=0x7f2860155520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f2860155520 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f285400c410 tx=0x7f285400c7d0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 shutdown_connections 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f284c066730 0x7f284c068bf0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f286015cc00 0x7f28600c8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28600d1610 0x7f28600c7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 --2- 172.21.15.119:0/2862119702 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28600ce8c0 0x7f2860155520 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 >> 172.21.15.119:0/2862119702 conn(0x7f286008c080 msgr2=0x7f28601607e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:59.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 shutdown_connections 2024-09-13T16:29:59.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.363+0000 7f2864a00700 1 -- 172.21.15.119:0/2862119702 wait complete. 2024-09-13T16:29:59.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:29:59.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.499+0000 7f9c1674a700 1 Processor -- start 2024-09-13T16:29:59.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.499+0000 7f9c1674a700 1 -- start start 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c1674a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c1015cc00 0x7f9c100d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c1674a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c100cdb00 con 0x7f9c100d0c20 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c1674a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c100cdde0 con 0x7f9c100d17e0 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c1674a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100ce0c0 con 0x7f9c1015cc00 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c14f47700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9c100d0c20 0x7f9c100d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56522/0 (socket says 172.21.15.119:56522) 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c14f47700 1 -- 172.21.15.119:0/2602440048 learned_addr learned my addr 172.21.15.119:0/2602440048 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c15748700 1 --2- 172.21.15.119:0/2602440048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c1015cc00 0x7f9c100d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 994253578 0 0) 0x7f9c100cdb00 con 0x7f9c100d0c20 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bfc0035d0 con 0x7f9c100d0c20 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102950395 0 0) 0x7f9c100cdde0 con 0x7f9c100d17e0 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c100cdb00 con 0x7f9c100d17e0 2024-09-13T16:29:59.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c15748700 1 -- 172.21.15.119:0/2602440048 >> v1:172.21.15.145:6789/0 conn(0x7f9c100d17e0 legacy=0x7f9c100b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c15748700 1 -- 172.21.15.119:0/2602440048 >> v1:172.21.15.119:6789/0 conn(0x7f9c100d0c20 legacy=0x7f9c100d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1690966976 0 0) 0x7f9bfc0035d0 con 0x7f9c100d0c20 2024-09-13T16:29:59.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c15748700 1 -- 172.21.15.119:0/2602440048 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c100ce3a0 con 0x7f9c1015cc00 2024-09-13T16:29:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c15748700 1 --2- 172.21.15.119:0/2602440048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c1015cc00 0x7f9c100d06e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009880 tx=0x7f9c0c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78666976b5942525 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c00001670 con 0x7f9c1015cc00 2024-09-13T16:29:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c004030 con 0x7f9c1015cc00 2024-09-13T16:29:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c07fff700 1 -- 172.21.15.119:0/2602440048 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c0043c0 con 0x7f9c1015cc00 2024-09-13T16:29:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c1674a700 1 -- 172.21.15.119:0/2602440048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c1015cc00 msgr2=0x7f9c100d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c1674a700 1 --2- 172.21.15.119:0/2602440048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c1015cc00 0x7f9c100d06e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c009880 tx=0x7f9c0c009b90 comp rx=0 tx=0).stop 2024-09-13T16:29:59.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.503+0000 7f9c1674a700 1 -- 172.21.15.119:0/2602440048 shutdown_connections 2024-09-13T16:29:59.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.507+0000 7f9c1674a700 1 --2- 172.21.15.119:0/2602440048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c1015cc00 0x7f9c100d06e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.507+0000 7f9c1674a700 1 -- 172.21.15.119:0/2602440048 >> 172.21.15.119:0/2602440048 conn(0x7f9c1008c080 msgr2=0x7f9c100d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:59.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.507+0000 7f9c1674a700 1 -- 172.21.15.119:0/2602440048 shutdown_connections 2024-09-13T16:29:59.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.507+0000 7f9c1674a700 1 -- 172.21.15.119:0/2602440048 wait complete. 2024-09-13T16:29:59.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.507+0000 7f9c1674a700 1 Processor -- start 2024-09-13T16:29:59.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.507+0000 7f9c1674a700 1 -- start start 2024-09-13T16:29:59.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100d0c20 0x7f9c10155eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c100d17e0 0x7f9c100c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c1015cc00 0x7f9c10156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100c66b0 con 0x7f9c100d0c20 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c101563f0 con 0x7f9c1015cc00 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c100ca070 con 0x7f9c100d17e0 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100d0c20 0x7f9c10155eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100d0c20 0x7f9c10155eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51608/0 (socket says 172.21.15.119:51608) 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 -- 172.21.15.119:0/1397237034 learned_addr learned my addr 172.21.15.119:0/1397237034 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c14f47700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c100d17e0 0x7f9c100c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15f49700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c1015cc00 0x7f9c10156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 -- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c100d17e0 msgr2=0x7f9c100c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c100d17e0 0x7f9c100c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 -- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c1015cc00 msgr2=0x7f9c10156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c1015cc00 0x7f9c10156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 -- 172.21.15.119:0/1397237034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c0094b0 con 0x7f9c100d0c20 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c15748700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100d0c20 0x7f9c10155eb0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0097d0 tx=0x7f9c0c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c05ffb700 1 -- 172.21.15.119:0/1397237034 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c003c50 con 0x7f9c100d0c20 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c05ffb700 1 -- 172.21.15.119:0/1397237034 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c003db0 con 0x7f9c100d0c20 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c100ca350 con 0x7f9c100d0c20 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c05ffb700 1 -- 172.21.15.119:0/1397237034 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c0c011870 con 0x7f9c100d0c20 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.511+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c100ca910 con 0x7f9c100d0c20 2024-09-13T16:29:59.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.515+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c100c67f0 con 0x7f9c100d0c20 2024-09-13T16:29:59.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.515+0000 7f9c05ffb700 1 -- 172.21.15.119:0/1397237034 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9c0c018050 con 0x7f9c100d0c20 2024-09-13T16:29:59.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.515+0000 7f9c05ffb700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.515+0000 7f9c05ffb700 1 -- 172.21.15.119:0/1397237034 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f9c0c084d30 con 0x7f9c100d0c20 2024-09-13T16:29:59.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.515+0000 7f9c14f47700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.515+0000 7f9c14f47700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9c00001500 tx=0x7f9c00001490 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.523+0000 7f9c05ffb700 1 -- 172.21.15.119:0/1397237034 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9c100c67f0 con 0x7f9c100d0c20 2024-09-13T16:29:59.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9c100c67f0 con 0x7f9c100d0c20 2024-09-13T16:29:59.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c05ffb700 1 -- 172.21.15.119:0/1397237034 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9c0c02b020 con 0x7f9c100d0c20 2024-09-13T16:29:59.696 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:29:59.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 msgr2=0x7f9bf4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f9c00001500 tx=0x7f9c00001490 comp rx=0 tx=0).stop 2024-09-13T16:29:59.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100d0c20 msgr2=0x7f9c10155eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100d0c20 0x7f9c10155eb0 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f9c0c0097d0 tx=0x7f9c0c004770 comp rx=0 tx=0).stop 2024-09-13T16:29:59.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 shutdown_connections 2024-09-13T16:29:59.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9bf4066730 0x7f9bf4068bf0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c1015cc00 0x7f9c10156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c100d17e0 0x7f9c100c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 --2- 172.21.15.119:0/1397237034 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c100d0c20 0x7f9c10155eb0 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 >> 172.21.15.119:0/1397237034 conn(0x7f9c1008c080 msgr2=0x7f9c100ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:59.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.691+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 shutdown_connections 2024-09-13T16:29:59.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.695+0000 7f9c1674a700 1 -- 172.21.15.119:0/1397237034 wait complete. 2024-09-13T16:29:59.711 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:29:59.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.823+0000 7fb3ca638700 1 Processor -- start 2024-09-13T16:29:59.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.823+0000 7fb3ca638700 1 -- start start 2024-09-13T16:29:59.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc00a5b0 0x7fb3bc0e18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb3bc01c740 0x7fb3bc01cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3bc0e1e00 con 0x7fb3bc005b10 2024-09-13T16:29:59.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3bc006ce0 con 0x7fb3bc01c740 2024-09-13T16:29:59.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0060d0 con 0x7fb3bc00a5b0 2024-09-13T16:29:59.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c8e35700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb3bc005b10 0x7fb3bc00a990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56536/0 (socket says 172.21.15.119:56536) 2024-09-13T16:29:59.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c8e35700 1 -- 172.21.15.119:0/1315832634 learned_addr learned my addr 172.21.15.119:0/1315832634 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c9e37700 1 --2- 172.21.15.119:0/1315832634 >> v2:172.21.15.145:3300/0 conn(0x7fb3bc01c740 0x7fb3bc01cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c9636700 1 --2- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc00a5b0 0x7fb3bc0e18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3b7fff700 1 -- 172.21.15.119:0/1315832634 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332937608 0 0) 0x7fb3b8001110 con 0x7fb3bc005b10 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3b7fff700 1 -- 172.21.15.119:0/1315832634 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b00035d0 con 0x7fb3bc005b10 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c9e37700 1 -- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc00a5b0 msgr2=0x7fb3bc0e18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c9e37700 1 --2- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc00a5b0 0x7fb3bc0e18c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c9e37700 1 -- 172.21.15.119:0/1315832634 >> v1:172.21.15.119:6789/0 conn(0x7fb3bc005b10 legacy=0x7fb3bc00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c9e37700 1 -- 172.21.15.119:0/1315832634 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc0063b0 con 0x7fb3bc01c740 2024-09-13T16:29:59.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3c9e37700 1 --2- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc01c740 0x7fb3bc01cbf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb3c404ff70 tx=0x7fb3c4068790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9504b60a7cb3851 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3b7fff700 1 -- 172.21.15.119:0/1315832634 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4067690 con 0x7fb3bc01c740 2024-09-13T16:29:59.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3b7fff700 1 -- 172.21.15.119:0/1315832634 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c404f010 con 0x7fb3bc01c740 2024-09-13T16:29:59.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3b7fff700 1 -- 172.21.15.119:0/1315832634 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c4067bb0 con 0x7fb3bc01c740 2024-09-13T16:29:59.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc01c740 msgr2=0x7fb3bc01cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc01c740 0x7fb3bc01cbf0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb3c404ff70 tx=0x7fb3c4068790 comp rx=0 tx=0).stop 2024-09-13T16:29:59.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- 172.21.15.119:0/1315832634 shutdown_connections 2024-09-13T16:29:59.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc01c740 0x7fb3bc01cbf0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1315832634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc00a5b0 0x7fb3bc0e18c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- 172.21.15.119:0/1315832634 >> 172.21.15.119:0/1315832634 conn(0x7fb3bc036140 msgr2=0x7fb3bc036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:29:59.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- 172.21.15.119:0/1315832634 shutdown_connections 2024-09-13T16:29:59.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.827+0000 7fb3ca638700 1 -- 172.21.15.119:0/1315832634 wait complete. 2024-09-13T16:29:59.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 Processor -- start 2024-09-13T16:29:59.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 -- start start 2024-09-13T16:29:59.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc005b10 0x7fb3bc0e3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3bc00a5b0 0x7fb3bc0d78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc01c740 0x7fb3bc04c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc04db60 con 0x7fb3bc00a5b0 2024-09-13T16:29:59.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0d80b0 con 0x7fb3bc005b10 2024-09-13T16:29:59.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3ca638700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc0e3e30 con 0x7fb3bc01c740 2024-09-13T16:29:59.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c8e35700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3bc00a5b0 0x7fb3bc0d78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c8e35700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3bc00a5b0 0x7fb3bc0d78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51612/0 (socket says 172.21.15.119:51612) 2024-09-13T16:29:59.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c8e35700 1 -- 172.21.15.119:0/1744306512 learned_addr learned my addr 172.21.15.119:0/1744306512 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:29:59.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9e37700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc01c740 0x7fb3bc04c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9636700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc005b10 0x7fb3bc0e3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9636700 1 -- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc01c740 msgr2=0x7fb3bc04c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9636700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc01c740 0x7fb3bc04c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9636700 1 -- 172.21.15.119:0/1744306512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3bc00a5b0 msgr2=0x7fb3bc0d78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:29:59.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9636700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3bc00a5b0 0x7fb3bc0d78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:29:59.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9636700 1 -- 172.21.15.119:0/1744306512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc04d510 con 0x7fb3bc005b10 2024-09-13T16:29:59.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c9e37700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc01c740 0x7fb3bc04c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:59.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.831+0000 7fb3c8e35700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3bc00a5b0 0x7fb3bc0d78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:29:59.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3c9636700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc005b10 0x7fb3bc0e3850 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000e900 tx=0x7fb3c000ec10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3b5ffb700 1 -- 172.21.15.119:0/1744306512 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0013680 con 0x7fb3bc005b10 2024-09-13T16:29:59.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3b5ffb700 1 -- 172.21.15.119:0/1744306512 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3c00137e0 con 0x7fb3bc005b10 2024-09-13T16:29:59.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc04d7f0 con 0x7fb3bc005b10 2024-09-13T16:29:59.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3b5ffb700 1 -- 172.21.15.119:0/1744306512 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3c0013b50 con 0x7fb3bc005b10 2024-09-13T16:29:59.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc0e3060 con 0x7fb3bc005b10 2024-09-13T16:29:59.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc04d9b0 con 0x7fb3bc005b10 2024-09-13T16:29:59.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3b5ffb700 1 -- 172.21.15.119:0/1744306512 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb3c0013cb0 con 0x7fb3bc005b10 2024-09-13T16:29:59.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.835+0000 7fb3b5ffb700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3a8066730 0x7fb3a8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:29:59.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.839+0000 7fb3b5ffb700 1 -- 172.21.15.119:0/1744306512 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fb3c0084720 con 0x7fb3bc005b10 2024-09-13T16:29:59.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.839+0000 7fb3c8e35700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3a8066730 0x7fb3a8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:29:59.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.839+0000 7fb3c8e35700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3a8066730 0x7fb3a8068bf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8007fd0 tx=0x7fb3b8007ce0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:29:59.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:29:59.851+0000 7fb3b5ffb700 1 -- 172.21.15.119:0/1744306512 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb3c00518f0 con 0x7fb3bc005b10 2024-09-13T16:29:59.936 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:29:59.931+0000 7fed948e1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-09-13T16:30:00.000 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:29:59.995+0000 7fed948e1700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 MDSs report damaged metadata 2024-09-13T16:30:00.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.023+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb3bc0e18c0 con 0x7fb3bc005b10 2024-09-13T16:30:00.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.023+0000 7fb3b5ffb700 1 -- 172.21.15.119:0/1744306512 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb3c00210a0 con 0x7fb3bc005b10 2024-09-13T16:30:00.030 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3a8066730 msgr2=0x7fb3a8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3a8066730 0x7fb3a8068bf0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fb3b8007fd0 tx=0x7fb3b8007ce0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc005b10 msgr2=0x7fb3bc0e3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc005b10 0x7fb3bc0e3850 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fb3c000e900 tx=0x7fb3c000ec10 comp rx=0 tx=0).stop 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 shutdown_connections 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb3a8066730 0x7fb3a8068bf0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb3bc01c740 0x7fb3bc04c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb3bc00a5b0 0x7fb3bc0d78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 --2- 172.21.15.119:0/1744306512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb3bc005b10 0x7fb3bc0e3850 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 >> 172.21.15.119:0/1744306512 conn(0x7fb3bc036140 msgr2=0x7fb3bc035660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:00.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 shutdown_connections 2024-09-13T16:30:00.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.027+0000 7fb3ca638700 1 -- 172.21.15.119:0/1744306512 wait complete. 2024-09-13T16:30:00.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:30:00.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.147+0000 7f3114e38700 1 Processor -- start 2024-09-13T16:30:00.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.147+0000 7f3114e38700 1 -- start start 2024-09-13T16:30:00.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f3114e38700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f31101657b0 0x7f3110167ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f3114e38700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100ceb60 con 0x7f311006b730 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f3114e38700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31100d01e0 con 0x7f311015cd60 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f3114e38700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f311015d140 con 0x7f31101657b0 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310effd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f311006b730 0x7f31100cfad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56548/0 (socket says 172.21.15.119:56548) 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310effd700 1 -- 172.21.15.119:0/1627216010 learned_addr learned my addr 172.21.15.119:0/1627216010 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 --2- 172.21.15.119:0/1627216010 >> v2:172.21.15.145:3300/0 conn(0x7f31101657b0 0x7f3110167ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028392085 0 0) 0x7f31100ceb60 con 0x7f311006b730 2024-09-13T16:30:00.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30f80035d0 con 0x7f311006b730 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568116130 0 0) 0x7f31100d01e0 con 0x7f311015cd60 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31100ceb60 con 0x7f311015cd60 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2255215941 0 0) 0x7f30f80035d0 con 0x7f311006b730 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 >> v2:172.21.15.145:3300/0 conn(0x7f31101657b0 msgr2=0x7f3110167ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 --2- 172.21.15.119:0/1627216010 >> v2:172.21.15.145:3300/0 conn(0x7f31101657b0 0x7f3110167ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 >> v1:172.21.15.145:6789/0 conn(0x7f311015cd60 legacy=0x7f31100cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100ce230 con 0x7f311006b730 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f3100001670 con 0x7f311006b730 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 >> v1:172.21.15.119:6789/0 conn(0x7f311006b730 legacy=0x7f31100cfad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30f8003bb0 0x7f30f8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8004870 0x7f30f8004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8005230 0x7f30f800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f311015d140 con 0x7f30f8005230 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f800c0b0 con 0x7f30f8004870 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30f800c270 con 0x7f30f8003bb0 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310effd700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8004870 0x7f30f8004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8005230 0x7f30f800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310f7fe700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30f8003bb0 0x7f30f8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 -- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30f8003bb0 msgr2=0x7f30f8003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f30f8003bb0 0x7f30f8003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 -- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8004870 msgr2=0x7f30f8004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8004870 0x7f30f8004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 -- 172.21.15.119:0/1627216010 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30f80035d0 con 0x7f30f8005230 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310effd700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8004870 0x7f30f8004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 -- 172.21.15.119:0/1627216010 reap_dead start 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310ffff700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8005230 0x7f30f800ba30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3108009fc0 tx=0x7f30f80058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cac61e6e7edab7ed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108003a40 con 0x7f30f8005230 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310800f040 con 0x7f30f8005230 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f310e7fc700 1 -- 172.21.15.119:0/1627216010 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3108003dc0 con 0x7f30f8005230 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f3114e38700 1 -- 172.21.15.119:0/1627216010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8005230 msgr2=0x7f30f800ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.151+0000 7f3114e38700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8005230 0x7f30f800ba30 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f3108009fc0 tx=0x7f30f80058a0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- 172.21.15.119:0/1627216010 shutdown_connections 2024-09-13T16:30:00.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 --2- 172.21.15.119:0/1627216010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f30f8005230 0x7f30f800ba30 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- 172.21.15.119:0/1627216010 >> 172.21.15.119:0/1627216010 conn(0x7f311008c240 msgr2=0x7f311009c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:00.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- 172.21.15.119:0/1627216010 shutdown_connections 2024-09-13T16:30:00.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- 172.21.15.119:0/1627216010 wait complete. 2024-09-13T16:30:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 Processor -- start 2024-09-13T16:30:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- start start 2024-09-13T16:30:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8005230 0x7f31101543d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3110153ea0 0x7f3110153680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31100b2b70 0x7f31100b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100b3530 con 0x7f3110153ea0 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31100caf10 con 0x7f30f8005230 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31100c9e70 con 0x7f31100b2b70 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3110153ea0 0x7f3110153680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3110153ea0 0x7f3110153680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51636/0 (socket says 172.21.15.119:51636) 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 -- 172.21.15.119:0/2636528292 learned_addr learned my addr 172.21.15.119:0/2636528292 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310ffff700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31100b2b70 0x7f31100b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310f7fe700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8005230 0x7f31101543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 -- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31100b2b70 msgr2=0x7f31100b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31100b2b70 0x7f31100b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 -- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8005230 msgr2=0x7f31101543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8005230 0x7f31101543d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 -- 172.21.15.119:0/2636528292 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31100ca150 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f310effd700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3110153ea0 0x7f3110153680 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3100007340 tx=0x7f3100007650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f30f7fff700 1 -- 172.21.15.119:0/2636528292 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31000077d0 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f30f7fff700 1 -- 172.21.15.119:0/2636528292 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f310000f870 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31100ca430 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f30f7fff700 1 -- 172.21.15.119:0/2636528292 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f310000fc20 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.155+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31100ca9f0 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.159+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31100ceff0 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.159+0000 7f30f7fff700 1 -- 172.21.15.119:0/2636528292 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f310000fd80 con 0x7f3110153ea0 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.159+0000 7f30f7fff700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f30fc066760 0x7f30fc068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.159+0000 7f30f7fff700 1 -- 172.21.15.119:0/2636528292 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f31000842c0 con 0x7f3110153ea0 2024-09-13T16:30:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.159+0000 7f310f7fe700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f30fc066760 0x7f30fc068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.159+0000 7f310f7fe700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f30fc066760 0x7f30fc068c20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3104007fd0 tx=0x7f3104009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:00.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.167+0000 7f30f7fff700 1 -- 172.21.15.119:0/2636528292 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f31000514a0 con 0x7f3110153ea0 2024-09-13T16:30:00.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f311006b510 con 0x7f3110153ea0 2024-09-13T16:30:00.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f30f7fff700 1 -- 172.21.15.119:0/2636528292 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v83) v1 ==== 80+0+122 (secure 0 0 0) 0x7f3100054ac0 con 0x7f3110153ea0 2024-09-13T16:30:00.352 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6835/491709455 2024-09-14T16:29:48.943151+0000 2024-09-13T16:30:00.352 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6834/491709455 2024-09-14T16:29:48.943151+0000 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f30fc066760 msgr2=0x7f30fc068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f30fc066760 0x7f30fc068c20 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f3104007fd0 tx=0x7f3104009040 comp rx=0 tx=0).stop 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3110153ea0 msgr2=0x7f3110153680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3110153ea0 0x7f3110153680 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f3100007340 tx=0x7f3100007650 comp rx=0 tx=0).stop 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 shutdown_connections 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f30fc066760 0x7f30fc068c20 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f31100b2b70 0x7f31100b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3110153ea0 0x7f3110153680 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 --2- 172.21.15.119:0/2636528292 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f30f8005230 0x7f31101543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:00.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 >> 172.21.15.119:0/2636528292 conn(0x7f311008c240 msgr2=0x7f31100cf200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:00.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 shutdown_connections 2024-09-13T16:30:00.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:00.347+0000 7f3114e38700 1 -- 172.21.15.119:0/2636528292 wait complete. 2024-09-13T16:30:00.356 INFO:teuthology.orchestra.run.smithi119.stderr:listed 2 entries 2024-09-13T16:30:00.366 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:30:00.366 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:30:00.366 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:30:00.378 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[44840]: fuse finished with error 0 and tester_r 0 2024-09-13T16:30:00.379 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:30:06.481 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:30:06.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:30:06.482 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:30:06.490 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub start /subdir recursive 2024-09-13T16:30:06.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.607+0000 7f2017191700 1 Processor -- start 2024-09-13T16:30:06.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.607+0000 7f2017191700 1 -- start start 2024-09-13T16:30:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f20100cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f20100d0720 0x7f20100d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20100d3260 con 0x7f20100d1670 2024-09-13T16:30:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20100d1b20 con 0x7f20100d0720 2024-09-13T16:30:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100d0130 con 0x7f201015cc40 2024-09-13T16:30:06.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f20100d0720 0x7f20100d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f20100d0720 0x7f20100d1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37838/0 (socket says 172.21.15.119:37838) 2024-09-13T16:30:06.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 -- 172.21.15.119:0/2958047416 learned_addr learned my addr 172.21.15.119:0/2958047416 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201618f700 1 --2- 172.21.15.119:0/2958047416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f20100cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 -- 172.21.15.119:0/2958047416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 msgr2=0x7f20100cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 --2- 172.21.15.119:0/2958047416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f20100cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 -- 172.21.15.119:0/2958047416 >> v1:172.21.15.145:6789/0 conn(0x7f20100d1670 legacy=0x7f2010160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 -- 172.21.15.119:0/2958047416 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20100cd640 con 0x7f20100d0720 2024-09-13T16:30:06.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201598e700 1 --2- 172.21.15.119:0/2958047416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d0720 0x7f20100d1130 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2008009e80 tx=0x7f200800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db4d1f19db3e614 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201518d700 1 -- 172.21.15.119:0/2958047416 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200800e070 con 0x7f20100d0720 2024-09-13T16:30:06.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201518d700 1 -- 172.21.15.119:0/2958047416 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200800b980 con 0x7f20100d0720 2024-09-13T16:30:06.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f201518d700 1 -- 172.21.15.119:0/2958047416 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200800bd30 con 0x7f20100d0720 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- 172.21.15.119:0/2958047416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d0720 msgr2=0x7f20100d1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 --2- 172.21.15.119:0/2958047416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d0720 0x7f20100d1130 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f2008009e80 tx=0x7f200800b330 comp rx=0 tx=0).stop 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- 172.21.15.119:0/2958047416 shutdown_connections 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 --2- 172.21.15.119:0/2958047416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d0720 0x7f20100d1130 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 --2- 172.21.15.119:0/2958047416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f20100cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- 172.21.15.119:0/2958047416 >> 172.21.15.119:0/2958047416 conn(0x7f201009be40 msgr2=0x7f20100d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- 172.21.15.119:0/2958047416 shutdown_connections 2024-09-13T16:30:06.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.611+0000 7f2017191700 1 -- 172.21.15.119:0/2958047416 wait complete. 2024-09-13T16:30:06.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 Processor -- start 2024-09-13T16:30:06.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 -- start start 2024-09-13T16:30:06.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20100d0720 0x7f2010156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d1670 0x7f201015d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f201015db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f201015fa60 con 0x7f20100d1670 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2010156660 con 0x7f20100d0720 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20100b4210 con 0x7f201015cc40 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d1670 0x7f201015d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d1670 0x7f201015d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37844/0 (socket says 172.21.15.119:37844) 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 -- 172.21.15.119:0/695043492 learned_addr learned my addr 172.21.15.119:0/695043492 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201618f700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20100d0720 0x7f2010156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2016990700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f201015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 -- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 msgr2=0x7f201015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f201015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 -- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20100d0720 msgr2=0x7f2010156120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20100d0720 0x7f2010156120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 -- 172.21.15.119:0/695043492 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2008009b30 con 0x7f20100d1670 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f201598e700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d1670 0x7f201015d5c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f200800b370 tx=0x7f2008004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2006ffd700 1 -- 172.21.15.119:0/695043492 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2008004880 con 0x7f20100d1670 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2006ffd700 1 -- 172.21.15.119:0/695043492 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20080099b0 con 0x7f20100d1670 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20100b44f0 con 0x7f20100d1670 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.615+0000 7f2006ffd700 1 -- 172.21.15.119:0/695043492 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200801b670 con 0x7f20100d1670 2024-09-13T16:30:06.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.619+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20100b4ab0 con 0x7f20100d1670 2024-09-13T16:30:06.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.619+0000 7f2006ffd700 1 -- 172.21.15.119:0/695043492 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f200801b7d0 con 0x7f20100d1670 2024-09-13T16:30:06.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.619+0000 7f2006ffd700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ff4066730 0x7f1ff4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.619+0000 7f2006ffd700 1 -- 172.21.15.119:0/695043492 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2008085710 con 0x7f20100d1670 2024-09-13T16:30:06.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.619+0000 7f201618f700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ff4066730 0x7f1ff4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.619+0000 7f201b6a4740 1 Processor -- start 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- start start 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201618f700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ff4066730 0x7f1ff4068bf0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2000009a00 tx=0x7f2000008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 0x24aa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24aac70 0x24ab050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x24f3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x241bda0 con 0x24ab590 2024-09-13T16:30:06.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2419530 con 0x24aac70 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2414e20 con 0x24a9ca0 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x24f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x24f3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37852/0 (socket says 172.21.15.119:37852) 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 -- 172.21.15.119:0/3999064093 learned_addr learned my addr 172.21.15.119:0/3999064093 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201618f700 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 0x24aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201598e700 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24aac70 0x24ab050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 -- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 msgr2=0x24aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 0x24aa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 -- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24aac70 msgr2=0x24ab050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24aac70 0x24ab050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 -- 172.21.15.119:0/3999064093 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2287bd0 con 0x24ab590 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201598e700 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24aac70 0x24ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:06.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f2016990700 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x24f3090 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f200c001820 tx=0x7f200c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b52b5d5d8f6c831 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f1ff3fff700 1 -- 172.21.15.119:0/3999064093 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c0052e0 con 0x24ab590 2024-09-13T16:30:06.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f1ff3fff700 1 -- 172.21.15.119:0/3999064093 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200c005440 con 0x24ab590 2024-09-13T16:30:06.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f1ff3fff700 1 -- 172.21.15.119:0/3999064093 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c0038a0 con 0x24ab590 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- 172.21.15.119:0/3999064093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 msgr2=0x24f3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x24f3090 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f200c001820 tx=0x7f200c000ed0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- 172.21.15.119:0/3999064093 shutdown_connections 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x24f3090 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24aac70 0x24ab050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 --2- 172.21.15.119:0/3999064093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 0x24aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- 172.21.15.119:0/3999064093 >> 172.21.15.119:0/3999064093 conn(0x223c740 msgr2=0x24d63f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- 172.21.15.119:0/3999064093 shutdown_connections 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- 172.21.15.119:0/3999064093 wait complete. 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.623+0000 7f201b6a4740 1 -- start start 2024-09-13T16:30:06.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 Processor -- start 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x24da9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24d9760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24ab590 0x24f3020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x23b3260 con 0x24aac70 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x23d85f0 con 0x24a9ca0 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x22c0d10 con 0x24ab590 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 10 client.0 _ll_register_callbacks cb 0x241d640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24d9760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24d9760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37860/0 (socket says 172.21.15.119:37860) 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 -- 172.21.15.119:0/2535353707 learned_addr learned my addr 172.21.15.119:0/2535353707 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f2016990700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24ab590 0x24f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201618f700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x24da9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 -- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24ab590 msgr2=0x24f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24ab590 0x24f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 -- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 msgr2=0x24da9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x24da9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 -- 172.21.15.119:0/2535353707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20100d3260 con 0x24aac70 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201618f700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x24da9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201598e700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24d9760 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f200805e4f0 tx=0x7f2008056e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd3be53e79a2a293 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f1ff17fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2535353707 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080092e0 con 0x24aac70 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2535353707 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f200806bcc0 con 0x24aac70 2024-09-13T16:30:06.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2249450 con 0x24aac70 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2535353707 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f200806be80 con 0x24aac70 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 10 client.7491 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x23db330 con 0x24aac70 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2535353707 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2249450 con 0x24aac70 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2535353707 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2008056890 con 0x24aac70 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 10 client.7491 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 10 client.7491 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 20 client.7491 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x24fdb00 0x24fdee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 4 client.7491 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x22936d0 con 0x24fdb00 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201618f700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x24fdb00 0x24fdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f1ff17fa700 10 client.7491 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.627+0000 7f201618f700 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x24fdb00 0x24fdee0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=177cd1a3a95bdf27 server_cookie=a863c35ab9d08193 in_seq=0 out_seq=0 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2535353707 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x22936d0 con 0x24fdb00 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f1ff17fa700 10 client.7491 handle_command_reply: tid=0 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 client.7491 shutdown 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 msgr2=0x24d9760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24d9760 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f200805e4f0 tx=0x7f2008056e60 comp rx=0 tx=0).stop 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 shutdown_connections 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x24fdb00 0x24fdee0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24ab590 0x24f3020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24d9760 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f1ff17fa700 0 client.7491 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2535353707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x24da9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 >> 172.21.15.119:0/2535353707 conn(0x223c740 msgr2=0x24a91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 shutdown_connections 2024-09-13T16:30:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 1 -- 172.21.15.119:0/2535353707 wait complete. 2024-09-13T16:30:06.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.631+0000 7f201b6a4740 20 client.7491 trim_cache size 0 max 16384 2024-09-13T16:30:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 Processor -- start 2024-09-13T16:30:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- start start 2024-09-13T16:30:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 0x24aefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24adb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24ab590 0x24ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2444b90 con 0x24aac70 2024-09-13T16:30:06.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x23a0140 con 0x24ab590 2024-09-13T16:30:06.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x23e55f0 con 0x24a9ca0 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24adb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37874/0 (socket says 172.21.15.119:37874) 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 -- 172.21.15.119:0/2887180809 learned_addr learned my addr 172.21.15.119:0/2887180809 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 -- 172.21.15.119:0/2887180809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 msgr2=0x24aefc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 --2- 172.21.15.119:0/2887180809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 0x24aefc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 -- 172.21.15.119:0/2887180809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24ab590 msgr2=0x24ae0d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 --2- 172.21.15.119:0/2887180809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24ab590 0x24ae0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 -- 172.21.15.119:0/2887180809 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x23c1560 con 0x24aac70 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201598e700 1 --2- 172.21.15.119:0/2887180809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24adb90 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f20080565c0 tx=0x7f2008085070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68baf7c5f2d7635 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f2004dad700 1 -- 172.21.15.119:0/2887180809 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20080092e0 con 0x24aac70 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f2004dad700 1 -- 172.21.15.119:0/2887180809 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f200805a0d0 con 0x24aac70 2024-09-13T16:30:06.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f2004dad700 1 -- 172.21.15.119:0/2887180809 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200805e7d0 con 0x24aac70 2024-09-13T16:30:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- 172.21.15.119:0/2887180809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 msgr2=0x24adb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2887180809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24adb90 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f20080565c0 tx=0x7f2008085070 comp rx=0 tx=0).stop 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- 172.21.15.119:0/2887180809 shutdown_connections 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2887180809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24ab590 0x24ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2887180809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24aac70 0x24adb90 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2887180809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24a9ca0 0x24aefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- 172.21.15.119:0/2887180809 >> 172.21.15.119:0/2887180809 conn(0x223c740 msgr2=0x24a8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.651+0000 7f201b6a4740 1 -- 172.21.15.119:0/2887180809 shutdown_connections 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 -- 172.21.15.119:0/2887180809 wait complete. 2024-09-13T16:30:06.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 -- start start 2024-09-13T16:30:06.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 Processor -- start 2024-09-13T16:30:06.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x2560f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24aac70 0x25481c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x255f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x23e40f0 con 0x24ab590 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x24fce40 con 0x24a9ca0 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x23bb6c0 con 0x24aac70 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201b6a4740 10 client.0 _ll_register_callbacks cb 0x23df320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f2016990700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x255f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f2016990700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x255f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37882/0 (socket says 172.21.15.119:37882) 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f2016990700 1 -- 172.21.15.119:0/2856317616 learned_addr learned my addr 172.21.15.119:0/2856317616 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201598e700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24aac70 0x25481c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.655+0000 7f201618f700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x2560f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f2016990700 1 -- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24aac70 msgr2=0x25481c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f2016990700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24aac70 0x25481c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f2016990700 1 -- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 msgr2=0x2560f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f2016990700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x2560f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f2016990700 1 -- 172.21.15.119:0/2856317616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x23db330 con 0x24ab590 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201598e700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24aac70 0x25481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f2016990700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x255f4e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f200c0017f0 tx=0x7f200c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d00b4535e7b00980 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f1ff17fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2856317616 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f200c0052e0 con 0x24ab590 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2856317616 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f200c006a80 con 0x24ab590 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x24dad90 con 0x24ab590 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2856317616 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f200c007ba0 con 0x24ab590 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 10 client.7497 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:06.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x23b5830 con 0x24ab590 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2856317616 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x24dad90 con 0x24ab590 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2856317616 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f200c012070 con 0x24ab590 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 10 client.7497 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 10 client.7497 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 20 client.7497 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2548700 0x255fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 4 client.7497 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}] 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["recursive"]}) v1 -- 0x7f200c005440 con 0x2548700 2024-09-13T16:30:06.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201618f700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2548700 0x255fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f1ff17fa700 10 client.7497 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.659+0000 7f201618f700 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2548700 0x255fa20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11096d5e3fb14bf2 server_cookie=db9e0da05cac83a8 in_seq=0 out_seq=0 2024-09-13T16:30:06.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f1ff17fa700 1 -- 172.21.15.119:0/2856317616 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x7f200c005440 con 0x2548700 2024-09-13T16:30:06.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f1ff17fa700 10 client.7497 handle_command_reply: tid=0 2024-09-13T16:30:06.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 client.7497 shutdown 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 msgr2=0x255f4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x255f4e0 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f200c0017f0 tx=0x7f200c007f90 comp rx=0 tx=0).stop 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 shutdown_connections 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2548700 0x255fa20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24ab590 0x255f4e0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24aac70 0x25481c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 --2- 172.21.15.119:0/2856317616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24a9ca0 0x2560f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f1ff17fa700 0 client.7497 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 >> 172.21.15.119:0/2856317616 conn(0x223c740 msgr2=0x24aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 shutdown_connections 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 1 -- 172.21.15.119:0/2856317616 wait complete. 2024-09-13T16:30:06.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f201b6a4740 20 client.7497 trim_cache size 0 max 16384 2024-09-13T16:30:06.670 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:30:06.670 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:30:06.670 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "f05da259-bd60-4ff7-a5b4-0af462d6d339", 2024-09-13T16:30:06.670 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:30:06.670 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ff4066730 msgr2=0x7f1ff4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ff4066730 0x7f1ff4068bf0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f2000009a00 tx=0x7f2000008040 comp rx=0 tx=0).stop 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d1670 msgr2=0x7f201015d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d1670 0x7f201015d5c0 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f200800b370 tx=0x7f2008004af0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 shutdown_connections 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ff4066730 0x7f1ff4068bf0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f201015cc40 0x7f201015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20100d1670 0x7f201015d5c0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 --2- 172.21.15.119:0/695043492 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20100d0720 0x7f2010156120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 >> 172.21.15.119:0/695043492 conn(0x7f201009be40 msgr2=0x7f20100cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 shutdown_connections 2024-09-13T16:30:06.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.663+0000 7f2017191700 1 -- 172.21.15.119:0/695043492 wait complete. 2024-09-13T16:30:06.684 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-09-13T16:30:06.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f1070b08700 1 Processor -- start 2024-09-13T16:30:06.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f1070b08700 1 -- start start 2024-09-13T16:30:06.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f1070b08700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f106c0b2f40 0x7f106c159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f1070b08700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c06dad0 0x7f106c14f7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f1070b08700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f106c0ade10 con 0x7f106c0b1760 2024-09-13T16:30:06.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f1070b08700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f106c0ce250 con 0x7f106c0b2f40 2024-09-13T16:30:06.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f1070b08700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c15a4e0 con 0x7f106c06dad0 2024-09-13T16:30:06.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c06dad0 0x7f106c14f7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c06dad0 0x7f106c14f7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:35870/0 (socket says 172.21.15.119:35870) 2024-09-13T16:30:06.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 -- 172.21.15.119:0/2529245561 learned_addr learned my addr 172.21.15.119:0/2529245561 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106b7fe700 1 --2- 172.21.15.119:0/2529245561 >> v2:172.21.15.145:3300/0 conn(0x7f106c0b2f40 0x7f106c159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 -- 172.21.15.119:0/2529245561 >> v2:172.21.15.145:3300/0 conn(0x7f106c0b2f40 msgr2=0x7f106c159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 --2- 172.21.15.119:0/2529245561 >> v2:172.21.15.145:3300/0 conn(0x7f106c0b2f40 0x7f106c159fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 -- 172.21.15.119:0/2529245561 >> v1:172.21.15.145:6789/0 conn(0x7f106c0b1760 legacy=0x7f106c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 -- 172.21.15.119:0/2529245561 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f106c0b1c90 con 0x7f106c06dad0 2024-09-13T16:30:06.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106bfff700 1 --2- 172.21.15.119:0/2529245561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c06dad0 0x7f106c14f7c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1064009e50 tx=0x7f106400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=551c16d2f9cdefe4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.799+0000 7f106a7fc700 1 -- 172.21.15.119:0/2529245561 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f106400bb10 con 0x7f106c06dad0 2024-09-13T16:30:06.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f106a7fc700 1 -- 172.21.15.119:0/2529245561 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f106400bc70 con 0x7f106c06dad0 2024-09-13T16:30:06.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f106a7fc700 1 -- 172.21.15.119:0/2529245561 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064011460 con 0x7f106c06dad0 2024-09-13T16:30:06.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- 172.21.15.119:0/2529245561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c06dad0 msgr2=0x7f106c14f7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 --2- 172.21.15.119:0/2529245561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c06dad0 0x7f106c14f7c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1064009e50 tx=0x7f106400b330 comp rx=0 tx=0).stop 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- 172.21.15.119:0/2529245561 shutdown_connections 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 --2- 172.21.15.119:0/2529245561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c06dad0 0x7f106c14f7c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 --2- 172.21.15.119:0/2529245561 >> v2:172.21.15.145:3300/0 conn(0x7f106c0b2f40 0x7f106c159fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- 172.21.15.119:0/2529245561 >> 172.21.15.119:0/2529245561 conn(0x7f106c0b0640 msgr2=0x7f106c0b0a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- 172.21.15.119:0/2529245561 shutdown_connections 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- 172.21.15.119:0/2529245561 wait complete. 2024-09-13T16:30:06.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 Processor -- start 2024-09-13T16:30:06.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- start start 2024-09-13T16:30:06.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f106c06dad0 0x7f106c158470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106c0b1760 0x7f106c0a8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c0b2f40 0x7f106c156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1589b0 con 0x7f106c0b1760 2024-09-13T16:30:06.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1573d0 con 0x7f106c06dad0 2024-09-13T16:30:06.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.803+0000 7f1070b08700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f106c1576b0 con 0x7f106c0b2f40 2024-09-13T16:30:06.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106c0b1760 0x7f106c0a8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106c0b1760 0x7f106c0a8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37896/0 (socket says 172.21.15.119:37896) 2024-09-13T16:30:06.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 -- 172.21.15.119:0/2747605620 learned_addr learned my addr 172.21.15.119:0/2747605620 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106bfff700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c0b2f40 0x7f106c156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106b7fe700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f106c06dad0 0x7f106c158470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 -- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c0b2f40 msgr2=0x7f106c156a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c0b2f40 0x7f106c156a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 -- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f106c06dad0 msgr2=0x7f106c158470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f106c06dad0 0x7f106c158470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 -- 172.21.15.119:0/2747605620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1064009b30 con 0x7f106c0b1760 2024-09-13T16:30:06.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f106affd700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106c0b1760 0x7f106c0a8ea0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f105400c500 tx=0x7f105400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f1063fff700 1 -- 172.21.15.119:0/2747605620 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10540052d0 con 0x7f106c0b1760 2024-09-13T16:30:06.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f1063fff700 1 -- 172.21.15.119:0/2747605620 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1054005430 con 0x7f106c0b1760 2024-09-13T16:30:06.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f106c157990 con 0x7f106c0b1760 2024-09-13T16:30:06.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f1063fff700 1 -- 172.21.15.119:0/2747605620 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10540055a0 con 0x7f106c0b1760 2024-09-13T16:30:06.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c157f50 con 0x7f106c0b1760 2024-09-13T16:30:06.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.807+0000 7f1063fff700 1 -- 172.21.15.119:0/2747605620 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f10540036a0 con 0x7f106c0b1760 2024-09-13T16:30:06.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.811+0000 7f1063fff700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1050066730 0x7f1050068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.811+0000 7f1063fff700 1 -- 172.21.15.119:0/2747605620 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1054083410 con 0x7f106c0b1760 2024-09-13T16:30:06.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.811+0000 7f106b7fe700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1050066730 0x7f1050068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.811+0000 7f106b7fe700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1050066730 0x7f1050068bf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f105c009b50 tx=0x7f105c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.811+0000 7f107501b740 1 Processor -- start 2024-09-13T16:30:06.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.811+0000 7f107501b740 1 -- start start 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ebd90 0x12a2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x12ca520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ec170 0x12ec550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1233010 con 0x12a1310 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x11c0f40 con 0x12ebd90 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1213e30 con 0x12ec170 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x12ca520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x12ca520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37900/0 (socket says 172.21.15.119:37900) 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 -- 172.21.15.119:0/1983308333 learned_addr learned my addr 172.21.15.119:0/1983308333 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106b7fe700 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ebd90 0x12a2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106bfff700 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ec170 0x12ec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 -- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ec170 msgr2=0x12ec550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ec170 0x12ec550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 -- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ebd90 msgr2=0x12a2810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ebd90 0x12a2810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 -- 172.21.15.119:0/1983308333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x107fa40 con 0x12a1310 2024-09-13T16:30:06.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f106affd700 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x12ca520 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f1054081830 tx=0x7f105400a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86471b4133bba8b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f10617fa700 1 -- 172.21.15.119:0/1983308333 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054054330 con 0x12a1310 2024-09-13T16:30:06.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f10617fa700 1 -- 172.21.15.119:0/1983308333 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1054058160 con 0x12a1310 2024-09-13T16:30:06.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f10617fa700 1 -- 172.21.15.119:0/1983308333 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105405c720 con 0x12a1310 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- 172.21.15.119:0/1983308333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 msgr2=0x12ca520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x12ca520 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f1054081830 tx=0x7f105400a000 comp rx=0 tx=0).stop 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- 172.21.15.119:0/1983308333 shutdown_connections 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ec170 0x12ec550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x12ca520 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 --2- 172.21.15.119:0/1983308333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ebd90 0x12a2810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- 172.21.15.119:0/1983308333 >> 172.21.15.119:0/1983308333 conn(0x10345b0 msgr2=0x12a5b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- 172.21.15.119:0/1983308333 shutdown_connections 2024-09-13T16:30:06.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- 172.21.15.119:0/1983308333 wait complete. 2024-09-13T16:30:06.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 -- start start 2024-09-13T16:30:06.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.815+0000 7f107501b740 1 Processor -- start 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12a1310 0x12c9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12c6a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ec170 0x12c8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x120f150 con 0x12ebd90 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x120ceb0 con 0x12ec170 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x11ca8e0 con 0x12a1310 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 10 client.0 _ll_register_callbacks cb 0x11a9670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12c6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12c6a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37904/0 (socket says 172.21.15.119:37904) 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 -- 172.21.15.119:0/952215933 learned_addr learned my addr 172.21.15.119:0/952215933 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106b7fe700 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12a1310 0x12c9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106bfff700 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ec170 0x12c8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 -- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12a1310 msgr2=0x12c9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12a1310 0x12c9c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 -- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ec170 msgr2=0x12c8de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ec170 0x12c8de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 -- 172.21.15.119:0/952215933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c157f50 con 0x12ebd90 2024-09-13T16:30:06.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106affd700 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12c6a40 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f105405c1f0 tx=0x7f1054000f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e8c3ab4d695615f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f105a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f105a7fc700 1 -- 172.21.15.119:0/952215933 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1054071070 con 0x12ebd90 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x10412c0 con 0x12ebd90 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f105a7fc700 1 -- 172.21.15.119:0/952215933 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f105405c720 con 0x12ebd90 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f105a7fc700 1 -- 172.21.15.119:0/952215933 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1054054330 con 0x12ebd90 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 10 client.7506 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x11d3290 con 0x12ebd90 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f105a7fc700 1 -- 172.21.15.119:0/952215933 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x10412c0 con 0x12ebd90 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f105a7fc700 1 -- 172.21.15.119:0/952215933 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f10540830f0 con 0x12ebd90 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 10 client.7506 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:06.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 10 client.7506 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:06.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 20 client.7506 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:06.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ea500 0x12ec550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 4 client.7506 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:30:06.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x108b540 con 0x12ea500 2024-09-13T16:30:06.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106b7fe700 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ea500 0x12ec550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f105a7fc700 10 client.7506 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.819+0000 7f106b7fe700 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ea500 0x12ec550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1f49c93bcd500cb server_cookie=d164f208af0bb73b in_seq=0 out_seq=0 2024-09-13T16:30:06.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f105a7fc700 1 -- 172.21.15.119:0/952215933 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x108b540 con 0x12ea500 2024-09-13T16:30:06.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f105a7fc700 10 client.7506 handle_command_reply: tid=0 2024-09-13T16:30:06.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 client.7506 shutdown 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 msgr2=0x12c6a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12c6a40 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f105405c1f0 tx=0x7f1054000f70 comp rx=0 tx=0).stop 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 shutdown_connections 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ea500 0x12ec550 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ec170 0x12c8de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f105a7fc700 0 client.7506 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12c6a40 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 --2- 172.21.15.119:0/952215933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12a1310 0x12c9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 >> 172.21.15.119:0/952215933 conn(0x10345b0 msgr2=0x12ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 shutdown_connections 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 1 -- 172.21.15.119:0/952215933 wait complete. 2024-09-13T16:30:06.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.823+0000 7f107501b740 20 client.7506 trim_cache size 0 max 16384 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 Processor -- start 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 -- start start 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12a1310 0x12e85a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea500 0x12e8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12e9020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x10b8b80 con 0x12ebd90 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x11cd990 con 0x12a1310 2024-09-13T16:30:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1145da0 con 0x12ea500 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12e9020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12e9020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37908/0 (socket says 172.21.15.119:37908) 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 -- 172.21.15.119:0/1260343572 learned_addr learned my addr 172.21.15.119:0/1260343572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106affd700 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea500 0x12e8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106b7fe700 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12a1310 0x12e85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 -- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea500 msgr2=0x12e8ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea500 0x12e8ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 -- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12a1310 msgr2=0x12e85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12a1310 0x12e85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 -- 172.21.15.119:0/1260343572 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x12a2b90 con 0x12ebd90 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106affd700 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea500 0x12e8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106b7fe700 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12a1310 0x12e85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:30:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f106bfff700 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12e9020 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1064009390 tx=0x7f1064012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d0e97cf1de8d901 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f1061ffb700 1 -- 172.21.15.119:0/1260343572 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064020070 con 0x12ebd90 2024-09-13T16:30:06.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f1061ffb700 1 -- 172.21.15.119:0/1260343572 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1064021040 con 0x12ebd90 2024-09-13T16:30:06.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.843+0000 7f1061ffb700 1 -- 172.21.15.119:0/1260343572 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1064003c60 con 0x12ebd90 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- 172.21.15.119:0/1260343572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 msgr2=0x12e9020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12e9020 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f1064009390 tx=0x7f1064012000 comp rx=0 tx=0).stop 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- 172.21.15.119:0/1260343572 shutdown_connections 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ebd90 0x12e9020 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ea500 0x12e8ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 --2- 172.21.15.119:0/1260343572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12a1310 0x12e85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- 172.21.15.119:0/1260343572 >> 172.21.15.119:0/1260343572 conn(0x10345b0 msgr2=0x12f4590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- 172.21.15.119:0/1260343572 shutdown_connections 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- 172.21.15.119:0/1260343572 wait complete. 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- start start 2024-09-13T16:30:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 Processor -- start 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x1304540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea500 0x1303930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ebd90 0x1303e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x12e90e0 con 0x12a1310 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x12f52d0 con 0x12ea500 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x12e9310 con 0x12ebd90 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f107501b740 10 client.0 _ll_register_callbacks cb 0x120b610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f106b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x1304540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f106b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x1304540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37910/0 (socket says 172.21.15.119:37910) 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.847+0000 7f106bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ebd90 0x1303e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106b7fe700 1 -- 172.21.15.119:0/518501000 learned_addr learned my addr 172.21.15.119:0/518501000 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106affd700 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea500 0x1303930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106b7fe700 1 -- 172.21.15.119:0/518501000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ebd90 msgr2=0x1303e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106b7fe700 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ebd90 0x1303e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106b7fe700 1 -- 172.21.15.119:0/518501000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea500 msgr2=0x1303930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106b7fe700 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea500 0x1303930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106b7fe700 1 -- 172.21.15.119:0/518501000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f106c0b1c90 con 0x12a1310 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106b7fe700 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x1304540 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f105c0040e0 tx=0x7f105c0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b055f8a9f71ba135 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 1 -- 172.21.15.119:0/518501000 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f105c029070 con 0x12a1310 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 1 -- 172.21.15.119:0/518501000 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f105c011e30 con 0x12a1310 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1054058160 con 0x12a1310 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 1 -- 172.21.15.119:0/518501000 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f105c01c040 con 0x12a1310 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 10 client.7512 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:06.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x11b94c0 con 0x12a1310 2024-09-13T16:30:06.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 1 -- 172.21.15.119:0/518501000 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1054058160 con 0x12a1310 2024-09-13T16:30:06.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 1 -- 172.21.15.119:0/518501000 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f105c005a40 con 0x12a1310 2024-09-13T16:30:06.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 10 client.7512 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:06.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 10 client.7512 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:06.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 20 client.7512 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:06.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ec170 0x1301c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:06.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 4 client.7512 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:30:06.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x10b02c0 con 0x12ec170 2024-09-13T16:30:06.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106affd700 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ec170 0x1301c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 10 client.7512 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f106affd700 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ec170 0x1301c70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e40ddc5303ceebf server_cookie=710ee1134b7ccd0 in_seq=0 out_seq=0 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 1 -- 172.21.15.119:0/518501000 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x10b02c0 con 0x12ec170 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 10 client.7512 handle_command_reply: tid=0 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 client.7512 shutdown 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 msgr2=0x1304540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x1304540 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7f105c0040e0 tx=0x7f105c0047e0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 shutdown_connections 2024-09-13T16:30:06.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x12ec170 0x1301c70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ebd90 0x1303e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f105a7fc700 0 client.7512 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ea500 0x1303930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 --2- 172.21.15.119:0/518501000 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12a1310 0x1304540 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 >> 172.21.15.119:0/518501000 conn(0x10345b0 msgr2=0x12eb440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 shutdown_connections 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 1 -- 172.21.15.119:0/518501000 wait complete. 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.851+0000 7f107501b740 20 client.7512 trim_cache size 0 max 16384 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:30:06.862 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout: "f05da259-bd60-4ff7-a5b4-0af462d6d339": { 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/subdir", 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "f05da259-bd60-4ff7-a5b4-0af462d6d339", 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive" 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:30:06.863 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1050066730 msgr2=0x7f1050068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1050066730 0x7f1050068bf0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f105c009b50 tx=0x7f105c009240 comp rx=0 tx=0).stop 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106c0b1760 msgr2=0x7f106c0a8ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106c0b1760 0x7f106c0a8ea0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f105400c500 tx=0x7f105400c810 comp rx=0 tx=0).stop 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 shutdown_connections 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1050066730 0x7f1050068bf0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f106c0b2f40 0x7f106c156a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f106c0b1760 0x7f106c0a8ea0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 --2- 172.21.15.119:0/2747605620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f106c06dad0 0x7f106c158470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 >> 172.21.15.119:0/2747605620 conn(0x7f106c0b0640 msgr2=0x7f106c152310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:06.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 shutdown_connections 2024-09-13T16:30:06.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:06.855+0000 7f1070b08700 1 -- 172.21.15.119:0/2747605620 wait complete. 2024-09-13T16:30:06.874 INFO:tasks.cephfs.filesystem:scrub status for tag:f05da259-bd60-4ff7-a5b4-0af462d6d339 - {'path': '/subdir', 'tag': 'f05da259-bd60-4ff7-a5b4-0af462d6d339', 'options': 'recursive'} 2024-09-13T16:30:27.452 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:30:27.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:30:36.875 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-09-13T16:30:37.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.002+0000 7fea47be9700 1 Processor -- start 2024-09-13T16:30:37.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.002+0000 7fea47be9700 1 -- start start 2024-09-13T16:30:37.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fea4006b710 0x7fea4006bb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea400d1640 0x7fea400d1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea4006c0a0 con 0x7fea4015cfb0 2024-09-13T16:30:37.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea400cee40 con 0x7fea4006b710 2024-09-13T16:30:37.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea400d1220 con 0x7fea400d1640 2024-09-13T16:30:37.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea45985700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fea4015cfb0 0x7fea4006ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55658/0 (socket says 172.21.15.119:55658) 2024-09-13T16:30:37.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea45985700 1 -- 172.21.15.119:0/483273259 learned_addr learned my addr 172.21.15.119:0/483273259 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:30:37.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea46186700 1 --2- 172.21.15.119:0/483273259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea400d1640 0x7fea400d1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea45184700 1 --2- 172.21.15.119:0/483273259 >> v2:172.21.15.145:3300/0 conn(0x7fea4006b710 0x7fea4006bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea44983700 1 -- 172.21.15.119:0/483273259 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540885709 0 0) 0x7fea30001110 con 0x7fea4015cfb0 2024-09-13T16:30:37.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea44983700 1 -- 172.21.15.119:0/483273259 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea2c0035d0 con 0x7fea4015cfb0 2024-09-13T16:30:37.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea46186700 1 -- 172.21.15.119:0/483273259 >> v2:172.21.15.145:3300/0 conn(0x7fea4006b710 msgr2=0x7fea4006bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea46186700 1 --2- 172.21.15.119:0/483273259 >> v2:172.21.15.145:3300/0 conn(0x7fea4006b710 0x7fea4006bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea46186700 1 -- 172.21.15.119:0/483273259 >> v1:172.21.15.119:6789/0 conn(0x7fea4015cfb0 legacy=0x7fea4006ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea44983700 1 -- 172.21.15.119:0/483273259 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714729026 0 0) 0x7fea2c0035d0 con 0x7fea4015cfb0 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea46186700 1 -- 172.21.15.119:0/483273259 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea400ce880 con 0x7fea400d1640 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea45184700 1 --2- 172.21.15.119:0/483273259 >> v2:172.21.15.145:3300/0 conn(0x7fea4006b710 0x7fea4006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea46186700 1 --2- 172.21.15.119:0/483273259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea400d1640 0x7fea400d1ad0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fea38009d90 tx=0x7fea3800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fb135ca37c24953 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea44983700 1 -- 172.21.15.119:0/483273259 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3800e070 con 0x7fea400d1640 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea44983700 1 -- 172.21.15.119:0/483273259 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea38003a40 con 0x7fea400d1640 2024-09-13T16:30:37.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea44983700 1 -- 172.21.15.119:0/483273259 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30001800 con 0x7fea400d1640 2024-09-13T16:30:37.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- 172.21.15.119:0/483273259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea400d1640 msgr2=0x7fea400d1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 --2- 172.21.15.119:0/483273259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea400d1640 0x7fea400d1ad0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fea38009d90 tx=0x7fea3800b330 comp rx=0 tx=0).stop 2024-09-13T16:30:37.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- 172.21.15.119:0/483273259 shutdown_connections 2024-09-13T16:30:37.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 --2- 172.21.15.119:0/483273259 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea400d1640 0x7fea400d1ad0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 --2- 172.21.15.119:0/483273259 >> v2:172.21.15.145:3300/0 conn(0x7fea4006b710 0x7fea4006bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- 172.21.15.119:0/483273259 >> 172.21.15.119:0/483273259 conn(0x7fea400d2da0 msgr2=0x7fea400d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- 172.21.15.119:0/483273259 shutdown_connections 2024-09-13T16:30:37.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.006+0000 7fea47be9700 1 -- 172.21.15.119:0/483273259 wait complete. 2024-09-13T16:30:37.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 Processor -- start 2024-09-13T16:30:37.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 -- start start 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea4006b710 0x7fea400b7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea400d1640 0x7fea401f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea4015cfb0 0x7fea401f6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b7e70 con 0x7fea4006b710 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b8150 con 0x7fea400d1640 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea47be9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea400b8430 con 0x7fea4015cfb0 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45985700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea4006b710 0x7fea400b7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea400d1640 0x7fea401f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea400d1640 0x7fea401f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:60430/0 (socket says 172.21.15.119:60430) 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 -- 172.21.15.119:0/2284338708 learned_addr learned my addr 172.21.15.119:0/2284338708 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45985700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea4006b710 0x7fea400b7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57190/0 (socket says 172.21.15.119:57190) 2024-09-13T16:30:37.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea46186700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea4015cfb0 0x7fea401f6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 -- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea4015cfb0 msgr2=0x7fea401f6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea4015cfb0 0x7fea401f6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 -- 172.21.15.119:0/2284338708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea4006b710 msgr2=0x7fea400b7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea4006b710 0x7fea400b7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45184700 1 -- 172.21.15.119:0/2284338708 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea3c0094b0 con 0x7fea400d1640 2024-09-13T16:30:37.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.010+0000 7fea45985700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea4006b710 0x7fea400b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:30:37.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea45184700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea400d1640 0x7fea401f6510 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00ebe0 tx=0x7fea3c00eef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea367fc700 1 -- 172.21.15.119:0/2284338708 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c009770 con 0x7fea400d1640 2024-09-13T16:30:37.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea38009a70 con 0x7fea400d1640 2024-09-13T16:30:37.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea367fc700 1 -- 172.21.15.119:0/2284338708 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3c00c850 con 0x7fea400d1640 2024-09-13T16:30:37.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea400b7a60 con 0x7fea400d1640 2024-09-13T16:30:37.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea367fc700 1 -- 172.21.15.119:0/2284338708 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c00cbc0 con 0x7fea400d1640 2024-09-13T16:30:37.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea367fc700 1 -- 172.21.15.119:0/2284338708 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fea3c00cd20 con 0x7fea400d1640 2024-09-13T16:30:37.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea367fc700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea24066730 0x7fea24068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea367fc700 1 -- 172.21.15.119:0/2284338708 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fea3c084600 con 0x7fea400d1640 2024-09-13T16:30:37.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea4ae9a740 1 Processor -- start 2024-09-13T16:30:37.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea4ae9a740 1 -- start start 2024-09-13T16:30:37.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.014+0000 7fea45985700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea24066730 0x7fea24068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aaf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ad0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad1270 0x2a845d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a18010 con 0x2ad0950 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x29a5f40 con 0x2ad1270 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x29f8e30 con 0x2aaed00 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aaf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aaf240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57206/0 (socket says 172.21.15.119:57206) 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 -- 172.21.15.119:0/3975547825 learned_addr learned my addr 172.21.15.119:0/3975547825 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea24066730 0x7fea24068bf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fea300019b0 tx=0x7fea300012f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45184700 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ad0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 -- 172.21.15.119:0/3975547825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 msgr2=0x2ad0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ad0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 -- 172.21.15.119:0/3975547825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad1270 msgr2=0x2a845d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad1270 0x2a845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 -- 172.21.15.119:0/3975547825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2864a40 con 0x2ad0950 2024-09-13T16:30:37.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea45985700 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aaf240 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fea3000fec0 tx=0x7fea30012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76dc277283d4c0c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea237fe700 1 -- 172.21.15.119:0/3975547825 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30015b00 con 0x2ad0950 2024-09-13T16:30:37.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea237fe700 1 -- 172.21.15.119:0/3975547825 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30015c60 con 0x2ad0950 2024-09-13T16:30:37.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea237fe700 1 -- 172.21.15.119:0/3975547825 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea30021400 con 0x2ad0950 2024-09-13T16:30:37.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- 172.21.15.119:0/3975547825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 msgr2=0x2aaf240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aaf240 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7fea3000fec0 tx=0x7fea30012000 comp rx=0 tx=0).stop 2024-09-13T16:30:37.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- 172.21.15.119:0/3975547825 shutdown_connections 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad1270 0x2a845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ad0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/3975547825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aaf240 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- 172.21.15.119:0/3975547825 >> 172.21.15.119:0/3975547825 conn(0x28195b0 msgr2=0x2a8a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- 172.21.15.119:0/3975547825 shutdown_connections 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.018+0000 7fea4ae9a740 1 -- 172.21.15.119:0/3975547825 wait complete. 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 -- start start 2024-09-13T16:30:37.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 Processor -- start 2024-09-13T16:30:37.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aaed00 0x2ab16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aae420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2ad1270 0x2acf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x29f4150 con 0x2ad0950 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x29f1eb0 con 0x2aaed00 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x29af8e0 con 0x2ad1270 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 10 client.0 _ll_register_callbacks cb 0x298e670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aae420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aae420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57214/0 (socket says 172.21.15.119:57214) 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 -- 172.21.15.119:0/2504064711 learned_addr learned my addr 172.21.15.119:0/2504064711 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45985700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aaed00 0x2ab16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea46186700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2ad1270 0x2acf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 -- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2ad1270 msgr2=0x2acf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2ad1270 0x2acf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 -- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aaed00 msgr2=0x2ab16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aaed00 0x2ab16c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 -- 172.21.15.119:0/2504064711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3000fb70 con 0x2ad0950 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea46186700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2ad1270 0x2acf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45985700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aaed00 0x2ab16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea45184700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aae420 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009fd0 tx=0x7fea3c055c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff964b55ca6456 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea20ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea20ff9700 1 -- 172.21.15.119:0/2504064711 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c054b90 con 0x2ad0950 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea20ff9700 1 -- 172.21.15.119:0/2504064711 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fea3c05d890 con 0x2ad0950 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x28262c0 con 0x2ad0950 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea20ff9700 1 -- 172.21.15.119:0/2504064711 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea3c05d400 con 0x2ad0950 2024-09-13T16:30:37.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 10 client.7524 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x29b8290 con 0x2ad0950 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.022+0000 7fea20ff9700 1 -- 172.21.15.119:0/2504064711 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3c080070 con 0x2ad0950 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea20ff9700 1 -- 172.21.15.119:0/2504064711 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fea3c05b260 con 0x2ad0950 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 10 client.7524 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 10 client.7524 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 20 client.7524 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2aafca0 0x2ab00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 4 client.7524 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2870540 con 0x2aafca0 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea45985700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2aafca0 0x2ab00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea45985700 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2aafca0 0x2ab00b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9992bb027a1ebf4b server_cookie=6126e51e7c7e8237 in_seq=0 out_seq=0 2024-09-13T16:30:37.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea20ff9700 10 client.7524 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea20ff9700 1 -- 172.21.15.119:0/2504064711 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x2870540 con 0x2aafca0 2024-09-13T16:30:37.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea20ff9700 10 client.7524 handle_command_reply: tid=0 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 client.7524 shutdown 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 msgr2=0x2aae420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aae420 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7fea3c009fd0 tx=0x7fea3c055c50 comp rx=0 tx=0).stop 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 shutdown_connections 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2aafca0 0x2ab00b0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2ad1270 0x2acf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea20ff9700 0 client.7524 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2aae420 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/2504064711 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aaed00 0x2ab16c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 >> 172.21.15.119:0/2504064711 conn(0x28195b0 msgr2=0x2a83db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 shutdown_connections 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 1 -- 172.21.15.119:0/2504064711 wait complete. 2024-09-13T16:30:37.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.026+0000 7fea4ae9a740 20 client.7524 trim_cache size 0 max 16384 2024-09-13T16:30:37.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 Processor -- start 2024-09-13T16:30:37.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 -- start start 2024-09-13T16:30:37.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2b26c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aafca0 0x2ab0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2ab05c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x289db80 con 0x2ad0950 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x29ef6c0 con 0x2aafca0 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a8ac10 con 0x2aaed00 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2ab05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea45985700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2b26c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2ab05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57220/0 (socket says 172.21.15.119:57220) 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea45985700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2b26c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:51018/0 (socket says 172.21.15.119:51018) 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 -- 172.21.15.119:0/49317868 learned_addr learned my addr 172.21.15.119:0/49317868 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea45184700 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aafca0 0x2ab0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 -- 172.21.15.119:0/49317868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 msgr2=0x2b26c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2b26c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 -- 172.21.15.119:0/49317868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aafca0 msgr2=0x2ab0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aafca0 0x2ab0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 -- 172.21.15.119:0/49317868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x299e4c0 con 0x2ad0950 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea46186700 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2ab05c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fea400cd480 tx=0x7fea3800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60ee5ee67f790dcd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea23fff700 1 -- 172.21.15.119:0/49317868 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3800ba20 con 0x2ad0950 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea23fff700 1 -- 172.21.15.119:0/49317868 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3800bb80 con 0x2ad0950 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.046+0000 7fea23fff700 1 -- 172.21.15.119:0/49317868 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea38004030 con 0x2ad0950 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- 172.21.15.119:0/49317868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 msgr2=0x2ab05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2ab05c0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fea400cd480 tx=0x7fea3800e000 comp rx=0 tx=0).stop 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- 172.21.15.119:0/49317868 shutdown_connections 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2ad0950 0x2ab05c0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2aafca0 0x2ab0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/49317868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2b26c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- 172.21.15.119:0/49317868 >> 172.21.15.119:0/49317868 conn(0x28195b0 msgr2=0x2ae0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- 172.21.15.119:0/49317868 shutdown_connections 2024-09-13T16:30:37.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- 172.21.15.119:0/49317868 wait complete. 2024-09-13T16:30:37.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- start start 2024-09-13T16:30:37.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 Processor -- start 2024-09-13T16:30:37.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ab1950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2aafca0 0x2b29b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad0950 0x2ae5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x297ba50 con 0x2aafca0 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2990620 con 0x2ad0950 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2ab1e90 con 0x2aaed00 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea4ae9a740 10 client.0 _ll_register_callbacks cb 0x29c18f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea45184700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2aafca0 0x2b29b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea45184700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2aafca0 0x2b29b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57228/0 (socket says 172.21.15.119:57228) 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea45184700 1 -- 172.21.15.119:0/1588416829 learned_addr learned my addr 172.21.15.119:0/1588416829 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea45985700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ab1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.050+0000 7fea46186700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad0950 0x2ae5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45184700 1 -- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 msgr2=0x2ab1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45184700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ab1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45184700 1 -- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad0950 msgr2=0x2ae5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45184700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad0950 0x2ae5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45184700 1 -- 172.21.15.119:0/1588416829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea400ce880 con 0x2aafca0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45985700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ab1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea46186700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad0950 0x2ae5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45184700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2aafca0 0x2b29b60 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fea3c051ec0 tx=0x7fea3c0828f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ce4ad42bec578c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 1 -- 172.21.15.119:0/1588416829 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3c054b90 con 0x2aafca0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 1 -- 172.21.15.119:0/1588416829 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fea3c05d890 con 0x2aafca0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fea30015c60 con 0x2aafca0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 1 -- 172.21.15.119:0/1588416829 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7fea3c0517e0 con 0x2aafca0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 10 client.7530 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x2992790 con 0x2aafca0 2024-09-13T16:30:37.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 1 -- 172.21.15.119:0/1588416829 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30015c60 con 0x2aafca0 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 1 -- 172.21.15.119:0/1588416829 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fea3c05d400 con 0x2aafca0 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 10 client.7530 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 10 client.7530 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 20 client.7530 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ad1270 0x2ae6410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 4 client.7530 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x28952c0 con 0x2ad1270 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45985700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ad1270 0x2ae6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 10 client.7530 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea45985700 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ad1270 0x2ae6410 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a98af52d1f2e09 server_cookie=efe9f997945e7485 in_seq=0 out_seq=0 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 1 -- 172.21.15.119:0/1588416829 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x28952c0 con 0x2ad1270 2024-09-13T16:30:37.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 10 client.7530 handle_command_reply: tid=0 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 client.7530 shutdown 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2aafca0 msgr2=0x2b29b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2aafca0 0x2b29b60 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fea3c051ec0 tx=0x7fea3c0828f0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 shutdown_connections 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ad1270 0x2ae6410 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2ad0950 0x2ae5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea20ff9700 0 client.7530 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2aafca0 0x2b29b60 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 --2- 172.21.15.119:0/1588416829 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2aaed00 0x2ab1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 >> 172.21.15.119:0/1588416829 conn(0x28195b0 msgr2=0x2ae8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 shutdown_connections 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 1 -- 172.21.15.119:0/1588416829 wait complete. 2024-09-13T16:30:37.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.054+0000 7fea4ae9a740 20 client.7530 trim_cache size 0 max 16384 2024-09-13T16:30:37.063 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:30:37.063 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:30:37.063 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:30:37.063 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea24066730 msgr2=0x7fea24068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea24066730 0x7fea24068bf0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fea300019b0 tx=0x7fea300012f0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea400d1640 msgr2=0x7fea401f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea400d1640 0x7fea401f6510 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fea3c00ebe0 tx=0x7fea3c00eef0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 shutdown_connections 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea24066730 0x7fea24068bf0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea4015cfb0 0x7fea401f6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea400d1640 0x7fea401f6510 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 --2- 172.21.15.119:0/2284338708 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea4006b710 0x7fea400b7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 >> 172.21.15.119:0/2284338708 conn(0x7fea400d2da0 msgr2=0x7fea400d3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 shutdown_connections 2024-09-13T16:30:37.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.058+0000 7fea47be9700 1 -- 172.21.15.119:0/2284338708 wait complete. 2024-09-13T16:30:37.077 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:30:37.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:30:37.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb587564700 1 Processor -- start 2024-09-13T16:30:37.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb587564700 1 -- start start 2024-09-13T16:30:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb587564700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 0x7fb5800ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb587564700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58015d020 con 0x7fb5800cd860 2024-09-13T16:30:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb587564700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5800d0720 con 0x7fb58015cc40 2024-09-13T16:30:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb587564700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800b44c0 con 0x7fb58008c6b0 2024-09-13T16:30:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb586d63700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb5800cd860 0x7fb580166a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55668/0 (socket says 172.21.15.119:55668) 2024-09-13T16:30:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb586d63700 1 -- 172.21.15.119:0/3791936545 learned_addr learned my addr 172.21.15.119:0/3791936545 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:30:37.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585d61700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 0x7fb5800ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600927206 0 0) 0x7fb58015d020 con 0x7fb5800cd860 2024-09-13T16:30:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb56c0035d0 con 0x7fb5800cd860 2024-09-13T16:30:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2874596204 0 0) 0x7fb56c0035d0 con 0x7fb5800cd860 2024-09-13T16:30:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 msgr2=0x7fb5800ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 0x7fb5800ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 >> v1:172.21.15.145:6789/0 conn(0x7fb58015cc40 legacy=0x7fb58008bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5800d09c0 con 0x7fb5800cd860 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fb5780019f0 con 0x7fb5800cd860 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 >> v1:172.21.15.119:6789/0 conn(0x7fb5800cd860 legacy=0x7fb580166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c003a50 0x7fb56c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb56c004870 0x7fb56c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 0x7fb56c005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800b44c0 con 0x7fb56c004870 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c00bb10 con 0x7fb56c003a50 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56c00bcd0 con 0x7fb58008c6b0 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.186+0000 7fb585d61700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb56c004870 0x7fb56c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb586562700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c003a50 0x7fb56c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb586d63700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 0x7fb56c005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585d61700 1 -- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 msgr2=0x7fb56c005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585d61700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 0x7fb56c005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585d61700 1 -- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c003a50 msgr2=0x7fb56c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585d61700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c003a50 0x7fb56c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585d61700 1 -- 172.21.15.119:0/3791936545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0035d0 con 0x7fb56c004870 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb586d63700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb58008c6b0 0x7fb56c005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585d61700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb56c004870 0x7fb56c004cf0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009b40 tx=0x7fb56c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0ec13127202bd84 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00e6c0 con 0x7fb56c004870 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c00e820 con 0x7fb56c004870 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb585560700 1 -- 172.21.15.119:0/3791936545 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00eb90 con 0x7fb56c004870 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb587564700 1 -- 172.21.15.119:0/3791936545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb56c004870 msgr2=0x7fb56c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb587564700 1 --2- 172.21.15.119:0/3791936545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb56c004870 0x7fb56c004cf0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fb57c009b40 tx=0x7fb56c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb586d63700 1 -- 172.21.15.119:0/3791936545 reap_dead start 2024-09-13T16:30:37.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb587564700 1 -- 172.21.15.119:0/3791936545 shutdown_connections 2024-09-13T16:30:37.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb587564700 1 -- 172.21.15.119:0/3791936545 >> 172.21.15.119:0/3791936545 conn(0x7fb5800b4640 msgr2=0x7fb5800b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb587564700 1 -- 172.21.15.119:0/3791936545 shutdown_connections 2024-09-13T16:30:37.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb587564700 1 -- 172.21.15.119:0/3791936545 wait complete. 2024-09-13T16:30:37.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.190+0000 7fb587564700 1 Processor -- start 2024-09-13T16:30:37.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 -- start start 2024-09-13T16:30:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c004870 0x7fb5801f3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5801f3b40 0x7fb58009fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5800a03e0 0x7fb58009cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5800a0890 con 0x7fb5801f3b40 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb58009eb60 con 0x7fb56c004870 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb58009ee00 con 0x7fb5800a03e0 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5801f3b40 0x7fb58009fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5801f3b40 0x7fb58009fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57240/0 (socket says 172.21.15.119:57240) 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 -- 172.21.15.119:0/3780821597 learned_addr learned my addr 172.21.15.119:0/3780821597 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb586562700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c004870 0x7fb5801f3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb586d63700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5800a03e0 0x7fb58009cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 -- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5800a03e0 msgr2=0x7fb58009cb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5800a03e0 0x7fb58009cb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 -- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c004870 msgr2=0x7fb5801f3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c004870 0x7fb5801f3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 -- 172.21.15.119:0/3780821597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58009f0e0 con 0x7fb5801f3b40 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb585d61700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5801f3b40 0x7fb58009fea0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00bae0 tx=0x7fb57c0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb576ffd700 1 -- 172.21.15.119:0/3780821597 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c00e6c0 con 0x7fb5801f3b40 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb576ffd700 1 -- 172.21.15.119:0/3780821597 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb57c013d50 con 0x7fb5801f3b40 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58009f3c0 con 0x7fb5801f3b40 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb576ffd700 1 -- 172.21.15.119:0/3780821597 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb57c004270 con 0x7fb5801f3b40 2024-09-13T16:30:37.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.194+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58009f980 con 0x7fb5801f3b40 2024-09-13T16:30:37.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.198+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58009fb10 con 0x7fb5801f3b40 2024-09-13T16:30:37.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.198+0000 7fb576ffd700 1 -- 172.21.15.119:0/3780821597 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb57c0043d0 con 0x7fb5801f3b40 2024-09-13T16:30:37.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.198+0000 7fb576ffd700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb564066760 0x7fb564068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.198+0000 7fb576ffd700 1 -- 172.21.15.119:0/3780821597 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fb57c085a50 con 0x7fb5801f3b40 2024-09-13T16:30:37.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.198+0000 7fb586562700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb564066760 0x7fb564068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.198+0000 7fb586562700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb564066760 0x7fb564068c20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb570007fd0 tx=0x7fb570009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.206+0000 7fb576ffd700 1 -- 172.21.15.119:0/3780821597 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb57c052dd0 con 0x7fb5801f3b40 2024-09-13T16:30:37.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.410+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5800a07f0 con 0x7fb5801f3b40 2024-09-13T16:30:37.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.410+0000 7fb576ffd700 1 -- 172.21.15.119:0/3780821597 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 48 v48) v1 ==== 94+0+4743 (secure 0 0 0) 0x7fb57c0563f0 con 0x7fb5801f3b40 2024-09-13T16:30:37.414 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:30:37.414 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":48,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":48,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:29:59.936379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":6,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb564066760 msgr2=0x7fb564068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb564066760 0x7fb564068c20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fb570007fd0 tx=0x7fb570009040 comp rx=0 tx=0).stop 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5801f3b40 msgr2=0x7fb58009fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5801f3b40 0x7fb58009fea0 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7fb56c00bae0 tx=0x7fb57c0047e0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 shutdown_connections 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb564066760 0x7fb564068c20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5800a03e0 0x7fb58009cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5801f3b40 0x7fb58009fea0 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 --2- 172.21.15.119:0/3780821597 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb56c004870 0x7fb5801f3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 >> 172.21.15.119:0/3780821597 conn(0x7fb5800b4640 msgr2=0x7fb5800ceac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 shutdown_connections 2024-09-13T16:30:37.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.414+0000 7fb587564700 1 -- 172.21.15.119:0/3780821597 wait complete. 2024-09-13T16:30:37.418 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 48 2024-09-13T16:30:37.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-09-13T16:30:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 Processor -- start 2024-09-13T16:30:37.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- start start 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbf300ce5f0 0x7fbf300cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf300d0fb0 0x7fbf300d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf300cd600 con 0x7fbf3015cae0 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf300ce1d0 con 0x7fbf300ce5f0 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300cea60 con 0x7fbf300d0fb0 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbf300ce5f0 0x7fbf300cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbf300ce5f0 0x7fbf300cff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57244/0 (socket says 172.21.15.119:57244) 2024-09-13T16:30:37.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2ed9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbf3015cae0 0x7fbf300d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55684/0 (socket says 172.21.15.119:55684) 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 -- 172.21.15.119:0/3897657154 learned_addr learned my addr 172.21.15.119:0/3897657154 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/3897657154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf300d0fb0 0x7fbf300d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 -- 172.21.15.119:0/3897657154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf300d0fb0 msgr2=0x7fbf300d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/3897657154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf300d0fb0 0x7fbf300d1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 -- 172.21.15.119:0/3897657154 >> v1:172.21.15.119:6789/0 conn(0x7fbf3015cae0 legacy=0x7fbf300d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 -- 172.21.15.119:0/3897657154 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf300d0dd0 con 0x7fbf300ce5f0 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/3897657154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300ce5f0 0x7fbf300cff60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fbf28009e80 tx=0x7fbf2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d97cb47eae6c994a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2dd9b700 1 -- 172.21.15.119:0/3897657154 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2800e070 con 0x7fbf300ce5f0 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2dd9b700 1 -- 172.21.15.119:0/3897657154 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf2800b980 con 0x7fbf300ce5f0 2024-09-13T16:30:37.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf2dd9b700 1 -- 172.21.15.119:0/3897657154 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2800bd30 con 0x7fbf300ce5f0 2024-09-13T16:30:37.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- 172.21.15.119:0/3897657154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300ce5f0 msgr2=0x7fbf300cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 --2- 172.21.15.119:0/3897657154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300ce5f0 0x7fbf300cff60 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7fbf28009e80 tx=0x7fbf2800b330 comp rx=0 tx=0).stop 2024-09-13T16:30:37.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- 172.21.15.119:0/3897657154 shutdown_connections 2024-09-13T16:30:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 --2- 172.21.15.119:0/3897657154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf300d0fb0 0x7fbf300d1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 --2- 172.21.15.119:0/3897657154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300ce5f0 0x7fbf300cff60 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- 172.21.15.119:0/3897657154 >> 172.21.15.119:0/3897657154 conn(0x7fbf300d29f0 msgr2=0x7fbf300d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.546+0000 7fbf34a00700 1 -- 172.21.15.119:0/3897657154 shutdown_connections 2024-09-13T16:30:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 -- 172.21.15.119:0/3897657154 wait complete. 2024-09-13T16:30:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 Processor -- start 2024-09-13T16:30:37.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 -- start start 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf300ce5f0 0x7fbf300c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300d0fb0 0x7fbf3015b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf3015cae0 0x7fbf300cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300caf90 con 0x7fbf300d0fb0 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300ca310 con 0x7fbf300ce5f0 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf34a00700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf300ca5f0 con 0x7fbf3015cae0 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300d0fb0 0x7fbf3015b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300d0fb0 0x7fbf3015b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57252/0 (socket says 172.21.15.119:57252) 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 -- 172.21.15.119:0/1308040166 learned_addr learned my addr 172.21.15.119:0/1308040166 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf300ce5f0 0x7fbf300c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf3015cae0 0x7fbf300cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 -- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf3015cae0 msgr2=0x7fbf300cb3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf3015cae0 0x7fbf300cb3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 -- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf300ce5f0 msgr2=0x7fbf300c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf300ce5f0 0x7fbf300c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.550+0000 7fbf2e59c700 1 -- 172.21.15.119:0/1308040166 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf28009b30 con 0x7fbf300d0fb0 2024-09-13T16:30:37.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.554+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300d0fb0 0x7fbf3015b8d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbf28000f80 tx=0x7fbf28004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.554+0000 7fbf277fe700 1 -- 172.21.15.119:0/1308040166 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf28004880 con 0x7fbf300d0fb0 2024-09-13T16:30:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.554+0000 7fbf277fe700 1 -- 172.21.15.119:0/1308040166 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf280099b0 con 0x7fbf300d0fb0 2024-09-13T16:30:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.554+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf300ca8d0 con 0x7fbf300d0fb0 2024-09-13T16:30:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.554+0000 7fbf277fe700 1 -- 172.21.15.119:0/1308040166 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2801b670 con 0x7fbf300d0fb0 2024-09-13T16:30:37.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.554+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf300ca000 con 0x7fbf300d0fb0 2024-09-13T16:30:37.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.558+0000 7fbf277fe700 1 -- 172.21.15.119:0/1308040166 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbf2801b7d0 con 0x7fbf300d0fb0 2024-09-13T16:30:37.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.558+0000 7fbf277fe700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf10066730 0x7fbf10068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.558+0000 7fbf277fe700 1 -- 172.21.15.119:0/1308040166 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fbf280857d0 con 0x7fbf300d0fb0 2024-09-13T16:30:37.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.558+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf10066730 0x7fbf10068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.558+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf10066730 0x7fbf10068bf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbf18000f60 tx=0x7fbf18007eb0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.558+0000 7fbf384b2740 1 Processor -- start 2024-09-13T16:30:37.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.558+0000 7fbf384b2740 1 -- start start 2024-09-13T16:30:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x206c520 0x2123410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x211fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123d30 0x2120210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x204b810 con 0x2123d30 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2048fa0 con 0x2123950 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2044890 con 0x206c520 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123d30 0x2120210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123d30 0x2120210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57262/0 (socket says 172.21.15.119:57262) 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 -- 172.21.15.119:0/99606306 learned_addr learned my addr 172.21.15.119:0/99606306 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x206c520 0x2123410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x211fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 -- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x206c520 msgr2=0x2123410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x206c520 0x2123410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 -- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 msgr2=0x211fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x211fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 -- 172.21.15.119:0/99606306 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1eb6ba0 con 0x2123d30 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123d30 0x2120210 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbf200099f0 tx=0x7fbf20009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c95d69e982367909 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf24ff9700 1 -- 172.21.15.119:0/99606306 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000c070 con 0x2123d30 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf24ff9700 1 -- 172.21.15.119:0/99606306 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf20009e90 con 0x2123d30 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf24ff9700 1 -- 172.21.15.119:0/99606306 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20011bb0 con 0x2123d30 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- 172.21.15.119:0/99606306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123d30 msgr2=0x2120210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123d30 0x2120210 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7fbf200099f0 tx=0x7fbf20009d00 comp rx=0 tx=0).stop 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- 172.21.15.119:0/99606306 shutdown_connections 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123d30 0x2120210 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x211fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 --2- 172.21.15.119:0/99606306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x206c520 0x2123410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- 172.21.15.119:0/99606306 >> 172.21.15.119:0/99606306 conn(0x1e6b710 msgr2=0x21056d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- 172.21.15.119:0/99606306 shutdown_connections 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- 172.21.15.119:0/99606306 wait complete. 2024-09-13T16:30:37.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.562+0000 7fbf384b2740 1 -- start start 2024-09-13T16:30:37.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 Processor -- start 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x206c520 0x2101e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123950 0x212d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x212d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2007d80 con 0x2123950 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2007a20 con 0x206c520 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f37770 con 0x2123d30 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 10 client.0 _ll_register_callbacks cb 0x203df00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123950 0x212d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123950 0x212d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57268/0 (socket says 172.21.15.119:57268) 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 -- 172.21.15.119:0/391232333 learned_addr learned my addr 172.21.15.119:0/391232333 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x206c520 0x2101e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x212d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 -- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 msgr2=0x212d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x212d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 -- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x206c520 msgr2=0x2101e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x206c520 0x2101e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 -- 172.21.15.119:0/391232333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf2000e040 con 0x2123950 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x212d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x206c520 0x2101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123950 0x212d420 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbf28009fd0 tx=0x7fbf28056650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=199e4439e4a9c2b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf0dffb700 1 -- 172.21.15.119:0/391232333 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2805a230 con 0x2123950 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf0dffb700 1 -- 172.21.15.119:0/391232333 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fbf280092e0 con 0x2123950 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1e78420 con 0x2123950 2024-09-13T16:30:37.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf0dffb700 1 -- 172.21.15.119:0/391232333 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf280568b0 con 0x2123950 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 10 client.7551 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x200a910 con 0x2123950 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf0dffb700 1 -- 172.21.15.119:0/391232333 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1e78420 con 0x2123950 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf0dffb700 1 -- 172.21.15.119:0/391232333 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbf28055b40 con 0x2123950 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 10 client.7551 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 10 client.7551 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 20 client.7551 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x21346e0 0x2136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 4 client.7551 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1ec26a0 con 0x21346e0 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.566+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x21346e0 0x2136ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf0dffb700 10 client.7551 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x21346e0 0x2136ad0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc0bd8906b4324f server_cookie=511e78d98aa7952d in_seq=0 out_seq=0 2024-09-13T16:30:37.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf0dffb700 1 -- 172.21.15.119:0/391232333 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1ec26a0 con 0x21346e0 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf0dffb700 10 client.7551 handle_command_reply: tid=0 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 client.7551 shutdown 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123950 msgr2=0x212d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123950 0x212d420 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7fbf28009fd0 tx=0x7fbf28056650 comp rx=0 tx=0).stop 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 shutdown_connections 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x21346e0 0x2136ad0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x212d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2123950 0x212d420 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf0dffb700 0 client.7551 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 --2- 172.21.15.119:0/391232333 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x206c520 0x2101e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 >> 172.21.15.119:0/391232333 conn(0x1e6b710 msgr2=0x2120310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 shutdown_connections 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 1 -- 172.21.15.119:0/391232333 wait complete. 2024-09-13T16:30:37.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.570+0000 7fbf384b2740 20 client.7551 trim_cache size 0 max 16384 2024-09-13T16:30:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 Processor -- start 2024-09-13T16:30:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 -- start start 2024-09-13T16:30:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x212d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123950 0x20fbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123d30 0x20fc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x204caa0 con 0x206c520 2024-09-13T16:30:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x20020e0 con 0x2123d30 2024-09-13T16:30:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x206b9a0 con 0x2123950 2024-09-13T16:30:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf2ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x212d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.582+0000 7fbf2ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x212d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57282/0 (socket says 172.21.15.119:57282) 2024-09-13T16:30:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/3212523017 learned_addr learned my addr 172.21.15.119:0/3212523017 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123950 0x20fbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123d30 0x20fc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123950 msgr2=0x20fbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123950 0x20fbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123d30 msgr2=0x20fc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123d30 0x20fc520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/3212523017 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1ff0b40 con 0x206c520 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123d30 0x20fc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x212d790 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbf3015c460 tx=0x7fbf18013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc6f4fecae2a517 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf257fa700 1 -- 172.21.15.119:0/3212523017 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1802b070 con 0x206c520 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf257fa700 1 -- 172.21.15.119:0/3212523017 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf18001cb0 con 0x206c520 2024-09-13T16:30:37.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf257fa700 1 -- 172.21.15.119:0/3212523017 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf18022650 con 0x206c520 2024-09-13T16:30:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 -- 172.21.15.119:0/3212523017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 msgr2=0x212d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x212d790 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7fbf3015c460 tx=0x7fbf18013000 comp rx=0 tx=0).stop 2024-09-13T16:30:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 -- 172.21.15.119:0/3212523017 shutdown_connections 2024-09-13T16:30:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123d30 0x20fc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123950 0x20fbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 --2- 172.21.15.119:0/3212523017 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x212d790 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 -- 172.21.15.119:0/3212523017 >> 172.21.15.119:0/3212523017 conn(0x1e6b710 msgr2=0x20fb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 -- 172.21.15.119:0/3212523017 shutdown_connections 2024-09-13T16:30:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 -- 172.21.15.119:0/3212523017 wait complete. 2024-09-13T16:30:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 -- start start 2024-09-13T16:30:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.586+0000 7fbf384b2740 1 Processor -- start 2024-09-13T16:30:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x2183e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x2140550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x2140a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2005660 con 0x206c520 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x20420a0 con 0x2123950 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x20473f0 con 0x2123d30 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 10 client.0 _ll_register_callbacks cb 0x2043460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x2183e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x2183e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57286/0 (socket says 172.21.15.119:57286) 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/4263110182 learned_addr learned my addr 172.21.15.119:0/4263110182 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x2140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x2140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 msgr2=0x2140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x2140a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 msgr2=0x2140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x2140550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 -- 172.21.15.119:0/4263110182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf300cd600 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2f59e700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x2140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2ed9d700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x2183e00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbf18025040 tx=0x7fbf18000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5eb5f89cfed39c21 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 1 -- 172.21.15.119:0/4263110182 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1802b070 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 1 -- 172.21.15.119:0/4263110182 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fbf1801b040 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf20009e90 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 1 -- 172.21.15.119:0/4263110182 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf18022b80 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 10 client.7557 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x1fe4c20 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 1 -- 172.21.15.119:0/4263110182 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf20009e90 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 1 -- 172.21.15.119:0/4263110182 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbf18022ce0 con 0x206c520 2024-09-13T16:30:37.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 10 client.7557 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 10 client.7557 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 20 client.7557 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2100940 0x2100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 4 client.7557 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x1ee7420 con 0x2100940 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2100940 0x2100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 10 client.7557 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf2e59c700 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2100940 0x2100d20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7d55a91f10841ac server_cookie=6e3badcc67319bee in_seq=0 out_seq=0 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 1 -- 172.21.15.119:0/4263110182 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+237 (crc 0 0 0) 0x1ee7420 con 0x2100940 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf0dffb700 10 client.7557 handle_command_reply: tid=0 2024-09-13T16:30:37.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 client.7557 shutdown 2024-09-13T16:30:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 msgr2=0x2183e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.590+0000 7fbf384b2740 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x2183e00 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7fbf18025040 tx=0x7fbf18000ed0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 shutdown_connections 2024-09-13T16:30:37.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2100940 0x2100d20 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2123d30 0x2140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf0dffb700 0 client.7557 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2123950 0x2140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 --2- 172.21.15.119:0/4263110182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x206c520 0x2183e00 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 >> 172.21.15.119:0/4263110182 conn(0x1e6b710 msgr2=0x20fcb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 shutdown_connections 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 1 -- 172.21.15.119:0/4263110182 wait complete. 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf384b2740 20 client.7557 trim_cache size 0 max 16384 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:30:37.599 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "damage_type": "dentry", 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 2094202721, 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627776, 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "frag": "*", 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "dname": "file_to_be_damaged", 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "snap_id": "head", 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/subdir/file_to_be_damaged" 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:30:37.600 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:30:37.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf10066730 msgr2=0x7fbf10068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf10066730 0x7fbf10068bf0 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fbf18000f60 tx=0x7fbf18007eb0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300d0fb0 msgr2=0x7fbf3015b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300d0fb0 0x7fbf3015b8d0 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7fbf28000f80 tx=0x7fbf28004af0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 shutdown_connections 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf10066730 0x7fbf10068bf0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf3015cae0 0x7fbf300cb3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf300d0fb0 0x7fbf3015b8d0 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 --2- 172.21.15.119:0/1308040166 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf300ce5f0 0x7fbf300c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 >> 172.21.15.119:0/1308040166 conn(0x7fbf300d29f0 msgr2=0x7fbf30159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 shutdown_connections 2024-09-13T16:30:37.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.594+0000 7fbf34a00700 1 -- 172.21.15.119:0/1308040166 wait complete. 2024-09-13T16:30:37.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:30:37.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.722+0000 7f755b172700 1 Processor -- start 2024-09-13T16:30:37.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.722+0000 7f755b172700 1 -- start start 2024-09-13T16:30:37.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f75540cd730 0x7f755406bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f755406fd00 0x7f755406c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75540adf20 con 0x7f755415c6c0 2024-09-13T16:30:37.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f755415caa0 con 0x7f75540cd730 2024-09-13T16:30:37.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75540ad7b0 con 0x7f755406fd00 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755996f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f75540cd730 0x7f755406bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755a170700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f755415c6c0 0x7f75540cd020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55698/0 (socket says 172.21.15.119:55698) 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755a170700 1 -- 172.21.15.119:0/2796651256 learned_addr learned my addr 172.21.15.119:0/2796651256 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755a971700 1 --2- 172.21.15.119:0/2796651256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f755406fd00 0x7f755406c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755996f700 1 -- 172.21.15.119:0/2796651256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f755406fd00 msgr2=0x7f755406c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755996f700 1 --2- 172.21.15.119:0/2796651256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f755406fd00 0x7f755406c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755996f700 1 -- 172.21.15.119:0/2796651256 >> v1:172.21.15.119:6789/0 conn(0x7f755415c6c0 legacy=0x7f75540cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755996f700 1 -- 172.21.15.119:0/2796651256 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75540ce140 con 0x7f75540cd730 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755996f700 1 --2- 172.21.15.119:0/2796651256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75540cd730 0x7f755406bc70 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f7544009850 tx=0x7f7544009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=985ce63b14ad25e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755916e700 1 -- 172.21.15.119:0/2796651256 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544011830 con 0x7f75540cd730 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755916e700 1 -- 172.21.15.119:0/2796651256 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7544011990 con 0x7f75540cd730 2024-09-13T16:30:37.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755916e700 1 -- 172.21.15.119:0/2796651256 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7544011d20 con 0x7f75540cd730 2024-09-13T16:30:37.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- 172.21.15.119:0/2796651256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75540cd730 msgr2=0x7f755406bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 --2- 172.21.15.119:0/2796651256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75540cd730 0x7f755406bc70 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f7544009850 tx=0x7f7544009b60 comp rx=0 tx=0).stop 2024-09-13T16:30:37.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- 172.21.15.119:0/2796651256 shutdown_connections 2024-09-13T16:30:37.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 --2- 172.21.15.119:0/2796651256 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f755406fd00 0x7f755406c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 --2- 172.21.15.119:0/2796651256 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75540cd730 0x7f755406bc70 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- 172.21.15.119:0/2796651256 >> 172.21.15.119:0/2796651256 conn(0x7f75540b0640 msgr2=0x7f75540b0a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- 172.21.15.119:0/2796651256 shutdown_connections 2024-09-13T16:30:37.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.726+0000 7f755b172700 1 -- 172.21.15.119:0/2796651256 wait complete. 2024-09-13T16:30:37.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 Processor -- start 2024-09-13T16:30:37.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 -- start start 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755406fd00 0x7f75541459c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75540cd730 0x7f7554148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f755415c6c0 0x7f75540cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7554158e20 con 0x7f755406fd00 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75541470f0 con 0x7f755415c6c0 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755b172700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75541473d0 con 0x7f75540cd730 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755406fd00 0x7f75541459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755406fd00 0x7f75541459c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57308/0 (socket says 172.21.15.119:57308) 2024-09-13T16:30:37.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 -- 172.21.15.119:0/3867556053 learned_addr learned my addr 172.21.15.119:0/3867556053 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755996f700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75540cd730 0x7f7554148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a971700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f755415c6c0 0x7f75540cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 -- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75540cd730 msgr2=0x7f7554148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75540cd730 0x7f7554148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 -- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f755415c6c0 msgr2=0x7f75540cb5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f755415c6c0 0x7f75540cb5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.730+0000 7f755a170700 1 -- 172.21.15.119:0/3867556053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75440094b0 con 0x7f755406fd00 2024-09-13T16:30:37.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f755a170700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755406fd00 0x7f75541459c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f755000c700 tx=0x7f755000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f754affd700 1 -- 172.21.15.119:0/3867556053 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550005d90 con 0x7f755406fd00 2024-09-13T16:30:37.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f754affd700 1 -- 172.21.15.119:0/3867556053 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75500070d0 con 0x7f755406fd00 2024-09-13T16:30:37.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7554159070 con 0x7f755406fd00 2024-09-13T16:30:37.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f754affd700 1 -- 172.21.15.119:0/3867556053 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7550007440 con 0x7f755406fd00 2024-09-13T16:30:37.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7554145f00 con 0x7f755406fd00 2024-09-13T16:30:37.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7554159590 con 0x7f755406fd00 2024-09-13T16:30:37.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.734+0000 7f754affd700 1 -- 172.21.15.119:0/3867556053 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f75500075a0 con 0x7f755406fd00 2024-09-13T16:30:37.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.738+0000 7f754affd700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7538066760 0x7f7538068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:37.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.738+0000 7f754affd700 1 -- 172.21.15.119:0/3867556053 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f75500833d0 con 0x7f755406fd00 2024-09-13T16:30:37.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.738+0000 7f755996f700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7538066760 0x7f7538068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:37.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.738+0000 7f755996f700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7538066760 0x7f7538068c20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7544009850 tx=0x7f7544008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:37.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.742+0000 7f754affd700 1 -- 172.21.15.119:0/3867556053 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7550050650 con 0x7f755406fd00 2024-09-13T16:30:37.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.950+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f75540adf20 con 0x7f755406fd00 2024-09-13T16:30:37.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.950+0000 7f754affd700 1 -- 172.21.15.119:0/3867556053 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f75540adf20 con 0x7f755406fd00 2024-09-13T16:30:37.953 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:30:37.953 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:30:37.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7538066760 msgr2=0x7f7538068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7538066760 0x7f7538068c20 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7544009850 tx=0x7f7544008030 comp rx=0 tx=0).stop 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755406fd00 msgr2=0x7f75541459c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755406fd00 0x7f75541459c0 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f755000c700 tx=0x7f755000ca10 comp rx=0 tx=0).stop 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 shutdown_connections 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7538066760 0x7f7538068c20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f755415c6c0 0x7f75540cb5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75540cd730 0x7f7554148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 --2- 172.21.15.119:0/3867556053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f755406fd00 0x7f75541459c0 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 >> 172.21.15.119:0/3867556053 conn(0x7f75540b0640 msgr2=0x7f7554157a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 shutdown_connections 2024-09-13T16:30:37.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:37.954+0000 7f755b172700 1 -- 172.21.15.119:0/3867556053 wait complete. 2024-09-13T16:30:37.968 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-13T16:30:37.968 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub start /subdir repair,recursive 2024-09-13T16:30:38.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.090+0000 7f36db291700 1 Processor -- start 2024-09-13T16:30:38.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.090+0000 7f36db291700 1 -- start start 2024-09-13T16:30:38.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.090+0000 7f36db291700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4156940 0x7f36d4070f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.090+0000 7f36db291700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d406fed0 con 0x7f36d4156d20 2024-09-13T16:30:38.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.090+0000 7f36db291700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40701b0 con 0x7f36d40714b0 2024-09-13T16:30:38.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.090+0000 7f36db291700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4070490 con 0x7f36d4156940 2024-09-13T16:30:38.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36d9a8e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f36d4156d20 0x7f36d40adbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55700/0 (socket says 172.21.15.119:55700) 2024-09-13T16:30:38.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36d9a8e700 1 -- 172.21.15.119:0/1643119084 learned_addr learned my addr 172.21.15.119:0/1643119084 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:30:38.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36da28f700 1 --2- 172.21.15.119:0/1643119084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4156940 0x7f36d4070f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36d928d700 1 -- 172.21.15.119:0/1643119084 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35802773 0 0) 0x7f36d406fed0 con 0x7f36d4156d20 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36d928d700 1 -- 172.21.15.119:0/1643119084 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36bc0035d0 con 0x7f36d4156d20 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36da28f700 1 -- 172.21.15.119:0/1643119084 >> v1:172.21.15.145:6789/0 conn(0x7f36d40714b0 legacy=0x7f36d40b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36da28f700 1 -- 172.21.15.119:0/1643119084 >> v1:172.21.15.119:6789/0 conn(0x7f36d4156d20 legacy=0x7f36d40adbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36da28f700 1 -- 172.21.15.119:0/1643119084 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4070770 con 0x7f36d4156940 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36da28f700 1 --2- 172.21.15.119:0/1643119084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4156940 0x7f36d4070f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009dc0 tx=0x7f36d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5920044125996ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36d928d700 1 -- 172.21.15.119:0/1643119084 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d000e070 con 0x7f36d4156940 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36d928d700 1 -- 172.21.15.119:0/1643119084 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d000bb20 con 0x7f36d4156940 2024-09-13T16:30:38.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36d928d700 1 -- 172.21.15.119:0/1643119084 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d000be90 con 0x7f36d4156940 2024-09-13T16:30:38.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36db291700 1 -- 172.21.15.119:0/1643119084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4156940 msgr2=0x7f36d4070f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36db291700 1 --2- 172.21.15.119:0/1643119084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4156940 0x7f36d4070f70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009dc0 tx=0x7f36d000b330 comp rx=0 tx=0).stop 2024-09-13T16:30:38.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36db291700 1 -- 172.21.15.119:0/1643119084 shutdown_connections 2024-09-13T16:30:38.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36db291700 1 --2- 172.21.15.119:0/1643119084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4156940 0x7f36d4070f70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36db291700 1 -- 172.21.15.119:0/1643119084 >> 172.21.15.119:0/1643119084 conn(0x7f36d406a350 msgr2=0x7f36d406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36db291700 1 -- 172.21.15.119:0/1643119084 shutdown_connections 2024-09-13T16:30:38.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.094+0000 7f36db291700 1 -- 172.21.15.119:0/1643119084 wait complete. 2024-09-13T16:30:38.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 Processor -- start 2024-09-13T16:30:38.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 -- start start 2024-09-13T16:30:38.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4156940 0x7f36d4146ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d40a99c0 0x7f36d4166ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4145b70 0x7f36d4148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40a8960 con 0x7f36d4156940 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40a8c20 con 0x7f36d40a99c0 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d40a8f00 con 0x7f36d4145b70 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36da28f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4156940 0x7f36d4146ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36da28f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4156940 0x7f36d4146ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57320/0 (socket says 172.21.15.119:57320) 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36d9a8e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d40a99c0 0x7f36d4166ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36daa90700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4145b70 0x7f36d4148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36da28f700 1 -- 172.21.15.119:0/3396010477 learned_addr learned my addr 172.21.15.119:0/3396010477 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36daa90700 1 -- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d40a99c0 msgr2=0x7f36d4166ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36daa90700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d40a99c0 0x7f36d4166ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36daa90700 1 -- 172.21.15.119:0/3396010477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4156940 msgr2=0x7f36d4146ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36daa90700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4156940 0x7f36d4146ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36daa90700 1 -- 172.21.15.119:0/3396010477 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d0009a70 con 0x7f36d4145b70 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36da28f700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4156940 0x7f36d4146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36daa90700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4145b70 0x7f36d4148c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f36cc001eb0 tx=0x7f36cc00c6e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36caffd700 1 -- 172.21.15.119:0/3396010477 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0052d0 con 0x7f36d4145b70 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36caffd700 1 -- 172.21.15.119:0/3396010477 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc005430 con 0x7f36d4145b70 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36caffd700 1 -- 172.21.15.119:0/3396010477 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0055a0 con 0x7f36d4145b70 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d40a91e0 con 0x7f36d4145b70 2024-09-13T16:30:38.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.098+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d4167420 con 0x7f36d4145b70 2024-09-13T16:30:38.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.102+0000 7f36caffd700 1 -- 172.21.15.119:0/3396010477 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f36cc003af0 con 0x7f36d4145b70 2024-09-13T16:30:38.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.102+0000 7f36caffd700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36c0066760 0x7f36c0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.102+0000 7f36caffd700 1 -- 172.21.15.119:0/3396010477 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f36cc083780 con 0x7f36d4145b70 2024-09-13T16:30:38.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36da28f700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36c0066760 0x7f36c0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36da28f700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36c0066760 0x7f36c0068c20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009d90 tx=0x7f36d0011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 Processor -- start 2024-09-13T16:30:38.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 -- start start 2024-09-13T16:30:38.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b97af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b9a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b9c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2abe530 con 0x2b9a580 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2abbcc0 con 0x2b525c0 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2ab75b0 con 0x2b521e0 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b9c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b9c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57330/0 (socket says 172.21.15.119:57330) 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 -- 172.21.15.119:0/3231193761 learned_addr learned my addr 172.21.15.119:0/3231193761 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b9a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36da28f700 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b97af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 -- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 msgr2=0x2b97af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b97af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 -- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 msgr2=0x2b9a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b9a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 -- 172.21.15.119:0/3231193761 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x29a9f10 con 0x2b9a580 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b9a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36daa90700 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b9c970 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f36cc081a80 tx=0x7f36cc053fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=800ea57e82ede0f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36bbfff700 1 -- 172.21.15.119:0/3231193761 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc0583c0 con 0x2b9a580 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36bbfff700 1 -- 172.21.15.119:0/3231193761 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc05c980 con 0x2b9a580 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.106+0000 7f36bbfff700 1 -- 172.21.15.119:0/3231193761 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc052930 con 0x2b9a580 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- 172.21.15.119:0/3231193761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 msgr2=0x2b9c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b9c970 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f36cc081a80 tx=0x7f36cc053fc0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- 172.21.15.119:0/3231193761 shutdown_connections 2024-09-13T16:30:38.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b9c970 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b9a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3231193761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b97af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- 172.21.15.119:0/3231193761 >> 172.21.15.119:0/3231193761 conn(0x28de7b0 msgr2=0x2b78ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- 172.21.15.119:0/3231193761 shutdown_connections 2024-09-13T16:30:38.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- 172.21.15.119:0/3231193761 wait complete. 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- start start 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 Processor -- start 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b502a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b75690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b75bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a7a950 con 0x2b9a580 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a7a5f0 con 0x2b525c0 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a4ce40 con 0x2b521e0 2024-09-13T16:30:38.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 10 client.0 _ll_register_callbacks cb 0x2ab0c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b75bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b75bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57340/0 (socket says 172.21.15.119:57340) 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 -- 172.21.15.119:0/3136052027 learned_addr learned my addr 172.21.15.119:0/3136052027 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36da28f700 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 -- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 msgr2=0x2b502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 -- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 msgr2=0x2b75690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b75690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b75690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 -- 172.21.15.119:0/3136052027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d4167420 con 0x2b9a580 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36daa90700 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b75bd0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f36cc05c450 tx=0x7f36cc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bf78507dea1434e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36b97fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36b97fa700 1 -- 172.21.15.119:0/3136052027 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36cc058440 con 0x2b9a580 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36b97fa700 1 -- 172.21.15.119:0/3136052027 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f36cc06acc0 con 0x2b9a580 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.110+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x28eb4c0 con 0x2b9a580 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36b97fa700 1 -- 172.21.15.119:0/3136052027 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36cc06ae80 con 0x2b9a580 2024-09-13T16:30:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 10 client.7578 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x2a7d630 con 0x2b9a580 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36b97fa700 1 -- 172.21.15.119:0/3136052027 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x28eb4c0 con 0x2b9a580 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36b97fa700 1 -- 172.21.15.119:0/3136052027 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f36cc053a80 con 0x2b9a580 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 10 client.7578 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 10 client.7578 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 20 client.7578 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b507e0 0x2b4d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 4 client.7578 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2935740 con 0x2b507e0 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36da28f700 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b507e0 0x2b4d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36da28f700 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b507e0 0x2b4d560 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9ad2bbbe97337b4 server_cookie=7ba158fccd2ef731 in_seq=0 out_seq=0 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36b97fa700 10 client.7578 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36b97fa700 1 -- 172.21.15.119:0/3136052027 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x2935740 con 0x2b507e0 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36b97fa700 10 client.7578 handle_command_reply: tid=0 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 client.7578 shutdown 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 msgr2=0x2b75bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b75bd0 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f36cc05c450 tx=0x7f36cc007f90 comp rx=0 tx=0).stop 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 shutdown_connections 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b507e0 0x2b4d560 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b9a580 0x2b75bd0 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2b75690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 --2- 172.21.15.119:0/3136052027 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36b97fa700 0 client.7578 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 >> 172.21.15.119:0/3136052027 conn(0x28de7b0 msgr2=0x2b98a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 shutdown_connections 2024-09-13T16:30:38.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 1 -- 172.21.15.119:0/3136052027 wait complete. 2024-09-13T16:30:38.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.114+0000 7f36df7a4740 20 client.7578 trim_cache size 0 max 16384 2024-09-13T16:30:38.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.134+0000 7f36df7a4740 1 Processor -- start 2024-09-13T16:30:38.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- start start 2024-09-13T16:30:38.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b92e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b525c0 0x2b4c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b9a580 0x2b4cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a555d0 con 0x2b525c0 2024-09-13T16:30:38.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a41910 con 0x2b9a580 2024-09-13T16:30:38.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a7b0a0 con 0x2b521e0 2024-09-13T16:30:38.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b525c0 0x2b4c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b525c0 0x2b4c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57348/0 (socket says 172.21.15.119:57348) 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 -- 172.21.15.119:0/4278828503 learned_addr learned my addr 172.21.15.119:0/4278828503 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36daa90700 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b9a580 0x2b4cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36da28f700 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b92e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 -- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 msgr2=0x2b92e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b92e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 -- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b9a580 msgr2=0x2b4cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b9a580 0x2b4cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 -- 172.21.15.119:0/4278828503 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2a63860 con 0x2b525c0 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b525c0 0x2b4c580 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f36c4006ba0 tx=0x7f36c4007de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8ec1b35c6305114 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36c8ff9700 1 -- 172.21.15.119:0/4278828503 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4016480 con 0x2b525c0 2024-09-13T16:30:38.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36c8ff9700 1 -- 172.21.15.119:0/4278828503 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c40165e0 con 0x2b525c0 2024-09-13T16:30:38.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36c8ff9700 1 -- 172.21.15.119:0/4278828503 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c4016990 con 0x2b525c0 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- 172.21.15.119:0/4278828503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b525c0 msgr2=0x2b4c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b525c0 0x2b4c580 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f36c4006ba0 tx=0x7f36c4007de0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- 172.21.15.119:0/4278828503 shutdown_connections 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b9a580 0x2b4cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b525c0 0x2b4c580 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 --2- 172.21.15.119:0/4278828503 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b521e0 0x2b92e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- 172.21.15.119:0/4278828503 >> 172.21.15.119:0/4278828503 conn(0x28de7b0 msgr2=0x2ba7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- 172.21.15.119:0/4278828503 shutdown_connections 2024-09-13T16:30:38.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.138+0000 7f36df7a4740 1 -- 172.21.15.119:0/4278828503 wait complete. 2024-09-13T16:30:38.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 -- start start 2024-09-13T16:30:38.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 Processor -- start 2024-09-13T16:30:38.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b521e0 0x2c00db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2c012f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b9a580 0x2b9fc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b9f590 con 0x2b521e0 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b9f770 con 0x2b525c0 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a71a50 con 0x2b9a580 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b521e0 0x2c00db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36df7a4740 10 client.0 _ll_register_callbacks cb 0x2ab6180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b521e0 0x2c00db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57352/0 (socket says 172.21.15.119:57352) 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 -- 172.21.15.119:0/1249328948 learned_addr learned my addr 172.21.15.119:0/1249328948 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2c012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36daa90700 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b9a580 0x2b9fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 -- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b9a580 msgr2=0x2b9fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b9a580 0x2b9fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 -- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 msgr2=0x2c012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2c012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.142+0000 7f36da28f700 1 -- 172.21.15.119:0/1249328948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36bc0035d0 con 0x2b521e0 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36da28f700 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b521e0 0x2c00db0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f36d000b370 tx=0x7f36d0004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153b372df05ad4ba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 1 -- 172.21.15.119:0/1249328948 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d002f040 con 0x2b521e0 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 1 -- 172.21.15.119:0/1249328948 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f36d0034670 con 0x2b521e0 2024-09-13T16:30:38.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f36cc05c980 con 0x2b521e0 2024-09-13T16:30:38.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 1 -- 172.21.15.119:0/1249328948 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f36d0034830 con 0x2b521e0 2024-09-13T16:30:38.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 10 client.7584 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:38.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x2929c40 con 0x2b521e0 2024-09-13T16:30:38.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 1 -- 172.21.15.119:0/1249328948 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d0043070 con 0x2b521e0 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 1 -- 172.21.15.119:0/1249328948 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f36d0012790 con 0x2b521e0 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 10 client.7584 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 10 client.7584 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 20 client.7584 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba0150 0x2b9dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 4 client.7584 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}] 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "scrub start", "path": "/subdir", "scrubops": ["repair", "recursive"]}) v1 -- 0x295c620 con 0x2ba0150 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba0150 0x2b9dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36d9a8e700 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba0150 0x2b9dbd0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45e454d90b6ce004 server_cookie=bf9f04edc380088b in_seq=0 out_seq=0 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 10 client.7584 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 1 -- 172.21.15.119:0/1249328948 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x295c620 con 0x2ba0150 2024-09-13T16:30:38.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 10 client.7584 handle_command_reply: tid=0 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 client.7584 shutdown 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b521e0 msgr2=0x2c00db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b521e0 0x2c00db0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7f36d000b370 tx=0x7f36d0004b60 comp rx=0 tx=0).stop 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 shutdown_connections 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba0150 0x2b9dbd0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b9a580 0x2b9fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36b97fa700 0 client.7584 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b525c0 0x2c012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 --2- 172.21.15.119:0/1249328948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b521e0 0x2c00db0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.146+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 >> 172.21.15.119:0/1249328948 conn(0x28de7b0 msgr2=0x2ba7590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 shutdown_connections 2024-09-13T16:30:38.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36df7a4740 1 -- 172.21.15.119:0/1249328948 wait complete. 2024-09-13T16:30:38.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36df7a4740 20 client.7584 trim_cache size 0 max 16384 2024-09-13T16:30:38.155 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:30:38.155 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:30:38.156 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "66bdb99f-5e09-4419-a3ef-fe2290416889", 2024-09-13T16:30:38.156 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:30:38.156 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:30:38.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36c0066760 msgr2=0x7f36c0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36c0066760 0x7f36c0068c20 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f36d0009d90 tx=0x7f36d0011000 comp rx=0 tx=0).stop 2024-09-13T16:30:38.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4145b70 msgr2=0x7f36d4148c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4145b70 0x7f36d4148c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f36cc001eb0 tx=0x7f36cc00c6e0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 shutdown_connections 2024-09-13T16:30:38.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36c0066760 0x7f36c0068c20 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d4145b70 0x7f36d4148c50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d40a99c0 0x7f36d4166ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 --2- 172.21.15.119:0/3396010477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d4156940 0x7f36d4146ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 >> 172.21.15.119:0/3396010477 conn(0x7f36d406a350 msgr2=0x7f36d40cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 shutdown_connections 2024-09-13T16:30:38.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.150+0000 7f36db291700 1 -- 172.21.15.119:0/3396010477 wait complete. 2024-09-13T16:30:38.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-09-13T16:30:38.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.290+0000 7f7ddb31d700 1 Processor -- start 2024-09-13T16:30:38.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.290+0000 7f7ddb31d700 1 -- start start 2024-09-13T16:30:38.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7dd40b3970 0x7f7dd40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd40cd4d0 con 0x7f7dd415dbe0 2024-09-13T16:30:38.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd415e130 con 0x7f7dd40add60 2024-09-13T16:30:38.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd4071be0 con 0x7f7dd40b3970 2024-09-13T16:30:38.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9b1a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7dd415dbe0 0x7f7dd40b3260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55716/0 (socket says 172.21.15.119:55716) 2024-09-13T16:30:38.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/2963795704 learned_addr learned my addr 172.21.15.119:0/2963795704 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:30:38.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddab1c700 1 --2- 172.21.15.119:0/2963795704 >> v2:172.21.15.145:3300/0 conn(0x7f7dd40b3970 0x7f7dd40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9319700 1 -- 172.21.15.119:0/2963795704 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054046699 0 0) 0x7f7dd40cd4d0 con 0x7f7dd415dbe0 2024-09-13T16:30:38.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9319700 1 -- 172.21.15.119:0/2963795704 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc00057e0 con 0x7f7dd415dbe0 2024-09-13T16:30:38.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddab1c700 1 -- 172.21.15.119:0/2963795704 >> v1:172.21.15.145:6789/0 conn(0x7f7dd40add60 legacy=0x7f7dd40cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddab1c700 1 -- 172.21.15.119:0/2963795704 >> v1:172.21.15.119:6789/0 conn(0x7f7dd415dbe0 legacy=0x7f7dd40b3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9319700 1 -- 172.21.15.119:0/2963795704 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273163013 0 0) 0x7f7dc00057e0 con 0x7f7dd415dbe0 2024-09-13T16:30:38.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddab1c700 1 -- 172.21.15.119:0/2963795704 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd40ccba0 con 0x7f7dd40b3970 2024-09-13T16:30:38.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddab1c700 1 --2- 172.21.15.119:0/2963795704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 0x7f7dd40ce0f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009850 tx=0x7f7dcc009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd7f49f0e9a4823b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9319700 1 -- 172.21.15.119:0/2963795704 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc011830 con 0x7f7dd40b3970 2024-09-13T16:30:38.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9319700 1 -- 172.21.15.119:0/2963795704 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc011990 con 0x7f7dd40b3970 2024-09-13T16:30:38.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7dd9319700 1 -- 172.21.15.119:0/2963795704 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc011d40 con 0x7f7dd40b3970 2024-09-13T16:30:38.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- 172.21.15.119:0/2963795704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 msgr2=0x7f7dd40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/2963795704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 0x7f7dd40ce0f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009850 tx=0x7f7dcc009b60 comp rx=0 tx=0).stop 2024-09-13T16:30:38.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- 172.21.15.119:0/2963795704 shutdown_connections 2024-09-13T16:30:38.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/2963795704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 0x7f7dd40ce0f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- 172.21.15.119:0/2963795704 >> 172.21.15.119:0/2963795704 conn(0x7f7dd40b0980 msgr2=0x7f7dd40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- 172.21.15.119:0/2963795704 shutdown_connections 2024-09-13T16:30:38.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.294+0000 7f7ddb31d700 1 -- 172.21.15.119:0/2963795704 wait complete. 2024-09-13T16:30:38.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 Processor -- start 2024-09-13T16:30:38.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 -- start start 2024-09-13T16:30:38.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dd40add60 0x7f7dd4150d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 0x7f7dd41537e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd415dbe0 0x7f7dd41ef070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd414f640 con 0x7f7dd40add60 2024-09-13T16:30:38.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd414f920 con 0x7f7dd40b3970 2024-09-13T16:30:38.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd414fc00 con 0x7f7dd415dbe0 2024-09-13T16:30:38.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dd40add60 0x7f7dd4150d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dd40add60 0x7f7dd4150d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57358/0 (socket says 172.21.15.119:57358) 2024-09-13T16:30:38.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 -- 172.21.15.119:0/1317901272 learned_addr learned my addr 172.21.15.119:0/1317901272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 0x7f7dd41537e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddab1c700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd415dbe0 0x7f7dd41ef070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 -- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd415dbe0 msgr2=0x7f7dd41ef070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd415dbe0 0x7f7dd41ef070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 -- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 msgr2=0x7f7dd41537e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 0x7f7dd41537e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 -- 172.21.15.119:0/1317901272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0094b0 con 0x7f7dd40add60 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dda31b700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dd40add60 0x7f7dd4150d90 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400c500 tx=0x7f7dc400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dcaffd700 1 -- 172.21.15.119:0/1317901272 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40052d0 con 0x7f7dd40add60 2024-09-13T16:30:38.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dcaffd700 1 -- 172.21.15.119:0/1317901272 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc4005430 con 0x7f7dd40add60 2024-09-13T16:30:38.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd414fee0 con 0x7f7dd40add60 2024-09-13T16:30:38.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7dcaffd700 1 -- 172.21.15.119:0/1317901272 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc40055a0 con 0x7f7dd40add60 2024-09-13T16:30:38.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.298+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd41512d0 con 0x7f7dd40add60 2024-09-13T16:30:38.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.302+0000 7f7dcaffd700 1 -- 172.21.15.119:0/1317901272 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7dc40038e0 con 0x7f7dd40add60 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.302+0000 7f7dcaffd700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7db8066730 0x7f7db8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.302+0000 7f7dcaffd700 1 -- 172.21.15.119:0/1317901272 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f7dc4083610 con 0x7f7dd40add60 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.302+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7db8066730 0x7f7db8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 Processor -- start 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 -- start start 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7db8066730 0x7f7db8068bf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4150880 tx=0x7f7dd0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3ed90 0x2af5810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b1d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a86010 con 0x2af4310 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a13f40 con 0x2b3ed90 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a66e30 con 0x2b3f170 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b1d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b1d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57364/0 (socket says 172.21.15.119:57364) 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/2226147882 learned_addr learned my addr 172.21.15.119:0/2226147882 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dda31b700 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3ed90 0x2af5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddab1c700 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 msgr2=0x2b3f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3ed90 msgr2=0x2af5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3ed90 0x2af5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/2226147882 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x28d2a40 con 0x2af4310 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b1d520 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000fbf0 tx=0x7f7dd0012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f979beece9e8498b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7db7fff700 1 -- 172.21.15.119:0/2226147882 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00146e0 con 0x2af4310 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7db7fff700 1 -- 172.21.15.119:0/2226147882 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd0014840 con 0x2af4310 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7db7fff700 1 -- 172.21.15.119:0/2226147882 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0014bb0 con 0x2af4310 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 -- 172.21.15.119:0/2226147882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 msgr2=0x2b1d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b1d520 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000fbf0 tx=0x7f7dd0012000 comp rx=0 tx=0).stop 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 -- 172.21.15.119:0/2226147882 shutdown_connections 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b1d520 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 --2- 172.21.15.119:0/2226147882 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3ed90 0x2af5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.306+0000 7f7ddf830740 1 -- 172.21.15.119:0/2226147882 >> 172.21.15.119:0/2226147882 conn(0x28875b0 msgr2=0x2af8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 -- 172.21.15.119:0/2226147882 shutdown_connections 2024-09-13T16:30:38.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 -- 172.21.15.119:0/2226147882 wait complete. 2024-09-13T16:30:38.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 -- start start 2024-09-13T16:30:38.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 Processor -- start 2024-09-13T16:30:38.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2af4310 0x2b1baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2b1f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a62150 con 0x2b3ed90 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a5feb0 con 0x2af4310 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a1d8e0 con 0x2b3f170 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddf830740 10 client.0 _ll_register_callbacks cb 0x29fc670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2b1f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2b1f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57370/0 (socket says 172.21.15.119:57370) 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/688788776 learned_addr learned my addr 172.21.15.119:0/688788776 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dda31b700 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2af4310 0x2b1baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7ddab1c700 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 msgr2=0x2b3d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2af4310 msgr2=0x2b1baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2af4310 0x2b1baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.310+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/688788776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd000f8a0 con 0x2b3ed90 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2b1f8d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0000f80 tx=0x7f7dd0005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca06ac51501ea221 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 1 -- 172.21.15.119:0/688788776 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd00208a0 con 0x2b3ed90 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 1 -- 172.21.15.119:0/688788776 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f7dd001c040 con 0x2b3ed90 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x28942c0 con 0x2b3ed90 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 1 -- 172.21.15.119:0/688788776 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dd0020a00 con 0x2b3ed90 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 10 client.7596 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x2a26290 con 0x2b3ed90 2024-09-13T16:30:38.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 1 -- 172.21.15.119:0/688788776 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x28942c0 con 0x2b3ed90 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 1 -- 172.21.15.119:0/688788776 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7dd0020b60 con 0x2b3ed90 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 10 client.7596 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 10 client.7596 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 20 client.7596 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b3da80 0x2b1deb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 4 client.7596 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x28de540 con 0x2b3da80 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7dda31b700 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b3da80 0x2b1deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 10 client.7596 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7dda31b700 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b3da80 0x2b1deb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eee86f32001a81a6 server_cookie=1ee604f323066a3c in_seq=0 out_seq=0 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 1 -- 172.21.15.119:0/688788776 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x28de540 con 0x2b3da80 2024-09-13T16:30:38.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 10 client.7596 handle_command_reply: tid=0 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 client.7596 shutdown 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 msgr2=0x2b1f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2b1f8d0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0000f80 tx=0x7f7dd0005c20 comp rx=0 tx=0).stop 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 shutdown_connections 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2b3da80 0x2b1deb0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3f170 0x2b3d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7db57fa700 0 client.7596 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2b1f8d0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 --2- 172.21.15.119:0/688788776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2af4310 0x2b1baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 >> 172.21.15.119:0/688788776 conn(0x28875b0 msgr2=0x2b1d4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 shutdown_connections 2024-09-13T16:30:38.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 1 -- 172.21.15.119:0/688788776 wait complete. 2024-09-13T16:30:38.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.314+0000 7f7ddf830740 20 client.7596 trim_cache size 0 max 16384 2024-09-13T16:30:38.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 Processor -- start 2024-09-13T16:30:38.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 -- start start 2024-09-13T16:30:38.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b19000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3ed90 0x2b51a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2b51f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x290bb80 con 0x2af4310 2024-09-13T16:30:38.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a30c00 con 0x2b3f170 2024-09-13T16:30:38.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a5d6c0 con 0x2b3ed90 2024-09-13T16:30:38.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b19000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b19000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57382/0 (socket says 172.21.15.119:57382) 2024-09-13T16:30:38.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dd9b1a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3ed90 0x2b51a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 -- 172.21.15.119:0/3178642243 learned_addr learned my addr 172.21.15.119:0/3178642243 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddab1c700 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2b51f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 -- 172.21.15.119:0/3178642243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3ed90 msgr2=0x2b51a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3ed90 0x2b51a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 -- 172.21.15.119:0/3178642243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 msgr2=0x2b51f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2b51f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 -- 172.21.15.119:0/3178642243 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2a0c4c0 con 0x2af4310 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dda31b700 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b19000 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40544c0 tx=0x7f7dc4083040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35e8c9ed349fa1c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dc8ff9700 1 -- 172.21.15.119:0/3178642243 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4058120 con 0x2af4310 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dc8ff9700 1 -- 172.21.15.119:0/3178642243 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc406d020 con 0x2af4310 2024-09-13T16:30:38.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7dc8ff9700 1 -- 172.21.15.119:0/3178642243 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4054530 con 0x2af4310 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 -- 172.21.15.119:0/3178642243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 msgr2=0x2b19000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b19000 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f7dc40544c0 tx=0x7f7dc4083040 comp rx=0 tx=0).stop 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 -- 172.21.15.119:0/3178642243 shutdown_connections 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2b51f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b3ed90 0x2b51a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3178642243 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2af4310 0x2b19000 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.338+0000 7f7ddf830740 1 -- 172.21.15.119:0/3178642243 >> 172.21.15.119:0/3178642243 conn(0x28875b0 msgr2=0x2b176f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- 172.21.15.119:0/3178642243 shutdown_connections 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- 172.21.15.119:0/3178642243 wait complete. 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- start start 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 Processor -- start 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2af4310 0x2ba7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2ba88e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2ba8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b578c0 con 0x2b3ed90 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2b19870 con 0x2b3f170 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2b57120 con 0x2af4310 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 10 client.0 _ll_register_callbacks cb 0x2a5e610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2ba88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dda31b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2af4310 0x2ba7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2ba88e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57390/0 (socket says 172.21.15.119:57390) 2024-09-13T16:30:38.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/3502816783 learned_addr learned my addr 172.21.15.119:0/3502816783 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:30:38.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddab1c700 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2ba8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2af4310 msgr2=0x2ba7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2af4310 0x2ba7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 msgr2=0x2ba8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2ba8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 -- 172.21.15.119:0/3502816783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd41512d0 con 0x2b3ed90 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dda31b700 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2af4310 0x2ba7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7dd9b1a700 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2ba88e0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000ff80 tx=0x7f7dd001e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2e2b7d8f9a09888 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:30:38.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7db57fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7db57fa700 1 -- 172.21.15.119:0/3502816783 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0034850 con 0x2b3ed90 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7db57fa700 1 -- 172.21.15.119:0/3502816783 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f7dd00206a0 con 0x2b3ed90 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7dd0014840 con 0x2b3ed90 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7db57fa700 1 -- 172.21.15.119:0/3502816783 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=48) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7dd0034a50 con 0x2b3ed90 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 10 client.7602 fetch_fsmap learned FSMap version 48 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.342+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=48,monmap=2+}) v3 -- 0x2a00790 con 0x2b3ed90 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7db57fa700 1 -- 172.21.15.119:0/3502816783 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd0014840 con 0x2b3ed90 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7db57fa700 1 -- 172.21.15.119:0/3502816783 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 48) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7dd0020860 con 0x2b3ed90 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 10 client.7602 fetch_fsmap finished waiting for FSMap version 48 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 10 client.7602 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 20 client.7602 populate_metadata read hostname 'smithi119' 2024-09-13T16:30:38.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba8160 0x2ba9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 4 client.7602 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x29032c0 con 0x2ba8160 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7dda31b700 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba8160 0x2ba9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7db57fa700 10 client.7602 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7dda31b700 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba8160 0x2ba9360 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2031b774bf77ef4 server_cookie=9e12fd1ae904687e in_seq=0 out_seq=0 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7db57fa700 1 -- 172.21.15.119:0/3502816783 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+266 (crc 0 0 0) 0x29032c0 con 0x2ba8160 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7db57fa700 10 client.7602 handle_command_reply: tid=0 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 client.7602 shutdown 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 msgr2=0x2ba88e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2ba88e0 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f7dd000ff80 tx=0x7f7dd001e000 comp rx=0 tx=0).stop 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 shutdown_connections 2024-09-13T16:30:38.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x2ba8160 0x2ba9360 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b3f170 0x2ba8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7db57fa700 0 client.7602 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b3ed90 0x2ba88e0 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 --2- 172.21.15.119:0/3502816783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2af4310 0x2ba7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 >> 172.21.15.119:0/3502816783 conn(0x28875b0 msgr2=0x2b3d010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 shutdown_connections 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 1 -- 172.21.15.119:0/3502816783 wait complete. 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddf830740 20 client.7602 trim_cache size 0 max 16384 2024-09-13T16:30:38.353 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: "66bdb99f-5e09-4419-a3ef-fe2290416889": { 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/subdir", 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "66bdb99f-5e09-4419-a3ef-fe2290416889", 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,repair" 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:30:38.354 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:30:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7db8066730 msgr2=0x7f7db8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7db8066730 0x7f7db8068bf0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f7dd4150880 tx=0x7f7dd0009040 comp rx=0 tx=0).stop 2024-09-13T16:30:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dd40add60 msgr2=0x7f7dd4150d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:30:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dd40add60 0x7f7dd4150d90 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f7dc400c500 tx=0x7f7dc400c810 comp rx=0 tx=0).stop 2024-09-13T16:30:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 shutdown_connections 2024-09-13T16:30:38.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7db8066730 0x7f7db8068bf0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dd415dbe0 0x7f7dd41ef070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dd40b3970 0x7f7dd41537e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 --2- 172.21.15.119:0/1317901272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dd40add60 0x7f7dd4150d90 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:30:38.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 >> 172.21.15.119:0/1317901272 conn(0x7f7dd40b0980 msgr2=0x7f7dd40b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:30:38.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 shutdown_connections 2024-09-13T16:30:38.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:30:38.346+0000 7f7ddb31d700 1 -- 172.21.15.119:0/1317901272 wait complete. 2024-09-13T16:30:38.367 INFO:tasks.cephfs.filesystem:scrub status for tag:66bdb99f-5e09-4419-a3ef-fe2290416889 - {'path': '/subdir', 'tag': '66bdb99f-5e09-4419-a3ef-fe2290416889', 'options': 'recursive,repair'} 2024-09-13T16:30:57.508 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:30:57.510 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:31:08.367 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.5:0 scrub status 2024-09-13T16:31:08.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.489+0000 7ff517519700 1 Processor -- start 2024-09-13T16:31:08.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.489+0000 7ff517519700 1 -- start start 2024-09-13T16:31:08.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.489+0000 7ff517519700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff51006ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.489+0000 7ff517519700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff51006be20 0x7ff5100d1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.489+0000 7ff517519700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5100ce880 con 0x7ff51006b2d0 2024-09-13T16:31:08.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.489+0000 7ff517519700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5100cee40 con 0x7ff51006be20 2024-09-13T16:31:08.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.489+0000 7ff517519700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100d1220 con 0x7ff51015cfb0 2024-09-13T16:31:08.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff51006be20 0x7ff5100d1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff51006be20 0x7ff5100d1640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47040/0 (socket says 172.21.15.119:47040) 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 -- 172.21.15.119:0/1644496962 learned_addr learned my addr 172.21.15.119:0/1644496962 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff5152b5700 1 --2- 172.21.15.119:0/1644496962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff51006ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 -- 172.21.15.119:0/1644496962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 msgr2=0x7ff51006ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 --2- 172.21.15.119:0/1644496962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff51006ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 -- 172.21.15.119:0/1644496962 >> v1:172.21.15.145:6789/0 conn(0x7ff51006b2d0 legacy=0x7ff51006b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 -- 172.21.15.119:0/1644496962 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5100ceb60 con 0x7ff51006be20 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff515ab6700 1 --2- 172.21.15.119:0/1644496962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006be20 0x7ff5100d1640 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff508009e50 tx=0x7ff50800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fd7acd311c891f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff507fff700 1 -- 172.21.15.119:0/1644496962 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800e070 con 0x7ff51006be20 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff507fff700 1 -- 172.21.15.119:0/1644496962 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff50800b9c0 con 0x7ff51006be20 2024-09-13T16:31:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff507fff700 1 -- 172.21.15.119:0/1644496962 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50800bd70 con 0x7ff51006be20 2024-09-13T16:31:08.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 -- 172.21.15.119:0/1644496962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006be20 msgr2=0x7ff5100d1640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 --2- 172.21.15.119:0/1644496962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006be20 0x7ff5100d1640 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7ff508009e50 tx=0x7ff50800b330 comp rx=0 tx=0).stop 2024-09-13T16:31:08.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 -- 172.21.15.119:0/1644496962 shutdown_connections 2024-09-13T16:31:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 --2- 172.21.15.119:0/1644496962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006be20 0x7ff5100d1640 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 --2- 172.21.15.119:0/1644496962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff51006ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 -- 172.21.15.119:0/1644496962 >> 172.21.15.119:0/1644496962 conn(0x7ff5100d2da0 msgr2=0x7ff5100d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 -- 172.21.15.119:0/1644496962 shutdown_connections 2024-09-13T16:31:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.493+0000 7ff517519700 1 -- 172.21.15.119:0/1644496962 wait complete. 2024-09-13T16:31:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 Processor -- start 2024-09-13T16:31:08.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 -- start start 2024-09-13T16:31:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006b2d0 0x7ff5100b7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006be20 0x7ff5100b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff5100c8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100b7790 con 0x7ff51006b2d0 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100b8a80 con 0x7ff51006be20 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5100b79e0 con 0x7ff51015cfb0 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006b2d0 0x7ff5100b7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006b2d0 0x7ff5100b7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47052/0 (socket says 172.21.15.119:47052) 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 -- 172.21.15.119:0/1141443532 learned_addr learned my addr 172.21.15.119:0/1141443532 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff514ab4700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006be20 0x7ff5100b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff515ab6700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff5100c8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 -- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 msgr2=0x7ff5100c8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff5100c8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 -- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006be20 msgr2=0x7ff5100b1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006be20 0x7ff5100b1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 -- 172.21.15.119:0/1141443532 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff508009b30 con 0x7ff51006b2d0 2024-09-13T16:31:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff5152b5700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006b2d0 0x7ff5100b7250 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff50000d800 tx=0x7ff50000dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff505ffb700 1 -- 172.21.15.119:0/1141443532 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5000125f0 con 0x7ff51006b2d0 2024-09-13T16:31:08.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff505ffb700 1 -- 172.21.15.119:0/1141443532 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff500012750 con 0x7ff51006b2d0 2024-09-13T16:31:08.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5100b7cc0 con 0x7ff51006b2d0 2024-09-13T16:31:08.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff505ffb700 1 -- 172.21.15.119:0/1141443532 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500012ac0 con 0x7ff51006b2d0 2024-09-13T16:31:08.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.497+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5100b8280 con 0x7ff51006b2d0 2024-09-13T16:31:08.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.501+0000 7ff505ffb700 1 -- 172.21.15.119:0/1141443532 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff500012c20 con 0x7ff51006b2d0 2024-09-13T16:31:08.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.501+0000 7ff505ffb700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4f4066730 0x7ff4f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.501+0000 7ff505ffb700 1 -- 172.21.15.119:0/1141443532 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff500084870 con 0x7ff51006b2d0 2024-09-13T16:31:08.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.501+0000 7ff514ab4700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4f4066730 0x7ff4f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.501+0000 7ff514ab4700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4f4066730 0x7ff4f4068bf0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff50c000f60 tx=0x7ff50c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 Processor -- start 2024-09-13T16:31:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 -- start start 2024-09-13T16:31:08.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a1f950 0x19fe240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a1fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x19d35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1967010 con 0x1a1f950 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x18f4f40 con 0x1a20270 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1947e30 con 0x19fdd00 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a1f950 0x19fe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a1f950 0x19fe240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47058/0 (socket says 172.21.15.119:47058) 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 -- 172.21.15.119:0/2827519888 learned_addr learned my addr 172.21.15.119:0/2827519888 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff514ab4700 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a1fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff515ab6700 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x19d35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 -- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 msgr2=0x1a1fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a1fd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 -- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 msgr2=0x19d35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x19d35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 -- 172.21.15.119:0/2827519888 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x17b3a40 con 0x1a1f950 2024-09-13T16:31:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff514ab4700 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a1fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff5152b5700 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a1f950 0x19fe240 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff50005d4a0 tx=0x7ff500012f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72504e12232b84b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff4faffd700 1 -- 172.21.15.119:0/2827519888 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50006d030 con 0x1a1f950 2024-09-13T16:31:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff4faffd700 1 -- 172.21.15.119:0/2827519888 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff500059170 con 0x1a1f950 2024-09-13T16:31:08.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.505+0000 7ff4faffd700 1 -- 172.21.15.119:0/2827519888 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff500055550 con 0x1a1f950 2024-09-13T16:31:08.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- 172.21.15.119:0/2827519888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a1f950 msgr2=0x19fe240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a1f950 0x19fe240 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7ff50005d4a0 tx=0x7ff500012f50 comp rx=0 tx=0).stop 2024-09-13T16:31:08.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- 172.21.15.119:0/2827519888 shutdown_connections 2024-09-13T16:31:08.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x19d35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a1fd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/2827519888 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a1f950 0x19fe240 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- 172.21.15.119:0/2827519888 >> 172.21.15.119:0/2827519888 conn(0x17685b0 msgr2=0x19d99b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- 172.21.15.119:0/2827519888 shutdown_connections 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- 172.21.15.119:0/2827519888 wait complete. 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- start start 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 Processor -- start 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a006c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a1f950 0x19fd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1943150 con 0x1a20270 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1940eb0 con 0x1a1f950 2024-09-13T16:31:08.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x18fe8e0 con 0x19fdd00 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 10 client.0 _ll_register_callbacks cb 0x18dd670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47068/0 (socket says 172.21.15.119:47068) 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 -- 172.21.15.119:0/3689996032 learned_addr learned my addr 172.21.15.119:0/3689996032 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff514ab4700 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a1f950 0x19fd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff5152b5700 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 -- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 msgr2=0x1a006c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a006c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 -- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a1f950 msgr2=0x19fd150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a1f950 0x19fd150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 -- 172.21.15.119:0/3689996032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5100b8280 con 0x1a20270 2024-09-13T16:31:08.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff515ab6700 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fd690 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ff50800b730 tx=0x7ff508004b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d9c6ff4fc841207 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff4effff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:08.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff4effff700 1 -- 172.21.15.119:0/3689996032 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff508004450 con 0x1a20270 2024-09-13T16:31:08.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff4effff700 1 -- 172.21.15.119:0/3689996032 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff5080045b0 con 0x1a20270 2024-09-13T16:31:08.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x17752c0 con 0x1a20270 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff4effff700 1 -- 172.21.15.119:0/3689996032 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff508004770 con 0x1a20270 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 10 client.7614 fetch_fsmap learned FSMap version 49 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x1907290 con 0x1a20270 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.509+0000 7ff4effff700 1 -- 172.21.15.119:0/3689996032 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x17752c0 con 0x1a20270 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff4effff700 1 -- 172.21.15.119:0/3689996032 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff50800b050 con 0x1a20270 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 10 client.7614 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 10 client.7614 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 20 client.7614 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:08.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19ff190 0x19ff5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 4 client.7614 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:31:08.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x17bf540 con 0x19ff190 2024-09-13T16:31:08.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff5152b5700 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19ff190 0x19ff5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff5152b5700 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19ff190 0x19ff5a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f18fb9165f45cdec server_cookie=d8679f471478cdfd in_seq=0 out_seq=0 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff4effff700 10 client.7614 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff4effff700 1 -- 172.21.15.119:0/3689996032 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x17bf540 con 0x19ff190 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff4effff700 10 client.7614 handle_command_reply: tid=0 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 client.7614 shutdown 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 msgr2=0x19fd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fd690 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7ff50800b730 tx=0x7ff508004b60 comp rx=0 tx=0).stop 2024-09-13T16:31:08.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 shutdown_connections 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19ff190 0x19ff5a0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fd690 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a1f950 0x19fd150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/3689996032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x19fdd00 0x1a006c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff4effff700 0 client.7614 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 >> 172.21.15.119:0/3689996032 conn(0x17685b0 msgr2=0x19d2db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 shutdown_connections 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 1 -- 172.21.15.119:0/3689996032 wait complete. 2024-09-13T16:31:08.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.513+0000 7ff51a7ca740 20 client.7614 trim_cache size 0 max 16384 2024-09-13T16:31:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 Processor -- start 2024-09-13T16:31:08.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- start start 2024-09-13T16:31:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19fdd00 0x1a37960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x19fc710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fcc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x17ecb80 con 0x1a20270 2024-09-13T16:31:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x18e5bd0 con 0x19fdd00 2024-09-13T16:31:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1911e80 con 0x1a1f950 2024-09-13T16:31:08.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fcc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fcc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47078/0 (socket says 172.21.15.119:47078) 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 -- 172.21.15.119:0/365215598 learned_addr learned my addr 172.21.15.119:0/365215598 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff5152b5700 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19fdd00 0x1a37960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff514ab4700 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x19fc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 -- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 msgr2=0x19fc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x19fc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 -- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19fdd00 msgr2=0x1a37960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19fdd00 0x1a37960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 -- 172.21.15.119:0/365215598 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x18ed4c0 con 0x1a20270 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff5152b5700 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19fdd00 0x1a37960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:08.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff515ab6700 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fcc50 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff508007120 tx=0x7ff508020070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce582f6cc5c59f02 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff4fb7fe700 1 -- 172.21.15.119:0/365215598 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50801eb10 con 0x1a20270 2024-09-13T16:31:08.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff4fb7fe700 1 -- 172.21.15.119:0/365215598 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff508022070 con 0x1a20270 2024-09-13T16:31:08.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff4fb7fe700 1 -- 172.21.15.119:0/365215598 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50801e430 con 0x1a20270 2024-09-13T16:31:08.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- 172.21.15.119:0/365215598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 msgr2=0x19fcc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fcc50 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7ff508007120 tx=0x7ff508020070 comp rx=0 tx=0).stop 2024-09-13T16:31:08.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- 172.21.15.119:0/365215598 shutdown_connections 2024-09-13T16:31:08.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a20270 0x19fcc50 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x19fc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/365215598 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x19fdd00 0x1a37960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- 172.21.15.119:0/365215598 >> 172.21.15.119:0/365215598 conn(0x17685b0 msgr2=0x1a2f710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- 172.21.15.119:0/365215598 shutdown_connections 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.533+0000 7ff51a7ca740 1 -- 172.21.15.119:0/365215598 wait complete. 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 -- start start 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 Processor -- start 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19fdd00 0x1a80530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x1a2f710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x1a2fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a3eee0 con 0x19fdd00 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19d9960 con 0x1a20270 2024-09-13T16:31:08.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a36330 con 0x1a1f950 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 10 client.0 _ll_register_callbacks cb 0x19108f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19fdd00 0x1a80530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19fdd00 0x1a80530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47092/0 (socket says 172.21.15.119:47092) 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 -- 172.21.15.119:0/88881155 learned_addr learned my addr 172.21.15.119:0/88881155 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff515ab6700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x1a2fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff514ab4700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x1a2f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 -- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 msgr2=0x1a2f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x1a2f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 -- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 msgr2=0x1a2fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x1a2fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 -- 172.21.15.119:0/88881155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1907290 con 0x19fdd00 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff514ab4700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x1a2f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:08.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff5152b5700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19fdd00 0x1a80530 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ff500055200 tx=0x7ff500007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3776d5fbd7da8b09 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff4effff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff4effff700 1 -- 172.21.15.119:0/88881155 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff50005d9d0 con 0x19fdd00 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff4effff700 1 -- 172.21.15.119:0/88881155 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff500055550 con 0x19fdd00 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff500059170 con 0x19fdd00 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff4effff700 1 -- 172.21.15.119:0/88881155 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5000557b0 con 0x19fdd00 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 10 client.7620 fetch_fsmap learned FSMap version 49 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x18e1790 con 0x19fdd00 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff4effff700 1 -- 172.21.15.119:0/88881155 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff500059170 con 0x19fdd00 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff4effff700 1 -- 172.21.15.119:0/88881155 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ff500054a40 con 0x19fdd00 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 10 client.7620 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 10 client.7620 resolve_mds: resolved 5:0 to role '5:0' aka daemon mds.c 2024-09-13T16:31:08.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 20 client.7620 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19fce20 0x1a34970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 4 client.7620 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:31:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x17e42c0 con 0x19fce20 2024-09-13T16:31:08.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff514ab4700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19fce20 0x1a34970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff514ab4700 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19fce20 0x1a34970 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feacb4e7fe82da6e server_cookie=5855532132358d64 in_seq=0 out_seq=0 2024-09-13T16:31:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.537+0000 7ff4effff700 10 client.7620 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff4effff700 1 -- 172.21.15.119:0/88881155 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x17e42c0 con 0x19fce20 2024-09-13T16:31:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff4effff700 10 client.7620 handle_command_reply: tid=0 2024-09-13T16:31:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 client.7620 shutdown 2024-09-13T16:31:08.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19fdd00 msgr2=0x1a80530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19fdd00 0x1a80530 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7ff500055200 tx=0x7ff500007f90 comp rx=0 tx=0).stop 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 shutdown_connections 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x19fce20 0x1a34970 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff4effff700 0 client.7620 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a20270 0x1a2fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1a1f950 0x1a2f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 --2- 172.21.15.119:0/88881155 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x19fdd00 0x1a80530 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 >> 172.21.15.119:0/88881155 conn(0x17685b0 msgr2=0x19fc710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 shutdown_connections 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 1 -- 172.21.15.119:0/88881155 wait complete. 2024-09-13T16:31:08.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff51a7ca740 20 client.7620 trim_cache size 0 max 16384 2024-09-13T16:31:08.549 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:08.549 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:31:08.549 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:31:08.549 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4f4066730 msgr2=0x7ff4f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4f4066730 0x7ff4f4068bf0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff50c000f60 tx=0x7ff50c009040 comp rx=0 tx=0).stop 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006b2d0 msgr2=0x7ff5100b7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006b2d0 0x7ff5100b7250 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7ff50000d800 tx=0x7ff50000dbc0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 shutdown_connections 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff4f4066730 0x7ff4f4068bf0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff51015cfb0 0x7ff5100c8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff51006be20 0x7ff5100b1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 --2- 172.21.15.119:0/1141443532 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff51006b2d0 0x7ff5100b7250 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 >> 172.21.15.119:0/1141443532 conn(0x7ff5100d2da0 msgr2=0x7ff5100b2800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 shutdown_connections 2024-09-13T16:31:08.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.541+0000 7ff517519700 1 -- 172.21.15.119:0/1141443532 wait complete. 2024-09-13T16:31:08.562 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:31:08.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:08.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99197d2700 1 Processor -- start 2024-09-13T16:31:08.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99197d2700 1 -- start start 2024-09-13T16:31:08.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99197d2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f9914165450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99197d2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f991406cf60 con 0x7f99140cc060 2024-09-13T16:31:08.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99197d2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99140cc4d0 con 0x7f99140adb30 2024-09-13T16:31:08.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99197d2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99140b3390 con 0x7f99140ce040 2024-09-13T16:31:08.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99137fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f99140cc060 0x7f99140cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48950/0 (socket says 172.21.15.119:48950) 2024-09-13T16:31:08.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f99137fe700 1 -- 172.21.15.119:0/48004200 learned_addr learned my addr 172.21.15.119:0/48004200 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:08.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f9918fd1700 1 --2- 172.21.15.119:0/48004200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f9914165450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f9912ffd700 1 -- 172.21.15.119:0/48004200 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145751289 0 0) 0x7f991406cf60 con 0x7f99140cc060 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.677+0000 7f9912ffd700 1 -- 172.21.15.119:0/48004200 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99000035d0 con 0x7f99140cc060 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9912ffd700 1 -- 172.21.15.119:0/48004200 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95640754 0 0) 0x7f99140cc4d0 con 0x7f99140adb30 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9918fd1700 1 -- 172.21.15.119:0/48004200 >> v1:172.21.15.145:6789/0 conn(0x7f99140adb30 legacy=0x7f991406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9918fd1700 1 -- 172.21.15.119:0/48004200 >> v1:172.21.15.119:6789/0 conn(0x7f99140cc060 legacy=0x7f99140cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9918fd1700 1 -- 172.21.15.119:0/48004200 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f991406c350 con 0x7f99140ce040 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9918fd1700 1 --2- 172.21.15.119:0/48004200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f9914165450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9904009880 tx=0x7f9904009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aebee3e931d0dcc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9912ffd700 1 -- 172.21.15.119:0/48004200 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904011830 con 0x7f99140ce040 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9912ffd700 1 -- 172.21.15.119:0/48004200 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9904011990 con 0x7f99140ce040 2024-09-13T16:31:08.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f9912ffd700 1 -- 172.21.15.119:0/48004200 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9904011d20 con 0x7f99140ce040 2024-09-13T16:31:08.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 -- 172.21.15.119:0/48004200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 msgr2=0x7f9914165450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 --2- 172.21.15.119:0/48004200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f9914165450 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f9904009880 tx=0x7f9904009b90 comp rx=0 tx=0).stop 2024-09-13T16:31:08.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 -- 172.21.15.119:0/48004200 shutdown_connections 2024-09-13T16:31:08.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 --2- 172.21.15.119:0/48004200 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f9914165450 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 -- 172.21.15.119:0/48004200 >> 172.21.15.119:0/48004200 conn(0x7f991406a350 msgr2=0x7f991406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 -- 172.21.15.119:0/48004200 shutdown_connections 2024-09-13T16:31:08.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 -- 172.21.15.119:0/48004200 wait complete. 2024-09-13T16:31:08.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 Processor -- start 2024-09-13T16:31:08.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.681+0000 7f99197d2700 1 -- start start 2024-09-13T16:31:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99140adb30 0x7f991413e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99140cc060 0x7f99141f11d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f99141f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99141f1ad0 con 0x7f99140adb30 2024-09-13T16:31:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99141f1db0 con 0x7f99140cc060 2024-09-13T16:31:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99141f2090 con 0x7f99140ce040 2024-09-13T16:31:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99140adb30 0x7f991413e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99140adb30 0x7f991413e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47108/0 (socket says 172.21.15.119:47108) 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 -- 172.21.15.119:0/2688884668 learned_addr learned my addr 172.21.15.119:0/2688884668 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99137fe700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99140cc060 0x7f99141f11d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9918fd1700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f99141f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 -- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 msgr2=0x7f99141f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f99141f2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 -- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99140cc060 msgr2=0x7f99141f11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99140cc060 0x7f99141f11d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 -- 172.21.15.119:0/2688884668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99040094b0 con 0x7f99140adb30 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99137fe700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99140cc060 0x7f99141f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9913fff700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99140adb30 0x7f991413e960 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f990c00cc50 tx=0x7f990c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9910ff9700 1 -- 172.21.15.119:0/2688884668 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c003e50 con 0x7f99140adb30 2024-09-13T16:31:08.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9910ff9700 1 -- 172.21.15.119:0/2688884668 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f990c00ce50 con 0x7f99140adb30 2024-09-13T16:31:08.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99141f2370 con 0x7f99140adb30 2024-09-13T16:31:08.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f9910ff9700 1 -- 172.21.15.119:0/2688884668 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f990c00ac10 con 0x7f99140adb30 2024-09-13T16:31:08.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99141f1710 con 0x7f99140adb30 2024-09-13T16:31:08.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.685+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99141f2890 con 0x7f99140adb30 2024-09-13T16:31:08.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.689+0000 7f9910ff9700 1 -- 172.21.15.119:0/2688884668 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f990c005070 con 0x7f99140adb30 2024-09-13T16:31:08.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.689+0000 7f9910ff9700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98f4066730 0x7f98f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:08.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.689+0000 7f9910ff9700 1 -- 172.21.15.119:0/2688884668 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f990c083f50 con 0x7f99140adb30 2024-09-13T16:31:08.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.689+0000 7f99137fe700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98f4066730 0x7f98f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:08.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.689+0000 7f99137fe700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98f4066730 0x7f98f4068bf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f991415ec80 tx=0x7f98fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:08.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.693+0000 7f9910ff9700 1 -- 172.21.15.119:0/2688884668 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f99141f2890 con 0x7f99140adb30 2024-09-13T16:31:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.897+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99141f2890 con 0x7f99140adb30 2024-09-13T16:31:08.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.897+0000 7f9910ff9700 1 -- 172.21.15.119:0/2688884668 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f990c017070 con 0x7f99140adb30 2024-09-13T16:31:08.903 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:08.903 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:30:40.052433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:31:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98f4066730 msgr2=0x7f98f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98f4066730 0x7f98f4068bf0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f991415ec80 tx=0x7f98fc009040 comp rx=0 tx=0).stop 2024-09-13T16:31:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99140adb30 msgr2=0x7f991413e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99140adb30 0x7f991413e960 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7f990c00cc50 tx=0x7f990c00a340 comp rx=0 tx=0).stop 2024-09-13T16:31:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 shutdown_connections 2024-09-13T16:31:08.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98f4066730 0x7f98f4068bf0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99140ce040 0x7f99141f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99140cc060 0x7f99141f11d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 --2- 172.21.15.119:0/2688884668 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99140adb30 0x7f991413e960 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 >> 172.21.15.119:0/2688884668 conn(0x7f991406a350 msgr2=0x7f9914066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 shutdown_connections 2024-09-13T16:31:08.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:08.901+0000 7f99197d2700 1 -- 172.21.15.119:0/2688884668 wait complete. 2024-09-13T16:31:08.907 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 49 2024-09-13T16:31:08.917 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-09-13T16:31:09.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7dbcd700 1 Processor -- start 2024-09-13T16:31:09.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7dbcd700 1 -- start start 2024-09-13T16:31:09.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7dbcd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d780add60 0x7f2d780ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7dbcd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2d780b3260 0x7f2d780b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7dbcd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d780cd4d0 con 0x7f2d7815dbe0 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7dbcd700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d78071be0 con 0x7f2d780b3260 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7dbcd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d78159af0 con 0x7f2d780add60 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7d3cc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2d780b3260 0x7f2d780b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7d3cc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2d780b3260 0x7f2d780b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47118/0 (socket says 172.21.15.119:47118) 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/1916637813 learned_addr learned my addr 172.21.15.119:0/1916637813 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/1916637813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d780add60 0x7f2d780ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/1916637813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d780add60 msgr2=0x7f2d780ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/1916637813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d780add60 0x7f2d780ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/1916637813 >> v1:172.21.15.145:6789/0 conn(0x7f2d7815dbe0 legacy=0x7f2d780cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.025+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/1916637813 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7815e130 con 0x7f2d780b3260 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/1916637813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780b3260 0x7f2d780b36f0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009e50 tx=0x7f2d7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d2e1d38484a247b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d777fe700 1 -- 172.21.15.119:0/1916637813 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7000e070 con 0x7f2d780b3260 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d777fe700 1 -- 172.21.15.119:0/1916637813 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d70003a40 con 0x7f2d780b3260 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d777fe700 1 -- 172.21.15.119:0/1916637813 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70003df0 con 0x7f2d780b3260 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/1916637813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780b3260 msgr2=0x7f2d780b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/1916637813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780b3260 0x7f2d780b36f0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f2d70009e50 tx=0x7f2d7000b330 comp rx=0 tx=0).stop 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/1916637813 shutdown_connections 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/1916637813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780b3260 0x7f2d780b36f0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/1916637813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d780add60 0x7f2d780ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/1916637813 >> 172.21.15.119:0/1916637813 conn(0x7f2d780b0980 msgr2=0x7f2d780b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/1916637813 shutdown_connections 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/1916637813 wait complete. 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 Processor -- start 2024-09-13T16:31:09.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.029+0000 7f2d7dbcd700 1 -- start start 2024-09-13T16:31:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780add60 0x7f2d7814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d780b3260 0x7f2d7815bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d7815dbe0 0x7f2d78148110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d78147630 con 0x7f2d780add60 2024-09-13T16:31:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d78150050 con 0x7f2d780b3260 2024-09-13T16:31:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d78146d90 con 0x7f2d7815dbe0 2024-09-13T16:31:09.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7cbcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780add60 0x7f2d7814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7d3cc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d7815dbe0 0x7f2d78148110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7cbcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780add60 0x7f2d7814fb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47122/0 (socket says 172.21.15.119:47122) 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/955677974 learned_addr learned my addr 172.21.15.119:0/955677974 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d77fff700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d780b3260 0x7f2d7815bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d780b3260 msgr2=0x7f2d7815bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d780b3260 0x7f2d7815bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/955677974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780add60 msgr2=0x7f2d7814fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780add60 0x7f2d7814fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/955677974 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d70009b30 con 0x7f2d7815dbe0 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d7815dbe0 0x7f2d78148110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2d70014000 tx=0x7f2d70004af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d757fa700 1 -- 172.21.15.119:0/955677974 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70004440 con 0x7f2d7815dbe0 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d757fa700 1 -- 172.21.15.119:0/955677974 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d700045c0 con 0x7f2d7815dbe0 2024-09-13T16:31:09.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d78147070 con 0x7f2d7815dbe0 2024-09-13T16:31:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d757fa700 1 -- 172.21.15.119:0/955677974 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70004970 con 0x7f2d7815dbe0 2024-09-13T16:31:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.033+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d781503a0 con 0x7f2d7815dbe0 2024-09-13T16:31:09.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.037+0000 7f2d757fa700 1 -- 172.21.15.119:0/955677974 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2d70012430 con 0x7f2d7815dbe0 2024-09-13T16:31:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.037+0000 7f2d757fa700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d64066730 0x7f2d64068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.037+0000 7f2d757fa700 1 -- 172.21.15.119:0/955677974 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2d700855c0 con 0x7f2d7815dbe0 2024-09-13T16:31:09.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.037+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d64066730 0x7f2d64068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.037+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d64066730 0x7f2d64068bf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2d78147b40 tx=0x7f2d6c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 Processor -- start 2024-09-13T16:31:09.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 -- start start 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f48410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f44cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48d30 0x1f45210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e70810 con 0x1f48d30 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e6dfa0 con 0x1f48950 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e69890 con 0x1e91520 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48d30 0x1f45210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48d30 0x1f45210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47126/0 (socket says 172.21.15.119:47126) 2024-09-13T16:31:09.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/3709802373 learned_addr learned my addr 172.21.15.119:0/3709802373 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f48410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d77fff700 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f44cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 msgr2=0x1f48410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f48410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 msgr2=0x1f44cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f44cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 -- 172.21.15.119:0/3709802373 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1cdbba0 con 0x1f48d30 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d77fff700 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f44cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48d30 0x1f45210 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2d7005e300 tx=0x7f2d70055e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a0547593909c8b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d5e7fc700 1 -- 172.21.15.119:0/3709802373 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d70056ba0 con 0x1f48d30 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d5e7fc700 1 -- 172.21.15.119:0/3709802373 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d70056440 con 0x1f48d30 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.041+0000 7f2d5e7fc700 1 -- 172.21.15.119:0/3709802373 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7005a270 con 0x1f48d30 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3709802373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48d30 msgr2=0x1f45210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48d30 0x1f45210 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f2d7005e300 tx=0x7f2d70055e20 comp rx=0 tx=0).stop 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3709802373 shutdown_connections 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48d30 0x1f45210 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f44cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3709802373 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f48410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3709802373 >> 172.21.15.119:0/3709802373 conn(0x1c90710 msgr2=0x1f2a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3709802373 shutdown_connections 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3709802373 wait complete. 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- start start 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 Processor -- start 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f26bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48d30 0x1f27100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e2cd80 con 0x1e91520 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e2ca20 con 0x1f48950 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d5c770 con 0x1f48d30 2024-09-13T16:31:09.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 10 client.0 _ll_register_callbacks cb 0x1e62f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47134/0 (socket says 172.21.15.119:47134) 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/3762583174 learned_addr learned my addr 172.21.15.119:0/3762583174 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48d30 0x1f27100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d77fff700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f26bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48d30 msgr2=0x1f27100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48d30 0x1f27100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 msgr2=0x1f26bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f26bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/3762583174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d781503a0 con 0x1e91520 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48d30 0x1f27100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d77fff700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f26bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21bf0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c016760 tx=0x7f2d6c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e36dbf96e6cf553 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d577fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d577fe700 1 -- 172.21.15.119:0/3762583174 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c014640 con 0x1e91520 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d577fe700 1 -- 172.21.15.119:0/3762583174 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2d6c0147a0 con 0x1e91520 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1c9d420 con 0x1e91520 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d577fe700 1 -- 172.21.15.119:0/3762583174 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d6c014960 con 0x1e91520 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 10 client.7638 fetch_fsmap learned FSMap version 49 2024-09-13T16:31:09.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x1e2f910 con 0x1e91520 2024-09-13T16:31:09.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d577fe700 1 -- 172.21.15.119:0/3762583174 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1c9d420 con 0x1e91520 2024-09-13T16:31:09.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d577fe700 1 -- 172.21.15.119:0/3762583174 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2d6c014ac0 con 0x1e91520 2024-09-13T16:31:09.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 10 client.7638 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T16:31:09.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 10 client.7638 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:31:09.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 20 client.7638 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:09.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f01540 0x1f01920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 4 client.7638 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:31:09.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.045+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1ce76a0 con 0x1f01540 2024-09-13T16:31:09.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d77fff700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f01540 0x1f01920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d577fe700 10 client.7638 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:09.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d77fff700 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f01540 0x1f01920 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=346cbc114bac0aa2 server_cookie=354e92b69dca863b in_seq=0 out_seq=0 2024-09-13T16:31:09.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d577fe700 1 -- 172.21.15.119:0/3762583174 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1ce76a0 con 0x1f01540 2024-09-13T16:31:09.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d577fe700 10 client.7638 handle_command_reply: tid=0 2024-09-13T16:31:09.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 client.7638 shutdown 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 msgr2=0x1f21bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21bf0 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c016760 tx=0x7f2d6c004770 comp rx=0 tx=0).stop 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 shutdown_connections 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f01540 0x1f01920 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48d30 0x1f27100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48950 0x1f26bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 --2- 172.21.15.119:0/3762583174 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21bf0 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d577fe700 0 client.7638 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 >> 172.21.15.119:0/3762583174 conn(0x1c90710 msgr2=0x1f45310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 shutdown_connections 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 1 -- 172.21.15.119:0/3762583174 wait complete. 2024-09-13T16:31:09.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.049+0000 7f2d820e0740 20 client.7638 trim_cache size 0 max 16384 2024-09-13T16:31:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 Processor -- start 2024-09-13T16:31:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 -- start start 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48950 0x1f265e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1f26b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e2a660 con 0x1e91520 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e71aa0 con 0x1f48d30 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e6c3f0 con 0x1f48950 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d7cbcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d7cbcb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47150/0 (socket says 172.21.15.119:47150) 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/2319189435 learned_addr learned my addr 172.21.15.119:0/2319189435 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1f26b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.061+0000 7f2d77fff700 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48950 0x1f265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48950 msgr2=0x1f265e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48950 0x1f265e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 msgr2=0x1f26b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1f26b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d7cbcb700 1 -- 172.21.15.119:0/2319189435 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1e15b40 con 0x1e91520 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21d50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c010c80 tx=0x7f2d6c023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea2ca65b8a0cab4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d5effd700 1 -- 172.21.15.119:0/2319189435 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c007890 con 0x1e91520 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d5effd700 1 -- 172.21.15.119:0/2319189435 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d6c006fe0 con 0x1e91520 2024-09-13T16:31:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d5effd700 1 -- 172.21.15.119:0/2319189435 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6c014860 con 0x1e91520 2024-09-13T16:31:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 -- 172.21.15.119:0/2319189435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 msgr2=0x1f21d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21d50 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f2d6c010c80 tx=0x7f2d6c023000 comp rx=0 tx=0).stop 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 -- 172.21.15.119:0/2319189435 shutdown_connections 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1f26b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1f48950 0x1f265e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 --2- 172.21.15.119:0/2319189435 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1e91520 0x1f21d50 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 -- 172.21.15.119:0/2319189435 >> 172.21.15.119:0/2319189435 conn(0x1c90710 msgr2=0x1f509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 -- 172.21.15.119:0/2319189435 shutdown_connections 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 -- 172.21.15.119:0/2319189435 wait complete. 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 -- start start 2024-09-13T16:31:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.065+0000 7f2d820e0740 1 Processor -- start 2024-09-13T16:31:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f4c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48950 0x1fa8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1fa8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e70810 con 0x1f48950 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1e909a0 con 0x1f48d30 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1e68720 con 0x1e91520 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48950 0x1fa8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 10 client.0 _ll_register_callbacks cb 0x1e68460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48950 0x1fa8990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47154/0 (socket says 172.21.15.119:47154) 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 -- 172.21.15.119:0/1617659550 learned_addr learned my addr 172.21.15.119:0/1617659550 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f4c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1fa8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 -- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 msgr2=0x1f4c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f4c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 -- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 msgr2=0x1fa8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1fa8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 -- 172.21.15.119:0/1617659550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1e2f910 con 0x1f48950 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f4c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d7d3cc700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1fa8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d77fff700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48950 0x1fa8990 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x1f280d0 tx=0x7f2d68003d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf28c6763f5b2517 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d577fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d577fe700 1 -- 172.21.15.119:0/1617659550 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d6801e070 con 0x1f48950 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d577fe700 1 -- 172.21.15.119:0/1617659550 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2d680198d0 con 0x1f48950 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d70056440 con 0x1f48950 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d577fe700 1 -- 172.21.15.119:0/1617659550 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d68004040 con 0x1f48950 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 10 client.7644 fetch_fsmap learned FSMap version 49 2024-09-13T16:31:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x1e09c20 con 0x1f48950 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d577fe700 1 -- 172.21.15.119:0/1617659550 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d70056440 con 0x1f48950 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d577fe700 1 -- 172.21.15.119:0/1617659550 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 49) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2d680041a0 con 0x1f48950 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 10 client.7644 fetch_fsmap finished waiting for FSMap version 49 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 10 client.7644 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 20 client.7644 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f4b170 0x1f4b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 4 client.7644 mds_command: new command op to 7341 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 --> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x1d0c420 con 0x1f4b170 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.069+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f4b170 0x1f4b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d577fe700 10 client.7644 ms_handle_connect on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7cbcb700 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f4b170 0x1f4b550 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7feab90e168db526 server_cookie=93a193e25b6409f3 in_seq=0 out_seq=0 2024-09-13T16:31:09.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d577fe700 1 -- 172.21.15.119:0/1617659550 <== mds.0 v2:172.21.15.119:6834/2534803404 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x1d0c420 con 0x1f4b170 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d577fe700 10 client.7644 handle_command_reply: tid=0 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 client.7644 shutdown 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48950 msgr2=0x1fa8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48950 0x1fa8990 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x1f280d0 tx=0x7f2d68003d90 comp rx=0 tx=0).stop 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 shutdown_connections 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:6834/2534803404,v1:172.21.15.119:6835/2534803404] conn(0x1f4b170 0x1f4b550 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1f48d30 0x1fa8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1f48950 0x1fa8990 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 --2- 172.21.15.119:0/1617659550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1e91520 0x1f4c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 >> 172.21.15.119:0/1617659550 conn(0x1c90710 msgr2=0x1f509c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d577fe700 0 client.7644 ms_handle_reset on v2:172.21.15.119:6834/2534803404 2024-09-13T16:31:09.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 shutdown_connections 2024-09-13T16:31:09.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 1 -- 172.21.15.119:0/1617659550 wait complete. 2024-09-13T16:31:09.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d820e0740 20 client.7644 trim_cache size 0 max 16384 2024-09-13T16:31:09.082 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:09.082 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-09-13T16:31:09.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d64066730 msgr2=0x7f2d64068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d64066730 0x7f2d64068bf0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f2d78147b40 tx=0x7f2d6c008040 comp rx=0 tx=0).stop 2024-09-13T16:31:09.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d7815dbe0 msgr2=0x7f2d78148110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d7815dbe0 0x7f2d78148110 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2d70014000 tx=0x7f2d70004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 shutdown_connections 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d64066730 0x7f2d64068bf0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2d7815dbe0 0x7f2d78148110 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2d780b3260 0x7f2d7815bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 --2- 172.21.15.119:0/955677974 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2d780add60 0x7f2d7814fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 >> 172.21.15.119:0/955677974 conn(0x7f2d780b0980 msgr2=0x7f2d781540b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 shutdown_connections 2024-09-13T16:31:09.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.073+0000 7f2d7dbcd700 1 -- 172.21.15.119:0/955677974 wait complete. 2024-09-13T16:31:09.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:31:09.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915c15700 1 Processor -- start 2024-09-13T16:31:09.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915c15700 1 -- start start 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915c15700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe910159160 0x7fe91006ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915c15700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9100adde0 0x7fe91006b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915c15700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9100ae210 con 0x7fe9100ae940 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915c15700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe910159540 con 0x7fe9100adde0 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915c15700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9100ce9f0 con 0x7fe910159160 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9100adde0 0x7fe91006b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9100adde0 0x7fe91006b2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47168/0 (socket says 172.21.15.119:47168) 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 -- 172.21.15.119:0/1339854397 learned_addr learned my addr 172.21.15.119:0/1339854397 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe914c13700 1 --2- 172.21.15.119:0/1339854397 >> v2:172.21.15.145:3300/0 conn(0x7fe910159160 0x7fe91006ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 -- 172.21.15.119:0/1339854397 >> v2:172.21.15.145:3300/0 conn(0x7fe910159160 msgr2=0x7fe91006ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 --2- 172.21.15.119:0/1339854397 >> v2:172.21.15.145:3300/0 conn(0x7fe910159160 0x7fe91006ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 -- 172.21.15.119:0/1339854397 >> v1:172.21.15.145:6789/0 conn(0x7fe9100ae940 legacy=0x7fe9100aef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 -- 172.21.15.119:0/1339854397 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9100cecd0 con 0x7fe9100adde0 2024-09-13T16:31:09.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe915414700 1 --2- 172.21.15.119:0/1339854397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9100adde0 0x7fe91006b2a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe9100700c0 tx=0x7fe90800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7bf895a00aac39 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe90f7fe700 1 -- 172.21.15.119:0/1339854397 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90800e070 con 0x7fe9100adde0 2024-09-13T16:31:09.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe90f7fe700 1 -- 172.21.15.119:0/1339854397 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe90800b9c0 con 0x7fe9100adde0 2024-09-13T16:31:09.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.201+0000 7fe90f7fe700 1 -- 172.21.15.119:0/1339854397 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90800bd70 con 0x7fe9100adde0 2024-09-13T16:31:09.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 -- 172.21.15.119:0/1339854397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9100adde0 msgr2=0x7fe91006b2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 --2- 172.21.15.119:0/1339854397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9100adde0 0x7fe91006b2a0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7fe9100700c0 tx=0x7fe90800b330 comp rx=0 tx=0).stop 2024-09-13T16:31:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 -- 172.21.15.119:0/1339854397 shutdown_connections 2024-09-13T16:31:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 --2- 172.21.15.119:0/1339854397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9100adde0 0x7fe91006b2a0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 --2- 172.21.15.119:0/1339854397 >> v2:172.21.15.145:3300/0 conn(0x7fe910159160 0x7fe91006ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 -- 172.21.15.119:0/1339854397 >> 172.21.15.119:0/1339854397 conn(0x7fe9100717e0 msgr2=0x7fe910071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 -- 172.21.15.119:0/1339854397 shutdown_connections 2024-09-13T16:31:09.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 -- 172.21.15.119:0/1339854397 wait complete. 2024-09-13T16:31:09.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 Processor -- start 2024-09-13T16:31:09.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.205+0000 7fe915c15700 1 -- start start 2024-09-13T16:31:09.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9100adde0 0x7fe910155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9100ae940 0x7fe910153d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe910159160 0x7fe9101484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe910155c50 con 0x7fe910159160 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe910154280 con 0x7fe9100ae940 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe910155ed0 con 0x7fe9100adde0 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe910159160 0x7fe9101484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe910159160 0x7fe9101484a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47172/0 (socket says 172.21.15.119:47172) 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 -- 172.21.15.119:0/3257191053 learned_addr learned my addr 172.21.15.119:0/3257191053 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe914c13700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9100adde0 0x7fe910155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe90ffff700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9100ae940 0x7fe910153d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 -- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9100adde0 msgr2=0x7fe910155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9100adde0 0x7fe910155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 -- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9100ae940 msgr2=0x7fe910153d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9100ae940 0x7fe910153d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 -- 172.21.15.119:0/3257191053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe908009b30 con 0x7fe910159160 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915414700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe910159160 0x7fe9101484a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe90800b370 tx=0x7fe908004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe90d7fa700 1 -- 172.21.15.119:0/3257191053 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe908004900 con 0x7fe910159160 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe90d7fa700 1 -- 172.21.15.119:0/3257191053 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9080099b0 con 0x7fe910159160 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9101561b0 con 0x7fe910159160 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe90d7fa700 1 -- 172.21.15.119:0/3257191053 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe90801c680 con 0x7fe910159160 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.209+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe91014a7c0 con 0x7fe910159160 2024-09-13T16:31:09.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.213+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe91014a340 con 0x7fe910159160 2024-09-13T16:31:09.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.213+0000 7fe90d7fa700 1 -- 172.21.15.119:0/3257191053 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe90801c7e0 con 0x7fe910159160 2024-09-13T16:31:09.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.213+0000 7fe90d7fa700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe8f4066730 0x7fe8f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.213+0000 7fe90d7fa700 1 -- 172.21.15.119:0/3257191053 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe908085ad0 con 0x7fe910159160 2024-09-13T16:31:09.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.213+0000 7fe914c13700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe8f4066730 0x7fe8f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.213+0000 7fe914c13700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe8f4066730 0x7fe8f4068bf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe91009d1d0 tx=0x7fe904008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.221+0000 7fe90d7fa700 1 -- 172.21.15.119:0/3257191053 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe91014a340 con 0x7fe910159160 2024-09-13T16:31:09.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.433+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe91014a340 con 0x7fe910159160 2024-09-13T16:31:09.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.433+0000 7fe90d7fa700 1 -- 172.21.15.119:0/3257191053 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7fe91014a340 con 0x7fe910159160 2024-09-13T16:31:09.437 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:09.437 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T16:31:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe8f4066730 msgr2=0x7fe8f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe8f4066730 0x7fe8f4068bf0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe91009d1d0 tx=0x7fe904008040 comp rx=0 tx=0).stop 2024-09-13T16:31:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe910159160 msgr2=0x7fe9101484a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe910159160 0x7fe9101484a0 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7fe90800b370 tx=0x7fe908004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 shutdown_connections 2024-09-13T16:31:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe8f4066730 0x7fe8f4068bf0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe910159160 0x7fe9101484a0 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9100ae940 0x7fe910153d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 --2- 172.21.15.119:0/3257191053 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9100adde0 0x7fe910155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 >> 172.21.15.119:0/3257191053 conn(0x7fe9100717e0 msgr2=0x7fe91006a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 shutdown_connections 2024-09-13T16:31:09.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.437+0000 7fe915c15700 1 -- 172.21.15.119:0/3257191053 wait complete. 2024-09-13T16:31:09.452 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-13T16:31:09.452 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:31:09.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077359e700 1 Processor -- start 2024-09-13T16:31:09.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077359e700 1 -- start start 2024-09-13T16:31:09.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f077406dc40 0x7f07740b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f077406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077359e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07740b2280 con 0x7f07741594f0 2024-09-13T16:31:09.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07740cc890 con 0x7f077406dc40 2024-09-13T16:31:09.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740b1970 con 0x7f07740b3630 2024-09-13T16:31:09.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077259c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f07741594f0 0x7f077406d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48956/0 (socket says 172.21.15.119:48956) 2024-09-13T16:31:09.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077259c700 1 -- 172.21.15.119:0/2309099915 learned_addr learned my addr 172.21.15.119:0/2309099915 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:09.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f0772d9d700 1 --2- 172.21.15.119:0/2309099915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f077406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f0771d9b700 1 --2- 172.21.15.119:0/2309099915 >> v2:172.21.15.145:3300/0 conn(0x7f077406dc40 0x7f07740b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.557+0000 7f077159a700 1 -- 172.21.15.119:0/2309099915 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189197842 0 0) 0x7f07740b2280 con 0x7f07741594f0 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077159a700 1 -- 172.21.15.119:0/2309099915 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07600035d0 con 0x7f07741594f0 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f0772d9d700 1 -- 172.21.15.119:0/2309099915 >> v2:172.21.15.145:3300/0 conn(0x7f077406dc40 msgr2=0x7f07740b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f0772d9d700 1 --2- 172.21.15.119:0/2309099915 >> v2:172.21.15.145:3300/0 conn(0x7f077406dc40 0x7f07740b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f0772d9d700 1 -- 172.21.15.119:0/2309099915 >> v1:172.21.15.119:6789/0 conn(0x7f07741594f0 legacy=0x7f077406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f0772d9d700 1 -- 172.21.15.119:0/2309099915 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07740b1c30 con 0x7f07740b3630 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f0771d9b700 1 --2- 172.21.15.119:0/2309099915 >> v2:172.21.15.145:3300/0 conn(0x7f077406dc40 0x7f07740b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f0772d9d700 1 --2- 172.21.15.119:0/2309099915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f077406fc00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f076c009850 tx=0x7f076c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e821ad525570cff2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077159a700 1 -- 172.21.15.119:0/2309099915 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c011830 con 0x7f07740b3630 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077159a700 1 -- 172.21.15.119:0/2309099915 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c011990 con 0x7f07740b3630 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077159a700 1 -- 172.21.15.119:0/2309099915 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c011d40 con 0x7f07740b3630 2024-09-13T16:31:09.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 -- 172.21.15.119:0/2309099915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 msgr2=0x7f077406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 --2- 172.21.15.119:0/2309099915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f077406fc00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f076c009850 tx=0x7f076c009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 -- 172.21.15.119:0/2309099915 shutdown_connections 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 --2- 172.21.15.119:0/2309099915 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f077406fc00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 --2- 172.21.15.119:0/2309099915 >> v2:172.21.15.145:3300/0 conn(0x7f077406dc40 0x7f07740b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 -- 172.21.15.119:0/2309099915 >> 172.21.15.119:0/2309099915 conn(0x7f077406a350 msgr2=0x7f077406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 -- 172.21.15.119:0/2309099915 shutdown_connections 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 -- 172.21.15.119:0/2309099915 wait complete. 2024-09-13T16:31:09.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 Processor -- start 2024-09-13T16:31:09.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.561+0000 7f077359e700 1 -- start start 2024-09-13T16:31:09.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f077406dc40 0x7f07741f25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f0774152260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07741594f0 0x7f07740c9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07740a84b0 con 0x7f07741594f0 2024-09-13T16:31:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07741527a0 con 0x7f077406dc40 2024-09-13T16:31:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07740a8700 con 0x7f07740b3630 2024-09-13T16:31:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07741594f0 0x7f07740c9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07741594f0 0x7f07740c9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47186/0 (socket says 172.21.15.119:47186) 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0771d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f0774152260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 -- 172.21.15.119:0/212491306 learned_addr learned my addr 172.21.15.119:0/212491306 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077259c700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f077406dc40 0x7f07741f25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 -- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 msgr2=0x7f0774152260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f0774152260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 -- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f077406dc40 msgr2=0x7f07741f25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f077406dc40 0x7f07741f25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 -- 172.21.15.119:0/212491306 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0764009a70 con 0x7f07741594f0 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f0772d9d700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07741594f0 0x7f07740c9cd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f07740b1ab0 tx=0x7f076c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f076affd700 1 -- 172.21.15.119:0/212491306 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c004300 con 0x7f07741594f0 2024-09-13T16:31:09.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f076affd700 1 -- 172.21.15.119:0/212491306 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f076c01f070 con 0x7f07741594f0 2024-09-13T16:31:09.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c0094b0 con 0x7f07741594f0 2024-09-13T16:31:09.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f076affd700 1 -- 172.21.15.119:0/212491306 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f076c0036c0 con 0x7f07741594f0 2024-09-13T16:31:09.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07740a8cc0 con 0x7f07741594f0 2024-09-13T16:31:09.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.565+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07740669f0 con 0x7f07741594f0 2024-09-13T16:31:09.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.569+0000 7f076affd700 1 -- 172.21.15.119:0/212491306 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f076c021030 con 0x7f07741594f0 2024-09-13T16:31:09.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.569+0000 7f076affd700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0754066760 0x7f0754068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:09.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.569+0000 7f076affd700 1 -- 172.21.15.119:0/212491306 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f076c0854c0 con 0x7f07741594f0 2024-09-13T16:31:09.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.569+0000 7f077259c700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0754066760 0x7f0754068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:09.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.569+0000 7f077259c700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0754066760 0x7f0754068c20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f075c007fd0 tx=0x7f075c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:09.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.577+0000 7f076affd700 1 -- 172.21.15.119:0/212491306 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f076c052730 con 0x7f07741594f0 2024-09-13T16:31:09.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f07740b30f0 con 0x7f07741594f0 2024-09-13T16:31:09.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f076affd700 1 -- 172.21.15.119:0/212491306 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) v1 ==== 53+0+73 (secure 0 0 0) 0x7f076c055d50 con 0x7f07741594f0 2024-09-13T16:31:09.788 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:31:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0754066760 msgr2=0x7f0754068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0754066760 0x7f0754068c20 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f075c007fd0 tx=0x7f075c009040 comp rx=0 tx=0).stop 2024-09-13T16:31:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07741594f0 msgr2=0x7f07740c9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07741594f0 0x7f07740c9cd0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7f07740b1ab0 tx=0x7f076c004770 comp rx=0 tx=0).stop 2024-09-13T16:31:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 shutdown_connections 2024-09-13T16:31:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0754066760 0x7f0754068c20 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07741594f0 0x7f07740c9cd0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f07740b3630 0x7f0774152260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 --2- 172.21.15.119:0/212491306 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f077406dc40 0x7f07741f25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:09.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 >> 172.21.15.119:0/212491306 conn(0x7f077406a350 msgr2=0x7f077415b6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:09.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 shutdown_connections 2024-09-13T16:31:09.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:09.785+0000 7f077359e700 1 -- 172.21.15.119:0/212491306 wait complete. 2024-09-13T16:31:09.804 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:31:09.804 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:31:09.804 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:31:09.804 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:31:09.804 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:09.805 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:31:09.805 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:31:09.805 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:31:09.805 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:09.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:09.806 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:31:09.815 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:31:09.815 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:31:09.815 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:31:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:31:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.816 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:09.816 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:09.817 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:09.817 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:09.817 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:31:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:09.817 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:31:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42271sec preferred_lft 42271sec 2024-09-13T16:31:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:31:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.818 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:31:09.818 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:09.819 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:31:09.819 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.819 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:31:09.819 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:09.819 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:31:09.820 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.820 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:31:09.820 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.820 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:31:09.821 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:31:09.821 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:31:09.821 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.821 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:31:09.821 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:31:09.822 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:31:09.822 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:09.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:09.822 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:31:09.873 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:31:09.875 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:09.875 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:31:09.884 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:31:09.884 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:31:09.884 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:31:09.884 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:31:09.884 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:31:09.884 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:31:09.885 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:31:09.885 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:31:09.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:09.885 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:31:09.885 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:31:09.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:31:09.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:31:09.885 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:31:09.886 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:31:09.924 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:09.924 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:31:09.974 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:31:09.974 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:31:09.974 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:09.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:09.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:31:09.987 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:31:09.987 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:31:09.987 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:09.993 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:09.994 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:31:09.994 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:10.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:10.057 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:10.057 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:10.068 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:10.069 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:10.070 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:10.070 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:10.074 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:31:10.075 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:31:10.077 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:10.094 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:10.094 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:10.149 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:10.150 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:10.150 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:10.150 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:10.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:31:10.161+0000 7fc3a507a640 -1 init, newargv = 0x558fb84ee540 newargc=15 2024-09-13T16:31:10.166 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[45772]: starting ceph client 2024-09-13T16:31:10.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[45772]: starting fuse 2024-09-13T16:31:11.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:11.275 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:11.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:11.333 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:11.333 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:11.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:11.334 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:11.341 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:31:11.341 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:11.342 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:31:11.343 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:31:11.422 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.45772.asok 2024-09-13T16:31:11.607 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45772.asok 2024-09-13T16:31:11.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:11.607 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45772.asok status 2024-09-13T16:31:11.753 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:11.753 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:31:11.753 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:31:11.753 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:31:11.753 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:31:11.753 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "45772", 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 7662, 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 7662 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:31:11.754 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2898933815 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2898933815 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.7662 192.168.0.1:0/2898933815", 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/2898933815", 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:31:11.755 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 49, 2024-09-13T16:31:11.756 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 83, 2024-09-13T16:31:11.756 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:31:11.756 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:31:11.756 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:31:11.756 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:11.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:11.763 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:31:11.763 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:11.768 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:31:11.768 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:11.768 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:11.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:11.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:11.782 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec getfattr --only-values -n ceph.dir.files ./subdir) 2024-09-13T16:31:11.840 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:31:11.840 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata setomapval 10000000000.00000000 file_to_be_damaged_head deadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeefdeadbeef 2024-09-13T16:31:11.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 Processor -- start 2024-09-13T16:31:11.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 -- start start 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c530 0x55636e5a8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c910 0x55636e61ad80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 0x55636e61b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636e4f02f0 con 0x55636e55c530 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636e4ac8e0 con 0x55636e619c60 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.877+0000 7fa0a4009140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55636e4ac580 con 0x55636e55c910 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a1b3e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c530 0x55636e5a8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a133d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c910 0x55636e61ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a1b3e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c530 0x55636e5a8890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47206/0 (socket says 172.21.15.119:47206) 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a1b3e700 1 -- 172.21.15.119:0/515391477 learned_addr learned my addr 172.21.15.119:0/515391477 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a233f700 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 0x55636e61b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a133d700 1 -- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 msgr2=0x55636e61b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a133d700 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 0x55636e61b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a133d700 1 -- 172.21.15.119:0/515391477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c530 msgr2=0x55636e5a8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a133d700 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c530 0x55636e5a8890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a133d700 1 -- 172.21.15.119:0/515391477 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa09c005de0 con 0x55636e55c910 2024-09-13T16:31:11.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a133d700 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c910 0x55636e61ad80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa08c001db0 tx=0x7fa08c00b4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5629a810b90ca6e5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa09bfff700 1 -- 172.21.15.119:0/515391477 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00f3a0 con 0x55636e55c910 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa09bfff700 1 -- 172.21.15.119:0/515391477 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c00a040 con 0x55636e55c910 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa09bfff700 1 -- 172.21.15.119:0/515391477 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00f760 con 0x55636e55c910 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 -- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c910 msgr2=0x55636e61ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c910 0x55636e61ad80 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa08c001db0 tx=0x7fa08c00b4a0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 -- 172.21.15.119:0/515391477 shutdown_connections 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 0x55636e61b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c910 0x55636e61ad80 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 --2- 172.21.15.119:0/515391477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c530 0x55636e5a8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 -- 172.21.15.119:0/515391477 >> 172.21.15.119:0/515391477 conn(0x55636e55db50 msgr2=0x55636e633dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:11.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 -- 172.21.15.119:0/515391477 shutdown_connections 2024-09-13T16:31:11.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.881+0000 7fa0a4009140 1 -- 172.21.15.119:0/515391477 wait complete. 2024-09-13T16:31:11.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 Processor -- start 2024-09-13T16:31:11.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 -- start start 2024-09-13T16:31:11.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c530 0x55636e5a68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c910 0x55636e626d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 0x55636e6c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636e4828d0 con 0x55636e55c910 2024-09-13T16:31:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55636e623d90 con 0x55636e619c60 2024-09-13T16:31:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a4009140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55636e624030 con 0x55636e55c530 2024-09-13T16:31:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c910 0x55636e626d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c910 0x55636e626d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47212/0 (socket says 172.21.15.119:47212) 2024-09-13T16:31:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 -- 172.21.15.119:0/3559876980 learned_addr learned my addr 172.21.15.119:0/3559876980 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a1b3e700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c530 0x55636e5a68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a233f700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 0x55636e6c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 -- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c530 msgr2=0x55636e5a68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:11.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55636e55c530 0x55636e5a68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 -- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 msgr2=0x55636e6c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55636e619c60 0x55636e6c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.885+0000 7fa0a133d700 1 -- 172.21.15.119:0/3559876980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa08c0019b0 con 0x55636e55c910 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa0a133d700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c910 0x55636e626d10 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fa08c000fb0 tx=0x7fa08c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa09affd700 1 -- 172.21.15.119:0/3559876980 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c00bee0 con 0x55636e55c910 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa09affd700 1 -- 172.21.15.119:0/3559876980 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa08c01d630 con 0x55636e55c910 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55636e49ab00 con 0x55636e55c910 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa09affd700 1 -- 172.21.15.119:0/3559876980 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa08c01d9e0 con 0x55636e55c910 2024-09-13T16:31:11.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55636e48df40 con 0x55636e55c910 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa09affd700 1 -- 172.21.15.119:0/3559876980 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa08c01db40 con 0x55636e55c910 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa09affd700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa084065060 0x7fa084067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa09affd700 1 -- 172.21.15.119:0/3559876980 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fa08c084740 con 0x55636e55c910 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa0a1b3e700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa084065060 0x7fa084067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa0a4009140 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55636e623880 0x55636e61dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.889+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:1 11.10 11:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=115b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e83) v8 -- 0x55636e624cb0 con 0x55636e623880 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a233f700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55636e623880 0x55636e61dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a233f700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55636e623880 0x55636e61dad0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:11.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a1b3e700 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa084065060 0x7fa084067520 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa09c000c00 tx=0x7fa09c00f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a233f700 1 -- 172.21.15.119:0/3559876980 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(1 10000000000.00000000 [omap-set-vals] v83'12 uv12 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fa094002a10 con 0x55636e623880 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55636e623880 msgr2=0x55636e61dad0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55636e623880 0x55636e61dad0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa084065060 msgr2=0x7fa084067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa084065060 0x7fa084067520 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa09c000c00 tx=0x7fa09c00f000 comp rx=0 tx=0).stop 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c910 msgr2=0x55636e626d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 --2- 172.21.15.119:0/3559876980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55636e55c910 0x55636e626d10 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7fa08c000fb0 tx=0x7fa08c000ed0 comp rx=0 tx=0).stop 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a233f700 1 -- 172.21.15.119:0/3559876980 reap_dead start 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 shutdown_connections 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 >> 172.21.15.119:0/3559876980 conn(0x55636e55db50 msgr2=0x55636e5a8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 shutdown_connections 2024-09-13T16:31:11.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:11.893+0000 7fa0a4009140 1 -- 172.21.15.119:0/3559876980 wait complete. 2024-09-13T16:31:11.902 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:31:11.902 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:31:11.920 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:31:11.940 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:31:11.940 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:31:11.940 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:11.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:12.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.081+0000 7f8631dca700 1 Processor -- start 2024-09-13T16:31:12.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.081+0000 7f8631dca700 1 -- start start 2024-09-13T16:31:12.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f862c0d2a30 0x7f862c0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f862c0b3a20 0x7f862c0cd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862c0d3350 con 0x7f862c0cf1e0 2024-09-13T16:31:12.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c0cdf90 con 0x7f862c0d2a30 2024-09-13T16:31:12.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862c0ce270 con 0x7f862c0b3a20 2024-09-13T16:31:12.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86315c9700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f862c0cf1e0 0x7f862c0cf650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48960/0 (socket says 172.21.15.119:48960) 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86315c9700 1 -- 172.21.15.119:0/3273814230 learned_addr learned my addr 172.21.15.119:0/3273814230 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8630dc8700 1 --2- 172.21.15.119:0/3273814230 >> v2:172.21.15.119:3300/0 conn(0x7f862c0d2a30 0x7f862c0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8623fff700 1 --2- 172.21.15.119:0/3273814230 >> v2:172.21.15.145:3300/0 conn(0x7f862c0b3a20 0x7f862c0cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86237fe700 1 -- 172.21.15.119:0/3273814230 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409252214 0 0) 0x7f862c0d3350 con 0x7f862c0cf1e0 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86237fe700 1 -- 172.21.15.119:0/3273814230 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86140035d0 con 0x7f862c0cf1e0 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86237fe700 1 -- 172.21.15.119:0/3273814230 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3358860354 0 0) 0x7f86140035d0 con 0x7f862c0cf1e0 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8630dc8700 1 -- 172.21.15.119:0/3273814230 >> v2:172.21.15.145:3300/0 conn(0x7f862c0b3a20 msgr2=0x7f862c0cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8630dc8700 1 --2- 172.21.15.119:0/3273814230 >> v2:172.21.15.145:3300/0 conn(0x7f862c0b3a20 0x7f862c0cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8630dc8700 1 -- 172.21.15.119:0/3273814230 >> v1:172.21.15.119:6789/0 conn(0x7f862c0cf1e0 legacy=0x7f862c0cf650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8630dc8700 1 -- 172.21.15.119:0/3273814230 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862c0ce550 con 0x7f862c0d2a30 2024-09-13T16:31:12.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8630dc8700 1 --2- 172.21.15.119:0/3273814230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0d2a30 0x7f862c0d2e10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8628009850 tx=0x7f8628009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4262f5ba545a8075 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86237fe700 1 -- 172.21.15.119:0/3273814230 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86240019f0 con 0x7f862c0d2a30 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86237fe700 1 -- 172.21.15.119:0/3273814230 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8628004030 con 0x7f862c0d2a30 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f86237fe700 1 -- 172.21.15.119:0/3273814230 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86280043a0 con 0x7f862c0d2a30 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- 172.21.15.119:0/3273814230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0d2a30 msgr2=0x7f862c0d2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 --2- 172.21.15.119:0/3273814230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0d2a30 0x7f862c0d2e10 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f8628009850 tx=0x7f8628009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- 172.21.15.119:0/3273814230 shutdown_connections 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 --2- 172.21.15.119:0/3273814230 >> v2:172.21.15.145:3300/0 conn(0x7f862c0b3a20 0x7f862c0cd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 --2- 172.21.15.119:0/3273814230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0d2a30 0x7f862c0d2e10 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- 172.21.15.119:0/3273814230 >> 172.21.15.119:0/3273814230 conn(0x7f862c08c0f0 msgr2=0x7f862c08c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:12.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- 172.21.15.119:0/3273814230 shutdown_connections 2024-09-13T16:31:12.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.085+0000 7f8631dca700 1 -- 172.21.15.119:0/3273814230 wait complete. 2024-09-13T16:31:12.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.089+0000 7f8631dca700 1 Processor -- start 2024-09-13T16:31:12.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.089+0000 7f8631dca700 1 -- start start 2024-09-13T16:31:12.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c0b3a20 0x7f862c0ca620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0cf1e0 0x7f862c0b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c0d2a30 0x7f862c0b6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0b5c40 con 0x7f862c0cf1e0 2024-09-13T16:31:12.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0b58f0 con 0x7f862c0b3a20 2024-09-13T16:31:12.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862c0cab60 con 0x7f862c0d2a30 2024-09-13T16:31:12.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0cf1e0 0x7f862c0b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0cf1e0 0x7f862c0b0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:47224/0 (socket says 172.21.15.119:47224) 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 -- 172.21.15.119:0/2096265012 learned_addr learned my addr 172.21.15.119:0/2096265012 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8630dc8700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c0b3a20 0x7f862c0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f86315c9700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c0d2a30 0x7f862c0b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 -- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c0d2a30 msgr2=0x7f862c0b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c0d2a30 0x7f862c0b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 -- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c0b3a20 msgr2=0x7f862c0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c0b3a20 0x7f862c0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 -- 172.21.15.119:0/2096265012 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86280094b0 con 0x7f862c0cf1e0 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8623fff700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0cf1e0 0x7f862c0b0c40 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f861c009fd0 tx=0x7f861c00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f86217fa700 1 -- 172.21.15.119:0/2096265012 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c0095a0 con 0x7f862c0cf1e0 2024-09-13T16:31:12.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f86217fa700 1 -- 172.21.15.119:0/2096265012 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c013680 con 0x7f862c0cf1e0 2024-09-13T16:31:12.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862c0cae00 con 0x7f862c0cf1e0 2024-09-13T16:31:12.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f86217fa700 1 -- 172.21.15.119:0/2096265012 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f861c013a10 con 0x7f862c0cf1e0 2024-09-13T16:31:12.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862c0cb3c0 con 0x7f862c0cf1e0 2024-09-13T16:31:12.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.093+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862c0675c0 con 0x7f862c0cf1e0 2024-09-13T16:31:12.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.097+0000 7f86217fa700 1 -- 172.21.15.119:0/2096265012 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f861c013b70 con 0x7f862c0cf1e0 2024-09-13T16:31:12.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.097+0000 7f86217fa700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8618066730 0x7f8618068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.097+0000 7f86217fa700 1 -- 172.21.15.119:0/2096265012 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f861c084920 con 0x7f862c0cf1e0 2024-09-13T16:31:12.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.097+0000 7f8630dc8700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8618066730 0x7f8618068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.097+0000 7f8630dc8700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8618066730 0x7f8618068bf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f86280090b0 tx=0x7f8628008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.105+0000 7f86217fa700 1 -- 172.21.15.119:0/2096265012 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f861c051b60 con 0x7f862c0cf1e0 2024-09-13T16:31:12.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.277+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f862c15cdb0 con 0x7f862c0cf1e0 2024-09-13T16:31:12.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.281+0000 7f86217fa700 1 -- 172.21.15.119:0/2096265012 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16966 (secure 0 0 0) 0x7f861c055180 con 0x7f862c0cf1e0 2024-09-13T16:31:12.286 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:12.287 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":83,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:48.943212+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/491709455":"2024-09-14T16:29:48.943151+0000","172.21.15.119:6834/491709455":"2024-09-14T16:29:48.943151+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:12.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8618066730 msgr2=0x7f8618068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8618066730 0x7f8618068bf0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f86280090b0 tx=0x7f8628008030 comp rx=0 tx=0).stop 2024-09-13T16:31:12.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0cf1e0 msgr2=0x7f862c0b0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0cf1e0 0x7f862c0b0c40 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 crypto rx=0x7f861c009fd0 tx=0x7f861c00eb20 comp rx=0 tx=0).stop 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 shutdown_connections 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8618066730 0x7f8618068bf0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f862c0d2a30 0x7f862c0b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f862c0cf1e0 0x7f862c0b0c40 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 --2- 172.21.15.119:0/2096265012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f862c0b3a20 0x7f862c0ca620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 >> 172.21.15.119:0/2096265012 conn(0x7f862c08c0f0 msgr2=0x7f862c06c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 shutdown_connections 2024-09-13T16:31:12.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.285+0000 7f8631dca700 1 -- 172.21.15.119:0/2096265012 wait complete. 2024-09-13T16:31:12.301 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:29:28.552550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:29:29.210253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:31:12.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:31:12.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.417+0000 7ff01e95d700 1 Processor -- start 2024-09-13T16:31:12.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.417+0000 7ff01e95d700 1 -- start start 2024-09-13T16:31:12.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff0180add60 0x7ff0180ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff01815dbe0 0x7ff0180cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0180cd4d0 con 0x7ff0180b3260 2024-09-13T16:31:12.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff018071be0 con 0x7ff0180add60 2024-09-13T16:31:12.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff018159af0 con 0x7ff01815dbe0 2024-09-13T16:31:12.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff0180add60 0x7ff0180ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff0180add60 0x7ff0180ce0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33964/0 (socket says 172.21.15.119:33964) 2024-09-13T16:31:12.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 -- 172.21.15.119:0/697608508 learned_addr learned my addr 172.21.15.119:0/697608508 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:12.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d15a700 1 --2- 172.21.15.119:0/697608508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff01815dbe0 0x7ff0180cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 -- 172.21.15.119:0/697608508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff01815dbe0 msgr2=0x7ff0180cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:12.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 --2- 172.21.15.119:0/697608508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff01815dbe0 0x7ff0180cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 -- 172.21.15.119:0/697608508 >> v1:172.21.15.145:6789/0 conn(0x7ff0180b3260 legacy=0x7ff01806fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01c959700 1 -- 172.21.15.119:0/697608508 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229993998 0 0) 0x7ff0180cd4d0 con 0x7ff0180b3260 2024-09-13T16:31:12.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 -- 172.21.15.119:0/697608508 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01815e130 con 0x7ff0180add60 2024-09-13T16:31:12.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01d95b700 1 --2- 172.21.15.119:0/697608508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180add60 0x7ff0180ce0f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff00c009910 tx=0x7ff00c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91fb5e03d3c79a71 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01c959700 1 -- 172.21.15.119:0/697608508 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c011830 con 0x7ff0180add60 2024-09-13T16:31:12.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01c959700 1 -- 172.21.15.119:0/697608508 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00c011990 con 0x7ff0180add60 2024-09-13T16:31:12.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01c959700 1 -- 172.21.15.119:0/697608508 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00c011d40 con 0x7ff0180add60 2024-09-13T16:31:12.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- 172.21.15.119:0/697608508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180add60 msgr2=0x7ff0180ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 --2- 172.21.15.119:0/697608508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180add60 0x7ff0180ce0f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7ff00c009910 tx=0x7ff00c009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:12.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- 172.21.15.119:0/697608508 shutdown_connections 2024-09-13T16:31:12.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 --2- 172.21.15.119:0/697608508 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff01815dbe0 0x7ff0180cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 --2- 172.21.15.119:0/697608508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180add60 0x7ff0180ce0f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- 172.21.15.119:0/697608508 >> 172.21.15.119:0/697608508 conn(0x7ff0180b0980 msgr2=0x7ff0180b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:12.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- 172.21.15.119:0/697608508 shutdown_connections 2024-09-13T16:31:12.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.421+0000 7ff01e95d700 1 -- 172.21.15.119:0/697608508 wait complete. 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 Processor -- start 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 -- start start 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0180add60 0x7ff01815c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180b3260 0x7ff01814dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff01815dbe0 0x7ff01806c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01815bc70 con 0x7ff0180b3260 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff01814e1d0 con 0x7ff01815dbe0 2024-09-13T16:31:12.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01815b060 con 0x7ff0180add60 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180b3260 0x7ff01814dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180b3260 0x7ff01814dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33970/0 (socket says 172.21.15.119:33970) 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 -- 172.21.15.119:0/308472776 learned_addr learned my addr 172.21.15.119:0/308472776 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e15c700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff01815dbe0 0x7ff01806c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d95b700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0180add60 0x7ff01815c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 -- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0180add60 msgr2=0x7ff01815c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0180add60 0x7ff01815c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 -- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff01815dbe0 msgr2=0x7ff01806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff01815dbe0 0x7ff01806c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 -- 172.21.15.119:0/308472776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff00c009570 con 0x7ff0180b3260 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01d15a700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180b3260 0x7ff01814dc90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff00800efc0 tx=0x7ff008009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff0167fc700 1 -- 172.21.15.119:0/308472776 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00800cae0 con 0x7ff0180b3260 2024-09-13T16:31:12.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01815b340 con 0x7ff0180b3260 2024-09-13T16:31:12.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01815b900 con 0x7ff0180b3260 2024-09-13T16:31:12.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff0167fc700 1 -- 172.21.15.119:0/308472776 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00800f040 con 0x7ff0180b3260 2024-09-13T16:31:12.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.425+0000 7ff0167fc700 1 -- 172.21.15.119:0/308472776 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff00800ce40 con 0x7ff0180b3260 2024-09-13T16:31:12.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.429+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff8005310 con 0x7ff0180b3260 2024-09-13T16:31:12.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.429+0000 7ff0167fc700 1 -- 172.21.15.119:0/308472776 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff008014480 con 0x7ff0180b3260 2024-09-13T16:31:12.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.429+0000 7ff0167fc700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff004066730 0x7ff004068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.429+0000 7ff0167fc700 1 -- 172.21.15.119:0/308472776 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff008084cd0 con 0x7ff0180b3260 2024-09-13T16:31:12.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.429+0000 7ff01d95b700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff004066730 0x7ff004068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.433+0000 7ff01d95b700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff004066730 0x7ff004068bf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff00c000c00 tx=0x7ff00c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.437+0000 7ff0167fc700 1 -- 172.21.15.119:0/308472776 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff008051eb0 con 0x7ff0180b3260 2024-09-13T16:31:12.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.605+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feff8005b40 con 0x7ff0180b3260 2024-09-13T16:31:12.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.605+0000 7ff0167fc700 1 -- 172.21.15.119:0/308472776 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff0080180a0 con 0x7ff0180b3260 2024-09-13T16:31:12.610 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:31:12.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff004066730 msgr2=0x7ff004068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff004066730 0x7ff004068bf0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7ff00c000c00 tx=0x7ff00c008030 comp rx=0 tx=0).stop 2024-09-13T16:31:12.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180b3260 msgr2=0x7ff01814dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180b3260 0x7ff01814dc90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7ff00800efc0 tx=0x7ff008009f80 comp rx=0 tx=0).stop 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 shutdown_connections 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff004066730 0x7ff004068bf0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff01815dbe0 0x7ff01806c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0180b3260 0x7ff01814dc90 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 --2- 172.21.15.119:0/308472776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0180add60 0x7ff01815c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 >> 172.21.15.119:0/308472776 conn(0x7ff0180b0980 msgr2=0x7ff018066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 shutdown_connections 2024-09-13T16:31:12.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.609+0000 7ff01e95d700 1 -- 172.21.15.119:0/308472776 wait complete. 2024-09-13T16:31:12.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:31:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1de1576700 1 Processor -- start 2024-09-13T16:31:12.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1de1576700 1 -- start start 2024-09-13T16:31:12.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1de1576700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ddc0ad9a0 0x7f1ddc159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1de1576700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ddc0b1dd0 con 0x7f1ddc0b1860 2024-09-13T16:31:12.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1de1576700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ddc0b20b0 con 0x7f1ddc06ca60 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1de1576700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ddc0b2fd0 con 0x7f1ddc0ad9a0 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1ddb7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1ddc0b1860 0x7f1ddc06c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41192/0 (socket says 172.21.15.119:41192) 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1ddb7fe700 1 -- 172.21.15.119:0/1078733714 learned_addr learned my addr 172.21.15.119:0/1078733714 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1ddbfff700 1 --2- 172.21.15.119:0/1078733714 >> v2:172.21.15.145:3300/0 conn(0x7f1ddc0ad9a0 0x7f1ddc159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1ddaffd700 1 -- 172.21.15.119:0/1078733714 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354555644 0 0) 0x7f1ddc0b1dd0 con 0x7f1ddc0b1860 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.733+0000 7f1ddaffd700 1 -- 172.21.15.119:0/1078733714 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1dc40035d0 con 0x7f1ddc0b1860 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddaffd700 1 -- 172.21.15.119:0/1078733714 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665510320 0 0) 0x7f1dc40035d0 con 0x7f1ddc0b1860 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddbfff700 1 -- 172.21.15.119:0/1078733714 >> v1:172.21.15.145:6789/0 conn(0x7f1ddc06ca60 legacy=0x7f1ddc15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddbfff700 1 -- 172.21.15.119:0/1078733714 >> v1:172.21.15.119:6789/0 conn(0x7f1ddc0b1860 legacy=0x7f1ddc06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddbfff700 1 -- 172.21.15.119:0/1078733714 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ddc0b2390 con 0x7f1ddc0ad9a0 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddbfff700 1 --2- 172.21.15.119:0/1078733714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc159de0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009850 tx=0x7f1dd0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ceb4fe995e9d8964 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddaffd700 1 -- 172.21.15.119:0/1078733714 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dcc0019f0 con 0x7f1ddc0ad9a0 2024-09-13T16:31:12.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddaffd700 1 -- 172.21.15.119:0/1078733714 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0004030 con 0x7f1ddc0ad9a0 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1ddaffd700 1 -- 172.21.15.119:0/1078733714 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00043a0 con 0x7f1ddc0ad9a0 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 -- 172.21.15.119:0/1078733714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 msgr2=0x7f1ddc159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 --2- 172.21.15.119:0/1078733714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc159de0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009850 tx=0x7f1dd0009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 -- 172.21.15.119:0/1078733714 shutdown_connections 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 --2- 172.21.15.119:0/1078733714 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc159de0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 -- 172.21.15.119:0/1078733714 >> 172.21.15.119:0/1078733714 conn(0x7f1ddc0b0980 msgr2=0x7f1ddc0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 -- 172.21.15.119:0/1078733714 shutdown_connections 2024-09-13T16:31:12.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 -- 172.21.15.119:0/1078733714 wait complete. 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 Processor -- start 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.737+0000 7f1de1576700 1 -- start start 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ddc06ca60 0x7f1ddc158a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc143ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ddc0b1860 0x7f1ddc140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc15d890 con 0x7f1ddc06ca60 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc1421d0 con 0x7f1ddc0ad9a0 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ddc0cd450 con 0x7f1ddc0b1860 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ddc06ca60 0x7f1ddc158a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ddc06ca60 0x7f1ddc158a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33982/0 (socket says 172.21.15.119:33982) 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 -- 172.21.15.119:0/4286340632 learned_addr learned my addr 172.21.15.119:0/4286340632 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddb7fe700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc143ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de0d75700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ddc0b1860 0x7f1ddc140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 -- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ddc0b1860 msgr2=0x7f1ddc140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ddc0b1860 0x7f1ddc140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 -- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 msgr2=0x7f1ddc143ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc143ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 -- 172.21.15.119:0/4286340632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dd00094b0 con 0x7f1ddc06ca60 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddb7fe700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1ddbfff700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ddc06ca60 0x7f1ddc158a90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009ba0 tx=0x7f1dd0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1dd8ff9700 1 -- 172.21.15.119:0/4286340632 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd0011630 con 0x7f1ddc06ca60 2024-09-13T16:31:12.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1dd8ff9700 1 -- 172.21.15.119:0/4286340632 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd001f070 con 0x7f1ddc06ca60 2024-09-13T16:31:12.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ddc0cd730 con 0x7f1ddc06ca60 2024-09-13T16:31:12.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1dd8ff9700 1 -- 172.21.15.119:0/4286340632 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dd00119b0 con 0x7f1ddc06ca60 2024-09-13T16:31:12.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.741+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ddc158fd0 con 0x7f1ddc06ca60 2024-09-13T16:31:12.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.745+0000 7f1dd8ff9700 1 -- 172.21.15.119:0/4286340632 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1dd0011b10 con 0x7f1ddc06ca60 2024-09-13T16:31:12.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.745+0000 7f1dd8ff9700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1dc8066730 0x7f1dc8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:12.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.745+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ddc0cdc50 con 0x7f1ddc06ca60 2024-09-13T16:31:12.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.745+0000 7f1dd8ff9700 1 -- 172.21.15.119:0/4286340632 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1dd0021070 con 0x7f1ddc06ca60 2024-09-13T16:31:12.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.745+0000 7f1ddb7fe700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1dc8066730 0x7f1dc8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:12.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.745+0000 7f1ddb7fe700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1dc8066730 0x7f1dc8068bf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc141af0 tx=0x7f1dcc002120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:12.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.753+0000 7f1dd8ff9700 1 -- 172.21.15.119:0/4286340632 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1ddc0cdc50 con 0x7f1ddc06ca60 2024-09-13T16:31:12.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.917+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1ddc0cdc50 con 0x7f1ddc06ca60 2024-09-13T16:31:12.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1dd8ff9700 1 -- 172.21.15.119:0/4286340632 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1ddc0cdc50 con 0x7f1ddc06ca60 2024-09-13T16:31:12.923 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1dc8066730 msgr2=0x7f1dc8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1dc8066730 0x7f1dc8068bf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc141af0 tx=0x7f1dcc002120 comp rx=0 tx=0).stop 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ddc06ca60 msgr2=0x7f1ddc158a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ddc06ca60 0x7f1ddc158a90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f1dd0009ba0 tx=0x7f1dd0004770 comp rx=0 tx=0).stop 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 shutdown_connections 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1dc8066730 0x7f1dc8068bf0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ddc0b1860 0x7f1ddc140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ddc0ad9a0 0x7f1ddc143ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 --2- 172.21.15.119:0/4286340632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ddc06ca60 0x7f1ddc158a90 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 >> 172.21.15.119:0/4286340632 conn(0x7f1ddc0b0980 msgr2=0x7f1ddc0b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:12.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 shutdown_connections 2024-09-13T16:31:12.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:12.921+0000 7f1de1576700 1 -- 172.21.15.119:0/4286340632 wait complete. 2024-09-13T16:31:12.938 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:31:13.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.053+0000 7fc5fee17700 1 Processor -- start 2024-09-13T16:31:13.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.053+0000 7fc5fee17700 1 -- start start 2024-09-13T16:31:13.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.053+0000 7fc5fee17700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc5f80b1d00 0x7fc5f8158950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.053+0000 7fc5fee17700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f806bbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.053+0000 7fc5fee17700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f80704d0 con 0x7fc5f806dbf0 2024-09-13T16:31:13.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.053+0000 7fc5fee17700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f8070720 con 0x7fc5f80b1d00 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.053+0000 7fc5fee17700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8070a00 con 0x7fc5f8159980 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc5f80b1d00 0x7fc5f8158950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc5f80b1d00 0x7fc5f8158950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33988/0 (socket says 172.21.15.119:33988) 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fde15700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc5f806dbf0 0x7fc5f815cf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41206/0 (socket says 172.21.15.119:41206) 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 -- 172.21.15.119:0/1639689769 learned_addr learned my addr 172.21.15.119:0/1639689769 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fe616700 1 --2- 172.21.15.119:0/1639689769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f806bbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 -- 172.21.15.119:0/1639689769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 msgr2=0x7fc5f806bbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 --2- 172.21.15.119:0/1639689769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f806bbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 -- 172.21.15.119:0/1639689769 >> v1:172.21.15.119:6789/0 conn(0x7fc5f806dbf0 legacy=0x7fc5f815cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fce13700 1 -- 172.21.15.119:0/1639689769 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3848484154 0 0) 0x7fc5f80704d0 con 0x7fc5f806dbf0 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 -- 172.21.15.119:0/1639689769 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f8159180 con 0x7fc5f80b1d00 2024-09-13T16:31:13.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fd614700 1 --2- 172.21.15.119:0/1639689769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f8158950 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009e50 tx=0x7fc5f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84bcf9ad126606b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fce13700 1 -- 172.21.15.119:0/1639689769 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f400e070 con 0x7fc5f80b1d00 2024-09-13T16:31:13.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fce13700 1 -- 172.21.15.119:0/1639689769 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f400b910 con 0x7fc5f80b1d00 2024-09-13T16:31:13.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fce13700 1 -- 172.21.15.119:0/1639689769 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f400bcc0 con 0x7fc5f80b1d00 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 -- 172.21.15.119:0/1639689769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 msgr2=0x7fc5f8158950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 --2- 172.21.15.119:0/1639689769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f8158950 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4009e50 tx=0x7fc5f400b330 comp rx=0 tx=0).stop 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 -- 172.21.15.119:0/1639689769 shutdown_connections 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 --2- 172.21.15.119:0/1639689769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f806bbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 --2- 172.21.15.119:0/1639689769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f8158950 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 -- 172.21.15.119:0/1639689769 >> 172.21.15.119:0/1639689769 conn(0x7fc5f80b0980 msgr2=0x7fc5f80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 -- 172.21.15.119:0/1639689769 shutdown_connections 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 -- 172.21.15.119:0/1639689769 wait complete. 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.057+0000 7fc5fee17700 1 Processor -- start 2024-09-13T16:31:13.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 -- start start 2024-09-13T16:31:13.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5f806dbf0 0x7fc5f814db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f814fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f81518f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f814e0a0 con 0x7fc5f80b1d00 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8147ad0 con 0x7fc5f806dbf0 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f8147d30 con 0x7fc5f8159980 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f814fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f814fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33990/0 (socket says 172.21.15.119:33990) 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 -- 172.21.15.119:0/4114379707 learned_addr learned my addr 172.21.15.119:0/4114379707 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fde15700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5f806dbf0 0x7fc5f814db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fe616700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f81518f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 -- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 msgr2=0x7fc5f81518f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f81518f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.066 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 -- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5f806dbf0 msgr2=0x7fc5f814db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5f806dbf0 0x7fc5f814db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 -- 172.21.15.119:0/4114379707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f4009b30 con 0x7fc5f80b1d00 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fd614700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f814fb50 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4012000 tx=0x7fc5f4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5ee7fc700 1 -- 172.21.15.119:0/4114379707 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f4004900 con 0x7fc5f80b1d00 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5ee7fc700 1 -- 172.21.15.119:0/4114379707 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f40092e0 con 0x7fc5f80b1d00 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f8148010 con 0x7fc5f80b1d00 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5ee7fc700 1 -- 172.21.15.119:0/4114379707 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f401c640 con 0x7fc5f80b1d00 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.061+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f8143f70 con 0x7fc5f80b1d00 2024-09-13T16:31:13.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.065+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f8158950 con 0x7fc5f80b1d00 2024-09-13T16:31:13.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.065+0000 7fc5ee7fc700 1 -- 172.21.15.119:0/4114379707 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc5f401c7a0 con 0x7fc5f80b1d00 2024-09-13T16:31:13.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.065+0000 7fc5ee7fc700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc5e4066730 0x7fc5e4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.065+0000 7fc5fde15700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc5e4066730 0x7fc5e4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.065+0000 7fc5ee7fc700 1 -- 172.21.15.119:0/4114379707 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fc5f4085be0 con 0x7fc5f80b1d00 2024-09-13T16:31:13.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.065+0000 7fc5fde15700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc5e4066730 0x7fc5e4068bf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8000f60 tx=0x7fc5e8007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.073+0000 7fc5ee7fc700 1 -- 172.21.15.119:0/4114379707 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc5f4052da0 con 0x7fc5f80b1d00 2024-09-13T16:31:13.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc5f8152560 con 0x7fc5f80b1d00 2024-09-13T16:31:13.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5ee7fc700 1 -- 172.21.15.119:0/4114379707 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7fc5f40563c0 con 0x7fc5f80b1d00 2024-09-13T16:31:13.248 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:31:13.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc5e4066730 msgr2=0x7fc5e4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc5e4066730 0x7fc5e4068bf0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fc5e8000f60 tx=0x7fc5e8007e90 comp rx=0 tx=0).stop 2024-09-13T16:31:13.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 msgr2=0x7fc5f814fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f814fb50 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fc5f4012000 tx=0x7fc5f4004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 shutdown_connections 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc5e4066730 0x7fc5e4068bf0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc5f8159980 0x7fc5f81518f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc5f80b1d00 0x7fc5f814fb50 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 --2- 172.21.15.119:0/4114379707 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc5f806dbf0 0x7fc5f814db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 >> 172.21.15.119:0/4114379707 conn(0x7fc5f80b0980 msgr2=0x7fc5f8071440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 shutdown_connections 2024-09-13T16:31:13.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.245+0000 7fc5fee17700 1 -- 172.21.15.119:0/4114379707 wait complete. 2024-09-13T16:31:13.263 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:31:13.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9150a15700 1 Processor -- start 2024-09-13T16:31:13.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9150a15700 1 -- start start 2024-09-13T16:31:13.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9150a15700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f914c0b17f0 0x7f914c0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9150a15700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c0705d0 con 0x7f914c0b2200 2024-09-13T16:31:13.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9150a15700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f914c06fca0 con 0x7f914c0cd8d0 2024-09-13T16:31:13.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9150a15700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f914c06ff80 con 0x7f914c0b17f0 2024-09-13T16:31:13.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f914b59e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f914c0b2200 0x7f914c070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41208/0 (socket says 172.21.15.119:41208) 2024-09-13T16:31:13.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f914a59c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f914c0b17f0 0x7f914c0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f914b59e700 1 -- 172.21.15.119:0/2222075728 learned_addr learned my addr 172.21.15.119:0/2222075728 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:13.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9149d9b700 1 -- 172.21.15.119:0/2222075728 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290415330 0 0) 0x7f914c0705d0 con 0x7f914c0b2200 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9149d9b700 1 -- 172.21.15.119:0/2222075728 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91380035d0 con 0x7f914c0b2200 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f914a59c700 1 -- 172.21.15.119:0/2222075728 >> v1:172.21.15.145:6789/0 conn(0x7f914c0cd8d0 legacy=0x7f914c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9149d9b700 1 -- 172.21.15.119:0/2222075728 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790995717 0 0) 0x7f914c06fca0 con 0x7f914c0cd8d0 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f914a59c700 1 -- 172.21.15.119:0/2222075728 >> v1:172.21.15.119:6789/0 conn(0x7f914c0b2200 legacy=0x7f914c070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f914a59c700 1 -- 172.21.15.119:0/2222075728 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f914c070260 con 0x7f914c0b17f0 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f914a59c700 1 --2- 172.21.15.119:0/2222075728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0b17f0 0x7f914c0cb800 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9144009d90 tx=0x7f914400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ca2e79a239c67c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9149d9b700 1 -- 172.21.15.119:0/2222075728 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914400e070 con 0x7f914c0b17f0 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9149d9b700 1 -- 172.21.15.119:0/2222075728 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f914400bb20 con 0x7f914c0b17f0 2024-09-13T16:31:13.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.377+0000 7f9149d9b700 1 -- 172.21.15.119:0/2222075728 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f914400be90 con 0x7f914c0b17f0 2024-09-13T16:31:13.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- 172.21.15.119:0/2222075728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0b17f0 msgr2=0x7f914c0cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 --2- 172.21.15.119:0/2222075728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0b17f0 0x7f914c0cb800 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f9144009d90 tx=0x7f914400b330 comp rx=0 tx=0).stop 2024-09-13T16:31:13.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- 172.21.15.119:0/2222075728 shutdown_connections 2024-09-13T16:31:13.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 --2- 172.21.15.119:0/2222075728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0b17f0 0x7f914c0cb800 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- 172.21.15.119:0/2222075728 >> 172.21.15.119:0/2222075728 conn(0x7f914c06a350 msgr2=0x7f914c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:13.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- 172.21.15.119:0/2222075728 shutdown_connections 2024-09-13T16:31:13.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- 172.21.15.119:0/2222075728 wait complete. 2024-09-13T16:31:13.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 Processor -- start 2024-09-13T16:31:13.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- start start 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0b17f0 0x7f914c0aaef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f914c0b2200 0x7f914c0ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0cd8d0 0x7f914c0ac8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0ab430 con 0x7f914c0b17f0 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0afc90 con 0x7f914c0cd8d0 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f9150a15700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0aebf0 con 0x7f914c0b2200 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f914ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0b17f0 0x7f914c0aaef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f914ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0b17f0 0x7f914c0aaef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34004/0 (socket says 172.21.15.119:34004) 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f914ad9d700 1 -- 172.21.15.119:0/2283039657 learned_addr learned my addr 172.21.15.119:0/2283039657 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f914a59c700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f914c0b2200 0x7f914c0ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.381+0000 7f914b59e700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0cd8d0 0x7f914c0ac8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f914ad9d700 1 -- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f914c0b2200 msgr2=0x7f914c0ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f914ad9d700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f914c0b2200 0x7f914c0ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f914ad9d700 1 -- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0cd8d0 msgr2=0x7f914c0ac8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f914ad9d700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0cd8d0 0x7f914c0ac8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f914ad9d700 1 -- 172.21.15.119:0/2283039657 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9144009a70 con 0x7f914c0b17f0 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f914b59e700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0cd8d0 0x7f914c0ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f914ad9d700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0b17f0 0x7f914c0aaef0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f913400c520 tx=0x7f913400c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f91437fe700 1 -- 172.21.15.119:0/2283039657 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91340054e0 con 0x7f914c0b17f0 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f91437fe700 1 -- 172.21.15.119:0/2283039657 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91340036a0 con 0x7f914c0b17f0 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c0aeed0 con 0x7f914c0b17f0 2024-09-13T16:31:13.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f91437fe700 1 -- 172.21.15.119:0/2283039657 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9134003a10 con 0x7f914c0b17f0 2024-09-13T16:31:13.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c0af490 con 0x7f914c0b17f0 2024-09-13T16:31:13.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c0cb800 con 0x7f914c0b17f0 2024-09-13T16:31:13.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f91437fe700 1 -- 172.21.15.119:0/2283039657 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9134003b70 con 0x7f914c0b17f0 2024-09-13T16:31:13.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f91437fe700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f912c066730 0x7f912c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.385+0000 7f91437fe700 1 -- 172.21.15.119:0/2283039657 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f91340835f0 con 0x7f914c0b17f0 2024-09-13T16:31:13.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.389+0000 7f914a59c700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f912c066730 0x7f912c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.389+0000 7f914a59c700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f912c066730 0x7f912c068bf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9144000f80 tx=0x7f914401d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.389+0000 7f91437fe700 1 -- 172.21.15.119:0/2283039657 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9134050900 con 0x7f914c0b17f0 2024-09-13T16:31:13.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.569+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f914c0cc0e0 con 0x7f914c0b17f0 2024-09-13T16:31:13.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f91437fe700 1 -- 172.21.15.119:0/2283039657 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v83) v1 ==== 80+0+122 (secure 0 0 0) 0x7f9134053f20 con 0x7f914c0b17f0 2024-09-13T16:31:13.575 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6835/491709455 2024-09-14T16:29:48.943151+0000 2024-09-13T16:31:13.576 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6834/491709455 2024-09-14T16:29:48.943151+0000 2024-09-13T16:31:13.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f912c066730 msgr2=0x7f912c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f912c066730 0x7f912c068bf0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f9144000f80 tx=0x7f914401d040 comp rx=0 tx=0).stop 2024-09-13T16:31:13.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0b17f0 msgr2=0x7f914c0aaef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0b17f0 0x7f914c0aaef0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f913400c520 tx=0x7f913400c830 comp rx=0 tx=0).stop 2024-09-13T16:31:13.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 shutdown_connections 2024-09-13T16:31:13.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f912c066730 0x7f912c068bf0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f914c0cd8d0 0x7f914c0ac8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f914c0b2200 0x7f914c0ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 --2- 172.21.15.119:0/2283039657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f914c0b17f0 0x7f914c0aaef0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 >> 172.21.15.119:0/2283039657 conn(0x7f914c06a350 msgr2=0x7f914c070a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:13.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 shutdown_connections 2024-09-13T16:31:13.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.573+0000 7f9150a15700 1 -- 172.21.15.119:0/2283039657 wait complete. 2024-09-13T16:31:13.580 INFO:teuthology.orchestra.run.smithi119.stderr:listed 2 entries 2024-09-13T16:31:13.591 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:31:13.591 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:13.592 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:13.607 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[45772]: fuse finished with error 0 and tester_r 0 2024-09-13T16:31:13.609 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:31:13.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:13.609 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:31:13.758 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:31:13.759 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:13.766 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:31:13.767 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:31:13.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:13.767 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:31:13.772 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-13T16:31:13.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:31:13.774 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:31:13.774 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:13.781 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:31:13.782 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:31:13.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:13.782 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:31:13.788 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-09-13T16:31:13.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:31:13.789 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:31:13.789 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:13.821 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:31:13.821 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:31:13.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:13.821 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:31:13.827 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-13T16:31:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:31:13.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:13.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.945+0000 7f5157fff700 1 Processor -- start 2024-09-13T16:31:13.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.945+0000 7f5157fff700 1 -- start start 2024-09-13T16:31:13.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.945+0000 7f5157fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f5158071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f515806d540 0x7f5158071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515806de00 con 0x7f5158158f10 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515806db20 con 0x7f515806d540 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51580b1920 con 0x7f5158158b30 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f515806d540 0x7f5158071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f515806d540 0x7f5158071570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34018/0 (socket says 172.21.15.119:34018) 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 -- 172.21.15.119:0/628077850 learned_addr learned my addr 172.21.15.119:0/628077850 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5156ffd700 1 --2- 172.21.15.119:0/628077850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f5158071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 -- 172.21.15.119:0/628077850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 msgr2=0x7f5158071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 --2- 172.21.15.119:0/628077850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f5158071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 -- 172.21.15.119:0/628077850 >> v1:172.21.15.145:6789/0 conn(0x7f5158158f10 legacy=0x7f515806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 -- 172.21.15.119:0/628077850 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51580b1c00 con 0x7f515806d540 2024-09-13T16:31:13.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f51577fe700 1 --2- 172.21.15.119:0/628077850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515806d540 0x7f5158071570 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f515000bbb0 tx=0x7f515000bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60a5b31bba8e83d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5155ffb700 1 -- 172.21.15.119:0/628077850 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f515000dd50 con 0x7f515806d540 2024-09-13T16:31:13.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5155ffb700 1 -- 172.21.15.119:0/628077850 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150004030 con 0x7f515806d540 2024-09-13T16:31:13.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5155ffb700 1 -- 172.21.15.119:0/628077850 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51500043e0 con 0x7f515806d540 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- 172.21.15.119:0/628077850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515806d540 msgr2=0x7f5158071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 --2- 172.21.15.119:0/628077850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515806d540 0x7f5158071570 secure :-1 s=READY pgs=1173 cs=0 l=1 rev1=1 crypto rx=0x7f515000bbb0 tx=0x7f515000bec0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- 172.21.15.119:0/628077850 shutdown_connections 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 --2- 172.21.15.119:0/628077850 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f515806d540 0x7f5158071570 unknown :-1 s=CLOSED pgs=1173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 --2- 172.21.15.119:0/628077850 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f5158071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- 172.21.15.119:0/628077850 >> 172.21.15.119:0/628077850 conn(0x7f515806a350 msgr2=0x7f515806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- 172.21.15.119:0/628077850 shutdown_connections 2024-09-13T16:31:13.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.949+0000 7f5157fff700 1 -- 172.21.15.119:0/628077850 wait complete. 2024-09-13T16:31:13.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 Processor -- start 2024-09-13T16:31:13.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 -- start start 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515806d540 0x7f5158159ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f51580a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5158158f10 0x7f51580a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51581f0080 con 0x7f5158158f10 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51581ef470 con 0x7f515806d540 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5157fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51581ef750 con 0x7f5158158b30 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f51577fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5158158f10 0x7f51580a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f51577fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5158158f10 0x7f51580a89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34024/0 (socket says 172.21.15.119:34024) 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f51577fe700 1 -- 172.21.15.119:0/1887066732 learned_addr learned my addr 172.21.15.119:0/1887066732 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:13.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f51567fc700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f51580a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.953+0000 7f5156ffd700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515806d540 0x7f5158159ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f51577fe700 1 -- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 msgr2=0x7f51580a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f51577fe700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f51580a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f51577fe700 1 -- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515806d540 msgr2=0x7f5158159ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f51577fe700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515806d540 0x7f5158159ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f51577fe700 1 -- 172.21.15.119:0/1887066732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515000b810 con 0x7f5158158f10 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f51577fe700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5158158f10 0x7f51580a89f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f51580b3ee0 tx=0x7f5150004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f513f7fe700 1 -- 172.21.15.119:0/1887066732 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150020070 con 0x7f5158158f10 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f513f7fe700 1 -- 172.21.15.119:0/1887066732 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5150003950 con 0x7f5158158f10 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51581efa30 con 0x7f5158158f10 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f513f7fe700 1 -- 172.21.15.119:0/1887066732 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5150003d00 con 0x7f5158158f10 2024-09-13T16:31:13.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51581f0510 con 0x7f5158158f10 2024-09-13T16:31:13.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515806ce30 con 0x7f5158158f10 2024-09-13T16:31:13.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.957+0000 7f513f7fe700 1 -- 172.21.15.119:0/1887066732 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f51500193f0 con 0x7f5158158f10 2024-09-13T16:31:13.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.961+0000 7f513f7fe700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5144066730 0x7f5144068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:13.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.961+0000 7f513f7fe700 1 -- 172.21.15.119:0/1887066732 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f5150087870 con 0x7f5158158f10 2024-09-13T16:31:13.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.961+0000 7f5156ffd700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5144066730 0x7f5144068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:13.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.961+0000 7f5156ffd700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5144066730 0x7f5144068bf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f514c009950 tx=0x7f514c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:13.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:13.969+0000 7f513f7fe700 1 -- 172.21.15.119:0/1887066732 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5150054a30 con 0x7f5158158f10 2024-09-13T16:31:14.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.177+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f515806a8d0 con 0x7f5158158f10 2024-09-13T16:31:14.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.177+0000 7f513f7fe700 1 -- 172.21.15.119:0/1887066732 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f5150003e60 con 0x7f5158158f10 2024-09-13T16:31:14.182 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:14.182 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:30:40.052433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:31:14.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5144066730 msgr2=0x7f5144068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5144066730 0x7f5144068bf0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f514c009950 tx=0x7f514c008040 comp rx=0 tx=0).stop 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5158158f10 msgr2=0x7f51580a89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5158158f10 0x7f51580a89f0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f51580b3ee0 tx=0x7f5150004770 comp rx=0 tx=0).stop 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 shutdown_connections 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5144066730 0x7f5144068bf0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5158158f10 0x7f51580a89f0 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5158158b30 0x7f51580a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 --2- 172.21.15.119:0/1887066732 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f515806d540 0x7f5158159ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 >> 172.21.15.119:0/1887066732 conn(0x7f515806a350 msgr2=0x7f51581554d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 shutdown_connections 2024-09-13T16:31:14.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.181+0000 7f5157fff700 1 -- 172.21.15.119:0/1887066732 wait complete. 2024-09-13T16:31:14.186 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 49 2024-09-13T16:31:14.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:14.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.301+0000 7fdeb6a29700 1 Processor -- start 2024-09-13T16:31:14.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.301+0000 7fdeb6a29700 1 -- start start 2024-09-13T16:31:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb01509e0 0x7fdeb0159cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdeb00cc220 0x7fdeb015a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdeb015a740 0x7fdeb015d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb015abf0 con 0x7fdeb015a740 2024-09-13T16:31:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdeb00b44e0 con 0x7fdeb00cc220 2024-09-13T16:31:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00cddf0 con 0x7fdeb01509e0 2024-09-13T16:31:14.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdeb015a740 0x7fdeb015d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdeb015a740 0x7fdeb015d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34036/0 (socket says 172.21.15.119:34036) 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 -- 172.21.15.119:0/3437021456 learned_addr learned my addr 172.21.15.119:0/3437021456 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb5a27700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb01509e0 0x7fdeb0159cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb5226700 1 --2- 172.21.15.119:0/3437021456 >> v2:172.21.15.145:3300/0 conn(0x7fdeb00cc220 0x7fdeb015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 -- 172.21.15.119:0/3437021456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb01509e0 msgr2=0x7fdeb0159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 --2- 172.21.15.119:0/3437021456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb01509e0 0x7fdeb0159cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 -- 172.21.15.119:0/3437021456 >> v2:172.21.15.145:3300/0 conn(0x7fdeb00cc220 msgr2=0x7fdeb015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 --2- 172.21.15.119:0/3437021456 >> v2:172.21.15.145:3300/0 conn(0x7fdeb00cc220 0x7fdeb015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 -- 172.21.15.119:0/3437021456 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb0071a50 con 0x7fdeb015a740 2024-09-13T16:31:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb5a27700 1 --2- 172.21.15.119:0/3437021456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb01509e0 0x7fdeb0159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:14.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6228700 1 --2- 172.21.15.119:0/3437021456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb015d2b0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00ccd80 tx=0x7fdea800d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76fb4d64fd40d07c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb4a25700 1 -- 172.21.15.119:0/3437021456 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8010070 con 0x7fdeb015a740 2024-09-13T16:31:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb4a25700 1 -- 172.21.15.119:0/3437021456 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea800dce0 con 0x7fdeb015a740 2024-09-13T16:31:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb4a25700 1 -- 172.21.15.119:0/3437021456 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea800b310 con 0x7fdeb015a740 2024-09-13T16:31:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- 172.21.15.119:0/3437021456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 msgr2=0x7fdeb015d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/3437021456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb015d2b0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00ccd80 tx=0x7fdea800d430 comp rx=0 tx=0).stop 2024-09-13T16:31:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- 172.21.15.119:0/3437021456 shutdown_connections 2024-09-13T16:31:14.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/3437021456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb015d2b0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/3437021456 >> v2:172.21.15.145:3300/0 conn(0x7fdeb00cc220 0x7fdeb015a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/3437021456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb01509e0 0x7fdeb0159cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- 172.21.15.119:0/3437021456 >> 172.21.15.119:0/3437021456 conn(0x7fdeb0070cc0 msgr2=0x7fdeb00710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:14.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- 172.21.15.119:0/3437021456 shutdown_connections 2024-09-13T16:31:14.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.305+0000 7fdeb6a29700 1 -- 172.21.15.119:0/3437021456 wait complete. 2024-09-13T16:31:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 Processor -- start 2024-09-13T16:31:14.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 -- start start 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00cc220 0x7fdeb01360f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb01509e0 0x7fdeb014afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb014b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0143650 con 0x7fdeb015a740 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0136630 con 0x7fdeb01509e0 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6a29700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0147d40 con 0x7fdeb00cc220 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6228700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb014b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6228700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb014b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34050/0 (socket says 172.21.15.119:34050) 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.309+0000 7fdeb6228700 1 -- 172.21.15.119:0/1207824405 learned_addr learned my addr 172.21.15.119:0/1207824405 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb5226700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb01509e0 0x7fdeb014afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb5a27700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00cc220 0x7fdeb01360f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6228700 1 -- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00cc220 msgr2=0x7fdeb01360f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6228700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00cc220 0x7fdeb01360f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6228700 1 -- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb01509e0 msgr2=0x7fdeb014afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6228700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb01509e0 0x7fdeb014afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6228700 1 -- 172.21.15.119:0/1207824405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac0094b0 con 0x7fdeb015a740 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6228700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb014b4f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00724c0 tx=0x7fdea8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdea67fc700 1 -- 172.21.15.119:0/1207824405 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea8009a10 con 0x7fdeb015a740 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdea67fc700 1 -- 172.21.15.119:0/1207824405 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea8009b90 con 0x7fdeb015a740 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea800bd40 con 0x7fdeb015a740 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdea67fc700 1 -- 172.21.15.119:0/1207824405 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea800b4c0 con 0x7fdeb015a740 2024-09-13T16:31:14.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb0142f90 con 0x7fdeb015a740 2024-09-13T16:31:14.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.313+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb0143790 con 0x7fdeb015a740 2024-09-13T16:31:14.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.317+0000 7fdea67fc700 1 -- 172.21.15.119:0/1207824405 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fdea8004040 con 0x7fdeb015a740 2024-09-13T16:31:14.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.317+0000 7fdea67fc700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fde94066730 0x7fde94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.317+0000 7fdea67fc700 1 -- 172.21.15.119:0/1207824405 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fdea8087c40 con 0x7fdeb015a740 2024-09-13T16:31:14.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.317+0000 7fdeb5a27700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fde94066730 0x7fde94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.317+0000 7fdeb5a27700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fde94066730 0x7fde94068bf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009d80 tx=0x7fdeac009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.321+0000 7fdea67fc700 1 -- 172.21.15.119:0/1207824405 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdeb0143790 con 0x7fdeb015a740 2024-09-13T16:31:14.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.497+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdeb0143790 con 0x7fdeb015a740 2024-09-13T16:31:14.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.497+0000 7fdea67fc700 1 -- 172.21.15.119:0/1207824405 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16966 (secure 0 0 0) 0x7fdeb0143790 con 0x7fdeb015a740 2024-09-13T16:31:14.503 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:14.503 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":83,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:48.943212+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/491709455":"2024-09-14T16:29:48.943151+0000","172.21.15.119:6834/491709455":"2024-09-14T16:29:48.943151+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:14.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fde94066730 msgr2=0x7fde94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fde94066730 0x7fde94068bf0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fdeac009d80 tx=0x7fdeac009240 comp rx=0 tx=0).stop 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 msgr2=0x7fdeb014b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb014b4f0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 crypto rx=0x7fdeb00724c0 tx=0x7fdea8004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 shutdown_connections 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fde94066730 0x7fde94068bf0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdeb015a740 0x7fdeb014b4f0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdeb01509e0 0x7fdeb014afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 --2- 172.21.15.119:0/1207824405 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdeb00cc220 0x7fdeb01360f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 >> 172.21.15.119:0/1207824405 conn(0x7fdeb0070cc0 msgr2=0x7fdeb006e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 shutdown_connections 2024-09-13T16:31:14.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.501+0000 7fdeb6a29700 1 -- 172.21.15.119:0/1207824405 wait complete. 2024-09-13T16:31:14.518 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:29:28.552550+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:29:29.210253+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:31:14.519 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:31:14.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1536ceb700 1 Processor -- start 2024-09-13T16:31:14.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1536ceb700 1 -- start start 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1536ceb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1530157280 0x7f153006ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1536ceb700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1530071210 0x7f153006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1536ceb700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153006e830 con 0x7f1530156ea0 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1536ceb700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15300b4810 con 0x7f1530157280 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1536ceb700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153006d4e0 con 0x7f1530071210 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1535ce9700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1530156ea0 0x7f1530070b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41210/0 (socket says 172.21.15.119:41210) 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f15354e8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1530157280 0x7f153006ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1535ce9700 1 -- 172.21.15.119:0/565501124 learned_addr learned my addr 172.21.15.119:0/565501124 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f15364ea700 1 --2- 172.21.15.119:0/565501124 >> v2:172.21.15.145:3300/0 conn(0x7f1530071210 0x7f153006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f15354e8700 1 -- 172.21.15.119:0/565501124 >> v2:172.21.15.145:3300/0 conn(0x7f1530071210 msgr2=0x7f153006cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:14.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f15354e8700 1 --2- 172.21.15.119:0/565501124 >> v2:172.21.15.145:3300/0 conn(0x7f1530071210 0x7f153006cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f15354e8700 1 -- 172.21.15.119:0/565501124 >> v1:172.21.15.119:6789/0 conn(0x7f1530156ea0 legacy=0x7f1530070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f15354e8700 1 -- 172.21.15.119:0/565501124 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153006d790 con 0x7f1530157280 2024-09-13T16:31:14.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f15354e8700 1 --2- 172.21.15.119:0/565501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530157280 0x7f153006ca60 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1530071f50 tx=0x7f152c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac7517d8c50f6c2b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1534ce7700 1 -- 172.21.15.119:0/565501124 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c011830 con 0x7f1530157280 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1534ce7700 1 -- 172.21.15.119:0/565501124 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f152c011990 con 0x7f1530157280 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.637+0000 7f1534ce7700 1 -- 172.21.15.119:0/565501124 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f152c011d20 con 0x7f1530157280 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 -- 172.21.15.119:0/565501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530157280 msgr2=0x7f153006ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 --2- 172.21.15.119:0/565501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530157280 0x7f153006ca60 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f1530071f50 tx=0x7f152c009b30 comp rx=0 tx=0).stop 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 -- 172.21.15.119:0/565501124 shutdown_connections 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 --2- 172.21.15.119:0/565501124 >> v2:172.21.15.145:3300/0 conn(0x7f1530071210 0x7f153006cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 --2- 172.21.15.119:0/565501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530157280 0x7f153006ca60 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 -- 172.21.15.119:0/565501124 >> 172.21.15.119:0/565501124 conn(0x7f15300b1500 msgr2=0x7f15300b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:14.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 -- 172.21.15.119:0/565501124 shutdown_connections 2024-09-13T16:31:14.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 -- 172.21.15.119:0/565501124 wait complete. 2024-09-13T16:31:14.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 Processor -- start 2024-09-13T16:31:14.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.641+0000 7f1536ceb700 1 -- start start 2024-09-13T16:31:14.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530071210 0x7f15300ae480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1530156ea0 0x7f1530157b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1530157280 0x7f153014f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15301f25c0 con 0x7f1530071210 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15301f28a0 con 0x7f1530157280 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15301f2b80 con 0x7f1530156ea0 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530071210 0x7f15300ae480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530071210 0x7f15300ae480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34078/0 (socket says 172.21.15.119:34078) 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 -- 172.21.15.119:0/3677428700 learned_addr learned my addr 172.21.15.119:0/3677428700 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f15364ea700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1530157280 0x7f153014f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f15354e8700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1530156ea0 0x7f1530157b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 -- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1530156ea0 msgr2=0x7f1530157b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1530156ea0 0x7f1530157b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 -- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1530157280 msgr2=0x7f153014f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1530157280 0x7f153014f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 -- 172.21.15.119:0/3677428700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c0094b0 con 0x7f1530071210 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1535ce9700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530071210 0x7f15300ae480 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f152400c700 tx=0x7f152400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f15227fc700 1 -- 172.21.15.119:0/3677428700 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524005d90 con 0x7f1530071210 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f15227fc700 1 -- 172.21.15.119:0/3677428700 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15240070d0 con 0x7f1530071210 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15301f2e60 con 0x7f1530071210 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f15227fc700 1 -- 172.21.15.119:0/3677428700 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524007440 con 0x7f1530071210 2024-09-13T16:31:14.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1530157760 con 0x7f1530071210 2024-09-13T16:31:14.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.645+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1530070b00 con 0x7f1530071210 2024-09-13T16:31:14.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.649+0000 7f15227fc700 1 -- 172.21.15.119:0/3677428700 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f15240075a0 con 0x7f1530071210 2024-09-13T16:31:14.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.649+0000 7f15227fc700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1514066730 0x7f1514068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.649+0000 7f15227fc700 1 -- 172.21.15.119:0/3677428700 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f15240834b0 con 0x7f1530071210 2024-09-13T16:31:14.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.649+0000 7f15354e8700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1514066730 0x7f1514068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.649+0000 7f15354e8700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1514066730 0x7f1514068bf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f152c000c00 tx=0x7f152c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.657+0000 7f15227fc700 1 -- 172.21.15.119:0/3677428700 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f15240507c0 con 0x7f1530071210 2024-09-13T16:31:14.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15301f3050 con 0x7f1530071210 2024-09-13T16:31:14.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f15227fc700 1 -- 172.21.15.119:0/3677428700 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1524017020 con 0x7f1530071210 2024-09-13T16:31:14.836 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:31:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1514066730 msgr2=0x7f1514068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1514066730 0x7f1514068bf0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f152c000c00 tx=0x7f152c008030 comp rx=0 tx=0).stop 2024-09-13T16:31:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530071210 msgr2=0x7f15300ae480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530071210 0x7f15300ae480 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f152400c700 tx=0x7f152400ca10 comp rx=0 tx=0).stop 2024-09-13T16:31:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 shutdown_connections 2024-09-13T16:31:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1514066730 0x7f1514068bf0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1530157280 0x7f153014f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1530156ea0 0x7f1530157b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 --2- 172.21.15.119:0/3677428700 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1530071210 0x7f15300ae480 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 >> 172.21.15.119:0/3677428700 conn(0x7f15300b1500 msgr2=0x7f153014fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:14.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.833+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 shutdown_connections 2024-09-13T16:31:14.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.837+0000 7f1536ceb700 1 -- 172.21.15.119:0/3677428700 wait complete. 2024-09-13T16:31:14.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:31:14.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.965+0000 7f3c71351700 1 Processor -- start 2024-09-13T16:31:14.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.965+0000 7f3c71351700 1 -- start start 2024-09-13T16:31:14.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3c6c0b1860 0x7f3c6c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c0b1dd0 con 0x7f3c6c0ad9a0 2024-09-13T16:31:14.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c6c0b20b0 con 0x7f3c6c06ca60 2024-09-13T16:31:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c6c0b2fd0 con 0x7f3c6c0b1860 2024-09-13T16:31:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c70b50700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3c6c0b1860 0x7f3c6c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c6bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3c6c0ad9a0 0x7f3c6c06c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41214/0 (socket says 172.21.15.119:41214) 2024-09-13T16:31:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c6bfff700 1 -- 172.21.15.119:0/2770144128 learned_addr learned my addr 172.21.15.119:0/2770144128 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:14.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c70b50700 1 -- 172.21.15.119:0/2770144128 >> v1:172.21.15.145:6789/0 conn(0x7f3c6c06ca60 legacy=0x7f3c6c15a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c6affd700 1 -- 172.21.15.119:0/2770144128 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3977768262 0 0) 0x7f3c6c0b1dd0 con 0x7f3c6c0ad9a0 2024-09-13T16:31:14.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c70b50700 1 -- 172.21.15.119:0/2770144128 >> v1:172.21.15.119:6789/0 conn(0x7f3c6c0ad9a0 legacy=0x7f3c6c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c70b50700 1 -- 172.21.15.119:0/2770144128 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c0b2390 con 0x7f3c6c0b1860 2024-09-13T16:31:14.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c70b50700 1 --2- 172.21.15.119:0/2770144128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c0b1860 0x7f3c6c159de0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009850 tx=0x7f3c64009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1f773099167e0fc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c6affd700 1 -- 172.21.15.119:0/2770144128 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64011830 con 0x7f3c6c0b1860 2024-09-13T16:31:14.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c6affd700 1 -- 172.21.15.119:0/2770144128 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c64011990 con 0x7f3c6c0b1860 2024-09-13T16:31:14.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c6affd700 1 -- 172.21.15.119:0/2770144128 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64011d20 con 0x7f3c6c0b1860 2024-09-13T16:31:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- 172.21.15.119:0/2770144128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c0b1860 msgr2=0x7f3c6c159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 --2- 172.21.15.119:0/2770144128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c0b1860 0x7f3c6c159de0 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7f3c64009850 tx=0x7f3c64009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- 172.21.15.119:0/2770144128 shutdown_connections 2024-09-13T16:31:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 --2- 172.21.15.119:0/2770144128 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c0b1860 0x7f3c6c159de0 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- 172.21.15.119:0/2770144128 >> 172.21.15.119:0/2770144128 conn(0x7f3c6c0b0980 msgr2=0x7f3c6c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- 172.21.15.119:0/2770144128 shutdown_connections 2024-09-13T16:31:14.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.969+0000 7f3c71351700 1 -- 172.21.15.119:0/2770144128 wait complete. 2024-09-13T16:31:14.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 Processor -- start 2024-09-13T16:31:14.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 -- start start 2024-09-13T16:31:14.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c06ca60 0x7f3c6c144070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c6c0ad9a0 0x7f3c6c144f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c6c0b1860 0x7f3c6c145440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0ccd90 con 0x7f3c6c06ca60 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0cd5a0 con 0x7f3c6c0ad9a0 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c0cc400 con 0x7f3c6c0b1860 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c06ca60 0x7f3c6c144070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c06ca60 0x7f3c6c144070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34088/0 (socket says 172.21.15.119:34088) 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 -- 172.21.15.119:0/2308291759 learned_addr learned my addr 172.21.15.119:0/2308291759 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c70b50700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c6c0b1860 0x7f3c6c145440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6b7fe700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c6c0ad9a0 0x7f3c6c144f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 -- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c6c0b1860 msgr2=0x7f3c6c145440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c6c0b1860 0x7f3c6c145440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 -- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c6c0ad9a0 msgr2=0x7f3c6c144f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c6c0ad9a0 0x7f3c6c144f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 -- 172.21.15.119:0/2308291759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c640094b0 con 0x7f3c6c06ca60 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c6bfff700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c06ca60 0x7f3c6c144070 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c550 tx=0x7f3c5400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c68ff9700 1 -- 172.21.15.119:0/2308291759 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54004ca0 con 0x7f3c6c06ca60 2024-09-13T16:31:14.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c68ff9700 1 -- 172.21.15.119:0/2308291759 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c54004e00 con 0x7f3c6c06ca60 2024-09-13T16:31:14.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c0ccfe0 con 0x7f3c6c06ca60 2024-09-13T16:31:14.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c68ff9700 1 -- 172.21.15.119:0/2308291759 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54007310 con 0x7f3c6c06ca60 2024-09-13T16:31:14.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.973+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c15b4b0 con 0x7f3c6c06ca60 2024-09-13T16:31:14.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.977+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c6c159de0 con 0x7f3c6c06ca60 2024-09-13T16:31:14.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.977+0000 7f3c68ff9700 1 -- 172.21.15.119:0/2308291759 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3c54007470 con 0x7f3c6c06ca60 2024-09-13T16:31:14.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.977+0000 7f3c68ff9700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c4c066730 0x7f3c4c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:14.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.977+0000 7f3c68ff9700 1 -- 172.21.15.119:0/2308291759 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f3c54083ef0 con 0x7f3c6c06ca60 2024-09-13T16:31:14.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.977+0000 7f3c6b7fe700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c4c066730 0x7f3c4c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:14.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.981+0000 7f3c6b7fe700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c4c066730 0x7f3c4c068bf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ccd20 tx=0x7f3c5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:14.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:14.985+0000 7f3c68ff9700 1 -- 172.21.15.119:0/2308291759 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3c540510c0 con 0x7f3c6c06ca60 2024-09-13T16:31:15.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.157+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3c6c148130 con 0x7f3c6c06ca60 2024-09-13T16:31:15.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.157+0000 7f3c68ff9700 1 -- 172.21.15.119:0/2308291759 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3c54007720 con 0x7f3c6c06ca60 2024-09-13T16:31:15.160 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:31:15.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c4c066730 msgr2=0x7f3c4c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c4c066730 0x7f3c4c068bf0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3c6c0ccd20 tx=0x7f3c5c009040 comp rx=0 tx=0).stop 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c06ca60 msgr2=0x7f3c6c144070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c06ca60 0x7f3c6c144070 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7f3c5400c550 tx=0x7f3c5400c860 comp rx=0 tx=0).stop 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 shutdown_connections 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c4c066730 0x7f3c4c068bf0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c6c0b1860 0x7f3c6c145440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c6c0ad9a0 0x7f3c6c144f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 --2- 172.21.15.119:0/2308291759 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c6c06ca60 0x7f3c6c144070 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 >> 172.21.15.119:0/2308291759 conn(0x7f3c6c0b0980 msgr2=0x7f3c6c0af730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 shutdown_connections 2024-09-13T16:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.161+0000 7f3c71351700 1 -- 172.21.15.119:0/2308291759 wait complete. 2024-09-13T16:31:15.175 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:31:15.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252f59e700 1 Processor -- start 2024-09-13T16:31:15.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252f59e700 1 -- start start 2024-09-13T16:31:15.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530157280 0x7f253006ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2530071210 0x7f253006cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f253006e830 con 0x7f2530156ea0 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252f59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25300b4810 con 0x7f2530071210 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f253006d4e0 con 0x7f2530157280 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2530071210 0x7f253006cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2530071210 0x7f253006cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34098/0 (socket says 172.21.15.119:34098) 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 -- 172.21.15.119:0/4267319172 learned_addr learned my addr 172.21.15.119:0/4267319172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252dd9b700 1 --2- 172.21.15.119:0/4267319172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530157280 0x7f253006ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 -- 172.21.15.119:0/4267319172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530157280 msgr2=0x7f253006ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 --2- 172.21.15.119:0/4267319172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530157280 0x7f253006ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 -- 172.21.15.119:0/4267319172 >> v1:172.21.15.145:6789/0 conn(0x7f2530156ea0 legacy=0x7f2530070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 -- 172.21.15.119:0/4267319172 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253006d790 con 0x7f2530071210 2024-09-13T16:31:15.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252ed9d700 1 --2- 172.21.15.119:0/4267319172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530071210 0x7f253006cfa0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f2528009e80 tx=0x7f252800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b816215c8a4182 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252d59a700 1 -- 172.21.15.119:0/4267319172 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f252800e070 con 0x7f2530071210 2024-09-13T16:31:15.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252d59a700 1 -- 172.21.15.119:0/4267319172 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528003a40 con 0x7f2530071210 2024-09-13T16:31:15.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.285+0000 7f252d59a700 1 -- 172.21.15.119:0/4267319172 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528003df0 con 0x7f2530071210 2024-09-13T16:31:15.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 -- 172.21.15.119:0/4267319172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530071210 msgr2=0x7f253006cfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 --2- 172.21.15.119:0/4267319172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530071210 0x7f253006cfa0 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7f2528009e80 tx=0x7f252800b330 comp rx=0 tx=0).stop 2024-09-13T16:31:15.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 -- 172.21.15.119:0/4267319172 shutdown_connections 2024-09-13T16:31:15.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 --2- 172.21.15.119:0/4267319172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530071210 0x7f253006cfa0 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 --2- 172.21.15.119:0/4267319172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530157280 0x7f253006ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 -- 172.21.15.119:0/4267319172 >> 172.21.15.119:0/4267319172 conn(0x7f25300b1500 msgr2=0x7f25300b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:15.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 -- 172.21.15.119:0/4267319172 shutdown_connections 2024-09-13T16:31:15.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 -- 172.21.15.119:0/4267319172 wait complete. 2024-09-13T16:31:15.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 Processor -- start 2024-09-13T16:31:15.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.289+0000 7f252f59e700 1 -- start start 2024-09-13T16:31:15.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2530071210 0x7f25301e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530156ea0 0x7f25300cc760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530157280 0x7f25301e3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25300ccca0 con 0x7f2530157280 2024-09-13T16:31:15.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25300ccf20 con 0x7f2530071210 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25300cd200 con 0x7f2530156ea0 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530157280 0x7f25301e3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530157280 0x7f25301e3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34114/0 (socket says 172.21.15.119:34114) 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 -- 172.21.15.119:0/3438736471 learned_addr learned my addr 172.21.15.119:0/3438736471 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252e59c700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2530071210 0x7f25301e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252dd9b700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530156ea0 0x7f25300cc760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 -- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530156ea0 msgr2=0x7f25300cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530156ea0 0x7f25300cc760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 -- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2530071210 msgr2=0x7f25301e4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2530071210 0x7f25301e4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 -- 172.21.15.119:0/3438736471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2528009b30 con 0x7f2530157280 2024-09-13T16:31:15.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252dd9b700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530156ea0 0x7f25300cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:15.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252ed9d700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530157280 0x7f25301e3b80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f2528015040 tx=0x7f2528004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f251effd700 1 -- 172.21.15.119:0/3438736471 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528004440 con 0x7f2530157280 2024-09-13T16:31:15.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f251effd700 1 -- 172.21.15.119:0/3438736471 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25280045c0 con 0x7f2530157280 2024-09-13T16:31:15.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25300cd4e0 con 0x7f2530157280 2024-09-13T16:31:15.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f251effd700 1 -- 172.21.15.119:0/3438736471 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2528004970 con 0x7f2530157280 2024-09-13T16:31:15.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.293+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25300cdaa0 con 0x7f2530157280 2024-09-13T16:31:15.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.297+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2530070b00 con 0x7f2530157280 2024-09-13T16:31:15.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.297+0000 7f251effd700 1 -- 172.21.15.119:0/3438736471 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2528012430 con 0x7f2530157280 2024-09-13T16:31:15.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.297+0000 7f251effd700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2510066730 0x7f2510068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.297+0000 7f251effd700 1 -- 172.21.15.119:0/3438736471 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2528086a30 con 0x7f2530157280 2024-09-13T16:31:15.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.297+0000 7f252e59c700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2510066730 0x7f2510068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.297+0000 7f252e59c700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2510066730 0x7f2510068bf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2524004f70 tx=0x7f2524009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.301+0000 7f251effd700 1 -- 172.21.15.119:0/3438736471 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2528053c10 con 0x7f2530157280 2024-09-13T16:31:15.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.465+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f25300b4290 con 0x7f2530157280 2024-09-13T16:31:15.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.465+0000 7f251effd700 1 -- 172.21.15.119:0/3438736471 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2528017030 con 0x7f2530157280 2024-09-13T16:31:15.471 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:31:15.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2510066730 msgr2=0x7f2510068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2510066730 0x7f2510068bf0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f2524004f70 tx=0x7f2524009040 comp rx=0 tx=0).stop 2024-09-13T16:31:15.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530157280 msgr2=0x7f25301e3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530157280 0x7f25301e3b80 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7f2528015040 tx=0x7f2528004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 shutdown_connections 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2510066730 0x7f2510068bf0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2530157280 0x7f25301e3b80 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2530156ea0 0x7f25300cc760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 --2- 172.21.15.119:0/3438736471 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2530071210 0x7f25301e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 >> 172.21.15.119:0/3438736471 conn(0x7f25300b1500 msgr2=0x7f25300af610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 shutdown_connections 2024-09-13T16:31:15.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.469+0000 7f252f59e700 1 -- 172.21.15.119:0/3438736471 wait complete. 2024-09-13T16:31:15.485 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:15.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.593+0000 7f69b359e700 1 Processor -- start 2024-09-13T16:31:15.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.593+0000 7f69b359e700 1 -- start start 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f69b40703d0 0x7f69b40b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f69b40707b0 0x7f69b406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b406b8d0 0x7f69b406d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b406c360 con 0x7f69b40707b0 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b406d180 con 0x7f69b40703d0 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b406bd30 con 0x7f69b406b8d0 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b1d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f69b40707b0 0x7f69b406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b2d9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b406b8d0 0x7f69b406d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b1d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f69b40707b0 0x7f69b406b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34120/0 (socket says 172.21.15.119:34120) 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b259c700 1 --2- 172.21.15.119:0/604206521 >> v2:172.21.15.145:3300/0 conn(0x7f69b40703d0 0x7f69b40b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b1d9b700 1 -- 172.21.15.119:0/604206521 learned_addr learned my addr 172.21.15.119:0/604206521 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:15.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b2d9d700 1 -- 172.21.15.119:0/604206521 >> v2:172.21.15.145:3300/0 conn(0x7f69b40703d0 msgr2=0x7f69b40b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b2d9d700 1 --2- 172.21.15.119:0/604206521 >> v2:172.21.15.145:3300/0 conn(0x7f69b40703d0 0x7f69b40b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b2d9d700 1 -- 172.21.15.119:0/604206521 >> v2:172.21.15.119:3300/0 conn(0x7f69b40707b0 msgr2=0x7f69b406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b2d9d700 1 --2- 172.21.15.119:0/604206521 >> v2:172.21.15.119:3300/0 conn(0x7f69b40707b0 0x7f69b406b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b2d9d700 1 -- 172.21.15.119:0/604206521 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b406c570 con 0x7f69b406b8d0 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b1d9b700 1 --2- 172.21.15.119:0/604206521 >> v2:172.21.15.119:3300/0 conn(0x7f69b40707b0 0x7f69b406b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b2d9d700 1 --2- 172.21.15.119:0/604206521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b406b8d0 0x7f69b406d610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f69b40cef30 tx=0x7f69ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd056c9e6a2a56fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b159a700 1 -- 172.21.15.119:0/604206521 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac00e070 con 0x7f69b406b8d0 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b159a700 1 -- 172.21.15.119:0/604206521 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac00bb20 con 0x7f69b406b8d0 2024-09-13T16:31:15.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b159a700 1 -- 172.21.15.119:0/604206521 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac012430 con 0x7f69b406b8d0 2024-09-13T16:31:15.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- 172.21.15.119:0/604206521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b406b8d0 msgr2=0x7f69b406d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 --2- 172.21.15.119:0/604206521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b406b8d0 0x7f69b406d610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f69b40cef30 tx=0x7f69ac00b330 comp rx=0 tx=0).stop 2024-09-13T16:31:15.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- 172.21.15.119:0/604206521 shutdown_connections 2024-09-13T16:31:15.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 --2- 172.21.15.119:0/604206521 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b406b8d0 0x7f69b406d610 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 --2- 172.21.15.119:0/604206521 >> v2:172.21.15.119:3300/0 conn(0x7f69b40707b0 0x7f69b406b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 --2- 172.21.15.119:0/604206521 >> v2:172.21.15.145:3300/0 conn(0x7f69b40703d0 0x7f69b40b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- 172.21.15.119:0/604206521 >> 172.21.15.119:0/604206521 conn(0x7f69b40ae750 msgr2=0x7f69b40b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:15.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- 172.21.15.119:0/604206521 shutdown_connections 2024-09-13T16:31:15.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.597+0000 7f69b359e700 1 -- 172.21.15.119:0/604206521 wait complete. 2024-09-13T16:31:15.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.601+0000 7f69b359e700 1 Processor -- start 2024-09-13T16:31:15.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.601+0000 7f69b359e700 1 -- start start 2024-09-13T16:31:15.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b40703d0 0x7f69b4142c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b40707b0 0x7f69b413ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b4141710 0x7f69b4159870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b4159510 con 0x7f69b4141710 2024-09-13T16:31:15.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b4140440 con 0x7f69b40703d0 2024-09-13T16:31:15.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b4140670 con 0x7f69b40707b0 2024-09-13T16:31:15.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b4141710 0x7f69b4159870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b4141710 0x7f69b4159870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34128/0 (socket says 172.21.15.119:34128) 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 -- 172.21.15.119:0/3041893172 learned_addr learned my addr 172.21.15.119:0/3041893172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b1d9b700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b40707b0 0x7f69b413ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b259c700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b40703d0 0x7f69b4142c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 -- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b40707b0 msgr2=0x7f69b413ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b40707b0 0x7f69b413ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 -- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b40703d0 msgr2=0x7f69b4142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b40703d0 0x7f69b4142c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 -- 172.21.15.119:0/3041893172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69a8009570 con 0x7f69b4141710 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b259c700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b40703d0 0x7f69b4142c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b2d9d700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b4141710 0x7f69b4159870 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f69ac014040 tx=0x7f69ac004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69a2ffd700 1 -- 172.21.15.119:0/3041893172 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac020070 con 0x7f69b4141710 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69a2ffd700 1 -- 172.21.15.119:0/3041893172 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac0098f0 con 0x7f69b4141710 2024-09-13T16:31:15.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac009a70 con 0x7f69b4141710 2024-09-13T16:31:15.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69a2ffd700 1 -- 172.21.15.119:0/3041893172 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac004640 con 0x7f69b4141710 2024-09-13T16:31:15.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b4140c30 con 0x7f69b4141710 2024-09-13T16:31:15.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.605+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b413fcf0 con 0x7f69b4141710 2024-09-13T16:31:15.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.609+0000 7f69a2ffd700 1 -- 172.21.15.119:0/3041893172 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f69ac024070 con 0x7f69b4141710 2024-09-13T16:31:15.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.609+0000 7f69a2ffd700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6994066730 0x7f6994068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.609+0000 7f69a2ffd700 1 -- 172.21.15.119:0/3041893172 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f69ac086fa0 con 0x7f69b4141710 2024-09-13T16:31:15.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.609+0000 7f69b259c700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6994066730 0x7f6994068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.609+0000 7f69b259c700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6994066730 0x7f6994068bf0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f69b409cac0 tx=0x7f69a4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.617+0000 7f69a2ffd700 1 -- 172.21.15.119:0/3041893172 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f69ac0b8db0 con 0x7f69b4141710 2024-09-13T16:31:15.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.813+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69b406e010 con 0x7f69b4141710 2024-09-13T16:31:15.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69a2ffd700 1 -- 172.21.15.119:0/3041893172 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f69ac02b090 con 0x7f69b4141710 2024-09-13T16:31:15.820 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:15.820 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:30:40.052433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6994066730 msgr2=0x7f6994068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6994066730 0x7f6994068bf0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f69b409cac0 tx=0x7f69a4008040 comp rx=0 tx=0).stop 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b4141710 msgr2=0x7f69b4159870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b4141710 0x7f69b4159870 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 crypto rx=0x7f69ac014040 tx=0x7f69ac004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 shutdown_connections 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6994066730 0x7f6994068bf0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b4141710 0x7f69b4159870 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b40707b0 0x7f69b413ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 --2- 172.21.15.119:0/3041893172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b40703d0 0x7f69b4142c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 >> 172.21.15.119:0/3041893172 conn(0x7f69b406b8d0 msgr2=0x7f69b40b27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:15.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 shutdown_connections 2024-09-13T16:31:15.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.817+0000 7f69b359e700 1 -- 172.21.15.119:0/3041893172 wait complete. 2024-09-13T16:31:15.824 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 49 2024-09-13T16:31:15.835 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0f0f1b700 1 Processor -- start 2024-09-13T16:31:15.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0f0f1b700 1 -- start start 2024-09-13T16:31:15.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0f0f1b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0ec0cd8d0 0x7fb0ec06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0f0f1b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0f0f1b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0ec0705d0 con 0x7fb0ec0b2200 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0f0f1b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0ec06d0d0 con 0x7fb0ec0cd8d0 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0f0f1b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec06fca0 con 0x7fb0ec0b17f0 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eb7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0ec0cd8d0 0x7fb0ec06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eb7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0ec0cd8d0 0x7fb0ec06cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34144/0 (socket says 172.21.15.119:34144) 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eb7fe700 1 -- 172.21.15.119:0/3477891304 learned_addr learned my addr 172.21.15.119:0/3477891304 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eaffd700 1 --2- 172.21.15.119:0/3477891304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eb7fe700 1 -- 172.21.15.119:0/3477891304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 msgr2=0x7fb0ec0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eb7fe700 1 --2- 172.21.15.119:0/3477891304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eb7fe700 1 -- 172.21.15.119:0/3477891304 >> v1:172.21.15.145:6789/0 conn(0x7fb0ec0b2200 legacy=0x7fb0ec070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.941+0000 7fb0eb7fe700 1 -- 172.21.15.119:0/3477891304 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ec06ff80 con 0x7fb0ec0cd8d0 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0eb7fe700 1 --2- 172.21.15.119:0/3477891304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0cd8d0 0x7fb0ec06cb90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009910 tx=0x7fb0dc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e73a1bae12f9c99d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0ea7fc700 1 -- 172.21.15.119:0/3477891304 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc011830 con 0x7fb0ec0cd8d0 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0ea7fc700 1 -- 172.21.15.119:0/3477891304 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0dc011990 con 0x7fb0ec0cd8d0 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0ea7fc700 1 -- 172.21.15.119:0/3477891304 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc011d40 con 0x7fb0ec0cd8d0 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/3477891304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0cd8d0 msgr2=0x7fb0ec06cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/3477891304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0cd8d0 0x7fb0ec06cb90 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009910 tx=0x7fb0dc009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/3477891304 shutdown_connections 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/3477891304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/3477891304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0cd8d0 0x7fb0ec06cb90 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/3477891304 >> 172.21.15.119:0/3477891304 conn(0x7fb0ec06a350 msgr2=0x7fb0ec06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:15.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/3477891304 shutdown_connections 2024-09-13T16:31:15.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/3477891304 wait complete. 2024-09-13T16:31:15.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 Processor -- start 2024-09-13T16:31:15.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.945+0000 7fb0f0f1b700 1 -- start start 2024-09-13T16:31:15.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0aa970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0b2200 0x7fb0ec0a9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ec0cd8d0 0x7fb0ec0ca070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec14f730 con 0x7fb0ec0b2200 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec14e860 con 0x7fb0ec0cd8d0 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ec14eab0 con 0x7fb0ec0b17f0 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0b2200 0x7fb0ec0a9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0b2200 0x7fb0ec0a9ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34156/0 (socket says 172.21.15.119:34156) 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 -- 172.21.15.119:0/2209732753 learned_addr learned my addr 172.21.15.119:0/2209732753 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eb7fe700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0aa970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0ebfff700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ec0cd8d0 0x7fb0ec0ca070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 -- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 msgr2=0x7fb0ec0aa970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0aa970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 -- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ec0cd8d0 msgr2=0x7fb0ec0ca070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ec0cd8d0 0x7fb0ec0ca070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 -- 172.21.15.119:0/2209732753 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc009570 con 0x7fb0ec0b2200 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0eaffd700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0b2200 0x7fb0ec0a9ec0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000e9a0 tx=0x7fb0e000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0d3fff700 1 -- 172.21.15.119:0/2209732753 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e0009720 con 0x7fb0ec0b2200 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0d3fff700 1 -- 172.21.15.119:0/2209732753 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e000c690 con 0x7fb0ec0b2200 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0d3fff700 1 -- 172.21.15.119:0/2209732753 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e000ca20 con 0x7fb0ec0b2200 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ec14ed90 con 0x7fb0ec0b2200 2024-09-13T16:31:15.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ec14f350 con 0x7fb0ec0b2200 2024-09-13T16:31:15.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ec0cdcb0 con 0x7fb0ec0b2200 2024-09-13T16:31:15.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.949+0000 7fb0d3fff700 1 -- 172.21.15.119:0/2209732753 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb0e000cb80 con 0x7fb0ec0b2200 2024-09-13T16:31:15.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.953+0000 7fb0d3fff700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb0d8066730 0x7fb0d8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:15.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.953+0000 7fb0d3fff700 1 -- 172.21.15.119:0/2209732753 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fb0e00842a0 con 0x7fb0ec0b2200 2024-09-13T16:31:15.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.953+0000 7fb0eb7fe700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb0d8066730 0x7fb0d8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:15.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.953+0000 7fb0eb7fe700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb0d8066730 0x7fb0d8068bf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009c60 tx=0x7fb0dc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:15.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:15.957+0000 7fb0d3fff700 1 -- 172.21.15.119:0/2209732753 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb0e0051500 con 0x7fb0ec0b2200 2024-09-13T16:31:16.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.117+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0ec0b3160 con 0x7fb0ec0b2200 2024-09-13T16:31:16.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0d3fff700 1 -- 172.21.15.119:0/2209732753 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16966 (secure 0 0 0) 0x7fb0e0054b20 con 0x7fb0ec0b2200 2024-09-13T16:31:16.124 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:16.124 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":83,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:48.943212+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/491709455":"2024-09-14T16:29:48.943151+0000","172.21.15.119:6834/491709455":"2024-09-14T16:29:48.943151+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:16.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb0d8066730 msgr2=0x7fb0d8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb0d8066730 0x7fb0d8068bf0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc009c60 tx=0x7fb0dc008030 comp rx=0 tx=0).stop 2024-09-13T16:31:16.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0b2200 msgr2=0x7fb0ec0a9ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0b2200 0x7fb0ec0a9ec0 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fb0e000e9a0 tx=0x7fb0e000ecb0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 shutdown_connections 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb0d8066730 0x7fb0d8068bf0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb0ec0cd8d0 0x7fb0ec0ca070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0ec0b2200 0x7fb0ec0a9ec0 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 --2- 172.21.15.119:0/2209732753 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0ec0b17f0 0x7fb0ec0aa970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 >> 172.21.15.119:0/2209732753 conn(0x7fb0ec06a350 msgr2=0x7fb0ec161340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 shutdown_connections 2024-09-13T16:31:16.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.121+0000 7fb0f0f1b700 1 -- 172.21.15.119:0/2209732753 wait complete. 2024-09-13T16:31:16.139 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:31:16.139 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:16.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 Processor -- start 2024-09-13T16:31:16.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 -- start start 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fed700cd8d0 0x7fed7006cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fed700b17f0 0x7fed700cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700b2200 0x7fed700b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed700b2bb0 con 0x7fed700cd8d0 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed700705d0 con 0x7fed700b17f0 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed768ed700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7006d0d0 con 0x7fed700b2200 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed758eb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fed700cd8d0 0x7fed7006cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed758eb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fed700cd8d0 0x7fed7006cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34162/0 (socket says 172.21.15.119:34162) 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed758eb700 1 -- 172.21.15.119:0/1601374081 learned_addr learned my addr 172.21.15.119:0/1601374081 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed750ea700 1 --2- 172.21.15.119:0/1601374081 >> v2:172.21.15.145:3300/0 conn(0x7fed700b17f0 0x7fed700cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.245+0000 7fed760ec700 1 --2- 172.21.15.119:0/1601374081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700b2200 0x7fed700b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed758eb700 1 -- 172.21.15.119:0/1601374081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700b2200 msgr2=0x7fed700b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed758eb700 1 --2- 172.21.15.119:0/1601374081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700b2200 0x7fed700b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed758eb700 1 -- 172.21.15.119:0/1601374081 >> v2:172.21.15.145:3300/0 conn(0x7fed700b17f0 msgr2=0x7fed700cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed758eb700 1 --2- 172.21.15.119:0/1601374081 >> v2:172.21.15.145:3300/0 conn(0x7fed700b17f0 0x7fed700cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed758eb700 1 -- 172.21.15.119:0/1601374081 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7006fca0 con 0x7fed700cd8d0 2024-09-13T16:31:16.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed758eb700 1 --2- 172.21.15.119:0/1601374081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700cd8d0 0x7fed7006cb90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fed700ae2c0 tx=0x7fed68009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68560c7bb1448177 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed748e9700 1 -- 172.21.15.119:0/1601374081 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68011830 con 0x7fed700cd8d0 2024-09-13T16:31:16.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed748e9700 1 -- 172.21.15.119:0/1601374081 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed68011990 con 0x7fed700cd8d0 2024-09-13T16:31:16.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed748e9700 1 -- 172.21.15.119:0/1601374081 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68011d20 con 0x7fed700cd8d0 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 -- 172.21.15.119:0/1601374081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700cd8d0 msgr2=0x7fed7006cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 --2- 172.21.15.119:0/1601374081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700cd8d0 0x7fed7006cb90 secure :-1 s=READY pgs=1188 cs=0 l=1 rev1=1 crypto rx=0x7fed700ae2c0 tx=0x7fed68009bf0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 -- 172.21.15.119:0/1601374081 shutdown_connections 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 --2- 172.21.15.119:0/1601374081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700b2200 0x7fed700b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 --2- 172.21.15.119:0/1601374081 >> v2:172.21.15.145:3300/0 conn(0x7fed700b17f0 0x7fed700cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 --2- 172.21.15.119:0/1601374081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700cd8d0 0x7fed7006cb90 unknown :-1 s=CLOSED pgs=1188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 -- 172.21.15.119:0/1601374081 >> 172.21.15.119:0/1601374081 conn(0x7fed7006a350 msgr2=0x7fed7006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 -- 172.21.15.119:0/1601374081 shutdown_connections 2024-09-13T16:31:16.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 -- 172.21.15.119:0/1601374081 wait complete. 2024-09-13T16:31:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 Processor -- start 2024-09-13T16:31:16.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.249+0000 7fed768ed700 1 -- start start 2024-09-13T16:31:16.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700b17f0 0x7fed7015ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700b2200 0x7fed700acc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700cd8d0 0x7fed700ad170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed700aac00 con 0x7fed700b17f0 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7015cf60 con 0x7fed700b2200 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed700a9c40 con 0x7fed700cd8d0 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700b17f0 0x7fed7015ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700b17f0 0x7fed7015ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34176/0 (socket says 172.21.15.119:34176) 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 -- 172.21.15.119:0/4021650648 learned_addr learned my addr 172.21.15.119:0/4021650648 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed750ea700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700b2200 0x7fed700acc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed760ec700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700cd8d0 0x7fed700ad170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 -- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700cd8d0 msgr2=0x7fed700ad170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700cd8d0 0x7fed700ad170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 -- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700b2200 msgr2=0x7fed700acc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700b2200 0x7fed700acc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 -- 172.21.15.119:0/4021650648 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68009570 con 0x7fed700b17f0 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed758eb700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700b17f0 0x7fed7015ca20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fed68000c00 tx=0x7fed68004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed6e7fc700 1 -- 172.21.15.119:0/4021650648 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed68004590 con 0x7fed700b17f0 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed6e7fc700 1 -- 172.21.15.119:0/4021650648 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6801e070 con 0x7fed700b17f0 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed6e7fc700 1 -- 172.21.15.119:0/4021650648 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680038e0 con 0x7fed700b17f0 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed700a90c0 con 0x7fed700b17f0 2024-09-13T16:31:16.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed700a9680 con 0x7fed700b17f0 2024-09-13T16:31:16.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed700ae2f0 con 0x7fed700b17f0 2024-09-13T16:31:16.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.253+0000 7fed6e7fc700 1 -- 172.21.15.119:0/4021650648 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fed68003a40 con 0x7fed700b17f0 2024-09-13T16:31:16.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.257+0000 7fed6e7fc700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed5c066760 0x7fed5c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.257+0000 7fed6e7fc700 1 -- 172.21.15.119:0/4021650648 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fed68085060 con 0x7fed700b17f0 2024-09-13T16:31:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.257+0000 7fed750ea700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed5c066760 0x7fed5c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.257+0000 7fed750ea700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed5c066760 0x7fed5c068c20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fed700ca9b0 tx=0x7fed6400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.261+0000 7fed6e7fc700 1 -- 172.21.15.119:0/4021650648 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fed680522c0 con 0x7fed700b17f0 2024-09-13T16:31:16.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.469+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed700cdcb0 con 0x7fed700b17f0 2024-09-13T16:31:16.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed6e7fc700 1 -- 172.21.15.119:0/4021650648 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4744 (secure 0 0 0) 0x7fed680558e0 con 0x7fed700b17f0 2024-09-13T16:31:16.475 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:16.475 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:30:40.052433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:31:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed5c066760 msgr2=0x7fed5c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed5c066760 0x7fed5c068c20 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fed700ca9b0 tx=0x7fed6400b040 comp rx=0 tx=0).stop 2024-09-13T16:31:16.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700b17f0 msgr2=0x7fed7015ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700b17f0 0x7fed7015ca20 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 crypto rx=0x7fed68000c00 tx=0x7fed68004770 comp rx=0 tx=0).stop 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 shutdown_connections 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fed5c066760 0x7fed5c068c20 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fed700cd8d0 0x7fed700ad170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fed700b2200 0x7fed700acc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 --2- 172.21.15.119:0/4021650648 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fed700b17f0 0x7fed7015ca20 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 >> 172.21.15.119:0/4021650648 conn(0x7fed7006a350 msgr2=0x7fed70164630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 shutdown_connections 2024-09-13T16:31:16.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.473+0000 7fed768ed700 1 -- 172.21.15.119:0/4021650648 wait complete. 2024-09-13T16:31:16.479 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 49 2024-09-13T16:31:16.489 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:16.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 Processor -- start 2024-09-13T16:31:16.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- start start 2024-09-13T16:31:16.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc15cc40 0x7f98fc0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98fc0cd860 0x7f98fc08bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f98fc08c4e0 0x7f98fc160720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc08c990 con 0x7f98fc0cd860 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98fc0b44c0 con 0x7f98fc08c4e0 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc15d020 con 0x7f98fc15cc40 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98fc0cd860 0x7f98fc08bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98fc0cd860 0x7f98fc08bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34178/0 (socket says 172.21.15.119:34178) 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 -- 172.21.15.119:0/1145376290 learned_addr learned my addr 172.21.15.119:0/1145376290 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9902a68700 1 --2- 172.21.15.119:0/1145376290 >> v2:172.21.15.145:3300/0 conn(0x7f98fc08c4e0 0x7f98fc160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9902267700 1 --2- 172.21.15.119:0/1145376290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc15cc40 0x7f98fc0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 -- 172.21.15.119:0/1145376290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc15cc40 msgr2=0x7f98fc0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 --2- 172.21.15.119:0/1145376290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc15cc40 0x7f98fc0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 -- 172.21.15.119:0/1145376290 >> v2:172.21.15.145:3300/0 conn(0x7f98fc08c4e0 msgr2=0x7f98fc160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 --2- 172.21.15.119:0/1145376290 >> v2:172.21.15.145:3300/0 conn(0x7f98fc08c4e0 0x7f98fc160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 -- 172.21.15.119:0/1145376290 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98fc0d0720 con 0x7f98fc0cd860 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9902267700 1 --2- 172.21.15.119:0/1145376290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc15cc40 0x7f98fc0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9902a68700 1 --2- 172.21.15.119:0/1145376290 >> v2:172.21.15.145:3300/0 conn(0x7f98fc08c4e0 0x7f98fc160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901a66700 1 --2- 172.21.15.119:0/1145376290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc0cd860 0x7f98fc08bfa0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f98fc08b1f0 tx=0x7f98f0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5291ba6e70b3a4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901265700 1 -- 172.21.15.119:0/1145376290 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f000ba20 con 0x7f98fc0cd860 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901265700 1 -- 172.21.15.119:0/1145376290 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f000bb80 con 0x7f98fc0cd860 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9901265700 1 -- 172.21.15.119:0/1145376290 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0009e60 con 0x7f98fc0cd860 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- 172.21.15.119:0/1145376290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc0cd860 msgr2=0x7f98fc08bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 --2- 172.21.15.119:0/1145376290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc0cd860 0x7f98fc08bfa0 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 crypto rx=0x7f98fc08b1f0 tx=0x7f98f0009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- 172.21.15.119:0/1145376290 shutdown_connections 2024-09-13T16:31:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 --2- 172.21.15.119:0/1145376290 >> v2:172.21.15.145:3300/0 conn(0x7f98fc08c4e0 0x7f98fc160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 --2- 172.21.15.119:0/1145376290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc0cd860 0x7f98fc08bfa0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 --2- 172.21.15.119:0/1145376290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc15cc40 0x7f98fc0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- 172.21.15.119:0/1145376290 >> 172.21.15.119:0/1145376290 conn(0x7f98fc0b4640 msgr2=0x7f98fc0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- 172.21.15.119:0/1145376290 shutdown_connections 2024-09-13T16:31:16.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.597+0000 7f9903269700 1 -- 172.21.15.119:0/1145376290 wait complete. 2024-09-13T16:31:16.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 Processor -- start 2024-09-13T16:31:16.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 -- start start 2024-09-13T16:31:16.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc08c4e0 0x7f98fc0cb650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98fc0cd860 0x7f98fc0ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc15cc40 0x7f98fc15e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc0ca380 con 0x7f98fc15cc40 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc15e7e0 con 0x7f98fc0cd860 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98fc15ea20 con 0x7f98fc08c4e0 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc15cc40 0x7f98fc15e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc15cc40 0x7f98fc15e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34192/0 (socket says 172.21.15.119:34192) 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 -- 172.21.15.119:0/4046710655 learned_addr learned my addr 172.21.15.119:0/4046710655 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9901a66700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98fc0cd860 0x7f98fc0ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902267700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc08c4e0 0x7f98fc0cb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 -- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc08c4e0 msgr2=0x7f98fc0cb650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc08c4e0 0x7f98fc0cb650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 -- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98fc0cd860 msgr2=0x7f98fc0ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98fc0cd860 0x7f98fc0ca7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 -- 172.21.15.119:0/4046710655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f0009570 con 0x7f98fc15cc40 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9902a68700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc15cc40 0x7f98fc15e2a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f98f400cd10 tx=0x7f98f400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f98eeffd700 1 -- 172.21.15.119:0/4046710655 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f400ad10 con 0x7f98fc15cc40 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f98eeffd700 1 -- 172.21.15.119:0/4046710655 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f400ae90 con 0x7f98fc15cc40 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98fc0c9cc0 con 0x7f98fc15cc40 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f98eeffd700 1 -- 172.21.15.119:0/4046710655 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f400f6c0 con 0x7f98fc15cc40 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.601+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98fc15ed00 con 0x7f98fc15cc40 2024-09-13T16:31:16.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.605+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98fc0ca4c0 con 0x7f98fc15cc40 2024-09-13T16:31:16.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.605+0000 7f98eeffd700 1 -- 172.21.15.119:0/4046710655 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f98f400b040 con 0x7f98fc15cc40 2024-09-13T16:31:16.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.605+0000 7f98eeffd700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98e0066730 0x7f98e0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.605+0000 7f98eeffd700 1 -- 172.21.15.119:0/4046710655 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f98f4084860 con 0x7f98fc15cc40 2024-09-13T16:31:16.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.605+0000 7f9902267700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98e0066730 0x7f98e0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.609+0000 7f9902267700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98e0066730 0x7f98e0068bf0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f98f800ab60 tx=0x7f98f8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.613+0000 7f98eeffd700 1 -- 172.21.15.119:0/4046710655 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f98fc0ca4c0 con 0x7f98fc15cc40 2024-09-13T16:31:16.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.817+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98fc0ca4c0 con 0x7f98fc15cc40 2024-09-13T16:31:16.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.817+0000 7f98eeffd700 1 -- 172.21.15.119:0/4046710655 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4744 (secure 0 0 0) 0x7f98fc0ca4c0 con 0x7f98fc15cc40 2024-09-13T16:31:16.821 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:16.821 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":49,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:29:30.227373+0000","modified":"2024-09-13T16:30:40.052433+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7341":{"gid":7341,"name":"c","rank":0,"incarnation":44,"state":"up:active","state_seq":16,"addr":"172.21.15.119:6835/2534803404","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2534803404},{"type":"v1","addr":"172.21.15.119:6835","nonce":2534803404}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98e0066730 msgr2=0x7f98e0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98e0066730 0x7f98e0068bf0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f98f800ab60 tx=0x7f98f8008040 comp rx=0 tx=0).stop 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc15cc40 msgr2=0x7f98fc15e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc15cc40 0x7f98fc15e2a0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f98f400cd10 tx=0x7f98f400a560 comp rx=0 tx=0).stop 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 shutdown_connections 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98e0066730 0x7f98e0068bf0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98fc15cc40 0x7f98fc15e2a0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98fc0cd860 0x7f98fc0ca7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 --2- 172.21.15.119:0/4046710655 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98fc08c4e0 0x7f98fc0cb650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 >> 172.21.15.119:0/4046710655 conn(0x7f98fc0b4640 msgr2=0x7f98fc0c9320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:16.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 shutdown_connections 2024-09-13T16:31:16.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.821+0000 7f9903269700 1 -- 172.21.15.119:0/4046710655 wait complete. 2024-09-13T16:31:16.825 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 49 2024-09-13T16:31:16.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:16.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3342ee0700 1 Processor -- start 2024-09-13T16:31:16.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3342ee0700 1 -- start start 2024-09-13T16:31:16.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3342ee0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c0adb30 0x7f333c0cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3342ee0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f333c0ce040 0x7f333c0b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3342ee0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f333c06cf60 con 0x7f333c06bb90 2024-09-13T16:31:16.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3342ee0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f333c06c350 con 0x7f333c0ce040 2024-09-13T16:31:16.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3342ee0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333c06c630 con 0x7f333c0adb30 2024-09-13T16:31:16.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f333c0ce040 0x7f333c0b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f333c0ce040 0x7f333c0b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34200/0 (socket says 172.21.15.119:34200) 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 -- 172.21.15.119:0/466381273 learned_addr learned my addr 172.21.15.119:0/466381273 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3341ede700 1 --2- 172.21.15.119:0/466381273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c0adb30 0x7f333c0cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 -- 172.21.15.119:0/466381273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c0adb30 msgr2=0x7f333c0cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 --2- 172.21.15.119:0/466381273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c0adb30 0x7f333c0cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 -- 172.21.15.119:0/466381273 >> v1:172.21.15.145:6789/0 conn(0x7f333c06bb90 legacy=0x7f333c0cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 -- 172.21.15.119:0/466381273 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c06c910 con 0x7f333c0ce040 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f33426df700 1 --2- 172.21.15.119:0/466381273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0ce040 0x7f333c0b2380 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f3334009940 tx=0x7f3334009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54b4ea1d901a2283 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3340edc700 1 -- 172.21.15.119:0/466381273 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333400ba20 con 0x7f333c0ce040 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3340edc700 1 -- 172.21.15.119:0/466381273 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3334009e90 con 0x7f333c0ce040 2024-09-13T16:31:16.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.949+0000 7f3340edc700 1 -- 172.21.15.119:0/466381273 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333400bda0 con 0x7f333c0ce040 2024-09-13T16:31:16.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 -- 172.21.15.119:0/466381273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0ce040 msgr2=0x7f333c0b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 --2- 172.21.15.119:0/466381273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0ce040 0x7f333c0b2380 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 crypto rx=0x7f3334009940 tx=0x7f3334009c50 comp rx=0 tx=0).stop 2024-09-13T16:31:16.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 -- 172.21.15.119:0/466381273 shutdown_connections 2024-09-13T16:31:16.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 --2- 172.21.15.119:0/466381273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0ce040 0x7f333c0b2380 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 --2- 172.21.15.119:0/466381273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c0adb30 0x7f333c0cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 -- 172.21.15.119:0/466381273 >> 172.21.15.119:0/466381273 conn(0x7f333c06a350 msgr2=0x7f333c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:16.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 -- 172.21.15.119:0/466381273 shutdown_connections 2024-09-13T16:31:16.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 -- 172.21.15.119:0/466381273 wait complete. 2024-09-13T16:31:16.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 Processor -- start 2024-09-13T16:31:16.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.953+0000 7f3342ee0700 1 -- start start 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c06bb90 0x7f333c15c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0adb30 0x7f333c153c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f333c0ce040 0x7f333c0b2db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c1541b0 con 0x7f333c0adb30 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f333c15b130 con 0x7f333c0ce040 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f333c15b410 con 0x7f333c06bb90 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0adb30 0x7f333c153c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3341ede700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c06bb90 0x7f333c15c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0adb30 0x7f333c153c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34212/0 (socket says 172.21.15.119:34212) 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33426df700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f333c0ce040 0x7f333c0b2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 -- 172.21.15.119:0/634129233 learned_addr learned my addr 172.21.15.119:0/634129233 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:16.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 -- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c06bb90 msgr2=0x7f333c15c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c06bb90 0x7f333c15c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 -- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f333c0ce040 msgr2=0x7f333c0b2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:16.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f333c0ce040 0x7f333c0b2db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:16.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 -- 172.21.15.119:0/634129233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3334009570 con 0x7f333c0adb30 2024-09-13T16:31:16.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33426df700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f333c0ce040 0x7f333c0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:16.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f33416dd700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0adb30 0x7f333c153c70 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f332c001ed0 tx=0x7f332c0021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f333a7fc700 1 -- 172.21.15.119:0/634129233 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c003c20 con 0x7f333c0adb30 2024-09-13T16:31:16.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f333a7fc700 1 -- 172.21.15.119:0/634129233 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f332c003d80 con 0x7f333c0adb30 2024-09-13T16:31:16.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f333c15b6f0 con 0x7f333c0adb30 2024-09-13T16:31:16.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f333a7fc700 1 -- 172.21.15.119:0/634129233 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f332c00ab40 con 0x7f333c0adb30 2024-09-13T16:31:16.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.957+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f333c15bcb0 con 0x7f333c0adb30 2024-09-13T16:31:16.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.961+0000 7f333a7fc700 1 -- 172.21.15.119:0/634129233 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f332c00aca0 con 0x7f333c0adb30 2024-09-13T16:31:16.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.961+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f333c15bed0 con 0x7f333c0adb30 2024-09-13T16:31:16.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.961+0000 7f333a7fc700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3328066730 0x7f3328068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:16.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.961+0000 7f333a7fc700 1 -- 172.21.15.119:0/634129233 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f332c0842e0 con 0x7f333c0adb30 2024-09-13T16:31:16.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.961+0000 7f3341ede700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3328066730 0x7f3328068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:16.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.961+0000 7f3341ede700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3328066730 0x7f3328068bf0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f333c13d0d0 tx=0x7f3330008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:16.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:16.973+0000 7f333a7fc700 1 -- 172.21.15.119:0/634129233 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f332c051540 con 0x7f333c0adb30 2024-09-13T16:31:17.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.137+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f333c0cc060 con 0x7f333c0adb30 2024-09-13T16:31:17.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.141+0000 7f333a7fc700 1 -- 172.21.15.119:0/634129233 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+16966 (secure 0 0 0) 0x7f332c054b60 con 0x7f333c0adb30 2024-09-13T16:31:17.146 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:17.146 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":83,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:29:48.943212+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:29:28.552550+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_data","create_time":"2024-09-13T16:29:29.210253+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/491709455":"2024-09-14T16:29:48.943151+0000","172.21.15.119:6834/491709455":"2024-09-14T16:29:48.943151+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3328066730 msgr2=0x7f3328068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3328066730 0x7f3328068bf0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f333c13d0d0 tx=0x7f3330008040 comp rx=0 tx=0).stop 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0adb30 msgr2=0x7f333c153c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0adb30 0x7f333c153c70 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f332c001ed0 tx=0x7f332c0021e0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 shutdown_connections 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3328066730 0x7f3328068bf0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f333c0ce040 0x7f333c0b2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f333c0adb30 0x7f333c153c70 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 --2- 172.21.15.119:0/634129233 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f333c06bb90 0x7f333c15c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 >> 172.21.15.119:0/634129233 conn(0x7f333c06a350 msgr2=0x7f333c0aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:17.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 shutdown_connections 2024-09-13T16:31:17.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.145+0000 7f3342ee0700 1 -- 172.21.15.119:0/634129233 wait complete. 2024-09-13T16:31:17.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:31:17.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 Processor -- start 2024-09-13T16:31:17.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- start start 2024-09-13T16:31:17.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27f40cef40 0x7f27f40d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40d0f60 0x7f27f415cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27f415c750 con 0x7f27f4154fd0 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27f40d1750 con 0x7f27f40cef40 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f415c100 con 0x7f27f40d0f60 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27f40cef40 0x7f27f40d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27f40cef40 0x7f27f40d0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34228/0 (socket says 172.21.15.119:34228) 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 -- 172.21.15.119:0/903719032 learned_addr learned my addr 172.21.15.119:0/903719032 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f88fb700 1 --2- 172.21.15.119:0/903719032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40d0f60 0x7f27f415cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 -- 172.21.15.119:0/903719032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40d0f60 msgr2=0x7f27f415cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 --2- 172.21.15.119:0/903719032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40d0f60 0x7f27f415cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 -- 172.21.15.119:0/903719032 >> v1:172.21.15.119:6789/0 conn(0x7f27f4154fd0 legacy=0x7f27f40ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 -- 172.21.15.119:0/903719032 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f415c3e0 con 0x7f27f40cef40 2024-09-13T16:31:17.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f37fe700 1 --2- 172.21.15.119:0/903719032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f40cef40 0x7f27f40d0a20 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009e50 tx=0x7f27ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcedb2953f1aeeb3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:17.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f2ffd700 1 -- 172.21.15.119:0/903719032 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00e070 con 0x7f27f40cef40 2024-09-13T16:31:17.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f2ffd700 1 -- 172.21.15.119:0/903719032 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec00b9c0 con 0x7f27f40cef40 2024-09-13T16:31:17.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f2ffd700 1 -- 172.21.15.119:0/903719032 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec00bd70 con 0x7f27f40cef40 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- 172.21.15.119:0/903719032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f40cef40 msgr2=0x7f27f40d0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 --2- 172.21.15.119:0/903719032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f40cef40 0x7f27f40d0a20 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009e50 tx=0x7f27ec00b330 comp rx=0 tx=0).stop 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- 172.21.15.119:0/903719032 shutdown_connections 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 --2- 172.21.15.119:0/903719032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40d0f60 0x7f27f415cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 --2- 172.21.15.119:0/903719032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f40cef40 0x7f27f40d0a20 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- 172.21.15.119:0/903719032 >> 172.21.15.119:0/903719032 conn(0x7f27f4099a50 msgr2=0x7f27f4099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.269+0000 7f27f90fc700 1 -- 172.21.15.119:0/903719032 shutdown_connections 2024-09-13T16:31:17.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 -- 172.21.15.119:0/903719032 wait complete. 2024-09-13T16:31:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 Processor -- start 2024-09-13T16:31:17.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 -- start start 2024-09-13T16:31:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40cef40 0x7f27f40b6090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27f40d0f60 0x7f27f414ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f4154fd0 0x7f27f414ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f41f10f0 con 0x7f27f4154fd0 2024-09-13T16:31:17.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f414f4d0 con 0x7f27f40d0f60 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f90fc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27f41f38e0 con 0x7f27f40cef40 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f88fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f4154fd0 0x7f27f414ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f88fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f4154fd0 0x7f27f414ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34238/0 (socket says 172.21.15.119:34238) 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f88fb700 1 -- 172.21.15.119:0/1749150365 learned_addr learned my addr 172.21.15.119:0/1749150365 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f37fe700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27f40d0f60 0x7f27f414ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.273+0000 7f27f3fff700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40cef40 0x7f27f40b6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f88fb700 1 -- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40cef40 msgr2=0x7f27f40b6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f88fb700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40cef40 0x7f27f40b6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f88fb700 1 -- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27f40d0f60 msgr2=0x7f27f414ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f88fb700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27f40d0f60 0x7f27f414ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f88fb700 1 -- 172.21.15.119:0/1749150365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec009b30 con 0x7f27f4154fd0 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f88fb700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f4154fd0 0x7f27f414ef90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f27e400e810 tx=0x7f27e400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:17.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f0ff9700 1 -- 172.21.15.119:0/1749150365 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e400c070 con 0x7f27f4154fd0 2024-09-13T16:31:17.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f0ff9700 1 -- 172.21.15.119:0/1749150365 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27e4013650 con 0x7f27f4154fd0 2024-09-13T16:31:17.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f41f3bc0 con 0x7f27f4154fd0 2024-09-13T16:31:17.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f0ff9700 1 -- 172.21.15.119:0/1749150365 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e4013a00 con 0x7f27f4154fd0 2024-09-13T16:31:17.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f41f4180 con 0x7f27f4154fd0 2024-09-13T16:31:17.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.277+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f41f4370 con 0x7f27f4154fd0 2024-09-13T16:31:17.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.281+0000 7f27f0ff9700 1 -- 172.21.15.119:0/1749150365 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f27e4013b60 con 0x7f27f4154fd0 2024-09-13T16:31:17.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.281+0000 7f27f0ff9700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27d4066730 0x7f27d4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.281+0000 7f27f0ff9700 1 -- 172.21.15.119:0/1749150365 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f27e4083d30 con 0x7f27f4154fd0 2024-09-13T16:31:17.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.281+0000 7f27f3fff700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27d4066730 0x7f27d4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.281+0000 7f27f3fff700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27d4066730 0x7f27d4068bf0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f27dc000f60 tx=0x7f27dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:17.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.285+0000 7f27f0ff9700 1 -- 172.21.15.119:0/1749150365 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f27e4050ef0 con 0x7f27f4154fd0 2024-09-13T16:31:17.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.493+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f27f40d0a20 con 0x7f27f4154fd0 2024-09-13T16:31:17.497 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:31:17.493+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:31:17.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.509+0000 7f27f0ff9700 1 -- 172.21.15.119:0/1749150365 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) v1 ==== 161+0+0 (secure 0 0 0) 0x7f27e4054510 con 0x7f27f4154fd0 2024-09-13T16:31:17.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27d4066730 msgr2=0x7f27d4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27d4066730 0x7f27d4068bf0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f27dc000f60 tx=0x7f27dc009040 comp rx=0 tx=0).stop 2024-09-13T16:31:17.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f4154fd0 msgr2=0x7f27f414ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f4154fd0 0x7f27f414ef90 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f27e400e810 tx=0x7f27e400eb20 comp rx=0 tx=0).stop 2024-09-13T16:31:17.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 shutdown_connections 2024-09-13T16:31:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f27d4066730 0x7f27d4068bf0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27f4154fd0 0x7f27f414ef90 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f27f40d0f60 0x7f27f414ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 --2- 172.21.15.119:0/1749150365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f27f40cef40 0x7f27f40b6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 >> 172.21.15.119:0/1749150365 conn(0x7f27f4099a50 msgr2=0x7f27f40adf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 shutdown_connections 2024-09-13T16:31:17.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.513+0000 7f27f90fc700 1 -- 172.21.15.119:0/1749150365 wait complete. 2024-09-13T16:31:17.518 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:31:17.520 INFO:tasks.ceph.mds.c.smithi119.stderr: -4070> 2024-09-13T16:29:54.891+0000 7f1b7691f700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:31:17.532 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:31:17.557 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:31:17.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.637+0000 7fea3759e700 1 Processor -- start 2024-09-13T16:31:17.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.637+0000 7fea3759e700 1 -- start start 2024-09-13T16:31:17.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea3815d950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea380712a0 con 0x7fea380cdd50 2024-09-13T16:31:17.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea38142490 con 0x7fea380acfd0 2024-09-13T16:31:17.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3806fb00 con 0x7fea38141fb0 2024-09-13T16:31:17.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3659c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fea380cdd50 0x7fea380ac8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41230/0 (socket says 172.21.15.119:41230) 2024-09-13T16:31:17.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3659c700 1 -- 172.21.15.119:0/1070688331 learned_addr learned my addr 172.21.15.119:0/1070688331 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:17.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea36d9d700 1 --2- 172.21.15.119:0/1070688331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea3815d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3559a700 1 -- 172.21.15.119:0/1070688331 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734533299 0 0) 0x7fea380712a0 con 0x7fea380cdd50 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3559a700 1 -- 172.21.15.119:0/1070688331 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea240035d0 con 0x7fea380cdd50 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea36d9d700 1 -- 172.21.15.119:0/1070688331 >> v1:172.21.15.145:6789/0 conn(0x7fea380acfd0 legacy=0x7fea3806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea36d9d700 1 -- 172.21.15.119:0/1070688331 >> v1:172.21.15.119:6789/0 conn(0x7fea380cdd50 legacy=0x7fea380ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3559a700 1 -- 172.21.15.119:0/1070688331 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2229451587 0 0) 0x7fea240035d0 con 0x7fea380cdd50 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea36d9d700 1 -- 172.21.15.119:0/1070688331 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38142770 con 0x7fea38141fb0 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea36d9d700 1 --2- 172.21.15.119:0/1070688331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea3815d950 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea28009880 tx=0x7fea28009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b45ccc5b9f68ab25 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3559a700 1 -- 172.21.15.119:0/1070688331 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28011830 con 0x7fea38141fb0 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3559a700 1 -- 172.21.15.119:0/1070688331 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea28011990 con 0x7fea38141fb0 2024-09-13T16:31:17.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3559a700 1 -- 172.21.15.119:0/1070688331 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea28011d00 con 0x7fea38141fb0 2024-09-13T16:31:17.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- 172.21.15.119:0/1070688331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 msgr2=0x7fea3815d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 --2- 172.21.15.119:0/1070688331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea3815d950 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea28009880 tx=0x7fea28009b90 comp rx=0 tx=0).stop 2024-09-13T16:31:17.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- 172.21.15.119:0/1070688331 shutdown_connections 2024-09-13T16:31:17.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 --2- 172.21.15.119:0/1070688331 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea3815d950 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- 172.21.15.119:0/1070688331 >> 172.21.15.119:0/1070688331 conn(0x7fea380b0980 msgr2=0x7fea380b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:17.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- 172.21.15.119:0/1070688331 shutdown_connections 2024-09-13T16:31:17.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.641+0000 7fea3759e700 1 -- 172.21.15.119:0/1070688331 wait complete. 2024-09-13T16:31:17.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 Processor -- start 2024-09-13T16:31:17.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 -- start start 2024-09-13T16:31:17.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea380acfd0 0x7fea381559c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea380cdd50 0x7fea38156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea381565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea38155f00 con 0x7fea380acfd0 2024-09-13T16:31:17.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3813bc20 con 0x7fea380cdd50 2024-09-13T16:31:17.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea381416b0 con 0x7fea38141fb0 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea380acfd0 0x7fea381559c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea380acfd0 0x7fea381559c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34256/0 (socket says 172.21.15.119:34256) 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 -- 172.21.15.119:0/1510859952 learned_addr learned my addr 172.21.15.119:0/1510859952 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea36d9d700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea381565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea35d9b700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea380cdd50 0x7fea38156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 -- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 msgr2=0x7fea381565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea381565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 -- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea380cdd50 msgr2=0x7fea38156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea380cdd50 0x7fea38156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea3659c700 1 -- 172.21.15.119:0/1510859952 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea280094b0 con 0x7fea380acfd0 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea36d9d700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea381565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:17.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.645+0000 7fea35d9b700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea380cdd50 0x7fea38156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea3659c700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea380acfd0 0x7fea381559c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fea3000cb60 tx=0x7fea3000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea2effd700 1 -- 172.21.15.119:0/1510859952 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000a9a0 con 0x7fea380acfd0 2024-09-13T16:31:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea2effd700 1 -- 172.21.15.119:0/1510859952 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3000ab20 con 0x7fea380acfd0 2024-09-13T16:31:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea3813b660 con 0x7fea380acfd0 2024-09-13T16:31:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea2effd700 1 -- 172.21.15.119:0/1510859952 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea3000ce80 con 0x7fea380acfd0 2024-09-13T16:31:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3806d890 con 0x7fea380acfd0 2024-09-13T16:31:17.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3815d950 con 0x7fea380acfd0 2024-09-13T16:31:17.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.649+0000 7fea2effd700 1 -- 172.21.15.119:0/1510859952 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fea3001b3b0 con 0x7fea380acfd0 2024-09-13T16:31:17.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.653+0000 7fea2effd700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea18066730 0x7fea18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:17.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.653+0000 7fea2effd700 1 -- 172.21.15.119:0/1510859952 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6201+0+0 (secure 0 0 0) 0x7fea30083e40 con 0x7fea380acfd0 2024-09-13T16:31:17.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.653+0000 7fea35d9b700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea18066730 0x7fea18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:17.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.653+0000 7fea35d9b700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea18066730 0x7fea18068bf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fea380b03e0 tx=0x7fea20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:17.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.657+0000 7fea2effd700 1 -- 172.21.15.119:0/1510859952 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fea30050fc0 con 0x7fea380acfd0 2024-09-13T16:31:17.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:17.865+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fea38070dc0 con 0x7fea380acfd0 2024-09-13T16:31:18.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.517+0000 7fea2effd700 1 -- 172.21.15.119:0/1510859952 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7fea300545e0 con 0x7fea380acfd0 2024-09-13T16:31:18.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea18066730 msgr2=0x7fea18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea18066730 0x7fea18068bf0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fea380b03e0 tx=0x7fea20009040 comp rx=0 tx=0).stop 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea380acfd0 msgr2=0x7fea381559c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea380acfd0 0x7fea381559c0 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 crypto rx=0x7fea3000cb60 tx=0x7fea3000cc40 comp rx=0 tx=0).stop 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 shutdown_connections 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fea18066730 0x7fea18068bf0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fea38141fb0 0x7fea381565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fea380cdd50 0x7fea38156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 --2- 172.21.15.119:0/1510859952 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fea380acfd0 0x7fea381559c0 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 >> 172.21.15.119:0/1510859952 conn(0x7fea380b0980 msgr2=0x7fea3815df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 shutdown_connections 2024-09-13T16:31:18.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.521+0000 7fea3759e700 1 -- 172.21.15.119:0/1510859952 wait complete. 2024-09-13T16:31:18.543 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:31:18.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.713+0000 7f8e3ee00700 1 Processor -- start 2024-09-13T16:31:18.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.713+0000 7f8e3ee00700 1 -- start start 2024-09-13T16:31:18.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.713+0000 7f8e3ee00700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8e381595d0 0x7f8e38070670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:18.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.713+0000 7f8e3ee00700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8e380cd960 0x7f8e380cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:18.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.713+0000 7f8e3ee00700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e3815a1e0 con 0x7f8e381599b0 2024-09-13T16:31:18.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.713+0000 7f8e3ee00700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e38070ed0 con 0x7f8e380cd960 2024-09-13T16:31:18.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.713+0000 7f8e3ee00700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e380ce310 con 0x7f8e381595d0 2024-09-13T16:31:18.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3e5ff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8e380cd960 0x7f8e380cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:18.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3d5fd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8e381599b0 0x7f8e3806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41232/0 (socket says 172.21.15.119:41232) 2024-09-13T16:31:18.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3d5fd700 1 -- 172.21.15.119:0/3853983674 learned_addr learned my addr 172.21.15.119:0/3853983674 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:18.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ddfe700 1 --2- 172.21.15.119:0/3853983674 >> v2:172.21.15.145:3300/0 conn(0x7f8e381595d0 0x7f8e38070670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:18.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3e5ff700 1 -- 172.21.15.119:0/3853983674 >> v2:172.21.15.145:3300/0 conn(0x7f8e381595d0 msgr2=0x7f8e38070670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3e5ff700 1 --2- 172.21.15.119:0/3853983674 >> v2:172.21.15.145:3300/0 conn(0x7f8e381595d0 0x7f8e38070670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3e5ff700 1 -- 172.21.15.119:0/3853983674 >> v1:172.21.15.119:6789/0 conn(0x7f8e381599b0 legacy=0x7f8e3806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3cdfc700 1 -- 172.21.15.119:0/3853983674 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310595182 0 0) 0x7f8e3815a1e0 con 0x7f8e381599b0 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3e5ff700 1 -- 172.21.15.119:0/3853983674 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e3815a4c0 con 0x7f8e380cd960 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3e5ff700 1 --2- 172.21.15.119:0/3853983674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cddd0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8e380cc430 tx=0x7f8e3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74826d02eb7dcb9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3cdfc700 1 -- 172.21.15.119:0/3853983674 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e3000e070 con 0x7f8e380cd960 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3cdfc700 1 -- 172.21.15.119:0/3853983674 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e3000bbd0 con 0x7f8e380cd960 2024-09-13T16:31:18.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3cdfc700 1 -- 172.21.15.119:0/3853983674 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e30012430 con 0x7f8e380cd960 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 -- 172.21.15.119:0/3853983674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 msgr2=0x7f8e380cddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/3853983674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cddd0 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f8e380cc430 tx=0x7f8e3000b330 comp rx=0 tx=0).stop 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 -- 172.21.15.119:0/3853983674 shutdown_connections 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/3853983674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cddd0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/3853983674 >> v2:172.21.15.145:3300/0 conn(0x7f8e381595d0 0x7f8e38070670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 -- 172.21.15.119:0/3853983674 >> 172.21.15.119:0/3853983674 conn(0x7f8e3806a350 msgr2=0x7f8e3806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 -- 172.21.15.119:0/3853983674 shutdown_connections 2024-09-13T16:31:18.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.717+0000 7f8e3ee00700 1 -- 172.21.15.119:0/3853983674 wait complete. 2024-09-13T16:31:18.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 Processor -- start 2024-09-13T16:31:18.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 -- start start 2024-09-13T16:31:18.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:18.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e381595d0 0x7f8e38150400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e381599b0 0x7f8e381f2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e38152030 con 0x7f8e380cd960 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e38153890 con 0x7f8e381599b0 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e381f26d0 con 0x7f8e381595d0 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34272/0 (socket says 172.21.15.119:34272) 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 -- 172.21.15.119:0/988598254 learned_addr learned my addr 172.21.15.119:0/988598254 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3d5fd700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e381595d0 0x7f8e38150400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3e5ff700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e381599b0 0x7f8e381f2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 -- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e381595d0 msgr2=0x7f8e38150400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e381595d0 0x7f8e38150400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 -- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e381599b0 msgr2=0x7f8e381f2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e381599b0 0x7f8e381f2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 -- 172.21.15.119:0/988598254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e30009a70 con 0x7f8e380cd960 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ddfe700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cd1a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800e790 tx=0x7f8e2800eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e2e7fc700 1 -- 172.21.15.119:0/988598254 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e2800c070 con 0x7f8e380cd960 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e2e7fc700 1 -- 172.21.15.119:0/988598254 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e28013610 con 0x7f8e380cd960 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e381f2950 con 0x7f8e380cd960 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e2e7fc700 1 -- 172.21.15.119:0/988598254 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e28013980 con 0x7f8e380cd960 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.721+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e381f2f10 con 0x7f8e380cd960 2024-09-13T16:31:18.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.725+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e380cddd0 con 0x7f8e380cd960 2024-09-13T16:31:18.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.725+0000 7f8e2e7fc700 1 -- 172.21.15.119:0/988598254 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8e28013ae0 con 0x7f8e380cd960 2024-09-13T16:31:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.725+0000 7f8e2e7fc700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e1c066730 0x7f8e1c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.725+0000 7f8e2e7fc700 1 -- 172.21.15.119:0/988598254 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f8e28084550 con 0x7f8e380cd960 2024-09-13T16:31:18.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.725+0000 7f8e3d5fd700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e1c066730 0x7f8e1c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:18.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.729+0000 7f8e3d5fd700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e1c066730 0x7f8e1c068bf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8e38067160 tx=0x7f8e34004a70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:18.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.733+0000 7f8e2e7fc700 1 -- 172.21.15.119:0/988598254 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8e280517b0 con 0x7f8e380cd960 2024-09-13T16:31:18.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:18.901+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8e3815b150 con 0x7f8e380cd960 2024-09-13T16:31:19.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.521+0000 7f8e2e7fc700 1 -- 172.21.15.119:0/988598254 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v85) v1 ==== 181+0+0 (secure 0 0 0) 0x7f8e28054dd0 con 0x7f8e380cd960 2024-09-13T16:31:19.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e1c066730 msgr2=0x7f8e1c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:19.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e1c066730 0x7f8e1c068bf0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f8e38067160 tx=0x7f8e34004a70 comp rx=0 tx=0).stop 2024-09-13T16:31:19.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 msgr2=0x7f8e380cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:19.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cd1a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 crypto rx=0x7f8e2800e790 tx=0x7f8e2800eaa0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 shutdown_connections 2024-09-13T16:31:19.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e1c066730 0x7f8e1c068bf0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e381599b0 0x7f8e381f2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e381595d0 0x7f8e38150400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 --2- 172.21.15.119:0/988598254 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e380cd960 0x7f8e380cd1a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 >> 172.21.15.119:0/988598254 conn(0x7f8e3806a350 msgr2=0x7f8e38158570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:19.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.525+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 shutdown_connections 2024-09-13T16:31:19.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.529+0000 7f8e3ee00700 1 -- 172.21.15.119:0/988598254 wait complete. 2024-09-13T16:31:19.533 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:31:19.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:31:19.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a925f700 1 Processor -- start 2024-09-13T16:31:19.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a925f700 1 -- start start 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a925f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f85a415cc40 0x7f85a40cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a925f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a40d3260 con 0x7f85a40d1840 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a925f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a40cd640 con 0x7f85a40d0720 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a925f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85a40d0130 con 0x7f85a415cc40 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a8a5e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f85a40d1840 0x7f85a40cdab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41236/0 (socket says 172.21.15.119:41236) 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f85a415cc40 0x7f85a40cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a8a5e700 1 -- 172.21.15.119:0/3567876327 learned_addr learned my addr 172.21.15.119:0/3567876327 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a2ffd700 1 -- 172.21.15.119:0/3567876327 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385932222 0 0) 0x7f85a40d3260 con 0x7f85a40d1840 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a2ffd700 1 -- 172.21.15.119:0/3567876327 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85900035d0 con 0x7f85a40d1840 2024-09-13T16:31:19.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a3fff700 1 -- 172.21.15.119:0/3567876327 >> v1:172.21.15.145:6789/0 conn(0x7f85a40d0720 legacy=0x7f85a40d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:19.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a3fff700 1 -- 172.21.15.119:0/3567876327 >> v1:172.21.15.119:6789/0 conn(0x7f85a40d1840 legacy=0x7f85a40cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:19.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a3fff700 1 -- 172.21.15.119:0/3567876327 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a40d0c70 con 0x7f85a415cc40 2024-09-13T16:31:19.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a3fff700 1 --2- 172.21.15.119:0/3567876327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a415cc40 0x7f85a40cfbf0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f858c009850 tx=0x7f858c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df701380f08b169 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:19.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a2ffd700 1 -- 172.21.15.119:0/3567876327 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c011830 con 0x7f85a415cc40 2024-09-13T16:31:19.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a2ffd700 1 -- 172.21.15.119:0/3567876327 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f858c011990 con 0x7f85a415cc40 2024-09-13T16:31:19.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.673+0000 7f85a2ffd700 1 -- 172.21.15.119:0/3567876327 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858c011d40 con 0x7f85a415cc40 2024-09-13T16:31:19.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 -- 172.21.15.119:0/3567876327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a415cc40 msgr2=0x7f85a40cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:19.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 --2- 172.21.15.119:0/3567876327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a415cc40 0x7f85a40cfbf0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7f858c009850 tx=0x7f858c009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 -- 172.21.15.119:0/3567876327 shutdown_connections 2024-09-13T16:31:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 --2- 172.21.15.119:0/3567876327 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a415cc40 0x7f85a40cfbf0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 -- 172.21.15.119:0/3567876327 >> 172.21.15.119:0/3567876327 conn(0x7f85a409be40 msgr2=0x7f85a40d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 -- 172.21.15.119:0/3567876327 shutdown_connections 2024-09-13T16:31:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 -- 172.21.15.119:0/3567876327 wait complete. 2024-09-13T16:31:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 Processor -- start 2024-09-13T16:31:19.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.677+0000 7f85a925f700 1 -- start start 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85a40d0720 0x7f85a41f5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a40d1840 0x7f85a40c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85a415cc40 0x7f85a40b4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a40ccd20 con 0x7f85a40d1840 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a40c7b70 con 0x7f85a415cc40 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a40cc760 con 0x7f85a40d0720 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a40d1840 0x7f85a40c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:19.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a40d1840 0x7f85a40c9c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34286/0 (socket says 172.21.15.119:34286) 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 -- 172.21.15.119:0/2750468927 learned_addr learned my addr 172.21.15.119:0/2750468927 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a3fff700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85a40d0720 0x7f85a41f5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a8a5e700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85a415cc40 0x7f85a40b4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 -- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85a40d0720 msgr2=0x7f85a41f5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85a40d0720 0x7f85a41f5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 -- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85a415cc40 msgr2=0x7f85a40b4630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85a415cc40 0x7f85a40b4630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 -- 172.21.15.119:0/2750468927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f858c0094b0 con 0x7f85a40d1840 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a8a5e700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85a415cc40 0x7f85a40b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:19.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a37fe700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a40d1840 0x7f85a40c9c20 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f859400c500 tx=0x7f859400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a0ff9700 1 -- 172.21.15.119:0/2750468927 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940052d0 con 0x7f85a40d1840 2024-09-13T16:31:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a0ff9700 1 -- 172.21.15.119:0/2750468927 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8594005430 con 0x7f85a40d1840 2024-09-13T16:31:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a40cca40 con 0x7f85a40d1840 2024-09-13T16:31:19.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a0ff9700 1 -- 172.21.15.119:0/2750468927 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85940055a0 con 0x7f85a40d1840 2024-09-13T16:31:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a4146a70 con 0x7f85a40d1840 2024-09-13T16:31:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.681+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a406ab20 con 0x7f85a40d1840 2024-09-13T16:31:19.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.685+0000 7f85a0ff9700 1 -- 172.21.15.119:0/2750468927 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f85940038e0 con 0x7f85a40d1840 2024-09-13T16:31:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.685+0000 7f85a0ff9700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8584066760 0x7f8584068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:19.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.685+0000 7f85a0ff9700 1 -- 172.21.15.119:0/2750468927 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5763+0+0 (secure 0 0 0) 0x7f8594083500 con 0x7f85a40d1840 2024-09-13T16:31:19.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.685+0000 7f85a3fff700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8584066760 0x7f8584068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:19.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.685+0000 7f85a3fff700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8584066760 0x7f8584068c20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f858c0090b0 tx=0x7f858c01c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:19.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.689+0000 7f85a0ff9700 1 -- 172.21.15.119:0/2750468927 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8594050820 con 0x7f85a40d1840 2024-09-13T16:31:19.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:19.861+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f85a40d0320 con 0x7f85a40d1840 2024-09-13T16:31:20.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.537+0000 7f85a0ff9700 1 -- 172.21.15.119:0/2750468927 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v86) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8594053e40 con 0x7f85a40d1840 2024-09-13T16:31:20.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8584066760 msgr2=0x7f8584068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8584066760 0x7f8584068c20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f858c0090b0 tx=0x7f858c01c040 comp rx=0 tx=0).stop 2024-09-13T16:31:20.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a40d1840 msgr2=0x7f85a40c9c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a40d1840 0x7f85a40c9c20 secure :-1 s=READY pgs=1207 cs=0 l=1 rev1=1 crypto rx=0x7f859400c500 tx=0x7f859400c810 comp rx=0 tx=0).stop 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 shutdown_connections 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8584066760 0x7f8584068c20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f85a415cc40 0x7f85a40b4630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f85a40d1840 0x7f85a40c9c20 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 --2- 172.21.15.119:0/2750468927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f85a40d0720 0x7f85a41f5e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 >> 172.21.15.119:0/2750468927 conn(0x7f85a409be40 msgr2=0x7f85a40d1130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 shutdown_connections 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.541+0000 7f85a925f700 1 -- 172.21.15.119:0/2750468927 wait complete. 2024-09-13T16:31:20.547 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:31:20.560 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T16:31:20.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.665+0000 7f1e8899d700 1 Processor -- start 2024-09-13T16:31:20.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.665+0000 7f1e8899d700 1 -- start start 2024-09-13T16:31:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e8899d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 0x7f1e8406c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e8899d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8406cb40 con 0x7f1e840cc010 2024-09-13T16:31:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e8899d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e8406ce20 con 0x7f1e840ce1d0 2024-09-13T16:31:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e8899d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e840b1c50 con 0x7f1e84144180 2024-09-13T16:31:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e82ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1e840cc010 0x7f1e840cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41242/0 (socket says 172.21.15.119:41242) 2024-09-13T16:31:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e82ffd700 1 -- 172.21.15.119:0/1074483852 learned_addr learned my addr 172.21.15.119:0/1074483852 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 0x7f1e8406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878713739 0 0) 0x7f1e8406cb40 con 0x7f1e840cc010 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e6c0035d0 con 0x7f1e840cc010 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2309264512 0 0) 0x7f1e6c0035d0 con 0x7f1e840cc010 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 msgr2=0x7f1e8406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 0x7f1e8406c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 >> v1:172.21.15.145:6789/0 conn(0x7f1e840ce1d0 legacy=0x7f1e840b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8406d100 con 0x7f1e840cc010 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f1e74001670 con 0x7f1e840cc010 2024-09-13T16:31:20.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 >> v1:172.21.15.119:6789/0 conn(0x7f1e840cc010 legacy=0x7f1e840cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e6c003a50 0x7f1e6c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c004870 0x7f1e6c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 0x7f1e6c005270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e840b1c50 con 0x7f1e6c003a50 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c00bb60 con 0x7f1e6c004870 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e6c00bd20 con 0x7f1e84144180 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e6c003a50 0x7f1e6c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e82ffd700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c004870 0x7f1e6c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e83fff700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 0x7f1e6c005270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 -- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 msgr2=0x7f1e6c005270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e84144180 0x7f1e6c005270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 -- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c004870 msgr2=0x7f1e6c004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:20.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c004870 0x7f1e6c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 -- 172.21.15.119:0/1074483852 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e6c0035d0 con 0x7f1e6c003a50 2024-09-13T16:31:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e837fe700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e6c003a50 0x7f1e6c003e70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1e780098b0 tx=0x7f1e6c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44dd85d743870262 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7800c070 con 0x7f1e6c003a50 2024-09-13T16:31:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e78013610 con 0x7f1e6c003a50 2024-09-13T16:31:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.669+0000 7f1e827fc700 1 -- 172.21.15.119:0/1074483852 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e780139c0 con 0x7f1e6c003a50 2024-09-13T16:31:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 -- 172.21.15.119:0/1074483852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e6c003a50 msgr2=0x7f1e6c003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 --2- 172.21.15.119:0/1074483852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e6c003a50 0x7f1e6c003e70 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f1e780098b0 tx=0x7f1e6c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e83fff700 1 -- 172.21.15.119:0/1074483852 reap_dead start 2024-09-13T16:31:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 -- 172.21.15.119:0/1074483852 shutdown_connections 2024-09-13T16:31:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 -- 172.21.15.119:0/1074483852 >> 172.21.15.119:0/1074483852 conn(0x7f1e840b0980 msgr2=0x7f1e840b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 -- 172.21.15.119:0/1074483852 shutdown_connections 2024-09-13T16:31:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 -- 172.21.15.119:0/1074483852 wait complete. 2024-09-13T16:31:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 Processor -- start 2024-09-13T16:31:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.673+0000 7f1e8899d700 1 -- start start 2024-09-13T16:31:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c003a50 0x7f1e841319b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e841f1040 0x7f1e84140c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e841f14c0 0x7f1e841411c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8412f360 con 0x7f1e841f14c0 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8412f640 con 0x7f1e6c003a50 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e8412f920 con 0x7f1e841f1040 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e83fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e841f14c0 0x7f1e841411c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e82ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e841f1040 0x7f1e84140c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e83fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e841f14c0 0x7f1e841411c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34302/0 (socket says 172.21.15.119:34302) 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e83fff700 1 -- 172.21.15.119:0/2970769823 learned_addr learned my addr 172.21.15.119:0/2970769823 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e837fe700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c003a50 0x7f1e841319b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e82ffd700 1 -- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c003a50 msgr2=0x7f1e841319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e82ffd700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c003a50 0x7f1e841319b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e82ffd700 1 -- 172.21.15.119:0/2970769823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e841f14c0 msgr2=0x7f1e841411c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e82ffd700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e841f14c0 0x7f1e841411c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e82ffd700 1 -- 172.21.15.119:0/2970769823 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e8412fc00 con 0x7f1e841f1040 2024-09-13T16:31:20.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e82ffd700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e841f1040 0x7f1e84140c80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1e74001b40 tx=0x7f1e74007570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:20.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e6bfff700 1 -- 172.21.15.119:0/2970769823 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74011580 con 0x7f1e841f1040 2024-09-13T16:31:20.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e6bfff700 1 -- 172.21.15.119:0/2970769823 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e74011720 con 0x7f1e841f1040 2024-09-13T16:31:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e6bfff700 1 -- 172.21.15.119:0/2970769823 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e74011ad0 con 0x7f1e841f1040 2024-09-13T16:31:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e8412fee0 con 0x7f1e841f1040 2024-09-13T16:31:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.677+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e840ac090 con 0x7f1e841f1040 2024-09-13T16:31:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.681+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e8406c480 con 0x7f1e841f1040 2024-09-13T16:31:20.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.681+0000 7f1e6bfff700 1 -- 172.21.15.119:0/2970769823 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1e74011c30 con 0x7f1e841f1040 2024-09-13T16:31:20.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.681+0000 7f1e6bfff700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e70066760 0x7f1e70068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:20.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.681+0000 7f1e6bfff700 1 -- 172.21.15.119:0/2970769823 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f1e74084a60 con 0x7f1e841f1040 2024-09-13T16:31:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.681+0000 7f1e837fe700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e70066760 0x7f1e70068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:20.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.681+0000 7f1e837fe700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e70066760 0x7f1e70068c20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1e78016040 tx=0x7f1e7801f040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:20.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.693+0000 7f1e6bfff700 1 -- 172.21.15.119:0/2970769823 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1e74051f40 con 0x7f1e841f1040 2024-09-13T16:31:20.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.861+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f1e840ae540 con 0x7f1e841f1040 2024-09-13T16:31:20.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.869+0000 7f1e6bfff700 1 -- 172.21.15.119:0/2970769823 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1e74055560 con 0x7f1e841f1040 2024-09-13T16:31:20.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e70066760 msgr2=0x7f1e70068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e70066760 0x7f1e70068c20 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f1e78016040 tx=0x7f1e7801f040 comp rx=0 tx=0).stop 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e841f1040 msgr2=0x7f1e84140c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e841f1040 0x7f1e84140c80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1e74001b40 tx=0x7f1e74007570 comp rx=0 tx=0).stop 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 shutdown_connections 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1e70066760 0x7f1e70068c20 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e841f14c0 0x7f1e841411c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e841f1040 0x7f1e84140c80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 --2- 172.21.15.119:0/2970769823 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e6c003a50 0x7f1e841319b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 >> 172.21.15.119:0/2970769823 conn(0x7f1e840b0980 msgr2=0x7f1e84066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 shutdown_connections 2024-09-13T16:31:20.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:20.873+0000 7f1e8899d700 1 -- 172.21.15.119:0/2970769823 wait complete. 2024-09-13T16:31:20.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair' 2024-09-13T16:31:21.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 Processor -- start 2024-09-13T16:31:21.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- start start 2024-09-13T16:31:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f37d80b17f0 0x7f37d80cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d80705d0 con 0x7f37d80b2200 2024-09-13T16:31:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37d806fca0 con 0x7f37d80cd8d0 2024-09-13T16:31:21.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37d806ff80 con 0x7f37d80b17f0 2024-09-13T16:31:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dd61a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f37d80b2200 0x7f37d8070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41258/0 (socket says 172.21.15.119:41258) 2024-09-13T16:31:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dd61a700 1 -- 172.21.15.119:0/162686283 learned_addr learned my addr 172.21.15.119:0/162686283 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:21.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cffff700 1 --2- 172.21.15.119:0/162686283 >> v2:172.21.15.119:3300/0 conn(0x7f37d80b17f0 0x7f37d80cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cf7fe700 1 -- 172.21.15.119:0/162686283 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649563697 0 0) 0x7f37d80705d0 con 0x7f37d80b2200 2024-09-13T16:31:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cf7fe700 1 -- 172.21.15.119:0/162686283 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37c00035d0 con 0x7f37d80b2200 2024-09-13T16:31:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cffff700 1 -- 172.21.15.119:0/162686283 >> v1:172.21.15.145:6789/0 conn(0x7f37d80cd8d0 legacy=0x7f37d806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cffff700 1 -- 172.21.15.119:0/162686283 >> v1:172.21.15.119:6789/0 conn(0x7f37d80b2200 legacy=0x7f37d8070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cffff700 1 -- 172.21.15.119:0/162686283 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37d8070260 con 0x7f37d80b17f0 2024-09-13T16:31:21.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cffff700 1 --2- 172.21.15.119:0/162686283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80b17f0 0x7f37d80cb800 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009850 tx=0x7f37c8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75a6142afe55965a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cf7fe700 1 -- 172.21.15.119:0/162686283 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8011830 con 0x7f37d80b17f0 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cf7fe700 1 -- 172.21.15.119:0/162686283 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37c8011990 con 0x7f37d80b17f0 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37cf7fe700 1 -- 172.21.15.119:0/162686283 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37c8011d00 con 0x7f37d80b17f0 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- 172.21.15.119:0/162686283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80b17f0 msgr2=0x7f37d80cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 --2- 172.21.15.119:0/162686283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80b17f0 0x7f37d80cb800 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f37c8009850 tx=0x7f37c8009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- 172.21.15.119:0/162686283 shutdown_connections 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 --2- 172.21.15.119:0/162686283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80b17f0 0x7f37d80cb800 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- 172.21.15.119:0/162686283 >> 172.21.15.119:0/162686283 conn(0x7f37d806a350 msgr2=0x7f37d806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:21.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- 172.21.15.119:0/162686283 shutdown_connections 2024-09-13T16:31:21.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.005+0000 7f37dde1b700 1 -- 172.21.15.119:0/162686283 wait complete. 2024-09-13T16:31:21.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 Processor -- start 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 -- start start 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37d80b17f0 0x7f37d80c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37d80b2200 0x7f37d81f5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80cd8d0 0x7f37d81f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80c5160 con 0x7f37d80cd8d0 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80c6800 con 0x7f37d80b2200 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37d80a8520 con 0x7f37d80b17f0 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80cd8d0 0x7f37d81f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80cd8d0 0x7f37d81f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34312/0 (socket says 172.21.15.119:34312) 2024-09-13T16:31:21.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 -- 172.21.15.119:0/3279899109 learned_addr learned my addr 172.21.15.119:0/3279899109 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:21.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 -- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37d80b17f0 msgr2=0x7f37d80c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37d80b17f0 0x7f37d80c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 -- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37d80b2200 msgr2=0x7f37d81f5ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37d80b2200 0x7f37d81f5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 -- 172.21.15.119:0/3279899109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37c80094b0 con 0x7f37d80cd8d0 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dd61a700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80cd8d0 0x7f37d81f6000 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f37d000cb40 tx=0x7f37d000cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37cd7fa700 1 -- 172.21.15.119:0/3279899109 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d000a9e0 con 0x7f37d80cd8d0 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37cd7fa700 1 -- 172.21.15.119:0/3279899109 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37d000ab40 con 0x7f37d80cd8d0 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37d81f6540 con 0x7f37d80cd8d0 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37cd7fa700 1 -- 172.21.15.119:0/3279899109 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37d000ce60 con 0x7f37d80cd8d0 2024-09-13T16:31:21.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.009+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37d80c6420 con 0x7f37d80cd8d0 2024-09-13T16:31:21.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.013+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37d80c6940 con 0x7f37d80cd8d0 2024-09-13T16:31:21.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.013+0000 7f37cd7fa700 1 -- 172.21.15.119:0/3279899109 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f37d001b3b0 con 0x7f37d80cd8d0 2024-09-13T16:31:21.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.013+0000 7f37cd7fa700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37c4066730 0x7f37c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.013+0000 7f37cd7fa700 1 -- 172.21.15.119:0/3279899109 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f37d0083930 con 0x7f37d80cd8d0 2024-09-13T16:31:21.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.013+0000 7f37dce19700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37c4066730 0x7f37c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.013+0000 7f37dce19700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37c4066730 0x7f37c4068bf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f37d4000f60 tx=0x7f37d4007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:21.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.021+0000 7f37cd7fa700 1 -- 172.21.15.119:0/3279899109 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f37d80c6940 con 0x7f37d80cd8d0 2024-09-13T16:31:21.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.253+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]} v 0) v1 -- 0x7f37d81f6540 con 0x7f37d80cd8d0 2024-09-13T16:31:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.533+0000 7f37cd7fa700 1 -- 172.21.15.119:0/3279899109 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dentry_repair"]}]=0 v298) v1 ==== 166+0+0 (secure 0 0 0) 0x7f37d81f6540 con 0x7f37d80cd8d0 2024-09-13T16:31:21.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37c4066730 msgr2=0x7f37c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37c4066730 0x7f37c4068bf0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f37d4000f60 tx=0x7f37d4007a80 comp rx=0 tx=0).stop 2024-09-13T16:31:21.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80cd8d0 msgr2=0x7f37d81f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80cd8d0 0x7f37d81f6000 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f37d000cb40 tx=0x7f37d000cc20 comp rx=0 tx=0).stop 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 shutdown_connections 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f37c4066730 0x7f37c4068bf0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f37d80cd8d0 0x7f37d81f6000 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f37d80b2200 0x7f37d81f5ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 --2- 172.21.15.119:0/3279899109 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f37d80b17f0 0x7f37d80c89c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 >> 172.21.15.119:0/3279899109 conn(0x7f37d806a350 msgr2=0x7f37d8070960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 shutdown_connections 2024-09-13T16:31:21.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.537+0000 7f37dde1b700 1 -- 172.21.15.119:0/3279899109 wait complete. 2024-09-13T16:31:21.559 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-09-13T16:31:21.559 INFO:tasks.cephfs_test_runner:Starting test: test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) Test that the damage health status is cleared 2024-09-13T16:31:21.559 INFO:tasks.cephfs_test_runner:test_health_status_after_dirfrag_repair (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-09-13T16:31:21.559 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:21.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 Processor -- start 2024-09-13T16:31:21.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- start start 2024-09-13T16:31:21.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f46a015e840 0x7f46a0070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a0165180 0x7f46a0161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a0071210 con 0x7f46a015ec20 2024-09-13T16:31:21.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a00714f0 con 0x7f46a015e840 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00717d0 con 0x7f46a0165180 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f46a015e840 0x7f46a0070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f46a015e840 0x7f46a0070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34316/0 (socket says 172.21.15.119:34316) 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 -- 172.21.15.119:0/1641209585 learned_addr learned my addr 172.21.15.119:0/1641209585 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a5ae1700 1 --2- 172.21.15.119:0/1641209585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a0165180 0x7f46a0161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 -- 172.21.15.119:0/1641209585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a0165180 msgr2=0x7f46a0161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 --2- 172.21.15.119:0/1641209585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a0165180 0x7f46a0161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 -- 172.21.15.119:0/1641209585 >> v1:172.21.15.145:6789/0 conn(0x7f46a015ec20 legacy=0x7f46a0164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 -- 172.21.15.119:0/1641209585 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a0071ab0 con 0x7f46a015e840 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a52e0700 1 --2- 172.21.15.119:0/1641209585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a0070b80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f4690009910 tx=0x7f4690009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef32b8ea994d3ac7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f4697fff700 1 -- 172.21.15.119:0/1641209585 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f469000ba60 con 0x7f46a015e840 2024-09-13T16:31:21.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f4697fff700 1 -- 172.21.15.119:0/1641209585 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469000bbc0 con 0x7f46a015e840 2024-09-13T16:31:21.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f4697fff700 1 -- 172.21.15.119:0/1641209585 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690009e60 con 0x7f46a015e840 2024-09-13T16:31:21.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- 172.21.15.119:0/1641209585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 msgr2=0x7f46a0070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 --2- 172.21.15.119:0/1641209585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a0070b80 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f4690009910 tx=0x7f4690009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:21.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- 172.21.15.119:0/1641209585 shutdown_connections 2024-09-13T16:31:21.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 --2- 172.21.15.119:0/1641209585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a0165180 0x7f46a0161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 --2- 172.21.15.119:0/1641209585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a0070b80 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- 172.21.15.119:0/1641209585 >> 172.21.15.119:0/1641209585 conn(0x7f46a006a350 msgr2=0x7f46a006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:21.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- 172.21.15.119:0/1641209585 shutdown_connections 2024-09-13T16:31:21.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.677+0000 7f46a7544700 1 -- 172.21.15.119:0/1641209585 wait complete. 2024-09-13T16:31:21.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.681+0000 7f46a7544700 1 Processor -- start 2024-09-13T16:31:21.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.681+0000 7f46a7544700 1 -- start start 2024-09-13T16:31:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a00cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0165180 0x7f46a00ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a015d6e0 0x7f46a015db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a015e0d0 con 0x7f46a015e840 2024-09-13T16:31:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00a8890 con 0x7f46a0165180 2024-09-13T16:31:21.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a00a8b70 con 0x7f46a015d6e0 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a00cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a00cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34330/0 (socket says 172.21.15.119:34330) 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 -- 172.21.15.119:0/2092217123 learned_addr learned my addr 172.21.15.119:0/2092217123 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a4adf700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0165180 0x7f46a00ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a5ae1700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a015d6e0 0x7f46a015db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 -- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a015d6e0 msgr2=0x7f46a015db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a015d6e0 0x7f46a015db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 -- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0165180 msgr2=0x7f46a00ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0165180 0x7f46a00ccee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 -- 172.21.15.119:0/2092217123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4690009570 con 0x7f46a015e840 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a5ae1700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a015d6e0 0x7f46a015db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a52e0700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a00cda00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f46a0069530 tx=0x7f4690004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f4695ffb700 1 -- 172.21.15.119:0/2092217123 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690004610 con 0x7f46a015e840 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f4695ffb700 1 -- 172.21.15.119:0/2092217123 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f469001f070 con 0x7f46a015e840 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a00a8e50 con 0x7f46a015e840 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f4695ffb700 1 -- 172.21.15.119:0/2092217123 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4690003900 con 0x7f46a015e840 2024-09-13T16:31:21.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a00a9410 con 0x7f46a015e840 2024-09-13T16:31:21.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.685+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a0067760 con 0x7f46a015e840 2024-09-13T16:31:21.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.689+0000 7f4695ffb700 1 -- 172.21.15.119:0/2092217123 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4690003a60 con 0x7f46a015e840 2024-09-13T16:31:21.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.689+0000 7f4695ffb700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4684066760 0x7f4684068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:21.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.689+0000 7f4695ffb700 1 -- 172.21.15.119:0/2092217123 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f4690021070 con 0x7f46a015e840 2024-09-13T16:31:21.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.689+0000 7f46a4adf700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4684066760 0x7f4684068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:21.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.689+0000 7f46a4adf700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4684066760 0x7f4684068c20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f46a00cd7c0 tx=0x7f469c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:21.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.693+0000 7f4695ffb700 1 -- 172.21.15.119:0/2092217123 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4690052de0 con 0x7f46a015e840 2024-09-13T16:31:21.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.869+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46a00a9600 con 0x7f46a015e840 2024-09-13T16:31:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.869+0000 7f4695ffb700 1 -- 172.21.15.119:0/2092217123 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+13384 (secure 0 0 0) 0x7f4690056400 con 0x7f46a015e840 2024-09-13T16:31:21.875 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:21.875 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":86,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:31:20.520859+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/2534803404":"2024-09-14T16:31:17.496938+0000","172.21.15.119:6834/2534803404":"2024-09-14T16:31:17.496938+0000","172.21.15.119:6835/491709455":"2024-09-14T16:29:48.943151+0000","172.21.15.119:6834/491709455":"2024-09-14T16:29:48.943151+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4684066760 msgr2=0x7f4684068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4684066760 0x7f4684068c20 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f46a00cd7c0 tx=0x7f469c009040 comp rx=0 tx=0).stop 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 msgr2=0x7f46a00cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a00cda00 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 crypto rx=0x7f46a0069530 tx=0x7f4690004770 comp rx=0 tx=0).stop 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 shutdown_connections 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4684066760 0x7f4684068c20 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f46a015d6e0 0x7f46a015db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f46a0165180 0x7f46a00ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 --2- 172.21.15.119:0/2092217123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46a015e840 0x7f46a00cda00 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:21.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 >> 172.21.15.119:0/2092217123 conn(0x7f46a006a350 msgr2=0x7f46a0162590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:21.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 shutdown_connections 2024-09-13T16:31:21.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:21.873+0000 7f46a7544700 1 -- 172.21.15.119:0/2092217123 wait complete. 2024-09-13T16:31:21.889 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:31:21.890 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:31:22.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db7fff700 1 Processor -- start 2024-09-13T16:31:22.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db7fff700 1 -- start start 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7db80ad9a0 0x7f7db8159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db7fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7db806ca60 0x7f7db815a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db7fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db80b1dd0 con 0x7f7db80b1860 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db7fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db80b2fd0 con 0x7f7db80ad9a0 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db7fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db815a520 con 0x7f7db806ca60 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7db6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7db80ad9a0 0x7f7db8159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7daffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7db80b1860 0x7f7db806c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41266/0 (socket says 172.21.15.119:41266) 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.005+0000 7f7daffff700 1 -- 172.21.15.119:0/3508458360 learned_addr learned my addr 172.21.15.119:0/3508458360 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db77fe700 1 --2- 172.21.15.119:0/3508458360 >> v2:172.21.15.145:3300/0 conn(0x7f7db806ca60 0x7f7db815a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db67fc700 1 -- 172.21.15.119:0/3508458360 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3239324271 0 0) 0x7f7db80b1dd0 con 0x7f7db80b1860 2024-09-13T16:31:22.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db67fc700 1 -- 172.21.15.119:0/3508458360 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da40035d0 con 0x7f7db80b1860 2024-09-13T16:31:22.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db67fc700 1 -- 172.21.15.119:0/3508458360 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4007408717 0 0) 0x7f7da40035d0 con 0x7f7db80b1860 2024-09-13T16:31:22.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db6ffd700 1 -- 172.21.15.119:0/3508458360 >> v2:172.21.15.145:3300/0 conn(0x7f7db806ca60 msgr2=0x7f7db815a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:22.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db6ffd700 1 --2- 172.21.15.119:0/3508458360 >> v2:172.21.15.145:3300/0 conn(0x7f7db806ca60 0x7f7db815a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db6ffd700 1 -- 172.21.15.119:0/3508458360 >> v1:172.21.15.119:6789/0 conn(0x7f7db80b1860 legacy=0x7f7db806c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db6ffd700 1 -- 172.21.15.119:0/3508458360 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db80b20b0 con 0x7f7db80ad9a0 2024-09-13T16:31:22.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db6ffd700 1 --2- 172.21.15.119:0/3508458360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80ad9a0 0x7f7db8159de0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7db8159360 tx=0x7f7da8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9f01f8a1f8fc974 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:22.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db67fc700 1 -- 172.21.15.119:0/3508458360 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da0001670 con 0x7f7db80ad9a0 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db67fc700 1 -- 172.21.15.119:0/3508458360 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da8004030 con 0x7f7db80ad9a0 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db67fc700 1 -- 172.21.15.119:0/3508458360 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7da80043c0 con 0x7f7db80ad9a0 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 -- 172.21.15.119:0/3508458360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80ad9a0 msgr2=0x7f7db8159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 --2- 172.21.15.119:0/3508458360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80ad9a0 0x7f7db8159de0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7f7db8159360 tx=0x7f7da8009b30 comp rx=0 tx=0).stop 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 -- 172.21.15.119:0/3508458360 shutdown_connections 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 --2- 172.21.15.119:0/3508458360 >> v2:172.21.15.145:3300/0 conn(0x7f7db806ca60 0x7f7db815a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 --2- 172.21.15.119:0/3508458360 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80ad9a0 0x7f7db8159de0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 -- 172.21.15.119:0/3508458360 >> 172.21.15.119:0/3508458360 conn(0x7f7db80b0980 msgr2=0x7f7db80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 -- 172.21.15.119:0/3508458360 shutdown_connections 2024-09-13T16:31:22.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 -- 172.21.15.119:0/3508458360 wait complete. 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 Processor -- start 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.009+0000 7f7db7fff700 1 -- start start 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7db806ca60 0x7f7db81421f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7db80ad9a0 0x7f7db81519d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80b1860 0x7f7db8150030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db81515b0 con 0x7f7db80b1860 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db8142730 con 0x7f7db806ca60 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db8151f10 con 0x7f7db80ad9a0 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80b1860 0x7f7db8150030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80b1860 0x7f7db8150030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34352/0 (socket says 172.21.15.119:34352) 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 -- 172.21.15.119:0/1655048401 learned_addr learned my addr 172.21.15.119:0/1655048401 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7daffff700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7db80ad9a0 0x7f7db81519d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db6ffd700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7db806ca60 0x7f7db81421f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 -- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7db80ad9a0 msgr2=0x7f7db81519d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7db80ad9a0 0x7f7db81519d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 -- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7db806ca60 msgr2=0x7f7db81421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7db806ca60 0x7f7db81421f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 -- 172.21.15.119:0/1655048401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da80094b0 con 0x7f7db80b1860 2024-09-13T16:31:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db77fe700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80b1860 0x7f7db8150030 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7db000efc0 tx=0x7f7db000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7daf7fe700 1 -- 172.21.15.119:0/1655048401 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0009ea0 con 0x7f7db80b1860 2024-09-13T16:31:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7daf7fe700 1 -- 172.21.15.119:0/1655048401 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db000f040 con 0x7f7db80b1860 2024-09-13T16:31:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db8150930 con 0x7f7db80b1860 2024-09-13T16:31:22.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7daf7fe700 1 -- 172.21.15.119:0/1655048401 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000ced0 con 0x7f7db80b1860 2024-09-13T16:31:22.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db8150ef0 con 0x7f7db80b1860 2024-09-13T16:31:22.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.013+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d9c005310 con 0x7f7db80b1860 2024-09-13T16:31:22.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.017+0000 7f7daf7fe700 1 -- 172.21.15.119:0/1655048401 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7db0014480 con 0x7f7db80b1860 2024-09-13T16:31:22.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.017+0000 7f7daf7fe700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d98066730 0x7f7d98068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.017+0000 7f7daf7fe700 1 -- 172.21.15.119:0/1655048401 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f7db00847f0 con 0x7f7db80b1860 2024-09-13T16:31:22.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.017+0000 7f7db6ffd700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d98066730 0x7f7d98068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.017+0000 7f7db6ffd700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d98066730 0x7f7d98068bf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7da80097a0 tx=0x7f7da8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:22.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.021+0000 7f7daf7fe700 1 -- 172.21.15.119:0/1655048401 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7db0051d60 con 0x7f7db80b1860 2024-09-13T16:31:22.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.189+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7d9c005b40 con 0x7f7db80b1860 2024-09-13T16:31:22.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7daf7fe700 1 -- 172.21.15.119:0/1655048401 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v86) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7db0018020 con 0x7f7db80b1860 2024-09-13T16:31:22.195 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d98066730 msgr2=0x7f7d98068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d98066730 0x7f7d98068bf0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f7da80097a0 tx=0x7f7da8008030 comp rx=0 tx=0).stop 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80b1860 msgr2=0x7f7db8150030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80b1860 0x7f7db8150030 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f7db000efc0 tx=0x7f7db000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 shutdown_connections 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d98066730 0x7f7d98068bf0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7db80b1860 0x7f7db8150030 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7db80ad9a0 0x7f7db81519d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 --2- 172.21.15.119:0/1655048401 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7db806ca60 0x7f7db81421f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 >> 172.21.15.119:0/1655048401 conn(0x7f7db80b0980 msgr2=0x7f7db806fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:22.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 shutdown_connections 2024-09-13T16:31:22.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.193+0000 7f7db7fff700 1 -- 172.21.15.119:0/1655048401 wait complete. 2024-09-13T16:31:22.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-09-13T16:31:22.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2f98700 1 Processor -- start 2024-09-13T16:31:22.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2f98700 1 -- start start 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2f98700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4aac154fd0 0x7f4aac0ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2f98700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac0ced70 0x7f4aac0d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2f98700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4aac15c750 con 0x7f4aac0d0f60 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2f98700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4aac0cf1b0 con 0x7f4aac154fd0 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2f98700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0d1750 con 0x7f4aac0ced70 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1f96700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4aac154fd0 0x7f4aac0ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2797700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4aac0d0f60 0x7f4aac06e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41282/0 (socket says 172.21.15.119:41282) 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1f96700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4aac154fd0 0x7f4aac0ce830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34354/0 (socket says 172.21.15.119:34354) 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab2797700 1 -- 172.21.15.119:0/2221751832 learned_addr learned my addr 172.21.15.119:0/2221751832 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1795700 1 --2- 172.21.15.119:0/2221751832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac0ced70 0x7f4aac0d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1f96700 1 -- 172.21.15.119:0/2221751832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac0ced70 msgr2=0x7f4aac0d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1f96700 1 --2- 172.21.15.119:0/2221751832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac0ced70 0x7f4aac0d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1f96700 1 -- 172.21.15.119:0/2221751832 >> v1:172.21.15.119:6789/0 conn(0x7f4aac0d0f60 legacy=0x7f4aac06e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1f96700 1 -- 172.21.15.119:0/2221751832 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4aac15c100 con 0x7f4aac154fd0 2024-09-13T16:31:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab1f96700 1 --2- 172.21.15.119:0/2221751832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac154fd0 0x7f4aac0ce830 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009d90 tx=0x7f4a9400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=194bac4f890cd914 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab0f94700 1 -- 172.21.15.119:0/2221751832 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a9400e070 con 0x7f4aac154fd0 2024-09-13T16:31:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab0f94700 1 -- 172.21.15.119:0/2221751832 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a9400bbd0 con 0x7f4aac154fd0 2024-09-13T16:31:22.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.321+0000 7f4ab0f94700 1 -- 172.21.15.119:0/2221751832 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94012430 con 0x7f4aac154fd0 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- 172.21.15.119:0/2221751832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac154fd0 msgr2=0x7f4aac0ce830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/2221751832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac154fd0 0x7f4aac0ce830 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009d90 tx=0x7f4a9400b330 comp rx=0 tx=0).stop 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- 172.21.15.119:0/2221751832 shutdown_connections 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/2221751832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac0ced70 0x7f4aac0d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/2221751832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac154fd0 0x7f4aac0ce830 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- 172.21.15.119:0/2221751832 >> 172.21.15.119:0/2221751832 conn(0x7f4aac099a50 msgr2=0x7f4aac099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- 172.21.15.119:0/2221751832 shutdown_connections 2024-09-13T16:31:22.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- 172.21.15.119:0/2221751832 wait complete. 2024-09-13T16:31:22.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 Processor -- start 2024-09-13T16:31:22.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- start start 2024-09-13T16:31:22.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac0ced70 0x7f4aac0b5860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aac0d0f60 0x7f4aac148000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac154fd0 0x7f4aac159850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac1f2560 con 0x7f4aac0ced70 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac148b40 con 0x7f4aac0d0f60 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2f98700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4aac0b5da0 con 0x7f4aac154fd0 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac0ced70 0x7f4aac0b5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac0ced70 0x7f4aac0b5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34368/0 (socket says 172.21.15.119:34368) 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 -- 172.21.15.119:0/643626861 learned_addr learned my addr 172.21.15.119:0/643626861 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1795700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aac0d0f60 0x7f4aac148000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab2797700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac154fd0 0x7f4aac159850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 -- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac154fd0 msgr2=0x7f4aac159850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac154fd0 0x7f4aac159850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 -- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aac0d0f60 msgr2=0x7f4aac148000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aac0d0f60 0x7f4aac148000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:22.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1f96700 1 -- 172.21.15.119:0/643626861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a94009a70 con 0x7f4aac0ced70 2024-09-13T16:31:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.325+0000 7f4ab1795700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aac0d0f60 0x7f4aac148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4ab1f96700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac0ced70 0x7f4aac0b5860 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4a94014040 tx=0x7f4a94004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4aaa7fc700 1 -- 172.21.15.119:0/643626861 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94012430 con 0x7f4aac0ced70 2024-09-13T16:31:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4aaa7fc700 1 -- 172.21.15.119:0/643626861 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a94020070 con 0x7f4aac0ced70 2024-09-13T16:31:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4aac0b6060 con 0x7f4aac0ced70 2024-09-13T16:31:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4aaa7fc700 1 -- 172.21.15.119:0/643626861 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a94004280 con 0x7f4aac0ced70 2024-09-13T16:31:22.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4aac0b6620 con 0x7f4aac0ced70 2024-09-13T16:31:22.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4aac0d0a20 con 0x7f4aac0ced70 2024-09-13T16:31:22.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4aaa7fc700 1 -- 172.21.15.119:0/643626861 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4a940043e0 con 0x7f4aac0ced70 2024-09-13T16:31:22.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4aaa7fc700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a8c066730 0x7f4a8c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:22.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4aaa7fc700 1 -- 172.21.15.119:0/643626861 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f4a94022070 con 0x7f4aac0ced70 2024-09-13T16:31:22.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4ab1795700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a8c066730 0x7f4a8c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:22.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.329+0000 7f4ab1795700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a8c066730 0x7f4a8c068bf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4aac06ce60 tx=0x7f4aa4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:22.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.337+0000 7f4aaa7fc700 1 -- 172.21.15.119:0/643626861 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4a94053990 con 0x7f4aac0ced70 2024-09-13T16:31:22.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:22.569+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) v1 -- 0x7f4aac1529e0 con 0x7f4aac0ced70 2024-09-13T16:31:23.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.541+0000 7f4aaa7fc700 1 -- 172.21.15.119:0/643626861 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v300) v1 ==== 170+0+0 (secure 0 0 0) 0x7f4a94056fb0 con 0x7f4aac0ced70 2024-09-13T16:31:23.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.545+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a8c066730 msgr2=0x7f4a8c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.545+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a8c066730 0x7f4a8c068bf0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f4aac06ce60 tx=0x7f4aa4008040 comp rx=0 tx=0).stop 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.545+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac0ced70 msgr2=0x7f4aac0b5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.545+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac0ced70 0x7f4aac0b5860 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7f4a94014040 tx=0x7f4a94004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 shutdown_connections 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a8c066730 0x7f4a8c068bf0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4aac154fd0 0x7f4aac159850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4aac0d0f60 0x7f4aac148000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 --2- 172.21.15.119:0/643626861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4aac0ced70 0x7f4aac0b5860 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 >> 172.21.15.119:0/643626861 conn(0x7f4aac099a50 msgr2=0x7f4aac0aea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 shutdown_connections 2024-09-13T16:31:23.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.549+0000 7f4ab2f98700 1 -- 172.21.15.119:0/643626861 wait complete. 2024-09-13T16:31:23.572 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T16:31:23.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.693+0000 7fcd7d66b700 1 Processor -- start 2024-09-13T16:31:23.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.693+0000 7fcd7d66b700 1 -- start start 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7806e700 0x7fcd7815d260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcd7815d7a0 0x7fcd7806b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd7815caa0 con 0x7fcd7815c610 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd780ae9b0 con 0x7fcd7815d7a0 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd780ce140 con 0x7fcd7806e700 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcd7815d7a0 0x7fcd7806b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcd7815d7a0 0x7fcd7806b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56722/0 (socket says 172.21.15.119:56722) 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 -- 172.21.15.119:0/1843631107 learned_addr learned my addr 172.21.15.119:0/1843631107 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd6f7fe700 1 --2- 172.21.15.119:0/1843631107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7806e700 0x7fcd7815d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:23.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 -- 172.21.15.119:0/1843631107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7806e700 msgr2=0x7fcd7815d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 --2- 172.21.15.119:0/1843631107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7806e700 0x7fcd7815d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 -- 172.21.15.119:0/1843631107 >> v1:172.21.15.119:6789/0 conn(0x7fcd7815c610 legacy=0x7fcd780b3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 -- 172.21.15.119:0/1843631107 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd780ce420 con 0x7fcd7815d7a0 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd6f7fe700 1 --2- 172.21.15.119:0/1843631107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7806e700 0x7fcd7815d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7ce6a700 1 --2- 172.21.15.119:0/1843631107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7815d7a0 0x7fcd7806b360 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000af60 tx=0x7fcd7000af90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70c4277cf8ea1d0c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd777fe700 1 -- 172.21.15.119:0/1843631107 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7000f070 con 0x7fcd7815d7a0 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd777fe700 1 -- 172.21.15.119:0/1843631107 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd7000a2e0 con 0x7fcd7815d7a0 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd777fe700 1 -- 172.21.15.119:0/1843631107 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd7000ccb0 con 0x7fcd7815d7a0 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- 172.21.15.119:0/1843631107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7815d7a0 msgr2=0x7fcd7806b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/1843631107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7815d7a0 0x7fcd7806b360 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7fcd7000af60 tx=0x7fcd7000af90 comp rx=0 tx=0).stop 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- 172.21.15.119:0/1843631107 shutdown_connections 2024-09-13T16:31:23.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/1843631107 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7815d7a0 0x7fcd7806b360 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/1843631107 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7806e700 0x7fcd7815d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- 172.21.15.119:0/1843631107 >> 172.21.15.119:0/1843631107 conn(0x7fcd780cb8f0 msgr2=0x7fcd780cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- 172.21.15.119:0/1843631107 shutdown_connections 2024-09-13T16:31:23.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.697+0000 7fcd7d66b700 1 -- 172.21.15.119:0/1843631107 wait complete. 2024-09-13T16:31:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 Processor -- start 2024-09-13T16:31:23.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 -- start start 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7806e700 0x7fcd7809a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd7815c610 0x7fcd78150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7815d7a0 0x7fcd78159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd78150e50 con 0x7fcd7806e700 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7809bb40 con 0x7fcd7815c610 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7d66b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7809af30 con 0x7fcd7815d7a0 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd77fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7806e700 0x7fcd7809a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd77fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7806e700 0x7fcd7809a710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56728/0 (socket says 172.21.15.119:56728) 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd77fff700 1 -- 172.21.15.119:0/2270894301 learned_addr learned my addr 172.21.15.119:0/2270894301 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:23.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd7ce6a700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7815d7a0 0x7fcd78159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.701+0000 7fcd6f7fe700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd7815c610 0x7fcd78150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd77fff700 1 -- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7815d7a0 msgr2=0x7fcd78159ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd77fff700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7815d7a0 0x7fcd78159ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd77fff700 1 -- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd7815c610 msgr2=0x7fcd78150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd77fff700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd7815c610 0x7fcd78150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd77fff700 1 -- 172.21.15.119:0/2270894301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd7000ac40 con 0x7fcd7806e700 2024-09-13T16:31:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd77fff700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7806e700 0x7fcd7809a710 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fcd6800e940 tx=0x7fcd6800ec50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd757fa700 1 -- 172.21.15.119:0/2270894301 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68006a80 con 0x7fcd7806e700 2024-09-13T16:31:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd757fa700 1 -- 172.21.15.119:0/2270894301 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd68006be0 con 0x7fcd7806e700 2024-09-13T16:31:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd757fa700 1 -- 172.21.15.119:0/2270894301 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd68007030 con 0x7fcd7806e700 2024-09-13T16:31:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7809b210 con 0x7fcd7806e700 2024-09-13T16:31:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd7809b7d0 con 0x7fcd7806e700 2024-09-13T16:31:23.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd7815d260 con 0x7fcd7806e700 2024-09-13T16:31:23.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.705+0000 7fcd757fa700 1 -- 172.21.15.119:0/2270894301 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcd68007190 con 0x7fcd7806e700 2024-09-13T16:31:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.709+0000 7fcd757fa700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd58066760 0x7fcd58068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.709+0000 7fcd757fa700 1 -- 172.21.15.119:0/2270894301 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5349+0+0 (secure 0 0 0) 0x7fcd68086230 con 0x7fcd7806e700 2024-09-13T16:31:23.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.709+0000 7fcd6f7fe700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd58066760 0x7fcd58068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:23.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.709+0000 7fcd6f7fe700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd58066760 0x7fcd58068c20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcd780b1ef0 tx=0x7fcd60008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:23.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.713+0000 7fcd757fa700 1 -- 172.21.15.119:0/2270894301 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcd68053870 con 0x7fcd7806e700 2024-09-13T16:31:23.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.885+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fcd780cca50 con 0x7fcd7806e700 2024-09-13T16:31:23.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd757fa700 1 -- 172.21.15.119:0/2270894301 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fcd6801f0c0 con 0x7fcd7806e700 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd58066760 msgr2=0x7fcd58068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd58066760 0x7fcd58068c20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fcd780b1ef0 tx=0x7fcd60008040 comp rx=0 tx=0).stop 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7806e700 msgr2=0x7fcd7809a710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7806e700 0x7fcd7809a710 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7fcd6800e940 tx=0x7fcd6800ec50 comp rx=0 tx=0).stop 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 shutdown_connections 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd58066760 0x7fcd58068c20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd7815d7a0 0x7fcd78159ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd7815c610 0x7fcd78150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 --2- 172.21.15.119:0/2270894301 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd7806e700 0x7fcd7809a710 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 >> 172.21.15.119:0/2270894301 conn(0x7fcd780cb8f0 msgr2=0x7fcd7806a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 shutdown_connections 2024-09-13T16:31:23.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:23.893+0000 7fcd7d66b700 1 -- 172.21.15.119:0/2270894301 wait complete. 2024-09-13T16:31:23.912 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:31:23.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:31:23.931 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:31:23.953 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:31:23.953 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:31:23.953 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:23.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:31:23.983 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:31:23.983 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:24.001 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:31:24.001 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:31:24.001 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:24.008 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:31:24.008 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:31:24.009 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:24.014 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:31:24.014 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T16:31:24.015 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:24.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bedf2700 1 Processor -- start 2024-09-13T16:31:24.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bedf2700 1 -- start start 2024-09-13T16:31:24.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bedf2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f69b80cc220 0x7f69b815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bedf2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b815a910 0x7f69b80ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bedf2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b80b44e0 con 0x7f69b81509e0 2024-09-13T16:31:24.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bedf2700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80cddf0 con 0x7f69b80cc220 2024-09-13T16:31:24.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bedf2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8071a50 con 0x7f69b815a910 2024-09-13T16:31:24.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b815a910 0x7f69b80ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bd5ef700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f69b80cc220 0x7f69b815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b815a910 0x7f69b80ce2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34296/0 (socket says 172.21.15.119:34296) 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 -- 172.21.15.119:0/2080179913 learned_addr learned my addr 172.21.15.119:0/2080179913 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 -- 172.21.15.119:0/2080179913 >> v2:172.21.15.145:3300/0 conn(0x7f69b80cc220 msgr2=0x7f69b815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 --2- 172.21.15.119:0/2080179913 >> v2:172.21.15.145:3300/0 conn(0x7f69b80cc220 0x7f69b815a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 -- 172.21.15.119:0/2080179913 >> v1:172.21.15.145:6789/0 conn(0x7f69b81509e0 legacy=0x7f69b8159cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 -- 172.21.15.119:0/2080179913 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b80cc750 con 0x7f69b815a910 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bd5ef700 1 --2- 172.21.15.119:0/2080179913 >> v2:172.21.15.145:3300/0 conn(0x7f69b80cc220 0x7f69b815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69be5f1700 1 --2- 172.21.15.119:0/2080179913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b815a910 0x7f69b80ce2d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009e50 tx=0x7f69b000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30ad51d742fba406 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.133+0000 7f69bcdee700 1 -- 172.21.15.119:0/2080179913 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000e070 con 0x7f69b815a910 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bcdee700 1 -- 172.21.15.119:0/2080179913 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b000b910 con 0x7f69b815a910 2024-09-13T16:31:24.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bcdee700 1 -- 172.21.15.119:0/2080179913 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000bcc0 con 0x7f69b815a910 2024-09-13T16:31:24.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 -- 172.21.15.119:0/2080179913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b815a910 msgr2=0x7f69b80ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2080179913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b815a910 0x7f69b80ce2d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009e50 tx=0x7f69b000b330 comp rx=0 tx=0).stop 2024-09-13T16:31:24.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 -- 172.21.15.119:0/2080179913 shutdown_connections 2024-09-13T16:31:24.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2080179913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b815a910 0x7f69b80ce2d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2080179913 >> v2:172.21.15.145:3300/0 conn(0x7f69b80cc220 0x7f69b815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 -- 172.21.15.119:0/2080179913 >> 172.21.15.119:0/2080179913 conn(0x7f69b8070cc0 msgr2=0x7f69b80710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:24.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 -- 172.21.15.119:0/2080179913 shutdown_connections 2024-09-13T16:31:24.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 -- 172.21.15.119:0/2080179913 wait complete. 2024-09-13T16:31:24.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 Processor -- start 2024-09-13T16:31:24.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.137+0000 7f69bedf2700 1 -- start start 2024-09-13T16:31:24.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b80cc220 0x7f69b8147c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b81509e0 0x7f69b8156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b815a910 0x7f69b8152ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8153010 con 0x7f69b815a910 2024-09-13T16:31:24.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8153220 con 0x7f69b81509e0 2024-09-13T16:31:24.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b8153500 con 0x7f69b80cc220 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b815a910 0x7f69b8152ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b815a910 0x7f69b8152ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56738/0 (socket says 172.21.15.119:56738) 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 -- 172.21.15.119:0/2258983918 learned_addr learned my addr 172.21.15.119:0/2258983918 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bddf0700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b80cc220 0x7f69b8147c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bd5ef700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b81509e0 0x7f69b8156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 -- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b80cc220 msgr2=0x7f69b8147c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b80cc220 0x7f69b8147c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 -- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b81509e0 msgr2=0x7f69b8156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b81509e0 0x7f69b8156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 -- 172.21.15.119:0/2258983918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b40094b0 con 0x7f69b815a910 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69be5f1700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b815a910 0x7f69b8152ad0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f69b0012000 tx=0x7f69b0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69ae7fc700 1 -- 172.21.15.119:0/2258983918 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0004700 con 0x7f69b815a910 2024-09-13T16:31:24.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69ae7fc700 1 -- 172.21.15.119:0/2258983918 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b0004860 con 0x7f69b815a910 2024-09-13T16:31:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b0009b30 con 0x7f69b815a910 2024-09-13T16:31:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69ae7fc700 1 -- 172.21.15.119:0/2258983918 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b001b770 con 0x7f69b815a910 2024-09-13T16:31:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.141+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b8153ac0 con 0x7f69b815a910 2024-09-13T16:31:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.145+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b80ce2d0 con 0x7f69b815a910 2024-09-13T16:31:24.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.145+0000 7f69ae7fc700 1 -- 172.21.15.119:0/2258983918 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f69b0021070 con 0x7f69b815a910 2024-09-13T16:31:24.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.145+0000 7f69ae7fc700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f699c066730 0x7f699c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.145+0000 7f69ae7fc700 1 -- 172.21.15.119:0/2258983918 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f69b0087740 con 0x7f69b815a910 2024-09-13T16:31:24.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.145+0000 7f69bddf0700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f699c066730 0x7f699c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.145+0000 7f69bddf0700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f699c066730 0x7f699c068bf0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f69a80016e0 tx=0x7f69a8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:24.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.149+0000 7f69ae7fc700 1 -- 172.21.15.119:0/2258983918 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f69b0054ca0 con 0x7f69b815a910 2024-09-13T16:31:24.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69b8072c50 con 0x7f69b815a910 2024-09-13T16:31:24.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69ae7fc700 1 -- 172.21.15.119:0/2258983918 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+3535 (secure 0 0 0) 0x7f69b00582c0 con 0x7f69b815a910 2024-09-13T16:31:24.360 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:24.360 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6206,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1451008584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1451008584},{"type":"v1","addr":"172.21.15.145:6835","nonce":1451008584}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":6838,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/3469294068","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":3469294068},{"type":"v1","addr":"172.21.15.145:6837","nonce":3469294068}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7098,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/25360333","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":25360333},{"type":"v1","addr":"172.21.15.119:6833","nonce":25360333}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":37},{"gid":7791,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2737651640","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2737651640},{"type":"v1","addr":"172.21.15.119:6835","nonce":2737651640}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":51}],"filesystems":[]} 2024-09-13T16:31:24.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f699c066730 msgr2=0x7f699c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f699c066730 0x7f699c068bf0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f69a80016e0 tx=0x7f69a8009040 comp rx=0 tx=0).stop 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b815a910 msgr2=0x7f69b8152ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b815a910 0x7f69b8152ad0 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f69b0012000 tx=0x7f69b0004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 shutdown_connections 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f699c066730 0x7f699c068bf0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f69b815a910 0x7f69b8152ad0 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f69b81509e0 0x7f69b8156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 --2- 172.21.15.119:0/2258983918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f69b80cc220 0x7f69b8147c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 >> 172.21.15.119:0/2258983918 conn(0x7f69b8070cc0 msgr2=0x7f69b806e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 shutdown_connections 2024-09-13T16:31:24.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.357+0000 7f69bedf2700 1 -- 172.21.15.119:0/2258983918 wait complete. 2024-09-13T16:31:24.364 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 51 2024-09-13T16:31:24.375 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:31:24.375 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T16:31:24.375 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:31:24.375 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:31:24.376 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:31:24.376 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T16:31:24.376 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:31:24.376 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:31:24.376 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:31:24.376 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T16:31:24.376 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:31:24.377 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:31:24.377 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:31:24.377 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T16:31:24.377 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:31:24.377 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:31:24.391 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:31:24.391 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:31:24.393 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:31:24.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:31:24.394 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:31:24.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:31:24.395 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:31:24.395 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:31:24.396 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:31:24.397 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:31:24.397 DEBUG:teuthology.parallel:result is None 2024-09-13T16:31:24.397 DEBUG:teuthology.parallel:result is None 2024-09-13T16:31:24.398 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:31:24.398 DEBUG:teuthology.parallel:result is None 2024-09-13T16:31:24.398 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:31:24.399 DEBUG:teuthology.parallel:result is None 2024-09-13T16:31:24.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T16:31:24.462 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:31:24.473 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:31:24.486 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:31:24.490 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:31:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.497+0000 7f78a2c2d700 1 Processor -- start 2024-09-13T16:31:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.497+0000 7f78a2c2d700 1 -- start start 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a2c2d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f789c158e00 0x7f789c0acd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a2c2d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f789c0cd9b0 0x7f789c0abed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a2c2d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c06e620 0x7f789c0ac410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a2c2d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c0ad270 con 0x7f789c0cd9b0 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a2c2d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f789c06eb90 con 0x7f789c158e00 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a2c2d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0cced0 con 0x7f789c06e620 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f789c0cd9b0 0x7f789c0abed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f789c0cd9b0 0x7f789c0abed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56760/0 (socket says 172.21.15.119:56760) 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 -- 172.21.15.119:0/2136079142 learned_addr learned my addr 172.21.15.119:0/2136079142 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a1c2b700 1 --2- 172.21.15.119:0/2136079142 >> v2:172.21.15.145:3300/0 conn(0x7f789c158e00 0x7f789c0acd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a242c700 1 --2- 172.21.15.119:0/2136079142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c06e620 0x7f789c0ac410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 -- 172.21.15.119:0/2136079142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c06e620 msgr2=0x7f789c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:24.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 --2- 172.21.15.119:0/2136079142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c06e620 0x7f789c0ac410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 -- 172.21.15.119:0/2136079142 >> v2:172.21.15.145:3300/0 conn(0x7f789c158e00 msgr2=0x7f789c0acd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:24.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 --2- 172.21.15.119:0/2136079142 >> v2:172.21.15.145:3300/0 conn(0x7f789c158e00 0x7f789c0acd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 -- 172.21.15.119:0/2136079142 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f789c0cd1b0 con 0x7f789c0cd9b0 2024-09-13T16:31:24.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a242c700 1 --2- 172.21.15.119:0/2136079142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c06e620 0x7f789c0ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:24.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a142a700 1 --2- 172.21.15.119:0/2136079142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c0cd9b0 0x7f789c0abed0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f789c0cff00 tx=0x7f788c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51163b60d0d126d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:24.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a0c29700 1 -- 172.21.15.119:0/2136079142 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c00da00 con 0x7f789c0cd9b0 2024-09-13T16:31:24.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a0c29700 1 -- 172.21.15.119:0/2136079142 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788c00db60 con 0x7f789c0cd9b0 2024-09-13T16:31:24.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.501+0000 7f78a0c29700 1 -- 172.21.15.119:0/2136079142 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788c010470 con 0x7f789c0cd9b0 2024-09-13T16:31:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2136079142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c0cd9b0 msgr2=0x7f789c0abed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2136079142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c0cd9b0 0x7f789c0abed0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f789c0cff00 tx=0x7f788c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2136079142 shutdown_connections 2024-09-13T16:31:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2136079142 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c06e620 0x7f789c0ac410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2136079142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c0cd9b0 0x7f789c0abed0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2136079142 >> v2:172.21.15.145:3300/0 conn(0x7f789c158e00 0x7f789c0acd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2136079142 >> 172.21.15.119:0/2136079142 conn(0x7f789c0ae4b0 msgr2=0x7f789c0b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2136079142 shutdown_connections 2024-09-13T16:31:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.505+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2136079142 wait complete. 2024-09-13T16:31:24.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 Processor -- start 2024-09-13T16:31:24.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 -- start start 2024-09-13T16:31:24.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f789c06e620 0x7f789c1e16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c0cd9b0 0x7f789c1e1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c158e00 0x7f789c143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c0b0480 con 0x7f789c158e00 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f789c1416e0 con 0x7f789c06e620 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a2c2d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f789c1419c0 con 0x7f789c0cd9b0 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c158e00 0x7f789c143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c158e00 0x7f789c143660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56768/0 (socket says 172.21.15.119:56768) 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 -- 172.21.15.119:0/2272433194 learned_addr learned my addr 172.21.15.119:0/2272433194 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a142a700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c0cd9b0 0x7f789c1e1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a1c2b700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f789c06e620 0x7f789c1e16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 -- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c0cd9b0 msgr2=0x7f789c1e1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c0cd9b0 0x7f789c1e1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 -- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f789c06e620 msgr2=0x7f789c1e16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f789c06e620 0x7f789c1e16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 -- 172.21.15.119:0/2272433194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f788c009570 con 0x7f789c158e00 2024-09-13T16:31:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.509+0000 7f78a242c700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c158e00 0x7f789c143660 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7894009fa0 tx=0x7f789400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78927fc700 1 -- 172.21.15.119:0/2272433194 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f789400ddb0 con 0x7f789c158e00 2024-09-13T16:31:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78927fc700 1 -- 172.21.15.119:0/2272433194 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f789400f040 con 0x7f789c158e00 2024-09-13T16:31:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f789c141ca0 con 0x7f789c158e00 2024-09-13T16:31:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78927fc700 1 -- 172.21.15.119:0/2272433194 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7894003c80 con 0x7f789c158e00 2024-09-13T16:31:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f789c159e90 con 0x7f789c158e00 2024-09-13T16:31:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f789c15a080 con 0x7f789c158e00 2024-09-13T16:31:24.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78927fc700 1 -- 172.21.15.119:0/2272433194 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7894004030 con 0x7f789c158e00 2024-09-13T16:31:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78927fc700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7880066760 0x7f7880068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78927fc700 1 -- 172.21.15.119:0/2272433194 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f7894084030 con 0x7f789c158e00 2024-09-13T16:31:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.513+0000 7f78a1c2b700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7880066760 0x7f7880068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:24.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.517+0000 7f78a1c2b700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7880066760 0x7f7880068c20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7898009ad0 tx=0x7f7898009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:24.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.525+0000 7f78927fc700 1 -- 172.21.15.119:0/2272433194 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7894051500 con 0x7f789c158e00 2024-09-13T16:31:24.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:24.697+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f789c1421c0 con 0x7f789c158e00 2024-09-13T16:31:25.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.561+0000 7f78927fc700 1 -- 172.21.15.119:0/2272433194 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v91) v1 ==== 97+0+0 (secure 0 0 0) 0x7f789c1421c0 con 0x7f789c158e00 2024-09-13T16:31:25.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7880066760 msgr2=0x7f7880068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7880066760 0x7f7880068c20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f7898009ad0 tx=0x7f7898009240 comp rx=0 tx=0).stop 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c158e00 msgr2=0x7f789c143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c158e00 0x7f789c143660 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 crypto rx=0x7f7894009fa0 tx=0x7f789400d3b0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 shutdown_connections 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7880066760 0x7f7880068c20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f789c158e00 0x7f789c143660 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f789c0cd9b0 0x7f789c1e1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 --2- 172.21.15.119:0/2272433194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f789c06e620 0x7f789c1e16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 >> 172.21.15.119:0/2272433194 conn(0x7f789c0ae4b0 msgr2=0x7f789c06d2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 shutdown_connections 2024-09-13T16:31:25.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.565+0000 7f78a2c2d700 1 -- 172.21.15.119:0/2272433194 wait complete. 2024-09-13T16:31:25.570 INFO:teuthology.orchestra.run.smithi119.stderr: removed all blocklist entries 2024-09-13T16:31:25.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T16:31:25.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 Processor -- start 2024-09-13T16:31:25.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 -- start start 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c740cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4c7406bb90 0x7f4c740cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4c740cde70 0x7f4c7415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7406cf60 con 0x7f4c7406bb90 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c7406c350 con 0x7f4c740cde70 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c7a06a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7406c630 con 0x7f4c740adb30 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c78867700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4c7406bb90 0x7f4c740cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c78867700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4c7406bb90 0x7f4c740cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56778/0 (socket says 172.21.15.119:56778) 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c78867700 1 -- 172.21.15.119:0/1485563400 learned_addr learned my addr 172.21.15.119:0/1485563400 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c79869700 1 --2- 172.21.15.119:0/1485563400 >> v2:172.21.15.145:3300/0 conn(0x7f4c740cde70 0x7f4c7415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:25.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c79068700 1 --2- 172.21.15.119:0/1485563400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c740cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:25.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c78867700 1 -- 172.21.15.119:0/1485563400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 msgr2=0x7f4c740cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:25.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c78867700 1 --2- 172.21.15.119:0/1485563400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c740cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c78867700 1 -- 172.21.15.119:0/1485563400 >> v2:172.21.15.145:3300/0 conn(0x7f4c740cde70 msgr2=0x7f4c7415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:25.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.697+0000 7f4c78867700 1 --2- 172.21.15.119:0/1485563400 >> v2:172.21.15.145:3300/0 conn(0x7f4c740cde70 0x7f4c7415e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c78867700 1 -- 172.21.15.119:0/1485563400 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7406c910 con 0x7f4c7406bb90 2024-09-13T16:31:25.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c79068700 1 --2- 172.21.15.119:0/1485563400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c740cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:25.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c78867700 1 --2- 172.21.15.119:0/1485563400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7406bb90 0x7f4c740cd930 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f4c740cb1e0 tx=0x7f4c7000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5867bedfa4ab2e08 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:25.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c6bfff700 1 -- 172.21.15.119:0/1485563400 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7000e070 con 0x7f4c7406bb90 2024-09-13T16:31:25.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c6bfff700 1 -- 172.21.15.119:0/1485563400 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7000b9c0 con 0x7f4c7406bb90 2024-09-13T16:31:25.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c6bfff700 1 -- 172.21.15.119:0/1485563400 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7000bd70 con 0x7f4c7406bb90 2024-09-13T16:31:25.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 -- 172.21.15.119:0/1485563400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7406bb90 msgr2=0x7f4c740cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:25.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/1485563400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7406bb90 0x7f4c740cd930 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f4c740cb1e0 tx=0x7f4c7000b330 comp rx=0 tx=0).stop 2024-09-13T16:31:25.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 -- 172.21.15.119:0/1485563400 shutdown_connections 2024-09-13T16:31:25.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/1485563400 >> v2:172.21.15.145:3300/0 conn(0x7f4c740cde70 0x7f4c7415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/1485563400 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c7406bb90 0x7f4c740cd930 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/1485563400 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c740cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 -- 172.21.15.119:0/1485563400 >> 172.21.15.119:0/1485563400 conn(0x7f4c7406a350 msgr2=0x7f4c7406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:25.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 -- 172.21.15.119:0/1485563400 shutdown_connections 2024-09-13T16:31:25.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 -- 172.21.15.119:0/1485563400 wait complete. 2024-09-13T16:31:25.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 Processor -- start 2024-09-13T16:31:25.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.701+0000 7f4c7a06a700 1 -- start start 2024-09-13T16:31:25.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c7406bb90 0x7f4c7415b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:25.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c741599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c740cde70 0x7f4c74159f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c741347a0 con 0x7f4c740cde70 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74133b90 con 0x7f4c7406bb90 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74133e70 con 0x7f4c740adb30 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c740cde70 0x7f4c74159f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c740cde70 0x7f4c74159f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56788/0 (socket says 172.21.15.119:56788) 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 -- 172.21.15.119:0/3335245769 learned_addr learned my addr 172.21.15.119:0/3335245769 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79068700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c7406bb90 0x7f4c7415b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c78867700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c741599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 -- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 msgr2=0x7f4c741599f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c741599f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 -- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c7406bb90 msgr2=0x7f4c7415b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:25.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c7406bb90 0x7f4c7415b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 -- 172.21.15.119:0/3335245769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c70009b30 con 0x7f4c740cde70 2024-09-13T16:31:25.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c79869700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c740cde70 0x7f4c74159f30 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00d800 tx=0x7f4c6c00db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:25.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c69ffb700 1 -- 172.21.15.119:0/3335245769 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c0125f0 con 0x7f4c740cde70 2024-09-13T16:31:25.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c69ffb700 1 -- 172.21.15.119:0/3335245769 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c012790 con 0x7f4c740cde70 2024-09-13T16:31:25.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c74134150 con 0x7f4c740cde70 2024-09-13T16:31:25.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c69ffb700 1 -- 172.21.15.119:0/3335245769 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c012b40 con 0x7f4c740cde70 2024-09-13T16:31:25.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.705+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7415bcf0 con 0x7f4c740cde70 2024-09-13T16:31:25.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.709+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c74134910 con 0x7f4c740cde70 2024-09-13T16:31:25.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.709+0000 7f4c69ffb700 1 -- 172.21.15.119:0/3335245769 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4c6c012ca0 con 0x7f4c740cde70 2024-09-13T16:31:25.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.709+0000 7f4c69ffb700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4c58066730 0x7f4c58068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:25.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.709+0000 7f4c69ffb700 1 -- 172.21.15.119:0/3335245769 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f4c6c083fa0 con 0x7f4c740cde70 2024-09-13T16:31:25.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.709+0000 7f4c79068700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4c58066730 0x7f4c58068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:25.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.709+0000 7f4c79068700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4c58066730 0x7f4c58068bf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4c64006fd0 tx=0x7f4c64008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:25.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.717+0000 7f4c69ffb700 1 -- 172.21.15.119:0/3335245769 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4c6c0b5db0 con 0x7f4c740cde70 2024-09-13T16:31:25.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.949+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4c740adf10 con 0x7f4c740cde70 2024-09-13T16:31:25.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.949+0000 7f4c69ffb700 1 -- 172.21.15.119:0/3335245769 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f4c6c051490 con 0x7f4c740cde70 2024-09-13T16:31:25.954 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:25.954 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_dump": [ 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.a", 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA==", 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.955 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.b", 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g==", 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.c", 2024-09-13T16:31:25.956 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ==", 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.d", 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw==", 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.957 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.0", 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA==", 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.958 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.1", 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw==", 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.2", 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ==", 2024-09-13T16:31:25.959 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.3", 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww==", 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.960 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.4", 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA==", 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.961 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.5", 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q==", 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.6", 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg==", 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.962 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.7", 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ==", 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.963 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.0", 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ==", 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow r", 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.1", 2024-09-13T16:31:25.964 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ==", 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.2", 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw==", 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.965 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.3", 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow==", 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.966 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.admin", 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmpEcAJRAAYorOK9AU5Q/2uOTIUrZhJw==", 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.x", 2024-09-13T16:31:25.967 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ==", 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.y", 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw==", 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:31:25.968 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:31:25.969 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.969 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:25.969 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:31:25.969 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4c58066730 msgr2=0x7f4c58068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4c58066730 0x7f4c58068bf0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f4c64006fd0 tx=0x7f4c64008040 comp rx=0 tx=0).stop 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c740cde70 msgr2=0x7f4c74159f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c740cde70 0x7f4c74159f30 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c00d800 tx=0x7f4c6c00db10 comp rx=0 tx=0).stop 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 shutdown_connections 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4c58066730 0x7f4c58068bf0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4c740cde70 0x7f4c74159f30 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4c740adb30 0x7f4c741599f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 --2- 172.21.15.119:0/3335245769 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4c7406bb90 0x7f4c7415b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:25.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 >> 172.21.15.119:0/3335245769 conn(0x7f4c7406a350 msgr2=0x7f4c74142080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:25.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 shutdown_connections 2024-09-13T16:31:25.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:25.953+0000 7f4c7a06a700 1 -- 172.21.15.119:0/3335245769 wait complete. 2024-09-13T16:31:25.975 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:26.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983ef4d700 1 Processor -- start 2024-09-13T16:31:26.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983ef4d700 1 -- start start 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983ef4d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 0x7f9838070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983ef4d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9838071210 con 0x7f983815ec20 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983ef4d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380714f0 con 0x7f983815e840 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983ef4d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98380717d0 con 0x7f9838165180 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f9837fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f983815ec20 0x7f98380ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48426/0 (socket says 172.21.15.119:48426) 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f9837fff700 1 -- 172.21.15.119:0/2893490825 learned_addr learned my addr 172.21.15.119:0/2893490825 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983d4ea700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 0x7f9838070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573748857 0 0) 0x7f9838071210 con 0x7f983815ec20 2024-09-13T16:31:26.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98240035d0 con 0x7f983815ec20 2024-09-13T16:31:26.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 57570636 0 0) 0x7f98240035d0 con 0x7f983815ec20 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 msgr2=0x7f9838070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 0x7f9838070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 >> v1:172.21.15.145:6789/0 conn(0x7f983815e840 legacy=0x7f9838164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838071ab0 con 0x7f983815ec20 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f98300019f0 con 0x7f983815ec20 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 >> v1:172.21.15.119:6789/0 conn(0x7f983815ec20 legacy=0x7f98380ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9824003a50 0x7f9824003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 0x7f9824004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824004db0 0x7f9824005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98380717d0 con 0x7f9824004db0 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982400bb10 con 0x7f9824003a50 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982400bcd0 con 0x7f9838165180 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983d4ea700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824004db0 0x7f9824005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f9837fff700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 0x7f9824004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983cce9700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9824003a50 0x7f9824003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983d4ea700 1 -- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 msgr2=0x7f9824004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983d4ea700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9838165180 0x7f9824004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983d4ea700 1 -- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9824003a50 msgr2=0x7f9824003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983d4ea700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9824003a50 0x7f9824003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.089+0000 7f983d4ea700 1 -- 172.21.15.119:0/2893490825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98240035d0 con 0x7f9824004db0 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983d4ea700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824004db0 0x7f9824005230 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f98381614c0 tx=0x7f98240058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e79f3f23715b2a7c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000e6c0 con 0x7f9824004db0 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f982000e820 con 0x7f9824004db0 2024-09-13T16:31:26.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f98377fe700 1 -- 172.21.15.119:0/2893490825 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f982000ebd0 con 0x7f9824004db0 2024-09-13T16:31:26.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983ef4d700 1 -- 172.21.15.119:0/2893490825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824004db0 msgr2=0x7f9824005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983ef4d700 1 --2- 172.21.15.119:0/2893490825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824004db0 0x7f9824005230 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f98381614c0 tx=0x7f98240058a0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983d4ea700 1 -- 172.21.15.119:0/2893490825 reap_dead start 2024-09-13T16:31:26.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983ef4d700 1 -- 172.21.15.119:0/2893490825 shutdown_connections 2024-09-13T16:31:26.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983ef4d700 1 -- 172.21.15.119:0/2893490825 >> 172.21.15.119:0/2893490825 conn(0x7f983806a350 msgr2=0x7f983806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:26.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983ef4d700 1 -- 172.21.15.119:0/2893490825 shutdown_connections 2024-09-13T16:31:26.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.093+0000 7f983ef4d700 1 -- 172.21.15.119:0/2893490825 wait complete. 2024-09-13T16:31:26.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 Processor -- start 2024-09-13T16:31:26.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 -- start start 2024-09-13T16:31:26.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98380adcb0 0x7f9838159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98381f6b30 0x7f983815b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98381f6fb0 0x7f983815b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9838159d80 con 0x7f98381f6b30 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983815bdd0 con 0x7f98381f6fb0 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9838159560 con 0x7f98380adcb0 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98381f6b30 0x7f983815b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98381f6b30 0x7f983815b350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56800/0 (socket says 172.21.15.119:56800) 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 -- 172.21.15.119:0/1676838902 learned_addr learned my addr 172.21.15.119:0/1676838902 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983cce9700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98380adcb0 0x7f9838159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983d4ea700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98381f6fb0 0x7f983815b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 -- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98380adcb0 msgr2=0x7f9838159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98380adcb0 0x7f9838159840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 -- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98381f6fb0 msgr2=0x7f983815b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98381f6fb0 0x7f983815b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 -- 172.21.15.119:0/1676838902 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f983815f140 con 0x7f98381f6b30 2024-09-13T16:31:26.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f9837fff700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98381f6b30 0x7f983815b350 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f9830001560 tx=0x7f9830007ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f98357fa700 1 -- 172.21.15.119:0/1676838902 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f983000aad0 con 0x7f98381f6b30 2024-09-13T16:31:26.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f98357fa700 1 -- 172.21.15.119:0/1676838902 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983000ac50 con 0x7f98381f6b30 2024-09-13T16:31:26.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983815ae30 con 0x7f98381f6b30 2024-09-13T16:31:26.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f98357fa700 1 -- 172.21.15.119:0/1676838902 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98300133f0 con 0x7f98381f6b30 2024-09-13T16:31:26.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.097+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9838161140 con 0x7f98381f6b30 2024-09-13T16:31:26.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.101+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983815b070 con 0x7f98381f6b30 2024-09-13T16:31:26.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.101+0000 7f98357fa700 1 -- 172.21.15.119:0/1676838902 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9830013550 con 0x7f98381f6b30 2024-09-13T16:31:26.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.101+0000 7f98357fa700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9818066730 0x7f9818068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.101+0000 7f98357fa700 1 -- 172.21.15.119:0/1676838902 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f9830084900 con 0x7f98381f6b30 2024-09-13T16:31:26.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.101+0000 7f983cce9700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9818066730 0x7f9818068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.101+0000 7f983cce9700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9818066730 0x7f9818068bf0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f983815c7a0 tx=0x7f9828009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.109+0000 7f98357fa700 1 -- 172.21.15.119:0/1676838902 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f983815b070 con 0x7f98381f6b30 2024-09-13T16:31:26.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.277+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f983815b070 con 0x7f98381f6b30 2024-09-13T16:31:26.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.277+0000 7f98357fa700 1 -- 172.21.15.119:0/1676838902 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+13123 (secure 0 0 0) 0x7f983815b070 con 0x7f98381f6b30 2024-09-13T16:31:26.282 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:26.282 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":91,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:31:25.549671+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:26.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9818066730 msgr2=0x7f9818068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9818066730 0x7f9818068bf0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f983815c7a0 tx=0x7f9828009040 comp rx=0 tx=0).stop 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98381f6b30 msgr2=0x7f983815b350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98381f6b30 0x7f983815b350 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f9830001560 tx=0x7f9830007ca0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 shutdown_connections 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9818066730 0x7f9818068bf0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98381f6fb0 0x7f983815b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98381f6b30 0x7f983815b350 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 --2- 172.21.15.119:0/1676838902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98380adcb0 0x7f9838159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 >> 172.21.15.119:0/1676838902 conn(0x7f983806a350 msgr2=0x7f983815d860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 shutdown_connections 2024-09-13T16:31:26.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.281+0000 7f983ef4d700 1 -- 172.21.15.119:0/1676838902 wait complete. 2024-09-13T16:31:26.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:31:26.297 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:31:26.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.409+0000 7efe34da0700 1 Processor -- start 2024-09-13T16:31:26.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.409+0000 7efe34da0700 1 -- start start 2024-09-13T16:31:26.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300ce8c0 0x7efe300d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe300d1610 0x7efe300b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe300d0df0 con 0x7efe3015cc00 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe300d1a80 con 0x7efe300d1610 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe300cdb00 con 0x7efe300ce8c0 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2ffff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe300d1610 0x7efe300b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe26ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300ce8c0 0x7efe300d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2f7fe700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7efe3015cc00 0x7efe300d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:52168/0 (socket says 172.21.15.119:52168) 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2ffff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe300d1610 0x7efe300b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:45428/0 (socket says 172.21.15.119:45428) 2024-09-13T16:31:26.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2f7fe700 1 -- 172.21.15.119:0/765314859 learned_addr learned my addr 172.21.15.119:0/765314859 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe26ffd700 1 -- 172.21.15.119:0/765314859 >> v2:172.21.15.145:3300/0 conn(0x7efe300d1610 msgr2=0x7efe300b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe26ffd700 1 --2- 172.21.15.119:0/765314859 >> v2:172.21.15.145:3300/0 conn(0x7efe300d1610 0x7efe300b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe26ffd700 1 -- 172.21.15.119:0/765314859 >> v1:172.21.15.145:6789/0 conn(0x7efe3015cc00 legacy=0x7efe300d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe26ffd700 1 -- 172.21.15.119:0/765314859 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe300cdde0 con 0x7efe300ce8c0 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2ffff700 1 --2- 172.21.15.119:0/765314859 >> v2:172.21.15.145:3300/0 conn(0x7efe300d1610 0x7efe300b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe26ffd700 1 --2- 172.21.15.119:0/765314859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300ce8c0 0x7efe300d10d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efe18009850 tx=0x7efe18009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c40cd28f3b90023 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2effd700 1 -- 172.21.15.119:0/765314859 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18011830 con 0x7efe300ce8c0 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2effd700 1 -- 172.21.15.119:0/765314859 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe18011990 con 0x7efe300ce8c0 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe2effd700 1 -- 172.21.15.119:0/765314859 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe18011d40 con 0x7efe300ce8c0 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 -- 172.21.15.119:0/765314859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300ce8c0 msgr2=0x7efe300d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 --2- 172.21.15.119:0/765314859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300ce8c0 0x7efe300d10d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efe18009850 tx=0x7efe18009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:26.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 -- 172.21.15.119:0/765314859 shutdown_connections 2024-09-13T16:31:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 --2- 172.21.15.119:0/765314859 >> v2:172.21.15.145:3300/0 conn(0x7efe300d1610 0x7efe300b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 --2- 172.21.15.119:0/765314859 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300ce8c0 0x7efe300d10d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.413+0000 7efe34da0700 1 -- 172.21.15.119:0/765314859 >> 172.21.15.119:0/765314859 conn(0x7efe3008c080 msgr2=0x7efe300d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 -- 172.21.15.119:0/765314859 shutdown_connections 2024-09-13T16:31:26.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 -- 172.21.15.119:0/765314859 wait complete. 2024-09-13T16:31:26.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 Processor -- start 2024-09-13T16:31:26.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 -- start start 2024-09-13T16:31:26.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe300ce8c0 0x7efe30151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300d1610 0x7efe3015f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe3015cc00 0x7efe300b7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3015cfe0 con 0x7efe3015cc00 2024-09-13T16:31:26.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3015d2c0 con 0x7efe300ce8c0 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe34da0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe300cbf30 con 0x7efe300d1610 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe2ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe3015cc00 0x7efe300b7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe26ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300d1610 0x7efe3015f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe2ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe3015cc00 0x7efe300b7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56806/0 (socket says 172.21.15.119:56806) 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe2ffff700 1 -- 172.21.15.119:0/2371102806 learned_addr learned my addr 172.21.15.119:0/2371102806 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.417+0000 7efe2f7fe700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe300ce8c0 0x7efe30151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2ffff700 1 -- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300d1610 msgr2=0x7efe3015f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2ffff700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300d1610 0x7efe3015f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2ffff700 1 -- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe300ce8c0 msgr2=0x7efe30151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2ffff700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe300ce8c0 0x7efe30151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2ffff700 1 -- 172.21.15.119:0/2371102806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe28009a70 con 0x7efe3015cc00 2024-09-13T16:31:26.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2f7fe700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe300ce8c0 0x7efe30151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe26ffd700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300d1610 0x7efe3015f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2ffff700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe3015cc00 0x7efe300b7d50 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7efe2800efc0 tx=0x7efe2800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2cff9700 1 -- 172.21.15.119:0/2371102806 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2800ca90 con 0x7efe3015cc00 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2cff9700 1 -- 172.21.15.119:0/2371102806 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2800f040 con 0x7efe3015cc00 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe180094b0 con 0x7efe3015cc00 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe2cff9700 1 -- 172.21.15.119:0/2371102806 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe2800ce10 con 0x7efe3015cc00 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe300cc4f0 con 0x7efe3015cc00 2024-09-13T16:31:26.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.421+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe300cc6e0 con 0x7efe3015cc00 2024-09-13T16:31:26.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.425+0000 7efe2cff9700 1 -- 172.21.15.119:0/2371102806 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efe28014480 con 0x7efe3015cc00 2024-09-13T16:31:26.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.425+0000 7efe2cff9700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe10066730 0x7efe10068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.425+0000 7efe2cff9700 1 -- 172.21.15.119:0/2371102806 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5177+0+0 (secure 0 0 0) 0x7efe280845d0 con 0x7efe3015cc00 2024-09-13T16:31:26.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.425+0000 7efe2f7fe700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe10066730 0x7efe10068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.425+0000 7efe2f7fe700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe10066730 0x7efe10068bf0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efe20005930 tx=0x7efe20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.433+0000 7efe2cff9700 1 -- 172.21.15.119:0/2371102806 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efe28051b50 con 0x7efe3015cc00 2024-09-13T16:31:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.597+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe300b7000 con 0x7efe3015cc00 2024-09-13T16:31:26.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.601+0000 7efe2cff9700 1 -- 172.21.15.119:0/2371102806 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe28009d70 con 0x7efe3015cc00 2024-09-13T16:31:26.603 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:31:26.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe10066730 msgr2=0x7efe10068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe10066730 0x7efe10068bf0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7efe20005930 tx=0x7efe20009040 comp rx=0 tx=0).stop 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe3015cc00 msgr2=0x7efe300b7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe3015cc00 0x7efe300b7d50 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7efe2800efc0 tx=0x7efe2800c3a0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 shutdown_connections 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe10066730 0x7efe10068bf0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe3015cc00 0x7efe300b7d50 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe300d1610 0x7efe3015f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 --2- 172.21.15.119:0/2371102806 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe300ce8c0 0x7efe30151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 >> 172.21.15.119:0/2371102806 conn(0x7efe3008c080 msgr2=0x7efe300b64e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 shutdown_connections 2024-09-13T16:31:26.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.605+0000 7efe34da0700 1 -- 172.21.15.119:0/2371102806 wait complete. 2024-09-13T16:31:26.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:31:26.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 Processor -- start 2024-09-13T16:31:26.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- start start 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feae806a350 0x7feae80cccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7feae80cc190 0x7feae806a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae806cd20 con 0x7feae8160f90 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae80cd1f0 con 0x7feae806a350 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feae806ac70 con 0x7feae80cc190 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feae806a350 0x7feae80cccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7feae806a350 0x7feae80cccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56818/0 (socket says 172.21.15.119:56818) 2024-09-13T16:31:26.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 -- 172.21.15.119:0/594414303 learned_addr learned my addr 172.21.15.119:0/594414303 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:26.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 -- 172.21.15.119:0/594414303 >> v2:172.21.15.145:3300/0 conn(0x7feae80cc190 msgr2=0x7feae806a730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 --2- 172.21.15.119:0/594414303 >> v2:172.21.15.145:3300/0 conn(0x7feae80cc190 0x7feae806a730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 -- 172.21.15.119:0/594414303 >> v1:172.21.15.119:6789/0 conn(0x7feae8160f90 legacy=0x7feae806d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae6ffd700 1 -- 172.21.15.119:0/594414303 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183687001 0 0) 0x7feae806cd20 con 0x7feae8160f90 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 -- 172.21.15.119:0/594414303 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae80b4040 con 0x7feae806a350 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae7fff700 1 --2- 172.21.15.119:0/594414303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae806a350 0x7feae80cccb0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fead0009910 tx=0x7fead0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbc7ede996937fa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae6ffd700 1 -- 172.21.15.119:0/594414303 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead000ba20 con 0x7feae806a350 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae6ffd700 1 -- 172.21.15.119:0/594414303 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead000bb80 con 0x7feae806a350 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feae6ffd700 1 -- 172.21.15.119:0/594414303 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead0009e60 con 0x7feae806a350 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- 172.21.15.119:0/594414303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae806a350 msgr2=0x7feae80cccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 --2- 172.21.15.119:0/594414303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae806a350 0x7feae80cccb0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7fead0009910 tx=0x7fead0009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- 172.21.15.119:0/594414303 shutdown_connections 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 --2- 172.21.15.119:0/594414303 >> v2:172.21.15.145:3300/0 conn(0x7feae80cc190 0x7feae806a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 --2- 172.21.15.119:0/594414303 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae806a350 0x7feae80cccb0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- 172.21.15.119:0/594414303 >> 172.21.15.119:0/594414303 conn(0x7feae8070cc0 msgr2=0x7feae80710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:26.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- 172.21.15.119:0/594414303 shutdown_connections 2024-09-13T16:31:26.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.733+0000 7feaed603700 1 -- 172.21.15.119:0/594414303 wait complete. 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 Processor -- start 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 -- start start 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae806a350 0x7feae814a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae80cc190 0x7feae815f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae8160f90 0x7feae8156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae815d1d0 con 0x7feae80cc190 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae814a8f0 con 0x7feae8160f90 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae815d470 con 0x7feae806a350 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae80cc190 0x7feae815f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae80cc190 0x7feae815f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56820/0 (socket says 172.21.15.119:56820) 2024-09-13T16:31:26.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 -- 172.21.15.119:0/767030910 learned_addr learned my addr 172.21.15.119:0/767030910 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:26.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 -- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae806a350 msgr2=0x7feae814a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae806a350 0x7feae814a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 -- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae8160f90 msgr2=0x7feae8156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae8160f90 0x7feae8156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 -- 172.21.15.119:0/767030910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fead0009570 con 0x7feae80cc190 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae77fe700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae80cc190 0x7feae815f520 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fead800ba50 tx=0x7fead800be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae4ff9700 1 -- 172.21.15.119:0/767030910 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead800cae0 con 0x7feae80cc190 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae4ff9700 1 -- 172.21.15.119:0/767030910 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead800cc40 con 0x7feae80cc190 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae815d750 con 0x7feae80cc190 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feae4ff9700 1 -- 172.21.15.119:0/767030910 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8014440 con 0x7feae80cc190 2024-09-13T16:31:26.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.737+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae815dd10 con 0x7feae80cc190 2024-09-13T16:31:26.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.741+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae815df30 con 0x7feae80cc190 2024-09-13T16:31:26.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.741+0000 7feae4ff9700 1 -- 172.21.15.119:0/767030910 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fead8007370 con 0x7feae80cc190 2024-09-13T16:31:26.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.741+0000 7feae4ff9700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feac8066730 0x7feac8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:26.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.741+0000 7feae4ff9700 1 -- 172.21.15.119:0/767030910 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fead8083d60 con 0x7feae80cc190 2024-09-13T16:31:26.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.741+0000 7feae7fff700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feac8066730 0x7feac8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:26.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.741+0000 7feae7fff700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feac8066730 0x7feac8068bf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fead0006f80 tx=0x7fead0010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:26.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.749+0000 7feae4ff9700 1 -- 172.21.15.119:0/767030910 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fead80512d0 con 0x7feae80cc190 2024-09-13T16:31:26.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.921+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feae80717b0 con 0x7feae80cc190 2024-09-13T16:31:26.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feae4ff9700 1 -- 172.21.15.119:0/767030910 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v91) v1 ==== 84+0+65 (secure 0 0 0) 0x7fead8019090 con 0x7feae80cc190 2024-09-13T16:31:26.928 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:26.928 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:31:26.928 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:31:26.928 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:31:26.928 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:31:26.928 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:26.928 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:31:26.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feac8066730 msgr2=0x7feac8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feac8066730 0x7feac8068bf0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fead0006f80 tx=0x7fead0010000 comp rx=0 tx=0).stop 2024-09-13T16:31:26.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae80cc190 msgr2=0x7feae815f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:26.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae80cc190 0x7feae815f520 secure :-1 s=READY pgs=1242 cs=0 l=1 rev1=1 crypto rx=0x7fead800ba50 tx=0x7fead800be10 comp rx=0 tx=0).stop 2024-09-13T16:31:26.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 shutdown_connections 2024-09-13T16:31:26.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feac8066730 0x7feac8068bf0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7feae8160f90 0x7feae8156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7feae80cc190 0x7feae815f520 unknown :-1 s=CLOSED pgs=1242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 --2- 172.21.15.119:0/767030910 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7feae806a350 0x7feae814a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:26.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 >> 172.21.15.119:0/767030910 conn(0x7feae8070cc0 msgr2=0x7feae81613d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:26.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 shutdown_connections 2024-09-13T16:31:26.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:26.925+0000 7feaed603700 1 -- 172.21.15.119:0/767030910 wait complete. 2024-09-13T16:31:26.945 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:31:26.946 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:31:27.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.065+0000 7fe9b9550700 1 Processor -- start 2024-09-13T16:31:27.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.065+0000 7fe9b9550700 1 -- start start 2024-09-13T16:31:27.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b9550700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b40b4b20 0x7fe9b406e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b9550700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b40afbb0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b9550700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b409a380 con 0x7fe9b415a880 2024-09-13T16:31:27.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b9550700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b40af230 con 0x7fe9b40b4b20 2024-09-13T16:31:27.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b37fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe9b40b38e0 0x7fe9b406e200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48444/0 (socket says 172.21.15.119:48444) 2024-09-13T16:31:27.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b37fe700 1 -- 172.21.15.119:0/2110548842 learned_addr learned my addr 172.21.15.119:0/2110548842 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:27.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b8d4f700 1 --2- 172.21.15.119:0/2110548842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b40b4b20 0x7fe9b406e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b2ffd700 1 -- 172.21.15.119:0/2110548842 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048422355 0 0) 0x7fe9b40afbb0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b2ffd700 1 -- 172.21.15.119:0/2110548842 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe99c0035d0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b2ffd700 1 -- 172.21.15.119:0/2110548842 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360054382 0 0) 0x7fe9b409a380 con 0x7fe9b415a880 2024-09-13T16:31:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b2ffd700 1 -- 172.21.15.119:0/2110548842 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9b40afbb0 con 0x7fe9b415a880 2024-09-13T16:31:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b8d4f700 1 -- 172.21.15.119:0/2110548842 >> v1:172.21.15.145:6789/0 conn(0x7fe9b415a880 legacy=0x7fe9b40b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b8d4f700 1 -- 172.21.15.119:0/2110548842 >> v1:172.21.15.119:6789/0 conn(0x7fe9b40b38e0 legacy=0x7fe9b406e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b8d4f700 1 -- 172.21.15.119:0/2110548842 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b40afe00 con 0x7fe9b40b4b20 2024-09-13T16:31:27.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b8d4f700 1 --2- 172.21.15.119:0/2110548842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b40b4b20 0x7fe9b406e910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009dc0 tx=0x7fe9ac00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edcb082ea1e3eefe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:27.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b2ffd700 1 -- 172.21.15.119:0/2110548842 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00e070 con 0x7fe9b40b4b20 2024-09-13T16:31:27.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b2ffd700 1 -- 172.21.15.119:0/2110548842 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9ac00bb20 con 0x7fe9b40b4b20 2024-09-13T16:31:27.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.069+0000 7fe9b2ffd700 1 -- 172.21.15.119:0/2110548842 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac00be90 con 0x7fe9b40b4b20 2024-09-13T16:31:27.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- 172.21.15.119:0/2110548842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b40b4b20 msgr2=0x7fe9b406e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 --2- 172.21.15.119:0/2110548842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b40b4b20 0x7fe9b406e910 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009dc0 tx=0x7fe9ac00b330 comp rx=0 tx=0).stop 2024-09-13T16:31:27.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- 172.21.15.119:0/2110548842 shutdown_connections 2024-09-13T16:31:27.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 --2- 172.21.15.119:0/2110548842 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b40b4b20 0x7fe9b406e910 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- 172.21.15.119:0/2110548842 >> 172.21.15.119:0/2110548842 conn(0x7fe9b40723c0 msgr2=0x7fe9b4099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:27.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- 172.21.15.119:0/2110548842 shutdown_connections 2024-09-13T16:31:27.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- 172.21.15.119:0/2110548842 wait complete. 2024-09-13T16:31:27.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 Processor -- start 2024-09-13T16:31:27.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- start start 2024-09-13T16:31:27.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b40b38e0 0x7fe9b41f28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9b40b4b20 0x7fe9b409e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b415a880 0x7fe9b4153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b4157e90 con 0x7fe9b40b38e0 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b41580e0 con 0x7fe9b40b4b20 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b9550700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b41583c0 con 0x7fe9b415a880 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b40b38e0 0x7fe9b41f28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b40b38e0 0x7fe9b41f28f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56834/0 (socket says 172.21.15.119:56834) 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b3fff700 1 -- 172.21.15.119:0/931330263 learned_addr learned my addr 172.21.15.119:0/931330263 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:27.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.073+0000 7fe9b37fe700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9b40b4b20 0x7fe9b409e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b8d4f700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b415a880 0x7fe9b4153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b3fff700 1 -- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b415a880 msgr2=0x7fe9b4153820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b3fff700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b415a880 0x7fe9b4153820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b3fff700 1 -- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9b40b4b20 msgr2=0x7fe9b409e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b3fff700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9b40b4b20 0x7fe9b409e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b3fff700 1 -- 172.21.15.119:0/931330263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9ac009a70 con 0x7fe9b40b38e0 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b8d4f700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b415a880 0x7fe9b4153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b3fff700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b40b38e0 0x7fe9b41f28f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400c500 tx=0x7fe9a400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b0ff9700 1 -- 172.21.15.119:0/931330263 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40052d0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b0ff9700 1 -- 172.21.15.119:0/931330263 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a4005450 con 0x7fe9b40b38e0 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b41586a0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b0ff9700 1 -- 172.21.15.119:0/931330263 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40038e0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b409e5e0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.077+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b40ae920 con 0x7fe9b40b38e0 2024-09-13T16:31:27.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.081+0000 7fe9b0ff9700 1 -- 172.21.15.119:0/931330263 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe9a4003a40 con 0x7fe9b40b38e0 2024-09-13T16:31:27.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.081+0000 7fe9b0ff9700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9a0066730 0x7fe9a0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.081+0000 7fe9b0ff9700 1 -- 172.21.15.119:0/931330263 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fe9a40836a0 con 0x7fe9b40b38e0 2024-09-13T16:31:27.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.081+0000 7fe9b37fe700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9a0066730 0x7fe9a0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.081+0000 7fe9b37fe700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9a0066730 0x7fe9a0068bf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4151020 tx=0x7fe9a8007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:27.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.085+0000 7fe9b0ff9700 1 -- 172.21.15.119:0/931330263 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe9a4050b80 con 0x7fe9b40b38e0 2024-09-13T16:31:27.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.253+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe9b40ae920 con 0x7fe9b40b38e0 2024-09-13T16:31:27.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.573+0000 7fe9b0ff9700 1 -- 172.21.15.119:0/931330263 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v92) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe9b40ae920 con 0x7fe9b40b38e0 2024-09-13T16:31:27.578 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:31:27.580 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:31:27.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9a0066730 msgr2=0x7fe9a0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9a0066730 0x7fe9a0068bf0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe9b4151020 tx=0x7fe9a8007de0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b40b38e0 msgr2=0x7fe9b41f28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b40b38e0 0x7fe9b41f28f0 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400c500 tx=0x7fe9a400c810 comp rx=0 tx=0).stop 2024-09-13T16:31:27.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 shutdown_connections 2024-09-13T16:31:27.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9a0066730 0x7fe9a0068bf0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9b415a880 0x7fe9b4153820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9b40b4b20 0x7fe9b409e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 --2- 172.21.15.119:0/931330263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9b40b38e0 0x7fe9b41f28f0 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 >> 172.21.15.119:0/931330263 conn(0x7fe9b40723c0 msgr2=0x7fe9b40b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:27.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 shutdown_connections 2024-09-13T16:31:27.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.581+0000 7fe9b9550700 1 -- 172.21.15.119:0/931330263 wait complete. 2024-09-13T16:31:27.587 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:31:27.599 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:31:27.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2b531a700 1 Processor -- start 2024-09-13T16:31:27.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2b531a700 1 -- start start 2024-09-13T16:31:27.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2b531a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff2b015cfb0 0x7ff2b006ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2b531a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006b2d0 0x7ff2b006b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2b531a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b00ce880 con 0x7ff2b006bc50 2024-09-13T16:31:27.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2b531a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b00cee40 con 0x7ff2b015cfb0 2024-09-13T16:31:27.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2b531a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b006c100 con 0x7ff2b006b2d0 2024-09-13T16:31:27.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2af7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff2b006bc50 0x7ff2b00cf5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48460/0 (socket says 172.21.15.119:48460) 2024-09-13T16:31:27.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2af7fe700 1 -- 172.21.15.119:0/3325357837 learned_addr learned my addr 172.21.15.119:0/3325357837 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:27.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2ae7fc700 1 --2- 172.21.15.119:0/3325357837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006b2d0 0x7ff2b006b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2aeffd700 1 --2- 172.21.15.119:0/3325357837 >> v2:172.21.15.145:3300/0 conn(0x7ff2b015cfb0 0x7ff2b006ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2adffb700 1 -- 172.21.15.119:0/3325357837 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603341019 0 0) 0x7ff2b00ce880 con 0x7ff2b006bc50 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.801+0000 7ff2adffb700 1 -- 172.21.15.119:0/3325357837 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2980035d0 con 0x7ff2b006bc50 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2ae7fc700 1 -- 172.21.15.119:0/3325357837 >> v2:172.21.15.145:3300/0 conn(0x7ff2b015cfb0 msgr2=0x7ff2b006ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2ae7fc700 1 --2- 172.21.15.119:0/3325357837 >> v2:172.21.15.145:3300/0 conn(0x7ff2b015cfb0 0x7ff2b006ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2ae7fc700 1 -- 172.21.15.119:0/3325357837 >> v1:172.21.15.119:6789/0 conn(0x7ff2b006bc50 legacy=0x7ff2b00cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2ae7fc700 1 -- 172.21.15.119:0/3325357837 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b00ceb60 con 0x7ff2b006b2d0 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2ae7fc700 1 --2- 172.21.15.119:0/3325357837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006b2d0 0x7ff2b006b710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009850 tx=0x7ff2a0009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8470fe739ae0fb48 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:27.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2adffb700 1 -- 172.21.15.119:0/3325357837 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0011830 con 0x7ff2b006b2d0 2024-09-13T16:31:27.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2adffb700 1 -- 172.21.15.119:0/3325357837 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a0011990 con 0x7ff2b006b2d0 2024-09-13T16:31:27.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2adffb700 1 -- 172.21.15.119:0/3325357837 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a0011d40 con 0x7ff2b006b2d0 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 -- 172.21.15.119:0/3325357837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006b2d0 msgr2=0x7ff2b006b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 --2- 172.21.15.119:0/3325357837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006b2d0 0x7ff2b006b710 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7ff2a0009850 tx=0x7ff2a0009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 -- 172.21.15.119:0/3325357837 shutdown_connections 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 --2- 172.21.15.119:0/3325357837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006b2d0 0x7ff2b006b710 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 --2- 172.21.15.119:0/3325357837 >> v2:172.21.15.145:3300/0 conn(0x7ff2b015cfb0 0x7ff2b006ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 -- 172.21.15.119:0/3325357837 >> 172.21.15.119:0/3325357837 conn(0x7ff2b00d2da0 msgr2=0x7ff2b00d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 -- 172.21.15.119:0/3325357837 shutdown_connections 2024-09-13T16:31:27.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.805+0000 7ff2b531a700 1 -- 172.21.15.119:0/3325357837 wait complete. 2024-09-13T16:31:27.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 Processor -- start 2024-09-13T16:31:27.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 -- start start 2024-09-13T16:31:27.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b006b2d0 0x7ff2b0152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006bc50 0x7ff2b015df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2b015cfb0 0x7ff2b00c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b00d1220 con 0x7ff2b015cfb0 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b015e950 con 0x7ff2b006b2d0 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.809+0000 7ff2b531a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b0152750 con 0x7ff2b006bc50 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2b015cfb0 0x7ff2b00c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2aeffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b006b2d0 0x7ff2b0152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2ae7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006bc50 0x7ff2b015df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2b015cfb0 0x7ff2b00c6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56840/0 (socket says 172.21.15.119:56840) 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 -- 172.21.15.119:0/610122789 learned_addr learned my addr 172.21.15.119:0/610122789 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 -- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006bc50 msgr2=0x7ff2b015df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006bc50 0x7ff2b015df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 -- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b006b2d0 msgr2=0x7ff2b0152210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b006b2d0 0x7ff2b0152210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:27.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 -- 172.21.15.119:0/610122789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a00094b0 con 0x7ff2b015cfb0 2024-09-13T16:31:27.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2af7fe700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2b015cfb0 0x7ff2b00c6ec0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800cb40 tx=0x7ff2a800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:27.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2977fe700 1 -- 172.21.15.119:0/610122789 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a8007500 con 0x7ff2b015cfb0 2024-09-13T16:31:27.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2977fe700 1 -- 172.21.15.119:0/610122789 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a8007660 con 0x7ff2b015cfb0 2024-09-13T16:31:27.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b00b5d00 con 0x7ff2b015cfb0 2024-09-13T16:31:27.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2977fe700 1 -- 172.21.15.119:0/610122789 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2a80079f0 con 0x7ff2b015cfb0 2024-09-13T16:31:27.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b00b62c0 con 0x7ff2b015cfb0 2024-09-13T16:31:27.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.813+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b006b710 con 0x7ff2b015cfb0 2024-09-13T16:31:27.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.817+0000 7ff2977fe700 1 -- 172.21.15.119:0/610122789 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff2a801b3b0 con 0x7ff2b015cfb0 2024-09-13T16:31:27.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.817+0000 7ff2977fe700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff29c066730 0x7ff29c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:27.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.817+0000 7ff2977fe700 1 -- 172.21.15.119:0/610122789 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 5547+0+0 (secure 0 0 0) 0x7ff2a8083ac0 con 0x7ff2b015cfb0 2024-09-13T16:31:27.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.817+0000 7ff2aeffd700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff29c066730 0x7ff29c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:27.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.817+0000 7ff2aeffd700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff29c066730 0x7ff29c068bf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009b50 tx=0x7ff2a4009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:27.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:27.821+0000 7ff2977fe700 1 -- 172.21.15.119:0/610122789 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff2a8050ee0 con 0x7ff2b015cfb0 2024-09-13T16:31:28.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.001+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff2b00d2060 con 0x7ff2b015cfb0 2024-09-13T16:31:28.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.585+0000 7ff2977fe700 1 -- 172.21.15.119:0/610122789 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v93) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff2a80170d0 con 0x7ff2b015cfb0 2024-09-13T16:31:28.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff29c066730 msgr2=0x7ff29c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff29c066730 0x7ff29c068bf0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009b50 tx=0x7ff2a4009240 comp rx=0 tx=0).stop 2024-09-13T16:31:28.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2b015cfb0 msgr2=0x7ff2b00c6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2b015cfb0 0x7ff2b00c6ec0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7ff2a800cb40 tx=0x7ff2a800cb70 comp rx=0 tx=0).stop 2024-09-13T16:31:28.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 shutdown_connections 2024-09-13T16:31:28.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff29c066730 0x7ff29c068bf0 secure :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7ff2a4009b50 tx=0x7ff2a4009240 comp rx=0 tx=0).stop 2024-09-13T16:31:28.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff2b015cfb0 0x7ff2b00c6ec0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff2b006bc50 0x7ff2b015df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 --2- 172.21.15.119:0/610122789 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff2b006b2d0 0x7ff2b0152210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.597+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 >> 172.21.15.119:0/610122789 conn(0x7ff2b00d2da0 msgr2=0x7ff2b00b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:28.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.605+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 shutdown_connections 2024-09-13T16:31:28.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.605+0000 7ff2b531a700 1 -- 172.21.15.119:0/610122789 wait complete. 2024-09-13T16:31:28.611 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:31:28.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:31:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d63fff700 1 Processor -- start 2024-09-13T16:31:28.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d63fff700 1 -- start start 2024-09-13T16:31:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d63fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5d640d10d0 0x7f5d640d1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d63fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d640b2240 con 0x7f5d64160210 2024-09-13T16:31:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d63fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d640b2520 con 0x7f5d640b29b0 2024-09-13T16:31:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d63fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d640d1aa0 con 0x7f5d640d10d0 2024-09-13T16:31:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d62ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5d64160210 0x7f5d640d3220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48468/0 (socket says 172.21.15.119:48468) 2024-09-13T16:31:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d62ffd700 1 -- 172.21.15.119:0/3602237148 learned_addr learned my addr 172.21.15.119:0/3602237148 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:28.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.733+0000 7f5d627fc700 1 --2- 172.21.15.119:0/3602237148 >> v2:172.21.15.145:3300/0 conn(0x7f5d640d10d0 0x7f5d640d1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:28.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d61ffb700 1 -- 172.21.15.119:0/3602237148 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1988984207 0 0) 0x7f5d640b2240 con 0x7f5d64160210 2024-09-13T16:31:28.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d61ffb700 1 -- 172.21.15.119:0/3602237148 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4c0035d0 con 0x7f5d64160210 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d627fc700 1 -- 172.21.15.119:0/3602237148 >> v1:172.21.15.145:6789/0 conn(0x7f5d640b29b0 legacy=0x7f5d640b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d627fc700 1 -- 172.21.15.119:0/3602237148 >> v1:172.21.15.119:6789/0 conn(0x7f5d64160210 legacy=0x7f5d640d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d627fc700 1 -- 172.21.15.119:0/3602237148 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d640d0780 con 0x7f5d640d10d0 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d627fc700 1 --2- 172.21.15.119:0/3602237148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640d10d0 0x7f5d640d1560 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009850 tx=0x7f5d54009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd6549bcb4062929 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d61ffb700 1 -- 172.21.15.119:0/3602237148 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d54011830 con 0x7f5d640d10d0 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d61ffb700 1 -- 172.21.15.119:0/3602237148 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d54011990 con 0x7f5d640d10d0 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d61ffb700 1 -- 172.21.15.119:0/3602237148 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d54011d00 con 0x7f5d640d10d0 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- 172.21.15.119:0/3602237148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640d10d0 msgr2=0x7f5d640d1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 --2- 172.21.15.119:0/3602237148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640d10d0 0x7f5d640d1560 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5d54009850 tx=0x7f5d54009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- 172.21.15.119:0/3602237148 shutdown_connections 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 --2- 172.21.15.119:0/3602237148 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640d10d0 0x7f5d640d1560 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- 172.21.15.119:0/3602237148 >> 172.21.15.119:0/3602237148 conn(0x7f5d6408c080 msgr2=0x7f5d640d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:28.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- 172.21.15.119:0/3602237148 shutdown_connections 2024-09-13T16:31:28.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- 172.21.15.119:0/3602237148 wait complete. 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 Processor -- start 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- start start 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640b29b0 0x7f5d64151ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d640d10d0 0x7f5d641529f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d64160210 0x7f5d640c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d640c72f0 con 0x7f5d64160210 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d640c66e0 con 0x7f5d640b29b0 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.737+0000 7f5d63fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d640c69c0 con 0x7f5d640d10d0 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d64160210 0x7f5d640c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:28.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d64160210 0x7f5d640c7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56856/0 (socket says 172.21.15.119:56856) 2024-09-13T16:31:28.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 -- 172.21.15.119:0/1702127795 learned_addr learned my addr 172.21.15.119:0/1702127795 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:28.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d62ffd700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640b29b0 0x7f5d64151ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:28.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d627fc700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d640d10d0 0x7f5d641529f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:28.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 -- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d640d10d0 msgr2=0x7f5d641529f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d640d10d0 0x7f5d641529f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 -- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640b29b0 msgr2=0x7f5d64151ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640b29b0 0x7f5d64151ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 -- 172.21.15.119:0/1702127795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540094b0 con 0x7f5d64160210 2024-09-13T16:31:28.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d637fe700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d64160210 0x7f5d640c7780 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00cb50 tx=0x7f5d5c00ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:28.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d4b7fe700 1 -- 172.21.15.119:0/1702127795 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c003c20 con 0x7f5d64160210 2024-09-13T16:31:28.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d4b7fe700 1 -- 172.21.15.119:0/1702127795 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d5c003d80 con 0x7f5d64160210 2024-09-13T16:31:28.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d640c6ca0 con 0x7f5d64160210 2024-09-13T16:31:28.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d4b7fe700 1 -- 172.21.15.119:0/1702127795 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d5c00aa20 con 0x7f5d64160210 2024-09-13T16:31:28.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d641523e0 con 0x7f5d64160210 2024-09-13T16:31:28.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.741+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d64073590 con 0x7f5d64160210 2024-09-13T16:31:28.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.745+0000 7f5d4b7fe700 1 -- 172.21.15.119:0/1702127795 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5d5c00ab80 con 0x7f5d64160210 2024-09-13T16:31:28.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.745+0000 7f5d4b7fe700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d50066730 0x7f5d50068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:28.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.745+0000 7f5d4b7fe700 1 -- 172.21.15.119:0/1702127795 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f5d5c084210 con 0x7f5d64160210 2024-09-13T16:31:28.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.745+0000 7f5d62ffd700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d50066730 0x7f5d50068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:28.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.745+0000 7f5d62ffd700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d50066730 0x7f5d50068bf0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5d58007fd0 tx=0x7f5d58007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:28.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.749+0000 7f5d4b7fe700 1 -- 172.21.15.119:0/1702127795 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5d5c0514a0 con 0x7f5d64160210 2024-09-13T16:31:28.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.953+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f5d640cfef0 con 0x7f5d64160210 2024-09-13T16:31:28.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.969+0000 7f5d4b7fe700 1 -- 172.21.15.119:0/1702127795 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 13 and data pool 14 v53) v1 ==== 174+0+0 (secure 0 0 0) 0x7f5d5c054ac0 con 0x7f5d64160210 2024-09-13T16:31:28.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d50066730 msgr2=0x7f5d50068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d50066730 0x7f5d50068bf0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f5d58007fd0 tx=0x7f5d58007de0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d64160210 msgr2=0x7f5d640c7780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:28.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d64160210 0x7f5d640c7780 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 crypto rx=0x7f5d5c00cb50 tx=0x7f5d5c00ce60 comp rx=0 tx=0).stop 2024-09-13T16:31:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 shutdown_connections 2024-09-13T16:31:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d50066730 0x7f5d50068bf0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5d64160210 0x7f5d640c7780 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.989+0000 7f5d63fff700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5d640d10d0 0x7f5d641529f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.993+0000 7f5d63fff700 1 --2- 172.21.15.119:0/1702127795 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5d640b29b0 0x7f5d64151ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:28.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.993+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 >> 172.21.15.119:0/1702127795 conn(0x7f5d6408c080 msgr2=0x7f5d640cd620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.993+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 shutdown_connections 2024-09-13T16:31:28.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:28.993+0000 7f5d63fff700 1 -- 172.21.15.119:0/1702127795 wait complete. 2024-09-13T16:31:28.999 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 13 and data pool 14 2024-09-13T16:31:29.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:29.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165b59e700 1 Processor -- start 2024-09-13T16:31:29.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165b59e700 1 -- start start 2024-09-13T16:31:29.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165b59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f165c1570d0 0x7f165c06b930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165b59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f165c06e270 0x7f165c06e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165b59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f165c06ebf0 con 0x7f165c0d0f70 2024-09-13T16:31:29.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165b59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165c0ce730 con 0x7f165c06e270 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165b59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f165c0cf840 con 0x7f165c1570d0 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f1659d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f165c06e270 0x7f165c06e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165ad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f165c0d0f70 0x7f165c0d1400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48472/0 (socket says 172.21.15.119:48472) 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165ad9d700 1 -- 172.21.15.119:0/2373829091 learned_addr learned my addr 172.21.15.119:0/2373829091 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165a59c700 1 --2- 172.21.15.119:0/2373829091 >> v2:172.21.15.145:3300/0 conn(0x7f165c1570d0 0x7f165c06b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f1659d9b700 1 -- 172.21.15.119:0/2373829091 >> v2:172.21.15.145:3300/0 conn(0x7f165c1570d0 msgr2=0x7f165c06b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f1659d9b700 1 --2- 172.21.15.119:0/2373829091 >> v2:172.21.15.145:3300/0 conn(0x7f165c1570d0 0x7f165c06b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f1659d9b700 1 -- 172.21.15.119:0/2373829091 >> v1:172.21.15.119:6789/0 conn(0x7f165c0d0f70 legacy=0x7f165c0d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165959a700 1 -- 172.21.15.119:0/2373829091 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588406630 0 0) 0x7f165c06ebf0 con 0x7f165c0d0f70 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f1659d9b700 1 -- 172.21.15.119:0/2373829091 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f165c0cfb20 con 0x7f165c06e270 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165a59c700 1 --2- 172.21.15.119:0/2373829091 >> v2:172.21.15.145:3300/0 conn(0x7f165c1570d0 0x7f165c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:29.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f1659d9b700 1 --2- 172.21.15.119:0/2373829091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c06e6b0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f165c0ce570 tx=0x7f164c009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c90b186ecb23ea server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165959a700 1 -- 172.21.15.119:0/2373829091 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c011830 con 0x7f165c06e270 2024-09-13T16:31:29.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165959a700 1 -- 172.21.15.119:0/2373829091 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f164c011990 con 0x7f165c06e270 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.109+0000 7f165959a700 1 -- 172.21.15.119:0/2373829091 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164c011d40 con 0x7f165c06e270 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- 172.21.15.119:0/2373829091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 msgr2=0x7f165c06e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 --2- 172.21.15.119:0/2373829091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c06e6b0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7f165c0ce570 tx=0x7f164c009b30 comp rx=0 tx=0).stop 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- 172.21.15.119:0/2373829091 shutdown_connections 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 --2- 172.21.15.119:0/2373829091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c06e6b0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 --2- 172.21.15.119:0/2373829091 >> v2:172.21.15.145:3300/0 conn(0x7f165c1570d0 0x7f165c06b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- 172.21.15.119:0/2373829091 >> 172.21.15.119:0/2373829091 conn(0x7f165c099a50 msgr2=0x7f165c099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- 172.21.15.119:0/2373829091 shutdown_connections 2024-09-13T16:31:29.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- 172.21.15.119:0/2373829091 wait complete. 2024-09-13T16:31:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 Processor -- start 2024-09-13T16:31:29.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- start start 2024-09-13T16:31:29.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c1f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f165c0d0f70 0x7f165c154ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f165c1570d0 0x7f165c142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c155900 con 0x7f165c06e270 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165c155be0 con 0x7f165c1570d0 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.113+0000 7f165b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165c155ec0 con 0x7f165c0d0f70 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c1f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c1f3020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56874/0 (socket says 172.21.15.119:56874) 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 -- 172.21.15.119:0/1304308390 learned_addr learned my addr 172.21.15.119:0/1304308390 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:29.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165ad9d700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f165c1570d0 0x7f165c142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f1659d9b700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f165c0d0f70 0x7f165c154ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 -- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f165c0d0f70 msgr2=0x7f165c154ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f165c0d0f70 0x7f165c154ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 -- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f165c1570d0 msgr2=0x7f165c142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f165c1570d0 0x7f165c142170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 -- 172.21.15.119:0/1304308390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f164c0094b0 con 0x7f165c06e270 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165a59c700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c1f3020 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f164400e810 tx=0x7f164400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f1652ffd700 1 -- 172.21.15.119:0/1304308390 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f164400c070 con 0x7f165c06e270 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f1652ffd700 1 -- 172.21.15.119:0/1304308390 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1644013610 con 0x7f165c06e270 2024-09-13T16:31:29.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f165c1561a0 con 0x7f165c06e270 2024-09-13T16:31:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f1652ffd700 1 -- 172.21.15.119:0/1304308390 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16440139a0 con 0x7f165c06e270 2024-09-13T16:31:29.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f165c161150 con 0x7f165c06e270 2024-09-13T16:31:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f165c0ae0c0 con 0x7f165c06e270 2024-09-13T16:31:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.117+0000 7f1652ffd700 1 -- 172.21.15.119:0/1304308390 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1644013b00 con 0x7f165c06e270 2024-09-13T16:31:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.121+0000 7f1652ffd700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f163c066730 0x7f163c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.121+0000 7f1652ffd700 1 -- 172.21.15.119:0/1304308390 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1644083f30 con 0x7f165c06e270 2024-09-13T16:31:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.121+0000 7f1659d9b700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f163c066730 0x7f163c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.121+0000 7f1659d9b700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f163c066730 0x7f163c068bf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f164c000c00 tx=0x7f164c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.125+0000 7f1652ffd700 1 -- 172.21.15.119:0/1304308390 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1644051150 con 0x7f165c06e270 2024-09-13T16:31:29.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.297+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f165c0adde0 con 0x7f165c06e270 2024-09-13T16:31:29.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.301+0000 7f1652ffd700 1 -- 172.21.15.119:0/1304308390 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f1644013db0 con 0x7f165c06e270 2024-09-13T16:31:29.305 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:29.305 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":94,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:31:28.959279+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f163c066730 msgr2=0x7f163c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f163c066730 0x7f163c068bf0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f164c000c00 tx=0x7f164c008030 comp rx=0 tx=0).stop 2024-09-13T16:31:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 msgr2=0x7f165c1f3020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c1f3020 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7f164400e810 tx=0x7f164400eb20 comp rx=0 tx=0).stop 2024-09-13T16:31:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 shutdown_connections 2024-09-13T16:31:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f163c066730 0x7f163c068bf0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f165c1570d0 0x7f165c142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f165c0d0f70 0x7f165c154ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 --2- 172.21.15.119:0/1304308390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f165c06e270 0x7f165c1f3020 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 >> 172.21.15.119:0/1304308390 conn(0x7f165c099a50 msgr2=0x7f165c06c1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 shutdown_connections 2024-09-13T16:31:29.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.305+0000 7f165b59e700 1 -- 172.21.15.119:0/1304308390 wait complete. 2024-09-13T16:31:29.326 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:29.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5ffff700 1 Processor -- start 2024-09-13T16:31:29.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5ffff700 1 -- start start 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3f60143f50 0x7f3f6006c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f600b3410 0x7f3f60164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5ffff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f600b1c40 con 0x7f3f60144330 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5ffff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f6006cdf0 con 0x7f3f60143f50 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6006d0d0 con 0x7f3f600b3410 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.433+0000 7f3f5effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3f60143f50 0x7f3f6006c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3f60143f50 0x7f3f6006c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56876/0 (socket says 172.21.15.119:56876) 2024-09-13T16:31:29.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f600b3410 0x7f3f60164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5effd700 1 -- 172.21.15.119:0/4227916120 learned_addr learned my addr 172.21.15.119:0/4227916120 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5effd700 1 -- 172.21.15.119:0/4227916120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f600b3410 msgr2=0x7f3f60164330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5effd700 1 --2- 172.21.15.119:0/4227916120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f600b3410 0x7f3f60164330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5effd700 1 -- 172.21.15.119:0/4227916120 >> v1:172.21.15.145:6789/0 conn(0x7f3f60144330 legacy=0x7f3f600cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5dffb700 1 -- 172.21.15.119:0/4227916120 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3885935125 0 0) 0x7f3f600b1c40 con 0x7f3f60144330 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5effd700 1 -- 172.21.15.119:0/4227916120 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6006d3b0 con 0x7f3f60143f50 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5effd700 1 --2- 172.21.15.119:0/4227916120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f60143f50 0x7f3f6006c5a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009910 tx=0x7f3f54009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20461bd934aeca8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5dffb700 1 -- 172.21.15.119:0/4227916120 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54011830 con 0x7f3f60143f50 2024-09-13T16:31:29.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5dffb700 1 -- 172.21.15.119:0/4227916120 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f54011990 con 0x7f3f60143f50 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5dffb700 1 -- 172.21.15.119:0/4227916120 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f54011d40 con 0x7f3f60143f50 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 -- 172.21.15.119:0/4227916120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f60143f50 msgr2=0x7f3f6006c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/4227916120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f60143f50 0x7f3f6006c5a0 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009910 tx=0x7f3f54009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 -- 172.21.15.119:0/4227916120 shutdown_connections 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/4227916120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f600b3410 0x7f3f60164330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/4227916120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f60143f50 0x7f3f6006c5a0 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 -- 172.21.15.119:0/4227916120 >> 172.21.15.119:0/4227916120 conn(0x7f3f6006a350 msgr2=0x7f3f6006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 -- 172.21.15.119:0/4227916120 shutdown_connections 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 -- 172.21.15.119:0/4227916120 wait complete. 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 Processor -- start 2024-09-13T16:31:29.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.437+0000 7f3f5ffff700 1 -- start start 2024-09-13T16:31:29.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f600b3410 0x7f3f600c4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f60143f50 0x7f3f601f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f60144330 0x7f3f600ad030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f600c5280 con 0x7f3f600b3410 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f600c3ca0 con 0x7f3f60143f50 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f600c3f80 con 0x7f3f60144330 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f600b3410 0x7f3f600c4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f600b3410 0x7f3f600c4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56892/0 (socket says 172.21.15.119:56892) 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 -- 172.21.15.119:0/2145144830 learned_addr learned my addr 172.21.15.119:0/2145144830 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5e7fc700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f60143f50 0x7f3f601f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5f7fe700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f60144330 0x7f3f600ad030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 -- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f60144330 msgr2=0x7f3f600ad030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f60144330 0x7f3f600ad030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 -- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f60143f50 msgr2=0x7f3f601f9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f60143f50 0x7f3f601f9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 -- 172.21.15.119:0/2145144830 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f54009570 con 0x7f3f600b3410 2024-09-13T16:31:29.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5effd700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f600b3410 0x7f3f600c4d40 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009f70 tx=0x7f3f54004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f477fe700 1 -- 172.21.15.119:0/2145144830 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400ba70 con 0x7f3f600b3410 2024-09-13T16:31:29.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f477fe700 1 -- 172.21.15.119:0/2145144830 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5401f070 con 0x7f3f600b3410 2024-09-13T16:31:29.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f600c4260 con 0x7f3f600b3410 2024-09-13T16:31:29.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f477fe700 1 -- 172.21.15.119:0/2145144830 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5400be10 con 0x7f3f600b3410 2024-09-13T16:31:29.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f600c4820 con 0x7f3f600b3410 2024-09-13T16:31:29.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.441+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6006c5a0 con 0x7f3f600b3410 2024-09-13T16:31:29.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.445+0000 7f3f477fe700 1 -- 172.21.15.119:0/2145144830 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3f5401a430 con 0x7f3f600b3410 2024-09-13T16:31:29.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.445+0000 7f3f477fe700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f4c066730 0x7f3f4c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.445+0000 7f3f477fe700 1 -- 172.21.15.119:0/2145144830 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3f54021070 con 0x7f3f600b3410 2024-09-13T16:31:29.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.445+0000 7f3f5e7fc700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f4c066730 0x7f3f4c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.445+0000 7f3f5e7fc700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f4c066730 0x7f3f4c068bf0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3f60143060 tx=0x7f3f50009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.453+0000 7f3f477fe700 1 -- 172.21.15.119:0/2145144830 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3f54052ea0 con 0x7f3f600b3410 2024-09-13T16:31:29.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.621+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f6006fca0 con 0x7f3f600b3410 2024-09-13T16:31:29.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.625+0000 7f3f477fe700 1 -- 172.21.15.119:0/2145144830 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v94) v1 ==== 74+0+16790 (secure 0 0 0) 0x7f3f54015020 con 0x7f3f600b3410 2024-09-13T16:31:29.629 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:29.629 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":94,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:31:28.959279+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f4c066730 msgr2=0x7f3f4c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f4c066730 0x7f3f4c068bf0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f3f60143060 tx=0x7f3f50009040 comp rx=0 tx=0).stop 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f600b3410 msgr2=0x7f3f600c4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f600b3410 0x7f3f600c4d40 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f3f54009f70 tx=0x7f3f54004770 comp rx=0 tx=0).stop 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 shutdown_connections 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f4c066730 0x7f3f4c068bf0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f60144330 0x7f3f600ad030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f60143f50 0x7f3f601f9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 --2- 172.21.15.119:0/2145144830 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f600b3410 0x7f3f600c4d40 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 >> 172.21.15.119:0/2145144830 conn(0x7f3f6006a350 msgr2=0x7f3f600a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:29.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 shutdown_connections 2024-09-13T16:31:29.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.629+0000 7f3f5ffff700 1 -- 172.21.15.119:0/2145144830 wait complete. 2024-09-13T16:31:29.644 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:31:29.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe606773700 1 Processor -- start 2024-09-13T16:31:29.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe606773700 1 -- start start 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe606773700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 0x7fe6000d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe606773700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000b2240 con 0x7fe6000d3760 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe606773700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6000b2520 con 0x7fe6000d17e0 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe606773700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6001605f0 con 0x7fe600160210 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe604f70700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe6000d3760 0x7fe6000d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:48484/0 (socket says 172.21.15.119:48484) 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.757+0000 7fe604f70700 1 -- 172.21.15.119:0/2016046923 learned_addr learned my addr 172.21.15.119:0/2016046923 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe605771700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 0x7fe6000d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361598684 0 0) 0x7fe6000b2240 con 0x7fe6000d3760 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec0035d0 con 0x7fe6000d3760 2024-09-13T16:31:29.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163574786 0 0) 0x7fe6000b2520 con 0x7fe6000d17e0 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6000b2240 con 0x7fe6000d17e0 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3978157440 0 0) 0x7fe5ec0035d0 con 0x7fe6000d3760 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 msgr2=0x7fe6000d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 0x7fe6000d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 >> v1:172.21.15.145:6789/0 conn(0x7fe6000d17e0 legacy=0x7fe6000b29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6000d0780 con 0x7fe6000d3760 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fe5fc001670 con 0x7fe6000d3760 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 >> v1:172.21.15.119:6789/0 conn(0x7fe6000d3760 legacy=0x7fe6000d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec003be0 0x7fe5ec003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5ec004870 0x7fe5ec004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 0x7fe5ec005290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6001605f0 con 0x7fe5ec004870 2024-09-13T16:31:29.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec00bbc0 con 0x7fe5ec003be0 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5ec00bd50 con 0x7fe600160210 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe604f70700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5ec004870 0x7fe5ec004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe605771700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec003be0 0x7fe5ec003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe605f72700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 0x7fe5ec005290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe604f70700 1 -- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 msgr2=0x7fe5ec005290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe604f70700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 0x7fe5ec005290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe604f70700 1 -- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec003be0 msgr2=0x7fe5ec003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe604f70700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec003be0 0x7fe5ec003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe604f70700 1 -- 172.21.15.119:0/2016046923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5ec0035d0 con 0x7fe5ec004870 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe605771700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec003be0 0x7fe5ec003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe605f72700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe600160210 0x7fe5ec005290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:29.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe604f70700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5ec004870 0x7fe5ec004cf0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fe60008cab0 tx=0x7fe5ec0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70d813bc3090a456 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc00cc50 con 0x7fe5ec004870 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5fc00cdb0 con 0x7fe5ec004870 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe5f7fff700 1 -- 172.21.15.119:0/2016046923 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5fc005b80 con 0x7fe5ec004870 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe606773700 1 -- 172.21.15.119:0/2016046923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5ec004870 msgr2=0x7fe5ec004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe606773700 1 --2- 172.21.15.119:0/2016046923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe5ec004870 0x7fe5ec004cf0 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fe60008cab0 tx=0x7fe5ec0058a0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe605f72700 1 -- 172.21.15.119:0/2016046923 reap_dead start 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe606773700 1 -- 172.21.15.119:0/2016046923 shutdown_connections 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe606773700 1 -- 172.21.15.119:0/2016046923 >> 172.21.15.119:0/2016046923 conn(0x7fe60008c080 msgr2=0x7fe6000d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe606773700 1 -- 172.21.15.119:0/2016046923 shutdown_connections 2024-09-13T16:31:29.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.761+0000 7fe606773700 1 -- 172.21.15.119:0/2016046923 wait complete. 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 Processor -- start 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 -- start start 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec004870 0x7fe6000ca7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6000cad30 0x7fe6001556f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe600155c30 0x7fe6000cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000c93e0 con 0x7fe600155c30 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6000b8350 con 0x7fe5ec004870 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe606773700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6001f33b0 con 0x7fe6000cad30 2024-09-13T16:31:29.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe605f72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe600155c30 0x7fe6000cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe605f72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe600155c30 0x7fe6000cb430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56902/0 (socket says 172.21.15.119:56902) 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe605771700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec004870 0x7fe6000ca7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe605f72700 1 -- 172.21.15.119:0/553335945 learned_addr learned my addr 172.21.15.119:0/553335945 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.765+0000 7fe604f70700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6000cad30 0x7fe6001556f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe605f72700 1 -- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6000cad30 msgr2=0x7fe6001556f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe605f72700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6000cad30 0x7fe6001556f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe605f72700 1 -- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec004870 msgr2=0x7fe6000ca7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe605f72700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec004870 0x7fe6000ca7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe605f72700 1 -- 172.21.15.119:0/553335945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6001f30d0 con 0x7fe600155c30 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe605771700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec004870 0x7fe6000ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe605f72700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe600155c30 0x7fe6000cb430 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fe6000d0560 tx=0x7fe5f800ddc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe5f5ffb700 1 -- 172.21.15.119:0/553335945 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8003e30 con 0x7fe600155c30 2024-09-13T16:31:29.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe5f5ffb700 1 -- 172.21.15.119:0/553335945 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f8005070 con 0x7fe600155c30 2024-09-13T16:31:29.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe5f5ffb700 1 -- 172.21.15.119:0/553335945 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5f8005420 con 0x7fe600155c30 2024-09-13T16:31:29.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6000c9d10 con 0x7fe600155c30 2024-09-13T16:31:29.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6000ca2d0 con 0x7fe600155c30 2024-09-13T16:31:29.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6000ca4f0 con 0x7fe600155c30 2024-09-13T16:31:29.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.769+0000 7fe5f5ffb700 1 -- 172.21.15.119:0/553335945 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe5f8005580 con 0x7fe600155c30 2024-09-13T16:31:29.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.773+0000 7fe5f5ffb700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe5e4066730 0x7fe5e4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:29.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.773+0000 7fe5f5ffb700 1 -- 172.21.15.119:0/553335945 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe5f8083d00 con 0x7fe600155c30 2024-09-13T16:31:29.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.773+0000 7fe605771700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe5e4066730 0x7fe5e4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:29.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.773+0000 7fe605771700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe5e4066730 0x7fe5e4068bf0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe6000b8710 tx=0x7fe5f0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:29.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.781+0000 7fe5f5ffb700 1 -- 172.21.15.119:0/553335945 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe5f8050fc0 con 0x7fe600155c30 2024-09-13T16:31:29.907 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-09-13T16:31:29.904+0000 7f5654b33700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:31:29.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:29.993+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe6000d3220 con 0x7fe600155c30 2024-09-13T16:31:30.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.973+0000 7fe5f5ffb700 1 -- 172.21.15.119:0/553335945 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v55) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe5f8018070 con 0x7fe600155c30 2024-09-13T16:31:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe5e4066730 msgr2=0x7fe5e4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe5e4066730 0x7fe5e4068bf0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fe6000b8710 tx=0x7fe5f0008040 comp rx=0 tx=0).stop 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe600155c30 msgr2=0x7fe6000cb430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe600155c30 0x7fe6000cb430 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fe6000d0560 tx=0x7fe5f800ddc0 comp rx=0 tx=0).stop 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 shutdown_connections 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe5e4066730 0x7fe5e4068bf0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe600155c30 0x7fe6000cb430 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6000cad30 0x7fe6001556f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 --2- 172.21.15.119:0/553335945 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe5ec004870 0x7fe6000ca7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 >> 172.21.15.119:0/553335945 conn(0x7fe60008c080 msgr2=0x7fe6000b1430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 shutdown_connections 2024-09-13T16:31:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:30.977+0000 7fe606773700 1 -- 172.21.15.119:0/553335945 wait complete. 2024-09-13T16:31:30.998 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T16:31:30.998 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:31:31.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc9364ba700 1 Processor -- start 2024-09-13T16:31:31.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc9364ba700 1 -- start start 2024-09-13T16:31:31.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc9364ba700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc93015bb70 0x7fc93015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:31.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc9364ba700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9300b4400 0x7fc9300b4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:31.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc9364ba700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc93015ae30 con 0x7fc93015aa50 2024-09-13T16:31:31.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc9364ba700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9300b4dd0 con 0x7fc93015bb70 2024-09-13T16:31:31.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc9364ba700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300ae480 con 0x7fc9300b4400 2024-09-13T16:31:31.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc934a57700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9300b4400 0x7fc9300b4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:31.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92f7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc93015bb70 0x7fc93015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:31.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc934a57700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9300b4400 0x7fc9300b4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34520/0 (socket says 172.21.15.119:34520) 2024-09-13T16:31:31.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92ffff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc93015aa50 0x7fc9300b39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:52206/0 (socket says 172.21.15.119:52206) 2024-09-13T16:31:31.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc934a57700 1 -- 172.21.15.119:0/1806850443 learned_addr learned my addr 172.21.15.119:0/1806850443 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:31.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92f7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc93015bb70 0x7fc93015c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:45562/0 (socket says 172.21.15.119:45562) 2024-09-13T16:31:31.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92f7fe700 1 -- 172.21.15.119:0/1806850443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9300b4400 msgr2=0x7fc9300b4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:31.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92f7fe700 1 --2- 172.21.15.119:0/1806850443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9300b4400 0x7fc9300b4890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92f7fe700 1 -- 172.21.15.119:0/1806850443 >> v1:172.21.15.145:6789/0 conn(0x7fc93015aa50 legacy=0x7fc9300b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:31.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92f7fe700 1 -- 172.21.15.119:0/1806850443 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9300ae760 con 0x7fc93015bb70 2024-09-13T16:31:31.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92f7fe700 1 --2- 172.21.15.119:0/1806850443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93015bb70 0x7fc93015c120 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc920009850 tx=0x7fc920009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab6fc66340d6bea8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:31.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.121+0000 7fc92effd700 1 -- 172.21.15.119:0/1806850443 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920011830 con 0x7fc93015bb70 2024-09-13T16:31:31.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc92effd700 1 -- 172.21.15.119:0/1806850443 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920011990 con 0x7fc93015bb70 2024-09-13T16:31:31.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc92effd700 1 -- 172.21.15.119:0/1806850443 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920011d40 con 0x7fc93015bb70 2024-09-13T16:31:31.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 -- 172.21.15.119:0/1806850443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93015bb70 msgr2=0x7fc93015c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:31.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 --2- 172.21.15.119:0/1806850443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93015bb70 0x7fc93015c120 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc920009850 tx=0x7fc920009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:31.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 -- 172.21.15.119:0/1806850443 shutdown_connections 2024-09-13T16:31:31.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 --2- 172.21.15.119:0/1806850443 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9300b4400 0x7fc9300b4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 --2- 172.21.15.119:0/1806850443 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc93015bb70 0x7fc93015c120 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 -- 172.21.15.119:0/1806850443 >> 172.21.15.119:0/1806850443 conn(0x7fc930099a50 msgr2=0x7fc930099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:31.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 -- 172.21.15.119:0/1806850443 shutdown_connections 2024-09-13T16:31:31.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 -- 172.21.15.119:0/1806850443 wait complete. 2024-09-13T16:31:31.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 Processor -- start 2024-09-13T16:31:31.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 -- start start 2024-09-13T16:31:31.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.125+0000 7fc9364ba700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9300b4400 0x7fc930151790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:31.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc93015aa50 0x7fc930159c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:31.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93015bb70 0x7fc930159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:31.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc930152000 con 0x7fc93015aa50 2024-09-13T16:31:31.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9301f3590 con 0x7fc9300b4400 2024-09-13T16:31:31.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9300af750 con 0x7fc93015bb70 2024-09-13T16:31:31.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc93015aa50 0x7fc930159c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:31.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc93015aa50 0x7fc930159c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56908/0 (socket says 172.21.15.119:56908) 2024-09-13T16:31:31.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 -- 172.21.15.119:0/3977174642 learned_addr learned my addr 172.21.15.119:0/3977174642 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:31.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc934a57700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93015bb70 0x7fc930159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:31.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92ffff700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9300b4400 0x7fc930151790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:31.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 -- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93015bb70 msgr2=0x7fc930159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:31.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93015bb70 0x7fc930159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 -- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9300b4400 msgr2=0x7fc930151790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9300b4400 0x7fc930151790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 -- 172.21.15.119:0/3977174642 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9200094b0 con 0x7fc93015aa50 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92ffff700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9300b4400 0x7fc930151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92f7fe700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc93015aa50 0x7fc930159c70 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc9200097a0 tx=0x7fc920004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92cff9700 1 -- 172.21.15.119:0/3977174642 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9200036c0 con 0x7fc93015aa50 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92cff9700 1 -- 172.21.15.119:0/3977174642 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc920003840 con 0x7fc93015aa50 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9300afa30 con 0x7fc93015aa50 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc92cff9700 1 -- 172.21.15.119:0/3977174642 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc920003bf0 con 0x7fc93015aa50 2024-09-13T16:31:31.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9300afff0 con 0x7fc93015aa50 2024-09-13T16:31:31.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.129+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9300b01e0 con 0x7fc93015aa50 2024-09-13T16:31:31.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.133+0000 7fc92cff9700 1 -- 172.21.15.119:0/3977174642 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc920003d50 con 0x7fc93015aa50 2024-09-13T16:31:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.133+0000 7fc92cff9700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc91c066730 0x7fc91c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.133+0000 7fc92cff9700 1 -- 172.21.15.119:0/3977174642 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc920085560 con 0x7fc93015aa50 2024-09-13T16:31:31.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.133+0000 7fc92ffff700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc91c066730 0x7fc91c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:31.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.133+0000 7fc92ffff700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc91c066730 0x7fc91c068bf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc924000f60 tx=0x7fc924007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:31.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.141+0000 7fc92cff9700 1 -- 172.21.15.119:0/3977174642 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc920052820 con 0x7fc93015aa50 2024-09-13T16:31:31.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.349+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc9300b4890 con 0x7fc93015aa50 2024-09-13T16:31:31.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.985+0000 7fc92cff9700 1 -- 172.21.15.119:0/3977174642 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v56) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc920055e40 con 0x7fc93015aa50 2024-09-13T16:31:31.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc91c066730 msgr2=0x7fc91c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:31.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc91c066730 0x7fc91c068bf0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc924000f60 tx=0x7fc924007ef0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc93015aa50 msgr2=0x7fc930159c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:31.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc93015aa50 0x7fc930159c70 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 crypto rx=0x7fc9200097a0 tx=0x7fc920004770 comp rx=0 tx=0).stop 2024-09-13T16:31:31.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 shutdown_connections 2024-09-13T16:31:31.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc91c066730 0x7fc91c068bf0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc93015bb70 0x7fc930159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc93015aa50 0x7fc930159c70 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 --2- 172.21.15.119:0/3977174642 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9300b4400 0x7fc930151790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:31.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 >> 172.21.15.119:0/3977174642 conn(0x7fc930099a50 msgr2=0x7fc930155c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:31.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 shutdown_connections 2024-09-13T16:31:31.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:31.989+0000 7fc9364ba700 1 -- 172.21.15.119:0/3977174642 wait complete. 2024-09-13T16:31:32.009 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:31:32.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:32.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.125+0000 7f15f359e700 1 Processor -- start 2024-09-13T16:31:32.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.125+0000 7f15f359e700 1 -- start start 2024-09-13T16:31:32.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f15f4158b30 0x7f15f4071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158f10 0x7f15f4071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15f406de00 con 0x7f15f406ce30 2024-09-13T16:31:32.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15f406db20 con 0x7f15f4158b30 2024-09-13T16:31:32.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f40b1920 con 0x7f15f4158f10 2024-09-13T16:31:32.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f1d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158f10 0x7f15f4071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f259c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f15f4158b30 0x7f15f4071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f2d9d700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f15f406ce30 0x7f15f406d2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:52212/0 (socket says 172.21.15.119:52212) 2024-09-13T16:31:32.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f2d9d700 1 -- 172.21.15.119:0/756692003 learned_addr learned my addr 172.21.15.119:0/756692003 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f259c700 1 -- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158f10 msgr2=0x7f15f4071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f259c700 1 --2- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158f10 0x7f15f4071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f259c700 1 -- 172.21.15.119:0/756692003 >> v1:172.21.15.145:6789/0 conn(0x7f15f406ce30 legacy=0x7f15f406d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f159a700 1 -- 172.21.15.119:0/756692003 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439314322 0 0) 0x7f15f406de00 con 0x7f15f406ce30 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f259c700 1 -- 172.21.15.119:0/756692003 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f40b1c00 con 0x7f15f4158b30 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f1d9b700 1 --2- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158f10 0x7f15f4071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f259c700 1 --2- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158b30 0x7f15f4071030 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f15e4009850 tx=0x7f15e4009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee0e8da0c2b1d837 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f159a700 1 -- 172.21.15.119:0/756692003 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4011830 con 0x7f15f4158b30 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f159a700 1 -- 172.21.15.119:0/756692003 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e4011990 con 0x7f15f4158b30 2024-09-13T16:31:32.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f159a700 1 -- 172.21.15.119:0/756692003 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4011d40 con 0x7f15f4158b30 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158b30 msgr2=0x7f15f4071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 --2- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158b30 0x7f15f4071030 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f15e4009850 tx=0x7f15e4009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- 172.21.15.119:0/756692003 shutdown_connections 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 --2- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158f10 0x7f15f4071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 --2- 172.21.15.119:0/756692003 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158b30 0x7f15f4071030 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- 172.21.15.119:0/756692003 >> 172.21.15.119:0/756692003 conn(0x7f15f406a350 msgr2=0x7f15f406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- 172.21.15.119:0/756692003 shutdown_connections 2024-09-13T16:31:32.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.129+0000 7f15f359e700 1 -- 172.21.15.119:0/756692003 wait complete. 2024-09-13T16:31:32.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 Processor -- start 2024-09-13T16:31:32.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 -- start start 2024-09-13T16:31:32.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15f406ce30 0x7f15f4164b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158b30 0x7f15f4146450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158f10 0x7f15f40ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4165f60 con 0x7f15f406ce30 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4165350 con 0x7f15f4158f10 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15f4165630 con 0x7f15f4158b30 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15f406ce30 0x7f15f4164b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15f406ce30 0x7f15f4164b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56914/0 (socket says 172.21.15.119:56914) 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 -- 172.21.15.119:0/1674213807 learned_addr learned my addr 172.21.15.119:0/1674213807 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f2d9d700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158f10 0x7f15f40ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f1d9b700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158b30 0x7f15f4146450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 -- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158b30 msgr2=0x7f15f4146450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158b30 0x7f15f4146450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 -- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158f10 msgr2=0x7f15f40ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158f10 0x7f15f40ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 -- 172.21.15.119:0/1674213807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15dc0094b0 con 0x7f15f406ce30 2024-09-13T16:31:32.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.133+0000 7f15f259c700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15f406ce30 0x7f15f4164b20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f15e4000c00 tx=0x7f15e4004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15eaffd700 1 -- 172.21.15.119:0/1674213807 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e401a990 con 0x7f15f406ce30 2024-09-13T16:31:32.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15eaffd700 1 -- 172.21.15.119:0/1674213807 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e401f070 con 0x7f15f406ce30 2024-09-13T16:31:32.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e40094b0 con 0x7f15f406ce30 2024-09-13T16:31:32.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15eaffd700 1 -- 172.21.15.119:0/1674213807 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e401ad30 con 0x7f15f406ce30 2024-09-13T16:31:32.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f4165bf0 con 0x7f15f406ce30 2024-09-13T16:31:32.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f406d2c0 con 0x7f15f406ce30 2024-09-13T16:31:32.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15eaffd700 1 -- 172.21.15.119:0/1674213807 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f15e401b430 con 0x7f15f406ce30 2024-09-13T16:31:32.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.137+0000 7f15eaffd700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f15d4066730 0x7f15d4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.141+0000 7f15eaffd700 1 -- 172.21.15.119:0/1674213807 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f15e4085280 con 0x7f15f406ce30 2024-09-13T16:31:32.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.141+0000 7f15f1d9b700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f15d4066730 0x7f15d4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.141+0000 7f15f1d9b700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f15d4066730 0x7f15d4068bf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f15f40b3550 tx=0x7f15dc009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.145+0000 7f15eaffd700 1 -- 172.21.15.119:0/1674213807 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f15e4052540 con 0x7f15f406ce30 2024-09-13T16:31:32.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.353+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f15f414c810 con 0x7f15f406ce30 2024-09-13T16:31:32.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.353+0000 7f15eaffd700 1 -- 172.21.15.119:0/1674213807 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f15e4055b60 con 0x7f15f406ce30 2024-09-13T16:31:32.358 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:32.359 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7881,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:31.973745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6835/3011171408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3011171408},{"type":"v1","addr":"172.21.15.145:6835","nonce":3011171408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:31:32.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f15d4066730 msgr2=0x7f15d4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f15d4066730 0x7f15d4068bf0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f15f40b3550 tx=0x7f15dc009240 comp rx=0 tx=0).stop 2024-09-13T16:31:32.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15f406ce30 msgr2=0x7f15f4164b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15f406ce30 0x7f15f4164b20 secure :-1 s=READY pgs=1258 cs=0 l=1 rev1=1 crypto rx=0x7f15e4000c00 tx=0x7f15e4004770 comp rx=0 tx=0).stop 2024-09-13T16:31:32.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 shutdown_connections 2024-09-13T16:31:32.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f15d4066730 0x7f15d4068bf0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15f4158f10 0x7f15f40ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f15f4158b30 0x7f15f4146450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 --2- 172.21.15.119:0/1674213807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15f406ce30 0x7f15f4164b20 unknown :-1 s=CLOSED pgs=1258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 >> 172.21.15.119:0/1674213807 conn(0x7f15f406a350 msgr2=0x7f15f406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:32.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 shutdown_connections 2024-09-13T16:31:32.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.357+0000 7f15f359e700 1 -- 172.21.15.119:0/1674213807 wait complete. 2024-09-13T16:31:32.362 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 56 2024-09-13T16:31:32.373 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:32.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6bfff700 1 Processor -- start 2024-09-13T16:31:32.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6bfff700 1 -- start start 2024-09-13T16:31:32.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd6c071740 0x7efd6c06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6bfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efd6c158f10 0x7efd6c06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6bfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd6c06de00 con 0x7efd6c158b30 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6bfff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6c06d8b0 con 0x7efd6c071740 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6bfff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6c06db20 con 0x7efd6c158f10 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6a7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd6c071740 0x7efd6c06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6a7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd6c071740 0x7efd6c06ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38186/0 (socket says 172.21.15.119:38186) 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2651181104 learned_addr learned my addr 172.21.15.119:0/2651181104 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6b7fe700 1 --2- 172.21.15.119:0/2651181104 >> v2:172.21.15.145:3300/0 conn(0x7efd6c158f10 0x7efd6c06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2651181104 >> v2:172.21.15.145:3300/0 conn(0x7efd6c158f10 msgr2=0x7efd6c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6a7fc700 1 --2- 172.21.15.119:0/2651181104 >> v2:172.21.15.145:3300/0 conn(0x7efd6c158f10 0x7efd6c06d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2651181104 >> v1:172.21.15.145:6789/0 conn(0x7efd6c158b30 legacy=0x7efd6c071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.481+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2651181104 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6c0b1920 con 0x7efd6c071740 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6a7fc700 1 --2- 172.21.15.119:0/2651181104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c071740 0x7efd6c06ce30 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0cbc10 tx=0x7efd54009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8b697f554a158c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd69ffb700 1 -- 172.21.15.119:0/2651181104 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54011830 con 0x7efd6c071740 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd69ffb700 1 -- 172.21.15.119:0/2651181104 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54011990 con 0x7efd6c071740 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd69ffb700 1 -- 172.21.15.119:0/2651181104 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54011d20 con 0x7efd6c071740 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 -- 172.21.15.119:0/2651181104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c071740 msgr2=0x7efd6c06ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2651181104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c071740 0x7efd6c06ce30 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0cbc10 tx=0x7efd54009bf0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 -- 172.21.15.119:0/2651181104 shutdown_connections 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2651181104 >> v2:172.21.15.145:3300/0 conn(0x7efd6c158f10 0x7efd6c06d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2651181104 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c071740 0x7efd6c06ce30 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 -- 172.21.15.119:0/2651181104 >> 172.21.15.119:0/2651181104 conn(0x7efd6c06a350 msgr2=0x7efd6c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:32.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 -- 172.21.15.119:0/2651181104 shutdown_connections 2024-09-13T16:31:32.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.485+0000 7efd6bfff700 1 -- 172.21.15.119:0/2651181104 wait complete. 2024-09-13T16:31:32.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 Processor -- start 2024-09-13T16:31:32.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 -- start start 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd6c071740 0x7efd6c0ce100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c158b30 0x7efd6c14ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd6c158f10 0x7efd6c1554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c15a270 con 0x7efd6c158b30 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c159660 con 0x7efd6c158f10 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6bfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd6c159940 con 0x7efd6c071740 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c158b30 0x7efd6c14ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c158b30 0x7efd6c14ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38200/0 (socket says 172.21.15.119:38200) 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2855621802 learned_addr learned my addr 172.21.15.119:0/2855621802 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6affd700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd6c071740 0x7efd6c0ce100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6b7fe700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd6c158f10 0x7efd6c1554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd6c071740 msgr2=0x7efd6c0ce100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd6c071740 0x7efd6c0ce100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd6c158f10 msgr2=0x7efd6c1554f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd6c158f10 0x7efd6c1554f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 -- 172.21.15.119:0/2855621802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd54009570 con 0x7efd6c158b30 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.489+0000 7efd6a7fc700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c158b30 0x7efd6c14ae50 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0b3ee0 tx=0x7efd54004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd637fe700 1 -- 172.21.15.119:0/2855621802 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd54004240 con 0x7efd6c158b30 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd637fe700 1 -- 172.21.15.119:0/2855621802 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5401f070 con 0x7efd6c158b30 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd6c159c20 con 0x7efd6c158b30 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd637fe700 1 -- 172.21.15.119:0/2855621802 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd540045e0 con 0x7efd6c158b30 2024-09-13T16:31:32.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6c159330 con 0x7efd6c158b30 2024-09-13T16:31:32.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd6c06c0d0 con 0x7efd6c158b30 2024-09-13T16:31:32.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd637fe700 1 -- 172.21.15.119:0/2855621802 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efd54021030 con 0x7efd6c158b30 2024-09-13T16:31:32.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.493+0000 7efd637fe700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd50066730 0x7efd50068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.497+0000 7efd637fe700 1 -- 172.21.15.119:0/2855621802 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efd54085190 con 0x7efd6c158b30 2024-09-13T16:31:32.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.497+0000 7efd6affd700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd50066730 0x7efd50068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.497+0000 7efd6affd700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd50066730 0x7efd50068bf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0cd1b0 tx=0x7efd5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.505+0000 7efd637fe700 1 -- 172.21.15.119:0/2855621802 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efd54052490 con 0x7efd6c158b30 2024-09-13T16:31:32.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.685+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efd6c1646d0 con 0x7efd6c158b30 2024-09-13T16:31:32.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.685+0000 7efd637fe700 1 -- 172.21.15.119:0/2855621802 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16777 (secure 0 0 0) 0x7efd54055ab0 con 0x7efd6c158b30 2024-09-13T16:31:32.691 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:32.691 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":95,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:31:29.963385+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd50066730 msgr2=0x7efd50068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd50066730 0x7efd50068bf0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0cd1b0 tx=0x7efd5c009040 comp rx=0 tx=0).stop 2024-09-13T16:31:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c158b30 msgr2=0x7efd6c14ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c158b30 0x7efd6c14ae50 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 crypto rx=0x7efd6c0b3ee0 tx=0x7efd54004770 comp rx=0 tx=0).stop 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 shutdown_connections 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd50066730 0x7efd50068bf0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd6c158f10 0x7efd6c1554f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd6c158b30 0x7efd6c14ae50 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 --2- 172.21.15.119:0/2855621802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd6c071740 0x7efd6c0ce100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 >> 172.21.15.119:0/2855621802 conn(0x7efd6c06a350 msgr2=0x7efd6c15d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 shutdown_connections 2024-09-13T16:31:32.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.689+0000 7efd6bfff700 1 -- 172.21.15.119:0/2855621802 wait complete. 2024-09-13T16:31:32.706 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:31:32.706 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:31:32.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 Processor -- start 2024-09-13T16:31:32.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 -- start start 2024-09-13T16:31:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff3840cd8d0 0x7ff38406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff3840cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3840705d0 con 0x7ff3840b2200 2024-09-13T16:31:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff38406d0d0 con 0x7ff3840cd8d0 2024-09-13T16:31:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff38406fca0 con 0x7ff3840b17f0 2024-09-13T16:31:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff3840cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff382d9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff3840cd8d0 0x7ff38406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff3840cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:33980/0 (socket says 172.21.15.119:33980) 2024-09-13T16:31:32.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 -- 172.21.15.119:0/4096116194 learned_addr learned my addr 172.21.15.119:0/4096116194 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:32.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38359e700 1 --1- 172.21.15.119:0/4096116194 >> v1:172.21.15.145:6789/0 conn(0x7ff3840b2200 0x7ff384070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:52980/0 (socket says 172.21.15.119:52980) 2024-09-13T16:31:32.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 -- 172.21.15.119:0/4096116194 >> v2:172.21.15.145:3300/0 conn(0x7ff3840cd8d0 msgr2=0x7ff38406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:32.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 --2- 172.21.15.119:0/4096116194 >> v2:172.21.15.145:3300/0 conn(0x7ff3840cd8d0 0x7ff38406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 -- 172.21.15.119:0/4096116194 >> v1:172.21.15.145:6789/0 conn(0x7ff3840b2200 legacy=0x7ff384070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 -- 172.21.15.119:0/4096116194 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff38406ff80 con 0x7ff3840b17f0 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff382d9d700 1 --2- 172.21.15.119:0/4096116194 >> v2:172.21.15.145:3300/0 conn(0x7ff3840cd8d0 0x7ff38406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff381d9b700 1 -- 172.21.15.119:0/4096116194 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212194580 0 0) 0x7ff3840705d0 con 0x7ff3840b2200 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38259c700 1 --2- 172.21.15.119:0/4096116194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff3840cb800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009e50 tx=0x7ff37c00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2eafa8ff627e8ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff381d9b700 1 -- 172.21.15.119:0/4096116194 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00e070 con 0x7ff3840b17f0 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff381d9b700 1 -- 172.21.15.119:0/4096116194 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c00b910 con 0x7ff3840b17f0 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff381d9b700 1 -- 172.21.15.119:0/4096116194 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c00bcc0 con 0x7ff3840b17f0 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 -- 172.21.15.119:0/4096116194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 msgr2=0x7ff3840cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.813+0000 7ff38888d700 1 --2- 172.21.15.119:0/4096116194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff3840cb800 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff37c009e50 tx=0x7ff37c00b330 comp rx=0 tx=0).stop 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- 172.21.15.119:0/4096116194 shutdown_connections 2024-09-13T16:31:32.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 --2- 172.21.15.119:0/4096116194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff3840cb800 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 --2- 172.21.15.119:0/4096116194 >> v2:172.21.15.145:3300/0 conn(0x7ff3840cd8d0 0x7ff38406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- 172.21.15.119:0/4096116194 >> 172.21.15.119:0/4096116194 conn(0x7ff38406a350 msgr2=0x7ff38406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:32.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- 172.21.15.119:0/4096116194 shutdown_connections 2024-09-13T16:31:32.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- 172.21.15.119:0/4096116194 wait complete. 2024-09-13T16:31:32.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 Processor -- start 2024-09-13T16:31:32.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- start start 2024-09-13T16:31:32.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff38415ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3840b2200 0x7ff3840ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3840cd8d0 0x7ff3840aa870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff38415cfd0 con 0x7ff3840b2200 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840ae830 con 0x7ff3840cd8d0 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38888d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3840ab2c0 con 0x7ff3840b17f0 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3840b2200 0x7ff3840ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3840b2200 0x7ff3840ae2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38212/0 (socket says 172.21.15.119:38212) 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38259c700 1 -- 172.21.15.119:0/3876466629 learned_addr learned my addr 172.21.15.119:0/3876466629 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff382d9d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff38415ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.817+0000 7ff38359e700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3840cd8d0 0x7ff3840aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38259c700 1 -- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 msgr2=0x7ff38415ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38259c700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff38415ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38259c700 1 -- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3840cd8d0 msgr2=0x7ff3840aa870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38259c700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3840cd8d0 0x7ff3840aa870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:32.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38259c700 1 -- 172.21.15.119:0/3876466629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3780094b0 con 0x7ff3840b2200 2024-09-13T16:31:32.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff382d9d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff38415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:32.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38259c700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3840b2200 0x7ff3840ae2f0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ff37c012000 tx=0x7ff37c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff36b7fe700 1 -- 172.21.15.119:0/3876466629 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c0099b0 con 0x7ff3840b2200 2024-09-13T16:31:32.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff36b7fe700 1 -- 172.21.15.119:0/3876466629 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff37c0092e0 con 0x7ff3840b2200 2024-09-13T16:31:32.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff37c009b30 con 0x7ff3840b2200 2024-09-13T16:31:32.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff36b7fe700 1 -- 172.21.15.119:0/3876466629 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff37c01c640 con 0x7ff3840b2200 2024-09-13T16:31:32.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3840a8930 con 0x7ff3840b2200 2024-09-13T16:31:32.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3840a84b0 con 0x7ff3840b2200 2024-09-13T16:31:32.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff36b7fe700 1 -- 172.21.15.119:0/3876466629 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff37c01c7a0 con 0x7ff3840b2200 2024-09-13T16:31:32.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.821+0000 7ff36b7fe700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff370066730 0x7ff370068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:32.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.825+0000 7ff36b7fe700 1 -- 172.21.15.119:0/3876466629 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff37c085b60 con 0x7ff3840b2200 2024-09-13T16:31:32.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.825+0000 7ff382d9d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff370066730 0x7ff370068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:32.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.825+0000 7ff382d9d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff370066730 0x7ff370068bf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff3840aff90 tx=0x7ff378008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:32.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.829+0000 7ff36b7fe700 1 -- 172.21.15.119:0/3876466629 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff3840a84b0 con 0x7ff3840b2200 2024-09-13T16:31:33.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:32.997+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3840a84b0 con 0x7ff370066730 2024-09-13T16:31:33.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.013+0000 7ff36b7fe700 1 -- 172.21.15.119:0/3876466629 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288123 (secure 0 0 0) 0x7ff3840a84b0 con 0x7ff370066730 2024-09-13T16:31:33.020 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff370066730 msgr2=0x7ff370068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff370066730 0x7ff370068bf0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7ff3840aff90 tx=0x7ff378008040 comp rx=0 tx=0).stop 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3840b2200 msgr2=0x7ff3840ae2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3840b2200 0x7ff3840ae2f0 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 crypto rx=0x7ff37c012000 tx=0x7ff37c004af0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 shutdown_connections 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff370066730 0x7ff370068bf0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3840cd8d0 0x7ff3840aa870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3840b2200 0x7ff3840ae2f0 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 --2- 172.21.15.119:0/3876466629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3840b17f0 0x7ff38415ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 >> 172.21.15.119:0/3876466629 conn(0x7ff38406a350 msgr2=0x7ff3840cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:33.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 shutdown_connections 2024-09-13T16:31:33.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.017+0000 7ff38888d700 1 -- 172.21.15.119:0/3876466629 wait complete. 2024-09-13T16:31:33.024 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:31:33.035 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-13T16:31:31.552134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246956,"kb_used_data":10272,"kb_used_omap":12,"kb_used_meta":236531,"kb_avail":754727764,"statfs":{"total":773094113280,"available":772841230336,"internally_reserved":0,"allocated":10518528,"data_stored":5404332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":242208313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001318"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987612+0000","last_change":"2024-09-13T16:31:28.986864+0000","last_active":"2024-09-13T16:31:28.987612+0000","last_peered":"2024-09-13T16:31:28.987612+0000","last_clean":"2024-09-13T16:31:28.987612+0000","last_became_active":"2024-09-13T16:31:28.986630+0000","last_became_peered":"2024-09-13T16:31:28.986630+0000","last_unstale":"2024-09-13T16:31:28.987612+0000","last_undegraded":"2024-09-13T16:31:28.987612+0000","last_fullsized":"2024-09-13T16:31:28.987612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:31.210815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976840+0000","last_change":"2024-09-13T16:31:28.989436+0000","last_active":"2024-09-13T16:31:29.976840+0000","last_peered":"2024-09-13T16:31:29.976840+0000","last_clean":"2024-09-13T16:31:29.976840+0000","last_became_active":"2024-09-13T16:31:28.989194+0000","last_became_peered":"2024-09-13T16:31:28.989194+0000","last_unstale":"2024-09-13T16:31:29.976840+0000","last_undegraded":"2024-09-13T16:31:29.976840+0000","last_fullsized":"2024-09-13T16:31:29.976840+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:40.385225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987569+0000","last_change":"2024-09-13T16:31:28.986161+0000","last_active":"2024-09-13T16:31:28.987569+0000","last_peered":"2024-09-13T16:31:28.987569+0000","last_clean":"2024-09-13T16:31:28.987569+0000","last_became_active":"2024-09-13T16:31:28.985946+0000","last_became_peered":"2024-09-13T16:31:28.985946+0000","last_unstale":"2024-09-13T16:31:28.987569+0000","last_undegraded":"2024-09-13T16:31:28.987569+0000","last_fullsized":"2024-09-13T16:31:28.987569+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:27.985372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.005380+0000","last_change":"2024-09-13T16:31:28.999374+0000","last_active":"2024-09-13T16:31:29.005380+0000","last_peered":"2024-09-13T16:31:29.005380+0000","last_clean":"2024-09-13T16:31:29.005380+0000","last_became_active":"2024-09-13T16:31:28.999123+0000","last_became_peered":"2024-09-13T16:31:28.999123+0000","last_unstale":"2024-09-13T16:31:29.005380+0000","last_undegraded":"2024-09-13T16:31:29.005380+0000","last_fullsized":"2024-09-13T16:31:29.005380+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:40:51.659631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.994847+0000","last_change":"2024-09-13T16:31:28.993117+0000","last_active":"2024-09-13T16:31:28.994847+0000","last_peered":"2024-09-13T16:31:28.994847+0000","last_clean":"2024-09-13T16:31:28.994847+0000","last_became_active":"2024-09-13T16:31:28.991936+0000","last_became_peered":"2024-09-13T16:31:28.991936+0000","last_unstale":"2024-09-13T16:31:28.994847+0000","last_undegraded":"2024-09-13T16:31:28.994847+0000","last_fullsized":"2024-09-13T16:31:28.994847+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:44.707877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973881+0000","last_change":"2024-09-13T16:31:28.993865+0000","last_active":"2024-09-13T16:31:29.973881+0000","last_peered":"2024-09-13T16:31:29.973881+0000","last_clean":"2024-09-13T16:31:29.973881+0000","last_became_active":"2024-09-13T16:31:28.993620+0000","last_became_peered":"2024-09-13T16:31:28.993620+0000","last_unstale":"2024-09-13T16:31:29.973881+0000","last_undegraded":"2024-09-13T16:31:29.973881+0000","last_fullsized":"2024-09-13T16:31:29.973881+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:23.952163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987337+0000","last_change":"2024-09-13T16:31:28.986192+0000","last_active":"2024-09-13T16:31:28.987337+0000","last_peered":"2024-09-13T16:31:28.987337+0000","last_clean":"2024-09-13T16:31:28.987337+0000","last_became_active":"2024-09-13T16:31:28.985940+0000","last_became_peered":"2024-09-13T16:31:28.985940+0000","last_unstale":"2024-09-13T16:31:28.987337+0000","last_undegraded":"2024-09-13T16:31:28.987337+0000","last_fullsized":"2024-09-13T16:31:28.987337+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:20.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000091+0000","last_change":"2024-09-13T16:31:28.995964+0000","last_active":"2024-09-13T16:31:29.000091+0000","last_peered":"2024-09-13T16:31:29.000091+0000","last_clean":"2024-09-13T16:31:29.000091+0000","last_became_active":"2024-09-13T16:31:28.995655+0000","last_became_peered":"2024-09-13T16:31:28.995655+0000","last_unstale":"2024-09-13T16:31:29.000091+0000","last_undegraded":"2024-09-13T16:31:29.000091+0000","last_fullsized":"2024-09-13T16:31:29.000091+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:42.752127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.986935+0000","last_change":"2024-09-13T16:31:28.984913+0000","last_active":"2024-09-13T16:31:28.986935+0000","last_peered":"2024-09-13T16:31:28.986935+0000","last_clean":"2024-09-13T16:31:28.986935+0000","last_became_active":"2024-09-13T16:31:28.984652+0000","last_became_peered":"2024-09-13T16:31:28.984652+0000","last_unstale":"2024-09-13T16:31:28.986935+0000","last_undegraded":"2024-09-13T16:31:28.986935+0000","last_fullsized":"2024-09-13T16:31:28.986935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:28.189762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999494+0000","last_change":"2024-09-13T16:31:28.997778+0000","last_active":"2024-09-13T16:31:28.999494+0000","last_peered":"2024-09-13T16:31:28.999494+0000","last_clean":"2024-09-13T16:31:28.999494+0000","last_became_active":"2024-09-13T16:31:28.997478+0000","last_became_peered":"2024-09-13T16:31:28.997478+0000","last_unstale":"2024-09-13T16:31:28.999494+0000","last_undegraded":"2024-09-13T16:31:28.999494+0000","last_fullsized":"2024-09-13T16:31:28.999494+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:39.125689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.985206+0000","last_change":"2024-09-13T16:31:28.981973+0000","last_active":"2024-09-13T16:31:28.985206+0000","last_peered":"2024-09-13T16:31:28.985206+0000","last_clean":"2024-09-13T16:31:28.985206+0000","last_became_active":"2024-09-13T16:31:28.981649+0000","last_became_peered":"2024-09-13T16:31:28.981649+0000","last_unstale":"2024-09-13T16:31:28.985206+0000","last_undegraded":"2024-09-13T16:31:28.985206+0000","last_fullsized":"2024-09-13T16:31:28.985206+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:35.100645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.992588+0000","last_change":"2024-09-13T16:31:28.988272+0000","last_active":"2024-09-13T16:31:28.992588+0000","last_peered":"2024-09-13T16:31:28.992588+0000","last_clean":"2024-09-13T16:31:28.992588+0000","last_became_active":"2024-09-13T16:31:28.988033+0000","last_became_peered":"2024-09-13T16:31:28.988033+0000","last_unstale":"2024-09-13T16:31:28.992588+0000","last_undegraded":"2024-09-13T16:31:28.992588+0000","last_fullsized":"2024-09-13T16:31:28.992588+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:13.778271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999613+0000","last_change":"2024-09-13T16:31:28.993596+0000","last_active":"2024-09-13T16:31:28.999613+0000","last_peered":"2024-09-13T16:31:28.999613+0000","last_clean":"2024-09-13T16:31:28.999613+0000","last_became_active":"2024-09-13T16:31:28.993346+0000","last_became_peered":"2024-09-13T16:31:28.993346+0000","last_unstale":"2024-09-13T16:31:28.999613+0000","last_undegraded":"2024-09-13T16:31:28.999613+0000","last_fullsized":"2024-09-13T16:31:28.999613+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:08.055096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.996696+0000","last_change":"2024-09-13T16:31:28.994048+0000","last_active":"2024-09-13T16:31:28.996696+0000","last_peered":"2024-09-13T16:31:28.996696+0000","last_clean":"2024-09-13T16:31:28.996696+0000","last_became_active":"2024-09-13T16:31:28.993752+0000","last_became_peered":"2024-09-13T16:31:28.993752+0000","last_unstale":"2024-09-13T16:31:28.996696+0000","last_undegraded":"2024-09-13T16:31:28.996696+0000","last_fullsized":"2024-09-13T16:31:28.996696+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:19.943107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.998744+0000","last_change":"2024-09-13T16:31:28.984539+0000","last_active":"2024-09-13T16:31:28.998744+0000","last_peered":"2024-09-13T16:31:28.998744+0000","last_clean":"2024-09-13T16:31:28.998744+0000","last_became_active":"2024-09-13T16:31:28.984301+0000","last_became_peered":"2024-09-13T16:31:28.984301+0000","last_unstale":"2024-09-13T16:31:28.998744+0000","last_undegraded":"2024-09-13T16:31:28.998744+0000","last_fullsized":"2024-09-13T16:31:28.998744+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:04.807834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973652+0000","last_change":"2024-09-13T16:31:28.989144+0000","last_active":"2024-09-13T16:31:29.973652+0000","last_peered":"2024-09-13T16:31:29.973652+0000","last_clean":"2024-09-13T16:31:29.973652+0000","last_became_active":"2024-09-13T16:31:28.988953+0000","last_became_peered":"2024-09-13T16:31:28.988953+0000","last_unstale":"2024-09-13T16:31:29.973652+0000","last_undegraded":"2024-09-13T16:31:29.973652+0000","last_fullsized":"2024-09-13T16:31:29.973652+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:14.307773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974938+0000","last_change":"2024-09-13T16:31:28.987501+0000","last_active":"2024-09-13T16:31:29.974938+0000","last_peered":"2024-09-13T16:31:29.974938+0000","last_clean":"2024-09-13T16:31:29.974938+0000","last_became_active":"2024-09-13T16:31:28.987252+0000","last_became_peered":"2024-09-13T16:31:28.987252+0000","last_unstale":"2024-09-13T16:31:29.974938+0000","last_undegraded":"2024-09-13T16:31:29.974938+0000","last_fullsized":"2024-09-13T16:31:29.974938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:25:15.223155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.997060+0000","last_change":"2024-09-13T16:31:28.990706+0000","last_active":"2024-09-13T16:31:28.997060+0000","last_peered":"2024-09-13T16:31:28.997060+0000","last_clean":"2024-09-13T16:31:28.997060+0000","last_became_active":"2024-09-13T16:31:28.989799+0000","last_became_peered":"2024-09-13T16:31:28.989799+0000","last_unstale":"2024-09-13T16:31:28.997060+0000","last_undegraded":"2024-09-13T16:31:28.997060+0000","last_fullsized":"2024-09-13T16:31:28.997060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:13:05.163905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.994831+0000","last_change":"2024-09-13T16:31:28.993492+0000","last_active":"2024-09-13T16:31:28.994831+0000","last_peered":"2024-09-13T16:31:28.994831+0000","last_clean":"2024-09-13T16:31:28.994831+0000","last_became_active":"2024-09-13T16:31:28.993209+0000","last_became_peered":"2024-09-13T16:31:28.993209+0000","last_unstale":"2024-09-13T16:31:28.994831+0000","last_undegraded":"2024-09-13T16:31:28.994831+0000","last_fullsized":"2024-09-13T16:31:28.994831+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:12.341081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999581+0000","last_change":"2024-09-13T16:31:28.993482+0000","last_active":"2024-09-13T16:31:28.999581+0000","last_peered":"2024-09-13T16:31:28.999581+0000","last_clean":"2024-09-13T16:31:28.999581+0000","last_became_active":"2024-09-13T16:31:28.993210+0000","last_became_peered":"2024-09-13T16:31:28.993210+0000","last_unstale":"2024-09-13T16:31:28.999581+0000","last_undegraded":"2024-09-13T16:31:28.999581+0000","last_fullsized":"2024-09-13T16:31:28.999581+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:01.604548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.982916+0000","last_change":"2024-09-13T16:31:28.980061+0000","last_active":"2024-09-13T16:31:28.982916+0000","last_peered":"2024-09-13T16:31:28.982916+0000","last_clean":"2024-09-13T16:31:28.982916+0000","last_became_active":"2024-09-13T16:31:28.979737+0000","last_became_peered":"2024-09-13T16:31:28.979737+0000","last_unstale":"2024-09-13T16:31:28.982916+0000","last_undegraded":"2024-09-13T16:31:28.982916+0000","last_fullsized":"2024-09-13T16:31:28.982916+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:54.464831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987835+0000","last_change":"2024-09-13T16:31:28.986415+0000","last_active":"2024-09-13T16:31:28.987835+0000","last_peered":"2024-09-13T16:31:28.987835+0000","last_clean":"2024-09-13T16:31:28.987835+0000","last_became_active":"2024-09-13T16:31:28.986078+0000","last_became_peered":"2024-09-13T16:31:28.986078+0000","last_unstale":"2024-09-13T16:31:28.987835+0000","last_undegraded":"2024-09-13T16:31:28.987835+0000","last_fullsized":"2024-09-13T16:31:28.987835+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:52.726980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975144+0000","last_change":"2024-09-13T16:31:28.998567+0000","last_active":"2024-09-13T16:31:29.975144+0000","last_peered":"2024-09-13T16:31:29.975144+0000","last_clean":"2024-09-13T16:31:29.975144+0000","last_became_active":"2024-09-13T16:31:28.996499+0000","last_became_peered":"2024-09-13T16:31:28.996499+0000","last_unstale":"2024-09-13T16:31:29.975144+0000","last_undegraded":"2024-09-13T16:31:29.975144+0000","last_fullsized":"2024-09-13T16:31:29.975144+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:06.203951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.993116+0000","last_change":"2024-09-13T16:31:28.989201+0000","last_active":"2024-09-13T16:31:28.993116+0000","last_peered":"2024-09-13T16:31:28.993116+0000","last_clean":"2024-09-13T16:31:28.993116+0000","last_became_active":"2024-09-13T16:31:28.988747+0000","last_became_peered":"2024-09-13T16:31:28.988747+0000","last_unstale":"2024-09-13T16:31:28.993116+0000","last_undegraded":"2024-09-13T16:31:28.993116+0000","last_fullsized":"2024-09-13T16:31:28.993116+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:49.183526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.977109+0000","last_change":"2024-09-13T16:31:28.999547+0000","last_active":"2024-09-13T16:31:29.977109+0000","last_peered":"2024-09-13T16:31:29.977109+0000","last_clean":"2024-09-13T16:31:29.977109+0000","last_became_active":"2024-09-13T16:31:28.999267+0000","last_became_peered":"2024-09-13T16:31:28.999267+0000","last_unstale":"2024-09-13T16:31:29.977109+0000","last_undegraded":"2024-09-13T16:31:29.977109+0000","last_fullsized":"2024-09-13T16:31:29.977109+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:53.477490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.986984+0000","last_change":"2024-09-13T16:31:28.984405+0000","last_active":"2024-09-13T16:31:28.986984+0000","last_peered":"2024-09-13T16:31:28.986984+0000","last_clean":"2024-09-13T16:31:28.986984+0000","last_became_active":"2024-09-13T16:31:28.984111+0000","last_became_peered":"2024-09-13T16:31:28.984111+0000","last_unstale":"2024-09-13T16:31:28.986984+0000","last_undegraded":"2024-09-13T16:31:28.986984+0000","last_fullsized":"2024-09-13T16:31:28.986984+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:20.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.995295+0000","last_change":"2024-09-13T16:31:28.992166+0000","last_active":"2024-09-13T16:31:28.995295+0000","last_peered":"2024-09-13T16:31:28.995295+0000","last_clean":"2024-09-13T16:31:28.995295+0000","last_became_active":"2024-09-13T16:31:28.991971+0000","last_became_peered":"2024-09-13T16:31:28.991971+0000","last_unstale":"2024-09-13T16:31:28.995295+0000","last_undegraded":"2024-09-13T16:31:28.995295+0000","last_fullsized":"2024-09-13T16:31:28.995295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:55.182947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999893+0000","last_change":"2024-09-13T16:31:28.994159+0000","last_active":"2024-09-13T16:31:28.999893+0000","last_peered":"2024-09-13T16:31:28.999893+0000","last_clean":"2024-09-13T16:31:28.999893+0000","last_became_active":"2024-09-13T16:31:28.993898+0000","last_became_peered":"2024-09-13T16:31:28.993898+0000","last_unstale":"2024-09-13T16:31:28.999893+0000","last_undegraded":"2024-09-13T16:31:28.999893+0000","last_fullsized":"2024-09-13T16:31:28.999893+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:45.243501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.983360+0000","last_change":"2024-09-13T16:31:28.981209+0000","last_active":"2024-09-13T16:31:28.983360+0000","last_peered":"2024-09-13T16:31:28.983360+0000","last_clean":"2024-09-13T16:31:28.983360+0000","last_became_active":"2024-09-13T16:31:28.980954+0000","last_became_peered":"2024-09-13T16:31:28.980954+0000","last_unstale":"2024-09-13T16:31:28.983360+0000","last_undegraded":"2024-09-13T16:31:28.983360+0000","last_fullsized":"2024-09-13T16:31:28.983360+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:40.162978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987026+0000","last_change":"2024-09-13T16:31:28.985248+0000","last_active":"2024-09-13T16:31:28.987026+0000","last_peered":"2024-09-13T16:31:28.987026+0000","last_clean":"2024-09-13T16:31:28.987026+0000","last_became_active":"2024-09-13T16:31:28.984624+0000","last_became_peered":"2024-09-13T16:31:28.984624+0000","last_unstale":"2024-09-13T16:31:28.987026+0000","last_undegraded":"2024-09-13T16:31:28.987026+0000","last_fullsized":"2024-09-13T16:31:28.987026+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:54.455329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976178+0000","last_change":"2024-09-13T16:31:28.993472+0000","last_active":"2024-09-13T16:31:29.976178+0000","last_peered":"2024-09-13T16:31:29.976178+0000","last_clean":"2024-09-13T16:31:29.976178+0000","last_became_active":"2024-09-13T16:31:28.993233+0000","last_became_peered":"2024-09-13T16:31:28.993233+0000","last_unstale":"2024-09-13T16:31:29.976178+0000","last_undegraded":"2024-09-13T16:31:29.976178+0000","last_fullsized":"2024-09-13T16:31:29.976178+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:10.870778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973576+0000","last_change":"2024-09-13T16:31:28.986953+0000","last_active":"2024-09-13T16:31:29.973576+0000","last_peered":"2024-09-13T16:31:29.973576+0000","last_clean":"2024-09-13T16:31:29.973576+0000","last_became_active":"2024-09-13T16:31:28.986715+0000","last_became_peered":"2024-09-13T16:31:28.986715+0000","last_unstale":"2024-09-13T16:31:29.973576+0000","last_undegraded":"2024-09-13T16:31:29.973576+0000","last_fullsized":"2024-09-13T16:31:29.973576+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:41.605098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974122+0000","last_change":"2024-09-13T16:31:28.977336+0000","last_active":"2024-09-13T16:31:29.974122+0000","last_peered":"2024-09-13T16:31:29.974122+0000","last_clean":"2024-09-13T16:31:29.974122+0000","last_became_active":"2024-09-13T16:31:28.976843+0000","last_became_peered":"2024-09-13T16:31:28.976843+0000","last_unstale":"2024-09-13T16:31:29.974122+0000","last_undegraded":"2024-09-13T16:31:29.974122+0000","last_fullsized":"2024-09-13T16:31:29.974122+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:24:22.340211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"94'4","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975646+0000","last_change":"2024-09-13T16:31:28.611651+0000","last_active":"2024-09-13T16:31:29.975646+0000","last_peered":"2024-09-13T16:31:29.975646+0000","last_clean":"2024-09-13T16:31:29.975646+0000","last_became_active":"2024-09-13T16:31:28.611310+0000","last_became_peered":"2024-09-13T16:31:28.611310+0000","last_unstale":"2024-09-13T16:31:29.975646+0000","last_undegraded":"2024-09-13T16:31:29.975646+0000","last_fullsized":"2024-09-13T16:31:29.975646+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:20.209081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974478+0000","last_change":"2024-09-13T16:31:28.983292+0000","last_active":"2024-09-13T16:31:29.974478+0000","last_peered":"2024-09-13T16:31:29.974478+0000","last_clean":"2024-09-13T16:31:29.974478+0000","last_became_active":"2024-09-13T16:31:28.983053+0000","last_became_peered":"2024-09-13T16:31:28.983053+0000","last_unstale":"2024-09-13T16:31:29.974478+0000","last_undegraded":"2024-09-13T16:31:29.974478+0000","last_fullsized":"2024-09-13T16:31:29.974478+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:03.519510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.001156+0000","last_change":"2024-09-13T16:31:28.604634+0000","last_active":"2024-09-13T16:31:29.001156+0000","last_peered":"2024-09-13T16:31:29.001156+0000","last_clean":"2024-09-13T16:31:29.001156+0000","last_became_active":"2024-09-13T16:31:28.603979+0000","last_became_peered":"2024-09-13T16:31:28.603979+0000","last_unstale":"2024-09-13T16:31:29.001156+0000","last_undegraded":"2024-09-13T16:31:29.001156+0000","last_fullsized":"2024-09-13T16:31:29.001156+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:52.354399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.996927+0000","last_change":"2024-09-13T16:31:28.995090+0000","last_active":"2024-09-13T16:31:28.996927+0000","last_peered":"2024-09-13T16:31:28.996927+0000","last_clean":"2024-09-13T16:31:28.996927+0000","last_became_active":"2024-09-13T16:31:28.994764+0000","last_became_peered":"2024-09-13T16:31:28.994764+0000","last_unstale":"2024-09-13T16:31:28.996927+0000","last_undegraded":"2024-09-13T16:31:28.996927+0000","last_fullsized":"2024-09-13T16:31:28.996927+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:19.943107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.993964+0000","last_change":"2024-09-13T16:31:28.609486+0000","last_active":"2024-09-13T16:31:28.993964+0000","last_peered":"2024-09-13T16:31:28.993964+0000","last_clean":"2024-09-13T16:31:28.993964+0000","last_became_active":"2024-09-13T16:31:28.609063+0000","last_became_peered":"2024-09-13T16:31:28.609063+0000","last_unstale":"2024-09-13T16:31:28.993964+0000","last_undegraded":"2024-09-13T16:31:28.993964+0000","last_fullsized":"2024-09-13T16:31:28.993964+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:36.879842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975116+0000","last_change":"2024-09-13T16:31:28.997207+0000","last_active":"2024-09-13T16:31:29.975116+0000","last_peered":"2024-09-13T16:31:29.975116+0000","last_clean":"2024-09-13T16:31:29.975116+0000","last_became_active":"2024-09-13T16:31:28.996929+0000","last_became_peered":"2024-09-13T16:31:28.996929+0000","last_unstale":"2024-09-13T16:31:29.975116+0000","last_undegraded":"2024-09-13T16:31:29.975116+0000","last_fullsized":"2024-09-13T16:31:29.975116+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:25.710493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"94'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975472+0000","last_change":"2024-09-13T16:31:28.607386+0000","last_active":"2024-09-13T16:31:29.975472+0000","last_peered":"2024-09-13T16:31:29.975472+0000","last_clean":"2024-09-13T16:31:29.975472+0000","last_became_active":"2024-09-13T16:31:28.607100+0000","last_became_peered":"2024-09-13T16:31:28.607100+0000","last_unstale":"2024-09-13T16:31:29.975472+0000","last_undegraded":"2024-09-13T16:31:29.975472+0000","last_fullsized":"2024-09-13T16:31:29.975472+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:01.296974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000110+0000","last_change":"2024-09-13T16:31:28.996128+0000","last_active":"2024-09-13T16:31:29.000110+0000","last_peered":"2024-09-13T16:31:29.000110+0000","last_clean":"2024-09-13T16:31:29.000110+0000","last_became_active":"2024-09-13T16:31:28.995707+0000","last_became_peered":"2024-09-13T16:31:28.995707+0000","last_unstale":"2024-09-13T16:31:29.000110+0000","last_undegraded":"2024-09-13T16:31:29.000110+0000","last_fullsized":"2024-09-13T16:31:29.000110+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:30.072934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.995315+0000","last_change":"2024-09-13T16:31:28.607186+0000","last_active":"2024-09-13T16:31:28.995315+0000","last_peered":"2024-09-13T16:31:28.995315+0000","last_clean":"2024-09-13T16:31:28.995315+0000","last_became_active":"2024-09-13T16:31:28.606835+0000","last_became_peered":"2024-09-13T16:31:28.606835+0000","last_unstale":"2024-09-13T16:31:28.995315+0000","last_undegraded":"2024-09-13T16:31:28.995315+0000","last_fullsized":"2024-09-13T16:31:28.995315+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:49.477532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975134+0000","last_change":"2024-09-13T16:31:28.996481+0000","last_active":"2024-09-13T16:31:29.975134+0000","last_peered":"2024-09-13T16:31:29.975134+0000","last_clean":"2024-09-13T16:31:29.975134+0000","last_became_active":"2024-09-13T16:31:28.996104+0000","last_became_peered":"2024-09-13T16:31:28.996104+0000","last_unstale":"2024-09-13T16:31:29.975134+0000","last_undegraded":"2024-09-13T16:31:29.975134+0000","last_fullsized":"2024-09-13T16:31:29.975134+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:09.621978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.998823+0000","last_change":"2024-09-13T16:31:28.609506+0000","last_active":"2024-09-13T16:31:28.998823+0000","last_peered":"2024-09-13T16:31:28.998823+0000","last_clean":"2024-09-13T16:31:28.998823+0000","last_became_active":"2024-09-13T16:31:28.609246+0000","last_became_peered":"2024-09-13T16:31:28.609246+0000","last_unstale":"2024-09-13T16:31:28.998823+0000","last_undegraded":"2024-09-13T16:31:28.998823+0000","last_fullsized":"2024-09-13T16:31:28.998823+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:17.805587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973834+0000","last_change":"2024-09-13T16:31:28.994339+0000","last_active":"2024-09-13T16:31:29.973834+0000","last_peered":"2024-09-13T16:31:29.973834+0000","last_clean":"2024-09-13T16:31:29.973834+0000","last_became_active":"2024-09-13T16:31:28.994105+0000","last_became_peered":"2024-09-13T16:31:28.994105+0000","last_unstale":"2024-09-13T16:31:29.973834+0000","last_undegraded":"2024-09-13T16:31:29.973834+0000","last_fullsized":"2024-09-13T16:31:29.973834+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:18.769898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.978582+0000","last_change":"2024-09-13T16:31:28.600186+0000","last_active":"2024-09-13T16:31:28.978582+0000","last_peered":"2024-09-13T16:31:28.978582+0000","last_clean":"2024-09-13T16:31:28.978582+0000","last_became_active":"2024-09-13T16:31:28.599926+0000","last_became_peered":"2024-09-13T16:31:28.599926+0000","last_unstale":"2024-09-13T16:31:28.978582+0000","last_undegraded":"2024-09-13T16:31:28.978582+0000","last_fullsized":"2024-09-13T16:31:28.978582+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:28.148770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000009+0000","last_change":"2024-09-13T16:31:28.996577+0000","last_active":"2024-09-13T16:31:29.000009+0000","last_peered":"2024-09-13T16:31:29.000009+0000","last_clean":"2024-09-13T16:31:29.000009+0000","last_became_active":"2024-09-13T16:31:28.996352+0000","last_became_peered":"2024-09-13T16:31:28.996352+0000","last_unstale":"2024-09-13T16:31:29.000009+0000","last_undegraded":"2024-09-13T16:31:29.000009+0000","last_fullsized":"2024-09-13T16:31:29.000009+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:43.147656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.977316+0000","last_change":"2024-09-13T16:31:28.608593+0000","last_active":"2024-09-13T16:31:28.977316+0000","last_peered":"2024-09-13T16:31:28.977316+0000","last_clean":"2024-09-13T16:31:28.977316+0000","last_became_active":"2024-09-13T16:31:28.608324+0000","last_became_peered":"2024-09-13T16:31:28.608324+0000","last_unstale":"2024-09-13T16:31:28.977316+0000","last_undegraded":"2024-09-13T16:31:28.977316+0000","last_fullsized":"2024-09-13T16:31:28.977316+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:29.062736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987035+0000","last_change":"2024-09-13T16:31:28.985889+0000","last_active":"2024-09-13T16:31:28.987035+0000","last_peered":"2024-09-13T16:31:28.987035+0000","last_clean":"2024-09-13T16:31:28.987035+0000","last_became_active":"2024-09-13T16:31:28.985610+0000","last_became_peered":"2024-09-13T16:31:28.985610+0000","last_unstale":"2024-09-13T16:31:28.987035+0000","last_undegraded":"2024-09-13T16:31:28.987035+0000","last_fullsized":"2024-09-13T16:31:28.987035+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:04.807834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.977200+0000","last_change":"2024-09-13T16:31:28.606601+0000","last_active":"2024-09-13T16:31:29.977200+0000","last_peered":"2024-09-13T16:31:29.977200+0000","last_clean":"2024-09-13T16:31:29.977200+0000","last_became_active":"2024-09-13T16:31:28.606330+0000","last_became_peered":"2024-09-13T16:31:28.606330+0000","last_unstale":"2024-09-13T16:31:29.977200+0000","last_undegraded":"2024-09-13T16:31:29.977200+0000","last_fullsized":"2024-09-13T16:31:29.977200+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:30.287705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.985874+0000","last_change":"2024-09-13T16:31:28.982778+0000","last_active":"2024-09-13T16:31:28.985874+0000","last_peered":"2024-09-13T16:31:28.985874+0000","last_clean":"2024-09-13T16:31:28.985874+0000","last_became_active":"2024-09-13T16:31:28.982486+0000","last_became_peered":"2024-09-13T16:31:28.982486+0000","last_unstale":"2024-09-13T16:31:28.985874+0000","last_undegraded":"2024-09-13T16:31:28.985874+0000","last_fullsized":"2024-09-13T16:31:28.985874+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:35.100645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.980217+0000","last_change":"2024-09-13T16:31:28.602408+0000","last_active":"2024-09-13T16:31:28.980217+0000","last_peered":"2024-09-13T16:31:28.980217+0000","last_clean":"2024-09-13T16:31:28.980217+0000","last_became_active":"2024-09-13T16:31:28.602164+0000","last_became_peered":"2024-09-13T16:31:28.602164+0000","last_unstale":"2024-09-13T16:31:28.980217+0000","last_undegraded":"2024-09-13T16:31:28.980217+0000","last_fullsized":"2024-09-13T16:31:28.980217+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:33.768254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987220+0000","last_change":"2024-09-13T16:31:28.985748+0000","last_active":"2024-09-13T16:31:28.987220+0000","last_peered":"2024-09-13T16:31:28.987220+0000","last_clean":"2024-09-13T16:31:28.987220+0000","last_became_active":"2024-09-13T16:31:28.985545+0000","last_became_peered":"2024-09-13T16:31:28.985545+0000","last_unstale":"2024-09-13T16:31:28.987220+0000","last_undegraded":"2024-09-13T16:31:28.987220+0000","last_fullsized":"2024-09-13T16:31:28.987220+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:29.318165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"94'3","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.997192+0000","last_change":"2024-09-13T16:31:28.608759+0000","last_active":"2024-09-13T16:31:28.997192+0000","last_peered":"2024-09-13T16:31:28.997192+0000","last_clean":"2024-09-13T16:31:28.997192+0000","last_became_active":"2024-09-13T16:31:28.608469+0000","last_became_peered":"2024-09-13T16:31:28.608469+0000","last_unstale":"2024-09-13T16:31:28.997192+0000","last_undegraded":"2024-09-13T16:31:28.997192+0000","last_fullsized":"2024-09-13T16:31:28.997192+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:42.720080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999920+0000","last_change":"2024-09-13T16:31:28.994056+0000","last_active":"2024-09-13T16:31:28.999920+0000","last_peered":"2024-09-13T16:31:28.999920+0000","last_clean":"2024-09-13T16:31:28.999920+0000","last_became_active":"2024-09-13T16:31:28.993800+0000","last_became_peered":"2024-09-13T16:31:28.993800+0000","last_unstale":"2024-09-13T16:31:28.999920+0000","last_undegraded":"2024-09-13T16:31:28.999920+0000","last_fullsized":"2024-09-13T16:31:28.999920+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:54.020569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"94'6","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974324+0000","last_change":"2024-09-13T16:31:28.611488+0000","last_active":"2024-09-13T16:31:29.974324+0000","last_peered":"2024-09-13T16:31:29.974324+0000","last_clean":"2024-09-13T16:31:29.974324+0000","last_became_active":"2024-09-13T16:31:28.611287+0000","last_became_peered":"2024-09-13T16:31:28.611287+0000","last_unstale":"2024-09-13T16:31:29.974324+0000","last_undegraded":"2024-09-13T16:31:29.974324+0000","last_fullsized":"2024-09-13T16:31:29.974324+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:14.999923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000127+0000","last_change":"2024-09-13T16:31:28.996368+0000","last_active":"2024-09-13T16:31:29.000127+0000","last_peered":"2024-09-13T16:31:29.000127+0000","last_clean":"2024-09-13T16:31:29.000127+0000","last_became_active":"2024-09-13T16:31:28.995902+0000","last_became_peered":"2024-09-13T16:31:28.995902+0000","last_unstale":"2024-09-13T16:31:29.000127+0000","last_undegraded":"2024-09-13T16:31:29.000127+0000","last_fullsized":"2024-09-13T16:31:29.000127+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:19.383061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.992983+0000","last_change":"2024-09-13T16:31:28.606615+0000","last_active":"2024-09-13T16:31:28.992983+0000","last_peered":"2024-09-13T16:31:28.992983+0000","last_clean":"2024-09-13T16:31:28.992983+0000","last_became_active":"2024-09-13T16:31:28.606351+0000","last_became_peered":"2024-09-13T16:31:28.606351+0000","last_unstale":"2024-09-13T16:31:28.992983+0000","last_undegraded":"2024-09-13T16:31:28.992983+0000","last_fullsized":"2024-09-13T16:31:28.992983+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:00.594349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.001351+0000","last_change":"2024-09-13T16:31:28.999890+0000","last_active":"2024-09-13T16:31:29.001351+0000","last_peered":"2024-09-13T16:31:29.001351+0000","last_clean":"2024-09-13T16:31:29.001351+0000","last_became_active":"2024-09-13T16:31:28.999619+0000","last_became_peered":"2024-09-13T16:31:28.999619+0000","last_unstale":"2024-09-13T16:31:29.001351+0000","last_undegraded":"2024-09-13T16:31:29.001351+0000","last_fullsized":"2024-09-13T16:31:29.001351+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:32.970089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973925+0000","last_change":"2024-09-13T16:31:28.610323+0000","last_active":"2024-09-13T16:31:29.973925+0000","last_peered":"2024-09-13T16:31:29.973925+0000","last_clean":"2024-09-13T16:31:29.973925+0000","last_became_active":"2024-09-13T16:31:28.610085+0000","last_became_peered":"2024-09-13T16:31:28.610085+0000","last_unstale":"2024-09-13T16:31:29.973925+0000","last_undegraded":"2024-09-13T16:31:29.973925+0000","last_fullsized":"2024-09-13T16:31:29.973925+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:50.439035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.998935+0000","last_change":"2024-09-13T16:31:28.997655+0000","last_active":"2024-09-13T16:31:28.998935+0000","last_peered":"2024-09-13T16:31:28.998935+0000","last_clean":"2024-09-13T16:31:28.998935+0000","last_became_active":"2024-09-13T16:31:28.997374+0000","last_became_peered":"2024-09-13T16:31:28.997374+0000","last_unstale":"2024-09-13T16:31:28.998935+0000","last_undegraded":"2024-09-13T16:31:28.998935+0000","last_fullsized":"2024-09-13T16:31:28.998935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:47.873917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.975100+0000","last_change":"2024-09-13T16:31:28.607220+0000","last_active":"2024-09-13T16:31:28.975100+0000","last_peered":"2024-09-13T16:31:28.975100+0000","last_clean":"2024-09-13T16:31:28.975100+0000","last_became_active":"2024-09-13T16:31:28.606872+0000","last_became_peered":"2024-09-13T16:31:28.606872+0000","last_unstale":"2024-09-13T16:31:28.975100+0000","last_undegraded":"2024-09-13T16:31:28.975100+0000","last_fullsized":"2024-09-13T16:31:28.975100+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:40.303706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.983815+0000","last_change":"2024-09-13T16:31:28.980615+0000","last_active":"2024-09-13T16:31:28.983815+0000","last_peered":"2024-09-13T16:31:28.983815+0000","last_clean":"2024-09-13T16:31:28.983815+0000","last_became_active":"2024-09-13T16:31:28.980246+0000","last_became_peered":"2024-09-13T16:31:28.980246+0000","last_unstale":"2024-09-13T16:31:28.983815+0000","last_undegraded":"2024-09-13T16:31:28.983815+0000","last_fullsized":"2024-09-13T16:31:28.983815+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:45.871680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"94'8","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999334+0000","last_change":"2024-09-13T16:31:28.609114+0000","last_active":"2024-09-13T16:31:28.999334+0000","last_peered":"2024-09-13T16:31:28.999334+0000","last_clean":"2024-09-13T16:31:28.999334+0000","last_became_active":"2024-09-13T16:31:28.608136+0000","last_became_peered":"2024-09-13T16:31:28.608136+0000","last_unstale":"2024-09-13T16:31:28.999334+0000","last_undegraded":"2024-09-13T16:31:28.999334+0000","last_fullsized":"2024-09-13T16:31:28.999334+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:47.319412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974225+0000","last_change":"2024-09-13T16:31:28.977196+0000","last_active":"2024-09-13T16:31:29.974225+0000","last_peered":"2024-09-13T16:31:29.974225+0000","last_clean":"2024-09-13T16:31:29.974225+0000","last_became_active":"2024-09-13T16:31:28.976364+0000","last_became_peered":"2024-09-13T16:31:28.976364+0000","last_unstale":"2024-09-13T16:31:29.974225+0000","last_undegraded":"2024-09-13T16:31:29.974225+0000","last_fullsized":"2024-09-13T16:31:29.974225+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:57.203509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"94'1","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975921+0000","last_change":"2024-09-13T16:31:28.607683+0000","last_active":"2024-09-13T16:31:29.975921+0000","last_peered":"2024-09-13T16:31:29.975921+0000","last_clean":"2024-09-13T16:31:29.975921+0000","last_became_active":"2024-09-13T16:31:28.607258+0000","last_became_peered":"2024-09-13T16:31:28.607258+0000","last_unstale":"2024-09-13T16:31:29.975921+0000","last_undegraded":"2024-09-13T16:31:29.975921+0000","last_fullsized":"2024-09-13T16:31:29.975921+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:41.523462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975472+0000","last_change":"2024-09-13T16:31:28.996123+0000","last_active":"2024-09-13T16:31:29.975472+0000","last_peered":"2024-09-13T16:31:29.975472+0000","last_clean":"2024-09-13T16:31:29.975472+0000","last_became_active":"2024-09-13T16:31:28.995799+0000","last_became_peered":"2024-09-13T16:31:28.995799+0000","last_unstale":"2024-09-13T16:31:29.975472+0000","last_undegraded":"2024-09-13T16:31:29.975472+0000","last_fullsized":"2024-09-13T16:31:29.975472+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:48.233359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.994696+0000","last_change":"2024-09-13T16:31:28.602961+0000","last_active":"2024-09-13T16:31:28.994696+0000","last_peered":"2024-09-13T16:31:28.994696+0000","last_clean":"2024-09-13T16:31:28.994696+0000","last_became_active":"2024-09-13T16:31:28.602678+0000","last_became_peered":"2024-09-13T16:31:28.602678+0000","last_unstale":"2024-09-13T16:31:28.994696+0000","last_undegraded":"2024-09-13T16:31:28.994696+0000","last_fullsized":"2024-09-13T16:31:28.994696+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:44.233302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.982946+0000","last_change":"2024-09-13T16:31:28.980653+0000","last_active":"2024-09-13T16:31:28.982946+0000","last_peered":"2024-09-13T16:31:28.982946+0000","last_clean":"2024-09-13T16:31:28.982946+0000","last_became_active":"2024-09-13T16:31:28.980270+0000","last_became_peered":"2024-09-13T16:31:28.980270+0000","last_unstale":"2024-09-13T16:31:28.982946+0000","last_undegraded":"2024-09-13T16:31:28.982946+0000","last_fullsized":"2024-09-13T16:31:28.982946+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:40.930927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999540+0000","last_change":"2024-09-13T16:31:28.607551+0000","last_active":"2024-09-13T16:31:28.999540+0000","last_peered":"2024-09-13T16:31:28.999540+0000","last_clean":"2024-09-13T16:31:28.999540+0000","last_became_active":"2024-09-13T16:31:28.607290+0000","last_became_peered":"2024-09-13T16:31:28.607290+0000","last_unstale":"2024-09-13T16:31:28.999540+0000","last_undegraded":"2024-09-13T16:31:28.999540+0000","last_fullsized":"2024-09-13T16:31:28.999540+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:05.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973684+0000","last_change":"2024-09-13T16:31:28.996100+0000","last_active":"2024-09-13T16:31:29.973684+0000","last_peered":"2024-09-13T16:31:29.973684+0000","last_clean":"2024-09-13T16:31:29.973684+0000","last_became_active":"2024-09-13T16:31:28.995838+0000","last_became_peered":"2024-09-13T16:31:28.995838+0000","last_unstale":"2024-09-13T16:31:29.973684+0000","last_undegraded":"2024-09-13T16:31:29.973684+0000","last_fullsized":"2024-09-13T16:31:29.973684+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:08.102040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"94'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976482+0000","last_change":"2024-09-13T16:31:28.602778+0000","last_active":"2024-09-13T16:31:29.976482+0000","last_peered":"2024-09-13T16:31:29.976482+0000","last_clean":"2024-09-13T16:31:29.976482+0000","last_became_active":"2024-09-13T16:31:28.602415+0000","last_became_peered":"2024-09-13T16:31:28.602415+0000","last_unstale":"2024-09-13T16:31:29.976482+0000","last_undegraded":"2024-09-13T16:31:29.976482+0000","last_fullsized":"2024-09-13T16:31:29.976482+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:59.521610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976362+0000","last_change":"2024-09-13T16:31:28.989696+0000","last_active":"2024-09-13T16:31:29.976362+0000","last_peered":"2024-09-13T16:31:29.976362+0000","last_clean":"2024-09-13T16:31:29.976362+0000","last_became_active":"2024-09-13T16:31:28.989443+0000","last_became_peered":"2024-09-13T16:31:28.989443+0000","last_unstale":"2024-09-13T16:31:29.976362+0000","last_undegraded":"2024-09-13T16:31:29.976362+0000","last_fullsized":"2024-09-13T16:31:29.976362+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:30.810042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.973445+0000","last_change":"2024-09-13T16:31:28.605291+0000","last_active":"2024-09-13T16:31:28.973445+0000","last_peered":"2024-09-13T16:31:28.973445+0000","last_clean":"2024-09-13T16:31:28.973445+0000","last_became_active":"2024-09-13T16:31:28.604986+0000","last_became_peered":"2024-09-13T16:31:28.604986+0000","last_unstale":"2024-09-13T16:31:28.973445+0000","last_undegraded":"2024-09-13T16:31:28.973445+0000","last_fullsized":"2024-09-13T16:31:28.973445+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.829434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976326+0000","last_change":"2024-09-13T16:31:28.993957+0000","last_active":"2024-09-13T16:31:29.976326+0000","last_peered":"2024-09-13T16:31:29.976326+0000","last_clean":"2024-09-13T16:31:29.976326+0000","last_became_active":"2024-09-13T16:31:28.993619+0000","last_became_peered":"2024-09-13T16:31:28.993619+0000","last_unstale":"2024-09-13T16:31:29.976326+0000","last_undegraded":"2024-09-13T16:31:29.976326+0000","last_fullsized":"2024-09-13T16:31:29.976326+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:58.626444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976255+0000","last_change":"2024-09-13T16:31:28.604158+0000","last_active":"2024-09-13T16:31:29.976255+0000","last_peered":"2024-09-13T16:31:29.976255+0000","last_clean":"2024-09-13T16:31:29.976255+0000","last_became_active":"2024-09-13T16:31:28.603864+0000","last_became_peered":"2024-09-13T16:31:28.603864+0000","last_unstale":"2024-09-13T16:31:29.976255+0000","last_undegraded":"2024-09-13T16:31:29.976255+0000","last_fullsized":"2024-09-13T16:31:29.976255+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:33.584847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973888+0000","last_change":"2024-09-13T16:31:28.983253+0000","last_active":"2024-09-13T16:31:29.973888+0000","last_peered":"2024-09-13T16:31:29.973888+0000","last_clean":"2024-09-13T16:31:29.973888+0000","last_became_active":"2024-09-13T16:31:28.983008+0000","last_became_peered":"2024-09-13T16:31:28.983008+0000","last_unstale":"2024-09-13T16:31:29.973888+0000","last_undegraded":"2024-09-13T16:31:29.973888+0000","last_fullsized":"2024-09-13T16:31:29.973888+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:39:44.101104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.971826+0000","last_change":"2024-09-13T16:31:28.605300+0000","last_active":"2024-09-13T16:31:28.971826+0000","last_peered":"2024-09-13T16:31:28.971826+0000","last_clean":"2024-09-13T16:31:28.971826+0000","last_became_active":"2024-09-13T16:31:28.605027+0000","last_became_peered":"2024-09-13T16:31:28.605027+0000","last_unstale":"2024-09-13T16:31:28.971826+0000","last_undegraded":"2024-09-13T16:31:28.971826+0000","last_fullsized":"2024-09-13T16:31:28.971826+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:25:46.128750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975489+0000","last_change":"2024-09-13T16:31:28.988236+0000","last_active":"2024-09-13T16:31:29.975489+0000","last_peered":"2024-09-13T16:31:29.975489+0000","last_clean":"2024-09-13T16:31:29.975489+0000","last_became_active":"2024-09-13T16:31:28.984106+0000","last_became_peered":"2024-09-13T16:31:28.984106+0000","last_unstale":"2024-09-13T16:31:29.975489+0000","last_undegraded":"2024-09-13T16:31:29.975489+0000","last_fullsized":"2024-09-13T16:31:29.975489+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:50.487730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999070+0000","last_change":"2024-09-13T16:31:28.606027+0000","last_active":"2024-09-13T16:31:28.999070+0000","last_peered":"2024-09-13T16:31:28.999070+0000","last_clean":"2024-09-13T16:31:28.999070+0000","last_became_active":"2024-09-13T16:31:28.605602+0000","last_became_peered":"2024-09-13T16:31:28.605602+0000","last_unstale":"2024-09-13T16:31:28.999070+0000","last_undegraded":"2024-09-13T16:31:28.999070+0000","last_fullsized":"2024-09-13T16:31:28.999070+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:53.445131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974522+0000","last_change":"2024-09-13T16:31:28.975660+0000","last_active":"2024-09-13T16:31:29.974522+0000","last_peered":"2024-09-13T16:31:29.974522+0000","last_clean":"2024-09-13T16:31:29.974522+0000","last_became_active":"2024-09-13T16:31:28.975169+0000","last_became_peered":"2024-09-13T16:31:28.975169+0000","last_unstale":"2024-09-13T16:31:29.974522+0000","last_undegraded":"2024-09-13T16:31:29.974522+0000","last_fullsized":"2024-09-13T16:31:29.974522+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:14:05.502528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.973177+0000","last_change":"2024-09-13T16:31:28.607851+0000","last_active":"2024-09-13T16:31:28.973177+0000","last_peered":"2024-09-13T16:31:28.973177+0000","last_clean":"2024-09-13T16:31:28.973177+0000","last_became_active":"2024-09-13T16:31:28.604623+0000","last_became_peered":"2024-09-13T16:31:28.604623+0000","last_unstale":"2024-09-13T16:31:28.973177+0000","last_undegraded":"2024-09-13T16:31:28.973177+0000","last_fullsized":"2024-09-13T16:31:28.973177+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:47.223161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974869+0000","last_change":"2024-09-13T16:31:28.980115+0000","last_active":"2024-09-13T16:31:29.974869+0000","last_peered":"2024-09-13T16:31:29.974869+0000","last_clean":"2024-09-13T16:31:29.974869+0000","last_became_active":"2024-09-13T16:31:28.979882+0000","last_became_peered":"2024-09-13T16:31:28.979882+0000","last_unstale":"2024-09-13T16:31:29.974869+0000","last_undegraded":"2024-09-13T16:31:29.974869+0000","last_fullsized":"2024-09-13T16:31:29.974869+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:38.621442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.973669+0000","last_change":"2024-09-13T16:31:28.607390+0000","last_active":"2024-09-13T16:31:28.973669+0000","last_peered":"2024-09-13T16:31:28.973669+0000","last_clean":"2024-09-13T16:31:28.973669+0000","last_became_active":"2024-09-13T16:31:28.606888+0000","last_became_peered":"2024-09-13T16:31:28.606888+0000","last_unstale":"2024-09-13T16:31:28.973669+0000","last_undegraded":"2024-09-13T16:31:28.973669+0000","last_fullsized":"2024-09-13T16:31:28.973669+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:47.176642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.977181+0000","last_change":"2024-09-13T16:31:28.980514+0000","last_active":"2024-09-13T16:31:29.977181+0000","last_peered":"2024-09-13T16:31:29.977181+0000","last_clean":"2024-09-13T16:31:29.977181+0000","last_became_active":"2024-09-13T16:31:28.980255+0000","last_became_peered":"2024-09-13T16:31:28.980255+0000","last_unstale":"2024-09-13T16:31:29.977181+0000","last_undegraded":"2024-09-13T16:31:29.977181+0000","last_fullsized":"2024-09-13T16:31:29.977181+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:18.033839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975648+0000","last_change":"2024-09-13T16:31:28.609741+0000","last_active":"2024-09-13T16:31:29.975648+0000","last_peered":"2024-09-13T16:31:29.975648+0000","last_clean":"2024-09-13T16:31:29.975648+0000","last_became_active":"2024-09-13T16:31:28.609500+0000","last_became_peered":"2024-09-13T16:31:28.609500+0000","last_unstale":"2024-09-13T16:31:29.975648+0000","last_undegraded":"2024-09-13T16:31:29.975648+0000","last_fullsized":"2024-09-13T16:31:29.975648+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:22.642570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.990388+0000","last_change":"2024-09-13T16:31:28.984233+0000","last_active":"2024-09-13T16:31:28.990388+0000","last_peered":"2024-09-13T16:31:28.990388+0000","last_clean":"2024-09-13T16:31:28.990388+0000","last_became_active":"2024-09-13T16:31:28.983943+0000","last_became_peered":"2024-09-13T16:31:28.983943+0000","last_unstale":"2024-09-13T16:31:28.990388+0000","last_undegraded":"2024-09-13T16:31:28.990388+0000","last_fullsized":"2024-09-13T16:31:28.990388+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:44.781084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974312+0000","last_change":"2024-09-13T16:31:28.611849+0000","last_active":"2024-09-13T16:31:29.974312+0000","last_peered":"2024-09-13T16:31:29.974312+0000","last_clean":"2024-09-13T16:31:29.974312+0000","last_became_active":"2024-09-13T16:31:28.611687+0000","last_became_peered":"2024-09-13T16:31:28.611687+0000","last_unstale":"2024-09-13T16:31:29.974312+0000","last_undegraded":"2024-09-13T16:31:29.974312+0000","last_fullsized":"2024-09-13T16:31:29.974312+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:01.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000855+0000","last_change":"2024-09-13T16:31:28.999444+0000","last_active":"2024-09-13T16:31:29.000855+0000","last_peered":"2024-09-13T16:31:29.000855+0000","last_clean":"2024-09-13T16:31:29.000855+0000","last_became_active":"2024-09-13T16:31:28.999146+0000","last_became_peered":"2024-09-13T16:31:28.999146+0000","last_unstale":"2024-09-13T16:31:29.000855+0000","last_undegraded":"2024-09-13T16:31:29.000855+0000","last_fullsized":"2024-09-13T16:31:29.000855+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:15.075638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976420+0000","last_change":"2024-09-13T16:31:28.602782+0000","last_active":"2024-09-13T16:31:29.976420+0000","last_peered":"2024-09-13T16:31:29.976420+0000","last_clean":"2024-09-13T16:31:29.976420+0000","last_became_active":"2024-09-13T16:31:28.602477+0000","last_became_peered":"2024-09-13T16:31:28.602477+0000","last_unstale":"2024-09-13T16:31:29.976420+0000","last_undegraded":"2024-09-13T16:31:29.976420+0000","last_fullsized":"2024-09-13T16:31:29.976420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:31:22.385998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974993+0000","last_change":"2024-09-13T16:31:28.996549+0000","last_active":"2024-09-13T16:31:29.974993+0000","last_peered":"2024-09-13T16:31:29.974993+0000","last_clean":"2024-09-13T16:31:29.974993+0000","last_became_active":"2024-09-13T16:31:28.996209+0000","last_became_peered":"2024-09-13T16:31:28.996209+0000","last_unstale":"2024-09-13T16:31:29.974993+0000","last_undegraded":"2024-09-13T16:31:29.974993+0000","last_fullsized":"2024-09-13T16:31:29.974993+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:53.364598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975307+0000","last_change":"2024-09-13T16:31:28.609266+0000","last_active":"2024-09-13T16:31:29.975307+0000","last_peered":"2024-09-13T16:31:29.975307+0000","last_clean":"2024-09-13T16:31:29.975307+0000","last_became_active":"2024-09-13T16:31:28.609002+0000","last_became_peered":"2024-09-13T16:31:28.609002+0000","last_unstale":"2024-09-13T16:31:29.975307+0000","last_undegraded":"2024-09-13T16:31:29.975307+0000","last_fullsized":"2024-09-13T16:31:29.975307+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:06.461863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974179+0000","last_change":"2024-09-13T16:31:28.987013+0000","last_active":"2024-09-13T16:31:29.974179+0000","last_peered":"2024-09-13T16:31:29.974179+0000","last_clean":"2024-09-13T16:31:29.974179+0000","last_became_active":"2024-09-13T16:31:28.986765+0000","last_became_peered":"2024-09-13T16:31:28.986765+0000","last_unstale":"2024-09-13T16:31:29.974179+0000","last_undegraded":"2024-09-13T16:31:29.974179+0000","last_fullsized":"2024-09-13T16:31:29.974179+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:04.155701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.977582+0000","last_change":"2024-09-13T16:31:28.602885+0000","last_active":"2024-09-13T16:31:28.977582+0000","last_peered":"2024-09-13T16:31:28.977582+0000","last_clean":"2024-09-13T16:31:28.977582+0000","last_became_active":"2024-09-13T16:31:28.602600+0000","last_became_peered":"2024-09-13T16:31:28.602600+0000","last_unstale":"2024-09-13T16:31:28.977582+0000","last_undegraded":"2024-09-13T16:31:28.977582+0000","last_fullsized":"2024-09-13T16:31:28.977582+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:14.555270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.989750+0000","last_change":"2024-09-13T16:31:28.988583+0000","last_active":"2024-09-13T16:31:28.989750+0000","last_peered":"2024-09-13T16:31:28.989750+0000","last_clean":"2024-09-13T16:31:28.989750+0000","last_became_active":"2024-09-13T16:31:28.988370+0000","last_became_peered":"2024-09-13T16:31:28.988370+0000","last_unstale":"2024-09-13T16:31:28.989750+0000","last_undegraded":"2024-09-13T16:31:28.989750+0000","last_fullsized":"2024-09-13T16:31:28.989750+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:12.341081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.982290+0000","last_change":"2024-09-13T16:31:28.603914+0000","last_active":"2024-09-13T16:31:28.982290+0000","last_peered":"2024-09-13T16:31:28.982290+0000","last_clean":"2024-09-13T16:31:28.982290+0000","last_became_active":"2024-09-13T16:31:28.603522+0000","last_became_peered":"2024-09-13T16:31:28.603522+0000","last_unstale":"2024-09-13T16:31:28.982290+0000","last_undegraded":"2024-09-13T16:31:28.982290+0000","last_fullsized":"2024-09-13T16:31:28.982290+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:22.642570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":228,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.593497+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:31:28.593497+0000","last_peered":"2024-09-13T16:31:28.593497+0000","last_clean":"2024-09-13T16:31:28.593497+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:31:28.593497+0000","last_undegraded":"2024-09-13T16:31:28.593497+0000","last_fullsized":"2024-09-13T16:31:28.593497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542223,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1757184,"data_stored":1116128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.85899999999999999,"15min":0.85299999999999998},"min":{"1min":0.59099999999999997,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1259999999999999,"5min":1.357,"15min":1.357},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.872,"15min":0.85199999999999998},"min":{"1min":0.65400000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.075,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.61799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.107,"5min":1.569,"15min":1.569},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.86299999999999999,"15min":0.84899999999999998},"min":{"1min":0.65500000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1020000000000001,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.623}]},{"osd":2,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.2330000000000001,"5min":1.242,"15min":1.242},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89000000000000001,"15min":0.84999999999999998},"min":{"1min":0.71699999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1919999999999999,"5min":1.339,"15min":1.339},"last":0.71399999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.93899999999999995,"15min":0.99299999999999999},"min":{"1min":0.64500000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.127,"5min":1.476,"15min":1.476},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99099999999999999,"15min":1.0049999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.21,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.79500000000000004}]},{"osd":4,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84799999999999998,"15min":0.88},"min":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.198,"5min":1.216,"15min":1.216},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.97099999999999997,"15min":1.0589999999999999},"min":{"1min":0.63900000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2030000000000001,"5min":1.5549999999999999,"15min":1.589},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.94199999999999995,"15min":1.012},"min":{"1min":0.70099999999999996,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.276,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0760000000000001,"15min":1.1080000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.242,"5min":1.3200000000000001,"15min":1.337},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.03,"15min":1.071},"min":{"1min":0.85599999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.3069999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.087,"15min":1.1579999999999999},"min":{"1min":0.84999999999999998,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.2569999999999999,"5min":1.623,"15min":1.6779999999999999},"last":0.57499999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542222,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1187840,"data_stored":534492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.59199999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93400000000000005,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88,"15min":0.89100000000000001},"min":{"1min":0.63400000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1399999999999999,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.71299999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1799999999999999,"15min":1.2030000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92700000000000005,"15min":0.91900000000000004},"min":{"1min":0.65500000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1140000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.85299999999999998,"15min":0.878},"min":{"1min":0.57499999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.101,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89100000000000001,"15min":0.85899999999999999},"min":{"1min":0.72699999999999998,"5min":0.56899999999999995,"15min":0.55500000000000005},"max":{"1min":1.097,"5min":1.276,"15min":1.276},"last":0.78000000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.95699999999999996,"15min":0.96499999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2410000000000001,"5min":1.5189999999999999,"15min":1.536},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.98099999999999998,"15min":0.98699999999999999},"min":{"1min":0.71299999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.133,"5min":1.3120000000000001,"15min":1.409},"last":0.83599999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.93100000000000005,"15min":0.96699999999999997},"min":{"1min":0.69499999999999995,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.177,"5min":1.4570000000000001,"15min":1.4750000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0009999999999999,"15min":0.95299999999999996},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.347,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.91600000000000004}]},{"osd":5,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.95599999999999996,"15min":0.92800000000000005},"min":{"1min":0.81799999999999995,"5min":0.628,"15min":0.628},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0629999999999999,"15min":1.1180000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.286,"5min":1.4259999999999999,"15min":1.6439999999999999},"last":0.96999999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.089,"15min":1.113},"min":{"1min":0.80000000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.256,"5min":1.522,"15min":1.522},"last":0.999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.0660000000000001,"15min":1.0609999999999999},"min":{"1min":0.78500000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.415,"5min":1.427,"15min":1.427},"last":1.0429999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542224,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1155072,"data_stored":525704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83799999999999997,"15min":0.85399999999999998},"min":{"1min":0.54600000000000004,"5min":0.47899999999999998,"15min":0.47599999999999998},"max":{"1min":1.0940000000000001,"5min":1.411,"15min":1.411},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82599999999999996,"15min":0.84799999999999998},"min":{"1min":0.59999999999999998,"5min":0.48299999999999998,"15min":0.41699999999999998},"max":{"1min":1.1499999999999999,"5min":1.3049999999999999,"15min":1.675},"last":0.70299999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.85199999999999998,"15min":0.876},"min":{"1min":0.59699999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.093,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.83599999999999997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.36499999999999999},"max":{"1min":1.0960000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.88900000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.90400000000000003,"15min":0.92800000000000005},"min":{"1min":0.68500000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1299999999999999,"5min":1.276,"15min":1.625},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.94299999999999995,"15min":0.97699999999999998},"min":{"1min":0.67900000000000005,"5min":0.57599999999999996,"15min":0.55000000000000004},"max":{"1min":1.1990000000000001,"5min":1.516,"15min":1.552},"last":1.036}]},{"osd":3,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.85299999999999998},"min":{"1min":0.71799999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.054,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90000000000000002,"15min":0.99199999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.60499999999999998},"max":{"1min":1.141,"5min":1.1819999999999999,"15min":3.8460000000000001},"last":0.85899999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.93600000000000005,"15min":1.0369999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.224,"5min":1.655,"15min":3.6869999999999998},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0629999999999999,"15min":1.0609999999999999},"min":{"1min":0.91200000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.244,"5min":1.47,"15min":1.601},"last":1.093}]},{"osd":6,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.93899999999999995,"15min":0.96599999999999997},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.123,"5min":1.319,"15min":1.421},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.0429999999999999,"15min":1.0669999999999999},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.28,"5min":2.21,"15min":2.21},"last":1.121}]},{"osd":7,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0369999999999999,"15min":1.0349999999999999},"min":{"1min":0.65900000000000003,"5min":0.621,"15min":0.45500000000000002},"max":{"1min":1.2909999999999999,"5min":1.3819999999999999,"15min":1.4530000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0660000000000001,"15min":1.1930000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.333,"5min":1.51,"15min":3.9830000000000001},"last":1.0629999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542222,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1163264,"data_stored":525864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83899999999999997,"15min":0.83699999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1990000000000001,"5min":1.411,"15min":1.411},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.89500000000000002,"15min":0.86499999999999999},"min":{"1min":0.58499999999999996,"5min":0.52800000000000002,"15min":0.44600000000000001},"max":{"1min":1.462,"5min":2.899,"15min":2.899},"last":0.96199999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92000000000000004,"15min":0.91800000000000004},"min":{"1min":0.66400000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.2490000000000001,"5min":3.085,"15min":3.085},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.91900000000000004,"15min":0.91700000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.3240000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.84399999999999997,"15min":0.83799999999999997},"min":{"1min":0.59599999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.1339999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.97299999999999998,"15min":0.96899999999999997},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.498},"max":{"1min":1.357,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":0.73499999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.92800000000000005,"15min":0.94899999999999995},"min":{"1min":0.56899999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.421,"5min":1.421,"15min":1.472},"last":0.876},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92700000000000005,"15min":0.95299999999999996},"min":{"1min":0.73699999999999999,"5min":0.55100000000000005,"15min":0.52200000000000002},"max":{"1min":1.1599999999999999,"5min":1.577,"15min":1.577},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.96599999999999997,"15min":0.97399999999999998},"min":{"1min":0.61599999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.2370000000000001,"5min":2.996,"15min":2.996},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.04,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.5189999999999999,"5min":1.79,"15min":2.8860000000000001},"last":0.98499999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.63300000000000001,"5min":0.623,"15min":0.623},"max":{"1min":1.3959999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.127,"15min":1.1100000000000001},"min":{"1min":0.878,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.2330000000000001,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.0109999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0940000000000001,"15min":1.129},"min":{"1min":0.68200000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2410000000000001,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0980000000000001,"15min":1.1120000000000001},"min":{"1min":0.81699999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3759999999999999,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.91900000000000004}]}]},{"osd":3,"up_from":14,"seq":60129542223,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1155072,"data_stored":525704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.70299999999999996,"5min":1.04,"15min":1.04},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67700000000000005,"15min":0.66300000000000003},"min":{"1min":0.51100000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":1.161,"15min":1.161},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.67400000000000004,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.378,"15min":0.35499999999999998},"max":{"1min":0.81799999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.51000000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0049999999999999,"5min":1.3029999999999999,"15min":1.4119999999999999},"last":0.69799999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.61199999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91300000000000003,"5min":1.0649999999999999,"15min":1.27},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78600000000000003,"15min":0.76200000000000001},"min":{"1min":0.58499999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.0509999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64100000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.91900000000000004,"15min":0.90300000000000002},"min":{"1min":0.73499999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.121,"5min":1.288,"15min":1.8460000000000001},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.94499999999999995,"15min":0.95699999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.091,"5min":1.538,"15min":1.782},"last":0.77400000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.59499999999999997},"max":{"1min":1.3029999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.046,"15min":1.087},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.329,"5min":1.484,"15min":1.8140000000000001},"last":0.79200000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0600000000000001,"15min":1.117},"min":{"1min":0.77600000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3919999999999999,"5min":1.6950000000000001,"15min":1.885},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.0029999999999999,"15min":1.048},"min":{"1min":0.82699999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.331,"5min":1.4770000000000001,"15min":1.95},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0780000000000001,"15min":1.0760000000000001},"min":{"1min":0.85499999999999998,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.284,"5min":1.889,"15min":1.889},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.0469999999999999,"15min":1.0569999999999999},"min":{"1min":0.84299999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.1899999999999999,"5min":1.756,"15min":1.756},"last":0.81299999999999994}]}]},{"osd":2,"up_from":14,"seq":60129542224,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1155072,"data_stored":525704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.39800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0229999999999999,"5min":1.069,"15min":1.069},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.433,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.079,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.42099999999999999,"15min":0.40100000000000002},"max":{"1min":0.995,"5min":0.997,"15min":0.997},"last":0.871},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68899999999999995,"15min":0.69199999999999995},"min":{"1min":0.44400000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.052,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.70799999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.68200000000000005,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.03,"5min":1.054,"15min":1.073},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.83399999999999996,"15min":0.80200000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.377},"max":{"1min":1.091,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.93600000000000005,"15min":0.95099999999999996},"min":{"1min":0.69299999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.272,"5min":1.4199999999999999,"15min":1.5149999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.92000000000000004,"15min":0.97199999999999998},"min":{"1min":0.76000000000000001,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.113,"5min":1.4330000000000001,"15min":1.8799999999999999},"last":0.98599999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.96399999999999997,"15min":0.97999999999999998},"min":{"1min":0.68400000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.452,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95999999999999996,"15min":0.96199999999999997},"min":{"1min":0.68600000000000005,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.21,"5min":1.3660000000000001,"15min":1.5720000000000001},"last":1.0980000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.93200000000000005,"15min":1.0149999999999999},"min":{"1min":0.70599999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1950000000000001,"5min":1.4690000000000001,"15min":2.024},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.069,"5min":1.0429999999999999,"15min":1.0609999999999999},"min":{"1min":0.87,"5min":0.78700000000000003,"15min":0.61799999999999999},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.6359999999999999},"last":1.0269999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.046,"15min":1.0740000000000001},"min":{"1min":0.71499999999999997,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0289999999999999,"15min":1.0609999999999999},"min":{"1min":0.81399999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.22,"5min":1.4430000000000001,"15min":1.55},"last":0.79700000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542223,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1159168,"data_stored":525774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.23300000000000001},"max":{"1min":0.86699999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.76300000000000001,"15min":0.73499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91700000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.627},"min":{"1min":0.496,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":0.755,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.871},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.45700000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":0.89900000000000002,"5min":1.3129999999999999,"15min":1.458},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.75600000000000001,"15min":0.70099999999999996},"min":{"1min":0.42499999999999999,"5min":0.39700000000000002,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.47099999999999997,"5min":0.44400000000000001,"15min":0.39600000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.403},"last":0.76500000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.93899999999999995,"15min":0.91100000000000003},"min":{"1min":0.78700000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.123,"5min":1.4930000000000001,"15min":1.546},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.90300000000000002,"15min":0.92500000000000004},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.51700000000000002},"max":{"1min":1.0529999999999999,"5min":1.377,"15min":2.2040000000000002},"last":0.91400000000000003}]},{"osd":5,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.96899999999999997,"15min":0.95099999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1870000000000001,"5min":1.389,"15min":1.389},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.91500000000000004,"15min":0.96299999999999997},"min":{"1min":0.56699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0740000000000001,"5min":1.3979999999999999,"15min":2.3029999999999999},"last":0.96399999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.999,"15min":1.0049999999999999},"min":{"1min":0.72899999999999998,"5min":0.70899999999999996,"15min":0.626},"max":{"1min":1.2070000000000001,"5min":1.4810000000000001,"15min":1.613},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.022,"15min":1.0149999999999999},"min":{"1min":0.83899999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.282,"5min":1.548,"15min":1.698},"last":1.002}]},{"osd":7,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0009999999999999,"15min":1.008},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.62},"max":{"1min":1.2849999999999999,"5min":1.5169999999999999,"15min":2.3999999999999999},"last":1.032},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0700000000000001,"15min":1.0469999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.59299999999999997},"max":{"1min":1.4059999999999999,"5min":1.546,"15min":1.7689999999999999},"last":1.181}]}]},{"osd":0,"up_from":14,"seq":60129542223,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1785856,"data_stored":1124962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63700000000000001,"15min":0.66500000000000004},"min":{"1min":0.47399999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.93500000000000005,"5min":1.0600000000000001,"15min":1.1100000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.442,"5min":0.42399999999999999,"15min":0.377},"max":{"1min":1.077,"5min":1.157,"15min":1.157},"last":0.442}]},{"osd":2,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.503,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0109999999999999,"5min":1.202,"15min":1.202},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.69199999999999995},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96299999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.74199999999999999,"15min":0.66800000000000004},"min":{"1min":0.54700000000000004,"5min":0.39200000000000002,"15min":0.312},"max":{"1min":1.01,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.875,"15min":0.83499999999999996},"min":{"1min":0.73899999999999999,"5min":0.45300000000000001,"15min":0.42899999999999999},"max":{"1min":1.073,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.86199999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95899999999999996},"min":{"1min":0.71799999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.321,"5min":1.486,"15min":1.486},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.97199999999999998,"15min":0.95899999999999996},"min":{"1min":0.68000000000000005,"5min":0.64600000000000002,"15min":0.59999999999999998},"max":{"1min":1.165,"5min":1.704,"15min":1.704},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.0169999999999999,"15min":0.98199999999999998},"min":{"1min":0.72799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.23,"5min":1.575,"15min":1.575},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.97699999999999998,"15min":0.99299999999999999},"min":{"1min":0.69299999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.5029999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.69299999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1,"15min":0.97999999999999998},"min":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.628},"max":{"1min":1.2629999999999999,"5min":1.944,"15min":1.944},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.089,"15min":1.0580000000000001},"min":{"1min":0.78400000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.363,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.97599999999999998}]},{"osd":7,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.077,"15min":1.054},"min":{"1min":0.747,"5min":0.747,"15min":0.61699999999999999},"max":{"1min":1.4059999999999999,"5min":1.659,"15min":1.659},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1160000000000001,"15min":1.056},"min":{"1min":0.88900000000000001,"5min":0.76700000000000002,"15min":0.65400000000000003},"max":{"1min":1.472,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.88900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:31:33.041 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:31:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.149+0000 7efd62125700 1 Processor -- start 2024-09-13T16:31:33.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.149+0000 7efd62125700 1 -- start start 2024-09-13T16:31:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.149+0000 7efd62125700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd5c0cd8d0 0x7efd5c06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.149+0000 7efd62125700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0b2200 0x7efd5c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.149+0000 7efd62125700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd5c0705d0 con 0x7efd5c0b17f0 2024-09-13T16:31:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.149+0000 7efd62125700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd5c0cbf10 con 0x7efd5c0cd8d0 2024-09-13T16:31:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.149+0000 7efd62125700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0b2bb0 con 0x7efd5c0b2200 2024-09-13T16:31:33.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd5c0cd8d0 0x7efd5c06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd5c0cd8d0 0x7efd5c06cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38220/0 (socket says 172.21.15.119:38220) 2024-09-13T16:31:33.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 -- 172.21.15.119:0/877779556 learned_addr learned my addr 172.21.15.119:0/877779556 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:33.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61924700 1 --2- 172.21.15.119:0/877779556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0b2200 0x7efd5c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 -- 172.21.15.119:0/877779556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0b2200 msgr2=0x7efd5c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 --2- 172.21.15.119:0/877779556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0b2200 0x7efd5c0b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 -- 172.21.15.119:0/877779556 >> v1:172.21.15.145:6789/0 conn(0x7efd5c0b17f0 legacy=0x7efd5c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 -- 172.21.15.119:0/877779556 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd5c06d0d0 con 0x7efd5c0cd8d0 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61924700 1 --2- 172.21.15.119:0/877779556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0b2200 0x7efd5c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd61123700 1 --2- 172.21.15.119:0/877779556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0cd8d0 0x7efd5c06cb90 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7efd50009940 tx=0x7efd50009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d87f54a7bd463c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd5bfff700 1 -- 172.21.15.119:0/877779556 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5000ba20 con 0x7efd5c0cd8d0 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd5bfff700 1 -- 172.21.15.119:0/877779556 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd50009e90 con 0x7efd5c0cd8d0 2024-09-13T16:31:33.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd5bfff700 1 -- 172.21.15.119:0/877779556 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd5000bda0 con 0x7efd5c0cd8d0 2024-09-13T16:31:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 -- 172.21.15.119:0/877779556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0cd8d0 msgr2=0x7efd5c06cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 --2- 172.21.15.119:0/877779556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0cd8d0 0x7efd5c06cb90 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 crypto rx=0x7efd50009940 tx=0x7efd50009c50 comp rx=0 tx=0).stop 2024-09-13T16:31:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 -- 172.21.15.119:0/877779556 shutdown_connections 2024-09-13T16:31:33.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 --2- 172.21.15.119:0/877779556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0b2200 0x7efd5c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 --2- 172.21.15.119:0/877779556 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0cd8d0 0x7efd5c06cb90 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 -- 172.21.15.119:0/877779556 >> 172.21.15.119:0/877779556 conn(0x7efd5c06a350 msgr2=0x7efd5c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 -- 172.21.15.119:0/877779556 shutdown_connections 2024-09-13T16:31:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.153+0000 7efd62125700 1 -- 172.21.15.119:0/877779556 wait complete. 2024-09-13T16:31:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 Processor -- start 2024-09-13T16:31:33.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 -- start start 2024-09-13T16:31:33.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0b17f0 0x7efd5c0a9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5c0b2200 0x7efd5c0af920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0cd8d0 0x7efd5c1526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0afe60 con 0x7efd5c0b17f0 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0a98d0 con 0x7efd5c0b2200 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd5c0a9bb0 con 0x7efd5c0cd8d0 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0b17f0 0x7efd5c0a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0b17f0 0x7efd5c0a9240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38222/0 (socket says 172.21.15.119:38222) 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 -- 172.21.15.119:0/3112920586 learned_addr learned my addr 172.21.15.119:0/3112920586 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61924700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0cd8d0 0x7efd5c1526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd60922700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5c0b2200 0x7efd5c0af920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 -- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0cd8d0 msgr2=0x7efd5c1526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0cd8d0 0x7efd5c1526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 -- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5c0b2200 msgr2=0x7efd5c0af920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5c0b2200 0x7efd5c0af920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 -- 172.21.15.119:0/3112920586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd50009570 con 0x7efd5c0b17f0 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61924700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0cd8d0 0x7efd5c1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd60922700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5c0b2200 0x7efd5c0af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd61123700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0b17f0 0x7efd5c0a9240 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7efd50009890 tx=0x7efd50004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd59ffb700 1 -- 172.21.15.119:0/3112920586 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd50004610 con 0x7efd5c0b17f0 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd59ffb700 1 -- 172.21.15.119:0/3112920586 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd5001e070 con 0x7efd5c0b17f0 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd5c0a9e90 con 0x7efd5c0b17f0 2024-09-13T16:31:33.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd59ffb700 1 -- 172.21.15.119:0/3112920586 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd500038e0 con 0x7efd5c0b17f0 2024-09-13T16:31:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.157+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd5c0aa450 con 0x7efd5c0b17f0 2024-09-13T16:31:33.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.161+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd5c0cdcb0 con 0x7efd5c0b17f0 2024-09-13T16:31:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.161+0000 7efd59ffb700 1 -- 172.21.15.119:0/3112920586 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efd50003a40 con 0x7efd5c0b17f0 2024-09-13T16:31:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.161+0000 7efd59ffb700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd48066730 0x7efd48068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.161+0000 7efd59ffb700 1 -- 172.21.15.119:0/3112920586 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efd50020070 con 0x7efd5c0b17f0 2024-09-13T16:31:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.161+0000 7efd60922700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd48066730 0x7efd48068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.161+0000 7efd60922700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd48066730 0x7efd48068bf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0cb280 tx=0x7efd4c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.169+0000 7efd59ffb700 1 -- 172.21.15.119:0/3112920586 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efd50052a10 con 0x7efd5c0b17f0 2024-09-13T16:31:33.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.333+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efd5c070260 con 0x7efd48066730 2024-09-13T16:31:33.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.349+0000 7efd59ffb700 1 -- 172.21.15.119:0/3112920586 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288123 (secure 0 0 0) 0x7efd4c00db50 con 0x7efd48066730 2024-09-13T16:31:33.358 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:33.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd48066730 msgr2=0x7efd48068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd48066730 0x7efd48068bf0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7efd5c0cb280 tx=0x7efd4c009040 comp rx=0 tx=0).stop 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0b17f0 msgr2=0x7efd5c0a9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0b17f0 0x7efd5c0a9240 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 crypto rx=0x7efd50009890 tx=0x7efd50004770 comp rx=0 tx=0).stop 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 shutdown_connections 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd48066730 0x7efd48068bf0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd5c0cd8d0 0x7efd5c1526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd5c0b2200 0x7efd5c0af920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 --2- 172.21.15.119:0/3112920586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd5c0b17f0 0x7efd5c0a9240 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 >> 172.21.15.119:0/3112920586 conn(0x7efd5c06a350 msgr2=0x7efd5c0ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 shutdown_connections 2024-09-13T16:31:33.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.357+0000 7efd62125700 1 -- 172.21.15.119:0/3112920586 wait complete. 2024-09-13T16:31:33.362 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:31:33.374 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":279,"stamp":"2024-09-13T16:31:31.552134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":169,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":246956,"kb_used_data":10272,"kb_used_omap":12,"kb_used_meta":236531,"kb_avail":754727764,"statfs":{"total":773094113280,"available":772841230336,"internally_reserved":0,"allocated":10518528,"data_stored":5404332,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":242208313},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001318"},"pg_stats":[{"pgid":"14.3f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987612+0000","last_change":"2024-09-13T16:31:28.986864+0000","last_active":"2024-09-13T16:31:28.987612+0000","last_peered":"2024-09-13T16:31:28.987612+0000","last_clean":"2024-09-13T16:31:28.987612+0000","last_became_active":"2024-09-13T16:31:28.986630+0000","last_became_peered":"2024-09-13T16:31:28.986630+0000","last_unstale":"2024-09-13T16:31:28.987612+0000","last_undegraded":"2024-09-13T16:31:28.987612+0000","last_fullsized":"2024-09-13T16:31:28.987612+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:31.210815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3e","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976840+0000","last_change":"2024-09-13T16:31:28.989436+0000","last_active":"2024-09-13T16:31:29.976840+0000","last_peered":"2024-09-13T16:31:29.976840+0000","last_clean":"2024-09-13T16:31:29.976840+0000","last_became_active":"2024-09-13T16:31:28.989194+0000","last_became_peered":"2024-09-13T16:31:28.989194+0000","last_unstale":"2024-09-13T16:31:29.976840+0000","last_undegraded":"2024-09-13T16:31:29.976840+0000","last_fullsized":"2024-09-13T16:31:29.976840+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:21:40.385225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.3d","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987569+0000","last_change":"2024-09-13T16:31:28.986161+0000","last_active":"2024-09-13T16:31:28.987569+0000","last_peered":"2024-09-13T16:31:28.987569+0000","last_clean":"2024-09-13T16:31:28.987569+0000","last_became_active":"2024-09-13T16:31:28.985946+0000","last_became_peered":"2024-09-13T16:31:28.985946+0000","last_unstale":"2024-09-13T16:31:28.987569+0000","last_undegraded":"2024-09-13T16:31:28.987569+0000","last_fullsized":"2024-09-13T16:31:28.987569+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:19:27.985372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.3c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.005380+0000","last_change":"2024-09-13T16:31:28.999374+0000","last_active":"2024-09-13T16:31:29.005380+0000","last_peered":"2024-09-13T16:31:29.005380+0000","last_clean":"2024-09-13T16:31:29.005380+0000","last_became_active":"2024-09-13T16:31:28.999123+0000","last_became_peered":"2024-09-13T16:31:28.999123+0000","last_unstale":"2024-09-13T16:31:29.005380+0000","last_undegraded":"2024-09-13T16:31:29.005380+0000","last_fullsized":"2024-09-13T16:31:29.005380+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:40:51.659631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.3b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.994847+0000","last_change":"2024-09-13T16:31:28.993117+0000","last_active":"2024-09-13T16:31:28.994847+0000","last_peered":"2024-09-13T16:31:28.994847+0000","last_clean":"2024-09-13T16:31:28.994847+0000","last_became_active":"2024-09-13T16:31:28.991936+0000","last_became_peered":"2024-09-13T16:31:28.991936+0000","last_unstale":"2024-09-13T16:31:28.994847+0000","last_undegraded":"2024-09-13T16:31:28.994847+0000","last_fullsized":"2024-09-13T16:31:28.994847+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:55:44.707877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.3a","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973881+0000","last_change":"2024-09-13T16:31:28.993865+0000","last_active":"2024-09-13T16:31:29.973881+0000","last_peered":"2024-09-13T16:31:29.973881+0000","last_clean":"2024-09-13T16:31:29.973881+0000","last_became_active":"2024-09-13T16:31:28.993620+0000","last_became_peered":"2024-09-13T16:31:28.993620+0000","last_unstale":"2024-09-13T16:31:29.973881+0000","last_undegraded":"2024-09-13T16:31:29.973881+0000","last_fullsized":"2024-09-13T16:31:29.973881+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:35:23.952163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.39","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987337+0000","last_change":"2024-09-13T16:31:28.986192+0000","last_active":"2024-09-13T16:31:28.987337+0000","last_peered":"2024-09-13T16:31:28.987337+0000","last_clean":"2024-09-13T16:31:28.987337+0000","last_became_active":"2024-09-13T16:31:28.985940+0000","last_became_peered":"2024-09-13T16:31:28.985940+0000","last_unstale":"2024-09-13T16:31:28.987337+0000","last_undegraded":"2024-09-13T16:31:28.987337+0000","last_fullsized":"2024-09-13T16:31:28.987337+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:20.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.38","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000091+0000","last_change":"2024-09-13T16:31:28.995964+0000","last_active":"2024-09-13T16:31:29.000091+0000","last_peered":"2024-09-13T16:31:29.000091+0000","last_clean":"2024-09-13T16:31:29.000091+0000","last_became_active":"2024-09-13T16:31:28.995655+0000","last_became_peered":"2024-09-13T16:31:28.995655+0000","last_unstale":"2024-09-13T16:31:29.000091+0000","last_undegraded":"2024-09-13T16:31:29.000091+0000","last_fullsized":"2024-09-13T16:31:29.000091+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:12:42.752127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.37","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.986935+0000","last_change":"2024-09-13T16:31:28.984913+0000","last_active":"2024-09-13T16:31:28.986935+0000","last_peered":"2024-09-13T16:31:28.986935+0000","last_clean":"2024-09-13T16:31:28.986935+0000","last_became_active":"2024-09-13T16:31:28.984652+0000","last_became_peered":"2024-09-13T16:31:28.984652+0000","last_unstale":"2024-09-13T16:31:28.986935+0000","last_undegraded":"2024-09-13T16:31:28.986935+0000","last_fullsized":"2024-09-13T16:31:28.986935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:13:28.189762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.36","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999494+0000","last_change":"2024-09-13T16:31:28.997778+0000","last_active":"2024-09-13T16:31:28.999494+0000","last_peered":"2024-09-13T16:31:28.999494+0000","last_clean":"2024-09-13T16:31:28.999494+0000","last_became_active":"2024-09-13T16:31:28.997478+0000","last_became_peered":"2024-09-13T16:31:28.997478+0000","last_unstale":"2024-09-13T16:31:28.999494+0000","last_undegraded":"2024-09-13T16:31:28.999494+0000","last_fullsized":"2024-09-13T16:31:28.999494+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:52:39.125689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.35","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.985206+0000","last_change":"2024-09-13T16:31:28.981973+0000","last_active":"2024-09-13T16:31:28.985206+0000","last_peered":"2024-09-13T16:31:28.985206+0000","last_clean":"2024-09-13T16:31:28.985206+0000","last_became_active":"2024-09-13T16:31:28.981649+0000","last_became_peered":"2024-09-13T16:31:28.981649+0000","last_unstale":"2024-09-13T16:31:28.985206+0000","last_undegraded":"2024-09-13T16:31:28.985206+0000","last_fullsized":"2024-09-13T16:31:28.985206+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:35.100645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.34","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.992588+0000","last_change":"2024-09-13T16:31:28.988272+0000","last_active":"2024-09-13T16:31:28.992588+0000","last_peered":"2024-09-13T16:31:28.992588+0000","last_clean":"2024-09-13T16:31:28.992588+0000","last_became_active":"2024-09-13T16:31:28.988033+0000","last_became_peered":"2024-09-13T16:31:28.988033+0000","last_unstale":"2024-09-13T16:31:28.992588+0000","last_undegraded":"2024-09-13T16:31:28.992588+0000","last_fullsized":"2024-09-13T16:31:28.992588+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:15:13.778271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.33","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999613+0000","last_change":"2024-09-13T16:31:28.993596+0000","last_active":"2024-09-13T16:31:28.999613+0000","last_peered":"2024-09-13T16:31:28.999613+0000","last_clean":"2024-09-13T16:31:28.999613+0000","last_became_active":"2024-09-13T16:31:28.993346+0000","last_became_peered":"2024-09-13T16:31:28.993346+0000","last_unstale":"2024-09-13T16:31:28.999613+0000","last_undegraded":"2024-09-13T16:31:28.999613+0000","last_fullsized":"2024-09-13T16:31:28.999613+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:06:08.055096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.32","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.996696+0000","last_change":"2024-09-13T16:31:28.994048+0000","last_active":"2024-09-13T16:31:28.996696+0000","last_peered":"2024-09-13T16:31:28.996696+0000","last_clean":"2024-09-13T16:31:28.996696+0000","last_became_active":"2024-09-13T16:31:28.993752+0000","last_became_peered":"2024-09-13T16:31:28.993752+0000","last_unstale":"2024-09-13T16:31:28.996696+0000","last_undegraded":"2024-09-13T16:31:28.996696+0000","last_fullsized":"2024-09-13T16:31:28.996696+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:19.943107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.31","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.998744+0000","last_change":"2024-09-13T16:31:28.984539+0000","last_active":"2024-09-13T16:31:28.998744+0000","last_peered":"2024-09-13T16:31:28.998744+0000","last_clean":"2024-09-13T16:31:28.998744+0000","last_became_active":"2024-09-13T16:31:28.984301+0000","last_became_peered":"2024-09-13T16:31:28.984301+0000","last_unstale":"2024-09-13T16:31:28.998744+0000","last_undegraded":"2024-09-13T16:31:28.998744+0000","last_fullsized":"2024-09-13T16:31:28.998744+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:04.807834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.30","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973652+0000","last_change":"2024-09-13T16:31:28.989144+0000","last_active":"2024-09-13T16:31:29.973652+0000","last_peered":"2024-09-13T16:31:29.973652+0000","last_clean":"2024-09-13T16:31:29.973652+0000","last_became_active":"2024-09-13T16:31:28.988953+0000","last_became_peered":"2024-09-13T16:31:28.988953+0000","last_unstale":"2024-09-13T16:31:29.973652+0000","last_undegraded":"2024-09-13T16:31:29.973652+0000","last_fullsized":"2024-09-13T16:31:29.973652+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:46:14.307773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2f","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974938+0000","last_change":"2024-09-13T16:31:28.987501+0000","last_active":"2024-09-13T16:31:29.974938+0000","last_peered":"2024-09-13T16:31:29.974938+0000","last_clean":"2024-09-13T16:31:29.974938+0000","last_became_active":"2024-09-13T16:31:28.987252+0000","last_became_peered":"2024-09-13T16:31:28.987252+0000","last_unstale":"2024-09-13T16:31:29.974938+0000","last_undegraded":"2024-09-13T16:31:29.974938+0000","last_fullsized":"2024-09-13T16:31:29.974938+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:25:15.223155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.2e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.997060+0000","last_change":"2024-09-13T16:31:28.990706+0000","last_active":"2024-09-13T16:31:28.997060+0000","last_peered":"2024-09-13T16:31:28.997060+0000","last_clean":"2024-09-13T16:31:28.997060+0000","last_became_active":"2024-09-13T16:31:28.989799+0000","last_became_peered":"2024-09-13T16:31:28.989799+0000","last_unstale":"2024-09-13T16:31:28.997060+0000","last_undegraded":"2024-09-13T16:31:28.997060+0000","last_fullsized":"2024-09-13T16:31:28.997060+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:13:05.163905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.2d","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.994831+0000","last_change":"2024-09-13T16:31:28.993492+0000","last_active":"2024-09-13T16:31:28.994831+0000","last_peered":"2024-09-13T16:31:28.994831+0000","last_clean":"2024-09-13T16:31:28.994831+0000","last_became_active":"2024-09-13T16:31:28.993209+0000","last_became_peered":"2024-09-13T16:31:28.993209+0000","last_unstale":"2024-09-13T16:31:28.994831+0000","last_undegraded":"2024-09-13T16:31:28.994831+0000","last_fullsized":"2024-09-13T16:31:28.994831+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:12.341081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2c","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999581+0000","last_change":"2024-09-13T16:31:28.993482+0000","last_active":"2024-09-13T16:31:28.999581+0000","last_peered":"2024-09-13T16:31:28.999581+0000","last_clean":"2024-09-13T16:31:28.999581+0000","last_became_active":"2024-09-13T16:31:28.993210+0000","last_became_peered":"2024-09-13T16:31:28.993210+0000","last_unstale":"2024-09-13T16:31:28.999581+0000","last_undegraded":"2024-09-13T16:31:28.999581+0000","last_fullsized":"2024-09-13T16:31:28.999581+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:01.604548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.2b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.982916+0000","last_change":"2024-09-13T16:31:28.980061+0000","last_active":"2024-09-13T16:31:28.982916+0000","last_peered":"2024-09-13T16:31:28.982916+0000","last_clean":"2024-09-13T16:31:28.982916+0000","last_became_active":"2024-09-13T16:31:28.979737+0000","last_became_peered":"2024-09-13T16:31:28.979737+0000","last_unstale":"2024-09-13T16:31:28.982916+0000","last_undegraded":"2024-09-13T16:31:28.982916+0000","last_fullsized":"2024-09-13T16:31:28.982916+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:56:54.464831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2a","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987835+0000","last_change":"2024-09-13T16:31:28.986415+0000","last_active":"2024-09-13T16:31:28.987835+0000","last_peered":"2024-09-13T16:31:28.987835+0000","last_clean":"2024-09-13T16:31:28.987835+0000","last_became_active":"2024-09-13T16:31:28.986078+0000","last_became_peered":"2024-09-13T16:31:28.986078+0000","last_unstale":"2024-09-13T16:31:28.987835+0000","last_undegraded":"2024-09-13T16:31:28.987835+0000","last_fullsized":"2024-09-13T16:31:28.987835+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:52.726980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.29","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975144+0000","last_change":"2024-09-13T16:31:28.998567+0000","last_active":"2024-09-13T16:31:29.975144+0000","last_peered":"2024-09-13T16:31:29.975144+0000","last_clean":"2024-09-13T16:31:29.975144+0000","last_became_active":"2024-09-13T16:31:28.996499+0000","last_became_peered":"2024-09-13T16:31:28.996499+0000","last_unstale":"2024-09-13T16:31:29.975144+0000","last_undegraded":"2024-09-13T16:31:29.975144+0000","last_fullsized":"2024-09-13T16:31:29.975144+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:06.203951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.28","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.993116+0000","last_change":"2024-09-13T16:31:28.989201+0000","last_active":"2024-09-13T16:31:28.993116+0000","last_peered":"2024-09-13T16:31:28.993116+0000","last_clean":"2024-09-13T16:31:28.993116+0000","last_became_active":"2024-09-13T16:31:28.988747+0000","last_became_peered":"2024-09-13T16:31:28.988747+0000","last_unstale":"2024-09-13T16:31:28.993116+0000","last_undegraded":"2024-09-13T16:31:28.993116+0000","last_fullsized":"2024-09-13T16:31:28.993116+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:12:49.183526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.27","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.977109+0000","last_change":"2024-09-13T16:31:28.999547+0000","last_active":"2024-09-13T16:31:29.977109+0000","last_peered":"2024-09-13T16:31:29.977109+0000","last_clean":"2024-09-13T16:31:29.977109+0000","last_became_active":"2024-09-13T16:31:28.999267+0000","last_became_peered":"2024-09-13T16:31:28.999267+0000","last_unstale":"2024-09-13T16:31:29.977109+0000","last_undegraded":"2024-09-13T16:31:29.977109+0000","last_fullsized":"2024-09-13T16:31:29.977109+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:19:53.477490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.26","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.986984+0000","last_change":"2024-09-13T16:31:28.984405+0000","last_active":"2024-09-13T16:31:28.986984+0000","last_peered":"2024-09-13T16:31:28.986984+0000","last_clean":"2024-09-13T16:31:28.986984+0000","last_became_active":"2024-09-13T16:31:28.984111+0000","last_became_peered":"2024-09-13T16:31:28.984111+0000","last_unstale":"2024-09-13T16:31:28.986984+0000","last_undegraded":"2024-09-13T16:31:28.986984+0000","last_fullsized":"2024-09-13T16:31:28.986984+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:13:20.502358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.25","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.995295+0000","last_change":"2024-09-13T16:31:28.992166+0000","last_active":"2024-09-13T16:31:28.995295+0000","last_peered":"2024-09-13T16:31:28.995295+0000","last_clean":"2024-09-13T16:31:28.995295+0000","last_became_active":"2024-09-13T16:31:28.991971+0000","last_became_peered":"2024-09-13T16:31:28.991971+0000","last_unstale":"2024-09-13T16:31:28.995295+0000","last_undegraded":"2024-09-13T16:31:28.995295+0000","last_fullsized":"2024-09-13T16:31:28.995295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:06:55.182947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.24","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999893+0000","last_change":"2024-09-13T16:31:28.994159+0000","last_active":"2024-09-13T16:31:28.999893+0000","last_peered":"2024-09-13T16:31:28.999893+0000","last_clean":"2024-09-13T16:31:28.999893+0000","last_became_active":"2024-09-13T16:31:28.993898+0000","last_became_peered":"2024-09-13T16:31:28.993898+0000","last_unstale":"2024-09-13T16:31:28.999893+0000","last_undegraded":"2024-09-13T16:31:28.999893+0000","last_fullsized":"2024-09-13T16:31:28.999893+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:45.243501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.23","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.983360+0000","last_change":"2024-09-13T16:31:28.981209+0000","last_active":"2024-09-13T16:31:28.983360+0000","last_peered":"2024-09-13T16:31:28.983360+0000","last_clean":"2024-09-13T16:31:28.983360+0000","last_became_active":"2024-09-13T16:31:28.980954+0000","last_became_peered":"2024-09-13T16:31:28.980954+0000","last_unstale":"2024-09-13T16:31:28.983360+0000","last_undegraded":"2024-09-13T16:31:28.983360+0000","last_fullsized":"2024-09-13T16:31:28.983360+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:40.162978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.22","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987026+0000","last_change":"2024-09-13T16:31:28.985248+0000","last_active":"2024-09-13T16:31:28.987026+0000","last_peered":"2024-09-13T16:31:28.987026+0000","last_clean":"2024-09-13T16:31:28.987026+0000","last_became_active":"2024-09-13T16:31:28.984624+0000","last_became_peered":"2024-09-13T16:31:28.984624+0000","last_unstale":"2024-09-13T16:31:28.987026+0000","last_undegraded":"2024-09-13T16:31:28.987026+0000","last_fullsized":"2024-09-13T16:31:28.987026+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:54.455329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.21","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976178+0000","last_change":"2024-09-13T16:31:28.993472+0000","last_active":"2024-09-13T16:31:29.976178+0000","last_peered":"2024-09-13T16:31:29.976178+0000","last_clean":"2024-09-13T16:31:29.976178+0000","last_became_active":"2024-09-13T16:31:28.993233+0000","last_became_peered":"2024-09-13T16:31:28.993233+0000","last_unstale":"2024-09-13T16:31:29.976178+0000","last_undegraded":"2024-09-13T16:31:29.976178+0000","last_fullsized":"2024-09-13T16:31:29.976178+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:20:10.870778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.20","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973576+0000","last_change":"2024-09-13T16:31:28.986953+0000","last_active":"2024-09-13T16:31:29.973576+0000","last_peered":"2024-09-13T16:31:29.973576+0000","last_clean":"2024-09-13T16:31:29.973576+0000","last_became_active":"2024-09-13T16:31:28.986715+0000","last_became_peered":"2024-09-13T16:31:28.986715+0000","last_unstale":"2024-09-13T16:31:29.973576+0000","last_undegraded":"2024-09-13T16:31:29.973576+0000","last_fullsized":"2024-09-13T16:31:29.973576+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:11:41.605098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1c","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974122+0000","last_change":"2024-09-13T16:31:28.977336+0000","last_active":"2024-09-13T16:31:29.974122+0000","last_peered":"2024-09-13T16:31:29.974122+0000","last_clean":"2024-09-13T16:31:29.974122+0000","last_became_active":"2024-09-13T16:31:28.976843+0000","last_became_peered":"2024-09-13T16:31:28.976843+0000","last_unstale":"2024-09-13T16:31:29.974122+0000","last_undegraded":"2024-09-13T16:31:29.974122+0000","last_fullsized":"2024-09-13T16:31:29.974122+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:24:22.340211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1f","version":"94'4","reported_seq":15,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975646+0000","last_change":"2024-09-13T16:31:28.611651+0000","last_active":"2024-09-13T16:31:29.975646+0000","last_peered":"2024-09-13T16:31:29.975646+0000","last_clean":"2024-09-13T16:31:29.975646+0000","last_became_active":"2024-09-13T16:31:28.611310+0000","last_became_peered":"2024-09-13T16:31:28.611310+0000","last_unstale":"2024-09-13T16:31:29.975646+0000","last_undegraded":"2024-09-13T16:31:29.975646+0000","last_fullsized":"2024-09-13T16:31:29.975646+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:20.209081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.1d","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974478+0000","last_change":"2024-09-13T16:31:28.983292+0000","last_active":"2024-09-13T16:31:29.974478+0000","last_peered":"2024-09-13T16:31:29.974478+0000","last_clean":"2024-09-13T16:31:29.974478+0000","last_became_active":"2024-09-13T16:31:28.983053+0000","last_became_peered":"2024-09-13T16:31:28.983053+0000","last_unstale":"2024-09-13T16:31:29.974478+0000","last_undegraded":"2024-09-13T16:31:29.974478+0000","last_fullsized":"2024-09-13T16:31:29.974478+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:27:03.519510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1e","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.001156+0000","last_change":"2024-09-13T16:31:28.604634+0000","last_active":"2024-09-13T16:31:29.001156+0000","last_peered":"2024-09-13T16:31:29.001156+0000","last_clean":"2024-09-13T16:31:29.001156+0000","last_became_active":"2024-09-13T16:31:28.603979+0000","last_became_peered":"2024-09-13T16:31:28.603979+0000","last_unstale":"2024-09-13T16:31:29.001156+0000","last_undegraded":"2024-09-13T16:31:29.001156+0000","last_fullsized":"2024-09-13T16:31:29.001156+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:52.354399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.996927+0000","last_change":"2024-09-13T16:31:28.995090+0000","last_active":"2024-09-13T16:31:28.996927+0000","last_peered":"2024-09-13T16:31:28.996927+0000","last_clean":"2024-09-13T16:31:28.996927+0000","last_became_active":"2024-09-13T16:31:28.994764+0000","last_became_peered":"2024-09-13T16:31:28.994764+0000","last_unstale":"2024-09-13T16:31:28.996927+0000","last_undegraded":"2024-09-13T16:31:28.996927+0000","last_fullsized":"2024-09-13T16:31:28.996927+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:00:19.943107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.1d","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.993964+0000","last_change":"2024-09-13T16:31:28.609486+0000","last_active":"2024-09-13T16:31:28.993964+0000","last_peered":"2024-09-13T16:31:28.993964+0000","last_clean":"2024-09-13T16:31:28.993964+0000","last_became_active":"2024-09-13T16:31:28.609063+0000","last_became_peered":"2024-09-13T16:31:28.609063+0000","last_unstale":"2024-09-13T16:31:28.993964+0000","last_undegraded":"2024-09-13T16:31:28.993964+0000","last_fullsized":"2024-09-13T16:31:28.993964+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:44:36.879842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1f","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975116+0000","last_change":"2024-09-13T16:31:28.997207+0000","last_active":"2024-09-13T16:31:29.975116+0000","last_peered":"2024-09-13T16:31:29.975116+0000","last_clean":"2024-09-13T16:31:29.975116+0000","last_became_active":"2024-09-13T16:31:28.996929+0000","last_became_peered":"2024-09-13T16:31:28.996929+0000","last_unstale":"2024-09-13T16:31:29.975116+0000","last_undegraded":"2024-09-13T16:31:29.975116+0000","last_fullsized":"2024-09-13T16:31:29.975116+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:25.710493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1c","version":"94'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975472+0000","last_change":"2024-09-13T16:31:28.607386+0000","last_active":"2024-09-13T16:31:29.975472+0000","last_peered":"2024-09-13T16:31:29.975472+0000","last_clean":"2024-09-13T16:31:29.975472+0000","last_became_active":"2024-09-13T16:31:28.607100+0000","last_became_peered":"2024-09-13T16:31:28.607100+0000","last_unstale":"2024-09-13T16:31:29.975472+0000","last_undegraded":"2024-09-13T16:31:29.975472+0000","last_fullsized":"2024-09-13T16:31:29.975472+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:32:01.296974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000110+0000","last_change":"2024-09-13T16:31:28.996128+0000","last_active":"2024-09-13T16:31:29.000110+0000","last_peered":"2024-09-13T16:31:29.000110+0000","last_clean":"2024-09-13T16:31:29.000110+0000","last_became_active":"2024-09-13T16:31:28.995707+0000","last_became_peered":"2024-09-13T16:31:28.995707+0000","last_unstale":"2024-09-13T16:31:29.000110+0000","last_undegraded":"2024-09-13T16:31:29.000110+0000","last_fullsized":"2024-09-13T16:31:29.000110+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:30.072934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.1b","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.995315+0000","last_change":"2024-09-13T16:31:28.607186+0000","last_active":"2024-09-13T16:31:28.995315+0000","last_peered":"2024-09-13T16:31:28.995315+0000","last_clean":"2024-09-13T16:31:28.995315+0000","last_became_active":"2024-09-13T16:31:28.606835+0000","last_became_peered":"2024-09-13T16:31:28.606835+0000","last_unstale":"2024-09-13T16:31:28.995315+0000","last_undegraded":"2024-09-13T16:31:28.995315+0000","last_fullsized":"2024-09-13T16:31:28.995315+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:49.477532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975134+0000","last_change":"2024-09-13T16:31:28.996481+0000","last_active":"2024-09-13T16:31:29.975134+0000","last_peered":"2024-09-13T16:31:29.975134+0000","last_clean":"2024-09-13T16:31:29.975134+0000","last_became_active":"2024-09-13T16:31:28.996104+0000","last_became_peered":"2024-09-13T16:31:28.996104+0000","last_unstale":"2024-09-13T16:31:29.975134+0000","last_undegraded":"2024-09-13T16:31:29.975134+0000","last_fullsized":"2024-09-13T16:31:29.975134+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:09.621978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1a","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.998823+0000","last_change":"2024-09-13T16:31:28.609506+0000","last_active":"2024-09-13T16:31:28.998823+0000","last_peered":"2024-09-13T16:31:28.998823+0000","last_clean":"2024-09-13T16:31:28.998823+0000","last_became_active":"2024-09-13T16:31:28.609246+0000","last_became_peered":"2024-09-13T16:31:28.609246+0000","last_unstale":"2024-09-13T16:31:28.998823+0000","last_undegraded":"2024-09-13T16:31:28.998823+0000","last_fullsized":"2024-09-13T16:31:28.998823+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:30:17.805587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1a","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973834+0000","last_change":"2024-09-13T16:31:28.994339+0000","last_active":"2024-09-13T16:31:29.973834+0000","last_peered":"2024-09-13T16:31:29.973834+0000","last_clean":"2024-09-13T16:31:29.973834+0000","last_became_active":"2024-09-13T16:31:28.994105+0000","last_became_peered":"2024-09-13T16:31:28.994105+0000","last_unstale":"2024-09-13T16:31:29.973834+0000","last_undegraded":"2024-09-13T16:31:29.973834+0000","last_fullsized":"2024-09-13T16:31:29.973834+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:37:18.769898+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.978582+0000","last_change":"2024-09-13T16:31:28.600186+0000","last_active":"2024-09-13T16:31:28.978582+0000","last_peered":"2024-09-13T16:31:28.978582+0000","last_clean":"2024-09-13T16:31:28.978582+0000","last_became_active":"2024-09-13T16:31:28.599926+0000","last_became_peered":"2024-09-13T16:31:28.599926+0000","last_unstale":"2024-09-13T16:31:28.978582+0000","last_undegraded":"2024-09-13T16:31:28.978582+0000","last_fullsized":"2024-09-13T16:31:28.978582+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:28.148770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1b","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000009+0000","last_change":"2024-09-13T16:31:28.996577+0000","last_active":"2024-09-13T16:31:29.000009+0000","last_peered":"2024-09-13T16:31:29.000009+0000","last_clean":"2024-09-13T16:31:29.000009+0000","last_became_active":"2024-09-13T16:31:28.996352+0000","last_became_peered":"2024-09-13T16:31:28.996352+0000","last_unstale":"2024-09-13T16:31:29.000009+0000","last_undegraded":"2024-09-13T16:31:29.000009+0000","last_fullsized":"2024-09-13T16:31:29.000009+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:43.147656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.977316+0000","last_change":"2024-09-13T16:31:28.608593+0000","last_active":"2024-09-13T16:31:28.977316+0000","last_peered":"2024-09-13T16:31:28.977316+0000","last_clean":"2024-09-13T16:31:28.977316+0000","last_became_active":"2024-09-13T16:31:28.608324+0000","last_became_peered":"2024-09-13T16:31:28.608324+0000","last_unstale":"2024-09-13T16:31:28.977316+0000","last_undegraded":"2024-09-13T16:31:28.977316+0000","last_fullsized":"2024-09-13T16:31:28.977316+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:23:29.062736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987035+0000","last_change":"2024-09-13T16:31:28.985889+0000","last_active":"2024-09-13T16:31:28.987035+0000","last_peered":"2024-09-13T16:31:28.987035+0000","last_clean":"2024-09-13T16:31:28.987035+0000","last_became_active":"2024-09-13T16:31:28.985610+0000","last_became_peered":"2024-09-13T16:31:28.985610+0000","last_unstale":"2024-09-13T16:31:28.987035+0000","last_undegraded":"2024-09-13T16:31:28.987035+0000","last_fullsized":"2024-09-13T16:31:28.987035+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:51:04.807834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.977200+0000","last_change":"2024-09-13T16:31:28.606601+0000","last_active":"2024-09-13T16:31:29.977200+0000","last_peered":"2024-09-13T16:31:29.977200+0000","last_clean":"2024-09-13T16:31:29.977200+0000","last_became_active":"2024-09-13T16:31:28.606330+0000","last_became_peered":"2024-09-13T16:31:28.606330+0000","last_unstale":"2024-09-13T16:31:29.977200+0000","last_undegraded":"2024-09-13T16:31:29.977200+0000","last_fullsized":"2024-09-13T16:31:29.977200+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:24:30.287705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.15","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.985874+0000","last_change":"2024-09-13T16:31:28.982778+0000","last_active":"2024-09-13T16:31:28.985874+0000","last_peered":"2024-09-13T16:31:28.985874+0000","last_clean":"2024-09-13T16:31:28.985874+0000","last_became_active":"2024-09-13T16:31:28.982486+0000","last_became_peered":"2024-09-13T16:31:28.982486+0000","last_unstale":"2024-09-13T16:31:28.985874+0000","last_undegraded":"2024-09-13T16:31:28.985874+0000","last_fullsized":"2024-09-13T16:31:28.985874+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:27:35.100645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.980217+0000","last_change":"2024-09-13T16:31:28.602408+0000","last_active":"2024-09-13T16:31:28.980217+0000","last_peered":"2024-09-13T16:31:28.980217+0000","last_clean":"2024-09-13T16:31:28.980217+0000","last_became_active":"2024-09-13T16:31:28.602164+0000","last_became_peered":"2024-09-13T16:31:28.602164+0000","last_unstale":"2024-09-13T16:31:28.980217+0000","last_undegraded":"2024-09-13T16:31:28.980217+0000","last_fullsized":"2024-09-13T16:31:28.980217+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:33.768254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.987220+0000","last_change":"2024-09-13T16:31:28.985748+0000","last_active":"2024-09-13T16:31:28.987220+0000","last_peered":"2024-09-13T16:31:28.987220+0000","last_clean":"2024-09-13T16:31:28.987220+0000","last_became_active":"2024-09-13T16:31:28.985545+0000","last_became_peered":"2024-09-13T16:31:28.985545+0000","last_unstale":"2024-09-13T16:31:28.987220+0000","last_undegraded":"2024-09-13T16:31:28.987220+0000","last_fullsized":"2024-09-13T16:31:28.987220+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:25:29.318165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.15","version":"94'3","reported_seq":13,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.997192+0000","last_change":"2024-09-13T16:31:28.608759+0000","last_active":"2024-09-13T16:31:28.997192+0000","last_peered":"2024-09-13T16:31:28.997192+0000","last_clean":"2024-09-13T16:31:28.997192+0000","last_became_active":"2024-09-13T16:31:28.608469+0000","last_became_peered":"2024-09-13T16:31:28.608469+0000","last_unstale":"2024-09-13T16:31:28.997192+0000","last_undegraded":"2024-09-13T16:31:28.997192+0000","last_fullsized":"2024-09-13T16:31:28.997192+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:58:42.720080+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999920+0000","last_change":"2024-09-13T16:31:28.994056+0000","last_active":"2024-09-13T16:31:28.999920+0000","last_peered":"2024-09-13T16:31:28.999920+0000","last_clean":"2024-09-13T16:31:28.999920+0000","last_became_active":"2024-09-13T16:31:28.993800+0000","last_became_peered":"2024-09-13T16:31:28.993800+0000","last_unstale":"2024-09-13T16:31:28.999920+0000","last_undegraded":"2024-09-13T16:31:28.999920+0000","last_fullsized":"2024-09-13T16:31:28.999920+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:17:54.020569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.14","version":"94'6","reported_seq":17,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974324+0000","last_change":"2024-09-13T16:31:28.611488+0000","last_active":"2024-09-13T16:31:29.974324+0000","last_peered":"2024-09-13T16:31:29.974324+0000","last_clean":"2024-09-13T16:31:29.974324+0000","last_became_active":"2024-09-13T16:31:28.611287+0000","last_became_peered":"2024-09-13T16:31:28.611287+0000","last_unstale":"2024-09-13T16:31:29.974324+0000","last_undegraded":"2024-09-13T16:31:29.974324+0000","last_fullsized":"2024-09-13T16:31:29.974324+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:36:14.999923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.10","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000127+0000","last_change":"2024-09-13T16:31:28.996368+0000","last_active":"2024-09-13T16:31:29.000127+0000","last_peered":"2024-09-13T16:31:29.000127+0000","last_clean":"2024-09-13T16:31:29.000127+0000","last_became_active":"2024-09-13T16:31:28.995902+0000","last_became_peered":"2024-09-13T16:31:28.995902+0000","last_unstale":"2024-09-13T16:31:29.000127+0000","last_undegraded":"2024-09-13T16:31:29.000127+0000","last_fullsized":"2024-09-13T16:31:29.000127+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:55:19.383061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.992983+0000","last_change":"2024-09-13T16:31:28.606615+0000","last_active":"2024-09-13T16:31:28.992983+0000","last_peered":"2024-09-13T16:31:28.992983+0000","last_clean":"2024-09-13T16:31:28.992983+0000","last_became_active":"2024-09-13T16:31:28.606351+0000","last_became_peered":"2024-09-13T16:31:28.606351+0000","last_unstale":"2024-09-13T16:31:28.992983+0000","last_undegraded":"2024-09-13T16:31:28.992983+0000","last_fullsized":"2024-09-13T16:31:28.992983+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:00.594349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.001351+0000","last_change":"2024-09-13T16:31:28.999890+0000","last_active":"2024-09-13T16:31:29.001351+0000","last_peered":"2024-09-13T16:31:29.001351+0000","last_clean":"2024-09-13T16:31:29.001351+0000","last_became_active":"2024-09-13T16:31:28.999619+0000","last_became_peered":"2024-09-13T16:31:28.999619+0000","last_unstale":"2024-09-13T16:31:29.001351+0000","last_undegraded":"2024-09-13T16:31:29.001351+0000","last_fullsized":"2024-09-13T16:31:29.001351+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:32.970089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973925+0000","last_change":"2024-09-13T16:31:28.610323+0000","last_active":"2024-09-13T16:31:29.973925+0000","last_peered":"2024-09-13T16:31:29.973925+0000","last_clean":"2024-09-13T16:31:29.973925+0000","last_became_active":"2024-09-13T16:31:28.610085+0000","last_became_peered":"2024-09-13T16:31:28.610085+0000","last_unstale":"2024-09-13T16:31:29.973925+0000","last_undegraded":"2024-09-13T16:31:29.973925+0000","last_fullsized":"2024-09-13T16:31:29.973925+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:19:50.439035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.998935+0000","last_change":"2024-09-13T16:31:28.997655+0000","last_active":"2024-09-13T16:31:28.998935+0000","last_peered":"2024-09-13T16:31:28.998935+0000","last_clean":"2024-09-13T16:31:28.998935+0000","last_became_active":"2024-09-13T16:31:28.997374+0000","last_became_peered":"2024-09-13T16:31:28.997374+0000","last_unstale":"2024-09-13T16:31:28.998935+0000","last_undegraded":"2024-09-13T16:31:28.998935+0000","last_fullsized":"2024-09-13T16:31:28.998935+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:49:47.873917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.975100+0000","last_change":"2024-09-13T16:31:28.607220+0000","last_active":"2024-09-13T16:31:28.975100+0000","last_peered":"2024-09-13T16:31:28.975100+0000","last_clean":"2024-09-13T16:31:28.975100+0000","last_became_active":"2024-09-13T16:31:28.606872+0000","last_became_peered":"2024-09-13T16:31:28.606872+0000","last_unstale":"2024-09-13T16:31:28.975100+0000","last_undegraded":"2024-09-13T16:31:28.975100+0000","last_fullsized":"2024-09-13T16:31:28.975100+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:50:40.303706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.13","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.983815+0000","last_change":"2024-09-13T16:31:28.980615+0000","last_active":"2024-09-13T16:31:28.983815+0000","last_peered":"2024-09-13T16:31:28.983815+0000","last_clean":"2024-09-13T16:31:28.983815+0000","last_became_active":"2024-09-13T16:31:28.980246+0000","last_became_peered":"2024-09-13T16:31:28.980246+0000","last_unstale":"2024-09-13T16:31:28.983815+0000","last_undegraded":"2024-09-13T16:31:28.983815+0000","last_fullsized":"2024-09-13T16:31:28.983815+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:59:45.871680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.10","version":"94'8","reported_seq":18,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999334+0000","last_change":"2024-09-13T16:31:28.609114+0000","last_active":"2024-09-13T16:31:28.999334+0000","last_peered":"2024-09-13T16:31:28.999334+0000","last_clean":"2024-09-13T16:31:28.999334+0000","last_became_active":"2024-09-13T16:31:28.608136+0000","last_became_peered":"2024-09-13T16:31:28.608136+0000","last_unstale":"2024-09-13T16:31:28.999334+0000","last_undegraded":"2024-09-13T16:31:28.999334+0000","last_fullsized":"2024-09-13T16:31:28.999334+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:47.319412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974225+0000","last_change":"2024-09-13T16:31:28.977196+0000","last_active":"2024-09-13T16:31:29.974225+0000","last_peered":"2024-09-13T16:31:29.974225+0000","last_clean":"2024-09-13T16:31:29.974225+0000","last_became_active":"2024-09-13T16:31:28.976364+0000","last_became_peered":"2024-09-13T16:31:28.976364+0000","last_unstale":"2024-09-13T16:31:29.974225+0000","last_undegraded":"2024-09-13T16:31:29.974225+0000","last_fullsized":"2024-09-13T16:31:29.974225+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:03:57.203509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.f","version":"94'1","reported_seq":12,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975921+0000","last_change":"2024-09-13T16:31:28.607683+0000","last_active":"2024-09-13T16:31:29.975921+0000","last_peered":"2024-09-13T16:31:29.975921+0000","last_clean":"2024-09-13T16:31:29.975921+0000","last_became_active":"2024-09-13T16:31:28.607258+0000","last_became_peered":"2024-09-13T16:31:28.607258+0000","last_unstale":"2024-09-13T16:31:29.975921+0000","last_undegraded":"2024-09-13T16:31:29.975921+0000","last_fullsized":"2024-09-13T16:31:29.975921+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:42:41.523462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.d","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975472+0000","last_change":"2024-09-13T16:31:28.996123+0000","last_active":"2024-09-13T16:31:29.975472+0000","last_peered":"2024-09-13T16:31:29.975472+0000","last_clean":"2024-09-13T16:31:29.975472+0000","last_became_active":"2024-09-13T16:31:28.995799+0000","last_became_peered":"2024-09-13T16:31:28.995799+0000","last_unstale":"2024-09-13T16:31:29.975472+0000","last_undegraded":"2024-09-13T16:31:29.975472+0000","last_fullsized":"2024-09-13T16:31:29.975472+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:48.233359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.e","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.994696+0000","last_change":"2024-09-13T16:31:28.602961+0000","last_active":"2024-09-13T16:31:28.994696+0000","last_peered":"2024-09-13T16:31:28.994696+0000","last_clean":"2024-09-13T16:31:28.994696+0000","last_became_active":"2024-09-13T16:31:28.602678+0000","last_became_peered":"2024-09-13T16:31:28.602678+0000","last_unstale":"2024-09-13T16:31:28.994696+0000","last_undegraded":"2024-09-13T16:31:28.994696+0000","last_fullsized":"2024-09-13T16:31:28.994696+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:45:44.233302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.e","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.982946+0000","last_change":"2024-09-13T16:31:28.980653+0000","last_active":"2024-09-13T16:31:28.982946+0000","last_peered":"2024-09-13T16:31:28.982946+0000","last_clean":"2024-09-13T16:31:28.982946+0000","last_became_active":"2024-09-13T16:31:28.980270+0000","last_became_peered":"2024-09-13T16:31:28.980270+0000","last_unstale":"2024-09-13T16:31:28.982946+0000","last_undegraded":"2024-09-13T16:31:28.982946+0000","last_fullsized":"2024-09-13T16:31:28.982946+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:32:40.930927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.d","version":"94'1","reported_seq":11,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999540+0000","last_change":"2024-09-13T16:31:28.607551+0000","last_active":"2024-09-13T16:31:28.999540+0000","last_peered":"2024-09-13T16:31:28.999540+0000","last_clean":"2024-09-13T16:31:28.999540+0000","last_became_active":"2024-09-13T16:31:28.607290+0000","last_became_peered":"2024-09-13T16:31:28.607290+0000","last_unstale":"2024-09-13T16:31:28.999540+0000","last_undegraded":"2024-09-13T16:31:28.999540+0000","last_fullsized":"2024-09-13T16:31:28.999540+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:48:05.193753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973684+0000","last_change":"2024-09-13T16:31:28.996100+0000","last_active":"2024-09-13T16:31:29.973684+0000","last_peered":"2024-09-13T16:31:29.973684+0000","last_clean":"2024-09-13T16:31:29.973684+0000","last_became_active":"2024-09-13T16:31:28.995838+0000","last_became_peered":"2024-09-13T16:31:28.995838+0000","last_unstale":"2024-09-13T16:31:29.973684+0000","last_undegraded":"2024-09-13T16:31:29.973684+0000","last_fullsized":"2024-09-13T16:31:29.973684+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:12:08.102040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.b","version":"94'2","reported_seq":13,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976482+0000","last_change":"2024-09-13T16:31:28.602778+0000","last_active":"2024-09-13T16:31:29.976482+0000","last_peered":"2024-09-13T16:31:29.976482+0000","last_clean":"2024-09-13T16:31:29.976482+0000","last_became_active":"2024-09-13T16:31:28.602415+0000","last_became_peered":"2024-09-13T16:31:28.602415+0000","last_unstale":"2024-09-13T16:31:29.976482+0000","last_undegraded":"2024-09-13T16:31:29.976482+0000","last_fullsized":"2024-09-13T16:31:29.976482+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:32:59.521610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976362+0000","last_change":"2024-09-13T16:31:28.989696+0000","last_active":"2024-09-13T16:31:29.976362+0000","last_peered":"2024-09-13T16:31:29.976362+0000","last_clean":"2024-09-13T16:31:29.976362+0000","last_became_active":"2024-09-13T16:31:28.989443+0000","last_became_peered":"2024-09-13T16:31:28.989443+0000","last_unstale":"2024-09-13T16:31:29.976362+0000","last_undegraded":"2024-09-13T16:31:29.976362+0000","last_fullsized":"2024-09-13T16:31:29.976362+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:57:30.810042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.973445+0000","last_change":"2024-09-13T16:31:28.605291+0000","last_active":"2024-09-13T16:31:28.973445+0000","last_peered":"2024-09-13T16:31:28.973445+0000","last_clean":"2024-09-13T16:31:28.973445+0000","last_became_active":"2024-09-13T16:31:28.604986+0000","last_became_peered":"2024-09-13T16:31:28.604986+0000","last_unstale":"2024-09-13T16:31:28.973445+0000","last_undegraded":"2024-09-13T16:31:28.973445+0000","last_fullsized":"2024-09-13T16:31:28.973445+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:34:48.829434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976326+0000","last_change":"2024-09-13T16:31:28.993957+0000","last_active":"2024-09-13T16:31:29.976326+0000","last_peered":"2024-09-13T16:31:29.976326+0000","last_clean":"2024-09-13T16:31:29.976326+0000","last_became_active":"2024-09-13T16:31:28.993619+0000","last_became_peered":"2024-09-13T16:31:28.993619+0000","last_unstale":"2024-09-13T16:31:29.976326+0000","last_undegraded":"2024-09-13T16:31:29.976326+0000","last_fullsized":"2024-09-13T16:31:29.976326+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:40:58.626444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976255+0000","last_change":"2024-09-13T16:31:28.604158+0000","last_active":"2024-09-13T16:31:29.976255+0000","last_peered":"2024-09-13T16:31:29.976255+0000","last_clean":"2024-09-13T16:31:29.976255+0000","last_became_active":"2024-09-13T16:31:28.603864+0000","last_became_peered":"2024-09-13T16:31:28.603864+0000","last_unstale":"2024-09-13T16:31:29.976255+0000","last_undegraded":"2024-09-13T16:31:29.976255+0000","last_fullsized":"2024-09-13T16:31:29.976255+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:33.584847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.b","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.973888+0000","last_change":"2024-09-13T16:31:28.983253+0000","last_active":"2024-09-13T16:31:29.973888+0000","last_peered":"2024-09-13T16:31:29.973888+0000","last_clean":"2024-09-13T16:31:29.973888+0000","last_became_active":"2024-09-13T16:31:28.983008+0000","last_became_peered":"2024-09-13T16:31:28.983008+0000","last_unstale":"2024-09-13T16:31:29.973888+0000","last_undegraded":"2024-09-13T16:31:29.973888+0000","last_fullsized":"2024-09-13T16:31:29.973888+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:39:44.101104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.971826+0000","last_change":"2024-09-13T16:31:28.605300+0000","last_active":"2024-09-13T16:31:28.971826+0000","last_peered":"2024-09-13T16:31:28.971826+0000","last_clean":"2024-09-13T16:31:28.971826+0000","last_became_active":"2024-09-13T16:31:28.605027+0000","last_became_peered":"2024-09-13T16:31:28.605027+0000","last_unstale":"2024-09-13T16:31:28.971826+0000","last_undegraded":"2024-09-13T16:31:28.971826+0000","last_fullsized":"2024-09-13T16:31:28.971826+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:25:46.128750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975489+0000","last_change":"2024-09-13T16:31:28.988236+0000","last_active":"2024-09-13T16:31:29.975489+0000","last_peered":"2024-09-13T16:31:29.975489+0000","last_clean":"2024-09-13T16:31:29.975489+0000","last_became_active":"2024-09-13T16:31:28.984106+0000","last_became_peered":"2024-09-13T16:31:28.984106+0000","last_unstale":"2024-09-13T16:31:29.975489+0000","last_undegraded":"2024-09-13T16:31:29.975489+0000","last_fullsized":"2024-09-13T16:31:29.975489+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:05:50.487730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.7","version":"94'2","reported_seq":12,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.999070+0000","last_change":"2024-09-13T16:31:28.606027+0000","last_active":"2024-09-13T16:31:28.999070+0000","last_peered":"2024-09-13T16:31:28.999070+0000","last_clean":"2024-09-13T16:31:28.999070+0000","last_became_active":"2024-09-13T16:31:28.605602+0000","last_became_peered":"2024-09-13T16:31:28.605602+0000","last_unstale":"2024-09-13T16:31:28.999070+0000","last_undegraded":"2024-09-13T16:31:28.999070+0000","last_fullsized":"2024-09-13T16:31:28.999070+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:12:53.445131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974522+0000","last_change":"2024-09-13T16:31:28.975660+0000","last_active":"2024-09-13T16:31:29.974522+0000","last_peered":"2024-09-13T16:31:29.974522+0000","last_clean":"2024-09-13T16:31:29.974522+0000","last_became_active":"2024-09-13T16:31:28.975169+0000","last_became_peered":"2024-09-13T16:31:28.975169+0000","last_unstale":"2024-09-13T16:31:29.974522+0000","last_undegraded":"2024-09-13T16:31:29.974522+0000","last_fullsized":"2024-09-13T16:31:29.974522+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:14:05.502528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.973177+0000","last_change":"2024-09-13T16:31:28.607851+0000","last_active":"2024-09-13T16:31:28.973177+0000","last_peered":"2024-09-13T16:31:28.973177+0000","last_clean":"2024-09-13T16:31:28.973177+0000","last_became_active":"2024-09-13T16:31:28.604623+0000","last_became_peered":"2024-09-13T16:31:28.604623+0000","last_unstale":"2024-09-13T16:31:28.973177+0000","last_undegraded":"2024-09-13T16:31:28.973177+0000","last_fullsized":"2024-09-13T16:31:28.973177+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:36:47.223161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974869+0000","last_change":"2024-09-13T16:31:28.980115+0000","last_active":"2024-09-13T16:31:29.974869+0000","last_peered":"2024-09-13T16:31:29.974869+0000","last_clean":"2024-09-13T16:31:29.974869+0000","last_became_active":"2024-09-13T16:31:28.979882+0000","last_became_peered":"2024-09-13T16:31:28.979882+0000","last_unstale":"2024-09-13T16:31:29.974869+0000","last_undegraded":"2024-09-13T16:31:29.974869+0000","last_fullsized":"2024-09-13T16:31:29.974869+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:25:38.621442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.973669+0000","last_change":"2024-09-13T16:31:28.607390+0000","last_active":"2024-09-13T16:31:28.973669+0000","last_peered":"2024-09-13T16:31:28.973669+0000","last_clean":"2024-09-13T16:31:28.973669+0000","last_became_active":"2024-09-13T16:31:28.606888+0000","last_became_peered":"2024-09-13T16:31:28.606888+0000","last_unstale":"2024-09-13T16:31:28.973669+0000","last_undegraded":"2024-09-13T16:31:28.973669+0000","last_fullsized":"2024-09-13T16:31:28.973669+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:07:47.176642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.7","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.977181+0000","last_change":"2024-09-13T16:31:28.980514+0000","last_active":"2024-09-13T16:31:29.977181+0000","last_peered":"2024-09-13T16:31:29.977181+0000","last_clean":"2024-09-13T16:31:29.977181+0000","last_became_active":"2024-09-13T16:31:28.980255+0000","last_became_peered":"2024-09-13T16:31:28.980255+0000","last_unstale":"2024-09-13T16:31:29.977181+0000","last_undegraded":"2024-09-13T16:31:29.977181+0000","last_fullsized":"2024-09-13T16:31:29.977181+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:17:18.033839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975648+0000","last_change":"2024-09-13T16:31:28.609741+0000","last_active":"2024-09-13T16:31:29.975648+0000","last_peered":"2024-09-13T16:31:29.975648+0000","last_clean":"2024-09-13T16:31:29.975648+0000","last_became_active":"2024-09-13T16:31:28.609500+0000","last_became_peered":"2024-09-13T16:31:28.609500+0000","last_unstale":"2024-09-13T16:31:29.975648+0000","last_undegraded":"2024-09-13T16:31:29.975648+0000","last_fullsized":"2024-09-13T16:31:29.975648+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:22.642570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.990388+0000","last_change":"2024-09-13T16:31:28.984233+0000","last_active":"2024-09-13T16:31:28.990388+0000","last_peered":"2024-09-13T16:31:28.990388+0000","last_clean":"2024-09-13T16:31:28.990388+0000","last_became_active":"2024-09-13T16:31:28.983943+0000","last_became_peered":"2024-09-13T16:31:28.983943+0000","last_unstale":"2024-09-13T16:31:28.990388+0000","last_undegraded":"2024-09-13T16:31:28.990388+0000","last_fullsized":"2024-09-13T16:31:28.990388+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:44.781084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974312+0000","last_change":"2024-09-13T16:31:28.611849+0000","last_active":"2024-09-13T16:31:29.974312+0000","last_peered":"2024-09-13T16:31:29.974312+0000","last_clean":"2024-09-13T16:31:29.974312+0000","last_became_active":"2024-09-13T16:31:28.611687+0000","last_became_peered":"2024-09-13T16:31:28.611687+0000","last_unstale":"2024-09-13T16:31:29.974312+0000","last_undegraded":"2024-09-13T16:31:29.974312+0000","last_fullsized":"2024-09-13T16:31:29.974312+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:45:01.180579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.000855+0000","last_change":"2024-09-13T16:31:28.999444+0000","last_active":"2024-09-13T16:31:29.000855+0000","last_peered":"2024-09-13T16:31:29.000855+0000","last_clean":"2024-09-13T16:31:29.000855+0000","last_became_active":"2024-09-13T16:31:28.999146+0000","last_became_peered":"2024-09-13T16:31:28.999146+0000","last_unstale":"2024-09-13T16:31:29.000855+0000","last_undegraded":"2024-09-13T16:31:29.000855+0000","last_fullsized":"2024-09-13T16:31:29.000855+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:38:15.075638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.976420+0000","last_change":"2024-09-13T16:31:28.602782+0000","last_active":"2024-09-13T16:31:29.976420+0000","last_peered":"2024-09-13T16:31:29.976420+0000","last_clean":"2024-09-13T16:31:29.976420+0000","last_became_active":"2024-09-13T16:31:28.602477+0000","last_became_peered":"2024-09-13T16:31:28.602477+0000","last_unstale":"2024-09-13T16:31:29.976420+0000","last_undegraded":"2024-09-13T16:31:29.976420+0000","last_fullsized":"2024-09-13T16:31:29.976420+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:31:22.385998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974993+0000","last_change":"2024-09-13T16:31:28.996549+0000","last_active":"2024-09-13T16:31:29.974993+0000","last_peered":"2024-09-13T16:31:29.974993+0000","last_clean":"2024-09-13T16:31:29.974993+0000","last_became_active":"2024-09-13T16:31:28.996209+0000","last_became_peered":"2024-09-13T16:31:28.996209+0000","last_unstale":"2024-09-13T16:31:29.974993+0000","last_undegraded":"2024-09-13T16:31:29.974993+0000","last_fullsized":"2024-09-13T16:31:29.974993+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:54:53.364598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.975307+0000","last_change":"2024-09-13T16:31:28.609266+0000","last_active":"2024-09-13T16:31:29.975307+0000","last_peered":"2024-09-13T16:31:29.975307+0000","last_clean":"2024-09-13T16:31:29.975307+0000","last_became_active":"2024-09-13T16:31:28.609002+0000","last_became_peered":"2024-09-13T16:31:28.609002+0000","last_unstale":"2024-09-13T16:31:29.975307+0000","last_undegraded":"2024-09-13T16:31:29.975307+0000","last_fullsized":"2024-09-13T16:31:29.975307+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:29:06.461863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":10,"reported_epoch":95,"state":"active+clean","last_fresh":"2024-09-13T16:31:29.974179+0000","last_change":"2024-09-13T16:31:28.987013+0000","last_active":"2024-09-13T16:31:29.974179+0000","last_peered":"2024-09-13T16:31:29.974179+0000","last_clean":"2024-09-13T16:31:29.974179+0000","last_became_active":"2024-09-13T16:31:28.986765+0000","last_became_peered":"2024-09-13T16:31:28.986765+0000","last_unstale":"2024-09-13T16:31:29.974179+0000","last_undegraded":"2024-09-13T16:31:29.974179+0000","last_fullsized":"2024-09-13T16:31:29.974179+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:04.155701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.977582+0000","last_change":"2024-09-13T16:31:28.602885+0000","last_active":"2024-09-13T16:31:28.977582+0000","last_peered":"2024-09-13T16:31:28.977582+0000","last_clean":"2024-09-13T16:31:28.977582+0000","last_became_active":"2024-09-13T16:31:28.602600+0000","last_became_peered":"2024-09-13T16:31:28.602600+0000","last_unstale":"2024-09-13T16:31:28.977582+0000","last_undegraded":"2024-09-13T16:31:28.977582+0000","last_fullsized":"2024-09-13T16:31:28.977582+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:59:14.555270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.f","version":"0'0","reported_seq":9,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.989750+0000","last_change":"2024-09-13T16:31:28.988583+0000","last_active":"2024-09-13T16:31:28.989750+0000","last_peered":"2024-09-13T16:31:28.989750+0000","last_clean":"2024-09-13T16:31:28.989750+0000","last_became_active":"2024-09-13T16:31:28.988370+0000","last_became_peered":"2024-09-13T16:31:28.988370+0000","last_unstale":"2024-09-13T16:31:28.989750+0000","last_undegraded":"2024-09-13T16:31:28.989750+0000","last_fullsized":"2024-09-13T16:31:28.989750+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:28.572720+0000","last_clean_scrub_stamp":"2024-09-13T16:31:28.572720+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:20:12.341081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":10,"reported_epoch":94,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.982290+0000","last_change":"2024-09-13T16:31:28.603914+0000","last_active":"2024-09-13T16:31:28.982290+0000","last_peered":"2024-09-13T16:31:28.982290+0000","last_clean":"2024-09-13T16:31:28.982290+0000","last_became_active":"2024-09-13T16:31:28.603522+0000","last_became_peered":"2024-09-13T16:31:28.603522+0000","last_unstale":"2024-09-13T16:31:28.982290+0000","last_undegraded":"2024-09-13T16:31:28.982290+0000","last_fullsized":"2024-09-13T16:31:28.982290+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":92,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:31:27.562521+0000","last_clean_scrub_stamp":"2024-09-13T16:31:27.562521+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:58:22.642570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":228,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-09-13T16:31:28.593497+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:31:28.593497+0000","last_peered":"2024-09-13T16:31:28.593497+0000","last_clean":"2024-09-13T16:31:28.593497+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:31:28.593497+0000","last_undegraded":"2024-09-13T16:31:28.593497+0000","last_fullsized":"2024-09-13T16:31:28.593497+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":14,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542223,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32836,"kb_used_data":1716,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339004,"statfs":{"total":96636764160,"available":96603140096,"internally_reserved":0,"allocated":1757184,"data_stored":1116128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.85899999999999999,"15min":0.85299999999999998},"min":{"1min":0.59099999999999997,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1259999999999999,"5min":1.357,"15min":1.357},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.872,"15min":0.85199999999999998},"min":{"1min":0.65400000000000003,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.075,"5min":1.3460000000000001,"15min":1.3460000000000001},"last":0.65200000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.84699999999999998,"15min":0.85299999999999998},"min":{"1min":0.61799999999999999,"5min":0.501,"15min":0.501},"max":{"1min":1.107,"5min":1.569,"15min":1.569},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.82199999999999995,"5min":0.86299999999999999,"15min":0.84899999999999998},"min":{"1min":0.65500000000000003,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.1020000000000001,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":0.623}]},{"osd":2,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.88600000000000001,"15min":0.88600000000000001},"min":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"max":{"1min":1.2330000000000001,"5min":1.242,"15min":1.242},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.89000000000000001,"15min":0.84999999999999998},"min":{"1min":0.71699999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1919999999999999,"5min":1.339,"15min":1.339},"last":0.71399999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.879,"5min":0.93899999999999995,"15min":0.99299999999999999},"min":{"1min":0.64500000000000002,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.127,"5min":1.476,"15min":1.476},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.98899999999999999,"5min":0.99099999999999999,"15min":1.0049999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.21,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.79500000000000004}]},{"osd":4,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.85599999999999998,"5min":0.84799999999999998,"15min":0.88},"min":{"1min":0.57899999999999996,"5min":0.56799999999999995,"15min":0.56799999999999995},"max":{"1min":1.198,"5min":1.216,"15min":1.216},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.97099999999999997,"15min":1.0589999999999999},"min":{"1min":0.63900000000000001,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.2030000000000001,"5min":1.5549999999999999,"15min":1.589},"last":0.47599999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.94199999999999995,"15min":1.012},"min":{"1min":0.70099999999999996,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.276,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.76300000000000001},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0760000000000001,"15min":1.1080000000000001},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"max":{"1min":1.242,"5min":1.3200000000000001,"15min":1.337},"last":0.58199999999999996}]},{"osd":6,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.056,"5min":1.03,"15min":1.071},"min":{"1min":0.85599999999999998,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.3069999999999999,"5min":1.4510000000000001,"15min":1.4510000000000001},"last":0.68799999999999994},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.087,"15min":1.1579999999999999},"min":{"1min":0.84999999999999998,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.2569999999999999,"5min":1.623,"15min":1.6779999999999999},"last":0.57499999999999996}]}]},{"osd":6,"up_from":14,"seq":60129542222,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28192,"kb_used_data":1160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343648,"statfs":{"total":96636764160,"available":96607895552,"internally_reserved":0,"allocated":1187840,"data_stored":534492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.77100000000000002,"5min":0.82099999999999995,"15min":0.84399999999999997},"min":{"1min":0.59199999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":0.93400000000000005,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.86099999999999999,"5min":0.88,"15min":0.89100000000000001},"min":{"1min":0.63400000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":1.1399999999999999,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.47199999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.84199999999999997,"15min":0.83999999999999997},"min":{"1min":0.71299999999999997,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.1339999999999999,"5min":1.1799999999999999,"15min":1.2030000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.92700000000000005,"15min":0.91900000000000004},"min":{"1min":0.65500000000000003,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1140000000000001,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.85299999999999998,"15min":0.878},"min":{"1min":0.57499999999999996,"5min":0.503,"15min":0.503},"max":{"1min":1.101,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.93700000000000006,"5min":0.89100000000000001,"15min":0.85899999999999999},"min":{"1min":0.72699999999999998,"5min":0.56899999999999995,"15min":0.55500000000000005},"max":{"1min":1.097,"5min":1.276,"15min":1.276},"last":0.78000000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.95699999999999996,"15min":0.96499999999999997},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.2410000000000001,"5min":1.5189999999999999,"15min":1.536},"last":1.0649999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.98099999999999998,"15min":0.98699999999999999},"min":{"1min":0.71299999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.133,"5min":1.3120000000000001,"15min":1.409},"last":0.83599999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.93100000000000005,"15min":0.96699999999999997},"min":{"1min":0.69499999999999995,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.177,"5min":1.4570000000000001,"15min":1.4750000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.93999999999999995,"5min":1.0009999999999999,"15min":0.95299999999999996},"min":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.347,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.91600000000000004}]},{"osd":5,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":0.95599999999999996,"15min":0.92800000000000005},"min":{"1min":0.81799999999999995,"5min":0.628,"15min":0.628},"max":{"1min":1.339,"5min":1.339,"15min":1.339},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0629999999999999,"15min":1.1180000000000001},"min":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.286,"5min":1.4259999999999999,"15min":1.6439999999999999},"last":0.96999999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:31:20 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.089,"15min":1.113},"min":{"1min":0.80000000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":1.256,"5min":1.522,"15min":1.522},"last":0.999},{"interface":"front","average":{"1min":1.1319999999999999,"5min":1.0660000000000001,"15min":1.0609999999999999},"min":{"1min":0.78500000000000003,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.415,"5min":1.427,"15min":1.427},"last":1.0429999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542224,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1155072,"data_stored":525704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81399999999999995,"5min":0.83799999999999997,"15min":0.85399999999999998},"min":{"1min":0.54600000000000004,"5min":0.47899999999999998,"15min":0.47599999999999998},"max":{"1min":1.0940000000000001,"5min":1.411,"15min":1.411},"last":1.1899999999999999},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.82599999999999996,"15min":0.84799999999999998},"min":{"1min":0.59999999999999998,"5min":0.48299999999999998,"15min":0.41699999999999998},"max":{"1min":1.1499999999999999,"5min":1.3049999999999999,"15min":1.675},"last":0.70299999999999996}]},{"osd":1,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.85199999999999998,"15min":0.876},"min":{"1min":0.59699999999999998,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.093,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.86499999999999999,"15min":0.83599999999999997},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.36499999999999999},"max":{"1min":1.0960000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"last":0.88900000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.90400000000000003,"15min":0.92800000000000005},"min":{"1min":0.68500000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1299999999999999,"5min":1.276,"15min":1.625},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.94299999999999995,"15min":0.97699999999999998},"min":{"1min":0.67900000000000005,"5min":0.57599999999999996,"15min":0.55000000000000004},"max":{"1min":1.1990000000000001,"5min":1.516,"15min":1.552},"last":1.036}]},{"osd":3,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.90300000000000002,"15min":0.85299999999999998},"min":{"1min":0.71799999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.054,"5min":1.3919999999999999,"15min":1.3919999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.89800000000000002,"5min":0.90000000000000002,"15min":0.99199999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.60499999999999998},"max":{"1min":1.141,"5min":1.1819999999999999,"15min":3.8460000000000001},"last":0.85899999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.90400000000000003,"5min":0.93600000000000005,"15min":1.0369999999999999},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.224,"5min":1.655,"15min":3.6869999999999998},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.0629999999999999,"15min":1.0609999999999999},"min":{"1min":0.91200000000000003,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.244,"5min":1.47,"15min":1.601},"last":1.093}]},{"osd":6,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.93899999999999995,"15min":0.96599999999999997},"min":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.123,"5min":1.319,"15min":1.421},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.0429999999999999,"15min":1.0669999999999999},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.28,"5min":2.21,"15min":2.21},"last":1.121}]},{"osd":7,"last update":"Fri Sep 13 16:31:16 2024","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.0369999999999999,"15min":1.0349999999999999},"min":{"1min":0.65900000000000003,"5min":0.621,"15min":0.45500000000000002},"max":{"1min":1.2909999999999999,"5min":1.3819999999999999,"15min":1.4530000000000001},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.042,"5min":1.0660000000000001,"15min":1.1930000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.333,"5min":1.51,"15min":3.9830000000000001},"last":1.0629999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542222,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32256,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339584,"statfs":{"total":96636764160,"available":96603734016,"internally_reserved":0,"allocated":1163264,"data_stored":525864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.83899999999999997,"15min":0.83699999999999997},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1990000000000001,"5min":1.411,"15min":1.411},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.89500000000000002,"15min":0.86499999999999999},"min":{"1min":0.58499999999999996,"5min":0.52800000000000002,"15min":0.44600000000000001},"max":{"1min":1.462,"5min":2.899,"15min":2.899},"last":0.96199999999999997}]},{"osd":1,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.92000000000000004,"15min":0.91800000000000004},"min":{"1min":0.66400000000000003,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":1.2490000000000001,"5min":3.085,"15min":3.085},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.85399999999999998,"5min":0.91900000000000004,"15min":0.91700000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.3240000000000001,"5min":1.5720000000000001,"15min":1.5720000000000001},"last":0.67000000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.84399999999999997,"15min":0.83799999999999997},"min":{"1min":0.59599999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.1339999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.93600000000000005,"5min":0.97299999999999998,"15min":0.96899999999999997},"min":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.498},"max":{"1min":1.357,"5min":3.2160000000000002,"15min":3.2160000000000002},"last":0.73499999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.92800000000000005,"15min":0.94899999999999995},"min":{"1min":0.56899999999999995,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.421,"5min":1.421,"15min":1.472},"last":0.876},{"interface":"front","average":{"1min":0.92100000000000004,"5min":0.92700000000000005,"15min":0.95299999999999996},"min":{"1min":0.73699999999999999,"5min":0.55100000000000005,"15min":0.52200000000000002},"max":{"1min":1.1599999999999999,"5min":1.577,"15min":1.577},"last":0.83399999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.96599999999999997,"15min":0.97399999999999998},"min":{"1min":0.61599999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.2370000000000001,"5min":2.996,"15min":2.996},"last":0.90000000000000002},{"interface":"front","average":{"1min":0.96399999999999997,"5min":1.04,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.5189999999999999,"5min":1.79,"15min":2.8860000000000001},"last":0.98499999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"min":{"1min":0.63300000000000001,"5min":0.623,"15min":0.623},"max":{"1min":1.3959999999999999,"5min":1.6499999999999999,"15min":1.6499999999999999},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.127,"15min":1.1100000000000001},"min":{"1min":0.878,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.2330000000000001,"5min":3.1469999999999998,"15min":3.1469999999999998},"last":1.0109999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0209999999999999,"5min":1.0940000000000001,"15min":1.129},"min":{"1min":0.68200000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.2410000000000001,"5min":2.1230000000000002,"15min":2.1230000000000002},"last":0.80100000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.0980000000000001,"15min":1.1120000000000001},"min":{"1min":0.81699999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.3759999999999999,"5min":3.1070000000000002,"15min":3.1070000000000002},"last":0.91900000000000004}]}]},{"osd":3,"up_from":14,"seq":60129542223,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1155072,"data_stored":525704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.60499999999999998,"15min":0.627},"min":{"1min":0.42999999999999999,"5min":0.373,"15min":0.373},"max":{"1min":0.70299999999999996,"5min":1.04,"15min":1.04},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.67700000000000005,"15min":0.66300000000000003},"min":{"1min":0.51100000000000001,"5min":0.433,"15min":0.433},"max":{"1min":0.874,"5min":1.161,"15min":1.161},"last":0.51300000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.67400000000000004,"15min":0.63600000000000001},"min":{"1min":0.44500000000000001,"5min":0.378,"15min":0.35499999999999998},"max":{"1min":0.81799999999999995,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.82499999999999996,"5min":0.78600000000000003,"15min":0.78800000000000003},"min":{"1min":0.51000000000000001,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":1.0049999999999999,"5min":1.3029999999999999,"15min":1.4119999999999999},"last":0.69799999999999995}]},{"osd":2,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.72399999999999998,"5min":0.69499999999999995,"15min":0.69899999999999995},"min":{"1min":0.61199999999999999,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.91300000000000003,"5min":1.0649999999999999,"15min":1.27},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.78600000000000003,"15min":0.76200000000000001},"min":{"1min":0.58499999999999996,"5min":0.437,"15min":0.437},"max":{"1min":1.0509999999999999,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.64100000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.91900000000000004,"15min":0.90300000000000002},"min":{"1min":0.73499999999999999,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.121,"5min":1.288,"15min":1.8460000000000001},"last":1.0369999999999999},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.94499999999999995,"15min":0.95699999999999996},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.091,"5min":1.538,"15min":1.782},"last":0.77400000000000002}]},{"osd":5,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":0.89700000000000002,"5min":0.94199999999999995,"15min":0.96999999999999997},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.59499999999999997},"max":{"1min":1.3029999999999999,"5min":1.6299999999999999,"15min":1.6299999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.046,"15min":1.087},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.329,"5min":1.484,"15min":1.8140000000000001},"last":0.79200000000000004}]},{"osd":6,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.0840000000000001,"5min":1.0600000000000001,"15min":1.117},"min":{"1min":0.77600000000000002,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3919999999999999,"5min":1.6950000000000001,"15min":1.885},"last":0.75600000000000001},{"interface":"front","average":{"1min":1.036,"5min":1.0029999999999999,"15min":1.048},"min":{"1min":0.82699999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.331,"5min":1.4770000000000001,"15min":1.95},"last":0.91700000000000004}]},{"osd":7,"last update":"Fri Sep 13 16:31:17 2024","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0780000000000001,"15min":1.0760000000000001},"min":{"1min":0.85499999999999998,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.284,"5min":1.889,"15min":1.889},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.04,"5min":1.0469999999999999,"15min":1.0569999999999999},"min":{"1min":0.84299999999999997,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.1899999999999999,"5min":1.756,"15min":1.756},"last":0.81299999999999994}]}]},{"osd":2,"up_from":14,"seq":60129542224,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32244,"kb_used_data":1128,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339596,"statfs":{"total":96636764160,"available":96603746304,"internally_reserved":0,"allocated":1155072,"data_stored":525704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.61799999999999999,"5min":0.61099999999999999,"15min":0.59099999999999997},"min":{"1min":0.39800000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.0229999999999999,"5min":1.069,"15min":1.069},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.69799999999999995,"15min":0.69599999999999995},"min":{"1min":0.433,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":1.079,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.61199999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70199999999999996,"5min":0.68000000000000005,"15min":0.70199999999999996},"min":{"1min":0.47999999999999998,"5min":0.42099999999999999,"15min":0.40100000000000002},"max":{"1min":0.995,"5min":0.997,"15min":0.997},"last":0.871},{"interface":"front","average":{"1min":0.68999999999999995,"5min":0.68899999999999995,"15min":0.69199999999999995},"min":{"1min":0.44400000000000001,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.052,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.70799999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.68200000000000005,"15min":0.67200000000000004},"min":{"1min":0.441,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.03,"5min":1.054,"15min":1.073},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.83399999999999996,"15min":0.80200000000000005},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.377},"max":{"1min":1.091,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.76600000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.93600000000000005,"15min":0.95099999999999996},"min":{"1min":0.69299999999999995,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.272,"5min":1.4199999999999999,"15min":1.5149999999999999},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.92000000000000004,"15min":0.97199999999999998},"min":{"1min":0.76000000000000001,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.113,"5min":1.4330000000000001,"15min":1.8799999999999999},"last":0.98599999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.99299999999999999,"5min":0.96399999999999997,"15min":0.97999999999999998},"min":{"1min":0.68400000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.452,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.95999999999999996,"15min":0.96199999999999997},"min":{"1min":0.68600000000000005,"5min":0.65200000000000002,"15min":0.65200000000000002},"max":{"1min":1.21,"5min":1.3660000000000001,"15min":1.5720000000000001},"last":1.0980000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.93200000000000005,"15min":1.0149999999999999},"min":{"1min":0.70599999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.1950000000000001,"5min":1.4690000000000001,"15min":2.024},"last":0.82899999999999996},{"interface":"front","average":{"1min":1.069,"5min":1.0429999999999999,"15min":1.0609999999999999},"min":{"1min":0.87,"5min":0.78700000000000003,"15min":0.61799999999999999},"max":{"1min":1.5249999999999999,"5min":1.5249999999999999,"15min":1.6359999999999999},"last":1.0269999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:31:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.046,"15min":1.0740000000000001},"min":{"1min":0.71499999999999997,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.075,"5min":1.0289999999999999,"15min":1.0609999999999999},"min":{"1min":0.81399999999999995,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.22,"5min":1.4430000000000001,"15min":1.55},"last":0.79700000000000004}]}]},{"osd":1,"up_from":14,"seq":60129542223,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28160,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343680,"statfs":{"total":96636764160,"available":96607928320,"internally_reserved":0,"allocated":1159168,"data_stored":525774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.62,"15min":0.60599999999999998},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.23300000000000001},"max":{"1min":0.86699999999999999,"5min":1.0880000000000001,"15min":1.0880000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.76300000000000001,"15min":0.73499999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.91700000000000004,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.623,"5min":0.67500000000000004,"15min":0.627},"min":{"1min":0.496,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":0.755,"5min":1.0629999999999999,"15min":1.0629999999999999},"last":0.871},{"interface":"front","average":{"1min":0.69299999999999995,"5min":0.70799999999999996,"15min":0.70399999999999996},"min":{"1min":0.45700000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":0.89900000000000002,"5min":1.3129999999999999,"15min":1.458},"last":0.58099999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.75600000000000001,"15min":0.70099999999999996},"min":{"1min":0.42499999999999999,"5min":0.39700000000000002,"15min":0.35399999999999998},"max":{"1min":0.85199999999999998,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69299999999999995,"15min":0.68300000000000005},"min":{"1min":0.47099999999999997,"5min":0.44400000000000001,"15min":0.39600000000000002},"max":{"1min":1.151,"5min":1.151,"15min":1.403},"last":0.76500000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.93899999999999995,"15min":0.91100000000000003},"min":{"1min":0.78700000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.123,"5min":1.4930000000000001,"15min":1.546},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.87,"5min":0.90300000000000002,"15min":0.92500000000000004},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.51700000000000002},"max":{"1min":1.0529999999999999,"5min":1.377,"15min":2.2040000000000002},"last":0.91400000000000003}]},{"osd":5,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.96899999999999997,"15min":0.95099999999999996},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1870000000000001,"5min":1.389,"15min":1.389},"last":1.1200000000000001},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.91500000000000004,"15min":0.96299999999999997},"min":{"1min":0.56699999999999995,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0740000000000001,"5min":1.3979999999999999,"15min":2.3029999999999999},"last":0.96399999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":0.93600000000000005,"5min":0.999,"15min":1.0049999999999999},"min":{"1min":0.72899999999999998,"5min":0.70899999999999996,"15min":0.626},"max":{"1min":1.2070000000000001,"5min":1.4810000000000001,"15min":1.613},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.0169999999999999,"5min":1.022,"15min":1.0149999999999999},"min":{"1min":0.83899999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.282,"5min":1.548,"15min":1.698},"last":1.002}]},{"osd":7,"last update":"Fri Sep 13 16:31:19 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0009999999999999,"15min":1.008},"min":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.62},"max":{"1min":1.2849999999999999,"5min":1.5169999999999999,"15min":2.3999999999999999},"last":1.032},{"interface":"front","average":{"1min":1.0389999999999999,"5min":1.0700000000000001,"15min":1.0469999999999999},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.59299999999999997},"max":{"1min":1.4059999999999999,"5min":1.546,"15min":1.7689999999999999},"last":1.181}]}]},{"osd":0,"up_from":14,"seq":60129542223,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32868,"kb_used_data":1744,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338972,"statfs":{"total":96636764160,"available":96603107328,"internally_reserved":0,"allocated":1785856,"data_stored":1124962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.63700000000000001,"15min":0.66500000000000004},"min":{"1min":0.47399999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.93500000000000005,"5min":1.0600000000000001,"15min":1.1100000000000001},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.71799999999999997,"5min":0.68000000000000005,"15min":0.66200000000000003},"min":{"1min":0.442,"5min":0.42399999999999999,"15min":0.377},"max":{"1min":1.077,"5min":1.157,"15min":1.157},"last":0.442}]},{"osd":2,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.71299999999999997,"15min":0.72099999999999997},"min":{"1min":0.503,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":1.0109999999999999,"5min":1.202,"15min":1.202},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.72499999999999998,"15min":0.69199999999999995},"min":{"1min":0.50900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.96299999999999997,"5min":1.2629999999999999,"15min":1.2629999999999999},"last":0.51500000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.74199999999999999,"15min":0.66800000000000004},"min":{"1min":0.54700000000000004,"5min":0.39200000000000002,"15min":0.312},"max":{"1min":1.01,"5min":1.4830000000000001,"15min":1.4830000000000001},"last":0.74299999999999999},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.875,"15min":0.83499999999999996},"min":{"1min":0.73899999999999999,"5min":0.45300000000000001,"15min":0.42899999999999999},"max":{"1min":1.073,"5min":1.6100000000000001,"15min":1.6100000000000001},"last":0.86199999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.95299999999999996,"5min":0.95299999999999996,"15min":0.95899999999999996},"min":{"1min":0.71799999999999997,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":1.321,"5min":1.486,"15min":1.486},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.92400000000000004,"5min":0.97199999999999998,"15min":0.95899999999999996},"min":{"1min":0.68000000000000005,"5min":0.64600000000000002,"15min":0.59999999999999998},"max":{"1min":1.165,"5min":1.704,"15min":1.704},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":1.0169999999999999,"15min":0.98199999999999998},"min":{"1min":0.72799999999999998,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.23,"5min":1.575,"15min":1.575},"last":0.78900000000000003},{"interface":"front","average":{"1min":1.0049999999999999,"5min":0.97699999999999998,"15min":0.99299999999999999},"min":{"1min":0.69299999999999995,"5min":0.443,"15min":0.443},"max":{"1min":1.5029999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.69299999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":1,"15min":0.97999999999999998},"min":{"1min":0.71499999999999997,"5min":0.69499999999999995,"15min":0.628},"max":{"1min":1.2629999999999999,"5min":1.944,"15min":1.944},"last":0.83699999999999997},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.089,"15min":1.0580000000000001},"min":{"1min":0.78400000000000003,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.363,"5min":1.5780000000000001,"15min":1.5780000000000001},"last":0.97599999999999998}]},{"osd":7,"last update":"Fri Sep 13 16:31:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0649999999999999,"5min":1.077,"15min":1.054},"min":{"1min":0.747,"5min":0.747,"15min":0.61699999999999999},"max":{"1min":1.4059999999999999,"5min":1.659,"15min":1.659},"last":0.93000000000000005},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.1160000000000001,"15min":1.056},"min":{"1min":0.88900000000000001,"5min":0.76700000000000002,"15min":0.65400000000000003},"max":{"1min":1.472,"5min":2.0139999999999998,"15min":2.0139999999999998},"last":0.88900000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:31:33.381 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:31:33.381 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T16:31:33.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2de660f700 1 Processor -- start 2024-09-13T16:31:33.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2de660f700 1 -- start start 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2de660f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2de00b1870 0x7f2de00b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2de660f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2de00cd190 0x7f2de00cd5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2de660f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2de00b2210 con 0x7f2de015aca0 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2de660f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de00b3540 con 0x7f2de00b1870 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2de660f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2de00b12e0 con 0x7f2de00cd190 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2ddf7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2de00b1870 0x7f2de00b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2ddf7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2de00b1870 0x7f2de00b1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38232/0 (socket says 172.21.15.119:38232) 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.505+0000 7f2ddf7fe700 1 -- 172.21.15.119:0/3218275856 learned_addr learned my addr 172.21.15.119:0/3218275856 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:33.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de4bac700 1 --2- 172.21.15.119:0/3218275856 >> v2:172.21.15.145:3300/0 conn(0x7f2de00cd190 0x7f2de00cd5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddf7fe700 1 -- 172.21.15.119:0/3218275856 >> v2:172.21.15.145:3300/0 conn(0x7f2de00cd190 msgr2=0x7f2de00cd5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:33.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddf7fe700 1 --2- 172.21.15.119:0/3218275856 >> v2:172.21.15.145:3300/0 conn(0x7f2de00cd190 0x7f2de00cd5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddf7fe700 1 -- 172.21.15.119:0/3218275856 >> v1:172.21.15.145:6789/0 conn(0x7f2de015aca0 legacy=0x7f2de00cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddf7fe700 1 -- 172.21.15.119:0/3218275856 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de00cfa30 con 0x7f2de00b1870 2024-09-13T16:31:33.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddf7fe700 1 --2- 172.21.15.119:0/3218275856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de00b1870 0x7f2de00b1cd0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f2de015f930 tx=0x7f2dd0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d4bced5a2c366b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddeffd700 1 -- 172.21.15.119:0/3218275856 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0011830 con 0x7f2de00b1870 2024-09-13T16:31:33.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddeffd700 1 -- 172.21.15.119:0/3218275856 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd0011990 con 0x7f2de00b1870 2024-09-13T16:31:33.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2ddeffd700 1 -- 172.21.15.119:0/3218275856 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd0011d20 con 0x7f2de00b1870 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 -- 172.21.15.119:0/3218275856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de00b1870 msgr2=0x7f2de00b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 --2- 172.21.15.119:0/3218275856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de00b1870 0x7f2de00b1cd0 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 crypto rx=0x7f2de015f930 tx=0x7f2dd0009bf0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 -- 172.21.15.119:0/3218275856 shutdown_connections 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 --2- 172.21.15.119:0/3218275856 >> v2:172.21.15.145:3300/0 conn(0x7f2de00cd190 0x7f2de00cd5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 --2- 172.21.15.119:0/3218275856 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de00b1870 0x7f2de00b1cd0 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 -- 172.21.15.119:0/3218275856 >> 172.21.15.119:0/3218275856 conn(0x7f2de00b26e0 msgr2=0x7f2de00b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 -- 172.21.15.119:0/3218275856 shutdown_connections 2024-09-13T16:31:33.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.509+0000 7f2de660f700 1 -- 172.21.15.119:0/3218275856 wait complete. 2024-09-13T16:31:33.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 Processor -- start 2024-09-13T16:31:33.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 -- start start 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2de00b1870 0x7f2de0157640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2de00cd190 0x7f2de01514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de015aca0 0x7f2de00b5c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01f6d50 con 0x7f2de015aca0 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2de0157b80 con 0x7f2de00b1870 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2de01f6a70 con 0x7f2de00cd190 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de015aca0 0x7f2de00b5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de015aca0 0x7f2de00b5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38248/0 (socket says 172.21.15.119:38248) 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 -- 172.21.15.119:0/3857179363 learned_addr learned my addr 172.21.15.119:0/3857179363 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:33.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2ddf7fe700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2de00cd190 0x7f2de01514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2ddffff700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2de00b1870 0x7f2de0157640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 -- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2de00cd190 msgr2=0x7f2de01514f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2de00cd190 0x7f2de01514f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 -- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2de00b1870 msgr2=0x7f2de0157640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2de00b1870 0x7f2de0157640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 -- 172.21.15.119:0/3857179363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dd0009570 con 0x7f2de015aca0 2024-09-13T16:31:33.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2ddffff700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2de00b1870 0x7f2de0157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de4bac700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de015aca0 0x7f2de00b5c80 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800efc0 tx=0x7f2dd800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2ddcff9700 1 -- 172.21.15.119:0/3857179363 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8009ea0 con 0x7f2de015aca0 2024-09-13T16:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2ddcff9700 1 -- 172.21.15.119:0/3857179363 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dd800f040 con 0x7f2de015aca0 2024-09-13T16:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2de00b3b80 con 0x7f2de015aca0 2024-09-13T16:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2ddcff9700 1 -- 172.21.15.119:0/3857179363 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dd8014480 con 0x7f2de015aca0 2024-09-13T16:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.513+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2de00b4140 con 0x7f2de015aca0 2024-09-13T16:31:33.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.517+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2de00b39e0 con 0x7f2de015aca0 2024-09-13T16:31:33.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.517+0000 7f2ddcff9700 1 -- 172.21.15.119:0/3857179363 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2dd80145e0 con 0x7f2de015aca0 2024-09-13T16:31:33.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.517+0000 7f2ddcff9700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2dc0066730 0x7f2dc0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.517+0000 7f2ddcff9700 1 -- 172.21.15.119:0/3857179363 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2dd8084b90 con 0x7f2de015aca0 2024-09-13T16:31:33.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.517+0000 7f2ddffff700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2dc0066730 0x7f2dc0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.517+0000 7f2ddffff700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2dc0066730 0x7f2dc0068bf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8000f60 tx=0x7f2dc8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.525+0000 7f2ddcff9700 1 -- 172.21.15.119:0/3857179363 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2de00b39e0 con 0x7f2de015aca0 2024-09-13T16:31:33.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.757+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f2de00b39e0 con 0x7f2de015aca0 2024-09-13T16:31:33.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.765+0000 7f2ddcff9700 1 -- 172.21.15.119:0/3857179363 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+0 (secure 0 0 0) 0x7f2de00b39e0 con 0x7f2de015aca0 2024-09-13T16:31:33.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2dc0066730 msgr2=0x7f2dc0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2dc0066730 0x7f2dc0068bf0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f2dc8000f60 tx=0x7f2dc8009040 comp rx=0 tx=0).stop 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de015aca0 msgr2=0x7f2de00b5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de015aca0 0x7f2de00b5c80 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 crypto rx=0x7f2dd800efc0 tx=0x7f2dd800c3a0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 shutdown_connections 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2dc0066730 0x7f2dc0068bf0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2de015aca0 0x7f2de00b5c80 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2de00cd190 0x7f2de01514f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 --2- 172.21.15.119:0/3857179363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2de00b1870 0x7f2de0157640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 >> 172.21.15.119:0/3857179363 conn(0x7f2de00b26e0 msgr2=0x7f2de006f1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 shutdown_connections 2024-09-13T16:31:33.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.769+0000 7f2de660f700 1 -- 172.21.15.119:0/3857179363 wait complete. 2024-09-13T16:31:33.774 INFO:teuthology.orchestra.run.smithi119.stderr:updated caps for client.0 2024-09-13T16:31:33.787 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:33.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49edb5e700 1 Processor -- start 2024-09-13T16:31:33.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49edb5e700 1 -- start start 2024-09-13T16:31:33.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49edb5e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f49e8071740 0x7f49e806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49edb5e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158f10 0x7f49e806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49edb5e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49e806de00 con 0x7f49e8158b30 2024-09-13T16:31:33.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49edb5e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49e806d8b0 con 0x7f49e8071740 2024-09-13T16:31:33.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49edb5e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e806db20 con 0x7f49e8158f10 2024-09-13T16:31:33.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49dffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f49e8071740 0x7f49e806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49ecb5c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f49e8158b30 0x7f49e8071030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45560/0 (socket says 172.21.15.119:45560) 2024-09-13T16:31:33.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49ecb5c700 1 -- 172.21.15.119:0/2509800941 learned_addr learned my addr 172.21.15.119:0/2509800941 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:33.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49ed35d700 1 --2- 172.21.15.119:0/2509800941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158f10 0x7f49e806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49dffff700 1 -- 172.21.15.119:0/2509800941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158f10 msgr2=0x7f49e806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:33.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49dffff700 1 --2- 172.21.15.119:0/2509800941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158f10 0x7f49e806d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49dffff700 1 -- 172.21.15.119:0/2509800941 >> v1:172.21.15.119:6789/0 conn(0x7f49e8158b30 legacy=0x7f49e8071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.893+0000 7f49df7fe700 1 -- 172.21.15.119:0/2509800941 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771924123 0 0) 0x7f49e806de00 con 0x7f49e8158b30 2024-09-13T16:31:33.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49dffff700 1 -- 172.21.15.119:0/2509800941 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49e80b1920 con 0x7f49e8071740 2024-09-13T16:31:33.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49dffff700 1 --2- 172.21.15.119:0/2509800941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8071740 0x7f49e806ce30 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009850 tx=0x7f49d8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=591a5c503edd42c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49df7fe700 1 -- 172.21.15.119:0/2509800941 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8011830 con 0x7f49e8071740 2024-09-13T16:31:33.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49df7fe700 1 -- 172.21.15.119:0/2509800941 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49d8011990 con 0x7f49e8071740 2024-09-13T16:31:33.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49df7fe700 1 -- 172.21.15.119:0/2509800941 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49d8011d40 con 0x7f49e8071740 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 -- 172.21.15.119:0/2509800941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8071740 msgr2=0x7f49e806ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2509800941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8071740 0x7f49e806ce30 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 crypto rx=0x7f49d8009850 tx=0x7f49d8009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 -- 172.21.15.119:0/2509800941 shutdown_connections 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2509800941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158f10 0x7f49e806d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2509800941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8071740 0x7f49e806ce30 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 -- 172.21.15.119:0/2509800941 >> 172.21.15.119:0/2509800941 conn(0x7f49e806a350 msgr2=0x7f49e806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 -- 172.21.15.119:0/2509800941 shutdown_connections 2024-09-13T16:31:33.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 -- 172.21.15.119:0/2509800941 wait complete. 2024-09-13T16:31:33.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.897+0000 7f49edb5e700 1 Processor -- start 2024-09-13T16:31:33.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 -- start start 2024-09-13T16:31:33.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e8071740 0x7f49e813bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158b30 0x7f49e814f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8158f10 0x7f49e814ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e815b260 con 0x7f49e8158f10 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49e813c4d0 con 0x7f49e8071740 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49e815a1c0 con 0x7f49e8158b30 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8158f10 0x7f49e814ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8158f10 0x7f49e814ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38260/0 (socket says 172.21.15.119:38260) 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 -- 172.21.15.119:0/2868228021 learned_addr learned my addr 172.21.15.119:0/2868228021 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ecb5c700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e8071740 0x7f49e813bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49dffff700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158b30 0x7f49e814f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 -- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158b30 msgr2=0x7f49e814f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158b30 0x7f49e814f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 -- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e8071740 msgr2=0x7f49e813bf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e8071740 0x7f49e813bf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 -- 172.21.15.119:0/2868228021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d80094b0 con 0x7f49e8158f10 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49ed35d700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8158f10 0x7f49e814ff20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f49e000efc0 tx=0x7f49e000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49dd7fa700 1 -- 172.21.15.119:0/2868228021 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0009ba0 con 0x7f49e8158f10 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49dd7fa700 1 -- 172.21.15.119:0/2868228021 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49e000f040 con 0x7f49e8158f10 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49e815a4a0 con 0x7f49e8158f10 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49dd7fa700 1 -- 172.21.15.119:0/2868228021 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49e0014480 con 0x7f49e8158f10 2024-09-13T16:31:33.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49e815aa60 con 0x7f49e8158f10 2024-09-13T16:31:33.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.901+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49e8071030 con 0x7f49e8158f10 2024-09-13T16:31:33.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.905+0000 7f49dd7fa700 1 -- 172.21.15.119:0/2868228021 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f49e00145e0 con 0x7f49e8158f10 2024-09-13T16:31:33.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.905+0000 7f49dd7fa700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49d4066730 0x7f49d4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:33.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.905+0000 7f49dd7fa700 1 -- 172.21.15.119:0/2868228021 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f49e0084c30 con 0x7f49e8158f10 2024-09-13T16:31:33.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.905+0000 7f49ecb5c700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49d4066730 0x7f49d4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:33.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.905+0000 7f49ecb5c700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49d4066730 0x7f49d4068bf0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f49e4000f60 tx=0x7f49e4007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:33.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:33.909+0000 7f49dd7fa700 1 -- 172.21.15.119:0/2868228021 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f49e0051d30 con 0x7f49e8158f10 2024-09-13T16:31:34.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.113+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49e80ca990 con 0x7f49e8158f10 2024-09-13T16:31:34.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49dd7fa700 1 -- 172.21.15.119:0/2868228021 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f49e0018070 con 0x7f49e8158f10 2024-09-13T16:31:34.120 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:34.120 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7881,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:31.973745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6835/3011171408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3011171408},{"type":"v1","addr":"172.21.15.145:6835","nonce":3011171408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:31:34.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49d4066730 msgr2=0x7f49d4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49d4066730 0x7f49d4068bf0 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f49e4000f60 tx=0x7f49e4007c00 comp rx=0 tx=0).stop 2024-09-13T16:31:34.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8158f10 msgr2=0x7f49e814ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8158f10 0x7f49e814ff20 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 crypto rx=0x7f49e000efc0 tx=0x7f49e000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 shutdown_connections 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f49d4066730 0x7f49d4068bf0 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f49e8158f10 0x7f49e814ff20 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f49e8158b30 0x7f49e814f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 --2- 172.21.15.119:0/2868228021 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49e8071740 0x7f49e813bf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 >> 172.21.15.119:0/2868228021 conn(0x7f49e806a350 msgr2=0x7f49e806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 shutdown_connections 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.117+0000 7f49edb5e700 1 -- 172.21.15.119:0/2868228021 wait complete. 2024-09-13T16:31:34.123 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 56 2024-09-13T16:31:34.537 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:31:28.960022+0000', 'modified': '2024-09-13T16:31:31.973745+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7890}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7890': {'gid': 7890, 'name': 'b', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6835/3011171408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3011171408}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 3011171408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:31:34.537 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:31:34.537 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-13T16:31:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 Processor -- start 2024-09-13T16:31:34.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 -- start start 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980cd8d0 0x7f9c9806cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c980b17f0 0x7f9c980cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9c980b2200 0x7f9c980b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c980b2bd0 con 0x7f9c980b17f0 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c980705d0 con 0x7f9c980b2200 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9806d0d0 con 0x7f9c980cd8d0 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c980b17f0 0x7f9c980cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c980b17f0 0x7f9c980cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38262/0 (socket says 172.21.15.119:38262) 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 -- 172.21.15.119:0/880279813 learned_addr learned my addr 172.21.15.119:0/880279813 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/880279813 >> v2:172.21.15.145:3300/0 conn(0x7f9c980b2200 0x7f9c980b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9659c700 1 --2- 172.21.15.119:0/880279813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980cd8d0 0x7f9c9806cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 -- 172.21.15.119:0/880279813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980cd8d0 msgr2=0x7f9c9806cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/880279813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980cd8d0 0x7f9c9806cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 -- 172.21.15.119:0/880279813 >> v2:172.21.15.145:3300/0 conn(0x7f9c980b2200 msgr2=0x7f9c980b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/880279813 >> v2:172.21.15.145:3300/0 conn(0x7f9c980b2200 0x7f9c980b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 -- 172.21.15.119:0/880279813 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9806fca0 con 0x7f9c980b17f0 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/880279813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980b17f0 0x7f9c980cb800 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f9c98165960 tx=0x7f9c88009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03947039fe909a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9559a700 1 -- 172.21.15.119:0/880279813 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88011830 con 0x7f9c980b17f0 2024-09-13T16:31:34.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9559a700 1 -- 172.21.15.119:0/880279813 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c88011990 con 0x7f9c980b17f0 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9559a700 1 -- 172.21.15.119:0/880279813 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88011d20 con 0x7f9c980b17f0 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 -- 172.21.15.119:0/880279813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980b17f0 msgr2=0x7f9c980cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.661+0000 7f9c9759e700 1 --2- 172.21.15.119:0/880279813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980b17f0 0x7f9c980cb800 secure :-1 s=READY pgs=1269 cs=0 l=1 rev1=1 crypto rx=0x7f9c98165960 tx=0x7f9c88009bf0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- 172.21.15.119:0/880279813 shutdown_connections 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 --2- 172.21.15.119:0/880279813 >> v2:172.21.15.145:3300/0 conn(0x7f9c980b2200 0x7f9c980b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 --2- 172.21.15.119:0/880279813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980b17f0 0x7f9c980cb800 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 --2- 172.21.15.119:0/880279813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980cd8d0 0x7f9c9806cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- 172.21.15.119:0/880279813 >> 172.21.15.119:0/880279813 conn(0x7f9c9806a350 msgr2=0x7f9c9806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- 172.21.15.119:0/880279813 shutdown_connections 2024-09-13T16:31:34.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- 172.21.15.119:0/880279813 wait complete. 2024-09-13T16:31:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 Processor -- start 2024-09-13T16:31:34.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- start start 2024-09-13T16:31:34.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c980b17f0 0x7f9c98151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980b2200 0x7f9c980ae2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980cd8d0 0x7f9c980aa390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980a84b0 con 0x7f9c980cd8d0 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980ad390 con 0x7f9c980b17f0 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c980ad670 con 0x7f9c980b2200 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c96d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980cd8d0 0x7f9c980aa390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c980b17f0 0x7f9c98151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c96d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980cd8d0 0x7f9c980aa390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38278/0 (socket says 172.21.15.119:38278) 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c95d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980b2200 0x7f9c980ae2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c9659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c980b17f0 0x7f9c98151520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:46360/0 (socket says 172.21.15.119:46360) 2024-09-13T16:31:34.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.665+0000 7f9c96d9d700 1 -- 172.21.15.119:0/408057695 learned_addr learned my addr 172.21.15.119:0/408057695 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c96d9d700 1 -- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980b2200 msgr2=0x7f9c980ae2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980b2200 0x7f9c980ae2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c96d9d700 1 -- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c980b17f0 msgr2=0x7f9c98151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c980b17f0 0x7f9c98151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c96d9d700 1 -- 172.21.15.119:0/408057695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c88009570 con 0x7f9c980cd8d0 2024-09-13T16:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980b2200 0x7f9c980ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980cd8d0 0x7f9c980aa390 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000efc0 tx=0x7f9c9000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c86ffd700 1 -- 172.21.15.119:0/408057695 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c90009ea0 con 0x7f9c980cd8d0 2024-09-13T16:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c86ffd700 1 -- 172.21.15.119:0/408057695 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9000f040 con 0x7f9c980cd8d0 2024-09-13T16:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c980a8700 con 0x7f9c980cd8d0 2024-09-13T16:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c86ffd700 1 -- 172.21.15.119:0/408057695 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9000ced0 con 0x7f9c980cd8d0 2024-09-13T16:31:34.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.669+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c980aa8d0 con 0x7f9c980cd8d0 2024-09-13T16:31:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c86ffd700 1 -- 172.21.15.119:0/408057695 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9c90014480 con 0x7f9c980cd8d0 2024-09-13T16:31:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c86ffd700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c78066730 0x7f9c78068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c86ffd700 1 -- 172.21.15.119:0/408057695 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9c90084a50 con 0x7f9c980cd8d0 2024-09-13T16:31:34.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9659c700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c78066730 0x7f9c78068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9659c700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c78066730 0x7f9c78068bf0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009ad0 tx=0x7f9c8c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 Processor -- start 2024-09-13T16:31:34.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 -- start start 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29fc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29c4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2923d80 con 0x29dae90 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2921510 con 0x29fcb50 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x291ce00 con 0x29fc230 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c95d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c95d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29c4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38294/0 (socket says 172.21.15.119:38294) 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.673+0000 7f9c95d9b700 1 -- 172.21.15.119:0/4006970950 learned_addr learned my addr 172.21.15.119:0/4006970950 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9659c700 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29fc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c95d9b700 1 -- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 msgr2=0x29fc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29fc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c95d9b700 1 -- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 msgr2=0x29c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29c4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c95d9b700 1 -- 172.21.15.119:0/4006970950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x278f960 con 0x29dae90 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29c4890 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f9c88000c00 tx=0x7f9c88010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee3b997b5cf3eef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c77fff700 1 -- 172.21.15.119:0/4006970950 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88020070 con 0x29dae90 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c77fff700 1 -- 172.21.15.119:0/4006970950 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c880116f0 con 0x29dae90 2024-09-13T16:31:34.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c77fff700 1 -- 172.21.15.119:0/4006970950 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c88011d20 con 0x29dae90 2024-09-13T16:31:34.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- 172.21.15.119:0/4006970950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 msgr2=0x29c4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29c4890 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7f9c88000c00 tx=0x7f9c88010000 comp rx=0 tx=0).stop 2024-09-13T16:31:34.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- 172.21.15.119:0/4006970950 shutdown_connections 2024-09-13T16:31:34.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29c4890 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/4006970950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29fc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- 172.21.15.119:0/4006970950 >> 172.21.15.119:0/4006970950 conn(0x27444d0 msgr2=0x29dde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- 172.21.15.119:0/4006970950 shutdown_connections 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- 172.21.15.119:0/4006970950 wait complete. 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- start start 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 Processor -- start 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29dae90 0x29fba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29dc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fcb50 0x29da040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x28da730 con 0x29fcb50 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x28e05b0 con 0x29dae90 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x27c8aa0 con 0x29fc230 2024-09-13T16:31:34.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9fd04740 10 client.0 _ll_register_callbacks cb 0x2916610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c96d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fcb50 0x29da040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c96d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fcb50 0x29da040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38310/0 (socket says 172.21.15.119:38310) 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c96d9d700 1 -- 172.21.15.119:0/3604480637 learned_addr learned my addr 172.21.15.119:0/3604480637 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c9659c700 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29dae90 0x29fba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.677+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29dc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c96d9d700 1 -- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 msgr2=0x29dc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29dc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c96d9d700 1 -- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29dae90 msgr2=0x29fba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29dae90 0x29fba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c96d9d700 1 -- 172.21.15.119:0/3604480637 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c9806fca0 con 0x29fcb50 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fcb50 0x29da040 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f9c90082e40 tx=0x7f9c90055f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d7055f9fe81a52 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:34.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 1 -- 172.21.15.119:0/3604480637 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c9006d380 con 0x29fcb50 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 1 -- 172.21.15.119:0/3604480637 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9c9005dd30 con 0x29fcb50 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x27511e0 con 0x29fcb50 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 1 -- 172.21.15.119:0/3604480637 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c9005d8a0 con 0x29fcb50 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 10 client.8010 fetch_fsmap learned FSMap version 56 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x28e3220 con 0x29fcb50 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 1 -- 172.21.15.119:0/3604480637 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x27511e0 con 0x29fcb50 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 1 -- 172.21.15.119:0/3604480637 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 56) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9c900517e0 con 0x29fcb50 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 10 client.8010 fetch_fsmap finished waiting for FSMap version 56 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 10 client.8010 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 20 client.8010 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x29db5e0 0x29b53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 4 client.8010 mds_command: new command op to 7890 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:31:34.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 --> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x279b460 con 0x29db5e0 2024-09-13T16:31:34.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9659c700 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x29db5e0 0x29b53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 10 client.8010 ms_handle_connect on v2:172.21.15.145:6834/3011171408 2024-09-13T16:31:34.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9659c700 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x29db5e0 0x29b53e0 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20c15901e87023d4 server_cookie=cd94bfc20b231771 in_seq=0 out_seq=0 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 1 -- 172.21.15.119:0/3604480637 <== mds.0 v2:172.21.15.145:6834/3011171408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x279b460 con 0x29db5e0 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c757fa700 10 client.8010 handle_command_reply: tid=0 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.681+0000 7f9c9fd04740 1 client.8010 shutdown 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fcb50 msgr2=0x29da040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fcb50 0x29da040 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 crypto rx=0x7f9c90082e40 tx=0x7f9c90055f30 comp rx=0 tx=0).stop 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 shutdown_connections 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x29db5e0 0x29b53e0 crc :-1 s=READY pgs=10 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fcb50 0x29da040 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c757fa700 0 client.8010 ms_handle_reset on v2:172.21.15.145:6834/3011171408 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29dc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/3604480637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29dae90 0x29fba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 >> 172.21.15.119:0/3604480637 conn(0x27444d0 msgr2=0x29b0700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 shutdown_connections 2024-09-13T16:31:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 1 -- 172.21.15.119:0/3604480637 wait complete. 2024-09-13T16:31:34.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.685+0000 7f9c9fd04740 20 client.8010 trim_cache size 0 max 16384 2024-09-13T16:31:34.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 Processor -- start 2024-09-13T16:31:34.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 -- start start 2024-09-13T16:31:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29fef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29ff480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29ff9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2974ec0 con 0x29dae90 2024-09-13T16:31:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x29ac320 con 0x29fcb50 2024-09-13T16:31:34.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x28c3950 con 0x29fc230 2024-09-13T16:31:34.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29fef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29fef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38318/0 (socket says 172.21.15.119:38318) 2024-09-13T16:31:34.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 -- 172.21.15.119:0/2096928563 learned_addr learned my addr 172.21.15.119:0/2096928563 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29ff9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 -- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 msgr2=0x29ff480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29ff480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 -- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 msgr2=0x29ff9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29ff9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.705+0000 7f9c9659c700 1 -- 172.21.15.119:0/2096928563 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x28c9450 con 0x29dae90 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9659c700 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29fef40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c004430 tx=0x7f9c8c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c448a072671c10c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c84ff9700 1 -- 172.21.15.119:0/2096928563 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c01c040 con 0x29dae90 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c84ff9700 1 -- 172.21.15.119:0/2096928563 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c8c011c80 con 0x29dae90 2024-09-13T16:31:34.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c84ff9700 1 -- 172.21.15.119:0/2096928563 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c02c3a0 con 0x29dae90 2024-09-13T16:31:34.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 -- 172.21.15.119:0/2096928563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 msgr2=0x29fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29fef40 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c004430 tx=0x7f9c8c018000 comp rx=0 tx=0).stop 2024-09-13T16:31:34.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 -- 172.21.15.119:0/2096928563 shutdown_connections 2024-09-13T16:31:34.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x29ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29fc230 0x29ff480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/2096928563 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29dae90 0x29fef40 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 -- 172.21.15.119:0/2096928563 >> 172.21.15.119:0/2096928563 conn(0x27444d0 msgr2=0x2a0b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 -- 172.21.15.119:0/2096928563 shutdown_connections 2024-09-13T16:31:34.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 -- 172.21.15.119:0/2096928563 wait complete. 2024-09-13T16:31:34.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 -- start start 2024-09-13T16:31:34.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.709+0000 7f9c9fd04740 1 Processor -- start 2024-09-13T16:31:34.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29dae90 0x2a5cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fc230 0x2a191d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x2a19710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a1a2e0 con 0x29fc230 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a01a50 con 0x29fcb50 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a19da0 con 0x29dae90 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 10 client.0 _ll_register_callbacks cb 0x28cece0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c95d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fc230 0x2a191d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29dae90 0x2a5cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29dae90 0x2a5cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34090/0 (socket says 172.21.15.119:34090) 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 -- 172.21.15.119:0/1953200402 learned_addr learned my addr 172.21.15.119:0/1953200402 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x2a19710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 -- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 msgr2=0x2a19710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x2a19710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 -- 172.21.15.119:0/1953200402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fc230 msgr2=0x2a191d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fc230 0x2a191d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 -- 172.21.15.119:0/1953200402 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8c004160 con 0x29dae90 2024-09-13T16:31:34.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c96d9d700 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x2a19710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9659c700 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29dae90 0x2a5cb80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c030040 tx=0x7f9c8c0047e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=632718f52b1443c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c757fa700 10 client.0 ms_handle_connect on v2:172.21.15.145:3301/0 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c757fa700 1 -- 172.21.15.119:0/1953200402 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c8c02c5a0 con 0x29dae90 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c757fa700 1 -- 172.21.15.119:0/1953200402 <== mon.2 v2:172.21.15.145:3301/0 2 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9c8c029040 con 0x29dae90 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c880116f0 con 0x29dae90 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c757fa700 1 -- 172.21.15.119:0/1953200402 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_get_version_reply(handle=1 version=56) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c8c005e40 con 0x29dae90 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 10 client.6881 fetch_fsmap learned FSMap version 56 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,fsmap=56,monmap=2+}) v3 -- 0x28bd720 con 0x29dae90 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c757fa700 1 -- 172.21.15.119:0/1953200402 <== mon.2 v2:172.21.15.145:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c880116f0 con 0x29dae90 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c757fa700 1 -- 172.21.15.119:0/1953200402 <== mon.2 v2:172.21.15.145:3301/0 5 ==== fsmap(e 56) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9c8c02c700 con 0x29dae90 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 10 client.6881 fetch_fsmap finished waiting for FSMap version 56 2024-09-13T16:31:34.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 10 client.6881 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:31:34.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 20 client.6881 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:34.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x2a00270 0x2a00650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 4 client.6881 mds_command: new command op to 7890 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:31:34.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.713+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 --> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] -- command(tid 0: {"prefix": "status"}) v1 -- 0x27c01e0 con 0x2a00270 2024-09-13T16:31:34.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x2a00270 0x2a00650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c757fa700 10 client.6881 ms_handle_connect on v2:172.21.15.145:6834/3011171408 2024-09-13T16:31:34.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c95d9b700 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x2a00270 0x2a00650 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d6ce352b9cf9c04 server_cookie=4687d3c9e4a4a3b0 in_seq=0 out_seq=0 2024-09-13T16:31:34.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c757fa700 1 -- 172.21.15.119:0/1953200402 <== mds.0 v2:172.21.15.145:6834/3011171408 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x27c01e0 con 0x2a00270 2024-09-13T16:31:34.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c757fa700 10 client.6881 handle_command_reply: tid=0 2024-09-13T16:31:34.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 client.6881 shutdown 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29dae90 msgr2=0x2a5cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29dae90 0x2a5cb80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c030040 tx=0x7f9c8c0047e0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 shutdown_connections 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:6834/3011171408,v1:172.21.15.145:6835/3011171408] conn(0x2a00270 0x2a00650 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29fcb50 0x2a19710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29fc230 0x2a191d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 --2- 172.21.15.119:0/1953200402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29dae90 0x2a5cb80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c757fa700 0 client.6881 ms_handle_reset on v2:172.21.15.145:6834/3011171408 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 >> 172.21.15.119:0/1953200402 conn(0x27444d0 msgr2=0x2a0b6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 shutdown_connections 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 1 -- 172.21.15.119:0/1953200402 wait complete. 2024-09-13T16:31:34.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9fd04740 20 client.6881 trim_cache size 0 max 16384 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 7890, 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 5.7566085449999997, 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 56, 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 95, 2024-09-13T16:31:34.724 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 95, 2024-09-13T16:31:34.725 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 10.228926008 2024-09-13T16:31:34.725 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c78066730 msgr2=0x7f9c78068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c78066730 0x7f9c78068bf0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f9c8c009ad0 tx=0x7f9c8c009240 comp rx=0 tx=0).stop 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980cd8d0 msgr2=0x7f9c980aa390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980cd8d0 0x7f9c980aa390 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000efc0 tx=0x7f9c9000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 shutdown_connections 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c78066730 0x7f9c78068bf0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c980cd8d0 0x7f9c980aa390 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c980b2200 0x7f9c980ae2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 --2- 172.21.15.119:0/408057695 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c980b17f0 0x7f9c98151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 >> 172.21.15.119:0/408057695 conn(0x7f9c9806a350 msgr2=0x7f9c98162320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 shutdown_connections 2024-09-13T16:31:34.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.717+0000 7f9c9759e700 1 -- 172.21.15.119:0/408057695 wait complete. 2024-09-13T16:31:34.736 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:31:34.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.849+0000 7f7b6f320700 1 Processor -- start 2024-09-13T16:31:34.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.849+0000 7f7b6f320700 1 -- start start 2024-09-13T16:31:34.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7b6815ec20 0x7f7b68070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b68161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b68071210 con 0x7f7b6815e840 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b680714f0 con 0x7f7b6815ec20 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680717d0 con 0x7f7b68165180 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7b6815ec20 0x7f7b68070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7b6815ec20 0x7f7b68070b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38332/0 (socket says 172.21.15.119:38332) 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 -- 172.21.15.119:0/3204255010 learned_addr learned my addr 172.21.15.119:0/3204255010 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6d8bd700 1 --2- 172.21.15.119:0/3204255010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b68161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b5ffff700 1 -- 172.21.15.119:0/3204255010 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3927057930 0 0) 0x7f7b68071210 con 0x7f7b6815e840 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b5ffff700 1 -- 172.21.15.119:0/3204255010 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b540035d0 con 0x7f7b6815e840 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 -- 172.21.15.119:0/3204255010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 msgr2=0x7f7b68161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:34.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 --2- 172.21.15.119:0/3204255010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b68161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b5ffff700 1 -- 172.21.15.119:0/3204255010 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 188594951 0 0) 0x7f7b540035d0 con 0x7f7b6815e840 2024-09-13T16:31:34.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 -- 172.21.15.119:0/3204255010 >> v1:172.21.15.119:6789/0 conn(0x7f7b6815e840 legacy=0x7f7b68164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 -- 172.21.15.119:0/3204255010 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b68071ab0 con 0x7f7b6815ec20 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6c8bb700 1 --2- 172.21.15.119:0/3204255010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b6815ec20 0x7f7b68070b80 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009e50 tx=0x7f7b6400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce725306ad8bb7a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b5ffff700 1 -- 172.21.15.119:0/3204255010 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b6400e070 con 0x7f7b6815ec20 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b5ffff700 1 -- 172.21.15.119:0/3204255010 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b64003a40 con 0x7f7b6815ec20 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b5ffff700 1 -- 172.21.15.119:0/3204255010 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b58001670 con 0x7f7b6815ec20 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- 172.21.15.119:0/3204255010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b6815ec20 msgr2=0x7f7b68070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 --2- 172.21.15.119:0/3204255010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b6815ec20 0x7f7b68070b80 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7b64009e50 tx=0x7f7b6400b330 comp rx=0 tx=0).stop 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- 172.21.15.119:0/3204255010 shutdown_connections 2024-09-13T16:31:34.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 --2- 172.21.15.119:0/3204255010 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b68161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 --2- 172.21.15.119:0/3204255010 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b6815ec20 0x7f7b68070b80 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- 172.21.15.119:0/3204255010 >> 172.21.15.119:0/3204255010 conn(0x7f7b6806a350 msgr2=0x7f7b6806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:34.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- 172.21.15.119:0/3204255010 shutdown_connections 2024-09-13T16:31:34.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.853+0000 7f7b6f320700 1 -- 172.21.15.119:0/3204255010 wait complete. 2024-09-13T16:31:34.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 Processor -- start 2024-09-13T16:31:34.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 -- start start 2024-09-13T16:31:34.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b6815e840 0x7f7b681f61a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b680cd0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b680cd620 0x7f7b680af160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680cdad0 con 0x7f7b680cd620 2024-09-13T16:31:34.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b681f66e0 con 0x7f7b6815e840 2024-09-13T16:31:34.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6f320700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b680b0450 con 0x7f7b68165180 2024-09-13T16:31:34.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d8bd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b680cd620 0x7f7b680af160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6c8bb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b680cd0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d0bc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b6815e840 0x7f7b681f61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6c8bb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b680cd0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34106/0 (socket says 172.21.15.119:34106) 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6c8bb700 1 -- 172.21.15.119:0/4111230475 learned_addr learned my addr 172.21.15.119:0/4111230475 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d8bd700 1 -- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 msgr2=0x7f7b680cd0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d8bd700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b680cd0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d8bd700 1 -- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b6815e840 msgr2=0x7f7b681f61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d8bd700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b6815e840 0x7f7b681f61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d8bd700 1 -- 172.21.15.119:0/4111230475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b64009b30 con 0x7f7b680cd620 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6c8bb700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b680cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.857+0000 7f7b6d8bd700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b680cd620 0x7f7b680af160 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000e7f0 tx=0x7f7b6000eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.861+0000 7f7b5dffb700 1 -- 172.21.15.119:0/4111230475 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60013680 con 0x7f7b680cd620 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.861+0000 7f7b5dffb700 1 -- 172.21.15.119:0/4111230475 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b60013800 con 0x7f7b680cd620 2024-09-13T16:31:34.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.861+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b680a84b0 con 0x7f7b680cd620 2024-09-13T16:31:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.861+0000 7f7b5dffb700 1 -- 172.21.15.119:0/4111230475 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b60013bb0 con 0x7f7b680cd620 2024-09-13T16:31:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.861+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b680ab9f0 con 0x7f7b680cd620 2024-09-13T16:31:34.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.861+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b68067760 con 0x7f7b680cd620 2024-09-13T16:31:34.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.861+0000 7f7b5dffb700 1 -- 172.21.15.119:0/4111230475 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7b60013d10 con 0x7f7b680cd620 2024-09-13T16:31:34.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.865+0000 7f7b5dffb700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b4c066730 0x7f7b4c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:34.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.865+0000 7f7b5dffb700 1 -- 172.21.15.119:0/4111230475 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7b60083c40 con 0x7f7b680cd620 2024-09-13T16:31:34.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.865+0000 7f7b6d0bc700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b4c066730 0x7f7b4c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:34.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.865+0000 7f7b6d0bc700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b4c066730 0x7f7b4c068bf0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b58001820 tx=0x7f7b58009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:34.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:34.869+0000 7f7b5dffb700 1 -- 172.21.15.119:0/4111230475 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7b60050dd0 con 0x7f7b680cd620 2024-09-13T16:31:35.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.077+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f7b68071210 con 0x7f7b680cd620 2024-09-13T16:31:35.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.077+0000 7f7b5dffb700 1 -- 172.21.15.119:0/4111230475 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v56) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7b60017060 con 0x7f7b680cd620 2024-09-13T16:31:35.081 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:31:35.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b4c066730 msgr2=0x7f7b4c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:35.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b4c066730 0x7f7b4c068bf0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f7b58001820 tx=0x7f7b58009040 comp rx=0 tx=0).stop 2024-09-13T16:31:35.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b680cd620 msgr2=0x7f7b680af160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:35.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b680cd620 0x7f7b680af160 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f7b6000e7f0 tx=0x7f7b6000eb00 comp rx=0 tx=0).stop 2024-09-13T16:31:35.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 shutdown_connections 2024-09-13T16:31:35.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7b4c066730 0x7f7b4c068bf0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:35.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7b680cd620 0x7f7b680af160 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:35.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7b68165180 0x7f7b680cd0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:35.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 --2- 172.21.15.119:0/4111230475 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7b6815e840 0x7f7b681f61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:35.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 >> 172.21.15.119:0/4111230475 conn(0x7f7b6806a350 msgr2=0x7f7b6806cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:35.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 shutdown_connections 2024-09-13T16:31:35.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:35.081+0000 7f7b6f320700 1 -- 172.21.15.119:0/4111230475 wait complete. 2024-09-13T16:31:35.096 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:31:35.096 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:31:35.096 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:31:35.096 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:31:35.097 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:35.097 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:31:35.097 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:31:35.097 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:31:35.097 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:35.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.097 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:31:35.105 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:35.106 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42246sec preferred_lft 42246sec 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.107 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:35.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.108 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:31:35.121 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:31:35.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.123 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:31:35.175 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:31:35.175 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:31:35.175 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:31:35.175 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:31:35.176 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:31:35.176 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:31:35.176 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:31:35.176 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:31:35.176 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.176 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:31:35.176 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:31:35.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:31:35.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:31:35.177 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:31:35.177 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:31:35.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.215 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:31:35.220 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:31:35.220 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:31:35.221 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:35.221 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.221 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:31:35.235 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:31:35.235 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:31:35.235 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:35.241 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:35.241 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:31:35.241 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:35.246 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:35.260 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:35.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:35.274 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:35.274 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:35.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.275 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:35.329 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:31:35.329 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:31:35.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:35.348 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:35.349 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:35.407 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:35.408 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:35.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:35.409 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:35.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:31:35.417+0000 7f7ff1f80640 -1 init, newargv = 0x55be677d15e0 newargc=15 2024-09-13T16:31:35.422 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[47236]: starting ceph client 2024-09-13T16:31:35.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[47236]: starting fuse 2024-09-13T16:31:36.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:36.536 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:36.536 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:36.550 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:36.550 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:36.551 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:36.551 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:36.558 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:31:36.558 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:31:36.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:31:36.558 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:31:36.558 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:31:36.558 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:31:36.558 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:36.559 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:31:36.560 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:31:36.682 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.47236.asok 2024-09-13T16:31:36.867 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47236.asok 2024-09-13T16:31:36.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:36.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47236.asok status 2024-09-13T16:31:36.973 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:31:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:31:36.973 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "47236", 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 8031, 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:31:36.974 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 8031 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2185729602 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2185729602 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:36.975 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.8031 192.168.0.1:0/2185729602", 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/2185729602", 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 56, 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 95, 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:31:36.976 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:36.983 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:36.983 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:31:36.983 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:36.989 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:31:36.989 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:36.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:36.990 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:37.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:37.047 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir dir) 2024-09-13T16:31:37.069 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:37.069 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch dir/file) 2024-09-13T16:31:37.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:37.082 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir testdir) 2024-09-13T16:31:37.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:37.135 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec ln dir/file testdir/hardlink) 2024-09-13T16:31:37.157 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:31:37.157 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:31:37.157 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:31:37.157 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:37.157 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/dir").st_ino) 2024-09-13T16:31:37.158 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:31:37.227 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627776 2024-09-13T16:31:37.414 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:31:37.414 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:37.463 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:37.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.577+0000 7f982a91e700 1 Processor -- start 2024-09-13T16:31:37.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.577+0000 7f982a91e700 1 -- start start 2024-09-13T16:31:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.577+0000 7f982a91e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 0x7f98240cfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.577+0000 7f982a91e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982415aa80 con 0x7f98241576c0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.577+0000 7f982a91e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240cf790 con 0x7f982406d3e0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.577+0000 7f982a91e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f982406be40 con 0x7f982406ba40 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9823fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f98241576c0 0x7f982406ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45590/0 (socket says 172.21.15.119:45590) 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9823fff700 1 -- 172.21.15.119:0/4002650234 learned_addr learned my addr 172.21.15.119:0/4002650234 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 0x7f98240cfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197739188 0 0) 0x7f982415aa80 con 0x7f98241576c0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100035d0 con 0x7f98241576c0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2596050842 0 0) 0x7f98240cf790 con 0x7f982406d3e0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f982415aa80 con 0x7f982406d3e0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930696341 0 0) 0x7f98100035d0 con 0x7f98241576c0 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 msgr2=0x7f98240cfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:37.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 0x7f98240cfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 >> v1:172.21.15.145:6789/0 conn(0x7f982406d3e0 legacy=0x7f982406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982415afb0 con 0x7f98241576c0 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f9814001670 con 0x7f98241576c0 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 0x7f98240cfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 >> v1:172.21.15.119:6789/0 conn(0x7f98241576c0 legacy=0x7f982406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9810003be0 0x7f9810003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 0x7f9810004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9810004db0 0x7f9810005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982406be40 con 0x7f9810004db0 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981000bb40 con 0x7f9810003be0 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f981000bd00 con 0x7f982406ba40 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9810004db0 0x7f9810005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9823fff700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9810003be0 0x7f9810003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f981bfff700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 0x7f9810004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 -- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 msgr2=0x7f9810004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982406ba40 0x7f9810004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 -- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9810003be0 msgr2=0x7f9810003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9810003be0 0x7f9810003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 -- 172.21.15.119:0/4002650234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f981c009a70 con 0x7f9810004db0 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9823fff700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9810003be0 0x7f9810003fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9810004db0 0x7f9810005230 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f981c00e350 tx=0x7f98100058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d90baacd35c9f707 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c00edf0 con 0x7f9810004db0 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c00f040 con 0x7f9810004db0 2024-09-13T16:31:37.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f98237fe700 1 -- 172.21.15.119:0/4002650234 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981c0146a0 con 0x7f9810004db0 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f982a91e700 1 -- 172.21.15.119:0/4002650234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9810004db0 msgr2=0x7f9810005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f982a91e700 1 --2- 172.21.15.119:0/4002650234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9810004db0 0x7f9810005230 secure :-1 s=READY pgs=1280 cs=0 l=1 rev1=1 crypto rx=0x7f981c00e350 tx=0x7f98100058a0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f9828ebb700 1 -- 172.21.15.119:0/4002650234 reap_dead start 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f982a91e700 1 -- 172.21.15.119:0/4002650234 shutdown_connections 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.581+0000 7f982a91e700 1 -- 172.21.15.119:0/4002650234 >> 172.21.15.119:0/4002650234 conn(0x7f98240b3fd0 msgr2=0x7f98240b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 -- 172.21.15.119:0/4002650234 shutdown_connections 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 -- 172.21.15.119:0/4002650234 wait complete. 2024-09-13T16:31:37.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 Processor -- start 2024-09-13T16:31:37.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 -- start start 2024-09-13T16:31:37.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824149c50 0x7f98241f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982414a030 0x7f9824158c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98241591b0 0x7f982414e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9824159630 con 0x7f9824149c50 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9824147710 con 0x7f98241591b0 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.585+0000 7f982a91e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9824145d20 con 0x7f982414a030 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824149c50 0x7f98241f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824149c50 0x7f98241f66a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38358/0 (socket says 172.21.15.119:38358) 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 -- 172.21.15.119:0/920796278 learned_addr learned my addr 172.21.15.119:0/920796278 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f981bfff700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982414a030 0x7f9824158c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9828ebb700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98241591b0 0x7f982414e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 -- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982414a030 msgr2=0x7f9824158c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982414a030 0x7f9824158c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 -- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98241591b0 msgr2=0x7f982414e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98241591b0 0x7f982414e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 -- 172.21.15.119:0/920796278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824146b90 con 0x7f9824149c50 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f981bfff700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982414a030 0x7f9824158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f9823fff700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824149c50 0x7f98241f66a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f981400e7f0 tx=0x7f981400ebb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f98217fa700 1 -- 172.21.15.119:0/920796278 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400c5f0 con 0x7f9824149c50 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f98217fa700 1 -- 172.21.15.119:0/920796278 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981400c750 con 0x7f9824149c50 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f98217fa700 1 -- 172.21.15.119:0/920796278 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981400cac0 con 0x7f9824149c50 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9824146e70 con 0x7f9824149c50 2024-09-13T16:31:37.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9824147430 con 0x7f9824149c50 2024-09-13T16:31:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98240cfc20 con 0x7f9824149c50 2024-09-13T16:31:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.589+0000 7f98217fa700 1 -- 172.21.15.119:0/920796278 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f981400cc20 con 0x7f9824149c50 2024-09-13T16:31:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.593+0000 7f98217fa700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9804066730 0x7f9804068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.593+0000 7f98217fa700 1 -- 172.21.15.119:0/920796278 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9814084070 con 0x7f9824149c50 2024-09-13T16:31:37.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.593+0000 7f981bfff700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9804066730 0x7f9804068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.593+0000 7f981bfff700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9804066730 0x7f9804068bf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9824156b90 tx=0x7f980c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:37.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.601+0000 7f98217fa700 1 -- 172.21.15.119:0/920796278 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9814051330 con 0x7f9824149c50 2024-09-13T16:31:37.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.777+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98241447d0 con 0x7f9824149c50 2024-09-13T16:31:37.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f98217fa700 1 -- 172.21.15.119:0/920796278 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16777 (secure 0 0 0) 0x7f9814054950 con 0x7f9824149c50 2024-09-13T16:31:37.784 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:37.784 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":95,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:31:29.963385+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9804066730 msgr2=0x7f9804068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9804066730 0x7f9804068bf0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f9824156b90 tx=0x7f980c009040 comp rx=0 tx=0).stop 2024-09-13T16:31:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824149c50 msgr2=0x7f98241f66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824149c50 0x7f98241f66a0 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7f981400e7f0 tx=0x7f981400ebb0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 shutdown_connections 2024-09-13T16:31:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9804066730 0x7f9804068bf0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98241591b0 0x7f982414e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f982414a030 0x7f9824158c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 --2- 172.21.15.119:0/920796278 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9824149c50 0x7f98241f66a0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 >> 172.21.15.119:0/920796278 conn(0x7f98240b3fd0 msgr2=0x7f982409aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:37.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 shutdown_connections 2024-09-13T16:31:37.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.781+0000 7f982a91e700 1 -- 172.21.15.119:0/920796278 wait complete. 2024-09-13T16:31:37.799 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:31:27.299109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:31:28.044326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:31:37.799 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:31:37.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c95aa8700 1 Processor -- start 2024-09-13T16:31:37.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c95aa8700 1 -- start start 2024-09-13T16:31:37.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c95aa8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 0x7f6c9006ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c95aa8700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900ce880 con 0x7f6c9006b2d0 2024-09-13T16:31:37.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c95aa8700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900ceb60 con 0x7f6c9006be20 2024-09-13T16:31:37.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c95aa8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900cee40 con 0x7f6c9015cfb0 2024-09-13T16:31:37.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c86ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6c9006b2d0 0x7f6c9006b710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45602/0 (socket says 172.21.15.119:45602) 2024-09-13T16:31:37.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c86ffd700 1 -- 172.21.15.119:0/444857421 learned_addr learned my addr 172.21.15.119:0/444857421 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:37.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c8f7fe700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 0x7f6c9006ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231803797 0 0) 0x7f6c900ceb60 con 0x7f6c9006be20 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c7c0035d0 con 0x7f6c9006be20 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123134330 0 0) 0x7f6c900ce880 con 0x7f6c9006b2d0 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c900ceb60 con 0x7f6c9006b2d0 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.921+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2352864500 0 0) 0x7f6c900ceb60 con 0x7f6c9006b2d0 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 msgr2=0x7f6c9006ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 0x7f6c9006ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 >> v1:172.21.15.145:6789/0 conn(0x7f6c9006be20 legacy=0x7f6c900d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900d0e40 con 0x7f6c9006b2d0 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f6c78001670 con 0x7f6c9006b2d0 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 >> v1:172.21.15.119:6789/0 conn(0x7f6c9006b2d0 legacy=0x7f6c9006b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c7c004c40 0x7f6c7c005040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c7c003810 0x7f6c7c003c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 0x7f6c7c0041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900cee40 con 0x7f6c7c003810 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c00bae0 con 0x7f6c7c004c40 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c00bc70 con 0x7f6c9015cfb0 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c86ffd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c7c003810 0x7f6c7c003c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8f7fe700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c7c004c40 0x7f6c7c005040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8ffff700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 0x7f6c7c0041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c86ffd700 1 -- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 msgr2=0x7f6c7c0041d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c86ffd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9015cfb0 0x7f6c7c0041d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c86ffd700 1 -- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c7c004c40 msgr2=0x7f6c7c005040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c86ffd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c7c004c40 0x7f6c7c005040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c86ffd700 1 -- 172.21.15.119:0/444857421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900ceb60 con 0x7f6c7c003810 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c86ffd700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c7c003810 0x7f6c7c003c90 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800c210 tx=0x7f6c7c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27d4afab598a02f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:37.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7800cd10 con 0x7f6c7c003810 2024-09-13T16:31:37.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7800ce70 con 0x7f6c7c003810 2024-09-13T16:31:37.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8effd700 1 -- 172.21.15.119:0/444857421 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78005ba0 con 0x7f6c7c003810 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c95aa8700 1 -- 172.21.15.119:0/444857421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c7c003810 msgr2=0x7f6c7c003c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c95aa8700 1 --2- 172.21.15.119:0/444857421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c7c003810 0x7f6c7c003c90 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800c210 tx=0x7f6c7c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c8ffff700 1 -- 172.21.15.119:0/444857421 reap_dead start 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c95aa8700 1 -- 172.21.15.119:0/444857421 shutdown_connections 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c95aa8700 1 -- 172.21.15.119:0/444857421 >> 172.21.15.119:0/444857421 conn(0x7f6c900d2da0 msgr2=0x7f6c900d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c95aa8700 1 -- 172.21.15.119:0/444857421 shutdown_connections 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.925+0000 7f6c95aa8700 1 -- 172.21.15.119:0/444857421 wait complete. 2024-09-13T16:31:37.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 Processor -- start 2024-09-13T16:31:37.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 -- start start 2024-09-13T16:31:37.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9014c560 0x7f6c900c66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c9014c940 0x7f6c900c6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c90155c00 0x7f6c90156080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900d1220 con 0x7f6c9014c940 2024-09-13T16:31:37.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c901f29a0 con 0x7f6c9014c560 2024-09-13T16:31:37.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c95aa8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900b77d0 con 0x7f6c90155c00 2024-09-13T16:31:37.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c9014c940 0x7f6c900c6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c9014c940 0x7f6c900c6c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38372/0 (socket says 172.21.15.119:38372) 2024-09-13T16:31:37.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 -- 172.21.15.119:0/1494015276 learned_addr learned my addr 172.21.15.119:0/1494015276 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:37.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c8f7fe700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9014c560 0x7f6c900c66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c8ffff700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c90155c00 0x7f6c90156080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 -- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c90155c00 msgr2=0x7f6c90156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c90155c00 0x7f6c90156080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 -- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9014c560 msgr2=0x7f6c900c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:37.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9014c560 0x7f6c900c66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:37.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c86ffd700 1 -- 172.21.15.119:0/1494015276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900b7ab0 con 0x7f6c9014c940 2024-09-13T16:31:37.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.929+0000 7f6c8ffff700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c90155c00 0x7f6c90156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:37.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c86ffd700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c9014c940 0x7f6c900c6c30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005810 tx=0x7f6c78001980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:37.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c8cff9700 1 -- 172.21.15.119:0/1494015276 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78000ed0 con 0x7f6c9014c940 2024-09-13T16:31:37.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c8cff9700 1 -- 172.21.15.119:0/1494015276 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c780070d0 con 0x7f6c9014c940 2024-09-13T16:31:37.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c8cff9700 1 -- 172.21.15.119:0/1494015276 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78007480 con 0x7f6c9014c940 2024-09-13T16:31:37.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c900b7d90 con 0x7f6c9014c940 2024-09-13T16:31:37.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c900b8350 con 0x7f6c9014c940 2024-09-13T16:31:37.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c900ce880 con 0x7f6c9014c940 2024-09-13T16:31:37.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.933+0000 7f6c8cff9700 1 -- 172.21.15.119:0/1494015276 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6c780075e0 con 0x7f6c9014c940 2024-09-13T16:31:37.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.937+0000 7f6c8cff9700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c70066760 0x7f6c70068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:37.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.937+0000 7f6c8cff9700 1 -- 172.21.15.119:0/1494015276 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6c78085270 con 0x7f6c9014c940 2024-09-13T16:31:37.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.937+0000 7f6c8f7fe700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c70066760 0x7f6c70068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:37.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.937+0000 7f6c8f7fe700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c70066760 0x7f6c70068c20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009b90 tx=0x7f6c80009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:37.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:37.941+0000 7f6c8cff9700 1 -- 172.21.15.119:0/1494015276 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6c780525b0 con 0x7f6c9014c940 2024-09-13T16:31:38.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.113+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c900afdf0 con 0x7f6c9014c940 2024-09-13T16:31:38.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.113+0000 7f6c8cff9700 1 -- 172.21.15.119:0/1494015276 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c78055bd0 con 0x7f6c9014c940 2024-09-13T16:31:38.117 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c70066760 msgr2=0x7f6c70068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c70066760 0x7f6c70068c20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009b90 tx=0x7f6c80009240 comp rx=0 tx=0).stop 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c9014c940 msgr2=0x7f6c900c6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c9014c940 0x7f6c900c6c30 secure :-1 s=READY pgs=1284 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005810 tx=0x7f6c78001980 comp rx=0 tx=0).stop 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 shutdown_connections 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c70066760 0x7f6c70068c20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c90155c00 0x7f6c90156080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c9014c940 0x7f6c900c6c30 unknown :-1 s=CLOSED pgs=1284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 --2- 172.21.15.119:0/1494015276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9014c560 0x7f6c900c66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 >> 172.21.15.119:0/1494015276 conn(0x7f6c900d2da0 msgr2=0x7f6c900b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:38.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 shutdown_connections 2024-09-13T16:31:38.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.117+0000 7f6c95aa8700 1 -- 172.21.15.119:0/1494015276 wait complete. 2024-09-13T16:31:38.131 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:31:38.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.241+0000 7ff481478700 1 Processor -- start 2024-09-13T16:31:38.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.241+0000 7ff481478700 1 -- start start 2024-09-13T16:31:38.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff47c0ad9a0 0x7ff47c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c0b1860 0x7ff47c06c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff47c06c890 0x7ff47c06bac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47c0b2fd0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff47c15a520 con 0x7ff47c06c890 2024-09-13T16:31:38.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c15a320 con 0x7ff47c0b1860 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff47c0ad9a0 0x7ff47c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff47c0ad9a0 0x7ff47c159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38388/0 (socket says 172.21.15.119:38388) 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 -- 172.21.15.119:0/2107552290 learned_addr learned my addr 172.21.15.119:0/2107552290 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47b7fe700 1 --2- 172.21.15.119:0/2107552290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c0b1860 0x7ff47c06c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff480c77700 1 --2- 172.21.15.119:0/2107552290 >> v2:172.21.15.145:3300/0 conn(0x7ff47c06c890 0x7ff47c06bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 -- 172.21.15.119:0/2107552290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c0b1860 msgr2=0x7ff47c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 --2- 172.21.15.119:0/2107552290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c0b1860 0x7ff47c06c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 -- 172.21.15.119:0/2107552290 >> v2:172.21.15.145:3300/0 conn(0x7ff47c06c890 msgr2=0x7ff47c06bac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 --2- 172.21.15.119:0/2107552290 >> v2:172.21.15.145:3300/0 conn(0x7ff47c06c890 0x7ff47c06bac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 -- 172.21.15.119:0/2107552290 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47c0b1dd0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47bfff700 1 --2- 172.21.15.119:0/2107552290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c159de0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0ae460 tx=0x7ff47400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d8d602f033fd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47affd700 1 -- 172.21.15.119:0/2107552290 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47400e070 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47affd700 1 -- 172.21.15.119:0/2107552290 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47400b910 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff47affd700 1 -- 172.21.15.119:0/2107552290 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47400bcc0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- 172.21.15.119:0/2107552290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 msgr2=0x7ff47c159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 --2- 172.21.15.119:0/2107552290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c159de0 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff47c0ae460 tx=0x7ff47400b330 comp rx=0 tx=0).stop 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- 172.21.15.119:0/2107552290 shutdown_connections 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 --2- 172.21.15.119:0/2107552290 >> v2:172.21.15.145:3300/0 conn(0x7ff47c06c890 0x7ff47c06bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 --2- 172.21.15.119:0/2107552290 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c0b1860 0x7ff47c06c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 --2- 172.21.15.119:0/2107552290 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c159de0 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- 172.21.15.119:0/2107552290 >> 172.21.15.119:0/2107552290 conn(0x7ff47c0b0980 msgr2=0x7ff47c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- 172.21.15.119:0/2107552290 shutdown_connections 2024-09-13T16:31:38.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.245+0000 7ff481478700 1 -- 172.21.15.119:0/2107552290 wait complete. 2024-09-13T16:31:38.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 Processor -- start 2024-09-13T16:31:38.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 -- start start 2024-09-13T16:31:38.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c06c890 0x7ff47c1f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff47c0b1860 0x7ff47c157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c1f60d0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c158270 con 0x7ff47c0b1860 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47c1f5a80 con 0x7ff47c06c890 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c157720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38390/0 (socket says 172.21.15.119:38390) 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 -- 172.21.15.119:0/3328289550 learned_addr learned my addr 172.21.15.119:0/3328289550 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47bfff700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c06c890 0x7ff47c1f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff480c77700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff47c0b1860 0x7ff47c157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 -- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c06c890 msgr2=0x7ff47c1f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c06c890 0x7ff47c1f6560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 -- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff47c0b1860 msgr2=0x7ff47c157c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff47c0b1860 0x7ff47c157c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 -- 172.21.15.119:0/3328289550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff474009b30 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff47b7fe700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c157720 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ff46400e7f0 tx=0x7ff46400eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff478ff9700 1 -- 172.21.15.119:0/3328289550 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff464009580 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff478ff9700 1 -- 172.21.15.119:0/3328289550 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff464004020 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff47c1f5d60 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff478ff9700 1 -- 172.21.15.119:0/3328289550 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4640043b0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.249+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff47c158550 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.253+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff47c1f6240 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.253+0000 7ff478ff9700 1 -- 172.21.15.119:0/3328289550 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff4640036c0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.253+0000 7ff478ff9700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff460066730 0x7ff460068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.253+0000 7ff478ff9700 1 -- 172.21.15.119:0/3328289550 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff464083d60 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.253+0000 7ff47bfff700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff460066730 0x7ff460068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.253+0000 7ff47bfff700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff460066730 0x7ff460068bf0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff474000f80 tx=0x7ff474009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.261+0000 7ff478ff9700 1 -- 172.21.15.119:0/3328289550 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff464050ef0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.425+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff47c159de0 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff478ff9700 1 -- 172.21.15.119:0/3328289550 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff464017020 con 0x7ff47c0ad9a0 2024-09-13T16:31:38.432 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:31:38.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff460066730 msgr2=0x7ff460068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff460066730 0x7ff460068bf0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7ff474000f80 tx=0x7ff474009000 comp rx=0 tx=0).stop 2024-09-13T16:31:38.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 msgr2=0x7ff47c157720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c157720 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ff46400e7f0 tx=0x7ff46400eb00 comp rx=0 tx=0).stop 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 shutdown_connections 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff460066730 0x7ff460068bf0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff47c0b1860 0x7ff47c157c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff47c0ad9a0 0x7ff47c157720 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 --2- 172.21.15.119:0/3328289550 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff47c06c890 0x7ff47c1f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 >> 172.21.15.119:0/3328289550 conn(0x7ff47c0b0980 msgr2=0x7ff47c0ae920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 shutdown_connections 2024-09-13T16:31:38.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.429+0000 7ff481478700 1 -- 172.21.15.119:0/3328289550 wait complete. 2024-09-13T16:31:38.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:31:38.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.569+0000 7ff18cb83700 1 Processor -- start 2024-09-13T16:31:38.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.569+0000 7ff18cb83700 1 -- start start 2024-09-13T16:31:38.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff1880d17e0 0x7ff188167bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1880d0df0 con 0x7ff18815cc00 2024-09-13T16:31:38.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1880cdb00 con 0x7ff1880ce8c0 2024-09-13T16:31:38.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1880cdde0 con 0x7ff1880d17e0 2024-09-13T16:31:38.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff187fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff1880d17e0 0x7ff188167bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff1877fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff18815cc00 0x7ff1880d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45610/0 (socket says 172.21.15.119:45610) 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff1877fe700 1 -- 172.21.15.119:0/1019020263 learned_addr learned my addr 172.21.15.119:0/1019020263 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff187fff700 1 -- 172.21.15.119:0/1019020263 >> v1:172.21.15.145:6789/0 conn(0x7ff1880ce8c0 legacy=0x7ff1880d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff187fff700 1 -- 172.21.15.119:0/1019020263 >> v1:172.21.15.119:6789/0 conn(0x7ff18815cc00 legacy=0x7ff1880d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff187fff700 1 -- 172.21.15.119:0/1019020263 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1880ce0c0 con 0x7ff1880d17e0 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff187fff700 1 --2- 172.21.15.119:0/1019020263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1880d17e0 0x7ff188167bb0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff180009d90 tx=0x7ff18000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3037edd0906d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff186ffd700 1 -- 172.21.15.119:0/1019020263 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18000e070 con 0x7ff1880d17e0 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff186ffd700 1 -- 172.21.15.119:0/1019020263 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff18000bb90 con 0x7ff1880d17e0 2024-09-13T16:31:38.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff186ffd700 1 -- 172.21.15.119:0/1019020263 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180003a40 con 0x7ff1880d17e0 2024-09-13T16:31:38.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- 172.21.15.119:0/1019020263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1880d17e0 msgr2=0x7ff188167bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 --2- 172.21.15.119:0/1019020263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1880d17e0 0x7ff188167bb0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7ff180009d90 tx=0x7ff18000b330 comp rx=0 tx=0).stop 2024-09-13T16:31:38.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- 172.21.15.119:0/1019020263 shutdown_connections 2024-09-13T16:31:38.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 --2- 172.21.15.119:0/1019020263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1880d17e0 0x7ff188167bb0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- 172.21.15.119:0/1019020263 >> 172.21.15.119:0/1019020263 conn(0x7ff18808c080 msgr2=0x7ff1880d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:38.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- 172.21.15.119:0/1019020263 shutdown_connections 2024-09-13T16:31:38.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- 172.21.15.119:0/1019020263 wait complete. 2024-09-13T16:31:38.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 Processor -- start 2024-09-13T16:31:38.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.573+0000 7ff18cb83700 1 -- start start 2024-09-13T16:31:38.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1880ce8c0 0x7ff188151e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1880d17e0 0x7ff18809c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff18815cc00 0x7ff1880c8770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880cae80 con 0x7ff18815cc00 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880cbca0 con 0x7ff1880ce8c0 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1880cb090 con 0x7ff1880d17e0 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff187fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff18815cc00 0x7ff1880c8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff187fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff18815cc00 0x7ff1880c8770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38408/0 (socket says 172.21.15.119:38408) 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff187fff700 1 -- 172.21.15.119:0/2988818728 learned_addr learned my addr 172.21.15.119:0/2988818728 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff1877fe700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1880ce8c0 0x7ff188151e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff17effd700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1880d17e0 0x7ff18809c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff1877fe700 1 -- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1880d17e0 msgr2=0x7ff18809c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff1877fe700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1880d17e0 0x7ff18809c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff1877fe700 1 -- 172.21.15.119:0/2988818728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff18815cc00 msgr2=0x7ff1880c8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff1877fe700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff18815cc00 0x7ff1880c8770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff1877fe700 1 -- 172.21.15.119:0/2988818728 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff180009a70 con 0x7ff1880ce8c0 2024-09-13T16:31:38.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff1877fe700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1880ce8c0 0x7ff188151e70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff178000f60 tx=0x7ff17800c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff184ff9700 1 -- 172.21.15.119:0/2988818728 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178005b60 con 0x7ff1880ce8c0 2024-09-13T16:31:38.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff184ff9700 1 -- 172.21.15.119:0/2988818728 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff178005cc0 con 0x7ff1880ce8c0 2024-09-13T16:31:38.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1880cb370 con 0x7ff1880ce8c0 2024-09-13T16:31:38.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff184ff9700 1 -- 172.21.15.119:0/2988818728 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1780072f0 con 0x7ff1880ce8c0 2024-09-13T16:31:38.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.577+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1880cb930 con 0x7ff1880ce8c0 2024-09-13T16:31:38.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.581+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18809f580 con 0x7ff1880ce8c0 2024-09-13T16:31:38.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.581+0000 7ff184ff9700 1 -- 172.21.15.119:0/2988818728 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff1780076e0 con 0x7ff1880ce8c0 2024-09-13T16:31:38.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.581+0000 7ff184ff9700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff168066730 0x7ff168068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.581+0000 7ff184ff9700 1 -- 172.21.15.119:0/2988818728 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff178083850 con 0x7ff1880ce8c0 2024-09-13T16:31:38.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.581+0000 7ff17effd700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff168066730 0x7ff168068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.585+0000 7ff17effd700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff168066730 0x7ff168068bf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff1880cc5c0 tx=0x7ff1700090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.589+0000 7ff184ff9700 1 -- 172.21.15.119:0/2988818728 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff178050960 con 0x7ff1880ce8c0 2024-09-13T16:31:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.761+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff18808c560 con 0x7ff1880ce8c0 2024-09-13T16:31:38.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.765+0000 7ff184ff9700 1 -- 172.21.15.119:0/2988818728 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff178053f80 con 0x7ff1880ce8c0 2024-09-13T16:31:38.768 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff168066730 msgr2=0x7ff168068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff168066730 0x7ff168068bf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7ff1880cc5c0 tx=0x7ff1700090e0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1880ce8c0 msgr2=0x7ff188151e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1880ce8c0 0x7ff188151e70 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7ff178000f60 tx=0x7ff17800c6e0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 shutdown_connections 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff168066730 0x7ff168068bf0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff18815cc00 0x7ff1880c8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1880d17e0 0x7ff18809c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 --2- 172.21.15.119:0/2988818728 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1880ce8c0 0x7ff188151e70 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 >> 172.21.15.119:0/2988818728 conn(0x7ff18808c080 msgr2=0x7ff18806c1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 shutdown_connections 2024-09-13T16:31:38.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.769+0000 7ff18cb83700 1 -- 172.21.15.119:0/2988818728 wait complete. 2024-09-13T16:31:38.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:31:38.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7afce37700 1 Processor -- start 2024-09-13T16:31:38.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7afce37700 1 -- start start 2024-09-13T16:31:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7afce37700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af815cc40 0x7f7af80ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7afce37700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7af80cd860 0x7f7af808bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7afce37700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7af815d020 con 0x7f7af808c4e0 2024-09-13T16:31:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7afce37700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7af808c990 con 0x7f7af80cd860 2024-09-13T16:31:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7afce37700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80b44c0 con 0x7f7af815cc40 2024-09-13T16:31:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af7fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7af808c4e0 0x7f7af8160720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45622/0 (socket says 172.21.15.119:45622) 2024-09-13T16:31:38.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af7fff700 1 -- 172.21.15.119:0/3882712941 learned_addr learned my addr 172.21.15.119:0/3882712941 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af6ffd700 1 --2- 172.21.15.119:0/3882712941 >> v2:172.21.15.119:3300/0 conn(0x7f7af80cd860 0x7f7af808bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af77fe700 1 --2- 172.21.15.119:0/3882712941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af815cc40 0x7f7af80ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af67fc700 1 -- 172.21.15.119:0/3882712941 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773527732 0 0) 0x7f7af815d020 con 0x7f7af808c4e0 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af67fc700 1 -- 172.21.15.119:0/3882712941 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ae40035d0 con 0x7f7af808c4e0 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af67fc700 1 -- 172.21.15.119:0/3882712941 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3540102973 0 0) 0x7f7ae40035d0 con 0x7f7af808c4e0 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af6ffd700 1 -- 172.21.15.119:0/3882712941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af815cc40 msgr2=0x7f7af80ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af6ffd700 1 --2- 172.21.15.119:0/3882712941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af815cc40 0x7f7af80ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af6ffd700 1 -- 172.21.15.119:0/3882712941 >> v1:172.21.15.119:6789/0 conn(0x7f7af808c4e0 legacy=0x7f7af8160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.897+0000 7f7af6ffd700 1 -- 172.21.15.119:0/3882712941 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7af80d0720 con 0x7f7af80cd860 2024-09-13T16:31:38.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7af6ffd700 1 --2- 172.21.15.119:0/3882712941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af80cd860 0x7f7af808bfa0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009880 tx=0x7f7ae0009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0cab18dd376c62d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7af67fc700 1 -- 172.21.15.119:0/3882712941 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0001670 con 0x7f7af80cd860 2024-09-13T16:31:38.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7af67fc700 1 -- 172.21.15.119:0/3882712941 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ae0004030 con 0x7f7af80cd860 2024-09-13T16:31:38.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7af67fc700 1 -- 172.21.15.119:0/3882712941 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ae00043a0 con 0x7f7af80cd860 2024-09-13T16:31:38.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 -- 172.21.15.119:0/3882712941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af80cd860 msgr2=0x7f7af808bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 --2- 172.21.15.119:0/3882712941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af80cd860 0x7f7af808bfa0 secure :-1 s=READY pgs=1290 cs=0 l=1 rev1=1 crypto rx=0x7f7ae0009880 tx=0x7f7ae0009b90 comp rx=0 tx=0).stop 2024-09-13T16:31:38.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 -- 172.21.15.119:0/3882712941 shutdown_connections 2024-09-13T16:31:38.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 --2- 172.21.15.119:0/3882712941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af80cd860 0x7f7af808bfa0 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 --2- 172.21.15.119:0/3882712941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af815cc40 0x7f7af80ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 -- 172.21.15.119:0/3882712941 >> 172.21.15.119:0/3882712941 conn(0x7f7af80b4640 msgr2=0x7f7af80b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:38.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 -- 172.21.15.119:0/3882712941 shutdown_connections 2024-09-13T16:31:38.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 -- 172.21.15.119:0/3882712941 wait complete. 2024-09-13T16:31:38.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 Processor -- start 2024-09-13T16:31:38.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.901+0000 7f7afce37700 1 -- start start 2024-09-13T16:31:38.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7af808c4e0 0x7f7af8153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af80cd860 0x7f7af8157e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af815cc40 0x7f7af815f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80cada0 con 0x7f7af815cc40 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7af81543a0 con 0x7f7af808c4e0 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7af80ca120 con 0x7f7af80cd860 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af815cc40 0x7f7af815f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af815cc40 0x7f7af815f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38436/0 (socket says 172.21.15.119:38436) 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 -- 172.21.15.119:0/1019398612 learned_addr learned my addr 172.21.15.119:0/1019398612 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af77fe700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7af808c4e0 0x7f7af8153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af6ffd700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af80cd860 0x7f7af8157e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 -- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af80cd860 msgr2=0x7f7af8157e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af80cd860 0x7f7af8157e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 -- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7af808c4e0 msgr2=0x7f7af8153e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7af808c4e0 0x7f7af8153e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 -- 172.21.15.119:0/1019398612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ae00094b0 con 0x7f7af815cc40 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7af7fff700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af815cc40 0x7f7af815f260 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c500 tx=0x7f7af000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7aeffff700 1 -- 172.21.15.119:0/1019398612 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0004ca0 con 0x7f7af815cc40 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7aeffff700 1 -- 172.21.15.119:0/1019398612 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7af0004e00 con 0x7f7af815cc40 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7af80ca400 con 0x7f7af815cc40 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7aeffff700 1 -- 172.21.15.119:0/1019398612 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7af0007310 con 0x7f7af815cc40 2024-09-13T16:31:38.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7af80ca9c0 con 0x7f7af815cc40 2024-09-13T16:31:38.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7af808cd70 con 0x7f7af815cc40 2024-09-13T16:31:38.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7aeffff700 1 -- 172.21.15.119:0/1019398612 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7af0007470 con 0x7f7af815cc40 2024-09-13T16:31:38.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.905+0000 7f7aeffff700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ad8066760 0x7f7ad8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:38.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.909+0000 7f7aeffff700 1 -- 172.21.15.119:0/1019398612 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7af0083ec0 con 0x7f7af815cc40 2024-09-13T16:31:38.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.909+0000 7f7af77fe700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ad8066760 0x7f7ad8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:38.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.909+0000 7f7af77fe700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ad8066760 0x7f7ad8068c20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7af81f5fa0 tx=0x7f7ae800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:38.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:38.917+0000 7f7aeffff700 1 -- 172.21.15.119:0/1019398612 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7af00511c0 con 0x7f7af815cc40 2024-09-13T16:31:39.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.097+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7af80d1540 con 0x7f7af815cc40 2024-09-13T16:31:39.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.097+0000 7f7aeffff700 1 -- 172.21.15.119:0/1019398612 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v95) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7af0007720 con 0x7f7af815cc40 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ad8066760 msgr2=0x7f7ad8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ad8066760 0x7f7ad8068c20 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f7af81f5fa0 tx=0x7f7ae800b040 comp rx=0 tx=0).stop 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af815cc40 msgr2=0x7f7af815f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af815cc40 0x7f7af815f260 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f7af000c500 tx=0x7f7af000c8c0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 shutdown_connections 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7ad8066760 0x7f7ad8068c20 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7af815cc40 0x7f7af815f260 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7af80cd860 0x7f7af8157e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 --2- 172.21.15.119:0/1019398612 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7af808c4e0 0x7f7af8153e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 >> 172.21.15.119:0/1019398612 conn(0x7f7af80b4640 msgr2=0x7f7af80b31e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:39.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 shutdown_connections 2024-09-13T16:31:39.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.101+0000 7f7afce37700 1 -- 172.21.15.119:0/1019398612 wait complete. 2024-09-13T16:31:39.107 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:31:39.117 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:31:39.117 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:31:39.125 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:31:39.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb5a53b5700 1 Processor -- start 2024-09-13T16:31:39.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb5a53b5700 1 -- start start 2024-09-13T16:31:39.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb5a53b5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb5a00d1110 0x7fb5a00d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb5a53b5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00c6ab0 con 0x7fb5a015ccd0 2024-09-13T16:31:39.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb5a53b5700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5a00d1a90 con 0x7fb5a00c6f40 2024-09-13T16:31:39.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb5a53b5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5a00cea10 con 0x7fb5a00d1110 2024-09-13T16:31:39.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb597fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb5a00d1110 0x7fb5a00d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb59ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb5a015ccd0 0x7fb5a00d0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45626/0 (socket says 172.21.15.119:45626) 2024-09-13T16:31:39.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb59ffff700 1 -- 172.21.15.119:0/1233338203 learned_addr learned my addr 172.21.15.119:0/1233338203 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb597fff700 1 -- 172.21.15.119:0/1233338203 >> v1:172.21.15.145:6789/0 conn(0x7fb5a00c6f40 legacy=0x7fb5a00c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb597fff700 1 -- 172.21.15.119:0/1233338203 >> v1:172.21.15.119:6789/0 conn(0x7fb5a015ccd0 legacy=0x7fb5a00d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb597fff700 1 -- 172.21.15.119:0/1233338203 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a00ce170 con 0x7fb5a00d1110 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb59f7fe700 1 -- 172.21.15.119:0/1233338203 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197476183 0 0) 0x7fb5a00c6ab0 con 0x7fb5a015ccd0 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb597fff700 1 --2- 172.21.15.119:0/1233338203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a00d1550 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb588009850 tx=0x7fb588009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9640cce017943e6a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb59f7fe700 1 -- 172.21.15.119:0/1233338203 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588011830 con 0x7fb5a00d1110 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb59f7fe700 1 -- 172.21.15.119:0/1233338203 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb588011990 con 0x7fb5a00d1110 2024-09-13T16:31:39.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.233+0000 7fb59f7fe700 1 -- 172.21.15.119:0/1233338203 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb588011d20 con 0x7fb5a00d1110 2024-09-13T16:31:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- 172.21.15.119:0/1233338203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 msgr2=0x7fb5a00d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/1233338203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a00d1550 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7fb588009850 tx=0x7fb588009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- 172.21.15.119:0/1233338203 shutdown_connections 2024-09-13T16:31:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/1233338203 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a00d1550 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- 172.21.15.119:0/1233338203 >> 172.21.15.119:0/1233338203 conn(0x7fb5a00b4640 msgr2=0x7fb5a00b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:39.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- 172.21.15.119:0/1233338203 shutdown_connections 2024-09-13T16:31:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- 172.21.15.119:0/1233338203 wait complete. 2024-09-13T16:31:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 Processor -- start 2024-09-13T16:31:39.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- start start 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5a00c6f40 0x7fb5a00b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a015ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5a015ccd0 0x7fb5a015ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a01f2230 con 0x7fb5a00d1110 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a00caf90 con 0x7fb5a015ccd0 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.237+0000 7fb5a53b5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a015f4c0 con 0x7fb5a00c6f40 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a015ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a015ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38464/0 (socket says 172.21.15.119:38464) 2024-09-13T16:31:39.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 -- 172.21.15.119:0/3596570799 learned_addr learned my addr 172.21.15.119:0/3596570799 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:39.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb59ffff700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5a00c6f40 0x7fb5a00b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb5a4bb4700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5a015ccd0 0x7fb5a015ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 -- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5a00c6f40 msgr2=0x7fb5a00b5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5a00c6f40 0x7fb5a00b5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 -- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5a015ccd0 msgr2=0x7fb5a015ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5a015ccd0 0x7fb5a015ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 -- 172.21.15.119:0/3596570799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5880094b0 con 0x7fb5a00d1110 2024-09-13T16:31:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb597fff700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a015ea40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fb588014040 tx=0x7fb588004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb59d7fa700 1 -- 172.21.15.119:0/3596570799 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58800ba70 con 0x7fb5a00d1110 2024-09-13T16:31:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb59d7fa700 1 -- 172.21.15.119:0/3596570799 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58801e070 con 0x7fb5a00d1110 2024-09-13T16:31:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a015f770 con 0x7fb5a00d1110 2024-09-13T16:31:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb59d7fa700 1 -- 172.21.15.119:0/3596570799 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb58800bdf0 con 0x7fb5a00d1110 2024-09-13T16:31:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a00cac20 con 0x7fb5a00d1110 2024-09-13T16:31:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a00caa80 con 0x7fb5a00d1110 2024-09-13T16:31:39.246 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb59d7fa700 1 -- 172.21.15.119:0/3596570799 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb58802d440 con 0x7fb5a00d1110 2024-09-13T16:31:39.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.241+0000 7fb59d7fa700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb580066730 0x7fb580068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.245+0000 7fb59d7fa700 1 -- 172.21.15.119:0/3596570799 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb5880854e0 con 0x7fb5a00d1110 2024-09-13T16:31:39.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.245+0000 7fb59ffff700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb580066730 0x7fb580068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.245+0000 7fb59ffff700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb580066730 0x7fb580068bf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb590007b30 tx=0x7fb590007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.249+0000 7fb59d7fa700 1 -- 172.21.15.119:0/3596570799 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb5a00caa80 con 0x7fb5a00d1110 2024-09-13T16:31:39.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.421+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5a00caa80 con 0x7fb5a00d1110 2024-09-13T16:31:39.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.421+0000 7fb59d7fa700 1 -- 172.21.15.119:0/3596570799 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16777 (secure 0 0 0) 0x7fb5a00caa80 con 0x7fb5a00d1110 2024-09-13T16:31:39.425 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:39.425 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":95,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:31:29.963385+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:31:39.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb580066730 msgr2=0x7fb580068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb580066730 0x7fb580068bf0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7fb590007b30 tx=0x7fb590007a80 comp rx=0 tx=0).stop 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 msgr2=0x7fb5a015ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a015ea40 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fb588014040 tx=0x7fb588004770 comp rx=0 tx=0).stop 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 shutdown_connections 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb580066730 0x7fb580068bf0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5a015ccd0 0x7fb5a015ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5a00d1110 0x7fb5a015ea40 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 --2- 172.21.15.119:0/3596570799 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5a00c6f40 0x7fb5a00b5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 >> 172.21.15.119:0/3596570799 conn(0x7fb5a00b4640 msgr2=0x7fb5a00c7460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 shutdown_connections 2024-09-13T16:31:39.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.425+0000 7fb5a53b5700 1 -- 172.21.15.119:0/3596570799 wait complete. 2024-09-13T16:31:39.441 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:31:27.299109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:31:28.044326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:31:39.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:31:39.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6d5ae1700 1 Processor -- start 2024-09-13T16:31:39.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6d5ae1700 1 -- start start 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6d5ae1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015bb70 0x7fe6d015c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6d5ae1700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe6d00b4400 0x7fe6d00b4860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6d5ae1700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d00b4da0 con 0x7fe6d015aa50 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6d5ae1700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d015ae30 con 0x7fe6d00b4400 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6d5ae1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00ae480 con 0x7fe6d015bb70 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6ceffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015bb70 0x7fe6d015c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6cffff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe6d00b4400 0x7fe6d00b4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6cf7fe700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fe6d015aa50 0x7fe6d00b39f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:53058/0 (socket says 172.21.15.119:53058) 2024-09-13T16:31:39.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6ceffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015bb70 0x7fe6d015c120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34240/0 (socket says 172.21.15.119:34240) 2024-09-13T16:31:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6cf7fe700 1 -- 172.21.15.119:0/3512726624 learned_addr learned my addr 172.21.15.119:0/3512726624 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:39.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6cffff700 1 -- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015bb70 msgr2=0x7fe6d015c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6cffff700 1 --2- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015bb70 0x7fe6d015c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6cffff700 1 -- 172.21.15.119:0/3512726624 >> v1:172.21.15.145:6789/0 conn(0x7fe6d015aa50 legacy=0x7fe6d00b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6cffff700 1 -- 172.21.15.119:0/3512726624 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d00ae760 con 0x7fe6d00b4400 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.561+0000 7fe6ceffd700 1 --2- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015bb70 0x7fe6d015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6cffff700 1 --2- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d00b4400 0x7fe6d00b4860 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009d90 tx=0x7fe6c800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9384baf1900a012b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6ce7fc700 1 -- 172.21.15.119:0/3512726624 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c800e070 con 0x7fe6d00b4400 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6ce7fc700 1 -- 172.21.15.119:0/3512726624 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c800bb20 con 0x7fe6d00b4400 2024-09-13T16:31:39.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6ce7fc700 1 -- 172.21.15.119:0/3512726624 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8012430 con 0x7fe6d00b4400 2024-09-13T16:31:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d00b4400 msgr2=0x7fe6d00b4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d00b4400 0x7fe6d00b4860 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009d90 tx=0x7fe6c800b330 comp rx=0 tx=0).stop 2024-09-13T16:31:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/3512726624 shutdown_connections 2024-09-13T16:31:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d00b4400 0x7fe6d00b4860 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/3512726624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015bb70 0x7fe6d015c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/3512726624 >> 172.21.15.119:0/3512726624 conn(0x7fe6d0099a50 msgr2=0x7fe6d0099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:39.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/3512726624 shutdown_connections 2024-09-13T16:31:39.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/3512726624 wait complete. 2024-09-13T16:31:39.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 Processor -- start 2024-09-13T16:31:39.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.565+0000 7fe6d5ae1700 1 -- start start 2024-09-13T16:31:39.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d00b4400 0x7fe6d014b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015aa50 0x7fe6d006b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d015bb70 0x7fe6d014cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d014cb00 con 0x7fe6d00b4400 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d006bdb0 con 0x7fe6d015bb70 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d006c090 con 0x7fe6d015aa50 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d00b4400 0x7fe6d014b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d00b4400 0x7fe6d014b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38476/0 (socket says 172.21.15.119:38476) 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 -- 172.21.15.119:0/427204927 learned_addr learned my addr 172.21.15.119:0/427204927 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cffff700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d015bb70 0x7fe6d014cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6ceffd700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015aa50 0x7fe6d006b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 -- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015aa50 msgr2=0x7fe6d006b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015aa50 0x7fe6d006b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 -- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d015bb70 msgr2=0x7fe6d014cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d015bb70 0x7fe6d014cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 -- 172.21.15.119:0/427204927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b80094b0 con 0x7fe6d00b4400 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6cf7fe700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d00b4400 0x7fe6d014b370 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000c700 tx=0x7fe6c000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6c7fff700 1 -- 172.21.15.119:0/427204927 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0005d90 con 0x7fe6d00b4400 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6c7fff700 1 -- 172.21.15.119:0/427204927 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c00070d0 con 0x7fe6d00b4400 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c8009a70 con 0x7fe6d00b4400 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6c7fff700 1 -- 172.21.15.119:0/427204927 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c0007440 con 0x7fe6d00b4400 2024-09-13T16:31:39.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d006c370 con 0x7fe6d00b4400 2024-09-13T16:31:39.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6c7fff700 1 -- 172.21.15.119:0/427204927 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe6c00075a0 con 0x7fe6d00b4400 2024-09-13T16:31:39.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.569+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d00b4b90 con 0x7fe6d00b4400 2024-09-13T16:31:39.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.573+0000 7fe6c7fff700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6b0066730 0x7fe6b0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.573+0000 7fe6c7fff700 1 -- 172.21.15.119:0/427204927 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe6c0083720 con 0x7fe6d00b4400 2024-09-13T16:31:39.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.573+0000 7fe6ceffd700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6b0066730 0x7fe6b0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.573+0000 7fe6ceffd700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6b0066730 0x7fe6b0068bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6d015e5e0 tx=0x7fe6b8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.577+0000 7fe6c7fff700 1 -- 172.21.15.119:0/427204927 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe6c0050970 con 0x7fe6d00b4400 2024-09-13T16:31:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.749+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe6d00b4860 con 0x7fe6d00b4400 2024-09-13T16:31:39.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.753+0000 7fe6c7fff700 1 -- 172.21.15.119:0/427204927 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v95) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe6c0053f90 con 0x7fe6d00b4400 2024-09-13T16:31:39.756 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6b0066730 msgr2=0x7fe6b0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6b0066730 0x7fe6b0068bf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7fe6d015e5e0 tx=0x7fe6b8008040 comp rx=0 tx=0).stop 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d00b4400 msgr2=0x7fe6d014b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d00b4400 0x7fe6d014b370 secure :-1 s=READY pgs=1295 cs=0 l=1 rev1=1 crypto rx=0x7fe6c000c700 tx=0x7fe6c000cac0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 shutdown_connections 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6b0066730 0x7fe6b0068bf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe6d015bb70 0x7fe6d014cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe6d015aa50 0x7fe6d006b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 --2- 172.21.15.119:0/427204927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe6d00b4400 0x7fe6d014b370 unknown :-1 s=CLOSED pgs=1295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 >> 172.21.15.119:0/427204927 conn(0x7fe6d0099a50 msgr2=0x7fe6d015e850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:39.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 shutdown_connections 2024-09-13T16:31:39.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.757+0000 7fe6d5ae1700 1 -- 172.21.15.119:0/427204927 wait complete. 2024-09-13T16:31:39.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:31:39.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f992716a700 1 Processor -- start 2024-09-13T16:31:39.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f992716a700 1 -- start start 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f992716a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9920073ab0 0x7f992015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f992716a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992006db20 con 0x7f99200b1fd0 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f992716a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f992006ddc0 con 0x7f992015d880 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f992716a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f992015d150 con 0x7f9920073ab0 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925967700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9920073ab0 0x7f992015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9926168700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f99200b1fd0 0x7f99200b23b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45636/0 (socket says 172.21.15.119:45636) 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9926168700 1 -- 172.21.15.119:0/869406006 learned_addr learned my addr 172.21.15.119:0/869406006 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925967700 1 -- 172.21.15.119:0/869406006 >> v1:172.21.15.145:6789/0 conn(0x7f992015d880 legacy=0x7f992006a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925967700 1 -- 172.21.15.119:0/869406006 >> v1:172.21.15.119:6789/0 conn(0x7f99200b1fd0 legacy=0x7f99200b23b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925967700 1 -- 172.21.15.119:0/869406006 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f992006e0a0 con 0x7f9920073ab0 2024-09-13T16:31:39.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925967700 1 --2- 172.21.15.119:0/869406006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992015a5e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9910009850 tx=0x7f9910009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee0c8097356e36f6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925166700 1 -- 172.21.15.119:0/869406006 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910011830 con 0x7f9920073ab0 2024-09-13T16:31:39.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925166700 1 -- 172.21.15.119:0/869406006 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9910011990 con 0x7f9920073ab0 2024-09-13T16:31:39.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.885+0000 7f9925166700 1 -- 172.21.15.119:0/869406006 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9910011d20 con 0x7f9920073ab0 2024-09-13T16:31:39.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- 172.21.15.119:0/869406006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 msgr2=0x7f992015a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 --2- 172.21.15.119:0/869406006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992015a5e0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f9910009850 tx=0x7f9910009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:39.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- 172.21.15.119:0/869406006 shutdown_connections 2024-09-13T16:31:39.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 --2- 172.21.15.119:0/869406006 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992015a5e0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- 172.21.15.119:0/869406006 >> 172.21.15.119:0/869406006 conn(0x7f99200b0cd0 msgr2=0x7f99200b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:39.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- 172.21.15.119:0/869406006 shutdown_connections 2024-09-13T16:31:39.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- 172.21.15.119:0/869406006 wait complete. 2024-09-13T16:31:39.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 Processor -- start 2024-09-13T16:31:39.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- start start 2024-09-13T16:31:39.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992014dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99200b1fd0 0x7f99201f40a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f992015d880 0x7f99200ce1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99200ce6f0 con 0x7f9920073ab0 2024-09-13T16:31:39.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9920151cd0 con 0x7f99200b1fd0 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f992716a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f992014b500 con 0x7f992015d880 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992014dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992014dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38494/0 (socket says 172.21.15.119:38494) 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 -- 172.21.15.119:0/3754435058 learned_addr learned my addr 172.21.15.119:0/3754435058 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926969700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f992015d880 0x7f99200ce1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9925967700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99200b1fd0 0x7f99201f40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 -- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f992015d880 msgr2=0x7f99200ce1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f992015d880 0x7f99200ce1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 -- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99200b1fd0 msgr2=0x7f99201f40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99200b1fd0 0x7f99201f40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.889+0000 7f9926168700 1 -- 172.21.15.119:0/3754435058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99100094b0 con 0x7f9920073ab0 2024-09-13T16:31:39.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9926168700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992014dfb0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c700 tx=0x7f991c00ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9916ffd700 1 -- 172.21.15.119:0/3754435058 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c005d90 con 0x7f9920073ab0 2024-09-13T16:31:39.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f992014b7e0 con 0x7f9920073ab0 2024-09-13T16:31:39.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9916ffd700 1 -- 172.21.15.119:0/3754435058 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f991c0070d0 con 0x7f9920073ab0 2024-09-13T16:31:39.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9916ffd700 1 -- 172.21.15.119:0/3754435058 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f991c007440 con 0x7f9920073ab0 2024-09-13T16:31:39.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f992014bda0 con 0x7f9920073ab0 2024-09-13T16:31:39.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f992015dd10 con 0x7f9920073ab0 2024-09-13T16:31:39.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9916ffd700 1 -- 172.21.15.119:0/3754435058 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f991c0075a0 con 0x7f9920073ab0 2024-09-13T16:31:39.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9916ffd700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9904066730 0x7f9904068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:39.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9916ffd700 1 -- 172.21.15.119:0/3754435058 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f991c083670 con 0x7f9920073ab0 2024-09-13T16:31:39.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.893+0000 7f9925967700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9904066730 0x7f9904068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:39.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.897+0000 7f9925967700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9904066730 0x7f9904068bf0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9910009fd0 tx=0x7f9910008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:39.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:39.901+0000 7f9916ffd700 1 -- 172.21.15.119:0/3754435058 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f991c050830 con 0x7f9920073ab0 2024-09-13T16:31:40.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.073+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9920074230 con 0x7f9920073ab0 2024-09-13T16:31:40.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.073+0000 7f9916ffd700 1 -- 172.21.15.119:0/3754435058 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v95) v1 ==== 104+0+11 (secure 0 0 0) 0x7f991c053e50 con 0x7f9920073ab0 2024-09-13T16:31:40.077 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:31:40.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9904066730 msgr2=0x7f9904068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9904066730 0x7f9904068bf0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f9910009fd0 tx=0x7f9910008030 comp rx=0 tx=0).stop 2024-09-13T16:31:40.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 msgr2=0x7f992014dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992014dfb0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f991c00c700 tx=0x7f991c00ca10 comp rx=0 tx=0).stop 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 shutdown_connections 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9904066730 0x7f9904068bf0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f992015d880 0x7f99200ce1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99200b1fd0 0x7f99201f40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 --2- 172.21.15.119:0/3754435058 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9920073ab0 0x7f992014dfb0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 >> 172.21.15.119:0/3754435058 conn(0x7f99200b0cd0 msgr2=0x7f992006c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 shutdown_connections 2024-09-13T16:31:40.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.077+0000 7f992716a700 1 -- 172.21.15.119:0/3754435058 wait complete. 2024-09-13T16:31:40.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:31:40.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.213+0000 7fcd36d19700 1 Processor -- start 2024-09-13T16:31:40.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.213+0000 7fcd36d19700 1 -- start start 2024-09-13T16:31:40.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd36d19700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcd300adbc0 0x7fcd30071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd36d19700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3015f420 con 0x7fcd3015a850 2024-09-13T16:31:40.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd36d19700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3015f180 con 0x7fcd3015ebc0 2024-09-13T16:31:40.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd36d19700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd30070db0 con 0x7fcd300adbc0 2024-09-13T16:31:40.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcd3015a850 0x7fcd3015e4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45648/0 (socket says 172.21.15.119:45648) 2024-09-13T16:31:40.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 -- 172.21.15.119:0/126780368 learned_addr learned my addr 172.21.15.119:0/126780368 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:40.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd352b6700 1 --2- 172.21.15.119:0/126780368 >> v2:172.21.15.145:3300/0 conn(0x7fcd300adbc0 0x7fcd30071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063525925 0 0) 0x7fcd3015f420 con 0x7fcd3015a850 2024-09-13T16:31:40.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd100035d0 con 0x7fcd3015a850 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2849609451 0 0) 0x7fcd3015f180 con 0x7fcd3015ebc0 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd3015f420 con 0x7fcd3015ebc0 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1535888887 0 0) 0x7fcd100035d0 con 0x7fcd3015a850 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 >> v2:172.21.15.145:3300/0 conn(0x7fcd300adbc0 msgr2=0x7fcd30071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 --2- 172.21.15.119:0/126780368 >> v2:172.21.15.145:3300/0 conn(0x7fcd300adbc0 0x7fcd30071bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 >> v1:172.21.15.145:6789/0 conn(0x7fcd3015ebc0 legacy=0x7fcd300cca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd300707f0 con 0x7fcd3015a850 2024-09-13T16:31:40.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fcd180019f0 con 0x7fcd3015a850 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 >> v1:172.21.15.119:6789/0 conn(0x7fcd3015a850 legacy=0x7fcd3015e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd10003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd10004870 0x7fcd10004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd10005230 0x7fcd1000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30070db0 con 0x7fcd10003bb0 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1000c0b0 con 0x7fcd10005230 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1000c270 con 0x7fcd10004870 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd10003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd352b6700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd10005230 0x7fcd1000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2ffff700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd10004870 0x7fcd10004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 -- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd10004870 msgr2=0x7fcd10004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:40.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd10004870 0x7fcd10004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 -- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd10005230 msgr2=0x7fcd1000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd10005230 0x7fcd1000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd352b6700 1 -- 172.21.15.119:0/126780368 reap_dead start 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 -- 172.21.15.119:0/126780368 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd100035d0 con 0x7fcd10003bb0 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd34ab5700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd10003fb0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800a7d0 tx=0x7fcd100058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d06007175f694bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd180072d0 con 0x7fcd10003bb0 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1800ae90 con 0x7fcd10003bb0 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.217+0000 7fcd2f7fe700 1 -- 172.21.15.119:0/126780368 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd18007670 con 0x7fcd10003bb0 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 -- 172.21.15.119:0/126780368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 msgr2=0x7fcd10003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd10003fb0 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fcd1800a7d0 tx=0x7fcd100058a0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 -- 172.21.15.119:0/126780368 shutdown_connections 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 --2- 172.21.15.119:0/126780368 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd10003fb0 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 -- 172.21.15.119:0/126780368 >> 172.21.15.119:0/126780368 conn(0x7fcd300b0560 msgr2=0x7fcd300b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:40.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 -- 172.21.15.119:0/126780368 shutdown_connections 2024-09-13T16:31:40.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 -- 172.21.15.119:0/126780368 wait complete. 2024-09-13T16:31:40.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 Processor -- start 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.221+0000 7fcd36d19700 1 -- start start 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd3014efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd3015af30 0x7fcd300ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd3015b390 0x7fcd300ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3015b960 con 0x7fcd10003bb0 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3015bc40 con 0x7fcd3015b390 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3015bf20 con 0x7fcd3015af30 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd3014efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd3014efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38508/0 (socket says 172.21.15.119:38508) 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 -- 172.21.15.119:0/1118556035 learned_addr learned my addr 172.21.15.119:0/1118556035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:40.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd2ffff700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd3015af30 0x7fcd300ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd352b6700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd3015b390 0x7fcd300ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 -- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd3015af30 msgr2=0x7fcd300ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd3015af30 0x7fcd300ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 -- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd3015b390 msgr2=0x7fcd300ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd3015b390 0x7fcd300ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 -- 172.21.15.119:0/1118556035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3015c200 con 0x7fcd10003bb0 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd34ab5700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd3014efe0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000c080 tx=0x7fcd3015f310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd2d7fa700 1 -- 172.21.15.119:0/1118556035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1801f630 con 0x7fcd10003bb0 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd2d7fa700 1 -- 172.21.15.119:0/1118556035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1801f790 con 0x7fcd10003bb0 2024-09-13T16:31:40.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3006bdc0 con 0x7fcd10003bb0 2024-09-13T16:31:40.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd2d7fa700 1 -- 172.21.15.119:0/1118556035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1801fb20 con 0x7fcd10003bb0 2024-09-13T16:31:40.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd301e5330 con 0x7fcd10003bb0 2024-09-13T16:31:40.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd2d7fa700 1 -- 172.21.15.119:0/1118556035 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcd1801fc80 con 0x7fcd10003bb0 2024-09-13T16:31:40.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd3015e230 con 0x7fcd10003bb0 2024-09-13T16:31:40.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.225+0000 7fcd2d7fa700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd14066760 0x7fcd14068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.229+0000 7fcd2d7fa700 1 -- 172.21.15.119:0/1118556035 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcd18085f70 con 0x7fcd10003bb0 2024-09-13T16:31:40.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.229+0000 7fcd2ffff700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd14066760 0x7fcd14068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.229+0000 7fcd2ffff700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd14066760 0x7fcd14068c20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcd301f6060 tx=0x7fcd20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.233+0000 7fcd2d7fa700 1 -- 172.21.15.119:0/1118556035 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcd180b7db0 con 0x7fcd10003bb0 2024-09-13T16:31:40.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fcd3015baa0 con 0x7fcd10003bb0 2024-09-13T16:31:40.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd2d7fa700 1 -- 172.21.15.119:0/1118556035 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v95) v1 ==== 100+0+11 (secure 0 0 0) 0x7fcd3015baa0 con 0x7fcd10003bb0 2024-09-13T16:31:40.408 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd14066760 msgr2=0x7fcd14068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd14066760 0x7fcd14068c20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fcd301f6060 tx=0x7fcd20009040 comp rx=0 tx=0).stop 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 msgr2=0x7fcd3014efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd3014efe0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fcd1000c080 tx=0x7fcd3015f310 comp rx=0 tx=0).stop 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 shutdown_connections 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd14066760 0x7fcd14068c20 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd3015b390 0x7fcd300ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd3015af30 0x7fcd300ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 --2- 172.21.15.119:0/1118556035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd10003bb0 0x7fcd3014efe0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 >> 172.21.15.119:0/1118556035 conn(0x7fcd300b0560 msgr2=0x7fcd300cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 shutdown_connections 2024-09-13T16:31:40.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.405+0000 7fcd36d19700 1 -- 172.21.15.119:0/1118556035 wait complete. 2024-09-13T16:31:40.424 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:31:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.529+0000 7fcd29dd0700 1 Processor -- start 2024-09-13T16:31:40.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.529+0000 7fcd29dd0700 1 -- start start 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.529+0000 7fcd29dd0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcd240b17f0 0x7fcd240cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.529+0000 7fcd29dd0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcd240b2200 0x7fcd240b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.529+0000 7fcd29dd0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd240705d0 con 0x7fcd240cd8d0 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.529+0000 7fcd29dd0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd240b2bb0 con 0x7fcd240b2200 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.529+0000 7fcd29dd0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd2406fca0 con 0x7fcd240b17f0 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcd240b2200 0x7fcd240b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcd240b2200 0x7fcd240b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38516/0 (socket says 172.21.15.119:38516) 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 -- 172.21.15.119:0/1587733665 learned_addr learned my addr 172.21.15.119:0/1587733665 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:40.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd23fff700 1 --2- 172.21.15.119:0/1587733665 >> v2:172.21.15.145:3300/0 conn(0x7fcd240b17f0 0x7fcd240cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 -- 172.21.15.119:0/1587733665 >> v2:172.21.15.145:3300/0 conn(0x7fcd240b17f0 msgr2=0x7fcd240cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 --2- 172.21.15.119:0/1587733665 >> v2:172.21.15.145:3300/0 conn(0x7fcd240b17f0 0x7fcd240cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 -- 172.21.15.119:0/1587733665 >> v1:172.21.15.145:6789/0 conn(0x7fcd240cd8d0 legacy=0x7fcd2406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 -- 172.21.15.119:0/1587733665 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2406ff80 con 0x7fcd240b2200 2024-09-13T16:31:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd295cf700 1 --2- 172.21.15.119:0/1587733665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b2200 0x7fcd240b2670 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcd240ae2c0 tx=0x7fcd1c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b16c7f63ce8fc217 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd237fe700 1 -- 172.21.15.119:0/1587733665 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c011830 con 0x7fcd240b2200 2024-09-13T16:31:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd237fe700 1 -- 172.21.15.119:0/1587733665 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd1c011990 con 0x7fcd240b2200 2024-09-13T16:31:40.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd237fe700 1 -- 172.21.15.119:0/1587733665 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd1c011d20 con 0x7fcd240b2200 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 -- 172.21.15.119:0/1587733665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b2200 msgr2=0x7fcd240b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/1587733665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b2200 0x7fcd240b2670 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 crypto rx=0x7fcd240ae2c0 tx=0x7fcd1c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 -- 172.21.15.119:0/1587733665 shutdown_connections 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/1587733665 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b2200 0x7fcd240b2670 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/1587733665 >> v2:172.21.15.145:3300/0 conn(0x7fcd240b17f0 0x7fcd240cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 -- 172.21.15.119:0/1587733665 >> 172.21.15.119:0/1587733665 conn(0x7fcd2406a350 msgr2=0x7fcd2406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 -- 172.21.15.119:0/1587733665 shutdown_connections 2024-09-13T16:31:40.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.533+0000 7fcd29dd0700 1 -- 172.21.15.119:0/1587733665 wait complete. 2024-09-13T16:31:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 Processor -- start 2024-09-13T16:31:40.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 -- start start 2024-09-13T16:31:40.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b17f0 0x7fcd24159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd240b2200 0x7fcd240c51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd240cd8d0 0x7fcd240c56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240c4400 con 0x7fcd240b17f0 2024-09-13T16:31:40.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240c46e0 con 0x7fcd240b2200 2024-09-13T16:31:40.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd240c49c0 con 0x7fcd240cd8d0 2024-09-13T16:31:40.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b17f0 0x7fcd24159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b17f0 0x7fcd24159bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38518/0 (socket says 172.21.15.119:38518) 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 -- 172.21.15.119:0/4179374117 learned_addr learned my addr 172.21.15.119:0/4179374117 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd295cf700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd240cd8d0 0x7fcd240c56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd23fff700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd240b2200 0x7fcd240c51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 -- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd240cd8d0 msgr2=0x7fcd240c56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd240cd8d0 0x7fcd240c56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 -- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd240b2200 msgr2=0x7fcd240c51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd240b2200 0x7fcd240c51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 -- 172.21.15.119:0/4179374117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd1c009570 con 0x7fcd240b17f0 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd23fff700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd240b2200 0x7fcd240c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:40.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd28dce700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b17f0 0x7fcd24159bf0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c500 tx=0x7fcd0c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd217fa700 1 -- 172.21.15.119:0/4179374117 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0052d0 con 0x7fcd240b17f0 2024-09-13T16:31:40.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd217fa700 1 -- 172.21.15.119:0/4179374117 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd0c005430 con 0x7fcd240b17f0 2024-09-13T16:31:40.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd240c4ca0 con 0x7fcd240b17f0 2024-09-13T16:31:40.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd217fa700 1 -- 172.21.15.119:0/4179374117 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd0c0055a0 con 0x7fcd240b17f0 2024-09-13T16:31:40.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.537+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd2415a310 con 0x7fcd240b17f0 2024-09-13T16:31:40.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.541+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd240c4ee0 con 0x7fcd240b17f0 2024-09-13T16:31:40.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.541+0000 7fcd217fa700 1 -- 172.21.15.119:0/4179374117 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcd0c0038e0 con 0x7fcd240b17f0 2024-09-13T16:31:40.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.541+0000 7fcd217fa700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd04066730 0x7fcd04068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.541+0000 7fcd217fa700 1 -- 172.21.15.119:0/4179374117 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fcd0c083a50 con 0x7fcd240b17f0 2024-09-13T16:31:40.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.541+0000 7fcd23fff700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd04066730 0x7fcd04068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.541+0000 7fcd23fff700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd04066730 0x7fcd04068bf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fcd2406ac80 tx=0x7fcd1400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.549+0000 7fcd217fa700 1 -- 172.21.15.119:0/4179374117 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcd0c050cf0 con 0x7fcd240b17f0 2024-09-13T16:31:40.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fcd240cdcb0 con 0x7fcd240b17f0 2024-09-13T16:31:40.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd217fa700 1 -- 172.21.15.119:0/4179374117 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v95) v1 ==== 80+0+0 (secure 0 0 0) 0x7fcd0c054310 con 0x7fcd240b17f0 2024-09-13T16:31:40.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd04066730 msgr2=0x7fcd04068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd04066730 0x7fcd04068bf0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fcd2406ac80 tx=0x7fcd1400b040 comp rx=0 tx=0).stop 2024-09-13T16:31:40.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b17f0 msgr2=0x7fcd24159bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b17f0 0x7fcd24159bf0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7fcd0c00c500 tx=0x7fcd0c00c810 comp rx=0 tx=0).stop 2024-09-13T16:31:40.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 shutdown_connections 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcd04066730 0x7fcd04068bf0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcd240cd8d0 0x7fcd240c56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcd240b2200 0x7fcd240c51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 --2- 172.21.15.119:0/4179374117 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcd240b17f0 0x7fcd24159bf0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.729+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 >> 172.21.15.119:0/4179374117 conn(0x7fcd2406a350 msgr2=0x7fcd24161f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.733+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 shutdown_connections 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.733+0000 7fcd29dd0700 1 -- 172.21.15.119:0/4179374117 wait complete. 2024-09-13T16:31:40.736 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:31:40.746 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:31:40.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:40.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:40.759 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[47236]: fuse finished with error 0 and tester_r 0 2024-09-13T16:31:40.760 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:31:40.812 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:31:40.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:40.813 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:31:40.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:40.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d05999700 1 Processor -- start 2024-09-13T16:31:40.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d05999700 1 -- start start 2024-09-13T16:31:40.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d05999700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1d0006e270 0x7f1d0006e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d05999700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0006c040 con 0x7f1d001570d0 2024-09-13T16:31:40.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d05999700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0006ebf0 con 0x7f1d000d0f70 2024-09-13T16:31:40.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d05999700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d000ce730 con 0x7f1d0006e270 2024-09-13T16:31:40.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d04997700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1d001570d0 0x7f1d0006b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45662/0 (socket says 172.21.15.119:45662) 2024-09-13T16:31:40.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1d04997700 1 -- 172.21.15.119:0/2450806191 learned_addr learned my addr 172.21.15.119:0/2450806191 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:40.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cfffff700 1 --2- 172.21.15.119:0/2450806191 >> v2:172.21.15.119:3300/0 conn(0x7f1d0006e270 0x7f1d0006e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cfffff700 1 -- 172.21.15.119:0/2450806191 >> v1:172.21.15.145:6789/0 conn(0x7f1d000d0f70 legacy=0x7f1d000d1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cfffff700 1 -- 172.21.15.119:0/2450806191 >> v1:172.21.15.119:6789/0 conn(0x7f1d001570d0 legacy=0x7f1d0006b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cfffff700 1 -- 172.21.15.119:0/2450806191 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d000cf840 con 0x7f1d0006e270 2024-09-13T16:31:40.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cfffff700 1 --2- 172.21.15.119:0/2450806191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d0006e270 0x7f1d0006e6b0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009850 tx=0x7f1cec009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66dccc793876b7bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cff7fe700 1 -- 172.21.15.119:0/2450806191 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec011830 con 0x7f1d0006e270 2024-09-13T16:31:40.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cff7fe700 1 -- 172.21.15.119:0/2450806191 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cec011990 con 0x7f1d0006e270 2024-09-13T16:31:40.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.981+0000 7f1cff7fe700 1 -- 172.21.15.119:0/2450806191 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec011d20 con 0x7f1d0006e270 2024-09-13T16:31:40.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 -- 172.21.15.119:0/2450806191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d0006e270 msgr2=0x7f1d0006e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 --2- 172.21.15.119:0/2450806191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d0006e270 0x7f1d0006e6b0 secure :-1 s=READY pgs=1305 cs=0 l=1 rev1=1 crypto rx=0x7f1cec009850 tx=0x7f1cec009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:40.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 -- 172.21.15.119:0/2450806191 shutdown_connections 2024-09-13T16:31:40.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 --2- 172.21.15.119:0/2450806191 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d0006e270 0x7f1d0006e6b0 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 -- 172.21.15.119:0/2450806191 >> 172.21.15.119:0/2450806191 conn(0x7f1d00099a50 msgr2=0x7f1d00099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:40.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 -- 172.21.15.119:0/2450806191 shutdown_connections 2024-09-13T16:31:40.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 -- 172.21.15.119:0/2450806191 wait complete. 2024-09-13T16:31:40.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 Processor -- start 2024-09-13T16:31:40.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.985+0000 7f1d05999700 1 -- start start 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0006e270 0x7f1d0009c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d000d0f70 0x7f1d0009db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d001570d0 0x7f1d00151d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0009e0c0 con 0x7f1d001570d0 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d00152950 con 0x7f1d0006e270 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0009cae0 con 0x7f1d000d0f70 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d001570d0 0x7f1d00151d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d001570d0 0x7f1d00151d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38524/0 (socket says 172.21.15.119:38524) 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 -- 172.21.15.119:0/2837515625 learned_addr learned my addr 172.21.15.119:0/2837515625 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d04997700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0006e270 0x7f1d0009c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1cfffff700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d000d0f70 0x7f1d0009db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 -- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d000d0f70 msgr2=0x7f1d0009db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d000d0f70 0x7f1d0009db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 -- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0006e270 msgr2=0x7f1d0009c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0006e270 0x7f1d0009c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:40.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 -- 172.21.15.119:0/2837515625 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cec0094b0 con 0x7f1d001570d0 2024-09-13T16:31:40.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05198700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d001570d0 0x7f1d00151d50 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800cc80 tx=0x7f1cf800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:40.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1cfd7fa700 1 -- 172.21.15.119:0/2837515625 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf8003c20 con 0x7f1d001570d0 2024-09-13T16:31:40.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1cfd7fa700 1 -- 172.21.15.119:0/2837515625 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf8003d80 con 0x7f1d001570d0 2024-09-13T16:31:40.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0009cdc0 con 0x7f1d001570d0 2024-09-13T16:31:40.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1cfd7fa700 1 -- 172.21.15.119:0/2837515625 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cf800aaa0 con 0x7f1d001570d0 2024-09-13T16:31:40.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.989+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0009d380 con 0x7f1d001570d0 2024-09-13T16:31:40.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.993+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0006e6b0 con 0x7f1d001570d0 2024-09-13T16:31:40.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.993+0000 7f1cfd7fa700 1 -- 172.21.15.119:0/2837515625 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1cf800ac00 con 0x7f1d001570d0 2024-09-13T16:31:40.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.993+0000 7f1cfd7fa700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ce4066730 0x7f1ce4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:40.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.993+0000 7f1cfd7fa700 1 -- 172.21.15.119:0/2837515625 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f1cf8084250 con 0x7f1d001570d0 2024-09-13T16:31:40.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.993+0000 7f1d04997700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ce4066730 0x7f1ce4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:40.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:40.993+0000 7f1d04997700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ce4066730 0x7f1ce4068bf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4005930 tx=0x7f1cf4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.001+0000 7f1cfd7fa700 1 -- 172.21.15.119:0/2837515625 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1cf8051480 con 0x7f1d001570d0 2024-09-13T16:31:41.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.205+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d00152dc0 con 0x7f1d001570d0 2024-09-13T16:31:41.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.205+0000 7f1cfd7fa700 1 -- 172.21.15.119:0/2837515625 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+4745 (secure 0 0 0) 0x7f1cf8054aa0 con 0x7f1d001570d0 2024-09-13T16:31:41.211 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:41.211 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":56,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7881,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:31.973745+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7890},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7890":{"gid":7890,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6835/3011171408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3011171408},{"type":"v1","addr":"172.21.15.145:6835","nonce":3011171408}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:31:41.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ce4066730 msgr2=0x7f1ce4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ce4066730 0x7f1ce4068bf0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f1cf4005930 tx=0x7f1cf4009040 comp rx=0 tx=0).stop 2024-09-13T16:31:41.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d001570d0 msgr2=0x7f1d00151d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d001570d0 0x7f1d00151d50 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 crypto rx=0x7f1cf800cc80 tx=0x7f1cf800cf90 comp rx=0 tx=0).stop 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 shutdown_connections 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1ce4066730 0x7f1ce4068bf0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d001570d0 0x7f1d00151d50 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d000d0f70 0x7f1d0009db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 --2- 172.21.15.119:0/2837515625 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d0006e270 0x7f1d0009c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 >> 172.21.15.119:0/2837515625 conn(0x7f1d00099a50 msgr2=0x7f1d000cfd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 shutdown_connections 2024-09-13T16:31:41.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.209+0000 7f1d05999700 1 -- 172.21.15.119:0/2837515625 wait complete. 2024-09-13T16:31:41.215 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 56 2024-09-13T16:31:41.226 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json flush journal 2024-09-13T16:31:41.370 INFO:teuthology.orchestra.run.smithi145.stdout:{"message":"","return_code":0} 2024-09-13T16:31:41.370 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:31:41.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:31:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac86b9b700 1 Processor -- start 2024-09-13T16:31:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac86b9b700 1 -- start start 2024-09-13T16:31:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac86b9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac86b9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fac800ce040 0x7fac800b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac86b9b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac8006cf60 con 0x7fac8006bb90 2024-09-13T16:31:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac86b9b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac8006c350 con 0x7fac800ce040 2024-09-13T16:31:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac86b9b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8006c630 con 0x7fac800adb30 2024-09-13T16:31:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac85398700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fac8006bb90 0x7fac800cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45664/0 (socket says 172.21.15.119:45664) 2024-09-13T16:31:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac85398700 1 -- 172.21.15.119:0/3130638596 learned_addr learned my addr 172.21.15.119:0/3130638596 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac8639a700 1 --2- 172.21.15.119:0/3130638596 >> v2:172.21.15.119:3300/0 conn(0x7fac800ce040 0x7fac800b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac85b99700 1 --2- 172.21.15.119:0/3130638596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac84b97700 1 -- 172.21.15.119:0/3130638596 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2020588874 0 0) 0x7fac8006cf60 con 0x7fac8006bb90 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac84b97700 1 -- 172.21.15.119:0/3130638596 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac6c0035d0 con 0x7fac8006bb90 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac8639a700 1 -- 172.21.15.119:0/3130638596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 msgr2=0x7fac800cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac8639a700 1 --2- 172.21.15.119:0/3130638596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac8639a700 1 -- 172.21.15.119:0/3130638596 >> v1:172.21.15.119:6789/0 conn(0x7fac8006bb90 legacy=0x7fac800cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac8639a700 1 -- 172.21.15.119:0/3130638596 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8006c910 con 0x7fac800ce040 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.489+0000 7fac8639a700 1 --2- 172.21.15.119:0/3130638596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac800b2380 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fac78009850 tx=0x7fac78009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587268541c76859a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac84b97700 1 -- 172.21.15.119:0/3130638596 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78011830 con 0x7fac800ce040 2024-09-13T16:31:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac84b97700 1 -- 172.21.15.119:0/3130638596 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac78011990 con 0x7fac800ce040 2024-09-13T16:31:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac84b97700 1 -- 172.21.15.119:0/3130638596 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78011d40 con 0x7fac800ce040 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 -- 172.21.15.119:0/3130638596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 msgr2=0x7fac800b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3130638596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac800b2380 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 crypto rx=0x7fac78009850 tx=0x7fac78009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 -- 172.21.15.119:0/3130638596 shutdown_connections 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3130638596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac800b2380 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3130638596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 -- 172.21.15.119:0/3130638596 >> 172.21.15.119:0/3130638596 conn(0x7fac8006a350 msgr2=0x7fac8006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 -- 172.21.15.119:0/3130638596 shutdown_connections 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 -- 172.21.15.119:0/3130638596 wait complete. 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 Processor -- start 2024-09-13T16:31:41.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.493+0000 7fac86b9b700 1 -- start start 2024-09-13T16:31:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8006bb90 0x7fac800abbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac8015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac800aab10 con 0x7fac800ce040 2024-09-13T16:31:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac8015b600 con 0x7fac8006bb90 2024-09-13T16:31:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac8015b8a0 con 0x7fac800adb30 2024-09-13T16:31:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac8015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac8015c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38544/0 (socket says 172.21.15.119:38544) 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 -- 172.21.15.119:0/3115425916 learned_addr learned my addr 172.21.15.119:0/3115425916 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac85398700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac85b99700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8006bb90 0x7fac800abbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 -- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 msgr2=0x7fac800ac110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800ac110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 -- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8006bb90 msgr2=0x7fac800abbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8006bb90 0x7fac800abbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 -- 172.21.15.119:0/3115425916 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac780094b0 con 0x7fac800ce040 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac85b99700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8006bb90 0x7fac800abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac8639a700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac8015c380 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fac80071ff0 tx=0x7fac78004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac7e7fc700 1 -- 172.21.15.119:0/3115425916 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac78004400 con 0x7fac800ce040 2024-09-13T16:31:41.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac7e7fc700 1 -- 172.21.15.119:0/3115425916 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac7801f070 con 0x7fac800ce040 2024-09-13T16:31:41.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac7e7fc700 1 -- 172.21.15.119:0/3115425916 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac780036c0 con 0x7fac800ce040 2024-09-13T16:31:41.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac8015bb80 con 0x7fac800ce040 2024-09-13T16:31:41.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac8015c8c0 con 0x7fac800ce040 2024-09-13T16:31:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac8015c0a0 con 0x7fac800ce040 2024-09-13T16:31:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.497+0000 7fac7e7fc700 1 -- 172.21.15.119:0/3115425916 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fac78021030 con 0x7fac800ce040 2024-09-13T16:31:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.501+0000 7fac7e7fc700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac60066730 0x7fac60068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.501+0000 7fac7e7fc700 1 -- 172.21.15.119:0/3115425916 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fac780856f0 con 0x7fac800ce040 2024-09-13T16:31:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.501+0000 7fac85b99700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac60066730 0x7fac60068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.501+0000 7fac85b99700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac60066730 0x7fac60068bf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fac70009b70 tx=0x7fac70008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.509+0000 7fac7e7fc700 1 -- 172.21.15.119:0/3115425916 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fac8015c0a0 con 0x7fac800ce040 2024-09-13T16:31:41.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.717+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fac8015c0a0 con 0x7fac800ce040 2024-09-13T16:31:41.723 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:31:41.721+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:31:41.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.733+0000 7fac7e7fc700 1 -- 172.21.15.119:0/3115425916 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7fac8015c0a0 con 0x7fac800ce040 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac60066730 msgr2=0x7fac60068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac60066730 0x7fac60068bf0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fac70009b70 tx=0x7fac70008040 comp rx=0 tx=0).stop 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 msgr2=0x7fac8015c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac8015c380 secure :-1 s=READY pgs=1309 cs=0 l=1 rev1=1 crypto rx=0x7fac80071ff0 tx=0x7fac78004770 comp rx=0 tx=0).stop 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 shutdown_connections 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fac60066730 0x7fac60068bf0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac800ce040 0x7fac8015c380 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fac800adb30 0x7fac800ac110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 --2- 172.21.15.119:0/3115425916 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fac8006bb90 0x7fac800abbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 >> 172.21.15.119:0/3115425916 conn(0x7fac8006a350 msgr2=0x7fac800b3610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 shutdown_connections 2024-09-13T16:31:41.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.737+0000 7fac86b9b700 1 -- 172.21.15.119:0/3115425916 wait complete. 2024-09-13T16:31:41.744 INFO:tasks.ceph.mds.b.smithi145.stderr: -3797> 2024-09-13T16:31:29.904+0000 7f5654b33700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:31:41.744 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:31:41.759 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rm 10000000000.00000000 2024-09-13T16:31:41.776 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:31:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 Processor -- start 2024-09-13T16:31:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- start start 2024-09-13T16:31:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8aa9ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab2c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 0x556e8ab2ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e8a9ec2f0 con 0x556e8aaa40f0 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e8a9a88e0 con 0x556e8ab304e0 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e8a9a8580 con 0x556e8ab30100 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab2c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc84e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8aa9ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab2c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34332/0 (socket says 172.21.15.119:34332) 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 -- 172.21.15.119:0/2013080055 learned_addr learned my addr 172.21.15.119:0/2013080055 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdd04f700 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 0x556e8ab2ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 -- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 msgr2=0x556e8ab2ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 0x556e8ab2ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 -- 172.21.15.119:0/2013080055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 msgr2=0x556e8aa9ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8aa9ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 -- 172.21.15.119:0/2013080055 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbccc005de0 con 0x556e8ab30100 2024-09-13T16:31:41.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc84e700 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8aa9ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:41.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdc04d700 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab2c440 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400b7e0 tx=0x7fbcd400baf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a76a9fb3748dbd4e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdb04b700 1 -- 172.21.15.119:0/2013080055 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4012680 con 0x556e8ab30100 2024-09-13T16:31:41.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdb04b700 1 -- 172.21.15.119:0/2013080055 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcd40127e0 con 0x556e8ab30100 2024-09-13T16:31:41.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcdb04b700 1 -- 172.21.15.119:0/2013080055 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcd4012b90 con 0x556e8ab30100 2024-09-13T16:31:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 msgr2=0x556e8ab2c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab2c440 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400b7e0 tx=0x7fbcd400baf0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- 172.21.15.119:0/2013080055 shutdown_connections 2024-09-13T16:31:41.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 0x556e8ab2ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab2c440 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 --2- 172.21.15.119:0/2013080055 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8aa9ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- 172.21.15.119:0/2013080055 >> 172.21.15.119:0/2013080055 conn(0x556e8aa5b110 msgr2=0x556e8ab2ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- 172.21.15.119:0/2013080055 shutdown_connections 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- 172.21.15.119:0/2013080055 wait complete. 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 Processor -- start 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.793+0000 7fbcded19140 1 -- start start 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8abc08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab328e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 0x556e8ab32e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e8a97e8d0 con 0x556e8aaa40f0 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556e8aa5d400 con 0x556e8ab304e0 2024-09-13T16:31:41.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556e8ab24950 con 0x556e8ab30100 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8abc08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc04d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc04d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab328e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34336/0 (socket says 172.21.15.119:34336) 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc04d700 1 -- 172.21.15.119:0/635342159 learned_addr learned my addr 172.21.15.119:0/635342159 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8abc08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38568/0 (socket says 172.21.15.119:38568) 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdd04f700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 0x556e8ab32e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 -- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 msgr2=0x556e8ab328e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab328e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 -- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 msgr2=0x556e8ab32e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556e8ab304e0 0x556e8ab32e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 -- 172.21.15.119:0/635342159 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc400a040 con 0x556e8aaa40f0 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc04d700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556e8ab30100 0x556e8ab328e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcdc84e700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8abc08a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400ec60 tx=0x7fbcc400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcda049700 1 -- 172.21.15.119:0/635342159 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4016030 con 0x556e8aaa40f0 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcda049700 1 -- 172.21.15.119:0/635342159 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcc4008070 con 0x556e8aaa40f0 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcd400b3e0 con 0x556e8aaa40f0 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcda049700 1 -- 172.21.15.119:0/635342159 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcc4001bf0 con 0x556e8aaa40f0 2024-09-13T16:31:41.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.797+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556e8a996b00 con 0x556e8aaa40f0 2024-09-13T16:31:41.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcda049700 1 -- 172.21.15.119:0/635342159 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbcc4019070 con 0x556e8aaa40f0 2024-09-13T16:31:41.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcda049700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbcbc065060 0x7fbcbc067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcda049700 1 -- 172.21.15.119:0/635342159 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fbcc4017070 con 0x556e8aaa40f0 2024-09-13T16:31:41.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcdc04d700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbcbc065060 0x7fbcbc067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcded19140 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x556e8abc0de0 0x556e8aa5b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 13.10 13:0cf6ad0f:::10000000000.00000000:head [delete] snapc 0=[] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e96) v8 -- 0x556e8aa5bc40 con 0x556e8abc0de0 2024-09-13T16:31:41.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcdd04f700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x556e8abc0de0 0x556e8aa5b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcdc04d700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbcbc065060 0x7fbcbc067520 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400bb30 tx=0x7fbcd401d040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.801+0000 7fbcdd04f700 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x556e8abc0de0 0x556e8aa5b700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.805+0000 7fbcdd04f700 1 -- 172.21.15.119:0/635342159 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 10000000000.00000000 [delete] v96'11 uv11 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7fbccc019040 con 0x556e8abc0de0 2024-09-13T16:31:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.805+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x556e8abc0de0 msgr2=0x556e8aa5b700 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.805+0000 7fbcded19140 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x556e8abc0de0 0x556e8aa5b700 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbcbc065060 msgr2=0x7fbcbc067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbcbc065060 0x7fbcbc067520 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fbcd400bb30 tx=0x7fbcd401d040 comp rx=0 tx=0).stop 2024-09-13T16:31:41.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 msgr2=0x556e8abc08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 --2- 172.21.15.119:0/635342159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556e8aaa40f0 0x556e8abc08a0 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 crypto rx=0x7fbcc400ec60 tx=0x7fbcc400ef70 comp rx=0 tx=0).stop 2024-09-13T16:31:41.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcdd04f700 1 -- 172.21.15.119:0/635342159 reap_dead start 2024-09-13T16:31:41.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 shutdown_connections 2024-09-13T16:31:41.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 >> 172.21.15.119:0/635342159 conn(0x556e8aa5b110 msgr2=0x556e8aa9e1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 shutdown_connections 2024-09-13T16:31:41.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.809+0000 7fbcded19140 1 -- 172.21.15.119:0/635342159 wait complete. 2024-09-13T16:31:41.816 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --yes-i-really-really-mean-it 2024-09-13T16:31:41.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 -- start start 2024-09-13T16:31:41.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 Processor -- start 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba433000 0x5563ba4d8580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba433400 0x5563ba4d8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba433800 0x5563ba4d9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4a00 con 0x5563ba433800 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4780 con 0x5563ba433000 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4640 con 0x5563ba433400 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba433800 0x5563ba4d9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba433800 0x5563ba4d9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38584/0 (socket says 172.21.15.119:38584) 2024-09-13T16:31:41.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 -- 172.21.15.119:0/1272256496 learned_addr learned my addr 172.21.15.119:0/1272256496 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:41.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba433400 0x5563ba4d8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9d435700 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba433000 0x5563ba4d8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 -- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba433400 msgr2=0x5563ba4d8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba433400 0x5563ba4d8b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 -- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba433000 msgr2=0x5563ba4d8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba433000 0x5563ba4d8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.897+0000 7f0a9dc36700 1 -- 172.21.15.119:0/1272256496 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563ba4569c0 con 0x5563ba433800 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba433800 0x5563ba4d9080 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x5563ba53b530 tx=0x5563bb212480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d3d2a76a06f1ef server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ac30700 1 -- 172.21.15.119:0/1272256496 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba483600 con 0x5563ba433800 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ac30700 1 -- 172.21.15.119:0/1272256496 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5563ba47e480 con 0x5563ba433800 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ac30700 1 -- 172.21.15.119:0/1272256496 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba4831e0 con 0x5563ba433800 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/1272256496 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563ba456b60 con 0x5563ba433800 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/1272256496 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5563ba456340 con 0x5563ba433800 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 4 waiting for MDS map... 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ac30700 1 -- 172.21.15.119:0/1272256496 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x5563ba46cc40 con 0x5563ba433800 2024-09-13T16:31:41.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ac30700 1 -- 172.21.15.119:0/1272256496 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 57) v1 ==== 2573+0+0 (secure 0 0 0) 0x5563bb222900 con 0x5563ba433800 2024-09-13T16:31:41.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 4 Got MDS map 57 2024-09-13T16:31:41.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 10 main: JournalTool::main 2024-09-13T16:31:41.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 4 main: JournalTool: connecting to RADOS... 2024-09-13T16:31:41.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 1 Processor -- start 2024-09-13T16:31:41.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.901+0000 7f0a9ec7b500 1 -- start start 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77a400 0x5563ba4d9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a800 0x5563ba4d9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77ac00 0x5563ba4da100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba469e00 con 0x5563ba77ac00 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4780 con 0x5563ba77a800 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4500 con 0x5563ba77a400 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77ac00 0x5563ba4da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77a400 0x5563ba4d9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77a400 0x5563ba4d9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34364/0 (socket says 172.21.15.119:34364) 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 -- 172.21.15.119:0/3600233459 learned_addr learned my addr 172.21.15.119:0/3600233459 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:41.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9d435700 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a800 0x5563ba4d9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 -- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a800 msgr2=0x5563ba4d9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a800 0x5563ba4d9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 -- 172.21.15.119:0/3600233459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77ac00 msgr2=0x5563ba4da100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77ac00 0x5563ba4da100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 -- 172.21.15.119:0/3600233459 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563ba456000 con 0x5563ba77a400 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77a400 0x5563ba4d9600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5563ba53b8f0 tx=0x5563bb212060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6efd73f7479ce623 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a98c2c700 1 -- 172.21.15.119:0/3600233459 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba483080 con 0x5563ba77a400 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a98c2c700 1 -- 172.21.15.119:0/3600233459 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5563ba47e480 con 0x5563ba77a400 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a98c2c700 1 -- 172.21.15.119:0/3600233459 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba483600 con 0x5563ba77a400 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77a400 msgr2=0x5563ba4d9600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77a400 0x5563ba4d9600 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x5563ba53b8f0 tx=0x5563bb212060 comp rx=0 tx=0).stop 2024-09-13T16:31:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/3600233459 shutdown_connections 2024-09-13T16:31:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77ac00 0x5563ba4da100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a800 0x5563ba4d9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/3600233459 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77a400 0x5563ba4d9600 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/3600233459 >> 172.21.15.119:0/3600233459 conn(0x5563ba77a000 msgr2=0x5563ba4ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/3600233459 shutdown_connections 2024-09-13T16:31:41.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/3600233459 wait complete. 2024-09-13T16:31:41.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 Processor -- start 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.905+0000 7f0a9ec7b500 1 -- start start 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a400 0x5563ba4db180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77a800 0x5563ba4dac00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77ac00 0x5563ba4da680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b43c0 con 0x5563ba77a800 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4280 con 0x5563ba77a400 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4140 con 0x5563ba77ac00 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77a800 0x5563ba4dac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77a800 0x5563ba4dac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38606/0 (socket says 172.21.15.119:38606) 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 -- 172.21.15.119:0/973616281 learned_addr learned my addr 172.21.15.119:0/973616281 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77ac00 0x5563ba4da680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a400 0x5563ba4db180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 -- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77ac00 msgr2=0x5563ba4da680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77ac00 0x5563ba4da680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 -- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a400 msgr2=0x5563ba4db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a400 0x5563ba4db180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 -- 172.21.15.119:0/973616281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563ba4561a0 con 0x5563ba77a800 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9d435700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77a800 0x5563ba4dac00 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x5563bb2254a0 tx=0x5563ba72cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a97429700 1 -- 172.21.15.119:0/973616281 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba483080 con 0x5563ba77a800 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a97429700 1 -- 172.21.15.119:0/973616281 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5563bb221800 con 0x5563ba77a800 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5563ba456000 con 0x5563ba77a800 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a97429700 1 -- 172.21.15.119:0/973616281 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba483600 con 0x5563ba77a800 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563ba4569c0 con 0x5563ba77a800 2024-09-13T16:31:41.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.909+0000 7f0a9ec7b500 4 main: JournalTool: resolving pool 13 2024-09-13T16:31:41.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a97429700 1 -- 172.21.15.119:0/973616281 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x5563ba492840 con 0x5563ba77a800 2024-09-13T16:31:41.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a97429700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x5563ba77b400 0x5563ba4db700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a97429700 1 -- 172.21.15.119:0/973616281 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x5563ba46ca80 con 0x5563ba77a800 2024-09-13T16:31:41.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a9ec7b500 4 main: JournalTool: creating IoCtx.. 2024-09-13T16:31:41.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a9ec7b500 4 main: Executing for rank 0 2024-09-13T16:31:41.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x5563ba77b400 0x5563ba4db700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x5563ba77b400 0x5563ba4db700 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x5563ba53b8f0 tx=0x5563bb212240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a9ec7b500 1 -- start start 2024-09-13T16:31:41.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.913+0000 7f0a9ec7b500 1 Processor -- start 2024-09-13T16:31:41.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77bc00 0x5563ba4da100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba8e8000 0x5563ba4d9b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba8e8400 0x5563ba4d9600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4140 con 0x5563ba8e8400 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4280 con 0x5563ba8e8000 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5563ba4b4780 con 0x5563ba77bc00 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9cc34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba8e8400 0x5563ba4d9600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9dc36700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77bc00 0x5563ba4da100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9cc34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba8e8400 0x5563ba4d9600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38608/0 (socket says 172.21.15.119:38608) 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9cc34700 1 -- 172.21.15.119:0/784379520 learned_addr learned my addr 172.21.15.119:0/784379520 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9d435700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba8e8000 0x5563ba4d9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba8e8000 msgr2=0x5563ba4d9b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba8e8000 0x5563ba4d9b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba8e8400 msgr2=0x5563ba4d9600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba8e8400 0x5563ba4d9600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5563ba7e5860 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba8e8400 0x5563ba4d9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77bc00 0x5563ba4da100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x5563ba53ba70 tx=0x5563bb212300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8afe80cb697bb771 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a94423700 1 -- 172.21.15.119:0/784379520 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba482f20 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a94423700 1 -- 172.21.15.119:0/784379520 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5563bb220900 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a94423700 1 -- 172.21.15.119:0/784379520 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x5563ba4826e0 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5563ba7e56c0 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({fsmap=0}) v3 -- 0x5563ba7e5520 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 4 waiting for MDS map... 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a94423700 1 -- 172.21.15.119:0/784379520 <== mon.2 v2:172.21.15.145:3301/0 4 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x5563ba46c8c0 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a94423700 1 -- 172.21.15.119:0/784379520 <== mon.2 v2:172.21.15.145:3301/0 5 ==== fsmap(e 57) v1 ==== 2573+0+0 (secure 0 0 0) 0x5563ba8f0600 con 0x5563ba77bc00 2024-09-13T16:31:41.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 4 Got MDS map 57 2024-09-13T16:31:41.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5563ba8e8800 0x5563ba90c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:1 13.f 13:f1f69726:::400.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8e8c00 con 0x5563ba8e8800 2024-09-13T16:31:41.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.917+0000 7f0a9d435700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5563ba8e8800 0x5563ba90c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.921+0000 7f0a9d435700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5563ba8e8800 0x5563ba90c000 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=8ddb6317c5220858 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.925+0000 7f0a9d435700 1 -- 172.21.15.119:0/784379520 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x5563bb1a0b40 con 0x5563ba8e8800 2024-09-13T16:31:41.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.925+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5563ba8e9000 0x5563ba90c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.925+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:2 13.14 13:292cf221:::200.00000000:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8e9400 con 0x5563ba8e9000 2024-09-13T16:31:41.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.925+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5563ba8e9000 0x5563ba90c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.925+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5563ba8e9000 0x5563ba90c580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=9c1483a31d4d2368 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.929+0000 7f0a9cc34700 1 -- 172.21.15.119:0/784379520 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv18 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x5563bb1a0d80 con 0x5563ba8e9000 2024-09-13T16:31:41.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.929+0000 7f0a92c20700 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:3 13.14 13:2e2fa760:::200.00000001:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8f6800 con 0x5563ba8e9000 2024-09-13T16:31:41.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.929+0000 7f0a92c20700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x5563ba8f6c00 0x5563ba90e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.929+0000 7f0a92c20700 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:4 13.1b 13:ddb4e4d7:::200.00000002:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8f7000 con 0x5563ba8f6c00 2024-09-13T16:31:41.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.929+0000 7f0a9cc34700 1 -- 172.21.15.119:0/784379520 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(3 200.00000001 [stat out=16b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+16 (crc 0 0 0) 0x5563bb1a0d80 con 0x5563ba8e9000 2024-09-13T16:31:41.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.929+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x5563ba8f6c00 0x5563ba90e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.933+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x5563ba8f6c00 0x5563ba90e680 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=ed459799b2b52463 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.933+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(4 200.00000002 [stat] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0fc0 con 0x5563ba8f6c00 2024-09-13T16:31:41.936 INFO:teuthology.orchestra.run.smithi119.stdout:old journal was 4208649~2030 2024-09-13T16:31:41.936 INFO:teuthology.orchestra.run.smithi119.stdout:new journal start will be 8388608 (4177929 bytes past old end) 2024-09-13T16:31:41.936 INFO:teuthology.orchestra.run.smithi119.stdout:writing journal head 2024-09-13T16:31:41.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.933+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:5 13.14 13:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8f7400 con 0x5563ba8e9000 2024-09-13T16:31:41.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.933+0000 7f0a9cc34700 1 -- 172.21.15.119:0/784379520 <== osd.0 v2:172.21.15.119:6800/2655437631 3 ==== osd_op_reply(5 200.00000000 [writefull 0~90] v96'19 uv19 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0d80 con 0x5563ba8e9000 2024-09-13T16:31:41.939 INFO:teuthology.orchestra.run.smithi119.stdout:writing EResetJournal entry 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.933+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:6 13.1b 13:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8f7800 con 0x5563ba8f6c00 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.933+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5563ba8f7c00 0x5563ba90cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:7 13.b 13:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8f6800 con 0x5563ba8f7c00 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:8 13.1f 13:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8f7400 con 0x5563ba8e8800 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5563ba8e9400 0x5563ba90d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:9 13.1a 13:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba920000 con 0x5563ba8e9400 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:10 13.1e 13:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba920400 con 0x5563ba8f6c00 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x5563ba920800 0x5563ba90d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:41.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:11 13.1d 13:bec1251c:::200.00000007:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba920c00 con 0x5563ba920800 2024-09-13T16:31:41.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5563ba8e9400 0x5563ba90d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9d435700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5563ba8f7c00 0x5563ba90cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x5563ba920800 0x5563ba90d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:41.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9cc34700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5563ba8e9400 0x5563ba90d080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=c828275ee7c45c43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9d435700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5563ba8f7c00 0x5563ba90cb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=890a3f6adb6c2c65 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9dc36700 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x5563ba920800 0x5563ba90d600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=d7ce055f4e50d9f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:41.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== osd_op_reply(6 200.00000002 [delete] v96'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0fc0 con 0x5563ba8f6c00 2024-09-13T16:31:41.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9cc34700 1 -- 172.21.15.119:0/784379520 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(9 200.00000005 [delete] v96'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0d80 con 0x5563ba8e9400 2024-09-13T16:31:41.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9d435700 1 -- 172.21.15.119:0/784379520 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(8 200.00000004 [delete] v96'7 uv6 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0b40 con 0x5563ba8e8800 2024-09-13T16:31:41.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.937+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 <== osd.4 v2:172.21.15.145:6816/4125431312 3 ==== osd_op_reply(10 200.00000006 [delete] v96'2 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0fc0 con 0x5563ba8f6c00 2024-09-13T16:31:41.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.941+0000 7f0a9d435700 1 -- 172.21.15.119:0/784379520 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(7 200.00000003 [delete] v96'4 uv3 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0b40 con 0x5563ba8f7c00 2024-09-13T16:31:41.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.941+0000 7f0a92c20700 1 -- 172.21.15.119:0/784379520 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:12 13.1b 13:ddb4e4d7:::200.00000002:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e96) v8 -- 0x5563ba8f7400 con 0x5563ba8f6c00 2024-09-13T16:31:41.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 <== osd.4 v2:172.21.15.145:6816/4125431312 4 ==== osd_op_reply(12 200.00000002 [write 0~48] v96'3 uv3 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0fc0 con 0x5563ba8f6c00 2024-09-13T16:31:41.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9dc36700 1 -- 172.21.15.119:0/784379520 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(11 200.00000007 [delete] v96'3 uv2 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x5563bb1a0fc0 con 0x5563ba920800 2024-09-13T16:31:41.949 INFO:teuthology.orchestra.run.smithi119.stdout:done 2024-09-13T16:31:41.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5563ba8e9000 msgr2=0x5563ba90c580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5563ba8e9000 0x5563ba90c580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5563ba8e9400 msgr2=0x5563ba90d080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5563ba8e9400 0x5563ba90d080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x5563ba8f6c00 msgr2=0x5563ba90e680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x5563ba8f6c00 0x5563ba90e680 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5563ba8f7c00 msgr2=0x5563ba90cb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5563ba8f7c00 0x5563ba90cb00 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5563ba8e8800 msgr2=0x5563ba90c000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5563ba8e8800 0x5563ba90c000 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x5563ba920800 msgr2=0x5563ba90d600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x5563ba920800 0x5563ba90d600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77bc00 msgr2=0x5563ba4da100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77bc00 0x5563ba4da100 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x5563ba53ba70 tx=0x5563bb212300 comp rx=0 tx=0).stop 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 shutdown_connections 2024-09-13T16:31:41.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x5563ba920800 0x5563ba90d600 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5563ba8e9400 0x5563ba90d080 crc :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5563ba8e9000 0x5563ba90c580 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x5563ba8f7c00 0x5563ba90cb00 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5563ba8e8800 0x5563ba90c000 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba8e8400 0x5563ba4d9600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba8e8000 0x5563ba4d9b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x5563ba8f6c00 0x5563ba90e680 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/784379520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77bc00 0x5563ba4da100 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 >> 172.21.15.119:0/784379520 conn(0x5563ba77b800 msgr2=0x5563ba4f0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9cc34700 1 -- 172.21.15.119:0/784379520 reap_dead start 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 shutdown_connections 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/784379520 wait complete. 2024-09-13T16:31:41.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x5563ba77b400 msgr2=0x5563ba4db700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x5563ba77b400 0x5563ba4db700 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x5563ba53b8f0 tx=0x5563bb212240 comp rx=0 tx=0).stop 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77a800 msgr2=0x5563ba4dac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77a800 0x5563ba4dac00 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 crypto rx=0x5563bb2254a0 tx=0x5563ba72cc00 comp rx=0 tx=0).stop 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 shutdown_connections 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x5563ba77b400 0x5563ba4db700 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba77ac00 0x5563ba4da680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba77a800 0x5563ba4dac00 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/973616281 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba77a400 0x5563ba4db180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 >> 172.21.15.119:0/973616281 conn(0x5563ba77a000 msgr2=0x5563ba4ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 shutdown_connections 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.945+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/973616281 wait complete. 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/1272256496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba433800 msgr2=0x5563ba4d9080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:41.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba433800 0x5563ba4d9080 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 crypto rx=0x5563ba53b530 tx=0x5563bb212480 comp rx=0 tx=0).stop 2024-09-13T16:31:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/1272256496 shutdown_connections 2024-09-13T16:31:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5563ba433800 0x5563ba4d9080 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5563ba433400 0x5563ba4d8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 --2- 172.21.15.119:0/1272256496 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5563ba433000 0x5563ba4d8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/1272256496 >> 172.21.15.119:0/1272256496 conn(0x5563ba432c00 msgr2=0x5563ba4ef000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/1272256496 shutdown_connections 2024-09-13T16:31:41.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:41.949+0000 7f0a9ec7b500 1 -- 172.21.15.119:0/1272256496 wait complete. 2024-09-13T16:31:41.956 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--yes-i-really-really-mean-it'] in time 0:00:00.140511, result: old journal was 4208649~2030 new journal start will be 8388608 (4177929 bytes past old end) writing journal head writing EResetJournal entry done 2024-09-13T16:31:41.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-13T16:31:42.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.073+0000 7ff586d39700 1 Processor -- start 2024-09-13T16:31:42.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586d39700 1 -- start start 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586d39700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff58006a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586d39700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5800ac9f0 con 0x7ff58006a350 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586d39700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5800accd0 con 0x7ff580146410 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586d39700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58006ad00 con 0x7ff58006ea20 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff585536700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff58006a350 0x7ff58006e310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:45668/0 (socket says 172.21.15.119:45668) 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff585536700 1 -- 172.21.15.119:0/2531422261 learned_addr learned my addr 172.21.15.119:0/2531422261 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586538700 1 --2- 172.21.15.119:0/2531422261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff58006a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff584d35700 1 -- 172.21.15.119:0/2531422261 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90617060 0 0) 0x7ff5800ac9f0 con 0x7ff58006a350 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff584d35700 1 -- 172.21.15.119:0/2531422261 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff56c0035d0 con 0x7ff58006a350 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff584d35700 1 -- 172.21.15.119:0/2531422261 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680545907 0 0) 0x7ff5800accd0 con 0x7ff580146410 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff584d35700 1 -- 172.21.15.119:0/2531422261 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5800ac9f0 con 0x7ff580146410 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586538700 1 -- 172.21.15.119:0/2531422261 >> v1:172.21.15.145:6789/0 conn(0x7ff580146410 legacy=0x7ff5800b28e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586538700 1 -- 172.21.15.119:0/2531422261 >> v1:172.21.15.119:6789/0 conn(0x7ff58006a350 legacy=0x7ff58006e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586538700 1 -- 172.21.15.119:0/2531422261 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5800acfb0 con 0x7ff58006ea20 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff586538700 1 --2- 172.21.15.119:0/2531422261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff58006a7c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff578009dc0 tx=0x7ff57800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed914e2c070f662 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.077+0000 7ff584d35700 1 -- 172.21.15.119:0/2531422261 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57800e070 con 0x7ff58006ea20 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff584d35700 1 -- 172.21.15.119:0/2531422261 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57800bb20 con 0x7ff58006ea20 2024-09-13T16:31:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff584d35700 1 -- 172.21.15.119:0/2531422261 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57800be90 con 0x7ff58006ea20 2024-09-13T16:31:42.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 -- 172.21.15.119:0/2531422261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 msgr2=0x7ff58006a7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 --2- 172.21.15.119:0/2531422261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff58006a7c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff578009dc0 tx=0x7ff57800b330 comp rx=0 tx=0).stop 2024-09-13T16:31:42.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 -- 172.21.15.119:0/2531422261 shutdown_connections 2024-09-13T16:31:42.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 --2- 172.21.15.119:0/2531422261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff58006a7c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 -- 172.21.15.119:0/2531422261 >> 172.21.15.119:0/2531422261 conn(0x7ff5800b15e0 msgr2=0x7ff5800b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:42.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 -- 172.21.15.119:0/2531422261 shutdown_connections 2024-09-13T16:31:42.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 -- 172.21.15.119:0/2531422261 wait complete. 2024-09-13T16:31:42.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 Processor -- start 2024-09-13T16:31:42.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.081+0000 7ff586d39700 1 -- start start 2024-09-13T16:31:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff58006a350 0x7ff5800c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff5800c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff580146410 0x7ff5801f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800c52a0 con 0x7ff58006a350 2024-09-13T16:31:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800c5580 con 0x7ff580146410 2024-09-13T16:31:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5800c5860 con 0x7ff58006ea20 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff58006a350 0x7ff5800c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff58006a350 0x7ff5800c6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38612/0 (socket says 172.21.15.119:38612) 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 -- 172.21.15.119:0/1719070678 learned_addr learned my addr 172.21.15.119:0/1719070678 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585536700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff5800c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586538700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff580146410 0x7ff5801f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 -- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 msgr2=0x7ff5800c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff5800c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 -- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff580146410 msgr2=0x7ff5801f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff580146410 0x7ff5801f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 -- 172.21.15.119:0/1719070678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff578009a70 con 0x7ff58006a350 2024-09-13T16:31:42.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff585d37700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff58006a350 0x7ff5800c6340 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00c490 tx=0x7ff57c00c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:42.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff5727fc700 1 -- 172.21.15.119:0/1719070678 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c00cea0 con 0x7ff58006a350 2024-09-13T16:31:42.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff5727fc700 1 -- 172.21.15.119:0/1719070678 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c0052f0 con 0x7ff58006a350 2024-09-13T16:31:42.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5800c5b40 con 0x7ff58006a350 2024-09-13T16:31:42.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff5727fc700 1 -- 172.21.15.119:0/1719070678 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c0036a0 con 0x7ff58006a350 2024-09-13T16:31:42.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58013fe00 con 0x7ff58006a350 2024-09-13T16:31:42.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.085+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58013ff90 con 0x7ff58006a350 2024-09-13T16:31:42.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.089+0000 7ff5727fc700 1 -- 172.21.15.119:0/1719070678 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff57c003800 con 0x7ff58006a350 2024-09-13T16:31:42.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.089+0000 7ff5727fc700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.089+0000 7ff5727fc700 1 -- 172.21.15.119:0/1719070678 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff57c0836d0 con 0x7ff58006a350 2024-09-13T16:31:42.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.089+0000 7ff585536700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.089+0000 7ff585536700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff580140f60 tx=0x7ff574007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:42.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.097+0000 7ff5727fc700 1 -- 172.21.15.119:0/1719070678 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff57c050890 con 0x7ff58006a350 2024-09-13T16:31:42.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.301+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7ff5800c6060 con 0x7ff58006a350 2024-09-13T16:31:42.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.741+0000 7ff5727fc700 1 -- 172.21.15.119:0/1719070678 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v58) v1 ==== 162+0+0 (secure 0 0 0) 0x7ff5800c6060 con 0x7ff58006a350 2024-09-13T16:31:42.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 msgr2=0x7ff564068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7ff580140f60 tx=0x7ff574007de0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff58006a350 msgr2=0x7ff5800c6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff58006a350 0x7ff5800c6340 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00c490 tx=0x7ff57c00c7a0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 shutdown_connections 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff564066730 0x7ff564068bf0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff580146410 0x7ff5801f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff58006ea20 0x7ff5800c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 --2- 172.21.15.119:0/1719070678 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff58006a350 0x7ff5800c6340 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 >> 172.21.15.119:0/1719070678 conn(0x7ff5800b15e0 msgr2=0x7ff580160df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 shutdown_connections 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.745+0000 7ff586d39700 1 -- 172.21.15.119:0/1719070678 wait complete. 2024-09-13T16:31:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-13T16:31:42.769 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:42.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f6787700 1 Processor -- start 2024-09-13T16:31:42.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f6787700 1 -- start start 2024-09-13T16:31:42.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f6787700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa2f01594f0 0x7fa2f006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f6787700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa2f006da70 0x7fa2f00b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f6787700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f00b2280 con 0x7fa2f00b3630 2024-09-13T16:31:42.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f6787700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f0070c10 con 0x7fa2f01594f0 2024-09-13T16:31:42.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f6787700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f006dee0 con 0x7fa2f006da70 2024-09-13T16:31:42.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.873+0000 7fa2f5785700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa2f01594f0 0x7fa2f006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f5785700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa2f01594f0 0x7fa2f006d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37848/0 (socket says 172.21.15.119:37848) 2024-09-13T16:31:42.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f5785700 1 -- 172.21.15.119:0/3848608913 learned_addr learned my addr 172.21.15.119:0/3848608913 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f4f84700 1 --2- 172.21.15.119:0/3848608913 >> v2:172.21.15.145:3300/0 conn(0x7fa2f006da70 0x7fa2f00b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f5785700 1 -- 172.21.15.119:0/3848608913 >> v2:172.21.15.145:3300/0 conn(0x7fa2f006da70 msgr2=0x7fa2f00b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f5785700 1 --2- 172.21.15.119:0/3848608913 >> v2:172.21.15.145:3300/0 conn(0x7fa2f006da70 0x7fa2f00b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f5785700 1 -- 172.21.15.119:0/3848608913 >> v1:172.21.15.145:6789/0 conn(0x7fa2f00b3630 legacy=0x7fa2f015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f5785700 1 -- 172.21.15.119:0/3848608913 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f00cc890 con 0x7fa2f01594f0 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f4f84700 1 --2- 172.21.15.119:0/3848608913 >> v2:172.21.15.145:3300/0 conn(0x7fa2f006da70 0x7fa2f00b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f5785700 1 --2- 172.21.15.119:0/3848608913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f006d530 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa2f006c330 tx=0x7fa2ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94c61a20e0a4d4b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2e7fff700 1 -- 172.21.15.119:0/3848608913 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00e070 con 0x7fa2f01594f0 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2e7fff700 1 -- 172.21.15.119:0/3848608913 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec00b9c0 con 0x7fa2f01594f0 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2e7fff700 1 -- 172.21.15.119:0/3848608913 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00bd70 con 0x7fa2f01594f0 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 -- 172.21.15.119:0/3848608913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 msgr2=0x7fa2f006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 --2- 172.21.15.119:0/3848608913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f006d530 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fa2f006c330 tx=0x7fa2ec00b330 comp rx=0 tx=0).stop 2024-09-13T16:31:42.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 -- 172.21.15.119:0/3848608913 shutdown_connections 2024-09-13T16:31:42.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 --2- 172.21.15.119:0/3848608913 >> v2:172.21.15.145:3300/0 conn(0x7fa2f006da70 0x7fa2f00b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 --2- 172.21.15.119:0/3848608913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f006d530 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 -- 172.21.15.119:0/3848608913 >> 172.21.15.119:0/3848608913 conn(0x7fa2f006a350 msgr2=0x7fa2f006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:42.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 -- 172.21.15.119:0/3848608913 shutdown_connections 2024-09-13T16:31:42.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 -- 172.21.15.119:0/3848608913 wait complete. 2024-09-13T16:31:42.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 Processor -- start 2024-09-13T16:31:42.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.877+0000 7fa2f6787700 1 -- start start 2024-09-13T16:31:42.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f006da70 0x7fa2f0158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b3630 0x7fa2f0150c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f0156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0158df0 con 0x7fa2f01594f0 2024-09-13T16:31:42.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0155d40 con 0x7fa2f00b3630 2024-09-13T16:31:42.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0156020 con 0x7fa2f006da70 2024-09-13T16:31:42.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f0156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f0156b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37850/0 (socket says 172.21.15.119:37850) 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 -- 172.21.15.119:0/807371758 learned_addr learned my addr 172.21.15.119:0/807371758 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5785700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f006da70 0x7fa2f0158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f4f84700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b3630 0x7fa2f0150c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 -- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f006da70 msgr2=0x7fa2f0158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f006da70 0x7fa2f0158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 -- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b3630 msgr2=0x7fa2f0150c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b3630 0x7fa2f0150c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 -- 172.21.15.119:0/807371758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec009b30 con 0x7fa2f01594f0 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f4f84700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b3630 0x7fa2f0150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f5f86700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f0156b00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800e730 tx=0x7fa2e800eaf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2e5ffb700 1 -- 172.21.15.119:0/807371758 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80052d0 con 0x7fa2f01594f0 2024-09-13T16:31:42.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2e5ffb700 1 -- 172.21.15.119:0/807371758 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e8005430 con 0x7fa2f01594f0 2024-09-13T16:31:42.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f0156300 con 0x7fa2f01594f0 2024-09-13T16:31:42.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f014f530 con 0x7fa2f01594f0 2024-09-13T16:31:42.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2e5ffb700 1 -- 172.21.15.119:0/807371758 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e80055a0 con 0x7fa2f01594f0 2024-09-13T16:31:42.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.881+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2d0005310 con 0x7fa2f01594f0 2024-09-13T16:31:42.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.885+0000 7fa2e5ffb700 1 -- 172.21.15.119:0/807371758 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa2e800c960 con 0x7fa2f01594f0 2024-09-13T16:31:42.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.885+0000 7fa2e5ffb700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d4066760 0x7fa2d4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:42.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.885+0000 7fa2e5ffb700 1 -- 172.21.15.119:0/807371758 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fa2e8086330 con 0x7fa2f01594f0 2024-09-13T16:31:42.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.885+0000 7fa2f5785700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d4066760 0x7fa2d4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:42.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.885+0000 7fa2f5785700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d4066760 0x7fa2d4068c20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec000f80 tx=0x7fa2ec009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:42.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:42.889+0000 7fa2e5ffb700 1 -- 172.21.15.119:0/807371758 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa2e8053590 con 0x7fa2f01594f0 2024-09-13T16:31:43.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.093+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2d00055c0 con 0x7fa2f01594f0 2024-09-13T16:31:43.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2e5ffb700 1 -- 172.21.15.119:0/807371758 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+4746 (secure 0 0 0) 0x7fa2e8056bb0 con 0x7fa2f01594f0 2024-09-13T16:31:43.099 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:43.099 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":59,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":59,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:42.743509+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7881":{"gid":7881,"name":"c","rank":0,"incarnation":59,"state":"up:replay","state_seq":1,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:31:43.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d4066760 msgr2=0x7fa2d4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:43.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d4066760 0x7fa2d4068c20 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec000f80 tx=0x7fa2ec009000 comp rx=0 tx=0).stop 2024-09-13T16:31:43.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 msgr2=0x7fa2f0156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:43.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f0156b00 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800e730 tx=0x7fa2e800eaf0 comp rx=0 tx=0).stop 2024-09-13T16:31:43.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 shutdown_connections 2024-09-13T16:31:43.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d4066760 0x7fa2d4068c20 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:43.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f01594f0 0x7fa2f0156b00 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b3630 0x7fa2f0150c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 --2- 172.21.15.119:0/807371758 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f006da70 0x7fa2f0158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 >> 172.21.15.119:0/807371758 conn(0x7fa2f006a350 msgr2=0x7fa2f015bea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 shutdown_connections 2024-09-13T16:31:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:43.097+0000 7fa2f6787700 1 -- 172.21.15.119:0/807371758 wait complete. 2024-09-13T16:31:43.103 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 59 2024-09-13T16:31:43.517 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 59, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:31:28.960022+0000', 'modified': '2024-09-13T16:31:42.743509+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7881}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7881': {'gid': 7881, 'name': 'c', 'rank': 0, 'incarnation': 59, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/3937841911', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 3937841911}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 3937841911}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:31:43.518 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7881:up:replay 2024-09-13T16:31:44.519 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:44.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf60843700 1 Processor -- start 2024-09-13T16:31:44.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf60843700 1 -- start start 2024-09-13T16:31:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf60843700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 0x7fbf5c06a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf60843700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c06e2a0 con 0x7fbf5c06af80 2024-09-13T16:31:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf60843700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c15d7a0 con 0x7fbf5c0724c0 2024-09-13T16:31:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf60843700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c06dce0 con 0x7fbf5c158d30 2024-09-13T16:31:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5affd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbf5c06af80 0x7fbf5c15c980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:39764/0 (socket says 172.21.15.119:39764) 2024-09-13T16:31:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5affd700 1 -- 172.21.15.119:0/3093052825 learned_addr learned my addr 172.21.15.119:0/3093052825 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2579354352 0 0) 0x7fbf5c06e2a0 con 0x7fbf5c06af80 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf480035d0 con 0x7fbf5c06af80 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5b7fe700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 0x7fbf5c06a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018573790 0 0) 0x7fbf5c15d7a0 con 0x7fbf5c0724c0 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c06e2a0 con 0x7fbf5c0724c0 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 773547340 0 0) 0x7fbf480035d0 con 0x7fbf5c06af80 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 msgr2=0x7fbf5c06a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 0x7fbf5c06a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 >> v1:172.21.15.145:6789/0 conn(0x7fbf5c0724c0 legacy=0x7fbf5c15d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c06dfc0 con 0x7fbf5c06af80 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fbf440019f0 con 0x7fbf5c06af80 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 >> v1:172.21.15.119:6789/0 conn(0x7fbf5c06af80 legacy=0x7fbf5c15c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf48003be0 0x7fbf48003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 0x7fbf48004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf48004db0 0x7fbf48005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c06dce0 con 0x7fbf48004db0 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4800bbf0 con 0x7fbf48003be0 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf4800bdb0 con 0x7fbf5c158d30 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5bfff700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf48004db0 0x7fbf48005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.645+0000 7fbf5affd700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 0x7fbf48004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5bfff700 1 -- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 msgr2=0x7fbf48004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5bfff700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c158d30 0x7fbf48004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5bfff700 1 -- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf48003be0 msgr2=0x7fbf48003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:44.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5bfff700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf48003be0 0x7fbf48003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5bfff700 1 -- 172.21.15.119:0/3093052825 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf480035d0 con 0x7fbf48004db0 2024-09-13T16:31:44.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5bfff700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf48004db0 0x7fbf48005230 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fbf54007fd0 tx=0x7fbf480058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d75c2bfa6ce8d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:44.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54001450 con 0x7fbf48004db0 2024-09-13T16:31:44.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf540052f0 con 0x7fbf48004db0 2024-09-13T16:31:44.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5a7fc700 1 -- 172.21.15.119:0/3093052825 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf54001d60 con 0x7fbf48004db0 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 -- 172.21.15.119:0/3093052825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf48004db0 msgr2=0x7fbf48005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 --2- 172.21.15.119:0/3093052825 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf48004db0 0x7fbf48005230 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7fbf54007fd0 tx=0x7fbf480058a0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf5bfff700 1 -- 172.21.15.119:0/3093052825 reap_dead start 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 -- 172.21.15.119:0/3093052825 shutdown_connections 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 -- 172.21.15.119:0/3093052825 >> 172.21.15.119:0/3093052825 conn(0x7fbf5c0ae6e0 msgr2=0x7fbf5c0b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 -- 172.21.15.119:0/3093052825 shutdown_connections 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 -- 172.21.15.119:0/3093052825 wait complete. 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 Processor -- start 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.649+0000 7fbf60843700 1 -- start start 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf5c149b20 0x7fbf5c0cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c0ca110 0x7fbf5c152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf5c1526a0 0x7fbf5c0cb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c152b20 con 0x7fbf5c149b20 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0cdbe0 con 0x7fbf5c1526a0 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf5c0c91d0 con 0x7fbf5c0ca110 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf5c149b20 0x7fbf5c0cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf5c149b20 0x7fbf5c0cd500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37870/0 (socket says 172.21.15.119:37870) 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 -- 172.21.15.119:0/4056966163 learned_addr learned my addr 172.21.15.119:0/4056966163 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5affd700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c0ca110 0x7fbf5c152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5bfff700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf5c1526a0 0x7fbf5c0cb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:44.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 -- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c0ca110 msgr2=0x7fbf5c152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:44.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c0ca110 0x7fbf5c152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 -- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf5c1526a0 msgr2=0x7fbf5c0cb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:44.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf5c1526a0 0x7fbf5c0cb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 -- 172.21.15.119:0/4056966163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c149f00 con 0x7fbf5c149b20 2024-09-13T16:31:44.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf5b7fe700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf5c149b20 0x7fbf5c0cd500 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00d6a0 tx=0x7fbf4c00d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:44.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf53fff700 1 -- 172.21.15.119:0/4056966163 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c011870 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf53fff700 1 -- 172.21.15.119:0/4056966163 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf4c011a30 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c0ca590 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf53fff700 1 -- 172.21.15.119:0/4056966163 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf4c011de0 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c0c86f0 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.653+0000 7fbf53fff700 1 -- 172.21.15.119:0/4056966163 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbf4c0103e0 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.657+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c074470 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.657+0000 7fbf53fff700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf3c066730 0x7fbf3c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.657+0000 7fbf53fff700 1 -- 172.21.15.119:0/4056966163 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fbf4c084500 con 0x7fbf5c149b20 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.657+0000 7fbf5affd700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf3c066730 0x7fbf3c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:44.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.657+0000 7fbf5affd700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf3c066730 0x7fbf3c068bf0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c154a80 tx=0x7fbf44006040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:44.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.665+0000 7fbf53fff700 1 -- 172.21.15.119:0/4056966163 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbf4c0551f0 con 0x7fbf5c149b20 2024-09-13T16:31:44.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.869+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf5c0703f0 con 0x7fbf5c149b20 2024-09-13T16:31:44.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf53fff700 1 -- 172.21.15.119:0/4056966163 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+4746 (secure 0 0 0) 0x7fbf4c05b070 con 0x7fbf5c149b20 2024-09-13T16:31:44.876 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:44.876 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":61,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:44.748203+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7881":{"gid":7881,"name":"c","rank":0,"incarnation":59,"state":"up:rejoin","state_seq":7,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:31:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf3c066730 msgr2=0x7fbf3c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf3c066730 0x7fbf3c068bf0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7fbf5c154a80 tx=0x7fbf44006040 comp rx=0 tx=0).stop 2024-09-13T16:31:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf5c149b20 msgr2=0x7fbf5c0cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf5c149b20 0x7fbf5c0cd500 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7fbf4c00d6a0 tx=0x7fbf4c00d9b0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 shutdown_connections 2024-09-13T16:31:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf3c066730 0x7fbf3c068bf0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbf5c1526a0 0x7fbf5c0cb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbf5c0ca110 0x7fbf5c152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 --2- 172.21.15.119:0/4056966163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbf5c149b20 0x7fbf5c0cd500 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 >> 172.21.15.119:0/4056966163 conn(0x7fbf48004db0 msgr2=0x7fbf5c157cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 shutdown_connections 2024-09-13T16:31:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:44.873+0000 7fbf60843700 1 -- 172.21.15.119:0/4056966163 wait complete. 2024-09-13T16:31:44.879 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 61 2024-09-13T16:31:45.294 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 61, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:31:28.960022+0000', 'modified': '2024-09-13T16:31:44.748203+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7881}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7881': {'gid': 7881, 'name': 'c', 'rank': 0, 'incarnation': 59, 'state': 'up:rejoin', 'state_seq': 7, 'addr': '172.21.15.119:6835/3937841911', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 3937841911}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 3937841911}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:31:45.294 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_7881:up:rejoin 2024-09-13T16:31:46.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:31:46.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc13fff700 1 Processor -- start 2024-09-13T16:31:46.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc13fff700 1 -- start start 2024-09-13T16:31:46.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc13fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc140b1940 0x7efc140b2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:46.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc13fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc1406dcf0 0x7efc14071480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:46.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc13fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc140b2c70 con 0x7efc14156530 2024-09-13T16:31:46.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc13fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc1406c2a0 con 0x7efc140b1940 2024-09-13T16:31:46.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc13fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc14156910 con 0x7efc1406dcf0 2024-09-13T16:31:46.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc140b1940 0x7efc140b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:46.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc140b1940 0x7efc140b2320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37884/0 (socket says 172.21.15.119:37884) 2024-09-13T16:31:46.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 -- 172.21.15.119:0/3266841486 learned_addr learned my addr 172.21.15.119:0/3266841486 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:46.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 -- 172.21.15.119:0/3266841486 >> v2:172.21.15.145:3300/0 conn(0x7efc1406dcf0 msgr2=0x7efc14071480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:46.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 --2- 172.21.15.119:0/3266841486 >> v2:172.21.15.145:3300/0 conn(0x7efc1406dcf0 0x7efc14071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 -- 172.21.15.119:0/3266841486 >> v1:172.21.15.145:6789/0 conn(0x7efc14156530 legacy=0x7efc1406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:46.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 -- 172.21.15.119:0/3266841486 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc140b2f50 con 0x7efc140b1940 2024-09-13T16:31:46.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc127fc700 1 --2- 172.21.15.119:0/3266841486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc140b1940 0x7efc140b2320 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7efc140700f0 tx=0x7efc0400d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30d52d4433c9970b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:46.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.413+0000 7efc11ffb700 1 -- 172.21.15.119:0/3266841486 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc04010070 con 0x7efc140b1940 2024-09-13T16:31:46.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc11ffb700 1 -- 172.21.15.119:0/3266841486 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0400db40 con 0x7efc140b1940 2024-09-13T16:31:46.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc11ffb700 1 -- 172.21.15.119:0/3266841486 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc04003a40 con 0x7efc140b1940 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- 172.21.15.119:0/3266841486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc140b1940 msgr2=0x7efc140b2320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 --2- 172.21.15.119:0/3266841486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc140b1940 0x7efc140b2320 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7efc140700f0 tx=0x7efc0400d430 comp rx=0 tx=0).stop 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- 172.21.15.119:0/3266841486 shutdown_connections 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 --2- 172.21.15.119:0/3266841486 >> v2:172.21.15.145:3300/0 conn(0x7efc1406dcf0 0x7efc14071480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 --2- 172.21.15.119:0/3266841486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc140b1940 0x7efc140b2320 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- 172.21.15.119:0/3266841486 >> 172.21.15.119:0/3266841486 conn(0x7efc1406a350 msgr2=0x7efc1406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- 172.21.15.119:0/3266841486 shutdown_connections 2024-09-13T16:31:46.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- 172.21.15.119:0/3266841486 wait complete. 2024-09-13T16:31:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 Processor -- start 2024-09-13T16:31:46.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- start start 2024-09-13T16:31:46.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc1406dcf0 0x7efc141521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc14156530 0x7efc1415a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc1415ac40 0x7efc1414bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc14146950 con 0x7efc1415ac40 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc141526e0 con 0x7efc14156530 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc13fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc14152970 con 0x7efc1406dcf0 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.417+0000 7efc137fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc1415ac40 0x7efc1414bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc1415ac40 0x7efc1414bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37896/0 (socket says 172.21.15.119:37896) 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 -- 172.21.15.119:0/2743351501 learned_addr learned my addr 172.21.15.119:0/2743351501 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc12ffd700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc1406dcf0 0x7efc141521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc127fc700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc14156530 0x7efc1415a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 -- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc1406dcf0 msgr2=0x7efc141521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:46.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc1406dcf0 0x7efc141521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 -- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc14156530 msgr2=0x7efc1415a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc14156530 0x7efc1415a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 -- 172.21.15.119:0/2743351501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc0400bd40 con 0x7efc1415ac40 2024-09-13T16:31:46.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc137fe700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc1415ac40 0x7efc1414bf60 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00b4d0 tx=0x7efc0c00b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efbfb7fe700 1 -- 172.21.15.119:0/2743351501 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c011830 con 0x7efc1415ac40 2024-09-13T16:31:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efbfb7fe700 1 -- 172.21.15.119:0/2743351501 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c011990 con 0x7efc1415ac40 2024-09-13T16:31:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc14149c40 con 0x7efc1415ac40 2024-09-13T16:31:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efbfb7fe700 1 -- 172.21.15.119:0/2743351501 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c011d40 con 0x7efc1415ac40 2024-09-13T16:31:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1414a200 con 0x7efc1415ac40 2024-09-13T16:31:46.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.421+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1406fd90 con 0x7efc1415ac40 2024-09-13T16:31:46.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.425+0000 7efbfb7fe700 1 -- 172.21.15.119:0/2743351501 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efc0c01a400 con 0x7efc1415ac40 2024-09-13T16:31:46.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.425+0000 7efbfb7fe700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc00066730 0x7efc00068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:46.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.425+0000 7efbfb7fe700 1 -- 172.21.15.119:0/2743351501 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7efc0c084240 con 0x7efc1415ac40 2024-09-13T16:31:46.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.425+0000 7efc12ffd700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc00066730 0x7efc00068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:46.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.425+0000 7efc12ffd700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc00066730 0x7efc00068bf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7efc08009980 tx=0x7efc080090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:46.435 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.433+0000 7efbfb7fe700 1 -- 172.21.15.119:0/2743351501 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efc0c051410 con 0x7efc1415ac40 2024-09-13T16:31:46.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc140af600 con 0x7efc1415ac40 2024-09-13T16:31:46.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efbfb7fe700 1 -- 172.21.15.119:0/2743351501 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+4746 (secure 0 0 0) 0x7efc0c054a30 con 0x7efc1415ac40 2024-09-13T16:31:46.640 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:31:46.640 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":62,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:45.750573+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7881":{"gid":7881,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":8,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc00066730 msgr2=0x7efc00068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc00066730 0x7efc00068bf0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7efc08009980 tx=0x7efc080090e0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc1415ac40 msgr2=0x7efc1414bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc1415ac40 0x7efc1414bf60 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00b4d0 tx=0x7efc0c00b7e0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 shutdown_connections 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc00066730 0x7efc00068bf0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc1415ac40 0x7efc1414bf60 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc14156530 0x7efc1415a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 --2- 172.21.15.119:0/2743351501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc1406dcf0 0x7efc141521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.637+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 >> 172.21.15.119:0/2743351501 conn(0x7efc1406a350 msgr2=0x7efc1406c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.641+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 shutdown_connections 2024-09-13T16:31:46.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:46.641+0000 7efc13fff700 1 -- 172.21.15.119:0/2743351501 wait complete. 2024-09-13T16:31:46.644 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 62 2024-09-13T16:31:47.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:31:28.960022+0000', 'modified': '2024-09-13T16:31:45.750573+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 96, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7881}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7881': {'gid': 7881, 'name': 'c', 'rank': 0, 'incarnation': 59, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.119:6835/3937841911', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 3937841911}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 3937841911}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:31:47.058 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:31:47.058 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:31:47.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a9c4700 1 Processor -- start 2024-09-13T16:31:47.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a9c4700 1 -- start start 2024-09-13T16:31:47.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a9c4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f340b3410 0x7f1f34165340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a9c4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f340b1c40 con 0x7f1f34143f50 2024-09-13T16:31:47.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a9c4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f3406cdf0 con 0x7f1f34144330 2024-09-13T16:31:47.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a9c4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f3406d0d0 con 0x7f1f340b3410 2024-09-13T16:31:47.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f399c2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1f34143f50 0x7f1f3406c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:39772/0 (socket says 172.21.15.119:39772) 2024-09-13T16:31:47.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f399c2700 1 -- 172.21.15.119:0/2541536873 learned_addr learned my addr 172.21.15.119:0/2541536873 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:47.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/2541536873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f340b3410 0x7f1f34165340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f389c0700 1 -- 172.21.15.119:0/2541536873 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 20373391 0 0) 0x7f1f340b1c40 con 0x7f1f34143f50 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f389c0700 1 -- 172.21.15.119:0/2541536873 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f200035d0 con 0x7f1f34143f50 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f389c0700 1 -- 172.21.15.119:0/2541536873 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911911257 0 0) 0x7f1f3406cdf0 con 0x7f1f34144330 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/2541536873 >> v1:172.21.15.145:6789/0 conn(0x7f1f34144330 legacy=0x7f1f340cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/2541536873 >> v1:172.21.15.119:6789/0 conn(0x7f1f34143f50 legacy=0x7f1f3406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/2541536873 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f3406d3b0 con 0x7f1f340b3410 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/2541536873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f340b3410 0x7f1f34165340 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009880 tx=0x7f1f2c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6bd27371dc14388 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.180+0000 7f1f389c0700 1 -- 172.21.15.119:0/2541536873 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c011830 con 0x7f1f340b3410 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f389c0700 1 -- 172.21.15.119:0/2541536873 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c011990 con 0x7f1f340b3410 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f389c0700 1 -- 172.21.15.119:0/2541536873 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c011d20 con 0x7f1f340b3410 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/2541536873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f340b3410 msgr2=0x7f1f34165340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/2541536873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f340b3410 0x7f1f34165340 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c009880 tx=0x7f1f2c009b90 comp rx=0 tx=0).stop 2024-09-13T16:31:47.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/2541536873 shutdown_connections 2024-09-13T16:31:47.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/2541536873 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f340b3410 0x7f1f34165340 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/2541536873 >> 172.21.15.119:0/2541536873 conn(0x7f1f3406a350 msgr2=0x7f1f3406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/2541536873 shutdown_connections 2024-09-13T16:31:47.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/2541536873 wait complete. 2024-09-13T16:31:47.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 Processor -- start 2024-09-13T16:31:47.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- start start 2024-09-13T16:31:47.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f340b3410 0x7f1f3415a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f34143f50 0x7f1f340c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f34144330 0x7f1f340ac480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340c3800 con 0x7f1f34143f50 2024-09-13T16:31:47.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340c3ae0 con 0x7f1f340b3410 2024-09-13T16:31:47.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f3a9c4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f340c3dc0 con 0x7f1f34144330 2024-09-13T16:31:47.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.184+0000 7f1f391c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f34143f50 0x7f1f340c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f34143f50 0x7f1f340c8010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37906/0 (socket says 172.21.15.119:37906) 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 -- 172.21.15.119:0/1083373270 learned_addr learned my addr 172.21.15.119:0/1083373270 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f399c2700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f340b3410 0x7f1f3415a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f34144330 0x7f1f340ac480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 -- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f34144330 msgr2=0x7f1f340ac480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f34144330 0x7f1f340ac480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 -- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f340b3410 msgr2=0x7f1f3415a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f340b3410 0x7f1f3415a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 -- 172.21.15.119:0/1083373270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f2c0094b0 con 0x7f1f34143f50 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f399c2700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f340b3410 0x7f1f3415a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:47.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f391c1700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f34143f50 0x7f1f340c8010 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f1f30001f90 tx=0x7f1f3000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f2a7fc700 1 -- 172.21.15.119:0/1083373270 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f30003e50 con 0x7f1f34143f50 2024-09-13T16:31:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f2a7fc700 1 -- 172.21.15.119:0/1083373270 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f3000a990 con 0x7f1f34143f50 2024-09-13T16:31:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f2a7fc700 1 -- 172.21.15.119:0/1083373270 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f3000ad40 con 0x7f1f34143f50 2024-09-13T16:31:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f340c40a0 con 0x7f1f34143f50 2024-09-13T16:31:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f341378d0 con 0x7f1f34143f50 2024-09-13T16:31:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f2a7fc700 1 -- 172.21.15.119:0/1083373270 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1f30005070 con 0x7f1f34143f50 2024-09-13T16:31:47.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.188+0000 7f1f2a7fc700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1f18066730 0x7f1f18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f2a7fc700 1 -- 172.21.15.119:0/1083373270 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1f30083f00 con 0x7f1f34143f50 2024-09-13T16:31:47.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f399c2700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1f18066730 0x7f1f18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f399c2700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1f18066730 0x7f1f18068bf0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1f24005930 tx=0x7f1f24009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 Processor -- start 2024-09-13T16:31:47.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 -- start start 2024-09-13T16:31:47.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baa230 0x2baa610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b88e90 0x2b72890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2b72dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2ad1d80 con 0x2baa230 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2acf510 con 0x2b88e90 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2acae00 con 0x2baab50 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f399c2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baa230 0x2baa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f399c2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baa230 0x2baa610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37914/0 (socket says 172.21.15.119:37914) 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f399c2700 1 -- 172.21.15.119:0/709511675 learned_addr learned my addr 172.21.15.119:0/709511675 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f391c1700 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b88e90 0x2b72890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.192+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2b72dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f399c2700 1 -- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 msgr2=0x2b72dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f399c2700 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2b72dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f399c2700 1 -- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b88e90 msgr2=0x2b72890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f399c2700 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b88e90 0x2b72890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f399c2700 1 -- 172.21.15.119:0/709511675 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x293d960 con 0x2baa230 2024-09-13T16:31:47.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2b72dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:47.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f399c2700 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baa230 0x2baa610 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f1f24005aa0 tx=0x7f1f24011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bf468af263a7cbf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f177fe700 1 -- 172.21.15.119:0/709511675 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f24007560 con 0x2baa230 2024-09-13T16:31:47.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f177fe700 1 -- 172.21.15.119:0/709511675 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f240076c0 con 0x2baa230 2024-09-13T16:31:47.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f177fe700 1 -- 172.21.15.119:0/709511675 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2400e820 con 0x2baa230 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 -- 172.21.15.119:0/709511675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baa230 msgr2=0x2baa610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baa230 0x2baa610 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7f1f24005aa0 tx=0x7f1f24011000 comp rx=0 tx=0).stop 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 -- 172.21.15.119:0/709511675 shutdown_connections 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2b72dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2b88e90 0x2b72890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/709511675 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baa230 0x2baa610 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 -- 172.21.15.119:0/709511675 >> 172.21.15.119:0/709511675 conn(0x28f24d0 msgr2=0x2b8be30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 -- 172.21.15.119:0/709511675 shutdown_connections 2024-09-13T16:31:47.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 -- 172.21.15.119:0/709511675 wait complete. 2024-09-13T16:31:47.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 -- start start 2024-09-13T16:31:47.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.196+0000 7f1f3eed7740 1 Processor -- start 2024-09-13T16:31:47.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b74f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba29c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a88730 con 0x2baab50 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a8e5b0 con 0x2baa230 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2976aa0 con 0x2b88e90 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 10 client.0 _ll_register_callbacks cb 0x2ac4610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f399c2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f391c1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b74f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37922/0 (socket says 172.21.15.119:37922) 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/2648834077 learned_addr learned my addr 172.21.15.119:0/2648834077 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/2648834077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 msgr2=0x2ba9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/2648834077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 msgr2=0x2b74f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b74f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/2648834077 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2400f040 con 0x2baab50 2024-09-13T16:31:47.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba29c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c012000 tx=0x7f1f2c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1be31291c2247276 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f14ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:47.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2648834077 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c01a800 con 0x2baab50 2024-09-13T16:31:47.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2648834077 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1f2c01f040 con 0x2baab50 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x28ff1e0 con 0x2baab50 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2648834077 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f2c011d30 con 0x2baab50 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 10 client.8184 fetch_fsmap learned FSMap version 62 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x2a91220 con 0x2baab50 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2648834077 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x28ff1e0 con 0x2baab50 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2648834077 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1f2c023050 con 0x2baab50 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 10 client.8184 fetch_fsmap finished waiting for FSMap version 62 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 10 client.8184 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 20 client.8184 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2b63050 0x2b5ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 4 client.8184 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2949460 con 0x2b63050 2024-09-13T16:31:47.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f399c2700 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2b63050 0x2b5ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f399c2700 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2b63050 0x2b5ff10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c82c725c9b61c20e server_cookie=b46c2c8b82130915 in_seq=0 out_seq=0 2024-09-13T16:31:47.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.200+0000 7f1f14ff9700 10 client.8184 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:47.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2648834077 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x2949460 con 0x2b63050 2024-09-13T16:31:47.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f14ff9700 10 client.8184 handle_command_reply: tid=0 2024-09-13T16:31:47.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 client.8184 shutdown 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 msgr2=0x2ba29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba29c0 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c012000 tx=0x7f1f2c00e000 comp rx=0 tx=0).stop 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 shutdown_connections 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2b63050 0x2b5ff10 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba29c0 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f14ff9700 0 client.8184 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b74f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2648834077 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 >> 172.21.15.119:0/2648834077 conn(0x28f24d0 msgr2=0x2ba32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 shutdown_connections 2024-09-13T16:31:47.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2648834077 wait complete. 2024-09-13T16:31:47.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.204+0000 7f1f3eed7740 20 client.8184 trim_cache size 0 max 16384 2024-09-13T16:31:47.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 Processor -- start 2024-09-13T16:31:47.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- start start 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b5ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a84c30 con 0x2baab50 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2ac2b70 con 0x2baa230 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a89e70 con 0x2b88e90 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37926/0 (socket says 172.21.15.119:37926) 2024-09-13T16:31:47.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/3991503431 learned_addr learned my addr 172.21.15.119:0/3991503431 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:47.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f399c2700 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f391c1700 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b5ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 msgr2=0x2ba9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 msgr2=0x2b5ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b5ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 -- 172.21.15.119:0/3991503431 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2a77450 con 0x2baab50 2024-09-13T16:31:47.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba16a0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0116d0 tx=0x7f1f2c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22c16dd2071cdba7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f17fff700 1 -- 172.21.15.119:0/3991503431 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c00b160 con 0x2baab50 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f17fff700 1 -- 172.21.15.119:0/3991503431 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f2c01a600 con 0x2baab50 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f17fff700 1 -- 172.21.15.119:0/3991503431 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c011df0 con 0x2baab50 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- 172.21.15.119:0/3991503431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 msgr2=0x2ba16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba16a0 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c0116d0 tx=0x7f1f2c008030 comp rx=0 tx=0).stop 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- 172.21.15.119:0/3991503431 shutdown_connections 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2baab50 0x2ba16a0 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2b5ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/3991503431 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2b88e90 0x2ba9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- 172.21.15.119:0/3991503431 >> 172.21.15.119:0/3991503431 conn(0x28f24d0 msgr2=0x2b5e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- 172.21.15.119:0/3991503431 shutdown_connections 2024-09-13T16:31:47.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.224+0000 7f1f3eed7740 1 -- 172.21.15.119:0/3991503431 wait complete. 2024-09-13T16:31:47.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 -- start start 2024-09-13T16:31:47.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 Processor -- start 2024-09-13T16:31:47.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b88e90 0x2baf960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2c0ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2c0f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2bafea0 con 0x2b88e90 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2ba6300 con 0x2baa230 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2ba6530 con 0x2baab50 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b88e90 0x2baf960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3eed7740 10 client.0 _ll_register_callbacks cb 0x2a7cce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b88e90 0x2baf960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37936/0 (socket says 172.21.15.119:37936) 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 -- 172.21.15.119:0/2004999024 learned_addr learned my addr 172.21.15.119:0/2004999024 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2c0f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f391c1700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2c0ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 -- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 msgr2=0x2c0f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2c0f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 -- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 msgr2=0x2c0ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2c0ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f399c2700 1 -- 172.21.15.119:0/2004999024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x2a91220 con 0x2b88e90 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.228+0000 7f1f3a1c3700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2c0f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f399c2700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b88e90 0x2baf960 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f1f24001eb0 tx=0x7f1f24000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2828812855cd7a0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2004999024 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f2402e690 con 0x2b88e90 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2004999024 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f1f2402e7f0 con 0x2b88e90 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1f240076c0 con 0x2b88e90 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2004999024 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1f2400e620 con 0x2b88e90 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 10 client.8190 fetch_fsmap learned FSMap version 62 2024-09-13T16:31:47.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x2a6b720 con 0x2b88e90 2024-09-13T16:31:47.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2004999024 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f240076c0 con 0x2b88e90 2024-09-13T16:31:47.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2004999024 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1f2400e780 con 0x2b88e90 2024-09-13T16:31:47.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 10 client.8190 fetch_fsmap finished waiting for FSMap version 62 2024-09-13T16:31:47.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 10 client.8190 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:31:47.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 20 client.8190 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:47.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2c0c3b0 0x2bacf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 4 client.8190 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "status"}) v1 -- 0x296e1e0 con 0x2c0c3b0 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f391c1700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2c0c3b0 0x2bacf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f391c1700 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2c0c3b0 0x2bacf40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc6956175d84be57 server_cookie=d08ab75443eca254 in_seq=0 out_seq=0 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 10 client.8190 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 1 -- 172.21.15.119:0/2004999024 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x296e1e0 con 0x2c0c3b0 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 10 client.8190 handle_command_reply: tid=0 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 client.8190 shutdown 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b88e90 msgr2=0x2baf960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b88e90 0x2baf960 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7f1f24001eb0 tx=0x7f1f24000ed0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 shutdown_connections 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2c0c3b0 0x2bacf40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2baab50 0x2c0f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2baa230 0x2c0ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 --2- 172.21.15.119:0/2004999024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2b88e90 0x2baf960 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f14ff9700 0 client.8190 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 >> 172.21.15.119:0/2004999024 conn(0x28f24d0 msgr2=0x2ba9290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 shutdown_connections 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 1 -- 172.21.15.119:0/2004999024 wait complete. 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3eed7740 20 client.8190 trim_cache size 0 max 16384 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:31:47.241 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 7881, 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 4.4905387980000002, 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 62, 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 96, 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 96, 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 22.761696613000002 2024-09-13T16:31:47.242 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1f18066730 msgr2=0x7f1f18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1f18066730 0x7f1f18068bf0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f1f24005930 tx=0x7f1f24009040 comp rx=0 tx=0).stop 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f34143f50 msgr2=0x7f1f340c8010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.232+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f34143f50 0x7f1f340c8010 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 crypto rx=0x7f1f30001f90 tx=0x7f1f3000a340 comp rx=0 tx=0).stop 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 shutdown_connections 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1f18066730 0x7f1f18068bf0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1f34144330 0x7f1f340ac480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1f34143f50 0x7f1f340c8010 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 --2- 172.21.15.119:0/1083373270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1f340b3410 0x7f1f3415a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 >> 172.21.15.119:0/1083373270 conn(0x7f1f3406a350 msgr2=0x7f1f340ae440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 shutdown_connections 2024-09-13T16:31:47.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.236+0000 7f1f3a9c4700 1 -- 172.21.15.119:0/1083373270 wait complete. 2024-09-13T16:31:47.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:31:47.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 Processor -- start 2024-09-13T16:31:47.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 -- start start 2024-09-13T16:31:47.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f232c15ec20 0x7f232c161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f232c071210 con 0x7f232c164a70 2024-09-13T16:31:47.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f232c0714f0 con 0x7f232c15ec20 2024-09-13T16:31:47.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c0717d0 con 0x7f232c15e840 2024-09-13T16:31:47.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331d85700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f232c164a70 0x7f232c0ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:39776/0 (socket says 172.21.15.119:39776) 2024-09-13T16:31:47.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331d85700 1 -- 172.21.15.119:0/1063260169 learned_addr learned my addr 172.21.15.119:0/1063260169 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:31:47.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331584700 1 --2- 172.21.15.119:0/1063260169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2330d83700 1 --2- 172.21.15.119:0/1063260169 >> v2:172.21.15.145:3300/0 conn(0x7f232c15ec20 0x7f232c161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2323fff700 1 -- 172.21.15.119:0/1063260169 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1974338630 0 0) 0x7f232c071210 con 0x7f232c164a70 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2323fff700 1 -- 172.21.15.119:0/1063260169 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23180035d0 con 0x7f232c164a70 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331584700 1 -- 172.21.15.119:0/1063260169 >> v2:172.21.15.145:3300/0 conn(0x7f232c15ec20 msgr2=0x7f232c161f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331584700 1 --2- 172.21.15.119:0/1063260169 >> v2:172.21.15.145:3300/0 conn(0x7f232c15ec20 0x7f232c161f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331584700 1 -- 172.21.15.119:0/1063260169 >> v1:172.21.15.119:6789/0 conn(0x7f232c164a70 legacy=0x7f232c0ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331584700 1 -- 172.21.15.119:0/1063260169 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f232c071ab0 con 0x7f232c15e840 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2331584700 1 --2- 172.21.15.119:0/1063260169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c070b80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2328009850 tx=0x7f2328009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66537a0c4b971cac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2323fff700 1 -- 172.21.15.119:0/1063260169 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328011830 con 0x7f232c15e840 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2323fff700 1 -- 172.21.15.119:0/1063260169 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2328011990 con 0x7f232c15e840 2024-09-13T16:31:47.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f2323fff700 1 -- 172.21.15.119:0/1063260169 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2328011d00 con 0x7f232c15e840 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 -- 172.21.15.119:0/1063260169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 msgr2=0x7f232c070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 --2- 172.21.15.119:0/1063260169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c070b80 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2328009850 tx=0x7f2328009b60 comp rx=0 tx=0).stop 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 -- 172.21.15.119:0/1063260169 shutdown_connections 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 --2- 172.21.15.119:0/1063260169 >> v2:172.21.15.145:3300/0 conn(0x7f232c15ec20 0x7f232c161f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.372+0000 7f23337e8700 1 --2- 172.21.15.119:0/1063260169 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c070b80 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.376+0000 7f23337e8700 1 -- 172.21.15.119:0/1063260169 >> 172.21.15.119:0/1063260169 conn(0x7f232c06a350 msgr2=0x7f232c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.376+0000 7f23337e8700 1 -- 172.21.15.119:0/1063260169 shutdown_connections 2024-09-13T16:31:47.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.376+0000 7f23337e8700 1 -- 172.21.15.119:0/1063260169 wait complete. 2024-09-13T16:31:47.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.376+0000 7f23337e8700 1 Processor -- start 2024-09-13T16:31:47.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.376+0000 7f23337e8700 1 -- start start 2024-09-13T16:31:47.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c1f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f232c164a70 0x7f232c153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f232c157100 0x7f232c160440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c0cc000 con 0x7f232c164a70 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f232c1f6e90 con 0x7f232c157100 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f232c1f7170 con 0x7f232c15e840 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f232c164a70 0x7f232c153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f232c164a70 0x7f232c153e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37948/0 (socket says 172.21.15.119:37948) 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 -- 172.21.15.119:0/39378977 learned_addr learned my addr 172.21.15.119:0/39378977 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2331584700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c1f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2331d85700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f232c157100 0x7f232c160440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 -- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 msgr2=0x7f232c1f6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c1f6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 -- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f232c157100 msgr2=0x7f232c160440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f232c157100 0x7f232c160440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 -- 172.21.15.119:0/39378977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23280094b0 con 0x7f232c164a70 2024-09-13T16:31:47.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2330d83700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f232c164a70 0x7f232c153e70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f231c00e930 tx=0x7f231c00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2321ffb700 1 -- 172.21.15.119:0/39378977 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c013680 con 0x7f232c164a70 2024-09-13T16:31:47.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f232c1f7450 con 0x7f232c164a70 2024-09-13T16:31:47.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232c1575b0 con 0x7f232c164a70 2024-09-13T16:31:47.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2321ffb700 1 -- 172.21.15.119:0/39378977 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f231c0137e0 con 0x7f232c164a70 2024-09-13T16:31:47.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f2321ffb700 1 -- 172.21.15.119:0/39378977 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f231c013b70 con 0x7f232c164a70 2024-09-13T16:31:47.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.380+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230c005310 con 0x7f232c164a70 2024-09-13T16:31:47.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.384+0000 7f2321ffb700 1 -- 172.21.15.119:0/39378977 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f231c013cd0 con 0x7f232c164a70 2024-09-13T16:31:47.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.384+0000 7f2321ffb700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2310066760 0x7f2310068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:47.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.384+0000 7f2321ffb700 1 -- 172.21.15.119:0/39378977 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f231c0842a0 con 0x7f232c164a70 2024-09-13T16:31:47.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.384+0000 7f2331584700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2310066760 0x7f2310068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:47.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.384+0000 7f2331584700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2310066760 0x7f2310068c20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f23280090b0 tx=0x7f2328008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:47.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.392+0000 7f2321ffb700 1 -- 172.21.15.119:0/39378977 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f231c051350 con 0x7f232c164a70 2024-09-13T16:31:47.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f230c005dd0 con 0x7f232c164a70 2024-09-13T16:31:47.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f2321ffb700 1 -- 172.21.15.119:0/39378977 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v62) v1 ==== 53+0+73 (secure 0 0 0) 0x7f231c054970 con 0x7f232c164a70 2024-09-13T16:31:47.604 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:31:47.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2310066760 msgr2=0x7f2310068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2310066760 0x7f2310068c20 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f23280090b0 tx=0x7f2328008030 comp rx=0 tx=0).stop 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f232c164a70 msgr2=0x7f232c153e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f232c164a70 0x7f232c153e70 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f231c00e930 tx=0x7f231c00ec40 comp rx=0 tx=0).stop 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 shutdown_connections 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2310066760 0x7f2310068c20 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f232c157100 0x7f232c160440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f232c164a70 0x7f232c153e70 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 --2- 172.21.15.119:0/39378977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f232c15e840 0x7f232c1f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 >> 172.21.15.119:0/39378977 conn(0x7f232c06a350 msgr2=0x7f232c15f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 shutdown_connections 2024-09-13T16:31:47.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:47.600+0000 7f23337e8700 1 -- 172.21.15.119:0/39378977 wait complete. 2024-09-13T16:31:47.618 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:31:47.618 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:31:47.619 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:31:47.619 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:31:47.619 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:47.619 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:31:47.619 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:31:47.619 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:31:47.619 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:47.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.619 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:31:47.627 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:31:47.627 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:31:47.627 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:47.628 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42233sec preferred_lft 42233sec 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:31:47.629 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:31:47.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.630 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:31:47.643 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:31:47.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.645 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:31:47.695 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:31:47.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.697 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:31:47.697 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:31:47.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:31:47.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:31:47.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:31:47.697 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:31:47.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.740 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:31:47.747 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:31:47.747 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:31:47.748 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:47.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.748 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:31:47.760 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:31:47.760 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:31:47.760 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:47.767 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:31:47.767 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:31:47.767 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:47.817 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:47.832 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:47.832 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:47.848 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:47.849 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:47.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.850 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:47.855 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:31:47.855 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:31:47.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:47.917 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:47.918 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:47.926 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:47.927 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:47.928 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:47.928 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:47.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:31:47.992+0000 7f6646524640 -1 init, newargv = 0x55c0de0656f0 newargc=15 2024-09-13T16:31:47.996 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[47917]: starting ceph client 2024-09-13T16:31:48.004 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[47917]: starting fuse 2024-09-13T16:31:49.033 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:31:49.052 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:31:49.052 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:31:49.111 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:31:49.111 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:31:49.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:49.112 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:31:49.117 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:31:49.118 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:31:49.118 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:31:49.119 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:31:49.120 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:31:49.120 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:31:49.120 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:31:49.202 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.47917.asok 2024-09-13T16:31:49.386 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.47917.asok 2024-09-13T16:31:49.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:49.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.47917.asok status 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "47917", 2024-09-13T16:31:49.531 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 8202, 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 8202 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:31:49.532 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 905506256 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 905506256 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.8202 192.168.0.1:0/905506256", 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/905506256", 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 62, 2024-09-13T16:31:49.533 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 96, 2024-09-13T16:31:49.534 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:31:49.534 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:31:49.534 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:31:49.534 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:49.541 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:49.541 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:31:49.541 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:49.548 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:31:49.549 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:49.549 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:49.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:31:49.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:31:49.564 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec stat testdir/hardlink) 2024-09-13T16:31:49.577 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:31:49.572+0000 7fa101b5c700 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-09-13T16:31:49.579 INFO:teuthology.orchestra.run.smithi119.stderr:stat: cannot stat 'testdir/hardlink': Input/output error 2024-09-13T16:31:49.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:31:49.580 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive 2024-09-13T16:31:49.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 Processor -- start 2024-09-13T16:31:49.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- start start 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9500b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe95006fe30 0x7fe9500cb320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9500705f0 con 0x7fe9500cb860 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9500cbcd0 con 0x7fe95006fe30 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9500708d0 con 0x7fe95006d9d0 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe95006fe30 0x7fe9500cb320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe95006fe30 0x7fe9500cb320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37956/0 (socket says 172.21.15.119:37956) 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 -- 172.21.15.119:0/1157735701 learned_addr learned my addr 172.21.15.119:0/1157735701 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9559b2700 1 --2- 172.21.15.119:0/1157735701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9500b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 -- 172.21.15.119:0/1157735701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 msgr2=0x7fe9500b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 --2- 172.21.15.119:0/1157735701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9500b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 -- 172.21.15.119:0/1157735701 >> v1:172.21.15.119:6789/0 conn(0x7fe9500cb860 legacy=0x7fe9500cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 -- 172.21.15.119:0/1157735701 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe950070bb0 con 0x7fe95006fe30 2024-09-13T16:31:49.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9551b1700 1 --2- 172.21.15.119:0/1157735701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe9500cb320 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009910 tx=0x7fe94c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21cf5aff618170b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9549b0700 1 -- 172.21.15.119:0/1157735701 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c00ba60 con 0x7fe95006fe30 2024-09-13T16:31:49.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9549b0700 1 -- 172.21.15.119:0/1157735701 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c00bbc0 con 0x7fe95006fe30 2024-09-13T16:31:49.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9549b0700 1 -- 172.21.15.119:0/1157735701 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c009e60 con 0x7fe95006fe30 2024-09-13T16:31:49.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- 172.21.15.119:0/1157735701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 msgr2=0x7fe9500cb320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1157735701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe9500cb320 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 crypto rx=0x7fe94c009910 tx=0x7fe94c009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:49.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- 172.21.15.119:0/1157735701 shutdown_connections 2024-09-13T16:31:49.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1157735701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe9500cb320 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1157735701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9500b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- 172.21.15.119:0/1157735701 >> 172.21.15.119:0/1157735701 conn(0x7fe95006a350 msgr2=0x7fe95006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- 172.21.15.119:0/1157735701 shutdown_connections 2024-09-13T16:31:49.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.696+0000 7fe9569b4700 1 -- 172.21.15.119:0/1157735701 wait complete. 2024-09-13T16:31:49.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 Processor -- start 2024-09-13T16:31:49.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 -- start start 2024-09-13T16:31:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9501e95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe95006c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9500cb860 0x7fe95006c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9501ec650 con 0x7fe95006fe30 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9501eaa70 con 0x7fe9500cb860 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9569b4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9501e9df0 con 0x7fe95006d9d0 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe95006c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe95006c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37972/0 (socket says 172.21.15.119:37972) 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 -- 172.21.15.119:0/1176667421 learned_addr learned my addr 172.21.15.119:0/1176667421 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9559b2700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9501e95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9561b3700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9500cb860 0x7fe95006c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 -- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 msgr2=0x7fe9501e95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9501e95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 -- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9500cb860 msgr2=0x7fe95006c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9500cb860 0x7fe95006c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9551b1700 1 -- 172.21.15.119:0/1176667421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe94c009570 con 0x7fe95006fe30 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.700+0000 7fe9561b3700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9500cb860 0x7fe95006c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9551b1700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe95006c440 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe94c000c00 tx=0x7fe94c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9427fc700 1 -- 172.21.15.119:0/1176667421 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c01a840 con 0x7fe95006fe30 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9427fc700 1 -- 172.21.15.119:0/1176667421 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c01f070 con 0x7fe95006fe30 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9501ea0d0 con 0x7fe95006fe30 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9427fc700 1 -- 172.21.15.119:0/1176667421 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c01ac20 con 0x7fe95006fe30 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9501ea690 con 0x7fe95006fe30 2024-09-13T16:31:49.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9427fc700 1 -- 172.21.15.119:0/1176667421 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe94c026480 con 0x7fe95006fe30 2024-09-13T16:31:49.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9427fc700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe934066730 0x7fe934068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9427fc700 1 -- 172.21.15.119:0/1176667421 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe94c021070 con 0x7fe95006fe30 2024-09-13T16:31:49.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.704+0000 7fe9559b2700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe934066730 0x7fe934068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9559b2700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe934066730 0x7fe934068bf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe950163670 tx=0x7fe944008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 Processor -- start 2024-09-13T16:31:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 -- start start 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0bd0 0xe26330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0fb0 0xe28880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe2b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xd4ceb0 con 0xde0fb0 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xd4a640 con 0xe28dc0 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xd45f30 con 0xde0bd0 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0fb0 0xe28880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9559b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0bd0 0xe26330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0fb0 0xe28880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37974/0 (socket says 172.21.15.119:37974) 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 -- 172.21.15.119:0/3041904345 learned_addr learned my addr 172.21.15.119:0/3041904345 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9561b3700 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe2b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 -- 172.21.15.119:0/3041904345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0bd0 msgr2=0xe26330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0bd0 0xe26330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 -- 172.21.15.119:0/3041904345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 msgr2=0xe2b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe2b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 -- 172.21.15.119:0/3041904345 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xd6c950 con 0xde0fb0 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0fb0 0xe28880 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fe94c083e60 tx=0x7fe94c0562d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62f15e439d0377bf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9337fe700 1 -- 172.21.15.119:0/3041904345 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c057070 con 0xde0fb0 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9337fe700 1 -- 172.21.15.119:0/3041904345 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c056970 con 0xde0fb0 2024-09-13T16:31:49.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.708+0000 7fe9337fe700 1 -- 172.21.15.119:0/3041904345 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe94c05a7a0 con 0xde0fb0 2024-09-13T16:31:49.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- 172.21.15.119:0/3041904345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0fb0 msgr2=0xe28880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0fb0 0xe28880 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 crypto rx=0x7fe94c083e60 tx=0x7fe94c0562d0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- 172.21.15.119:0/3041904345 shutdown_connections 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe2b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0fb0 0xe28880 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3041904345 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0bd0 0xe26330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- 172.21.15.119:0/3041904345 >> 172.21.15.119:0/3041904345 conn(0xb6d700 msgr2=0xe07470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- 172.21.15.119:0/3041904345 shutdown_connections 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- 172.21.15.119:0/3041904345 wait complete. 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- start start 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 Processor -- start 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0bd0 0xe03dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe04310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe28dc0 0xe09260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xd09740 con 0xe28dc0 2024-09-13T16:31:49.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xd093e0 con 0xde0bd0 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xbf1cd0 con 0xde0fb0 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe95aec7740 10 client.0 _ll_register_callbacks cb 0xd97ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe28dc0 0xe09260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe28dc0 0xe09260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37976/0 (socket says 172.21.15.119:37976) 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 -- 172.21.15.119:0/3056686457 learned_addr learned my addr 172.21.15.119:0/3056686457 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe04310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0bd0 0xe03dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 -- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 msgr2=0xe04310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe04310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 -- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0bd0 msgr2=0xe03dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0bd0 0xe03dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 -- 172.21.15.119:0/3056686457 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9501ea690 con 0xe28dc0 2024-09-13T16:31:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe9561b3700 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe28dc0 0xe09260 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe9480014d0 tx=0x7fe948011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc1ce34ea8aef694 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe930ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:49.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.712+0000 7fe930ff9700 1 -- 172.21.15.119:0/3056686457 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe948003c30 con 0xe28dc0 2024-09-13T16:31:49.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 1 -- 172.21.15.119:0/3056686457 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe948003d90 con 0xe28dc0 2024-09-13T16:31:49.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xb7a410 con 0xe28dc0 2024-09-13T16:31:49.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 1 -- 172.21.15.119:0/3056686457 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe9480050d0 con 0xe28dc0 2024-09-13T16:31:49.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 10 client.8217 fetch_fsmap learned FSMap version 62 2024-09-13T16:31:49.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0xbb8b90 con 0xe28dc0 2024-09-13T16:31:49.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 1 -- 172.21.15.119:0/3056686457 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xb7a410 con 0xe28dc0 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 1 -- 172.21.15.119:0/3056686457 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe948005230 con 0xe28dc0 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 10 client.8217 fetch_fsmap finished waiting for FSMap version 62 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 10 client.8217 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 20 client.8217 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe09c90 0xddaaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 4 client.8217 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xbc4690 con 0xe09c90 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe09c90 0xddaaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 10 client.8217 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe09c90 0xddaaa0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29236c5584adbeb6 server_cookie=8f959b988fe06268 in_seq=0 out_seq=0 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 1 -- 172.21.15.119:0/3056686457 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0xbc4690 con 0xe09c90 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 10 client.8217 handle_command_reply: tid=0 2024-09-13T16:31:49.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 client.8217 shutdown 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe28dc0 msgr2=0xe09260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe28dc0 0xe09260 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe9480014d0 tx=0x7fe948011ca0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 shutdown_connections 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe09c90 0xddaaa0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe28dc0 0xe09260 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe04310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3056686457 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0bd0 0xe03dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe930ff9700 0 client.8217 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 >> 172.21.15.119:0/3056686457 conn(0xb6d700 msgr2=0xe281e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 shutdown_connections 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 1 -- 172.21.15.119:0/3056686457 wait complete. 2024-09-13T16:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.716+0000 7fe95aec7740 20 client.8217 trim_cache size 0 max 16384 2024-09-13T16:31:49.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 Processor -- start 2024-09-13T16:31:49.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 -- start start 2024-09-13T16:31:49.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe81550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe81a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe028b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xd0ae90 con 0xde0bd0 2024-09-13T16:31:49.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xdd71b0 con 0xe28dc0 2024-09-13T16:31:49.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xcedfb0 con 0xde0fb0 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe81550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe81550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:37992/0 (socket says 172.21.15.119:37992) 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 -- 172.21.15.119:0/3462184025 learned_addr learned my addr 172.21.15.119:0/3462184025 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe81a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9561b3700 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe028b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 -- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 msgr2=0xe81a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe81a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 -- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 msgr2=0xe028b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe028b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 -- 172.21.15.119:0/3462184025 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xd0c350 con 0xde0bd0 2024-09-13T16:31:49.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe81550 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe944011540 tx=0x7fe944017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244271b1fb639bf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe933fff700 1 -- 172.21.15.119:0/3462184025 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe944028040 con 0xde0bd0 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe933fff700 1 -- 172.21.15.119:0/3462184025 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94402ad70 con 0xde0bd0 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.740+0000 7fe933fff700 1 -- 172.21.15.119:0/3462184025 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9440042f0 con 0xde0bd0 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 -- 172.21.15.119:0/3462184025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 msgr2=0xe81550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe81550 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe944011540 tx=0x7fe944017000 comp rx=0 tx=0).stop 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 -- 172.21.15.119:0/3462184025 shutdown_connections 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xe28dc0 0xe028b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xde0fb0 0xe81a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3462184025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe81550 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 -- 172.21.15.119:0/3462184025 >> 172.21.15.119:0/3462184025 conn(0xb6d700 msgr2=0xe28160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 -- 172.21.15.119:0/3462184025 shutdown_connections 2024-09-13T16:31:49.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 -- 172.21.15.119:0/3462184025 wait complete. 2024-09-13T16:31:49.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 -- start start 2024-09-13T16:31:49.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.744+0000 7fe95aec7740 1 Processor -- start 2024-09-13T16:31:49.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe7e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0fb0 0xe86a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe28dc0 0xe86f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xd41720 con 0xde0bd0 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xd07030 con 0xde0fb0 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xe80890 con 0xe28dc0 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 10 client.0 _ll_register_callbacks cb 0xcf7e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe7e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe7e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38008/0 (socket says 172.21.15.119:38008) 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 -- 172.21.15.119:0/3612977933 learned_addr learned my addr 172.21.15.119:0/3612977933 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9561b3700 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe28dc0 0xe86f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0fb0 0xe86a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 -- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe28dc0 msgr2=0xe86f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe28dc0 0xe86f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 -- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0fb0 msgr2=0xe86a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0fb0 0xe86a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 -- 172.21.15.119:0/3612977933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe94400e040 con 0xde0bd0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9559b2700 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe7e880 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe9440118d0 tx=0x7fe9440047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eddbbcd0332e3a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe930ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe930ff9700 1 -- 172.21.15.119:0/3612977933 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9440157d0 con 0xde0bd0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe930ff9700 1 -- 172.21.15.119:0/3612977933 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe94401b040 con 0xde0bd0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe94c056970 con 0xde0bd0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe930ff9700 1 -- 172.21.15.119:0/3612977933 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe944015930 con 0xde0bd0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 10 client.8223 fetch_fsmap learned FSMap version 62 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0xcf2580 con 0xde0bd0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe930ff9700 1 -- 172.21.15.119:0/3612977933 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c056970 con 0xde0bd0 2024-09-13T16:31:49.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe930ff9700 1 -- 172.21.15.119:0/3612977933 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe944015a90 con 0xde0bd0 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 10 client.8223 fetch_fsmap finished waiting for FSMap version 62 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 10 client.8223 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 20 client.8223 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe874b0 0xe87890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 4 client.8223 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}] 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive"]}) v1 -- 0xbe9410 con 0xe874b0 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe874b0 0xe87890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe930ff9700 10 client.8223 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.748+0000 7fe9551b1700 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe874b0 0xe87890 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5aecc57f71ae0fd server_cookie=e190c8fbcab684b5 in_seq=0 out_seq=0 2024-09-13T16:31:49.757 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fa101b5c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0) n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1) (iversion lock) | dirfrag=0 authpin=1 scrubqueue=0 0x5589ea977b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe930ff9700 1 -- 172.21.15.119:0/3612977933 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0xbe9410 con 0xe874b0 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe930ff9700 10 client.8223 handle_command_reply: tid=0 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 client.8223 shutdown 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 msgr2=0xe7e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe7e880 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7fe9440118d0 tx=0x7fe9440047e0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 shutdown_connections 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0xe874b0 0xe87890 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe28dc0 0xe86f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe930ff9700 0 client.8223 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xde0fb0 0xe86a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 --2- 172.21.15.119:0/3612977933 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xde0bd0 0xe7e880 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 >> 172.21.15.119:0/3612977933 conn(0xb6d700 msgr2=0xe03b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 shutdown_connections 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 1 -- 172.21.15.119:0/3612977933 wait complete. 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe95aec7740 20 client.8223 trim_cache size 0 max 16384 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "695d5313-cc5c-43cb-aec0-996bc8c00804", 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:31:49.759 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:49.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe934066730 msgr2=0x7fe934068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe934066730 0x7fe934068bf0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7fe950163670 tx=0x7fe944008040 comp rx=0 tx=0).stop 2024-09-13T16:31:49.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 msgr2=0x7fe95006c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe95006c440 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7fe94c000c00 tx=0x7fe94c004770 comp rx=0 tx=0).stop 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 shutdown_connections 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe934066730 0x7fe934068bf0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9500cb860 0x7fe95006c980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe95006fe30 0x7fe95006c440 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 --2- 172.21.15.119:0/1176667421 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe95006d9d0 0x7fe9501e95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 >> 172.21.15.119:0/1176667421 conn(0x7fe95006a350 msgr2=0x7fe9501637c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 shutdown_connections 2024-09-13T16:31:49.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.752+0000 7fe9569b4700 1 -- 172.21.15.119:0/1176667421 wait complete. 2024-09-13T16:31:49.773 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-09-13T16:31:49.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.892+0000 7fe70cdc0700 1 Processor -- start 2024-09-13T16:31:49.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.892+0000 7fe70cdc0700 1 -- start start 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70806bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe7080d1640 0x7fe7080d1af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe70806c070 con 0x7fe70815cfb0 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7080cee40 con 0x7fe7080d1640 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7080d1220 con 0x7fe70806b710 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70806bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe706d9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe7080d1640 0x7fe7080d1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70806bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:52192/0 (socket says 172.21.15.119:52192) 2024-09-13T16:31:49.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 -- 172.21.15.119:0/1104464659 learned_addr learned my addr 172.21.15.119:0/1104464659 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 -- 172.21.15.119:0/1104464659 >> v2:172.21.15.145:3300/0 conn(0x7fe7080d1640 msgr2=0x7fe7080d1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1104464659 >> v2:172.21.15.145:3300/0 conn(0x7fe7080d1640 0x7fe7080d1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 -- 172.21.15.119:0/1104464659 >> v1:172.21.15.145:6789/0 conn(0x7fe70815cfb0 legacy=0x7fe70806ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 -- 172.21.15.119:0/1104464659 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7080ce880 con 0x7fe70806b710 2024-09-13T16:31:49.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1104464659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70806bb30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009910 tx=0x7fe6fc009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8c0debdb9beb9b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70559a700 1 -- 172.21.15.119:0/1104464659 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc011830 con 0x7fe70806b710 2024-09-13T16:31:49.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70559a700 1 -- 172.21.15.119:0/1104464659 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc011990 con 0x7fe70806b710 2024-09-13T16:31:49.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70559a700 1 -- 172.21.15.119:0/1104464659 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc011d40 con 0x7fe70806b710 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- 172.21.15.119:0/1104464659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 msgr2=0x7fe70806bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/1104464659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70806bb30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009910 tx=0x7fe6fc009c20 comp rx=0 tx=0).stop 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- 172.21.15.119:0/1104464659 shutdown_connections 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/1104464659 >> v2:172.21.15.145:3300/0 conn(0x7fe7080d1640 0x7fe7080d1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/1104464659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70806bb30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- 172.21.15.119:0/1104464659 >> 172.21.15.119:0/1104464659 conn(0x7fe7080d2da0 msgr2=0x7fe7080d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- 172.21.15.119:0/1104464659 shutdown_connections 2024-09-13T16:31:49.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.896+0000 7fe70cdc0700 1 -- 172.21.15.119:0/1104464659 wait complete. 2024-09-13T16:31:49.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 Processor -- start 2024-09-13T16:31:49.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 -- start start 2024-09-13T16:31:49.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70815dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7080d1640 0x7fe7080b7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe70815cfb0 0x7fe7081f61f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe70815e260 con 0x7fe70815cfb0 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7080b8120 con 0x7fe7080d1640 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70cdc0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7081f4310 con 0x7fe70806b710 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe70815cfb0 0x7fe7081f61f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe70815cfb0 0x7fe7081f61f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38020/0 (socket says 172.21.15.119:38020) 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 -- 172.21.15.119:0/3781970358 learned_addr learned my addr 172.21.15.119:0/3781970358 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe70659c700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70815dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe705d9b700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7080d1640 0x7fe7080b7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 -- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 msgr2=0x7fe70815dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70815dd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 -- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7080d1640 msgr2=0x7fe7080b7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7080d1640 0x7fe7080b7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 -- 172.21.15.119:0/3781970358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6fc009570 con 0x7fe70815cfb0 2024-09-13T16:31:49.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.900+0000 7fe706d9d700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe70815cfb0 0x7fe7081f61f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fe70000efc0 tx=0x7fe70000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.909 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fa105363700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:31:49.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe6f6ffd700 1 -- 172.21.15.119:0/3781970358 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe700009ea0 con 0x7fe70815cfb0 2024-09-13T16:31:49.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe6f6ffd700 1 -- 172.21.15.119:0/3781970358 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe70000f040 con 0x7fe70815cfb0 2024-09-13T16:31:49.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7081f45f0 con 0x7fe70815cfb0 2024-09-13T16:31:49.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe6f6ffd700 1 -- 172.21.15.119:0/3781970358 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70000ced0 con 0x7fe70815cfb0 2024-09-13T16:31:49.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7081f4bb0 con 0x7fe70815cfb0 2024-09-13T16:31:49.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe6f6ffd700 1 -- 172.21.15.119:0/3781970358 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe700014480 con 0x7fe70815cfb0 2024-09-13T16:31:49.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe6f6ffd700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6e8066730 0x7fe6e8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe6f6ffd700 1 -- 172.21.15.119:0/3781970358 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe700084b70 con 0x7fe70815cfb0 2024-09-13T16:31:49.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.904+0000 7fe70659c700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6e8066730 0x7fe6e8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6e8066730 0x7fe6e8068bf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe7080b7b70 tx=0x7fe6f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 Processor -- start 2024-09-13T16:31:49.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 -- start start 2024-09-13T16:31:49.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c48240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c47d00 0x1c69d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c6a270 0x1c1d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bb1010 con 0x1c69950 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b3ef40 con 0x1c47d00 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b91e30 con 0x1c6a270 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c48240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c48240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38024/0 (socket says 172.21.15.119:38024) 2024-09-13T16:31:49.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 -- 172.21.15.119:0/2797008957 learned_addr learned my addr 172.21.15.119:0/2797008957 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe706d9d700 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c6a270 0x1c1d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe705d9b700 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c47d00 0x1c69d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 -- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c6a270 msgr2=0x1c1d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c6a270 0x1c1d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 -- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c47d00 msgr2=0x1c69d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c47d00 0x1c69d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 -- 172.21.15.119:0/2797008957 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x19fda40 con 0x1c69950 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe705d9b700 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c47d00 0x1c69d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe70659c700 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c48240 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800f6c0 tx=0x7fe6f8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5f49028be7af8a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe6e7fff700 1 -- 172.21.15.119:0/2797008957 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f800fd90 con 0x1c69950 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe6e7fff700 1 -- 172.21.15.119:0/2797008957 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8014450 con 0x1c69950 2024-09-13T16:31:49.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.908+0000 7fe6e7fff700 1 -- 172.21.15.119:0/2797008957 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f80147c0 con 0x1c69950 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- 172.21.15.119:0/2797008957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 msgr2=0x1c48240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c48240 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 crypto rx=0x7fe6f800f6c0 tx=0x7fe6f8012000 comp rx=0 tx=0).stop 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- 172.21.15.119:0/2797008957 shutdown_connections 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c6a270 0x1c1d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c47d00 0x1c69d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 --2- 172.21.15.119:0/2797008957 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c48240 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- 172.21.15.119:0/2797008957 >> 172.21.15.119:0/2797008957 conn(0x19b25b0 msgr2=0x1c239b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- 172.21.15.119:0/2797008957 shutdown_connections 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- 172.21.15.119:0/2797008957 wait complete. 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- start start 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 Processor -- start 2024-09-13T16:31:49.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c65620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c31010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b8d150 con 0x1c69950 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b8aeb0 con 0x1c6a270 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b488e0 con 0x1c47d00 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe710071740 10 client.0 _ll_register_callbacks cb 0x1b27670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c31010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c31010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38040/0 (socket says 172.21.15.119:38040) 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 -- 172.21.15.119:0/1867372052 learned_addr learned my addr 172.21.15.119:0/1867372052 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe70659c700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c65620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe706d9d700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 -- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 msgr2=0x1c65620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c65620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 -- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 msgr2=0x1c31550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe705d9b700 1 -- 172.21.15.119:0/1867372052 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe70806c070 con 0x1c69950 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.912+0000 7fe706d9d700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c31010 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc011860 tx=0x7fe6fc0047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855e8055e8ff2d8f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1867372052 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc011d40 con 0x1c69950 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1867372052 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe6fc004450 con 0x1c69950 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x19bf2c0 con 0x1c69950 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1867372052 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6fc0036c0 con 0x1c69950 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 10 client.8232 fetch_fsmap learned FSMap version 62 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x1b51290 con 0x1c69950 2024-09-13T16:31:49.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1867372052 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x19bf2c0 con 0x1c69950 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1867372052 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe6fc024070 con 0x1c69950 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 10 client.8232 fetch_fsmap finished waiting for FSMap version 62 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 10 client.8232 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 20 client.8232 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c465a0 0x1c469b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 4 client.8232 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1a09540 con 0x1c465a0 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe70659c700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c465a0 0x1c469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 10 client.8232 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe70659c700 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c465a0 0x1c469b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d32d0c31984643e9 server_cookie=b4ee522dc0f41214 in_seq=0 out_seq=0 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1867372052 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1a09540 con 0x1c465a0 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 10 client.8232 handle_command_reply: tid=0 2024-09-13T16:31:49.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 client.8232 shutdown 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 msgr2=0x1c31010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c31010 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc011860 tx=0x7fe6fc0047e0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 shutdown_connections 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c465a0 0x1c469b0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c31010 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 --2- 172.21.15.119:0/1867372052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c65620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe6e57fa700 0 client.8232 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 >> 172.21.15.119:0/1867372052 conn(0x19b25b0 msgr2=0x1c1edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 shutdown_connections 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 1 -- 172.21.15.119:0/1867372052 wait complete. 2024-09-13T16:31:49.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.916+0000 7fe710071740 20 client.8232 trim_cache size 0 max 16384 2024-09-13T16:31:49.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 Processor -- start 2024-09-13T16:31:49.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 -- start start 2024-09-13T16:31:49.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c476a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c2ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c30440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a36b80 con 0x1c69950 2024-09-13T16:31:49.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c2d8a0 con 0x1c6a270 2024-09-13T16:31:49.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b84330 con 0x1c47d00 2024-09-13T16:31:49.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c2ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c2ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38050/0 (socket says 172.21.15.119:38050) 2024-09-13T16:31:49.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 -- 172.21.15.119:0/1364431546 learned_addr learned my addr 172.21.15.119:0/1364431546 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe70659c700 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c476a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe706d9d700 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c30440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 -- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 msgr2=0x1c476a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c476a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 -- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 msgr2=0x1c30440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c30440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.936+0000 7fe705d9b700 1 -- 172.21.15.119:0/1364431546 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1b374c0 con 0x1c69950 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c2ff00 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009150 tx=0x7fe6fc01d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669dffe883a434c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe6f4ff9700 1 -- 172.21.15.119:0/1364431546 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc004450 con 0x1c69950 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe6f4ff9700 1 -- 172.21.15.119:0/1364431546 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6fc021040 con 0x1c69950 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe6f4ff9700 1 -- 172.21.15.119:0/1364431546 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6fc003c50 con 0x1c69950 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 -- 172.21.15.119:0/1364431546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 msgr2=0x1c2ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c2ff00 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7fe6fc009150 tx=0x7fe6fc01d070 comp rx=0 tx=0).stop 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 -- 172.21.15.119:0/1364431546 shutdown_connections 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c30440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c69950 0x1c2ff00 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 --2- 172.21.15.119:0/1364431546 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c47d00 0x1c476a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 -- 172.21.15.119:0/1364431546 >> 172.21.15.119:0/1364431546 conn(0x19b25b0 msgr2=0x1c48710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 -- 172.21.15.119:0/1364431546 shutdown_connections 2024-09-13T16:31:49.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 -- 172.21.15.119:0/1364431546 wait complete. 2024-09-13T16:31:49.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 -- start start 2024-09-13T16:31:49.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.940+0000 7fe710071740 1 Processor -- start 2024-09-13T16:31:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c47d00 0x1cd3490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c69950 0x1c30af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b29620 con 0x1c47d00 2024-09-13T16:31:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bb2610 con 0x1c6a270 2024-09-13T16:31:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b886c0 con 0x1c69950 2024-09-13T16:31:49.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 10 client.0 _ll_register_callbacks cb 0x1b5a8f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c47d00 0x1cd3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c47d00 0x1cd3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:38058/0 (socket says 172.21.15.119:38058) 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 -- 172.21.15.119:0/1324436406 learned_addr learned my addr 172.21.15.119:0/1324436406 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe706d9d700 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c69950 0x1c30af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 -- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c69950 msgr2=0x1c30af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c69950 0x1c30af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 -- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 msgr2=0x1c31030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 -- 172.21.15.119:0/1324436406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1b51290 con 0x1c47d00 2024-09-13T16:31:49.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe70659c700 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c47d00 0x1cd3490 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8011010 tx=0x7fe6f8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57c5de3e1fed588d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:31:49.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe6e57fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:31:49.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1324436406 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6f8031040 con 0x1c47d00 2024-09-13T16:31:49.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1324436406 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe6f801c040 con 0x1c47d00 2024-09-13T16:31:49.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe6f8014450 con 0x1c47d00 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1324436406 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=62) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6f8007ac0 con 0x1c47d00 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 10 client.8238 fetch_fsmap learned FSMap version 62 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=62,monmap=2+}) v3 -- 0x1b2b790 con 0x1c47d00 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.944+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1324436406 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f8014450 con 0x1c47d00 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1324436406 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 62) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe6f80148e0 con 0x1c47d00 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 10 client.8238 fetch_fsmap finished waiting for FSMap version 62 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 10 client.8238 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 20 client.8238 populate_metadata read hostname 'smithi119' 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c63df0 0x1c64200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 4 client.8238 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:31:49.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x1a2e2c0 con 0x1c63df0 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c63df0 0x1c64200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe6e57fa700 10 client.8238 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe705d9b700 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c63df0 0x1c64200 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e3b301c96bac1acc server_cookie=7b1a3149fcceaf8 in_seq=0 out_seq=0 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe6e57fa700 1 -- 172.21.15.119:0/1324436406 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+256 (crc 0 0 0) 0x1a2e2c0 con 0x1c63df0 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe6e57fa700 10 client.8238 handle_command_reply: tid=0 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 client.8238 shutdown 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c47d00 msgr2=0x1cd3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c47d00 0x1cd3490 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 crypto rx=0x7fe6f8011010 tx=0x7fe6f8001080 comp rx=0 tx=0).stop 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 shutdown_connections 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1c63df0 0x1c64200 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c6a270 0x1c31030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c69950 0x1c30af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 --2- 172.21.15.119:0/1324436406 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c47d00 0x1cd3490 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe6e57fa700 0 client.8238 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 >> 172.21.15.119:0/1324436406 conn(0x19b25b0 msgr2=0x1c226a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 shutdown_connections 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 1 -- 172.21.15.119:0/1324436406 wait complete. 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe710071740 20 client.8238 trim_cache size 0 max 16384 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stdout: "695d5313-cc5c-43cb-aec0-996bc8c00804": { 2024-09-13T16:31:49.957 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/dir", 2024-09-13T16:31:49.958 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "695d5313-cc5c-43cb-aec0-996bc8c00804", 2024-09-13T16:31:49.958 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive" 2024-09-13T16:31:49.958 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:49.958 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:31:49.958 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:31:49.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6e8066730 msgr2=0x7fe6e8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6e8066730 0x7fe6e8068bf0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fe7080b7b70 tx=0x7fe6f8009040 comp rx=0 tx=0).stop 2024-09-13T16:31:49.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe70815cfb0 msgr2=0x7fe7081f61f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:31:49.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe70815cfb0 0x7fe7081f61f0 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7fe70000efc0 tx=0x7fe70000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 shutdown_connections 2024-09-13T16:31:49.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe6e8066730 0x7fe6e8068bf0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe70815cfb0 0x7fe7081f61f0 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe7080d1640 0x7fe7080b7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 --2- 172.21.15.119:0/3781970358 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe70806b710 0x7fe70815dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:31:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 >> 172.21.15.119:0/3781970358 conn(0x7fe7080d2da0 msgr2=0x7fe7080d3180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:31:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 shutdown_connections 2024-09-13T16:31:49.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:31:49.948+0000 7fe70cdc0700 1 -- 172.21.15.119:0/3781970358 wait complete. 2024-09-13T16:31:49.969 INFO:tasks.cephfs.filesystem:scrub status for tag:695d5313-cc5c-43cb-aec0-996bc8c00804 - {'path': '/dir', 'tag': '695d5313-cc5c-43cb-aec0-996bc8c00804', 'options': 'recursive'} 2024-09-13T16:31:57.085 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:31:57.080+0000 7fed948e1700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-09-13T16:31:57.630 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:31:57.632 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:32:19.970 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-09-13T16:32:20.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.092+0000 7f2217fff700 1 Processor -- start 2024-09-13T16:32:20.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.092+0000 7f2217fff700 1 -- start start 2024-09-13T16:32:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22181652b0 0x7f221806c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218071ac0 con 0x7f22180b2bb0 2024-09-13T16:32:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2218165760 con 0x7f22180ade80 2024-09-13T16:32:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2218070eb0 con 0x7f22181652b0 2024-09-13T16:32:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f22167fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f22180b2bb0 0x7f221806bda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49762/0 (socket says 172.21.15.119:49762) 2024-09-13T16:32:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f22167fc700 1 -- 172.21.15.119:0/2498252385 learned_addr learned my addr 172.21.15.119:0/2498252385 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:20.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f22177fe700 1 --2- 172.21.15.119:0/2498252385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22181652b0 0x7f221806c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2215ffb700 1 -- 172.21.15.119:0/2498252385 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684210516 0 0) 0x7f2218071ac0 con 0x7f22180b2bb0 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2215ffb700 1 -- 172.21.15.119:0/2498252385 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22040035d0 con 0x7f22180b2bb0 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2215ffb700 1 -- 172.21.15.119:0/2498252385 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2835195243 0 0) 0x7f2218165760 con 0x7f22180ade80 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2215ffb700 1 -- 172.21.15.119:0/2498252385 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2218071ac0 con 0x7f22180ade80 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f22177fe700 1 -- 172.21.15.119:0/2498252385 >> v1:172.21.15.145:6789/0 conn(0x7f22180ade80 legacy=0x7f22180b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f22177fe700 1 -- 172.21.15.119:0/2498252385 >> v1:172.21.15.119:6789/0 conn(0x7f22180b2bb0 legacy=0x7f221806bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f22177fe700 1 -- 172.21.15.119:0/2498252385 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2218071190 con 0x7f22181652b0 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f22177fe700 1 --2- 172.21.15.119:0/2498252385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22181652b0 0x7f221806c4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2210009dc0 tx=0x7f221000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b73d25c942071a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2215ffb700 1 -- 172.21.15.119:0/2498252385 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221000e070 con 0x7f22181652b0 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2215ffb700 1 -- 172.21.15.119:0/2498252385 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221000bb20 con 0x7f22181652b0 2024-09-13T16:32:20.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2215ffb700 1 -- 172.21.15.119:0/2498252385 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221000be90 con 0x7f22181652b0 2024-09-13T16:32:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- 172.21.15.119:0/2498252385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22181652b0 msgr2=0x7f221806c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 --2- 172.21.15.119:0/2498252385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22181652b0 0x7f221806c4b0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f2210009dc0 tx=0x7f221000b330 comp rx=0 tx=0).stop 2024-09-13T16:32:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- 172.21.15.119:0/2498252385 shutdown_connections 2024-09-13T16:32:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 --2- 172.21.15.119:0/2498252385 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22181652b0 0x7f221806c4b0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- 172.21.15.119:0/2498252385 >> 172.21.15.119:0/2498252385 conn(0x7f22180b0980 msgr2=0x7f22180b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- 172.21.15.119:0/2498252385 shutdown_connections 2024-09-13T16:32:20.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.096+0000 7f2217fff700 1 -- 172.21.15.119:0/2498252385 wait complete. 2024-09-13T16:32:20.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 Processor -- start 2024-09-13T16:32:20.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 -- start start 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22180ade80 0x7f221815c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22180b2bb0 0x7f22180ab250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22181652b0 0x7f22180ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221815cbb0 con 0x7f22181652b0 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22180cc880 con 0x7f22180b2bb0 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2217fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22180ccb60 con 0x7f22180ade80 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f22177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22181652b0 0x7f22180ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f22177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22181652b0 0x7f22180ace10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48122/0 (socket says 172.21.15.119:48122) 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f22177fe700 1 -- 172.21.15.119:0/4253685478 learned_addr learned my addr 172.21.15.119:0/4253685478 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f22167fc700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22180b2bb0 0x7f22180ab250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.100+0000 7f2216ffd700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22180ade80 0x7f221815c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f22177fe700 1 -- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22180ade80 msgr2=0x7f221815c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f22177fe700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22180ade80 0x7f221815c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f22177fe700 1 -- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22180b2bb0 msgr2=0x7f22180ab250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f22177fe700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22180b2bb0 0x7f22180ab250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f22177fe700 1 -- 172.21.15.119:0/4253685478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2210009a70 con 0x7f22181652b0 2024-09-13T16:32:20.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f22177fe700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22181652b0 0x7f22180ace10 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2210009fd0 tx=0x7f2210004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f220f7fe700 1 -- 172.21.15.119:0/4253685478 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22100098f0 con 0x7f22181652b0 2024-09-13T16:32:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f220f7fe700 1 -- 172.21.15.119:0/4253685478 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221001f070 con 0x7f22181652b0 2024-09-13T16:32:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22180cce40 con 0x7f22181652b0 2024-09-13T16:32:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f220f7fe700 1 -- 172.21.15.119:0/4253685478 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2210003c80 con 0x7f22181652b0 2024-09-13T16:32:20.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22180cd400 con 0x7f22181652b0 2024-09-13T16:32:20.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.104+0000 7f220f7fe700 1 -- 172.21.15.119:0/4253685478 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2210004030 con 0x7f22181652b0 2024-09-13T16:32:20.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 Processor -- start 2024-09-13T16:32:20.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- start start 2024-09-13T16:32:20.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f220f7fe700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f21f8066760 0x7f21f8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f220f7fe700 1 -- 172.21.15.119:0/4253685478 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f22100859e0 con 0x7f22181652b0 2024-09-13T16:32:20.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f21f8066760 0x7f21f8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x158fd90 0x1546810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1545310 0x156e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x1590550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x14d7010 con 0x1545310 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1464f40 con 0x1590170 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x14b7e30 con 0x158fd90 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1545310 0x156e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1545310 0x156e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48134/0 (socket says 172.21.15.119:48134) 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 -- 172.21.15.119:0/1778544649 learned_addr learned my addr 172.21.15.119:0/1778544649 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x158fd90 0x1546810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22177fe700 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x1590550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f21f8066760 0x7f21f8068c20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f22180cdd80 tx=0x7f2200007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 -- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x158fd90 msgr2=0x1546810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x158fd90 0x1546810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 -- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 msgr2=0x1590550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x1590550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 -- 172.21.15.119:0/1778544649 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1323a40 con 0x1545310 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22177fe700 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x1590550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22167fc700 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1545310 0x156e520 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f22000014d0 tx=0x7f220001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a9970c995161db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f220cff9700 1 -- 172.21.15.119:0/1778544649 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200001b20 con 0x1545310 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f220cff9700 1 -- 172.21.15.119:0/1778544649 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2200001c80 con 0x1545310 2024-09-13T16:32:20.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f220cff9700 1 -- 172.21.15.119:0/1778544649 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200005b00 con 0x1545310 2024-09-13T16:32:20.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- 172.21.15.119:0/1778544649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1545310 msgr2=0x156e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1545310 0x156e520 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 crypto rx=0x7f22000014d0 tx=0x7f220001a000 comp rx=0 tx=0).stop 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- 172.21.15.119:0/1778544649 shutdown_connections 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x1590550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1545310 0x156e520 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 --2- 172.21.15.119:0/1778544649 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x158fd90 0x1546810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- 172.21.15.119:0/1778544649 >> 172.21.15.119:0/1778544649 conn(0x12d85b0 msgr2=0x1549b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- 172.21.15.119:0/1778544649 shutdown_connections 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.108+0000 7f22207e9740 1 -- 172.21.15.119:0/1778544649 wait complete. 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 -- start start 2024-09-13T16:32:20.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 Processor -- start 2024-09-13T16:32:20.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x156ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x158e370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x156cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x14b3150 con 0x158fd90 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x14b0eb0 con 0x1590170 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x146e8e0 con 0x1545310 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x158e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 10 client.0 _ll_register_callbacks cb 0x144d670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x158e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48136/0 (socket says 172.21.15.119:48136) 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 -- 172.21.15.119:0/1552691014 learned_addr learned my addr 172.21.15.119:0/1552691014 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x156ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22177fe700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x156cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 -- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 msgr2=0x156ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x156ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 -- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 msgr2=0x156cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x156cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 -- 172.21.15.119:0/1552691014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22040035d0 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22177fe700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x156cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22167fc700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x158e370 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x1481fa0 tx=0x7f2200000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6913119da188640d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f21f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1552691014 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200001b20 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1552691014 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f2200006250 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.112+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x12e52c0 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1552691014 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2200006410 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 10 client.8250 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x1477290 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1552691014 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x12e52c0 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1552691014 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2200006570 con 0x158fd90 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 10 client.8250 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 10 client.8250 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:20.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 20 client.8250 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:20.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x1592fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 4 client.8250 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:32:20.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x132f540 con 0x15916d0 2024-09-13T16:32:20.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x1592fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f21f5ffb700 10 client.8250 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x1592fa0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1296c43685fceff9 server_cookie=98f89d72f071c4b2 in_seq=0 out_seq=0 2024-09-13T16:32:20.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1552691014 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x132f540 con 0x15916d0 2024-09-13T16:32:20.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f21f5ffb700 10 client.8250 handle_command_reply: tid=0 2024-09-13T16:32:20.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 client.8250 shutdown 2024-09-13T16:32:20.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 msgr2=0x158e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x158e370 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x1481fa0 tx=0x7f2200000ed0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 shutdown_connections 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x1592fa0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1590170 0x156cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f21f5ffb700 0 client.8250 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x158e370 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 --2- 172.21.15.119:0/1552691014 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x156ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 >> 172.21.15.119:0/1552691014 conn(0x12d85b0 msgr2=0x156e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 shutdown_connections 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 1 -- 172.21.15.119:0/1552691014 wait complete. 2024-09-13T16:32:20.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.116+0000 7f22207e9740 20 client.8250 trim_cache size 0 max 16384 2024-09-13T16:32:20.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 Processor -- start 2024-09-13T16:32:20.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 -- start start 2024-09-13T16:32:20.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x15e97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x158fd90 0x15e9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1590170 0x15ea230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x135cb80 con 0x1590170 2024-09-13T16:32:20.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x14ae6c0 con 0x158fd90 2024-09-13T16:32:20.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x14b5f30 con 0x1545310 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1590170 0x15ea230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1590170 0x15ea230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48150/0 (socket says 172.21.15.119:48150) 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 -- 172.21.15.119:0/744639159 learned_addr learned my addr 172.21.15.119:0/744639159 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f2216ffd700 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x15e97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22167fc700 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x158fd90 0x15e9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 -- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 msgr2=0x15e97b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x15e97b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 -- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x158fd90 msgr2=0x15e9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x158fd90 0x15e9cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 -- 172.21.15.119:0/744639159 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x145d4c0 con 0x1590170 2024-09-13T16:32:20.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22177fe700 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1590170 0x15ea230 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f2210056820 tx=0x7f2210056f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6fdb4c3b8d9558d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f220d7fa700 1 -- 172.21.15.119:0/744639159 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221006e070 con 0x1590170 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f220d7fa700 1 -- 172.21.15.119:0/744639159 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221005a6c0 con 0x1590170 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f220d7fa700 1 -- 172.21.15.119:0/744639159 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221005ec80 con 0x1590170 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 -- 172.21.15.119:0/744639159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1590170 msgr2=0x15ea230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.140+0000 7f22207e9740 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1590170 0x15ea230 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f2210056820 tx=0x7f2210056f90 comp rx=0 tx=0).stop 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- 172.21.15.119:0/744639159 shutdown_connections 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1590170 0x15ea230 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x158fd90 0x15e9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 --2- 172.21.15.119:0/744639159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1545310 0x15e97b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- 172.21.15.119:0/744639159 >> 172.21.15.119:0/744639159 conn(0x12d85b0 msgr2=0x159f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- 172.21.15.119:0/744639159 shutdown_connections 2024-09-13T16:32:20.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- 172.21.15.119:0/744639159 wait complete. 2024-09-13T16:32:20.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- start start 2024-09-13T16:32:20.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 Processor -- start 2024-09-13T16:32:20.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1545310 0x15e8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x15ace60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1590170 0x15ad3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x15a75a0 con 0x158fd90 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x15a6920 con 0x1545310 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x15a6680 con 0x1590170 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22207e9740 10 client.0 _ll_register_callbacks cb 0x14af610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22167fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x15ace60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22167fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x15ace60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48156/0 (socket says 172.21.15.119:48156) 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22167fc700 1 -- 172.21.15.119:0/1835638273 learned_addr learned my addr 172.21.15.119:0/1835638273 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1545310 0x15e8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.144+0000 7f22177fe700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1590170 0x15ad3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22167fc700 1 -- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1590170 msgr2=0x15ad3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22167fc700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1590170 0x15ad3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22167fc700 1 -- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1545310 msgr2=0x15e8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22167fc700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1545310 0x15e8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22167fc700 1 -- 172.21.15.119:0/1835638273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22180cd400 con 0x158fd90 2024-09-13T16:32:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1545310 0x15e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22167fc700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x15ace60 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x1481fa0 tx=0x7f2200017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c1b1528dbaa741c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1835638273 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2200001b20 con 0x158fd90 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1835638273 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f2200006650 con 0x158fd90 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2200001c80 con 0x158fd90 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1835638273 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22000078f0 con 0x158fd90 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 10 client.8256 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x1451790 con 0x158fd90 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1835638273 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2200001c80 con 0x158fd90 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1835638273 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f2200029d10 con 0x158fd90 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 10 client.8256 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 10 client.8256 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:20.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 20 client.8256 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:20.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x15a55d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 4 client.8256 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:32:20.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x13542c0 con 0x15916d0 2024-09-13T16:32:20.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x15a55d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 10 client.8256 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f2216ffd700 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x15a55d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1251104ec6c77847 server_cookie=e7d48f87c8e7939a in_seq=0 out_seq=0 2024-09-13T16:32:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 1 -- 172.21.15.119:0/1835638273 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x13542c0 con 0x15916d0 2024-09-13T16:32:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 10 client.8256 handle_command_reply: tid=0 2024-09-13T16:32:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 client.8256 shutdown 2024-09-13T16:32:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 msgr2=0x15ace60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x15ace60 secure :-1 s=READY pgs=1352 cs=0 l=1 rev1=1 crypto rx=0x1481fa0 tx=0x7f2200017000 comp rx=0 tx=0).stop 2024-09-13T16:32:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 shutdown_connections 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x15916d0 0x15a55d0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1590170 0x15ad3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f21f5ffb700 0 client.8256 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x158fd90 0x15ace60 unknown :-1 s=CLOSED pgs=1352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 --2- 172.21.15.119:0/1835638273 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1545310 0x15e8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 >> 172.21.15.119:0/1835638273 conn(0x12d85b0 msgr2=0x15a7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 shutdown_connections 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 1 -- 172.21.15.119:0/1835638273 wait complete. 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.148+0000 7f22207e9740 20 client.8256 trim_cache size 0 max 16384 2024-09-13T16:32:20.158 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:32:20.159 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:32:20.159 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:32:20.159 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:32:20.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f21f8066760 msgr2=0x7f21f8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f21f8066760 0x7f21f8068c20 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f22180cdd80 tx=0x7f2200007de0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22181652b0 msgr2=0x7f22180ace10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22181652b0 0x7f22180ace10 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7f2210009fd0 tx=0x7f2210004af0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 shutdown_connections 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f21f8066760 0x7f21f8068c20 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f22181652b0 0x7f22180ace10 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f22180b2bb0 0x7f22180ab250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 --2- 172.21.15.119:0/4253685478 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f22180ade80 0x7f221815c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 >> 172.21.15.119:0/4253685478 conn(0x7f22180b0980 msgr2=0x7f22180afde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 shutdown_connections 2024-09-13T16:32:20.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.152+0000 7f2217fff700 1 -- 172.21.15.119:0/4253685478 wait complete. 2024-09-13T16:32:20.174 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:32:20.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:32:20.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.276+0000 7fdf9d006700 1 Processor -- start 2024-09-13T16:32:20.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.276+0000 7fdf9d006700 1 -- start start 2024-09-13T16:32:20.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf9d006700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fdf980d2a30 0x7fdf980d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf9d006700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980b3a20 0x7fdf980cd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf9d006700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf980d3350 con 0x7fdf980cf1e0 2024-09-13T16:32:20.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf9d006700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf980cdf90 con 0x7fdf980d2a30 2024-09-13T16:32:20.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf9d006700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf980ce270 con 0x7fdf980b3a20 2024-09-13T16:32:20.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf9c805700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fdf980cf1e0 0x7fdf980cf670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49776/0 (socket says 172.21.15.119:49776) 2024-09-13T16:32:20.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf9c805700 1 -- 172.21.15.119:0/3533233861 learned_addr learned my addr 172.21.15.119:0/3533233861 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:20.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf977fe700 1 --2- 172.21.15.119:0/3533233861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980b3a20 0x7fdf980cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf97fff700 1 --2- 172.21.15.119:0/3533233861 >> v2:172.21.15.145:3300/0 conn(0x7fdf980d2a30 0x7fdf980d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf96ffd700 1 -- 172.21.15.119:0/3533233861 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234569553 0 0) 0x7fdf980d3350 con 0x7fdf980cf1e0 2024-09-13T16:32:20.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf977fe700 1 -- 172.21.15.119:0/3533233861 >> v2:172.21.15.145:3300/0 conn(0x7fdf980d2a30 msgr2=0x7fdf980d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf977fe700 1 --2- 172.21.15.119:0/3533233861 >> v2:172.21.15.145:3300/0 conn(0x7fdf980d2a30 0x7fdf980d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf977fe700 1 -- 172.21.15.119:0/3533233861 >> v1:172.21.15.119:6789/0 conn(0x7fdf980cf1e0 legacy=0x7fdf980cf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf977fe700 1 -- 172.21.15.119:0/3533233861 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf980ce550 con 0x7fdf980b3a20 2024-09-13T16:32:20.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf97fff700 1 --2- 172.21.15.119:0/3533233861 >> v2:172.21.15.145:3300/0 conn(0x7fdf980d2a30 0x7fdf980d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:20.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf977fe700 1 --2- 172.21.15.119:0/3533233861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980b3a20 0x7fdf980cd640 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009850 tx=0x7fdf88009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d90fc950d298138e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf96ffd700 1 -- 172.21.15.119:0/3533233861 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88011830 con 0x7fdf980b3a20 2024-09-13T16:32:20.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf96ffd700 1 -- 172.21.15.119:0/3533233861 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf88011990 con 0x7fdf980b3a20 2024-09-13T16:32:20.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.280+0000 7fdf96ffd700 1 -- 172.21.15.119:0/3533233861 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf88011d40 con 0x7fdf980b3a20 2024-09-13T16:32:20.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 -- 172.21.15.119:0/3533233861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980b3a20 msgr2=0x7fdf980cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 --2- 172.21.15.119:0/3533233861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980b3a20 0x7fdf980cd640 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fdf88009850 tx=0x7fdf88009b60 comp rx=0 tx=0).stop 2024-09-13T16:32:20.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 -- 172.21.15.119:0/3533233861 shutdown_connections 2024-09-13T16:32:20.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 --2- 172.21.15.119:0/3533233861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980b3a20 0x7fdf980cd640 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 --2- 172.21.15.119:0/3533233861 >> v2:172.21.15.145:3300/0 conn(0x7fdf980d2a30 0x7fdf980d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 -- 172.21.15.119:0/3533233861 >> 172.21.15.119:0/3533233861 conn(0x7fdf9808c0f0 msgr2=0x7fdf9808c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 -- 172.21.15.119:0/3533233861 shutdown_connections 2024-09-13T16:32:20.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 -- 172.21.15.119:0/3533233861 wait complete. 2024-09-13T16:32:20.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 Processor -- start 2024-09-13T16:32:20.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.284+0000 7fdf9d006700 1 -- start start 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf980b3a20 0x7fdf9814d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980cf1e0 0x7fdf9815cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf980d2a30 0x7fdf980cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf980cbc90 con 0x7fdf980d2a30 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf9814dd90 con 0x7fdf980b3a20 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf980b6b20 con 0x7fdf980cf1e0 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9c805700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf980d2a30 0x7fdf980cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9c805700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf980d2a30 0x7fdf980cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48158/0 (socket says 172.21.15.119:48158) 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9c805700 1 -- 172.21.15.119:0/1903666235 learned_addr learned my addr 172.21.15.119:0/1903666235 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf97fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf980b3a20 0x7fdf9814d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf977fe700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980cf1e0 0x7fdf9815cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf97fff700 1 -- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980cf1e0 msgr2=0x7fdf9815cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf97fff700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980cf1e0 0x7fdf9815cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf97fff700 1 -- 172.21.15.119:0/1903666235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf980d2a30 msgr2=0x7fdf980cc120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf97fff700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf980d2a30 0x7fdf980cc120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf97fff700 1 -- 172.21.15.119:0/1903666235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf8c0094b0 con 0x7fdf980b3a20 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf977fe700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980cf1e0 0x7fdf9815cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9c805700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf980d2a30 0x7fdf980cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf97fff700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf980b3a20 0x7fdf9814d850 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00e9b0 tx=0x7fdf8c00ecc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf94ff9700 1 -- 172.21.15.119:0/1903666235 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00ee50 con 0x7fdf980b3a20 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf94ff9700 1 -- 172.21.15.119:0/1903666235 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c00c850 con 0x7fdf980b3a20 2024-09-13T16:32:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf880094b0 con 0x7fdf980b3a20 2024-09-13T16:32:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf94ff9700 1 -- 172.21.15.119:0/1903666235 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00cc00 con 0x7fdf980b3a20 2024-09-13T16:32:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.288+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf980cb920 con 0x7fdf980b3a20 2024-09-13T16:32:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.292+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf980675c0 con 0x7fdf980b3a20 2024-09-13T16:32:20.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.292+0000 7fdf94ff9700 1 -- 172.21.15.119:0/1903666235 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fdf8c013470 con 0x7fdf980b3a20 2024-09-13T16:32:20.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.292+0000 7fdf94ff9700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdf84066730 0x7fdf84068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.292+0000 7fdf94ff9700 1 -- 172.21.15.119:0/1903666235 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fdf8c0842f0 con 0x7fdf980b3a20 2024-09-13T16:32:20.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.292+0000 7fdf977fe700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdf84066730 0x7fdf84068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.296+0000 7fdf977fe700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdf84066730 0x7fdf84068bf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdf880090b0 tx=0x7fdf8801c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.304+0000 7fdf94ff9700 1 -- 172.21.15.119:0/1903666235 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdf8c0514e0 con 0x7fdf980b3a20 2024-09-13T16:32:20.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.512+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf9815cdb0 con 0x7fdf980b3a20 2024-09-13T16:32:20.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf94ff9700 1 -- 172.21.15.119:0/1903666235 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4747 (secure 0 0 0) 0x7fdf8c054b00 con 0x7fdf980b3a20 2024-09-13T16:32:20.520 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:32:20.520 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":63,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:57.084733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7881":{"gid":7881,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":11,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:32:20.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdf84066730 msgr2=0x7fdf84068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdf84066730 0x7fdf84068bf0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fdf880090b0 tx=0x7fdf8801c040 comp rx=0 tx=0).stop 2024-09-13T16:32:20.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf980b3a20 msgr2=0x7fdf9814d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf980b3a20 0x7fdf9814d850 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fdf8c00e9b0 tx=0x7fdf8c00ecc0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 shutdown_connections 2024-09-13T16:32:20.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdf84066730 0x7fdf84068bf0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdf980d2a30 0x7fdf980cc120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdf980cf1e0 0x7fdf9815cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 --2- 172.21.15.119:0/1903666235 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdf980b3a20 0x7fdf9814d850 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 >> 172.21.15.119:0/1903666235 conn(0x7fdf9808c0f0 msgr2=0x7fdf9808c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 shutdown_connections 2024-09-13T16:32:20.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.516+0000 7fdf9d006700 1 -- 172.21.15.119:0/1903666235 wait complete. 2024-09-13T16:32:20.526 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 63 2024-09-13T16:32:20.534 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-09-13T16:32:20.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d21f0700 1 Processor -- start 2024-09-13T16:32:20.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d21f0700 1 -- start start 2024-09-13T16:32:20.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d21f0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc06c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d21f0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45cc0b3fd0 0x7f45cc06d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d21f0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45cc0b2fe0 con 0x7f45cc0abf40 2024-09-13T16:32:20.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d21f0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45cc06de40 con 0x7f45cc0b3fd0 2024-09-13T16:32:20.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d21f0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc06db80 con 0x7f45cc160020 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d11ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc06c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d09ed700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45cc0b3fd0 0x7f45cc06d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d19ef700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f45cc0abf40 0x7f45cc06a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:44704/0 (socket says 172.21.15.119:44704) 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d19ef700 1 -- 172.21.15.119:0/3420902082 learned_addr learned my addr 172.21.15.119:0/3420902082 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d09ed700 1 -- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 msgr2=0x7f45cc06c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc06c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d09ed700 1 -- 172.21.15.119:0/3420902082 >> v1:172.21.15.145:6789/0 conn(0x7f45cc0abf40 legacy=0x7f45cc06a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.648+0000 7f45d09ed700 1 -- 172.21.15.119:0/3420902082 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45cc0b3210 con 0x7f45cc0b3fd0 2024-09-13T16:32:20.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc06c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:20.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 0x7f45cc06d640 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009850 tx=0x7f45c0009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14c1c19ca61563ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45cbfff700 1 -- 172.21.15.119:0/3420902082 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0011830 con 0x7f45cc0b3fd0 2024-09-13T16:32:20.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45cbfff700 1 -- 172.21.15.119:0/3420902082 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c0011990 con 0x7f45cc0b3fd0 2024-09-13T16:32:20.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45cbfff700 1 -- 172.21.15.119:0/3420902082 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0011d40 con 0x7f45cc0b3fd0 2024-09-13T16:32:20.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 -- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 msgr2=0x7f45cc06d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 0x7f45cc06d640 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009850 tx=0x7f45c0009b60 comp rx=0 tx=0).stop 2024-09-13T16:32:20.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 -- 172.21.15.119:0/3420902082 shutdown_connections 2024-09-13T16:32:20.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 0x7f45cc06d640 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3420902082 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc06c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 -- 172.21.15.119:0/3420902082 >> 172.21.15.119:0/3420902082 conn(0x7f45cc0b0cd0 msgr2=0x7f45cc0b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 -- 172.21.15.119:0/3420902082 shutdown_connections 2024-09-13T16:32:20.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 -- 172.21.15.119:0/3420902082 wait complete. 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 Processor -- start 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.652+0000 7f45d21f0700 1 -- start start 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45cc0abf40 0x7f45cc071aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 0x7f45cc0afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc15d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc099d00 con 0x7f45cc0abf40 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc099fa0 con 0x7f45cc0b3fd0 2024-09-13T16:32:20.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45cc09a280 con 0x7f45cc160020 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45cc0abf40 0x7f45cc071aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d09ed700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 0x7f45cc0afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45cc0abf40 0x7f45cc071aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48170/0 (socket says 172.21.15.119:48170) 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 -- 172.21.15.119:0/3913633046 learned_addr learned my addr 172.21.15.119:0/3913633046 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d19ef700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc15d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 -- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 msgr2=0x7f45cc15d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc15d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 -- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 msgr2=0x7f45cc0afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 0x7f45cc0afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 -- 172.21.15.119:0/3913633046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45bc009a70 con 0x7f45cc0abf40 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45cc0abf40 0x7f45cc071aa0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00efc0 tx=0x7f45bc00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45c9ffb700 1 -- 172.21.15.119:0/3913633046 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc00cd10 con 0x7f45cc0abf40 2024-09-13T16:32:20.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45c9ffb700 1 -- 172.21.15.119:0/3913633046 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45bc00f040 con 0x7f45cc0abf40 2024-09-13T16:32:20.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45c00094b0 con 0x7f45cc0abf40 2024-09-13T16:32:20.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45c9ffb700 1 -- 172.21.15.119:0/3913633046 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45bc0146a0 con 0x7f45cc0abf40 2024-09-13T16:32:20.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.656+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc1f6dd0 con 0x7f45cc0abf40 2024-09-13T16:32:20.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.660+0000 7f45c9ffb700 1 -- 172.21.15.119:0/3913633046 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f45bc014800 con 0x7f45cc0abf40 2024-09-13T16:32:20.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.660+0000 7f45c9ffb700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45b8066730 0x7f45b8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.660+0000 7f45c9ffb700 1 -- 172.21.15.119:0/3913633046 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f45bc084f60 con 0x7f45cc0abf40 2024-09-13T16:32:20.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.660+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45b8066730 0x7f45b8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.660+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45b8066730 0x7f45b8068bf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f45c00090b0 tx=0x7f45c0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.660+0000 7f45d6703740 1 Processor -- start 2024-09-13T16:32:20.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d6703740 1 -- start start 2024-09-13T16:32:20.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x211d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d6703740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x2119cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x211a210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d6703740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2045810 con 0x211d950 2024-09-13T16:32:20.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2042fa0 con 0x2066520 2024-09-13T16:32:20.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x203e890 con 0x211dd30 2024-09-13T16:32:20.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x2119cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x2119cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48180/0 (socket says 172.21.15.119:48180) 2024-09-13T16:32:20.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 -- 172.21.15.119:0/3201700615 learned_addr learned my addr 172.21.15.119:0/3201700615 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x211d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d19ef700 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x211a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 -- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 msgr2=0x211a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x211a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 -- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 msgr2=0x211d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x211d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 -- 172.21.15.119:0/3201700615 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1eb0ba0 con 0x211d950 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x211d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x2119cd0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009eb0 tx=0x7f45c0010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95621ca7630863b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45b2ffd700 1 -- 172.21.15.119:0/3201700615 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c002c070 con 0x211d950 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45b2ffd700 1 -- 172.21.15.119:0/3201700615 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c00036c0 con 0x211d950 2024-09-13T16:32:20.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.664+0000 7f45b2ffd700 1 -- 172.21.15.119:0/3201700615 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0020070 con 0x211d950 2024-09-13T16:32:20.673 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- 172.21.15.119:0/3201700615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 msgr2=0x2119cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x2119cd0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7f45c0009eb0 tx=0x7f45c0010000 comp rx=0 tx=0).stop 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- 172.21.15.119:0/3201700615 shutdown_connections 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x211a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x2119cd0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 --2- 172.21.15.119:0/3201700615 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x211d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- 172.21.15.119:0/3201700615 >> 172.21.15.119:0/3201700615 conn(0x1e65710 msgr2=0x20ff6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- 172.21.15.119:0/3201700615 shutdown_connections 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- 172.21.15.119:0/3201700615 wait complete. 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- start start 2024-09-13T16:32:20.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 Processor -- start 2024-09-13T16:32:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d7e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211d950 0x20fee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211dd30 0x2100c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2001d80 con 0x211dd30 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2001a20 con 0x211d950 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f31770 con 0x2066520 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d6703740 10 client.0 _ll_register_callbacks cb 0x2037f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d19ef700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211dd30 0x2100c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d11ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.668+0000 7f45d19ef700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211dd30 0x2100c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48186/0 (socket says 172.21.15.119:48186) 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d19ef700 1 -- 172.21.15.119:0/1466483614 learned_addr learned my addr 172.21.15.119:0/1466483614 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d09ed700 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211d950 0x20fee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d19ef700 1 -- 172.21.15.119:0/1466483614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 msgr2=0x20d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d19ef700 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d7e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d19ef700 1 -- 172.21.15.119:0/1466483614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211d950 msgr2=0x20fee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d19ef700 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211d950 0x20fee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d19ef700 1 -- 172.21.15.119:0/1466483614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45cc0b3210 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d19ef700 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211dd30 0x2100c40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f45c4007700 tx=0x7f45c40015c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6e4945fd603d323 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 1 -- 172.21.15.119:0/1466483614 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c4001d30 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 1 -- 172.21.15.119:0/1466483614 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f45c4015470 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1e72420 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 1 -- 172.21.15.119:0/1466483614 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45c4015630 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 10 client.8271 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x2004910 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 1 -- 172.21.15.119:0/1466483614 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1e72420 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 1 -- 172.21.15.119:0/1466483614 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f45c4015790 con 0x211dd30 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 10 client.8271 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 10 client.8271 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:32:20.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 20 client.8271 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20ff340 0x2127490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 4 client.8271 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:32:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1ebc6a0 con 0x20ff340 2024-09-13T16:32:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d11ee700 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20ff340 0x2127490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 10 client.8271 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d11ee700 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20ff340 0x2127490 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d6bedf3bbf80af5 server_cookie=40978121b2e0c109 in_seq=0 out_seq=0 2024-09-13T16:32:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 1 -- 172.21.15.119:0/1466483614 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1ebc6a0 con 0x20ff340 2024-09-13T16:32:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45abfff700 10 client.8271 handle_command_reply: tid=0 2024-09-13T16:32:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 1 client.8271 shutdown 2024-09-13T16:32:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211dd30 msgr2=0x2100c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.672+0000 7f45d6703740 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211dd30 0x2100c40 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7f45c4007700 tx=0x7f45c40015c0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 shutdown_connections 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20ff340 0x2127490 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211dd30 0x2100c40 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211d950 0x20fee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 --2- 172.21.15.119:0/1466483614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d7e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45abfff700 0 client.8271 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 >> 172.21.15.119:0/1466483614 conn(0x1e65710 msgr2=0x20d1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 shutdown_connections 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 1 -- 172.21.15.119:0/1466483614 wait complete. 2024-09-13T16:32:20.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.676+0000 7f45d6703740 20 client.8271 trim_cache size 0 max 16384 2024-09-13T16:32:20.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 Processor -- start 2024-09-13T16:32:20.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 -- start start 2024-09-13T16:32:20.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x20d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211dd30 0x20d2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x20659a0 con 0x211d950 2024-09-13T16:32:20.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x203c0a0 con 0x211dd30 2024-09-13T16:32:20.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x20413f0 con 0x2066520 2024-09-13T16:32:20.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x20d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x20d24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48192/0 (socket says 172.21.15.119:48192) 2024-09-13T16:32:20.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 -- 172.21.15.119:0/2826361012 learned_addr learned my addr 172.21.15.119:0/2826361012 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d11ee700 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d19ef700 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211dd30 0x20d2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 -- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 msgr2=0x20d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 -- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211dd30 msgr2=0x20d2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211dd30 0x20d2a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.692+0000 7f45d09ed700 1 -- 172.21.15.119:0/2826361012 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1feab40 con 0x211d950 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x20d24e0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f45c0022040 tx=0x7f45c0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e427b6fc4646c41 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45b37fe700 1 -- 172.21.15.119:0/2826361012 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0040070 con 0x211d950 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45b37fe700 1 -- 172.21.15.119:0/2826361012 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c0003830 con 0x211d950 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45b37fe700 1 -- 172.21.15.119:0/2826361012 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c002c040 con 0x211d950 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- 172.21.15.119:0/2826361012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 msgr2=0x20d24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x20d24e0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 crypto rx=0x7f45c0022040 tx=0x7f45c0020000 comp rx=0 tx=0).stop 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- 172.21.15.119:0/2826361012 shutdown_connections 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x211dd30 0x20d2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x20d24e0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 --2- 172.21.15.119:0/2826361012 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2066520 0x20d1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- 172.21.15.119:0/2826361012 >> 172.21.15.119:0/2826361012 conn(0x1e65710 msgr2=0x21183f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- 172.21.15.119:0/2826361012 shutdown_connections 2024-09-13T16:32:20.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- 172.21.15.119:0/2826361012 wait complete. 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- start start 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 Processor -- start 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x20d61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x21865a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x2187890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ffc0e0 con 0x211d950 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2046aa0 con 0x2066520 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1fff660 con 0x211dd30 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d6703740 10 client.0 _ll_register_callbacks cb 0x203d460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x21865a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x21865a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48206/0 (socket says 172.21.15.119:48206) 2024-09-13T16:32:20.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 -- 172.21.15.119:0/3437111005 learned_addr learned my addr 172.21.15.119:0/3437111005 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d19ef700 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x2187890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x20d61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 -- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 msgr2=0x2187890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x2187890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 -- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 msgr2=0x20d61a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x20d61a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.696+0000 7f45d09ed700 1 -- 172.21.15.119:0/3437111005 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1eb0ba0 con 0x211d950 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d09ed700 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x21865a0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f45c003e040 tx=0x7f45c00047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6cd1da72d8277a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 1 -- 172.21.15.119:0/3437111005 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c0040030 con 0x211d950 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45c00036c0 con 0x211d950 2024-09-13T16:32:20.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 1 -- 172.21.15.119:0/3437111005 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f45c0041070 con 0x211d950 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 1 -- 172.21.15.119:0/3437111005 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45c002c040 con 0x211d950 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 10 client.8277 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x1fdec20 con 0x211d950 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 1 -- 172.21.15.119:0/3437111005 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c00036c0 con 0x211d950 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 1 -- 172.21.15.119:0/3437111005 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f45c0031470 con 0x211d950 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 10 client.8277 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 10 client.8277 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 20 client.8277 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2187dd0 0x21881b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 4 client.8277 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x1ee1420 con 0x2187dd0 2024-09-13T16:32:20.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2187dd0 0x21881b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 10 client.8277 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d11ee700 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2187dd0 0x21881b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa57290cf73757ae server_cookie=5c5f21c361c8325c in_seq=0 out_seq=0 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 1 -- 172.21.15.119:0/3437111005 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+418 (crc 0 0 0) 0x1ee1420 con 0x2187dd0 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 10 client.8277 handle_command_reply: tid=0 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 client.8277 shutdown 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 msgr2=0x21865a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x21865a0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f45c003e040 tx=0x7f45c00047e0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 shutdown_connections 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x2187dd0 0x21881b0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x211dd30 0x2187890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45abfff700 0 client.8277 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x211d950 0x21865a0 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 --2- 172.21.15.119:0/3437111005 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2066520 0x20d61a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 >> 172.21.15.119:0/3437111005 conn(0x1e65710 msgr2=0x20fb900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 shutdown_connections 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 1 -- 172.21.15.119:0/3437111005 wait complete. 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.700+0000 7f45d6703740 20 client.8277 trim_cache size 0 max 16384 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:32:20.710 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: "damage_type": "backtrace", 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 278457655, 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627777, 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/testdir/hardlink" 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:32:20.711 INFO:teuthology.orchestra.run.smithi119.stdout: "damage_type": "backtrace", 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 347399855, 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627776, 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/dir" 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "damage_type": "dir_frag", 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 3434036730, 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627776, 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "frag": "*", 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/dir" 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:32:20.712 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:32:20.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45b8066730 msgr2=0x7f45b8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45b8066730 0x7f45b8068bf0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f45c00090b0 tx=0x7f45c0008030 comp rx=0 tx=0).stop 2024-09-13T16:32:20.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45cc0abf40 msgr2=0x7f45cc071aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45cc0abf40 0x7f45cc071aa0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f45bc00efc0 tx=0x7f45bc00c3a0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 shutdown_connections 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45b8066730 0x7f45b8068bf0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45cc160020 0x7f45cc15d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45cc0b3fd0 0x7f45cc0afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 --2- 172.21.15.119:0/3913633046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45cc0abf40 0x7f45cc071aa0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 >> 172.21.15.119:0/3913633046 conn(0x7f45cc0b0cd0 msgr2=0x7f45cc0b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 shutdown_connections 2024-09-13T16:32:20.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.704+0000 7f45d21f0700 1 -- 172.21.15.119:0/3913633046 wait complete. 2024-09-13T16:32:20.724 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:32:20.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82ed968700 1 Processor -- start 2024-09-13T16:32:20.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82ed968700 1 -- start start 2024-09-13T16:32:20.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82ed968700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f82e815ab00 0x7f82e80cee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82ed968700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e806c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82ed968700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82e80cd5e0 con 0x7f82e80b2470 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82ed968700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82e80cd8c0 con 0x7f82e815ab00 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82ed968700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80cdba0 con 0x7f82e80cd100 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e6ffd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f82e815ab00 0x7f82e80cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e67fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e806c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e77fe700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f82e80b2470 0x7f82e815e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:44712/0 (socket says 172.21.15.119:44712) 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e77fe700 1 -- 172.21.15.119:0/3312227425 learned_addr learned my addr 172.21.15.119:0/3312227425 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e67fc700 1 -- 172.21.15.119:0/3312227425 >> v2:172.21.15.145:3300/0 conn(0x7f82e815ab00 msgr2=0x7f82e80cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e67fc700 1 --2- 172.21.15.119:0/3312227425 >> v2:172.21.15.145:3300/0 conn(0x7f82e815ab00 0x7f82e80cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e67fc700 1 -- 172.21.15.119:0/3312227425 >> v1:172.21.15.145:6789/0 conn(0x7f82e80b2470 legacy=0x7f82e815e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e67fc700 1 -- 172.21.15.119:0/3312227425 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82e80cde80 con 0x7f82e80cd100 2024-09-13T16:32:20.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e5ffb700 1 -- 172.21.15.119:0/3312227425 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909473509 0 0) 0x7f82e80cd5e0 con 0x7f82e80b2470 2024-09-13T16:32:20.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.840+0000 7f82e67fc700 1 --2- 172.21.15.119:0/3312227425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e806c990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009d90 tx=0x7f82d800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10a75fd8f66e1276 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82e5ffb700 1 -- 172.21.15.119:0/3312227425 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d800e070 con 0x7f82e80cd100 2024-09-13T16:32:20.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82e5ffb700 1 -- 172.21.15.119:0/3312227425 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82d800bb20 con 0x7f82e80cd100 2024-09-13T16:32:20.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82e5ffb700 1 -- 172.21.15.119:0/3312227425 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82d8012430 con 0x7f82e80cd100 2024-09-13T16:32:20.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- 172.21.15.119:0/3312227425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 msgr2=0x7f82e806c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 --2- 172.21.15.119:0/3312227425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e806c990 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f82d8009d90 tx=0x7f82d800b330 comp rx=0 tx=0).stop 2024-09-13T16:32:20.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- 172.21.15.119:0/3312227425 shutdown_connections 2024-09-13T16:32:20.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 --2- 172.21.15.119:0/3312227425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e806c990 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 --2- 172.21.15.119:0/3312227425 >> v2:172.21.15.145:3300/0 conn(0x7f82e815ab00 0x7f82e80cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- 172.21.15.119:0/3312227425 >> 172.21.15.119:0/3312227425 conn(0x7f82e806a350 msgr2=0x7f82e806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:20.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- 172.21.15.119:0/3312227425 shutdown_connections 2024-09-13T16:32:20.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- 172.21.15.119:0/3312227425 wait complete. 2024-09-13T16:32:20.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 Processor -- start 2024-09-13T16:32:20.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- start start 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82e80b2470 0x7f82e806ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e814e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e815ab00 0x7f82e8159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e8070490 con 0x7f82e815ab00 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82e814fe10 con 0x7f82e80b2470 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.844+0000 7f82ed968700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82e80ace30 con 0x7f82e80cd100 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e815ab00 0x7f82e8159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e815ab00 0x7f82e8159140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48220/0 (socket says 172.21.15.119:48220) 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 -- 172.21.15.119:0/3447801483 learned_addr learned my addr 172.21.15.119:0/3447801483 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:20.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e67fc700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e814e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e6ffd700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82e80b2470 0x7f82e806ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 -- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 msgr2=0x7f82e814e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e814e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 -- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82e80b2470 msgr2=0x7f82e806ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82e80b2470 0x7f82e806ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 -- 172.21.15.119:0/3447801483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82d8009a70 con 0x7f82e815ab00 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82e77fe700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e815ab00 0x7f82e8159140 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c550 tx=0x7f82e000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82dffff700 1 -- 172.21.15.119:0/3447801483 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0004ca0 con 0x7f82e815ab00 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82dffff700 1 -- 172.21.15.119:0/3447801483 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82e0004e40 con 0x7f82e815ab00 2024-09-13T16:32:20.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82e80ad110 con 0x7f82e815ab00 2024-09-13T16:32:20.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82dffff700 1 -- 172.21.15.119:0/3447801483 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0007310 con 0x7f82e815ab00 2024-09-13T16:32:20.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82e80ad6d0 con 0x7f82e815ab00 2024-09-13T16:32:20.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82e8067760 con 0x7f82e815ab00 2024-09-13T16:32:20.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.848+0000 7f82dffff700 1 -- 172.21.15.119:0/3447801483 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f82e0007470 con 0x7f82e815ab00 2024-09-13T16:32:20.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.852+0000 7f82dffff700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f82c8066730 0x7f82c8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:20.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.852+0000 7f82dffff700 1 -- 172.21.15.119:0/3447801483 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f82e0084600 con 0x7f82e815ab00 2024-09-13T16:32:20.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.852+0000 7f82e6ffd700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f82c8066730 0x7f82c8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:20.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.852+0000 7f82e6ffd700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f82c8066730 0x7f82c8068bf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f82e81f5b60 tx=0x7f82d0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:20.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:20.856+0000 7f82dffff700 1 -- 172.21.15.119:0/3447801483 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f82e0051750 con 0x7f82e815ab00 2024-09-13T16:32:21.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.064+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f82e814ff80 con 0x7f82e815ab00 2024-09-13T16:32:21.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.064+0000 7f82dffff700 1 -- 172.21.15.119:0/3447801483 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+164 (secure 0 0 0) 0x7f82e0007720 con 0x7f82e815ab00 2024-09-13T16:32:21.069 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:32:21.069 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_ERR","checks":{"MDS_DAMAGE":{"severity":"HEALTH_ERR","summary":{"message":"1 MDSs report damaged metadata","count":1},"muted":false}},"mutes":[]} 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f82c8066730 msgr2=0x7f82c8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f82c8066730 0x7f82c8068bf0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f82e81f5b60 tx=0x7f82d0009240 comp rx=0 tx=0).stop 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e815ab00 msgr2=0x7f82e8159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e815ab00 0x7f82e8159140 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c550 tx=0x7f82e000c860 comp rx=0 tx=0).stop 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 shutdown_connections 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f82c8066730 0x7f82c8068bf0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f82e815ab00 0x7f82e8159140 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f82e80cd100 0x7f82e814e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 --2- 172.21.15.119:0/3447801483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f82e80b2470 0x7f82e806ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 >> 172.21.15.119:0/3447801483 conn(0x7f82e806a350 msgr2=0x7f82e80cb290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 shutdown_connections 2024-09-13T16:32:21.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.068+0000 7f82ed968700 1 -- 172.21.15.119:0/3447801483 wait complete. 2024-09-13T16:32:21.084 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-13T16:32:21.085 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-09-13T16:32:21.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9857e700 1 Processor -- start 2024-09-13T16:32:21.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9857e700 1 -- start start 2024-09-13T16:32:21.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9857e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fce9015cc60 0x7fce900cf480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9857e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce900b44d0 con 0x7fce900cfe30 2024-09-13T16:32:21.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9857e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce900d02e0 con 0x7fce900cf9c0 2024-09-13T16:32:21.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9857e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce900d12d0 con 0x7fce9015cc60 2024-09-13T16:32:21.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce96b1b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fce900cfe30 0x7fce900ce970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49778/0 (socket says 172.21.15.119:49778) 2024-09-13T16:32:21.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce96b1b700 1 -- 172.21.15.119:0/1810880858 learned_addr learned my addr 172.21.15.119:0/1810880858 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:21.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9631a700 1 --2- 172.21.15.119:0/1810880858 >> v2:172.21.15.145:3300/0 conn(0x7fce9015cc60 0x7fce900cf480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce95318700 1 -- 172.21.15.119:0/1810880858 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503555421 0 0) 0x7fce900b44d0 con 0x7fce900cfe30 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce95318700 1 -- 172.21.15.119:0/1810880858 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce7c0035d0 con 0x7fce900cfe30 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce95318700 1 -- 172.21.15.119:0/1810880858 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 105419347 0 0) 0x7fce7c0035d0 con 0x7fce900cfe30 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9631a700 1 -- 172.21.15.119:0/1810880858 >> v1:172.21.15.145:6789/0 conn(0x7fce900cf9c0 legacy=0x7fce900b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9631a700 1 -- 172.21.15.119:0/1810880858 >> v1:172.21.15.119:6789/0 conn(0x7fce900cfe30 legacy=0x7fce900ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.200+0000 7fce9631a700 1 -- 172.21.15.119:0/1810880858 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce900d15b0 con 0x7fce9015cc60 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9631a700 1 --2- 172.21.15.119:0/1810880858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 0x7fce900cf480 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fce84009850 tx=0x7fce84009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8de7a9f690f2a1db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce95318700 1 -- 172.21.15.119:0/1810880858 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce880019f0 con 0x7fce9015cc60 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce95318700 1 -- 172.21.15.119:0/1810880858 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce84004030 con 0x7fce9015cc60 2024-09-13T16:32:21.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce95318700 1 -- 172.21.15.119:0/1810880858 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce840043a0 con 0x7fce9015cc60 2024-09-13T16:32:21.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 -- 172.21.15.119:0/1810880858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 msgr2=0x7fce900cf480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 --2- 172.21.15.119:0/1810880858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 0x7fce900cf480 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7fce84009850 tx=0x7fce84009b60 comp rx=0 tx=0).stop 2024-09-13T16:32:21.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 -- 172.21.15.119:0/1810880858 shutdown_connections 2024-09-13T16:32:21.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 --2- 172.21.15.119:0/1810880858 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 0x7fce900cf480 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 -- 172.21.15.119:0/1810880858 >> 172.21.15.119:0/1810880858 conn(0x7fce900d2b70 msgr2=0x7fce900d2f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 -- 172.21.15.119:0/1810880858 shutdown_connections 2024-09-13T16:32:21.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 -- 172.21.15.119:0/1810880858 wait complete. 2024-09-13T16:32:21.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 Processor -- start 2024-09-13T16:32:21.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.204+0000 7fce9857e700 1 -- start start 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce900cf9c0 0x7fce9015ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce900cfe30 0x7fce9015f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 0x7fce900b77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce9015f670 con 0x7fce900cf9c0 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce90160470 con 0x7fce9015cc60 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce9015f860 con 0x7fce900cfe30 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce900cf9c0 0x7fce9015ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce900cf9c0 0x7fce9015ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48226/0 (socket says 172.21.15.119:48226) 2024-09-13T16:32:21.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 -- 172.21.15.119:0/1930785908 learned_addr learned my addr 172.21.15.119:0/1930785908 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce95b19700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce900cfe30 0x7fce9015f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce96b1b700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 0x7fce900b77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 -- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce900cfe30 msgr2=0x7fce9015f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce900cfe30 0x7fce9015f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 -- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 msgr2=0x7fce900b77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 0x7fce900b77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 -- 172.21.15.119:0/1930785908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840094b0 con 0x7fce900cf9c0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9631a700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce900cf9c0 0x7fce9015ebf0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fce9008b420 tx=0x7fce84004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce82ffd700 1 -- 172.21.15.119:0/1930785908 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84003c70 con 0x7fce900cf9c0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce82ffd700 1 -- 172.21.15.119:0/1930785908 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8401f070 con 0x7fce900cf9c0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce9015fb40 con 0x7fce900cf9c0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce82ffd700 1 -- 172.21.15.119:0/1930785908 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84011630 con 0x7fce900cf9c0 2024-09-13T16:32:21.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.208+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce90160100 con 0x7fce900cf9c0 2024-09-13T16:32:21.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.212+0000 7fce82ffd700 1 -- 172.21.15.119:0/1930785908 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fce84021030 con 0x7fce900cf9c0 2024-09-13T16:32:21.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.212+0000 7fce82ffd700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce74066730 0x7fce74068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.212+0000 7fce82ffd700 1 -- 172.21.15.119:0/1930785908 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fce840855b0 con 0x7fce900cf9c0 2024-09-13T16:32:21.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.212+0000 7fce95b19700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce74066730 0x7fce74068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.212+0000 7fce95b19700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce74066730 0x7fce74068bf0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fce900b7590 tx=0x7fce8c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.212+0000 7fce9b82f740 1 Processor -- start 2024-09-13T16:32:21.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.212+0000 7fce9b82f740 1 -- start start 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20dd630 0x20dda10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20de330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20de870 0x20dec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2008500 con 0x20ddf50 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2005c90 con 0x20dd630 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2001580 con 0x20de870 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce95b19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20de330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20dd630 0x20dda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20dd630 0x20dda10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:55376/0 (socket says 172.21.15.119:55376) 2024-09-13T16:32:21.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce95b19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20de330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48228/0 (socket says 172.21.15.119:48228) 2024-09-13T16:32:21.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 -- 172.21.15.119:0/520956451 learned_addr learned my addr 172.21.15.119:0/520956451 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce96b1b700 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20de870 0x20dec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 -- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20de870 msgr2=0x20dec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20de870 0x20dec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 -- 172.21.15.119:0/520956451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 msgr2=0x20de330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20de330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 -- 172.21.15.119:0/520956451 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1ef3ee0 con 0x20dd630 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce95b19700 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20de330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9631a700 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20dd630 0x20dda10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fce840090b0 tx=0x7fce840528d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a089a2cfd61bf057 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce73fff700 1 -- 172.21.15.119:0/520956451 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8405a290 con 0x20dd630 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce73fff700 1 -- 172.21.15.119:0/520956451 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8405e850 con 0x20dd630 2024-09-13T16:32:21.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce73fff700 1 -- 172.21.15.119:0/520956451 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce84054800 con 0x20dd630 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20dd630 msgr2=0x20dda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20dd630 0x20dda10 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7fce840090b0 tx=0x7fce840528d0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- 172.21.15.119:0/520956451 shutdown_connections 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20de870 0x20dec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20de330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 --2- 172.21.15.119:0/520956451 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20dd630 0x20dda10 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- 172.21.15.119:0/520956451 >> 172.21.15.119:0/520956451 conn(0x1e28780 msgr2=0x20c2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- 172.21.15.119:0/520956451 shutdown_connections 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- 172.21.15.119:0/520956451 wait complete. 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.216+0000 7fce9b82f740 1 -- start start 2024-09-13T16:32:21.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 Processor -- start 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20bfcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20de870 0x20c6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1fc4920 con 0x20ddf50 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1fc45c0 con 0x20de870 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f96e10 con 0x20dd630 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 10 client.0 _ll_register_callbacks cb 0x1ffabf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:21.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48242/0 (socket says 172.21.15.119:48242) 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 -- 172.21.15.119:0/3164970221 learned_addr learned my addr 172.21.15.119:0/3164970221 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9631a700 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20bfcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce96b1b700 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20de870 0x20c6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 -- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 msgr2=0x20bfcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20bfcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 -- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20de870 msgr2=0x20c6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20de870 0x20c6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 -- 172.21.15.119:0/3164970221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce900d02e0 con 0x20ddf50 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce95b19700 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c62d0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fce8c014480 tx=0x7fce8c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf03da1acfbc41b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce717fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce717fa700 1 -- 172.21.15.119:0/3164970221 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c014c00 con 0x20ddf50 2024-09-13T16:32:21.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce717fa700 1 -- 172.21.15.119:0/3164970221 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fce8c01c040 con 0x20ddf50 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1e35490 con 0x20ddf50 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce717fa700 1 -- 172.21.15.119:0/3164970221 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce8c014d60 con 0x20ddf50 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 10 client.8292 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x7fce90160100 con 0x20ddf50 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce717fa700 1 -- 172.21.15.119:0/3164970221 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1e35490 con 0x20ddf50 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce717fa700 1 -- 172.21.15.119:0/3164970221 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fce8c02a3f0 con 0x20ddf50 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 10 client.8292 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 10 client.8292 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 20 client.8292 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20973b0 0x20977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 4 client.8292 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1e7f710 con 0x20973b0 2024-09-13T16:32:21.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.220+0000 7fce9631a700 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20973b0 0x20977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce717fa700 10 client.8292 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9631a700 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20973b0 0x20977c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65fcf2b96b54c6fb server_cookie=b14eb87789293c1b in_seq=0 out_seq=0 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce717fa700 1 -- 172.21.15.119:0/3164970221 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1e7f710 con 0x20973b0 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce717fa700 10 client.8292 handle_command_reply: tid=0 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 client.8292 shutdown 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 msgr2=0x20c62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c62d0 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7fce8c014480 tx=0x7fce8c001120 comp rx=0 tx=0).stop 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 shutdown_connections 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20973b0 0x20977c0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20de870 0x20c6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c62d0 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce717fa700 0 client.8292 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 --2- 172.21.15.119:0/3164970221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20bfcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 >> 172.21.15.119:0/3164970221 conn(0x1e28780 msgr2=0x20c7280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 shutdown_connections 2024-09-13T16:32:21.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 1 -- 172.21.15.119:0/3164970221 wait complete. 2024-09-13T16:32:21.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.224+0000 7fce9b82f740 20 client.8292 trim_cache size 0 max 16384 2024-09-13T16:32:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 Processor -- start 2024-09-13T16:32:21.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- start start 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20973b0 0x20f0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ffcd70 con 0x20ddf50 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1fbb9b0 con 0x20973b0 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2001e30 con 0x20dd630 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9631a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20973b0 0x20f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48254/0 (socket says 172.21.15.119:48254) 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 -- 172.21.15.119:0/664806296 learned_addr learned my addr 172.21.15.119:0/664806296 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce95b19700 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 -- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 msgr2=0x20f0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20f0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 -- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20973b0 msgr2=0x20f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20973b0 0x20f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 -- 172.21.15.119:0/664806296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1fc7600 con 0x20ddf50 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9631a700 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20973b0 0x20f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:21.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce96b1b700 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c60f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fce88007420 tx=0x7fce880020e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f04b11a15fbc85cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce80ff9700 1 -- 172.21.15.119:0/664806296 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce880166f0 con 0x20ddf50 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce80ff9700 1 -- 172.21.15.119:0/664806296 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8801a070 con 0x20ddf50 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce80ff9700 1 -- 172.21.15.119:0/664806296 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88016a70 con 0x20ddf50 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- 172.21.15.119:0/664806296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 msgr2=0x20c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c60f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7fce88007420 tx=0x7fce880020e0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- 172.21.15.119:0/664806296 shutdown_connections 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20ddf50 0x20c60f0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20dd630 0x20f0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 --2- 172.21.15.119:0/664806296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20973b0 0x20f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- 172.21.15.119:0/664806296 >> 172.21.15.119:0/664806296 conn(0x1e28780 msgr2=0x2095d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- 172.21.15.119:0/664806296 shutdown_connections 2024-09-13T16:32:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.248+0000 7fce9b82f740 1 -- 172.21.15.119:0/664806296 wait complete. 2024-09-13T16:32:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 -- start start 2024-09-13T16:32:21.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 Processor -- start 2024-09-13T16:32:21.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20973b0 0x20f86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20dd630 0x214a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20ddf50 0x214abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x20f9d30 con 0x20dd630 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x20f9f60 con 0x20ddf50 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x20fa200 con 0x20973b0 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 10 client.0 _ll_register_callbacks cb 0x2000150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce95b19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20dd630 0x214a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce96b1b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20ddf50 0x214abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce95b19700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20dd630 0x214a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48256/0 (socket says 172.21.15.119:48256) 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9631a700 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20973b0 0x20f86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce95b19700 1 -- 172.21.15.119:0/1472999261 learned_addr learned my addr 172.21.15.119:0/1472999261 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce96b1b700 1 -- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20973b0 msgr2=0x20f86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce96b1b700 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20973b0 0x20f86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce96b1b700 1 -- 172.21.15.119:0/1472999261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20dd630 msgr2=0x214a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce96b1b700 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20dd630 0x214a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce96b1b700 1 -- 172.21.15.119:0/1472999261 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1ef3ee0 con 0x20ddf50 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce96b1b700 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20ddf50 0x214abc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fce8801d000 tx=0x7fce88000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca354a8ab88fcb70 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce717fa700 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce717fa700 1 -- 172.21.15.119:0/1472999261 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88001800 con 0x20ddf50 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce8405e850 con 0x20ddf50 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.252+0000 7fce717fa700 1 -- 172.21.15.119:0/1472999261 <== mon.1 v2:172.21.15.145:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fce88020430 con 0x20ddf50 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce717fa700 1 -- 172.21.15.119:0/1472999261 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce880205f0 con 0x20ddf50 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 10 client.7933 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x7fce88007100 con 0x20ddf50 2024-09-13T16:32:21.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce717fa700 1 -- 172.21.15.119:0/1472999261 <== mon.1 v2:172.21.15.145:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8405e850 con 0x20ddf50 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce717fa700 1 -- 172.21.15.119:0/1472999261 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fce88012070 con 0x20ddf50 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 10 client.7933 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 10 client.7933 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 20 client.7933 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20de870 0x214b100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 4 client.7933 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) v1 -- 0x1ea65f0 con 0x20de870 2024-09-13T16:32:21.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9631a700 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20de870 0x214b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce717fa700 10 client.7933 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9631a700 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20de870 0x214b100 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd949595d581cf03 server_cookie=25ef5305eeaabbf2 in_seq=0 out_seq=0 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce717fa700 1 -- 172.21.15.119:0/1472999261 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x1ea65f0 con 0x20de870 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce717fa700 10 client.7933 handle_command_reply: tid=0 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 client.7933 shutdown 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20ddf50 msgr2=0x214abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20ddf50 0x214abc0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fce8801d000 tx=0x7fce88000f40 comp rx=0 tx=0).stop 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 shutdown_connections 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x20de870 0x214b100 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x20ddf50 0x214abc0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x20dd630 0x214a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce717fa700 0 client.7933 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 --2- 172.21.15.119:0/1472999261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x20973b0 0x20f86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 >> 172.21.15.119:0/1472999261 conn(0x1e28780 msgr2=0x20f0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 shutdown_connections 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 1 -- 172.21.15.119:0/1472999261 wait complete. 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fce9b82f740 20 client.7933 trim_cache size 0 max 16384 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "3dba5a5b-693d-413e-9227-b192e1496a85", 2024-09-13T16:32:21.265 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:32:21.266 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:32:21.266 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:32:21.256+0000 7fa101b5c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv16 ap=3 DIRTYPARENT f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0)->f(v1 m2024-09-13T16:31:37.078416+0000) n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1)->n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x5589ea977b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce74066730 msgr2=0x7fce74068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce74066730 0x7fce74068bf0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fce900b7590 tx=0x7fce8c009040 comp rx=0 tx=0).stop 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce900cf9c0 msgr2=0x7fce9015ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce900cf9c0 0x7fce9015ebf0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7fce9008b420 tx=0x7fce84004770 comp rx=0 tx=0).stop 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 shutdown_connections 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce74066730 0x7fce74068bf0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce9015cc60 0x7fce900b77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce900cfe30 0x7fce9015f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 --2- 172.21.15.119:0/1930785908 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce900cf9c0 0x7fce9015ebf0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 >> 172.21.15.119:0/1930785908 conn(0x7fce900d2b70 msgr2=0x7fce9006b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 shutdown_connections 2024-09-13T16:32:21.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.260+0000 7fce9857e700 1 -- 172.21.15.119:0/1930785908 wait complete. 2024-09-13T16:32:21.279 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-09-13T16:32:21.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.392+0000 7f9b54b4b700 1 Processor -- start 2024-09-13T16:32:21.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.392+0000 7f9b54b4b700 1 -- start start 2024-09-13T16:32:21.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9b5006c480 0x7f9b500cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5006c8c0 con 0x7f9b500ce000 2024-09-13T16:32:21.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b5006cb40 con 0x7f9b50144180 2024-09-13T16:32:21.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b500b1c50 con 0x7f9b5006c480 2024-09-13T16:32:21.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9b500ce000 0x7f9b500b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49786/0 (socket says 172.21.15.119:49786) 2024-09-13T16:32:21.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4ffff700 1 -- 172.21.15.119:0/826581572 learned_addr learned my addr 172.21.15.119:0/826581572 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:21.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4effd700 1 --2- 172.21.15.119:0/826581572 >> v2:172.21.15.145:3300/0 conn(0x7f9b5006c480 0x7f9b500cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4e7fc700 1 -- 172.21.15.119:0/826581572 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401198307 0 0) 0x7f9b5006c8c0 con 0x7f9b500ce000 2024-09-13T16:32:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4e7fc700 1 -- 172.21.15.119:0/826581572 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b3c0035d0 con 0x7f9b500ce000 2024-09-13T16:32:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4effd700 1 -- 172.21.15.119:0/826581572 >> v1:172.21.15.145:6789/0 conn(0x7f9b50144180 legacy=0x7f9b500cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4effd700 1 -- 172.21.15.119:0/826581572 >> v1:172.21.15.119:6789/0 conn(0x7f9b500ce000 legacy=0x7f9b500b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4effd700 1 -- 172.21.15.119:0/826581572 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b5006ce20 con 0x7f9b5006c480 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4effd700 1 --2- 172.21.15.119:0/826581572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 0x7f9b500cdac0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9b500b1d90 tx=0x7f9b3800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=813afcc78a697845 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4e7fc700 1 -- 172.21.15.119:0/826581572 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38016070 con 0x7f9b5006c480 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4e7fc700 1 -- 172.21.15.119:0/826581572 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b3800bb20 con 0x7f9b5006c480 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b4e7fc700 1 -- 172.21.15.119:0/826581572 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b3800e070 con 0x7f9b5006c480 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- 172.21.15.119:0/826581572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 msgr2=0x7f9b500cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/826581572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 0x7f9b500cdac0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f9b500b1d90 tx=0x7f9b3800b330 comp rx=0 tx=0).stop 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- 172.21.15.119:0/826581572 shutdown_connections 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/826581572 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 0x7f9b500cdac0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- 172.21.15.119:0/826581572 >> 172.21.15.119:0/826581572 conn(0x7f9b500b0980 msgr2=0x7f9b500b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- 172.21.15.119:0/826581572 shutdown_connections 2024-09-13T16:32:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- 172.21.15.119:0/826581572 wait complete. 2024-09-13T16:32:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 Processor -- start 2024-09-13T16:32:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.396+0000 7f9b54b4b700 1 -- start start 2024-09-13T16:32:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 0x7f9b5015a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b500ce000 0x7f9b5013ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b50144180 0x7f9b501406c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5015a6c0 con 0x7f9b50144180 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b50139bb0 con 0x7f9b5006c480 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5013f620 con 0x7f9b500ce000 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b50144180 0x7f9b501406c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b50144180 0x7f9b501406c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48260/0 (socket says 172.21.15.119:48260) 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 -- 172.21.15.119:0/544409543 learned_addr learned my addr 172.21.15.119:0/544409543 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4effd700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b500ce000 0x7f9b5013ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 0x7f9b5015a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 -- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b500ce000 msgr2=0x7f9b5013ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b500ce000 0x7f9b5013ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 -- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 msgr2=0x7f9b5015a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 0x7f9b5015a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 -- 172.21.15.119:0/544409543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b38009a70 con 0x7f9b50144180 2024-09-13T16:32:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b50144180 0x7f9b501406c0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c4d0 tx=0x7f9b4800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b47fff700 1 -- 172.21.15.119:0/544409543 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48005d90 con 0x7f9b50144180 2024-09-13T16:32:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b47fff700 1 -- 172.21.15.119:0/544409543 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b480070d0 con 0x7f9b50144180 2024-09-13T16:32:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b5013f900 con 0x7f9b50144180 2024-09-13T16:32:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b47fff700 1 -- 172.21.15.119:0/544409543 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48007460 con 0x7f9b50144180 2024-09-13T16:32:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b5013fec0 con 0x7f9b50144180 2024-09-13T16:32:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.400+0000 7f9b47fff700 1 -- 172.21.15.119:0/544409543 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9b480075c0 con 0x7f9b50144180 2024-09-13T16:32:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.404+0000 7f9b47fff700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b30066730 0x7f9b30068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.404+0000 7f9b47fff700 1 -- 172.21.15.119:0/544409543 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f9b480837b0 con 0x7f9b50144180 2024-09-13T16:32:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.404+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b30066730 0x7f9b30068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.404+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b30066730 0x7f9b30068bf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9b40000f60 tx=0x7f9b40009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.404+0000 7f9b5905e740 1 Processor -- start 2024-09-13T16:32:21.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.404+0000 7f9b5905e740 1 -- start start 2024-09-13T16:32:21.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b3d90 0x256a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2569310 0x2592520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x25b4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x24fb010 con 0x25b3d90 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2488f40 con 0x2569310 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x24dbe30 con 0x25b4170 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b3d90 0x256a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b3d90 0x256a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48262/0 (socket says 172.21.15.119:48262) 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4effd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2569310 0x2592520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/3195414656 learned_addr learned my addr 172.21.15.119:0/3195414656 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x25b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/3195414656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 msgr2=0x25b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x25b4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/3195414656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2569310 msgr2=0x2592520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2569310 0x2592520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/3195414656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2347a40 con 0x25b3d90 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b3d90 0x256a810 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f9b40007c30 tx=0x7f9b40011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f94794ae2fca17c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b457fa700 1 -- 172.21.15.119:0/3195414656 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40005100 con 0x25b3d90 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b457fa700 1 -- 172.21.15.119:0/3195414656 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40003cc0 con 0x25b3d90 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b457fa700 1 -- 172.21.15.119:0/3195414656 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40021650 con 0x25b3d90 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- 172.21.15.119:0/3195414656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b3d90 msgr2=0x256a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b3d90 0x256a810 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7f9b40007c30 tx=0x7f9b40011000 comp rx=0 tx=0).stop 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- 172.21.15.119:0/3195414656 shutdown_connections 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x25b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2569310 0x2592520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 --2- 172.21.15.119:0/3195414656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b3d90 0x256a810 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- 172.21.15.119:0/3195414656 >> 172.21.15.119:0/3195414656 conn(0x22fc5b0 msgr2=0x256db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- 172.21.15.119:0/3195414656 shutdown_connections 2024-09-13T16:32:21.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- 172.21.15.119:0/3195414656 wait complete. 2024-09-13T16:32:21.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.408+0000 7f9b5905e740 1 -- start start 2024-09-13T16:32:21.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 Processor -- start 2024-09-13T16:32:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2594dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x25b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x2590d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x24d7150 con 0x25b4170 2024-09-13T16:32:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x24d4eb0 con 0x25b3d90 2024-09-13T16:32:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x24928e0 con 0x2569310 2024-09-13T16:32:21.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 10 client.0 _ll_register_callbacks cb 0x2471670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x2590d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x2590d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48272/0 (socket says 172.21.15.119:48272) 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 -- 172.21.15.119:0/2539185998 learned_addr learned my addr 172.21.15.119:0/2539185998 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2594dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4effd700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x25b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 -- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 msgr2=0x2594dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2594dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 -- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 msgr2=0x25b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x25b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 -- 172.21.15.119:0/2539185998 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b4000f040 con 0x25b4170 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2594dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x2590d60 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800cbe0 tx=0x7f9b48054b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b787f58fa14f10 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b2e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/2539185998 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4806c020 con 0x25b4170 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/2539185998 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f9b4805c9c0 con 0x25b4170 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x23092c0 con 0x25b4170 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/2539185998 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b480581b0 con 0x25b4170 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 10 client.8310 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x249b290 con 0x25b4170 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/2539185998 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x23092c0 con 0x25b4170 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/2539185998 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9b4805a3a0 con 0x25b4170 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 10 client.8310 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 10 client.8310 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 20 client.8310 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:21.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x25b4a40 0x25b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 4 client.8310 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:32:21.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2353540 con 0x25b4a40 2024-09-13T16:32:21.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x25b4a40 0x25b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x25b4a40 0x25b4e50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6e9d610e822a3366 server_cookie=a5102966fbe0f3d in_seq=0 out_seq=0 2024-09-13T16:32:21.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.412+0000 7f9b2e7fc700 10 client.8310 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/2539185998 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x2353540 con 0x25b4a40 2024-09-13T16:32:21.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b2e7fc700 10 client.8310 handle_command_reply: tid=0 2024-09-13T16:32:21.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 client.8310 shutdown 2024-09-13T16:32:21.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 msgr2=0x2590d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x2590d60 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800cbe0 tx=0x7f9b48054b10 comp rx=0 tx=0).stop 2024-09-13T16:32:21.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 shutdown_connections 2024-09-13T16:32:21.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x25b4a40 0x25b4e50 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x2590d60 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x25b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 --2- 172.21.15.119:0/2539185998 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2594dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 >> 172.21.15.119:0/2539185998 conn(0x22fc5b0 msgr2=0x25924b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b2e7fc700 0 client.8310 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 shutdown_connections 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 1 -- 172.21.15.119:0/2539185998 wait complete. 2024-09-13T16:32:21.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.416+0000 7f9b5905e740 20 client.8310 trim_cache size 0 max 16384 2024-09-13T16:32:21.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 Processor -- start 2024-09-13T16:32:21.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 -- start start 2024-09-13T16:32:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2617f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26184b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x26140e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2380b80 con 0x25b4170 2024-09-13T16:32:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x245f630 con 0x25b3d90 2024-09-13T16:32:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2479bd0 con 0x2569310 2024-09-13T16:32:21.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x26140e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2617f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x26140e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48288/0 (socket says 172.21.15.119:48288) 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 -- 172.21.15.119:0/1332935267 learned_addr learned my addr 172.21.15.119:0/1332935267 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4effd700 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26184b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 -- 172.21.15.119:0/1332935267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 msgr2=0x2617f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2617f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 -- 172.21.15.119:0/1332935267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 msgr2=0x26184b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26184b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.436+0000 7f9b4ffff700 1 -- 172.21.15.119:0/1332935267 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x24814c0 con 0x25b4170 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x26140e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f9b48054560 tx=0x7f9b48083070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee64e25088fc6370 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b45ffb700 1 -- 172.21.15.119:0/1332935267 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b48054110 con 0x25b4170 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b45ffb700 1 -- 172.21.15.119:0/1332935267 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b480543a0 con 0x25b4170 2024-09-13T16:32:21.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b45ffb700 1 -- 172.21.15.119:0/1332935267 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4805c770 con 0x25b4170 2024-09-13T16:32:21.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 -- 172.21.15.119:0/1332935267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 msgr2=0x26140e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x26140e0 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 crypto rx=0x7f9b48054560 tx=0x7f9b48083070 comp rx=0 tx=0).stop 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 -- 172.21.15.119:0/1332935267 shutdown_connections 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25b4170 0x26140e0 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26184b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1332935267 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2569310 0x2617f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 -- 172.21.15.119:0/1332935267 >> 172.21.15.119:0/1332935267 conn(0x22fc5b0 msgr2=0x25c2ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 -- 172.21.15.119:0/1332935267 shutdown_connections 2024-09-13T16:32:21.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 -- 172.21.15.119:0/1332935267 wait complete. 2024-09-13T16:32:21.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 -- start start 2024-09-13T16:32:21.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.440+0000 7f9b5905e740 1 Processor -- start 2024-09-13T16:32:21.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2569310 0x26183a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26212c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x2621800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2613600 con 0x2569310 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x25c8d20 con 0x25b3d90 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2613810 con 0x25b4170 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 10 client.0 _ll_register_callbacks cb 0x24d3610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2569310 0x26183a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2569310 0x26183a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48298/0 (socket says 172.21.15.119:48298) 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/1331099874 learned_addr learned my addr 172.21.15.119:0/1331099874 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4ffff700 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x2621800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4effd700 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26212c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 msgr2=0x2621800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x2621800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 msgr2=0x26212c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26212c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 -- 172.21.15.119:0/1331099874 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x249b290 con 0x2569310 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4f7fe700 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2569310 0x26183a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f9b40014000 tx=0x7f9b40000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d9dd79754f33853 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b2e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/1331099874 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b40028040 con 0x2569310 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/1331099874 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f9b40003e30 con 0x2569310 2024-09-13T16:32:21.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9b40003cc0 con 0x2569310 2024-09-13T16:32:21.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/1331099874 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9b4002e590 con 0x2569310 2024-09-13T16:32:21.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 10 client.8316 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x2475790 con 0x2569310 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/1331099874 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40003cc0 con 0x2569310 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/1331099874 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9b40035060 con 0x2569310 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 10 client.8316 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 10 client.8316 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 20 client.8316 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x26188e0 0x2621d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 4 client.8316 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x23782c0 con 0x26188e0 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.444+0000 7f9b4effd700 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x26188e0 0x2621d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b2e7fc700 10 client.8316 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b4effd700 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x26188e0 0x2621d40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a27093db9036c3c3 server_cookie=ee2535a8e13ac444 in_seq=0 out_seq=0 2024-09-13T16:32:21.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b2e7fc700 1 -- 172.21.15.119:0/1331099874 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+263 (crc 0 0 0) 0x23782c0 con 0x26188e0 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b2e7fc700 10 client.8316 handle_command_reply: tid=0 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 client.8316 shutdown 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2569310 msgr2=0x26183a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2569310 0x26183a0 secure :-1 s=READY pgs=1369 cs=0 l=1 rev1=1 crypto rx=0x7f9b40014000 tx=0x7f9b40000ed0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 shutdown_connections 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x26188e0 0x2621d40 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25b4170 0x2621800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25b3d90 0x26212c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b2e7fc700 0 client.8316 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 --2- 172.21.15.119:0/1331099874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2569310 0x26183a0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 >> 172.21.15.119:0/1331099874 conn(0x22fc5b0 msgr2=0x25c32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 shutdown_connections 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 1 -- 172.21.15.119:0/1331099874 wait complete. 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b5905e740 20 client.8316 trim_cache size 0 max 16384 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: "3dba5a5b-693d-413e-9227-b192e1496a85": { 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/dir", 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "3dba5a5b-693d-413e-9227-b192e1496a85", 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,repair" 2024-09-13T16:32:21.456 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:32:21.457 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:32:21.457 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b30066730 msgr2=0x7f9b30068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b30066730 0x7f9b30068bf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9b40000f60 tx=0x7f9b40009040 comp rx=0 tx=0).stop 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b50144180 msgr2=0x7f9b501406c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b50144180 0x7f9b501406c0 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 crypto rx=0x7f9b4800c4d0 tx=0x7f9b4800c7e0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 shutdown_connections 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b30066730 0x7f9b30068bf0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b50144180 0x7f9b501406c0 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b500ce000 0x7f9b5013ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 --2- 172.21.15.119:0/544409543 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b5006c480 0x7f9b5015a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 >> 172.21.15.119:0/544409543 conn(0x7f9b500b0980 msgr2=0x7f9b500afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:21.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 shutdown_connections 2024-09-13T16:32:21.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:21.448+0000 7f9b54b4b700 1 -- 172.21.15.119:0/544409543 wait complete. 2024-09-13T16:32:21.469 INFO:tasks.cephfs.filesystem:scrub status for tag:3dba5a5b-693d-413e-9227-b192e1496a85 - {'path': '/dir', 'tag': '3dba5a5b-693d-413e-9227-b192e1496a85', 'options': 'recursive,repair'} 2024-09-13T16:32:27.778 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:32:27.781 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:32:51.471 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-09-13T16:32:51.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa762ad2700 1 Processor -- start 2024-09-13T16:32:51.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa762ad2700 1 -- start start 2024-09-13T16:32:51.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa762ad2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c06c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa762ad2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa75c0ae0b0 0x7fa75c0b1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa762ad2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa75c06cb90 con 0x7fa75c06c680 2024-09-13T16:32:51.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa762ad2700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa75c06ce70 con 0x7fa75c0ae0b0 2024-09-13T16:32:51.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa762ad2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c06d150 con 0x7fa75c158f10 2024-09-13T16:32:51.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76106f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa75c0ae0b0 0x7fa75c0b1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76086e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c06c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76106f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa75c0ae0b0 0x7fa75c0b1860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:51860/0 (socket says 172.21.15.119:51860) 2024-09-13T16:32:51.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa75bfff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fa75c06c680 0x7fa75c0ad9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:42756/0 (socket says 172.21.15.119:42756) 2024-09-13T16:32:51.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76106f700 1 -- 172.21.15.119:0/1431487720 learned_addr learned my addr 172.21.15.119:0/1431487720 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76106f700 1 -- 172.21.15.119:0/1431487720 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 msgr2=0x7fa75c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76106f700 1 --2- 172.21.15.119:0/1431487720 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c06c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76106f700 1 -- 172.21.15.119:0/1431487720 >> v1:172.21.15.145:6789/0 conn(0x7fa75c06c680 legacy=0x7fa75c0ad9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.595+0000 7fa76106f700 1 -- 172.21.15.119:0/1431487720 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa75c06d430 con 0x7fa75c0ae0b0 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa76106f700 1 --2- 172.21.15.119:0/1431487720 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c0b1860 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa754009910 tx=0x7fa754009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e01cd1698786e15 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa75b7fe700 1 -- 172.21.15.119:0/1431487720 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754011830 con 0x7fa75c0ae0b0 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa75b7fe700 1 -- 172.21.15.119:0/1431487720 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa754011990 con 0x7fa75c0ae0b0 2024-09-13T16:32:51.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa75b7fe700 1 -- 172.21.15.119:0/1431487720 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754011d40 con 0x7fa75c0ae0b0 2024-09-13T16:32:51.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 -- 172.21.15.119:0/1431487720 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c0ae0b0 msgr2=0x7fa75c0b1860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1431487720 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c0b1860 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fa754009910 tx=0x7fa754009c20 comp rx=0 tx=0).stop 2024-09-13T16:32:51.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 -- 172.21.15.119:0/1431487720 shutdown_connections 2024-09-13T16:32:51.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1431487720 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c0b1860 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1431487720 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c06c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 -- 172.21.15.119:0/1431487720 >> 172.21.15.119:0/1431487720 conn(0x7fa75c0b0560 msgr2=0x7fa75c0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:51.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 -- 172.21.15.119:0/1431487720 shutdown_connections 2024-09-13T16:32:51.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 -- 172.21.15.119:0/1431487720 wait complete. 2024-09-13T16:32:51.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 Processor -- start 2024-09-13T16:32:51.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.599+0000 7fa762ad2700 1 -- start start 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c06c680 0x7fa75c14a140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c1e4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c15af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c156d00 con 0x7fa75c0ae0b0 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15be30 con 0x7fa75c06c680 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa75c15c110 con 0x7fa75c158f10 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa75bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c1e4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76086e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c06c680 0x7fa75c14a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76106f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c15af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa75bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c1e4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36264/0 (socket says 172.21.15.119:36264) 2024-09-13T16:32:51.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa75bfff700 1 -- 172.21.15.119:0/1335772512 learned_addr learned my addr 172.21.15.119:0/1335772512 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76086e700 1 -- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 msgr2=0x7fa75c15af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76086e700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c15af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76086e700 1 -- 172.21.15.119:0/1335772512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa75c0ae0b0 msgr2=0x7fa75c1e4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76086e700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c1e4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76086e700 1 -- 172.21.15.119:0/1335772512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa754009570 con 0x7fa75c06c680 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa76086e700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c06c680 0x7fa75c14a140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa74400e6e0 tx=0x7fa74400e9f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa7597fa700 1 -- 172.21.15.119:0/1335772512 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa744010820 con 0x7fa75c06c680 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa7597fa700 1 -- 172.21.15.119:0/1335772512 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa744010980 con 0x7fa75c06c680 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa7597fa700 1 -- 172.21.15.119:0/1335772512 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa744010cf0 con 0x7fa75c06c680 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa75c15c3f0 con 0x7fa75c06c680 2024-09-13T16:32:51.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.603+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c1e1de0 con 0x7fa75c06c680 2024-09-13T16:32:51.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.607+0000 7fa7597fa700 1 -- 172.21.15.119:0/1335772512 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa7440113f0 con 0x7fa75c06c680 2024-09-13T16:32:51.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.607+0000 7fa7597fa700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa73c066730 0x7fa73c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.607+0000 7fa7597fa700 1 -- 172.21.15.119:0/1335772512 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fa744083d40 con 0x7fa75c06c680 2024-09-13T16:32:51.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.607+0000 7fa75bfff700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa73c066730 0x7fa73c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.607+0000 7fa75bfff700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa73c066730 0x7fa73c068bf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0cd5e0 tx=0x7fa74c001b50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 Processor -- start 2024-09-13T16:32:51.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 -- start start 2024-09-13T16:32:51.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x1302240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1301d00 0x1323d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1324270 0x12d75d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x126b010 con 0x1324270 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x11f8f40 con 0x1323950 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x124be30 con 0x1301d00 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa76106f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1324270 0x12d75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa76106f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1324270 0x12d75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36266/0 (socket says 172.21.15.119:36266) 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa76106f700 1 -- 172.21.15.119:0/4040810936 learned_addr learned my addr 172.21.15.119:0/4040810936 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa76086e700 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x1302240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.611+0000 7fa75bfff700 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1301d00 0x1323d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa76106f700 1 -- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1301d00 msgr2=0x1323d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa76106f700 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1301d00 0x1323d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa76106f700 1 -- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 msgr2=0x1302240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa76106f700 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x1302240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa76106f700 1 -- 172.21.15.119:0/4040810936 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x10b7a40 con 0x1324270 2024-09-13T16:32:51.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa76086e700 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x1302240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:51.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa76106f700 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1324270 0x12d75d0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fa754013000 tx=0x7fa754010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d0b5f19b6f8456 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa7527fc700 1 -- 172.21.15.119:0/4040810936 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754020070 con 0x1324270 2024-09-13T16:32:51.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa7527fc700 1 -- 172.21.15.119:0/4040810936 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7540117b0 con 0x1324270 2024-09-13T16:32:51.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa7527fc700 1 -- 172.21.15.119:0/4040810936 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa754011d20 con 0x1324270 2024-09-13T16:32:51.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 -- 172.21.15.119:0/4040810936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1324270 msgr2=0x12d75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1324270 0x12d75d0 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 crypto rx=0x7fa754013000 tx=0x7fa754010000 comp rx=0 tx=0).stop 2024-09-13T16:32:51.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 -- 172.21.15.119:0/4040810936 shutdown_connections 2024-09-13T16:32:51.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1324270 0x12d75d0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1301d00 0x1323d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 --2- 172.21.15.119:0/4040810936 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x1302240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 -- 172.21.15.119:0/4040810936 >> 172.21.15.119:0/4040810936 conn(0x106c5b0 msgr2=0x12dd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:51.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 -- 172.21.15.119:0/4040810936 shutdown_connections 2024-09-13T16:32:51.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 -- 172.21.15.119:0/4040810936 wait complete. 2024-09-13T16:32:51.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 -- start start 2024-09-13T16:32:51.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.615+0000 7fa765d83740 1 Processor -- start 2024-09-13T16:32:51.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ed8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x12eac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1324270 0x12ec3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1247150 con 0x1301d00 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1244eb0 con 0x1323950 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x12028e0 con 0x1324270 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ed8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 10 client.0 _ll_register_callbacks cb 0x11e1670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ed8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36270/0 (socket says 172.21.15.119:36270) 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 -- 172.21.15.119:0/2583578523 learned_addr learned my addr 172.21.15.119:0/2583578523 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa75bfff700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x12eac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76106f700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1324270 0x12ec3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 -- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1324270 msgr2=0x12ec3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1324270 0x12ec3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 -- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 msgr2=0x12eac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x12eac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 -- 172.21.15.119:0/2583578523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c06d430 con 0x1301d00 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa75bfff700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x12eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa76086e700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ed8c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fa744000c00 tx=0x7fa744054b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72aba7fbfc9640b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa73b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2583578523 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74406c070 con 0x1301d00 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2583578523 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fa74405cf40 con 0x1301d00 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x10792c0 con 0x1301d00 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2583578523 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa744054240 con 0x1301d00 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 10 client.8325 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:51.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x120b290 con 0x1301d00 2024-09-13T16:32:51.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2583578523 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x10792c0 con 0x1301d00 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2583578523 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa74405cab0 con 0x1301d00 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 10 client.8325 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 10 client.8325 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 20 client.8325 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x12ed1c0 0x1302ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 4 client.8325 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x10c3540 con 0x12ed1c0 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa75bfff700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x12ed1c0 0x1302ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa73b7fe700 10 client.8325 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.619+0000 7fa75bfff700 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x12ed1c0 0x1302ca0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbb67e15a54988b1 server_cookie=214ae1f67146d88e in_seq=0 out_seq=0 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2583578523 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x10c3540 con 0x12ed1c0 2024-09-13T16:32:51.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa73b7fe700 10 client.8325 handle_command_reply: tid=0 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 client.8325 shutdown 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 msgr2=0x12ed8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ed8c0 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 crypto rx=0x7fa744000c00 tx=0x7fa744054b50 comp rx=0 tx=0).stop 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 shutdown_connections 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x12ed1c0 0x1302ca0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1324270 0x12ec3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1323950 0x12eac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa73b7fe700 0 client.8325 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 --2- 172.21.15.119:0/2583578523 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ed8c0 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 >> 172.21.15.119:0/2583578523 conn(0x106c5b0 msgr2=0x12d8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 shutdown_connections 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 1 -- 172.21.15.119:0/2583578523 wait complete. 2024-09-13T16:32:51.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.623+0000 7fa765d83740 20 client.8325 trim_cache size 0 max 16384 2024-09-13T16:32:51.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.639+0000 7fa765d83740 1 Processor -- start 2024-09-13T16:32:51.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.639+0000 7fa765d83740 1 -- start start 2024-09-13T16:32:51.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ed1c0 0x13010f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ff810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x12ffd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x10f0b80 con 0x1301d00 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1243bd0 con 0x12ed1c0 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x12426c0 con 0x1323950 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ff810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ff810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36280/0 (socket says 172.21.15.119:36280) 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa76106f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x12ffd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 -- 172.21.15.119:0/1099447749 learned_addr learned my addr 172.21.15.119:0/1099447749 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa76086e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ed1c0 0x13010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 -- 172.21.15.119:0/1099447749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 msgr2=0x12ffd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 --2- 172.21.15.119:0/1099447749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x12ffd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 -- 172.21.15.119:0/1099447749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ed1c0 msgr2=0x13010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 --2- 172.21.15.119:0/1099447749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ed1c0 0x13010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 -- 172.21.15.119:0/1099447749 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x11f14c0 con 0x1301d00 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa75bfff700 1 --2- 172.21.15.119:0/1099447749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ff810 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0135d0 tx=0x7fa74c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ffa9ba7fc185904 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa752ffd700 1 -- 172.21.15.119:0/1099447749 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c013ba0 con 0x1301d00 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa752ffd700 1 -- 172.21.15.119:0/1099447749 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa74c013d00 con 0x1301d00 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa752ffd700 1 -- 172.21.15.119:0/1099447749 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74c013e70 con 0x1301d00 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- 172.21.15.119:0/1099447749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 msgr2=0x12ff810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 --2- 172.21.15.119:0/1099447749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ff810 secure :-1 s=READY pgs=1372 cs=0 l=1 rev1=1 crypto rx=0x7fa74c0135d0 tx=0x7fa74c013000 comp rx=0 tx=0).stop 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- 172.21.15.119:0/1099447749 shutdown_connections 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 --2- 172.21.15.119:0/1099447749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x12ffd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 --2- 172.21.15.119:0/1099447749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1301d00 0x12ff810 unknown :-1 s=CLOSED pgs=1372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 --2- 172.21.15.119:0/1099447749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ed1c0 0x13010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- 172.21.15.119:0/1099447749 >> 172.21.15.119:0/1099447749 conn(0x106c5b0 msgr2=0x12dd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- 172.21.15.119:0/1099447749 shutdown_connections 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.643+0000 7fa765d83740 1 -- 172.21.15.119:0/1099447749 wait complete. 2024-09-13T16:32:51.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 -- start start 2024-09-13T16:32:51.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 Processor -- start 2024-09-13T16:32:51.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ed1c0 0x133bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1301d00 0x1384a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x1384fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x11e3620 con 0x12ed1c0 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1215c00 con 0x1301d00 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x11cea50 con 0x1323950 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 10 client.0 _ll_register_callbacks cb 0x12148f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ed1c0 0x133bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ed1c0 0x133bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36284/0 (socket says 172.21.15.119:36284) 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 -- 172.21.15.119:0/2857252260 learned_addr learned my addr 172.21.15.119:0/2857252260 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76106f700 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x1384fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa75bfff700 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1301d00 0x1384a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 -- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 msgr2=0x1384fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x1384fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 -- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1301d00 msgr2=0x1384a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1301d00 0x1384a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 -- 172.21.15.119:0/2857252260 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa75c06cb90 con 0x12ed1c0 2024-09-13T16:32:51.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa76086e700 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ed1c0 0x133bf90 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fa744052ea0 tx=0x7fa744081f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6dcca2a5749ff3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa73b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:51.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2857252260 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa74406c040 con 0x12ed1c0 2024-09-13T16:32:51.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2857252260 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fa74405cf40 con 0x12ed1c0 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7540117b0 con 0x12ed1c0 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2857252260 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa744050e90 con 0x12ed1c0 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 10 client.8331 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x11e5790 con 0x12ed1c0 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.647+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2857252260 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7540117b0 con 0x12ed1c0 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2857252260 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa744060020 con 0x12ed1c0 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 10 client.8331 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 10 client.8331 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 20 client.8331 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1324270 0x1339b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 4 client.8331 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x10e82c0 con 0x1324270 2024-09-13T16:32:51.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa75bfff700 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1324270 0x1339b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa73b7fe700 10 client.8331 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa75bfff700 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1324270 0x1339b60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f4aa1177c025b83 server_cookie=c9046dd5df7fa08b in_seq=0 out_seq=0 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa73b7fe700 1 -- 172.21.15.119:0/2857252260 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x10e82c0 con 0x1324270 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa73b7fe700 10 client.8331 handle_command_reply: tid=0 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 client.8331 shutdown 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ed1c0 msgr2=0x133bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ed1c0 0x133bf90 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7fa744052ea0 tx=0x7fa744081f20 comp rx=0 tx=0).stop 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 shutdown_connections 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1324270 0x1339b60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1323950 0x1384fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1301d00 0x1384a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa73b7fe700 0 client.8331 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 --2- 172.21.15.119:0/2857252260 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ed1c0 0x133bf90 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 >> 172.21.15.119:0/2857252260 conn(0x106c5b0 msgr2=0x12dd590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 shutdown_connections 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 1 -- 172.21.15.119:0/2857252260 wait complete. 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa765d83740 20 client.8331 trim_cache size 0 max 16384 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:32:51.660 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:32:51.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa73c066730 msgr2=0x7fa73c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa73c066730 0x7fa73c068bf0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fa75c0cd5e0 tx=0x7fa74c001b50 comp rx=0 tx=0).stop 2024-09-13T16:32:51.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c06c680 msgr2=0x7fa75c14a140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c06c680 0x7fa75c14a140 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fa74400e6e0 tx=0x7fa74400e9f0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 shutdown_connections 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa73c066730 0x7fa73c068bf0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa75c158f10 0x7fa75c15af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa75c0ae0b0 0x7fa75c1e4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 --2- 172.21.15.119:0/1335772512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa75c06c680 0x7fa75c14a140 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 >> 172.21.15.119:0/1335772512 conn(0x7fa75c0b0560 msgr2=0x7fa75c0ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 shutdown_connections 2024-09-13T16:32:51.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.651+0000 7fa762ad2700 1 -- 172.21.15.119:0/1335772512 wait complete. 2024-09-13T16:32:51.673 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:32:51.673 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:32:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 Processor -- start 2024-09-13T16:32:51.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- start start 2024-09-13T16:32:51.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f02781444b0 0x7f027806d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f02780cbb10 0x7f02780cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027806bd60 con 0x7f02781440d0 2024-09-13T16:32:51.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02780cc4c0 con 0x7f02780cbb10 2024-09-13T16:32:51.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027806bfe0 con 0x7f02781444b0 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f02780cbb10 0x7f02780cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f02780cbb10 0x7f02780cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36290/0 (socket says 172.21.15.119:36290) 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 -- 172.21.15.119:0/741390536 learned_addr learned my addr 172.21.15.119:0/741390536 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f026dd9b700 1 --2- 172.21.15.119:0/741390536 >> v2:172.21.15.145:3300/0 conn(0x7f02781444b0 0x7f027806d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 -- 172.21.15.119:0/741390536 >> v2:172.21.15.145:3300/0 conn(0x7f02781444b0 msgr2=0x7f027806d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 --2- 172.21.15.119:0/741390536 >> v2:172.21.15.145:3300/0 conn(0x7f02781444b0 0x7f027806d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 -- 172.21.15.119:0/741390536 >> v1:172.21.15.145:6789/0 conn(0x7f02781440d0 legacy=0x7f027815b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 -- 172.21.15.119:0/741390536 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027806c2c0 con 0x7f02780cbb10 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f026dd9b700 1 --2- 172.21.15.119:0/741390536 >> v2:172.21.15.145:3300/0 conn(0x7f02781444b0 0x7f027806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0276d9d700 1 --2- 172.21.15.119:0/741390536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02780cbb10 0x7f02780cbf80 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f02780b4030 tx=0x7f027000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e50c154b646d2bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0275d9b700 1 -- 172.21.15.119:0/741390536 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027000e070 con 0x7f02780cbb10 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0275d9b700 1 -- 172.21.15.119:0/741390536 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f027000b9c0 con 0x7f02780cbb10 2024-09-13T16:32:51.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f0275d9b700 1 -- 172.21.15.119:0/741390536 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027000bd70 con 0x7f02780cbb10 2024-09-13T16:32:51.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- 172.21.15.119:0/741390536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02780cbb10 msgr2=0x7f02780cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 --2- 172.21.15.119:0/741390536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02780cbb10 0x7f02780cbf80 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f02780b4030 tx=0x7f027000b330 comp rx=0 tx=0).stop 2024-09-13T16:32:51.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- 172.21.15.119:0/741390536 shutdown_connections 2024-09-13T16:32:51.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 --2- 172.21.15.119:0/741390536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02780cbb10 0x7f02780cbf80 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 --2- 172.21.15.119:0/741390536 >> v2:172.21.15.145:3300/0 conn(0x7f02781444b0 0x7f027806d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- 172.21.15.119:0/741390536 >> 172.21.15.119:0/741390536 conn(0x7f027806a350 msgr2=0x7f027806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:51.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- 172.21.15.119:0/741390536 shutdown_connections 2024-09-13T16:32:51.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.791+0000 7f027c985700 1 -- 172.21.15.119:0/741390536 wait complete. 2024-09-13T16:32:51.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 Processor -- start 2024-09-13T16:32:51.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 -- start start 2024-09-13T16:32:51.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02780cbb10 0x7f027815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02781440d0 0x7f02780acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02781444b0 0x7f02781f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02781f18c0 con 0x7f02781444b0 2024-09-13T16:32:51.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027815d720 con 0x7f02781440d0 2024-09-13T16:32:51.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.795+0000 7f027c985700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027815da00 con 0x7f02780cbb10 2024-09-13T16:32:51.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02781444b0 0x7f02781f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f027659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02780cbb10 0x7f027815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02781444b0 0x7f02781f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36294/0 (socket says 172.21.15.119:36294) 2024-09-13T16:32:51.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 -- 172.21.15.119:0/2824838092 learned_addr learned my addr 172.21.15.119:0/2824838092 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:51.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f026dd9b700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02781440d0 0x7f02780acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 -- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02780cbb10 msgr2=0x7f027815ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02780cbb10 0x7f027815ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 -- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02781440d0 msgr2=0x7f02780acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02781440d0 0x7f02780acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 -- 172.21.15.119:0/2824838092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0270009b30 con 0x7f02781444b0 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f0276d9d700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02781444b0 0x7f02781f3a10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0278072100 tx=0x7f0270004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f026f7fe700 1 -- 172.21.15.119:0/2824838092 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0270004880 con 0x7f02781444b0 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f026f7fe700 1 -- 172.21.15.119:0/2824838092 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02700099b0 con 0x7f02781444b0 2024-09-13T16:32:51.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027815dce0 con 0x7f02781444b0 2024-09-13T16:32:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f026f7fe700 1 -- 172.21.15.119:0/2824838092 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f027001c680 con 0x7f02781444b0 2024-09-13T16:32:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027815d350 con 0x7f02781444b0 2024-09-13T16:32:51.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.799+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02781f1a70 con 0x7f02781444b0 2024-09-13T16:32:51.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.803+0000 7f026f7fe700 1 -- 172.21.15.119:0/2824838092 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f027001c7e0 con 0x7f02781444b0 2024-09-13T16:32:51.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.803+0000 7f026f7fe700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0258066760 0x7f0258068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:51.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.803+0000 7f026f7fe700 1 -- 172.21.15.119:0/2824838092 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f0270085f40 con 0x7f02781444b0 2024-09-13T16:32:51.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.803+0000 7f027659c700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0258066760 0x7f0258068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:51.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.803+0000 7f027659c700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0258066760 0x7f0258068c20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0268000f60 tx=0x7f0268009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:51.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:51.811+0000 7f026f7fe700 1 -- 172.21.15.119:0/2824838092 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f02781f1a70 con 0x7f02781444b0 2024-09-13T16:32:52.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.015+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0278067760 con 0x7f02781444b0 2024-09-13T16:32:52.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.015+0000 7f026f7fe700 1 -- 172.21.15.119:0/2824838092 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f0270053010 con 0x7f02781444b0 2024-09-13T16:32:52.022 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:32:52.022 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":63,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:57.084733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7881":{"gid":7881,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":11,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0258066760 msgr2=0x7f0258068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0258066760 0x7f0258068c20 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0268000f60 tx=0x7f0268009040 comp rx=0 tx=0).stop 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02781444b0 msgr2=0x7f02781f3a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02781444b0 0x7f02781f3a10 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f0278072100 tx=0x7f0270004af0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 shutdown_connections 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0258066760 0x7f0258068c20 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02781444b0 0x7f02781f3a10 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02781440d0 0x7f02780acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 --2- 172.21.15.119:0/2824838092 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02780cbb10 0x7f027815ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 >> 172.21.15.119:0/2824838092 conn(0x7f027806a350 msgr2=0x7f027806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 shutdown_connections 2024-09-13T16:32:52.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.019+0000 7f027c985700 1 -- 172.21.15.119:0/2824838092 wait complete. 2024-09-13T16:32:52.026 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 63 2024-09-13T16:32:52.036 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-09-13T16:32:52.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfd32f700 1 Processor -- start 2024-09-13T16:32:52.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfd32f700 1 -- start start 2024-09-13T16:32:52.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfd32f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf8144180 0x7f6cf806c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfd32f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6cf80cc010 0x7f6cf80cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfd32f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf806cb40 con 0x7f6cf80ce000 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfd32f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf80b1c50 con 0x7f6cf80cc010 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfd32f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf806ce20 con 0x7f6cf8144180 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cf77fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6cf80cc010 0x7f6cf80cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfcb2e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6cf80ce000 0x7f6cf815e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:39758/0 (socket says 172.21.15.119:39758) 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cfcb2e700 1 -- 172.21.15.119:0/541676087 learned_addr learned my addr 172.21.15.119:0/541676087 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:52.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.147+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/541676087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf8144180 0x7f6cf806c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf77fe700 1 -- 172.21.15.119:0/541676087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf8144180 msgr2=0x7f6cf806c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/541676087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf8144180 0x7f6cf806c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf77fe700 1 -- 172.21.15.119:0/541676087 >> v1:172.21.15.119:6789/0 conn(0x7f6cf80ce000 legacy=0x7f6cf815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf77fe700 1 -- 172.21.15.119:0/541676087 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf806d100 con 0x7f6cf80cc010 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf6ffd700 1 -- 172.21.15.119:0/541676087 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723243250 0 0) 0x7f6cf806cb40 con 0x7f6cf80ce000 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/541676087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf80cc010 0x7f6cf80cdac0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009850 tx=0x7f6ce0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b574eaed4a1ded server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf6ffd700 1 -- 172.21.15.119:0/541676087 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0011830 con 0x7f6cf80cc010 2024-09-13T16:32:52.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf6ffd700 1 -- 172.21.15.119:0/541676087 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce0011990 con 0x7f6cf80cc010 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cf6ffd700 1 -- 172.21.15.119:0/541676087 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0011d40 con 0x7f6cf80cc010 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 -- 172.21.15.119:0/541676087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf80cc010 msgr2=0x7f6cf80cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/541676087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf80cc010 0x7f6cf80cdac0 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009850 tx=0x7f6ce0009b60 comp rx=0 tx=0).stop 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 -- 172.21.15.119:0/541676087 shutdown_connections 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/541676087 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf80cc010 0x7f6cf80cdac0 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/541676087 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf8144180 0x7f6cf806c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 -- 172.21.15.119:0/541676087 >> 172.21.15.119:0/541676087 conn(0x7f6cf80b0980 msgr2=0x7f6cf80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 -- 172.21.15.119:0/541676087 shutdown_connections 2024-09-13T16:32:52.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.151+0000 7f6cfd32f700 1 -- 172.21.15.119:0/541676087 wait complete. 2024-09-13T16:32:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 Processor -- start 2024-09-13T16:32:52.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 -- start start 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf80cc010 0x7f6cf80ac120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf80ce000 0x7f6cf80ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf8144180 0x7f6cf81318f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf80ab040 con 0x7f6cf8144180 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf80aa3c0 con 0x7f6cf80ce000 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf80aa6a0 con 0x7f6cf80cc010 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf8144180 0x7f6cf81318f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf8144180 0x7f6cf81318f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36318/0 (socket says 172.21.15.119:36318) 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cf7fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf80cc010 0x7f6cf80ac120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 -- 172.21.15.119:0/4102674305 learned_addr learned my addr 172.21.15.119:0/4102674305 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf80ce000 0x7f6cf80ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 -- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf80cc010 msgr2=0x7f6cf80ac120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf80cc010 0x7f6cf80ac120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 -- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf80ce000 msgr2=0x7f6cf80ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf80ce000 0x7f6cf80ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 -- 172.21.15.119:0/4102674305 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00094b0 con 0x7f6cf8144180 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfcb2e700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf8144180 0x7f6cf81318f0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0002230 tx=0x7f6cf000cf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cf4ff9700 1 -- 172.21.15.119:0/4102674305 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf000aac0 con 0x7f6cf8144180 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cf4ff9700 1 -- 172.21.15.119:0/4102674305 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cf000ac40 con 0x7f6cf8144180 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf80aa980 con 0x7f6cf8144180 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.155+0000 7f6cf4ff9700 1 -- 172.21.15.119:0/4102674305 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cf00070d0 con 0x7f6cf8144180 2024-09-13T16:32:52.164 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.159+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf80ac660 con 0x7f6cf8144180 2024-09-13T16:32:52.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.159+0000 7f6cf4ff9700 1 -- 172.21.15.119:0/4102674305 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6cf0007250 con 0x7f6cf8144180 2024-09-13T16:32:52.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.159+0000 7f6cf4ff9700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cdc066730 0x7f6cdc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.159+0000 7f6cf4ff9700 1 -- 172.21.15.119:0/4102674305 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f6cf00840d0 con 0x7f6cf8144180 2024-09-13T16:32:52.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.159+0000 7f6d01842740 1 Processor -- start 2024-09-13T16:32:52.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.159+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cdc066730 0x7f6cdc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.159+0000 7f6d01842740 1 -- start start 2024-09-13T16:32:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cdc066730 0x7f6cdc068bf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009b50 tx=0x7f6ce8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c41520 0x1cf8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8d30 0x1cf5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c20810 con 0x1cf8d30 2024-09-13T16:32:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c1dfa0 con 0x1c41520 2024-09-13T16:32:52.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c19890 con 0x1cf8950 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cfcb2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8d30 0x1cf5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cfcb2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8d30 0x1cf5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36326/0 (socket says 172.21.15.119:36326) 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cfcb2e700 1 -- 172.21.15.119:0/509669425 learned_addr learned my addr 172.21.15.119:0/509669425 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c41520 0x1cf8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 -- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 msgr2=0x1cf4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 -- 172.21.15.119:0/509669425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8d30 msgr2=0x1cf5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8d30 0x1cf5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 -- 172.21.15.119:0/509669425 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1a8bba0 con 0x1c41520 2024-09-13T16:32:52.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cfcb2e700 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8d30 0x1cf5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c41520 0x1cf8410 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8004440 tx=0x7f6ce8014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99efa98a230c51a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cedffb700 1 -- 172.21.15.119:0/509669425 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce801c040 con 0x1c41520 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cedffb700 1 -- 172.21.15.119:0/509669425 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce80036c0 con 0x1c41520 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6cedffb700 1 -- 172.21.15.119:0/509669425 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8003a30 con 0x1c41520 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c41520 msgr2=0x1cf8410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c41520 0x1cf8410 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8004440 tx=0x7f6ce8014000 comp rx=0 tx=0).stop 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- 172.21.15.119:0/509669425 shutdown_connections 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8d30 0x1cf5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 --2- 172.21.15.119:0/509669425 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c41520 0x1cf8410 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- 172.21.15.119:0/509669425 >> 172.21.15.119:0/509669425 conn(0x1a40710 msgr2=0x1cda6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- 172.21.15.119:0/509669425 shutdown_connections 2024-09-13T16:32:52.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- 172.21.15.119:0/509669425 wait complete. 2024-09-13T16:32:52.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.163+0000 7f6d01842740 1 -- start start 2024-09-13T16:32:52.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 Processor -- start 2024-09-13T16:32:52.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1cb4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1cb0150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cb0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bdcd80 con 0x1cf8950 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bdca20 con 0x1cf8d30 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b0c770 con 0x1c41520 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 10 client.0 _ll_register_callbacks cb 0x1c12f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1cb0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1cb0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36330/0 (socket says 172.21.15.119:36330) 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 -- 172.21.15.119:0/3246153072 learned_addr learned my addr 172.21.15.119:0/3246153072 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1cb4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cfcb2e700 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cb0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 -- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 msgr2=0x1cb4a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1cb4a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 -- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 msgr2=0x1cb0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cb0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 -- 172.21.15.119:0/3246153072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf80ac660 con 0x1cf8950 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1cb0150 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00090b0 tx=0x7f6ce000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ccb4adcda51290 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6ce6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/3246153072 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce0020040 con 0x1cf8950 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1a4d420 con 0x1cf8950 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/3246153072 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f6ce001b840 con 0x1cf8950 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/3246153072 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ce001ba00 con 0x1cf8950 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 10 client.8352 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x7f6ce8004120 con 0x1cf8950 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/3246153072 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1a4d420 con 0x1cf8950 2024-09-13T16:32:52.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/3246153072 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6ce0023030 con 0x1cf8950 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 10 client.8352 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 10 client.8352 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 20 client.8352 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cb1cd0 0x1cadff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 4 client.8352 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1a976a0 con 0x1cb1cd0 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.167+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cb1cd0 0x1cadff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cb1cd0 0x1cadff0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c34b5c3883fceaa5 server_cookie=9e0108cb33cf1af1 in_seq=0 out_seq=0 2024-09-13T16:32:52.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6ce6ffd700 10 client.8352 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:52.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/3246153072 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1a976a0 con 0x1cb1cd0 2024-09-13T16:32:52.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6ce6ffd700 10 client.8352 handle_command_reply: tid=0 2024-09-13T16:32:52.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 client.8352 shutdown 2024-09-13T16:32:52.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 msgr2=0x1cb0150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1cb0150 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7f6ce00090b0 tx=0x7f6ce000e000 comp rx=0 tx=0).stop 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 shutdown_connections 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cb1cd0 0x1cadff0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cb0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1cb0150 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 --2- 172.21.15.119:0/3246153072 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1cb4a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 >> 172.21.15.119:0/3246153072 conn(0x1a40710 msgr2=0x1cf7320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6ce6ffd700 0 client.8352 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 shutdown_connections 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 1 -- 172.21.15.119:0/3246153072 wait complete. 2024-09-13T16:32:52.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.171+0000 7f6d01842740 20 client.8352 trim_cache size 0 max 16384 2024-09-13T16:32:52.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.187+0000 7f6d01842740 1 Processor -- start 2024-09-13T16:32:52.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.187+0000 7f6d01842740 1 -- start start 2024-09-13T16:32:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c41520 0x1cf4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cf4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bda660 con 0x1c41520 2024-09-13T16:32:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c21aa0 con 0x1cf8d30 2024-09-13T16:32:52.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c1c3f0 con 0x1cf8950 2024-09-13T16:32:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c41520 0x1cf4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf77fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c41520 0x1cf4560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36332/0 (socket says 172.21.15.119:36332) 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 -- 172.21.15.119:0/3283238488 learned_addr learned my addr 172.21.15.119:0/3283238488 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cfcb2e700 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cf4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 -- 172.21.15.119:0/3283238488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 msgr2=0x1cf4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 -- 172.21.15.119:0/3283238488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 msgr2=0x1cf4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cf4fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 -- 172.21.15.119:0/3283238488 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1cb0a40 con 0x1c41520 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c41520 0x1cf4560 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8023000 tx=0x7f6ce8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b9b448c6ef82145 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cee7fc700 1 -- 172.21.15.119:0/3283238488 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8031040 con 0x1c41520 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cee7fc700 1 -- 172.21.15.119:0/3283238488 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8037620 con 0x1c41520 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6cee7fc700 1 -- 172.21.15.119:0/3283238488 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce80038b0 con 0x1c41520 2024-09-13T16:32:52.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- 172.21.15.119:0/3283238488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c41520 msgr2=0x1cf4560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c41520 0x1cf4560 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8023000 tx=0x7f6ce8016000 comp rx=0 tx=0).stop 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- 172.21.15.119:0/3283238488 shutdown_connections 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1cf4fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1cf8950 0x1cf4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 --2- 172.21.15.119:0/3283238488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c41520 0x1cf4560 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- 172.21.15.119:0/3283238488 >> 172.21.15.119:0/3283238488 conn(0x1a40710 msgr2=0x1cd66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- 172.21.15.119:0/3283238488 shutdown_connections 2024-09-13T16:32:52.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- 172.21.15.119:0/3283238488 wait complete. 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 -- start start 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.191+0000 7f6d01842740 1 Processor -- start 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1d19800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1d5c180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1d5c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bd70e0 con 0x1cf8950 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1cb40a0 con 0x1cf8d30 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1d19d40 con 0x1c41520 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 10 client.0 _ll_register_callbacks cb 0x1c18460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1d5c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1d5c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36342/0 (socket says 172.21.15.119:36342) 2024-09-13T16:32:52.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 -- 172.21.15.119:0/122756809 learned_addr learned my addr 172.21.15.119:0/122756809 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1d19800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cfcb2e700 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1d5c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 -- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 msgr2=0x1d19800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1d19800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 -- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 msgr2=0x1d5c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1d5c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 -- 172.21.15.119:0/122756809 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1a8bba0 con 0x1cf8950 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf77fe700 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1d5c180 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009850 tx=0x7f6ce00047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e41b463e64d7c409 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/122756809 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce001ba50 con 0x1cf8950 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6ce80036c0 con 0x1cf8950 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/122756809 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f6ce0000e40 con 0x1cf8950 2024-09-13T16:32:52.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/122756809 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=63) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ce001bc00 con 0x1cf8950 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 10 client.8358 fetch_fsmap learned FSMap version 63 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=63,monmap=2+}) v3 -- 0x1bdf910 con 0x1cf8950 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/122756809 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce80036c0 con 0x1cf8950 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/122756809 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 63) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f6ce0009bd0 con 0x1cf8950 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 10 client.8358 fetch_fsmap finished waiting for FSMap version 63 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 10 client.8358 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 20 client.8358 populate_metadata read hostname 'smithi119' 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cf7640 0x1d5d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 4 client.8358 mds_command: new command op to 7881 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 --> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x1abc420 con 0x1cf7640 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cf7640 0x1d5d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 10 client.8358 ms_handle_connect on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:52.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6cf7fff700 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cf7640 0x1d5d740 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7e9d8d853934884 server_cookie=84280ce6b230013b in_seq=0 out_seq=0 2024-09-13T16:32:52.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 1 -- 172.21.15.119:0/122756809 <== mds.0 v2:172.21.15.119:6834/3937841911 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+143 (crc 0 0 0) 0x1abc420 con 0x1cf7640 2024-09-13T16:32:52.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.195+0000 7f6ce6ffd700 10 client.8358 handle_command_reply: tid=0 2024-09-13T16:32:52.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 client.8358 shutdown 2024-09-13T16:32:52.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 msgr2=0x1d5c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1d5c180 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0009850 tx=0x7f6ce00047e0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 shutdown_connections 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:6834/3937841911,v1:172.21.15.119:6835/3937841911] conn(0x1cf7640 0x1d5d740 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1cf8d30 0x1d5c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1cf8950 0x1d5c180 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 --2- 172.21.15.119:0/122756809 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c41520 0x1d19800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 >> 172.21.15.119:0/122756809 conn(0x1a40710 msgr2=0x1cd6ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6ce6ffd700 0 client.8358 ms_handle_reset on v2:172.21.15.119:6834/3937841911 2024-09-13T16:32:52.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 shutdown_connections 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 1 -- 172.21.15.119:0/122756809 wait complete. 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6d01842740 20 client.8358 trim_cache size 0 max 16384 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stdout: "damage_type": "backtrace", 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 278457655, 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stdout: "ino": 1099511627777, 2024-09-13T16:32:52.208 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/testdir/hardlink" 2024-09-13T16:32:52.209 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:32:52.209 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:32:52.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cdc066730 msgr2=0x7f6cdc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cdc066730 0x7f6cdc068bf0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f6ce8009b50 tx=0x7f6ce8009240 comp rx=0 tx=0).stop 2024-09-13T16:32:52.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf8144180 msgr2=0x7f6cf81318f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf8144180 0x7f6cf81318f0 secure :-1 s=READY pgs=1378 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0002230 tx=0x7f6cf000cf50 comp rx=0 tx=0).stop 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 shutdown_connections 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cdc066730 0x7f6cdc068bf0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf8144180 0x7f6cf81318f0 unknown :-1 s=CLOSED pgs=1378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf80ce000 0x7f6cf80ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 --2- 172.21.15.119:0/4102674305 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf80cc010 0x7f6cf80ac120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 >> 172.21.15.119:0/4102674305 conn(0x7f6cf80b0980 msgr2=0x7f6cf8066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 shutdown_connections 2024-09-13T16:32:52.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.199+0000 7f6cfd32f700 1 -- 172.21.15.119:0/4102674305 wait complete. 2024-09-13T16:32:52.218 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:32:52.218 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:32:52.225 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:32:52.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.335+0000 7f9b4ab0d700 1 Processor -- start 2024-09-13T16:32:52.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.335+0000 7f9b4ab0d700 1 -- start start 2024-09-13T16:32:52.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9b3c033c50 0x7f9b3c005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c0df440 con 0x7f9b3c01ce60 2024-09-13T16:32:52.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3c006050 con 0x7f9b3c04fc90 2024-09-13T16:32:52.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3c0e39f0 con 0x7f9b3c033c50 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4a30c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9b3c01ce60 0x7f9b3c01d270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:39774/0 (socket says 172.21.15.119:39774) 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4a30c700 1 -- 172.21.15.119:0/1326252230 learned_addr learned my addr 172.21.15.119:0/1326252230 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b49b0b700 1 --2- 172.21.15.119:0/1326252230 >> v2:172.21.15.145:3300/0 conn(0x7f9b3c033c50 0x7f9b3c005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909311769 0 0) 0x7f9b44068670 con 0x7f9b3c01ce60 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b300035d0 con 0x7f9b3c01ce60 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 733546693 0 0) 0x7f9b300035d0 con 0x7f9b3c01ce60 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 >> v2:172.21.15.145:3300/0 conn(0x7f9b3c033c50 msgr2=0x7f9b3c005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 --2- 172.21.15.119:0/1326252230 >> v2:172.21.15.145:3300/0 conn(0x7f9b3c033c50 0x7f9b3c005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 >> v1:172.21.15.145:6789/0 conn(0x7f9b3c04fc90 legacy=0x7f9b3c0e32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c0062f0 con 0x7f9b3c01ce60 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f9b4404ef50 con 0x7f9b3c01ce60 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 >> v1:172.21.15.119:6789/0 conn(0x7f9b3c01ce60 legacy=0x7f9b3c01d270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b30003a20 0x7f9b30003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b30004890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b30005090 0x7f9b3000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0e39f0 con 0x7f9b300043c0 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3000bfd0 con 0x7f9b30003a20 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3000c190 con 0x7f9b30005090 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4930a700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b30004890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b49b0b700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b30003a20 0x7f9b30003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4a30c700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b30005090 0x7f9b3000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4930a700 1 -- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b30005090 msgr2=0x7f9b3000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4930a700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b30005090 0x7f9b3000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4930a700 1 -- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b30003a20 msgr2=0x7f9b30003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4930a700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b30003a20 0x7f9b30003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4a30c700 1 -- 172.21.15.119:0/1326252230 reap_dead start 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4930a700 1 -- 172.21.15.119:0/1326252230 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b300035d0 con 0x7f9b300043c0 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b49b0b700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b30003a20 0x7f9b30003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4930a700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b30004890 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9b400017e0 tx=0x7f9b300058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=883f7d93df9f6015 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400052d0 con 0x7f9b300043c0 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b40005430 con 0x7f9b300043c0 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b48b09700 1 -- 172.21.15.119:0/1326252230 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b400055a0 con 0x7f9b300043c0 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/1326252230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 msgr2=0x7f9b30004890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b30004890 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f9b400017e0 tx=0x7f9b300058a0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/1326252230 shutdown_connections 2024-09-13T16:32:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/1326252230 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b30004890 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/1326252230 >> 172.21.15.119:0/1326252230 conn(0x7f9b3c036140 msgr2=0x7f9b3c036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/1326252230 shutdown_connections 2024-09-13T16:32:52.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.339+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/1326252230 wait complete. 2024-09-13T16:32:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 Processor -- start 2024-09-13T16:32:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 -- start start 2024-09-13T16:32:52.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b3c0d2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b3c0d34e0 0x7f9b3c0dfc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b3c0d7ad0 0x7f9b3c0e0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0d12a0 con 0x7f9b300043c0 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c0e0690 con 0x7f9b3c0d7ad0 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b4ab0d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3c03a540 con 0x7f9b3c0d34e0 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b49b0b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b3c0d2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b49b0b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b3c0d2fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36356/0 (socket says 172.21.15.119:36356) 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.343+0000 7f9b49b0b700 1 -- 172.21.15.119:0/3732674745 learned_addr learned my addr 172.21.15.119:0/3732674745 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b4930a700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b3c0d34e0 0x7f9b3c0dfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b4a30c700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b3c0d7ad0 0x7f9b3c0e0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b49b0b700 1 -- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b3c0d34e0 msgr2=0x7f9b3c0dfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b49b0b700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b3c0d34e0 0x7f9b3c0dfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b49b0b700 1 -- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b3c0d7ad0 msgr2=0x7f9b3c0e0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b49b0b700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b3c0d7ad0 0x7f9b3c0e0150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b49b0b700 1 -- 172.21.15.119:0/3732674745 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3c0d0fc0 con 0x7f9b300043c0 2024-09-13T16:32:52.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b49b0b700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b3c0d2fa0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f9b380115c0 tx=0x7f9b380118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b367fc700 1 -- 172.21.15.119:0/3732674745 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38005c50 con 0x7f9b300043c0 2024-09-13T16:32:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b367fc700 1 -- 172.21.15.119:0/3732674745 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b38005db0 con 0x7f9b300043c0 2024-09-13T16:32:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b3c03a260 con 0x7f9b300043c0 2024-09-13T16:32:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b367fc700 1 -- 172.21.15.119:0/3732674745 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b38004240 con 0x7f9b300043c0 2024-09-13T16:32:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b3c035b40 con 0x7f9b300043c0 2024-09-13T16:32:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.347+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b3c03a0c0 con 0x7f9b300043c0 2024-09-13T16:32:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.351+0000 7f9b367fc700 1 -- 172.21.15.119:0/3732674745 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9b380043a0 con 0x7f9b300043c0 2024-09-13T16:32:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.351+0000 7f9b367fc700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b28066730 0x7f9b28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.351+0000 7f9b367fc700 1 -- 172.21.15.119:0/3732674745 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f9b38084680 con 0x7f9b300043c0 2024-09-13T16:32:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.351+0000 7f9b4930a700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b28066730 0x7f9b28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.351+0000 7f9b4930a700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b28066730 0x7f9b28068bf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9b40005040 tx=0x7f9b40009030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.359+0000 7f9b367fc700 1 -- 172.21.15.119:0/3732674745 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9b3c03a0c0 con 0x7f9b300043c0 2024-09-13T16:32:52.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.531+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b3c03a0c0 con 0x7f9b300043c0 2024-09-13T16:32:52.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.535+0000 7f9b367fc700 1 -- 172.21.15.119:0/3732674745 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17101 (secure 0 0 0) 0x7f9b3c03a0c0 con 0x7f9b300043c0 2024-09-13T16:32:52.541 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:32:52.542 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":98,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:32:09.134785+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1e","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3011171408":"2024-09-14T16:31:41.722614+0000","172.21.15.145:6834/3011171408":"2024-09-14T16:31:41.722614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:32:52.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b28066730 msgr2=0x7f9b28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b28066730 0x7f9b28068bf0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f9b40005040 tx=0x7f9b40009030 comp rx=0 tx=0).stop 2024-09-13T16:32:52.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 msgr2=0x7f9b3c0d2fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b3c0d2fa0 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f9b380115c0 tx=0x7f9b380118d0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 shutdown_connections 2024-09-13T16:32:52.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b28066730 0x7f9b28068bf0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9b3c0d7ad0 0x7f9b3c0e0150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9b3c0d34e0 0x7f9b3c0dfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 --2- 172.21.15.119:0/3732674745 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9b300043c0 0x7f9b3c0d2fa0 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 >> 172.21.15.119:0/3732674745 conn(0x7f9b3c036140 msgr2=0x7f9b3c031240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 shutdown_connections 2024-09-13T16:32:52.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.539+0000 7f9b4ab0d700 1 -- 172.21.15.119:0/3732674745 wait complete. 2024-09-13T16:32:52.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:31:27.299109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:31:28.044326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:32:52.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:32:52.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.671+0000 7f3f94cdf700 1 Processor -- start 2024-09-13T16:32:52.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.671+0000 7f3f94cdf700 1 -- start start 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.671+0000 7f3f94cdf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3f9006ca60 0x7f3f90159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.671+0000 7f3f94cdf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3f900b1860 0x7f3f9015a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.671+0000 7f3f94cdf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f900b1dd0 con 0x7f3f900ad9a0 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.671+0000 7f3f94cdf700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f900b2fd0 con 0x7f3f9006ca60 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.671+0000 7f3f94cdf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f9015a520 con 0x7f3f900b1860 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3f9006ca60 0x7f3f90159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3f9006ca60 0x7f3f90159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50414/0 (socket says 172.21.15.119:50414) 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 -- 172.21.15.119:0/1556478536 learned_addr learned my addr 172.21.15.119:0/1556478536 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f8ffff700 1 --2- 172.21.15.119:0/1556478536 >> v2:172.21.15.145:3300/0 conn(0x7f3f900b1860 0x7f3f9015a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 -- 172.21.15.119:0/1556478536 >> v2:172.21.15.145:3300/0 conn(0x7f3f900b1860 msgr2=0x7f3f9015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 --2- 172.21.15.119:0/1556478536 >> v2:172.21.15.145:3300/0 conn(0x7f3f900b1860 0x7f3f9015a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 -- 172.21.15.119:0/1556478536 >> v1:172.21.15.119:6789/0 conn(0x7f3f900ad9a0 legacy=0x7f3f9006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 -- 172.21.15.119:0/1556478536 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f900b20b0 con 0x7f3f9006ca60 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f86ffd700 1 --2- 172.21.15.119:0/1556478536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f90159de0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3f90159360 tx=0x7f3f78009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4aaa12ae9ced0e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f8effd700 1 -- 172.21.15.119:0/1556478536 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800b8a0 con 0x7f3f9006ca60 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f8effd700 1 -- 172.21.15.119:0/1556478536 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7800ba00 con 0x7f3f9006ca60 2024-09-13T16:32:52.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f8effd700 1 -- 172.21.15.119:0/1556478536 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f7800bd90 con 0x7f3f9006ca60 2024-09-13T16:32:52.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 -- 172.21.15.119:0/1556478536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 msgr2=0x7f3f90159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/1556478536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f90159de0 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7f3f90159360 tx=0x7f3f78009bf0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 -- 172.21.15.119:0/1556478536 shutdown_connections 2024-09-13T16:32:52.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/1556478536 >> v2:172.21.15.145:3300/0 conn(0x7f3f900b1860 0x7f3f9015a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/1556478536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f90159de0 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 -- 172.21.15.119:0/1556478536 >> 172.21.15.119:0/1556478536 conn(0x7f3f900b0980 msgr2=0x7f3f900b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 -- 172.21.15.119:0/1556478536 shutdown_connections 2024-09-13T16:32:52.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.675+0000 7f3f94cdf700 1 -- 172.21.15.119:0/1556478536 wait complete. 2024-09-13T16:32:52.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 Processor -- start 2024-09-13T16:32:52.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 -- start start 2024-09-13T16:32:52.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f9015cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f900ad9a0 0x7f3f90146f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f900b1860 0x7f3f901474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9014ad90 con 0x7f3f9006ca60 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f9014a900 con 0x7f3f900b1860 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f90149cf0 con 0x7f3f900ad9a0 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f9015cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f9015cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50416/0 (socket says 172.21.15.119:50416) 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 -- 172.21.15.119:0/4141053501 learned_addr learned my addr 172.21.15.119:0/4141053501 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f86ffd700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f900ad9a0 0x7f3f90146f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8ffff700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f900b1860 0x7f3f901474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 -- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f900ad9a0 msgr2=0x7f3f90146f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f900ad9a0 0x7f3f90146f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 -- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f900b1860 msgr2=0x7f3f901474d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f900b1860 0x7f3f901474d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 -- 172.21.15.119:0/4141053501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f78009570 con 0x7f3f9006ca60 2024-09-13T16:32:52.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8f7fe700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f9015cc00 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000c500 tx=0x7f3f8000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8cff9700 1 -- 172.21.15.119:0/4141053501 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f800052d0 con 0x7f3f9006ca60 2024-09-13T16:32:52.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8cff9700 1 -- 172.21.15.119:0/4141053501 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f80005430 con 0x7f3f9006ca60 2024-09-13T16:32:52.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f90149fd0 con 0x7f3f9006ca60 2024-09-13T16:32:52.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f8cff9700 1 -- 172.21.15.119:0/4141053501 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f800055a0 con 0x7f3f9006ca60 2024-09-13T16:32:52.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.679+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f9014a590 con 0x7f3f9006ca60 2024-09-13T16:32:52.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.683+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f90159de0 con 0x7f3f9006ca60 2024-09-13T16:32:52.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.683+0000 7f3f8cff9700 1 -- 172.21.15.119:0/4141053501 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3f800036a0 con 0x7f3f9006ca60 2024-09-13T16:32:52.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.683+0000 7f3f8cff9700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f70066730 0x7f3f70068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:52.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.687+0000 7f3f8cff9700 1 -- 172.21.15.119:0/4141053501 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f3f800834d0 con 0x7f3f9006ca60 2024-09-13T16:32:52.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.687+0000 7f3f86ffd700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f70066730 0x7f3f70068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:52.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.687+0000 7f3f86ffd700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f70066730 0x7f3f70068bf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3f78009860 tx=0x7f3f78008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:52.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.691+0000 7f3f8cff9700 1 -- 172.21.15.119:0/4141053501 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3f800506c0 con 0x7f3f9006ca60 2024-09-13T16:32:52.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.859+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3f9006bbf0 con 0x7f3f9006ca60 2024-09-13T16:32:52.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.859+0000 7f3f8cff9700 1 -- 172.21.15.119:0/4141053501 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3f80053ce0 con 0x7f3f9006ca60 2024-09-13T16:32:52.867 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:32:52.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f70066730 msgr2=0x7f3f70068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f70066730 0x7f3f70068bf0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3f78009860 tx=0x7f3f78008030 comp rx=0 tx=0).stop 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 msgr2=0x7f3f9015cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f9015cc00 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 crypto rx=0x7f3f8000c500 tx=0x7f3f8000c810 comp rx=0 tx=0).stop 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 shutdown_connections 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3f70066730 0x7f3f70068bf0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3f900b1860 0x7f3f901474d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3f900ad9a0 0x7f3f90146f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 --2- 172.21.15.119:0/4141053501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3f9006ca60 0x7f3f9015cc00 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 >> 172.21.15.119:0/4141053501 conn(0x7f3f900b0980 msgr2=0x7f3f90155660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 shutdown_connections 2024-09-13T16:32:52.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.863+0000 7f3f94cdf700 1 -- 172.21.15.119:0/4141053501 wait complete. 2024-09-13T16:32:52.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:32:53.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc17d0700 1 Processor -- start 2024-09-13T16:32:53.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc17d0700 1 -- start start 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc17d0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 0x7fafb400be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc17d0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fafb4054b90 0x7fafb4055020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc17d0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40516c0 con 0x7fafb40e0b00 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc17d0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40545d0 con 0x7fafb4054b90 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc17d0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40548b0 con 0x7fafb4035950 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbbfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fafb40e0b00 0x7fafb4056f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49834/0 (socket says 172.21.15.119:49834) 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbbfff700 1 -- 172.21.15.119:0/4229101263 learned_addr learned my addr 172.21.15.119:0/4229101263 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbb7fe700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 0x7fafb400be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafc0fcf700 1 --2- 172.21.15.119:0/4229101263 >> v2:172.21.15.145:3300/0 conn(0x7fafb4054b90 0x7fafb4055020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3251377970 0 0) 0x7fafb40516c0 con 0x7fafb40e0b00 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafac0035d0 con 0x7fafb40e0b00 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2775322963 0 0) 0x7fafac0035d0 con 0x7fafb40e0b00 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 msgr2=0x7fafb400be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:52.999+0000 7fafbaffd700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 0x7fafb400be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 >> v2:172.21.15.145:3300/0 conn(0x7fafb4054b90 msgr2=0x7fafb4055020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 --2- 172.21.15.119:0/4229101263 >> v2:172.21.15.145:3300/0 conn(0x7fafb4054b90 0x7fafb4055020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb4055560 con 0x7fafb40e0b00 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fafb00017f0 con 0x7fafb40e0b00 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 >> v1:172.21.15.119:6789/0 conn(0x7fafb40e0b00 legacy=0x7fafb4056f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafac003a20 0x7fafac003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 0x7fafac004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafac004db0 0x7fafac005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40545d0 con 0x7fafac004db0 2024-09-13T16:32:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb40548b0 con 0x7fafac003a20 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafac00ba30 con 0x7fafb4035950 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafac004db0 0x7fafac005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbbfff700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafac003a20 0x7fafac003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbb7fe700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 0x7fafac004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 -- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 msgr2=0x7fafac004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb4035950 0x7fafac004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 -- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafac003a20 msgr2=0x7fafac003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafac003a20 0x7fafac003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 -- 172.21.15.119:0/4229101263 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafac0035d0 con 0x7fafac004db0 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafac004db0 0x7fafac005230 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fafbc0641b0 tx=0x7fafac0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4332bef864656736 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc08f040 con 0x7fafac004db0 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafbc067600 con 0x7fafac004db0 2024-09-13T16:32:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafbaffd700 1 -- 172.21.15.119:0/4229101263 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafbc067b90 con 0x7fafac004db0 2024-09-13T16:32:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc17d0700 1 -- 172.21.15.119:0/4229101263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafac004db0 msgr2=0x7fafac005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc17d0700 1 --2- 172.21.15.119:0/4229101263 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafac004db0 0x7fafac005230 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7fafbc0641b0 tx=0x7fafac0058a0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc0fcf700 1 -- 172.21.15.119:0/4229101263 reap_dead start 2024-09-13T16:32:53.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc17d0700 1 -- 172.21.15.119:0/4229101263 shutdown_connections 2024-09-13T16:32:53.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc17d0700 1 -- 172.21.15.119:0/4229101263 >> 172.21.15.119:0/4229101263 conn(0x7fafb401fbf0 msgr2=0x7fafb4056860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:53.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc17d0700 1 -- 172.21.15.119:0/4229101263 shutdown_connections 2024-09-13T16:32:53.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.003+0000 7fafc17d0700 1 -- 172.21.15.119:0/4229101263 wait complete. 2024-09-13T16:32:53.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 Processor -- start 2024-09-13T16:32:53.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 -- start start 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb40cc7c0 0x7fafb40ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb40cd0e0 0x7fafb40da5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb40cd560 0x7fafb40dab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb417a5f0 con 0x7fafb40cd0e0 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4179970 con 0x7fafb40cd560 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc17d0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4179c50 con 0x7fafb40cc7c0 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafbb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb40cd0e0 0x7fafb40da5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafbb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb40cd0e0 0x7fafb40da5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50428/0 (socket says 172.21.15.119:50428) 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafbb7fe700 1 -- 172.21.15.119:0/1816905270 learned_addr learned my addr 172.21.15.119:0/1816905270 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafbbfff700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb40cc7c0 0x7fafb40ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.007+0000 7fafc0fcf700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb40cd560 0x7fafb40dab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafbb7fe700 1 -- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb40cc7c0 msgr2=0x7fafb40ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafbb7fe700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb40cc7c0 0x7fafb40ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafbb7fe700 1 -- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb40cd560 msgr2=0x7fafb40dab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafbb7fe700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb40cd560 0x7fafb40dab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafbb7fe700 1 -- 172.21.15.119:0/1816905270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40557b0 con 0x7fafb40cd0e0 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafbb7fe700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb40cd0e0 0x7fafb40da5e0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fafa8011a20 tx=0x7fafa8011d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafb8ff9700 1 -- 172.21.15.119:0/1816905270 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8011ec0 con 0x7fafb40cd0e0 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafb8ff9700 1 -- 172.21.15.119:0/1816905270 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa800f940 con 0x7fafb40cd0e0 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb4179f30 con 0x7fafb40cd0e0 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafb8ff9700 1 -- 172.21.15.119:0/1816905270 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa800fcd0 con 0x7fafb40cd0e0 2024-09-13T16:32:53.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb40db060 con 0x7fafb40cd0e0 2024-09-13T16:32:53.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb4055020 con 0x7fafb40cd0e0 2024-09-13T16:32:53.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafb8ff9700 1 -- 172.21.15.119:0/1816905270 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fafa8014440 con 0x7fafb40cd0e0 2024-09-13T16:32:53.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafb8ff9700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafa0066730 0x7fafa0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafb8ff9700 1 -- 172.21.15.119:0/1816905270 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fafa80849c0 con 0x7fafb40cd0e0 2024-09-13T16:32:53.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.011+0000 7fafbbfff700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafa0066730 0x7fafa0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.015+0000 7fafbbfff700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafa0066730 0x7fafa0068bf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fafb0001e40 tx=0x7fafb0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.019+0000 7fafb8ff9700 1 -- 172.21.15.119:0/1816905270 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fafa8051c70 con 0x7fafb40cd0e0 2024-09-13T16:32:53.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.195+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fafb4033580 con 0x7fafb40cd0e0 2024-09-13T16:32:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.195+0000 7fafb8ff9700 1 -- 172.21.15.119:0/1816905270 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7fafa8055290 con 0x7fafb40cd0e0 2024-09-13T16:32:53.202 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:32:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.199+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafa0066730 msgr2=0x7fafa0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.199+0000 7fafc17d0700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafa0066730 0x7fafa0068bf0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7fafb0001e40 tx=0x7fafb0009040 comp rx=0 tx=0).stop 2024-09-13T16:32:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.199+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb40cd0e0 msgr2=0x7fafb40da5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.199+0000 7fafc17d0700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb40cd0e0 0x7fafb40da5e0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 crypto rx=0x7fafa8011a20 tx=0x7fafa8011d30 comp rx=0 tx=0).stop 2024-09-13T16:32:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 shutdown_connections 2024-09-13T16:32:53.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafa0066730 0x7fafa0068bf0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fafb40cd560 0x7fafb40dab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fafb40cd0e0 0x7fafb40da5e0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 --2- 172.21.15.119:0/1816905270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fafb40cc7c0 0x7fafb40ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 >> 172.21.15.119:0/1816905270 conn(0x7fafb401fbf0 msgr2=0x7fafb400c3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 shutdown_connections 2024-09-13T16:32:53.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.203+0000 7fafc17d0700 1 -- 172.21.15.119:0/1816905270 wait complete. 2024-09-13T16:32:53.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:32:53.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.335+0000 7f0247669700 1 Processor -- start 2024-09-13T16:32:53.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.335+0000 7f0247669700 1 -- start start 2024-09-13T16:32:53.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f024015aa50 0x7f024015bb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f024015c120 0x7f02400b39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f024015ae30 con 0x7f02400b4400 2024-09-13T16:32:53.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f024015c580 con 0x7f024015aa50 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02400b3f30 con 0x7f024015c120 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f024015aa50 0x7f024015bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f024015aa50 0x7f024015bb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50432/0 (socket says 172.21.15.119:50432) 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 -- 172.21.15.119:0/3740331833 learned_addr learned my addr 172.21.15.119:0/3740331833 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0244c04700 1 --2- 172.21.15.119:0/3740331833 >> v2:172.21.15.145:3300/0 conn(0x7f024015c120 0x7f02400b39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 -- 172.21.15.119:0/3740331833 >> v2:172.21.15.145:3300/0 conn(0x7f024015c120 msgr2=0x7f02400b39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 --2- 172.21.15.119:0/3740331833 >> v2:172.21.15.145:3300/0 conn(0x7f024015c120 0x7f02400b39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 -- 172.21.15.119:0/3740331833 >> v1:172.21.15.145:6789/0 conn(0x7f02400b4400 legacy=0x7f02400af240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 -- 172.21.15.119:0/3740331833 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02400ae480 con 0x7f024015aa50 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0244c04700 1 --2- 172.21.15.119:0/3740331833 >> v2:172.21.15.145:3300/0 conn(0x7f024015c120 0x7f02400b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:32:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0245405700 1 --2- 172.21.15.119:0/3740331833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015aa50 0x7f024015bb70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f02400b5ec0 tx=0x7f0230009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e95d032e8d877873 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0237fff700 1 -- 172.21.15.119:0/3740331833 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230011830 con 0x7f024015aa50 2024-09-13T16:32:53.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0237fff700 1 -- 172.21.15.119:0/3740331833 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0230011990 con 0x7f024015aa50 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0237fff700 1 -- 172.21.15.119:0/3740331833 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0230011d20 con 0x7f024015aa50 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- 172.21.15.119:0/3740331833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015aa50 msgr2=0x7f024015bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 --2- 172.21.15.119:0/3740331833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015aa50 0x7f024015bb70 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7f02400b5ec0 tx=0x7f0230009bf0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- 172.21.15.119:0/3740331833 shutdown_connections 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 --2- 172.21.15.119:0/3740331833 >> v2:172.21.15.145:3300/0 conn(0x7f024015c120 0x7f02400b39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 --2- 172.21.15.119:0/3740331833 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015aa50 0x7f024015bb70 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- 172.21.15.119:0/3740331833 >> 172.21.15.119:0/3740331833 conn(0x7f0240099a50 msgr2=0x7f0240099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- 172.21.15.119:0/3740331833 shutdown_connections 2024-09-13T16:32:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.339+0000 7f0247669700 1 -- 172.21.15.119:0/3740331833 wait complete. 2024-09-13T16:32:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.343+0000 7f0247669700 1 Processor -- start 2024-09-13T16:32:53.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.343+0000 7f0247669700 1 -- start start 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02400b4400 0x7f02401efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f024015aa50 0x7f024014ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015c120 0x7f02400b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02401f0c50 con 0x7f024015c120 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024014f970 con 0x7f02400b4400 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02401f0590 con 0x7f024015aa50 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015c120 0x7f02400b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015c120 0x7f02400b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50444/0 (socket says 172.21.15.119:50444) 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 -- 172.21.15.119:0/3127218060 learned_addr learned my addr 172.21.15.119:0/3127218060 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0244c04700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f024015aa50 0x7f024014ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245405700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02400b4400 0x7f02401efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 -- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f024015aa50 msgr2=0x7f024014ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f024015aa50 0x7f024014ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 -- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02400b4400 msgr2=0x7f02401efcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02400b4400 0x7f02401efcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 -- 172.21.15.119:0/3127218060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f023c0094b0 con 0x7f024015c120 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0245c06700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015c120 0x7f02400b1410 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f023800cc50 tx=0x7f023800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0235ffb700 1 -- 172.21.15.119:0/3127218060 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0238003c20 con 0x7f024015c120 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0235ffb700 1 -- 172.21.15.119:0/3127218060 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0238003d80 con 0x7f024015c120 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0235ffb700 1 -- 172.21.15.119:0/3127218060 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f023800aab0 con 0x7f024015c120 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0230009570 con 0x7f024015c120 2024-09-13T16:32:53.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02401f0200 con 0x7f024015c120 2024-09-13T16:32:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.347+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02401f0d90 con 0x7f024015c120 2024-09-13T16:32:53.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.351+0000 7f0235ffb700 1 -- 172.21.15.119:0/3127218060 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f023800ac10 con 0x7f024015c120 2024-09-13T16:32:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.351+0000 7f0235ffb700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0224066730 0x7f0224068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.351+0000 7f0235ffb700 1 -- 172.21.15.119:0/3127218060 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f0238084100 con 0x7f024015c120 2024-09-13T16:32:53.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.351+0000 7f0245405700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0224066730 0x7f0224068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.351+0000 7f0245405700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0224066730 0x7f0224068bf0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0230009c30 tx=0x7f0230008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.355+0000 7f0235ffb700 1 -- 172.21.15.119:0/3127218060 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f02401f0d90 con 0x7f024015c120 2024-09-13T16:32:53.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.523+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f024015ae30 con 0x7f024015c120 2024-09-13T16:32:53.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.523+0000 7f0235ffb700 1 -- 172.21.15.119:0/3127218060 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f024015ae30 con 0x7f024015c120 2024-09-13T16:32:53.530 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0224066730 msgr2=0x7f0224068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0224066730 0x7f0224068bf0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f0230009c30 tx=0x7f0230008030 comp rx=0 tx=0).stop 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015c120 msgr2=0x7f02400b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015c120 0x7f02400b1410 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7f023800cc50 tx=0x7f023800cf60 comp rx=0 tx=0).stop 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 shutdown_connections 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0224066730 0x7f0224068bf0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f024015c120 0x7f02400b1410 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f024015aa50 0x7f024014ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 --2- 172.21.15.119:0/3127218060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02400b4400 0x7f02401efcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 >> 172.21.15.119:0/3127218060 conn(0x7f0240099a50 msgr2=0x7f024006eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:53.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 shutdown_connections 2024-09-13T16:32:53.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.527+0000 7f0247669700 1 -- 172.21.15.119:0/3127218060 wait complete. 2024-09-13T16:32:53.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:32:53.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.651+0000 7f9073fff700 1 Processor -- start 2024-09-13T16:32:53.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.651+0000 7f9073fff700 1 -- start start 2024-09-13T16:32:53.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f907415da80 0x7f90741408f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9074140e30 0x7f907406a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f907415d230 con 0x7f907406a890 2024-09-13T16:32:53.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90740acad0 con 0x7f907415da80 2024-09-13T16:32:53.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9074141270 con 0x7f9074140e30 2024-09-13T16:32:53.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f90737fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f907406a890 0x7f90740ce9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49842/0 (socket says 172.21.15.119:49842) 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f90737fe700 1 -- 172.21.15.119:0/3006292453 learned_addr learned my addr 172.21.15.119:0/3006292453 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9072ffd700 1 --2- 172.21.15.119:0/3006292453 >> v2:172.21.15.119:3300/0 conn(0x7f907415da80 0x7f90741408f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f906bfff700 1 --2- 172.21.15.119:0/3006292453 >> v2:172.21.15.145:3300/0 conn(0x7f9074140e30 0x7f907406a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f90727fc700 1 -- 172.21.15.119:0/3006292453 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402729045 0 0) 0x7f907415d230 con 0x7f907406a890 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f90727fc700 1 -- 172.21.15.119:0/3006292453 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90540035d0 con 0x7f907406a890 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9072ffd700 1 -- 172.21.15.119:0/3006292453 >> v2:172.21.15.145:3300/0 conn(0x7f9074140e30 msgr2=0x7f907406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9072ffd700 1 --2- 172.21.15.119:0/3006292453 >> v2:172.21.15.145:3300/0 conn(0x7f9074140e30 0x7f907406a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9072ffd700 1 -- 172.21.15.119:0/3006292453 >> v1:172.21.15.119:6789/0 conn(0x7f907406a890 legacy=0x7f90740ce9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9072ffd700 1 -- 172.21.15.119:0/3006292453 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907415de60 con 0x7f907415da80 2024-09-13T16:32:53.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9072ffd700 1 --2- 172.21.15.119:0/3006292453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f907415da80 0x7f90741408f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f907406cc70 tx=0x7f9064009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=244de036d83f5bca server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f90727fc700 1 -- 172.21.15.119:0/3006292453 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064011830 con 0x7f907415da80 2024-09-13T16:32:53.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f90727fc700 1 -- 172.21.15.119:0/3006292453 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9064011990 con 0x7f907415da80 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f90727fc700 1 -- 172.21.15.119:0/3006292453 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9064011d20 con 0x7f907415da80 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- 172.21.15.119:0/3006292453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f907415da80 msgr2=0x7f90741408f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 --2- 172.21.15.119:0/3006292453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f907415da80 0x7f90741408f0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 crypto rx=0x7f907406cc70 tx=0x7f9064009b30 comp rx=0 tx=0).stop 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- 172.21.15.119:0/3006292453 shutdown_connections 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 --2- 172.21.15.119:0/3006292453 >> v2:172.21.15.145:3300/0 conn(0x7f9074140e30 0x7f907406a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 --2- 172.21.15.119:0/3006292453 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f907415da80 0x7f90741408f0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- 172.21.15.119:0/3006292453 >> 172.21.15.119:0/3006292453 conn(0x7f907406b460 msgr2=0x7f907406b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- 172.21.15.119:0/3006292453 shutdown_connections 2024-09-13T16:32:53.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.655+0000 7f9073fff700 1 -- 172.21.15.119:0/3006292453 wait complete. 2024-09-13T16:32:53.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.659+0000 7f9073fff700 1 Processor -- start 2024-09-13T16:32:53.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.659+0000 7f9073fff700 1 -- start start 2024-09-13T16:32:53.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.659+0000 7f9073fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f907406a890 0x7f907413ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9074140e30 0x7f90740b3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f907415da80 0x7f90740b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90740cea80 con 0x7f9074140e30 2024-09-13T16:32:53.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90741f20a0 con 0x7f907415da80 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90741f2360 con 0x7f907406a890 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9074140e30 0x7f90740b3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9074140e30 0x7f90740b3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50460/0 (socket says 172.21.15.119:50460) 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9072ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f907406a890 0x7f907413ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 -- 172.21.15.119:0/1243655577 learned_addr learned my addr 172.21.15.119:0/1243655577 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f90737fe700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f907415da80 0x7f90740b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 -- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f907406a890 msgr2=0x7f907413ef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f907406a890 0x7f907413ef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 -- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f907415da80 msgr2=0x7f90740b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f907415da80 0x7f90740b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 -- 172.21.15.119:0/1243655577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90640094b0 con 0x7f9074140e30 2024-09-13T16:32:53.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906bfff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9074140e30 0x7f90740b3c90 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e9d0 tx=0x7f905c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906b7fe700 1 -- 172.21.15.119:0/1243655577 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c009720 con 0x7f9074140e30 2024-09-13T16:32:53.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906b7fe700 1 -- 172.21.15.119:0/1243655577 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f905c00c850 con 0x7f9074140e30 2024-09-13T16:32:53.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906b7fe700 1 -- 172.21.15.119:0/1243655577 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905c00cbc0 con 0x7f9074140e30 2024-09-13T16:32:53.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90741f2640 con 0x7f9074140e30 2024-09-13T16:32:53.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90741f2c00 con 0x7f9074140e30 2024-09-13T16:32:53.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90740677a0 con 0x7f9074140e30 2024-09-13T16:32:53.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.663+0000 7f906b7fe700 1 -- 172.21.15.119:0/1243655577 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f905c00cd20 con 0x7f9074140e30 2024-09-13T16:32:53.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.667+0000 7f906b7fe700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9058066730 0x7f9058068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:53.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.667+0000 7f906b7fe700 1 -- 172.21.15.119:0/1243655577 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f905c0843a0 con 0x7f9074140e30 2024-09-13T16:32:53.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.667+0000 7f9072ffd700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9058066730 0x7f9058068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:53.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.667+0000 7f9072ffd700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9058066730 0x7f9058068bf0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9064000c00 tx=0x7f9064008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:53.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.671+0000 7f906b7fe700 1 -- 172.21.15.119:0/1243655577 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f905c051590 con 0x7f9074140e30 2024-09-13T16:32:53.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.855+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f90741f2f40 con 0x7f9074140e30 2024-09-13T16:32:53.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.855+0000 7f906b7fe700 1 -- 172.21.15.119:0/1243655577 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v98) v1 ==== 80+0+124 (secure 0 0 0) 0x7f905c054bb0 con 0x7f9074140e30 2024-09-13T16:32:53.860 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6835/3011171408 2024-09-14T16:31:41.722614+0000 2024-09-13T16:32:53.860 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6834/3011171408 2024-09-14T16:31:41.722614+0000 2024-09-13T16:32:53.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9058066730 msgr2=0x7f9058068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9058066730 0x7f9058068bf0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f9064000c00 tx=0x7f9064008030 comp rx=0 tx=0).stop 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9074140e30 msgr2=0x7f90740b3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9074140e30 0x7f90740b3c90 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7f905c00e9d0 tx=0x7f905c00ece0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 shutdown_connections 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9058066730 0x7f9058068bf0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f907415da80 0x7f90740b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9074140e30 0x7f90740b3c90 unknown :-1 s=CLOSED pgs=1395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 --2- 172.21.15.119:0/1243655577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f907406a890 0x7f907413ef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 >> 172.21.15.119:0/1243655577 conn(0x7f907406b460 msgr2=0x7f90740af5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 shutdown_connections 2024-09-13T16:32:53.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.859+0000 7f9073fff700 1 -- 172.21.15.119:0/1243655577 wait complete. 2024-09-13T16:32:53.866 INFO:teuthology.orchestra.run.smithi119.stderr:listed 2 entries 2024-09-13T16:32:53.879 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:32:53.879 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:32:53.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:32:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 Processor -- start 2024-09-13T16:32:54.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 -- start start 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ff40cd8d0 0x7f9ff406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9ff40b17f0 0x7f9ff40cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ff40705d0 con 0x7f9ff40b2200 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff406d0d0 con 0x7f9ff40cd8d0 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ff406fca0 con 0x7f9ff40b17f0 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ff40cd8d0 0x7f9ff406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ff40cd8d0 0x7f9ff406cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50474/0 (socket says 172.21.15.119:50474) 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 -- 172.21.15.119:0/2822372635 learned_addr learned my addr 172.21.15.119:0/2822372635 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:54.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff27fc700 1 --2- 172.21.15.119:0/2822372635 >> v2:172.21.15.145:3300/0 conn(0x7f9ff40b17f0 0x7f9ff40cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 -- 172.21.15.119:0/2822372635 >> v2:172.21.15.145:3300/0 conn(0x7f9ff40b17f0 msgr2=0x7f9ff40cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 --2- 172.21.15.119:0/2822372635 >> v2:172.21.15.145:3300/0 conn(0x7f9ff40b17f0 0x7f9ff40cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 -- 172.21.15.119:0/2822372635 >> v1:172.21.15.145:6789/0 conn(0x7f9ff40b2200 legacy=0x7f9ff4070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 -- 172.21.15.119:0/2822372635 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ff406ff80 con 0x7f9ff40cd8d0 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff27fc700 1 --2- 172.21.15.119:0/2822372635 >> v2:172.21.15.145:3300/0 conn(0x7f9ff40b17f0 0x7f9ff40cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff2ffd700 1 --2- 172.21.15.119:0/2822372635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff406cb90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40ae2c0 tx=0x7f9fec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7e5577924119182 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff1ffb700 1 -- 172.21.15.119:0/2822372635 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00e070 con 0x7f9ff40cd8d0 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff1ffb700 1 -- 172.21.15.119:0/2822372635 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fec00b9c0 con 0x7f9ff40cd8d0 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff1ffb700 1 -- 172.21.15.119:0/2822372635 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fec00bd70 con 0x7f9ff40cd8d0 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 -- 172.21.15.119:0/2822372635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40cd8d0 msgr2=0x7f9ff406cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/2822372635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff406cb90 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7f9ff40ae2c0 tx=0x7f9fec00b330 comp rx=0 tx=0).stop 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 -- 172.21.15.119:0/2822372635 shutdown_connections 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/2822372635 >> v2:172.21.15.145:3300/0 conn(0x7f9ff40b17f0 0x7f9ff40cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/2822372635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff406cb90 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.995+0000 7f9ff3fff700 1 -- 172.21.15.119:0/2822372635 >> 172.21.15.119:0/2822372635 conn(0x7f9ff406a350 msgr2=0x7f9ff406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- 172.21.15.119:0/2822372635 shutdown_connections 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- 172.21.15.119:0/2822372635 wait complete. 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 Processor -- start 2024-09-13T16:32:54.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- start start 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ff40b17f0 0x7f9ff4159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40b2200 0x7f9ff40aca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff40a9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff4159b70 con 0x7f9ff40b2200 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40a93a0 con 0x7f9ff40cd8d0 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ff40a84b0 con 0x7f9ff40b17f0 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40b2200 0x7f9ff40aca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40b2200 0x7f9ff40aca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50484/0 (socket says 172.21.15.119:50484) 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 -- 172.21.15.119:0/383921415 learned_addr learned my addr 172.21.15.119:0/383921415 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff37fe700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff40a9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff2ffd700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ff40b17f0 0x7f9ff4159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 -- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ff40b17f0 msgr2=0x7f9ff4159630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ff40b17f0 0x7f9ff4159630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 -- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ff40cd8d0 msgr2=0x7f9ff40a9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff40a9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 -- 172.21.15.119:0/383921415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec009b30 con 0x7f9ff40b2200 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff2ffd700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ff40b17f0 0x7f9ff4159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff37fe700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff40a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff27fc700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40b2200 0x7f9ff40aca40 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e880 tx=0x7f9fdc00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9febfff700 1 -- 172.21.15.119:0/383921415 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00c780 con 0x7f9ff40b2200 2024-09-13T16:32:54.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9febfff700 1 -- 172.21.15.119:0/383921415 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fdc00c8e0 con 0x7f9ff40b2200 2024-09-13T16:32:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ff40a8790 con 0x7f9ff40b2200 2024-09-13T16:32:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9febfff700 1 -- 172.21.15.119:0/383921415 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9fdc00cc70 con 0x7f9ff40b2200 2024-09-13T16:32:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:53.999+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ff40a8d50 con 0x7f9ff40b2200 2024-09-13T16:32:54.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.003+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ff40cdcb0 con 0x7f9ff40b2200 2024-09-13T16:32:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.003+0000 7f9febfff700 1 -- 172.21.15.119:0/383921415 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9fdc013410 con 0x7f9ff40b2200 2024-09-13T16:32:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.003+0000 7f9febfff700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fd4066730 0x7f9fd4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.003+0000 7f9febfff700 1 -- 172.21.15.119:0/383921415 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f9fdc084020 con 0x7f9ff40b2200 2024-09-13T16:32:54.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.003+0000 7f9ff2ffd700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fd4066730 0x7f9fd4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.003+0000 7f9ff2ffd700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fd4066730 0x7f9fd4068bf0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00b370 tx=0x7f9fec009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.007+0000 7f9febfff700 1 -- 172.21.15.119:0/383921415 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9fdc051180 con 0x7f9ff40b2200 2024-09-13T16:32:54.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.179+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ff40ae2f0 con 0x7f9ff40b2200 2024-09-13T16:32:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.183+0000 7f9febfff700 1 -- 172.21.15.119:0/383921415 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17101 (secure 0 0 0) 0x7f9fdc0547a0 con 0x7f9ff40b2200 2024-09-13T16:32:54.189 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:32:54.189 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":98,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:32:09.134785+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1e","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3011171408":"2024-09-14T16:31:41.722614+0000","172.21.15.145:6834/3011171408":"2024-09-14T16:31:41.722614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:32:54.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fd4066730 msgr2=0x7f9fd4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fd4066730 0x7f9fd4068bf0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9fec00b370 tx=0x7f9fec009000 comp rx=0 tx=0).stop 2024-09-13T16:32:54.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40b2200 msgr2=0x7f9ff40aca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40b2200 0x7f9ff40aca40 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 crypto rx=0x7f9fdc00e880 tx=0x7f9fdc00ec40 comp rx=0 tx=0).stop 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 shutdown_connections 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9fd4066730 0x7f9fd4068bf0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ff40cd8d0 0x7f9ff40a9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ff40b2200 0x7f9ff40aca40 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 --2- 172.21.15.119:0/383921415 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ff40b17f0 0x7f9ff4159630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 >> 172.21.15.119:0/383921415 conn(0x7f9ff406a350 msgr2=0x7f9ff4161310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 shutdown_connections 2024-09-13T16:32:54.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.187+0000 7f9ff3fff700 1 -- 172.21.15.119:0/383921415 wait complete. 2024-09-13T16:32:54.204 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:31:27.299109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:31:28.044326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:32:54.204 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:32:54.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d897f700 1 Processor -- start 2024-09-13T16:32:54.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d897f700 1 -- start start 2024-09-13T16:32:54.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d897f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40ce8c0 0x7f25d40d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d897f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f25d40d1610 0x7f25d40b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d897f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25d40d0df0 con 0x7f25d415cc00 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d897f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25d40d1a80 con 0x7f25d40d1610 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d897f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d40cdb00 con 0x7f25d40ce8c0 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f25d40d1610 0x7f25d40b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d37fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f25d415cc00 0x7f25d40d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49846/0 (socket says 172.21.15.119:49846) 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d37fe700 1 -- 172.21.15.119:0/1173862988 learned_addr learned my addr 172.21.15.119:0/1173862988 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:32:54.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.319+0000 7f25d2ffd700 1 --2- 172.21.15.119:0/1173862988 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40ce8c0 0x7f25d40d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d3fff700 1 -- 172.21.15.119:0/1173862988 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40ce8c0 msgr2=0x7f25d40d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d3fff700 1 --2- 172.21.15.119:0/1173862988 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40ce8c0 0x7f25d40d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d3fff700 1 -- 172.21.15.119:0/1173862988 >> v1:172.21.15.119:6789/0 conn(0x7f25d415cc00 legacy=0x7f25d40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d27fc700 1 -- 172.21.15.119:0/1173862988 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551924760 0 0) 0x7f25d40d0df0 con 0x7f25d415cc00 2024-09-13T16:32:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d3fff700 1 -- 172.21.15.119:0/1173862988 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25d40cdde0 con 0x7f25d40d1610 2024-09-13T16:32:54.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d3fff700 1 --2- 172.21.15.119:0/1173862988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d40d1610 0x7f25d40b5960 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009d90 tx=0x7f25c400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b90fa8d581eed93b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d27fc700 1 -- 172.21.15.119:0/1173862988 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c400e070 con 0x7f25d40d1610 2024-09-13T16:32:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d27fc700 1 -- 172.21.15.119:0/1173862988 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c400bb20 con 0x7f25d40d1610 2024-09-13T16:32:54.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d27fc700 1 -- 172.21.15.119:0/1173862988 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4012430 con 0x7f25d40d1610 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 -- 172.21.15.119:0/1173862988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d40d1610 msgr2=0x7f25d40b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 --2- 172.21.15.119:0/1173862988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d40d1610 0x7f25d40b5960 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f25c4009d90 tx=0x7f25c400b330 comp rx=0 tx=0).stop 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 -- 172.21.15.119:0/1173862988 shutdown_connections 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 --2- 172.21.15.119:0/1173862988 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d40d1610 0x7f25d40b5960 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 --2- 172.21.15.119:0/1173862988 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40ce8c0 0x7f25d40d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 -- 172.21.15.119:0/1173862988 >> 172.21.15.119:0/1173862988 conn(0x7f25d408c080 msgr2=0x7f25d40d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 -- 172.21.15.119:0/1173862988 shutdown_connections 2024-09-13T16:32:54.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.323+0000 7f25d897f700 1 -- 172.21.15.119:0/1173862988 wait complete. 2024-09-13T16:32:54.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 Processor -- start 2024-09-13T16:32:54.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 -- start start 2024-09-13T16:32:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d40ce8c0 0x7f25d41f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40d1610 0x7f25d41550f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d415cc00 0x7f25d41f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d415d510 con 0x7f25d415cc00 2024-09-13T16:32:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25d41f3e70 con 0x7f25d40ce8c0 2024-09-13T16:32:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25d41f2d90 con 0x7f25d40d1610 2024-09-13T16:32:54.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d415cc00 0x7f25d41f3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d415cc00 0x7f25d41f3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50510/0 (socket says 172.21.15.119:50510) 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 -- 172.21.15.119:0/3170936220 learned_addr learned my addr 172.21.15.119:0/3170936220 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d2ffd700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40d1610 0x7f25d41550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d37fe700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d40ce8c0 0x7f25d41f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 -- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40d1610 msgr2=0x7f25d41550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40d1610 0x7f25d41550f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 -- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d40ce8c0 msgr2=0x7f25d41f27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d40ce8c0 0x7f25d41f27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 -- 172.21.15.119:0/3170936220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25c4009a70 con 0x7f25d415cc00 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d3fff700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d415cc00 0x7f25d41f3930 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f25d408bdf0 tx=0x7f25c4004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25cbfff700 1 -- 172.21.15.119:0/3170936220 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4012430 con 0x7f25d415cc00 2024-09-13T16:32:54.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25cbfff700 1 -- 172.21.15.119:0/3170936220 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c4020070 con 0x7f25d415cc00 2024-09-13T16:32:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25d41f3070 con 0x7f25d415cc00 2024-09-13T16:32:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25cbfff700 1 -- 172.21.15.119:0/3170936220 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25c4004280 con 0x7f25d415cc00 2024-09-13T16:32:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.327+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25d41f3630 con 0x7f25d415cc00 2024-09-13T16:32:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.331+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25d40d3670 con 0x7f25d415cc00 2024-09-13T16:32:54.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.331+0000 7f25cbfff700 1 -- 172.21.15.119:0/3170936220 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f25c40043e0 con 0x7f25d415cc00 2024-09-13T16:32:54.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.331+0000 7f25cbfff700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f25b4066730 0x7f25b4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.331+0000 7f25cbfff700 1 -- 172.21.15.119:0/3170936220 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f25c4022070 con 0x7f25d415cc00 2024-09-13T16:32:54.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.331+0000 7f25d37fe700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f25b4066730 0x7f25b4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.331+0000 7f25d37fe700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f25b4066730 0x7f25b4068bf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f25cc000f60 tx=0x7f25cc007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.339+0000 7f25cbfff700 1 -- 172.21.15.119:0/3170936220 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f25c40530d0 con 0x7f25d415cc00 2024-09-13T16:32:54.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.507+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f25d415d650 con 0x7f25d415cc00 2024-09-13T16:32:54.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25cbfff700 1 -- 172.21.15.119:0/3170936220 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) v1 ==== 93+0+10 (secure 0 0 0) 0x7f25c402c0a0 con 0x7f25d415cc00 2024-09-13T16:32:54.515 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f25b4066730 msgr2=0x7f25b4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f25b4066730 0x7f25b4068bf0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f25cc000f60 tx=0x7f25cc007e90 comp rx=0 tx=0).stop 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d415cc00 msgr2=0x7f25d41f3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d415cc00 0x7f25d41f3930 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 crypto rx=0x7f25d408bdf0 tx=0x7f25c4004af0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 shutdown_connections 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f25b4066730 0x7f25b4068bf0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f25d415cc00 0x7f25d41f3930 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f25d40d1610 0x7f25d41550f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 --2- 172.21.15.119:0/3170936220 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f25d40ce8c0 0x7f25d41f27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 >> 172.21.15.119:0/3170936220 conn(0x7f25d408c080 msgr2=0x7f25d415b350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 shutdown_connections 2024-09-13T16:32:54.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.511+0000 7f25d897f700 1 -- 172.21.15.119:0/3170936220 wait complete. 2024-09-13T16:32:54.530 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:32:54.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 Processor -- start 2024-09-13T16:32:54.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 -- start start 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db415cc00 0x7f1db40d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1db40d0c20 0x7f1db40d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1db40ce8c0 0x7f1db40d1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db40cdb00 con 0x7f1db40d0c20 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db40cdde0 con 0x7f1db40ce8c0 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1dba8d1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db40ce0c0 con 0x7f1db415cc00 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.643+0000 7f1db90ce700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1db40d0c20 0x7f1db40d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1db40d0c20 0x7f1db40d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50514/0 (socket says 172.21.15.119:50514) 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 -- 172.21.15.119:0/1720592586 learned_addr learned my addr 172.21.15.119:0/1720592586 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db98cf700 1 --2- 172.21.15.119:0/1720592586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db415cc00 0x7f1db40d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba0d0700 1 --2- 172.21.15.119:0/1720592586 >> v2:172.21.15.145:3300/0 conn(0x7f1db40ce8c0 0x7f1db40d1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 -- 172.21.15.119:0/1720592586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db415cc00 msgr2=0x7f1db40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 --2- 172.21.15.119:0/1720592586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db415cc00 0x7f1db40d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 -- 172.21.15.119:0/1720592586 >> v2:172.21.15.145:3300/0 conn(0x7f1db40ce8c0 msgr2=0x7f1db40d1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 --2- 172.21.15.119:0/1720592586 >> v2:172.21.15.145:3300/0 conn(0x7f1db40ce8c0 0x7f1db40d1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 -- 172.21.15.119:0/1720592586 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db40ce3a0 con 0x7f1db40d0c20 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db90ce700 1 --2- 172.21.15.119:0/1720592586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db40d0c20 0x7f1db40d10d0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1db40d0490 tx=0x7f1db0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=392f051a17348578 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db88cd700 1 -- 172.21.15.119:0/1720592586 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000b8a0 con 0x7f1db40d0c20 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db88cd700 1 -- 172.21.15.119:0/1720592586 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1db000ba00 con 0x7f1db40d0c20 2024-09-13T16:32:54.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1db88cd700 1 -- 172.21.15.119:0/1720592586 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db000bd90 con 0x7f1db40d0c20 2024-09-13T16:32:54.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 -- 172.21.15.119:0/1720592586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db40d0c20 msgr2=0x7f1db40d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/1720592586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db40d0c20 0x7f1db40d10d0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f1db40d0490 tx=0x7f1db0009bf0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 -- 172.21.15.119:0/1720592586 shutdown_connections 2024-09-13T16:32:54.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/1720592586 >> v2:172.21.15.145:3300/0 conn(0x7f1db40ce8c0 0x7f1db40d1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/1720592586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db40d0c20 0x7f1db40d10d0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/1720592586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db415cc00 0x7f1db40d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 -- 172.21.15.119:0/1720592586 >> 172.21.15.119:0/1720592586 conn(0x7f1db408c080 msgr2=0x7f1db40d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:54.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 -- 172.21.15.119:0/1720592586 shutdown_connections 2024-09-13T16:32:54.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.647+0000 7f1dba8d1700 1 -- 172.21.15.119:0/1720592586 wait complete. 2024-09-13T16:32:54.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 Processor -- start 2024-09-13T16:32:54.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 -- start start 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db40ce8c0 0x7f1db41f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db40d0c20 0x7f1db4155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db415cc00 0x7f1db40cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41f2be0 con 0x7f1db415cc00 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db41f5a10 con 0x7f1db40d0c20 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db40c9d10 con 0x7f1db40ce8c0 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db415cc00 0x7f1db40cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db415cc00 0x7f1db40cadb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50524/0 (socket says 172.21.15.119:50524) 2024-09-13T16:32:54.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 -- 172.21.15.119:0/2242186501 learned_addr learned my addr 172.21.15.119:0/2242186501 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:54.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1db98cf700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db40ce8c0 0x7f1db41f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1db90ce700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db40d0c20 0x7f1db4155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 -- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db40ce8c0 msgr2=0x7f1db41f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db40ce8c0 0x7f1db41f26a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 -- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db40d0c20 msgr2=0x7f1db4155960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db40d0c20 0x7f1db4155960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 -- 172.21.15.119:0/2242186501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db0009570 con 0x7f1db415cc00 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba0d0700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db415cc00 0x7f1db40cadb0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f1dac007e60 tx=0x7f1dac00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1daa7fc700 1 -- 172.21.15.119:0/2242186501 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac00ed30 con 0x7f1db415cc00 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1daa7fc700 1 -- 172.21.15.119:0/2242186501 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dac014420 con 0x7f1db415cc00 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db40c9ff0 con 0x7f1db415cc00 2024-09-13T16:32:54.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1daa7fc700 1 -- 172.21.15.119:0/2242186501 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1dac0147d0 con 0x7f1db415cc00 2024-09-13T16:32:54.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.651+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db40ca5b0 con 0x7f1db415cc00 2024-09-13T16:32:54.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.655+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db40d06e0 con 0x7f1db415cc00 2024-09-13T16:32:54.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.655+0000 7f1daa7fc700 1 -- 172.21.15.119:0/2242186501 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1dac014930 con 0x7f1db415cc00 2024-09-13T16:32:54.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.655+0000 7f1daa7fc700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d98066760 0x7f1d98068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.655+0000 7f1daa7fc700 1 -- 172.21.15.119:0/2242186501 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f1dac0849a0 con 0x7f1db415cc00 2024-09-13T16:32:54.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.655+0000 7f1db98cf700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d98066760 0x7f1d98068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.655+0000 7f1db98cf700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d98066760 0x7f1d98068c20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1da4007a60 tx=0x7f1da4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.663+0000 7f1daa7fc700 1 -- 172.21.15.119:0/2242186501 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1dac051ba0 con 0x7f1db415cc00 2024-09-13T16:32:54.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1db408c560 con 0x7f1db415cc00 2024-09-13T16:32:54.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1daa7fc700 1 -- 172.21.15.119:0/2242186501 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f1dac0551c0 con 0x7f1db415cc00 2024-09-13T16:32:54.840 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d98066760 msgr2=0x7f1d98068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d98066760 0x7f1d98068c20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f1da4007a60 tx=0x7f1da4008040 comp rx=0 tx=0).stop 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db415cc00 msgr2=0x7f1db40cadb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db415cc00 0x7f1db40cadb0 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f1dac007e60 tx=0x7f1dac00e3c0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 shutdown_connections 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d98066760 0x7f1d98068c20 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1db415cc00 0x7f1db40cadb0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1db40d0c20 0x7f1db4155960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 --2- 172.21.15.119:0/2242186501 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1db40ce8c0 0x7f1db41f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 >> 172.21.15.119:0/2242186501 conn(0x7f1db408c080 msgr2=0x7f1db406ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:54.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 shutdown_connections 2024-09-13T16:32:54.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.835+0000 7f1dba8d1700 1 -- 172.21.15.119:0/2242186501 wait complete. 2024-09-13T16:32:54.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:32:54.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd442de0700 1 Processor -- start 2024-09-13T16:32:54.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd442de0700 1 -- start start 2024-09-13T16:32:54.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd442de0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd43c160210 0x7fd43c0d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd442de0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d17e0 0x7fd43c0b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd442de0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd43c0b2240 con 0x7fd43c0d3760 2024-09-13T16:32:54.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd442de0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd43c1605f0 con 0x7fd43c160210 2024-09-13T16:32:54.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd442de0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c0b2520 con 0x7fd43c0d17e0 2024-09-13T16:32:54.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd4425df700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d17e0 0x7fd43c0b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd441dde700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd43c160210 0x7fd43c0d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd4425df700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d17e0 0x7fd43c0b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:38710/0 (socket says 172.21.15.119:38710) 2024-09-13T16:32:54.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd4415dd700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd43c0d3760 0x7fd43c0d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:56264/0 (socket says 172.21.15.119:56264) 2024-09-13T16:32:54.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd4425df700 1 -- 172.21.15.119:0/1771144579 learned_addr learned my addr 172.21.15.119:0/1771144579 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:54.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd440ddc700 1 -- 172.21.15.119:0/1771144579 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853208879 0 0) 0x7fd43c0b2240 con 0x7fd43c0d3760 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd440ddc700 1 -- 172.21.15.119:0/1771144579 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4280035d0 con 0x7fd43c0d3760 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd4425df700 1 -- 172.21.15.119:0/1771144579 >> v2:172.21.15.145:3300/0 conn(0x7fd43c160210 msgr2=0x7fd43c0d3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd4425df700 1 --2- 172.21.15.119:0/1771144579 >> v2:172.21.15.145:3300/0 conn(0x7fd43c160210 0x7fd43c0d3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.975+0000 7fd4425df700 1 -- 172.21.15.119:0/1771144579 >> v1:172.21.15.145:6789/0 conn(0x7fd43c0d3760 legacy=0x7fd43c0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd4425df700 1 -- 172.21.15.119:0/1771144579 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd43c0d0780 con 0x7fd43c0d17e0 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd4425df700 1 --2- 172.21.15.119:0/1771144579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d17e0 0x7fd43c0b29b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd434009e50 tx=0x7fd43400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1719b681d5d3a0ed server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd440ddc700 1 -- 172.21.15.119:0/1771144579 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43400e070 con 0x7fd43c0d17e0 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd440ddc700 1 -- 172.21.15.119:0/1771144579 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd43400b910 con 0x7fd43c0d17e0 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd440ddc700 1 -- 172.21.15.119:0/1771144579 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43400bc80 con 0x7fd43c0d17e0 2024-09-13T16:32:54.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 -- 172.21.15.119:0/1771144579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d17e0 msgr2=0x7fd43c0b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 --2- 172.21.15.119:0/1771144579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d17e0 0x7fd43c0b29b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd434009e50 tx=0x7fd43400b330 comp rx=0 tx=0).stop 2024-09-13T16:32:54.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 -- 172.21.15.119:0/1771144579 shutdown_connections 2024-09-13T16:32:54.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 --2- 172.21.15.119:0/1771144579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d17e0 0x7fd43c0b29b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 --2- 172.21.15.119:0/1771144579 >> v2:172.21.15.145:3300/0 conn(0x7fd43c160210 0x7fd43c0d3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 -- 172.21.15.119:0/1771144579 >> 172.21.15.119:0/1771144579 conn(0x7fd43c08c080 msgr2=0x7fd43c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:54.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 -- 172.21.15.119:0/1771144579 shutdown_connections 2024-09-13T16:32:54.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 -- 172.21.15.119:0/1771144579 wait complete. 2024-09-13T16:32:54.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 Processor -- start 2024-09-13T16:32:54.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.979+0000 7fd442de0700 1 -- start start 2024-09-13T16:32:54.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd43c0d17e0 0x7fd43c0b82b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d3760 0x7fd43c0b68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd43c160210 0x7fd43c0c7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c0b8a20 con 0x7fd43c160210 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c0b87f0 con 0x7fd43c0d17e0 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd43c0b6410 con 0x7fd43c0d3760 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd43c160210 0x7fd43c0c7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd43c160210 0x7fd43c0c7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50536/0 (socket says 172.21.15.119:50536) 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 -- 172.21.15.119:0/5294661 learned_addr learned my addr 172.21.15.119:0/5294661 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4415dd700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d3760 0x7fd43c0b68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd441dde700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd43c0d17e0 0x7fd43c0b82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 -- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d3760 msgr2=0x7fd43c0b68a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d3760 0x7fd43c0b68a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 -- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd43c0d17e0 msgr2=0x7fd43c0b82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd43c0d17e0 0x7fd43c0b82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 -- 172.21.15.119:0/5294661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd434009b30 con 0x7fd43c160210 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4425df700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd43c160210 0x7fd43c0c7690 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fd434012000 tx=0x7fd434004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4327fc700 1 -- 172.21.15.119:0/5294661 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4340048c0 con 0x7fd43c160210 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4327fc700 1 -- 172.21.15.119:0/5294661 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4340099b0 con 0x7fd43c160210 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd43c0b5ae0 con 0x7fd43c160210 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd4327fc700 1 -- 172.21.15.119:0/5294661 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd43401b670 con 0x7fd43c160210 2024-09-13T16:32:54.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.983+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd43c0b60a0 con 0x7fd43c160210 2024-09-13T16:32:54.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.987+0000 7fd4327fc700 1 -- 172.21.15.119:0/5294661 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd43401b7d0 con 0x7fd43c160210 2024-09-13T16:32:54.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.987+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd43c0b6260 con 0x7fd43c160210 2024-09-13T16:32:54.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.987+0000 7fd4327fc700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd420066730 0x7fd420068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:54.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.987+0000 7fd4327fc700 1 -- 172.21.15.119:0/5294661 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fd434085b60 con 0x7fd43c160210 2024-09-13T16:32:54.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.987+0000 7fd441dde700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd420066730 0x7fd420068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:54.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.987+0000 7fd441dde700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd420066730 0x7fd420068bf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009bf0 tx=0x7fd42c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:55.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:54.995+0000 7fd4327fc700 1 -- 172.21.15.119:0/5294661 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd434052cb0 con 0x7fd43c160210 2024-09-13T16:32:55.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.163+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd43c0b6580 con 0x7fd43c160210 2024-09-13T16:32:55.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.163+0000 7fd4327fc700 1 -- 172.21.15.119:0/5294661 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd43401f020 con 0x7fd43c160210 2024-09-13T16:32:55.171 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:32:55.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.167+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd420066730 msgr2=0x7fd420068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd420066730 0x7fd420068bf0 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009bf0 tx=0x7fd42c009240 comp rx=0 tx=0).stop 2024-09-13T16:32:55.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd43c160210 msgr2=0x7fd43c0c7690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd43c160210 0x7fd43c0c7690 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7fd434012000 tx=0x7fd434004af0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 shutdown_connections 2024-09-13T16:32:55.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd420066730 0x7fd420068bf0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd43c160210 0x7fd43c0c7690 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd43c0d3760 0x7fd43c0b68a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 --2- 172.21.15.119:0/5294661 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd43c0d17e0 0x7fd43c0b82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 >> 172.21.15.119:0/5294661 conn(0x7fd43c08c080 msgr2=0x7fd43c15ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:55.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 shutdown_connections 2024-09-13T16:32:55.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.171+0000 7fd442de0700 1 -- 172.21.15.119:0/5294661 wait complete. 2024-09-13T16:32:55.192 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:32:55.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 Processor -- start 2024-09-13T16:32:55.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- start start 2024-09-13T16:32:55.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc6f0164780 0x7fc6f00b4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:55.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f00ac040 0x7fc6f006b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:55.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f006a8a0 con 0x7fc6f006b5c0 2024-09-13T16:32:55.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f00ce690 con 0x7fc6f0164780 2024-09-13T16:32:55.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f006ba70 con 0x7fc6f00ac040 2024-09-13T16:32:55.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc6f0164780 0x7fc6f00b4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:55.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc6f0164780 0x7fc6f00b4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50540/0 (socket says 172.21.15.119:50540) 2024-09-13T16:32:55.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 -- 172.21.15.119:0/2430807817 learned_addr learned my addr 172.21.15.119:0/2430807817 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:55.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6edd9b700 1 --2- 172.21.15.119:0/2430807817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f00ac040 0x7fc6f006b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:55.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 -- 172.21.15.119:0/2430807817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f00ac040 msgr2=0x7fc6f006b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:32:55.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 --2- 172.21.15.119:0/2430807817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f00ac040 0x7fc6f006b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 -- 172.21.15.119:0/2430807817 >> v1:172.21.15.145:6789/0 conn(0x7fc6f006b5c0 legacy=0x7fc6f00b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 -- 172.21.15.119:0/2430807817 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f006ab80 con 0x7fc6f0164780 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ee59c700 1 --2- 172.21.15.119:0/2430807817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f0164780 0x7fc6f00b4d00 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009910 tx=0x7fc6d8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5de7cdc1c34d2700 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ed59a700 1 -- 172.21.15.119:0/2430807817 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011830 con 0x7fc6f0164780 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ed59a700 1 -- 172.21.15.119:0/2430807817 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6d8011990 con 0x7fc6f0164780 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ed59a700 1 -- 172.21.15.119:0/2430807817 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6d8011d40 con 0x7fc6f0164780 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- 172.21.15.119:0/2430807817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f0164780 msgr2=0x7fc6f00b4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/2430807817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f0164780 0x7fc6f00b4d00 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009910 tx=0x7fc6d8009c20 comp rx=0 tx=0).stop 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- 172.21.15.119:0/2430807817 shutdown_connections 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/2430807817 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f00ac040 0x7fc6f006b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/2430807817 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f0164780 0x7fc6f00b4d00 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- 172.21.15.119:0/2430807817 >> 172.21.15.119:0/2430807817 conn(0x7fc6f00b3ee0 msgr2=0x7fc6f00b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- 172.21.15.119:0/2430807817 shutdown_connections 2024-09-13T16:32:55.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.303+0000 7fc6ef59e700 1 -- 172.21.15.119:0/2430807817 wait complete. 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 Processor -- start 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 -- start start 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b5c0 0x7fc6f0098980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f00ac040 0x7fc6f009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f0164780 0x7fc6f009c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f014dc10 con 0x7fc6f00ac040 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f015fd70 con 0x7fc6f0164780 2024-09-13T16:32:55.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f0160050 con 0x7fc6f006b5c0 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f00ac040 0x7fc6f009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f00ac040 0x7fc6f009bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50550/0 (socket says 172.21.15.119:50550) 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 -- 172.21.15.119:0/3155197042 learned_addr learned my addr 172.21.15.119:0/3155197042 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6eed9d700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f0164780 0x7fc6f009c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ee59c700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b5c0 0x7fc6f0098980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 -- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b5c0 msgr2=0x7fc6f0098980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b5c0 0x7fc6f0098980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 -- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f0164780 msgr2=0x7fc6f009c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f0164780 0x7fc6f009c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 -- 172.21.15.119:0/3155197042 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d8009570 con 0x7fc6f00ac040 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6edd9b700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f00ac040 0x7fc6f009bc90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009fd0 tx=0x7fc6e000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6e6ffd700 1 -- 172.21.15.119:0/3155197042 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e00095a0 con 0x7fc6f00ac040 2024-09-13T16:32:55.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6e6ffd700 1 -- 172.21.15.119:0/3155197042 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e0013680 con 0x7fc6f00ac040 2024-09-13T16:32:55.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f0160330 con 0x7fc6f00ac040 2024-09-13T16:32:55.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6e6ffd700 1 -- 172.21.15.119:0/3155197042 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0013a10 con 0x7fc6f00ac040 2024-09-13T16:32:55.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f01608f0 con 0x7fc6f00ac040 2024-09-13T16:32:55.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.307+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f00cec80 con 0x7fc6f00ac040 2024-09-13T16:32:55.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.311+0000 7fc6e6ffd700 1 -- 172.21.15.119:0/3155197042 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc6e0013b70 con 0x7fc6f00ac040 2024-09-13T16:32:55.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.311+0000 7fc6e6ffd700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:32:55.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.311+0000 7fc6e6ffd700 1 -- 172.21.15.119:0/3155197042 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fc6e00846b0 con 0x7fc6f00ac040 2024-09-13T16:32:55.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.311+0000 7fc6ee59c700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:32:55.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.311+0000 7fc6ee59c700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009c60 tx=0x7fc6d8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:32:55.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.319+0000 7fc6e6ffd700 1 -- 172.21.15.119:0/3155197042 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc6e0051810 con 0x7fc6f00ac040 2024-09-13T16:32:55.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc6f0074230 con 0x7fc6f00ac040 2024-09-13T16:32:55.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6e6ffd700 1 -- 172.21.15.119:0/3155197042 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v98) v1 ==== 80+0+124 (secure 0 0 0) 0x7fc6e0017090 con 0x7fc6f00ac040 2024-09-13T16:32:55.499 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6835/3011171408 2024-09-14T16:31:41.722614+0000 2024-09-13T16:32:55.500 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6834/3011171408 2024-09-14T16:31:41.722614+0000 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 msgr2=0x7fc6d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc6d8009c60 tx=0x7fc6d8008030 comp rx=0 tx=0).stop 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f00ac040 msgr2=0x7fc6f009bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f00ac040 0x7fc6f009bc90 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0009fd0 tx=0x7fc6e000ebd0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 shutdown_connections 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6d0066730 0x7fc6d0068bf0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc6f0164780 0x7fc6f009c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc6f00ac040 0x7fc6f009bc90 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 --2- 172.21.15.119:0/3155197042 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc6f006b5c0 0x7fc6f0098980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:32:55.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 >> 172.21.15.119:0/3155197042 conn(0x7fc6f00b3ee0 msgr2=0x7fc6f006dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:32:55.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 shutdown_connections 2024-09-13T16:32:55.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:32:55.495+0000 7fc6ef59e700 1 -- 172.21.15.119:0/3155197042 wait complete. 2024-09-13T16:32:55.503 INFO:teuthology.orchestra.run.smithi119.stderr:listed 2 entries 2024-09-13T16:32:55.514 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:32:55.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:32:55.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:32:55.527 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[47917]: fuse finished with error 0 and tester_r 0 2024-09-13T16:32:55.529 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:32:57.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:32:57.846 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:33:01.630 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:33:01.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:01.631 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:33:01.639 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:01.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 Processor -- start 2024-09-13T16:33:01.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 -- start start 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f11e806c5f0 0x7f11e80cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e815a010 0x7f11e806c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f11e80cd200 0x7f11e806bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e806cf10 con 0x7f11e80cd200 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11e80ce540 con 0x7f11e806c5f0 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ef4da700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e80cc720 con 0x7f11e815a010 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f11e80cd200 0x7f11e806bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f11e80cd200 0x7f11e806bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50552/0 (socket says 172.21.15.119:50552) 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11ed276700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f11e806c5f0 0x7f11e80cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eca75700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e815a010 0x7f11e806c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:01.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 -- 172.21.15.119:0/3759189827 learned_addr learned my addr 172.21.15.119:0/3759189827 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:01.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 -- 172.21.15.119:0/3759189827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e815a010 msgr2=0x7f11e806c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:01.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 --2- 172.21.15.119:0/3759189827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e815a010 0x7f11e806c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 -- 172.21.15.119:0/3759189827 >> v2:172.21.15.145:3300/0 conn(0x7f11e806c5f0 msgr2=0x7f11e80cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 --2- 172.21.15.119:0/3759189827 >> v2:172.21.15.145:3300/0 conn(0x7f11e806c5f0 0x7f11e80cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.759+0000 7f11eda77700 1 -- 172.21.15.119:0/3759189827 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e80cca00 con 0x7f11e80cd200 2024-09-13T16:33:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11eda77700 1 --2- 172.21.15.119:0/3759189827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e806bd60 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f11e806dfe0 tx=0x7f11e000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7667c555d3fbce8f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11dffff700 1 -- 172.21.15.119:0/3759189827 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e000e070 con 0x7f11e80cd200 2024-09-13T16:33:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11dffff700 1 -- 172.21.15.119:0/3759189827 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11e000b910 con 0x7f11e80cd200 2024-09-13T16:33:01.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11dffff700 1 -- 172.21.15.119:0/3759189827 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11e000bcc0 con 0x7f11e80cd200 2024-09-13T16:33:01.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 -- 172.21.15.119:0/3759189827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 msgr2=0x7f11e806bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:01.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3759189827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e806bd60 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f11e806dfe0 tx=0x7f11e000b330 comp rx=0 tx=0).stop 2024-09-13T16:33:01.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 -- 172.21.15.119:0/3759189827 shutdown_connections 2024-09-13T16:33:01.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3759189827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e806bd60 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3759189827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e815a010 0x7f11e806c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3759189827 >> v2:172.21.15.145:3300/0 conn(0x7f11e806c5f0 0x7f11e80cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 -- 172.21.15.119:0/3759189827 >> 172.21.15.119:0/3759189827 conn(0x7f11e806a350 msgr2=0x7f11e806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:01.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 -- 172.21.15.119:0/3759189827 shutdown_connections 2024-09-13T16:33:01.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 -- 172.21.15.119:0/3759189827 wait complete. 2024-09-13T16:33:01.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 Processor -- start 2024-09-13T16:33:01.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.763+0000 7f11ef4da700 1 -- start start 2024-09-13T16:33:01.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e806c5f0 0x7f11e8148f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e814ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e815a010 0x7f11e8152d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e814ab00 con 0x7f11e80cd200 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e814ade0 con 0x7f11e815a010 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e814b0c0 con 0x7f11e806c5f0 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e814ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e814ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50556/0 (socket says 172.21.15.119:50556) 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 -- 172.21.15.119:0/3252244891 learned_addr learned my addr 172.21.15.119:0/3252244891 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ed276700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e806c5f0 0x7f11e8148f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eda77700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e815a010 0x7f11e8152d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 -- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e806c5f0 msgr2=0x7f11e8148f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e806c5f0 0x7f11e8148f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 -- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e815a010 msgr2=0x7f11e8152d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:01.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e815a010 0x7f11e8152d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 -- 172.21.15.119:0/3252244891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e0009b30 con 0x7f11e80cd200 2024-09-13T16:33:01.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ed276700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e806c5f0 0x7f11e8148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:01.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11eca75700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e814ff00 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f11d800e7f0 tx=0x7f11d800eb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:01.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ddffb700 1 -- 172.21.15.119:0/3252244891 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d8009580 con 0x7f11e80cd200 2024-09-13T16:33:01.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ddffb700 1 -- 172.21.15.119:0/3252244891 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11d8004020 con 0x7f11e80cd200 2024-09-13T16:33:01.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ddffb700 1 -- 172.21.15.119:0/3252244891 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11d80043b0 con 0x7f11e80cd200 2024-09-13T16:33:01.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11e814b3a0 con 0x7f11e80cd200 2024-09-13T16:33:01.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11e81494c0 con 0x7f11e80cd200 2024-09-13T16:33:01.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.767+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11e814b8c0 con 0x7f11e80cd200 2024-09-13T16:33:01.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.771+0000 7f11ddffb700 1 -- 172.21.15.119:0/3252244891 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f11d80036c0 con 0x7f11e80cd200 2024-09-13T16:33:01.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.771+0000 7f11ddffb700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f11cc066730 0x7f11cc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:01.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.771+0000 7f11ddffb700 1 -- 172.21.15.119:0/3252244891 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f11d8084490 con 0x7f11e80cd200 2024-09-13T16:33:01.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.771+0000 7f11ed276700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f11cc066730 0x7f11cc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:01.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.771+0000 7f11ed276700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f11cc066730 0x7f11cc068bf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009f60 tx=0x7f11e4009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:01.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.775+0000 7f11ddffb700 1 -- 172.21.15.119:0/3252244891 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f11d8051640 con 0x7f11e80cd200 2024-09-13T16:33:01.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.983+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11e814b8c0 con 0x7f11e80cd200 2024-09-13T16:33:01.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.983+0000 7f11ddffb700 1 -- 172.21.15.119:0/3252244891 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f11e814b8c0 con 0x7f11e80cd200 2024-09-13T16:33:01.987 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:01.987 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":63,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:31:57.084733+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":96,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7881},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7881":{"gid":7881,"name":"c","rank":0,"incarnation":59,"state":"up:active","state_seq":11,"addr":"172.21.15.119:6835/3937841911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3937841911},{"type":"v1","addr":"172.21.15.119:6835","nonce":3937841911}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:01.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f11cc066730 msgr2=0x7f11cc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f11cc066730 0x7f11cc068bf0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7f11e4009f60 tx=0x7f11e4009240 comp rx=0 tx=0).stop 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 msgr2=0x7f11e814ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e814ff00 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7f11d800e7f0 tx=0x7f11d800eb00 comp rx=0 tx=0).stop 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 shutdown_connections 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f11cc066730 0x7f11cc068bf0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f11e815a010 0x7f11e8152d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f11e80cd200 0x7f11e814ff00 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 --2- 172.21.15.119:0/3252244891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f11e806c5f0 0x7f11e8148f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 >> 172.21.15.119:0/3252244891 conn(0x7f11e806a350 msgr2=0x7f11e80aee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 shutdown_connections 2024-09-13T16:33:01.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:01.987+0000 7f11ef4da700 1 -- 172.21.15.119:0/3252244891 wait complete. 2024-09-13T16:33:01.991 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 63 2024-09-13T16:33:02.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-13T16:33:02.137 INFO:teuthology.orchestra.run.smithi119.stdout:{"message":"","return_code":0} 2024-09-13T16:33:02.138 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:33:02.138 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:33:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.247+0000 7fa866415700 1 Processor -- start 2024-09-13T16:33:02.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.247+0000 7fa866415700 1 -- start start 2024-09-13T16:33:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.247+0000 7fa866415700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b3ad0 0x7fa860159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.247+0000 7fa866415700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa860159880 0x7fa8600cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.247+0000 7fa866415700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa860159cf0 con 0x7fa8600b36f0 2024-09-13T16:33:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.247+0000 7fa866415700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8600b2050 con 0x7fa860159880 2024-09-13T16:33:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.247+0000 7fa866415700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600b2320 con 0x7fa8600b3ad0 2024-09-13T16:33:02.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b3ad0 0x7fa860159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa8649b2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa860159880 0x7fa8600cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b3ad0 0x7fa860159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:38764/0 (socket says 172.21.15.119:38764) 2024-09-13T16:33:02.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa8649b2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa860159880 0x7fa8600cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:48816/0 (socket says 172.21.15.119:48816) 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 -- 172.21.15.119:0/2358234465 learned_addr learned my addr 172.21.15.119:0/2358234465 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85ffff700 1 --1- 172.21.15.119:0/2358234465 >> v1:172.21.15.145:6789/0 conn(0x7fa8600b36f0 0x7fa8600b1940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:56280/0 (socket says 172.21.15.119:56280) 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 -- 172.21.15.119:0/2358234465 >> v2:172.21.15.145:3300/0 conn(0x7fa860159880 msgr2=0x7fa8600cd2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 --2- 172.21.15.119:0/2358234465 >> v2:172.21.15.145:3300/0 conn(0x7fa860159880 0x7fa8600cd2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 -- 172.21.15.119:0/2358234465 >> v1:172.21.15.145:6789/0 conn(0x7fa8600b36f0 legacy=0x7fa8600b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 -- 172.21.15.119:0/2358234465 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8600b2600 con 0x7fa8600b3ad0 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85f7fe700 1 --2- 172.21.15.119:0/2358234465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b3ad0 0x7fa860159340 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa850009e50 tx=0x7fa85000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=33892b9d16f8aa8b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85effd700 1 -- 172.21.15.119:0/2358234465 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85000e070 con 0x7fa8600b3ad0 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85effd700 1 -- 172.21.15.119:0/2358234465 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85000b910 con 0x7fa8600b3ad0 2024-09-13T16:33:02.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa85effd700 1 -- 172.21.15.119:0/2358234465 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85000bcc0 con 0x7fa8600b3ad0 2024-09-13T16:33:02.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 -- 172.21.15.119:0/2358234465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b3ad0 msgr2=0x7fa860159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 --2- 172.21.15.119:0/2358234465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b3ad0 0x7fa860159340 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fa850009e50 tx=0x7fa85000b330 comp rx=0 tx=0).stop 2024-09-13T16:33:02.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 -- 172.21.15.119:0/2358234465 shutdown_connections 2024-09-13T16:33:02.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 --2- 172.21.15.119:0/2358234465 >> v2:172.21.15.145:3300/0 conn(0x7fa860159880 0x7fa8600cd2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 --2- 172.21.15.119:0/2358234465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b3ad0 0x7fa860159340 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 -- 172.21.15.119:0/2358234465 >> 172.21.15.119:0/2358234465 conn(0x7fa86006a350 msgr2=0x7fa86006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 -- 172.21.15.119:0/2358234465 shutdown_connections 2024-09-13T16:33:02.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.251+0000 7fa866415700 1 -- 172.21.15.119:0/2358234465 wait complete. 2024-09-13T16:33:02.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 Processor -- start 2024-09-13T16:33:02.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 -- start start 2024-09-13T16:33:02.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b36f0 0x7fa860139840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8600b3ad0 0x7fa860066090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa860159880 0x7fa8600665d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86014e860 con 0x7fa860159880 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa86013a380 con 0x7fa8600b3ad0 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8600af7b0 con 0x7fa8600b36f0 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa860159880 0x7fa8600665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa85f7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8600b3ad0 0x7fa860066090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa85ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b36f0 0x7fa860139840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa860159880 0x7fa8600665d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:50562/0 (socket says 172.21.15.119:50562) 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 -- 172.21.15.119:0/438354845 learned_addr learned my addr 172.21.15.119:0/438354845 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 -- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b36f0 msgr2=0x7fa860139840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b36f0 0x7fa860139840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 -- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8600b3ad0 msgr2=0x7fa860066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8600b3ad0 0x7fa860066090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 -- 172.21.15.119:0/438354845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa850009b30 con 0x7fa860159880 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa8649b2700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa860159880 0x7fa8600665d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa858010e90 tx=0x7fa85800e640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa85cff9700 1 -- 172.21.15.119:0/438354845 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85800b9e0 con 0x7fa860159880 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa85cff9700 1 -- 172.21.15.119:0/438354845 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa858011450 con 0x7fa860159880 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa85cff9700 1 -- 172.21.15.119:0/438354845 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8580117e0 con 0x7fa860159880 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa86014dfc0 con 0x7fa860159880 2024-09-13T16:33:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.255+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86014e580 con 0x7fa860159880 2024-09-13T16:33:02.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.259+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86013a1e0 con 0x7fa860159880 2024-09-13T16:33:02.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.259+0000 7fa85cff9700 1 -- 172.21.15.119:0/438354845 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa858005a40 con 0x7fa860159880 2024-09-13T16:33:02.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.259+0000 7fa85cff9700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa84c066730 0x7fa84c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.259+0000 7fa85cff9700 1 -- 172.21.15.119:0/438354845 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fa858086f00 con 0x7fa860159880 2024-09-13T16:33:02.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.259+0000 7fa85ffff700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa84c066730 0x7fa84c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.259+0000 7fa85ffff700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa84c066730 0x7fa84c068bf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa854000f60 tx=0x7fa854007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.267+0000 7fa85cff9700 1 -- 172.21.15.119:0/438354845 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa858053fb0 con 0x7fa860159880 2024-09-13T16:33:02.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.471+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa86013a1e0 con 0x7fa860159880 2024-09-13T16:33:02.473 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:33:02.471+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:33:02.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.483+0000 7fa85cff9700 1 -- 172.21.15.119:0/438354845 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v64) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa86013a1e0 con 0x7fa860159880 2024-09-13T16:33:02.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa84c066730 msgr2=0x7fa84c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa84c066730 0x7fa84c068bf0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa854000f60 tx=0x7fa854007f30 comp rx=0 tx=0).stop 2024-09-13T16:33:02.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa860159880 msgr2=0x7fa8600665d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa860159880 0x7fa8600665d0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7fa858010e90 tx=0x7fa85800e640 comp rx=0 tx=0).stop 2024-09-13T16:33:02.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 shutdown_connections 2024-09-13T16:33:02.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa84c066730 0x7fa84c068bf0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa860159880 0x7fa8600665d0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa8600b3ad0 0x7fa860066090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 --2- 172.21.15.119:0/438354845 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa8600b36f0 0x7fa860139840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 >> 172.21.15.119:0/438354845 conn(0x7fa86006a350 msgr2=0x7fa860070630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 shutdown_connections 2024-09-13T16:33:02.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.487+0000 7fa866415700 1 -- 172.21.15.119:0/438354845 wait complete. 2024-09-13T16:33:02.491 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:33:02.495 INFO:tasks.ceph.mds.c.smithi119.stderr: -4063> 2024-09-13T16:31:49.572+0000 7fa101b5c700 -1 log_channel(cluster) log [ERR] : dir 0x10000000000 object missing on disk; some files may be lost (/dir) 2024-09-13T16:33:02.496 INFO:tasks.ceph.mds.c.smithi119.stderr: -3886> 2024-09-13T16:31:49.752+0000 7fa101b5c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 ap=1 f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0) n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1) (iversion lock) | dirfrag=0 authpin=1 scrubqueue=0 0x5589ea977b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones"},"return_code":-2} 2024-09-13T16:33:02.496 INFO:tasks.ceph.mds.c.smithi119.stderr: -3869> 2024-09-13T16:31:49.904+0000 7fa105363700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:33:02.498 INFO:tasks.ceph.mds.c.smithi119.stderr: -2287> 2024-09-13T16:32:21.256+0000 7fa101b5c700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000000 [...2,head] /dir/ auth v13 pv16 ap=3 DIRTYPARENT f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0)->f(v1 m2024-09-13T16:31:37.078416+0000) n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1)->n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1) (inest lock dirty) (ifile lock->sync w=1 flushing) (iversion lock) | dirtyscattered=2 lock=1 dirfrag=1 dirtyrstat=1 dirtyparent=1 waiter=1 authpin=1 scrubqueue=0 0x5589ea977b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":-2,"ondisk_value":"(-1)0x0:[]//[]","memoryvalue":"(13)0x10000000000:[<0x1/dir v13>]//[]","error_str":"failed to decode on-disk backtrace (0 bytes)!"},"raw_stats":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n(v0 1=0+1)","memory_value.dirstat":"f(v0 m2024-09-13T16:31:37.078416+0000 1=1+0)","memory_value.rstat":"n(v0 rc2024-09-13T16:31:37.152203+0000 2=1+1)","error_str":"freshly-calculated rstats don't match existing ones (will be fixed)"},"return_code":-2} 2024-09-13T16:33:02.503 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-09-13T16:33:02.528 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:33:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.539+0000 7ff406ea99c0 1 -- start start 2024-09-13T16:33:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.539+0000 7ff406ea99c0 1 Processor -- start 2024-09-13T16:33:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6ac7800 0x555ac6b6c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6ac7400 0x555ac6b6cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6ac7000 0x555ac6b6d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b48a00 con 0x555ac6ac7800 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b48780 con 0x555ac6ac7000 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b48640 con 0x555ac6ac7400 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6ac7800 0x555ac6b6c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6ac7800 0x555ac6b6c580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33208/0 (socket says 172.21.15.119:33208) 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 -- 172.21.15.119:0/2540923950 learned_addr learned my addr 172.21.15.119:0/2540923950 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff404e62700 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6ac7400 0x555ac6b6cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405e64700 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6ac7000 0x555ac6b6d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 -- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6ac7400 msgr2=0x555ac6b6cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6ac7400 0x555ac6b6cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 -- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6ac7000 msgr2=0x555ac6b6d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6ac7000 0x555ac6b6d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 -- 172.21.15.119:0/2540923950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac6aea9c0 con 0x555ac6ac7800 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405e64700 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6ac7000 0x555ac6b6d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff405663700 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6ac7800 0x555ac6b6c580 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x555ac6b87920 tx=0x555ac6b7ac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd0750cca27dcfe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff402e5e700 1 -- 172.21.15.119:0/2540923950 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555ac6b17600 con 0x555ac6ac7800 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff402e5e700 1 -- 172.21.15.119:0/2540923950 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555ac6b12180 con 0x555ac6ac7800 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff402e5e700 1 -- 172.21.15.119:0/2540923950 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555ac6b171e0 con 0x555ac6ac7800 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 -- 172.21.15.119:0/2540923950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ac6aeab60 con 0x555ac6ac7800 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 1 -- 172.21.15.119:0/2540923950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x555ac6aea340 con 0x555ac6ac7800 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 4 waiting for MDS map... 2024-09-13T16:33:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff402e5e700 1 -- 172.21.15.119:0/2540923950 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x555ac6b00c40 con 0x555ac6ac7800 2024-09-13T16:33:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff402e5e700 1 -- 172.21.15.119:0/2540923950 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 64) v1 ==== 2573+0+0 (secure 0 0 0) 0x555ac78b4900 con 0x555ac6ac7800 2024-09-13T16:33:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.543+0000 7ff406ea99c0 4 Got MDS map 64 2024-09-13T16:33:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 4 datascan.main: Using metadata pool output 2024-09-13T16:33:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 4 datascan.main: connecting to RADOS... 2024-09-13T16:33:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 Processor -- start 2024-09-13T16:33:02.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 -- start start 2024-09-13T16:33:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db6800 0x555ac6b6d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6c00 0x555ac6b6db80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db7000 0x555ac6b6e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac6afde00 con 0x555ac6db7000 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b48780 con 0x555ac6db6800 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b48500 con 0x555ac6db6c00 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db7000 0x555ac6b6e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db7000 0x555ac6b6e100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33210/0 (socket says 172.21.15.119:33210) 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 -- 172.21.15.119:0/3963315578 learned_addr learned my addr 172.21.15.119:0/3963315578 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff405663700 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6c00 0x555ac6b6db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff405e64700 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db6800 0x555ac6b6d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 -- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6c00 msgr2=0x555ac6b6db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6c00 0x555ac6b6db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 -- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db6800 msgr2=0x555ac6b6d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db6800 0x555ac6b6d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 -- 172.21.15.119:0/3963315578 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac6aea000 con 0x555ac6db7000 2024-09-13T16:33:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.547+0000 7ff404e62700 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db7000 0x555ac6b6e100 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x555ac78a26f0 tx=0x555ac78a4c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c39ffd036abb0504 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff400e5a700 1 -- 172.21.15.119:0/3963315578 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555ac6b17600 con 0x555ac6db7000 2024-09-13T16:33:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff400e5a700 1 -- 172.21.15.119:0/3963315578 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555ac6b12480 con 0x555ac6db7000 2024-09-13T16:33:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff400e5a700 1 -- 172.21.15.119:0/3963315578 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555ac6b171e0 con 0x555ac6db7000 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 -- 172.21.15.119:0/3963315578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db7000 msgr2=0x555ac6b6e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db7000 0x555ac6b6e100 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x555ac78a26f0 tx=0x555ac78a4c00 comp rx=0 tx=0).stop 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 -- 172.21.15.119:0/3963315578 shutdown_connections 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db7000 0x555ac6b6e100 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6c00 0x555ac6b6db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/3963315578 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db6800 0x555ac6b6d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 -- 172.21.15.119:0/3963315578 >> 172.21.15.119:0/3963315578 conn(0x555ac6ac6c00 msgr2=0x555ac6b83800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 -- 172.21.15.119:0/3963315578 shutdown_connections 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 -- 172.21.15.119:0/3963315578 wait complete. 2024-09-13T16:33:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 Processor -- start 2024-09-13T16:33:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 -- start start 2024-09-13T16:33:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6800 0x555ac6b6f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.551+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db6c00 0x555ac6b6ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db7000 0x555ac6b6e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b483c0 con 0x555ac6db6c00 2024-09-13T16:33:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b48280 con 0x555ac6db7000 2024-09-13T16:33:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x555ac6b48140 con 0x555ac6db6800 2024-09-13T16:33:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db6c00 0x555ac6b6ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db6c00 0x555ac6b6ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33218/0 (socket says 172.21.15.119:33218) 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 learned_addr learned my addr 172.21.15.119:0/4265003647 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6800 0x555ac6b6f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff404e62700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db7000 0x555ac6b6e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6800 msgr2=0x555ac6b6f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6db6800 0x555ac6b6f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db7000 msgr2=0x555ac6b6e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6db7000 0x555ac6b6e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555ac6aea9c0 con 0x555ac6db6c00 2024-09-13T16:33:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405663700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db6c00 0x555ac6b6ec00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x555ac78a3950 tx=0x555ac6b7a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff3ff657700 1 -- 172.21.15.119:0/4265003647 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555ac6b16f20 con 0x555ac6db6c00 2024-09-13T16:33:02.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff3ff657700 1 -- 172.21.15.119:0/4265003647 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555ac78b9800 con 0x555ac6db6c00 2024-09-13T16:33:02.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x555ac6aeab60 con 0x555ac6db6c00 2024-09-13T16:33:02.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff3ff657700 1 -- 172.21.15.119:0/4265003647 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x555ac6b17600 con 0x555ac6db6c00 2024-09-13T16:33:02.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555ac6aeaea0 con 0x555ac6db6c00 2024-09-13T16:33:02.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 4 datascan.init: resolving metadata pool 13 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff3ff657700 1 -- 172.21.15.119:0/4265003647 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x555ac6b26840 con 0x555ac6db6c00 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff3ff657700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x555ac6e36800 0x555ac6b6f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff3ff657700 1 -- 172.21.15.119:0/4265003647 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x555ac6b00a80 con 0x555ac6db6c00 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 4 datascan.main: data pool 'cephfs_data' has ID 14 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 4 datascan.main: opening data pool 'cephfs_data' 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x555ac6db7400 0x555ac6b6e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:1 14.0 14:00000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6db7800 con 0x555ac6db7400 2024-09-13T16:33:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x555ac6e36800 0x555ac6b6f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff404e62700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x555ac6db7400 0x555ac6b6e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.555+0000 7ff404e62700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x555ac6db7400 0x555ac6b6e100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.559+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x555ac6e36800 0x555ac6b6f700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x555ac6b86210 tx=0x555ac78ae000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.559+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== osd_op_reply(1 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6db7400 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.559+0000 7ff400659700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x555ac6e36c00 0x555ac6f2e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.559+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:2 14.20 14:04000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37000 con 0x555ac6e36c00 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.559+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x555ac6e36c00 0x555ac6f2e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.559+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x555ac6e36c00 0x555ac6f2e000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.563+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(2 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.563+0000 7ff400e5a700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x555ac6e37800 0x555ac6f2e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.563+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:3 14.10 14:08000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37c00 con 0x555ac6e37800 2024-09-13T16:33:02.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.563+0000 7ff405663700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x555ac6e37800 0x555ac6f2e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.563+0000 7ff405663700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x555ac6e37800 0x555ac6f2e580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.567+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(3 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.567+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:4 14.30 14:0c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37400 con 0x555ac6e36c00 2024-09-13T16:33:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.567+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(4 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.567+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:5 14.8 14:10000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f36000 con 0x555ac6e36c00 2024-09-13T16:33:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.567+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 3 ==== osd_op_reply(5 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.567+0000 7ff400659700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x555ac6f36000 0x555ac6f2eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.567+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:6 14.28 14:14000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37000 con 0x555ac6f36000 2024-09-13T16:33:02.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff404e62700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x555ac6f36000 0x555ac6f2eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff404e62700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x555ac6f36000 0x555ac6f2eb00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== osd_op_reply(6 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36000 2024-09-13T16:33:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:7 14.18 14:18000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f36400 con 0x555ac6e37800 2024-09-13T16:33:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== osd_op_reply(7 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:8 14.38 14:1c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37400 con 0x555ac6e37800 2024-09-13T16:33:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 3 ==== osd_op_reply(8 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff400e5a700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x555ac6f36800 0x555ac6f2f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:9 14.4 14:20000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e36400 con 0x555ac6f36800 2024-09-13T16:33:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x555ac6f36800 0x555ac6f2f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.571+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x555ac6f36800 0x555ac6f2f600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.575+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(9 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f36800 2024-09-13T16:33:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:10 14.24 14:24000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f36400 con 0x555ac6e37800 2024-09-13T16:33:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 4 ==== osd_op_reply(10 [pgnls start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff400e5a700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x555ac6e37c00 0x555ac6f2fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:11 14.14 14:28000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37400 con 0x555ac6e37c00 2024-09-13T16:33:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff405663700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x555ac6e37c00 0x555ac6f2fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff405663700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x555ac6e37c00 0x555ac6f2fb80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(11 [pgnls start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:12 14.34 14:2c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f36c00 con 0x555ac6db7400 2024-09-13T16:33:02.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== osd_op_reply(12 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6db7400 2024-09-13T16:33:02.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:13 14.c 14:30000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6db7800 con 0x555ac6e36c00 2024-09-13T16:33:02.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 4 ==== osd_op_reply(13 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:14 14.2c 14:34000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37000 con 0x555ac6e37800 2024-09-13T16:33:02.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 5 ==== osd_op_reply(14 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.579+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:15 14.1c 14:38000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37400 con 0x555ac6e36c00 2024-09-13T16:33:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 5 ==== osd_op_reply(15 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:16 14.3c 14:3c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37000 con 0x555ac6db7400 2024-09-13T16:33:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.2 v2:172.21.15.119:6824/3093261484 3 ==== osd_op_reply(16 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6db7400 2024-09-13T16:33:02.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:17 14.2 14:40000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37800 con 0x555ac6f36800 2024-09-13T16:33:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(17 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f36800 2024-09-13T16:33:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:18 14.22 14:44000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37400 con 0x555ac6e37c00 2024-09-13T16:33:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== osd_op_reply(18 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:19 14.12 14:48000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37c00 con 0x555ac6db7400 2024-09-13T16:33:02.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.2 v2:172.21.15.119:6824/3093261484 4 ==== osd_op_reply(19 [pgnls start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6db7400 2024-09-13T16:33:02.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:20 14.32 14:4c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37800 con 0x555ac6e37c00 2024-09-13T16:33:02.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 3 ==== osd_op_reply(20 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff400e5a700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x555ac6f36400 0x555ac6f30100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:21 14.a 14:50000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37000 con 0x555ac6f36400 2024-09-13T16:33:02.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.583+0000 7ff404e62700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x555ac6f36400 0x555ac6f30100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.587 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff404e62700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x555ac6f36400 0x555ac6f30100 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(21 [pgnls start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:22 14.2a 14:54000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6db7800 con 0x555ac6e37c00 2024-09-13T16:33:02.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 4 ==== osd_op_reply(22 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:23 14.1a 14:58000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37400 con 0x555ac6e36c00 2024-09-13T16:33:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 6 ==== osd_op_reply(23 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:24 14.3a 14:5c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e36400 con 0x555ac6e36c00 2024-09-13T16:33:02.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 7 ==== osd_op_reply(24 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:25 14.6 14:60000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37800 con 0x555ac6e36c00 2024-09-13T16:33:02.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 8 ==== osd_op_reply(25 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.587+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:26 14.26 14:64000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f36c00 con 0x555ac6e37c00 2024-09-13T16:33:02.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 5 ==== osd_op_reply(26 [pgnls start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:27 14.16 14:68000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37400 con 0x555ac6f36000 2024-09-13T16:33:02.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== osd_op_reply(27 [pgnls start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36000 2024-09-13T16:33:02.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:28 14.36 14:6c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37c00 con 0x555ac6db7400 2024-09-13T16:33:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.2 v2:172.21.15.119:6824/3093261484 5 ==== osd_op_reply(28 [pgnls start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6db7400 2024-09-13T16:33:02.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:29 14.e 14:70000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37000 con 0x555ac6e37c00 2024-09-13T16:33:02.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 6 ==== osd_op_reply(29 [pgnls start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:30 14.2e 14:74000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52000 con 0x555ac6f36000 2024-09-13T16:33:02.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.3 v2:172.21.15.119:6808/3803418857 3 ==== osd_op_reply(30 [pgnls start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36000 2024-09-13T16:33:02.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff400e5a700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x555ac6f52000 0x555ac6f30680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:31 14.1e 14:78000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53c00 con 0x555ac6f52000 2024-09-13T16:33:02.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.591+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x555ac6f52000 0x555ac6f30680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.595+0000 7ff405e64700 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x555ac6f52000 0x555ac6f30680 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(31 [pgnls start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f52000 2024-09-13T16:33:02.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:32 14.3e 14:7c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52400 con 0x555ac6f36400 2024-09-13T16:33:02.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== osd_op_reply(32 [pgnls start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:33 14.1 14:80000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37000 con 0x555ac6f36000 2024-09-13T16:33:02.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.3 v2:172.21.15.119:6808/3803418857 4 ==== osd_op_reply(33 [pgnls start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36000 2024-09-13T16:33:02.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:34 14.21 14:84000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52800 con 0x555ac6f36400 2024-09-13T16:33:02.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 3 ==== osd_op_reply(34 [pgnls start_epoch 99 out=49b] v95'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.599+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:35 14.11 14:88000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37c00 con 0x555ac6e37c00 2024-09-13T16:33:02.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 7 ==== osd_op_reply(35 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:36 14.31 14:8c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52c00 con 0x555ac6e37800 2024-09-13T16:33:02.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 6 ==== osd_op_reply(36 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:37 14.9 14:90000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37400 con 0x555ac6f36400 2024-09-13T16:33:02.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 4 ==== osd_op_reply(37 [pgnls start_epoch 99 out=49b] v95'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:38 14.29 14:94000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53000 con 0x555ac6f36800 2024-09-13T16:33:02.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(38 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f36800 2024-09-13T16:33:02.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:39 14.19 14:98000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53c00 con 0x555ac6f36800 2024-09-13T16:33:02.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.6 v2:172.21.15.145:6800/3152548227 4 ==== osd_op_reply(39 [pgnls start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f36800 2024-09-13T16:33:02.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.603+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:40 14.39 14:9c000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53400 con 0x555ac6f52000 2024-09-13T16:33:02.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== osd_op_reply(40 [pgnls start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f52000 2024-09-13T16:33:02.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:41 14.5 14:a0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53000 con 0x555ac6e36c00 2024-09-13T16:33:02.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 9 ==== osd_op_reply(41 [pgnls start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:42 14.25 14:a4000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53800 con 0x555ac6db7400 2024-09-13T16:33:02.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.2 v2:172.21.15.119:6824/3093261484 6 ==== osd_op_reply(42 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6db7400 2024-09-13T16:33:02.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:43 14.15 14:a8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37400 con 0x555ac6f52000 2024-09-13T16:33:02.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.7 v2:172.21.15.145:6808/1049327118 3 ==== osd_op_reply(43 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f52000 2024-09-13T16:33:02.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:44 14.35 14:ac000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52400 con 0x555ac6e37c00 2024-09-13T16:33:02.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 8 ==== osd_op_reply(44 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.607+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:45 14.d 14:b0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37800 con 0x555ac6f36800 2024-09-13T16:33:02.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.6 v2:172.21.15.145:6800/3152548227 5 ==== osd_op_reply(45 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f36800 2024-09-13T16:33:02.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:46 14.2d 14:b4000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52800 con 0x555ac6e37c00 2024-09-13T16:33:02.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 9 ==== osd_op_reply(46 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:47 14.1d 14:b8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e36400 con 0x555ac6e36c00 2024-09-13T16:33:02.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 10 ==== osd_op_reply(47 [pgnls start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:48 14.3d 14:bc000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53800 con 0x555ac6f36000 2024-09-13T16:33:02.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.3 v2:172.21.15.119:6808/3803418857 5 ==== osd_op_reply(48 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36000 2024-09-13T16:33:02.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:49 14.3 14:c0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53c00 con 0x555ac6e36c00 2024-09-13T16:33:02.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 11 ==== osd_op_reply(49 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.611+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:50 14.23 14:c4000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37400 con 0x555ac6f52000 2024-09-13T16:33:02.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.7 v2:172.21.15.145:6808/1049327118 4 ==== osd_op_reply(50 [pgnls start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f52000 2024-09-13T16:33:02.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:51 14.13 14:c8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53400 con 0x555ac6db7400 2024-09-13T16:33:02.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.2 v2:172.21.15.119:6824/3093261484 7 ==== osd_op_reply(51 [pgnls start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6db7400 2024-09-13T16:33:02.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:52 14.33 14:cc000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37000 con 0x555ac6e37800 2024-09-13T16:33:02.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 7 ==== osd_op_reply(52 [pgnls start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:53 14.b 14:d0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37400 con 0x555ac6e36c00 2024-09-13T16:33:02.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 12 ==== osd_op_reply(53 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:54 14.2b 14:d4000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f53000 con 0x555ac6e37c00 2024-09-13T16:33:02.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 10 ==== osd_op_reply(54 [pgnls start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.615+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:55 14.1b 14:d8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37800 con 0x555ac6e37800 2024-09-13T16:33:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 8 ==== osd_op_reply(55 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:56 14.3b 14:dc000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52c00 con 0x555ac6e37c00 2024-09-13T16:33:02.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 11 ==== osd_op_reply(56 [pgnls start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:57 14.7 14:e0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f36c00 con 0x555ac6f36400 2024-09-13T16:33:02.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 5 ==== osd_op_reply(57 [pgnls start_epoch 99 out=81b] v95'6 uv6 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:58 14.7 14:e2e07930:::10000000001.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6db7c00 con 0x555ac6f36400 2024-09-13T16:33:02.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 6 ==== osd_op_reply(58 10000000001.00000000 [stat out=16b] v0'0 uv6 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff406ea99c0 10 datascan.operator(): handling object 1099511627777.0 2024-09-13T16:33:02.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.619+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:59 14.7 14:e2e07930:::10000000001.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f64000 con 0x555ac6f36400 2024-09-13T16:33:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.623+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 7 ==== osd_op_reply(59 10000000001.00000000 [call] v99'7 uv7 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.623+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:60 14.27 14:e4000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f64400 con 0x555ac6f36400 2024-09-13T16:33:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.623+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.5 v2:172.21.15.145:6824/4068956621 8 ==== osd_op_reply(60 [pgnls start_epoch 99 out=49b] v95'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36400 2024-09-13T16:33:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.623+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:61 14.17 14:e8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37c00 con 0x555ac6e37800 2024-09-13T16:33:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.1 v2:172.21.15.119:6816/1495422598 9 ==== osd_op_reply(61 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37800 2024-09-13T16:33:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:62 14.37 14:ec000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f52400 con 0x555ac6e37c00 2024-09-13T16:33:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff405663700 1 -- 172.21.15.119:0/4265003647 <== osd.4 v2:172.21.15.145:6816/4125431312 12 ==== osd_op_reply(62 [pgnls start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acfc0 con 0x555ac6e37c00 2024-09-13T16:33:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:63 14.f 14:f0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e36400 con 0x555ac6f52000 2024-09-13T16:33:02.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.7 v2:172.21.15.145:6808/1049327118 5 ==== osd_op_reply(63 [pgnls start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6f52000 2024-09-13T16:33:02.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:64 14.2f 14:f4000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37000 con 0x555ac6e36c00 2024-09-13T16:33:02.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 13 ==== osd_op_reply(64 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff400e5a700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:65 14.1f 14:f8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6e37000 con 0x555ac6e36c00 2024-09-13T16:33:02.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff405e64700 1 -- 172.21.15.119:0/4265003647 <== osd.0 v2:172.21.15.119:6800/2655437631 14 ==== osd_op_reply(65 [pgnls start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acd80 con 0x555ac6e36c00 2024-09-13T16:33:02.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.627+0000 7ff400659700 1 -- 172.21.15.119:0/4265003647 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:66 14.3f 14:fc000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x555ac6f37800 con 0x555ac6f36000 2024-09-13T16:33:02.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 <== osd.3 v2:172.21.15.119:6808/3803418857 6 ==== osd_op_reply(66 [pgnls start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x555ac78acb40 con 0x555ac6f36000 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x555ac6e36c00 msgr2=0x555ac6f2e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x555ac6e36c00 0x555ac6f2e000 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x555ac6e37800 msgr2=0x555ac6f2e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x555ac6e37800 0x555ac6f2e580 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x555ac6db7400 msgr2=0x555ac6b6e100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x555ac6db7400 0x555ac6b6e100 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x555ac6f36000 msgr2=0x555ac6f2eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x555ac6f36000 0x555ac6f2eb00 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 reap_dead start 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x555ac6e37c00 msgr2=0x555ac6f2fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x555ac6e37c00 0x555ac6f2fb80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x555ac6f36400 msgr2=0x555ac6f30100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x555ac6f36400 0x555ac6f30100 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x555ac6f36800 msgr2=0x555ac6f2f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x555ac6f36800 0x555ac6f2f600 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x555ac6f52000 msgr2=0x555ac6f30680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x555ac6f52000 0x555ac6f30680 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x555ac6e36800 msgr2=0x555ac6b6f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff404e62700 1 -- 172.21.15.119:0/4265003647 reap_dead start 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x555ac6e36800 0x555ac6b6f700 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x555ac6b86210 tx=0x555ac78ae000 comp rx=0 tx=0).stop 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db6c00 msgr2=0x555ac6b6ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db6c00 0x555ac6b6ec00 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x555ac78a3950 tx=0x555ac6b7a060 comp rx=0 tx=0).stop 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 shutdown_connections 2024-09-13T16:33:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x555ac6e36800 0x555ac6b6f700 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/4265003647 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6db6c00 0x555ac6b6ec00 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 >> 172.21.15.119:0/4265003647 conn(0x555ac6ac6c00 msgr2=0x555ac6b83800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 shutdown_connections 2024-09-13T16:33:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/4265003647 wait complete. 2024-09-13T16:33:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/2540923950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6ac7800 msgr2=0x555ac6b6c580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6ac7800 0x555ac6b6c580 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x555ac6b87920 tx=0x555ac6b7ac00 comp rx=0 tx=0).stop 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/2540923950 shutdown_connections 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x555ac6ac7000 0x555ac6b6d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x555ac6ac7400 0x555ac6b6cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 --2- 172.21.15.119:0/2540923950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x555ac6ac7800 0x555ac6b6c580 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/2540923950 >> 172.21.15.119:0/2540923950 conn(0x555ac6ac7c00 msgr2=0x555ac6b83000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/2540923950 shutdown_connections 2024-09-13T16:33:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.631+0000 7ff406ea99c0 1 -- 172.21.15.119:0/2540923950 wait complete. 2024-09-13T16:33:02.639 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.135683, result: 2024-09-13T16:33:02.639 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes cephfs_data 2024-09-13T16:33:02.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 -- start start 2024-09-13T16:33:02.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 Processor -- start 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59c911800 0x55c59c9b6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59c911400 0x55c59c9b6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59c911000 0x55c59c9b7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59c992a00 con 0x55c59c911000 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59c992780 con 0x55c59c911800 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c59c992640 con 0x55c59c911400 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.675+0000 7fdd8aea9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59c911000 0x55c59c9b7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8a6a8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59c911800 0x55c59c9b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59c911000 0x55c59c9b7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33226/0 (socket says 172.21.15.119:33226) 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 -- 172.21.15.119:0/678529079 learned_addr learned my addr 172.21.15.119:0/678529079 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59c911400 0x55c59c9b6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 -- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59c911400 msgr2=0x55c59c9b6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59c911400 0x55c59c9b6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 -- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59c911800 msgr2=0x55c59c9b6580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59c911800 0x55c59c9b6580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 -- 172.21.15.119:0/678529079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c59c9349c0 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59c911800 0x55c59c9b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59c911400 0x55c59c9b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59c911000 0x55c59c9b7080 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x55c59c9d1920 tx=0x55c59c9c4840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6f69f71970b721 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd87ea3700 1 -- 172.21.15.119:0/678529079 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c59c961600 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/678529079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c59c934b60 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd87ea3700 1 -- 172.21.15.119:0/678529079 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c59c95c180 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd87ea3700 1 -- 172.21.15.119:0/678529079 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c59c9611e0 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/678529079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55c59c934340 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8beee9c0 4 waiting for MDS map... 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd87ea3700 1 -- 172.21.15.119:0/678529079 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x55c59c94ac40 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd87ea3700 1 -- 172.21.15.119:0/678529079 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 64) v1 ==== 2573+0+0 (secure 0 0 0) 0x55c59d6f0900 con 0x55c59c911000 2024-09-13T16:33:02.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8beee9c0 4 Got MDS map 64 2024-09-13T16:33:02.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8beee9c0 4 datascan.main: Using metadata pool output 2024-09-13T16:33:02.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8beee9c0 4 datascan.main: connecting to RADOS... 2024-09-13T16:33:02.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.679+0000 7fdd8beee9c0 1 Processor -- start 2024-09-13T16:33:02.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- start start 2024-09-13T16:33:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc04c00 0x55c59c9b7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc05000 0x55c59c9b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 0x55c59c9b8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59c947e00 con 0x55c59cc04c00 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59c992780 con 0x55c59cc05400 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c59c992500 con 0x55c59cc05000 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc04c00 0x55c59c9b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc04c00 0x55c59c9b7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33242/0 (socket says 172.21.15.119:33242) 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1815232525 learned_addr learned my addr 172.21.15.119:0/1815232525 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc05000 0x55c59c9b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 0x55c59c9b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc05000 msgr2=0x55c59c9b7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc05000 0x55c59c9b7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 msgr2=0x55c59c9b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 0x55c59c9b8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1815232525 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c59c934000 con 0x55c59cc04c00 2024-09-13T16:33:02.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc04c00 0x55c59c9b7600 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x55c59d6f8690 tx=0x55c59ca762a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f91cf8f255c7df43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1815232525 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c59c961080 con 0x55c59cc04c00 2024-09-13T16:33:02.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1815232525 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c59c95c900 con 0x55c59cc04c00 2024-09-13T16:33:02.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1815232525 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c59c961600 con 0x55c59cc04c00 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1815232525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc04c00 msgr2=0x55c59c9b7600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc04c00 0x55c59c9b7600 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x55c59d6f8690 tx=0x55c59ca762a0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1815232525 shutdown_connections 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 0x55c59c9b8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc05000 0x55c59c9b7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1815232525 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc04c00 0x55c59c9b7600 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1815232525 >> 172.21.15.119:0/1815232525 conn(0x55c59c910c00 msgr2=0x55c59c9cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1815232525 shutdown_connections 2024-09-13T16:33:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.683+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1815232525 wait complete. 2024-09-13T16:33:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 Processor -- start 2024-09-13T16:33:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 -- start start 2024-09-13T16:33:02.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc04c00 0x55c59c9b9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc05000 0x55c59c9b8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 0x55c59c9b8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59c9923c0 con 0x55c59cc05000 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c59c992280 con 0x55c59cc05400 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55c59c992140 con 0x55c59cc04c00 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc05000 0x55c59c9b8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc05000 0x55c59c9b8c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33258/0 (socket says 172.21.15.119:33258) 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 learned_addr learned my addr 172.21.15.119:0/1288211379 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc04c00 0x55c59c9b9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 0x55c59c9b8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc04c00 msgr2=0x55c59c9b9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59cc04c00 0x55c59c9b9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 msgr2=0x55c59c9b8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59cc05400 0x55c59c9b8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c59c9341a0 con 0x55c59cc05000 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc05000 0x55c59c9b8c00 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x55c59d6f9950 tx=0x55c59d6fac00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8469c700 1 -- 172.21.15.119:0/1288211379 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c59c961080 con 0x55c59cc05000 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8469c700 1 -- 172.21.15.119:0/1288211379 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55c59d6fd800 con 0x55c59cc05000 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55c59c934000 con 0x55c59cc05000 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8469c700 1 -- 172.21.15.119:0/1288211379 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55c59c961600 con 0x55c59cc05000 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c59c9349c0 con 0x55c59cc05000 2024-09-13T16:33:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.687+0000 7fdd8beee9c0 4 datascan.init: resolving metadata pool 13 2024-09-13T16:33:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8469c700 1 -- 172.21.15.119:0/1288211379 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x55c59c970840 con 0x55c59cc05000 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8469c700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55c59cc8c800 0x55c59c9b9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8469c700 1 -- 172.21.15.119:0/1288211379 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x55c59c94aa80 con 0x55c59cc05000 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8beee9c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8beee9c0 4 datascan.main: data pool 'cephfs_data' has ID 14 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8beee9c0 4 datascan.main: opening data pool 'cephfs_data' 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55c59cc05800 0x55c59c9b8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:1 13.14 13:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc05c00 con 0x55c59cc05800 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55c59cc8c800 0x55c59c9b9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55c59cc05800 0x55c59c9b8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.691+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55c59cc05800 0x55c59c9b8100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55c59cc8c800 0x55c59c9b9700 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x55c59d6f87e0 tx=0x55c59d6fbec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv23 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55c59cc8cc00 0x55c59c9b7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:2 13.1f 13:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d000 con 0x55c59cc8cc00 2024-09-13T16:33:02.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55c59cc8cc00 0x55c59c9b7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55c59cc8cc00 0x55c59c9b7b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55c59cc8d400 0x55c59c9b7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:3 14.0 14:00000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d800 con 0x55c59cc8d400 2024-09-13T16:33:02.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55c59cc8d400 0x55c59c9b7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55c59cc8d400 0x55c59c9b7600 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.695+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:4 14.20 14:04000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94000 con 0x55c59cc05800 2024-09-13T16:33:02.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(4 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd85e9f700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55c59cd94400 0x55c59cd9a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:5 14.10 14:08000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94800 con 0x55c59cd94400 2024-09-13T16:33:02.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55c59cd94400 0x55c59cd9a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55c59cd94400 0x55c59cd9a000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(5 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:6 14.30 14:0c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94000 con 0x55c59cc05800 2024-09-13T16:33:02.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 3 ==== osd_op_reply(6 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:7 14.8 14:10000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc05800 2024-09-13T16:33:02.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 4 ==== osd_op_reply(7 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd8569e700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55c59cc05c00 0x55c59cd9a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:8 14.28 14:14000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94800 con 0x55c59cc05c00 2024-09-13T16:33:02.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55c59cc05c00 0x55c59cd9a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.699+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55c59cc05c00 0x55c59cd9a580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== osd_op_reply(8 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:9 14.18 14:18000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95800 con 0x55c59cd94400 2024-09-13T16:33:02.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== osd_op_reply(9 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:10 14.38 14:1c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95000 con 0x55c59cd94400 2024-09-13T16:33:02.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 3 ==== osd_op_reply(10 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:11 14.4 14:20000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95c00 con 0x55c59cc8cc00 2024-09-13T16:33:02.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(11 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:12 14.24 14:24000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95400 con 0x55c59cd94400 2024-09-13T16:33:02.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 4 ==== osd_op_reply(12 [pgnls-filter start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd85e9f700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55c59cd94800 0x55c59cd9ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:13 14.14 14:28000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95c00 con 0x55c59cd94800 2024-09-13T16:33:02.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.703+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55c59cd94800 0x55c59cd9ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd8a6a8700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55c59cd94800 0x55c59cd9ab00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(13 [pgnls-filter start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:14 14.34 14:2c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc8d400 2024-09-13T16:33:02.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== osd_op_reply(14 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:15 14.c 14:30000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94000 con 0x55c59cc05800 2024-09-13T16:33:02.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 5 ==== osd_op_reply(15 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:16 14.2c 14:34000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95800 con 0x55c59cd94400 2024-09-13T16:33:02.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 5 ==== osd_op_reply(16 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:17 14.1c 14:38000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d000 con 0x55c59cc05800 2024-09-13T16:33:02.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 6 ==== osd_op_reply(17 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:18 14.3c 14:3c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95000 con 0x55c59cc8d400 2024-09-13T16:33:02.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.707+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 3 ==== osd_op_reply(18 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:19 14.2 14:40000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95c00 con 0x55c59cc8cc00 2024-09-13T16:33:02.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(19 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:20 14.22 14:44000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6000 con 0x55c59cd94800 2024-09-13T16:33:02.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== osd_op_reply(20 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:21 14.12 14:48000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc8d400 2024-09-13T16:33:02.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 4 ==== osd_op_reply(21 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:22 14.32 14:4c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6400 con 0x55c59cd94800 2024-09-13T16:33:02.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 3 ==== osd_op_reply(22 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd85e9f700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55c59cc8c400 0x55c59cd9b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:23 14.a 14:50000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d800 con 0x55c59cc8c400 2024-09-13T16:33:02.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.711+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55c59cc8c400 0x55c59cd9b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd89ea7700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55c59cc8c400 0x55c59cd9b080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(23 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:24 14.2a 14:54000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6800 con 0x55c59cd94800 2024-09-13T16:33:02.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:25 14.1a 14:58000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d000 con 0x55c59cc05800 2024-09-13T16:33:02.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 7 ==== osd_op_reply(25 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:26 14.3a 14:5c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6c00 con 0x55c59cc05800 2024-09-13T16:33:02.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 8 ==== osd_op_reply(26 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.715+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:27 14.6 14:60000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95000 con 0x55c59cc05800 2024-09-13T16:33:02.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 9 ==== osd_op_reply(27 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:28 14.26 14:64000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7000 con 0x55c59cd94800 2024-09-13T16:33:02.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 5 ==== osd_op_reply(28 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:29 14.16 14:68000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc05c00 2024-09-13T16:33:02.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== osd_op_reply(29 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:30 14.36 14:6c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6000 con 0x55c59cc8d400 2024-09-13T16:33:02.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 5 ==== osd_op_reply(30 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:31 14.e 14:70000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95400 con 0x55c59cd94800 2024-09-13T16:33:02.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 6 ==== osd_op_reply(31 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.719+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:32 14.2e 14:74000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6400 con 0x55c59cc05c00 2024-09-13T16:33:02.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 3 ==== osd_op_reply(32 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd85e9f700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c59cd94000 0x55c59cd9b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:33 14.1e 14:78000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95800 con 0x55c59cd94000 2024-09-13T16:33:02.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c59cd94000 0x55c59cd9b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd8aea9700 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c59cd94000 0x55c59cd9b600 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(33 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:34 14.3e 14:7c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6800 con 0x55c59cc8c400 2024-09-13T16:33:02.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== osd_op_reply(34 [pgnls-filter start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:35 14.1 14:80000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6400 con 0x55c59cc05c00 2024-09-13T16:33:02.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 4 ==== osd_op_reply(35 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.723+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:36 14.21 14:84000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7400 con 0x55c59cc8c400 2024-09-13T16:33:02.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 3 ==== osd_op_reply(36 [pgnls-filter start_epoch 99 out=49b] v95'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:37 14.11 14:88000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7c00 con 0x55c59cd94800 2024-09-13T16:33:02.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 7 ==== osd_op_reply(37 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:38 14.31 14:8c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7800 con 0x55c59cd94400 2024-09-13T16:33:02.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 6 ==== osd_op_reply(38 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:39 14.9 14:90000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95c00 con 0x55c59cc8c400 2024-09-13T16:33:02.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 4 ==== osd_op_reply(39 [pgnls-filter start_epoch 99 out=49b] v95'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:40 14.29 14:94000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d800 con 0x55c59cc8cc00 2024-09-13T16:33:02.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 4 ==== osd_op_reply(40 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.727+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:41 14.19 14:98000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc8cc00 2024-09-13T16:33:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 5 ==== osd_op_reply(41 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:42 14.39 14:9c000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6c00 con 0x55c59cd94000 2024-09-13T16:33:02.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== osd_op_reply(42 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:43 14.5 14:a0000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95800 con 0x55c59cc05800 2024-09-13T16:33:02.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 10 ==== osd_op_reply(43 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:44 14.25 14:a4000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6800 con 0x55c59cc8d400 2024-09-13T16:33:02.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 6 ==== osd_op_reply(44 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:45 14.15 14:a8000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb2000 con 0x55c59cd94000 2024-09-13T16:33:02.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 3 ==== osd_op_reply(45 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.731+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:46 14.35 14:ac000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7400 con 0x55c59cd94800 2024-09-13T16:33:02.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 8 ==== osd_op_reply(46 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:47 14.d 14:b0000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb2400 con 0x55c59cc8cc00 2024-09-13T16:33:02.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 6 ==== osd_op_reply(47 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:48 14.2d 14:b4000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7800 con 0x55c59cd94800 2024-09-13T16:33:02.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 9 ==== osd_op_reply(48 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:49 14.1d 14:b8000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb2800 con 0x55c59cc05800 2024-09-13T16:33:02.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 11 ==== osd_op_reply(49 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:50 14.3d 14:bc000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95000 con 0x55c59cc05c00 2024-09-13T16:33:02.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 5 ==== osd_op_reply(50 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.735+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:51 14.3 14:c0000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb2c00 con 0x55c59cc05800 2024-09-13T16:33:02.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 12 ==== osd_op_reply(51 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:52 14.23 14:c4000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d000 con 0x55c59cd94000 2024-09-13T16:33:02.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 4 ==== osd_op_reply(52 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:53 14.13 14:c8000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb3000 con 0x55c59cc8d400 2024-09-13T16:33:02.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 7 ==== osd_op_reply(53 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:54 14.33 14:cc000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7000 con 0x55c59cd94400 2024-09-13T16:33:02.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 7 ==== osd_op_reply(54 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:55 14.b 14:d0000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb3400 con 0x55c59cc05800 2024-09-13T16:33:02.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 13 ==== osd_op_reply(55 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.739+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:56 14.2b 14:d4000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6000 con 0x55c59cd94800 2024-09-13T16:33:02.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 10 ==== osd_op_reply(56 [pgnls-filter start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:57 14.1b 14:d8000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb3800 con 0x55c59cd94400 2024-09-13T16:33:02.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 8 ==== osd_op_reply(57 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:58 14.3b 14:dc000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7c00 con 0x55c59cd94800 2024-09-13T16:33:02.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 11 ==== osd_op_reply(58 [pgnls-filter start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:59 14.7 14:e0000000::::head [pgnls-filter start_epoch 99 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb3c00 con 0x55c59cc8c400 2024-09-13T16:33:02.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 5 ==== osd_op_reply(59 [pgnls-filter start_epoch 99 out=81b] v99'7 uv7 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:60 14.0 14:00000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8dc00 con 0x55c59cc8d400 2024-09-13T16:33:02.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 8 ==== osd_op_reply(60 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.743+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:61 14.20 14:04000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb2800 con 0x55c59cc05800 2024-09-13T16:33:02.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 14 ==== osd_op_reply(61 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:62 14.10 14:08000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6800 con 0x55c59cd94400 2024-09-13T16:33:02.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 9 ==== osd_op_reply(62 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:63 14.30 14:0c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb2c00 con 0x55c59cc05800 2024-09-13T16:33:02.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 15 ==== osd_op_reply(63 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:64 14.8 14:10000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7400 con 0x55c59cc05800 2024-09-13T16:33:02.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 16 ==== osd_op_reply(64 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:65 14.28 14:14000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb3400 con 0x55c59cc05c00 2024-09-13T16:33:02.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:66 14.18 14:18000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95400 con 0x55c59cd94400 2024-09-13T16:33:02.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 10 ==== osd_op_reply(66 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.747+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:67 14.38 14:1c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdb3800 con 0x55c59cd94400 2024-09-13T16:33:02.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 11 ==== osd_op_reply(67 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:68 14.4 14:20000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc8cc00 2024-09-13T16:33:02.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 7 ==== osd_op_reply(68 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:69 14.24 14:24000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7000 con 0x55c59cd94400 2024-09-13T16:33:02.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 12 ==== osd_op_reply(69 [pgnls-filter start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:70 14.14 14:28000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d800 con 0x55c59cd94800 2024-09-13T16:33:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 12 ==== osd_op_reply(70 [pgnls-filter start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:71 14.34 14:2c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6400 con 0x55c59cc8d400 2024-09-13T16:33:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 9 ==== osd_op_reply(71 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:72 14.c 14:30000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7800 con 0x55c59cc05800 2024-09-13T16:33:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 17 ==== osd_op_reply(72 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.751+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:73 14.2c 14:34000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6c00 con 0x55c59cd94400 2024-09-13T16:33:02.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 13 ==== osd_op_reply(73 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:74 14.1c 14:38000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6000 con 0x55c59cc05800 2024-09-13T16:33:02.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 18 ==== osd_op_reply(74 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:75 14.3c 14:3c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95c00 con 0x55c59cc8d400 2024-09-13T16:33:02.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 10 ==== osd_op_reply(75 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:76 14.2 14:40000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7c00 con 0x55c59cc8cc00 2024-09-13T16:33:02.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 8 ==== osd_op_reply(76 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:77 14.22 14:44000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95800 con 0x55c59cd94800 2024-09-13T16:33:02.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 13 ==== osd_op_reply(77 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.755+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:78 14.12 14:48000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6400 con 0x55c59cc8d400 2024-09-13T16:33:02.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 11 ==== osd_op_reply(78 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:79 14.32 14:4c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7c00 con 0x55c59cd94800 2024-09-13T16:33:02.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 14 ==== osd_op_reply(79 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:80 14.a 14:50000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6800 con 0x55c59cc8c400 2024-09-13T16:33:02.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 6 ==== osd_op_reply(80 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:81 14.2a 14:54000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7000 con 0x55c59cd94800 2024-09-13T16:33:02.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 15 ==== osd_op_reply(81 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:82 14.1a 14:58000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7400 con 0x55c59cc05800 2024-09-13T16:33:02.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 19 ==== osd_op_reply(82 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:83 14.3a 14:5c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95000 con 0x55c59cc05800 2024-09-13T16:33:02.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 20 ==== osd_op_reply(83 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.759+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:84 14.6 14:60000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95400 con 0x55c59cc05800 2024-09-13T16:33:02.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 21 ==== osd_op_reply(84 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:85 14.26 14:64000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cd94800 2024-09-13T16:33:02.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 16 ==== osd_op_reply(85 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:86 14.16 14:68000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8dc00 con 0x55c59cc05c00 2024-09-13T16:33:02.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 7 ==== osd_op_reply(86 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:87 14.36 14:6c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d000 con 0x55c59cc8d400 2024-09-13T16:33:02.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 12 ==== osd_op_reply(87 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:88 14.e 14:70000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d800 con 0x55c59cd94800 2024-09-13T16:33:02.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 17 ==== osd_op_reply(88 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:89 14.2e 14:74000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d000 con 0x55c59cc05c00 2024-09-13T16:33:02.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 8 ==== osd_op_reply(89 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.763+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:90 14.1e 14:78000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc0000 con 0x55c59cd94000 2024-09-13T16:33:02.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 5 ==== osd_op_reply(90 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:91 14.3e 14:7c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d800 con 0x55c59cc8c400 2024-09-13T16:33:02.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 7 ==== osd_op_reply(91 [pgnls-filter start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:92 14.1 14:80000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc0400 con 0x55c59cc05c00 2024-09-13T16:33:02.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 9 ==== osd_op_reply(92 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:93 14.21 14:84000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95c00 con 0x55c59cc8c400 2024-09-13T16:33:02.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 8 ==== osd_op_reply(93 [pgnls-filter start_epoch 99 out=49b] v95'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:94 14.11 14:88000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc0800 con 0x55c59cd94800 2024-09-13T16:33:02.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 18 ==== osd_op_reply(94 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.767+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:95 14.31 14:8c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95800 con 0x55c59cd94400 2024-09-13T16:33:02.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 14 ==== osd_op_reply(95 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:96 14.9 14:90000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc0c00 con 0x55c59cc8c400 2024-09-13T16:33:02.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 9 ==== osd_op_reply(96 [pgnls-filter start_epoch 99 out=49b] v95'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:97 14.29 14:94000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc8cc00 2024-09-13T16:33:02.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 9 ==== osd_op_reply(97 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.772 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:98 14.19 14:98000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc1000 con 0x55c59cc8cc00 2024-09-13T16:33:02.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 10 ==== osd_op_reply(98 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:99 14.39 14:9c000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6400 con 0x55c59cd94000 2024-09-13T16:33:02.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 6 ==== osd_op_reply(99 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:100 14.5 14:a0000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc1400 con 0x55c59cc05800 2024-09-13T16:33:02.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.771+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 22 ==== osd_op_reply(100 [pgnls-filter start_epoch 99 out=49b] v95'5 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:101 14.25 14:a4000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7c00 con 0x55c59cc8d400 2024-09-13T16:33:02.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 13 ==== osd_op_reply(101 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:102 14.15 14:a8000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc1800 con 0x55c59cd94000 2024-09-13T16:33:02.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 7 ==== osd_op_reply(102 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:103 14.35 14:ac000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7000 con 0x55c59cd94800 2024-09-13T16:33:02.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 19 ==== osd_op_reply(103 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:104 14.d 14:b0000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc1c00 con 0x55c59cc8cc00 2024-09-13T16:33:02.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 11 ==== osd_op_reply(104 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.775+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:105 14.2d 14:b4000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8dc00 con 0x55c59cd94800 2024-09-13T16:33:02.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 20 ==== osd_op_reply(105 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:106 14.1d 14:b8000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd94c00 con 0x55c59cc05800 2024-09-13T16:33:02.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 23 ==== osd_op_reply(106 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:107 14.3d 14:bc000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d000 con 0x55c59cc05c00 2024-09-13T16:33:02.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 10 ==== osd_op_reply(107 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:108 14.3 14:c0000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95000 con 0x55c59cc05800 2024-09-13T16:33:02.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 24 ==== osd_op_reply(108 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:109 14.23 14:c4000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cc8d800 con 0x55c59cd94000 2024-09-13T16:33:02.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 8 ==== osd_op_reply(109 [pgnls-filter start_epoch 99 out=49b] v95'7 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:110 14.13 14:c8000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95400 con 0x55c59cc8d400 2024-09-13T16:33:02.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.779+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.2 v2:172.21.15.119:6824/3093261484 14 ==== osd_op_reply(110 [pgnls-filter start_epoch 99 out=49b] v95'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cc8d400 2024-09-13T16:33:02.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:111 14.33 14:cc000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6400 con 0x55c59cd94400 2024-09-13T16:33:02.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 15 ==== osd_op_reply(111 [pgnls-filter start_epoch 99 out=49b] v95'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:112 14.b 14:d0000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cd95c00 con 0x55c59cc05800 2024-09-13T16:33:02.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 25 ==== osd_op_reply(112 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:113 14.2b 14:d4000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7800 con 0x55c59cd94800 2024-09-13T16:33:02.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 21 ==== osd_op_reply(113 [pgnls-filter start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:114 14.1b 14:d8000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc6000 con 0x55c59cd94400 2024-09-13T16:33:02.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 16 ==== osd_op_reply(114 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:115 14.3b 14:dc000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6c00 con 0x55c59cd94800 2024-09-13T16:33:02.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 22 ==== osd_op_reply(115 [pgnls-filter start_epoch 99 out=49b] v95'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.783+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:116 14.7 14:e0000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc6400 con 0x55c59cc8c400 2024-09-13T16:33:02.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 10 ==== osd_op_reply(116 [pgnls-filter start_epoch 99 out=81b] v99'7 uv7 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 20 datascan.forall_objects: OSD matched oid 10000000001.00000000 2024-09-13T16:33:02.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 10 datascan.operator(): handling object 10000000001.0 2024-09-13T16:33:02.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:117 14.7 14:e2e07930:::10000000001.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdcc000 con 0x55c59cc8c400 2024-09-13T16:33:02.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 11 ==== osd_op_reply(117 10000000001.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (89) out=89b,getxattr (30) out=30b,getxattr] v0'0 uv7 ondisk = 0) v8 ==== 416+0+157 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 10 datascan.inject_with_backtrace: inode: 0x0x10000000001 2024-09-13T16:33:02.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 10 datascan.inject_with_backtrace: backptr: 0x0x10000000000/file 2024-09-13T16:33:02.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 20 datascan.get_frag_of: dirino=0x10000000000 target_dname=file 2024-09-13T16:33:02.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:118 13.10 13:0cf6ad0f:::10000000000.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdcc400 con 0x55c59cd94400 2024-09-13T16:33:02.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 17 ==== osd_op_reply(118 10000000000.00000000 [getxattr (59) out=59b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+59 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 20 datascan.get_frag_of: got backtrace parent 0x1/dir 2024-09-13T16:33:02.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:119 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals-by-keys in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdcc800 con 0x55c59cc8cc00 2024-09-13T16:33:02.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.6 v2:172.21.15.145:6800/3152548227 12 ==== osd_op_reply(119 1.00000000 [omap-get-vals-by-keys out=520b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+520 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc8cc00 2024-09-13T16:33:02.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 20 datascan.read_dentry: oid=1.00000000 dname=dir frag=*, r=0 2024-09-13T16:33:02.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 20 datascan.get_frag_of: fast path, fragtree is fragtree_t() 2024-09-13T16:33:02.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 20 datascan.get_frag_of: frag is * 2024-09-13T16:33:02.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.787+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:120 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdccc00 con 0x55c59cd94400 2024-09-13T16:33:02.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 18 ==== osd_op_reply(120 10000000000.00000000 [omap-get-header out=274b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 10 datascan.find_or_create_dirfrag: read_version = 17 2024-09-13T16:33:02.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000000 * 2024-09-13T16:33:02.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:121 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdcd000 con 0x55c59cd94400 2024-09-13T16:33:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 19 ==== osd_op_reply(121 10000000000.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-09-13T16:33:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 20 datascan.read_dentry: file_head not found in result 2024-09-13T16:33:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 10 datascan.inject_with_backtrace: Linking inode 0x0x10000000001 at 0x0x10000000000/file with size=0 bytes 2024-09-13T16:33:02.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:122 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdcd400 con 0x55c59cd94400 2024-09-13T16:33:02.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 20 ==== osd_op_reply(122 10000000000.00000000 [omap-set-vals] v99'18 uv18 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.791+0000 7fdd8beee9c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-09-13T16:33:02.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:123 14.27 14:e4000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdcd800 con 0x55c59cc8c400 2024-09-13T16:33:02.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.5 v2:172.21.15.145:6824/4068956621 12 ==== osd_op_reply(123 [pgnls-filter start_epoch 99 out=49b] v95'2 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc8c400 2024-09-13T16:33:02.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:124 14.17 14:e8000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc6800 con 0x55c59cd94400 2024-09-13T16:33:02.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.1 v2:172.21.15.119:6816/1495422598 21 ==== osd_op_reply(124 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cd94400 2024-09-13T16:33:02.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:125 14.37 14:ec000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6000 con 0x55c59cd94800 2024-09-13T16:33:02.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 <== osd.4 v2:172.21.15.145:6816/4125431312 23 ==== osd_op_reply(125 [pgnls-filter start_epoch 99 out=49b] v95'11 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6fc0 con 0x55c59cd94800 2024-09-13T16:33:02.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:126 14.f 14:f0000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc6c00 con 0x55c59cd94000 2024-09-13T16:33:02.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.7 v2:172.21.15.145:6808/1049327118 9 ==== osd_op_reply(126 [pgnls-filter start_epoch 99 out=49b] v95'9 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cd94000 2024-09-13T16:33:02.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.795+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:127 14.2f 14:f4000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda6800 con 0x55c59cc05800 2024-09-13T16:33:02.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 26 ==== osd_op_reply(127 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8569e700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:128 14.1f 14:f8000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cdc7000 con 0x55c59cc05800 2024-09-13T16:33:02.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd89ea7700 1 -- 172.21.15.119:0/1288211379 <== osd.0 v2:172.21.15.119:6800/2655437631 27 ==== osd_op_reply(128 [pgnls-filter start_epoch 99 out=49b] v95'6 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6b40 con 0x55c59cc05800 2024-09-13T16:33:02.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd85e9f700 1 -- 172.21.15.119:0/1288211379 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:129 14.3f 14:fc000000::::head [pgnls-filter start_epoch 99 in=85b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55c59cda7400 con 0x55c59cc05c00 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8aea9700 1 -- 172.21.15.119:0/1288211379 <== osd.3 v2:172.21.15.119:6808/3803418857 11 ==== osd_op_reply(129 [pgnls-filter start_epoch 99 out=49b] v95'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55c59d6f6d80 con 0x55c59cc05c00 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55c59cc05800 msgr2=0x55c59c9b8100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55c59cc05800 0x55c59c9b8100 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55c59cd94400 msgr2=0x55c59cd9a000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55c59cd94400 0x55c59cd9a000 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55c59cc8d400 msgr2=0x55c59c9b7600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55c59cc8d400 0x55c59c9b7600 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55c59cc05c00 msgr2=0x55c59cd9a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55c59cc05c00 0x55c59cd9a580 crc :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 reap_dead start 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55c59cd94800 msgr2=0x55c59cd9ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55c59cd94800 0x55c59cd9ab00 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55c59cc8c400 msgr2=0x55c59cd9b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55c59cc8c400 0x55c59cd9b080 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 reap_dead start 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55c59cc8cc00 msgr2=0x55c59c9b7b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55c59cc8cc00 0x55c59c9b7b80 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c59cd94000 msgr2=0x55c59cd9b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55c59cd94000 0x55c59cd9b600 crc :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55c59cc8c800 msgr2=0x55c59c9b9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55c59cc8c800 0x55c59c9b9700 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x55c59d6f87e0 tx=0x55c59d6fbec0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc05000 msgr2=0x55c59c9b8c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/1288211379 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59cc05000 0x55c59c9b8c00 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x55c59d6f9950 tx=0x55c59d6fac00 comp rx=0 tx=0).stop 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8a6a8700 1 -- 172.21.15.119:0/1288211379 reap_dead start 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 shutdown_connections 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 >> 172.21.15.119:0/1288211379 conn(0x55c59c910c00 msgr2=0x55c59c9cd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 shutdown_connections 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/1288211379 wait complete. 2024-09-13T16:33:02.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/678529079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59c911000 msgr2=0x55c59c9b7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59c911000 0x55c59c9b7080 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 crypto rx=0x55c59c9d1920 tx=0x55c59c9c4840 comp rx=0 tx=0).stop 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/678529079 shutdown_connections 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55c59c911000 0x55c59c9b7080 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55c59c911400 0x55c59c9b6b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 --2- 172.21.15.119:0/678529079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55c59c911800 0x55c59c9b6580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.799+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/678529079 >> 172.21.15.119:0/678529079 conn(0x55c59c911c00 msgr2=0x55c59c9cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.803+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/678529079 shutdown_connections 2024-09-13T16:33:02.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.803+0000 7fdd8beee9c0 1 -- 172.21.15.119:0/678529079 wait complete. 2024-09-13T16:33:02.808 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', 'cephfs_data'] in time 0:00:00.169224, result: 2024-09-13T16:33:02.809 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-09-13T16:33:02.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.847+0000 7fa0871429c0 1 -- start start 2024-09-13T16:33:02.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.847+0000 7fa0871429c0 1 Processor -- start 2024-09-13T16:33:02.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdaf2f800 0x55bfdafd4580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdaf2f400 0x55bfdafd4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdaf2f000 0x55bfdafd5080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdaf65e00 con 0x55bfdaf2f400 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb0a00 con 0x55bfdaf2f800 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb0640 con 0x55bfdaf2f000 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdaf2f400 0x55bfdafd4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdaf2f400 0x55bfdafd4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33266/0 (socket says 172.21.15.119:33266) 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 -- 172.21.15.119:0/1936674818 learned_addr learned my addr 172.21.15.119:0/1936674818 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0858fc700 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdaf2f800 0x55bfdafd4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0860fd700 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdaf2f000 0x55bfdafd5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 -- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdaf2f000 msgr2=0x55bfdafd5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdaf2f000 0x55bfdafd5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 -- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdaf2f800 msgr2=0x55bfdafd4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdaf2f800 0x55bfdafd4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 -- 172.21.15.119:0/1936674818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfdaf529c0 con 0x55bfdaf2f400 2024-09-13T16:33:02.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0860fd700 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdaf2f000 0x55bfdafd5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:02.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0850fb700 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdaf2f400 0x55bfdafd4b00 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x55bfdafef800 tx=0x55bfdb094000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ff96476fca6d99 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0830f7700 1 -- 172.21.15.119:0/1936674818 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55bfdaf7f080 con 0x55bfdaf2f400 2024-09-13T16:33:02.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0830f7700 1 -- 172.21.15.119:0/1936674818 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bfdaf7a480 con 0x55bfdaf2f400 2024-09-13T16:33:02.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0830f7700 1 -- 172.21.15.119:0/1936674818 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55bfdaf7f600 con 0x55bfdaf2f400 2024-09-13T16:33:02.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 -- 172.21.15.119:0/1936674818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfdaf521a0 con 0x55bfdaf2f400 2024-09-13T16:33:02.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 1 -- 172.21.15.119:0/1936674818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55bfdaf529c0 con 0x55bfdaf2f400 2024-09-13T16:33:02.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 4 waiting for MDS map... 2024-09-13T16:33:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0830f7700 1 -- 172.21.15.119:0/1936674818 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x55bfdaf68c40 con 0x55bfdaf2f400 2024-09-13T16:33:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0830f7700 1 -- 172.21.15.119:0/1936674818 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 64) v1 ==== 2573+0+0 (secure 0 0 0) 0x55bfdaf70f00 con 0x55bfdaf2f400 2024-09-13T16:33:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.851+0000 7fa0871429c0 4 Got MDS map 64 2024-09-13T16:33:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 4 datascan.main: Using metadata pool output 2024-09-13T16:33:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 4 datascan.main: connecting to RADOS... 2024-09-13T16:33:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 Processor -- start 2024-09-13T16:33:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 -- start start 2024-09-13T16:33:02.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8800 0x55bfdafd5b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb0500 con 0x55bfdbdb8c00 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb03c0 con 0x55bfdbdb8800 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0871429c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb0640 con 0x55bfdbdb8400 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0850fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0850fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33282/0 (socket says 172.21.15.119:33282) 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0850fb700 1 -- 172.21.15.119:0/1090529794 learned_addr learned my addr 172.21.15.119:0/1090529794 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0858fc700 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8800 0x55bfdafd5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.855+0000 7fa0860fd700 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0850fb700 1 -- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 msgr2=0x55bfdafd5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0850fb700 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0850fb700 1 -- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8800 msgr2=0x55bfdafd5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0850fb700 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8800 0x55bfdafd5b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0850fb700 1 -- 172.21.15.119:0/1090529794 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfdaf52b60 con 0x55bfdbdb8c00 2024-09-13T16:33:02.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0860fd700 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:02.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0850fb700 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6100 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x55bfdb240570 tx=0x55bfdb094060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9778ab6d089cbb6c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0810f3700 1 -- 172.21.15.119:0/1090529794 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55bfdaf7f1e0 con 0x55bfdbdb8c00 2024-09-13T16:33:02.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0810f3700 1 -- 172.21.15.119:0/1090529794 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bfdbd12180 con 0x55bfdbdb8c00 2024-09-13T16:33:02.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0810f3700 1 -- 172.21.15.119:0/1090529794 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55bfdaf7ef20 con 0x55bfdbdb8c00 2024-09-13T16:33:02.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- 172.21.15.119:0/1090529794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8c00 msgr2=0x55bfdafd6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6100 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x55bfdb240570 tx=0x55bfdb094060 comp rx=0 tx=0).stop 2024-09-13T16:33:02.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- 172.21.15.119:0/1090529794 shutdown_connections 2024-09-13T16:33:02.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6100 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8800 0x55bfdafd5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1090529794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- 172.21.15.119:0/1090529794 >> 172.21.15.119:0/1090529794 conn(0x55bfdaf2ec00 msgr2=0x55bfdafeb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- 172.21.15.119:0/1090529794 shutdown_connections 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- 172.21.15.119:0/1090529794 wait complete. 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 Processor -- start 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- start start 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd7180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8800 0x55bfdafd6c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb0280 con 0x55bfdbdb8800 2024-09-13T16:33:02.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb0140 con 0x55bfdbdb8c00 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0871429c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55bfdafb0000 con 0x55bfdbdb8400 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8800 0x55bfdafd6c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8800 0x55bfdafd6c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33296/0 (socket says 172.21.15.119:33296) 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 learned_addr learned my addr 172.21.15.119:0/2248403941 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 msgr2=0x55bfdafd7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdbdb8400 0x55bfdafd7180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8c00 msgr2=0x55bfdafd6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdbdb8c00 0x55bfdafd6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bfdaf521a0 con 0x55bfdbdb8800 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.859+0000 7fa0858fc700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8800 0x55bfdafd6c00 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x55bfdb2418c0 tx=0x55bfdb244e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa07f8f0700 1 -- 172.21.15.119:0/2248403941 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55bfdaf7f1e0 con 0x55bfdbdb8800 2024-09-13T16:33:02.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa07f8f0700 1 -- 172.21.15.119:0/2248403941 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55bfdbd13680 con 0x55bfdbdb8800 2024-09-13T16:33:02.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55bfdaf529c0 con 0x55bfdbdb8800 2024-09-13T16:33:02.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa07f8f0700 1 -- 172.21.15.119:0/2248403941 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55bfdaf7ef20 con 0x55bfdbdb8800 2024-09-13T16:33:02.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bfdaf52b60 con 0x55bfdbdb8800 2024-09-13T16:33:02.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0871429c0 4 datascan.init: resolving metadata pool 13 2024-09-13T16:33:02.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa07f8f0700 1 -- 172.21.15.119:0/2248403941 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x55bfdaf8e840 con 0x55bfdbdb8800 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa07f8f0700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55bfdb396000 0x55bfdafd7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa07f8f0700 1 -- 172.21.15.119:0/2248403941 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x55bfdaf68a80 con 0x55bfdbdb8800 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0871429c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0871429c0 4 datascan.main: resolving metadata pool 13 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55bfdbdb9000 0x55bfdafd6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:1 13.0 13:00000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdbdb9400 con 0x55bfdbdb9000 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55bfdb396000 0x55bfdafd7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55bfdbdb9000 0x55bfdafd6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.863+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55bfdbdb9000 0x55bfdafd6100 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55bfdb396000 0x55bfdafd7700 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55bfdafeee70 tx=0x55bfdb398480 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(1 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bfdbdb9400 0x55bfdb3a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:2 13.10 13:08000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb396400 con 0x55bfdbdb9400 2024-09-13T16:33:02.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bfdbdb9400 0x55bfdb3a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bfdbdb9400 0x55bfdb3a0000 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(2 [pgnls start_epoch 99 out=177b] v99'18 uv18 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55bfdb396400 0x55bfdb3a0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:3 13.8 13:10000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb396c00 con 0x55bfdb396400 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0858fc700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55bfdb396400 0x55bfdb3a0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0858fc700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55bfdb396400 0x55bfdb3a0580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== osd_op_reply(3 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb396400 2024-09-13T16:33:02.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:4 13.18 13:18000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb396c00 con 0x55bfdbdb9000 2024-09-13T16:33:02.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== osd_op_reply(4 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55bfdb396c00 0x55bfdb3a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.867+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:5 13.4 13:20000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb396800 con 0x55bfdb396c00 2024-09-13T16:33:02.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55bfdb396c00 0x55bfdb3a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55bfdb396c00 0x55bfdb3a0b00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(5 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55bfdb396800 0x55bfdb3a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:6 13.14 13:28000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397000 con 0x55bfdb396800 2024-09-13T16:33:02.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55bfdb396800 0x55bfdb3a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55bfdb396800 0x55bfdb3a1080 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(6 [pgnls start_epoch 99 out=149b] v98'26 uv26 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:7 13.c 13:30000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397000 con 0x55bfdbdb9400 2024-09-13T16:33:02.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== osd_op_reply(7 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.871+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:8 13.1c 13:38000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397000 con 0x55bfdb396c00 2024-09-13T16:33:02.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(8 [pgnls start_epoch 99 out=73b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55bfdb397000 0x55bfdb3a1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:9 13.2 13:40000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397400 con 0x55bfdb397000 2024-09-13T16:33:02.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0858fc700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55bfdb397000 0x55bfdb3a1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0858fc700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55bfdb397000 0x55bfdb3a1600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(9 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:10 13.12 13:48000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397400 con 0x55bfdb396800 2024-09-13T16:33:02.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(10 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:11 13.a 13:50000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397400 con 0x55bfdb396400 2024-09-13T16:33:02.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== osd_op_reply(11 [pgnls start_epoch 99 out=81b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb396400 2024-09-13T16:33:02.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:12 13.1a 13:58000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397400 con 0x55bfdbdb9400 2024-09-13T16:33:02.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 3 ==== osd_op_reply(12 [pgnls start_epoch 99 out=49b] v96'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55bfdb397400 0x55bfdb3a2680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.875+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:13 13.6 13:60000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397c00 con 0x55bfdb397400 2024-09-13T16:33:02.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55bfdb397400 0x55bfdb3a2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55bfdb397400 0x55bfdb3a2680 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(13 [pgnls start_epoch 99 out=77b] v98'7 uv7 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0850fb700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55bfdb397c00 0x55bfdb3a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:02.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:14 13.16 13:68000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397c00 2024-09-13T16:33:02.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55bfdb397c00 0x55bfdb3a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:02.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0860fd700 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55bfdb397c00 0x55bfdb3a1b80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== osd_op_reply(14 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb397c00 2024-09-13T16:33:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:15 13.e 13:70000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdbdb9000 2024-09-13T16:33:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 3 ==== osd_op_reply(15 [pgnls start_epoch 99 out=73b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.879+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:16 13.1e 13:78000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397400 2024-09-13T16:33:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== osd_op_reply(16 [pgnls start_epoch 99 out=49b] v96'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:17 13.1 13:80000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb396c00 2024-09-13T16:33:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(17 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:18 13.11 13:88000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdbdb9400 2024-09-13T16:33:02.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 4 ==== osd_op_reply(18 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:19 13.9 13:90000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397000 2024-09-13T16:33:02.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== osd_op_reply(19 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:20 13.19 13:98000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397c00 2024-09-13T16:33:02.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== osd_op_reply(20 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb397c00 2024-09-13T16:33:02.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.883+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:21 13.5 13:a0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397400 2024-09-13T16:33:02.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 3 ==== osd_op_reply(21 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:22 13.15 13:a8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397400 2024-09-13T16:33:02.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 4 ==== osd_op_reply(22 [pgnls start_epoch 99 out=97b] v98'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:23 13.d 13:b0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397400 2024-09-13T16:33:02.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 5 ==== osd_op_reply(23 [pgnls start_epoch 99 out=74b] v94'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:24 13.1d 13:b8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdbdb9000 2024-09-13T16:33:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 4 ==== osd_op_reply(24 [pgnls start_epoch 99 out=73b] v98'5 uv5 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:25 13.3 13:c0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb396800 2024-09-13T16:33:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 3 ==== osd_op_reply(25 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:26 13.13 13:c8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdbdb9000 2024-09-13T16:33:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 5 ==== osd_op_reply(26 [pgnls start_epoch 99 out=98b] v98'4 uv4 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.887+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:27 13.b 13:d0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397000 2024-09-13T16:33:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 3 ==== osd_op_reply(27 [pgnls start_epoch 99 out=76b] v98'6 uv6 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:28 13.1b 13:d8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397400 2024-09-13T16:33:02.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 6 ==== osd_op_reply(28 [pgnls start_epoch 99 out=73b] v98'10 uv10 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:29 13.7 13:e0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdbdb9000 2024-09-13T16:33:02.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 6 ==== osd_op_reply(29 [pgnls start_epoch 99 out=73b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:30 13.17 13:e8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb397000 2024-09-13T16:33:02.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 4 ==== osd_op_reply(30 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.891+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:31 13.f 13:f0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb396c00 2024-09-13T16:33:02.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 4 ==== osd_op_reply(31 [pgnls start_epoch 99 out=73b] v94'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:32 13.1f 13:f8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdb396c00 2024-09-13T16:33:02.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 5 ==== osd_op_reply(32 [pgnls start_epoch 99 out=101b] v98'9 uv9 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 601.00000000 2024-09-13T16:33:02.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:33 13.10 13:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdbdb9800 con 0x55bfdbdb9400 2024-09-13T16:33:02.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 5 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 602.00000000 2024-09-13T16:33:02.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:34 13.10 13:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9400 2024-09-13T16:33:02.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 6 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-09-13T16:33:02.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:35 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4400 con 0x55bfdbdb9400 2024-09-13T16:33:02.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 7 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 20 datascan.scan_links: injected ino detected 2024-09-13T16:33:02.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 600.00000000 2024-09-13T16:33:02.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:36 13.10 13:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4800 con 0x55bfdbdb9400 2024-09-13T16:33:02.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 8 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 603.00000000 2024-09-13T16:33:02.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.895+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:37 13.10 13:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4c00 con 0x55bfdbdb9400 2024-09-13T16:33:02.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 9 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-09-13T16:33:02.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-09-13T16:33:02.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 200.00000000 2024-09-13T16:33:02.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 200.00000001 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 606.00000000 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:38 13.14 13:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5000 con 0x55bfdb396800 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 4 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 607.00000000 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:39 13.1c 13:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5400 con 0x55bfdb396c00 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 6 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 100000001f6.00000000 2024-09-13T16:33:02.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:40 13.a 13:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5800 con 0x55bfdb396400 2024-09-13T16:33:02.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.2 v2:172.21.15.119:6824/3093261484 3 ==== osd_op_reply(40 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+54 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb396400 2024-09-13T16:33:02.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-09-13T16:33:02.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-09-13T16:33:02.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 608.00000000 2024-09-13T16:33:02.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:41 13.e 13:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 7 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 604.00000000 2024-09-13T16:33:02.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.899+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:42 13.15 13:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4400 con 0x55bfdb397400 2024-09-13T16:33:02.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 7 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 500.00000000 2024-09-13T16:33:02.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-09-13T16:33:02.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-09-13T16:33:02.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-09-13T16:33:02.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 605.00000000 2024-09-13T16:33:02.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:43 13.1d 13:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4800 con 0x55bfdbdb9000 2024-09-13T16:33:02.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 8 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-09-13T16:33:02.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-09-13T16:33:02.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 100.00000000 2024-09-13T16:33:02.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:44 13.13 13:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5c00 con 0x55bfdbdb9000 2024-09-13T16:33:02.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 9 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 200.00000002 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 609.00000000 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:45 13.7 13:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdbdb9800 con 0x55bfdbdb9000 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 10 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 400.00000000 2024-09-13T16:33:02.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-09-13T16:33:02.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-09-13T16:33:02.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-09-13T16:33:02.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 10 datascan.scan_links: step 1: handling object 1.00000000 2024-09-13T16:33:02.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.903+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:46 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5400 con 0x55bfdb396c00 2024-09-13T16:33:02.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 7 ==== osd_op_reply(46 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+1041 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:47 13.0 13:00000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 11 ==== osd_op_reply(47 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:48 13.10 13:08000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9400 2024-09-13T16:33:02.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 10 ==== osd_op_reply(48 [pgnls start_epoch 99 out=177b] v99'18 uv18 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:49 13.8 13:10000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396400 2024-09-13T16:33:02.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.2 v2:172.21.15.119:6824/3093261484 4 ==== osd_op_reply(49 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb396400 2024-09-13T16:33:02.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:50 13.18 13:18000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 12 ==== osd_op_reply(50 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.907+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:51 13.4 13:20000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396c00 2024-09-13T16:33:02.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 8 ==== osd_op_reply(51 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:52 13.14 13:28000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396800 2024-09-13T16:33:02.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 5 ==== osd_op_reply(52 [pgnls start_epoch 99 out=149b] v98'26 uv26 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:53 13.c 13:30000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9400 2024-09-13T16:33:02.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 11 ==== osd_op_reply(53 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:54 13.1c 13:38000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396c00 2024-09-13T16:33:02.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 9 ==== osd_op_reply(54 [pgnls start_epoch 99 out=73b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:55 13.2 13:40000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397000 2024-09-13T16:33:02.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 5 ==== osd_op_reply(55 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:56 13.12 13:48000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396800 2024-09-13T16:33:02.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 6 ==== osd_op_reply(56 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.911+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:57 13.a 13:50000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396400 2024-09-13T16:33:02.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.2 v2:172.21.15.119:6824/3093261484 5 ==== osd_op_reply(57 [pgnls start_epoch 99 out=81b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb396400 2024-09-13T16:33:02.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:58 13.1a 13:58000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9400 2024-09-13T16:33:02.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 12 ==== osd_op_reply(58 [pgnls start_epoch 99 out=49b] v96'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:59 13.6 13:60000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397400 2024-09-13T16:33:02.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 8 ==== osd_op_reply(59 [pgnls start_epoch 99 out=77b] v98'7 uv7 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:60 13.16 13:68000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397c00 2024-09-13T16:33:02.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.3 v2:172.21.15.119:6808/3803418857 3 ==== osd_op_reply(60 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb397c00 2024-09-13T16:33:02.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:61 13.e 13:70000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 13 ==== osd_op_reply(61 [pgnls start_epoch 99 out=73b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.915+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:62 13.1e 13:78000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397400 2024-09-13T16:33:02.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 9 ==== osd_op_reply(62 [pgnls start_epoch 99 out=49b] v96'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.919 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:63 13.1 13:80000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396c00 2024-09-13T16:33:02.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 10 ==== osd_op_reply(63 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:64 13.11 13:88000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9400 2024-09-13T16:33:02.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 13 ==== osd_op_reply(64 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:65 13.9 13:90000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397000 2024-09-13T16:33:02.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 6 ==== osd_op_reply(65 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.919+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:66 13.19 13:98000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397c00 2024-09-13T16:33:02.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.3 v2:172.21.15.119:6808/3803418857 4 ==== osd_op_reply(66 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb397c00 2024-09-13T16:33:02.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:67 13.5 13:a0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397400 2024-09-13T16:33:02.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 10 ==== osd_op_reply(67 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:68 13.15 13:a8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397400 2024-09-13T16:33:02.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 11 ==== osd_op_reply(68 [pgnls start_epoch 99 out=97b] v98'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:69 13.d 13:b0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397400 2024-09-13T16:33:02.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 12 ==== osd_op_reply(69 [pgnls start_epoch 99 out=74b] v94'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.923+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:70 13.1d 13:b8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 14 ==== osd_op_reply(70 [pgnls start_epoch 99 out=73b] v98'5 uv5 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:71 13.3 13:c0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396800 2024-09-13T16:33:02.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 7 ==== osd_op_reply(71 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:72 13.13 13:c8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 15 ==== osd_op_reply(72 [pgnls start_epoch 99 out=98b] v98'4 uv4 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.928 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:73 13.b 13:d0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397000 2024-09-13T16:33:02.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 7 ==== osd_op_reply(73 [pgnls start_epoch 99 out=76b] v98'6 uv6 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:74 13.1b 13:d8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397400 2024-09-13T16:33:02.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 13 ==== osd_op_reply(74 [pgnls start_epoch 99 out=73b] v98'10 uv10 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.930 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.927+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:75 13.7 13:e0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 16 ==== osd_op_reply(75 [pgnls start_epoch 99 out=73b] v98'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.931 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:76 13.17 13:e8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb397000 2024-09-13T16:33:02.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.5 v2:172.21.15.145:6824/4068956621 8 ==== osd_op_reply(76 [pgnls start_epoch 99 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb397000 2024-09-13T16:33:02.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:77 13.f 13:f0000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396c00 2024-09-13T16:33:02.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 11 ==== osd_op_reply(77 [pgnls start_epoch 99 out=73b] v94'1 uv1 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:78 13.1f 13:f8000000::::head [pgnls start_epoch 99 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdb396c00 2024-09-13T16:33:02.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 12 ==== osd_op_reply(78 [pgnls start_epoch 99 out=101b] v98'9 uv9 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 601.00000000 2024-09-13T16:33:02.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:79 13.10 13:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4400 con 0x55bfdbdb9400 2024-09-13T16:33:02.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 14 ==== osd_op_reply(79 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.931+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 602.00000000 2024-09-13T16:33:02.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:80 13.10 13:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4800 con 0x55bfdbdb9400 2024-09-13T16:33:02.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 15 ==== osd_op_reply(80 602.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-09-13T16:33:02.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:81 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdbdb9400 2024-09-13T16:33:02.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 16 ==== osd_op_reply(81 10000000000.00000000 [omap-get-vals out=506b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 20 datascan.scan_links: injected ino detected 2024-09-13T16:33:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 -1 datascan.scan_links: Bad nlink on 0x10000000001 expected 2 has 1 2024-09-13T16:33:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 20 datascan.scan_links: adding 0x10000000001 for future processing to fix dnfirst 2024-09-13T16:33:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 600.00000000 2024-09-13T16:33:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:82 13.10 13:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb397800 con 0x55bfdbdb9400 2024-09-13T16:33:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 17 ==== osd_op_reply(82 600.00000000 [omap-get-vals out=5b] v0'0 uv12 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 603.00000000 2024-09-13T16:33:02.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:83 13.10 13:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4c00 con 0x55bfdbdb9400 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 18 ==== osd_op_reply(83 603.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 200.00000000 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 200.00000001 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-09-13T16:33:02.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 606.00000000 2024-09-13T16:33:02.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:84 13.14 13:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5000 con 0x55bfdb396800 2024-09-13T16:33:02.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.0 v2:172.21.15.119:6800/2655437631 8 ==== osd_op_reply(84 606.00000000 [omap-get-vals out=5b] v0'0 uv24 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdb396800 2024-09-13T16:33:02.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.935+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 607.00000000 2024-09-13T16:33:02.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:85 13.1c 13:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4400 con 0x55bfdb396c00 2024-09-13T16:33:02.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 13 ==== osd_op_reply(85 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 100000001f6.00000000 2024-09-13T16:33:02.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:86 13.a 13:57fe7535:::100000001f6.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4800 con 0x55bfdb396400 2024-09-13T16:33:02.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 <== osd.2 v2:172.21.15.119:6824/3093261484 6 ==== osd_op_reply(86 100000001f6.00000000 [omap-get-vals out=54b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+54 (crc 0 0 0) 0x55bfdbd0cfc0 con 0x55bfdb396400 2024-09-13T16:33:02.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-09-13T16:33:02.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-09-13T16:33:02.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 608.00000000 2024-09-13T16:33:02.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:87 13.e 13:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cc000 con 0x55bfdbdb9000 2024-09-13T16:33:02.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 17 ==== osd_op_reply(87 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 604.00000000 2024-09-13T16:33:02.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.939+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:88 13.15 13:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cc400 con 0x55bfdb397400 2024-09-13T16:33:02.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 14 ==== osd_op_reply(88 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 500.00000000 2024-09-13T16:33:02.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-09-13T16:33:02.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-09-13T16:33:02.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-09-13T16:33:02.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 605.00000000 2024-09-13T16:33:02.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:89 13.1d 13:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cc800 con 0x55bfdbdb9000 2024-09-13T16:33:02.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 18 ==== osd_op_reply(89 605.00000000 [omap-get-vals out=5b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-09-13T16:33:02.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-09-13T16:33:02.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 100.00000000 2024-09-13T16:33:02.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:90 13.13 13:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3ccc00 con 0x55bfdbdb9000 2024-09-13T16:33:02.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 19 ==== osd_op_reply(90 100.00000000 [omap-get-vals out=5195b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-09-13T16:33:02.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-09-13T16:33:02.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 200.00000002 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000002' 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 609.00000000 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:91 13.7 13:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cd000 con 0x55bfdbdb9000 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 20 ==== osd_op_reply(91 609.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 400.00000000 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-09-13T16:33:02.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 10 datascan.scan_links: step 2: handling object 1.00000000 2024-09-13T16:33:02.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.943+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:92 13.1f 13:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cd400 con 0x55bfdb396c00 2024-09-13T16:33:02.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.6 v2:172.21.15.145:6800/3152548227 14 ==== osd_op_reply(92 1.00000000 [omap-get-vals out=1041b] v0'0 uv9 ondisk = 0) v8 ==== 154+0+1041 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb396c00 2024-09-13T16:33:02.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 10 datascan.scan_links: processing 0 dup_primaries, 1 remote_links 2024-09-13T16:33:02.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 10 datascan.scan_links: removing dup dentries from 0 objects 2024-09-13T16:33:02.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 10 datascan.scan_links: processing 1 bad_nlink_inos 2024-09-13T16:33:02.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 10 datascan.scan_links: handling bad_nlink_ino 0x10000000001 2024-09-13T16:33:02.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:93 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cd800 con 0x55bfdbdb9400 2024-09-13T16:33:02.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 19 ==== osd_op_reply(93 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv18 ondisk = 0) v8 ==== 164+0+505 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-09-13T16:33:02.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:94 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cdc00 con 0x55bfdbdb9400 2024-09-13T16:33:02.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 20 ==== osd_op_reply(94 10000000000.00000000 [omap-set-vals] v99'19 uv19 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-09-13T16:33:02.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 10 datascan.scan_links: processing 1 injected_inos 2024-09-13T16:33:02.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 10 datascan.scan_links: handling injected_ino 0x10000000001 2024-09-13T16:33:02.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000000:file 2024-09-13T16:33:02.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.947+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:95 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=17b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3cc000 con 0x55bfdbdb9400 2024-09-13T16:33:02.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 21 ==== osd_op_reply(95 10000000000.00000000 [omap-get-vals-by-keys out=505b] v0'0 uv19 ondisk = 0) v8 ==== 164+0+505 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0871429c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=file frag=*, r=0 2024-09-13T16:33:02.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0871429c0 20 datascan.scan_links: first is now 2 2024-09-13T16:33:02.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:96 13.10 13:0cf6ad0f:::10000000000.00000000:head [omap-set-vals in=505b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5c00 con 0x55bfdbdb9400 2024-09-13T16:33:02.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0860fd700 1 -- 172.21.15.119:0/2248403941 <== osd.1 v2:172.21.15.119:6816/1495422598 22 ==== osd_op_reply(96 10000000000.00000000 [omap-set-vals] v99'20 uv20 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55bfdbd0cd80 con 0x55bfdbdb9400 2024-09-13T16:33:02.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0871429c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000000/file pointing to 0x0x10000000001 2024-09-13T16:33:02.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0871429c0 10 datascan.scan_links: updating inotable 2024-09-13T16:33:02.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.951+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:97 13.13 13:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c5400 con 0x55bfdbdb9000 2024-09-13T16:33:02.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.955+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 21 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+50 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.955+0000 7fa0871429c0 10 mds.0.inotable: skip_inos was [0x10000000002~0x1f4,0x100000001f7~0xfffffffe09] 2024-09-13T16:33:02.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.955+0000 7fa0871429c0 10 mds.0.inotable: skip_inos now [0x100000001f7~0xfffffffe09] 2024-09-13T16:33:02.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.955+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:98 13.13 13:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4000 con 0x55bfdbdb9000 2024-09-13T16:33:02.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.955+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.7 v2:172.21.15.145:6808/1049327118 22 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v99'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdbdb9000 2024-09-13T16:33:02.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.955+0000 7fa0871429c0 10 datascan.scan_links: updating snaptable 2024-09-13T16:33:02.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.955+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:99 13.d 13:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e99) v8 -- 0x55bfdb3c4400 con 0x55bfdb397400 2024-09-13T16:33:02.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0850fb700 1 -- 172.21.15.119:0/2248403941 <== osd.4 v2:172.21.15.145:6816/4125431312 15 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x55bfdbd0cb40 con 0x55bfdb397400 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55bfdb396800 msgr2=0x55bfdb3a1080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55bfdb396800 0x55bfdb3a1080 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bfdbdb9400 msgr2=0x55bfdb3a0000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55bfdbdb9400 0x55bfdb3a0000 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55bfdb396400 msgr2=0x55bfdb3a0580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55bfdb396400 0x55bfdb3a0580 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55bfdb397c00 msgr2=0x55bfdb3a1b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55bfdb397c00 0x55bfdb3a1b80 crc :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55bfdb397400 msgr2=0x55bfdb3a2680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55bfdb397400 0x55bfdb3a2680 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55bfdb397000 msgr2=0x55bfdb3a1600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55bfdb397000 0x55bfdb3a1600 crc :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55bfdb396c00 msgr2=0x55bfdb3a0b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55bfdb396c00 0x55bfdb3a0b00 crc :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55bfdbdb9000 msgr2=0x55bfdafd6100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55bfdbdb9000 0x55bfdafd6100 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 reap_dead start 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55bfdb396000 msgr2=0x55bfdafd7700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55bfdb396000 0x55bfdafd7700 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x55bfdafeee70 tx=0x55bfdb398480 comp rx=0 tx=0).stop 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8800 msgr2=0x55bfdafd6c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/2248403941 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdbdb8800 0x55bfdafd6c00 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x55bfdb2418c0 tx=0x55bfdb244e40 comp rx=0 tx=0).stop 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 reap_dead start 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 reap_dead start 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 reap_dead start 2024-09-13T16:33:02.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0858fc700 1 -- 172.21.15.119:0/2248403941 reap_dead start 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 shutdown_connections 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 >> 172.21.15.119:0/2248403941 conn(0x55bfdaf2ec00 msgr2=0x55bfdafeb800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 shutdown_connections 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/2248403941 wait complete. 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/1936674818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdaf2f400 msgr2=0x55bfdafd4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdaf2f400 0x55bfdafd4b00 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x55bfdafef800 tx=0x55bfdb094000 comp rx=0 tx=0).stop 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/1936674818 shutdown_connections 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55bfdaf2f000 0x55bfdafd5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55bfdaf2f400 0x55bfdafd4b00 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 --2- 172.21.15.119:0/1936674818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55bfdaf2f800 0x55bfdafd4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/1936674818 >> 172.21.15.119:0/1936674818 conn(0x55bfdaf2fc00 msgr2=0x55bfdafeb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/1936674818 shutdown_connections 2024-09-13T16:33:02.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:02.959+0000 7fa0871429c0 1 -- 172.21.15.119:0/1936674818 wait complete. 2024-09-13T16:33:02.967 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.158056, result: 2024-09-13T16:33:02.967 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-13T16:33:03.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35fd6cd700 1 Processor -- start 2024-09-13T16:33:03.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35fd6cd700 1 -- start start 2024-09-13T16:33:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35fd6cd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35f8146670 0x7f35f80aca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35fd6cd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f80acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35fd6cd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f80ac040 con 0x7f35f80b0460 2024-09-13T16:33:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35fd6cd700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f80aed20 con 0x7f35f8146670 2024-09-13T16:33:03.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35fd6cd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f80ac600 con 0x7f35f80b0840 2024-09-13T16:33:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35f67fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35f8146670 0x7f35f80aca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35f77fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f80acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35f77fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f80acfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:58674/0 (socket says 172.21.15.119:58674) 2024-09-13T16:33:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35f77fe700 1 -- 172.21.15.119:0/2945452705 learned_addr learned my addr 172.21.15.119:0/2945452705 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35f67fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f35f8146670 0x7f35f80aca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:42868/0 (socket says 172.21.15.119:42868) 2024-09-13T16:33:03.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.087+0000 7f35f6ffd700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f35f80b0460 0x7f35f8071930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:56466/0 (socket says 172.21.15.119:56466) 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f67fc700 1 -- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 msgr2=0x7f35f80acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f67fc700 1 --2- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f80acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f67fc700 1 -- 172.21.15.119:0/2945452705 >> v1:172.21.15.145:6789/0 conn(0x7f35f80b0460 legacy=0x7f35f8071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f67fc700 1 -- 172.21.15.119:0/2945452705 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f815d120 con 0x7f35f8146670 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f77fe700 1 --2- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f80acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f67fc700 1 --2- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f8146670 0x7f35f80aca90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009850 tx=0x7f35e8009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f8deae11e38603d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f5ffb700 1 -- 172.21.15.119:0/2945452705 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8011830 con 0x7f35f8146670 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f5ffb700 1 -- 172.21.15.119:0/2945452705 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e8011990 con 0x7f35f8146670 2024-09-13T16:33:03.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35f5ffb700 1 -- 172.21.15.119:0/2945452705 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8011d40 con 0x7f35f8146670 2024-09-13T16:33:03.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 -- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f8146670 msgr2=0x7f35f80aca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f8146670 0x7f35f80aca90 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009850 tx=0x7f35e8009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:03.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 -- 172.21.15.119:0/2945452705 shutdown_connections 2024-09-13T16:33:03.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f80acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/2945452705 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f8146670 0x7f35f80aca90 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 -- 172.21.15.119:0/2945452705 >> 172.21.15.119:0/2945452705 conn(0x7f35f80b3ec0 msgr2=0x7f35f80b4580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 -- 172.21.15.119:0/2945452705 shutdown_connections 2024-09-13T16:33:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.091+0000 7f35fd6cd700 1 -- 172.21.15.119:0/2945452705 wait complete. 2024-09-13T16:33:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 Processor -- start 2024-09-13T16:33:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 -- start start 2024-09-13T16:33:03.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f80b0460 0x7f35f8156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f8145090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f8146670 0x7f35f8154830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f815a6e0 con 0x7f35f8146670 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f8155a60 con 0x7f35f80b0460 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f8155d40 con 0x7f35f80b0840 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f8146670 0x7f35f8154830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f67fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f8145090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f8146670 0x7f35f8154830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33312/0 (socket says 172.21.15.119:33312) 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f6ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f80b0460 0x7f35f8156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f77fe700 1 -- 172.21.15.119:0/1322214137 learned_addr learned my addr 172.21.15.119:0/1322214137 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f6ffd700 1 -- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 msgr2=0x7f35f8145090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f6ffd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f8145090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f6ffd700 1 -- 172.21.15.119:0/1322214137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f8146670 msgr2=0x7f35f8154830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f6ffd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f8146670 0x7f35f8154830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f6ffd700 1 -- 172.21.15.119:0/1322214137 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f0009b30 con 0x7f35f80b0460 2024-09-13T16:33:03.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f77fe700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f8146670 0x7f35f8154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:03.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35f6ffd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f80b0460 0x7f35f8156b00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f35e000c500 tx=0x7f35e000c810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35ef7fe700 1 -- 172.21.15.119:0/1322214137 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e000cea0 con 0x7f35f80b0460 2024-09-13T16:33:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e80094b0 con 0x7f35f80b0460 2024-09-13T16:33:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35ef7fe700 1 -- 172.21.15.119:0/1322214137 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0005cd0 con 0x7f35f80b0460 2024-09-13T16:33:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35ef7fe700 1 -- 172.21.15.119:0/1322214137 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e00072d0 con 0x7f35f80b0460 2024-09-13T16:33:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.095+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f8156300 con 0x7f35f80b0460 2024-09-13T16:33:03.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.099+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f8071930 con 0x7f35f80b0460 2024-09-13T16:33:03.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.099+0000 7f35ef7fe700 1 -- 172.21.15.119:0/1322214137 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f35e0007430 con 0x7f35f80b0460 2024-09-13T16:33:03.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.099+0000 7f35ef7fe700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35d8066730 0x7f35d8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.099+0000 7f35ef7fe700 1 -- 172.21.15.119:0/1322214137 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f35e0083fc0 con 0x7f35f80b0460 2024-09-13T16:33:03.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.099+0000 7f35f67fc700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35d8066730 0x7f35d8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.103+0000 7f35f67fc700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35d8066730 0x7f35d8068bf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f35f8146a90 tx=0x7f35e8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:03.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.111+0000 7f35ef7fe700 1 -- 172.21.15.119:0/1322214137 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f35e0051170 con 0x7f35f80b0460 2024-09-13T16:33:03.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.315+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f35f80cff60 con 0x7f35f80b0460 2024-09-13T16:33:03.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.491+0000 7f35ef7fe700 1 -- 172.21.15.119:0/1322214137 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v65) v1 ==== 162+0+0 (secure 0 0 0) 0x7f35e0017020 con 0x7f35f80b0460 2024-09-13T16:33:03.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35d8066730 msgr2=0x7f35d8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35d8066730 0x7f35d8068bf0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f35f8146a90 tx=0x7f35e8008030 comp rx=0 tx=0).stop 2024-09-13T16:33:03.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f80b0460 msgr2=0x7f35f8156b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f80b0460 0x7f35f8156b00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f35e000c500 tx=0x7f35e000c810 comp rx=0 tx=0).stop 2024-09-13T16:33:03.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 shutdown_connections 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f35d8066730 0x7f35d8068bf0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f35f8146670 0x7f35f8154830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f35f80b0840 0x7f35f8145090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 --2- 172.21.15.119:0/1322214137 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f35f80b0460 0x7f35f8156b00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 >> 172.21.15.119:0/1322214137 conn(0x7f35f80b3ec0 msgr2=0x7f35f80aca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 shutdown_connections 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.499+0000 7f35fd6cd700 1 -- 172.21.15.119:0/1322214137 wait complete. 2024-09-13T16:33:03.501 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-13T16:33:03.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:03.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.623+0000 7f6c62fbd700 1 Processor -- start 2024-09-13T16:33:03.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.623+0000 7f6c62fbd700 1 -- start start 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6c5c143f50 0x7f6c5c06c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c0b1c40 con 0x7f6c5c144330 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c5c06cdf0 con 0x7f6c5c0b3410 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c5c06d0d0 con 0x7f6c5c143f50 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c617ba700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6c5c144330 0x7f6c5c0cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42322/0 (socket says 172.21.15.119:42322) 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c617ba700 1 -- 172.21.15.119:0/2739811968 learned_addr learned my addr 172.21.15.119:0/2739811968 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c61fbb700 1 --2- 172.21.15.119:0/2739811968 >> v2:172.21.15.119:3300/0 conn(0x7f6c5c143f50 0x7f6c5c06c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c60fb9700 1 -- 172.21.15.119:0/2739811968 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257575551 0 0) 0x7f6c5c0b1c40 con 0x7f6c5c144330 2024-09-13T16:33:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c60fb9700 1 -- 172.21.15.119:0/2739811968 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c480035d0 con 0x7f6c5c144330 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c60fb9700 1 -- 172.21.15.119:0/2739811968 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3503785739 0 0) 0x7f6c480035d0 con 0x7f6c5c144330 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c61fbb700 1 -- 172.21.15.119:0/2739811968 >> v1:172.21.15.145:6789/0 conn(0x7f6c5c0b3410 legacy=0x7f6c5c164330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c61fbb700 1 -- 172.21.15.119:0/2739811968 >> v1:172.21.15.119:6789/0 conn(0x7f6c5c144330 legacy=0x7f6c5c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c61fbb700 1 -- 172.21.15.119:0/2739811968 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c5c06d3b0 con 0x7f6c5c143f50 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c61fbb700 1 --2- 172.21.15.119:0/2739811968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c06c5a0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009850 tx=0x7f6c4c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee52d9084905dc8e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c60fb9700 1 -- 172.21.15.119:0/2739811968 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58001670 con 0x7f6c5c143f50 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c60fb9700 1 -- 172.21.15.119:0/2739811968 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c4c010590 con 0x7f6c5c143f50 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c60fb9700 1 -- 172.21.15.119:0/2739811968 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c4c010900 con 0x7f6c5c143f50 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2739811968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 msgr2=0x7f6c5c06c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2739811968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c06c5a0 secure :-1 s=READY pgs=1425 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c009850 tx=0x7f6c4c009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2739811968 shutdown_connections 2024-09-13T16:33:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2739811968 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c06c5a0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2739811968 >> 172.21.15.119:0/2739811968 conn(0x7f6c5c06a350 msgr2=0x7f6c5c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2739811968 shutdown_connections 2024-09-13T16:33:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.627+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2739811968 wait complete. 2024-09-13T16:33:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.631+0000 7f6c62fbd700 1 Processor -- start 2024-09-13T16:33:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.631+0000 7f6c62fbd700 1 -- start start 2024-09-13T16:33:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c5c0b3410 0x7f6c5c0ae440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c5c144330 0x7f6c5c0acf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c0ca3b0 con 0x7f6c5c143f50 2024-09-13T16:33:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c1f0660 con 0x7f6c5c0b3410 2024-09-13T16:33:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c5c1f0920 con 0x7f6c5c144330 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c0ae980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33320/0 (socket says 172.21.15.119:33320) 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 -- 172.21.15.119:0/2793481376 learned_addr learned my addr 172.21.15.119:0/2793481376 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c61fbb700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c5c0b3410 0x7f6c5c0ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c627bc700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c5c144330 0x7f6c5c0acf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 -- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c5c144330 msgr2=0x7f6c5c0acf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c5c144330 0x7f6c5c0acf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 -- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c5c0b3410 msgr2=0x7f6c5c0ae440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c5c0b3410 0x7f6c5c0ae440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 -- 172.21.15.119:0/2793481376 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c4c0094b0 con 0x7f6c5c143f50 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c61fbb700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c5c0b3410 0x7f6c5c0ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c617ba700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c0ae980 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f6c580017a0 tx=0x7f6c5800cae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c527fc700 1 -- 172.21.15.119:0/2793481376 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58004ca0 con 0x7f6c5c143f50 2024-09-13T16:33:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c527fc700 1 -- 172.21.15.119:0/2793481376 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c58004e00 con 0x7f6c5c143f50 2024-09-13T16:33:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c527fc700 1 -- 172.21.15.119:0/2793481376 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c58007310 con 0x7f6c5c143f50 2024-09-13T16:33:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c5c1f0c00 con 0x7f6c5c143f50 2024-09-13T16:33:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c5c1f11c0 con 0x7f6c5c143f50 2024-09-13T16:33:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.635+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c5c0ccdc0 con 0x7f6c5c143f50 2024-09-13T16:33:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.639+0000 7f6c527fc700 1 -- 172.21.15.119:0/2793481376 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6c58007470 con 0x7f6c5c143f50 2024-09-13T16:33:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.639+0000 7f6c527fc700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c40066730 0x7f6c40068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.639+0000 7f6c527fc700 1 -- 172.21.15.119:0/2793481376 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f6c58084210 con 0x7f6c5c143f50 2024-09-13T16:33:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.639+0000 7f6c61fbb700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c40066730 0x7f6c40068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.639+0000 7f6c61fbb700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c40066730 0x7f6c40068bf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c013040 tx=0x7f6c4c008000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:03.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.643+0000 7f6c527fc700 1 -- 172.21.15.119:0/2793481376 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6c58051280 con 0x7f6c5c143f50 2024-09-13T16:33:03.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.855+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c5c06c5a0 con 0x7f6c5c143f50 2024-09-13T16:33:03.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.859+0000 7f6c527fc700 1 -- 172.21.15.119:0/2793481376 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f6c58007720 con 0x7f6c5c143f50 2024-09-13T16:33:03.859 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:03.859 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":66,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:03.492206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:replay","state_seq":1,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.859+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c40066730 msgr2=0x7f6c40068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.859+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c40066730 0x7f6c40068bf0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f6c4c013040 tx=0x7f6c4c008000 comp rx=0 tx=0).stop 2024-09-13T16:33:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.859+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 msgr2=0x7f6c5c0ae980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.859+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c0ae980 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 crypto rx=0x7f6c580017a0 tx=0x7f6c5800cae0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 shutdown_connections 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c40066730 0x7f6c40068bf0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c5c144330 0x7f6c5c0acf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c5c143f50 0x7f6c5c0ae980 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 --2- 172.21.15.119:0/2793481376 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c5c0b3410 0x7f6c5c0ae440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 >> 172.21.15.119:0/2793481376 conn(0x7f6c5c06a350 msgr2=0x7f6c5c0cda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 shutdown_connections 2024-09-13T16:33:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:03.863+0000 7f6c62fbd700 1 -- 172.21.15.119:0/2793481376 wait complete. 2024-09-13T16:33:03.865 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 66 2024-09-13T16:33:04.283 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:31:28.960022+0000', 'modified': '2024-09-13T16:33:03.492206+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8442': {'gid': 8442, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/1434266893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 1434266893}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 1434266893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:33:04.284 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8442:up:replay 2024-09-13T16:33:05.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:05.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.403+0000 7fe175811700 1 Processor -- start 2024-09-13T16:33:05.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.403+0000 7fe175811700 1 -- start start 2024-09-13T16:33:05.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.403+0000 7fe175811700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe17006dc40 0x7fe1700b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:05.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.403+0000 7fe175811700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1700b2280 con 0x7fe1701594f0 2024-09-13T16:33:05.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.403+0000 7fe175811700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1700cc890 con 0x7fe1700b3630 2024-09-13T16:33:05.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.403+0000 7fe175811700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1700b1970 con 0x7fe17006dc40 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe17480f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe1701594f0 0x7fe17006d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42332/0 (socket says 172.21.15.119:42332) 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe17480f700 1 -- 172.21.15.119:0/3132249968 learned_addr learned my addr 172.21.15.119:0/3132249968 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe167fff700 1 --2- 172.21.15.119:0/3132249968 >> v2:172.21.15.145:3300/0 conn(0x7fe17006dc40 0x7fe1700b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe1677fe700 1 -- 172.21.15.119:0/3132249968 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736526457 0 0) 0x7fe1700b2280 con 0x7fe1701594f0 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe1677fe700 1 -- 172.21.15.119:0/3132249968 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1580035d0 con 0x7fe1701594f0 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe1677fe700 1 -- 172.21.15.119:0/3132249968 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3369506125 0 0) 0x7fe1700cc890 con 0x7fe1700b3630 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe1677fe700 1 -- 172.21.15.119:0/3132249968 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1700b2280 con 0x7fe1700b3630 2024-09-13T16:33:05.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe167fff700 1 -- 172.21.15.119:0/3132249968 >> v1:172.21.15.145:6789/0 conn(0x7fe1700b3630 legacy=0x7fe17006fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:05.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe167fff700 1 -- 172.21.15.119:0/3132249968 >> v1:172.21.15.119:6789/0 conn(0x7fe1701594f0 legacy=0x7fe17006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:05.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe167fff700 1 -- 172.21.15.119:0/3132249968 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1700b1c30 con 0x7fe17006dc40 2024-09-13T16:33:05.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe167fff700 1 --2- 172.21.15.119:0/3132249968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe17006dc40 0x7fe1700b30f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe160009850 tx=0x7fe160009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ad4096ce21d9c073 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe1677fe700 1 -- 172.21.15.119:0/3132249968 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160011830 con 0x7fe17006dc40 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe1677fe700 1 -- 172.21.15.119:0/3132249968 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe160011990 con 0x7fe17006dc40 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe1677fe700 1 -- 172.21.15.119:0/3132249968 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160011d00 con 0x7fe17006dc40 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe175811700 1 -- 172.21.15.119:0/3132249968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe17006dc40 msgr2=0x7fe1700b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe175811700 1 --2- 172.21.15.119:0/3132249968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe17006dc40 0x7fe1700b30f0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fe160009850 tx=0x7fe160009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe175811700 1 -- 172.21.15.119:0/3132249968 shutdown_connections 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe175811700 1 --2- 172.21.15.119:0/3132249968 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe17006dc40 0x7fe1700b30f0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe175811700 1 -- 172.21.15.119:0/3132249968 >> 172.21.15.119:0/3132249968 conn(0x7fe17006a350 msgr2=0x7fe17006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe175811700 1 -- 172.21.15.119:0/3132249968 shutdown_connections 2024-09-13T16:33:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.407+0000 7fe175811700 1 -- 172.21.15.119:0/3132249968 wait complete. 2024-09-13T16:33:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 Processor -- start 2024-09-13T16:33:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 -- start start 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe17006dc40 0x7fe170152410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1700b3630 0x7fe1700cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1701594f0 0x7fe1700cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701f66c0 con 0x7fe1700b3630 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe170152950 con 0x7fe1701594f0 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1701f5900 con 0x7fe17006dc40 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1700b3630 0x7fe1700cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1700b3630 0x7fe1700cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33334/0 (socket says 172.21.15.119:33334) 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 -- 172.21.15.119:0/2097630387 learned_addr learned my addr 172.21.15.119:0/2097630387 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175010700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1701594f0 0x7fe1700cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe17480f700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe17006dc40 0x7fe170152410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:05.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 -- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe17006dc40 msgr2=0x7fe170152410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:05.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe17006dc40 0x7fe170152410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 -- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1701594f0 msgr2=0x7fe1700cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1701594f0 0x7fe1700cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 -- 172.21.15.119:0/2097630387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1600094b0 con 0x7fe1700b3630 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe167fff700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1700b3630 0x7fe1700cd4c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe1600090b0 tx=0x7fe160004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe1657fa700 1 -- 172.21.15.119:0/2097630387 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe160004380 con 0x7fe1700b3630 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe1657fa700 1 -- 172.21.15.119:0/2097630387 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe16001f070 con 0x7fe1700b3630 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1701f5be0 con 0x7fe1700b3630 2024-09-13T16:33:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe1657fa700 1 -- 172.21.15.119:0/2097630387 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1600036c0 con 0x7fe1700b3630 2024-09-13T16:33:05.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.411+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1701f61a0 con 0x7fe1700b3630 2024-09-13T16:33:05.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.415+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1700b30f0 con 0x7fe1700b3630 2024-09-13T16:33:05.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.415+0000 7fe1657fa700 1 -- 172.21.15.119:0/2097630387 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe160021030 con 0x7fe1700b3630 2024-09-13T16:33:05.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.415+0000 7fe1657fa700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe15c066730 0x7fe15c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:05.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.415+0000 7fe1657fa700 1 -- 172.21.15.119:0/2097630387 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fe160085870 con 0x7fe1700b3630 2024-09-13T16:33:05.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.419+0000 7fe17480f700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe15c066730 0x7fe15c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:05.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.419+0000 7fe17480f700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe15c066730 0x7fe15c068bf0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe16c000f60 tx=0x7fe16c007e00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:05.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.427+0000 7fe1657fa700 1 -- 172.21.15.119:0/2097630387 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe160052b30 con 0x7fe1700b3630 2024-09-13T16:33:05.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.631+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1700706e0 con 0x7fe1700b3630 2024-09-13T16:33:05.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe1657fa700 1 -- 172.21.15.119:0/2097630387 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+4746 (secure 0 0 0) 0x7fe160056150 con 0x7fe1700b3630 2024-09-13T16:33:05.636 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:05.636 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":68,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:05.503663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:05.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe15c066730 msgr2=0x7fe15c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe15c066730 0x7fe15c068bf0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7fe16c000f60 tx=0x7fe16c007e00 comp rx=0 tx=0).stop 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1700b3630 msgr2=0x7fe1700cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1700b3630 0x7fe1700cd4c0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7fe1600090b0 tx=0x7fe160004770 comp rx=0 tx=0).stop 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 shutdown_connections 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe15c066730 0x7fe15c068bf0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe1701594f0 0x7fe1700cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe1700b3630 0x7fe1700cd4c0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 --2- 172.21.15.119:0/2097630387 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe17006dc40 0x7fe170152410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 >> 172.21.15.119:0/2097630387 conn(0x7fe17006a350 msgr2=0x7fe17015ce30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 shutdown_connections 2024-09-13T16:33:05.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:05.635+0000 7fe175811700 1 -- 172.21.15.119:0/2097630387 wait complete. 2024-09-13T16:33:05.640 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 68 2024-09-13T16:33:06.054 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 68, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:31:28.960022+0000', 'modified': '2024-09-13T16:33:05.503663+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8442': {'gid': 8442, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.119:6835/1434266893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 1434266893}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 1434266893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:33:06.054 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8442:up:rejoin 2024-09-13T16:33:07.055 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:07.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.175+0000 7efe524b6700 1 Processor -- start 2024-09-13T16:33:07.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.175+0000 7efe524b6700 1 -- start start 2024-09-13T16:33:07.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efe4c0b17f0 0x7efe4c0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efe4c0b2200 0x7efe4c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c0705d0 con 0x7efe4c0cd8d0 2024-09-13T16:33:07.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c0b2bb0 con 0x7efe4c0b17f0 2024-09-13T16:33:07.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4c06fca0 con 0x7efe4c0b2200 2024-09-13T16:33:07.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe50cb3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efe4c0b17f0 0x7efe4c0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe514b4700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7efe4c0cd8d0 0x7efe4c06cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42348/0 (socket says 172.21.15.119:42348) 2024-09-13T16:33:07.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe514b4700 1 -- 172.21.15.119:0/963473823 learned_addr learned my addr 172.21.15.119:0/963473823 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:07.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe51cb5700 1 --2- 172.21.15.119:0/963473823 >> v2:172.21.15.145:3300/0 conn(0x7efe4c0b2200 0x7efe4c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe50cb3700 1 -- 172.21.15.119:0/963473823 >> v2:172.21.15.145:3300/0 conn(0x7efe4c0b2200 msgr2=0x7efe4c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:07.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe50cb3700 1 --2- 172.21.15.119:0/963473823 >> v2:172.21.15.145:3300/0 conn(0x7efe4c0b2200 0x7efe4c0b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe50cb3700 1 -- 172.21.15.119:0/963473823 >> v1:172.21.15.119:6789/0 conn(0x7efe4c0cd8d0 legacy=0x7efe4c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe43fff700 1 -- 172.21.15.119:0/963473823 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768249168 0 0) 0x7efe4c0705d0 con 0x7efe4c0cd8d0 2024-09-13T16:33:07.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe50cb3700 1 -- 172.21.15.119:0/963473823 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c06ff80 con 0x7efe4c0b17f0 2024-09-13T16:33:07.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe50cb3700 1 --2- 172.21.15.119:0/963473823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b17f0 0x7efe4c0cb800 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7efe4c071f00 tx=0x7efe4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c3dc4dd259953af server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe43fff700 1 -- 172.21.15.119:0/963473823 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4800e070 con 0x7efe4c0b17f0 2024-09-13T16:33:07.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe43fff700 1 -- 172.21.15.119:0/963473823 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4800bb20 con 0x7efe4c0b17f0 2024-09-13T16:33:07.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe43fff700 1 -- 172.21.15.119:0/963473823 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48012430 con 0x7efe4c0b17f0 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- 172.21.15.119:0/963473823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b17f0 msgr2=0x7efe4c0cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 --2- 172.21.15.119:0/963473823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b17f0 0x7efe4c0cb800 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7efe4c071f00 tx=0x7efe4800b330 comp rx=0 tx=0).stop 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- 172.21.15.119:0/963473823 shutdown_connections 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 --2- 172.21.15.119:0/963473823 >> v2:172.21.15.145:3300/0 conn(0x7efe4c0b2200 0x7efe4c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 --2- 172.21.15.119:0/963473823 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b17f0 0x7efe4c0cb800 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- 172.21.15.119:0/963473823 >> 172.21.15.119:0/963473823 conn(0x7efe4c06a350 msgr2=0x7efe4c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- 172.21.15.119:0/963473823 shutdown_connections 2024-09-13T16:33:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.179+0000 7efe524b6700 1 -- 172.21.15.119:0/963473823 wait complete. 2024-09-13T16:33:07.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 Processor -- start 2024-09-13T16:33:07.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 -- start start 2024-09-13T16:33:07.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4c0b17f0 0x7efe4c0aa330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b2200 0x7efe4c1f6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe4c0cd8d0 0x7efe4c163370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c1f76e0 con 0x7efe4c0b2200 2024-09-13T16:33:07.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c1f7120 con 0x7efe4c0cd8d0 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.183+0000 7efe524b6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe4c1f7400 con 0x7efe4c0b17f0 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b2200 0x7efe4c1f6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b2200 0x7efe4c1f6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33358/0 (socket says 172.21.15.119:33358) 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 -- 172.21.15.119:0/1356192250 learned_addr learned my addr 172.21.15.119:0/1356192250 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe514b4700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4c0b17f0 0x7efe4c0aa330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe51cb5700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe4c0cd8d0 0x7efe4c163370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 -- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4c0b17f0 msgr2=0x7efe4c0aa330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4c0b17f0 0x7efe4c0aa330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 -- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe4c0cd8d0 msgr2=0x7efe4c163370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe4c0cd8d0 0x7efe4c163370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 -- 172.21.15.119:0/1356192250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe48009a70 con 0x7efe4c0b2200 2024-09-13T16:33:07.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe50cb3700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b2200 0x7efe4c1f6710 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7efe48014040 tx=0x7efe48004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe41ffb700 1 -- 172.21.15.119:0/1356192250 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48012430 con 0x7efe4c0b2200 2024-09-13T16:33:07.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe41ffb700 1 -- 172.21.15.119:0/1356192250 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe48020070 con 0x7efe4c0b2200 2024-09-13T16:33:07.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c0a9b30 con 0x7efe4c0b2200 2024-09-13T16:33:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe41ffb700 1 -- 172.21.15.119:0/1356192250 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe48004280 con 0x7efe4c0b2200 2024-09-13T16:33:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c0aa870 con 0x7efe4c0b2200 2024-09-13T16:33:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c0aa050 con 0x7efe4c0b2200 2024-09-13T16:33:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.187+0000 7efe41ffb700 1 -- 172.21.15.119:0/1356192250 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efe480043e0 con 0x7efe4c0b2200 2024-09-13T16:33:07.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.191+0000 7efe41ffb700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe30066730 0x7efe30068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.191+0000 7efe41ffb700 1 -- 172.21.15.119:0/1356192250 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7efe48022070 con 0x7efe4c0b2200 2024-09-13T16:33:07.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.191+0000 7efe514b4700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe30066730 0x7efe30068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.191+0000 7efe514b4700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe30066730 0x7efe30068bf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000f60 tx=0x7efe3c007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.195+0000 7efe41ffb700 1 -- 172.21.15.119:0/1356192250 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efe4c0aa050 con 0x7efe4c0b2200 2024-09-13T16:33:07.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe4c0aa050 con 0x7efe4c0b2200 2024-09-13T16:33:07.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe41ffb700 1 -- 172.21.15.119:0/1356192250 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4746 (secure 0 0 0) 0x7efe4c0aa050 con 0x7efe4c0b2200 2024-09-13T16:33:07.408 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:07.408 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":69,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:06.506474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:07.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe30066730 msgr2=0x7efe30068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe30066730 0x7efe30068bf0 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7efe3c000f60 tx=0x7efe3c007e90 comp rx=0 tx=0).stop 2024-09-13T16:33:07.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b2200 msgr2=0x7efe4c1f6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b2200 0x7efe4c1f6710 secure :-1 s=READY pgs=1431 cs=0 l=1 rev1=1 crypto rx=0x7efe48014040 tx=0x7efe48004af0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 shutdown_connections 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe30066730 0x7efe30068bf0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe4c0cd8d0 0x7efe4c163370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe4c0b2200 0x7efe4c1f6710 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 --2- 172.21.15.119:0/1356192250 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe4c0b17f0 0x7efe4c0aa330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.407+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 >> 172.21.15.119:0/1356192250 conn(0x7efe4c06a350 msgr2=0x7efe4c06d500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.411+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 shutdown_connections 2024-09-13T16:33:07.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.411+0000 7efe524b6700 1 -- 172.21.15.119:0/1356192250 wait complete. 2024-09-13T16:33:07.413 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 69 2024-09-13T16:33:07.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 69, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:31:28.960022+0000', 'modified': '2024-09-13T16:33:06.506474+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 99, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8442}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8442': {'gid': 8442, 'name': 'c', 'rank': 0, 'incarnation': 66, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.119:6835/1434266893', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 1434266893}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 1434266893}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:33:07.826 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:33:07.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:33:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.951+0000 7fa7a48a7700 1 Processor -- start 2024-09-13T16:33:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.951+0000 7fa7a48a7700 1 -- start start 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.951+0000 7fa7a48a7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa7980e18c0 0x7fa79801ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.951+0000 7fa7a48a7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa798006ce0 con 0x7fa798005b10 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.951+0000 7fa7a48a7700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7980060d0 con 0x7fa79800a5b0 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.951+0000 7fa7a48a7700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7980063b0 con 0x7fa7980e18c0 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa7980e18c0 0x7fa79801ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa7980e18c0 0x7fa79801ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33362/0 (socket says 172.21.15.119:33362) 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79e59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa798005b10 0x7fa79801c740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42358/0 (socket says 172.21.15.119:42358) 2024-09-13T16:33:07.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79f59e700 1 -- 172.21.15.119:0/2259537946 learned_addr learned my addr 172.21.15.119:0/2259537946 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79f59e700 1 -- 172.21.15.119:0/2259537946 >> v1:172.21.15.145:6789/0 conn(0x7fa79800a5b0 legacy=0x7fa79800a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79f59e700 1 -- 172.21.15.119:0/2259537946 >> v1:172.21.15.119:6789/0 conn(0x7fa798005b10 legacy=0x7fa79801c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79f59e700 1 -- 172.21.15.119:0/2259537946 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798006690 con 0x7fa7980e18c0 2024-09-13T16:33:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79f59e700 1 --2- 172.21.15.119:0/2259537946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7980e18c0 0x7fa79801ce50 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00698b0 tx=0x7fa7a004f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb3b8669753da2c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79dd9b700 1 -- 172.21.15.119:0/2259537946 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0067690 con 0x7fa7980e18c0 2024-09-13T16:33:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79dd9b700 1 -- 172.21.15.119:0/2259537946 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a004f010 con 0x7fa7980e18c0 2024-09-13T16:33:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa79dd9b700 1 -- 172.21.15.119:0/2259537946 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0067b90 con 0x7fa7980e18c0 2024-09-13T16:33:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa7a48a7700 1 -- 172.21.15.119:0/2259537946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7980e18c0 msgr2=0x7fa79801ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/2259537946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7980e18c0 0x7fa79801ce50 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00698b0 tx=0x7fa7a004f710 comp rx=0 tx=0).stop 2024-09-13T16:33:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa7a48a7700 1 -- 172.21.15.119:0/2259537946 shutdown_connections 2024-09-13T16:33:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/2259537946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa7980e18c0 0x7fa79801ce50 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa7a48a7700 1 -- 172.21.15.119:0/2259537946 >> 172.21.15.119:0/2259537946 conn(0x7fa798036140 msgr2=0x7fa798036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa7a48a7700 1 -- 172.21.15.119:0/2259537946 shutdown_connections 2024-09-13T16:33:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.955+0000 7fa7a48a7700 1 -- 172.21.15.119:0/2259537946 wait complete. 2024-09-13T16:33:07.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 Processor -- start 2024-09-13T16:33:07.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 -- start start 2024-09-13T16:33:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa798005b10 0x7fa7980d96f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa79800a5b0 0x7fa7980da670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7980e18c0 0x7fa79817ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980dabb0 con 0x7fa79800a5b0 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980dbe40 con 0x7fa7980e18c0 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa7a48a7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7980dada0 con 0x7fa798005b10 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa79e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa79800a5b0 0x7fa7980da670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa79e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa79800a5b0 0x7fa7980da670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33366/0 (socket says 172.21.15.119:33366) 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa79e59c700 1 -- 172.21.15.119:0/832311073 learned_addr learned my addr 172.21.15.119:0/832311073 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa79f59e700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7980e18c0 0x7fa79817ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.959+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa798005b10 0x7fa7980d96f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa79e59c700 1 -- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa798005b10 msgr2=0x7fa7980d96f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa79e59c700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa798005b10 0x7fa7980d96f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa79e59c700 1 -- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7980e18c0 msgr2=0x7fa79817ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa79e59c700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7980e18c0 0x7fa79817ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa79e59c700 1 -- 172.21.15.119:0/832311073 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7980db080 con 0x7fa79800a5b0 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa79e59c700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa79800a5b0 0x7fa7980da670 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c500 tx=0x7fa78800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa7977fe700 1 -- 172.21.15.119:0/832311073 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788005cd0 con 0x7fa79800a5b0 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa7977fe700 1 -- 172.21.15.119:0/832311073 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78800ce90 con 0x7fa79800a5b0 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa7977fe700 1 -- 172.21.15.119:0/832311073 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880072d0 con 0x7fa79800a5b0 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7980db360 con 0x7fa79800a5b0 2024-09-13T16:33:07.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.963+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7980db920 con 0x7fa79800a5b0 2024-09-13T16:33:07.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7977fe700 1 -- 172.21.15.119:0/832311073 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa788007430 con 0x7fa79800a5b0 2024-09-13T16:33:07.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7977fe700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa780066760 0x7fa780068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 Processor -- start 2024-09-13T16:33:07.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7977fe700 1 -- 172.21.15.119:0/832311073 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fa788083500 con 0x7fa79800a5b0 2024-09-13T16:33:07.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 -- start start 2024-09-13T16:33:07.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa780066760 0x7fa780068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xccc610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xc94890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xc94dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xbf3d80 con 0xcaae90 2024-09-13T16:33:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xbf1510 con 0xccc230 2024-09-13T16:33:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xbece00 con 0xcccb50 2024-09-13T16:33:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa780066760 0x7fa780068c20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa790001870 tx=0x7fa790009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xc94890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xc94890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33372/0 (socket says 172.21.15.119:33372) 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79ed9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xccc610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 -- 172.21.15.119:0/1924162659 learned_addr learned my addr 172.21.15.119:0/1924162659 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79f59e700 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xc94dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 -- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 msgr2=0xc94dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xc94dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 -- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 msgr2=0xccc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xccc610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 -- 172.21.15.119:0/1924162659 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xa5f960 con 0xcaae90 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79f59e700 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xc94dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa79e59c700 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xc94890 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa788050910 tx=0x7fa788053ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c50996f8dd22014 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa794ff9700 1 -- 172.21.15.119:0/1924162659 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788058190 con 0xcaae90 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa794ff9700 1 -- 172.21.15.119:0/1924162659 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78805c750 con 0xcaae90 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa794ff9700 1 -- 172.21.15.119:0/1924162659 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7880527b0 con 0xcaae90 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 -- 172.21.15.119:0/1924162659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 msgr2=0xc94890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.967+0000 7fa7a8359740 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xc94890 secure :-1 s=READY pgs=1435 cs=0 l=1 rev1=1 crypto rx=0x7fa788050910 tx=0x7fa788053ea0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- 172.21.15.119:0/1924162659 shutdown_connections 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xc94dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xc94890 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 --2- 172.21.15.119:0/1924162659 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xccc610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- 172.21.15.119:0/1924162659 >> 172.21.15.119:0/1924162659 conn(0xa144d0 msgr2=0xcade30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- 172.21.15.119:0/1924162659 shutdown_connections 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- 172.21.15.119:0/1924162659 wait complete. 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- start start 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 Processor -- start 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xccba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xccc230 0xcc4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcccb50 0xc86b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xbaa730 con 0xcaae90 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xbb05b0 con 0xcccb50 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xa98aa0 con 0xccc230 2024-09-13T16:33:07.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 10 client.0 _ll_register_callbacks cb 0xbe6610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xccba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xccba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33374/0 (socket says 172.21.15.119:33374) 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 -- 172.21.15.119:0/854991007 learned_addr learned my addr 172.21.15.119:0/854991007 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79f59e700 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcccb50 0xc86b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79e59c700 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xccc230 0xcc4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 -- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xccc230 msgr2=0xcc4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xccc230 0xcc4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 -- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcccb50 msgr2=0xc86b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcccb50 0xc86b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 -- 172.21.15.119:0/854991007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7980db920 con 0xcaae90 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xccba00 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa790014450 tx=0x7fa790000f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3551d41a49662b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:07.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa77dffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa77dffb700 1 -- 172.21.15.119:0/854991007 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa790014b20 con 0xcaae90 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa77dffb700 1 -- 172.21.15.119:0/854991007 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fa79001c040 con 0xcaae90 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xa211e0 con 0xcaae90 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa77dffb700 1 -- 172.21.15.119:0/854991007 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa790014c80 con 0xcaae90 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 10 client.8520 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.971+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0xbb3220 con 0xcaae90 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa77dffb700 1 -- 172.21.15.119:0/854991007 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xa211e0 con 0xcaae90 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa77dffb700 1 -- 172.21.15.119:0/854991007 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa79002a3f0 con 0xcaae90 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 10 client.8520 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 10 client.8520 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 20 client.8520 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xc82030 0xc82410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 4 client.8520 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:33:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xa6b460 con 0xc82030 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa79e59c700 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xc82030 0xc82410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa77dffb700 10 client.8520 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa79e59c700 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xc82030 0xc82410 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25df38db2c9f2a73 server_cookie=3c67a67200233436 in_seq=0 out_seq=0 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa77dffb700 1 -- 172.21.15.119:0/854991007 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0xa6b460 con 0xc82030 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa77dffb700 10 client.8520 handle_command_reply: tid=0 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 client.8520 shutdown 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 msgr2=0xccba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xccba00 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 crypto rx=0x7fa790014450 tx=0x7fa790000f20 comp rx=0 tx=0).stop 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 shutdown_connections 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xc82030 0xc82410 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcccb50 0xc86b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xccc230 0xcc4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 --2- 172.21.15.119:0/854991007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcaae90 0xccba00 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa77dffb700 0 client.8520 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:07.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 >> 172.21.15.119:0/854991007 conn(0xa144d0 msgr2=0xcc52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:07.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 shutdown_connections 2024-09-13T16:33:07.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 1 -- 172.21.15.119:0/854991007 wait complete. 2024-09-13T16:33:07.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.975+0000 7fa7a8359740 20 client.8520 trim_cache size 0 max 16384 2024-09-13T16:33:08.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 Processor -- start 2024-09-13T16:33:08.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 -- start start 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcaae90 0xc84560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xc84aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcccb50 0xcc36a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xba6c30 con 0xcccb50 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xc1bec0 con 0xccc230 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xbbb5b0 con 0xcaae90 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcccb50 0xcc36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcccb50 0xcc36a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33384/0 (socket says 172.21.15.119:33384) 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 -- 172.21.15.119:0/3735797784 learned_addr learned my addr 172.21.15.119:0/3735797784 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 -- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcaae90 msgr2=0xc84560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcaae90 0xc84560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79e59c700 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xc84aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcaae90 0xc84560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 -- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 msgr2=0xc84aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xc84aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 -- 172.21.15.119:0/3735797784 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xb99450 con 0xcccb50 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcaae90 0xc84560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79e59c700 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xc84aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa79f59e700 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcccb50 0xcc36a0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa7a004f180 tx=0x7fa7a008d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e201dc32483af66a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:07.999+0000 7fa7957fa700 1 -- 172.21.15.119:0/3735797784 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a009a040 con 0xcccb50 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7957fa700 1 -- 172.21.15.119:0/3735797784 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7a0067990 con 0xcccb50 2024-09-13T16:33:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7957fa700 1 -- 172.21.15.119:0/3735797784 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0067d40 con 0xcccb50 2024-09-13T16:33:08.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- 172.21.15.119:0/3735797784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcccb50 msgr2=0xcc36a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcccb50 0xcc36a0 secure :-1 s=READY pgs=1437 cs=0 l=1 rev1=1 crypto rx=0x7fa7a004f180 tx=0x7fa7a008d000 comp rx=0 tx=0).stop 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- 172.21.15.119:0/3735797784 shutdown_connections 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xcccb50 0xcc36a0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xccc230 0xc84aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 --2- 172.21.15.119:0/3735797784 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcaae90 0xc84560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- 172.21.15.119:0/3735797784 >> 172.21.15.119:0/3735797784 conn(0xa144d0 msgr2=0xc80f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- 172.21.15.119:0/3735797784 shutdown_connections 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- 172.21.15.119:0/3735797784 wait complete. 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- start start 2024-09-13T16:33:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 Processor -- start 2024-09-13T16:33:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcaae90 0xce9850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xccc230 0xce9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xd2cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xccbba0 con 0xccc230 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xceb180 con 0xcaae90 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.003+0000 7fa7a8359740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xc97490 con 0xcccb50 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 10 client.0 _ll_register_callbacks cb 0xb9ece0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xccc230 0xce9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xccc230 0xce9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33390/0 (socket says 172.21.15.119:33390) 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 -- 172.21.15.119:0/833025822 learned_addr learned my addr 172.21.15.119:0/833025822 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79f59e700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xd2cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcaae90 0xce9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 -- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 msgr2=0xd2cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xd2cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 -- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcaae90 msgr2=0xce9850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcaae90 0xce9850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 -- 172.21.15.119:0/833025822 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa798006690 con 0xccc230 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79f59e700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xd2cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcaae90 0xce9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79e59c700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xccc230 0xce9d90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fa7880818b0 tx=0x7fa78801e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afbe4cc12e4d3e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 1 -- 172.21.15.119:0/833025822 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa788081040 con 0xccc230 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 1 -- 172.21.15.119:0/833025822 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fa7880527b0 con 0xccc230 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa78805c750 con 0xccc230 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 1 -- 172.21.15.119:0/833025822 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa788050750 con 0xccc230 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 10 client.8526 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0xb8d720 con 0xccc230 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 1 -- 172.21.15.119:0/833025822 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa78805c750 con 0xccc230 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 1 -- 172.21.15.119:0/833025822 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa788053900 con 0xccc230 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 10 client.8526 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 10 client.8526 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 20 client.8526 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xcd0270 0xcd0650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 4 client.8526 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:33:08.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "status"}) v1 -- 0xa901e0 con 0xcd0270 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xcd0270 0xcd0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 10 client.8526 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa79ed9d700 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xcd0270 0xcd0650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c95f78ff1086d5c2 server_cookie=8cd042164606b857 in_seq=0 out_seq=0 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 1 -- 172.21.15.119:0/833025822 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0xa901e0 con 0xcd0270 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 10 client.8526 handle_command_reply: tid=0 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 client.8526 shutdown 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xccc230 msgr2=0xce9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xccc230 0xce9d90 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 crypto rx=0x7fa7880818b0 tx=0x7fa78801e040 comp rx=0 tx=0).stop 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 shutdown_connections 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0xcd0270 0xcd0650 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xcccb50 0xd2cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa77dffb700 0 client.8526 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xccc230 0xce9d90 unknown :-1 s=CLOSED pgs=1438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 --2- 172.21.15.119:0/833025822 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0xcaae90 0xce9850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 >> 172.21.15.119:0/833025822 conn(0xa144d0 msgr2=0xca9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 shutdown_connections 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.007+0000 7fa7a8359740 1 -- 172.21.15.119:0/833025822 wait complete. 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a8359740 20 client.8526 trim_cache size 0 max 16384 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:33:08.014 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 8442, 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 4.5093735349999999, 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 69, 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 99, 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 99, 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 5.4793663400000003 2024-09-13T16:33:08.015 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:33:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa780066760 msgr2=0x7fa780068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa780066760 0x7fa780068c20 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7fa790001870 tx=0x7fa790009040 comp rx=0 tx=0).stop 2024-09-13T16:33:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa79800a5b0 msgr2=0x7fa7980da670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa79800a5b0 0x7fa7980da670 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 crypto rx=0x7fa78800c500 tx=0x7fa78800c810 comp rx=0 tx=0).stop 2024-09-13T16:33:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 shutdown_connections 2024-09-13T16:33:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa780066760 0x7fa780068c20 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa7980e18c0 0x7fa79817ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa79800a5b0 0x7fa7980da670 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 --2- 172.21.15.119:0/832311073 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa798005b10 0x7fa7980d96f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 >> 172.21.15.119:0/832311073 conn(0x7fa798036140 msgr2=0x7fa7980350f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 shutdown_connections 2024-09-13T16:33:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.011+0000 7fa7a48a7700 1 -- 172.21.15.119:0/832311073 wait complete. 2024-09-13T16:33:08.027 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:33:08.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.147+0000 7fab92478700 1 Processor -- start 2024-09-13T16:33:08.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.147+0000 7fab92478700 1 -- start start 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c158fd0 0x7fab8c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab8c0721f0 0x7fab8c06dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8c071d50 con 0x7fab8c0abf40 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8c0726a0 con 0x7fab8c0721f0 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c06d350 con 0x7fab8c158fd0 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab90c75700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c158fd0 0x7fab8c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91c77700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab8c0721f0 0x7fab8c06dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91476700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fab8c0abf40 0x7fab8c0ac320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:56516/0 (socket says 172.21.15.119:56516) 2024-09-13T16:33:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab90c75700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c158fd0 0x7fab8c06cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:58774/0 (socket says 172.21.15.119:58774) 2024-09-13T16:33:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91c77700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab8c0721f0 0x7fab8c06dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:42978/0 (socket says 172.21.15.119:42978) 2024-09-13T16:33:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91476700 1 -- 172.21.15.119:0/3433300208 learned_addr learned my addr 172.21.15.119:0/3433300208 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91c77700 1 -- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c158fd0 msgr2=0x7fab8c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91c77700 1 --2- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c158fd0 0x7fab8c06cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91c77700 1 -- 172.21.15.119:0/3433300208 >> v1:172.21.15.145:6789/0 conn(0x7fab8c0abf40 legacy=0x7fab8c0ac320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91c77700 1 -- 172.21.15.119:0/3433300208 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8c0acec0 con 0x7fab8c0721f0 2024-09-13T16:33:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab83fff700 1 -- 172.21.15.119:0/3433300208 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823495781 0 0) 0x7fab8c071d50 con 0x7fab8c0abf40 2024-09-13T16:33:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab90c75700 1 --2- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c158fd0 0x7fab8c06cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab91c77700 1 --2- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c0721f0 0x7fab8c06dc80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fab84009d90 tx=0x7fab8400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=386fb2fef8e2f7e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab83fff700 1 -- 172.21.15.119:0/3433300208 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab8400e070 con 0x7fab8c0721f0 2024-09-13T16:33:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab83fff700 1 -- 172.21.15.119:0/3433300208 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab8400bb20 con 0x7fab8c0721f0 2024-09-13T16:33:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab83fff700 1 -- 172.21.15.119:0/3433300208 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab84012430 con 0x7fab8c0721f0 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 -- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c0721f0 msgr2=0x7fab8c06dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 --2- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c0721f0 0x7fab8c06dc80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fab84009d90 tx=0x7fab8400b330 comp rx=0 tx=0).stop 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 -- 172.21.15.119:0/3433300208 shutdown_connections 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 --2- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c0721f0 0x7fab8c06dc80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 --2- 172.21.15.119:0/3433300208 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c158fd0 0x7fab8c06cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.151+0000 7fab92478700 1 -- 172.21.15.119:0/3433300208 >> 172.21.15.119:0/3433300208 conn(0x7fab8c0ae750 msgr2=0x7fab8c0b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 -- 172.21.15.119:0/3433300208 shutdown_connections 2024-09-13T16:33:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 -- 172.21.15.119:0/3433300208 wait complete. 2024-09-13T16:33:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 Processor -- start 2024-09-13T16:33:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 -- start start 2024-09-13T16:33:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab8c0abf40 0x7fab8c0b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c158fd0 0x7fab8c14e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c145f70 0x7fab8c148880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c1f20c0 con 0x7fab8c0abf40 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c1452f0 con 0x7fab8c158fd0 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab92478700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab8c0b41d0 con 0x7fab8c145f70 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab8c0abf40 0x7fab8c0b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab8c0abf40 0x7fab8c0b3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33406/0 (socket says 172.21.15.119:33406) 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 -- 172.21.15.119:0/648671866 learned_addr learned my addr 172.21.15.119:0/648671866 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab90c75700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c158fd0 0x7fab8c14e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91c77700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c145f70 0x7fab8c148880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 -- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c145f70 msgr2=0x7fab8c148880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c145f70 0x7fab8c148880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 -- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c158fd0 msgr2=0x7fab8c14e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c158fd0 0x7fab8c14e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91476700 1 -- 172.21.15.119:0/648671866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0094b0 con 0x7fab8c0abf40 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.155+0000 7fab91c77700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c145f70 0x7fab8c148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab91476700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab8c0abf40 0x7fab8c0b3150 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fab8800c550 tx=0x7fab8800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab81ffb700 1 -- 172.21.15.119:0/648671866 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88004ca0 con 0x7fab8c0abf40 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab81ffb700 1 -- 172.21.15.119:0/648671866 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab88004e00 con 0x7fab8c0abf40 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab84009a70 con 0x7fab8c0abf40 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab81ffb700 1 -- 172.21.15.119:0/648671866 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab88007310 con 0x7fab8c0abf40 2024-09-13T16:33:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab8c0b3930 con 0x7fab8c0abf40 2024-09-13T16:33:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8c073ec0 con 0x7fab8c0abf40 2024-09-13T16:33:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab81ffb700 1 -- 172.21.15.119:0/648671866 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fab88007470 con 0x7fab8c0abf40 2024-09-13T16:33:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab81ffb700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab70066760 0x7fab70068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab81ffb700 1 -- 172.21.15.119:0/648671866 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fab88084100 con 0x7fab8c0abf40 2024-09-13T16:33:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.159+0000 7fab90c75700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab70066760 0x7fab70068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:08.163 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.163+0000 7fab90c75700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab70066760 0x7fab70068c20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fab8c066290 tx=0x7fab7c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.167+0000 7fab81ffb700 1 -- 172.21.15.119:0/648671866 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fab88051100 con 0x7fab8c0abf40 2024-09-13T16:33:08.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.375+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fab8c0cae70 con 0x7fab8c0abf40 2024-09-13T16:33:08.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.375+0000 7fab81ffb700 1 -- 172.21.15.119:0/648671866 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v69) v1 ==== 53+0+73 (secure 0 0 0) 0x7fab88007720 con 0x7fab8c0abf40 2024-09-13T16:33:08.377 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:33:08.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.375+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab70066760 msgr2=0x7fab70068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.375+0000 7fab92478700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab70066760 0x7fab70068c20 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7fab8c066290 tx=0x7fab7c009240 comp rx=0 tx=0).stop 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab8c0abf40 msgr2=0x7fab8c0b3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab8c0abf40 0x7fab8c0b3150 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 crypto rx=0x7fab8800c550 tx=0x7fab8800c860 comp rx=0 tx=0).stop 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 shutdown_connections 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab70066760 0x7fab70068c20 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab8c145f70 0x7fab8c148880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab8c158fd0 0x7fab8c14e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 --2- 172.21.15.119:0/648671866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab8c0abf40 0x7fab8c0b3150 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 >> 172.21.15.119:0/648671866 conn(0x7fab8c0721f0 msgr2=0x7fab8c0aec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 shutdown_connections 2024-09-13T16:33:08.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:08.379+0000 7fab92478700 1 -- 172.21.15.119:0/648671866 wait complete. 2024-09-13T16:33:08.391 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:33:08.392 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:33:08.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.393 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:33:08.400 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:33:08.401 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42153sec preferred_lft 42153sec 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:33:08.402 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:33:08.403 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:33:08.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.404 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:33:08.417 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:33:08.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.419 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:33:08.428 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:33:08.428 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:33:08.428 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:33:08.428 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:33:08.428 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:33:08.428 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:33:08.429 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:33:08.429 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:33:08.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.429 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:33:08.429 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:33:08.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:33:08.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:33:08.429 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:33:08.430 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:33:08.503 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:33:08.503+0000 7f6b98b92700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:33:08.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.508 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:33:08.516 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:33:08.516 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:33:08.517 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:33:08.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:33:08.529 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:33:08.529 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:33:08.529 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:08.534 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:33:08.534 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:33:08.534 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:08.539 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:33:08.599 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:33:08.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:33:08.609 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:33:08.610 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:33:08.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.611 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:33:08.616 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:33:08.616 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:33:08.618 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:33:08.635 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:33:08.635 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:33:08.693 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:33:08.694 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:33:08.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:08.695 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:33:08.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:33:08.703+0000 7f3ee7759640 -1 init, newargv = 0x559e03609250 newargc=15 2024-09-13T16:33:08.704 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[49131]: starting ceph client 2024-09-13T16:33:08.711 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[49131]: starting fuse 2024-09-13T16:33:09.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:33:09.819 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:33:09.819 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:33:09.878 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:33:09.879 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:33:09.880 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:09.880 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:33:09.886 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:33:09.887 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:33:09.887 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:33:09.887 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:33:09.887 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:33:09.887 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:33:09.887 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:33:09.887 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:33:09.888 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:33:09.889 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:33:09.889 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:33:09.889 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:33:09.889 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:33:09.889 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:33:09.889 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:33:09.889 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:33:09.970 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.49131.asok 2024-09-13T16:33:10.153 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.49131.asok 2024-09-13T16:33:10.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:10.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.49131.asok status 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "49131", 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:33:10.300 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 8538, 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 8538 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2388457104 2024-09-13T16:33:10.301 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2388457104 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.8538 192.168.0.1:0/2388457104", 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/2388457104", 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 69, 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 99, 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:33:10.302 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:33:10.303 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:33:10.303 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:33:10.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:10.309 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:33:10.309 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:10.317 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:33:10.317 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:10.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:10.318 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:10.333 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub start /dir recursive,repair 2024-09-13T16:33:10.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 Processor -- start 2024-09-13T16:33:10.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- start start 2024-09-13T16:33:10.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4ee806d9d0 0x7f4ee80b3a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4ee80cba30 0x7f4ee806bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ee80705f0 con 0x7f4ee806fe30 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee80708d0 con 0x7f4ee80cba30 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ee8070bb0 con 0x7f4ee806d9d0 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eedba2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4ee806fe30 0x7f4ee80cb320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:42366/0 (socket says 172.21.15.119:42366) 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eeeba4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4ee80cba30 0x7f4ee806bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eedba2700 1 -- 172.21.15.119:0/3012353657 learned_addr learned my addr 172.21.15.119:0/3012353657 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/3012353657 >> v2:172.21.15.145:3300/0 conn(0x7f4ee806d9d0 0x7f4ee80b3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eeeba4700 1 -- 172.21.15.119:0/3012353657 >> v2:172.21.15.145:3300/0 conn(0x7f4ee806d9d0 msgr2=0x7f4ee80b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/3012353657 >> v2:172.21.15.145:3300/0 conn(0x7f4ee806d9d0 0x7f4ee80b3a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eeeba4700 1 -- 172.21.15.119:0/3012353657 >> v1:172.21.15.119:6789/0 conn(0x7f4ee806fe30 legacy=0x7f4ee80cb320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eeeba4700 1 -- 172.21.15.119:0/3012353657 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee8070e90 con 0x7f4ee80cba30 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/3012353657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee80cba30 0x7f4ee806bb80 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8071f80 tx=0x7f4ee4009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bee643fe835db412 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eed3a1700 1 -- 172.21.15.119:0/3012353657 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4011830 con 0x7f4ee80cba30 2024-09-13T16:33:10.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eed3a1700 1 -- 172.21.15.119:0/3012353657 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee4011990 con 0x7f4ee80cba30 2024-09-13T16:33:10.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eed3a1700 1 -- 172.21.15.119:0/3012353657 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4011d40 con 0x7f4ee80cba30 2024-09-13T16:33:10.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- 172.21.15.119:0/3012353657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee80cba30 msgr2=0x7f4ee806bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/3012353657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee80cba30 0x7f4ee806bb80 secure :-1 s=READY pgs=1444 cs=0 l=1 rev1=1 crypto rx=0x7f4ee8071f80 tx=0x7f4ee4009b30 comp rx=0 tx=0).stop 2024-09-13T16:33:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- 172.21.15.119:0/3012353657 shutdown_connections 2024-09-13T16:33:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/3012353657 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee80cba30 0x7f4ee806bb80 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/3012353657 >> v2:172.21.15.145:3300/0 conn(0x7f4ee806d9d0 0x7f4ee80b3a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- 172.21.15.119:0/3012353657 >> 172.21.15.119:0/3012353657 conn(0x7f4ee806a350 msgr2=0x7f4ee806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- 172.21.15.119:0/3012353657 shutdown_connections 2024-09-13T16:33:10.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.502+0000 7f4eef3a5700 1 -- 172.21.15.119:0/3012353657 wait complete. 2024-09-13T16:33:10.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 Processor -- start 2024-09-13T16:33:10.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 -- start start 2024-09-13T16:33:10.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ee806d9d0 0x7f4ee80a85d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee806fe30 0x7f4ee815ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80cba30 0x7f4ee80acf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80addf0 con 0x7f4ee806fe30 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee815c360 con 0x7f4ee806d9d0 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eef3a5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee80adb10 con 0x7f4ee80cba30 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eedba2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee806fe30 0x7f4ee815ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eedba2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee806fe30 0x7f4ee815ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33422/0 (socket says 172.21.15.119:33422) 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eedba2700 1 -- 172.21.15.119:0/2617706354 learned_addr learned my addr 172.21.15.119:0/2617706354 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.506+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ee806d9d0 0x7f4ee80a85d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80cba30 0x7f4ee80acf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eedba2700 1 -- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80cba30 msgr2=0x7f4ee80acf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eedba2700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80cba30 0x7f4ee80acf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eedba2700 1 -- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ee806d9d0 msgr2=0x7f4ee80a85d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eedba2700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ee806d9d0 0x7f4ee80a85d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eedba2700 1 -- 172.21.15.119:0/2617706354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee40094b0 con 0x7f4ee806fe30 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80cba30 0x7f4ee80acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eedba2700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee806fe30 0x7f4ee815ad50 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000cba0 tx=0x7f4ee000cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4edeffd700 1 -- 172.21.15.119:0/2617706354 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee000a7f0 con 0x7f4ee806fe30 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4edeffd700 1 -- 172.21.15.119:0/2617706354 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee000a950 con 0x7f4ee806fe30 2024-09-13T16:33:10.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee815bd10 con 0x7f4ee806fe30 2024-09-13T16:33:10.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4edeffd700 1 -- 172.21.15.119:0/2617706354 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee000acc0 con 0x7f4ee806fe30 2024-09-13T16:33:10.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee81f7500 con 0x7f4ee806fe30 2024-09-13T16:33:10.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4edeffd700 1 -- 172.21.15.119:0/2617706354 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4ee001b3b0 con 0x7f4ee806fe30 2024-09-13T16:33:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4edeffd700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ecc066760 0x7f4ecc068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4edeffd700 1 -- 172.21.15.119:0/2617706354 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f4ee0083c90 con 0x7f4ee806fe30 2024-09-13T16:33:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.510+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ecc066760 0x7f4ecc068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ecc066760 0x7f4ecc068c20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009a00 tx=0x7f4ed8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 Processor -- start 2024-09-13T16:33:10.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 -- start start 2024-09-13T16:33:10.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x262f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x25e8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25e9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x255a500 con 0x26121f0 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2557c90 con 0x25e92b0 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2553580 con 0x26125d0 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4eee3a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x262f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4eee3a3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x262f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33432/0 (socket says 172.21.15.119:33432) 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4eee3a3700 1 -- 172.21.15.119:0/3237386045 learned_addr learned my addr 172.21.15.119:0/3237386045 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25e9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.514+0000 7f4eedba2700 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x25e8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4eee3a3700 1 -- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 msgr2=0x25e8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x25e8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4eee3a3700 1 -- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 msgr2=0x25e9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25e9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4eee3a3700 1 -- 172.21.15.119:0/3237386045 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2445ee0 con 0x26121f0 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x262f730 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8011950 tx=0x7f4ed8014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9665731ae33c7c9e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ecbfff700 1 -- 172.21.15.119:0/3237386045 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8015740 con 0x26121f0 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ecbfff700 1 -- 172.21.15.119:0/3237386045 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed80158a0 con 0x26121f0 2024-09-13T16:33:10.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ecbfff700 1 -- 172.21.15.119:0/3237386045 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8015c10 con 0x26121f0 2024-09-13T16:33:10.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3237386045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 msgr2=0x262f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x262f730 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8011950 tx=0x7f4ed8014000 comp rx=0 tx=0).stop 2024-09-13T16:33:10.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3237386045 shutdown_connections 2024-09-13T16:33:10.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25e9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x25e8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3237386045 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x262f730 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3237386045 >> 172.21.15.119:0/3237386045 conn(0x237a780 msgr2=0x2614c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3237386045 shutdown_connections 2024-09-13T16:33:10.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3237386045 wait complete. 2024-09-13T16:33:10.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 -- start start 2024-09-13T16:33:10.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.518+0000 7f4ef38b8740 1 Processor -- start 2024-09-13T16:33:10.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x25ed870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26121f0 0x2614640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26125d0 0x263c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2516920 con 0x26125d0 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x25165c0 con 0x26121f0 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x24e8e10 con 0x25e92b0 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 10 client.0 _ll_register_callbacks cb 0x254cbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26125d0 0x263c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26125d0 0x263c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33448/0 (socket says 172.21.15.119:33448) 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 -- 172.21.15.119:0/671501139 learned_addr learned my addr 172.21.15.119:0/671501139 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x25ed870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eedba2700 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26121f0 0x2614640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 -- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 msgr2=0x25ed870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x25ed870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 -- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26121f0 msgr2=0x2614640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26121f0 0x2614640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 -- 172.21.15.119:0/671501139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed800e040 con 0x26125d0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26125d0 0x263c080 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4014040 tx=0x7f4ee400e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=733f5da43d3dfd83 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ec97fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ec97fa700 1 -- 172.21.15.119:0/671501139 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4020040 con 0x26125d0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ec97fa700 1 -- 172.21.15.119:0/671501139 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f4ee4004450 con 0x26125d0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x2387490 con 0x26125d0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ec97fa700 1 -- 172.21.15.119:0/671501139 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ee4004610 con 0x26125d0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 10 client.8553 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x2519600 con 0x26125d0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ec97fa700 1 -- 172.21.15.119:0/671501139 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x2387490 con 0x26125d0 2024-09-13T16:33:10.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ec97fa700 1 -- 172.21.15.119:0/671501139 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4ee40036c0 con 0x26125d0 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 10 client.8553 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 10 client.8553 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 20 client.8553 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2614b80 0x2612c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 4 client.8553 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x23d1710 con 0x2614b80 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2614b80 0x2612c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4ec97fa700 10 client.8553 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.522+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2614b80 0x2612c60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cef27d19e830c92 server_cookie=c7a7fb782c4075f9 in_seq=0 out_seq=0 2024-09-13T16:33:10.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ec97fa700 1 -- 172.21.15.119:0/671501139 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x23d1710 con 0x2614b80 2024-09-13T16:33:10.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ec97fa700 10 client.8553 handle_command_reply: tid=0 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 client.8553 shutdown 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26125d0 msgr2=0x263c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26125d0 0x263c080 secure :-1 s=READY pgs=1447 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4014040 tx=0x7f4ee400e000 comp rx=0 tx=0).stop 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 shutdown_connections 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2614b80 0x2612c60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26125d0 0x263c080 unknown :-1 s=CLOSED pgs=1447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26121f0 0x2614640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/671501139 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x25ed870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 >> 172.21.15.119:0/671501139 conn(0x237a780 msgr2=0x262ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ec97fa700 0 client.8553 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 shutdown_connections 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 1 -- 172.21.15.119:0/671501139 wait complete. 2024-09-13T16:33:10.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.526+0000 7f4ef38b8740 20 client.8553 trim_cache size 0 max 16384 2024-09-13T16:33:10.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 Processor -- start 2024-09-13T16:33:10.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 -- start start 2024-09-13T16:33:10.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25ed1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x25ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x26141e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2522db0 con 0x26121f0 2024-09-13T16:33:10.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x254d980 con 0x25e92b0 2024-09-13T16:33:10.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x250d010 con 0x26125d0 2024-09-13T16:33:10.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x25ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x25ed6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33464/0 (socket says 172.21.15.119:33464) 2024-09-13T16:33:10.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 -- 172.21.15.119:0/3673056427 learned_addr learned my addr 172.21.15.119:0/3673056427 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x26141e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25ed1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 -- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 msgr2=0x26141e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x26141e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 -- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 msgr2=0x25ed1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25ed1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 -- 172.21.15.119:0/3673056427 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x24ff830 con 0x26121f0 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eeeba4700 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x26141e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4eedba2700 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x25ed6f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00510a0 tx=0x7f4ee0051010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=447dc5a0287d4d1a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4edcff9700 1 -- 172.21.15.119:0/3673056427 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee005cee0 con 0x26121f0 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4edcff9700 1 -- 172.21.15.119:0/3673056427 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ee0052f40 con 0x26121f0 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.546+0000 7f4edcff9700 1 -- 172.21.15.119:0/3673056427 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee00542b0 con 0x26121f0 2024-09-13T16:33:10.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3673056427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 msgr2=0x25ed6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x25ed6f0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7f4ee00510a0 tx=0x7f4ee0051010 comp rx=0 tx=0).stop 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3673056427 shutdown_connections 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x26125d0 0x26141e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x25ed6f0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/3673056427 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25e92b0 0x25ed1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3673056427 >> 172.21.15.119:0/3673056427 conn(0x237a780 msgr2=0x263b490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3673056427 shutdown_connections 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 -- 172.21.15.119:0/3673056427 wait complete. 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 -- start start 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 Processor -- start 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x2649230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.550+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x2649970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26125d0 0x264a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2579480 con 0x26121f0 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2648d00 con 0x26125d0 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2615530 con 0x25e92b0 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 10 client.0 _ll_register_callbacks cb 0x2552150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x2649970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eee3a3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x2649230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x2649970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33474/0 (socket says 172.21.15.119:33474) 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eeeba4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26125d0 0x264a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 -- 172.21.15.119:0/1973601836 learned_addr learned my addr 172.21.15.119:0/1973601836 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 -- 172.21.15.119:0/1973601836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 msgr2=0x2649230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x2649230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 -- 172.21.15.119:0/1973601836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26125d0 msgr2=0x264a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26125d0 0x264a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 -- 172.21.15.119:0/1973601836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee81f7500 con 0x26121f0 2024-09-13T16:33:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eedba2700 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x2649970 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000cfd0 tx=0x7f4ee0001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c55988bbdcdcd39e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ec97fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ec97fa700 1 -- 172.21.15.119:0/1973601836 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ee0001c80 con 0x26121f0 2024-09-13T16:33:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ec97fa700 1 -- 172.21.15.119:0/1973601836 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f4ee005cee0 con 0x26121f0 2024-09-13T16:33:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4ed80158a0 con 0x26121f0 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ec97fa700 1 -- 172.21.15.119:0/1973601836 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4ee006c210 con 0x26121f0 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 10 client.8559 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x23c5c10 con 0x26121f0 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ec97fa700 1 -- 172.21.15.119:0/1973601836 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed80158a0 con 0x26121f0 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ec97fa700 1 -- 172.21.15.119:0/1973601836 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f4ee006c370 con 0x26121f0 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 10 client.8559 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 10 client.8559 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 20 client.8559 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2694520 0x2694930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 4 client.8559 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}] 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "scrub start", "path": "/dir", "scrubops": ["recursive", "repair"]}) v1 -- 0x23f85f0 con 0x2694520 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2694520 0x2694930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4ec97fa700 10 client.8559 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.554+0000 7f4eee3a3700 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2694520 0x2694930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5f1bcbf7376a0433 server_cookie=d793a552de4ab09c in_seq=0 out_seq=0 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ec97fa700 1 -- 172.21.15.119:0/1973601836 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x23f85f0 con 0x2694520 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ec97fa700 10 client.8559 handle_command_reply: tid=0 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 client.8559 shutdown 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 msgr2=0x2649970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x2649970 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000cfd0 tx=0x7f4ee0001250 comp rx=0 tx=0).stop 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 shutdown_connections 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2694520 0x2694930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x26125d0 0x264a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x26121f0 0x2649970 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 --2- 172.21.15.119:0/1973601836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x25e92b0 0x2649230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 >> 172.21.15.119:0/1973601836 conn(0x237a780 msgr2=0x263bb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ec97fa700 0 client.8559 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 shutdown_connections 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 1 -- 172.21.15.119:0/1973601836 wait complete. 2024-09-13T16:33:10.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4ef38b8740 20 client.8559 trim_cache size 0 max 16384 2024-09-13T16:33:10.564 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:33:10.564 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:33:10.564 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "fff368ea-9e5a-4ac3-9ffe-fceb635f3265", 2024-09-13T16:33:10.564 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:33:10.564 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:33:10.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ecc066760 msgr2=0x7f4ecc068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ecc066760 0x7f4ecc068c20 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f4ed8009a00 tx=0x7f4ed8008040 comp rx=0 tx=0).stop 2024-09-13T16:33:10.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee806fe30 msgr2=0x7f4ee815ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee806fe30 0x7f4ee815ad50 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000cba0 tx=0x7f4ee000cbd0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 shutdown_connections 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4ecc066760 0x7f4ecc068c20 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4ee80cba30 0x7f4ee80acf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4ee806fe30 0x7f4ee815ad50 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 --2- 172.21.15.119:0/2617706354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4ee806d9d0 0x7f4ee80a85d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 >> 172.21.15.119:0/2617706354 conn(0x7f4ee806a350 msgr2=0x7f4ee80ae550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 shutdown_connections 2024-09-13T16:33:10.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f4eef3a5700 1 -- 172.21.15.119:0/2617706354 wait complete. 2024-09-13T16:33:10.566 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:33:10.558+0000 7f6b9538b700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55d31711c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(14)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(14)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v20>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:33:10.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-09-13T16:33:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.694+0000 7fe4a89d4700 1 Processor -- start 2024-09-13T16:33:10.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- start start 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a415ccb0 0x7fe4a40d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe4a40d0da0 0x7fe4a40b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe4a40b54a0 0x7fe4a406bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a40d16c0 con 0x7fe4a40d0da0 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a40d11e0 con 0x7fe4a40b54a0 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40b5950 con 0x7fe4a415ccb0 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe4a40d0da0 0x7fe4a40b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe4a40d0da0 0x7fe4a40b4f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33488/0 (socket says 172.21.15.119:33488) 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 -- 172.21.15.119:0/348247389 learned_addr learned my addr 172.21.15.119:0/348247389 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/348247389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a415ccb0 0x7fe4a40d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a359e700 1 --2- 172.21.15.119:0/348247389 >> v2:172.21.15.145:3300/0 conn(0x7fe4a40b54a0 0x7fe4a406bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 -- 172.21.15.119:0/348247389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a415ccb0 msgr2=0x7fe4a40d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 --2- 172.21.15.119:0/348247389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a415ccb0 0x7fe4a40d0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 -- 172.21.15.119:0/348247389 >> v2:172.21.15.145:3300/0 conn(0x7fe4a40b54a0 msgr2=0x7fe4a406bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 --2- 172.21.15.119:0/348247389 >> v2:172.21.15.145:3300/0 conn(0x7fe4a40b54a0 0x7fe4a406bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 -- 172.21.15.119:0/348247389 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40b5bb0 con 0x7fe4a40d0da0 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a259c700 1 --2- 172.21.15.119:0/348247389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a40d0da0 0x7fe4a40b4f60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40721a0 tx=0x7fe494009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fcad3799ada0889 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a1d9b700 1 -- 172.21.15.119:0/348247389 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49400b8a0 con 0x7fe4a40d0da0 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a1d9b700 1 -- 172.21.15.119:0/348247389 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49400ba00 con 0x7fe4a40d0da0 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a1d9b700 1 -- 172.21.15.119:0/348247389 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49400bd90 con 0x7fe4a40d0da0 2024-09-13T16:33:10.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- 172.21.15.119:0/348247389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a40d0da0 msgr2=0x7fe4a40b4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/348247389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a40d0da0 0x7fe4a40b4f60 secure :-1 s=READY pgs=1450 cs=0 l=1 rev1=1 crypto rx=0x7fe4a40721a0 tx=0x7fe494009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- 172.21.15.119:0/348247389 shutdown_connections 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/348247389 >> v2:172.21.15.145:3300/0 conn(0x7fe4a40b54a0 0x7fe4a406bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/348247389 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a40d0da0 0x7fe4a40b4f60 unknown :-1 s=CLOSED pgs=1450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/348247389 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a415ccb0 0x7fe4a40d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- 172.21.15.119:0/348247389 >> 172.21.15.119:0/348247389 conn(0x7fe4a4072240 msgr2=0x7fe4a4099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.698+0000 7fe4a89d4700 1 -- 172.21.15.119:0/348247389 shutdown_connections 2024-09-13T16:33:10.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 -- 172.21.15.119:0/348247389 wait complete. 2024-09-13T16:33:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 Processor -- start 2024-09-13T16:33:10.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 -- start start 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4a40b54a0 0x7fe4a414b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a40d0da0 0x7fe4a415b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a415ccb0 0x7fe4a415bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a409b370 con 0x7fe4a415ccb0 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4152820 con 0x7fe4a40b54a0 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a89d4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a414b900 con 0x7fe4a40d0da0 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a415ccb0 0x7fe4a415bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a415ccb0 0x7fe4a415bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33490/0 (socket says 172.21.15.119:33490) 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 -- 172.21.15.119:0/1746588718 learned_addr learned my addr 172.21.15.119:0/1746588718 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4a40b54a0 0x7fe4a414b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a259c700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a40d0da0 0x7fe4a415b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 -- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a40d0da0 msgr2=0x7fe4a415b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a40d0da0 0x7fe4a415b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 -- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4a40b54a0 msgr2=0x7fe4a414b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4a40b54a0 0x7fe4a414b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 -- 172.21.15.119:0/1746588718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe494009570 con 0x7fe4a415ccb0 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.702+0000 7fe4a359e700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a415ccb0 0x7fe4a415bc20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fe49c007e90 tx=0x7fe49c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe48b7fe700 1 -- 172.21.15.119:0/1746588718 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c00ed30 con 0x7fe4a415ccb0 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe48b7fe700 1 -- 172.21.15.119:0/1746588718 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c00eeb0 con 0x7fe4a415ccb0 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a41f6750 con 0x7fe4a415ccb0 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe48b7fe700 1 -- 172.21.15.119:0/1746588718 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c014640 con 0x7fe4a415ccb0 2024-09-13T16:33:10.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a409acb0 con 0x7fe4a415ccb0 2024-09-13T16:33:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe48b7fe700 1 -- 172.21.15.119:0/1746588718 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe49c0147a0 con 0x7fe4a415ccb0 2024-09-13T16:33:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe48b7fe700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe490066730 0x7fe490068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe48b7fe700 1 -- 172.21.15.119:0/1746588718 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fe49c084a40 con 0x7fe4a415ccb0 2024-09-13T16:33:10.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.706+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe490066730 0x7fe490068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe490066730 0x7fe490068bf0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe498006b40 tx=0x7fe498008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 Processor -- start 2024-09-13T16:33:10.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 -- start start 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x24bf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x24e7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2509170 0x2509550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2450010 con 0x2509170 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x23ddf40 con 0x24be310 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2430e30 con 0x2508d90 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2509170 0x2509550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x24bf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x24bf810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:58864/0 (socket says 172.21.15.119:58864) 2024-09-13T16:33:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/2301694701 learned_addr learned my addr 172.21.15.119:0/2301694701 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a259c700 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x24e7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a359e700 1 -- 172.21.15.119:0/2301694701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 msgr2=0x24bf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a359e700 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x24bf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a359e700 1 -- 172.21.15.119:0/2301694701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 msgr2=0x24e7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a359e700 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x24e7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a359e700 1 -- 172.21.15.119:0/2301694701 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x229ca40 con 0x2509170 2024-09-13T16:33:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe4a359e700 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2509170 0x2509550 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fe49c082d60 tx=0x7fe49c055f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f43c43696fff328 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe488ff9700 1 -- 172.21.15.119:0/2301694701 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c084040 con 0x2509170 2024-09-13T16:33:10.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe488ff9700 1 -- 172.21.15.119:0/2301694701 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c059640 con 0x2509170 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.710+0000 7fe488ff9700 1 -- 172.21.15.119:0/2301694701 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c05dc00 con 0x2509170 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- 172.21.15.119:0/2301694701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2509170 msgr2=0x2509550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2509170 0x2509550 secure :-1 s=READY pgs=1452 cs=0 l=1 rev1=1 crypto rx=0x7fe49c082d60 tx=0x7fe49c055f90 comp rx=0 tx=0).stop 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- 172.21.15.119:0/2301694701 shutdown_connections 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2509170 0x2509550 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x24e7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2301694701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x24bf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- 172.21.15.119:0/2301694701 >> 172.21.15.119:0/2301694701 conn(0x22515b0 msgr2=0x24c2b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- 172.21.15.119:0/2301694701 shutdown_connections 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- 172.21.15.119:0/2301694701 wait complete. 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- start start 2024-09-13T16:33:10.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 Processor -- start 2024-09-13T16:33:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24be310 0x24e3410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x24e3950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2509170 0x25074a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x242c150 con 0x24be310 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2429eb0 con 0x2508d90 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x23e78e0 con 0x2509170 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 10 client.0 _ll_register_callbacks cb 0x23c6670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24be310 0x24e3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24be310 0x24e3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33512/0 (socket says 172.21.15.119:33512) 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1834956949 learned_addr learned my addr 172.21.15.119:0/1834956949 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a359e700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2509170 0x25074a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a259c700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x24e3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2509170 msgr2=0x25074a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2509170 0x25074a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 msgr2=0x24e3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x24e3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1834956949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a409acb0 con 0x24be310 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a359e700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2509170 0x25074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24be310 0x24e3410 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fe498010cf0 tx=0x7fe498004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6dd2d700a3341bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe481ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe481ffb700 1 -- 172.21.15.119:0/1834956949 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4980045d0 con 0x24be310 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe481ffb700 1 -- 172.21.15.119:0/1834956949 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fe498016480 con 0x24be310 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x225e2c0 con 0x24be310 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe481ffb700 1 -- 172.21.15.119:0/1834956949 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe498016690 con 0x24be310 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.714+0000 7fe4ac486740 10 client.8571 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:10.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x23f0290 con 0x24be310 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe481ffb700 1 -- 172.21.15.119:0/1834956949 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x225e2c0 con 0x24be310 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe481ffb700 1 -- 172.21.15.119:0/1834956949 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe498022070 con 0x24be310 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 10 client.8571 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 10 client.8571 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 20 client.8571 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2507ed0 0x24e5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 4 client.8571 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x22a8540 con 0x2507ed0 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4a259c700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2507ed0 0x24e5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe481ffb700 10 client.8571 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4a259c700 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2507ed0 0x24e5dd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f18dc7d8e04399a3 server_cookie=3cde0a1b375bc02f in_seq=0 out_seq=0 2024-09-13T16:33:10.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe481ffb700 1 -- 172.21.15.119:0/1834956949 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x22a8540 con 0x2507ed0 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe481ffb700 10 client.8571 handle_command_reply: tid=0 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 client.8571 shutdown 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24be310 msgr2=0x24e3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24be310 0x24e3410 secure :-1 s=READY pgs=1453 cs=0 l=1 rev1=1 crypto rx=0x7fe498010cf0 tx=0x7fe498004770 comp rx=0 tx=0).stop 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 shutdown_connections 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2507ed0 0x24e5dd0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2509170 0x25074a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x24e3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1834956949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x24be310 0x24e3410 unknown :-1 s=CLOSED pgs=1453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe481ffb700 0 client.8571 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 >> 172.21.15.119:0/1834956949 conn(0x22515b0 msgr2=0x24e74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 shutdown_connections 2024-09-13T16:33:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 1 -- 172.21.15.119:0/1834956949 wait complete. 2024-09-13T16:33:10.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.718+0000 7fe4ac486740 20 client.8571 trim_cache size 0 max 16384 2024-09-13T16:33:10.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.738+0000 7fe4ac486740 1 Processor -- start 2024-09-13T16:33:10.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.738+0000 7fe4ac486740 1 -- start start 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x25120a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24be310 0x2505fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x25064e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x22d5b80 con 0x22515b0 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x23b3a50 con 0x2508d90 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x24276c0 con 0x24be310 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x25120a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x25120a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33514/0 (socket says 172.21.15.119:33514) 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/2950476959 learned_addr learned my addr 172.21.15.119:0/2950476959 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a259c700 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24be310 0x2505fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a359e700 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x25064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24be310 msgr2=0x2505fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24be310 0x2505fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 msgr2=0x25064e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x25064e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/2950476959 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x24c38d0 con 0x22515b0 2024-09-13T16:33:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x25120a0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fe498006eb0 tx=0x7fe49801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0be1193f9f2e554 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4897fa700 1 -- 172.21.15.119:0/2950476959 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498038030 con 0x22515b0 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4897fa700 1 -- 172.21.15.119:0/2950476959 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49801b070 con 0x22515b0 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4897fa700 1 -- 172.21.15.119:0/2950476959 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498016a70 con 0x22515b0 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- 172.21.15.119:0/2950476959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 msgr2=0x25120a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x25120a0 secure :-1 s=READY pgs=1454 cs=0 l=1 rev1=1 crypto rx=0x7fe498006eb0 tx=0x7fe49801f000 comp rx=0 tx=0).stop 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- 172.21.15.119:0/2950476959 shutdown_connections 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2508d90 0x25064e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x24be310 0x2505fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 --2- 172.21.15.119:0/2950476959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x25120a0 unknown :-1 s=CLOSED pgs=1454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- 172.21.15.119:0/2950476959 >> 172.21.15.119:0/2950476959 conn(0x24c2f10 msgr2=0x24e2ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- 172.21.15.119:0/2950476959 shutdown_connections 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- 172.21.15.119:0/2950476959 wait complete. 2024-09-13T16:33:10.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 -- start start 2024-09-13T16:33:10.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.742+0000 7fe4ac486740 1 Processor -- start 2024-09-13T16:33:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x2510b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x250ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x250f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x23fac00 con 0x22515b0 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2530e10 con 0x24be310 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x25068d0 con 0x2508d90 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 10 client.0 _ll_register_callbacks cb 0x2428610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x2510b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x2510b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:33516/0 (socket says 172.21.15.119:33516) 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1924975062 learned_addr learned my addr 172.21.15.119:0/1924975062 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a359e700 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x250f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a259c700 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x250ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 msgr2=0x250f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x250f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 msgr2=0x250ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x250ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 -- 172.21.15.119:0/1924975062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x23f0290 con 0x22515b0 2024-09-13T16:33:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a2d9d700 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x2510b10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fe498006ac0 tx=0x7fe498015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31408255bf5864ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe481ffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe481ffb700 1 -- 172.21.15.119:0/1924975062 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49802b910 con 0x22515b0 2024-09-13T16:33:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe481ffb700 1 -- 172.21.15.119:0/1924975062 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fe498039070 con 0x22515b0 2024-09-13T16:33:10.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe49c059640 con 0x22515b0 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe481ffb700 1 -- 172.21.15.119:0/1924975062 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe49802ba70 con 0x22515b0 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 10 client.8577 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x23d64c0 con 0x22515b0 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe481ffb700 1 -- 172.21.15.119:0/1924975062 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c059640 con 0x22515b0 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe481ffb700 1 -- 172.21.15.119:0/1924975062 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fe498016480 con 0x22515b0 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 10 client.8577 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 10 client.8577 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 20 client.8577 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2509170 0x250f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 4 client.8577 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x22cd2c0 con 0x2509170 2024-09-13T16:33:10.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a259c700 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2509170 0x250f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe481ffb700 10 client.8577 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.746+0000 7fe4a259c700 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2509170 0x250f650 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b16d75d84d07d8f server_cookie=9f5f175d3a74088c in_seq=0 out_seq=0 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe481ffb700 1 -- 172.21.15.119:0/1924975062 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+263 (crc 0 0 0) 0x22cd2c0 con 0x2509170 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe481ffb700 10 client.8577 handle_command_reply: tid=0 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 client.8577 shutdown 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 msgr2=0x2510b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x2510b10 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7fe498006ac0 tx=0x7fe498015000 comp rx=0 tx=0).stop 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 shutdown_connections 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2509170 0x250f650 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2508d90 0x250f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x24be310 0x250ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe481ffb700 0 client.8577 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:10.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 --2- 172.21.15.119:0/1924975062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x22515b0 0x2510b10 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 >> 172.21.15.119:0/1924975062 conn(0x24c2f10 msgr2=0x24e2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 shutdown_connections 2024-09-13T16:33:10.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 1 -- 172.21.15.119:0/1924975062 wait complete. 2024-09-13T16:33:10.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4ac486740 20 client.8577 trim_cache size 0 max 16384 2024-09-13T16:33:10.758 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: "fff368ea-9e5a-4ac3-9ffe-fceb635f3265": { 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/dir", 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "fff368ea-9e5a-4ac3-9ffe-fceb635f3265", 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,repair" 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:10.759 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:33:10.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe490066730 msgr2=0x7fe490068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe490066730 0x7fe490068bf0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7fe498006b40 tx=0x7fe498008040 comp rx=0 tx=0).stop 2024-09-13T16:33:10.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a415ccb0 msgr2=0x7fe4a415bc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:10.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a415ccb0 0x7fe4a415bc20 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 crypto rx=0x7fe49c007e90 tx=0x7fe49c00e3c0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 shutdown_connections 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe490066730 0x7fe490068bf0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe4a415ccb0 0x7fe4a415bc20 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe4a40d0da0 0x7fe4a415b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 --2- 172.21.15.119:0/1746588718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe4a40b54a0 0x7fe4a414b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 >> 172.21.15.119:0/1746588718 conn(0x7fe4a4072240 msgr2=0x7fe4a4159d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 shutdown_connections 2024-09-13T16:33:10.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:10.750+0000 7fe4a89d4700 1 -- 172.21.15.119:0/1746588718 wait complete. 2024-09-13T16:33:10.770 INFO:tasks.cephfs.filesystem:scrub status for tag:fff368ea-9e5a-4ac3-9ffe-fceb635f3265 - {'path': '/dir', 'tag': 'fff368ea-9e5a-4ac3-9ffe-fceb635f3265', 'options': 'recursive,repair'} 2024-09-13T16:33:28.099 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:33:28.102 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:33:40.770 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.6:0 scrub status 2024-09-13T16:33:40.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f3485abf700 1 Processor -- start 2024-09-13T16:33:40.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f3485abf700 1 -- start start 2024-09-13T16:33:40.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f3485abf700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3480144180 0x7f348006c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f3485abf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cc010 0x7f34800cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f3485abf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f348006cb40 con 0x7f34800ce000 2024-09-13T16:33:40.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f3485abf700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34800b1c50 con 0x7f3480144180 2024-09-13T16:33:40.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f3485abf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f348006ce20 con 0x7f34800cc010 2024-09-13T16:33:40.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f34852be700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f34800ce000 0x7f348015e350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54598/0 (socket says 172.21.15.119:54598) 2024-09-13T16:33:40.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.894+0000 7f34852be700 1 -- 172.21.15.119:0/1328555378 learned_addr learned my addr 172.21.15.119:0/1328555378 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:40.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3484abd700 1 --2- 172.21.15.119:0/1328555378 >> v2:172.21.15.145:3300/0 conn(0x7f3480144180 0x7f348006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f347ffff700 1 --2- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cc010 0x7f34800cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f347f7fe700 1 -- 172.21.15.119:0/1328555378 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424622759 0 0) 0x7f348006cb40 con 0x7f34800ce000 2024-09-13T16:33:40.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f347f7fe700 1 -- 172.21.15.119:0/1328555378 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34680035d0 con 0x7f34800ce000 2024-09-13T16:33:40.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f347f7fe700 1 -- 172.21.15.119:0/1328555378 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3632732490 0 0) 0x7f34680035d0 con 0x7f34800ce000 2024-09-13T16:33:40.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3484abd700 1 -- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cc010 msgr2=0x7f34800cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3484abd700 1 --2- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cc010 0x7f34800cdac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3484abd700 1 -- 172.21.15.119:0/1328555378 >> v1:172.21.15.119:6789/0 conn(0x7f34800ce000 legacy=0x7f348015e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3484abd700 1 -- 172.21.15.119:0/1328555378 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f348006d100 con 0x7f3480144180 2024-09-13T16:33:40.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3484abd700 1 --2- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3480144180 0x7f348006c480 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3474009850 tx=0x7f3474009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f9a9f0f42174b0d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:40.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f347f7fe700 1 -- 172.21.15.119:0/1328555378 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3478001670 con 0x7f3480144180 2024-09-13T16:33:40.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f347f7fe700 1 -- 172.21.15.119:0/1328555378 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3474004030 con 0x7f3480144180 2024-09-13T16:33:40.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f347f7fe700 1 -- 172.21.15.119:0/1328555378 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34740043a0 con 0x7f3480144180 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 -- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3480144180 msgr2=0x7f348006c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 --2- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3480144180 0x7f348006c480 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f3474009850 tx=0x7f3474009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 -- 172.21.15.119:0/1328555378 shutdown_connections 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 --2- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34800cc010 0x7f34800cdac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 --2- 172.21.15.119:0/1328555378 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3480144180 0x7f348006c480 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 -- 172.21.15.119:0/1328555378 >> 172.21.15.119:0/1328555378 conn(0x7f34800b0980 msgr2=0x7f34800b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 -- 172.21.15.119:0/1328555378 shutdown_connections 2024-09-13T16:33:40.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.898+0000 7f3485abf700 1 -- 172.21.15.119:0/1328555378 wait complete. 2024-09-13T16:33:40.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 Processor -- start 2024-09-13T16:33:40.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 -- start start 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34800cc010 0x7f34800acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34800ce000 0x7f348012d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3480144180 0x7f34801e36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34801e4390 con 0x7f34800ce000 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34801e4670 con 0x7f34800cc010 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34801e4950 con 0x7f3480144180 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34800ce000 0x7f348012d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34800ce000 0x7f348012d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48208/0 (socket says 172.21.15.119:48208) 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 -- 172.21.15.119:0/2010954194 learned_addr learned my addr 172.21.15.119:0/2010954194 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f34852be700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3480144180 0x7f34801e36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3484abd700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34800cc010 0x7f34800acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 -- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3480144180 msgr2=0x7f34801e36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3480144180 0x7f34801e36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 -- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34800cc010 msgr2=0x7f34800acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34800cc010 0x7f34800acec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 -- 172.21.15.119:0/2010954194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34740094b0 con 0x7f34800ce000 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347ffff700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34800ce000 0x7f348012d7a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f347000efc0 tx=0x7f347000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347d7fa700 1 -- 172.21.15.119:0/2010954194 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347000cd10 con 0x7f34800ce000 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347d7fa700 1 -- 172.21.15.119:0/2010954194 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347000f040 con 0x7f34800ce000 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34801e4c30 con 0x7f34800ce000 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f347d7fa700 1 -- 172.21.15.119:0/2010954194 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34700146a0 con 0x7f34800ce000 2024-09-13T16:33:40.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.902+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34800ad400 con 0x7f34800ce000 2024-09-13T16:33:40.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.906+0000 7f347d7fa700 1 -- 172.21.15.119:0/2010954194 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3470014800 con 0x7f34800ce000 2024-09-13T16:33:40.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.906+0000 7f347d7fa700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f346c066730 0x7f346c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.908 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.906+0000 7f347d7fa700 1 -- 172.21.15.119:0/2010954194 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f3470084ea0 con 0x7f34800ce000 2024-09-13T16:33:40.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.906+0000 7f3484abd700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f346c066730 0x7f346c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.906+0000 7f3484abd700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f346c066730 0x7f346c068bf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f34740090b0 tx=0x7f3474008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:40.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 Processor -- start 2024-09-13T16:33:40.912 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 -- start start 2024-09-13T16:33:40.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fcc810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff4520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x2016550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f5d010 con 0x2016170 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1eeaf40 con 0x2015d90 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1f3de30 con 0x1fcb310 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x2016550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x2016550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48220/0 (socket says 172.21.15.119:48220) 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 -- 172.21.15.119:0/3667356564 learned_addr learned my addr 172.21.15.119:0/3667356564 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3484abd700 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fcc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f347ffff700 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 -- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 msgr2=0x1ff4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 -- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 msgr2=0x1fcc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fcc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 -- 172.21.15.119:0/3667356564 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1da9a40 con 0x2016170 2024-09-13T16:33:40.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f3484abd700 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fcc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:40.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34852be700 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x2016550 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f3478004c60 tx=0x7f3478001dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5df9a5b9e9f97cc2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:40.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34667fc700 1 -- 172.21.15.119:0/3667356564 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347800f930 con 0x2016170 2024-09-13T16:33:40.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34667fc700 1 -- 172.21.15.119:0/3667356564 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347800fab0 con 0x2016170 2024-09-13T16:33:40.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.910+0000 7f34667fc700 1 -- 172.21.15.119:0/3667356564 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347800fe60 con 0x2016170 2024-09-13T16:33:40.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- 172.21.15.119:0/3667356564 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 msgr2=0x2016550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x2016550 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7f3478004c60 tx=0x7f3478001dc0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- 172.21.15.119:0/3667356564 shutdown_connections 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x2016550 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3667356564 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fcc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- 172.21.15.119:0/3667356564 >> 172.21.15.119:0/3667356564 conn(0x1d5e5b0 msgr2=0x1fcfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- 172.21.15.119:0/3667356564 shutdown_connections 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- 172.21.15.119:0/3667356564 wait complete. 2024-09-13T16:33:40.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- start start 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 Processor -- start 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fef830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x20140a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f39150 con 0x2016170 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f36eb0 con 0x2015d90 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ef48e0 con 0x1fcb310 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3489fd2740 10 client.0 _ll_register_callbacks cb 0x1ed3670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x20140a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f3484abd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x20140a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48236/0 (socket says 172.21.15.119:48236) 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 -- 172.21.15.119:0/3056016517 learned_addr learned my addr 172.21.15.119:0/3056016517 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f347ffff700 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fef830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 -- 172.21.15.119:0/3056016517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 msgr2=0x1ff2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 -- 172.21.15.119:0/3056016517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 msgr2=0x1fef830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fef830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 -- 172.21.15.119:0/3056016517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3478004890 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f34852be700 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x20140a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f347800f4b0 tx=0x7f3478000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9273774b6b6f037 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.914+0000 7f345f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 1 -- 172.21.15.119:0/3056016517 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347800f930 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 1 -- 172.21.15.119:0/3056016517 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f34780013f0 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1d6b2c0 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 1 -- 172.21.15.119:0/3056016517 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f347801f410 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 10 client.8589 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x1efd290 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 1 -- 172.21.15.119:0/3056016517 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1d6b2c0 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 1 -- 172.21.15.119:0/3056016517 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f3478012070 con 0x2016170 2024-09-13T16:33:40.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 10 client.8589 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:40.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 10 client.8589 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:33:40.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 20 client.8589 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:40.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1fefd70 0x1fede50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 4 client.8589 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:33:40.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1db5540 con 0x1fefd70 2024-09-13T16:33:40.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3484abd700 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1fefd70 0x1fede50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3484abd700 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1fefd70 0x1fede50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a537f435642b0ab server_cookie=a12ffeba273c88f in_seq=0 out_seq=0 2024-09-13T16:33:40.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 10 client.8589 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:40.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 1 -- 172.21.15.119:0/3056016517 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1db5540 con 0x1fefd70 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 10 client.8589 handle_command_reply: tid=0 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 client.8589 shutdown 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 msgr2=0x20140a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x20140a0 secure :-1 s=READY pgs=1459 cs=0 l=1 rev1=1 crypto rx=0x7f347800f4b0 tx=0x7f3478000f40 comp rx=0 tx=0).stop 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 shutdown_connections 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1fefd70 0x1fede50 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2016170 0x20140a0 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f345f7fe700 0 client.8589 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2015d90 0x1fef830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3056016517 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fcb310 0x1ff2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 >> 172.21.15.119:0/3056016517 conn(0x1d5e5b0 msgr2=0x1ff44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 shutdown_connections 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 1 -- 172.21.15.119:0/3056016517 wait complete. 2024-09-13T16:33:40.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.918+0000 7f3489fd2740 20 client.8589 trim_cache size 0 max 16384 2024-09-13T16:33:40.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 Processor -- start 2024-09-13T16:33:40.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 -- start start 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d5e5b0 0x2072d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1fcb310 0x1ff2420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x1ff2960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1de2b80 con 0x1fcb310 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ed5620 con 0x1d5e5b0 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ec0a50 con 0x1fefd70 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1fcb310 0x1ff2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1fcb310 0x1ff2420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48244/0 (socket says 172.21.15.119:48244) 2024-09-13T16:33:40.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 -- 172.21.15.119:0/869891973 learned_addr learned my addr 172.21.15.119:0/869891973 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f34852be700 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x1ff2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3484abd700 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d5e5b0 0x2072d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 -- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 msgr2=0x1ff2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x1ff2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 -- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d5e5b0 msgr2=0x2072d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d5e5b0 0x2072d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 -- 172.21.15.119:0/869891973 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1fd08d0 con 0x1fcb310 2024-09-13T16:33:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3484abd700 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d5e5b0 0x2072d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f347ffff700 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1fcb310 0x1ff2420 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f347000c6f0 tx=0x7f3470016040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3124a5e6757d9dd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:40.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3466ffd700 1 -- 172.21.15.119:0/869891973 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470059a00 con 0x1fcb310 2024-09-13T16:33:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.942+0000 7f3466ffd700 1 -- 172.21.15.119:0/869891973 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347005dfc0 con 0x1fcb310 2024-09-13T16:33:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3466ffd700 1 -- 172.21.15.119:0/869891973 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3470024300 con 0x1fcb310 2024-09-13T16:33:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- 172.21.15.119:0/869891973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1fcb310 msgr2=0x1ff2420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1fcb310 0x1ff2420 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 crypto rx=0x7f347000c6f0 tx=0x7f3470016040 comp rx=0 tx=0).stop 2024-09-13T16:33:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- 172.21.15.119:0/869891973 shutdown_connections 2024-09-13T16:33:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x1ff2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1fcb310 0x1ff2420 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 --2- 172.21.15.119:0/869891973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1d5e5b0 0x2072d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- 172.21.15.119:0/869891973 >> 172.21.15.119:0/869891973 conn(0x1fcff10 msgr2=0x202db80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- 172.21.15.119:0/869891973 shutdown_connections 2024-09-13T16:33:40.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- 172.21.15.119:0/869891973 wait complete. 2024-09-13T16:33:40.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- start start 2024-09-13T16:33:40.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 Processor -- start 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d5e5b0 0x202c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1fcb310 0x2073f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x202b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1edc2a0 con 0x1d5e5b0 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1f346c0 con 0x1fcb310 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x202ae50 con 0x1fefd70 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3489fd2740 10 client.0 _ll_register_callbacks cb 0x1f35610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d5e5b0 0x202c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d5e5b0 0x202c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48260/0 (socket says 172.21.15.119:48260) 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 -- 172.21.15.119:0/3818081674 learned_addr learned my addr 172.21.15.119:0/3818081674 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f347ffff700 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1fcb310 0x2073f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f34852be700 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x202b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 -- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 msgr2=0x202b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x202b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 -- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1fcb310 msgr2=0x2073f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1fcb310 0x2073f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 -- 172.21.15.119:0/3818081674 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34800ad400 con 0x1d5e5b0 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f3484abd700 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d5e5b0 0x202c370 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f34740041f0 tx=0x7f34740047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cf701df455c4878 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f345f7fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.946+0000 7f345f7fe700 1 -- 172.21.15.119:0/3818081674 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f347402e040 con 0x1d5e5b0 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 1 -- 172.21.15.119:0/3818081674 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f3474035420 con 0x1d5e5b0 2024-09-13T16:33:40.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f347800fab0 con 0x1d5e5b0 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 1 -- 172.21.15.119:0/3818081674 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3474020070 con 0x1d5e5b0 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 10 client.8595 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x1ee34c0 con 0x1d5e5b0 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 1 -- 172.21.15.119:0/3818081674 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f347800fab0 con 0x1d5e5b0 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 1 -- 172.21.15.119:0/3818081674 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f347403c060 con 0x1d5e5b0 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 10 client.8595 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 10 client.8595 resolve_mds: resolved 6:0 to role '6:0' aka daemon mds.c 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 20 client.8595 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2015d90 0x202b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 4 client.8595 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x1dda2c0 con 0x2015d90 2024-09-13T16:33:40.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f347ffff700 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2015d90 0x202b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 10 client.8595 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f347ffff700 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2015d90 0x202b730 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4cb5f01146716d32 server_cookie=92122b16854f5961 in_seq=0 out_seq=0 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 1 -- 172.21.15.119:0/3818081674 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x1dda2c0 con 0x2015d90 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 10 client.8595 handle_command_reply: tid=0 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 client.8595 shutdown 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d5e5b0 msgr2=0x202c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d5e5b0 0x202c370 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 crypto rx=0x7f34740041f0 tx=0x7f34740047e0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 shutdown_connections 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x2015d90 0x202b730 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1fefd70 0x202b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1fcb310 0x2073f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 --2- 172.21.15.119:0/3818081674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1d5e5b0 0x202c370 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f345f7fe700 0 client.8595 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:40.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 >> 172.21.15.119:0/3818081674 conn(0x1fcff10 msgr2=0x2072d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:40.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 shutdown_connections 2024-09-13T16:33:40.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 1 -- 172.21.15.119:0/3818081674 wait complete. 2024-09-13T16:33:40.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3489fd2740 20 client.8595 trim_cache size 0 max 16384 2024-09-13T16:33:40.957 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:33:40.957 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:33:40.957 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:33:40.957 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:33:40.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f346c066730 msgr2=0x7f346c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f346c066730 0x7f346c068bf0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f34740090b0 tx=0x7f3474008030 comp rx=0 tx=0).stop 2024-09-13T16:33:40.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34800ce000 msgr2=0x7f348012d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:40.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34800ce000 0x7f348012d7a0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7f347000efc0 tx=0x7f347000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 shutdown_connections 2024-09-13T16:33:40.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f346c066730 0x7f346c068bf0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3480144180 0x7f34801e36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34800ce000 0x7f348012d7a0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 --2- 172.21.15.119:0/2010954194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34800cc010 0x7f34800acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:40.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 >> 172.21.15.119:0/2010954194 conn(0x7f34800b0980 msgr2=0x7f348015b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:40.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 shutdown_connections 2024-09-13T16:33:40.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:40.950+0000 7f3485abf700 1 -- 172.21.15.119:0/2010954194 wait complete. 2024-09-13T16:33:40.968 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:33:40.969 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:41.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a355f700 1 Processor -- start 2024-09-13T16:33:41.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a355f700 1 -- start start 2024-09-13T16:33:41.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a355f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f649c1594f0 0x7f649c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a355f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c06da70 0x7f649c0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a355f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649c0b2280 con 0x7f649c0b3630 2024-09-13T16:33:41.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a355f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f649c070c10 con 0x7f649c1594f0 2024-09-13T16:33:41.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a355f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c06deb0 con 0x7f649c06da70 2024-09-13T16:33:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a1d5c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c06da70 0x7f649c0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a255d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f649c1594f0 0x7f649c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a1d5c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c06da70 0x7f649c0b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34126/0 (socket says 172.21.15.119:34126) 2024-09-13T16:33:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.078+0000 7f64a1d5c700 1 -- 172.21.15.119:0/2326872207 learned_addr learned my addr 172.21.15.119:0/2326872207 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a1d5c700 1 -- 172.21.15.119:0/2326872207 >> v2:172.21.15.145:3300/0 conn(0x7f649c1594f0 msgr2=0x7f649c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a155b700 1 -- 172.21.15.119:0/2326872207 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 638394742 0 0) 0x7f649c0b2280 con 0x7f649c0b3630 2024-09-13T16:33:41.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a1d5c700 1 --2- 172.21.15.119:0/2326872207 >> v2:172.21.15.145:3300/0 conn(0x7f649c1594f0 0x7f649c06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a1d5c700 1 -- 172.21.15.119:0/2326872207 >> v1:172.21.15.145:6789/0 conn(0x7f649c0b3630 legacy=0x7f649c15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a1d5c700 1 -- 172.21.15.119:0/2326872207 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649c0cc890 con 0x7f649c06da70 2024-09-13T16:33:41.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a1d5c700 1 --2- 172.21.15.119:0/2326872207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c06da70 0x7f649c0b30f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f649c070d50 tx=0x7f6490009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5872d6a7d50b65e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a155b700 1 -- 172.21.15.119:0/2326872207 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490004230 con 0x7f649c06da70 2024-09-13T16:33:41.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a155b700 1 -- 172.21.15.119:0/2326872207 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6490004390 con 0x7f649c06da70 2024-09-13T16:33:41.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a155b700 1 -- 172.21.15.119:0/2326872207 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64900036c0 con 0x7f649c06da70 2024-09-13T16:33:41.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 -- 172.21.15.119:0/2326872207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c06da70 msgr2=0x7f649c0b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 --2- 172.21.15.119:0/2326872207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c06da70 0x7f649c0b30f0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f649c070d50 tx=0x7f6490009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 -- 172.21.15.119:0/2326872207 shutdown_connections 2024-09-13T16:33:41.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 --2- 172.21.15.119:0/2326872207 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c06da70 0x7f649c0b30f0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 --2- 172.21.15.119:0/2326872207 >> v2:172.21.15.145:3300/0 conn(0x7f649c1594f0 0x7f649c06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 -- 172.21.15.119:0/2326872207 >> 172.21.15.119:0/2326872207 conn(0x7f649c06a350 msgr2=0x7f649c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 -- 172.21.15.119:0/2326872207 shutdown_connections 2024-09-13T16:33:41.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 -- 172.21.15.119:0/2326872207 wait complete. 2024-09-13T16:33:41.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 Processor -- start 2024-09-13T16:33:41.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.082+0000 7f64a355f700 1 -- start start 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f649c06da70 0x7f649c149630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f649c0b3630 0x7f649c0a9550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c1594f0 0x7f649c156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c158b60 con 0x7f649c06da70 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0a9a90 con 0x7f649c0b3630 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f649c0a84b0 con 0x7f649c1594f0 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a255d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f649c06da70 0x7f649c149630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f649c0b3630 0x7f649c0a9550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a2d5e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c1594f0 0x7f649c156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f649c0b3630 0x7f649c0a9550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40094/0 (socket says 172.21.15.119:40094) 2024-09-13T16:33:41.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a2d5e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c1594f0 0x7f649c156ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:34128/0 (socket says 172.21.15.119:34128) 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 -- 172.21.15.119:0/784356609 learned_addr learned my addr 172.21.15.119:0/784356609 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a255d700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f649c06da70 0x7f649c149630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48270/0 (socket says 172.21.15.119:48270) 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 -- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c1594f0 msgr2=0x7f649c156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c1594f0 0x7f649c156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 -- 172.21.15.119:0/784356609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f649c06da70 msgr2=0x7f649c149630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f649c06da70 0x7f649c149630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 -- 172.21.15.119:0/784356609 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490009570 con 0x7f649c0b3630 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a1d5c700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f649c0b3630 0x7f649c0a9550 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6490009c30 tx=0x7f6490004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f648effd700 1 -- 172.21.15.119:0/784356609 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649001f070 con 0x7f649c0b3630 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f648effd700 1 -- 172.21.15.119:0/784356609 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6490004500 con 0x7f649c0b3630 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f649c0a8790 con 0x7f649c0b3630 2024-09-13T16:33:41.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f648effd700 1 -- 172.21.15.119:0/784356609 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649000fb10 con 0x7f649c0b3630 2024-09-13T16:33:41.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f649c0a8d50 con 0x7f649c0b3630 2024-09-13T16:33:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.086+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f649c0b2710 con 0x7f649c0b3630 2024-09-13T16:33:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.090+0000 7f648effd700 1 -- 172.21.15.119:0/784356609 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f649000fc70 con 0x7f649c0b3630 2024-09-13T16:33:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.090+0000 7f648effd700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6480066730 0x7f6480068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.090+0000 7f648effd700 1 -- 172.21.15.119:0/784356609 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f6490087090 con 0x7f649c0b3630 2024-09-13T16:33:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.090+0000 7f64a255d700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6480066730 0x7f6480068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.090+0000 7f64a255d700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6480066730 0x7f6480068bf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6498009b50 tx=0x7f6498009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.098+0000 7f648effd700 1 -- 172.21.15.119:0/784356609 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6490054190 con 0x7f649c0b3630 2024-09-13T16:33:41.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.302+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f649c155170 con 0x7f649c0b3630 2024-09-13T16:33:41.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f648effd700 1 -- 172.21.15.119:0/784356609 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f6490058e60 con 0x7f649c0b3630 2024-09-13T16:33:41.308 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:41.308 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":69,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:06.506474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6480066730 msgr2=0x7f6480068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6480066730 0x7f6480068bf0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f6498009b50 tx=0x7f6498009240 comp rx=0 tx=0).stop 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f649c0b3630 msgr2=0x7f649c0a9550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f649c0b3630 0x7f649c0a9550 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6490009c30 tx=0x7f6490004770 comp rx=0 tx=0).stop 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 shutdown_connections 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6480066730 0x7f6480068bf0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f649c1594f0 0x7f649c156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f649c0b3630 0x7f649c0a9550 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 --2- 172.21.15.119:0/784356609 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f649c06da70 0x7f649c149630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 >> 172.21.15.119:0/784356609 conn(0x7f649c06a350 msgr2=0x7f649c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 shutdown_connections 2024-09-13T16:33:41.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.306+0000 7f64a355f700 1 -- 172.21.15.119:0/784356609 wait complete. 2024-09-13T16:33:41.312 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 69 2024-09-13T16:33:41.323 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c damage ls --format=json-pretty 2024-09-13T16:33:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efe04e0b700 1 Processor -- start 2024-09-13T16:33:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efe04e0b700 1 -- start start 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efe04e0b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efe000b1860 0x7efe00159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efe04e0b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe000b1dd0 con 0x7efe0006ca60 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efe04e0b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe000b20b0 con 0x7efe000ad9a0 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efe04e0b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe000b2fd0 con 0x7efe000b1860 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efdfffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efe000b1860 0x7efe00159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efdf7fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7efe0006ca60 0x7efe0015a970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54606/0 (socket says 172.21.15.119:54606) 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efdfffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efe000b1860 0x7efe00159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48278/0 (socket says 172.21.15.119:48278) 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efdf7fff700 1 -- 172.21.15.119:0/2441541322 learned_addr learned my addr 172.21.15.119:0/2441541322 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efdfffff700 1 -- 172.21.15.119:0/2441541322 >> v1:172.21.15.145:6789/0 conn(0x7efe000ad9a0 legacy=0x7efe0006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efdfffff700 1 -- 172.21.15.119:0/2441541322 >> v1:172.21.15.119:6789/0 conn(0x7efe0006ca60 legacy=0x7efe0015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.438+0000 7efdfffff700 1 -- 172.21.15.119:0/2441541322 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe000b2390 con 0x7efe000b1860 2024-09-13T16:33:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdfffff700 1 --2- 172.21.15.119:0/2441541322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe000b1860 0x7efe00159de0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009d90 tx=0x7efdf800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43f4dd877225abf9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdfeffd700 1 -- 172.21.15.119:0/2441541322 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf800e070 con 0x7efe000b1860 2024-09-13T16:33:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdfeffd700 1 -- 172.21.15.119:0/2441541322 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf800bbd0 con 0x7efe000b1860 2024-09-13T16:33:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdfeffd700 1 -- 172.21.15.119:0/2441541322 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8012430 con 0x7efe000b1860 2024-09-13T16:33:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- 172.21.15.119:0/2441541322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe000b1860 msgr2=0x7efe00159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 --2- 172.21.15.119:0/2441541322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe000b1860 0x7efe00159de0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7efdf8009d90 tx=0x7efdf800b330 comp rx=0 tx=0).stop 2024-09-13T16:33:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- 172.21.15.119:0/2441541322 shutdown_connections 2024-09-13T16:33:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 --2- 172.21.15.119:0/2441541322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe000b1860 0x7efe00159de0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- 172.21.15.119:0/2441541322 >> 172.21.15.119:0/2441541322 conn(0x7efe000b0980 msgr2=0x7efe000b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- 172.21.15.119:0/2441541322 shutdown_connections 2024-09-13T16:33:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- 172.21.15.119:0/2441541322 wait complete. 2024-09-13T16:33:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 Processor -- start 2024-09-13T16:33:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- start start 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe0006ca60 0x7efe0012cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe000ad9a0 0x7efe0012f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe000b1860 0x7efe00143a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0015c4f0 con 0x7efe0006ca60 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0012d0b0 con 0x7efe000b1860 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efe04e0b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe0012fb10 con 0x7efe000ad9a0 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe0006ca60 0x7efe0012cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe0006ca60 0x7efe0012cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48294/0 (socket says 172.21.15.119:48294) 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 -- 172.21.15.119:0/1183894395 learned_addr learned my addr 172.21.15.119:0/1183894395 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdf7fff700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe000ad9a0 0x7efe0012f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdfffff700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe000b1860 0x7efe00143a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 -- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe000ad9a0 msgr2=0x7efe0012f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe000ad9a0 0x7efe0012f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 -- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe000b1860 msgr2=0x7efe00143a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe000b1860 0x7efe00143a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdff7fe700 1 -- 172.21.15.119:0/1183894395 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf8009a70 con 0x7efe0006ca60 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.442+0000 7efdf7fff700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe000ad9a0 0x7efe0012f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdff7fe700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe0006ca60 0x7efe0012cb70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7efde800c500 tx=0x7efde800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdfcff9700 1 -- 172.21.15.119:0/1183894395 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde80052d0 con 0x7efe0006ca60 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdfcff9700 1 -- 172.21.15.119:0/1183894395 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde8005430 con 0x7efe0006ca60 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdfcff9700 1 -- 172.21.15.119:0/1183894395 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde80055a0 con 0x7efe0006ca60 2024-09-13T16:33:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe0015bc50 con 0x7efe0006ca60 2024-09-13T16:33:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0015c210 con 0x7efe0006ca60 2024-09-13T16:33:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdfcff9700 1 -- 172.21.15.119:0/1183894395 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efde80038e0 con 0x7efe0006ca60 2024-09-13T16:33:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdfcff9700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efde0066760 0x7efde0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdfcff9700 1 -- 172.21.15.119:0/1183894395 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7efde80838d0 con 0x7efe0006ca60 2024-09-13T16:33:41.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.446+0000 7efdf7fff700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efde0066760 0x7efde0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.450+0000 7efdf7fff700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efde0066760 0x7efde0068c20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efdf0007fd0 tx=0x7efdf0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.450+0000 7efe0931e740 1 Processor -- start 2024-09-13T16:33:41.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.450+0000 7efe0931e740 1 -- start start 2024-09-13T16:33:41.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c94410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1c90cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c91210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bbc810 con 0x1c94950 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bb9fa0 con 0x1bdd520 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1bb5890 con 0x1c94d30 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1c90cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1c90cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48296/0 (socket says 172.21.15.119:48296) 2024-09-13T16:33:41.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 -- 172.21.15.119:0/328525101 learned_addr learned my addr 172.21.15.119:0/328525101 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdff7fe700 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c94410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdfffff700 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c91210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 -- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 msgr2=0x1c91210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c91210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 -- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 msgr2=0x1c94410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c94410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 -- 172.21.15.119:0/328525101 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1a27ba0 con 0x1c94950 2024-09-13T16:33:41.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf7fff700 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1c90cd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000f60 tx=0x7efdf0011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315cbf29baa27613 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf57fa700 1 -- 172.21.15.119:0/328525101 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0007560 con 0x1c94950 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf57fa700 1 -- 172.21.15.119:0/328525101 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdf00076c0 con 0x1c94950 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efdf57fa700 1 -- 172.21.15.119:0/328525101 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf000e820 con 0x1c94950 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- 172.21.15.119:0/328525101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 msgr2=0x1c90cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1c90cd0 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 crypto rx=0x7efdf0000f60 tx=0x7efdf0011000 comp rx=0 tx=0).stop 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- 172.21.15.119:0/328525101 shutdown_connections 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c91210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1c90cd0 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 --2- 172.21.15.119:0/328525101 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c94410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- 172.21.15.119:0/328525101 >> 172.21.15.119:0/328525101 conn(0x19dc710 msgr2=0x1c766d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- 172.21.15.119:0/328525101 shutdown_connections 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- 172.21.15.119:0/328525101 wait complete. 2024-09-13T16:33:41.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 -- start start 2024-09-13T16:33:41.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.454+0000 7efe0931e740 1 Processor -- start 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bdd520 0x1c75c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94950 0x1c4ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94d30 0x1c77ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b78d80 con 0x1c94d30 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b78a20 con 0x1c94950 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1aa8770 con 0x1bdd520 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 10 client.0 _ll_register_callbacks cb 0x1baef00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94d30 0x1c77ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94d30 0x1c77ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48306/0 (socket says 172.21.15.119:48306) 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 -- 172.21.15.119:0/3127124891 learned_addr learned my addr 172.21.15.119:0/3127124891 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdf7fff700 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94950 0x1c4ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdff7fe700 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bdd520 0x1c75c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 -- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bdd520 msgr2=0x1c75c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bdd520 0x1c75c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 -- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94950 msgr2=0x1c4ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94950 0x1c4ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 -- 172.21.15.119:0/3127124891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf000f040 con 0x1c94d30 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdfffff700 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94d30 0x1c77ab0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7efdf8013000 tx=0x7efdf800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a60e5e7eda84c396 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdde7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdde7fc700 1 -- 172.21.15.119:0/3127124891 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf8020040 con 0x1c94d30 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdde7fc700 1 -- 172.21.15.119:0/3127124891 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7efdf8003e40 con 0x1c94d30 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x19e9420 con 0x1c94d30 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdde7fc700 1 -- 172.21.15.119:0/3127124891 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf8012d20 con 0x1c94d30 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 10 client.8613 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x1b7b910 con 0x1c94d30 2024-09-13T16:33:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdde7fc700 1 -- 172.21.15.119:0/3127124891 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x19e9420 con 0x1c94d30 2024-09-13T16:33:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdde7fc700 1 -- 172.21.15.119:0/3127124891 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efdf8004040 con 0x1c94d30 2024-09-13T16:33:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 10 client.8613 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 10 client.8613 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:33:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 20 client.8613 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c761c0 0x1c746c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 4 client.8613 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:33:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1a336a0 con 0x1c761c0 2024-09-13T16:33:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdff7fe700 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c761c0 0x1c746c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdff7fe700 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c761c0 0x1c746c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12eb676e6df9c680 server_cookie=84513c93d17d5067 in_seq=0 out_seq=0 2024-09-13T16:33:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.458+0000 7efdde7fc700 10 client.8613 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efdde7fc700 1 -- 172.21.15.119:0/3127124891 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1a336a0 con 0x1c761c0 2024-09-13T16:33:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efdde7fc700 10 client.8613 handle_command_reply: tid=0 2024-09-13T16:33:41.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 client.8613 shutdown 2024-09-13T16:33:41.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94d30 msgr2=0x1c77ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94d30 0x1c77ab0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7efdf8013000 tx=0x7efdf800e000 comp rx=0 tx=0).stop 2024-09-13T16:33:41.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 shutdown_connections 2024-09-13T16:33:41.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c761c0 0x1c746c0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94d30 0x1c77ab0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94950 0x1c4ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 --2- 172.21.15.119:0/3127124891 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bdd520 0x1c75c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 >> 172.21.15.119:0/3127124891 conn(0x19dc710 msgr2=0x1c93800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efdde7fc700 0 client.8613 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:41.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 shutdown_connections 2024-09-13T16:33:41.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 1 -- 172.21.15.119:0/3127124891 wait complete. 2024-09-13T16:33:41.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.462+0000 7efe0931e740 20 client.8613 trim_cache size 0 max 16384 2024-09-13T16:33:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 Processor -- start 2024-09-13T16:33:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 -- start start 2024-09-13T16:33:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bdd520 0x1c8e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94950 0x1c8ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94d30 0x1c72900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bb4720 con 0x1bdd520 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bb83f0 con 0x1c94d30 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1bb30a0 con 0x1c94950 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bdd520 0x1c8e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bdd520 0x1c8e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48322/0 (socket says 172.21.15.119:48322) 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efdff7fe700 1 -- 172.21.15.119:0/3261868558 learned_addr learned my addr 172.21.15.119:0/3261868558 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efdf7fff700 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94950 0x1c8ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.474+0000 7efdfffff700 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94d30 0x1c72900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 -- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94950 msgr2=0x1c8ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94950 0x1c8ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 -- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94d30 msgr2=0x1c72900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94d30 0x1c72900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 -- 172.21.15.119:0/3261868558 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1b61b40 con 0x1bdd520 2024-09-13T16:33:41.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bdd520 0x1c8e6b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7efde800ccc0 tx=0x7efde8050b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9568b3abc46efe4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf5ffb700 1 -- 172.21.15.119:0/3261868558 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde807f070 con 0x1bdd520 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf5ffb700 1 -- 172.21.15.119:0/3261868558 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde805ca80 con 0x1bdd520 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf5ffb700 1 -- 172.21.15.119:0/3261868558 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efde8081c10 con 0x1bdd520 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- 172.21.15.119:0/3261868558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bdd520 msgr2=0x1c8e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bdd520 0x1c8e6b0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7efde800ccc0 tx=0x7efde8050b00 comp rx=0 tx=0).stop 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- 172.21.15.119:0/3261868558 shutdown_connections 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1c94d30 0x1c72900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94950 0x1c8ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 --2- 172.21.15.119:0/3261868558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bdd520 0x1c8e6b0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- 172.21.15.119:0/3261868558 >> 172.21.15.119:0/3261868558 conn(0x19dc710 msgr2=0x1c76f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- 172.21.15.119:0/3261868558 shutdown_connections 2024-09-13T16:33:41.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- 172.21.15.119:0/3261868558 wait complete. 2024-09-13T16:33:41.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- start start 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 Processor -- start 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c4dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1cf4da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c4c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1bbdaa0 con 0x1c94950 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b76660 con 0x1bdd520 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1bdc9a0 con 0x1c94d30 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efe0931e740 10 client.0 _ll_register_callbacks cb 0x1bb4460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1cf4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c4dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1cf4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48336/0 (socket says 172.21.15.119:48336) 2024-09-13T16:33:41.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdfffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c4c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 -- 172.21.15.119:0/315602621 learned_addr learned my addr 172.21.15.119:0/315602621 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c4dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:40134/0 (socket says 172.21.15.119:40134) 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 -- 172.21.15.119:0/315602621 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 msgr2=0x1c4c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c4c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 -- 172.21.15.119:0/315602621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 msgr2=0x1c4dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c4dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 -- 172.21.15.119:0/315602621 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe0015c210 con 0x1c94950 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdf7fff700 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1cf4da0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7efdf000e850 tx=0x7efdf0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc33417961558bfb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.478+0000 7efdde7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 1 -- 172.21.15.119:0/315602621 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0028070 con 0x1c94950 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1c931b0 con 0x1c94950 2024-09-13T16:33:41.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 1 -- 172.21.15.119:0/315602621 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7efdf0001ce0 con 0x1c94950 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 1 -- 172.21.15.119:0/315602621 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=69) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf0007830 con 0x1c94950 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 10 client.8619 fetch_fsmap learned FSMap version 69 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=69,monmap=2+}) v3 -- 0x1b55c20 con 0x1c94950 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 1 -- 172.21.15.119:0/315602621 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1c931b0 con 0x1c94950 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 1 -- 172.21.15.119:0/315602621 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 69) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efdf000ec40 con 0x1c94950 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 10 client.8619 fetch_fsmap finished waiting for FSMap version 69 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 10 client.8619 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 20 client.8619 populate_metadata read hostname 'smithi119' 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c4a200 0x1c4a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 4 client.8619 mds_command: new command op to 8442 tid=0 multi_id=0 [{"prefix": "damage ls", "format": "json-pretty"}] 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 --> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] -- command(tid 0: {"prefix": "damage ls", "format": "json-pretty"}) v1 -- 0x7efdf00076c0 con 0x1c4a200 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdff7fe700 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c4a200 0x1c4a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 10 client.8619 ms_handle_connect on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdff7fe700 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c4a200 0x1c4a5e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd6b16168d44f723 server_cookie=917d9c3e55458659 in_seq=0 out_seq=0 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 1 -- 172.21.15.119:0/315602621 <== mds.0 v2:172.21.15.119:6834/1434266893 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7efdf00076c0 con 0x1c4a200 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 10 client.8619 handle_command_reply: tid=0 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 client.8619 shutdown 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 msgr2=0x1cf4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1cf4da0 secure :-1 s=READY pgs=1468 cs=0 l=1 rev1=1 crypto rx=0x7efdf000e850 tx=0x7efdf0000ed0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 shutdown_connections 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:6834/1434266893,v1:172.21.15.119:6835/1434266893] conn(0x1c4a200 0x1c4a5e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1c94d30 0x1c4c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efdde7fc700 0 client.8619 ms_handle_reset on v2:172.21.15.119:6834/1434266893 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1c94950 0x1cf4da0 unknown :-1 s=CLOSED pgs=1468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 --2- 172.21.15.119:0/315602621 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bdd520 0x1c4dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 >> 172.21.15.119:0/315602621 conn(0x19dc710 msgr2=0x1c74160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 shutdown_connections 2024-09-13T16:33:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 1 -- 172.21.15.119:0/315602621 wait complete. 2024-09-13T16:33:41.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.482+0000 7efe0931e740 20 client.8619 trim_cache size 0 max 16384 2024-09-13T16:33:41.491 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:41.491 INFO:teuthology.orchestra.run.smithi119.stdout:[] 2024-09-13T16:33:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efde0066760 msgr2=0x7efde0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efde0066760 0x7efde0068c20 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7efdf0007fd0 tx=0x7efdf0009040 comp rx=0 tx=0).stop 2024-09-13T16:33:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe0006ca60 msgr2=0x7efe0012cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe0006ca60 0x7efe0012cb70 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7efde800c500 tx=0x7efde800c8c0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 shutdown_connections 2024-09-13T16:33:41.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efde0066760 0x7efde0068c20 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe000b1860 0x7efe00143a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe000ad9a0 0x7efe0012f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 --2- 172.21.15.119:0/1183894395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe0006ca60 0x7efe0012cb70 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 >> 172.21.15.119:0/1183894395 conn(0x7efe000b0980 msgr2=0x7efe00066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 shutdown_connections 2024-09-13T16:33:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.486+0000 7efe04e0b700 1 -- 172.21.15.119:0/1183894395 wait complete. 2024-09-13T16:33:41.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-13T16:33:41.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e3fff700 1 Processor -- start 2024-09-13T16:33:41.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e3fff700 1 -- start start 2024-09-13T16:33:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e3fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff6e40d1060 0x7ff6e40d14c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff6e40b3070 0x7ff6e40b3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e3fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e40d1a00 con 0x7ff6e415ccb0 2024-09-13T16:33:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e3fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40b2be0 con 0x7ff6e40b3070 2024-09-13T16:33:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e3fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6e40ce150 con 0x7ff6e40d1060 2024-09-13T16:33:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff6e40b3070 0x7ff6e40b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e27fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff6e40d1060 0x7ff6e40d14c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff6e40b3070 0x7ff6e40b3500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48352/0 (socket says 172.21.15.119:48352) 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 -- 172.21.15.119:0/1717095404 learned_addr learned my addr 172.21.15.119:0/1717095404 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e2ffd700 1 --1- 172.21.15.119:0/1717095404 >> v1:172.21.15.145:6789/0 conn(0x7ff6e415ccb0 0x7ff6e40cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:54630/0 (socket says 172.21.15.119:54630) 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 -- 172.21.15.119:0/1717095404 >> v2:172.21.15.145:3300/0 conn(0x7ff6e40d1060 msgr2=0x7ff6e40d14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 --2- 172.21.15.119:0/1717095404 >> v2:172.21.15.145:3300/0 conn(0x7ff6e40d1060 0x7ff6e40d14c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 -- 172.21.15.119:0/1717095404 >> v1:172.21.15.145:6789/0 conn(0x7ff6e415ccb0 legacy=0x7ff6e40cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 -- 172.21.15.119:0/1717095404 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e40ce430 con 0x7ff6e40b3070 2024-09-13T16:33:41.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e37fe700 1 --2- 172.21.15.119:0/1717095404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e40b3070 0x7ff6e40b3500 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40b22b0 tx=0x7ff6dc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f761ffbca83f73d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e1ffb700 1 -- 172.21.15.119:0/1717095404 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00e070 con 0x7ff6e40b3070 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e1ffb700 1 -- 172.21.15.119:0/1717095404 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc00b910 con 0x7ff6e40b3070 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.614+0000 7ff6e1ffb700 1 -- 172.21.15.119:0/1717095404 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc00bcc0 con 0x7ff6e40b3070 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 -- 172.21.15.119:0/1717095404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e40b3070 msgr2=0x7ff6e40b3500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/1717095404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e40b3070 0x7ff6e40b3500 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40b22b0 tx=0x7ff6dc00b330 comp rx=0 tx=0).stop 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 -- 172.21.15.119:0/1717095404 shutdown_connections 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/1717095404 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e40b3070 0x7ff6e40b3500 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/1717095404 >> v2:172.21.15.145:3300/0 conn(0x7ff6e40d1060 0x7ff6e40d14c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 -- 172.21.15.119:0/1717095404 >> 172.21.15.119:0/1717095404 conn(0x7ff6e408c0f0 msgr2=0x7ff6e408c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 -- 172.21.15.119:0/1717095404 shutdown_connections 2024-09-13T16:33:41.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 -- 172.21.15.119:0/1717095404 wait complete. 2024-09-13T16:33:41.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 Processor -- start 2024-09-13T16:33:41.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.618+0000 7ff6e3fff700 1 -- start start 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6e40b3070 0x7ff6e415fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6e40d1060 0x7ff6e40c8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e415ccb0 0x7ff6e40c8550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40c65d0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40c67e0 con 0x7ff6e40d1060 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6e40c8bc0 con 0x7ff6e40b3070 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e415ccb0 0x7ff6e40c8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e415ccb0 0x7ff6e40c8550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48358/0 (socket says 172.21.15.119:48358) 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 -- 172.21.15.119:0/3532505194 learned_addr learned my addr 172.21.15.119:0/3532505194 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e27fc700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6e40d1060 0x7ff6e40c8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e2ffd700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6e40b3070 0x7ff6e415fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 -- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6e40b3070 msgr2=0x7ff6e415fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6e40b3070 0x7ff6e415fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 -- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6e40d1060 msgr2=0x7ff6e40c8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6e40d1060 0x7ff6e40c8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 -- 172.21.15.119:0/3532505194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc009b30 con 0x7ff6e415ccb0 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e27fc700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6e40d1060 0x7ff6e40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:41.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e37fe700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e415ccb0 0x7ff6e40c8550 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40b5660 tx=0x7ff6dc004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6db7fe700 1 -- 172.21.15.119:0/3532505194 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc004930 con 0x7ff6e415ccb0 2024-09-13T16:33:41.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6db7fe700 1 -- 172.21.15.119:0/3532505194 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc0099b0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e40c8ea0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6db7fe700 1 -- 172.21.15.119:0/3532505194 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc01b670 con 0x7ff6e415ccb0 2024-09-13T16:33:41.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.622+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e40c93c0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.626+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6e40c90e0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.626+0000 7ff6db7fe700 1 -- 172.21.15.119:0/3532505194 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff6dc01b7d0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.626+0000 7ff6db7fe700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6c4066730 0x7ff6c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:41.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.626+0000 7ff6db7fe700 1 -- 172.21.15.119:0/3532505194 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7ff6dc085c20 con 0x7ff6e415ccb0 2024-09-13T16:33:41.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.626+0000 7ff6e2ffd700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6c4066730 0x7ff6c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:41.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.626+0000 7ff6e2ffd700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6c4066730 0x7ff6c4068bf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000f60 tx=0x7ff6d4007ef0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:41.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.634+0000 7ff6db7fe700 1 -- 172.21.15.119:0/3532505194 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff6e40c90e0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.846+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff6e40d0c90 con 0x7ff6e415ccb0 2024-09-13T16:33:41.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.846+0000 7ff6db7fe700 1 -- 172.21.15.119:0/3532505194 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7ff6dc052dc0 con 0x7ff6e415ccb0 2024-09-13T16:33:41.849 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:41.849 INFO:teuthology.orchestra.run.smithi119.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-13T16:33:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6c4066730 msgr2=0x7ff6c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6c4066730 0x7ff6c4068bf0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4000f60 tx=0x7ff6d4007ef0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e415ccb0 msgr2=0x7ff6e40c8550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e415ccb0 0x7ff6e40c8550 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7ff6e40b5660 tx=0x7ff6dc004af0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 shutdown_connections 2024-09-13T16:33:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff6c4066730 0x7ff6c4068bf0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff6e415ccb0 0x7ff6e40c8550 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6e40d1060 0x7ff6e40c8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 --2- 172.21.15.119:0/3532505194 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6e40b3070 0x7ff6e415fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:41.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 >> 172.21.15.119:0/3532505194 conn(0x7ff6e408c0f0 msgr2=0x7ff6e40b1920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:41.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 shutdown_connections 2024-09-13T16:33:41.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:41.850+0000 7ff6e3fff700 1 -- 172.21.15.119:0/3532505194 wait complete. 2024-09-13T16:33:41.863 DEBUG:tasks.ceph_test_case:wait_until_true: success in 0s and 0 retries 2024-09-13T16:33:41.863 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:33:41.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:33:41.882 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:33:41.904 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:33:41.904 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:33:41.904 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:41.912 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:42.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.074+0000 7f8984fbf700 1 Processor -- start 2024-09-13T16:33:42.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- start start 2024-09-13T16:33:42.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f89800cc060 0x7f89800cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f898006cf60 con 0x7f89800cde70 2024-09-13T16:33:42.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89800cc4d0 con 0x7f89800adb30 2024-09-13T16:33:42.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f898006c350 con 0x7f89800cc060 2024-09-13T16:33:42.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f89800cc060 0x7f89800cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f89800cc060 0x7f89800cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48360/0 (socket says 172.21.15.119:48360) 2024-09-13T16:33:42.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897effd700 1 -- 172.21.15.119:0/1066762743 learned_addr learned my addr 172.21.15.119:0/1066762743 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:42.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897effd700 1 -- 172.21.15.119:0/1066762743 >> v1:172.21.15.145:6789/0 conn(0x7f89800adb30 legacy=0x7f898006bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:42.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897effd700 1 -- 172.21.15.119:0/1066762743 >> v1:172.21.15.119:6789/0 conn(0x7f89800cde70 legacy=0x7f89800b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897e7fc700 1 -- 172.21.15.119:0/1066762743 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796391928 0 0) 0x7f898006cf60 con 0x7f89800cde70 2024-09-13T16:33:42.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897effd700 1 -- 172.21.15.119:0/1066762743 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898006c630 con 0x7f89800cc060 2024-09-13T16:33:42.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897effd700 1 --2- 172.21.15.119:0/1066762743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cc060 0x7f89800cd930 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f8974009910 tx=0x7f8974009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a633b5a480e8252 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897e7fc700 1 -- 172.21.15.119:0/1066762743 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897400ba60 con 0x7f89800cc060 2024-09-13T16:33:42.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897e7fc700 1 -- 172.21.15.119:0/1066762743 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897400bbc0 con 0x7f89800cc060 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f897e7fc700 1 -- 172.21.15.119:0/1066762743 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8974009e60 con 0x7f89800cc060 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- 172.21.15.119:0/1066762743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cc060 msgr2=0x7f89800cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 --2- 172.21.15.119:0/1066762743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cc060 0x7f89800cd930 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7f8974009910 tx=0x7f8974009c20 comp rx=0 tx=0).stop 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- 172.21.15.119:0/1066762743 shutdown_connections 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 --2- 172.21.15.119:0/1066762743 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cc060 0x7f89800cd930 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- 172.21.15.119:0/1066762743 >> 172.21.15.119:0/1066762743 conn(0x7f898006a350 msgr2=0x7f898006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- 172.21.15.119:0/1066762743 shutdown_connections 2024-09-13T16:33:42.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.078+0000 7f8984fbf700 1 -- 172.21.15.119:0/1066762743 wait complete. 2024-09-13T16:33:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 Processor -- start 2024-09-13T16:33:42.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 -- start start 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89800adb30 0x7f89800a9f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89800cc060 0x7f89801413e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cde70 0x7f89800c4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89800ab5c0 con 0x7f89800cde70 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89800aa4f0 con 0x7f89800cc060 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89800aa7d0 con 0x7f89800adb30 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cde70 0x7f89800c4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cde70 0x7f89800c4320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:48372/0 (socket says 172.21.15.119:48372) 2024-09-13T16:33:42.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 -- 172.21.15.119:0/3428544616 learned_addr learned my addr 172.21.15.119:0/3428544616 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897effd700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89800cc060 0x7f89801413e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897f7fe700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89800adb30 0x7f89800a9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 -- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89800adb30 msgr2=0x7f89800a9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89800adb30 0x7f89800a9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 -- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89800cc060 msgr2=0x7f89801413e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89800cc060 0x7f89801413e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 -- 172.21.15.119:0/3428544616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8974009570 con 0x7f89800cde70 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f897ffff700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cde70 0x7f89800c4320 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f897800cc10 tx=0x7f897800cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8967fff700 1 -- 172.21.15.119:0/3428544616 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8978003e50 con 0x7f89800cde70 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8967fff700 1 -- 172.21.15.119:0/3428544616 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f897800a740 con 0x7f89800cde70 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89800aaab0 con 0x7f89800cde70 2024-09-13T16:33:42.088 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8967fff700 1 -- 172.21.15.119:0/3428544616 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f897800aad0 con 0x7f89800cde70 2024-09-13T16:33:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.082+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89800c4860 con 0x7f89800cde70 2024-09-13T16:33:42.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.086+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89800ab730 con 0x7f89800cde70 2024-09-13T16:33:42.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.086+0000 7f8967fff700 1 -- 172.21.15.119:0/3428544616 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f897800ac30 con 0x7f89800cde70 2024-09-13T16:33:42.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.086+0000 7f8967fff700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f896c066730 0x7f896c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.090+0000 7f8967fff700 1 -- 172.21.15.119:0/3428544616 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f8978083df0 con 0x7f89800cde70 2024-09-13T16:33:42.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.090+0000 7f897f7fe700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f896c066730 0x7f896c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.090+0000 7f897f7fe700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f896c066730 0x7f896c068bf0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8970000f60 tx=0x7f8970009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.098+0000 7f8967fff700 1 -- 172.21.15.119:0/3428544616 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f89800ab730 con 0x7f89800cde70 2024-09-13T16:33:42.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.274+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89800ab730 con 0x7f89800cde70 2024-09-13T16:33:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8967fff700 1 -- 172.21.15.119:0/3428544616 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17233 (secure 0 0 0) 0x7f89800ab730 con 0x7f89800cde70 2024-09-13T16:33:42.280 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:42.280 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":99,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:02.472922+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1e","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.119:6834/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.145:6835/3011171408":"2024-09-14T16:31:41.722614+0000","172.21.15.145:6834/3011171408":"2024-09-14T16:31:41.722614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f896c066730 msgr2=0x7f896c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f896c066730 0x7f896c068bf0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8970000f60 tx=0x7f8970009040 comp rx=0 tx=0).stop 2024-09-13T16:33:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cde70 msgr2=0x7f89800c4320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cde70 0x7f89800c4320 secure :-1 s=READY pgs=1473 cs=0 l=1 rev1=1 crypto rx=0x7f897800cc10 tx=0x7f897800cf20 comp rx=0 tx=0).stop 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 shutdown_connections 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f896c066730 0x7f896c068bf0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f89800cde70 0x7f89800c4320 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f89800cc060 0x7f89801413e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 --2- 172.21.15.119:0/3428544616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f89800adb30 0x7f89800a9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 >> 172.21.15.119:0/3428544616 conn(0x7f898006a350 msgr2=0x7f89800b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 shutdown_connections 2024-09-13T16:33:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.278+0000 7f8984fbf700 1 -- 172.21.15.119:0/3428544616 wait complete. 2024-09-13T16:33:42.295 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:31:27.299109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:31:28.044326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:33:42.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe23fff700 1 Processor -- start 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe23fff700 1 -- start start 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe23fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbe2415abb0 0x7fbe240cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe23fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbe240cd920 0x7fbe240cbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe23fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2415af90 con 0x7fbe240b31e0 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe23fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe240cb6d0 con 0x7fbe2415abb0 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe23fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe240d0060 con 0x7fbe240cd920 2024-09-13T16:33:42.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe22ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbe2415abb0 0x7fbe240cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe227fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbe240b31e0 0x7fbe240cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56612/0 (socket says 172.21.15.119:56612) 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe227fc700 1 -- 172.21.15.119:0/4048551123 learned_addr learned my addr 172.21.15.119:0/4048551123 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.414+0000 7fbe237fe700 1 --2- 172.21.15.119:0/4048551123 >> v2:172.21.15.145:3300/0 conn(0x7fbe240cd920 0x7fbe240cbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 -- 172.21.15.119:0/4048551123 >> v2:172.21.15.145:3300/0 conn(0x7fbe240cd920 msgr2=0x7fbe240cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 --2- 172.21.15.119:0/4048551123 >> v2:172.21.15.145:3300/0 conn(0x7fbe240cd920 0x7fbe240cbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 -- 172.21.15.119:0/4048551123 >> v1:172.21.15.119:6789/0 conn(0x7fbe240b31e0 legacy=0x7fbe240cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe21ffb700 1 -- 172.21.15.119:0/4048551123 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222137857 0 0) 0x7fbe1c001490 con 0x7fbe240b31e0 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 -- 172.21.15.119:0/4048551123 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe240cfd80 con 0x7fbe2415abb0 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 --2- 172.21.15.119:0/4048551123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe2415abb0 0x7fbe240cf530 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fbe240714e0 tx=0x7fbe14009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84c309f22e7a09ad server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe21ffb700 1 -- 172.21.15.119:0/4048551123 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14011830 con 0x7fbe2415abb0 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe21ffb700 1 -- 172.21.15.119:0/4048551123 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe14011990 con 0x7fbe2415abb0 2024-09-13T16:33:42.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe21ffb700 1 -- 172.21.15.119:0/4048551123 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14011d20 con 0x7fbe2415abb0 2024-09-13T16:33:42.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- 172.21.15.119:0/4048551123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe2415abb0 msgr2=0x7fbe240cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 --2- 172.21.15.119:0/4048551123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe2415abb0 0x7fbe240cf530 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7fbe240714e0 tx=0x7fbe14009b30 comp rx=0 tx=0).stop 2024-09-13T16:33:42.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- 172.21.15.119:0/4048551123 shutdown_connections 2024-09-13T16:33:42.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 --2- 172.21.15.119:0/4048551123 >> v2:172.21.15.145:3300/0 conn(0x7fbe240cd920 0x7fbe240cbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 --2- 172.21.15.119:0/4048551123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe2415abb0 0x7fbe240cf530 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- 172.21.15.119:0/4048551123 >> 172.21.15.119:0/4048551123 conn(0x7fbe240b25e0 msgr2=0x7fbe240b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:42.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- 172.21.15.119:0/4048551123 shutdown_connections 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- 172.21.15.119:0/4048551123 wait complete. 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 Processor -- start 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- start start 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe240b31e0 0x7fbe24149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe240cd920 0x7fbe240b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe2415abb0 0x7fbe2414bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406ced0 con 0x7fbe240b31e0 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406c2c0 con 0x7fbe2415abb0 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe23fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe2406c5a0 con 0x7fbe240cd920 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe240b31e0 0x7fbe24149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe240b31e0 0x7fbe24149190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35192/0 (socket says 172.21.15.119:35192) 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.418+0000 7fbe22ffd700 1 -- 172.21.15.119:0/1444474865 learned_addr learned my addr 172.21.15.119:0/1444474865 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:42.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe237fe700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe2415abb0 0x7fbe2414bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe227fc700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe240cd920 0x7fbe240b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe22ffd700 1 -- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe240cd920 msgr2=0x7fbe240b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe22ffd700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe240cd920 0x7fbe240b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe22ffd700 1 -- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe2415abb0 msgr2=0x7fbe2414bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe22ffd700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe2415abb0 0x7fbe2414bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe22ffd700 1 -- 172.21.15.119:0/1444474865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe140094b0 con 0x7fbe240b31e0 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe227fc700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe240cd920 0x7fbe240b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe22ffd700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe240b31e0 0x7fbe24149190 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fbe14014040 tx=0x7fbe14004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe1b7fe700 1 -- 172.21.15.119:0/1444474865 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe140036c0 con 0x7fbe240b31e0 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe1b7fe700 1 -- 172.21.15.119:0/1444474865 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe1401e070 con 0x7fbe240b31e0 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2406c880 con 0x7fbe240b31e0 2024-09-13T16:33:42.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe1b7fe700 1 -- 172.21.15.119:0/1444474865 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe14003a80 con 0x7fbe240b31e0 2024-09-13T16:33:42.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe241496d0 con 0x7fbe240b31e0 2024-09-13T16:33:42.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.422+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe240cbb60 con 0x7fbe240b31e0 2024-09-13T16:33:42.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.426+0000 7fbe1b7fe700 1 -- 172.21.15.119:0/1444474865 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbe14003be0 con 0x7fbe240b31e0 2024-09-13T16:33:42.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.426+0000 7fbe1b7fe700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe08066760 0x7fbe08068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.426+0000 7fbe1b7fe700 1 -- 172.21.15.119:0/1444474865 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fbe140853a0 con 0x7fbe240b31e0 2024-09-13T16:33:42.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.426+0000 7fbe227fc700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe08066760 0x7fbe08068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.426+0000 7fbe227fc700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe08066760 0x7fbe08068c20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbe2415e450 tx=0x7fbe1c001a00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.430+0000 7fbe1b7fe700 1 -- 172.21.15.119:0/1444474865 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbe14052390 con 0x7fbe240b31e0 2024-09-13T16:33:42.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.606+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe240c7760 con 0x7fbe240b31e0 2024-09-13T16:33:42.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.606+0000 7fbe1b7fe700 1 -- 172.21.15.119:0/1444474865 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe140559b0 con 0x7fbe240b31e0 2024-09-13T16:33:42.609 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:33:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.606+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe08066760 msgr2=0x7fbe08068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.606+0000 7fbe23fff700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe08066760 0x7fbe08068c20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fbe2415e450 tx=0x7fbe1c001a00 comp rx=0 tx=0).stop 2024-09-13T16:33:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.606+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe240b31e0 msgr2=0x7fbe24149190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.606+0000 7fbe23fff700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe240b31e0 0x7fbe24149190 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 crypto rx=0x7fbe14014040 tx=0x7fbe14004770 comp rx=0 tx=0).stop 2024-09-13T16:33:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 shutdown_connections 2024-09-13T16:33:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe08066760 0x7fbe08068c20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe2415abb0 0x7fbe2414bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe240cd920 0x7fbe240b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 --2- 172.21.15.119:0/1444474865 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe240b31e0 0x7fbe24149190 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 >> 172.21.15.119:0/1444474865 conn(0x7fbe240b25e0 msgr2=0x7fbe24157b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 shutdown_connections 2024-09-13T16:33:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.610+0000 7fbe23fff700 1 -- 172.21.15.119:0/1444474865 wait complete. 2024-09-13T16:33:42.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:33:42.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.746+0000 7f8b1ae9a700 1 Processor -- start 2024-09-13T16:33:42.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.746+0000 7f8b1ae9a700 1 -- start start 2024-09-13T16:33:42.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.746+0000 7f8b1ae9a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140ce8c0 0x7f8b140d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.746+0000 7f8b1ae9a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8b140d1610 0x7f8b140b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.746+0000 7f8b1ae9a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b140d0df0 con 0x7f8b1415cc00 2024-09-13T16:33:42.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b140d1a80 con 0x7f8b140d1610 2024-09-13T16:33:42.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b140cdb00 con 0x7f8b140ce8c0 2024-09-13T16:33:42.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8b140d1610 0x7f8b140b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8b140d1610 0x7f8b140b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35200/0 (socket says 172.21.15.119:35200) 2024-09-13T16:33:42.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 -- 172.21.15.119:0/1578605949 learned_addr learned my addr 172.21.15.119:0/1578605949 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:42.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b19697700 1 --2- 172.21.15.119:0/1578605949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140ce8c0 0x7f8b140d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 -- 172.21.15.119:0/1578605949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140ce8c0 msgr2=0x7f8b140d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:42.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 --2- 172.21.15.119:0/1578605949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140ce8c0 0x7f8b140d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 -- 172.21.15.119:0/1578605949 >> v1:172.21.15.145:6789/0 conn(0x7f8b1415cc00 legacy=0x7f8b140d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 -- 172.21.15.119:0/1578605949 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b140cdde0 con 0x7f8b140d1610 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1a699700 1 --2- 172.21.15.119:0/1578605949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b140d1610 0x7f8b140b5960 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009e50 tx=0x7f8b0c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7991833e1a3ab083 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b18e96700 1 -- 172.21.15.119:0/1578605949 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c00e070 con 0x7f8b140d1610 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b18e96700 1 -- 172.21.15.119:0/1578605949 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c00b910 con 0x7f8b140d1610 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b18e96700 1 -- 172.21.15.119:0/1578605949 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c00bcc0 con 0x7f8b140d1610 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/1578605949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b140d1610 msgr2=0x7f8b140b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/1578605949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b140d1610 0x7f8b140b5960 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f8b0c009e50 tx=0x7f8b0c00b330 comp rx=0 tx=0).stop 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/1578605949 shutdown_connections 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/1578605949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b140d1610 0x7f8b140b5960 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/1578605949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140ce8c0 0x7f8b140d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/1578605949 >> 172.21.15.119:0/1578605949 conn(0x7f8b1408c080 msgr2=0x7f8b140d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/1578605949 shutdown_connections 2024-09-13T16:33:42.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/1578605949 wait complete. 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 Processor -- start 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.750+0000 7f8b1ae9a700 1 -- start start 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b140ce8c0 0x7f8b1415b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140d1610 0x7f8b140c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b1415cc00 0x7f8b1415bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1415f020 con 0x7f8b1415cc00 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1415f290 con 0x7f8b140ce8c0 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b1415f570 con 0x7f8b140d1610 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b1415cc00 0x7f8b1415bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b1415cc00 0x7f8b1415bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35212/0 (socket says 172.21.15.119:35212) 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 -- 172.21.15.119:0/2532976852 learned_addr learned my addr 172.21.15.119:0/2532976852 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b19697700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140d1610 0x7f8b140c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b19e98700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b140ce8c0 0x7f8b1415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 -- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140d1610 msgr2=0x7f8b140c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140d1610 0x7f8b140c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 -- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b140ce8c0 msgr2=0x7f8b1415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b140ce8c0 0x7f8b1415b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 -- 172.21.15.119:0/2532976852 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b0c009b30 con 0x7f8b1415cc00 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1a699700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b1415cc00 0x7f8b1415bc80 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f8b1408bdf0 tx=0x7f8b0c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b067fc700 1 -- 172.21.15.119:0/2532976852 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c004800 con 0x7f8b1415cc00 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b067fc700 1 -- 172.21.15.119:0/2532976852 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b0c0099b0 con 0x7f8b1415cc00 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b1415f850 con 0x7f8b1415cc00 2024-09-13T16:33:42.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b067fc700 1 -- 172.21.15.119:0/2532976852 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b0c01c680 con 0x7f8b1415cc00 2024-09-13T16:33:42.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b141f6990 con 0x7f8b1415cc00 2024-09-13T16:33:42.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.754+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b140d3670 con 0x7f8b1415cc00 2024-09-13T16:33:42.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.758+0000 7f8b067fc700 1 -- 172.21.15.119:0/2532976852 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8b0c01c7e0 con 0x7f8b1415cc00 2024-09-13T16:33:42.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.758+0000 7f8b067fc700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af8066730 0x7f8af8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:42.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.758+0000 7f8b067fc700 1 -- 172.21.15.119:0/2532976852 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f8b0c085f80 con 0x7f8b1415cc00 2024-09-13T16:33:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.758+0000 7f8b19e98700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af8066730 0x7f8af8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:42.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.758+0000 7f8b19e98700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af8066730 0x7f8af8068bf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8b08000f60 tx=0x7f8b08009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:42.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.762+0000 7f8b067fc700 1 -- 172.21.15.119:0/2532976852 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8b0c053080 con 0x7f8b1415cc00 2024-09-13T16:33:42.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8b140b6ac0 con 0x7f8b1415cc00 2024-09-13T16:33:42.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b067fc700 1 -- 172.21.15.119:0/2532976852 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8b0c0566a0 con 0x7f8b1415cc00 2024-09-13T16:33:42.937 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af8066730 msgr2=0x7f8af8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af8066730 0x7f8af8068bf0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f8b08000f60 tx=0x7f8b08009040 comp rx=0 tx=0).stop 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b1415cc00 msgr2=0x7f8b1415bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b1415cc00 0x7f8b1415bc80 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7f8b1408bdf0 tx=0x7f8b0c004af0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 shutdown_connections 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8af8066730 0x7f8af8068bf0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8b1415cc00 0x7f8b1415bc80 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8b140d1610 0x7f8b140c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.934+0000 7f8b1ae9a700 1 --2- 172.21.15.119:0/2532976852 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8b140ce8c0 0x7f8b1415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:42.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.938+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 >> 172.21.15.119:0/2532976852 conn(0x7f8b1408c080 msgr2=0x7f8b1406b730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:42.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.938+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 shutdown_connections 2024-09-13T16:33:42.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:42.938+0000 7f8b1ae9a700 1 -- 172.21.15.119:0/2532976852 wait complete. 2024-09-13T16:33:42.951 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:33:43.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f206a793700 1 Processor -- start 2024-09-13T16:33:43.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f206a793700 1 -- start start 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f206a793700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f206415cad0 0x7f20640d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f206a793700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f20640cf930 0x7f20640ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f206a793700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20640d05b0 con 0x7f20640ce0d0 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f206a793700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20640d0870 con 0x7f206415cad0 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f206a793700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20640d0b50 con 0x7f20640cf930 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f2069791700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f206415cad0 0x7f20640d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f2069791700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f206415cad0 0x7f20640d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35214/0 (socket says 172.21.15.119:35214) 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f2069791700 1 -- 172.21.15.119:0/1192436137 learned_addr learned my addr 172.21.15.119:0/1192436137 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.066+0000 7f2069f92700 1 --2- 172.21.15.119:0/1192436137 >> v2:172.21.15.145:3300/0 conn(0x7f20640cf930 0x7f20640ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2069791700 1 -- 172.21.15.119:0/1192436137 >> v2:172.21.15.145:3300/0 conn(0x7f20640cf930 msgr2=0x7f20640ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:43.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2069791700 1 --2- 172.21.15.119:0/1192436137 >> v2:172.21.15.145:3300/0 conn(0x7f20640cf930 0x7f20640ce510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2069791700 1 -- 172.21.15.119:0/1192436137 >> v1:172.21.15.119:6789/0 conn(0x7f20640ce0d0 legacy=0x7f20640cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2069791700 1 -- 172.21.15.119:0/1192436137 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20640d0e30 con 0x7f206415cad0 2024-09-13T16:33:43.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2069791700 1 --2- 172.21.15.119:0/1192436137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f206415cad0 0x7f20640d1630 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f20640d2760 tx=0x7f205c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b461d33bb3618990 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2063fff700 1 -- 172.21.15.119:0/1192436137 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00b8a0 con 0x7f206415cad0 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2063fff700 1 -- 172.21.15.119:0/1192436137 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f205c00ba00 con 0x7f206415cad0 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f2063fff700 1 -- 172.21.15.119:0/1192436137 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f205c00bd90 con 0x7f206415cad0 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 -- 172.21.15.119:0/1192436137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f206415cad0 msgr2=0x7f20640d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 --2- 172.21.15.119:0/1192436137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f206415cad0 0x7f20640d1630 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 crypto rx=0x7f20640d2760 tx=0x7f205c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 -- 172.21.15.119:0/1192436137 shutdown_connections 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 --2- 172.21.15.119:0/1192436137 >> v2:172.21.15.145:3300/0 conn(0x7f20640cf930 0x7f20640ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 --2- 172.21.15.119:0/1192436137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f206415cad0 0x7f20640d1630 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 -- 172.21.15.119:0/1192436137 >> 172.21.15.119:0/1192436137 conn(0x7f206408bf70 msgr2=0x7f20640d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:43.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 -- 172.21.15.119:0/1192436137 shutdown_connections 2024-09-13T16:33:43.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 -- 172.21.15.119:0/1192436137 wait complete. 2024-09-13T16:33:43.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 Processor -- start 2024-09-13T16:33:43.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.070+0000 7f206a793700 1 -- start start 2024-09-13T16:33:43.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20640ce0d0 0x7f20640cc970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20640cf930 0x7f20640ca770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f206415cad0 0x7f206415dfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f206415d030 con 0x7f20640cf930 2024-09-13T16:33:43.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20640c8960 con 0x7f20640ce0d0 2024-09-13T16:33:43.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f206415d210 con 0x7f206415cad0 2024-09-13T16:33:43.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20640cf930 0x7f20640ca770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2069791700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20640ce0d0 0x7f20640cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20640cf930 0x7f20640ca770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35230/0 (socket says 172.21.15.119:35230) 2024-09-13T16:33:43.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 -- 172.21.15.119:0/1587592704 learned_addr learned my addr 172.21.15.119:0/1587592704 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:43.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2069f92700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f206415cad0 0x7f206415dfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 -- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f206415cad0 msgr2=0x7f206415dfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f206415cad0 0x7f206415dfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 -- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20640ce0d0 msgr2=0x7f20640cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20640ce0d0 0x7f20640cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 -- 172.21.15.119:0/1587592704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205c009570 con 0x7f20640cf930 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2068f90700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20640cf930 0x7f20640ca770 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f205800c500 tx=0x7f205800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2061ffb700 1 -- 172.21.15.119:0/1587592704 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20580052d0 con 0x7f20640cf930 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2061ffb700 1 -- 172.21.15.119:0/1587592704 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2058005430 con 0x7f20640cf930 2024-09-13T16:33:43.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f2061ffb700 1 -- 172.21.15.119:0/1587592704 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20580055a0 con 0x7f20640cf930 2024-09-13T16:33:43.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206415d4f0 con 0x7f20640cf930 2024-09-13T16:33:43.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.074+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206415dab0 con 0x7f20640cf930 2024-09-13T16:33:43.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.078+0000 7f2061ffb700 1 -- 172.21.15.119:0/1587592704 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f20580036a0 con 0x7f20640cf930 2024-09-13T16:33:43.080 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.078+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206415dcd0 con 0x7f20640cf930 2024-09-13T16:33:43.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.078+0000 7f2061ffb700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2050066730 0x7f2050068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.078+0000 7f2061ffb700 1 -- 172.21.15.119:0/1587592704 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f2058083590 con 0x7f20640cf930 2024-09-13T16:33:43.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.078+0000 7f2069791700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2050066730 0x7f2050068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.078+0000 7f2069791700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2050066730 0x7f2050068bf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f205c000c00 tx=0x7f205c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.086+0000 7f2061ffb700 1 -- 172.21.15.119:0/1587592704 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2058050690 con 0x7f20640cf930 2024-09-13T16:33:43.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f20640ce510 con 0x7f20640cf930 2024-09-13T16:33:43.256 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f2061ffb700 1 -- 172.21.15.119:0/1587592704 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2058053cb0 con 0x7f20640cf930 2024-09-13T16:33:43.256 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:33:43.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2050066730 msgr2=0x7f2050068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2050066730 0x7f2050068bf0 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7f205c000c00 tx=0x7f205c008030 comp rx=0 tx=0).stop 2024-09-13T16:33:43.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20640cf930 msgr2=0x7f20640ca770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20640cf930 0x7f20640ca770 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 crypto rx=0x7f205800c500 tx=0x7f205800c810 comp rx=0 tx=0).stop 2024-09-13T16:33:43.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 shutdown_connections 2024-09-13T16:33:43.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2050066730 0x7f2050068bf0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f206415cad0 0x7f206415dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f20640cf930 0x7f20640ca770 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 --2- 172.21.15.119:0/1587592704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f20640ce0d0 0x7f20640cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.254+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 >> 172.21.15.119:0/1587592704 conn(0x7f206408bf70 msgr2=0x7f206406b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:43.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.258+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 shutdown_connections 2024-09-13T16:33:43.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.258+0000 7f206a793700 1 -- 172.21.15.119:0/1587592704 wait complete. 2024-09-13T16:33:43.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:33:43.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.382+0000 7f64f8985700 1 Processor -- start 2024-09-13T16:33:43.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.382+0000 7f64f8985700 1 -- start start 2024-09-13T16:33:43.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.382+0000 7f64f8985700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64f40cd860 0x7f64f40ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.382+0000 7f64f8985700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f415d020 con 0x7f64f415cc40 2024-09-13T16:33:43.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.382+0000 7f64f8985700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f40d0720 con 0x7f64f408c6b0 2024-09-13T16:33:43.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.382+0000 7f64f8985700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f40b44c0 con 0x7f64f40cd860 2024-09-13T16:33:43.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.382+0000 7f64f359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f64f40cd860 0x7f64f40ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f2d9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f64f415cc40 0x7f64f408bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56642/0 (socket says 172.21.15.119:56642) 2024-09-13T16:33:43.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f2d9d700 1 -- 172.21.15.119:0/1637291761 learned_addr learned my addr 172.21.15.119:0/1637291761 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f359e700 1 -- 172.21.15.119:0/1637291761 >> v1:172.21.15.145:6789/0 conn(0x7f64f408c6b0 legacy=0x7f64f4164a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f359e700 1 -- 172.21.15.119:0/1637291761 >> v1:172.21.15.119:6789/0 conn(0x7f64f415cc40 legacy=0x7f64f408bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f359e700 1 -- 172.21.15.119:0/1637291761 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f40d09c0 con 0x7f64f40cd860 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f259c700 1 -- 172.21.15.119:0/1637291761 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2813491214 0 0) 0x7f64f415d020 con 0x7f64f415cc40 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f359e700 1 --2- 172.21.15.119:0/1637291761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f40cd860 0x7f64f40ce710 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009dc0 tx=0x7f64ec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a05837c97e0eacc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f259c700 1 -- 172.21.15.119:0/1637291761 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec00e070 con 0x7f64f40cd860 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f259c700 1 -- 172.21.15.119:0/1637291761 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64ec00bb90 con 0x7f64f40cd860 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f259c700 1 -- 172.21.15.119:0/1637291761 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64ec012430 con 0x7f64f40cd860 2024-09-13T16:33:43.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 -- 172.21.15.119:0/1637291761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f40cd860 msgr2=0x7f64f40ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 --2- 172.21.15.119:0/1637291761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f40cd860 0x7f64f40ce710 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 crypto rx=0x7f64ec009dc0 tx=0x7f64ec00b330 comp rx=0 tx=0).stop 2024-09-13T16:33:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 -- 172.21.15.119:0/1637291761 shutdown_connections 2024-09-13T16:33:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 --2- 172.21.15.119:0/1637291761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f40cd860 0x7f64f40ce710 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 -- 172.21.15.119:0/1637291761 >> 172.21.15.119:0/1637291761 conn(0x7f64f40b4640 msgr2=0x7f64f40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 -- 172.21.15.119:0/1637291761 shutdown_connections 2024-09-13T16:33:43.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 -- 172.21.15.119:0/1637291761 wait complete. 2024-09-13T16:33:43.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.386+0000 7f64f8985700 1 Processor -- start 2024-09-13T16:33:43.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 -- start start 2024-09-13T16:33:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f408c6b0 0x7f64f415b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f40cd860 0x7f64f40c8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f415cc40 0x7f64f41f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40b3fe0 con 0x7f64f408c6b0 2024-09-13T16:33:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40b3a60 con 0x7f64f415cc40 2024-09-13T16:33:43.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f40b3d00 con 0x7f64f40cd860 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f408c6b0 0x7f64f415b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f408c6b0 0x7f64f415b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35244/0 (socket says 172.21.15.119:35244) 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 -- 172.21.15.119:0/3077979545 learned_addr learned my addr 172.21.15.119:0/3077979545 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64ea59c700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f40cd860 0x7f64f40c8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f359e700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f415cc40 0x7f64f41f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 -- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f40cd860 msgr2=0x7f64f40c8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f40cd860 0x7f64f40c8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 -- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f415cc40 msgr2=0x7f64f41f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f415cc40 0x7f64f41f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 -- 172.21.15.119:0/3077979545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64ec009a70 con 0x7f64f408c6b0 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f2d9d700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f408c6b0 0x7f64f415b970 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c550 tx=0x7f64e400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64ebfff700 1 -- 172.21.15.119:0/3077979545 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4004ca0 con 0x7f64f408c6b0 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64ebfff700 1 -- 172.21.15.119:0/3077979545 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64e4004e00 con 0x7f64f408c6b0 2024-09-13T16:33:43.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f40cc890 con 0x7f64f408c6b0 2024-09-13T16:33:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64ebfff700 1 -- 172.21.15.119:0/3077979545 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64e4007310 con 0x7f64f408c6b0 2024-09-13T16:33:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f415d570 con 0x7f64f408c6b0 2024-09-13T16:33:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.390+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f40d1a60 con 0x7f64f408c6b0 2024-09-13T16:33:43.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.394+0000 7f64ebfff700 1 -- 172.21.15.119:0/3077979545 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f64e4007470 con 0x7f64f408c6b0 2024-09-13T16:33:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.394+0000 7f64ebfff700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d4066730 0x7f64d4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.394+0000 7f64ebfff700 1 -- 172.21.15.119:0/3077979545 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f64e4084160 con 0x7f64f408c6b0 2024-09-13T16:33:43.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.394+0000 7f64ea59c700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d4066730 0x7f64d4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.394+0000 7f64ea59c700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d4066730 0x7f64d4068bf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f64f40b30f0 tx=0x7f64dc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.398+0000 7f64ebfff700 1 -- 172.21.15.119:0/3077979545 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f64e4051260 con 0x7f64f408c6b0 2024-09-13T16:33:43.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.578+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f64f40ceff0 con 0x7f64f408c6b0 2024-09-13T16:33:43.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.578+0000 7f64ebfff700 1 -- 172.21.15.119:0/3077979545 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v99) v1 ==== 80+0+248 (secure 0 0 0) 0x7f64e4007720 con 0x7f64f408c6b0 2024-09-13T16:33:43.581 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6835/3937841911 2024-09-14T16:33:02.472861+0000 2024-09-13T16:33:43.581 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.119:6834/3937841911 2024-09-14T16:33:02.472861+0000 2024-09-13T16:33:43.581 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6835/3011171408 2024-09-14T16:31:41.722614+0000 2024-09-13T16:33:43.581 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6834/3011171408 2024-09-14T16:31:41.722614+0000 2024-09-13T16:33:43.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d4066730 msgr2=0x7f64d4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d4066730 0x7f64d4068bf0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7f64f40b30f0 tx=0x7f64dc009040 comp rx=0 tx=0).stop 2024-09-13T16:33:43.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f408c6b0 msgr2=0x7f64f415b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f408c6b0 0x7f64f415b970 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f64e400c550 tx=0x7f64e400c910 comp rx=0 tx=0).stop 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 shutdown_connections 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f64d4066730 0x7f64d4068bf0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f64f415cc40 0x7f64f41f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f64f40cd860 0x7f64f40c8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 --2- 172.21.15.119:0/3077979545 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f64f408c6b0 0x7f64f415b970 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 >> 172.21.15.119:0/3077979545 conn(0x7f64f40b4640 msgr2=0x7f64f40b7990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 shutdown_connections 2024-09-13T16:33:43.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.582+0000 7f64f8985700 1 -- 172.21.15.119:0/3077979545 wait complete. 2024-09-13T16:33:43.585 INFO:teuthology.orchestra.run.smithi119.stderr:listed 4 entries 2024-09-13T16:33:43.595 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:33:43.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:43.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:33:43.610 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[49131]: fuse finished with error 0 and tester_r 0 2024-09-13T16:33:43.612 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:33:43.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:43.612 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:33:43.737 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:33:43.737 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:43.744 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:33:43.745 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:33:43.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:43.745 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:33:43.751 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-13T16:33:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:33:43.752 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:33:43.752 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:43.760 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:33:43.760 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:33:43.760 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:43.760 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:33:43.808 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-09-13T16:33:43.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:33:43.810 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:33:43.810 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:43.817 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:33:43.817 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:33:43.817 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:33:43.817 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:33:43.824 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-13T16:33:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:33:43.825 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:43.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.942+0000 7f5f1df5f700 1 Processor -- start 2024-09-13T16:33:43.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.942+0000 7f5f1df5f700 1 -- start start 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1df5f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f18156e00 0x7f5f1815a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1df5f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5f180cc8d0 0x7f5f180ae490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1df5f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1815a610 con 0x7f5f1806b950 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1df5f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f180cbb10 con 0x7f5f180cc8d0 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1df5f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180cbdf0 con 0x7f5f18156e00 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1d75e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5f1806b950 0x7f5f1806a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56648/0 (socket says 172.21.15.119:56648) 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1d75e700 1 -- 172.21.15.119:0/502298844 learned_addr learned my addr 172.21.15.119:0/502298844 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f17fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5f180cc8d0 0x7f5f180ae490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1cf5d700 1 --2- 172.21.15.119:0/502298844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f18156e00 0x7f5f1815a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f177fe700 1 -- 172.21.15.119:0/502298844 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092639930 0 0) 0x7f5f1815a610 con 0x7f5f1806b950 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f177fe700 1 -- 172.21.15.119:0/502298844 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f080035d0 con 0x7f5f1806b950 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f17fff700 1 -- 172.21.15.119:0/502298844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f18156e00 msgr2=0x7f5f1815a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f17fff700 1 --2- 172.21.15.119:0/502298844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f18156e00 0x7f5f1815a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f17fff700 1 -- 172.21.15.119:0/502298844 >> v1:172.21.15.119:6789/0 conn(0x7f5f1806b950 legacy=0x7f5f1806a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f17fff700 1 -- 172.21.15.119:0/502298844 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f180cc0d0 con 0x7f5f180cc8d0 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1cf5d700 1 --2- 172.21.15.119:0/502298844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f18156e00 0x7f5f1815a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f17fff700 1 --2- 172.21.15.119:0/502298844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f180cc8d0 0x7f5f180ae490 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400a9f0 tx=0x7f5f0400ad00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cee024627f71b35 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f177fe700 1 -- 172.21.15.119:0/502298844 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0400cb70 con 0x7f5f180cc8d0 2024-09-13T16:33:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f177fe700 1 -- 172.21.15.119:0/502298844 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0400ae90 con 0x7f5f180cc8d0 2024-09-13T16:33:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f177fe700 1 -- 172.21.15.119:0/502298844 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f04018400 con 0x7f5f180cc8d0 2024-09-13T16:33:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1df5f700 1 -- 172.21.15.119:0/502298844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f180cc8d0 msgr2=0x7f5f180ae490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.946+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/502298844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f180cc8d0 0x7f5f180ae490 secure :-1 s=READY pgs=1486 cs=0 l=1 rev1=1 crypto rx=0x7f5f0400a9f0 tx=0x7f5f0400ad00 comp rx=0 tx=0).stop 2024-09-13T16:33:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- 172.21.15.119:0/502298844 shutdown_connections 2024-09-13T16:33:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/502298844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f180cc8d0 0x7f5f180ae490 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/502298844 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f18156e00 0x7f5f1815a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- 172.21.15.119:0/502298844 >> 172.21.15.119:0/502298844 conn(0x7f5f180b3ee0 msgr2=0x7f5f180b45f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- 172.21.15.119:0/502298844 shutdown_connections 2024-09-13T16:33:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- 172.21.15.119:0/502298844 wait complete. 2024-09-13T16:33:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 Processor -- start 2024-09-13T16:33:43.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- start start 2024-09-13T16:33:43.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f1806b950 0x7f5f18152cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f180cc8d0 0x7f5f180ce640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f18156e00 0x7f5f1814fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f18147940 con 0x7f5f18156e00 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180acd90 con 0x7f5f1806b950 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1df5f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180ad070 con 0x7f5f180cc8d0 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1d75e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f18156e00 0x7f5f1814fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1d75e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f18156e00 0x7f5f1814fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35262/0 (socket says 172.21.15.119:35262) 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1d75e700 1 -- 172.21.15.119:0/650171111 learned_addr learned my addr 172.21.15.119:0/650171111 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f1cf5d700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f1806b950 0x7f5f18152cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.950+0000 7f5f17fff700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f180cc8d0 0x7f5f180ce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1d75e700 1 -- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f180cc8d0 msgr2=0x7f5f180ce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1d75e700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f180cc8d0 0x7f5f180ce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1d75e700 1 -- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f1806b950 msgr2=0x7f5f18152cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1d75e700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f1806b950 0x7f5f18152cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1d75e700 1 -- 172.21.15.119:0/650171111 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f0c0094b0 con 0x7f5f18156e00 2024-09-13T16:33:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1d75e700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f18156e00 0x7f5f1814fb00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000cb40 tx=0x7f5f1000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f157fa700 1 -- 172.21.15.119:0/650171111 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f10007500 con 0x7f5f18156e00 2024-09-13T16:33:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f157fa700 1 -- 172.21.15.119:0/650171111 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f10007660 con 0x7f5f18156e00 2024-09-13T16:33:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f0400a650 con 0x7f5f18156e00 2024-09-13T16:33:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f157fa700 1 -- 172.21.15.119:0/650171111 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f100079f0 con 0x7f5f18156e00 2024-09-13T16:33:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.954+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f180ad910 con 0x7f5f18156e00 2024-09-13T16:33:43.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.958+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f180ae490 con 0x7f5f18156e00 2024-09-13T16:33:43.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.958+0000 7f5f157fa700 1 -- 172.21.15.119:0/650171111 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5f1001b3b0 con 0x7f5f18156e00 2024-09-13T16:33:43.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.958+0000 7f5f157fa700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5efc066730 0x7f5efc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:43.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.958+0000 7f5f157fa700 1 -- 172.21.15.119:0/650171111 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f5f10083d50 con 0x7f5f18156e00 2024-09-13T16:33:43.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.958+0000 7f5f1cf5d700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5efc066730 0x7f5efc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:43.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.958+0000 7f5f1cf5d700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5efc066730 0x7f5efc068bf0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009ce0 tx=0x7f5f0c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:43.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:43.966+0000 7f5f157fa700 1 -- 172.21.15.119:0/650171111 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5f10087030 con 0x7f5f18156e00 2024-09-13T16:33:44.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.178+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5f181513a0 con 0x7f5f18156e00 2024-09-13T16:33:44.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.178+0000 7f5f157fa700 1 -- 172.21.15.119:0/650171111 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f5f10017090 con 0x7f5f18156e00 2024-09-13T16:33:44.182 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:44.182 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":69,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:06.506474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5efc066730 msgr2=0x7f5efc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5efc066730 0x7f5efc068bf0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c009ce0 tx=0x7f5f0c009240 comp rx=0 tx=0).stop 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f18156e00 msgr2=0x7f5f1814fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f18156e00 0x7f5f1814fb00 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 crypto rx=0x7f5f1000cb40 tx=0x7f5f1000cb70 comp rx=0 tx=0).stop 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 shutdown_connections 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5efc066730 0x7f5efc068bf0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f18156e00 0x7f5f1814fb00 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f180cc8d0 0x7f5f180ce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 --2- 172.21.15.119:0/650171111 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f1806b950 0x7f5f18152cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 >> 172.21.15.119:0/650171111 conn(0x7f5f180b3ee0 msgr2=0x7f5f180b45d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:44.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 shutdown_connections 2024-09-13T16:33:44.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.182+0000 7f5f1df5f700 1 -- 172.21.15.119:0/650171111 wait complete. 2024-09-13T16:33:44.186 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 69 2024-09-13T16:33:44.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:44.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 Processor -- start 2024-09-13T16:33:44.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 -- start start 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6e200adb30 0x7f6e200cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e2006bb90 0x7f6e200cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6e200cde70 0x7f6e2015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2006cf60 con 0x7f6e200cde70 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e2006c350 con 0x7f6e200adb30 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e2578a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2006c630 con 0x7f6e2006bb90 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6e200cde70 0x7f6e2015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6e200cde70 0x7f6e2015e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35270/0 (socket says 172.21.15.119:35270) 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 -- 172.21.15.119:0/1995078348 learned_addr learned my addr 172.21.15.119:0/1995078348 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e1f7fe700 1 --2- 172.21.15.119:0/1995078348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e2006bb90 0x7f6e200cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e1ffff700 1 --2- 172.21.15.119:0/1995078348 >> v2:172.21.15.145:3300/0 conn(0x7f6e200adb30 0x7f6e200cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 -- 172.21.15.119:0/1995078348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e2006bb90 msgr2=0x7f6e200cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 --2- 172.21.15.119:0/1995078348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e2006bb90 0x7f6e200cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 -- 172.21.15.119:0/1995078348 >> v2:172.21.15.145:3300/0 conn(0x7f6e200adb30 msgr2=0x7f6e200cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 --2- 172.21.15.119:0/1995078348 >> v2:172.21.15.145:3300/0 conn(0x7f6e200adb30 0x7f6e200cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e24f89700 1 -- 172.21.15.119:0/1995078348 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e2006c910 con 0x7f6e200cde70 2024-09-13T16:33:44.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.302+0000 7f6e1ffff700 1 --2- 172.21.15.119:0/1995078348 >> v2:172.21.15.145:3300/0 conn(0x7f6e200adb30 0x7f6e200cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:44.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e24f89700 1 --2- 172.21.15.119:0/1995078348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e2015e2f0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6e20071fa0 tx=0x7f6e18009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72f8f32ea068d51e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e1effd700 1 -- 172.21.15.119:0/1995078348 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18011830 con 0x7f6e200cde70 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e1effd700 1 -- 172.21.15.119:0/1995078348 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e18011990 con 0x7f6e200cde70 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e1effd700 1 -- 172.21.15.119:0/1995078348 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18011d20 con 0x7f6e200cde70 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 -- 172.21.15.119:0/1995078348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 msgr2=0x7f6e2015e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 --2- 172.21.15.119:0/1995078348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e2015e2f0 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 crypto rx=0x7f6e20071fa0 tx=0x7f6e18009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 -- 172.21.15.119:0/1995078348 shutdown_connections 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 --2- 172.21.15.119:0/1995078348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e2015e2f0 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 --2- 172.21.15.119:0/1995078348 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e2006bb90 0x7f6e200cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 --2- 172.21.15.119:0/1995078348 >> v2:172.21.15.145:3300/0 conn(0x7f6e200adb30 0x7f6e200cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 -- 172.21.15.119:0/1995078348 >> 172.21.15.119:0/1995078348 conn(0x7f6e2006a350 msgr2=0x7f6e2006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:44.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 -- 172.21.15.119:0/1995078348 shutdown_connections 2024-09-13T16:33:44.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.306+0000 7f6e2578a700 1 -- 172.21.15.119:0/1995078348 wait complete. 2024-09-13T16:33:44.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 Processor -- start 2024-09-13T16:33:44.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 -- start start 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e2006bb90 0x7f6e2015a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e200adb30 0x7f6e200af3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e200af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2015a5d0 con 0x7f6e200cde70 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e20130790 con 0x7f6e2006bb90 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e2012f6f0 con 0x7f6e200adb30 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e200af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e200af8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35278/0 (socket says 172.21.15.119:35278) 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 -- 172.21.15.119:0/3459155485 learned_addr learned my addr 172.21.15.119:0/3459155485 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e1ffff700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e2006bb90 0x7f6e2015a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e1f7fe700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e200adb30 0x7f6e200af3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 -- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e200adb30 msgr2=0x7f6e200af3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e200adb30 0x7f6e200af3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 -- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e2006bb90 msgr2=0x7f6e2015a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e2006bb90 0x7f6e2015a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 -- 172.21.15.119:0/3459155485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18009570 con 0x7f6e200cde70 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e1ffff700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e2006bb90 0x7f6e2015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e24f89700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e200af8f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f6e18014040 tx=0x7f6e18004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e1cff9700 1 -- 172.21.15.119:0/3459155485 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18004510 con 0x7f6e200cde70 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e1cff9700 1 -- 172.21.15.119:0/3459155485 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1801f070 con 0x7f6e200cde70 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e1cff9700 1 -- 172.21.15.119:0/3459155485 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18003900 con 0x7f6e200cde70 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e2012f9d0 con 0x7f6e200cde70 2024-09-13T16:33:44.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.310+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e2012ff90 con 0x7f6e200cde70 2024-09-13T16:33:44.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.314+0000 7f6e1cff9700 1 -- 172.21.15.119:0/3459155485 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6e18003a60 con 0x7f6e200cde70 2024-09-13T16:33:44.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.314+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e200cc990 con 0x7f6e200cde70 2024-09-13T16:33:44.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.314+0000 7f6e1cff9700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e00066730 0x7f6e00068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.314+0000 7f6e1cff9700 1 -- 172.21.15.119:0/3459155485 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f6e18021070 con 0x7f6e200cde70 2024-09-13T16:33:44.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.314+0000 7f6e1ffff700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e00066730 0x7f6e00068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.314+0000 7f6e1ffff700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e00066730 0x7f6e00068bf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6e100037d0 tx=0x7f6e10008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.322+0000 7f6e1cff9700 1 -- 172.21.15.119:0/3459155485 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6e18052bd0 con 0x7f6e200cde70 2024-09-13T16:33:44.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.494+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e200adf10 con 0x7f6e200cde70 2024-09-13T16:33:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e1cff9700 1 -- 172.21.15.119:0/3459155485 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17233 (secure 0 0 0) 0x7f6e18015090 con 0x7f6e200cde70 2024-09-13T16:33:44.500 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:44.501 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":99,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:02.472922+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1e","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.119:6834/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.145:6835/3011171408":"2024-09-14T16:31:41.722614+0000","172.21.15.145:6834/3011171408":"2024-09-14T16:31:41.722614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:44.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e00066730 msgr2=0x7f6e00068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e00066730 0x7f6e00068bf0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6e100037d0 tx=0x7f6e10008040 comp rx=0 tx=0).stop 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 msgr2=0x7f6e200af8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e200af8f0 secure :-1 s=READY pgs=1489 cs=0 l=1 rev1=1 crypto rx=0x7f6e18014040 tx=0x7f6e18004770 comp rx=0 tx=0).stop 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 shutdown_connections 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e00066730 0x7f6e00068bf0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e200cde70 0x7f6e200af8f0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e200adb30 0x7f6e200af3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 --2- 172.21.15.119:0/3459155485 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e2006bb90 0x7f6e2015a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 >> 172.21.15.119:0/3459155485 conn(0x7f6e2006a350 msgr2=0x7f6e2015ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 shutdown_connections 2024-09-13T16:33:44.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.498+0000 7f6e2578a700 1 -- 172.21.15.119:0/3459155485 wait complete. 2024-09-13T16:33:44.516 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:31:27.299109+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '94', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:31:28.044326+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:33:44.517 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:33:44.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.634+0000 7f9ead480700 1 Processor -- start 2024-09-13T16:33:44.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.634+0000 7f9ead480700 1 -- start start 2024-09-13T16:33:44.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ead480700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 0x7f9ea8071c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ead480700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea815d300 con 0x7f9ea806bb70 2024-09-13T16:33:44.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ead480700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ea815d5e0 con 0x7f9ea8072170 2024-09-13T16:33:44.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ead480700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea815df80 con 0x7f9ea813fa40 2024-09-13T16:33:44.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9eacc7f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9ea806bb70 0x7f9ea80b3100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56662/0 (socket says 172.21.15.119:56662) 2024-09-13T16:33:44.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9eacc7f700 1 -- 172.21.15.119:0/1807262537 learned_addr learned my addr 172.21.15.119:0/1807262537 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 0x7f9ea8071c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45251125 0 0) 0x7f9ea815d300 con 0x7f9ea806bb70 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e900035d0 con 0x7f9ea806bb70 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 811206208 0 0) 0x7f9e900035d0 con 0x7f9ea806bb70 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 msgr2=0x7f9ea8071c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 0x7f9ea8071c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 >> v1:172.21.15.145:6789/0 conn(0x7f9ea8072170 legacy=0x7f9ea806b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea815d8c0 con 0x7f9ea806bb70 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f9ea00019f0 con 0x7f9ea806bb70 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 >> v1:172.21.15.119:6789/0 conn(0x7f9ea806bb70 legacy=0x7f9ea80b3100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e90003a50 0x7f9e90003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 0x7f9e90004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e90004db0 0x7f9e90005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea815df80 con 0x7f9e90003a50 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9000bbc0 con 0x7f9e90004db0 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e9000bd50 con 0x7f9ea813fa40 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e90003a50 0x7f9e90003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea77fe700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 0x7f9e90004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9eacc7f700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e90004db0 0x7f9e90005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 -- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 msgr2=0x7f9e90004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea813fa40 0x7f9e90004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 -- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e90004db0 msgr2=0x7f9e90005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e90004db0 0x7f9e90005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 -- 172.21.15.119:0/1807262537 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e900035d0 con 0x7f9e90003a50 2024-09-13T16:33:44.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.638+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e90003a50 0x7f9e90003e30 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009ba0 tx=0x7f9e900058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17bb33c0a79912c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800e6c0 con 0x7f9e90003a50 2024-09-13T16:33:44.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e9800e820 con 0x7f9e90003a50 2024-09-13T16:33:44.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ea6ffd700 1 -- 172.21.15.119:0/1807262537 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e9800eb90 con 0x7f9e90003a50 2024-09-13T16:33:44.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 -- 172.21.15.119:0/1807262537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e90003a50 msgr2=0x7f9e90003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 --2- 172.21.15.119:0/1807262537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e90003a50 0x7f9e90003e30 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 crypto rx=0x7f9e98009ba0 tx=0x7f9e900058a0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9eacc7f700 1 -- 172.21.15.119:0/1807262537 reap_dead start 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 -- 172.21.15.119:0/1807262537 shutdown_connections 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 -- 172.21.15.119:0/1807262537 >> 172.21.15.119:0/1807262537 conn(0x7f9ea80ae6e0 msgr2=0x7f9ea80b0c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 -- 172.21.15.119:0/1807262537 shutdown_connections 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 -- 172.21.15.119:0/1807262537 wait complete. 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 Processor -- start 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.642+0000 7f9ead480700 1 -- start start 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ea813c320 0x7f9ea8156d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea8153630 0x7f9ea8154c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea81551a0 0x7f9ea813cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea815cb50 con 0x7f9ea81551a0 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8155650 con 0x7f9ea813c320 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea815bc80 con 0x7f9ea8153630 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea81551a0 0x7f9ea813cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea81551a0 0x7f9ea813cd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35290/0 (socket says 172.21.15.119:35290) 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 -- 172.21.15.119:0/830276455 learned_addr learned my addr 172.21.15.119:0/830276455 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ea813c320 0x7f9ea8156d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ea77fe700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea8153630 0x7f9ea8154c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 -- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea8153630 msgr2=0x7f9ea8154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea8153630 0x7f9ea8154c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 -- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ea813c320 msgr2=0x7f9ea8156d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ea813c320 0x7f9ea8156d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 -- 172.21.15.119:0/830276455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea815bed0 con 0x7f9ea81551a0 2024-09-13T16:33:44.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ea813c320 0x7f9ea8156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:44.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9eacc7f700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea81551a0 0x7f9ea813cd60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8154240 tx=0x7f9ea00019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ea4ff9700 1 -- 172.21.15.119:0/830276455 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea000a8f0 con 0x7f9ea81551a0 2024-09-13T16:33:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ea4ff9700 1 -- 172.21.15.119:0/830276455 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ea000aa70 con 0x7f9ea81551a0 2024-09-13T16:33:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea815c1b0 con 0x7f9ea81551a0 2024-09-13T16:33:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ea4ff9700 1 -- 172.21.15.119:0/830276455 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ea000ae20 con 0x7f9ea81551a0 2024-09-13T16:33:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea815c770 con 0x7f9ea81551a0 2024-09-13T16:33:44.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.646+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea815cc90 con 0x7f9ea81551a0 2024-09-13T16:33:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.650+0000 7f9ea4ff9700 1 -- 172.21.15.119:0/830276455 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9ea001c400 con 0x7f9ea81551a0 2024-09-13T16:33:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.650+0000 7f9ea4ff9700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e94066730 0x7f9e94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.650+0000 7f9ea4ff9700 1 -- 172.21.15.119:0/830276455 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f9ea0084a30 con 0x7f9ea81551a0 2024-09-13T16:33:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.650+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e94066730 0x7f9e94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.650+0000 7f9ea7fff700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e94066730 0x7f9e94068bf0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9e980090b0 tx=0x7f9e9801e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.654+0000 7f9ea4ff9700 1 -- 172.21.15.119:0/830276455 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9ea815cc90 con 0x7f9ea81551a0 2024-09-13T16:33:44.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9ea815cc90 con 0x7f9ea81551a0 2024-09-13T16:33:44.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ea4ff9700 1 -- 172.21.15.119:0/830276455 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9ea815cc90 con 0x7f9ea81551a0 2024-09-13T16:33:44.828 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:33:44.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e94066730 msgr2=0x7f9e94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e94066730 0x7f9e94068bf0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f9e980090b0 tx=0x7f9e9801e040 comp rx=0 tx=0).stop 2024-09-13T16:33:44.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea81551a0 msgr2=0x7f9ea813cd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea81551a0 0x7f9ea813cd60 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f9ea8154240 tx=0x7f9ea00019e0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 shutdown_connections 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9e94066730 0x7f9e94068bf0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ea81551a0 0x7f9ea813cd60 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ea8153630 0x7f9ea8154c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 --2- 172.21.15.119:0/830276455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ea813c320 0x7f9ea8156d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 >> 172.21.15.119:0/830276455 conn(0x7f9e90003a50 msgr2=0x7f9ea8070c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 shutdown_connections 2024-09-13T16:33:44.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.826+0000 7f9ead480700 1 -- 172.21.15.119:0/830276455 wait complete. 2024-09-13T16:33:44.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:33:44.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb041295700 1 Processor -- start 2024-09-13T16:33:44.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb041295700 1 -- start start 2024-09-13T16:33:44.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb041295700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c0ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb041295700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb03c0b3260 0x7fb03c0b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb041295700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb03c0cd4d0 con 0x7fb03c15dbe0 2024-09-13T16:33:44.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb041295700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03c071be0 con 0x7fb03c0b3260 2024-09-13T16:33:44.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb041295700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c159af0 con 0x7fb03c0add60 2024-09-13T16:33:44.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb03b7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb03c15dbe0 0x7fb03c0cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56674/0 (socket says 172.21.15.119:56674) 2024-09-13T16:33:44.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb03b7fe700 1 -- 172.21.15.119:0/1538114917 learned_addr learned my addr 172.21.15.119:0/1538114917 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:44.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb040a94700 1 --2- 172.21.15.119:0/1538114917 >> v2:172.21.15.145:3300/0 conn(0x7fb03c0b3260 0x7fb03c0b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb03bfff700 1 --2- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c0ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb03affd700 1 -- 172.21.15.119:0/1538114917 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 464339401 0 0) 0x7fb03c0cd4d0 con 0x7fb03c15dbe0 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb03affd700 1 -- 172.21.15.119:0/1538114917 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0240035d0 con 0x7fb03c15dbe0 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb040a94700 1 -- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 msgr2=0x7fb03c0ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb040a94700 1 --2- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c0ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb040a94700 1 -- 172.21.15.119:0/1538114917 >> v1:172.21.15.119:6789/0 conn(0x7fb03c15dbe0 legacy=0x7fb03c0cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb040a94700 1 -- 172.21.15.119:0/1538114917 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03c15e130 con 0x7fb03c0b3260 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.966+0000 7fb040a94700 1 --2- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c0b36f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb034009d90 tx=0x7fb03400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=628c07482435a024 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb03affd700 1 -- 172.21.15.119:0/1538114917 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03400e070 con 0x7fb03c0b3260 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb03affd700 1 -- 172.21.15.119:0/1538114917 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb03400bb20 con 0x7fb03c0b3260 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb03affd700 1 -- 172.21.15.119:0/1538114917 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034012430 con 0x7fb03c0b3260 2024-09-13T16:33:44.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 msgr2=0x7fb03c0b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 --2- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c0b36f0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7fb034009d90 tx=0x7fb03400b330 comp rx=0 tx=0).stop 2024-09-13T16:33:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- 172.21.15.119:0/1538114917 shutdown_connections 2024-09-13T16:33:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 --2- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c0b36f0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 --2- 172.21.15.119:0/1538114917 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c0ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- 172.21.15.119:0/1538114917 >> 172.21.15.119:0/1538114917 conn(0x7fb03c0b0980 msgr2=0x7fb03c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- 172.21.15.119:0/1538114917 shutdown_connections 2024-09-13T16:33:44.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- 172.21.15.119:0/1538114917 wait complete. 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 Processor -- start 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- start start 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c14f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c15bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb03c15dbe0 0x7fb03c0af3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c1f1f30 con 0x7fb03c15dbe0 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c1f2170 con 0x7fb03c0b3260 2024-09-13T16:33:44.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb041295700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03c1f2450 con 0x7fb03c0add60 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb03c15dbe0 0x7fb03c0af3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb03c15dbe0 0x7fb03c0af3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35306/0 (socket says 172.21.15.119:35306) 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 -- 172.21.15.119:0/3982073100 learned_addr learned my addr 172.21.15.119:0/3982073100 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb03bfff700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c14f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb03b7fe700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c15bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 -- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 msgr2=0x7fb03c14f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c14f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 -- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 msgr2=0x7fb03c15bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c15bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 -- 172.21.15.119:0/3982073100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034009a70 con 0x7fb03c15dbe0 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb03b7fe700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c15bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:44.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.970+0000 7fb040a94700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb03c15dbe0 0x7fb03c0af3a0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fb034014040 tx=0x7fb034004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb038ff9700 1 -- 172.21.15.119:0/3982073100 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034012430 con 0x7fb03c15dbe0 2024-09-13T16:33:44.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb03c1f2730 con 0x7fb03c15dbe0 2024-09-13T16:33:44.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb03c1f2cf0 con 0x7fb03c15dbe0 2024-09-13T16:33:44.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb038ff9700 1 -- 172.21.15.119:0/3982073100 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034020070 con 0x7fb03c15dbe0 2024-09-13T16:33:44.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb038ff9700 1 -- 172.21.15.119:0/3982073100 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034004240 con 0x7fb03c15dbe0 2024-09-13T16:33:44.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01c005310 con 0x7fb03c15dbe0 2024-09-13T16:33:44.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb038ff9700 1 -- 172.21.15.119:0/3982073100 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb0340043a0 con 0x7fb03c15dbe0 2024-09-13T16:33:44.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb038ff9700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb028066730 0x7fb028068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:44.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb038ff9700 1 -- 172.21.15.119:0/3982073100 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fb034022070 con 0x7fb03c15dbe0 2024-09-13T16:33:44.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.974+0000 7fb03bfff700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb028066730 0x7fb028068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:44.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.978+0000 7fb03bfff700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb028066730 0x7fb028068bf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb030009ac0 tx=0x7fb030008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:44.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:44.982+0000 7fb038ff9700 1 -- 172.21.15.119:0/3982073100 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb0340539d0 con 0x7fb03c15dbe0 2024-09-13T16:33:45.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb01c005b40 con 0x7fb03c15dbe0 2024-09-13T16:33:45.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb038ff9700 1 -- 172.21.15.119:0/3982073100 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb034056ff0 con 0x7fb03c15dbe0 2024-09-13T16:33:45.152 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:33:45.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb028066730 msgr2=0x7fb028068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb028066730 0x7fb028068bf0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fb030009ac0 tx=0x7fb030008040 comp rx=0 tx=0).stop 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb03c15dbe0 msgr2=0x7fb03c0af3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb03c15dbe0 0x7fb03c0af3a0 secure :-1 s=READY pgs=1494 cs=0 l=1 rev1=1 crypto rx=0x7fb034014040 tx=0x7fb034004af0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 shutdown_connections 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb028066730 0x7fb028068bf0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb03c15dbe0 0x7fb03c0af3a0 unknown :-1 s=CLOSED pgs=1494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb03c0b3260 0x7fb03c15bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 --2- 172.21.15.119:0/3982073100 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb03c0add60 0x7fb03c14f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 >> 172.21.15.119:0/3982073100 conn(0x7fb03c0b0980 msgr2=0x7fb03c066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 shutdown_connections 2024-09-13T16:33:45.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.150+0000 7fb041295700 1 -- 172.21.15.119:0/3982073100 wait complete. 2024-09-13T16:33:45.167 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:33:45.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.278+0000 7f99e0e0b700 1 Processor -- start 2024-09-13T16:33:45.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.278+0000 7f99e0e0b700 1 -- start start 2024-09-13T16:33:45.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f99dc144180 0x7f99dc06c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f99dc0ce1d0 0x7f99dc0b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99dc06cb40 con 0x7f99dc0cc010 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99dc0b1c50 con 0x7f99dc0ce1d0 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99dc06ce20 con 0x7f99dc144180 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f99dc0ce1d0 0x7f99dc0b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f99dc0ce1d0 0x7f99dc0b20a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35322/0 (socket says 172.21.15.119:35322) 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 -- 172.21.15.119:0/961582053 learned_addr learned my addr 172.21.15.119:0/961582053 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99db7fe700 1 --2- 172.21.15.119:0/961582053 >> v2:172.21.15.145:3300/0 conn(0x7f99dc144180 0x7f99dc06c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 -- 172.21.15.119:0/961582053 >> v2:172.21.15.145:3300/0 conn(0x7f99dc144180 msgr2=0x7f99dc06c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 --2- 172.21.15.119:0/961582053 >> v2:172.21.15.145:3300/0 conn(0x7f99dc144180 0x7f99dc06c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 -- 172.21.15.119:0/961582053 >> v1:172.21.15.145:6789/0 conn(0x7f99dc0cc010 legacy=0x7f99dc0cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 -- 172.21.15.119:0/961582053 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc06d100 con 0x7f99dc0ce1d0 2024-09-13T16:33:45.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99dbfff700 1 --2- 172.21.15.119:0/961582053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0ce1d0 0x7f99dc0b20a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0b1560 tx=0x7f99d4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f788945b716984fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99da7fc700 1 -- 172.21.15.119:0/961582053 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4011830 con 0x7f99dc0ce1d0 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99da7fc700 1 -- 172.21.15.119:0/961582053 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99d4011990 con 0x7f99dc0ce1d0 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99da7fc700 1 -- 172.21.15.119:0/961582053 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99d4011d20 con 0x7f99dc0ce1d0 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- 172.21.15.119:0/961582053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0ce1d0 msgr2=0x7f99dc0b20a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/961582053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0ce1d0 0x7f99dc0b20a0 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0b1560 tx=0x7f99d4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- 172.21.15.119:0/961582053 shutdown_connections 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/961582053 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0ce1d0 0x7f99dc0b20a0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/961582053 >> v2:172.21.15.145:3300/0 conn(0x7f99dc144180 0x7f99dc06c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- 172.21.15.119:0/961582053 >> 172.21.15.119:0/961582053 conn(0x7f99dc0b0980 msgr2=0x7f99dc0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- 172.21.15.119:0/961582053 shutdown_connections 2024-09-13T16:33:45.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.282+0000 7f99e0e0b700 1 -- 172.21.15.119:0/961582053 wait complete. 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 Processor -- start 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 -- start start 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0cc010 0x7f99dc0c3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99dc0ce1d0 0x7f99dc0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99dc144180 0x7f99dc0a89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc0a8f30 con 0x7f99dc0cc010 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc0a9130 con 0x7f99dc0ce1d0 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99dc0a9410 con 0x7f99dc144180 2024-09-13T16:33:45.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0cc010 0x7f99dc0c3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0cc010 0x7f99dc0c3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35328/0 (socket says 172.21.15.119:35328) 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 -- 172.21.15.119:0/3666206747 learned_addr learned my addr 172.21.15.119:0/3666206747 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99dbfff700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99dc144180 0x7f99dc0a89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99daffd700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99dc0ce1d0 0x7f99dc0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 -- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99dc144180 msgr2=0x7f99dc0a89f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99dc144180 0x7f99dc0a89f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 -- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99dc0ce1d0 msgr2=0x7f99dc0a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99dc0ce1d0 0x7f99dc0a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 -- 172.21.15.119:0/3666206747 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4009570 con 0x7f99dc0cc010 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99dbfff700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99dc144180 0x7f99dc0a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99db7fe700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0cc010 0x7f99dc0c3ce0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00efc0 tx=0x7f99cc00c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99d3fff700 1 -- 172.21.15.119:0/3666206747 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc00cc00 con 0x7f99dc0cc010 2024-09-13T16:33:45.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99dc0a96f0 con 0x7f99dc0cc010 2024-09-13T16:33:45.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99d3fff700 1 -- 172.21.15.119:0/3666206747 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99cc00f040 con 0x7f99dc0cc010 2024-09-13T16:33:45.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99d3fff700 1 -- 172.21.15.119:0/3666206747 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99cc014480 con 0x7f99dc0cc010 2024-09-13T16:33:45.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.286+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99dc0a9cb0 con 0x7f99dc0cc010 2024-09-13T16:33:45.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.290+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99dc06c480 con 0x7f99dc0cc010 2024-09-13T16:33:45.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.290+0000 7f99d3fff700 1 -- 172.21.15.119:0/3666206747 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f99cc0145e0 con 0x7f99dc0cc010 2024-09-13T16:33:45.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.290+0000 7f99d3fff700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99c0066730 0x7f99c0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.290+0000 7f99d3fff700 1 -- 172.21.15.119:0/3666206747 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f99cc084bb0 con 0x7f99dc0cc010 2024-09-13T16:33:45.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.290+0000 7f99daffd700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99c0066730 0x7f99c0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.290+0000 7f99daffd700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99c0066730 0x7f99c0068bf0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0c3a20 tx=0x7f99c4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.298+0000 7f99d3fff700 1 -- 172.21.15.119:0/3666206747 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f99cc051cd0 con 0x7f99dc0cc010 2024-09-13T16:33:45.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.466+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99dc072c50 con 0x7f99dc0cc010 2024-09-13T16:33:45.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.466+0000 7f99d3fff700 1 -- 172.21.15.119:0/3666206747 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) v1 ==== 100+0+11 (secure 0 0 0) 0x7f99cc0552f0 con 0x7f99dc0cc010 2024-09-13T16:33:45.470 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:33:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99c0066730 msgr2=0x7f99c0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99c0066730 0x7f99c0068bf0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f99dc0c3a20 tx=0x7f99c4009040 comp rx=0 tx=0).stop 2024-09-13T16:33:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0cc010 msgr2=0x7f99dc0c3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0cc010 0x7f99dc0c3ce0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f99cc00efc0 tx=0x7f99cc00c0c0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 shutdown_connections 2024-09-13T16:33:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99c0066730 0x7f99c0068bf0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99dc144180 0x7f99dc0a89f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99dc0ce1d0 0x7f99dc0a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 --2- 172.21.15.119:0/3666206747 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99dc0cc010 0x7f99dc0c3ce0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 >> 172.21.15.119:0/3666206747 conn(0x7f99dc0b0980 msgr2=0x7f99dc1e2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:45.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 shutdown_connections 2024-09-13T16:33:45.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.470+0000 7f99e0e0b700 1 -- 172.21.15.119:0/3666206747 wait complete. 2024-09-13T16:33:45.485 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:45.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7318bae700 1 Processor -- start 2024-09-13T16:33:45.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7318bae700 1 -- start start 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7318bae700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f73140cc060 0x7f73140cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7318bae700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f73140cde70 0x7f73140b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7318bae700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731406cf60 con 0x7f73140adb30 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7318bae700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73140cc4d0 con 0x7f73140cde70 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7318bae700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f731406c350 con 0x7f73140cc060 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f73140cde70 0x7f73140b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f73140cde70 0x7f73140b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35344/0 (socket says 172.21.15.119:35344) 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 -- 172.21.15.119:0/397034780 learned_addr learned my addr 172.21.15.119:0/397034780 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:45.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7312ffd700 1 --2- 172.21.15.119:0/397034780 >> v2:172.21.15.145:3300/0 conn(0x7f73140cc060 0x7f73140cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 -- 172.21.15.119:0/397034780 >> v2:172.21.15.145:3300/0 conn(0x7f73140cc060 msgr2=0x7f73140cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 --2- 172.21.15.119:0/397034780 >> v2:172.21.15.145:3300/0 conn(0x7f73140cc060 0x7f73140cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 -- 172.21.15.119:0/397034780 >> v1:172.21.15.145:6789/0 conn(0x7f73140adb30 legacy=0x7f731406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 -- 172.21.15.119:0/397034780 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731406c630 con 0x7f73140cde70 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.594+0000 7f7313fff700 1 --2- 172.21.15.119:0/397034780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140cde70 0x7f73140b2380 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f73140cb210 tx=0x7f730c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b9d8def5b42e0b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f73127fc700 1 -- 172.21.15.119:0/397034780 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00e070 con 0x7f73140cde70 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f73127fc700 1 -- 172.21.15.119:0/397034780 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730c00b910 con 0x7f73140cde70 2024-09-13T16:33:45.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f73127fc700 1 -- 172.21.15.119:0/397034780 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730c00bcc0 con 0x7f73140cde70 2024-09-13T16:33:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- 172.21.15.119:0/397034780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140cde70 msgr2=0x7f73140b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 --2- 172.21.15.119:0/397034780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140cde70 0x7f73140b2380 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 crypto rx=0x7f73140cb210 tx=0x7f730c00b330 comp rx=0 tx=0).stop 2024-09-13T16:33:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- 172.21.15.119:0/397034780 shutdown_connections 2024-09-13T16:33:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 --2- 172.21.15.119:0/397034780 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140cde70 0x7f73140b2380 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 --2- 172.21.15.119:0/397034780 >> v2:172.21.15.145:3300/0 conn(0x7f73140cc060 0x7f73140cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- 172.21.15.119:0/397034780 >> 172.21.15.119:0/397034780 conn(0x7f731406a350 msgr2=0x7f731406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:45.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- 172.21.15.119:0/397034780 shutdown_connections 2024-09-13T16:33:45.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- 172.21.15.119:0/397034780 wait complete. 2024-09-13T16:33:45.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 Processor -- start 2024-09-13T16:33:45.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- start start 2024-09-13T16:33:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140adb30 0x7f731415d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73140cc060 0x7f731415bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73140cde70 0x7f7314139e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f731415c1b0 con 0x7f73140adb30 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7314138050 con 0x7f73140cde70 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f7318bae700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73141382f0 con 0x7f73140cc060 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f73137fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140adb30 0x7f731415d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f73137fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140adb30 0x7f731415d6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35360/0 (socket says 172.21.15.119:35360) 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.598+0000 7f73137fe700 1 -- 172.21.15.119:0/710607079 learned_addr learned my addr 172.21.15.119:0/710607079 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f7312ffd700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73140cc060 0x7f731415bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f7313fff700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73140cde70 0x7f7314139e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f73137fe700 1 -- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73140cc060 msgr2=0x7f731415bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f73137fe700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73140cc060 0x7f731415bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f73137fe700 1 -- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73140cde70 msgr2=0x7f7314139e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f73137fe700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73140cde70 0x7f7314139e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f73137fe700 1 -- 172.21.15.119:0/710607079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c009b30 con 0x7f73140adb30 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f7312ffd700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73140cc060 0x7f731415bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f7313fff700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73140cde70 0x7f7314139e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f73137fe700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140adb30 0x7f731415d6b0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00c500 tx=0x7f72fc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f730bfff700 1 -- 172.21.15.119:0/710607079 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0052d0 con 0x7f73140adb30 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f730bfff700 1 -- 172.21.15.119:0/710607079 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72fc005430 con 0x7f73140adb30 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73141385d0 con 0x7f73140adb30 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f730bfff700 1 -- 172.21.15.119:0/710607079 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72fc0055a0 con 0x7f73140adb30 2024-09-13T16:33:45.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7314138b90 con 0x7f73140adb30 2024-09-13T16:33:45.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7314138db0 con 0x7f73140adb30 2024-09-13T16:33:45.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.602+0000 7f730bfff700 1 -- 172.21.15.119:0/710607079 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f72fc0036a0 con 0x7f73140adb30 2024-09-13T16:33:45.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.606+0000 7f730bfff700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f72f4066730 0x7f72f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.606+0000 7f730bfff700 1 -- 172.21.15.119:0/710607079 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f72fc083b50 con 0x7f73140adb30 2024-09-13T16:33:45.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.606+0000 7f7312ffd700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f72f4066730 0x7f72f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.606+0000 7f7312ffd700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f72f4066730 0x7f72f4068bf0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7314137e40 tx=0x7f7304008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.610+0000 7f730bfff700 1 -- 172.21.15.119:0/710607079 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f72fc083e60 con 0x7f73140adb30 2024-09-13T16:33:45.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.810+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73140adf10 con 0x7f73140adb30 2024-09-13T16:33:45.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f730bfff700 1 -- 172.21.15.119:0/710607079 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f72fc020090 con 0x7f73140adb30 2024-09-13T16:33:45.816 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:45.816 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":69,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:06.506474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:45.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f72f4066730 msgr2=0x7f72f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f72f4066730 0x7f72f4068bf0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f7314137e40 tx=0x7f7304008040 comp rx=0 tx=0).stop 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140adb30 msgr2=0x7f731415d6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140adb30 0x7f731415d6b0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7f72fc00c500 tx=0x7f72fc00c8c0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 shutdown_connections 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f72f4066730 0x7f72f4068bf0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73140cde70 0x7f7314139e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73140cc060 0x7f731415bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 --2- 172.21.15.119:0/710607079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73140adb30 0x7f731415d6b0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 >> 172.21.15.119:0/710607079 conn(0x7f731406a350 msgr2=0x7f73140b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 shutdown_connections 2024-09-13T16:33:45.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.814+0000 7f7318bae700 1 -- 172.21.15.119:0/710607079 wait complete. 2024-09-13T16:33:45.820 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 69 2024-09-13T16:33:45.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:45.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcdc98e700 1 Processor -- start 2024-09-13T16:33:45.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcdc98e700 1 -- start start 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcdc98e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fdcd81656a0 0x7fdcd8167a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcdc98e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80cdd50 con 0x7fdcd80d2a30 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcdc98e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd80cdf90 con 0x7fdcd80b3a20 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcdc98e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd80ce270 con 0x7fdcd81656a0 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fdcd80d2a30 0x7fdcd80d2e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56690/0 (socket says 172.21.15.119:56690) 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 -- 172.21.15.119:0/722038813 learned_addr learned my addr 172.21.15.119:0/722038813 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd7fff700 1 --2- 172.21.15.119:0/722038813 >> v2:172.21.15.119:3300/0 conn(0x7fdcd81656a0 0x7fdcd8167a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447347082 0 0) 0x7fdcc8001110 con 0x7fdcd80d2a30 2024-09-13T16:33:45.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc00035d0 con 0x7fdcd80d2a30 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4056031779 0 0) 0x7fdcc00035d0 con 0x7fdcd80d2a30 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 >> v2:172.21.15.119:3300/0 conn(0x7fdcd81656a0 msgr2=0x7fdcd8167a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 --2- 172.21.15.119:0/722038813 >> v2:172.21.15.119:3300/0 conn(0x7fdcd81656a0 0x7fdcd8167a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 >> v1:172.21.15.145:6789/0 conn(0x7fdcd80b3a20 legacy=0x7fdcd80cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80ce550 con 0x7fdcd80d2a30 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fdcc8001800 con 0x7fdcd80d2a30 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd7fff700 1 --2- 172.21.15.119:0/722038813 >> v2:172.21.15.119:3300/0 conn(0x7fdcd81656a0 0x7fdcd8167a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 >> v1:172.21.15.119:6789/0 conn(0x7fdcd80d2a30 legacy=0x7fdcd80d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcc0003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcc0004420 0x7fdcc00048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcc00050f0 0x7fdcc000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80ce270 con 0x7fdcc0003a20 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc000c050 con 0x7fdcc00050f0 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcc000c210 con 0x7fdcc0004420 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcc0003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd7fff700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcc00050f0 0x7fdcc000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd6ffd700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcc0004420 0x7fdcc00048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 -- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcc0004420 msgr2=0x7fdcc00048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcc0004420 0x7fdcc00048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 -- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcc00050f0 msgr2=0x7fdcc000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:45.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcc00050f0 0x7fdcc000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 -- 172.21.15.119:0/722038813 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc00035d0 con 0x7fdcc0003a20 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd7fff700 1 -- 172.21.15.119:0/722038813 reap_dead start 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd77fe700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcc0003e00 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c320 tx=0x7fdcc00058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67898266cb7bd793 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc800ce20 con 0x7fdcc0003a20 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8005980 con 0x7fdcc0003a20 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.938+0000 7fdcd67fc700 1 -- 172.21.15.119:0/722038813 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8005d10 con 0x7fdcc0003a20 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- 172.21.15.119:0/722038813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 msgr2=0x7fdcc0003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcc0003e00 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 crypto rx=0x7fdcc800c320 tx=0x7fdcc00058a0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- 172.21.15.119:0/722038813 shutdown_connections 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/722038813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcc0003e00 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- 172.21.15.119:0/722038813 >> 172.21.15.119:0/722038813 conn(0x7fdcd808c0f0 msgr2=0x7fdcd808c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- 172.21.15.119:0/722038813 shutdown_connections 2024-09-13T16:33:45.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- 172.21.15.119:0/722038813 wait complete. 2024-09-13T16:33:45.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 Processor -- start 2024-09-13T16:33:45.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- start start 2024-09-13T16:33:45.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcd80b58f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcd80c9160 0x7fdcd80b5e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcd814c2f0 0x7fdcd814c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80b70a0 con 0x7fdcc0003a20 2024-09-13T16:33:45.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80c9920 con 0x7fdcd814c2f0 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcdc98e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd80c9640 con 0x7fdcd80c9160 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcd80b58f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcd77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcd80b58f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35398/0 (socket says 172.21.15.119:35398) 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.942+0000 7fdcd77fe700 1 -- 172.21.15.119:0/3007663414 learned_addr learned my addr 172.21.15.119:0/3007663414 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd6ffd700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcd80c9160 0x7fdcd80b5e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd7fff700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcd814c2f0 0x7fdcd814c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd77fe700 1 -- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcd80c9160 msgr2=0x7fdcd80b5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd77fe700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcd80c9160 0x7fdcd80b5e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd77fe700 1 -- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcd814c2f0 msgr2=0x7fdcd814c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd77fe700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcd814c2f0 0x7fdcd814c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd77fe700 1 -- 172.21.15.119:0/3007663414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd80b65c0 con 0x7fdcc0003a20 2024-09-13T16:33:45.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcd77fe700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcd80b58f0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000c020 tx=0x7fdcc8001b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcbffff700 1 -- 172.21.15.119:0/3007663414 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8004890 con 0x7fdcc0003a20 2024-09-13T16:33:45.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcbffff700 1 -- 172.21.15.119:0/3007663414 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc8004a10 con 0x7fdcc0003a20 2024-09-13T16:33:45.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd80b68a0 con 0x7fdcc0003a20 2024-09-13T16:33:45.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcbffff700 1 -- 172.21.15.119:0/3007663414 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdcc8004dc0 con 0x7fdcc0003a20 2024-09-13T16:33:45.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd814ccc0 con 0x7fdcc0003a20 2024-09-13T16:33:45.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd814ce50 con 0x7fdcc0003a20 2024-09-13T16:33:45.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcbffff700 1 -- 172.21.15.119:0/3007663414 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fdcc8019070 con 0x7fdcc0003a20 2024-09-13T16:33:45.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcbffff700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdcc4066730 0x7fdcc4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:45.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.946+0000 7fdcbffff700 1 -- 172.21.15.119:0/3007663414 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7fdcc8084fa0 con 0x7fdcc0003a20 2024-09-13T16:33:45.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.950+0000 7fdcd6ffd700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdcc4066730 0x7fdcc4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:45.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.950+0000 7fdcd6ffd700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdcc4066730 0x7fdcc4068bf0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdcd80b3e90 tx=0x7fdccc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:45.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:45.954+0000 7fdcbffff700 1 -- 172.21.15.119:0/3007663414 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdcc80520a0 con 0x7fdcc0003a20 2024-09-13T16:33:46.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.126+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcd80cdd50 con 0x7fdcc0003a20 2024-09-13T16:33:46.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcbffff700 1 -- 172.21.15.119:0/3007663414 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17233 (secure 0 0 0) 0x7fdcd80cdd50 con 0x7fdcc0003a20 2024-09-13T16:33:46.133 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:46.133 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":99,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:02.472922+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1e","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.119:6834/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.145:6835/3011171408":"2024-09-14T16:31:41.722614+0000","172.21.15.145:6834/3011171408":"2024-09-14T16:31:41.722614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:46.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdcc4066730 msgr2=0x7fdcc4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdcc4066730 0x7fdcc4068bf0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fdcd80b3e90 tx=0x7fdccc009040 comp rx=0 tx=0).stop 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 msgr2=0x7fdcd80b58f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcd80b58f0 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7fdcc000c020 tx=0x7fdcc8001b90 comp rx=0 tx=0).stop 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 shutdown_connections 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdcc4066730 0x7fdcc4068bf0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fdcd814c2f0 0x7fdcd814c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fdcd80c9160 0x7fdcd80b5e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 --2- 172.21.15.119:0/3007663414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fdcc0003a20 0x7fdcd80b58f0 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 >> 172.21.15.119:0/3007663414 conn(0x7fdcd808c0f0 msgr2=0x7fdcd80d19a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 shutdown_connections 2024-09-13T16:33:46.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.130+0000 7fdcdc98e700 1 -- 172.21.15.119:0/3007663414 wait complete. 2024-09-13T16:33:46.148 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:33:46.148 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:46.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac7fff700 1 Processor -- start 2024-09-13T16:33:46.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac7fff700 1 -- start start 2024-09-13T16:33:46.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac7fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffac80b3800 0x7ffac81642b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac7fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac80b2280 con 0x7ffac806dc40 2024-09-13T16:33:46.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac7fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffac80cc890 con 0x7ffac81594f0 2024-09-13T16:33:46.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac7fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffac8070c10 con 0x7ffac80b3800 2024-09-13T16:33:46.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac67fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ffac806dc40 0x7ffac80b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56696/0 (socket says 172.21.15.119:56696) 2024-09-13T16:33:46.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac67fc700 1 -- 172.21.15.119:0/3024030289 learned_addr learned my addr 172.21.15.119:0/3024030289 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:46.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.254+0000 7ffac77fe700 1 --2- 172.21.15.119:0/3024030289 >> v2:172.21.15.145:3300/0 conn(0x7ffac80b3800 0x7ffac81642b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3481446176 0 0) 0x7ffac80b2280 con 0x7ffac806dc40 2024-09-13T16:33:46.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab00035d0 con 0x7ffac806dc40 2024-09-13T16:33:46.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924484829 0 0) 0x7ffac80cc890 con 0x7ffac81594f0 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac80b2280 con 0x7ffac81594f0 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 647007793 0 0) 0x7ffab00035d0 con 0x7ffac806dc40 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac77fe700 1 -- 172.21.15.119:0/3024030289 >> v1:172.21.15.145:6789/0 conn(0x7ffac81594f0 legacy=0x7ffac806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac77fe700 1 -- 172.21.15.119:0/3024030289 >> v1:172.21.15.119:6789/0 conn(0x7ffac806dc40 legacy=0x7ffac80b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac77fe700 1 -- 172.21.15.119:0/3024030289 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac80b1970 con 0x7ffac80b3800 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac77fe700 1 --2- 172.21.15.119:0/3024030289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac80b3800 0x7ffac81642b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffac8070d50 tx=0x7ffac000b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=831a3e455375a0f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffac000e070 con 0x7ffac80b3800 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffac0003a40 con 0x7ffac80b3800 2024-09-13T16:33:46.261 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac5ffb700 1 -- 172.21.15.119:0/3024030289 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffabc001670 con 0x7ffac80b3800 2024-09-13T16:33:46.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac7fff700 1 -- 172.21.15.119:0/3024030289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac80b3800 msgr2=0x7ffac81642b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac7fff700 1 --2- 172.21.15.119:0/3024030289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac80b3800 0x7ffac81642b0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7ffac8070d50 tx=0x7ffac000b330 comp rx=0 tx=0).stop 2024-09-13T16:33:46.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac7fff700 1 -- 172.21.15.119:0/3024030289 shutdown_connections 2024-09-13T16:33:46.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac7fff700 1 --2- 172.21.15.119:0/3024030289 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac80b3800 0x7ffac81642b0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac7fff700 1 -- 172.21.15.119:0/3024030289 >> 172.21.15.119:0/3024030289 conn(0x7ffac806a350 msgr2=0x7ffac806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:46.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac7fff700 1 -- 172.21.15.119:0/3024030289 shutdown_connections 2024-09-13T16:33:46.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.258+0000 7ffac7fff700 1 -- 172.21.15.119:0/3024030289 wait complete. 2024-09-13T16:33:46.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 Processor -- start 2024-09-13T16:33:46.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 -- start start 2024-09-13T16:33:46.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffac806dc40 0x7ffac8147600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffac80b3800 0x7ffac81441f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac81594f0 0x7ffac8157b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac8147b40 con 0x7ffac806dc40 2024-09-13T16:33:46.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac81f0070 con 0x7ffac81594f0 2024-09-13T16:33:46.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac7fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffac81ef460 con 0x7ffac80b3800 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffac806dc40 0x7ffac8147600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffac806dc40 0x7ffac8147600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35408/0 (socket says 172.21.15.119:35408) 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac6ffd700 1 -- 172.21.15.119:0/2821262040 learned_addr learned my addr 172.21.15.119:0/2821262040 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac67fc700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffac80b3800 0x7ffac81441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.262+0000 7ffac77fe700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac81594f0 0x7ffac8157b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac6ffd700 1 -- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffac80b3800 msgr2=0x7ffac81441f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac6ffd700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffac80b3800 0x7ffac81441f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac6ffd700 1 -- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac81594f0 msgr2=0x7ffac8157b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac6ffd700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac81594f0 0x7ffac8157b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac6ffd700 1 -- 172.21.15.119:0/2821262040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac0009a70 con 0x7ffac806dc40 2024-09-13T16:33:46.269 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac67fc700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffac80b3800 0x7ffac81441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:46.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac77fe700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac81594f0 0x7ffac8157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:46.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac6ffd700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffac806dc40 0x7ffac8147600 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c490 tx=0x7ffab800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffaaf7fe700 1 -- 172.21.15.119:0/2821262040 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab80052d0 con 0x7ffac806dc40 2024-09-13T16:33:46.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffaaf7fe700 1 -- 172.21.15.119:0/2821262040 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffab8005450 con 0x7ffac806dc40 2024-09-13T16:33:46.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac81ef740 con 0x7ffac806dc40 2024-09-13T16:33:46.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffaaf7fe700 1 -- 172.21.15.119:0/2821262040 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab80038e0 con 0x7ffac806dc40 2024-09-13T16:33:46.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac81efd00 con 0x7ffac806dc40 2024-09-13T16:33:46.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffac80b30f0 con 0x7ffac806dc40 2024-09-13T16:33:46.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffaaf7fe700 1 -- 172.21.15.119:0/2821262040 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ffab8003a40 con 0x7ffac806dc40 2024-09-13T16:33:46.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffaaf7fe700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffab4066730 0x7ffab4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffaaf7fe700 1 -- 172.21.15.119:0/2821262040 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7ffab8083940 con 0x7ffac806dc40 2024-09-13T16:33:46.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.266+0000 7ffac67fc700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffab4066730 0x7ffab4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.270+0000 7ffac67fc700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffab4066730 0x7ffab4068bf0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ffabc001eb0 tx=0x7ffabc001c80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.274+0000 7ffaaf7fe700 1 -- 172.21.15.119:0/2821262040 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ffab80509d0 con 0x7ffac806dc40 2024-09-13T16:33:46.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.482+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffac81efec0 con 0x7ffac806dc40 2024-09-13T16:33:46.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.482+0000 7ffaaf7fe700 1 -- 172.21.15.119:0/2821262040 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4746 (secure 0 0 0) 0x7ffab8053ff0 con 0x7ffac806dc40 2024-09-13T16:33:46.486 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:46.486 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":69,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:06.506474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:46.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffab4066730 msgr2=0x7ffab4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffab4066730 0x7ffab4068bf0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7ffabc001eb0 tx=0x7ffabc001c80 comp rx=0 tx=0).stop 2024-09-13T16:33:46.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffac806dc40 msgr2=0x7ffac8147600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffac806dc40 0x7ffac8147600 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 crypto rx=0x7ffab800c490 tx=0x7ffab800c850 comp rx=0 tx=0).stop 2024-09-13T16:33:46.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 shutdown_connections 2024-09-13T16:33:46.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffab4066730 0x7ffab4068bf0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffac81594f0 0x7ffac8157b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffac80b3800 0x7ffac81441f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 --2- 172.21.15.119:0/2821262040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffac806dc40 0x7ffac8147600 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 >> 172.21.15.119:0/2821262040 conn(0x7ffac806a350 msgr2=0x7ffac815c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:46.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 shutdown_connections 2024-09-13T16:33:46.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.486+0000 7ffac7fff700 1 -- 172.21.15.119:0/2821262040 wait complete. 2024-09-13T16:33:46.492 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 69 2024-09-13T16:33:46.507 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:46.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.618+0000 7f6cf767d700 1 Processor -- start 2024-09-13T16:33:46.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.618+0000 7f6cf767d700 1 -- start start 2024-09-13T16:33:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf01591e0 0x7f6cf006ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6cf006b600 0x7f6cf006c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf00cfd00 con 0x7f6cf006cc30 2024-09-13T16:33:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf006d0e0 con 0x7f6cf006b600 2024-09-13T16:33:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf006b2a0 con 0x7f6cf01591e0 2024-09-13T16:33:46.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf4c18700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6cf006b600 0x7f6cf006c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5419700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf01591e0 0x7f6cf006ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5c1a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6cf006cc30 0x7f6cf0071a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56708/0 (socket says 172.21.15.119:56708) 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5c1a700 1 -- 172.21.15.119:0/3480738028 learned_addr learned my addr 172.21.15.119:0/3480738028 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5419700 1 -- 172.21.15.119:0/3480738028 >> v2:172.21.15.119:3300/0 conn(0x7f6cf006b600 msgr2=0x7f6cf006c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5419700 1 --2- 172.21.15.119:0/3480738028 >> v2:172.21.15.119:3300/0 conn(0x7f6cf006b600 0x7f6cf006c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5419700 1 -- 172.21.15.119:0/3480738028 >> v1:172.21.15.119:6789/0 conn(0x7f6cf006cc30 legacy=0x7f6cf0071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6ce7fff700 1 -- 172.21.15.119:0/3480738028 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152491360 0 0) 0x7f6cf00cfd00 con 0x7f6cf006cc30 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5419700 1 -- 172.21.15.119:0/3480738028 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf01595c0 con 0x7f6cf01591e0 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf4c18700 1 --2- 172.21.15.119:0/3480738028 >> v2:172.21.15.119:3300/0 conn(0x7f6cf006b600 0x7f6cf006c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:46.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf5419700 1 --2- 172.21.15.119:0/3480738028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf01591e0 0x7f6cf006ad60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009850 tx=0x7f6cec009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c1de008ea47bc18 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6ce7fff700 1 -- 172.21.15.119:0/3480738028 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec011830 con 0x7f6cf01591e0 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6ce7fff700 1 -- 172.21.15.119:0/3480738028 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec011990 con 0x7f6cf01591e0 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6ce7fff700 1 -- 172.21.15.119:0/3480738028 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec011d40 con 0x7f6cf01591e0 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- 172.21.15.119:0/3480738028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf01591e0 msgr2=0x7f6cf006ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 --2- 172.21.15.119:0/3480738028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf01591e0 0x7f6cf006ad60 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6cec009850 tx=0x7f6cec009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- 172.21.15.119:0/3480738028 shutdown_connections 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 --2- 172.21.15.119:0/3480738028 >> v2:172.21.15.119:3300/0 conn(0x7f6cf006b600 0x7f6cf006c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 --2- 172.21.15.119:0/3480738028 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf01591e0 0x7f6cf006ad60 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- 172.21.15.119:0/3480738028 >> 172.21.15.119:0/3480738028 conn(0x7f6cf00b3fd0 msgr2=0x7f6cf00b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- 172.21.15.119:0/3480738028 shutdown_connections 2024-09-13T16:33:46.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.622+0000 7f6cf767d700 1 -- 172.21.15.119:0/3480738028 wait complete. 2024-09-13T16:33:46.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 Processor -- start 2024-09-13T16:33:46.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 -- start start 2024-09-13T16:33:46.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf006b600 0x7f6cf0152ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf006cc30 0x7f6cf0149620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf01591e0 0x7f6cf01e0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0152a60 con 0x7f6cf01591e0 2024-09-13T16:33:46.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0153430 con 0x7f6cf006cc30 2024-09-13T16:33:46.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf767d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf0151b70 con 0x7f6cf006b600 2024-09-13T16:33:46.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf006b600 0x7f6cf0152ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf006b600 0x7f6cf0152ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:55830/0 (socket says 172.21.15.119:55830) 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 -- 172.21.15.119:0/1839595956 learned_addr learned my addr 172.21.15.119:0/1839595956 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5c1a700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf01591e0 0x7f6cf01e0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf4c18700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf006cc30 0x7f6cf0149620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 -- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf006cc30 msgr2=0x7f6cf0149620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf006cc30 0x7f6cf0149620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 -- 172.21.15.119:0/1839595956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf01591e0 msgr2=0x7f6cf01e0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf01591e0 0x7f6cf01e0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5419700 1 -- 172.21.15.119:0/1839595956 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce00094b0 con 0x7f6cf006b600 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.626+0000 7f6cf5c1a700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf01591e0 0x7f6cf01e0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:46.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6cf5419700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf006b600 0x7f6cf0152ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6cec014040 tx=0x7f6cec004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6ce5ffb700 1 -- 172.21.15.119:0/1839595956 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec019980 con 0x7f6cf006b600 2024-09-13T16:33:46.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6ce5ffb700 1 -- 172.21.15.119:0/1839595956 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cec01e070 con 0x7f6cf006b600 2024-09-13T16:33:46.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cec0094b0 con 0x7f6cf006b600 2024-09-13T16:33:46.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6ce5ffb700 1 -- 172.21.15.119:0/1839595956 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cec019d20 con 0x7f6cf006b600 2024-09-13T16:33:46.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf0152130 con 0x7f6cf006b600 2024-09-13T16:33:46.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf006ba40 con 0x7f6cf006b600 2024-09-13T16:33:46.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6ce5ffb700 1 -- 172.21.15.119:0/1839595956 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6cec02d450 con 0x7f6cf006b600 2024-09-13T16:33:46.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.630+0000 7f6ce5ffb700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cd4066730 0x7f6cd4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.634+0000 7f6ce5ffb700 1 -- 172.21.15.119:0/1839595956 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f6cec085720 con 0x7f6cf006b600 2024-09-13T16:33:46.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.634+0000 7f6cf4c18700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cd4066730 0x7f6cd4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.634+0000 7f6cf4c18700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cd4066730 0x7f6cd4068bf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0153fd0 tx=0x7f6ce0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.642+0000 7f6ce5ffb700 1 -- 172.21.15.119:0/1839595956 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6cec0529a0 con 0x7f6cf006b600 2024-09-13T16:33:46.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.846+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cf0156330 con 0x7f6cf006b600 2024-09-13T16:33:46.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.850+0000 7f6ce5ffb700 1 -- 172.21.15.119:0/1839595956 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+4746 (secure 0 0 0) 0x7f6cec020090 con 0x7f6cf006b600 2024-09-13T16:33:46.851 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:46.852 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":69,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:31:28.960022+0000","modified":"2024-09-13T16:33:06.506474+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":99,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8442},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8442":{"gid":8442,"name":"c","rank":0,"incarnation":66,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/1434266893","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1434266893},{"type":"v1","addr":"172.21.15.119:6835","nonce":1434266893}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2024-09-13T16:33:46.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cd4066730 msgr2=0x7f6cd4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cd4066730 0x7f6cd4068bf0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0153fd0 tx=0x7f6ce0009240 comp rx=0 tx=0).stop 2024-09-13T16:33:46.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf006b600 msgr2=0x7f6cf0152ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf006b600 0x7f6cf0152ef0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f6cec014040 tx=0x7f6cec004770 comp rx=0 tx=0).stop 2024-09-13T16:33:46.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 shutdown_connections 2024-09-13T16:33:46.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6cd4066730 0x7f6cd4068bf0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6cf01591e0 0x7f6cf01e0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6cf006cc30 0x7f6cf0149620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 --2- 172.21.15.119:0/1839595956 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6cf006b600 0x7f6cf0152ef0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 >> 172.21.15.119:0/1839595956 conn(0x7f6cf00b3fd0 msgr2=0x7f6cf00cd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 shutdown_connections 2024-09-13T16:33:46.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.854+0000 7f6cf767d700 1 -- 172.21.15.119:0/1839595956 wait complete. 2024-09-13T16:33:46.858 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 69 2024-09-13T16:33:46.873 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:46.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7989d700 1 Processor -- start 2024-09-13T16:33:46.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7989d700 1 -- start start 2024-09-13T16:33:46.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7989d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffa7406dc40 0x7ffa740b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7989d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffa740b3630 0x7ffa7406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7989d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa740b2280 con 0x7ffa741594f0 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7989d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa740cc890 con 0x7ffa740b3630 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7989d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa740b1970 con 0x7ffa7406dc40 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffa740b3630 0x7ffa7406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ffa740b3630 0x7ffa7406fc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35448/0 (socket says 172.21.15.119:35448) 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 -- 172.21.15.119:0/1731590095 learned_addr learned my addr 172.21.15.119:0/1731590095 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:46.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa73fff700 1 --2- 172.21.15.119:0/1731590095 >> v2:172.21.15.145:3300/0 conn(0x7ffa7406dc40 0x7ffa740b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 -- 172.21.15.119:0/1731590095 >> v2:172.21.15.145:3300/0 conn(0x7ffa7406dc40 msgr2=0x7ffa740b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:46.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 --2- 172.21.15.119:0/1731590095 >> v2:172.21.15.145:3300/0 conn(0x7ffa7406dc40 0x7ffa740b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 -- 172.21.15.119:0/1731590095 >> v1:172.21.15.145:6789/0 conn(0x7ffa741594f0 legacy=0x7ffa7406d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 -- 172.21.15.119:0/1731590095 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa740b1c30 con 0x7ffa740b3630 2024-09-13T16:33:46.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa7909c700 1 --2- 172.21.15.119:0/1731590095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7406fc00 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cc010 tx=0x7ffa6c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26825ec7d83026f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa737fe700 1 -- 172.21.15.119:0/1731590095 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c00e070 con 0x7ffa740b3630 2024-09-13T16:33:46.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa737fe700 1 -- 172.21.15.119:0/1731590095 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6c00b910 con 0x7ffa740b3630 2024-09-13T16:33:46.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.986+0000 7ffa737fe700 1 -- 172.21.15.119:0/1731590095 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c00bcc0 con 0x7ffa740b3630 2024-09-13T16:33:46.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 -- 172.21.15.119:0/1731590095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 msgr2=0x7ffa7406fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 --2- 172.21.15.119:0/1731590095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7406fc00 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cc010 tx=0x7ffa6c00b330 comp rx=0 tx=0).stop 2024-09-13T16:33:46.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 -- 172.21.15.119:0/1731590095 shutdown_connections 2024-09-13T16:33:46.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 --2- 172.21.15.119:0/1731590095 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7406fc00 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 --2- 172.21.15.119:0/1731590095 >> v2:172.21.15.145:3300/0 conn(0x7ffa7406dc40 0x7ffa740b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 -- 172.21.15.119:0/1731590095 >> 172.21.15.119:0/1731590095 conn(0x7ffa7406a350 msgr2=0x7ffa7406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:46.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 -- 172.21.15.119:0/1731590095 shutdown_connections 2024-09-13T16:33:46.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.990+0000 7ffa7989d700 1 -- 172.21.15.119:0/1731590095 wait complete. 2024-09-13T16:33:46.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 Processor -- start 2024-09-13T16:33:46.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 -- start start 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa7406dc40 0x7ffa7413db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7413c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa741594f0 0x7ffa7406cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7413ca70 con 0x7ffa740b3630 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7413cd50 con 0x7ffa741594f0 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7413d030 con 0x7ffa7406dc40 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7413c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa73fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7413c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35450/0 (socket says 172.21.15.119:35450) 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa73fff700 1 -- 172.21.15.119:0/185005662 learned_addr learned my addr 172.21.15.119:0/185005662 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:46.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7889b700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa7406dc40 0x7ffa7413db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7909c700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa741594f0 0x7ffa7406cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7889b700 1 -- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa741594f0 msgr2=0x7ffa7406cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7889b700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa741594f0 0x7ffa7406cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7889b700 1 -- 172.21.15.119:0/185005662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 msgr2=0x7ffa7413c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7889b700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7413c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7889b700 1 -- 172.21.15.119:0/185005662 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c009b30 con 0x7ffa7406dc40 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa73fff700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7413c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7889b700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa7406dc40 0x7ffa7413db10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800c410 tx=0x7ffa68001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa717fa700 1 -- 172.21.15.119:0/185005662 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa680052d0 con 0x7ffa7406dc40 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa717fa700 1 -- 172.21.15.119:0/185005662 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa68005430 con 0x7ffa7406dc40 2024-09-13T16:33:46.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7413d310 con 0x7ffa7406dc40 2024-09-13T16:33:47.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa717fa700 1 -- 172.21.15.119:0/185005662 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa680055a0 con 0x7ffa7406dc40 2024-09-13T16:33:47.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.994+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7413c680 con 0x7ffa7406dc40 2024-09-13T16:33:47.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.998+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa7413d830 con 0x7ffa7406dc40 2024-09-13T16:33:47.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.998+0000 7ffa717fa700 1 -- 172.21.15.119:0/185005662 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ffa680038e0 con 0x7ffa7406dc40 2024-09-13T16:33:47.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.998+0000 7ffa717fa700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa58066730 0x7ffa58068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.998+0000 7ffa717fa700 1 -- 172.21.15.119:0/185005662 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7ffa68083660 con 0x7ffa7406dc40 2024-09-13T16:33:47.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.998+0000 7ffa73fff700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa58066730 0x7ffa58068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:46.998+0000 7ffa73fff700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa58066730 0x7ffa58068bf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cb220 tx=0x7ffa60009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:47.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.006+0000 7ffa717fa700 1 -- 172.21.15.119:0/185005662 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ffa7413d830 con 0x7ffa7406dc40 2024-09-13T16:33:47.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.182+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa7413d830 con 0x7ffa7406dc40 2024-09-13T16:33:47.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa717fa700 1 -- 172.21.15.119:0/185005662 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) v1 ==== 74+0+17233 (secure 0 0 0) 0x7ffa7413d830 con 0x7ffa7406dc40 2024-09-13T16:33:47.188 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:47.188 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":99,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:02.472922+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:31:27.299109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"94","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_data","create_time":"2024-09-13T16:31:28.044326+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"14.2","mappings":[{"from":1,"to":7}]},{"pgid":"14.12","mappings":[{"from":1,"to":7}]},{"pgid":"14.1e","mappings":[{"from":1,"to":5}]},{"pgid":"14.3c","mappings":[{"from":1,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.119:6834/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.145:6835/3011171408":"2024-09-14T16:31:41.722614+0000","172.21.15.145:6834/3011171408":"2024-09-14T16:31:41.722614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa58066730 msgr2=0x7ffa58068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa58066730 0x7ffa58068bf0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ffa740cb220 tx=0x7ffa60009240 comp rx=0 tx=0).stop 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa7406dc40 msgr2=0x7ffa7413db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa7406dc40 0x7ffa7413db10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ffa6800c410 tx=0x7ffa68001e70 comp rx=0 tx=0).stop 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 shutdown_connections 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa58066730 0x7ffa58068bf0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa741594f0 0x7ffa7406cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa740b3630 0x7ffa7413c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 --2- 172.21.15.119:0/185005662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa7406dc40 0x7ffa7413db10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 >> 172.21.15.119:0/185005662 conn(0x7ffa7406a350 msgr2=0x7ffa7406fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:47.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 shutdown_connections 2024-09-13T16:33:47.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.186+0000 7ffa7989d700 1 -- 172.21.15.119:0/185005662 wait complete. 2024-09-13T16:33:47.203 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:33:47.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 Processor -- start 2024-09-13T16:33:47.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- start start 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb00adb30 0x7f0fb00cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0fb006bb90 0x7f0fb00cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f0fb00cde70 0x7f0fb015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb006cf60 con 0x7f0fb006bb90 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb006c350 con 0x7f0fb00cde70 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb006c630 con 0x7f0fb00adb30 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0fb006bb90 0x7f0fb00cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0fb006bb90 0x7f0fb00cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35452/0 (socket says 172.21.15.119:35452) 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 -- 172.21.15.119:0/2049109265 learned_addr learned my addr 172.21.15.119:0/2049109265 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb56a9700 1 --2- 172.21.15.119:0/2049109265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb00adb30 0x7f0fb00cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb5eaa700 1 --2- 172.21.15.119:0/2049109265 >> v2:172.21.15.145:3300/0 conn(0x7f0fb00cde70 0x7f0fb015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 -- 172.21.15.119:0/2049109265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb00adb30 msgr2=0x7f0fb00cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 --2- 172.21.15.119:0/2049109265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb00adb30 0x7f0fb00cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 -- 172.21.15.119:0/2049109265 >> v2:172.21.15.145:3300/0 conn(0x7f0fb00cde70 msgr2=0x7f0fb015e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 --2- 172.21.15.119:0/2049109265 >> v2:172.21.15.145:3300/0 conn(0x7f0fb00cde70 0x7f0fb015e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 -- 172.21.15.119:0/2049109265 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb006c910 con 0x7f0fb006bb90 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb5eaa700 1 --2- 172.21.15.119:0/2049109265 >> v2:172.21.15.145:3300/0 conn(0x7f0fb00cde70 0x7f0fb015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb4ea8700 1 --2- 172.21.15.119:0/2049109265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb006bb90 0x7f0fb00cd930 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00cb1e0 tx=0x7f0fa000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56d18fce552b9cd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fa7fff700 1 -- 172.21.15.119:0/2049109265 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa000e070 con 0x7f0fb006bb90 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fa7fff700 1 -- 172.21.15.119:0/2049109265 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa000b9c0 con 0x7f0fb006bb90 2024-09-13T16:33:47.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fa7fff700 1 -- 172.21.15.119:0/2049109265 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa000bd70 con 0x7f0fb006bb90 2024-09-13T16:33:47.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- 172.21.15.119:0/2049109265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb006bb90 msgr2=0x7f0fb00cd930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/2049109265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb006bb90 0x7f0fb00cd930 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7f0fb00cb1e0 tx=0x7f0fa000b330 comp rx=0 tx=0).stop 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- 172.21.15.119:0/2049109265 shutdown_connections 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/2049109265 >> v2:172.21.15.145:3300/0 conn(0x7f0fb00cde70 0x7f0fb015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/2049109265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb006bb90 0x7f0fb00cd930 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/2049109265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb00adb30 0x7f0fb00cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- 172.21.15.119:0/2049109265 >> 172.21.15.119:0/2049109265 conn(0x7f0fb006a350 msgr2=0x7f0fb006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- 172.21.15.119:0/2049109265 shutdown_connections 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.314+0000 7f0fb66ab700 1 -- 172.21.15.119:0/2049109265 wait complete. 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 Processor -- start 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 -- start start 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb006bb90 0x7f0fb0070d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb00adb30 0x7f0fb00ad180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb00cde70 0x7f0fb013e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb00ae490 con 0x7f0fb00cde70 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb0071bc0 con 0x7f0fb00adb30 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb015b540 con 0x7f0fb006bb90 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb00cde70 0x7f0fb013e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb00cde70 0x7f0fb013e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35464/0 (socket says 172.21.15.119:35464) 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 -- 172.21.15.119:0/1893956205 learned_addr learned my addr 172.21.15.119:0/1893956205 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb56a9700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb006bb90 0x7f0fb0070d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 -- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb006bb90 msgr2=0x7f0fb0070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb006bb90 0x7f0fb0070d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 -- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb00adb30 msgr2=0x7f0fb00ad180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb4ea8700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb00adb30 0x7f0fb00ad180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb00adb30 0x7f0fb00ad180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 -- 172.21.15.119:0/1893956205 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fa0009b30 con 0x7f0fb00cde70 2024-09-13T16:33:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb4ea8700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb00adb30 0x7f0fb00ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb5eaa700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb00cde70 0x7f0fb013e700 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800c7f0 tx=0x7f0fa800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fa5ffb700 1 -- 172.21.15.119:0/1893956205 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8011800 con 0x7f0fb00cde70 2024-09-13T16:33:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fa5ffb700 1 -- 172.21.15.119:0/1893956205 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fa8011980 con 0x7f0fb00cde70 2024-09-13T16:33:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb00718e0 con 0x7f0fb00cde70 2024-09-13T16:33:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fa5ffb700 1 -- 172.21.15.119:0/1893956205 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fa8011d30 con 0x7f0fb00cde70 2024-09-13T16:33:47.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.318+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb013ec40 con 0x7f0fb00cde70 2024-09-13T16:33:47.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.322+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb00ae640 con 0x7f0fb00cde70 2024-09-13T16:33:47.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.322+0000 7f0fa5ffb700 1 -- 172.21.15.119:0/1893956205 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0fa8022350 con 0x7f0fb00cde70 2024-09-13T16:33:47.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.322+0000 7f0fa5ffb700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f9c066730 0x7f0f9c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.322+0000 7f0fa5ffb700 1 -- 172.21.15.119:0/1893956205 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6317+0+0 (secure 0 0 0) 0x7f0fa8084310 con 0x7f0fb00cde70 2024-09-13T16:33:47.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.322+0000 7f0fb56a9700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f9c066730 0x7f0f9c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.322+0000 7f0fb56a9700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f9c066730 0x7f0f9c068bf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01f3090 tx=0x7f0fac008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:47.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.326+0000 7f0fa5ffb700 1 -- 172.21.15.119:0/1893956205 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0fb00ae640 con 0x7f0fb00cde70 2024-09-13T16:33:47.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.530+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0fb00ae640 con 0x7f0fb00cde70 2024-09-13T16:33:47.533 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:33:47.530+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:33:47.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.542+0000 7f0fa5ffb700 1 -- 172.21.15.119:0/1893956205 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v70) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0fb00ae640 con 0x7f0fb00cde70 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f9c066730 msgr2=0x7f0f9c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f9c066730 0x7f0f9c068bf0 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7f0fb01f3090 tx=0x7f0fac008040 comp rx=0 tx=0).stop 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb00cde70 msgr2=0x7f0fb013e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb00cde70 0x7f0fb013e700 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7f0fa800c7f0 tx=0x7f0fa800cbb0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 shutdown_connections 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f9c066730 0x7f0f9c068bf0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb00cde70 0x7f0fb013e700 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb00adb30 0x7f0fb00ad180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 --2- 172.21.15.119:0/1893956205 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb006bb90 0x7f0fb0070d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 >> 172.21.15.119:0/1893956205 conn(0x7f0fb006a350 msgr2=0x7f0fb015c410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:47.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 shutdown_connections 2024-09-13T16:33:47.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.546+0000 7f0fb66ab700 1 -- 172.21.15.119:0/1893956205 wait complete. 2024-09-13T16:33:47.550 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:33:47.550 INFO:tasks.ceph.mds.c.smithi119.stderr: -2163> 2024-09-13T16:33:08.503+0000 7f6b98b92700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:33:47.551 INFO:tasks.ceph.mds.c.smithi119.stderr: -1757> 2024-09-13T16:33:10.558+0000 7f6b9538b700 -1 mds.0.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000001 [...2,head] /dir/file auth v1 ap=1 DIRTYPARENT s=0 nl=2 n() (iversion lock) | dirtyparent=1 authpin=1 scrubqueue=0 0x55d31711c100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(14)0x10000000001:[<0x10000000000/file v6>,<0x1/dir v13>]//[]","memoryvalue":"(14)0x10000000001:[<0x10000000000/file v1>,<0x1/dir v20>]//[]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-09-13T16:33:47.563 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:33:47.581 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:33:47.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.674+0000 7f8de3fff700 1 Processor -- start 2024-09-13T16:33:47.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.674+0000 7f8de3fff700 1 -- start start 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de41652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8de40b24a0 0x7f8de40b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8de4071ac0 con 0x7f8de41612d0 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8de41657f0 con 0x7f8de40b24a0 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40b2e20 con 0x7f8de40ade80 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de2ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de41652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8dda7fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8de40b24a0 0x7f8de40b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de2ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de41652b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:55882/0 (socket says 172.21.15.119:55882) 2024-09-13T16:33:47.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de2ffd700 1 -- 172.21.15.119:0/455225988 learned_addr learned my addr 172.21.15.119:0/455225988 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:47.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8dda7fc700 1 -- 172.21.15.119:0/455225988 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 msgr2=0x7f8de41652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:47.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8dda7fc700 1 --2- 172.21.15.119:0/455225988 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de41652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8dda7fc700 1 -- 172.21.15.119:0/455225988 >> v1:172.21.15.145:6789/0 conn(0x7f8de41612d0 legacy=0x7f8de4161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8dda7fc700 1 -- 172.21.15.119:0/455225988 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4070eb0 con 0x7f8de40b24a0 2024-09-13T16:33:47.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8dda7fc700 1 --2- 172.21.15.119:0/455225988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de40b28e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009850 tx=0x7f8dcc009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5e8189321aa7e7a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:47.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de27fc700 1 -- 172.21.15.119:0/455225988 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc011830 con 0x7f8de40b24a0 2024-09-13T16:33:47.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de27fc700 1 -- 172.21.15.119:0/455225988 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8dcc011990 con 0x7f8de40b24a0 2024-09-13T16:33:47.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de27fc700 1 -- 172.21.15.119:0/455225988 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dcc011d40 con 0x7f8de40b24a0 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- 172.21.15.119:0/455225988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 msgr2=0x7f8de40b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 --2- 172.21.15.119:0/455225988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de40b28e0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8dcc009850 tx=0x7f8dcc009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- 172.21.15.119:0/455225988 shutdown_connections 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 --2- 172.21.15.119:0/455225988 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de40b28e0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 --2- 172.21.15.119:0/455225988 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de41652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- 172.21.15.119:0/455225988 >> 172.21.15.119:0/455225988 conn(0x7f8de40b0980 msgr2=0x7f8de40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- 172.21.15.119:0/455225988 shutdown_connections 2024-09-13T16:33:47.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.678+0000 7f8de3fff700 1 -- 172.21.15.119:0/455225988 wait complete. 2024-09-13T16:33:47.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 Processor -- start 2024-09-13T16:33:47.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 -- start start 2024-09-13T16:33:47.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de40acb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de415c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41612d0 0x7f8de40aab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40ad0d0 con 0x7f8de41612d0 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40ad870 con 0x7f8de40b24a0 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8de40cbe10 con 0x7f8de40ade80 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41612d0 0x7f8de40aab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8dda7fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de415c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de2ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de40acb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41612d0 0x7f8de40aab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35478/0 (socket says 172.21.15.119:35478) 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8dda7fc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de415c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:59242/0 (socket says 172.21.15.119:59242) 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 -- 172.21.15.119:0/3245756486 learned_addr learned my addr 172.21.15.119:0/3245756486 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:47.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 -- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 msgr2=0x7f8de40acb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de40acb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 -- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 msgr2=0x7f8de415c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de415c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 -- 172.21.15.119:0/3245756486 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dcc0094b0 con 0x7f8de41612d0 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de37fe700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41612d0 0x7f8de40aab00 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00cc50 tx=0x7f8ddc00cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8ddbfff700 1 -- 172.21.15.119:0/3245756486 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc003c20 con 0x7f8de41612d0 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8ddbfff700 1 -- 172.21.15.119:0/3245756486 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ddc003d80 con 0x7f8de41612d0 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de415ca40 con 0x7f8de41612d0 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8ddbfff700 1 -- 172.21.15.119:0/3245756486 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ddc00aab0 con 0x7f8de41612d0 2024-09-13T16:33:47.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.682+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de40ad2b0 con 0x7f8de41612d0 2024-09-13T16:33:47.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.686+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8de40cbf50 con 0x7f8de41612d0 2024-09-13T16:33:47.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.686+0000 7f8ddbfff700 1 -- 172.21.15.119:0/3245756486 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8ddc00ac10 con 0x7f8de41612d0 2024-09-13T16:33:47.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.686+0000 7f8ddbfff700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8dc4066760 0x7f8dc4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:47.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.686+0000 7f8ddbfff700 1 -- 172.21.15.119:0/3245756486 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6403+0+0 (secure 0 0 0) 0x7f8ddc0842c0 con 0x7f8de41612d0 2024-09-13T16:33:47.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.686+0000 7f8de2ffd700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8dc4066760 0x7f8dc4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:47.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.686+0000 7f8de2ffd700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8dc4066760 0x7f8dc4068c20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009b50 tx=0x7f8dd40092e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:47.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.694+0000 7f8ddbfff700 1 -- 172.21.15.119:0/3245756486 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8ddc051400 con 0x7f8de41612d0 2024-09-13T16:33:47.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:47.902+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8de4161a60 con 0x7f8de41612d0 2024-09-13T16:33:48.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.554+0000 7f8ddbfff700 1 -- 172.21.15.119:0/3245756486 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v71) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8ddc054a20 con 0x7f8de41612d0 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8dc4066760 msgr2=0x7f8dc4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8dc4066760 0x7f8dc4068c20 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009b50 tx=0x7f8dd40092e0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41612d0 msgr2=0x7f8de40aab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41612d0 0x7f8de40aab00 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc00cc50 tx=0x7f8ddc00cf60 comp rx=0 tx=0).stop 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 shutdown_connections 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8dc4066760 0x7f8dc4068c20 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8de41612d0 0x7f8de40aab00 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8de40b24a0 0x7f8de415c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 --2- 172.21.15.119:0/3245756486 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8de40ade80 0x7f8de40acb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 >> 172.21.15.119:0/3245756486 conn(0x7f8de40b0980 msgr2=0x7f8de406bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:48.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 shutdown_connections 2024-09-13T16:33:48.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.558+0000 7f8de3fff700 1 -- 172.21.15.119:0/3245756486 wait complete. 2024-09-13T16:33:48.579 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:33:48.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136c36700 1 Processor -- start 2024-09-13T16:33:48.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136c36700 1 -- start start 2024-09-13T16:33:48.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136c36700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013c6f0 0x7ff1300b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:48.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136c36700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff13013ed60 0x7ff13013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:48.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136c36700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff13013f6e0 con 0x7ff13013f990 2024-09-13T16:33:48.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136c36700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1300cdce0 con 0x7ff13013ed60 2024-09-13T16:33:48.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136c36700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1300cdfc0 con 0x7ff13013c6f0 2024-09-13T16:33:48.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff136435700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff13013ed60 0x7ff13013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:48.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff135c34700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013c6f0 0x7ff1300b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:48.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff135433700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff13013f990 0x7ff13006eec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:46866/0 (socket says 172.21.15.119:46866) 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff135433700 1 -- 172.21.15.119:0/147689902 learned_addr learned my addr 172.21.15.119:0/147689902 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.702+0000 7ff135c34700 1 --2- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013c6f0 0x7ff1300b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:55904/0 (socket says 172.21.15.119:55904) 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136435700 1 -- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013c6f0 msgr2=0x7ff1300b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136435700 1 --2- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013c6f0 0x7ff1300b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136435700 1 -- 172.21.15.119:0/147689902 >> v1:172.21.15.145:6789/0 conn(0x7ff13013f990 legacy=0x7ff13006eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136435700 1 -- 172.21.15.119:0/147689902 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1300ce2a0 con 0x7ff13013ed60 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff135c34700 1 --2- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013c6f0 0x7ff1300b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:48.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136435700 1 --2- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013ed60 0x7ff13013f1a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff128009d90 tx=0x7ff12800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=582ea049d9819471 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:48.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff134c32700 1 -- 172.21.15.119:0/147689902 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12800e070 con 0x7ff13013ed60 2024-09-13T16:33:48.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff134c32700 1 -- 172.21.15.119:0/147689902 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12800bb20 con 0x7ff13013ed60 2024-09-13T16:33:48.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff134c32700 1 -- 172.21.15.119:0/147689902 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff128012430 con 0x7ff13013ed60 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 -- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013ed60 msgr2=0x7ff13013f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 --2- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013ed60 0x7ff13013f1a0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7ff128009d90 tx=0x7ff12800b330 comp rx=0 tx=0).stop 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 -- 172.21.15.119:0/147689902 shutdown_connections 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 --2- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013ed60 0x7ff13013f1a0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 --2- 172.21.15.119:0/147689902 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013c6f0 0x7ff1300b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 -- 172.21.15.119:0/147689902 >> 172.21.15.119:0/147689902 conn(0x7ff1300b1f40 msgr2=0x7ff13006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 -- 172.21.15.119:0/147689902 shutdown_connections 2024-09-13T16:33:48.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.706+0000 7ff136c36700 1 -- 172.21.15.119:0/147689902 wait complete. 2024-09-13T16:33:48.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 Processor -- start 2024-09-13T16:33:48.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 -- start start 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff13013c6f0 0x7ff13013ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013ed60 0x7ff130137900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013f990 0x7ff1301414a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130137e40 con 0x7ff13013c6f0 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130141fd0 con 0x7ff13013f990 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.710+0000 7ff136c36700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff130141cf0 con 0x7ff13013ed60 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff13013c6f0 0x7ff13013ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:48.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135433700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013ed60 0x7ff130137900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff13013c6f0 0x7ff13013ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35492/0 (socket says 172.21.15.119:35492) 2024-09-13T16:33:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff136435700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013f990 0x7ff1301414a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:48.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 -- 172.21.15.119:0/3118212729 learned_addr learned my addr 172.21.15.119:0/3118212729 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 -- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013ed60 msgr2=0x7ff130137900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013ed60 0x7ff130137900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 -- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013f990 msgr2=0x7ff1301414a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013f990 0x7ff1301414a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 -- 172.21.15.119:0/3118212729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0094b0 con 0x7ff13013c6f0 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff135c34700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff13013c6f0 0x7ff13013ae40 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00ec10 tx=0x7ff12c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff1227fc700 1 -- 172.21.15.119:0/3118212729 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c009770 con 0x7ff13013c6f0 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff1227fc700 1 -- 172.21.15.119:0/3118212729 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff12c00c850 con 0x7ff13013c6f0 2024-09-13T16:33:48.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff128009a70 con 0x7ff13013c6f0 2024-09-13T16:33:48.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff1227fc700 1 -- 172.21.15.119:0/3118212729 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff12c00cbc0 con 0x7ff13013c6f0 2024-09-13T16:33:48.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff130140240 con 0x7ff13013c6f0 2024-09-13T16:33:48.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff130073590 con 0x7ff13013c6f0 2024-09-13T16:33:48.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.714+0000 7ff1227fc700 1 -- 172.21.15.119:0/3118212729 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff12c00cd20 con 0x7ff13013c6f0 2024-09-13T16:33:48.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.718+0000 7ff1227fc700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff114066730 0x7ff114068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:48.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.718+0000 7ff1227fc700 1 -- 172.21.15.119:0/3118212729 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6403+0+0 (secure 0 0 0) 0x7ff12c0842f0 con 0x7ff13013c6f0 2024-09-13T16:33:48.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.718+0000 7ff135433700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff114066730 0x7ff114068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:48.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.718+0000 7ff135433700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff114066730 0x7ff114068bf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff1301597d0 tx=0x7ff124009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:48.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.726+0000 7ff1227fc700 1 -- 172.21.15.119:0/3118212729 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff12c051430 con 0x7ff13013c6f0 2024-09-13T16:33:48.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:48.902+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff13006eb80 con 0x7ff13013c6f0 2024-09-13T16:33:49.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.562+0000 7ff1227fc700 1 -- 172.21.15.119:0/3118212729 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v101) v1 ==== 181+0+0 (secure 0 0 0) 0x7ff12c054a50 con 0x7ff13013c6f0 2024-09-13T16:33:49.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff114066730 msgr2=0x7ff114068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff114066730 0x7ff114068bf0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7ff1301597d0 tx=0x7ff124009040 comp rx=0 tx=0).stop 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff13013c6f0 msgr2=0x7ff13013ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff13013c6f0 0x7ff13013ae40 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7ff12c00ec10 tx=0x7ff12c00ef20 comp rx=0 tx=0).stop 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 shutdown_connections 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff114066730 0x7ff114068bf0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff13013f990 0x7ff1301414a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff13013ed60 0x7ff130137900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 --2- 172.21.15.119:0/3118212729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff13013c6f0 0x7ff13013ae40 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 >> 172.21.15.119:0/3118212729 conn(0x7ff1300b1f40 msgr2=0x7ff1300cec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 shutdown_connections 2024-09-13T16:33:49.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.566+0000 7ff136c36700 1 -- 172.21.15.119:0/3118212729 wait complete. 2024-09-13T16:33:49.571 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:33:49.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:33:49.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.698+0000 7fcbcc4dc700 1 Processor -- start 2024-09-13T16:33:49.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.698+0000 7fcbcc4dc700 1 -- start start 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcbc415a870 0x7fcbc415eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc40ce950 0x7fcbc406ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc40cdbf0 con 0x7fcbc40ced90 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc415f3f0 con 0x7fcbc415a870 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4071490 con 0x7fcbc40ce950 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcaa79700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcbc40ced90 0x7fcbc406a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56724/0 (socket says 172.21.15.119:56724) 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcaa79700 1 -- 172.21.15.119:0/1369970643 learned_addr learned my addr 172.21.15.119:0/1369970643 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbca278700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcbc415a870 0x7fcbc415eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbc9a77700 1 --2- 172.21.15.119:0/1369970643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc40ce950 0x7fcbc406ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbc9276700 1 -- 172.21.15.119:0/1369970643 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3945227483 0 0) 0x7fcbc40cdbf0 con 0x7fcbc40ced90 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbc9276700 1 -- 172.21.15.119:0/1369970643 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbb00035d0 con 0x7fcbc40ced90 2024-09-13T16:33:49.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbca278700 1 -- 172.21.15.119:0/1369970643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc40ce950 msgr2=0x7fcbc406ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:49.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbca278700 1 --2- 172.21.15.119:0/1369970643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc40ce950 0x7fcbc406ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbca278700 1 -- 172.21.15.119:0/1369970643 >> v1:172.21.15.119:6789/0 conn(0x7fcbc40ced90 legacy=0x7fcbc406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:49.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbca278700 1 -- 172.21.15.119:0/1369970643 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc40cde70 con 0x7fcbc415a870 2024-09-13T16:33:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbca278700 1 --2- 172.21.15.119:0/1369970643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc415a870 0x7fcbc415eeb0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009850 tx=0x7fcbc0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ebe0d72c3e5ecd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbc9276700 1 -- 172.21.15.119:0/1369970643 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0011830 con 0x7fcbc415a870 2024-09-13T16:33:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbc9276700 1 -- 172.21.15.119:0/1369970643 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc0011990 con 0x7fcbc415a870 2024-09-13T16:33:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbc9276700 1 -- 172.21.15.119:0/1369970643 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0011d40 con 0x7fcbc415a870 2024-09-13T16:33:49.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/1369970643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc415a870 msgr2=0x7fcbc415eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/1369970643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc415a870 0x7fcbc415eeb0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0009850 tx=0x7fcbc0009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/1369970643 shutdown_connections 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/1369970643 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc40ce950 0x7fcbc406ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/1369970643 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc415a870 0x7fcbc415eeb0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/1369970643 >> 172.21.15.119:0/1369970643 conn(0x7fcbc406b460 msgr2=0x7fcbc406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/1369970643 shutdown_connections 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.702+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/1369970643 wait complete. 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 Processor -- start 2024-09-13T16:33:49.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 -- start start 2024-09-13T16:33:49.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc40ce950 0x7fcbc414f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:49.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbc40ced90 0x7fcbc41f0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc415a870 0x7fcbc4153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc4152530 con 0x7fcbc40ce950 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc414f900 con 0x7fcbc40ced90 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41f1410 con 0x7fcbc415a870 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc40ce950 0x7fcbc414f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc40ce950 0x7fcbc414f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35510/0 (socket says 172.21.15.119:35510) 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 -- 172.21.15.119:0/3271213164 learned_addr learned my addr 172.21.15.119:0/3271213164 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcaa79700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc415a870 0x7fcbc4153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbc9a77700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbc40ced90 0x7fcbc41f0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 -- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc415a870 msgr2=0x7fcbc4153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:49.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc415a870 0x7fcbc4153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 -- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbc40ced90 msgr2=0x7fcbc41f0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbc40ced90 0x7fcbc41f0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 -- 172.21.15.119:0/3271213164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc00094b0 con 0x7fcbc40ce950 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbc9a77700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbc40ced90 0x7fcbc41f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbca278700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc40ce950 0x7fcbc414f3c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0014040 tx=0x7fcbc0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbb6ffd700 1 -- 172.21.15.119:0/3271213164 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc00036c0 con 0x7fcbc40ce950 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc4151490 con 0x7fcbc40ce950 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbb6ffd700 1 -- 172.21.15.119:0/3271213164 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbc001e070 con 0x7fcbc40ce950 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc4151a50 con 0x7fcbc40ce950 2024-09-13T16:33:49.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.706+0000 7fcbb6ffd700 1 -- 172.21.15.119:0/3271213164 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbc0003a40 con 0x7fcbc40ce950 2024-09-13T16:33:49.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.710+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc40cc5c0 con 0x7fcbc40ce950 2024-09-13T16:33:49.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.710+0000 7fcbb6ffd700 1 -- 172.21.15.119:0/3271213164 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcbc0003ba0 con 0x7fcbc40ce950 2024-09-13T16:33:49.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.710+0000 7fcbb6ffd700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcba8066730 0x7fcba8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:49.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.710+0000 7fcbb6ffd700 1 -- 172.21.15.119:0/3271213164 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5965+0+0 (secure 0 0 0) 0x7fcbc0020070 con 0x7fcbc40ce950 2024-09-13T16:33:49.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.710+0000 7fcbc9a77700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcba8066730 0x7fcba8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:49.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.710+0000 7fcbc9a77700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcba8066730 0x7fcba8068bf0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4153f20 tx=0x7fcbb8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:49.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.718+0000 7fcbb6ffd700 1 -- 172.21.15.119:0/3271213164 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcbc00529a0 con 0x7fcbc40ce950 2024-09-13T16:33:49.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:49.886+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fcbc4071890 con 0x7fcbc40ce950 2024-09-13T16:33:50.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.570+0000 7fcbb6ffd700 1 -- 172.21.15.119:0/3271213164 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v102) v1 ==== 169+0+0 (secure 0 0 0) 0x7fcbc0006ce0 con 0x7fcbc40ce950 2024-09-13T16:33:50.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcba8066730 msgr2=0x7fcba8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcba8066730 0x7fcba8068bf0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcbc4153f20 tx=0x7fcbb8008040 comp rx=0 tx=0).stop 2024-09-13T16:33:50.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc40ce950 msgr2=0x7fcbc414f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc40ce950 0x7fcbc414f3c0 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 crypto rx=0x7fcbc0014040 tx=0x7fcbc0004770 comp rx=0 tx=0).stop 2024-09-13T16:33:50.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 shutdown_connections 2024-09-13T16:33:50.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcba8066730 0x7fcba8068bf0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbc415a870 0x7fcbc4153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbc40ced90 0x7fcbc41f0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 --2- 172.21.15.119:0/3271213164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbc40ce950 0x7fcbc414f3c0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 >> 172.21.15.119:0/3271213164 conn(0x7fcbc406b460 msgr2=0x7fcbc40b0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:50.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 shutdown_connections 2024-09-13T16:33:50.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.574+0000 7fcbcc4dc700 1 -- 172.21.15.119:0/3271213164 wait complete. 2024-09-13T16:33:50.578 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:33:50.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T16:33:50.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.698+0000 7f3c351f9700 1 Processor -- start 2024-09-13T16:33:50.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.698+0000 7f3c351f9700 1 -- start start 2024-09-13T16:33:50.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3c300b1860 0x7f3c30159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:50.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c300b1dd0 con 0x7f3c300ad9a0 2024-09-13T16:33:50.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c300b20b0 con 0x7f3c3006ca60 2024-09-13T16:33:50.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c300b2fd0 con 0x7f3c300b1860 2024-09-13T16:33:50.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c349f8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3c300b1860 0x7f3c30159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:50.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c2ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3c300ad9a0 0x7f3c3006c350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56736/0 (socket says 172.21.15.119:56736) 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c349f8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3c300b1860 0x7f3c30159de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35516/0 (socket says 172.21.15.119:35516) 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c2ffff700 1 -- 172.21.15.119:0/604444272 learned_addr learned my addr 172.21.15.119:0/604444272 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c349f8700 1 -- 172.21.15.119:0/604444272 >> v1:172.21.15.145:6789/0 conn(0x7f3c3006ca60 legacy=0x7f3c3015a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c349f8700 1 -- 172.21.15.119:0/604444272 >> v1:172.21.15.119:6789/0 conn(0x7f3c300ad9a0 legacy=0x7f3c3006c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c2f7fe700 1 -- 172.21.15.119:0/604444272 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 597909597 0 0) 0x7f3c300b1dd0 con 0x7f3c300ad9a0 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c349f8700 1 -- 172.21.15.119:0/604444272 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c300b2390 con 0x7f3c300b1860 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c349f8700 1 --2- 172.21.15.119:0/604444272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c30159de0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800bfa0 tx=0x7f3c2800d330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c89afb634b99a5e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c2f7fe700 1 -- 172.21.15.119:0/604444272 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28010070 con 0x7f3c300b1860 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c2f7fe700 1 -- 172.21.15.119:0/604444272 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c2800dab0 con 0x7f3c300b1860 2024-09-13T16:33:50.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c2f7fe700 1 -- 172.21.15.119:0/604444272 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c2800de60 con 0x7f3c300b1860 2024-09-13T16:33:50.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- 172.21.15.119:0/604444272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 msgr2=0x7f3c30159de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 --2- 172.21.15.119:0/604444272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c30159de0 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 crypto rx=0x7f3c2800bfa0 tx=0x7f3c2800d330 comp rx=0 tx=0).stop 2024-09-13T16:33:50.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- 172.21.15.119:0/604444272 shutdown_connections 2024-09-13T16:33:50.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 --2- 172.21.15.119:0/604444272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c30159de0 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- 172.21.15.119:0/604444272 >> 172.21.15.119:0/604444272 conn(0x7f3c300b0980 msgr2=0x7f3c300b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:50.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- 172.21.15.119:0/604444272 shutdown_connections 2024-09-13T16:33:50.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.702+0000 7f3c351f9700 1 -- 172.21.15.119:0/604444272 wait complete. 2024-09-13T16:33:50.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 Processor -- start 2024-09-13T16:33:50.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 -- start start 2024-09-13T16:33:50.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c3006ca60 0x7f3c3015b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:50.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c300ad9a0 0x7f3c301e2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:50.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c300acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c301e4290 con 0x7f3c300b1860 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c301e2fe0 con 0x7f3c3006ca60 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c301e3210 con 0x7f3c300ad9a0 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c300acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c300acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35526/0 (socket says 172.21.15.119:35526) 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 -- 172.21.15.119:0/2443615164 learned_addr learned my addr 172.21.15.119:0/2443615164 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c2ffff700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c3006ca60 0x7f3c3015b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 -- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c300ad9a0 msgr2=0x7f3c301e2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c27fff700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c300ad9a0 0x7f3c301e2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c300ad9a0 0x7f3c301e2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 -- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c3006ca60 msgr2=0x7f3c3015b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c3006ca60 0x7f3c3015b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 -- 172.21.15.119:0/2443615164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2800bc80 con 0x7f3c300b1860 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c2ffff700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c3006ca60 0x7f3c3015b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c27fff700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c300ad9a0 0x7f3c301e2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c349f8700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c300acb30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f3c3006dcb0 tx=0x7f3c28004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c2d7fa700 1 -- 172.21.15.119:0/2443615164 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c280156a0 con 0x7f3c300b1860 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c2d7fa700 1 -- 172.21.15.119:0/2443615164 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c28015800 con 0x7f3c300b1860 2024-09-13T16:33:50.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c301e34f0 con 0x7f3c300b1860 2024-09-13T16:33:50.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c2d7fa700 1 -- 172.21.15.119:0/2443615164 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c28015bb0 con 0x7f3c300b1860 2024-09-13T16:33:50.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c301e3ab0 con 0x7f3c300b1860 2024-09-13T16:33:50.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.706+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c30159de0 con 0x7f3c300b1860 2024-09-13T16:33:50.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.710+0000 7f3c2d7fa700 1 -- 172.21.15.119:0/2443615164 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3c28015d10 con 0x7f3c300b1860 2024-09-13T16:33:50.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.710+0000 7f3c2d7fa700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c10066730 0x7f3c10068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:50.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.710+0000 7f3c2d7fa700 1 -- 172.21.15.119:0/2443615164 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f3c28086a40 con 0x7f3c300b1860 2024-09-13T16:33:50.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.710+0000 7f3c2ffff700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c10066730 0x7f3c10068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:50.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.710+0000 7f3c2ffff700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c10066730 0x7f3c10068bf0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3c20000f60 tx=0x7f3c20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:50.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.718+0000 7f3c2d7fa700 1 -- 172.21.15.119:0/2443615164 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3c28053f50 con 0x7f3c300b1860 2024-09-13T16:33:50.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.882+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3c30158d10 con 0x7f3c300b1860 2024-09-13T16:33:50.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c2d7fa700 1 -- 172.21.15.119:0/2443615164 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3c28057570 con 0x7f3c300b1860 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c10066730 msgr2=0x7f3c10068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c10066730 0x7f3c10068bf0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f3c20000f60 tx=0x7f3c20009040 comp rx=0 tx=0).stop 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 msgr2=0x7f3c300acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c300acb30 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f3c3006dcb0 tx=0x7f3c28004af0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 shutdown_connections 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3c10066730 0x7f3c10068bf0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3c300b1860 0x7f3c300acb30 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3c300ad9a0 0x7f3c301e2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 --2- 172.21.15.119:0/2443615164 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3c3006ca60 0x7f3c3015b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:50.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 >> 172.21.15.119:0/2443615164 conn(0x7f3c300b0980 msgr2=0x7f3c3006fb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:50.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 shutdown_connections 2024-09-13T16:33:50.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:50.890+0000 7f3c351f9700 1 -- 172.21.15.119:0/2443615164 wait complete. 2024-09-13T16:33:50.907 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair' 2024-09-13T16:33:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2223f2d700 1 Processor -- start 2024-09-13T16:33:51.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2223f2d700 1 -- start start 2024-09-13T16:33:51.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2223f2d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f221c0ad910 0x7f221c0b3380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2223f2d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f221c0712e0 0x7f221c1627a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2223f2d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f221c0b1a70 con 0x7f221c06de30 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2223f2d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f221c0b1d50 con 0x7f221c0ad910 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2223f2d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f221c0b2030 con 0x7f221c0712e0 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f22214c8700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f221c06de30 0x7f221c0cbf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56748/0 (socket says 172.21.15.119:56748) 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f22214c8700 1 -- 172.21.15.119:0/3222892159 learned_addr learned my addr 172.21.15.119:0/3222892159 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2221cc9700 1 --2- 172.21.15.119:0/3222892159 >> v2:172.21.15.119:3300/0 conn(0x7f221c0ad910 0x7f221c0b3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f22224ca700 1 --2- 172.21.15.119:0/3222892159 >> v2:172.21.15.145:3300/0 conn(0x7f221c0712e0 0x7f221c1627a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2220cc7700 1 -- 172.21.15.119:0/3222892159 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985250402 0 0) 0x7f221c0b1a70 con 0x7f221c06de30 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2220cc7700 1 -- 172.21.15.119:0/3222892159 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22080035d0 con 0x7f221c06de30 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2221cc9700 1 -- 172.21.15.119:0/3222892159 >> v2:172.21.15.145:3300/0 conn(0x7f221c0712e0 msgr2=0x7f221c1627a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2221cc9700 1 --2- 172.21.15.119:0/3222892159 >> v2:172.21.15.145:3300/0 conn(0x7f221c0712e0 0x7f221c1627a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2221cc9700 1 -- 172.21.15.119:0/3222892159 >> v1:172.21.15.119:6789/0 conn(0x7f221c06de30 legacy=0x7f221c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.014+0000 7f2221cc9700 1 -- 172.21.15.119:0/3222892159 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f221c0b2310 con 0x7f221c0ad910 2024-09-13T16:33:51.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2221cc9700 1 --2- 172.21.15.119:0/3222892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b3380 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f221c0cb7e0 tx=0x7f2210009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49225252340d1351 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2220cc7700 1 -- 172.21.15.119:0/3222892159 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2210011830 con 0x7f221c0ad910 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2220cc7700 1 -- 172.21.15.119:0/3222892159 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2210011990 con 0x7f221c0ad910 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2220cc7700 1 -- 172.21.15.119:0/3222892159 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2210011d20 con 0x7f221c0ad910 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 -- 172.21.15.119:0/3222892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 msgr2=0x7f221c0b3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 --2- 172.21.15.119:0/3222892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b3380 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 crypto rx=0x7f221c0cb7e0 tx=0x7f2210009b30 comp rx=0 tx=0).stop 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 -- 172.21.15.119:0/3222892159 shutdown_connections 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 --2- 172.21.15.119:0/3222892159 >> v2:172.21.15.145:3300/0 conn(0x7f221c0712e0 0x7f221c1627a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 --2- 172.21.15.119:0/3222892159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b3380 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 -- 172.21.15.119:0/3222892159 >> 172.21.15.119:0/3222892159 conn(0x7f221c06a350 msgr2=0x7f221c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:51.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 -- 172.21.15.119:0/3222892159 shutdown_connections 2024-09-13T16:33:51.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 -- 172.21.15.119:0/3222892159 wait complete. 2024-09-13T16:33:51.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 Processor -- start 2024-09-13T16:33:51.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.018+0000 7f2223f2d700 1 -- start start 2024-09-13T16:33:51.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221c06de30 0x7f221c141320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221c0712e0 0x7f221c0ae590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c0afc60 con 0x7f221c0ad910 2024-09-13T16:33:51.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f221c141860 con 0x7f221c06de30 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f221c0af050 con 0x7f221c0712e0 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35546/0 (socket says 172.21.15.119:35546) 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 -- 172.21.15.119:0/4153717195 learned_addr learned my addr 172.21.15.119:0/4153717195 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2221cc9700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221c06de30 0x7f221c141320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22214c8700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221c0712e0 0x7f221c0ae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 -- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221c0712e0 msgr2=0x7f221c0ae590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221c0712e0 0x7f221c0ae590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 -- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221c06de30 msgr2=0x7f221c141320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221c06de30 0x7f221c141320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 -- 172.21.15.119:0/4153717195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22100094b0 con 0x7f221c0ad910 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f22224ca700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b08a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f221400eb60 tx=0x7f221400ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:51.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f220e7fc700 1 -- 172.21.15.119:0/4153717195 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2214009740 con 0x7f221c0ad910 2024-09-13T16:33:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f220e7fc700 1 -- 172.21.15.119:0/4153717195 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221400c850 con 0x7f221c0ad910 2024-09-13T16:33:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f220e7fc700 1 -- 172.21.15.119:0/4153717195 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f221400cbe0 con 0x7f221c0ad910 2024-09-13T16:33:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f221c0af330 con 0x7f221c0ad910 2024-09-13T16:33:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f221c0af8f0 con 0x7f221c0ad910 2024-09-13T16:33:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f221c067760 con 0x7f221c0ad910 2024-09-13T16:33:51.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.022+0000 7f220e7fc700 1 -- 172.21.15.119:0/4153717195 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f221400cd40 con 0x7f221c0ad910 2024-09-13T16:33:51.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.026+0000 7f220e7fc700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2200066760 0x7f2200068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.026+0000 7f220e7fc700 1 -- 172.21.15.119:0/4153717195 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f22140841d0 con 0x7f221c0ad910 2024-09-13T16:33:51.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.026+0000 7f2221cc9700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2200066760 0x7f2200068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.026+0000 7f2221cc9700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2200066760 0x7f2200068c20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2210000c00 tx=0x7f2210008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:51.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.034+0000 7f220e7fc700 1 -- 172.21.15.119:0/4153717195 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f22140515a0 con 0x7f221c0ad910 2024-09-13T16:33:51.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.266+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]} v 0) v1 -- 0x7f221c0cbf80 con 0x7f221c0ad910 2024-09-13T16:33:51.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.570+0000 7f220e7fc700 1 -- 172.21.15.119:0/4153717195 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_health_status_after_dirfrag_repair"]}]=0 v400) v1 ==== 167+0+0 (secure 0 0 0) 0x7f22140170e0 con 0x7f221c0ad910 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2200066760 msgr2=0x7f2200068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2200066760 0x7f2200068c20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2210000c00 tx=0x7f2210008030 comp rx=0 tx=0).stop 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 msgr2=0x7f221c0b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b08a0 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f221400eb60 tx=0x7f221400ef20 comp rx=0 tx=0).stop 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 shutdown_connections 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2200066760 0x7f2200068c20 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f221c0ad910 0x7f221c0b08a0 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f221c0712e0 0x7f221c0ae590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 --2- 172.21.15.119:0/4153717195 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f221c06de30 0x7f221c141320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 >> 172.21.15.119:0/4153717195 conn(0x7f221c06a350 msgr2=0x7f221c0ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:51.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 shutdown_connections 2024-09-13T16:33:51.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.574+0000 7f2223f2d700 1 -- 172.21.15.119:0/4153717195 wait complete. 2024-09-13T16:33:51.594 INFO:tasks.cephfs_test_runner:Test that the damage health status is cleared ... ok 2024-09-13T16:33:51.594 INFO:tasks.cephfs_test_runner:Starting test: test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-09-13T16:33:51.595 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:51.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f629a1bf700 1 Processor -- start 2024-09-13T16:33:51.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f629a1bf700 1 -- start start 2024-09-13T16:33:51.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f629a1bf700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62940d1110 0x7f62940d1550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f629a1bf700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62940c6ab0 con 0x7f629415ccd0 2024-09-13T16:33:51.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f629a1bf700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62940d1a90 con 0x7f62940c6f40 2024-09-13T16:33:51.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f629a1bf700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62940cea10 con 0x7f62940d1110 2024-09-13T16:33:51.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f62989bc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f62940d1110 0x7f62940d1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f62991bd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f629415ccd0 0x7f62940d0720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56756/0 (socket says 172.21.15.119:56756) 2024-09-13T16:33:51.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f62991bd700 1 -- 172.21.15.119:0/2193707101 learned_addr learned my addr 172.21.15.119:0/2193707101 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f62989bc700 1 -- 172.21.15.119:0/2193707101 >> v1:172.21.15.145:6789/0 conn(0x7f62940c6f40 legacy=0x7f62940c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f62989bc700 1 -- 172.21.15.119:0/2193707101 >> v1:172.21.15.119:6789/0 conn(0x7f629415ccd0 legacy=0x7f62940d0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f62989bc700 1 -- 172.21.15.119:0/2193707101 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62940ce170 con 0x7f62940d1110 2024-09-13T16:33:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.706+0000 7f62989bc700 1 --2- 172.21.15.119:0/2193707101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d1110 0x7f62940d1550 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f629000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ae5e8606c83d81 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f6287fff700 1 -- 172.21.15.119:0/2193707101 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f629000e070 con 0x7f62940d1110 2024-09-13T16:33:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f6287fff700 1 -- 172.21.15.119:0/2193707101 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f629000bbd0 con 0x7f62940d1110 2024-09-13T16:33:51.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f6287fff700 1 -- 172.21.15.119:0/2193707101 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6290012430 con 0x7f62940d1110 2024-09-13T16:33:51.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 -- 172.21.15.119:0/2193707101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d1110 msgr2=0x7f62940d1550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 --2- 172.21.15.119:0/2193707101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d1110 0x7f62940d1550 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7f6290009d90 tx=0x7f629000b330 comp rx=0 tx=0).stop 2024-09-13T16:33:51.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 -- 172.21.15.119:0/2193707101 shutdown_connections 2024-09-13T16:33:51.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 --2- 172.21.15.119:0/2193707101 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f62940d1110 0x7f62940d1550 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 -- 172.21.15.119:0/2193707101 >> 172.21.15.119:0/2193707101 conn(0x7f62940b4640 msgr2=0x7f62940b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:51.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 -- 172.21.15.119:0/2193707101 shutdown_connections 2024-09-13T16:33:51.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 -- 172.21.15.119:0/2193707101 wait complete. 2024-09-13T16:33:51.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 Processor -- start 2024-09-13T16:33:51.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.710+0000 7f629a1bf700 1 -- start start 2024-09-13T16:33:51.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62940c6f40 0x7f629415b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940d1110 0x7f62940a0070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f629415ccd0 0x7f62940b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629415bcd0 con 0x7f629415ccd0 2024-09-13T16:33:51.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f629409f2b0 con 0x7f62940c6f40 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f629409f590 con 0x7f62940d1110 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f629415ccd0 0x7f62940b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f629415ccd0 0x7f62940b3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35560/0 (socket says 172.21.15.119:35560) 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 -- 172.21.15.119:0/1258041738 learned_addr learned my addr 172.21.15.119:0/1258041738 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62991bd700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62940c6f40 0x7f629415b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62989bc700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940d1110 0x7f62940a0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 -- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940d1110 msgr2=0x7f62940a0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940d1110 0x7f62940a0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 -- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62940c6f40 msgr2=0x7f629415b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62940c6f40 0x7f629415b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 -- 172.21.15.119:0/1258041738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6290009a70 con 0x7f629415ccd0 2024-09-13T16:33:51.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f62999be700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f629415ccd0 0x7f62940b3960 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f628c00c390 tx=0x7f628c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:51.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f6285ffb700 1 -- 172.21.15.119:0/1258041738 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c0052d0 con 0x7f629415ccd0 2024-09-13T16:33:51.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f6285ffb700 1 -- 172.21.15.119:0/1258041738 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f628c005430 con 0x7f629415ccd0 2024-09-13T16:33:51.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f629409f870 con 0x7f629415ccd0 2024-09-13T16:33:51.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f6285ffb700 1 -- 172.21.15.119:0/1258041738 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f628c0055a0 con 0x7f629415ccd0 2024-09-13T16:33:51.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.714+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62941f35a0 con 0x7f629415ccd0 2024-09-13T16:33:51.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.718+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62941f3730 con 0x7f629415ccd0 2024-09-13T16:33:51.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.718+0000 7f6285ffb700 1 -- 172.21.15.119:0/1258041738 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f628c0038c0 con 0x7f629415ccd0 2024-09-13T16:33:51.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.718+0000 7f6285ffb700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6278066730 0x7f6278068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:51.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.718+0000 7f6285ffb700 1 -- 172.21.15.119:0/1258041738 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f628c0831f0 con 0x7f629415ccd0 2024-09-13T16:33:51.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.718+0000 7f62991bd700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6278066730 0x7f6278068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:51.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.718+0000 7f62991bd700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6278066730 0x7f6278068bf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6288005930 tx=0x7f6288009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:51.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.722+0000 7f6285ffb700 1 -- 172.21.15.119:0/1258041738 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f628c050700 con 0x7f629415ccd0 2024-09-13T16:33:51.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.906+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f629409fd90 con 0x7f629415ccd0 2024-09-13T16:33:51.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f6285ffb700 1 -- 172.21.15.119:0/1258041738 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) v1 ==== 74+0+13519 (secure 0 0 0) 0x7f629409fd90 con 0x7f629415ccd0 2024-09-13T16:33:51.912 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:51.912 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":102,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:50.555931+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/1434266893":"2024-09-14T16:33:47.532902+0000","172.21.15.119:6834/1434266893":"2024-09-14T16:33:47.532902+0000","172.21.15.119:6835/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.119:6834/3937841911":"2024-09-14T16:33:02.472861+0000","172.21.15.145:6835/3011171408":"2024-09-14T16:31:41.722614+0000","172.21.15.145:6834/3011171408":"2024-09-14T16:31:41.722614+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:51.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6278066730 msgr2=0x7f6278068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6278066730 0x7f6278068bf0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f6288005930 tx=0x7f6288009040 comp rx=0 tx=0).stop 2024-09-13T16:33:51.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f629415ccd0 msgr2=0x7f62940b3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f629415ccd0 0x7f62940b3960 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 crypto rx=0x7f628c00c390 tx=0x7f628c00c6a0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 shutdown_connections 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6278066730 0x7f6278068bf0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f629415ccd0 0x7f62940b3960 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f62940d1110 0x7f62940a0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 --2- 172.21.15.119:0/1258041738 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f62940c6f40 0x7f629415b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 >> 172.21.15.119:0/1258041738 conn(0x7f62940b4640 msgr2=0x7f62940c88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 shutdown_connections 2024-09-13T16:33:51.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:51.910+0000 7f629a1bf700 1 -- 172.21.15.119:0/1258041738 wait complete. 2024-09-13T16:33:51.926 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:33:51.926 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:33:52.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.042+0000 7f1c757fa700 1 Processor -- start 2024-09-13T16:33:52.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.042+0000 7f1c757fa700 1 -- start start 2024-09-13T16:33:52.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1c7015aa40 0x7f1c700ce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c700ad4b0 con 0x7f1c7006ba70 2024-09-13T16:33:52.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c700ce9e0 con 0x7f1c70070380 2024-09-13T16:33:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c700cde00 con 0x7f1c7015aa40 2024-09-13T16:33:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1c7015aa40 0x7f1c700ce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c74ff9700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1c7006ba70 0x7f1c700ad940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56762/0 (socket says 172.21.15.119:56762) 2024-09-13T16:33:52.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c74ff9700 1 -- 172.21.15.119:0/1026523465 learned_addr learned my addr 172.21.15.119:0/1026523465 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6ffff700 1 -- 172.21.15.119:0/1026523465 >> v1:172.21.15.145:6789/0 conn(0x7f1c70070380 legacy=0x7f1c7006b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6ffff700 1 -- 172.21.15.119:0/1026523465 >> v1:172.21.15.119:6789/0 conn(0x7f1c7006ba70 legacy=0x7f1c700ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6ffff700 1 -- 172.21.15.119:0/1026523465 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c700cdb20 con 0x7f1c7015aa40 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6f7fe700 1 -- 172.21.15.119:0/1026523465 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722694581 0 0) 0x7f1c700ad4b0 con 0x7f1c7006ba70 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6ffff700 1 --2- 172.21.15.119:0/1026523465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c700ce4a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009850 tx=0x7f1c60009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=875343c9b9c40b19 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6f7fe700 1 -- 172.21.15.119:0/1026523465 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60011830 con 0x7f1c7015aa40 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6f7fe700 1 -- 172.21.15.119:0/1026523465 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c60011990 con 0x7f1c7015aa40 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c6f7fe700 1 -- 172.21.15.119:0/1026523465 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c60011d20 con 0x7f1c7015aa40 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- 172.21.15.119:0/1026523465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 msgr2=0x7f1c700ce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 --2- 172.21.15.119:0/1026523465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c700ce4a0 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7f1c60009850 tx=0x7f1c60009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- 172.21.15.119:0/1026523465 shutdown_connections 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 --2- 172.21.15.119:0/1026523465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c700ce4a0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- 172.21.15.119:0/1026523465 >> 172.21.15.119:0/1026523465 conn(0x7f1c700cb8f0 msgr2=0x7f1c700cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- 172.21.15.119:0/1026523465 shutdown_connections 2024-09-13T16:33:52.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.046+0000 7f1c757fa700 1 -- 172.21.15.119:0/1026523465 wait complete. 2024-09-13T16:33:52.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 Processor -- start 2024-09-13T16:33:52.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 -- start start 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7006ba70 0x7f1c7009a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c70070380 0x7f1c7009c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c7006d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7009a520 con 0x7f1c7015aa40 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7009c8c0 con 0x7f1c7006ba70 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c757fa700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7009c620 con 0x7f1c70070380 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c7006d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c7006d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35592/0 (socket says 172.21.15.119:35592) 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 -- 172.21.15.119:0/2149365436 learned_addr learned my addr 172.21.15.119:0/2149365436 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:52.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c6ffff700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7006ba70 0x7f1c7009a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c677fe700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c70070380 0x7f1c7009c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 -- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c70070380 msgr2=0x7f1c7009c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c70070380 0x7f1c7009c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 -- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7006ba70 msgr2=0x7f1c7009a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7006ba70 0x7f1c7009a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 -- 172.21.15.119:0/2149365436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c600094b0 con 0x7f1c7015aa40 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c6ffff700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7006ba70 0x7f1c7009a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.050+0000 7f1c74ff9700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c7006d0d0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f1c68002230 tx=0x7f1c6800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c6d7fa700 1 -- 172.21.15.119:0/2149365436 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c6800ac20 con 0x7f1c7015aa40 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c6d7fa700 1 -- 172.21.15.119:0/2149365436 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c6800ada0 con 0x7f1c7015aa40 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c70099b80 con 0x7f1c7015aa40 2024-09-13T16:33:52.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c6d7fa700 1 -- 172.21.15.119:0/2149365436 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c68007310 con 0x7f1c7015aa40 2024-09-13T16:33:52.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c7009a140 con 0x7f1c7015aa40 2024-09-13T16:33:52.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c6d7fa700 1 -- 172.21.15.119:0/2149365436 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1c68007470 con 0x7f1c7015aa40 2024-09-13T16:33:52.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c7009a660 con 0x7f1c7015aa40 2024-09-13T16:33:52.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c6d7fa700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c50066730 0x7f1c50068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c6d7fa700 1 -- 172.21.15.119:0/2149365436 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f1c68083f10 con 0x7f1c7015aa40 2024-09-13T16:33:52.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.054+0000 7f1c6ffff700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c50066730 0x7f1c50068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.058+0000 7f1c6ffff700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c50066730 0x7f1c50068bf0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1c600090b0 tx=0x7f1c6001c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:52.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.062+0000 7f1c6d7fa700 1 -- 172.21.15.119:0/2149365436 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1c7009a660 con 0x7f1c7015aa40 2024-09-13T16:33:52.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.225+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1c7009a660 con 0x7f1c7015aa40 2024-09-13T16:33:52.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.225+0000 7f1c6d7fa700 1 -- 172.21.15.119:0/2149365436 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v102) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1c7009a660 con 0x7f1c7015aa40 2024-09-13T16:33:52.230 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:33:52.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c50066730 msgr2=0x7f1c50068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c50066730 0x7f1c50068bf0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f1c600090b0 tx=0x7f1c6001c040 comp rx=0 tx=0).stop 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 msgr2=0x7f1c7006d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c7006d0d0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f1c68002230 tx=0x7f1c6800a060 comp rx=0 tx=0).stop 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 shutdown_connections 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1c50066730 0x7f1c50068bf0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1c7015aa40 0x7f1c7006d0d0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1c70070380 0x7f1c7009c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 --2- 172.21.15.119:0/2149365436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1c7006ba70 0x7f1c7009a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 >> 172.21.15.119:0/2149365436 conn(0x7f1c700cb8f0 msgr2=0x7f1c700afe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 shutdown_connections 2024-09-13T16:33:52.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.229+0000 7f1c757fa700 1 -- 172.21.15.119:0/2149365436 wait complete. 2024-09-13T16:33:52.245 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-09-13T16:33:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe3fff700 1 Processor -- start 2024-09-13T16:33:52.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe3fff700 1 -- start start 2024-09-13T16:33:52.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcbe415cbe0 0x7fcbe4161390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe3fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcbe40d1030 0x7fcbe40d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe3fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe40cdae0 con 0x7fcbe40d3420 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe3fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe406b210 con 0x7fcbe415cbe0 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe3fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe40d19e0 con 0x7fcbe40d1030 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe27fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcbe40d3420 0x7fcbe406ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56772/0 (socket says 172.21.15.119:56772) 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe2ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcbe415cbe0 0x7fcbe4161390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe27fc700 1 -- 172.21.15.119:0/3381909348 learned_addr learned my addr 172.21.15.119:0/3381909348 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:52.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe37fe700 1 --2- 172.21.15.119:0/3381909348 >> v2:172.21.15.145:3300/0 conn(0x7fcbe40d1030 0x7fcbe40d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe2ffd700 1 -- 172.21.15.119:0/3381909348 >> v2:172.21.15.145:3300/0 conn(0x7fcbe40d1030 msgr2=0x7fcbe40d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe2ffd700 1 --2- 172.21.15.119:0/3381909348 >> v2:172.21.15.145:3300/0 conn(0x7fcbe40d1030 0x7fcbe40d14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe2ffd700 1 -- 172.21.15.119:0/3381909348 >> v1:172.21.15.119:6789/0 conn(0x7fcbe40d3420 legacy=0x7fcbe406ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe2ffd700 1 -- 172.21.15.119:0/3381909348 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe40d0760 con 0x7fcbe415cbe0 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe37fe700 1 --2- 172.21.15.119:0/3381909348 >> v2:172.21.15.145:3300/0 conn(0x7fcbe40d1030 0x7fcbe40d14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe2ffd700 1 --2- 172.21.15.119:0/3381909348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe415cbe0 0x7fcbe4161390 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fcbe408bd80 tx=0x7fcbd4009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4821102fc50de56 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe1ffb700 1 -- 172.21.15.119:0/3381909348 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011830 con 0x7fcbe415cbe0 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.349+0000 7fcbe1ffb700 1 -- 172.21.15.119:0/3381909348 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbd4011990 con 0x7fcbe415cbe0 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe1ffb700 1 -- 172.21.15.119:0/3381909348 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4011d20 con 0x7fcbe415cbe0 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 -- 172.21.15.119:0/3381909348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe415cbe0 msgr2=0x7fcbe4161390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/3381909348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe415cbe0 0x7fcbe4161390 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 crypto rx=0x7fcbe408bd80 tx=0x7fcbd4009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:52.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 -- 172.21.15.119:0/3381909348 shutdown_connections 2024-09-13T16:33:52.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/3381909348 >> v2:172.21.15.145:3300/0 conn(0x7fcbe40d1030 0x7fcbe40d14a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/3381909348 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe415cbe0 0x7fcbe4161390 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 -- 172.21.15.119:0/3381909348 >> 172.21.15.119:0/3381909348 conn(0x7fcbe408c080 msgr2=0x7fcbe40d2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:52.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 -- 172.21.15.119:0/3381909348 shutdown_connections 2024-09-13T16:33:52.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.353+0000 7fcbe3fff700 1 -- 172.21.15.119:0/3381909348 wait complete. 2024-09-13T16:33:52.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 Processor -- start 2024-09-13T16:33:52.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 -- start start 2024-09-13T16:33:52.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbe40d1030 0x7fcbe4154480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe40d3420 0x7fcbe40cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbe415cbe0 0x7fcbe40b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40b5920 con 0x7fcbe40d3420 2024-09-13T16:33:52.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40cc8f0 con 0x7fcbe40d1030 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40b5b70 con 0x7fcbe415cbe0 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe40d3420 0x7fcbe40cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe40d3420 0x7fcbe40cbf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35614/0 (socket says 172.21.15.119:35614) 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 -- 172.21.15.119:0/2664377587 learned_addr learned my addr 172.21.15.119:0/2664377587 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe2ffd700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbe40d1030 0x7fcbe4154480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe37fe700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbe415cbe0 0x7fcbe40b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 -- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbe415cbe0 msgr2=0x7fcbe40b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbe415cbe0 0x7fcbe40b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 -- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbe40d1030 msgr2=0x7fcbe4154480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbe40d1030 0x7fcbe4154480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 -- 172.21.15.119:0/2664377587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd40094b0 con 0x7fcbe40d3420 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe27fc700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe40d3420 0x7fcbe40cbf80 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00c500 tx=0x7fcbcc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:52.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbdb7fe700 1 -- 172.21.15.119:0/2664377587 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc005d90 con 0x7fcbe40d3420 2024-09-13T16:33:52.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbdb7fe700 1 -- 172.21.15.119:0/2664377587 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbcc0070d0 con 0x7fcbe40d3420 2024-09-13T16:33:52.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe40b5e50 con 0x7fcbe40d3420 2024-09-13T16:33:52.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbdb7fe700 1 -- 172.21.15.119:0/2664377587 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbcc007460 con 0x7fcbe40d3420 2024-09-13T16:33:52.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.357+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe41549c0 con 0x7fcbe40d3420 2024-09-13T16:33:52.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.361+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe40b6370 con 0x7fcbe40d3420 2024-09-13T16:33:52.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.361+0000 7fcbdb7fe700 1 -- 172.21.15.119:0/2664377587 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcbcc0075c0 con 0x7fcbe40d3420 2024-09-13T16:33:52.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.361+0000 7fcbdb7fe700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcbc4066730 0x7fcbc4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:52.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.361+0000 7fcbdb7fe700 1 -- 172.21.15.119:0/2664377587 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fcbcc083ad0 con 0x7fcbe40d3420 2024-09-13T16:33:52.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.361+0000 7fcbe2ffd700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcbc4066730 0x7fcbc4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:52.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.365+0000 7fcbe2ffd700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcbc4066730 0x7fcbc4068bf0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4000c00 tx=0x7fcbd4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:52.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.369+0000 7fcbdb7fe700 1 -- 172.21.15.119:0/2664377587 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcbcc050f30 con 0x7fcbe40d3420 2024-09-13T16:33:52.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:52.593+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) v1 -- 0x7fcbe40b6370 con 0x7fcbe40d3420 2024-09-13T16:33:53.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.577+0000 7fcbdb7fe700 1 -- 172.21.15.119:0/2664377587 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v402) v1 ==== 149+0+0 (secure 0 0 0) 0x7fcbe40b6370 con 0x7fcbe40d3420 2024-09-13T16:33:53.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcbc4066730 msgr2=0x7fcbc4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcbc4066730 0x7fcbc4068bf0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4000c00 tx=0x7fcbd4008030 comp rx=0 tx=0).stop 2024-09-13T16:33:53.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe40d3420 msgr2=0x7fcbe40cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe40d3420 0x7fcbe40cbf80 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7fcbcc00c500 tx=0x7fcbcc00c810 comp rx=0 tx=0).stop 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 shutdown_connections 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcbc4066730 0x7fcbc4068bf0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcbe415cbe0 0x7fcbe40b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcbe40d3420 0x7fcbe40cbf80 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 --2- 172.21.15.119:0/2664377587 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcbe40d1030 0x7fcbe4154480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 >> 172.21.15.119:0/2664377587 conn(0x7fcbe408c080 msgr2=0x7fcbe406ab00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 shutdown_connections 2024-09-13T16:33:53.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.585+0000 7fcbe3fff700 1 -- 172.21.15.119:0/2664377587 wait complete. 2024-09-13T16:33:53.605 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T16:33:53.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.729+0000 7ff5bd452700 1 Processor -- start 2024-09-13T16:33:53.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bd452700 1 -- start start 2024-09-13T16:33:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bd452700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff5b80d17e0 0x7ff5b8167bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bd452700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80d0df0 con 0x7ff5b815cc00 2024-09-13T16:33:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bd452700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5b80cdb00 con 0x7ff5b80ce8c0 2024-09-13T16:33:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bd452700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5b80cdde0 con 0x7ff5b80d17e0 2024-09-13T16:33:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b7fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff5b815cc00 0x7ff5b80d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59156/0 (socket says 172.21.15.119:59156) 2024-09-13T16:33:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b7fff700 1 -- 172.21.15.119:0/53382980 learned_addr learned my addr 172.21.15.119:0/53382980 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:53.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 --2- 172.21.15.119:0/53382980 >> v2:172.21.15.145:3300/0 conn(0x7ff5b80d17e0 0x7ff5b8167bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924502678 0 0) 0x7ff5b80d0df0 con 0x7ff5b815cc00 2024-09-13T16:33:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5a00035d0 con 0x7ff5b815cc00 2024-09-13T16:33:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408397258 0 0) 0x7ff5b80cdb00 con 0x7ff5b80ce8c0 2024-09-13T16:33:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5b80d0df0 con 0x7ff5b80ce8c0 2024-09-13T16:33:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4087728555 0 0) 0x7ff5a00035d0 con 0x7ff5b815cc00 2024-09-13T16:33:53.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 >> v2:172.21.15.145:3300/0 conn(0x7ff5b80d17e0 msgr2=0x7ff5b8167bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 --2- 172.21.15.119:0/53382980 >> v2:172.21.15.145:3300/0 conn(0x7ff5b80d17e0 0x7ff5b8167bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 >> v1:172.21.15.145:6789/0 conn(0x7ff5b80ce8c0 legacy=0x7ff5b80d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b80ce0c0 con 0x7ff5b815cc00 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7ff5ac001670 con 0x7ff5b815cc00 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 >> v1:172.21.15.119:6789/0 conn(0x7ff5b815cc00 legacy=0x7ff5b80d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5a0003bb0 0x7ff5a0003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0004870 0x7ff5a0004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0005230 0x7ff5a000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b80cdde0 con 0x7ff5a0005230 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a000c0b0 con 0x7ff5a0004870 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a000c270 con 0x7ff5a0003bb0 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0005230 0x7ff5a000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b7fff700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5a0003bb0 0x7ff5a0003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5b77fe700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0004870 0x7ff5a0004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 -- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5a0003bb0 msgr2=0x7ff5a0003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5a0003bb0 0x7ff5a0003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 -- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0004870 msgr2=0x7ff5a0004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0004870 0x7ff5a0004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 -- 172.21.15.119:0/53382980 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a00035d0 con 0x7ff5a0005230 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 -- 172.21.15.119:0/53382980 reap_dead start 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.733+0000 7ff5bcc51700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0005230 0x7ff5a000ba30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009fc0 tx=0x7ff5a00058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5b9079a1663be4a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:53.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0003a40 con 0x7ff5a0005230 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5b000f040 con 0x7ff5a0005230 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5b6ffd700 1 -- 172.21.15.119:0/53382980 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5b0003e00 con 0x7ff5a0005230 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5bd452700 1 -- 172.21.15.119:0/53382980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0005230 msgr2=0x7ff5a000ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5bd452700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0005230 0x7ff5a000ba30 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 crypto rx=0x7ff5b0009fc0 tx=0x7ff5a00058a0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5bd452700 1 -- 172.21.15.119:0/53382980 shutdown_connections 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5bd452700 1 --2- 172.21.15.119:0/53382980 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5a0005230 0x7ff5a000ba30 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5bd452700 1 -- 172.21.15.119:0/53382980 >> 172.21.15.119:0/53382980 conn(0x7ff5b808c080 msgr2=0x7ff5b80d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5bd452700 1 -- 172.21.15.119:0/53382980 shutdown_connections 2024-09-13T16:33:53.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.737+0000 7ff5bd452700 1 -- 172.21.15.119:0/53382980 wait complete. 2024-09-13T16:33:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 Processor -- start 2024-09-13T16:33:53.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 -- start start 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0005230 0x7ff5b809c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5b81f6dc0 0x7ff5b80cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5b80c7dd0 0x7ff5b814bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b809cdb0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b81595e0 con 0x7ff5a0005230 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5b81f6300 con 0x7ff5b80c7dd0 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5b81f6dc0 0x7ff5b80cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5b81f6dc0 0x7ff5b80cbab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34286/0 (socket says 172.21.15.119:34286) 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 -- 172.21.15.119:0/577671131 learned_addr learned my addr 172.21.15.119:0/577671131 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:53.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0005230 0x7ff5b809c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bcc51700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5b80c7dd0 0x7ff5b814bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 -- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5b80c7dd0 msgr2=0x7ff5b814bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5b80c7dd0 0x7ff5b814bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 -- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0005230 msgr2=0x7ff5b809c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0005230 0x7ff5b809c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 -- 172.21.15.119:0/577671131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5b80c8340 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b77fe700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5b81f6dc0 0x7ff5b80cbab0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7ff5a800f7a0 tx=0x7ff5a800fab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b4ff9700 1 -- 172.21.15.119:0/577671131 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8005320 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b4ff9700 1 -- 172.21.15.119:0/577671131 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a8005480 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5b81f65c0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5b4ff9700 1 -- 172.21.15.119:0/577671131 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a8006ac0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.741+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5b80cbff0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.745+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5b81f6ae0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.745+0000 7ff5b4ff9700 1 -- 172.21.15.119:0/577671131 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff5a8006c20 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.745+0000 7ff5b4ff9700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5a4066760 0x7ff5a4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.745+0000 7ff5b4ff9700 1 -- 172.21.15.119:0/577671131 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5435+0+0 (secure 0 0 0) 0x7ff5a8083670 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.750 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.745+0000 7ff5b7fff700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5a4066760 0x7ff5a4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:53.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.749+0000 7ff5b7fff700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5a4066760 0x7ff5a4068c20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac001c40 tx=0x7ff5ac001e20 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:53.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.753+0000 7ff5b4ff9700 1 -- 172.21.15.119:0/577671131 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff5b81f6ae0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.929 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.925+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff5b81f6ae0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.933+0000 7ff5b4ff9700 1 -- 172.21.15.119:0/577671131 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff5b81f6ae0 con 0x7ff5b81f6dc0 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5a4066760 msgr2=0x7ff5a4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5a4066760 0x7ff5a4068c20 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7ff5ac001c40 tx=0x7ff5ac001e20 comp rx=0 tx=0).stop 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5b81f6dc0 msgr2=0x7ff5b80cbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5b81f6dc0 0x7ff5b80cbab0 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7ff5a800f7a0 tx=0x7ff5a800fab0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 shutdown_connections 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5a4066760 0x7ff5a4068c20 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff5b80c7dd0 0x7ff5b814bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff5b81f6dc0 0x7ff5b80cbab0 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 --2- 172.21.15.119:0/577671131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff5a0005230 0x7ff5b809c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:53.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 >> 172.21.15.119:0/577671131 conn(0x7ff5b808c080 msgr2=0x7ff5b80660d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:53.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 shutdown_connections 2024-09-13T16:33:53.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:53.937+0000 7ff5bd452700 1 -- 172.21.15.119:0/577671131 wait complete. 2024-09-13T16:33:53.958 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:33:53.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:33:53.975 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:33:53.997 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:33:53.997 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:33:53.997 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:54.024 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:33:54.025 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:33:54.025 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:54.046 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:33:54.046 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:33:54.046 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:54.052 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:33:54.052 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:33:54.053 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:33:54.060 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:33:54.060 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T16:33:54.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:33:54.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4ad70700 1 Processor -- start 2024-09-13T16:33:54.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4ad70700 1 -- start start 2024-09-13T16:33:54.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4ad70700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fad440717b0 0x7fad44160f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4ad70700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fad4406a9a0 0x7fad4415d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4ad70700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad440b4880 con 0x7fad4406a5c0 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4ad70700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad440b3c70 con 0x7fad4406a9a0 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4ad70700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad440b3f50 con 0x7fad440717b0 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fad4406a9a0 0x7fad4415d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fad4406a9a0 0x7fad4415d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34290/0 (socket says 172.21.15.119:34290) 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 -- 172.21.15.119:0/851177607 learned_addr learned my addr 172.21.15.119:0/851177607 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:54.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4956d700 1 --2- 172.21.15.119:0/851177607 >> v2:172.21.15.145:3300/0 conn(0x7fad440717b0 0x7fad44160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 -- 172.21.15.119:0/851177607 >> v2:172.21.15.145:3300/0 conn(0x7fad440717b0 msgr2=0x7fad44160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 --2- 172.21.15.119:0/851177607 >> v2:172.21.15.145:3300/0 conn(0x7fad440717b0 0x7fad44160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 -- 172.21.15.119:0/851177607 >> v1:172.21.15.145:6789/0 conn(0x7fad4406a5c0 legacy=0x7fad4406cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 -- 172.21.15.119:0/851177607 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad440b4230 con 0x7fad4406a9a0 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4956d700 1 --2- 172.21.15.119:0/851177607 >> v2:172.21.15.145:3300/0 conn(0x7fad440717b0 0x7fad44160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.177+0000 7fad4a56f700 1 --2- 172.21.15.119:0/851177607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a9a0 0x7fad4415d7b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fad440cf8e0 tx=0x7fad3c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bc618aec5e6c8db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad48d6c700 1 -- 172.21.15.119:0/851177607 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c011830 con 0x7fad4406a9a0 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad48d6c700 1 -- 172.21.15.119:0/851177607 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad3c011990 con 0x7fad4406a9a0 2024-09-13T16:33:54.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad48d6c700 1 -- 172.21.15.119:0/851177607 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad3c011d20 con 0x7fad4406a9a0 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- 172.21.15.119:0/851177607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a9a0 msgr2=0x7fad4415d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 --2- 172.21.15.119:0/851177607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a9a0 0x7fad4415d7b0 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7fad440cf8e0 tx=0x7fad3c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- 172.21.15.119:0/851177607 shutdown_connections 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 --2- 172.21.15.119:0/851177607 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a9a0 0x7fad4415d7b0 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 --2- 172.21.15.119:0/851177607 >> v2:172.21.15.145:3300/0 conn(0x7fad440717b0 0x7fad44160f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- 172.21.15.119:0/851177607 >> 172.21.15.119:0/851177607 conn(0x7fad44070cc0 msgr2=0x7fad440710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- 172.21.15.119:0/851177607 shutdown_connections 2024-09-13T16:33:54.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- 172.21.15.119:0/851177607 wait complete. 2024-09-13T16:33:54.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 Processor -- start 2024-09-13T16:33:54.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- start start 2024-09-13T16:33:54.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a5c0 0x7fad440ccf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4406a9a0 0x7fad4414d6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad440717b0 0x7fad440cee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad4414dbf0 con 0x7fad4406a5c0 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad440cd500 con 0x7fad4406a9a0 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4ad70700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad440cd7e0 con 0x7fad440717b0 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad49d6e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a5c0 0x7fad440ccf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad49d6e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a5c0 0x7fad440ccf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34304/0 (socket says 172.21.15.119:34304) 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad49d6e700 1 -- 172.21.15.119:0/390808688 learned_addr learned my addr 172.21.15.119:0/390808688 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4a56f700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad440717b0 0x7fad440cee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.181+0000 7fad4956d700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4406a9a0 0x7fad4414d6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad49d6e700 1 -- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad440717b0 msgr2=0x7fad440cee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad49d6e700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad440717b0 0x7fad440cee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad49d6e700 1 -- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4406a9a0 msgr2=0x7fad4414d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad49d6e700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4406a9a0 0x7fad4414d6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad49d6e700 1 -- 172.21.15.119:0/390808688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad38009a70 con 0x7fad4406a5c0 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad49d6e700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a5c0 0x7fad440ccf20 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c500 tx=0x7fad4000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad367fc700 1 -- 172.21.15.119:0/390808688 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad400052d0 con 0x7fad4406a5c0 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad367fc700 1 -- 172.21.15.119:0/390808688 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad40005430 con 0x7fad4406a5c0 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad3c009570 con 0x7fad4406a5c0 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad367fc700 1 -- 172.21.15.119:0/390808688 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad400055a0 con 0x7fad4406a5c0 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad4415de70 con 0x7fad4406a5c0 2024-09-13T16:33:54.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad44160f60 con 0x7fad4406a5c0 2024-09-13T16:33:54.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad367fc700 1 -- 172.21.15.119:0/390808688 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fad400038e0 con 0x7fad4406a5c0 2024-09-13T16:33:54.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad367fc700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad28066730 0x7fad28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad367fc700 1 -- 172.21.15.119:0/390808688 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fad40083580 con 0x7fad4406a5c0 2024-09-13T16:33:54.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.185+0000 7fad4956d700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad28066730 0x7fad28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.189+0000 7fad4956d700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad28066730 0x7fad28068bf0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fad4415c1a0 tx=0x7fad3800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:54.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.197+0000 7fad367fc700 1 -- 172.21.15.119:0/390808688 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fad40050a00 con 0x7fad4406a5c0 2024-09-13T16:33:54.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.405+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad440cf630 con 0x7fad4406a5c0 2024-09-13T16:33:54.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.405+0000 7fad367fc700 1 -- 172.21.15.119:0/390808688 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+3538 (secure 0 0 0) 0x7fad40017df0 con 0x7fad4406a5c0 2024-09-13T16:33:54.409 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:54.409 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":71,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6947,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/1767850579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":1767850579},{"type":"v1","addr":"172.21.15.145:6835","nonce":1767850579}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":58},{"gid":7510,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/2871142846","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":2871142846},{"type":"v1","addr":"172.21.15.119:6833","nonce":2871142846}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":7522,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/764668337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":764668337},{"type":"v1","addr":"172.21.15.145:6837","nonce":764668337}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":52},{"gid":8748,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/1298701506","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":1298701506},{"type":"v1","addr":"172.21.15.119:6835","nonce":1298701506}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":71}],"filesystems":[]} 2024-09-13T16:33:54.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad28066730 msgr2=0x7fad28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad28066730 0x7fad28068bf0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fad4415c1a0 tx=0x7fad3800b040 comp rx=0 tx=0).stop 2024-09-13T16:33:54.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a5c0 msgr2=0x7fad440ccf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a5c0 0x7fad440ccf20 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 crypto rx=0x7fad4000c500 tx=0x7fad4000c8c0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 shutdown_connections 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad28066730 0x7fad28068bf0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad440717b0 0x7fad440cee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad4406a9a0 0x7fad4414d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 --2- 172.21.15.119:0/390808688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad4406a5c0 0x7fad440ccf20 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 >> 172.21.15.119:0/390808688 conn(0x7fad44070cc0 msgr2=0x7fad4415d3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 shutdown_connections 2024-09-13T16:33:54.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.409+0000 7fad4ad70700 1 -- 172.21.15.119:0/390808688 wait complete. 2024-09-13T16:33:54.413 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 71 2024-09-13T16:33:54.428 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:33:54.429 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T16:33:54.429 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:33:54.429 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:33:54.429 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:33:54.429 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T16:33:54.429 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:33:54.429 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:33:54.430 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:33:54.430 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T16:33:54.430 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:33:54.430 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:33:54.431 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:33:54.431 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T16:33:54.431 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:33:54.431 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:33:54.439 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:33:54.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:33:54.442 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:33:54.442 DEBUG:teuthology.parallel:result is None 2024-09-13T16:33:54.444 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:33:54.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:33:54.446 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:33:54.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:33:54.447 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:33:54.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:33:54.448 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:33:54.448 DEBUG:teuthology.parallel:result is None 2024-09-13T16:33:54.497 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:33:54.498 DEBUG:teuthology.parallel:result is None 2024-09-13T16:33:54.499 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:33:54.499 DEBUG:teuthology.parallel:result is None 2024-09-13T16:33:54.499 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T16:33:54.510 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:33:54.529 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:33:54.596 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:33:54.596 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:33:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.605+0000 7f99f24b4700 1 Processor -- start 2024-09-13T16:33:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.605+0000 7f99f24b4700 1 -- start start 2024-09-13T16:33:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.605+0000 7f99f24b4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec06d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.605+0000 7f99f24b4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f99ec06da70 0x7f99ec0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f24b4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec0b2280 con 0x7f99ec0b3630 2024-09-13T16:33:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f24b4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99ec070c10 con 0x7f99ec06da70 2024-09-13T16:33:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f24b4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec06deb0 con 0x7f99ec1594f0 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f1cb3700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f99ec0b3630 0x7f99ec15d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59170/0 (socket says 172.21.15.119:59170) 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f1cb3700 1 -- 172.21.15.119:0/2125454294 learned_addr learned my addr 172.21.15.119:0/2125454294 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f0cb1700 1 --2- 172.21.15.119:0/2125454294 >> v2:172.21.15.119:3300/0 conn(0x7f99ec06da70 0x7f99ec0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f14b2700 1 --2- 172.21.15.119:0/2125454294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f0cb1700 1 -- 172.21.15.119:0/2125454294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 msgr2=0x7f99ec06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f0cb1700 1 --2- 172.21.15.119:0/2125454294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec06d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f0cb1700 1 -- 172.21.15.119:0/2125454294 >> v1:172.21.15.119:6789/0 conn(0x7f99ec0b3630 legacy=0x7f99ec15d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f0cb1700 1 -- 172.21.15.119:0/2125454294 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec0cc890 con 0x7f99ec06da70 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f14b2700 1 --2- 172.21.15.119:0/2125454294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.609+0000 7f99f0cb1700 1 --2- 172.21.15.119:0/2125454294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec06da70 0x7f99ec0b30f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f99ec070d50 tx=0x7f99e0009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c759ed22a45a9072 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:54.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99ebfff700 1 -- 172.21.15.119:0/2125454294 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0004020 con 0x7f99ec06da70 2024-09-13T16:33:54.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99ebfff700 1 -- 172.21.15.119:0/2125454294 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e0004180 con 0x7f99ec06da70 2024-09-13T16:33:54.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99ebfff700 1 -- 172.21.15.119:0/2125454294 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0004530 con 0x7f99ec06da70 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 -- 172.21.15.119:0/2125454294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec06da70 msgr2=0x7f99ec0b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 --2- 172.21.15.119:0/2125454294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec06da70 0x7f99ec0b30f0 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 crypto rx=0x7f99ec070d50 tx=0x7f99e0009b30 comp rx=0 tx=0).stop 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 -- 172.21.15.119:0/2125454294 shutdown_connections 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 --2- 172.21.15.119:0/2125454294 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec06da70 0x7f99ec0b30f0 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 --2- 172.21.15.119:0/2125454294 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec06d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 -- 172.21.15.119:0/2125454294 >> 172.21.15.119:0/2125454294 conn(0x7f99ec06a350 msgr2=0x7f99ec06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 -- 172.21.15.119:0/2125454294 shutdown_connections 2024-09-13T16:33:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.613+0000 7f99f24b4700 1 -- 172.21.15.119:0/2125454294 wait complete. 2024-09-13T16:33:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 Processor -- start 2024-09-13T16:33:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 -- start start 2024-09-13T16:33:54.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ec06da70 0x7f99ec0acd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec0b3630 0x7f99ec1f0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0ad2d0 con 0x7f99ec0b3630 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec1f1750 con 0x7f99ec06da70 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f24b4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec1f1520 con 0x7f99ec1594f0 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec0b3630 0x7f99ec1f0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec0b3630 0x7f99ec1f0fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34364/0 (socket says 172.21.15.119:34364) 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 -- 172.21.15.119:0/155979596 learned_addr learned my addr 172.21.15.119:0/155979596 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f1cb3700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 -- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 msgr2=0x7f99ec151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 -- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ec06da70 msgr2=0x7f99ec0acd90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ec06da70 0x7f99ec0acd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 -- 172.21.15.119:0/155979596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99dc009a70 con 0x7f99ec0b3630 2024-09-13T16:33:54.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.617+0000 7f99f0cb1700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec0b3630 0x7f99ec1f0fe0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f99e00090b0 tx=0x7f99e0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.621+0000 7f99e9ffb700 1 -- 172.21.15.119:0/155979596 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0011da0 con 0x7f99ec0b3630 2024-09-13T16:33:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.621+0000 7f99e9ffb700 1 -- 172.21.15.119:0/155979596 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e00036c0 con 0x7f99ec0b3630 2024-09-13T16:33:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.621+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99e00094b0 con 0x7f99ec0b3630 2024-09-13T16:33:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.621+0000 7f99e9ffb700 1 -- 172.21.15.119:0/155979596 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e0003a70 con 0x7f99ec0b3630 2024-09-13T16:33:54.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.621+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec1f1cd0 con 0x7f99ec0b3630 2024-09-13T16:33:54.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.621+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ec0b2710 con 0x7f99ec0b3630 2024-09-13T16:33:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.625+0000 7f99e9ffb700 1 -- 172.21.15.119:0/155979596 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f99e0003bd0 con 0x7f99ec0b3630 2024-09-13T16:33:54.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.625+0000 7f99e9ffb700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99d8066760 0x7f99d8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:54.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.625+0000 7f99e9ffb700 1 -- 172.21.15.119:0/155979596 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f99e0084fc0 con 0x7f99ec0b3630 2024-09-13T16:33:54.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.625+0000 7f99f14b2700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99d8066760 0x7f99d8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:54.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.625+0000 7f99f14b2700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99d8066760 0x7f99d8068c20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f99ec06d4c0 tx=0x7f99dc00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:54.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.629+0000 7f99e9ffb700 1 -- 172.21.15.119:0/155979596 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f99e00b6db0 con 0x7f99ec0b3630 2024-09-13T16:33:54.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:54.809+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f99ec1598d0 con 0x7f99ec0b3630 2024-09-13T16:33:55.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.625+0000 7f99e9ffb700 1 -- 172.21.15.119:0/155979596 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f99e002a080 con 0x7f99ec0b3630 2024-09-13T16:33:55.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99d8066760 msgr2=0x7f99d8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:55.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99d8066760 0x7f99d8068c20 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f99ec06d4c0 tx=0x7f99dc00b040 comp rx=0 tx=0).stop 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec0b3630 msgr2=0x7f99ec1f0fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec0b3630 0x7f99ec1f0fe0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f99e00090b0 tx=0x7f99e0004770 comp rx=0 tx=0).stop 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 shutdown_connections 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f99d8066760 0x7f99d8068c20 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f99ec1594f0 0x7f99ec151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f99ec0b3630 0x7f99ec1f0fe0 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 --2- 172.21.15.119:0/155979596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f99ec06da70 0x7f99ec0acd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 >> 172.21.15.119:0/155979596 conn(0x7f99ec06a350 msgr2=0x7f99ec06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 shutdown_connections 2024-09-13T16:33:55.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.629+0000 7f99f24b4700 1 -- 172.21.15.119:0/155979596 wait complete. 2024-09-13T16:33:55.636 INFO:teuthology.orchestra.run.smithi119.stderr: removed all blocklist entries 2024-09-13T16:33:55.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T16:33:55.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 Processor -- start 2024-09-13T16:33:55.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 -- start start 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6ec80adb30 0x7f6ec80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec806bb90 0x7f6ec80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6ec80cde70 0x7f6ec815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec806cf60 con 0x7f6ec80cde70 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec806c350 con 0x7f6ec80adb30 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ece45e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec806c630 con 0x7f6ec806bb90 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ecdc5d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6ec80cde70 0x7f6ec815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ecdc5d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6ec80cde70 0x7f6ec815e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34372/0 (socket says 172.21.15.119:34372) 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.769+0000 7f6ecdc5d700 1 -- 172.21.15.119:0/2386392043 learned_addr learned my addr 172.21.15.119:0/2386392043 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6eccc5b700 1 --2- 172.21.15.119:0/2386392043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec806bb90 0x7f6ec80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:55.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ecd45c700 1 --2- 172.21.15.119:0/2386392043 >> v2:172.21.15.145:3300/0 conn(0x7f6ec80adb30 0x7f6ec80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:55.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ecdc5d700 1 -- 172.21.15.119:0/2386392043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec806bb90 msgr2=0x7f6ec80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:55.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ecdc5d700 1 --2- 172.21.15.119:0/2386392043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec806bb90 0x7f6ec80cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ecdc5d700 1 -- 172.21.15.119:0/2386392043 >> v2:172.21.15.145:3300/0 conn(0x7f6ec80adb30 msgr2=0x7f6ec80cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ecdc5d700 1 --2- 172.21.15.119:0/2386392043 >> v2:172.21.15.145:3300/0 conn(0x7f6ec80adb30 0x7f6ec80cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ecdc5d700 1 -- 172.21.15.119:0/2386392043 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec806c910 con 0x7f6ec80cde70 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6eccc5b700 1 --2- 172.21.15.119:0/2386392043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec806bb90 0x7f6ec80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ecdc5d700 1 --2- 172.21.15.119:0/2386392043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80cde70 0x7f6ec815e2f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8071fa0 tx=0x7f6ec0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75422fa1c3836b5a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ebffff700 1 -- 172.21.15.119:0/2386392043 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0011830 con 0x7f6ec80cde70 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ebffff700 1 -- 172.21.15.119:0/2386392043 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec0011990 con 0x7f6ec80cde70 2024-09-13T16:33:55.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ebffff700 1 -- 172.21.15.119:0/2386392043 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec0011d20 con 0x7f6ec80cde70 2024-09-13T16:33:55.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 -- 172.21.15.119:0/2386392043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80cde70 msgr2=0x7f6ec815e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:55.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2386392043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80cde70 0x7f6ec815e2f0 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f6ec8071fa0 tx=0x7f6ec0009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 -- 172.21.15.119:0/2386392043 shutdown_connections 2024-09-13T16:33:55.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2386392043 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80cde70 0x7f6ec815e2f0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2386392043 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec806bb90 0x7f6ec80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2386392043 >> v2:172.21.15.145:3300/0 conn(0x7f6ec80adb30 0x7f6ec80cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 -- 172.21.15.119:0/2386392043 >> 172.21.15.119:0/2386392043 conn(0x7f6ec806a350 msgr2=0x7f6ec806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:55.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 -- 172.21.15.119:0/2386392043 shutdown_connections 2024-09-13T16:33:55.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.773+0000 7f6ece45e700 1 -- 172.21.15.119:0/2386392043 wait complete. 2024-09-13T16:33:55.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 Processor -- start 2024-09-13T16:33:55.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 -- start start 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ec806bb90 0x7f6ec80c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80adb30 0x7f6ec8132920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec80cde70 0x7f6ec80afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80af3b0 con 0x7f6ec80adb30 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8132e60 con 0x7f6ec806bb90 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8133100 con 0x7f6ec80cde70 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80adb30 0x7f6ec8132920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80adb30 0x7f6ec8132920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34378/0 (socket says 172.21.15.119:34378) 2024-09-13T16:33:55.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 -- 172.21.15.119:0/2197880673 learned_addr learned my addr 172.21.15.119:0/2197880673 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ecd45c700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ec806bb90 0x7f6ec80c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ecdc5d700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec80cde70 0x7f6ec80afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 -- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec80cde70 msgr2=0x7f6ec80afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec80cde70 0x7f6ec80afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 -- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ec806bb90 msgr2=0x7f6ec80c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:55.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ec806bb90 0x7f6ec80c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:55.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 -- 172.21.15.119:0/2197880673 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec4009a70 con 0x7f6ec80adb30 2024-09-13T16:33:55.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6eccc5b700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80adb30 0x7f6ec8132920 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400efc0 tx=0x7f6ec400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ebdffb700 1 -- 172.21.15.119:0/2197880673 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec400cdd0 con 0x7f6ec80adb30 2024-09-13T16:33:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ebdffb700 1 -- 172.21.15.119:0/2197880673 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec400f040 con 0x7f6ec80adb30 2024-09-13T16:33:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec0009570 con 0x7f6ec80adb30 2024-09-13T16:33:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ebdffb700 1 -- 172.21.15.119:0/2197880673 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ec40146a0 con 0x7f6ec80adb30 2024-09-13T16:33:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.777+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec81336c0 con 0x7f6ec80adb30 2024-09-13T16:33:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.781+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec80adf10 con 0x7f6ec80adb30 2024-09-13T16:33:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.781+0000 7f6ebdffb700 1 -- 172.21.15.119:0/2197880673 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6ec4014800 con 0x7f6ec80adb30 2024-09-13T16:33:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.781+0000 7f6ebdffb700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6eac066730 0x7f6eac068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.781+0000 7f6ebdffb700 1 -- 172.21.15.119:0/2197880673 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6ec4084c20 con 0x7f6ec80adb30 2024-09-13T16:33:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.781+0000 7f6ecd45c700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6eac066730 0x7f6eac068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.781+0000 7f6ecd45c700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6eac066730 0x7f6eac068bf0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009b70 tx=0x7f6eb8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:55.789+0000 7f6ebdffb700 1 -- 172.21.15.119:0/2197880673 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6ec4052190 con 0x7f6ec80adb30 2024-09-13T16:33:56.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.013+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6ec80ce2d0 con 0x7f6ec80adb30 2024-09-13T16:33:56.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.013+0000 7f6ebdffb700 1 -- 172.21.15.119:0/2197880673 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6ec40557b0 con 0x7f6ec80adb30 2024-09-13T16:33:56.018 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:56.018 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_dump": [ 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.a", 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA==", 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.019 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.b", 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g==", 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.c", 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ==", 2024-09-13T16:33:56.020 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.d", 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw==", 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:33:56.021 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.0", 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA==", 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.022 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.1", 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw==", 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.2", 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ==", 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.023 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.3", 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww==", 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.024 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.4", 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA==", 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.5", 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q==", 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.025 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.6", 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg==", 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.026 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.7", 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ==", 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.0", 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ==", 2024-09-13T16:33:56.027 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow r", 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.1", 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ==", 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:33:56.028 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.2", 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw==", 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.029 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.3", 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow==", 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.admin", 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmpEcAJRAAYorOK9AU5Q/2uOTIUrZhJw==", 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.030 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.x", 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ==", 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.031 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.y", 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw==", 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:33:56.032 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6eac066730 msgr2=0x7f6eac068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6eac066730 0x7f6eac068bf0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8009b70 tx=0x7f6eb8009240 comp rx=0 tx=0).stop 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80adb30 msgr2=0x7f6ec8132920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80adb30 0x7f6ec8132920 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 crypto rx=0x7f6ec400efc0 tx=0x7f6ec400c3a0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 shutdown_connections 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6eac066730 0x7f6eac068bf0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6ec80cde70 0x7f6ec80afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6ec80adb30 0x7f6ec8132920 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 --2- 172.21.15.119:0/2197880673 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6ec806bb90 0x7f6ec80c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 >> 172.21.15.119:0/2197880673 conn(0x7f6ec806a350 msgr2=0x7f6ec815ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:56.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 shutdown_connections 2024-09-13T16:33:56.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.017+0000 7f6ece45e700 1 -- 172.21.15.119:0/2197880673 wait complete. 2024-09-13T16:33:56.035 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:56.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e63244700 1 Processor -- start 2024-09-13T16:33:56.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e63244700 1 -- start start 2024-09-13T16:33:56.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e63244700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c06dc40 0x7f8e5c0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e63244700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0b2280 con 0x7f8e5c1594f0 2024-09-13T16:33:56.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e63244700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e5c0cc890 con 0x7f8e5c0b3630 2024-09-13T16:33:56.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e63244700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c0b3ac0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e62242700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8e5c1594f0 0x7f8e5c06d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59186/0 (socket says 172.21.15.119:59186) 2024-09-13T16:33:56.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e62242700 1 -- 172.21.15.119:0/2515662646 learned_addr learned my addr 172.21.15.119:0/2515662646 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:56.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61a41700 1 --2- 172.21.15.119:0/2515662646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c06dc40 0x7f8e5c0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744640399 0 0) 0x7f8e5c0b2280 con 0x7f8e5c1594f0 2024-09-13T16:33:56.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e480057e0 con 0x7f8e5c1594f0 2024-09-13T16:33:56.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2508958843 0 0) 0x7f8e5c0cc890 con 0x7f8e5c0b3630 2024-09-13T16:33:56.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e5c0b2280 con 0x7f8e5c0b3630 2024-09-13T16:33:56.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61a41700 1 -- 172.21.15.119:0/2515662646 >> v1:172.21.15.145:6789/0 conn(0x7f8e5c0b3630 legacy=0x7f8e5c06fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61a41700 1 -- 172.21.15.119:0/2515662646 >> v1:172.21.15.119:6789/0 conn(0x7f8e5c1594f0 legacy=0x7f8e5c06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 917698174 0 0) 0x7f8e480057e0 con 0x7f8e5c1594f0 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.145+0000 7f8e61a41700 1 -- 172.21.15.119:0/2515662646 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e5c0b1970 con 0x7f8e5c06dc40 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e61a41700 1 --2- 172.21.15.119:0/2515662646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c06dc40 0x7f8e5c0b30f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0b1b60 tx=0x7f8e4c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e5ad7ad02a1a8dc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58001670 con 0x7f8e5c06dc40 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e4c004030 con 0x7f8e5c06dc40 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e61240700 1 -- 172.21.15.119:0/2515662646 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e4c0043c0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 -- 172.21.15.119:0/2515662646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c06dc40 msgr2=0x7f8e5c0b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 --2- 172.21.15.119:0/2515662646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c06dc40 0x7f8e5c0b30f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c0b1b60 tx=0x7f8e4c009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 -- 172.21.15.119:0/2515662646 shutdown_connections 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 --2- 172.21.15.119:0/2515662646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c06dc40 0x7f8e5c0b30f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 -- 172.21.15.119:0/2515662646 >> 172.21.15.119:0/2515662646 conn(0x7f8e5c06a350 msgr2=0x7f8e5c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 -- 172.21.15.119:0/2515662646 shutdown_connections 2024-09-13T16:33:56.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 -- 172.21.15.119:0/2515662646 wait complete. 2024-09-13T16:33:56.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 Processor -- start 2024-09-13T16:33:56.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.149+0000 7f8e63244700 1 -- start start 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e5c06dc40 0x7f8e5c15b160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c0b3630 0x7f8e5c155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c1594f0 0x7f8e5c1f1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1f23d0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c1edd80 con 0x7f8e5c1594f0 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e5c156680 con 0x7f8e5c0b3630 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e5c06dc40 0x7f8e5c15b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e5c06dc40 0x7f8e5c15b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34384/0 (socket says 172.21.15.119:34384) 2024-09-13T16:33:56.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 -- 172.21.15.119:0/4027825836 learned_addr learned my addr 172.21.15.119:0/4027825836 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e61a41700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c0b3630 0x7f8e5c155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62a43700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c1594f0 0x7f8e5c1f1e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 -- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c0b3630 msgr2=0x7f8e5c155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c0b3630 0x7f8e5c155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 -- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c1594f0 msgr2=0x7f8e5c1f1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c1594f0 0x7f8e5c1f1e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 -- 172.21.15.119:0/4027825836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0094b0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62a43700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c1594f0 0x7f8e5c1f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e62242700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e5c06dc40 0x7f8e5c15b160 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800c700 tx=0x7f8e5800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e52ffd700 1 -- 172.21.15.119:0/4027825836 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58004ca0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e52ffd700 1 -- 172.21.15.119:0/4027825836 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e58004e20 con 0x7f8e5c06dc40 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e5c156960 con 0x7f8e5c06dc40 2024-09-13T16:33:56.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e52ffd700 1 -- 172.21.15.119:0/4027825836 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e58007310 con 0x7f8e5c06dc40 2024-09-13T16:33:56.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.153+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e5c14f530 con 0x7f8e5c06dc40 2024-09-13T16:33:56.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.157+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e5c0b26f0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.157+0000 7f8e52ffd700 1 -- 172.21.15.119:0/4027825836 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8e58007470 con 0x7f8e5c06dc40 2024-09-13T16:33:56.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.157+0000 7f8e52ffd700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e40066730 0x7f8e40068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.157+0000 7f8e52ffd700 1 -- 172.21.15.119:0/4027825836 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f8e58083ac0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.157+0000 7f8e61a41700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e40066730 0x7f8e40068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.157+0000 7f8e61a41700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e40066730 0x7f8e40068bf0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0090b0 tx=0x7f8e4c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.161+0000 7f8e52ffd700 1 -- 172.21.15.119:0/4027825836 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8e5c0b26f0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.337+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e5c0b26f0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.341+0000 7f8e52ffd700 1 -- 172.21.15.119:0/4027825836 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+13124 (secure 0 0 0) 0x7f8e5c0b26f0 con 0x7f8e5c06dc40 2024-09-13T16:33:56.344 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:56.344 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":107,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:55.621512+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":93,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":97,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e40066730 msgr2=0x7f8e40068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e40066730 0x7f8e40068bf0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c0090b0 tx=0x7f8e4c008030 comp rx=0 tx=0).stop 2024-09-13T16:33:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e5c06dc40 msgr2=0x7f8e5c15b160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e5c06dc40 0x7f8e5c15b160 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f8e5800c700 tx=0x7f8e5800ca10 comp rx=0 tx=0).stop 2024-09-13T16:33:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 shutdown_connections 2024-09-13T16:33:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8e40066730 0x7f8e40068bf0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8e5c1594f0 0x7f8e5c1f1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8e5c0b3630 0x7f8e5c155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 --2- 172.21.15.119:0/4027825836 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8e5c06dc40 0x7f8e5c15b160 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 >> 172.21.15.119:0/4027825836 conn(0x7f8e5c06a350 msgr2=0x7f8e5c0add10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:56.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 shutdown_connections 2024-09-13T16:33:56.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.345+0000 7f8e63244700 1 -- 172.21.15.119:0/4027825836 wait complete. 2024-09-13T16:33:56.364 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:33:56.364 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:33:56.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.473+0000 7fb4a7538700 1 Processor -- start 2024-09-13T16:33:56.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.473+0000 7fb4a7538700 1 -- start start 2024-09-13T16:33:56.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb4a015ccb0 0x7fb4a00d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a00d0da0 0x7fb4a00b4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a00d11e0 con 0x7fb4a00b54a0 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a00d16c0 con 0x7fb4a015ccb0 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00b5950 con 0x7fb4a00d0da0 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6d37700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb4a00b54a0 0x7fb4a006bbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59202/0 (socket says 172.21.15.119:59202) 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6d37700 1 -- 172.21.15.119:0/1847919895 learned_addr learned my addr 172.21.15.119:0/1847919895 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6536700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb4a015ccb0 0x7fb4a00d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a5d35700 1 --2- 172.21.15.119:0/1847919895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a00d0da0 0x7fb4a00b4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a5534700 1 -- 172.21.15.119:0/1847919895 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477734509 0 0) 0x7fb4a00d11e0 con 0x7fb4a00b54a0 2024-09-13T16:33:56.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a5534700 1 -- 172.21.15.119:0/1847919895 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c0035d0 con 0x7fb4a00b54a0 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6536700 1 -- 172.21.15.119:0/1847919895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a00d0da0 msgr2=0x7fb4a00b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6536700 1 --2- 172.21.15.119:0/1847919895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a00d0da0 0x7fb4a00b4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6536700 1 -- 172.21.15.119:0/1847919895 >> v1:172.21.15.119:6789/0 conn(0x7fb4a00b54a0 legacy=0x7fb4a006bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6536700 1 -- 172.21.15.119:0/1847919895 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a00b5bb0 con 0x7fb4a015ccb0 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a5d35700 1 --2- 172.21.15.119:0/1847919895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a00d0da0 0x7fb4a00b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a6536700 1 --2- 172.21.15.119:0/1847919895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a015ccb0 0x7fb4a00d0860 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fb490009850 tx=0x7fb490009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de7a07d2414a6f0b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a5534700 1 -- 172.21.15.119:0/1847919895 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490011830 con 0x7fb4a015ccb0 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a5534700 1 -- 172.21.15.119:0/1847919895 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb490011990 con 0x7fb4a015ccb0 2024-09-13T16:33:56.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a5534700 1 -- 172.21.15.119:0/1847919895 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490011d40 con 0x7fb4a015ccb0 2024-09-13T16:33:56.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- 172.21.15.119:0/1847919895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a015ccb0 msgr2=0x7fb4a00d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 --2- 172.21.15.119:0/1847919895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a015ccb0 0x7fb4a00d0860 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 crypto rx=0x7fb490009850 tx=0x7fb490009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:56.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- 172.21.15.119:0/1847919895 shutdown_connections 2024-09-13T16:33:56.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 --2- 172.21.15.119:0/1847919895 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a00d0da0 0x7fb4a00b4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 --2- 172.21.15.119:0/1847919895 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a015ccb0 0x7fb4a00d0860 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- 172.21.15.119:0/1847919895 >> 172.21.15.119:0/1847919895 conn(0x7fb4a0072240 msgr2=0x7fb4a0099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:56.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- 172.21.15.119:0/1847919895 shutdown_connections 2024-09-13T16:33:56.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.477+0000 7fb4a7538700 1 -- 172.21.15.119:0/1847919895 wait complete. 2024-09-13T16:33:56.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 Processor -- start 2024-09-13T16:33:56.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 -- start start 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a00b54a0 0x7fb4a009da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4a00d0da0 0x7fb4a009df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a015ccb0 0x7fb4a0147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0158b90 con 0x7fb4a00b54a0 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a01588b0 con 0x7fb4a00d0da0 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a0145180 con 0x7fb4a015ccb0 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a00b54a0 0x7fb4a009da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a00b54a0 0x7fb4a009da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34392/0 (socket says 172.21.15.119:34392) 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 -- 172.21.15.119:0/2623220846 learned_addr learned my addr 172.21.15.119:0/2623220846 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:56.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6d37700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a015ccb0 0x7fb4a0147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a5d35700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4a00d0da0 0x7fb4a009df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 -- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a015ccb0 msgr2=0x7fb4a0147080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a015ccb0 0x7fb4a0147080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 -- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4a00d0da0 msgr2=0x7fb4a009df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4a00d0da0 0x7fb4a009df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 -- 172.21.15.119:0/2623220846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4900094b0 con 0x7fb4a00b54a0 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a6536700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a00b54a0 0x7fb4a009da00 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb4900090b0 tx=0x7fb490004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb496ffd700 1 -- 172.21.15.119:0/2623220846 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb490004610 con 0x7fb4a00b54a0 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a0145460 con 0x7fb4a00b54a0 2024-09-13T16:33:56.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb496ffd700 1 -- 172.21.15.119:0/2623220846 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb49001e070 con 0x7fb4a00b54a0 2024-09-13T16:33:56.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb496ffd700 1 -- 172.21.15.119:0/2623220846 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4900038c0 con 0x7fb4a00b54a0 2024-09-13T16:33:56.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.481+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a01475c0 con 0x7fb4a00b54a0 2024-09-13T16:33:56.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.485+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a0073590 con 0x7fb4a00b54a0 2024-09-13T16:33:56.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.485+0000 7fb496ffd700 1 -- 172.21.15.119:0/2623220846 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb490003a20 con 0x7fb4a00b54a0 2024-09-13T16:33:56.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.489+0000 7fb496ffd700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb484066730 0x7fb484068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.489+0000 7fb496ffd700 1 -- 172.21.15.119:0/2623220846 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb490020070 con 0x7fb4a00b54a0 2024-09-13T16:33:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.489+0000 7fb4a5d35700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb484066730 0x7fb484068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.489+0000 7fb4a5d35700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb484066730 0x7fb484068bf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb4a01f57f0 tx=0x7fb49c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.493+0000 7fb496ffd700 1 -- 172.21.15.119:0/2623220846 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb490052c70 con 0x7fb4a00b54a0 2024-09-13T16:33:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.661+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb4a00d1390 con 0x7fb4a00b54a0 2024-09-13T16:33:56.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.661+0000 7fb496ffd700 1 -- 172.21.15.119:0/2623220846 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v107) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb490056290 con 0x7fb4a00b54a0 2024-09-13T16:33:56.667 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb484066730 msgr2=0x7fb484068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb484066730 0x7fb484068bf0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7fb4a01f57f0 tx=0x7fb49c008040 comp rx=0 tx=0).stop 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a00b54a0 msgr2=0x7fb4a009da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a00b54a0 0x7fb4a009da00 secure :-1 s=READY pgs=1549 cs=0 l=1 rev1=1 crypto rx=0x7fb4900090b0 tx=0x7fb490004770 comp rx=0 tx=0).stop 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 shutdown_connections 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb484066730 0x7fb484068bf0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb4a015ccb0 0x7fb4a0147080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb4a00d0da0 0x7fb4a009df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 --2- 172.21.15.119:0/2623220846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb4a00b54a0 0x7fb4a009da00 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 >> 172.21.15.119:0/2623220846 conn(0x7fb4a0072240 msgr2=0x7fb4a00b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:56.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 shutdown_connections 2024-09-13T16:33:56.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.665+0000 7fb4a7538700 1 -- 172.21.15.119:0/2623220846 wait complete. 2024-09-13T16:33:56.682 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:33:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.785+0000 7fa1e359e700 1 Processor -- start 2024-09-13T16:33:56.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.785+0000 7fa1e359e700 1 -- start start 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa1e40adb30 0x7fa1e40cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e406bb90 0x7fa1e40cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e406cf60 con 0x7fa1e40cde70 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e40b3390 con 0x7fa1e40adb30 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e406c350 con 0x7fa1e406bb90 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa1e40adb30 0x7fa1e40cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e2d9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa1e40cde70 0x7fa1e415e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59208/0 (socket says 172.21.15.119:59208) 2024-09-13T16:33:56.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e2d9d700 1 -- 172.21.15.119:0/3530692182 learned_addr learned my addr 172.21.15.119:0/3530692182 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:56.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e1d9b700 1 --2- 172.21.15.119:0/3530692182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e406bb90 0x7fa1e40cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e159a700 1 -- 172.21.15.119:0/3530692182 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232297407 0 0) 0x7fa1e406cf60 con 0x7fa1e40cde70 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e259c700 1 -- 172.21.15.119:0/3530692182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e406bb90 msgr2=0x7fa1e40cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e259c700 1 --2- 172.21.15.119:0/3530692182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e406bb90 0x7fa1e40cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e259c700 1 -- 172.21.15.119:0/3530692182 >> v1:172.21.15.119:6789/0 conn(0x7fa1e40cde70 legacy=0x7fa1e415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e259c700 1 -- 172.21.15.119:0/3530692182 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e406c630 con 0x7fa1e40adb30 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e259c700 1 --2- 172.21.15.119:0/3530692182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40adb30 0x7fa1e40cc060 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009850 tx=0x7fa1cc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d2544c1bf10104 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e159a700 1 -- 172.21.15.119:0/3530692182 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc011830 con 0x7fa1e40adb30 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e159a700 1 -- 172.21.15.119:0/3530692182 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc011990 con 0x7fa1e40adb30 2024-09-13T16:33:56.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e159a700 1 -- 172.21.15.119:0/3530692182 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc011d40 con 0x7fa1e40adb30 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- 172.21.15.119:0/3530692182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40adb30 msgr2=0x7fa1e40cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 --2- 172.21.15.119:0/3530692182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40adb30 0x7fa1e40cc060 secure :-1 s=READY pgs=1551 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009850 tx=0x7fa1cc009b60 comp rx=0 tx=0).stop 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- 172.21.15.119:0/3530692182 shutdown_connections 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 --2- 172.21.15.119:0/3530692182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e406bb90 0x7fa1e40cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 --2- 172.21.15.119:0/3530692182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40adb30 0x7fa1e40cc060 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- 172.21.15.119:0/3530692182 >> 172.21.15.119:0/3530692182 conn(0x7fa1e406a350 msgr2=0x7fa1e406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- 172.21.15.119:0/3530692182 shutdown_connections 2024-09-13T16:33:56.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.789+0000 7fa1e359e700 1 -- 172.21.15.119:0/3530692182 wait complete. 2024-09-13T16:33:56.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 Processor -- start 2024-09-13T16:33:56.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 -- start start 2024-09-13T16:33:56.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1e406bb90 0x7fa1e41ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e40adb30 0x7fa1e415e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40cde70 0x7fa1e415eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e41346a0 con 0x7fa1e40cde70 2024-09-13T16:33:56.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e41332c0 con 0x7fa1e406bb90 2024-09-13T16:33:56.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e41335a0 con 0x7fa1e40adb30 2024-09-13T16:33:56.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40cde70 0x7fa1e415eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e1d9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e40adb30 0x7fa1e415e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40cde70 0x7fa1e415eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34410/0 (socket says 172.21.15.119:34410) 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e259c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1e406bb90 0x7fa1e41ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 -- 172.21.15.119:0/770049127 learned_addr learned my addr 172.21.15.119:0/770049127 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 -- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e40adb30 msgr2=0x7fa1e415e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e40adb30 0x7fa1e415e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 -- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1e406bb90 msgr2=0x7fa1e41ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1e406bb90 0x7fa1e41ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 -- 172.21.15.119:0/770049127 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc0094b0 con 0x7fa1e40cde70 2024-09-13T16:33:56.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1e2d9d700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40cde70 0x7fa1e415eb80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c460 tx=0x7fa1dc00c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.793+0000 7fa1daffd700 1 -- 172.21.15.119:0/770049127 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc004ca0 con 0x7fa1e40cde70 2024-09-13T16:33:56.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1daffd700 1 -- 172.21.15.119:0/770049127 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1dc004e20 con 0x7fa1e40cde70 2024-09-13T16:33:56.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e4133880 con 0x7fa1e40cde70 2024-09-13T16:33:56.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1daffd700 1 -- 172.21.15.119:0/770049127 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1dc007310 con 0x7fa1e40cde70 2024-09-13T16:33:56.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e4133e40 con 0x7fa1e40cde70 2024-09-13T16:33:56.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e406d3d0 con 0x7fa1e40cde70 2024-09-13T16:33:56.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1daffd700 1 -- 172.21.15.119:0/770049127 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa1dc007470 con 0x7fa1e40cde70 2024-09-13T16:33:56.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1daffd700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:56.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1daffd700 1 -- 172.21.15.119:0/770049127 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fa1dc083ad0 con 0x7fa1e40cde70 2024-09-13T16:33:56.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.797+0000 7fa1e259c700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:56.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.801+0000 7fa1e259c700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0090b0 tx=0x7fa1cc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:56.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.805+0000 7fa1daffd700 1 -- 172.21.15.119:0/770049127 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa1e406d3d0 con 0x7fa1e40cde70 2024-09-13T16:33:56.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.977+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa1e406d3d0 con 0x7fa1e40cde70 2024-09-13T16:33:56.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.977+0000 7fa1daffd700 1 -- 172.21.15.119:0/770049127 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+65 (secure 0 0 0) 0x7fa1e406d3d0 con 0x7fa1e40cde70 2024-09-13T16:33:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:56.981 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:33:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:33:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:33:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:33:56.981 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:33:56.981 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:33:56.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 msgr2=0x7fa1c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc0090b0 tx=0x7fa1cc008030 comp rx=0 tx=0).stop 2024-09-13T16:33:56.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40cde70 msgr2=0x7fa1e415eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:56.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40cde70 0x7fa1e415eb80 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 crypto rx=0x7fa1dc00c460 tx=0x7fa1dc00c770 comp rx=0 tx=0).stop 2024-09-13T16:33:56.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 shutdown_connections 2024-09-13T16:33:56.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1e40cde70 0x7fa1e415eb80 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1e40adb30 0x7fa1e415e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 --2- 172.21.15.119:0/770049127 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1e406bb90 0x7fa1e41ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:56.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 >> 172.21.15.119:0/770049127 conn(0x7fa1e406a350 msgr2=0x7fa1e40ca9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:56.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 shutdown_connections 2024-09-13T16:33:56.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:56.981+0000 7fa1e359e700 1 -- 172.21.15.119:0/770049127 wait complete. 2024-09-13T16:33:57.001 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:33:57.002 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:33:57.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6294383700 1 Processor -- start 2024-09-13T16:33:57.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6294383700 1 -- start start 2024-09-13T16:33:57.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6294383700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f628c159830 0x7f628c0b3360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6294383700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c071560 0x7f628c0cd860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6294383700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628c0cd3d0 con 0x7f628c06e820 2024-09-13T16:33:57.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6294383700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628c0ccb00 con 0x7f628c159830 2024-09-13T16:33:57.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6294383700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c0b3a20 con 0x7f628c071560 2024-09-13T16:33:57.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6292920700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c071560 0x7f628c0cd860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6292920700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c071560 0x7f628c0cd860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:36694/0 (socket says 172.21.15.119:36694) 2024-09-13T16:33:57.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6292920700 1 -- 172.21.15.119:0/580842894 learned_addr learned my addr 172.21.15.119:0/580842894 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f629211f700 1 --2- 172.21.15.119:0/580842894 >> v2:172.21.15.145:3300/0 conn(0x7f628c159830 0x7f628c0b3360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6292920700 1 -- 172.21.15.119:0/580842894 >> v2:172.21.15.145:3300/0 conn(0x7f628c159830 msgr2=0x7f628c0b3360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6292920700 1 --2- 172.21.15.119:0/580842894 >> v2:172.21.15.145:3300/0 conn(0x7f628c159830 0x7f628c0b3360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6292920700 1 -- 172.21.15.119:0/580842894 >> v1:172.21.15.145:6789/0 conn(0x7f628c06e820 legacy=0x7f628c070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.113+0000 7f6292920700 1 -- 172.21.15.119:0/580842894 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628c0ccd80 con 0x7f628c071560 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6292920700 1 --2- 172.21.15.119:0/580842894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c071560 0x7f628c0cd860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6284009910 tx=0x7f6284009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d498ba2c9b6e00ec server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f629111d700 1 -- 172.21.15.119:0/580842894 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284011830 con 0x7f628c071560 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f629111d700 1 -- 172.21.15.119:0/580842894 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6284011990 con 0x7f628c071560 2024-09-13T16:33:57.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f629111d700 1 -- 172.21.15.119:0/580842894 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284011d40 con 0x7f628c071560 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 -- 172.21.15.119:0/580842894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c071560 msgr2=0x7f628c0cd860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 --2- 172.21.15.119:0/580842894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c071560 0x7f628c0cd860 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6284009910 tx=0x7f6284009c20 comp rx=0 tx=0).stop 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 -- 172.21.15.119:0/580842894 shutdown_connections 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 --2- 172.21.15.119:0/580842894 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c071560 0x7f628c0cd860 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 --2- 172.21.15.119:0/580842894 >> v2:172.21.15.145:3300/0 conn(0x7f628c159830 0x7f628c0b3360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 -- 172.21.15.119:0/580842894 >> 172.21.15.119:0/580842894 conn(0x7f628c0b1650 msgr2=0x7f628c0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 -- 172.21.15.119:0/580842894 shutdown_connections 2024-09-13T16:33:57.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 -- 172.21.15.119:0/580842894 wait complete. 2024-09-13T16:33:57.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 Processor -- start 2024-09-13T16:33:57.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.117+0000 7f6294383700 1 -- start start 2024-09-13T16:33:57.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c06e820 0x7f628c156d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f628c071560 0x7f628c1470f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f628c159830 0x7f628c0cde00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c142fa0 con 0x7f628c071560 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f628c143280 con 0x7f628c159830 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f628c143560 con 0x7f628c06e820 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f628c071560 0x7f628c1470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f628c071560 0x7f628c1470f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34420/0 (socket says 172.21.15.119:34420) 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 -- 172.21.15.119:0/521186434 learned_addr learned my addr 172.21.15.119:0/521186434 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6292920700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f628c159830 0x7f628c0cde00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629211f700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c06e820 0x7f628c156d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 -- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c06e820 msgr2=0x7f628c156d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c06e820 0x7f628c156d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 -- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f628c159830 msgr2=0x7f628c0cde00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f628c159830 0x7f628c0cde00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 -- 172.21.15.119:0/521186434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6284009570 con 0x7f628c071560 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629211f700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c06e820 0x7f628c156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f629191e700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f628c071560 0x7f628c1470f0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f628000c500 tx=0x7f628000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f627effd700 1 -- 172.21.15.119:0/521186434 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62800052d0 con 0x7f628c071560 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f627effd700 1 -- 172.21.15.119:0/521186434 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6280005430 con 0x7f628c071560 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f628c143840 con 0x7f628c071560 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f627effd700 1 -- 172.21.15.119:0/521186434 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62800055a0 con 0x7f628c071560 2024-09-13T16:33:57.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f628c144040 con 0x7f628c071560 2024-09-13T16:33:57.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628c143d60 con 0x7f628c071560 2024-09-13T16:33:57.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.121+0000 7f627effd700 1 -- 172.21.15.119:0/521186434 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f62800038e0 con 0x7f628c071560 2024-09-13T16:33:57.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.125+0000 7f627effd700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6270066730 0x7f6270068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.125+0000 7f627effd700 1 -- 172.21.15.119:0/521186434 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f6280083450 con 0x7f628c071560 2024-09-13T16:33:57.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.125+0000 7f629211f700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6270066730 0x7f6270068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.125+0000 7f629211f700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6270066730 0x7f6270068bf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f628c147a30 tx=0x7f628800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:57.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.133+0000 7f627effd700 1 -- 172.21.15.119:0/521186434 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f628c143d60 con 0x7f628c071560 2024-09-13T16:33:57.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.301+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f628c143d60 con 0x7f628c071560 2024-09-13T16:33:57.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.641+0000 7f627effd700 1 -- 172.21.15.119:0/521186434 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v108) v1 ==== 138+0+0 (secure 0 0 0) 0x7f628c143d60 con 0x7f628c071560 2024-09-13T16:33:57.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6270066730 msgr2=0x7f6270068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6270066730 0x7f6270068bf0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f628c147a30 tx=0x7f628800b040 comp rx=0 tx=0).stop 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f628c071560 msgr2=0x7f628c1470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f628c071560 0x7f628c1470f0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f628000c500 tx=0x7f628000c810 comp rx=0 tx=0).stop 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 shutdown_connections 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6270066730 0x7f6270068bf0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f628c159830 0x7f628c0cde00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f628c071560 0x7f628c1470f0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 --2- 172.21.15.119:0/521186434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f628c06e820 0x7f628c156d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 >> 172.21.15.119:0/521186434 conn(0x7f628c0b1650 msgr2=0x7f628c1512c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.645+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 shutdown_connections 2024-09-13T16:33:57.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.649+0000 7f6294383700 1 -- 172.21.15.119:0/521186434 wait complete. 2024-09-13T16:33:57.653 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:33:57.667 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:33:57.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 Processor -- start 2024-09-13T16:33:57.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- start start 2024-09-13T16:33:57.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f68c40ce8c0 0x7f68c40d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68c40d1a60 con 0x7f68c415cc00 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68c40d0df0 con 0x7f68c40ce8c0 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40cdb00 con 0x7f68c40d1610 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f68c40ce8c0 0x7f68c40d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f68c40ce8c0 0x7f68c40d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34422/0 (socket says 172.21.15.119:34422) 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 -- 172.21.15.119:0/2098777052 learned_addr learned my addr 172.21.15.119:0/2098777052 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9714700 1 --2- 172.21.15.119:0/2098777052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 -- 172.21.15.119:0/2098777052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 msgr2=0x7f68c40b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 --2- 172.21.15.119:0/2098777052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 -- 172.21.15.119:0/2098777052 >> v1:172.21.15.145:6789/0 conn(0x7f68c415cc00 legacy=0x7f68c40d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 -- 172.21.15.119:0/2098777052 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c40cdde0 con 0x7f68c40ce8c0 2024-09-13T16:33:57.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c3fff700 1 --2- 172.21.15.119:0/2098777052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c40ce8c0 0x7f68c40d10d0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009910 tx=0x7f68b4009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a282ebcbf88440db server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:57.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c37fe700 1 -- 172.21.15.119:0/2098777052 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b400ba60 con 0x7f68c40ce8c0 2024-09-13T16:33:57.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c37fe700 1 -- 172.21.15.119:0/2098777052 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68b400bbc0 con 0x7f68c40ce8c0 2024-09-13T16:33:57.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c37fe700 1 -- 172.21.15.119:0/2098777052 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68b4009e60 con 0x7f68c40ce8c0 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- 172.21.15.119:0/2098777052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c40ce8c0 msgr2=0x7f68c40d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 --2- 172.21.15.119:0/2098777052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c40ce8c0 0x7f68c40d10d0 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f68b4009910 tx=0x7f68b4009c20 comp rx=0 tx=0).stop 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- 172.21.15.119:0/2098777052 shutdown_connections 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 --2- 172.21.15.119:0/2098777052 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 --2- 172.21.15.119:0/2098777052 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c40ce8c0 0x7f68c40d10d0 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- 172.21.15.119:0/2098777052 >> 172.21.15.119:0/2098777052 conn(0x7f68c408c080 msgr2=0x7f68c40d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- 172.21.15.119:0/2098777052 shutdown_connections 2024-09-13T16:33:57.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.785+0000 7f68c9f15700 1 -- 172.21.15.119:0/2098777052 wait complete. 2024-09-13T16:33:57.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 Processor -- start 2024-09-13T16:33:57.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 -- start start 2024-09-13T16:33:57.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40ce8c0 0x7f68c4157f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c415cc00 0x7f68c40ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c415dbb0 con 0x7f68c415cc00 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68c415fa60 con 0x7f68c40ce8c0 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9f15700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68c40ccd60 con 0x7f68c40d1610 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c415cc00 0x7f68c40ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c415cc00 0x7f68c40ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34424/0 (socket says 172.21.15.119:34424) 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 -- 172.21.15.119:0/125348778 learned_addr learned my addr 172.21.15.119:0/125348778 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c8f13700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40ce8c0 0x7f68c4157f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c3fff700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 -- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 msgr2=0x7f68c40c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 -- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40ce8c0 msgr2=0x7f68c4157f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40ce8c0 0x7f68c4157f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.789+0000 7f68c9714700 1 -- 172.21.15.119:0/125348778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b4009570 con 0x7f68c415cc00 2024-09-13T16:33:57.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c9714700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c415cc00 0x7f68c40ca120 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f68bc007e40 tx=0x7f68bc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:57.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c17fa700 1 -- 172.21.15.119:0/125348778 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc00ede0 con 0x7f68c415cc00 2024-09-13T16:33:57.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c17fa700 1 -- 172.21.15.119:0/125348778 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68bc014420 con 0x7f68c415cc00 2024-09-13T16:33:57.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68c415fcb0 con 0x7f68c415cc00 2024-09-13T16:33:57.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c17fa700 1 -- 172.21.15.119:0/125348778 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68bc0147d0 con 0x7f68c415cc00 2024-09-13T16:33:57.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68c40cc950 con 0x7f68c415cc00 2024-09-13T16:33:57.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68c415dcf0 con 0x7f68c415cc00 2024-09-13T16:33:57.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c17fa700 1 -- 172.21.15.119:0/125348778 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f68bc014930 con 0x7f68c415cc00 2024-09-13T16:33:57.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.793+0000 7f68c17fa700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f68a4066730 0x7f68a4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:57.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.797+0000 7f68c17fa700 1 -- 172.21.15.119:0/125348778 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f68bc084830 con 0x7f68c415cc00 2024-09-13T16:33:57.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.797+0000 7f68c8f13700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f68a4066730 0x7f68a4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:57.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.797+0000 7f68c8f13700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f68a4066730 0x7f68a4068bf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f68ac000f60 tx=0x7f68ac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:57.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.801+0000 7f68c17fa700 1 -- 172.21.15.119:0/125348778 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f68c415dcf0 con 0x7f68c415cc00 2024-09-13T16:33:57.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:57.973+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f68c415dcf0 con 0x7f68c415cc00 2024-09-13T16:33:58.151 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:33:58.153 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:33:58.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.669+0000 7f68c17fa700 1 -- 172.21.15.119:0/125348778 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v109) v1 ==== 170+0+0 (secure 0 0 0) 0x7f68c415dcf0 con 0x7f68c415cc00 2024-09-13T16:33:58.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f68a4066730 msgr2=0x7f68a4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:58.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f68a4066730 0x7f68a4068bf0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f68ac000f60 tx=0x7f68ac009040 comp rx=0 tx=0).stop 2024-09-13T16:33:58.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c415cc00 msgr2=0x7f68c40ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:58.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c415cc00 0x7f68c40ca120 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 crypto rx=0x7f68bc007e40 tx=0x7f68bc00e3c0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 shutdown_connections 2024-09-13T16:33:58.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f68a4066730 0x7f68a4068bf0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f68c415cc00 0x7f68c40ca120 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f68c40d1610 0x7f68c40c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 --2- 172.21.15.119:0/125348778 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f68c40ce8c0 0x7f68c4157f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 >> 172.21.15.119:0/125348778 conn(0x7f68c408c080 msgr2=0x7f68c4161000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:58.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 shutdown_connections 2024-09-13T16:33:58.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.677+0000 7f68c9f15700 1 -- 172.21.15.119:0/125348778 wait complete. 2024-09-13T16:33:58.683 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:33:58.694 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:33:58.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.801+0000 7fd90ad2d700 1 Processor -- start 2024-09-13T16:33:58.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.801+0000 7fd90ad2d700 1 -- start start 2024-09-13T16:33:58.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd9040b2e00 0x7fd9040ad550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:58.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90406d100 con 0x7fd90406ccb0 2024-09-13T16:33:58.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90406e110 con 0x7fd9041591e0 2024-09-13T16:33:58.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90406d350 con 0x7fd9040b2e00 2024-09-13T16:33:58.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd903fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd90406ccb0 0x7fd9040ace40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59216/0 (socket says 172.21.15.119:59216) 2024-09-13T16:33:58.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd903fff700 1 -- 172.21.15.119:0/1486616940 learned_addr learned my addr 172.21.15.119:0/1486616940 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:58.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9092ca700 1 --2- 172.21.15.119:0/1486616940 >> v2:172.21.15.145:3300/0 conn(0x7fd9040b2e00 0x7fd9040ad550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9037fe700 1 -- 172.21.15.119:0/1486616940 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3056392439 0 0) 0x7fd90406d100 con 0x7fd90406ccb0 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9037fe700 1 -- 172.21.15.119:0/1486616940 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8f40035d0 con 0x7fd90406ccb0 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9092ca700 1 -- 172.21.15.119:0/1486616940 >> v1:172.21.15.145:6789/0 conn(0x7fd9041591e0 legacy=0x7fd9040b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9092ca700 1 -- 172.21.15.119:0/1486616940 >> v1:172.21.15.119:6789/0 conn(0x7fd90406ccb0 legacy=0x7fd9040ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9092ca700 1 -- 172.21.15.119:0/1486616940 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90406d630 con 0x7fd9040b2e00 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9092ca700 1 --2- 172.21.15.119:0/1486616940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 0x7fd9040ad550 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009d90 tx=0x7fd8fc00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c33f3b6bcbb28856 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9037fe700 1 -- 172.21.15.119:0/1486616940 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc00e070 con 0x7fd9040b2e00 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9037fe700 1 -- 172.21.15.119:0/1486616940 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc00bb20 con 0x7fd9040b2e00 2024-09-13T16:33:58.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd9037fe700 1 -- 172.21.15.119:0/1486616940 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc012430 con 0x7fd9040b2e00 2024-09-13T16:33:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- 172.21.15.119:0/1486616940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 msgr2=0x7fd9040ad550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/1486616940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 0x7fd9040ad550 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009d90 tx=0x7fd8fc00b330 comp rx=0 tx=0).stop 2024-09-13T16:33:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- 172.21.15.119:0/1486616940 shutdown_connections 2024-09-13T16:33:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/1486616940 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 0x7fd9040ad550 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- 172.21.15.119:0/1486616940 >> 172.21.15.119:0/1486616940 conn(0x7fd9040b22e0 msgr2=0x7fd9040b26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:58.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- 172.21.15.119:0/1486616940 shutdown_connections 2024-09-13T16:33:58.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.805+0000 7fd90ad2d700 1 -- 172.21.15.119:0/1486616940 wait complete. 2024-09-13T16:33:58.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 Processor -- start 2024-09-13T16:33:58.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 -- start start 2024-09-13T16:33:58.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd90406ccb0 0x7fd9040cbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:58.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 0x7fd9041f1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:58.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9041591e0 0x7fd9041f1fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:58.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040ce320 con 0x7fd9041591e0 2024-09-13T16:33:58.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040ceb30 con 0x7fd9040b2e00 2024-09-13T16:33:58.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd90ad2d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040ce570 con 0x7fd90406ccb0 2024-09-13T16:33:58.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd9092ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9041591e0 0x7fd9041f1fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:58.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd9092ca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9041591e0 0x7fd9041f1fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34432/0 (socket says 172.21.15.119:34432) 2024-09-13T16:33:58.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd9092ca700 1 -- 172.21.15.119:0/3045311633 learned_addr learned my addr 172.21.15.119:0/3045311633 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:58.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd903fff700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 0x7fd9041f1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:58.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd908ac9700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd90406ccb0 0x7fd9040cbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:58.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd9092ca700 1 -- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd90406ccb0 msgr2=0x7fd9040cbbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:58.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd9092ca700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd90406ccb0 0x7fd9040cbbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd9092ca700 1 -- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 msgr2=0x7fd9041f1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:58.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.809+0000 7fd9092ca700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 0x7fd9041f1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9092ca700 1 -- 172.21.15.119:0/3045311633 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc009a70 con 0x7fd9041591e0 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9092ca700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9041591e0 0x7fd9041f1fb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc013000 tx=0x7fd8fc004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9017fa700 1 -- 172.21.15.119:0/3045311633 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc012430 con 0x7fd9041591e0 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9017fa700 1 -- 172.21.15.119:0/3045311633 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc020070 con 0x7fd9041591e0 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9040ce850 con 0x7fd9041591e0 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9017fa700 1 -- 172.21.15.119:0/3045311633 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc004280 con 0x7fd9041591e0 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd904156020 con 0x7fd9041591e0 2024-09-13T16:33:58.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90406c660 con 0x7fd9041591e0 2024-09-13T16:33:58.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9017fa700 1 -- 172.21.15.119:0/3045311633 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd8fc0043e0 con 0x7fd9041591e0 2024-09-13T16:33:58.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9017fa700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066730 0x7fd8e8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:58.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd9017fa700 1 -- 172.21.15.119:0/3045311633 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5929+0+0 (secure 0 0 0) 0x7fd8fc022070 con 0x7fd9041591e0 2024-09-13T16:33:58.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.813+0000 7fd908ac9700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066730 0x7fd8e8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:58.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.817+0000 7fd908ac9700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066730 0x7fd8e8068bf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8000f60 tx=0x7fd8f8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:58.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:58.825+0000 7fd9017fa700 1 -- 172.21.15.119:0/3045311633 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd8fc053390 con 0x7fd9041591e0 2024-09-13T16:33:59.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.029+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fd9041561e0 con 0x7fd9041591e0 2024-09-13T16:33:59.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.041+0000 7fd9017fa700 1 -- 172.21.15.119:0/3045311633 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 15 and data pool 16 v73) v1 ==== 174+0+0 (secure 0 0 0) 0x7fd8fc0569b0 con 0x7fd9041591e0 2024-09-13T16:33:59.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.045+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066730 msgr2=0x7fd8e8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.045+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066730 0x7fd8e8068bf0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7fd8f8000f60 tx=0x7fd8f8009040 comp rx=0 tx=0).stop 2024-09-13T16:33:59.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.045+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9041591e0 msgr2=0x7fd9041f1fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.045+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9041591e0 0x7fd9041f1fb0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc013000 tx=0x7fd8fc004af0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 shutdown_connections 2024-09-13T16:33:59.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066730 0x7fd8e8068bf0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9041591e0 0x7fd9041f1fb0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9040b2e00 0x7fd9041f1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 --2- 172.21.15.119:0/3045311633 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd90406ccb0 0x7fd9040cbbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 >> 172.21.15.119:0/3045311633 conn(0x7fd9040b22e0 msgr2=0x7fd9041504c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:59.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 shutdown_connections 2024-09-13T16:33:59.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.049+0000 7fd90ad2d700 1 -- 172.21.15.119:0/3045311633 wait complete. 2024-09-13T16:33:59.055 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 15 and data pool 16 2024-09-13T16:33:59.072 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:59.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.165+0000 7f4b74aea700 1 Processor -- start 2024-09-13T16:33:59.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.165+0000 7f4b74aea700 1 -- start start 2024-09-13T16:33:59.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4b700cd200 0x7f4b7015a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b700ce540 con 0x7f4b7015a010 2024-09-13T16:33:59.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b700cc720 con 0x7f4b700cd200 2024-09-13T16:33:59.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700cca00 con 0x7f4b7006c5f0 2024-09-13T16:33:59.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6e59c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6ed9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4b700cd200 0x7f4b7015a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b67fff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f4b7015a010 0x7f4b7006c9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:38148/0 (socket says 172.21.15.119:38148) 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b67fff700 1 -- 172.21.15.119:0/3760863631 learned_addr learned my addr 172.21.15.119:0/3760863631 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6ed9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4b700cd200 0x7f4b7015a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:42964/0 (socket says 172.21.15.119:42964) 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6ed9d700 1 -- 172.21.15.119:0/3760863631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 msgr2=0x7f4b700cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6ed9d700 1 --2- 172.21.15.119:0/3760863631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6ed9d700 1 -- 172.21.15.119:0/3760863631 >> v1:172.21.15.145:6789/0 conn(0x7f4b7015a010 legacy=0x7f4b7006c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6ed9d700 1 -- 172.21.15.119:0/3760863631 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b700ccce0 con 0x7f4b700cd200 2024-09-13T16:33:59.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6ed9d700 1 --2- 172.21.15.119:0/3760863631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 0x7f4b7015a760 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b6800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16295cc5926f40a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6dd9b700 1 -- 172.21.15.119:0/3760863631 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b6800e070 con 0x7f4b700cd200 2024-09-13T16:33:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6dd9b700 1 -- 172.21.15.119:0/3760863631 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b6800bb20 con 0x7f4b700cd200 2024-09-13T16:33:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b6dd9b700 1 -- 172.21.15.119:0/3760863631 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68012430 con 0x7f4b700cd200 2024-09-13T16:33:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- 172.21.15.119:0/3760863631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 msgr2=0x7f4b7015a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3760863631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 0x7f4b7015a760 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009d90 tx=0x7f4b6800b330 comp rx=0 tx=0).stop 2024-09-13T16:33:59.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- 172.21.15.119:0/3760863631 shutdown_connections 2024-09-13T16:33:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3760863631 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 0x7f4b7015a760 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3760863631 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700cdeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- 172.21.15.119:0/3760863631 >> 172.21.15.119:0/3760863631 conn(0x7f4b7006a350 msgr2=0x7f4b7006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- 172.21.15.119:0/3760863631 shutdown_connections 2024-09-13T16:33:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.169+0000 7f4b74aea700 1 -- 172.21.15.119:0/3760863631 wait complete. 2024-09-13T16:33:59.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 Processor -- start 2024-09-13T16:33:59.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 -- start start 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 0x7f4b700b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7015a010 0x7f4b70135900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700b2bf0 con 0x7f4b7015a010 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b7014e2d0 con 0x7f4b700cd200 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b700b1f70 con 0x7f4b7006c5f0 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7015a010 0x7f4b70135900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b67fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 0x7f4b700b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7015a010 0x7f4b70135900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34436/0 (socket says 172.21.15.119:34436) 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 -- 172.21.15.119:0/3363117635 learned_addr learned my addr 172.21.15.119:0/3363117635 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:59.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6e59c700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 -- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 msgr2=0x7f4b700b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 -- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 msgr2=0x7f4b700b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 0x7f4b700b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 -- 172.21.15.119:0/3363117635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68009a70 con 0x7f4b7015a010 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b6ed9d700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7015a010 0x7f4b70135900 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4b7006dda0 tx=0x7f4b68004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b66ffd700 1 -- 172.21.15.119:0/3363117635 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68012430 con 0x7f4b7015a010 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b66ffd700 1 -- 172.21.15.119:0/3363117635 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b68020070 con 0x7f4b7015a010 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b700b2250 con 0x7f4b7015a010 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b66ffd700 1 -- 172.21.15.119:0/3363117635 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b68004280 con 0x7f4b7015a010 2024-09-13T16:33:59.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.173+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b700b2810 con 0x7f4b7015a010 2024-09-13T16:33:59.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.177+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b700b2d30 con 0x7f4b7015a010 2024-09-13T16:33:59.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.177+0000 7f4b66ffd700 1 -- 172.21.15.119:0/3363117635 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4b680043e0 con 0x7f4b7015a010 2024-09-13T16:33:59.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.177+0000 7f4b66ffd700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4b50066730 0x7f4b50068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.177+0000 7f4b66ffd700 1 -- 172.21.15.119:0/3363117635 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4b68022070 con 0x7f4b7015a010 2024-09-13T16:33:59.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.177+0000 7f4b6e59c700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4b50066730 0x7f4b50068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.177+0000 7f4b6e59c700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4b50066730 0x7f4b50068bf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009b50 tx=0x7f4b60009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.185+0000 7f4b66ffd700 1 -- 172.21.15.119:0/3363117635 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4b700b2d30 con 0x7f4b7015a010 2024-09-13T16:33:59.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.353+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b700b2d30 con 0x7f4b7015a010 2024-09-13T16:33:59.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.357+0000 7f4b66ffd700 1 -- 172.21.15.119:0/3363117635 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16789 (secure 0 0 0) 0x7f4b700b2d30 con 0x7f4b7015a010 2024-09-13T16:33:59.362 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:59.362 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":110,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:59.031881+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:59.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4b50066730 msgr2=0x7f4b50068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4b50066730 0x7f4b50068bf0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009b50 tx=0x7f4b60009240 comp rx=0 tx=0).stop 2024-09-13T16:33:59.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7015a010 msgr2=0x7f4b70135900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7015a010 0x7f4b70135900 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 crypto rx=0x7f4b7006dda0 tx=0x7f4b68004af0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 shutdown_connections 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4b50066730 0x7f4b50068bf0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4b7015a010 0x7f4b70135900 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4b700cd200 0x7f4b700b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 --2- 172.21.15.119:0/3363117635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4b7006c5f0 0x7f4b700b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 >> 172.21.15.119:0/3363117635 conn(0x7f4b7006a350 msgr2=0x7f4b7015bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 shutdown_connections 2024-09-13T16:33:59.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.361+0000 7f4b74aea700 1 -- 172.21.15.119:0/3363117635 wait complete. 2024-09-13T16:33:59.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:33:59.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.485+0000 7ff0ab03d700 1 Processor -- start 2024-09-13T16:33:59.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.485+0000 7ff0ab03d700 1 -- start start 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff0a415ccd0 0x7ff0a40d0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff0a40d1820 0x7ff0a40c6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0a40c6ab0 con 0x7ff0a40d0c60 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a40cea10 con 0x7ff0a415ccd0 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0a40ce170 con 0x7ff0a40d1820 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff0a415ccd0 0x7ff0a40d0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff0a415ccd0 0x7ff0a40d0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34444/0 (socket says 172.21.15.119:34444) 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 -- 172.21.15.119:0/3169815683 learned_addr learned my addr 172.21.15.119:0/3169815683 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:59.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa83c700 1 --2- 172.21.15.119:0/3169815683 >> v2:172.21.15.145:3300/0 conn(0x7ff0a40d1820 0x7ff0a40c6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 -- 172.21.15.119:0/3169815683 >> v2:172.21.15.145:3300/0 conn(0x7ff0a40d1820 msgr2=0x7ff0a40c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:59.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 --2- 172.21.15.119:0/3169815683 >> v2:172.21.15.145:3300/0 conn(0x7ff0a40d1820 0x7ff0a40c6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 -- 172.21.15.119:0/3169815683 >> v1:172.21.15.145:6789/0 conn(0x7ff0a40d0c60 legacy=0x7ff0a40d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 -- 172.21.15.119:0/3169815683 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a40ce450 con 0x7ff0a415ccd0 2024-09-13T16:33:59.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0aa03b700 1 --2- 172.21.15.119:0/3169815683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a415ccd0 0x7ff0a40d0720 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4073a20 tx=0x7ff094009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ca0f3d3634c7720 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0a9039700 1 -- 172.21.15.119:0/3169815683 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094011830 con 0x7ff0a415ccd0 2024-09-13T16:33:59.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0a9039700 1 -- 172.21.15.119:0/3169815683 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff094011990 con 0x7ff0a415ccd0 2024-09-13T16:33:59.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0a9039700 1 -- 172.21.15.119:0/3169815683 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff094011d20 con 0x7ff0a415ccd0 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- 172.21.15.119:0/3169815683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a415ccd0 msgr2=0x7ff0a40d0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/3169815683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a415ccd0 0x7ff0a40d0720 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4073a20 tx=0x7ff094009bf0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- 172.21.15.119:0/3169815683 shutdown_connections 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/3169815683 >> v2:172.21.15.145:3300/0 conn(0x7ff0a40d1820 0x7ff0a40c6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/3169815683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a415ccd0 0x7ff0a40d0720 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- 172.21.15.119:0/3169815683 >> 172.21.15.119:0/3169815683 conn(0x7ff0a40b4640 msgr2=0x7ff0a40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- 172.21.15.119:0/3169815683 shutdown_connections 2024-09-13T16:33:59.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.489+0000 7ff0ab03d700 1 -- 172.21.15.119:0/3169815683 wait complete. 2024-09-13T16:33:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 Processor -- start 2024-09-13T16:33:59.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 -- start start 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a40d0c60 0x7ff0a41f6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a40d1820 0x7ff0a40c8750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0a415ccd0 0x7ff0a40c8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a41f3150 con 0x7ff0a40d1820 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a41f2820 con 0x7ff0a40d0c60 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0ab03d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a41f2b00 con 0x7ff0a415ccd0 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0a983a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a40d1820 0x7ff0a40c8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0a983a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a40d1820 0x7ff0a40c8750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34458/0 (socket says 172.21.15.119:34458) 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.493+0000 7ff0a983a700 1 -- 172.21.15.119:0/2545344391 learned_addr learned my addr 172.21.15.119:0/2545344391 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:59.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0aa03b700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a40d0c60 0x7ff0a41f6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0aa83c700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0a415ccd0 0x7ff0a40c8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0a983a700 1 -- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0a415ccd0 msgr2=0x7ff0a40c8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0a983a700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0a415ccd0 0x7ff0a40c8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0a983a700 1 -- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a40d0c60 msgr2=0x7ff0a41f6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0a983a700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a40d0c60 0x7ff0a41f6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0a983a700 1 -- 172.21.15.119:0/2545344391 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff094009570 con 0x7ff0a40d1820 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0a983a700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a40d1820 0x7ff0a40c8750 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000c500 tx=0x7ff0a000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff09affd700 1 -- 172.21.15.119:0/2545344391 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a00052d0 con 0x7ff0a40d1820 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff09affd700 1 -- 172.21.15.119:0/2545344391 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a0005430 con 0x7ff0a40d1820 2024-09-13T16:33:59.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a41f2de0 con 0x7ff0a40d1820 2024-09-13T16:33:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff09affd700 1 -- 172.21.15.119:0/2545344391 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a00055a0 con 0x7ff0a40d1820 2024-09-13T16:33:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a41f6880 con 0x7ff0a40d1820 2024-09-13T16:33:59.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff09affd700 1 -- 172.21.15.119:0/2545344391 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff0a00036a0 con 0x7ff0a40d1820 2024-09-13T16:33:59.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a41f3300 con 0x7ff0a40d1820 2024-09-13T16:33:59.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff09affd700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff088066760 0x7ff088068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.497+0000 7ff09affd700 1 -- 172.21.15.119:0/2545344391 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff0a00835b0 con 0x7ff0a40d1820 2024-09-13T16:33:59.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.501+0000 7ff0aa03b700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff088066760 0x7ff088068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.501+0000 7ff0aa03b700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff088066760 0x7ff088068c20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff094000c00 tx=0x7ff094008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.509+0000 7ff09affd700 1 -- 172.21.15.119:0/2545344391 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff0a41f3300 con 0x7ff0a40d1820 2024-09-13T16:33:59.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.677+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff0a41f3290 con 0x7ff0a40d1820 2024-09-13T16:33:59.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.681+0000 7ff09affd700 1 -- 172.21.15.119:0/2545344391 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16789 (secure 0 0 0) 0x7ff0a005a030 con 0x7ff0a40d1820 2024-09-13T16:33:59.685 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:33:59.685 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":110,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:33:59.031881+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:33:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.681+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff088066760 msgr2=0x7ff088068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.681+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff088066760 0x7ff088068c20 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7ff094000c00 tx=0x7ff094008030 comp rx=0 tx=0).stop 2024-09-13T16:33:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.681+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a40d1820 msgr2=0x7ff0a40c8750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.681+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a40d1820 0x7ff0a40c8750 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7ff0a000c500 tx=0x7ff0a000c810 comp rx=0 tx=0).stop 2024-09-13T16:33:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 shutdown_connections 2024-09-13T16:33:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff088066760 0x7ff088068c20 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0a415ccd0 0x7ff0a40c8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0a40d1820 0x7ff0a40c8750 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 --2- 172.21.15.119:0/2545344391 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0a40d0c60 0x7ff0a41f6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 >> 172.21.15.119:0/2545344391 conn(0x7ff0a40b4640 msgr2=0x7ff0a40c7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 shutdown_connections 2024-09-13T16:33:59.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.685+0000 7ff0ab03d700 1 -- 172.21.15.119:0/2545344391 wait complete. 2024-09-13T16:33:59.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:33:59.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee2dca700 1 Processor -- start 2024-09-13T16:33:59.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee2dca700 1 -- start start 2024-09-13T16:33:59.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee2dca700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbedc06ee00 0x7fbedc159be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee2dca700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc0b3c70 0x7fbedc0b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee2dca700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbedc06f3a0 con 0x7fbedc159800 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee2dca700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbedc0b2a80 con 0x7fbedc06ee00 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee2dca700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc0b2d60 con 0x7fbedc0b3c70 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee15c7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbedc06ee00 0x7fbedc159be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee15c7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbedc06ee00 0x7fbedc159be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34470/0 (socket says 172.21.15.119:34470) 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee15c7700 1 -- 172.21.15.119:0/2359695846 learned_addr learned my addr 172.21.15.119:0/2359695846 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.813+0000 7fbee25c9700 1 --2- 172.21.15.119:0/2359695846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc0b3c70 0x7fbedc0b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee15c7700 1 -- 172.21.15.119:0/2359695846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc0b3c70 msgr2=0x7fbedc0b40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:33:59.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee15c7700 1 --2- 172.21.15.119:0/2359695846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc0b3c70 0x7fbedc0b40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee15c7700 1 -- 172.21.15.119:0/2359695846 >> v1:172.21.15.145:6789/0 conn(0x7fbedc159800 legacy=0x7fbedc0b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee15c7700 1 -- 172.21.15.119:0/2359695846 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbedc0b3040 con 0x7fbedc06ee00 2024-09-13T16:33:59.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee15c7700 1 --2- 172.21.15.119:0/2359695846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc159be0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009910 tx=0x7fbecc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be47fa0d55c600c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee0dc6700 1 -- 172.21.15.119:0/2359695846 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc011830 con 0x7fbedc06ee00 2024-09-13T16:33:59.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee0dc6700 1 -- 172.21.15.119:0/2359695846 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbecc011990 con 0x7fbedc06ee00 2024-09-13T16:33:59.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee0dc6700 1 -- 172.21.15.119:0/2359695846 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbecc011d40 con 0x7fbedc06ee00 2024-09-13T16:33:59.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- 172.21.15.119:0/2359695846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 msgr2=0x7fbedc159be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 --2- 172.21.15.119:0/2359695846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc159be0 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009910 tx=0x7fbecc009c20 comp rx=0 tx=0).stop 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- 172.21.15.119:0/2359695846 shutdown_connections 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 --2- 172.21.15.119:0/2359695846 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc0b3c70 0x7fbedc0b40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 --2- 172.21.15.119:0/2359695846 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc159be0 unknown :-1 s=CLOSED pgs=1561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- 172.21.15.119:0/2359695846 >> 172.21.15.119:0/2359695846 conn(0x7fbedc0b15e0 msgr2=0x7fbedc0b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- 172.21.15.119:0/2359695846 shutdown_connections 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- 172.21.15.119:0/2359695846 wait complete. 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 Processor -- start 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- start start 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbedc0b3c70 0x7fbedc0cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc159800 0x7fbedc0ce080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc0a96d0 con 0x7fbedc06ee00 2024-09-13T16:33:59.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc152b60 con 0x7fbedc0b3c70 2024-09-13T16:33:59.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee2dca700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbedc0a8e30 con 0x7fbedc159800 2024-09-13T16:33:59.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee1dc8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee1dc8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc0a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34480/0 (socket says 172.21.15.119:34480) 2024-09-13T16:33:59.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.817+0000 7fbee1dc8700 1 -- 172.21.15.119:0/1476294004 learned_addr learned my addr 172.21.15.119:0/1476294004 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee25c9700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc159800 0x7fbedc0ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee15c7700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbedc0b3c70 0x7fbedc0cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee1dc8700 1 -- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc159800 msgr2=0x7fbedc0ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee1dc8700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc159800 0x7fbedc0ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee1dc8700 1 -- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbedc0b3c70 msgr2=0x7fbedc0cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee1dc8700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbedc0b3c70 0x7fbedc0cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee1dc8700 1 -- 172.21.15.119:0/1476294004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbecc009570 con 0x7fbedc06ee00 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee15c7700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbedc0b3c70 0x7fbedc0cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee25c9700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc159800 0x7fbedc0ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:33:59.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee1dc8700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc0a84b0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbed800c500 tx=0x7fbed800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbed27fc700 1 -- 172.21.15.119:0/1476294004 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80052d0 con 0x7fbedc06ee00 2024-09-13T16:33:59.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbed27fc700 1 -- 172.21.15.119:0/1476294004 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbed8005430 con 0x7fbedc06ee00 2024-09-13T16:33:59.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbedc0a9110 con 0x7fbedc06ee00 2024-09-13T16:33:59.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbed27fc700 1 -- 172.21.15.119:0/1476294004 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbed80055a0 con 0x7fbedc06ee00 2024-09-13T16:33:59.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbedc152d70 con 0x7fbedc06ee00 2024-09-13T16:33:59.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbedc0b4690 con 0x7fbedc06ee00 2024-09-13T16:33:59.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbed27fc700 1 -- 172.21.15.119:0/1476294004 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbed80038e0 con 0x7fbedc06ee00 2024-09-13T16:33:59.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbed27fc700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbec0066730 0x7fbec0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:33:59.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbed27fc700 1 -- 172.21.15.119:0/1476294004 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbed80836f0 con 0x7fbedc06ee00 2024-09-13T16:33:59.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.821+0000 7fbee15c7700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbec0066730 0x7fbec0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:33:59.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.825+0000 7fbee15c7700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbec0066730 0x7fbec0068bf0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009910 tx=0x7fbecc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:33:59.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:33:59.829+0000 7fbed27fc700 1 -- 172.21.15.119:0/1476294004 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbed8050930 con 0x7fbedc06ee00 2024-09-13T16:33:59.948 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:33:59.945+0000 7f339e892700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:34:00.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:00.037+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fbedc0b1c80 con 0x7fbedc06ee00 2024-09-13T16:34:01.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.041+0000 7fbed27fc700 1 -- 172.21.15.119:0/1476294004 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v75) v1 ==== 118+0+0 (secure 0 0 0) 0x7fbed8053f50 con 0x7fbedc06ee00 2024-09-13T16:34:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbec0066730 msgr2=0x7fbec0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbec0066730 0x7fbec0068bf0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fbecc009910 tx=0x7fbecc008030 comp rx=0 tx=0).stop 2024-09-13T16:34:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 msgr2=0x7fbedc0a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc0a84b0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7fbed800c500 tx=0x7fbed800c8c0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 shutdown_connections 2024-09-13T16:34:01.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbec0066730 0x7fbec0068bf0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbedc159800 0x7fbedc0ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbedc0b3c70 0x7fbedc0cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 --2- 172.21.15.119:0/1476294004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbedc06ee00 0x7fbedc0a84b0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 >> 172.21.15.119:0/1476294004 conn(0x7fbedc0b15e0 msgr2=0x7fbedc155be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:01.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 shutdown_connections 2024-09-13T16:34:01.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.045+0000 7fbee2dca700 1 -- 172.21.15.119:0/1476294004 wait complete. 2024-09-13T16:34:01.066 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T16:34:01.066 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:34:01.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2badf3d700 1 Processor -- start 2024-09-13T16:34:01.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2badf3d700 1 -- start start 2024-09-13T16:34:01.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2badf3d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2ba815cc00 0x7f2ba80d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:01.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2badf3d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2ba80d17e0 0x7f2ba80b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:01.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2badf3d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ba80cdb00 con 0x7f2ba80d0c20 2024-09-13T16:34:01.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2badf3d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba80cdde0 con 0x7f2ba815cc00 2024-09-13T16:34:01.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2badf3d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ba80ce0c0 con 0x7f2ba80d17e0 2024-09-13T16:34:01.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2ba815cc00 0x7f2ba80d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2ba815cc00 0x7f2ba80d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34482/0 (socket says 172.21.15.119:34482) 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1355489578 learned_addr learned my addr 172.21.15.119:0/1355489578 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bad73c700 1 --2- 172.21.15.119:0/1355489578 >> v2:172.21.15.145:3300/0 conn(0x7f2ba80d17e0 0x7f2ba80b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1355489578 >> v2:172.21.15.145:3300/0 conn(0x7f2ba80d17e0 msgr2=0x7f2ba80b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 --2- 172.21.15.119:0/1355489578 >> v2:172.21.15.145:3300/0 conn(0x7f2ba80d17e0 0x7f2ba80b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1355489578 >> v1:172.21.15.145:6789/0 conn(0x7f2ba80d0c20 legacy=0x7f2ba80d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1355489578 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba80ce3a0 con 0x7f2ba815cc00 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2bacf3b700 1 --2- 172.21.15.119:0/1355489578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba815cc00 0x7f2ba80d06e0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80a0a20 tx=0x7f2b98009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11fb4aa649d71bba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2ba77fe700 1 -- 172.21.15.119:0/1355489578 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98004230 con 0x7f2ba815cc00 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2ba77fe700 1 -- 172.21.15.119:0/1355489578 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b98004390 con 0x7f2ba815cc00 2024-09-13T16:34:01.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.189+0000 7f2ba77fe700 1 -- 172.21.15.119:0/1355489578 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980036c0 con 0x7f2ba815cc00 2024-09-13T16:34:01.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- 172.21.15.119:0/1355489578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba815cc00 msgr2=0x7f2ba80d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:01.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1355489578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba815cc00 0x7f2ba80d06e0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80a0a20 tx=0x7f2b98009bf0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- 172.21.15.119:0/1355489578 shutdown_connections 2024-09-13T16:34:01.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1355489578 >> v2:172.21.15.145:3300/0 conn(0x7f2ba80d17e0 0x7f2ba80b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1355489578 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba815cc00 0x7f2ba80d06e0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- 172.21.15.119:0/1355489578 >> 172.21.15.119:0/1355489578 conn(0x7f2ba808c080 msgr2=0x7f2ba80d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:01.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- 172.21.15.119:0/1355489578 shutdown_connections 2024-09-13T16:34:01.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- 172.21.15.119:0/1355489578 wait complete. 2024-09-13T16:34:01.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 Processor -- start 2024-09-13T16:34:01.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- start start 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba80d0c20 0x7f2ba8150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ba80d17e0 0x7f2ba815d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ba81f3a30 0x7f2ba815d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba81513a0 con 0x7f2ba80d0c20 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba815e190 con 0x7f2ba80d17e0 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2badf3d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ba8158a70 con 0x7f2ba81f3a30 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2bacf3b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba80d0c20 0x7f2ba8150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:01.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2bacf3b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba80d0c20 0x7f2ba8150e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34498/0 (socket says 172.21.15.119:34498) 2024-09-13T16:34:01.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2bad73c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ba81f3a30 0x7f2ba815d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:01.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1681625067 learned_addr learned my addr 172.21.15.119:0/1681625067 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:01.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.193+0000 7f2ba7fff700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ba80d17e0 0x7f2ba815d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:01.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ba81f3a30 msgr2=0x7f2ba815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:01.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2bacf3b700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ba81f3a30 0x7f2ba815d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ba80d17e0 msgr2=0x7f2ba815d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:01.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2bacf3b700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ba80d17e0 0x7f2ba815d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:01.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2bacf3b700 1 -- 172.21.15.119:0/1681625067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b98009570 con 0x7f2ba80d0c20 2024-09-13T16:34:01.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2ba7fff700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ba80d17e0 0x7f2ba815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:01.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2bacf3b700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba80d0c20 0x7f2ba8150e60 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f2b98004000 tx=0x7f2b98004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:01.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2ba57fa700 1 -- 172.21.15.119:0/1681625067 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b980036e0 con 0x7f2ba80d0c20 2024-09-13T16:34:01.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2ba57fa700 1 -- 172.21.15.119:0/1681625067 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b9801f070 con 0x7f2ba80d0c20 2024-09-13T16:34:01.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ba815deb0 con 0x7f2ba80d0c20 2024-09-13T16:34:01.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2ba57fa700 1 -- 172.21.15.119:0/1681625067 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b9800b660 con 0x7f2ba80d0c20 2024-09-13T16:34:01.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ba81f3230 con 0x7f2ba80d0c20 2024-09-13T16:34:01.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.197+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ba81f2db0 con 0x7f2ba80d0c20 2024-09-13T16:34:01.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.201+0000 7f2ba57fa700 1 -- 172.21.15.119:0/1681625067 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2b98021030 con 0x7f2ba80d0c20 2024-09-13T16:34:01.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.201+0000 7f2ba57fa700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2b88066730 0x7f2b88068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:01.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.201+0000 7f2ba57fa700 1 -- 172.21.15.119:0/1681625067 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2b980856c0 con 0x7f2ba80d0c20 2024-09-13T16:34:01.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.201+0000 7f2ba7fff700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2b88066730 0x7f2b88068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:01.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.201+0000 7f2ba7fff700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2b88066730 0x7f2b88068bf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80cd0f0 tx=0x7f2b90007ee0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:01.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.205+0000 7f2ba57fa700 1 -- 172.21.15.119:0/1681625067 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2ba81f2db0 con 0x7f2ba80d0c20 2024-09-13T16:34:01.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:01.409+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2ba81f2db0 con 0x7f2ba80d0c20 2024-09-13T16:34:02.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.053+0000 7f2ba57fa700 1 -- 172.21.15.119:0/1681625067 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2ba81f2db0 con 0x7f2ba80d0c20 2024-09-13T16:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2b88066730 msgr2=0x7f2b88068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2b88066730 0x7f2b88068bf0 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80cd0f0 tx=0x7f2b90007ee0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba80d0c20 msgr2=0x7f2ba8150e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba80d0c20 0x7f2ba8150e60 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f2b98004000 tx=0x7f2b98004770 comp rx=0 tx=0).stop 2024-09-13T16:34:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 shutdown_connections 2024-09-13T16:34:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2b88066730 0x7f2b88068bf0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2ba81f3a30 0x7f2ba815d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2ba80d17e0 0x7f2ba815d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 --2- 172.21.15.119:0/1681625067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2ba80d0c20 0x7f2ba8150e60 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 >> 172.21.15.119:0/1681625067 conn(0x7f2ba808c080 msgr2=0x7f2ba80b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:02.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 shutdown_connections 2024-09-13T16:34:02.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.057+0000 7f2badf3d700 1 -- 172.21.15.119:0/1681625067 wait complete. 2024-09-13T16:34:02.077 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:34:02.078 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:34:02.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0069f1f700 1 Processor -- start 2024-09-13T16:34:02.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0069f1f700 1 -- start start 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0069f1f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f00640b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0069f1f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f00640707b0 0x7f006406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0069f1f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f006406d180 con 0x7f006406b8d0 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0069f1f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f006406c360 con 0x7f00640707b0 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0069f1f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006406bd50 con 0x7f00640703d0 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0063fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f00640707b0 0x7f006406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f006971e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f006406b8d0 0x7f006406d610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:59224/0 (socket says 172.21.15.119:59224) 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f006971e700 1 -- 172.21.15.119:0/2252364160 learned_addr learned my addr 172.21.15.119:0/2252364160 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0068f1d700 1 --2- 172.21.15.119:0/2252364160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f00640b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0063fff700 1 -- 172.21.15.119:0/2252364160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 msgr2=0x7f00640b1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:02.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0063fff700 1 --2- 172.21.15.119:0/2252364160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f00640b1fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0063fff700 1 -- 172.21.15.119:0/2252364160 >> v1:172.21.15.119:6789/0 conn(0x7f006406b8d0 legacy=0x7f006406d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f00637fe700 1 -- 172.21.15.119:0/2252364160 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 713207631 0 0) 0x7f006406d180 con 0x7f006406b8d0 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0063fff700 1 -- 172.21.15.119:0/2252364160 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f006406c570 con 0x7f00640707b0 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0068f1d700 1 --2- 172.21.15.119:0/2252364160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f00640b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.193+0000 7f0063fff700 1 --2- 172.21.15.119:0/2252364160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00640707b0 0x7f006406b390 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f0050009850 tx=0x7f0050009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd28779602e7b64a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f00637fe700 1 -- 172.21.15.119:0/2252364160 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050011830 con 0x7f00640707b0 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f00637fe700 1 -- 172.21.15.119:0/2252364160 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0050011990 con 0x7f00640707b0 2024-09-13T16:34:02.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f00637fe700 1 -- 172.21.15.119:0/2252364160 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0050011d20 con 0x7f00640707b0 2024-09-13T16:34:02.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- 172.21.15.119:0/2252364160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00640707b0 msgr2=0x7f006406b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 --2- 172.21.15.119:0/2252364160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00640707b0 0x7f006406b390 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7f0050009850 tx=0x7f0050009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:02.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- 172.21.15.119:0/2252364160 shutdown_connections 2024-09-13T16:34:02.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 --2- 172.21.15.119:0/2252364160 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f00640707b0 0x7f006406b390 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 --2- 172.21.15.119:0/2252364160 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f00640b1fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- 172.21.15.119:0/2252364160 >> 172.21.15.119:0/2252364160 conn(0x7f00640ae750 msgr2=0x7f00640b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:02.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- 172.21.15.119:0/2252364160 shutdown_connections 2024-09-13T16:34:02.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- 172.21.15.119:0/2252364160 wait complete. 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 Processor -- start 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- start start 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f006409cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00640707b0 0x7f00640c3f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006414ab10 0x7f00640abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00640c4480 con 0x7f006414ab10 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f006414b360 con 0x7f00640707b0 2024-09-13T16:34:02.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.197+0000 7f0069f1f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f006414b640 con 0x7f00640703d0 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006414ab10 0x7f00640abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006414ab10 0x7f00640abfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:34520/0 (socket says 172.21.15.119:34520) 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 -- 172.21.15.119:0/561120622 learned_addr learned my addr 172.21.15.119:0/561120622 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f0068f1d700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f006409cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f0063fff700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00640707b0 0x7f00640c3f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 -- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 msgr2=0x7f006409cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f006409cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 -- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00640707b0 msgr2=0x7f00640c3f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00640707b0 0x7f00640c3f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 -- 172.21.15.119:0/561120622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00500094b0 con 0x7f006414ab10 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f006971e700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006414ab10 0x7f00640abfb0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f005c00cb60 tx=0x7f005c00ce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f00617fa700 1 -- 172.21.15.119:0/561120622 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c00a9e0 con 0x7f006414ab10 2024-09-13T16:34:02.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f00617fa700 1 -- 172.21.15.119:0/561120622 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f005c00ab40 con 0x7f006414ab10 2024-09-13T16:34:02.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00641489a0 con 0x7f006414ab10 2024-09-13T16:34:02.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f00617fa700 1 -- 172.21.15.119:0/561120622 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f005c0070d0 con 0x7f006414ab10 2024-09-13T16:34:02.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f006414afc0 con 0x7f006414ab10 2024-09-13T16:34:02.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0064148ec0 con 0x7f006414ab10 2024-09-13T16:34:02.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f00617fa700 1 -- 172.21.15.119:0/561120622 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f005c007230 con 0x7f006414ab10 2024-09-13T16:34:02.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f00617fa700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.201+0000 7f00617fa700 1 -- 172.21.15.119:0/561120622 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f005c083ee0 con 0x7f006414ab10 2024-09-13T16:34:02.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.205+0000 7f0068f1d700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.205+0000 7f0068f1d700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f005800c960 tx=0x7f0058008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.209+0000 7f00617fa700 1 -- 172.21.15.119:0/561120622 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f005c0511a0 con 0x7f006414ab10 2024-09-13T16:34:02.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.413+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f006413fcf0 con 0x7f006414ab10 2024-09-13T16:34:02.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f00617fa700 1 -- 172.21.15.119:0/561120622 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f005c018090 con 0x7f006414ab10 2024-09-13T16:34:02.419 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:02.419 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":76,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:34:02.039855+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8829":{"gid":8829,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3700300033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3700300033},{"type":"v1","addr":"172.21.15.119:6835","nonce":3700300033}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:34:02.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 msgr2=0x7f0048068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f005800c960 tx=0x7f0058008040 comp rx=0 tx=0).stop 2024-09-13T16:34:02.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006414ab10 msgr2=0x7f00640abfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006414ab10 0x7f00640abfb0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f005c00cb60 tx=0x7f005c00ce70 comp rx=0 tx=0).stop 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 shutdown_connections 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0048066730 0x7f0048068bf0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f006414ab10 0x7f00640abfb0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f00640707b0 0x7f00640c3f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 --2- 172.21.15.119:0/561120622 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f00640703d0 0x7f006409cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 >> 172.21.15.119:0/561120622 conn(0x7f006406b8d0 msgr2=0x7f00640b28f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 shutdown_connections 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.417+0000 7f0069f1f700 1 -- 172.21.15.119:0/561120622 wait complete. 2024-09-13T16:34:02.423 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 76 2024-09-13T16:34:02.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:34:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 Processor -- start 2024-09-13T16:34:02.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- start start 2024-09-13T16:34:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe25c1440d0 0x7fe25c15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe25c06bd60 con 0x7fe25c1444b0 2024-09-13T16:34:02.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25c0cc4c0 con 0x7fe25c1440d0 2024-09-13T16:34:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c15bf10 con 0x7fe25c0cbb10 2024-09-13T16:34:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe25c1440d0 0x7fe25c15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe25c1440d0 0x7fe25c15b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36468/0 (socket says 172.21.15.119:36468) 2024-09-13T16:34:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe2608a0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 -- 172.21.15.119:0/1181940866 learned_addr learned my addr 172.21.15.119:0/1181940866 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 -- 172.21.15.119:0/1181940866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 msgr2=0x7fe25c0cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 --2- 172.21.15.119:0/1181940866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 -- 172.21.15.119:0/1181940866 >> v1:172.21.15.145:6789/0 conn(0x7fe25c1444b0 legacy=0x7fe25c06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 -- 172.21.15.119:0/1181940866 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe25c06d790 con 0x7fe25c1440d0 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25bfff700 1 --2- 172.21.15.119:0/1181940866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c15b9d0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009910 tx=0x7fe24c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f57b55c6928f2cd1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25b7fe700 1 -- 172.21.15.119:0/1181940866 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c011830 con 0x7fe25c1440d0 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25b7fe700 1 -- 172.21.15.119:0/1181940866 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24c011990 con 0x7fe25c1440d0 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe25b7fe700 1 -- 172.21.15.119:0/1181940866 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24c011d40 con 0x7fe25c1440d0 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- 172.21.15.119:0/1181940866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 msgr2=0x7fe25c15b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 --2- 172.21.15.119:0/1181940866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c15b9d0 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 crypto rx=0x7fe24c009910 tx=0x7fe24c009c20 comp rx=0 tx=0).stop 2024-09-13T16:34:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- 172.21.15.119:0/1181940866 shutdown_connections 2024-09-13T16:34:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 --2- 172.21.15.119:0/1181940866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 --2- 172.21.15.119:0/1181940866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c15b9d0 unknown :-1 s=CLOSED pgs=1568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- 172.21.15.119:0/1181940866 >> 172.21.15.119:0/1181940866 conn(0x7fe25c06a350 msgr2=0x7fe25c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- 172.21.15.119:0/1181940866 shutdown_connections 2024-09-13T16:34:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.545+0000 7fe262303700 1 -- 172.21.15.119:0/1181940866 wait complete. 2024-09-13T16:34:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.549+0000 7fe262303700 1 Processor -- start 2024-09-13T16:34:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.549+0000 7fe262303700 1 -- start start 2024-09-13T16:34:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c1efb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe25c1444b0 0x7fe25c141070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c13c970 con 0x7fe25c1440d0 2024-09-13T16:34:02.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c13cbc0 con 0x7fe25c1444b0 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25c13cea0 con 0x7fe25c0cbb10 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c1efb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c1efb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36476/0 (socket says 172.21.15.119:36476) 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 -- 172.21.15.119:0/2772255540 learned_addr learned my addr 172.21.15.119:0/2772255540 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe25bfff700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe2608a0700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe25c1444b0 0x7fe25c141070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 -- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 msgr2=0x7fe25c0a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 -- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe25c1444b0 msgr2=0x7fe25c141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe25c1444b0 0x7fe25c141070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 -- 172.21.15.119:0/2772255540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe24c009570 con 0x7fe25c1440d0 2024-09-13T16:34:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe253fff700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c1efb90 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c4d0 tx=0x7fe24400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe2597fa700 1 -- 172.21.15.119:0/2772255540 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2440052d0 con 0x7fe25c1440d0 2024-09-13T16:34:02.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe2597fa700 1 -- 172.21.15.119:0/2772255540 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe244005430 con 0x7fe25c1440d0 2024-09-13T16:34:02.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe25c13d180 con 0x7fe25c1440d0 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe2597fa700 1 -- 172.21.15.119:0/2772255540 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2440055a0 con 0x7fe25c1440d0 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe25c13d740 con 0x7fe25c1440d0 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.553+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe25c13d900 con 0x7fe25c1440d0 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.557+0000 7fe2597fa700 1 -- 172.21.15.119:0/2772255540 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe2440038c0 con 0x7fe25c1440d0 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.557+0000 7fe2597fa700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe23c066730 0x7fe23c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.557+0000 7fe2597fa700 1 -- 172.21.15.119:0/2772255540 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe244083610 con 0x7fe25c1440d0 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.557+0000 7fe25bfff700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe23c066730 0x7fe23c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.557+0000 7fe25bfff700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe23c066730 0x7fe23c068bf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.561+0000 7fe2597fa700 1 -- 172.21.15.119:0/2772255540 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe2440507a0 con 0x7fe25c1440d0 2024-09-13T16:34:02.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.741+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe25c067760 con 0x7fe25c1440d0 2024-09-13T16:34:02.749 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.745+0000 7fe2597fa700 1 -- 172.21.15.119:0/2772255540 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16776 (secure 0 0 0) 0x7fe244053dc0 con 0x7fe25c1440d0 2024-09-13T16:34:02.750 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:02.750 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":111,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:34:00.035557+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:34:02.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe23c066730 msgr2=0x7fe23c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe23c066730 0x7fe23c068bf0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7fe24c000c00 tx=0x7fe24c008030 comp rx=0 tx=0).stop 2024-09-13T16:34:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 msgr2=0x7fe25c1efb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c1efb90 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7fe24400c4d0 tx=0x7fe24400c7e0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 shutdown_connections 2024-09-13T16:34:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe23c066730 0x7fe23c068bf0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe25c1444b0 0x7fe25c141070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe25c1440d0 0x7fe25c1efb90 unknown :-1 s=CLOSED pgs=1569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 --2- 172.21.15.119:0/2772255540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe25c0cbb10 0x7fe25c0a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 >> 172.21.15.119:0/2772255540 conn(0x7fe25c06a350 msgr2=0x7fe25c0b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 shutdown_connections 2024-09-13T16:34:02.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.749+0000 7fe262303700 1 -- 172.21.15.119:0/2772255540 wait complete. 2024-09-13T16:34:02.764 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:34:02.765 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:34:02.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 Processor -- start 2024-09-13T16:34:02.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- start start 2024-09-13T16:34:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8ad806e8c0 0x7f8ad80cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8ad81613a0 0x7f8ad806eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ad80afbb0 con 0x7f8ad806e450 2024-09-13T16:34:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad80afe90 con 0x7f8ad806e8c0 2024-09-13T16:34:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ad80b33d0 con 0x7f8ad81613a0 2024-09-13T16:34:02.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8ad806e8c0 0x7f8ad80cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8ad806e8c0 0x7f8ad80cb090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36478/0 (socket says 172.21.15.119:36478) 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 -- 172.21.15.119:0/3907588811 learned_addr learned my addr 172.21.15.119:0/3907588811 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad67fc700 1 --2- 172.21.15.119:0/3907588811 >> v2:172.21.15.145:3300/0 conn(0x7f8ad81613a0 0x7f8ad806eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 -- 172.21.15.119:0/3907588811 >> v2:172.21.15.145:3300/0 conn(0x7f8ad81613a0 msgr2=0x7f8ad806eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 --2- 172.21.15.119:0/3907588811 >> v2:172.21.15.145:3300/0 conn(0x7f8ad81613a0 0x7f8ad806eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 -- 172.21.15.119:0/3907588811 >> v1:172.21.15.145:6789/0 conn(0x7f8ad806e450 legacy=0x7f8ad80cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 -- 172.21.15.119:0/3907588811 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad80ceee0 con 0x7f8ad806e8c0 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad6ffd700 1 --2- 172.21.15.119:0/3907588811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad806e8c0 0x7f8ad80cb090 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80b6cd0 tx=0x7f8ac0009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d413955a5319e5b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad5ffb700 1 -- 172.21.15.119:0/3907588811 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac000b8a0 con 0x7f8ad806e8c0 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad5ffb700 1 -- 172.21.15.119:0/3907588811 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac000ba00 con 0x7f8ad806e8c0 2024-09-13T16:34:02.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8ad5ffb700 1 -- 172.21.15.119:0/3907588811 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac000bd90 con 0x7f8ad806e8c0 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- 172.21.15.119:0/3907588811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad806e8c0 msgr2=0x7f8ad80cb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 --2- 172.21.15.119:0/3907588811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad806e8c0 0x7f8ad80cb090 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f8ad80b6cd0 tx=0x7f8ac0009bf0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- 172.21.15.119:0/3907588811 shutdown_connections 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 --2- 172.21.15.119:0/3907588811 >> v2:172.21.15.145:3300/0 conn(0x7f8ad81613a0 0x7f8ad806eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 --2- 172.21.15.119:0/3907588811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad806e8c0 0x7f8ad80cb090 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- 172.21.15.119:0/3907588811 >> 172.21.15.119:0/3907588811 conn(0x7f8ad80b2710 msgr2=0x7f8ad80b2b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- 172.21.15.119:0/3907588811 shutdown_connections 2024-09-13T16:34:02.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.877+0000 7f8add56a700 1 -- 172.21.15.119:0/3907588811 wait complete. 2024-09-13T16:34:02.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.881+0000 7f8add56a700 1 Processor -- start 2024-09-13T16:34:02.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.881+0000 7f8add56a700 1 -- start start 2024-09-13T16:34:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad806e450 0x7f8ad80b3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad806e8c0 0x7f8ad8154eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad81613a0 0x7f8ad81f7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81553f0 con 0x7f8ad81613a0 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81f80a0 con 0x7f8ad806e8c0 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad8157140 con 0x7f8ad806e450 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad81613a0 0x7f8ad81f7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad81613a0 0x7f8ad81f7b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36494/0 (socket says 172.21.15.119:36494) 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 -- 172.21.15.119:0/2688916995 learned_addr learned my addr 172.21.15.119:0/2688916995 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad6ffd700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad806e450 0x7f8ad80b3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad67fc700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad806e8c0 0x7f8ad8154eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 -- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad806e450 msgr2=0x7f8ad80b3be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad806e450 0x7f8ad80b3be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 -- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad806e8c0 msgr2=0x7f8ad8154eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad806e8c0 0x7f8ad8154eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 -- 172.21.15.119:0/2688916995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac0009570 con 0x7f8ad81613a0 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8ad77fe700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad81613a0 0x7f8ad81f7b60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800c4d0 tx=0x7f8ac800c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8acf7fe700 1 -- 172.21.15.119:0/2688916995 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80052d0 con 0x7f8ad81613a0 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8acf7fe700 1 -- 172.21.15.119:0/2688916995 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac8005430 con 0x7f8ad81613a0 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad81573e0 con 0x7f8ad81613a0 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8acf7fe700 1 -- 172.21.15.119:0/2688916995 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac80055a0 con 0x7f8ad81613a0 2024-09-13T16:34:02.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad81579a0 con 0x7f8ad81613a0 2024-09-13T16:34:02.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.885+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad80b0060 con 0x7f8ad81613a0 2024-09-13T16:34:02.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.889+0000 7f8acf7fe700 1 -- 172.21.15.119:0/2688916995 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8ac80036a0 con 0x7f8ad81613a0 2024-09-13T16:34:02.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.889+0000 7f8acf7fe700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ab8066760 0x7f8ab8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:02.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.889+0000 7f8acf7fe700 1 -- 172.21.15.119:0/2688916995 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8ac80834e0 con 0x7f8ad81613a0 2024-09-13T16:34:02.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.889+0000 7f8ad6ffd700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ab8066760 0x7f8ab8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:02.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.889+0000 7f8ad6ffd700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ab8066760 0x7f8ab8068c20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:02.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:02.897+0000 7f8acf7fe700 1 -- 172.21.15.119:0/2688916995 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8ac8050670 con 0x7f8ad81613a0 2024-09-13T16:34:03.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.065+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ad806fec0 con 0x7f8ab8066760 2024-09-13T16:34:03.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.081+0000 7f8acf7fe700 1 -- 172.21.15.119:0/2688916995 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288585 (secure 0 0 0) 0x7f8ad806fec0 con 0x7f8ab8066760 2024-09-13T16:34:03.093 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ab8066760 msgr2=0x7f8ab8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ab8066760 0x7f8ab8068c20 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0000c00 tx=0x7f8ac0008030 comp rx=0 tx=0).stop 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad81613a0 msgr2=0x7f8ad81f7b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad81613a0 0x7f8ad81f7b60 secure :-1 s=READY pgs=1571 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800c4d0 tx=0x7f8ac800c7e0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 shutdown_connections 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ab8066760 0x7f8ab8068c20 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8ad81613a0 0x7f8ad81f7b60 unknown :-1 s=CLOSED pgs=1571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8ad806e8c0 0x7f8ad8154eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 --2- 172.21.15.119:0/2688916995 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8ad806e450 0x7f8ad80b3be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 >> 172.21.15.119:0/2688916995 conn(0x7f8ad80b2710 msgr2=0x7f8ad80d03d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:03.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 shutdown_connections 2024-09-13T16:34:03.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.089+0000 7f8add56a700 1 -- 172.21.15.119:0/2688916995 wait complete. 2024-09-13T16:34:03.097 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:34:03.109 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-13T16:34:01.608232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260828,"kb_used_data":11860,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754713892,"statfs":{"total":773094113280,"available":772827025408,"internally_reserved":0,"allocated":12144640,"data_stored":6197752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001124"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070114+0000","last_change":"2024-09-13T16:33:59.066124+0000","last_active":"2024-09-13T16:33:59.070114+0000","last_peered":"2024-09-13T16:33:59.070114+0000","last_clean":"2024-09-13T16:33:59.070114+0000","last_became_active":"2024-09-13T16:33:59.064047+0000","last_became_peered":"2024-09-13T16:33:59.064047+0000","last_unstale":"2024-09-13T16:33:59.070114+0000","last_undegraded":"2024-09-13T16:33:59.070114+0000","last_fullsized":"2024-09-13T16:33:59.070114+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:58.249905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.068005+0000","last_change":"2024-09-13T16:33:59.060600+0000","last_active":"2024-09-13T16:33:59.068005+0000","last_peered":"2024-09-13T16:33:59.068005+0000","last_clean":"2024-09-13T16:33:59.068005+0000","last_became_active":"2024-09-13T16:33:59.060336+0000","last_became_peered":"2024-09-13T16:33:59.060336+0000","last_unstale":"2024-09-13T16:33:59.068005+0000","last_undegraded":"2024-09-13T16:33:59.068005+0000","last_fullsized":"2024-09-13T16:33:59.068005+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:29.912089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070399+0000","last_change":"2024-09-13T16:33:59.066828+0000","last_active":"2024-09-13T16:33:59.070399+0000","last_peered":"2024-09-13T16:33:59.070399+0000","last_clean":"2024-09-13T16:33:59.070399+0000","last_became_active":"2024-09-13T16:33:59.066566+0000","last_became_peered":"2024-09-13T16:33:59.066566+0000","last_unstale":"2024-09-13T16:33:59.070399+0000","last_undegraded":"2024-09-13T16:33:59.070399+0000","last_fullsized":"2024-09-13T16:33:59.070399+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:47:04.673288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048421+0000","last_change":"2024-09-13T16:33:59.056833+0000","last_active":"2024-09-13T16:34:00.048421+0000","last_peered":"2024-09-13T16:34:00.048421+0000","last_clean":"2024-09-13T16:34:00.048421+0000","last_became_active":"2024-09-13T16:33:59.056582+0000","last_became_peered":"2024-09-13T16:33:59.056582+0000","last_unstale":"2024-09-13T16:34:00.048421+0000","last_undegraded":"2024-09-13T16:34:00.048421+0000","last_fullsized":"2024-09-13T16:34:00.048421+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:18.392057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073463+0000","last_change":"2024-09-13T16:33:59.068592+0000","last_active":"2024-09-13T16:33:59.073463+0000","last_peered":"2024-09-13T16:33:59.073463+0000","last_clean":"2024-09-13T16:33:59.073463+0000","last_became_active":"2024-09-13T16:33:59.064783+0000","last_became_peered":"2024-09-13T16:33:59.064783+0000","last_unstale":"2024-09-13T16:33:59.073463+0000","last_undegraded":"2024-09-13T16:33:59.073463+0000","last_fullsized":"2024-09-13T16:33:59.073463+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:29.912089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.067499+0000","last_change":"2024-09-13T16:33:59.060319+0000","last_active":"2024-09-13T16:33:59.067499+0000","last_peered":"2024-09-13T16:33:59.067499+0000","last_clean":"2024-09-13T16:33:59.067499+0000","last_became_active":"2024-09-13T16:33:59.060048+0000","last_became_peered":"2024-09-13T16:33:59.060048+0000","last_unstale":"2024-09-13T16:33:59.067499+0000","last_undegraded":"2024-09-13T16:33:59.067499+0000","last_fullsized":"2024-09-13T16:33:59.067499+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:45.989345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.067402+0000","last_change":"2024-09-13T16:33:59.060299+0000","last_active":"2024-09-13T16:33:59.067402+0000","last_peered":"2024-09-13T16:33:59.067402+0000","last_clean":"2024-09-13T16:33:59.067402+0000","last_became_active":"2024-09-13T16:33:59.059999+0000","last_became_peered":"2024-09-13T16:33:59.059999+0000","last_unstale":"2024-09-13T16:33:59.067402+0000","last_undegraded":"2024-09-13T16:33:59.067402+0000","last_fullsized":"2024-09-13T16:33:59.067402+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:11.323458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356303+0000","last_change":"2024-09-13T16:33:59.059359+0000","last_active":"2024-09-13T16:34:00.356303+0000","last_peered":"2024-09-13T16:34:00.356303+0000","last_clean":"2024-09-13T16:34:00.356303+0000","last_became_active":"2024-09-13T16:33:59.059027+0000","last_became_peered":"2024-09-13T16:33:59.059027+0000","last_unstale":"2024-09-13T16:34:00.356303+0000","last_undegraded":"2024-09-13T16:34:00.356303+0000","last_fullsized":"2024-09-13T16:34:00.356303+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:23.865851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.052062+0000","last_change":"2024-09-13T16:33:59.048164+0000","last_active":"2024-09-13T16:33:59.052062+0000","last_peered":"2024-09-13T16:33:59.052062+0000","last_clean":"2024-09-13T16:33:59.052062+0000","last_became_active":"2024-09-13T16:33:59.047585+0000","last_became_peered":"2024-09-13T16:33:59.047585+0000","last_unstale":"2024-09-13T16:33:59.052062+0000","last_undegraded":"2024-09-13T16:33:59.052062+0000","last_fullsized":"2024-09-13T16:33:59.052062+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:24:39.261364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073329+0000","last_change":"2024-09-13T16:33:59.063750+0000","last_active":"2024-09-13T16:33:59.073329+0000","last_peered":"2024-09-13T16:33:59.073329+0000","last_clean":"2024-09-13T16:33:59.073329+0000","last_became_active":"2024-09-13T16:33:59.061925+0000","last_became_peered":"2024-09-13T16:33:59.061925+0000","last_unstale":"2024-09-13T16:33:59.073329+0000","last_undegraded":"2024-09-13T16:33:59.073329+0000","last_fullsized":"2024-09-13T16:33:59.073329+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:11.776642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250492+0000","last_change":"2024-09-13T16:33:59.068555+0000","last_active":"2024-09-13T16:34:00.250492+0000","last_peered":"2024-09-13T16:34:00.250492+0000","last_clean":"2024-09-13T16:34:00.250492+0000","last_became_active":"2024-09-13T16:33:59.068231+0000","last_became_peered":"2024-09-13T16:33:59.068231+0000","last_unstale":"2024-09-13T16:34:00.250492+0000","last_undegraded":"2024-09-13T16:34:00.250492+0000","last_fullsized":"2024-09-13T16:34:00.250492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:17.602566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049470+0000","last_change":"2024-09-13T16:33:59.060294+0000","last_active":"2024-09-13T16:34:00.049470+0000","last_peered":"2024-09-13T16:34:00.049470+0000","last_clean":"2024-09-13T16:34:00.049470+0000","last_became_active":"2024-09-13T16:33:59.059510+0000","last_became_peered":"2024-09-13T16:33:59.059510+0000","last_unstale":"2024-09-13T16:34:00.049470+0000","last_undegraded":"2024-09-13T16:34:00.049470+0000","last_fullsized":"2024-09-13T16:34:00.049470+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:30.844442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048226+0000","last_change":"2024-09-13T16:33:59.066489+0000","last_active":"2024-09-13T16:34:00.048226+0000","last_peered":"2024-09-13T16:34:00.048226+0000","last_clean":"2024-09-13T16:34:00.048226+0000","last_became_active":"2024-09-13T16:33:59.066256+0000","last_became_peered":"2024-09-13T16:33:59.066256+0000","last_unstale":"2024-09-13T16:34:00.048226+0000","last_undegraded":"2024-09-13T16:34:00.048226+0000","last_fullsized":"2024-09-13T16:34:00.048226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:27.994724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050426+0000","last_change":"2024-09-13T16:33:59.064257+0000","last_active":"2024-09-13T16:34:00.050426+0000","last_peered":"2024-09-13T16:34:00.050426+0000","last_clean":"2024-09-13T16:34:00.050426+0000","last_became_active":"2024-09-13T16:33:59.063906+0000","last_became_peered":"2024-09-13T16:33:59.063906+0000","last_unstale":"2024-09-13T16:34:00.050426+0000","last_undegraded":"2024-09-13T16:34:00.050426+0000","last_fullsized":"2024-09-13T16:34:00.050426+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:10.886581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050532+0000","last_change":"2024-09-13T16:33:59.069361+0000","last_active":"2024-09-13T16:34:00.050532+0000","last_peered":"2024-09-13T16:34:00.050532+0000","last_clean":"2024-09-13T16:34:00.050532+0000","last_became_active":"2024-09-13T16:33:59.069122+0000","last_became_peered":"2024-09-13T16:33:59.069122+0000","last_unstale":"2024-09-13T16:34:00.050532+0000","last_undegraded":"2024-09-13T16:34:00.050532+0000","last_fullsized":"2024-09-13T16:34:00.050532+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:17.902369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050362+0000","last_change":"2024-09-13T16:33:59.059847+0000","last_active":"2024-09-13T16:34:00.050362+0000","last_peered":"2024-09-13T16:34:00.050362+0000","last_clean":"2024-09-13T16:34:00.050362+0000","last_became_active":"2024-09-13T16:33:59.059546+0000","last_became_peered":"2024-09-13T16:33:59.059546+0000","last_unstale":"2024-09-13T16:34:00.050362+0000","last_undegraded":"2024-09-13T16:34:00.050362+0000","last_fullsized":"2024-09-13T16:34:00.050362+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:06.230871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070133+0000","last_change":"2024-09-13T16:33:59.066611+0000","last_active":"2024-09-13T16:33:59.070133+0000","last_peered":"2024-09-13T16:33:59.070133+0000","last_clean":"2024-09-13T16:33:59.070133+0000","last_became_active":"2024-09-13T16:33:59.066370+0000","last_became_peered":"2024-09-13T16:33:59.066370+0000","last_unstale":"2024-09-13T16:33:59.070133+0000","last_undegraded":"2024-09-13T16:33:59.070133+0000","last_fullsized":"2024-09-13T16:33:59.070133+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:56.153678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070192+0000","last_change":"2024-09-13T16:33:59.064150+0000","last_active":"2024-09-13T16:33:59.070192+0000","last_peered":"2024-09-13T16:33:59.070192+0000","last_clean":"2024-09-13T16:33:59.070192+0000","last_became_active":"2024-09-13T16:33:59.063883+0000","last_became_peered":"2024-09-13T16:33:59.063883+0000","last_unstale":"2024-09-13T16:33:59.070192+0000","last_undegraded":"2024-09-13T16:33:59.070192+0000","last_fullsized":"2024-09-13T16:33:59.070192+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:53.651102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070414+0000","last_change":"2024-09-13T16:33:59.069179+0000","last_active":"2024-09-13T16:33:59.070414+0000","last_peered":"2024-09-13T16:33:59.070414+0000","last_clean":"2024-09-13T16:33:59.070414+0000","last_became_active":"2024-09-13T16:33:59.067982+0000","last_became_peered":"2024-09-13T16:33:59.067982+0000","last_unstale":"2024-09-13T16:33:59.070414+0000","last_undegraded":"2024-09-13T16:33:59.070414+0000","last_fullsized":"2024-09-13T16:33:59.070414+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:08.465601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049444+0000","last_change":"2024-09-13T16:33:59.060484+0000","last_active":"2024-09-13T16:34:00.049444+0000","last_peered":"2024-09-13T16:34:00.049444+0000","last_clean":"2024-09-13T16:34:00.049444+0000","last_became_active":"2024-09-13T16:33:59.059850+0000","last_became_peered":"2024-09-13T16:33:59.059850+0000","last_unstale":"2024-09-13T16:34:00.049444+0000","last_undegraded":"2024-09-13T16:34:00.049444+0000","last_fullsized":"2024-09-13T16:34:00.049444+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:04.750350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357033+0000","last_change":"2024-09-13T16:33:59.069430+0000","last_active":"2024-09-13T16:34:00.357033+0000","last_peered":"2024-09-13T16:34:00.357033+0000","last_clean":"2024-09-13T16:34:00.357033+0000","last_became_active":"2024-09-13T16:33:59.069176+0000","last_became_peered":"2024-09-13T16:33:59.069176+0000","last_unstale":"2024-09-13T16:34:00.357033+0000","last_undegraded":"2024-09-13T16:34:00.357033+0000","last_fullsized":"2024-09-13T16:34:00.357033+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:46.662453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073533+0000","last_change":"2024-09-13T16:33:59.069799+0000","last_active":"2024-09-13T16:33:59.073533+0000","last_peered":"2024-09-13T16:33:59.073533+0000","last_clean":"2024-09-13T16:33:59.073533+0000","last_became_active":"2024-09-13T16:33:59.069570+0000","last_became_peered":"2024-09-13T16:33:59.069570+0000","last_unstale":"2024-09-13T16:33:59.073533+0000","last_undegraded":"2024-09-13T16:33:59.073533+0000","last_fullsized":"2024-09-13T16:33:59.073533+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:11.323458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070372+0000","last_change":"2024-09-13T16:33:59.069644+0000","last_active":"2024-09-13T16:33:59.070372+0000","last_peered":"2024-09-13T16:33:59.070372+0000","last_clean":"2024-09-13T16:33:59.070372+0000","last_became_active":"2024-09-13T16:33:59.069411+0000","last_became_peered":"2024-09-13T16:33:59.069411+0000","last_unstale":"2024-09-13T16:33:59.070372+0000","last_undegraded":"2024-09-13T16:33:59.070372+0000","last_fullsized":"2024-09-13T16:33:59.070372+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:20.575702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070551+0000","last_change":"2024-09-13T16:33:59.057279+0000","last_active":"2024-09-13T16:33:59.070551+0000","last_peered":"2024-09-13T16:33:59.070551+0000","last_clean":"2024-09-13T16:33:59.070551+0000","last_became_active":"2024-09-13T16:33:59.057003+0000","last_became_peered":"2024-09-13T16:33:59.057003+0000","last_unstale":"2024-09-13T16:33:59.070551+0000","last_undegraded":"2024-09-13T16:33:59.070551+0000","last_fullsized":"2024-09-13T16:33:59.070551+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:53.651102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049552+0000","last_change":"2024-09-13T16:33:59.069583+0000","last_active":"2024-09-13T16:34:00.049552+0000","last_peered":"2024-09-13T16:34:00.049552+0000","last_clean":"2024-09-13T16:34:00.049552+0000","last_became_active":"2024-09-13T16:33:59.069247+0000","last_became_peered":"2024-09-13T16:33:59.069247+0000","last_unstale":"2024-09-13T16:34:00.049552+0000","last_undegraded":"2024-09-13T16:34:00.049552+0000","last_fullsized":"2024-09-13T16:34:00.049552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:56.299851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049486+0000","last_change":"2024-09-13T16:33:59.051893+0000","last_active":"2024-09-13T16:34:00.049486+0000","last_peered":"2024-09-13T16:34:00.049486+0000","last_clean":"2024-09-13T16:34:00.049486+0000","last_became_active":"2024-09-13T16:33:59.051666+0000","last_became_peered":"2024-09-13T16:33:59.051666+0000","last_unstale":"2024-09-13T16:34:00.049486+0000","last_undegraded":"2024-09-13T16:34:00.049486+0000","last_fullsized":"2024-09-13T16:34:00.049486+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:28.934344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250649+0000","last_change":"2024-09-13T16:33:59.065849+0000","last_active":"2024-09-13T16:34:00.250649+0000","last_peered":"2024-09-13T16:34:00.250649+0000","last_clean":"2024-09-13T16:34:00.250649+0000","last_became_active":"2024-09-13T16:33:59.065494+0000","last_became_peered":"2024-09-13T16:33:59.065494+0000","last_unstale":"2024-09-13T16:34:00.250649+0000","last_undegraded":"2024-09-13T16:34:00.250649+0000","last_fullsized":"2024-09-13T16:34:00.250649+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:49.348627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070199+0000","last_change":"2024-09-13T16:33:59.066580+0000","last_active":"2024-09-13T16:33:59.070199+0000","last_peered":"2024-09-13T16:33:59.070199+0000","last_clean":"2024-09-13T16:33:59.070199+0000","last_became_active":"2024-09-13T16:33:59.066289+0000","last_became_peered":"2024-09-13T16:33:59.066289+0000","last_unstale":"2024-09-13T16:33:59.070199+0000","last_undegraded":"2024-09-13T16:33:59.070199+0000","last_fullsized":"2024-09-13T16:33:59.070199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:13.717073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357237+0000","last_change":"2024-09-13T16:33:59.060821+0000","last_active":"2024-09-13T16:34:00.357237+0000","last_peered":"2024-09-13T16:34:00.357237+0000","last_clean":"2024-09-13T16:34:00.357237+0000","last_became_active":"2024-09-13T16:33:59.060577+0000","last_became_peered":"2024-09-13T16:33:59.060577+0000","last_unstale":"2024-09-13T16:34:00.357237+0000","last_undegraded":"2024-09-13T16:34:00.357237+0000","last_fullsized":"2024-09-13T16:34:00.357237+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:22.417008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049039+0000","last_change":"2024-09-13T16:33:59.068373+0000","last_active":"2024-09-13T16:34:00.049039+0000","last_peered":"2024-09-13T16:34:00.049039+0000","last_clean":"2024-09-13T16:34:00.049039+0000","last_became_active":"2024-09-13T16:33:59.068122+0000","last_became_peered":"2024-09-13T16:33:59.068122+0000","last_unstale":"2024-09-13T16:34:00.049039+0000","last_undegraded":"2024-09-13T16:34:00.049039+0000","last_fullsized":"2024-09-13T16:34:00.049039+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:37.284055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.057809+0000","last_change":"2024-09-13T16:33:59.054579+0000","last_active":"2024-09-13T16:33:59.057809+0000","last_peered":"2024-09-13T16:33:59.057809+0000","last_clean":"2024-09-13T16:33:59.057809+0000","last_became_active":"2024-09-13T16:33:59.054259+0000","last_became_peered":"2024-09-13T16:33:59.054259+0000","last_unstale":"2024-09-13T16:33:59.057809+0000","last_undegraded":"2024-09-13T16:33:59.057809+0000","last_fullsized":"2024-09-13T16:33:59.057809+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:45.288502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356513+0000","last_change":"2024-09-13T16:33:59.050714+0000","last_active":"2024-09-13T16:34:00.356513+0000","last_peered":"2024-09-13T16:34:00.356513+0000","last_clean":"2024-09-13T16:34:00.356513+0000","last_became_active":"2024-09-13T16:33:59.050466+0000","last_became_peered":"2024-09-13T16:33:59.050466+0000","last_unstale":"2024-09-13T16:34:00.356513+0000","last_undegraded":"2024-09-13T16:34:00.356513+0000","last_fullsized":"2024-09-13T16:34:00.356513+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:40.134229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.063399+0000","last_change":"2024-09-13T16:33:59.058917+0000","last_active":"2024-09-13T16:33:59.063399+0000","last_peered":"2024-09-13T16:33:59.063399+0000","last_clean":"2024-09-13T16:33:59.063399+0000","last_became_active":"2024-09-13T16:33:59.058649+0000","last_became_peered":"2024-09-13T16:33:59.058649+0000","last_unstale":"2024-09-13T16:33:59.063399+0000","last_undegraded":"2024-09-13T16:33:59.063399+0000","last_fullsized":"2024-09-13T16:33:59.063399+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:48.715850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"110'4","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048575+0000","last_change":"2024-09-13T16:33:58.676086+0000","last_active":"2024-09-13T16:34:00.048575+0000","last_peered":"2024-09-13T16:34:00.048575+0000","last_clean":"2024-09-13T16:34:00.048575+0000","last_became_active":"2024-09-13T16:33:58.675839+0000","last_became_peered":"2024-09-13T16:33:58.675839+0000","last_unstale":"2024-09-13T16:34:00.048575+0000","last_undegraded":"2024-09-13T16:34:00.048575+0000","last_fullsized":"2024-09-13T16:34:00.048575+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:00.120019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356525+0000","last_change":"2024-09-13T16:33:59.059874+0000","last_active":"2024-09-13T16:34:00.356525+0000","last_peered":"2024-09-13T16:34:00.356525+0000","last_clean":"2024-09-13T16:34:00.356525+0000","last_became_active":"2024-09-13T16:33:59.059623+0000","last_became_peered":"2024-09-13T16:33:59.059623+0000","last_unstale":"2024-09-13T16:34:00.356525+0000","last_undegraded":"2024-09-13T16:34:00.356525+0000","last_fullsized":"2024-09-13T16:34:00.356525+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:43.311770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357145+0000","last_change":"2024-09-13T16:33:58.673512+0000","last_active":"2024-09-13T16:34:00.357145+0000","last_peered":"2024-09-13T16:34:00.357145+0000","last_clean":"2024-09-13T16:34:00.357145+0000","last_became_active":"2024-09-13T16:33:58.673259+0000","last_became_peered":"2024-09-13T16:33:58.673259+0000","last_unstale":"2024-09-13T16:34:00.357145+0000","last_undegraded":"2024-09-13T16:34:00.357145+0000","last_fullsized":"2024-09-13T16:34:00.357145+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:50.075397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049105+0000","last_change":"2024-09-13T16:33:59.063570+0000","last_active":"2024-09-13T16:34:00.049105+0000","last_peered":"2024-09-13T16:34:00.049105+0000","last_clean":"2024-09-13T16:34:00.049105+0000","last_became_active":"2024-09-13T16:33:59.063235+0000","last_became_peered":"2024-09-13T16:33:59.063235+0000","last_unstale":"2024-09-13T16:34:00.049105+0000","last_undegraded":"2024-09-13T16:34:00.049105+0000","last_fullsized":"2024-09-13T16:34:00.049105+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:14.487933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048983+0000","last_change":"2024-09-13T16:33:58.672877+0000","last_active":"2024-09-13T16:34:00.048983+0000","last_peered":"2024-09-13T16:34:00.048983+0000","last_clean":"2024-09-13T16:34:00.048983+0000","last_became_active":"2024-09-13T16:33:58.672445+0000","last_became_peered":"2024-09-13T16:33:58.672445+0000","last_unstale":"2024-09-13T16:34:00.048983+0000","last_undegraded":"2024-09-13T16:34:00.048983+0000","last_fullsized":"2024-09-13T16:34:00.048983+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:03.444757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250584+0000","last_change":"2024-09-13T16:33:59.068418+0000","last_active":"2024-09-13T16:34:00.250584+0000","last_peered":"2024-09-13T16:34:00.250584+0000","last_clean":"2024-09-13T16:34:00.250584+0000","last_became_active":"2024-09-13T16:33:59.068064+0000","last_became_peered":"2024-09-13T16:33:59.068064+0000","last_unstale":"2024-09-13T16:34:00.250584+0000","last_undegraded":"2024-09-13T16:34:00.250584+0000","last_fullsized":"2024-09-13T16:34:00.250584+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:55.047884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070577+0000","last_change":"2024-09-13T16:33:58.679320+0000","last_active":"2024-09-13T16:33:59.070577+0000","last_peered":"2024-09-13T16:33:59.070577+0000","last_clean":"2024-09-13T16:33:59.070577+0000","last_became_active":"2024-09-13T16:33:58.678986+0000","last_became_peered":"2024-09-13T16:33:58.678986+0000","last_unstale":"2024-09-13T16:33:59.070577+0000","last_undegraded":"2024-09-13T16:33:59.070577+0000","last_fullsized":"2024-09-13T16:33:59.070577+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:15.661709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048628+0000","last_change":"2024-09-13T16:33:59.062397+0000","last_active":"2024-09-13T16:34:00.048628+0000","last_peered":"2024-09-13T16:34:00.048628+0000","last_clean":"2024-09-13T16:34:00.048628+0000","last_became_active":"2024-09-13T16:33:59.062169+0000","last_became_peered":"2024-09-13T16:33:59.062169+0000","last_unstale":"2024-09-13T16:34:00.048628+0000","last_undegraded":"2024-09-13T16:34:00.048628+0000","last_fullsized":"2024-09-13T16:34:00.048628+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:22.178142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050658+0000","last_change":"2024-09-13T16:33:58.678062+0000","last_active":"2024-09-13T16:34:00.050658+0000","last_peered":"2024-09-13T16:34:00.050658+0000","last_clean":"2024-09-13T16:34:00.050658+0000","last_became_active":"2024-09-13T16:33:58.677518+0000","last_became_peered":"2024-09-13T16:33:58.677518+0000","last_unstale":"2024-09-13T16:34:00.050658+0000","last_undegraded":"2024-09-13T16:34:00.050658+0000","last_fullsized":"2024-09-13T16:34:00.050658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:11.642624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.067000+0000","last_change":"2024-09-13T16:33:59.065747+0000","last_active":"2024-09-13T16:33:59.067000+0000","last_peered":"2024-09-13T16:33:59.067000+0000","last_clean":"2024-09-13T16:33:59.067000+0000","last_became_active":"2024-09-13T16:33:59.064140+0000","last_became_peered":"2024-09-13T16:33:59.064140+0000","last_unstale":"2024-09-13T16:33:59.067000+0000","last_undegraded":"2024-09-13T16:33:59.067000+0000","last_fullsized":"2024-09-13T16:33:59.067000+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:38.995988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250183+0000","last_change":"2024-09-13T16:33:58.675556+0000","last_active":"2024-09-13T16:34:00.250183+0000","last_peered":"2024-09-13T16:34:00.250183+0000","last_clean":"2024-09-13T16:34:00.250183+0000","last_became_active":"2024-09-13T16:33:58.675268+0000","last_became_peered":"2024-09-13T16:33:58.675268+0000","last_unstale":"2024-09-13T16:34:00.250183+0000","last_undegraded":"2024-09-13T16:34:00.250183+0000","last_fullsized":"2024-09-13T16:34:00.250183+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:39.120119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070943+0000","last_change":"2024-09-13T16:33:59.068111+0000","last_active":"2024-09-13T16:33:59.070943+0000","last_peered":"2024-09-13T16:33:59.070943+0000","last_clean":"2024-09-13T16:33:59.070943+0000","last_became_active":"2024-09-13T16:33:59.067829+0000","last_became_peered":"2024-09-13T16:33:59.067829+0000","last_unstale":"2024-09-13T16:33:59.070943+0000","last_undegraded":"2024-09-13T16:33:59.070943+0000","last_fullsized":"2024-09-13T16:33:59.070943+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:50.292575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049652+0000","last_change":"2024-09-13T16:33:58.672874+0000","last_active":"2024-09-13T16:34:00.049652+0000","last_peered":"2024-09-13T16:34:00.049652+0000","last_clean":"2024-09-13T16:34:00.049652+0000","last_became_active":"2024-09-13T16:33:58.672515+0000","last_became_peered":"2024-09-13T16:33:58.672515+0000","last_unstale":"2024-09-13T16:34:00.049652+0000","last_undegraded":"2024-09-13T16:34:00.049652+0000","last_fullsized":"2024-09-13T16:34:00.049652+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:07.404188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048450+0000","last_change":"2024-09-13T16:33:59.058324+0000","last_active":"2024-09-13T16:34:00.048450+0000","last_peered":"2024-09-13T16:34:00.048450+0000","last_clean":"2024-09-13T16:34:00.048450+0000","last_became_active":"2024-09-13T16:33:59.058112+0000","last_became_peered":"2024-09-13T16:33:59.058112+0000","last_unstale":"2024-09-13T16:34:00.048450+0000","last_undegraded":"2024-09-13T16:34:00.048450+0000","last_fullsized":"2024-09-13T16:34:00.048450+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:44.131355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048532+0000","last_change":"2024-09-13T16:33:58.669983+0000","last_active":"2024-09-13T16:34:00.048532+0000","last_peered":"2024-09-13T16:34:00.048532+0000","last_clean":"2024-09-13T16:34:00.048532+0000","last_became_active":"2024-09-13T16:33:58.669696+0000","last_became_peered":"2024-09-13T16:33:58.669696+0000","last_unstale":"2024-09-13T16:34:00.048532+0000","last_undegraded":"2024-09-13T16:34:00.048532+0000","last_fullsized":"2024-09-13T16:34:00.048532+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:10.360488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.063423+0000","last_change":"2024-09-13T16:33:59.059064+0000","last_active":"2024-09-13T16:33:59.063423+0000","last_peered":"2024-09-13T16:33:59.063423+0000","last_clean":"2024-09-13T16:33:59.063423+0000","last_became_active":"2024-09-13T16:33:59.058775+0000","last_became_peered":"2024-09-13T16:33:59.058775+0000","last_unstale":"2024-09-13T16:33:59.063423+0000","last_undegraded":"2024-09-13T16:33:59.063423+0000","last_fullsized":"2024-09-13T16:33:59.063423+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:44.111538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049621+0000","last_change":"2024-09-13T16:33:58.673000+0000","last_active":"2024-09-13T16:34:00.049621+0000","last_peered":"2024-09-13T16:34:00.049621+0000","last_clean":"2024-09-13T16:34:00.049621+0000","last_became_active":"2024-09-13T16:33:58.672745+0000","last_became_peered":"2024-09-13T16:33:58.672745+0000","last_unstale":"2024-09-13T16:34:00.049621+0000","last_undegraded":"2024-09-13T16:34:00.049621+0000","last_fullsized":"2024-09-13T16:34:00.049621+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:14.708873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073195+0000","last_change":"2024-09-13T16:33:59.061564+0000","last_active":"2024-09-13T16:33:59.073195+0000","last_peered":"2024-09-13T16:33:59.073195+0000","last_clean":"2024-09-13T16:33:59.073195+0000","last_became_active":"2024-09-13T16:33:59.061201+0000","last_became_peered":"2024-09-13T16:33:59.061201+0000","last_unstale":"2024-09-13T16:33:59.073195+0000","last_undegraded":"2024-09-13T16:33:59.073195+0000","last_fullsized":"2024-09-13T16:33:59.073195+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:42.285817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.048993+0000","last_change":"2024-09-13T16:33:58.674091+0000","last_active":"2024-09-13T16:33:59.048993+0000","last_peered":"2024-09-13T16:33:59.048993+0000","last_clean":"2024-09-13T16:33:59.048993+0000","last_became_active":"2024-09-13T16:33:58.673844+0000","last_became_peered":"2024-09-13T16:33:58.673844+0000","last_unstale":"2024-09-13T16:33:59.048993+0000","last_undegraded":"2024-09-13T16:33:59.048993+0000","last_fullsized":"2024-09-13T16:33:59.048993+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:20.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356628+0000","last_change":"2024-09-13T16:33:59.059555+0000","last_active":"2024-09-13T16:34:00.356628+0000","last_peered":"2024-09-13T16:34:00.356628+0000","last_clean":"2024-09-13T16:34:00.356628+0000","last_became_active":"2024-09-13T16:33:59.058898+0000","last_became_peered":"2024-09-13T16:33:59.058898+0000","last_unstale":"2024-09-13T16:34:00.356628+0000","last_undegraded":"2024-09-13T16:34:00.356628+0000","last_fullsized":"2024-09-13T16:34:00.356628+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:00.974927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"110'3","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050918+0000","last_change":"2024-09-13T16:33:58.674922+0000","last_active":"2024-09-13T16:34:00.050918+0000","last_peered":"2024-09-13T16:34:00.050918+0000","last_clean":"2024-09-13T16:34:00.050918+0000","last_became_active":"2024-09-13T16:33:58.674541+0000","last_became_peered":"2024-09-13T16:33:58.674541+0000","last_unstale":"2024-09-13T16:34:00.050918+0000","last_undegraded":"2024-09-13T16:34:00.050918+0000","last_fullsized":"2024-09-13T16:34:00.050918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:39:11.486110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073431+0000","last_change":"2024-09-13T16:33:59.064997+0000","last_active":"2024-09-13T16:33:59.073431+0000","last_peered":"2024-09-13T16:33:59.073431+0000","last_clean":"2024-09-13T16:33:59.073431+0000","last_became_active":"2024-09-13T16:33:59.064202+0000","last_became_peered":"2024-09-13T16:33:59.064202+0000","last_unstale":"2024-09-13T16:33:59.073431+0000","last_undegraded":"2024-09-13T16:33:59.073431+0000","last_fullsized":"2024-09-13T16:33:59.073431+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:45.288502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"110'6","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073103+0000","last_change":"2024-09-13T16:33:58.675526+0000","last_active":"2024-09-13T16:33:59.073103+0000","last_peered":"2024-09-13T16:33:59.073103+0000","last_clean":"2024-09-13T16:33:59.073103+0000","last_became_active":"2024-09-13T16:33:58.675267+0000","last_became_peered":"2024-09-13T16:33:58.675267+0000","last_unstale":"2024-09-13T16:33:59.073103+0000","last_undegraded":"2024-09-13T16:33:59.073103+0000","last_fullsized":"2024-09-13T16:33:59.073103+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:14.456004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250814+0000","last_change":"2024-09-13T16:33:59.057018+0000","last_active":"2024-09-13T16:34:00.250814+0000","last_peered":"2024-09-13T16:34:00.250814+0000","last_clean":"2024-09-13T16:34:00.250814+0000","last_became_active":"2024-09-13T16:33:59.056616+0000","last_became_peered":"2024-09-13T16:33:59.056616+0000","last_unstale":"2024-09-13T16:34:00.250814+0000","last_undegraded":"2024-09-13T16:34:00.250814+0000","last_fullsized":"2024-09-13T16:34:00.250814+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:04.986467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.064386+0000","last_change":"2024-09-13T16:33:58.675819+0000","last_active":"2024-09-13T16:33:59.064386+0000","last_peered":"2024-09-13T16:33:59.064386+0000","last_clean":"2024-09-13T16:33:59.064386+0000","last_became_active":"2024-09-13T16:33:58.675567+0000","last_became_peered":"2024-09-13T16:33:58.675567+0000","last_unstale":"2024-09-13T16:33:59.064386+0000","last_undegraded":"2024-09-13T16:33:59.064386+0000","last_fullsized":"2024-09-13T16:33:59.064386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:53.340550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.050922+0000","last_change":"2024-09-13T16:33:59.048159+0000","last_active":"2024-09-13T16:33:59.050922+0000","last_peered":"2024-09-13T16:33:59.050922+0000","last_clean":"2024-09-13T16:33:59.050922+0000","last_became_active":"2024-09-13T16:33:59.047959+0000","last_became_peered":"2024-09-13T16:33:59.047959+0000","last_unstale":"2024-09-13T16:33:59.050922+0000","last_undegraded":"2024-09-13T16:33:59.050922+0000","last_fullsized":"2024-09-13T16:33:59.050922+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:31.176030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048501+0000","last_change":"2024-09-13T16:33:58.672737+0000","last_active":"2024-09-13T16:34:00.048501+0000","last_peered":"2024-09-13T16:34:00.048501+0000","last_clean":"2024-09-13T16:34:00.048501+0000","last_became_active":"2024-09-13T16:33:58.672489+0000","last_became_peered":"2024-09-13T16:33:58.672489+0000","last_unstale":"2024-09-13T16:34:00.048501+0000","last_undegraded":"2024-09-13T16:34:00.048501+0000","last_fullsized":"2024-09-13T16:34:00.048501+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:15.267273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357082+0000","last_change":"2024-09-13T16:33:59.053852+0000","last_active":"2024-09-13T16:34:00.357082+0000","last_peered":"2024-09-13T16:34:00.357082+0000","last_clean":"2024-09-13T16:34:00.357082+0000","last_became_active":"2024-09-13T16:33:59.053642+0000","last_became_peered":"2024-09-13T16:33:59.053642+0000","last_unstale":"2024-09-13T16:34:00.357082+0000","last_undegraded":"2024-09-13T16:34:00.357082+0000","last_fullsized":"2024-09-13T16:34:00.357082+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:34.748525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356549+0000","last_change":"2024-09-13T16:33:58.672773+0000","last_active":"2024-09-13T16:34:00.356549+0000","last_peered":"2024-09-13T16:34:00.356549+0000","last_clean":"2024-09-13T16:34:00.356549+0000","last_became_active":"2024-09-13T16:33:58.672501+0000","last_became_peered":"2024-09-13T16:33:58.672501+0000","last_unstale":"2024-09-13T16:34:00.356549+0000","last_undegraded":"2024-09-13T16:34:00.356549+0000","last_fullsized":"2024-09-13T16:34:00.356549+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:45.549224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073029+0000","last_change":"2024-09-13T16:33:59.069960+0000","last_active":"2024-09-13T16:33:59.073029+0000","last_peered":"2024-09-13T16:33:59.073029+0000","last_clean":"2024-09-13T16:33:59.073029+0000","last_became_active":"2024-09-13T16:33:59.069710+0000","last_became_peered":"2024-09-13T16:33:59.069710+0000","last_unstale":"2024-09-13T16:33:59.073029+0000","last_undegraded":"2024-09-13T16:33:59.073029+0000","last_fullsized":"2024-09-13T16:33:59.073029+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:53.506939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"110'8","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048740+0000","last_change":"2024-09-13T16:33:58.671685+0000","last_active":"2024-09-13T16:34:00.048740+0000","last_peered":"2024-09-13T16:34:00.048740+0000","last_clean":"2024-09-13T16:34:00.048740+0000","last_became_active":"2024-09-13T16:33:58.671087+0000","last_became_peered":"2024-09-13T16:33:58.671087+0000","last_unstale":"2024-09-13T16:34:00.048740+0000","last_undegraded":"2024-09-13T16:34:00.048740+0000","last_fullsized":"2024-09-13T16:34:00.048740+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.816446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356579+0000","last_change":"2024-09-13T16:33:59.059756+0000","last_active":"2024-09-13T16:34:00.356579+0000","last_peered":"2024-09-13T16:34:00.356579+0000","last_clean":"2024-09-13T16:34:00.356579+0000","last_became_active":"2024-09-13T16:33:59.059378+0000","last_became_peered":"2024-09-13T16:33:59.059378+0000","last_unstale":"2024-09-13T16:34:00.356579+0000","last_undegraded":"2024-09-13T16:34:00.356579+0000","last_fullsized":"2024-09-13T16:34:00.356579+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:04.036480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.062699+0000","last_change":"2024-09-13T16:33:58.673399+0000","last_active":"2024-09-13T16:33:59.062699+0000","last_peered":"2024-09-13T16:33:59.062699+0000","last_clean":"2024-09-13T16:33:59.062699+0000","last_became_active":"2024-09-13T16:33:58.673124+0000","last_became_peered":"2024-09-13T16:33:58.673124+0000","last_unstale":"2024-09-13T16:33:59.062699+0000","last_undegraded":"2024-09-13T16:33:59.062699+0000","last_fullsized":"2024-09-13T16:33:59.062699+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:10.801326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.057784+0000","last_change":"2024-09-13T16:33:59.054489+0000","last_active":"2024-09-13T16:33:59.057784+0000","last_peered":"2024-09-13T16:33:59.057784+0000","last_clean":"2024-09-13T16:33:59.057784+0000","last_became_active":"2024-09-13T16:33:59.054165+0000","last_became_peered":"2024-09-13T16:33:59.054165+0000","last_unstale":"2024-09-13T16:33:59.057784+0000","last_undegraded":"2024-09-13T16:33:59.057784+0000","last_fullsized":"2024-09-13T16:33:59.057784+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:42.285817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050885+0000","last_change":"2024-09-13T16:33:58.678147+0000","last_active":"2024-09-13T16:34:00.050885+0000","last_peered":"2024-09-13T16:34:00.050885+0000","last_clean":"2024-09-13T16:34:00.050885+0000","last_became_active":"2024-09-13T16:33:58.677904+0000","last_became_peered":"2024-09-13T16:33:58.677904+0000","last_unstale":"2024-09-13T16:34:00.050885+0000","last_undegraded":"2024-09-13T16:34:00.050885+0000","last_fullsized":"2024-09-13T16:34:00.050885+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:14.708873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073452+0000","last_change":"2024-09-13T16:33:59.068458+0000","last_active":"2024-09-13T16:33:59.073452+0000","last_peered":"2024-09-13T16:33:59.073452+0000","last_clean":"2024-09-13T16:33:59.073452+0000","last_became_active":"2024-09-13T16:33:59.064541+0000","last_became_peered":"2024-09-13T16:33:59.064541+0000","last_unstale":"2024-09-13T16:33:59.073452+0000","last_undegraded":"2024-09-13T16:33:59.073452+0000","last_fullsized":"2024-09-13T16:33:59.073452+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:45.989345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250842+0000","last_change":"2024-09-13T16:33:58.675527+0000","last_active":"2024-09-13T16:34:00.250842+0000","last_peered":"2024-09-13T16:34:00.250842+0000","last_clean":"2024-09-13T16:34:00.250842+0000","last_became_active":"2024-09-13T16:33:58.675099+0000","last_became_peered":"2024-09-13T16:33:58.675099+0000","last_unstale":"2024-09-13T16:34:00.250842+0000","last_undegraded":"2024-09-13T16:34:00.250842+0000","last_fullsized":"2024-09-13T16:34:00.250842+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:19.505280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070283+0000","last_change":"2024-09-13T16:33:59.065925+0000","last_active":"2024-09-13T16:33:59.070283+0000","last_peered":"2024-09-13T16:33:59.070283+0000","last_clean":"2024-09-13T16:33:59.070283+0000","last_became_active":"2024-09-13T16:33:59.065675+0000","last_became_peered":"2024-09-13T16:33:59.065675+0000","last_unstale":"2024-09-13T16:33:59.070283+0000","last_undegraded":"2024-09-13T16:33:59.070283+0000","last_fullsized":"2024-09-13T16:33:59.070283+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:17.605895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357356+0000","last_change":"2024-09-13T16:33:58.674979+0000","last_active":"2024-09-13T16:34:00.357356+0000","last_peered":"2024-09-13T16:34:00.357356+0000","last_clean":"2024-09-13T16:34:00.357356+0000","last_became_active":"2024-09-13T16:33:58.674706+0000","last_became_peered":"2024-09-13T16:33:58.674706+0000","last_unstale":"2024-09-13T16:34:00.357356+0000","last_undegraded":"2024-09-13T16:34:00.357356+0000","last_fullsized":"2024-09-13T16:34:00.357356+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:11.872087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.066983+0000","last_change":"2024-09-13T16:33:59.063871+0000","last_active":"2024-09-13T16:33:59.066983+0000","last_peered":"2024-09-13T16:33:59.066983+0000","last_clean":"2024-09-13T16:33:59.066983+0000","last_became_active":"2024-09-13T16:33:59.063378+0000","last_became_peered":"2024-09-13T16:33:59.063378+0000","last_unstale":"2024-09-13T16:33:59.066983+0000","last_undegraded":"2024-09-13T16:33:59.066983+0000","last_fullsized":"2024-09-13T16:33:59.066983+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:10.897741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356534+0000","last_change":"2024-09-13T16:33:58.667018+0000","last_active":"2024-09-13T16:34:00.356534+0000","last_peered":"2024-09-13T16:34:00.356534+0000","last_clean":"2024-09-13T16:34:00.356534+0000","last_became_active":"2024-09-13T16:33:58.666768+0000","last_became_peered":"2024-09-13T16:33:58.666768+0000","last_unstale":"2024-09-13T16:34:00.356534+0000","last_undegraded":"2024-09-13T16:34:00.356534+0000","last_fullsized":"2024-09-13T16:34:00.356534+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:24:35.079021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048215+0000","last_change":"2024-09-13T16:33:59.052338+0000","last_active":"2024-09-13T16:34:00.048215+0000","last_peered":"2024-09-13T16:34:00.048215+0000","last_clean":"2024-09-13T16:34:00.048215+0000","last_became_active":"2024-09-13T16:33:59.052094+0000","last_became_peered":"2024-09-13T16:33:59.052094+0000","last_unstale":"2024-09-13T16:34:00.048215+0000","last_undegraded":"2024-09-13T16:34:00.048215+0000","last_fullsized":"2024-09-13T16:34:00.048215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:28.168051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.042041+0000","last_change":"2024-09-13T16:33:58.677489+0000","last_active":"2024-09-13T16:33:59.042041+0000","last_peered":"2024-09-13T16:33:59.042041+0000","last_clean":"2024-09-13T16:33:59.042041+0000","last_became_active":"2024-09-13T16:33:58.676929+0000","last_became_peered":"2024-09-13T16:33:58.676929+0000","last_unstale":"2024-09-13T16:33:59.042041+0000","last_undegraded":"2024-09-13T16:33:59.042041+0000","last_fullsized":"2024-09-13T16:33:59.042041+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:20.735767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357064+0000","last_change":"2024-09-13T16:33:59.064700+0000","last_active":"2024-09-13T16:34:00.357064+0000","last_peered":"2024-09-13T16:34:00.357064+0000","last_clean":"2024-09-13T16:34:00.357064+0000","last_became_active":"2024-09-13T16:33:59.064417+0000","last_became_peered":"2024-09-13T16:33:59.064417+0000","last_unstale":"2024-09-13T16:34:00.357064+0000","last_undegraded":"2024-09-13T16:34:00.357064+0000","last_fullsized":"2024-09-13T16:34:00.357064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:41.675312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357037+0000","last_change":"2024-09-13T16:33:58.666731+0000","last_active":"2024-09-13T16:34:00.357037+0000","last_peered":"2024-09-13T16:34:00.357037+0000","last_clean":"2024-09-13T16:34:00.357037+0000","last_became_active":"2024-09-13T16:33:58.666452+0000","last_became_peered":"2024-09-13T16:33:58.666452+0000","last_unstale":"2024-09-13T16:34:00.357037+0000","last_undegraded":"2024-09-13T16:34:00.357037+0000","last_fullsized":"2024-09-13T16:34:00.357037+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:38.543263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.066939+0000","last_change":"2024-09-13T16:33:59.063297+0000","last_active":"2024-09-13T16:33:59.066939+0000","last_peered":"2024-09-13T16:33:59.066939+0000","last_clean":"2024-09-13T16:33:59.066939+0000","last_became_active":"2024-09-13T16:33:59.063048+0000","last_became_peered":"2024-09-13T16:33:59.063048+0000","last_unstale":"2024-09-13T16:33:59.066939+0000","last_undegraded":"2024-09-13T16:33:59.066939+0000","last_fullsized":"2024-09-13T16:33:59.066939+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:36:51.336980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.063274+0000","last_change":"2024-09-13T16:33:58.673962+0000","last_active":"2024-09-13T16:33:59.063274+0000","last_peered":"2024-09-13T16:33:59.063274+0000","last_clean":"2024-09-13T16:33:59.063274+0000","last_became_active":"2024-09-13T16:33:58.673699+0000","last_became_peered":"2024-09-13T16:33:58.673699+0000","last_unstale":"2024-09-13T16:33:59.063274+0000","last_undegraded":"2024-09-13T16:33:59.063274+0000","last_fullsized":"2024-09-13T16:33:59.063274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:32.457259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048996+0000","last_change":"2024-09-13T16:33:59.058435+0000","last_active":"2024-09-13T16:34:00.048996+0000","last_peered":"2024-09-13T16:34:00.048996+0000","last_clean":"2024-09-13T16:34:00.048996+0000","last_became_active":"2024-09-13T16:33:59.058192+0000","last_became_peered":"2024-09-13T16:33:59.058192+0000","last_unstale":"2024-09-13T16:34:00.048996+0000","last_undegraded":"2024-09-13T16:34:00.048996+0000","last_fullsized":"2024-09-13T16:34:00.048996+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:15.680793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.049087+0000","last_change":"2024-09-13T16:33:58.675846+0000","last_active":"2024-09-13T16:33:59.049087+0000","last_peered":"2024-09-13T16:33:59.049087+0000","last_clean":"2024-09-13T16:33:59.049087+0000","last_became_active":"2024-09-13T16:33:58.675582+0000","last_became_peered":"2024-09-13T16:33:58.675582+0000","last_unstale":"2024-09-13T16:33:59.049087+0000","last_undegraded":"2024-09-13T16:33:59.049087+0000","last_fullsized":"2024-09-13T16:33:59.049087+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:08.439890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048261+0000","last_change":"2024-09-13T16:33:59.052217+0000","last_active":"2024-09-13T16:34:00.048261+0000","last_peered":"2024-09-13T16:34:00.048261+0000","last_clean":"2024-09-13T16:34:00.048261+0000","last_became_active":"2024-09-13T16:33:59.051961+0000","last_became_peered":"2024-09-13T16:33:59.051961+0000","last_unstale":"2024-09-13T16:34:00.048261+0000","last_undegraded":"2024-09-13T16:34:00.048261+0000","last_fullsized":"2024-09-13T16:34:00.048261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:24.319606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.042977+0000","last_change":"2024-09-13T16:33:58.675765+0000","last_active":"2024-09-13T16:33:59.042977+0000","last_peered":"2024-09-13T16:33:59.042977+0000","last_clean":"2024-09-13T16:33:59.042977+0000","last_became_active":"2024-09-13T16:33:58.675506+0000","last_became_peered":"2024-09-13T16:33:58.675506+0000","last_unstale":"2024-09-13T16:33:59.042977+0000","last_undegraded":"2024-09-13T16:33:59.042977+0000","last_fullsized":"2024-09-13T16:33:59.042977+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:40.216628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250450+0000","last_change":"2024-09-13T16:33:59.073444+0000","last_active":"2024-09-13T16:34:00.250450+0000","last_peered":"2024-09-13T16:34:00.250450+0000","last_clean":"2024-09-13T16:34:00.250450+0000","last_became_active":"2024-09-13T16:33:59.073253+0000","last_became_peered":"2024-09-13T16:33:59.073253+0000","last_unstale":"2024-09-13T16:34:00.250450+0000","last_undegraded":"2024-09-13T16:34:00.250450+0000","last_fullsized":"2024-09-13T16:34:00.250450+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:02.185471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.051551+0000","last_change":"2024-09-13T16:33:58.677431+0000","last_active":"2024-09-13T16:33:59.051551+0000","last_peered":"2024-09-13T16:33:59.051551+0000","last_clean":"2024-09-13T16:33:59.051551+0000","last_became_active":"2024-09-13T16:33:58.677102+0000","last_became_peered":"2024-09-13T16:33:58.677102+0000","last_unstale":"2024-09-13T16:33:59.051551+0000","last_undegraded":"2024-09-13T16:33:59.051551+0000","last_fullsized":"2024-09-13T16:33:59.051551+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:29.138171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356861+0000","last_change":"2024-09-13T16:33:59.063747+0000","last_active":"2024-09-13T16:34:00.356861+0000","last_peered":"2024-09-13T16:34:00.356861+0000","last_clean":"2024-09-13T16:34:00.356861+0000","last_became_active":"2024-09-13T16:33:59.063517+0000","last_became_peered":"2024-09-13T16:33:59.063517+0000","last_unstale":"2024-09-13T16:34:00.356861+0000","last_undegraded":"2024-09-13T16:34:00.356861+0000","last_fullsized":"2024-09-13T16:34:00.356861+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:28.881958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049278+0000","last_change":"2024-09-13T16:33:58.670084+0000","last_active":"2024-09-13T16:34:00.049278+0000","last_peered":"2024-09-13T16:34:00.049278+0000","last_clean":"2024-09-13T16:34:00.049278+0000","last_became_active":"2024-09-13T16:33:58.669773+0000","last_became_peered":"2024-09-13T16:33:58.669773+0000","last_unstale":"2024-09-13T16:34:00.049278+0000","last_undegraded":"2024-09-13T16:34:00.049278+0000","last_fullsized":"2024-09-13T16:34:00.049278+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:24.815735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356853+0000","last_change":"2024-09-13T16:33:59.064218+0000","last_active":"2024-09-13T16:34:00.356853+0000","last_peered":"2024-09-13T16:34:00.356853+0000","last_clean":"2024-09-13T16:34:00.356853+0000","last_became_active":"2024-09-13T16:33:59.063719+0000","last_became_peered":"2024-09-13T16:33:59.063719+0000","last_unstale":"2024-09-13T16:34:00.356853+0000","last_undegraded":"2024-09-13T16:34:00.356853+0000","last_fullsized":"2024-09-13T16:34:00.356853+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:20.519391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356773+0000","last_change":"2024-09-13T16:33:58.674249+0000","last_active":"2024-09-13T16:34:00.356773+0000","last_peered":"2024-09-13T16:34:00.356773+0000","last_clean":"2024-09-13T16:34:00.356773+0000","last_became_active":"2024-09-13T16:33:58.674009+0000","last_became_peered":"2024-09-13T16:33:58.674009+0000","last_unstale":"2024-09-13T16:34:00.356773+0000","last_undegraded":"2024-09-13T16:34:00.356773+0000","last_fullsized":"2024-09-13T16:34:00.356773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:30.785733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050817+0000","last_change":"2024-09-13T16:33:59.068317+0000","last_active":"2024-09-13T16:34:00.050817+0000","last_peered":"2024-09-13T16:34:00.050817+0000","last_clean":"2024-09-13T16:34:00.050817+0000","last_became_active":"2024-09-13T16:33:59.068069+0000","last_became_peered":"2024-09-13T16:33:59.068069+0000","last_unstale":"2024-09-13T16:34:00.050817+0000","last_undegraded":"2024-09-13T16:34:00.050817+0000","last_fullsized":"2024-09-13T16:34:00.050817+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:39.928275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.042315+0000","last_change":"2024-09-13T16:33:58.675706+0000","last_active":"2024-09-13T16:33:59.042315+0000","last_peered":"2024-09-13T16:33:59.042315+0000","last_clean":"2024-09-13T16:33:59.042315+0000","last_became_active":"2024-09-13T16:33:58.675438+0000","last_became_peered":"2024-09-13T16:33:58.675438+0000","last_unstale":"2024-09-13T16:33:59.042315+0000","last_undegraded":"2024-09-13T16:33:59.042315+0000","last_fullsized":"2024-09-13T16:33:59.042315+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:49.586862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250892+0000","last_change":"2024-09-13T16:33:59.072306+0000","last_active":"2024-09-13T16:34:00.250892+0000","last_peered":"2024-09-13T16:34:00.250892+0000","last_clean":"2024-09-13T16:34:00.250892+0000","last_became_active":"2024-09-13T16:33:59.071890+0000","last_became_peered":"2024-09-13T16:33:59.071890+0000","last_unstale":"2024-09-13T16:34:00.250892+0000","last_undegraded":"2024-09-13T16:34:00.250892+0000","last_fullsized":"2024-09-13T16:34:00.250892+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:34.150180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356812+0000","last_change":"2024-09-13T16:33:58.672758+0000","last_active":"2024-09-13T16:34:00.356812+0000","last_peered":"2024-09-13T16:34:00.356812+0000","last_clean":"2024-09-13T16:34:00.356812+0000","last_became_active":"2024-09-13T16:33:58.672465+0000","last_became_peered":"2024-09-13T16:33:58.672465+0000","last_unstale":"2024-09-13T16:34:00.356812+0000","last_undegraded":"2024-09-13T16:34:00.356812+0000","last_fullsized":"2024-09-13T16:34:00.356812+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:08.560580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250183+0000","last_change":"2024-09-13T16:33:59.056838+0000","last_active":"2024-09-13T16:34:00.250183+0000","last_peered":"2024-09-13T16:34:00.250183+0000","last_clean":"2024-09-13T16:34:00.250183+0000","last_became_active":"2024-09-13T16:33:59.056411+0000","last_became_peered":"2024-09-13T16:33:59.056411+0000","last_unstale":"2024-09-13T16:34:00.250183+0000","last_undegraded":"2024-09-13T16:34:00.250183+0000","last_fullsized":"2024-09-13T16:34:00.250183+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:58.823269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357271+0000","last_change":"2024-09-13T16:33:58.667319+0000","last_active":"2024-09-13T16:34:00.357271+0000","last_peered":"2024-09-13T16:34:00.357271+0000","last_clean":"2024-09-13T16:34:00.357271+0000","last_became_active":"2024-09-13T16:33:58.666989+0000","last_became_peered":"2024-09-13T16:33:58.666989+0000","last_unstale":"2024-09-13T16:34:00.357271+0000","last_undegraded":"2024-09-13T16:34:00.357271+0000","last_fullsized":"2024-09-13T16:34:00.357271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:38.406060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.050832+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:33:59.050832+0000","last_peered":"2024-09-13T16:33:59.050832+0000","last_clean":"2024-09-13T16:33:59.050832+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:33:59.050832+0000","last_undegraded":"2024-09-13T16:33:59.050832+0000","last_fullsized":"2024-09-13T16:33:59.050832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1974272,"data_stored":1222857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83799999999999997,"15min":0.84999999999999998},"min":{"1min":0.55100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1100000000000001,"5min":1.357,"15min":1.357},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85299999999999998},"min":{"1min":0.59699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0609999999999999,"5min":1.163,"15min":1.3460000000000001},"last":0.94399999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.58499999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.1459999999999999,"5min":1.569,"15min":1.569},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.85399999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1719999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":1.0900000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85199999999999998,"15min":0.878},"min":{"1min":0.67200000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1919999999999999,"5min":1.242,"15min":1.242},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.89700000000000002,"15min":0.85499999999999998},"min":{"1min":0.60499999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.024,"5min":1.339,"15min":1.339},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.92400000000000004,"15min":0.98199999999999998},"min":{"1min":0.78300000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1579999999999999,"5min":1.476,"15min":1.476},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.995,"15min":1.0009999999999999},"min":{"1min":0.754,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1299999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.85599999999999998,"15min":0.873},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.145,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.94699999999999995,"15min":1.036},"min":{"1min":0.57899999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.264,"5min":1.5549999999999999,"15min":1.589},"last":1.0189999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.208,"5min":1.425,"15min":1.425},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.054,"15min":1.097},"min":{"1min":0.626,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2889999999999999,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.997}]},{"osd":6,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0149999999999999,"15min":1.0609999999999999},"min":{"1min":0.74099999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.1950000000000001,"5min":1.72,"15min":1.72},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0489999999999999,"15min":1.1359999999999999},"min":{"1min":0.79300000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.218,"5min":1.623,"15min":1.6779999999999999},"last":1.0700000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542252,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1380352,"data_stored":627877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.52900000000000003,"5min":0.496,"15min":0.48299999999999998},"max":{"1min":1.256,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85499999999999998,"15min":0.88200000000000001},"min":{"1min":0.49099999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.151,"5min":1.202,"15min":1.4410000000000001},"last":0.309}]},{"osd":1,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.86099999999999999,"15min":0.84299999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.119,"5min":1.1799999999999999,"15min":1.2030000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.90200000000000002,"15min":0.91400000000000003},"min":{"1min":0.64000000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2010000000000001,"5min":1.411,"15min":1.411},"last":0.25900000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85199999999999998,"15min":0.87},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.503},"max":{"1min":1.1839999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91800000000000004,"15min":0.86699999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.55500000000000005},"max":{"1min":1.167,"5min":1.276,"15min":1.276},"last":0.42699999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.95099999999999996,"15min":0.95699999999999996},"min":{"1min":0.62,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.151,"5min":1.5309999999999999,"15min":1.536},"last":0.379},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.97099999999999997,"15min":0.98299999999999998},"min":{"1min":0.71799999999999997,"5min":0.71299999999999997,"15min":0.56399999999999995},"max":{"1min":1.2410000000000001,"5min":1.3120000000000001,"15min":1.409},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.93799999999999994,"15min":0.96199999999999997},"min":{"1min":0.63500000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2549999999999999,"5min":1.4570000000000001,"15min":1.4750000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.95599999999999996},"min":{"1min":0.66100000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.3149999999999999,"5min":1.347,"15min":1.6140000000000001},"last":0.40600000000000003}]},{"osd":5,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.94899999999999995,"15min":0.92700000000000005},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.204,"5min":1.339,"15min":1.339},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.05,"15min":1.1060000000000001},"min":{"1min":0.68700000000000006,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2070000000000001,"5min":1.4490000000000001,"15min":1.6439999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0489999999999999,"15min":1.099},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3819999999999999,"5min":1.4530000000000001,"15min":1.522},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.071,"15min":1.0600000000000001},"min":{"1min":0.746,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.2509999999999999,"5min":1.427,"15min":1.427},"last":0.50800000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542254,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1359872,"data_stored":625709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.83399999999999996,"15min":0.85299999999999998},"min":{"1min":0.56100000000000005,"5min":0.47899999999999998,"15min":0.47599999999999998},"max":{"1min":1.1479999999999999,"5min":1.1899999999999999,"15min":1.411},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.84499999999999997},"min":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.41699999999999998},"max":{"1min":1.1659999999999999,"5min":1.2909999999999999,"15min":1.675},"last":0.69899999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83599999999999997,"15min":0.86699999999999999},"min":{"1min":0.65800000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0880000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85099999999999998,"15min":0.83999999999999997},"min":{"1min":0.65600000000000003,"5min":0.52300000000000002,"15min":0.36499999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.96799999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.90400000000000003,"15min":0.92500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.135,"5min":1.258,"15min":1.625},"last":1.123},{"interface":"front","average":{"1min":1,"5min":0.96699999999999997,"15min":0.97799999999999998},"min":{"1min":0.78600000000000003,"5min":0.57599999999999996,"15min":0.55000000000000004},"max":{"1min":1.2010000000000001,"5min":1.516,"15min":1.552},"last":1.054}]},{"osd":3,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.91500000000000004,"15min":0.86599999999999999},"min":{"1min":0.69599999999999995,"5min":0.624,"15min":0.56100000000000005},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.3919999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90200000000000002,"15min":0.97099999999999997},"min":{"1min":0.67900000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1020000000000001,"5min":1.1419999999999999,"15min":3.8460000000000001},"last":1.23}]},{"osd":4,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.89600000000000002,"15min":1.008},"min":{"1min":0.59699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0369999999999999,"5min":1.655,"15min":3.6869999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0609999999999999,"15min":1.0620000000000001},"min":{"1min":0.88800000000000001,"5min":0.76100000000000001,"15min":0.57499999999999996},"max":{"1min":1.2669999999999999,"5min":1.347,"15min":1.601},"last":1.1619999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.92800000000000005,"15min":0.95599999999999996},"min":{"1min":0.68000000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.1679999999999999,"5min":1.3540000000000001,"15min":1.421},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.002,"15min":1.052},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.1899999999999999,"5min":1.73,"15min":2.21},"last":0.90300000000000002}]},{"osd":7,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0329999999999999,"15min":1.0349999999999999},"min":{"1min":0.64000000000000001,"5min":0.621,"15min":0.45500000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0629999999999999,"15min":1.165},"min":{"1min":0.76100000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.2350000000000001,"5min":1.51,"15min":3.9830000000000001},"last":1.2969999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1368064,"data_stored":625801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85299999999999998,"15min":0.84399999999999997},"min":{"1min":0.58699999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1819999999999999,"5min":1.2270000000000001,"15min":1.411},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89900000000000002,"15min":0.874},"min":{"1min":0.65500000000000003,"5min":0.52800000000000002,"15min":0.44600000000000001},"max":{"1min":1.175,"5min":1.462,"15min":2.899},"last":0.90900000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.90100000000000002,"15min":0.91600000000000004},"min":{"1min":0.627,"5min":0.55000000000000004,"15min":0.46200000000000002},"max":{"1min":1.1879999999999999,"5min":1.2490000000000001,"15min":3.085},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88400000000000001,"15min":0.91000000000000003},"min":{"1min":0.623,"5min":0.497,"15min":0.497},"max":{"1min":1.0649999999999999,"5min":1.367,"15min":1.5720000000000001},"last":0.96799999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.83199999999999996,"15min":0.84199999999999997},"min":{"1min":0.59299999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.228,"5min":1.228,"15min":1.4419999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93700000000000006,"15min":0.95999999999999996},"min":{"1min":0.70099999999999996,"5min":0.60199999999999998,"15min":0.498},"max":{"1min":1.2390000000000001,"5min":1.357,"15min":3.2160000000000002},"last":0.84799999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.91200000000000003,"15min":0.94299999999999995},"min":{"1min":0.60399999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91200000000000003,"15min":0.94599999999999995},"min":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.52200000000000002},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.577},"last":0.80300000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.95999999999999996},"min":{"1min":0.65500000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0900000000000001,"5min":1.268,"15min":2.996},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.008,"5min":0.99099999999999999,"15min":1.0940000000000001},"min":{"1min":0.67100000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.3440000000000001,"5min":1.5189999999999999,"15min":2.8860000000000001},"last":1.1140000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.92200000000000004,"15min":0.93899999999999995},"min":{"1min":0.63800000000000001,"5min":0.62,"15min":0.62},"max":{"1min":1.522,"5min":1.522,"15min":1.6499999999999999},"last":1.083},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0600000000000001,"15min":1.103},"min":{"1min":0.84899999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.3939999999999999,"5min":1.4450000000000001,"15min":3.1469999999999998},"last":0.998}]},{"osd":7,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0309999999999999,"15min":1.1140000000000001},"min":{"1min":0.80500000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.556,"5min":1.556,"15min":2.1230000000000002},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0409999999999999,"15min":1.1040000000000001},"min":{"1min":0.81799999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":1.018}]}]},{"osd":3,"up_from":14,"seq":60129542253,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1363968,"data_stored":626291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.63},"min":{"1min":0.497,"5min":0.42999999999999999,"15min":0.373},"max":{"1min":0.98499999999999999,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69199999999999995,"15min":0.66800000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.161,"15min":1.161},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67400000000000004,"15min":0.64100000000000001},"min":{"1min":0.46800000000000003,"5min":0.40699999999999997,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.50900000000000001,"5min":0.47899999999999998,"15min":0.46800000000000003},"max":{"1min":0.98499999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.70299999999999996},"min":{"1min":0.48099999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0609999999999999,"5min":1.0649999999999999,"15min":1.27},"last":0.439},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79700000000000004,"15min":0.77000000000000002},"min":{"1min":0.61399999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.0740000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.379}]},{"osd":4,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.93400000000000005,"15min":0.91000000000000003},"min":{"1min":0.752,"5min":0.65900000000000003,"15min":0.60499999999999998},"max":{"1min":1.23,"5min":1.29,"15min":1.8460000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.93700000000000006,"15min":0.95299999999999996},"min":{"1min":0.72999999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.782},"last":0.748}]},{"osd":5,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.96799999999999997},"min":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.59499999999999997},"max":{"1min":1.2649999999999999,"5min":1.3520000000000001,"15min":1.6299999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0369999999999999,"15min":1.079},"min":{"1min":0.74099999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.296,"5min":1.581,"15min":1.8140000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0620000000000001,"15min":1.109},"min":{"1min":0.69799999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3660000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":1,"5min":1.044,"15min":1.0489999999999999},"min":{"1min":0.64900000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2350000000000001,"5min":1.7030000000000001,"15min":1.95},"last":0.80300000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0940000000000001,"15min":1.0820000000000001},"min":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.71399999999999997},"max":{"1min":1.306,"5min":1.657,"15min":1.889},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0489999999999999,"15min":1.056},"min":{"1min":0.85199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.2350000000000001,"5min":1.5289999999999999,"15min":1.756},"last":0.85399999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542254,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1363968,"data_stored":625743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60799999999999998,"15min":0.59399999999999997},"min":{"1min":0.47699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.628},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.52000000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.92600000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67100000000000004,"15min":0.69499999999999995},"min":{"1min":0.45400000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85099999999999998,"5min":0.995,"15min":0.997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.71399999999999997,"15min":0.69499999999999995},"min":{"1min":0.45200000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.0529999999999999,"5min":1.147,"15min":1.147},"last":0.78800000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70699999999999996,"15min":0.67800000000000005},"min":{"1min":0.51800000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0389999999999999,"5min":1.119,"15min":1.119},"last":0.745},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.80400000000000005},"min":{"1min":0.627,"5min":0.37,"15min":0.37},"max":{"1min":0.98299999999999998,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.99099999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.93700000000000006,"15min":0.94799999999999995},"min":{"1min":0.73899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.4370000000000001,"5min":1.448,"15min":1.5149999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.95299999999999996,"15min":0.96899999999999997},"min":{"1min":0.82999999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1819999999999999,"5min":1.4330000000000001,"15min":1.8799999999999999},"last":1.0289999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95199999999999996,"15min":0.97299999999999998},"min":{"1min":0.63700000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.413,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.97399999999999998,"15min":0.96399999999999997},"min":{"1min":0.60699999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.22,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.3799999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.94699999999999995,"15min":1.006},"min":{"1min":0.79300000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.28,"5min":1.4690000000000001,"15min":2.024},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.028,"15min":1.0529999999999999},"min":{"1min":0.69699999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.3129999999999999,"5min":1.5249999999999999,"15min":1.6359999999999999},"last":1.3109999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0580000000000001,"15min":1.071},"min":{"1min":0.82499999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.357,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.0389999999999999,"15min":1.0580000000000001},"min":{"1min":0.79100000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.1459999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542253,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1363968,"data_stored":625743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.371,"15min":0.23300000000000001},"max":{"1min":0.78200000000000003,"5min":0.95899999999999996,"15min":1.0880000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.75,"15min":0.73899999999999999},"min":{"1min":0.51000000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93000000000000005,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.94999999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67500000000000004,"15min":0.64100000000000001},"min":{"1min":0.54200000000000004,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":0.96099999999999997,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.433},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71899999999999997,"15min":0.70899999999999996},"min":{"1min":0.52200000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":0.877,"5min":1.3129999999999999,"15min":1.458},"last":0.86099999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.74199999999999999,"15min":0.71399999999999997},"min":{"1min":0.54800000000000004,"5min":0.39700000000000002,"15min":0.35399999999999998},"max":{"1min":1.1100000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.753},{"interface":"front","average":{"1min":0.745,"5min":0.70799999999999996,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.44400000000000001,"15min":0.39600000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.403},"last":0.66800000000000004}]},{"osd":4,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.93600000000000005,"15min":0.91900000000000004},"min":{"1min":0.78700000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.2190000000000001,"5min":1.4930000000000001,"15min":1.546},"last":1.02},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90200000000000002,"15min":0.92200000000000004},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.51700000000000002},"max":{"1min":1.0660000000000001,"5min":1.377,"15min":2.2040000000000002},"last":0.98799999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.96999999999999997,"15min":0.95599999999999996},"min":{"1min":0.70399999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.389},"last":1.006},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.91400000000000003,"15min":0.95699999999999996},"min":{"1min":0.78300000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.131,"5min":1.238,"15min":2.3029999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.006,"15min":1.0069999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.357,"5min":1.4810000000000001,"15min":1.613},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0369999999999999,"15min":1.0209999999999999},"min":{"1min":0.80000000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3380000000000001,"5min":1.548,"15min":1.698},"last":1.5029999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.016,"15min":1.012},"min":{"1min":0.71599999999999997,"5min":0.64400000000000002,"15min":0.62},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.075,"15min":1.0549999999999999},"min":{"1min":0.76700000000000002,"5min":0.61599999999999999,"15min":0.59299999999999997},"max":{"1min":1.302,"5min":1.546,"15min":1.7689999999999999},"last":0.90600000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542253,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1970176,"data_stored":1217731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.63100000000000001,"15min":0.66000000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68899999999999995,"15min":0.66700000000000004},"min":{"1min":0.53400000000000003,"5min":0.442,"15min":0.377},"max":{"1min":0.872,"5min":1.077,"15min":1.157},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.56899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88600000000000001,"5min":1.0609999999999999,"15min":1.202},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73199999999999998,"15min":0.70099999999999996},"min":{"1min":0.52300000000000002,"5min":0.42799999999999999,"15min":0.39800000000000002},"max":{"1min":1.069,"5min":1.1160000000000001,"15min":1.2629999999999999},"last":0.71999999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73099999999999998,"15min":0.68100000000000005},"min":{"1min":0.45800000000000002,"5min":0.39200000000000002,"15min":0.312},"max":{"1min":0.97499999999999998,"5min":1.0589999999999999,"15min":1.4830000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88,"15min":0.84499999999999997},"min":{"1min":0.53300000000000003,"5min":0.48899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0660000000000001,"5min":1.284,"15min":1.6100000000000001},"last":0.74299999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95599999999999996,"15min":0.95899999999999996},"min":{"1min":0.73399999999999999,"5min":0.61299999999999999,"15min":0.57699999999999996},"max":{"1min":1.1539999999999999,"5min":1.321,"15min":1.486},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95399999999999996,"15min":0.96099999999999997},"min":{"1min":0.70499999999999996,"5min":0.64600000000000002,"15min":0.59999999999999998},"max":{"1min":1.167,"5min":1.3169999999999999,"15min":1.704},"last":0.85099999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.997,"15min":0.98599999999999999},"min":{"1min":0.752,"5min":0.71799999999999997,"15min":0.59599999999999997},"max":{"1min":1.2689999999999999,"5min":1.5609999999999999,"15min":1.575},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97899999999999998,"15min":0.99199999999999999},"min":{"1min":0.77400000000000002,"5min":0.52700000000000002,"15min":0.443},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.96299999999999997,"15min":0.97699999999999998},"min":{"1min":0.76900000000000002,"5min":0.69499999999999995,"15min":0.628},"max":{"1min":1.2190000000000001,"5min":1.3700000000000001,"15min":1.944},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.087,"15min":1.0629999999999999},"min":{"1min":0.77300000000000002,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3480000000000001,"5min":1.5,"15min":1.5780000000000001},"last":0.80900000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0780000000000001,"15min":1.0609999999999999},"min":{"1min":0.96199999999999997,"5min":0.747,"15min":0.61699999999999999},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.012},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0880000000000001,"15min":1.0629999999999999},"min":{"1min":0.91700000000000004,"5min":0.76700000000000002,"15min":0.65400000000000003},"max":{"1min":1.3460000000000001,"5min":1.51,"15min":2.0139999999999998},"last":0.89800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:34:03.158 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:34:03.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.273+0000 7f8039941700 1 Processor -- start 2024-09-13T16:34:03.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.273+0000 7f8039941700 1 -- start start 2024-09-13T16:34:03.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.273+0000 7f8039941700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80340cd7d0 0x7f803415adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803406ef30 con 0x7f8034158580 2024-09-13T16:34:03.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803406f180 con 0x7f80340cf130 2024-09-13T16:34:03.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f803415e980 con 0x7f80340cd7d0 2024-09-13T16:34:03.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039140700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80340cd7d0 0x7f803415adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039140700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f80340cd7d0 0x7f803415adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36502/0 (socket says 172.21.15.119:36502) 2024-09-13T16:34:03.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f803893f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8034158580 0x7f803415eda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54216/0 (socket says 172.21.15.119:54216) 2024-09-13T16:34:03.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039140700 1 -- 172.21.15.119:0/144851998 learned_addr learned my addr 172.21.15.119:0/144851998 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:03.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039140700 1 -- 172.21.15.119:0/144851998 >> v1:172.21.15.145:6789/0 conn(0x7f80340cf130 legacy=0x7f80340cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039140700 1 -- 172.21.15.119:0/144851998 >> v1:172.21.15.119:6789/0 conn(0x7f8034158580 legacy=0x7f803415eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f802b7fe700 1 -- 172.21.15.119:0/144851998 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4180099121 0 0) 0x7f803406ef30 con 0x7f8034158580 2024-09-13T16:34:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039140700 1 -- 172.21.15.119:0/144851998 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803406f460 con 0x7f80340cd7d0 2024-09-13T16:34:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039140700 1 --2- 172.21.15.119:0/144851998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80340cd7d0 0x7f803415adb0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f802c009e80 tx=0x7f802c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89046078d8a3e46b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f802b7fe700 1 -- 172.21.15.119:0/144851998 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00e070 con 0x7f80340cd7d0 2024-09-13T16:34:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f802b7fe700 1 -- 172.21.15.119:0/144851998 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c00b980 con 0x7f80340cd7d0 2024-09-13T16:34:03.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f802b7fe700 1 -- 172.21.15.119:0/144851998 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c00bd30 con 0x7f80340cd7d0 2024-09-13T16:34:03.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- 172.21.15.119:0/144851998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80340cd7d0 msgr2=0x7f803415adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 --2- 172.21.15.119:0/144851998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80340cd7d0 0x7f803415adb0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f802c009e80 tx=0x7f802c00b330 comp rx=0 tx=0).stop 2024-09-13T16:34:03.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- 172.21.15.119:0/144851998 shutdown_connections 2024-09-13T16:34:03.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 --2- 172.21.15.119:0/144851998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f80340cd7d0 0x7f803415adb0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- 172.21.15.119:0/144851998 >> 172.21.15.119:0/144851998 conn(0x7f8034070cc0 msgr2=0x7f80340710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:03.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- 172.21.15.119:0/144851998 shutdown_connections 2024-09-13T16:34:03.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.277+0000 7f8039941700 1 -- 172.21.15.119:0/144851998 wait complete. 2024-09-13T16:34:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 Processor -- start 2024-09-13T16:34:03.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 -- start start 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80340cd7d0 0x7f803414fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80340cf130 0x7f8034151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034158580 0x7f8034156660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8034152350 con 0x7f8034158580 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8034156ba0 con 0x7f80340cd7d0 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8034150290 con 0x7f80340cf130 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034158580 0x7f8034156660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034158580 0x7f8034156660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36514/0 (socket says 172.21.15.119:36514) 2024-09-13T16:34:03.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 -- 172.21.15.119:0/3954285559 learned_addr learned my addr 172.21.15.119:0/3954285559 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f803893f700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80340cd7d0 0x7f803414fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f802bfff700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80340cf130 0x7f8034151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 -- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80340cf130 msgr2=0x7f8034151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80340cf130 0x7f8034151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 -- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80340cd7d0 msgr2=0x7f803414fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80340cd7d0 0x7f803414fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 -- 172.21.15.119:0/3954285559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802c009b30 con 0x7f8034158580 2024-09-13T16:34:03.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039140700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034158580 0x7f8034156660 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f802c012000 tx=0x7f802c004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:03.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f80297fa700 1 -- 172.21.15.119:0/3954285559 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c01c640 con 0x7f8034158580 2024-09-13T16:34:03.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f80297fa700 1 -- 172.21.15.119:0/3954285559 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f802c01c7a0 con 0x7f8034158580 2024-09-13T16:34:03.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8034150570 con 0x7f8034158580 2024-09-13T16:34:03.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f80297fa700 1 -- 172.21.15.119:0/3954285559 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f802c01cb50 con 0x7f8034158580 2024-09-13T16:34:03.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.281+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f803414e230 con 0x7f8034158580 2024-09-13T16:34:03.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.285+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8034158a00 con 0x7f8034158580 2024-09-13T16:34:03.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.285+0000 7f80297fa700 1 -- 172.21.15.119:0/3954285559 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f802c01ccb0 con 0x7f8034158580 2024-09-13T16:34:03.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.285+0000 7f80297fa700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8020066730 0x7f8020068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.285+0000 7f80297fa700 1 -- 172.21.15.119:0/3954285559 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f802c085ad0 con 0x7f8034158580 2024-09-13T16:34:03.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.285+0000 7f803893f700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8020066730 0x7f8020068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.289+0000 7f803893f700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8020066730 0x7f8020068bf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8030000f60 tx=0x7f8030004a70 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:03.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.293+0000 7f80297fa700 1 -- 172.21.15.119:0/3954285559 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f802c052d90 con 0x7f8034158580 2024-09-13T16:34:03.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.461+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80340cf590 con 0x7f8020066730 2024-09-13T16:34:03.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.473+0000 7f80297fa700 1 -- 172.21.15.119:0/3954285559 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288585 (secure 0 0 0) 0x7f80300012a0 con 0x7f8020066730 2024-09-13T16:34:03.481 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8020066730 msgr2=0x7f8020068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8020066730 0x7f8020068bf0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f8030000f60 tx=0x7f8030004a70 comp rx=0 tx=0).stop 2024-09-13T16:34:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034158580 msgr2=0x7f8034156660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034158580 0x7f8034156660 secure :-1 s=READY pgs=1574 cs=0 l=1 rev1=1 crypto rx=0x7f802c012000 tx=0x7f802c004af0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 shutdown_connections 2024-09-13T16:34:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8020066730 0x7f8020068bf0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8034158580 0x7f8034156660 unknown :-1 s=CLOSED pgs=1574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f80340cf130 0x7f8034151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 --2- 172.21.15.119:0/3954285559 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f80340cd7d0 0x7f803414fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 >> 172.21.15.119:0/3954285559 conn(0x7f8034070cc0 msgr2=0x7f803406ab70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.477+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 shutdown_connections 2024-09-13T16:34:03.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.481+0000 7f8039941700 1 -- 172.21.15.119:0/3954285559 wait complete. 2024-09-13T16:34:03.486 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:34:03.497 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":370,"stamp":"2024-09-13T16:34:01.608232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":154,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":260828,"kb_used_data":11860,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754713892,"statfs":{"total":773094113280,"available":772827025408,"internally_reserved":0,"allocated":12144640,"data_stored":6197752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001124"},"pg_stats":[{"pgid":"16.3f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070114+0000","last_change":"2024-09-13T16:33:59.066124+0000","last_active":"2024-09-13T16:33:59.070114+0000","last_peered":"2024-09-13T16:33:59.070114+0000","last_clean":"2024-09-13T16:33:59.070114+0000","last_became_active":"2024-09-13T16:33:59.064047+0000","last_became_peered":"2024-09-13T16:33:59.064047+0000","last_unstale":"2024-09-13T16:33:59.070114+0000","last_undegraded":"2024-09-13T16:33:59.070114+0000","last_fullsized":"2024-09-13T16:33:59.070114+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:58.249905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.068005+0000","last_change":"2024-09-13T16:33:59.060600+0000","last_active":"2024-09-13T16:33:59.068005+0000","last_peered":"2024-09-13T16:33:59.068005+0000","last_clean":"2024-09-13T16:33:59.068005+0000","last_became_active":"2024-09-13T16:33:59.060336+0000","last_became_peered":"2024-09-13T16:33:59.060336+0000","last_unstale":"2024-09-13T16:33:59.068005+0000","last_undegraded":"2024-09-13T16:33:59.068005+0000","last_fullsized":"2024-09-13T16:33:59.068005+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:29.912089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.3d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070399+0000","last_change":"2024-09-13T16:33:59.066828+0000","last_active":"2024-09-13T16:33:59.070399+0000","last_peered":"2024-09-13T16:33:59.070399+0000","last_clean":"2024-09-13T16:33:59.070399+0000","last_became_active":"2024-09-13T16:33:59.066566+0000","last_became_peered":"2024-09-13T16:33:59.066566+0000","last_unstale":"2024-09-13T16:33:59.070399+0000","last_undegraded":"2024-09-13T16:33:59.070399+0000","last_fullsized":"2024-09-13T16:33:59.070399+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:47:04.673288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.3c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048421+0000","last_change":"2024-09-13T16:33:59.056833+0000","last_active":"2024-09-13T16:34:00.048421+0000","last_peered":"2024-09-13T16:34:00.048421+0000","last_clean":"2024-09-13T16:34:00.048421+0000","last_became_active":"2024-09-13T16:33:59.056582+0000","last_became_peered":"2024-09-13T16:33:59.056582+0000","last_unstale":"2024-09-13T16:34:00.048421+0000","last_undegraded":"2024-09-13T16:34:00.048421+0000","last_fullsized":"2024-09-13T16:34:00.048421+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:09:18.392057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.3b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073463+0000","last_change":"2024-09-13T16:33:59.068592+0000","last_active":"2024-09-13T16:33:59.073463+0000","last_peered":"2024-09-13T16:33:59.073463+0000","last_clean":"2024-09-13T16:33:59.073463+0000","last_became_active":"2024-09-13T16:33:59.064783+0000","last_became_peered":"2024-09-13T16:33:59.064783+0000","last_unstale":"2024-09-13T16:33:59.073463+0000","last_undegraded":"2024-09-13T16:33:59.073463+0000","last_fullsized":"2024-09-13T16:33:59.073463+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:28:29.912089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.3a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.067499+0000","last_change":"2024-09-13T16:33:59.060319+0000","last_active":"2024-09-13T16:33:59.067499+0000","last_peered":"2024-09-13T16:33:59.067499+0000","last_clean":"2024-09-13T16:33:59.067499+0000","last_became_active":"2024-09-13T16:33:59.060048+0000","last_became_peered":"2024-09-13T16:33:59.060048+0000","last_unstale":"2024-09-13T16:33:59.067499+0000","last_undegraded":"2024-09-13T16:33:59.067499+0000","last_fullsized":"2024-09-13T16:33:59.067499+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:45.989345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.39","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.067402+0000","last_change":"2024-09-13T16:33:59.060299+0000","last_active":"2024-09-13T16:33:59.067402+0000","last_peered":"2024-09-13T16:33:59.067402+0000","last_clean":"2024-09-13T16:33:59.067402+0000","last_became_active":"2024-09-13T16:33:59.059999+0000","last_became_peered":"2024-09-13T16:33:59.059999+0000","last_unstale":"2024-09-13T16:33:59.067402+0000","last_undegraded":"2024-09-13T16:33:59.067402+0000","last_fullsized":"2024-09-13T16:33:59.067402+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:11.323458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.38","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356303+0000","last_change":"2024-09-13T16:33:59.059359+0000","last_active":"2024-09-13T16:34:00.356303+0000","last_peered":"2024-09-13T16:34:00.356303+0000","last_clean":"2024-09-13T16:34:00.356303+0000","last_became_active":"2024-09-13T16:33:59.059027+0000","last_became_peered":"2024-09-13T16:33:59.059027+0000","last_unstale":"2024-09-13T16:34:00.356303+0000","last_undegraded":"2024-09-13T16:34:00.356303+0000","last_fullsized":"2024-09-13T16:34:00.356303+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:52:23.865851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.37","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.052062+0000","last_change":"2024-09-13T16:33:59.048164+0000","last_active":"2024-09-13T16:33:59.052062+0000","last_peered":"2024-09-13T16:33:59.052062+0000","last_clean":"2024-09-13T16:33:59.052062+0000","last_became_active":"2024-09-13T16:33:59.047585+0000","last_became_peered":"2024-09-13T16:33:59.047585+0000","last_unstale":"2024-09-13T16:33:59.052062+0000","last_undegraded":"2024-09-13T16:33:59.052062+0000","last_fullsized":"2024-09-13T16:33:59.052062+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:24:39.261364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.36","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073329+0000","last_change":"2024-09-13T16:33:59.063750+0000","last_active":"2024-09-13T16:33:59.073329+0000","last_peered":"2024-09-13T16:33:59.073329+0000","last_clean":"2024-09-13T16:33:59.073329+0000","last_became_active":"2024-09-13T16:33:59.061925+0000","last_became_peered":"2024-09-13T16:33:59.061925+0000","last_unstale":"2024-09-13T16:33:59.073329+0000","last_undegraded":"2024-09-13T16:33:59.073329+0000","last_fullsized":"2024-09-13T16:33:59.073329+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:50:11.776642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.35","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250492+0000","last_change":"2024-09-13T16:33:59.068555+0000","last_active":"2024-09-13T16:34:00.250492+0000","last_peered":"2024-09-13T16:34:00.250492+0000","last_clean":"2024-09-13T16:34:00.250492+0000","last_became_active":"2024-09-13T16:33:59.068231+0000","last_became_peered":"2024-09-13T16:33:59.068231+0000","last_unstale":"2024-09-13T16:34:00.250492+0000","last_undegraded":"2024-09-13T16:34:00.250492+0000","last_fullsized":"2024-09-13T16:34:00.250492+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:11:17.602566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.34","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049470+0000","last_change":"2024-09-13T16:33:59.060294+0000","last_active":"2024-09-13T16:34:00.049470+0000","last_peered":"2024-09-13T16:34:00.049470+0000","last_clean":"2024-09-13T16:34:00.049470+0000","last_became_active":"2024-09-13T16:33:59.059510+0000","last_became_peered":"2024-09-13T16:33:59.059510+0000","last_unstale":"2024-09-13T16:34:00.049470+0000","last_undegraded":"2024-09-13T16:34:00.049470+0000","last_fullsized":"2024-09-13T16:34:00.049470+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:30.844442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.33","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048226+0000","last_change":"2024-09-13T16:33:59.066489+0000","last_active":"2024-09-13T16:34:00.048226+0000","last_peered":"2024-09-13T16:34:00.048226+0000","last_clean":"2024-09-13T16:34:00.048226+0000","last_became_active":"2024-09-13T16:33:59.066256+0000","last_became_peered":"2024-09-13T16:33:59.066256+0000","last_unstale":"2024-09-13T16:34:00.048226+0000","last_undegraded":"2024-09-13T16:34:00.048226+0000","last_fullsized":"2024-09-13T16:34:00.048226+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:27.994724+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.32","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050426+0000","last_change":"2024-09-13T16:33:59.064257+0000","last_active":"2024-09-13T16:34:00.050426+0000","last_peered":"2024-09-13T16:34:00.050426+0000","last_clean":"2024-09-13T16:34:00.050426+0000","last_became_active":"2024-09-13T16:33:59.063906+0000","last_became_peered":"2024-09-13T16:33:59.063906+0000","last_unstale":"2024-09-13T16:34:00.050426+0000","last_undegraded":"2024-09-13T16:34:00.050426+0000","last_fullsized":"2024-09-13T16:34:00.050426+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:10.886581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.31","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050532+0000","last_change":"2024-09-13T16:33:59.069361+0000","last_active":"2024-09-13T16:34:00.050532+0000","last_peered":"2024-09-13T16:34:00.050532+0000","last_clean":"2024-09-13T16:34:00.050532+0000","last_became_active":"2024-09-13T16:33:59.069122+0000","last_became_peered":"2024-09-13T16:33:59.069122+0000","last_unstale":"2024-09-13T16:34:00.050532+0000","last_undegraded":"2024-09-13T16:34:00.050532+0000","last_fullsized":"2024-09-13T16:34:00.050532+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:17.902369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.30","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050362+0000","last_change":"2024-09-13T16:33:59.059847+0000","last_active":"2024-09-13T16:34:00.050362+0000","last_peered":"2024-09-13T16:34:00.050362+0000","last_clean":"2024-09-13T16:34:00.050362+0000","last_became_active":"2024-09-13T16:33:59.059546+0000","last_became_peered":"2024-09-13T16:33:59.059546+0000","last_unstale":"2024-09-13T16:34:00.050362+0000","last_undegraded":"2024-09-13T16:34:00.050362+0000","last_fullsized":"2024-09-13T16:34:00.050362+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:03:06.230871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.2f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070133+0000","last_change":"2024-09-13T16:33:59.066611+0000","last_active":"2024-09-13T16:33:59.070133+0000","last_peered":"2024-09-13T16:33:59.070133+0000","last_clean":"2024-09-13T16:33:59.070133+0000","last_became_active":"2024-09-13T16:33:59.066370+0000","last_became_peered":"2024-09-13T16:33:59.066370+0000","last_unstale":"2024-09-13T16:33:59.070133+0000","last_undegraded":"2024-09-13T16:33:59.070133+0000","last_fullsized":"2024-09-13T16:33:59.070133+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:18:56.153678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2e","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070192+0000","last_change":"2024-09-13T16:33:59.064150+0000","last_active":"2024-09-13T16:33:59.070192+0000","last_peered":"2024-09-13T16:33:59.070192+0000","last_clean":"2024-09-13T16:33:59.070192+0000","last_became_active":"2024-09-13T16:33:59.063883+0000","last_became_peered":"2024-09-13T16:33:59.063883+0000","last_unstale":"2024-09-13T16:33:59.070192+0000","last_undegraded":"2024-09-13T16:33:59.070192+0000","last_fullsized":"2024-09-13T16:33:59.070192+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:53.651102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.2d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070414+0000","last_change":"2024-09-13T16:33:59.069179+0000","last_active":"2024-09-13T16:33:59.070414+0000","last_peered":"2024-09-13T16:33:59.070414+0000","last_clean":"2024-09-13T16:33:59.070414+0000","last_became_active":"2024-09-13T16:33:59.067982+0000","last_became_peered":"2024-09-13T16:33:59.067982+0000","last_unstale":"2024-09-13T16:33:59.070414+0000","last_undegraded":"2024-09-13T16:33:59.070414+0000","last_fullsized":"2024-09-13T16:33:59.070414+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:06:08.465601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.2c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049444+0000","last_change":"2024-09-13T16:33:59.060484+0000","last_active":"2024-09-13T16:34:00.049444+0000","last_peered":"2024-09-13T16:34:00.049444+0000","last_clean":"2024-09-13T16:34:00.049444+0000","last_became_active":"2024-09-13T16:33:59.059850+0000","last_became_peered":"2024-09-13T16:33:59.059850+0000","last_unstale":"2024-09-13T16:34:00.049444+0000","last_undegraded":"2024-09-13T16:34:00.049444+0000","last_fullsized":"2024-09-13T16:34:00.049444+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:20:04.750350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357033+0000","last_change":"2024-09-13T16:33:59.069430+0000","last_active":"2024-09-13T16:34:00.357033+0000","last_peered":"2024-09-13T16:34:00.357033+0000","last_clean":"2024-09-13T16:34:00.357033+0000","last_became_active":"2024-09-13T16:33:59.069176+0000","last_became_peered":"2024-09-13T16:33:59.069176+0000","last_unstale":"2024-09-13T16:34:00.357033+0000","last_undegraded":"2024-09-13T16:34:00.357033+0000","last_fullsized":"2024-09-13T16:34:00.357033+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:44:46.662453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2a","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073533+0000","last_change":"2024-09-13T16:33:59.069799+0000","last_active":"2024-09-13T16:33:59.073533+0000","last_peered":"2024-09-13T16:33:59.073533+0000","last_clean":"2024-09-13T16:33:59.073533+0000","last_became_active":"2024-09-13T16:33:59.069570+0000","last_became_peered":"2024-09-13T16:33:59.069570+0000","last_unstale":"2024-09-13T16:33:59.073533+0000","last_undegraded":"2024-09-13T16:33:59.073533+0000","last_fullsized":"2024-09-13T16:33:59.073533+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:32:11.323458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.29","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070372+0000","last_change":"2024-09-13T16:33:59.069644+0000","last_active":"2024-09-13T16:33:59.070372+0000","last_peered":"2024-09-13T16:33:59.070372+0000","last_clean":"2024-09-13T16:33:59.070372+0000","last_became_active":"2024-09-13T16:33:59.069411+0000","last_became_peered":"2024-09-13T16:33:59.069411+0000","last_unstale":"2024-09-13T16:33:59.070372+0000","last_undegraded":"2024-09-13T16:33:59.070372+0000","last_fullsized":"2024-09-13T16:33:59.070372+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:55:20.575702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.28","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070551+0000","last_change":"2024-09-13T16:33:59.057279+0000","last_active":"2024-09-13T16:33:59.070551+0000","last_peered":"2024-09-13T16:33:59.070551+0000","last_clean":"2024-09-13T16:33:59.070551+0000","last_became_active":"2024-09-13T16:33:59.057003+0000","last_became_peered":"2024-09-13T16:33:59.057003+0000","last_unstale":"2024-09-13T16:33:59.070551+0000","last_undegraded":"2024-09-13T16:33:59.070551+0000","last_fullsized":"2024-09-13T16:33:59.070551+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:53.651102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.27","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049552+0000","last_change":"2024-09-13T16:33:59.069583+0000","last_active":"2024-09-13T16:34:00.049552+0000","last_peered":"2024-09-13T16:34:00.049552+0000","last_clean":"2024-09-13T16:34:00.049552+0000","last_became_active":"2024-09-13T16:33:59.069247+0000","last_became_peered":"2024-09-13T16:33:59.069247+0000","last_unstale":"2024-09-13T16:34:00.049552+0000","last_undegraded":"2024-09-13T16:34:00.049552+0000","last_fullsized":"2024-09-13T16:34:00.049552+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:56.299851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.26","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049486+0000","last_change":"2024-09-13T16:33:59.051893+0000","last_active":"2024-09-13T16:34:00.049486+0000","last_peered":"2024-09-13T16:34:00.049486+0000","last_clean":"2024-09-13T16:34:00.049486+0000","last_became_active":"2024-09-13T16:33:59.051666+0000","last_became_peered":"2024-09-13T16:33:59.051666+0000","last_unstale":"2024-09-13T16:34:00.049486+0000","last_undegraded":"2024-09-13T16:34:00.049486+0000","last_fullsized":"2024-09-13T16:34:00.049486+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:18:28.934344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.25","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250649+0000","last_change":"2024-09-13T16:33:59.065849+0000","last_active":"2024-09-13T16:34:00.250649+0000","last_peered":"2024-09-13T16:34:00.250649+0000","last_clean":"2024-09-13T16:34:00.250649+0000","last_became_active":"2024-09-13T16:33:59.065494+0000","last_became_peered":"2024-09-13T16:33:59.065494+0000","last_unstale":"2024-09-13T16:34:00.250649+0000","last_undegraded":"2024-09-13T16:34:00.250649+0000","last_fullsized":"2024-09-13T16:34:00.250649+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:09:49.348627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.24","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070199+0000","last_change":"2024-09-13T16:33:59.066580+0000","last_active":"2024-09-13T16:33:59.070199+0000","last_peered":"2024-09-13T16:33:59.070199+0000","last_clean":"2024-09-13T16:33:59.070199+0000","last_became_active":"2024-09-13T16:33:59.066289+0000","last_became_peered":"2024-09-13T16:33:59.066289+0000","last_unstale":"2024-09-13T16:33:59.070199+0000","last_undegraded":"2024-09-13T16:33:59.070199+0000","last_fullsized":"2024-09-13T16:33:59.070199+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:33:13.717073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.23","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357237+0000","last_change":"2024-09-13T16:33:59.060821+0000","last_active":"2024-09-13T16:34:00.357237+0000","last_peered":"2024-09-13T16:34:00.357237+0000","last_clean":"2024-09-13T16:34:00.357237+0000","last_became_active":"2024-09-13T16:33:59.060577+0000","last_became_peered":"2024-09-13T16:33:59.060577+0000","last_unstale":"2024-09-13T16:34:00.357237+0000","last_undegraded":"2024-09-13T16:34:00.357237+0000","last_fullsized":"2024-09-13T16:34:00.357237+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:11:22.417008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.22","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049039+0000","last_change":"2024-09-13T16:33:59.068373+0000","last_active":"2024-09-13T16:34:00.049039+0000","last_peered":"2024-09-13T16:34:00.049039+0000","last_clean":"2024-09-13T16:34:00.049039+0000","last_became_active":"2024-09-13T16:33:59.068122+0000","last_became_peered":"2024-09-13T16:33:59.068122+0000","last_unstale":"2024-09-13T16:34:00.049039+0000","last_undegraded":"2024-09-13T16:34:00.049039+0000","last_fullsized":"2024-09-13T16:34:00.049039+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:58:37.284055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.21","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.057809+0000","last_change":"2024-09-13T16:33:59.054579+0000","last_active":"2024-09-13T16:33:59.057809+0000","last_peered":"2024-09-13T16:33:59.057809+0000","last_clean":"2024-09-13T16:33:59.057809+0000","last_became_active":"2024-09-13T16:33:59.054259+0000","last_became_peered":"2024-09-13T16:33:59.054259+0000","last_unstale":"2024-09-13T16:33:59.057809+0000","last_undegraded":"2024-09-13T16:33:59.057809+0000","last_fullsized":"2024-09-13T16:33:59.057809+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:45.288502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.20","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356513+0000","last_change":"2024-09-13T16:33:59.050714+0000","last_active":"2024-09-13T16:34:00.356513+0000","last_peered":"2024-09-13T16:34:00.356513+0000","last_clean":"2024-09-13T16:34:00.356513+0000","last_became_active":"2024-09-13T16:33:59.050466+0000","last_became_peered":"2024-09-13T16:33:59.050466+0000","last_unstale":"2024-09-13T16:34:00.356513+0000","last_undegraded":"2024-09-13T16:34:00.356513+0000","last_fullsized":"2024-09-13T16:34:00.356513+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:40.134229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.063399+0000","last_change":"2024-09-13T16:33:59.058917+0000","last_active":"2024-09-13T16:33:59.063399+0000","last_peered":"2024-09-13T16:33:59.063399+0000","last_clean":"2024-09-13T16:33:59.063399+0000","last_became_active":"2024-09-13T16:33:59.058649+0000","last_became_peered":"2024-09-13T16:33:59.058649+0000","last_unstale":"2024-09-13T16:33:59.063399+0000","last_undegraded":"2024-09-13T16:33:59.063399+0000","last_fullsized":"2024-09-13T16:33:59.063399+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:48.715850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"110'4","reported_seq":15,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048575+0000","last_change":"2024-09-13T16:33:58.676086+0000","last_active":"2024-09-13T16:34:00.048575+0000","last_peered":"2024-09-13T16:34:00.048575+0000","last_clean":"2024-09-13T16:34:00.048575+0000","last_became_active":"2024-09-13T16:33:58.675839+0000","last_became_peered":"2024-09-13T16:33:58.675839+0000","last_unstale":"2024-09-13T16:34:00.048575+0000","last_undegraded":"2024-09-13T16:34:00.048575+0000","last_fullsized":"2024-09-13T16:34:00.048575+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:41:00.120019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356525+0000","last_change":"2024-09-13T16:33:59.059874+0000","last_active":"2024-09-13T16:34:00.356525+0000","last_peered":"2024-09-13T16:34:00.356525+0000","last_clean":"2024-09-13T16:34:00.356525+0000","last_became_active":"2024-09-13T16:33:59.059623+0000","last_became_peered":"2024-09-13T16:33:59.059623+0000","last_unstale":"2024-09-13T16:34:00.356525+0000","last_undegraded":"2024-09-13T16:34:00.356525+0000","last_fullsized":"2024-09-13T16:34:00.356525+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:05:43.311770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.1e","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357145+0000","last_change":"2024-09-13T16:33:58.673512+0000","last_active":"2024-09-13T16:34:00.357145+0000","last_peered":"2024-09-13T16:34:00.357145+0000","last_clean":"2024-09-13T16:34:00.357145+0000","last_became_active":"2024-09-13T16:33:58.673259+0000","last_became_peered":"2024-09-13T16:33:58.673259+0000","last_unstale":"2024-09-13T16:34:00.357145+0000","last_undegraded":"2024-09-13T16:34:00.357145+0000","last_fullsized":"2024-09-13T16:34:00.357145+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:09:50.075397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049105+0000","last_change":"2024-09-13T16:33:59.063570+0000","last_active":"2024-09-13T16:34:00.049105+0000","last_peered":"2024-09-13T16:34:00.049105+0000","last_clean":"2024-09-13T16:34:00.049105+0000","last_became_active":"2024-09-13T16:33:59.063235+0000","last_became_peered":"2024-09-13T16:33:59.063235+0000","last_unstale":"2024-09-13T16:34:00.049105+0000","last_undegraded":"2024-09-13T16:34:00.049105+0000","last_fullsized":"2024-09-13T16:34:00.049105+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:34:14.487933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.1d","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048983+0000","last_change":"2024-09-13T16:33:58.672877+0000","last_active":"2024-09-13T16:34:00.048983+0000","last_peered":"2024-09-13T16:34:00.048983+0000","last_clean":"2024-09-13T16:34:00.048983+0000","last_became_active":"2024-09-13T16:33:58.672445+0000","last_became_peered":"2024-09-13T16:33:58.672445+0000","last_unstale":"2024-09-13T16:34:00.048983+0000","last_undegraded":"2024-09-13T16:34:00.048983+0000","last_fullsized":"2024-09-13T16:34:00.048983+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:05:03.444757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250584+0000","last_change":"2024-09-13T16:33:59.068418+0000","last_active":"2024-09-13T16:34:00.250584+0000","last_peered":"2024-09-13T16:34:00.250584+0000","last_clean":"2024-09-13T16:34:00.250584+0000","last_became_active":"2024-09-13T16:33:59.068064+0000","last_became_peered":"2024-09-13T16:33:59.068064+0000","last_unstale":"2024-09-13T16:34:00.250584+0000","last_undegraded":"2024-09-13T16:34:00.250584+0000","last_fullsized":"2024-09-13T16:34:00.250584+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:48:55.047884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.1c","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070577+0000","last_change":"2024-09-13T16:33:58.679320+0000","last_active":"2024-09-13T16:33:59.070577+0000","last_peered":"2024-09-13T16:33:59.070577+0000","last_clean":"2024-09-13T16:33:59.070577+0000","last_became_active":"2024-09-13T16:33:58.678986+0000","last_became_peered":"2024-09-13T16:33:58.678986+0000","last_unstale":"2024-09-13T16:33:59.070577+0000","last_undegraded":"2024-09-13T16:33:59.070577+0000","last_fullsized":"2024-09-13T16:33:59.070577+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:16:15.661709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048628+0000","last_change":"2024-09-13T16:33:59.062397+0000","last_active":"2024-09-13T16:34:00.048628+0000","last_peered":"2024-09-13T16:34:00.048628+0000","last_clean":"2024-09-13T16:34:00.048628+0000","last_became_active":"2024-09-13T16:33:59.062169+0000","last_became_peered":"2024-09-13T16:33:59.062169+0000","last_unstale":"2024-09-13T16:34:00.048628+0000","last_undegraded":"2024-09-13T16:34:00.048628+0000","last_fullsized":"2024-09-13T16:34:00.048628+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:32:22.178142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1b","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050658+0000","last_change":"2024-09-13T16:33:58.678062+0000","last_active":"2024-09-13T16:34:00.050658+0000","last_peered":"2024-09-13T16:34:00.050658+0000","last_clean":"2024-09-13T16:34:00.050658+0000","last_became_active":"2024-09-13T16:33:58.677518+0000","last_became_peered":"2024-09-13T16:33:58.677518+0000","last_unstale":"2024-09-13T16:34:00.050658+0000","last_undegraded":"2024-09-13T16:34:00.050658+0000","last_fullsized":"2024-09-13T16:34:00.050658+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:37:11.642624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.067000+0000","last_change":"2024-09-13T16:33:59.065747+0000","last_active":"2024-09-13T16:33:59.067000+0000","last_peered":"2024-09-13T16:33:59.067000+0000","last_clean":"2024-09-13T16:33:59.067000+0000","last_became_active":"2024-09-13T16:33:59.064140+0000","last_became_peered":"2024-09-13T16:33:59.064140+0000","last_unstale":"2024-09-13T16:33:59.067000+0000","last_undegraded":"2024-09-13T16:33:59.067000+0000","last_fullsized":"2024-09-13T16:33:59.067000+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:13:38.995988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250183+0000","last_change":"2024-09-13T16:33:58.675556+0000","last_active":"2024-09-13T16:34:00.250183+0000","last_peered":"2024-09-13T16:34:00.250183+0000","last_clean":"2024-09-13T16:34:00.250183+0000","last_became_active":"2024-09-13T16:33:58.675268+0000","last_became_peered":"2024-09-13T16:33:58.675268+0000","last_unstale":"2024-09-13T16:34:00.250183+0000","last_undegraded":"2024-09-13T16:34:00.250183+0000","last_fullsized":"2024-09-13T16:34:00.250183+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:39.120119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070943+0000","last_change":"2024-09-13T16:33:59.068111+0000","last_active":"2024-09-13T16:33:59.070943+0000","last_peered":"2024-09-13T16:33:59.070943+0000","last_clean":"2024-09-13T16:33:59.070943+0000","last_became_active":"2024-09-13T16:33:59.067829+0000","last_became_peered":"2024-09-13T16:33:59.067829+0000","last_unstale":"2024-09-13T16:33:59.070943+0000","last_undegraded":"2024-09-13T16:33:59.070943+0000","last_fullsized":"2024-09-13T16:33:59.070943+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:50.292575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049652+0000","last_change":"2024-09-13T16:33:58.672874+0000","last_active":"2024-09-13T16:34:00.049652+0000","last_peered":"2024-09-13T16:34:00.049652+0000","last_clean":"2024-09-13T16:34:00.049652+0000","last_became_active":"2024-09-13T16:33:58.672515+0000","last_became_peered":"2024-09-13T16:33:58.672515+0000","last_unstale":"2024-09-13T16:34:00.049652+0000","last_undegraded":"2024-09-13T16:34:00.049652+0000","last_fullsized":"2024-09-13T16:34:00.049652+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:07.404188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.7","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048450+0000","last_change":"2024-09-13T16:33:59.058324+0000","last_active":"2024-09-13T16:34:00.048450+0000","last_peered":"2024-09-13T16:34:00.048450+0000","last_clean":"2024-09-13T16:34:00.048450+0000","last_became_active":"2024-09-13T16:33:59.058112+0000","last_became_peered":"2024-09-13T16:33:59.058112+0000","last_unstale":"2024-09-13T16:34:00.048450+0000","last_undegraded":"2024-09-13T16:34:00.048450+0000","last_fullsized":"2024-09-13T16:34:00.048450+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:10:44.131355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048532+0000","last_change":"2024-09-13T16:33:58.669983+0000","last_active":"2024-09-13T16:34:00.048532+0000","last_peered":"2024-09-13T16:34:00.048532+0000","last_clean":"2024-09-13T16:34:00.048532+0000","last_became_active":"2024-09-13T16:33:58.669696+0000","last_became_peered":"2024-09-13T16:33:58.669696+0000","last_unstale":"2024-09-13T16:34:00.048532+0000","last_undegraded":"2024-09-13T16:34:00.048532+0000","last_fullsized":"2024-09-13T16:34:00.048532+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:28:10.360488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.063423+0000","last_change":"2024-09-13T16:33:59.059064+0000","last_active":"2024-09-13T16:33:59.063423+0000","last_peered":"2024-09-13T16:33:59.063423+0000","last_clean":"2024-09-13T16:33:59.063423+0000","last_became_active":"2024-09-13T16:33:59.058775+0000","last_became_peered":"2024-09-13T16:33:59.058775+0000","last_unstale":"2024-09-13T16:33:59.063423+0000","last_undegraded":"2024-09-13T16:33:59.063423+0000","last_fullsized":"2024-09-13T16:33:59.063423+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:44.111538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049621+0000","last_change":"2024-09-13T16:33:58.673000+0000","last_active":"2024-09-13T16:34:00.049621+0000","last_peered":"2024-09-13T16:34:00.049621+0000","last_clean":"2024-09-13T16:34:00.049621+0000","last_became_active":"2024-09-13T16:33:58.672745+0000","last_became_peered":"2024-09-13T16:33:58.672745+0000","last_unstale":"2024-09-13T16:34:00.049621+0000","last_undegraded":"2024-09-13T16:34:00.049621+0000","last_fullsized":"2024-09-13T16:34:00.049621+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:14.708873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073195+0000","last_change":"2024-09-13T16:33:59.061564+0000","last_active":"2024-09-13T16:33:59.073195+0000","last_peered":"2024-09-13T16:33:59.073195+0000","last_clean":"2024-09-13T16:33:59.073195+0000","last_became_active":"2024-09-13T16:33:59.061201+0000","last_became_peered":"2024-09-13T16:33:59.061201+0000","last_unstale":"2024-09-13T16:33:59.073195+0000","last_undegraded":"2024-09-13T16:33:59.073195+0000","last_fullsized":"2024-09-13T16:33:59.073195+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:42.285817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.048993+0000","last_change":"2024-09-13T16:33:58.674091+0000","last_active":"2024-09-13T16:33:59.048993+0000","last_peered":"2024-09-13T16:33:59.048993+0000","last_clean":"2024-09-13T16:33:59.048993+0000","last_became_active":"2024-09-13T16:33:58.673844+0000","last_became_peered":"2024-09-13T16:33:58.673844+0000","last_unstale":"2024-09-13T16:33:59.048993+0000","last_undegraded":"2024-09-13T16:33:59.048993+0000","last_fullsized":"2024-09-13T16:33:59.048993+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:23:20.449112+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356628+0000","last_change":"2024-09-13T16:33:59.059555+0000","last_active":"2024-09-13T16:34:00.356628+0000","last_peered":"2024-09-13T16:34:00.356628+0000","last_clean":"2024-09-13T16:34:00.356628+0000","last_became_active":"2024-09-13T16:33:59.058898+0000","last_became_peered":"2024-09-13T16:33:59.058898+0000","last_unstale":"2024-09-13T16:34:00.356628+0000","last_undegraded":"2024-09-13T16:34:00.356628+0000","last_fullsized":"2024-09-13T16:34:00.356628+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:28:00.974927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.15","version":"110'3","reported_seq":14,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050918+0000","last_change":"2024-09-13T16:33:58.674922+0000","last_active":"2024-09-13T16:34:00.050918+0000","last_peered":"2024-09-13T16:34:00.050918+0000","last_clean":"2024-09-13T16:34:00.050918+0000","last_became_active":"2024-09-13T16:33:58.674541+0000","last_became_peered":"2024-09-13T16:33:58.674541+0000","last_unstale":"2024-09-13T16:34:00.050918+0000","last_undegraded":"2024-09-13T16:34:00.050918+0000","last_fullsized":"2024-09-13T16:34:00.050918+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:39:11.486110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073431+0000","last_change":"2024-09-13T16:33:59.064997+0000","last_active":"2024-09-13T16:33:59.073431+0000","last_peered":"2024-09-13T16:33:59.073431+0000","last_clean":"2024-09-13T16:33:59.073431+0000","last_became_active":"2024-09-13T16:33:59.064202+0000","last_became_peered":"2024-09-13T16:33:59.064202+0000","last_unstale":"2024-09-13T16:33:59.073431+0000","last_undegraded":"2024-09-13T16:33:59.073431+0000","last_fullsized":"2024-09-13T16:33:59.073431+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:40:45.288502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.14","version":"110'6","reported_seq":16,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073103+0000","last_change":"2024-09-13T16:33:58.675526+0000","last_active":"2024-09-13T16:33:59.073103+0000","last_peered":"2024-09-13T16:33:59.073103+0000","last_clean":"2024-09-13T16:33:59.073103+0000","last_became_active":"2024-09-13T16:33:58.675267+0000","last_became_peered":"2024-09-13T16:33:58.675267+0000","last_unstale":"2024-09-13T16:33:59.073103+0000","last_undegraded":"2024-09-13T16:33:59.073103+0000","last_fullsized":"2024-09-13T16:33:59.073103+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:42:14.456004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250814+0000","last_change":"2024-09-13T16:33:59.057018+0000","last_active":"2024-09-13T16:34:00.250814+0000","last_peered":"2024-09-13T16:34:00.250814+0000","last_clean":"2024-09-13T16:34:00.250814+0000","last_became_active":"2024-09-13T16:33:59.056616+0000","last_became_peered":"2024-09-13T16:33:59.056616+0000","last_unstale":"2024-09-13T16:34:00.250814+0000","last_undegraded":"2024-09-13T16:34:00.250814+0000","last_fullsized":"2024-09-13T16:34:00.250814+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:30:04.986467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.13","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.064386+0000","last_change":"2024-09-13T16:33:58.675819+0000","last_active":"2024-09-13T16:33:59.064386+0000","last_peered":"2024-09-13T16:33:59.064386+0000","last_clean":"2024-09-13T16:33:59.064386+0000","last_became_active":"2024-09-13T16:33:58.675567+0000","last_became_peered":"2024-09-13T16:33:58.675567+0000","last_unstale":"2024-09-13T16:33:59.064386+0000","last_undegraded":"2024-09-13T16:33:59.064386+0000","last_fullsized":"2024-09-13T16:33:59.064386+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:05:53.340550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.d","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.050922+0000","last_change":"2024-09-13T16:33:59.048159+0000","last_active":"2024-09-13T16:33:59.050922+0000","last_peered":"2024-09-13T16:33:59.050922+0000","last_clean":"2024-09-13T16:33:59.050922+0000","last_became_active":"2024-09-13T16:33:59.047959+0000","last_became_peered":"2024-09-13T16:33:59.047959+0000","last_unstale":"2024-09-13T16:33:59.050922+0000","last_undegraded":"2024-09-13T16:33:59.050922+0000","last_fullsized":"2024-09-13T16:33:59.050922+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:12:31.176030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048501+0000","last_change":"2024-09-13T16:33:58.672737+0000","last_active":"2024-09-13T16:34:00.048501+0000","last_peered":"2024-09-13T16:34:00.048501+0000","last_clean":"2024-09-13T16:34:00.048501+0000","last_became_active":"2024-09-13T16:33:58.672489+0000","last_became_peered":"2024-09-13T16:33:58.672489+0000","last_unstale":"2024-09-13T16:34:00.048501+0000","last_undegraded":"2024-09-13T16:34:00.048501+0000","last_fullsized":"2024-09-13T16:34:00.048501+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:41:15.267273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357082+0000","last_change":"2024-09-13T16:33:59.053852+0000","last_active":"2024-09-13T16:34:00.357082+0000","last_peered":"2024-09-13T16:34:00.357082+0000","last_clean":"2024-09-13T16:34:00.357082+0000","last_became_active":"2024-09-13T16:33:59.053642+0000","last_became_peered":"2024-09-13T16:33:59.053642+0000","last_unstale":"2024-09-13T16:34:00.357082+0000","last_undegraded":"2024-09-13T16:34:00.357082+0000","last_fullsized":"2024-09-13T16:34:00.357082+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:34.748525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356549+0000","last_change":"2024-09-13T16:33:58.672773+0000","last_active":"2024-09-13T16:34:00.356549+0000","last_peered":"2024-09-13T16:34:00.356549+0000","last_clean":"2024-09-13T16:34:00.356549+0000","last_became_active":"2024-09-13T16:33:58.672501+0000","last_became_peered":"2024-09-13T16:33:58.672501+0000","last_unstale":"2024-09-13T16:34:00.356549+0000","last_undegraded":"2024-09-13T16:34:00.356549+0000","last_fullsized":"2024-09-13T16:34:00.356549+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:15:45.549224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073029+0000","last_change":"2024-09-13T16:33:59.069960+0000","last_active":"2024-09-13T16:33:59.073029+0000","last_peered":"2024-09-13T16:33:59.073029+0000","last_clean":"2024-09-13T16:33:59.073029+0000","last_became_active":"2024-09-13T16:33:59.069710+0000","last_became_peered":"2024-09-13T16:33:59.069710+0000","last_unstale":"2024-09-13T16:33:59.073029+0000","last_undegraded":"2024-09-13T16:33:59.073029+0000","last_fullsized":"2024-09-13T16:33:59.073029+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:53.506939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.10","version":"110'8","reported_seq":19,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048740+0000","last_change":"2024-09-13T16:33:58.671685+0000","last_active":"2024-09-13T16:34:00.048740+0000","last_peered":"2024-09-13T16:34:00.048740+0000","last_clean":"2024-09-13T16:34:00.048740+0000","last_became_active":"2024-09-13T16:33:58.671087+0000","last_became_peered":"2024-09-13T16:33:58.671087+0000","last_unstale":"2024-09-13T16:34:00.048740+0000","last_undegraded":"2024-09-13T16:34:00.048740+0000","last_fullsized":"2024-09-13T16:34:00.048740+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:30:37.816446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.10","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356579+0000","last_change":"2024-09-13T16:33:59.059756+0000","last_active":"2024-09-13T16:34:00.356579+0000","last_peered":"2024-09-13T16:34:00.356579+0000","last_clean":"2024-09-13T16:34:00.356579+0000","last_became_active":"2024-09-13T16:33:59.059378+0000","last_became_peered":"2024-09-13T16:33:59.059378+0000","last_unstale":"2024-09-13T16:34:00.356579+0000","last_undegraded":"2024-09-13T16:34:00.356579+0000","last_fullsized":"2024-09-13T16:34:00.356579+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:03:04.036480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.f","version":"110'1","reported_seq":11,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.062699+0000","last_change":"2024-09-13T16:33:58.673399+0000","last_active":"2024-09-13T16:33:59.062699+0000","last_peered":"2024-09-13T16:33:59.062699+0000","last_clean":"2024-09-13T16:33:59.062699+0000","last_became_active":"2024-09-13T16:33:58.673124+0000","last_became_peered":"2024-09-13T16:33:58.673124+0000","last_unstale":"2024-09-13T16:33:59.062699+0000","last_undegraded":"2024-09-13T16:33:59.062699+0000","last_fullsized":"2024-09-13T16:33:59.062699+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:54:10.801326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.057784+0000","last_change":"2024-09-13T16:33:59.054489+0000","last_active":"2024-09-13T16:33:59.057784+0000","last_peered":"2024-09-13T16:33:59.057784+0000","last_clean":"2024-09-13T16:33:59.057784+0000","last_became_active":"2024-09-13T16:33:59.054165+0000","last_became_peered":"2024-09-13T16:33:59.054165+0000","last_unstale":"2024-09-13T16:33:59.057784+0000","last_undegraded":"2024-09-13T16:33:59.057784+0000","last_fullsized":"2024-09-13T16:33:59.057784+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:36:42.285817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"110'1","reported_seq":12,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050885+0000","last_change":"2024-09-13T16:33:58.678147+0000","last_active":"2024-09-13T16:34:00.050885+0000","last_peered":"2024-09-13T16:34:00.050885+0000","last_clean":"2024-09-13T16:34:00.050885+0000","last_became_active":"2024-09-13T16:33:58.677904+0000","last_became_peered":"2024-09-13T16:33:58.677904+0000","last_unstale":"2024-09-13T16:34:00.050885+0000","last_undegraded":"2024-09-13T16:34:00.050885+0000","last_fullsized":"2024-09-13T16:34:00.050885+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:12:14.708873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.13","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.073452+0000","last_change":"2024-09-13T16:33:59.068458+0000","last_active":"2024-09-13T16:33:59.073452+0000","last_peered":"2024-09-13T16:33:59.073452+0000","last_clean":"2024-09-13T16:33:59.073452+0000","last_became_active":"2024-09-13T16:33:59.064541+0000","last_became_peered":"2024-09-13T16:33:59.064541+0000","last_unstale":"2024-09-13T16:33:59.073452+0000","last_undegraded":"2024-09-13T16:33:59.073452+0000","last_fullsized":"2024-09-13T16:33:59.073452+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:45.989345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250842+0000","last_change":"2024-09-13T16:33:58.675527+0000","last_active":"2024-09-13T16:34:00.250842+0000","last_peered":"2024-09-13T16:34:00.250842+0000","last_clean":"2024-09-13T16:34:00.250842+0000","last_became_active":"2024-09-13T16:33:58.675099+0000","last_became_peered":"2024-09-13T16:33:58.675099+0000","last_unstale":"2024-09-13T16:34:00.250842+0000","last_undegraded":"2024-09-13T16:34:00.250842+0000","last_fullsized":"2024-09-13T16:34:00.250842+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:19.505280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.070283+0000","last_change":"2024-09-13T16:33:59.065925+0000","last_active":"2024-09-13T16:33:59.070283+0000","last_peered":"2024-09-13T16:33:59.070283+0000","last_clean":"2024-09-13T16:33:59.070283+0000","last_became_active":"2024-09-13T16:33:59.065675+0000","last_became_peered":"2024-09-13T16:33:59.065675+0000","last_unstale":"2024-09-13T16:33:59.070283+0000","last_undegraded":"2024-09-13T16:33:59.070283+0000","last_fullsized":"2024-09-13T16:33:59.070283+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:10:17.605895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357356+0000","last_change":"2024-09-13T16:33:58.674979+0000","last_active":"2024-09-13T16:34:00.357356+0000","last_peered":"2024-09-13T16:34:00.357356+0000","last_clean":"2024-09-13T16:34:00.357356+0000","last_became_active":"2024-09-13T16:33:58.674706+0000","last_became_peered":"2024-09-13T16:33:58.674706+0000","last_unstale":"2024-09-13T16:34:00.357356+0000","last_undegraded":"2024-09-13T16:34:00.357356+0000","last_fullsized":"2024-09-13T16:34:00.357356+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:11.872087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.15","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.066983+0000","last_change":"2024-09-13T16:33:59.063871+0000","last_active":"2024-09-13T16:33:59.066983+0000","last_peered":"2024-09-13T16:33:59.066983+0000","last_clean":"2024-09-13T16:33:59.066983+0000","last_became_active":"2024-09-13T16:33:59.063378+0000","last_became_peered":"2024-09-13T16:33:59.063378+0000","last_unstale":"2024-09-13T16:33:59.066983+0000","last_undegraded":"2024-09-13T16:33:59.066983+0000","last_fullsized":"2024-09-13T16:33:59.066983+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:02:10.897741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356534+0000","last_change":"2024-09-13T16:33:58.667018+0000","last_active":"2024-09-13T16:34:00.356534+0000","last_peered":"2024-09-13T16:34:00.356534+0000","last_clean":"2024-09-13T16:34:00.356534+0000","last_became_active":"2024-09-13T16:33:58.666768+0000","last_became_peered":"2024-09-13T16:33:58.666768+0000","last_unstale":"2024-09-13T16:34:00.356534+0000","last_undegraded":"2024-09-13T16:34:00.356534+0000","last_fullsized":"2024-09-13T16:34:00.356534+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:24:35.079021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048215+0000","last_change":"2024-09-13T16:33:59.052338+0000","last_active":"2024-09-13T16:34:00.048215+0000","last_peered":"2024-09-13T16:34:00.048215+0000","last_clean":"2024-09-13T16:34:00.048215+0000","last_became_active":"2024-09-13T16:33:59.052094+0000","last_became_peered":"2024-09-13T16:33:59.052094+0000","last_unstale":"2024-09-13T16:34:00.048215+0000","last_undegraded":"2024-09-13T16:34:00.048215+0000","last_fullsized":"2024-09-13T16:34:00.048215+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:28.168051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.042041+0000","last_change":"2024-09-13T16:33:58.677489+0000","last_active":"2024-09-13T16:33:59.042041+0000","last_peered":"2024-09-13T16:33:59.042041+0000","last_clean":"2024-09-13T16:33:59.042041+0000","last_became_active":"2024-09-13T16:33:58.676929+0000","last_became_peered":"2024-09-13T16:33:58.676929+0000","last_unstale":"2024-09-13T16:33:59.042041+0000","last_undegraded":"2024-09-13T16:33:59.042041+0000","last_fullsized":"2024-09-13T16:33:59.042041+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:52:20.735767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357064+0000","last_change":"2024-09-13T16:33:59.064700+0000","last_active":"2024-09-13T16:34:00.357064+0000","last_peered":"2024-09-13T16:34:00.357064+0000","last_clean":"2024-09-13T16:34:00.357064+0000","last_became_active":"2024-09-13T16:33:59.064417+0000","last_became_peered":"2024-09-13T16:33:59.064417+0000","last_unstale":"2024-09-13T16:34:00.357064+0000","last_undegraded":"2024-09-13T16:34:00.357064+0000","last_fullsized":"2024-09-13T16:34:00.357064+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:07:41.675312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357037+0000","last_change":"2024-09-13T16:33:58.666731+0000","last_active":"2024-09-13T16:34:00.357037+0000","last_peered":"2024-09-13T16:34:00.357037+0000","last_clean":"2024-09-13T16:34:00.357037+0000","last_became_active":"2024-09-13T16:33:58.666452+0000","last_became_peered":"2024-09-13T16:33:58.666452+0000","last_unstale":"2024-09-13T16:34:00.357037+0000","last_undegraded":"2024-09-13T16:34:00.357037+0000","last_fullsized":"2024-09-13T16:34:00.357037+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:16:38.543263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":9,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.066939+0000","last_change":"2024-09-13T16:33:59.063297+0000","last_active":"2024-09-13T16:33:59.066939+0000","last_peered":"2024-09-13T16:33:59.066939+0000","last_clean":"2024-09-13T16:33:59.066939+0000","last_became_active":"2024-09-13T16:33:59.063048+0000","last_became_peered":"2024-09-13T16:33:59.063048+0000","last_unstale":"2024-09-13T16:33:59.066939+0000","last_undegraded":"2024-09-13T16:33:59.066939+0000","last_fullsized":"2024-09-13T16:33:59.066939+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:36:51.336980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.7","version":"110'2","reported_seq":12,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.063274+0000","last_change":"2024-09-13T16:33:58.673962+0000","last_active":"2024-09-13T16:33:59.063274+0000","last_peered":"2024-09-13T16:33:59.063274+0000","last_clean":"2024-09-13T16:33:59.063274+0000","last_became_active":"2024-09-13T16:33:58.673699+0000","last_became_peered":"2024-09-13T16:33:58.673699+0000","last_unstale":"2024-09-13T16:33:59.063274+0000","last_undegraded":"2024-09-13T16:33:59.063274+0000","last_fullsized":"2024-09-13T16:33:59.063274+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:32.457259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048996+0000","last_change":"2024-09-13T16:33:59.058435+0000","last_active":"2024-09-13T16:34:00.048996+0000","last_peered":"2024-09-13T16:34:00.048996+0000","last_clean":"2024-09-13T16:34:00.048996+0000","last_became_active":"2024-09-13T16:33:59.058192+0000","last_became_peered":"2024-09-13T16:33:59.058192+0000","last_unstale":"2024-09-13T16:34:00.048996+0000","last_undegraded":"2024-09-13T16:34:00.048996+0000","last_fullsized":"2024-09-13T16:34:00.048996+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:38:15.680793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.049087+0000","last_change":"2024-09-13T16:33:58.675846+0000","last_active":"2024-09-13T16:33:59.049087+0000","last_peered":"2024-09-13T16:33:59.049087+0000","last_clean":"2024-09-13T16:33:59.049087+0000","last_became_active":"2024-09-13T16:33:58.675582+0000","last_became_peered":"2024-09-13T16:33:58.675582+0000","last_unstale":"2024-09-13T16:33:59.049087+0000","last_undegraded":"2024-09-13T16:33:59.049087+0000","last_fullsized":"2024-09-13T16:33:59.049087+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:08.439890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1a","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.048261+0000","last_change":"2024-09-13T16:33:59.052217+0000","last_active":"2024-09-13T16:34:00.048261+0000","last_peered":"2024-09-13T16:34:00.048261+0000","last_clean":"2024-09-13T16:34:00.048261+0000","last_became_active":"2024-09-13T16:33:59.051961+0000","last_became_peered":"2024-09-13T16:33:59.051961+0000","last_unstale":"2024-09-13T16:34:00.048261+0000","last_undegraded":"2024-09-13T16:34:00.048261+0000","last_fullsized":"2024-09-13T16:34:00.048261+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:07:24.319606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.042977+0000","last_change":"2024-09-13T16:33:58.675765+0000","last_active":"2024-09-13T16:33:59.042977+0000","last_peered":"2024-09-13T16:33:59.042977+0000","last_clean":"2024-09-13T16:33:59.042977+0000","last_became_active":"2024-09-13T16:33:58.675506+0000","last_became_peered":"2024-09-13T16:33:58.675506+0000","last_unstale":"2024-09-13T16:33:59.042977+0000","last_undegraded":"2024-09-13T16:33:59.042977+0000","last_fullsized":"2024-09-13T16:33:59.042977+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:51:40.216628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1b","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250450+0000","last_change":"2024-09-13T16:33:59.073444+0000","last_active":"2024-09-13T16:34:00.250450+0000","last_peered":"2024-09-13T16:34:00.250450+0000","last_clean":"2024-09-13T16:34:00.250450+0000","last_became_active":"2024-09-13T16:33:59.073253+0000","last_became_peered":"2024-09-13T16:33:59.073253+0000","last_unstale":"2024-09-13T16:34:00.250450+0000","last_undegraded":"2024-09-13T16:34:00.250450+0000","last_fullsized":"2024-09-13T16:34:00.250450+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:45:02.185471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.051551+0000","last_change":"2024-09-13T16:33:58.677431+0000","last_active":"2024-09-13T16:33:59.051551+0000","last_peered":"2024-09-13T16:33:59.051551+0000","last_clean":"2024-09-13T16:33:59.051551+0000","last_became_active":"2024-09-13T16:33:58.677102+0000","last_became_peered":"2024-09-13T16:33:58.677102+0000","last_unstale":"2024-09-13T16:33:59.051551+0000","last_undegraded":"2024-09-13T16:33:59.051551+0000","last_fullsized":"2024-09-13T16:33:59.051551+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:59:29.138171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1c","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356861+0000","last_change":"2024-09-13T16:33:59.063747+0000","last_active":"2024-09-13T16:34:00.356861+0000","last_peered":"2024-09-13T16:34:00.356861+0000","last_clean":"2024-09-13T16:34:00.356861+0000","last_became_active":"2024-09-13T16:33:59.063517+0000","last_became_peered":"2024-09-13T16:33:59.063517+0000","last_unstale":"2024-09-13T16:34:00.356861+0000","last_undegraded":"2024-09-13T16:34:00.356861+0000","last_fullsized":"2024-09-13T16:34:00.356861+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:21:28.881958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.049278+0000","last_change":"2024-09-13T16:33:58.670084+0000","last_active":"2024-09-13T16:34:00.049278+0000","last_peered":"2024-09-13T16:34:00.049278+0000","last_clean":"2024-09-13T16:34:00.049278+0000","last_became_active":"2024-09-13T16:33:58.669773+0000","last_became_peered":"2024-09-13T16:33:58.669773+0000","last_unstale":"2024-09-13T16:34:00.049278+0000","last_undegraded":"2024-09-13T16:34:00.049278+0000","last_fullsized":"2024-09-13T16:34:00.049278+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:57:24.815735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1d","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356853+0000","last_change":"2024-09-13T16:33:59.064218+0000","last_active":"2024-09-13T16:34:00.356853+0000","last_peered":"2024-09-13T16:34:00.356853+0000","last_clean":"2024-09-13T16:34:00.356853+0000","last_became_active":"2024-09-13T16:33:59.063719+0000","last_became_peered":"2024-09-13T16:33:59.063719+0000","last_unstale":"2024-09-13T16:34:00.356853+0000","last_undegraded":"2024-09-13T16:34:00.356853+0000","last_fullsized":"2024-09-13T16:34:00.356853+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:53:20.519391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356773+0000","last_change":"2024-09-13T16:33:58.674249+0000","last_active":"2024-09-13T16:34:00.356773+0000","last_peered":"2024-09-13T16:34:00.356773+0000","last_clean":"2024-09-13T16:34:00.356773+0000","last_became_active":"2024-09-13T16:33:58.674009+0000","last_became_peered":"2024-09-13T16:33:58.674009+0000","last_unstale":"2024-09-13T16:34:00.356773+0000","last_undegraded":"2024-09-13T16:34:00.356773+0000","last_fullsized":"2024-09-13T16:34:00.356773+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:33:30.785733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.1e","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.050817+0000","last_change":"2024-09-13T16:33:59.068317+0000","last_active":"2024-09-13T16:34:00.050817+0000","last_peered":"2024-09-13T16:34:00.050817+0000","last_clean":"2024-09-13T16:34:00.050817+0000","last_became_active":"2024-09-13T16:33:59.068069+0000","last_became_peered":"2024-09-13T16:33:59.068069+0000","last_unstale":"2024-09-13T16:34:00.050817+0000","last_undegraded":"2024-09-13T16:34:00.050817+0000","last_fullsized":"2024-09-13T16:34:00.050817+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:40:39.928275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":10,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.042315+0000","last_change":"2024-09-13T16:33:58.675706+0000","last_active":"2024-09-13T16:33:59.042315+0000","last_peered":"2024-09-13T16:33:59.042315+0000","last_clean":"2024-09-13T16:33:59.042315+0000","last_became_active":"2024-09-13T16:33:58.675438+0000","last_became_peered":"2024-09-13T16:33:58.675438+0000","last_unstale":"2024-09-13T16:33:59.042315+0000","last_undegraded":"2024-09-13T16:33:59.042315+0000","last_fullsized":"2024-09-13T16:33:59.042315+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:59:49.586862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1f","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250892+0000","last_change":"2024-09-13T16:33:59.072306+0000","last_active":"2024-09-13T16:34:00.250892+0000","last_peered":"2024-09-13T16:34:00.250892+0000","last_clean":"2024-09-13T16:34:00.250892+0000","last_became_active":"2024-09-13T16:33:59.071890+0000","last_became_peered":"2024-09-13T16:33:59.071890+0000","last_unstale":"2024-09-13T16:34:00.250892+0000","last_undegraded":"2024-09-13T16:34:00.250892+0000","last_fullsized":"2024-09-13T16:34:00.250892+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:58:34.150180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.356812+0000","last_change":"2024-09-13T16:33:58.672758+0000","last_active":"2024-09-13T16:34:00.356812+0000","last_peered":"2024-09-13T16:34:00.356812+0000","last_clean":"2024-09-13T16:34:00.356812+0000","last_became_active":"2024-09-13T16:33:58.672465+0000","last_became_peered":"2024-09-13T16:33:58.672465+0000","last_unstale":"2024-09-13T16:34:00.356812+0000","last_undegraded":"2024-09-13T16:34:00.356812+0000","last_fullsized":"2024-09-13T16:34:00.356812+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:43:08.560580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":10,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.250183+0000","last_change":"2024-09-13T16:33:59.056838+0000","last_active":"2024-09-13T16:34:00.250183+0000","last_peered":"2024-09-13T16:34:00.250183+0000","last_clean":"2024-09-13T16:34:00.250183+0000","last_became_active":"2024-09-13T16:33:59.056411+0000","last_became_peered":"2024-09-13T16:33:59.056411+0000","last_unstale":"2024-09-13T16:34:00.250183+0000","last_undegraded":"2024-09-13T16:34:00.250183+0000","last_fullsized":"2024-09-13T16:34:00.250183+0000","mapping_epoch":109,"log_start":"0'0","ondisk_log_start":"0'0","created":109,"last_epoch_clean":110,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:58.642306+0000","last_clean_scrub_stamp":"2024-09-13T16:33:58.642306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:51:58.823269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.e","version":"110'2","reported_seq":13,"reported_epoch":111,"state":"active+clean","last_fresh":"2024-09-13T16:34:00.357271+0000","last_change":"2024-09-13T16:33:58.667319+0000","last_active":"2024-09-13T16:34:00.357271+0000","last_peered":"2024-09-13T16:34:00.357271+0000","last_clean":"2024-09-13T16:34:00.357271+0000","last_became_active":"2024-09-13T16:33:58.666989+0000","last_became_peered":"2024-09-13T16:33:58.666989+0000","last_unstale":"2024-09-13T16:34:00.357271+0000","last_undegraded":"2024-09-13T16:34:00.357271+0000","last_fullsized":"2024-09-13T16:34:00.357271+0000","mapping_epoch":108,"log_start":"0'0","ondisk_log_start":"0'0","created":108,"last_epoch_clean":109,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:33:57.628195+0000","last_clean_scrub_stamp":"2024-09-13T16:33:57.628195+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:26:38.406060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":244,"reported_epoch":110,"state":"active+clean","last_fresh":"2024-09-13T16:33:59.050832+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:33:59.050832+0000","last_peered":"2024-09-13T16:33:59.050832+0000","last_clean":"2024-09-13T16:33:59.050832+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:33:59.050832+0000","last_undegraded":"2024-09-13T16:33:59.050832+0000","last_fullsized":"2024-09-13T16:33:59.050832+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":16,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":15,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542253,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33048,"kb_used_data":1928,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338792,"statfs":{"total":96636764160,"available":96602923008,"internally_reserved":0,"allocated":1974272,"data_stored":1222857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.83799999999999997,"15min":0.84999999999999998},"min":{"1min":0.55100000000000005,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1100000000000001,"5min":1.357,"15min":1.357},"last":0.91800000000000004},{"interface":"front","average":{"1min":0.85999999999999999,"5min":0.85599999999999998,"15min":0.85299999999999998},"min":{"1min":0.59699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.0609999999999999,"5min":1.163,"15min":1.3460000000000001},"last":0.94399999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.81200000000000006,"5min":0.85099999999999998,"15min":0.85099999999999998},"min":{"1min":0.58499999999999996,"5min":0.501,"15min":0.501},"max":{"1min":1.1459999999999999,"5min":1.569,"15min":1.569},"last":0.89100000000000001},{"interface":"front","average":{"1min":0.874,"5min":0.85999999999999999,"15min":0.85399999999999998},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1719999999999999,"5min":1.4219999999999999,"15min":1.4219999999999999},"last":1.0900000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.85199999999999998,"15min":0.878},"min":{"1min":0.67200000000000004,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.1919999999999999,"5min":1.242,"15min":1.242},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.89700000000000002,"15min":0.85499999999999998},"min":{"1min":0.60499999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.024,"5min":1.339,"15min":1.339},"last":0.96699999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.92400000000000004,"15min":0.98199999999999998},"min":{"1min":0.78300000000000003,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.1579999999999999,"5min":1.476,"15min":1.476},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.995,"15min":1.0009999999999999},"min":{"1min":0.754,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.1299999999999999,"5min":1.3939999999999999,"15min":1.3939999999999999},"last":0.78100000000000003}]},{"osd":4,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.85599999999999998,"15min":0.873},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.145,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.94699999999999995,"15min":1.036},"min":{"1min":0.57899999999999996,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.264,"5min":1.5549999999999999,"15min":1.589},"last":1.0189999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.95999999999999996,"15min":1.0009999999999999},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.208,"5min":1.425,"15min":1.425},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.01,"5min":1.054,"15min":1.097},"min":{"1min":0.626,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.2889999999999999,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.997}]},{"osd":6,"last update":"Fri Sep 13 16:33:27 2024","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":1.0149999999999999,"15min":1.0609999999999999},"min":{"1min":0.74099999999999999,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.1950000000000001,"5min":1.72,"15min":1.72},"last":0.86899999999999999},{"interface":"front","average":{"1min":1.0209999999999999,"5min":1.0489999999999999,"15min":1.1359999999999999},"min":{"1min":0.79300000000000004,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.218,"5min":1.623,"15min":1.6779999999999999},"last":1.0700000000000001}]}]},{"osd":6,"up_from":14,"seq":60129542252,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32468,"kb_used_data":1348,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339372,"statfs":{"total":96636764160,"available":96603516928,"internally_reserved":0,"allocated":1380352,"data_stored":627877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82099999999999995,"5min":0.82599999999999996,"15min":0.84199999999999997},"min":{"1min":0.52900000000000003,"5min":0.496,"15min":0.48299999999999998},"max":{"1min":1.256,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.84399999999999997,"5min":0.85499999999999998,"15min":0.88200000000000001},"min":{"1min":0.49099999999999999,"5min":0.47199999999999998,"15min":0.47199999999999998},"max":{"1min":1.151,"5min":1.202,"15min":1.4410000000000001},"last":0.309}]},{"osd":1,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.86099999999999999,"15min":0.84299999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.119,"5min":1.1799999999999999,"15min":1.2030000000000001},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.86399999999999999,"5min":0.90200000000000002,"15min":0.91400000000000003},"min":{"1min":0.64000000000000001,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.2010000000000001,"5min":1.411,"15min":1.411},"last":0.25900000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85199999999999998,"15min":0.87},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.503},"max":{"1min":1.1839999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.91800000000000004,"15min":0.86699999999999999},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.55500000000000005},"max":{"1min":1.167,"5min":1.276,"15min":1.276},"last":0.42699999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.88,"5min":0.95099999999999996,"15min":0.95699999999999996},"min":{"1min":0.62,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.151,"5min":1.5309999999999999,"15min":1.536},"last":0.379},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.97099999999999997,"15min":0.98299999999999998},"min":{"1min":0.71799999999999997,"5min":0.71299999999999997,"15min":0.56399999999999995},"max":{"1min":1.2410000000000001,"5min":1.3120000000000001,"15min":1.409},"last":0.48199999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.93799999999999994,"15min":0.96199999999999997},"min":{"1min":0.63500000000000001,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.2549999999999999,"5min":1.4570000000000001,"15min":1.4750000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.96599999999999997,"15min":0.95599999999999996},"min":{"1min":0.66100000000000003,"5min":0.63500000000000001,"15min":0.63500000000000001},"max":{"1min":1.3149999999999999,"5min":1.347,"15min":1.6140000000000001},"last":0.40600000000000003}]},{"osd":5,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.94899999999999995,"15min":0.92700000000000005},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.204,"5min":1.339,"15min":1.339},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.0149999999999999,"5min":1.05,"15min":1.1060000000000001},"min":{"1min":0.68700000000000006,"5min":0.67800000000000005,"15min":0.67800000000000005},"max":{"1min":1.2070000000000001,"5min":1.4490000000000001,"15min":1.6439999999999999},"last":0.44700000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:33:28 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0489999999999999,"15min":1.099},"min":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3819999999999999,"5min":1.4530000000000001,"15min":1.522},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.071,"15min":1.0600000000000001},"min":{"1min":0.746,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.2509999999999999,"5min":1.427,"15min":1.427},"last":0.50800000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542254,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1328,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1359872,"data_stored":625709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85799999999999998,"5min":0.83399999999999996,"15min":0.85299999999999998},"min":{"1min":0.56100000000000005,"5min":0.47899999999999998,"15min":0.47599999999999998},"max":{"1min":1.1479999999999999,"5min":1.1899999999999999,"15min":1.411},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.81799999999999995,"15min":0.84499999999999997},"min":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.41699999999999998},"max":{"1min":1.1659999999999999,"5min":1.2909999999999999,"15min":1.675},"last":0.69899999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.83599999999999997,"15min":0.86699999999999999},"min":{"1min":0.65800000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":1.0880000000000001,"5min":1.4079999999999999,"15min":1.4079999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.85099999999999998,"15min":0.83999999999999997},"min":{"1min":0.65600000000000003,"5min":0.52300000000000002,"15min":0.36499999999999999},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.96799999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.94899999999999995,"5min":0.90400000000000003,"15min":0.92500000000000004},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.135,"5min":1.258,"15min":1.625},"last":1.123},{"interface":"front","average":{"1min":1,"5min":0.96699999999999997,"15min":0.97799999999999998},"min":{"1min":0.78600000000000003,"5min":0.57599999999999996,"15min":0.55000000000000004},"max":{"1min":1.2010000000000001,"5min":1.516,"15min":1.552},"last":1.054}]},{"osd":3,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.91500000000000004,"15min":0.86599999999999999},"min":{"1min":0.69599999999999995,"5min":0.624,"15min":0.56100000000000005},"max":{"1min":1.2290000000000001,"5min":1.2290000000000001,"15min":1.3919999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.90200000000000002,"15min":0.97099999999999997},"min":{"1min":0.67900000000000005,"5min":0.58399999999999996,"15min":0.58399999999999996},"max":{"1min":1.1020000000000001,"5min":1.1419999999999999,"15min":3.8460000000000001},"last":1.23}]},{"osd":4,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.89600000000000002,"15min":1.008},"min":{"1min":0.59699999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.0369999999999999,"5min":1.655,"15min":3.6869999999999998},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0609999999999999,"15min":1.0620000000000001},"min":{"1min":0.88800000000000001,"5min":0.76100000000000001,"15min":0.57499999999999996},"max":{"1min":1.2669999999999999,"5min":1.347,"15min":1.601},"last":1.1619999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":0.876,"5min":0.92800000000000005,"15min":0.95599999999999996},"min":{"1min":0.68000000000000005,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.1679999999999999,"5min":1.3540000000000001,"15min":1.421},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.93500000000000005,"5min":1.002,"15min":1.052},"min":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.1899999999999999,"5min":1.73,"15min":2.21},"last":0.90300000000000002}]},{"osd":7,"last update":"Fri Sep 13 16:33:18 2024","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":1.0329999999999999,"15min":1.0349999999999999},"min":{"1min":0.64000000000000001,"5min":0.621,"15min":0.45500000000000002},"max":{"1min":1.3100000000000001,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.0289999999999999,"5min":1.0629999999999999,"15min":1.165},"min":{"1min":0.76100000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.2350000000000001,"5min":1.51,"15min":3.9830000000000001},"last":1.2969999999999999}]}]},{"osd":4,"up_from":14,"seq":60129542252,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1336,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1368064,"data_stored":625801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.90300000000000002,"5min":0.85299999999999998,"15min":0.84399999999999997},"min":{"1min":0.58699999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.1819999999999999,"5min":1.2270000000000001,"15min":1.411},"last":0.88100000000000001},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.89900000000000002,"15min":0.874},"min":{"1min":0.65500000000000003,"5min":0.52800000000000002,"15min":0.44600000000000001},"max":{"1min":1.175,"5min":1.462,"15min":2.899},"last":0.90900000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91200000000000003,"5min":0.90100000000000002,"15min":0.91600000000000004},"min":{"1min":0.627,"5min":0.55000000000000004,"15min":0.46200000000000002},"max":{"1min":1.1879999999999999,"5min":1.2490000000000001,"15min":3.085},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.88400000000000001,"15min":0.91000000000000003},"min":{"1min":0.623,"5min":0.497,"15min":0.497},"max":{"1min":1.0649999999999999,"5min":1.367,"15min":1.5720000000000001},"last":0.96799999999999997}]},{"osd":2,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.83199999999999996,"15min":0.84199999999999997},"min":{"1min":0.59299999999999997,"5min":0.433,"15min":0.433},"max":{"1min":1.228,"5min":1.228,"15min":1.4419999999999999},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.93700000000000006,"15min":0.95999999999999996},"min":{"1min":0.70099999999999996,"5min":0.60199999999999998,"15min":0.498},"max":{"1min":1.2390000000000001,"5min":1.357,"15min":3.2160000000000002},"last":0.84799999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.91200000000000003,"15min":0.94299999999999995},"min":{"1min":0.60399999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.91200000000000003,"15min":0.94599999999999995},"min":{"1min":0.65100000000000002,"5min":0.58999999999999997,"15min":0.52200000000000002},"max":{"1min":1.3979999999999999,"5min":1.3979999999999999,"15min":1.577},"last":0.80300000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.90400000000000003,"15min":0.95999999999999996},"min":{"1min":0.65500000000000003,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.0900000000000001,"5min":1.268,"15min":2.996},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.008,"5min":0.99099999999999999,"15min":1.0940000000000001},"min":{"1min":0.67100000000000004,"5min":0.495,"15min":0.495},"max":{"1min":1.3440000000000001,"5min":1.5189999999999999,"15min":2.8860000000000001},"last":1.1140000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.92200000000000004,"15min":0.93899999999999995},"min":{"1min":0.63800000000000001,"5min":0.62,"15min":0.62},"max":{"1min":1.522,"5min":1.522,"15min":1.6499999999999999},"last":1.083},{"interface":"front","average":{"1min":1.0589999999999999,"5min":1.0600000000000001,"15min":1.103},"min":{"1min":0.84899999999999998,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":1.3939999999999999,"5min":1.4450000000000001,"15min":3.1469999999999998},"last":0.998}]},{"osd":7,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0309999999999999,"15min":1.1140000000000001},"min":{"1min":0.80500000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.556,"5min":1.556,"15min":2.1230000000000002},"last":1.0449999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0409999999999999,"15min":1.1040000000000001},"min":{"1min":0.81799999999999995,"5min":0.67000000000000004,"15min":0.67000000000000004},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":1.018}]}]},{"osd":3,"up_from":14,"seq":60129542253,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32452,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339388,"statfs":{"total":96636764160,"available":96603533312,"internally_reserved":0,"allocated":1363968,"data_stored":626291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.626,"15min":0.63},"min":{"1min":0.497,"5min":0.42999999999999999,"15min":0.373},"max":{"1min":0.98499999999999999,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.69499999999999995,"5min":0.69199999999999995,"15min":0.66800000000000004},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.433},"max":{"1min":1.0469999999999999,"5min":1.161,"15min":1.161},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67400000000000004,"15min":0.64100000000000001},"min":{"1min":0.46800000000000003,"5min":0.40699999999999997,"15min":0.35499999999999998},"max":{"1min":0.85299999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.50900000000000001,"5min":0.47899999999999998,"15min":0.46800000000000003},"max":{"1min":0.98499999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.42499999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.71899999999999997,"5min":0.72199999999999998,"15min":0.70299999999999996},"min":{"1min":0.48099999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.0609999999999999,"5min":1.0649999999999999,"15min":1.27},"last":0.439},{"interface":"front","average":{"1min":0.80900000000000005,"5min":0.79700000000000004,"15min":0.77000000000000002},"min":{"1min":0.61399999999999999,"5min":0.433,"15min":0.433},"max":{"1min":1.0740000000000001,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.379}]},{"osd":4,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.93400000000000005,"15min":0.91000000000000003},"min":{"1min":0.752,"5min":0.65900000000000003,"15min":0.60499999999999998},"max":{"1min":1.23,"5min":1.29,"15min":1.8460000000000001},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.91900000000000004,"5min":0.93700000000000006,"15min":0.95299999999999996},"min":{"1min":0.72999999999999998,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.782},"last":0.748}]},{"osd":5,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.96799999999999997},"min":{"1min":0.67600000000000005,"5min":0.65900000000000003,"15min":0.59499999999999997},"max":{"1min":1.2649999999999999,"5min":1.3520000000000001,"15min":1.6299999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":1.0229999999999999,"5min":1.0369999999999999,"15min":1.079},"min":{"1min":0.74099999999999999,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.296,"5min":1.581,"15min":1.8140000000000001},"last":0.72799999999999998}]},{"osd":6,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.0620000000000001,"15min":1.109},"min":{"1min":0.69799999999999995,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":1.3660000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":0.61599999999999999},{"interface":"front","average":{"1min":1,"5min":1.044,"15min":1.0489999999999999},"min":{"1min":0.64900000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.2350000000000001,"5min":1.7030000000000001,"15min":1.95},"last":0.80300000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:33:19 2024","interfaces":[{"interface":"back","average":{"1min":1.0509999999999999,"5min":1.0940000000000001,"15min":1.0820000000000001},"min":{"1min":0.74199999999999999,"5min":0.72999999999999998,"15min":0.71399999999999997},"max":{"1min":1.306,"5min":1.657,"15min":1.889},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0489999999999999,"15min":1.056},"min":{"1min":0.85199999999999998,"5min":0.70799999999999996,"15min":0.70799999999999996},"max":{"1min":1.2350000000000001,"5min":1.5289999999999999,"15min":1.756},"last":0.85399999999999998}]}]},{"osd":2,"up_from":14,"seq":60129542254,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1363968,"data_stored":625743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.60699999999999998,"5min":0.60799999999999998,"15min":0.59399999999999997},"min":{"1min":0.47699999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.628},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.70599999999999996,"15min":0.69399999999999995},"min":{"1min":0.52000000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.169,"5min":1.169,"15min":1.169},"last":0.92600000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.67100000000000004,"15min":0.69499999999999995},"min":{"1min":0.45400000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.85099999999999998,"5min":0.995,"15min":0.997},"last":0.84199999999999997},{"interface":"front","average":{"1min":0.69899999999999995,"5min":0.71399999999999997,"15min":0.69499999999999995},"min":{"1min":0.45200000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.0529999999999999,"5min":1.147,"15min":1.147},"last":0.78800000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.70699999999999996,"15min":0.67800000000000005},"min":{"1min":0.51800000000000002,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.0389999999999999,"5min":1.119,"15min":1.119},"last":0.745},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.83399999999999996,"15min":0.80400000000000005},"min":{"1min":0.627,"5min":0.37,"15min":0.37},"max":{"1min":0.98299999999999998,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.99099999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.93700000000000006,"15min":0.94799999999999995},"min":{"1min":0.73899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.4370000000000001,"5min":1.448,"15min":1.5149999999999999},"last":1.2849999999999999},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.95299999999999996,"15min":0.96899999999999997},"min":{"1min":0.82999999999999996,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.1819999999999999,"5min":1.4330000000000001,"15min":1.8799999999999999},"last":1.0289999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95199999999999996,"15min":0.97299999999999998},"min":{"1min":0.63700000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.413,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.1799999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.97399999999999998,"15min":0.96399999999999997},"min":{"1min":0.60699999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.22,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.3799999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.94699999999999995,"15min":1.006},"min":{"1min":0.79300000000000004,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.28,"5min":1.4690000000000001,"15min":2.024},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.99299999999999999,"5min":1.028,"15min":1.0529999999999999},"min":{"1min":0.69699999999999995,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.3129999999999999,"5min":1.5249999999999999,"15min":1.6359999999999999},"last":1.3109999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:33:30 2024","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.0580000000000001,"15min":1.071},"min":{"1min":0.82499999999999996,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.357,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.079,"5min":1.0389999999999999,"15min":1.0580000000000001},"min":{"1min":0.79100000000000004,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.587,"5min":1.587,"15min":1.587},"last":1.1459999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542253,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32456,"kb_used_data":1332,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339384,"statfs":{"total":96636764160,"available":96603529216,"internally_reserved":0,"allocated":1363968,"data_stored":625743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.59599999999999997,"5min":0.61299999999999999,"15min":0.60899999999999999},"min":{"1min":0.442,"5min":0.371,"15min":0.23300000000000001},"max":{"1min":0.78200000000000003,"5min":0.95899999999999996,"15min":1.0880000000000001},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.75,"15min":0.73899999999999999},"min":{"1min":0.51000000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.93000000000000005,"5min":1.2010000000000001,"15min":1.2010000000000001},"last":0.94999999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.68999999999999995,"5min":0.67500000000000004,"15min":0.64100000000000001},"min":{"1min":0.54200000000000004,"5min":0.29699999999999999,"15min":0.26700000000000002},"max":{"1min":0.96099999999999997,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.433},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.71899999999999997,"15min":0.70899999999999996},"min":{"1min":0.52200000000000002,"5min":0.33300000000000002,"15min":0.29699999999999999},"max":{"1min":0.877,"5min":1.3129999999999999,"15min":1.458},"last":0.86099999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.79900000000000004,"5min":0.74199999999999999,"15min":0.71399999999999997},"min":{"1min":0.54800000000000004,"5min":0.39700000000000002,"15min":0.35399999999999998},"max":{"1min":1.1100000000000001,"5min":1.4019999999999999,"15min":1.4019999999999999},"last":0.753},{"interface":"front","average":{"1min":0.745,"5min":0.70799999999999996,"15min":0.68999999999999995},"min":{"1min":0.501,"5min":0.44400000000000001,"15min":0.39600000000000002},"max":{"1min":1.175,"5min":1.175,"15min":1.403},"last":0.66800000000000004}]},{"osd":4,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.96999999999999997,"5min":0.93600000000000005,"15min":0.91900000000000004},"min":{"1min":0.78700000000000003,"5min":0.505,"15min":0.505},"max":{"1min":1.2190000000000001,"5min":1.4930000000000001,"15min":1.546},"last":1.02},{"interface":"front","average":{"1min":0.90400000000000003,"5min":0.90200000000000002,"15min":0.92200000000000004},"min":{"1min":0.64700000000000002,"5min":0.58099999999999996,"15min":0.51700000000000002},"max":{"1min":1.0660000000000001,"5min":1.377,"15min":2.2040000000000002},"last":0.98799999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.96999999999999997,"15min":0.95599999999999996},"min":{"1min":0.70399999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.389},"last":1.006},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.91400000000000003,"15min":0.95699999999999996},"min":{"1min":0.78300000000000003,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.131,"5min":1.238,"15min":2.3029999999999999},"last":0.82099999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":0.99399999999999999,"5min":1.006,"15min":1.0069999999999999},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.357,"5min":1.4810000000000001,"15min":1.613},"last":0.97699999999999998},{"interface":"front","average":{"1min":1.0529999999999999,"5min":1.0369999999999999,"15min":1.0209999999999999},"min":{"1min":0.80000000000000004,"5min":0.59299999999999997,"15min":0.59299999999999997},"max":{"1min":1.3380000000000001,"5min":1.548,"15min":1.698},"last":1.5029999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:33:22 2024","interfaces":[{"interface":"back","average":{"1min":1.0580000000000001,"5min":1.016,"15min":1.012},"min":{"1min":0.71599999999999997,"5min":0.64400000000000002,"15min":0.62},"max":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.0649999999999999,"5min":1.075,"15min":1.0549999999999999},"min":{"1min":0.76700000000000002,"5min":0.61599999999999999,"15min":0.59299999999999997},"max":{"1min":1.302,"5min":1.546,"15min":1.7689999999999999},"last":0.90600000000000003}]}]},{"osd":0,"up_from":14,"seq":60129542253,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33040,"kb_used_data":1924,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338800,"statfs":{"total":96636764160,"available":96602931200,"internally_reserved":0,"allocated":1970176,"data_stored":1217731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.63100000000000001,"15min":0.66000000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68899999999999995,"15min":0.66700000000000004},"min":{"1min":0.53400000000000003,"5min":0.442,"15min":0.377},"max":{"1min":0.872,"5min":1.077,"15min":1.157},"last":0.60799999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.73299999999999998,"5min":0.71399999999999997,"15min":0.71899999999999997},"min":{"1min":0.56899999999999995,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.88600000000000001,"5min":1.0609999999999999,"15min":1.202},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.78100000000000003,"5min":0.73199999999999998,"15min":0.70099999999999996},"min":{"1min":0.52300000000000002,"5min":0.42799999999999999,"15min":0.39800000000000002},"max":{"1min":1.069,"5min":1.1160000000000001,"15min":1.2629999999999999},"last":0.71999999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.73099999999999998,"15min":0.68100000000000005},"min":{"1min":0.45800000000000002,"5min":0.39200000000000002,"15min":0.312},"max":{"1min":0.97499999999999998,"5min":1.0589999999999999,"15min":1.4830000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88,"15min":0.84499999999999997},"min":{"1min":0.53300000000000003,"5min":0.48899999999999999,"15min":0.42899999999999999},"max":{"1min":1.0660000000000001,"5min":1.284,"15min":1.6100000000000001},"last":0.74299999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95799999999999996,"5min":0.95599999999999996,"15min":0.95899999999999996},"min":{"1min":0.73399999999999999,"5min":0.61299999999999999,"15min":0.57699999999999996},"max":{"1min":1.1539999999999999,"5min":1.321,"15min":1.486},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.95399999999999996,"15min":0.96099999999999997},"min":{"1min":0.70499999999999996,"5min":0.64600000000000002,"15min":0.59999999999999998},"max":{"1min":1.167,"5min":1.3169999999999999,"15min":1.704},"last":0.85099999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.997,"15min":0.98599999999999999},"min":{"1min":0.752,"5min":0.71799999999999997,"15min":0.59599999999999997},"max":{"1min":1.2689999999999999,"5min":1.5609999999999999,"15min":1.575},"last":0.67100000000000004},{"interface":"front","average":{"1min":1.0289999999999999,"5min":0.97899999999999998,"15min":0.99199999999999999},"min":{"1min":0.77400000000000002,"5min":0.52700000000000002,"15min":0.443},"max":{"1min":1.742,"5min":1.742,"15min":1.742},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":0.96299999999999997,"15min":0.97699999999999998},"min":{"1min":0.76900000000000002,"5min":0.69499999999999995,"15min":0.628},"max":{"1min":1.2190000000000001,"5min":1.3700000000000001,"15min":1.944},"last":0.77200000000000002},{"interface":"front","average":{"1min":1.0900000000000001,"5min":1.087,"15min":1.0629999999999999},"min":{"1min":0.77300000000000002,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.3480000000000001,"5min":1.5,"15min":1.5780000000000001},"last":0.80900000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:33:33 2024","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.0780000000000001,"15min":1.0609999999999999},"min":{"1min":0.96199999999999997,"5min":0.747,"15min":0.61699999999999999},"max":{"1min":1.6639999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.012},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0880000000000001,"15min":1.0629999999999999},"min":{"1min":0.91700000000000004,"5min":0.76700000000000002,"15min":0.65400000000000003},"max":{"1min":1.3460000000000001,"5min":1.51,"15min":2.0139999999999998},"last":0.89800000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:34:03.503 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:34:03.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T16:34:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05a25700 1 Processor -- start 2024-09-13T16:34:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05a25700 1 -- start start 2024-09-13T16:34:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05a25700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0f000b5670 0x7f0f00167880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05a25700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f000b5bb0 con 0x7f0f0015ccb0 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05a25700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f0015d3e0 con 0x7f0f000d0f70 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05a25700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f000d16c0 con 0x7f0f000b5670 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05224700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0f000b5670 0x7f0f00167880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05224700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0f000b5670 0x7f0f00167880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36528/0 (socket says 172.21.15.119:36528) 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05224700 1 -- 172.21.15.119:0/510877862 learned_addr learned my addr 172.21.15.119:0/510877862 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05224700 1 -- 172.21.15.119:0/510877862 >> v1:172.21.15.145:6789/0 conn(0x7f0f000d0f70 legacy=0x7f0f000b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05224700 1 -- 172.21.15.119:0/510877862 >> v1:172.21.15.119:6789/0 conn(0x7f0f0015ccb0 legacy=0x7f0f000d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.625+0000 7f0f05224700 1 -- 172.21.15.119:0/510877862 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f0015d6c0 con 0x7f0f000b5670 2024-09-13T16:34:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05224700 1 --2- 172.21.15.119:0/510877862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000b5670 0x7f0f00167880 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009e50 tx=0x7f0ef800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8765035f53eeece2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0eff7fe700 1 -- 172.21.15.119:0/510877862 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef800e070 con 0x7f0f000b5670 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0eff7fe700 1 -- 172.21.15.119:0/510877862 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef8003a40 con 0x7f0f000b5670 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0eff7fe700 1 -- 172.21.15.119:0/510877862 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8003df0 con 0x7f0f000b5670 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 -- 172.21.15.119:0/510877862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000b5670 msgr2=0x7f0f00167880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 --2- 172.21.15.119:0/510877862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000b5670 0x7f0f00167880 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f0ef8009e50 tx=0x7f0ef800b330 comp rx=0 tx=0).stop 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 -- 172.21.15.119:0/510877862 shutdown_connections 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 --2- 172.21.15.119:0/510877862 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000b5670 0x7f0f00167880 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 -- 172.21.15.119:0/510877862 >> 172.21.15.119:0/510877862 conn(0x7f0f00072240 msgr2=0x7f0f00099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 -- 172.21.15.119:0/510877862 shutdown_connections 2024-09-13T16:34:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 -- 172.21.15.119:0/510877862 wait complete. 2024-09-13T16:34:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 Processor -- start 2024-09-13T16:34:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.629+0000 7f0f05a25700 1 -- start start 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f000b5670 0x7f0f0015a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000d0f70 0x7f0f0014e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f0015ccb0 0x7f0f00155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0014eb50 con 0x7f0f000d0f70 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0015af40 con 0x7f0f0015ccb0 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00157d30 con 0x7f0f000b5670 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000d0f70 0x7f0f0014e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000d0f70 0x7f0f0014e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36542/0 (socket says 172.21.15.119:36542) 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 -- 172.21.15.119:0/1576326242 learned_addr learned my addr 172.21.15.119:0/1576326242 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05224700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f0015ccb0 0x7f0f00155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f04a23700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f000b5670 0x7f0f0015a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 -- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f000b5670 msgr2=0x7f0f0015a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f000b5670 0x7f0f0015a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 -- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f0015ccb0 msgr2=0x7f0f00155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f0015ccb0 0x7f0f00155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 -- 172.21.15.119:0/1576326242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef8009b30 con 0x7f0f000d0f70 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efffff700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000d0f70 0x7f0f0014e610 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c4e0 tx=0x7f0ef000c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efd7fa700 1 -- 172.21.15.119:0/1576326242 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0011790 con 0x7f0f000d0f70 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efd7fa700 1 -- 172.21.15.119:0/1576326242 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef0011910 con 0x7f0f000d0f70 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f00158010 con 0x7f0f000d0f70 2024-09-13T16:34:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0efd7fa700 1 -- 172.21.15.119:0/1576326242 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0011cc0 con 0x7f0f000d0f70 2024-09-13T16:34:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f001585d0 con 0x7f0f000d0f70 2024-09-13T16:34:03.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.633+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f000b4f60 con 0x7f0f000d0f70 2024-09-13T16:34:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.637+0000 7f0efd7fa700 1 -- 172.21.15.119:0/1576326242 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0ef001c460 con 0x7f0f000d0f70 2024-09-13T16:34:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.637+0000 7f0efd7fa700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0eec066730 0x7f0eec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.637+0000 7f0efd7fa700 1 -- 172.21.15.119:0/1576326242 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0ef0084060 con 0x7f0f000d0f70 2024-09-13T16:34:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.637+0000 7f0f04a23700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0eec066730 0x7f0eec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:03.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.637+0000 7f0f04a23700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0eec066730 0x7f0eec068bf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0f0006c5b0 tx=0x7f0ef4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:03.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.641+0000 7f0efd7fa700 1 -- 172.21.15.119:0/1576326242 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0ef00b5db0 con 0x7f0f000d0f70 2024-09-13T16:34:03.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.869+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f0f0015b940 con 0x7f0f000d0f70 2024-09-13T16:34:03.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.877+0000 7f0efd7fa700 1 -- 172.21.15.119:0/1576326242 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0ef00511f0 con 0x7f0f000d0f70 2024-09-13T16:34:03.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0eec066730 msgr2=0x7f0eec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0eec066730 0x7f0eec068bf0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f0f0006c5b0 tx=0x7f0ef4009040 comp rx=0 tx=0).stop 2024-09-13T16:34:03.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000d0f70 msgr2=0x7f0f0014e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:03.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000d0f70 0x7f0f0014e610 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000c4e0 tx=0x7f0ef000c7f0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 shutdown_connections 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0eec066730 0x7f0eec068bf0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0f0015ccb0 0x7f0f00155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0f000d0f70 0x7f0f0014e610 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 --2- 172.21.15.119:0/1576326242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0f000b5670 0x7f0f0015a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 >> 172.21.15.119:0/1576326242 conn(0x7f0f00072240 msgr2=0x7f0f00066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 shutdown_connections 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:03.881+0000 7f0f05a25700 1 -- 172.21.15.119:0/1576326242 wait complete. 2024-09-13T16:34:03.886 INFO:teuthology.orchestra.run.smithi119.stderr:updated caps for client.0 2024-09-13T16:34:03.899 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:34:04.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389974700 1 Processor -- start 2024-09-13T16:34:04.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389974700 1 -- start start 2024-09-13T16:34:04.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389974700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe3840cd8d0 0x7fe38406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389974700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3840705d0 con 0x7fe3840b17f0 2024-09-13T16:34:04.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389974700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe38406d0d0 con 0x7fe3840b2200 2024-09-13T16:34:04.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389974700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3840cbf10 con 0x7fe3840cd8d0 2024-09-13T16:34:04.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe383fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fe3840b17f0 0x7fe3840cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54236/0 (socket says 172.21.15.119:54236) 2024-09-13T16:34:04.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe383fff700 1 -- 172.21.15.119:0/2752927616 learned_addr learned my addr 172.21.15.119:0/2752927616 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:04.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 --2- 172.21.15.119:0/2752927616 >> v2:172.21.15.119:3300/0 conn(0x7fe3840cd8d0 0x7fe38406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3845398611 0 0) 0x7fe3840705d0 con 0x7fe3840b17f0 2024-09-13T16:34:04.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe36c0035d0 con 0x7fe3840b17f0 2024-09-13T16:34:04.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3507283947 0 0) 0x7fe36c0035d0 con 0x7fe3840b17f0 2024-09-13T16:34:04.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 >> v2:172.21.15.119:3300/0 conn(0x7fe3840cd8d0 msgr2=0x7fe38406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 --2- 172.21.15.119:0/2752927616 >> v2:172.21.15.119:3300/0 conn(0x7fe3840cd8d0 0x7fe38406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 >> v1:172.21.15.145:6789/0 conn(0x7fe3840b2200 legacy=0x7fe384070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe38406fca0 con 0x7fe3840b17f0 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fe374001670 con 0x7fe3840b17f0 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 >> v1:172.21.15.119:6789/0 conn(0x7fe3840b17f0 legacy=0x7fe3840cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe36c003a20 0x7fe36c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe36c004420 0x7fe36c0048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c005190 0x7fe36c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c00c0a0 con 0x7fe36c003a20 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c00c210 con 0x7fe36c005190 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe36c00c3d0 con 0x7fe36c004420 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 --2- 172.21.15.119:0/2752927616 >> v2:172.21.15.119:3300/0 conn(0x7fe3840cd8d0 0x7fe38406cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe36c003a20 0x7fe36c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe383fff700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe36c004420 0x7fe36c0048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389173700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c005190 0x7fe36c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 -- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe36c004420 msgr2=0x7fe36c0048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe36c004420 0x7fe36c0048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 -- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c005190 msgr2=0x7fe36c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c005190 0x7fe36c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 -- 172.21.15.119:0/2752927616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe36c0035d0 con 0x7fe36c003a20 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe389173700 1 -- 172.21.15.119:0/2752927616 reap_dead start 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe388972700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe36c003a20 0x7fe36c003e00 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe378009d20 tx=0x7fe36c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7054edf3f12547 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe378009e70 con 0x7fe36c003a20 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe37800e9a0 con 0x7fe36c003a20 2024-09-13T16:34:04.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.005+0000 7fe3837fe700 1 -- 172.21.15.119:0/2752927616 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe37800ed10 con 0x7fe36c003a20 2024-09-13T16:34:04.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 -- 172.21.15.119:0/2752927616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe36c003a20 msgr2=0x7fe36c003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe36c003a20 0x7fe36c003e00 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 crypto rx=0x7fe378009d20 tx=0x7fe36c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 -- 172.21.15.119:0/2752927616 shutdown_connections 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 --2- 172.21.15.119:0/2752927616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe36c003a20 0x7fe36c003e00 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 -- 172.21.15.119:0/2752927616 >> 172.21.15.119:0/2752927616 conn(0x7fe38406a350 msgr2=0x7fe38406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 -- 172.21.15.119:0/2752927616 shutdown_connections 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 -- 172.21.15.119:0/2752927616 wait complete. 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 Processor -- start 2024-09-13T16:34:04.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.009+0000 7fe389974700 1 -- start start 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c003a20 0x7fe38415a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3840b0a70 0x7fe384158d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe384159290 0x7fe3840ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3840b1100 con 0x7fe3840b0a70 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3840b0ef0 con 0x7fe36c003a20 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe384145690 con 0x7fe384159290 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3840b0a70 0x7fe384158d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3840b0a70 0x7fe384158d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36576/0 (socket says 172.21.15.119:36576) 2024-09-13T16:34:04.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 -- 172.21.15.119:0/381392152 learned_addr learned my addr 172.21.15.119:0/381392152 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe388972700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c003a20 0x7fe38415a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389173700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe384159290 0x7fe3840ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 -- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe384159290 msgr2=0x7fe3840ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe384159290 0x7fe3840ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 -- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c003a20 msgr2=0x7fe38415a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c003a20 0x7fe38415a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 -- 172.21.15.119:0/381392152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe384145970 con 0x7fe3840b0a70 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe383fff700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3840b0a70 0x7fe384158d50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe374007290 tx=0x7fe3740075a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe3817fa700 1 -- 172.21.15.119:0/381392152 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe374011580 con 0x7fe3840b0a70 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe3817fa700 1 -- 172.21.15.119:0/381392152 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe374011700 con 0x7fe3840b0a70 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe384145c50 con 0x7fe3840b0a70 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe3817fa700 1 -- 172.21.15.119:0/381392152 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe374011ab0 con 0x7fe3840b0a70 2024-09-13T16:34:04.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3840ae670 con 0x7fe3840b0a70 2024-09-13T16:34:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe384164610 con 0x7fe3840b0a70 2024-09-13T16:34:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.013+0000 7fe3817fa700 1 -- 172.21.15.119:0/381392152 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe374011c10 con 0x7fe3840b0a70 2024-09-13T16:34:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.017+0000 7fe3817fa700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe370066730 0x7fe370068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.017+0000 7fe3817fa700 1 -- 172.21.15.119:0/381392152 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe3740842d0 con 0x7fe3840b0a70 2024-09-13T16:34:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.017+0000 7fe388972700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe370066730 0x7fe370068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.017+0000 7fe388972700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe370066730 0x7fe370068bf0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe3780090b0 tx=0x7fe37801e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.021+0000 7fe3817fa700 1 -- 172.21.15.119:0/381392152 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe374051510 con 0x7fe3840b0a70 2024-09-13T16:34:04.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.233+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3840b2670 con 0x7fe3840b0a70 2024-09-13T16:34:04.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.233+0000 7fe3817fa700 1 -- 172.21.15.119:0/381392152 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fe374054b30 con 0x7fe3840b0a70 2024-09-13T16:34:04.237 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:04.238 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":76,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:34:02.039855+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8829":{"gid":8829,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3700300033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3700300033},{"type":"v1","addr":"172.21.15.119:6835","nonce":3700300033}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:34:04.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe370066730 msgr2=0x7fe370068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe370066730 0x7fe370068bf0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7fe3780090b0 tx=0x7fe37801e040 comp rx=0 tx=0).stop 2024-09-13T16:34:04.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3840b0a70 msgr2=0x7fe384158d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3840b0a70 0x7fe384158d50 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fe374007290 tx=0x7fe3740075a0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 shutdown_connections 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe370066730 0x7fe370068bf0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe384159290 0x7fe3840ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe3840b0a70 0x7fe384158d50 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 --2- 172.21.15.119:0/381392152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe36c003a20 0x7fe38415a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 >> 172.21.15.119:0/381392152 conn(0x7fe38406a350 msgr2=0x7fe384071080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 shutdown_connections 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.237+0000 7fe389974700 1 -- 172.21.15.119:0/381392152 wait complete. 2024-09-13T16:34:04.241 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 76 2024-09-13T16:34:04.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 76, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:33:59.032499+0000', 'modified': '2024-09-13T16:34:02.039855+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8829}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8829': {'gid': 8829, 'name': 'c', 'rank': 0, 'incarnation': 73, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.119:6835/3700300033', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 3700300033}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 3700300033}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:34:04.655 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:34:04.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:34:04.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04b4a44700 1 Processor -- start 2024-09-13T16:34:04.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04b4a44700 1 -- start start 2024-09-13T16:34:04.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04b4a44700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04b006be70 0x7f04b00b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04b4a44700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b006cff0 con 0x7f04b006e6a0 2024-09-13T16:34:04.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04b4a44700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04b006d480 con 0x7f04b00ad7e0 2024-09-13T16:34:04.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04b4a44700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04b006c3e0 con 0x7f04b006be70 2024-09-13T16:34:04.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04aed9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f04b006e6a0 0x7f04b006ea80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54252/0 (socket says 172.21.15.119:54252) 2024-09-13T16:34:04.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04aed9d700 1 -- 172.21.15.119:0/2847342754 learned_addr learned my addr 172.21.15.119:0/2847342754 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:04.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04ae59c700 1 --2- 172.21.15.119:0/2847342754 >> v2:172.21.15.145:3300/0 conn(0x7f04b006be70 0x7f04b00b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 44625476 0 0) 0x7f04b006cff0 con 0x7f04b006e6a0 2024-09-13T16:34:04.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f049c0035d0 con 0x7f04b006e6a0 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3634804633 0 0) 0x7f049c0035d0 con 0x7f04b006e6a0 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 >> v2:172.21.15.145:3300/0 conn(0x7f04b006be70 msgr2=0x7f04b00b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 --2- 172.21.15.119:0/2847342754 >> v2:172.21.15.145:3300/0 conn(0x7f04b006be70 0x7f04b00b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 >> v1:172.21.15.145:6789/0 conn(0x7f04b00ad7e0 legacy=0x7f04b00adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b006c6c0 con 0x7f04b006e6a0 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f0498001670 con 0x7f04b006e6a0 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 >> v1:172.21.15.119:6789/0 conn(0x7f04b006e6a0 legacy=0x7f04b006ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c003a20 0x7f049c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f049c004870 0x7f049c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f049c005230 0x7f049c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b006c3e0 con 0x7f049c005230 2024-09-13T16:34:04.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049c00c0d0 con 0x7f049c003a20 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c00c290 con 0x7f049c004870 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04aed9d700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c003a20 0x7f049c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04af59e700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f049c005230 0x7f049c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.777+0000 7f04ae59c700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f049c004870 0x7f049c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04af59e700 1 -- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f049c004870 msgr2=0x7f049c004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04af59e700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f049c004870 0x7f049c004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04af59e700 1 -- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c003a20 msgr2=0x7f049c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04af59e700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c003a20 0x7f049c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04af59e700 1 -- 172.21.15.119:0/2847342754 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c0035d0 con 0x7f049c005230 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04aed9d700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c003a20 0x7f049c003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04af59e700 1 -- 172.21.15.119:0/2847342754 reap_dead start 2024-09-13T16:34:04.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04af59e700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f049c005230 0x7f049c00ba30 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f04a80017e0 tx=0x7f049c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=216c64f420feb01d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80052d0 con 0x7f049c005230 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8005430 con 0x7f049c005230 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04add9b700 1 -- 172.21.15.119:0/2847342754 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80055a0 con 0x7f049c005230 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- 172.21.15.119:0/2847342754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f049c005230 msgr2=0x7f049c00ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f049c005230 0x7f049c00ba30 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7f04a80017e0 tx=0x7f049c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- 172.21.15.119:0/2847342754 shutdown_connections 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 --2- 172.21.15.119:0/2847342754 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f049c005230 0x7f049c00ba30 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- 172.21.15.119:0/2847342754 >> 172.21.15.119:0/2847342754 conn(0x7f04b00b5d20 msgr2=0x7f04b00b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- 172.21.15.119:0/2847342754 shutdown_connections 2024-09-13T16:34:04.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- 172.21.15.119:0/2847342754 wait complete. 2024-09-13T16:34:04.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 Processor -- start 2024-09-13T16:34:04.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- start start 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c005230 0x7f04b00b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04b014ffa0 0x7f04b015ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0150400 0x7f04b015b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01f2460 con 0x7f04b014ffa0 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01f2670 con 0x7f049c005230 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.781+0000 7f04b4a44700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04b01f2950 con 0x7f04b0150400 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04b014ffa0 0x7f04b015ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04b014ffa0 0x7f04b015ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36590/0 (socket says 172.21.15.119:36590) 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 -- 172.21.15.119:0/3750850735 learned_addr learned my addr 172.21.15.119:0/3750850735 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04aed9d700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c005230 0x7f04b00b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04af59e700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0150400 0x7f04b015b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 -- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0150400 msgr2=0x7f04b015b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0150400 0x7f04b015b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 -- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c005230 msgr2=0x7f04b00b0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c005230 0x7f04b00b0430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 -- 172.21.15.119:0/3750850735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04b01f2c30 con 0x7f04b014ffa0 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04ae59c700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04b014ffa0 0x7f04b015ae90 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009f90 tx=0x7f04a000f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04a77fe700 1 -- 172.21.15.119:0/3750850735 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a000e040 con 0x7f04b014ffa0 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04b01f2f10 con 0x7f04b014ffa0 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04a77fe700 1 -- 172.21.15.119:0/3750850735 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a000f040 con 0x7f04b014ffa0 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04a77fe700 1 -- 172.21.15.119:0/3750850735 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0015670 con 0x7f04b014ffa0 2024-09-13T16:34:04.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.785+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b0152990 con 0x7f04b014ffa0 2024-09-13T16:34:04.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04a77fe700 1 -- 172.21.15.119:0/3750850735 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f04a00157d0 con 0x7f04b014ffa0 2024-09-13T16:34:04.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04a77fe700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0490066730 0x7f0490068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04a77fe700 1 -- 172.21.15.119:0/3750850735 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f04a0084d50 con 0x7f04b014ffa0 2024-09-13T16:34:04.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04aed9d700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0490066730 0x7f0490068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04aed9d700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0490066730 0x7f0490068bf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0498007fa0 tx=0x7f0498007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 Processor -- start 2024-09-13T16:34:04.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 -- start start 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a88610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a50890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a50dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x29afd80 con 0x2a66e90 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x29ad510 con 0x2a88b50 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x29a8e00 con 0x2a88230 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a50890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a50890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36592/0 (socket says 172.21.15.119:36592) 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04ae59c700 1 -- 172.21.15.119:0/726026470 learned_addr learned my addr 172.21.15.119:0/726026470 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04aed9d700 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a88610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04af59e700 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a50dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04ae59c700 1 -- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 msgr2=0x2a88610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04ae59c700 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a88610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04ae59c700 1 -- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 msgr2=0x2a50dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.789+0000 7f04ae59c700 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a50dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04ae59c700 1 -- 172.21.15.119:0/726026470 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x281b960 con 0x2a66e90 2024-09-13T16:34:04.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04af59e700 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a50dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:04.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04ae59c700 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a50890 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f04a0007130 tx=0x7f04a0055620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7feaefba329910d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04a4ff9700 1 -- 172.21.15.119:0/726026470 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a005dfb0 con 0x2a66e90 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04a4ff9700 1 -- 172.21.15.119:0/726026470 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0011e00 con 0x2a66e90 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04a4ff9700 1 -- 172.21.15.119:0/726026470 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0055e20 con 0x2a66e90 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 -- 172.21.15.119:0/726026470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 msgr2=0x2a50890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a50890 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f04a0007130 tx=0x7f04a0055620 comp rx=0 tx=0).stop 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 -- 172.21.15.119:0/726026470 shutdown_connections 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a50dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a50890 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 --2- 172.21.15.119:0/726026470 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a88610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 -- 172.21.15.119:0/726026470 >> 172.21.15.119:0/726026470 conn(0x27d04d0 msgr2=0x2a69e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 -- 172.21.15.119:0/726026470 shutdown_connections 2024-09-13T16:34:04.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 -- 172.21.15.119:0/726026470 wait complete. 2024-09-13T16:34:04.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 -- start start 2024-09-13T16:34:04.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.793+0000 7f04b84f6740 1 Processor -- start 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a80b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a40a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a42a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2966730 con 0x2a66e90 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x296c5b0 con 0x2a88b50 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2854aa0 con 0x2a88230 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a80b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 10 client.0 _ll_register_callbacks cb 0x29a2610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a80b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36594/0 (socket says 172.21.15.119:36594) 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 -- 172.21.15.119:0/591120616 learned_addr learned my addr 172.21.15.119:0/591120616 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:04.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04ae59c700 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a40a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04af59e700 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a42a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 -- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 msgr2=0x2a40a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a40a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 -- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 msgr2=0x2a42a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a42a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 -- 172.21.15.119:0/591120616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04b0152990 con 0x2a66e90 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04aed9d700 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a80b30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0498015a20 tx=0x7f0498001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21151a44dc8b4a88 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f048dffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f048dffb700 1 -- 172.21.15.119:0/591120616 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f049801a040 con 0x2a66e90 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f048dffb700 1 -- 172.21.15.119:0/591120616 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0498027960 con 0x2a66e90 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x27dd1e0 con 0x2a66e90 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f048dffb700 1 -- 172.21.15.119:0/591120616 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0498027b60 con 0x2a66e90 2024-09-13T16:34:04.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 10 client.8973 fetch_fsmap learned FSMap version 76 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x296f220 con 0x2a66e90 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f048dffb700 1 -- 172.21.15.119:0/591120616 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x27dd1e0 con 0x2a66e90 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f048dffb700 1 -- 172.21.15.119:0/591120616 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0498027cc0 con 0x2a66e90 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 10 client.8973 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 10 client.8973 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 20 client.8973 populate_metadata read hostname 'smithi119' 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a3e150 0x2a9bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 4 client.8973 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.797+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2827460 con 0x2a3e150 2024-09-13T16:34:04.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04ae59c700 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a3e150 0x2a9bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04ae59c700 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a3e150 0x2a9bc90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4ba370befff1a865 server_cookie=d8067715c0d21c70 in_seq=0 out_seq=0 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f048dffb700 10 client.8973 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f048dffb700 1 -- 172.21.15.119:0/591120616 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x2827460 con 0x2a3e150 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f048dffb700 10 client.8973 handle_command_reply: tid=0 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 client.8973 shutdown 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 msgr2=0x2a80b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a80b30 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f0498015a20 tx=0x7f0498001a00 comp rx=0 tx=0).stop 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 shutdown_connections 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a3e150 0x2a9bc90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a88b50 0x2a42a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a40a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 --2- 172.21.15.119:0/591120616 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a66e90 0x2a80b30 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f048dffb700 0 client.8973 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:04.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 >> 172.21.15.119:0/591120616 conn(0x27d04d0 msgr2=0x2a81230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 shutdown_connections 2024-09-13T16:34:04.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 1 -- 172.21.15.119:0/591120616 wait complete. 2024-09-13T16:34:04.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.801+0000 7f04b84f6740 20 client.8973 trim_cache size 0 max 16384 2024-09-13T16:34:04.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.817+0000 7f04b84f6740 1 Processor -- start 2024-09-13T16:34:04.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.817+0000 7f04b84f6740 1 -- start start 2024-09-13T16:34:04.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a9e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88b50 0x2a3ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x29a0b70 con 0x2a88b50 2024-09-13T16:34:04.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x29d7ec0 con 0x2a66e90 2024-09-13T16:34:04.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2962c30 con 0x2a88230 2024-09-13T16:34:04.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88b50 0x2a3ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88b50 0x2a3ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36604/0 (socket says 172.21.15.119:36604) 2024-09-13T16:34:04.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 -- 172.21.15.119:0/1369505365 learned_addr learned my addr 172.21.15.119:0/1369505365 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:04.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04aed9d700 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04ae59c700 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a9e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 -- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 msgr2=0x2a9e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a9e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 -- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 msgr2=0x2a9dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 -- 172.21.15.119:0/1369505365 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2a53190 con 0x2a88b50 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04aed9d700 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04af59e700 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88b50 0x2a3ce10 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f04a8001dd0 tx=0x7f04a8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a892874c3599ba12 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04a57fa700 1 -- 172.21.15.119:0/1369505365 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a80052d0 con 0x2a88b50 2024-09-13T16:34:04.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04a57fa700 1 -- 172.21.15.119:0/1369505365 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a8001290 con 0x2a88b50 2024-09-13T16:34:04.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04a57fa700 1 -- 172.21.15.119:0/1369505365 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a8006aa0 con 0x2a88b50 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- 172.21.15.119:0/1369505365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88b50 msgr2=0x2a3ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88b50 0x2a3ce10 secure :-1 s=READY pgs=1586 cs=0 l=1 rev1=1 crypto rx=0x7f04a8001dd0 tx=0x7f04a8011000 comp rx=0 tx=0).stop 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- 172.21.15.119:0/1369505365 shutdown_connections 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88b50 0x2a3ce10 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88230 0x2a9e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 --2- 172.21.15.119:0/1369505365 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- 172.21.15.119:0/1369505365 >> 172.21.15.119:0/1369505365 conn(0x27d04d0 msgr2=0x2a3c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- 172.21.15.119:0/1369505365 shutdown_connections 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.821+0000 7f04b84f6740 1 -- 172.21.15.119:0/1369505365 wait complete. 2024-09-13T16:34:04.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 -- start start 2024-09-13T16:34:04.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 Processor -- start 2024-09-13T16:34:04.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88230 0x2af14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88b50 0x2a97ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a43600 con 0x2a88230 2024-09-13T16:34:04.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a9d030 con 0x2a66e90 2024-09-13T16:34:04.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2a9d2d0 con 0x2a88b50 2024-09-13T16:34:04.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 10 client.0 _ll_register_callbacks cb 0x295ace0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:34:04.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88230 0x2af14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04aed9d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04af59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88b50 0x2a97ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88230 0x2af14a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36614/0 (socket says 172.21.15.119:36614) 2024-09-13T16:34:04.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 -- 172.21.15.119:0/3845337843 learned_addr learned my addr 172.21.15.119:0/3845337843 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:04.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 -- 172.21.15.119:0/3845337843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88b50 msgr2=0x2a97ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88b50 0x2a97ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 -- 172.21.15.119:0/3845337843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 msgr2=0x2a9dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 -- 172.21.15.119:0/3845337843 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x296f220 con 0x2a88230 2024-09-13T16:34:04.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04af59e700 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88b50 0x2a97ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:04.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04ae59c700 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88230 0x2af14a0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f04a000f890 tx=0x7f04a00070c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa46949a7675d704 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f048dffb700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f048dffb700 1 -- 172.21.15.119:0/3845337843 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04a0072070 con 0x2a88230 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f048dffb700 1 -- 172.21.15.119:0/3845337843 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f04a005dfb0 con 0x2a88230 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.825+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f04a0011e00 con 0x2a88230 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f048dffb700 1 -- 172.21.15.119:0/3845337843 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04a00550b0 con 0x2a88230 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 10 client.8979 fetch_fsmap learned FSMap version 76 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x7f049c0035d0 con 0x2a88230 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f048dffb700 1 -- 172.21.15.119:0/3845337843 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04a0011e00 con 0x2a88230 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f048dffb700 1 -- 172.21.15.119:0/3845337843 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f04a0055e20 con 0x2a88230 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 10 client.8979 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 10 client.8979 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 20 client.8979 populate_metadata read hostname 'smithi119' 2024-09-13T16:34:04.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a806a0 0x2a80a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 4 client.8979 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "status"}) v1 -- 0x284c1e0 con 0x2a806a0 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04aed9d700 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a806a0 0x2a80a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f048dffb700 10 client.8979 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04aed9d700 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a806a0 0x2a80a80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=849c86326dedfc5 server_cookie=dd08f026f77cd37d in_seq=0 out_seq=0 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f048dffb700 1 -- 172.21.15.119:0/3845337843 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x284c1e0 con 0x2a806a0 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f048dffb700 10 client.8979 handle_command_reply: tid=0 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 client.8979 shutdown 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88230 msgr2=0x2af14a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88230 0x2af14a0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f04a000f890 tx=0x7f04a00070c0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 shutdown_connections 2024-09-13T16:34:04.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x2a806a0 0x2a80a80 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2a88b50 0x2a97ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x2a88230 0x2af14a0 unknown :-1 s=CLOSED pgs=1587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 --2- 172.21.15.119:0/3845337843 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x2a66e90 0x2a9dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 >> 172.21.15.119:0/3845337843 conn(0x27d04d0 msgr2=0x2a871c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f048dffb700 0 client.8979 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 shutdown_connections 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 1 -- 172.21.15.119:0/3845337843 wait complete. 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b84f6740 20 client.8979 trim_cache size 0 max 16384 2024-09-13T16:34:04.836 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 8829, 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 5.7973689960000003, 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 76, 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 111, 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 111, 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 10.302491521 2024-09-13T16:34:04.837 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:34:04.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0490066730 msgr2=0x7f0490068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0490066730 0x7f0490068bf0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f0498007fa0 tx=0x7f0498007f30 comp rx=0 tx=0).stop 2024-09-13T16:34:04.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04b014ffa0 msgr2=0x7f04b015ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04b014ffa0 0x7f04b015ae90 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7f04a0009f90 tx=0x7f04a000f490 comp rx=0 tx=0).stop 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 shutdown_connections 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0490066730 0x7f0490068bf0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04b0150400 0x7f04b015b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04b014ffa0 0x7f04b015ae90 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 --2- 172.21.15.119:0/3750850735 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f049c005230 0x7f04b00b0430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 >> 172.21.15.119:0/3750850735 conn(0x7f04b00b5d20 msgr2=0x7f04b0158fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 shutdown_connections 2024-09-13T16:34:04.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.829+0000 7f04b4a44700 1 -- 172.21.15.119:0/3750850735 wait complete. 2024-09-13T16:34:04.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:34:04.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.953+0000 7f6c9e55e700 1 Processor -- start 2024-09-13T16:34:04.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.953+0000 7f6c9e55e700 1 -- start start 2024-09-13T16:34:04.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6c9806dc40 0x7f6c980b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c980b3630 0x7f6c9806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c980b2280 con 0x7f6c981594f0 2024-09-13T16:34:04.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c980cc890 con 0x7f6c9806dc40 2024-09-13T16:34:04.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980b1970 con 0x7f6c980b3630 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9d55c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6c981594f0 0x7f6c9806d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54256/0 (socket says 172.21.15.119:54256) 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9d55c700 1 -- 172.21.15.119:0/408281356 learned_addr learned my addr 172.21.15.119:0/408281356 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9cd5b700 1 --2- 172.21.15.119:0/408281356 >> v2:172.21.15.145:3300/0 conn(0x7f6c9806dc40 0x7f6c980b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9dd5d700 1 --2- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c980b3630 0x7f6c9806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c8ffff700 1 -- 172.21.15.119:0/408281356 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834747038 0 0) 0x7f6c980b2280 con 0x7f6c981594f0 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c8ffff700 1 -- 172.21.15.119:0/408281356 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c840035d0 con 0x7f6c981594f0 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c8ffff700 1 -- 172.21.15.119:0/408281356 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1715587417 0 0) 0x7f6c840035d0 con 0x7f6c981594f0 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9cd5b700 1 -- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c980b3630 msgr2=0x7f6c9806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9cd5b700 1 --2- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c980b3630 0x7f6c9806fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9cd5b700 1 -- 172.21.15.119:0/408281356 >> v1:172.21.15.119:6789/0 conn(0x7f6c981594f0 legacy=0x7f6c9806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9cd5b700 1 -- 172.21.15.119:0/408281356 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980b1c30 con 0x7f6c9806dc40 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9cd5b700 1 --2- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9806dc40 0x7f6c980b30f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009850 tx=0x7f6c88009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa7986b5f712ceaa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c8ffff700 1 -- 172.21.15.119:0/408281356 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c94001670 con 0x7f6c9806dc40 2024-09-13T16:34:04.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c8ffff700 1 -- 172.21.15.119:0/408281356 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c88004030 con 0x7f6c9806dc40 2024-09-13T16:34:04.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c8ffff700 1 -- 172.21.15.119:0/408281356 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c880043a0 con 0x7f6c9806dc40 2024-09-13T16:34:04.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9806dc40 msgr2=0x7f6c980b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9806dc40 0x7f6c980b30f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f6c88009850 tx=0x7f6c88009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:04.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- 172.21.15.119:0/408281356 shutdown_connections 2024-09-13T16:34:04.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c980b3630 0x7f6c9806fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/408281356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c9806dc40 0x7f6c980b30f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- 172.21.15.119:0/408281356 >> 172.21.15.119:0/408281356 conn(0x7f6c9806a350 msgr2=0x7f6c9806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- 172.21.15.119:0/408281356 shutdown_connections 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.957+0000 7f6c9e55e700 1 -- 172.21.15.119:0/408281356 wait complete. 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 Processor -- start 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 -- start start 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9806dc40 0x7f6c98150610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c980b3630 0x7f6c9814d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c981594f0 0x7f6c9814ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980ad550 con 0x7f6c980b3630 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9814d8d0 con 0x7f6c981594f0 2024-09-13T16:34:04.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98153440 con 0x7f6c9806dc40 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c980b3630 0x7f6c9814d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c980b3630 0x7f6c9814d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36624/0 (socket says 172.21.15.119:36624) 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 -- 172.21.15.119:0/27859596 learned_addr learned my addr 172.21.15.119:0/27859596 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9d55c700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9806dc40 0x7f6c98150610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9dd5d700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c981594f0 0x7f6c9814ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 -- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9806dc40 msgr2=0x7f6c98150610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9806dc40 0x7f6c98150610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 -- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c981594f0 msgr2=0x7f6c9814ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c981594f0 0x7f6c9814ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 -- 172.21.15.119:0/27859596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c880094b0 con 0x7f6c980b3630 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9dd5d700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c981594f0 0x7f6c9814ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9cd5b700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c980b3630 0x7f6c9814d390 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f6c880090b0 tx=0x7f6c88004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c8dffb700 1 -- 172.21.15.119:0/27859596 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88004590 con 0x7f6c980b3630 2024-09-13T16:34:04.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c8dffb700 1 -- 172.21.15.119:0/27859596 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c8801f070 con 0x7f6c980b3630 2024-09-13T16:34:04.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9814f1b0 con 0x7f6c980b3630 2024-09-13T16:34:04.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c8dffb700 1 -- 172.21.15.119:0/27859596 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c88011870 con 0x7f6c980b3630 2024-09-13T16:34:04.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c98152b10 con 0x7f6c980b3630 2024-09-13T16:34:04.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.961+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c980b26f0 con 0x7f6c980b3630 2024-09-13T16:34:04.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.965+0000 7f6c8dffb700 1 -- 172.21.15.119:0/27859596 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6c880119d0 con 0x7f6c980b3630 2024-09-13T16:34:04.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.965+0000 7f6c8dffb700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c7c066730 0x7f6c7c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:04.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.965+0000 7f6c8dffb700 1 -- 172.21.15.119:0/27859596 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f6c88021070 con 0x7f6c980b3630 2024-09-13T16:34:04.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.965+0000 7f6c9d55c700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c7c066730 0x7f6c7c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:04.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.965+0000 7f6c9d55c700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c7c066730 0x7f6c7c068bf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6c940011e0 tx=0x7f6c94001da0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:04.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:04.973+0000 7f6c8dffb700 1 -- 172.21.15.119:0/27859596 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6c980b26f0 con 0x7f6c980b3630 2024-09-13T16:34:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.181+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6c980b26f0 con 0x7f6c980b3630 2024-09-13T16:34:05.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.181+0000 7f6c8dffb700 1 -- 172.21.15.119:0/27859596 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) v1 ==== 53+0+73 (secure 0 0 0) 0x7f6c980b26f0 con 0x7f6c980b3630 2024-09-13T16:34:05.185 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:34:05.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c7c066730 msgr2=0x7f6c7c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:05.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c7c066730 0x7f6c7c068bf0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6c940011e0 tx=0x7f6c94001da0 comp rx=0 tx=0).stop 2024-09-13T16:34:05.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c980b3630 msgr2=0x7f6c9814d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:05.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c980b3630 0x7f6c9814d390 secure :-1 s=READY pgs=1589 cs=0 l=1 rev1=1 crypto rx=0x7f6c880090b0 tx=0x7f6c88004770 comp rx=0 tx=0).stop 2024-09-13T16:34:05.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 shutdown_connections 2024-09-13T16:34:05.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6c7c066730 0x7f6c7c068bf0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:05.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6c981594f0 0x7f6c9814ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:05.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6c980b3630 0x7f6c9814d390 unknown :-1 s=CLOSED pgs=1589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:05.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 --2- 172.21.15.119:0/27859596 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6c9806dc40 0x7f6c98150610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:05.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 >> 172.21.15.119:0/27859596 conn(0x7f6c9806a350 msgr2=0x7f6c98159c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:05.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 shutdown_connections 2024-09-13T16:34:05.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:05.185+0000 7f6c9e55e700 1 -- 172.21.15.119:0/27859596 wait complete. 2024-09-13T16:34:05.200 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:34:05.200 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:34:05.200 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:34:05.201 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:34:05.201 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:05.201 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:34:05.201 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:34:05.201 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:34:05.201 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:34:05.201 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.201 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:34:05.207 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:34:05.207 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:34:05.207 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:34:05.207 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.207 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:34:05.207 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42096sec preferred_lft 42096sec 2024-09-13T16:34:05.208 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:34:05.209 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:34:05.210 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:34:05.210 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.210 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:34:05.210 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:34:05.210 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:34:05.210 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:05.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.210 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:34:05.223 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:34:05.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.225 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:34:05.233 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:34:05.234 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:34:05.234 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:34:05.234 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:34:05.234 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:34:05.234 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:34:05.234 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:34:05.234 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:34:05.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.235 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:34:05.235 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:34:05.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:34:05.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:34:05.235 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:34:05.235 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:34:05.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.314 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:34:05.321 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:34:05.321 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:34:05.322 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:34:05.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.322 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:34:05.335 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:34:05.335 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:34:05.335 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:05.386 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:34:05.387 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:34:05.387 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:05.393 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:34:05.410 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:34:05.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:34:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:34:05.425 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:34:05.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.426 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:34:05.473 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:34:05.473 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:34:05.475 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:34:05.491 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:34:05.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:34:05.504 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:34:05.505 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:34:05.505 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:05.505 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:34:05.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:34:05.557+0000 7f7c25d8e640 -1 init, newargv = 0x55745b0014f0 newargc=15 2024-09-13T16:34:05.562 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[50845]: starting ceph client 2024-09-13T16:34:05.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[50845]: starting fuse 2024-09-13T16:34:06.614 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:34:06.632 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:34:06.632 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:34:06.648 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:34:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:34:06.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:06.650 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:34:06.656 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:34:06.657 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:34:06.657 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:34:06.658 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:34:06.659 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:34:06.659 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:34:06.659 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:06.659 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:34:06.659 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:34:06.778 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.50845.asok 2024-09-13T16:34:06.962 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.50845.asok 2024-09-13T16:34:06.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:06.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.50845.asok status 2024-09-13T16:34:07.071 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:34:07.071 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:34:07.071 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:34:07.071 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:34:07.071 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:34:07.071 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "50845", 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 8593, 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:34:07.072 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 8593 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 1065203181 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "172.21.15.119:0", 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 1065203181 2024-09-13T16:34:07.073 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.8593 172.21.15.119:0/1065203181", 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "172.21.15.119:0/1065203181", 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 76, 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 111, 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:34:07.074 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:34:07.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:07.082 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:34:07.082 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:07.087 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:34:07.087 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:07.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:07.087 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:07.148 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:07.148 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:07.155 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:07.155 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file1_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-09-13T16:34:07.250 INFO:teuthology.orchestra.run.smithi119.stderr:6+0 records in 2024-09-13T16:34:07.250 INFO:teuthology.orchestra.run.smithi119.stderr:6+0 records out 2024-09-13T16:34:07.250 INFO:teuthology.orchestra.run.smithi119.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0771016 s, 81.6 MB/s 2024-09-13T16:34:07.250 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:07.250 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:07.258 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:34:07.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.413+0000 7f569b59e700 1 Processor -- start 2024-09-13T16:34:07.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.413+0000 7f569b59e700 1 -- start start 2024-09-13T16:34:07.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.413+0000 7f569b59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f569c0cd8d0 0x7f569c06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0b17f0 0x7f569c0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569c0705d0 con 0x7f569c0b2200 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569c06d0d0 con 0x7f569c0cd8d0 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c06fca0 con 0x7f569c0b17f0 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569ad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f569c0b2200 0x7f569c070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54262/0 (socket says 172.21.15.119:54262) 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569ad9d700 1 -- 172.21.15.119:0/1175609506 learned_addr learned my addr 172.21.15.119:0/1175609506 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0b17f0 0x7f569c0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569a59c700 1 --2- 172.21.15.119:0/1175609506 >> v2:172.21.15.145:3300/0 conn(0x7f569c0cd8d0 0x7f569c06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004081668 0 0) 0x7f569c0705d0 con 0x7f569c0b2200 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f567c0035d0 con 0x7f569c0b2200 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3944210130 0 0) 0x7f567c0035d0 con 0x7f569c0b2200 2024-09-13T16:34:07.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0b17f0 msgr2=0x7f569c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0b17f0 0x7f569c0cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 >> v2:172.21.15.145:3300/0 conn(0x7f569c0cd8d0 msgr2=0x7f569c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 --2- 172.21.15.119:0/1175609506 >> v2:172.21.15.145:3300/0 conn(0x7f569c0cd8d0 0x7f569c06cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c06ff80 con 0x7f569c0b2200 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f5694001670 con 0x7f569c0b2200 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 >> v1:172.21.15.119:6789/0 conn(0x7f569c0b2200 legacy=0x7f569c070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c003a20 0x7f567c003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567c004870 0x7f567c004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f567c005230 0x7f567c00ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c06d0d0 con 0x7f567c004870 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c06fca0 con 0x7f567c003a20 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c00bfa0 con 0x7f567c005230 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0b17f0 0x7f569c0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:07.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567c004870 0x7f567c004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569ad9d700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f567c005230 0x7f567c00ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569a59c700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c003a20 0x7f567c003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 -- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f567c005230 msgr2=0x7f567c00ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f567c005230 0x7f567c00ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 -- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c003a20 msgr2=0x7f567c003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c003a20 0x7f567c003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 -- 172.21.15.119:0/1175609506 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0035d0 con 0x7f567c004870 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f5699d9b700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567c004870 0x7f567c004cf0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f569c06cb60 tx=0x7f567c0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=895f1b35606c966b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684004030 con 0x7f567c004870 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56840041b0 con 0x7f567c004870 2024-09-13T16:34:07.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569959a700 1 -- 172.21.15.119:0/1175609506 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5684004560 con 0x7f567c004870 2024-09-13T16:34:07.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- 172.21.15.119:0/1175609506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567c004870 msgr2=0x7f567c004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 --2- 172.21.15.119:0/1175609506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f567c004870 0x7f567c004cf0 secure :-1 s=READY pgs=1592 cs=0 l=1 rev1=1 crypto rx=0x7f569c06cb60 tx=0x7f567c0058a0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569ad9d700 1 -- 172.21.15.119:0/1175609506 reap_dead start 2024-09-13T16:34:07.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- 172.21.15.119:0/1175609506 shutdown_connections 2024-09-13T16:34:07.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- 172.21.15.119:0/1175609506 >> 172.21.15.119:0/1175609506 conn(0x7f569c06a350 msgr2=0x7f569c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:07.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- 172.21.15.119:0/1175609506 shutdown_connections 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.417+0000 7f569b59e700 1 -- 172.21.15.119:0/1175609506 wait complete. 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 Processor -- start 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 -- start start 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c004870 0x7f569c15c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0a84b0 0x7f569c0a8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f569c15eed0 0x7f569c15f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c158e30 con 0x7f569c15eed0 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569c0a8e50 con 0x7f567c004870 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569c15f890 con 0x7f569c0a84b0 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f569c15eed0 0x7f569c15f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f569c15eed0 0x7f569c15f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36640/0 (socket says 172.21.15.119:36640) 2024-09-13T16:34:07.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 -- 172.21.15.119:0/622465065 learned_addr learned my addr 172.21.15.119:0/622465065 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f5699d9b700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0a84b0 0x7f569c0a8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569a59c700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c004870 0x7f569c15c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 -- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0a84b0 msgr2=0x7f569c0a8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0a84b0 0x7f569c0a8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 -- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c004870 msgr2=0x7f569c15c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c004870 0x7f569c15c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 -- 172.21.15.119:0/622465065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569c15fb30 con 0x7f569c15eed0 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.421+0000 7f569ad9d700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f569c15eed0 0x7f569c15f350 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f569c0ce660 tx=0x7f5694004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f5692ffd700 1 -- 172.21.15.119:0/622465065 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694012560 con 0x7f569c15eed0 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f5692ffd700 1 -- 172.21.15.119:0/622465065 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5694012700 con 0x7f569c15eed0 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f569c15fe10 con 0x7f569c15eed0 2024-09-13T16:34:07.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f5692ffd700 1 -- 172.21.15.119:0/622465065 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5694012ab0 con 0x7f569c15eed0 2024-09-13T16:34:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f569c0aa780 con 0x7f569c15eed0 2024-09-13T16:34:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f569c06cb90 con 0x7f569c15eed0 2024-09-13T16:34:07.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f5692ffd700 1 -- 172.21.15.119:0/622465065 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5694012c10 con 0x7f569c15eed0 2024-09-13T16:34:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.425+0000 7f5692ffd700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5680066760 0x7f5680068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.429+0000 7f5692ffd700 1 -- 172.21.15.119:0/622465065 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f5694083fd0 con 0x7f569c15eed0 2024-09-13T16:34:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.429+0000 7f569a59c700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5680066760 0x7f5680068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.429+0000 7f569a59c700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5680066760 0x7f5680068c20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f568c014450 tx=0x7f568c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:07.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.433+0000 7f5692ffd700 1 -- 172.21.15.119:0/622465065 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5694051290 con 0x7f569c15eed0 2024-09-13T16:34:07.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.609+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f569c1619a0 con 0x7f569c15eed0 2024-09-13T16:34:07.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.609+0000 7f5692ffd700 1 -- 172.21.15.119:0/622465065 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16776 (secure 0 0 0) 0x7f56940548b0 con 0x7f569c15eed0 2024-09-13T16:34:07.614 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:07.615 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":111,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:34:00.035557+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5680066760 msgr2=0x7f5680068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5680066760 0x7f5680068c20 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f568c014450 tx=0x7f568c00b040 comp rx=0 tx=0).stop 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f569c15eed0 msgr2=0x7f569c15f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f569c15eed0 0x7f569c15f350 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7f569c0ce660 tx=0x7f5694004f90 comp rx=0 tx=0).stop 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 shutdown_connections 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5680066760 0x7f5680068c20 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f569c15eed0 0x7f569c15f350 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f569c0a84b0 0x7f569c0a8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 --2- 172.21.15.119:0/622465065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f567c004870 0x7f569c15c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 >> 172.21.15.119:0/622465065 conn(0x7f569c06a350 msgr2=0x7f569c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:07.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 shutdown_connections 2024-09-13T16:34:07.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.613+0000 7f569b59e700 1 -- 172.21.15.119:0/622465065 wait complete. 2024-09-13T16:34:07.634 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:33:57.346379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:33:58.017381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:34:07.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:34:07.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f080e0e6700 1 Processor -- start 2024-09-13T16:34:07.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f080e0e6700 1 -- start start 2024-09-13T16:34:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f080e0e6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f080815e430 0x7f08080cb210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f080e0e6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f080815e810 0x7f080806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f080e0e6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08080b2690 con 0x7f08080701f0 2024-09-13T16:34:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f080e0e6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08080b2970 con 0x7f080815e430 2024-09-13T16:34:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f080e0e6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08080cb750 con 0x7f080815e810 2024-09-13T16:34:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08077fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f080815e430 0x7f08080cb210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f0807fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f08080701f0 0x7f08080ccf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54278/0 (socket says 172.21.15.119:54278) 2024-09-13T16:34:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f0807fff700 1 -- 172.21.15.119:0/1364432477 learned_addr learned my addr 172.21.15.119:0/1364432477 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f0806ffd700 1 --2- 172.21.15.119:0/1364432477 >> v2:172.21.15.145:3300/0 conn(0x7f080815e810 0x7f080806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08077fe700 1 -- 172.21.15.119:0/1364432477 >> v2:172.21.15.145:3300/0 conn(0x7f080815e810 msgr2=0x7f080806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08077fe700 1 --2- 172.21.15.119:0/1364432477 >> v2:172.21.15.145:3300/0 conn(0x7f080815e810 0x7f080806fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08077fe700 1 -- 172.21.15.119:0/1364432477 >> v1:172.21.15.119:6789/0 conn(0x7f08080701f0 legacy=0x7f08080ccf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08077fe700 1 -- 172.21.15.119:0/1364432477 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0808071a10 con 0x7f080815e430 2024-09-13T16:34:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08067fc700 1 -- 172.21.15.119:0/1364432477 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1650406625 0 0) 0x7f08080b2690 con 0x7f08080701f0 2024-09-13T16:34:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08077fe700 1 --2- 172.21.15.119:0/1364432477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f08080cb210 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f080806e0a0 tx=0x7f07f8009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eda06483c3eb4d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08067fc700 1 -- 172.21.15.119:0/1364432477 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8011830 con 0x7f080815e430 2024-09-13T16:34:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08067fc700 1 -- 172.21.15.119:0/1364432477 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f8011990 con 0x7f080815e430 2024-09-13T16:34:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.749+0000 7f08067fc700 1 -- 172.21.15.119:0/1364432477 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8011d40 con 0x7f080815e430 2024-09-13T16:34:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- 172.21.15.119:0/1364432477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 msgr2=0x7f08080cb210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 --2- 172.21.15.119:0/1364432477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f08080cb210 secure :-1 s=READY pgs=1595 cs=0 l=1 rev1=1 crypto rx=0x7f080806e0a0 tx=0x7f07f8009b30 comp rx=0 tx=0).stop 2024-09-13T16:34:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- 172.21.15.119:0/1364432477 shutdown_connections 2024-09-13T16:34:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 --2- 172.21.15.119:0/1364432477 >> v2:172.21.15.145:3300/0 conn(0x7f080815e810 0x7f080806fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 --2- 172.21.15.119:0/1364432477 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f08080cb210 unknown :-1 s=CLOSED pgs=1595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- 172.21.15.119:0/1364432477 >> 172.21.15.119:0/1364432477 conn(0x7f08080b0560 msgr2=0x7f08080b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- 172.21.15.119:0/1364432477 shutdown_connections 2024-09-13T16:34:07.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- 172.21.15.119:0/1364432477 wait complete. 2024-09-13T16:34:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 Processor -- start 2024-09-13T16:34:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- start start 2024-09-13T16:34:07.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08080701f0 0x7f080814c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f080815cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080815e810 0x7f08081f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080815cb20 con 0x7f080815e430 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080815bc70 con 0x7f08080701f0 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.753+0000 7f080e0e6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080814cef0 con 0x7f080815e810 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f080815cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f080815cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36662/0 (socket says 172.21.15.119:36662) 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 -- 172.21.15.119:0/3380317670 learned_addr learned my addr 172.21.15.119:0/3380317670 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0807fff700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080815e810 0x7f08081f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f08077fe700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08080701f0 0x7f080814c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 -- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080815e810 msgr2=0x7f08081f18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080815e810 0x7f08081f18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 -- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08080701f0 msgr2=0x7f080814c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08080701f0 0x7f080814c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 -- 172.21.15.119:0/3380317670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07f80094b0 con 0x7f080815e430 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0807fff700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080815e810 0x7f08081f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f08077fe700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08080701f0 0x7f080814c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f0806ffd700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f080815cfb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00d800 tx=0x7f07fc00dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f07effff700 1 -- 172.21.15.119:0/3380317670 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc0125f0 con 0x7f080815e430 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f07effff700 1 -- 172.21.15.119:0/3380317670 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07fc012750 con 0x7f080815e430 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080815bf10 con 0x7f080815e430 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f07effff700 1 -- 172.21.15.119:0/3380317670 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07fc012ac0 con 0x7f080815e430 2024-09-13T16:34:07.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080815c4d0 con 0x7f080815e430 2024-09-13T16:34:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080815ccd0 con 0x7f080815e430 2024-09-13T16:34:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.757+0000 7f07effff700 1 -- 172.21.15.119:0/3380317670 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f07fc012c20 con 0x7f080815e430 2024-09-13T16:34:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.761+0000 7f07effff700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07f4066760 0x7f07f4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:07.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.761+0000 7f07effff700 1 -- 172.21.15.119:0/3380317670 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f07fc0845d0 con 0x7f080815e430 2024-09-13T16:34:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.761+0000 7f08077fe700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07f4066760 0x7f07f4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:07.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.761+0000 7f08077fe700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07f4066760 0x7f07f4068c20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f07f80090b0 tx=0x7f07f8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:07.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.769+0000 7f07effff700 1 -- 172.21.15.119:0/3380317670 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f080815ccd0 con 0x7f080815e430 2024-09-13T16:34:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.937+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f08080b2690 con 0x7f080815e430 2024-09-13T16:34:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f07effff700 1 -- 172.21.15.119:0/3380317670 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v111) v1 ==== 93+0+10 (secure 0 0 0) 0x7f07fc055290 con 0x7f080815e430 2024-09-13T16:34:07.944 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:34:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07f4066760 msgr2=0x7f07f4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07f4066760 0x7f07f4068c20 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f07f80090b0 tx=0x7f07f8008030 comp rx=0 tx=0).stop 2024-09-13T16:34:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 msgr2=0x7f080815cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f080815cfb0 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7f07fc00d800 tx=0x7f07fc00dbc0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 shutdown_connections 2024-09-13T16:34:07.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f07f4066760 0x7f07f4068c20 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080815e810 0x7f08081f18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080815e430 0x7f080815cfb0 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 --2- 172.21.15.119:0/3380317670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08080701f0 0x7f080814c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 >> 172.21.15.119:0/3380317670 conn(0x7f08080b0560 msgr2=0x7f08080afbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 shutdown_connections 2024-09-13T16:34:07.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:07.941+0000 7f080e0e6700 1 -- 172.21.15.119:0/3380317670 wait complete. 2024-09-13T16:34:07.958 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:34:08.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 Processor -- start 2024-09-13T16:34:08.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 -- start start 2024-09-13T16:34:08.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd2a015cfb0 0x7fd2a006ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd2a006b2d0 0x7fd2a006b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006bc50 0x7fd2a00cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a00cee40 con 0x7fd2a006b2d0 2024-09-13T16:34:08.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2a00d1220 con 0x7fd2a015cfb0 2024-09-13T16:34:08.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a836c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a006c0e0 con 0x7fd2a006bc50 2024-09-13T16:34:08.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd2a006b2d0 0x7fd2a006b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd2a006b2d0 0x7fd2a006b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36668/0 (socket says 172.21.15.119:36668) 2024-09-13T16:34:08.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 -- 172.21.15.119:0/1668272962 learned_addr learned my addr 172.21.15.119:0/1668272962 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a6909700 1 --2- 172.21.15.119:0/1668272962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006bc50 0x7fd2a00cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a6108700 1 --2- 172.21.15.119:0/1668272962 >> v2:172.21.15.145:3300/0 conn(0x7fd2a015cfb0 0x7fd2a006ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 -- 172.21.15.119:0/1668272962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006bc50 msgr2=0x7fd2a00cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 --2- 172.21.15.119:0/1668272962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006bc50 0x7fd2a00cf5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 -- 172.21.15.119:0/1668272962 >> v2:172.21.15.145:3300/0 conn(0x7fd2a015cfb0 msgr2=0x7fd2a006ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 --2- 172.21.15.119:0/1668272962 >> v2:172.21.15.145:3300/0 conn(0x7fd2a015cfb0 0x7fd2a006ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 -- 172.21.15.119:0/1668272962 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a00ce880 con 0x7fd2a006b2d0 2024-09-13T16:34:08.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a6909700 1 --2- 172.21.15.119:0/1668272962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006bc50 0x7fd2a00cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:08.087 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.077+0000 7fd2a5907700 1 --2- 172.21.15.119:0/1668272962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a006b2d0 0x7fd2a006b710 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fd2a008b240 tx=0x7fd290009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=816cf1612904b957 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a5106700 1 -- 172.21.15.119:0/1668272962 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290011830 con 0x7fd2a006b2d0 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a5106700 1 -- 172.21.15.119:0/1668272962 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd290011990 con 0x7fd2a006b2d0 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a5106700 1 -- 172.21.15.119:0/1668272962 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd290011d20 con 0x7fd2a006b2d0 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 -- 172.21.15.119:0/1668272962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a006b2d0 msgr2=0x7fd2a006b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 --2- 172.21.15.119:0/1668272962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a006b2d0 0x7fd2a006b710 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fd2a008b240 tx=0x7fd290009bf0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 -- 172.21.15.119:0/1668272962 shutdown_connections 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 --2- 172.21.15.119:0/1668272962 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006bc50 0x7fd2a00cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 --2- 172.21.15.119:0/1668272962 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a006b2d0 0x7fd2a006b710 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 --2- 172.21.15.119:0/1668272962 >> v2:172.21.15.145:3300/0 conn(0x7fd2a015cfb0 0x7fd2a006ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 -- 172.21.15.119:0/1668272962 >> 172.21.15.119:0/1668272962 conn(0x7fd2a00d2da0 msgr2=0x7fd2a00d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 -- 172.21.15.119:0/1668272962 shutdown_connections 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.081+0000 7fd2a836c700 1 -- 172.21.15.119:0/1668272962 wait complete. 2024-09-13T16:34:08.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 Processor -- start 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 -- start start 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006b2d0 0x7fd2a00ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2a006bc50 0x7fd2a0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a015cfb0 0x7fd2a00cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a00cb120 con 0x7fd2a015cfb0 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a00cba00 con 0x7fd2a006bc50 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a836c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a00cbce0 con 0x7fd2a006b2d0 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a015cfb0 0x7fd2a00cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a015cfb0 0x7fd2a00cc7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36672/0 (socket says 172.21.15.119:36672) 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 -- 172.21.15.119:0/759601794 learned_addr learned my addr 172.21.15.119:0/759601794 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6108700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006b2d0 0x7fd2a00ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a5907700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2a006bc50 0x7fd2a0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 -- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006b2d0 msgr2=0x7fd2a00ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006b2d0 0x7fd2a00ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 -- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2a006bc50 msgr2=0x7fd2a0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2a006bc50 0x7fd2a0158e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 -- 172.21.15.119:0/759601794 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290009570 con 0x7fd2a015cfb0 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.085+0000 7fd2a6909700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a015cfb0 0x7fd2a00cc7c0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fd298007e10 tx=0x7fd29800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd296ffd700 1 -- 172.21.15.119:0/759601794 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29800dd20 con 0x7fd2a015cfb0 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd296ffd700 1 -- 172.21.15.119:0/759601794 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29800f040 con 0x7fd2a015cfb0 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd296ffd700 1 -- 172.21.15.119:0/759601794 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd298003c80 con 0x7fd2a015cfb0 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2a00cbfc0 con 0x7fd2a015cfb0 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2a00cad70 con 0x7fd2a015cfb0 2024-09-13T16:34:08.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2a006b710 con 0x7fd2a015cfb0 2024-09-13T16:34:08.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd296ffd700 1 -- 172.21.15.119:0/759601794 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd298004030 con 0x7fd2a015cfb0 2024-09-13T16:34:08.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd296ffd700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd284066730 0x7fd284068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd296ffd700 1 -- 172.21.15.119:0/759601794 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd298084330 con 0x7fd2a015cfb0 2024-09-13T16:34:08.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.089+0000 7fd2a6108700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd284066730 0x7fd284068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.096 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.093+0000 7fd2a6108700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd284066730 0x7fd284068bf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009bf0 tx=0x7fd29c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.097+0000 7fd296ffd700 1 -- 172.21.15.119:0/759601794 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd2980514e0 con 0x7fd2a015cfb0 2024-09-13T16:34:08.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.265+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2a015c8d0 con 0x7fd2a015cfb0 2024-09-13T16:34:08.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.265+0000 7fd296ffd700 1 -- 172.21.15.119:0/759601794 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd298054b00 con 0x7fd2a015cfb0 2024-09-13T16:34:08.271 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:34:08.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd284066730 msgr2=0x7fd284068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd284066730 0x7fd284068bf0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fd29c009bf0 tx=0x7fd29c009240 comp rx=0 tx=0).stop 2024-09-13T16:34:08.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a015cfb0 msgr2=0x7fd2a00cc7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a015cfb0 0x7fd2a00cc7c0 secure :-1 s=READY pgs=1598 cs=0 l=1 rev1=1 crypto rx=0x7fd298007e10 tx=0x7fd29800d3b0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 shutdown_connections 2024-09-13T16:34:08.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd284066730 0x7fd284068bf0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd2a015cfb0 0x7fd2a00cc7c0 unknown :-1 s=CLOSED pgs=1598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd2a006bc50 0x7fd2a0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 --2- 172.21.15.119:0/759601794 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2a006b2d0 0x7fd2a00ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 >> 172.21.15.119:0/759601794 conn(0x7fd2a00d2da0 msgr2=0x7fd2a00b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:08.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 shutdown_connections 2024-09-13T16:34:08.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.269+0000 7fd2a836c700 1 -- 172.21.15.119:0/759601794 wait complete. 2024-09-13T16:34:08.285 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:34:08.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.401+0000 7f9ab54be700 1 Processor -- start 2024-09-13T16:34:08.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.401+0000 7f9ab54be700 1 -- start start 2024-09-13T16:34:08.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9ab015ccb0 0x7f9ab00d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab015e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab00b5bb0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab00d16c0 con 0x7f9ab015ccb0 2024-09-13T16:34:08.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab00d1210 con 0x7f9ab00b5670 2024-09-13T16:34:08.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaf7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9ab00d0da0 0x7f9ab00b4f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54284/0 (socket says 172.21.15.119:54284) 2024-09-13T16:34:08.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaf7fe700 1 -- 172.21.15.119:0/2212024434 learned_addr learned my addr 172.21.15.119:0/2212024434 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:08.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaffff700 1 --2- 172.21.15.119:0/2212024434 >> v2:172.21.15.145:3300/0 conn(0x7f9ab015ccb0 0x7f9ab00d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab4cbd700 1 --2- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab015e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaeffd700 1 -- 172.21.15.119:0/2212024434 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550918288 0 0) 0x7f9ab00b5bb0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaeffd700 1 -- 172.21.15.119:0/2212024434 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a980035d0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaffff700 1 -- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 msgr2=0x7f9ab015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaffff700 1 --2- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab015e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaffff700 1 -- 172.21.15.119:0/2212024434 >> v1:172.21.15.119:6789/0 conn(0x7f9ab00d0da0 legacy=0x7f9ab00b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaffff700 1 -- 172.21.15.119:0/2212024434 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab015d3e0 con 0x7f9ab015ccb0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab4cbd700 1 --2- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab015e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaffff700 1 --2- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 0x7f9ab00d0860 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009880 tx=0x7f9aa4009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1155417fa360e4e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaeffd700 1 -- 172.21.15.119:0/2212024434 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4011830 con 0x7f9ab015ccb0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaeffd700 1 -- 172.21.15.119:0/2212024434 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa4011990 con 0x7f9ab015ccb0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9aaeffd700 1 -- 172.21.15.119:0/2212024434 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa4011d00 con 0x7f9ab015ccb0 2024-09-13T16:34:08.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 -- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 msgr2=0x7f9ab00d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 --2- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 0x7f9ab00d0860 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009880 tx=0x7f9aa4009b90 comp rx=0 tx=0).stop 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 -- 172.21.15.119:0/2212024434 shutdown_connections 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 --2- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab015e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 --2- 172.21.15.119:0/2212024434 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 0x7f9ab00d0860 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.405+0000 7f9ab54be700 1 -- 172.21.15.119:0/2212024434 >> 172.21.15.119:0/2212024434 conn(0x7f9ab0072240 msgr2=0x7f9ab0099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 -- 172.21.15.119:0/2212024434 shutdown_connections 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 -- 172.21.15.119:0/2212024434 wait complete. 2024-09-13T16:34:08.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 Processor -- start 2024-09-13T16:34:08.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 -- start start 2024-09-13T16:34:08.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab01473a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab00d0da0 0x7f9ab01478e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 0x7f9ab00b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0146f10 con 0x7f9ab00d0da0 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab0146300 con 0x7f9ab015ccb0 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab54be700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab01465e0 con 0x7f9ab00b5670 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9aaf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab00d0da0 0x7f9ab01478e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9aaf7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab00d0da0 0x7f9ab01478e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36680/0 (socket says 172.21.15.119:36680) 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9aaf7fe700 1 -- 172.21.15.119:0/3912234560 learned_addr learned my addr 172.21.15.119:0/3912234560 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9aaffff700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab01473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.409+0000 7f9ab4cbd700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 0x7f9ab00b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aaf7fe700 1 -- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 msgr2=0x7f9ab01473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aaf7fe700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab01473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aaf7fe700 1 -- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 msgr2=0x7f9ab00b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aaf7fe700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 0x7f9ab00b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aaf7fe700 1 -- 172.21.15.119:0/3912234560 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8009a70 con 0x7f9ab00d0da0 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aaf7fe700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab00d0da0 0x7f9ab01478e0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000c4f0 tx=0x7f9aa000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aacff9700 1 -- 172.21.15.119:0/3912234560 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0005d90 con 0x7f9ab00d0da0 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aacff9700 1 -- 172.21.15.119:0/3912234560 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa00070d0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa40094b0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aacff9700 1 -- 172.21.15.119:0/3912234560 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa0007440 con 0x7f9ab00d0da0 2024-09-13T16:34:08.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab0146ba0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab00d0860 con 0x7f9ab00d0da0 2024-09-13T16:34:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aacff9700 1 -- 172.21.15.119:0/3912234560 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9aa00075a0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aacff9700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a9c066730 0x7f9a9c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aacff9700 1 -- 172.21.15.119:0/3912234560 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f9aa0083680 con 0x7f9ab00d0da0 2024-09-13T16:34:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.413+0000 7f9aaffff700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a9c066730 0x7f9a9c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.417+0000 7f9aaffff700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a9c066730 0x7f9a9c068bf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4000c00 tx=0x7f9aa4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.425+0000 7f9aacff9700 1 -- 172.21.15.119:0/3912234560 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9aa0050830 con 0x7f9ab00d0da0 2024-09-13T16:34:08.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ab006c140 con 0x7f9ab00d0da0 2024-09-13T16:34:08.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9aacff9700 1 -- 172.21.15.119:0/3912234560 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9aa00170a0 con 0x7f9ab00d0da0 2024-09-13T16:34:08.592 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:34:08.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a9c066730 msgr2=0x7f9a9c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a9c066730 0x7f9a9c068bf0 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4000c00 tx=0x7f9aa4008030 comp rx=0 tx=0).stop 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab00d0da0 msgr2=0x7f9ab01478e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab00d0da0 0x7f9ab01478e0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f9aa000c4f0 tx=0x7f9aa000c800 comp rx=0 tx=0).stop 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 shutdown_connections 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9a9c066730 0x7f9a9c068bf0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ab015ccb0 0x7f9ab00b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ab00d0da0 0x7f9ab01478e0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 --2- 172.21.15.119:0/3912234560 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ab00b5670 0x7f9ab01473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 >> 172.21.15.119:0/3912234560 conn(0x7f9ab0072240 msgr2=0x7f9ab015b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:08.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.589+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 shutdown_connections 2024-09-13T16:34:08.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.593+0000 7f9ab54be700 1 -- 172.21.15.119:0/3912234560 wait complete. 2024-09-13T16:34:08.612 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:34:08.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.721+0000 7f837df0e700 1 Processor -- start 2024-09-13T16:34:08.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.721+0000 7f837df0e700 1 -- start start 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.721+0000 7f837df0e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8378154fd0 0x7f83780ce830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.721+0000 7f837df0e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83780d1130 0x7f837815cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.721+0000 7f837df0e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837815c750 con 0x7f83780ced70 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.721+0000 7f837df0e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83780cf1e0 con 0x7f8378154fd0 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.721+0000 7f837df0e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83780d1750 con 0x7f83780d1130 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f8377fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f83780ced70 0x7f83780d0a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54288/0 (socket says 172.21.15.119:54288) 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f8377fff700 1 -- 172.21.15.119:0/3071255851 learned_addr learned my addr 172.21.15.119:0/3071255851 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837cf0c700 1 --2- 172.21.15.119:0/3071255851 >> v2:172.21.15.145:3300/0 conn(0x7f8378154fd0 0x7f83780ce830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837d70d700 1 --2- 172.21.15.119:0/3071255851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83780d1130 0x7f837815cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f83777fe700 1 -- 172.21.15.119:0/3071255851 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533766176 0 0) 0x7f837815c750 con 0x7f83780ced70 2024-09-13T16:34:08.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f83777fe700 1 -- 172.21.15.119:0/3071255851 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83580035d0 con 0x7f83780ced70 2024-09-13T16:34:08.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837d70d700 1 -- 172.21.15.119:0/3071255851 >> v2:172.21.15.145:3300/0 conn(0x7f8378154fd0 msgr2=0x7f83780ce830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:08.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837d70d700 1 --2- 172.21.15.119:0/3071255851 >> v2:172.21.15.145:3300/0 conn(0x7f8378154fd0 0x7f83780ce830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837d70d700 1 -- 172.21.15.119:0/3071255851 >> v1:172.21.15.119:6789/0 conn(0x7f83780ced70 legacy=0x7f83780d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837d70d700 1 -- 172.21.15.119:0/3071255851 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837815c100 con 0x7f83780d1130 2024-09-13T16:34:08.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837d70d700 1 --2- 172.21.15.119:0/3071255851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83780d1130 0x7f837815cbe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8370009850 tx=0x7f8370009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5618705d64ce792 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f83777fe700 1 -- 172.21.15.119:0/3071255851 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370011830 con 0x7f83780d1130 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f83777fe700 1 -- 172.21.15.119:0/3071255851 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8370011990 con 0x7f83780d1130 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f83777fe700 1 -- 172.21.15.119:0/3071255851 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370011d00 con 0x7f83780d1130 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 -- 172.21.15.119:0/3071255851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83780d1130 msgr2=0x7f837815cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 --2- 172.21.15.119:0/3071255851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83780d1130 0x7f837815cbe0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8370009850 tx=0x7f8370009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 -- 172.21.15.119:0/3071255851 shutdown_connections 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 --2- 172.21.15.119:0/3071255851 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83780d1130 0x7f837815cbe0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 --2- 172.21.15.119:0/3071255851 >> v2:172.21.15.145:3300/0 conn(0x7f8378154fd0 0x7f83780ce830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 -- 172.21.15.119:0/3071255851 >> 172.21.15.119:0/3071255851 conn(0x7f8378099a50 msgr2=0x7f8378099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 -- 172.21.15.119:0/3071255851 shutdown_connections 2024-09-13T16:34:08.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.725+0000 7f837df0e700 1 -- 172.21.15.119:0/3071255851 wait complete. 2024-09-13T16:34:08.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 Processor -- start 2024-09-13T16:34:08.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 -- start start 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83780ced70 0x7f83781485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83780d1130 0x7f83781f0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8378154fd0 0x7f83780b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83780b4d40 con 0x7f83780d1130 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8378148b20 con 0x7f83780ced70 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83781f0850 con 0x7f8378154fd0 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f8377fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83780d1130 0x7f83781f0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f8377fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83780d1130 0x7f83781f0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36688/0 (socket says 172.21.15.119:36688) 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f8377fff700 1 -- 172.21.15.119:0/1567714537 learned_addr learned my addr 172.21.15.119:0/1567714537 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837d70d700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8378154fd0 0x7f83780b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837cf0c700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83780ced70 0x7f83781485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837d70d700 1 -- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83780ced70 msgr2=0x7f83781485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837d70d700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83780ced70 0x7f83781485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837d70d700 1 -- 172.21.15.119:0/1567714537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83780d1130 msgr2=0x7f83781f0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837d70d700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83780d1130 0x7f83781f0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837d70d700 1 -- 172.21.15.119:0/1567714537 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83700094b0 con 0x7f8378154fd0 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837d70d700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8378154fd0 0x7f83780b2630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8370016000 tx=0x7f8370004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f83757fa700 1 -- 172.21.15.119:0/1567714537 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370004510 con 0x7f8378154fd0 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f83757fa700 1 -- 172.21.15.119:0/1567714537 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837001f070 con 0x7f8378154fd0 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f83757fa700 1 -- 172.21.15.119:0/1567714537 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8370003900 con 0x7f8378154fd0 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83780b30e0 con 0x7f8378154fd0 2024-09-13T16:34:08.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.729+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83780b4680 con 0x7f8378154fd0 2024-09-13T16:34:08.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.733+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837815cbe0 con 0x7f8378154fd0 2024-09-13T16:34:08.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.733+0000 7f83757fa700 1 -- 172.21.15.119:0/1567714537 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8370003a60 con 0x7f8378154fd0 2024-09-13T16:34:08.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.733+0000 7f83757fa700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f835c066760 0x7f835c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:08.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.733+0000 7f83757fa700 1 -- 172.21.15.119:0/1567714537 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8370021070 con 0x7f8378154fd0 2024-09-13T16:34:08.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.733+0000 7f837cf0c700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f835c066760 0x7f835c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:08.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.737+0000 7f837cf0c700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f835c066760 0x7f835c068c20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8368009b50 tx=0x7f8368009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:08.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.745+0000 7f83757fa700 1 -- 172.21.15.119:0/1567714537 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8370053080 con 0x7f8378154fd0 2024-09-13T16:34:08.932 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.929+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f83780b30e0 con 0x7f8378154fd0 2024-09-13T16:34:08.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.929+0000 7f83757fa700 1 -- 172.21.15.119:0/1567714537 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v111) v1 ==== 80+0+0 (secure 0 0 0) 0x7f837002b020 con 0x7f8378154fd0 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f835c066760 msgr2=0x7f835c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f835c066760 0x7f835c068c20 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8368009b50 tx=0x7f8368009240 comp rx=0 tx=0).stop 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8378154fd0 msgr2=0x7f83780b2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8378154fd0 0x7f83780b2630 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8370016000 tx=0x7f8370004770 comp rx=0 tx=0).stop 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 shutdown_connections 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f835c066760 0x7f835c068c20 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8378154fd0 0x7f83780b2630 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83780d1130 0x7f83781f0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 --2- 172.21.15.119:0/1567714537 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83780ced70 0x7f83781485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:08.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 >> 172.21.15.119:0/1567714537 conn(0x7f8378099a50 msgr2=0x7f83781542c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:08.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 shutdown_connections 2024-09-13T16:34:08.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:08.933+0000 7f837df0e700 1 -- 172.21.15.119:0/1567714537 wait complete. 2024-09-13T16:34:08.937 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:34:08.948 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:08.948 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:08.955 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:34:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.061+0000 7fd2549e6700 1 Processor -- start 2024-09-13T16:34:09.067 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- start start 2024-09-13T16:34:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd25006a5c0 0x7fd250160f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500b4880 con 0x7fd25006a9a0 2024-09-13T16:34:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2500b3c70 con 0x7fd2500717b0 2024-09-13T16:34:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd25015dec0 con 0x7fd25006a5c0 2024-09-13T16:34:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24f59e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd25006a9a0 0x7fd25015d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54290/0 (socket says 172.21.15.119:54290) 2024-09-13T16:34:09.068 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24f59e700 1 -- 172.21.15.119:0/1043247344 learned_addr learned my addr 172.21.15.119:0/1043247344 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:09.069 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24ed9d700 1 --2- 172.21.15.119:0/1043247344 >> v2:172.21.15.119:3300/0 conn(0x7fd25006a5c0 0x7fd250160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24dd9b700 1 -- 172.21.15.119:0/1043247344 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277875154 0 0) 0x7fd2500b4880 con 0x7fd25006a9a0 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24dd9b700 1 -- 172.21.15.119:0/1043247344 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd23c0035d0 con 0x7fd25006a9a0 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24ed9d700 1 -- 172.21.15.119:0/1043247344 >> v1:172.21.15.145:6789/0 conn(0x7fd2500717b0 legacy=0x7fd25006cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24ed9d700 1 -- 172.21.15.119:0/1043247344 >> v1:172.21.15.119:6789/0 conn(0x7fd25006a9a0 legacy=0x7fd25015d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24ed9d700 1 -- 172.21.15.119:0/1043247344 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2500b3f50 con 0x7fd25006a5c0 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24ed9d700 1 --2- 172.21.15.119:0/1043247344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a5c0 0x7fd250160f60 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fd240009850 tx=0x7fd240009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cc548e2c555a6f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24dd9b700 1 -- 172.21.15.119:0/1043247344 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240011830 con 0x7fd25006a5c0 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24dd9b700 1 -- 172.21.15.119:0/1043247344 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd240011990 con 0x7fd25006a5c0 2024-09-13T16:34:09.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd24dd9b700 1 -- 172.21.15.119:0/1043247344 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd240011d00 con 0x7fd25006a5c0 2024-09-13T16:34:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- 172.21.15.119:0/1043247344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a5c0 msgr2=0x7fd250160f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 --2- 172.21.15.119:0/1043247344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a5c0 0x7fd250160f60 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7fd240009850 tx=0x7fd240009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- 172.21.15.119:0/1043247344 shutdown_connections 2024-09-13T16:34:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 --2- 172.21.15.119:0/1043247344 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a5c0 0x7fd250160f60 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- 172.21.15.119:0/1043247344 >> 172.21.15.119:0/1043247344 conn(0x7fd250070cc0 msgr2=0x7fd2500710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- 172.21.15.119:0/1043247344 shutdown_connections 2024-09-13T16:34:09.071 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.065+0000 7fd2549e6700 1 -- 172.21.15.119:0/1043247344 wait complete. 2024-09-13T16:34:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 Processor -- start 2024-09-13T16:34:09.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 -- start start 2024-09-13T16:34:09.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25006a5c0 0x7fd25015c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a9a0 0x7fd25015cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2500717b0 0x7fd2501e7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25015a330 con 0x7fd25006a9a0 2024-09-13T16:34:09.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25015a5b0 con 0x7fd25006a5c0 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd2549e6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd250146b40 con 0x7fd2500717b0 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a9a0 0x7fd25015cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a9a0 0x7fd25015cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36714/0 (socket says 172.21.15.119:36714) 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 -- 172.21.15.119:0/4206063057 learned_addr learned my addr 172.21.15.119:0/4206063057 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24ed9d700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25006a5c0 0x7fd25015c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24f59e700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2500717b0 0x7fd2501e7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 -- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2500717b0 msgr2=0x7fd2501e7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2500717b0 0x7fd2501e7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 -- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25006a5c0 msgr2=0x7fd25015c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25006a5c0 0x7fd25015c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.069+0000 7fd24e59c700 1 -- 172.21.15.119:0/4206063057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2400094b0 con 0x7fd25006a9a0 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd24e59c700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a9a0 0x7fd25015cab0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fd23800c410 tx=0x7fd23800c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2477fe700 1 -- 172.21.15.119:0/4206063057 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2380052d0 con 0x7fd25006a9a0 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25015a870 con 0x7fd25006a9a0 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2477fe700 1 -- 172.21.15.119:0/4206063057 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd238005430 con 0x7fd25006a9a0 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2477fe700 1 -- 172.21.15.119:0/4206063057 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2380055a0 con 0x7fd25006a9a0 2024-09-13T16:34:09.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2501e7540 con 0x7fd25006a9a0 2024-09-13T16:34:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25015ad90 con 0x7fd25006a9a0 2024-09-13T16:34:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2477fe700 1 -- 172.21.15.119:0/4206063057 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd2380036a0 con 0x7fd25006a9a0 2024-09-13T16:34:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2477fe700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd230066730 0x7fd230068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd2477fe700 1 -- 172.21.15.119:0/4206063057 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd238083430 con 0x7fd25006a9a0 2024-09-13T16:34:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.073+0000 7fd24ed9d700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd230066730 0x7fd230068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.077+0000 7fd24ed9d700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd230066730 0x7fd230068bf0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd2400090b0 tx=0x7fd240008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.081+0000 7fd2477fe700 1 -- 172.21.15.119:0/4206063057 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd25015ad90 con 0x7fd25006a9a0 2024-09-13T16:34:09.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.257+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd25015ad90 con 0x7fd25006a9a0 2024-09-13T16:34:09.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.257+0000 7fd2477fe700 1 -- 172.21.15.119:0/4206063057 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fd25015ad90 con 0x7fd25006a9a0 2024-09-13T16:34:09.263 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:09.263 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":112,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:34:09.089587+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd230066730 msgr2=0x7fd230068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd230066730 0x7fd230068bf0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7fd2400090b0 tx=0x7fd240008030 comp rx=0 tx=0).stop 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a9a0 msgr2=0x7fd25015cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a9a0 0x7fd25015cab0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7fd23800c410 tx=0x7fd23800c720 comp rx=0 tx=0).stop 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 shutdown_connections 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd230066730 0x7fd230068bf0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd2500717b0 0x7fd2501e7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd25006a9a0 0x7fd25015cab0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 --2- 172.21.15.119:0/4206063057 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd25006a5c0 0x7fd25015c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 >> 172.21.15.119:0/4206063057 conn(0x7fd250070cc0 msgr2=0x7fd2500af630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:09.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 shutdown_connections 2024-09-13T16:34:09.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.261+0000 7fd2549e6700 1 -- 172.21.15.119:0/4206063057 wait complete. 2024-09-13T16:34:09.276 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:33:57.346379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:33:58.017381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:34:09.276 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:34:09.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.393+0000 7f9140d54700 1 Processor -- start 2024-09-13T16:34:09.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.393+0000 7f9140d54700 1 -- start start 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 0x7f913c06ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0ce880 con 0x7f913c06be20 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f913c0ceb60 con 0x7f913c06b2d0 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0cee40 con 0x7f913c15cfb0 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913ad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f913c06be20 0x7f913c0d1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54296/0 (socket says 172.21.15.119:54296) 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913ad9d700 1 -- 172.21.15.119:0/615574683 learned_addr learned my addr 172.21.15.119:0/615574683 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 0x7f913c06ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180699252 0 0) 0x7f913c0ce880 con 0x7f913c06be20 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91240035d0 con 0x7f913c06be20 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1847836921 0 0) 0x7f91240035d0 con 0x7f913c06be20 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 msgr2=0x7f913c06ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 0x7f913c06ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 >> v1:172.21.15.145:6789/0 conn(0x7f913c06b2d0 legacy=0x7f913c06b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0d0e40 con 0x7f913c06be20 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f91340019f0 con 0x7f913c06be20 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 >> v1:172.21.15.119:6789/0 conn(0x7f913c06be20 legacy=0x7f913c0d1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9124003a50 0x7f9124003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 0x7f9124004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124004db0 0x7f9124005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0cee40 con 0x7f9124003a50 2024-09-13T16:34:09.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912400ba30 con 0x7f9124004db0 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912400bbf0 con 0x7f913c15cfb0 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9124003a50 0x7f9124003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913ad9d700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124004db0 0x7f9124005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9139d9b700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 0x7f9124004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 -- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 msgr2=0x7f9124004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c15cfb0 0x7f9124004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 -- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124004db0 msgr2=0x7f9124005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124004db0 0x7f9124005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 -- 172.21.15.119:0/615574683 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91240035d0 con 0x7f9124003a50 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913a59c700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9124003a50 0x7f9124003e30 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9130009b70 tx=0x7f91240058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6682920cf27c9027 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913000e6c0 con 0x7f9124003a50 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f913000e820 con 0x7f9124003a50 2024-09-13T16:34:09.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913959a700 1 -- 172.21.15.119:0/615574683 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f913000eb90 con 0x7f9124003a50 2024-09-13T16:34:09.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- 172.21.15.119:0/615574683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9124003a50 msgr2=0x7f9124003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 --2- 172.21.15.119:0/615574683 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9124003a50 0x7f9124003e30 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f9130009b70 tx=0x7f91240058a0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f913ad9d700 1 -- 172.21.15.119:0/615574683 reap_dead start 2024-09-13T16:34:09.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- 172.21.15.119:0/615574683 shutdown_connections 2024-09-13T16:34:09.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- 172.21.15.119:0/615574683 >> 172.21.15.119:0/615574683 conn(0x7f913c0d2da0 msgr2=0x7f913c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- 172.21.15.119:0/615574683 shutdown_connections 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.397+0000 7f9140d54700 1 -- 172.21.15.119:0/615574683 wait complete. 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 Processor -- start 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 -- start start 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124003a50 0x7f913c15b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f913c0b6590 0x7f913c0b8640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c0c88a0 0x7f913c0c8d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0d1220 con 0x7f913c0b6590 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c0c9260 con 0x7f9124003a50 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f913c15be50 con 0x7f913c0c88a0 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f913c0b6590 0x7f913c0b8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f913c0b6590 0x7f913c0b8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36718/0 (socket says 172.21.15.119:36718) 2024-09-13T16:34:09.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 -- 172.21.15.119:0/369602992 learned_addr learned my addr 172.21.15.119:0/369602992 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f913ad9d700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c0c88a0 0x7f913c0c8d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f913a59c700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124003a50 0x7f913c15b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 -- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c0c88a0 msgr2=0x7f913c0c8d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c0c88a0 0x7f913c0c8d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 -- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124003a50 msgr2=0x7f913c15b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124003a50 0x7f913c15b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 -- 172.21.15.119:0/369602992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c0b6b30 con 0x7f913c0b6590 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9139d9b700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f913c0b6590 0x7f913c0b8640 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f912c00e830 tx=0x7f912c00eb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f912affd700 1 -- 172.21.15.119:0/369602992 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c0052e0 con 0x7f913c0b6590 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f912affd700 1 -- 172.21.15.119:0/369602992 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912c005460 con 0x7f913c0b6590 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c0b6e10 con 0x7f913c0b6590 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f912affd700 1 -- 172.21.15.119:0/369602992 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912c0038e0 con 0x7f913c0b6590 2024-09-13T16:34:09.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.401+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c0b73d0 con 0x7f913c0b6590 2024-09-13T16:34:09.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.405+0000 7f912affd700 1 -- 172.21.15.119:0/369602992 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f912c003a40 con 0x7f913c0b6590 2024-09-13T16:34:09.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.405+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c0ceb60 con 0x7f913c0b6590 2024-09-13T16:34:09.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.405+0000 7f912affd700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f911c066730 0x7f911c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.405+0000 7f912affd700 1 -- 172.21.15.119:0/369602992 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f912c0839e0 con 0x7f913c0b6590 2024-09-13T16:34:09.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.405+0000 7f913a59c700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f911c066730 0x7f911c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.405+0000 7f913a59c700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f911c066730 0x7f911c068bf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f91300090b0 tx=0x7f913001e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.413+0000 7f912affd700 1 -- 172.21.15.119:0/369602992 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f912c050c00 con 0x7f913c0b6590 2024-09-13T16:34:09.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.577+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f913c0b7560 con 0x7f913c0b6590 2024-09-13T16:34:09.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.577+0000 7f912affd700 1 -- 172.21.15.119:0/369602992 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7f912c0190a0 con 0x7f913c0b6590 2024-09-13T16:34:09.583 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:34:09.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f911c066730 msgr2=0x7f911c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f911c066730 0x7f911c068bf0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f91300090b0 tx=0x7f913001e040 comp rx=0 tx=0).stop 2024-09-13T16:34:09.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f913c0b6590 msgr2=0x7f913c0b8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f913c0b6590 0x7f913c0b8640 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f912c00e830 tx=0x7f912c00eb40 comp rx=0 tx=0).stop 2024-09-13T16:34:09.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 shutdown_connections 2024-09-13T16:34:09.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f911c066730 0x7f911c068bf0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f913c0c88a0 0x7f913c0c8d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f913c0b6590 0x7f913c0b8640 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 --2- 172.21.15.119:0/369602992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9124003a50 0x7f913c15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 >> 172.21.15.119:0/369602992 conn(0x7f913c0d2da0 msgr2=0x7f913c0b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:09.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 shutdown_connections 2024-09-13T16:34:09.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.581+0000 7f9140d54700 1 -- 172.21.15.119:0/369602992 wait complete. 2024-09-13T16:34:09.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:34:09.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.709+0000 7f08e492c700 1 Processor -- start 2024-09-13T16:34:09.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.709+0000 7f08e492c700 1 -- start start 2024-09-13T16:34:09.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e00ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f08e015dbe0 0x7f08e00cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08e00cd4d0 con 0x7f08e00b3260 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08e0071be0 con 0x7f08e015dbe0 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e0159af0 con 0x7f08e00add60 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f08e015dbe0 0x7f08e00cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f08e015dbe0 0x7f08e00cd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36734/0 (socket says 172.21.15.119:36734) 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 -- 172.21.15.119:0/1817355402 learned_addr learned my addr 172.21.15.119:0/1817355402 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08ded9d700 1 --2- 172.21.15.119:0/1817355402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e00ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 -- 172.21.15.119:0/1817355402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 msgr2=0x7f08e00ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 --2- 172.21.15.119:0/1817355402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e00ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 -- 172.21.15.119:0/1817355402 >> v1:172.21.15.145:6789/0 conn(0x7f08e00b3260 legacy=0x7f08e006fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 -- 172.21.15.119:0/1817355402 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e015e130 con 0x7f08e015dbe0 2024-09-13T16:34:09.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08ded9d700 1 --2- 172.21.15.119:0/1817355402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:34:09.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08de59c700 1 --2- 172.21.15.119:0/1817355402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e00cd960 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009e50 tx=0x7f08d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dd1ec136a7d8e8f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08ddd9b700 1 -- 172.21.15.119:0/1817355402 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d000e070 con 0x7f08e015dbe0 2024-09-13T16:34:09.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08ddd9b700 1 -- 172.21.15.119:0/1817355402 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d000b9c0 con 0x7f08e015dbe0 2024-09-13T16:34:09.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08ddd9b700 1 -- 172.21.15.119:0/1817355402 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d000bd70 con 0x7f08e015dbe0 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- 172.21.15.119:0/1817355402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 msgr2=0x7f08e00cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 --2- 172.21.15.119:0/1817355402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e00cd960 secure :-1 s=READY pgs=1608 cs=0 l=1 rev1=1 crypto rx=0x7f08d0009e50 tx=0x7f08d000b330 comp rx=0 tx=0).stop 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- 172.21.15.119:0/1817355402 shutdown_connections 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 --2- 172.21.15.119:0/1817355402 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e00cd960 unknown :-1 s=CLOSED pgs=1608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 --2- 172.21.15.119:0/1817355402 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e00ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- 172.21.15.119:0/1817355402 >> 172.21.15.119:0/1817355402 conn(0x7f08e00b0980 msgr2=0x7f08e00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- 172.21.15.119:0/1817355402 shutdown_connections 2024-09-13T16:34:09.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.713+0000 7f08e492c700 1 -- 172.21.15.119:0/1817355402 wait complete. 2024-09-13T16:34:09.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 Processor -- start 2024-09-13T16:34:09.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 -- start start 2024-09-13T16:34:09.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e014d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e00b3260 0x7f08e01505b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e0150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e014fc50 con 0x7f08e015dbe0 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08e014b0a0 con 0x7f08e00b3260 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08e014f040 con 0x7f08e00add60 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e0150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e0150af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36736/0 (socket says 172.21.15.119:36736) 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 -- 172.21.15.119:0/1679024868 learned_addr learned my addr 172.21.15.119:0/1679024868 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08de59c700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e00b3260 0x7f08e01505b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08ded9d700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e014d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 -- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 msgr2=0x7f08e014d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e014d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 -- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e00b3260 msgr2=0x7f08e01505b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e00b3260 0x7f08e01505b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 -- 172.21.15.119:0/1679024868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08d0009b30 con 0x7f08e015dbe0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08df59e700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e0150af0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f08d800c4b0 tx=0x7f08d800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08d77fe700 1 -- 172.21.15.119:0/1679024868 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80052d0 con 0x7f08e015dbe0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08d77fe700 1 -- 172.21.15.119:0/1679024868 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08d8005430 con 0x7f08e015dbe0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08e014f320 con 0x7f08e015dbe0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08d77fe700 1 -- 172.21.15.119:0/1679024868 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08d80055a0 con 0x7f08e015dbe0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.717+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08e014f8e0 con 0x7f08e015dbe0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.721+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08e0071260 con 0x7f08e015dbe0 2024-09-13T16:34:09.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.721+0000 7f08d77fe700 1 -- 172.21.15.119:0/1679024868 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f08d80038e0 con 0x7f08e015dbe0 2024-09-13T16:34:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.721+0000 7f08d77fe700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f08c0066730 0x7f08c0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:09.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.721+0000 7f08d77fe700 1 -- 172.21.15.119:0/1679024868 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f08d8083520 con 0x7f08e015dbe0 2024-09-13T16:34:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.721+0000 7f08ded9d700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f08c0066730 0x7f08c0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:09.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.721+0000 7f08ded9d700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f08c0066730 0x7f08c0068bf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f08c8006fd0 tx=0x7f08c8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:09.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.729+0000 7f08d77fe700 1 -- 172.21.15.119:0/1679024868 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f08d80507c0 con 0x7f08e015dbe0 2024-09-13T16:34:09.899 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.897+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f08e014b1e0 con 0x7f08e015dbe0 2024-09-13T16:34:09.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.897+0000 7f08d77fe700 1 -- 172.21.15.119:0/1679024868 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f08d8053de0 con 0x7f08e015dbe0 2024-09-13T16:34:09.900 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:34:09.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f08c0066730 msgr2=0x7f08c0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f08c0066730 0x7f08c0068bf0 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f08c8006fd0 tx=0x7f08c8008040 comp rx=0 tx=0).stop 2024-09-13T16:34:09.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 msgr2=0x7f08e0150af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:09.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e0150af0 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f08d800c4b0 tx=0x7f08d800c7c0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 shutdown_connections 2024-09-13T16:34:09.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f08c0066730 0x7f08c0068bf0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f08e015dbe0 0x7f08e0150af0 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f08e00b3260 0x7f08e01505b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 --2- 172.21.15.119:0/1679024868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f08e00add60 0x7f08e014d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:09.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 >> 172.21.15.119:0/1679024868 conn(0x7f08e00b0980 msgr2=0x7f08e00b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:09.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 shutdown_connections 2024-09-13T16:34:09.905 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:09.901+0000 7f08e492c700 1 -- 172.21.15.119:0/1679024868 wait complete. 2024-09-13T16:34:09.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:34:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff623335700 1 Processor -- start 2024-09-13T16:34:10.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff623335700 1 -- start start 2024-09-13T16:34:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff623335700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff61c0ce5f0 0x7ff61c0cff60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff623335700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0d0fb0 0x7ff61c0d1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff623335700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61c0cd600 con 0x7ff61c15cae0 2024-09-13T16:34:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff623335700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff61c0ce1d0 con 0x7ff61c0ce5f0 2024-09-13T16:34:10.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff623335700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0cea60 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff621b32700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff61c0ce5f0 0x7ff61c0cff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff622b34700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0d0fb0 0x7ff61c0d1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff621b32700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff61c0ce5f0 0x7ff61c0cff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:60026/0 (socket says 172.21.15.119:60026) 2024-09-13T16:34:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff621b32700 1 -- 172.21.15.119:0/1351319388 learned_addr learned my addr 172.21.15.119:0/1351319388 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff622b34700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0d0fb0 0x7ff61c0d1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:59188/0 (socket says 172.21.15.119:59188) 2024-09-13T16:34:10.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.041+0000 7ff622333700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff61c15cae0 0x7ff61c0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:51040/0 (socket says 172.21.15.119:51040) 2024-09-13T16:34:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621b32700 1 -- 172.21.15.119:0/1351319388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0d0fb0 msgr2=0x7ff61c0d1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621b32700 1 --2- 172.21.15.119:0/1351319388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0d0fb0 0x7ff61c0d1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621b32700 1 -- 172.21.15.119:0/1351319388 >> v1:172.21.15.145:6789/0 conn(0x7ff61c15cae0 legacy=0x7ff61c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621331700 1 -- 172.21.15.119:0/1351319388 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434791116 0 0) 0x7ff61c0cd600 con 0x7ff61c15cae0 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621b32700 1 -- 172.21.15.119:0/1351319388 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff61c0d0dd0 con 0x7ff61c0ce5f0 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621b32700 1 --2- 172.21.15.119:0/1351319388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c0ce5f0 0x7ff61c0cff60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff618009e80 tx=0x7ff61800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=159fb9207be77cba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621331700 1 -- 172.21.15.119:0/1351319388 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61800e070 con 0x7ff61c0ce5f0 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621331700 1 -- 172.21.15.119:0/1351319388 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61800b8d0 con 0x7ff61c0ce5f0 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff621331700 1 -- 172.21.15.119:0/1351319388 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61800bc80 con 0x7ff61c0ce5f0 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 -- 172.21.15.119:0/1351319388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c0ce5f0 msgr2=0x7ff61c0cff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 --2- 172.21.15.119:0/1351319388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c0ce5f0 0x7ff61c0cff60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff618009e80 tx=0x7ff61800b330 comp rx=0 tx=0).stop 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 -- 172.21.15.119:0/1351319388 shutdown_connections 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 --2- 172.21.15.119:0/1351319388 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0d0fb0 0x7ff61c0d1420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 --2- 172.21.15.119:0/1351319388 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c0ce5f0 0x7ff61c0cff60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 -- 172.21.15.119:0/1351319388 >> 172.21.15.119:0/1351319388 conn(0x7ff61c0d29f0 msgr2=0x7ff61c0d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:10.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 -- 172.21.15.119:0/1351319388 shutdown_connections 2024-09-13T16:34:10.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 -- 172.21.15.119:0/1351319388 wait complete. 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 Processor -- start 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.045+0000 7ff623335700 1 -- start start 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0ce5f0 0x7ff61c15f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61c0d0fb0 0x7ff61c151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c15cae0 0x7ff61c15db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c15e680 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c15e3a0 con 0x7ff61c15cae0 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff61c0c9c60 con 0x7ff61c0ce5f0 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61c0d0fb0 0x7ff61c151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61c0d0fb0 0x7ff61c151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36746/0 (socket says 172.21.15.119:36746) 2024-09-13T16:34:10.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 -- 172.21.15.119:0/302212670 learned_addr learned my addr 172.21.15.119:0/302212670 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff622b34700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c15cae0 0x7ff61c15db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff622333700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0ce5f0 0x7ff61c15f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 -- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0ce5f0 msgr2=0x7ff61c15f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0ce5f0 0x7ff61c15f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 -- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c15cae0 msgr2=0x7ff61c15db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c15cae0 0x7ff61c15db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 -- 172.21.15.119:0/302212670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff618009b30 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff621b32700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61c0d0fb0 0x7ff61c151ef0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff61c08b2a0 tx=0x7ff618004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff612ffd700 1 -- 172.21.15.119:0/302212670 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61801b630 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff612ffd700 1 -- 172.21.15.119:0/302212670 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61801b790 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff612ffd700 1 -- 172.21.15.119:0/302212670 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff61801bb40 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff61c0c9f40 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff61c15e060 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.049+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61c0cdaf0 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.053+0000 7ff612ffd700 1 -- 172.21.15.119:0/302212670 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff61801bca0 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.053+0000 7ff612ffd700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff600066730 0x7ff600068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.053+0000 7ff612ffd700 1 -- 172.21.15.119:0/302212670 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff618085820 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.053+0000 7ff622333700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff600066730 0x7ff600068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.053+0000 7ff622333700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff600066730 0x7ff600068bf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff60c000f60 tx=0x7ff60c007e00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:10.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.057+0000 7ff612ffd700 1 -- 172.21.15.119:0/302212670 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff61c0cdaf0 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.233+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff61c0ca460 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.233+0000 7ff612ffd700 1 -- 172.21.15.119:0/302212670 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff618021020 con 0x7ff61c0d0fb0 2024-09-13T16:34:10.237 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff600066730 msgr2=0x7ff600068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff600066730 0x7ff600068bf0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7ff60c000f60 tx=0x7ff60c007e00 comp rx=0 tx=0).stop 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61c0d0fb0 msgr2=0x7ff61c151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61c0d0fb0 0x7ff61c151ef0 secure :-1 s=READY pgs=1610 cs=0 l=1 rev1=1 crypto rx=0x7ff61c08b2a0 tx=0x7ff618004af0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 shutdown_connections 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff600066730 0x7ff600068bf0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61c15cae0 0x7ff61c15db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61c0d0fb0 0x7ff61c151ef0 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 --2- 172.21.15.119:0/302212670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff61c0ce5f0 0x7ff61c15f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 >> 172.21.15.119:0/302212670 conn(0x7ff61c0d29f0 msgr2=0x7ff61c0b2ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:10.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 shutdown_connections 2024-09-13T16:34:10.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.237+0000 7ff623335700 1 -- 172.21.15.119:0/302212670 wait complete. 2024-09-13T16:34:10.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:34:10.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5021cc700 1 Processor -- start 2024-09-13T16:34:10.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5021cc700 1 -- start start 2024-09-13T16:34:10.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5021cc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd4fc0b17f0 0x7fd4fc0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5021cc700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc0705d0 con 0x7fd4fc0b2200 2024-09-13T16:34:10.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5021cc700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4fc06fca0 con 0x7fd4fc0cd8d0 2024-09-13T16:34:10.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5021cc700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4fc06ff80 con 0x7fd4fc0b17f0 2024-09-13T16:34:10.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5019cb700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd4fc0b2200 0x7fd4fc070a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54304/0 (socket says 172.21.15.119:54304) 2024-09-13T16:34:10.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5019cb700 1 -- 172.21.15.119:0/2608157701 learned_addr learned my addr 172.21.15.119:0/2608157701 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:10.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd5009c9700 1 --2- 172.21.15.119:0/2608157701 >> v2:172.21.15.145:3300/0 conn(0x7fd4fc0b17f0 0x7fd4fc0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 425578113 0 0) 0x7fd4fc0705d0 con 0x7fd4fc0b2200 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4e80035d0 con 0x7fd4fc0b2200 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.361+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940021276 0 0) 0x7fd4fc06fca0 con 0x7fd4fc0cd8d0 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4fc0705d0 con 0x7fd4fc0cd8d0 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2497403684 0 0) 0x7fd4e80035d0 con 0x7fd4fc0b2200 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 >> v2:172.21.15.145:3300/0 conn(0x7fd4fc0b17f0 msgr2=0x7fd4fc0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 --2- 172.21.15.119:0/2608157701 >> v2:172.21.15.145:3300/0 conn(0x7fd4fc0b17f0 0x7fd4fc0cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 >> v1:172.21.15.145:6789/0 conn(0x7fd4fc0cd8d0 legacy=0x7fd4fc06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc070260 con 0x7fd4fc0b2200 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fd4f4001670 con 0x7fd4fc0b2200 2024-09-13T16:34:10.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5009c9700 1 --2- 172.21.15.119:0/2608157701 >> v2:172.21.15.145:3300/0 conn(0x7fd4fc0b17f0 0x7fd4fc0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 >> v1:172.21.15.119:6789/0 conn(0x7fd4fc0b2200 legacy=0x7fd4fc070a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4e8003bb0 0x7fd4e8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e8004870 0x7fd4e8004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8005230 0x7fd4e800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc06ff80 con 0x7fd4e8003bb0 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e800c160 con 0x7fd4e8005230 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e800c320 con 0x7fd4e8004870 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5011ca700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4e8003bb0 0x7fd4e8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5019cb700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8005230 0x7fd4e800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5009c9700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e8004870 0x7fd4e8004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5011ca700 1 -- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e8004870 msgr2=0x7fd4e8004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5011ca700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e8004870 0x7fd4e8004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5011ca700 1 -- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8005230 msgr2=0x7fd4e800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:10.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5011ca700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8005230 0x7fd4e800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5011ca700 1 -- 172.21.15.119:0/2608157701 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e80035d0 con 0x7fd4e8003bb0 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5019cb700 1 -- 172.21.15.119:0/2608157701 reap_dead start 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5009c9700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4e8004870 0x7fd4e8004cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5011ca700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4e8003bb0 0x7fd4e8003fb0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec001960 tx=0x7fd4e80058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e207b9314a0f62a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec001450 con 0x7fd4e8003bb0 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ec006aa0 con 0x7fd4e8003bb0 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd4f3fff700 1 -- 172.21.15.119:0/2608157701 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4ec006e50 con 0x7fd4e8003bb0 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 -- 172.21.15.119:0/2608157701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4e8003bb0 msgr2=0x7fd4e8003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4e8003bb0 0x7fd4e8003fb0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec001960 tx=0x7fd4e80058a0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 -- 172.21.15.119:0/2608157701 shutdown_connections 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 --2- 172.21.15.119:0/2608157701 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4e8003bb0 0x7fd4e8003fb0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 -- 172.21.15.119:0/2608157701 >> 172.21.15.119:0/2608157701 conn(0x7fd4fc06a350 msgr2=0x7fd4fc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:10.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 -- 172.21.15.119:0/2608157701 shutdown_connections 2024-09-13T16:34:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 -- 172.21.15.119:0/2608157701 wait complete. 2024-09-13T16:34:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 Processor -- start 2024-09-13T16:34:10.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.365+0000 7fd5021cc700 1 -- start start 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8003bb0 0x7fd4fc1f7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0ab810 0x7fd4fc0abc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4fc0af650 0x7fd4fc0afad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc14d6c0 con 0x7fd4fc0ab810 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc0ac1d0 con 0x7fd4e8003bb0 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4fc0b0380 con 0x7fd4fc0af650 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0ab810 0x7fd4fc0abc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0ab810 0x7fd4fc0abc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36758/0 (socket says 172.21.15.119:36758) 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 -- 172.21.15.119:0/1000577497 learned_addr learned my addr 172.21.15.119:0/1000577497 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:10.375 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5019cb700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4fc0af650 0x7fd4fc0afad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5011ca700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8003bb0 0x7fd4fc1f7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 -- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4fc0af650 msgr2=0x7fd4fc0afad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4fc0af650 0x7fd4fc0afad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 -- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8003bb0 msgr2=0x7fd4fc1f7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8003bb0 0x7fd4fc1f7200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 -- 172.21.15.119:0/1000577497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc0b0660 con 0x7fd4fc0ab810 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5019cb700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4fc0af650 0x7fd4fc0afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5009c9700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0ab810 0x7fd4fc0abc90 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800f380 tx=0x7fd4f800f740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd4f1ffb700 1 -- 172.21.15.119:0/1000577497 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f800fee0 con 0x7fd4fc0ab810 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd4f1ffb700 1 -- 172.21.15.119:0/1000577497 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4f8016620 con 0x7fd4fc0ab810 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4fc0b0940 con 0x7fd4fc0ab810 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd4f1ffb700 1 -- 172.21.15.119:0/1000577497 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4f80169b0 con 0x7fd4fc0ab810 2024-09-13T16:34:10.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4fc0b0c20 con 0x7fd4fc0ab810 2024-09-13T16:34:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd4f1ffb700 1 -- 172.21.15.119:0/1000577497 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd4f8016b10 con 0x7fd4fc0ab810 2024-09-13T16:34:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4fc14d830 con 0x7fd4fc0ab810 2024-09-13T16:34:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd4f1ffb700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd4e0066760 0x7fd4e0068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd4f1ffb700 1 -- 172.21.15.119:0/1000577497 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd4f80852a0 con 0x7fd4fc0ab810 2024-09-13T16:34:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.369+0000 7fd5011ca700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd4e0066760 0x7fd4e0068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:10.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.373+0000 7fd5011ca700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd4e0066760 0x7fd4e0068c20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec015040 tx=0x7fd4ec005040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:10.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.381+0000 7fd4f1ffb700 1 -- 172.21.15.119:0/1000577497 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd4fc14d830 con 0x7fd4fc0ab810 2024-09-13T16:34:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.553+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd4fc14d800 con 0x7fd4fc0ab810 2024-09-13T16:34:10.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.553+0000 7fd4f1ffb700 1 -- 172.21.15.119:0/1000577497 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7fd4f8052420 con 0x7fd4fc0ab810 2024-09-13T16:34:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd4e0066760 msgr2=0x7fd4e0068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd4e0066760 0x7fd4e0068c20 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fd4ec015040 tx=0x7fd4ec005040 comp rx=0 tx=0).stop 2024-09-13T16:34:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0ab810 msgr2=0x7fd4fc0abc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0ab810 0x7fd4fc0abc90 secure :-1 s=READY pgs=1613 cs=0 l=1 rev1=1 crypto rx=0x7fd4f800f380 tx=0x7fd4f800f740 comp rx=0 tx=0).stop 2024-09-13T16:34:10.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 shutdown_connections 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd4e0066760 0x7fd4e0068c20 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4fc0af650 0x7fd4fc0afad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd4fc0ab810 0x7fd4fc0abc90 unknown :-1 s=CLOSED pgs=1613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 --2- 172.21.15.119:0/1000577497 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd4e8003bb0 0x7fd4fc1f7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 >> 172.21.15.119:0/1000577497 conn(0x7fd4fc06a350 msgr2=0x7fd4fc0ad7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 shutdown_connections 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:10.557+0000 7fd5021cc700 1 -- 172.21.15.119:0/1000577497 wait complete. 2024-09-13T16:34:10.562 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:34:10.574 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:34:10.574 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:10.574 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:10.587 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[50845]: fuse finished with error 0 and tester_r 0 2024-09-13T16:34:10.590 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:34:16.691 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:34:16.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:16.692 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:34:16.700 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:34:16.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f6566830700 1 Processor -- start 2024-09-13T16:34:16.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f6566830700 1 -- start start 2024-09-13T16:34:16.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f6566830700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f65600b1fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:16.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f6566830700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f65600707b0 0x7f656006b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:16.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f6566830700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656006d180 con 0x7f656006b8d0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f6566830700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656006c360 con 0x7f65600707b0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f6566830700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f656006bd50 con 0x7f65600703d0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656602f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f656006b8d0 0x7f656006d610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58386/0 (socket says 172.21.15.119:58386) 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656602f700 1 -- 172.21.15.119:0/3980300900 learned_addr learned my addr 172.21.15.119:0/3980300900 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656582e700 1 --2- 172.21.15.119:0/3980300900 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f65600b1fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656502d700 1 --2- 172.21.15.119:0/3980300900 >> v2:172.21.15.145:3300/0 conn(0x7f65600707b0 0x7f656006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656482c700 1 -- 172.21.15.119:0/3980300900 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2876624363 0 0) 0x7f656006d180 con 0x7f656006b8d0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656482c700 1 -- 172.21.15.119:0/3980300900 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c0035d0 con 0x7f656006b8d0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656582e700 1 -- 172.21.15.119:0/3980300900 >> v2:172.21.15.145:3300/0 conn(0x7f65600707b0 msgr2=0x7f656006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656582e700 1 --2- 172.21.15.119:0/3980300900 >> v2:172.21.15.145:3300/0 conn(0x7f65600707b0 0x7f656006b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656582e700 1 -- 172.21.15.119:0/3980300900 >> v1:172.21.15.119:6789/0 conn(0x7f656006b8d0 legacy=0x7f656006d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656582e700 1 -- 172.21.15.119:0/3980300900 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656006c570 con 0x7f65600703d0 2024-09-13T16:34:16.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.861+0000 7f656582e700 1 --2- 172.21.15.119:0/3980300900 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f65600b1fd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f655c009850 tx=0x7f655c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a65f2252c613a03 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f656482c700 1 -- 172.21.15.119:0/3980300900 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c011830 con 0x7f65600703d0 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f656482c700 1 -- 172.21.15.119:0/3980300900 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655c011990 con 0x7f65600703d0 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f656482c700 1 -- 172.21.15.119:0/3980300900 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f655c011d00 con 0x7f65600703d0 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 -- 172.21.15.119:0/3980300900 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 msgr2=0x7f65600b1fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 --2- 172.21.15.119:0/3980300900 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f65600b1fd0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f655c009850 tx=0x7f655c009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 -- 172.21.15.119:0/3980300900 shutdown_connections 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 --2- 172.21.15.119:0/3980300900 >> v2:172.21.15.145:3300/0 conn(0x7f65600707b0 0x7f656006b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 --2- 172.21.15.119:0/3980300900 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f65600b1fd0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 -- 172.21.15.119:0/3980300900 >> 172.21.15.119:0/3980300900 conn(0x7f65600ae750 msgr2=0x7f65600b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 -- 172.21.15.119:0/3980300900 shutdown_connections 2024-09-13T16:34:16.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 -- 172.21.15.119:0/3980300900 wait complete. 2024-09-13T16:34:16.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 Processor -- start 2024-09-13T16:34:16.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.865+0000 7f6566830700 1 -- start start 2024-09-13T16:34:16.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f6560159bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:16.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65600707b0 0x7f65600ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:16.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65600b3bb0 0x7f656015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:16.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f656015a100 con 0x7f65600b3bb0 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65601f3860 con 0x7f65600707b0 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65601f3b40 con 0x7f65600703d0 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65600b3bb0 0x7f656015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656582e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f6560159bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65600b3bb0 0x7f656015b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53668/0 (socket says 172.21.15.119:53668) 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 -- 172.21.15.119:0/1478421118 learned_addr learned my addr 172.21.15.119:0/1478421118 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656502d700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65600707b0 0x7f65600ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 -- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 msgr2=0x7f6560159bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f6560159bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 -- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65600707b0 msgr2=0x7f65600ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65600707b0 0x7f65600ac1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 -- 172.21.15.119:0/1478421118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655c0094b0 con 0x7f65600b3bb0 2024-09-13T16:34:16.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f656602f700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65600b3bb0 0x7f656015b920 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f655800caa0 tx=0x7f655800cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:16.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f65567fc700 1 -- 172.21.15.119:0/1478421118 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6558007480 con 0x7f65600b3bb0 2024-09-13T16:34:16.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f65567fc700 1 -- 172.21.15.119:0/1478421118 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65580075e0 con 0x7f65600b3bb0 2024-09-13T16:34:16.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65601f3e20 con 0x7f65600b3bb0 2024-09-13T16:34:16.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f65567fc700 1 -- 172.21.15.119:0/1478421118 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6558007970 con 0x7f65600b3bb0 2024-09-13T16:34:16.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.869+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65601f34c0 con 0x7f65600b3bb0 2024-09-13T16:34:16.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.873+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656013fcf0 con 0x7f65600b3bb0 2024-09-13T16:34:16.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.873+0000 7f65567fc700 1 -- 172.21.15.119:0/1478421118 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f655801b3b0 con 0x7f65600b3bb0 2024-09-13T16:34:16.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.873+0000 7f65567fc700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6544066730 0x7f6544068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:16.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.873+0000 7f65567fc700 1 -- 172.21.15.119:0/1478421118 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f6558083ba0 con 0x7f65600b3bb0 2024-09-13T16:34:16.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.873+0000 7f656582e700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6544066730 0x7f6544068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:16.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.873+0000 7f656582e700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6544066730 0x7f6544068bf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f655c0090b0 tx=0x7f655c01c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:16.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:16.881+0000 7f65567fc700 1 -- 172.21.15.119:0/1478421118 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6558050da0 con 0x7f65600b3bb0 2024-09-13T16:34:17.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.085+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65600cfbe0 con 0x7f65600b3bb0 2024-09-13T16:34:17.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.085+0000 7f65567fc700 1 -- 172.21.15.119:0/1478421118 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4748 (secure 0 0 0) 0x7f65580543c0 con 0x7f65600b3bb0 2024-09-13T16:34:17.090 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:17.090 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":76,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:34:02.039855+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8829":{"gid":8829,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3700300033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3700300033},{"type":"v1","addr":"172.21.15.119:6835","nonce":3700300033}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:34:17.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6544066730 msgr2=0x7f6544068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6544066730 0x7f6544068bf0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f655c0090b0 tx=0x7f655c01c040 comp rx=0 tx=0).stop 2024-09-13T16:34:17.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65600b3bb0 msgr2=0x7f656015b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65600b3bb0 0x7f656015b920 secure :-1 s=READY pgs=1615 cs=0 l=1 rev1=1 crypto rx=0x7f655800caa0 tx=0x7f655800cad0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 shutdown_connections 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6544066730 0x7f6544068bf0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f65600b3bb0 0x7f656015b920 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f65600707b0 0x7f65600ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 --2- 172.21.15.119:0/1478421118 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f65600703d0 0x7f6560159bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 >> 172.21.15.119:0/1478421118 conn(0x7f656006b8d0 msgr2=0x7f65600ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 shutdown_connections 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.089+0000 7f6566830700 1 -- 172.21.15.119:0/1478421118 wait complete. 2024-09-13T16:34:17.094 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 76 2024-09-13T16:34:17.106 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-13T16:34:17.239 INFO:teuthology.orchestra.run.smithi119.stdout:{"message":"","return_code":0} 2024-09-13T16:34:17.239 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:34:17.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:34:17.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1998ef700 1 Processor -- start 2024-09-13T16:34:17.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1998ef700 1 -- start start 2024-09-13T16:34:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1998ef700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd194071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1998ef700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd19406d540 0x7fd194071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1998ef700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19406de00 con 0x7fd194158f10 2024-09-13T16:34:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1998ef700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19406db20 con 0x7fd19406d540 2024-09-13T16:34:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1998ef700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1940b1920 con 0x7fd194158b30 2024-09-13T16:34:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd193fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd194158f10 0x7fd19406ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58396/0 (socket says 172.21.15.119:58396) 2024-09-13T16:34:17.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd193fff700 1 -- 172.21.15.119:0/3074521989 learned_addr learned my addr 172.21.15.119:0/3074521989 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1990ee700 1 --2- 172.21.15.119:0/3074521989 >> v2:172.21.15.145:3300/0 conn(0x7fd19406d540 0x7fd194071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1988ed700 1 --2- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd194071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1937fe700 1 -- 172.21.15.119:0/3074521989 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286405631 0 0) 0x7fd19406de00 con 0x7fd194158f10 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.345+0000 7fd1937fe700 1 -- 172.21.15.119:0/3074521989 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1800057e0 con 0x7fd194158f10 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1990ee700 1 -- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 msgr2=0x7fd194071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1990ee700 1 --2- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd194071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1990ee700 1 -- 172.21.15.119:0/3074521989 >> v1:172.21.15.119:6789/0 conn(0x7fd194158f10 legacy=0x7fd19406ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1990ee700 1 -- 172.21.15.119:0/3074521989 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1940b1c00 con 0x7fd19406d540 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1990ee700 1 --2- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19406d540 0x7fd194071570 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009850 tx=0x7fd18c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17f789db9919862f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1937fe700 1 -- 172.21.15.119:0/3074521989 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c011830 con 0x7fd19406d540 2024-09-13T16:34:17.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1937fe700 1 -- 172.21.15.119:0/3074521989 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18c011990 con 0x7fd19406d540 2024-09-13T16:34:17.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1937fe700 1 -- 172.21.15.119:0/3074521989 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd18c011d00 con 0x7fd19406d540 2024-09-13T16:34:17.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 -- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19406d540 msgr2=0x7fd194071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 --2- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19406d540 0x7fd194071570 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7fd18c009850 tx=0x7fd18c009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:17.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 -- 172.21.15.119:0/3074521989 shutdown_connections 2024-09-13T16:34:17.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 --2- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd19406d540 0x7fd194071570 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 --2- 172.21.15.119:0/3074521989 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd194071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 -- 172.21.15.119:0/3074521989 >> 172.21.15.119:0/3074521989 conn(0x7fd19406a350 msgr2=0x7fd19406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:17.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 -- 172.21.15.119:0/3074521989 shutdown_connections 2024-09-13T16:34:17.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 -- 172.21.15.119:0/3074521989 wait complete. 2024-09-13T16:34:17.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 Processor -- start 2024-09-13T16:34:17.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.349+0000 7fd1998ef700 1 -- start start 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd19406d540 0x7fd194147090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd1940aa430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd194158f10 0x7fd1940a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19415b440 con 0x7fd19406d540 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd19415a7c0 con 0x7fd194158f10 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd19415aaa0 con 0x7fd194158b30 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd19406d540 0x7fd194147090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd19406d540 0x7fd194147090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53676/0 (socket says 172.21.15.119:53676) 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 -- 172.21.15.119:0/2253175991 learned_addr learned my addr 172.21.15.119:0/2253175991 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd193fff700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd1940aa430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1990ee700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd194158f10 0x7fd1940a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 -- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 msgr2=0x7fd1940aa430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd1940aa430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 -- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd194158f10 msgr2=0x7fd1940a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd194158f10 0x7fd1940a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 -- 172.21.15.119:0/2253175991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18c0094b0 con 0x7fd19406d540 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd193fff700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd1940aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1990ee700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd194158f10 0x7fd1940a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1988ed700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd19406d540 0x7fd194147090 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00e9d0 tx=0x7fd17c00ed90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1917fa700 1 -- 172.21.15.119:0/2253175991 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c009720 con 0x7fd19406d540 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19415ad80 con 0x7fd19406d540 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1917fa700 1 -- 172.21.15.119:0/2253175991 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17c00c850 con 0x7fd19406d540 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1917fa700 1 -- 172.21.15.119:0/2253175991 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17c00cbc0 con 0x7fd19406d540 2024-09-13T16:34:17.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1941475d0 con 0x7fd19406d540 2024-09-13T16:34:17.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.353+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19406ce30 con 0x7fd19406d540 2024-09-13T16:34:17.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.357+0000 7fd1917fa700 1 -- 172.21.15.119:0/2253175991 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd17c00cd20 con 0x7fd19406d540 2024-09-13T16:34:17.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.357+0000 7fd1917fa700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd174066730 0x7fd174068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.357+0000 7fd1917fa700 1 -- 172.21.15.119:0/2253175991 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd17c084480 con 0x7fd19406d540 2024-09-13T16:34:17.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.357+0000 7fd193fff700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd174066730 0x7fd174068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.357+0000 7fd193fff700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd174066730 0x7fd174068bf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd19406ab60 tx=0x7fd184009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.361+0000 7fd1917fa700 1 -- 172.21.15.119:0/2253175991 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd17c051600 con 0x7fd19406d540 2024-09-13T16:34:17.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.561+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd1940af8b0 con 0x7fd19406d540 2024-09-13T16:34:17.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.561+0000 7fd1917fa700 1 -- 172.21.15.119:0/2253175991 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fd17c054c20 con 0x7fd19406d540 2024-09-13T16:34:17.683 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:17.683 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":76,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:34:02.039855+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8829":{"gid":8829,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3700300033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3700300033},{"type":"v1","addr":"172.21.15.119:6835","nonce":3700300033}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd174066730 msgr2=0x7fd174068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd174066730 0x7fd174068bf0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fd19406ab60 tx=0x7fd184009040 comp rx=0 tx=0).stop 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd19406d540 msgr2=0x7fd194147090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd19406d540 0x7fd194147090 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7fd17c00e9d0 tx=0x7fd17c00ed90 comp rx=0 tx=0).stop 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 shutdown_connections 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd174066730 0x7fd174068bf0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd194158f10 0x7fd1940a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd194158b30 0x7fd1940aa430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 --2- 172.21.15.119:0/2253175991 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd19406d540 0x7fd194147090 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 >> 172.21.15.119:0/2253175991 conn(0x7fd19406a350 msgr2=0x7fd1941522a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:17.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 shutdown_connections 2024-09-13T16:34:17.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.565+0000 7fd1998ef700 1 -- 172.21.15.119:0/2253175991 wait complete. 2024-09-13T16:34:17.686 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 76 2024-09-13T16:34:17.686 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata get mds0_inotable - 2024-09-13T16:34:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.725+0000 7fbf2b383140 1 Processor -- start 2024-09-13T16:34:17.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.725+0000 7fbf2b383140 1 -- start start 2024-09-13T16:34:17.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf2b383140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112e94e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf2b383140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e93c7a0 0x56112e94e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf2b383140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e9e0620 0x56112ea78020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf2b383140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56112e85a8e0 con 0x56112e93be80 2024-09-13T16:34:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf2b383140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56112e85a580 con 0x56112e93c7a0 2024-09-13T16:34:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf2b383140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56112e8308d0 con 0x56112e9e0620 2024-09-13T16:34:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112e94e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112e94e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53692/0 (socket says 172.21.15.119:53692) 2024-09-13T16:34:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 -- 172.21.15.119:0/1671430081 learned_addr learned my addr 172.21.15.119:0/1671430081 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf286b7700 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e93c7a0 0x56112e94e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf296b9700 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e9e0620 0x56112ea78020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 -- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e9e0620 msgr2=0x56112ea78020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e9e0620 0x56112ea78020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 -- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e93c7a0 msgr2=0x56112e94e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e93c7a0 0x56112e94e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 -- 172.21.15.119:0/1671430081 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf10005de0 con 0x56112e93be80 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf286b7700 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e93c7a0 0x56112e94e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf28eb8700 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112e94e0b0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009890 tx=0x7fbf20005a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=161aaee88b37e108 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf276b5700 1 -- 172.21.15.119:0/1671430081 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000f040 con 0x56112e93be80 2024-09-13T16:34:17.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf276b5700 1 -- 172.21.15.119:0/1671430081 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf200089b0 con 0x56112e93be80 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.729+0000 7fbf276b5700 1 -- 172.21.15.119:0/1671430081 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20008d60 con 0x56112e93be80 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- 172.21.15.119:0/1671430081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 msgr2=0x56112e94e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112e94e0b0 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009890 tx=0x7fbf20005a00 comp rx=0 tx=0).stop 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- 172.21.15.119:0/1671430081 shutdown_connections 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e9e0620 0x56112ea78020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e93c7a0 0x56112e94e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 --2- 172.21.15.119:0/1671430081 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112e94e0b0 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- 172.21.15.119:0/1671430081 >> 172.21.15.119:0/1671430081 conn(0x56112e90d110 msgr2=0x56112e9c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:17.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- 172.21.15.119:0/1671430081 shutdown_connections 2024-09-13T16:34:17.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- 172.21.15.119:0/1671430081 wait complete. 2024-09-13T16:34:17.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 Processor -- start 2024-09-13T16:34:17.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- start start 2024-09-13T16:34:17.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112ea65740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e93c7a0 0x56112ea68ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e9e0620 0x56112e90e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56112ea67070 con 0x56112e93be80 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56112e9e0330 con 0x56112e9e0620 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.733+0000 7fbf2b383140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56112ea6b950 con 0x56112e93c7a0 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112ea65740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf286b7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e93c7a0 0x56112ea68ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112ea65740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53696/0 (socket says 172.21.15.119:53696) 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 -- 172.21.15.119:0/2796025807 learned_addr learned my addr 172.21.15.119:0/2796025807 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf296b9700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e9e0620 0x56112e90e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 -- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e93c7a0 msgr2=0x56112ea68ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e93c7a0 0x56112ea68ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 -- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e9e0620 msgr2=0x56112e90e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56112e9e0620 0x56112e90e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 -- 172.21.15.119:0/2796025807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56112e83bbb0 con 0x56112e93be80 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf286b7700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56112e93c7a0 0x56112ea68ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf28eb8700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112ea65740 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009e80 tx=0x7fbf20003020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf266b3700 1 -- 172.21.15.119:0/2796025807 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf2000f040 con 0x56112e93be80 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf266b3700 1 -- 172.21.15.119:0/2796025807 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf20021070 con 0x56112e93be80 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf20009490 con 0x56112e93be80 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf266b3700 1 -- 172.21.15.119:0/2796025807 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf20013a90 con 0x56112e93be80 2024-09-13T16:34:17.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.737+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56112e862ca0 con 0x56112e93be80 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf266b3700 1 -- 172.21.15.119:0/2796025807 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbf20013bf0 con 0x56112e93be80 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf266b3700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf08065090 0x7fbf08067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf266b3700 1 -- 172.21.15.119:0/2796025807 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fbf20023070 con 0x56112e93be80 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf286b7700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf08065090 0x7fbf08067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56112ea69fa0 0x56112ea6a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [read 0~4194304] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e113) v8 -- 0x56112ea6a8c0 con 0x56112ea69fa0 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf296b9700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56112ea69fa0 0x56112ea6a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf286b7700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf08065090 0x7fbf08067550 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbf10000c00 tx=0x7fbf1000f000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf296b9700 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56112ea69fa0 0x56112ea6a380 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf296b9700 1 -- 172.21.15.119:0/2796025807 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv3 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x7fbf18002ac0 con 0x56112ea69fa0 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56112ea69fa0 msgr2=0x56112ea6a380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56112ea69fa0 0x56112ea6a380 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf08065090 msgr2=0x7fbf08067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbf08065090 0x7fbf08067550 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fbf10000c00 tx=0x7fbf1000f000 comp rx=0 tx=0).stop 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 msgr2=0x56112ea65740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 --2- 172.21.15.119:0/2796025807 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56112e93be80 0x56112ea65740 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7fbf20009e80 tx=0x7fbf20003020 comp rx=0 tx=0).stop 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf296b9700 1 -- 172.21.15.119:0/2796025807 reap_dead start 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 shutdown_connections 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 >> 172.21.15.119:0/2796025807 conn(0x56112e90d110 msgr2=0x56112e9c6b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 shutdown_connections 2024-09-13T16:34:17.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.741+0000 7fbf2b383140 1 -- 172.21.15.119:0/2796025807 wait complete. 2024-09-13T16:34:17.749 INFO:teuthology.orchestra.run.smithi119.stdout:����� 2024-09-13T16:34:17.750 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:34:17.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.909+0000 7f7854e92700 1 Processor -- start 2024-09-13T16:34:17.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.909+0000 7f7854e92700 1 -- start start 2024-09-13T16:34:17.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.909+0000 7f7854e92700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f78480eb420 0x7f784804ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7848007260 0x7f784804f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784801cfa0 con 0x7f7848033ac0 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784801c670 con 0x7f78480eb420 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784801c950 con 0x7f7848007260 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f78480eb420 0x7f784804ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f78480eb420 0x7f784804ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53710/0 (socket says 172.21.15.119:53710) 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 -- 172.21.15.119:0/2840661302 learned_addr learned my addr 172.21.15.119:0/2840661302 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784ffff700 1 --2- 172.21.15.119:0/2840661302 >> v2:172.21.15.145:3300/0 conn(0x7f7848007260 0x7f784804f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 -- 172.21.15.119:0/2840661302 >> v2:172.21.15.145:3300/0 conn(0x7f7848007260 msgr2=0x7f784804f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 --2- 172.21.15.119:0/2840661302 >> v2:172.21.15.145:3300/0 conn(0x7f7848007260 0x7f784804f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 -- 172.21.15.119:0/2840661302 >> v1:172.21.15.145:6789/0 conn(0x7f7848033ac0 legacy=0x7f7848009ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 -- 172.21.15.119:0/2840661302 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784801cc30 con 0x7f78480eb420 2024-09-13T16:34:17.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784ffff700 1 --2- 172.21.15.119:0/2840661302 >> v2:172.21.15.145:3300/0 conn(0x7f7848007260 0x7f784804f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:17.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784effd700 1 --2- 172.21.15.119:0/2840661302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480eb420 0x7f784804ec10 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f784800db10 tx=0x7f7844009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=839ccbf4c7b364b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784e7fc700 1 -- 172.21.15.119:0/2840661302 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784400b8a0 con 0x7f78480eb420 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784e7fc700 1 -- 172.21.15.119:0/2840661302 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784400ba00 con 0x7f78480eb420 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f784e7fc700 1 -- 172.21.15.119:0/2840661302 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784400bd90 con 0x7f78480eb420 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- 172.21.15.119:0/2840661302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480eb420 msgr2=0x7f784804ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 --2- 172.21.15.119:0/2840661302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480eb420 0x7f784804ec10 secure :-1 s=READY pgs=1621 cs=0 l=1 rev1=1 crypto rx=0x7f784800db10 tx=0x7f7844009bf0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- 172.21.15.119:0/2840661302 shutdown_connections 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 --2- 172.21.15.119:0/2840661302 >> v2:172.21.15.145:3300/0 conn(0x7f7848007260 0x7f784804f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 --2- 172.21.15.119:0/2840661302 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f78480eb420 0x7f784804ec10 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- 172.21.15.119:0/2840661302 >> 172.21.15.119:0/2840661302 conn(0x7f7848036140 msgr2=0x7f7848036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- 172.21.15.119:0/2840661302 shutdown_connections 2024-09-13T16:34:17.918 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.913+0000 7f7854e92700 1 -- 172.21.15.119:0/2840661302 wait complete. 2024-09-13T16:34:17.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 Processor -- start 2024-09-13T16:34:17.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 -- start start 2024-09-13T16:34:17.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7848007260 0x7f78480da440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7848033ac0 0x7f7848170850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f784817fd90 0x7f7848180240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7848180780 con 0x7f7848033ac0 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784804cbd0 con 0x7f784817fd90 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f7854e92700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7848049b70 con 0x7f7848007260 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7848033ac0 0x7f7848170850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7848033ac0 0x7f7848170850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53726/0 (socket says 172.21.15.119:53726) 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 -- 172.21.15.119:0/3633314736 learned_addr learned my addr 172.21.15.119:0/3633314736 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784ffff700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f784817fd90 0x7f7848180240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784f7fe700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7848007260 0x7f78480da440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 -- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7848007260 msgr2=0x7f78480da440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.924 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7848007260 0x7f78480da440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 -- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f784817fd90 msgr2=0x7f7848180240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f784817fd90 0x7f7848180240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784effd700 1 -- 172.21.15.119:0/3633314736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7844009570 con 0x7f7848033ac0 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.917+0000 7f784ffff700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f784817fd90 0x7f7848180240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f784effd700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7848033ac0 0x7f7848170850 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7844000c00 tx=0x7f7844004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f783bfff700 1 -- 172.21.15.119:0/3633314736 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844003ad0 con 0x7f7848033ac0 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f783bfff700 1 -- 172.21.15.119:0/3633314736 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784401e070 con 0x7f7848033ac0 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784804ce00 con 0x7f7848033ac0 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f783bfff700 1 -- 172.21.15.119:0/3633314736 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7844003e50 con 0x7f7848033ac0 2024-09-13T16:34:17.925 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784804d3c0 con 0x7f7848033ac0 2024-09-13T16:34:17.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7848009ba0 con 0x7f7848033ac0 2024-09-13T16:34:17.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f783bfff700 1 -- 172.21.15.119:0/3633314736 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7844011420 con 0x7f7848033ac0 2024-09-13T16:34:17.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f783bfff700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7834066730 0x7f7834068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:17.926 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f783bfff700 1 -- 172.21.15.119:0/3633314736 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7844020070 con 0x7f7848033ac0 2024-09-13T16:34:17.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.921+0000 7f784f7fe700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7834066730 0x7f7834068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:17.927 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.925+0000 7f784f7fe700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7834066730 0x7f7834068bf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f783c007fd0 tx=0x7f783c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:17.933 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:17.929+0000 7f783bfff700 1 -- 172.21.15.119:0/3633314736 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7844052dc0 con 0x7f7848033ac0 2024-09-13T16:34:18.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.141+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f784800ba70 con 0x7f7848033ac0 2024-09-13T16:34:18.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.141+0000 7f783bfff700 1 -- 172.21.15.119:0/3633314736 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v76) v1 ==== 53+0+73 (secure 0 0 0) 0x7f7844015070 con 0x7f7848033ac0 2024-09-13T16:34:18.144 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:34:18.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7834066730 msgr2=0x7f7834068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:18.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7834066730 0x7f7834068bf0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f783c007fd0 tx=0x7f783c009040 comp rx=0 tx=0).stop 2024-09-13T16:34:18.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7848033ac0 msgr2=0x7f7848170850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:18.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7848033ac0 0x7f7848170850 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7f7844000c00 tx=0x7f7844004770 comp rx=0 tx=0).stop 2024-09-13T16:34:18.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 shutdown_connections 2024-09-13T16:34:18.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7834066730 0x7f7834068bf0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f784817fd90 0x7f7848180240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7848033ac0 0x7f7848170850 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 --2- 172.21.15.119:0/3633314736 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7848007260 0x7f78480da440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 >> 172.21.15.119:0/3633314736 conn(0x7f7848036140 msgr2=0x7f7848035500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 shutdown_connections 2024-09-13T16:34:18.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:18.145+0000 7f7854e92700 1 -- 172.21.15.119:0/3633314736 wait complete. 2024-09-13T16:34:18.159 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:34:18.160 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:34:18.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.160 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:34:18.167 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:34:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:34:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:34:18.167 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:34:18.168 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 42083sec preferred_lft 42083sec 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.169 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:34:18.170 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:34:18.171 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.171 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:34:18.185 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:34:18.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.187 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:34:18.196 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:34:18.196 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:34:18.196 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:34:18.196 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:34:18.196 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:34:18.196 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:34:18.196 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:34:18.197 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:34:18.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.197 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:34:18.197 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:34:18.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:34:18.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:34:18.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:34:18.197 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:34:18.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.276 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:34:18.282 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:34:18.282 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:34:18.283 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:34:18.283 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.283 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:34:18.295 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:34:18.295 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:34:18.295 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:18.346 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:34:18.346 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:34:18.346 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:18.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:34:18.367 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:34:18.368 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:34:18.379 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:34:18.380 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:34:18.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.381 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:34:18.431 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:34:18.431 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:34:18.434 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:34:18.451 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:34:18.451 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:34:18.508 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:34:18.508 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:34:18.509 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:18.510 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:34:18.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:34:18.521+0000 7f1ba98e8640 -1 init, newargv = 0x55de9eb72640 newargc=15 2024-09-13T16:34:18.525 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[51312]: starting ceph client 2024-09-13T16:34:18.533 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[51312]: starting fuse 2024-09-13T16:34:19.617 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:34:19.634 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:34:19.634 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:34:19.690 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:34:19.691 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:34:19.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:19.692 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:34:19.699 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:34:19.699 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:34:19.699 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:34:19.700 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:34:19.701 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:34:19.782 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.51312.asok 2024-09-13T16:34:19.965 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51312.asok 2024-09-13T16:34:19.965 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:19.965 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51312.asok status 2024-09-13T16:34:20.110 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:34:20.110 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:34:20.110 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:34:20.110 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:34:20.111 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:34:20.111 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:34:20.111 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:34:20.111 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "51312", 2024-09-13T16:34:20.111 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:34:20.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:20.112 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:34:20.112 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:34:20.112 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9087, 2024-09-13T16:34:20.112 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:34:20.113 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:34:20.113 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:34:20.113 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 9087 2024-09-13T16:34:20.113 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:20.113 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3935968437 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3935968437 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.9087 192.168.0.1:0/3935968437", 2024-09-13T16:34:20.114 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/3935968437", 2024-09-13T16:34:20.115 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:34:20.115 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 76, 2024-09-13T16:34:20.115 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 113, 2024-09-13T16:34:20.115 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:34:20.115 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:34:20.115 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:34:20.115 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:34:20.120 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:20.121 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:34:20.121 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:20.126 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:34:20.127 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:20.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:20.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:20.139 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:20.139 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:20.146 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:20.147 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file2_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-09-13T16:34:20.232 INFO:teuthology.orchestra.run.smithi119.stderr:6+0 records in 2024-09-13T16:34:20.232 INFO:teuthology.orchestra.run.smithi119.stderr:6+0 records out 2024-09-13T16:34:20.232 INFO:teuthology.orchestra.run.smithi119.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0679687 s, 92.6 MB/s 2024-09-13T16:34:20.233 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:20.233 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:20.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:20.238 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=file3_sixmegs bs=1M conv=fdatasync count=6 seek=0) 2024-09-13T16:34:20.317 INFO:teuthology.orchestra.run.smithi119.stderr:6+0 records in 2024-09-13T16:34:20.317 INFO:teuthology.orchestra.run.smithi119.stderr:6+0 records out 2024-09-13T16:34:20.317 INFO:teuthology.orchestra.run.smithi119.stderr:6291456 bytes (6.3 MB, 6.0 MiB) copied, 0.0689558 s, 91.2 MB/s 2024-09-13T16:34:20.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:20.318 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-09-13T16:34:20.332 INFO:teuthology.orchestra.run.smithi119.stdout:./ 2024-09-13T16:34:20.332 INFO:teuthology.orchestra.run.smithi119.stdout:./file2_sixmegs 2024-09-13T16:34:20.332 INFO:teuthology.orchestra.run.smithi119.stdout:./file3_sixmegs 2024-09-13T16:34:20.332 INFO:teuthology.orchestra.run.smithi119.stdout:./file1_sixmegs 2024-09-13T16:34:20.333 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:34:20.333 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:34:20.333 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:34:20.333 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:20.333 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-09-13T16:34:20.333 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:34:20.398 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:34:20.589 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:34:20.589 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:34:20.589 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:34:20.589 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:20.589 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file2_sixmegs").st_ino) 2024-09-13T16:34:20.589 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:34:20.659 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627777 2024-09-13T16:34:20.849 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:34:20.849 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:34:20.849 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:34:20.849 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:20.849 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file3_sixmegs").st_ino) 2024-09-13T16:34:20.849 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:34:20.956 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627778 2024-09-13T16:34:21.147 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:34:21.148 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:34:21.148 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:34:21.148 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:34:21.148 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./file1_sixmegs").st_ino) 2024-09-13T16:34:21.148 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:34:21.213 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627776 2024-09-13T16:34:21.400 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:21.401 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:21.409 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:34:21.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 Processor -- start 2024-09-13T16:34:21.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 -- start start 2024-09-13T16:34:21.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c15ec20 0x7f522c070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c071210 con 0x7f522c165180 2024-09-13T16:34:21.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c0714f0 con 0x7f522c15e840 2024-09-13T16:34:21.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c0717d0 con 0x7f522c15ec20 2024-09-13T16:34:21.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f523265a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f522c165180 0x7f522c0ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58404/0 (socket says 172.21.15.119:58404) 2024-09-13T16:34:21.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f523265a700 1 -- 172.21.15.119:0/1661519730 learned_addr learned my addr 172.21.15.119:0/1661519730 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:21.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5231658700 1 --2- 172.21.15.119:0/1661519730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c15ec20 0x7f522c070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5230e57700 1 -- 172.21.15.119:0/1661519730 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 180815777 0 0) 0x7f522c071210 con 0x7f522c165180 2024-09-13T16:34:21.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5230e57700 1 -- 172.21.15.119:0/1661519730 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52180035d0 con 0x7f522c165180 2024-09-13T16:34:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5230e57700 1 -- 172.21.15.119:0/1661519730 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482535158 0 0) 0x7f522c0714f0 con 0x7f522c15e840 2024-09-13T16:34:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5230e57700 1 -- 172.21.15.119:0/1661519730 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f522c071210 con 0x7f522c15e840 2024-09-13T16:34:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5231658700 1 -- 172.21.15.119:0/1661519730 >> v1:172.21.15.145:6789/0 conn(0x7f522c15e840 legacy=0x7f522c164a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5231658700 1 -- 172.21.15.119:0/1661519730 >> v1:172.21.15.119:6789/0 conn(0x7f522c165180 legacy=0x7f522c0ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5231658700 1 -- 172.21.15.119:0/1661519730 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c071ab0 con 0x7f522c15ec20 2024-09-13T16:34:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5231658700 1 --2- 172.21.15.119:0/1661519730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c15ec20 0x7f522c070b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f521c009880 tx=0x7f521c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4986a1dda5c0748b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5230e57700 1 -- 172.21.15.119:0/1661519730 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c011830 con 0x7f522c15ec20 2024-09-13T16:34:21.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5230e57700 1 -- 172.21.15.119:0/1661519730 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521c011990 con 0x7f522c15ec20 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f5230e57700 1 -- 172.21.15.119:0/1661519730 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521c011d00 con 0x7f522c15ec20 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 -- 172.21.15.119:0/1661519730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c15ec20 msgr2=0x7f522c070b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.525+0000 7f52340bd700 1 --2- 172.21.15.119:0/1661519730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c15ec20 0x7f522c070b80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f521c009880 tx=0x7f521c009b90 comp rx=0 tx=0).stop 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- 172.21.15.119:0/1661519730 shutdown_connections 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 --2- 172.21.15.119:0/1661519730 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c15ec20 0x7f522c070b80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- 172.21.15.119:0/1661519730 >> 172.21.15.119:0/1661519730 conn(0x7f522c06a350 msgr2=0x7f522c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- 172.21.15.119:0/1661519730 shutdown_connections 2024-09-13T16:34:21.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- 172.21.15.119:0/1661519730 wait complete. 2024-09-13T16:34:21.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 Processor -- start 2024-09-13T16:34:21.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- start start 2024-09-13T16:34:21.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f522c15e840 0x7f522c155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c165180 0x7f522c160db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c156770 0x7f522c162c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c15f840 con 0x7f522c15e840 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f522c15fb20 con 0x7f522c156770 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f52340bd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f522c15fe00 con 0x7f522c165180 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f522c15e840 0x7f522c155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f522c15e840 0x7f522c155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53742/0 (socket says 172.21.15.119:53742) 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 -- 172.21.15.119:0/781687040 learned_addr learned my addr 172.21.15.119:0/781687040 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231658700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c165180 0x7f522c160db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f523265a700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c156770 0x7f522c162c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 -- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c165180 msgr2=0x7f522c160db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c165180 0x7f522c160db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 -- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c156770 msgr2=0x7f522c162c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c156770 0x7f522c162c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f5231e59700 1 -- 172.21.15.119:0/781687040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c0094b0 con 0x7f522c15e840 2024-09-13T16:34:21.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.529+0000 7f523265a700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c156770 0x7f522c162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:21.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f5231e59700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f522c15e840 0x7f522c155af0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f522800c520 tx=0x7f522800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:21.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52227fc700 1 -- 172.21.15.119:0/781687040 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52280052d0 con 0x7f522c15e840 2024-09-13T16:34:21.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52227fc700 1 -- 172.21.15.119:0/781687040 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5228005450 con 0x7f522c15e840 2024-09-13T16:34:21.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c1600e0 con 0x7f522c15e840 2024-09-13T16:34:21.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52227fc700 1 -- 172.21.15.119:0/781687040 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52280038e0 con 0x7f522c15e840 2024-09-13T16:34:21.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c15f4a0 con 0x7f522c15e840 2024-09-13T16:34:21.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522c15b5e0 con 0x7f522c15e840 2024-09-13T16:34:21.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52227fc700 1 -- 172.21.15.119:0/781687040 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5228003a40 con 0x7f522c15e840 2024-09-13T16:34:21.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52227fc700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5210066730 0x7f5210068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f52227fc700 1 -- 172.21.15.119:0/781687040 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5228083720 con 0x7f522c15e840 2024-09-13T16:34:21.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.533+0000 7f5231658700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5210066730 0x7f5210068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.537+0000 7f5231658700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5210066730 0x7f5210068bf0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f521c0090b0 tx=0x7f521c010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:21.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.545+0000 7f52227fc700 1 -- 172.21.15.119:0/781687040 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5228050930 con 0x7f522c15e840 2024-09-13T16:34:21.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.717+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f522c067760 con 0x7f522c15e840 2024-09-13T16:34:21.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.717+0000 7f52227fc700 1 -- 172.21.15.119:0/781687040 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16826 (secure 0 0 0) 0x7f5228053f50 con 0x7f522c15e840 2024-09-13T16:34:21.721 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:21.722 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":113,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:34:10.103391+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:34:21.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5210066730 msgr2=0x7f5210068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5210066730 0x7f5210068bf0 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f521c0090b0 tx=0x7f521c010000 comp rx=0 tx=0).stop 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f522c15e840 msgr2=0x7f522c155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f522c15e840 0x7f522c155af0 secure :-1 s=READY pgs=1627 cs=0 l=1 rev1=1 crypto rx=0x7f522800c520 tx=0x7f522800c830 comp rx=0 tx=0).stop 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 shutdown_connections 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5210066730 0x7f5210068bf0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f522c156770 0x7f522c162c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f522c165180 0x7f522c160db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 --2- 172.21.15.119:0/781687040 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f522c15e840 0x7f522c155af0 unknown :-1 s=CLOSED pgs=1627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 >> 172.21.15.119:0/781687040 conn(0x7f522c06a350 msgr2=0x7f522c1532f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 shutdown_connections 2024-09-13T16:34:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.721+0000 7f52340bd700 1 -- 172.21.15.119:0/781687040 wait complete. 2024-09-13T16:34:21.736 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:33:57.346379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:33:58.017381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:34:21.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:34:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3f064700 1 Processor -- start 2024-09-13T16:34:21.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3f064700 1 -- start start 2024-09-13T16:34:21.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3f064700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5a3815cb90 0x7f5a380b1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3f064700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3806b990 0x7f5a3806bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3f064700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a380b4390 con 0x7f5a3806c310 2024-09-13T16:34:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3f064700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a380b1be0 con 0x7f5a3815cb90 2024-09-13T16:34:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3f064700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a3806c780 con 0x7f5a3806b990 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a37fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3806b990 0x7f5a3806bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3ce00700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5a3815cb90 0x7f5a380b1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3d601700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5a3806c310 0x7f5a38160780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:55446/0 (socket says 172.21.15.119:55446) 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.853+0000 7f5a3d601700 1 -- 172.21.15.119:0/2287282161 learned_addr learned my addr 172.21.15.119:0/2287282161 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a377fe700 1 -- 172.21.15.119:0/2287282161 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3355344054 0 0) 0x7f5a380b4390 con 0x7f5a3806c310 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a377fe700 1 -- 172.21.15.119:0/2287282161 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a200035d0 con 0x7f5a3806c310 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a37fff700 1 -- 172.21.15.119:0/2287282161 >> v2:172.21.15.145:3300/0 conn(0x7f5a3815cb90 msgr2=0x7f5a380b1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a37fff700 1 --2- 172.21.15.119:0/2287282161 >> v2:172.21.15.145:3300/0 conn(0x7f5a3815cb90 0x7f5a380b1d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a37fff700 1 -- 172.21.15.119:0/2287282161 >> v1:172.21.15.145:6789/0 conn(0x7f5a3806c310 legacy=0x7f5a38160780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a37fff700 1 -- 172.21.15.119:0/2287282161 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a380b1060 con 0x7f5a3806b990 2024-09-13T16:34:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a37fff700 1 --2- 172.21.15.119:0/2287282161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3806b990 0x7f5a3806bdd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009850 tx=0x7f5a28009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9aaf1e2e5348461 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a377fe700 1 -- 172.21.15.119:0/2287282161 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28011830 con 0x7f5a3806b990 2024-09-13T16:34:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a377fe700 1 -- 172.21.15.119:0/2287282161 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a28011990 con 0x7f5a3806b990 2024-09-13T16:34:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a377fe700 1 -- 172.21.15.119:0/2287282161 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28011d00 con 0x7f5a3806b990 2024-09-13T16:34:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 -- 172.21.15.119:0/2287282161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3806b990 msgr2=0x7f5a3806bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2287282161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3806b990 0x7f5a3806bdd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f5a28009850 tx=0x7f5a28009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 -- 172.21.15.119:0/2287282161 shutdown_connections 2024-09-13T16:34:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2287282161 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3806b990 0x7f5a3806bdd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2287282161 >> v2:172.21.15.145:3300/0 conn(0x7f5a3815cb90 0x7f5a380b1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 -- 172.21.15.119:0/2287282161 >> 172.21.15.119:0/2287282161 conn(0x7f5a38099a50 msgr2=0x7f5a38099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 -- 172.21.15.119:0/2287282161 shutdown_connections 2024-09-13T16:34:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 -- 172.21.15.119:0/2287282161 wait complete. 2024-09-13T16:34:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.857+0000 7f5a3f064700 1 Processor -- start 2024-09-13T16:34:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 -- start start 2024-09-13T16:34:21.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a3806b990 0x7f5a38146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a3806c310 0x7f5a38152a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3815cb90 0x7f5a381464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a38146a00 con 0x7f5a3806c310 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a381530f0 con 0x7f5a3806b990 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a380cc330 con 0x7f5a3815cb90 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a3806c310 0x7f5a38152a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a3806c310 0x7f5a38152a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53754/0 (socket says 172.21.15.119:53754) 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 -- 172.21.15.119:0/2032681103 learned_addr learned my addr 172.21.15.119:0/2032681103 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3d601700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3815cb90 0x7f5a381464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3ce00700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a3806b990 0x7f5a38146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 -- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3815cb90 msgr2=0x7f5a381464c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3815cb90 0x7f5a381464c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 -- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a3806b990 msgr2=0x7f5a38146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a3806b990 0x7f5a38146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 -- 172.21.15.119:0/2032681103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a280094b0 con 0x7f5a3806c310 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a37fff700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a3806c310 0x7f5a38152a00 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5a280090b0 tx=0x7f5a28004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a357fa700 1 -- 172.21.15.119:0/2032681103 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28004590 con 0x7f5a3806c310 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a357fa700 1 -- 172.21.15.119:0/2032681103 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a2801f070 con 0x7f5a3806c310 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a357fa700 1 -- 172.21.15.119:0/2032681103 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a28003900 con 0x7f5a3806c310 2024-09-13T16:34:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a380cc610 con 0x7f5a3806c310 2024-09-13T16:34:21.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a380ccbd0 con 0x7f5a3806c310 2024-09-13T16:34:21.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.861+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a3815cf70 con 0x7f5a3806c310 2024-09-13T16:34:21.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.865+0000 7f5a357fa700 1 -- 172.21.15.119:0/2032681103 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5a28003a60 con 0x7f5a3806c310 2024-09-13T16:34:21.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.865+0000 7f5a357fa700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5a24066730 0x7f5a24068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.865+0000 7f5a357fa700 1 -- 172.21.15.119:0/2032681103 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5a28021070 con 0x7f5a3806c310 2024-09-13T16:34:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.865+0000 7f5a3ce00700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5a24066730 0x7f5a24068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:21.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.869+0000 7f5a3ce00700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5a24066730 0x7f5a24068bf0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009b50 tx=0x7f5a2c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:21.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:21.873+0000 7f5a357fa700 1 -- 172.21.15.119:0/2032681103 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5a28052e00 con 0x7f5a3806c310 2024-09-13T16:34:22.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5a38153230 con 0x7f5a3806c310 2024-09-13T16:34:22.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a357fa700 1 -- 172.21.15.119:0/2032681103 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5a28015020 con 0x7f5a3806c310 2024-09-13T16:34:22.044 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:34:22.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5a24066730 msgr2=0x7f5a24068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5a24066730 0x7f5a24068bf0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c009b50 tx=0x7f5a2c009240 comp rx=0 tx=0).stop 2024-09-13T16:34:22.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a3806c310 msgr2=0x7f5a38152a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a3806c310 0x7f5a38152a00 secure :-1 s=READY pgs=1628 cs=0 l=1 rev1=1 crypto rx=0x7f5a280090b0 tx=0x7f5a28004770 comp rx=0 tx=0).stop 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 shutdown_connections 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5a24066730 0x7f5a24068bf0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5a3815cb90 0x7f5a381464c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5a3806c310 0x7f5a38152a00 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 --2- 172.21.15.119:0/2032681103 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5a3806b990 0x7f5a38146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.041+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 >> 172.21.15.119:0/2032681103 conn(0x7f5a38099a50 msgr2=0x7f5a3806e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.045+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 shutdown_connections 2024-09-13T16:34:22.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.045+0000 7f5a3f064700 1 -- 172.21.15.119:0/2032681103 wait complete. 2024-09-13T16:34:22.060 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:34:22.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff6149e7700 1 Processor -- start 2024-09-13T16:34:22.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff6149e7700 1 -- start start 2024-09-13T16:34:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff6149e7700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff61015a970 0x7ff61015d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff6149e7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100d0280 0x7ff61006b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff6149e7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff61015ad50 con 0x7ff6100b3200 2024-09-13T16:34:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff6149e7700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6100721b0 con 0x7ff61015a970 2024-09-13T16:34:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff6149e7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6100c9d60 con 0x7ff6100d0280 2024-09-13T16:34:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff60dd9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff6100b3200 0x7ff6100cfb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:58412/0 (socket says 172.21.15.119:58412) 2024-09-13T16:34:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff60dd9b700 1 -- 172.21.15.119:0/1647358646 learned_addr learned my addr 172.21.15.119:0/1647358646 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff60e59c700 1 --2- 172.21.15.119:0/1647358646 >> v2:172.21.15.145:3300/0 conn(0x7ff61015a970 0x7ff61015d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff60ed9d700 1 --2- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100d0280 0x7ff61006b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.173+0000 7ff60d59a700 1 -- 172.21.15.119:0/1647358646 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528611076 0 0) 0x7ff61015ad50 con 0x7ff6100b3200 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60d59a700 1 -- 172.21.15.119:0/1647358646 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff5f80035d0 con 0x7ff6100b3200 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60e59c700 1 -- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100d0280 msgr2=0x7ff61006b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60e59c700 1 --2- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100d0280 0x7ff61006b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60e59c700 1 -- 172.21.15.119:0/1647358646 >> v1:172.21.15.119:6789/0 conn(0x7ff6100b3200 legacy=0x7ff6100cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60e59c700 1 -- 172.21.15.119:0/1647358646 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6100ca040 con 0x7ff61015a970 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60ed9d700 1 --2- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100d0280 0x7ff61006b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60e59c700 1 --2- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61015a970 0x7ff61015d2f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff600009850 tx=0x7ff600009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2e56a5a1bbf90b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60d59a700 1 -- 172.21.15.119:0/1647358646 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600011830 con 0x7ff61015a970 2024-09-13T16:34:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60d59a700 1 -- 172.21.15.119:0/1647358646 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff600011990 con 0x7ff61015a970 2024-09-13T16:34:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff60d59a700 1 -- 172.21.15.119:0/1647358646 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600011d40 con 0x7ff61015a970 2024-09-13T16:34:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 -- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61015a970 msgr2=0x7ff61015d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 --2- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61015a970 0x7ff61015d2f0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7ff600009850 tx=0x7ff600009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 -- 172.21.15.119:0/1647358646 shutdown_connections 2024-09-13T16:34:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 --2- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100d0280 0x7ff61006b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 --2- 172.21.15.119:0/1647358646 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff61015a970 0x7ff61015d2f0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 -- 172.21.15.119:0/1647358646 >> 172.21.15.119:0/1647358646 conn(0x7ff6100b26e0 msgr2=0x7ff6100b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 -- 172.21.15.119:0/1647358646 shutdown_connections 2024-09-13T16:34:22.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 -- 172.21.15.119:0/1647358646 wait complete. 2024-09-13T16:34:22.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.177+0000 7ff6149e7700 1 Processor -- start 2024-09-13T16:34:22.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 -- start start 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100b3200 0x7ff61013f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6100d0280 0x7ff6100cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61015a970 0x7ff61014f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6100cdc70 con 0x7ff61015a970 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6100cd340 con 0x7ff6100d0280 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6100cd620 con 0x7ff6100b3200 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61015a970 0x7ff61014f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61015a970 0x7ff61014f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:53766/0 (socket says 172.21.15.119:53766) 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 -- 172.21.15.119:0/2035753838 learned_addr learned my addr 172.21.15.119:0/2035753838 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60e59c700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100b3200 0x7ff61013f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60dd9b700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6100d0280 0x7ff6100cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 -- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100b3200 msgr2=0x7ff61013f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100b3200 0x7ff61013f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 -- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6100d0280 msgr2=0x7ff6100cb270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6100d0280 0x7ff6100cb270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 -- 172.21.15.119:0/2035753838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff608009a70 con 0x7ff61015a970 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff60ed9d700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61015a970 0x7ff61014f000 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ff60800efc0 tx=0x7ff60800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff5feffd700 1 -- 172.21.15.119:0/2035753838 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60800ca90 con 0x7ff61015a970 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff5feffd700 1 -- 172.21.15.119:0/2035753838 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60800f040 con 0x7ff61015a970 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6000094b0 con 0x7ff61015a970 2024-09-13T16:34:22.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff5feffd700 1 -- 172.21.15.119:0/2035753838 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60800ce10 con 0x7ff61015a970 2024-09-13T16:34:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.181+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6100ccfa0 con 0x7ff61015a970 2024-09-13T16:34:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.185+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff61015d2f0 con 0x7ff61015a970 2024-09-13T16:34:22.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.185+0000 7ff5feffd700 1 -- 172.21.15.119:0/2035753838 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff608014480 con 0x7ff61015a970 2024-09-13T16:34:22.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.185+0000 7ff5feffd700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5f0066730 0x7ff5f0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.185+0000 7ff5feffd700 1 -- 172.21.15.119:0/2035753838 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff608084990 con 0x7ff61015a970 2024-09-13T16:34:22.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.185+0000 7ff60e59c700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5f0066730 0x7ff5f0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.185+0000 7ff60e59c700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5f0066730 0x7ff5f0068bf0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff6000090b0 tx=0x7ff60001c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.193+0000 7ff5feffd700 1 -- 172.21.15.119:0/2035753838 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff608051ba0 con 0x7ff61015a970 2024-09-13T16:34:22.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.361+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff6100cddb0 con 0x7ff61015a970 2024-09-13T16:34:22.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff5feffd700 1 -- 172.21.15.119:0/2035753838 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff608009d70 con 0x7ff61015a970 2024-09-13T16:34:22.368 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5f0066730 msgr2=0x7ff5f0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5f0066730 0x7ff5f0068bf0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7ff6000090b0 tx=0x7ff60001c040 comp rx=0 tx=0).stop 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61015a970 msgr2=0x7ff61014f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61015a970 0x7ff61014f000 secure :-1 s=READY pgs=1630 cs=0 l=1 rev1=1 crypto rx=0x7ff60800efc0 tx=0x7ff60800c3a0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 shutdown_connections 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff5f0066730 0x7ff5f0068bf0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff61015a970 0x7ff61014f000 unknown :-1 s=CLOSED pgs=1630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff6100d0280 0x7ff6100cb270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 --2- 172.21.15.119:0/2035753838 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff6100b3200 0x7ff61013f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 >> 172.21.15.119:0/2035753838 conn(0x7ff6100b26e0 msgr2=0x7ff61006c070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:22.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 shutdown_connections 2024-09-13T16:34:22.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.365+0000 7ff6149e7700 1 -- 172.21.15.119:0/2035753838 wait complete. 2024-09-13T16:34:22.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:34:22.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 Processor -- start 2024-09-13T16:34:22.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 -- start start 2024-09-13T16:34:22.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7effc4144180 0x7effc406c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7effc40cc010 0x7effc40cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40ce000 0x7effc415e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc40b1c50 con 0x7effc40cc010 2024-09-13T16:34:22.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effc406cb40 con 0x7effc4144180 2024-09-13T16:34:22.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc9d3d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc406ce20 con 0x7effc40ce000 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7effc40cc010 0x7effc40cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7effc40cc010 0x7effc40cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55458/0 (socket says 172.21.15.119:55458) 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 -- 172.21.15.119:0/3899259408 learned_addr learned my addr 172.21.15.119:0/3899259408 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc953c700 1 --2- 172.21.15.119:0/3899259408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40ce000 0x7effc415e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc8d3b700 1 --2- 172.21.15.119:0/3899259408 >> v2:172.21.15.145:3300/0 conn(0x7effc4144180 0x7effc406c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 -- 172.21.15.119:0/3899259408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40ce000 msgr2=0x7effc415e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 --2- 172.21.15.119:0/3899259408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40ce000 0x7effc415e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 -- 172.21.15.119:0/3899259408 >> v2:172.21.15.145:3300/0 conn(0x7effc4144180 msgr2=0x7effc406c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 --2- 172.21.15.119:0/3899259408 >> v2:172.21.15.145:3300/0 conn(0x7effc4144180 0x7effc406c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 -- 172.21.15.119:0/3899259408 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc406d100 con 0x7effc40cc010 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc3fff700 1 --2- 172.21.15.119:0/3899259408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc40cc010 0x7effc40cdac0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7effc40b1560 tx=0x7effb4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b38eaa2de858948 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc37fe700 1 -- 172.21.15.119:0/3899259408 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb400b8a0 con 0x7effc40cc010 2024-09-13T16:34:22.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc37fe700 1 -- 172.21.15.119:0/3899259408 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb400ba00 con 0x7effc40cc010 2024-09-13T16:34:22.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.501+0000 7effc37fe700 1 -- 172.21.15.119:0/3899259408 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb400bd90 con 0x7effc40cc010 2024-09-13T16:34:22.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- 172.21.15.119:0/3899259408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc40cc010 msgr2=0x7effc40cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 --2- 172.21.15.119:0/3899259408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc40cc010 0x7effc40cdac0 secure :-1 s=READY pgs=1631 cs=0 l=1 rev1=1 crypto rx=0x7effc40b1560 tx=0x7effb4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- 172.21.15.119:0/3899259408 shutdown_connections 2024-09-13T16:34:22.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 --2- 172.21.15.119:0/3899259408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40ce000 0x7effc415e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 --2- 172.21.15.119:0/3899259408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc40cc010 0x7effc40cdac0 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 --2- 172.21.15.119:0/3899259408 >> v2:172.21.15.145:3300/0 conn(0x7effc4144180 0x7effc406c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- 172.21.15.119:0/3899259408 >> 172.21.15.119:0/3899259408 conn(0x7effc40b0980 msgr2=0x7effc40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:22.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- 172.21.15.119:0/3899259408 shutdown_connections 2024-09-13T16:34:22.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- 172.21.15.119:0/3899259408 wait complete. 2024-09-13T16:34:22.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 Processor -- start 2024-09-13T16:34:22.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- start start 2024-09-13T16:34:22.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40cc010 0x7effc4159a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effc40ce000 0x7effc415ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc4144180 0x7effc415d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc415b860 con 0x7effc4144180 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effc415bb40 con 0x7effc40ce000 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc9d3d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effc415be20 con 0x7effc40cc010 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc953c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc4144180 0x7effc415d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc953c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc4144180 0x7effc415d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55466/0 (socket says 172.21.15.119:55466) 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.505+0000 7effc953c700 1 -- 172.21.15.119:0/633140645 learned_addr learned my addr 172.21.15.119:0/633140645 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc3fff700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effc40ce000 0x7effc415ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc8d3b700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40cc010 0x7effc4159a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc953c700 1 -- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40cc010 msgr2=0x7effc4159a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc953c700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40cc010 0x7effc4159a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc953c700 1 -- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effc40ce000 msgr2=0x7effc415ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc953c700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effc40ce000 0x7effc415ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc953c700 1 -- 172.21.15.119:0/633140645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb4009570 con 0x7effc4144180 2024-09-13T16:34:22.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc953c700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc4144180 0x7effc415d020 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7effbc007e40 tx=0x7effbc00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc17fa700 1 -- 172.21.15.119:0/633140645 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc00ed30 con 0x7effc4144180 2024-09-13T16:34:22.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc415c100 con 0x7effc4144180 2024-09-13T16:34:22.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc17fa700 1 -- 172.21.15.119:0/633140645 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effbc014420 con 0x7effc4144180 2024-09-13T16:34:22.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc17fa700 1 -- 172.21.15.119:0/633140645 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effbc0147d0 con 0x7effc4144180 2024-09-13T16:34:22.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effc4139c20 con 0x7effc4144180 2024-09-13T16:34:22.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc415c620 con 0x7effc4144180 2024-09-13T16:34:22.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.509+0000 7effc17fa700 1 -- 172.21.15.119:0/633140645 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7effbc014930 con 0x7effc4144180 2024-09-13T16:34:22.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.513+0000 7effc17fa700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7effa4066730 0x7effa4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.513+0000 7effc17fa700 1 -- 172.21.15.119:0/633140645 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7effbc084950 con 0x7effc4144180 2024-09-13T16:34:22.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.513+0000 7effc8d3b700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7effa4066730 0x7effa4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.513+0000 7effc8d3b700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7effa4066730 0x7effa4068bf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7effc40b1ab0 tx=0x7effac00b660 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.517+0000 7effc17fa700 1 -- 172.21.15.119:0/633140645 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7effbc051b70 con 0x7effc4144180 2024-09-13T16:34:22.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.689+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7effc40cdac0 con 0x7effc4144180 2024-09-13T16:34:22.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.689+0000 7effc17fa700 1 -- 172.21.15.119:0/633140645 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7effbc055190 con 0x7effc4144180 2024-09-13T16:34:22.693 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:34:22.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7effa4066730 msgr2=0x7effa4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7effa4066730 0x7effa4068bf0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7effc40b1ab0 tx=0x7effac00b660 comp rx=0 tx=0).stop 2024-09-13T16:34:22.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc4144180 msgr2=0x7effc415d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc4144180 0x7effc415d020 secure :-1 s=READY pgs=1632 cs=0 l=1 rev1=1 crypto rx=0x7effbc007e40 tx=0x7effbc00e3c0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 shutdown_connections 2024-09-13T16:34:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7effa4066730 0x7effa4068bf0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7effc4144180 0x7effc415d020 unknown :-1 s=CLOSED pgs=1632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7effc40ce000 0x7effc415ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 --2- 172.21.15.119:0/633140645 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7effc40cc010 0x7effc4159a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 >> 172.21.15.119:0/633140645 conn(0x7effc40b0980 msgr2=0x7effc406ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 shutdown_connections 2024-09-13T16:34:22.698 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.693+0000 7effc9d3d700 1 -- 172.21.15.119:0/633140645 wait complete. 2024-09-13T16:34:22.708 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:34:22.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7759e700 1 Processor -- start 2024-09-13T16:34:22.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7759e700 1 -- start start 2024-09-13T16:34:22.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc781594f0 0x7efc7806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7759e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc7806da70 0x7efc780b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc780b2280 con 0x7efc780b3630 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc78070c10 con 0x7efc781594f0 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7759e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc7806dee0 con 0x7efc7806da70 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7659c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc781594f0 0x7efc7806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7659c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc781594f0 0x7efc7806d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55482/0 (socket says 172.21.15.119:55482) 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7659c700 1 -- 172.21.15.119:0/3686029022 learned_addr learned my addr 172.21.15.119:0/3686029022 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc75d9b700 1 --2- 172.21.15.119:0/3686029022 >> v2:172.21.15.145:3300/0 conn(0x7efc7806da70 0x7efc780b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7659c700 1 -- 172.21.15.119:0/3686029022 >> v2:172.21.15.145:3300/0 conn(0x7efc7806da70 msgr2=0x7efc780b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.817+0000 7efc7659c700 1 --2- 172.21.15.119:0/3686029022 >> v2:172.21.15.145:3300/0 conn(0x7efc7806da70 0x7efc780b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7659c700 1 -- 172.21.15.119:0/3686029022 >> v1:172.21.15.145:6789/0 conn(0x7efc780b3630 legacy=0x7efc7815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7659c700 1 -- 172.21.15.119:0/3686029022 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc780cc890 con 0x7efc781594f0 2024-09-13T16:34:22.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7659c700 1 --2- 172.21.15.119:0/3686029022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc7806d530 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7efc7806c330 tx=0x7efc60009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1791378a9f66475c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7559a700 1 -- 172.21.15.119:0/3686029022 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60011830 con 0x7efc781594f0 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7559a700 1 -- 172.21.15.119:0/3686029022 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc60011990 con 0x7efc781594f0 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7559a700 1 -- 172.21.15.119:0/3686029022 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc60011d20 con 0x7efc781594f0 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 -- 172.21.15.119:0/3686029022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 msgr2=0x7efc7806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 --2- 172.21.15.119:0/3686029022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc7806d530 secure :-1 s=READY pgs=1633 cs=0 l=1 rev1=1 crypto rx=0x7efc7806c330 tx=0x7efc60009bf0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 -- 172.21.15.119:0/3686029022 shutdown_connections 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 --2- 172.21.15.119:0/3686029022 >> v2:172.21.15.145:3300/0 conn(0x7efc7806da70 0x7efc780b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 --2- 172.21.15.119:0/3686029022 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc7806d530 unknown :-1 s=CLOSED pgs=1633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 -- 172.21.15.119:0/3686029022 >> 172.21.15.119:0/3686029022 conn(0x7efc7806a350 msgr2=0x7efc7806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 -- 172.21.15.119:0/3686029022 shutdown_connections 2024-09-13T16:34:22.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 -- 172.21.15.119:0/3686029022 wait complete. 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 Processor -- start 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.821+0000 7efc7759e700 1 -- start start 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc7806da70 0x7efc78157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc780b3630 0x7efc781559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc781514f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc78155ef0 con 0x7efc781594f0 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc78158550 con 0x7efc7806da70 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc78158830 con 0x7efc780b3630 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc781514f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc7806da70 0x7efc78157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc781514f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55486/0 (socket says 172.21.15.119:55486) 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 -- 172.21.15.119:0/3648291394 learned_addr learned my addr 172.21.15.119:0/3648291394 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc75d9b700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc780b3630 0x7efc781559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 -- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc780b3630 msgr2=0x7efc781559b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc780b3630 0x7efc781559b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 -- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc7806da70 msgr2=0x7efc78157e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc7806da70 0x7efc78157e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 -- 172.21.15.119:0/3648291394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60009570 con 0x7efc781594f0 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc76d9d700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc781514f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7efc7000cba0 tx=0x7efc7000ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc6effd700 1 -- 172.21.15.119:0/3648291394 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc70003c20 con 0x7efc781594f0 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc6effd700 1 -- 172.21.15.119:0/3648291394 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc70003d80 con 0x7efc781594f0 2024-09-13T16:34:22.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc78158b10 con 0x7efc781594f0 2024-09-13T16:34:22.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc6effd700 1 -- 172.21.15.119:0/3648291394 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc7000ab10 con 0x7efc781594f0 2024-09-13T16:34:22.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc78151a30 con 0x7efc781594f0 2024-09-13T16:34:22.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc7814f4c0 con 0x7efc781594f0 2024-09-13T16:34:22.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc6effd700 1 -- 172.21.15.119:0/3648291394 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efc7000ac70 con 0x7efc781594f0 2024-09-13T16:34:22.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc6effd700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc58066780 0x7efc58068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:22.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.825+0000 7efc6effd700 1 -- 172.21.15.119:0/3648291394 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7efc70084130 con 0x7efc781594f0 2024-09-13T16:34:22.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.829+0000 7efc7659c700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc58066780 0x7efc58068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:22.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.829+0000 7efc7659c700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc58066780 0x7efc58068c40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc60008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:22.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:22.837+0000 7efc6effd700 1 -- 172.21.15.119:0/3648291394 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efc7814f4c0 con 0x7efc781594f0 2024-09-13T16:34:23.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.013+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7efc7814f4c0 con 0x7efc781594f0 2024-09-13T16:34:23.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.013+0000 7efc6effd700 1 -- 172.21.15.119:0/3648291394 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7efc7814f4c0 con 0x7efc781594f0 2024-09-13T16:34:23.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc58066780 msgr2=0x7efc58068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc58066780 0x7efc58068c40 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc60008030 comp rx=0 tx=0).stop 2024-09-13T16:34:23.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 msgr2=0x7efc781514f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc781514f0 secure :-1 s=READY pgs=1634 cs=0 l=1 rev1=1 crypto rx=0x7efc7000cba0 tx=0x7efc7000ceb0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 shutdown_connections 2024-09-13T16:34:23.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efc58066780 0x7efc58068c40 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc781594f0 0x7efc781514f0 unknown :-1 s=CLOSED pgs=1634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc780b3630 0x7efc781559b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 --2- 172.21.15.119:0/3648291394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc7806da70 0x7efc78157e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 >> 172.21.15.119:0/3648291394 conn(0x7efc7806a350 msgr2=0x7efc780aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 shutdown_connections 2024-09-13T16:34:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.017+0000 7efc7759e700 1 -- 172.21.15.119:0/3648291394 wait complete. 2024-09-13T16:34:23.022 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:34:23.033 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:23.033 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:23.038 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:34:23.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd9cf9700 1 Processor -- start 2024-09-13T16:34:23.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd9cf9700 1 -- start start 2024-09-13T16:34:23.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd9cf9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd41594f0 0x7fccd406d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd9cf9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fccd40b3800 0x7fccd406fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd9cf9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccd40b2280 con 0x7fccd406da70 2024-09-13T16:34:23.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd9cf9700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccd406deb0 con 0x7fccd40b3800 2024-09-13T16:34:23.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd9cf9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd40cc890 con 0x7fccd41594f0 2024-09-13T16:34:23.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd94f8700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fccd40b3800 0x7fccd406fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd8cf7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd41594f0 0x7fccd406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd3fff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fccd406da70 0x7fccd40b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:58740/0 (socket says 172.21.15.119:58740) 2024-09-13T16:34:23.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.189+0000 7fccd3fff700 1 -- 172.21.15.119:0/2800433534 learned_addr learned my addr 172.21.15.119:0/2800433534 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:23.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd8cf7700 1 -- 172.21.15.119:0/2800433534 >> v2:172.21.15.145:3300/0 conn(0x7fccd40b3800 msgr2=0x7fccd406fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:23.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd8cf7700 1 --2- 172.21.15.119:0/2800433534 >> v2:172.21.15.145:3300/0 conn(0x7fccd40b3800 0x7fccd406fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd8cf7700 1 -- 172.21.15.119:0/2800433534 >> v1:172.21.15.145:6789/0 conn(0x7fccd406da70 legacy=0x7fccd40b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd8cf7700 1 -- 172.21.15.119:0/2800433534 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccd40b1970 con 0x7fccd41594f0 2024-09-13T16:34:23.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd94f8700 1 --2- 172.21.15.119:0/2800433534 >> v2:172.21.15.145:3300/0 conn(0x7fccd40b3800 0x7fccd406fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:23.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd8cf7700 1 --2- 172.21.15.119:0/2800433534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd41594f0 0x7fccd406d530 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fccd40b1b60 tx=0x7fccc8009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bf2b5e9473f7051 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd37fe700 1 -- 172.21.15.119:0/2800433534 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8004020 con 0x7fccd41594f0 2024-09-13T16:34:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd37fe700 1 -- 172.21.15.119:0/2800433534 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc8004180 con 0x7fccd41594f0 2024-09-13T16:34:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd37fe700 1 -- 172.21.15.119:0/2800433534 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc8004530 con 0x7fccd41594f0 2024-09-13T16:34:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 -- 172.21.15.119:0/2800433534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd41594f0 msgr2=0x7fccd406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/2800433534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd41594f0 0x7fccd406d530 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fccd40b1b60 tx=0x7fccc8009b30 comp rx=0 tx=0).stop 2024-09-13T16:34:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 -- 172.21.15.119:0/2800433534 shutdown_connections 2024-09-13T16:34:23.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/2800433534 >> v2:172.21.15.145:3300/0 conn(0x7fccd40b3800 0x7fccd406fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/2800433534 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd41594f0 0x7fccd406d530 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 -- 172.21.15.119:0/2800433534 >> 172.21.15.119:0/2800433534 conn(0x7fccd406a350 msgr2=0x7fccd406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:23.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 -- 172.21.15.119:0/2800433534 shutdown_connections 2024-09-13T16:34:23.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 -- 172.21.15.119:0/2800433534 wait complete. 2024-09-13T16:34:23.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 Processor -- start 2024-09-13T16:34:23.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.193+0000 7fccd9cf9700 1 -- start start 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd406da70 0x7fccd41585e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccd40b3800 0x7fccd4147040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd41594f0 0x7fccd40ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd4150530 con 0x7fccd40b3800 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccd414cc80 con 0x7fccd41594f0 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccd414cf60 con 0x7fccd406da70 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccd40b3800 0x7fccd4147040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccd40b3800 0x7fccd4147040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55496/0 (socket says 172.21.15.119:55496) 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd94f8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd41594f0 0x7fccd40ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 -- 172.21.15.119:0/1452948085 learned_addr learned my addr 172.21.15.119:0/1452948085 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd8cf7700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd406da70 0x7fccd41585e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 -- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd406da70 msgr2=0x7fccd41585e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd406da70 0x7fccd41585e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 -- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd41594f0 msgr2=0x7fccd40ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd41594f0 0x7fccd40ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 -- 172.21.15.119:0/1452948085 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcccc009a70 con 0x7fccd40b3800 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd8cf7700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd406da70 0x7fccd41585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd3fff700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccd40b3800 0x7fccd4147040 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fccc0001eb0 tx=0x7fccc000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd17fa700 1 -- 172.21.15.119:0/1452948085 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0005d90 con 0x7fccd40b3800 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd17fa700 1 -- 172.21.15.119:0/1452948085 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccc00070d0 con 0x7fccd40b3800 2024-09-13T16:34:23.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc80094b0 con 0x7fccd40b3800 2024-09-13T16:34:23.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd17fa700 1 -- 172.21.15.119:0/1452948085 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccc0007440 con 0x7fccd40b3800 2024-09-13T16:34:23.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccd414d520 con 0x7fccd40b3800 2024-09-13T16:34:23.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.197+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccd40b26f0 con 0x7fccd40b3800 2024-09-13T16:34:23.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.201+0000 7fccd17fa700 1 -- 172.21.15.119:0/1452948085 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fccc00075a0 con 0x7fccd40b3800 2024-09-13T16:34:23.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.201+0000 7fccd17fa700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fccb8066760 0x7fccb8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.201+0000 7fccd17fa700 1 -- 172.21.15.119:0/1452948085 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fccc0083570 con 0x7fccd40b3800 2024-09-13T16:34:23.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.201+0000 7fccd8cf7700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fccb8066760 0x7fccb8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.201+0000 7fccd8cf7700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fccb8066760 0x7fccb8068c20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fccc8000c00 tx=0x7fccc8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.209+0000 7fccd17fa700 1 -- 172.21.15.119:0/1452948085 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fccd40b26f0 con 0x7fccd40b3800 2024-09-13T16:34:23.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.385+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccd40b26f0 con 0x7fccd40b3800 2024-09-13T16:34:23.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd17fa700 1 -- 172.21.15.119:0/1452948085 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+16826 (secure 0 0 0) 0x7fccd40b26f0 con 0x7fccd40b3800 2024-09-13T16:34:23.392 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:23.392 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":113,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:34:10.103391+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:34:23.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fccb8066760 msgr2=0x7fccb8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fccb8066760 0x7fccb8068c20 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7fccc8000c00 tx=0x7fccc8008030 comp rx=0 tx=0).stop 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccd40b3800 msgr2=0x7fccd4147040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccd40b3800 0x7fccd4147040 secure :-1 s=READY pgs=1635 cs=0 l=1 rev1=1 crypto rx=0x7fccc0001eb0 tx=0x7fccc000c860 comp rx=0 tx=0).stop 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 shutdown_connections 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fccb8066760 0x7fccb8068c20 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fccd41594f0 0x7fccd40ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fccd40b3800 0x7fccd4147040 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 --2- 172.21.15.119:0/1452948085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fccd406da70 0x7fccd41585e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 >> 172.21.15.119:0/1452948085 conn(0x7fccd406a350 msgr2=0x7fccd40718d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 shutdown_connections 2024-09-13T16:34:23.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.389+0000 7fccd9cf9700 1 -- 172.21.15.119:0/1452948085 wait complete. 2024-09-13T16:34:23.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:33:57.346379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:33:58.017381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:34:23.408 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:34:23.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 Processor -- start 2024-09-13T16:34:23.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 -- start start 2024-09-13T16:34:23.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0554160210 0x7f05540d3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f05540d3760 0x7f05540d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d1610 0x7f05540cd620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05541605f0 con 0x7f0554160210 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05540d1a80 con 0x7f05540d3760 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0553fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05540b2240 con 0x7f05540d1610 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0552ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0554160210 0x7f05540d3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0552ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0554160210 0x7f05540d3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55508/0 (socket says 172.21.15.119:55508) 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0552ffd700 1 -- 172.21.15.119:0/1467321913 learned_addr learned my addr 172.21.15.119:0/1467321913 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f05527fc700 1 --2- 172.21.15.119:0/1467321913 >> v2:172.21.15.145:3300/0 conn(0x7f05540d3760 0x7f05540d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f05537fe700 1 --2- 172.21.15.119:0/1467321913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d1610 0x7f05540cd620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0552ffd700 1 -- 172.21.15.119:0/1467321913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d1610 msgr2=0x7f05540cd620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0552ffd700 1 --2- 172.21.15.119:0/1467321913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d1610 0x7f05540cd620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.525+0000 7f0552ffd700 1 -- 172.21.15.119:0/1467321913 >> v2:172.21.15.145:3300/0 conn(0x7f05540d3760 msgr2=0x7f05540d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:23.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0552ffd700 1 --2- 172.21.15.119:0/1467321913 >> v2:172.21.15.145:3300/0 conn(0x7f05540d3760 0x7f05540d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0552ffd700 1 -- 172.21.15.119:0/1467321913 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05540b2520 con 0x7f0554160210 2024-09-13T16:34:23.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f05537fe700 1 --2- 172.21.15.119:0/1467321913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d1610 0x7f05540cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:34:23.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f05527fc700 1 --2- 172.21.15.119:0/1467321913 >> v2:172.21.15.145:3300/0 conn(0x7f05540d3760 0x7f05540d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:34:23.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0552ffd700 1 --2- 172.21.15.119:0/1467321913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0554160210 0x7f05540d3220 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f055408cab0 tx=0x7f0544009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db05865b235d1a1a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0551ffb700 1 -- 172.21.15.119:0/1467321913 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054400ba20 con 0x7f0554160210 2024-09-13T16:34:23.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0551ffb700 1 -- 172.21.15.119:0/1467321913 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054400bb80 con 0x7f0554160210 2024-09-13T16:34:23.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0551ffb700 1 -- 172.21.15.119:0/1467321913 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544009e60 con 0x7f0554160210 2024-09-13T16:34:23.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 -- 172.21.15.119:0/1467321913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0554160210 msgr2=0x7f05540d3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 --2- 172.21.15.119:0/1467321913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0554160210 0x7f05540d3220 secure :-1 s=READY pgs=1636 cs=0 l=1 rev1=1 crypto rx=0x7f055408cab0 tx=0x7f0544009c20 comp rx=0 tx=0).stop 2024-09-13T16:34:23.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 -- 172.21.15.119:0/1467321913 shutdown_connections 2024-09-13T16:34:23.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 --2- 172.21.15.119:0/1467321913 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d1610 0x7f05540cd620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 --2- 172.21.15.119:0/1467321913 >> v2:172.21.15.145:3300/0 conn(0x7f05540d3760 0x7f05540d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 --2- 172.21.15.119:0/1467321913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0554160210 0x7f05540d3220 unknown :-1 s=CLOSED pgs=1636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 -- 172.21.15.119:0/1467321913 >> 172.21.15.119:0/1467321913 conn(0x7f055408c080 msgr2=0x7f05540d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:23.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 -- 172.21.15.119:0/1467321913 shutdown_connections 2024-09-13T16:34:23.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 -- 172.21.15.119:0/1467321913 wait complete. 2024-09-13T16:34:23.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 Processor -- start 2024-09-13T16:34:23.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.529+0000 7f0553fff700 1 -- start start 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05540d1610 0x7f05541f1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d3760 0x7f05540cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554160210 0x7f05540cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541f6870 con 0x7f05540d1610 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05540cb740 con 0x7f0554160210 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05540b84c0 con 0x7f05540d3760 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05540d1610 0x7f05541f1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05540d1610 0x7f05541f1b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55512/0 (socket says 172.21.15.119:55512) 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 -- 172.21.15.119:0/2749088937 learned_addr learned my addr 172.21.15.119:0/2749088937 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f05527fc700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d3760 0x7f05540cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f05537fe700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554160210 0x7f05540cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 -- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d3760 msgr2=0x7f05540cd5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d3760 0x7f05540cd5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 -- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554160210 msgr2=0x7f05540cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554160210 0x7f05540cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 -- 172.21.15.119:0/2749088937 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0544009570 con 0x7f05540d1610 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f05527fc700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d3760 0x7f05540cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f05537fe700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554160210 0x7f05540cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0552ffd700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05540d1610 0x7f05541f1b30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f055408b5e0 tx=0x7f0544004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f053b7fe700 1 -- 172.21.15.119:0/2749088937 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544004400 con 0x7f05540d1610 2024-09-13T16:34:23.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f053b7fe700 1 -- 172.21.15.119:0/2749088937 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054401f070 con 0x7f05540d1610 2024-09-13T16:34:23.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05540b87a0 con 0x7f05540d1610 2024-09-13T16:34:23.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f053b7fe700 1 -- 172.21.15.119:0/2749088937 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05440036c0 con 0x7f05540d1610 2024-09-13T16:34:23.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0554149960 con 0x7f05540d1610 2024-09-13T16:34:23.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.533+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05540cfed0 con 0x7f05540d1610 2024-09-13T16:34:23.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.537+0000 7f053b7fe700 1 -- 172.21.15.119:0/2749088937 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0544021030 con 0x7f05540d1610 2024-09-13T16:34:23.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.537+0000 7f053b7fe700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0540066780 0x7f0540068c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.537+0000 7f053b7fe700 1 -- 172.21.15.119:0/2749088937 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0544085660 con 0x7f05540d1610 2024-09-13T16:34:23.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.537+0000 7f05527fc700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0540066780 0x7f0540068c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.537+0000 7f05527fc700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0540066780 0x7f0540068c40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f055415ce10 tx=0x7f0548008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.545+0000 7f053b7fe700 1 -- 172.21.15.119:0/2749088937 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f05540cfed0 con 0x7f05540d1610 2024-09-13T16:34:23.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.717+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f05540cfed0 con 0x7f05540d1610 2024-09-13T16:34:23.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.717+0000 7f053b7fe700 1 -- 172.21.15.119:0/2749088937 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f05540cfed0 con 0x7f05540d1610 2024-09-13T16:34:23.722 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:34:23.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0540066780 msgr2=0x7f0540068c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0540066780 0x7f0540068c40 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f055415ce10 tx=0x7f0548008040 comp rx=0 tx=0).stop 2024-09-13T16:34:23.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05540d1610 msgr2=0x7f05541f1b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05540d1610 0x7f05541f1b30 secure :-1 s=READY pgs=1637 cs=0 l=1 rev1=1 crypto rx=0x7f055408b5e0 tx=0x7f0544004770 comp rx=0 tx=0).stop 2024-09-13T16:34:23.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 shutdown_connections 2024-09-13T16:34:23.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0540066780 0x7f0540068c40 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0554160210 0x7f05540cdb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f05540d3760 0x7f05540cd5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 --2- 172.21.15.119:0/2749088937 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05540d1610 0x7f05541f1b30 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 >> 172.21.15.119:0/2749088937 conn(0x7f055408c080 msgr2=0x7f055415eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:23.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 shutdown_connections 2024-09-13T16:34:23.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.721+0000 7f0553fff700 1 -- 172.21.15.119:0/2749088937 wait complete. 2024-09-13T16:34:23.742 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:34:23.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f478a3dd700 1 Processor -- start 2024-09-13T16:34:23.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f478a3dd700 1 -- start start 2024-09-13T16:34:23.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f478a3dd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f47840ade80 0x7f47841652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f478a3dd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4784071ac0 con 0x7f478406cdb0 2024-09-13T16:34:23.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f478a3dd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4784070eb0 con 0x7f47840b24a0 2024-09-13T16:34:23.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f478a3dd700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47841657f0 con 0x7f47840ade80 2024-09-13T16:34:23.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f4789bdc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f478406cdb0 0x7f47841619e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33132/0 (socket says 172.21.15.119:33132) 2024-09-13T16:34:23.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f4789bdc700 1 -- 172.21.15.119:0/329398219 learned_addr learned my addr 172.21.15.119:0/329398219 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:23.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f47893db700 1 --2- 172.21.15.119:0/329398219 >> v2:172.21.15.145:3300/0 conn(0x7f47840ade80 0x7f47841652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f477bfff700 1 -- 172.21.15.119:0/329398219 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939049050 0 0) 0x7f4784071ac0 con 0x7f478406cdb0 2024-09-13T16:34:23.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f477bfff700 1 -- 172.21.15.119:0/329398219 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47700035d0 con 0x7f478406cdb0 2024-09-13T16:34:23.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f477bfff700 1 -- 172.21.15.119:0/329398219 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23571468 0 0) 0x7f4784070eb0 con 0x7f47840b24a0 2024-09-13T16:34:23.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f477bfff700 1 -- 172.21.15.119:0/329398219 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4784071ac0 con 0x7f47840b24a0 2024-09-13T16:34:23.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f47893db700 1 -- 172.21.15.119:0/329398219 >> v1:172.21.15.145:6789/0 conn(0x7f47840b24a0 legacy=0x7f47841612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f47893db700 1 -- 172.21.15.119:0/329398219 >> v1:172.21.15.119:6789/0 conn(0x7f478406cdb0 legacy=0x7f47841619e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f47893db700 1 -- 172.21.15.119:0/329398219 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4784071190 con 0x7f47840ade80 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.861+0000 7f47893db700 1 --2- 172.21.15.119:0/329398219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47841652b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4774009850 tx=0x7f4774009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f35993ce2e8aef4c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f477bfff700 1 -- 172.21.15.119:0/329398219 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774011830 con 0x7f47840ade80 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f477bfff700 1 -- 172.21.15.119:0/329398219 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4774011990 con 0x7f47840ade80 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f477bfff700 1 -- 172.21.15.119:0/329398219 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4774011d40 con 0x7f47840ade80 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 -- 172.21.15.119:0/329398219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 msgr2=0x7f47841652b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 --2- 172.21.15.119:0/329398219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47841652b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f4774009850 tx=0x7f4774009b60 comp rx=0 tx=0).stop 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 -- 172.21.15.119:0/329398219 shutdown_connections 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 --2- 172.21.15.119:0/329398219 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47841652b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 -- 172.21.15.119:0/329398219 >> 172.21.15.119:0/329398219 conn(0x7f47840b0980 msgr2=0x7f47840b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 -- 172.21.15.119:0/329398219 shutdown_connections 2024-09-13T16:34:23.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 -- 172.21.15.119:0/329398219 wait complete. 2024-09-13T16:34:23.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.865+0000 7f478a3dd700 1 Processor -- start 2024-09-13T16:34:23.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 -- start start 2024-09-13T16:34:23.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f478406cdb0 0x7f478414c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47840ab770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840b24a0 0x7f47840abcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f478414c7c0 con 0x7f478406cdb0 2024-09-13T16:34:23.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47840cd780 con 0x7f47840ade80 2024-09-13T16:34:23.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47840cd4a0 con 0x7f47840b24a0 2024-09-13T16:34:23.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f478406cdb0 0x7f478414c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f478406cdb0 0x7f478414c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55522/0 (socket says 172.21.15.119:55522) 2024-09-13T16:34:23.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 -- 172.21.15.119:0/1856141381 learned_addr learned my addr 172.21.15.119:0/1856141381 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:23.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f4789bdc700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840b24a0 0x7f47840abcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f4788bda700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47840ab770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 -- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840b24a0 msgr2=0x7f47840abcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840b24a0 0x7f47840abcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 -- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 msgr2=0x7f47840ab770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:23.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47840ab770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:23.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 -- 172.21.15.119:0/1856141381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47740094b0 con 0x7f478406cdb0 2024-09-13T16:34:23.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f4788bda700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47840ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:34:23.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f47893db700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f478406cdb0 0x7f478414c280 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f47840b15b0 tx=0x7f4774004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f4779ffb700 1 -- 172.21.15.119:0/1856141381 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477401aac0 con 0x7f478406cdb0 2024-09-13T16:34:23.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f4779ffb700 1 -- 172.21.15.119:0/1856141381 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f477401f070 con 0x7f478406cdb0 2024-09-13T16:34:23.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f478414cb00 con 0x7f478406cdb0 2024-09-13T16:34:23.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f4779ffb700 1 -- 172.21.15.119:0/1856141381 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f477401ae60 con 0x7f478406cdb0 2024-09-13T16:34:23.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.869+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478414d0c0 con 0x7f478406cdb0 2024-09-13T16:34:23.880 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.873+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f478414d2b0 con 0x7f478406cdb0 2024-09-13T16:34:23.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.873+0000 7f4779ffb700 1 -- 172.21.15.119:0/1856141381 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f477401b430 con 0x7f478406cdb0 2024-09-13T16:34:23.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.873+0000 7f4779ffb700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4768066730 0x7f4768068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:23.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.873+0000 7f4779ffb700 1 -- 172.21.15.119:0/1856141381 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f47740853b0 con 0x7f478406cdb0 2024-09-13T16:34:23.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.873+0000 7f4788bda700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4768066730 0x7f4768068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:23.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.873+0000 7f4788bda700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4768066730 0x7f4768068bf0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f47840a8d20 tx=0x7f4780009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:23.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:23.881+0000 7f4779ffb700 1 -- 172.21.15.119:0/1856141381 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f47740525d0 con 0x7f478406cdb0 2024-09-13T16:34:24.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.045+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4784163090 con 0x7f478406cdb0 2024-09-13T16:34:24.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.045+0000 7f4779ffb700 1 -- 172.21.15.119:0/1856141381 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4774055bf0 con 0x7f478406cdb0 2024-09-13T16:34:24.049 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:34:24.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4768066730 msgr2=0x7f4768068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4768066730 0x7f4768068bf0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f47840a8d20 tx=0x7f4780009040 comp rx=0 tx=0).stop 2024-09-13T16:34:24.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f478406cdb0 msgr2=0x7f478414c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f478406cdb0 0x7f478414c280 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7f47840b15b0 tx=0x7f4774004770 comp rx=0 tx=0).stop 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 shutdown_connections 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4768066730 0x7f4768068bf0 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f47840b24a0 0x7f47840abcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f47840ade80 0x7f47840ab770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 --2- 172.21.15.119:0/1856141381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f478406cdb0 0x7f478414c280 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 >> 172.21.15.119:0/1856141381 conn(0x7f47840b0980 msgr2=0x7f47841601c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 shutdown_connections 2024-09-13T16:34:24.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.049+0000 7f478a3dd700 1 -- 172.21.15.119:0/1856141381 wait complete. 2024-09-13T16:34:24.064 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:34:24.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59af37e700 1 Processor -- start 2024-09-13T16:34:24.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59af37e700 1 -- start start 2024-09-13T16:34:24.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59af37e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f59a806ee70 0x7f59a815bb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59af37e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f59a815e580 0x7f59a815ea10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59af37e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a80cd580 con 0x7f59a806f250 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59af37e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a815b0a0 con 0x7f59a806ee70 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59af37e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a815b380 con 0x7f59a815e580 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59ac919700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f59a806f250 0x7f59a80ccb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33136/0 (socket says 172.21.15.119:33136) 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59ac919700 1 -- 172.21.15.119:0/1196308899 learned_addr learned my addr 172.21.15.119:0/1196308899 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59ad11a700 1 --2- 172.21.15.119:0/1196308899 >> v2:172.21.15.119:3300/0 conn(0x7f59a806ee70 0x7f59a815bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59ad91b700 1 --2- 172.21.15.119:0/1196308899 >> v2:172.21.15.145:3300/0 conn(0x7f59a815e580 0x7f59a815ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59a7fff700 1 -- 172.21.15.119:0/1196308899 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924316128 0 0) 0x7f59a80cd580 con 0x7f59a806f250 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59a7fff700 1 -- 172.21.15.119:0/1196308899 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59900035d0 con 0x7f59a806f250 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59a7fff700 1 -- 172.21.15.119:0/1196308899 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3954609250 0 0) 0x7f59900035d0 con 0x7f59a806f250 2024-09-13T16:34:24.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59ad11a700 1 -- 172.21.15.119:0/1196308899 >> v2:172.21.15.145:3300/0 conn(0x7f59a815e580 msgr2=0x7f59a815ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:24.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59ad11a700 1 --2- 172.21.15.119:0/1196308899 >> v2:172.21.15.145:3300/0 conn(0x7f59a815e580 0x7f59a815ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.177+0000 7f59ad11a700 1 -- 172.21.15.119:0/1196308899 >> v1:172.21.15.119:6789/0 conn(0x7f59a806f250 legacy=0x7f59a80ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59ad11a700 1 -- 172.21.15.119:0/1196308899 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a815b660 con 0x7f59a806ee70 2024-09-13T16:34:24.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59ad11a700 1 --2- 172.21.15.119:0/1196308899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806ee70 0x7f59a815bb80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f59a8073170 tx=0x7f5998009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5c3480df85e42b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:24.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59a7fff700 1 -- 172.21.15.119:0/1196308899 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c0019f0 con 0x7f59a806ee70 2024-09-13T16:34:24.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59a7fff700 1 -- 172.21.15.119:0/1196308899 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5998004030 con 0x7f59a806ee70 2024-09-13T16:34:24.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59a7fff700 1 -- 172.21.15.119:0/1196308899 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59980043c0 con 0x7f59a806ee70 2024-09-13T16:34:24.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 -- 172.21.15.119:0/1196308899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806ee70 msgr2=0x7f59a815bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 --2- 172.21.15.119:0/1196308899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806ee70 0x7f59a815bb80 secure :-1 s=READY pgs=1641 cs=0 l=1 rev1=1 crypto rx=0x7f59a8073170 tx=0x7f5998009b30 comp rx=0 tx=0).stop 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 -- 172.21.15.119:0/1196308899 shutdown_connections 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 --2- 172.21.15.119:0/1196308899 >> v2:172.21.15.145:3300/0 conn(0x7f59a815e580 0x7f59a815ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 --2- 172.21.15.119:0/1196308899 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806ee70 0x7f59a815bb80 unknown :-1 s=CLOSED pgs=1641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 -- 172.21.15.119:0/1196308899 >> 172.21.15.119:0/1196308899 conn(0x7f59a8070dc0 msgr2=0x7f59a80711d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 -- 172.21.15.119:0/1196308899 shutdown_connections 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 -- 172.21.15.119:0/1196308899 wait complete. 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 Processor -- start 2024-09-13T16:34:24.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.181+0000 7f59af37e700 1 -- start start 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a806ee70 0x7f59a81443d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806f250 0x7f59a80ad200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a815e580 0x7f59a80ad740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80ab4c0 con 0x7f59a806f250 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80aa8b0 con 0x7f59a806ee70 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59a80aab90 con 0x7f59a815e580 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806f250 0x7f59a80ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806f250 0x7f59a80ad200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55534/0 (socket says 172.21.15.119:55534) 2024-09-13T16:34:24.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 -- 172.21.15.119:0/1684578967 learned_addr learned my addr 172.21.15.119:0/1684578967 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:24.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ad11a700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a806ee70 0x7f59a81443d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ad91b700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a815e580 0x7f59a80ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 -- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a815e580 msgr2=0x7f59a80ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a815e580 0x7f59a80ad740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 -- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a806ee70 msgr2=0x7f59a81443d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a806ee70 0x7f59a81443d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 -- 172.21.15.119:0/1684578967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59980094b0 con 0x7f59a806f250 2024-09-13T16:34:24.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59ac919700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806f250 0x7f59a80ad200 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f599c0019b0 tx=0x7f599c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:24.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59a5ffb700 1 -- 172.21.15.119:0/1684578967 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c00acd0 con 0x7f59a806f250 2024-09-13T16:34:24.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59a5ffb700 1 -- 172.21.15.119:0/1684578967 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f599c00ae30 con 0x7f59a806f250 2024-09-13T16:34:24.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59a5ffb700 1 -- 172.21.15.119:0/1684578967 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f599c007310 con 0x7f59a806f250 2024-09-13T16:34:24.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a80aae70 con 0x7f59a806f250 2024-09-13T16:34:24.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.185+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a80a96e0 con 0x7f59a806f250 2024-09-13T16:34:24.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.189+0000 7f59a5ffb700 1 -- 172.21.15.119:0/1684578967 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f599c007470 con 0x7f59a806f250 2024-09-13T16:34:24.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.189+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a80ab670 con 0x7f59a806f250 2024-09-13T16:34:24.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.189+0000 7f59a5ffb700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5994066760 0x7f5994068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.189+0000 7f59a5ffb700 1 -- 172.21.15.119:0/1684578967 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f599c083f00 con 0x7f59a806f250 2024-09-13T16:34:24.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.189+0000 7f59ad11a700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5994066760 0x7f5994068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.189+0000 7f59ad11a700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5994066760 0x7f5994068c20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f59980090b0 tx=0x7f5998008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:24.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.197+0000 7f59a5ffb700 1 -- 172.21.15.119:0/1684578967 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f59a80ab670 con 0x7f59a806f250 2024-09-13T16:34:24.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.369+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f59a80aec40 con 0x7f59a806f250 2024-09-13T16:34:24.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.369+0000 7f59a5ffb700 1 -- 172.21.15.119:0/1684578967 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7f599c018020 con 0x7f59a806f250 2024-09-13T16:34:24.373 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:34:24.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.369+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5994066760 msgr2=0x7f5994068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.369+0000 7f59af37e700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5994066760 0x7f5994068c20 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f59980090b0 tx=0x7f5998008030 comp rx=0 tx=0).stop 2024-09-13T16:34:24.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.369+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806f250 msgr2=0x7f59a80ad200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.369+0000 7f59af37e700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806f250 0x7f59a80ad200 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f599c0019b0 tx=0x7f599c00a340 comp rx=0 tx=0).stop 2024-09-13T16:34:24.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 shutdown_connections 2024-09-13T16:34:24.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5994066760 0x7f5994068c20 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59a815e580 0x7f59a80ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59a806f250 0x7f59a80ad200 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 --2- 172.21.15.119:0/1684578967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59a806ee70 0x7f59a81443d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 >> 172.21.15.119:0/1684578967 conn(0x7f59a8070dc0 msgr2=0x7f59a80b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:24.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 shutdown_connections 2024-09-13T16:34:24.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.373+0000 7f59af37e700 1 -- 172.21.15.119:0/1684578967 wait complete. 2024-09-13T16:34:24.386 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:34:24.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.493+0000 7f55383f4700 1 Processor -- start 2024-09-13T16:34:24.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.493+0000 7f55383f4700 1 -- start start 2024-09-13T16:34:24.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b36f0 0x7f55300b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5530159a50 0x7f55300cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55300b2320 con 0x7f55300b1e80 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55300b2600 con 0x7f5530159a50 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55300b28e0 con 0x7f55300b36f0 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536991700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5530159a50 0x7f55300cd2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f553598f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f55300b1e80 0x7f5530159340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33138/0 (socket says 172.21.15.119:33138) 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536991700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5530159a50 0x7f55300cd2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55544/0 (socket says 172.21.15.119:55544) 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f553598f700 1 -- 172.21.15.119:0/51757093 learned_addr learned my addr 172.21.15.119:0/51757093 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:24.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536190700 1 --2- 172.21.15.119:0/51757093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b36f0 0x7f55300b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536991700 1 -- 172.21.15.119:0/51757093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b36f0 msgr2=0x7f55300b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536991700 1 --2- 172.21.15.119:0/51757093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b36f0 0x7f55300b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536991700 1 -- 172.21.15.119:0/51757093 >> v1:172.21.15.119:6789/0 conn(0x7f55300b1e80 legacy=0x7f5530159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536991700 1 -- 172.21.15.119:0/51757093 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55300b2bc0 con 0x7f5530159a50 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f553518e700 1 -- 172.21.15.119:0/51757093 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1579941160 0 0) 0x7f55300b2320 con 0x7f55300b1e80 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f5536991700 1 --2- 172.21.15.119:0/51757093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5530159a50 0x7f55300cd2f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f5528009d90 tx=0x7f552800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7b83164fc8b676 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f553518e700 1 -- 172.21.15.119:0/51757093 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552800e070 con 0x7f5530159a50 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f553518e700 1 -- 172.21.15.119:0/51757093 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552800bb20 con 0x7f5530159a50 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f553518e700 1 -- 172.21.15.119:0/51757093 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5528012430 con 0x7f5530159a50 2024-09-13T16:34:24.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- 172.21.15.119:0/51757093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5530159a50 msgr2=0x7f55300cd2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 --2- 172.21.15.119:0/51757093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5530159a50 0x7f55300cd2f0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f5528009d90 tx=0x7f552800b330 comp rx=0 tx=0).stop 2024-09-13T16:34:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- 172.21.15.119:0/51757093 shutdown_connections 2024-09-13T16:34:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 --2- 172.21.15.119:0/51757093 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5530159a50 0x7f55300cd2f0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 --2- 172.21.15.119:0/51757093 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b36f0 0x7f55300b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- 172.21.15.119:0/51757093 >> 172.21.15.119:0/51757093 conn(0x7f553006a350 msgr2=0x7f553006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- 172.21.15.119:0/51757093 shutdown_connections 2024-09-13T16:34:24.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.497+0000 7f55383f4700 1 -- 172.21.15.119:0/51757093 wait complete. 2024-09-13T16:34:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 Processor -- start 2024-09-13T16:34:24.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 -- start start 2024-09-13T16:34:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b1e80 0x7f5530142ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55300b36f0 0x7f553012eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5530159a50 0x7f553014f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553014f3d0 con 0x7f55300b36f0 2024-09-13T16:34:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5530143530 con 0x7f5530159a50 2024-09-13T16:34:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f55383f4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553012f050 con 0x7f55300b1e80 2024-09-13T16:34:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55300b36f0 0x7f553012eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55300b36f0 0x7f553012eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55548/0 (socket says 172.21.15.119:55548) 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 -- 172.21.15.119:0/903945994 learned_addr learned my addr 172.21.15.119:0/903945994 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f5536190700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b1e80 0x7f5530142ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f5536991700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5530159a50 0x7f553014f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 -- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b1e80 msgr2=0x7f5530142ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b1e80 0x7f5530142ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 -- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5530159a50 msgr2=0x7f553014f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5530159a50 0x7f553014f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.501+0000 7f553598f700 1 -- 172.21.15.119:0/903945994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5528009a70 con 0x7f55300b36f0 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f553598f700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55300b36f0 0x7f553012eb10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f552000c5b0 tx=0x7f552000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:24.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f5526ffd700 1 -- 172.21.15.119:0/903945994 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520004ca0 con 0x7f55300b36f0 2024-09-13T16:34:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f5526ffd700 1 -- 172.21.15.119:0/903945994 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5520004e20 con 0x7f55300b36f0 2024-09-13T16:34:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f553014fda0 con 0x7f55300b36f0 2024-09-13T16:34:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f5526ffd700 1 -- 172.21.15.119:0/903945994 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5520007310 con 0x7f55300b36f0 2024-09-13T16:34:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f553014eaa0 con 0x7f55300b36f0 2024-09-13T16:34:24.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5530067760 con 0x7f55300b36f0 2024-09-13T16:34:24.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.505+0000 7f5526ffd700 1 -- 172.21.15.119:0/903945994 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5520007470 con 0x7f55300b36f0 2024-09-13T16:34:24.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.509+0000 7f5526ffd700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5514066730 0x7f5514068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:24.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.509+0000 7f5526ffd700 1 -- 172.21.15.119:0/903945994 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5520083f40 con 0x7f55300b36f0 2024-09-13T16:34:24.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.509+0000 7f5536190700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5514066730 0x7f5514068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:24.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.509+0000 7f5536190700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5514066730 0x7f5514068bf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5530157c00 tx=0x7f552c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:24.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.513+0000 7f5526ffd700 1 -- 172.21.15.119:0/903945994 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f55200511e0 con 0x7f55300b36f0 2024-09-13T16:34:24.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.693+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f55300b1940 con 0x7f55300b36f0 2024-09-13T16:34:24.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.693+0000 7f5526ffd700 1 -- 172.21.15.119:0/903945994 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v113) v1 ==== 80+0+0 (secure 0 0 0) 0x7f5520007720 con 0x7f55300b36f0 2024-09-13T16:34:24.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5514066730 msgr2=0x7f5514068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5514066730 0x7f5514068bf0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7f5530157c00 tx=0x7f552c009240 comp rx=0 tx=0).stop 2024-09-13T16:34:24.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55300b36f0 msgr2=0x7f553012eb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:24.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55300b36f0 0x7f553012eb10 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f552000c5b0 tx=0x7f552000c8c0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 shutdown_connections 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5514066730 0x7f5514068bf0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5530159a50 0x7f553014f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f55300b36f0 0x7f553012eb10 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 --2- 172.21.15.119:0/903945994 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f55300b1e80 0x7f5530142ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 >> 172.21.15.119:0/903945994 conn(0x7f553006a350 msgr2=0x7f5530156f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 shutdown_connections 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:24.697+0000 7f55383f4700 1 -- 172.21.15.119:0/903945994 wait complete. 2024-09-13T16:34:24.701 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:34:24.712 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:34:24.713 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:24.713 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:24.727 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[51312]: fuse finished with error 0 and tester_r 0 2024-09-13T16:34:24.731 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:34:28.247 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:34:28.249 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:34:30.833 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:34:30.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:30.833 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:34:30.841 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:34:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:30.997+0000 7fd2a2438700 1 Processor -- start 2024-09-13T16:34:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:30.997+0000 7fd2a2438700 1 -- start start 2024-09-13T16:34:31.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd29c143f50 0x7fd29c06c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c0b3410 0x7fd29c164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd29c0b1c40 con 0x7fd29c144330 2024-09-13T16:34:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29c06cdf0 con 0x7fd29c143f50 2024-09-13T16:34:31.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c06d0d0 con 0x7fd29c0b3410 2024-09-13T16:34:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a1436700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd29c143f50 0x7fd29c06c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a0c35700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fd29c144330 0x7fd29c0cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:58754/0 (socket says 172.21.15.119:58754) 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a0c35700 1 -- 172.21.15.119:0/2284845624 learned_addr learned my addr 172.21.15.119:0/2284845624 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a1c37700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c0b3410 0x7fd29c164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd293fff700 1 -- 172.21.15.119:0/2284845624 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514355438 0 0) 0x7fd29c0b1c40 con 0x7fd29c144330 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd293fff700 1 -- 172.21.15.119:0/2284845624 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2880035d0 con 0x7fd29c144330 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a1c37700 1 -- 172.21.15.119:0/2284845624 >> v2:172.21.15.145:3300/0 conn(0x7fd29c143f50 msgr2=0x7fd29c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a1c37700 1 --2- 172.21.15.119:0/2284845624 >> v2:172.21.15.145:3300/0 conn(0x7fd29c143f50 0x7fd29c06c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a1c37700 1 -- 172.21.15.119:0/2284845624 >> v1:172.21.15.145:6789/0 conn(0x7fd29c144330 legacy=0x7fd29c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a1c37700 1 -- 172.21.15.119:0/2284845624 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29c06d3b0 con 0x7fd29c0b3410 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a1c37700 1 --2- 172.21.15.119:0/2284845624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c0b3410 0x7fd29c164330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd294009d90 tx=0x7fd29400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d8774f1ba00b86e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd293fff700 1 -- 172.21.15.119:0/2284845624 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd29400e070 con 0x7fd29c0b3410 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd293fff700 1 -- 172.21.15.119:0/2284845624 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd29400bb20 con 0x7fd29c0b3410 2024-09-13T16:34:31.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd293fff700 1 -- 172.21.15.119:0/2284845624 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd294012430 con 0x7fd29c0b3410 2024-09-13T16:34:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- 172.21.15.119:0/2284845624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c0b3410 msgr2=0x7fd29c164330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 --2- 172.21.15.119:0/2284845624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c0b3410 0x7fd29c164330 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd294009d90 tx=0x7fd29400b330 comp rx=0 tx=0).stop 2024-09-13T16:34:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- 172.21.15.119:0/2284845624 shutdown_connections 2024-09-13T16:34:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 --2- 172.21.15.119:0/2284845624 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c0b3410 0x7fd29c164330 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 --2- 172.21.15.119:0/2284845624 >> v2:172.21.15.145:3300/0 conn(0x7fd29c143f50 0x7fd29c06c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- 172.21.15.119:0/2284845624 >> 172.21.15.119:0/2284845624 conn(0x7fd29c06a350 msgr2=0x7fd29c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- 172.21.15.119:0/2284845624 shutdown_connections 2024-09-13T16:34:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.001+0000 7fd2a2438700 1 -- 172.21.15.119:0/2284845624 wait complete. 2024-09-13T16:34:31.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 Processor -- start 2024-09-13T16:34:31.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 -- start start 2024-09-13T16:34:31.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29c0b3410 0x7fd29c15a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c143f50 0x7fd29c0aa180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd29c144330 0x7fd29c0abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c1e4ab0 con 0x7fd29c0b3410 2024-09-13T16:34:31.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c1e4d90 con 0x7fd29c144330 2024-09-13T16:34:31.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a2438700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29c1e5070 con 0x7fd29c143f50 2024-09-13T16:34:31.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a1436700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29c0b3410 0x7fd29c15a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a1436700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29c0b3410 0x7fd29c15a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55554/0 (socket says 172.21.15.119:55554) 2024-09-13T16:34:31.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a1436700 1 -- 172.21.15.119:0/3337945314 learned_addr learned my addr 172.21.15.119:0/3337945314 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a0c35700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c143f50 0x7fd29c0aa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.005+0000 7fd2a1c37700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd29c144330 0x7fd29c0abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a1436700 1 -- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c143f50 msgr2=0x7fd29c0aa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a1436700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c143f50 0x7fd29c0aa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a1436700 1 -- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd29c144330 msgr2=0x7fd29c0abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a1436700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd29c144330 0x7fd29c0abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a1436700 1 -- 172.21.15.119:0/3337945314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd294009a70 con 0x7fd29c0b3410 2024-09-13T16:34:31.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a1436700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29c0b3410 0x7fd29c15a430 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00e9d0 tx=0x7fd28c00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd291ffb700 1 -- 172.21.15.119:0/3337945314 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c009720 con 0x7fd29c0b3410 2024-09-13T16:34:31.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd291ffb700 1 -- 172.21.15.119:0/3337945314 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c00ee70 con 0x7fd29c0b3410 2024-09-13T16:34:31.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd29c1e5350 con 0x7fd29c0b3410 2024-09-13T16:34:31.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd291ffb700 1 -- 172.21.15.119:0/3337945314 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28c00ca50 con 0x7fd29c0b3410 2024-09-13T16:34:31.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29c15a970 con 0x7fd29c0b3410 2024-09-13T16:34:31.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd29c0add50 con 0x7fd29c0b3410 2024-09-13T16:34:31.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.009+0000 7fd291ffb700 1 -- 172.21.15.119:0/3337945314 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd28c00cbb0 con 0x7fd29c0b3410 2024-09-13T16:34:31.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.013+0000 7fd291ffb700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd280066730 0x7fd280068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.013+0000 7fd291ffb700 1 -- 172.21.15.119:0/3337945314 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd28c0842a0 con 0x7fd29c0b3410 2024-09-13T16:34:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.013+0000 7fd2a0c35700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd280066730 0x7fd280068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.013+0000 7fd2a0c35700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd280066730 0x7fd280068bf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd29c141040 tx=0x7fd298007de0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.021+0000 7fd291ffb700 1 -- 172.21.15.119:0/3337945314 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd28c0514b0 con 0x7fd29c0b3410 2024-09-13T16:34:31.237 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.233+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd29c0cc7d0 con 0x7fd29c0b3410 2024-09-13T16:34:31.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd291ffb700 1 -- 172.21.15.119:0/3337945314 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+4748 (secure 0 0 0) 0x7fd28c00ce60 con 0x7fd29c0b3410 2024-09-13T16:34:31.239 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:31.239 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":76,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:34:02.039855+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8829},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8829":{"gid":8829,"name":"c","rank":0,"incarnation":73,"state":"up:active","state_seq":3,"addr":"172.21.15.119:6835/3700300033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":3700300033},{"type":"v1","addr":"172.21.15.119:6835","nonce":3700300033}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:34:31.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd280066730 msgr2=0x7fd280068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd280066730 0x7fd280068bf0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd29c141040 tx=0x7fd298007de0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29c0b3410 msgr2=0x7fd29c15a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29c0b3410 0x7fd29c15a430 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fd28c00e9d0 tx=0x7fd28c00ece0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 shutdown_connections 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd280066730 0x7fd280068bf0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd29c144330 0x7fd29c0abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd29c143f50 0x7fd29c0aa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 --2- 172.21.15.119:0/3337945314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd29c0b3410 0x7fd29c15a430 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 >> 172.21.15.119:0/3337945314 conn(0x7fd29c06a350 msgr2=0x7fd29c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 shutdown_connections 2024-09-13T16:34:31.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.237+0000 7fd2a2438700 1 -- 172.21.15.119:0/3337945314 wait complete. 2024-09-13T16:34:31.243 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 76 2024-09-13T16:34:31.255 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.c.asok --format=json flush journal 2024-09-13T16:34:31.380 INFO:teuthology.orchestra.run.smithi119.stdout:{"message":"","return_code":0} 2024-09-13T16:34:31.380 DEBUG:tasks.cephfs.filesystem:_json_asok output { "message": "", "return_code": 0 } 2024-09-13T16:34:31.380 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:34:31.380 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:34:31.388 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:34:31.388 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi119.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-09-13T16:34:31.388 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:34:31.388 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:34:31.388 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:34:31.436 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T16:34:31.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:34:31.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo daemon-helper kill ceph --cluster ceph -w 2024-09-13T16:34:31.440 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-09-13T16:34:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.561+0000 7f7c4ae77700 1 Processor -- start 2024-09-13T16:34:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.561+0000 7f7c4ae77700 1 -- start start 2024-09-13T16:34:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7c440b0e00 0x7f7c440b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c440b52b0 con 0x7f7c44073ab0 2024-09-13T16:34:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c440acf70 con 0x7f7c440cb480 2024-09-13T16:34:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c440ad2c0 con 0x7f7c440b0e00 2024-09-13T16:34:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7c440b0e00 0x7f7c440b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49674700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f7c44073ab0 0x7f7c4406a450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33142/0 (socket says 172.21.15.119:33142) 2024-09-13T16:34:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49674700 1 -- 172.21.15.119:0/3437217251 learned_addr learned my addr 172.21.15.119:0/3437217251 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317876089 0 0) 0x7f7c440b52b0 con 0x7f7c44073ab0 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c300035d0 con 0x7f7c44073ab0 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360767179 0 0) 0x7f7c300035d0 con 0x7f7c44073ab0 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 >> v2:172.21.15.119:3300/0 conn(0x7f7c440b0e00 msgr2=0x7f7c440b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 --2- 172.21.15.119:0/3437217251 >> v2:172.21.15.119:3300/0 conn(0x7f7c440b0e00 0x7f7c440b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 >> v1:172.21.15.145:6789/0 conn(0x7f7c440cb480 legacy=0x7f7c4406ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c440aead0 con 0x7f7c44073ab0 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f7c34001670 con 0x7f7c44073ab0 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3437217251 >> v2:172.21.15.119:3300/0 conn(0x7f7c440b0e00 0x7f7c440b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 >> v1:172.21.15.119:6789/0 conn(0x7f7c44073ab0 legacy=0x7f7c4406a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c30003a20 0x7f7c30003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30004870 0x7f7c30004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c30005230 0x7f7c3000ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c440ad2c0 con 0x7f7c30003a20 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c3000c1d0 con 0x7f7c30005230 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c300107a0 con 0x7f7c30004870 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c30003a20 0x7f7c30003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4a676700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c30005230 0x7f7c3000ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49674700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30004870 0x7f7c30004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 -- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30004870 msgr2=0x7f7c30004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30004870 0x7f7c30004cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 -- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c30005230 msgr2=0x7f7c3000ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c30005230 0x7f7c3000ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4a676700 1 -- 172.21.15.119:0/3437217251 reap_dead start 2024-09-13T16:34:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 -- 172.21.15.119:0/3437217251 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c300035d0 con 0x7f7c30003a20 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c30003a20 0x7f7c30003e40 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009d40 tx=0x7f7c300058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6407f9e4ff305bc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40009e90 con 0x7f7c30003a20 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4000e9a0 con 0x7f7c30003a20 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c48e73700 1 -- 172.21.15.119:0/3437217251 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4000ed10 con 0x7f7c30003a20 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3437217251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c30003a20 msgr2=0x7f7c30003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c30003a20 0x7f7c30003e40 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009d40 tx=0x7f7c300058a0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3437217251 shutdown_connections 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3437217251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c30003a20 0x7f7c30003e40 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3437217251 >> 172.21.15.119:0/3437217251 conn(0x7f7c440b4ba0 msgr2=0x7f7c440b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3437217251 shutdown_connections 2024-09-13T16:34:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.565+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3437217251 wait complete. 2024-09-13T16:34:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 Processor -- start 2024-09-13T16:34:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 -- start start 2024-09-13T16:34:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30003a20 0x7f7c4415c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c4414a4f0 0x7f7c4415c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4415ba90 0x7f7c4415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c44151ec0 con 0x7f7c4414a4f0 2024-09-13T16:34:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c441512d0 con 0x7f7c4415ba90 2024-09-13T16:34:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51336e700 1 Processor -- start 2024-09-13T16:34:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c44151590 con 0x7f7c30003a20 2024-09-13T16:34:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51336e700 1 -- start start 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c4414a4f0 0x7f7c4415c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c4414a4f0 0x7f7c4415c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55580/0 (socket says 172.21.15.119:55580) 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 -- 172.21.15.119:0/3600137898 learned_addr learned my addr 172.21.15.119:0/3600137898 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4a676700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4415ba90 0x7f7c4415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30003a20 0x7f7c4415c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 -- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30003a20 msgr2=0x7f7c4415c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30003a20 0x7f7c4415c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 -- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4415ba90 msgr2=0x7f7c4415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4415ba90 0x7f7c4415cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 -- 172.21.15.119:0/3600137898 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c44151870 con 0x7f7c4414a4f0 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4a676700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4415ba90 0x7f7c4415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c49674700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c4414a4f0 0x7f7c4415c960 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400dae0 tx=0x7f7c3400dea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c3a7fc700 1 -- 172.21.15.119:0/3600137898 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34004ca0 con 0x7f7c4414a4f0 2024-09-13T16:34:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c3a7fc700 1 -- 172.21.15.119:0/3600137898 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c34004e00 con 0x7f7c4414a4f0 2024-09-13T16:34:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c44151b50 con 0x7f7c4414a4f0 2024-09-13T16:34:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c3a7fc700 1 -- 172.21.15.119:0/3600137898 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c340072d0 con 0x7f7c4414a4f0 2024-09-13T16:34:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4414ee30 con 0x7f7c4414a4f0 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51336e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc50c0b3a50 0x7fc50c0703f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51336e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc50c15d070 0x7fc50c0b1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51336e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c15d5e0 con 0x7fc50c0ce9c0 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51336e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc50c0b3e30 con 0x7fc50c15d070 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51336e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc50c0ac8c0 con 0x7fc50c0b3a50 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51110a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc50c0b3a50 0x7fc50c0703f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc510909700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc50c15d070 0x7fc50c0b1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51110a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc50c0b3a50 0x7fc50c0703f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:56198/0 (socket says 172.21.15.119:56198) 2024-09-13T16:34:31.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51190b700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fc50c0ce9c0 0x7fc50c0acd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:58774/0 (socket says 172.21.15.119:58774) 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.569+0000 7fc51110a700 1 -- 172.21.15.119:0/722840170 learned_addr learned my addr 172.21.15.119:0/722840170 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc510909700 1 -- 172.21.15.119:0/722840170 >> v2:172.21.15.145:3300/0 conn(0x7fc50c0b3a50 msgr2=0x7fc50c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc510909700 1 --2- 172.21.15.119:0/722840170 >> v2:172.21.15.145:3300/0 conn(0x7fc50c0b3a50 0x7fc50c0703f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc510909700 1 -- 172.21.15.119:0/722840170 >> v1:172.21.15.145:6789/0 conn(0x7fc50c0ce9c0 legacy=0x7fc50c0acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc510909700 1 -- 172.21.15.119:0/722840170 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c15d8c0 con 0x7fc50c15d070 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc510909700 1 --2- 172.21.15.119:0/722840170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15d070 0x7fc50c0b1aa0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0724b0 tx=0x7fc4fc009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe34df4e09918f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc503fff700 1 -- 172.21.15.119:0/722840170 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc011830 con 0x7fc50c15d070 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc503fff700 1 -- 172.21.15.119:0/722840170 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4fc011990 con 0x7fc50c15d070 2024-09-13T16:34:31.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc503fff700 1 -- 172.21.15.119:0/722840170 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc011d20 con 0x7fc50c15d070 2024-09-13T16:34:31.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7f7c3a7fc700 1 -- 172.21.15.119:0/3600137898 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7c34007430 con 0x7f7c4414a4f0 2024-09-13T16:34:31.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- 172.21.15.119:0/722840170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15d070 msgr2=0x7fc50c0b1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 --2- 172.21.15.119:0/722840170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15d070 0x7fc50c0b1aa0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0724b0 tx=0x7fc4fc009b30 comp rx=0 tx=0).stop 2024-09-13T16:34:31.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7f7c3a7fc700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c28066730 0x7f7c28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- 172.21.15.119:0/722840170 shutdown_connections 2024-09-13T16:34:31.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 --2- 172.21.15.119:0/722840170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15d070 0x7fc50c0b1aa0 secure :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fc50c0724b0 tx=0x7fc4fc009b30 comp rx=0 tx=0).stop 2024-09-13T16:34:31.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 --2- 172.21.15.119:0/722840170 >> v2:172.21.15.145:3300/0 conn(0x7fc50c0b3a50 0x7fc50c0703f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- 172.21.15.119:0/722840170 >> 172.21.15.119:0/722840170 conn(0x7fc50c0b0560 msgr2=0x7fc50c0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- 172.21.15.119:0/722840170 shutdown_connections 2024-09-13T16:34:31.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- 172.21.15.119:0/722840170 wait complete. 2024-09-13T16:34:31.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7f7c3a7fc700 1 -- 172.21.15.119:0/3600137898 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7c34083e50 con 0x7f7c4414a4f0 2024-09-13T16:34:31.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c28066730 0x7f7c28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7f7c49e75700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c28066730 0x7f7c28068bf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7c400090b0 tx=0x7f7c40008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 Processor -- start 2024-09-13T16:34:31.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- start start 2024-09-13T16:34:31.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc50c0b3a50 0x7fc50c156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc50c0ce9c0 0x7fc50c144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15be90 0x7fc50c15c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c15c880 con 0x7fc50c15be90 2024-09-13T16:34:31.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c157140 con 0x7fc50c0b3a50 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51336e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c159510 con 0x7fc50c0ce9c0 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51190b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15be90 0x7fc50c15c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51190b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15be90 0x7fc50c15c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55598/0 (socket says 172.21.15.119:55598) 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.573+0000 7fc51190b700 1 -- 172.21.15.119:0/2082818802 learned_addr learned my addr 172.21.15.119:0/2082818802 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc510909700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc50c0ce9c0 0x7fc50c144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51110a700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc50c0b3a50 0x7fc50c156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51190b700 1 -- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc50c0ce9c0 msgr2=0x7fc50c144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51190b700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc50c0ce9c0 0x7fc50c144160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51190b700 1 -- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc50c0b3a50 msgr2=0x7fc50c156490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51190b700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc50c0b3a50 0x7fc50c156490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51190b700 1 -- 172.21.15.119:0/2082818802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4fc0094b0 con 0x7fc50c15be90 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51190b700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc50c15be90 0x7fc50c15c340 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fc50400cbc0 tx=0x7fc50400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc50400ab50 con 0x7fc50c15be90 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc50400acb0 con 0x7fc50c15be90 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51336e700 1 -- 172.21.15.119:0/2082818802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50c1597f0 con 0x7fc50c15be90 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc504007270 con 0x7fc50c15be90 2024-09-13T16:34:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51336e700 1 -- 172.21.15.119:0/2082818802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50c1576e0 con 0x7fc50c15be90 2024-09-13T16:34:31.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 Processor -- start 2024-09-13T16:34:31.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 -- start start 2024-09-13T16:34:31.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc51336e700 1 -- 172.21.15.119:0/2082818802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fc50c071fc0 con 0x7fc50c15be90 2024-09-13T16:34:31.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29b51f0 0x29d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b55d0 0x298bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x298c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x28fd500 con 0x298c2b0 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x28fac90 con 0x29b51f0 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x28f6580 con 0x29b55d0 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x298c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x298c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55612/0 (socket says 172.21.15.119:55612) 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 -- 172.21.15.119:0/820907266 learned_addr learned my addr 172.21.15.119:0/820907266 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c49674700 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b55d0 0x298bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c49e75700 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29b51f0 0x29d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 -- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b55d0 msgr2=0x298bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b55d0 0x298bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 -- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29b51f0 msgr2=0x29d2730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29b51f0 0x29d2730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 -- 172.21.15.119:0/820907266 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x27e8ee0 con 0x298c2b0 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c49e75700 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29b51f0 0x29d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4a676700 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x298c690 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7c440b1f10 tx=0x7f7c3c001d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=683d3bbff74838ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c277fe700 1 -- 172.21.15.119:0/820907266 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c005170 con 0x298c2b0 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c277fe700 1 -- 172.21.15.119:0/820907266 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3c0052d0 con 0x298c2b0 2024-09-13T16:34:31.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c277fe700 1 -- 172.21.15.119:0/820907266 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c005660 con 0x298c2b0 2024-09-13T16:34:31.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc5040073d0 con 0x7fc50c15be90 2024-09-13T16:34:31.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc501ffb700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4f0066730 0x7fc4f0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 -- 172.21.15.119:0/820907266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 msgr2=0x298c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x298c690 secure :-1 s=READY pgs=1652 cs=0 l=1 rev1=1 crypto rx=0x7f7c440b1f10 tx=0x7f7c3c001d40 comp rx=0 tx=0).stop 2024-09-13T16:34:31.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 -- 172.21.15.119:0/820907266 shutdown_connections 2024-09-13T16:34:31.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc504084a50 con 0x7fc50c15be90 2024-09-13T16:34:31.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x298c690 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b55d0 0x298bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/820907266 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x29b51f0 0x29d2730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.577+0000 7f7c4f38a740 1 -- 172.21.15.119:0/820907266 >> 172.21.15.119:0/820907266 conn(0x271d780 msgr2=0x29b7c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- 172.21.15.119:0/820907266 shutdown_connections 2024-09-13T16:34:31.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- 172.21.15.119:0/820907266 wait complete. 2024-09-13T16:34:31.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7fc51110a700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4f0066730 0x7fc4f0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+511 (secure 0 0 0) 0x7fc5040852d0 con 0x7fc50c15be90 2024-09-13T16:34:31.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7fc51336e700 1 -- 172.21.15.119:0/2082818802 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7fc50c0703f0 con 0x7fc50c15be90 2024-09-13T16:34:31.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7fc51110a700 1 --2- 172.21.15.119:0/2082818802 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc4f0066730 0x7fc4f0068bf0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fc50800fca0 tx=0x7fc508013040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- start start 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 Processor -- start 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x29df370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2990350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29b55d0 0x2990890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x28b9920 con 0x29b55d0 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x28b95c0 con 0x298c2b0 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x288be10 con 0x29b51f0 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 10 client.0 _ll_register_callbacks cb 0x28efbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29b55d0 0x2990890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29b55d0 0x2990890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55618/0 (socket says 172.21.15.119:55618) 2024-09-13T16:34:31.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 -- 172.21.15.119:0/1373743869 learned_addr learned my addr 172.21.15.119:0/1373743869 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c49e75700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x29df370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c49674700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2990350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 -- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 msgr2=0x2990350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2990350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 -- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 msgr2=0x29df370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x29df370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 -- 172.21.15.119:0/1373743869 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c3c00f040 con 0x29b55d0 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c49674700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2990350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4a676700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29b55d0 0x2990890 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7c440adb50 tx=0x7f7c3c001510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=181c4b0ff2973d02 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1373743869 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c003a20 con 0x29b55d0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1373743869 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7c3c005860 con 0x29b55d0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x272a490 con 0x29b55d0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1373743869 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c3c014aa0 con 0x29b55d0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 10 client.9171 fetch_fsmap learned FSMap version 76 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x28bc600 con 0x29b55d0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1373743869 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x272a490 con 0x29b55d0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1373743869 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7c3c014c00 con 0x29b55d0 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 10 client.9171 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 10 client.9171 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 20 client.9171 populate_metadata read hostname 'smithi119' 2024-09-13T16:34:31.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29dcd00 0x29dd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 4 client.9171 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x2774710 con 0x29dcd00 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c49e75700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29dcd00 0x29dd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c24ff9700 10 client.9171 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.581+0000 7f7c49e75700 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29dcd00 0x29dd110 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64261e70f49a4ea3 server_cookie=b16e8f75dd619d4d in_seq=0 out_seq=0 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1373743869 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x2774710 con 0x29dcd00 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c24ff9700 10 client.9171 handle_command_reply: tid=0 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 client.9171 shutdown 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29b55d0 msgr2=0x2990890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29b55d0 0x2990890 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7f7c440adb50 tx=0x7f7c3c001510 comp rx=0 tx=0).stop 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 shutdown_connections 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29dcd00 0x29dd110 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x29b55d0 0x2990890 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c24ff9700 0 client.9171 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2990350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1373743869 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x29df370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 >> 172.21.15.119:0/1373743869 conn(0x271d780 msgr2=0x29d2f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 shutdown_connections 2024-09-13T16:34:31.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1373743869 wait complete. 2024-09-13T16:34:31.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.585+0000 7f7c4f38a740 20 client.9171 trim_cache size 0 max 16384 2024-09-13T16:34:31.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.601+0000 7f7c4f38a740 1 Processor -- start 2024-09-13T16:34:31.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.601+0000 7f7c4f38a740 1 -- start start 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x271d780 0x298b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x29e9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x29ea2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x28b0010 con 0x298c2b0 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x28b63c0 con 0x271d780 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x294df30 con 0x29b51f0 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x29e9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x29e9d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55628/0 (socket says 172.21.15.119:55628) 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 -- 172.21.15.119:0/2585848964 learned_addr learned my addr 172.21.15.119:0/2585848964 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49e75700 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x271d780 0x298b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 -- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 msgr2=0x29ea2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4a676700 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x29ea2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x29ea2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 -- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x271d780 msgr2=0x298b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x271d780 0x298b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 -- 172.21.15.119:0/2585848964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x28a2830 con 0x298c2b0 2024-09-13T16:34:31.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49e75700 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x271d780 0x298b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:34:31.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c49674700 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x29e9d90 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7c3405cca0 tx=0x7f7c34054810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=587ca3a9dbc8ae81 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c27fff700 1 -- 172.21.15.119:0/2585848964 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c340554e0 con 0x298c2b0 2024-09-13T16:34:31.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c27fff700 1 -- 172.21.15.119:0/2585848964 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c34054de0 con 0x298c2b0 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c27fff700 1 -- 172.21.15.119:0/2585848964 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c34058c10 con 0x298c2b0 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- 172.21.15.119:0/2585848964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 msgr2=0x29e9d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x29e9d90 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7f7c3405cca0 tx=0x7f7c34054810 comp rx=0 tx=0).stop 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- 172.21.15.119:0/2585848964 shutdown_connections 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x29ea2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x298c2b0 0x29e9d90 unknown :-1 s=CLOSED pgs=1654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/2585848964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x271d780 0x298b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- 172.21.15.119:0/2585848964 >> 172.21.15.119:0/2585848964 conn(0x29b4e10 msgr2=0x29e63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- 172.21.15.119:0/2585848964 shutdown_connections 2024-09-13T16:34:31.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- 172.21.15.119:0/2585848964 wait complete. 2024-09-13T16:34:31.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 -- start start 2024-09-13T16:34:31.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.605+0000 7f7c4f38a740 1 Processor -- start 2024-09-13T16:34:31.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x271d780 0x29f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x2a370b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2a375f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2a37b30 con 0x271d780 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x29ee180 con 0x298c2b0 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x29f4120 con 0x29b51f0 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x271d780 0x29f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 10 client.0 _ll_register_callbacks cb 0x28f5150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x271d780 0x29f36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55644/0 (socket says 172.21.15.119:55644) 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 -- 172.21.15.119:0/1684337729 learned_addr learned my addr 172.21.15.119:0/1684337729 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49674700 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x2a370b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4a676700 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2a375f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 -- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 msgr2=0x2a375f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2a375f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 -- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 msgr2=0x2a370b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x2a370b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 -- 172.21.15.119:0/1684337729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c4414ee30 con 0x271d780 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49e75700 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x271d780 0x29f36d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f7c40016000 tx=0x7f7c40004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6258d8d964a766d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c24ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1684337729 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40021070 con 0x271d780 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1684337729 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f7c400044e0 con 0x271d780 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7c3c0052d0 con 0x271d780 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1684337729 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7c40034320 con 0x271d780 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 10 client.9177 fetch_fsmap learned FSMap version 76 2024-09-13T16:34:31.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x2768c10 con 0x271d780 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1684337729 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3c0052d0 con 0x271d780 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1684337729 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f7c40034040 con 0x271d780 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 10 client.9177 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 10 client.9177 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 20 client.9177 populate_metadata read hostname 'smithi119' 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29b55d0 0x29ec5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 4 client.9177 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x279b5f0 con 0x29b55d0 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.609+0000 7f7c49674700 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29b55d0 0x29ec5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c24ff9700 10 client.9177 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c49674700 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29b55d0 0x29ec5e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c26ec136e0e6152e server_cookie=df96b8d9c22c72da in_seq=0 out_seq=0 2024-09-13T16:34:31.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c24ff9700 1 -- 172.21.15.119:0/1684337729 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x279b5f0 con 0x29b55d0 2024-09-13T16:34:31.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c24ff9700 10 client.9177 handle_command_reply: tid=0 2024-09-13T16:34:31.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 client.9177 shutdown 2024-09-13T16:34:31.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x271d780 msgr2=0x29f36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x271d780 0x29f36d0 secure :-1 s=READY pgs=1655 cs=0 l=1 rev1=1 crypto rx=0x7f7c40016000 tx=0x7f7c40004770 comp rx=0 tx=0).stop 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 shutdown_connections 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x29b55d0 0x29ec5e0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c24ff9700 0 client.9177 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x29b51f0 0x2a375f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x298c2b0 0x2a370b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 --2- 172.21.15.119:0/1684337729 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x271d780 0x29f36d0 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 >> 172.21.15.119:0/1684337729 conn(0x29b4e10 msgr2=0x298a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 shutdown_connections 2024-09-13T16:34:31.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 1 -- 172.21.15.119:0/1684337729 wait complete. 2024-09-13T16:34:31.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4f38a740 20 client.9177 trim_cache size 0 max 16384 2024-09-13T16:34:31.623 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:34:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:34:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "457309f2-1a77-4742-bbbf-2f31b549bf6d", 2024-09-13T16:34:31.623 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:34:31.623 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:34:31.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c28066730 msgr2=0x7f7c28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c28066730 0x7f7c28068bf0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f7c400090b0 tx=0x7f7c40008030 comp rx=0 tx=0).stop 2024-09-13T16:34:31.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c4414a4f0 msgr2=0x7f7c4415c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c4414a4f0 0x7f7c4415c960 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7f7c3400dae0 tx=0x7f7c3400dea0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 shutdown_connections 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7c28066730 0x7f7c28068bf0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7c4415ba90 0x7f7c4415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7c4414a4f0 0x7f7c4415c960 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 --2- 172.21.15.119:0/3600137898 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7c30003a20 0x7f7c4415c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 >> 172.21.15.119:0/3600137898 conn(0x7f7c440b4ba0 msgr2=0x7f7c4406d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 shutdown_connections 2024-09-13T16:34:31.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.613+0000 7f7c4ae77700 1 -- 172.21.15.119:0/3600137898 wait complete. 2024-09-13T16:34:31.633 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-09-13T16:34:31.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffab2081700 1 Processor -- start 2024-09-13T16:34:31.751 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffab2081700 1 -- start start 2024-09-13T16:34:31.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffab2081700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0adf60 0x7ffaac0b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.752 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffab2081700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaac0ae340 con 0x7ffaac159ca0 2024-09-13T16:34:31.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffab2081700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaac15b1d0 con 0x7ffaac070d40 2024-09-13T16:34:31.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffab2081700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac15ae40 con 0x7ffaac0adf60 2024-09-13T16:34:31.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaa2ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ffaac159ca0 0x7ffaac070630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33144/0 (socket says 172.21.15.119:33144) 2024-09-13T16:34:31.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/1737816947 learned_addr learned my addr 172.21.15.119:0/1737816947 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:34:31.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/1737816947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0adf60 0x7ffaac0b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaaaffd700 1 -- 172.21.15.119:0/1737816947 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956696779 0 0) 0x7ffaac0ae340 con 0x7ffaac159ca0 2024-09-13T16:34:31.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaaaffd700 1 -- 172.21.15.119:0/1737816947 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa980035d0 con 0x7ffaac159ca0 2024-09-13T16:34:31.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaaaffd700 1 -- 172.21.15.119:0/1737816947 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2672732901 0 0) 0x7ffaac15b1d0 con 0x7ffaac070d40 2024-09-13T16:34:31.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaaaffd700 1 -- 172.21.15.119:0/1737816947 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaac0ae340 con 0x7ffaac070d40 2024-09-13T16:34:31.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaab7fe700 1 -- 172.21.15.119:0/1737816947 >> v1:172.21.15.145:6789/0 conn(0x7ffaac070d40 legacy=0x7ffaac15b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaab7fe700 1 -- 172.21.15.119:0/1737816947 >> v1:172.21.15.119:6789/0 conn(0x7ffaac159ca0 legacy=0x7ffaac070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaab7fe700 1 -- 172.21.15.119:0/1737816947 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaac15a5a0 con 0x7ffaac0adf60 2024-09-13T16:34:31.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/1737816947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0adf60 0x7ffaac0b3450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009880 tx=0x7ffa9c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ec537739064f2d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaaaffd700 1 -- 172.21.15.119:0/1737816947 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c011830 con 0x7ffaac0adf60 2024-09-13T16:34:31.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaaaffd700 1 -- 172.21.15.119:0/1737816947 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c011990 con 0x7ffaac0adf60 2024-09-13T16:34:31.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.749+0000 7ffaaaffd700 1 -- 172.21.15.119:0/1737816947 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c011d20 con 0x7ffaac0adf60 2024-09-13T16:34:31.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 -- 172.21.15.119:0/1737816947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0adf60 msgr2=0x7ffaac0b3450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 --2- 172.21.15.119:0/1737816947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0adf60 0x7ffaac0b3450 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009880 tx=0x7ffa9c009b90 comp rx=0 tx=0).stop 2024-09-13T16:34:31.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 -- 172.21.15.119:0/1737816947 shutdown_connections 2024-09-13T16:34:31.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 --2- 172.21.15.119:0/1737816947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac0adf60 0x7ffaac0b3450 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 -- 172.21.15.119:0/1737816947 >> 172.21.15.119:0/1737816947 conn(0x7ffaac0b0560 msgr2=0x7ffaac0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 -- 172.21.15.119:0/1737816947 shutdown_connections 2024-09-13T16:34:31.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 -- 172.21.15.119:0/1737816947 wait complete. 2024-09-13T16:34:31.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 Processor -- start 2024-09-13T16:34:31.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.753+0000 7ffab2081700 1 -- start start 2024-09-13T16:34:31.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac070d40 0x7ffaac0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac0adf60 0x7ffaac152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac159ca0 0x7ffaac14ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac0cec30 con 0x7ffaac0adf60 2024-09-13T16:34:31.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac1f51f0 con 0x7ffaac159ca0 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffaac0ce290 con 0x7ffaac070d40 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac0adf60 0x7ffaac152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac0adf60 0x7ffaac152750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55656/0 (socket says 172.21.15.119:55656) 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/2406900718 learned_addr learned my addr 172.21.15.119:0/2406900718 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac070d40 0x7ffaac0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaabfff700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac159ca0 0x7ffaac14ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac070d40 msgr2=0x7ffaac0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac070d40 0x7ffaac0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac159ca0 msgr2=0x7ffaac14ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac159ca0 0x7ffaac14ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/2406900718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0094b0 con 0x7ffaac0adf60 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac0adf60 0x7ffaac152750 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400c4d0 tx=0x7ffa9400c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa8ff9700 1 -- 172.21.15.119:0/2406900718 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94005d90 con 0x7ffaac0adf60 2024-09-13T16:34:31.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa8ff9700 1 -- 172.21.15.119:0/2406900718 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa940070d0 con 0x7ffaac0adf60 2024-09-13T16:34:31.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaac0ce570 con 0x7ffaac0adf60 2024-09-13T16:34:31.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffaa8ff9700 1 -- 172.21.15.119:0/2406900718 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94007460 con 0x7ffaac0adf60 2024-09-13T16:34:31.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.757+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac152c90 con 0x7ffaac0adf60 2024-09-13T16:34:31.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa8ff9700 1 -- 172.21.15.119:0/2406900718 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ffa940075c0 con 0x7ffaac0adf60 2024-09-13T16:34:31.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa8ff9700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa8c066730 0x7ffa8c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa8ff9700 1 -- 172.21.15.119:0/2406900718 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ffa94083600 con 0x7ffaac0adf60 2024-09-13T16:34:31.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 Processor -- start 2024-09-13T16:34:31.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 -- start start 2024-09-13T16:34:31.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa8c066730 0x7ffa8c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa8c066730 0x7ffa8c068bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0090b0 tx=0x7ffa9c01c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1225950 0x1204240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x1225d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1226270 0x11d95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x116d010 con 0x1203d00 2024-09-13T16:34:31.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x10faf40 con 0x1225950 2024-09-13T16:34:31.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x114de30 con 0x1226270 2024-09-13T16:34:31.766 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x1225d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaab7fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1225950 0x1204240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x1225d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55664/0 (socket says 172.21.15.119:55664) 2024-09-13T16:34:31.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/1136022356 learned_addr learned my addr 172.21.15.119:0/1136022356 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaabfff700 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1226270 0x11d95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/1136022356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1226270 msgr2=0x11d95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1226270 0x11d95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/1136022356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1225950 msgr2=0x1204240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1225950 0x1204240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.761+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/1136022356 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xfb9a40 con 0x1203d00 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x1225d30 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7ffa9405c420 tx=0x7ffa94054120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f2e43c17040aee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffaa17fa700 1 -- 172.21.15.119:0/1136022356 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94058170 con 0x1203d00 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffaa17fa700 1 -- 172.21.15.119:0/1136022356 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa94054c60 con 0x1203d00 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffaa17fa700 1 -- 172.21.15.119:0/1136022356 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94054560 con 0x1203d00 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- 172.21.15.119:0/1136022356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 msgr2=0x1225d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x1225d30 secure :-1 s=READY pgs=1658 cs=0 l=1 rev1=1 crypto rx=0x7ffa9405c420 tx=0x7ffa94054120 comp rx=0 tx=0).stop 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- 172.21.15.119:0/1136022356 shutdown_connections 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1226270 0x11d95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x1225d30 unknown :-1 s=CLOSED pgs=1658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 --2- 172.21.15.119:0/1136022356 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1225950 0x1204240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- 172.21.15.119:0/1136022356 >> 172.21.15.119:0/1136022356 conn(0xf6e5b0 msgr2=0x11df9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- 172.21.15.119:0/1136022356 shutdown_connections 2024-09-13T16:34:31.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- 172.21.15.119:0/1136022356 wait complete. 2024-09-13T16:34:31.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- start start 2024-09-13T16:34:31.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 Processor -- start 2024-09-13T16:34:31.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x12036f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x1223c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1202840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1149150 con 0x1203d00 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1146eb0 con 0x1226270 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.765+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x11048e0 con 0x1225950 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x12036f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 10 client.0 _ll_register_callbacks cb 0x10e3670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x12036f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55668/0 (socket says 172.21.15.119:55668) 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2274882635 learned_addr learned my addr 172.21.15.119:0/2274882635 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaabfff700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1202840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x1223c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 msgr2=0x1223c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x1223c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 msgr2=0x1202840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1202840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2274882635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaac152c90 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaabfff700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1202840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x1223c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x12036f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c013040 tx=0x7ffa9c00e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6290aa3c84f05bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/2274882635 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c02c040 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/2274882635 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ffa9c01f070 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xf7b2c0 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/2274882635 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa9c024ea0 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 10 client.9189 fetch_fsmap learned FSMap version 76 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x110d290 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/2274882635 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xf7b2c0 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/2274882635 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ffa9c01a800 con 0x1203d00 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 10 client.9189 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:34:31.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 10 client.9189 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:34:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 20 client.9189 populate_metadata read hostname 'smithi119' 2024-09-13T16:34:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x1200010 0x12221f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 4 client.9189 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:34:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xfc5540 con 0x1200010 2024-09-13T16:34:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x1200010 0x12221f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 10 client.9189 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x1200010 0x12221f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c97a12e2b4f6bc20 server_cookie=9e7f34afe8bdb8ff in_seq=0 out_seq=0 2024-09-13T16:34:31.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/2274882635 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0xfc5540 con 0x1200010 2024-09-13T16:34:31.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffa8a7fc700 10 client.9189 handle_command_reply: tid=0 2024-09-13T16:34:31.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 1 client.9189 shutdown 2024-09-13T16:34:31.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 msgr2=0x12036f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.769+0000 7ffab5332740 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x12036f0 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c013040 tx=0x7ffa9c00e000 comp rx=0 tx=0).stop 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 shutdown_connections 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x1200010 0x12221f0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1202840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffa8a7fc700 0 client.9189 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x1223c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 --2- 172.21.15.119:0/2274882635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x12036f0 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 >> 172.21.15.119:0/2274882635 conn(0xf6e5b0 msgr2=0x11de6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 shutdown_connections 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 1 -- 172.21.15.119:0/2274882635 wait complete. 2024-09-13T16:34:31.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.773+0000 7ffab5332740 20 client.9189 trim_cache size 0 max 16384 2024-09-13T16:34:31.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 Processor -- start 2024-09-13T16:34:31.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- start start 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x121ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x12204f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1220a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x11df1c0 con 0x1203d00 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xff2b80 con 0x1226270 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x10d1630 con 0x1225950 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x121ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x121ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55682/0 (socket says 172.21.15.119:55682) 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2283639764 learned_addr learned my addr 172.21.15.119:0/2283639764 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaabfff700 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1220a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 msgr2=0x12204f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x12204f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x12204f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 msgr2=0x1220a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1220a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 -- 172.21.15.119:0/2283639764 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x10f34c0 con 0x1203d00 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x12204f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x121ffb0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c011690 tx=0x7ffa9c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2efaebc4fb2ea96f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaa1ffb700 1 -- 172.21.15.119:0/2283639764 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c02c070 con 0x1203d00 2024-09-13T16:34:31.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaa1ffb700 1 -- 172.21.15.119:0/2283639764 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa9c01a600 con 0x1203d00 2024-09-13T16:34:31.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffaa1ffb700 1 -- 172.21.15.119:0/2283639764 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c01a970 con 0x1203d00 2024-09-13T16:34:31.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- 172.21.15.119:0/2283639764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 msgr2=0x121ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x121ffb0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c011690 tx=0x7ffa9c009240 comp rx=0 tx=0).stop 2024-09-13T16:34:31.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- 172.21.15.119:0/2283639764 shutdown_connections 2024-09-13T16:34:31.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x1220a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1225950 0x12204f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 --2- 172.21.15.119:0/2283639764 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1203d00 0x121ffb0 unknown :-1 s=CLOSED pgs=1660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- 172.21.15.119:0/2283639764 >> 172.21.15.119:0/2283639764 conn(0xf6e5b0 msgr2=0x122e690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- 172.21.15.119:0/2283639764 shutdown_connections 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- 172.21.15.119:0/2283639764 wait complete. 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 -- start start 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.793+0000 7ffab5332740 1 Processor -- start 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1203d00 0x1248480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1225950 0x128b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x128ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1236530 con 0x1225950 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x12204c0 con 0x1226270 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1236760 con 0x1203d00 2024-09-13T16:34:31.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 10 client.0 _ll_register_callbacks cb 0x11168f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1225950 0x128b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1225950 0x128b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55688/0 (socket says 172.21.15.119:55688) 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaabfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x128ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/736481883 learned_addr learned my addr 172.21.15.119:0/736481883 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1203d00 0x1248480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1203d00 msgr2=0x1248480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1203d00 0x1248480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 msgr2=0x128ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x128ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 -- 172.21.15.119:0/736481883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x110d290 con 0x1225950 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaabfff700 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x128ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaa2ffd700 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1225950 0x128b4f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7ffa9405c420 tx=0x7ffa94001150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b722f65dd4301a37 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:34:31.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffa8a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/736481883 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa94001250 con 0x1225950 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/736481883 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ffa94058390 con 0x1225950 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffa94054c60 con 0x1225950 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/736481883 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffa9405c950 con 0x1225950 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 10 client.9195 fetch_fsmap learned FSMap version 76 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0x10e7790 con 0x1225950 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/736481883 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa94054c60 con 0x1225950 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/736481883 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7ffa9405a050 con 0x1225950 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 10 client.9195 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 10 client.9195 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 20 client.9195 populate_metadata read hostname 'smithi119' 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x128bf70 0x128c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:34:31.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 4 client.9195 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0xfea2c0 con 0x128bf70 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x128bf70 0x128c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffa8a7fc700 10 client.9195 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.797+0000 7ffaab7fe700 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x128bf70 0x128c380 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=19bc365240344766 server_cookie=854f9d8fc6a64e44 in_seq=0 out_seq=0 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffa8a7fc700 1 -- 172.21.15.119:0/736481883 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0xfea2c0 con 0x128bf70 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffa8a7fc700 10 client.9195 handle_command_reply: tid=0 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 client.9195 shutdown 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1225950 msgr2=0x128b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1225950 0x128b4f0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7ffa9405c420 tx=0x7ffa94001150 comp rx=0 tx=0).stop 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 shutdown_connections 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x128bf70 0x128c380 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1226270 0x128ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffa8a7fc700 0 client.9195 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1225950 0x128b4f0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 --2- 172.21.15.119:0/736481883 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1203d00 0x1248480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 >> 172.21.15.119:0/736481883 conn(0xf6e5b0 msgr2=0x1235880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 shutdown_connections 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 1 -- 172.21.15.119:0/736481883 wait complete. 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab5332740 20 client.9195 trim_cache size 0 max 16384 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:34:31.809 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "457309f2-1a77-4742-bbbf-2f31b549bf6d": { 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "457309f2-1a77-4742-bbbf-2f31b549bf6d", 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,repair" 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:34:31.810 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:34:31.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa8c066730 msgr2=0x7ffa8c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa8c066730 0x7ffa8c068bf0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c0090b0 tx=0x7ffa9c01c040 comp rx=0 tx=0).stop 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac0adf60 msgr2=0x7ffaac152750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac0adf60 0x7ffaac152750 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7ffa9400c4d0 tx=0x7ffa9400c7e0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 shutdown_connections 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa8c066730 0x7ffa8c068bf0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffaac159ca0 0x7ffaac14ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffaac0adf60 0x7ffaac152750 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 --2- 172.21.15.119:0/2406900718 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffaac070d40 0x7ffaac0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 >> 172.21.15.119:0/2406900718 conn(0x7ffaac0b0560 msgr2=0x7ffaac0afc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 shutdown_connections 2024-09-13T16:34:31.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:31.801+0000 7ffab2081700 1 -- 172.21.15.119:0/2406900718 wait complete. 2024-09-13T16:34:31.820 INFO:tasks.cephfs.filesystem:scrub status for tag:457309f2-1a77-4742-bbbf-2f31b549bf6d - {'path': '/', 'tag': '457309f2-1a77-4742-bbbf-2f31b549bf6d', 'options': 'recursive,repair'} 2024-09-13T16:34:33.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:33.193+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 7 ==== log(4 entries from seq 1 at 2024-09-13T16:34:31.615832+0000) v1 ==== 695+0+0 (secure 0 0 0) 0x7fc50400ae20 con 0x7fc50c15be90 2024-09-13T16:34:33.197 INFO:teuthology.orchestra.run.smithi119.stdout: cluster: 2024-09-13T16:34:33.197 INFO:teuthology.orchestra.run.smithi119.stdout: id: 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:34:33.197 INFO:teuthology.orchestra.run.smithi119.stdout: health: HEALTH_OK 2024-09-13T16:34:33.197 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:33.197 INFO:teuthology.orchestra.run.smithi119.stdout: services: 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: mon: 3 daemons, quorum a,b,c (age 9m) 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: mgr: x(active, since 9m), standbys: y 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: mds: 1/1 daemons up, 3 standby 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 9m) 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: data: 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: volumes: 1/1 healthy 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: pools: 3 pools, 97 pgs 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: objects: 30 objects, 19 MiB 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: usage: 291 MiB used, 720 GiB / 720 GiB avail 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: pgs: 97 active+clean 2024-09-13T16:34:33.198 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout: io: 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout: client: 85 B/s rd, 1.0 MiB/s wr, 0 op/s rd, 2 op/s wr 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:34:31.615832+0000 mds.c [INF] scrub queued for path: / 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:34:31.615844+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:34:31.615853+0000 mds.c [INF] scrub summary: active paths [/] 2024-09-13T16:34:33.199 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:34:31.618041+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-09-13T16:34:35.201 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:34:34.533461+0000 mds.c [INF] scrub summary: idle 2024-09-13T16:34:35.201 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:34:35.196+0000 7fc501ffb700 1 -- 172.21.15.119:0/2082818802 <== mon.0 v2:172.21.15.119:3300/0 8 ==== log(1 entries from seq 5 at 2024-09-13T16:34:34.533461+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc504055a60 con 0x7fc50c15be90 2024-09-13T16:34:58.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:34:58.308 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:35:01.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-09-13T16:35:01.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.940+0000 7f75cef3f700 1 Processor -- start 2024-09-13T16:35:01.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.940+0000 7f75cef3f700 1 -- start start 2024-09-13T16:35:01.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f75c8144180 0x7f75c806c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f75c80cc010 0x7f75c80cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75c806cb40 con 0x7f75c80ce000 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c80b1c50 con 0x7f75c80cc010 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75c806ce20 con 0x7f75c8144180 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f75c80cc010 0x7f75c80cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f75c80cc010 0x7f75c80cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35776/0 (socket says 172.21.15.119:35776) 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 -- 172.21.15.119:0/926556515 learned_addr learned my addr 172.21.15.119:0/926556515 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/926556515 >> v2:172.21.15.145:3300/0 conn(0x7f75c8144180 0x7f75c806c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 -- 172.21.15.119:0/926556515 >> v2:172.21.15.145:3300/0 conn(0x7f75c8144180 msgr2=0x7f75c806c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 --2- 172.21.15.119:0/926556515 >> v2:172.21.15.145:3300/0 conn(0x7f75c8144180 0x7f75c806c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 -- 172.21.15.119:0/926556515 >> v1:172.21.15.145:6789/0 conn(0x7f75c80ce000 legacy=0x7f75c815e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 -- 172.21.15.119:0/926556515 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c806d100 con 0x7f75c80cc010 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cd73c700 1 --2- 172.21.15.119:0/926556515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c80cc010 0x7f75c80cdac0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f75c80b1560 tx=0x7f75b8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e75513df67def78 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:01.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75ccf3b700 1 -- 172.21.15.119:0/926556515 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b800b8a0 con 0x7f75c80cc010 2024-09-13T16:35:01.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75ccf3b700 1 -- 172.21.15.119:0/926556515 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b800ba00 con 0x7f75c80cc010 2024-09-13T16:35:01.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75ccf3b700 1 -- 172.21.15.119:0/926556515 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b800bd90 con 0x7f75c80cc010 2024-09-13T16:35:01.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- 172.21.15.119:0/926556515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c80cc010 msgr2=0x7f75c80cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 --2- 172.21.15.119:0/926556515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c80cc010 0x7f75c80cdac0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7f75c80b1560 tx=0x7f75b8009bf0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- 172.21.15.119:0/926556515 shutdown_connections 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 --2- 172.21.15.119:0/926556515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c80cc010 0x7f75c80cdac0 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 --2- 172.21.15.119:0/926556515 >> v2:172.21.15.145:3300/0 conn(0x7f75c8144180 0x7f75c806c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- 172.21.15.119:0/926556515 >> 172.21.15.119:0/926556515 conn(0x7f75c80b0980 msgr2=0x7f75c80b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- 172.21.15.119:0/926556515 shutdown_connections 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.944+0000 7f75cef3f700 1 -- 172.21.15.119:0/926556515 wait complete. 2024-09-13T16:35:01.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 Processor -- start 2024-09-13T16:35:01.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 -- start start 2024-09-13T16:35:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75c80cc010 0x7f75c813d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75c80ce000 0x7f75c80aa9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c8144180 0x7f75c80aaf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c81f00e0 con 0x7f75c8144180 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75c81f10e0 con 0x7f75c80cc010 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cef3f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75c80ab530 con 0x7f75c80ce000 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75ce73e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c8144180 0x7f75c80aaf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cdf3d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75c80cc010 0x7f75c813d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cdf3d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75c80cc010 0x7f75c813d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:56432/0 (socket says 172.21.15.119:56432) 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cdf3d700 1 -- 172.21.15.119:0/628948176 learned_addr learned my addr 172.21.15.119:0/628948176 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75cd73c700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75c80ce000 0x7f75c80aa9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75ce73e700 1 -- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75c80ce000 msgr2=0x7f75c80aa9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75ce73e700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75c80ce000 0x7f75c80aa9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75ce73e700 1 -- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75c80cc010 msgr2=0x7f75c813d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75ce73e700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75c80cc010 0x7f75c813d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.948+0000 7f75ce73e700 1 -- 172.21.15.119:0/628948176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8009570 con 0x7f75c8144180 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75ce73e700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c8144180 0x7f75c80aaf00 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f75c000cae0 tx=0x7f75c000cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75be7fc700 1 -- 172.21.15.119:0/628948176 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0003c20 con 0x7f75c8144180 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75be7fc700 1 -- 172.21.15.119:0/628948176 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c0003d80 con 0x7f75c8144180 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75c81f0320 con 0x7f75c8144180 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75be7fc700 1 -- 172.21.15.119:0/628948176 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c000aa20 con 0x7f75c8144180 2024-09-13T16:35:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c81f08e0 con 0x7f75c8144180 2024-09-13T16:35:01.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75be7fc700 1 -- 172.21.15.119:0/628948176 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f75c000ab80 con 0x7f75c8144180 2024-09-13T16:35:01.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75be7fc700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75ac066730 0x7f75ac068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.952+0000 7f75be7fc700 1 -- 172.21.15.119:0/628948176 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f75c0084180 con 0x7f75c8144180 2024-09-13T16:35:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75ac066730 0x7f75ac068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75ac066730 0x7f75ac068bf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f75c80b30a0 tx=0x7f75c4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:01.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 Processor -- start 2024-09-13T16:35:01.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 -- start start 2024-09-13T16:35:01.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x108e810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x10b6520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10d8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x101f010 con 0x10d7d90 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xfacf40 con 0x10d8170 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xfffe30 con 0x108d310 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x108e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x108e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35798/0 (socket says 172.21.15.119:35798) 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 -- 172.21.15.119:0/2951455308 learned_addr learned my addr 172.21.15.119:0/2951455308 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cd73c700 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x10b6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75ce73e700 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10d8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 -- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 msgr2=0x10b6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x10b6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 -- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 msgr2=0x10d8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:01.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10d8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.956+0000 7f75cdf3d700 1 -- 172.21.15.119:0/2951455308 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xe6ba40 con 0x10d7d90 2024-09-13T16:35:01.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x108e810 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f75c4010a10 tx=0x7f75c4013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d202d0a8a599f21d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:01.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75ab7fe700 1 -- 172.21.15.119:0/2951455308 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4016900 con 0x10d7d90 2024-09-13T16:35:01.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75ab7fe700 1 -- 172.21.15.119:0/2951455308 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c401a070 con 0x10d7d90 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75ab7fe700 1 -- 172.21.15.119:0/2951455308 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4016c60 con 0x10d7d90 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- 172.21.15.119:0/2951455308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 msgr2=0x108e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x108e810 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7f75c4010a10 tx=0x7f75c4013000 comp rx=0 tx=0).stop 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- 172.21.15.119:0/2951455308 shutdown_connections 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10d8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x10b6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 --2- 172.21.15.119:0/2951455308 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x108e810 unknown :-1 s=CLOSED pgs=1664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- 172.21.15.119:0/2951455308 >> 172.21.15.119:0/2951455308 conn(0xe205b0 msgr2=0x1091b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- 172.21.15.119:0/2951455308 shutdown_connections 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- 172.21.15.119:0/2951455308 wait complete. 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- start start 2024-09-13T16:35:01.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 Processor -- start 2024-09-13T16:35:01.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x1093c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x10b2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xffb150 con 0x10d7d90 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xff8eb0 con 0x10d8170 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xfb68e0 con 0x108d310 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75d3452740 10 client.0 _ll_register_callbacks cb 0xf95670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75cd73c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x10b2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75cd73c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x10b2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35806/0 (socket says 172.21.15.119:35806) 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75cd73c700 1 -- 172.21.15.119:0/3508821832 learned_addr learned my addr 172.21.15.119:0/3508821832 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x1093c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.960+0000 7f75ce73e700 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cd73c700 1 -- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 msgr2=0x1093c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cd73c700 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x1093c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cd73c700 1 -- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 msgr2=0x10b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cd73c700 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cd73c700 1 -- 172.21.15.119:0/3508821832 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75c400e040 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cd73c700 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x10b2800 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f75b8006eb0 tx=0x7f75b80047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a7647d36d25c50 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 1 -- 172.21.15.119:0/3508821832 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b80038c0 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 1 -- 172.21.15.119:0/3508821832 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f75b80045b0 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xe2d2c0 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 1 -- 172.21.15.119:0/3508821832 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75b800b860 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 10 client.9207 fetch_fsmap learned FSMap version 76 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0xfbf290 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 1 -- 172.21.15.119:0/3508821832 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xe2d2c0 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 1 -- 172.21.15.119:0/3508821832 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f75b8003a20 con 0x10d7d90 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 10 client.9207 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 10 client.9207 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:01.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 20 client.9207 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:01.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x10b55d0 0x10e1860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 4 client.9207 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:35:01.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xe77540 con 0x10b55d0 2024-09-13T16:35:01.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x10b55d0 0x10e1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 10 client.9207 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:35:01.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x10b55d0 0x10e1860 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fae65341633268ee server_cookie=272b57a1005f6304 in_seq=0 out_seq=0 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 1 -- 172.21.15.119:0/3508821832 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0xe77540 con 0x10b55d0 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 10 client.9207 handle_command_reply: tid=0 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 client.9207 shutdown 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 msgr2=0x10b2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x10b2800 secure :-1 s=READY pgs=1665 cs=0 l=1 rev1=1 crypto rx=0x7f75b8006eb0 tx=0x7f75b80047e0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 shutdown_connections 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x10b55d0 0x10e1860 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10d8170 0x10b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d7d90 0x10b2800 unknown :-1 s=CLOSED pgs=1665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75a8ff9700 0 client.9207 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:35:01.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 --2- 172.21.15.119:0/3508821832 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x1093c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 >> 172.21.15.119:0/3508821832 conn(0xe205b0 msgr2=0x10b64b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:01.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 shutdown_connections 2024-09-13T16:35:01.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.964+0000 7f75d3452740 1 -- 172.21.15.119:0/3508821832 wait complete. 2024-09-13T16:35:01.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.968+0000 7f75d3452740 20 client.9207 trim_cache size 0 max 16384 2024-09-13T16:35:01.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 Processor -- start 2024-09-13T16:35:01.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- start start 2024-09-13T16:35:01.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x108d310 0x10ae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10d8170 0x10aec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x10ee490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xea4b80 con 0x108d310 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xfb9810 con 0x10ee0b0 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1054a30 con 0x10d8170 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x108d310 0x10ae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x108d310 0x10ae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35812/0 (socket says 172.21.15.119:35812) 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 -- 172.21.15.119:0/676140035 learned_addr learned my addr 172.21.15.119:0/676140035 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75ce73e700 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x10ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cd73c700 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10d8170 0x10aec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 -- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10d8170 msgr2=0x10aec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10d8170 0x10aec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 -- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 msgr2=0x10ee490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x10ee490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 -- 172.21.15.119:0/676140035 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xfa54c0 con 0x108d310 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cd73c700 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10d8170 0x10aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x108d310 0x10ae710 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f75c4010a10 tx=0x7f75c4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9575574f3f9b5055 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75abfff700 1 -- 172.21.15.119:0/676140035 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4005a40 con 0x108d310 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75abfff700 1 -- 172.21.15.119:0/676140035 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c4032040 con 0x108d310 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75abfff700 1 -- 172.21.15.119:0/676140035 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4033d20 con 0x108d310 2024-09-13T16:35:01.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- 172.21.15.119:0/676140035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x108d310 msgr2=0x10ae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x108d310 0x10ae710 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f75c4010a10 tx=0x7f75c4015000 comp rx=0 tx=0).stop 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- 172.21.15.119:0/676140035 shutdown_connections 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x10ee490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10d8170 0x10aec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 --2- 172.21.15.119:0/676140035 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x108d310 0x10ae710 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- 172.21.15.119:0/676140035 >> 172.21.15.119:0/676140035 conn(0xe205b0 msgr2=0x1091750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- 172.21.15.119:0/676140035 shutdown_connections 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.988+0000 7f75d3452740 1 -- 172.21.15.119:0/676140035 wait complete. 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 -- start start 2024-09-13T16:35:01.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 Processor -- start 2024-09-13T16:35:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x113d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d8170 0x113e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x113e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:01.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x10fa740 con 0x10d8170 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x10b4180 con 0x10ee0b0 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x10b1690 con 0x108d310 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 10 client.0 _ll_register_callbacks cb 0xff7610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75cd73c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d8170 0x113e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x113e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75cdf3d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x113d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x113e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:56462/0 (socket says 172.21.15.119:56462) 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75cd73c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d8170 0x113e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35814/0 (socket says 172.21.15.119:35814) 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 -- 172.21.15.119:0/514388311 learned_addr learned my addr 172.21.15.119:0/514388311 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 -- 172.21.15.119:0/514388311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 msgr2=0x113d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x113d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 -- 172.21.15.119:0/514388311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d8170 msgr2=0x113e130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d8170 0x113e130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 -- 172.21.15.119:0/514388311 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0xe6ba40 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75ce73e700 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x113e670 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f75c000a340 tx=0x7f75c0001250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5eee96ff883e0dfa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75a8ff9700 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75a8ff9700 1 -- 172.21.15.119:0/514388311 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c0069ea0 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75a8ff9700 1 -- 172.21.15.119:0/514388311 <== mon.1 v2:172.21.15.145:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f75c0055050 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f75c401a070 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75a8ff9700 1 -- 172.21.15.119:0/514388311 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=76) v2 ==== 24+0+0 (secure 0 0 0) 0x7f75c0073080 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 10 client.8830 fetch_fsmap learned FSMap version 76 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=76,monmap=2+}) v3 -- 0xf99790 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75a8ff9700 1 -- 172.21.15.119:0/514388311 <== mon.1 v2:172.21.15.145:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c401a070 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75a8ff9700 1 -- 172.21.15.119:0/514388311 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 76) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f75c00552b0 con 0x10ee0b0 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 10 client.8830 fetch_fsmap finished waiting for FSMap version 76 2024-09-13T16:35:02.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 10 client.8830 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 20 client.8830 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x113da20 0x113ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 4 client.8830 mds_command: new command op to 8829 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:35:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.992+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 --> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0xe9c2c0 con 0x113da20 2024-09-13T16:35:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x113da20 0x113ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75a8ff9700 10 client.8830 ms_handle_connect on v2:172.21.15.119:6834/3700300033 2024-09-13T16:35:02.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cdf3d700 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x113da20 0x113ebb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=70b492e47b5f1f67 server_cookie=5504d6e50d5eab00 in_seq=0 out_seq=0 2024-09-13T16:35:02.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75a8ff9700 1 -- 172.21.15.119:0/514388311 <== mds.0 v2:172.21.15.119:6834/3700300033 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0xe9c2c0 con 0x113da20 2024-09-13T16:35:02.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75a8ff9700 10 client.8830 handle_command_reply: tid=0 2024-09-13T16:35:02.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 client.8830 shutdown 2024-09-13T16:35:02.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 msgr2=0x113e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x113e670 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7f75c000a340 tx=0x7f75c0001250 comp rx=0 tx=0).stop 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 shutdown_connections 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.119:6834/3700300033,v1:172.21.15.119:6835/3700300033] conn(0x113da20 0x113ebb0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10ee0b0 0x113e670 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10d8170 0x113e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75a8ff9700 0 client.8830 ms_handle_reset on v2:172.21.15.119:6834/3700300033 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 --2- 172.21.15.119:0/514388311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x108d310 0x113d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 >> 172.21.15.119:0/514388311 conn(0xe205b0 msgr2=0x10d71c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 shutdown_connections 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 1 -- 172.21.15.119:0/514388311 wait complete. 2024-09-13T16:35:02.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75d3452740 20 client.8830 trim_cache size 0 max 16384 2024-09-13T16:35:02.005 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:35:02.005 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:35:02.005 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:35:02.005 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75ac066730 msgr2=0x7f75ac068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75ac066730 0x7f75ac068bf0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f75c80b30a0 tx=0x7f75c4008040 comp rx=0 tx=0).stop 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c8144180 msgr2=0x7f75c80aaf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c8144180 0x7f75c80aaf00 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7f75c000cae0 tx=0x7f75c000cdf0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 shutdown_connections 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75ac066730 0x7f75ac068bf0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f75c8144180 0x7f75c80aaf00 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f75c80ce000 0x7f75c80aa9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 --2- 172.21.15.119:0/628948176 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f75c80cc010 0x7f75c813d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 >> 172.21.15.119:0/628948176 conn(0x7f75c80b0980 msgr2=0x7f75c815b4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 shutdown_connections 2024-09-13T16:35:02.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:01.996+0000 7f75cef3f700 1 -- 172.21.15.119:0/628948176 wait complete. 2024-09-13T16:35:02.019 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:35:02.120 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:35:02.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.232+0000 7fc00dc65700 1 Processor -- start 2024-09-13T16:35:02.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.232+0000 7fc00dc65700 1 -- start start 2024-09-13T16:35:02.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc0080adf00 0x7fc00806da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc008159740 0x7fc00806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0080cc890 con 0x7fc0080adf00 2024-09-13T16:35:02.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc008071770 con 0x7fc008159740 2024-09-13T16:35:02.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc00815e0b0 con 0x7fc008159360 2024-09-13T16:35:02.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc0080adf00 0x7fc00806da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc0080adf00 0x7fc00806da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35830/0 (socket says 172.21.15.119:35830) 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 -- 172.21.15.119:0/495778201 learned_addr learned my addr 172.21.15.119:0/495778201 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00d464700 1 --2- 172.21.15.119:0/495778201 >> v2:172.21.15.145:3300/0 conn(0x7fc008159740 0x7fc00806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00cc63700 1 --2- 172.21.15.119:0/495778201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 -- 172.21.15.119:0/495778201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 msgr2=0x7fc00806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 --2- 172.21.15.119:0/495778201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 -- 172.21.15.119:0/495778201 >> v2:172.21.15.145:3300/0 conn(0x7fc008159740 msgr2=0x7fc00806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 --2- 172.21.15.119:0/495778201 >> v2:172.21.15.145:3300/0 conn(0x7fc008159740 0x7fc00806fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 -- 172.21.15.119:0/495778201 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc00815d2f0 con 0x7fc0080adf00 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00d464700 1 --2- 172.21.15.119:0/495778201 >> v2:172.21.15.145:3300/0 conn(0x7fc008159740 0x7fc00806fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:02.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc007fff700 1 --2- 172.21.15.119:0/495778201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0080adf00 0x7fc00806da70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fc0080ade90 tx=0x7fbff4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd606ee596a28fee server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc0077fe700 1 -- 172.21.15.119:0/495778201 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4011830 con 0x7fc0080adf00 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc0077fe700 1 -- 172.21.15.119:0/495778201 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff4011990 con 0x7fc0080adf00 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc0077fe700 1 -- 172.21.15.119:0/495778201 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbff4011d20 con 0x7fc0080adf00 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- 172.21.15.119:0/495778201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0080adf00 msgr2=0x7fc00806da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 --2- 172.21.15.119:0/495778201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0080adf00 0x7fc00806da70 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7fc0080ade90 tx=0x7fbff4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- 172.21.15.119:0/495778201 shutdown_connections 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 --2- 172.21.15.119:0/495778201 >> v2:172.21.15.145:3300/0 conn(0x7fc008159740 0x7fc00806fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 --2- 172.21.15.119:0/495778201 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc0080adf00 0x7fc00806da70 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 --2- 172.21.15.119:0/495778201 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- 172.21.15.119:0/495778201 >> 172.21.15.119:0/495778201 conn(0x7fc00806a350 msgr2=0x7fc00806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- 172.21.15.119:0/495778201 shutdown_connections 2024-09-13T16:35:02.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.236+0000 7fc00dc65700 1 -- 172.21.15.119:0/495778201 wait complete. 2024-09-13T16:35:02.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 Processor -- start 2024-09-13T16:35:02.245 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 -- start start 2024-09-13T16:35:02.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0080adf00 0x7fc008151250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00815be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc008159740 0x7fc00815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080a89f0 con 0x7fc008159740 2024-09-13T16:35:02.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc008151790 con 0x7fc0080adf00 2024-09-13T16:35:02.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00dc65700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080a8630 con 0x7fc008159360 2024-09-13T16:35:02.248 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00d464700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc008159740 0x7fc00815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00d464700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc008159740 0x7fc00815c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:35846/0 (socket says 172.21.15.119:35846) 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00d464700 1 -- 172.21.15.119:0/3753427172 learned_addr learned my addr 172.21.15.119:0/3753427172 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc007fff700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00815be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.240+0000 7fc00cc63700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0080adf00 0x7fc008151250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00d464700 1 -- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 msgr2=0x7fc00815be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00d464700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00815be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00d464700 1 -- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0080adf00 msgr2=0x7fc008151250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00d464700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0080adf00 0x7fc008151250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00d464700 1 -- 172.21.15.119:0/3753427172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc000009a70 con 0x7fc008159740 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00d464700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc008159740 0x7fc00815c3c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fc000009e60 tx=0x7fc00000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.249 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc0057fa700 1 -- 172.21.15.119:0/3753427172 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc000009cd0 con 0x7fc008159740 2024-09-13T16:35:02.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc0057fa700 1 -- 172.21.15.119:0/3753427172 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc00000f040 con 0x7fc008159740 2024-09-13T16:35:02.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbff4009570 con 0x7fc008159740 2024-09-13T16:35:02.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc0057fa700 1 -- 172.21.15.119:0/3753427172 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00000ccb0 con 0x7fc008159740 2024-09-13T16:35:02.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0080acec0 con 0x7fc008159740 2024-09-13T16:35:02.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0080a8b30 con 0x7fc008159740 2024-09-13T16:35:02.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.244+0000 7fc0057fa700 1 -- 172.21.15.119:0/3753427172 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc000014480 con 0x7fc008159740 2024-09-13T16:35:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.248+0000 7fc0057fa700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbfec066730 0x7fbfec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.248+0000 7fc0057fa700 1 -- 172.21.15.119:0/3753427172 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fc000084860 con 0x7fc008159740 2024-09-13T16:35:02.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.248+0000 7fc00cc63700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbfec066730 0x7fbfec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.248+0000 7fc00cc63700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbfec066730 0x7fbfec068bf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009b50 tx=0x7fbffc009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.256+0000 7fc0057fa700 1 -- 172.21.15.119:0/3753427172 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc0080a8b30 con 0x7fc008159740 2024-09-13T16:35:02.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.460+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc0080a8b30 con 0x7fc008159740 2024-09-13T16:35:02.468 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:35:02.464+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:35:02.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.476+0000 7fc0057fa700 1 -- 172.21.15.119:0/3753427172 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v77) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc0080a8b30 con 0x7fc008159740 2024-09-13T16:35:02.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbfec066730 msgr2=0x7fbfec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbfec066730 0x7fbfec068bf0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fbffc009b50 tx=0x7fbffc009240 comp rx=0 tx=0).stop 2024-09-13T16:35:02.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc008159740 msgr2=0x7fc00815c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc008159740 0x7fc00815c3c0 secure :-1 s=READY pgs=1668 cs=0 l=1 rev1=1 crypto rx=0x7fc000009e60 tx=0x7fc00000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 shutdown_connections 2024-09-13T16:35:02.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbfec066730 0x7fbfec068bf0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc008159740 0x7fc00815c3c0 unknown :-1 s=CLOSED pgs=1668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc008159360 0x7fc00815be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 --2- 172.21.15.119:0/3753427172 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc0080adf00 0x7fc008151250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 >> 172.21.15.119:0/3753427172 conn(0x7fc00806a350 msgr2=0x7fc0080b2990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 shutdown_connections 2024-09-13T16:35:02.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.480+0000 7fc00dc65700 1 -- 172.21.15.119:0/3753427172 wait complete. 2024-09-13T16:35:02.489 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:35:02.489 INFO:tasks.ceph.mds.c.smithi119.stderr: -6185> 2024-09-13T16:33:59.945+0000 7f339e892700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:35:02.499 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627777 summary 2024-09-13T16:35:02.521 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:35:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 -- start start 2024-09-13T16:35:02.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 Processor -- start 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561607df9800 0x561607e9e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561607df9400 0x561607e9eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561607df9000 0x561607e9f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561607e2fe00 con 0x561607df9000 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561607e7aa00 con 0x561607df9400 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561607e7a640 con 0x561607df9800 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561607df9000 0x561607e9f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561607df9000 0x561607e9f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49322/0 (socket says 172.21.15.119:49322) 2024-09-13T16:35:02.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 -- 172.21.15.119:0/3391653436 learned_addr learned my addr 172.21.15.119:0/3391653436 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561607df9400 0x561607e9eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dacbb700 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561607df9800 0x561607e9e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 -- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561607df9800 msgr2=0x561607e9e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561607df9800 0x561607e9e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 -- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561607df9400 msgr2=0x561607e9eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561607df9400 0x561607e9eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 -- 172.21.15.119:0/3391653436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561607e1c9c0 con 0x561607df9000 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7db4bc700 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561607df9000 0x561607e9f080 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x561607eb9230 tx=0x561608bd6000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06a1ea44c5dcb33 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7d84b6700 1 -- 172.21.15.119:0/3391653436 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561607e49080 con 0x561607df9000 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7d84b6700 1 -- 172.21.15.119:0/3391653436 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561608be0300 con 0x561607df9000 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7d84b6700 1 -- 172.21.15.119:0/3391653436 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561607e49600 con 0x561607df9000 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 -- 172.21.15.119:0/3391653436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561607e1c1a0 con 0x561607df9000 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 1 -- 172.21.15.119:0/3391653436 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x561607e1c9c0 con 0x561607df9000 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 4 waiting for MDS map... 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7d84b6700 1 -- 172.21.15.119:0/3391653436 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x561607e32c40 con 0x561607df9000 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7d84b6700 1 -- 172.21.15.119:0/3391653436 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 77) v1 ==== 2573+0+0 (secure 0 0 0) 0x561607e3af00 con 0x561607df9000 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 4 Got MDS map 77 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 10 main: JournalTool::main 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.536+0000 7ff7dc501500 4 main: JournalTool: connecting to RADOS... 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 Processor -- start 2024-09-13T16:35:02.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- start start 2024-09-13T16:35:02.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2400 0x561607e9f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2800 0x561607e9fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561607e7a500 con 0x5616080f2c00 2024-09-13T16:35:02.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561607e7a3c0 con 0x5616080f2400 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561607e7a640 con 0x5616080f2800 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49330/0 (socket says 172.21.15.119:49330) 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 -- 172.21.15.119:0/1512251541 learned_addr learned my addr 172.21.15.119:0/1512251541 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2400 0x561607e9f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dacbb700 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2800 0x561607e9fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 -- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2800 msgr2=0x561607e9fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2800 0x561607e9fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 -- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2400 msgr2=0x561607e9f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.551 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2400 0x561607e9f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 -- 172.21.15.119:0/1512251541 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561607e1cb60 con 0x5616080f2c00 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0100 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x56160813e720 tx=0x561608104cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea7df7983e6205 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7d64b2700 1 -- 172.21.15.119:0/1512251541 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561607e49080 con 0x5616080f2c00 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7d64b2700 1 -- 172.21.15.119:0/1512251541 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561608be0f00 con 0x5616080f2c00 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7d64b2700 1 -- 172.21.15.119:0/1512251541 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561607e49600 con 0x5616080f2c00 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- 172.21.15.119:0/1512251541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 msgr2=0x561607ea0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0100 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x56160813e720 tx=0x561608104cc0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- 172.21.15.119:0/1512251541 shutdown_connections 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0100 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2800 0x561607e9fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 --2- 172.21.15.119:0/1512251541 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2400 0x561607e9f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- 172.21.15.119:0/1512251541 >> 172.21.15.119:0/1512251541 conn(0x561607df8c00 msgr2=0x561607ef3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- 172.21.15.119:0/1512251541 shutdown_connections 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.540+0000 7ff7dc501500 1 -- 172.21.15.119:0/1512251541 wait complete. 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 Processor -- start 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 -- start start 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2400 0x561607ea1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2800 0x561607ea0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x561607e7a280 con 0x5616080f2c00 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x561607e7a140 con 0x5616080f2800 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x561607e7a000 con 0x5616080f2400 2024-09-13T16:35:02.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7da4ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2800 0x561607ea0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7da4ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49344/0 (socket says 172.21.15.119:49344) 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2800 0x561607ea0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:48048/0 (socket says 172.21.15.119:48048) 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7da4ba700 1 -- 172.21.15.119:0/3410211015 learned_addr learned my addr 172.21.15.119:0/3410211015 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2400 0x561607ea1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 -- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2400 msgr2=0x561607ea1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5616080f2400 0x561607ea1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 -- 172.21.15.119:0/3410211015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 msgr2=0x561607ea0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 -- 172.21.15.119:0/3410211015 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561607e1c340 con 0x5616080f2800 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5616080f2c00 0x561607ea0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:02.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dacbb700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2800 0x561607ea0c00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x56160813f470 tx=0x561608104d20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7d4caf700 1 -- 172.21.15.119:0/3410211015 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561607e49080 con 0x5616080f2800 2024-09-13T16:35:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7d4caf700 1 -- 172.21.15.119:0/3410211015 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x561607e44180 con 0x5616080f2800 2024-09-13T16:35:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7d4caf700 1 -- 172.21.15.119:0/3410211015 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x561607e49600 con 0x5616080f2800 2024-09-13T16:35:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x561607e1c000 con 0x5616080f2800 2024-09-13T16:35:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561607e1cea0 con 0x5616080f2800 2024-09-13T16:35:02.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.544+0000 7ff7dc501500 4 main: JournalTool: resolving pool 15 2024-09-13T16:35:02.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7d4caf700 1 -- 172.21.15.119:0/3410211015 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x561607e58840 con 0x5616080f2800 2024-09-13T16:35:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7d4caf700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56160817c400 0x561607ea1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7d4caf700 1 -- 172.21.15.119:0/3410211015 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x561607e32a80 con 0x5616080f2800 2024-09-13T16:35:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 4 main: JournalTool: creating IoCtx.. 2024-09-13T16:35:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 4 main: Executing for rank 0 2024-09-13T16:35:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 4 Filtering by inode '1099511627777' 2024-09-13T16:35:02.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5616080f3000 0x561607ea0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:1 15.f 15:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x5616080f3400 con 0x5616080f3000 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56160817c400 0x561607ea1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5616080f3000 0x561607ea0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56160817c400 0x561607ea1700 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x56160813f380 tx=0x56160826a000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7da4ba700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5616080f3000 0x561607ea0100 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7da4ba700 1 -- 172.21.15.119:0/3410211015 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x561608c72b40 con 0x5616080f3000 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 4 Pointer 400.00000000 is readable 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 4 JournalScanner::scan: reading header object '200.00000000' 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:2 15.14 15:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x5616080f3800 con 0x5616080f3000 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7da4ba700 1 -- 172.21.15.119:0/3410211015 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv30 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x561608c72b40 con 0x5616080f3000 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 10 Header 0x400000 0x405ee0 0x40625c 2024-09-13T16:35:02.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 10 Starting journal scan from offset 0x405ee0 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:3 15.14 15:2e2fa760:::200.00000001:head [read 24288~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x56160817c800 con 0x5616080f3000 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7da4ba700 1 -- 172.21.15.119:0/3410211015 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(3 200.00000001 [read 24288~892 out=892b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+892 (crc 0 0 0) 0x561608c72b40 con 0x5616080f3000 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 4 Read 0x37c bytes from 200.00000001 gap=0 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 10 Parsing data, 0x37c bytes available 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 10 Attempting decode at 0x405ee0 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 10 Consumed 0x37c bytes 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 10 Valid entry at 0x405ee0 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56160817cc00 0x561607e9fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:4 15.1b 15:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x56160817d000 con 0x56160817cc00 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.548+0000 7ff7db4bc700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56160817cc00 0x561607e9fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7db4bc700 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56160817cc00 0x561607e9fb80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7db4bc700 1 -- 172.21.15.119:0/3410211015 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x561608c72d80 con 0x56160817cc00 2024-09-13T16:35:02.561 INFO:teuthology.orchestra.run.smithi119.stdout:Events by type: 2024-09-13T16:35:02.561 INFO:teuthology.orchestra.run.smithi119.stdout:Errors: 0 2024-09-13T16:35:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 4 Reached end of journal objects 2024-09-13T16:35:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 4 Scanned objects, 0 missing, 1 valid 2024-09-13T16:35:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 4 Events scanned, 0 gaps 2024-09-13T16:35:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 4 Found 1 valid events 2024-09-13T16:35:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 4 Selected 0 events events for processing 2024-09-13T16:35:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 4 0 events 2024-09-13T16:35:02.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5616080f3000 msgr2=0x561607ea0100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x5616080f3000 0x561607ea0100 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56160817cc00 msgr2=0x561607e9fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56160817cc00 0x561607e9fb80 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56160817c400 msgr2=0x561607ea1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56160817c400 0x561607ea1700 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x56160813f380 tx=0x56160826a000 comp rx=0 tx=0).stop 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2800 msgr2=0x561607ea0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3410211015 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5616080f2800 0x561607ea0c00 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x56160813f470 tx=0x561608104d20 comp rx=0 tx=0).stop 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dacbb700 1 -- 172.21.15.119:0/3410211015 reap_dead start 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 shutdown_connections 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 >> 172.21.15.119:0/3410211015 conn(0x561607df8c00 msgr2=0x561607ef3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dacbb700 1 -- 172.21.15.119:0/3410211015 reap_dead start 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 shutdown_connections 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3410211015 wait complete. 2024-09-13T16:35:02.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3391653436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561607df9000 msgr2=0x561607e9f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561607df9000 0x561607e9f080 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x561607eb9230 tx=0x561608bd6000 comp rx=0 tx=0).stop 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3391653436 shutdown_connections 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x561607df9000 0x561607e9f080 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x561607df9400 0x561607e9eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 --2- 172.21.15.119:0/3391653436 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x561607df9800 0x561607e9e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3391653436 >> 172.21.15.119:0/3391653436 conn(0x561607df9c00 msgr2=0x561607ef3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3391653436 shutdown_connections 2024-09-13T16:35:02.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.552+0000 7ff7dc501500 1 -- 172.21.15.119:0/3391653436 wait complete. 2024-09-13T16:35:02.565 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627777', 'summary'] in time 0:00:00.065906, result: Events by type: Errors: 0 2024-09-13T16:35:02.565 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 event splice --inode=1099511627778 summary 2024-09-13T16:35:02.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.600+0000 7f3cdc68a500 1 -- start start 2024-09-13T16:35:02.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.600+0000 7f3cdc68a500 1 Processor -- start 2024-09-13T16:35:02.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f5fd07800 0x556f5fdadb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f5fd07400 0x556f5fdad600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f5fd07000 0x556f5fdad080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd88a00 con 0x556f5fd07800 2024-09-13T16:35:02.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd88640 con 0x556f5fd07400 2024-09-13T16:35:02.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd88500 con 0x556f5fd07000 2024-09-13T16:35:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdae44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f5fd07800 0x556f5fdadb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cda643700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f5fd07400 0x556f5fdad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdae44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f5fd07800 0x556f5fdadb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49362/0 (socket says 172.21.15.119:49362) 2024-09-13T16:35:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdae44700 1 -- 172.21.15.119:0/445536524 learned_addr learned my addr 172.21.15.119:0/445536524 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdb645700 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f5fd07000 0x556f5fdad080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cda643700 1 -- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f5fd07000 msgr2=0x556f5fdad080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cda643700 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f5fd07000 0x556f5fdad080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cda643700 1 -- 172.21.15.119:0/445536524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f5fd07800 msgr2=0x556f5fdadb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cda643700 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f5fd07800 0x556f5fdadb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cda643700 1 -- 172.21.15.119:0/445536524 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f5fd2a9c0 con 0x556f5fd07400 2024-09-13T16:35:02.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdae44700 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f5fd07800 0x556f5fdadb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:02.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cda643700 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f5fd07400 0x556f5fdad600 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x556f5fe0f290 tx=0x556f5fddac00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4d9aae969644ed1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cd863f700 1 -- 172.21.15.119:0/445536524 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f5fd57600 con 0x556f5fd07400 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cd863f700 1 -- 172.21.15.119:0/445536524 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556f60ae6000 con 0x556f5fd07400 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cd863f700 1 -- 172.21.15.119:0/445536524 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f5fd571e0 con 0x556f5fd07400 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 -- 172.21.15.119:0/445536524 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f5fd2ab60 con 0x556f5fd07400 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 1 -- 172.21.15.119:0/445536524 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x556f5fd2a340 con 0x556f5fd07400 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 4 waiting for MDS map... 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cd863f700 1 -- 172.21.15.119:0/445536524 <== mon.1 v2:172.21.15.145:3300/0 4 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x556f5fd40c40 con 0x556f5fd07400 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cd863f700 1 -- 172.21.15.119:0/445536524 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 77) v1 ==== 2573+0+0 (secure 0 0 0) 0x556f60ae8900 con 0x556f5fd07400 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 4 Got MDS map 77 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.604+0000 7f3cdc68a500 10 main: JournalTool::main 2024-09-13T16:35:02.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 4 main: JournalTool: connecting to RADOS... 2024-09-13T16:35:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 Processor -- start 2024-09-13T16:35:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 -- start start 2024-09-13T16:35:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b90800 0x556f5fdacb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90c00 0x556f5fdac580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd3de00 con 0x556f60b91000 2024-09-13T16:35:02.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd88a00 con 0x556f60b90800 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd883c0 con 0x556f60b90c00 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49368/0 (socket says 172.21.15.119:49368) 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 -- 172.21.15.119:0/977140731 learned_addr learned my addr 172.21.15.119:0/977140731 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdae44700 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90c00 0x556f5fdac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cdb645700 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b90800 0x556f5fdacb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 -- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90c00 msgr2=0x556f5fdac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90c00 0x556f5fdac580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 -- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b90800 msgr2=0x556f5fdacb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b90800 0x556f5fdacb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 -- 172.21.15.119:0/977140731 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f5fd2a000 con 0x556f60b91000 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cda643700 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b91000 0x556f5fdae100 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x556f60052720 tx=0x556f5fddc180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebcea6b4690e51b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cd663b700 1 -- 172.21.15.119:0/977140731 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f5fd56f20 con 0x556f60b91000 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cd663b700 1 -- 172.21.15.119:0/977140731 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556f60ae6c00 con 0x556f60b91000 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.608+0000 7f3cd663b700 1 -- 172.21.15.119:0/977140731 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f5fd57600 con 0x556f60b91000 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 -- 172.21.15.119:0/977140731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b91000 msgr2=0x556f5fdae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b91000 0x556f5fdae100 secure :-1 s=READY pgs=1677 cs=0 l=1 rev1=1 crypto rx=0x556f60052720 tx=0x556f5fddc180 comp rx=0 tx=0).stop 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 -- 172.21.15.119:0/977140731 shutdown_connections 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=CLOSED pgs=1677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90c00 0x556f5fdac580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/977140731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b90800 0x556f5fdacb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 -- 172.21.15.119:0/977140731 >> 172.21.15.119:0/977140731 conn(0x556f5fd06c00 msgr2=0x556f5fdc3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 -- 172.21.15.119:0/977140731 shutdown_connections 2024-09-13T16:35:02.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 -- 172.21.15.119:0/977140731 wait complete. 2024-09-13T16:35:02.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 Processor -- start 2024-09-13T16:35:02.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 -- start start 2024-09-13T16:35:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.612+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90800 0x556f5fdacb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b90c00 0x556f5fdac580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd88280 con 0x556f60b90c00 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd88140 con 0x556f60b91000 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x556f5fd88000 con 0x556f60b90800 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b90c00 0x556f5fdac580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b90c00 0x556f5fdac580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49378/0 (socket says 172.21.15.119:49378) 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 -- 172.21.15.119:0/785838679 learned_addr learned my addr 172.21.15.119:0/785838679 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdb645700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90800 0x556f5fdacb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cda643700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 -- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90800 msgr2=0x556f5fdacb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f60b90800 0x556f5fdacb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 -- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b91000 msgr2=0x556f5fdae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 -- 172.21.15.119:0/785838679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556f5fd2a9c0 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cda643700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f60b91000 0x556f5fdae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdae44700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b90c00 0x556f5fdac580 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x556f60053470 tx=0x556f5fddd860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cd4e38700 1 -- 172.21.15.119:0/785838679 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f5fd57080 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cd4e38700 1 -- 172.21.15.119:0/785838679 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556f5fd52900 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x556f5fd2ab60 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cd4e38700 1 -- 172.21.15.119:0/785838679 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x556f5fd56f20 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556f5fd2aea0 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.616+0000 7f3cdc68a500 4 main: JournalTool: resolving pool 15 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cd4e38700 1 -- 172.21.15.119:0/785838679 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x556f5fd66840 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cd4e38700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x556f6006e800 0x556f60156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cd4e38700 1 -- 172.21.15.119:0/785838679 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x556f5fd40a80 con 0x556f60b90c00 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 4 main: JournalTool: creating IoCtx.. 2024-09-13T16:35:02.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdb645700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x556f6006e800 0x556f60156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 4 main: Executing for rank 0 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 4 Filtering by inode '1099511627778' 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x556f60b91400 0x556f5fdae680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:1 15.f 15:f1f69726:::400.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x556f60b91800 con 0x556f60b91400 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cda643700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x556f60b91400 0x556f5fdae680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdb645700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x556f6006e800 0x556f60156000 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x556f5fe0e750 tx=0x556f6015e000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cda643700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x556f60b91400 0x556f5fdae680 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cda643700 1 -- 172.21.15.119:0/785838679 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(1 400.00000000 [read 0~22 out=22b] v0'0 uv1 ondisk = 0) v8 ==== 156+0+22 (crc 0 0 0) 0x556f60b88b40 con 0x556f60b91400 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 4 Pointer 400.00000000 is readable 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 4 JournalScanner::scan: reading header object '200.00000000' 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.620+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:2 15.14 15:292cf221:::200.00000000:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x556f60b91c00 con 0x556f60b91400 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cda643700 1 -- 172.21.15.119:0/785838679 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(2 200.00000000 [read 0~90 out=90b] v0'0 uv30 ondisk = 0) v8 ==== 156+0+90 (crc 0 0 0) 0x556f60b88b40 con 0x556f60b91400 2024-09-13T16:35:02.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 10 Header 0x400000 0x405ee0 0x40625c 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 10 Starting journal scan from offset 0x405ee0 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:3 15.14 15:2e2fa760:::200.00000001:head [read 24288~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x556f6006ec00 con 0x556f60b91400 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cda643700 1 -- 172.21.15.119:0/785838679 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(3 200.00000001 [read 24288~892 out=892b] v0'0 uv27 ondisk = 0) v8 ==== 156+0+892 (crc 0 0 0) 0x556f60b88b40 con 0x556f60b91400 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 4 Read 0x37c bytes from 200.00000001 gap=0 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 10 Parsing data, 0x37c bytes available 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 10 Attempting decode at 0x405ee0 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 10 Consumed 0x37c bytes 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 10 Valid entry at 0x405ee0 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x556f6006f000 0x556f5fdaec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:4 15.1b 15:ddb4e4d7:::200.00000002:head [read 0~2147483647] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e114) v8 -- 0x556f6006f400 con 0x556f6006f000 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdb645700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x556f6006f000 0x556f5fdaec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdb645700 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x556f6006f000 0x556f5fdaec00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdb645700 1 -- 172.21.15.119:0/785838679 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(4 200.00000002 [read 0~2147483647] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x556f60b88d80 con 0x556f6006f000 2024-09-13T16:35:02.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 4 Reached end of journal objects 2024-09-13T16:35:02.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 4 Scanned objects, 0 missing, 1 valid 2024-09-13T16:35:02.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 4 Events scanned, 0 gaps 2024-09-13T16:35:02.638 INFO:teuthology.orchestra.run.smithi119.stdout:Events by type: 2024-09-13T16:35:02.638 INFO:teuthology.orchestra.run.smithi119.stdout:Errors: 0 2024-09-13T16:35:02.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 4 Found 1 valid events 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 4 Selected 0 events events for processing 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 4 0 events 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x556f60b91400 msgr2=0x556f5fdae680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x556f60b91400 0x556f5fdae680 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x556f6006f000 msgr2=0x556f5fdaec00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x556f6006f000 0x556f5fdaec00 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x556f6006e800 msgr2=0x556f60156000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x556f6006e800 0x556f60156000 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x556f5fe0e750 tx=0x556f6015e000 comp rx=0 tx=0).stop 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b90c00 msgr2=0x556f5fdac580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/785838679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f60b90c00 0x556f5fdac580 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x556f60053470 tx=0x556f5fddd860 comp rx=0 tx=0).stop 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdae44700 1 -- 172.21.15.119:0/785838679 reap_dead start 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 shutdown_connections 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdae44700 1 -- 172.21.15.119:0/785838679 reap_dead start 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 >> 172.21.15.119:0/785838679 conn(0x556f5fd06c00 msgr2=0x556f5fdc3800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 shutdown_connections 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/785838679 wait complete. 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f5fd07400 msgr2=0x556f5fdad600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f5fd07400 0x556f5fdad600 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x556f5fe0f290 tx=0x556f5fddac00 comp rx=0 tx=0).stop 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/445536524 shutdown_connections 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x556f5fd07000 0x556f5fdad080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x556f5fd07400 0x556f5fdad600 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 --2- 172.21.15.119:0/445536524 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x556f5fd07800 0x556f5fdadb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/445536524 >> 172.21.15.119:0/445536524 conn(0x556f5fd07c00 msgr2=0x556f5fdc3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/445536524 shutdown_connections 2024-09-13T16:35:02.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.624+0000 7f3cdc68a500 1 -- 172.21.15.119:0/445536524 wait complete. 2024-09-13T16:35:02.643 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'event', 'splice', '--inode=1099511627778', 'summary'] in time 0:00:00.077513, result: Events by type: Errors: 0 2024-09-13T16:35:02.643 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata put mds0_inotable - 2024-09-13T16:35:02.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 Processor -- start 2024-09-13T16:35:02.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- start start 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf8d0 0x55870770d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 0x55870777f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587076532f0 con 0x5587076bf4f0 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55870760f8e0 con 0x5587076bf8d0 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55870760f580 con 0x55870777d330 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49390/0 (socket says 172.21.15.119:49390) 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 -- 172.21.15.119:0/3687871839 learned_addr learned my addr 172.21.15.119:0/3687871839 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e97937700 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf8d0 0x55870770d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98939700 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 0x55870777f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 -- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 msgr2=0x55870777f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 0x55870777f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 -- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf8d0 msgr2=0x55870770d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf8d0 0x55870770d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 -- 172.21.15.119:0/3687871839 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e90005de0 con 0x5587076bf4f0 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e98138700 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf4f0 0x55870770b640 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001550 tx=0x7f6e88001860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c31e1ac6ce84900 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e96935700 1 -- 172.21.15.119:0/3687871839 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e88004070 con 0x5587076bf4f0 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e96935700 1 -- 172.21.15.119:0/3687871839 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e88012620 con 0x5587076bf4f0 2024-09-13T16:35:02.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e96935700 1 -- 172.21.15.119:0/3687871839 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e880129d0 con 0x5587076bf4f0 2024-09-13T16:35:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- 172.21.15.119:0/3687871839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf4f0 msgr2=0x55870770b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf4f0 0x55870770b640 secure :-1 s=READY pgs=1680 cs=0 l=1 rev1=1 crypto rx=0x7f6e88001550 tx=0x7f6e88001860 comp rx=0 tx=0).stop 2024-09-13T16:35:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- 172.21.15.119:0/3687871839 shutdown_connections 2024-09-13T16:35:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 0x55870777f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf8d0 0x55870770d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 --2- 172.21.15.119:0/3687871839 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- 172.21.15.119:0/3687871839 >> 172.21.15.119:0/3687871839 conn(0x5587076c0b10 msgr2=0x558707796df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- 172.21.15.119:0/3687871839 shutdown_connections 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 -- 172.21.15.119:0/3687871839 wait complete. 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.680+0000 7f6e9a603140 1 Processor -- start 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 -- start start 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf8d0 0x55870770bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 0x55870782fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587075e58d0 con 0x5587076bf8d0 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x558707708ca0 con 0x5587076bf4f0 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x558707708f40 con 0x55870777d330 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf8d0 0x55870770bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf8d0 0x55870770bb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49394/0 (socket says 172.21.15.119:49394) 2024-09-13T16:35:02.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e98138700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 -- 172.21.15.119:0/1166320265 learned_addr learned my addr 172.21.15.119:0/1166320265 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e98939700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 0x55870782fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 -- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 msgr2=0x55870782fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55870777d330 0x55870782fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 -- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf4f0 msgr2=0x55870770b640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 -- 172.21.15.119:0/1166320265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587075f0bb0 con 0x5587076bf8d0 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e98138700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587076bf4f0 0x55870770b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e97937700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf8d0 0x55870770bb80 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0086b0 tx=0x7f6e8c0089c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e95933700 1 -- 172.21.15.119:0/1166320265 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c00f040 con 0x5587076bf8d0 2024-09-13T16:35:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e95933700 1 -- 172.21.15.119:0/1166320265 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e8c015880 con 0x5587076bf8d0 2024-09-13T16:35:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5587075fdb00 con 0x5587076bf8d0 2024-09-13T16:35:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e95933700 1 -- 172.21.15.119:0/1166320265 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c015c10 con 0x5587076bf8d0 2024-09-13T16:35:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.684+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587075f0f40 con 0x5587076bf8d0 2024-09-13T16:35:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e95933700 1 -- 172.21.15.119:0/1166320265 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6e8c01e400 con 0x5587076bf8d0 2024-09-13T16:35:02.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e95933700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e84065090 0x7f6e84067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e95933700 1 -- 172.21.15.119:0/1166320265 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f6e8c0839e0 con 0x5587076bf8d0 2024-09-13T16:35:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e98138700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e84065090 0x7f6e84067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e9a603140 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5587077063c0 0x5587078304c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114) v8 -- 0x558707705a90 con 0x5587077063c0 2024-09-13T16:35:02.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e98939700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5587077063c0 0x5587078304c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e98138700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e84065090 0x7f6e84067550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6e88014040 tx=0x7f6e88008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.688+0000 7f6e98939700 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5587077063c0 0x5587078304c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e98939700 1 -- 172.21.15.119:0/1166320265 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 mds0_inotable [writefull 0~34] v114'5 uv5 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x7f6e90008960 con 0x5587077063c0 2024-09-13T16:35:02.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5587077063c0 msgr2=0x5587078304c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x5587077063c0 0x5587078304c0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e84065090 msgr2=0x7f6e84067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6e84065090 0x7f6e84067550 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f6e88014040 tx=0x7f6e88008030 comp rx=0 tx=0).stop 2024-09-13T16:35:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf8d0 msgr2=0x55870770bb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 --2- 172.21.15.119:0/1166320265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587076bf8d0 0x55870770bb80 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c0086b0 tx=0x7f6e8c0089c0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e98939700 1 -- 172.21.15.119:0/1166320265 reap_dead start 2024-09-13T16:35:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 shutdown_connections 2024-09-13T16:35:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 >> 172.21.15.119:0/1166320265 conn(0x5587076c0b10 msgr2=0x5587077969b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 shutdown_connections 2024-09-13T16:35:02.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.692+0000 7f6e9a603140 1 -- 172.21.15.119:0/1166320265 wait complete. 2024-09-13T16:35:02.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-13T16:35:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a328e700 1 Processor -- start 2024-09-13T16:35:02.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a328e700 1 -- start start 2024-09-13T16:35:02.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a328e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f199c15cc00 0x7f199c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a328e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f199c0d17e0 0x7f199c0b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a328e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199c0cdb00 con 0x7f199c0d0c20 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a328e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199c0cdde0 con 0x7f199c0d17e0 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a328e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f199c0ce0c0 con 0x7f199c15cc00 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a2a8d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f199c0d17e0 0x7f199c0b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a2a8d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f199c0d17e0 0x7f199c0b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49398/0 (socket says 172.21.15.119:49398) 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.816+0000 7f19a2a8d700 1 -- 172.21.15.119:0/2111157540 learned_addr learned my addr 172.21.15.119:0/2111157540 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a228c700 1 --2- 172.21.15.119:0/2111157540 >> v2:172.21.15.145:3300/0 conn(0x7f199c15cc00 0x7f199c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a2a8d700 1 -- 172.21.15.119:0/2111157540 >> v2:172.21.15.145:3300/0 conn(0x7f199c15cc00 msgr2=0x7f199c0d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:02.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a2a8d700 1 --2- 172.21.15.119:0/2111157540 >> v2:172.21.15.145:3300/0 conn(0x7f199c15cc00 0x7f199c0d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a2a8d700 1 -- 172.21.15.119:0/2111157540 >> v1:172.21.15.145:6789/0 conn(0x7f199c0d0c20 legacy=0x7f199c0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a2a8d700 1 -- 172.21.15.119:0/2111157540 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f199c0ce3a0 con 0x7f199c0d17e0 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a2a8d700 1 --2- 172.21.15.119:0/2111157540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c0b5960 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f199c0b5610 tx=0x7f199400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f57cba8dc99b1d6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a128a700 1 -- 172.21.15.119:0/2111157540 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199400e070 con 0x7f199c0d17e0 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a128a700 1 -- 172.21.15.119:0/2111157540 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f199400b910 con 0x7f199c0d17e0 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a128a700 1 -- 172.21.15.119:0/2111157540 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f199400bcc0 con 0x7f199c0d17e0 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 -- 172.21.15.119:0/2111157540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 msgr2=0x7f199c0b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 --2- 172.21.15.119:0/2111157540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c0b5960 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f199c0b5610 tx=0x7f199400b330 comp rx=0 tx=0).stop 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 -- 172.21.15.119:0/2111157540 shutdown_connections 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 --2- 172.21.15.119:0/2111157540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c0b5960 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 --2- 172.21.15.119:0/2111157540 >> v2:172.21.15.145:3300/0 conn(0x7f199c15cc00 0x7f199c0d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 -- 172.21.15.119:0/2111157540 >> 172.21.15.119:0/2111157540 conn(0x7f199c08c080 msgr2=0x7f199c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:02.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 -- 172.21.15.119:0/2111157540 shutdown_connections 2024-09-13T16:35:02.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.820+0000 7f19a328e700 1 -- 172.21.15.119:0/2111157540 wait complete. 2024-09-13T16:35:02.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 Processor -- start 2024-09-13T16:35:02.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 -- start start 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f199c0d0c20 0x7f199c0c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c147f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f199c15cc00 0x7f199c06b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c148490 con 0x7f199c0d17e0 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f199c1e6820 con 0x7f199c0d0c20 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f199c1e6b00 con 0x7f199c15cc00 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c147f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c147f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49400/0 (socket says 172.21.15.119:49400) 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 -- 172.21.15.119:0/1284840499 learned_addr learned my addr 172.21.15.119:0/1284840499 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a228c700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f199c0d0c20 0x7f199c0c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a2a8d700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f199c15cc00 0x7f199c06b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 -- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f199c15cc00 msgr2=0x7f199c06b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f199c15cc00 0x7f199c06b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 -- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f199c0d0c20 msgr2=0x7f199c0c88a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f199c0d0c20 0x7f199c0c88a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 -- 172.21.15.119:0/1284840499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1994009b30 con 0x7f199c0d17e0 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a2a8d700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f199c15cc00 0x7f199c06b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a1a8b700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c147f50 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f199800c500 tx=0x7f199800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f1992ffd700 1 -- 172.21.15.119:0/1284840499 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19980052d0 con 0x7f199c0d17e0 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f1992ffd700 1 -- 172.21.15.119:0/1284840499 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1998005430 con 0x7f199c0d17e0 2024-09-13T16:35:02.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f199c1e6de0 con 0x7f199c0d17e0 2024-09-13T16:35:02.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.824+0000 7f1992ffd700 1 -- 172.21.15.119:0/1284840499 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19980055a0 con 0x7f199c0d17e0 2024-09-13T16:35:02.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.828+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f199c1e73a0 con 0x7f199c0d17e0 2024-09-13T16:35:02.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.828+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f199c0ce8c0 con 0x7f199c0d17e0 2024-09-13T16:35:02.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.828+0000 7f1992ffd700 1 -- 172.21.15.119:0/1284840499 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f19980038e0 con 0x7f199c0d17e0 2024-09-13T16:35:02.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.828+0000 7f1992ffd700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1980066730 0x7f1980068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:02.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.828+0000 7f1992ffd700 1 -- 172.21.15.119:0/1284840499 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f19980836f0 con 0x7f199c0d17e0 2024-09-13T16:35:02.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.828+0000 7f19a228c700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1980066730 0x7f1980068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:02.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.832+0000 7f19a228c700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1980066730 0x7f1980068bf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f199c1f6100 tx=0x7f198c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:02.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:02.836+0000 7f1992ffd700 1 -- 172.21.15.119:0/1284840499 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1998050940 con 0x7f199c0d17e0 2024-09-13T16:35:03.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.044+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f199c06c2f0 con 0x7f199c0d17e0 2024-09-13T16:35:03.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.480+0000 7f1992ffd700 1 -- 172.21.15.119:0/1284840499 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v78) v1 ==== 162+0+0 (secure 0 0 0) 0x7f1998053f60 con 0x7f199c0d17e0 2024-09-13T16:35:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1980066730 msgr2=0x7f1980068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1980066730 0x7f1980068bf0 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f199c1f6100 tx=0x7f198c009240 comp rx=0 tx=0).stop 2024-09-13T16:35:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 msgr2=0x7f199c147f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c147f50 secure :-1 s=READY pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f199800c500 tx=0x7f199800c810 comp rx=0 tx=0).stop 2024-09-13T16:35:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 shutdown_connections 2024-09-13T16:35:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1980066730 0x7f1980068bf0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f199c15cc00 0x7f199c06b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f199c0d17e0 0x7f199c147f50 secure :-1 s=CLOSED pgs=1683 cs=0 l=1 rev1=1 crypto rx=0x7f199800c500 tx=0x7f199800c810 comp rx=0 tx=0).stop 2024-09-13T16:35:03.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 --2- 172.21.15.119:0/1284840499 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f199c0d0c20 0x7f199c0c88a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 >> 172.21.15.119:0/1284840499 conn(0x7f199c08c080 msgr2=0x7f199c066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:03.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 shutdown_connections 2024-09-13T16:35:03.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.488+0000 7f19a328e700 1 -- 172.21.15.119:0/1284840499 wait complete. 2024-09-13T16:35:03.493 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-13T16:35:03.509 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:03.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.620+0000 7fc345069700 1 Processor -- start 2024-09-13T16:35:03.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.620+0000 7fc345069700 1 -- start start 2024-09-13T16:35:03.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3401594f0 0x7fc34006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc34006da70 0x7fc3400b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3400b2280 con 0x7fc3400b3630 2024-09-13T16:35:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc340070c10 con 0x7fc34006da70 2024-09-13T16:35:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34006deb0 con 0x7fc3401594f0 2024-09-13T16:35:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc344868700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc3400b3630 0x7fc34015d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:43460/0 (socket says 172.21.15.119:43460) 2024-09-13T16:35:03.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc344868700 1 -- 172.21.15.119:0/4028863022 learned_addr learned my addr 172.21.15.119:0/4028863022 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc3377fe700 1 --2- 172.21.15.119:0/4028863022 >> v2:172.21.15.145:3300/0 conn(0x7fc34006da70 0x7fc3400b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc33ffff700 1 --2- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3401594f0 0x7fc34006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc33f7fe700 1 -- 172.21.15.119:0/4028863022 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293064621 0 0) 0x7fc3400b2280 con 0x7fc3400b3630 2024-09-13T16:35:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc33f7fe700 1 -- 172.21.15.119:0/4028863022 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc32c0035d0 con 0x7fc3400b3630 2024-09-13T16:35:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc3377fe700 1 -- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3401594f0 msgr2=0x7fc34006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:03.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc3377fe700 1 --2- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3401594f0 0x7fc34006d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc3377fe700 1 -- 172.21.15.119:0/4028863022 >> v1:172.21.15.119:6789/0 conn(0x7fc3400b3630 legacy=0x7fc34015d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc3377fe700 1 -- 172.21.15.119:0/4028863022 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3400cc890 con 0x7fc34006da70 2024-09-13T16:35:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc3377fe700 1 --2- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc34006da70 0x7fc3400b30f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc340070d50 tx=0x7fc328009b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6054aae0abebf173 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc33f7fe700 1 -- 172.21.15.119:0/4028863022 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc328004020 con 0x7fc34006da70 2024-09-13T16:35:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc33f7fe700 1 -- 172.21.15.119:0/4028863022 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc328004180 con 0x7fc34006da70 2024-09-13T16:35:03.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc33f7fe700 1 -- 172.21.15.119:0/4028863022 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3280044f0 con 0x7fc34006da70 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc34006da70 msgr2=0x7fc3400b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 --2- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc34006da70 0x7fc3400b30f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc340070d50 tx=0x7fc328009b30 comp rx=0 tx=0).stop 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- 172.21.15.119:0/4028863022 shutdown_connections 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 --2- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc34006da70 0x7fc3400b30f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 --2- 172.21.15.119:0/4028863022 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc3401594f0 0x7fc34006d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- 172.21.15.119:0/4028863022 >> 172.21.15.119:0/4028863022 conn(0x7fc34006a350 msgr2=0x7fc34006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- 172.21.15.119:0/4028863022 shutdown_connections 2024-09-13T16:35:03.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.624+0000 7fc345069700 1 -- 172.21.15.119:0/4028863022 wait complete. 2024-09-13T16:35:03.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 Processor -- start 2024-09-13T16:35:03.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 -- start start 2024-09-13T16:35:03.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc34006da70 0x7fc34014d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3400b3630 0x7fc340147af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3401594f0 0x7fc3400acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34014eb80 con 0x7fc3401594f0 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34014d0d0 con 0x7fc3400b3630 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc345069700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3401466a0 con 0x7fc34006da70 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3401594f0 0x7fc3400acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc3377fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3400b3630 0x7fc340147af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3401594f0 0x7fc3400acb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49416/0 (socket says 172.21.15.119:49416) 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc33ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc34006da70 0x7fc34014d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:03.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 -- 172.21.15.119:0/3234769007 learned_addr learned my addr 172.21.15.119:0/3234769007 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:03.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 -- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc34006da70 msgr2=0x7fc34014d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc34006da70 0x7fc34014d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 -- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3400b3630 msgr2=0x7fc340147af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3400b3630 0x7fc340147af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.628+0000 7fc344868700 1 -- 172.21.15.119:0/3234769007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3280094b0 con 0x7fc3401594f0 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc344868700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3401594f0 0x7fc3400acb30 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc33800cb10 tx=0x7fc33800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc33d7fa700 1 -- 172.21.15.119:0/3234769007 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc338007480 con 0x7fc3401594f0 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc340165cd0 con 0x7fc3401594f0 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc34014d590 con 0x7fc3401594f0 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc33d7fa700 1 -- 172.21.15.119:0/3234769007 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3380075e0 con 0x7fc3401594f0 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc33d7fa700 1 -- 172.21.15.119:0/3234769007 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc338007970 con 0x7fc3401594f0 2024-09-13T16:35:03.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc324005310 con 0x7fc3401594f0 2024-09-13T16:35:03.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc33d7fa700 1 -- 172.21.15.119:0/3234769007 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc33801b3b0 con 0x7fc3401594f0 2024-09-13T16:35:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc33d7fa700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.632+0000 7fc33d7fa700 1 -- 172.21.15.119:0/3234769007 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7fc3380840b0 con 0x7fc3401594f0 2024-09-13T16:35:03.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.636+0000 7fc33ffff700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:03.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.636+0000 7fc33ffff700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc330009b50 tx=0x7fc330009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:03.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.640+0000 7fc33d7fa700 1 -- 172.21.15.119:0/3234769007 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc338051270 con 0x7fc3401594f0 2024-09-13T16:35:03.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.844+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc324005890 con 0x7fc3401594f0 2024-09-13T16:35:03.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.844+0000 7fc33d7fa700 1 -- 172.21.15.119:0/3234769007 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 79 v79) v1 ==== 94+0+4747 (secure 0 0 0) 0x7fc338017020 con 0x7fc3401594f0 2024-09-13T16:35:03.850 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:03.850 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":79,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":79,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:35:03.486321+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9228},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9228":{"gid":9228,"name":"c","rank":0,"incarnation":79,"state":"up:replay","state_seq":1,"addr":"172.21.15.119:6835/912842245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":912842245},{"type":"v1","addr":"172.21.15.119:6835","nonce":912842245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:35:03.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 msgr2=0x7fc320068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7fc330009b50 tx=0x7fc330009240 comp rx=0 tx=0).stop 2024-09-13T16:35:03.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3401594f0 msgr2=0x7fc3400acb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:03.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3401594f0 0x7fc3400acb30 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7fc33800cb10 tx=0x7fc33800cb40 comp rx=0 tx=0).stop 2024-09-13T16:35:03.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 shutdown_connections 2024-09-13T16:35:03.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc320066730 0x7fc320068bf0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc3401594f0 0x7fc3400acb30 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc3400b3630 0x7fc340147af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 --2- 172.21.15.119:0/3234769007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc34006da70 0x7fc34014d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:03.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 >> 172.21.15.119:0/3234769007 conn(0x7fc34006a350 msgr2=0x7fc3400ae440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:03.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 shutdown_connections 2024-09-13T16:35:03.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:03.848+0000 7fc345069700 1 -- 172.21.15.119:0/3234769007 wait complete. 2024-09-13T16:35:03.855 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 79 2024-09-13T16:35:04.274 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 79, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:33:59.032499+0000', 'modified': '2024-09-13T16:35:03.486321+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9228}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9228': {'gid': 9228, 'name': 'c', 'rank': 0, 'incarnation': 79, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.119:6835/912842245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 912842245}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 912842245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:35:04.274 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9228:up:replay 2024-09-13T16:35:05.275 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042e696700 1 Processor -- start 2024-09-13T16:35:05.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042e696700 1 -- start start 2024-09-13T16:35:05.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042e696700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04281594f0 0x7f042806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042e696700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04280b3800 0x7f042806fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042e696700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04280b2280 con 0x7f042806da70 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042e696700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042806dee0 con 0x7f04281594f0 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042e696700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04280cc890 con 0x7f04280b3800 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042ce93700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f042806da70 0x7f04280b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:43470/0 (socket says 172.21.15.119:43470) 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.388+0000 7f042ce93700 1 -- 172.21.15.119:0/832059805 learned_addr learned my addr 172.21.15.119:0/832059805 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042d694700 1 --2- 172.21.15.119:0/832059805 >> v2:172.21.15.145:3300/0 conn(0x7f04281594f0 0x7f042806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:05.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042de95700 1 --2- 172.21.15.119:0/832059805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04280b3800 0x7f042806fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:05.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f041bfff700 1 -- 172.21.15.119:0/832059805 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561503225 0 0) 0x7f04280b2280 con 0x7f042806da70 2024-09-13T16:35:05.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f041bfff700 1 -- 172.21.15.119:0/832059805 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04140035d0 con 0x7f042806da70 2024-09-13T16:35:05.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042de95700 1 -- 172.21.15.119:0/832059805 >> v2:172.21.15.145:3300/0 conn(0x7f04281594f0 msgr2=0x7f042806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:05.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042de95700 1 --2- 172.21.15.119:0/832059805 >> v2:172.21.15.145:3300/0 conn(0x7f04281594f0 0x7f042806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042de95700 1 -- 172.21.15.119:0/832059805 >> v1:172.21.15.119:6789/0 conn(0x7f042806da70 legacy=0x7f04280b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:05.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f041bfff700 1 -- 172.21.15.119:0/832059805 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 365732583 0 0) 0x7f04140035d0 con 0x7f042806da70 2024-09-13T16:35:05.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042de95700 1 -- 172.21.15.119:0/832059805 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280b1970 con 0x7f04280b3800 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042de95700 1 --2- 172.21.15.119:0/832059805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04280b3800 0x7f042806fc00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f04280b1b60 tx=0x7f042000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd698eb7c48f3f43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f041bfff700 1 -- 172.21.15.119:0/832059805 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042000e070 con 0x7f04280b3800 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f041bfff700 1 -- 172.21.15.119:0/832059805 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0420003a40 con 0x7f04280b3800 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f041bfff700 1 -- 172.21.15.119:0/832059805 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0424001670 con 0x7f04280b3800 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 -- 172.21.15.119:0/832059805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04280b3800 msgr2=0x7f042806fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 --2- 172.21.15.119:0/832059805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04280b3800 0x7f042806fc00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f04280b1b60 tx=0x7f042000b330 comp rx=0 tx=0).stop 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 -- 172.21.15.119:0/832059805 shutdown_connections 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 --2- 172.21.15.119:0/832059805 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04280b3800 0x7f042806fc00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 --2- 172.21.15.119:0/832059805 >> v2:172.21.15.145:3300/0 conn(0x7f04281594f0 0x7f042806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 -- 172.21.15.119:0/832059805 >> 172.21.15.119:0/832059805 conn(0x7f042806a350 msgr2=0x7f042806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 -- 172.21.15.119:0/832059805 shutdown_connections 2024-09-13T16:35:05.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 -- 172.21.15.119:0/832059805 wait complete. 2024-09-13T16:35:05.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.392+0000 7f042e696700 1 Processor -- start 2024-09-13T16:35:05.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 -- start start 2024-09-13T16:35:05.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f042806da70 0x7f042814f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:05.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280b3800 0x7f04281e39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04281594f0 0x7f04281545b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280af210 con 0x7f04281594f0 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04280af4f0 con 0x7f04280b3800 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04280af7d0 con 0x7f042806da70 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04281594f0 0x7f04281545b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04281594f0 0x7f04281545b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49422/0 (socket says 172.21.15.119:49422) 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 -- 172.21.15.119:0/3288550964 learned_addr learned my addr 172.21.15.119:0/3288550964 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042d694700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f042806da70 0x7f042814f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042ce93700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280b3800 0x7f04281e39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 -- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f042806da70 msgr2=0x7f042814f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f042806da70 0x7f042814f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 -- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280b3800 msgr2=0x7f04281e39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:05.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280b3800 0x7f04281e39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 -- 172.21.15.119:0/3288550964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0420009a70 con 0x7f04281594f0 2024-09-13T16:35:05.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042de95700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04281594f0 0x7f04281545b0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f042806b1a0 tx=0x7f0420004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:05.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f0419ffb700 1 -- 172.21.15.119:0/3288550964 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420009470 con 0x7f04281594f0 2024-09-13T16:35:05.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f0419ffb700 1 -- 172.21.15.119:0/3288550964 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0420004600 con 0x7f04281594f0 2024-09-13T16:35:05.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04280afab0 con 0x7f04281594f0 2024-09-13T16:35:05.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f0419ffb700 1 -- 172.21.15.119:0/3288550964 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f042001b430 con 0x7f04281594f0 2024-09-13T16:35:05.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042814f650 con 0x7f04281594f0 2024-09-13T16:35:05.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.396+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04280affd0 con 0x7f04281594f0 2024-09-13T16:35:05.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.400+0000 7f0419ffb700 1 -- 172.21.15.119:0/3288550964 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f042001b590 con 0x7f04281594f0 2024-09-13T16:35:05.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.400+0000 7f0419ffb700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f040c066730 0x7f040c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:05.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.400+0000 7f0419ffb700 1 -- 172.21.15.119:0/3288550964 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f0420085980 con 0x7f04281594f0 2024-09-13T16:35:05.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.400+0000 7f042d694700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f040c066730 0x7f040c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:05.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.400+0000 7f042d694700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f040c066730 0x7f040c068bf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f04281e31a0 tx=0x7f041c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:05.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.408+0000 7f0419ffb700 1 -- 172.21.15.119:0/3288550964 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f04280affd0 con 0x7f04281594f0 2024-09-13T16:35:05.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.612+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04280affd0 con 0x7f04281594f0 2024-09-13T16:35:05.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.612+0000 7f0419ffb700 1 -- 172.21.15.119:0/3288550964 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f04280affd0 con 0x7f04281594f0 2024-09-13T16:35:05.617 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:05.617 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":81,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:35:05.498131+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9228},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9228":{"gid":9228,"name":"c","rank":0,"incarnation":79,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.119:6835/912842245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":912842245},{"type":"v1","addr":"172.21.15.119:6835","nonce":912842245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f040c066730 msgr2=0x7f040c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f040c066730 0x7f040c068bf0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f04281e31a0 tx=0x7f041c009240 comp rx=0 tx=0).stop 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04281594f0 msgr2=0x7f04281545b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04281594f0 0x7f04281545b0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f042806b1a0 tx=0x7f0420004af0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 shutdown_connections 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f040c066730 0x7f040c068bf0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f04281594f0 0x7f04281545b0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04280b3800 0x7f04281e39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 --2- 172.21.15.119:0/3288550964 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f042806da70 0x7f042814f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:05.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 >> 172.21.15.119:0/3288550964 conn(0x7f042806a350 msgr2=0x7f042815bdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:05.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 shutdown_connections 2024-09-13T16:35:05.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:05.616+0000 7f042e696700 1 -- 172.21.15.119:0/3288550964 wait complete. 2024-09-13T16:35:05.622 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 81 2024-09-13T16:35:06.041 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 81, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:33:59.032499+0000', 'modified': '2024-09-13T16:35:05.498131+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9228}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9228': {'gid': 9228, 'name': 'c', 'rank': 0, 'incarnation': 79, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.119:6835/912842245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 912842245}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 912842245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:35:06.041 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9228:up:rejoin 2024-09-13T16:35:07.043 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:07.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.160+0000 7f2122575700 1 Processor -- start 2024-09-13T16:35:07.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.160+0000 7f2122575700 1 -- start start 2024-09-13T16:35:07.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f211c158f10 0x7f211c06d540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c06de00 con 0x7f211c158b30 2024-09-13T16:35:07.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f211c06db20 con 0x7f211c071740 2024-09-13T16:35:07.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f211c0b1920 con 0x7f211c158f10 2024-09-13T16:35:07.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2121d74700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f211c158f10 0x7f211c06d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2121d74700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f211c158f10 0x7f211c06d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49424/0 (socket says 172.21.15.119:49424) 2024-09-13T16:35:07.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2121d74700 1 -- 172.21.15.119:0/4104374998 learned_addr learned my addr 172.21.15.119:0/4104374998 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2121d74700 1 -- 172.21.15.119:0/4104374998 >> v1:172.21.15.145:6789/0 conn(0x7f211c071740 legacy=0x7f211c06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2121d74700 1 -- 172.21.15.119:0/4104374998 >> v1:172.21.15.119:6789/0 conn(0x7f211c158b30 legacy=0x7f211c071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2121d74700 1 -- 172.21.15.119:0/4104374998 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211c0b1c00 con 0x7f211c158f10 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2113fff700 1 -- 172.21.15.119:0/4104374998 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680734606 0 0) 0x7f211c06de00 con 0x7f211c158b30 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2121d74700 1 --2- 172.21.15.119:0/4104374998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158f10 0x7f211c06d540 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f2114009e50 tx=0x7f211400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=813217c47aa67ab8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2113fff700 1 -- 172.21.15.119:0/4104374998 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211400e070 con 0x7f211c158f10 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2113fff700 1 -- 172.21.15.119:0/4104374998 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f211400b9c0 con 0x7f211c158f10 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2113fff700 1 -- 172.21.15.119:0/4104374998 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f211400bd70 con 0x7f211c158f10 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- 172.21.15.119:0/4104374998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158f10 msgr2=0x7f211c06d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 --2- 172.21.15.119:0/4104374998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158f10 0x7f211c06d540 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f2114009e50 tx=0x7f211400b330 comp rx=0 tx=0).stop 2024-09-13T16:35:07.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- 172.21.15.119:0/4104374998 shutdown_connections 2024-09-13T16:35:07.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 --2- 172.21.15.119:0/4104374998 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158f10 0x7f211c06d540 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- 172.21.15.119:0/4104374998 >> 172.21.15.119:0/4104374998 conn(0x7f211c06a350 msgr2=0x7f211c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- 172.21.15.119:0/4104374998 shutdown_connections 2024-09-13T16:35:07.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.164+0000 7f2122575700 1 -- 172.21.15.119:0/4104374998 wait complete. 2024-09-13T16:35:07.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 Processor -- start 2024-09-13T16:35:07.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 -- start start 2024-09-13T16:35:07.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f211c071740 0x7f211c1573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158b30 0x7f211c14ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f211c158f10 0x7f211c15b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c147c50 con 0x7f211c158b30 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f211c14a050 con 0x7f211c071740 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f211c14a330 con 0x7f211c158f10 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158b30 0x7f211c14ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158b30 0x7f211c14ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49440/0 (socket says 172.21.15.119:49440) 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 -- 172.21.15.119:0/3138268861 learned_addr learned my addr 172.21.15.119:0/3138268861 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2121573700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f211c071740 0x7f211c1573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2121d74700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f211c158f10 0x7f211c15b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 -- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f211c158f10 msgr2=0x7f211c15b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f211c158f10 0x7f211c15b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 -- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f211c071740 msgr2=0x7f211c1573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f211c071740 0x7f211c1573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 -- 172.21.15.119:0/3138268861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2114009b30 con 0x7f211c158b30 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2120d72700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158b30 0x7f211c14ae10 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f210c00c500 tx=0x7f210c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2111ffb700 1 -- 172.21.15.119:0/3138268861 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c0052d0 con 0x7f211c158b30 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2111ffb700 1 -- 172.21.15.119:0/3138268861 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210c005430 con 0x7f211c158b30 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f211c14a610 con 0x7f211c158b30 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2111ffb700 1 -- 172.21.15.119:0/3138268861 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f210c0055a0 con 0x7f211c158b30 2024-09-13T16:35:07.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.168+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211c147840 con 0x7f211c158b30 2024-09-13T16:35:07.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.172+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f211c14ab30 con 0x7f211c158b30 2024-09-13T16:35:07.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.172+0000 7f2111ffb700 1 -- 172.21.15.119:0/3138268861 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f210c0038e0 con 0x7f211c158b30 2024-09-13T16:35:07.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.172+0000 7f2111ffb700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2100066730 0x7f2100068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.172+0000 7f2111ffb700 1 -- 172.21.15.119:0/3138268861 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f210c0837d0 con 0x7f211c158b30 2024-09-13T16:35:07.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.172+0000 7f2121573700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2100066730 0x7f2100068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.172+0000 7f2121573700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2100066730 0x7f2100068bf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2118000f60 tx=0x7f2118009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.180+0000 7f2111ffb700 1 -- 172.21.15.119:0/3138268861 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f211c14ab30 con 0x7f211c158b30 2024-09-13T16:35:07.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.384+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f211c14ab30 con 0x7f211c158b30 2024-09-13T16:35:07.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.384+0000 7f2111ffb700 1 -- 172.21.15.119:0/3138268861 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+4747 (secure 0 0 0) 0x7f211c14ab30 con 0x7f211c158b30 2024-09-13T16:35:07.390 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:07.390 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":82,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":82,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:35:06.501057+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":114,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9228},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9228":{"gid":9228,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":4,"addr":"172.21.15.119:6835/912842245","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":912842245},{"type":"v1","addr":"172.21.15.119:6835","nonce":912842245}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:35:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2100066730 msgr2=0x7f2100068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2100066730 0x7f2100068bf0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f2118000f60 tx=0x7f2118009040 comp rx=0 tx=0).stop 2024-09-13T16:35:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158b30 msgr2=0x7f211c14ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158b30 0x7f211c14ae10 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f210c00c500 tx=0x7f210c00c810 comp rx=0 tx=0).stop 2024-09-13T16:35:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 shutdown_connections 2024-09-13T16:35:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2100066730 0x7f2100068bf0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f211c158f10 0x7f211c15b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f211c158b30 0x7f211c14ae10 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 --2- 172.21.15.119:0/3138268861 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f211c071740 0x7f211c1573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 >> 172.21.15.119:0/3138268861 conn(0x7f211c06a350 msgr2=0x7f211c0b2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 shutdown_connections 2024-09-13T16:35:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.388+0000 7f2122575700 1 -- 172.21.15.119:0/3138268861 wait complete. 2024-09-13T16:35:07.394 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 82 2024-09-13T16:35:07.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 82, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:33:59.032499+0000', 'modified': '2024-09-13T16:35:06.501057+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 114, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9228}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9228': {'gid': 9228, 'name': 'c', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.119:6835/912842245', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.119:6834', 'nonce': 912842245}, {'type': 'v1', 'addr': '172.21.15.119:6835', 'nonce': 912842245}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:35:07.808 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:35:07.809 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-09-13T16:35:07.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.928+0000 7f52d320d700 1 Processor -- start 2024-09-13T16:35:07.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.928+0000 7f52d320d700 1 -- start start 2024-09-13T16:35:07.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f52cc06dc40 0x7f52cc0b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc0b3630 0x7f52cc06fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52cc0b2280 con 0x7f52cc1594f0 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52cc0cc890 con 0x7f52cc06dc40 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0b1970 con 0x7f52cc0b3630 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f52cc06dc40 0x7f52cc0b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f52cc06dc40 0x7f52cc0b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49450/0 (socket says 172.21.15.119:49450) 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 -- 172.21.15.119:0/3033370959 learned_addr learned my addr 172.21.15.119:0/3033370959 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/3033370959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc0b3630 0x7f52cc06fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 -- 172.21.15.119:0/3033370959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc0b3630 msgr2=0x7f52cc06fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/3033370959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc0b3630 0x7f52cc06fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 -- 172.21.15.119:0/3033370959 >> v1:172.21.15.145:6789/0 conn(0x7f52cc1594f0 legacy=0x7f52cc06d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 -- 172.21.15.119:0/3033370959 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc0b1c30 con 0x7f52cc06dc40 2024-09-13T16:35:07.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/3033370959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc06dc40 0x7f52cc0b30f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009910 tx=0x7f52bc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3376d27eae104c27 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1209700 1 -- 172.21.15.119:0/3033370959 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc00ba60 con 0x7f52cc06dc40 2024-09-13T16:35:07.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1209700 1 -- 172.21.15.119:0/3033370959 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc00bbc0 con 0x7f52cc06dc40 2024-09-13T16:35:07.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d1209700 1 -- 172.21.15.119:0/3033370959 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc009e60 con 0x7f52cc06dc40 2024-09-13T16:35:07.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- 172.21.15.119:0/3033370959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc06dc40 msgr2=0x7f52cc0b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 --2- 172.21.15.119:0/3033370959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc06dc40 0x7f52cc0b30f0 secure :-1 s=READY pgs=1691 cs=0 l=1 rev1=1 crypto rx=0x7f52bc009910 tx=0x7f52bc009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- 172.21.15.119:0/3033370959 shutdown_connections 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 --2- 172.21.15.119:0/3033370959 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc0b3630 0x7f52cc06fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 --2- 172.21.15.119:0/3033370959 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc06dc40 0x7f52cc0b30f0 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- 172.21.15.119:0/3033370959 >> 172.21.15.119:0/3033370959 conn(0x7f52cc06a350 msgr2=0x7f52cc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- 172.21.15.119:0/3033370959 shutdown_connections 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.932+0000 7f52d320d700 1 -- 172.21.15.119:0/3033370959 wait complete. 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 Processor -- start 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 -- start start 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52cc06dc40 0x7f52cc143a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc0b3630 0x7f52cc15b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc1594f0 0x7f52cc148b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0a9130 con 0x7f52cc0b3630 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0a84b0 con 0x7f52cc06dc40 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52cc0a8790 con 0x7f52cc1594f0 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc0b3630 0x7f52cc15b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc0b3630 0x7f52cc15b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49464/0 (socket says 172.21.15.119:49464) 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 -- 172.21.15.119:0/1171314811 learned_addr learned my addr 172.21.15.119:0/1171314811 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d220b700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52cc06dc40 0x7f52cc143a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc1594f0 0x7f52cc148b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 -- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc1594f0 msgr2=0x7f52cc148b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc1594f0 0x7f52cc148b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 -- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52cc06dc40 msgr2=0x7f52cc143a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52cc06dc40 0x7f52cc143a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 -- 172.21.15.119:0/1171314811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52bc009570 con 0x7f52cc0b3630 2024-09-13T16:35:07.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc1594f0 0x7f52cc148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d220b700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52cc06dc40 0x7f52cc143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc0b3630 0x7f52cc15b870 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f52bc000c00 tx=0x7f52bc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52c2ffd700 1 -- 172.21.15.119:0/1171314811 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc0118b0 con 0x7f52cc0b3630 2024-09-13T16:35:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52c2ffd700 1 -- 172.21.15.119:0/1171314811 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52bc01f070 con 0x7f52cc0b3630 2024-09-13T16:35:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52cc0a8a70 con 0x7f52cc0b3630 2024-09-13T16:35:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52c2ffd700 1 -- 172.21.15.119:0/1171314811 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52bc011c30 con 0x7f52cc0b3630 2024-09-13T16:35:07.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.936+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52cc143f40 con 0x7f52cc0b3630 2024-09-13T16:35:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.940+0000 7f52c2ffd700 1 -- 172.21.15.119:0/1171314811 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f52bc026480 con 0x7f52cc0b3630 2024-09-13T16:35:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.940+0000 7f52c2ffd700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f52b0066730 0x7f52b0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.940+0000 7f52c2ffd700 1 -- 172.21.15.119:0/1171314811 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f52bc021070 con 0x7f52cc0b3630 2024-09-13T16:35:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.940+0000 7f52d220b700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f52b0066730 0x7f52b0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.940+0000 7f52d220b700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f52b0066730 0x7f52b0068bf0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f52cc1f5ae0 tx=0x7f52c8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 Processor -- start 2024-09-13T16:35:07.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 -- start start 2024-09-13T16:35:07.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bd7d00 0x1bc1890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bf96c0 0x1bc1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b20d80 con 0x1bf96c0 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b1e510 con 0x1bd7d00 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b19e00 con 0x1bf8da0 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bf96c0 0x1bc1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bf96c0 0x1bc1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49466/0 (socket says 172.21.15.119:49466) 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 -- 172.21.15.119:0/2511018966 learned_addr learned my addr 172.21.15.119:0/2511018966 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d220b700 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bd7d00 0x1bc1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 -- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 msgr2=0x1bf9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 -- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bd7d00 msgr2=0x1bc1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bd7d00 0x1bc1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 -- 172.21.15.119:0/2511018966 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x198c960 con 0x1bf96c0 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bf96c0 0x1bc1dd0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f52c40074e0 tx=0x7f52c40077f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca43262ad3f427be server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.944+0000 7f52affff700 1 -- 172.21.15.119:0/2511018966 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c4015670 con 0x1bf96c0 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52affff700 1 -- 172.21.15.119:0/2511018966 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c400f050 con 0x1bf96c0 2024-09-13T16:35:07.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52affff700 1 -- 172.21.15.119:0/2511018966 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c4015a10 con 0x1bf96c0 2024-09-13T16:35:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 -- 172.21.15.119:0/2511018966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bf96c0 msgr2=0x1bc1dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bf96c0 0x1bc1dd0 secure :-1 s=READY pgs=1693 cs=0 l=1 rev1=1 crypto rx=0x7f52c40074e0 tx=0x7f52c40077f0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 -- 172.21.15.119:0/2511018966 shutdown_connections 2024-09-13T16:35:07.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bf96c0 0x1bc1dd0 unknown :-1 s=CLOSED pgs=1693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bd7d00 0x1bc1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 --2- 172.21.15.119:0/2511018966 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 -- 172.21.15.119:0/2511018966 >> 172.21.15.119:0/2511018966 conn(0x19414d0 msgr2=0x1bdac20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 -- 172.21.15.119:0/2511018966 shutdown_connections 2024-09-13T16:35:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 -- 172.21.15.119:0/2511018966 wait complete. 2024-09-13T16:35:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 -- start start 2024-09-13T16:35:07.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.948+0000 7f52d7720740 1 Processor -- start 2024-09-13T16:35:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bb2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1bb10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1bd8c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ad7730 con 0x1bd7d00 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1add5b0 con 0x1bf8da0 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x19c5aa0 con 0x1bf96c0 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bb2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bb2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49482/0 (socket says 172.21.15.119:49482) 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 -- 172.21.15.119:0/566494749 learned_addr learned my addr 172.21.15.119:0/566494749 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 10 client.0 _ll_register_callbacks cb 0x1b13610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1bb10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1bd8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 -- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 msgr2=0x1bd8c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1bd8c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 -- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 msgr2=0x1bb10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1bb10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 -- 172.21.15.119:0/566494749 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52c4009d50 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1bb10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d220b700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bb2390 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f52c8003c80 tx=0x7f52c8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2b7284682911019 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 1 -- 172.21.15.119:0/566494749 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c80054e0 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 1 -- 172.21.15.119:0/566494749 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f52c8022450 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x194e1e0 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 1 -- 172.21.15.119:0/566494749 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52c8022610 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 10 client.9303 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x1ae0220 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 1 -- 172.21.15.119:0/566494749 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x194e1e0 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 1 -- 172.21.15.119:0/566494749 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f52c8020070 con 0x1bd7d00 2024-09-13T16:35:07.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 10 client.9303 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 10 client.9303 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 20 client.9303 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1bd4180 0x1baecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 4 client.9303 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1998460 con 0x1bd4180 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1bd4180 0x1baecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1bd4180 0x1baecf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5056c62e20802c4d server_cookie=a1fd91d514eed9ec in_seq=0 out_seq=0 2024-09-13T16:35:07.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 10 client.9303 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:07.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 1 -- 172.21.15.119:0/566494749 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1998460 con 0x1bd4180 2024-09-13T16:35:07.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52ad7fa700 10 client.9303 handle_command_reply: tid=0 2024-09-13T16:35:07.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.952+0000 7f52d7720740 1 client.9303 shutdown 2024-09-13T16:35:07.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 msgr2=0x1bb2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bb2390 secure :-1 s=READY pgs=1694 cs=0 l=1 rev1=1 crypto rx=0x7f52c8003c80 tx=0x7f52c8001080 comp rx=0 tx=0).stop 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 shutdown_connections 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1bd4180 0x1baecf0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1bd8c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1bb10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 --2- 172.21.15.119:0/566494749 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bb2390 unknown :-1 s=CLOSED pgs=1694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52ad7fa700 0 client.9303 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 >> 172.21.15.119:0/566494749 conn(0x19414d0 msgr2=0x1bd4860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 shutdown_connections 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 1 -- 172.21.15.119:0/566494749 wait complete. 2024-09-13T16:35:07.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.956+0000 7f52d7720740 20 client.9303 trim_cache size 0 max 16384 2024-09-13T16:35:07.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 Processor -- start 2024-09-13T16:35:07.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 -- start start 2024-09-13T16:35:07.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bd7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf34e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf96c0 0x1bf3a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1b19610 con 0x1bd7d00 2024-09-13T16:35:07.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ac2020 con 0x1bf96c0 2024-09-13T16:35:07.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ba9320 con 0x1bf8da0 2024-09-13T16:35:07.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bd7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bd7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49498/0 (socket says 172.21.15.119:49498) 2024-09-13T16:35:07.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 -- 172.21.15.119:0/3402331704 learned_addr learned my addr 172.21.15.119:0/3402331704 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf96c0 0x1bf3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 -- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 msgr2=0x1bf34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf34e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 -- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf96c0 msgr2=0x1bf3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf96c0 0x1bf3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.972+0000 7f52d220b700 1 -- 172.21.15.119:0/3402331704 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1bb36b0 con 0x1bd7d00 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf96c0 0x1bf3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d220b700 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bd7580 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f52c8005690 tx=0x7f52c801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f7810bf69631b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52c0dad700 1 -- 172.21.15.119:0/3402331704 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8005070 con 0x1bd7d00 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52c0dad700 1 -- 172.21.15.119:0/3402331704 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c8036070 con 0x1bd7d00 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52c0dad700 1 -- 172.21.15.119:0/3402331704 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8022450 con 0x1bd7d00 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 -- 172.21.15.119:0/3402331704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 msgr2=0x1bd7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bd7580 secure :-1 s=READY pgs=1695 cs=0 l=1 rev1=1 crypto rx=0x7f52c8005690 tx=0x7f52c801d000 comp rx=0 tx=0).stop 2024-09-13T16:35:07.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 -- 172.21.15.119:0/3402331704 shutdown_connections 2024-09-13T16:35:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf96c0 0x1bf3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf8da0 0x1bf34e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 --2- 172.21.15.119:0/3402331704 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1bd7580 unknown :-1 s=CLOSED pgs=1695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 -- 172.21.15.119:0/3402331704 >> 172.21.15.119:0/3402331704 conn(0x19414d0 msgr2=0x1bf7e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 -- 172.21.15.119:0/3402331704 shutdown_connections 2024-09-13T16:35:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 -- 172.21.15.119:0/3402331704 wait complete. 2024-09-13T16:35:07.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 -- start start 2024-09-13T16:35:07.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.976+0000 7f52d7720740 1 Processor -- start 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1c11580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1c11ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1c59dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c10b50 con 0x1bd7d00 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1c120f0 con 0x1bf8da0 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1c10db0 con 0x1bf96c0 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1c11580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 10 client.0 _ll_register_callbacks cb 0x1b70710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1c11580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49502/0 (socket says 172.21.15.119:49502) 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 -- 172.21.15.119:0/2321894946 learned_addr learned my addr 172.21.15.119:0/2321894946 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d2a0c700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1c59dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1c11ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 -- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 msgr2=0x1c59dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1c59dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 -- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 msgr2=0x1c11ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1c11ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 -- 172.21.15.119:0/2321894946 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1ae0220 con 0x1bd7d00 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1c11ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d220b700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1c11580 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f52c8034040 tx=0x7f52c8013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b369410cb8deaab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52ad7fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52ad7fa700 1 -- 172.21.15.119:0/2321894946 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8005390 con 0x1bd7d00 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52ad7fa700 1 -- 172.21.15.119:0/2321894946 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f52c80229a0 con 0x1bd7d00 2024-09-13T16:35:07.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f52c400f050 con 0x1bd7d00 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52ad7fa700 1 -- 172.21.15.119:0/2321894946 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52c8022b60 con 0x1bd7d00 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 10 client.9309 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x1ac6450 con 0x1bd7d00 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52ad7fa700 1 -- 172.21.15.119:0/2321894946 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52c400f050 con 0x1bd7d00 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.980+0000 7f52ad7fa700 1 -- 172.21.15.119:0/2321894946 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f52c8006880 con 0x1bd7d00 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 10 client.9309 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 10 client.9309 resolve_mds: resolved name 'c' to daemon mds.c 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 20 client.9309 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1c0f200 0x1c0f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 4 client.9309 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "status"}) v1 -- 0x19bd1e0 con 0x1c0f200 2024-09-13T16:35:07.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1c0f200 0x1c0f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52ad7fa700 10 client.9309 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d1a0a700 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1c0f200 0x1c0f5e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df46fe365e9ed4c5 server_cookie=a740d99e88208e4 in_seq=0 out_seq=0 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52ad7fa700 1 -- 172.21.15.119:0/2321894946 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x19bd1e0 con 0x1c0f200 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52ad7fa700 10 client.9309 handle_command_reply: tid=0 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 client.9309 shutdown 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 msgr2=0x1c11580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1c11580 secure :-1 s=READY pgs=1696 cs=0 l=1 rev1=1 crypto rx=0x7f52c8034040 tx=0x7f52c8013000 comp rx=0 tx=0).stop 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 shutdown_connections 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1c0f200 0x1c0f5e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1bf96c0 0x1c59dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52ad7fa700 0 client.9309 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:07.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1bf8da0 0x1c11ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 --2- 172.21.15.119:0/2321894946 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1bd7d00 0x1c11580 unknown :-1 s=CLOSED pgs=1696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 >> 172.21.15.119:0/2321894946 conn(0x19414d0 msgr2=0x1bb3e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 shutdown_connections 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 1 -- 172.21.15.119:0/2321894946 wait complete. 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d7720740 20 client.9309 trim_cache size 0 max 16384 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:35:07.992 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9228, 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 4.4939035289999998, 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 82, 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 114, 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 114, 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 5.4651634680000001 2024-09-13T16:35:07.993 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:35:07.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f52b0066730 msgr2=0x7f52b0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f52b0066730 0x7f52b0068bf0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f52cc1f5ae0 tx=0x7f52c8009040 comp rx=0 tx=0).stop 2024-09-13T16:35:07.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc0b3630 msgr2=0x7f52cc15b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc0b3630 0x7f52cc15b870 secure :-1 s=READY pgs=1692 cs=0 l=1 rev1=1 crypto rx=0x7f52bc000c00 tx=0x7f52bc004770 comp rx=0 tx=0).stop 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 shutdown_connections 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f52b0066730 0x7f52b0068bf0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f52cc1594f0 0x7f52cc148b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f52cc0b3630 0x7f52cc15b870 unknown :-1 s=CLOSED pgs=1692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 --2- 172.21.15.119:0/1171314811 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f52cc06dc40 0x7f52cc143a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 >> 172.21.15.119:0/1171314811 conn(0x7f52cc06a350 msgr2=0x7f52cc0aebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 shutdown_connections 2024-09-13T16:35:07.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:07.984+0000 7f52d320d700 1 -- 172.21.15.119:0/1171314811 wait complete. 2024-09-13T16:35:08.006 DEBUG:teuthology.orchestra.run.smithi119:> sudo daemon-helper kill ceph --cluster ceph -w 2024-09-13T16:35:08.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub start / repair,recursive 2024-09-13T16:35:08.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf3990700 1 Processor -- start 2024-09-13T16:35:08.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf3990700 1 -- start start 2024-09-13T16:35:08.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf3990700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 0x7efcec15f190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf3990700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efcec0b4e10 0x7efcec073ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf3990700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcec06b360 con 0x7efcec073ff0 2024-09-13T16:35:08.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf3990700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcec15f770 con 0x7efcec0b4e10 2024-09-13T16:35:08.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf3990700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0744a0 con 0x7efcec15aae0 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf1f2d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7efcec073ff0 0x7efcec071a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:43490/0 (socket says 172.21.15.119:43490) 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf1f2d700 1 -- 172.21.15.119:0/2792394412 learned_addr learned my addr 172.21.15.119:0/2792394412 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2792394412 >> v2:172.21.15.145:3300/0 conn(0x7efcec0b4e10 0x7efcec073ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf172c700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 0x7efcec15f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2603375676 0 0) 0x7efcec06b360 con 0x7efcec073ff0 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcd40035d0 con 0x7efcec073ff0 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 25160895 0 0) 0x7efcd40035d0 con 0x7efcec073ff0 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 msgr2=0x7efcec15f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 0x7efcec15f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 >> v2:172.21.15.145:3300/0 conn(0x7efcec0b4e10 msgr2=0x7efcec073ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 --2- 172.21.15.119:0/2792394412 >> v2:172.21.15.145:3300/0 conn(0x7efcec0b4e10 0x7efcec073ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec0b5250 con 0x7efcec073ff0 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7efce4001670 con 0x7efcec073ff0 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 >> v1:172.21.15.119:6789/0 conn(0x7efcec073ff0 legacy=0x7efcec071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcd4003a20 0x7efcd4003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcd4004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 0x7efcd4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec15f770 con 0x7efcd4004870 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec0744a0 con 0x7efcd4003a20 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcd400ba30 con 0x7efcec15aae0 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcd4004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf172c700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcd4003a20 0x7efcd4003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 0x7efcd4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf0f2b700 1 -- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 msgr2=0x7efcd4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec15aae0 0x7efcd4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.124+0000 7efcf0f2b700 1 -- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcd4003a20 msgr2=0x7efcd4003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcd4003a20 0x7efcd4003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf0f2b700 1 -- 172.21.15.119:0/2792394412 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd40035d0 con 0x7efcd4004870 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf172c700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcd4003a20 0x7efcd4003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcd4004cf0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7efce0009b40 tx=0x7efcd40058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68bdaafa55530df2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce000e6c0 con 0x7efcd4004870 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce000e820 con 0x7efcd4004870 2024-09-13T16:35:08.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcebfff700 1 -- 172.21.15.119:0/2792394412 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce000eb90 con 0x7efcd4004870 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 -- 172.21.15.119:0/2792394412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 msgr2=0x7efcd4004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 --2- 172.21.15.119:0/2792394412 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcd4004cf0 secure :-1 s=READY pgs=1698 cs=0 l=1 rev1=1 crypto rx=0x7efce0009b40 tx=0x7efcd40058a0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf1f2d700 1 -- 172.21.15.119:0/2792394412 reap_dead start 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 -- 172.21.15.119:0/2792394412 shutdown_connections 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 -- 172.21.15.119:0/2792394412 >> 172.21.15.119:0/2792394412 conn(0x7efcec0b4040 msgr2=0x7efcec0b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 -- 172.21.15.119:0/2792394412 shutdown_connections 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 -- 172.21.15.119:0/2792394412 wait complete. 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 Processor -- start 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.128+0000 7efcf3990700 1 -- start start 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcec1586e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcec156940 0x7efcec14f420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec06ab20 0x7efcec06afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec06b4e0 con 0x7efcd4004870 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcec156dc0 con 0x7efcec156940 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcec157040 con 0x7efcec06ab20 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcec1586e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcec1586e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49530/0 (socket says 172.21.15.119:49530) 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 -- 172.21.15.119:0/2054875493 learned_addr learned my addr 172.21.15.119:0/2054875493 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcec156940 0x7efcec14f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec06ab20 0x7efcec06afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 -- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec06ab20 msgr2=0x7efcec06afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec06ab20 0x7efcec06afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 -- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcec156940 msgr2=0x7efcec14f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcec156940 0x7efcec14f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 -- 172.21.15.119:0/2054875493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcec157320 con 0x7efcd4004870 2024-09-13T16:35:08.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf172c700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcec1586e0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0101c0 tx=0x7efcdc0104d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efce9ffb700 1 -- 172.21.15.119:0/2054875493 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc00e050 con 0x7efcd4004870 2024-09-13T16:35:08.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efce9ffb700 1 -- 172.21.15.119:0/2054875493 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc00f040 con 0x7efcd4004870 2024-09-13T16:35:08.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcec157600 con 0x7efcd4004870 2024-09-13T16:35:08.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efce9ffb700 1 -- 172.21.15.119:0/2054875493 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc003a40 con 0x7efcd4004870 2024-09-13T16:35:08.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.132+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec157bc0 con 0x7efcd4004870 2024-09-13T16:35:08.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42dcb08700 1 Processor -- start 2024-09-13T16:35:08.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42dcb08700 1 -- start start 2024-09-13T16:35:08.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7efce9ffb700 1 -- 172.21.15.119:0/2054875493 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efcdc003ba0 con 0x7efcd4004870 2024-09-13T16:35:08.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7efce9ffb700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efcd8066730 0x7efcd8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7efce9ffb700 1 -- 172.21.15.119:0/2054875493 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7efcdc084df0 con 0x7efcd4004870 2024-09-13T16:35:08.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efcd8066730 0x7efcd8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42dcb08700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f42d80abed0 0x7f42d80ac2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42dcb08700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d80cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42dcb08700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42d806b360 con 0x7f42d8159110 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42dcb08700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42d806ed30 con 0x7f42d80abed0 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42dcb08700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d80ac7f0 con 0x7f42d80cb4a0 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d6ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f42d8159110 0x7f42d80ccdd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:43496/0 (socket says 172.21.15.119:43496) 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d6ffd700 1 -- 172.21.15.119:0/212312392 learned_addr learned my addr 172.21.15.119:0/212312392 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d77fe700 1 --2- 172.21.15.119:0/212312392 >> v2:172.21.15.145:3300/0 conn(0x7f42d80abed0 0x7f42d80ac2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d7fff700 1 --2- 172.21.15.119:0/212312392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d80cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d67fc700 1 -- 172.21.15.119:0/212312392 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550261987 0 0) 0x7f42d806b360 con 0x7f42d8159110 2024-09-13T16:35:08.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d67fc700 1 -- 172.21.15.119:0/212312392 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42c00079e0 con 0x7f42d8159110 2024-09-13T16:35:08.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d7fff700 1 -- 172.21.15.119:0/212312392 >> v2:172.21.15.145:3300/0 conn(0x7f42d80abed0 msgr2=0x7f42d80ac2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d7fff700 1 --2- 172.21.15.119:0/212312392 >> v2:172.21.15.145:3300/0 conn(0x7f42d80abed0 0x7f42d80ac2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d7fff700 1 -- 172.21.15.119:0/212312392 >> v1:172.21.15.119:6789/0 conn(0x7f42d8159110 legacy=0x7f42d80ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efcd8066730 0x7efcd8068bf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7efce00090b0 tx=0x7efce001e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d67fc700 1 -- 172.21.15.119:0/212312392 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339087002 0 0) 0x7f42c00079e0 con 0x7f42d8159110 2024-09-13T16:35:08.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d7fff700 1 -- 172.21.15.119:0/212312392 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d815d970 con 0x7f42d80cb4a0 2024-09-13T16:35:08.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d7fff700 1 --2- 172.21.15.119:0/212312392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d80cd4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009d90 tx=0x7f42d000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41a044127b2a6ec3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d67fc700 1 -- 172.21.15.119:0/212312392 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42d000e070 con 0x7f42d80cb4a0 2024-09-13T16:35:08.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d67fc700 1 -- 172.21.15.119:0/212312392 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42d0003a40 con 0x7f42d80cb4a0 2024-09-13T16:35:08.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7f42d67fc700 1 -- 172.21.15.119:0/212312392 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42cc001670 con 0x7f42d80cb4a0 2024-09-13T16:35:08.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7efcf6c41740 1 Processor -- start 2024-09-13T16:35:08.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.136+0000 7efcf6c41740 1 -- start start 2024-09-13T16:35:08.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- 172.21.15.119:0/212312392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 msgr2=0x7f42d80cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 --2- 172.21.15.119:0/212312392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d80cd4e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f42d0009d90 tx=0x7f42d000b330 comp rx=0 tx=0).stop 2024-09-13T16:35:08.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- 172.21.15.119:0/212312392 shutdown_connections 2024-09-13T16:35:08.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 --2- 172.21.15.119:0/212312392 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d80cd4e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 --2- 172.21.15.119:0/212312392 >> v2:172.21.15.145:3300/0 conn(0x7f42d80abed0 0x7f42d80ac2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- 172.21.15.119:0/212312392 >> 172.21.15.119:0/212312392 conn(0x7f42d80ae4b0 msgr2=0x7f42d80b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.146 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- 172.21.15.119:0/212312392 shutdown_connections 2024-09-13T16:35:08.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8630 0x21c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21c9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c9870 0x21c9c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x20f3500 con 0x21c8630 2024-09-13T16:35:08.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x20f0c90 con 0x21c9870 2024-09-13T16:35:08.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x20ec580 con 0x21c8f50 2024-09-13T16:35:08.147 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- 172.21.15.119:0/212312392 wait complete. 2024-09-13T16:35:08.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8630 0x21c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8630 0x21c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49534/0 (socket says 172.21.15.119:49534) 2024-09-13T16:35:08.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 -- 172.21.15.119:0/3177490800 learned_addr learned my addr 172.21.15.119:0/3177490800 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.149 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c9870 0x21c9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 -- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 msgr2=0x21c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21c9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 -- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c9870 msgr2=0x21c9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c9870 0x21c9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 -- 172.21.15.119:0/3177490800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1fdeee0 con 0x21c8630 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf172c700 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8630 0x21c8a10 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0520f0 tx=0x7efcdc055650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87c2492129975d56 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcd2ffd700 1 -- 172.21.15.119:0/3177490800 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc055bf0 con 0x21c8630 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcd2ffd700 1 -- 172.21.15.119:0/3177490800 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc081070 con 0x21c8630 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcd2ffd700 1 -- 172.21.15.119:0/3177490800 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcdc059a20 con 0x21c8630 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- 172.21.15.119:0/3177490800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8630 msgr2=0x21c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8630 0x21c8a10 secure :-1 s=READY pgs=1701 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0520f0 tx=0x7efcdc055650 comp rx=0 tx=0).stop 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- 172.21.15.119:0/3177490800 shutdown_connections 2024-09-13T16:35:08.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c9870 0x21c9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 --2- 172.21.15.119:0/3177490800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8630 0x21c8a10 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- 172.21.15.119:0/3177490800 >> 172.21.15.119:0/3177490800 conn(0x1f13780 msgr2=0x21adc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- 172.21.15.119:0/3177490800 shutdown_connections 2024-09-13T16:35:08.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- 172.21.15.119:0/3177490800 wait complete. 2024-09-13T16:35:08.151 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 Processor -- start 2024-09-13T16:35:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- start start 2024-09-13T16:35:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42d80abed0 0x7f42d81eeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d8141360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.152 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7efcf6c41740 1 -- start start 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d81443f0 0x7f42d81f1630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d8143f60 con 0x7f42d80abed0 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d81f24f0 con 0x7f42d81443f0 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42dcb08700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d81f1f30 con 0x7f42d80cb4a0 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42d77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42d80abed0 0x7f42d81eeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42d77fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42d80abed0 0x7f42d81eeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49544/0 (socket says 172.21.15.119:49544) 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.140+0000 7f42d77fe700 1 -- 172.21.15.119:0/3880728381 learned_addr learned my addr 172.21.15.119:0/3880728381 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d6ffd700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d8141360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d7fff700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d81443f0 0x7f42d81f1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d77fe700 1 -- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 msgr2=0x7f42d8141360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 Processor -- start 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d77fe700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d8141360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d77fe700 1 -- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d81443f0 msgr2=0x7f42d81f1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d77fe700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d81443f0 0x7f42d81f1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d77fe700 1 -- 172.21.15.119:0/3880728381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d0009a70 con 0x7f42d80abed0 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d6ffd700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f42d80cb4a0 0x7f42d8141360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d7fff700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f42d81443f0 0x7f42d81f1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d77fe700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f42d80abed0 0x7f42d81eeb50 secure :-1 s=READY pgs=1702 cs=0 l=1 rev1=1 crypto rx=0x7f42c800fde0 tx=0x7f42c800d630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8018640 con 0x7f42d80abed0 2024-09-13T16:35:08.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42c80187a0 con 0x7f42d80abed0 2024-09-13T16:35:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42dcb08700 1 -- 172.21.15.119:0/3880728381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d81f2210 con 0x7f42d80abed0 2024-09-13T16:35:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c8018b10 con 0x7f42d80abed0 2024-09-13T16:35:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42dcb08700 1 -- 172.21.15.119:0/3880728381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d81f1b70 con 0x7f42d80abed0 2024-09-13T16:35:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42dcb08700 1 -- 172.21.15.119:0/3880728381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f42d80cbb50 con 0x7f42d80abed0 2024-09-13T16:35:08.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21839d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8f50 0x2183f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c9870 0x21aa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x20af920 con 0x21c8f50 2024-09-13T16:35:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x20af5c0 con 0x21c8630 2024-09-13T16:35:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2081e10 con 0x21c9870 2024-09-13T16:35:08.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f42c8018c70 con 0x7f42d80abed0 2024-09-13T16:35:08.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8f50 0x2183f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8f50 0x2183f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49558/0 (socket says 172.21.15.119:49558) 2024-09-13T16:35:08.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 -- 172.21.15.119:0/1480073319 learned_addr learned my addr 172.21.15.119:0/1480073319 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 10 client.0 _ll_register_callbacks cb 0x20e5bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:08.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf172c700 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21839d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c9870 0x21aa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 -- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c9870 msgr2=0x21aa7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c9870 0x21aa7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 -- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 msgr2=0x21839d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21839d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f42c4066730 0x7f42c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 -- 172.21.15.119:0/1480073319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcec157bc0 con 0x21c8f50 2024-09-13T16:35:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f42c8084970 con 0x7f42d80abed0 2024-09-13T16:35:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8f50 0x2183f10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7efce000e4b0 tx=0x7efce0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3dc09317651b214 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efccbfff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:08.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+515 (secure 0 0 0) 0x7f42c8051c70 con 0x7f42d80abed0 2024-09-13T16:35:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42d6ffd700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f42c4066730 0x7f42c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efccbfff700 1 -- 172.21.15.119:0/1480073319 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce0034070 con 0x21c8f50 2024-09-13T16:35:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x1f20490 con 0x21c8f50 2024-09-13T16:35:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efccbfff700 1 -- 172.21.15.119:0/1480073319 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7efce0003950 con 0x21c8f50 2024-09-13T16:35:08.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42dcb08700 1 -- 172.21.15.119:0/3880728381 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({log-info=0+}) v3 -- 0x7f42d80cbe60 con 0x7f42d80abed0 2024-09-13T16:35:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efccbfff700 1 -- 172.21.15.119:0/1480073319 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7efce0021040 con 0x21c8f50 2024-09-13T16:35:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 10 client.9327 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x20b2600 con 0x21c8f50 2024-09-13T16:35:08.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 7 ==== log(2 entries from seq 1112 at 2024-09-13T16:35:06.501145+0000) v1 ==== 389+0+0 (secure 0 0 0) 0x7f42c800be20 con 0x7f42d80abed0 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: cluster: 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: id: 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: health: HEALTH_OK 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: services: 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: mon: 3 daemons, quorum a,b,c (age 10m) 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: mgr: x(active, since 10m), standbys: y 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: mds: 1/1 daemons up, 3 standby 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: osd: 8 osds: 8 up (since 9m), 8 in (since 10m) 2024-09-13T16:35:08.163 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: data: 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: volumes: 1/1 healthy 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: pools: 3 pools, 97 pgs 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: objects: 30 objects, 19 MiB 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: usage: 291 MiB used, 720 GiB / 720 GiB avail 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: pgs: 97 active+clean 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: io: 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: client: 1.6 KiB/s rd, 102 B/s wr, 1 op/s rd, 0 op/s wr 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:06.501145+0000 mon.a [INF] Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-13T16:35:08.164 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:06.501198+0000 mon.a [INF] Cluster is now healthy 2024-09-13T16:35:08.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efccbfff700 1 -- 172.21.15.119:0/1480073319 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x1f20490 con 0x21c8f50 2024-09-13T16:35:08.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efccbfff700 1 -- 172.21.15.119:0/1480073319 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efce0003b10 con 0x21c8f50 2024-09-13T16:35:08.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 10 client.9327 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:08.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 10 client.9327 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:08.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.144+0000 7efcf6c41740 20 client.9327 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:08.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x2185bd0 0x2185fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 4 client.9327 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:35:08.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x1f6a710 con 0x2185bd0 2024-09-13T16:35:08.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7f42d6ffd700 1 --2- 172.21.15.119:0/3880728381 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f42c4066730 0x7f42c4068bf0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f42cc007fd0 tx=0x7f42cc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf172c700 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x2185bd0 0x2185fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efccbfff700 10 client.9327 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf172c700 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x2185bd0 0x2185fe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54a6fea7ec96f1fe server_cookie=c6dae02cfbff3079 in_seq=0 out_seq=0 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efccbfff700 1 -- 172.21.15.119:0/1480073319 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x1f6a710 con 0x2185bd0 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efccbfff700 10 client.9327 handle_command_reply: tid=0 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 client.9327 shutdown 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8f50 msgr2=0x2183f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8f50 0x2183f10 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7efce000e4b0 tx=0x7efce0004770 comp rx=0 tx=0).stop 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 shutdown_connections 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x2185bd0 0x2185fe0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c9870 0x21aa7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efccbfff700 0 client.9327 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c8f50 0x2183f10 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1480073319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21839d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 >> 172.21.15.119:0/1480073319 conn(0x1f13780 msgr2=0x21b2500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 shutdown_connections 2024-09-13T16:35:08.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 1 -- 172.21.15.119:0/1480073319 wait complete. 2024-09-13T16:35:08.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.148+0000 7efcf6c41740 20 client.9327 trim_cache size 0 max 16384 2024-09-13T16:35:08.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.164+0000 7efcf6c41740 1 Processor -- start 2024-09-13T16:35:08.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.164+0000 7efcf6c41740 1 -- start start 2024-09-13T16:35:08.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21dba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21dfcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x20b0070 con 0x21c9870 2024-09-13T16:35:08.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x20ebd90 con 0x21c8630 2024-09-13T16:35:08.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x209bef0 con 0x21c8f50 2024-09-13T16:35:08.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49574/0 (socket says 172.21.15.119:49574) 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1163152879 learned_addr learned my addr 172.21.15.119:0/1163152879 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf172c700 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21dba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21dfcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 msgr2=0x21dfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21dfcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 msgr2=0x21dba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21dba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1163152879 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2098830 con 0x21c9870 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21dfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf172c700 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21dba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e0210 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7efcec06f0c0 tx=0x7efce4007040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c18a32fbaa0cc636 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcd37fe700 1 -- 172.21.15.119:0/1163152879 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce40016f0 con 0x21c9870 2024-09-13T16:35:08.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcd37fe700 1 -- 172.21.15.119:0/1163152879 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efce4005e90 con 0x21c9870 2024-09-13T16:35:08.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcd37fe700 1 -- 172.21.15.119:0/1163152879 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4014bb0 con 0x21c9870 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- 172.21.15.119:0/1163152879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 msgr2=0x21e0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e0210 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7efcec06f0c0 tx=0x7efce4007040 comp rx=0 tx=0).stop 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- 172.21.15.119:0/1163152879 shutdown_connections 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e0210 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8f50 0x21dfcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1163152879 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8630 0x21dba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- 172.21.15.119:0/1163152879 >> 172.21.15.119:0/1163152879 conn(0x1f13780 msgr2=0x21c76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- 172.21.15.119:0/1163152879 shutdown_connections 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- 172.21.15.119:0/1163152879 wait complete. 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 -- start start 2024-09-13T16:35:08.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.168+0000 7efcf6c41740 1 Processor -- start 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8630 0x222d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8f50 0x21e9420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x21ea6c0 con 0x21c9870 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x21c8120 con 0x21c8f50 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x21ea180 con 0x21c8630 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 10 client.0 _ll_register_callbacks cb 0x20eb150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49584/0 (socket says 172.21.15.119:49584) 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1232869965 learned_addr learned my addr 172.21.15.119:0/1232869965 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf172c700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8630 0x222d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8f50 0x21e9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8630 msgr2=0x222d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8630 0x222d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8f50 msgr2=0x21e9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8f50 0x21e9420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 -- 172.21.15.119:0/1232869965 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce400f040 con 0x21c9870 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf172c700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8630 0x222d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf0f2b700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8f50 0x21e9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf1f2d700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e9960 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x1ef35f0 tx=0x7efce4000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dc9ddcfc8ddeb1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efccbfff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efccbfff700 1 -- 172.21.15.119:0/1232869965 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efce4023070 con 0x21c9870 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efccbfff700 1 -- 172.21.15.119:0/1232869965 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7efce4016810 con 0x21c9870 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efcdc081070 con 0x21c9870 2024-09-13T16:35:08.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efccbfff700 1 -- 172.21.15.119:0/1232869965 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7efce40169d0 con 0x21c9870 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 10 client.9333 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x1f5ec10 con 0x21c9870 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efccbfff700 1 -- 172.21.15.119:0/1232869965 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcdc081070 con 0x21c9870 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efccbfff700 1 -- 172.21.15.119:0/1232869965 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7efce4012050 con 0x21c9870 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 10 client.9333 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 10 client.9333 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 20 client.9333 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x222d650 0x21e2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 4 client.9333 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}] 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["repair", "recursive"]}) v1 -- 0x1f915f0 con 0x222d650 2024-09-13T16:35:08.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf172c700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x222d650 0x21e2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efccbfff700 10 client.9333 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.172+0000 7efcf172c700 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x222d650 0x21e2520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4024ce27e023164f server_cookie=f82923f4c94e9482 in_seq=0 out_seq=0 2024-09-13T16:35:08.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efccbfff700 1 -- 172.21.15.119:0/1232869965 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x1f915f0 con 0x222d650 2024-09-13T16:35:08.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efccbfff700 10 client.9333 handle_command_reply: tid=0 2024-09-13T16:35:08.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 client.9333 shutdown 2024-09-13T16:35:08.182 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-09-13T16:35:08.182 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-09-13T16:35:08.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 msgr2=0x21e9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e9960 secure :-1 s=READY pgs=1705 cs=0 l=1 rev1=1 crypto rx=0x1ef35f0 tx=0x7efce4000fe0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.183 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-09-13T16:35:08.183 INFO:tasks.ceph.mds.c.smithi119.stderr:2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-09-13T16:35:08.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 shutdown_connections 2024-09-13T16:35:08.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x222d650 0x21e2520 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x21c9870 0x21e9960 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x21c8f50 0x21e9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 --2- 172.21.15.119:0/1232869965 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x21c8630 0x222d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efccbfff700 0 client.9333 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 >> 172.21.15.119:0/1232869965 conn(0x1f13780 msgr2=0x21b2670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 shutdown_connections 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 1 -- 172.21.15.119:0/1232869965 wait complete. 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf6c41740 20 client.9333 trim_cache size 0 max 16384 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0, 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "scrub_tag": "89b34e08-6f35-4706-9fcd-ca95b56ed49f", 2024-09-13T16:35:08.184 INFO:teuthology.orchestra.run.smithi119.stdout: "mode": "asynchronous" 2024-09-13T16:35:08.185 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:35:08.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efcd8066730 msgr2=0x7efcd8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efcd8066730 0x7efcd8068bf0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7efce00090b0 tx=0x7efce001e040 comp rx=0 tx=0).stop 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 msgr2=0x7efcec1586e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcec1586e0 secure :-1 s=READY pgs=1699 cs=0 l=1 rev1=1 crypto rx=0x7efcdc0101c0 tx=0x7efcdc0104d0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 shutdown_connections 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efcd8066730 0x7efcd8068bf0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efcec06ab20 0x7efcec06afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efcec156940 0x7efcec14f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.176+0000 7efcf3990700 1 --2- 172.21.15.119:0/2054875493 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efcd4004870 0x7efcec1586e0 unknown :-1 s=CLOSED pgs=1699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.180+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 >> 172.21.15.119:0/2054875493 conn(0x7efcec0b4040 msgr2=0x7efcec1552c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.180+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 shutdown_connections 2024-09-13T16:35:08.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.180+0000 7efcf3990700 1 -- 172.21.15.119:0/2054875493 wait complete. 2024-09-13T16:35:08.197 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-09-13T16:35:08.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6ede6700 1 Processor -- start 2024-09-13T16:35:08.316 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6ede6700 1 -- start start 2024-09-13T16:35:08.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6ede6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c6806c480 0x7f9c680cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6ede6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9c680ce000 0x7f9c680b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6ede6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c6806c8c0 con 0x7f9c68144180 2024-09-13T16:35:08.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6ede6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c680b1c50 con 0x7f9c6806c480 2024-09-13T16:35:08.317 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6ede6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c6806cb40 con 0x7f9c680ce000 2024-09-13T16:35:08.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c6806c480 0x7f9c680cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9c6806c480 0x7f9c680cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49594/0 (socket says 172.21.15.119:49594) 2024-09-13T16:35:08.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/437191508 learned_addr learned my addr 172.21.15.119:0/437191508 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.318 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/437191508 >> v2:172.21.15.145:3300/0 conn(0x7f9c680ce000 0x7f9c680b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/437191508 >> v2:172.21.15.145:3300/0 conn(0x7f9c680ce000 msgr2=0x7f9c680b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/437191508 >> v2:172.21.15.145:3300/0 conn(0x7f9c680ce000 0x7f9c680b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/437191508 >> v1:172.21.15.145:6789/0 conn(0x7f9c68144180 legacy=0x7f9c680cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/437191508 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c6806ce20 con 0x7f9c6806c480 2024-09-13T16:35:08.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.312+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/437191508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c6806c480 0x7f9c680cdac0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f9c680ce760 tx=0x7f9c58009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21d63faa6a859bb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6cde2700 1 -- 172.21.15.119:0/437191508 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58011830 con 0x7f9c6806c480 2024-09-13T16:35:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6cde2700 1 -- 172.21.15.119:0/437191508 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c58011990 con 0x7f9c6806c480 2024-09-13T16:35:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6cde2700 1 -- 172.21.15.119:0/437191508 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58011d20 con 0x7f9c6806c480 2024-09-13T16:35:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- 172.21.15.119:0/437191508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c6806c480 msgr2=0x7f9c680cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/437191508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c6806c480 0x7f9c680cdac0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f9c680ce760 tx=0x7f9c58009bf0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- 172.21.15.119:0/437191508 shutdown_connections 2024-09-13T16:35:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/437191508 >> v2:172.21.15.145:3300/0 conn(0x7f9c680ce000 0x7f9c680b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/437191508 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c6806c480 0x7f9c680cdac0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- 172.21.15.119:0/437191508 >> 172.21.15.119:0/437191508 conn(0x7f9c680b0980 msgr2=0x7f9c680b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- 172.21.15.119:0/437191508 shutdown_connections 2024-09-13T16:35:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- 172.21.15.119:0/437191508 wait complete. 2024-09-13T16:35:08.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 Processor -- start 2024-09-13T16:35:08.322 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- start start 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c6806c480 0x7f9c6813cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c680ce000 0x7f9c680ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c68144180 0x7f9c68130090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6813d490 con 0x7f9c680ce000 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6813f8d0 con 0x7f9c6806c480 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6ede6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6813fbb0 con 0x7f9c68144180 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6d5e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c680ce000 0x7f9c680ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.316+0000 7f9c6d5e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c680ce000 0x7f9c680ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49602/0 (socket says 172.21.15.119:49602) 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/3175009390 learned_addr learned my addr 172.21.15.119:0/3175009390 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c6806c480 0x7f9c6813cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c68144180 0x7f9c68130090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c68144180 msgr2=0x7f9c68130090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c68144180 0x7f9c68130090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c6806c480 msgr2=0x7f9c6813cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c6806c480 0x7f9c6813cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/3175009390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c58009570 con 0x7f9c680ce000 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c68144180 0x7f9c68130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c680ce000 0x7f9c680ad310 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f9c58000c00 tx=0x7f9c58004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c667fc700 1 -- 172.21.15.119:0/3175009390 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58004610 con 0x7f9c680ce000 2024-09-13T16:35:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c667fc700 1 -- 172.21.15.119:0/3175009390 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5801e070 con 0x7f9c680ce000 2024-09-13T16:35:08.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c6813fe90 con 0x7f9c680ce000 2024-09-13T16:35:08.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c667fc700 1 -- 172.21.15.119:0/3175009390 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c580038e0 con 0x7f9c680ce000 2024-09-13T16:35:08.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.320+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c68140450 con 0x7f9c680ce000 2024-09-13T16:35:08.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 Processor -- start 2024-09-13T16:35:08.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 -- start start 2024-09-13T16:35:08.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c667fc700 1 -- 172.21.15.119:0/3175009390 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9c58003a40 con 0x7f9c680ce000 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1ada810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b02520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b24550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a6b010 con 0x1b23d90 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19f8f40 con 0x1ad9310 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a4be30 con 0x1b24170 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1ada810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1ada810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49604/0 (socket says 172.21.15.119:49604) 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 -- 172.21.15.119:0/639913065 learned_addr learned my addr 172.21.15.119:0/639913065 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b24550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b02520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 -- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 msgr2=0x1b24550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b24550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 -- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 msgr2=0x1b02520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b02520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 -- 172.21.15.119:0/639913065 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x18b7a40 con 0x1b23d90 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b02520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c667fc700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c54066730 0x7f9c54068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1ada810 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f9c600074c0 tx=0x7f9c60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24e7cef949ebe821 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c667fc700 1 -- 172.21.15.119:0/3175009390 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f9c58085200 con 0x7f9c680ce000 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.324+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c54066730 0x7f9c54068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c4f7fe700 1 -- 172.21.15.119:0/639913065 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c60015670 con 0x1b23d90 2024-09-13T16:35:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c4f7fe700 1 -- 172.21.15.119:0/639913065 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6000f050 con 0x1b23d90 2024-09-13T16:35:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c4f7fe700 1 -- 172.21.15.119:0/639913065 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c600159f0 con 0x1b23d90 2024-09-13T16:35:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- 172.21.15.119:0/639913065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 msgr2=0x1ada810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1ada810 secure :-1 s=READY pgs=1708 cs=0 l=1 rev1=1 crypto rx=0x7f9c600074c0 tx=0x7f9c60009750 comp rx=0 tx=0).stop 2024-09-13T16:35:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c54066730 0x7f9c54068bf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9c680a8b30 tx=0x7f9c5c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- 172.21.15.119:0/639913065 shutdown_connections 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b24550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b02520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 --2- 172.21.15.119:0/639913065 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1ada810 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- 172.21.15.119:0/639913065 >> 172.21.15.119:0/639913065 conn(0x186c5b0 msgr2=0x1addb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- 172.21.15.119:0/639913065 shutdown_connections 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- 172.21.15.119:0/639913065 wait complete. 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- start start 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 Processor -- start 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ad9310 0x1b01d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b23d90 0x1b223e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1b24170 0x1b00dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a47150 con 0x1ad9310 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a44eb0 con 0x1b24170 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a028e0 con 0x1b23d90 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 10 client.0 _ll_register_callbacks cb 0x19e1670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ad9310 0x1b01d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ad9310 0x1b01d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49614/0 (socket says 172.21.15.119:49614) 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 -- 172.21.15.119:0/787909579 learned_addr learned my addr 172.21.15.119:0/787909579 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b23d90 0x1b223e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1b24170 0x1b00dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 -- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b23d90 msgr2=0x1b223e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b23d90 0x1b223e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 -- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1b24170 msgr2=0x1b00dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1b24170 0x1b00dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 -- 172.21.15.119:0/787909579 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c60009df0 con 0x1ad9310 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b23d90 0x1b223e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1b24170 0x1b00dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ad9310 0x1b01d60 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c001a60 tx=0x7f9c5c000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e47d2e541ae81cca server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c4cff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c4cff9700 1 -- 172.21.15.119:0/787909579 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c014ca0 con 0x1ad9310 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c4cff9700 1 -- 172.21.15.119:0/787909579 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f9c5c01c050 con 0x1ad9310 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.328+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x18792c0 con 0x1ad9310 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c4cff9700 1 -- 172.21.15.119:0/787909579 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c5c014e00 con 0x1ad9310 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 10 client.9345 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x1a0b290 con 0x1ad9310 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c4cff9700 1 -- 172.21.15.119:0/787909579 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x18792c0 con 0x1ad9310 2024-09-13T16:35:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c4cff9700 1 -- 172.21.15.119:0/787909579 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9c5c02c3e0 con 0x1ad9310 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 10 client.9345 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 10 client.9345 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 20 client.9345 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b26eb0 0x1b272c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 4 client.9345 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x18c3540 con 0x1b26eb0 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b26eb0 0x1b272c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b26eb0 0x1b272c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adbd036bf166009f server_cookie=62a0423e043a8d55 in_seq=0 out_seq=0 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c4cff9700 10 client.9345 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c4cff9700 1 -- 172.21.15.119:0/787909579 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x18c3540 con 0x1b26eb0 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c4cff9700 10 client.9345 handle_command_reply: tid=0 2024-09-13T16:35:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 client.9345 shutdown 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ad9310 msgr2=0x1b01d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ad9310 0x1b01d60 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c001a60 tx=0x7f9c5c000ed0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 shutdown_connections 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b26eb0 0x1b272c0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1b24170 0x1b00dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b23d90 0x1b223e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 --2- 172.21.15.119:0/787909579 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1ad9310 0x1b01d60 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c4cff9700 0 client.9345 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 >> 172.21.15.119:0/787909579 conn(0x186c5b0 msgr2=0x1b024b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 shutdown_connections 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 1 -- 172.21.15.119:0/787909579 wait complete. 2024-09-13T16:35:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.332+0000 7f9c732f9740 20 client.9345 trim_cache size 0 max 16384 2024-09-13T16:35:08.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 Processor -- start 2024-09-13T16:35:08.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 -- start start 2024-09-13T16:35:08.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b005c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b00b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b36890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x18f0b80 con 0x1b23d90 2024-09-13T16:35:08.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19e3620 con 0x1ad9310 2024-09-13T16:35:08.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.348+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a15c00 con 0x1b24170 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b00b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b00b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49630/0 (socket says 172.21.15.119:49630) 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6dde4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b005c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/1074601011 learned_addr learned my addr 172.21.15.119:0/1074601011 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b36890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 msgr2=0x1b36890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b36890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 msgr2=0x1b005c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b005c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/1074601011 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x19f14c0 con 0x1b23d90 2024-09-13T16:35:08.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b36890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:08.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b00b00 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009150 tx=0x7f9c5805c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab42b60d2474039e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c4ffff700 1 -- 172.21.15.119:0/1074601011 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58055bd0 con 0x1b23d90 2024-09-13T16:35:08.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c4ffff700 1 -- 172.21.15.119:0/1074601011 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c58059030 con 0x1b23d90 2024-09-13T16:35:08.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c4ffff700 1 -- 172.21.15.119:0/1074601011 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c580559f0 con 0x1b23d90 2024-09-13T16:35:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 -- 172.21.15.119:0/1074601011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 msgr2=0x1b00b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b00b00 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f9c58009150 tx=0x7f9c5805c040 comp rx=0 tx=0).stop 2024-09-13T16:35:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 -- 172.21.15.119:0/1074601011 shutdown_connections 2024-09-13T16:35:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b36890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b00b00 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 --2- 172.21.15.119:0/1074601011 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 -- 172.21.15.119:0/1074601011 >> 172.21.15.119:0/1074601011 conn(0x186c5b0 msgr2=0x1affeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 -- 172.21.15.119:0/1074601011 shutdown_connections 2024-09-13T16:35:08.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 -- 172.21.15.119:0/1074601011 wait complete. 2024-09-13T16:35:08.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 -- start start 2024-09-13T16:35:08.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.352+0000 7f9c732f9740 1 Processor -- start 2024-09-13T16:35:08.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b83ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b84420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b495e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a426c0 con 0x1b23d90 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19cea50 con 0x1ad9310 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1b227d0 con 0x1b24170 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b84420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 10 client.0 _ll_register_callbacks cb 0x1a43610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b84420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49634/0 (socket says 172.21.15.119:49634) 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/745854455 learned_addr learned my addr 172.21.15.119:0/745854455 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b83ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6e5e5700 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 msgr2=0x1b495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b495e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 msgr2=0x1b83ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b83ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 -- 172.21.15.119:0/745854455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c68140450 con 0x1b23d90 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6d5e3700 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b84420 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f9c58056020 tx=0x7f9c5c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a10e7af0f71b4d69 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c4cff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c4cff9700 1 -- 172.21.15.119:0/745854455 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c58056090 con 0x1b23d90 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c4cff9700 1 -- 172.21.15.119:0/745854455 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f9c58059ec0 con 0x1b23d90 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c6000f050 con 0x1b23d90 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c4cff9700 1 -- 172.21.15.119:0/745854455 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c5805e480 con 0x1b23d90 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 10 client.9351 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x19e5790 con 0x1b23d90 2024-09-13T16:35:08.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c4cff9700 1 -- 172.21.15.119:0/745854455 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c6000f050 con 0x1b23d90 2024-09-13T16:35:08.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c4cff9700 1 -- 172.21.15.119:0/745854455 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f9c580527a0 con 0x1b23d90 2024-09-13T16:35:08.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 10 client.9351 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:08.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 10 client.9351 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:08.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 20 client.9351 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b49b20 0x1b39980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 4 client.9351 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x18e82c0 con 0x1b49b20 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.356+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b49b20 0x1b39980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c4cff9700 10 client.9351 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6dde4700 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b49b20 0x1b39980 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72a9eac79843b207 server_cookie=9cfcb6a5b2862a8f in_seq=0 out_seq=0 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c4cff9700 1 -- 172.21.15.119:0/745854455 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+260 (crc 0 0 0) 0x18e82c0 con 0x1b49b20 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c4cff9700 10 client.9351 handle_command_reply: tid=0 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 client.9351 shutdown 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 msgr2=0x1b84420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b84420 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f9c58056020 tx=0x7f9c5c013000 comp rx=0 tx=0).stop 2024-09-13T16:35:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 shutdown_connections 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x1b49b20 0x1b39980 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1b24170 0x1b495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c4cff9700 0 client.9351 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1b23d90 0x1b84420 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 --2- 172.21.15.119:0/745854455 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1ad9310 0x1b83ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 >> 172.21.15.119:0/745854455 conn(0x186c5b0 msgr2=0x1affeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 shutdown_connections 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 1 -- 172.21.15.119:0/745854455 wait complete. 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c732f9740 20 client.9351 trim_cache size 0 max 16384 2024-09-13T16:35:08.367 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "IDLE (waiting for more scrubs)", 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": { 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: "89b34e08-6f35-4706-9fcd-ca95b56ed49f": { 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: "path": "/", 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: "tag": "89b34e08-6f35-4706-9fcd-ca95b56ed49f", 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: "options": "recursive,repair" 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:08.368 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:35:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c54066730 msgr2=0x7f9c54068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c54066730 0x7f9c54068bf0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9c680a8b30 tx=0x7f9c5c009040 comp rx=0 tx=0).stop 2024-09-13T16:35:08.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c680ce000 msgr2=0x7f9c680ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c680ce000 0x7f9c680ad310 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f9c58000c00 tx=0x7f9c58004770 comp rx=0 tx=0).stop 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 shutdown_connections 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9c54066730 0x7f9c54068bf0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9c68144180 0x7f9c68130090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9c680ce000 0x7f9c680ad310 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 --2- 172.21.15.119:0/3175009390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9c6806c480 0x7f9c6813cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 >> 172.21.15.119:0/3175009390 conn(0x7f9c680b0980 msgr2=0x7f9c6815c460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 shutdown_connections 2024-09-13T16:35:08.370 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:08.360+0000 7f9c6ede6700 1 -- 172.21.15.119:0/3175009390 wait complete. 2024-09-13T16:35:08.379 INFO:tasks.cephfs.filesystem:scrub status for tag:89b34e08-6f35-4706-9fcd-ca95b56ed49f - {'path': '/', 'tag': '89b34e08-6f35-4706-9fcd-ca95b56ed49f', 'options': 'recursive,repair'} 2024-09-13T16:35:09.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:09.520+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 8 ==== log(8 entries from seq 1 at 2024-09-13T16:35:08.179173+0000) v1 ==== 1407+0+0 (secure 0 0 0) 0x7f42c8084790 con 0x7f42d80abed0 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.179173+0000 mds.c [INF] scrub queued for path: / 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.179182+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.179195+0000 mds.c [INF] scrub summary: active paths [/] 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.180167+0000 mds.c [ERR] scrub: inode wrongly marked free: 0x10000000001 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.180179+0000 mds.c [ERR] inode table repaired for inode: 0x10000000001 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.180709+0000 mds.c [ERR] scrub: inode wrongly marked free: 0x10000000002 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.180722+0000 mds.c [ERR] inode table repaired for inode: 0x10000000002 2024-09-13T16:35:09.525 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:08.181918+0000 mds.c [INF] scrub summary: idle+waiting paths [/] 2024-09-13T16:35:13.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:13.692+0000 7f42bffff700 1 -- 172.21.15.119:0/3880728381 <== mon.0 v2:172.21.15.119:3300/0 9 ==== log(1 entries from seq 9 at 2024-09-13T16:35:12.526071+0000) v1 ==== 189+0+0 (secure 0 0 0) 0x7f42c8082e50 con 0x7f42d80abed0 2024-09-13T16:35:13.697 INFO:teuthology.orchestra.run.smithi119.stdout:2024-09-13T16:35:12.526071+0000 mds.c [INF] scrub summary: idle 2024-09-13T16:35:28.441 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:35:28.444 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:35:38.380 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.7:0 scrub status 2024-09-13T16:35:38.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.503+0000 7f1fe6b79700 1 Processor -- start 2024-09-13T16:35:38.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.503+0000 7f1fe6b79700 1 -- start start 2024-09-13T16:35:38.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.503+0000 7f1fe6b79700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1fe00ade80 0x7f1fe01652b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.503+0000 7f1fe6b79700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe00b28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.503+0000 7f1fe6b79700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe0071ac0 con 0x7f1fe01612d0 2024-09-13T16:35:38.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.503+0000 7f1fe6b79700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe01657f0 con 0x7f1fe00ade80 2024-09-13T16:35:38.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.503+0000 7f1fe6b79700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe00b2e20 con 0x7f1fe00b24a0 2024-09-13T16:35:38.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe00b28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5b77700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1fe00ade80 0x7f1fe01652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe00b28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:48108/0 (socket says 172.21.15.119:48108) 2024-09-13T16:35:38.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 -- 172.21.15.119:0/2067603779 learned_addr learned my addr 172.21.15.119:0/2067603779 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 -- 172.21.15.119:0/2067603779 >> v2:172.21.15.145:3300/0 conn(0x7f1fe00ade80 msgr2=0x7f1fe01652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 --2- 172.21.15.119:0/2067603779 >> v2:172.21.15.145:3300/0 conn(0x7f1fe00ade80 0x7f1fe01652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 -- 172.21.15.119:0/2067603779 >> v1:172.21.15.145:6789/0 conn(0x7f1fe01612d0 legacy=0x7f1fe0161780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 -- 172.21.15.119:0/2067603779 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe0070eb0 con 0x7f1fe00b24a0 2024-09-13T16:35:38.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe5376700 1 --2- 172.21.15.119:0/2067603779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe00b28e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009910 tx=0x7f1fd0009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94959dde030aa020 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe4b75700 1 -- 172.21.15.119:0/2067603779 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0011830 con 0x7f1fe00b24a0 2024-09-13T16:35:38.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe4b75700 1 -- 172.21.15.119:0/2067603779 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd0011990 con 0x7f1fe00b24a0 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe4b75700 1 -- 172.21.15.119:0/2067603779 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0011d40 con 0x7f1fe00b24a0 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2067603779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 msgr2=0x7f1fe00b28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2067603779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe00b28e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f1fd0009910 tx=0x7f1fd0009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2067603779 shutdown_connections 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2067603779 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe00b28e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2067603779 >> v2:172.21.15.145:3300/0 conn(0x7f1fe00ade80 0x7f1fe01652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2067603779 >> 172.21.15.119:0/2067603779 conn(0x7f1fe00b0980 msgr2=0x7f1fe00b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2067603779 shutdown_connections 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.507+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2067603779 wait complete. 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 Processor -- start 2024-09-13T16:35:38.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 -- start start 2024-09-13T16:35:38.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe00ade80 0x7f1fe01eaa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe01593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe01612d0 0x7f1fe0146400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe01598f0 con 0x7f1fe01612d0 2024-09-13T16:35:38.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe0159df0 con 0x7f1fe00ade80 2024-09-13T16:35:38.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe015d5c0 con 0x7f1fe00b24a0 2024-09-13T16:35:38.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe01612d0 0x7f1fe0146400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe01612d0 0x7f1fe0146400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56756/0 (socket says 172.21.15.119:56756) 2024-09-13T16:35:38.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 -- 172.21.15.119:0/2821205152 learned_addr learned my addr 172.21.15.119:0/2821205152 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe5376700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe01593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe00ade80 0x7f1fe01eaa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 -- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 msgr2=0x7f1fe01593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe01593b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 -- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe00ade80 msgr2=0x7f1fe01eaa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe00ade80 0x7f1fe01eaa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 -- 172.21.15.119:0/2821205152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fd0009570 con 0x7f1fe01612d0 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6378700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe01612d0 0x7f1fe0146400 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800cc50 tx=0x7f1fd800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fd67fc700 1 -- 172.21.15.119:0/2821205152 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd8003c20 con 0x7f1fe01612d0 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fd67fc700 1 -- 172.21.15.119:0/2821205152 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8003d80 con 0x7f1fe01612d0 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe015a340 con 0x7f1fe01612d0 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fd67fc700 1 -- 172.21.15.119:0/2821205152 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd800aab0 con 0x7f1fe01612d0 2024-09-13T16:35:38.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.511+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe00cc3d0 con 0x7f1fe01612d0 2024-09-13T16:35:38.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.515+0000 7f1fd67fc700 1 -- 172.21.15.119:0/2821205152 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1fd800ac10 con 0x7f1fe01612d0 2024-09-13T16:35:38.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.515+0000 7f1fd67fc700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fc4066730 0x7f1fc4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.515+0000 7f1fd67fc700 1 -- 172.21.15.119:0/2821205152 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f1fd80842b0 con 0x7f1fe01612d0 2024-09-13T16:35:38.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fc4066730 0x7f1fc4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fc4066730 0x7f1fc4068bf0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1fe006cbb0 tx=0x7f1fdc009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 Processor -- start 2024-09-13T16:35:38.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 -- start start 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25bfd90 0x2576810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25c0170 0x25c0550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2507010 con 0x25c0170 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2494f40 con 0x25bfd90 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x24e7e30 con 0x2575310 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25c0170 0x25c0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25c0170 0x25c0550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56762/0 (socket says 172.21.15.119:56762) 2024-09-13T16:35:38.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 -- 172.21.15.119:0/1715971923 learned_addr learned my addr 172.21.15.119:0/1715971923 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25bfd90 0x2576810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe5376700 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 -- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 msgr2=0x259e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 -- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25bfd90 msgr2=0x2576810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25bfd90 0x2576810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 -- 172.21.15.119:0/1715971923 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2353a40 con 0x25c0170 2024-09-13T16:35:38.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25bfd90 0x2576810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:38.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fe6378700 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25c0170 0x25c0550 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f1fd805d000 tx=0x7f1fd8054d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5bf93da320ca658 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.519+0000 7f1fc37fe700 1 -- 172.21.15.119:0/1715971923 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd806c070 con 0x25c0170 2024-09-13T16:35:38.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1fc37fe700 1 -- 172.21.15.119:0/1715971923 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8059e90 con 0x25c0170 2024-09-13T16:35:38.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1fc37fe700 1 -- 172.21.15.119:0/1715971923 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80840c0 con 0x25c0170 2024-09-13T16:35:38.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- 172.21.15.119:0/1715971923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25c0170 msgr2=0x25c0550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25c0170 0x25c0550 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f1fd805d000 tx=0x7f1fd8054d80 comp rx=0 tx=0).stop 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- 172.21.15.119:0/1715971923 shutdown_connections 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25c0170 0x25c0550 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 --2- 172.21.15.119:0/1715971923 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25bfd90 0x2576810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- 172.21.15.119:0/1715971923 >> 172.21.15.119:0/1715971923 conn(0x23085b0 msgr2=0x2579b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- 172.21.15.119:0/1715971923 shutdown_connections 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- 172.21.15.119:0/1715971923 wait complete. 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- start start 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 Processor -- start 2024-09-13T16:35:38.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2599920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25c0170 0x2599e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x24e3150 con 0x25bfd90 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x24e0eb0 con 0x25c0170 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x249e8e0 con 0x2575310 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1feb08c740 10 client.0 _ll_register_callbacks cb 0x247d670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1fe5376700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2599920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1fe5376700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2599920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56772/0 (socket says 172.21.15.119:56772) 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1fe5376700 1 -- 172.21.15.119:0/352588687 learned_addr learned my addr 172.21.15.119:0/352588687 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.523+0000 7f1fe6378700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25c0170 0x2599e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5376700 1 -- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 msgr2=0x259cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5376700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259cb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5376700 1 -- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25c0170 msgr2=0x2599e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5376700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25c0170 0x2599e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5376700 1 -- 172.21.15.119:0/352588687 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe00cc3d0 con 0x25bfd90 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe6378700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25c0170 0x2599e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5376700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2599920 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0165930 tx=0x7f1fd00047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e20097c1a1edaa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/352588687 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0004450 con 0x25bfd90 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/352588687 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f1fd00045b0 con 0x25bfd90 2024-09-13T16:35:38.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x23152c0 con 0x25bfd90 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/352588687 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd00036c0 con 0x25bfd90 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 10 client.9360 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x24a7290 con 0x25bfd90 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/352588687 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x23152c0 con 0x25bfd90 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/352588687 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1fd0023070 con 0x25bfd90 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 10 client.9360 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 10 client.9360 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 20 client.9360 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:38.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25984b0 0x25c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 4 client.9360 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:35:38.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x235f540 con 0x25984b0 2024-09-13T16:35:38.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25984b0 0x25c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 10 client.9360 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:38.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25984b0 0x25c98a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=107587e666848eaa server_cookie=f6ba05f102d3d9e1 in_seq=0 out_seq=0 2024-09-13T16:35:38.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/352588687 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x235f540 con 0x25984b0 2024-09-13T16:35:38.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 10 client.9360 handle_command_reply: tid=0 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 client.9360 shutdown 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 msgr2=0x2599920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2599920 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f1fe0165930 tx=0x7f1fd00047e0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 shutdown_connections 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25984b0 0x25c98a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x25c0170 0x2599e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2599920 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 --2- 172.21.15.119:0/352588687 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x259cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1fc0ff9700 0 client.9360 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:38.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 >> 172.21.15.119:0/352588687 conn(0x23085b0 msgr2=0x259e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:38.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 shutdown_connections 2024-09-13T16:35:38.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.527+0000 7f1feb08c740 1 -- 172.21.15.119:0/352588687 wait complete. 2024-09-13T16:35:38.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.531+0000 7f1feb08c740 20 client.9360 trim_cache size 0 max 16384 2024-09-13T16:35:38.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 Processor -- start 2024-09-13T16:35:38.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 -- start start 2024-09-13T16:35:38.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x25d8210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x2595de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2596320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x238cb80 con 0x25bfd90 2024-09-13T16:35:38.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x257af40 con 0x23085b0 2024-09-13T16:35:38.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x24b1c00 con 0x2575310 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2596320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2596320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56784/0 (socket says 172.21.15.119:56784) 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 -- 172.21.15.119:0/4100248070 learned_addr learned my addr 172.21.15.119:0/4100248070 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x25d8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe5376700 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x2595de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 -- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 msgr2=0x2595de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x2595de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 -- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 msgr2=0x25d8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x25d8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 -- 172.21.15.119:0/4100248070 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x248d4c0 con 0x25bfd90 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe5376700 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x2595de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fe6378700 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2596320 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800cc20 tx=0x7f1fd8084050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27011dca5891a5b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fc3fff700 1 -- 172.21.15.119:0/4100248070 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd806c040 con 0x25bfd90 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fc3fff700 1 -- 172.21.15.119:0/4100248070 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8051480 con 0x25bfd90 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1fc3fff700 1 -- 172.21.15.119:0/4100248070 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd80840c0 con 0x25bfd90 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 -- 172.21.15.119:0/4100248070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 msgr2=0x2596320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.547+0000 7f1feb08c740 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2596320 secure :-1 s=READY pgs=1715 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800cc20 tx=0x7f1fd8084050 comp rx=0 tx=0).stop 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- 172.21.15.119:0/4100248070 shutdown_connections 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2596320 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x2595de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 --2- 172.21.15.119:0/4100248070 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x25d8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- 172.21.15.119:0/4100248070 >> 172.21.15.119:0/4100248070 conn(0x2579f10 msgr2=0x2598b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:38.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- 172.21.15.119:0/4100248070 shutdown_connections 2024-09-13T16:35:38.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- 172.21.15.119:0/4100248070 wait complete. 2024-09-13T16:35:38.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- start start 2024-09-13T16:35:38.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 Processor -- start 2024-09-13T16:35:38.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x2620d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x26212b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2621c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x25df620 con 0x25bfd90 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x26217f0 con 0x23085b0 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x25d7cd0 con 0x2575310 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 10 client.0 _ll_register_callbacks cb 0x24df610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe5b77700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x2620d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe6378700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2621c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe5b77700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x2620d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:43318/0 (socket says 172.21.15.119:43318) 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe5b77700 1 -- 172.21.15.119:0/3651268972 learned_addr learned my addr 172.21.15.119:0/3651268972 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe5376700 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x26212b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe6378700 1 -- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 msgr2=0x26212b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe6378700 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x26212b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe6378700 1 -- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 msgr2=0x2620d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe6378700 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x2620d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe6378700 1 -- 172.21.15.119:0/3651268972 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x2353a40 con 0x25bfd90 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x2620d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fe6378700 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2621c20 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8005010 tx=0x7f1fd8001250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b8a79d1d25f1133 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fc0ff9700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/3651268972 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fd806c040 con 0x25bfd90 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/3651268972 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f1fd8058f70 con 0x25bfd90 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1fd8059e90 con 0x25bfd90 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/3651268972 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=82) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fd805d530 con 0x25bfd90 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 10 client.9366 fetch_fsmap learned FSMap version 82 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=82,monmap=2+}) v3 -- 0x2481790 con 0x25bfd90 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/3651268972 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fd8059e90 con 0x25bfd90 2024-09-13T16:35:38.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.551+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/3651268972 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 82) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f1fd8082200 con 0x25bfd90 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 10 client.9366 fetch_fsmap finished waiting for FSMap version 82 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 10 client.9366 resolve_mds: resolved 7:0 to role '7:0' aka daemon mds.c 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 20 client.9366 populate_metadata read hostname 'smithi119' 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25c0170 0x2622160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 4 client.9366 mds_command: new command op to 9228 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 --> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x23842c0 con 0x25c0170 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25c0170 0x2622160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fc0ff9700 10 client.9366 ms_handle_connect on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe5b77700 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25c0170 0x2622160 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=289179d5ab662520 server_cookie=cd53bb87332dad20 in_seq=0 out_seq=0 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fc0ff9700 1 -- 172.21.15.119:0/3651268972 <== mds.0 v2:172.21.15.119:6834/912842245 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x23842c0 con 0x25c0170 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fc0ff9700 10 client.9366 handle_command_reply: tid=0 2024-09-13T16:35:38.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 client.9366 shutdown 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 msgr2=0x2621c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2621c20 secure :-1 s=READY pgs=1716 cs=0 l=1 rev1=1 crypto rx=0x7f1fd8005010 tx=0x7f1fd8001250 comp rx=0 tx=0).stop 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 shutdown_connections 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:6834/912842245,v1:172.21.15.119:6835/912842245] conn(0x25c0170 0x2622160 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x25bfd90 0x2621c20 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x2575310 0x26212b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 --2- 172.21.15.119:0/3651268972 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x23085b0 0x2620d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fc0ff9700 0 client.9366 ms_handle_reset on v2:172.21.15.119:6834/912842245 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 >> 172.21.15.119:0/3651268972 conn(0x2579f10 msgr2=0x2595de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:38.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 shutdown_connections 2024-09-13T16:35:38.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 1 -- 172.21.15.119:0/3651268972 wait complete. 2024-09-13T16:35:38.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1feb08c740 20 client.9366 trim_cache size 0 max 16384 2024-09-13T16:35:38.566 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:35:38.566 INFO:teuthology.orchestra.run.smithi119.stdout: "status": "no active scrubs running", 2024-09-13T16:35:38.566 INFO:teuthology.orchestra.run.smithi119.stdout: "scrubs": {} 2024-09-13T16:35:38.566 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:35:38.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fc4066730 msgr2=0x7f1fc4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fc4066730 0x7f1fc4068bf0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f1fe006cbb0 tx=0x7f1fdc009240 comp rx=0 tx=0).stop 2024-09-13T16:35:38.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe01612d0 msgr2=0x7f1fe0146400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe01612d0 0x7f1fe0146400 secure :-1 s=READY pgs=1712 cs=0 l=1 rev1=1 crypto rx=0x7f1fd800cc50 tx=0x7f1fd800cf60 comp rx=0 tx=0).stop 2024-09-13T16:35:38.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 shutdown_connections 2024-09-13T16:35:38.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fc4066730 0x7f1fc4068bf0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1fe01612d0 0x7f1fe0146400 unknown :-1 s=CLOSED pgs=1712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1fe00b24a0 0x7f1fe01593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 --2- 172.21.15.119:0/2821205152 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1fe00ade80 0x7f1fe01eaa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 >> 172.21.15.119:0/2821205152 conn(0x7f1fe00b0980 msgr2=0x7f1fe00b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:38.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 shutdown_connections 2024-09-13T16:35:38.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.555+0000 7f1fe6b79700 1 -- 172.21.15.119:0/2821205152 wait complete. 2024-09-13T16:35:38.576 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-09-13T16:35:38.678 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:35:38.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.791+0000 7f3e59e8c700 1 Processor -- start 2024-09-13T16:35:38.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.791+0000 7f3e59e8c700 1 -- start start 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.791+0000 7f3e59e8c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3e54156ed0 0x7f3e5406e520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.791+0000 7f3e59e8c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3e540b42d0 0x7f3e54071620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.791+0000 7f3e59e8c700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e540aec60 con 0x7f3e541572b0 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.791+0000 7f3e59e8c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5415e0a0 con 0x7f3e540b42d0 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.791+0000 7f3e59e8c700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e540b47b0 con 0x7f3e54156ed0 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3e540b42d0 0x7f3e54071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3e540b42d0 0x7f3e54071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56806/0 (socket says 172.21.15.119:56806) 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 -- 172.21.15.119:0/2621285322 learned_addr learned my addr 172.21.15.119:0/2621285322 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e58e8a700 1 --2- 172.21.15.119:0/2621285322 >> v2:172.21.15.145:3300/0 conn(0x7f3e54156ed0 0x7f3e5406e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 -- 172.21.15.119:0/2621285322 >> v2:172.21.15.145:3300/0 conn(0x7f3e54156ed0 msgr2=0x7f3e5406e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:38.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 --2- 172.21.15.119:0/2621285322 >> v2:172.21.15.145:3300/0 conn(0x7f3e54156ed0 0x7f3e5406e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 -- 172.21.15.119:0/2621285322 >> v1:172.21.15.145:6789/0 conn(0x7f3e541572b0 legacy=0x7f3e5406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e537fe700 1 -- 172.21.15.119:0/2621285322 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 333741573 0 0) 0x7f3e540aec60 con 0x7f3e541572b0 2024-09-13T16:35:38.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 -- 172.21.15.119:0/2621285322 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e540b4a90 con 0x7f3e540b42d0 2024-09-13T16:35:38.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e53fff700 1 --2- 172.21.15.119:0/2621285322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540b42d0 0x7f3e54071620 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f3e54070840 tx=0x7f3e48009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc41bb15adbf76b8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e537fe700 1 -- 172.21.15.119:0/2621285322 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011830 con 0x7f3e540b42d0 2024-09-13T16:35:38.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e537fe700 1 -- 172.21.15.119:0/2621285322 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e48011990 con 0x7f3e540b42d0 2024-09-13T16:35:38.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e537fe700 1 -- 172.21.15.119:0/2621285322 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e48011d20 con 0x7f3e540b42d0 2024-09-13T16:35:38.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 -- 172.21.15.119:0/2621285322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540b42d0 msgr2=0x7f3e54071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/2621285322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540b42d0 0x7f3e54071620 secure :-1 s=READY pgs=1717 cs=0 l=1 rev1=1 crypto rx=0x7f3e54070840 tx=0x7f3e48009bf0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 -- 172.21.15.119:0/2621285322 shutdown_connections 2024-09-13T16:35:38.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/2621285322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540b42d0 0x7f3e54071620 unknown :-1 s=CLOSED pgs=1717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/2621285322 >> v2:172.21.15.145:3300/0 conn(0x7f3e54156ed0 0x7f3e5406e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 -- 172.21.15.119:0/2621285322 >> 172.21.15.119:0/2621285322 conn(0x7f3e5406b460 msgr2=0x7f3e5406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:38.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 -- 172.21.15.119:0/2621285322 shutdown_connections 2024-09-13T16:35:38.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.795+0000 7f3e59e8c700 1 -- 172.21.15.119:0/2621285322 wait complete. 2024-09-13T16:35:38.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 Processor -- start 2024-09-13T16:35:38.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 -- start start 2024-09-13T16:35:38.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e54156ed0 0x7f3e54144160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e541572b0 0x7f3e54158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540ce980 0x7f3e54159350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e540cf190 con 0x7f3e540ce980 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e54159890 con 0x7f3e541572b0 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e541446a0 con 0x7f3e54156ed0 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540ce980 0x7f3e54159350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540ce980 0x7f3e54159350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56814/0 (socket says 172.21.15.119:56814) 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 -- 172.21.15.119:0/1029718163 learned_addr learned my addr 172.21.15.119:0/1029718163 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e53fff700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e541572b0 0x7f3e54158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e58e8a700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e54156ed0 0x7f3e54144160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 -- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e54156ed0 msgr2=0x7f3e54144160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e54156ed0 0x7f3e54144160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 -- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e541572b0 msgr2=0x7f3e54158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e541572b0 0x7f3e54158e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 -- 172.21.15.119:0/1029718163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e48009570 con 0x7f3e540ce980 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e58e8a700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e54156ed0 0x7f3e54144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e5968b700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540ce980 0x7f3e54159350 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00cba0 tx=0x7f3e4c00ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e517fa700 1 -- 172.21.15.119:0/1029718163 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c003e50 con 0x7f3e540ce980 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e540ceeb0 con 0x7f3e540ce980 2024-09-13T16:35:38.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e517fa700 1 -- 172.21.15.119:0/1029718163 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e4c00a6c0 con 0x7f3e540ce980 2024-09-13T16:35:38.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e517fa700 1 -- 172.21.15.119:0/1029718163 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e4c00aa30 con 0x7f3e540ce980 2024-09-13T16:35:38.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.799+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e540b0870 con 0x7f3e540ce980 2024-09-13T16:35:38.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.803+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e541569f0 con 0x7f3e540ce980 2024-09-13T16:35:38.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.803+0000 7f3e517fa700 1 -- 172.21.15.119:0/1029718163 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3e4c00ab90 con 0x7f3e540ce980 2024-09-13T16:35:38.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.803+0000 7f3e517fa700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e40066730 0x7f3e40068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:38.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.803+0000 7f3e517fa700 1 -- 172.21.15.119:0/1029718163 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6144+0+0 (secure 0 0 0) 0x7f3e4c083d10 con 0x7f3e540ce980 2024-09-13T16:35:38.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.803+0000 7f3e58e8a700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e40066730 0x7f3e40068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:38.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.803+0000 7f3e58e8a700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e40066730 0x7f3e40068bf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009c40 tx=0x7f3e44008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:38.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:38.811+0000 7f3e517fa700 1 -- 172.21.15.119:0/1029718163 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3e4c050e50 con 0x7f3e540ce980 2024-09-13T16:35:39.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.011+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3e5406e230 con 0x7f3e540ce980 2024-09-13T16:35:39.019 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:35:39.015+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:35:39.035 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.027+0000 7f3e517fa700 1 -- 172.21.15.119:0/1029718163 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v83) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3e4c054470 con 0x7f3e540ce980 2024-09-13T16:35:39.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e40066730 msgr2=0x7f3e40068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e40066730 0x7f3e40068bf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f3e44009c40 tx=0x7f3e44008040 comp rx=0 tx=0).stop 2024-09-13T16:35:39.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540ce980 msgr2=0x7f3e54159350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540ce980 0x7f3e54159350 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f3e4c00cba0 tx=0x7f3e4c00ceb0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.038 INFO:tasks.ceph.mds.c.smithi119.stderr: -1010> 2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000001 2024-09-13T16:35:39.038 INFO:tasks.ceph.mds.c.smithi119.stderr: -1007> 2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000001 2024-09-13T16:35:39.038 INFO:tasks.ceph.mds.c.smithi119.stderr: -990> 2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : scrub: inode wrongly marked free: 0x10000000002 2024-09-13T16:35:39.038 INFO:tasks.ceph.mds.c.smithi119.stderr: -986> 2024-09-13T16:35:08.176+0000 7f4ec11d7700 -1 log_channel(cluster) log [ERR] : inode table repaired for inode: 0x10000000002 2024-09-13T16:35:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 shutdown_connections 2024-09-13T16:35:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3e40066730 0x7f3e40068bf0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3e540ce980 0x7f3e54159350 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3e541572b0 0x7f3e54158e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 --2- 172.21.15.119:0/1029718163 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3e54156ed0 0x7f3e54144160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 >> 172.21.15.119:0/1029718163 conn(0x7f3e5406b460 msgr2=0x7f3e540cfe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 shutdown_connections 2024-09-13T16:35:39.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.031+0000 7f3e59e8c700 1 -- 172.21.15.119:0/1029718163 wait complete. 2024-09-13T16:35:39.040 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:35:39.052 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-table-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 0 show inode 2024-09-13T16:35:39.066 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:35:39.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.083+0000 7fbe9e8a9500 1 -- start start 2024-09-13T16:35:39.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.083+0000 7fbe9e8a9500 1 Processor -- start 2024-09-13T16:35:39.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830000 0x55f416af7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51c00 0x55f416af6b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f417830800 0x55f416af7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f416a87e00 con 0x55f416a51c00 2024-09-13T16:35:39.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad2a00 con 0x55f417830000 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad2280 con 0x55f417830800 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51c00 0x55f416af6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51c00 0x55f416af6b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56850/0 (socket says 172.21.15.119:56850) 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 -- 172.21.15.119:0/4028774866 learned_addr learned my addr 172.21.15.119:0/4028774866 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9d864700 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f417830800 0x55f416af7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9d063700 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830000 0x55f416af7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 -- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f417830800 msgr2=0x55f416af7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f417830800 0x55f416af7600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 -- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830000 msgr2=0x55f416af7080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830000 0x55f416af7080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 -- 172.21.15.119:0/4028774866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f416a74000 con 0x55f416a51c00 2024-09-13T16:35:39.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9c862700 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51c00 0x55f416af6b00 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x55f417822d50 tx=0x55f417818ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d716b442cad1d17c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9a85e700 1 -- 172.21.15.119:0/4028774866 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f416aa1080 con 0x55f416a51c00 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9a85e700 1 -- 172.21.15.119:0/4028774866 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f416d4a000 con 0x55f416a51c00 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9a85e700 1 -- 172.21.15.119:0/4028774866 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f416aa1600 con 0x55f416a51c00 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/4028774866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f416a74ea0 con 0x55f416a51c00 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/4028774866 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55f416d4c000 con 0x55f416a51c00 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 4 waiting for MDS map... 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9a85e700 1 -- 172.21.15.119:0/4028774866 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x55f416a8ac40 con 0x55f416a51c00 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9a85e700 1 -- 172.21.15.119:0/4028774866 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 83) v1 ==== 2573+0+0 (secure 0 0 0) 0x55f416d5e300 con 0x55f416a51c00 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 4 Got MDS map 83 2024-09-13T16:35:39.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 10 main: main 2024-09-13T16:35:39.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 4 main: connecting to RADOS... 2024-09-13T16:35:39.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.087+0000 7fbe9e8a9500 1 Processor -- start 2024-09-13T16:35:39.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- start start 2024-09-13T16:35:39.097 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51000 0x55f416af6580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51800 0x55f416af8100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830c00 0x55f416af8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad23c0 con 0x55f416a51000 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad2500 con 0x55f417830c00 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad2a00 con 0x55f416a51800 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51000 0x55f416af6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51000 0x55f416af6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56856/0 (socket says 172.21.15.119:56856) 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 -- 172.21.15.119:0/876958990 learned_addr learned my addr 172.21.15.119:0/876958990 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d063700 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51800 0x55f416af8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9c862700 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830c00 0x55f416af8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 -- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51800 msgr2=0x55f416af8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51800 0x55f416af8100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.098 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 -- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830c00 msgr2=0x55f416af8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:39.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830c00 0x55f416af8680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 -- 172.21.15.119:0/876958990 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f416d4c1a0 con 0x55f416a51000 2024-09-13T16:35:39.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d063700 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51800 0x55f416af8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:39.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9d864700 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51000 0x55f416af6580 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x55f4178202a0 tx=0x55f416d6cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae2f28ca7ba0a0e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9885a700 1 -- 172.21.15.119:0/876958990 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f416aa1080 con 0x55f416a51000 2024-09-13T16:35:39.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9885a700 1 -- 172.21.15.119:0/876958990 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f416d4a480 con 0x55f416a51000 2024-09-13T16:35:39.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9885a700 1 -- 172.21.15.119:0/876958990 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f416aa1600 con 0x55f416a51000 2024-09-13T16:35:39.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/876958990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51000 msgr2=0x55f416af6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51000 0x55f416af6580 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x55f4178202a0 tx=0x55f416d6cc00 comp rx=0 tx=0).stop 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/876958990 shutdown_connections 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830c00 0x55f416af8680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51800 0x55f416af8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/876958990 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51000 0x55f416af6580 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/876958990 >> 172.21.15.119:0/876958990 conn(0x55f416a51400 msgr2=0x55f416b0d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/876958990 shutdown_connections 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.091+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/876958990 wait complete. 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 Processor -- start 2024-09-13T16:35:39.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 -- start start 2024-09-13T16:35:39.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51800 0x55f416af9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51000 0x55f416af9180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417831c00 0x55f416af8c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad2640 con 0x55f416a51800 2024-09-13T16:35:39.104 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad2140 con 0x55f417831c00 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55f416ad2000 con 0x55f416a51000 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51800 0x55f416af9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51800 0x55f416af9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56872/0 (socket says 172.21.15.119:56872) 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 -- 172.21.15.119:0/3846317296 learned_addr learned my addr 172.21.15.119:0/3846317296 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d063700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51000 0x55f416af9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9c862700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417831c00 0x55f416af8c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 -- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51000 msgr2=0x55f416af9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f416a51000 0x55f416af9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 -- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417831c00 msgr2=0x55f416af8c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417831c00 0x55f416af8c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 -- 172.21.15.119:0/3846317296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f416d4c340 con 0x55f416a51800 2024-09-13T16:35:39.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9d864700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51800 0x55f416af9700 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x55f417821560 tx=0x55f416b17ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe97057700 1 -- 172.21.15.119:0/3846317296 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f416aa0f20 con 0x55f416a51800 2024-09-13T16:35:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe97057700 1 -- 172.21.15.119:0/3846317296 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55f416d4b980 con 0x55f416a51800 2024-09-13T16:35:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe97057700 1 -- 172.21.15.119:0/3846317296 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55f416aa1080 con 0x55f416a51800 2024-09-13T16:35:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55f416d4c4e0 con 0x55f416a51800 2024-09-13T16:35:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f416d4c1a0 con 0x55f416a51800 2024-09-13T16:35:39.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.095+0000 7fbe9e8a9500 4 main: resolving pool 15 2024-09-13T16:35:39.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe97057700 1 -- 172.21.15.119:0/3846317296 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x55f416ab0840 con 0x55f416a51800 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe97057700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55f416dd0c00 0x55f416eb4000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe97057700 1 -- 172.21.15.119:0/3846317296 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x55f416a8aa80 con 0x55f416a51800 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe9e8a9500 4 main: creating IoCtx.. 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55f417831800 0x55f416af8680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 15.13 15:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e115) v8 -- 0x55f417831400 con 0x55f417831800 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe9d063700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55f416dd0c00 0x55f416eb4000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe9c862700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55f417831800 0x55f416af8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe9c862700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55f417831800 0x55f416af8680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.099+0000 7fbe9d063700 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55f416dd0c00 0x55f416eb4000 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x55f417822360 tx=0x55f416d6dec0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9c862700 1 -- 172.21.15.119:0/3846317296 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 mds0_inotable [read 0~34 out=34b] v0'0 uv7 ondisk = 0) v8 ==== 157+0+34 (crc 0 0 0) 0x55f417840b40 con 0x55f417831800 2024-09-13T16:35:39.110 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:35:39.110 INFO:teuthology.orchestra.run.smithi119.stdout: "0": { 2024-09-13T16:35:39.110 INFO:teuthology.orchestra.run.smithi119.stdout: "data": { 2024-09-13T16:35:39.110 INFO:teuthology.orchestra.run.smithi119.stdout: "version": 5, 2024-09-13T16:35:39.110 INFO:teuthology.orchestra.run.smithi119.stdout: "inotable": { 2024-09-13T16:35:39.110 INFO:teuthology.orchestra.run.smithi119.stdout: "projected_free": [ 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: "start": 1099511627779, 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: "len": 1099511627773 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: ], 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: "free": [ 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: "start": 1099511627779, 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: "len": 1099511627773 2024-09-13T16:35:39.111 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:39.112 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:35:39.112 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:39.112 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:39.112 INFO:teuthology.orchestra.run.smithi119.stdout: "result": 0 2024-09-13T16:35:39.112 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:39.112 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:35:39.112 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55f417831800 msgr2=0x55f416af8680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55f417831800 0x55f416af8680 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55f416dd0c00 msgr2=0x55f416eb4000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55f416dd0c00 0x55f416eb4000 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x55f417822360 tx=0x55f416d6dec0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51800 msgr2=0x55f416af9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/3846317296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51800 0x55f416af9700 secure :-1 s=READY pgs=1726 cs=0 l=1 rev1=1 crypto rx=0x55f417821560 tx=0x55f416b17ce0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9d063700 1 -- 172.21.15.119:0/3846317296 reap_dead start 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 shutdown_connections 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 >> 172.21.15.119:0/3846317296 conn(0x55f416a51400 msgr2=0x55f416b0d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 shutdown_connections 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.103+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/3846317296 wait complete. 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/4028774866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51c00 msgr2=0x55f416af6b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51c00 0x55f416af6b00 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x55f417822d50 tx=0x55f417818ae0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/4028774866 shutdown_connections 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55f417830800 0x55f416af7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55f416a51c00 0x55f416af6b00 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 --2- 172.21.15.119:0/4028774866 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55f417830000 0x55f416af7080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/4028774866 >> 172.21.15.119:0/4028774866 conn(0x55f416a50c00 msgr2=0x55f416b0e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/4028774866 shutdown_connections 2024-09-13T16:35:39.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.107+0000 7fbe9e8a9500 1 -- 172.21.15.119:0/4028774866 wait complete. 2024-09-13T16:35:39.116 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-table-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '0', 'show', 'inode'] in time 0:00:00.064076, result: { "0": { "data": { "version": 5, "inotable": { "projected_free": [ { "start": 1099511627779, "len": 1099511627773 } ], "free": [ { "start": 1099511627779, "len": 1099511627773 } ] } }, "result": 0 } } 2024-09-13T16:35:39.117 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:35:39.117 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:35:39.176 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:35:39.196 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:35:39.197 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:35:39.197 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:39.205 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:35:39.205 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:35:39.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:35:39.206 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:35:39.212 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T16:35:39.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:35:39.213 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:35:39.213 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:39.220 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:35:39.220 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:35:39.220 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:35:39.221 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:35:39.268 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-13T16:35:39.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:35:39.269 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:35:39.269 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:39.277 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:35:39.277 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:35:39.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:35:39.277 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:35:39.283 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-09-13T16:35:39.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:35:39.285 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:35:39.285 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:39.292 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:35:39.292 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:35:39.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:35:39.292 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:35:39.299 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-13T16:35:39.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:35:39.300 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:39.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e87b2700 1 Processor -- start 2024-09-13T16:35:39.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e87b2700 1 -- start start 2024-09-13T16:35:39.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e87b2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f28e01440d0 0x7f28e015b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e87b2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e006bd60 con 0x7f28e00cbb10 2024-09-13T16:35:39.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e87b2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28e015bf10 con 0x7f28e01444b0 2024-09-13T16:35:39.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e87b2700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28e006d790 con 0x7f28e01440d0 2024-09-13T16:35:39.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e6d4f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f28e00cbb10 0x7f28e00b1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55596/0 (socket says 172.21.15.119:55596) 2024-09-13T16:35:39.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e6d4f700 1 -- 172.21.15.119:0/166994538 learned_addr learned my addr 172.21.15.119:0/166994538 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:39.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e654e700 1 --2- 172.21.15.119:0/166994538 >> v2:172.21.15.145:3300/0 conn(0x7f28e01440d0 0x7f28e015b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e554c700 1 -- 172.21.15.119:0/166994538 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175092500 0 0) 0x7f28e006bd60 con 0x7f28e00cbb10 2024-09-13T16:35:39.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e554c700 1 -- 172.21.15.119:0/166994538 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28cc0035d0 con 0x7f28e00cbb10 2024-09-13T16:35:39.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e654e700 1 -- 172.21.15.119:0/166994538 >> v1:172.21.15.145:6789/0 conn(0x7f28e01444b0 legacy=0x7f28e006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.450 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e654e700 1 -- 172.21.15.119:0/166994538 >> v1:172.21.15.119:6789/0 conn(0x7f28e00cbb10 legacy=0x7f28e00b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e654e700 1 -- 172.21.15.119:0/166994538 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e006bfe0 con 0x7f28e01440d0 2024-09-13T16:35:39.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e654e700 1 --2- 172.21.15.119:0/166994538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 0x7f28e015b9d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009850 tx=0x7f28d4009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8801a588a8310a02 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e554c700 1 -- 172.21.15.119:0/166994538 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4011830 con 0x7f28e01440d0 2024-09-13T16:35:39.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e554c700 1 -- 172.21.15.119:0/166994538 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28d4011990 con 0x7f28e01440d0 2024-09-13T16:35:39.451 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.443+0000 7f28e554c700 1 -- 172.21.15.119:0/166994538 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28d4011d40 con 0x7f28e01440d0 2024-09-13T16:35:39.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- 172.21.15.119:0/166994538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 msgr2=0x7f28e015b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 --2- 172.21.15.119:0/166994538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 0x7f28e015b9d0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f28d4009850 tx=0x7f28d4009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:39.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- 172.21.15.119:0/166994538 shutdown_connections 2024-09-13T16:35:39.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 --2- 172.21.15.119:0/166994538 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 0x7f28e015b9d0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- 172.21.15.119:0/166994538 >> 172.21.15.119:0/166994538 conn(0x7f28e006a350 msgr2=0x7f28e006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- 172.21.15.119:0/166994538 shutdown_connections 2024-09-13T16:35:39.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- 172.21.15.119:0/166994538 wait complete. 2024-09-13T16:35:39.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 Processor -- start 2024-09-13T16:35:39.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- start start 2024-09-13T16:35:39.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28e00cbb10 0x7f28e013df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 0x7f28e00718b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28e01444b0 0x7f28e013c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e0071df0 con 0x7f28e01444b0 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e013ce40 con 0x7f28e01440d0 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e87b2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28e0070810 con 0x7f28e00cbb10 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e6d4f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28e01444b0 0x7f28e013c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e6d4f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28e01444b0 0x7f28e013c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56882/0 (socket says 172.21.15.119:56882) 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e6d4f700 1 -- 172.21.15.119:0/1304912090 learned_addr learned my addr 172.21.15.119:0/1304912090 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:39.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e654e700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28e00cbb10 0x7f28e013df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.447+0000 7f28e5d4d700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 0x7f28e00718b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e6d4f700 1 -- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28e00cbb10 msgr2=0x7f28e013df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e6d4f700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28e00cbb10 0x7f28e013df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e6d4f700 1 -- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 msgr2=0x7f28e00718b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e6d4f700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 0x7f28e00718b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e6d4f700 1 -- 172.21.15.119:0/1304912090 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d40094b0 con 0x7f28e01444b0 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e6d4f700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28e01444b0 0x7f28e013c900 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00c4f0 tx=0x7f28dc00c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28d2ffd700 1 -- 172.21.15.119:0/1304912090 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc005d90 con 0x7f28e01444b0 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28d2ffd700 1 -- 172.21.15.119:0/1304912090 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28dc0070d0 con 0x7f28e01444b0 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e0070af0 con 0x7f28e01444b0 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28d2ffd700 1 -- 172.21.15.119:0/1304912090 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc007460 con 0x7f28e01444b0 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e00710b0 con 0x7f28e01444b0 2024-09-13T16:35:39.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.451+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28e0067760 con 0x7f28e01444b0 2024-09-13T16:35:39.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.455+0000 7f28d2ffd700 1 -- 172.21.15.119:0/1304912090 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f28dc0075c0 con 0x7f28e01444b0 2024-09-13T16:35:39.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.455+0000 7f28d2ffd700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28c4066730 0x7f28c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.455+0000 7f28d2ffd700 1 -- 172.21.15.119:0/1304912090 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f28dc083860 con 0x7f28e01444b0 2024-09-13T16:35:39.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.455+0000 7f28e654e700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28c4066730 0x7f28c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.455+0000 7f28e654e700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28c4066730 0x7f28c4068bf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f28d40090b0 tx=0x7f28d4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.463+0000 7f28d2ffd700 1 -- 172.21.15.119:0/1304912090 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f28dc050920 con 0x7f28e01444b0 2024-09-13T16:35:39.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.671+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28e00cdfe0 con 0x7f28e01444b0 2024-09-13T16:35:39.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.671+0000 7f28d2ffd700 1 -- 172.21.15.119:0/1304912090 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+3982 (secure 0 0 0) 0x7f28dc053f40 con 0x7f28e01444b0 2024-09-13T16:35:39.677 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:39.677 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":83,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72}],"filesystems":[{"mdsmap":{"epoch":83,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:35:39.019737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:35:39.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28c4066730 msgr2=0x7f28c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28c4066730 0x7f28c4068bf0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f28d40090b0 tx=0x7f28d4008030 comp rx=0 tx=0).stop 2024-09-13T16:35:39.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28e01444b0 msgr2=0x7f28e013c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28e01444b0 0x7f28e013c900 secure :-1 s=READY pgs=1728 cs=0 l=1 rev1=1 crypto rx=0x7f28dc00c4f0 tx=0x7f28dc00c800 comp rx=0 tx=0).stop 2024-09-13T16:35:39.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 shutdown_connections 2024-09-13T16:35:39.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f28c4066730 0x7f28c4068bf0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f28e01444b0 0x7f28e013c900 unknown :-1 s=CLOSED pgs=1728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f28e01440d0 0x7f28e00718b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 --2- 172.21.15.119:0/1304912090 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f28e00cbb10 0x7f28e013df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 >> 172.21.15.119:0/1304912090 conn(0x7f28e006a350 msgr2=0x7f28e015cae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 shutdown_connections 2024-09-13T16:35:39.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.675+0000 7f28e87b2700 1 -- 172.21.15.119:0/1304912090 wait complete. 2024-09-13T16:35:39.681 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 83 2024-09-13T16:35:39.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:39.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 Processor -- start 2024-09-13T16:35:39.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 -- start start 2024-09-13T16:35:39.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbd380adb30 0x7fbd380cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3806bb90 0x7fbd380cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbd380cde70 0x7fbd3815e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3806cf60 con 0x7fbd380cde70 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd3806c350 con 0x7fbd380adb30 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3e246700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd3806c630 con 0x7fbd3806bb90 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3da45700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbd380cde70 0x7fbd3815e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3da45700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbd380cde70 0x7fbd3815e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56886/0 (socket says 172.21.15.119:56886) 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.791+0000 7fbd3da45700 1 -- 172.21.15.119:0/2263351618 learned_addr learned my addr 172.21.15.119:0/2263351618 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3d244700 1 --2- 172.21.15.119:0/2263351618 >> v2:172.21.15.145:3300/0 conn(0x7fbd380adb30 0x7fbd380cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3ca43700 1 --2- 172.21.15.119:0/2263351618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3806bb90 0x7fbd380cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3da45700 1 -- 172.21.15.119:0/2263351618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3806bb90 msgr2=0x7fbd380cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:39.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3da45700 1 --2- 172.21.15.119:0/2263351618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3806bb90 0x7fbd380cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3da45700 1 -- 172.21.15.119:0/2263351618 >> v2:172.21.15.145:3300/0 conn(0x7fbd380adb30 msgr2=0x7fbd380cc060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:39.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3da45700 1 --2- 172.21.15.119:0/2263351618 >> v2:172.21.15.145:3300/0 conn(0x7fbd380adb30 0x7fbd380cc060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3da45700 1 -- 172.21.15.119:0/2263351618 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd3806c910 con 0x7fbd380cde70 2024-09-13T16:35:39.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3da45700 1 --2- 172.21.15.119:0/2263351618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380cde70 0x7fbd3815e2f0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fbd38071fa0 tx=0x7fbd3000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5cb7f934f1e74de server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd2ffff700 1 -- 172.21.15.119:0/2263351618 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000e070 con 0x7fbd380cde70 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd2ffff700 1 -- 172.21.15.119:0/2263351618 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3000b910 con 0x7fbd380cde70 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd2ffff700 1 -- 172.21.15.119:0/2263351618 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3000bcc0 con 0x7fbd380cde70 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 -- 172.21.15.119:0/2263351618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380cde70 msgr2=0x7fbd3815e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 --2- 172.21.15.119:0/2263351618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380cde70 0x7fbd3815e2f0 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7fbd38071fa0 tx=0x7fbd3000b330 comp rx=0 tx=0).stop 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 -- 172.21.15.119:0/2263351618 shutdown_connections 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 --2- 172.21.15.119:0/2263351618 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380cde70 0x7fbd3815e2f0 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 --2- 172.21.15.119:0/2263351618 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd3806bb90 0x7fbd380cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 --2- 172.21.15.119:0/2263351618 >> v2:172.21.15.145:3300/0 conn(0x7fbd380adb30 0x7fbd380cc060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 -- 172.21.15.119:0/2263351618 >> 172.21.15.119:0/2263351618 conn(0x7fbd3806a350 msgr2=0x7fbd3806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 -- 172.21.15.119:0/2263351618 shutdown_connections 2024-09-13T16:35:39.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 -- 172.21.15.119:0/2263351618 wait complete. 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 Processor -- start 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.795+0000 7fbd3e246700 1 -- start start 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd3806bb90 0x7fbd380af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380adb30 0x7fbd380c3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd380cde70 0x7fbd380acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd380af940 con 0x7fbd380adb30 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd380afc20 con 0x7fbd3806bb90 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd380aff00 con 0x7fbd380cde70 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380adb30 0x7fbd380c3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380adb30 0x7fbd380c3d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56888/0 (socket says 172.21.15.119:56888) 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 -- 172.21.15.119:0/484171535 learned_addr learned my addr 172.21.15.119:0/484171535 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3d244700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd3806bb90 0x7fbd380af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3da45700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd380cde70 0x7fbd380acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 -- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd380cde70 msgr2=0x7fbd380acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd380cde70 0x7fbd380acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 -- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd3806bb90 msgr2=0x7fbd380af360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd3806bb90 0x7fbd380af360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 -- 172.21.15.119:0/484171535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd30009b30 con 0x7fbd380adb30 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3ca43700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380adb30 0x7fbd380c3d90 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400e9d0 tx=0x7fbd3400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd2dffb700 1 -- 172.21.15.119:0/484171535 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd34009720 con 0x7fbd380adb30 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd380b01e0 con 0x7fbd380adb30 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd2dffb700 1 -- 172.21.15.119:0/484171535 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd3400ee70 con 0x7fbd380adb30 2024-09-13T16:35:39.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd2dffb700 1 -- 172.21.15.119:0/484171535 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd3400ca50 con 0x7fbd380adb30 2024-09-13T16:35:39.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd380b09e0 con 0x7fbd380adb30 2024-09-13T16:35:39.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.799+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd380ce2d0 con 0x7fbd380adb30 2024-09-13T16:35:39.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.803+0000 7fbd2dffb700 1 -- 172.21.15.119:0/484171535 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbd3400cbb0 con 0x7fbd380adb30 2024-09-13T16:35:39.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.803+0000 7fbd2dffb700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbd1c066730 0x7fbd1c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:39.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.803+0000 7fbd2dffb700 1 -- 172.21.15.119:0/484171535 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7fbd34084ae0 con 0x7fbd380adb30 2024-09-13T16:35:39.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.803+0000 7fbd3d244700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbd1c066730 0x7fbd1c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:39.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.803+0000 7fbd3d244700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbd1c066730 0x7fbd1c068bf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbd28009b70 tx=0x7fbd28009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:39.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.807+0000 7fbd2dffb700 1 -- 172.21.15.119:0/484171535 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbd34051cb0 con 0x7fbd380adb30 2024-09-13T16:35:39.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.979+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbd380adf10 con 0x7fbd380adb30 2024-09-13T16:35:39.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.983+0000 7fbd2dffb700 1 -- 172.21.15.119:0/484171535 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17087 (secure 0 0 0) 0x7fbd34017090 con 0x7fbd380adb30 2024-09-13T16:35:39.988 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:39.988 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":115,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:39.019035+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6834/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6835/3700300033":"2024-09-14T16:35:02.467890+0000","172.21.15.119:6834/3700300033":"2024-09-14T16:35:02.467890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbd1c066730 msgr2=0x7fbd1c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbd1c066730 0x7fbd1c068bf0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbd28009b70 tx=0x7fbd28009240 comp rx=0 tx=0).stop 2024-09-13T16:35:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380adb30 msgr2=0x7fbd380c3d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380adb30 0x7fbd380c3d90 secure :-1 s=READY pgs=1730 cs=0 l=1 rev1=1 crypto rx=0x7fbd3400e9d0 tx=0x7fbd3400ece0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 shutdown_connections 2024-09-13T16:35:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbd1c066730 0x7fbd1c068bf0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbd380cde70 0x7fbd380acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbd380adb30 0x7fbd380c3d90 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 --2- 172.21.15.119:0/484171535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbd3806bb90 0x7fbd380af360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:39.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 >> 172.21.15.119:0/484171535 conn(0x7fbd3806a350 msgr2=0x7fbd3815d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:39.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 shutdown_connections 2024-09-13T16:35:39.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:39.987+0000 7fbd3e246700 1 -- 172.21.15.119:0/484171535 wait complete. 2024-09-13T16:35:40.009 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:33:57.346379+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:33:58.017381+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:35:40.010 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:35:40.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.119+0000 7f1e1759e700 1 Processor -- start 2024-09-13T16:35:40.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1759e700 1 -- start start 2024-09-13T16:35:40.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 0x7f1e180d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1759e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1e180d17e0 0x7f1e180b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e180cdb00 con 0x7f1e180d0c20 2024-09-13T16:35:40.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1759e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e180cdde0 con 0x7f1e180d17e0 2024-09-13T16:35:40.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180ce0c0 con 0x7f1e1815cc00 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e15d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f1e180d0c20 0x7f1e180d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55608/0 (socket says 172.21.15.119:55608) 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e15d9b700 1 -- 172.21.15.119:0/807017144 learned_addr learned my addr 172.21.15.119:0/807017144 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 --2- 172.21.15.119:0/807017144 >> v2:172.21.15.145:3300/0 conn(0x7f1e180d17e0 0x7f1e180b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1659c700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 0x7f1e180d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258432018 0 0) 0x7f1e180cdb00 con 0x7f1e180d0c20 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e000035d0 con 0x7f1e180d0c20 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1110903965 0 0) 0x7f1e000035d0 con 0x7f1e180d0c20 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 msgr2=0x7f1e180d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 0x7f1e180d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 >> v2:172.21.15.145:3300/0 conn(0x7f1e180d17e0 msgr2=0x7f1e180b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 --2- 172.21.15.119:0/807017144 >> v2:172.21.15.145:3300/0 conn(0x7f1e180d17e0 0x7f1e180b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180ce3a0 con 0x7f1e180d0c20 2024-09-13T16:35:40.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f1e08001670 con 0x7f1e180d0c20 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 >> v1:172.21.15.119:6789/0 conn(0x7f1e180d0c20 legacy=0x7f1e180d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e00003a20 0x7f1e00003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 0x7f1e00004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e00004db0 0x7f1e00005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180cdde0 con 0x7f1e00004db0 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180ce0c0 con 0x7f1e00003a20 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0000ba30 con 0x7f1e1815cc00 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e00004db0 0x7f1e00005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1659c700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e00003a20 0x7f1e00003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e15d9b700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 0x7f1e00004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 -- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 msgr2=0x7f1e00004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e1815cc00 0x7f1e00004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 -- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e00003a20 msgr2=0x7f1e00003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e00003a20 0x7f1e00003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 -- 172.21.15.119:0/807017144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e000035d0 con 0x7f1e00004db0 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1659c700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e00003a20 0x7f1e00003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e16d9d700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e00004db0 0x7f1e00005230 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009f90 tx=0x7f1e000058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97abf384d553da91 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.123+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e1000ec30 con 0x7f1e00004db0 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1000f040 con 0x7f1e00004db0 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1559a700 1 -- 172.21.15.119:0/807017144 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e10014480 con 0x7f1e00004db0 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 -- 172.21.15.119:0/807017144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e00004db0 msgr2=0x7f1e00005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 --2- 172.21.15.119:0/807017144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e00004db0 0x7f1e00005230 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f1e10009f90 tx=0x7f1e000058a0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e16d9d700 1 -- 172.21.15.119:0/807017144 reap_dead start 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 -- 172.21.15.119:0/807017144 shutdown_connections 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 -- 172.21.15.119:0/807017144 >> 172.21.15.119:0/807017144 conn(0x7f1e1808c080 msgr2=0x7f1e180d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:40.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 -- 172.21.15.119:0/807017144 shutdown_connections 2024-09-13T16:35:40.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 -- 172.21.15.119:0/807017144 wait complete. 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 Processor -- start 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.127+0000 7f1e1759e700 1 -- start start 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e18149f90 0x7f1e1815b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180c9320 0x7f1e1815b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e181f6320 0x7f1e181f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180c85a0 con 0x7f1e180c9320 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e181f7050 con 0x7f1e181f6320 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180c8840 con 0x7f1e18149f90 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180c9320 0x7f1e1815b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180c9320 0x7f1e1815b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56912/0 (socket says 172.21.15.119:56912) 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 -- 172.21.15.119:0/1139371106 learned_addr learned my addr 172.21.15.119:0/1139371106 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e16d9d700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e181f6320 0x7f1e181f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1659c700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e18149f90 0x7f1e1815b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 -- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e18149f90 msgr2=0x7f1e1815b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e18149f90 0x7f1e1815b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 -- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e181f6320 msgr2=0x7f1e181f67d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e181f6320 0x7f1e181f67d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 -- 172.21.15.119:0/1139371106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e180c8b20 con 0x7f1e180c9320 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e15d9b700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180c9320 0x7f1e1815b660 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f1e08007310 tx=0x7f1e08007620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e06ffd700 1 -- 172.21.15.119:0/1139371106 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0800f660 con 0x7f1e180c9320 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e06ffd700 1 -- 172.21.15.119:0/1139371106 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0800f7c0 con 0x7f1e180c9320 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e06ffd700 1 -- 172.21.15.119:0/1139371106 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0800fb70 con 0x7f1e180c9320 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e180c8e00 con 0x7f1e180c9320 2024-09-13T16:35:40.139 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.131+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1814a370 con 0x7f1e180c9320 2024-09-13T16:35:40.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.135+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e180cdb00 con 0x7f1e180c9320 2024-09-13T16:35:40.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.135+0000 7f1e06ffd700 1 -- 172.21.15.119:0/1139371106 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1e0800fcd0 con 0x7f1e180c9320 2024-09-13T16:35:40.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.135+0000 7f1e06ffd700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df8066760 0x7f1df8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.135+0000 7f1e06ffd700 1 -- 172.21.15.119:0/1139371106 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f1e08084260 con 0x7f1e180c9320 2024-09-13T16:35:40.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.135+0000 7f1e1659c700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df8066760 0x7f1df8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.135+0000 7f1e1659c700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df8066760 0x7f1df8068c20 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009e10 tx=0x7f1e0c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.143+0000 7f1e06ffd700 1 -- 172.21.15.119:0/1139371106 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1e08051330 con 0x7f1e180c9320 2024-09-13T16:35:40.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e1808c560 con 0x7f1e180c9320 2024-09-13T16:35:40.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e06ffd700 1 -- 172.21.15.119:0/1139371106 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v115) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e08054950 con 0x7f1e180c9320 2024-09-13T16:35:40.320 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df8066760 msgr2=0x7f1df8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df8066760 0x7f1df8068c20 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c009e10 tx=0x7f1e0c009240 comp rx=0 tx=0).stop 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180c9320 msgr2=0x7f1e1815b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180c9320 0x7f1e1815b660 secure :-1 s=READY pgs=1733 cs=0 l=1 rev1=1 crypto rx=0x7f1e08007310 tx=0x7f1e08007620 comp rx=0 tx=0).stop 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 shutdown_connections 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df8066760 0x7f1df8068c20 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e181f6320 0x7f1e181f67d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180c9320 0x7f1e1815b660 unknown :-1 s=CLOSED pgs=1733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 --2- 172.21.15.119:0/1139371106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e18149f90 0x7f1e1815b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 >> 172.21.15.119:0/1139371106 conn(0x7f1e1808c080 msgr2=0x7f1e180661e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:40.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.315+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 shutdown_connections 2024-09-13T16:35:40.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.319+0000 7f1e1759e700 1 -- 172.21.15.119:0/1139371106 wait complete. 2024-09-13T16:35:40.335 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:35:40.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 Processor -- start 2024-09-13T16:35:40.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- start start 2024-09-13T16:35:40.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f735415ccb0 0x7f73540d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f73540b5670 0x7f735415e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73540d11e0 con 0x7f73540d0da0 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73540d16c0 con 0x7f73540b5670 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73540b5bb0 con 0x7f735415ccb0 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73537fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f73540d0da0 0x7f73540b4f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55614/0 (socket says 172.21.15.119:55614) 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7358fe6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f73540b5670 0x7f735415e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73537fe700 1 -- 172.21.15.119:0/3001648396 learned_addr learned my addr 172.21.15.119:0/3001648396 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7353fff700 1 --2- 172.21.15.119:0/3001648396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f735415ccb0 0x7f73540d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7358fe6700 1 -- 172.21.15.119:0/3001648396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f735415ccb0 msgr2=0x7f73540d0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7358fe6700 1 --2- 172.21.15.119:0/3001648396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f735415ccb0 0x7f73540d0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7358fe6700 1 -- 172.21.15.119:0/3001648396 >> v1:172.21.15.119:6789/0 conn(0x7f73540d0da0 legacy=0x7f73540b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7358fe6700 1 -- 172.21.15.119:0/3001648396 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f735415d3e0 con 0x7f73540b5670 2024-09-13T16:35:40.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7353fff700 1 --2- 172.21.15.119:0/3001648396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f735415ccb0 0x7f73540d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:40.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7358fe6700 1 --2- 172.21.15.119:0/3001648396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73540b5670 0x7f735415e1a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f734c009850 tx=0x7f734c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89403b6b8afc0b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7352ffd700 1 -- 172.21.15.119:0/3001648396 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c011830 con 0x7f73540b5670 2024-09-13T16:35:40.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7352ffd700 1 -- 172.21.15.119:0/3001648396 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c011990 con 0x7f73540b5670 2024-09-13T16:35:40.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f7352ffd700 1 -- 172.21.15.119:0/3001648396 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c011d20 con 0x7f73540b5670 2024-09-13T16:35:40.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- 172.21.15.119:0/3001648396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73540b5670 msgr2=0x7f735415e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 --2- 172.21.15.119:0/3001648396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73540b5670 0x7f735415e1a0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f734c009850 tx=0x7f734c009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- 172.21.15.119:0/3001648396 shutdown_connections 2024-09-13T16:35:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 --2- 172.21.15.119:0/3001648396 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f73540b5670 0x7f735415e1a0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 --2- 172.21.15.119:0/3001648396 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f735415ccb0 0x7f73540d0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- 172.21.15.119:0/3001648396 >> 172.21.15.119:0/3001648396 conn(0x7f7354072240 msgr2=0x7f7354099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- 172.21.15.119:0/3001648396 shutdown_connections 2024-09-13T16:35:40.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.467+0000 7f73597e7700 1 -- 172.21.15.119:0/3001648396 wait complete. 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 Processor -- start 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 -- start start 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73540b5670 0x7f735414dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73540d0da0 0x7f7354152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f735415ccb0 0x7f735414fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7354153f90 con 0x7f735415ccb0 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7354154270 con 0x7f73540b5670 2024-09-13T16:35:40.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73597e7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7354154550 con 0x7f73540d0da0 2024-09-13T16:35:40.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f735415ccb0 0x7f735414fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f735415ccb0 0x7f735414fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56942/0 (socket says 172.21.15.119:56942) 2024-09-13T16:35:40.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 -- 172.21.15.119:0/758842131 learned_addr learned my addr 172.21.15.119:0/758842131 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f73537fe700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73540d0da0 0x7f7354152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7353fff700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73540b5670 0x7f735414dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 -- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73540d0da0 msgr2=0x7f7354152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73540d0da0 0x7f7354152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 -- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73540b5670 msgr2=0x7f735414dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73540b5670 0x7f735414dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 -- 172.21.15.119:0/758842131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f734c0094b0 con 0x7f735415ccb0 2024-09-13T16:35:40.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.471+0000 7f7358fe6700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f735415ccb0 0x7f735414fcc0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f734c009fd0 tx=0x7f734c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f7350ff9700 1 -- 172.21.15.119:0/758842131 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c004480 con 0x7f735415ccb0 2024-09-13T16:35:40.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f7350ff9700 1 -- 172.21.15.119:0/758842131 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f734c01f070 con 0x7f735415ccb0 2024-09-13T16:35:40.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f7350ff9700 1 -- 172.21.15.119:0/758842131 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f734c0038e0 con 0x7f735415ccb0 2024-09-13T16:35:40.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7354154830 con 0x7f735415ccb0 2024-09-13T16:35:40.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7354154df0 con 0x7f735415ccb0 2024-09-13T16:35:40.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73540d0600 con 0x7f735415ccb0 2024-09-13T16:35:40.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f7350ff9700 1 -- 172.21.15.119:0/758842131 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f734c021030 con 0x7f735415ccb0 2024-09-13T16:35:40.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f7350ff9700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7340066730 0x7f7340068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.475+0000 7f7350ff9700 1 -- 172.21.15.119:0/758842131 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f734c0857c0 con 0x7f735415ccb0 2024-09-13T16:35:40.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.479+0000 7f7353fff700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7340066730 0x7f7340068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.479+0000 7f7353fff700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7340066730 0x7f7340068bf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f734800ab60 tx=0x7f7348008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.483+0000 7f7350ff9700 1 -- 172.21.15.119:0/758842131 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f734c052930 con 0x7f735415ccb0 2024-09-13T16:35:40.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.655+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7354154fe0 con 0x7f735415ccb0 2024-09-13T16:35:40.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.655+0000 7f7350ff9700 1 -- 172.21.15.119:0/758842131 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v115) v1 ==== 104+0+11 (secure 0 0 0) 0x7f734c055f50 con 0x7f735415ccb0 2024-09-13T16:35:40.665 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:35:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7340066730 msgr2=0x7f7340068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7340066730 0x7f7340068bf0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f734800ab60 tx=0x7f7348008040 comp rx=0 tx=0).stop 2024-09-13T16:35:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f735415ccb0 msgr2=0x7f735414fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f735415ccb0 0x7f735414fcc0 secure :-1 s=READY pgs=1736 cs=0 l=1 rev1=1 crypto rx=0x7f734c009fd0 tx=0x7f734c004770 comp rx=0 tx=0).stop 2024-09-13T16:35:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 shutdown_connections 2024-09-13T16:35:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7340066730 0x7f7340068bf0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f735415ccb0 0x7f735414fcc0 unknown :-1 s=CLOSED pgs=1736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f73540d0da0 0x7f7354152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 --2- 172.21.15.119:0/758842131 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f73540b5670 0x7f735414dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 >> 172.21.15.119:0/758842131 conn(0x7f7354072240 msgr2=0x7f735406de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:40.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 shutdown_connections 2024-09-13T16:35:40.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.659+0000 7f73597e7700 1 -- 172.21.15.119:0/758842131 wait complete. 2024-09-13T16:35:40.677 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:35:40.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.783+0000 7efe33ac5700 1 Processor -- start 2024-09-13T16:35:40.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.783+0000 7efe33ac5700 1 -- start start 2024-09-13T16:35:40.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efe2c0cb210 0x7efe2c0ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2c0b2690 con 0x7efe2c15e430 2024-09-13T16:35:40.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe2c071a10 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe2c0b2970 con 0x7efe2c0cb210 2024-09-13T16:35:40.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe31861700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7efe2c15e430 0x7efe2c15e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55626/0 (socket says 172.21.15.119:55626) 2024-09-13T16:35:40.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe31861700 1 -- 172.21.15.119:0/1059231956 learned_addr learned my addr 172.21.15.119:0/1059231956 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:40.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe31060700 1 --2- 172.21.15.119:0/1059231956 >> v2:172.21.15.119:3300/0 conn(0x7efe2c0cb210 0x7efe2c0ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe3085f700 1 -- 172.21.15.119:0/1059231956 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689668548 0 0) 0x7efe2c0b2690 con 0x7efe2c15e430 2024-09-13T16:35:40.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe3085f700 1 -- 172.21.15.119:0/1059231956 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe140035d0 con 0x7efe2c15e430 2024-09-13T16:35:40.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe31060700 1 -- 172.21.15.119:0/1059231956 >> v1:172.21.15.145:6789/0 conn(0x7efe2c0cd4c0 legacy=0x7efe2c0cd950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe31060700 1 -- 172.21.15.119:0/1059231956 >> v1:172.21.15.119:6789/0 conn(0x7efe2c15e430 legacy=0x7efe2c15e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe31060700 1 -- 172.21.15.119:0/1059231956 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe2c071cf0 con 0x7efe2c0cb210 2024-09-13T16:35:40.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe31060700 1 --2- 172.21.15.119:0/1059231956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cb210 0x7efe2c0ccf80 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7efe20009850 tx=0x7efe20009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba0cf3d468acb75 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe3085f700 1 -- 172.21.15.119:0/1059231956 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20011830 con 0x7efe2c0cb210 2024-09-13T16:35:40.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe3085f700 1 -- 172.21.15.119:0/1059231956 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe20011990 con 0x7efe2c0cb210 2024-09-13T16:35:40.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe3085f700 1 -- 172.21.15.119:0/1059231956 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20011d00 con 0x7efe2c0cb210 2024-09-13T16:35:40.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- 172.21.15.119:0/1059231956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cb210 msgr2=0x7efe2c0ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 --2- 172.21.15.119:0/1059231956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cb210 0x7efe2c0ccf80 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7efe20009850 tx=0x7efe20009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- 172.21.15.119:0/1059231956 shutdown_connections 2024-09-13T16:35:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 --2- 172.21.15.119:0/1059231956 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cb210 0x7efe2c0ccf80 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- 172.21.15.119:0/1059231956 >> 172.21.15.119:0/1059231956 conn(0x7efe2c0b0560 msgr2=0x7efe2c0b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- 172.21.15.119:0/1059231956 shutdown_connections 2024-09-13T16:35:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.787+0000 7efe33ac5700 1 -- 172.21.15.119:0/1059231956 wait complete. 2024-09-13T16:35:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 Processor -- start 2024-09-13T16:35:40.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 -- start start 2024-09-13T16:35:40.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe2c0cb210 0x7efe2c0a8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cd4c0 0x7efe2c0ca100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe2c15e430 0x7efe2c0ac040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0a9830 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0a9aa0 con 0x7efe2c15e430 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c0a9d80 con 0x7efe2c0cb210 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cd4c0 0x7efe2c0ca100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cd4c0 0x7efe2c0ca100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56954/0 (socket says 172.21.15.119:56954) 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 -- 172.21.15.119:0/222088473 learned_addr learned my addr 172.21.15.119:0/222088473 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31861700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe2c0cb210 0x7efe2c0a8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe32062700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe2c15e430 0x7efe2c0ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 -- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe2c0cb210 msgr2=0x7efe2c0a8ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe2c0cb210 0x7efe2c0a8ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 -- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe2c15e430 msgr2=0x7efe2c0ac040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe2c15e430 0x7efe2c0ac040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 -- 172.21.15.119:0/222088473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe200094b0 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe32062700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe2c15e430 0x7efe2c0ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe31060700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cd4c0 0x7efe2c0ca100 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7efe20000c00 tx=0x7efe20004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe2a7fc700 1 -- 172.21.15.119:0/222088473 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20004550 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe2a7fc700 1 -- 172.21.15.119:0/222088473 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe2001f070 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c0aa060 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe2a7fc700 1 -- 172.21.15.119:0/222088473 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe20003900 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.791+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c0a9400 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.795+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c0aa580 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.795+0000 7efe2a7fc700 1 -- 172.21.15.119:0/222088473 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efe20003a60 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.795+0000 7efe2a7fc700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe18066730 0x7efe18068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:40.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.795+0000 7efe2a7fc700 1 -- 172.21.15.119:0/222088473 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7efe20021070 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.795+0000 7efe31861700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe18066730 0x7efe18068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:40.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.799+0000 7efe31861700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe18066730 0x7efe18068bf0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efe24001560 tx=0x7efe240019f0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:40.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.803+0000 7efe2a7fc700 1 -- 172.21.15.119:0/222088473 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efe2c0aa580 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.971+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efe2c0aa580 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.971+0000 7efe2a7fc700 1 -- 172.21.15.119:0/222088473 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v115) v1 ==== 100+0+11 (secure 0 0 0) 0x7efe2c0aa580 con 0x7efe2c0cd4c0 2024-09-13T16:35:40.977 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:35:40.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe18066730 msgr2=0x7efe18068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe18066730 0x7efe18068bf0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7efe24001560 tx=0x7efe240019f0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cd4c0 msgr2=0x7efe2c0ca100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:40.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cd4c0 0x7efe2c0ca100 secure :-1 s=READY pgs=1739 cs=0 l=1 rev1=1 crypto rx=0x7efe20000c00 tx=0x7efe20004770 comp rx=0 tx=0).stop 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 shutdown_connections 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efe18066730 0x7efe18068bf0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efe2c15e430 0x7efe2c0ac040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efe2c0cd4c0 0x7efe2c0ca100 unknown :-1 s=CLOSED pgs=1739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 --2- 172.21.15.119:0/222088473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efe2c0cb210 0x7efe2c0a8ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 >> 172.21.15.119:0/222088473 conn(0x7efe2c0b0560 msgr2=0x7efe2c159a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 shutdown_connections 2024-09-13T16:35:40.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:40.975+0000 7efe33ac5700 1 -- 172.21.15.119:0/222088473 wait complete. 2024-09-13T16:35:40.992 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:41.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 Processor -- start 2024-09-13T16:35:41.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 -- start start 2024-09-13T16:35:41.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2a0c15cc40 0x7f2a0c0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c0cd860 0x7f2a0c08bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2a0c08c4e0 0x7f2a0c160720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c08c970 con 0x7f2a0c15cc40 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a0c0b44c0 con 0x7f2a0c08c4e0 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c15d020 con 0x7f2a0c0cd860 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0a59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2a0c15cc40 0x7f2a0c0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0a59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2a0c15cc40 0x7f2a0c0ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56958/0 (socket says 172.21.15.119:56958) 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0a59c700 1 -- 172.21.15.119:0/4126142210 learned_addr learned my addr 172.21.15.119:0/4126142210 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0ad9d700 1 --2- 172.21.15.119:0/4126142210 >> v2:172.21.15.145:3300/0 conn(0x7f2a0c08c4e0 0x7f2a0c160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a09d9b700 1 --2- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c0cd860 0x7f2a0c08bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0ad9d700 1 -- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c0cd860 msgr2=0x7f2a0c08bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0ad9d700 1 --2- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c0cd860 0x7f2a0c08bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0ad9d700 1 -- 172.21.15.119:0/4126142210 >> v2:172.21.15.119:3300/0 conn(0x7f2a0c15cc40 msgr2=0x7f2a0c0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0ad9d700 1 --2- 172.21.15.119:0/4126142210 >> v2:172.21.15.119:3300/0 conn(0x7f2a0c15cc40 0x7f2a0c0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0ad9d700 1 -- 172.21.15.119:0/4126142210 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0c0d0720 con 0x7f2a0c08c4e0 2024-09-13T16:35:41.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.103+0000 7f2a0a59c700 1 --2- 172.21.15.119:0/4126142210 >> v2:172.21.15.119:3300/0 conn(0x7f2a0c15cc40 0x7f2a0c0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:41.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0ad9d700 1 --2- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c08c4e0 0x7f2a0c160720 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c08b1f0 tx=0x7f2a0400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72bab893b66ed4b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0959a700 1 -- 172.21.15.119:0/4126142210 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a0400e070 con 0x7f2a0c08c4e0 2024-09-13T16:35:41.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0959a700 1 -- 172.21.15.119:0/4126142210 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a0400bb20 con 0x7f2a0c08c4e0 2024-09-13T16:35:41.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0959a700 1 -- 172.21.15.119:0/4126142210 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a04012430 con 0x7f2a0c08c4e0 2024-09-13T16:35:41.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 -- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c08c4e0 msgr2=0x7f2a0c160720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c08c4e0 0x7f2a0c160720 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f2a0c08b1f0 tx=0x7f2a0400b330 comp rx=0 tx=0).stop 2024-09-13T16:35:41.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 -- 172.21.15.119:0/4126142210 shutdown_connections 2024-09-13T16:35:41.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c08c4e0 0x7f2a0c160720 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/4126142210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c0cd860 0x7f2a0c08bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/4126142210 >> v2:172.21.15.119:3300/0 conn(0x7f2a0c15cc40 0x7f2a0c0ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 -- 172.21.15.119:0/4126142210 >> 172.21.15.119:0/4126142210 conn(0x7f2a0c0b4640 msgr2=0x7f2a0c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:41.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 -- 172.21.15.119:0/4126142210 shutdown_connections 2024-09-13T16:35:41.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 -- 172.21.15.119:0/4126142210 wait complete. 2024-09-13T16:35:41.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 Processor -- start 2024-09-13T16:35:41.115 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.107+0000 7f2a0b59e700 1 -- start start 2024-09-13T16:35:41.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c08c4e0 0x7f2a0c1f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c0cd860 0x7f2a0c0ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a0c15cc40 0x7f2a0c06b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c1f2260 con 0x7f2a0c15cc40 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c1f2da0 con 0x7f2a0c0cd860 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a0c1f2470 con 0x7f2a0c08c4e0 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a0c15cc40 0x7f2a0c06b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0ad9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a0c15cc40 0x7f2a0c06b060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56966/0 (socket says 172.21.15.119:56966) 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a09d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c0cd860 0x7f2a0c0ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0ad9d700 1 -- 172.21.15.119:0/3598101592 learned_addr learned my addr 172.21.15.119:0/3598101592 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0a59c700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c08c4e0 0x7f2a0c1f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a09d9b700 1 -- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c08c4e0 msgr2=0x7f2a0c1f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a09d9b700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c08c4e0 0x7f2a0c1f1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a09d9b700 1 -- 172.21.15.119:0/3598101592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a0c15cc40 msgr2=0x7f2a0c06b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a09d9b700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a0c15cc40 0x7f2a0c06b060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a09d9b700 1 -- 172.21.15.119:0/3598101592 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a00009570 con 0x7f2a0c0cd860 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a09d9b700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c0cd860 0x7f2a0c0ca600 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00e870 tx=0x7f29fc00eb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f29faffd700 1 -- 172.21.15.119:0/3598101592 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc013680 con 0x7f2a0c0cd860 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f29faffd700 1 -- 172.21.15.119:0/3598101592 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29fc0137e0 con 0x7f2a0c0cd860 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a04009a70 con 0x7f2a0c0cd860 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f29faffd700 1 -- 172.21.15.119:0/3598101592 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29fc013b90 con 0x7f2a0c0cd860 2024-09-13T16:35:41.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a0c1f2a30 con 0x7f2a0c0cd860 2024-09-13T16:35:41.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.111+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a0c1f2f50 con 0x7f2a0c0cd860 2024-09-13T16:35:41.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.115+0000 7f29faffd700 1 -- 172.21.15.119:0/3598101592 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f29fc013cf0 con 0x7f2a0c0cd860 2024-09-13T16:35:41.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.115+0000 7f29faffd700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f29ec066730 0x7f29ec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.115+0000 7f29faffd700 1 -- 172.21.15.119:0/3598101592 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f29fc0847e0 con 0x7f2a0c0cd860 2024-09-13T16:35:41.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.115+0000 7f2a0a59c700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f29ec066730 0x7f29ec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.115+0000 7f2a0a59c700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f29ec066730 0x7f29ec068bf0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f2a00006d50 tx=0x7f2a000092e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.123+0000 7f29faffd700 1 -- 172.21.15.119:0/3598101592 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f29fc051940 con 0x7f2a0c0cd860 2024-09-13T16:35:41.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.331+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a0c0d1540 con 0x7f2a0c0cd860 2024-09-13T16:35:41.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.331+0000 7f29faffd700 1 -- 172.21.15.119:0/3598101592 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f29fc054f60 con 0x7f2a0c0cd860 2024-09-13T16:35:41.337 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:41.337 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/734046953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":734046953},{"type":"v1","addr":"172.21.15.119:6835","nonce":734046953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:35:39.019737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:35:41.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f29ec066730 msgr2=0x7f29ec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f29ec066730 0x7f29ec068bf0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f2a00006d50 tx=0x7f2a000092e0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c0cd860 msgr2=0x7f2a0c0ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c0cd860 0x7f2a0c0ca600 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f29fc00e870 tx=0x7f29fc00eb80 comp rx=0 tx=0).stop 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 shutdown_connections 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f29ec066730 0x7f29ec068bf0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a0c15cc40 0x7f2a0c06b060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a0c0cd860 0x7f2a0c0ca600 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 --2- 172.21.15.119:0/3598101592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a0c08c4e0 0x7f2a0c1f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 >> 172.21.15.119:0/3598101592 conn(0x7f2a0c0b4640 msgr2=0x7f2a0c0b0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 shutdown_connections 2024-09-13T16:35:41.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.335+0000 7f2a0b59e700 1 -- 172.21.15.119:0/3598101592 wait complete. 2024-09-13T16:35:41.341 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 84 2024-09-13T16:35:41.351 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:41.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 Processor -- start 2024-09-13T16:35:41.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 -- start start 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f3af40b17f0 0x7f3af40cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3af40b2200 0x7f3af40b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40705d0 con 0x7f3af40cd8d0 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af40b2bb0 con 0x7f3af40b2200 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af406fca0 con 0x7f3af40b17f0 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3af40b2200 0x7f3af40b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3af40b2200 0x7f3af40b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56982/0 (socket says 172.21.15.119:56982) 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 -- 172.21.15.119:0/4133581226 learned_addr learned my addr 172.21.15.119:0/4133581226 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af3fff700 1 --1- 172.21.15.119:0/4133581226 >> v1:172.21.15.119:6789/0 conn(0x7f3af40cd8d0 0x7f3af406cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55642/0 (socket says 172.21.15.119:55642) 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af37fe700 1 --2- 172.21.15.119:0/4133581226 >> v2:172.21.15.145:3300/0 conn(0x7f3af40b17f0 0x7f3af40cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 -- 172.21.15.119:0/4133581226 >> v2:172.21.15.145:3300/0 conn(0x7f3af40b17f0 msgr2=0x7f3af40cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:41.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 --2- 172.21.15.119:0/4133581226 >> v2:172.21.15.145:3300/0 conn(0x7f3af40b17f0 0x7f3af40cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 -- 172.21.15.119:0/4133581226 >> v1:172.21.15.119:6789/0 conn(0x7f3af40cd8d0 legacy=0x7f3af406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 -- 172.21.15.119:0/4133581226 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af406ff80 con 0x7f3af40b2200 2024-09-13T16:35:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af37fe700 1 --2- 172.21.15.119:0/4133581226 >> v2:172.21.15.145:3300/0 conn(0x7f3af40b17f0 0x7f3af40cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af8e12700 1 --2- 172.21.15.119:0/4133581226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b2200 0x7f3af40b2670 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f3af40ae2c0 tx=0x7f3aec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81bade821fe32831 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af2ffd700 1 -- 172.21.15.119:0/4133581226 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec00e070 con 0x7f3af40b2200 2024-09-13T16:35:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af2ffd700 1 -- 172.21.15.119:0/4133581226 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aec00b9c0 con 0x7f3af40b2200 2024-09-13T16:35:41.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af2ffd700 1 -- 172.21.15.119:0/4133581226 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec00bd70 con 0x7f3af40b2200 2024-09-13T16:35:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 -- 172.21.15.119:0/4133581226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b2200 msgr2=0x7f3af40b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.459+0000 7f3af9613700 1 --2- 172.21.15.119:0/4133581226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b2200 0x7f3af40b2670 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7f3af40ae2c0 tx=0x7f3aec00b330 comp rx=0 tx=0).stop 2024-09-13T16:35:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- 172.21.15.119:0/4133581226 shutdown_connections 2024-09-13T16:35:41.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 --2- 172.21.15.119:0/4133581226 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b2200 0x7f3af40b2670 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 --2- 172.21.15.119:0/4133581226 >> v2:172.21.15.145:3300/0 conn(0x7f3af40b17f0 0x7f3af40cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- 172.21.15.119:0/4133581226 >> 172.21.15.119:0/4133581226 conn(0x7f3af406a350 msgr2=0x7f3af406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- 172.21.15.119:0/4133581226 shutdown_connections 2024-09-13T16:35:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- 172.21.15.119:0/4133581226 wait complete. 2024-09-13T16:35:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 Processor -- start 2024-09-13T16:35:41.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- start start 2024-09-13T16:35:41.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b17f0 0x7f3af4160460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3af40b2200 0x7f3af40acda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3af40cd8d0 0x7f3af40cc040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40c3760 con 0x7f3af40b17f0 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af41609a0 con 0x7f3af40cd8d0 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af9613700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af40ad2e0 con 0x7f3af40b2200 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b17f0 0x7f3af4160460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b17f0 0x7f3af4160460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56988/0 (socket says 172.21.15.119:56988) 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 -- 172.21.15.119:0/3013604515 learned_addr learned my addr 172.21.15.119:0/3013604515 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af8e12700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3af40cd8d0 0x7f3af40cc040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af37fe700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3af40b2200 0x7f3af40acda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 -- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3af40b2200 msgr2=0x7f3af40acda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3af40b2200 0x7f3af40acda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 -- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3af40cd8d0 msgr2=0x7f3af40cc040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3af40cd8d0 0x7f3af40cc040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af3fff700 1 -- 172.21.15.119:0/3013604515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aec009b30 con 0x7f3af40b17f0 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.463+0000 7f3af8e12700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3af40cd8d0 0x7f3af40cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af3fff700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b17f0 0x7f3af4160460 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c410 tx=0x7f3ae400c7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af0ff9700 1 -- 172.21.15.119:0/3013604515 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4005cd0 con 0x7f3af40b17f0 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af0ff9700 1 -- 172.21.15.119:0/3013604515 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4005e30 con 0x7f3af40b17f0 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af40c3940 con 0x7f3af40b17f0 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af0ff9700 1 -- 172.21.15.119:0/3013604515 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae40072f0 con 0x7f3af40b17f0 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af40c3f00 con 0x7f3af40b17f0 2024-09-13T16:35:41.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af40cdcb0 con 0x7f3af40b17f0 2024-09-13T16:35:41.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af0ff9700 1 -- 172.21.15.119:0/3013604515 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3ae4007450 con 0x7f3af40b17f0 2024-09-13T16:35:41.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af0ff9700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ad8066730 0x7f3ad8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af0ff9700 1 -- 172.21.15.119:0/3013604515 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f3ae4083790 con 0x7f3af40b17f0 2024-09-13T16:35:41.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.467+0000 7f3af37fe700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ad8066730 0x7f3ad8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.471+0000 7f3af37fe700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ad8066730 0x7f3ad8068bf0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3af40ac320 tx=0x7f3adc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.475+0000 7f3af0ff9700 1 -- 172.21.15.119:0/3013604515 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3ae40508f0 con 0x7f3af40b17f0 2024-09-13T16:35:41.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.647+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3af40ccdc0 con 0x7f3af40b17f0 2024-09-13T16:35:41.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.647+0000 7f3af0ff9700 1 -- 172.21.15.119:0/3013604515 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17087 (secure 0 0 0) 0x7f3ae4053f10 con 0x7f3af40b17f0 2024-09-13T16:35:41.653 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:41.653 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":115,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:39.019035+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6834/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6835/3700300033":"2024-09-14T16:35:02.467890+0000","172.21.15.119:6834/3700300033":"2024-09-14T16:35:02.467890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:41.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ad8066730 msgr2=0x7f3ad8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ad8066730 0x7f3ad8068bf0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f3af40ac320 tx=0x7f3adc008040 comp rx=0 tx=0).stop 2024-09-13T16:35:41.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b17f0 msgr2=0x7f3af4160460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b17f0 0x7f3af4160460 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c410 tx=0x7f3ae400c7d0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 shutdown_connections 2024-09-13T16:35:41.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3ad8066730 0x7f3ad8068bf0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3af40cd8d0 0x7f3af40cc040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3af40b2200 0x7f3af40acda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 --2- 172.21.15.119:0/3013604515 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af40b17f0 0x7f3af4160460 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 >> 172.21.15.119:0/3013604515 conn(0x7f3af406a350 msgr2=0x7f3af4164600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:41.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 shutdown_connections 2024-09-13T16:35:41.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.651+0000 7f3af9613700 1 -- 172.21.15.119:0/3013604515 wait complete. 2024-09-13T16:35:41.669 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:35:41.669 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:41.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e8d9fe700 1 Processor -- start 2024-09-13T16:35:41.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e8d9fe700 1 -- start start 2024-09-13T16:35:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e8d9fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 0x7f5e880ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e8d9fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5e8806aa60 0x7f5e8806d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e8d9fe700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e880b2e40 con 0x7f5e8806cef0 2024-09-13T16:35:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e8d9fe700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e8806d9f0 con 0x7f5e8806aa60 2024-09-13T16:35:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e8d9fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880b3090 con 0x7f5e8815cbb0 2024-09-13T16:35:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e867fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5e8806cef0 0x7f5e8806a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55656/0 (socket says 172.21.15.119:55656) 2024-09-13T16:35:41.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e867fc700 1 -- 172.21.15.119:0/2218935804 learned_addr learned my addr 172.21.15.119:0/2218935804 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:41.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e877fe700 1 --2- 172.21.15.119:0/2218935804 >> v2:172.21.15.145:3300/0 conn(0x7f5e8806aa60 0x7f5e8806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 0x7f5e880ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60963911 0 0) 0x7f5e880b2e40 con 0x7f5e8806cef0 2024-09-13T16:35:41.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e740035d0 con 0x7f5e8806cef0 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3147986421 0 0) 0x7f5e740035d0 con 0x7f5e8806cef0 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 msgr2=0x7f5e880ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 0x7f5e880ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 >> v2:172.21.15.145:3300/0 conn(0x7f5e8806aa60 msgr2=0x7f5e8806d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 --2- 172.21.15.119:0/2218935804 >> v2:172.21.15.145:3300/0 conn(0x7f5e8806aa60 0x7f5e8806d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e880b3370 con 0x7f5e8806cef0 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.775+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f5e70001670 con 0x7f5e8806cef0 2024-09-13T16:35:41.785 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 >> v1:172.21.15.119:6789/0 conn(0x7f5e8806cef0 legacy=0x7f5e8806a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e74003a20 0x7f5e74003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 0x7f5e74004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e74004db0 0x7f5e74005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e8806d9f0 con 0x7f5e74003a20 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880b3090 con 0x7f5e74004db0 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7400ba30 con 0x7f5e8815cbb0 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e74003a20 0x7f5e74003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e867fc700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 0x7f5e74004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e877fe700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e74004db0 0x7f5e74005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e86ffd700 1 -- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 msgr2=0x7f5e74004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e8815cbb0 0x7f5e74004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e86ffd700 1 -- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e74004db0 msgr2=0x7f5e74005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e74004db0 0x7f5e74005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e86ffd700 1 -- 172.21.15.119:0/2218935804 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e740035d0 con 0x7f5e74003a20 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e74003a20 0x7f5e74003e40 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009b70 tx=0x7f5e740058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b863156c00a3672 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800e6c0 con 0x7f5e74003a20 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e7800e820 con 0x7f5e74003a20 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e85ffb700 1 -- 172.21.15.119:0/2218935804 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e7800eb90 con 0x7f5e74003a20 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/2218935804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e74003a20 msgr2=0x7f5e74003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 --2- 172.21.15.119:0/2218935804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e74003a20 0x7f5e74003e40 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7f5e78009b70 tx=0x7f5e740058a0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e877fe700 1 -- 172.21.15.119:0/2218935804 reap_dead start 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/2218935804 shutdown_connections 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/2218935804 >> 172.21.15.119:0/2218935804 conn(0x7f5e880b1650 msgr2=0x7f5e880b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/2218935804 shutdown_connections 2024-09-13T16:35:41.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/2218935804 wait complete. 2024-09-13T16:35:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 Processor -- start 2024-09-13T16:35:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.779+0000 7f5e8d9fe700 1 -- start start 2024-09-13T16:35:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e88154610 0x7f5e881527d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e880b0800 0x7f5e880b0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e88157480 0x7f5e88157900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e881557d0 con 0x7f5e880b0800 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e880b11c0 con 0x7f5e88157480 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e88157e40 con 0x7f5e88154610 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e880b0800 0x7f5e880b0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e880b0800 0x7f5e880b0c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57012/0 (socket says 172.21.15.119:57012) 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 -- 172.21.15.119:0/1240994662 learned_addr learned my addr 172.21.15.119:0/1240994662 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e88154610 0x7f5e881527d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e877fe700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e88157480 0x7f5e88157900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 -- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e88154610 msgr2=0x7f5e881527d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e88154610 0x7f5e881527d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 -- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e88157480 msgr2=0x7f5e88157900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e88157480 0x7f5e88157900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:41.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 -- 172.21.15.119:0/1240994662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e88158060 con 0x7f5e880b0800 2024-09-13T16:35:41.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e867fc700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e880b0800 0x7f5e880b0c80 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5e700048f0 tx=0x7f5e70007570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e7ffff700 1 -- 172.21.15.119:0/1240994662 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70011580 con 0x7f5e880b0800 2024-09-13T16:35:41.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e7ffff700 1 -- 172.21.15.119:0/1240994662 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e70011700 con 0x7f5e880b0800 2024-09-13T16:35:41.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e881f6e20 con 0x7f5e880b0800 2024-09-13T16:35:41.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e7ffff700 1 -- 172.21.15.119:0/1240994662 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e70011ab0 con 0x7f5e880b0800 2024-09-13T16:35:41.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e88155110 con 0x7f5e880b0800 2024-09-13T16:35:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e88155910 con 0x7f5e880b0800 2024-09-13T16:35:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.783+0000 7f5e7ffff700 1 -- 172.21.15.119:0/1240994662 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5e70011c10 con 0x7f5e880b0800 2024-09-13T16:35:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.787+0000 7f5e7ffff700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5e68066730 0x7f5e68068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.787+0000 7f5e7ffff700 1 -- 172.21.15.119:0/1240994662 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f5e700842b0 con 0x7f5e880b0800 2024-09-13T16:35:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.787+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5e68066730 0x7f5e68068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:41.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.787+0000 7f5e86ffd700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5e68066730 0x7f5e68068bf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5e88072b50 tx=0x7f5e7801f040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:41.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.791+0000 7f5e7ffff700 1 -- 172.21.15.119:0/1240994662 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5e88155910 con 0x7f5e880b0800 2024-09-13T16:35:42.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.995+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e88155910 con 0x7f5e880b0800 2024-09-13T16:35:42.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.995+0000 7f5e7ffff700 1 -- 172.21.15.119:0/1240994662 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f5e88155910 con 0x7f5e880b0800 2024-09-13T16:35:42.002 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:42.002 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/734046953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":734046953},{"type":"v1","addr":"172.21.15.119:6835","nonce":734046953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:35:39.019737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:35:42.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5e68066730 msgr2=0x7f5e68068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5e68066730 0x7f5e68068bf0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f5e88072b50 tx=0x7f5e7801f040 comp rx=0 tx=0).stop 2024-09-13T16:35:42.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e880b0800 msgr2=0x7f5e880b0c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e880b0800 0x7f5e880b0c80 secure :-1 s=READY pgs=1745 cs=0 l=1 rev1=1 crypto rx=0x7f5e700048f0 tx=0x7f5e70007570 comp rx=0 tx=0).stop 2024-09-13T16:35:42.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 shutdown_connections 2024-09-13T16:35:42.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5e68066730 0x7f5e68068bf0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5e88157480 0x7f5e88157900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5e880b0800 0x7f5e880b0c80 unknown :-1 s=CLOSED pgs=1745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 --2- 172.21.15.119:0/1240994662 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5e88154610 0x7f5e881527d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 >> 172.21.15.119:0/1240994662 conn(0x7f5e880b1650 msgr2=0x7f5e880ac8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:42.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 shutdown_connections 2024-09-13T16:35:42.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:41.999+0000 7f5e8d9fe700 1 -- 172.21.15.119:0/1240994662 wait complete. 2024-09-13T16:35:42.007 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 84 2024-09-13T16:35:42.018 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:42.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871d5d7700 1 Processor -- start 2024-09-13T16:35:42.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871d5d7700 1 -- start start 2024-09-13T16:35:42.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871d5d7700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f871815cc40 0x7f87180ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871d5d7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f871808bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871d5d7700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f871815d020 con 0x7f871808c4e0 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871d5d7700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f871808c970 con 0x7f871815cc40 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871d5d7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b44c0 con 0x7f87180cd860 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f871808bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f8717fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f871815cc40 0x7f87180ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f871808bfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:48356/0 (socket says 172.21.15.119:48356) 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f871cdd6700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f871808c4e0 0x7f8718160720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:43760/0 (socket says 172.21.15.119:43760) 2024-09-13T16:35:42.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 -- 172.21.15.119:0/1262198407 learned_addr learned my addr 172.21.15.119:0/1262198407 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 -- 172.21.15.119:0/1262198407 >> v2:172.21.15.145:3300/0 conn(0x7f871815cc40 msgr2=0x7f87180ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 --2- 172.21.15.119:0/1262198407 >> v2:172.21.15.145:3300/0 conn(0x7f871815cc40 0x7f87180ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 -- 172.21.15.119:0/1262198407 >> v1:172.21.15.145:6789/0 conn(0x7f871808c4e0 legacy=0x7f8718160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 -- 172.21.15.119:0/1262198407 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87180d0720 con 0x7f87180cd860 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f87177fe700 1 --2- 172.21.15.119:0/1262198407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f871808bfa0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8708009910 tx=0x7f8708009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b21b089351727a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f8716ffd700 1 -- 172.21.15.119:0/1262198407 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708011830 con 0x7f87180cd860 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f8716ffd700 1 -- 172.21.15.119:0/1262198407 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8708011990 con 0x7f87180cd860 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.127+0000 7f8716ffd700 1 -- 172.21.15.119:0/1262198407 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8708011d40 con 0x7f87180cd860 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- 172.21.15.119:0/1262198407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 msgr2=0x7f871808bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 --2- 172.21.15.119:0/1262198407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f871808bfa0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8708009910 tx=0x7f8708009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- 172.21.15.119:0/1262198407 shutdown_connections 2024-09-13T16:35:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 --2- 172.21.15.119:0/1262198407 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f871808bfa0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 --2- 172.21.15.119:0/1262198407 >> v2:172.21.15.145:3300/0 conn(0x7f871815cc40 0x7f87180ce710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- 172.21.15.119:0/1262198407 >> 172.21.15.119:0/1262198407 conn(0x7f87180b4640 msgr2=0x7f87180b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- 172.21.15.119:0/1262198407 shutdown_connections 2024-09-13T16:35:42.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- 172.21.15.119:0/1262198407 wait complete. 2024-09-13T16:35:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 Processor -- start 2024-09-13T16:35:42.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- start start 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f871808c4e0 0x7f8718154220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f87180c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f871815cc40 0x7f87180c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180c8de0 con 0x7f871815cc40 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b5980 con 0x7f871808c4e0 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871d5d7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87180b5c20 con 0x7f87180cd860 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871cdd6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f871815cc40 0x7f87180c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871cdd6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f871815cc40 0x7f87180c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57020/0 (socket says 172.21.15.119:57020) 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f871cdd6700 1 -- 172.21.15.119:0/514724689 learned_addr learned my addr 172.21.15.119:0/514724689 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f87177fe700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f87180c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.131+0000 7f8717fff700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f871808c4e0 0x7f8718154220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f87177fe700 1 -- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f871808c4e0 msgr2=0x7f8718154220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f87177fe700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f871808c4e0 0x7f8718154220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f87177fe700 1 -- 172.21.15.119:0/514724689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f871815cc40 msgr2=0x7f87180c6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f87177fe700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f871815cc40 0x7f87180c6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f87177fe700 1 -- 172.21.15.119:0/514724689 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8708009570 con 0x7f87180cd860 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f87177fe700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f87180c6660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8708016000 tx=0x7f8708004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f8714ff9700 1 -- 172.21.15.119:0/514724689 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870801a8f0 con 0x7f87180cd860 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f8714ff9700 1 -- 172.21.15.119:0/514724689 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f870801e070 con 0x7f87180cd860 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87180c8540 con 0x7f87180cd860 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f8714ff9700 1 -- 172.21.15.119:0/514724689 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f870801acd0 con 0x7f87180cd860 2024-09-13T16:35:42.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87180c8b00 con 0x7f87180cd860 2024-09-13T16:35:42.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87180d1540 con 0x7f87180cd860 2024-09-13T16:35:42.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f8714ff9700 1 -- 172.21.15.119:0/514724689 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f870802d450 con 0x7f87180cd860 2024-09-13T16:35:42.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f8714ff9700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8704066730 0x7f8704068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.142 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.135+0000 7f8714ff9700 1 -- 172.21.15.119:0/514724689 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f8708085580 con 0x7f87180cd860 2024-09-13T16:35:42.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.139+0000 7f8717fff700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8704066730 0x7f8704068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.143 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.139+0000 7f8717fff700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8704066730 0x7f8704068bf0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f87180b85c0 tx=0x7f870c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.150 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.143+0000 7f8714ff9700 1 -- 172.21.15.119:0/514724689 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8708052850 con 0x7f87180cd860 2024-09-13T16:35:42.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.347+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87180d38e0 con 0x7f87180cd860 2024-09-13T16:35:42.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f8714ff9700 1 -- 172.21.15.119:0/514724689 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f8708020090 con 0x7f87180cd860 2024-09-13T16:35:42.355 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:42.355 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":84,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/734046953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":734046953},{"type":"v1","addr":"172.21.15.119:6835","nonce":734046953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[{"mdsmap":{"epoch":83,"flags":19,"flags_state":{"joinable":false,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:33:59.032499+0000","modified":"2024-09-13T16:35:39.019737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":115,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8704066730 msgr2=0x7f8704068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8704066730 0x7f8704068bf0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f87180b85c0 tx=0x7f870c009240 comp rx=0 tx=0).stop 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 msgr2=0x7f87180c6660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f87180c6660 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8708016000 tx=0x7f8708004770 comp rx=0 tx=0).stop 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 shutdown_connections 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8704066730 0x7f8704068bf0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f871815cc40 0x7f87180c6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f87180cd860 0x7f87180c6660 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 --2- 172.21.15.119:0/514724689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f871808c4e0 0x7f8718154220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 >> 172.21.15.119:0/514724689 conn(0x7f87180b4640 msgr2=0x7f87180b4a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 shutdown_connections 2024-09-13T16:35:42.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.351+0000 7f871d5d7700 1 -- 172.21.15.119:0/514724689 wait complete. 2024-09-13T16:35:42.359 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 84 2024-09-13T16:35:42.371 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:42.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.471+0000 7ff40f59e700 1 Processor -- start 2024-09-13T16:35:42.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.471+0000 7ff40f59e700 1 -- start start 2024-09-13T16:35:42.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff410158f10 0x7ff41006c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff410070980 0x7ff41006cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4100b2940 con 0x7ff4100b2c10 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4100b2380 con 0x7ff410070980 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100b2660 con 0x7ff410158f10 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40ed9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff410070980 0x7ff41006cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40e59c700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7ff4100b2c10 0x7ff4100b2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:33672/0 (socket says 172.21.15.119:33672) 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40dd9b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff410158f10 0x7ff41006c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40e59c700 1 -- 172.21.15.119:0/364827260 learned_addr learned my addr 172.21.15.119:0/364827260 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:42.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40ed9d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff410070980 0x7ff41006cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:43682/0 (socket says 172.21.15.119:43682) 2024-09-13T16:35:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40ed9d700 1 -- 172.21.15.119:0/364827260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff410158f10 msgr2=0x7ff41006c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40ed9d700 1 --2- 172.21.15.119:0/364827260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff410158f10 0x7ff41006c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40ed9d700 1 -- 172.21.15.119:0/364827260 >> v1:172.21.15.145:6789/0 conn(0x7ff4100b2c10 legacy=0x7ff4100b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40ed9d700 1 -- 172.21.15.119:0/364827260 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41006c4c0 con 0x7ff410070980 2024-09-13T16:35:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40ed9d700 1 --2- 172.21.15.119:0/364827260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410070980 0x7ff41006cf20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff40800aea0 tx=0x7ff40800c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de5e92eeac8a3aac server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40d59a700 1 -- 172.21.15.119:0/364827260 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800f070 con 0x7ff410070980 2024-09-13T16:35:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40d59a700 1 -- 172.21.15.119:0/364827260 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40800a2e0 con 0x7ff410070980 2024-09-13T16:35:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40d59a700 1 -- 172.21.15.119:0/364827260 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40800cb50 con 0x7ff410070980 2024-09-13T16:35:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- 172.21.15.119:0/364827260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410070980 msgr2=0x7ff41006cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 --2- 172.21.15.119:0/364827260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410070980 0x7ff41006cf20 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7ff40800aea0 tx=0x7ff40800c330 comp rx=0 tx=0).stop 2024-09-13T16:35:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- 172.21.15.119:0/364827260 shutdown_connections 2024-09-13T16:35:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 --2- 172.21.15.119:0/364827260 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410070980 0x7ff41006cf20 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 --2- 172.21.15.119:0/364827260 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff410158f10 0x7ff41006c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- 172.21.15.119:0/364827260 >> 172.21.15.119:0/364827260 conn(0x7ff41006a350 msgr2=0x7ff41006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- 172.21.15.119:0/364827260 shutdown_connections 2024-09-13T16:35:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.475+0000 7ff40f59e700 1 -- 172.21.15.119:0/364827260 wait complete. 2024-09-13T16:35:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 Processor -- start 2024-09-13T16:35:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 -- start start 2024-09-13T16:35:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff410070980 0x7ff410144210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4100b2c10 0x7ff410155e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410158f10 0x7ff41015b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100af660 con 0x7ff410070980 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff410147690 con 0x7ff410158f10 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4100af010 con 0x7ff4100b2c10 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff410070980 0x7ff410144210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff410070980 0x7ff410144210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41684/0 (socket says 172.21.15.119:41684) 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40e59c700 1 -- 172.21.15.119:0/1630705996 learned_addr learned my addr 172.21.15.119:0/1630705996 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40ed9d700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410158f10 0x7ff41015b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.479+0000 7ff40dd9b700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4100b2c10 0x7ff410155e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40e59c700 1 -- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4100b2c10 msgr2=0x7ff410155e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40e59c700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4100b2c10 0x7ff410155e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40e59c700 1 -- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410158f10 msgr2=0x7ff41015b650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40e59c700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410158f10 0x7ff41015b650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40e59c700 1 -- 172.21.15.119:0/1630705996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40800ab80 con 0x7ff410070980 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40e59c700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff410070980 0x7ff410144210 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c700 tx=0x7ff40000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff406ffd700 1 -- 172.21.15.119:0/1630705996 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400005d90 con 0x7ff410070980 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff406ffd700 1 -- 172.21.15.119:0/1630705996 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4000070d0 con 0x7ff410070980 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4100af2f0 con 0x7ff410070980 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff406ffd700 1 -- 172.21.15.119:0/1630705996 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff400007440 con 0x7ff410070980 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4101563d0 con 0x7ff410070980 2024-09-13T16:35:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4100aee70 con 0x7ff410070980 2024-09-13T16:35:42.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff406ffd700 1 -- 172.21.15.119:0/1630705996 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff4000075a0 con 0x7ff410070980 2024-09-13T16:35:42.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.483+0000 7ff406ffd700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3f0066730 0x7ff3f0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.487+0000 7ff406ffd700 1 -- 172.21.15.119:0/1630705996 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7ff400083640 con 0x7ff410070980 2024-09-13T16:35:42.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.487+0000 7ff40dd9b700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3f0066730 0x7ff3f0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.487+0000 7ff40dd9b700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3f0066730 0x7ff3f0068bf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff41014b6a0 tx=0x7ff3f8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.495+0000 7ff406ffd700 1 -- 172.21.15.119:0/1630705996 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff4100aee70 con 0x7ff410070980 2024-09-13T16:35:42.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.663+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff4100aee70 con 0x7ff410070980 2024-09-13T16:35:42.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff406ffd700 1 -- 172.21.15.119:0/1630705996 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v115) v1 ==== 74+0+17087 (secure 0 0 0) 0x7ff4100aee70 con 0x7ff410070980 2024-09-13T16:35:42.672 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:42.672 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":115,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:39.019035+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:33:57.346379+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_data","create_time":"2024-09-13T16:33:58.017381+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"16.2b","mappings":[{"from":5,"to":7}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6834/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6835/3700300033":"2024-09-14T16:35:02.467890+0000","172.21.15.119:6834/3700300033":"2024-09-14T16:35:02.467890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3f0066730 msgr2=0x7ff3f0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3f0066730 0x7ff3f0068bf0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7ff41014b6a0 tx=0x7ff3f8009240 comp rx=0 tx=0).stop 2024-09-13T16:35:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff410070980 msgr2=0x7ff410144210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff410070980 0x7ff410144210 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7ff40000c700 tx=0x7ff40000ca10 comp rx=0 tx=0).stop 2024-09-13T16:35:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 shutdown_connections 2024-09-13T16:35:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3f0066730 0x7ff3f0068bf0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff410158f10 0x7ff41015b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff4100b2c10 0x7ff410155e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 --2- 172.21.15.119:0/1630705996 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff410070980 0x7ff410144210 unknown :-1 s=CLOSED pgs=1746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 >> 172.21.15.119:0/1630705996 conn(0x7ff41006a350 msgr2=0x7ff4101573a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 shutdown_connections 2024-09-13T16:35:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.667+0000 7ff40f59e700 1 -- 172.21.15.119:0/1630705996 wait complete. 2024-09-13T16:35:42.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:35:42.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.791+0000 7f2db522b700 1 Processor -- start 2024-09-13T16:35:42.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.791+0000 7f2db522b700 1 -- start start 2024-09-13T16:35:42.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.791+0000 7f2db522b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db00b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.791+0000 7f2db522b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2db00b1e80 0x7f2db0159340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.791+0000 7f2db522b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db00b2320 con 0x7f2db00b3ad0 2024-09-13T16:35:42.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.791+0000 7f2db522b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db00b2600 con 0x7f2db00b1e80 2024-09-13T16:35:42.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.791+0000 7f2db522b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db00b28e0 con 0x7f2db00b36f0 2024-09-13T16:35:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2da659c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2db00b1e80 0x7f2db0159340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2daed9d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db00b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2daf59e700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2db00b3ad0 0x7f2db015d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:33688/0 (socket says 172.21.15.119:33688) 2024-09-13T16:35:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2da659c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2db00b1e80 0x7f2db0159340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:43702/0 (socket says 172.21.15.119:43702) 2024-09-13T16:35:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2daf59e700 1 -- 172.21.15.119:0/365250124 learned_addr learned my addr 172.21.15.119:0/365250124 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2da659c700 1 -- 172.21.15.119:0/365250124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 msgr2=0x7f2db00b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:42.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2da659c700 1 --2- 172.21.15.119:0/365250124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db00b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2da659c700 1 -- 172.21.15.119:0/365250124 >> v1:172.21.15.145:6789/0 conn(0x7f2db00b3ad0 legacy=0x7f2db015d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2da659c700 1 -- 172.21.15.119:0/365250124 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db00b2bc0 con 0x7f2db00b1e80 2024-09-13T16:35:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2da659c700 1 --2- 172.21.15.119:0/365250124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b1e80 0x7f2db0159340 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009850 tx=0x7f2d98009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d20c06f2df31b73 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2dae59c700 1 -- 172.21.15.119:0/365250124 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98011830 con 0x7f2db00b1e80 2024-09-13T16:35:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2dae59c700 1 -- 172.21.15.119:0/365250124 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d98011990 con 0x7f2db00b1e80 2024-09-13T16:35:42.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2dae59c700 1 -- 172.21.15.119:0/365250124 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98011d40 con 0x7f2db00b1e80 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 -- 172.21.15.119:0/365250124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b1e80 msgr2=0x7f2db0159340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/365250124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b1e80 0x7f2db0159340 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7f2d98009850 tx=0x7f2d98009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 -- 172.21.15.119:0/365250124 shutdown_connections 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/365250124 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b1e80 0x7f2db0159340 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/365250124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db00b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 -- 172.21.15.119:0/365250124 >> 172.21.15.119:0/365250124 conn(0x7f2db006a350 msgr2=0x7f2db006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 -- 172.21.15.119:0/365250124 shutdown_connections 2024-09-13T16:35:42.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.795+0000 7f2db522b700 1 -- 172.21.15.119:0/365250124 wait complete. 2024-09-13T16:35:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 Processor -- start 2024-09-13T16:35:42.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 -- start start 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db00b1e80 0x7f2db015b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db014e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b3ad0 0x7f2db0141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db015c0f0 con 0x7f2db00b1e80 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db0165d20 con 0x7f2db00b3ad0 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db015be10 con 0x7f2db00b36f0 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2daed9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db00b1e80 0x7f2db015b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2daf59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b3ad0 0x7f2db0141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db014e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db014e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:52546/0 (socket says 172.21.15.119:52546) 2024-09-13T16:35:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2daf59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b3ad0 0x7f2db0141cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:43708/0 (socket says 172.21.15.119:43708) 2024-09-13T16:35:42.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 -- 172.21.15.119:0/2577538456 learned_addr learned my addr 172.21.15.119:0/2577538456 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 -- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b3ad0 msgr2=0x7f2db0141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b3ad0 0x7f2db0141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 -- 172.21.15.119:0/2577538456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db00b1e80 msgr2=0x7f2db015b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db00b1e80 0x7f2db015b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 -- 172.21.15.119:0/2577538456 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980094b0 con 0x7f2db00b36f0 2024-09-13T16:35:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2daed9d700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db00b1e80 0x7f2db015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:42.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da659c700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db014e640 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2db006b380 tx=0x7f2d98004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da7fff700 1 -- 172.21.15.119:0/2577538456 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98004400 con 0x7f2db00b36f0 2024-09-13T16:35:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da7fff700 1 -- 172.21.15.119:0/2577538456 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d9801f070 con 0x7f2db00b36f0 2024-09-13T16:35:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db01656d0 con 0x7f2db00b36f0 2024-09-13T16:35:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2da7fff700 1 -- 172.21.15.119:0/2577538456 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d98003900 con 0x7f2db00b36f0 2024-09-13T16:35:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.799+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db015ba00 con 0x7f2db00b36f0 2024-09-13T16:35:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.803+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db0165530 con 0x7f2db00b36f0 2024-09-13T16:35:42.808 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.803+0000 7f2da7fff700 1 -- 172.21.15.119:0/2577538456 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2d98021030 con 0x7f2db00b36f0 2024-09-13T16:35:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.803+0000 7f2da7fff700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d90066760 0x7f2d90068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.803+0000 7f2da7fff700 1 -- 172.21.15.119:0/2577538456 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f2d98085640 con 0x7f2db00b36f0 2024-09-13T16:35:42.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.803+0000 7f2daed9d700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d90066760 0x7f2d90068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:42.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.807+0000 7f2daed9d700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d90066760 0x7f2d90068c20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2da00112c0 tx=0x7f2da0009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:42.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:42.815+0000 7f2da7fff700 1 -- 172.21.15.119:0/2577538456 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2db0165530 con 0x7f2db00b36f0 2024-09-13T16:35:43.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.023+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2db0165530 con 0x7f2db00b36f0 2024-09-13T16:35:43.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.791+0000 7f2da7fff700 1 -- 172.21.15.119:0/2577538456 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v85) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2db0165530 con 0x7f2db00b36f0 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d90066760 msgr2=0x7f2d90068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d90066760 0x7f2d90068c20 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f2da00112c0 tx=0x7f2da0009240 comp rx=0 tx=0).stop 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 msgr2=0x7f2db014e640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db014e640 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f2db006b380 tx=0x7f2d98004770 comp rx=0 tx=0).stop 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 shutdown_connections 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d90066760 0x7f2d90068c20 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db00b3ad0 0x7f2db0141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db00b36f0 0x7f2db014e640 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 --2- 172.21.15.119:0/2577538456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db00b1e80 0x7f2db015b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 >> 172.21.15.119:0/2577538456 conn(0x7f2db006a350 msgr2=0x7f2db0158040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:43.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 shutdown_connections 2024-09-13T16:35:43.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.795+0000 7f2db522b700 1 -- 172.21.15.119:0/2577538456 wait complete. 2024-09-13T16:35:43.803 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:35:43.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:35:43.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 Processor -- start 2024-09-13T16:35:43.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- start start 2024-09-13T16:35:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080eb420 0x7f1e0804ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e08007260 0x7f1e0804f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e0801cfa0 con 0x7f1e08033ac0 2024-09-13T16:35:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e0801c670 con 0x7f1e08007260 2024-09-13T16:35:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e0801c950 con 0x7f1e080eb420 2024-09-13T16:35:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e08007260 0x7f1e0804f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:43.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e08007260 0x7f1e0804f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41706/0 (socket says 172.21.15.119:41706) 2024-09-13T16:35:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 -- 172.21.15.119:0/2431633125 learned_addr learned my addr 172.21.15.119:0/2431633125 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0dd9b700 1 --2- 172.21.15.119:0/2431633125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080eb420 0x7f1e0804ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 -- 172.21.15.119:0/2431633125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080eb420 msgr2=0x7f1e0804ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 --2- 172.21.15.119:0/2431633125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080eb420 0x7f1e0804ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 -- 172.21.15.119:0/2431633125 >> v1:172.21.15.145:6789/0 conn(0x7f1e08033ac0 legacy=0x7f1e08009ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 -- 172.21.15.119:0/2431633125 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0801cc30 con 0x7f1e08007260 2024-09-13T16:35:43.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0ed9d700 1 --2- 172.21.15.119:0/2431633125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e0804f150 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009910 tx=0x7f1e00009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dea1d91face7981b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0d59a700 1 -- 172.21.15.119:0/2431633125 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0000ba60 con 0x7f1e08007260 2024-09-13T16:35:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0d59a700 1 -- 172.21.15.119:0/2431633125 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0000bbc0 con 0x7f1e08007260 2024-09-13T16:35:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0d59a700 1 -- 172.21.15.119:0/2431633125 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e00009e60 con 0x7f1e08007260 2024-09-13T16:35:43.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- 172.21.15.119:0/2431633125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 msgr2=0x7f1e0804f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/2431633125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e0804f150 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7f1e00009910 tx=0x7f1e00009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- 172.21.15.119:0/2431633125 shutdown_connections 2024-09-13T16:35:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/2431633125 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e0804f150 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/2431633125 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080eb420 0x7f1e0804ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- 172.21.15.119:0/2431633125 >> 172.21.15.119:0/2431633125 conn(0x7f1e08036140 msgr2=0x7f1e08036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.943+0000 7f1e0f59e700 1 -- 172.21.15.119:0/2431633125 shutdown_connections 2024-09-13T16:35:43.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 -- 172.21.15.119:0/2431633125 wait complete. 2024-09-13T16:35:43.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 Processor -- start 2024-09-13T16:35:43.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 -- start start 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e08049cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e08033ac0 0x7f1e080da420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080ddd10 0x7f1e080ea030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080480f0 con 0x7f1e08007260 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e080488c0 con 0x7f1e08033ac0 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e08037890 con 0x7f1e080ddd10 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e08049cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e08049cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41712/0 (socket says 172.21.15.119:41712) 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 -- 172.21.15.119:0/734672576 learned_addr learned my addr 172.21.15.119:0/734672576 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0dd9b700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e08033ac0 0x7f1e080da420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:43.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0ed9d700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080ddd10 0x7f1e080ea030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 -- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080ddd10 msgr2=0x7f1e080ea030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080ddd10 0x7f1e080ea030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 -- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e08033ac0 msgr2=0x7f1e080da420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e08033ac0 0x7f1e080da420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 -- 172.21.15.119:0/734672576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e00009570 con 0x7f1e08007260 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0dd9b700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e08033ac0 0x7f1e080da420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.947+0000 7f1e0e59c700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e08049cb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f1df800c4f0 tx=0x7f1df800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e06ffd700 1 -- 172.21.15.119:0/734672576 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df80052d0 con 0x7f1e08007260 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e06ffd700 1 -- 172.21.15.119:0/734672576 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1df8005430 con 0x7f1e08007260 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e080da9c0 con 0x7f1e08007260 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e06ffd700 1 -- 172.21.15.119:0/734672576 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1df8003930 con 0x7f1e08007260 2024-09-13T16:35:43.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e08037aa0 con 0x7f1e08007260 2024-09-13T16:35:43.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e08009ba0 con 0x7f1e08007260 2024-09-13T16:35:43.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e06ffd700 1 -- 172.21.15.119:0/734672576 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1df8003a90 con 0x7f1e08007260 2024-09-13T16:35:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e06ffd700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df0066730 0x7f1df0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e06ffd700 1 -- 172.21.15.119:0/734672576 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7f1df8083900 con 0x7f1e08007260 2024-09-13T16:35:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.951+0000 7f1e0dd9b700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df0066730 0x7f1df0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:43.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.955+0000 7f1e0dd9b700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df0066730 0x7f1df0068bf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1e1004f940 tx=0x7f1e1006f040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:43.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:43.959+0000 7f1e06ffd700 1 -- 172.21.15.119:0/734672576 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1df8050a60 con 0x7f1e08007260 2024-09-13T16:35:44.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.163+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1e08006730 con 0x7f1e08007260 2024-09-13T16:35:44.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.791+0000 7f1e06ffd700 1 -- 172.21.15.119:0/734672576 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v86) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1df8054080 con 0x7f1e08007260 2024-09-13T16:35:44.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df0066730 msgr2=0x7f1df0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:44.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df0066730 0x7f1df0068bf0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f1e1004f940 tx=0x7f1e1006f040 comp rx=0 tx=0).stop 2024-09-13T16:35:44.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 msgr2=0x7f1e08049cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:44.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e08049cb0 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7f1df800c4f0 tx=0x7f1df800c8b0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 shutdown_connections 2024-09-13T16:35:44.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1df0066730 0x7f1df0068bf0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1e080ddd10 0x7f1e080ea030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1e08033ac0 0x7f1e080da420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 --2- 172.21.15.119:0/734672576 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e08007260 0x7f1e08049cb0 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 >> 172.21.15.119:0/734672576 conn(0x7f1e08036140 msgr2=0x7f1e08035130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:44.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 shutdown_connections 2024-09-13T16:35:44.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.799+0000 7f1e0f59e700 1 -- 172.21.15.119:0/734672576 wait complete. 2024-09-13T16:35:44.818 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:35:44.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.939+0000 7fa0f0be4700 1 Processor -- start 2024-09-13T16:35:44.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.939+0000 7fa0f0be4700 1 -- start start 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec15cc40 0x7fa0ec0cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa0ec0d1840 0x7fa0ec0cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0d3260 con 0x7fa0ec0d0720 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0ec0d0130 con 0x7fa0ec0d1840 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec0d0c70 con 0x7fa0ec15cc40 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa0ec0d1840 0x7fa0ec0cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa0ec0d1840 0x7fa0ec0cdab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41722/0 (socket says 172.21.15.119:41722) 2024-09-13T16:35:44.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 -- 172.21.15.119:0/2696454133 learned_addr learned my addr 172.21.15.119:0/2696454133 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0eb7fe700 1 --2- 172.21.15.119:0/2696454133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec15cc40 0x7fa0ec0cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 -- 172.21.15.119:0/2696454133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec15cc40 msgr2=0x7fa0ec0cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 --2- 172.21.15.119:0/2696454133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec15cc40 0x7fa0ec0cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 -- 172.21.15.119:0/2696454133 >> v1:172.21.15.145:6789/0 conn(0x7fa0ec0d0720 legacy=0x7fa0ec0d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 -- 172.21.15.119:0/2696454133 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec0cd640 con 0x7fa0ec0d1840 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0eb7fe700 1 --2- 172.21.15.119:0/2696454133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec15cc40 0x7fa0ec0cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ebfff700 1 --2- 172.21.15.119:0/2696454133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec0cdab0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009e80 tx=0x7fa0e400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba55be023d07008c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ea7fc700 1 -- 172.21.15.119:0/2696454133 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e400e070 con 0x7fa0ec0d1840 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ea7fc700 1 -- 172.21.15.119:0/2696454133 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0e400b980 con 0x7fa0ec0d1840 2024-09-13T16:35:44.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0ea7fc700 1 -- 172.21.15.119:0/2696454133 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0e400bd30 con 0x7fa0ec0d1840 2024-09-13T16:35:44.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- 172.21.15.119:0/2696454133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 msgr2=0x7fa0ec0cdab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:44.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/2696454133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec0cdab0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fa0e4009e80 tx=0x7fa0e400b330 comp rx=0 tx=0).stop 2024-09-13T16:35:44.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- 172.21.15.119:0/2696454133 shutdown_connections 2024-09-13T16:35:44.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/2696454133 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec0cdab0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/2696454133 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec15cc40 0x7fa0ec0cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- 172.21.15.119:0/2696454133 >> 172.21.15.119:0/2696454133 conn(0x7fa0ec09be40 msgr2=0x7fa0ec0d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:44.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- 172.21.15.119:0/2696454133 shutdown_connections 2024-09-13T16:35:44.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.943+0000 7fa0f0be4700 1 -- 172.21.15.119:0/2696454133 wait complete. 2024-09-13T16:35:44.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 Processor -- start 2024-09-13T16:35:44.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 -- start start 2024-09-13T16:35:44.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec0d0720 0x7fa0ec0cc900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:44.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec155bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:44.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0ec15cc40 0x7fa0ec0c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:44.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec156580 con 0x7fa0ec0d1840 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec0b5530 con 0x7fa0ec15cc40 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0f0be4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0ec0cacb0 con 0x7fa0ec0d0720 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0eaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec155bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0eaffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec155bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41728/0 (socket says 172.21.15.119:41728) 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0eaffd700 1 -- 172.21.15.119:0/3613738272 learned_addr learned my addr 172.21.15.119:0/3613738272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0ebfff700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0ec15cc40 0x7fa0ec0c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.947+0000 7fa0eb7fe700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec0d0720 0x7fa0ec0cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0eaffd700 1 -- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec0d0720 msgr2=0x7fa0ec0cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0eaffd700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec0d0720 0x7fa0ec0cc900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0eaffd700 1 -- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0ec15cc40 msgr2=0x7fa0ec0c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:44.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0eaffd700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0ec15cc40 0x7fa0ec0c6f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0eaffd700 1 -- 172.21.15.119:0/3613738272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0e4009b30 con 0x7fa0ec0d1840 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0eaffd700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec155bd0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c500 tx=0x7fa0dc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0d3fff700 1 -- 172.21.15.119:0/3613738272 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc0052d0 con 0x7fa0ec0d1840 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0d3fff700 1 -- 172.21.15.119:0/3613738272 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0dc005430 con 0x7fa0ec0d1840 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec0caf50 con 0x7fa0ec0d1840 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0d3fff700 1 -- 172.21.15.119:0/3613738272 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0dc0055a0 con 0x7fa0ec0d1840 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec0cb510 con 0x7fa0ec0d1840 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0ec06ab20 con 0x7fa0ec0d1840 2024-09-13T16:35:44.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.951+0000 7fa0d3fff700 1 -- 172.21.15.119:0/3613738272 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa0dc0038c0 con 0x7fa0ec0d1840 2024-09-13T16:35:44.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.955+0000 7fa0d3fff700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0d8066760 0x7fa0d8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:44.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.955+0000 7fa0d3fff700 1 -- 172.21.15.119:0/3613738272 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6230+0+0 (secure 0 0 0) 0x7fa0dc0836b0 con 0x7fa0ec0d1840 2024-09-13T16:35:44.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.955+0000 7fa0eb7fe700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0d8066760 0x7fa0d8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:44.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.955+0000 7fa0eb7fe700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0d8066760 0x7fa0d8068c20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c960 tx=0x7fa0e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:44.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:44.959+0000 7fa0d3fff700 1 -- 172.21.15.119:0/3613738272 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa0dc050800 con 0x7fa0ec0d1840 2024-09-13T16:35:45.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.131+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa0ec0d0320 con 0x7fa0ec0d1840 2024-09-13T16:35:45.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.815+0000 7fa0d3fff700 1 -- 172.21.15.119:0/3613738272 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v116) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa0dc053e20 con 0x7fa0ec0d1840 2024-09-13T16:35:45.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0d8066760 msgr2=0x7fa0d8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0d8066760 0x7fa0d8068c20 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fa0e000c960 tx=0x7fa0e0008040 comp rx=0 tx=0).stop 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 msgr2=0x7fa0ec155bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec155bd0 secure :-1 s=READY pgs=1750 cs=0 l=1 rev1=1 crypto rx=0x7fa0dc00c500 tx=0x7fa0dc00c810 comp rx=0 tx=0).stop 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 shutdown_connections 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa0d8066760 0x7fa0d8068c20 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa0ec15cc40 0x7fa0ec0c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0ec0d1840 0x7fa0ec155bd0 unknown :-1 s=CLOSED pgs=1750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 --2- 172.21.15.119:0/3613738272 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0ec0d0720 0x7fa0ec0cc900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 >> 172.21.15.119:0/3613738272 conn(0x7fa0ec09be40 msgr2=0x7fa0ec0b5980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 shutdown_connections 2024-09-13T16:35:45.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.823+0000 7fa0f0be4700 1 -- 172.21.15.119:0/3613738272 wait complete. 2024-09-13T16:35:45.830 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:35:45.842 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:35:45.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91db57700 1 Processor -- start 2024-09-13T16:35:45.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91db57700 1 -- start start 2024-09-13T16:35:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91db57700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9180d0720 0x7fd9180d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91db57700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd9180d1670 0x7fd9180cdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91db57700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9180d3260 con 0x7fd91815cc40 2024-09-13T16:35:45.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91db57700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180d0c70 con 0x7fd9180d0720 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91db57700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9180d3490 con 0x7fd9180d1670 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9180d0720 0x7fd9180d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd9180d0720 0x7fd9180d1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41736/0 (socket says 172.21.15.119:41736) 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91cb55700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd91815cc40 0x7fd9180cfbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:60304/0 (socket says 172.21.15.119:60304) 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 -- 172.21.15.119:0/475121966 learned_addr learned my addr 172.21.15.119:0/475121966 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd91d356700 1 --2- 172.21.15.119:0/475121966 >> v2:172.21.15.145:3300/0 conn(0x7fd9180d1670 0x7fd9180cdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 -- 172.21.15.119:0/475121966 >> v2:172.21.15.145:3300/0 conn(0x7fd9180d1670 msgr2=0x7fd9180cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 --2- 172.21.15.119:0/475121966 >> v2:172.21.15.145:3300/0 conn(0x7fd9180d1670 0x7fd9180cdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 -- 172.21.15.119:0/475121966 >> v1:172.21.15.119:6789/0 conn(0x7fd91815cc40 legacy=0x7fd9180cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 -- 172.21.15.119:0/475121966 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9180cd640 con 0x7fd9180d0720 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd917fff700 1 --2- 172.21.15.119:0/475121966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9180d0720 0x7fd9180d1130 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fd91809c370 tx=0x7fd900009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206efa602a342451 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:45.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd9177fe700 1 -- 172.21.15.119:0/475121966 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000b8a0 con 0x7fd9180d0720 2024-09-13T16:35:45.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd9177fe700 1 -- 172.21.15.119:0/475121966 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd90000ba00 con 0x7fd9180d0720 2024-09-13T16:35:45.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.951+0000 7fd9177fe700 1 -- 172.21.15.119:0/475121966 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd90000bd90 con 0x7fd9180d0720 2024-09-13T16:35:45.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 -- 172.21.15.119:0/475121966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9180d0720 msgr2=0x7fd9180d1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:45.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 --2- 172.21.15.119:0/475121966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9180d0720 0x7fd9180d1130 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fd91809c370 tx=0x7fd900009bf0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 -- 172.21.15.119:0/475121966 shutdown_connections 2024-09-13T16:35:45.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 --2- 172.21.15.119:0/475121966 >> v2:172.21.15.145:3300/0 conn(0x7fd9180d1670 0x7fd9180cdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 --2- 172.21.15.119:0/475121966 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd9180d0720 0x7fd9180d1130 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 -- 172.21.15.119:0/475121966 >> 172.21.15.119:0/475121966 conn(0x7fd91809be40 msgr2=0x7fd9180d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:45.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 -- 172.21.15.119:0/475121966 shutdown_connections 2024-09-13T16:35:45.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.955+0000 7fd91db57700 1 -- 172.21.15.119:0/475121966 wait complete. 2024-09-13T16:35:45.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 Processor -- start 2024-09-13T16:35:45.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 -- start start 2024-09-13T16:35:45.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9180d0720 0x7fd9180c8e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:45.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9180d1670 0x7fd918155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd91815cc40 0x7fd918156260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180c7750 con 0x7fd91815cc40 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180cc3b0 con 0x7fd9180d0720 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9180cba70 con 0x7fd9180d1670 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd91815cc40 0x7fd918156260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd91815cc40 0x7fd918156260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41748/0 (socket says 172.21.15.119:41748) 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 -- 172.21.15.119:0/3629584479 learned_addr learned my addr 172.21.15.119:0/3629584479 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91cb55700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9180d0720 0x7fd9180c8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd917fff700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9180d1670 0x7fd918155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 -- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9180d1670 msgr2=0x7fd918155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9180d1670 0x7fd918155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 -- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9180d0720 msgr2=0x7fd9180c8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9180d0720 0x7fd9180c8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 -- 172.21.15.119:0/3629584479 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd900009570 con 0x7fd91815cc40 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91d356700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd91815cc40 0x7fd918156260 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fd910007e40 tx=0x7fd91000e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd9157fa700 1 -- 172.21.15.119:0/3629584479 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd91000ed30 con 0x7fd91815cc40 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd9157fa700 1 -- 172.21.15.119:0/3629584479 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd910014420 con 0x7fd91815cc40 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9180c7470 con 0x7fd91815cc40 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd9157fa700 1 -- 172.21.15.119:0/3629584479 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9100147d0 con 0x7fd91815cc40 2024-09-13T16:35:45.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.959+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9180cbfd0 con 0x7fd91815cc40 2024-09-13T16:35:45.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.963+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9180cc4f0 con 0x7fd91815cc40 2024-09-13T16:35:45.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.963+0000 7fd9157fa700 1 -- 172.21.15.119:0/3629584479 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd910014930 con 0x7fd91815cc40 2024-09-13T16:35:45.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.963+0000 7fd9157fa700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8f8066730 0x7fd8f8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:45.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.963+0000 7fd9157fa700 1 -- 172.21.15.119:0/3629584479 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5792+0+0 (secure 0 0 0) 0x7fd910084910 con 0x7fd91815cc40 2024-09-13T16:35:45.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.963+0000 7fd91cb55700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8f8066730 0x7fd8f8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:45.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.963+0000 7fd91cb55700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8f8066730 0x7fd8f8068bf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd908000f60 tx=0x7fd908007f30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:45.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:45.971+0000 7fd9157fa700 1 -- 172.21.15.119:0/3629584479 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd9180cc4f0 con 0x7fd91815cc40 2024-09-13T16:35:46.148 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.143+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd9180cc4f0 con 0x7fd91815cc40 2024-09-13T16:35:46.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.823+0000 7fd9157fa700 1 -- 172.21.15.119:0/3629584479 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v117) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd9180cc4f0 con 0x7fd91815cc40 2024-09-13T16:35:46.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8f8066730 msgr2=0x7fd8f8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:46.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8f8066730 0x7fd8f8068bf0 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fd908000f60 tx=0x7fd908007f30 comp rx=0 tx=0).stop 2024-09-13T16:35:46.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd91815cc40 msgr2=0x7fd918156260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:46.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd91815cc40 0x7fd918156260 secure :-1 s=READY pgs=1753 cs=0 l=1 rev1=1 crypto rx=0x7fd910007e40 tx=0x7fd91000e3c0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 shutdown_connections 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8f8066730 0x7fd8f8068bf0 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd91815cc40 0x7fd918156260 unknown :-1 s=CLOSED pgs=1753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd9180d1670 0x7fd918155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 --2- 172.21.15.119:0/3629584479 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd9180d0720 0x7fd9180c8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 >> 172.21.15.119:0/3629584479 conn(0x7fd91809be40 msgr2=0x7fd91806b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 shutdown_connections 2024-09-13T16:35:46.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.831+0000 7fd91db57700 1 -- 172.21.15.119:0/3629584479 wait complete. 2024-09-13T16:35:46.838 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:35:46.851 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T16:35:46.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.963+0000 7fc590816700 1 Processor -- start 2024-09-13T16:35:46.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.963+0000 7fc590816700 1 -- start start 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc58c0cde80 0x7fc58c06b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c0b2e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58c0b1e80 con 0x7fc58c159890 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc58c06bd30 con 0x7fc58c0cde80 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c0b2a00 con 0x7fc58c06b8a0 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58ad9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc58c159890 0x7fc58c0ce5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:60318/0 (socket says 172.21.15.119:60318) 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58ad9d700 1 -- 172.21.15.119:0/2691799091 learned_addr learned my addr 172.21.15.119:0/2691799091 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc583fff700 1 --2- 172.21.15.119:0/2691799091 >> v2:172.21.15.119:3300/0 conn(0x7fc58c0cde80 0x7fc58c06b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58b59e700 1 --2- 172.21.15.119:0/2691799091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c0b2e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:46.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58a59c700 1 -- 172.21.15.119:0/2691799091 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3753913148 0 0) 0x7fc58c0b1e80 con 0x7fc58c159890 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58a59c700 1 -- 172.21.15.119:0/2691799091 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5780035d0 con 0x7fc58c159890 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc583fff700 1 -- 172.21.15.119:0/2691799091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 msgr2=0x7fc58c0b2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc583fff700 1 --2- 172.21.15.119:0/2691799091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c0b2e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc583fff700 1 -- 172.21.15.119:0/2691799091 >> v1:172.21.15.119:6789/0 conn(0x7fc58c159890 legacy=0x7fc58c0ce5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc583fff700 1 -- 172.21.15.119:0/2691799091 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc58c0b20d0 con 0x7fc58c0cde80 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58b59e700 1 --2- 172.21.15.119:0/2691799091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc583fff700 1 --2- 172.21.15.119:0/2691799091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c06b360 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc574009850 tx=0x7fc574009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25729aa1f842428a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58a59c700 1 -- 172.21.15.119:0/2691799091 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574011830 con 0x7fc58c0cde80 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58a59c700 1 -- 172.21.15.119:0/2691799091 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc574011990 con 0x7fc58c0cde80 2024-09-13T16:35:46.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc58a59c700 1 -- 172.21.15.119:0/2691799091 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574011d00 con 0x7fc58c0cde80 2024-09-13T16:35:46.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 -- 172.21.15.119:0/2691799091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 msgr2=0x7fc58c06b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:46.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 --2- 172.21.15.119:0/2691799091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c06b360 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7fc574009850 tx=0x7fc574009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:46.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 -- 172.21.15.119:0/2691799091 shutdown_connections 2024-09-13T16:35:46.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 --2- 172.21.15.119:0/2691799091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c0b2e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 --2- 172.21.15.119:0/2691799091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c06b360 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.967+0000 7fc590816700 1 -- 172.21.15.119:0/2691799091 >> 172.21.15.119:0/2691799091 conn(0x7fc58c0cb8f0 msgr2=0x7fc58c0cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:46.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 -- 172.21.15.119:0/2691799091 shutdown_connections 2024-09-13T16:35:46.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 -- 172.21.15.119:0/2691799091 wait complete. 2024-09-13T16:35:46.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 Processor -- start 2024-09-13T16:35:46.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 -- start start 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c09b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c144040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc58c159890 0x7fc58c15c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c09abf0 con 0x7fc58c0cde80 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c09b650 con 0x7fc58c159890 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc590816700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc58c1478f0 con 0x7fc58c06b8a0 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc583fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c144040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc583fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c144040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41780/0 (socket says 172.21.15.119:41780) 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc583fff700 1 -- 172.21.15.119:0/2373555123 learned_addr learned my addr 172.21.15.119:0/2373555123 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc58ad9d700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c09b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:46.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.971+0000 7fc58b59e700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc58c159890 0x7fc58c15c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc583fff700 1 -- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 msgr2=0x7fc58c09b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc583fff700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c09b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc583fff700 1 -- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc58c159890 msgr2=0x7fc58c15c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc583fff700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc58c159890 0x7fc58c15c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc583fff700 1 -- 172.21.15.119:0/2373555123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5740094b0 con 0x7fc58c0cde80 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc58b59e700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc58c159890 0x7fc58c15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc583fff700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c144040 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc58c072c70 tx=0x7fc574004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc5837fe700 1 -- 172.21.15.119:0/2373555123 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc574004550 con 0x7fc58c0cde80 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc5837fe700 1 -- 172.21.15.119:0/2373555123 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc57401e070 con 0x7fc58c0cde80 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc58c146d70 con 0x7fc58c0cde80 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc5837fe700 1 -- 172.21.15.119:0/2373555123 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5740038e0 con 0x7fc58c0cde80 2024-09-13T16:35:46.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc58c147330 con 0x7fc58c0cde80 2024-09-13T16:35:46.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc58c073ce0 con 0x7fc58c0cde80 2024-09-13T16:35:46.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.975+0000 7fc5837fe700 1 -- 172.21.15.119:0/2373555123 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc574003a40 con 0x7fc58c0cde80 2024-09-13T16:35:46.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.979+0000 7fc5837fe700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc56c066730 0x7fc56c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:46.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.979+0000 7fc5837fe700 1 -- 172.21.15.119:0/2373555123 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5349+0+0 (secure 0 0 0) 0x7fc574084e90 con 0x7fc58c0cde80 2024-09-13T16:35:46.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.979+0000 7fc58ad9d700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc56c066730 0x7fc56c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:46.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.979+0000 7fc58ad9d700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc56c066730 0x7fc56c068bf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:46.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:46.983+0000 7fc5837fe700 1 -- 172.21.15.119:0/2373555123 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc5740523f0 con 0x7fc58c0cde80 2024-09-13T16:35:47.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.151+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fc58c0745b0 con 0x7fc58c0cde80 2024-09-13T16:35:47.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.155+0000 7fc5837fe700 1 -- 172.21.15.119:0/2373555123 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc574055a10 con 0x7fc58c0cde80 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc56c066730 msgr2=0x7fc56c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc56c066730 0x7fc56c068bf0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7fc57c007fd0 tx=0x7fc57c009040 comp rx=0 tx=0).stop 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 msgr2=0x7fc58c144040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c144040 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7fc58c072c70 tx=0x7fc574004770 comp rx=0 tx=0).stop 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 shutdown_connections 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc56c066730 0x7fc56c068bf0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc58c159890 0x7fc58c15c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc58c0cde80 0x7fc58c144040 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 --2- 172.21.15.119:0/2373555123 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc58c06b8a0 0x7fc58c09b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 >> 172.21.15.119:0/2373555123 conn(0x7fc58c0cb8f0 msgr2=0x7fc58c157d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:47.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 shutdown_connections 2024-09-13T16:35:47.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.159+0000 7fc590816700 1 -- 172.21.15.119:0/2373555123 wait complete. 2024-09-13T16:35:47.181 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync' 2024-09-13T16:35:47.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 Processor -- start 2024-09-13T16:35:47.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 -- start start 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c0d37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa54c0d1620 0x7fa54c0ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa54c1607c0 0x7fa54c160c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c1611b0 con 0x7fa54c1607c0 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa54c0d1190 con 0x7fa54c0d1620 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa5525c1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0ceaf0 con 0x7fa54c15ce70 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa550b5e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa54c1607c0 0x7fa54c160c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa54c0d1620 0x7fa54c0ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa54c0d1620 0x7fa54c0ce2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:43786/0 (socket says 172.21.15.119:43786) 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 -- 172.21.15.119:0/1159626545 learned_addr learned my addr 172.21.15.119:0/1159626545 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:47.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54bfff700 1 --2- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c0d37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 -- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 msgr2=0x7fa54c0d37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 --2- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c0d37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 -- 172.21.15.119:0/1159626545 >> v2:172.21.15.119:3300/0 conn(0x7fa54c1607c0 msgr2=0x7fa54c160c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 --2- 172.21.15.119:0/1159626545 >> v2:172.21.15.119:3300/0 conn(0x7fa54c1607c0 0x7fa54c160c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 -- 172.21.15.119:0/1159626545 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa54c0d0860 con 0x7fa54c0d1620 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa550b5e700 1 --2- 172.21.15.119:0/1159626545 >> v2:172.21.15.119:3300/0 conn(0x7fa54c1607c0 0x7fa54c160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54b7fe700 1 --2- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0ce2a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa544009e50 tx=0x7fa54400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47d9259b6304c1bf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:47.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54affd700 1 -- 172.21.15.119:0/1159626545 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54400e070 con 0x7fa54c0d1620 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54affd700 1 -- 172.21.15.119:0/1159626545 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa54400b8d0 con 0x7fa54c0d1620 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.287+0000 7fa54affd700 1 -- 172.21.15.119:0/1159626545 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa54400bc80 con 0x7fa54c0d1620 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 msgr2=0x7fa54c0ce2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0ce2a0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fa544009e50 tx=0x7fa54400b330 comp rx=0 tx=0).stop 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- 172.21.15.119:0/1159626545 shutdown_connections 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1159626545 >> v2:172.21.15.119:3300/0 conn(0x7fa54c1607c0 0x7fa54c160c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0ce2a0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1159626545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c0d37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- 172.21.15.119:0/1159626545 >> 172.21.15.119:0/1159626545 conn(0x7fa54c0d2bf0 msgr2=0x7fa54c0d3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- 172.21.15.119:0/1159626545 shutdown_connections 2024-09-13T16:35:47.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- 172.21.15.119:0/1159626545 wait complete. 2024-09-13T16:35:47.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 Processor -- start 2024-09-13T16:35:47.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- start start 2024-09-13T16:35:47.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0c8b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa54c1607c0 0x7fa54c0b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0c7ae0 con 0x7fa54c1607c0 2024-09-13T16:35:47.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0c7dc0 con 0x7fa54c0d1620 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa5525c1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa54c0c80a0 con 0x7fa54c15ce70 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa550b5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa54c1607c0 0x7fa54c0b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa550b5e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa54c1607c0 0x7fa54c0b74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41800/0 (socket says 172.21.15.119:41800) 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa550b5e700 1 -- 172.21.15.119:0/1871319279 learned_addr learned my addr 172.21.15.119:0/1871319279 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa54b7fe700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.291+0000 7fa54bfff700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0c8b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa550b5e700 1 -- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 msgr2=0x7fa54c151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa550b5e700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa550b5e700 1 -- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 msgr2=0x7fa54c0c8b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa550b5e700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0c8b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa550b5e700 1 -- 172.21.15.119:0/1871319279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53c0094b0 con 0x7fa54c1607c0 2024-09-13T16:35:47.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa54bfff700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:47.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa550b5e700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa54c1607c0 0x7fa54c0b74a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e930 tx=0x7fa53c00ecf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:47.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa548ff9700 1 -- 172.21.15.119:0/1871319279 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c009710 con 0x7fa54c1607c0 2024-09-13T16:35:47.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa548ff9700 1 -- 172.21.15.119:0/1871319279 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa53c004020 con 0x7fa54c1607c0 2024-09-13T16:35:47.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa544009b30 con 0x7fa54c1607c0 2024-09-13T16:35:47.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa548ff9700 1 -- 172.21.15.119:0/1871319279 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa53c0043b0 con 0x7fa54c1607c0 2024-09-13T16:35:47.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa54c0c8660 con 0x7fa54c1607c0 2024-09-13T16:35:47.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa54c0c8880 con 0x7fa54c1607c0 2024-09-13T16:35:47.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa548ff9700 1 -- 172.21.15.119:0/1871319279 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa53c00c850 con 0x7fa54c1607c0 2024-09-13T16:35:47.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa548ff9700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa538066730 0x7fa538068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.295+0000 7fa548ff9700 1 -- 172.21.15.119:0/1871319279 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5349+0+0 (secure 0 0 0) 0x7fa53c084100 con 0x7fa54c1607c0 2024-09-13T16:35:47.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.299+0000 7fa54bfff700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa538066730 0x7fa538068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.299+0000 7fa54bfff700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa538066730 0x7fa538068bf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa54c15b340 tx=0x7fa540008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:47.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.307+0000 7fa548ff9700 1 -- 172.21.15.119:0/1871319279 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa53c0515d0 con 0x7fa54c1607c0 2024-09-13T16:35:47.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.547+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]} v 0) v1 -- 0x7fa54c072730 con 0x7fa54c1607c0 2024-09-13T16:35:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.823+0000 7fa548ff9700 1 -- 172.21.15.119:0/1871319279 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_inotable_sync"]}]=0 v480) v1 ==== 146+0+0 (secure 0 0 0) 0x7fa53c054bf0 con 0x7fa54c1607c0 2024-09-13T16:35:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa538066730 msgr2=0x7fa538068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa538066730 0x7fa538068bf0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7fa54c15b340 tx=0x7fa540008040 comp rx=0 tx=0).stop 2024-09-13T16:35:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa54c1607c0 msgr2=0x7fa54c0b74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa54c1607c0 0x7fa54c0b74a0 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7fa53c00e930 tx=0x7fa53c00ecf0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 shutdown_connections 2024-09-13T16:35:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa538066730 0x7fa538068bf0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa54c1607c0 0x7fa54c0b74a0 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa54c15ce70 0x7fa54c151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 --2- 172.21.15.119:0/1871319279 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa54c0d1620 0x7fa54c0c8b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 >> 172.21.15.119:0/1871319279 conn(0x7fa54c0d2bf0 msgr2=0x7fa54c0b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 shutdown_connections 2024-09-13T16:35:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.827+0000 7fa5525c1700 1 -- 172.21.15.119:0/1871319279 wait complete. 2024-09-13T16:35:47.853 INFO:tasks.cephfs_test_runner:test_inotable_sync (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-09-13T16:35:47.854 INFO:tasks.cephfs_test_runner:Starting test: test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) 2024-09-13T16:35:47.854 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:47.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.967+0000 7f657ffff700 1 Processor -- start 2024-09-13T16:35:47.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.967+0000 7f657ffff700 1 -- start start 2024-09-13T16:35:47.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6580158b30 0x7f6580071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f6580071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f658006de00 con 0x7f6580158f10 2024-09-13T16:35:47.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f658006db20 con 0x7f6580158b30 2024-09-13T16:35:47.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65800b1920 con 0x7f658006d540 2024-09-13T16:35:47.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6580158b30 0x7f6580071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f6577fff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f6580158f10 0x7f658006ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:33722/0 (socket says 172.21.15.119:33722) 2024-09-13T16:35:47.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6580158b30 0x7f6580071030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41804/0 (socket says 172.21.15.119:41804) 2024-09-13T16:35:47.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f6577fff700 1 -- 172.21.15.119:0/732610814 learned_addr learned my addr 172.21.15.119:0/732610814 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:47.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657f7fe700 1 --2- 172.21.15.119:0/732610814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f6580071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657e7fc700 1 -- 172.21.15.119:0/732610814 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352999457 0 0) 0x7f658006de00 con 0x7f6580158f10 2024-09-13T16:35:47.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657e7fc700 1 -- 172.21.15.119:0/732610814 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656c0035d0 con 0x7f6580158f10 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657f7fe700 1 -- 172.21.15.119:0/732610814 >> v2:172.21.15.119:3300/0 conn(0x7f6580158b30 msgr2=0x7f6580071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657f7fe700 1 --2- 172.21.15.119:0/732610814 >> v2:172.21.15.119:3300/0 conn(0x7f6580158b30 0x7f6580071030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657f7fe700 1 -- 172.21.15.119:0/732610814 >> v1:172.21.15.145:6789/0 conn(0x7f6580158f10 legacy=0x7f658006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657f7fe700 1 -- 172.21.15.119:0/732610814 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65800b1c00 con 0x7f658006d540 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657effd700 1 --2- 172.21.15.119:0/732610814 >> v2:172.21.15.119:3300/0 conn(0x7f6580158b30 0x7f6580071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657f7fe700 1 --2- 172.21.15.119:0/732610814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f6580071570 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6578009d90 tx=0x7f657800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d79d7cdeea6fa31e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657e7fc700 1 -- 172.21.15.119:0/732610814 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f657800e070 con 0x7f658006d540 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657e7fc700 1 -- 172.21.15.119:0/732610814 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f657800bb20 con 0x7f658006d540 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657e7fc700 1 -- 172.21.15.119:0/732610814 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578012430 con 0x7f658006d540 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- 172.21.15.119:0/732610814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 msgr2=0x7f6580071570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 --2- 172.21.15.119:0/732610814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f6580071570 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f6578009d90 tx=0x7f657800b330 comp rx=0 tx=0).stop 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- 172.21.15.119:0/732610814 shutdown_connections 2024-09-13T16:35:47.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 --2- 172.21.15.119:0/732610814 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f6580071570 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 --2- 172.21.15.119:0/732610814 >> v2:172.21.15.119:3300/0 conn(0x7f6580158b30 0x7f6580071030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- 172.21.15.119:0/732610814 >> 172.21.15.119:0/732610814 conn(0x7f658006a350 msgr2=0x7f658006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:47.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- 172.21.15.119:0/732610814 shutdown_connections 2024-09-13T16:35:47.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.971+0000 7f657ffff700 1 -- 172.21.15.119:0/732610814 wait complete. 2024-09-13T16:35:47.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 Processor -- start 2024-09-13T16:35:47.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 -- start start 2024-09-13T16:35:47.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f65801535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6580158b30 0x7f65800abbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580158f10 0x7f65800ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658014d7a0 con 0x7f6580158f10 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65800aabc0 con 0x7f6580158b30 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65800ab720 con 0x7f658006d540 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580158f10 0x7f65800ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580158f10 0x7f65800ac0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41806/0 (socket says 172.21.15.119:41806) 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 -- 172.21.15.119:0/1384750963 learned_addr learned my addr 172.21.15.119:0/1384750963 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f6577fff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6580158b30 0x7f65800abbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657effd700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f65801535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 -- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 msgr2=0x7f65801535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f65801535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 -- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6580158b30 msgr2=0x7f65800abbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6580158b30 0x7f65800abbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 -- 172.21.15.119:0/1384750963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65680094b0 con 0x7f6580158f10 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f6577fff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6580158b30 0x7f65800abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657f7fe700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580158f10 0x7f65800ac0f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f65800cb620 tx=0x7f6578004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f65777fe700 1 -- 172.21.15.119:0/1384750963 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578012430 con 0x7f6580158f10 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6578009a70 con 0x7f6580158f10 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f65777fe700 1 -- 172.21.15.119:0/1384750963 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6578020070 con 0x7f6580158f10 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f65777fe700 1 -- 172.21.15.119:0/1384750963 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6578004280 con 0x7f6580158f10 2024-09-13T16:35:47.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.975+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65800ab0d0 con 0x7f6580158f10 2024-09-13T16:35:47.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.979+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65800ab8d0 con 0x7f6580158f10 2024-09-13T16:35:47.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.979+0000 7f65777fe700 1 -- 172.21.15.119:0/1384750963 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f65780043e0 con 0x7f6580158f10 2024-09-13T16:35:47.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.979+0000 7f65777fe700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6560066760 0x7f6560068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:47.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.979+0000 7f65777fe700 1 -- 172.21.15.119:0/1384750963 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f6578022070 con 0x7f6580158f10 2024-09-13T16:35:47.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.979+0000 7f657effd700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6560066760 0x7f6560068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:47.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.979+0000 7f657effd700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6560066760 0x7f6560068c20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6568009d10 tx=0x7f6568008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:47.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:47.991+0000 7f65777fe700 1 -- 172.21.15.119:0/1384750963 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f65800ab8d0 con 0x7f6580158f10 2024-09-13T16:35:48.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.159+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65800ab890 con 0x7f6580158f10 2024-09-13T16:35:48.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f65777fe700 1 -- 172.21.15.119:0/1384750963 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v117) v1 ==== 74+0+13393 (secure 0 0 0) 0x7f6578053130 con 0x7f6580158f10 2024-09-13T16:35:48.168 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:48.168 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":117,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:46.812630+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.119:6835/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6834/912842245":"2024-09-14T16:35:39.019011+0000","172.21.15.119:6835/3700300033":"2024-09-14T16:35:02.467890+0000","172.21.15.119:6834/3700300033":"2024-09-14T16:35:02.467890+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:48.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6560066760 msgr2=0x7f6560068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6560066760 0x7f6560068c20 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f6568009d10 tx=0x7f6568008040 comp rx=0 tx=0).stop 2024-09-13T16:35:48.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580158f10 msgr2=0x7f65800ac0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580158f10 0x7f65800ac0f0 secure :-1 s=READY pgs=1758 cs=0 l=1 rev1=1 crypto rx=0x7f65800cb620 tx=0x7f6578004af0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 shutdown_connections 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6560066760 0x7f6560068c20 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6580158f10 0x7f65800ac0f0 unknown :-1 s=CLOSED pgs=1758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6580158b30 0x7f65800abbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 --2- 172.21.15.119:0/1384750963 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f658006d540 0x7f65801535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 >> 172.21.15.119:0/1384750963 conn(0x7f658006a350 msgr2=0x7f6580154e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 shutdown_connections 2024-09-13T16:35:48.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.163+0000 7f657ffff700 1 -- 172.21.15.119:0/1384750963 wait complete. 2024-09-13T16:35:48.182 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:35:48.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:35:48.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.295+0000 7efec9ddd700 1 Processor -- start 2024-09-13T16:35:48.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.295+0000 7efec9ddd700 1 -- start start 2024-09-13T16:35:48.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.295+0000 7efec9ddd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec415d1c0 0x7efec415d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.295+0000 7efec9ddd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec415db20 con 0x7efec4159530 2024-09-13T16:35:48.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.295+0000 7efec9ddd700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efec40ce700 con 0x7efec4159910 2024-09-13T16:35:48.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.295+0000 7efec9ddd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec40ce9e0 con 0x7efec415d1c0 2024-09-13T16:35:48.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.295+0000 7efec8ddb700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7efec4159530 0x7efec40cdbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:60322/0 (socket says 172.21.15.119:60322) 2024-09-13T16:35:48.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec8ddb700 1 -- 172.21.15.119:0/2157762556 learned_addr learned my addr 172.21.15.119:0/2157762556 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:48.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec3fff700 1 --2- 172.21.15.119:0/2157762556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec415d1c0 0x7efec415d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec37fe700 1 -- 172.21.15.119:0/2157762556 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286349638 0 0) 0x7efec415db20 con 0x7efec4159530 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec37fe700 1 -- 172.21.15.119:0/2157762556 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efeb40035d0 con 0x7efec4159530 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec37fe700 1 -- 172.21.15.119:0/2157762556 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 32110684 0 0) 0x7efec40ce700 con 0x7efec4159910 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec37fe700 1 -- 172.21.15.119:0/2157762556 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efec415db20 con 0x7efec4159910 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec3fff700 1 -- 172.21.15.119:0/2157762556 >> v1:172.21.15.145:6789/0 conn(0x7efec4159910 legacy=0x7efec4159da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec3fff700 1 -- 172.21.15.119:0/2157762556 >> v1:172.21.15.119:6789/0 conn(0x7efec4159530 legacy=0x7efec40cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec3fff700 1 -- 172.21.15.119:0/2157762556 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efec40cecc0 con 0x7efec415d1c0 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec3fff700 1 --2- 172.21.15.119:0/2157762556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec415d1c0 0x7efec415d5e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009880 tx=0x7efeb0009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e33b3c17fee36228 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec37fe700 1 -- 172.21.15.119:0/2157762556 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0011830 con 0x7efec415d1c0 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec37fe700 1 -- 172.21.15.119:0/2157762556 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb0011990 con 0x7efec415d1c0 2024-09-13T16:35:48.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec37fe700 1 -- 172.21.15.119:0/2157762556 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb0011d00 con 0x7efec415d1c0 2024-09-13T16:35:48.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec9ddd700 1 -- 172.21.15.119:0/2157762556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec415d1c0 msgr2=0x7efec415d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec9ddd700 1 --2- 172.21.15.119:0/2157762556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec415d1c0 0x7efec415d5e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7efeb0009880 tx=0x7efeb0009b90 comp rx=0 tx=0).stop 2024-09-13T16:35:48.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec9ddd700 1 -- 172.21.15.119:0/2157762556 shutdown_connections 2024-09-13T16:35:48.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec9ddd700 1 --2- 172.21.15.119:0/2157762556 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec415d1c0 0x7efec415d5e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec9ddd700 1 -- 172.21.15.119:0/2157762556 >> 172.21.15.119:0/2157762556 conn(0x7efec406b460 msgr2=0x7efec406b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:48.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec9ddd700 1 -- 172.21.15.119:0/2157762556 shutdown_connections 2024-09-13T16:35:48.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.299+0000 7efec9ddd700 1 -- 172.21.15.119:0/2157762556 wait complete. 2024-09-13T16:35:48.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 Processor -- start 2024-09-13T16:35:48.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 -- start start 2024-09-13T16:35:48.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efec4159530 0x7efec406ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec4159910 0x7efec414cca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efec415d1c0 0x7efec4157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec415be80 con 0x7efec4159530 2024-09-13T16:35:48.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efec40cb050 con 0x7efec415d1c0 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efec40cb330 con 0x7efec4159910 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efec4159530 0x7efec406ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efec4159530 0x7efec406ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41810/0 (socket says 172.21.15.119:41810) 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 -- 172.21.15.119:0/123525781 learned_addr learned my addr 172.21.15.119:0/123525781 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec3fff700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec4159910 0x7efec414cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec95dc700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efec415d1c0 0x7efec4157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 -- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec4159910 msgr2=0x7efec414cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec4159910 0x7efec414cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 -- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efec415d1c0 msgr2=0x7efec4157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efec415d1c0 0x7efec4157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 -- 172.21.15.119:0/123525781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb00094b0 con 0x7efec4159530 2024-09-13T16:35:48.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec8ddb700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efec4159530 0x7efec406ce10 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7efeb800c580 tx=0x7efeb800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:48.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec17fa700 1 -- 172.21.15.119:0/123525781 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8005d90 con 0x7efec4159530 2024-09-13T16:35:48.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec17fa700 1 -- 172.21.15.119:0/123525781 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb80070d0 con 0x7efec4159530 2024-09-13T16:35:48.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efec40cb610 con 0x7efec4159530 2024-09-13T16:35:48.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec17fa700 1 -- 172.21.15.119:0/123525781 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeb8007460 con 0x7efec4159530 2024-09-13T16:35:48.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efec40cac90 con 0x7efec4159530 2024-09-13T16:35:48.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efec4073590 con 0x7efec4159530 2024-09-13T16:35:48.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec17fa700 1 -- 172.21.15.119:0/123525781 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efeb80075c0 con 0x7efec4159530 2024-09-13T16:35:48.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec17fa700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efea8066730 0x7efea8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec17fa700 1 -- 172.21.15.119:0/123525781 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5349+0+0 (secure 0 0 0) 0x7efeb8083570 con 0x7efec4159530 2024-09-13T16:35:48.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec3fff700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efea8066730 0x7efea8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.307+0000 7efec3fff700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efea8066730 0x7efea8068bf0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efeb00090b0 tx=0x7efeb0010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:48.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.319+0000 7efec17fa700 1 -- 172.21.15.119:0/123525781 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efeb8050ad0 con 0x7efec4159530 2024-09-13T16:35:48.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.479+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efec40b4d30 con 0x7efec4159530 2024-09-13T16:35:48.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.479+0000 7efec17fa700 1 -- 172.21.15.119:0/123525781 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v117) v1 ==== 93+0+10 (secure 0 0 0) 0x7efeb80540f0 con 0x7efec4159530 2024-09-13T16:35:48.485 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efea8066730 msgr2=0x7efea8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efea8066730 0x7efea8068bf0 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7efeb00090b0 tx=0x7efeb0010000 comp rx=0 tx=0).stop 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efec4159530 msgr2=0x7efec406ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efec4159530 0x7efec406ce10 secure :-1 s=READY pgs=1760 cs=0 l=1 rev1=1 crypto rx=0x7efeb800c580 tx=0x7efeb800c940 comp rx=0 tx=0).stop 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 shutdown_connections 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efea8066730 0x7efea8068bf0 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efec415d1c0 0x7efec4157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efec4159910 0x7efec414cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 --2- 172.21.15.119:0/123525781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efec4159530 0x7efec406ce10 unknown :-1 s=CLOSED pgs=1760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 >> 172.21.15.119:0/123525781 conn(0x7efec406b460 msgr2=0x7efec406a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:48.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 shutdown_connections 2024-09-13T16:35:48.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.483+0000 7efec9ddd700 1 -- 172.21.15.119:0/123525781 wait complete. 2024-09-13T16:35:48.500 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-09-13T16:35:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 Processor -- start 2024-09-13T16:35:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- start start 2024-09-13T16:35:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 0x7fa0480b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0480b2280 con 0x7fa0481594f0 2024-09-13T16:35:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0480cc890 con 0x7fa0480b3630 2024-09-13T16:35:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0480b3ac0 con 0x7fa04806dc40 2024-09-13T16:35:48.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa0481594f0 0x7fa04806d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:60332/0 (socket says 172.21.15.119:60332) 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 -- 172.21.15.119:0/8414530 learned_addr learned my addr 172.21.15.119:0/8414530 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d040700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 0x7fa0480b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2171448651 0 0) 0x7fa0480b2280 con 0x7fa0481594f0 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0340057e0 con 0x7fa0481594f0 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1450395764 0 0) 0x7fa0340057e0 con 0x7fa0481594f0 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 msgr2=0x7fa0480b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 0x7fa0480b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 >> v1:172.21.15.145:6789/0 conn(0x7fa0480b3630 legacy=0x7fa04806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0480b1970 con 0x7fa0481594f0 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fa03c001670 con 0x7fa0481594f0 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 >> v1:172.21.15.119:6789/0 conn(0x7fa0481594f0 legacy=0x7fa04806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa034005c60 0x7fa0340060a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 0x7fa034006a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034006fc0 0x7fa034007440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0480b3ac0 con 0x7fa034005c60 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa03400dd20 con 0x7fa034006fc0 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03400dee0 con 0x7fa04806dc40 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa034005c60 0x7fa0340060a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d040700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 0x7fa034006a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e042700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034006fc0 0x7fa034007440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 -- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 msgr2=0x7fa034006a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa04806dc40 0x7fa034006a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 -- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034006fc0 msgr2=0x7fa034007440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034006fc0 0x7fa034007440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 -- 172.21.15.119:0/8414530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0340057e0 con 0x7fa034005c60 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04d841700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa034005c60 0x7fa0340060a0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00c270 tx=0x7fa034007ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=527e483577759b2e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c007c30 con 0x7fa034005c60 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa03c00cdd0 con 0x7fa034005c60 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04c83f700 1 -- 172.21.15.119:0/8414530 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa03c005bc0 con 0x7fa034005c60 2024-09-13T16:35:48.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- 172.21.15.119:0/8414530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa034005c60 msgr2=0x7fa0340060a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 --2- 172.21.15.119:0/8414530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa034005c60 0x7fa0340060a0 secure :-1 s=READY pgs=1762 cs=0 l=1 rev1=1 crypto rx=0x7fa03c00c270 tx=0x7fa034007ab0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e042700 1 -- 172.21.15.119:0/8414530 reap_dead start 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- 172.21.15.119:0/8414530 shutdown_connections 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- 172.21.15.119:0/8414530 >> 172.21.15.119:0/8414530 conn(0x7fa04806a350 msgr2=0x7fa04806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- 172.21.15.119:0/8414530 shutdown_connections 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- 172.21.15.119:0/8414530 wait complete. 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 Processor -- start 2024-09-13T16:35:48.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- start start 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034005c60 0x7fa04814d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa048152540 0x7fa048141ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0480ac8c0 0x7fa0480acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa04815ab60 con 0x7fa048152540 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0480ad2b0 con 0x7fa034005c60 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.615+0000 7fa04e843700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0481f2650 con 0x7fa0480ac8c0 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa048152540 0x7fa048141ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa048152540 0x7fa048141ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41834/0 (socket says 172.21.15.119:41834) 2024-09-13T16:35:48.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 -- 172.21.15.119:0/1780050456 learned_addr learned my addr 172.21.15.119:0/1780050456 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04e042700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0480ac8c0 0x7fa0480acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d841700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034005c60 0x7fa04814d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 -- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0480ac8c0 msgr2=0x7fa0480acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0480ac8c0 0x7fa0480acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 -- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034005c60 msgr2=0x7fa04814d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034005c60 0x7fa04814d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 -- 172.21.15.119:0/1780050456 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa04815a550 con 0x7fa048152540 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04e042700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0480ac8c0 0x7fa0480acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04d040700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa048152540 0x7fa048141ff0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa044011e30 tx=0x7fa04400f490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa03a7fc700 1 -- 172.21.15.119:0/1780050456 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa04400e040 con 0x7fa048152540 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa03a7fc700 1 -- 172.21.15.119:0/1780050456 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa04400f040 con 0x7fa048152540 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa048153eb0 con 0x7fa048152540 2024-09-13T16:35:48.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa03a7fc700 1 -- 172.21.15.119:0/1780050456 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa044009c00 con 0x7fa048152540 2024-09-13T16:35:48.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa04814e1b0 con 0x7fa048152540 2024-09-13T16:35:48.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa04815aca0 con 0x7fa048152540 2024-09-13T16:35:48.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.619+0000 7fa03a7fc700 1 -- 172.21.15.119:0/1780050456 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa044015430 con 0x7fa048152540 2024-09-13T16:35:48.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.623+0000 7fa03a7fc700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa02c066760 0x7fa02c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:48.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.623+0000 7fa03a7fc700 1 -- 172.21.15.119:0/1780050456 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5349+0+0 (secure 0 0 0) 0x7fa0440847d0 con 0x7fa048152540 2024-09-13T16:35:48.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.623+0000 7fa04d841700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa02c066760 0x7fa02c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:48.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.623+0000 7fa04d841700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa02c066760 0x7fa02c068c20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa03c001930 tx=0x7fa03c007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:48.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.627+0000 7fa03a7fc700 1 -- 172.21.15.119:0/1780050456 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa04815aca0 con 0x7fa048152540 2024-09-13T16:35:48.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:48.863+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) v1 -- 0x7fa048153eb0 con 0x7fa048152540 2024-09-13T16:35:49.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.839+0000 7fa03a7fc700 1 -- 172.21.15.119:0/1780050456 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v482) v1 ==== 147+0+0 (secure 0 0 0) 0x7fa048153eb0 con 0x7fa048152540 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa02c066760 msgr2=0x7fa02c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa02c066760 0x7fa02c068c20 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7fa03c001930 tx=0x7fa03c007000 comp rx=0 tx=0).stop 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa048152540 msgr2=0x7fa048141ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa048152540 0x7fa048141ff0 secure :-1 s=READY pgs=1763 cs=0 l=1 rev1=1 crypto rx=0x7fa044011e30 tx=0x7fa04400f490 comp rx=0 tx=0).stop 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 shutdown_connections 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa02c066760 0x7fa02c068c20 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa0480ac8c0 0x7fa0480acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:49.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa048152540 0x7fa048141ff0 unknown :-1 s=CLOSED pgs=1763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:49.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 --2- 172.21.15.119:0/1780050456 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa034005c60 0x7fa04814d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:49.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 >> 172.21.15.119:0/1780050456 conn(0x7fa04806a350 msgr2=0x7fa04806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:49.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 shutdown_connections 2024-09-13T16:35:49.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.843+0000 7fa04e843700 1 -- 172.21.15.119:0/1780050456 wait complete. 2024-09-13T16:35:49.862 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-09-13T16:35:49.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2923fff700 1 Processor -- start 2024-09-13T16:35:49.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2923fff700 1 -- start start 2024-09-13T16:35:49.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2923fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241599c0 0x7f292406b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:49.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2923fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f292406b8a0 0x7f292406c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:49.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2923fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f292406bd00 con 0x7f29241595e0 2024-09-13T16:35:49.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2923fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f292406c3f0 con 0x7f292406b8a0 2024-09-13T16:35:49.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2923fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292406ee20 con 0x7f29241599c0 2024-09-13T16:35:49.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2922ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f29241595e0 0x7f292406e710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:60342/0 (socket says 172.21.15.119:60342) 2024-09-13T16:35:49.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f2922ffd700 1 -- 172.21.15.119:0/656948629 learned_addr learned my addr 172.21.15.119:0/656948629 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:49.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f29237fe700 1 --2- 172.21.15.119:0/656948629 >> v2:172.21.15.145:3300/0 conn(0x7f292406b8a0 0x7f292406c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:49.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f291a7fc700 1 --2- 172.21.15.119:0/656948629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241599c0 0x7f292406b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f29227fc700 1 -- 172.21.15.119:0/656948629 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2827370642 0 0) 0x7f292406bd00 con 0x7f29241595e0 2024-09-13T16:35:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f29227fc700 1 -- 172.21.15.119:0/656948629 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29100035d0 con 0x7f29241595e0 2024-09-13T16:35:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f291a7fc700 1 -- 172.21.15.119:0/656948629 >> v2:172.21.15.145:3300/0 conn(0x7f292406b8a0 msgr2=0x7f292406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f291a7fc700 1 --2- 172.21.15.119:0/656948629 >> v2:172.21.15.145:3300/0 conn(0x7f292406b8a0 0x7f292406c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f291a7fc700 1 -- 172.21.15.119:0/656948629 >> v1:172.21.15.119:6789/0 conn(0x7f29241595e0 legacy=0x7f292406e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f29227fc700 1 -- 172.21.15.119:0/656948629 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 736155106 0 0) 0x7f29100035d0 con 0x7f29241595e0 2024-09-13T16:35:49.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f291a7fc700 1 -- 172.21.15.119:0/656948629 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29240710b0 con 0x7f29241599c0 2024-09-13T16:35:49.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f29237fe700 1 --2- 172.21.15.119:0/656948629 >> v2:172.21.15.145:3300/0 conn(0x7f292406b8a0 0x7f292406c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:49.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.991+0000 7f291a7fc700 1 --2- 172.21.15.119:0/656948629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241599c0 0x7f292406b360 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f290c009850 tx=0x7f290c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2e8297104710ff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:49.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f29227fc700 1 -- 172.21.15.119:0/656948629 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2914001670 con 0x7f29241599c0 2024-09-13T16:35:49.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f29227fc700 1 -- 172.21.15.119:0/656948629 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290c004030 con 0x7f29241599c0 2024-09-13T16:35:49.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f29227fc700 1 -- 172.21.15.119:0/656948629 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f290c0043a0 con 0x7f29241599c0 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 -- 172.21.15.119:0/656948629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241599c0 msgr2=0x7f292406b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 --2- 172.21.15.119:0/656948629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241599c0 0x7f292406b360 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f290c009850 tx=0x7f290c009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 -- 172.21.15.119:0/656948629 shutdown_connections 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 --2- 172.21.15.119:0/656948629 >> v2:172.21.15.145:3300/0 conn(0x7f292406b8a0 0x7f292406c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 --2- 172.21.15.119:0/656948629 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241599c0 0x7f292406b360 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 -- 172.21.15.119:0/656948629 >> 172.21.15.119:0/656948629 conn(0x7f29240cb8f0 msgr2=0x7f29240cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 -- 172.21.15.119:0/656948629 shutdown_connections 2024-09-13T16:35:49.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 -- 172.21.15.119:0/656948629 wait complete. 2024-09-13T16:35:49.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 Processor -- start 2024-09-13T16:35:49.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.995+0000 7f2923fff700 1 -- start start 2024-09-13T16:35:50.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f292406b8a0 0x7f29241f3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241595e0 0x7f2924151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29241599c0 0x7f292415ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f292415b170 con 0x7f29241599c0 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29241f4240 con 0x7f292406b8a0 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29241f4520 con 0x7f29241595e0 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29241599c0 0x7f292415ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29241599c0 0x7f292415ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41836/0 (socket says 172.21.15.119:41836) 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 -- 172.21.15.119:0/4182221013 learned_addr learned my addr 172.21.15.119:0/4182221013 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2922ffd700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f292406b8a0 0x7f29241f3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f291a7fc700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241595e0 0x7f2924151d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 -- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241595e0 msgr2=0x7f2924151d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241595e0 0x7f2924151d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 -- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f292406b8a0 msgr2=0x7f29241f3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f292406b8a0 0x7f29241f3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 -- 172.21.15.119:0/4182221013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c009a70 con 0x7f29241599c0 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f29237fe700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29241599c0 0x7f292415ac30 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f291c00efc0 tx=0x7f291c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f291bfff700 1 -- 172.21.15.119:0/4182221013 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c00ca90 con 0x7f29241599c0 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f291bfff700 1 -- 172.21.15.119:0/4182221013 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291c00f040 con 0x7f29241599c0 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f291bfff700 1 -- 172.21.15.119:0/4182221013 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291c00ce10 con 0x7f29241599c0 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290c0094b0 con 0x7f29241599c0 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:49.999+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29241f4ae0 con 0x7f29241599c0 2024-09-13T16:35:50.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.003+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29240b3e10 con 0x7f29241599c0 2024-09-13T16:35:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.003+0000 7f291bfff700 1 -- 172.21.15.119:0/4182221013 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f291c014480 con 0x7f29241599c0 2024-09-13T16:35:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.003+0000 7f291bfff700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2904066730 0x7f2904068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.003+0000 7f291bfff700 1 -- 172.21.15.119:0/4182221013 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f291c0847a0 con 0x7f29241599c0 2024-09-13T16:35:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.003+0000 7f2922ffd700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2904066730 0x7f2904068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.003+0000 7f2922ffd700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2904066730 0x7f2904068bf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f29240b3d90 tx=0x7f2914001e40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.011+0000 7f291bfff700 1 -- 172.21.15.119:0/4182221013 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f291c051b50 con 0x7f29241599c0 2024-09-13T16:35:50.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.183+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f2924157980 con 0x7f29241599c0 2024-09-13T16:35:50.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.191+0000 7f291bfff700 1 -- 172.21.15.119:0/4182221013 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f291c009d70 con 0x7f29241599c0 2024-09-13T16:35:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2904066730 msgr2=0x7f2904068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2904066730 0x7f2904068bf0 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f29240b3d90 tx=0x7f2914001e40 comp rx=0 tx=0).stop 2024-09-13T16:35:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29241599c0 msgr2=0x7f292415ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29241599c0 0x7f292415ac30 secure :-1 s=READY pgs=1765 cs=0 l=1 rev1=1 crypto rx=0x7f291c00efc0 tx=0x7f291c00c3a0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 shutdown_connections 2024-09-13T16:35:50.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2904066730 0x7f2904068bf0 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f29241599c0 0x7f292415ac30 unknown :-1 s=CLOSED pgs=1765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f29241595e0 0x7f2924151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 --2- 172.21.15.119:0/4182221013 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f292406b8a0 0x7f29241f3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 >> 172.21.15.119:0/4182221013 conn(0x7f29240cb8f0 msgr2=0x7f29240ccd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 shutdown_connections 2024-09-13T16:35:50.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.195+0000 7f2923fff700 1 -- 172.21.15.119:0/4182221013 wait complete. 2024-09-13T16:35:50.210 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:35:50.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:35:50.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:35:50.247 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:35:50.247 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:35:50.247 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:50.278 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:35:50.278 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:35:50.278 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:50.286 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:35:50.286 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:35:50.286 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:50.293 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:35:50.294 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:35:50.294 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:35:50.333 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:35:50.333 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-09-13T16:35:50.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:50.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc98759e700 1 Processor -- start 2024-09-13T16:35:50.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc98759e700 1 -- start start 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc98759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc98815cd60 0x7fc98806b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc98759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc9880cf3c0 0x7fc9880cf7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc98759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9880ceb60 con 0x7fc9880cfd00 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc98759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc98815d140 con 0x7fc9880cf3c0 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc98759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880d3ca0 con 0x7fc98815cd60 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.451+0000 7fc985d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc9880cf3c0 0x7fc9880cf7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc985d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc9880cf3c0 0x7fc9880cf7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41840/0 (socket says 172.21.15.119:41840) 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc985d9b700 1 -- 172.21.15.119:0/3782569912 learned_addr learned my addr 172.21.15.119:0/3782569912 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:50.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98659c700 1 --2- 172.21.15.119:0/3782569912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc98815cd60 0x7fc98806b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc985d9b700 1 -- 172.21.15.119:0/3782569912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc98815cd60 msgr2=0x7fc98806b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc985d9b700 1 --2- 172.21.15.119:0/3782569912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc98815cd60 0x7fc98806b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc985d9b700 1 -- 172.21.15.119:0/3782569912 >> v1:172.21.15.119:6789/0 conn(0x7fc9880cfd00 legacy=0x7fc9880d0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98559a700 1 -- 172.21.15.119:0/3782569912 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676375252 0 0) 0x7fc9880ceb60 con 0x7fc9880cfd00 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc985d9b700 1 -- 172.21.15.119:0/3782569912 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9880d01b0 con 0x7fc9880cf3c0 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98659c700 1 --2- 172.21.15.119:0/3782569912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc98815cd60 0x7fc98806b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc985d9b700 1 --2- 172.21.15.119:0/3782569912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9880cf3c0 0x7fc9880cf7c0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fc978009910 tx=0x7fc978009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c07e0a1a239d8f74 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98559a700 1 -- 172.21.15.119:0/3782569912 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc97800ba60 con 0x7fc9880cf3c0 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98559a700 1 -- 172.21.15.119:0/3782569912 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc97800bbc0 con 0x7fc9880cf3c0 2024-09-13T16:35:50.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98559a700 1 -- 172.21.15.119:0/3782569912 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc978009e60 con 0x7fc9880cf3c0 2024-09-13T16:35:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 -- 172.21.15.119:0/3782569912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9880cf3c0 msgr2=0x7fc9880cf7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 --2- 172.21.15.119:0/3782569912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9880cf3c0 0x7fc9880cf7c0 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fc978009910 tx=0x7fc978009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 -- 172.21.15.119:0/3782569912 shutdown_connections 2024-09-13T16:35:50.458 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 --2- 172.21.15.119:0/3782569912 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc9880cf3c0 0x7fc9880cf7c0 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 --2- 172.21.15.119:0/3782569912 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc98815cd60 0x7fc98806b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 -- 172.21.15.119:0/3782569912 >> 172.21.15.119:0/3782569912 conn(0x7fc98808c240 msgr2=0x7fc98809c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:50.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 -- 172.21.15.119:0/3782569912 shutdown_connections 2024-09-13T16:35:50.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 -- 172.21.15.119:0/3782569912 wait complete. 2024-09-13T16:35:50.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 Processor -- start 2024-09-13T16:35:50.459 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.455+0000 7fc98759e700 1 -- start start 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9880cf3c0 0x7fc9881f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9880cfd00 0x7fc98815fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc98815cd60 0x7fc9880c8850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880cc9f0 con 0x7fc98815cd60 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880cce10 con 0x7fc9880cf3c0 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9880cc050 con 0x7fc9880cfd00 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc98815cd60 0x7fc9880c8850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc98815cd60 0x7fc9880c8850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41854/0 (socket says 172.21.15.119:41854) 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98659c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9880cf3c0 0x7fc9881f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 -- 172.21.15.119:0/3801776067 learned_addr learned my addr 172.21.15.119:0/3801776067 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:50.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc985d9b700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9880cfd00 0x7fc98815fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 -- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9880cfd00 msgr2=0x7fc98815fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9880cfd00 0x7fc98815fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 -- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9880cf3c0 msgr2=0x7fc9881f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9880cf3c0 0x7fc9881f6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 -- 172.21.15.119:0/3801776067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc978009570 con 0x7fc98815cd60 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc986d9d700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc98815cd60 0x7fc9880c8850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fc98000cc70 tx=0x7fc98000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc976ffd700 1 -- 172.21.15.119:0/3801776067 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc980003e50 con 0x7fc98815cd60 2024-09-13T16:35:50.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc976ffd700 1 -- 172.21.15.119:0/3801776067 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc98000a6f0 con 0x7fc98815cd60 2024-09-13T16:35:50.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9880cc330 con 0x7fc98815cd60 2024-09-13T16:35:50.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc976ffd700 1 -- 172.21.15.119:0/3801776067 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc98000aaa0 con 0x7fc98815cd60 2024-09-13T16:35:50.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.459+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9880cbd40 con 0x7fc98815cd60 2024-09-13T16:35:50.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.463+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9880ccb30 con 0x7fc98815cd60 2024-09-13T16:35:50.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.463+0000 7fc976ffd700 1 -- 172.21.15.119:0/3801776067 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc98000ac00 con 0x7fc98815cd60 2024-09-13T16:35:50.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.463+0000 7fc976ffd700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc968066730 0x7fc968068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.463+0000 7fc976ffd700 1 -- 172.21.15.119:0/3801776067 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 5349+0+0 (secure 0 0 0) 0x7fc980083bd0 con 0x7fc98815cd60 2024-09-13T16:35:50.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.463+0000 7fc98659c700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc968066730 0x7fc968068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.467+0000 7fc98659c700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc968066730 0x7fc968068bf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0044b0 tx=0x7fc97c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.471+0000 7fc976ffd700 1 -- 172.21.15.119:0/3801776067 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc9880ccb30 con 0x7fc98815cd60 2024-09-13T16:35:50.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.675+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9880ccb30 con 0x7fc98815cd60 2024-09-13T16:35:50.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc976ffd700 1 -- 172.21.15.119:0/3801776067 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+3538 (secure 0 0 0) 0x7fc9880ccb30 con 0x7fc98815cd60 2024-09-13T16:35:50.679 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:50.679 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":86,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7502,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6836/4091693192","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":4091693192},{"type":"v1","addr":"172.21.15.145:6836","nonce":4091693192}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":7505,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1944414797","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6835","nonce":1944414797},{"type":"v1","addr":"172.21.15.145:6837","nonce":1944414797}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":8823,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/3964942482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":3964942482},{"type":"v1","addr":"172.21.15.119:6833","nonce":3964942482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":72},{"gid":9381,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/734046953","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":734046953},{"type":"v1","addr":"172.21.15.119:6835","nonce":734046953}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":84}],"filesystems":[]} 2024-09-13T16:35:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc968066730 msgr2=0x7fc968068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc968066730 0x7fc968068bf0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fc97c0044b0 tx=0x7fc97c008040 comp rx=0 tx=0).stop 2024-09-13T16:35:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc98815cd60 msgr2=0x7fc9880c8850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc98815cd60 0x7fc9880c8850 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fc98000cc70 tx=0x7fc98000cf80 comp rx=0 tx=0).stop 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 shutdown_connections 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc968066730 0x7fc968068bf0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc98815cd60 0x7fc9880c8850 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9880cfd00 0x7fc98815fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 --2- 172.21.15.119:0/3801776067 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9880cf3c0 0x7fc9881f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 >> 172.21.15.119:0/3801776067 conn(0x7fc98808c240 msgr2=0x7fc9880b1570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 shutdown_connections 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.679+0000 7fc98759e700 1 -- 172.21.15.119:0/3801776067 wait complete. 2024-09-13T16:35:50.682 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 86 2024-09-13T16:35:50.698 INFO:tasks.ceph.mds.a:Restarting daemon 2024-09-13T16:35:50.698 INFO:tasks.ceph.mds.a:Stopping old one... 2024-09-13T16:35:50.698 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:35:50.698 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:35:50.699 INFO:tasks.ceph.mds.c:Restarting daemon 2024-09-13T16:35:50.699 INFO:tasks.ceph.mds.c:Stopping old one... 2024-09-13T16:35:50.699 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:35:50.699 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:35:50.699 INFO:tasks.ceph.mds.b:Restarting daemon 2024-09-13T16:35:50.699 INFO:tasks.ceph.mds.b:Stopping old one... 2024-09-13T16:35:50.699 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:35:50.700 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:35:50.700 INFO:tasks.ceph.mds.d:Restarting daemon 2024-09-13T16:35:50.700 INFO:tasks.ceph.mds.d:Stopping old one... 2024-09-13T16:35:50.700 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:35:50.700 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:35:50.715 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:35:50.715 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-09-13T16:35:50.716 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:35:50.716 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-09-13T16:35:50.718 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:35:50.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-09-13T16:35:50.719 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:35:50.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-09-13T16:35:50.720 INFO:tasks.ceph.mds.b:Started 2024-09-13T16:35:50.720 DEBUG:teuthology.parallel:result is None 2024-09-13T16:35:50.721 INFO:tasks.ceph.mds.a:Started 2024-09-13T16:35:50.721 DEBUG:teuthology.parallel:result is None 2024-09-13T16:35:50.722 INFO:tasks.ceph.mds.d:Started 2024-09-13T16:35:50.722 DEBUG:teuthology.parallel:result is None 2024-09-13T16:35:50.723 INFO:tasks.ceph.mds.c:Started 2024-09-13T16:35:50.723 DEBUG:teuthology.parallel:result is None 2024-09-13T16:35:50.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-09-13T16:35:50.789 INFO:tasks.ceph.mds.c.smithi119.stdout:starting mds.c at 2024-09-13T16:35:50.792 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:35:50.795 INFO:tasks.ceph.mds.a.smithi119.stdout:starting mds.a at 2024-09-13T16:35:50.804 INFO:tasks.ceph.mds.d.smithi145.stdout:starting mds.d at 2024-09-13T16:35:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.835+0000 7f2e58ba7700 1 Processor -- start 2024-09-13T16:35:50.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.835+0000 7f2e58ba7700 1 -- start start 2024-09-13T16:35:50.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2e5415a770 0x7f2e5415b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2e5415ab50 0x7f2e54070e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e5406a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e540cdbd0 con 0x7f2e5415a770 2024-09-13T16:35:50.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e54071dd0 con 0x7f2e5415ab50 2024-09-13T16:35:50.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e54158fd0 con 0x7f2e54158b40 2024-09-13T16:35:50.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2e5415a770 0x7f2e5415b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2e5415a770 0x7f2e5415b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41878/0 (socket says 172.21.15.119:41878) 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 -- 172.21.15.119:0/527817490 learned_addr learned my addr 172.21.15.119:0/527817490 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e52d9d700 1 --2- 172.21.15.119:0/527817490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e5406a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 -- 172.21.15.119:0/527817490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 msgr2=0x7f2e5406a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e51d9b700 1 --2- 172.21.15.119:0/527817490 >> v2:172.21.15.145:3300/0 conn(0x7f2e5415ab50 0x7f2e54070e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 --2- 172.21.15.119:0/527817490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e5406a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 -- 172.21.15.119:0/527817490 >> v2:172.21.15.145:3300/0 conn(0x7f2e5415ab50 msgr2=0x7f2e54070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 --2- 172.21.15.119:0/527817490 >> v2:172.21.15.145:3300/0 conn(0x7f2e5415ab50 0x7f2e54070e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 -- 172.21.15.119:0/527817490 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e54071530 con 0x7f2e5415a770 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e51d9b700 1 --2- 172.21.15.119:0/527817490 >> v2:172.21.15.145:3300/0 conn(0x7f2e5415ab50 0x7f2e54070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:50.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5259c700 1 --2- 172.21.15.119:0/527817490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e5415b980 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f2e540cf170 tx=0x7f2e4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=753ca55c6c27e2bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5159a700 1 -- 172.21.15.119:0/527817490 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c00e070 con 0x7f2e5415a770 2024-09-13T16:35:50.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5159a700 1 -- 172.21.15.119:0/527817490 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e4c00b9c0 con 0x7f2e5415a770 2024-09-13T16:35:50.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e5159a700 1 -- 172.21.15.119:0/527817490 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e4c00bd70 con 0x7f2e5415a770 2024-09-13T16:35:50.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- 172.21.15.119:0/527817490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 msgr2=0x7f2e5415b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/527817490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e5415b980 secure :-1 s=READY pgs=1774 cs=0 l=1 rev1=1 crypto rx=0x7f2e540cf170 tx=0x7f2e4c00b330 comp rx=0 tx=0).stop 2024-09-13T16:35:50.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- 172.21.15.119:0/527817490 shutdown_connections 2024-09-13T16:35:50.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/527817490 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e5406a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/527817490 >> v2:172.21.15.145:3300/0 conn(0x7f2e5415ab50 0x7f2e54070e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/527817490 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e5415b980 unknown :-1 s=CLOSED pgs=1774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- 172.21.15.119:0/527817490 >> 172.21.15.119:0/527817490 conn(0x7f2e540b1650 msgr2=0x7f2e540b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:50.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- 172.21.15.119:0/527817490 shutdown_connections 2024-09-13T16:35:50.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.839+0000 7f2e58ba7700 1 -- 172.21.15.119:0/527817490 wait complete. 2024-09-13T16:35:50.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 Processor -- start 2024-09-13T16:35:50.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 -- start start 2024-09-13T16:35:50.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e54153540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e541591a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e5415ab50 0x7f2e541596e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e540b3140 con 0x7f2e5415a770 2024-09-13T16:35:50.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e54157b80 con 0x7f2e5415ab50 2024-09-13T16:35:50.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e541578a0 con 0x7f2e54158b40 2024-09-13T16:35:50.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e541591a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e541591a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41882/0 (socket says 172.21.15.119:41882) 2024-09-13T16:35:50.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 -- 172.21.15.119:0/649370007 learned_addr learned my addr 172.21.15.119:0/649370007 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:50.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e52d9d700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e5415ab50 0x7f2e541596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e5259c700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e54153540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 -- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 msgr2=0x7f2e54153540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e54153540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 -- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e5415ab50 msgr2=0x7f2e541596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:50.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e5415ab50 0x7f2e541596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:50.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 -- 172.21.15.119:0/649370007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c009b30 con 0x7f2e5415a770 2024-09-13T16:35:50.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e51d9b700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e541591a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400c7c0 tx=0x7f2e4400cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e4affd700 1 -- 172.21.15.119:0/649370007 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e44011800 con 0x7f2e5415a770 2024-09-13T16:35:50.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e4affd700 1 -- 172.21.15.119:0/649370007 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e44011960 con 0x7f2e5415a770 2024-09-13T16:35:50.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e540b2af0 con 0x7f2e5415a770 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e4affd700 1 -- 172.21.15.119:0/649370007 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e44011cf0 con 0x7f2e5415a770 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.843+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e54153a80 con 0x7f2e5415a770 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.847+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e540b32f0 con 0x7f2e5415a770 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.847+0000 7f2e4affd700 1 -- 172.21.15.119:0/649370007 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2e44021350 con 0x7f2e5415a770 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.847+0000 7f2e4affd700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e34066730 0x7f2e34068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.847+0000 7f2e4affd700 1 -- 172.21.15.119:0/649370007 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 5349+0+0 (secure 0 0 0) 0x7f2e44083d30 con 0x7f2e5415a770 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.847+0000 7f2e5259c700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e34066730 0x7f2e34068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:50.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.847+0000 7f2e5259c700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e34066730 0x7f2e34068bf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00b370 tx=0x7f2e4c009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:50.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:50.855+0000 7f2e4affd700 1 -- 172.21.15.119:0/649370007 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2e540b32f0 con 0x7f2e5415a770 2024-09-13T16:35:51.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.035+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2e54067a40 con 0x7f2e5415a770 2024-09-13T16:35:51.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.851+0000 7f2e4affd700 1 -- 172.21.15.119:0/649370007 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v122) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2e44051210 con 0x7f2e5415a770 2024-09-13T16:35:51.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e34066730 msgr2=0x7f2e34068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:51.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e34066730 0x7f2e34068bf0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c00b370 tx=0x7f2e4c009000 comp rx=0 tx=0).stop 2024-09-13T16:35:51.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 msgr2=0x7f2e541591a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:51.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e541591a0 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7f2e4400c7c0 tx=0x7f2e4400cad0 comp rx=0 tx=0).stop 2024-09-13T16:35:51.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 shutdown_connections 2024-09-13T16:35:51.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e34066730 0x7f2e34068bf0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:51.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e5415ab50 0x7f2e541596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:51.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e5415a770 0x7f2e541591a0 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:51.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 --2- 172.21.15.119:0/649370007 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e54158b40 0x7f2e54153540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:51.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 >> 172.21.15.119:0/649370007 conn(0x7f2e540b1650 msgr2=0x7f2e541547f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:51.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 shutdown_connections 2024-09-13T16:35:51.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.855+0000 7f2e58ba7700 1 -- 172.21.15.119:0/649370007 wait complete. 2024-09-13T16:35:51.858 INFO:teuthology.orchestra.run.smithi119.stderr: removed all blocklist entries 2024-09-13T16:35:51.873 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-09-13T16:35:51.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff875c0a700 1 Processor -- start 2024-09-13T16:35:51.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff875c0a700 1 -- start start 2024-09-13T16:35:51.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff875c0a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff870070980 0x7ff87006d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:51.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff875c0a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8700b2940 con 0x7ff870158f10 2024-09-13T16:35:51.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff875c0a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8700b2380 con 0x7ff8700b2c10 2024-09-13T16:35:51.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff875c0a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8700b2660 con 0x7ff870070980 2024-09-13T16:35:51.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff867fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff870158f10 0x7ff87006c9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:60348/0 (socket says 172.21.15.119:60348) 2024-09-13T16:35:51.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff867fff700 1 -- 172.21.15.119:0/2949285821 learned_addr learned my addr 172.21.15.119:0/2949285821 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:51.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.987+0000 7ff875409700 1 --2- 172.21.15.119:0/2949285821 >> v2:172.21.15.145:3300/0 conn(0x7ff870070980 0x7ff87006d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:51.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff8677fe700 1 -- 172.21.15.119:0/2949285821 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712964529 0 0) 0x7ff8700b2940 con 0x7ff870158f10 2024-09-13T16:35:51.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff8677fe700 1 -- 172.21.15.119:0/2949285821 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8580035d0 con 0x7ff870158f10 2024-09-13T16:35:51.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875409700 1 -- 172.21.15.119:0/2949285821 >> v1:172.21.15.145:6789/0 conn(0x7ff8700b2c10 legacy=0x7ff8700b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:51.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875409700 1 -- 172.21.15.119:0/2949285821 >> v1:172.21.15.119:6789/0 conn(0x7ff870158f10 legacy=0x7ff87006c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:51.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875409700 1 -- 172.21.15.119:0/2949285821 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87006c4c0 con 0x7ff870070980 2024-09-13T16:35:51.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875409700 1 --2- 172.21.15.119:0/2949285821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff870070980 0x7ff87006d0f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff868009d90 tx=0x7ff86800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=873024c036bb35b6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:51.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff8677fe700 1 -- 172.21.15.119:0/2949285821 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86800e070 con 0x7ff870070980 2024-09-13T16:35:51.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff8677fe700 1 -- 172.21.15.119:0/2949285821 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86800bb20 con 0x7ff870070980 2024-09-13T16:35:51.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff8677fe700 1 -- 172.21.15.119:0/2949285821 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff868012430 con 0x7ff870070980 2024-09-13T16:35:51.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875c0a700 1 -- 172.21.15.119:0/2949285821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff870070980 msgr2=0x7ff87006d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:51.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875c0a700 1 --2- 172.21.15.119:0/2949285821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff870070980 0x7ff87006d0f0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7ff868009d90 tx=0x7ff86800b330 comp rx=0 tx=0).stop 2024-09-13T16:35:51.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875c0a700 1 -- 172.21.15.119:0/2949285821 shutdown_connections 2024-09-13T16:35:51.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875c0a700 1 --2- 172.21.15.119:0/2949285821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff870070980 0x7ff87006d0f0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:51.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875c0a700 1 -- 172.21.15.119:0/2949285821 >> 172.21.15.119:0/2949285821 conn(0x7ff87006a350 msgr2=0x7ff87006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:51.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875c0a700 1 -- 172.21.15.119:0/2949285821 shutdown_connections 2024-09-13T16:35:51.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.991+0000 7ff875c0a700 1 -- 172.21.15.119:0/2949285821 wait complete. 2024-09-13T16:35:51.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 Processor -- start 2024-09-13T16:35:51.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 -- start start 2024-09-13T16:35:51.999 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff870070980 0x7ff87014e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8700b2c10 0x7ff8700b0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff870158f10 0x7ff87015b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700af4b0 con 0x7ff870070980 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff870152260 con 0x7ff8700b2c10 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff875c0a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8700af770 con 0x7ff870158f10 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff874c08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff870070980 0x7ff87014e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff874c08700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff870070980 0x7ff87014e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41894/0 (socket says 172.21.15.119:41894) 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.995+0000 7ff874c08700 1 -- 172.21.15.119:0/1000501632 learned_addr learned my addr 172.21.15.119:0/1000501632 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff867fff700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8700b2c10 0x7ff8700b0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff875409700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff870158f10 0x7ff87015b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff874c08700 1 -- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff870158f10 msgr2=0x7ff87015b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff874c08700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff870158f10 0x7ff87015b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff874c08700 1 -- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8700b2c10 msgr2=0x7ff8700b0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff874c08700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8700b2c10 0x7ff8700b0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff874c08700 1 -- 172.21.15.119:0/1000501632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff868009a70 con 0x7ff870070980 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff875409700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff870158f10 0x7ff87015b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff867fff700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8700b2c10 0x7ff8700b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff874c08700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff870070980 0x7ff87014e450 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c500 tx=0x7ff86c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff8657fa700 1 -- 172.21.15.119:0/1000501632 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c0052d0 con 0x7ff870070980 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8700afa50 con 0x7ff870070980 2024-09-13T16:35:52.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff8657fa700 1 -- 172.21.15.119:0/1000501632 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff86c005430 con 0x7ff870070980 2024-09-13T16:35:52.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8700b0790 con 0x7ff870070980 2024-09-13T16:35:52.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff8657fa700 1 -- 172.21.15.119:0/1000501632 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff86c0055a0 con 0x7ff870070980 2024-09-13T16:35:52.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff8657fa700 1 -- 172.21.15.119:0/1000501632 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff86c0038a0 con 0x7ff870070980 2024-09-13T16:35:52.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8700b2ff0 con 0x7ff870070980 2024-09-13T16:35:52.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:51.999+0000 7ff8657fa700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff85c066730 0x7ff85c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.003+0000 7ff8657fa700 1 -- 172.21.15.119:0/1000501632 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5177+0+0 (secure 0 0 0) 0x7ff86c0832e0 con 0x7ff870070980 2024-09-13T16:35:52.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.003+0000 7ff867fff700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff85c066730 0x7ff85c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.003+0000 7ff867fff700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff85c066730 0x7ff85c068bf0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff8700ad640 tx=0x7ff860009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.007+0000 7ff8657fa700 1 -- 172.21.15.119:0/1000501632 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff86c050900 con 0x7ff870070980 2024-09-13T16:35:52.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.239+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ff87006aaf0 con 0x7ff870070980 2024-09-13T16:35:52.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.239+0000 7ff8657fa700 1 -- 172.21.15.119:0/1000501632 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v17) v1 ==== 80+0+5304 (secure 0 0 0) 0x7ff86c053f20 con 0x7ff870070980 2024-09-13T16:35:52.241 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:52.241 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:35:52.241 INFO:teuthology.orchestra.run.smithi119.stdout: "auth_dump": [ 2024-09-13T16:35:52.241 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.241 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.a", 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmxtYxNhAA2/LuWjOgzEX6GYFZXh15dA==", 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.b", 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm8/wPAhAArxItA1NeaBXhEJkrCLRy4g==", 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.242 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.c", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmrG/JORAAtsv+Cccy1+ZawD4uB5hCoQ==", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:35:52.243 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mds.d", 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+Rm3Ae4BRAA2rsbd+/peSR3oeL8JnxTYw==", 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.244 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.0", 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA+Z+RmhkSkLxAAxdl+aMM2hRp/xcRudHBJqA==", 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.1", 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBBZ+RmEeZMMhAAWJQbfekaJrImN9yI7B+hzw==", 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.245 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.2", 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBEZ+RmiEjfORAAo7e8AthzVahZz6NEK32WeQ==", 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.246 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.3", 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBIZ+Rmk9IWARAAg4BuQ7j8RL9MT1HQXKf8ww==", 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.247 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.4", 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBNZ+RmnugiHRAAFyuiBIcotdih4a6o6oxZbA==", 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.5", 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBQZ+RmL1AEIRAA5PLSTYHAbQnEcmoPBdG84Q==", 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.248 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.6", 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBTZ+RmT6m4KRAA5jgjivgCvsnkI9Sq0KrWFg==", 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.249 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "osd.7", 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQBWZ+RmhRZbLhAAXAkB0tYduvU6U3r1UrCAfQ==", 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow profile osd", 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile osd", 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.250 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.251 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.0", 2024-09-13T16:35:52.251 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmvNgFCRAArtn7VEDtT89478yQ2FZfWQ==", 2024-09-13T16:35:52.251 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.251 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.251 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow r", 2024-09-13T16:35:52.251 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rw tag cephfs data=cephfs" 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.1", 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmcnZPDxAA7/OdOLp0se+QGVptqivCuQ==", 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.252 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.2", 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfLoeDBAAKiV4r8o1dRH7F2oylY6eWw==", 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.253 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.3", 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA8Z+RmfmRgEhAADKbQE+KeKM3CD8OnHDjZow==", 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow", 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow r", 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow rw", 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow rwx" 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "client.admin", 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmpEcAJRAAYorOK9AU5Q/2uOTIUrZhJw==", 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.254 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mgr": "allow *", 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow *", 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.x", 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+RmkwtAMhAARGuSZy2IscRHaOdnrClzKQ==", 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.255 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: "entity": "mgr.y", 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: "key": "AQA7Z+Rm6qQSLxAAFdoypw8mFCcs8QDoRJVVyw==", 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: "caps": { 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mds": "allow *", 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: "mon": "allow profile mgr", 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: "osd": "allow *" 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout: ] 2024-09-13T16:35:52.256 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:35:52.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff85c066730 msgr2=0x7ff85c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff85c066730 0x7ff85c068bf0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7ff8700ad640 tx=0x7ff860009040 comp rx=0 tx=0).stop 2024-09-13T16:35:52.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff870070980 msgr2=0x7ff87014e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff870070980 0x7ff87014e450 secure :-1 s=READY pgs=1777 cs=0 l=1 rev1=1 crypto rx=0x7ff86c00c500 tx=0x7ff86c00c8c0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 shutdown_connections 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff85c066730 0x7ff85c068bf0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff870158f10 0x7ff87015b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff8700b2c10 0x7ff8700b0250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 --2- 172.21.15.119:0/1000501632 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff870070980 0x7ff87014e450 unknown :-1 s=CLOSED pgs=1777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 >> 172.21.15.119:0/1000501632 conn(0x7ff87006a350 msgr2=0x7ff87006ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 shutdown_connections 2024-09-13T16:35:52.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.243+0000 7ff875c0a700 1 -- 172.21.15.119:0/1000501632 wait complete. 2024-09-13T16:35:52.260 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:52.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bcad42700 1 Processor -- start 2024-09-13T16:35:52.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bcad42700 1 -- start start 2024-09-13T16:35:52.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bcad42700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cc060 0x7f0bc40cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bcad42700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0bc40cde70 0x7f0bc40b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bcad42700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bc406cf60 con 0x7f0bc40adb30 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bcad42700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bc40cc4a0 con 0x7f0bc40cde70 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bcad42700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc406c350 con 0x7f0bc40cc060 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.375+0000 7f0bca541700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0bc40cde70 0x7f0bc40b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bca541700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0bc40cde70 0x7f0bc40b2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41904/0 (socket says 172.21.15.119:41904) 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bca541700 1 -- 172.21.15.119:0/709564565 learned_addr learned my addr 172.21.15.119:0/709564565 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bc953f700 1 --2- 172.21.15.119:0/709564565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cc060 0x7f0bc40cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bca541700 1 -- 172.21.15.119:0/709564565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cc060 msgr2=0x7f0bc40cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bca541700 1 --2- 172.21.15.119:0/709564565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cc060 0x7f0bc40cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bca541700 1 -- 172.21.15.119:0/709564565 >> v1:172.21.15.145:6789/0 conn(0x7f0bc40adb30 legacy=0x7f0bc406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bca541700 1 -- 172.21.15.119:0/709564565 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bc406c630 con 0x7f0bc40cde70 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bc953f700 1 --2- 172.21.15.119:0/709564565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cc060 0x7f0bc40cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:52.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bca541700 1 --2- 172.21.15.119:0/709564565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40cde70 0x7f0bc40b2380 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc009910 tx=0x7f0bbc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15d80e6adfd543fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bc8d3e700 1 -- 172.21.15.119:0/709564565 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc00ba60 con 0x7f0bc40cde70 2024-09-13T16:35:52.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bc8d3e700 1 -- 172.21.15.119:0/709564565 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bbc00bbc0 con 0x7f0bc40cde70 2024-09-13T16:35:52.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bc8d3e700 1 -- 172.21.15.119:0/709564565 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bbc009e60 con 0x7f0bc40cde70 2024-09-13T16:35:52.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 -- 172.21.15.119:0/709564565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40cde70 msgr2=0x7f0bc40b2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 --2- 172.21.15.119:0/709564565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40cde70 0x7f0bc40b2380 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7f0bbc009910 tx=0x7f0bbc009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:52.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 -- 172.21.15.119:0/709564565 shutdown_connections 2024-09-13T16:35:52.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 --2- 172.21.15.119:0/709564565 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40cde70 0x7f0bc40b2380 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 --2- 172.21.15.119:0/709564565 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cc060 0x7f0bc40cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 -- 172.21.15.119:0/709564565 >> 172.21.15.119:0/709564565 conn(0x7f0bc406a350 msgr2=0x7f0bc406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:52.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 -- 172.21.15.119:0/709564565 shutdown_connections 2024-09-13T16:35:52.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 -- 172.21.15.119:0/709564565 wait complete. 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 Processor -- start 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.379+0000 7f0bcad42700 1 -- start start 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40adb30 0x7f0bc40ca9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bc40cc060 0x7f0bc40acc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cde70 0x7f0bc415c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc413ebc0 con 0x7f0bc40adb30 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc413edf0 con 0x7f0bc40cc060 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bc413f090 con 0x7f0bc40cde70 2024-09-13T16:35:52.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40adb30 0x7f0bc40ca9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40adb30 0x7f0bc40ca9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:41916/0 (socket says 172.21.15.119:41916) 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 -- 172.21.15.119:0/3281033630 learned_addr learned my addr 172.21.15.119:0/3281033630 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc953f700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bc40cc060 0x7f0bc40acc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bca541700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cde70 0x7f0bc415c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 -- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cde70 msgr2=0x7f0bc415c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cde70 0x7f0bc415c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 -- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bc40cc060 msgr2=0x7f0bc40acc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bc40cc060 0x7f0bc40acc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 -- 172.21.15.119:0/3281033630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bbc009570 con 0x7f0bc40adb30 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bc9d40700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40adb30 0x7f0bc40ca9e0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800c500 tx=0x7f0bb800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bb67fc700 1 -- 172.21.15.119:0/3281033630 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80052d0 con 0x7f0bc40adb30 2024-09-13T16:35:52.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bb67fc700 1 -- 172.21.15.119:0/3281033630 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bb8005430 con 0x7f0bc40adb30 2024-09-13T16:35:52.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bc413f370 con 0x7f0bc40adb30 2024-09-13T16:35:52.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bb67fc700 1 -- 172.21.15.119:0/3281033630 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bb80055a0 con 0x7f0bc40adb30 2024-09-13T16:35:52.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bc40caf20 con 0x7f0bc40adb30 2024-09-13T16:35:52.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.383+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bc40adf10 con 0x7f0bc40adb30 2024-09-13T16:35:52.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.387+0000 7f0bb67fc700 1 -- 172.21.15.119:0/3281033630 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0bb80038e0 con 0x7f0bc40adb30 2024-09-13T16:35:52.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.387+0000 7f0bb67fc700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ba8066730 0x7f0ba8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.387+0000 7f0bb67fc700 1 -- 172.21.15.119:0/3281033630 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5177+0+0 (secure 0 0 0) 0x7f0bb80834b0 con 0x7f0bc40adb30 2024-09-13T16:35:52.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.387+0000 7f0bc953f700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ba8066730 0x7f0ba8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.387+0000 7f0bc953f700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ba8066730 0x7f0ba8068bf0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0bc41f3520 tx=0x7f0bc0007810 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.391+0000 7f0bb67fc700 1 -- 172.21.15.119:0/3281033630 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0bb8050900 con 0x7f0bc40adb30 2024-09-13T16:35:52.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.563+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0bc413a770 con 0x7f0bc40adb30 2024-09-13T16:35:52.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bb67fc700 1 -- 172.21.15.119:0/3281033630 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v122) v1 ==== 74+0+13132 (secure 0 0 0) 0x7f0bb8053f20 con 0x7f0bc40adb30 2024-09-13T16:35:52.567 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:52.568 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":122,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:51.838295+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":112,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ba8066730 msgr2=0x7f0ba8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ba8066730 0x7f0ba8068bf0 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f0bc41f3520 tx=0x7f0bc0007810 comp rx=0 tx=0).stop 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40adb30 msgr2=0x7f0bc40ca9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40adb30 0x7f0bc40ca9e0 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7f0bb800c500 tx=0x7f0bb800c810 comp rx=0 tx=0).stop 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 shutdown_connections 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ba8066730 0x7f0ba8068bf0 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0bc40cde70 0x7f0bc415c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0bc40cc060 0x7f0bc40acc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 --2- 172.21.15.119:0/3281033630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0bc40adb30 0x7f0bc40ca9e0 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 >> 172.21.15.119:0/3281033630 conn(0x7f0bc406a350 msgr2=0x7f0bc40aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:52.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 shutdown_connections 2024-09-13T16:35:52.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.567+0000 7f0bcad42700 1 -- 172.21.15.119:0/3281033630 wait complete. 2024-09-13T16:35:52.583 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:35:52.583 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:35:52.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 Processor -- start 2024-09-13T16:35:52.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 -- start start 2024-09-13T16:35:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7faa5c06c480 0x7faa5c0cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c06c8c0 con 0x7faa5c0ce000 2024-09-13T16:35:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa5c06cb40 con 0x7faa5c144180 2024-09-13T16:35:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa5c0b1c50 con 0x7faa5c06c480 2024-09-13T16:35:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7faa5c06c480 0x7faa5c0cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61522700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7faa5c0ce000 0x7faa5c0b20a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57432/0 (socket says 172.21.15.119:57432) 2024-09-13T16:35:52.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61522700 1 -- 172.21.15.119:0/3083138328 learned_addr learned my addr 172.21.15.119:0/3083138328 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:52.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5bfff700 1 -- 172.21.15.119:0/3083138328 >> v1:172.21.15.145:6789/0 conn(0x7faa5c144180 legacy=0x7faa5c0cc010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5bfff700 1 -- 172.21.15.119:0/3083138328 >> v1:172.21.15.119:6789/0 conn(0x7faa5c0ce000 legacy=0x7faa5c0b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5bfff700 1 -- 172.21.15.119:0/3083138328 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa5c06ce20 con 0x7faa5c06c480 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5b7fe700 1 -- 172.21.15.119:0/3083138328 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429047476 0 0) 0x7faa5c06c8c0 con 0x7faa5c0ce000 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5bfff700 1 --2- 172.21.15.119:0/3083138328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0cdac0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0b1d90 tx=0x7faa48009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a51e931ae01fcbb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5b7fe700 1 -- 172.21.15.119:0/3083138328 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa48004230 con 0x7faa5c06c480 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5b7fe700 1 -- 172.21.15.119:0/3083138328 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa48004390 con 0x7faa5c06c480 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa5b7fe700 1 -- 172.21.15.119:0/3083138328 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa480036c0 con 0x7faa5c06c480 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 -- 172.21.15.119:0/3083138328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 msgr2=0x7faa5c0cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 --2- 172.21.15.119:0/3083138328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0cdac0 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0b1d90 tx=0x7faa48009b30 comp rx=0 tx=0).stop 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 -- 172.21.15.119:0/3083138328 shutdown_connections 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 --2- 172.21.15.119:0/3083138328 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0cdac0 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.699+0000 7faa61d23700 1 -- 172.21.15.119:0/3083138328 >> 172.21.15.119:0/3083138328 conn(0x7faa5c0b0980 msgr2=0x7faa5c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- 172.21.15.119:0/3083138328 shutdown_connections 2024-09-13T16:35:52.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- 172.21.15.119:0/3083138328 wait complete. 2024-09-13T16:35:52.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 Processor -- start 2024-09-13T16:35:52.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- start start 2024-09-13T16:35:52.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa5c0ce000 0x7faa5c0acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa5c144180 0x7faa5c1359e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c13fb00 con 0x7faa5c06c480 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c140d80 con 0x7faa5c144180 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa5c071750 con 0x7faa5c0ce000 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0c37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56178/0 (socket says 172.21.15.119:56178) 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 -- 172.21.15.119:0/3555059229 learned_addr learned my addr 172.21.15.119:0/3555059229 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61522700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa5c144180 0x7faa5c1359e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa5bfff700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa5c0ce000 0x7faa5c0acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 -- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa5c0ce000 msgr2=0x7faa5c0acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa5c0ce000 0x7faa5c0acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 -- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa5c144180 msgr2=0x7faa5c1359e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa5c144180 0x7faa5c1359e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 -- 172.21.15.119:0/3555059229 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa480094b0 con 0x7faa5c06c480 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61522700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa5c144180 0x7faa5c1359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa60d21700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0c37d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7faa5000c530 tx=0x7faa5000c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa597fa700 1 -- 172.21.15.119:0/3555059229 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa500052d0 con 0x7faa5c06c480 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa597fa700 1 -- 172.21.15.119:0/3555059229 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa50005430 con 0x7faa5c06c480 2024-09-13T16:35:52.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa5c071a30 con 0x7faa5c06c480 2024-09-13T16:35:52.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa597fa700 1 -- 172.21.15.119:0/3555059229 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa500055a0 con 0x7faa5c06c480 2024-09-13T16:35:52.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.703+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5c13f790 con 0x7faa5c06c480 2024-09-13T16:35:52.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.707+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa5c0cdac0 con 0x7faa5c06c480 2024-09-13T16:35:52.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.707+0000 7faa597fa700 1 -- 172.21.15.119:0/3555059229 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7faa500038e0 con 0x7faa5c06c480 2024-09-13T16:35:52.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.707+0000 7faa597fa700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa40066730 0x7faa40068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.707+0000 7faa597fa700 1 -- 172.21.15.119:0/3555059229 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5177+0+0 (secure 0 0 0) 0x7faa500833c0 con 0x7faa5c06c480 2024-09-13T16:35:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.707+0000 7faa5bfff700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa40066730 0x7faa40068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:52.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.711+0000 7faa5bfff700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa40066730 0x7faa40068bf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0b1d90 tx=0x7faa48008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:52.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.711+0000 7faa597fa700 1 -- 172.21.15.119:0/3555059229 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7faa500508a0 con 0x7faa5c06c480 2024-09-13T16:35:52.881 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.879+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faa5c15d020 con 0x7faa5c06c480 2024-09-13T16:35:52.882 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.879+0000 7faa597fa700 1 -- 172.21.15.119:0/3555059229 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v122) v1 ==== 93+0+10 (secure 0 0 0) 0x7faa50053ec0 con 0x7faa5c06c480 2024-09-13T16:35:52.882 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:35:52.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa40066730 msgr2=0x7faa40068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa40066730 0x7faa40068bf0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7faa5c0b1d90 tx=0x7faa48008030 comp rx=0 tx=0).stop 2024-09-13T16:35:52.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 msgr2=0x7faa5c0c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:52.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0c37d0 secure :-1 s=READY pgs=1782 cs=0 l=1 rev1=1 crypto rx=0x7faa5000c530 tx=0x7faa5000c840 comp rx=0 tx=0).stop 2024-09-13T16:35:52.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 shutdown_connections 2024-09-13T16:35:52.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faa40066730 0x7faa40068bf0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7faa5c144180 0x7faa5c1359e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7faa5c0ce000 0x7faa5c0acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 --2- 172.21.15.119:0/3555059229 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7faa5c06c480 0x7faa5c0c37d0 unknown :-1 s=CLOSED pgs=1782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:52.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 >> 172.21.15.119:0/3555059229 conn(0x7faa5c0b0980 msgr2=0x7faa5c15b690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:52.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 shutdown_connections 2024-09-13T16:35:52.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:52.883+0000 7faa61d23700 1 -- 172.21.15.119:0/3555059229 wait complete. 2024-09-13T16:35:52.897 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-09-13T16:35:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fd341700 1 Processor -- start 2024-09-13T16:35:53.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fd341700 1 -- start start 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fd341700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb9f00df8e0 0x7fb9f001c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fd341700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f0005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fd341700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f00e3310 con 0x7fb9f004fe30 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fd341700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9f00e35f0 con 0x7fb9f00df8e0 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fd341700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00e38d0 con 0x7fb9f0050b90 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f77fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb9f004fe30 0x7fb9f0050480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57438/0 (socket says 172.21.15.119:57438) 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f77fe700 1 -- 172.21.15.119:0/2153352797 learned_addr learned my addr 172.21.15.119:0/2153352797 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:53.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f7fff700 1 --2- 172.21.15.119:0/2153352797 >> v2:172.21.15.145:3300/0 conn(0x7fb9f00df8e0 0x7fb9f001c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9fcb40700 1 --2- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f0005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f6ffd700 1 -- 172.21.15.119:0/2153352797 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120084615 0 0) 0x7fb9f00e3310 con 0x7fb9f004fe30 2024-09-13T16:35:53.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f6ffd700 1 -- 172.21.15.119:0/2153352797 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e00035d0 con 0x7fb9f004fe30 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f7fff700 1 -- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 msgr2=0x7fb9f0005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f7fff700 1 --2- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f0005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f7fff700 1 -- 172.21.15.119:0/2153352797 >> v1:172.21.15.119:6789/0 conn(0x7fb9f004fe30 legacy=0x7fb9f0050480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f6ffd700 1 -- 172.21.15.119:0/2153352797 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 14748765 0 0) 0x7fb9e00035d0 con 0x7fb9f004fe30 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.007+0000 7fb9f7fff700 1 -- 172.21.15.119:0/2153352797 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f00e3bb0 con 0x7fb9f00df8e0 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9f7fff700 1 --2- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 0x7fb9f001c5d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009840 tx=0x7fb9ec009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecdc1b5a19a2bb1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9f6ffd700 1 -- 172.21.15.119:0/2153352797 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9e8001670 con 0x7fb9f00df8e0 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9f6ffd700 1 -- 172.21.15.119:0/2153352797 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec004030 con 0x7fb9f00df8e0 2024-09-13T16:35:53.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9f6ffd700 1 -- 172.21.15.119:0/2153352797 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0043a0 con 0x7fb9f00df8e0 2024-09-13T16:35:53.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 -- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 msgr2=0x7fb9f001c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 --2- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 0x7fb9f001c5d0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fb9ec009840 tx=0x7fb9ec009b50 comp rx=0 tx=0).stop 2024-09-13T16:35:53.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 -- 172.21.15.119:0/2153352797 shutdown_connections 2024-09-13T16:35:53.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 --2- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f0005b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 --2- 172.21.15.119:0/2153352797 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 0x7fb9f001c5d0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 -- 172.21.15.119:0/2153352797 >> 172.21.15.119:0/2153352797 conn(0x7fb9f0036140 msgr2=0x7fb9f0036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:53.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 -- 172.21.15.119:0/2153352797 shutdown_connections 2024-09-13T16:35:53.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.011+0000 7fb9fd341700 1 -- 172.21.15.119:0/2153352797 wait complete. 2024-09-13T16:35:53.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 Processor -- start 2024-09-13T16:35:53.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 -- start start 2024-09-13T16:35:53.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9f004fe30 0x7fb9f004c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f00cfa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 0x7fb9f00e2830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f004d8e0 con 0x7fb9f004fe30 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00cff70 con 0x7fb9f00df8e0 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fd341700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9f00e2d70 con 0x7fb9f0050b90 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9f004fe30 0x7fb9f004c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9f004fe30 0x7fb9f004c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56182/0 (socket says 172.21.15.119:56182) 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 -- 172.21.15.119:0/1363339041 learned_addr learned my addr 172.21.15.119:0/1363339041 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9fcb40700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 0x7fb9f00e2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f77fe700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f00cfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 -- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 msgr2=0x7fb9f00cfa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f00cfa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 -- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 msgr2=0x7fb9f00e2830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 0x7fb9f00e2830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.015+0000 7fb9f7fff700 1 -- 172.21.15.119:0/1363339041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9ec009450 con 0x7fb9f004fe30 2024-09-13T16:35:53.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f77fe700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f00cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:53.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f7fff700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9f004fe30 0x7fb9f004c5d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb9f001ce30 tx=0x7fb9ec004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:53.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f4ff9700 1 -- 172.21.15.119:0/1363339041 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec003dc0 con 0x7fb9f004fe30 2024-09-13T16:35:53.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f4ff9700 1 -- 172.21.15.119:0/1363339041 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9ec011630 con 0x7fb9f004fe30 2024-09-13T16:35:53.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f004cc60 con 0x7fb9f004fe30 2024-09-13T16:35:53.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f4ff9700 1 -- 172.21.15.119:0/1363339041 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9ec0119e0 con 0x7fb9f004fe30 2024-09-13T16:35:53.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f004d220 con 0x7fb9f004fe30 2024-09-13T16:35:53.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f4ff9700 1 -- 172.21.15.119:0/1363339041 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb9ec022070 con 0x7fb9f004fe30 2024-09-13T16:35:53.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f0007b40 con 0x7fb9f004fe30 2024-09-13T16:35:53.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f4ff9700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb9e4066730 0x7fb9e4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f4ff9700 1 -- 172.21.15.119:0/1363339041 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5177+0+0 (secure 0 0 0) 0x7fb9ec085770 con 0x7fb9f004fe30 2024-09-13T16:35:53.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.019+0000 7fb9f77fe700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb9e4066730 0x7fb9e4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.023+0000 7fb9f77fe700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb9e4066730 0x7fb9e4068bf0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8001eb0 tx=0x7fb9e8001d00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:53.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.031+0000 7fb9f4ff9700 1 -- 172.21.15.119:0/1363339041 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb9ec052d80 con 0x7fb9f004fe30 2024-09-13T16:35:53.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb9f00e3310 con 0x7fb9f004fe30 2024-09-13T16:35:53.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9f4ff9700 1 -- 172.21.15.119:0/1363339041 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v122) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb9ec01aeb0 con 0x7fb9f004fe30 2024-09-13T16:35:53.200 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:53.200 INFO:teuthology.orchestra.run.smithi119.stdout:[ 2024-09-13T16:35:53.200 INFO:teuthology.orchestra.run.smithi119.stdout: { 2024-09-13T16:35:53.200 INFO:teuthology.orchestra.run.smithi119.stdout: "poolnum": 1, 2024-09-13T16:35:53.200 INFO:teuthology.orchestra.run.smithi119.stdout: "poolname": ".mgr" 2024-09-13T16:35:53.200 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:35:53.200 INFO:teuthology.orchestra.run.smithi119.stdout:] 2024-09-13T16:35:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb9e4066730 msgr2=0x7fb9e4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb9e4066730 0x7fb9e4068bf0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fb9e8001eb0 tx=0x7fb9e8001d00 comp rx=0 tx=0).stop 2024-09-13T16:35:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9f004fe30 msgr2=0x7fb9f004c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9f004fe30 0x7fb9f004c5d0 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7fb9f001ce30 tx=0x7fb9ec004770 comp rx=0 tx=0).stop 2024-09-13T16:35:53.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 shutdown_connections 2024-09-13T16:35:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb9e4066730 0x7fb9e4068bf0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb9f00df8e0 0x7fb9f00e2830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb9f0050b90 0x7fb9f00cfa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 --2- 172.21.15.119:0/1363339041 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb9f004fe30 0x7fb9f004c5d0 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 >> 172.21.15.119:0/1363339041 conn(0x7fb9f0036140 msgr2=0x7fb9f004af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 shutdown_connections 2024-09-13T16:35:53.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.199+0000 7fb9fd341700 1 -- 172.21.15.119:0/1363339041 wait complete. 2024-09-13T16:35:53.215 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-09-13T16:35:53.215 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-09-13T16:35:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.331+0000 7fb0056d4700 1 Processor -- start 2024-09-13T16:35:53.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.331+0000 7fb0056d4700 1 -- start start 2024-09-13T16:35:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0000cfb90 0x7fb0000cd850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0000b5a90 con 0x7fb000160340 2024-09-13T16:35:53.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0000d1220 con 0x7fb0000cfb90 2024-09-13T16:35:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000d1500 con 0x7fb0000d3760 2024-09-13T16:35:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0000cfb90 0x7fb0000cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0000cfb90 0x7fb0000cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56186/0 (socket says 172.21.15.119:56186) 2024-09-13T16:35:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 -- 172.21.15.119:0/3572737847 learned_addr learned my addr 172.21.15.119:0/3572737847 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:53.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fafff7fe700 1 --2- 172.21.15.119:0/3572737847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 -- 172.21.15.119:0/3572737847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 msgr2=0x7fb0000cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:53.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 --2- 172.21.15.119:0/3572737847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 -- 172.21.15.119:0/3572737847 >> v1:172.21.15.145:6789/0 conn(0x7fb000160340 legacy=0x7fb0000cf480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 -- 172.21.15.119:0/3572737847 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0000d17e0 con 0x7fb0000cfb90 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffe7fc700 1 --2- 172.21.15.119:0/3572737847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0000cd850 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7faff0009910 tx=0x7faff0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae33eb55ec71c8bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffdffb700 1 -- 172.21.15.119:0/3572737847 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0011830 con 0x7fb0000cfb90 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffdffb700 1 -- 172.21.15.119:0/3572737847 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff0011990 con 0x7fb0000cfb90 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7faffdffb700 1 -- 172.21.15.119:0/3572737847 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff0011d40 con 0x7fb0000cfb90 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- 172.21.15.119:0/3572737847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 msgr2=0x7fb0000cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 --2- 172.21.15.119:0/3572737847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0000cd850 secure :-1 s=READY pgs=1785 cs=0 l=1 rev1=1 crypto rx=0x7faff0009910 tx=0x7faff0009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- 172.21.15.119:0/3572737847 shutdown_connections 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 --2- 172.21.15.119:0/3572737847 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 --2- 172.21.15.119:0/3572737847 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0000cd850 unknown :-1 s=CLOSED pgs=1785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- 172.21.15.119:0/3572737847 >> 172.21.15.119:0/3572737847 conn(0x7fb0000d2c40 msgr2=0x7fb0000d3050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:53.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- 172.21.15.119:0/3572737847 shutdown_connections 2024-09-13T16:35:53.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.335+0000 7fb0056d4700 1 -- 172.21.15.119:0/3572737847 wait complete. 2024-09-13T16:35:53.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 Processor -- start 2024-09-13T16:35:53.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 -- start start 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0001594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb000160340 0x7fb00015a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000caf00 con 0x7fb0000cfb90 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb000159a10 con 0x7fb000160340 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0000ca280 con 0x7fb0000d3760 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0001594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0001594d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56188/0 (socket says 172.21.15.119:56188) 2024-09-13T16:35:53.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 -- 172.21.15.119:0/4200438827 learned_addr learned my addr 172.21.15.119:0/4200438827 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:53.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fafff7fe700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb000160340 0x7fb00015a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffe7fc700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 -- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 msgr2=0x7fb0000c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 -- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb000160340 msgr2=0x7fb00015a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb000160340 0x7fb00015a580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 -- 172.21.15.119:0/4200438827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff0009570 con 0x7fb0000cfb90 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7faffeffd700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0001594d0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7faff400c500 tx=0x7faff400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fafe77fe700 1 -- 172.21.15.119:0/4200438827 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40052d0 con 0x7fb0000cfb90 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fafe77fe700 1 -- 172.21.15.119:0/4200438827 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faff4005430 con 0x7fb0000cfb90 2024-09-13T16:35:53.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0000ca560 con 0x7fb0000cfb90 2024-09-13T16:35:53.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fafe77fe700 1 -- 172.21.15.119:0/4200438827 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faff40055a0 con 0x7fb0000cfb90 2024-09-13T16:35:53.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.339+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0000cab20 con 0x7fb0000cfb90 2024-09-13T16:35:53.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.343+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0000cb040 con 0x7fb0000cfb90 2024-09-13T16:35:53.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.343+0000 7fafe77fe700 1 -- 172.21.15.119:0/4200438827 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7faff40038e0 con 0x7fb0000cfb90 2024-09-13T16:35:53.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.343+0000 7fafe77fe700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafec066730 0x7fafec068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:53.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.343+0000 7fafe77fe700 1 -- 172.21.15.119:0/4200438827 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5177+0+0 (secure 0 0 0) 0x7faff40832a0 con 0x7fb0000cfb90 2024-09-13T16:35:53.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.347+0000 7faffe7fc700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafec066730 0x7fafec068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:53.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.347+0000 7faffe7fc700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafec066730 0x7fafec068bf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7faff0000c00 tx=0x7faff0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:53.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.351+0000 7fafe77fe700 1 -- 172.21.15.119:0/4200438827 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7faff4050810 con 0x7fb0000cfb90 2024-09-13T16:35:53.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.519+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fb000160720 con 0x7fb0000cfb90 2024-09-13T16:35:53.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.863+0000 7fafe77fe700 1 -- 172.21.15.119:0/4200438827 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v123) v1 ==== 138+0+0 (secure 0 0 0) 0x7faff4053e30 con 0x7fb0000cfb90 2024-09-13T16:35:53.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafec066730 msgr2=0x7fafec068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafec066730 0x7fafec068bf0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7faff0000c00 tx=0x7faff0008030 comp rx=0 tx=0).stop 2024-09-13T16:35:53.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 msgr2=0x7fb0001594d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:53.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0001594d0 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7faff400c500 tx=0x7faff400c810 comp rx=0 tx=0).stop 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 shutdown_connections 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fafec066730 0x7fafec068bf0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb000160340 0x7fb00015a580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb0000d3760 0x7fb0000c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 --2- 172.21.15.119:0/4200438827 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0000cfb90 0x7fb0001594d0 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 >> 172.21.15.119:0/4200438827 conn(0x7fb0000d2c40 msgr2=0x7fb0000cd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 shutdown_connections 2024-09-13T16:35:53.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:53.875+0000 7fb0056d4700 1 -- 172.21.15.119:0/4200438827 wait complete. 2024-09-13T16:35:53.877 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' created 2024-09-13T16:35:53.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-09-13T16:35:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f105c893700 1 Processor -- start 2024-09-13T16:35:54.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f105c893700 1 -- start start 2024-09-13T16:35:54.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f105c893700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cf250 0x7f10580ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:54.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f105c893700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580cf690 con 0x7f105815a6e0 2024-09-13T16:35:54.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f105c893700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10580cd670 con 0x7f10580cd240 2024-09-13T16:35:54.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f105c893700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f105815eb10 con 0x7f10580cf250 2024-09-13T16:35:54.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f1056d9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f105815a6e0 0x7f105815e2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57440/0 (socket says 172.21.15.119:57440) 2024-09-13T16:35:54.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f1056d9d700 1 -- 172.21.15.119:0/1167038654 learned_addr learned my addr 172.21.15.119:0/1167038654 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:54.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f105659c700 1 --2- 172.21.15.119:0/1167038654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cf250 0x7f10580ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:54.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f1055d9b700 1 -- 172.21.15.119:0/1167038654 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122355590 0 0) 0x7f1048001110 con 0x7f105815a6e0 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f1055d9b700 1 -- 172.21.15.119:0/1167038654 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10400035d0 con 0x7f105815a6e0 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f1055d9b700 1 -- 172.21.15.119:0/1167038654 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357189236 0 0) 0x7f10580cd670 con 0x7f10580cd240 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.011+0000 7f1055d9b700 1 -- 172.21.15.119:0/1167038654 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1048001110 con 0x7f10580cd240 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105659c700 1 -- 172.21.15.119:0/1167038654 >> v1:172.21.15.145:6789/0 conn(0x7f10580cd240 legacy=0x7f105815c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105659c700 1 -- 172.21.15.119:0/1167038654 >> v1:172.21.15.119:6789/0 conn(0x7f105815a6e0 legacy=0x7f105815e2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105659c700 1 -- 172.21.15.119:0/1167038654 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105815edf0 con 0x7f10580cf250 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105659c700 1 --2- 172.21.15.119:0/1167038654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cf250 0x7f10580ccd00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f104c009880 tx=0x7f104c009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52dc063d0acc6039 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f1055d9b700 1 -- 172.21.15.119:0/1167038654 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c011830 con 0x7f10580cf250 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f1055d9b700 1 -- 172.21.15.119:0/1167038654 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c011990 con 0x7f10580cf250 2024-09-13T16:35:54.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f1055d9b700 1 -- 172.21.15.119:0/1167038654 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c011d00 con 0x7f10580cf250 2024-09-13T16:35:54.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105c893700 1 -- 172.21.15.119:0/1167038654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cf250 msgr2=0x7f10580ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:54.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105c893700 1 --2- 172.21.15.119:0/1167038654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cf250 0x7f10580ccd00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f104c009880 tx=0x7f104c009b90 comp rx=0 tx=0).stop 2024-09-13T16:35:54.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105c893700 1 -- 172.21.15.119:0/1167038654 shutdown_connections 2024-09-13T16:35:54.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105c893700 1 --2- 172.21.15.119:0/1167038654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cf250 0x7f10580ccd00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:54.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105c893700 1 -- 172.21.15.119:0/1167038654 >> 172.21.15.119:0/1167038654 conn(0x7f10580b15e0 msgr2=0x7f10580b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:54.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105c893700 1 -- 172.21.15.119:0/1167038654 shutdown_connections 2024-09-13T16:35:54.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.015+0000 7f105c893700 1 -- 172.21.15.119:0/1167038654 wait complete. 2024-09-13T16:35:54.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 Processor -- start 2024-09-13T16:35:54.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 -- start start 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cd240 0x7f105814adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10580cf250 0x7f105814fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f105815a6e0 0x7f1058152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f105814b330 con 0x7f10580cf250 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1058153850 con 0x7f105815a6e0 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10580cf250 0x7f105814fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.019+0000 7f105c893700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1058153b30 con 0x7f10580cd240 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10580cf250 0x7f105814fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56190/0 (socket says 172.21.15.119:56190) 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 -- 172.21.15.119:0/3465340004 learned_addr learned my addr 172.21.15.119:0/3465340004 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f1056d9d700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cd240 0x7f105814adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:54.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105759e700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f105815a6e0 0x7f1058152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 -- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cd240 msgr2=0x7f105814adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cd240 0x7f105814adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 -- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f105815a6e0 msgr2=0x7f1058152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f105815a6e0 0x7f1058152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 -- 172.21.15.119:0/3465340004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f104c0094b0 con 0x7f10580cf250 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f1056d9d700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cd240 0x7f105814adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105759e700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f105815a6e0 0x7f1058152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105659c700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10580cf250 0x7f105814fde0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f104c013040 tx=0x7f104c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f103f7fe700 1 -- 172.21.15.119:0/3465340004 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c004610 con 0x7f10580cf250 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f103f7fe700 1 -- 172.21.15.119:0/3465340004 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f104c0036c0 con 0x7f10580cf250 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f105814c4a0 con 0x7f10580cf250 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f103f7fe700 1 -- 172.21.15.119:0/3465340004 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f104c003a50 con 0x7f10580cf250 2024-09-13T16:35:54.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10581529d0 con 0x7f10580cf250 2024-09-13T16:35:54.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.023+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f105814c300 con 0x7f10580cf250 2024-09-13T16:35:54.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.027+0000 7f103f7fe700 1 -- 172.21.15.119:0/3465340004 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f104c003bb0 con 0x7f10580cf250 2024-09-13T16:35:54.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.027+0000 7f103f7fe700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1044066760 0x7f1044068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:54.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.027+0000 7f103f7fe700 1 -- 172.21.15.119:0/3465340004 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5547+0+0 (secure 0 0 0) 0x7f104c0855e0 con 0x7f10580cf250 2024-09-13T16:35:54.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.027+0000 7f1056d9d700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1044066760 0x7f1044068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:54.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.027+0000 7f1056d9d700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1044066760 0x7f1044068c20 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1048000f60 tx=0x7f1048009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:54.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.035+0000 7f103f7fe700 1 -- 172.21.15.119:0/3465340004 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f105814c300 con 0x7f10580cf250 2024-09-13T16:35:54.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.207+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f105814c9c0 con 0x7f10580cf250 2024-09-13T16:35:54.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.871+0000 7f103f7fe700 1 -- 172.21.15.119:0/3465340004 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v124) v1 ==== 170+0+0 (secure 0 0 0) 0x7f105814c9c0 con 0x7f10580cf250 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1044066760 msgr2=0x7f1044068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1044066760 0x7f1044068c20 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f1048000f60 tx=0x7f1048009040 comp rx=0 tx=0).stop 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10580cf250 msgr2=0x7f105814fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10580cf250 0x7f105814fde0 secure :-1 s=READY pgs=1788 cs=0 l=1 rev1=1 crypto rx=0x7f104c013040 tx=0x7f104c004770 comp rx=0 tx=0).stop 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 shutdown_connections 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1044066760 0x7f1044068c20 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f105815a6e0 0x7f1058152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f10580cf250 0x7f105814fde0 unknown :-1 s=CLOSED pgs=1788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:54.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 --2- 172.21.15.119:0/3465340004 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f10580cd240 0x7f105814adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 >> 172.21.15.119:0/3465340004 conn(0x7f10580b15e0 msgr2=0x7f10580b0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 shutdown_connections 2024-09-13T16:35:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:54.891+0000 7f105c893700 1 -- 172.21.15.119:0/3465340004 wait complete. 2024-09-13T16:35:54.894 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' created 2024-09-13T16:35:54.905 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-09-13T16:35:55.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.015+0000 7f2000ee3700 1 Processor -- start 2024-09-13T16:35:55.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.015+0000 7f2000ee3700 1 -- start start 2024-09-13T16:35:55.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1ffc15d150 0x7f1ffc15d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ffc06f480 0x7f1ffc0b4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ffc0b5b20 con 0x7f1ffc15da70 2024-09-13T16:35:55.020 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0b5280 con 0x7f1ffc15d150 2024-09-13T16:35:55.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ffc0b5560 con 0x7f1ffc06f480 2024-09-13T16:35:55.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffb7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1ffc15d150 0x7f1ffc15d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffbfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ffc06f480 0x7f1ffc0b4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffb7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1ffc15d150 0x7f1ffc15d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56196/0 (socket says 172.21.15.119:56196) 2024-09-13T16:35:55.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffbfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f1ffc06f480 0x7f1ffc0b4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:52748/0 (socket says 172.21.15.119:52748) 2024-09-13T16:35:55.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffb7fe700 1 -- 172.21.15.119:0/3075787044 learned_addr learned my addr 172.21.15.119:0/3075787044 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:55.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffbfff700 1 -- 172.21.15.119:0/3075787044 >> v2:172.21.15.119:3300/0 conn(0x7f1ffc15d150 msgr2=0x7f1ffc15d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffbfff700 1 --2- 172.21.15.119:0/3075787044 >> v2:172.21.15.119:3300/0 conn(0x7f1ffc15d150 0x7f1ffc15d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffbfff700 1 -- 172.21.15.119:0/3075787044 >> v1:172.21.15.145:6789/0 conn(0x7f1ffc15da70 legacy=0x7f1ffc0b4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffbfff700 1 -- 172.21.15.119:0/3075787044 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ffc0b5840 con 0x7f1ffc06f480 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffbfff700 1 --2- 172.21.15.119:0/3075787044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc06f480 0x7f1ffc0b4c20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc06cdd0 tx=0x7f1ff400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c419fe8bfcc8921c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffa7fc700 1 -- 172.21.15.119:0/3075787044 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff400e070 con 0x7f1ffc06f480 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffa7fc700 1 -- 172.21.15.119:0/3075787044 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff400bb20 con 0x7f1ffc06f480 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f1ffa7fc700 1 -- 172.21.15.119:0/3075787044 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4012430 con 0x7f1ffc06f480 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- 172.21.15.119:0/3075787044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc06f480 msgr2=0x7f1ffc0b4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 --2- 172.21.15.119:0/3075787044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc06f480 0x7f1ffc0b4c20 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc06cdd0 tx=0x7f1ff400b330 comp rx=0 tx=0).stop 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- 172.21.15.119:0/3075787044 shutdown_connections 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 --2- 172.21.15.119:0/3075787044 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc06f480 0x7f1ffc0b4c20 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 --2- 172.21.15.119:0/3075787044 >> v2:172.21.15.119:3300/0 conn(0x7f1ffc15d150 0x7f1ffc15d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- 172.21.15.119:0/3075787044 >> 172.21.15.119:0/3075787044 conn(0x7f1ffc0b5d90 msgr2=0x7f1ffc0b61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:55.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- 172.21.15.119:0/3075787044 shutdown_connections 2024-09-13T16:35:55.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.019+0000 7f2000ee3700 1 -- 172.21.15.119:0/3075787044 wait complete. 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 Processor -- start 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 -- start start 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ffc06f480 0x7f1ffc153270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ffc15d150 0x7f1ffc15f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc15da70 0x7f1ffc14a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc1537b0 con 0x7f1ffc06f480 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc149c30 con 0x7f1ffc15da70 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ffc149060 con 0x7f1ffc15d150 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ffc06f480 0x7f1ffc153270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ffc06f480 0x7f1ffc153270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56212/0 (socket says 172.21.15.119:56212) 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffb7fe700 1 -- 172.21.15.119:0/157346767 learned_addr learned my addr 172.21.15.119:0/157346767 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:55.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffbfff700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc15da70 0x7f1ffc14a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffaffd700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ffc15d150 0x7f1ffc15f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffbfff700 1 -- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ffc15d150 msgr2=0x7f1ffc15f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffbfff700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ffc15d150 0x7f1ffc15f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffbfff700 1 -- 172.21.15.119:0/157346767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ffc06f480 msgr2=0x7f1ffc153270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffbfff700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ffc06f480 0x7f1ffc153270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffbfff700 1 -- 172.21.15.119:0/157346767 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff4009a70 con 0x7f1ffc15da70 2024-09-13T16:35:55.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1ffbfff700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc15da70 0x7f1ffc14a0c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0cd0b0 tx=0x7f1ff4004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1fe3fff700 1 -- 172.21.15.119:0/157346767 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff4012430 con 0x7f1ffc15da70 2024-09-13T16:35:55.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1fe3fff700 1 -- 172.21.15.119:0/157346767 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ff4020070 con 0x7f1ffc15da70 2024-09-13T16:35:55.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ffc149300 con 0x7f1ffc15da70 2024-09-13T16:35:55.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f1fe3fff700 1 -- 172.21.15.119:0/157346767 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ff40098f0 con 0x7f1ffc15da70 2024-09-13T16:35:55.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.023+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ffc1498c0 con 0x7f1ffc15da70 2024-09-13T16:35:55.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.027+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ffc0d1770 con 0x7f1ffc15da70 2024-09-13T16:35:55.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.027+0000 7f1fe3fff700 1 -- 172.21.15.119:0/157346767 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1ff4004040 con 0x7f1ffc15da70 2024-09-13T16:35:55.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.027+0000 7f1fe3fff700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fe8066730 0x7f1fe8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.027+0000 7f1fe3fff700 1 -- 172.21.15.119:0/157346767 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5929+0+0 (secure 0 0 0) 0x7f1ff4022070 con 0x7f1ffc15da70 2024-09-13T16:35:55.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.027+0000 7f1ffb7fe700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fe8066730 0x7f1fe8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.031+0000 7f1ffb7fe700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fe8066730 0x7f1fe8068bf0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0006fd0 tx=0x7f1ff00092e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.035+0000 7f1fe3fff700 1 -- 172.21.15.119:0/157346767 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1ff4053320 con 0x7f1ffc15da70 2024-09-13T16:35:55.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.239+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1ffc15d530 con 0x7f1ffc15da70 2024-09-13T16:35:55.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.263+0000 7f1fe3fff700 1 -- 172.21.15.119:0/157346767 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 17 and data pool 18 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1ff402c020 con 0x7f1ffc15da70 2024-09-13T16:35:55.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.271+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fe8066730 msgr2=0x7f1fe8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.271+0000 7f2000ee3700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fe8066730 0x7f1fe8068bf0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0006fd0 tx=0x7f1ff00092e0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.271+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc15da70 msgr2=0x7f1ffc14a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.271+0000 7f2000ee3700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc15da70 0x7f1ffc14a0c0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f1ffc0cd0b0 tx=0x7f1ff4004af0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.271+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 shutdown_connections 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.271+0000 7f2000ee3700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1fe8066730 0x7f1fe8068bf0 secure :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f1ff0006fd0 tx=0x7f1ff00092e0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.271+0000 7f2000ee3700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1ffc15da70 0x7f1ffc14a0c0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.275+0000 7f2000ee3700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1ffc15d150 0x7f1ffc15f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.275+0000 7f2000ee3700 1 --2- 172.21.15.119:0/157346767 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1ffc06f480 0x7f1ffc153270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.275+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 >> 172.21.15.119:0/157346767 conn(0x7f1ffc0b5d90 msgr2=0x7f1ffc153ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:55.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.275+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 shutdown_connections 2024-09-13T16:35:55.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.275+0000 7f2000ee3700 1 -- 172.21.15.119:0/157346767 wait complete. 2024-09-13T16:35:55.280 INFO:teuthology.orchestra.run.smithi119.stderr:new fs with metadata pool 17 and data pool 18 2024-09-13T16:35:55.291 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:55.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f4650840700 1 Processor -- start 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f4650840700 1 -- start start 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f4650840700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f464c15cd60 0x7f464c06b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f4650840700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c0ceb60 con 0x7f464c0cf3c0 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f4650840700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f464c0ce230 con 0x7f464c0cfed0 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f4650840700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464c15d140 con 0x7f464c15cd60 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464b7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f464c15cd60 0x7f464c06b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464b7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f464c15cd60 0x7f464c06b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56216/0 (socket says 172.21.15.119:56216) 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464b7fe700 1 -- 172.21.15.119:0/2717040638 learned_addr learned my addr 172.21.15.119:0/2717040638 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464b7fe700 1 -- 172.21.15.119:0/2717040638 >> v1:172.21.15.145:6789/0 conn(0x7f464c0cfed0 legacy=0x7f464c0d2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464b7fe700 1 -- 172.21.15.119:0/2717040638 >> v1:172.21.15.119:6789/0 conn(0x7f464c0cf3c0 legacy=0x7f464c0cf7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464b7fe700 1 -- 172.21.15.119:0/2717040638 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464c0ce510 con 0x7f464c15cd60 2024-09-13T16:35:55.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464b7fe700 1 --2- 172.21.15.119:0/2717040638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c15cd60 0x7f464c06b730 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f463c009910 tx=0x7f463c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfaf2bb6cc04bdd5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464a7fc700 1 -- 172.21.15.119:0/2717040638 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c004230 con 0x7f464c15cd60 2024-09-13T16:35:55.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.391+0000 7f464a7fc700 1 -- 172.21.15.119:0/2717040638 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c004390 con 0x7f464c15cd60 2024-09-13T16:35:55.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f464a7fc700 1 -- 172.21.15.119:0/2717040638 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c009e60 con 0x7f464c15cd60 2024-09-13T16:35:55.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 -- 172.21.15.119:0/2717040638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c15cd60 msgr2=0x7f464c06b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 --2- 172.21.15.119:0/2717040638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c15cd60 0x7f464c06b730 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f463c009910 tx=0x7f463c009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:55.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 -- 172.21.15.119:0/2717040638 shutdown_connections 2024-09-13T16:35:55.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 --2- 172.21.15.119:0/2717040638 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c15cd60 0x7f464c06b730 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 -- 172.21.15.119:0/2717040638 >> 172.21.15.119:0/2717040638 conn(0x7f464c08c240 msgr2=0x7f464c09c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:55.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 -- 172.21.15.119:0/2717040638 shutdown_connections 2024-09-13T16:35:55.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 -- 172.21.15.119:0/2717040638 wait complete. 2024-09-13T16:35:55.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.395+0000 7f4650840700 1 Processor -- start 2024-09-13T16:35:55.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 -- start start 2024-09-13T16:35:55.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c0cf3c0 0x7f464c152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464c0cfed0 0x7f464c1f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f464c15cd60 0x7f464c15ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1f6c80 con 0x7f464c0cf3c0 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f464c0ccf60 con 0x7f464c0cfed0 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464c1f6000 con 0x7f464c15cd60 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c0cf3c0 0x7f464c152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c0cf3c0 0x7f464c152b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56220/0 (socket says 172.21.15.119:56220) 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 -- 172.21.15.119:0/3587516071 learned_addr learned my addr 172.21.15.119:0/3587516071 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464affd700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464c0cfed0 0x7f464c1f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464bfff700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f464c15cd60 0x7f464c15ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 -- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f464c15cd60 msgr2=0x7f464c15ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f464c15cd60 0x7f464c15ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 -- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464c0cfed0 msgr2=0x7f464c1f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464c0cfed0 0x7f464c1f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 -- 172.21.15.119:0/3587516071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f463c009570 con 0x7f464c0cf3c0 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464bfff700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f464c15cd60 0x7f464c15ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f464b7fe700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c0cf3c0 0x7f464c152b20 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f463c004000 tx=0x7f463c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4643fff700 1 -- 172.21.15.119:0/3587516071 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c0036c0 con 0x7f464c0cf3c0 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4643fff700 1 -- 172.21.15.119:0/3587516071 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f463c003840 con 0x7f464c0cf3c0 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464c1f62e0 con 0x7f464c0cf3c0 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4643fff700 1 -- 172.21.15.119:0/3587516071 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f463c003bf0 con 0x7f464c0cf3c0 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.399+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464c1f68a0 con 0x7f464c0cf3c0 2024-09-13T16:35:55.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.403+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464c0aff90 con 0x7f464c0cf3c0 2024-09-13T16:35:55.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.403+0000 7f4643fff700 1 -- 172.21.15.119:0/3587516071 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f463c003d50 con 0x7f464c0cf3c0 2024-09-13T16:35:55.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.403+0000 7f4643fff700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f462c066760 0x7f462c068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.403+0000 7f4643fff700 1 -- 172.21.15.119:0/3587516071 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f463c085ba0 con 0x7f464c0cf3c0 2024-09-13T16:35:55.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.403+0000 7f464affd700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f462c066760 0x7f462c068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.403+0000 7f464affd700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f462c066760 0x7f462c068c20 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f46340018a0 tx=0x7f4634009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.411+0000 7f4643fff700 1 -- 172.21.15.119:0/3587516071 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f463c052e60 con 0x7f464c0cf3c0 2024-09-13T16:35:55.588 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.587+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f464c0b1520 con 0x7f464c0cf3c0 2024-09-13T16:35:55.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.587+0000 7f4643fff700 1 -- 172.21.15.119:0/3587516071 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16857 (secure 0 0 0) 0x7f463c01f090 con 0x7f464c0cf3c0 2024-09-13T16:35:55.591 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:55.591 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":125,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:55.244089+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:55.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f462c066760 msgr2=0x7f462c068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f462c066760 0x7f462c068c20 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f46340018a0 tx=0x7f4634009040 comp rx=0 tx=0).stop 2024-09-13T16:35:55.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c0cf3c0 msgr2=0x7f464c152b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c0cf3c0 0x7f464c152b20 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f463c004000 tx=0x7f463c004770 comp rx=0 tx=0).stop 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 shutdown_connections 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f462c066760 0x7f462c068c20 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f464c15cd60 0x7f464c15ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f464c0cfed0 0x7f464c1f3280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 --2- 172.21.15.119:0/3587516071 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f464c0cf3c0 0x7f464c152b20 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 >> 172.21.15.119:0/3587516071 conn(0x7f464c08c240 msgr2=0x7f464c0b1800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 shutdown_connections 2024-09-13T16:35:55.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.591+0000 7f4650840700 1 -- 172.21.15.119:0/3587516071 wait complete. 2024-09-13T16:35:55.606 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:55.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3434b53700 1 Processor -- start 2024-09-13T16:35:55.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3434b53700 1 -- start start 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3434b53700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f34300b17f0 0x7f34300cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3434b53700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f34300b2200 0x7f34300b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3434b53700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34300705d0 con 0x7f34300cd8d0 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3434b53700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34300b2bb0 con 0x7f34300b2200 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3434b53700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f343006fca0 con 0x7f34300b17f0 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f34300b2200 0x7f34300b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3426ffd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f34300b17f0 0x7f34300cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3426ffd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f34300b17f0 0x7f34300cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:52788/0 (socket says 172.21.15.119:52788) 2024-09-13T16:35:55.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f3426ffd700 1 -- 172.21.15.119:0/2228017272 learned_addr learned my addr 172.21.15.119:0/2228017272 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:55.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342ffff700 1 -- 172.21.15.119:0/2228017272 >> v2:172.21.15.145:3300/0 conn(0x7f34300b17f0 msgr2=0x7f34300cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:55.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342ffff700 1 --2- 172.21.15.119:0/2228017272 >> v2:172.21.15.145:3300/0 conn(0x7f34300b17f0 0x7f34300cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342ffff700 1 -- 172.21.15.119:0/2228017272 >> v1:172.21.15.145:6789/0 conn(0x7f34300cd8d0 legacy=0x7f343006cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342ffff700 1 -- 172.21.15.119:0/2228017272 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f343006ff80 con 0x7f34300b2200 2024-09-13T16:35:55.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342effd700 1 -- 172.21.15.119:0/2228017272 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998804834 0 0) 0x7f34300705d0 con 0x7f34300cd8d0 2024-09-13T16:35:55.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342ffff700 1 --2- 172.21.15.119:0/2228017272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300b2670 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f34300ae2c0 tx=0x7f342800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84ebfffdb0390baa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342effd700 1 -- 172.21.15.119:0/2228017272 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f342800e070 con 0x7f34300b2200 2024-09-13T16:35:55.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342effd700 1 -- 172.21.15.119:0/2228017272 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f342800bb20 con 0x7f34300b2200 2024-09-13T16:35:55.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.715+0000 7f342effd700 1 -- 172.21.15.119:0/2228017272 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3428012430 con 0x7f34300b2200 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 -- 172.21.15.119:0/2228017272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 msgr2=0x7f34300b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 --2- 172.21.15.119:0/2228017272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300b2670 secure :-1 s=READY pgs=1792 cs=0 l=1 rev1=1 crypto rx=0x7f34300ae2c0 tx=0x7f342800b330 comp rx=0 tx=0).stop 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 -- 172.21.15.119:0/2228017272 shutdown_connections 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 --2- 172.21.15.119:0/2228017272 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300b2670 unknown :-1 s=CLOSED pgs=1792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 --2- 172.21.15.119:0/2228017272 >> v2:172.21.15.145:3300/0 conn(0x7f34300b17f0 0x7f34300cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 -- 172.21.15.119:0/2228017272 >> 172.21.15.119:0/2228017272 conn(0x7f343006a350 msgr2=0x7f343006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 -- 172.21.15.119:0/2228017272 shutdown_connections 2024-09-13T16:35:55.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 -- 172.21.15.119:0/2228017272 wait complete. 2024-09-13T16:35:55.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 Processor -- start 2024-09-13T16:35:55.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.719+0000 7f3434b53700 1 -- start start 2024-09-13T16:35:55.723 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34300b17f0 0x7f343014f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300af7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34300cd8d0 0x7f34300afd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f343014f970 con 0x7f34300b2200 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34300ad5c0 con 0x7f34300cd8d0 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34300ad000 con 0x7f34300b17f0 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300af7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300af7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56240/0 (socket says 172.21.15.119:56240) 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 -- 172.21.15.119:0/187227234 learned_addr learned my addr 172.21.15.119:0/187227234 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f342ffff700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34300cd8d0 0x7f34300afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f342f7fe700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34300b17f0 0x7f343014f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 -- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34300b17f0 msgr2=0x7f343014f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34300b17f0 0x7f343014f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 -- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34300cd8d0 msgr2=0x7f34300afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34300cd8d0 0x7f34300afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 -- 172.21.15.119:0/187227234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3428009a70 con 0x7f34300b2200 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3426ffd700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300af7e0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f341800e960 tx=0x7f341800ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f342cff9700 1 -- 172.21.15.119:0/187227234 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34180096c0 con 0x7f34300b2200 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f342cff9700 1 -- 172.21.15.119:0/187227234 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3418013680 con 0x7f34300b2200 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34300ad2e0 con 0x7f34300b2200 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f342cff9700 1 -- 172.21.15.119:0/187227234 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3418013a30 con 0x7f34300b2200 2024-09-13T16:35:55.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.723+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34300c3b50 con 0x7f34300b2200 2024-09-13T16:35:55.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.727+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34300c3d10 con 0x7f34300b2200 2024-09-13T16:35:55.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.727+0000 7f342cff9700 1 -- 172.21.15.119:0/187227234 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3418013b90 con 0x7f34300b2200 2024-09-13T16:35:55.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.727+0000 7f342cff9700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3410066760 0x7f3410068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:55.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.727+0000 7f342cff9700 1 -- 172.21.15.119:0/187227234 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f3418084180 con 0x7f34300b2200 2024-09-13T16:35:55.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.727+0000 7f342f7fe700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3410066760 0x7f3410068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:55.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.727+0000 7f342f7fe700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3410066760 0x7f3410068c20 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3420000f60 tx=0x7f3420009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:55.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.739+0000 7f342cff9700 1 -- 172.21.15.119:0/187227234 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3418051330 con 0x7f34300b2200 2024-09-13T16:35:55.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.911+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34300cdcb0 con 0x7f34300b2200 2024-09-13T16:35:55.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.915+0000 7f342cff9700 1 -- 172.21.15.119:0/187227234 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16857 (secure 0 0 0) 0x7f3418054950 con 0x7f34300b2200 2024-09-13T16:35:55.918 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:55.918 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":125,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:55.244089+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:55.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3410066760 msgr2=0x7f3410068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3410066760 0x7f3410068c20 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f3420000f60 tx=0x7f3420009040 comp rx=0 tx=0).stop 2024-09-13T16:35:55.920 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 msgr2=0x7f34300af7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:55.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300af7e0 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7f341800e960 tx=0x7f341800ec70 comp rx=0 tx=0).stop 2024-09-13T16:35:55.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 shutdown_connections 2024-09-13T16:35:55.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3410066760 0x7f3410068c20 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f34300cd8d0 0x7f34300afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.921 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f34300b2200 0x7f34300af7e0 unknown :-1 s=CLOSED pgs=1793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 --2- 172.21.15.119:0/187227234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f34300b17f0 0x7f343014f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:55.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 >> 172.21.15.119:0/187227234 conn(0x7f343006a350 msgr2=0x7f34301626e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:55.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 shutdown_connections 2024-09-13T16:35:55.922 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:55.919+0000 7f3434b53700 1 -- 172.21.15.119:0/187227234 wait complete. 2024-09-13T16:35:55.933 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-09-13T16:35:56.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd326547700 1 Processor -- start 2024-09-13T16:35:56.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd326547700 1 -- start start 2024-09-13T16:35:56.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd326547700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd32006c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:56.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd326547700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd3200cc010 0x7fd3200cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:56.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd326547700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd32006cb40 con 0x7fd3200ce000 2024-09-13T16:35:56.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd326547700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3200b1c50 con 0x7fd3200cc010 2024-09-13T16:35:56.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd326547700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32006ce20 con 0x7fd320144180 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd3200cc010 0x7fd3200cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd3200cc010 0x7fd3200cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56248/0 (socket says 172.21.15.119:56248) 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 -- 172.21.15.119:0/3207259159 learned_addr learned my addr 172.21.15.119:0/3207259159 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd325545700 1 --2- 172.21.15.119:0/3207259159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd32006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 -- 172.21.15.119:0/3207259159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 msgr2=0x7fd32006c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 --2- 172.21.15.119:0/3207259159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd32006c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 -- 172.21.15.119:0/3207259159 >> v1:172.21.15.145:6789/0 conn(0x7fd3200ce000 legacy=0x7fd32015e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 -- 172.21.15.119:0/3207259159 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32006d100 con 0x7fd3200cc010 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd324d44700 1 --2- 172.21.15.119:0/3207259159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd3200cdac0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fd308009e50 tx=0x7fd30800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b216d3df291b1ff server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd31ffff700 1 -- 172.21.15.119:0/3207259159 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800e070 con 0x7fd3200cc010 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd31ffff700 1 -- 172.21.15.119:0/3207259159 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30800b9c0 con 0x7fd3200cc010 2024-09-13T16:35:56.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.051+0000 7fd31ffff700 1 -- 172.21.15.119:0/3207259159 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd30800bd70 con 0x7fd3200cc010 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- 172.21.15.119:0/3207259159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 msgr2=0x7fd3200cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 --2- 172.21.15.119:0/3207259159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd3200cdac0 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7fd308009e50 tx=0x7fd30800b330 comp rx=0 tx=0).stop 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- 172.21.15.119:0/3207259159 shutdown_connections 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 --2- 172.21.15.119:0/3207259159 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd3200cdac0 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 --2- 172.21.15.119:0/3207259159 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd32006c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- 172.21.15.119:0/3207259159 >> 172.21.15.119:0/3207259159 conn(0x7fd3200b0980 msgr2=0x7fd3200b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- 172.21.15.119:0/3207259159 shutdown_connections 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- 172.21.15.119:0/3207259159 wait complete. 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 Processor -- start 2024-09-13T16:35:56.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- start start 2024-09-13T16:35:56.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd32015a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3200ce000 0x7fd3201408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd3201398f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32013bb80 con 0x7fd3200cc010 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd32013be60 con 0x7fd3200ce000 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd326547700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32013c140 con 0x7fd320144180 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd32015a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd32015a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56260/0 (socket says 172.21.15.119:56260) 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 -- 172.21.15.119:0/4149690762 learned_addr learned my addr 172.21.15.119:0/4149690762 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325d46700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd3201398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd324d44700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3200ce000 0x7fd3201408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:56.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 -- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 msgr2=0x7fd3201398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd3201398f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 -- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3200ce000 msgr2=0x7fd3201408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3200ce000 0x7fd3201408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.055+0000 7fd325545700 1 -- 172.21.15.119:0/4149690762 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd308009b30 con 0x7fd3200cc010 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd325545700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd32015a120 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fd31800d800 tx=0x7fd31800db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd31dffb700 1 -- 172.21.15.119:0/4149690762 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3180125f0 con 0x7fd3200cc010 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd31dffb700 1 -- 172.21.15.119:0/4149690762 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd318012750 con 0x7fd3200cc010 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd32013c420 con 0x7fd3200cc010 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd31dffb700 1 -- 172.21.15.119:0/4149690762 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd318012ac0 con 0x7fd3200cc010 2024-09-13T16:35:56.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd32015a660 con 0x7fd3200cc010 2024-09-13T16:35:56.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3200cdac0 con 0x7fd3200cc010 2024-09-13T16:35:56.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.059+0000 7fd31dffb700 1 -- 172.21.15.119:0/4149690762 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd318012c20 con 0x7fd3200cc010 2024-09-13T16:35:56.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.063+0000 7fd31dffb700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd300066730 0x7fd300068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:56.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.063+0000 7fd31dffb700 1 -- 172.21.15.119:0/4149690762 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd3180843a0 con 0x7fd3200cc010 2024-09-13T16:35:56.063 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.063+0000 7fd324d44700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd300066730 0x7fd300068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:56.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.063+0000 7fd324d44700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd300066730 0x7fd300068bf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd308009e50 tx=0x7fd308009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:56.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.067+0000 7fd31dffb700 1 -- 172.21.15.119:0/4149690762 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd318051530 con 0x7fd3200cc010 2024-09-13T16:35:56.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:56.275+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fd32015a8d0 con 0x7fd3200cc010 2024-09-13T16:35:57.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.263+0000 7fd31dffb700 1 -- 172.21.15.119:0/4149690762 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v91) v1 ==== 118+0+0 (secure 0 0 0) 0x7fd318054b50 con 0x7fd3200cc010 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd300066730 msgr2=0x7fd300068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd300066730 0x7fd300068bf0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fd308009e50 tx=0x7fd308009000 comp rx=0 tx=0).stop 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 msgr2=0x7fd32015a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd32015a120 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7fd31800d800 tx=0x7fd31800db10 comp rx=0 tx=0).stop 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 shutdown_connections 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd300066730 0x7fd300068bf0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd320144180 0x7fd3201398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd3200ce000 0x7fd3201408c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 --2- 172.21.15.119:0/4149690762 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd3200cc010 0x7fd32015a120 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 >> 172.21.15.119:0/4149690762 conn(0x7fd3200b0980 msgr2=0x7fd32015bfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:57.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 shutdown_connections 2024-09-13T16:35:57.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.267+0000 7fd326547700 1 -- 172.21.15.119:0/4149690762 wait complete. 2024-09-13T16:35:57.287 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-09-13T16:35:57.287 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-09-13T16:35:57.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.411+0000 7fc47cfa7700 1 Processor -- start 2024-09-13T16:35:57.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.411+0000 7fc47cfa7700 1 -- start start 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4780ce8c0 0x7fc4780d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc4780d1610 0x7fc4780b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4780d1a20 con 0x7fc47815cc00 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4780d0df0 con 0x7fc4780ce8c0 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4780cdb00 con 0x7fc4780d1610 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc476ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4780ce8c0 0x7fc4780d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc4777fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc47815cc00 0x7fc4780d06e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57464/0 (socket says 172.21.15.119:57464) 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc476ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4780ce8c0 0x7fc4780d10d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56266/0 (socket says 172.21.15.119:56266) 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc4777fe700 1 -- 172.21.15.119:0/2923443813 learned_addr learned my addr 172.21.15.119:0/2923443813 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc477fff700 1 --2- 172.21.15.119:0/2923443813 >> v2:172.21.15.145:3300/0 conn(0x7fc4780d1610 0x7fc4780b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:57.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc476ffd700 1 -- 172.21.15.119:0/2923443813 >> v2:172.21.15.145:3300/0 conn(0x7fc4780d1610 msgr2=0x7fc4780b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:57.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc476ffd700 1 --2- 172.21.15.119:0/2923443813 >> v2:172.21.15.145:3300/0 conn(0x7fc4780d1610 0x7fc4780b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc476ffd700 1 -- 172.21.15.119:0/2923443813 >> v1:172.21.15.119:6789/0 conn(0x7fc47815cc00 legacy=0x7fc4780d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:57.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc476ffd700 1 -- 172.21.15.119:0/2923443813 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4780cdde0 con 0x7fc4780ce8c0 2024-09-13T16:35:57.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc476ffd700 1 --2- 172.21.15.119:0/2923443813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4780ce8c0 0x7fc4780d10d0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc4780a0a20 tx=0x7fc468009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9528d62b03684df9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc4767fc700 1 -- 172.21.15.119:0/2923443813 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc468011830 con 0x7fc4780ce8c0 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc4767fc700 1 -- 172.21.15.119:0/2923443813 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc468011990 con 0x7fc4780ce8c0 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc4767fc700 1 -- 172.21.15.119:0/2923443813 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc468011d20 con 0x7fc4780ce8c0 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- 172.21.15.119:0/2923443813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4780ce8c0 msgr2=0x7fc4780d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/2923443813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4780ce8c0 0x7fc4780d10d0 secure :-1 s=READY pgs=1797 cs=0 l=1 rev1=1 crypto rx=0x7fc4780a0a20 tx=0x7fc468009b30 comp rx=0 tx=0).stop 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- 172.21.15.119:0/2923443813 shutdown_connections 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/2923443813 >> v2:172.21.15.145:3300/0 conn(0x7fc4780d1610 0x7fc4780b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/2923443813 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4780ce8c0 0x7fc4780d10d0 unknown :-1 s=CLOSED pgs=1797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- 172.21.15.119:0/2923443813 >> 172.21.15.119:0/2923443813 conn(0x7fc47808c080 msgr2=0x7fc4780d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:57.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- 172.21.15.119:0/2923443813 shutdown_connections 2024-09-13T16:35:57.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 -- 172.21.15.119:0/2923443813 wait complete. 2024-09-13T16:35:57.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.415+0000 7fc47cfa7700 1 Processor -- start 2024-09-13T16:35:57.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 -- start start 2024-09-13T16:35:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4780ce8c0 0x7fc4780b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4780d1610 0x7fc47815ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc47815cc00 0x7fc47806bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:57.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780b1b10 con 0x7fc47815cc00 2024-09-13T16:35:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780cb5b0 con 0x7fc4780d1610 2024-09-13T16:35:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4780cb260 con 0x7fc4780ce8c0 2024-09-13T16:35:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc47815cc00 0x7fc47806bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc47815cc00 0x7fc47806bb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56282/0 (socket says 172.21.15.119:56282) 2024-09-13T16:35:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 -- 172.21.15.119:0/4143103540 learned_addr learned my addr 172.21.15.119:0/4143103540 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc4777fe700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4780ce8c0 0x7fc4780b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:57.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc476ffd700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4780d1610 0x7fc47815ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:57.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 -- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4780ce8c0 msgr2=0x7fc4780b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:57.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4780ce8c0 0x7fc4780b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 -- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4780d1610 msgr2=0x7fc47815ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:57.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4780d1610 0x7fc47815ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:57.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 -- 172.21.15.119:0/4143103540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4680094b0 con 0x7fc47815cc00 2024-09-13T16:35:57.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc477fff700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc47815cc00 0x7fc47806bb50 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fc47000efc0 tx=0x7fc47000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:57.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc45ffff700 1 -- 172.21.15.119:0/4143103540 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc470009ea0 con 0x7fc47815cc00 2024-09-13T16:35:57.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc45ffff700 1 -- 172.21.15.119:0/4143103540 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc47000f040 con 0x7fc47815cc00 2024-09-13T16:35:57.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4780b0710 con 0x7fc47815cc00 2024-09-13T16:35:57.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc45ffff700 1 -- 172.21.15.119:0/4143103540 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc47000ced0 con 0x7fc47815cc00 2024-09-13T16:35:57.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.419+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4780b0cd0 con 0x7fc47815cc00 2024-09-13T16:35:57.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.423+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4780cb720 con 0x7fc47815cc00 2024-09-13T16:35:57.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.423+0000 7fc45ffff700 1 -- 172.21.15.119:0/4143103540 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc470014480 con 0x7fc47815cc00 2024-09-13T16:35:57.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.423+0000 7fc45ffff700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc464066730 0x7fc464068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:57.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.423+0000 7fc45ffff700 1 -- 172.21.15.119:0/4143103540 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc470084d70 con 0x7fc47815cc00 2024-09-13T16:35:57.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.423+0000 7fc4777fe700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc464066730 0x7fc464068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:57.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.423+0000 7fc4777fe700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc464066730 0x7fc464068bf0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005930 tx=0x7fc46c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:57.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.431+0000 7fc45ffff700 1 -- 172.21.15.119:0/4143103540 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc470052040 con 0x7fc47815cc00 2024-09-13T16:35:57.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:57.639+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc4780d0970 con 0x7fc47815cc00 2024-09-13T16:35:58.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.271+0000 7fc45ffff700 1 -- 172.21.15.119:0/4143103540 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v92) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc470018070 con 0x7fc47815cc00 2024-09-13T16:35:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.275+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc464066730 msgr2=0x7fc464068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.275+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc464066730 0x7fc464068bf0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fc46c005930 tx=0x7fc46c009040 comp rx=0 tx=0).stop 2024-09-13T16:35:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.275+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc47815cc00 msgr2=0x7fc47806bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.275+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc47815cc00 0x7fc47806bb50 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fc47000efc0 tx=0x7fc47000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 shutdown_connections 2024-09-13T16:35:58.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc464066730 0x7fc464068bf0 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc47815cc00 0x7fc47806bb50 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc4780d1610 0x7fc47815ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 --2- 172.21.15.119:0/4143103540 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc4780ce8c0 0x7fc4780b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 >> 172.21.15.119:0/4143103540 conn(0x7fc47808c080 msgr2=0x7fc4780b7740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 shutdown_connections 2024-09-13T16:35:58.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.279+0000 7fc47cfa7700 1 -- 172.21.15.119:0/4143103540 wait complete. 2024-09-13T16:35:58.295 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-09-13T16:35:58.296 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:35:58.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.411+0000 7f760759e700 1 Processor -- start 2024-09-13T16:35:58.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.411+0000 7f760759e700 1 -- start start 2024-09-13T16:35:58.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f760806dc40 0x7f76080b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080b2280 con 0x7f76081594f0 2024-09-13T16:35:58.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080cc890 con 0x7f76080b3630 2024-09-13T16:35:58.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76080b1970 con 0x7f760806dc40 2024-09-13T16:35:58.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f7605d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f760806dc40 0x7f76080b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760659c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f76081594f0 0x7f760806d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57468/0 (socket says 172.21.15.119:57468) 2024-09-13T16:35:58.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f7605d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f760806dc40 0x7f76080b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56286/0 (socket says 172.21.15.119:56286) 2024-09-13T16:35:58.416 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760659c700 1 -- 172.21.15.119:0/563096892 learned_addr learned my addr 172.21.15.119:0/563096892 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:58.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f7605d9b700 1 -- 172.21.15.119:0/563096892 >> v1:172.21.15.145:6789/0 conn(0x7f76080b3630 legacy=0x7f760806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f7605d9b700 1 -- 172.21.15.119:0/563096892 >> v1:172.21.15.119:6789/0 conn(0x7f76081594f0 legacy=0x7f760806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f7605d9b700 1 -- 172.21.15.119:0/563096892 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080b1c30 con 0x7f760806dc40 2024-09-13T16:35:58.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f7605d9b700 1 --2- 172.21.15.119:0/563096892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760806dc40 0x7f76080b30f0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009850 tx=0x7f75f8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa2ea0bf0de2538 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:58.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760559a700 1 -- 172.21.15.119:0/563096892 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8011830 con 0x7f760806dc40 2024-09-13T16:35:58.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760559a700 1 -- 172.21.15.119:0/563096892 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75f8011990 con 0x7f760806dc40 2024-09-13T16:35:58.417 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760559a700 1 -- 172.21.15.119:0/563096892 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75f8011d20 con 0x7f760806dc40 2024-09-13T16:35:58.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- 172.21.15.119:0/563096892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760806dc40 msgr2=0x7f76080b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 --2- 172.21.15.119:0/563096892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760806dc40 0x7f76080b30f0 secure :-1 s=READY pgs=1800 cs=0 l=1 rev1=1 crypto rx=0x7f75f8009850 tx=0x7f75f8009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:58.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- 172.21.15.119:0/563096892 shutdown_connections 2024-09-13T16:35:58.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 --2- 172.21.15.119:0/563096892 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f760806dc40 0x7f76080b30f0 unknown :-1 s=CLOSED pgs=1800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.418 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- 172.21.15.119:0/563096892 >> 172.21.15.119:0/563096892 conn(0x7f760806a350 msgr2=0x7f760806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:58.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- 172.21.15.119:0/563096892 shutdown_connections 2024-09-13T16:35:58.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.415+0000 7f760759e700 1 -- 172.21.15.119:0/563096892 wait complete. 2024-09-13T16:35:58.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 Processor -- start 2024-09-13T16:35:58.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 -- start start 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f760806dc40 0x7f760814fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76080b3630 0x7f76080ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76081594f0 0x7f76080ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608158940 con 0x7f76081594f0 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7608158d60 con 0x7f76080b3630 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7608157ad0 con 0x7f760806dc40 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76081594f0 0x7f76080ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7605d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76080b3630 0x7f76080ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f760806dc40 0x7f760814fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76081594f0 0x7f76080ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56298/0 (socket says 172.21.15.119:56298) 2024-09-13T16:35:58.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7605d9b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76080b3630 0x7f76080ac8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:52830/0 (socket says 172.21.15.119:52830) 2024-09-13T16:35:58.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 -- 172.21.15.119:0/1516625818 learned_addr learned my addr 172.21.15.119:0/1516625818 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 -- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f760806dc40 msgr2=0x7f760814fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f760806dc40 0x7f760814fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 -- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76080b3630 msgr2=0x7f76080ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76080b3630 0x7f76080ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 -- 172.21.15.119:0/1516625818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f80094b0 con 0x7f76081594f0 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f7606d9d700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76081594f0 0x7f76080ace00 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f760000cc80 tx=0x7f760000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f75f6ffd700 1 -- 172.21.15.119:0/1516625818 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600003c20 con 0x7f76081594f0 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f75f6ffd700 1 -- 172.21.15.119:0/1516625818 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7600003d80 con 0x7f76081594f0 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f75f6ffd700 1 -- 172.21.15.119:0/1516625818 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760000aaa0 con 0x7f76081594f0 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7608157d00 con 0x7f76081594f0 2024-09-13T16:35:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.419+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7608158280 con 0x7f76081594f0 2024-09-13T16:35:58.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.423+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7608158a80 con 0x7f76081594f0 2024-09-13T16:35:58.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.423+0000 7f75f6ffd700 1 -- 172.21.15.119:0/1516625818 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f760000ac00 con 0x7f76081594f0 2024-09-13T16:35:58.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.423+0000 7f75f6ffd700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75e8066730 0x7f75e8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.423+0000 7f75f6ffd700 1 -- 172.21.15.119:0/1516625818 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f76000842f0 con 0x7f76081594f0 2024-09-13T16:35:58.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.423+0000 7f760659c700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75e8066730 0x7f75e8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.427+0000 7f760659c700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75e8066730 0x7f75e8068bf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f75fc005930 tx=0x7f75fc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:58.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.427+0000 7f75f6ffd700 1 -- 172.21.15.119:0/1516625818 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7608158a80 con 0x7f76081594f0 2024-09-13T16:35:58.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.635+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7608158a80 con 0x7f76081594f0 2024-09-13T16:35:58.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f75f6ffd700 1 -- 172.21.15.119:0/1516625818 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4739 (secure 0 0 0) 0x7f7608158a80 con 0x7f76081594f0 2024-09-13T16:35:58.639 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:58.639 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":92,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:35:58.262367+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9546":{"gid":9546,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6835/3901706437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3901706437},{"type":"v1","addr":"172.21.15.145:6835","nonce":3901706437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:35:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75e8066730 msgr2=0x7f75e8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75e8066730 0x7f75e8068bf0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f75fc005930 tx=0x7f75fc009040 comp rx=0 tx=0).stop 2024-09-13T16:35:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76081594f0 msgr2=0x7f76080ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76081594f0 0x7f76080ace00 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f760000cc80 tx=0x7f760000cf90 comp rx=0 tx=0).stop 2024-09-13T16:35:58.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 shutdown_connections 2024-09-13T16:35:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f75e8066730 0x7f75e8068bf0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f76081594f0 0x7f76080ace00 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76080b3630 0x7f76080ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 --2- 172.21.15.119:0/1516625818 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f760806dc40 0x7f760814fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 >> 172.21.15.119:0/1516625818 conn(0x7f760806a350 msgr2=0x7f76080cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 shutdown_connections 2024-09-13T16:35:58.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.639+0000 7f760759e700 1 -- 172.21.15.119:0/1516625818 wait complete. 2024-09-13T16:35:58.644 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 92 2024-09-13T16:35:58.655 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:35:58.716 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:35:58.719 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:35:58.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f511cb94700 1 Processor -- start 2024-09-13T16:35:58.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f511cb94700 1 -- start start 2024-09-13T16:35:58.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f511cb94700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 0x7f5118160f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f511cb94700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180b4880 con 0x7f511806a9a0 2024-09-13T16:35:58.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f511cb94700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51180b3c70 con 0x7f511806a5c0 2024-09-13T16:35:58.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f511cb94700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f511815dec0 con 0x7f51180717b0 2024-09-13T16:35:58.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f5117fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f511806a9a0 0x7f511815d7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57480/0 (socket says 172.21.15.119:57480) 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f5117fff700 1 -- 172.21.15.119:0/4113401377 learned_addr learned my addr 172.21.15.119:0/4113401377 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f5116ffd700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 0x7f5118160f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3279022327 0 0) 0x7f51180b4880 con 0x7f511806a9a0 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51000035d0 con 0x7f511806a9a0 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3047140055 0 0) 0x7f51000035d0 con 0x7f511806a9a0 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 msgr2=0x7f5118160f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 0x7f5118160f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 >> v1:172.21.15.145:6789/0 conn(0x7f511806a5c0 legacy=0x7f511806cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51180b3f50 con 0x7f511806a9a0 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f51100019f0 con 0x7f511806a9a0 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 >> v1:172.21.15.119:6789/0 conn(0x7f511806a9a0 legacy=0x7f511815d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.768 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5100003a50 0x7f5100003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 0x7f5100004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5100004db0 0x7f5100005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511815dec0 con 0x7f5100004db0 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510000bae0 con 0x7f5100003a50 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f510000bca0 con 0x7f51180717b0 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f51177fe700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5100003a50 0x7f5100003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f5116ffd700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 0x7f5100004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.763+0000 7f5117fff700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5100004db0 0x7f5100005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f5116ffd700 1 -- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5100003a50 msgr2=0x7f5100003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f5116ffd700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5100003a50 0x7f5100003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f5116ffd700 1 -- 172.21.15.119:0/4113401377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5100004db0 msgr2=0x7f5100005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:58.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f5116ffd700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5100004db0 0x7f5100005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f5116ffd700 1 -- 172.21.15.119:0/4113401377 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000035d0 con 0x7f51180717b0 2024-09-13T16:35:58.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f51177fe700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5100003a50 0x7f5100003e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:35:58.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f5116ffd700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 0x7f5100004870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f51180cf8e0 tx=0x7f51000058a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85008cf1ba4aa647 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:58.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c00e6c0 con 0x7f51180717b0 2024-09-13T16:35:58.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f510c00e820 con 0x7f51180717b0 2024-09-13T16:35:58.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f51167fc700 1 -- 172.21.15.119:0/4113401377 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f510c00eb90 con 0x7f51180717b0 2024-09-13T16:35:58.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 -- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 msgr2=0x7f5100004870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 --2- 172.21.15.119:0/4113401377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f51180717b0 0x7f5100004870 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f51180cf8e0 tx=0x7f51000058a0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f5117fff700 1 -- 172.21.15.119:0/4113401377 reap_dead start 2024-09-13T16:35:58.773 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 -- 172.21.15.119:0/4113401377 shutdown_connections 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 -- 172.21.15.119:0/4113401377 >> 172.21.15.119:0/4113401377 conn(0x7f5118070cc0 msgr2=0x7f51180710a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 -- 172.21.15.119:0/4113401377 shutdown_connections 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 -- 172.21.15.119:0/4113401377 wait complete. 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 Processor -- start 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.767+0000 7f511cb94700 1 -- start start 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f511814d9f0 0x7f5118159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f511814ddd0 0x7f51181f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5118151330 0x7f511814f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51181600d0 con 0x7f511814d9f0 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5118151b10 con 0x7f511814ddd0 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5118151df0 con 0x7f5118151330 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f511814d9f0 0x7f5118159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f511814d9f0 0x7f5118159e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56320/0 (socket says 172.21.15.119:56320) 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 -- 172.21.15.119:0/1638867296 learned_addr learned my addr 172.21.15.119:0/1638867296 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f5116ffd700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f511814ddd0 0x7f51181f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f5117fff700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5118151330 0x7f511814f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 -- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5118151330 msgr2=0x7f511814f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5118151330 0x7f511814f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 -- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f511814ddd0 msgr2=0x7f51181f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f511814ddd0 0x7f51181f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 -- 172.21.15.119:0/1638867296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51181f2cd0 con 0x7f511814d9f0 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f5116ffd700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f511814ddd0 0x7f51181f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f5117fff700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5118151330 0x7f511814f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f51177fe700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f511814d9f0 0x7f5118159e80 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f51080078c0 tx=0x7f510800fbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f50fffff700 1 -- 172.21.15.119:0/1638867296 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080052e0 con 0x7f511814d9f0 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f50fffff700 1 -- 172.21.15.119:0/1638867296 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5108005440 con 0x7f511814d9f0 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51181f2fb0 con 0x7f511814d9f0 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f50fffff700 1 -- 172.21.15.119:0/1638867296 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51080038e0 con 0x7f511814d9f0 2024-09-13T16:35:58.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.771+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51181517e0 con 0x7f511814d9f0 2024-09-13T16:35:58.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.775+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f511806cdb0 con 0x7f511814d9f0 2024-09-13T16:35:58.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.775+0000 7f50fffff700 1 -- 172.21.15.119:0/1638867296 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5108003a40 con 0x7f511814d9f0 2024-09-13T16:35:58.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.775+0000 7f50fffff700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5104066760 0x7f5104068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:58.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.775+0000 7f50fffff700 1 -- 172.21.15.119:0/1638867296 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f51080841a0 con 0x7f511814d9f0 2024-09-13T16:35:58.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.775+0000 7f5116ffd700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5104066760 0x7f5104068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:58.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.775+0000 7f5116ffd700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5104066760 0x7f5104068c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5118160900 tx=0x7f510c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:58.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.783+0000 7f50fffff700 1 -- 172.21.15.119:0/1638867296 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f51080513d0 con 0x7f511814d9f0 2024-09-13T16:35:58.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.954+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f511806e6d0 con 0x7f511814d9f0 2024-09-13T16:35:58.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f50fffff700 1 -- 172.21.15.119:0/1638867296 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16844 (secure 0 0 0) 0x7f51080549f0 con 0x7f511814d9f0 2024-09-13T16:35:58.959 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:58.959 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":126,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:56.248923+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:35:58.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5104066760 msgr2=0x7f5104068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5104066760 0x7f5104068c20 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f5118160900 tx=0x7f510c008030 comp rx=0 tx=0).stop 2024-09-13T16:35:58.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f511814d9f0 msgr2=0x7f5118159e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:58.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f511814d9f0 0x7f5118159e80 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f51080078c0 tx=0x7f510800fbf0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 shutdown_connections 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5104066760 0x7f5104068c20 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5118151330 0x7f511814f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f511814ddd0 0x7f51181f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 --2- 172.21.15.119:0/1638867296 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f511814d9f0 0x7f5118159e80 unknown :-1 s=CLOSED pgs=1803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 >> 172.21.15.119:0/1638867296 conn(0x7f5118070cc0 msgr2=0x7f51180af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 shutdown_connections 2024-09-13T16:35:58.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:58.958+0000 7f511cb94700 1 -- 172.21.15.119:0/1638867296 wait complete. 2024-09-13T16:35:58.973 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-09-13T16:35:58.973 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:35:59.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc68675b700 1 Processor -- start 2024-09-13T16:35:59.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc68675b700 1 -- start start 2024-09-13T16:35:59.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc68675b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc680158b30 0x7fc680071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc68675b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc680158f10 0x7fc680071570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc68675b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc68006de00 con 0x7fc68006bb90 2024-09-13T16:35:59.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc68675b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc68006db20 con 0x7fc680158b30 2024-09-13T16:35:59.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc68675b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6800b1920 con 0x7fc680158f10 2024-09-13T16:35:59.085 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685759700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc680158b30 0x7fc680071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685f5a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc68006bb90 0x7fc68006ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57492/0 (socket says 172.21.15.119:57492) 2024-09-13T16:35:59.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685f5a700 1 -- 172.21.15.119:0/2155814317 learned_addr learned my addr 172.21.15.119:0/2155814317 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:59.086 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc684f58700 1 --2- 172.21.15.119:0/2155814317 >> v2:172.21.15.145:3300/0 conn(0x7fc680158f10 0x7fc680071570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685759700 1 -- 172.21.15.119:0/2155814317 >> v2:172.21.15.145:3300/0 conn(0x7fc680158f10 msgr2=0x7fc680071570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc677fff700 1 -- 172.21.15.119:0/2155814317 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403515991 0 0) 0x7fc68006de00 con 0x7fc68006bb90 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685759700 1 --2- 172.21.15.119:0/2155814317 >> v2:172.21.15.145:3300/0 conn(0x7fc680158f10 0x7fc680071570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685759700 1 -- 172.21.15.119:0/2155814317 >> v1:172.21.15.119:6789/0 conn(0x7fc68006bb90 legacy=0x7fc68006ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685759700 1 -- 172.21.15.119:0/2155814317 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6800b1c00 con 0x7fc680158b30 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc685759700 1 --2- 172.21.15.119:0/2155814317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc680158b30 0x7fc680071030 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fc680070dd0 tx=0x7fc670009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938bf2b2c3a823a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc677fff700 1 -- 172.21.15.119:0/2155814317 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670011830 con 0x7fc680158b30 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc677fff700 1 -- 172.21.15.119:0/2155814317 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc670011990 con 0x7fc680158b30 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.082+0000 7fc677fff700 1 -- 172.21.15.119:0/2155814317 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc670011d20 con 0x7fc680158b30 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- 172.21.15.119:0/2155814317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc680158b30 msgr2=0x7fc680071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 --2- 172.21.15.119:0/2155814317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc680158b30 0x7fc680071030 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7fc680070dd0 tx=0x7fc670009b30 comp rx=0 tx=0).stop 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- 172.21.15.119:0/2155814317 shutdown_connections 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 --2- 172.21.15.119:0/2155814317 >> v2:172.21.15.145:3300/0 conn(0x7fc680158f10 0x7fc680071570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 --2- 172.21.15.119:0/2155814317 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc680158b30 0x7fc680071030 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- 172.21.15.119:0/2155814317 >> 172.21.15.119:0/2155814317 conn(0x7fc68006a350 msgr2=0x7fc68006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- 172.21.15.119:0/2155814317 shutdown_connections 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- 172.21.15.119:0/2155814317 wait complete. 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 Processor -- start 2024-09-13T16:35:59.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- start start 2024-09-13T16:35:59.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc68006bb90 0x7fc6801efcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc680158b30 0x7fc6801568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.092 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc680158f10 0x7fc680156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6801f0200 con 0x7fc68006bb90 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc680152840 con 0x7fc680158b30 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc68675b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc680140ee0 con 0x7fc680158f10 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc685759700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc68006bb90 0x7fc6801efcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc685759700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc68006bb90 0x7fc6801efcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56344/0 (socket says 172.21.15.119:56344) 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc685759700 1 -- 172.21.15.119:0/2744037330 learned_addr learned my addr 172.21.15.119:0/2744037330 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc685f5a700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc680158f10 0x7fc680156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc684f58700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc680158b30 0x7fc6801568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc685759700 1 -- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc680158f10 msgr2=0x7fc680156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc685759700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc680158f10 0x7fc680156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.086+0000 7fc685759700 1 -- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc680158b30 msgr2=0x7fc6801568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc685759700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc680158b30 0x7fc6801568e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc685759700 1 -- 172.21.15.119:0/2744037330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6700094b0 con 0x7fc68006bb90 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc685f5a700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc680158f10 0x7fc680156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc684f58700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc680158b30 0x7fc6801568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc685759700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc68006bb90 0x7fc6801efcc0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc670009fd0 tx=0x7fc670004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc675ffb700 1 -- 172.21.15.119:0/2744037330 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700045b0 con 0x7fc68006bb90 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc675ffb700 1 -- 172.21.15.119:0/2744037330 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc67001f070 con 0x7fc68006bb90 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6801411c0 con 0x7fc68006bb90 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc675ffb700 1 -- 172.21.15.119:0/2744037330 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6700038e0 con 0x7fc68006bb90 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc680141780 con 0x7fc68006bb90 2024-09-13T16:35:59.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc680071030 con 0x7fc68006bb90 2024-09-13T16:35:59.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc675ffb700 1 -- 172.21.15.119:0/2744037330 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc670021030 con 0x7fc68006bb90 2024-09-13T16:35:59.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc675ffb700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc664066730 0x7fc664068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc675ffb700 1 -- 172.21.15.119:0/2744037330 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fc670085630 con 0x7fc68006bb90 2024-09-13T16:35:59.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.090+0000 7fc684f58700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc664066730 0x7fc664068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.095 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.094+0000 7fc684f58700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc664066730 0x7fc664068bf0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc6801574d0 tx=0x7fc67c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.098+0000 7fc675ffb700 1 -- 172.21.15.119:0/2744037330 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc670052a10 con 0x7fc68006bb90 2024-09-13T16:35:59.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.266+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc68006bfe0 con 0x7fc664066730 2024-09-13T16:35:59.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.282+0000 7fc675ffb700 1 -- 172.21.15.119:0/2744037330 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288169 (secure 0 0 0) 0x7fc67c007d20 con 0x7fc664066730 2024-09-13T16:35:59.291 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:59.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc664066730 msgr2=0x7fc664068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc664066730 0x7fc664068bf0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc6801574d0 tx=0x7fc67c00b040 comp rx=0 tx=0).stop 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc68006bb90 msgr2=0x7fc6801efcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc68006bb90 0x7fc6801efcc0 secure :-1 s=READY pgs=1806 cs=0 l=1 rev1=1 crypto rx=0x7fc670009fd0 tx=0x7fc670004770 comp rx=0 tx=0).stop 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 shutdown_connections 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc664066730 0x7fc664068bf0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc680158f10 0x7fc680156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc680158b30 0x7fc6801568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 --2- 172.21.15.119:0/2744037330 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc68006bb90 0x7fc6801efcc0 unknown :-1 s=CLOSED pgs=1806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 >> 172.21.15.119:0/2744037330 conn(0x7fc68006a350 msgr2=0x7fc68006cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 shutdown_connections 2024-09-13T16:35:59.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.290+0000 7fc68675b700 1 -- 172.21.15.119:0/2744037330 wait complete. 2024-09-13T16:35:59.295 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:35:59.307 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-13T16:35:57.650585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262188,"kb_used_data":13220,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754712532,"statfs":{"total":773094113280,"available":772825632768,"internally_reserved":0,"allocated":13537280,"data_stored":6874627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001341"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284254+0000","last_change":"2024-09-13T16:35:55.279603+0000","last_active":"2024-09-13T16:35:55.284254+0000","last_peered":"2024-09-13T16:35:55.284254+0000","last_clean":"2024-09-13T16:35:55.284254+0000","last_became_active":"2024-09-13T16:35:55.279310+0000","last_became_peered":"2024-09-13T16:35:55.279310+0000","last_unstale":"2024-09-13T16:35:55.284254+0000","last_undegraded":"2024-09-13T16:35:55.284254+0000","last_fullsized":"2024-09-13T16:35:55.284254+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:37.136914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280009+0000","last_change":"2024-09-13T16:35:55.277134+0000","last_active":"2024-09-13T16:35:55.280009+0000","last_peered":"2024-09-13T16:35:55.280009+0000","last_clean":"2024-09-13T16:35:55.280009+0000","last_became_active":"2024-09-13T16:35:55.276859+0000","last_became_peered":"2024-09-13T16:35:55.276859+0000","last_unstale":"2024-09-13T16:35:55.280009+0000","last_undegraded":"2024-09-13T16:35:55.280009+0000","last_fullsized":"2024-09-13T16:35:55.280009+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:19.852842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280131+0000","last_change":"2024-09-13T16:35:55.278080+0000","last_active":"2024-09-13T16:35:55.280131+0000","last_peered":"2024-09-13T16:35:55.280131+0000","last_clean":"2024-09-13T16:35:55.280131+0000","last_became_active":"2024-09-13T16:35:55.277853+0000","last_became_peered":"2024-09-13T16:35:55.277853+0000","last_unstale":"2024-09-13T16:35:55.280131+0000","last_undegraded":"2024-09-13T16:35:55.280131+0000","last_fullsized":"2024-09-13T16:35:55.280131+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:49.561531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279876+0000","last_change":"2024-09-13T16:35:55.276626+0000","last_active":"2024-09-13T16:35:55.279876+0000","last_peered":"2024-09-13T16:35:55.279876+0000","last_clean":"2024-09-13T16:35:55.279876+0000","last_became_active":"2024-09-13T16:35:55.276311+0000","last_became_peered":"2024-09-13T16:35:55.276311+0000","last_unstale":"2024-09-13T16:35:55.279876+0000","last_undegraded":"2024-09-13T16:35:55.279876+0000","last_fullsized":"2024-09-13T16:35:55.279876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:06.782215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284207+0000","last_change":"2024-09-13T16:35:55.279767+0000","last_active":"2024-09-13T16:35:55.284207+0000","last_peered":"2024-09-13T16:35:55.284207+0000","last_clean":"2024-09-13T16:35:55.284207+0000","last_became_active":"2024-09-13T16:35:55.279415+0000","last_became_peered":"2024-09-13T16:35:55.279415+0000","last_unstale":"2024-09-13T16:35:55.284207+0000","last_undegraded":"2024-09-13T16:35:55.284207+0000","last_fullsized":"2024-09-13T16:35:55.284207+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:53.391694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275710+0000","last_change":"2024-09-13T16:35:55.273486+0000","last_active":"2024-09-13T16:35:55.275710+0000","last_peered":"2024-09-13T16:35:55.275710+0000","last_clean":"2024-09-13T16:35:55.275710+0000","last_became_active":"2024-09-13T16:35:55.272868+0000","last_became_peered":"2024-09-13T16:35:55.272868+0000","last_unstale":"2024-09-13T16:35:55.275710+0000","last_undegraded":"2024-09-13T16:35:55.275710+0000","last_fullsized":"2024-09-13T16:35:55.275710+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:08.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278856+0000","last_change":"2024-09-13T16:35:55.276620+0000","last_active":"2024-09-13T16:35:55.278856+0000","last_peered":"2024-09-13T16:35:55.278856+0000","last_clean":"2024-09-13T16:35:55.278856+0000","last_became_active":"2024-09-13T16:35:55.276089+0000","last_became_peered":"2024-09-13T16:35:55.276089+0000","last_unstale":"2024-09-13T16:35:55.278856+0000","last_undegraded":"2024-09-13T16:35:55.278856+0000","last_fullsized":"2024-09-13T16:35:55.278856+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:31.583929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282997+0000","last_change":"2024-09-13T16:35:55.282402+0000","last_active":"2024-09-13T16:35:55.282997+0000","last_peered":"2024-09-13T16:35:55.282997+0000","last_clean":"2024-09-13T16:35:55.282997+0000","last_became_active":"2024-09-13T16:35:55.282151+0000","last_became_peered":"2024-09-13T16:35:55.282151+0000","last_unstale":"2024-09-13T16:35:55.282997+0000","last_undegraded":"2024-09-13T16:35:55.282997+0000","last_fullsized":"2024-09-13T16:35:55.282997+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:58.964322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276504+0000","last_change":"2024-09-13T16:35:55.271390+0000","last_active":"2024-09-13T16:35:55.276504+0000","last_peered":"2024-09-13T16:35:55.276504+0000","last_clean":"2024-09-13T16:35:55.276504+0000","last_became_active":"2024-09-13T16:35:55.271078+0000","last_became_peered":"2024-09-13T16:35:55.271078+0000","last_unstale":"2024-09-13T16:35:55.276504+0000","last_undegraded":"2024-09-13T16:35:55.276504+0000","last_fullsized":"2024-09-13T16:35:55.276504+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:55.505021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280370+0000","last_change":"2024-09-13T16:35:55.276030+0000","last_active":"2024-09-13T16:35:55.280370+0000","last_peered":"2024-09-13T16:35:55.280370+0000","last_clean":"2024-09-13T16:35:55.280370+0000","last_became_active":"2024-09-13T16:35:55.275608+0000","last_became_peered":"2024-09-13T16:35:55.275608+0000","last_unstale":"2024-09-13T16:35:55.280370+0000","last_undegraded":"2024-09-13T16:35:55.280370+0000","last_fullsized":"2024-09-13T16:35:55.280370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:11.770066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282249+0000","last_change":"2024-09-13T16:35:55.273486+0000","last_active":"2024-09-13T16:35:55.282249+0000","last_peered":"2024-09-13T16:35:55.282249+0000","last_clean":"2024-09-13T16:35:55.282249+0000","last_became_active":"2024-09-13T16:35:55.273165+0000","last_became_peered":"2024-09-13T16:35:55.273165+0000","last_unstale":"2024-09-13T16:35:55.282249+0000","last_undegraded":"2024-09-13T16:35:55.282249+0000","last_fullsized":"2024-09-13T16:35:55.282249+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:55.505021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281569+0000","last_change":"2024-09-13T16:35:55.280916+0000","last_active":"2024-09-13T16:35:55.281569+0000","last_peered":"2024-09-13T16:35:55.281569+0000","last_clean":"2024-09-13T16:35:55.281569+0000","last_became_active":"2024-09-13T16:35:55.280672+0000","last_became_peered":"2024-09-13T16:35:55.280672+0000","last_unstale":"2024-09-13T16:35:55.281569+0000","last_undegraded":"2024-09-13T16:35:55.281569+0000","last_fullsized":"2024-09-13T16:35:55.281569+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:58.174716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278191+0000","last_change":"2024-09-13T16:35:55.270541+0000","last_active":"2024-09-13T16:35:55.278191+0000","last_peered":"2024-09-13T16:35:55.278191+0000","last_clean":"2024-09-13T16:35:55.278191+0000","last_became_active":"2024-09-13T16:35:55.269759+0000","last_became_peered":"2024-09-13T16:35:55.269759+0000","last_unstale":"2024-09-13T16:35:55.278191+0000","last_undegraded":"2024-09-13T16:35:55.278191+0000","last_fullsized":"2024-09-13T16:35:55.278191+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:58.150363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275509+0000","last_change":"2024-09-13T16:35:55.271638+0000","last_active":"2024-09-13T16:35:55.275509+0000","last_peered":"2024-09-13T16:35:55.275509+0000","last_clean":"2024-09-13T16:35:55.275509+0000","last_became_active":"2024-09-13T16:35:55.271410+0000","last_became_peered":"2024-09-13T16:35:55.271410+0000","last_unstale":"2024-09-13T16:35:55.275509+0000","last_undegraded":"2024-09-13T16:35:55.275509+0000","last_fullsized":"2024-09-13T16:35:55.275509+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:06.468091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282279+0000","last_change":"2024-09-13T16:35:55.273535+0000","last_active":"2024-09-13T16:35:55.282279+0000","last_peered":"2024-09-13T16:35:55.282279+0000","last_clean":"2024-09-13T16:35:55.282279+0000","last_became_active":"2024-09-13T16:35:55.273256+0000","last_became_peered":"2024-09-13T16:35:55.273256+0000","last_unstale":"2024-09-13T16:35:55.282279+0000","last_undegraded":"2024-09-13T16:35:55.282279+0000","last_fullsized":"2024-09-13T16:35:55.282279+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:48.360160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280333+0000","last_change":"2024-09-13T16:35:55.279169+0000","last_active":"2024-09-13T16:35:55.280333+0000","last_peered":"2024-09-13T16:35:55.280333+0000","last_clean":"2024-09-13T16:35:55.280333+0000","last_became_active":"2024-09-13T16:35:55.278953+0000","last_became_peered":"2024-09-13T16:35:55.278953+0000","last_unstale":"2024-09-13T16:35:55.280333+0000","last_undegraded":"2024-09-13T16:35:55.280333+0000","last_fullsized":"2024-09-13T16:35:55.280333+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:55.507098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278085+0000","last_change":"2024-09-13T16:35:55.269537+0000","last_active":"2024-09-13T16:35:55.278085+0000","last_peered":"2024-09-13T16:35:55.278085+0000","last_clean":"2024-09-13T16:35:55.278085+0000","last_became_active":"2024-09-13T16:35:55.269296+0000","last_became_peered":"2024-09-13T16:35:55.269296+0000","last_unstale":"2024-09-13T16:35:55.278085+0000","last_undegraded":"2024-09-13T16:35:55.278085+0000","last_fullsized":"2024-09-13T16:35:55.278085+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:06.292463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.277026+0000","last_change":"2024-09-13T16:35:55.273483+0000","last_active":"2024-09-13T16:35:55.277026+0000","last_peered":"2024-09-13T16:35:55.277026+0000","last_clean":"2024-09-13T16:35:55.277026+0000","last_became_active":"2024-09-13T16:35:55.273241+0000","last_became_peered":"2024-09-13T16:35:55.273241+0000","last_unstale":"2024-09-13T16:35:55.277026+0000","last_undegraded":"2024-09-13T16:35:55.277026+0000","last_fullsized":"2024-09-13T16:35:55.277026+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:24.023139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.286694+0000","last_change":"2024-09-13T16:35:55.283864+0000","last_active":"2024-09-13T16:35:55.286694+0000","last_peered":"2024-09-13T16:35:55.286694+0000","last_clean":"2024-09-13T16:35:55.286694+0000","last_became_active":"2024-09-13T16:35:55.281152+0000","last_became_peered":"2024-09-13T16:35:55.281152+0000","last_unstale":"2024-09-13T16:35:55.286694+0000","last_undegraded":"2024-09-13T16:35:55.286694+0000","last_fullsized":"2024-09-13T16:35:55.286694+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:31:43.636829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280275+0000","last_change":"2024-09-13T16:35:55.276770+0000","last_active":"2024-09-13T16:35:55.280275+0000","last_peered":"2024-09-13T16:35:55.280275+0000","last_clean":"2024-09-13T16:35:55.280275+0000","last_became_active":"2024-09-13T16:35:55.276498+0000","last_became_peered":"2024-09-13T16:35:55.276498+0000","last_unstale":"2024-09-13T16:35:55.280275+0000","last_undegraded":"2024-09-13T16:35:55.280275+0000","last_fullsized":"2024-09-13T16:35:55.280275+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:45.212600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279925+0000","last_change":"2024-09-13T16:35:55.271943+0000","last_active":"2024-09-13T16:35:55.279925+0000","last_peered":"2024-09-13T16:35:55.279925+0000","last_clean":"2024-09-13T16:35:55.279925+0000","last_became_active":"2024-09-13T16:35:55.271447+0000","last_became_peered":"2024-09-13T16:35:55.271447+0000","last_unstale":"2024-09-13T16:35:55.279925+0000","last_undegraded":"2024-09-13T16:35:55.279925+0000","last_fullsized":"2024-09-13T16:35:55.279925+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:08.476350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284413+0000","last_change":"2024-09-13T16:35:55.279946+0000","last_active":"2024-09-13T16:35:55.284413+0000","last_peered":"2024-09-13T16:35:55.284413+0000","last_clean":"2024-09-13T16:35:55.284413+0000","last_became_active":"2024-09-13T16:35:55.279420+0000","last_became_peered":"2024-09-13T16:35:55.279420+0000","last_unstale":"2024-09-13T16:35:55.284413+0000","last_undegraded":"2024-09-13T16:35:55.284413+0000","last_fullsized":"2024-09-13T16:35:55.284413+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:47.356307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275001+0000","last_change":"2024-09-13T16:35:55.270766+0000","last_active":"2024-09-13T16:35:55.275001+0000","last_peered":"2024-09-13T16:35:55.275001+0000","last_clean":"2024-09-13T16:35:55.275001+0000","last_became_active":"2024-09-13T16:35:55.264869+0000","last_became_peered":"2024-09-13T16:35:55.264869+0000","last_unstale":"2024-09-13T16:35:55.275001+0000","last_undegraded":"2024-09-13T16:35:55.275001+0000","last_fullsized":"2024-09-13T16:35:55.275001+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:11.766616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282194+0000","last_change":"2024-09-13T16:35:55.277130+0000","last_active":"2024-09-13T16:35:55.282194+0000","last_peered":"2024-09-13T16:35:55.282194+0000","last_clean":"2024-09-13T16:35:55.282194+0000","last_became_active":"2024-09-13T16:35:55.276503+0000","last_became_peered":"2024-09-13T16:35:55.276503+0000","last_unstale":"2024-09-13T16:35:55.282194+0000","last_undegraded":"2024-09-13T16:35:55.282194+0000","last_fullsized":"2024-09-13T16:35:55.282194+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:07.717972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284132+0000","last_change":"2024-09-13T16:35:55.281497+0000","last_active":"2024-09-13T16:35:55.284132+0000","last_peered":"2024-09-13T16:35:55.284132+0000","last_clean":"2024-09-13T16:35:55.284132+0000","last_became_active":"2024-09-13T16:35:55.281213+0000","last_became_peered":"2024-09-13T16:35:55.281213+0000","last_unstale":"2024-09-13T16:35:55.284132+0000","last_undegraded":"2024-09-13T16:35:55.284132+0000","last_fullsized":"2024-09-13T16:35:55.284132+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:39.179995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280412+0000","last_change":"2024-09-13T16:35:55.275779+0000","last_active":"2024-09-13T16:35:55.280412+0000","last_peered":"2024-09-13T16:35:55.280412+0000","last_clean":"2024-09-13T16:35:55.280412+0000","last_became_active":"2024-09-13T16:35:55.275563+0000","last_became_peered":"2024-09-13T16:35:55.275563+0000","last_unstale":"2024-09-13T16:35:55.280412+0000","last_undegraded":"2024-09-13T16:35:55.280412+0000","last_fullsized":"2024-09-13T16:35:55.280412+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:50.333030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.285121+0000","last_change":"2024-09-13T16:35:55.283609+0000","last_active":"2024-09-13T16:35:55.285121+0000","last_peered":"2024-09-13T16:35:55.285121+0000","last_clean":"2024-09-13T16:35:55.285121+0000","last_became_active":"2024-09-13T16:35:55.283230+0000","last_became_peered":"2024-09-13T16:35:55.283230+0000","last_unstale":"2024-09-13T16:35:55.285121+0000","last_undegraded":"2024-09-13T16:35:55.285121+0000","last_fullsized":"2024-09-13T16:35:55.285121+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:58.150363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275112+0000","last_change":"2024-09-13T16:35:55.273034+0000","last_active":"2024-09-13T16:35:55.275112+0000","last_peered":"2024-09-13T16:35:55.275112+0000","last_clean":"2024-09-13T16:35:55.275112+0000","last_became_active":"2024-09-13T16:35:55.272674+0000","last_became_peered":"2024-09-13T16:35:55.272674+0000","last_unstale":"2024-09-13T16:35:55.275112+0000","last_undegraded":"2024-09-13T16:35:55.275112+0000","last_fullsized":"2024-09-13T16:35:55.275112+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:25.410612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278488+0000","last_change":"2024-09-13T16:35:55.274390+0000","last_active":"2024-09-13T16:35:55.278488+0000","last_peered":"2024-09-13T16:35:55.278488+0000","last_clean":"2024-09-13T16:35:55.278488+0000","last_became_active":"2024-09-13T16:35:55.274153+0000","last_became_peered":"2024-09-13T16:35:55.274153+0000","last_unstale":"2024-09-13T16:35:55.278488+0000","last_undegraded":"2024-09-13T16:35:55.278488+0000","last_fullsized":"2024-09-13T16:35:55.278488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:09.187095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282245+0000","last_change":"2024-09-13T16:35:55.281148+0000","last_active":"2024-09-13T16:35:55.282245+0000","last_peered":"2024-09-13T16:35:55.282245+0000","last_clean":"2024-09-13T16:35:55.282245+0000","last_became_active":"2024-09-13T16:35:55.280818+0000","last_became_peered":"2024-09-13T16:35:55.280818+0000","last_unstale":"2024-09-13T16:35:55.282245+0000","last_undegraded":"2024-09-13T16:35:55.282245+0000","last_fullsized":"2024-09-13T16:35:55.282245+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:12.278199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279559+0000","last_change":"2024-09-13T16:35:55.270018+0000","last_active":"2024-09-13T16:35:55.279559+0000","last_peered":"2024-09-13T16:35:55.279559+0000","last_clean":"2024-09-13T16:35:55.279559+0000","last_became_active":"2024-09-13T16:35:55.269738+0000","last_became_peered":"2024-09-13T16:35:55.269738+0000","last_unstale":"2024-09-13T16:35:55.279559+0000","last_undegraded":"2024-09-13T16:35:55.279559+0000","last_fullsized":"2024-09-13T16:35:55.279559+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:00.636075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282183+0000","last_change":"2024-09-13T16:35:55.276944+0000","last_active":"2024-09-13T16:35:55.282183+0000","last_peered":"2024-09-13T16:35:55.282183+0000","last_clean":"2024-09-13T16:35:55.282183+0000","last_became_active":"2024-09-13T16:35:55.276043+0000","last_became_peered":"2024-09-13T16:35:55.276043+0000","last_unstale":"2024-09-13T16:35:55.282183+0000","last_undegraded":"2024-09-13T16:35:55.282183+0000","last_fullsized":"2024-09-13T16:35:55.282183+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:37.460203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280444+0000","last_change":"2024-09-13T16:35:55.274402+0000","last_active":"2024-09-13T16:35:55.280444+0000","last_peered":"2024-09-13T16:35:55.280444+0000","last_clean":"2024-09-13T16:35:55.280444+0000","last_became_active":"2024-09-13T16:35:55.274155+0000","last_became_peered":"2024-09-13T16:35:55.274155+0000","last_unstale":"2024-09-13T16:35:55.280444+0000","last_undegraded":"2024-09-13T16:35:55.280444+0000","last_fullsized":"2024-09-13T16:35:55.280444+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:32.967289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"125'4","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284799+0000","last_change":"2024-09-13T16:35:54.889433+0000","last_active":"2024-09-13T16:35:55.284799+0000","last_peered":"2024-09-13T16:35:55.284799+0000","last_clean":"2024-09-13T16:35:55.284799+0000","last_became_active":"2024-09-13T16:35:54.889176+0000","last_became_peered":"2024-09-13T16:35:54.889176+0000","last_unstale":"2024-09-13T16:35:55.284799+0000","last_undegraded":"2024-09-13T16:35:55.284799+0000","last_fullsized":"2024-09-13T16:35:55.284799+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:00.104258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275062+0000","last_change":"2024-09-13T16:35:55.272130+0000","last_active":"2024-09-13T16:35:55.275062+0000","last_peered":"2024-09-13T16:35:55.275062+0000","last_clean":"2024-09-13T16:35:55.275062+0000","last_became_active":"2024-09-13T16:35:55.271881+0000","last_became_peered":"2024-09-13T16:35:55.271881+0000","last_unstale":"2024-09-13T16:35:55.275062+0000","last_undegraded":"2024-09-13T16:35:55.275062+0000","last_fullsized":"2024-09-13T16:35:55.275062+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:35.658877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.271023+0000","last_change":"2024-09-13T16:35:54.888195+0000","last_active":"2024-09-13T16:35:55.271023+0000","last_peered":"2024-09-13T16:35:55.271023+0000","last_clean":"2024-09-13T16:35:55.271023+0000","last_became_active":"2024-09-13T16:35:54.887939+0000","last_became_peered":"2024-09-13T16:35:54.887939+0000","last_unstale":"2024-09-13T16:35:55.271023+0000","last_undegraded":"2024-09-13T16:35:55.271023+0000","last_fullsized":"2024-09-13T16:35:55.271023+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:43.039403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282095+0000","last_change":"2024-09-13T16:35:55.273233+0000","last_active":"2024-09-13T16:35:55.282095+0000","last_peered":"2024-09-13T16:35:55.282095+0000","last_clean":"2024-09-13T16:35:55.282095+0000","last_became_active":"2024-09-13T16:35:55.272891+0000","last_became_peered":"2024-09-13T16:35:55.272891+0000","last_unstale":"2024-09-13T16:35:55.282095+0000","last_undegraded":"2024-09-13T16:35:55.282095+0000","last_fullsized":"2024-09-13T16:35:55.282095+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:52.424969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275160+0000","last_change":"2024-09-13T16:35:54.891686+0000","last_active":"2024-09-13T16:35:55.275160+0000","last_peered":"2024-09-13T16:35:55.275160+0000","last_clean":"2024-09-13T16:35:55.275160+0000","last_became_active":"2024-09-13T16:35:54.891407+0000","last_became_peered":"2024-09-13T16:35:54.891407+0000","last_unstale":"2024-09-13T16:35:55.275160+0000","last_undegraded":"2024-09-13T16:35:55.275160+0000","last_fullsized":"2024-09-13T16:35:55.275160+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:00.453766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279313+0000","last_change":"2024-09-13T16:35:55.272704+0000","last_active":"2024-09-13T16:35:55.279313+0000","last_peered":"2024-09-13T16:35:55.279313+0000","last_clean":"2024-09-13T16:35:55.279313+0000","last_became_active":"2024-09-13T16:35:55.272451+0000","last_became_peered":"2024-09-13T16:35:55.272451+0000","last_unstale":"2024-09-13T16:35:55.279313+0000","last_undegraded":"2024-09-13T16:35:55.279313+0000","last_fullsized":"2024-09-13T16:35:55.279313+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:44.312036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.271605+0000","last_change":"2024-09-13T16:35:54.888401+0000","last_active":"2024-09-13T16:35:55.271605+0000","last_peered":"2024-09-13T16:35:55.271605+0000","last_clean":"2024-09-13T16:35:55.271605+0000","last_became_active":"2024-09-13T16:35:54.888092+0000","last_became_peered":"2024-09-13T16:35:54.888092+0000","last_unstale":"2024-09-13T16:35:55.271605+0000","last_undegraded":"2024-09-13T16:35:55.271605+0000","last_fullsized":"2024-09-13T16:35:55.271605+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:13.571296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280251+0000","last_change":"2024-09-13T16:35:55.276918+0000","last_active":"2024-09-13T16:35:55.280251+0000","last_peered":"2024-09-13T16:35:55.280251+0000","last_clean":"2024-09-13T16:35:55.280251+0000","last_became_active":"2024-09-13T16:35:55.276687+0000","last_became_peered":"2024-09-13T16:35:55.276687+0000","last_unstale":"2024-09-13T16:35:55.280251+0000","last_undegraded":"2024-09-13T16:35:55.280251+0000","last_fullsized":"2024-09-13T16:35:55.280251+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:01.463116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279857+0000","last_change":"2024-09-13T16:35:54.887074+0000","last_active":"2024-09-13T16:35:55.279857+0000","last_peered":"2024-09-13T16:35:55.279857+0000","last_clean":"2024-09-13T16:35:55.279857+0000","last_became_active":"2024-09-13T16:35:54.886669+0000","last_became_peered":"2024-09-13T16:35:54.886669+0000","last_unstale":"2024-09-13T16:35:55.279857+0000","last_undegraded":"2024-09-13T16:35:55.279857+0000","last_fullsized":"2024-09-13T16:35:55.279857+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:34.524636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284218+0000","last_change":"2024-09-13T16:35:55.279187+0000","last_active":"2024-09-13T16:35:55.284218+0000","last_peered":"2024-09-13T16:35:55.284218+0000","last_clean":"2024-09-13T16:35:55.284218+0000","last_became_active":"2024-09-13T16:35:55.278844+0000","last_became_peered":"2024-09-13T16:35:55.278844+0000","last_unstale":"2024-09-13T16:35:55.284218+0000","last_undegraded":"2024-09-13T16:35:55.284218+0000","last_fullsized":"2024-09-13T16:35:55.284218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:11.766616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.274527+0000","last_change":"2024-09-13T16:35:54.891182+0000","last_active":"2024-09-13T16:35:55.274527+0000","last_peered":"2024-09-13T16:35:55.274527+0000","last_clean":"2024-09-13T16:35:55.274527+0000","last_became_active":"2024-09-13T16:35:54.890865+0000","last_became_peered":"2024-09-13T16:35:54.890865+0000","last_unstale":"2024-09-13T16:35:55.274527+0000","last_undegraded":"2024-09-13T16:35:55.274527+0000","last_fullsized":"2024-09-13T16:35:55.274527+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:42.872900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278604+0000","last_change":"2024-09-13T16:35:55.275612+0000","last_active":"2024-09-13T16:35:55.278604+0000","last_peered":"2024-09-13T16:35:55.278604+0000","last_clean":"2024-09-13T16:35:55.278604+0000","last_became_active":"2024-09-13T16:35:55.275344+0000","last_became_peered":"2024-09-13T16:35:55.275344+0000","last_unstale":"2024-09-13T16:35:55.278604+0000","last_undegraded":"2024-09-13T16:35:55.278604+0000","last_fullsized":"2024-09-13T16:35:55.278604+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:08.105328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.254216+0000","last_change":"2024-09-13T16:35:54.885936+0000","last_active":"2024-09-13T16:35:55.254216+0000","last_peered":"2024-09-13T16:35:55.254216+0000","last_clean":"2024-09-13T16:35:55.254216+0000","last_became_active":"2024-09-13T16:35:54.885422+0000","last_became_peered":"2024-09-13T16:35:54.885422+0000","last_unstale":"2024-09-13T16:35:55.254216+0000","last_undegraded":"2024-09-13T16:35:55.254216+0000","last_fullsized":"2024-09-13T16:35:55.254216+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:59.841074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278230+0000","last_change":"2024-09-13T16:35:55.270876+0000","last_active":"2024-09-13T16:35:55.278230+0000","last_peered":"2024-09-13T16:35:55.278230+0000","last_clean":"2024-09-13T16:35:55.278230+0000","last_became_active":"2024-09-13T16:35:55.270324+0000","last_became_peered":"2024-09-13T16:35:55.270324+0000","last_unstale":"2024-09-13T16:35:55.278230+0000","last_undegraded":"2024-09-13T16:35:55.278230+0000","last_fullsized":"2024-09-13T16:35:55.278230+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:02:39.323231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.266012+0000","last_change":"2024-09-13T16:35:54.884585+0000","last_active":"2024-09-13T16:35:55.266012+0000","last_peered":"2024-09-13T16:35:55.266012+0000","last_clean":"2024-09-13T16:35:55.266012+0000","last_became_active":"2024-09-13T16:35:54.884265+0000","last_became_peered":"2024-09-13T16:35:54.884265+0000","last_unstale":"2024-09-13T16:35:55.266012+0000","last_undegraded":"2024-09-13T16:35:55.266012+0000","last_fullsized":"2024-09-13T16:35:55.266012+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:57.954972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284085+0000","last_change":"2024-09-13T16:35:55.279229+0000","last_active":"2024-09-13T16:35:55.284085+0000","last_peered":"2024-09-13T16:35:55.284085+0000","last_clean":"2024-09-13T16:35:55.284085+0000","last_became_active":"2024-09-13T16:35:55.275486+0000","last_became_peered":"2024-09-13T16:35:55.275486+0000","last_unstale":"2024-09-13T16:35:55.284085+0000","last_undegraded":"2024-09-13T16:35:55.284085+0000","last_fullsized":"2024-09-13T16:35:55.284085+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:03.079664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.267483+0000","last_change":"2024-09-13T16:35:54.892587+0000","last_active":"2024-09-13T16:35:55.267483+0000","last_peered":"2024-09-13T16:35:55.267483+0000","last_clean":"2024-09-13T16:35:55.267483+0000","last_became_active":"2024-09-13T16:35:54.892278+0000","last_became_peered":"2024-09-13T16:35:54.892278+0000","last_unstale":"2024-09-13T16:35:55.267483+0000","last_undegraded":"2024-09-13T16:35:55.267483+0000","last_fullsized":"2024-09-13T16:35:55.267483+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:54.111959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.268876+0000","last_change":"2024-09-13T16:35:55.263965+0000","last_active":"2024-09-13T16:35:55.268876+0000","last_peered":"2024-09-13T16:35:55.268876+0000","last_clean":"2024-09-13T16:35:55.268876+0000","last_became_active":"2024-09-13T16:35:55.263689+0000","last_became_peered":"2024-09-13T16:35:55.263689+0000","last_unstale":"2024-09-13T16:35:55.268876+0000","last_undegraded":"2024-09-13T16:35:55.268876+0000","last_fullsized":"2024-09-13T16:35:55.268876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:39.844424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.255803+0000","last_change":"2024-09-13T16:35:54.884085+0000","last_active":"2024-09-13T16:35:55.255803+0000","last_peered":"2024-09-13T16:35:55.255803+0000","last_clean":"2024-09-13T16:35:55.255803+0000","last_became_active":"2024-09-13T16:35:54.883730+0000","last_became_peered":"2024-09-13T16:35:54.883730+0000","last_unstale":"2024-09-13T16:35:55.255803+0000","last_undegraded":"2024-09-13T16:35:55.255803+0000","last_fullsized":"2024-09-13T16:35:55.255803+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:23:26.541011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280118+0000","last_change":"2024-09-13T16:35:55.277876+0000","last_active":"2024-09-13T16:35:55.280118+0000","last_peered":"2024-09-13T16:35:55.280118+0000","last_clean":"2024-09-13T16:35:55.280118+0000","last_became_active":"2024-09-13T16:35:55.277620+0000","last_became_peered":"2024-09-13T16:35:55.277620+0000","last_unstale":"2024-09-13T16:35:55.280118+0000","last_undegraded":"2024-09-13T16:35:55.280118+0000","last_fullsized":"2024-09-13T16:35:55.280118+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:12.923144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"125'3","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276917+0000","last_change":"2024-09-13T16:35:54.886486+0000","last_active":"2024-09-13T16:35:55.276917+0000","last_peered":"2024-09-13T16:35:55.276917+0000","last_clean":"2024-09-13T16:35:55.276917+0000","last_became_active":"2024-09-13T16:35:54.886191+0000","last_became_peered":"2024-09-13T16:35:54.886191+0000","last_unstale":"2024-09-13T16:35:55.276917+0000","last_undegraded":"2024-09-13T16:35:55.276917+0000","last_fullsized":"2024-09-13T16:35:55.276917+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:50.494802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281614+0000","last_change":"2024-09-13T16:35:55.270334+0000","last_active":"2024-09-13T16:35:55.281614+0000","last_peered":"2024-09-13T16:35:55.281614+0000","last_clean":"2024-09-13T16:35:55.281614+0000","last_became_active":"2024-09-13T16:35:55.269786+0000","last_became_peered":"2024-09-13T16:35:55.269786+0000","last_unstale":"2024-09-13T16:35:55.281614+0000","last_undegraded":"2024-09-13T16:35:55.281614+0000","last_fullsized":"2024-09-13T16:35:55.281614+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:33.253752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"125'6","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.287478+0000","last_change":"2024-09-13T16:35:54.889719+0000","last_active":"2024-09-13T16:35:55.287478+0000","last_peered":"2024-09-13T16:35:55.287478+0000","last_clean":"2024-09-13T16:35:55.287478+0000","last_became_active":"2024-09-13T16:35:54.889460+0000","last_became_peered":"2024-09-13T16:35:54.889460+0000","last_unstale":"2024-09-13T16:35:55.287478+0000","last_undegraded":"2024-09-13T16:35:55.287478+0000","last_fullsized":"2024-09-13T16:35:55.287478+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:17.919296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284954+0000","last_change":"2024-09-13T16:35:55.280602+0000","last_active":"2024-09-13T16:35:55.284954+0000","last_peered":"2024-09-13T16:35:55.284954+0000","last_clean":"2024-09-13T16:35:55.284954+0000","last_became_active":"2024-09-13T16:35:55.280267+0000","last_became_peered":"2024-09-13T16:35:55.280267+0000","last_unstale":"2024-09-13T16:35:55.284954+0000","last_undegraded":"2024-09-13T16:35:55.284954+0000","last_fullsized":"2024-09-13T16:35:55.284954+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:52.746769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276182+0000","last_change":"2024-09-13T16:35:54.887642+0000","last_active":"2024-09-13T16:35:55.276182+0000","last_peered":"2024-09-13T16:35:55.276182+0000","last_clean":"2024-09-13T16:35:55.276182+0000","last_became_active":"2024-09-13T16:35:54.886949+0000","last_became_peered":"2024-09-13T16:35:54.886949+0000","last_unstale":"2024-09-13T16:35:55.276182+0000","last_undegraded":"2024-09-13T16:35:55.276182+0000","last_fullsized":"2024-09-13T16:35:55.276182+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:15.061384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278043+0000","last_change":"2024-09-13T16:35:55.272281+0000","last_active":"2024-09-13T16:35:55.278043+0000","last_peered":"2024-09-13T16:35:55.278043+0000","last_clean":"2024-09-13T16:35:55.278043+0000","last_became_active":"2024-09-13T16:35:55.272007+0000","last_became_peered":"2024-09-13T16:35:55.272007+0000","last_unstale":"2024-09-13T16:35:55.278043+0000","last_undegraded":"2024-09-13T16:35:55.278043+0000","last_fullsized":"2024-09-13T16:35:55.278043+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:08.309452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.264841+0000","last_change":"2024-09-13T16:35:54.888765+0000","last_active":"2024-09-13T16:35:55.264841+0000","last_peered":"2024-09-13T16:35:55.264841+0000","last_clean":"2024-09-13T16:35:55.264841+0000","last_became_active":"2024-09-13T16:35:54.888497+0000","last_became_peered":"2024-09-13T16:35:54.888497+0000","last_unstale":"2024-09-13T16:35:55.264841+0000","last_undegraded":"2024-09-13T16:35:55.264841+0000","last_fullsized":"2024-09-13T16:35:55.264841+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:01.009326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281556+0000","last_change":"2024-09-13T16:35:55.281036+0000","last_active":"2024-09-13T16:35:55.281556+0000","last_peered":"2024-09-13T16:35:55.281556+0000","last_clean":"2024-09-13T16:35:55.281556+0000","last_became_active":"2024-09-13T16:35:55.280804+0000","last_became_peered":"2024-09-13T16:35:55.280804+0000","last_unstale":"2024-09-13T16:35:55.281556+0000","last_undegraded":"2024-09-13T16:35:55.281556+0000","last_fullsized":"2024-09-13T16:35:55.281556+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:23:27.550361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.261131+0000","last_change":"2024-09-13T16:35:54.890278+0000","last_active":"2024-09-13T16:35:55.261131+0000","last_peered":"2024-09-13T16:35:55.261131+0000","last_clean":"2024-09-13T16:35:55.261131+0000","last_became_active":"2024-09-13T16:35:54.889876+0000","last_became_peered":"2024-09-13T16:35:54.889876+0000","last_unstale":"2024-09-13T16:35:55.261131+0000","last_undegraded":"2024-09-13T16:35:55.261131+0000","last_fullsized":"2024-09-13T16:35:55.261131+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:12.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282151+0000","last_change":"2024-09-13T16:35:55.273223+0000","last_active":"2024-09-13T16:35:55.282151+0000","last_peered":"2024-09-13T16:35:55.282151+0000","last_clean":"2024-09-13T16:35:55.282151+0000","last_became_active":"2024-09-13T16:35:55.272802+0000","last_became_peered":"2024-09-13T16:35:55.272802+0000","last_unstale":"2024-09-13T16:35:55.282151+0000","last_undegraded":"2024-09-13T16:35:55.282151+0000","last_fullsized":"2024-09-13T16:35:55.282151+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:37.143833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.273910+0000","last_change":"2024-09-13T16:35:54.890074+0000","last_active":"2024-09-13T16:35:55.273910+0000","last_peered":"2024-09-13T16:35:55.273910+0000","last_clean":"2024-09-13T16:35:55.273910+0000","last_became_active":"2024-09-13T16:35:54.889832+0000","last_became_peered":"2024-09-13T16:35:54.889832+0000","last_unstale":"2024-09-13T16:35:55.273910+0000","last_undegraded":"2024-09-13T16:35:55.273910+0000","last_fullsized":"2024-09-13T16:35:55.273910+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:54.111959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282322+0000","last_change":"2024-09-13T16:35:55.273977+0000","last_active":"2024-09-13T16:35:55.282322+0000","last_peered":"2024-09-13T16:35:55.282322+0000","last_clean":"2024-09-13T16:35:55.282322+0000","last_became_active":"2024-09-13T16:35:55.273737+0000","last_became_peered":"2024-09-13T16:35:55.273737+0000","last_unstale":"2024-09-13T16:35:55.282322+0000","last_undegraded":"2024-09-13T16:35:55.282322+0000","last_fullsized":"2024-09-13T16:35:55.282322+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:50.586289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276400+0000","last_change":"2024-09-13T16:35:54.884544+0000","last_active":"2024-09-13T16:35:55.276400+0000","last_peered":"2024-09-13T16:35:55.276400+0000","last_clean":"2024-09-13T16:35:55.276400+0000","last_became_active":"2024-09-13T16:35:54.884221+0000","last_became_peered":"2024-09-13T16:35:54.884221+0000","last_unstale":"2024-09-13T16:35:55.276400+0000","last_undegraded":"2024-09-13T16:35:55.276400+0000","last_fullsized":"2024-09-13T16:35:55.276400+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:21.497053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278250+0000","last_change":"2024-09-13T16:35:55.276103+0000","last_active":"2024-09-13T16:35:55.278250+0000","last_peered":"2024-09-13T16:35:55.278250+0000","last_clean":"2024-09-13T16:35:55.278250+0000","last_became_active":"2024-09-13T16:35:55.275862+0000","last_became_peered":"2024-09-13T16:35:55.275862+0000","last_unstale":"2024-09-13T16:35:55.278250+0000","last_undegraded":"2024-09-13T16:35:55.278250+0000","last_fullsized":"2024-09-13T16:35:55.278250+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:55.377221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.273800+0000","last_change":"2024-09-13T16:35:54.888841+0000","last_active":"2024-09-13T16:35:55.273800+0000","last_peered":"2024-09-13T16:35:55.273800+0000","last_clean":"2024-09-13T16:35:55.273800+0000","last_became_active":"2024-09-13T16:35:54.888567+0000","last_became_peered":"2024-09-13T16:35:54.888567+0000","last_unstale":"2024-09-13T16:35:55.273800+0000","last_undegraded":"2024-09-13T16:35:55.273800+0000","last_fullsized":"2024-09-13T16:35:55.273800+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:51.467281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282275+0000","last_change":"2024-09-13T16:35:55.277459+0000","last_active":"2024-09-13T16:35:55.282275+0000","last_peered":"2024-09-13T16:35:55.282275+0000","last_clean":"2024-09-13T16:35:55.282275+0000","last_became_active":"2024-09-13T16:35:55.276362+0000","last_became_peered":"2024-09-13T16:35:55.276362+0000","last_unstale":"2024-09-13T16:35:55.282275+0000","last_undegraded":"2024-09-13T16:35:55.282275+0000","last_fullsized":"2024-09-13T16:35:55.282275+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:43.278864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.258193+0000","last_change":"2024-09-13T16:35:54.890899+0000","last_active":"2024-09-13T16:35:55.258193+0000","last_peered":"2024-09-13T16:35:55.258193+0000","last_clean":"2024-09-13T16:35:55.258193+0000","last_became_active":"2024-09-13T16:35:54.890626+0000","last_became_peered":"2024-09-13T16:35:54.890626+0000","last_unstale":"2024-09-13T16:35:55.258193+0000","last_undegraded":"2024-09-13T16:35:55.258193+0000","last_fullsized":"2024-09-13T16:35:55.258193+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:51.563617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282901+0000","last_change":"2024-09-13T16:35:55.282369+0000","last_active":"2024-09-13T16:35:55.282901+0000","last_peered":"2024-09-13T16:35:55.282901+0000","last_clean":"2024-09-13T16:35:55.282901+0000","last_became_active":"2024-09-13T16:35:55.282129+0000","last_became_peered":"2024-09-13T16:35:55.282129+0000","last_unstale":"2024-09-13T16:35:55.282901+0000","last_undegraded":"2024-09-13T16:35:55.282901+0000","last_fullsized":"2024-09-13T16:35:55.282901+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:23.167482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276074+0000","last_change":"2024-09-13T16:35:54.887365+0000","last_active":"2024-09-13T16:35:55.276074+0000","last_peered":"2024-09-13T16:35:55.276074+0000","last_clean":"2024-09-13T16:35:55.276074+0000","last_became_active":"2024-09-13T16:35:54.886927+0000","last_became_peered":"2024-09-13T16:35:54.886927+0000","last_unstale":"2024-09-13T16:35:55.276074+0000","last_undegraded":"2024-09-13T16:35:55.276074+0000","last_fullsized":"2024-09-13T16:35:55.276074+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:05.458741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.273434+0000","last_change":"2024-09-13T16:35:55.268501+0000","last_active":"2024-09-13T16:35:55.273434+0000","last_peered":"2024-09-13T16:35:55.273434+0000","last_clean":"2024-09-13T16:35:55.273434+0000","last_became_active":"2024-09-13T16:35:55.268261+0000","last_became_peered":"2024-09-13T16:35:55.268261+0000","last_unstale":"2024-09-13T16:35:55.273434+0000","last_undegraded":"2024-09-13T16:35:55.273434+0000","last_fullsized":"2024-09-13T16:35:55.273434+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:11.454742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.257972+0000","last_change":"2024-09-13T16:35:54.889084+0000","last_active":"2024-09-13T16:35:55.257972+0000","last_peered":"2024-09-13T16:35:55.257972+0000","last_clean":"2024-09-13T16:35:55.257972+0000","last_became_active":"2024-09-13T16:35:54.888765+0000","last_became_peered":"2024-09-13T16:35:54.888765+0000","last_unstale":"2024-09-13T16:35:55.257972+0000","last_undegraded":"2024-09-13T16:35:55.257972+0000","last_fullsized":"2024-09-13T16:35:55.257972+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:12.359450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280100+0000","last_change":"2024-09-13T16:35:55.278232+0000","last_active":"2024-09-13T16:35:55.280100+0000","last_peered":"2024-09-13T16:35:55.280100+0000","last_clean":"2024-09-13T16:35:55.280100+0000","last_became_active":"2024-09-13T16:35:55.277525+0000","last_became_peered":"2024-09-13T16:35:55.277525+0000","last_unstale":"2024-09-13T16:35:55.280100+0000","last_undegraded":"2024-09-13T16:35:55.280100+0000","last_fullsized":"2024-09-13T16:35:55.280100+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:01.605412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.261408+0000","last_change":"2024-09-13T16:35:54.890110+0000","last_active":"2024-09-13T16:35:55.261408+0000","last_peered":"2024-09-13T16:35:55.261408+0000","last_clean":"2024-09-13T16:35:55.261408+0000","last_became_active":"2024-09-13T16:35:54.889149+0000","last_became_peered":"2024-09-13T16:35:54.889149+0000","last_unstale":"2024-09-13T16:35:55.261408+0000","last_undegraded":"2024-09-13T16:35:55.261408+0000","last_fullsized":"2024-09-13T16:35:55.261408+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:59.524539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282359+0000","last_change":"2024-09-13T16:35:55.277317+0000","last_active":"2024-09-13T16:35:55.282359+0000","last_peered":"2024-09-13T16:35:55.282359+0000","last_clean":"2024-09-13T16:35:55.282359+0000","last_became_active":"2024-09-13T16:35:55.276491+0000","last_became_peered":"2024-09-13T16:35:55.276491+0000","last_unstale":"2024-09-13T16:35:55.282359+0000","last_undegraded":"2024-09-13T16:35:55.282359+0000","last_fullsized":"2024-09-13T16:35:55.282359+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:43.626192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.257841+0000","last_change":"2024-09-13T16:35:54.886698+0000","last_active":"2024-09-13T16:35:55.257841+0000","last_peered":"2024-09-13T16:35:55.257841+0000","last_clean":"2024-09-13T16:35:55.257841+0000","last_became_active":"2024-09-13T16:35:54.886421+0000","last_became_peered":"2024-09-13T16:35:54.886421+0000","last_unstale":"2024-09-13T16:35:55.257841+0000","last_undegraded":"2024-09-13T16:35:55.257841+0000","last_fullsized":"2024-09-13T16:35:55.257841+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:31.957939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275096+0000","last_change":"2024-09-13T16:35:55.271988+0000","last_active":"2024-09-13T16:35:55.275096+0000","last_peered":"2024-09-13T16:35:55.275096+0000","last_clean":"2024-09-13T16:35:55.275096+0000","last_became_active":"2024-09-13T16:35:55.271718+0000","last_became_peered":"2024-09-13T16:35:55.271718+0000","last_unstale":"2024-09-13T16:35:55.275096+0000","last_undegraded":"2024-09-13T16:35:55.275096+0000","last_fullsized":"2024-09-13T16:35:55.275096+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:59:54.573695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281181+0000","last_change":"2024-09-13T16:35:54.881475+0000","last_active":"2024-09-13T16:35:55.281181+0000","last_peered":"2024-09-13T16:35:55.281181+0000","last_clean":"2024-09-13T16:35:55.281181+0000","last_became_active":"2024-09-13T16:35:54.881201+0000","last_became_peered":"2024-09-13T16:35:54.881201+0000","last_unstale":"2024-09-13T16:35:55.281181+0000","last_undegraded":"2024-09-13T16:35:55.281181+0000","last_fullsized":"2024-09-13T16:35:55.281181+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:10.445392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280217+0000","last_change":"2024-09-13T16:35:55.276287+0000","last_active":"2024-09-13T16:35:55.280217+0000","last_peered":"2024-09-13T16:35:55.280217+0000","last_clean":"2024-09-13T16:35:55.280217+0000","last_became_active":"2024-09-13T16:35:55.275984+0000","last_became_peered":"2024-09-13T16:35:55.275984+0000","last_unstale":"2024-09-13T16:35:55.280217+0000","last_undegraded":"2024-09-13T16:35:55.280217+0000","last_fullsized":"2024-09-13T16:35:55.280217+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:38.801815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.264589+0000","last_change":"2024-09-13T16:35:54.890898+0000","last_active":"2024-09-13T16:35:55.264589+0000","last_peered":"2024-09-13T16:35:55.264589+0000","last_clean":"2024-09-13T16:35:55.264589+0000","last_became_active":"2024-09-13T16:35:54.890640+0000","last_became_peered":"2024-09-13T16:35:54.890640+0000","last_unstale":"2024-09-13T16:35:55.264589+0000","last_undegraded":"2024-09-13T16:35:55.264589+0000","last_fullsized":"2024-09-13T16:35:55.264589+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:43.039403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284204+0000","last_change":"2024-09-13T16:35:55.281012+0000","last_active":"2024-09-13T16:35:55.284204+0000","last_peered":"2024-09-13T16:35:55.284204+0000","last_clean":"2024-09-13T16:35:55.284204+0000","last_became_active":"2024-09-13T16:35:55.280756+0000","last_became_peered":"2024-09-13T16:35:55.280756+0000","last_unstale":"2024-09-13T16:35:55.284204+0000","last_undegraded":"2024-09-13T16:35:55.284204+0000","last_fullsized":"2024-09-13T16:35:55.284204+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:48.360160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.270405+0000","last_change":"2024-09-13T16:35:54.892603+0000","last_active":"2024-09-13T16:35:55.270405+0000","last_peered":"2024-09-13T16:35:55.270405+0000","last_clean":"2024-09-13T16:35:55.270405+0000","last_became_active":"2024-09-13T16:35:54.892255+0000","last_became_peered":"2024-09-13T16:35:54.892255+0000","last_unstale":"2024-09-13T16:35:55.270405+0000","last_undegraded":"2024-09-13T16:35:55.270405+0000","last_fullsized":"2024-09-13T16:35:55.270405+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:13.571296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.272623+0000","last_change":"2024-09-13T16:35:55.270232+0000","last_active":"2024-09-13T16:35:55.272623+0000","last_peered":"2024-09-13T16:35:55.272623+0000","last_clean":"2024-09-13T16:35:55.272623+0000","last_became_active":"2024-09-13T16:35:55.269974+0000","last_became_peered":"2024-09-13T16:35:55.269974+0000","last_unstale":"2024-09-13T16:35:55.272623+0000","last_undegraded":"2024-09-13T16:35:55.272623+0000","last_fullsized":"2024-09-13T16:35:55.272623+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:51.504152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.260016+0000","last_change":"2024-09-13T16:35:54.888833+0000","last_active":"2024-09-13T16:35:55.260016+0000","last_peered":"2024-09-13T16:35:55.260016+0000","last_clean":"2024-09-13T16:35:55.260016+0000","last_became_active":"2024-09-13T16:35:54.888568+0000","last_became_peered":"2024-09-13T16:35:54.888568+0000","last_unstale":"2024-09-13T16:35:55.260016+0000","last_undegraded":"2024-09-13T16:35:55.260016+0000","last_fullsized":"2024-09-13T16:35:55.260016+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:07.508642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281545+0000","last_change":"2024-09-13T16:35:55.280416+0000","last_active":"2024-09-13T16:35:55.281545+0000","last_peered":"2024-09-13T16:35:55.281545+0000","last_clean":"2024-09-13T16:35:55.281545+0000","last_became_active":"2024-09-13T16:35:55.280169+0000","last_became_peered":"2024-09-13T16:35:55.280169+0000","last_unstale":"2024-09-13T16:35:55.281545+0000","last_undegraded":"2024-09-13T16:35:55.281545+0000","last_fullsized":"2024-09-13T16:35:55.281545+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:54.455858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.267560+0000","last_change":"2024-09-13T16:35:54.883820+0000","last_active":"2024-09-13T16:35:55.267560+0000","last_peered":"2024-09-13T16:35:55.267560+0000","last_clean":"2024-09-13T16:35:55.267560+0000","last_became_active":"2024-09-13T16:35:54.883505+0000","last_became_peered":"2024-09-13T16:35:54.883505+0000","last_unstale":"2024-09-13T16:35:55.267560+0000","last_undegraded":"2024-09-13T16:35:55.267560+0000","last_fullsized":"2024-09-13T16:35:55.267560+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:02.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.274190+0000","last_change":"2024-09-13T16:35:55.270760+0000","last_active":"2024-09-13T16:35:55.274190+0000","last_peered":"2024-09-13T16:35:55.274190+0000","last_clean":"2024-09-13T16:35:55.274190+0000","last_became_active":"2024-09-13T16:35:55.270503+0000","last_became_peered":"2024-09-13T16:35:55.270503+0000","last_unstale":"2024-09-13T16:35:55.274190+0000","last_undegraded":"2024-09-13T16:35:55.274190+0000","last_fullsized":"2024-09-13T16:35:55.274190+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:43.882250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.267624+0000","last_change":"2024-09-13T16:35:54.877539+0000","last_active":"2024-09-13T16:35:55.267624+0000","last_peered":"2024-09-13T16:35:55.267624+0000","last_clean":"2024-09-13T16:35:55.267624+0000","last_became_active":"2024-09-13T16:35:54.877243+0000","last_became_peered":"2024-09-13T16:35:54.877243+0000","last_unstale":"2024-09-13T16:35:55.267624+0000","last_undegraded":"2024-09-13T16:35:55.267624+0000","last_fullsized":"2024-09-13T16:35:55.267624+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:09.661131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279805+0000","last_change":"2024-09-13T16:35:55.275460+0000","last_active":"2024-09-13T16:35:55.279805+0000","last_peered":"2024-09-13T16:35:55.279805+0000","last_clean":"2024-09-13T16:35:55.279805+0000","last_became_active":"2024-09-13T16:35:55.275191+0000","last_became_peered":"2024-09-13T16:35:55.275191+0000","last_unstale":"2024-09-13T16:35:55.279805+0000","last_undegraded":"2024-09-13T16:35:55.279805+0000","last_fullsized":"2024-09-13T16:35:55.279805+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:37.136914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.258845+0000","last_change":"2024-09-13T16:35:54.887053+0000","last_active":"2024-09-13T16:35:55.258845+0000","last_peered":"2024-09-13T16:35:55.258845+0000","last_clean":"2024-09-13T16:35:55.258845+0000","last_became_active":"2024-09-13T16:35:54.886576+0000","last_became_peered":"2024-09-13T16:35:54.886576+0000","last_unstale":"2024-09-13T16:35:55.258845+0000","last_undegraded":"2024-09-13T16:35:55.258845+0000","last_fullsized":"2024-09-13T16:35:55.258845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:00:38.599113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278211+0000","last_change":"2024-09-13T16:35:55.270698+0000","last_active":"2024-09-13T16:35:55.278211+0000","last_peered":"2024-09-13T16:35:55.278211+0000","last_clean":"2024-09-13T16:35:55.278211+0000","last_became_active":"2024-09-13T16:35:55.270046+0000","last_became_peered":"2024-09-13T16:35:55.270046+0000","last_unstale":"2024-09-13T16:35:55.278211+0000","last_undegraded":"2024-09-13T16:35:55.278211+0000","last_fullsized":"2024-09-13T16:35:55.278211+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:53.391689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.257942+0000","last_change":"2024-09-13T16:35:54.888362+0000","last_active":"2024-09-13T16:35:55.257942+0000","last_peered":"2024-09-13T16:35:55.257942+0000","last_clean":"2024-09-13T16:35:55.257942+0000","last_became_active":"2024-09-13T16:35:54.888090+0000","last_became_peered":"2024-09-13T16:35:54.888090+0000","last_unstale":"2024-09-13T16:35:55.257942+0000","last_undegraded":"2024-09-13T16:35:55.257942+0000","last_fullsized":"2024-09-13T16:35:55.257942+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:10.760716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279945+0000","last_change":"2024-09-13T16:35:55.272268+0000","last_active":"2024-09-13T16:35:55.279945+0000","last_peered":"2024-09-13T16:35:55.279945+0000","last_clean":"2024-09-13T16:35:55.279945+0000","last_became_active":"2024-09-13T16:35:55.271334+0000","last_became_peered":"2024-09-13T16:35:55.271334+0000","last_unstale":"2024-09-13T16:35:55.279945+0000","last_undegraded":"2024-09-13T16:35:55.279945+0000","last_fullsized":"2024-09-13T16:35:55.279945+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:47.285453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"125'8","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.277694+0000","last_change":"2024-09-13T16:35:54.887314+0000","last_active":"2024-09-13T16:35:55.277694+0000","last_peered":"2024-09-13T16:35:55.277694+0000","last_clean":"2024-09-13T16:35:55.277694+0000","last_became_active":"2024-09-13T16:35:54.886972+0000","last_became_peered":"2024-09-13T16:35:54.886972+0000","last_unstale":"2024-09-13T16:35:55.277694+0000","last_undegraded":"2024-09-13T16:35:55.277694+0000","last_fullsized":"2024-09-13T16:35:55.277694+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:49.576939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":257,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T16:35:54.881450+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:35:54.881450+0000","last_peered":"2024-09-13T16:35:54.881450+0000","last_clean":"2024-09-13T16:35:54.881450+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:35:54.881450+0000","last_undegraded":"2024-09-13T16:35:54.881450+0000","last_fullsized":"2024-09-13T16:35:54.881450+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2134016,"data_stored":1301067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81799999999999995,"15min":0.84199999999999997},"min":{"1min":0.47799999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.262,"5min":1.262,"15min":1.357},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.629,"5min":0.52000000000000002,"15min":0.47699999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.3460000000000001},"last":0.79300000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.56699999999999995,"5min":0.56000000000000005,"15min":0.501},"max":{"1min":1.1619999999999999,"5min":1.212,"15min":1.569},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.88200000000000001,"15min":0.86299999999999999},"min":{"1min":0.623,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1719999999999999,"5min":1.2290000000000001,"15min":1.4219999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.86099999999999999,"15min":0.876},"min":{"1min":0.70799999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.2330000000000001,"15min":1.242},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88600000000000001,"15min":0.86299999999999999},"min":{"1min":0.67300000000000004,"5min":0.60499999999999998,"15min":0.53900000000000003},"max":{"1min":1.24,"5min":1.24,"15min":1.339},"last":0.97399999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91000000000000003,"15min":0.96599999999999997},"min":{"1min":0.67400000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.476},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.006,"5min":0.999,"15min":1.002},"min":{"1min":0.76200000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.3939999999999999},"last":1.0600000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.85999999999999999,"15min":0.871},"min":{"1min":0.60699999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.214,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.94199999999999995,"15min":1.014},"min":{"1min":0.65100000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.3720000000000001,"5min":1.381,"15min":1.589},"last":1.1140000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.98499999999999999,"15min":0.99399999999999999},"min":{"1min":0.745,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.173,"5min":1.444,"15min":1.444},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0600000000000001,"15min":1.091},"min":{"1min":0.72799999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4910000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.149}]},{"osd":6,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.032,"15min":1.054},"min":{"1min":0.745,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.4279999999999999,"5min":1.72,"15min":1.72},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.048,"15min":1.1180000000000001},"min":{"1min":0.71399999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.367,"5min":1.542,"15min":1.6779999999999999},"last":1.264}]}]},{"osd":6,"up_from":14,"seq":60129542275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1540096,"data_stored":710699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.83499999999999996,"15min":0.84299999999999997},"min":{"1min":0.65400000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.1319999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.113},{"interface":"front","average":{"1min":0.872,"5min":0.85299999999999998,"15min":0.879},"min":{"1min":0.70199999999999996,"5min":0.309,"15min":0.309},"max":{"1min":1.1639999999999999,"5min":1.2949999999999999,"15min":1.4410000000000001},"last":0.92700000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.871,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.2030000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.90400000000000003,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.2569999999999999,"5min":1.411,"15min":1.411},"last":0.57699999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84299999999999997,"15min":0.86799999999999999},"min":{"1min":0.502,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.103,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.91600000000000004,"15min":0.877},"min":{"1min":0.82799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1990000000000001,"5min":1.399,"15min":1.399},"last":0.88500000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.92500000000000004,"15min":0.95299999999999996},"min":{"1min":0.69499999999999995,"5min":0.379,"15min":0.379},"max":{"1min":1.2110000000000001,"5min":1.544,"15min":1.544},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.997,"5min":0.96599999999999997,"15min":0.98099999999999998},"min":{"1min":0.68899999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1970000000000001,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.1639999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.94399999999999995,"15min":0.95799999999999996},"min":{"1min":0.81000000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.133,"5min":1.508,"15min":1.508},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.93899999999999995,"15min":0.95499999999999996},"min":{"1min":0.69099999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1970000000000001,"5min":1.55,"15min":1.6140000000000001},"last":1.343}]},{"osd":5,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.94099999999999995,"15min":0.92900000000000005},"min":{"1min":0.73799999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.25,"5min":1.339,"15min":1.339},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0569999999999999,"15min":1.095},"min":{"1min":0.97999999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3029999999999999,"5min":1.615,"15min":1.6439999999999999},"last":1.2829999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0369999999999999,"15min":1.0880000000000001},"min":{"1min":0.76400000000000001,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3380000000000001,"5min":1.3819999999999999,"15min":1.522},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.0649999999999999,"15min":1.0580000000000001},"min":{"1min":0.79400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.319,"5min":1.415,"15min":1.427},"last":1.0349999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1560576,"data_stored":717954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.84999999999999998},"min":{"1min":0.58299999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.1619999999999999,"5min":1.1899999999999999,"15min":1.411},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"min":{"1min":0.56100000000000005,"5min":0.497,"15min":0.41699999999999998},"max":{"1min":1.173,"5min":1.3029999999999999,"15min":1.675},"last":0.88700000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.83999999999999997,"15min":0.86499999999999999},"min":{"1min":0.66200000000000003,"5min":0.51900000000000002,"15min":0.46600000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.4079999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.83499999999999996,"15min":0.83799999999999997},"min":{"1min":0.54900000000000004,"5min":0.52300000000000002,"15min":0.36499999999999999},"max":{"1min":1.1579999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.81000000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.91000000000000003,"15min":0.91900000000000004},"min":{"1min":0.61899999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.129,"5min":1.135,"15min":1.625},"last":0.873},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.97799999999999998,"15min":0.97599999999999998},"min":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.55000000000000004},"max":{"1min":1.1599999999999999,"5min":1.331,"15min":1.552},"last":0.90300000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91200000000000003,"15min":0.876},"min":{"1min":0.67500000000000004,"5min":0.5,"15min":0.5},"max":{"1min":1.143,"5min":1.2290000000000001,"15min":1.3919999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88800000000000001,"15min":0.95099999999999996},"min":{"1min":0.66100000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.1160000000000001,"5min":1.2929999999999999,"15min":3.8460000000000001},"last":0.93400000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.88500000000000001,"15min":0.98199999999999998},"min":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1439999999999999,"5min":1.3540000000000001,"15min":3.6869999999999998},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0640000000000001,"15min":1.0629999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.57499999999999996},"max":{"1min":1.244,"5min":1.365,"15min":1.601},"last":1.05}]},{"osd":6,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.91000000000000003,"15min":0.94699999999999995},"min":{"1min":0.65200000000000002,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2390000000000001,"5min":1.3540000000000001,"15min":1.421},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.98999999999999999,"15min":1.0469999999999999},"min":{"1min":0.77100000000000002,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.2150000000000001,"5min":1.3740000000000001,"15min":2.21},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0069999999999999,"15min":1.03},"min":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.45500000000000002},"max":{"1min":1.294,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":0.629},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.054,"15min":1.1399999999999999},"min":{"1min":0.80200000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.208,"5min":1.502,"15min":3.9830000000000001},"last":1.03}]}]},{"osd":4,"up_from":14,"seq":60129542275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1527808,"data_stored":704104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84899999999999998,"15min":0.84399999999999997},"min":{"1min":0.63700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.093,"5min":1.21,"15min":1.411},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.873},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.44600000000000001},"max":{"1min":1.127,"5min":1.462,"15min":2.899},"last":1.1699999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.89300000000000002,"15min":0.91000000000000003},"min":{"1min":0.59699999999999998,"5min":0.53800000000000003,"15min":0.46200000000000002},"max":{"1min":1.151,"5min":1.2490000000000001,"15min":3.085},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.90400000000000003},"min":{"1min":0.58999999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.1930000000000001,"5min":1.3240000000000001,"15min":1.5720000000000001},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.86099999999999999,"15min":0.84799999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.433},"max":{"1min":1.177,"5min":1.228,"15min":1.4419999999999999},"last":0.755},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.92300000000000004,"15min":0.95599999999999996},"min":{"1min":0.78000000000000003,"5min":0.60199999999999998,"15min":0.498},"max":{"1min":1.165,"5min":1.3640000000000001,"15min":3.2160000000000002},"last":1.3220000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.90700000000000003,"15min":0.93700000000000006},"min":{"1min":0.66300000000000003,"5min":0.56899999999999995,"15min":0.47999999999999998},"max":{"1min":1.1859999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.92900000000000005,"15min":0.94399999999999995},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.52200000000000002},"max":{"1min":1.2110000000000001,"5min":1.3979999999999999,"15min":1.577},"last":0.99299999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.90600000000000003,"15min":0.95399999999999996},"min":{"1min":0.70599999999999996,"5min":0.59999999999999998,"15min":0.49299999999999999},"max":{"1min":1.252,"5min":1.252,"15min":2.996},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.98599999999999999,"15min":1.073},"min":{"1min":0.61099999999999999,"5min":0.495,"15min":0.495},"max":{"1min":1.232,"5min":1.5189999999999999,"15min":2.8860000000000001},"last":1.0569999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.93799999999999994,"15min":0.93899999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.169,"5min":1.522,"15min":1.6499999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0669999999999999,"15min":1.101},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.55500000000000005},"max":{"1min":1.3220000000000001,"5min":1.3939999999999999,"15min":3.1469999999999998},"last":1.375}]},{"osd":7,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0469999999999999,"15min":1.1040000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.28,"5min":1.556,"15min":2.1230000000000002},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.071,"15min":1.099},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.67000000000000004},"max":{"1min":1.242,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":1.288}]}]},{"osd":3,"up_from":14,"seq":60129542276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32612,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1527808,"data_stored":704104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.5,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83199999999999996,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.90100000000000002,"5min":1.107,"15min":1.161},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66100000000000003,"15min":0.64800000000000002},"min":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.35499999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82199999999999995,"15min":0.79700000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.163,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.77300000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.70199999999999996},"min":{"1min":0.53700000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.27},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81200000000000006,"15min":0.77800000000000002},"min":{"1min":0.58999999999999997,"5min":0.379,"15min":0.379},"max":{"1min":1.069,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.96499999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95099999999999996,"15min":0.92000000000000004},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.60499999999999998},"max":{"1min":1.2929999999999999,"5min":1.3380000000000001,"15min":1.8460000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.93000000000000005,"15min":0.95099999999999996},"min":{"1min":0.747,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1930000000000001,"5min":1.3360000000000001,"15min":1.782},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95299999999999996,"15min":0.96599999999999997},"min":{"1min":0.78000000000000003,"5min":0.65900000000000003,"15min":0.59499999999999997},"max":{"1min":1.3100000000000001,"5min":1.397,"15min":1.6299999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.038,"5min":1.0409999999999999,"15min":1.069},"min":{"1min":0.75800000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.367,"5min":1.581,"15min":1.8140000000000001},"last":1.3049999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.079,"15min":1.099},"min":{"1min":0.79400000000000004,"5min":0.61599999999999999,"15min":0.58699999999999997},"max":{"1min":1.4159999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.25},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0669999999999999,"15min":1.0489999999999999},"min":{"1min":0.76900000000000002,"5min":0.64900000000000002,"15min":0.57499999999999996},"max":{"1min":1.4530000000000001,"5min":1.7030000000000001,"15min":1.95},"last":1.153}]},{"osd":7,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0900000000000001,"15min":1.081},"min":{"1min":0.76900000000000002,"5min":0.74199999999999999,"15min":0.71399999999999997},"max":{"1min":1.419,"5min":1.657,"15min":1.889},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0580000000000001,"15min":1.0569999999999999},"min":{"1min":0.81299999999999994,"5min":0.77200000000000002,"15min":0.70799999999999996},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.756},"last":0.82499999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1540096,"data_stored":710699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.75700000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.46800000000000003,"5min":0.377,"15min":0.377},"max":{"1min":1.0920000000000001,"5min":1.169,"15min":1.169},"last":0.751}]},{"osd":1,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.51600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.96199999999999997,"5min":0.995,"15min":0.997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71299999999999997,"15min":0.69899999999999995},"min":{"1min":0.59499999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.88300000000000001,"5min":1.147,"15min":1.147},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.71299999999999997,"15min":0.68500000000000005},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.38100000000000001},"max":{"1min":0.91400000000000003,"5min":1.119,"15min":1.119},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.82999999999999996,"15min":0.81200000000000006},"min":{"1min":0.59299999999999997,"5min":0.37,"15min":0.37},"max":{"1min":1.0589999999999999,"5min":1.1000000000000001,"15min":1.2330000000000001},"last":0.88600000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.93300000000000005,"15min":0.94299999999999995},"min":{"1min":0.70899999999999996,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.131,"5min":1.448,"15min":1.5149999999999999},"last":1.091},{"interface":"front","average":{"1min":1.004,"5min":0.97699999999999998,"15min":0.96899999999999997},"min":{"1min":0.77600000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.8799999999999999},"last":1.4219999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.95499999999999996,"15min":0.96899999999999997},"min":{"1min":0.79700000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2,"5min":1.452,"15min":1.4750000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"min":{"1min":0.70799999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1839999999999999,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.155}]},{"osd":6,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.98899999999999999,"15min":0.999},"min":{"1min":0.68300000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.347,"5min":1.347,"15min":2.024},"last":1.302},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.052},"min":{"1min":0.85099999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.214,"5min":1.5249999999999999,"15min":1.6359999999999999},"last":0.93100000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0489999999999999,"15min":1.0649999999999999},"min":{"1min":0.754,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.321,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.27},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0620000000000001,"15min":1.0569999999999999},"min":{"1min":0.81399999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.423,"5min":1.587,"15min":1.587},"last":1.482}]}]},{"osd":1,"up_from":14,"seq":60129542276,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":717816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.45100000000000001,"5min":0.42999999999999999,"15min":0.23300000000000001},"max":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":1.0880000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.56200000000000006,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.2010000000000001},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.48699999999999999,"5min":0.38300000000000001,"15min":0.26700000000000002},"max":{"1min":0.94099999999999995,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.48999999999999999,"5min":0.45700000000000002,"15min":0.29699999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.458},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.74299999999999999,"15min":0.72299999999999998},"min":{"1min":0.56299999999999994,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":0.98099999999999998,"5min":1.1359999999999999,"15min":1.4019999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70599999999999996,"15min":0.68899999999999995},"min":{"1min":0.52700000000000002,"5min":0.47099999999999997,"15min":0.39600000000000002},"max":{"1min":0.92200000000000004,"5min":1.175,"15min":1.403},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.95399999999999996,"15min":0.92700000000000005},"min":{"1min":0.68500000000000005,"5min":0.67800000000000005,"15min":0.505},"max":{"1min":1.1559999999999999,"5min":1.2190000000000001,"15min":1.546},"last":0.88},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.81499999999999995,"5min":0.628,"15min":0.51700000000000002},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":2.2040000000000002},"last":1.1459999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96599999999999997,"15min":0.95999999999999996},"min":{"1min":0.68799999999999994,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1279999999999999,"5min":1.3819999999999999,"15min":1.389},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93100000000000005,"15min":0.95099999999999996},"min":{"1min":0.77600000000000002,"5min":0.56699999999999995,"15min":0.52000000000000002},"max":{"1min":1.0760000000000001,"5min":1.238,"15min":2.3029999999999999},"last":0.98699999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.998,"15min":1.006},"min":{"1min":0.82299999999999995,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.1599999999999999,"5min":1.4330000000000001,"15min":1.613},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0529999999999999,"15min":1.028},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.59299999999999997},"max":{"1min":1.667,"5min":1.667,"15min":1.698},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0249999999999999,"15min":1.0129999999999999},"min":{"1min":0.83199999999999996,"5min":0.64400000000000002,"15min":0.62},"max":{"1min":1.224,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.071,"5min":1.069,"15min":1.0580000000000001},"min":{"1min":0.94599999999999995,"5min":0.61599999999999999,"15min":0.59299999999999997},"max":{"1min":1.2829999999999999,"5min":1.4059999999999999,"15min":1.7689999999999999},"last":1.006}]}]},{"osd":0,"up_from":14,"seq":60129542276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33224,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338616,"statfs":{"total":96636764160,"available":96602742784,"internally_reserved":0,"allocated":2150400,"data_stored":1308184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.496,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":0.91400000000000003,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69699999999999995,"15min":0.67300000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.377},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.157},"last":0.78600000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.70199999999999996,"15min":0.71599999999999997},"min":{"1min":0.55400000000000005,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.202},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.72699999999999998,"15min":0.70599999999999996},"min":{"1min":0.51800000000000002,"5min":0.443,"15min":0.39800000000000002},"max":{"1min":1.0760000000000001,"5min":1.0780000000000001,"15min":1.2629999999999999},"last":0.90800000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.52800000000000002,"5min":0.42799999999999999,"15min":0.312},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.4830000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88100000000000001,"15min":0.85399999999999998},"min":{"1min":0.72899999999999998,"5min":0.53300000000000003,"15min":0.42899999999999999},"max":{"1min":1.155,"5min":1.155,"15min":1.6100000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.96399999999999997,"15min":0.95999999999999996},"min":{"1min":0.66900000000000004,"5min":0.629,"15min":0.57699999999999996},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.486},"last":1.3080000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":0.95999999999999996},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.59999999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.704},"last":0.88700000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.98699999999999999,"15min":0.99099999999999999},"min":{"1min":0.72899999999999998,"5min":0.67100000000000004,"15min":0.59599999999999997},"max":{"1min":1.532,"5min":1.532,"15min":1.575},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.99299999999999999,"15min":0.98999999999999999},"min":{"1min":0.72799999999999998,"5min":0.61199999999999999,"15min":0.443},"max":{"1min":1.49,"5min":1.742,"15min":1.742},"last":1.399}]},{"osd":6,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96699999999999997,"15min":0.97999999999999998},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.628},"max":{"1min":1.218,"5min":1.4099999999999999,"15min":1.944},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.0840000000000001,"15min":1.0700000000000001},"min":{"1min":0.92900000000000005,"5min":0.77300000000000002,"15min":0.58899999999999997},"max":{"1min":1.53,"5min":1.53,"15min":1.5780000000000001},"last":1.0980000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0860000000000001,"15min":1.0669999999999999},"min":{"1min":0.76300000000000001,"5min":0.747,"15min":0.61699999999999999},"max":{"1min":1.421,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.359},{"interface":"front","average":{"1min":1.042,"5min":1.0800000000000001,"15min":1.0700000000000001},"min":{"1min":0.84899999999999998,"5min":0.80900000000000005,"15min":0.65400000000000003},"max":{"1min":1.274,"5min":1.472,"15min":2.0139999999999998},"last":0.98799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:35:59.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:35:59.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fbaea5700 1 Processor -- start 2024-09-13T16:35:59.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fbaea5700 1 -- start start 2024-09-13T16:35:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fbaea5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0fb406b730 0x7f0fb40cfad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fbaea5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb40d2c90 0x7f0fb40d3120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fbaea5700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fb40ceb60 con 0x7f0fb415cd60 2024-09-13T16:35:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fbaea5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fb415d140 con 0x7f0fb406b730 2024-09-13T16:35:59.431 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fbaea5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40ce230 con 0x7f0fb40d2c90 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.426+0000 7f0fb96a2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0fb406b730 0x7f0fb40cfad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb96a2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0fb406b730 0x7f0fb40cfad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56348/0 (socket says 172.21.15.119:56348) 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb96a2700 1 -- 172.21.15.119:0/3688872234 learned_addr learned my addr 172.21.15.119:0/3688872234 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fba6a4700 1 --2- 172.21.15.119:0/3688872234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb40d2c90 0x7f0fb40d3120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb96a2700 1 -- 172.21.15.119:0/3688872234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb40d2c90 msgr2=0x7f0fb40d3120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb96a2700 1 --2- 172.21.15.119:0/3688872234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb40d2c90 0x7f0fb40d3120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb96a2700 1 -- 172.21.15.119:0/3688872234 >> v1:172.21.15.119:6789/0 conn(0x7f0fb415cd60 legacy=0x7f0fb40cf3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.432 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb96a2700 1 -- 172.21.15.119:0/3688872234 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb40ce510 con 0x7f0fb406b730 2024-09-13T16:35:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb96a2700 1 --2- 172.21.15.119:0/3688872234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb406b730 0x7f0fb40cfad0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009910 tx=0x7f0f9c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dcae168adffaca1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.433 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb8ea1700 1 -- 172.21.15.119:0/3688872234 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c00ba60 con 0x7f0fb406b730 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb8ea1700 1 -- 172.21.15.119:0/3688872234 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c00bbc0 con 0x7f0fb406b730 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fb8ea1700 1 -- 172.21.15.119:0/3688872234 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c009e60 con 0x7f0fb406b730 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 -- 172.21.15.119:0/3688872234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb406b730 msgr2=0x7f0fb40cfad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/3688872234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb406b730 0x7f0fb40cfad0 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7f0f9c009910 tx=0x7f0f9c009c20 comp rx=0 tx=0).stop 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 -- 172.21.15.119:0/3688872234 shutdown_connections 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/3688872234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb40d2c90 0x7f0fb40d3120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/3688872234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb406b730 0x7f0fb40cfad0 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 -- 172.21.15.119:0/3688872234 >> 172.21.15.119:0/3688872234 conn(0x7f0fb408c240 msgr2=0x7f0fb409c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 -- 172.21.15.119:0/3688872234 shutdown_connections 2024-09-13T16:35:59.434 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.430+0000 7f0fbaea5700 1 -- 172.21.15.119:0/3688872234 wait complete. 2024-09-13T16:35:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 Processor -- start 2024-09-13T16:35:59.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 -- start start 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb406b730 0x7f0fb40b7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb40d2c90 0x7f0fb4150200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb415cd60 0x7f0fb40c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40cd2e0 con 0x7f0fb40d2c90 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb415e2f0 con 0x7f0fb415cd60 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fbaea5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb40b7fb0 con 0x7f0fb406b730 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fb96a2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb40d2c90 0x7f0fb4150200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fb96a2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb40d2c90 0x7f0fb4150200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56358/0 (socket says 172.21.15.119:56358) 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fb96a2700 1 -- 172.21.15.119:0/1892856949 learned_addr learned my addr 172.21.15.119:0/1892856949 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fb9ea3700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb406b730 0x7f0fb40b7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.434+0000 7f0fba6a4700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb415cd60 0x7f0fb40c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb96a2700 1 -- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb406b730 msgr2=0x7f0fb40b7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb96a2700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb406b730 0x7f0fb40b7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb96a2700 1 -- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb415cd60 msgr2=0x7f0fb40c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb96a2700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb415cd60 0x7f0fb40c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb96a2700 1 -- 172.21.15.119:0/1892856949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f9c009570 con 0x7f0fb40d2c90 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb9ea3700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb406b730 0x7f0fb40b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb96a2700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb40d2c90 0x7f0fb4150200 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40b58c0 tx=0x7f0f9c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb27fc700 1 -- 172.21.15.119:0/1892856949 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c004610 con 0x7f0fb40d2c90 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb27fc700 1 -- 172.21.15.119:0/1892856949 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9c01f070 con 0x7f0fb40d2c90 2024-09-13T16:35:59.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb4150740 con 0x7f0fb40d2c90 2024-09-13T16:35:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb27fc700 1 -- 172.21.15.119:0/1892856949 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9c003900 con 0x7f0fb40d2c90 2024-09-13T16:35:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb40b8210 con 0x7f0fb40d2c90 2024-09-13T16:35:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb40d3120 con 0x7f0fb40d2c90 2024-09-13T16:35:59.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.438+0000 7f0fb27fc700 1 -- 172.21.15.119:0/1892856949 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0f9c003a60 con 0x7f0fb40d2c90 2024-09-13T16:35:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.442+0000 7f0fb27fc700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f94066730 0x7f0f94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.442+0000 7f0fb27fc700 1 -- 172.21.15.119:0/1892856949 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0f9c021070 con 0x7f0fb40d2c90 2024-09-13T16:35:59.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.442+0000 7f0fb9ea3700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f94066730 0x7f0f94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.442+0000 7f0fb9ea3700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f94066730 0x7f0f94068bf0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0fb415c670 tx=0x7f0fac009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.450+0000 7f0fb27fc700 1 -- 172.21.15.119:0/1892856949 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0f9c052dc0 con 0x7f0fb40d2c90 2024-09-13T16:35:59.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.614+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0fb406be10 con 0x7f0f94066730 2024-09-13T16:35:59.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.630+0000 7f0fb27fc700 1 -- 172.21.15.119:0/1892856949 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+288169 (secure 0 0 0) 0x7f0fac00ea70 con 0x7f0f94066730 2024-09-13T16:35:59.639 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:35:59.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f94066730 msgr2=0x7f0f94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f94066730 0x7f0f94068bf0 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f0fb415c670 tx=0x7f0fac009040 comp rx=0 tx=0).stop 2024-09-13T16:35:59.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb40d2c90 msgr2=0x7f0fb4150200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb40d2c90 0x7f0fb4150200 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7f0fb40b58c0 tx=0x7f0f9c004770 comp rx=0 tx=0).stop 2024-09-13T16:35:59.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 shutdown_connections 2024-09-13T16:35:59.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0f94066730 0x7f0f94068bf0 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0fb415cd60 0x7f0fb40c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0fb40d2c90 0x7f0fb4150200 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 --2- 172.21.15.119:0/1892856949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0fb406b730 0x7f0fb40b7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 >> 172.21.15.119:0/1892856949 conn(0x7f0fb408c240 msgr2=0x7f0fb41e7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:59.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 shutdown_connections 2024-09-13T16:35:59.643 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.638+0000 7f0fbaea5700 1 -- 172.21.15.119:0/1892856949 wait complete. 2024-09-13T16:35:59.643 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:35:59.654 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2024-09-13T16:35:57.650585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592730,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":5,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":262188,"kb_used_data":13220,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754712532,"statfs":{"total":773094113280,"available":772825632768,"internally_reserved":0,"allocated":13537280,"data_stored":6874627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":34,"apply_latency_ms":34,"commit_latency_ns":34000000,"apply_latency_ns":34000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001341"},"pg_stats":[{"pgid":"18.3f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284254+0000","last_change":"2024-09-13T16:35:55.279603+0000","last_active":"2024-09-13T16:35:55.284254+0000","last_peered":"2024-09-13T16:35:55.284254+0000","last_clean":"2024-09-13T16:35:55.284254+0000","last_became_active":"2024-09-13T16:35:55.279310+0000","last_became_peered":"2024-09-13T16:35:55.279310+0000","last_unstale":"2024-09-13T16:35:55.284254+0000","last_undegraded":"2024-09-13T16:35:55.284254+0000","last_fullsized":"2024-09-13T16:35:55.284254+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:37.136914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280009+0000","last_change":"2024-09-13T16:35:55.277134+0000","last_active":"2024-09-13T16:35:55.280009+0000","last_peered":"2024-09-13T16:35:55.280009+0000","last_clean":"2024-09-13T16:35:55.280009+0000","last_became_active":"2024-09-13T16:35:55.276859+0000","last_became_peered":"2024-09-13T16:35:55.276859+0000","last_unstale":"2024-09-13T16:35:55.280009+0000","last_undegraded":"2024-09-13T16:35:55.280009+0000","last_fullsized":"2024-09-13T16:35:55.280009+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:21:19.852842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280131+0000","last_change":"2024-09-13T16:35:55.278080+0000","last_active":"2024-09-13T16:35:55.280131+0000","last_peered":"2024-09-13T16:35:55.280131+0000","last_clean":"2024-09-13T16:35:55.280131+0000","last_became_active":"2024-09-13T16:35:55.277853+0000","last_became_peered":"2024-09-13T16:35:55.277853+0000","last_unstale":"2024-09-13T16:35:55.280131+0000","last_undegraded":"2024-09-13T16:35:55.280131+0000","last_fullsized":"2024-09-13T16:35:55.280131+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:55:49.561531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.3c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279876+0000","last_change":"2024-09-13T16:35:55.276626+0000","last_active":"2024-09-13T16:35:55.279876+0000","last_peered":"2024-09-13T16:35:55.279876+0000","last_clean":"2024-09-13T16:35:55.279876+0000","last_became_active":"2024-09-13T16:35:55.276311+0000","last_became_peered":"2024-09-13T16:35:55.276311+0000","last_unstale":"2024-09-13T16:35:55.279876+0000","last_undegraded":"2024-09-13T16:35:55.279876+0000","last_fullsized":"2024-09-13T16:35:55.279876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:30:06.782215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284207+0000","last_change":"2024-09-13T16:35:55.279767+0000","last_active":"2024-09-13T16:35:55.284207+0000","last_peered":"2024-09-13T16:35:55.284207+0000","last_clean":"2024-09-13T16:35:55.284207+0000","last_became_active":"2024-09-13T16:35:55.279415+0000","last_became_peered":"2024-09-13T16:35:55.279415+0000","last_unstale":"2024-09-13T16:35:55.284207+0000","last_undegraded":"2024-09-13T16:35:55.284207+0000","last_fullsized":"2024-09-13T16:35:55.284207+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:28:53.391694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.3a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275710+0000","last_change":"2024-09-13T16:35:55.273486+0000","last_active":"2024-09-13T16:35:55.275710+0000","last_peered":"2024-09-13T16:35:55.275710+0000","last_clean":"2024-09-13T16:35:55.275710+0000","last_became_active":"2024-09-13T16:35:55.272868+0000","last_became_peered":"2024-09-13T16:35:55.272868+0000","last_unstale":"2024-09-13T16:35:55.275710+0000","last_undegraded":"2024-09-13T16:35:55.275710+0000","last_fullsized":"2024-09-13T16:35:55.275710+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:33:08.622453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.39","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278856+0000","last_change":"2024-09-13T16:35:55.276620+0000","last_active":"2024-09-13T16:35:55.278856+0000","last_peered":"2024-09-13T16:35:55.278856+0000","last_clean":"2024-09-13T16:35:55.278856+0000","last_became_active":"2024-09-13T16:35:55.276089+0000","last_became_peered":"2024-09-13T16:35:55.276089+0000","last_unstale":"2024-09-13T16:35:55.278856+0000","last_undegraded":"2024-09-13T16:35:55.278856+0000","last_fullsized":"2024-09-13T16:35:55.278856+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:57:31.583929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.38","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282997+0000","last_change":"2024-09-13T16:35:55.282402+0000","last_active":"2024-09-13T16:35:55.282997+0000","last_peered":"2024-09-13T16:35:55.282997+0000","last_clean":"2024-09-13T16:35:55.282997+0000","last_became_active":"2024-09-13T16:35:55.282151+0000","last_became_peered":"2024-09-13T16:35:55.282151+0000","last_unstale":"2024-09-13T16:35:55.282997+0000","last_undegraded":"2024-09-13T16:35:55.282997+0000","last_fullsized":"2024-09-13T16:35:55.282997+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:58.964322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.37","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276504+0000","last_change":"2024-09-13T16:35:55.271390+0000","last_active":"2024-09-13T16:35:55.276504+0000","last_peered":"2024-09-13T16:35:55.276504+0000","last_clean":"2024-09-13T16:35:55.276504+0000","last_became_active":"2024-09-13T16:35:55.271078+0000","last_became_peered":"2024-09-13T16:35:55.271078+0000","last_unstale":"2024-09-13T16:35:55.276504+0000","last_undegraded":"2024-09-13T16:35:55.276504+0000","last_fullsized":"2024-09-13T16:35:55.276504+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:55.505021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.36","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280370+0000","last_change":"2024-09-13T16:35:55.276030+0000","last_active":"2024-09-13T16:35:55.280370+0000","last_peered":"2024-09-13T16:35:55.280370+0000","last_clean":"2024-09-13T16:35:55.280370+0000","last_became_active":"2024-09-13T16:35:55.275608+0000","last_became_peered":"2024-09-13T16:35:55.275608+0000","last_unstale":"2024-09-13T16:35:55.280370+0000","last_undegraded":"2024-09-13T16:35:55.280370+0000","last_fullsized":"2024-09-13T16:35:55.280370+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:11.770066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.35","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282249+0000","last_change":"2024-09-13T16:35:55.273486+0000","last_active":"2024-09-13T16:35:55.282249+0000","last_peered":"2024-09-13T16:35:55.282249+0000","last_clean":"2024-09-13T16:35:55.282249+0000","last_became_active":"2024-09-13T16:35:55.273165+0000","last_became_peered":"2024-09-13T16:35:55.273165+0000","last_unstale":"2024-09-13T16:35:55.282249+0000","last_undegraded":"2024-09-13T16:35:55.282249+0000","last_fullsized":"2024-09-13T16:35:55.282249+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:13:55.505021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.34","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281569+0000","last_change":"2024-09-13T16:35:55.280916+0000","last_active":"2024-09-13T16:35:55.281569+0000","last_peered":"2024-09-13T16:35:55.281569+0000","last_clean":"2024-09-13T16:35:55.281569+0000","last_became_active":"2024-09-13T16:35:55.280672+0000","last_became_peered":"2024-09-13T16:35:55.280672+0000","last_unstale":"2024-09-13T16:35:55.281569+0000","last_undegraded":"2024-09-13T16:35:55.281569+0000","last_fullsized":"2024-09-13T16:35:55.281569+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:58.174716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.33","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278191+0000","last_change":"2024-09-13T16:35:55.270541+0000","last_active":"2024-09-13T16:35:55.278191+0000","last_peered":"2024-09-13T16:35:55.278191+0000","last_clean":"2024-09-13T16:35:55.278191+0000","last_became_active":"2024-09-13T16:35:55.269759+0000","last_became_peered":"2024-09-13T16:35:55.269759+0000","last_unstale":"2024-09-13T16:35:55.278191+0000","last_undegraded":"2024-09-13T16:35:55.278191+0000","last_fullsized":"2024-09-13T16:35:55.278191+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:58.150363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.32","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275509+0000","last_change":"2024-09-13T16:35:55.271638+0000","last_active":"2024-09-13T16:35:55.275509+0000","last_peered":"2024-09-13T16:35:55.275509+0000","last_clean":"2024-09-13T16:35:55.275509+0000","last_became_active":"2024-09-13T16:35:55.271410+0000","last_became_peered":"2024-09-13T16:35:55.271410+0000","last_unstale":"2024-09-13T16:35:55.275509+0000","last_undegraded":"2024-09-13T16:35:55.275509+0000","last_fullsized":"2024-09-13T16:35:55.275509+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:06.468091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.31","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282279+0000","last_change":"2024-09-13T16:35:55.273535+0000","last_active":"2024-09-13T16:35:55.282279+0000","last_peered":"2024-09-13T16:35:55.282279+0000","last_clean":"2024-09-13T16:35:55.282279+0000","last_became_active":"2024-09-13T16:35:55.273256+0000","last_became_peered":"2024-09-13T16:35:55.273256+0000","last_unstale":"2024-09-13T16:35:55.282279+0000","last_undegraded":"2024-09-13T16:35:55.282279+0000","last_fullsized":"2024-09-13T16:35:55.282279+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:48.360160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.30","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280333+0000","last_change":"2024-09-13T16:35:55.279169+0000","last_active":"2024-09-13T16:35:55.280333+0000","last_peered":"2024-09-13T16:35:55.280333+0000","last_clean":"2024-09-13T16:35:55.280333+0000","last_became_active":"2024-09-13T16:35:55.278953+0000","last_became_peered":"2024-09-13T16:35:55.278953+0000","last_unstale":"2024-09-13T16:35:55.280333+0000","last_undegraded":"2024-09-13T16:35:55.280333+0000","last_fullsized":"2024-09-13T16:35:55.280333+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:37:55.507098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.2f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278085+0000","last_change":"2024-09-13T16:35:55.269537+0000","last_active":"2024-09-13T16:35:55.278085+0000","last_peered":"2024-09-13T16:35:55.278085+0000","last_clean":"2024-09-13T16:35:55.278085+0000","last_became_active":"2024-09-13T16:35:55.269296+0000","last_became_peered":"2024-09-13T16:35:55.269296+0000","last_unstale":"2024-09-13T16:35:55.278085+0000","last_undegraded":"2024-09-13T16:35:55.278085+0000","last_fullsized":"2024-09-13T16:35:55.278085+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:09:06.292463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.2e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.277026+0000","last_change":"2024-09-13T16:35:55.273483+0000","last_active":"2024-09-13T16:35:55.277026+0000","last_peered":"2024-09-13T16:35:55.277026+0000","last_clean":"2024-09-13T16:35:55.277026+0000","last_became_active":"2024-09-13T16:35:55.273241+0000","last_became_peered":"2024-09-13T16:35:55.273241+0000","last_unstale":"2024-09-13T16:35:55.277026+0000","last_undegraded":"2024-09-13T16:35:55.277026+0000","last_fullsized":"2024-09-13T16:35:55.277026+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:05:24.023139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.2d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.286694+0000","last_change":"2024-09-13T16:35:55.283864+0000","last_active":"2024-09-13T16:35:55.286694+0000","last_peered":"2024-09-13T16:35:55.286694+0000","last_clean":"2024-09-13T16:35:55.286694+0000","last_became_active":"2024-09-13T16:35:55.281152+0000","last_became_peered":"2024-09-13T16:35:55.281152+0000","last_unstale":"2024-09-13T16:35:55.286694+0000","last_undegraded":"2024-09-13T16:35:55.286694+0000","last_fullsized":"2024-09-13T16:35:55.286694+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:31:43.636829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.2c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280275+0000","last_change":"2024-09-13T16:35:55.276770+0000","last_active":"2024-09-13T16:35:55.280275+0000","last_peered":"2024-09-13T16:35:55.280275+0000","last_clean":"2024-09-13T16:35:55.280275+0000","last_became_active":"2024-09-13T16:35:55.276498+0000","last_became_peered":"2024-09-13T16:35:55.276498+0000","last_unstale":"2024-09-13T16:35:55.280275+0000","last_undegraded":"2024-09-13T16:35:55.280275+0000","last_fullsized":"2024-09-13T16:35:55.280275+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:18:45.212600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.2b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279925+0000","last_change":"2024-09-13T16:35:55.271943+0000","last_active":"2024-09-13T16:35:55.279925+0000","last_peered":"2024-09-13T16:35:55.279925+0000","last_clean":"2024-09-13T16:35:55.279925+0000","last_became_active":"2024-09-13T16:35:55.271447+0000","last_became_peered":"2024-09-13T16:35:55.271447+0000","last_unstale":"2024-09-13T16:35:55.279925+0000","last_undegraded":"2024-09-13T16:35:55.279925+0000","last_fullsized":"2024-09-13T16:35:55.279925+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:32:08.476350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.2a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284413+0000","last_change":"2024-09-13T16:35:55.279946+0000","last_active":"2024-09-13T16:35:55.284413+0000","last_peered":"2024-09-13T16:35:55.284413+0000","last_clean":"2024-09-13T16:35:55.284413+0000","last_became_active":"2024-09-13T16:35:55.279420+0000","last_became_peered":"2024-09-13T16:35:55.279420+0000","last_unstale":"2024-09-13T16:35:55.284413+0000","last_undegraded":"2024-09-13T16:35:55.284413+0000","last_fullsized":"2024-09-13T16:35:55.284413+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:31:47.356307+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.29","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275001+0000","last_change":"2024-09-13T16:35:55.270766+0000","last_active":"2024-09-13T16:35:55.275001+0000","last_peered":"2024-09-13T16:35:55.275001+0000","last_clean":"2024-09-13T16:35:55.275001+0000","last_became_active":"2024-09-13T16:35:55.264869+0000","last_became_peered":"2024-09-13T16:35:55.264869+0000","last_unstale":"2024-09-13T16:35:55.275001+0000","last_undegraded":"2024-09-13T16:35:55.275001+0000","last_fullsized":"2024-09-13T16:35:55.275001+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:11.766616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.28","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282194+0000","last_change":"2024-09-13T16:35:55.277130+0000","last_active":"2024-09-13T16:35:55.282194+0000","last_peered":"2024-09-13T16:35:55.282194+0000","last_clean":"2024-09-13T16:35:55.282194+0000","last_became_active":"2024-09-13T16:35:55.276503+0000","last_became_peered":"2024-09-13T16:35:55.276503+0000","last_unstale":"2024-09-13T16:35:55.282194+0000","last_undegraded":"2024-09-13T16:35:55.282194+0000","last_fullsized":"2024-09-13T16:35:55.282194+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:59:07.717972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.27","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284132+0000","last_change":"2024-09-13T16:35:55.281497+0000","last_active":"2024-09-13T16:35:55.284132+0000","last_peered":"2024-09-13T16:35:55.284132+0000","last_clean":"2024-09-13T16:35:55.284132+0000","last_became_active":"2024-09-13T16:35:55.281213+0000","last_became_peered":"2024-09-13T16:35:55.281213+0000","last_unstale":"2024-09-13T16:35:55.284132+0000","last_undegraded":"2024-09-13T16:35:55.284132+0000","last_fullsized":"2024-09-13T16:35:55.284132+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:14:39.179995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.26","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280412+0000","last_change":"2024-09-13T16:35:55.275779+0000","last_active":"2024-09-13T16:35:55.280412+0000","last_peered":"2024-09-13T16:35:55.280412+0000","last_clean":"2024-09-13T16:35:55.280412+0000","last_became_active":"2024-09-13T16:35:55.275563+0000","last_became_peered":"2024-09-13T16:35:55.275563+0000","last_unstale":"2024-09-13T16:35:55.280412+0000","last_undegraded":"2024-09-13T16:35:55.280412+0000","last_fullsized":"2024-09-13T16:35:55.280412+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:19:50.333030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.25","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.285121+0000","last_change":"2024-09-13T16:35:55.283609+0000","last_active":"2024-09-13T16:35:55.285121+0000","last_peered":"2024-09-13T16:35:55.285121+0000","last_clean":"2024-09-13T16:35:55.285121+0000","last_became_active":"2024-09-13T16:35:55.283230+0000","last_became_peered":"2024-09-13T16:35:55.283230+0000","last_unstale":"2024-09-13T16:35:55.285121+0000","last_undegraded":"2024-09-13T16:35:55.285121+0000","last_fullsized":"2024-09-13T16:35:55.285121+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:31:58.150363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.24","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275112+0000","last_change":"2024-09-13T16:35:55.273034+0000","last_active":"2024-09-13T16:35:55.275112+0000","last_peered":"2024-09-13T16:35:55.275112+0000","last_clean":"2024-09-13T16:35:55.275112+0000","last_became_active":"2024-09-13T16:35:55.272674+0000","last_became_peered":"2024-09-13T16:35:55.272674+0000","last_unstale":"2024-09-13T16:35:55.275112+0000","last_undegraded":"2024-09-13T16:35:55.275112+0000","last_fullsized":"2024-09-13T16:35:55.275112+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:30:25.410612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.23","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278488+0000","last_change":"2024-09-13T16:35:55.274390+0000","last_active":"2024-09-13T16:35:55.278488+0000","last_peered":"2024-09-13T16:35:55.278488+0000","last_clean":"2024-09-13T16:35:55.278488+0000","last_became_active":"2024-09-13T16:35:55.274153+0000","last_became_peered":"2024-09-13T16:35:55.274153+0000","last_unstale":"2024-09-13T16:35:55.278488+0000","last_undegraded":"2024-09-13T16:35:55.278488+0000","last_fullsized":"2024-09-13T16:35:55.278488+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:58:09.187095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.22","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282245+0000","last_change":"2024-09-13T16:35:55.281148+0000","last_active":"2024-09-13T16:35:55.282245+0000","last_peered":"2024-09-13T16:35:55.282245+0000","last_clean":"2024-09-13T16:35:55.282245+0000","last_became_active":"2024-09-13T16:35:55.280818+0000","last_became_peered":"2024-09-13T16:35:55.280818+0000","last_unstale":"2024-09-13T16:35:55.282245+0000","last_undegraded":"2024-09-13T16:35:55.282245+0000","last_fullsized":"2024-09-13T16:35:55.282245+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:17:12.278199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.21","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279559+0000","last_change":"2024-09-13T16:35:55.270018+0000","last_active":"2024-09-13T16:35:55.279559+0000","last_peered":"2024-09-13T16:35:55.279559+0000","last_clean":"2024-09-13T16:35:55.279559+0000","last_became_active":"2024-09-13T16:35:55.269738+0000","last_became_peered":"2024-09-13T16:35:55.269738+0000","last_unstale":"2024-09-13T16:35:55.279559+0000","last_undegraded":"2024-09-13T16:35:55.279559+0000","last_fullsized":"2024-09-13T16:35:55.279559+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:24:00.636075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.20","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282183+0000","last_change":"2024-09-13T16:35:55.276944+0000","last_active":"2024-09-13T16:35:55.282183+0000","last_peered":"2024-09-13T16:35:55.282183+0000","last_clean":"2024-09-13T16:35:55.282183+0000","last_became_active":"2024-09-13T16:35:55.276043+0000","last_became_peered":"2024-09-13T16:35:55.276043+0000","last_unstale":"2024-09-13T16:35:55.282183+0000","last_undegraded":"2024-09-13T16:35:55.282183+0000","last_fullsized":"2024-09-13T16:35:55.282183+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:37.460203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280444+0000","last_change":"2024-09-13T16:35:55.274402+0000","last_active":"2024-09-13T16:35:55.280444+0000","last_peered":"2024-09-13T16:35:55.280444+0000","last_clean":"2024-09-13T16:35:55.280444+0000","last_became_active":"2024-09-13T16:35:55.274155+0000","last_became_peered":"2024-09-13T16:35:55.274155+0000","last_unstale":"2024-09-13T16:35:55.280444+0000","last_undegraded":"2024-09-13T16:35:55.280444+0000","last_fullsized":"2024-09-13T16:35:55.280444+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:32.967289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1f","version":"125'4","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284799+0000","last_change":"2024-09-13T16:35:54.889433+0000","last_active":"2024-09-13T16:35:55.284799+0000","last_peered":"2024-09-13T16:35:55.284799+0000","last_clean":"2024-09-13T16:35:55.284799+0000","last_became_active":"2024-09-13T16:35:54.889176+0000","last_became_peered":"2024-09-13T16:35:54.889176+0000","last_unstale":"2024-09-13T16:35:55.284799+0000","last_undegraded":"2024-09-13T16:35:55.284799+0000","last_fullsized":"2024-09-13T16:35:55.284799+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:37:00.104258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275062+0000","last_change":"2024-09-13T16:35:55.272130+0000","last_active":"2024-09-13T16:35:55.275062+0000","last_peered":"2024-09-13T16:35:55.275062+0000","last_clean":"2024-09-13T16:35:55.275062+0000","last_became_active":"2024-09-13T16:35:55.271881+0000","last_became_peered":"2024-09-13T16:35:55.271881+0000","last_unstale":"2024-09-13T16:35:55.275062+0000","last_undegraded":"2024-09-13T16:35:55.275062+0000","last_fullsized":"2024-09-13T16:35:55.275062+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:23:35.658877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1e","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.271023+0000","last_change":"2024-09-13T16:35:54.888195+0000","last_active":"2024-09-13T16:35:55.271023+0000","last_peered":"2024-09-13T16:35:55.271023+0000","last_clean":"2024-09-13T16:35:55.271023+0000","last_became_active":"2024-09-13T16:35:54.887939+0000","last_became_peered":"2024-09-13T16:35:54.887939+0000","last_unstale":"2024-09-13T16:35:55.271023+0000","last_undegraded":"2024-09-13T16:35:55.271023+0000","last_fullsized":"2024-09-13T16:35:55.271023+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:43.039403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.1e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282095+0000","last_change":"2024-09-13T16:35:55.273233+0000","last_active":"2024-09-13T16:35:55.282095+0000","last_peered":"2024-09-13T16:35:55.282095+0000","last_clean":"2024-09-13T16:35:55.282095+0000","last_became_active":"2024-09-13T16:35:55.272891+0000","last_became_peered":"2024-09-13T16:35:55.272891+0000","last_unstale":"2024-09-13T16:35:55.282095+0000","last_undegraded":"2024-09-13T16:35:55.282095+0000","last_fullsized":"2024-09-13T16:35:55.282095+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:29:52.424969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275160+0000","last_change":"2024-09-13T16:35:54.891686+0000","last_active":"2024-09-13T16:35:55.275160+0000","last_peered":"2024-09-13T16:35:55.275160+0000","last_clean":"2024-09-13T16:35:55.275160+0000","last_became_active":"2024-09-13T16:35:54.891407+0000","last_became_peered":"2024-09-13T16:35:54.891407+0000","last_unstale":"2024-09-13T16:35:55.275160+0000","last_undegraded":"2024-09-13T16:35:55.275160+0000","last_fullsized":"2024-09-13T16:35:55.275160+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:00.453766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279313+0000","last_change":"2024-09-13T16:35:55.272704+0000","last_active":"2024-09-13T16:35:55.279313+0000","last_peered":"2024-09-13T16:35:55.279313+0000","last_clean":"2024-09-13T16:35:55.279313+0000","last_became_active":"2024-09-13T16:35:55.272451+0000","last_became_peered":"2024-09-13T16:35:55.272451+0000","last_unstale":"2024-09-13T16:35:55.279313+0000","last_undegraded":"2024-09-13T16:35:55.279313+0000","last_fullsized":"2024-09-13T16:35:55.279313+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:44:44.312036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.271605+0000","last_change":"2024-09-13T16:35:54.888401+0000","last_active":"2024-09-13T16:35:55.271605+0000","last_peered":"2024-09-13T16:35:55.271605+0000","last_clean":"2024-09-13T16:35:55.271605+0000","last_became_active":"2024-09-13T16:35:54.888092+0000","last_became_peered":"2024-09-13T16:35:54.888092+0000","last_unstale":"2024-09-13T16:35:55.271605+0000","last_undegraded":"2024-09-13T16:35:55.271605+0000","last_fullsized":"2024-09-13T16:35:55.271605+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:13.571296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280251+0000","last_change":"2024-09-13T16:35:55.276918+0000","last_active":"2024-09-13T16:35:55.280251+0000","last_peered":"2024-09-13T16:35:55.280251+0000","last_clean":"2024-09-13T16:35:55.280251+0000","last_became_active":"2024-09-13T16:35:55.276687+0000","last_became_peered":"2024-09-13T16:35:55.276687+0000","last_unstale":"2024-09-13T16:35:55.280251+0000","last_undegraded":"2024-09-13T16:35:55.280251+0000","last_fullsized":"2024-09-13T16:35:55.280251+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:57:01.463116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1b","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279857+0000","last_change":"2024-09-13T16:35:54.887074+0000","last_active":"2024-09-13T16:35:55.279857+0000","last_peered":"2024-09-13T16:35:55.279857+0000","last_clean":"2024-09-13T16:35:55.279857+0000","last_became_active":"2024-09-13T16:35:54.886669+0000","last_became_peered":"2024-09-13T16:35:54.886669+0000","last_unstale":"2024-09-13T16:35:55.279857+0000","last_undegraded":"2024-09-13T16:35:55.279857+0000","last_fullsized":"2024-09-13T16:35:55.279857+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:34.524636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284218+0000","last_change":"2024-09-13T16:35:55.279187+0000","last_active":"2024-09-13T16:35:55.284218+0000","last_peered":"2024-09-13T16:35:55.284218+0000","last_clean":"2024-09-13T16:35:55.284218+0000","last_became_active":"2024-09-13T16:35:55.278844+0000","last_became_peered":"2024-09-13T16:35:55.278844+0000","last_unstale":"2024-09-13T16:35:55.284218+0000","last_undegraded":"2024-09-13T16:35:55.284218+0000","last_fullsized":"2024-09-13T16:35:55.284218+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:28:11.766616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.274527+0000","last_change":"2024-09-13T16:35:54.891182+0000","last_active":"2024-09-13T16:35:55.274527+0000","last_peered":"2024-09-13T16:35:55.274527+0000","last_clean":"2024-09-13T16:35:55.274527+0000","last_became_active":"2024-09-13T16:35:54.890865+0000","last_became_peered":"2024-09-13T16:35:54.890865+0000","last_unstale":"2024-09-13T16:35:55.274527+0000","last_undegraded":"2024-09-13T16:35:55.274527+0000","last_fullsized":"2024-09-13T16:35:55.274527+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:42.872900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.1a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278604+0000","last_change":"2024-09-13T16:35:55.275612+0000","last_active":"2024-09-13T16:35:55.278604+0000","last_peered":"2024-09-13T16:35:55.278604+0000","last_clean":"2024-09-13T16:35:55.278604+0000","last_became_active":"2024-09-13T16:35:55.275344+0000","last_became_peered":"2024-09-13T16:35:55.275344+0000","last_unstale":"2024-09-13T16:35:55.278604+0000","last_undegraded":"2024-09-13T16:35:55.278604+0000","last_fullsized":"2024-09-13T16:35:55.278604+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:58:08.105328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.254216+0000","last_change":"2024-09-13T16:35:54.885936+0000","last_active":"2024-09-13T16:35:55.254216+0000","last_peered":"2024-09-13T16:35:55.254216+0000","last_clean":"2024-09-13T16:35:55.254216+0000","last_became_active":"2024-09-13T16:35:54.885422+0000","last_became_peered":"2024-09-13T16:35:54.885422+0000","last_unstale":"2024-09-13T16:35:55.254216+0000","last_undegraded":"2024-09-13T16:35:55.254216+0000","last_fullsized":"2024-09-13T16:35:55.254216+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:32:59.841074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.1b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278230+0000","last_change":"2024-09-13T16:35:55.270876+0000","last_active":"2024-09-13T16:35:55.278230+0000","last_peered":"2024-09-13T16:35:55.278230+0000","last_clean":"2024-09-13T16:35:55.278230+0000","last_became_active":"2024-09-13T16:35:55.270324+0000","last_became_peered":"2024-09-13T16:35:55.270324+0000","last_unstale":"2024-09-13T16:35:55.278230+0000","last_undegraded":"2024-09-13T16:35:55.278230+0000","last_fullsized":"2024-09-13T16:35:55.278230+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:02:39.323231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.266012+0000","last_change":"2024-09-13T16:35:54.884585+0000","last_active":"2024-09-13T16:35:55.266012+0000","last_peered":"2024-09-13T16:35:55.266012+0000","last_clean":"2024-09-13T16:35:55.266012+0000","last_became_active":"2024-09-13T16:35:54.884265+0000","last_became_peered":"2024-09-13T16:35:54.884265+0000","last_unstale":"2024-09-13T16:35:55.266012+0000","last_undegraded":"2024-09-13T16:35:55.266012+0000","last_fullsized":"2024-09-13T16:35:55.266012+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:43:57.954972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.14","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284085+0000","last_change":"2024-09-13T16:35:55.279229+0000","last_active":"2024-09-13T16:35:55.284085+0000","last_peered":"2024-09-13T16:35:55.284085+0000","last_clean":"2024-09-13T16:35:55.284085+0000","last_became_active":"2024-09-13T16:35:55.275486+0000","last_became_peered":"2024-09-13T16:35:55.275486+0000","last_unstale":"2024-09-13T16:35:55.284085+0000","last_undegraded":"2024-09-13T16:35:55.284085+0000","last_fullsized":"2024-09-13T16:35:55.284085+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:18:03.079664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.267483+0000","last_change":"2024-09-13T16:35:54.892587+0000","last_active":"2024-09-13T16:35:55.267483+0000","last_peered":"2024-09-13T16:35:55.267483+0000","last_clean":"2024-09-13T16:35:55.267483+0000","last_became_active":"2024-09-13T16:35:54.892278+0000","last_became_peered":"2024-09-13T16:35:54.892278+0000","last_unstale":"2024-09-13T16:35:55.267483+0000","last_undegraded":"2024-09-13T16:35:55.267483+0000","last_fullsized":"2024-09-13T16:35:55.267483+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:54.111959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.15","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.268876+0000","last_change":"2024-09-13T16:35:55.263965+0000","last_active":"2024-09-13T16:35:55.268876+0000","last_peered":"2024-09-13T16:35:55.268876+0000","last_clean":"2024-09-13T16:35:55.268876+0000","last_became_active":"2024-09-13T16:35:55.263689+0000","last_became_peered":"2024-09-13T16:35:55.263689+0000","last_unstale":"2024-09-13T16:35:55.268876+0000","last_undegraded":"2024-09-13T16:35:55.268876+0000","last_fullsized":"2024-09-13T16:35:55.268876+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:48:39.844424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.255803+0000","last_change":"2024-09-13T16:35:54.884085+0000","last_active":"2024-09-13T16:35:55.255803+0000","last_peered":"2024-09-13T16:35:55.255803+0000","last_clean":"2024-09-13T16:35:55.255803+0000","last_became_active":"2024-09-13T16:35:54.883730+0000","last_became_peered":"2024-09-13T16:35:54.883730+0000","last_unstale":"2024-09-13T16:35:55.255803+0000","last_undegraded":"2024-09-13T16:35:55.255803+0000","last_fullsized":"2024-09-13T16:35:55.255803+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:23:26.541011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280118+0000","last_change":"2024-09-13T16:35:55.277876+0000","last_active":"2024-09-13T16:35:55.280118+0000","last_peered":"2024-09-13T16:35:55.280118+0000","last_clean":"2024-09-13T16:35:55.280118+0000","last_became_active":"2024-09-13T16:35:55.277620+0000","last_became_peered":"2024-09-13T16:35:55.277620+0000","last_unstale":"2024-09-13T16:35:55.280118+0000","last_undegraded":"2024-09-13T16:35:55.280118+0000","last_fullsized":"2024-09-13T16:35:55.280118+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:12.923144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.15","version":"125'3","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276917+0000","last_change":"2024-09-13T16:35:54.886486+0000","last_active":"2024-09-13T16:35:55.276917+0000","last_peered":"2024-09-13T16:35:55.276917+0000","last_clean":"2024-09-13T16:35:55.276917+0000","last_became_active":"2024-09-13T16:35:54.886191+0000","last_became_peered":"2024-09-13T16:35:54.886191+0000","last_unstale":"2024-09-13T16:35:55.276917+0000","last_undegraded":"2024-09-13T16:35:55.276917+0000","last_fullsized":"2024-09-13T16:35:55.276917+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:50.494802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281614+0000","last_change":"2024-09-13T16:35:55.270334+0000","last_active":"2024-09-13T16:35:55.281614+0000","last_peered":"2024-09-13T16:35:55.281614+0000","last_clean":"2024-09-13T16:35:55.281614+0000","last_became_active":"2024-09-13T16:35:55.269786+0000","last_became_peered":"2024-09-13T16:35:55.269786+0000","last_unstale":"2024-09-13T16:35:55.281614+0000","last_undegraded":"2024-09-13T16:35:55.281614+0000","last_fullsized":"2024-09-13T16:35:55.281614+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:33.253752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.14","version":"125'6","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.287478+0000","last_change":"2024-09-13T16:35:54.889719+0000","last_active":"2024-09-13T16:35:55.287478+0000","last_peered":"2024-09-13T16:35:55.287478+0000","last_clean":"2024-09-13T16:35:55.287478+0000","last_became_active":"2024-09-13T16:35:54.889460+0000","last_became_peered":"2024-09-13T16:35:54.889460+0000","last_unstale":"2024-09-13T16:35:55.287478+0000","last_undegraded":"2024-09-13T16:35:55.287478+0000","last_fullsized":"2024-09-13T16:35:55.287478+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:03:17.919296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1564,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.10","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284954+0000","last_change":"2024-09-13T16:35:55.280602+0000","last_active":"2024-09-13T16:35:55.284954+0000","last_peered":"2024-09-13T16:35:55.284954+0000","last_clean":"2024-09-13T16:35:55.284954+0000","last_became_active":"2024-09-13T16:35:55.280267+0000","last_became_peered":"2024-09-13T16:35:55.280267+0000","last_unstale":"2024-09-13T16:35:55.284954+0000","last_undegraded":"2024-09-13T16:35:55.284954+0000","last_fullsized":"2024-09-13T16:35:55.284954+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:50:52.746769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.13","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276182+0000","last_change":"2024-09-13T16:35:54.887642+0000","last_active":"2024-09-13T16:35:55.276182+0000","last_peered":"2024-09-13T16:35:55.276182+0000","last_clean":"2024-09-13T16:35:55.276182+0000","last_became_active":"2024-09-13T16:35:54.886949+0000","last_became_peered":"2024-09-13T16:35:54.886949+0000","last_unstale":"2024-09-13T16:35:55.276182+0000","last_undegraded":"2024-09-13T16:35:55.276182+0000","last_fullsized":"2024-09-13T16:35:55.276182+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:53:15.061384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278043+0000","last_change":"2024-09-13T16:35:55.272281+0000","last_active":"2024-09-13T16:35:55.278043+0000","last_peered":"2024-09-13T16:35:55.278043+0000","last_clean":"2024-09-13T16:35:55.278043+0000","last_became_active":"2024-09-13T16:35:55.272007+0000","last_became_peered":"2024-09-13T16:35:55.272007+0000","last_unstale":"2024-09-13T16:35:55.278043+0000","last_undegraded":"2024-09-13T16:35:55.278043+0000","last_fullsized":"2024-09-13T16:35:55.278043+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:35:08.309452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.264841+0000","last_change":"2024-09-13T16:35:54.888765+0000","last_active":"2024-09-13T16:35:55.264841+0000","last_peered":"2024-09-13T16:35:55.264841+0000","last_clean":"2024-09-13T16:35:55.264841+0000","last_became_active":"2024-09-13T16:35:54.888497+0000","last_became_peered":"2024-09-13T16:35:54.888497+0000","last_unstale":"2024-09-13T16:35:55.264841+0000","last_undegraded":"2024-09-13T16:35:55.264841+0000","last_fullsized":"2024-09-13T16:35:55.264841+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:28:01.009326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281556+0000","last_change":"2024-09-13T16:35:55.281036+0000","last_active":"2024-09-13T16:35:55.281556+0000","last_peered":"2024-09-13T16:35:55.281556+0000","last_clean":"2024-09-13T16:35:55.281556+0000","last_became_active":"2024-09-13T16:35:55.280804+0000","last_became_peered":"2024-09-13T16:35:55.280804+0000","last_unstale":"2024-09-13T16:35:55.281556+0000","last_undegraded":"2024-09-13T16:35:55.281556+0000","last_fullsized":"2024-09-13T16:35:55.281556+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T03:23:27.550361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.261131+0000","last_change":"2024-09-13T16:35:54.890278+0000","last_active":"2024-09-13T16:35:55.261131+0000","last_peered":"2024-09-13T16:35:55.261131+0000","last_clean":"2024-09-13T16:35:55.261131+0000","last_became_active":"2024-09-13T16:35:54.889876+0000","last_became_peered":"2024-09-13T16:35:54.889876+0000","last_unstale":"2024-09-13T16:35:55.261131+0000","last_undegraded":"2024-09-13T16:35:55.261131+0000","last_fullsized":"2024-09-13T16:35:55.261131+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:01:12.962907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282151+0000","last_change":"2024-09-13T16:35:55.273223+0000","last_active":"2024-09-13T16:35:55.282151+0000","last_peered":"2024-09-13T16:35:55.282151+0000","last_clean":"2024-09-13T16:35:55.282151+0000","last_became_active":"2024-09-13T16:35:55.272802+0000","last_became_peered":"2024-09-13T16:35:55.272802+0000","last_unstale":"2024-09-13T16:35:55.282151+0000","last_undegraded":"2024-09-13T16:35:55.282151+0000","last_fullsized":"2024-09-13T16:35:55.282151+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:22:37.143833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.f","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.273910+0000","last_change":"2024-09-13T16:35:54.890074+0000","last_active":"2024-09-13T16:35:55.273910+0000","last_peered":"2024-09-13T16:35:55.273910+0000","last_clean":"2024-09-13T16:35:55.273910+0000","last_became_active":"2024-09-13T16:35:54.889832+0000","last_became_peered":"2024-09-13T16:35:54.889832+0000","last_unstale":"2024-09-13T16:35:55.273910+0000","last_undegraded":"2024-09-13T16:35:55.273910+0000","last_fullsized":"2024-09-13T16:35:55.273910+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:05:54.111959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.d","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282322+0000","last_change":"2024-09-13T16:35:55.273977+0000","last_active":"2024-09-13T16:35:55.282322+0000","last_peered":"2024-09-13T16:35:55.282322+0000","last_clean":"2024-09-13T16:35:55.282322+0000","last_became_active":"2024-09-13T16:35:55.273737+0000","last_became_peered":"2024-09-13T16:35:55.273737+0000","last_unstale":"2024-09-13T16:35:55.282322+0000","last_undegraded":"2024-09-13T16:35:55.282322+0000","last_fullsized":"2024-09-13T16:35:55.282322+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:50.586289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276400+0000","last_change":"2024-09-13T16:35:54.884544+0000","last_active":"2024-09-13T16:35:55.276400+0000","last_peered":"2024-09-13T16:35:55.276400+0000","last_clean":"2024-09-13T16:35:55.276400+0000","last_became_active":"2024-09-13T16:35:54.884221+0000","last_became_peered":"2024-09-13T16:35:54.884221+0000","last_unstale":"2024-09-13T16:35:55.276400+0000","last_undegraded":"2024-09-13T16:35:55.276400+0000","last_fullsized":"2024-09-13T16:35:55.276400+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T21:06:21.497053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.e","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278250+0000","last_change":"2024-09-13T16:35:55.276103+0000","last_active":"2024-09-13T16:35:55.278250+0000","last_peered":"2024-09-13T16:35:55.278250+0000","last_clean":"2024-09-13T16:35:55.278250+0000","last_became_active":"2024-09-13T16:35:55.275862+0000","last_became_peered":"2024-09-13T16:35:55.275862+0000","last_unstale":"2024-09-13T16:35:55.278250+0000","last_undegraded":"2024-09-13T16:35:55.278250+0000","last_fullsized":"2024-09-13T16:35:55.278250+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:10:55.377221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.d","version":"125'1","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.273800+0000","last_change":"2024-09-13T16:35:54.888841+0000","last_active":"2024-09-13T16:35:55.273800+0000","last_peered":"2024-09-13T16:35:55.273800+0000","last_clean":"2024-09-13T16:35:55.273800+0000","last_became_active":"2024-09-13T16:35:54.888567+0000","last_became_peered":"2024-09-13T16:35:54.888567+0000","last_unstale":"2024-09-13T16:35:55.273800+0000","last_undegraded":"2024-09-13T16:35:55.273800+0000","last_fullsized":"2024-09-13T16:35:55.273800+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:51.467281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.f","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282275+0000","last_change":"2024-09-13T16:35:55.277459+0000","last_active":"2024-09-13T16:35:55.282275+0000","last_peered":"2024-09-13T16:35:55.282275+0000","last_clean":"2024-09-13T16:35:55.282275+0000","last_became_active":"2024-09-13T16:35:55.276362+0000","last_became_peered":"2024-09-13T16:35:55.276362+0000","last_unstale":"2024-09-13T16:35:55.282275+0000","last_undegraded":"2024-09-13T16:35:55.282275+0000","last_fullsized":"2024-09-13T16:35:55.282275+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:43:43.278864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.258193+0000","last_change":"2024-09-13T16:35:54.890899+0000","last_active":"2024-09-13T16:35:55.258193+0000","last_peered":"2024-09-13T16:35:55.258193+0000","last_clean":"2024-09-13T16:35:55.258193+0000","last_became_active":"2024-09-13T16:35:54.890626+0000","last_became_peered":"2024-09-13T16:35:54.890626+0000","last_unstale":"2024-09-13T16:35:55.258193+0000","last_undegraded":"2024-09-13T16:35:55.258193+0000","last_fullsized":"2024-09-13T16:35:55.258193+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:35:51.563617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282901+0000","last_change":"2024-09-13T16:35:55.282369+0000","last_active":"2024-09-13T16:35:55.282901+0000","last_peered":"2024-09-13T16:35:55.282901+0000","last_clean":"2024-09-13T16:35:55.282901+0000","last_became_active":"2024-09-13T16:35:55.282129+0000","last_became_peered":"2024-09-13T16:35:55.282129+0000","last_unstale":"2024-09-13T16:35:55.282901+0000","last_undegraded":"2024-09-13T16:35:55.282901+0000","last_fullsized":"2024-09-13T16:35:55.282901+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:17:23.167482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.276074+0000","last_change":"2024-09-13T16:35:54.887365+0000","last_active":"2024-09-13T16:35:55.276074+0000","last_peered":"2024-09-13T16:35:55.276074+0000","last_clean":"2024-09-13T16:35:55.276074+0000","last_became_active":"2024-09-13T16:35:54.886927+0000","last_became_peered":"2024-09-13T16:35:54.886927+0000","last_unstale":"2024-09-13T16:35:55.276074+0000","last_undegraded":"2024-09-13T16:35:55.276074+0000","last_fullsized":"2024-09-13T16:35:55.276074+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:48:05.458741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.273434+0000","last_change":"2024-09-13T16:35:55.268501+0000","last_active":"2024-09-13T16:35:55.273434+0000","last_peered":"2024-09-13T16:35:55.273434+0000","last_clean":"2024-09-13T16:35:55.273434+0000","last_became_active":"2024-09-13T16:35:55.268261+0000","last_became_peered":"2024-09-13T16:35:55.268261+0000","last_unstale":"2024-09-13T16:35:55.273434+0000","last_undegraded":"2024-09-13T16:35:55.273434+0000","last_fullsized":"2024-09-13T16:35:55.273434+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:11.454742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.257972+0000","last_change":"2024-09-13T16:35:54.889084+0000","last_active":"2024-09-13T16:35:55.257972+0000","last_peered":"2024-09-13T16:35:55.257972+0000","last_clean":"2024-09-13T16:35:55.257972+0000","last_became_active":"2024-09-13T16:35:54.888765+0000","last_became_peered":"2024-09-13T16:35:54.888765+0000","last_unstale":"2024-09-13T16:35:55.257972+0000","last_undegraded":"2024-09-13T16:35:55.257972+0000","last_fullsized":"2024-09-13T16:35:55.257972+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:14:12.359450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280100+0000","last_change":"2024-09-13T16:35:55.278232+0000","last_active":"2024-09-13T16:35:55.280100+0000","last_peered":"2024-09-13T16:35:55.280100+0000","last_clean":"2024-09-13T16:35:55.280100+0000","last_became_active":"2024-09-13T16:35:55.277525+0000","last_became_peered":"2024-09-13T16:35:55.277525+0000","last_unstale":"2024-09-13T16:35:55.280100+0000","last_undegraded":"2024-09-13T16:35:55.280100+0000","last_fullsized":"2024-09-13T16:35:55.280100+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:22:01.605412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.261408+0000","last_change":"2024-09-13T16:35:54.890110+0000","last_active":"2024-09-13T16:35:55.261408+0000","last_peered":"2024-09-13T16:35:55.261408+0000","last_clean":"2024-09-13T16:35:55.261408+0000","last_became_active":"2024-09-13T16:35:54.889149+0000","last_became_peered":"2024-09-13T16:35:54.889149+0000","last_unstale":"2024-09-13T16:35:55.261408+0000","last_undegraded":"2024-09-13T16:35:55.261408+0000","last_fullsized":"2024-09-13T16:35:55.261408+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:19:59.524539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.b","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.282359+0000","last_change":"2024-09-13T16:35:55.277317+0000","last_active":"2024-09-13T16:35:55.282359+0000","last_peered":"2024-09-13T16:35:55.282359+0000","last_clean":"2024-09-13T16:35:55.282359+0000","last_became_active":"2024-09-13T16:35:55.276491+0000","last_became_peered":"2024-09-13T16:35:55.276491+0000","last_unstale":"2024-09-13T16:35:55.282359+0000","last_undegraded":"2024-09-13T16:35:55.282359+0000","last_fullsized":"2024-09-13T16:35:55.282359+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:33:43.626192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.257841+0000","last_change":"2024-09-13T16:35:54.886698+0000","last_active":"2024-09-13T16:35:55.257841+0000","last_peered":"2024-09-13T16:35:55.257841+0000","last_clean":"2024-09-13T16:35:55.257841+0000","last_became_active":"2024-09-13T16:35:54.886421+0000","last_became_peered":"2024-09-13T16:35:54.886421+0000","last_unstale":"2024-09-13T16:35:55.257841+0000","last_undegraded":"2024-09-13T16:35:55.257841+0000","last_fullsized":"2024-09-13T16:35:55.257841+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:48:31.957939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.275096+0000","last_change":"2024-09-13T16:35:55.271988+0000","last_active":"2024-09-13T16:35:55.275096+0000","last_peered":"2024-09-13T16:35:55.275096+0000","last_clean":"2024-09-13T16:35:55.275096+0000","last_became_active":"2024-09-13T16:35:55.271718+0000","last_became_peered":"2024-09-13T16:35:55.271718+0000","last_unstale":"2024-09-13T16:35:55.275096+0000","last_undegraded":"2024-09-13T16:35:55.275096+0000","last_fullsized":"2024-09-13T16:35:55.275096+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T02:59:54.573695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.7","version":"125'2","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281181+0000","last_change":"2024-09-13T16:35:54.881475+0000","last_active":"2024-09-13T16:35:55.281181+0000","last_peered":"2024-09-13T16:35:55.281181+0000","last_clean":"2024-09-13T16:35:55.281181+0000","last_became_active":"2024-09-13T16:35:54.881201+0000","last_became_peered":"2024-09-13T16:35:54.881201+0000","last_unstale":"2024-09-13T16:35:55.281181+0000","last_undegraded":"2024-09-13T16:35:55.281181+0000","last_fullsized":"2024-09-13T16:35:55.281181+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:48:10.445392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.280217+0000","last_change":"2024-09-13T16:35:55.276287+0000","last_active":"2024-09-13T16:35:55.280217+0000","last_peered":"2024-09-13T16:35:55.280217+0000","last_clean":"2024-09-13T16:35:55.280217+0000","last_became_active":"2024-09-13T16:35:55.275984+0000","last_became_peered":"2024-09-13T16:35:55.275984+0000","last_unstale":"2024-09-13T16:35:55.280217+0000","last_undegraded":"2024-09-13T16:35:55.280217+0000","last_fullsized":"2024-09-13T16:35:55.280217+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:53:38.801815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.264589+0000","last_change":"2024-09-13T16:35:54.890898+0000","last_active":"2024-09-13T16:35:55.264589+0000","last_peered":"2024-09-13T16:35:55.264589+0000","last_clean":"2024-09-13T16:35:55.264589+0000","last_became_active":"2024-09-13T16:35:54.890640+0000","last_became_peered":"2024-09-13T16:35:54.890640+0000","last_unstale":"2024-09-13T16:35:55.264589+0000","last_undegraded":"2024-09-13T16:35:55.264589+0000","last_fullsized":"2024-09-13T16:35:55.264589+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:43.039403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.284204+0000","last_change":"2024-09-13T16:35:55.281012+0000","last_active":"2024-09-13T16:35:55.284204+0000","last_peered":"2024-09-13T16:35:55.284204+0000","last_clean":"2024-09-13T16:35:55.284204+0000","last_became_active":"2024-09-13T16:35:55.280756+0000","last_became_peered":"2024-09-13T16:35:55.280756+0000","last_unstale":"2024-09-13T16:35:55.284204+0000","last_undegraded":"2024-09-13T16:35:55.284204+0000","last_fullsized":"2024-09-13T16:35:55.284204+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:59:48.360160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.270405+0000","last_change":"2024-09-13T16:35:54.892603+0000","last_active":"2024-09-13T16:35:55.270405+0000","last_peered":"2024-09-13T16:35:55.270405+0000","last_clean":"2024-09-13T16:35:55.270405+0000","last_became_active":"2024-09-13T16:35:54.892255+0000","last_became_peered":"2024-09-13T16:35:54.892255+0000","last_unstale":"2024-09-13T16:35:55.270405+0000","last_undegraded":"2024-09-13T16:35:55.270405+0000","last_fullsized":"2024-09-13T16:35:55.270405+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T20:35:13.571296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.7","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.272623+0000","last_change":"2024-09-13T16:35:55.270232+0000","last_active":"2024-09-13T16:35:55.272623+0000","last_peered":"2024-09-13T16:35:55.272623+0000","last_clean":"2024-09-13T16:35:55.272623+0000","last_became_active":"2024-09-13T16:35:55.269974+0000","last_became_peered":"2024-09-13T16:35:55.269974+0000","last_unstale":"2024-09-13T16:35:55.272623+0000","last_undegraded":"2024-09-13T16:35:55.272623+0000","last_fullsized":"2024-09-13T16:35:55.272623+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:40:51.504152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.260016+0000","last_change":"2024-09-13T16:35:54.888833+0000","last_active":"2024-09-13T16:35:55.260016+0000","last_peered":"2024-09-13T16:35:55.260016+0000","last_clean":"2024-09-13T16:35:55.260016+0000","last_became_active":"2024-09-13T16:35:54.888568+0000","last_became_peered":"2024-09-13T16:35:54.888568+0000","last_unstale":"2024-09-13T16:35:55.260016+0000","last_undegraded":"2024-09-13T16:35:55.260016+0000","last_fullsized":"2024-09-13T16:35:55.260016+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:43:07.508642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.281545+0000","last_change":"2024-09-13T16:35:55.280416+0000","last_active":"2024-09-13T16:35:55.281545+0000","last_peered":"2024-09-13T16:35:55.281545+0000","last_clean":"2024-09-13T16:35:55.281545+0000","last_became_active":"2024-09-13T16:35:55.280169+0000","last_became_peered":"2024-09-13T16:35:55.280169+0000","last_unstale":"2024-09-13T16:35:55.281545+0000","last_undegraded":"2024-09-13T16:35:55.281545+0000","last_fullsized":"2024-09-13T16:35:55.281545+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:14:54.455858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.267560+0000","last_change":"2024-09-13T16:35:54.883820+0000","last_active":"2024-09-13T16:35:55.267560+0000","last_peered":"2024-09-13T16:35:55.267560+0000","last_clean":"2024-09-13T16:35:55.267560+0000","last_became_active":"2024-09-13T16:35:54.883505+0000","last_became_peered":"2024-09-13T16:35:54.883505+0000","last_unstale":"2024-09-13T16:35:55.267560+0000","last_undegraded":"2024-09-13T16:35:55.267560+0000","last_fullsized":"2024-09-13T16:35:55.267560+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T23:11:02.870378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.274190+0000","last_change":"2024-09-13T16:35:55.270760+0000","last_active":"2024-09-13T16:35:55.274190+0000","last_peered":"2024-09-13T16:35:55.274190+0000","last_clean":"2024-09-13T16:35:55.274190+0000","last_became_active":"2024-09-13T16:35:55.270503+0000","last_became_peered":"2024-09-13T16:35:55.270503+0000","last_unstale":"2024-09-13T16:35:55.274190+0000","last_undegraded":"2024-09-13T16:35:55.274190+0000","last_fullsized":"2024-09-13T16:35:55.274190+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:41:43.882250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.267624+0000","last_change":"2024-09-13T16:35:54.877539+0000","last_active":"2024-09-13T16:35:55.267624+0000","last_peered":"2024-09-13T16:35:55.267624+0000","last_clean":"2024-09-13T16:35:55.267624+0000","last_became_active":"2024-09-13T16:35:54.877243+0000","last_became_peered":"2024-09-13T16:35:54.877243+0000","last_unstale":"2024-09-13T16:35:55.267624+0000","last_undegraded":"2024-09-13T16:35:55.267624+0000","last_fullsized":"2024-09-13T16:35:55.267624+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T16:56:09.661131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279805+0000","last_change":"2024-09-13T16:35:55.275460+0000","last_active":"2024-09-13T16:35:55.279805+0000","last_peered":"2024-09-13T16:35:55.279805+0000","last_clean":"2024-09-13T16:35:55.279805+0000","last_became_active":"2024-09-13T16:35:55.275191+0000","last_became_peered":"2024-09-13T16:35:55.275191+0000","last_unstale":"2024-09-13T16:35:55.279805+0000","last_undegraded":"2024-09-13T16:35:55.279805+0000","last_fullsized":"2024-09-13T16:35:55.279805+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T00:55:37.136914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.258845+0000","last_change":"2024-09-13T16:35:54.887053+0000","last_active":"2024-09-13T16:35:55.258845+0000","last_peered":"2024-09-13T16:35:55.258845+0000","last_clean":"2024-09-13T16:35:55.258845+0000","last_became_active":"2024-09-13T16:35:54.886576+0000","last_became_peered":"2024-09-13T16:35:54.886576+0000","last_unstale":"2024-09-13T16:35:55.258845+0000","last_undegraded":"2024-09-13T16:35:55.258845+0000","last_fullsized":"2024-09-13T16:35:55.258845+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T04:00:38.599113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.278211+0000","last_change":"2024-09-13T16:35:55.270698+0000","last_active":"2024-09-13T16:35:55.278211+0000","last_peered":"2024-09-13T16:35:55.278211+0000","last_clean":"2024-09-13T16:35:55.278211+0000","last_became_active":"2024-09-13T16:35:55.270046+0000","last_became_peered":"2024-09-13T16:35:55.270046+0000","last_unstale":"2024-09-13T16:35:55.278211+0000","last_undegraded":"2024-09-13T16:35:55.278211+0000","last_fullsized":"2024-09-13T16:35:55.278211+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T17:37:53.391689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":10,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.257942+0000","last_change":"2024-09-13T16:35:54.888362+0000","last_active":"2024-09-13T16:35:55.257942+0000","last_peered":"2024-09-13T16:35:55.257942+0000","last_clean":"2024-09-13T16:35:55.257942+0000","last_became_active":"2024-09-13T16:35:54.888090+0000","last_became_peered":"2024-09-13T16:35:54.888090+0000","last_unstale":"2024-09-13T16:35:55.257942+0000","last_undegraded":"2024-09-13T16:35:55.257942+0000","last_fullsized":"2024-09-13T16:35:55.257942+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:11:10.760716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"0'0","reported_seq":9,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.279945+0000","last_change":"2024-09-13T16:35:55.272268+0000","last_active":"2024-09-13T16:35:55.279945+0000","last_peered":"2024-09-13T16:35:55.279945+0000","last_clean":"2024-09-13T16:35:55.279945+0000","last_became_active":"2024-09-13T16:35:55.271334+0000","last_became_peered":"2024-09-13T16:35:55.271334+0000","last_unstale":"2024-09-13T16:35:55.279945+0000","last_undegraded":"2024-09-13T16:35:55.279945+0000","last_fullsized":"2024-09-13T16:35:55.279945+0000","mapping_epoch":124,"log_start":"0'0","ondisk_log_start":"0'0","created":124,"last_epoch_clean":125,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:54.860076+0000","last_clean_scrub_stamp":"2024-09-13T16:35:54.860076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-15T01:31:47.285453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.10","version":"125'8","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2024-09-13T16:35:55.277694+0000","last_change":"2024-09-13T16:35:54.887314+0000","last_active":"2024-09-13T16:35:55.277694+0000","last_peered":"2024-09-13T16:35:55.277694+0000","last_clean":"2024-09-13T16:35:55.277694+0000","last_became_active":"2024-09-13T16:35:54.886972+0000","last_became_peered":"2024-09-13T16:35:54.886972+0000","last_unstale":"2024-09-13T16:35:55.277694+0000","last_undegraded":"2024-09-13T16:35:55.277694+0000","last_fullsized":"2024-09-13T16:35:55.277694+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:35:53.850726+0000","last_clean_scrub_stamp":"2024-09-13T16:35:53.850726+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T19:54:49.576939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"17'117","reported_seq":257,"reported_epoch":124,"state":"active+clean","last_fresh":"2024-09-13T16:35:54.881450+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:35:54.881450+0000","last_peered":"2024-09-13T16:35:54.881450+0000","last_clean":"2024-09-13T16:35:54.881450+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:35:54.881450+0000","last_undegraded":"2024-09-13T16:35:54.881450+0000","last_fullsized":"2024-09-13T16:35:54.881450+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":18,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":2362,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33204,"kb_used_data":2084,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338636,"statfs":{"total":96636764160,"available":96602763264,"internally_reserved":0,"allocated":2134016,"data_stored":1301067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81799999999999995,"15min":0.84199999999999997},"min":{"1min":0.47799999999999998,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.262,"5min":1.262,"15min":1.357},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.85299999999999998,"15min":0.85299999999999998},"min":{"1min":0.629,"5min":0.52000000000000002,"15min":0.47699999999999998},"max":{"1min":1.214,"5min":1.214,"15min":1.3460000000000001},"last":0.79300000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84899999999999998},"min":{"1min":0.56699999999999995,"5min":0.56000000000000005,"15min":0.501},"max":{"1min":1.1619999999999999,"5min":1.212,"15min":1.569},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.88200000000000001,"15min":0.86299999999999999},"min":{"1min":0.623,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.1719999999999999,"5min":1.2290000000000001,"15min":1.4219999999999999},"last":0.69999999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.90900000000000003,"5min":0.86099999999999999,"15min":0.876},"min":{"1min":0.70799999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.111,"5min":1.2330000000000001,"15min":1.242},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.88400000000000001,"5min":0.88600000000000001,"15min":0.86299999999999999},"min":{"1min":0.67300000000000004,"5min":0.60499999999999998,"15min":0.53900000000000003},"max":{"1min":1.24,"5min":1.24,"15min":1.339},"last":0.97399999999999998}]},{"osd":3,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91000000000000003,"15min":0.96599999999999997},"min":{"1min":0.67400000000000004,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.202,"5min":1.202,"15min":1.476},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.006,"5min":0.999,"15min":1.002},"min":{"1min":0.76200000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.3939999999999999},"last":1.0600000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.85999999999999999,"15min":0.871},"min":{"1min":0.60699999999999998,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.214,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.92800000000000005},{"interface":"front","average":{"1min":0.96199999999999997,"5min":0.94199999999999995,"15min":1.014},"min":{"1min":0.65100000000000002,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.3720000000000001,"5min":1.381,"15min":1.589},"last":1.1140000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.98499999999999999,"15min":0.99399999999999999},"min":{"1min":0.745,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.173,"5min":1.444,"15min":1.444},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.0600000000000001,"15min":1.091},"min":{"1min":0.72799999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.4910000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.149}]},{"osd":6,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0489999999999999,"5min":1.032,"15min":1.054},"min":{"1min":0.745,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.4279999999999999,"5min":1.72,"15min":1.72},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.083,"5min":1.048,"15min":1.1180000000000001},"min":{"1min":0.71399999999999997,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.367,"5min":1.542,"15min":1.6779999999999999},"last":1.264}]}]},{"osd":6,"up_from":14,"seq":60129542275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32628,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339212,"statfs":{"total":96636764160,"available":96603353088,"internally_reserved":0,"allocated":1540096,"data_stored":710699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.83499999999999996,"15min":0.84299999999999997},"min":{"1min":0.65400000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.1319999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.113},{"interface":"front","average":{"1min":0.872,"5min":0.85299999999999998,"15min":0.879},"min":{"1min":0.70199999999999996,"5min":0.309,"15min":0.309},"max":{"1min":1.1639999999999999,"5min":1.2949999999999999,"15min":1.4410000000000001},"last":0.92700000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.871,"15min":0.84699999999999998},"min":{"1min":0.64800000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.153,"5min":1.153,"15min":1.2030000000000001},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.90400000000000003,"15min":0.91500000000000004},"min":{"1min":0.64700000000000002,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.2569999999999999,"5min":1.411,"15min":1.411},"last":0.57699999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.82699999999999996,"5min":0.84299999999999997,"15min":0.86799999999999999},"min":{"1min":0.502,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.103,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.97799999999999998,"5min":0.91600000000000004,"15min":0.877},"min":{"1min":0.82799999999999996,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.1990000000000001,"5min":1.399,"15min":1.399},"last":0.88500000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.94199999999999995,"5min":0.92500000000000004,"15min":0.95299999999999996},"min":{"1min":0.69499999999999995,"5min":0.379,"15min":0.379},"max":{"1min":1.2110000000000001,"5min":1.544,"15min":1.544},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.997,"5min":0.96599999999999997,"15min":0.98099999999999998},"min":{"1min":0.68899999999999995,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1970000000000001,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.1639999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.94399999999999995,"15min":0.95799999999999996},"min":{"1min":0.81000000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.133,"5min":1.508,"15min":1.508},"last":0.98799999999999999},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.93899999999999995,"15min":0.95499999999999996},"min":{"1min":0.69099999999999995,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.1970000000000001,"5min":1.55,"15min":1.6140000000000001},"last":1.343}]},{"osd":5,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.995,"5min":0.94099999999999995,"15min":0.92900000000000005},"min":{"1min":0.73799999999999999,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.25,"5min":1.339,"15min":1.339},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0569999999999999,"15min":1.095},"min":{"1min":0.97999999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.3029999999999999,"5min":1.615,"15min":1.6439999999999999},"last":1.2829999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.1100000000000001,"5min":1.0369999999999999,"15min":1.0880000000000001},"min":{"1min":0.76400000000000001,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.3380000000000001,"5min":1.3819999999999999,"15min":1.522},"last":1.4350000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.0649999999999999,"15min":1.0580000000000001},"min":{"1min":0.79400000000000004,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.319,"5min":1.415,"15min":1.427},"last":1.0349999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32648,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339192,"statfs":{"total":96636764160,"available":96603332608,"internally_reserved":0,"allocated":1560576,"data_stored":717954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82499999999999996,"5min":0.83999999999999997,"15min":0.84999999999999998},"min":{"1min":0.58299999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.1619999999999999,"5min":1.1899999999999999,"15min":1.411},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.83999999999999997,"15min":0.84199999999999997},"min":{"1min":0.56100000000000005,"5min":0.497,"15min":0.41699999999999998},"max":{"1min":1.173,"5min":1.3029999999999999,"15min":1.675},"last":0.88700000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89500000000000002,"5min":0.83999999999999997,"15min":0.86499999999999999},"min":{"1min":0.66200000000000003,"5min":0.51900000000000002,"15min":0.46600000000000003},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.4079999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.79200000000000004,"5min":0.83499999999999996,"15min":0.83799999999999997},"min":{"1min":0.54900000000000004,"5min":0.52300000000000002,"15min":0.36499999999999999},"max":{"1min":1.1579999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.81000000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.91000000000000003,"15min":0.91900000000000004},"min":{"1min":0.61899999999999999,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.129,"5min":1.135,"15min":1.625},"last":0.873},{"interface":"front","average":{"1min":0.93500000000000005,"5min":0.97799999999999998,"15min":0.97599999999999998},"min":{"1min":0.68200000000000005,"5min":0.67900000000000005,"15min":0.55000000000000004},"max":{"1min":1.1599999999999999,"5min":1.331,"15min":1.552},"last":0.90300000000000002}]},{"osd":3,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.93500000000000005,"5min":0.91200000000000003,"15min":0.876},"min":{"1min":0.67500000000000004,"5min":0.5,"15min":0.5},"max":{"1min":1.143,"5min":1.2290000000000001,"15min":1.3919999999999999},"last":0.98099999999999998},{"interface":"front","average":{"1min":0.88900000000000001,"5min":0.88800000000000001,"15min":0.95099999999999996},"min":{"1min":0.66100000000000003,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.1160000000000001,"5min":1.2929999999999999,"15min":3.8460000000000001},"last":0.93400000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.88500000000000001,"15min":0.98199999999999998},"min":{"1min":0.58199999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.1439999999999999,"5min":1.3540000000000001,"15min":3.6869999999999998},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.0600000000000001,"5min":1.0640000000000001,"15min":1.0629999999999999},"min":{"1min":0.753,"5min":0.753,"15min":0.57499999999999996},"max":{"1min":1.244,"5min":1.365,"15min":1.601},"last":1.05}]},{"osd":6,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.91000000000000003,"15min":0.94699999999999995},"min":{"1min":0.65200000000000002,"5min":0.60899999999999999,"15min":0.60899999999999999},"max":{"1min":1.2390000000000001,"5min":1.3540000000000001,"15min":1.421},"last":0.76000000000000001},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.98999999999999999,"15min":1.0469999999999999},"min":{"1min":0.77100000000000002,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.2150000000000001,"5min":1.3740000000000001,"15min":2.21},"last":0.65800000000000003}]},{"osd":7,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0069999999999999,"15min":1.03},"min":{"1min":0.64100000000000001,"5min":0.64000000000000001,"15min":0.45500000000000002},"max":{"1min":1.294,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":0.629},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.054,"15min":1.1399999999999999},"min":{"1min":0.80200000000000005,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.208,"5min":1.502,"15min":3.9830000000000001},"last":1.03}]}]},{"osd":4,"up_from":14,"seq":60129542275,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32608,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339232,"statfs":{"total":96636764160,"available":96603373568,"internally_reserved":0,"allocated":1527808,"data_stored":704104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.84899999999999998,"15min":0.84399999999999997},"min":{"1min":0.63700000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.093,"5min":1.21,"15min":1.411},"last":0.88900000000000001},{"interface":"front","average":{"1min":0.80800000000000005,"5min":0.88900000000000001,"15min":0.873},"min":{"1min":0.59899999999999998,"5min":0.54900000000000004,"15min":0.44600000000000001},"max":{"1min":1.127,"5min":1.462,"15min":2.899},"last":1.1699999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90000000000000002,"5min":0.89300000000000002,"15min":0.91000000000000003},"min":{"1min":0.59699999999999998,"5min":0.53800000000000003,"15min":0.46200000000000002},"max":{"1min":1.151,"5min":1.2490000000000001,"15min":3.085},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.86399999999999999,"15min":0.90400000000000003},"min":{"1min":0.58999999999999997,"5min":0.497,"15min":0.497},"max":{"1min":1.1930000000000001,"5min":1.3240000000000001,"15min":1.5720000000000001},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.86099999999999999,"15min":0.84799999999999998},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.433},"max":{"1min":1.177,"5min":1.228,"15min":1.4419999999999999},"last":0.755},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.92300000000000004,"15min":0.95599999999999996},"min":{"1min":0.78000000000000003,"5min":0.60199999999999998,"15min":0.498},"max":{"1min":1.165,"5min":1.3640000000000001,"15min":3.2160000000000002},"last":1.3220000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.90700000000000003,"15min":0.93700000000000006},"min":{"1min":0.66300000000000003,"5min":0.56899999999999995,"15min":0.47999999999999998},"max":{"1min":1.1859999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.96499999999999997},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.92900000000000005,"15min":0.94399999999999995},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.52200000000000002},"max":{"1min":1.2110000000000001,"5min":1.3979999999999999,"15min":1.577},"last":0.99299999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.90600000000000003,"15min":0.95399999999999996},"min":{"1min":0.70599999999999996,"5min":0.59999999999999998,"15min":0.49299999999999999},"max":{"1min":1.252,"5min":1.252,"15min":2.996},"last":1.2390000000000001},{"interface":"front","average":{"1min":0.95599999999999996,"5min":0.98599999999999999,"15min":1.073},"min":{"1min":0.61099999999999999,"5min":0.495,"15min":0.495},"max":{"1min":1.232,"5min":1.5189999999999999,"15min":2.8860000000000001},"last":1.0569999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.93799999999999994,"15min":0.93899999999999995},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.169,"5min":1.522,"15min":1.6499999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.0549999999999999,"5min":1.0669999999999999,"15min":1.101},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.55500000000000005},"max":{"1min":1.3220000000000001,"5min":1.3939999999999999,"15min":3.1469999999999998},"last":1.375}]},{"osd":7,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":1.0469999999999999,"15min":1.1040000000000001},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.28,"5min":1.556,"15min":2.1230000000000002},"last":1.1919999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.071,"15min":1.099},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.67000000000000004},"max":{"1min":1.242,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":1.288}]}]},{"osd":3,"up_from":14,"seq":60129542276,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32612,"kb_used_data":1492,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339228,"statfs":{"total":96636764160,"available":96603369472,"internally_reserved":0,"allocated":1527808,"data_stored":704104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.63300000000000001,"15min":0.629},"min":{"1min":0.5,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83199999999999996,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.68899999999999995,"15min":0.67300000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.90100000000000002,"5min":1.107,"15min":1.161},"last":0.54700000000000004}]},{"osd":1,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66100000000000003,"15min":0.64800000000000002},"min":{"1min":0.46899999999999997,"5min":0.40799999999999997,"15min":0.35499999999999998},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.82699999999999996,"5min":0.82199999999999995,"15min":0.79700000000000004},"min":{"1min":0.60499999999999998,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.163,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.77300000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71499999999999997,"15min":0.70199999999999996},"min":{"1min":0.53700000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.133,"5min":1.133,"15min":1.27},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.80500000000000005,"5min":0.81200000000000006,"15min":0.77800000000000002},"min":{"1min":0.58999999999999997,"5min":0.379,"15min":0.379},"max":{"1min":1.069,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.96499999999999997}]},{"osd":4,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95099999999999996,"15min":0.92000000000000004},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.60499999999999998},"max":{"1min":1.2929999999999999,"5min":1.3380000000000001,"15min":1.8460000000000001},"last":1.2210000000000001},{"interface":"front","average":{"1min":0.96899999999999997,"5min":0.93000000000000005,"15min":0.95099999999999996},"min":{"1min":0.747,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.1930000000000001,"5min":1.3360000000000001,"15min":1.782},"last":0.99099999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":0.95299999999999996,"15min":0.96599999999999997},"min":{"1min":0.78000000000000003,"5min":0.65900000000000003,"15min":0.59499999999999997},"max":{"1min":1.3100000000000001,"5min":1.397,"15min":1.6299999999999999},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.038,"5min":1.0409999999999999,"15min":1.069},"min":{"1min":0.75800000000000001,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.367,"5min":1.581,"15min":1.8140000000000001},"last":1.3049999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.079,"15min":1.099},"min":{"1min":0.79400000000000004,"5min":0.61599999999999999,"15min":0.58699999999999997},"max":{"1min":1.4159999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.25},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.0669999999999999,"15min":1.0489999999999999},"min":{"1min":0.76900000000000002,"5min":0.64900000000000002,"15min":0.57499999999999996},"max":{"1min":1.4530000000000001,"5min":1.7030000000000001,"15min":1.95},"last":1.153}]},{"osd":7,"last update":"Fri Sep 13 16:35:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0900000000000001,"15min":1.081},"min":{"1min":0.76900000000000002,"5min":0.74199999999999999,"15min":0.71399999999999997},"max":{"1min":1.419,"5min":1.657,"15min":1.889},"last":1.0700000000000001},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0580000000000001,"15min":1.0569999999999999},"min":{"1min":0.81299999999999994,"5min":0.77200000000000002,"15min":0.70799999999999996},"max":{"1min":1.6240000000000001,"5min":1.6240000000000001,"15min":1.756},"last":0.82499999999999996}]}]},{"osd":2,"up_from":14,"seq":60129542277,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32624,"kb_used_data":1504,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339216,"statfs":{"total":96636764160,"available":96603357184,"internally_reserved":0,"allocated":1540096,"data_stored":710699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60599999999999998,"15min":0.59699999999999998},"min":{"1min":0.44500000000000001,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.75700000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.78900000000000003},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.67800000000000005,"15min":0.68899999999999995},"min":{"1min":0.46800000000000003,"5min":0.377,"15min":0.377},"max":{"1min":1.0920000000000001,"5min":1.169,"15min":1.169},"last":0.751}]},{"osd":1,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.68899999999999995,"15min":0.69399999999999995},"min":{"1min":0.51600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.96199999999999997,"5min":0.995,"15min":0.997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.71299999999999997,"15min":0.69899999999999995},"min":{"1min":0.59499999999999997,"5min":0.316,"15min":0.316},"max":{"1min":0.88300000000000001,"5min":1.147,"15min":1.147},"last":0.55700000000000005}]},{"osd":3,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.71299999999999997,"15min":0.68500000000000005},"min":{"1min":0.438,"5min":0.39000000000000001,"15min":0.38100000000000001},"max":{"1min":0.91400000000000003,"5min":1.119,"15min":1.119},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.85499999999999998,"5min":0.82999999999999996,"15min":0.81200000000000006},"min":{"1min":0.59299999999999997,"5min":0.37,"15min":0.37},"max":{"1min":1.0589999999999999,"5min":1.1000000000000001,"15min":1.2330000000000001},"last":0.88600000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.90500000000000003,"5min":0.93300000000000005,"15min":0.94299999999999995},"min":{"1min":0.70899999999999996,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.131,"5min":1.448,"15min":1.5149999999999999},"last":1.091},{"interface":"front","average":{"1min":1.004,"5min":0.97699999999999998,"15min":0.96899999999999997},"min":{"1min":0.77600000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.3740000000000001,"5min":1.3740000000000001,"15min":1.8799999999999999},"last":1.4219999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.95499999999999996,"15min":0.96899999999999997},"min":{"1min":0.79700000000000004,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.2,"5min":1.452,"15min":1.4750000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":0.95399999999999996,"5min":0.95699999999999996,"15min":0.96099999999999997},"min":{"1min":0.70799999999999996,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.1839999999999999,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.155}]},{"osd":6,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":1.0189999999999999,"5min":0.98899999999999999,"15min":0.999},"min":{"1min":0.68300000000000005,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.347,"5min":1.347,"15min":2.024},"last":1.302},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.0429999999999999,"15min":1.052},"min":{"1min":0.85099999999999998,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.214,"5min":1.5249999999999999,"15min":1.6359999999999999},"last":0.93100000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:35:36 2024","interfaces":[{"interface":"back","average":{"1min":1.05,"5min":1.0489999999999999,"15min":1.0649999999999999},"min":{"1min":0.754,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.321,"5min":1.4810000000000001,"15min":1.4810000000000001},"last":1.27},{"interface":"front","average":{"1min":1.0669999999999999,"5min":1.0620000000000001,"15min":1.0569999999999999},"min":{"1min":0.81399999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.423,"5min":1.587,"15min":1.587},"last":1.482}]}]},{"osd":1,"up_from":14,"seq":60129542276,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32640,"kb_used_data":1520,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339200,"statfs":{"total":96636764160,"available":96603340800,"internally_reserved":0,"allocated":1556480,"data_stored":717816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.629,"15min":0.61499999999999999},"min":{"1min":0.45100000000000001,"5min":0.42999999999999999,"15min":0.23300000000000001},"max":{"1min":0.82999999999999996,"5min":0.88900000000000001,"15min":1.0880000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.75,"15min":0.74399999999999999},"min":{"1min":0.56200000000000006,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.2010000000000001},"last":0.67400000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.66800000000000004,"15min":0.64700000000000002},"min":{"1min":0.48699999999999999,"5min":0.38300000000000001,"15min":0.26700000000000002},"max":{"1min":0.94099999999999995,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.74399999999999999,"5min":0.72799999999999998,"15min":0.71299999999999997},"min":{"1min":0.48999999999999999,"5min":0.45700000000000002,"15min":0.29699999999999999},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.458},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.74299999999999999,"15min":0.72299999999999998},"min":{"1min":0.56299999999999994,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":0.98099999999999998,"5min":1.1359999999999999,"15min":1.4019999999999999},"last":0.95099999999999996},{"interface":"front","average":{"1min":0.70299999999999996,"5min":0.70599999999999996,"15min":0.68899999999999995},"min":{"1min":0.52700000000000002,"5min":0.47099999999999997,"15min":0.39600000000000002},"max":{"1min":0.92200000000000004,"5min":1.175,"15min":1.403},"last":0.63800000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":0.95399999999999996,"15min":0.92700000000000005},"min":{"1min":0.68500000000000005,"5min":0.67800000000000005,"15min":0.505},"max":{"1min":1.1559999999999999,"5min":1.2190000000000001,"15min":1.546},"last":0.88},{"interface":"front","average":{"1min":0.95499999999999996,"5min":0.91500000000000004,"15min":0.92200000000000004},"min":{"1min":0.81499999999999995,"5min":0.628,"15min":0.51700000000000002},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":2.2040000000000002},"last":1.1459999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96599999999999997,"15min":0.95999999999999996},"min":{"1min":0.68799999999999994,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":1.1279999999999999,"5min":1.3819999999999999,"15min":1.389},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.93300000000000005,"5min":0.93100000000000005,"15min":0.95099999999999996},"min":{"1min":0.77600000000000002,"5min":0.56699999999999995,"15min":0.52000000000000002},"max":{"1min":1.0760000000000001,"5min":1.238,"15min":2.3029999999999999},"last":0.98699999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":0.97699999999999998,"5min":0.998,"15min":1.006},"min":{"1min":0.82299999999999995,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.1599999999999999,"5min":1.4330000000000001,"15min":1.613},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0529999999999999,"15min":1.028},"min":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.59299999999999997},"max":{"1min":1.667,"5min":1.667,"15min":1.698},"last":0.80800000000000005}]},{"osd":7,"last update":"Fri Sep 13 16:35:29 2024","interfaces":[{"interface":"back","average":{"1min":1.038,"5min":1.0249999999999999,"15min":1.0129999999999999},"min":{"1min":0.83199999999999996,"5min":0.64400000000000002,"15min":0.62},"max":{"1min":1.224,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.071,"5min":1.069,"15min":1.0580000000000001},"min":{"1min":0.94599999999999995,"5min":0.61599999999999999,"15min":0.59299999999999997},"max":{"1min":1.2829999999999999,"5min":1.4059999999999999,"15min":1.7689999999999999},"last":1.006}]}]},{"osd":0,"up_from":14,"seq":60129542276,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33224,"kb_used_data":2100,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338616,"statfs":{"total":96636764160,"available":96602742784,"internally_reserved":0,"allocated":2150400,"data_stored":1308184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.63300000000000001,"15min":0.65500000000000003},"min":{"1min":0.496,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":0.91400000000000003,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.69699999999999995,"15min":0.67300000000000004},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.377},"max":{"1min":1.1559999999999999,"5min":1.1559999999999999,"15min":1.157},"last":0.78600000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.755,"5min":0.70199999999999996,"15min":0.71599999999999997},"min":{"1min":0.55400000000000005,"5min":0.503,"15min":0.45500000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.202},"last":0.71299999999999997},{"interface":"front","average":{"1min":0.746,"5min":0.72699999999999998,"15min":0.70599999999999996},"min":{"1min":0.51800000000000002,"5min":0.443,"15min":0.39800000000000002},"max":{"1min":1.0760000000000001,"5min":1.0780000000000001,"15min":1.2629999999999999},"last":0.90800000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.72999999999999998,"15min":0.69499999999999995},"min":{"1min":0.52800000000000002,"5min":0.42799999999999999,"15min":0.312},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.4830000000000001},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.88100000000000001,"15min":0.85399999999999998},"min":{"1min":0.72899999999999998,"5min":0.53300000000000003,"15min":0.42899999999999999},"max":{"1min":1.155,"5min":1.155,"15min":1.6100000000000001},"last":0.81899999999999995}]},{"osd":4,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":0.96399999999999997,"15min":0.95999999999999996},"min":{"1min":0.66900000000000004,"5min":0.629,"15min":0.57699999999999996},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.486},"last":1.3080000000000001},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.94899999999999995,"15min":0.95999999999999996},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.59999999999999998},"max":{"1min":1.274,"5min":1.274,"15min":1.704},"last":0.88700000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.98699999999999999,"15min":0.99099999999999999},"min":{"1min":0.72899999999999998,"5min":0.67100000000000004,"15min":0.59599999999999997},"max":{"1min":1.532,"5min":1.532,"15min":1.575},"last":1.1419999999999999},{"interface":"front","average":{"1min":1.0109999999999999,"5min":0.99299999999999999,"15min":0.98999999999999999},"min":{"1min":0.72799999999999998,"5min":0.61199999999999999,"15min":0.443},"max":{"1min":1.49,"5min":1.742,"15min":1.742},"last":1.399}]},{"osd":6,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":0.93999999999999995,"5min":0.96699999999999997,"15min":0.97999999999999998},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.628},"max":{"1min":1.218,"5min":1.4099999999999999,"15min":1.944},"last":0.94299999999999995},{"interface":"front","average":{"1min":1.1379999999999999,"5min":1.0840000000000001,"15min":1.0700000000000001},"min":{"1min":0.92900000000000005,"5min":0.77300000000000002,"15min":0.58899999999999997},"max":{"1min":1.53,"5min":1.53,"15min":1.5780000000000001},"last":1.0980000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:35:43 2024","interfaces":[{"interface":"back","average":{"1min":1.113,"5min":1.0860000000000001,"15min":1.0669999999999999},"min":{"1min":0.76300000000000001,"5min":0.747,"15min":0.61699999999999999},"max":{"1min":1.421,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.359},{"interface":"front","average":{"1min":1.042,"5min":1.0800000000000001,"15min":1.0700000000000001},"min":{"1min":0.84899999999999998,"5min":0.80900000000000005,"15min":0.65400000000000003},"max":{"1min":1.274,"5min":1.472,"15min":2.0139999999999998},"last":0.98799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:35:59.658 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-09-13T16:35:59.659 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-09-13T16:35:59.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cee1a700 1 Processor -- start 2024-09-13T16:35:59.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cee1a700 1 -- start start 2024-09-13T16:35:59.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cee1a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff3c8152fd0 0x7ff3c80b3090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cee1a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cee1a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c80b3970 con 0x7ff3c806dda0 2024-09-13T16:35:59.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cee1a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c806be90 con 0x7ff3c8152fd0 2024-09-13T16:35:59.786 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cee1a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c806c170 con 0x7ff3c81533b0 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cd3b7700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff3c806dda0 0x7ff3c80b1860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57510/0 (socket says 172.21.15.119:57510) 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3cd3b7700 1 -- 172.21.15.119:0/1512420828 learned_addr learned my addr 172.21.15.119:0/1512420828 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3ccbb6700 1 --2- 172.21.15.119:0/1512420828 >> v2:172.21.15.119:3300/0 conn(0x7ff3c8152fd0 0x7ff3c80b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3c7fff700 1 --2- 172.21.15.119:0/1512420828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3c77fe700 1 -- 172.21.15.119:0/1512420828 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1596972731 0 0) 0x7ff3c80b3970 con 0x7ff3c806dda0 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3c77fe700 1 -- 172.21.15.119:0/1512420828 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3b00057e0 con 0x7ff3c806dda0 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3ccbb6700 1 -- 172.21.15.119:0/1512420828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 msgr2=0x7ff3c806fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3ccbb6700 1 --2- 172.21.15.119:0/1512420828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c806fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3ccbb6700 1 -- 172.21.15.119:0/1512420828 >> v1:172.21.15.119:6789/0 conn(0x7ff3c806dda0 legacy=0x7ff3c80b1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.782+0000 7ff3ccbb6700 1 -- 172.21.15.119:0/1512420828 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c806c450 con 0x7ff3c8152fd0 2024-09-13T16:35:59.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3ccbb6700 1 --2- 172.21.15.119:0/1512420828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c80b3090 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009850 tx=0x7ff3bc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8228582944c8c7e7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3c77fe700 1 -- 172.21.15.119:0/1512420828 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc011830 con 0x7ff3c8152fd0 2024-09-13T16:35:59.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3c77fe700 1 -- 172.21.15.119:0/1512420828 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc011990 con 0x7ff3c8152fd0 2024-09-13T16:35:59.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3c77fe700 1 -- 172.21.15.119:0/1512420828 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3bc011d00 con 0x7ff3c8152fd0 2024-09-13T16:35:59.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1512420828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 msgr2=0x7ff3c80b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1512420828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c80b3090 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009850 tx=0x7ff3bc009b60 comp rx=0 tx=0).stop 2024-09-13T16:35:59.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1512420828 shutdown_connections 2024-09-13T16:35:59.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1512420828 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c806fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1512420828 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c80b3090 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1512420828 >> 172.21.15.119:0/1512420828 conn(0x7ff3c80b0560 msgr2=0x7ff3c80b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:35:59.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1512420828 shutdown_connections 2024-09-13T16:35:59.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1512420828 wait complete. 2024-09-13T16:35:59.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 Processor -- start 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- start start 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3c806dda0 0x7ff3c814d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c815a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c80ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c815cae0 con 0x7ff3c8152fd0 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c815d580 con 0x7ff3c806dda0 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3cee1a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c815b0e0 con 0x7ff3c81533b0 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.786+0000 7ff3c7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c815a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c815a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56368/0 (socket says 172.21.15.119:56368) 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 -- 172.21.15.119:0/1650562311 learned_addr learned my addr 172.21.15.119:0/1650562311 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3cd3b7700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c80ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3ccbb6700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3c806dda0 0x7ff3c814d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 -- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 msgr2=0x7ff3c80ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c80ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 -- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3c806dda0 msgr2=0x7ff3c814d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3c806dda0 0x7ff3c814d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 -- 172.21.15.119:0/1650562311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc0094b0 con 0x7ff3c8152fd0 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3ccbb6700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3c806dda0 0x7ff3c814d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c7fff700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c815a300 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800e840 tx=0x7ff3b800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c57fa700 1 -- 172.21.15.119:0/1650562311 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b80095a0 con 0x7ff3c8152fd0 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c57fa700 1 -- 172.21.15.119:0/1650562311 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b800c690 con 0x7ff3c8152fd0 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c815a100 con 0x7ff3c8152fd0 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c57fa700 1 -- 172.21.15.119:0/1650562311 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b800ca00 con 0x7ff3c8152fd0 2024-09-13T16:35:59.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c815d060 con 0x7ff3c8152fd0 2024-09-13T16:35:59.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c815c940 con 0x7ff3c8152fd0 2024-09-13T16:35:59.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c57fa700 1 -- 172.21.15.119:0/1650562311 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff3b800cbe0 con 0x7ff3c8152fd0 2024-09-13T16:35:59.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.790+0000 7ff3c57fa700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3b4066730 0x7ff3b4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:35:59.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.794+0000 7ff3c57fa700 1 -- 172.21.15.119:0/1650562311 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff3b80842a0 con 0x7ff3c8152fd0 2024-09-13T16:35:59.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.794+0000 7ff3ccbb6700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3b4066730 0x7ff3b4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:35:59.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.794+0000 7ff3ccbb6700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3b4066730 0x7ff3b4068bf0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009ba0 tx=0x7ff3bc008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:35:59.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:35:59.798+0000 7ff3c57fa700 1 -- 172.21.15.119:0/1650562311 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff3c815c940 con 0x7ff3c8152fd0 2024-09-13T16:36:00.034 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.030+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff3c815c940 con 0x7ff3c8152fd0 2024-09-13T16:36:00.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.038+0000 7ff3c57fa700 1 -- 172.21.15.119:0/1650562311 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v18) v1 ==== 190+0+0 (secure 0 0 0) 0x7ff3c815c940 con 0x7ff3c8152fd0 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3b4066730 msgr2=0x7ff3b4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3b4066730 0x7ff3b4068bf0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc009ba0 tx=0x7ff3bc008030 comp rx=0 tx=0).stop 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 msgr2=0x7ff3c815a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c815a300 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7ff3b800e840 tx=0x7ff3b800ec00 comp rx=0 tx=0).stop 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 shutdown_connections 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff3b4066730 0x7ff3b4068bf0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff3c81533b0 0x7ff3c80ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff3c8152fd0 0x7ff3c815a300 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 --2- 172.21.15.119:0/1650562311 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff3c806dda0 0x7ff3c814d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 >> 172.21.15.119:0/1650562311 conn(0x7ff3c80b0560 msgr2=0x7ff3c81593d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 shutdown_connections 2024-09-13T16:36:00.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.042+0000 7ff3cee1a700 1 -- 172.21.15.119:0/1650562311 wait complete. 2024-09-13T16:36:00.047 INFO:teuthology.orchestra.run.smithi119.stderr:updated caps for client.0 2024-09-13T16:36:00.059 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:00.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 Processor -- start 2024-09-13T16:36:00.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- start start 2024-09-13T16:36:00.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe14415e840 0x7fe144070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe14415ec20 0x7fe144161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe144071210 con 0x7fe144164a70 2024-09-13T16:36:00.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1440714f0 con 0x7fe14415e840 2024-09-13T16:36:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1440717d0 con 0x7fe14415ec20 2024-09-13T16:36:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe14415ec20 0x7fe144161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe149d0d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe14415e840 0x7fe144070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe14415ec20 0x7fe144161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:56190/0 (socket says 172.21.15.119:56190) 2024-09-13T16:36:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 -- 172.21.15.119:0/2797594232 learned_addr learned my addr 172.21.15.119:0/2797594232 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14a50e700 1 --1- 172.21.15.119:0/2797594232 >> v1:172.21.15.145:6789/0 conn(0x7fe144164a70 0x7fe1440ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:52722/0 (socket says 172.21.15.119:52722) 2024-09-13T16:36:00.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 -- 172.21.15.119:0/2797594232 >> v2:172.21.15.145:3300/0 conn(0x7fe14415e840 msgr2=0x7fe144070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 --2- 172.21.15.119:0/2797594232 >> v2:172.21.15.145:3300/0 conn(0x7fe14415e840 0x7fe144070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 -- 172.21.15.119:0/2797594232 >> v1:172.21.15.145:6789/0 conn(0x7fe144164a70 legacy=0x7fe1440ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 -- 172.21.15.119:0/2797594232 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144071ab0 con 0x7fe14415ec20 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe149d0d700 1 --2- 172.21.15.119:0/2797594232 >> v2:172.21.15.145:3300/0 conn(0x7fe14415e840 0x7fe144070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14950c700 1 --2- 172.21.15.119:0/2797594232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe14415ec20 0x7fe144161f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe134009910 tx=0x7fe134009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49683d250f6ced97 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe148d0b700 1 -- 172.21.15.119:0/2797594232 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134011830 con 0x7fe14415ec20 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe148d0b700 1 -- 172.21.15.119:0/2797594232 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe134011990 con 0x7fe14415ec20 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe148d0b700 1 -- 172.21.15.119:0/2797594232 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe134011d40 con 0x7fe14415ec20 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- 172.21.15.119:0/2797594232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe14415ec20 msgr2=0x7fe144161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 --2- 172.21.15.119:0/2797594232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe14415ec20 0x7fe144161f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe134009910 tx=0x7fe134009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:00.171 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- 172.21.15.119:0/2797594232 shutdown_connections 2024-09-13T16:36:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 --2- 172.21.15.119:0/2797594232 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe14415ec20 0x7fe144161f00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 --2- 172.21.15.119:0/2797594232 >> v2:172.21.15.145:3300/0 conn(0x7fe14415e840 0x7fe144070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- 172.21.15.119:0/2797594232 >> 172.21.15.119:0/2797594232 conn(0x7fe14406a350 msgr2=0x7fe14406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- 172.21.15.119:0/2797594232 shutdown_connections 2024-09-13T16:36:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.166+0000 7fe14bf71700 1 -- 172.21.15.119:0/2797594232 wait complete. 2024-09-13T16:36:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 Processor -- start 2024-09-13T16:36:00.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 -- start start 2024-09-13T16:36:00.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe14415e840 0x7fe14415bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe144164a70 0x7fe1440ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.174 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1440a8e90 0x7fe1440a9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1440acba0 con 0x7fe14415e840 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1440ace80 con 0x7fe144164a70 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1440ad160 con 0x7fe1440a8e90 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe14415e840 0x7fe14415bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe14415e840 0x7fe14415bef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56374/0 (socket says 172.21.15.119:56374) 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14950c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe144164a70 0x7fe1440ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 -- 172.21.15.119:0/399981276 learned_addr learned my addr 172.21.15.119:0/399981276 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14a50e700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1440a8e90 0x7fe1440a9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 -- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1440a8e90 msgr2=0x7fe1440a9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1440a8e90 0x7fe1440a9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 -- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe144164a70 msgr2=0x7fe1440ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe144164a70 0x7fe1440ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 -- 172.21.15.119:0/399981276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe140009a70 con 0x7fe14415e840 2024-09-13T16:36:00.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe149d0d700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe14415e840 0x7fe14415bef0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fe14000efc0 tx=0x7fe14000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe13a7fc700 1 -- 172.21.15.119:0/399981276 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe14000cd10 con 0x7fe14415e840 2024-09-13T16:36:00.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe13a7fc700 1 -- 172.21.15.119:0/399981276 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe14000f040 con 0x7fe14415e840 2024-09-13T16:36:00.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe13a7fc700 1 -- 172.21.15.119:0/399981276 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1400146a0 con 0x7fe14415e840 2024-09-13T16:36:00.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe134009570 con 0x7fe14415e840 2024-09-13T16:36:00.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.170+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1440aa340 con 0x7fe14415e840 2024-09-13T16:36:00.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.174+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe144067760 con 0x7fe14415e840 2024-09-13T16:36:00.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.174+0000 7fe13a7fc700 1 -- 172.21.15.119:0/399981276 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe140014800 con 0x7fe14415e840 2024-09-13T16:36:00.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.174+0000 7fe13a7fc700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe128066760 0x7fe128068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.178+0000 7fe13a7fc700 1 -- 172.21.15.119:0/399981276 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fe140084e20 con 0x7fe14415e840 2024-09-13T16:36:00.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.178+0000 7fe14950c700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe128066760 0x7fe128068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.178+0000 7fe14950c700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe128066760 0x7fe128068c20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe134000c00 tx=0x7fe134008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.182+0000 7fe13a7fc700 1 -- 172.21.15.119:0/399981276 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe140052050 con 0x7fe14415e840 2024-09-13T16:36:00.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.386+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe1440acce0 con 0x7fe14415e840 2024-09-13T16:36:00.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.386+0000 7fe13a7fc700 1 -- 172.21.15.119:0/399981276 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+4739 (secure 0 0 0) 0x7fe1440acce0 con 0x7fe14415e840 2024-09-13T16:36:00.390 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:00.390 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":92,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":92,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:35:58.262367+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9546":{"gid":9546,"name":"b","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.145:6835/3901706437","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3901706437},{"type":"v1","addr":"172.21.15.145:6835","nonce":3901706437}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:00.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe128066760 msgr2=0x7fe128068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe128066760 0x7fe128068c20 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fe134000c00 tx=0x7fe134008030 comp rx=0 tx=0).stop 2024-09-13T16:36:00.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe14415e840 msgr2=0x7fe14415bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe14415e840 0x7fe14415bef0 secure :-1 s=READY pgs=1813 cs=0 l=1 rev1=1 crypto rx=0x7fe14000efc0 tx=0x7fe14000c3a0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 shutdown_connections 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe128066760 0x7fe128068c20 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe1440a8e90 0x7fe1440a9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe144164a70 0x7fe1440ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 --2- 172.21.15.119:0/399981276 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe14415e840 0x7fe14415bef0 unknown :-1 s=CLOSED pgs=1813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 >> 172.21.15.119:0/399981276 conn(0x7fe14406a350 msgr2=0x7fe1441635d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 shutdown_connections 2024-09-13T16:36:00.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.390+0000 7fe14bf71700 1 -- 172.21.15.119:0/399981276 wait complete. 2024-09-13T16:36:00.396 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 92 2024-09-13T16:36:00.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 92, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:35:55.244835+0000', 'modified': '2024-09-13T16:35:58.262367+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9546}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9546': {'gid': 9546, 'name': 'b', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.145:6835/3901706437', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3901706437}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 3901706437}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:36:00.813 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:36:00.813 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-13T16:36:00.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 Processor -- start 2024-09-13T16:36:00.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- start start 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f008c0cd8d0 0x7f008c06cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f008c0b2200 0x7f008c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f008c0705d0 con 0x7f008c0b17f0 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c0cbf10 con 0x7f008c0cd8d0 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008c0b2bb0 con 0x7f008c0b2200 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f008c0cd8d0 0x7f008c06cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f008c0cd8d0 0x7f008c06cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56380/0 (socket says 172.21.15.119:56380) 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 -- 172.21.15.119:0/1205008948 learned_addr learned my addr 172.21.15.119:0/1205008948 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00918b4700 1 --2- 172.21.15.119:0/1205008948 >> v2:172.21.15.145:3300/0 conn(0x7f008c0b2200 0x7f008c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 -- 172.21.15.119:0/1205008948 >> v2:172.21.15.145:3300/0 conn(0x7f008c0b2200 msgr2=0x7f008c0b2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 --2- 172.21.15.119:0/1205008948 >> v2:172.21.15.145:3300/0 conn(0x7f008c0b2200 0x7f008c0b2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 -- 172.21.15.119:0/1205008948 >> v1:172.21.15.145:6789/0 conn(0x7f008c0b17f0 legacy=0x7f008c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 -- 172.21.15.119:0/1205008948 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008c06d0d0 con 0x7f008c0cd8d0 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00918b4700 1 --2- 172.21.15.119:0/1205008948 >> v2:172.21.15.145:3300/0 conn(0x7f008c0b2200 0x7f008c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00910b3700 1 --2- 172.21.15.119:0/1205008948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0cd8d0 0x7f008c06cb90 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f008c0b3a60 tx=0x7f0088009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92d05c6a0f8756c7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f0083fff700 1 -- 172.21.15.119:0/1205008948 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088004230 con 0x7f008c0cd8d0 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f0083fff700 1 -- 172.21.15.119:0/1205008948 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088004390 con 0x7f008c0cd8d0 2024-09-13T16:36:00.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f0083fff700 1 -- 172.21.15.119:0/1205008948 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088009e60 con 0x7f008c0cd8d0 2024-09-13T16:36:00.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- 172.21.15.119:0/1205008948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0cd8d0 msgr2=0x7f008c06cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 --2- 172.21.15.119:0/1205008948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0cd8d0 0x7f008c06cb90 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f008c0b3a60 tx=0x7f0088009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:00.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- 172.21.15.119:0/1205008948 shutdown_connections 2024-09-13T16:36:00.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 --2- 172.21.15.119:0/1205008948 >> v2:172.21.15.145:3300/0 conn(0x7f008c0b2200 0x7f008c0b2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 --2- 172.21.15.119:0/1205008948 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0cd8d0 0x7f008c06cb90 unknown :-1 s=CLOSED pgs=1814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- 172.21.15.119:0/1205008948 >> 172.21.15.119:0/1205008948 conn(0x7f008c06a350 msgr2=0x7f008c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- 172.21.15.119:0/1205008948 shutdown_connections 2024-09-13T16:36:00.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.938+0000 7f00920b5700 1 -- 172.21.15.119:0/1205008948 wait complete. 2024-09-13T16:36:00.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 Processor -- start 2024-09-13T16:36:00.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 -- start start 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f008c0b17f0 0x7f008c0a9db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0b2200 0x7f008c0ac5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f008c0cd8d0 0x7f008c0afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c0acae0 con 0x7f008c0b2200 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1f7410 con 0x7f008c0b17f0 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00920b5700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f008c1f6890 con 0x7f008c0cd8d0 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0b2200 0x7f008c0ac5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0b2200 0x7f008c0ac5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56392/0 (socket says 172.21.15.119:56392) 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 -- 172.21.15.119:0/2340283656 learned_addr learned my addr 172.21.15.119:0/2340283656 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00910b3700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f008c0b17f0 0x7f008c0a9db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00918b4700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f008c0cd8d0 0x7f008c0afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 -- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f008c0cd8d0 msgr2=0x7f008c0afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f008c0cd8d0 0x7f008c0afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 -- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f008c0b17f0 msgr2=0x7f008c0a9db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f008c0b17f0 0x7f008c0a9db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.942+0000 7f00908b2700 1 -- 172.21.15.119:0/2340283656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0088009570 con 0x7f008c0b2200 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f00908b2700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0b2200 0x7f008c0ac5a0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f007c00c4e0 tx=0x7f007c00c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f0081ffb700 1 -- 172.21.15.119:0/2340283656 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c011790 con 0x7f008c0b2200 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c1f6b70 con 0x7f008c0b2200 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f0081ffb700 1 -- 172.21.15.119:0/2340283656 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c011910 con 0x7f008c0b2200 2024-09-13T16:36:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f0081ffb700 1 -- 172.21.15.119:0/2340283656 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c011cc0 con 0x7f008c0b2200 2024-09-13T16:36:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c1f7130 con 0x7f008c0b2200 2024-09-13T16:36:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f0081ffb700 1 -- 172.21.15.119:0/2340283656 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f007c01c460 con 0x7f008c0b2200 2024-09-13T16:36:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f0081ffb700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0070066730 0x7f0070068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.946+0000 7f0081ffb700 1 -- 172.21.15.119:0/2340283656 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f007c084040 con 0x7f008c0b2200 2024-09-13T16:36:00.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0070066730 0x7f0070068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0070066730 0x7f0070068bf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f008c0b3a60 tx=0x7f0088008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 Processor -- start 2024-09-13T16:36:00.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 -- start start 2024-09-13T16:36:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x12ac610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x1274890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12acb50 0x1274dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x11d3d80 con 0x12ac230 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x11d1510 con 0x128ae90 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x11cce00 con 0x12acb50 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x12ac610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x12ac610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56400/0 (socket says 172.21.15.119:56400) 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 -- 172.21.15.119:0/4283971582 learned_addr learned my addr 172.21.15.119:0/4283971582 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00918b4700 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12acb50 0x1274dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00908b2700 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x1274890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 -- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12acb50 msgr2=0x1274dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12acb50 0x1274dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 -- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 msgr2=0x1274890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x1274890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 -- 172.21.15.119:0/4283971582 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x103f960 con 0x12ac230 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00918b4700 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12acb50 0x1274dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.950+0000 7f00910b3700 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x12ac610 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f008c0b3ee0 tx=0x7f0088010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=562bb704c74bd56f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f0076ffd700 1 -- 172.21.15.119:0/4283971582 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00880038c0 con 0x12ac230 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f0076ffd700 1 -- 172.21.15.119:0/4283971582 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088003a20 con 0x12ac230 2024-09-13T16:36:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f0076ffd700 1 -- 172.21.15.119:0/4283971582 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0088003d90 con 0x12ac230 2024-09-13T16:36:00.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- 172.21.15.119:0/4283971582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 msgr2=0x12ac610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x12ac610 secure :-1 s=READY pgs=1816 cs=0 l=1 rev1=1 crypto rx=0x7f008c0b3ee0 tx=0x7f0088010000 comp rx=0 tx=0).stop 2024-09-13T16:36:00.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- 172.21.15.119:0/4283971582 shutdown_connections 2024-09-13T16:36:00.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12acb50 0x1274dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x1274890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 --2- 172.21.15.119:0/4283971582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x12ac610 unknown :-1 s=CLOSED pgs=1816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- 172.21.15.119:0/4283971582 >> 172.21.15.119:0/4283971582 conn(0xff44d0 msgr2=0x128de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- 172.21.15.119:0/4283971582 shutdown_connections 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- 172.21.15.119:0/4283971582 wait complete. 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- start start 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 Processor -- start 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x1286c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ac230 0x1264910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x1264e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x118a730 con 0x12acb50 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x11905b0 con 0x12ac230 2024-09-13T16:36:00.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1078aa0 con 0x128ae90 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00965c8740 10 client.0 _ll_register_callbacks cb 0x11c6610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00918b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x1264e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00918b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x1264e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56402/0 (socket says 172.21.15.119:56402) 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00918b4700 1 -- 172.21.15.119:0/522452144 learned_addr learned my addr 172.21.15.119:0/522452144 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00908b2700 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ac230 0x1264910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.954+0000 7f00910b3700 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x1286c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00918b4700 1 -- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 msgr2=0x1286c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00918b4700 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x1286c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00918b4700 1 -- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ac230 msgr2=0x1264910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00918b4700 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ac230 0x1264910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00918b4700 1 -- 172.21.15.119:0/522452144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c06d0d0 con 0x12acb50 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00918b4700 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x1264e50 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f008400df40 tx=0x7f008400df70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb42df4a599a784f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f006bfff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f006bfff700 1 -- 172.21.15.119:0/522452144 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008400cd20 con 0x12acb50 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f006bfff700 1 -- 172.21.15.119:0/522452144 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f008400a410 con 0x12acb50 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x10011e0 con 0x12acb50 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f006bfff700 1 -- 172.21.15.119:0/522452144 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f008400ce80 con 0x12acb50 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 10 client.9690 fetch_fsmap learned FSMap version 92 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x1193220 con 0x12acb50 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f006bfff700 1 -- 172.21.15.119:0/522452144 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x10011e0 con 0x12acb50 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f006bfff700 1 -- 172.21.15.119:0/522452144 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 92) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f008400a5d0 con 0x12acb50 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 10 client.9690 fetch_fsmap finished waiting for FSMap version 92 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 10 client.9690 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 20 client.9690 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x1266880 0x12620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.964 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 4 client.9690 mds_command: new command op to 9546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:36:00.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 --> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x104b460 con 0x1266880 2024-09-13T16:36:00.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00910b3700 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x1266880 0x12620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f006bfff700 10 client.9690 ms_handle_connect on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:00.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.958+0000 7f00910b3700 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x1266880 0x12620f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d0b20d43705b160 server_cookie=f7ee991a4354a19d in_seq=0 out_seq=0 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f006bfff700 1 -- 172.21.15.119:0/522452144 <== mds.0 v2:172.21.15.145:6834/3901706437 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x104b460 con 0x1266880 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f006bfff700 10 client.9690 handle_command_reply: tid=0 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 client.9690 shutdown 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 msgr2=0x1264e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x1264e50 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7f008400df40 tx=0x7f008400df70 comp rx=0 tx=0).stop 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 shutdown_connections 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x1266880 0x12620f0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x1264e50 unknown :-1 s=CLOSED pgs=1817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f006bfff700 0 client.9690 ms_handle_reset on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:00.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12ac230 0x1264910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 --2- 172.21.15.119:0/522452144 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x1286c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 >> 172.21.15.119:0/522452144 conn(0xff44d0 msgr2=0x1287470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 shutdown_connections 2024-09-13T16:36:00.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 1 -- 172.21.15.119:0/522452144 wait complete. 2024-09-13T16:36:00.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.962+0000 7f00965c8740 20 client.9690 trim_cache size 0 max 16384 2024-09-13T16:36:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 Processor -- start 2024-09-13T16:36:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- start start 2024-09-13T16:36:00.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x12bc190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x1263e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12acb50 0x1264360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x11fbec0 con 0x12ac230 2024-09-13T16:36:00.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x11974f0 con 0x12acb50 2024-09-13T16:36:00.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x118f0e0 con 0x128ae90 2024-09-13T16:36:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x1263e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x1263e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56404/0 (socket says 172.21.15.119:56404) 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 -- 172.21.15.119:0/2335163530 learned_addr learned my addr 172.21.15.119:0/2335163530 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00918b4700 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12acb50 0x1264360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00910b3700 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x12bc190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 -- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 msgr2=0x12bc190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x12bc190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 -- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12acb50 msgr2=0x1264360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12acb50 0x1264360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 -- 172.21.15.119:0/2335163530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1179450 con 0x12ac230 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00908b2700 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x1263e20 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f007c00ce00 tx=0x7f007c073040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eedde32a976630f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00777fe700 1 -- 172.21.15.119:0/2335163530 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c011e20 con 0x12ac230 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00777fe700 1 -- 172.21.15.119:0/2335163530 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f007c0555b0 con 0x12ac230 2024-09-13T16:36:00.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00777fe700 1 -- 172.21.15.119:0/2335163530 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f007c054eb0 con 0x12ac230 2024-09-13T16:36:00.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- 172.21.15.119:0/2335163530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 msgr2=0x1263e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x1263e20 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7f007c00ce00 tx=0x7f007c073040 comp rx=0 tx=0).stop 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- 172.21.15.119:0/2335163530 shutdown_connections 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x12acb50 0x1264360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12ac230 0x1263e20 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 --2- 172.21.15.119:0/2335163530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x128ae90 0x12bc190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- 172.21.15.119:0/2335163530 >> 172.21.15.119:0/2335163530 conn(0xff44d0 msgr2=0x12bba80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- 172.21.15.119:0/2335163530 shutdown_connections 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.982+0000 7f00965c8740 1 -- 172.21.15.119:0/2335163530 wait complete. 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 -- start start 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 Processor -- start 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x130cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ac230 0x130d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x130d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x12b1730 con 0x12acb50 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1264ff0 con 0x128ae90 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x128a650 con 0x12ac230 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 10 client.0 _ll_register_callbacks cb 0x117ece0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x130d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x130d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56408/0 (socket says 172.21.15.119:56408) 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 -- 172.21.15.119:0/4096728614 learned_addr learned my addr 172.21.15.119:0/4096728614 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00910b3700 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x130cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00908b2700 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ac230 0x130d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 -- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ac230 msgr2=0x130d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ac230 0x130d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 -- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 msgr2=0x130cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:00.991 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x130cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 -- 172.21.15.119:0/4096728614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c1f7130 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00918b4700 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x130d790 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f008400a6f0 tx=0x7f0084004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f380f2cb5f8cbaf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f006bfff700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f006bfff700 1 -- 172.21.15.119:0/4096728614 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084007ac0 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f006bfff700 1 -- 172.21.15.119:0/4096728614 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f0084007e60 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0088003a20 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f006bfff700 1 -- 172.21.15.119:0/4096728614 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7f008400c700 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 10 client.9696 fetch_fsmap learned FSMap version 92 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0x116d720 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f006bfff700 1 -- 172.21.15.119:0/4096728614 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0088003a20 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f006bfff700 1 -- 172.21.15.119:0/4096728614 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 92) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0084003a40 con 0x12acb50 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 10 client.9696 fetch_fsmap finished waiting for FSMap version 92 2024-09-13T16:36:00.992 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 10 client.9696 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:36:00.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 20 client.9696 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:00.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x130e650 0x12aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:00.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 4 client.9696 mds_command: new command op to 9546 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:36:00.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 --> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] -- command(tid 0: {"prefix": "status"}) v1 -- 0x10701e0 con 0x130e650 2024-09-13T16:36:00.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f00910b3700 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x130e650 0x12aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:00.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.986+0000 7f006bfff700 10 client.9696 ms_handle_connect on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:00.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00910b3700 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x130e650 0x12aef40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=599f1f14d4e81d99 server_cookie=695804e1fdf2c3a1 in_seq=0 out_seq=0 2024-09-13T16:36:00.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f006bfff700 1 -- 172.21.15.119:0/4096728614 <== mds.0 v2:172.21.15.145:6834/3901706437 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x10701e0 con 0x130e650 2024-09-13T16:36:00.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f006bfff700 10 client.9696 handle_command_reply: tid=0 2024-09-13T16:36:00.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 client.9696 shutdown 2024-09-13T16:36:00.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 msgr2=0x130d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x130d790 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7f008400a6f0 tx=0x7f0084004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 shutdown_connections 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x130e650 0x12aef40 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x12acb50 0x130d790 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x12ac230 0x130d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f006bfff700 0 client.9696 ms_handle_reset on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 --2- 172.21.15.119:0/4096728614 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x128ae90 0x130cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 >> 172.21.15.119:0/4096728614 conn(0xff44d0 msgr2=0x12bc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 shutdown_connections 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 1 -- 172.21.15.119:0/4096728614 wait complete. 2024-09-13T16:36:00.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00965c8740 20 client.9696 trim_cache size 0 max 16384 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9546, 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 5.7425035229999999, 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 92, 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 126, 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 126, 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 10.196578709000001 2024-09-13T16:36:00.996 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0070066730 msgr2=0x7f0070068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0070066730 0x7f0070068bf0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f008c0b3a60 tx=0x7f0088008030 comp rx=0 tx=0).stop 2024-09-13T16:36:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0b2200 msgr2=0x7f008c0ac5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0b2200 0x7f008c0ac5a0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f007c00c4e0 tx=0x7f007c00c7f0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 shutdown_connections 2024-09-13T16:36:00.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0070066730 0x7f0070068bf0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f008c0cd8d0 0x7f008c0afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f008c0b2200 0x7f008c0ac5a0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 --2- 172.21.15.119:0/2340283656 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f008c0b17f0 0x7f008c0a9db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:00.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 >> 172.21.15.119:0/2340283656 conn(0x7f008c06a350 msgr2=0x7f008c0caa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:00.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 shutdown_connections 2024-09-13T16:36:00.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:00.990+0000 7f00920b5700 1 -- 172.21.15.119:0/2340283656 wait complete. 2024-09-13T16:36:01.009 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:36:01.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.118+0000 7fcea95a2700 1 Processor -- start 2024-09-13T16:36:01.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.118+0000 7fcea95a2700 1 -- start start 2024-09-13T16:36:01.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcea40ce300 0x7fcea4164520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:01.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40b2050 con 0x7fcea40b36f0 2024-09-13T16:36:01.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcea40b2320 con 0x7fcea40b3ad0 2024-09-13T16:36:01.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcea40b2600 con 0x7fcea40ce300 2024-09-13T16:36:01.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea2ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcea40b36f0 0x7fcea40b1940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57516/0 (socket says 172.21.15.119:57516) 2024-09-13T16:36:01.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea2ffd700 1 -- 172.21.15.119:0/655899468 learned_addr learned my addr 172.21.15.119:0/655899468 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:01.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea37fe700 1 --2- 172.21.15.119:0/655899468 >> v2:172.21.15.145:3300/0 conn(0x7fcea40ce300 0x7fcea4164520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:01.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea1ffb700 1 -- 172.21.15.119:0/655899468 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133595034 0 0) 0x7fcea40b2050 con 0x7fcea40b36f0 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea1ffb700 1 -- 172.21.15.119:0/655899468 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce8c0057e0 con 0x7fcea40b36f0 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea37fe700 1 -- 172.21.15.119:0/655899468 >> v1:172.21.15.145:6789/0 conn(0x7fcea40b3ad0 legacy=0x7fcea4159340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea37fe700 1 -- 172.21.15.119:0/655899468 >> v1:172.21.15.119:6789/0 conn(0x7fcea40b36f0 legacy=0x7fcea40b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea37fe700 1 -- 172.21.15.119:0/655899468 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea40b28e0 con 0x7fcea40ce300 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea37fe700 1 --2- 172.21.15.119:0/655899468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 0x7fcea4164520 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009d90 tx=0x7fce9c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=442afcf9b3e21924 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea1ffb700 1 -- 172.21.15.119:0/655899468 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c00e070 con 0x7fcea40ce300 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea1ffb700 1 -- 172.21.15.119:0/655899468 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce9c00bb20 con 0x7fcea40ce300 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea1ffb700 1 -- 172.21.15.119:0/655899468 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9c012430 con 0x7fcea40ce300 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- 172.21.15.119:0/655899468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 msgr2=0x7fcea4164520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 --2- 172.21.15.119:0/655899468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 0x7fcea4164520 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fce9c009d90 tx=0x7fce9c00b330 comp rx=0 tx=0).stop 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- 172.21.15.119:0/655899468 shutdown_connections 2024-09-13T16:36:01.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 --2- 172.21.15.119:0/655899468 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 0x7fcea4164520 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:01.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- 172.21.15.119:0/655899468 >> 172.21.15.119:0/655899468 conn(0x7fcea406a350 msgr2=0x7fcea406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:01.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- 172.21.15.119:0/655899468 shutdown_connections 2024-09-13T16:36:01.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.122+0000 7fcea95a2700 1 -- 172.21.15.119:0/655899468 wait complete. 2024-09-13T16:36:01.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 Processor -- start 2024-09-13T16:36:01.128 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 -- start start 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcea40b36f0 0x7fcea414f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40b3ad0 0x7fcea414e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 0x7fcea413cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4153610 con 0x7fcea40b36f0 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4153830 con 0x7fcea40ce300 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea95a2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcea4153b10 con 0x7fcea40b3ad0 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcea40b36f0 0x7fcea414f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcea40b36f0 0x7fcea414f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56412/0 (socket says 172.21.15.119:56412) 2024-09-13T16:36:01.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea2ffd700 1 -- 172.21.15.119:0/2244642227 learned_addr learned my addr 172.21.15.119:0/2244642227 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea27fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40b3ad0 0x7fcea414e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.126+0000 7fcea37fe700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 0x7fcea413cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea2ffd700 1 -- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40b3ad0 msgr2=0x7fcea414e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea2ffd700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40b3ad0 0x7fcea414e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea2ffd700 1 -- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 msgr2=0x7fcea413cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea2ffd700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 0x7fcea413cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea2ffd700 1 -- 172.21.15.119:0/2244642227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9c009a70 con 0x7fcea40b36f0 2024-09-13T16:36:01.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea2ffd700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcea40b36f0 0x7fcea414f140 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fce9400c700 tx=0x7fce9400ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:01.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fce8b7fe700 1 -- 172.21.15.119:0/2244642227 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94005d90 con 0x7fcea40b36f0 2024-09-13T16:36:01.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fce8b7fe700 1 -- 172.21.15.119:0/2244642227 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce940070d0 con 0x7fcea40b36f0 2024-09-13T16:36:01.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea4153df0 con 0x7fcea40b36f0 2024-09-13T16:36:01.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fce8b7fe700 1 -- 172.21.15.119:0/2244642227 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce94007480 con 0x7fcea40b36f0 2024-09-13T16:36:01.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea414a830 con 0x7fcea40b36f0 2024-09-13T16:36:01.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.130+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea4067760 con 0x7fcea40b36f0 2024-09-13T16:36:01.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.134+0000 7fce8b7fe700 1 -- 172.21.15.119:0/2244642227 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fce940075e0 con 0x7fcea40b36f0 2024-09-13T16:36:01.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.134+0000 7fce8b7fe700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce90066730 0x7fce90068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:01.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.134+0000 7fce8b7fe700 1 -- 172.21.15.119:0/2244642227 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fce94083690 con 0x7fcea40b36f0 2024-09-13T16:36:01.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.134+0000 7fcea27fc700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce90066730 0x7fce90068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:01.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.134+0000 7fcea27fc700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce90066730 0x7fce90068bf0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcea40b1f80 tx=0x7fce98009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:01.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.142+0000 7fce8b7fe700 1 -- 172.21.15.119:0/2244642227 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fce940508b0 con 0x7fcea40b36f0 2024-09-13T16:36:01.203 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-09-13T16:36:01.202+0000 7f55623e3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:36:01.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.350+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcea40cb440 con 0x7fcea40b36f0 2024-09-13T16:36:01.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.350+0000 7fce8b7fe700 1 -- 172.21.15.119:0/2244642227 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7fce94017060 con 0x7fcea40b36f0 2024-09-13T16:36:01.352 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:36:01.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce90066730 msgr2=0x7fce90068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:01.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce90066730 0x7fce90068bf0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fcea40b1f80 tx=0x7fce98009040 comp rx=0 tx=0).stop 2024-09-13T16:36:01.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcea40b36f0 msgr2=0x7fcea414f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:01.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcea40b36f0 0x7fcea414f140 secure :-1 s=READY pgs=1821 cs=0 l=1 rev1=1 crypto rx=0x7fce9400c700 tx=0x7fce9400ca10 comp rx=0 tx=0).stop 2024-09-13T16:36:01.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 shutdown_connections 2024-09-13T16:36:01.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce90066730 0x7fce90068bf0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:01.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcea40ce300 0x7fcea413cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:01.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcea40b3ad0 0x7fcea414e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:01.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 --2- 172.21.15.119:0/2244642227 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcea40b36f0 0x7fcea414f140 unknown :-1 s=CLOSED pgs=1821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:01.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 >> 172.21.15.119:0/2244642227 conn(0x7fcea406a350 msgr2=0x7fcea40af4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:01.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 shutdown_connections 2024-09-13T16:36:01.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:01.354+0000 7fcea95a2700 1 -- 172.21.15.119:0/2244642227 wait complete. 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:36:01.368 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:01.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.369 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:36:01.373 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:36:01.374 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 41980sec preferred_lft 41980sec 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.375 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:01.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.377 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:36:01.389 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:36:01.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.390 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:36:01.443 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:36:01.443 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:36:01.443 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:36:01.443 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:36:01.444 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:36:01.444 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:36:01.444 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:36:01.444 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:36:01.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.444 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:36:01.444 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:36:01.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:36:01.444 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:36:01.445 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:36:01.445 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:36:01.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.484 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:36:01.491 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:36:01.491 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:36:01.492 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:01.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.492 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:36:01.506 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:36:01.507 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:01.507 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:01.559 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:01.559 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:01.559 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:01.566 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:01.624 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:01.624 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:01.638 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:01.638 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:01.639 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.639 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:01.645 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:36:01.645 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:36:01.648 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:01.664 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:01.665 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:01.674 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:01.675 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:01.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:01.675 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:01.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:36:01.726+0000 7f1398c4c640 -1 init, newargv = 0x56226f8d5490 newargc=15 2024-09-13T16:36:01.731 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[53667]: starting ceph client 2024-09-13T16:36:01.741 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[53667]: starting fuse 2024-09-13T16:36:02.782 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:02.800 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:02.800 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:02.857 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:02.859 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:02.859 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:02.864 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:36:02.864 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:36:02.864 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:02.864 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:36:02.864 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:36:02.864 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:02.864 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:36:02.865 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:36:02.866 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:02.947 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.53667.asok 2024-09-13T16:36:03.132 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53667.asok 2024-09-13T16:36:03.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:03.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53667.asok status 2024-09-13T16:36:03.283 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:03.283 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:36:03.283 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:36:03.283 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "53667", 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9705, 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:36:03.284 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 9705 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4246575945 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4246575945 2024-09-13T16:36:03.285 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.9705 192.168.0.1:0/4246575945", 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/4246575945", 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 92, 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 126, 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:36:03.286 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:03.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:03.294 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:36:03.294 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:03.301 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:36:03.301 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:03.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:03.301 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:03.316 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:03.316 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:03.316 DEBUG:teuthology.orchestra.run.smithi119:> mount_point = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:36:03.316 DEBUG:teuthology.orchestra.run.smithi119:> parent = os.path.join(mount_point, "parent") 2024-09-13T16:36:03.316 DEBUG:teuthology.orchestra.run.smithi119:> os.mkdir(parent) 2024-09-13T16:36:03.316 DEBUG:teuthology.orchestra.run.smithi119:> flushed = os.path.join(parent, "flushed") 2024-09-13T16:36:03.316 DEBUG:teuthology.orchestra.run.smithi119:> os.mkdir(flushed) 2024-09-13T16:36:03.317 DEBUG:teuthology.orchestra.run.smithi119:> for f in ["alpha", "bravo", "charlie"]: 2024-09-13T16:36:03.317 DEBUG:teuthology.orchestra.run.smithi119:> open(os.path.join(flushed, f), '"'"'w'"'"').write(f) 2024-09-13T16:36:03.317 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:03.571 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:03.571 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-09-13T16:36:03.605 INFO:teuthology.orchestra.run.smithi119.stdout:./ 2024-09-13T16:36:03.605 INFO:teuthology.orchestra.run.smithi119.stdout:./parent 2024-09-13T16:36:03.605 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed 2024-09-13T16:36:03.605 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/charlie 2024-09-13T16:36:03.605 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/alpha 2024-09-13T16:36:03.605 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/bravo 2024-09-13T16:36:03.606 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:03.606 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:03.606 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:03.606 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:03.606 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-09-13T16:36:03.606 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:03.714 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:36:03.902 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:03.903 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:03.903 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:03.903 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:03.903 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-09-13T16:36:03.903 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:03.972 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627776 2024-09-13T16:36:04.160 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:04.161 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:04.161 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:04.161 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:04.161 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-09-13T16:36:04.161 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:04.228 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627777 2024-09-13T16:36:04.417 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:04.417 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:04.418 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:04.418 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:04.418 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-09-13T16:36:04.418 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:04.487 INFO:teuthology.orchestra.run.smithi119.stdout:1099511628280 2024-09-13T16:36:04.676 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:04.676 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:04.676 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:04.676 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:04.676 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-09-13T16:36:04.676 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:04.788 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627778 2024-09-13T16:36:04.977 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:04.977 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:04.977 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:04.977 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:04.977 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-09-13T16:36:04.977 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:05.040 INFO:teuthology.orchestra.run.smithi119.stdout:1099511628279 2024-09-13T16:36:05.230 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:05.231 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:05.239 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:05.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f157e731700 1 Processor -- start 2024-09-13T16:36:05.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f157e731700 1 -- start start 2024-09-13T16:36:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f157e731700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f15780cfb60 0x7f15780cee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f157e731700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806bae0 0x7f1578160dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f157e731700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1578160940 con 0x7f157806a350 2024-09-13T16:36:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f157e731700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f157806a850 con 0x7f15780cfb60 2024-09-13T16:36:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f157e731700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f157806ab30 con 0x7f157806bae0 2024-09-13T16:36:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f1577fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f15780cfb60 0x7f15780cee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f1577fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f15780cfb60 0x7f15780cee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49830/0 (socket says 172.21.15.119:49830) 2024-09-13T16:36:05.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.402+0000 7f1577fff700 1 -- 172.21.15.119:0/3862218634 learned_addr learned my addr 172.21.15.119:0/3862218634 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:05.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157ccce700 1 --2- 172.21.15.119:0/3862218634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806bae0 0x7f1578160dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1577fff700 1 -- 172.21.15.119:0/3862218634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806bae0 msgr2=0x7f1578160dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:05.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1577fff700 1 --2- 172.21.15.119:0/3862218634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806bae0 0x7f1578160dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.412 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1577fff700 1 -- 172.21.15.119:0/3862218634 >> v1:172.21.15.145:6789/0 conn(0x7f157806a350 legacy=0x7f157806b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1577fff700 1 -- 172.21.15.119:0/3862218634 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157806ae10 con 0x7f15780cfb60 2024-09-13T16:36:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1576ffd700 1 -- 172.21.15.119:0/3862218634 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2049628442 0 0) 0x7f1578160940 con 0x7f157806a350 2024-09-13T16:36:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1577fff700 1 --2- 172.21.15.119:0/3862218634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15780cfb60 0x7f15780cee00 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1568009910 tx=0x7f1568009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241177115cfac635 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1576ffd700 1 -- 172.21.15.119:0/3862218634 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568011830 con 0x7f15780cfb60 2024-09-13T16:36:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1576ffd700 1 -- 172.21.15.119:0/3862218634 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1568011990 con 0x7f15780cfb60 2024-09-13T16:36:05.413 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f1576ffd700 1 -- 172.21.15.119:0/3862218634 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568011d40 con 0x7f15780cfb60 2024-09-13T16:36:05.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 -- 172.21.15.119:0/3862218634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15780cfb60 msgr2=0x7f15780cee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 --2- 172.21.15.119:0/3862218634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15780cfb60 0x7f15780cee00 secure :-1 s=READY pgs=1823 cs=0 l=1 rev1=1 crypto rx=0x7f1568009910 tx=0x7f1568009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:05.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 -- 172.21.15.119:0/3862218634 shutdown_connections 2024-09-13T16:36:05.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 --2- 172.21.15.119:0/3862218634 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806bae0 0x7f1578160dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.414 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 --2- 172.21.15.119:0/3862218634 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f15780cfb60 0x7f15780cee00 unknown :-1 s=CLOSED pgs=1823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 -- 172.21.15.119:0/3862218634 >> 172.21.15.119:0/3862218634 conn(0x7f15780b3fd0 msgr2=0x7f15780b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 -- 172.21.15.119:0/3862218634 shutdown_connections 2024-09-13T16:36:05.415 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.406+0000 7f157e731700 1 -- 172.21.15.119:0/3862218634 wait complete. 2024-09-13T16:36:05.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 Processor -- start 2024-09-13T16:36:05.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 -- start start 2024-09-13T16:36:05.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806a350 0x7f15780afe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f157806bae0 0x7f157815fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15780cfb60 0x7f15781552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1578157b70 con 0x7f157806bae0 2024-09-13T16:36:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15781f28b0 con 0x7f15780cfb60 2024-09-13T16:36:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15780cf940 con 0x7f157806a350 2024-09-13T16:36:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f157806bae0 0x7f157815fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f157806bae0 0x7f157815fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49832/0 (socket says 172.21.15.119:49832) 2024-09-13T16:36:05.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 -- 172.21.15.119:0/1760980547 learned_addr learned my addr 172.21.15.119:0/1760980547 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:05.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f1577fff700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806a350 0x7f15780afe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157ccce700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15780cfb60 0x7f15781552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 -- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806a350 msgr2=0x7f15780afe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806a350 0x7f15780afe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 -- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15780cfb60 msgr2=0x7f15781552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15780cfb60 0x7f15781552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 -- 172.21.15.119:0/1760980547 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1568009570 con 0x7f157806bae0 2024-09-13T16:36:05.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f15777fe700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f157806bae0 0x7f157815fa30 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c550 tx=0x7f156c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:05.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f1574ff9700 1 -- 172.21.15.119:0/1760980547 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c0054e0 con 0x7f157806bae0 2024-09-13T16:36:05.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f1574ff9700 1 -- 172.21.15.119:0/1760980547 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f156c0036a0 con 0x7f157806bae0 2024-09-13T16:36:05.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1578153330 con 0x7f157806bae0 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f1574ff9700 1 -- 172.21.15.119:0/1760980547 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156c003a50 con 0x7f157806bae0 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.410+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15781538b0 con 0x7f157806bae0 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.414+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1578157cb0 con 0x7f157806bae0 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.414+0000 7f1574ff9700 1 -- 172.21.15.119:0/1760980547 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f156c003bb0 con 0x7f157806bae0 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.414+0000 7f1574ff9700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1564066730 0x7f1564068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.414+0000 7f1574ff9700 1 -- 172.21.15.119:0/1760980547 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f156c083590 con 0x7f157806bae0 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.414+0000 7f1577fff700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1564066730 0x7f1564068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.414+0000 7f1577fff700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1564066730 0x7f1564068bf0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1568000c00 tx=0x7f1568008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:05.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.422+0000 7f1574ff9700 1 -- 172.21.15.119:0/1760980547 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1578157cb0 con 0x7f157806bae0 2024-09-13T16:36:05.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.602+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1578157cb0 con 0x7f157806bae0 2024-09-13T16:36:05.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f1574ff9700 1 -- 172.21.15.119:0/1760980547 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16844 (secure 0 0 0) 0x7f1578157cb0 con 0x7f157806bae0 2024-09-13T16:36:05.607 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:05.607 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":126,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:56.248923+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:05.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1564066730 msgr2=0x7f1564068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1564066730 0x7f1564068bf0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f1568000c00 tx=0x7f1568008030 comp rx=0 tx=0).stop 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f157806bae0 msgr2=0x7f157815fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f157806bae0 0x7f157815fa30 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7f156c00c550 tx=0x7f156c00c860 comp rx=0 tx=0).stop 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 shutdown_connections 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1564066730 0x7f1564068bf0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f15780cfb60 0x7f15781552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f157806bae0 0x7f157815fa30 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 --2- 172.21.15.119:0/1760980547 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f157806a350 0x7f15780afe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 >> 172.21.15.119:0/1760980547 conn(0x7f15780b3fd0 msgr2=0x7f157815b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 shutdown_connections 2024-09-13T16:36:05.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.606+0000 7f157e731700 1 -- 172.21.15.119:0/1760980547 wait complete. 2024-09-13T16:36:05.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:05.623 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:05.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36dd753700 1 Processor -- start 2024-09-13T16:36:05.736 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36dd753700 1 -- start start 2024-09-13T16:36:05.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36dd753700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f36d80b4360 0x7f36d815b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36dd753700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b1d50 0x7f36d80b4740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36dd753700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d815cb70 con 0x7f36d80b4c80 2024-09-13T16:36:05.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36dd753700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d815bf60 con 0x7f36d80b4360 2024-09-13T16:36:05.737 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36dd753700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d815c240 con 0x7f36d80b1d50 2024-09-13T16:36:05.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b1d50 0x7f36d80b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d7fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f36d80b4360 0x7f36d815b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b1d50 0x7f36d80b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:33126/0 (socket says 172.21.15.119:33126) 2024-09-13T16:36:05.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 -- 172.21.15.119:0/3465561002 learned_addr learned my addr 172.21.15.119:0/3465561002 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:05.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 -- 172.21.15.119:0/3465561002 >> v2:172.21.15.145:3300/0 conn(0x7f36d80b4360 msgr2=0x7f36d815b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:05.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 --2- 172.21.15.119:0/3465561002 >> v2:172.21.15.145:3300/0 conn(0x7f36d80b4360 0x7f36d815b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 -- 172.21.15.119:0/3465561002 >> v1:172.21.15.145:6789/0 conn(0x7f36d80b4c80 legacy=0x7f36d815d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 -- 172.21.15.119:0/3465561002 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d815c520 con 0x7f36d80b1d50 2024-09-13T16:36:05.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.734+0000 7f36d77fe700 1 --2- 172.21.15.119:0/3465561002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b1d50 0x7f36d80b4740 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009910 tx=0x7f36c0009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=189d2b4e0a97c972 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:05.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36d6ffd700 1 -- 172.21.15.119:0/3465561002 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0011830 con 0x7f36d80b1d50 2024-09-13T16:36:05.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36d6ffd700 1 -- 172.21.15.119:0/3465561002 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c0011990 con 0x7f36d80b1d50 2024-09-13T16:36:05.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36d6ffd700 1 -- 172.21.15.119:0/3465561002 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0011d40 con 0x7f36d80b1d50 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 -- 172.21.15.119:0/3465561002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b1d50 msgr2=0x7f36d80b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 --2- 172.21.15.119:0/3465561002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b1d50 0x7f36d80b4740 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f36c0009910 tx=0x7f36c0009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 -- 172.21.15.119:0/3465561002 shutdown_connections 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 --2- 172.21.15.119:0/3465561002 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b1d50 0x7f36d80b4740 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 --2- 172.21.15.119:0/3465561002 >> v2:172.21.15.145:3300/0 conn(0x7f36d80b4360 0x7f36d815b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 -- 172.21.15.119:0/3465561002 >> 172.21.15.119:0/3465561002 conn(0x7f36d8072240 msgr2=0x7f36d8099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 -- 172.21.15.119:0/3465561002 shutdown_connections 2024-09-13T16:36:05.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 -- 172.21.15.119:0/3465561002 wait complete. 2024-09-13T16:36:05.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 Processor -- start 2024-09-13T16:36:05.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.738+0000 7f36dd753700 1 -- start start 2024-09-13T16:36:05.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d80b1d50 0x7f36d8154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b4360 0x7f36d81f0190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d80b4c80 0x7f36d81f4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81557e0 con 0x7f36d80b4c80 2024-09-13T16:36:05.743 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81f3d10 con 0x7f36d80b1d50 2024-09-13T16:36:05.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d81f42d0 con 0x7f36d80b4360 2024-09-13T16:36:05.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dcf52700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d80b4c80 0x7f36d81f4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d80b1d50 0x7f36d8154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d77fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b4360 0x7f36d81f0190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dcf52700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d80b4c80 0x7f36d81f4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49846/0 (socket says 172.21.15.119:49846) 2024-09-13T16:36:05.744 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dcf52700 1 -- 172.21.15.119:0/3034538357 learned_addr learned my addr 172.21.15.119:0/3034538357 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:05.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d77fe700 1 -- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d80b1d50 msgr2=0x7f36d8154bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d77fe700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d80b1d50 0x7f36d8154bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d77fe700 1 -- 172.21.15.119:0/3034538357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d80b4c80 msgr2=0x7f36d81f4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.745 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d77fe700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d80b4c80 0x7f36d81f4760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d77fe700 1 -- 172.21.15.119:0/3034538357 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c0009570 con 0x7f36d80b4360 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dcf52700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d80b4c80 0x7f36d81f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d77fe700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b4360 0x7f36d81f0190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f36d809a640 tx=0x7f36c0004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d4ff9700 1 -- 172.21.15.119:0/3034538357 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c0004300 con 0x7f36d80b4360 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d4ff9700 1 -- 172.21.15.119:0/3034538357 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36c001f070 con 0x7f36d80b4360 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36d4ff9700 1 -- 172.21.15.119:0/3034538357 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36c00036c0 con 0x7f36d80b4360 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d8155220 con 0x7f36d80b4360 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d81f3f60 con 0x7f36d80b4360 2024-09-13T16:36:05.746 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.742+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d80d18a0 con 0x7f36d80b4360 2024-09-13T16:36:05.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.746+0000 7f36d4ff9700 1 -- 172.21.15.119:0/3034538357 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f36c0021030 con 0x7f36d80b4360 2024-09-13T16:36:05.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.746+0000 7f36d4ff9700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36b8066730 0x7f36b8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:05.747 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.746+0000 7f36d4ff9700 1 -- 172.21.15.119:0/3034538357 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f36c0085480 con 0x7f36d80b4360 2024-09-13T16:36:05.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.746+0000 7f36d7fff700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36b8066730 0x7f36b8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:05.748 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.746+0000 7f36d7fff700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36b8066730 0x7f36b8068bf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f36d8157ae0 tx=0x7f36c800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:05.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.758+0000 7f36d4ff9700 1 -- 172.21.15.119:0/3034538357 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f36c0052740 con 0x7f36d80b4360 2024-09-13T16:36:05.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.934+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f36d81f40f0 con 0x7f36d80b4360 2024-09-13T16:36:05.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.934+0000 7f36d4ff9700 1 -- 172.21.15.119:0/3034538357 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f36c0006da0 con 0x7f36d80b4360 2024-09-13T16:36:05.938 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36b8066730 msgr2=0x7f36b8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36b8066730 0x7f36b8068bf0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f36d8157ae0 tx=0x7f36c800b040 comp rx=0 tx=0).stop 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b4360 msgr2=0x7f36d81f0190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b4360 0x7f36d81f0190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f36d809a640 tx=0x7f36c0004770 comp rx=0 tx=0).stop 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 shutdown_connections 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f36b8066730 0x7f36b8068bf0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f36d80b4c80 0x7f36d81f4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f36d80b4360 0x7f36d81f0190 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 --2- 172.21.15.119:0/3034538357 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f36d80b1d50 0x7f36d8154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:05.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 >> 172.21.15.119:0/3034538357 conn(0x7f36d8072240 msgr2=0x7f36d8159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 shutdown_connections 2024-09-13T16:36:05.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:05.938+0000 7f36dd753700 1 -- 172.21.15.119:0/3034538357 wait complete. 2024-09-13T16:36:05.952 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.066+0000 7fd902da0700 1 Processor -- start 2024-09-13T16:36:06.070 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.066+0000 7fd902da0700 1 -- start start 2024-09-13T16:36:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd8fc0b1cd0 0x7fd8fc15efa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.072 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8fc0b3540 con 0x7fd8fc0b1870 2024-09-13T16:36:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8fc0b12e0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0cb520 con 0x7fd8fc15aca0 2024-09-13T16:36:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd8fc0b1cd0 0x7fd8fc15efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd8fc0b1cd0 0x7fd8fc15efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49862/0 (socket says 172.21.15.119:49862) 2024-09-13T16:36:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd8f3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd8fc0b1870 0x7fd8fc15e890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41920/0 (socket says 172.21.15.119:41920) 2024-09-13T16:36:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 -- 172.21.15.119:0/676650992 learned_addr learned my addr 172.21.15.119:0/676650992 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:06.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd900b3c700 1 --2- 172.21.15.119:0/676650992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 -- 172.21.15.119:0/676650992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 msgr2=0x7fd8fc0cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 --2- 172.21.15.119:0/676650992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd8f37fe700 1 -- 172.21.15.119:0/676650992 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851922017 0 0) 0x7fd8fc0b3540 con 0x7fd8fc0b1870 2024-09-13T16:36:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 -- 172.21.15.119:0/676650992 >> v1:172.21.15.119:6789/0 conn(0x7fd8fc0b1870 legacy=0x7fd8fc15e890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 -- 172.21.15.119:0/676650992 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc0b06d0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd90133d700 1 --2- 172.21.15.119:0/676650992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc15efa0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009e50 tx=0x7fd8f400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51eec6915ea33533 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd8f37fe700 1 -- 172.21.15.119:0/676650992 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400e070 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd8f37fe700 1 -- 172.21.15.119:0/676650992 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f400b910 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd8f37fe700 1 -- 172.21.15.119:0/676650992 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f400bcc0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- 172.21.15.119:0/676650992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 msgr2=0x7fd8fc15efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 --2- 172.21.15.119:0/676650992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc15efa0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fd8f4009e50 tx=0x7fd8f400b330 comp rx=0 tx=0).stop 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- 172.21.15.119:0/676650992 shutdown_connections 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 --2- 172.21.15.119:0/676650992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc15efa0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 --2- 172.21.15.119:0/676650992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- 172.21.15.119:0/676650992 >> 172.21.15.119:0/676650992 conn(0x7fd8fc0b26e0 msgr2=0x7fd8fc0b2af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- 172.21.15.119:0/676650992 shutdown_connections 2024-09-13T16:36:06.076 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.070+0000 7fd902da0700 1 -- 172.21.15.119:0/676650992 wait complete. 2024-09-13T16:36:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 Processor -- start 2024-09-13T16:36:06.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 -- start start 2024-09-13T16:36:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc0b1870 0x7fd8fc159f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc1523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0b4ef0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0b4ce0 con 0x7fd8fc0b1870 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd902da0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8fc0b5140 con 0x7fd8fc15aca0 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd8f3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc1523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd8f3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc1523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49870/0 (socket says 172.21.15.119:49870) 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd8f3fff700 1 -- 172.21.15.119:0/50038416 learned_addr learned my addr 172.21.15.119:0/50038416 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd900b3c700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc0b1870 0x7fd8fc159f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.079 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.074+0000 7fd90133d700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f3fff700 1 -- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 msgr2=0x7fd8fc0ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f3fff700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f3fff700 1 -- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc0b1870 msgr2=0x7fd8fc159f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f3fff700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc0b1870 0x7fd8fc159f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f3fff700 1 -- 172.21.15.119:0/50038416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8f4009b30 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.081 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd90133d700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd900b3c700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc0b1870 0x7fd8fc159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f3fff700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc1523a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c4b0 tx=0x7fd8ec00c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f17fa700 1 -- 172.21.15.119:0/50038416 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec004ca0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f17fa700 1 -- 172.21.15.119:0/50038416 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8ec004e00 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8fc0b5420 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f17fa700 1 -- 172.21.15.119:0/50038416 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8ec0072d0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.082 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8fc0b59e0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8fc0cf6f0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f17fa700 1 -- 172.21.15.119:0/50038416 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd8ec007430 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.078+0000 7fd8f17fa700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066760 0x7fd8e8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.082+0000 7fd8f17fa700 1 -- 172.21.15.119:0/50038416 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd8ec083d90 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.083 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.082+0000 7fd900b3c700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066760 0x7fd8e8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.084 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.082+0000 7fd900b3c700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066760 0x7fd8e8068c20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80112c0 tx=0x7fd8f8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.089 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.086+0000 7fd8f17fa700 1 -- 172.21.15.119:0/50038416 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd8ec050fd0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.250+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd8fc0b5b70 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.250+0000 7fd8f17fa700 1 -- 172.21.15.119:0/50038416 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd8ec0545f0 con 0x7fd8fc0b1cd0 2024-09-13T16:36:06.254 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066760 msgr2=0x7fd8e8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066760 0x7fd8e8068c20 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80112c0 tx=0x7fd8f8009240 comp rx=0 tx=0).stop 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 msgr2=0x7fd8fc1523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc1523a0 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fd8ec00c4b0 tx=0x7fd8ec00c7c0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 shutdown_connections 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd8e8066760 0x7fd8e8068c20 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd8fc15aca0 0x7fd8fc0ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8fc0b1cd0 0x7fd8fc1523a0 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 --2- 172.21.15.119:0/50038416 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd8fc0b1870 0x7fd8fc159f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 >> 172.21.15.119:0/50038416 conn(0x7fd8fc0b26e0 msgr2=0x7fd8fc06e550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 shutdown_connections 2024-09-13T16:36:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.254+0000 7fd902da0700 1 -- 172.21.15.119:0/50038416 wait complete. 2024-09-13T16:36:06.269 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:06.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.386+0000 7f066ecc2700 1 Processor -- start 2024-09-13T16:36:06.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.386+0000 7f066ecc2700 1 -- start start 2024-09-13T16:36:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f066815cae0 0x7f06680ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06680cd600 con 0x7f06680d0fb0 2024-09-13T16:36:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06680d0dd0 con 0x7f06680cff60 2024-09-13T16:36:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06680ce1d0 con 0x7f066815cae0 2024-09-13T16:36:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066dcc0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f066815cae0 0x7f06680ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066e4c1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f06680d0fb0 0x7f06680b20f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41924/0 (socket says 172.21.15.119:41924) 2024-09-13T16:36:06.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066e4c1700 1 -- 172.21.15.119:0/498821424 learned_addr learned my addr 172.21.15.119:0/498821424 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:06.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ccbe700 1 -- 172.21.15.119:0/498821424 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 279116457 0 0) 0x7f06680cd600 con 0x7f06680d0fb0 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ccbe700 1 -- 172.21.15.119:0/498821424 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06500035d0 con 0x7f06680d0fb0 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ccbe700 1 -- 172.21.15.119:0/498821424 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617857597 0 0) 0x7f06680d0dd0 con 0x7f06680cff60 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ccbe700 1 -- 172.21.15.119:0/498821424 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06680cd600 con 0x7f06680cff60 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066dcc0700 1 -- 172.21.15.119:0/498821424 >> v1:172.21.15.145:6789/0 conn(0x7f06680cff60 legacy=0x7f06680d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066dcc0700 1 -- 172.21.15.119:0/498821424 >> v1:172.21.15.119:6789/0 conn(0x7f06680d0fb0 legacy=0x7f06680b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066dcc0700 1 -- 172.21.15.119:0/498821424 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06680cd830 con 0x7f066815cae0 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066dcc0700 1 --2- 172.21.15.119:0/498821424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f066815cae0 0x7f06680ce5f0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f0658009850 tx=0x7f0658009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444268b936ca3d17 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ccbe700 1 -- 172.21.15.119:0/498821424 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658011830 con 0x7f066815cae0 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ccbe700 1 -- 172.21.15.119:0/498821424 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0658011990 con 0x7f066815cae0 2024-09-13T16:36:06.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ccbe700 1 -- 172.21.15.119:0/498821424 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0658011d40 con 0x7f066815cae0 2024-09-13T16:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- 172.21.15.119:0/498821424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f066815cae0 msgr2=0x7f06680ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 --2- 172.21.15.119:0/498821424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f066815cae0 0x7f06680ce5f0 secure :-1 s=READY pgs=1829 cs=0 l=1 rev1=1 crypto rx=0x7f0658009850 tx=0x7f0658009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- 172.21.15.119:0/498821424 shutdown_connections 2024-09-13T16:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 --2- 172.21.15.119:0/498821424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f066815cae0 0x7f06680ce5f0 unknown :-1 s=CLOSED pgs=1829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- 172.21.15.119:0/498821424 >> 172.21.15.119:0/498821424 conn(0x7f06680d29f0 msgr2=0x7f06680d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- 172.21.15.119:0/498821424 shutdown_connections 2024-09-13T16:36:06.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- 172.21.15.119:0/498821424 wait complete. 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 Processor -- start 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.390+0000 7f066ecc2700 1 -- start start 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06680cff60 0x7f066815a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06680d0fb0 0x7f06681f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f066815cae0 0x7f06680afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f066815a860 con 0x7f06680d0fb0 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06681f38d0 con 0x7f06680cff60 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06681f35a0 con 0x7f066815cae0 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06680d0fb0 0x7f06681f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06680d0fb0 0x7f06681f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49896/0 (socket says 172.21.15.119:49896) 2024-09-13T16:36:06.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 -- 172.21.15.119:0/3366702465 learned_addr learned my addr 172.21.15.119:0/3366702465 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066dcc0700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06680cff60 0x7f066815a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066e4c1700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f066815cae0 0x7f06680afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 -- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f066815cae0 msgr2=0x7f06680afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f066815cae0 0x7f06680afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 -- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06680cff60 msgr2=0x7f066815a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06680cff60 0x7f066815a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 -- 172.21.15.119:0/3366702465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06580094b0 con 0x7f06680d0fb0 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066dcc0700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06680cff60 0x7f066815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:06.397 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066e4c1700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f066815cae0 0x7f06680afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:06.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066d4bf700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06680d0fb0 0x7f06681f2380 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f065c00c4c0 tx=0x7f065c00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f06667fc700 1 -- 172.21.15.119:0/3366702465 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c0052d0 con 0x7f06680d0fb0 2024-09-13T16:36:06.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f06667fc700 1 -- 172.21.15.119:0/3366702465 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f065c005430 con 0x7f06680d0fb0 2024-09-13T16:36:06.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06681f2ca0 con 0x7f06680d0fb0 2024-09-13T16:36:06.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f06667fc700 1 -- 172.21.15.119:0/3366702465 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f065c0055a0 con 0x7f06680d0fb0 2024-09-13T16:36:06.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06681f3260 con 0x7f06680d0fb0 2024-09-13T16:36:06.398 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.394+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06680ce5f0 con 0x7f06680d0fb0 2024-09-13T16:36:06.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.398+0000 7f06667fc700 1 -- 172.21.15.119:0/3366702465 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f065c0038e0 con 0x7f06680d0fb0 2024-09-13T16:36:06.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.398+0000 7f06667fc700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0654066730 0x7f0654068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.398+0000 7f06667fc700 1 -- 172.21.15.119:0/3366702465 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f065c083810 con 0x7f06680d0fb0 2024-09-13T16:36:06.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.398+0000 7f066dcc0700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0654066730 0x7f0654068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.398+0000 7f066dcc0700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0654066730 0x7f0654068bf0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f06580090b0 tx=0x7f0658008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.406+0000 7f06667fc700 1 -- 172.21.15.119:0/3366702465 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f065c050a40 con 0x7f06680d0fb0 2024-09-13T16:36:06.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f06681f3a10 con 0x7f06680d0fb0 2024-09-13T16:36:06.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f06667fc700 1 -- 172.21.15.119:0/3366702465 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7f065c0170a0 con 0x7f06680d0fb0 2024-09-13T16:36:06.576 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0654066730 msgr2=0x7f0654068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0654066730 0x7f0654068bf0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f06580090b0 tx=0x7f0658008030 comp rx=0 tx=0).stop 2024-09-13T16:36:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06680d0fb0 msgr2=0x7f06681f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06680d0fb0 0x7f06681f2380 secure :-1 s=READY pgs=1830 cs=0 l=1 rev1=1 crypto rx=0x7f065c00c4c0 tx=0x7f065c00c880 comp rx=0 tx=0).stop 2024-09-13T16:36:06.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 shutdown_connections 2024-09-13T16:36:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0654066730 0x7f0654068bf0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f066815cae0 0x7f06680afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f06680d0fb0 0x7f06681f2380 unknown :-1 s=CLOSED pgs=1830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 --2- 172.21.15.119:0/3366702465 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f06680cff60 0x7f066815a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 >> 172.21.15.119:0/3366702465 conn(0x7f06680d29f0 msgr2=0x7f06680b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 shutdown_connections 2024-09-13T16:36:06.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.574+0000 7f066ecc2700 1 -- 172.21.15.119:0/3366702465 wait complete. 2024-09-13T16:36:06.591 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:36:06.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.706+0000 7f7d4ccc0700 1 Processor -- start 2024-09-13T16:36:06.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.706+0000 7f7d4ccc0700 1 -- start start 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.706+0000 7f7d4ccc0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7d4815e840 0x7f7d48070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.706+0000 7f7d4ccc0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7d4815ec20 0x7f7d48161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.706+0000 7f7d4ccc0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d48071210 con 0x7f7d48164a70 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.706+0000 7f7d4ccc0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480714f0 con 0x7f7d4815ec20 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.706+0000 7f7d4ccc0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d480717d0 con 0x7f7d4815e840 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7d4815ec20 0x7f7d48161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7d4815ec20 0x7f7d48161f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49906/0 (socket says 172.21.15.119:49906) 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 -- 172.21.15.119:0/322209940 learned_addr learned my addr 172.21.15.119:0/322209940 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:06.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4659c700 1 --2- 172.21.15.119:0/322209940 >> v2:172.21.15.145:3300/0 conn(0x7f7d4815e840 0x7f7d48070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 -- 172.21.15.119:0/322209940 >> v2:172.21.15.145:3300/0 conn(0x7f7d4815e840 msgr2=0x7f7d48070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:06.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 --2- 172.21.15.119:0/322209940 >> v2:172.21.15.145:3300/0 conn(0x7f7d4815e840 0x7f7d48070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 -- 172.21.15.119:0/322209940 >> v1:172.21.15.145:6789/0 conn(0x7f7d48164a70 legacy=0x7f7d480ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 -- 172.21.15.119:0/322209940 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d48071ab0 con 0x7f7d4815ec20 2024-09-13T16:36:06.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4659c700 1 --2- 172.21.15.119:0/322209940 >> v2:172.21.15.145:3300/0 conn(0x7f7d4815e840 0x7f7d48070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:06.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d45d9b700 1 --2- 172.21.15.119:0/322209940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4815ec20 0x7f7d48161f00 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f7d4806dec0 tx=0x7f7d3800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61e1941339fc3323 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4559a700 1 -- 172.21.15.119:0/322209940 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3800e070 con 0x7f7d4815ec20 2024-09-13T16:36:06.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4559a700 1 -- 172.21.15.119:0/322209940 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d3800b9c0 con 0x7f7d4815ec20 2024-09-13T16:36:06.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4559a700 1 -- 172.21.15.119:0/322209940 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3800bd70 con 0x7f7d4815ec20 2024-09-13T16:36:06.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/322209940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4815ec20 msgr2=0x7f7d48161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/322209940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4815ec20 0x7f7d48161f00 secure :-1 s=READY pgs=1831 cs=0 l=1 rev1=1 crypto rx=0x7f7d4806dec0 tx=0x7f7d3800b330 comp rx=0 tx=0).stop 2024-09-13T16:36:06.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/322209940 shutdown_connections 2024-09-13T16:36:06.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/322209940 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d4815ec20 0x7f7d48161f00 unknown :-1 s=CLOSED pgs=1831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/322209940 >> v2:172.21.15.145:3300/0 conn(0x7f7d4815e840 0x7f7d48070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/322209940 >> 172.21.15.119:0/322209940 conn(0x7f7d4806a350 msgr2=0x7f7d4806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:06.714 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/322209940 shutdown_connections 2024-09-13T16:36:06.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.710+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/322209940 wait complete. 2024-09-13T16:36:06.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.714+0000 7f7d4ccc0700 1 Processor -- start 2024-09-13T16:36:06.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.714+0000 7f7d4ccc0700 1 -- start start 2024-09-13T16:36:06.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.714+0000 7f7d4ccc0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4815e840 0x7f7d4814f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4ccc0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d48164a70 0x7f7d4815f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4ccc0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4815f550 0x7f7d48160a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4ccc0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815f9f0 con 0x7f7d48164a70 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4ccc0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815fcd0 con 0x7f7d4815f550 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4ccc0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d4815ffb0 con 0x7f7d4815e840 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d48164a70 0x7f7d4815f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4815e840 0x7f7d4814f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4815e840 0x7f7d4814f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:33164/0 (socket says 172.21.15.119:33164) 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4659c700 1 -- 172.21.15.119:0/3710195535 learned_addr learned my addr 172.21.15.119:0/3710195535 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d48164a70 0x7f7d4815f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49914/0 (socket says 172.21.15.119:49914) 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d46d9d700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4815f550 0x7f7d48160a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 -- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4815e840 msgr2=0x7f7d4814f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4815e840 0x7f7d4814f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 -- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4815f550 msgr2=0x7f7d48160a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4815f550 0x7f7d48160a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 -- 172.21.15.119:0/3710195535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d38009b30 con 0x7f7d48164a70 2024-09-13T16:36:06.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d4659c700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4815e840 0x7f7d4814f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:06.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.718+0000 7f7d45d9b700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d48164a70 0x7f7d4815f010 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f7d38015000 tx=0x7f7d38004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d3effd700 1 -- 172.21.15.119:0/3710195535 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d380048c0 con 0x7f7d48164a70 2024-09-13T16:36:06.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d48160290 con 0x7f7d48164a70 2024-09-13T16:36:06.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d3effd700 1 -- 172.21.15.119:0/3710195535 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d380099b0 con 0x7f7d48164a70 2024-09-13T16:36:06.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d3effd700 1 -- 172.21.15.119:0/3710195535 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d3801c680 con 0x7f7d48164a70 2024-09-13T16:36:06.724 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4814fe50 con 0x7f7d48164a70 2024-09-13T16:36:06.725 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d48067760 con 0x7f7d48164a70 2024-09-13T16:36:06.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d3effd700 1 -- 172.21.15.119:0/3710195535 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7d3801c7e0 con 0x7f7d48164a70 2024-09-13T16:36:06.726 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.722+0000 7f7d3effd700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d28066730 0x7f7d28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:06.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.726+0000 7f7d3effd700 1 -- 172.21.15.119:0/3710195535 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7d38085b60 con 0x7f7d48164a70 2024-09-13T16:36:06.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.726+0000 7f7d4659c700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d28066730 0x7f7d28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:06.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.726+0000 7f7d4659c700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d28066730 0x7f7d28068bf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009ec0 tx=0x7f7d30008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:06.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.730+0000 7f7d3effd700 1 -- 172.21.15.119:0/3710195535 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7d38052d80 con 0x7f7d48164a70 2024-09-13T16:36:06.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f7d4814ffe0 con 0x7f7d48164a70 2024-09-13T16:36:06.911 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d3effd700 1 -- 172.21.15.119:0/3710195535 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v126) v1 ==== 80+0+0 (secure 0 0 0) 0x7f7d380563a0 con 0x7f7d48164a70 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d28066730 msgr2=0x7f7d28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d28066730 0x7f7d28068bf0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f7d30009ec0 tx=0x7f7d30008040 comp rx=0 tx=0).stop 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d48164a70 msgr2=0x7f7d4815f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d48164a70 0x7f7d4815f010 secure :-1 s=READY pgs=1832 cs=0 l=1 rev1=1 crypto rx=0x7f7d38015000 tx=0x7f7d38004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 shutdown_connections 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d28066730 0x7f7d28068bf0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7d4815f550 0x7f7d48160a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7d48164a70 0x7f7d4815f010 unknown :-1 s=CLOSED pgs=1832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 --2- 172.21.15.119:0/3710195535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7d4815e840 0x7f7d4814f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:06.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 >> 172.21.15.119:0/3710195535 conn(0x7f7d4806a350 msgr2=0x7f7d480b1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:06.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 shutdown_connections 2024-09-13T16:36:06.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:06.910+0000 7f7d4ccc0700 1 -- 172.21.15.119:0/3710195535 wait complete. 2024-09-13T16:36:06.915 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:36:06.926 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:06.926 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:06.934 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:07.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.038+0000 7fbe9c9c2700 1 Processor -- start 2024-09-13T16:36:07.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- start start 2024-09-13T16:36:07.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbe980ce040 0x7fbe98165450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9806cf60 con 0x7fbe980cc060 2024-09-13T16:36:07.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe980cc4d0 con 0x7fbe980adb30 2024-09-13T16:36:07.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe980b3390 con 0x7fbe980ce040 2024-09-13T16:36:07.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe96ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbe980cc060 0x7fbe980cd930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41934/0 (socket says 172.21.15.119:41934) 2024-09-13T16:36:07.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe96ffd700 1 -- 172.21.15.119:0/2128476506 learned_addr learned my addr 172.21.15.119:0/2128476506 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:07.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe97fff700 1 --2- 172.21.15.119:0/2128476506 >> v2:172.21.15.119:3300/0 conn(0x7fbe980ce040 0x7fbe98165450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe967fc700 1 -- 172.21.15.119:0/2128476506 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289988992 0 0) 0x7fbe9806cf60 con 0x7fbe980cc060 2024-09-13T16:36:07.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe967fc700 1 -- 172.21.15.119:0/2128476506 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe840035d0 con 0x7fbe980cc060 2024-09-13T16:36:07.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe97fff700 1 -- 172.21.15.119:0/2128476506 >> v1:172.21.15.145:6789/0 conn(0x7fbe980adb30 legacy=0x7fbe9806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe97fff700 1 -- 172.21.15.119:0/2128476506 >> v1:172.21.15.119:6789/0 conn(0x7fbe980cc060 legacy=0x7fbe980cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe97fff700 1 -- 172.21.15.119:0/2128476506 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9806c350 con 0x7fbe980ce040 2024-09-13T16:36:07.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe97fff700 1 --2- 172.21.15.119:0/2128476506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980ce040 0x7fbe98165450 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009d90 tx=0x7fbe9000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc32179488aa4eb server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe967fc700 1 -- 172.21.15.119:0/2128476506 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9000e070 con 0x7fbe980ce040 2024-09-13T16:36:07.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe967fc700 1 -- 172.21.15.119:0/2128476506 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9000bb20 con 0x7fbe980ce040 2024-09-13T16:36:07.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe967fc700 1 -- 172.21.15.119:0/2128476506 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe90012430 con 0x7fbe980ce040 2024-09-13T16:36:07.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/2128476506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980ce040 msgr2=0x7fbe98165450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/2128476506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980ce040 0x7fbe98165450 secure :-1 s=READY pgs=1834 cs=0 l=1 rev1=1 crypto rx=0x7fbe90009d90 tx=0x7fbe9000b330 comp rx=0 tx=0).stop 2024-09-13T16:36:07.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/2128476506 shutdown_connections 2024-09-13T16:36:07.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/2128476506 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980ce040 0x7fbe98165450 unknown :-1 s=CLOSED pgs=1834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/2128476506 >> 172.21.15.119:0/2128476506 conn(0x7fbe9806a350 msgr2=0x7fbe9806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:07.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/2128476506 shutdown_connections 2024-09-13T16:36:07.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.042+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/2128476506 wait complete. 2024-09-13T16:36:07.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.046+0000 7fbe9c9c2700 1 Processor -- start 2024-09-13T16:36:07.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.046+0000 7fbe9c9c2700 1 -- start start 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980adb30 0x7fbe981efa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe980cc060 0x7fbe9815d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe980ce040 0x7fbe980c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe981419b0 con 0x7fbe980adb30 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980abaf0 con 0x7fbe980cc060 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980adb30 0x7fbe981efa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe980ab1c0 con 0x7fbe980ce040 2024-09-13T16:36:07.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980adb30 0x7fbe981efa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49926/0 (socket says 172.21.15.119:49926) 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 -- 172.21.15.119:0/3268929820 learned_addr learned my addr 172.21.15.119:0/3268929820 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe96ffd700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe980cc060 0x7fbe9815d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe97fff700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe980ce040 0x7fbe980c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 -- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe980ce040 msgr2=0x7fbe980c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe980ce040 0x7fbe980c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 -- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe980cc060 msgr2=0x7fbe9815d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe980cc060 0x7fbe9815d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 -- 172.21.15.119:0/3268929820 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe90009a70 con 0x7fbe980adb30 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe96ffd700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe980cc060 0x7fbe9815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe97fff700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe980ce040 0x7fbe980c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe977fe700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980adb30 0x7fbe981efa60 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800c500 tx=0x7fbe8800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe8ffff700 1 -- 172.21.15.119:0/3268929820 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe880052d0 con 0x7fbe980adb30 2024-09-13T16:36:07.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe8ffff700 1 -- 172.21.15.119:0/3268929820 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe88005430 con 0x7fbe980adb30 2024-09-13T16:36:07.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe8ffff700 1 -- 172.21.15.119:0/3268929820 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe880055a0 con 0x7fbe980adb30 2024-09-13T16:36:07.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe980ab4a0 con 0x7fbe980adb30 2024-09-13T16:36:07.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe980c4800 con 0x7fbe980adb30 2024-09-13T16:36:07.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.050+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe980adf10 con 0x7fbe980adb30 2024-09-13T16:36:07.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.054+0000 7fbe8ffff700 1 -- 172.21.15.119:0/3268929820 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbe880038e0 con 0x7fbe980adb30 2024-09-13T16:36:07.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.054+0000 7fbe8ffff700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe78066730 0x7fbe78068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.054+0000 7fbe8ffff700 1 -- 172.21.15.119:0/3268929820 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbe88083710 con 0x7fbe980adb30 2024-09-13T16:36:07.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.054+0000 7fbe96ffd700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe78066730 0x7fbe78068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.054+0000 7fbe96ffd700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe78066730 0x7fbe78068bf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbe980ad810 tx=0x7fbe80009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.058+0000 7fbe8ffff700 1 -- 172.21.15.119:0/3268929820 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbe880509d0 con 0x7fbe980adb30 2024-09-13T16:36:07.238 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.234+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe98142190 con 0x7fbe980adb30 2024-09-13T16:36:07.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.238+0000 7fbe8ffff700 1 -- 172.21.15.119:0/3268929820 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16844 (secure 0 0 0) 0x7fbe88053ff0 con 0x7fbe980adb30 2024-09-13T16:36:07.241 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:07.241 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":126,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:56.248923+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe78066730 msgr2=0x7fbe78068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe78066730 0x7fbe78068bf0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fbe980ad810 tx=0x7fbe80009040 comp rx=0 tx=0).stop 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980adb30 msgr2=0x7fbe981efa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980adb30 0x7fbe981efa60 secure :-1 s=READY pgs=1835 cs=0 l=1 rev1=1 crypto rx=0x7fbe8800c500 tx=0x7fbe8800c8c0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 shutdown_connections 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe78066730 0x7fbe78068bf0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe980ce040 0x7fbe980c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe980cc060 0x7fbe9815d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 --2- 172.21.15.119:0/3268929820 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe980adb30 0x7fbe981efa60 unknown :-1 s=CLOSED pgs=1835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 >> 172.21.15.119:0/3268929820 conn(0x7fbe9806a350 msgr2=0x7fbe980af400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 shutdown_connections 2024-09-13T16:36:07.244 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.242+0000 7fbe9c9c2700 1 -- 172.21.15.119:0/3268929820 wait complete. 2024-09-13T16:36:07.256 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:07.256 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:07.368 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0ce5c700 1 Processor -- start 2024-09-13T16:36:07.369 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0ce5c700 1 -- start start 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0ce5c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab080b3450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0ce5c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fab08070d40 0x7fab0815b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0ce5c700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab080ae340 con 0x7fab08159ca0 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0ce5c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0815ae40 con 0x7fab08070d40 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0ce5c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab0815b1d0 con 0x7fab080adf60 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab06d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fab08070d40 0x7fab0815b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab06d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fab08070d40 0x7fab0815b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49938/0 (socket says 172.21.15.119:49938) 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab06d9d700 1 -- 172.21.15.119:0/2985835085 learned_addr learned my addr 172.21.15.119:0/2985835085 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:07.371 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.366+0000 7fab0659c700 1 --2- 172.21.15.119:0/2985835085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab080b3450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab06d9d700 1 -- 172.21.15.119:0/2985835085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 msgr2=0x7fab080b3450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:07.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab06d9d700 1 --2- 172.21.15.119:0/2985835085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab080b3450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab06d9d700 1 -- 172.21.15.119:0/2985835085 >> v1:172.21.15.145:6789/0 conn(0x7fab08159ca0 legacy=0x7fab08070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab06d9d700 1 -- 172.21.15.119:0/2985835085 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0815a5a0 con 0x7fab08070d40 2024-09-13T16:36:07.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab06d9d700 1 --2- 172.21.15.119:0/2985835085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08070d40 0x7fab0815b660 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fab00009e50 tx=0x7fab0000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8977e63017a8fdd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0559a700 1 -- 172.21.15.119:0/2985835085 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0000e070 con 0x7fab08070d40 2024-09-13T16:36:07.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0559a700 1 -- 172.21.15.119:0/2985835085 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0000b910 con 0x7fab08070d40 2024-09-13T16:36:07.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0559a700 1 -- 172.21.15.119:0/2985835085 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0000bcc0 con 0x7fab08070d40 2024-09-13T16:36:07.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2985835085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08070d40 msgr2=0x7fab0815b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2985835085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08070d40 0x7fab0815b660 secure :-1 s=READY pgs=1836 cs=0 l=1 rev1=1 crypto rx=0x7fab00009e50 tx=0x7fab0000b330 comp rx=0 tx=0).stop 2024-09-13T16:36:07.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2985835085 shutdown_connections 2024-09-13T16:36:07.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2985835085 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08070d40 0x7fab0815b660 unknown :-1 s=CLOSED pgs=1836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2985835085 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab080b3450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2985835085 >> 172.21.15.119:0/2985835085 conn(0x7fab080b0560 msgr2=0x7fab080b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:07.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2985835085 shutdown_connections 2024-09-13T16:36:07.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2985835085 wait complete. 2024-09-13T16:36:07.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 Processor -- start 2024-09-13T16:36:07.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.370+0000 7fab0ce5c700 1 -- start start 2024-09-13T16:36:07.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab08070d40 0x7fab08143cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab08147d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.376 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08159ca0 0x7fab081459d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab080cd570 con 0x7fab08159ca0 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab08152f30 con 0x7fab08070d40 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab080ccbd0 con 0x7fab080adf60 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08159ca0 0x7fab081459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08159ca0 0x7fab081459d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49944/0 (socket says 172.21.15.119:49944) 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 -- 172.21.15.119:0/2482154931 learned_addr learned my addr 172.21.15.119:0/2482154931 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab05d9b700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab08147d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0659c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab08070d40 0x7fab08143cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 -- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 msgr2=0x7fab08147d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab08147d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 -- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab08070d40 msgr2=0x7fab08143cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab08070d40 0x7fab08143cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 -- 172.21.15.119:0/2482154931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab00009b30 con 0x7fab08159ca0 2024-09-13T16:36:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab06d9d700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08159ca0 0x7fab081459d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fab00012000 tx=0x7fab00004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7faaf6ffd700 1 -- 172.21.15.119:0/2482154931 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab000048c0 con 0x7fab08159ca0 2024-09-13T16:36:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7faaf6ffd700 1 -- 172.21.15.119:0/2482154931 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab000092e0 con 0x7fab08159ca0 2024-09-13T16:36:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7faaf6ffd700 1 -- 172.21.15.119:0/2482154931 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab0001c660 con 0x7fab08159ca0 2024-09-13T16:36:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab080cceb0 con 0x7fab08159ca0 2024-09-13T16:36:07.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab08146790 con 0x7fab08159ca0 2024-09-13T16:36:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab080cd6b0 con 0x7fab08159ca0 2024-09-13T16:36:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.374+0000 7faaf6ffd700 1 -- 172.21.15.119:0/2482154931 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fab0001c7c0 con 0x7fab08159ca0 2024-09-13T16:36:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.378+0000 7faaf6ffd700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faae8066730 0x7faae8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.378+0000 7faaf6ffd700 1 -- 172.21.15.119:0/2482154931 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fab00085de0 con 0x7fab08159ca0 2024-09-13T16:36:07.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.378+0000 7fab0659c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faae8066730 0x7faae8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.378+0000 7fab0659c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faae8066730 0x7faae8068bf0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7faafc009bf0 tx=0x7faafc009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.382+0000 7faaf6ffd700 1 -- 172.21.15.119:0/2482154931 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fab080cd6b0 con 0x7fab08159ca0 2024-09-13T16:36:07.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.542+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fab080cd6b0 con 0x7fab08159ca0 2024-09-13T16:36:07.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7faaf6ffd700 1 -- 172.21.15.119:0/2482154931 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7fab080cd6b0 con 0x7fab08159ca0 2024-09-13T16:36:07.547 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:07.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faae8066730 msgr2=0x7faae8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faae8066730 0x7faae8068bf0 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7faafc009bf0 tx=0x7faafc009240 comp rx=0 tx=0).stop 2024-09-13T16:36:07.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08159ca0 msgr2=0x7fab081459d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.549 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08159ca0 0x7fab081459d0 secure :-1 s=READY pgs=1837 cs=0 l=1 rev1=1 crypto rx=0x7fab00012000 tx=0x7fab00004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 shutdown_connections 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7faae8066730 0x7faae8068bf0 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab08159ca0 0x7fab081459d0 unknown :-1 s=CLOSED pgs=1837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab080adf60 0x7fab08147d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 --2- 172.21.15.119:0/2482154931 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab08070d40 0x7fab08143cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 >> 172.21.15.119:0/2482154931 conn(0x7fab080b0560 msgr2=0x7fab080ae7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 shutdown_connections 2024-09-13T16:36:07.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.546+0000 7fab0ce5c700 1 -- 172.21.15.119:0/2482154931 wait complete. 2024-09-13T16:36:07.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:07.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.678+0000 7f8bc676b700 1 Processor -- start 2024-09-13T16:36:07.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.678+0000 7f8bc676b700 1 -- start start 2024-09-13T16:36:07.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.678+0000 7f8bc676b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8bc00d17e0 0x7f8bc00b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc00cdb00 con 0x7f8bc00d0c20 2024-09-13T16:36:07.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc00cdde0 con 0x7f8bc00d17e0 2024-09-13T16:36:07.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc00ce0c0 con 0x7f8bc015cc00 2024-09-13T16:36:07.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5f6a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f8bc00d17e0 0x7f8bc00b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:33210/0 (socket says 172.21.15.119:33210) 2024-09-13T16:36:07.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 -- 172.21.15.119:0/1114945731 learned_addr learned my addr 172.21.15.119:0/1114945731 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:07.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 -- 172.21.15.119:0/1114945731 >> v2:172.21.15.145:3300/0 conn(0x7f8bc00d17e0 msgr2=0x7f8bc00b5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:07.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 --2- 172.21.15.119:0/1114945731 >> v2:172.21.15.145:3300/0 conn(0x7f8bc00d17e0 0x7f8bc00b5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bb7fff700 1 -- 172.21.15.119:0/1114945731 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951947215 0 0) 0x7f8bc00cdb00 con 0x7f8bc00d0c20 2024-09-13T16:36:07.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 -- 172.21.15.119:0/1114945731 >> v1:172.21.15.145:6789/0 conn(0x7f8bc00d0c20 legacy=0x7f8bc00d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 -- 172.21.15.119:0/1114945731 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc00ce3a0 con 0x7f8bc015cc00 2024-09-13T16:36:07.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc5769700 1 --2- 172.21.15.119:0/1114945731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00d06e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009910 tx=0x7f8bb0009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6602170c8e67bffe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bb7fff700 1 -- 172.21.15.119:0/1114945731 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0011830 con 0x7f8bc015cc00 2024-09-13T16:36:07.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bb7fff700 1 -- 172.21.15.119:0/1114945731 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb0011990 con 0x7f8bc015cc00 2024-09-13T16:36:07.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bb7fff700 1 -- 172.21.15.119:0/1114945731 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb0011d40 con 0x7f8bc015cc00 2024-09-13T16:36:07.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- 172.21.15.119:0/1114945731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 msgr2=0x7f8bc00d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 --2- 172.21.15.119:0/1114945731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00d06e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0009910 tx=0x7f8bb0009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:07.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- 172.21.15.119:0/1114945731 shutdown_connections 2024-09-13T16:36:07.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 --2- 172.21.15.119:0/1114945731 >> v2:172.21.15.145:3300/0 conn(0x7f8bc00d17e0 0x7f8bc00b5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 --2- 172.21.15.119:0/1114945731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00d06e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- 172.21.15.119:0/1114945731 >> 172.21.15.119:0/1114945731 conn(0x7f8bc008c080 msgr2=0x7f8bc00d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:07.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- 172.21.15.119:0/1114945731 shutdown_connections 2024-09-13T16:36:07.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.682+0000 7f8bc676b700 1 -- 172.21.15.119:0/1114945731 wait complete. 2024-09-13T16:36:07.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 Processor -- start 2024-09-13T16:36:07.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 -- start start 2024-09-13T16:36:07.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bc00d0c20 0x7f8bc0155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8bc00d17e0 0x7f8bc00cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00ccb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc0156b60 con 0x7f8bc00d0c20 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc0156dd0 con 0x7f8bc00d17e0 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc01570b0 con 0x7f8bc015cc00 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bc00d0c20 0x7f8bc0155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5f6a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00ccb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bc00d0c20 0x7f8bc0155a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49948/0 (socket says 172.21.15.119:49948) 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc4f68700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8bc00d17e0 0x7f8bc00cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 -- 172.21.15.119:0/3846188750 learned_addr learned my addr 172.21.15.119:0/3846188750 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5f6a700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00ccb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:33214/0 (socket says 172.21.15.119:33214) 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 -- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 msgr2=0x7f8bc00ccb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00ccb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 -- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8bc00d17e0 msgr2=0x7f8bc00cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8bc00d17e0 0x7f8bc00cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 -- 172.21.15.119:0/3846188750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb0009570 con 0x7f8bc00d0c20 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc5769700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bc00d0c20 0x7f8bc0155a70 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0000c00 tx=0x7f8bb0004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bb5ffb700 1 -- 172.21.15.119:0/3846188750 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb001a8f0 con 0x7f8bc00d0c20 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bb5ffb700 1 -- 172.21.15.119:0/3846188750 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb001e070 con 0x7f8bc00d0c20 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bb5ffb700 1 -- 172.21.15.119:0/3846188750 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bb001acd0 con 0x7f8bc00d0c20 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc0157390 con 0x7f8bc00d0c20 2024-09-13T16:36:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.686+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc0155fb0 con 0x7f8bc00d0c20 2024-09-13T16:36:07.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.690+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc00ce8c0 con 0x7f8bc00d0c20 2024-09-13T16:36:07.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.690+0000 7f8bb5ffb700 1 -- 172.21.15.119:0/3846188750 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8bb0024400 con 0x7f8bc00d0c20 2024-09-13T16:36:07.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.690+0000 7f8bb5ffb700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ba4066730 0x7f8ba4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:07.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.690+0000 7f8bb5ffb700 1 -- 172.21.15.119:0/3846188750 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f8bb00852e0 con 0x7f8bc00d0c20 2024-09-13T16:36:07.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.690+0000 7f8bc4f68700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ba4066730 0x7f8ba4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:07.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.690+0000 7f8bc4f68700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ba4066730 0x7f8ba4068bf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8bc00b7770 tx=0x7f8bbc009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:07.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.698+0000 7f8bb5ffb700 1 -- 172.21.15.119:0/3846188750 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f8bb0052500 con 0x7f8bc00d0c20 2024-09-13T16:36:07.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.866+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8bc006c400 con 0x7f8bc00d0c20 2024-09-13T16:36:07.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.866+0000 7f8bb5ffb700 1 -- 172.21.15.119:0/3846188750 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8bb0055b20 con 0x7f8bc00d0c20 2024-09-13T16:36:07.869 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ba4066730 msgr2=0x7f8ba4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ba4066730 0x7f8ba4068bf0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f8bc00b7770 tx=0x7f8bbc009040 comp rx=0 tx=0).stop 2024-09-13T16:36:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bc00d0c20 msgr2=0x7f8bc0155a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:07.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bc00d0c20 0x7f8bc0155a70 secure :-1 s=READY pgs=1838 cs=0 l=1 rev1=1 crypto rx=0x7f8bb0000c00 tx=0x7f8bb0004770 comp rx=0 tx=0).stop 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 shutdown_connections 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8ba4066730 0x7f8ba4068bf0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8bc015cc00 0x7f8bc00ccb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8bc00d17e0 0x7f8bc00cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 --2- 172.21.15.119:0/3846188750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8bc00d0c20 0x7f8bc0155a70 unknown :-1 s=CLOSED pgs=1838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 >> 172.21.15.119:0/3846188750 conn(0x7f8bc008c080 msgr2=0x7f8bc00b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 shutdown_connections 2024-09-13T16:36:07.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:07.870+0000 7f8bc676b700 1 -- 172.21.15.119:0/3846188750 wait complete. 2024-09-13T16:36:07.888 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e4a573700 1 Processor -- start 2024-09-13T16:36:08.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e4a573700 1 -- start start 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e4a573700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4e4415dbe0 0x7f4e440ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e4a573700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4e440b3260 0x7f4e440b3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e4a573700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e440cd4d0 con 0x7f4e440add60 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e4a573700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e44071be0 con 0x7f4e440b3260 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e4a573700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e44159af0 con 0x7f4e4415dbe0 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4e440b3260 0x7f4e440b3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4e440b3260 0x7f4e440b3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49956/0 (socket says 172.21.15.119:49956) 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 -- 172.21.15.119:0/1722491913 learned_addr learned my addr 172.21.15.119:0/1722491913 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e48d70700 1 --2- 172.21.15.119:0/1722491913 >> v2:172.21.15.145:3300/0 conn(0x7f4e4415dbe0 0x7f4e440ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 -- 172.21.15.119:0/1722491913 >> v2:172.21.15.145:3300/0 conn(0x7f4e4415dbe0 msgr2=0x7f4e440ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 --2- 172.21.15.119:0/1722491913 >> v2:172.21.15.145:3300/0 conn(0x7f4e4415dbe0 0x7f4e440ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 -- 172.21.15.119:0/1722491913 >> v1:172.21.15.145:6789/0 conn(0x7f4e440add60 legacy=0x7f4e440cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 -- 172.21.15.119:0/1722491913 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e4415e130 con 0x7f4e440b3260 2024-09-13T16:36:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.002+0000 7f4e49d72700 1 --2- 172.21.15.119:0/1722491913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440b3670 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f4e440b2ab0 tx=0x7f4e3c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=682392354e1b1562 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e3bfff700 1 -- 172.21.15.119:0/1722491913 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c00e070 con 0x7f4e440b3260 2024-09-13T16:36:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e3bfff700 1 -- 172.21.15.119:0/1722491913 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3c003a40 con 0x7f4e440b3260 2024-09-13T16:36:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e3bfff700 1 -- 172.21.15.119:0/1722491913 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3c003df0 con 0x7f4e440b3260 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 -- 172.21.15.119:0/1722491913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 msgr2=0x7f4e440b3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 --2- 172.21.15.119:0/1722491913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440b3670 secure :-1 s=READY pgs=1839 cs=0 l=1 rev1=1 crypto rx=0x7f4e440b2ab0 tx=0x7f4e3c00b330 comp rx=0 tx=0).stop 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 -- 172.21.15.119:0/1722491913 shutdown_connections 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 --2- 172.21.15.119:0/1722491913 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440b3670 unknown :-1 s=CLOSED pgs=1839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 --2- 172.21.15.119:0/1722491913 >> v2:172.21.15.145:3300/0 conn(0x7f4e4415dbe0 0x7f4e440ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 -- 172.21.15.119:0/1722491913 >> 172.21.15.119:0/1722491913 conn(0x7f4e440b0980 msgr2=0x7f4e440b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 -- 172.21.15.119:0/1722491913 shutdown_connections 2024-09-13T16:36:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 -- 172.21.15.119:0/1722491913 wait complete. 2024-09-13T16:36:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 Processor -- start 2024-09-13T16:36:08.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.006+0000 7f4e4a573700 1 -- start start 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e440add60 0x7f4e44155490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440c37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e4415dbe0 0x7f4e440c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440c40d0 con 0x7f4e440b3260 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440c43b0 con 0x7f4e440add60 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440c4690 con 0x7f4e4415dbe0 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440c37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440c37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49958/0 (socket says 172.21.15.119:49958) 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 -- 172.21.15.119:0/3508748151 learned_addr learned my addr 172.21.15.119:0/3508748151 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e49571700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e440add60 0x7f4e44155490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e49d72700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e4415dbe0 0x7f4e440c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 -- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e4415dbe0 msgr2=0x7f4e440c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e4415dbe0 0x7f4e440c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 -- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e440add60 msgr2=0x7f4e44155490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e440add60 0x7f4e44155490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 -- 172.21.15.119:0/3508748151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3c009b30 con 0x7f4e440b3260 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e48d70700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440c37d0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400d800 tx=0x7f4e3400db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e39ffb700 1 -- 172.21.15.119:0/3508748151 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e340125f0 con 0x7f4e440b3260 2024-09-13T16:36:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e39ffb700 1 -- 172.21.15.119:0/3508748151 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e34012750 con 0x7f4e440b3260 2024-09-13T16:36:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e440c4970 con 0x7f4e440b3260 2024-09-13T16:36:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e39ffb700 1 -- 172.21.15.119:0/3508748151 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e34012ac0 con 0x7f4e440b3260 2024-09-13T16:36:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e440c3d10 con 0x7f4e440b3260 2024-09-13T16:36:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.010+0000 7f4e39ffb700 1 -- 172.21.15.119:0/3508748151 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4e34012c20 con 0x7f4e440b3260 2024-09-13T16:36:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.014+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e440ae140 con 0x7f4e440b3260 2024-09-13T16:36:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.014+0000 7f4e39ffb700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e28066730 0x7f4e28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.014+0000 7f4e39ffb700 1 -- 172.21.15.119:0/3508748151 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4e340842b0 con 0x7f4e440b3260 2024-09-13T16:36:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.014+0000 7f4e49571700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e28066730 0x7f4e28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.014+0000 7f4e49571700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e28066730 0x7f4e28068bf0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4e40005150 tx=0x7f4e40007ee0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:08.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.022+0000 7f4e39ffb700 1 -- 172.21.15.119:0/3508748151 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4e340515a0 con 0x7f4e440b3260 2024-09-13T16:36:08.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.190+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4e44072c50 con 0x7f4e440b3260 2024-09-13T16:36:08.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e39ffb700 1 -- 172.21.15.119:0/3508748151 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4e34054bc0 con 0x7f4e440b3260 2024-09-13T16:36:08.195 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e28066730 msgr2=0x7f4e28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e28066730 0x7f4e28068bf0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f4e40005150 tx=0x7f4e40007ee0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 msgr2=0x7f4e440c37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440c37d0 secure :-1 s=READY pgs=1840 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400d800 tx=0x7f4e3400db10 comp rx=0 tx=0).stop 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 shutdown_connections 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4e28066730 0x7f4e28068bf0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4e4415dbe0 0x7f4e440c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4e440b3260 0x7f4e440c37d0 unknown :-1 s=CLOSED pgs=1840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 --2- 172.21.15.119:0/3508748151 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4e440add60 0x7f4e44155490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 >> 172.21.15.119:0/3508748151 conn(0x7f4e440b0980 msgr2=0x7f4e440b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:08.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 shutdown_connections 2024-09-13T16:36:08.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.194+0000 7f4e4a573700 1 -- 172.21.15.119:0/3508748151 wait complete. 2024-09-13T16:36:08.210 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:36:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771995700 1 Processor -- start 2024-09-13T16:36:08.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771995700 1 -- start start 2024-09-13T16:36:08.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771995700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 0x7ff76c0ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771995700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff76c08c6b0 0x7ff76c166a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771995700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff76c15d020 con 0x7ff76c0cd860 2024-09-13T16:36:08.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771995700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff76c0b44c0 con 0x7ff76c08c6b0 2024-09-13T16:36:08.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771995700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0d0720 con 0x7ff76c15cc40 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff763fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff76c0cd860 0x7ff76c08bfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41942/0 (socket says 172.21.15.119:41942) 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff763fff700 1 -- 172.21.15.119:0/446787488 learned_addr learned my addr 172.21.15.119:0/446787488 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff771194700 1 --2- 172.21.15.119:0/446787488 >> v2:172.21.15.145:3300/0 conn(0x7ff76c08c6b0 0x7ff76c166a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff770993700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 0x7ff76c0ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.322+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 927104510 0 0) 0x7ff76c15d020 con 0x7ff76c0cd860 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7540035d0 con 0x7ff76c0cd860 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3463004845 0 0) 0x7ff7540035d0 con 0x7ff76c0cd860 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 msgr2=0x7ff76c0ce710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 0x7ff76c0ce710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 >> v2:172.21.15.145:3300/0 conn(0x7ff76c08c6b0 msgr2=0x7ff76c166a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 --2- 172.21.15.119:0/446787488 >> v2:172.21.15.145:3300/0 conn(0x7ff76c08c6b0 0x7ff76c166a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0d09c0 con 0x7ff76c0cd860 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7ff75c001670 con 0x7ff76c0cd860 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 >> v1:172.21.15.119:6789/0 conn(0x7ff76c0cd860 legacy=0x7ff76c08bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff754003a20 0x7ff754003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 0x7ff754004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff754004db0 0x7ff754005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0b44c0 con 0x7ff754003a20 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0d0720 con 0x7ff754004db0 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff75400ba30 con 0x7ff76c15cc40 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771194700 1 --2- 172.21.15.119:0/446787488 >> v2:172.21.15.145:3300/0 conn(0x7ff76c08c6b0 0x7ff76c166a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff770993700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff754003a20 0x7ff754003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff763fff700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 0x7ff754004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771194700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff754004db0 0x7ff754005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff770993700 1 -- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 msgr2=0x7ff754004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff770993700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c15cc40 0x7ff754004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff770993700 1 -- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff754004db0 msgr2=0x7ff754005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff770993700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff754004db0 0x7ff754005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff770993700 1 -- 172.21.15.119:0/446787488 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7540035d0 con 0x7ff754003a20 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771194700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff754004db0 0x7ff754005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff770993700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff754003a20 0x7ff754003e40 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff768009b70 tx=0x7ff7540058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef503ba43bb9f104 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76800e6c0 con 0x7ff754003a20 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff76800e820 con 0x7ff754003a20 2024-09-13T16:36:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff7637fe700 1 -- 172.21.15.119:0/446787488 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76800eb90 con 0x7ff754003a20 2024-09-13T16:36:08.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771995700 1 -- 172.21.15.119:0/446787488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff754003a20 msgr2=0x7ff754003e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771995700 1 --2- 172.21.15.119:0/446787488 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff754003a20 0x7ff754003e40 secure :-1 s=READY pgs=1842 cs=0 l=1 rev1=1 crypto rx=0x7ff768009b70 tx=0x7ff7540058a0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771194700 1 -- 172.21.15.119:0/446787488 reap_dead start 2024-09-13T16:36:08.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771995700 1 -- 172.21.15.119:0/446787488 shutdown_connections 2024-09-13T16:36:08.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771995700 1 -- 172.21.15.119:0/446787488 >> 172.21.15.119:0/446787488 conn(0x7ff76c0b4640 msgr2=0x7ff76c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:08.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771995700 1 -- 172.21.15.119:0/446787488 shutdown_connections 2024-09-13T16:36:08.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.326+0000 7ff771995700 1 -- 172.21.15.119:0/446787488 wait complete. 2024-09-13T16:36:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 Processor -- start 2024-09-13T16:36:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 -- start start 2024-09-13T16:36:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff76c155d60 0x7ff76c0b5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff76c156560 0x7ff76c1e88e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c1e8e20 0x7ff76c1f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c1e92a0 con 0x7ff76c155d60 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0b89a0 con 0x7ff76c156560 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff76c0b8100 con 0x7ff76c1e8e20 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff76c155d60 0x7ff76c0b5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff76c155d60 0x7ff76c0b5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:49982/0 (socket says 172.21.15.119:49982) 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 -- 172.21.15.119:0/2067408383 learned_addr learned my addr 172.21.15.119:0/2067408383 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771194700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c1e8e20 0x7ff76c1f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff763fff700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff76c156560 0x7ff76c1e88e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 -- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c1e8e20 msgr2=0x7ff76c1f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c1e8e20 0x7ff76c1f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 -- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff76c156560 msgr2=0x7ff76c1e88e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff76c156560 0x7ff76c1e88e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 -- 172.21.15.119:0/2067408383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff76c0b83e0 con 0x7ff76c155d60 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff770993700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff76c155d60 0x7ff76c0b5980 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff768016000 tx=0x7ff7680047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff7617fa700 1 -- 172.21.15.119:0/2067408383 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff76800e6c0 con 0x7ff76c155d60 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff7617fa700 1 -- 172.21.15.119:0/2067408383 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7680038e0 con 0x7ff76c155d60 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff76c0b86c0 con 0x7ff76c155d60 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff7617fa700 1 -- 172.21.15.119:0/2067408383 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff768003c90 con 0x7ff76c155d60 2024-09-13T16:36:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.330+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff76c0b5ec0 con 0x7ff76c155d60 2024-09-13T16:36:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.334+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff76c0b6050 con 0x7ff76c155d60 2024-09-13T16:36:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.334+0000 7ff7617fa700 1 -- 172.21.15.119:0/2067408383 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff768013470 con 0x7ff76c155d60 2024-09-13T16:36:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.334+0000 7ff7617fa700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff758066730 0x7ff758068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.334+0000 7ff7617fa700 1 -- 172.21.15.119:0/2067408383 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7ff7680856e0 con 0x7ff76c155d60 2024-09-13T16:36:08.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.334+0000 7ff763fff700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff758066730 0x7ff758068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:08.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.334+0000 7ff763fff700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff758066730 0x7ff758068bf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff76c06b230 tx=0x7ff75c0079a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:08.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.342+0000 7ff7617fa700 1 -- 172.21.15.119:0/2067408383 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff768052ac0 con 0x7ff76c155d60 2024-09-13T16:36:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.526+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ff76c1e9200 con 0x7ff76c155d60 2024-09-13T16:36:08.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.526+0000 7ff7617fa700 1 -- 172.21.15.119:0/2067408383 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v126) v1 ==== 80+0+0 (secure 0 0 0) 0x7ff768000e60 con 0x7ff76c155d60 2024-09-13T16:36:08.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff758066730 msgr2=0x7ff758068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff758066730 0x7ff758068bf0 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff76c06b230 tx=0x7ff75c0079a0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff76c155d60 msgr2=0x7ff76c0b5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:08.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff76c155d60 0x7ff76c0b5980 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7ff768016000 tx=0x7ff7680047e0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 shutdown_connections 2024-09-13T16:36:08.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff758066730 0x7ff758068bf0 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff76c1e8e20 0x7ff76c1f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff76c156560 0x7ff76c1e88e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 --2- 172.21.15.119:0/2067408383 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff76c155d60 0x7ff76c0b5980 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 >> 172.21.15.119:0/2067408383 conn(0x7ff76c0b4640 msgr2=0x7ff76c0b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 shutdown_connections 2024-09-13T16:36:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:08.530+0000 7ff771995700 1 -- 172.21.15.119:0/2067408383 wait complete. 2024-09-13T16:36:08.534 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:36:08.548 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:36:08.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:08.549 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:08.559 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[53667]: fuse finished with error 0 and tester_r 0 2024-09-13T16:36:08.560 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:36:14.662 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:36:14.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:14.662 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:36:14.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-09-13T16:36:14.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.786+0000 7fbb531e8700 1 Processor -- start 2024-09-13T16:36:14.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.786+0000 7fbb531e8700 1 -- start start 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c06ca60 0x7fbb4c159de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbb4c0b1860 0x7fbb4c15a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb4c0b1dd0 con 0x7fbb4c0ad9a0 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb4c0b2fd0 con 0x7fbb4c0b1860 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c15a520 con 0x7fbb4c06ca60 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbb4c0b1860 0x7fbb4c15a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbb4c0b1860 0x7fbb4c15a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60394/0 (socket says 172.21.15.119:60394) 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 -- 172.21.15.119:0/319038874 learned_addr learned my addr 172.21.15.119:0/319038874 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:14.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb519e5700 1 --2- 172.21.15.119:0/319038874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c06ca60 0x7fbb4c159de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 -- 172.21.15.119:0/319038874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c06ca60 msgr2=0x7fbb4c159de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 --2- 172.21.15.119:0/319038874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c06ca60 0x7fbb4c159de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 -- 172.21.15.119:0/319038874 >> v1:172.21.15.145:6789/0 conn(0x7fbb4c0ad9a0 legacy=0x7fbb4c06c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 -- 172.21.15.119:0/319038874 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c0b20b0 con 0x7fbb4c0b1860 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb529e7700 1 --2- 172.21.15.119:0/319038874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c0b1860 0x7fbb4c15a970 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbb44006fd0 tx=0x7fbb4400bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4afb6a4f5414bcfe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb511e4700 1 -- 172.21.15.119:0/319038874 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400dc20 con 0x7fbb4c0b1860 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb511e4700 1 -- 172.21.15.119:0/319038874 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4400dd80 con 0x7fbb4c0b1860 2024-09-13T16:36:14.794 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb511e4700 1 -- 172.21.15.119:0/319038874 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44004250 con 0x7fbb4c0b1860 2024-09-13T16:36:14.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- 172.21.15.119:0/319038874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c0b1860 msgr2=0x7fbb4c15a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 --2- 172.21.15.119:0/319038874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c0b1860 0x7fbb4c15a970 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7fbb44006fd0 tx=0x7fbb4400bea0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- 172.21.15.119:0/319038874 shutdown_connections 2024-09-13T16:36:14.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 --2- 172.21.15.119:0/319038874 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c0b1860 0x7fbb4c15a970 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 --2- 172.21.15.119:0/319038874 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c06ca60 0x7fbb4c159de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- 172.21.15.119:0/319038874 >> 172.21.15.119:0/319038874 conn(0x7fbb4c0b0980 msgr2=0x7fbb4c0b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:14.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- 172.21.15.119:0/319038874 shutdown_connections 2024-09-13T16:36:14.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.790+0000 7fbb531e8700 1 -- 172.21.15.119:0/319038874 wait complete. 2024-09-13T16:36:14.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.794+0000 7fbb531e8700 1 Processor -- start 2024-09-13T16:36:14.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.794+0000 7fbb531e8700 1 -- start start 2024-09-13T16:36:14.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c06ca60 0x7fbb4c14e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c0ad9a0 0x7fbb4c1e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb4c0b1860 0x7fbb4c14ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c1522b0 con 0x7fbb4c06ca60 2024-09-13T16:36:14.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c1f2980 con 0x7fbb4c0b1860 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb4c1f2c50 con 0x7fbb4c0ad9a0 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c06ca60 0x7fbb4c14e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c06ca60 0x7fbb4c14e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60396/0 (socket says 172.21.15.119:60396) 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 -- 172.21.15.119:0/2917660236 learned_addr learned my addr 172.21.15.119:0/2917660236 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb519e5700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c0ad9a0 0x7fbb4c1e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb529e7700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb4c0b1860 0x7fbb4c14ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 -- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c0ad9a0 msgr2=0x7fbb4c1e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c0ad9a0 0x7fbb4c1e5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 -- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb4c0b1860 msgr2=0x7fbb4c14ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb4c0b1860 0x7fbb4c14ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 -- 172.21.15.119:0/2917660236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4400b820 con 0x7fbb4c06ca60 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb521e6700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c06ca60 0x7fbb4c14e190 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c500 tx=0x7fbb3c00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:14.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb42ffd700 1 -- 172.21.15.119:0/2917660236 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c0052d0 con 0x7fbb4c06ca60 2024-09-13T16:36:14.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb4c15d760 con 0x7fbb4c06ca60 2024-09-13T16:36:14.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb42ffd700 1 -- 172.21.15.119:0/2917660236 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3c005430 con 0x7fbb4c06ca60 2024-09-13T16:36:14.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb42ffd700 1 -- 172.21.15.119:0/2917660236 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3c0055a0 con 0x7fbb4c06ca60 2024-09-13T16:36:14.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.798+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c151fd0 con 0x7fbb4c06ca60 2024-09-13T16:36:14.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.802+0000 7fbb42ffd700 1 -- 172.21.15.119:0/2917660236 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbb3c0038e0 con 0x7fbb4c06ca60 2024-09-13T16:36:14.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.802+0000 7fbb42ffd700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb30066760 0x7fbb30068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.802+0000 7fbb42ffd700 1 -- 172.21.15.119:0/2917660236 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbb3c0835a0 con 0x7fbb4c06ca60 2024-09-13T16:36:14.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.802+0000 7fbb519e5700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb30066760 0x7fbb30068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.802+0000 7fbb519e5700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb30066760 0x7fbb30068c20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c159050 tx=0x7fbb4800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:14.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 Processor -- start 2024-09-13T16:36:14.807 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 -- start start 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x10b0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x10d8520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10fa170 0x10fa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1041010 con 0x10fa170 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xfcef40 con 0x10f9d90 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1021e30 con 0x10af310 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10fa170 0x10fa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10fa170 0x10fa550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60398/0 (socket says 172.21.15.119:60398) 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 -- 172.21.15.119:0/1916245032 learned_addr learned my addr 172.21.15.119:0/1916245032 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb519e5700 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x10d8520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb521e6700 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x10b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 -- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 msgr2=0x10d8520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x10d8520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 -- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 msgr2=0x10b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x10b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 -- 172.21.15.119:0/1916245032 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xe8da40 con 0x10fa170 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb529e7700 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10fa170 0x10fa550 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c06dcb0 tx=0x7fbb44010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d94b8cd778ec6c2c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb2ffff700 1 -- 172.21.15.119:0/1916245032 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44006e20 con 0x10fa170 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb2ffff700 1 -- 172.21.15.119:0/1916245032 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4400da20 con 0x10fa170 2024-09-13T16:36:14.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb2ffff700 1 -- 172.21.15.119:0/1916245032 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400db90 con 0x10fa170 2024-09-13T16:36:14.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 -- 172.21.15.119:0/1916245032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10fa170 msgr2=0x10fa550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.806+0000 7fbb576fb740 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10fa170 0x10fa550 secure :-1 s=READY pgs=1846 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c06dcb0 tx=0x7fbb44010000 comp rx=0 tx=0).stop 2024-09-13T16:36:14.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- 172.21.15.119:0/1916245032 shutdown_connections 2024-09-13T16:36:14.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10fa170 0x10fa550 unknown :-1 s=CLOSED pgs=1846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x10d8520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 --2- 172.21.15.119:0/1916245032 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x10b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- 172.21.15.119:0/1916245032 >> 172.21.15.119:0/1916245032 conn(0xe425b0 msgr2=0x10b3b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:14.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- 172.21.15.119:0/1916245032 shutdown_connections 2024-09-13T16:36:14.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- 172.21.15.119:0/1916245032 wait complete. 2024-09-13T16:36:14.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- start start 2024-09-13T16:36:14.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 Processor -- start 2024-09-13T16:36:14.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x10d7560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10fa170 0x10fa550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x101d150 con 0x10f9d90 2024-09-13T16:36:14.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x101aeb0 con 0x10af310 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xfd88e0 con 0x10fa170 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 10 client.0 _ll_register_callbacks cb 0xfb7670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x10d7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x10d7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60404/0 (socket says 172.21.15.119:60404) 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 -- 172.21.15.119:0/3383921920 learned_addr learned my addr 172.21.15.119:0/3383921920 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb521e6700 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb529e7700 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10fa170 0x10fa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 -- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10fa170 msgr2=0x10fa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10fa170 0x10fa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 -- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 msgr2=0x10b6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 -- 172.21.15.119:0/3383921920 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c0b20b0 con 0x10f9d90 2024-09-13T16:36:14.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb519e5700 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x10d7560 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fbb480045f0 tx=0x7fbb48004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eee364696a2c960 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb2d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/3383921920 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4801c040 con 0x10f9d90 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/3383921920 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbb48017ad0 con 0x10f9d90 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xe4f2c0 con 0x10f9d90 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/3383921920 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb4800e040 con 0x10f9d90 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 10 client.9780 fetch_fsmap learned FSMap version 92 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.810+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0xfe1290 con 0x10f9d90 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/3383921920 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xe4f2c0 con 0x10f9d90 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/3383921920 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 92) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbb48023420 con 0x10f9d90 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 10 client.9780 fetch_fsmap finished waiting for FSMap version 92 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 10 client.9780 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-09-13T16:36:14.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 20 client.9780 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:14.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10b84a0 0x10d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 4 client.9780 mds_command: new command op to 9546 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:36:14.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 --> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xe99540 con 0x10b84a0 2024-09-13T16:36:14.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb521e6700 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10b84a0 0x10d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb2d7fa700 10 client.9780 ms_handle_connect on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:14.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb521e6700 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10b84a0 0x10d40c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f118fe7d7b779238 server_cookie=335a4cc38d62f2fc in_seq=0 out_seq=0 2024-09-13T16:36:14.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/3383921920 <== mds.0 v2:172.21.15.145:6834/3901706437 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0xe99540 con 0x10b84a0 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb2d7fa700 10 client.9780 handle_command_reply: tid=0 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 client.9780 shutdown 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 msgr2=0x10d7560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x10d7560 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fbb480045f0 tx=0x7fbb48004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 shutdown_connections 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10b84a0 0x10d40c0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10fa170 0x10fa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x10d7560 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 --2- 172.21.15.119:0/3383921920 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb2d7fa700 0 client.9780 ms_handle_reset on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 >> 172.21.15.119:0/3383921920 conn(0xe425b0 msgr2=0x10d84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 shutdown_connections 2024-09-13T16:36:14.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 1 -- 172.21.15.119:0/3383921920 wait complete. 2024-09-13T16:36:14.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.814+0000 7fbb576fb740 20 client.9780 trim_cache size 0 max 16384 2024-09-13T16:36:14.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.834+0000 7fbb576fb740 1 Processor -- start 2024-09-13T16:36:14.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.834+0000 7fbb576fb740 1 -- start start 2024-09-13T16:36:14.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe425b0 0x10b4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x1108da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0xec6b80 con 0x10f9d90 2024-09-13T16:36:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0xfdb810 con 0x10af310 2024-09-13T16:36:14.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0xfbfbd0 con 0xe425b0 2024-09-13T16:36:14.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x1108da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb521e6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe425b0 0x10b4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x1108da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60416/0 (socket says 172.21.15.119:60416) 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb519e5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 -- 172.21.15.119:0/4177660877 learned_addr learned my addr 172.21.15.119:0/4177660877 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 -- 172.21.15.119:0/4177660877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe425b0 msgr2=0x10b4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe425b0 0x10b4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 -- 172.21.15.119:0/4177660877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 msgr2=0x10b5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 -- 172.21.15.119:0/4177660877 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xfc74c0 con 0x10f9d90 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb521e6700 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe425b0 0x10b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb529e7700 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x1108da0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400f010 tx=0x7fbb4400b520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98b717cbd6534c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:14.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb40dad700 1 -- 172.21.15.119:0/4177660877 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44004230 con 0x10f9d90 2024-09-13T16:36:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb40dad700 1 -- 172.21.15.119:0/4177660877 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb440043b0 con 0x10f9d90 2024-09-13T16:36:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb40dad700 1 -- 172.21.15.119:0/4177660877 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb4400db90 con 0x10f9d90 2024-09-13T16:36:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- 172.21.15.119:0/4177660877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 msgr2=0x1108da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x1108da0 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fbb4400f010 tx=0x7fbb4400b520 comp rx=0 tx=0).stop 2024-09-13T16:36:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- 172.21.15.119:0/4177660877 shutdown_connections 2024-09-13T16:36:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x10f9d90 0x1108da0 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.843 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10af310 0x10b5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4177660877 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0xe425b0 0x10b4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- 172.21.15.119:0/4177660877 >> 172.21.15.119:0/4177660877 conn(0x10b3f10 msgr2=0x10b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:14.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- 172.21.15.119:0/4177660877 shutdown_connections 2024-09-13T16:36:14.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- 172.21.15.119:0/4177660877 wait complete. 2024-09-13T16:36:14.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 -- start start 2024-09-13T16:36:14.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 Processor -- start 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe425b0 0x115ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.838+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x115b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x115ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1111d40 con 0xe425b0 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1118060 con 0x10f9d90 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1111aa0 con 0x10af310 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 10 client.0 _ll_register_callbacks cb 0x1019610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb521e6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe425b0 0x115ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb529e7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x115ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb519e5700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x115b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb521e6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe425b0 0x115ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60424/0 (socket says 172.21.15.119:60424) 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb521e6700 1 -- 172.21.15.119:0/4171116558 learned_addr learned my addr 172.21.15.119:0/4171116558 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb529e7700 1 -- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 msgr2=0x115b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb529e7700 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x115b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb529e7700 1 -- 172.21.15.119:0/4171116558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe425b0 msgr2=0x115ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb529e7700 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe425b0 0x115ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb529e7700 1 -- 172.21.15.119:0/4171116558 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb4c151fd0 con 0x10f9d90 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb521e6700 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe425b0 0x115ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb519e5700 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x115b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb529e7700 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x115ba90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x101a730 tx=0x7fbb440047e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=844788bf57a99770 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb2d7fa700 10 client.0 ms_handle_connect on v2:172.21.15.145:3300/0 2024-09-13T16:36:14.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/4171116558 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb44017b50 con 0x10f9d90 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/4171116558 <== mon.1 v2:172.21.15.145:3300/0 2 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fbb4400b040 con 0x10f9d90 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb4400da20 con 0x10f9d90 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/4171116558 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_get_version_reply(handle=1 version=92) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb44017cb0 con 0x10f9d90 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 10 client.9358 fetch_fsmap learned FSMap version 92 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,fsmap=92,monmap=2+}) v3 -- 0xe8da40 con 0x10f9d90 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/4171116558 <== mon.1 v2:172.21.15.145:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4400da20 con 0x10f9d90 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/4171116558 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 92) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fbb44046020 con 0x10f9d90 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 10 client.9358 fetch_fsmap finished waiting for FSMap version 92 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 10 client.9358 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 20 client.9358 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10fa170 0x115bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 4 client.9358 mds_command: new command op to 9546 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-13T16:36:14.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 --> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0xebe2c0 con 0x10fa170 2024-09-13T16:36:14.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.842+0000 7fbb521e6700 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10fa170 0x115bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:14.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.846+0000 7fbb2d7fa700 10 client.9358 ms_handle_connect on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:14.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.846+0000 7fbb521e6700 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10fa170 0x115bfd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e627261e0fd61ce3 server_cookie=2a640294ee3bacda in_seq=0 out_seq=0 2024-09-13T16:36:14.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb2d7fa700 1 -- 172.21.15.119:0/4171116558 <== mds.0 v2:172.21.15.145:6834/3901706437 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0xebe2c0 con 0x10fa170 2024-09-13T16:36:14.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb2d7fa700 10 client.9358 handle_command_reply: tid=0 2024-09-13T16:36:14.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 client.9358 shutdown 2024-09-13T16:36:14.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 msgr2=0x115ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x115ba90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x101a730 tx=0x7fbb440047e0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 shutdown_connections 2024-09-13T16:36:14.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:6834/3901706437,v1:172.21.15.145:6835/3901706437] conn(0x10fa170 0x115bfd0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x10f9d90 0x115ba90 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x10af310 0x115b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb2d7fa700 0 client.9358 ms_handle_reset on v2:172.21.15.145:6834/3901706437 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 --2- 172.21.15.119:0/4171116558 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0xe425b0 0x115ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 >> 172.21.15.119:0/4171116558 conn(0x10b3f10 msgr2=0x1109f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 shutdown_connections 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 1 -- 172.21.15.119:0/4171116558 wait complete. 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb576fb740 20 client.9358 trim_cache size 0 max 16384 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:14.866 INFO:teuthology.orchestra.run.smithi119.stdout: "message": "", 2024-09-13T16:36:14.867 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0 2024-09-13T16:36:14.867 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:14.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb30066760 msgr2=0x7fbb30068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb30066760 0x7fbb30068c20 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7fbb4c159050 tx=0x7fbb4800b040 comp rx=0 tx=0).stop 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c06ca60 msgr2=0x7fbb4c14e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c06ca60 0x7fbb4c14e190 secure :-1 s=READY pgs=1845 cs=0 l=1 rev1=1 crypto rx=0x7fbb3c00c500 tx=0x7fbb3c00c810 comp rx=0 tx=0).stop 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 shutdown_connections 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbb30066760 0x7fbb30068c20 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbb4c0b1860 0x7fbb4c14ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbb4c0ad9a0 0x7fbb4c1e5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 --2- 172.21.15.119:0/2917660236 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbb4c06ca60 0x7fbb4c14e190 unknown :-1 s=CLOSED pgs=1845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 >> 172.21.15.119:0/2917660236 conn(0x7fbb4c0b0980 msgr2=0x7fbb4c0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 shutdown_connections 2024-09-13T16:36:14.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:14.862+0000 7fbb531e8700 1 -- 172.21.15.119:0/2917660236 wait complete. 2024-09-13T16:36:14.882 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:36:15.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603da05700 1 Processor -- start 2024-09-13T16:36:15.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603da05700 1 -- start start 2024-09-13T16:36:15.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603da05700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f6038159360 0x7f603806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603da05700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60380adf00 0x7f603806da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603da05700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6038071770 con 0x7f6038159740 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603da05700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60380cc890 con 0x7f6038159360 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603da05700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603815e0b0 con 0x7f60380adf00 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603d204700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f6038159740 0x7f603806fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:32980/0 (socket says 172.21.15.119:32980) 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603d204700 1 -- 172.21.15.119:0/4060675922 learned_addr learned my addr 172.21.15.119:0/4060675922 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f6037fff700 1 --2- 172.21.15.119:0/4060675922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60380adf00 0x7f603806da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:15.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f603ca03700 1 --2- 172.21.15.119:0/4060675922 >> v2:172.21.15.145:3300/0 conn(0x7f6038159360 0x7f603806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f6037fff700 1 -- 172.21.15.119:0/4060675922 >> v2:172.21.15.145:3300/0 conn(0x7f6038159360 msgr2=0x7f603806d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:15.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f6037fff700 1 --2- 172.21.15.119:0/4060675922 >> v2:172.21.15.145:3300/0 conn(0x7f6038159360 0x7f603806d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f6037fff700 1 -- 172.21.15.119:0/4060675922 >> v1:172.21.15.119:6789/0 conn(0x7f6038159740 legacy=0x7f603806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f60377fe700 1 -- 172.21.15.119:0/4060675922 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105605273 0 0) 0x7f6038071770 con 0x7f6038159740 2024-09-13T16:36:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.002+0000 7f6037fff700 1 -- 172.21.15.119:0/4060675922 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60380b2800 con 0x7f60380adf00 2024-09-13T16:36:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f6037fff700 1 --2- 172.21.15.119:0/4060675922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60380adf00 0x7f603806da70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6028009850 tx=0x7f6028009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5fe869b1a77a5cc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f60377fe700 1 -- 172.21.15.119:0/4060675922 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028011830 con 0x7f60380adf00 2024-09-13T16:36:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f60377fe700 1 -- 172.21.15.119:0/4060675922 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6028011990 con 0x7f60380adf00 2024-09-13T16:36:15.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f60377fe700 1 -- 172.21.15.119:0/4060675922 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6028011d40 con 0x7f60380adf00 2024-09-13T16:36:15.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 -- 172.21.15.119:0/4060675922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60380adf00 msgr2=0x7f603806da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:15.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 --2- 172.21.15.119:0/4060675922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60380adf00 0x7f603806da70 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f6028009850 tx=0x7f6028009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:15.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 -- 172.21.15.119:0/4060675922 shutdown_connections 2024-09-13T16:36:15.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 --2- 172.21.15.119:0/4060675922 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f60380adf00 0x7f603806da70 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 --2- 172.21.15.119:0/4060675922 >> v2:172.21.15.145:3300/0 conn(0x7f6038159360 0x7f603806d530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 -- 172.21.15.119:0/4060675922 >> 172.21.15.119:0/4060675922 conn(0x7f603806a350 msgr2=0x7f603806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:15.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 -- 172.21.15.119:0/4060675922 shutdown_connections 2024-09-13T16:36:15.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.006+0000 7f603da05700 1 -- 172.21.15.119:0/4060675922 wait complete. 2024-09-13T16:36:15.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 Processor -- start 2024-09-13T16:36:15.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 -- start start 2024-09-13T16:36:15.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60380adf00 0x7f603814e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:15.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6038159360 0x7f60380b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:15.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6038159740 0x7f603815be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:15.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603814d860 con 0x7f60380adf00 2024-09-13T16:36:15.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f603814e260 con 0x7f6038159360 2024-09-13T16:36:15.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603da05700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f603815ca60 con 0x7f6038159740 2024-09-13T16:36:15.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603ca03700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60380adf00 0x7f603814e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:15.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603ca03700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60380adf00 0x7f603814e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60438/0 (socket says 172.21.15.119:60438) 2024-09-13T16:36:15.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603ca03700 1 -- 172.21.15.119:0/2899107927 learned_addr learned my addr 172.21.15.119:0/2899107927 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:15.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f6037fff700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6038159360 0x7f60380b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:15.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.010+0000 7f603d204700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6038159740 0x7f603815be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603ca03700 1 -- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6038159740 msgr2=0x7f603815be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603ca03700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6038159740 0x7f603815be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603ca03700 1 -- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6038159360 msgr2=0x7f60380b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603ca03700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6038159360 0x7f60380b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603ca03700 1 -- 172.21.15.119:0/2899107927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60280094b0 con 0x7f60380adf00 2024-09-13T16:36:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603ca03700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60380adf00 0x7f603814e6f0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f602c00e840 tx=0x7f602c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f60357fa700 1 -- 172.21.15.119:0/2899107927 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c0095a0 con 0x7f60380adf00 2024-09-13T16:36:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f60357fa700 1 -- 172.21.15.119:0/2899107927 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f602c013680 con 0x7f60380adf00 2024-09-13T16:36:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f603814dc10 con 0x7f60380adf00 2024-09-13T16:36:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f60357fa700 1 -- 172.21.15.119:0/2899107927 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f602c0139f0 con 0x7f60380adf00 2024-09-13T16:36:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f603814ec30 con 0x7f60380adf00 2024-09-13T16:36:15.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6038071ba0 con 0x7f60380adf00 2024-09-13T16:36:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f60357fa700 1 -- 172.21.15.119:0/2899107927 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f602c013b50 con 0x7f60380adf00 2024-09-13T16:36:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f60357fa700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6024066730 0x7f6024068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f60357fa700 1 -- 172.21.15.119:0/2899107927 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f602c0841d0 con 0x7f60380adf00 2024-09-13T16:36:15.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.014+0000 7f6037fff700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6024066730 0x7f6024068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:15.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.018+0000 7f6037fff700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6024066730 0x7f6024068bf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f60280090b0 tx=0x7f602801c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:15.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.022+0000 7f60357fa700 1 -- 172.21.15.119:0/2899107927 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f602c0512d0 con 0x7f60380adf00 2024-09-13T16:36:15.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.234+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f6038071de0 con 0x7f60380adf00 2024-09-13T16:36:15.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.234+0000 7f60357fa700 1 -- 172.21.15.119:0/2899107927 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v92) v1 ==== 53+0+73 (secure 0 0 0) 0x7f602c0548f0 con 0x7f60380adf00 2024-09-13T16:36:15.237 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6024066730 msgr2=0x7f6024068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6024066730 0x7f6024068bf0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f60280090b0 tx=0x7f602801c040 comp rx=0 tx=0).stop 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60380adf00 msgr2=0x7f603814e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60380adf00 0x7f603814e6f0 secure :-1 s=READY pgs=1850 cs=0 l=1 rev1=1 crypto rx=0x7f602c00e840 tx=0x7f602c00eb50 comp rx=0 tx=0).stop 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 shutdown_connections 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6024066730 0x7f6024068bf0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6038159740 0x7f603815be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6038159360 0x7f60380b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 --2- 172.21.15.119:0/2899107927 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f60380adf00 0x7f603814e6f0 unknown :-1 s=CLOSED pgs=1850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:15.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 >> 172.21.15.119:0/2899107927 conn(0x7f603806a350 msgr2=0x7f603806fc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:15.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 shutdown_connections 2024-09-13T16:36:15.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:15.238+0000 7f603da05700 1 -- 172.21.15.119:0/2899107927 wait complete. 2024-09-13T16:36:15.257 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:36:15.257 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:36:15.257 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:36:15.257 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:36:15.257 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:15.257 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:36:15.258 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:36:15.258 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:36:15.258 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:15.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.258 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:15.267 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 41966sec preferred_lft 41966sec 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:15.268 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:36:15.269 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:15.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.270 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:36:15.284 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:36:15.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.286 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:36:15.339 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:36:15.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.340 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:36:15.341 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:36:15.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:36:15.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:36:15.341 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:36:15.341 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:36:15.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.382 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:36:15.390 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:36:15.390 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:36:15.390 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:15.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.390 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:36:15.447 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:36:15.447 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:15.447 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:15.455 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:15.455 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:15.455 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:15.461 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:15.474 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:15.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:15.489 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:15.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.490 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:15.496 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:36:15.496 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:36:15.540 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:15.555 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:15.555 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:15.564 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:15.565 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:15.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:15.565 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:15.630 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:36:15.626+0000 7fcb43e59640 -1 init, newargv = 0x5586023af240 newargc=15 2024-09-13T16:36:15.630 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[54130]: starting ceph client 2024-09-13T16:36:15.638 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[54130]: starting fuse 2024-09-13T16:36:16.672 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:16.691 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:16.691 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:16.747 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:16.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:16.748 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:16.755 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:36:16.755 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:16.756 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:16.757 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:36:16.758 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:16.842 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.54130.asok 2024-09-13T16:36:17.025 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54130.asok 2024-09-13T16:36:17.026 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:17.026 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54130.asok status 2024-09-13T16:36:17.169 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:17.169 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:36:17.169 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:36:17.169 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:36:17.169 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:36:17.169 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:36:17.169 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "54130", 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9801, 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 9801 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:17.170 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4020048728 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 4020048728 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.9801 192.168.0.1:0/4020048728", 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/4020048728", 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:36:17.171 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 92, 2024-09-13T16:36:17.172 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 126, 2024-09-13T16:36:17.172 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:36:17.172 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:36:17.172 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:36:17.172 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:17.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:17.178 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:36:17.179 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:17.186 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:36:17.186 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:17.186 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:17.187 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:17.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:17.202 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir parent/unflushed) 2024-09-13T16:36:17.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:17.261 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec dd if=/dev/urandom of=./parent/unflushed/jfile bs=1M count=8) 2024-09-13T16:36:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:8+0 records in 2024-09-13T16:36:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:8+0 records out 2024-09-13T16:36:17.343 INFO:teuthology.orchestra.run.smithi119.stderr:8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.07154 s, 117 MB/s 2024-09-13T16:36:17.345 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:17.345 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:17.345 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:17.345 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:17.345 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-09-13T16:36:17.345 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:17.407 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627779 2024-09-13T16:36:17.598 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:17.598 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:17.598 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:17.598 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:17.598 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-09-13T16:36:17.598 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:17.669 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627780 2024-09-13T16:36:17.858 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:17.858 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:17.908 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:18.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd48562f700 1 Processor -- start 2024-09-13T16:36:18.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd48562f700 1 -- start start 2024-09-13T16:36:18.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd48562f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd48006cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd48562f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd48006d520 0x7fd4800b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd48562f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd48006d940 con 0x7fd480160cc0 2024-09-13T16:36:18.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd48562f700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4800b2aa0 con 0x7fd48006d520 2024-09-13T16:36:18.036 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd48562f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800b4e30 con 0x7fd4800b49f0 2024-09-13T16:36:18.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd47ffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd480160cc0 0x7fd480164960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:32996/0 (socket says 172.21.15.119:32996) 2024-09-13T16:36:18.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd47ffff700 1 -- 172.21.15.119:0/3167906795 learned_addr learned my addr 172.21.15.119:0/3167906795 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:18.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd484e2e700 1 --2- 172.21.15.119:0/3167906795 >> v2:172.21.15.119:3300/0 conn(0x7fd48006d520 0x7fd4800b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd47f7fe700 1 --2- 172.21.15.119:0/3167906795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd48006cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd47effd700 1 -- 172.21.15.119:0/3167906795 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3053642725 0 0) 0x7fd48006d940 con 0x7fd480160cc0 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd484e2e700 1 -- 172.21.15.119:0/3167906795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 msgr2=0x7fd48006cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd484e2e700 1 --2- 172.21.15.119:0/3167906795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd48006cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd484e2e700 1 -- 172.21.15.119:0/3167906795 >> v1:172.21.15.119:6789/0 conn(0x7fd480160cc0 legacy=0x7fd480164960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd484e2e700 1 -- 172.21.15.119:0/3167906795 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd480165120 con 0x7fd48006d520 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd484e2e700 1 --2- 172.21.15.119:0/3167906795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd48006d520 0x7fd4800b31d0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd470009850 tx=0x7fd470009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e686deb5a6ffb7a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd47effd700 1 -- 172.21.15.119:0/3167906795 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470011830 con 0x7fd48006d520 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd47effd700 1 -- 172.21.15.119:0/3167906795 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd470011990 con 0x7fd48006d520 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.030+0000 7fd47effd700 1 -- 172.21.15.119:0/3167906795 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470011d40 con 0x7fd48006d520 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 -- 172.21.15.119:0/3167906795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd48006d520 msgr2=0x7fd4800b31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 --2- 172.21.15.119:0/3167906795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd48006d520 0x7fd4800b31d0 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7fd470009850 tx=0x7fd470009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:18.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 -- 172.21.15.119:0/3167906795 shutdown_connections 2024-09-13T16:36:18.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 --2- 172.21.15.119:0/3167906795 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd48006d520 0x7fd4800b31d0 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 --2- 172.21.15.119:0/3167906795 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd48006cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 -- 172.21.15.119:0/3167906795 >> 172.21.15.119:0/3167906795 conn(0x7fd4800b17a0 msgr2=0x7fd4800b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:18.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 -- 172.21.15.119:0/3167906795 shutdown_connections 2024-09-13T16:36:18.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 -- 172.21.15.119:0/3167906795 wait complete. 2024-09-13T16:36:18.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 Processor -- start 2024-09-13T16:36:18.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.034+0000 7fd48562f700 1 -- start start 2024-09-13T16:36:18.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd48006d520 0x7fd48014cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd4800cbe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd480160cc0 0x7fd4800ce5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800cc780 con 0x7fd480160cc0 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800cca60 con 0x7fd48006d520 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4800ccd40 con 0x7fd4800b49f0 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd480160cc0 0x7fd4800ce5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd47ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd48006d520 0x7fd48014cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd480160cc0 0x7fd4800ce5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60460/0 (socket says 172.21.15.119:60460) 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd47f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd4800cbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 -- 172.21.15.119:0/3119319242 learned_addr learned my addr 172.21.15.119:0/3119319242 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 -- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 msgr2=0x7fd4800cbe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd4800cbe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 -- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd48006d520 msgr2=0x7fd48014cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd48006d520 0x7fd48014cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 -- 172.21.15.119:0/3119319242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4700094b0 con 0x7fd480160cc0 2024-09-13T16:36:18.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd484e2e700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd480160cc0 0x7fd4800ce5c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fd470009ba0 tx=0x7fd470004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd47cff9700 1 -- 172.21.15.119:0/3119319242 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470004550 con 0x7fd480160cc0 2024-09-13T16:36:18.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd47cff9700 1 -- 172.21.15.119:0/3119319242 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd47001f070 con 0x7fd480160cc0 2024-09-13T16:36:18.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd47cff9700 1 -- 172.21.15.119:0/3119319242 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd470003900 con 0x7fd480160cc0 2024-09-13T16:36:18.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4800cd020 con 0x7fd480160cc0 2024-09-13T16:36:18.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.038+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd48014d170 con 0x7fd480160cc0 2024-09-13T16:36:18.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.042+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4800cd540 con 0x7fd480160cc0 2024-09-13T16:36:18.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.042+0000 7fd47cff9700 1 -- 172.21.15.119:0/3119319242 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd470003a60 con 0x7fd480160cc0 2024-09-13T16:36:18.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.042+0000 7fd47cff9700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd460066730 0x7fd460068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.042+0000 7fd47cff9700 1 -- 172.21.15.119:0/3119319242 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fd470021070 con 0x7fd480160cc0 2024-09-13T16:36:18.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.042+0000 7fd47ffff700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd460066730 0x7fd460068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.042+0000 7fd47ffff700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd460066730 0x7fd460068bf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd468000f60 tx=0x7fd468007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.050+0000 7fd47cff9700 1 -- 172.21.15.119:0/3119319242 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd4800cd540 con 0x7fd480160cc0 2024-09-13T16:36:18.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.218+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4800cd540 con 0x7fd480160cc0 2024-09-13T16:36:18.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd47cff9700 1 -- 172.21.15.119:0/3119319242 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16844 (secure 0 0 0) 0x7fd4800cd540 con 0x7fd480160cc0 2024-09-13T16:36:18.225 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:18.225 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":126,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:56.248923+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:18.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd460066730 msgr2=0x7fd460068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd460066730 0x7fd460068bf0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7fd468000f60 tx=0x7fd468007e90 comp rx=0 tx=0).stop 2024-09-13T16:36:18.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd480160cc0 msgr2=0x7fd4800ce5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd480160cc0 0x7fd4800ce5c0 secure :-1 s=READY pgs=1856 cs=0 l=1 rev1=1 crypto rx=0x7fd470009ba0 tx=0x7fd470004770 comp rx=0 tx=0).stop 2024-09-13T16:36:18.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 shutdown_connections 2024-09-13T16:36:18.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd460066730 0x7fd460068bf0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd480160cc0 0x7fd4800ce5c0 unknown :-1 s=CLOSED pgs=1856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd4800b49f0 0x7fd4800cbe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 --2- 172.21.15.119:0/3119319242 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd48006d520 0x7fd48014cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.222+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 >> 172.21.15.119:0/3119319242 conn(0x7fd4800b17a0 msgr2=0x7fd48015e530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:18.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.226+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 shutdown_connections 2024-09-13T16:36:18.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.226+0000 7fd48562f700 1 -- 172.21.15.119:0/3119319242 wait complete. 2024-09-13T16:36:18.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:18.240 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:18.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413c94c700 1 Processor -- start 2024-09-13T16:36:18.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413c94c700 1 -- start start 2024-09-13T16:36:18.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413c94c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f413815cc00 0x7f41380d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413c94c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41380d0c20 0x7f41380d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413c94c700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41380cdb00 con 0x7f41380ce8c0 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413c94c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41380cdde0 con 0x7f413815cc00 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413c94c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41380ce0c0 con 0x7f41380d0c20 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f413815cc00 0x7f41380d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f413815cc00 0x7f41380d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60464/0 (socket says 172.21.15.119:60464) 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 -- 172.21.15.119:0/629106949 learned_addr learned my addr 172.21.15.119:0/629106949 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:18.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f413659c700 1 --2- 172.21.15.119:0/629106949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41380d0c20 0x7f41380d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 -- 172.21.15.119:0/629106949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41380d0c20 msgr2=0x7f41380d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 --2- 172.21.15.119:0/629106949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41380d0c20 0x7f41380d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 -- 172.21.15.119:0/629106949 >> v1:172.21.15.145:6789/0 conn(0x7f41380ce8c0 legacy=0x7f4138160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 -- 172.21.15.119:0/629106949 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41380ce3a0 con 0x7f413815cc00 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4136d9d700 1 --2- 172.21.15.119:0/629106949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f413815cc00 0x7f41380d06e0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f4128009910 tx=0x7f4128009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44d08a1d1af61091 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4135d9b700 1 -- 172.21.15.119:0/629106949 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f412800ba60 con 0x7f413815cc00 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4135d9b700 1 -- 172.21.15.119:0/629106949 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f412800bbc0 con 0x7f413815cc00 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.350+0000 7f4135d9b700 1 -- 172.21.15.119:0/629106949 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4128009e60 con 0x7f413815cc00 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 -- 172.21.15.119:0/629106949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f413815cc00 msgr2=0x7f41380d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 --2- 172.21.15.119:0/629106949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f413815cc00 0x7f41380d06e0 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7f4128009910 tx=0x7f4128009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:18.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 -- 172.21.15.119:0/629106949 shutdown_connections 2024-09-13T16:36:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 --2- 172.21.15.119:0/629106949 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f41380d0c20 0x7f41380d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 --2- 172.21.15.119:0/629106949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f413815cc00 0x7f41380d06e0 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 -- 172.21.15.119:0/629106949 >> 172.21.15.119:0/629106949 conn(0x7f413808c080 msgr2=0x7f41380d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 -- 172.21.15.119:0/629106949 shutdown_connections 2024-09-13T16:36:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 -- 172.21.15.119:0/629106949 wait complete. 2024-09-13T16:36:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 Processor -- start 2024-09-13T16:36:18.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.354+0000 7f413c94c700 1 -- start start 2024-09-13T16:36:18.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41380ce8c0 0x7f413815d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41380d0c20 0x7f413815ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f413815cc00 0x7f413815f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41380b8550 con 0x7f41380d0c20 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41381605c0 con 0x7f41380ce8c0 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f413815fc90 con 0x7f413815cc00 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41380d0c20 0x7f413815ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41380d0c20 0x7f413815ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60470/0 (socket says 172.21.15.119:60470) 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 -- 172.21.15.119:0/297800180 learned_addr learned my addr 172.21.15.119:0/297800180 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f4136d9d700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41380ce8c0 0x7f413815d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413759e700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f413815cc00 0x7f413815f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.362 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 -- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f413815cc00 msgr2=0x7f413815f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f413815cc00 0x7f413815f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 -- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41380ce8c0 msgr2=0x7f413815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41380ce8c0 0x7f413815d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 -- 172.21.15.119:0/297800180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4128009570 con 0x7f41380d0c20 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413659c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41380d0c20 0x7f413815ef30 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f412000e760 tx=0x7f412000ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f412f7fe700 1 -- 172.21.15.119:0/297800180 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120013610 con 0x7f41380d0c20 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f412f7fe700 1 -- 172.21.15.119:0/297800180 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4120013770 con 0x7f41380d0c20 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f413815ff70 con 0x7f41380d0c20 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f412f7fe700 1 -- 172.21.15.119:0/297800180 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4120013ae0 con 0x7f41380d0c20 2024-09-13T16:36:18.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f413815f9b0 con 0x7f41380d0c20 2024-09-13T16:36:18.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.358+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41380d06e0 con 0x7f41380d0c20 2024-09-13T16:36:18.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.362+0000 7f412f7fe700 1 -- 172.21.15.119:0/297800180 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4120013c40 con 0x7f41380d0c20 2024-09-13T16:36:18.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.362+0000 7f412f7fe700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4118066730 0x7f4118068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.362+0000 7f412f7fe700 1 -- 172.21.15.119:0/297800180 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4120083f30 con 0x7f41380d0c20 2024-09-13T16:36:18.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.362+0000 7f4136d9d700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4118066730 0x7f4118068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.362+0000 7f4136d9d700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4118066730 0x7f4118068bf0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4128000c00 tx=0x7f4128008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.370+0000 7f412f7fe700 1 -- 172.21.15.119:0/297800180 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f41200510c0 con 0x7f41380d0c20 2024-09-13T16:36:18.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.538+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41380b5450 con 0x7f41380d0c20 2024-09-13T16:36:18.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.538+0000 7f412f7fe700 1 -- 172.21.15.119:0/297800180 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4120021020 con 0x7f41380d0c20 2024-09-13T16:36:18.541 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4118066730 msgr2=0x7f4118068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4118066730 0x7f4118068bf0 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f4128000c00 tx=0x7f4128008030 comp rx=0 tx=0).stop 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41380d0c20 msgr2=0x7f413815ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41380d0c20 0x7f413815ef30 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7f412000e760 tx=0x7f412000ea70 comp rx=0 tx=0).stop 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 shutdown_connections 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4118066730 0x7f4118068bf0 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f413815cc00 0x7f413815f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f41380d0c20 0x7f413815ef30 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 --2- 172.21.15.119:0/297800180 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f41380ce8c0 0x7f413815d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 >> 172.21.15.119:0/297800180 conn(0x7f413808c080 msgr2=0x7f413806c3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:18.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 shutdown_connections 2024-09-13T16:36:18.545 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.542+0000 7f413c94c700 1 -- 172.21.15.119:0/297800180 wait complete. 2024-09-13T16:36:18.556 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:18.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7dabfff700 1 Processor -- start 2024-09-13T16:36:18.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7dabfff700 1 -- start start 2024-09-13T16:36:18.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7dabfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7dac15abb0 0x7f7dac0cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7dabfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f7dac0b31e0 0x7f7dac0cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7dabfff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dac15af90 con 0x7f7dac0cd750 2024-09-13T16:36:18.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7dabfff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dac0cdbc0 con 0x7f7dac15abb0 2024-09-13T16:36:18.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7dabfff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dac0cb6d0 con 0x7f7dac0b31e0 2024-09-13T16:36:18.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7dac15abb0 0x7f7dac0cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.680 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f7dac15abb0 0x7f7dac0cf530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60484/0 (socket says 172.21.15.119:60484) 2024-09-13T16:36:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 -- 172.21.15.119:0/2931434572 learned_addr learned my addr 172.21.15.119:0/2931434572 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daa7fc700 1 --2- 172.21.15.119:0/2931434572 >> v2:172.21.15.145:3300/0 conn(0x7f7dac0b31e0 0x7f7dac0cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 -- 172.21.15.119:0/2931434572 >> v2:172.21.15.145:3300/0 conn(0x7f7dac0b31e0 msgr2=0x7f7dac0cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 --2- 172.21.15.119:0/2931434572 >> v2:172.21.15.145:3300/0 conn(0x7f7dac0b31e0 0x7f7dac0cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 -- 172.21.15.119:0/2931434572 >> v1:172.21.15.119:6789/0 conn(0x7f7dac0cd750 legacy=0x7f7dac0c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 -- 172.21.15.119:0/2931434572 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dac0d0060 con 0x7f7dac15abb0 2024-09-13T16:36:18.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daa7fc700 1 --2- 172.21.15.119:0/2931434572 >> v2:172.21.15.145:3300/0 conn(0x7f7dac0b31e0 0x7f7dac0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7daaffd700 1 --2- 172.21.15.119:0/2931434572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac15abb0 0x7f7dac0cf530 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7dac071810 tx=0x7f7d9c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92af281a08294785 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7da9ffb700 1 -- 172.21.15.119:0/2931434572 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c00b8a0 con 0x7f7dac15abb0 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7da9ffb700 1 -- 172.21.15.119:0/2931434572 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c00ba00 con 0x7f7dac15abb0 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.674+0000 7f7da9ffb700 1 -- 172.21.15.119:0/2931434572 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c00bd90 con 0x7f7dac15abb0 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- 172.21.15.119:0/2931434572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac15abb0 msgr2=0x7f7dac0cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2931434572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac15abb0 0x7f7dac0cf530 secure :-1 s=READY pgs=1860 cs=0 l=1 rev1=1 crypto rx=0x7f7dac071810 tx=0x7f7d9c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- 172.21.15.119:0/2931434572 shutdown_connections 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2931434572 >> v2:172.21.15.145:3300/0 conn(0x7f7dac0b31e0 0x7f7dac0cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2931434572 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac15abb0 0x7f7dac0cf530 unknown :-1 s=CLOSED pgs=1860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- 172.21.15.119:0/2931434572 >> 172.21.15.119:0/2931434572 conn(0x7f7dac0b25e0 msgr2=0x7f7dac0b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:18.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- 172.21.15.119:0/2931434572 shutdown_connections 2024-09-13T16:36:18.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- 172.21.15.119:0/2931434572 wait complete. 2024-09-13T16:36:18.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 Processor -- start 2024-09-13T16:36:18.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- start start 2024-09-13T16:36:18.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dac0b31e0 0x7f7dac15bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac0cd750 0x7f7dac1f0b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dac15abb0 0x7f7dac1ef8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac1f1050 con 0x7f7dac0cd750 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0b4db0 con 0x7f7dac0b31e0 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dac0b4830 con 0x7f7dac15abb0 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac0cd750 0x7f7dac1f0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac0cd750 0x7f7dac1f0b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60500/0 (socket says 172.21.15.119:60500) 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 -- 172.21.15.119:0/2687129750 learned_addr learned my addr 172.21.15.119:0/2687129750 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daaffd700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dac0b31e0 0x7f7dac15bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dab7fe700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dac15abb0 0x7f7dac1ef8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 -- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dac15abb0 msgr2=0x7f7dac1ef8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.686 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dac15abb0 0x7f7dac1ef8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 -- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dac0b31e0 msgr2=0x7f7dac15bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dac0b31e0 0x7f7dac15bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 -- 172.21.15.119:0/2687129750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9c009570 con 0x7f7dac0cd750 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dab7fe700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dac15abb0 0x7f7dac1ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7daa7fc700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac0cd750 0x7f7dac1f0b10 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c7f0 tx=0x7f7d9400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7da37fe700 1 -- 172.21.15.119:0/2687129750 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94011800 con 0x7f7dac0cd750 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7da37fe700 1 -- 172.21.15.119:0/2687129750 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d94011960 con 0x7f7dac0cd750 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dac0b4ad0 con 0x7f7dac0cd750 2024-09-13T16:36:18.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.678+0000 7f7da37fe700 1 -- 172.21.15.119:0/2687129750 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d94011d10 con 0x7f7dac0cd750 2024-09-13T16:36:18.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.682+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dac15c190 con 0x7f7dac0cd750 2024-09-13T16:36:18.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.682+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dac0cb040 con 0x7f7dac0cd750 2024-09-13T16:36:18.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.682+0000 7f7da37fe700 1 -- 172.21.15.119:0/2687129750 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f7d94021350 con 0x7f7dac0cd750 2024-09-13T16:36:18.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.682+0000 7f7da37fe700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d8c066730 0x7f7d8c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.682+0000 7f7da37fe700 1 -- 172.21.15.119:0/2687129750 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f7d94084210 con 0x7f7dac0cd750 2024-09-13T16:36:18.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.682+0000 7f7daaffd700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d8c066730 0x7f7d8c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.686+0000 7f7daaffd700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d8c066730 0x7f7d8c068bf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c000c00 tx=0x7f7d9c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.695 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.694+0000 7f7da37fe700 1 -- 172.21.15.119:0/2687129750 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f7d94051450 con 0x7f7dac0cd750 2024-09-13T16:36:18.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.862+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7dac0cd210 con 0x7f7dac0cd750 2024-09-13T16:36:18.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7da37fe700 1 -- 172.21.15.119:0/2687129750 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7d94059e60 con 0x7f7dac0cd750 2024-09-13T16:36:18.867 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:18.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d8c066730 msgr2=0x7f7d8c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d8c066730 0x7f7d8c068bf0 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c000c00 tx=0x7f7d9c008030 comp rx=0 tx=0).stop 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac0cd750 msgr2=0x7f7dac1f0b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac0cd750 0x7f7dac1f0b10 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7f7d9400c7f0 tx=0x7f7d9400cbb0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 shutdown_connections 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7d8c066730 0x7f7d8c068bf0 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f7dac15abb0 0x7f7dac1ef8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f7dac0cd750 0x7f7dac1f0b10 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 --2- 172.21.15.119:0/2687129750 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f7dac0b31e0 0x7f7dac15bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 >> 172.21.15.119:0/2687129750 conn(0x7f7dac0b25e0 msgr2=0x7f7dac0cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 shutdown_connections 2024-09-13T16:36:18.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.866+0000 7f7dabfff700 1 -- 172.21.15.119:0/2687129750 wait complete. 2024-09-13T16:36:18.881 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:18.993 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.990+0000 7f4d7dac7700 1 Processor -- start 2024-09-13T16:36:18.994 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.990+0000 7f4d7dac7700 1 -- start start 2024-09-13T16:36:18.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d7815cc00 0x7f4d780d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d17e0 0x7f4d780b5960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:18.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d780cdb00 con 0x7f4d780d0c20 2024-09-13T16:36:18.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d780cdde0 con 0x7f4d7815cc00 2024-09-13T16:36:18.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d780ce0c0 con 0x7f4d780d17e0 2024-09-13T16:36:18.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7d2c6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d17e0 0x7f4d780b5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.995 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d77fff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f4d780d0c20 0x7f4d780d10d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:43486/0 (socket says 172.21.15.119:43486) 2024-09-13T16:36:18.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7cac5700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4d7815cc00 0x7f4d780d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:18.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7d2c6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d17e0 0x7f4d780b5960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:42248/0 (socket says 172.21.15.119:42248) 2024-09-13T16:36:18.996 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d77fff700 1 -- 172.21.15.119:0/4259433173 learned_addr learned my addr 172.21.15.119:0/4259433173 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:18.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7d2c6700 1 -- 172.21.15.119:0/4259433173 >> v2:172.21.15.145:3300/0 conn(0x7f4d7815cc00 msgr2=0x7f4d780d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:18.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7d2c6700 1 --2- 172.21.15.119:0/4259433173 >> v2:172.21.15.145:3300/0 conn(0x7f4d7815cc00 0x7f4d780d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7d2c6700 1 -- 172.21.15.119:0/4259433173 >> v1:172.21.15.145:6789/0 conn(0x7f4d780d0c20 legacy=0x7f4d780d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.997 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7d2c6700 1 -- 172.21.15.119:0/4259433173 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d780ce3a0 con 0x7f4d780d17e0 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7d2c6700 1 --2- 172.21.15.119:0/4259433173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d17e0 0x7f4d780b5960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009d90 tx=0x7f4d7000b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dbc50125569f32f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d777fe700 1 -- 172.21.15.119:0/4259433173 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d7000e070 con 0x7f4d780d17e0 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d777fe700 1 -- 172.21.15.119:0/4259433173 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d7000bb20 con 0x7f4d780d17e0 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d777fe700 1 -- 172.21.15.119:0/4259433173 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70012430 con 0x7f4d780d17e0 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4259433173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d17e0 msgr2=0x7f4d780b5960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4259433173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d17e0 0x7f4d780b5960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d70009d90 tx=0x7f4d7000b330 comp rx=0 tx=0).stop 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4259433173 shutdown_connections 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4259433173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d17e0 0x7f4d780b5960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4259433173 >> v2:172.21.15.145:3300/0 conn(0x7f4d7815cc00 0x7f4d780d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4259433173 >> 172.21.15.119:0/4259433173 conn(0x7f4d7808c080 msgr2=0x7f4d780d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4259433173 shutdown_connections 2024-09-13T16:36:18.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4259433173 wait complete. 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 Processor -- start 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.994+0000 7f4d7dac7700 1 -- start start 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d0c20 0x7f4d7815d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d780d17e0 0x7f4d7814ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d7815cc00 0x7f4d780b5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7815e5b0 con 0x7f4d7815cc00 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7815d930 con 0x7f4d780d17e0 2024-09-13T16:36:19.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7815dc10 con 0x7f4d780d0c20 2024-09-13T16:36:19.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d7815cc00 0x7f4d780b5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d7815cc00 0x7f4d780b5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60504/0 (socket says 172.21.15.119:60504) 2024-09-13T16:36:19.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 -- 172.21.15.119:0/4264073257 learned_addr learned my addr 172.21.15.119:0/4264073257 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:19.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7cac5700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d0c20 0x7f4d7815d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d77fff700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d780d17e0 0x7f4d7814ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 -- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d0c20 msgr2=0x7f4d7815d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d0c20 0x7f4d7815d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 -- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d780d17e0 msgr2=0x7f4d7814ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d780d17e0 0x7f4d7814ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 -- 172.21.15.119:0/4264073257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70009a70 con 0x7f4d7815cc00 2024-09-13T16:36:19.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7d2c6700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d7815cc00 0x7f4d780b5fc0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f4d70014040 tx=0x7f4d70004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d757fa700 1 -- 172.21.15.119:0/4264073257 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d70012430 con 0x7f4d7815cc00 2024-09-13T16:36:19.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d757fa700 1 -- 172.21.15.119:0/4264073257 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d70020070 con 0x7f4d7815cc00 2024-09-13T16:36:19.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7815def0 con 0x7f4d7815cc00 2024-09-13T16:36:19.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d757fa700 1 -- 172.21.15.119:0/4264073257 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d700098f0 con 0x7f4d7815cc00 2024-09-13T16:36:19.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:18.998+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7814b3c0 con 0x7f4d7815cc00 2024-09-13T16:36:19.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.002+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d7815e6f0 con 0x7f4d7815cc00 2024-09-13T16:36:19.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.002+0000 7f4d757fa700 1 -- 172.21.15.119:0/4264073257 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4d70004040 con 0x7f4d7815cc00 2024-09-13T16:36:19.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.002+0000 7f4d757fa700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d64066730 0x7f4d64068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.002+0000 7f4d757fa700 1 -- 172.21.15.119:0/4264073257 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f4d70022070 con 0x7f4d7815cc00 2024-09-13T16:36:19.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.002+0000 7f4d7cac5700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d64066730 0x7f4d64068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.002+0000 7f4d7cac5700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d64066730 0x7f4d64068bf0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009b50 tx=0x7f4d6c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.006+0000 7f4d757fa700 1 -- 172.21.15.119:0/4264073257 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4d7815e6f0 con 0x7f4d7815cc00 2024-09-13T16:36:19.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.174+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4d7815e6f0 con 0x7f4d7815cc00 2024-09-13T16:36:19.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.174+0000 7f4d757fa700 1 -- 172.21.15.119:0/4264073257 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4d7815e6f0 con 0x7f4d7815cc00 2024-09-13T16:36:19.179 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:19.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d64066730 msgr2=0x7f4d64068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d64066730 0x7f4d64068bf0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c009b50 tx=0x7f4d6c009240 comp rx=0 tx=0).stop 2024-09-13T16:36:19.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d7815cc00 msgr2=0x7f4d780b5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d7815cc00 0x7f4d780b5fc0 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7f4d70014040 tx=0x7f4d70004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 shutdown_connections 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d64066730 0x7f4d64068bf0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d7815cc00 0x7f4d780b5fc0 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d780d17e0 0x7f4d7814ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 --2- 172.21.15.119:0/4264073257 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d780d0c20 0x7f4d7815d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 >> 172.21.15.119:0/4264073257 conn(0x7f4d7808c080 msgr2=0x7f4d781609f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:19.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 shutdown_connections 2024-09-13T16:36:19.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.178+0000 7f4d7dac7700 1 -- 172.21.15.119:0/4264073257 wait complete. 2024-09-13T16:36:19.195 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:36:19.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb42222f700 1 Processor -- start 2024-09-13T16:36:19.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb42222f700 1 -- start start 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb42222f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb41c0d2a30 0x7fb41c0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb42222f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c0ce030 con 0x7fb41c0afae0 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb42222f700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41c0b3980 con 0x7fb41c0af670 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb42222f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb41c0cd9e0 con 0x7fb41c0d2a30 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb421a2e700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb41c0afae0 0x7fb41c0b2c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33006/0 (socket says 172.21.15.119:33006) 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.298+0000 7fb421a2e700 1 -- 172.21.15.119:0/3064373036 learned_addr learned my addr 172.21.15.119:0/3064373036 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42122d700 1 --2- 172.21.15.119:0/3064373036 >> v2:172.21.15.145:3300/0 conn(0x7fb41c0d2a30 0x7fb41c0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226773826 0 0) 0x7fb41c0ce030 con 0x7fb41c0afae0 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4080035d0 con 0x7fb41c0afae0 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3501833053 0 0) 0x7fb41c0b3980 con 0x7fb41c0af670 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb41c0ce030 con 0x7fb41c0af670 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2327905396 0 0) 0x7fb4080035d0 con 0x7fb41c0afae0 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 >> v2:172.21.15.145:3300/0 conn(0x7fb41c0d2a30 msgr2=0x7fb41c0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 --2- 172.21.15.119:0/3064373036 >> v2:172.21.15.145:3300/0 conn(0x7fb41c0d2a30 0x7fb41c0d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 >> v1:172.21.15.145:6789/0 conn(0x7fb41c0af670 legacy=0x7fb41c0ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c0cdcc0 con 0x7fb41c0afae0 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7fb4140019f0 con 0x7fb41c0afae0 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 >> v1:172.21.15.119:6789/0 conn(0x7fb41c0afae0 legacy=0x7fb41c0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb408003c00 0x7fb408004000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb408004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb408005280 0x7fb40800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c0cd9e0 con 0x7fb408004870 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40800c1b0 con 0x7fb408003c00 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40800c370 con 0x7fb408005280 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb420a2c700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb408004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42122d700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb408003c00 0x7fb408004000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb421a2e700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb408005280 0x7fb40800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb420a2c700 1 -- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb408005280 msgr2=0x7fb40800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb420a2c700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb408005280 0x7fb40800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb420a2c700 1 -- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb408003c00 msgr2=0x7fb408004000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb420a2c700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb408003c00 0x7fb408004000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb421a2e700 1 -- 172.21.15.119:0/3064373036 reap_dead start 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb420a2c700 1 -- 172.21.15.119:0/3064373036 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4080035d0 con 0x7fb408004870 2024-09-13T16:36:19.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb420a2c700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb408004cf0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fb418001960 tx=0x7fb4080058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93524b8f327666fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418001450 con 0x7fb408004870 2024-09-13T16:36:19.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb418006aa0 con 0x7fb408004870 2024-09-13T16:36:19.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb413fff700 1 -- 172.21.15.119:0/3064373036 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb418006e50 con 0x7fb408004870 2024-09-13T16:36:19.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42222f700 1 -- 172.21.15.119:0/3064373036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 msgr2=0x7fb408004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42222f700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb408004cf0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7fb418001960 tx=0x7fb4080058a0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42222f700 1 -- 172.21.15.119:0/3064373036 shutdown_connections 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42222f700 1 --2- 172.21.15.119:0/3064373036 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb408004cf0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42222f700 1 -- 172.21.15.119:0/3064373036 >> 172.21.15.119:0/3064373036 conn(0x7fb41c08c0f0 msgr2=0x7fb41c08c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42222f700 1 -- 172.21.15.119:0/3064373036 shutdown_connections 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.302+0000 7fb42222f700 1 -- 172.21.15.119:0/3064373036 wait complete. 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 Processor -- start 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 -- start start 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb41c1563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb41c15a520 0x7fb41c155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb41c0b7f00 0x7fb41c0b8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c15c760 con 0x7fb408004870 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c15aae0 con 0x7fb41c15a520 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41c15c110 con 0x7fb41c0b7f00 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb41c1563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb41c1563b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60514/0 (socket says 172.21.15.119:60514) 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 -- 172.21.15.119:0/2858988918 learned_addr learned my addr 172.21.15.119:0/2858988918 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb420a2c700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb41c15a520 0x7fb41c155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb421a2e700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb41c0b7f00 0x7fb41c0b8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 -- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb41c0b7f00 msgr2=0x7fb41c0b8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb41c0b7f00 0x7fb41c0b8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 -- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb41c15a520 msgr2=0x7fb41c155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb41c15a520 0x7fb41c155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 -- 172.21.15.119:0/2858988918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41c15c3f0 con 0x7fb408004870 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42122d700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb41c1563b0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fb40c011920 tx=0x7fb40c011c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb411ffb700 1 -- 172.21.15.119:0/2858988918 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c004240 con 0x7fb408004870 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb411ffb700 1 -- 172.21.15.119:0/2858988918 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c0043c0 con 0x7fb408004870 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb411ffb700 1 -- 172.21.15.119:0/2858988918 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40c0036c0 con 0x7fb408004870 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb41c1560f0 con 0x7fb408004870 2024-09-13T16:36:19.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.306+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41c0caf00 con 0x7fb408004870 2024-09-13T16:36:19.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.310+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb41c15bf70 con 0x7fb408004870 2024-09-13T16:36:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.310+0000 7fb411ffb700 1 -- 172.21.15.119:0/2858988918 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb40c003820 con 0x7fb408004870 2024-09-13T16:36:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.310+0000 7fb411ffb700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb400066730 0x7fb400068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.310+0000 7fb411ffb700 1 -- 172.21.15.119:0/2858988918 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb40c0845b0 con 0x7fb408004870 2024-09-13T16:36:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.310+0000 7fb420a2c700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb400066730 0x7fb400068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.310+0000 7fb420a2c700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb400066730 0x7fb400068bf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb418015040 tx=0x7fb418005040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.318+0000 7fb411ffb700 1 -- 172.21.15.119:0/2858988918 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb41c15bf70 con 0x7fb408004870 2024-09-13T16:36:19.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.502+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb41c15bf70 con 0x7fb408004870 2024-09-13T16:36:19.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.502+0000 7fb411ffb700 1 -- 172.21.15.119:0/2858988918 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v126) v1 ==== 80+0+0 (secure 0 0 0) 0x7fb41c15bf70 con 0x7fb408004870 2024-09-13T16:36:19.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb400066730 msgr2=0x7fb400068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb400066730 0x7fb400068bf0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fb418015040 tx=0x7fb418005040 comp rx=0 tx=0).stop 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 msgr2=0x7fb41c1563b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb41c1563b0 secure :-1 s=READY pgs=1865 cs=0 l=1 rev1=1 crypto rx=0x7fb40c011920 tx=0x7fb40c011c30 comp rx=0 tx=0).stop 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 shutdown_connections 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb400066730 0x7fb400068bf0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb41c0b7f00 0x7fb41c0b8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb41c15a520 0x7fb41c155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 --2- 172.21.15.119:0/2858988918 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb408004870 0x7fb41c1563b0 unknown :-1 s=CLOSED pgs=1865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 >> 172.21.15.119:0/2858988918 conn(0x7fb41c08c0f0 msgr2=0x7fb41c156b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 shutdown_connections 2024-09-13T16:36:19.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.506+0000 7fb42222f700 1 -- 172.21.15.119:0/2858988918 wait complete. 2024-09-13T16:36:19.509 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:36:19.519 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:19.519 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:19.527 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:19.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.634+0000 7f93d2c58700 1 Processor -- start 2024-09-13T16:36:19.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.634+0000 7f93d2c58700 1 -- start start 2024-09-13T16:36:19.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f93cc071740 0x7f93cc06ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f93cc158f10 0x7f93cc06d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc06de00 con 0x7f93cc158b30 2024-09-13T16:36:19.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc06d8b0 con 0x7f93cc158f10 2024-09-13T16:36:19.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc06db20 con 0x7f93cc071740 2024-09-13T16:36:19.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f93cc158f10 0x7f93cc06d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f93cc158f10 0x7f93cc06d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60516/0 (socket says 172.21.15.119:60516) 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 -- 172.21.15.119:0/2570983772 learned_addr learned my addr 172.21.15.119:0/2570983772 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d1455700 1 --2- 172.21.15.119:0/2570983772 >> v2:172.21.15.145:3300/0 conn(0x7f93cc071740 0x7f93cc06ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 -- 172.21.15.119:0/2570983772 >> v2:172.21.15.145:3300/0 conn(0x7f93cc071740 msgr2=0x7f93cc06ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 --2- 172.21.15.119:0/2570983772 >> v2:172.21.15.145:3300/0 conn(0x7f93cc071740 0x7f93cc06ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 -- 172.21.15.119:0/2570983772 >> v1:172.21.15.145:6789/0 conn(0x7f93cc158b30 legacy=0x7f93cc071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 -- 172.21.15.119:0/2570983772 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc0b1920 con 0x7f93cc158f10 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2457700 1 --2- 172.21.15.119:0/2570983772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc158f10 0x7f93cc06d370 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f93cc070dd0 tx=0x7f93c4009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=794da9824e328be5 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d0c54700 1 -- 172.21.15.119:0/2570983772 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4011830 con 0x7f93cc158f10 2024-09-13T16:36:19.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d0c54700 1 -- 172.21.15.119:0/2570983772 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c4011990 con 0x7f93cc158f10 2024-09-13T16:36:19.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d0c54700 1 -- 172.21.15.119:0/2570983772 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c4011d20 con 0x7f93cc158f10 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- 172.21.15.119:0/2570983772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc158f10 msgr2=0x7f93cc06d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2570983772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc158f10 0x7f93cc06d370 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f93cc070dd0 tx=0x7f93c4009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- 172.21.15.119:0/2570983772 shutdown_connections 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2570983772 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc158f10 0x7f93cc06d370 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2570983772 >> v2:172.21.15.145:3300/0 conn(0x7f93cc071740 0x7f93cc06ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- 172.21.15.119:0/2570983772 >> 172.21.15.119:0/2570983772 conn(0x7f93cc06a350 msgr2=0x7f93cc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- 172.21.15.119:0/2570983772 shutdown_connections 2024-09-13T16:36:19.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- 172.21.15.119:0/2570983772 wait complete. 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 Processor -- start 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.638+0000 7f93d2c58700 1 -- start start 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc071740 0x7f93cc156120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93cc158b30 0x7f93cc15a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93cc158f10 0x7f93cc0ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc14a4d0 con 0x7f93cc071740 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc156740 con 0x7f93cc158b30 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93cc156a20 con 0x7f93cc158f10 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc071740 0x7f93cc156120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc071740 0x7f93cc156120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60522/0 (socket says 172.21.15.119:60522) 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 -- 172.21.15.119:0/2385907154 learned_addr learned my addr 172.21.15.119:0/2385907154 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1455700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93cc158b30 0x7f93cc15a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2457700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93cc158f10 0x7f93cc0ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 -- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93cc158f10 msgr2=0x7f93cc0ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93cc158f10 0x7f93cc0ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 -- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93cc158b30 msgr2=0x7f93cc15a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93cc158b30 0x7f93cc15a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 -- 172.21.15.119:0/2385907154 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c4009570 con 0x7f93cc071740 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1455700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93cc158b30 0x7f93cc15a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d1c56700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc071740 0x7f93cc156120 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f93c000c500 tx=0x7f93c000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93be7fc700 1 -- 172.21.15.119:0/2385907154 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c00052d0 con 0x7f93cc071740 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93be7fc700 1 -- 172.21.15.119:0/2385907154 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93c0005430 con 0x7f93cc071740 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc156d00 con 0x7f93cc071740 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93be7fc700 1 -- 172.21.15.119:0/2385907154 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93c00055a0 con 0x7f93cc071740 2024-09-13T16:36:19.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc1572c0 con 0x7f93cc071740 2024-09-13T16:36:19.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.642+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc06c0d0 con 0x7f93cc071740 2024-09-13T16:36:19.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.646+0000 7f93be7fc700 1 -- 172.21.15.119:0/2385907154 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f93c00036a0 con 0x7f93cc071740 2024-09-13T16:36:19.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.646+0000 7f93be7fc700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f93b0066730 0x7f93b0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.646+0000 7f93be7fc700 1 -- 172.21.15.119:0/2385907154 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f93c00835d0 con 0x7f93cc071740 2024-09-13T16:36:19.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.646+0000 7f93d1455700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f93b0066730 0x7f93b0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.646+0000 7f93d1455700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f93b0066730 0x7f93b0068bf0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f93cc139200 tx=0x7f93c800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.654+0000 7f93be7fc700 1 -- 172.21.15.119:0/2385907154 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f93c0050800 con 0x7f93cc071740 2024-09-13T16:36:19.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.826+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93cc06abf0 con 0x7f93cc071740 2024-09-13T16:36:19.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.830+0000 7f93be7fc700 1 -- 172.21.15.119:0/2385907154 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+16844 (secure 0 0 0) 0x7f93c0053e20 con 0x7f93cc071740 2024-09-13T16:36:19.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:19.833 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":126,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:35:56.248923+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:19.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f93b0066730 msgr2=0x7f93b0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f93b0066730 0x7f93b0068bf0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f93cc139200 tx=0x7f93c800b040 comp rx=0 tx=0).stop 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc071740 msgr2=0x7f93cc156120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc071740 0x7f93cc156120 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f93c000c500 tx=0x7f93c000c8c0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 shutdown_connections 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f93b0066730 0x7f93b0068bf0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f93cc158f10 0x7f93cc0ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f93cc158b30 0x7f93cc15a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 --2- 172.21.15.119:0/2385907154 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f93cc071740 0x7f93cc156120 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 >> 172.21.15.119:0/2385907154 conn(0x7f93cc06a350 msgr2=0x7f93cc1642b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 shutdown_connections 2024-09-13T16:36:19.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.834+0000 7f93d2c58700 1 -- 172.21.15.119:0/2385907154 wait complete. 2024-09-13T16:36:19.848 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:19.848 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:19.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.966+0000 7fb2e29d8700 1 Processor -- start 2024-09-13T16:36:19.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.966+0000 7fb2e29d8700 1 -- start start 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb2dc158e60 0x7fb2dc15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb2dc0ac6b0 0x7fb2dc0acb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc0abf40 con 0x7fb2dc0cb1c0 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc06db50 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc06ddc0 con 0x7fb2dc158e60 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e21d7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb2dc0ac6b0 0x7fb2dc0acb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e11d5700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb2dc0cb1c0 0x7fb2dc06c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33014/0 (socket says 172.21.15.119:33014) 2024-09-13T16:36:19.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e11d5700 1 -- 172.21.15.119:0/4289902382 learned_addr learned my addr 172.21.15.119:0/4289902382 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:19.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e19d6700 1 --2- 172.21.15.119:0/4289902382 >> v2:172.21.15.145:3300/0 conn(0x7fb2dc158e60 0x7fb2dc15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e21d7700 1 -- 172.21.15.119:0/4289902382 >> v2:172.21.15.145:3300/0 conn(0x7fb2dc158e60 msgr2=0x7fb2dc15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:19.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e21d7700 1 --2- 172.21.15.119:0/4289902382 >> v2:172.21.15.145:3300/0 conn(0x7fb2dc158e60 0x7fb2dc15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e21d7700 1 -- 172.21.15.119:0/4289902382 >> v1:172.21.15.119:6789/0 conn(0x7fb2dc0cb1c0 legacy=0x7fb2dc06c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e09d4700 1 -- 172.21.15.119:0/4289902382 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111529196 0 0) 0x7fb2dc0abf40 con 0x7fb2dc0cb1c0 2024-09-13T16:36:19.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e21d7700 1 -- 172.21.15.119:0/4289902382 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc06e0a0 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e21d7700 1 --2- 172.21.15.119:0/4289902382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc0acb40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc074780 tx=0x7fb2d400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=210e532762b1a37a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e09d4700 1 -- 172.21.15.119:0/4289902382 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d400f070 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e09d4700 1 -- 172.21.15.119:0/4289902382 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d400a2e0 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e09d4700 1 -- 172.21.15.119:0/4289902382 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d400cb50 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- 172.21.15.119:0/4289902382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 msgr2=0x7fb2dc0acb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/4289902382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc0acb40 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc074780 tx=0x7fb2d400c330 comp rx=0 tx=0).stop 2024-09-13T16:36:19.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- 172.21.15.119:0/4289902382 shutdown_connections 2024-09-13T16:36:19.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/4289902382 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc0acb40 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/4289902382 >> v2:172.21.15.145:3300/0 conn(0x7fb2dc158e60 0x7fb2dc15d300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- 172.21.15.119:0/4289902382 >> 172.21.15.119:0/4289902382 conn(0x7fb2dc0b0cd0 msgr2=0x7fb2dc0b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:19.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- 172.21.15.119:0/4289902382 shutdown_connections 2024-09-13T16:36:19.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.970+0000 7fb2e29d8700 1 -- 172.21.15.119:0/4289902382 wait complete. 2024-09-13T16:36:19.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.974+0000 7fb2e29d8700 1 Processor -- start 2024-09-13T16:36:19.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.974+0000 7fb2e29d8700 1 -- start start 2024-09-13T16:36:19.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.974+0000 7fb2e29d8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc15c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2dc0cb1c0 0x7fb2dc14fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2dc158e60 0x7fb2dc0cd4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc151030 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0cb8f0 con 0x7fb2dc0cb1c0 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc151240 con 0x7fb2dc158e60 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc15c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc15c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60540/0 (socket says 172.21.15.119:60540) 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 -- 172.21.15.119:0/2726304681 learned_addr learned my addr 172.21.15.119:0/2726304681 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e21d7700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2dc158e60 0x7fb2dc0cd4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e11d5700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2dc0cb1c0 0x7fb2dc14fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 -- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2dc158e60 msgr2=0x7fb2dc0cd4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2dc158e60 0x7fb2dc0cd4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 -- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2dc0cb1c0 msgr2=0x7fb2dc14fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:19.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2dc0cb1c0 0x7fb2dc14fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:19.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 -- 172.21.15.119:0/2726304681 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d400ab80 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e19d6700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc15c030 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800e840 tx=0x7fb2d800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2d27fc700 1 -- 172.21.15.119:0/2726304681 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d80095a0 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2d27fc700 1 -- 172.21.15.119:0/2726304681 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d8013680 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc06cbf0 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2d27fc700 1 -- 172.21.15.119:0/2726304681 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d80139f0 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc151800 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc15d300 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2d27fc700 1 -- 172.21.15.119:0/2726304681 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb2d8013b50 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.978+0000 7fb2d27fc700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb2c0066730 0x7fb2c0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:19.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.982+0000 7fb2d27fc700 1 -- 172.21.15.119:0/2726304681 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7fb2d8084150 con 0x7fb2dc0ac6b0 2024-09-13T16:36:19.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.982+0000 7fb2e11d5700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb2c0066730 0x7fb2c0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:19.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.982+0000 7fb2e11d5700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb2c0066730 0x7fb2c0068bf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc15b8a0 tx=0x7fb2cc007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:19.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:19.990+0000 7fb2d27fc700 1 -- 172.21.15.119:0/2726304681 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb2d80512f0 con 0x7fb2dc0ac6b0 2024-09-13T16:36:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.150+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2dc0b4290 con 0x7fb2dc0ac6b0 2024-09-13T16:36:20.154 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.150+0000 7fb2d27fc700 1 -- 172.21.15.119:0/2726304681 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2d8054910 con 0x7fb2dc0ac6b0 2024-09-13T16:36:20.154 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb2c0066730 msgr2=0x7fb2c0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb2c0066730 0x7fb2c0068bf0 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc15b8a0 tx=0x7fb2cc007e90 comp rx=0 tx=0).stop 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 msgr2=0x7fb2dc15c030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc15c030 secure :-1 s=READY pgs=1870 cs=0 l=1 rev1=1 crypto rx=0x7fb2d800e840 tx=0x7fb2d800eb50 comp rx=0 tx=0).stop 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 shutdown_connections 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb2c0066730 0x7fb2c0068bf0 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb2dc158e60 0x7fb2dc0cd4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb2dc0cb1c0 0x7fb2dc14fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 --2- 172.21.15.119:0/2726304681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb2dc0ac6b0 0x7fb2dc15c030 unknown :-1 s=CLOSED pgs=1870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 >> 172.21.15.119:0/2726304681 conn(0x7fb2dc0b0cd0 msgr2=0x7fb2dc06c360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 shutdown_connections 2024-09-13T16:36:20.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.154+0000 7fb2e29d8700 1 -- 172.21.15.119:0/2726304681 wait complete. 2024-09-13T16:36:20.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:20.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.286+0000 7f2e4d9d1700 1 Processor -- start 2024-09-13T16:36:20.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.286+0000 7f2e4d9d1700 1 -- start start 2024-09-13T16:36:20.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4d9d1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2e4815abb0 0x7f2e480cf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4d9d1700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e4815af90 con 0x7f2e480b31e0 2024-09-13T16:36:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4d9d1700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e480cb6d0 con 0x7f2e480cd920 2024-09-13T16:36:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4d9d1700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e480d0060 con 0x7f2e4815abb0 2024-09-13T16:36:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e47fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2e480b31e0 0x7f2e480cd210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33018/0 (socket says 172.21.15.119:33018) 2024-09-13T16:36:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e47fff700 1 -- 172.21.15.119:0/393988279 learned_addr learned my addr 172.21.15.119:0/393988279 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:20.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4c9cf700 1 --2- 172.21.15.119:0/393988279 >> v2:172.21.15.119:3300/0 conn(0x7f2e4815abb0 0x7f2e480cf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e477fe700 1 -- 172.21.15.119:0/393988279 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322451972 0 0) 0x7f2e30001110 con 0x7f2e480b31e0 2024-09-13T16:36:20.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e477fe700 1 -- 172.21.15.119:0/393988279 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e340035d0 con 0x7f2e480b31e0 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e477fe700 1 -- 172.21.15.119:0/393988279 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427189615 0 0) 0x7f2e480cb6d0 con 0x7f2e480cd920 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4c9cf700 1 -- 172.21.15.119:0/393988279 >> v1:172.21.15.145:6789/0 conn(0x7f2e480cd920 legacy=0x7f2e480cbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4c9cf700 1 -- 172.21.15.119:0/393988279 >> v1:172.21.15.119:6789/0 conn(0x7f2e480b31e0 legacy=0x7f2e480cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4c9cf700 1 -- 172.21.15.119:0/393988279 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e480cfd80 con 0x7f2e4815abb0 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e4c9cf700 1 --2- 172.21.15.119:0/393988279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e4815abb0 0x7f2e480cf530 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009850 tx=0x7f2e38009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=270fc3325b71add1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e477fe700 1 -- 172.21.15.119:0/393988279 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38011830 con 0x7f2e4815abb0 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e477fe700 1 -- 172.21.15.119:0/393988279 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e38011990 con 0x7f2e4815abb0 2024-09-13T16:36:20.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.290+0000 7f2e477fe700 1 -- 172.21.15.119:0/393988279 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38011d40 con 0x7f2e4815abb0 2024-09-13T16:36:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/393988279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e4815abb0 msgr2=0x7f2e480cf530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/393988279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e4815abb0 0x7f2e480cf530 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f2e38009850 tx=0x7f2e38009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:20.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/393988279 shutdown_connections 2024-09-13T16:36:20.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/393988279 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e4815abb0 0x7f2e480cf530 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/393988279 >> 172.21.15.119:0/393988279 conn(0x7f2e480b25e0 msgr2=0x7f2e480b29f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:20.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/393988279 shutdown_connections 2024-09-13T16:36:20.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/393988279 wait complete. 2024-09-13T16:36:20.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 Processor -- start 2024-09-13T16:36:20.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.294+0000 7f2e4d9d1700 1 -- start start 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e480b31e0 0x7f2e480c89f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e480cd920 0x7f2e481f00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e4815abb0 0x7f2e481f05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e480c9c50 con 0x7f2e480b31e0 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e480b4b60 con 0x7f2e4815abb0 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e480c93b0 con 0x7f2e480cd920 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e480b31e0 0x7f2e480c89f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e480b31e0 0x7f2e480c89f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60556/0 (socket says 172.21.15.119:60556) 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 -- 172.21.15.119:0/4115410037 learned_addr learned my addr 172.21.15.119:0/4115410037 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:20.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d1d0700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e4815abb0 0x7f2e481f05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e47fff700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e480cd920 0x7f2e481f00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 -- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e480cd920 msgr2=0x7f2e481f00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e480cd920 0x7f2e481f00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 -- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e4815abb0 msgr2=0x7f2e481f05f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e4815abb0 0x7f2e481f05f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 -- 172.21.15.119:0/4115410037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e380094b0 con 0x7f2e480b31e0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d1d0700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e4815abb0 0x7f2e481f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4c9cf700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e480b31e0 0x7f2e480c89f0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2e380090b0 tx=0x7f2e38004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e457fa700 1 -- 172.21.15.119:0/4115410037 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e38019b60 con 0x7f2e480b31e0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e457fa700 1 -- 172.21.15.119:0/4115410037 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e3801e070 con 0x7f2e480b31e0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e480c9690 con 0x7f2e480b31e0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e457fa700 1 -- 172.21.15.119:0/4115410037 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e3801a420 con 0x7f2e480b31e0 2024-09-13T16:36:20.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.298+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e480cacc0 con 0x7f2e480b31e0 2024-09-13T16:36:20.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.302+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e480b4930 con 0x7f2e480b31e0 2024-09-13T16:36:20.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.302+0000 7f2e457fa700 1 -- 172.21.15.119:0/4115410037 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2e3801a580 con 0x7f2e480b31e0 2024-09-13T16:36:20.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.302+0000 7f2e457fa700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e28066730 0x7f2e28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.302+0000 7f2e457fa700 1 -- 172.21.15.119:0/4115410037 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f2e380853d0 con 0x7f2e480b31e0 2024-09-13T16:36:20.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.302+0000 7f2e47fff700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e28066730 0x7f2e28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.302+0000 7f2e47fff700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e28066730 0x7f2e28068bf0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2e480ca050 tx=0x7f2e30009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.310+0000 7f2e457fa700 1 -- 172.21.15.119:0/4115410037 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2e480b4930 con 0x7f2e480b31e0 2024-09-13T16:36:20.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.478+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2e480b4930 con 0x7f2e480b31e0 2024-09-13T16:36:20.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.478+0000 7f2e457fa700 1 -- 172.21.15.119:0/4115410037 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2e480b4930 con 0x7f2e480b31e0 2024-09-13T16:36:20.482 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:20.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e28066730 msgr2=0x7f2e28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e28066730 0x7f2e28068bf0 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f2e480ca050 tx=0x7f2e30009040 comp rx=0 tx=0).stop 2024-09-13T16:36:20.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e480b31e0 msgr2=0x7f2e480c89f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e480b31e0 0x7f2e480c89f0 secure :-1 s=READY pgs=1873 cs=0 l=1 rev1=1 crypto rx=0x7f2e380090b0 tx=0x7f2e38004770 comp rx=0 tx=0).stop 2024-09-13T16:36:20.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 shutdown_connections 2024-09-13T16:36:20.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2e28066730 0x7f2e28068bf0 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2e4815abb0 0x7f2e481f05f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2e480cd920 0x7f2e481f00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 --2- 172.21.15.119:0/4115410037 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e480b31e0 0x7f2e480c89f0 unknown :-1 s=CLOSED pgs=1873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 >> 172.21.15.119:0/4115410037 conn(0x7f2e480b25e0 msgr2=0x7f2e48066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:20.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 shutdown_connections 2024-09-13T16:36:20.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.482+0000 7f2e4d9d1700 1 -- 172.21.15.119:0/4115410037 wait complete. 2024-09-13T16:36:20.496 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:20.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f77715ec700 1 Processor -- start 2024-09-13T16:36:20.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f77715ec700 1 -- start start 2024-09-13T16:36:20.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f77715ec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c0ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f77715ec700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c0cd600 con 0x7f776c0cff60 2024-09-13T16:36:20.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f77715ec700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c0d0dd0 con 0x7f776c0d0fb0 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f77715ec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0ce1d0 con 0x7f776c15cae0 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f776b7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f776c0cff60 0x7f776c0d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:33028/0 (socket says 172.21.15.119:33028) 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f776b7fe700 1 -- 172.21.15.119:0/1015598266 learned_addr learned my addr 172.21.15.119:0/1015598266 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f776bfff700 1 --2- 172.21.15.119:0/1015598266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c0ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.618+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602886187 0 0) 0x7f776c0cd600 con 0x7f776c0cff60 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77540035d0 con 0x7f776c0cff60 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782708048 0 0) 0x7f776c0d0dd0 con 0x7f776c0d0fb0 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f776c0cd600 con 0x7f776c0d0fb0 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776bfff700 1 -- 172.21.15.119:0/1015598266 >> v1:172.21.15.145:6789/0 conn(0x7f776c0d0fb0 legacy=0x7f776c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776bfff700 1 -- 172.21.15.119:0/1015598266 >> v1:172.21.15.119:6789/0 conn(0x7f776c0cff60 legacy=0x7f776c0d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3581649103 0 0) 0x7f77540035d0 con 0x7f776c0cff60 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776bfff700 1 -- 172.21.15.119:0/1015598266 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c0cd830 con 0x7f776c15cae0 2024-09-13T16:36:20.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776bfff700 1 --2- 172.21.15.119:0/1015598266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c0ce5f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7760009880 tx=0x7f7760009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15c7a7bef68d9234 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c001670 con 0x7f776c15cae0 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7760004030 con 0x7f776c15cae0 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f776affd700 1 -- 172.21.15.119:0/1015598266 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77600043c0 con 0x7f776c15cae0 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f77715ec700 1 -- 172.21.15.119:0/1015598266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 msgr2=0x7f776c0ce5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f77715ec700 1 --2- 172.21.15.119:0/1015598266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c0ce5f0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f7760009880 tx=0x7f7760009b90 comp rx=0 tx=0).stop 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f77715ec700 1 -- 172.21.15.119:0/1015598266 shutdown_connections 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f77715ec700 1 --2- 172.21.15.119:0/1015598266 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c0ce5f0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f77715ec700 1 -- 172.21.15.119:0/1015598266 >> 172.21.15.119:0/1015598266 conn(0x7f776c0d29f0 msgr2=0x7f776c0d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f77715ec700 1 -- 172.21.15.119:0/1015598266 shutdown_connections 2024-09-13T16:36:20.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.622+0000 7f77715ec700 1 -- 172.21.15.119:0/1015598266 wait complete. 2024-09-13T16:36:20.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 Processor -- start 2024-09-13T16:36:20.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 -- start start 2024-09-13T16:36:20.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f776c0cff60 0x7f776c159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f776c0d0fb0 0x7f776c15f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c152cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c1601b0 con 0x7f776c0d0fb0 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c153210 con 0x7f776c0cff60 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f77715ec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c15fed0 con 0x7f776c15cae0 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f776c0d0fb0 0x7f776c15f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f776c0d0fb0 0x7f776c15f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60572/0 (socket says 172.21.15.119:60572) 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 -- 172.21.15.119:0/383922555 learned_addr learned my addr 172.21.15.119:0/383922555 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776bfff700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f776c0cff60 0x7f776c159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f7770deb700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c152cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 -- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 msgr2=0x7f776c152cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c152cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 -- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f776c0cff60 msgr2=0x7f776c159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f776c0cff60 0x7f776c159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 -- 172.21.15.119:0/383922555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77600094b0 con 0x7f776c0d0fb0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.626+0000 7f776b7fe700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f776c0d0fb0 0x7f776c15f210 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f775c00c580 tx=0x7f775c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f7768ff9700 1 -- 172.21.15.119:0/383922555 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c004ca0 con 0x7f776c0d0fb0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f7768ff9700 1 -- 172.21.15.119:0/383922555 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f775c004e00 con 0x7f776c0d0fb0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f7768ff9700 1 -- 172.21.15.119:0/383922555 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c0072d0 con 0x7f776c0d0fb0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c15e730 con 0x7f776c0d0fb0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c15ecf0 con 0x7f776c0d0fb0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c15e590 con 0x7f776c0d0fb0 2024-09-13T16:36:20.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f7768ff9700 1 -- 172.21.15.119:0/383922555 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f775c007430 con 0x7f776c0d0fb0 2024-09-13T16:36:20.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f7768ff9700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7758066730 0x7f7758068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f7768ff9700 1 -- 172.21.15.119:0/383922555 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f775c083f40 con 0x7f776c0d0fb0 2024-09-13T16:36:20.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.630+0000 7f776bfff700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7758066730 0x7f7758068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.634+0000 7f776bfff700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7758066730 0x7f7758068bf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f77600090b0 tx=0x7f7760010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.642+0000 7f7768ff9700 1 -- 172.21.15.119:0/383922555 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f776c15e590 con 0x7f776c0d0fb0 2024-09-13T16:36:20.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.810+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f776c15e590 con 0x7f776c0d0fb0 2024-09-13T16:36:20.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.810+0000 7f7768ff9700 1 -- 172.21.15.119:0/383922555 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7f776c15e590 con 0x7f776c0d0fb0 2024-09-13T16:36:20.814 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:20.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7758066730 msgr2=0x7f7758068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7758066730 0x7f7758068bf0 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f77600090b0 tx=0x7f7760010000 comp rx=0 tx=0).stop 2024-09-13T16:36:20.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f776c0d0fb0 msgr2=0x7f776c15f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f776c0d0fb0 0x7f776c15f210 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7f775c00c580 tx=0x7f775c00c890 comp rx=0 tx=0).stop 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 shutdown_connections 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7758066730 0x7f7758068bf0 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f776c15cae0 0x7f776c152cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f776c0d0fb0 0x7f776c15f210 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 --2- 172.21.15.119:0/383922555 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f776c0cff60 0x7f776c159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 >> 172.21.15.119:0/383922555 conn(0x7f776c0d29f0 msgr2=0x7f776c0b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 shutdown_connections 2024-09-13T16:36:20.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.814+0000 7f77715ec700 1 -- 172.21.15.119:0/383922555 wait complete. 2024-09-13T16:36:20.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:36:20.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d1447700 1 Processor -- start 2024-09-13T16:36:20.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d1447700 1 -- start start 2024-09-13T16:36:20.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d1447700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98cc144330 0x7f98cc0cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d1447700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f98cc0b3410 0x7f98cc0cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.941 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d1447700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0ce4d0 con 0x7f98cc143f50 2024-09-13T16:36:20.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d1447700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc0b1c40 con 0x7f98cc144330 2024-09-13T16:36:20.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d1447700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc06cdf0 con 0x7f98cc0b3410 2024-09-13T16:36:20.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98cc144330 0x7f98cc0cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f98cc144330 0x7f98cc0cda50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60580/0 (socket says 172.21.15.119:60580) 2024-09-13T16:36:20.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 -- 172.21.15.119:0/3935940681 learned_addr learned my addr 172.21.15.119:0/3935940681 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:20.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98d0c46700 1 --2- 172.21.15.119:0/3935940681 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0b3410 0x7f98cc0cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 -- 172.21.15.119:0/3935940681 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0b3410 msgr2=0x7f98cc0cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:20.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 --2- 172.21.15.119:0/3935940681 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0b3410 0x7f98cc0cdf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 -- 172.21.15.119:0/3935940681 >> v1:172.21.15.145:6789/0 conn(0x7f98cc143f50 legacy=0x7f98cc06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 -- 172.21.15.119:0/3935940681 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc06d0d0 con 0x7f98cc144330 2024-09-13T16:36:20.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.938+0000 7f98cb7fe700 1 --2- 172.21.15.119:0/3935940681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc144330 0x7f98cc0cda50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0cc730 tx=0x7f98bc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=814fc734b401c2dc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98caffd700 1 -- 172.21.15.119:0/3935940681 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00e070 con 0x7f98cc144330 2024-09-13T16:36:20.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98caffd700 1 -- 172.21.15.119:0/3935940681 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc00b910 con 0x7f98cc144330 2024-09-13T16:36:20.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98caffd700 1 -- 172.21.15.119:0/3935940681 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00bcc0 con 0x7f98cc144330 2024-09-13T16:36:20.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 -- 172.21.15.119:0/3935940681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc144330 msgr2=0x7f98cc0cda50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 --2- 172.21.15.119:0/3935940681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc144330 0x7f98cc0cda50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f98cc0cc730 tx=0x7f98bc00b330 comp rx=0 tx=0).stop 2024-09-13T16:36:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 -- 172.21.15.119:0/3935940681 shutdown_connections 2024-09-13T16:36:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 --2- 172.21.15.119:0/3935940681 >> v2:172.21.15.145:3300/0 conn(0x7f98cc0b3410 0x7f98cc0cdf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 --2- 172.21.15.119:0/3935940681 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc144330 0x7f98cc0cda50 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 -- 172.21.15.119:0/3935940681 >> 172.21.15.119:0/3935940681 conn(0x7f98cc06a350 msgr2=0x7f98cc06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 -- 172.21.15.119:0/3935940681 shutdown_connections 2024-09-13T16:36:20.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 -- 172.21.15.119:0/3935940681 wait complete. 2024-09-13T16:36:20.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 Processor -- start 2024-09-13T16:36:20.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.942+0000 7f98d1447700 1 -- start start 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b3410 0x7f98cc1f1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc143f50 0x7f98cc0c3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc144330 0x7f98cc0c4300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc15aa70 con 0x7f98cc143f50 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0ab660 con 0x7f98cc0b3410 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0ab900 con 0x7f98cc144330 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc143f50 0x7f98cc0c3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc143f50 0x7f98cc0c3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60590/0 (socket says 172.21.15.119:60590) 2024-09-13T16:36:20.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 -- 172.21.15.119:0/1806753489 learned_addr learned my addr 172.21.15.119:0/1806753489 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cbfff700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b3410 0x7f98cc1f1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d0c46700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc144330 0x7f98cc0c4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 -- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc144330 msgr2=0x7f98cc0c4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc144330 0x7f98cc0c4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 -- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b3410 msgr2=0x7f98cc1f1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b3410 0x7f98cc1f1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 -- 172.21.15.119:0/1806753489 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc009b30 con 0x7f98cc143f50 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98cb7fe700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc143f50 0x7f98cc0c3dc0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f98cc06b240 tx=0x7f98bc004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98c8ff9700 1 -- 172.21.15.119:0/1806753489 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc0048c0 con 0x7f98cc143f50 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98c8ff9700 1 -- 172.21.15.119:0/1806753489 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc0099b0 con 0x7f98cc143f50 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc0abbe0 con 0x7f98cc143f50 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98c8ff9700 1 -- 172.21.15.119:0/1806753489 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc01b670 con 0x7f98cc143f50 2024-09-13T16:36:20.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc0ac1a0 con 0x7f98cc143f50 2024-09-13T16:36:20.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.946+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc0ac390 con 0x7f98cc143f50 2024-09-13T16:36:20.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.950+0000 7f98c8ff9700 1 -- 172.21.15.119:0/1806753489 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f98bc01b7d0 con 0x7f98cc143f50 2024-09-13T16:36:20.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.950+0000 7f98c8ff9700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98b8066760 0x7f98b8068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:20.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.950+0000 7f98c8ff9700 1 -- 172.21.15.119:0/1806753489 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7f98bc0857f0 con 0x7f98cc143f50 2024-09-13T16:36:20.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.950+0000 7f98cbfff700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98b8066760 0x7f98b8068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:20.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.950+0000 7f98cbfff700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98b8066760 0x7f98b8068c20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f98c0000f60 tx=0x7f98c0009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:20.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:20.958+0000 7f98c8ff9700 1 -- 172.21.15.119:0/1806753489 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f98bc052a10 con 0x7f98cc143f50 2024-09-13T16:36:21.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.138+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f98cc06c5a0 con 0x7f98cc143f50 2024-09-13T16:36:21.141 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.138+0000 7f98c8ff9700 1 -- 172.21.15.119:0/1806753489 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v126) v1 ==== 80+0+0 (secure 0 0 0) 0x7f98bc056030 con 0x7f98cc143f50 2024-09-13T16:36:21.144 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98b8066760 msgr2=0x7f98b8068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98b8066760 0x7f98b8068c20 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f98c0000f60 tx=0x7f98c0009040 comp rx=0 tx=0).stop 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc143f50 msgr2=0x7f98cc0c3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc143f50 0x7f98cc0c3dc0 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f98cc06b240 tx=0x7f98bc004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 shutdown_connections 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f98b8066760 0x7f98b8068c20 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f98cc144330 0x7f98cc0c4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f98cc143f50 0x7f98cc0c3dc0 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 --2- 172.21.15.119:0/1806753489 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f98cc0b3410 0x7f98cc1f1340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 >> 172.21.15.119:0/1806753489 conn(0x7f98cc06a350 msgr2=0x7f98cc0b2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 shutdown_connections 2024-09-13T16:36:21.145 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.142+0000 7f98d1447700 1 -- 172.21.15.119:0/1806753489 wait complete. 2024-09-13T16:36:21.146 INFO:teuthology.orchestra.run.smithi119.stderr:listed 0 entries 2024-09-13T16:36:21.156 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:36:21.157 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:21.157 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:21.169 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[54130]: fuse finished with error 0 and tester_r 0 2024-09-13T16:36:21.171 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:36:21.222 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:36:21.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:21.222 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:36:21.226 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:21.226 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:21.231 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:36:21.231 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@smithi119.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:21.231 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:36:21.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:21.231 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:36:21.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:21.235 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-13T16:36:21.236 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:36:21.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 Processor -- start 2024-09-13T16:36:21.399 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- start start 2024-09-13T16:36:21.400 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7efc08033c50 0x7efc08005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc0800ab40 0x7efc0800af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc080df440 con 0x7efc0804fc90 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc08006050 con 0x7efc0800ab40 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc080e39f0 con 0x7efc08033c50 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc0800ab40 0x7efc0800af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efc0800ab40 0x7efc0800af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60594/0 (socket says 172.21.15.119:60594) 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 -- 172.21.15.119:0/1568208907 learned_addr learned my addr 172.21.15.119:0/1568208907 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc151ea700 1 --2- 172.21.15.119:0/1568208907 >> v2:172.21.15.145:3300/0 conn(0x7efc08033c50 0x7efc08005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 -- 172.21.15.119:0/1568208907 >> v2:172.21.15.145:3300/0 conn(0x7efc08033c50 msgr2=0x7efc08005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 --2- 172.21.15.119:0/1568208907 >> v2:172.21.15.145:3300/0 conn(0x7efc08033c50 0x7efc08005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 -- 172.21.15.119:0/1568208907 >> v1:172.21.15.145:6789/0 conn(0x7efc0804fc90 legacy=0x7efc080e32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.401 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 -- 172.21.15.119:0/1568208907 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080062f0 con 0x7efc0800ab40 2024-09-13T16:36:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc159eb700 1 --2- 172.21.15.119:0/1568208907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc0800af50 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7efc100698b0 tx=0x7efc1004f710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4e7b9641ee2d21 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:21.402 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc07fff700 1 -- 172.21.15.119:0/1568208907 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10067480 con 0x7efc0800ab40 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc07fff700 1 -- 172.21.15.119:0/1568208907 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc100675e0 con 0x7efc0800ab40 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc07fff700 1 -- 172.21.15.119:0/1568208907 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc10067b90 con 0x7efc0800ab40 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- 172.21.15.119:0/1568208907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 msgr2=0x7efc0800af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 --2- 172.21.15.119:0/1568208907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc0800af50 secure :-1 s=READY pgs=1878 cs=0 l=1 rev1=1 crypto rx=0x7efc100698b0 tx=0x7efc1004f710 comp rx=0 tx=0).stop 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- 172.21.15.119:0/1568208907 shutdown_connections 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 --2- 172.21.15.119:0/1568208907 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc0800af50 unknown :-1 s=CLOSED pgs=1878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 --2- 172.21.15.119:0/1568208907 >> v2:172.21.15.145:3300/0 conn(0x7efc08033c50 0x7efc08005b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- 172.21.15.119:0/1568208907 >> 172.21.15.119:0/1568208907 conn(0x7efc08036140 msgr2=0x7efc08036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- 172.21.15.119:0/1568208907 shutdown_connections 2024-09-13T16:36:21.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.398+0000 7efc161ec700 1 -- 172.21.15.119:0/1568208907 wait complete. 2024-09-13T16:36:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 Processor -- start 2024-09-13T16:36:21.404 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 -- start start 2024-09-13T16:36:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc080d4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc08033c50 0x7efc08038830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc0804fc90 0x7efc08031b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08039560 con 0x7efc0800ab40 2024-09-13T16:36:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc08039280 con 0x7efc08033c50 2024-09-13T16:36:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc08179a80 con 0x7efc0804fc90 2024-09-13T16:36:21.405 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc080d4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc080d4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60608/0 (socket says 172.21.15.119:60608) 2024-09-13T16:36:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 -- 172.21.15.119:0/2000692950 learned_addr learned my addr 172.21.15.119:0/2000692950 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc149e9700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc08033c50 0x7efc08038830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc159eb700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc0804fc90 0x7efc08031b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 -- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc0804fc90 msgr2=0x7efc08031b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc0804fc90 0x7efc08031b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 -- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc08033c50 msgr2=0x7efc08038830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc08033c50 0x7efc08038830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 -- 172.21.15.119:0/2000692950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc08179d60 con 0x7efc0800ab40 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc151ea700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc080d4830 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00e9b0 tx=0x7efc0c00ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc05ffb700 1 -- 172.21.15.119:0/2000692950 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c009710 con 0x7efc0800ab40 2024-09-13T16:36:21.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc05ffb700 1 -- 172.21.15.119:0/2000692950 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc0c00ee50 con 0x7efc0800ab40 2024-09-13T16:36:21.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc0817a040 con 0x7efc0800ab40 2024-09-13T16:36:21.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc05ffb700 1 -- 172.21.15.119:0/2000692950 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc0c00ca50 con 0x7efc0800ab40 2024-09-13T16:36:21.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.402+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc08179560 con 0x7efc0800ab40 2024-09-13T16:36:21.408 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.406+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc081798e0 con 0x7efc0800ab40 2024-09-13T16:36:21.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.406+0000 7efc05ffb700 1 -- 172.21.15.119:0/2000692950 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efc0c00cbb0 con 0x7efc0800ab40 2024-09-13T16:36:21.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.406+0000 7efc05ffb700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efbf4066730 0x7efbf4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.406+0000 7efc05ffb700 1 -- 172.21.15.119:0/2000692950 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6029+0+0 (secure 0 0 0) 0x7efc0c084720 con 0x7efc0800ab40 2024-09-13T16:36:21.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.406+0000 7efc149e9700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efbf4066730 0x7efbf4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.411 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.410+0000 7efc149e9700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efbf4066730 0x7efbf4068bf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7efc08052c10 tx=0x7efc00009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:21.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.418+0000 7efc05ffb700 1 -- 172.21.15.119:0/2000692950 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efc081798e0 con 0x7efc0800ab40 2024-09-13T16:36:21.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.626+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7efc081798e0 con 0x7efc0800ab40 2024-09-13T16:36:21.632 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:36:21.630+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:36:21.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.642+0000 7efc05ffb700 1 -- 172.21.15.119:0/2000692950 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v93) v1 ==== 161+0+0 (secure 0 0 0) 0x7efc081798e0 con 0x7efc0800ab40 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efbf4066730 msgr2=0x7efbf4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efbf4066730 0x7efbf4068bf0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7efc08052c10 tx=0x7efc00009040 comp rx=0 tx=0).stop 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 msgr2=0x7efc080d4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc080d4830 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00e9b0 tx=0x7efc0c00ecc0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 shutdown_connections 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efbf4066730 0x7efbf4068bf0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efc0804fc90 0x7efc08031b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efc08033c50 0x7efc08038830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 --2- 172.21.15.119:0/2000692950 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efc0800ab40 0x7efc080d4830 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 >> 172.21.15.119:0/2000692950 conn(0x7efc08036140 msgr2=0x7efc08036520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:21.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 shutdown_connections 2024-09-13T16:36:21.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.646+0000 7efc161ec700 1 -- 172.21.15.119:0/2000692950 wait complete. 2024-09-13T16:36:21.649 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:36:21.651 INFO:tasks.ceph.mds.b.smithi145.stderr: -7458> 2024-09-13T16:36:01.202+0000 7f55623e3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:36:21.663 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:21.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:21.688 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.694 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-09-13T16:36:21.695 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:21.696 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:21.697 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:21.698 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-09-13T16:36:21.699 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-09-13T16:36:21.700 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-09-13T16:36:21.701 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.702 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-09-13T16:36:21.703 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:21.704 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: mon op complaint time = 120 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on pool no app = False 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:21.705 INFO:teuthology.orchestra.run.smithi145.stdout: client mount timeout = 600 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = "" 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-09-13T16:36:21.706 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout:[mds] 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds = 20 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds balancer = 20 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug frag = True 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug scatterstat = True 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout: mds op complaint time = 180 2024-09-13T16:36:21.707 INFO:teuthology.orchestra.run.smithi145.stdout: mds verify scatter = False 2024-09-13T16:36:21.708 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:21.708 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.708 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:21.710 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.711 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-09-13T16:36:21.712 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-09-13T16:36:21.713 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:21.714 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.715 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-09-13T16:36:21.716 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.717 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:21.718 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.719 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: mon op complaint time = 120 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on pool no app = False 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: client mount timeout = 600 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:21.720 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = "" 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds = 20 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds balancer = 20 2024-09-13T16:36:21.721 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.722 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug frag = True 2024-09-13T16:36:21.722 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug scatterstat = True 2024-09-13T16:36:21.722 INFO:teuthology.orchestra.run.smithi119.stdout: mds op complaint time = 180 2024-09-13T16:36:21.722 INFO:teuthology.orchestra.run.smithi119.stdout: mds verify scatter = False 2024-09-13T16:36:21.722 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:21.722 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.722 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.723 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:21.752 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:21.786 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.787 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-09-13T16:36:21.788 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.789 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-09-13T16:36:21.790 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.791 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2024-09-13T16:36:21.792 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-09-13T16:36:21.793 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:21.794 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.795 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: mon op complaint time = 120 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on pool no app = False 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-09-13T16:36:21.796 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: client mount timeout = 600 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = "" 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-09-13T16:36:21.797 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout:[mds] 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds = 20 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds balancer = 20 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug frag = True 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug scatterstat = False 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: mds op complaint time = 180 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: mds verify scatter = False 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:21.798 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.799 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.800 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-13T16:36:21.800 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-09-13T16:36:21.800 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:21.800 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-09-13T16:36:21.800 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:21.801 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:21.802 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:21.803 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-09-13T16:36:21.804 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:21.805 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:21.806 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:21.807 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.808 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon op complaint time = 120 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on pool no app = False 2024-09-13T16:36:21.809 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: client mount timeout = 600 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = "" 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.810 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds = 20 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds balancer = 20 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug frag = True 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug scatterstat = False 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout: mds op complaint time = 180 2024-09-13T16:36:21.811 INFO:teuthology.orchestra.run.smithi119.stdout: mds verify scatter = False 2024-09-13T16:36:21.812 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:21.812 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:21.812 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:21.812 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_metadata rmomapkey 10000000001.00000000 bravo_head 2024-09-13T16:36:21.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 Processor -- start 2024-09-13T16:36:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 -- start start 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1eee300 0x5587e1fbffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1fc4100 0x5587e1f1be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc44e0 0x5587e1f1c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e1e84a00 con 0x5587e1fc4100 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e1e802f0 con 0x5587e1fc44e0 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f856003e140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587e1e3c8e0 con 0x5587e1eee300 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f855d372700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1fc4100 0x5587e1f1be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f855d372700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1fc4100 0x5587e1f1be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60612/0 (socket says 172.21.15.119:60612) 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f855d372700 1 -- 172.21.15.119:0/1383852557 learned_addr learned my addr 172.21.15.119:0/1383852557 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f855e374700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc44e0 0x5587e1f1c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.854+0000 7f855db73700 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1eee300 0x5587e1fbffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f855d372700 1 -- 172.21.15.119:0/1383852557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1eee300 msgr2=0x5587e1fbffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f855d372700 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1eee300 0x5587e1fbffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f855d372700 1 -- 172.21.15.119:0/1383852557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc44e0 msgr2=0x5587e1f1c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f855d372700 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc44e0 0x5587e1f1c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f855d372700 1 -- 172.21.15.119:0/1383852557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8558005d20 con 0x5587e1fc4100 2024-09-13T16:36:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f855d372700 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1fc4100 0x5587e1f1be40 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f8548001ce0 tx=0x7f854800b4c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab113e6f45f195bf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f8557fff700 1 -- 172.21.15.119:0/1383852557 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8548014070 con 0x5587e1fc4100 2024-09-13T16:36:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f8557fff700 1 -- 172.21.15.119:0/1383852557 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8548008030 con 0x5587e1fc4100 2024-09-13T16:36:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f8557fff700 1 -- 172.21.15.119:0/1383852557 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854800bce0 con 0x5587e1fc4100 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 -- 172.21.15.119:0/1383852557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1fc4100 msgr2=0x5587e1f1be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1fc4100 0x5587e1f1be40 secure :-1 s=READY pgs=1881 cs=0 l=1 rev1=1 crypto rx=0x7f8548001ce0 tx=0x7f854800b4c0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 -- 172.21.15.119:0/1383852557 shutdown_connections 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc44e0 0x5587e1f1c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1fc4100 0x5587e1f1be40 unknown :-1 s=CLOSED pgs=1881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 --2- 172.21.15.119:0/1383852557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1eee300 0x5587e1fbffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 -- 172.21.15.119:0/1383852557 >> 172.21.15.119:0/1383852557 conn(0x5587e1eef110 msgr2=0x5587e1fc0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 -- 172.21.15.119:0/1383852557 shutdown_connections 2024-09-13T16:36:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 -- 172.21.15.119:0/1383852557 wait complete. 2024-09-13T16:36:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.858+0000 7f856003e140 1 Processor -- start 2024-09-13T16:36:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 -- start start 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1eee300 0x5587e1f33400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc4100 0x5587e204d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1fc44e0 0x5587e204d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e1e3c580 con 0x5587e1eee300 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587e1e128d0 con 0x5587e1fc4100 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x5587e1fbf8e0 con 0x5587e1fc44e0 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1eee300 0x5587e1f33400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1eee300 0x5587e1f33400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60616/0 (socket says 172.21.15.119:60616) 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 -- 172.21.15.119:0/2834496141 learned_addr learned my addr 172.21.15.119:0/2834496141 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855e374700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1fc44e0 0x5587e204d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855d372700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc4100 0x5587e204d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 -- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1fc44e0 msgr2=0x5587e204d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5587e1fc44e0 0x5587e204d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 -- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc4100 msgr2=0x5587e204d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5587e1fc4100 0x5587e204d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 -- 172.21.15.119:0/2834496141 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854800a040 con 0x5587e1eee300 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f855db73700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1eee300 0x5587e1f33400 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x5587e1f1ef50 tx=0x7f8558002fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f8556ffd700 1 -- 172.21.15.119:0/2834496141 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85580054c0 con 0x5587e1eee300 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f8556ffd700 1 -- 172.21.15.119:0/2834496141 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8558012820 con 0x5587e1eee300 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x5587e1e1df40 con 0x5587e1eee300 2024-09-13T16:36:21.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f8556ffd700 1 -- 172.21.15.119:0/2834496141 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8558012bd0 con 0x5587e1eee300 2024-09-13T16:36:21.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.862+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587e1e1d9a0 con 0x5587e1eee300 2024-09-13T16:36:21.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f8556ffd700 1 -- 172.21.15.119:0/2834496141 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8558012d30 con 0x5587e1eee300 2024-09-13T16:36:21.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f8556ffd700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8540065090 0x7f8540067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f8556ffd700 1 -- 172.21.15.119:0/2834496141 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f8558084ae0 con 0x5587e1eee300 2024-09-13T16:36:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f855d372700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8540065090 0x7f8540067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f856003e140 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5587e1f38150 0x5587e2063ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:1 17.7 17:e2e07930:::10000000001.00000000:head [omap-rm-keys in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e127) v8 -- 0x5587e20641e0 con 0x5587e1f38150 2024-09-13T16:36:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f855e374700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5587e1f38150 0x5587e2063ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f855d372700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8540065090 0x7f8540067550 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8548000fd0 tx=0x7f8548009000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:21.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.866+0000 7f855e374700 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5587e1f38150 0x5587e2063ca0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f855e374700 1 -- 172.21.15.119:0/2834496141 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(1 10000000001.00000000 [omap-rm-keys] v127'5 uv5 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x7f8550001650 con 0x5587e1f38150 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5587e1f38150 msgr2=0x5587e2063ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x5587e1f38150 0x5587e2063ca0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8540065090 msgr2=0x7f8540067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8540065090 0x7f8540067550 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f8548000fd0 tx=0x7f8548009000 comp rx=0 tx=0).stop 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1eee300 msgr2=0x5587e1f33400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 --2- 172.21.15.119:0/2834496141 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5587e1eee300 0x5587e1f33400 secure :-1 s=READY pgs=1882 cs=0 l=1 rev1=1 crypto rx=0x5587e1f1ef50 tx=0x7f8558002fb0 comp rx=0 tx=0).stop 2024-09-13T16:36:21.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f855e374700 1 -- 172.21.15.119:0/2834496141 reap_dead start 2024-09-13T16:36:21.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 shutdown_connections 2024-09-13T16:36:21.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 >> 172.21.15.119:0/2834496141 conn(0x5587e1eef110 msgr2=0x5587e1f34bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:21.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 shutdown_connections 2024-09-13T16:36:21.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:21.870+0000 7f856003e140 1 -- 172.21.15.119:0/2834496141 wait complete. 2024-09-13T16:36:21.879 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-13T16:36:22.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e9939700 1 Processor -- start 2024-09-13T16:36:22.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e9939700 1 -- start start 2024-09-13T16:36:22.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e9939700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f59e4071c10 0x7f59e406db40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e9939700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f59e4168a10 0x7f59e416ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e9939700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e4162270 con 0x7f59e40b1ad0 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e9939700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e406e080 con 0x7f59e4071c10 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e9939700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e406ce40 con 0x7f59e4168a10 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f59e4071c10 0x7f59e406db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f59e4071c10 0x7f59e406db40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60618/0 (socket says 172.21.15.119:60618) 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 -- 172.21.15.119:0/743426468 learned_addr learned my addr 172.21.15.119:0/743426468 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e37fe700 1 --2- 172.21.15.119:0/743426468 >> v2:172.21.15.145:3300/0 conn(0x7f59e4168a10 0x7f59e416ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 -- 172.21.15.119:0/743426468 >> v2:172.21.15.145:3300/0 conn(0x7f59e4168a10 msgr2=0x7f59e416ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 --2- 172.21.15.119:0/743426468 >> v2:172.21.15.145:3300/0 conn(0x7f59e4168a10 0x7f59e416ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 -- 172.21.15.119:0/743426468 >> v1:172.21.15.145:6789/0 conn(0x7f59e40b1ad0 legacy=0x7f59e40b1eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 -- 172.21.15.119:0/743426468 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e40702f0 con 0x7f59e4071c10 2024-09-13T16:36:22.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e27fc700 1 --2- 172.21.15.119:0/743426468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e4071c10 0x7f59e406db40 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f59e40b2c80 tx=0x7f59d8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd44ceaf6bd3e7a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e1ffb700 1 -- 172.21.15.119:0/743426468 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d8011830 con 0x7f59e4071c10 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e1ffb700 1 -- 172.21.15.119:0/743426468 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d8011990 con 0x7f59e4071c10 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.046+0000 7f59e1ffb700 1 -- 172.21.15.119:0/743426468 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d8011d20 con 0x7f59e4071c10 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 -- 172.21.15.119:0/743426468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e4071c10 msgr2=0x7f59e406db40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 --2- 172.21.15.119:0/743426468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e4071c10 0x7f59e406db40 secure :-1 s=READY pgs=1883 cs=0 l=1 rev1=1 crypto rx=0x7f59e40b2c80 tx=0x7f59d8009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 -- 172.21.15.119:0/743426468 shutdown_connections 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 --2- 172.21.15.119:0/743426468 >> v2:172.21.15.145:3300/0 conn(0x7f59e4168a10 0x7f59e416ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 --2- 172.21.15.119:0/743426468 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e4071c10 0x7f59e406db40 unknown :-1 s=CLOSED pgs=1883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 -- 172.21.15.119:0/743426468 >> 172.21.15.119:0/743426468 conn(0x7f59e40b0560 msgr2=0x7f59e40b0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 -- 172.21.15.119:0/743426468 shutdown_connections 2024-09-13T16:36:22.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 -- 172.21.15.119:0/743426468 wait complete. 2024-09-13T16:36:22.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 Processor -- start 2024-09-13T16:36:22.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.050+0000 7f59e9939700 1 -- start start 2024-09-13T16:36:22.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59e4071c10 0x7f59e4161b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e40b1ad0 0x7f59e40a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e4168a10 0x7f59e41592e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e40a89f0 con 0x7f59e40b1ad0 2024-09-13T16:36:22.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4167c40 con 0x7f59e4168a10 2024-09-13T16:36:22.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59e4166170 con 0x7f59e4071c10 2024-09-13T16:36:22.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e40b1ad0 0x7f59e40a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e40b1ad0 0x7f59e40a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:60626/0 (socket says 172.21.15.119:60626) 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 -- 172.21.15.119:0/2679574377 learned_addr learned my addr 172.21.15.119:0/2679574377 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e37fe700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e4168a10 0x7f59e41592e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e2ffd700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59e4071c10 0x7f59e4161b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 -- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59e4071c10 msgr2=0x7f59e4161b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59e4071c10 0x7f59e4161b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 -- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e4168a10 msgr2=0x7f59e41592e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e4168a10 0x7f59e41592e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 -- 172.21.15.119:0/2679574377 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d8009570 con 0x7f59e40b1ad0 2024-09-13T16:36:22.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e27fc700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e40b1ad0 0x7f59e40a84b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f59d8000c00 tx=0x7f59d8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59cbfff700 1 -- 172.21.15.119:0/2679574377 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d8004510 con 0x7f59e40b1ad0 2024-09-13T16:36:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59cbfff700 1 -- 172.21.15.119:0/2679574377 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d801e070 con 0x7f59e40b1ad0 2024-09-13T16:36:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e4167680 con 0x7f59e40b1ad0 2024-09-13T16:36:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59cbfff700 1 -- 172.21.15.119:0/2679574377 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d80038c0 con 0x7f59e40b1ad0 2024-09-13T16:36:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e4166410 con 0x7f59e40b1ad0 2024-09-13T16:36:22.058 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.054+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e406fcb0 con 0x7f59e40b1ad0 2024-09-13T16:36:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.058+0000 7f59cbfff700 1 -- 172.21.15.119:0/2679574377 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f59d8003a20 con 0x7f59e40b1ad0 2024-09-13T16:36:22.059 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.058+0000 7f59cbfff700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f59d0066730 0x7f59d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.058+0000 7f59cbfff700 1 -- 172.21.15.119:0/2679574377 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f59d8020070 con 0x7f59e40b1ad0 2024-09-13T16:36:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.058+0000 7f59e2ffd700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f59d0066730 0x7f59d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.060 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.058+0000 7f59e2ffd700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f59d0066730 0x7f59d0068bf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f59e40ce9d0 tx=0x7f59d4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:22.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.062+0000 7f59cbfff700 1 -- 172.21.15.119:0/2679574377 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f59d8052a30 con 0x7f59e40b1ad0 2024-09-13T16:36:22.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.270+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7f59e406d8e0 con 0x7f59e40b1ad0 2024-09-13T16:36:22.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.654+0000 7f59cbfff700 1 -- 172.21.15.119:0/2679574377 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v94) v1 ==== 162+0+0 (secure 0 0 0) 0x7f59d8056050 con 0x7f59e40b1ad0 2024-09-13T16:36:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f59d0066730 msgr2=0x7f59d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f59d0066730 0x7f59d0068bf0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f59e40ce9d0 tx=0x7f59d4009040 comp rx=0 tx=0).stop 2024-09-13T16:36:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e40b1ad0 msgr2=0x7f59e40a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e40b1ad0 0x7f59e40a84b0 secure :-1 s=READY pgs=1884 cs=0 l=1 rev1=1 crypto rx=0x7f59d8000c00 tx=0x7f59d8004770 comp rx=0 tx=0).stop 2024-09-13T16:36:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 shutdown_connections 2024-09-13T16:36:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f59d0066730 0x7f59d0068bf0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f59e4168a10 0x7f59e41592e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59e40b1ad0 0x7f59e40a84b0 unknown :-1 s=CLOSED pgs=1884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 --2- 172.21.15.119:0/2679574377 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59e4071c10 0x7f59e4161b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 >> 172.21.15.119:0/2679574377 conn(0x7f59e40b0560 msgr2=0x7f59e415aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:22.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 shutdown_connections 2024-09-13T16:36:22.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.658+0000 7f59e9939700 1 -- 172.21.15.119:0/2679574377 wait complete. 2024-09-13T16:36:22.664 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-13T16:36:22.680 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:22.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc907fff700 1 Processor -- start 2024-09-13T16:36:22.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc907fff700 1 -- start start 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc907fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908071740 0x7fc90806ce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc907fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc908158f10 0x7fc90806d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc907fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc90806de00 con 0x7fc908158b30 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc907fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90806d8b0 con 0x7fc908158f10 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc907fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90806db20 con 0x7fc908071740 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc908158f10 0x7fc90806d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc908158f10 0x7fc90806d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36014/0 (socket says 172.21.15.119:36014) 2024-09-13T16:36:22.803 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 -- 172.21.15.119:0/3591256397 learned_addr learned my addr 172.21.15.119:0/3591256397 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9067fc700 1 --2- 172.21.15.119:0/3591256397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908071740 0x7fc90806ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 -- 172.21.15.119:0/3591256397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908071740 msgr2=0x7fc90806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 --2- 172.21.15.119:0/3591256397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908071740 0x7fc90806ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 -- 172.21.15.119:0/3591256397 >> v1:172.21.15.145:6789/0 conn(0x7fc908158b30 legacy=0x7fc908071030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 -- 172.21.15.119:0/3591256397 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9080b1920 con 0x7fc908158f10 2024-09-13T16:36:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9067fc700 1 --2- 172.21.15.119:0/3591256397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908071740 0x7fc90806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:22.804 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.794+0000 7fc9077fe700 1 --2- 172.21.15.119:0/3591256397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158f10 0x7fc90806d370 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009910 tx=0x7fc8f8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8d6559f1bc8ee57 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:22.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc905ffb700 1 -- 172.21.15.119:0/3591256397 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f800ba60 con 0x7fc908158f10 2024-09-13T16:36:22.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc905ffb700 1 -- 172.21.15.119:0/3591256397 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f800bbc0 con 0x7fc908158f10 2024-09-13T16:36:22.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc905ffb700 1 -- 172.21.15.119:0/3591256397 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f8009e60 con 0x7fc908158f10 2024-09-13T16:36:22.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 -- 172.21.15.119:0/3591256397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158f10 msgr2=0x7fc90806d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 --2- 172.21.15.119:0/3591256397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158f10 0x7fc90806d370 secure :-1 s=READY pgs=1885 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009910 tx=0x7fc8f8009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:22.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 -- 172.21.15.119:0/3591256397 shutdown_connections 2024-09-13T16:36:22.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 --2- 172.21.15.119:0/3591256397 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158f10 0x7fc90806d370 unknown :-1 s=CLOSED pgs=1885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 --2- 172.21.15.119:0/3591256397 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908071740 0x7fc90806ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 -- 172.21.15.119:0/3591256397 >> 172.21.15.119:0/3591256397 conn(0x7fc90806a350 msgr2=0x7fc90806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:22.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 -- 172.21.15.119:0/3591256397 shutdown_connections 2024-09-13T16:36:22.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 -- 172.21.15.119:0/3591256397 wait complete. 2024-09-13T16:36:22.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 Processor -- start 2024-09-13T16:36:22.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.798+0000 7fc907fff700 1 -- start start 2024-09-13T16:36:22.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc908071740 0x7fc9081482c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158b30 0x7fc908148800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908158f10 0x7fc90814d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90815ad40 con 0x7fc908158b30 2024-09-13T16:36:22.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc908149910 con 0x7fc908071740 2024-09-13T16:36:22.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90815a730 con 0x7fc908158f10 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158b30 0x7fc908148800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158b30 0x7fc908148800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36016/0 (socket says 172.21.15.119:36016) 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 -- 172.21.15.119:0/1922677731 learned_addr learned my addr 172.21.15.119:0/1922677731 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc906ffd700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc908071740 0x7fc9081482c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9077fe700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908158f10 0x7fc90814d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 -- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908158f10 msgr2=0x7fc90814d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908158f10 0x7fc90814d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 -- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc908071740 msgr2=0x7fc9081482c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc908071740 0x7fc9081482c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 -- 172.21.15.119:0/1922677731 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8009570 con 0x7fc908158b30 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc906ffd700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc908071740 0x7fc9081482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:22.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc9067fc700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158b30 0x7fc908148800 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00e880 tx=0x7fc8fc00ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc8effff700 1 -- 172.21.15.119:0/1922677731 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc00c580 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc8effff700 1 -- 172.21.15.119:0/1922677731 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc00c6e0 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90815bb50 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc8effff700 1 -- 172.21.15.119:0/1922677731 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc00ca50 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90814f110 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.802+0000 7fc8effff700 1 -- 172.21.15.119:0/1922677731 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc8fc00cbb0 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.806+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc90806c120 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.806+0000 7fc8effff700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc8f4066730 0x7fc8f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.806+0000 7fc8effff700 1 -- 172.21.15.119:0/1922677731 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fc8fc083fa0 con 0x7fc908158b30 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.806+0000 7fc906ffd700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc8f4066730 0x7fc8f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:22.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.806+0000 7fc906ffd700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc8f4066730 0x7fc8f4068bf0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc908148f70 tx=0x7fc900009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:22.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:22.814+0000 7fc8effff700 1 -- 172.21.15.119:0/1922677731 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc8fc0b5db0 con 0x7fc908158b30 2024-09-13T16:36:23.026 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.022+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9080ce520 con 0x7fc908158b30 2024-09-13T16:36:23.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc8effff700 1 -- 172.21.15.119:0/1922677731 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 95 v95) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fc8fc0510f0 con 0x7fc908158b30 2024-09-13T16:36:23.028 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:23.028 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":95,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":95,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:22.656411+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8264":{"gid":8264,"name":"b","rank":0,"incarnation":95,"state":"up:replay","state_seq":1,"addr":"172.21.15.145:6835/3898509964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3898509964},{"type":"v1","addr":"172.21.15.145:6835","nonce":3898509964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:23.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc8f4066730 msgr2=0x7fc8f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:23.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc8f4066730 0x7fc8f4068bf0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fc908148f70 tx=0x7fc900009040 comp rx=0 tx=0).stop 2024-09-13T16:36:23.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158b30 msgr2=0x7fc908148800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:23.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158b30 0x7fc908148800 secure :-1 s=READY pgs=1886 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00e880 tx=0x7fc8fc00ec40 comp rx=0 tx=0).stop 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 shutdown_connections 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc8f4066730 0x7fc8f4068bf0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc908158f10 0x7fc90814d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc908158b30 0x7fc908148800 unknown :-1 s=CLOSED pgs=1886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 --2- 172.21.15.119:0/1922677731 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc908071740 0x7fc9081482c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 >> 172.21.15.119:0/1922677731 conn(0x7fc90806a350 msgr2=0x7fc908071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 shutdown_connections 2024-09-13T16:36:23.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:23.026+0000 7fc907fff700 1 -- 172.21.15.119:0/1922677731 wait complete. 2024-09-13T16:36:23.032 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 95 2024-09-13T16:36:23.445 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 95, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:35:55.244835+0000', 'modified': '2024-09-13T16:36:22.656411+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 127, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8264': {'gid': 8264, 'name': 'b', 'rank': 0, 'incarnation': 95, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.145:6835/3898509964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3898509964}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 3898509964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:36:23.446 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8264:up:replay 2024-09-13T16:36:24.447 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:24.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.566+0000 7f24f3fff700 1 Processor -- start 2024-09-13T16:36:24.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f3fff700 1 -- start start 2024-09-13T16:36:24.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 0x7f24f406cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f3fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f24f40b2200 0x7f24f40b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f3fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f40705d0 con 0x7f24f40b17f0 2024-09-13T16:36:24.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f3fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f40cbf10 con 0x7f24f40b2200 2024-09-13T16:36:24.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40b2bd0 con 0x7f24f40cd8d0 2024-09-13T16:36:24.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f27fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f24f40b17f0 0x7f24f40cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:55810/0 (socket says 172.21.15.119:55810) 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f27fc700 1 -- 172.21.15.119:0/980655981 learned_addr learned my addr 172.21.15.119:0/980655981 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f37fe700 1 --2- 172.21.15.119:0/980655981 >> v2:172.21.15.145:3300/0 conn(0x7f24f40b2200 0x7f24f40b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 0x7f24f406cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 36725831 0 0) 0x7f24f40705d0 con 0x7f24f40b17f0 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24dc0035d0 con 0x7f24f40b17f0 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 493127704 0 0) 0x7f24dc0035d0 con 0x7f24f40b17f0 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 msgr2=0x7f24f406cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 0x7f24f406cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 >> v2:172.21.15.145:3300/0 conn(0x7f24f40b2200 msgr2=0x7f24f40b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 --2- 172.21.15.119:0/980655981 >> v2:172.21.15.145:3300/0 conn(0x7f24f40b2200 0x7f24f40b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f406d0d0 con 0x7f24f40b17f0 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f24e40019f0 con 0x7f24f40b17f0 2024-09-13T16:36:24.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 >> v1:172.21.15.119:6789/0 conn(0x7f24f40b17f0 legacy=0x7f24f40cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24dc003a50 0x7f24dc003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc004870 0x7f24dc004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 0x7f24dc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f40b2bd0 con 0x7f24dc003a50 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc00ba30 con 0x7f24dc004870 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc00bba0 con 0x7f24f40cd8d0 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24dc003a50 0x7f24dc003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f27fc700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc004870 0x7f24dc004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f37fe700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 0x7f24dc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 -- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 msgr2=0x7f24dc005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f40cd8d0 0x7f24dc005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 -- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc004870 msgr2=0x7f24dc004cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc004870 0x7f24dc004cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 -- 172.21.15.119:0/980655981 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc0035d0 con 0x7f24dc003a50 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24dc003a50 0x7f24dc003e70 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f24e8009b70 tx=0x7f24dc0058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=febce3334ae0ded2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e800e660 con 0x7f24dc003a50 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24e800e7c0 con 0x7f24dc003a50 2024-09-13T16:36:24.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.570+0000 7f24f1ffb700 1 -- 172.21.15.119:0/980655981 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24e800eb30 con 0x7f24dc003a50 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- 172.21.15.119:0/980655981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24dc003a50 msgr2=0x7f24dc003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 --2- 172.21.15.119:0/980655981 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24dc003a50 0x7f24dc003e70 secure :-1 s=READY pgs=1888 cs=0 l=1 rev1=1 crypto rx=0x7f24e8009b70 tx=0x7f24dc0058a0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f37fe700 1 -- 172.21.15.119:0/980655981 reap_dead start 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- 172.21.15.119:0/980655981 shutdown_connections 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- 172.21.15.119:0/980655981 >> 172.21.15.119:0/980655981 conn(0x7f24f406a350 msgr2=0x7f24f406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- 172.21.15.119:0/980655981 shutdown_connections 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- 172.21.15.119:0/980655981 wait complete. 2024-09-13T16:36:24.580 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 Processor -- start 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- start start 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc003a50 0x7f24f4155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f41567c0 0x7f24f4156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f41589c0 0x7f24f4158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4157180 con 0x7f24f41589c0 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24f4157410 con 0x7f24dc003a50 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f3fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f41f6de0 con 0x7f24f41567c0 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f41589c0 0x7f24f4158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f37fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f41589c0 0x7f24f4158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36044/0 (socket says 172.21.15.119:36044) 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f37fe700 1 -- 172.21.15.119:0/4096361304 learned_addr learned my addr 172.21.15.119:0/4096361304 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f27fc700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f41567c0 0x7f24f4156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.581 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.574+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc003a50 0x7f24f4155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f37fe700 1 -- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f41567c0 msgr2=0x7f24f4156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f37fe700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f41567c0 0x7f24f4156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f37fe700 1 -- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc003a50 msgr2=0x7f24f4155890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f37fe700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc003a50 0x7f24f4155890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f37fe700 1 -- 172.21.15.119:0/4096361304 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f41f70c0 con 0x7f24f41589c0 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f27fc700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f41567c0 0x7f24f4156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f37fe700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f41589c0 0x7f24f4158e70 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f24f4071850 tx=0x7f24ec011bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24db7fe700 1 -- 172.21.15.119:0/4096361304 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec004240 con 0x7f24f41589c0 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24db7fe700 1 -- 172.21.15.119:0/4096361304 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ec0043a0 con 0x7f24f41589c0 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24db7fe700 1 -- 172.21.15.119:0/4096361304 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24ec0036c0 con 0x7f24f41589c0 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f41f73a0 con 0x7f24f41589c0 2024-09-13T16:36:24.582 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f41593b0 con 0x7f24f41589c0 2024-09-13T16:36:24.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f41f78c0 con 0x7f24f41589c0 2024-09-13T16:36:24.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.578+0000 7f24db7fe700 1 -- 172.21.15.119:0/4096361304 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f24ec003820 con 0x7f24f41589c0 2024-09-13T16:36:24.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.582+0000 7f24db7fe700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f24e0066730 0x7f24e0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:24.583 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.582+0000 7f24db7fe700 1 -- 172.21.15.119:0/4096361304 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f24ec0842d0 con 0x7f24f41589c0 2024-09-13T16:36:24.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.582+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f24e0066730 0x7f24e0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:24.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.582+0000 7f24f2ffd700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f24e0066730 0x7f24e0068bf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f24e80090b0 tx=0x7f24e801e040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:24.589 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.586+0000 7f24db7fe700 1 -- 172.21.15.119:0/4096361304 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f24f41f78c0 con 0x7f24f41589c0 2024-09-13T16:36:24.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.794+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24f41f78c0 con 0x7f24f41589c0 2024-09-13T16:36:24.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.794+0000 7f24db7fe700 1 -- 172.21.15.119:0/4096361304 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f24ec0582a0 con 0x7f24f41589c0 2024-09-13T16:36:24.799 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:24.799 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":97,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":97,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:24.662407+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8264":{"gid":8264,"name":"b","rank":0,"incarnation":95,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.145:6835/3898509964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3898509964},{"type":"v1","addr":"172.21.15.145:6835","nonce":3898509964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f24e0066730 msgr2=0x7f24e0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f24e0066730 0x7f24e0068bf0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f24e80090b0 tx=0x7f24e801e040 comp rx=0 tx=0).stop 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f41589c0 msgr2=0x7f24f4158e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f41589c0 0x7f24f4158e70 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7f24f4071850 tx=0x7f24ec011bb0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 shutdown_connections 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f24e0066730 0x7f24e0068bf0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24f41589c0 0x7f24f4158e70 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f24f41567c0 0x7f24f4156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 --2- 172.21.15.119:0/4096361304 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24dc003a50 0x7f24f4155890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:24.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 >> 172.21.15.119:0/4096361304 conn(0x7f24f406a350 msgr2=0x7f24f4163640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:24.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 shutdown_connections 2024-09-13T16:36:24.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:24.798+0000 7f24f3fff700 1 -- 172.21.15.119:0/4096361304 wait complete. 2024-09-13T16:36:24.802 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 97 2024-09-13T16:36:25.215 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 97, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:35:55.244835+0000', 'modified': '2024-09-13T16:36:24.662407+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 127, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8264': {'gid': 8264, 'name': 'b', 'rank': 0, 'incarnation': 95, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.145:6835/3898509964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3898509964}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 3898509964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:36:25.216 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_8264:up:rejoin 2024-09-13T16:36:26.217 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:26.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 Processor -- start 2024-09-13T16:36:26.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 -- start start 2024-09-13T16:36:26.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f33181594f0 0x7f331806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:26.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f331806da70 0x7f33180b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:26.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f33180b3630 0x7f331815d150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:26.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33180b2280 con 0x7f33181594f0 2024-09-13T16:36:26.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f331806deb0 con 0x7f33180b3630 2024-09-13T16:36:26.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331f04d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33180cc890 con 0x7f331806da70 2024-09-13T16:36:26.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331e04b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f33181594f0 0x7f331806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:26.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331e04b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f33181594f0 0x7f331806d530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36058/0 (socket says 172.21.15.119:36058) 2024-09-13T16:36:26.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331e04b700 1 -- 172.21.15.119:0/4244420947 learned_addr learned my addr 172.21.15.119:0/4244420947 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:26.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331e84c700 1 --2- 172.21.15.119:0/4244420947 >> v2:172.21.15.145:3300/0 conn(0x7f33180b3630 0x7f331815d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:26.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.334+0000 7f331d84a700 1 --2- 172.21.15.119:0/4244420947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f331806da70 0x7f33180b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:26.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331e04b700 1 -- 172.21.15.119:0/4244420947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f331806da70 msgr2=0x7f33180b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:26.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331e04b700 1 --2- 172.21.15.119:0/4244420947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f331806da70 0x7f33180b30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331e04b700 1 -- 172.21.15.119:0/4244420947 >> v2:172.21.15.145:3300/0 conn(0x7f33180b3630 msgr2=0x7f331815d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:26.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331e04b700 1 --2- 172.21.15.119:0/4244420947 >> v2:172.21.15.145:3300/0 conn(0x7f33180b3630 0x7f331815d150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331e04b700 1 -- 172.21.15.119:0/4244420947 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33180b1970 con 0x7f33181594f0 2024-09-13T16:36:26.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331e84c700 1 --2- 172.21.15.119:0/4244420947 >> v2:172.21.15.145:3300/0 conn(0x7f33180b3630 0x7f331815d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:26.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331e04b700 1 --2- 172.21.15.119:0/4244420947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33181594f0 0x7f331806d530 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f33180b1b60 tx=0x7f3308009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79e39a98b413108 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:26.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331d049700 1 -- 172.21.15.119:0/4244420947 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3308004230 con 0x7f33181594f0 2024-09-13T16:36:26.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331d049700 1 -- 172.21.15.119:0/4244420947 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3308004390 con 0x7f33181594f0 2024-09-13T16:36:26.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331d049700 1 -- 172.21.15.119:0/4244420947 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33080036c0 con 0x7f33181594f0 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 -- 172.21.15.119:0/4244420947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33181594f0 msgr2=0x7f331806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 --2- 172.21.15.119:0/4244420947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33181594f0 0x7f331806d530 secure :-1 s=READY pgs=1890 cs=0 l=1 rev1=1 crypto rx=0x7f33180b1b60 tx=0x7f3308009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 -- 172.21.15.119:0/4244420947 shutdown_connections 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 --2- 172.21.15.119:0/4244420947 >> v2:172.21.15.145:3300/0 conn(0x7f33180b3630 0x7f331815d150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 --2- 172.21.15.119:0/4244420947 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f331806da70 0x7f33180b30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 --2- 172.21.15.119:0/4244420947 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33181594f0 0x7f331806d530 unknown :-1 s=CLOSED pgs=1890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 -- 172.21.15.119:0/4244420947 >> 172.21.15.119:0/4244420947 conn(0x7f331806a350 msgr2=0x7f331806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 -- 172.21.15.119:0/4244420947 shutdown_connections 2024-09-13T16:36:26.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.338+0000 7f331f04d700 1 -- 172.21.15.119:0/4244420947 wait complete. 2024-09-13T16:36:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 Processor -- start 2024-09-13T16:36:26.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 -- start start 2024-09-13T16:36:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f331806da70 0x7f33180acb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33180b3630 0x7f33181576d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33181594f0 0x7f33181f4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318156930 con 0x7f33180b3630 2024-09-13T16:36:26.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3318155d20 con 0x7f331806da70 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3318156000 con 0x7f33181594f0 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33180b3630 0x7f33181576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33180b3630 0x7f33181576d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36064/0 (socket says 172.21.15.119:36064) 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 -- 172.21.15.119:0/1608962186 learned_addr learned my addr 172.21.15.119:0/1608962186 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331e04b700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f331806da70 0x7f33180acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331e84c700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33181594f0 0x7f33181f4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 -- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33181594f0 msgr2=0x7f33181f4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33181594f0 0x7f33181f4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 -- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f331806da70 msgr2=0x7f33180acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f331806da70 0x7f33180acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 -- 172.21.15.119:0/1608962186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3310009a70 con 0x7f33180b3630 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331e04b700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f331806da70 0x7f33180acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:26.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331d84a700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33180b3630 0x7f33181576d0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f331400e960 tx=0x7f331400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f330effd700 1 -- 172.21.15.119:0/1608962186 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33140096c0 con 0x7f33180b3630 2024-09-13T16:36:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3308009570 con 0x7f33180b3630 2024-09-13T16:36:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f330effd700 1 -- 172.21.15.119:0/1608962186 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331400c850 con 0x7f33180b3630 2024-09-13T16:36:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f330effd700 1 -- 172.21.15.119:0/1608962186 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f331400cbe0 con 0x7f33180b3630 2024-09-13T16:36:26.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33181565c0 con 0x7f33180b3630 2024-09-13T16:36:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.342+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33180b26f0 con 0x7f33180b3630 2024-09-13T16:36:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.346+0000 7f330effd700 1 -- 172.21.15.119:0/1608962186 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f331400cd40 con 0x7f33180b3630 2024-09-13T16:36:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.346+0000 7f330effd700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32fc066760 0x7f32fc068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.346+0000 7f330effd700 1 -- 172.21.15.119:0/1608962186 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f33140844f0 con 0x7f33180b3630 2024-09-13T16:36:26.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.346+0000 7f331e04b700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32fc066760 0x7f32fc068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:26.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.346+0000 7f331e04b700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32fc066760 0x7f32fc068c20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f33180b1b60 tx=0x7f3308008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:26.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.350+0000 7f330effd700 1 -- 172.21.15.119:0/1608962186 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f33180b26f0 con 0x7f33180b3630 2024-09-13T16:36:26.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.562+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f33180b26f0 con 0x7f33180b3630 2024-09-13T16:36:26.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.562+0000 7f330effd700 1 -- 172.21.15.119:0/1608962186 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4741 (secure 0 0 0) 0x7f33180b26f0 con 0x7f33180b3630 2024-09-13T16:36:26.565 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:26.565 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:25.670376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8264":{"gid":8264,"name":"b","rank":0,"incarnation":95,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6835/3898509964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3898509964},{"type":"v1","addr":"172.21.15.145:6835","nonce":3898509964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32fc066760 msgr2=0x7f32fc068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32fc066760 0x7f32fc068c20 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f33180b1b60 tx=0x7f3308008030 comp rx=0 tx=0).stop 2024-09-13T16:36:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33180b3630 msgr2=0x7f33181576d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:26.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33180b3630 0x7f33181576d0 secure :-1 s=READY pgs=1891 cs=0 l=1 rev1=1 crypto rx=0x7f331400e960 tx=0x7f331400ed20 comp rx=0 tx=0).stop 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 shutdown_connections 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f32fc066760 0x7f32fc068c20 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f33181594f0 0x7f33181f4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f33180b3630 0x7f33181576d0 unknown :-1 s=CLOSED pgs=1891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 --2- 172.21.15.119:0/1608962186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f331806da70 0x7f33180acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 >> 172.21.15.119:0/1608962186 conn(0x7f331806a350 msgr2=0x7f33180b0a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 shutdown_connections 2024-09-13T16:36:26.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:26.566+0000 7f331f04d700 1 -- 172.21.15.119:0/1608962186 wait complete. 2024-09-13T16:36:26.569 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 98 2024-09-13T16:36:26.982 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:35:55.244835+0000', 'modified': '2024-09-13T16:36:25.670376+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 127, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8264}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8264': {'gid': 8264, 'name': 'b', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.145:6835/3898509964', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 3898509964}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 3898509964}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:36:26.983 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:36:26.983 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-13T16:36:27.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 Processor -- start 2024-09-13T16:36:27.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 -- start start 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa2f00cd8d0 0x7fa2f006cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00b17f0 0x7fa2f00cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa2f00b2200 0x7fa2f00b2690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f00b2bd0 con 0x7fa2f00cd8d0 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2f00705d0 con 0x7fa2f00b2200 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f5a5d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f006d0d0 con 0x7fa2f00b17f0 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa2f00cd8d0 0x7fa2f006cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa2f00cd8d0 0x7fa2f006cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36080/0 (socket says 172.21.15.119:36080) 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3259730025 learned_addr learned my addr 172.21.15.119:0/3259730025 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f525c700 1 --2- 172.21.15.119:0/3259730025 >> v2:172.21.15.145:3300/0 conn(0x7fa2f00b2200 0x7fa2f00b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2effff700 1 --2- 172.21.15.119:0/3259730025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00b17f0 0x7fa2f00cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3259730025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00b17f0 msgr2=0x7fa2f00cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3259730025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00b17f0 0x7fa2f00cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3259730025 >> v2:172.21.15.145:3300/0 conn(0x7fa2f00b2200 msgr2=0x7fa2f00b2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3259730025 >> v2:172.21.15.145:3300/0 conn(0x7fa2f00b2200 0x7fa2f00b2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3259730025 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f006fca0 con 0x7fa2f00cd8d0 2024-09-13T16:36:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3259730025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00cd8d0 0x7fa2f006cb90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0165960 tx=0x7fa2d8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce116f927d2c04f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2ef7fe700 1 -- 172.21.15.119:0/3259730025 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8011830 con 0x7fa2f00cd8d0 2024-09-13T16:36:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2ef7fe700 1 -- 172.21.15.119:0/3259730025 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2d8011990 con 0x7fa2f00cd8d0 2024-09-13T16:36:27.112 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.106+0000 7fa2ef7fe700 1 -- 172.21.15.119:0/3259730025 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8011d20 con 0x7fa2f00cd8d0 2024-09-13T16:36:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3259730025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00cd8d0 msgr2=0x7fa2f006cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3259730025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00cd8d0 0x7fa2f006cb90 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7fa2f0165960 tx=0x7fa2d8009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3259730025 shutdown_connections 2024-09-13T16:36:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3259730025 >> v2:172.21.15.145:3300/0 conn(0x7fa2f00b2200 0x7fa2f00b2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3259730025 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00b17f0 0x7fa2f00cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.113 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3259730025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00cd8d0 0x7fa2f006cb90 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3259730025 >> 172.21.15.119:0/3259730025 conn(0x7fa2f006a350 msgr2=0x7fa2f006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3259730025 shutdown_connections 2024-09-13T16:36:27.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3259730025 wait complete. 2024-09-13T16:36:27.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 Processor -- start 2024-09-13T16:36:27.114 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- start start 2024-09-13T16:36:27.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b17f0 0x7fa2f014e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00b2200 0x7fa2f0164870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00cd8d0 0x7fa2f0164db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f01504a0 con 0x7fa2f00b2200 2024-09-13T16:36:27.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f01fa350 con 0x7fa2f00b17f0 2024-09-13T16:36:27.116 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f5a5d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f01fa630 con 0x7fa2f00cd8d0 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00b2200 0x7fa2f0164870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00b2200 0x7fa2f0164870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36094/0 (socket says 172.21.15.119:36094) 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2effff700 1 -- 172.21.15.119:0/3692090394 learned_addr learned my addr 172.21.15.119:0/3692090394 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f525c700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00cd8d0 0x7fa2f0164db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.110+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b17f0 0x7fa2f014e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2effff700 1 -- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00cd8d0 msgr2=0x7fa2f0164db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2effff700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00cd8d0 0x7fa2f0164db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2effff700 1 -- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b17f0 msgr2=0x7fa2f014e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2effff700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b17f0 0x7fa2f014e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2effff700 1 -- 172.21.15.119:0/3692090394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8009570 con 0x7fa2f00b2200 2024-09-13T16:36:27.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2effff700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00b2200 0x7fa2f0164870 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800efc0 tx=0x7fa2e800c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2ed7fa700 1 -- 172.21.15.119:0/3692090394 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8003a40 con 0x7fa2f00b2200 2024-09-13T16:36:27.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2ed7fa700 1 -- 172.21.15.119:0/3692090394 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e800f040 con 0x7fa2f00b2200 2024-09-13T16:36:27.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f01fa910 con 0x7fa2f00b2200 2024-09-13T16:36:27.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2ed7fa700 1 -- 172.21.15.119:0/3692090394 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8003de0 con 0x7fa2f00b2200 2024-09-13T16:36:27.118 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.114+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f00ad350 con 0x7fa2f00b2200 2024-09-13T16:36:27.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2ed7fa700 1 -- 172.21.15.119:0/3692090394 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa2e800c950 con 0x7fa2f00b2200 2024-09-13T16:36:27.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 Processor -- start 2024-09-13T16:36:27.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- start start 2024-09-13T16:36:27.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2ed7fa700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d0066730 0x7fa2d0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.120 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2ed7fa700 1 -- 172.21.15.119:0/3692090394 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fa2e8084ad0 con 0x7fa2f00b2200 2024-09-13T16:36:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f525c700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d0066730 0x7fa2d0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x284d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2815890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x2815dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.121 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2774d80 con 0x282be90 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2772510 con 0x284db50 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x276de00 con 0x284d230 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2815890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2815890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36106/0 (socket says 172.21.15.119:36106) 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f525c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x2815dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 -- 172.21.15.119:0/3382337390 learned_addr learned my addr 172.21.15.119:0/3382337390 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x284d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f525c700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d0066730 0x7fa2d0068bf0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009b80 tx=0x7fa2e0009f40 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.124 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 -- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 msgr2=0x284d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x284d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 -- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 msgr2=0x2815dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x2815dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 -- 172.21.15.119:0/3382337390 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x25e0960 con 0x282be90 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f525c700 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x2815dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2effff700 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2815890 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8082e00 tx=0x7fa2e8012410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa2974d18703d42 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2e67fc700 1 -- 172.21.15.119:0/3382337390 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8059710 con 0x282be90 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2e67fc700 1 -- 172.21.15.119:0/3382337390 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e805dcd0 con 0x282be90 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2e67fc700 1 -- 172.21.15.119:0/3382337390 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e806d1c0 con 0x282be90 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3382337390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 msgr2=0x2815890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.125 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2815890 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7fa2e8082e00 tx=0x7fa2e8012410 comp rx=0 tx=0).stop 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3382337390 shutdown_connections 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x2815dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2815890 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3382337390 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x284d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3382337390 >> 172.21.15.119:0/3382337390 conn(0x25954d0 msgr2=0x282ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.118+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3382337390 shutdown_connections 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3382337390 wait complete. 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 -- start start 2024-09-13T16:36:27.126 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 Processor -- start 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2807120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x282b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x282d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x272b730 con 0x282be90 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x27315b0 con 0x284db50 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2619aa0 con 0x284d230 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 10 client.0 _ll_register_callbacks cb 0x2767610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2807120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2807120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36116/0 (socket says 172.21.15.119:36116) 2024-09-13T16:36:27.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3837992528 learned_addr learned my addr 172.21.15.119:0/3837992528 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2effff700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x282b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f525c700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x282d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 msgr2=0x282b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x282b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 msgr2=0x282d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x282d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/3837992528 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e0011930 con 0x282be90 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f525c700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x282d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2effff700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x282b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2807120 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8014040 tx=0x7fa2d800e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d486d809ef8afb61 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2cf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/3837992528 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8021040 con 0x282be90 2024-09-13T16:36:27.130 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/3837992528 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fa2d8004450 con 0x282be90 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.122+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x25a21e0 con 0x282be90 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/3837992528 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2d8004610 con 0x282be90 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 10 client.9927 fetch_fsmap learned FSMap version 98 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x7fa2f00ad350 con 0x282be90 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/3837992528 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x25a21e0 con 0x282be90 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/3837992528 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 98) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa2d80036c0 con 0x282be90 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 10 client.9927 fetch_fsmap finished waiting for FSMap version 98 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 10 client.9927 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 20 client.9927 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2805f00 0x2860c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 4 client.9927 mds_command: new command op to 8264 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:36:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 --> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x25ec460 con 0x2805f00 2024-09-13T16:36:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2effff700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2805f00 0x2860c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2cf7fe700 10 client.9927 ms_handle_connect on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:27.132 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2effff700 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2805f00 0x2860c00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bfbce7f3176b6c7 server_cookie=f98b63a8f64e1bf5 in_seq=0 out_seq=0 2024-09-13T16:36:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/3837992528 <== mds.0 v2:172.21.15.145:6834/3898509964 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x25ec460 con 0x2805f00 2024-09-13T16:36:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2cf7fe700 10 client.9927 handle_command_reply: tid=0 2024-09-13T16:36:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 client.9927 shutdown 2024-09-13T16:36:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 msgr2=0x2807120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2807120 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8014040 tx=0x7fa2d800e000 comp rx=0 tx=0).stop 2024-09-13T16:36:27.133 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 shutdown_connections 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x282d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284d230 0x282b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2805f00 0x2860c00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/3837992528 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x2807120 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2cf7fe700 0 client.9927 ms_handle_reset on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 >> 172.21.15.119:0/3837992528 conn(0x25954d0 msgr2=0x2801700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 shutdown_connections 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.126+0000 7fa2f9f70740 1 -- 172.21.15.119:0/3837992528 wait complete. 2024-09-13T16:36:27.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.130+0000 7fa2f9f70740 20 client.9927 trim_cache size 0 max 16384 2024-09-13T16:36:27.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 Processor -- start 2024-09-13T16:36:27.153 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 -- start start 2024-09-13T16:36:27.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x282be90 0x285d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.155 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x284d230 0x284b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x284bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x2716020 con 0x284d230 2024-09-13T16:36:27.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x272fe90 con 0x284db50 2024-09-13T16:36:27.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2793f50 con 0x282be90 2024-09-13T16:36:27.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x284d230 0x284b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x284d230 0x284b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36128/0 (socket says 172.21.15.119:36128) 2024-09-13T16:36:27.156 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 -- 172.21.15.119:0/1634474586 learned_addr learned my addr 172.21.15.119:0/1634474586 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x282be90 0x285d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f525c700 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x284bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 -- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x282be90 msgr2=0x285d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x282be90 0x285d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 -- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 msgr2=0x284bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x284bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 -- 172.21.15.119:0/1634474586 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2734220 con 0x284d230 2024-09-13T16:36:27.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2f525c700 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x284bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:27.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2effff700 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x284d230 0x284b9b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800ef90 tx=0x7fa2e8073000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34c5c27373ac0951 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2e6ffd700 1 -- 172.21.15.119:0/1634474586 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8051c20 con 0x284d230 2024-09-13T16:36:27.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2e6ffd700 1 -- 172.21.15.119:0/1634474586 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e8059710 con 0x284d230 2024-09-13T16:36:27.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.150+0000 7fa2e6ffd700 1 -- 172.21.15.119:0/1634474586 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e8072230 con 0x284d230 2024-09-13T16:36:27.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1634474586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x284d230 msgr2=0x284b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x284d230 0x284b9b0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800ef90 tx=0x7fa2e8073000 comp rx=0 tx=0).stop 2024-09-13T16:36:27.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1634474586 shutdown_connections 2024-09-13T16:36:27.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284db50 0x284bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x284d230 0x284b9b0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1634474586 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x282be90 0x285d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1634474586 >> 172.21.15.119:0/1634474586 conn(0x25954d0 msgr2=0x285c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1634474586 shutdown_connections 2024-09-13T16:36:27.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1634474586 wait complete. 2024-09-13T16:36:27.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 -- start start 2024-09-13T16:36:27.165 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.154+0000 7fa2f9f70740 1 Processor -- start 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x28adc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284d230 0x28049e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284db50 0x2804f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x28ae140 con 0x282be90 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x2851180 con 0x284d230 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x2851420 con 0x284db50 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 10 client.0 _ll_register_callbacks cb 0x271fce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x28adc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x28adc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36142/0 (socket says 172.21.15.119:36142) 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/1303556024 learned_addr learned my addr 172.21.15.119:0/1303556024 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.166 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2effff700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284d230 0x28049e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f525c700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284db50 0x2804f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284db50 msgr2=0x2804f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284db50 0x2804f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284d230 msgr2=0x28049e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284d230 0x28049e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 -- 172.21.15.119:0/1303556024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x25e0960 con 0x282be90 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f525c700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284db50 0x2804f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f4a5b700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x28adc00 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d80092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2245421b590cdd0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2cf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/1303556024 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2d8021040 con 0x282be90 2024-09-13T16:36:27.167 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/1303556024 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fa2d8000e40 con 0x282be90 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2e805dcd0 con 0x282be90 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/1303556024 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2d8011630 con 0x282be90 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 10 client.9933 fetch_fsmap learned FSMap version 98 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x271a450 con 0x282be90 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/1303556024 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2e805dcd0 con 0x282be90 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/1303556024 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 98) v1 ==== 3114+0+0 (secure 0 0 0) 0x7fa2d8025420 con 0x282be90 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 10 client.9933 fetch_fsmap finished waiting for FSMap version 98 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 10 client.9933 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 20 client.9933 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2850060 0x2850440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 4 client.9933 mds_command: new command op to 8264 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:36:27.168 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 --> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] -- command(tid 0: {"prefix": "status"}) v1 -- 0x26111e0 con 0x2850060 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2effff700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2850060 0x2850440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2cf7fe700 10 client.9933 ms_handle_connect on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.158+0000 7fa2effff700 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2850060 0x2850440 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa6035fe1b74aa07 server_cookie=9c6cddeced0ad981 in_seq=0 out_seq=0 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2cf7fe700 1 -- 172.21.15.119:0/1303556024 <== mds.0 v2:172.21.15.145:6834/3898509964 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x26111e0 con 0x2850060 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2cf7fe700 10 client.9933 handle_command_reply: tid=0 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 client.9933 shutdown 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 msgr2=0x28adc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x28adc00 secure :-1 s=READY pgs=1897 cs=0 l=1 rev1=1 crypto rx=0x7fa2d8000c00 tx=0x7fa2d80092e0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 shutdown_connections 2024-09-13T16:36:27.169 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x284db50 0x2804f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x284d230 0x28049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x2850060 0x2850440 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 --2- 172.21.15.119:0/1303556024 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x282be90 0x28adc00 unknown :-1 s=CLOSED pgs=1897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 >> 172.21.15.119:0/1303556024 conn(0x25954d0 msgr2=0x2808240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2cf7fe700 0 client.9933 ms_handle_reset on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 shutdown_connections 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 1 -- 172.21.15.119:0/1303556024 wait complete. 2024-09-13T16:36:27.170 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f9f70740 20 client.9933 trim_cache size 0 max 16384 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 8264, 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 4.5007143000000003, 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 98, 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 127, 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 127, 2024-09-13T16:36:27.171 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 5.4726145329999998 2024-09-13T16:36:27.172 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:27.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d0066730 msgr2=0x7fa2d0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.172 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d0066730 0x7fa2d0068bf0 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7fa2e0009b80 tx=0x7fa2e0009f40 comp rx=0 tx=0).stop 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00b2200 msgr2=0x7fa2f0164870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00b2200 0x7fa2f0164870 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7fa2e800efc0 tx=0x7fa2e800c0c0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 shutdown_connections 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa2d0066730 0x7fa2d0068bf0 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa2f00cd8d0 0x7fa2f0164db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa2f00b2200 0x7fa2f0164870 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 --2- 172.21.15.119:0/3692090394 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa2f00b17f0 0x7fa2f014e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 >> 172.21.15.119:0/3692090394 conn(0x7fa2f006a350 msgr2=0x7fa2f0162690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 shutdown_connections 2024-09-13T16:36:27.173 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.162+0000 7fa2f5a5d700 1 -- 172.21.15.119:0/3692090394 wait complete. 2024-09-13T16:36:27.181 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:36:27.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.290+0000 7ff1a7391700 1 Processor -- start 2024-09-13T16:36:27.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.290+0000 7ff1a7391700 1 -- start start 2024-09-13T16:36:27.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff1a015cc40 0x7ff1a00ce710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a008bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff1a008c4e0 0x7ff1a0160720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a008c970 con 0x7ff1a015cc40 2024-09-13T16:36:27.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a00b44c0 con 0x7ff1a008c4e0 2024-09-13T16:36:27.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a015d020 con 0x7ff1a00cd860 2024-09-13T16:36:27.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff1a015cc40 0x7ff1a00ce710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff1a015cc40 0x7ff1a00ce710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36148/0 (socket says 172.21.15.119:36148) 2024-09-13T16:36:27.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 -- 172.21.15.119:0/3480137837 learned_addr learned my addr 172.21.15.119:0/3480137837 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a5b8e700 1 --2- 172.21.15.119:0/3480137837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a008bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a6b90700 1 --2- 172.21.15.119:0/3480137837 >> v2:172.21.15.145:3300/0 conn(0x7ff1a008c4e0 0x7ff1a0160720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 -- 172.21.15.119:0/3480137837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 msgr2=0x7ff1a008bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 --2- 172.21.15.119:0/3480137837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a008bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 -- 172.21.15.119:0/3480137837 >> v2:172.21.15.145:3300/0 conn(0x7ff1a008c4e0 msgr2=0x7ff1a0160720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 --2- 172.21.15.119:0/3480137837 >> v2:172.21.15.145:3300/0 conn(0x7ff1a008c4e0 0x7ff1a0160720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 -- 172.21.15.119:0/3480137837 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a00d0720 con 0x7ff1a015cc40 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a638f700 1 --2- 172.21.15.119:0/3480137837 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a015cc40 0x7ff1a00ce710 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ff1a008b1f0 tx=0x7ff19c009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be52d7136803f68d server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a538d700 1 -- 172.21.15.119:0/3480137837 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c011830 con 0x7ff1a015cc40 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a538d700 1 -- 172.21.15.119:0/3480137837 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c011990 con 0x7ff1a015cc40 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a538d700 1 -- 172.21.15.119:0/3480137837 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c011d40 con 0x7ff1a015cc40 2024-09-13T16:36:27.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- 172.21.15.119:0/3480137837 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a015cc40 msgr2=0x7ff1a00ce710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3480137837 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a015cc40 0x7ff1a00ce710 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7ff1a008b1f0 tx=0x7ff19c009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:27.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- 172.21.15.119:0/3480137837 shutdown_connections 2024-09-13T16:36:27.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3480137837 >> v2:172.21.15.145:3300/0 conn(0x7ff1a008c4e0 0x7ff1a0160720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3480137837 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a008bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3480137837 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a015cc40 0x7ff1a00ce710 unknown :-1 s=CLOSED pgs=1898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- 172.21.15.119:0/3480137837 >> 172.21.15.119:0/3480137837 conn(0x7ff1a00b4640 msgr2=0x7ff1a00b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- 172.21.15.119:0/3480137837 shutdown_connections 2024-09-13T16:36:27.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.294+0000 7ff1a7391700 1 -- 172.21.15.119:0/3480137837 wait complete. 2024-09-13T16:36:27.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 Processor -- start 2024-09-13T16:36:27.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 -- start start 2024-09-13T16:36:27.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a008c4e0 0x7ff1a00cb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a0159310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1a015cc40 0x7ff1a00b76f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00cbd40 con 0x7ff1a008c4e0 2024-09-13T16:36:27.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00ca760 con 0x7ff1a015cc40 2024-09-13T16:36:27.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00caa40 con 0x7ff1a00cd860 2024-09-13T16:36:27.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a008c4e0 0x7ff1a00cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a008c4e0 0x7ff1a00cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36156/0 (socket says 172.21.15.119:36156) 2024-09-13T16:36:27.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 -- 172.21.15.119:0/3948890046 learned_addr learned my addr 172.21.15.119:0/3948890046 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:27.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a5b8e700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a0159310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a6b90700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1a015cc40 0x7ff1a00b76f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 -- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 msgr2=0x7ff1a0159310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a0159310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 -- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1a015cc40 msgr2=0x7ff1a00b76f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1a015cc40 0x7ff1a00b76f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 -- 172.21.15.119:0/3948890046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c009570 con 0x7ff1a008c4e0 2024-09-13T16:36:27.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a5b8e700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a0159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:27.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a6b90700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1a015cc40 0x7ff1a00b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:27.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a638f700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a008c4e0 0x7ff1a00cb800 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7ff19c014040 tx=0x7ff19c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff196ffd700 1 -- 172.21.15.119:0/3948890046 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c004510 con 0x7ff1a008c4e0 2024-09-13T16:36:27.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff196ffd700 1 -- 172.21.15.119:0/3948890046 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19c01e070 con 0x7ff1a008c4e0 2024-09-13T16:36:27.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a00cad20 con 0x7ff1a008c4e0 2024-09-13T16:36:27.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff196ffd700 1 -- 172.21.15.119:0/3948890046 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19c003900 con 0x7ff1a008c4e0 2024-09-13T16:36:27.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.298+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a00cb2e0 con 0x7ff1a008c4e0 2024-09-13T16:36:27.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.302+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a00b42e0 con 0x7ff1a008c4e0 2024-09-13T16:36:27.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.302+0000 7ff196ffd700 1 -- 172.21.15.119:0/3948890046 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff19c003a60 con 0x7ff1a008c4e0 2024-09-13T16:36:27.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.302+0000 7ff196ffd700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff184066730 0x7ff184068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:27.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.302+0000 7ff196ffd700 1 -- 172.21.15.119:0/3948890046 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff19c0852a0 con 0x7ff1a008c4e0 2024-09-13T16:36:27.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.302+0000 7ff1a5b8e700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff184066730 0x7ff184068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:27.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.302+0000 7ff1a5b8e700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff184066730 0x7ff184068bf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00a0130 tx=0x7ff190008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:27.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.310+0000 7ff196ffd700 1 -- 172.21.15.119:0/3948890046 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff19c052540 con 0x7ff1a008c4e0 2024-09-13T16:36:27.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff1a00d1540 con 0x7ff1a008c4e0 2024-09-13T16:36:27.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff196ffd700 1 -- 172.21.15.119:0/3948890046 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v98) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff19c006da0 con 0x7ff1a008c4e0 2024-09-13T16:36:27.520 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:36:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff184066730 msgr2=0x7ff184068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff184066730 0x7ff184068bf0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7ff1a00a0130 tx=0x7ff190008040 comp rx=0 tx=0).stop 2024-09-13T16:36:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a008c4e0 msgr2=0x7ff1a00cb800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a008c4e0 0x7ff1a00cb800 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7ff19c014040 tx=0x7ff19c004770 comp rx=0 tx=0).stop 2024-09-13T16:36:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 shutdown_connections 2024-09-13T16:36:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff184066730 0x7ff184068bf0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff1a015cc40 0x7ff1a00b76f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff1a00cd860 0x7ff1a0159310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 --2- 172.21.15.119:0/3948890046 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff1a008c4e0 0x7ff1a00cb800 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:27.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.518+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 >> 172.21.15.119:0/3948890046 conn(0x7ff1a00b4640 msgr2=0x7ff1a00b3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:27.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.522+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 shutdown_connections 2024-09-13T16:36:27.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:27.522+0000 7ff1a7391700 1 -- 172.21.15.119:0/3948890046 wait complete. 2024-09-13T16:36:27.535 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:36:27.536 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:27.536 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.537 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:27.543 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 41954sec preferred_lft 41954sec 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:36:27.544 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:36:27.545 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.546 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:27.546 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:36:27.546 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:36:27.546 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:27.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.546 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:36:27.602 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:36:27.604 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.604 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:36:27.614 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:36:27.614 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:36:27.614 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:36:27.614 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:36:27.614 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:36:27.614 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:36:27.614 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:36:27.615 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:36:27.615 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.615 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:36:27.615 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:36:27.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:36:27.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:36:27.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:36:27.616 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:36:27.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.655 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:36:27.664 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-09-13T16:36:27.661+0000 7fdde7ae0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:36:27.709 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:36:27.709 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:36:27.710 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:27.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:36:27.723 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:36:27.723 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:27.724 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:27.728 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:27.728 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:27.729 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:27.776 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:27.793 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:27.793 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:27.807 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:27.807 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:27.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.808 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:27.814 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:36:27.814 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:36:27.856 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:27.871 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:27.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:27.883 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:27.884 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:27.884 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:27.884 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:27.946 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:36:27.942+0000 7f4411129640 -1 init, newargv = 0x55d9c3b8d540 newargc=15 2024-09-13T16:36:27.947 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[54756]: starting ceph client 2024-09-13T16:36:27.955 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[54756]: starting fuse 2024-09-13T16:36:28.791 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:36:28.794 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:36:28.993 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:29.011 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:29.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:29.022 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:29.023 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:29.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:29.024 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:29.028 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:36:29.029 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:36:29.029 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:36:29.030 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:36:29.031 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:36:29.031 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:29.031 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:36:29.031 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:29.155 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.54756.asok 2024-09-13T16:36:29.339 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.54756.asok 2024-09-13T16:36:29.340 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:29.340 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.54756.asok status 2024-09-13T16:36:29.439 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:29.439 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "54756", 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9945, 2024-09-13T16:36:29.440 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 9945 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3225992180 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:29.441 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 3225992180 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.9945 192.168.0.1:0/3225992180", 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/3225992180", 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 98, 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 127, 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:36:29.442 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:29.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:29.450 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:36:29.450 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:29.459 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:36:29.459 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:29.459 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:29.459 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:29.471 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:29.471 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-09-13T16:36:29.505 INFO:teuthology.orchestra.run.smithi119.stdout:./ 2024-09-13T16:36:29.505 INFO:teuthology.orchestra.run.smithi119.stdout:./parent 2024-09-13T16:36:29.505 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed 2024-09-13T16:36:29.505 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/charlie 2024-09-13T16:36:29.506 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/alpha 2024-09-13T16:36:29.506 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/unflushed 2024-09-13T16:36:29.506 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/unflushed/jfile 2024-09-13T16:36:29.507 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:29.507 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:29.507 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:29.507 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:29.507 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-09-13T16:36:29.507 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:29.618 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:36:29.807 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:29.807 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:29.807 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:29.808 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:29.808 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-09-13T16:36:29.808 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:29.870 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627776 2024-09-13T16:36:30.060 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:30.060 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:30.060 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:30.061 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:30.061 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-09-13T16:36:30.061 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:30.130 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627777 2024-09-13T16:36:30.318 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:30.318 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:30.319 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:30.319 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:30.319 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-09-13T16:36:30.319 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:30.432 INFO:teuthology.orchestra.run.smithi119.stdout:1099511628280 2024-09-13T16:36:30.621 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:30.621 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:30.621 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:30.621 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:30.621 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-09-13T16:36:30.621 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:30.691 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627778 2024-09-13T16:36:30.879 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:30.879 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:30.879 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:30.879 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:30.879 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-09-13T16:36:30.879 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:30.948 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627779 2024-09-13T16:36:31.139 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:31.139 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:31.139 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:31.140 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:31.140 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-09-13T16:36:31.140 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:31.210 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627780 2024-09-13T16:36:31.396 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:31.396 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:31.446 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 Processor -- start 2024-09-13T16:36:31.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 -- start start 2024-09-13T16:36:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45540adb30 0x7f45540cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f45540cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45540cde70 0x7f455415e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f455406cf60 con 0x7f45540cde70 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f455406c350 con 0x7f45540adb30 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4559387700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455406c630 con 0x7f455406bb90 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4558b86700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45540cde70 0x7f455415e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4553fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45540adb30 0x7f45540cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f45537fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f45540cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4553fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45540adb30 0x7f45540cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:54132/0 (socket says 172.21.15.119:54132) 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4553fff700 1 -- 172.21.15.119:0/1309150826 learned_addr learned my addr 172.21.15.119:0/1309150826 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:31.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.562+0000 7f4558b86700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45540cde70 0x7f455415e2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36158/0 (socket says 172.21.15.119:36158) 2024-09-13T16:36:31.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4553fff700 1 -- 172.21.15.119:0/1309150826 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 msgr2=0x7f45540cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:31.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4553fff700 1 --2- 172.21.15.119:0/1309150826 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f45540cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4553fff700 1 -- 172.21.15.119:0/1309150826 >> v2:172.21.15.119:3300/0 conn(0x7f45540cde70 msgr2=0x7f455415e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:31.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4553fff700 1 --2- 172.21.15.119:0/1309150826 >> v2:172.21.15.119:3300/0 conn(0x7f45540cde70 0x7f455415e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4553fff700 1 -- 172.21.15.119:0/1309150826 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f455406c910 con 0x7f45540adb30 2024-09-13T16:36:31.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4558b86700 1 --2- 172.21.15.119:0/1309150826 >> v2:172.21.15.119:3300/0 conn(0x7f45540cde70 0x7f455415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:31.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4553fff700 1 --2- 172.21.15.119:0/1309150826 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540adb30 0x7f45540cc060 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4554071fa0 tx=0x7f453c009bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=602427864aae680 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4552ffd700 1 -- 172.21.15.119:0/1309150826 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c011830 con 0x7f45540adb30 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4552ffd700 1 -- 172.21.15.119:0/1309150826 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c011990 con 0x7f45540adb30 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4552ffd700 1 -- 172.21.15.119:0/1309150826 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c011d20 con 0x7f45540adb30 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 -- 172.21.15.119:0/1309150826 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540adb30 msgr2=0x7f45540cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 --2- 172.21.15.119:0/1309150826 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540adb30 0x7f45540cc060 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f4554071fa0 tx=0x7f453c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 -- 172.21.15.119:0/1309150826 shutdown_connections 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 --2- 172.21.15.119:0/1309150826 >> v2:172.21.15.119:3300/0 conn(0x7f45540cde70 0x7f455415e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 --2- 172.21.15.119:0/1309150826 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f45540cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 --2- 172.21.15.119:0/1309150826 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540adb30 0x7f45540cc060 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 -- 172.21.15.119:0/1309150826 >> 172.21.15.119:0/1309150826 conn(0x7f455406a350 msgr2=0x7f455406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 -- 172.21.15.119:0/1309150826 shutdown_connections 2024-09-13T16:36:31.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 -- 172.21.15.119:0/1309150826 wait complete. 2024-09-13T16:36:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 Processor -- start 2024-09-13T16:36:31.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.566+0000 7f4559387700 1 -- start start 2024-09-13T16:36:31.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f455415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45540adb30 0x7f455413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540cde70 0x7f455414b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455415bd10 con 0x7f45540adb30 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455415bff0 con 0x7f45540cde70 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455415c2d0 con 0x7f455406bb90 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f45537fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45540adb30 0x7f455413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f45537fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45540adb30 0x7f455413bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36160/0 (socket says 172.21.15.119:36160) 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f45537fe700 1 -- 172.21.15.119:0/1844161672 learned_addr learned my addr 172.21.15.119:0/1844161672 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4553fff700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f455415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4558b86700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540cde70 0x7f455414b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4553fff700 1 -- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540cde70 msgr2=0x7f455414b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4553fff700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540cde70 0x7f455414b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4553fff700 1 -- 172.21.15.119:0/1844161672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45540adb30 msgr2=0x7f455413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4553fff700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45540adb30 0x7f455413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4553fff700 1 -- 172.21.15.119:0/1844161672 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f454c0094b0 con 0x7f455406bb90 2024-09-13T16:36:31.574 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f45537fe700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45540adb30 0x7f455413bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4553fff700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f455415cdb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f453c016000 tx=0x7f453c004770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4550ff9700 1 -- 172.21.15.119:0/1844161672 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c004240 con 0x7f455406bb90 2024-09-13T16:36:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4550ff9700 1 -- 172.21.15.119:0/1844161672 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453c01f070 con 0x7f455406bb90 2024-09-13T16:36:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4550ff9700 1 -- 172.21.15.119:0/1844161672 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f453c0045e0 con 0x7f455406bb90 2024-09-13T16:36:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f453c009570 con 0x7f455406bb90 2024-09-13T16:36:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.570+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455415c890 con 0x7f455406bb90 2024-09-13T16:36:31.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.574+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45540adf10 con 0x7f455406bb90 2024-09-13T16:36:31.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.574+0000 7f4550ff9700 1 -- 172.21.15.119:0/1844161672 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f453c021030 con 0x7f455406bb90 2024-09-13T16:36:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.574+0000 7f4550ff9700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4534066730 0x7f4534068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.574+0000 7f4550ff9700 1 -- 172.21.15.119:0/1844161672 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f453c085150 con 0x7f455406bb90 2024-09-13T16:36:31.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.574+0000 7f45537fe700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4534066730 0x7f4534068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.578+0000 7f45537fe700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4534066730 0x7f4534068bf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f45540cb660 tx=0x7f454400b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:31.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.586+0000 7f4550ff9700 1 -- 172.21.15.119:0/1844161672 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f453c052470 con 0x7f455406bb90 2024-09-13T16:36:31.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.762+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45540b2520 con 0x7f455406bb90 2024-09-13T16:36:31.767 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4550ff9700 1 -- 172.21.15.119:0/1844161672 <== mon.2 v2:172.21.15.145:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16975 (secure 0 0 0) 0x7f453c055a90 con 0x7f455406bb90 2024-09-13T16:36:31.768 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:31.769 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":127,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:36:21.631839+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3901706437":"2024-09-14T16:36:21.631788+0000","172.21.15.145:6834/3901706437":"2024-09-14T16:36:21.631788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:31.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4534066730 msgr2=0x7f4534068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4534066730 0x7f4534068bf0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f45540cb660 tx=0x7f454400b040 comp rx=0 tx=0).stop 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 msgr2=0x7f455415cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f455415cdb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f453c016000 tx=0x7f453c004770 comp rx=0 tx=0).stop 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 shutdown_connections 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4534066730 0x7f4534068bf0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45540cde70 0x7f455414b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45540adb30 0x7f455413bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 --2- 172.21.15.119:0/1844161672 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f455406bb90 0x7f455415cdb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 >> 172.21.15.119:0/1844161672 conn(0x7f455406a350 msgr2=0x7f4554141fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 shutdown_connections 2024-09-13T16:36:31.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.766+0000 7f4559387700 1 -- 172.21.15.119:0/1844161672 wait complete. 2024-09-13T16:36:31.783 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:31.783 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:31.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf7a184700 1 Processor -- start 2024-09-13T16:36:31.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf7a184700 1 -- start start 2024-09-13T16:36:31.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf7a184700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcf6c0df850 0x7fcf6c008170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.906 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf7a184700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcf6c050430 0x7fcf6c0562a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf7a184700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf6c00b160 con 0x7fcf6c055e70 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf7a184700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c009170 con 0x7fcf6c050430 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf7a184700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf6c008bb0 con 0x7fcf6c0df850 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf79983700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcf6c050430 0x7fcf6c0562a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf79182700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcf6c0df850 0x7fcf6c008170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf79983700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcf6c050430 0x7fcf6c0562a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36176/0 (socket says 172.21.15.119:36176) 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf78981700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fcf6c055e70 0x7fcf6c04fd20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:53440/0 (socket says 172.21.15.119:53440) 2024-09-13T16:36:31.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.902+0000 7fcf79983700 1 -- 172.21.15.119:0/2271312973 learned_addr learned my addr 172.21.15.119:0/2271312973 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:31.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf79182700 1 -- 172.21.15.119:0/2271312973 >> v2:172.21.15.119:3300/0 conn(0x7fcf6c050430 msgr2=0x7fcf6c0562a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:31.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf79182700 1 --2- 172.21.15.119:0/2271312973 >> v2:172.21.15.119:3300/0 conn(0x7fcf6c050430 0x7fcf6c0562a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf79182700 1 -- 172.21.15.119:0/2271312973 >> v1:172.21.15.145:6789/0 conn(0x7fcf6c055e70 legacy=0x7fcf6c04fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf79182700 1 -- 172.21.15.119:0/2271312973 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c008e90 con 0x7fcf6c0df850 2024-09-13T16:36:31.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf79983700 1 --2- 172.21.15.119:0/2271312973 >> v2:172.21.15.119:3300/0 conn(0x7fcf6c050430 0x7fcf6c0562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:31.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf79182700 1 --2- 172.21.15.119:0/2271312973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c0df850 0x7fcf6c008170 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00d680 tx=0x7fcf70009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e3d05da37a343cd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:31.909 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf67fff700 1 -- 172.21.15.119:0/2271312973 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf70005e50 con 0x7fcf6c0df850 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf67fff700 1 -- 172.21.15.119:0/2271312973 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf70006030 con 0x7fcf6c0df850 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf67fff700 1 -- 172.21.15.119:0/2271312973 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf700063c0 con 0x7fcf6c0df850 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 -- 172.21.15.119:0/2271312973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c0df850 msgr2=0x7fcf6c008170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 --2- 172.21.15.119:0/2271312973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c0df850 0x7fcf6c008170 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fcf6c00d680 tx=0x7fcf70009b20 comp rx=0 tx=0).stop 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 -- 172.21.15.119:0/2271312973 shutdown_connections 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 --2- 172.21.15.119:0/2271312973 >> v2:172.21.15.119:3300/0 conn(0x7fcf6c050430 0x7fcf6c0562a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 --2- 172.21.15.119:0/2271312973 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c0df850 0x7fcf6c008170 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 -- 172.21.15.119:0/2271312973 >> 172.21.15.119:0/2271312973 conn(0x7fcf6c01eaf0 msgr2=0x7fcf6c055760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 -- 172.21.15.119:0/2271312973 shutdown_connections 2024-09-13T16:36:31.910 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 -- 172.21.15.119:0/2271312973 wait complete. 2024-09-13T16:36:31.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 Processor -- start 2024-09-13T16:36:31.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.906+0000 7fcf7a184700 1 -- start start 2024-09-13T16:36:31.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf6c050430 0x7fcf6c0d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c055e70 0x7fcf6c1763b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.913 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf6c0df850 0x7fcf6c0e04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0dfdb0 con 0x7fcf6c0df850 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c0d6620 con 0x7fcf6c055e70 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c176be0 con 0x7fcf6c050430 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf6c0df850 0x7fcf6c0e04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf6c0df850 0x7fcf6c0e04e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36190/0 (socket says 172.21.15.119:36190) 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79182700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf6c050430 0x7fcf6c0d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 -- 172.21.15.119:0/3130527062 learned_addr learned my addr 172.21.15.119:0/3130527062 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf78981700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c055e70 0x7fcf6c1763b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 -- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf6c050430 msgr2=0x7fcf6c0d60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf6c050430 0x7fcf6c0d60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.914 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 -- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c055e70 msgr2=0x7fcf6c1763b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c055e70 0x7fcf6c1763b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 -- 172.21.15.119:0/3130527062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf70009450 con 0x7fcf6c0df850 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf79983700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf6c0df850 0x7fcf6c0e04e0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fcf7402ce60 tx=0x7fcf74067790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf65ffb700 1 -- 172.21.15.119:0/3130527062 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf7408f040 con 0x7fcf6c0df850 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf65ffb700 1 -- 172.21.15.119:0/3130527062 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf74073120 con 0x7fcf6c0df850 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf6c176ec0 con 0x7fcf6c0df850 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf65ffb700 1 -- 172.21.15.119:0/3130527062 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf74072370 con 0x7fcf6c0df850 2024-09-13T16:36:31.915 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf6c0dffc0 con 0x7fcf6c0df850 2024-09-13T16:36:31.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf6c0562a0 con 0x7fcf6c0df850 2024-09-13T16:36:31.916 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.910+0000 7fcf65ffb700 1 -- 172.21.15.119:0/3130527062 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcf740724d0 con 0x7fcf6c0df850 2024-09-13T16:36:31.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.914+0000 7fcf65ffb700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf58066730 0x7fcf58068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:31.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.914+0000 7fcf65ffb700 1 -- 172.21.15.119:0/3130527062 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fcf740fdc70 con 0x7fcf6c0df850 2024-09-13T16:36:31.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.914+0000 7fcf79182700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf58066730 0x7fcf58068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:31.917 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.914+0000 7fcf79182700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf58066730 0x7fcf58068bf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009b60 tx=0x7fcf70008070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:31.923 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:31.918+0000 7fcf65ffb700 1 -- 172.21.15.119:0/3130527062 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcf740caf80 con 0x7fcf6c0df850 2024-09-13T16:36:32.090 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.086+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcf6c0066f0 con 0x7fcf6c0df850 2024-09-13T16:36:32.091 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf65ffb700 1 -- 172.21.15.119:0/3130527062 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcf740ce5a0 con 0x7fcf6c0df850 2024-09-13T16:36:32.091 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:32.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf58066730 msgr2=0x7fcf58068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.093 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf58066730 0x7fcf58068bf0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fcf70009b60 tx=0x7fcf70008070 comp rx=0 tx=0).stop 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf6c0df850 msgr2=0x7fcf6c0e04e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf6c0df850 0x7fcf6c0e04e0 secure :-1 s=READY pgs=1902 cs=0 l=1 rev1=1 crypto rx=0x7fcf7402ce60 tx=0x7fcf74067790 comp rx=0 tx=0).stop 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 shutdown_connections 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf58066730 0x7fcf58068bf0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf6c0df850 0x7fcf6c0e04e0 unknown :-1 s=CLOSED pgs=1902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf6c055e70 0x7fcf6c1763b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 --2- 172.21.15.119:0/3130527062 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf6c050430 0x7fcf6c0d60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 >> 172.21.15.119:0/3130527062 conn(0x7fcf6c01eaf0 msgr2=0x7fcf6c036160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 shutdown_connections 2024-09-13T16:36:32.094 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.090+0000 7fcf7a184700 1 -- 172.21.15.119:0/3130527062 wait complete. 2024-09-13T16:36:32.105 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:32.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 Processor -- start 2024-09-13T16:36:32.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 -- start start 2024-09-13T16:36:32.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fbe7c15cfb0 0x7fbe7c06ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbe7c06b2d0 0x7fbe7c06b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06bc50 0x7fbe7c0cf5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7c0cee40 con 0x7fbe7c06b2d0 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe7c0d1220 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe839ee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c06c0e0 con 0x7fbe7c06bc50 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbe7c06b2d0 0x7fbe7c06b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbe7c06b2d0 0x7fbe7c06b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36196/0 (socket says 172.21.15.119:36196) 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 -- 172.21.15.119:0/149661644 learned_addr learned my addr 172.21.15.119:0/149661644 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe8178a700 1 --2- 172.21.15.119:0/149661644 >> v2:172.21.15.145:3300/0 conn(0x7fbe7c15cfb0 0x7fbe7c06ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe81f8b700 1 --2- 172.21.15.119:0/149661644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06bc50 0x7fbe7c0cf5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 -- 172.21.15.119:0/149661644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06bc50 msgr2=0x7fbe7c0cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 --2- 172.21.15.119:0/149661644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06bc50 0x7fbe7c0cf5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 -- 172.21.15.119:0/149661644 >> v2:172.21.15.145:3300/0 conn(0x7fbe7c15cfb0 msgr2=0x7fbe7c06ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 --2- 172.21.15.119:0/149661644 >> v2:172.21.15.145:3300/0 conn(0x7fbe7c15cfb0 0x7fbe7c06ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 -- 172.21.15.119:0/149661644 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c0ce880 con 0x7fbe7c06b2d0 2024-09-13T16:36:32.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe80f89700 1 --2- 172.21.15.119:0/149661644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c06b2d0 0x7fbe7c06b710 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c08b240 tx=0x7fbe70009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=595ffd73f47301ce server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe6ffff700 1 -- 172.21.15.119:0/149661644 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7000b8a0 con 0x7fbe7c06b2d0 2024-09-13T16:36:32.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe6ffff700 1 -- 172.21.15.119:0/149661644 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7000ba00 con 0x7fbe7c06b2d0 2024-09-13T16:36:32.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.218+0000 7fbe6ffff700 1 -- 172.21.15.119:0/149661644 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7000bd90 con 0x7fbe7c06b2d0 2024-09-13T16:36:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- 172.21.15.119:0/149661644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c06b2d0 msgr2=0x7fbe7c06b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 --2- 172.21.15.119:0/149661644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c06b2d0 0x7fbe7c06b710 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c08b240 tx=0x7fbe70009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- 172.21.15.119:0/149661644 shutdown_connections 2024-09-13T16:36:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 --2- 172.21.15.119:0/149661644 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06bc50 0x7fbe7c0cf5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 --2- 172.21.15.119:0/149661644 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c06b2d0 0x7fbe7c06b710 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 --2- 172.21.15.119:0/149661644 >> v2:172.21.15.145:3300/0 conn(0x7fbe7c15cfb0 0x7fbe7c06ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- 172.21.15.119:0/149661644 >> 172.21.15.119:0/149661644 conn(0x7fbe7c0d2da0 msgr2=0x7fbe7c0d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- 172.21.15.119:0/149661644 shutdown_connections 2024-09-13T16:36:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- 172.21.15.119:0/149661644 wait complete. 2024-09-13T16:36:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 Processor -- start 2024-09-13T16:36:32.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- start start 2024-09-13T16:36:32.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06b2d0 0x7fbe7c1521b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7c06bc50 0x7fbe7c0cb430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c15cfb0 0x7fbe7c0b74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c0cb010 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c152000 con 0x7fbe7c06bc50 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe839ee700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe7c1559c0 con 0x7fbe7c06b2d0 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.222+0000 7fbe81f8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c15cfb0 0x7fbe7c0b74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c15cfb0 0x7fbe7c0b74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:36206/0 (socket says 172.21.15.119:36206) 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 -- 172.21.15.119:0/2006631192 learned_addr learned my addr 172.21.15.119:0/2006631192 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe80f89700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7c06bc50 0x7fbe7c0cb430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe8178a700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06b2d0 0x7fbe7c1521b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 -- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06b2d0 msgr2=0x7fbe7c1521b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06b2d0 0x7fbe7c1521b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 -- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7c06bc50 msgr2=0x7fbe7c0cb430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7c06bc50 0x7fbe7c0cb430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.229 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 -- 172.21.15.119:0/2006631192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe70009570 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe81f8b700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c15cfb0 0x7fbe7c0b74e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400ccf0 tx=0x7fbe7400a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe6dffb700 1 -- 172.21.15.119:0/2006631192 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7400ad60 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe6dffb700 1 -- 172.21.15.119:0/2006631192 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe74003a40 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe7c155ca0 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe6dffb700 1 -- 172.21.15.119:0/2006631192 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74003dd0 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.230 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe7c0b7a20 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe7c0b7c70 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.226+0000 7fbe6dffb700 1 -- 172.21.15.119:0/2006631192 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbe74003f50 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.230+0000 7fbe6dffb700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe60066730 0x7fbe60068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.230+0000 7fbe6dffb700 1 -- 172.21.15.119:0/2006631192 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fbe74012070 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.230+0000 7fbe8178a700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe60066730 0x7fbe60068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.232 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.230+0000 7fbe8178a700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe60066730 0x7fbe60068bf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0ca010 tx=0x7fbe78008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.234+0000 7fbe6dffb700 1 -- 172.21.15.119:0/2006631192 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbe74051b80 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.402+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbe7c155ee0 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.402+0000 7fbe6dffb700 1 -- 172.21.15.119:0/2006631192 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbe7c155ee0 con 0x7fbe7c15cfb0 2024-09-13T16:36:32.406 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe60066730 msgr2=0x7fbe60068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe60066730 0x7fbe60068bf0 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0ca010 tx=0x7fbe78008040 comp rx=0 tx=0).stop 2024-09-13T16:36:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c15cfb0 msgr2=0x7fbe7c0b74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c15cfb0 0x7fbe7c0b74e0 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400ccf0 tx=0x7fbe7400a290 comp rx=0 tx=0).stop 2024-09-13T16:36:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 shutdown_connections 2024-09-13T16:36:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe60066730 0x7fbe60068bf0 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.409 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbe7c15cfb0 0x7fbe7c0b74e0 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbe7c06bc50 0x7fbe7c0cb430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 --2- 172.21.15.119:0/2006631192 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbe7c06b2d0 0x7fbe7c1521b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 >> 172.21.15.119:0/2006631192 conn(0x7fbe7c0d2da0 msgr2=0x7fbe7c0b2890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:32.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 shutdown_connections 2024-09-13T16:36:32.410 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.406+0000 7fbe839ee700 1 -- 172.21.15.119:0/2006631192 wait complete. 2024-09-13T16:36:32.421 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:32.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2dbe0fb700 1 Processor -- start 2024-09-13T16:36:32.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2dbe0fb700 1 -- start start 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2dbe0fb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db815cfb0 0x7f2db806ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2dbe0fb700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2db806b2d0 0x7f2db806b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2dbe0fb700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2db80ce880 con 0x7f2db806bc50 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2dbe0fb700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2db80cee40 con 0x7f2db806b2d0 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2dbe0fb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db806c100 con 0x7f2db815cfb0 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2db806b2d0 0x7f2db806b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2db806b2d0 0x7f2db806b710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46040/0 (socket says 172.21.15.119:46040) 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 -- 172.21.15.119:0/41026529 learned_addr learned my addr 172.21.15.119:0/41026529 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2db77fe700 1 --2- 172.21.15.119:0/41026529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db815cfb0 0x7f2db806ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.546 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 -- 172.21.15.119:0/41026529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db815cfb0 msgr2=0x7f2db806ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 --2- 172.21.15.119:0/41026529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db815cfb0 0x7f2db806ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 -- 172.21.15.119:0/41026529 >> v1:172.21.15.145:6789/0 conn(0x7f2db806bc50 legacy=0x7f2db80cf5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 -- 172.21.15.119:0/41026529 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2db80ceb60 con 0x7f2db806b2d0 2024-09-13T16:36:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2daeffd700 1 --2- 172.21.15.119:0/41026529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db806b2d0 0x7f2db806b710 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009910 tx=0x7f2da0009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbb5ce14eb05b284 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2db6ffd700 1 -- 172.21.15.119:0/41026529 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da000ba60 con 0x7f2db806b2d0 2024-09-13T16:36:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2db6ffd700 1 -- 172.21.15.119:0/41026529 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2da000bbc0 con 0x7f2db806b2d0 2024-09-13T16:36:32.547 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.542+0000 7f2db6ffd700 1 -- 172.21.15.119:0/41026529 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2da0009e60 con 0x7f2db806b2d0 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/41026529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db806b2d0 msgr2=0x7f2db806b710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/41026529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db806b2d0 0x7f2db806b710 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009910 tx=0x7f2da0009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/41026529 shutdown_connections 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/41026529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db806b2d0 0x7f2db806b710 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/41026529 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db815cfb0 0x7f2db806ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/41026529 >> 172.21.15.119:0/41026529 conn(0x7f2db80d2da0 msgr2=0x7f2db80d31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/41026529 shutdown_connections 2024-09-13T16:36:32.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/41026529 wait complete. 2024-09-13T16:36:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 Processor -- start 2024-09-13T16:36:32.550 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.546+0000 7f2dbe0fb700 1 -- start start 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db806b2d0 0x7f2db80b8820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db806bc50 0x7f2db80cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db815cfb0 0x7f2db8155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db80d1220 con 0x7f2db815cfb0 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db8159e70 con 0x7f2db806bc50 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2db81565e0 con 0x7f2db806b2d0 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db815cfb0 0x7f2db8155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db815cfb0 0x7f2db8155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46056/0 (socket says 172.21.15.119:46056) 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 -- 172.21.15.119:0/4258831608 learned_addr learned my addr 172.21.15.119:0/4258831608 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2daeffd700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db806bc50 0x7f2db80cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.553 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db77fe700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db806b2d0 0x7f2db80b8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 -- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db806b2d0 msgr2=0x7f2db80b8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db806b2d0 0x7f2db80b8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 -- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db806bc50 msgr2=0x7f2db80cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db806bc50 0x7f2db80cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 -- 172.21.15.119:0/4258831608 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da0009570 con 0x7f2db815cfb0 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2daeffd700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db806bc50 0x7f2db80cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db77fe700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db806b2d0 0x7f2db80b8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db7fff700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db815cfb0 0x7f2db8155980 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f2db000cc80 tx=0x7f2db000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db4ff9700 1 -- 172.21.15.119:0/4258831608 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000ad40 con 0x7f2db815cfb0 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db4ff9700 1 -- 172.21.15.119:0/4258831608 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2db000ce20 con 0x7f2db815cfb0 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db81e75a0 con 0x7f2db815cfb0 2024-09-13T16:36:32.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2db4ff9700 1 -- 172.21.15.119:0/4258831608 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2db000aea0 con 0x7f2db815cfb0 2024-09-13T16:36:32.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db80ccca0 con 0x7f2db815cfb0 2024-09-13T16:36:32.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.550+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db806b710 con 0x7f2db815cfb0 2024-09-13T16:36:32.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.554+0000 7f2db4ff9700 1 -- 172.21.15.119:0/4258831608 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2db0003c60 con 0x7f2db815cfb0 2024-09-13T16:36:32.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.554+0000 7f2db4ff9700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d98066730 0x7f2d98068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.554+0000 7f2db4ff9700 1 -- 172.21.15.119:0/4258831608 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2db00843d0 con 0x7f2db815cfb0 2024-09-13T16:36:32.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.554+0000 7f2db77fe700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d98066730 0x7f2d98068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.554+0000 7f2db77fe700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d98066730 0x7f2d98068bf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009840 tx=0x7f2da8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.558+0000 7f2db4ff9700 1 -- 172.21.15.119:0/4258831608 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2db0051580 con 0x7f2db815cfb0 2024-09-13T16:36:32.728 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.726+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2db80cce30 con 0x7f2db815cfb0 2024-09-13T16:36:32.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.726+0000 7f2db4ff9700 1 -- 172.21.15.119:0/4258831608 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2db80cce30 con 0x7f2db815cfb0 2024-09-13T16:36:32.729 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:32.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d98066730 msgr2=0x7f2d98068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d98066730 0x7f2d98068bf0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2da8009840 tx=0x7f2da8008040 comp rx=0 tx=0).stop 2024-09-13T16:36:32.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db815cfb0 msgr2=0x7f2db8155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db815cfb0 0x7f2db8155980 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f2db000cc80 tx=0x7f2db000a340 comp rx=0 tx=0).stop 2024-09-13T16:36:32.732 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 shutdown_connections 2024-09-13T16:36:32.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2d98066730 0x7f2d98068bf0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2db815cfb0 0x7f2db8155980 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2db806bc50 0x7f2db80cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 --2- 172.21.15.119:0/4258831608 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2db806b2d0 0x7f2db80b8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 >> 172.21.15.119:0/4258831608 conn(0x7f2db80d2da0 msgr2=0x7f2db80d1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:32.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 shutdown_connections 2024-09-13T16:36:32.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.730+0000 7f2dbe0fb700 1 -- 172.21.15.119:0/4258831608 wait complete. 2024-09-13T16:36:32.744 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:36:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae213a9700 1 Processor -- start 2024-09-13T16:36:32.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae213a9700 1 -- start start 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae213a9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b17f0 0x7fae1c0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae213a9700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fae1c0b2200 0x7fae1c0b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae213a9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae1c0705d0 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae213a9700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1c0b2bf0 con 0x7fae1c0b2200 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae213a9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c06fca0 con 0x7fae1c0b17f0 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1bfff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fae1c0cd8d0 0x7fae1c06cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56050/0 (socket says 172.21.15.119:56050) 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1bfff700 1 -- 172.21.15.119:0/3489852393 learned_addr learned my addr 172.21.15.119:0/3489852393 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae20ba8700 1 --2- 172.21.15.119:0/3489852393 >> v2:172.21.15.145:3300/0 conn(0x7fae1c0b2200 0x7fae1c0b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1b7fe700 1 --2- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b17f0 0x7fae1c0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1affd700 1 -- 172.21.15.119:0/3489852393 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495514342 0 0) 0x7fae1c0705d0 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1affd700 1 -- 172.21.15.119:0/3489852393 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae040035d0 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae20ba8700 1 -- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b17f0 msgr2=0x7fae1c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae20ba8700 1 --2- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b17f0 0x7fae1c0cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae20ba8700 1 -- 172.21.15.119:0/3489852393 >> v1:172.21.15.119:6789/0 conn(0x7fae1c0cd8d0 legacy=0x7fae1c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae20ba8700 1 -- 172.21.15.119:0/3489852393 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1c06ff80 con 0x7fae1c0b2200 2024-09-13T16:36:32.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae20ba8700 1 --2- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b2200 0x7fae1c0b26b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fae14009d90 tx=0x7fae1400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f2154e10e124f45 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1affd700 1 -- 172.21.15.119:0/3489852393 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1400e070 con 0x7fae1c0b2200 2024-09-13T16:36:32.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1affd700 1 -- 172.21.15.119:0/3489852393 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae1400bb20 con 0x7fae1c0b2200 2024-09-13T16:36:32.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.862+0000 7fae1affd700 1 -- 172.21.15.119:0/3489852393 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae1400be90 con 0x7fae1c0b2200 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 -- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b2200 msgr2=0x7fae1c0b26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 --2- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b2200 0x7fae1c0b26b0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fae14009d90 tx=0x7fae1400b330 comp rx=0 tx=0).stop 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 -- 172.21.15.119:0/3489852393 shutdown_connections 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 --2- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b2200 0x7fae1c0b26b0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 --2- 172.21.15.119:0/3489852393 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b17f0 0x7fae1c0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 -- 172.21.15.119:0/3489852393 >> 172.21.15.119:0/3489852393 conn(0x7fae1c06a350 msgr2=0x7fae1c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 -- 172.21.15.119:0/3489852393 shutdown_connections 2024-09-13T16:36:32.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 -- 172.21.15.119:0/3489852393 wait complete. 2024-09-13T16:36:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 Processor -- start 2024-09-13T16:36:32.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.866+0000 7fae213a9700 1 -- start start 2024-09-13T16:36:32.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b17f0 0x7fae1c14da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b2200 0x7fae1c0a8e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1c0cd8d0 0x7fae1c153820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0a97c0 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0a9aa0 con 0x7fae1c0b17f0 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1c0a9d80 con 0x7fae1c0b2200 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1c0cd8d0 0x7fae1c153820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1c0cd8d0 0x7fae1c153820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46070/0 (socket says 172.21.15.119:46070) 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 -- 172.21.15.119:0/1056117186 learned_addr learned my addr 172.21.15.119:0/1056117186 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae1bfff700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b17f0 0x7fae1c14da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae1b7fe700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b2200 0x7fae1c0a8e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 -- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b2200 msgr2=0x7fae1c0a8e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b2200 0x7fae1c0a8e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 -- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b17f0 msgr2=0x7fae1c14da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:32.874 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b17f0 0x7fae1c14da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 -- 172.21.15.119:0/1056117186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae14009a70 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae20ba8700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1c0cd8d0 0x7fae1c153820 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fae14014040 tx=0x7fae14004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae18ff9700 1 -- 172.21.15.119:0/1056117186 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae14004960 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae18ff9700 1 -- 172.21.15.119:0/1056117186 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae14020070 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae18ff9700 1 -- 172.21.15.119:0/1056117186 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae140276c0 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae1c151810 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.870+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae1c14df70 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.874+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1c151d30 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.876 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.874+0000 7fae18ff9700 1 -- 172.21.15.119:0/1056117186 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fae14012430 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.874+0000 7fae18ff9700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae08066760 0x7fae08068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:32.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.874+0000 7fae18ff9700 1 -- 172.21.15.119:0/1056117186 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fae14022070 con 0x7fae1c0cd8d0 2024-09-13T16:36:32.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.874+0000 7fae1bfff700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae08066760 0x7fae08068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:32.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.874+0000 7fae1bfff700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae08066760 0x7fae08068c20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fae10005930 tx=0x7fae10009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:32.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:32.882+0000 7fae18ff9700 1 -- 172.21.15.119:0/1056117186 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fae1c151d30 con 0x7fae1c0cd8d0 2024-09-13T16:36:33.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.058+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fae1c151d30 con 0x7fae1c0cd8d0 2024-09-13T16:36:33.061 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.058+0000 7fae18ff9700 1 -- 172.21.15.119:0/1056117186 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v127) v1 ==== 80+0+124 (secure 0 0 0) 0x7fae1c151d30 con 0x7fae1c0cd8d0 2024-09-13T16:36:33.062 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6835/3901706437 2024-09-14T16:36:21.631788+0000 2024-09-13T16:36:33.062 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6834/3901706437 2024-09-14T16:36:21.631788+0000 2024-09-13T16:36:33.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae08066760 msgr2=0x7fae08068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae08066760 0x7fae08068c20 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fae10005930 tx=0x7fae10009040 comp rx=0 tx=0).stop 2024-09-13T16:36:33.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1c0cd8d0 msgr2=0x7fae1c153820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1c0cd8d0 0x7fae1c153820 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fae14014040 tx=0x7fae14004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.064 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 shutdown_connections 2024-09-13T16:36:33.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae08066760 0x7fae08068c20 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae1c0cd8d0 0x7fae1c153820 unknown :-1 s=CLOSED pgs=1908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae1c0b2200 0x7fae1c0a8e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 --2- 172.21.15.119:0/1056117186 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae1c0b17f0 0x7fae1c14da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 >> 172.21.15.119:0/1056117186 conn(0x7fae1c06a350 msgr2=0x7fae1c161b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:33.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 shutdown_connections 2024-09-13T16:36:33.065 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.062+0000 7fae213a9700 1 -- 172.21.15.119:0/1056117186 wait complete. 2024-09-13T16:36:33.066 INFO:teuthology.orchestra.run.smithi119.stderr:listed 2 entries 2024-09-13T16:36:33.076 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:33.077 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:33.084 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:33.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.190+0000 7f86149d6700 1 Processor -- start 2024-09-13T16:36:33.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.190+0000 7f86149d6700 1 -- start start 2024-09-13T16:36:33.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.190+0000 7f86149d6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f861006dc40 0x7f86100b30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.190+0000 7f86149d6700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f86100b3630 0x7f861006fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.190+0000 7f86149d6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86100b2280 con 0x7f86101594f0 2024-09-13T16:36:33.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.190+0000 7f86149d6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86100cc890 con 0x7f861006dc40 2024-09-13T16:36:33.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.190+0000 7f86149d6700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86100b1970 con 0x7f86100b3630 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f861006dc40 0x7f86100b30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f861006dc40 0x7f86100b30f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46086/0 (socket says 172.21.15.119:46086) 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 -- 172.21.15.119:0/4031272878 learned_addr learned my addr 172.21.15.119:0/4031272878 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860f59e700 1 --2- 172.21.15.119:0/4031272878 >> v2:172.21.15.145:3300/0 conn(0x7f86100b3630 0x7f861006fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 -- 172.21.15.119:0/4031272878 >> v2:172.21.15.145:3300/0 conn(0x7f86100b3630 msgr2=0x7f861006fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 --2- 172.21.15.119:0/4031272878 >> v2:172.21.15.145:3300/0 conn(0x7f86100b3630 0x7f861006fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 -- 172.21.15.119:0/4031272878 >> v1:172.21.15.145:6789/0 conn(0x7f86101594f0 legacy=0x7f861006d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 -- 172.21.15.119:0/4031272878 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86100b1c30 con 0x7f861006dc40 2024-09-13T16:36:33.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860e59c700 1 --2- 172.21.15.119:0/4031272878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f861006dc40 0x7f86100b30f0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f86100ce6e0 tx=0x7f85f8009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd359de0214c53ab server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860dd9b700 1 -- 172.21.15.119:0/4031272878 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f8004230 con 0x7f861006dc40 2024-09-13T16:36:33.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860dd9b700 1 -- 172.21.15.119:0/4031272878 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8004390 con 0x7f861006dc40 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f860dd9b700 1 -- 172.21.15.119:0/4031272878 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f80036c0 con 0x7f861006dc40 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 -- 172.21.15.119:0/4031272878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f861006dc40 msgr2=0x7f86100b30f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 --2- 172.21.15.119:0/4031272878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f861006dc40 0x7f86100b30f0 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7f86100ce6e0 tx=0x7f85f8009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 -- 172.21.15.119:0/4031272878 shutdown_connections 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 --2- 172.21.15.119:0/4031272878 >> v2:172.21.15.145:3300/0 conn(0x7f86100b3630 0x7f861006fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 --2- 172.21.15.119:0/4031272878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f861006dc40 0x7f86100b30f0 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 -- 172.21.15.119:0/4031272878 >> 172.21.15.119:0/4031272878 conn(0x7f861006a350 msgr2=0x7f861006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 -- 172.21.15.119:0/4031272878 shutdown_connections 2024-09-13T16:36:33.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.194+0000 7f86149d6700 1 -- 172.21.15.119:0/4031272878 wait complete. 2024-09-13T16:36:33.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 Processor -- start 2024-09-13T16:36:33.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 -- start start 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f861006dc40 0x7f861015a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86100b3630 0x7f861015a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86101594f0 0x7f861015ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8610150c90 con 0x7f86100b3630 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86101579b0 con 0x7f86101594f0 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f86149d6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86101570c0 con 0x7f861006dc40 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86100b3630 0x7f861015a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86100b3630 0x7f861015a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46088/0 (socket says 172.21.15.119:46088) 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 -- 172.21.15.119:0/4171248384 learned_addr learned my addr 172.21.15.119:0/4171248384 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860ed9d700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f861006dc40 0x7f861015a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860f59e700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86101594f0 0x7f861015ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 -- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f861006dc40 msgr2=0x7f861015a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f861006dc40 0x7f861015a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 -- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86101594f0 msgr2=0x7f861015ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86101594f0 0x7f861015ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.198+0000 7f860e59c700 1 -- 172.21.15.119:0/4171248384 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f8009570 con 0x7f86100b3630 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f860e59c700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86100b3630 0x7f861015a8a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f85f8004000 tx=0x7f85f8004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86077fe700 1 -- 172.21.15.119:0/4171248384 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f801f070 con 0x7f86100b3630 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86077fe700 1 -- 172.21.15.119:0/4171248384 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8003e90 con 0x7f86100b3630 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86077fe700 1 -- 172.21.15.119:0/4171248384 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85f800b660 con 0x7f86100b3630 2024-09-13T16:36:33.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86101509b0 con 0x7f86100b3630 2024-09-13T16:36:33.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8610157640 con 0x7f86100b3630 2024-09-13T16:36:33.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86077fe700 1 -- 172.21.15.119:0/4171248384 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f85f800b7c0 con 0x7f86100b3630 2024-09-13T16:36:33.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86100b26f0 con 0x7f86100b3630 2024-09-13T16:36:33.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.202+0000 7f86077fe700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85f0066730 0x7f85f0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.206+0000 7f86077fe700 1 -- 172.21.15.119:0/4171248384 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f85f8087120 con 0x7f86100b3630 2024-09-13T16:36:33.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.206+0000 7f860ed9d700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85f0066730 0x7f85f0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.206+0000 7f860ed9d700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85f0066730 0x7f85f0068bf0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f8600009980 tx=0x7f86000090e0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.214+0000 7f86077fe700 1 -- 172.21.15.119:0/4171248384 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f86100b26f0 con 0x7f86100b3630 2024-09-13T16:36:33.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.386+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86100b26f0 con 0x7f86100b3630 2024-09-13T16:36:33.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86077fe700 1 -- 172.21.15.119:0/4171248384 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v127) v1 ==== 74+0+16975 (secure 0 0 0) 0x7f86100b26f0 con 0x7f86100b3630 2024-09-13T16:36:33.391 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:33.392 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":127,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:36:21.631839+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3901706437":"2024-09-14T16:36:21.631788+0000","172.21.15.145:6834/3901706437":"2024-09-14T16:36:21.631788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85f0066730 msgr2=0x7f85f0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85f0066730 0x7f85f0068bf0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7f8600009980 tx=0x7f86000090e0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86100b3630 msgr2=0x7f861015a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86100b3630 0x7f861015a8a0 secure :-1 s=READY pgs=1910 cs=0 l=1 rev1=1 crypto rx=0x7f85f8004000 tx=0x7f85f8004770 comp rx=0 tx=0).stop 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 shutdown_connections 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f85f0066730 0x7f85f0068bf0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f86101594f0 0x7f861015ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f86100b3630 0x7f861015a8a0 unknown :-1 s=CLOSED pgs=1910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 --2- 172.21.15.119:0/4171248384 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f861006dc40 0x7f861015a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 >> 172.21.15.119:0/4171248384 conn(0x7f861006a350 msgr2=0x7f86100ade50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:33.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 shutdown_connections 2024-09-13T16:36:33.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.390+0000 7f86149d6700 1 -- 172.21.15.119:0/4171248384 wait complete. 2024-09-13T16:36:33.406 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:33.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:33.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.522+0000 7f7751f92700 1 Processor -- start 2024-09-13T16:36:33.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.522+0000 7f7751f92700 1 -- start start 2024-09-13T16:36:33.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f774c15cc00 0x7f774c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f774c0d0c20 0x7f774c0d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f774c0cdb00 con 0x7f774c0ce8c0 2024-09-13T16:36:33.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774c0cdde0 con 0x7f774c15cc00 2024-09-13T16:36:33.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f774c0ce0c0 con 0x7f774c0d0c20 2024-09-13T16:36:33.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7750f90700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f774c15cc00 0x7f774c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751791700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f774c0ce8c0 0x7f774c160880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56064/0 (socket says 172.21.15.119:56064) 2024-09-13T16:36:33.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751791700 1 -- 172.21.15.119:0/2856501124 learned_addr learned my addr 172.21.15.119:0/2856501124 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:33.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f774bfff700 1 --2- 172.21.15.119:0/2856501124 >> v2:172.21.15.145:3300/0 conn(0x7f774c0d0c20 0x7f774c0d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7750f90700 1 -- 172.21.15.119:0/2856501124 >> v2:172.21.15.145:3300/0 conn(0x7f774c0d0c20 msgr2=0x7f774c0d10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7750f90700 1 --2- 172.21.15.119:0/2856501124 >> v2:172.21.15.145:3300/0 conn(0x7f774c0d0c20 0x7f774c0d10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7750f90700 1 -- 172.21.15.119:0/2856501124 >> v1:172.21.15.119:6789/0 conn(0x7f774c0ce8c0 legacy=0x7f774c160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7750f90700 1 -- 172.21.15.119:0/2856501124 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f774c0ce3a0 con 0x7f774c15cc00 2024-09-13T16:36:33.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f774bfff700 1 --2- 172.21.15.119:0/2856501124 >> v2:172.21.15.145:3300/0 conn(0x7f774c0d0c20 0x7f774c0d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7750f90700 1 --2- 172.21.15.119:0/2856501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c15cc00 0x7f774c0d06e0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f774c08c050 tx=0x7f7740009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3da01238b99931fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f774b7fe700 1 -- 172.21.15.119:0/2856501124 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7740011830 con 0x7f774c15cc00 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f774b7fe700 1 -- 172.21.15.119:0/2856501124 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7740011990 con 0x7f774c15cc00 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f774b7fe700 1 -- 172.21.15.119:0/2856501124 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7740011d40 con 0x7f774c15cc00 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- 172.21.15.119:0/2856501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c15cc00 msgr2=0x7f774c0d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 --2- 172.21.15.119:0/2856501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c15cc00 0x7f774c0d06e0 secure :-1 s=READY pgs=1912 cs=0 l=1 rev1=1 crypto rx=0x7f774c08c050 tx=0x7f7740009b30 comp rx=0 tx=0).stop 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- 172.21.15.119:0/2856501124 shutdown_connections 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 --2- 172.21.15.119:0/2856501124 >> v2:172.21.15.145:3300/0 conn(0x7f774c0d0c20 0x7f774c0d10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 --2- 172.21.15.119:0/2856501124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c15cc00 0x7f774c0d06e0 unknown :-1 s=CLOSED pgs=1912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- 172.21.15.119:0/2856501124 >> 172.21.15.119:0/2856501124 conn(0x7f774c08c080 msgr2=0x7f774c0d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- 172.21.15.119:0/2856501124 shutdown_connections 2024-09-13T16:36:33.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- 172.21.15.119:0/2856501124 wait complete. 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 Processor -- start 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.526+0000 7f7751f92700 1 -- start start 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f774c0ce8c0 0x7f774c0b3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c0d0c20 0x7f774c0ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f774c15cc00 0x7f774c0c8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c0cc9b0 con 0x7f774c0d0c20 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f774c0ccc90 con 0x7f774c0ce8c0 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f774c0ccf70 con 0x7f774c15cc00 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c0d0c20 0x7f774c0ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c0d0c20 0x7f774c0ca820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46116/0 (socket says 172.21.15.119:46116) 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 -- 172.21.15.119:0/3520335419 learned_addr learned my addr 172.21.15.119:0/3520335419 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:33.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7750f90700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f774c0ce8c0 0x7f774c0b3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751791700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f774c15cc00 0x7f774c0c8390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 -- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f774c15cc00 msgr2=0x7f774c0c8390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f774c15cc00 0x7f774c0c8390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 -- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f774c0ce8c0 msgr2=0x7f774c0b3c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f774c0ce8c0 0x7f774c0b3c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 -- 172.21.15.119:0/3520335419 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77400094b0 con 0x7f774c0d0c20 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751791700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f774c15cc00 0x7f774c0c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f774bfff700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c0d0c20 0x7f774c0ca820 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f773c00e760 tx=0x7f773c00ea70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f77497fa700 1 -- 172.21.15.119:0/3520335419 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c004230 con 0x7f774c0d0c20 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f77497fa700 1 -- 172.21.15.119:0/3520335419 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f773c004390 con 0x7f774c0d0c20 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f774c156d20 con 0x7f774c0d0c20 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f77497fa700 1 -- 172.21.15.119:0/3520335419 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f773c0036c0 con 0x7f774c0d0c20 2024-09-13T16:36:33.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f774c0b41a0 con 0x7f774c0d0c20 2024-09-13T16:36:33.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.530+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f774c0d06e0 con 0x7f774c0d0c20 2024-09-13T16:36:33.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.534+0000 7f77497fa700 1 -- 172.21.15.119:0/3520335419 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f773c003840 con 0x7f774c0d0c20 2024-09-13T16:36:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.534+0000 7f77497fa700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7738066730 0x7f7738068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.534+0000 7f77497fa700 1 -- 172.21.15.119:0/3520335419 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f773c083d30 con 0x7f774c0d0c20 2024-09-13T16:36:33.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.534+0000 7f7750f90700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7738066730 0x7f7738068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.534+0000 7f7750f90700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7738066730 0x7f7738068bf0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f77400090b0 tx=0x7f774001c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.542+0000 7f77497fa700 1 -- 172.21.15.119:0/3520335419 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f773c050f90 con 0x7f774c0d0c20 2024-09-13T16:36:33.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.714+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f774c08c560 con 0x7f774c0d0c20 2024-09-13T16:36:33.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.714+0000 7f77497fa700 1 -- 172.21.15.119:0/3520335419 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v127) v1 ==== 93+0+10 (secure 0 0 0) 0x7f773c0200a0 con 0x7f774c0d0c20 2024-09-13T16:36:33.717 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:33.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7738066730 msgr2=0x7f7738068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7738066730 0x7f7738068bf0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7f77400090b0 tx=0x7f774001c040 comp rx=0 tx=0).stop 2024-09-13T16:36:33.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c0d0c20 msgr2=0x7f774c0ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c0d0c20 0x7f774c0ca820 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f773c00e760 tx=0x7f773c00ea70 comp rx=0 tx=0).stop 2024-09-13T16:36:33.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 shutdown_connections 2024-09-13T16:36:33.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7738066730 0x7f7738068bf0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f774c15cc00 0x7f774c0c8390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f774c0d0c20 0x7f774c0ca820 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 --2- 172.21.15.119:0/3520335419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f774c0ce8c0 0x7f774c0b3c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 >> 172.21.15.119:0/3520335419 conn(0x7f774c08c080 msgr2=0x7f774c06b250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:33.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 shutdown_connections 2024-09-13T16:36:33.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.718+0000 7f7751f92700 1 -- 172.21.15.119:0/3520335419 wait complete. 2024-09-13T16:36:33.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497f59e700 1 Processor -- start 2024-09-13T16:36:33.859 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497f59e700 1 -- start start 2024-09-13T16:36:33.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4978033c10 0x7f49780e9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497f59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f497803a9b0 con 0x7f49780df160 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49780df540 con 0x7f4978055b70 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497e59c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f49780df160 0x7f4978039b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56080/0 (socket says 172.21.15.119:56080) 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497e59c700 1 -- 172.21.15.119:0/4069223583 learned_addr learned my addr 172.21.15.119:0/4069223583 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497f59e700 1 -- 172.21.15.119:0/4069223583 --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4978055720 con 0x7f4978033c10 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497ed9d700 1 --2- 172.21.15.119:0/4069223583 >> v2:172.21.15.119:3300/0 conn(0x7f4978033c10 0x7f49780e9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497ed9d700 1 -- 172.21.15.119:0/4069223583 >> v1:172.21.15.145:6789/0 conn(0x7f4978055b70 legacy=0x7f497803a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497ed9d700 1 -- 172.21.15.119:0/4069223583 >> v1:172.21.15.119:6789/0 conn(0x7f49780df160 legacy=0x7f4978039b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497ed9d700 1 -- 172.21.15.119:0/4069223583 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f497800c3f0 con 0x7f4978033c10 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497ed9d700 1 --2- 172.21.15.119:0/4069223583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780e9c90 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f498004ff70 tx=0x7f4980068790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfda1d5d92166928 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497d59a700 1 -- 172.21.15.119:0/4069223583 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4980093070 con 0x7f4978033c10 2024-09-13T16:36:33.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.858+0000 7f497d59a700 1 -- 172.21.15.119:0/4069223583 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498004f010 con 0x7f4978033c10 2024-09-13T16:36:33.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497d59a700 1 -- 172.21.15.119:0/4069223583 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49800676b0 con 0x7f4978033c10 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- 172.21.15.119:0/4069223583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 msgr2=0x7f49780e9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 --2- 172.21.15.119:0/4069223583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780e9c90 secure :-1 s=READY pgs=1915 cs=0 l=1 rev1=1 crypto rx=0x7f498004ff70 tx=0x7f4980068790 comp rx=0 tx=0).stop 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- 172.21.15.119:0/4069223583 shutdown_connections 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 --2- 172.21.15.119:0/4069223583 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780e9c90 unknown :-1 s=CLOSED pgs=1915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- 172.21.15.119:0/4069223583 >> 172.21.15.119:0/4069223583 conn(0x7f497801e270 msgr2=0x7f4978054ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- 172.21.15.119:0/4069223583 shutdown_connections 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- 172.21.15.119:0/4069223583 wait complete. 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 Processor -- start 2024-09-13T16:36:33.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- start start 2024-09-13T16:36:33.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780dc7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4978055b70 0x7f4978047930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49780df160 0x7f4978005000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4978048510 con 0x7f4978033c10 2024-09-13T16:36:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49780dcd30 con 0x7f49780df160 2024-09-13T16:36:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4978048230 con 0x7f4978055b70 2024-09-13T16:36:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780dc7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780dc7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46144/0 (socket says 172.21.15.119:46144) 2024-09-13T16:36:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497e59c700 1 -- 172.21.15.119:0/11020654 learned_addr learned my addr 172.21.15.119:0/11020654 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497ed9d700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49780df160 0x7f4978005000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.862+0000 7f497dd9b700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4978055b70 0x7f4978047930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497e59c700 1 -- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4978055b70 msgr2=0x7f4978047930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497e59c700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4978055b70 0x7f4978047930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497e59c700 1 -- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49780df160 msgr2=0x7f4978005000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497e59c700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49780df160 0x7f4978005000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497e59c700 1 -- 172.21.15.119:0/11020654 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49780e08c0 con 0x7f4978033c10 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497e59c700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780dc7f0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f497000c700 tx=0x7f497000ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f496effd700 1 -- 172.21.15.119:0/11020654 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49700074e0 con 0x7f4978033c10 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f496effd700 1 -- 172.21.15.119:0/11020654 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4970007640 con 0x7f4978033c10 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49780e0ba0 con 0x7f4978033c10 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f496effd700 1 -- 172.21.15.119:0/11020654 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4970005a90 con 0x7f4978033c10 2024-09-13T16:36:33.869 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4978047e70 con 0x7f4978033c10 2024-09-13T16:36:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49780e0720 con 0x7f4978033c10 2024-09-13T16:36:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f496effd700 1 -- 172.21.15.119:0/11020654 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4970005bf0 con 0x7f4978033c10 2024-09-13T16:36:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f496effd700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4960066730 0x7f4960068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:33.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f496effd700 1 -- 172.21.15.119:0/11020654 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f49700836c0 con 0x7f4978033c10 2024-09-13T16:36:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.866+0000 7f497dd9b700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4960066730 0x7f4960068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:33.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.870+0000 7f497dd9b700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4960066730 0x7f4960068bf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f497804e320 tx=0x7f4974009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:33.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:33.874+0000 7f496effd700 1 -- 172.21.15.119:0/11020654 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f49780e0720 con 0x7f4978033c10 2024-09-13T16:36:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.038+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f49780e0720 con 0x7f4978033c10 2024-09-13T16:36:34.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.038+0000 7f496effd700 1 -- 172.21.15.119:0/11020654 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) v1 ==== 104+0+11 (secure 0 0 0) 0x7f49780e0720 con 0x7f4978033c10 2024-09-13T16:36:34.042 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:34.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4960066730 msgr2=0x7f4960068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4960066730 0x7f4960068bf0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f497804e320 tx=0x7f4974009040 comp rx=0 tx=0).stop 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 msgr2=0x7f49780dc7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780dc7f0 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f497000c700 tx=0x7f497000ca10 comp rx=0 tx=0).stop 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 shutdown_connections 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4960066730 0x7f4960068bf0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f49780df160 0x7f4978005000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4978055b70 0x7f4978047930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 --2- 172.21.15.119:0/11020654 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4978033c10 0x7f49780dc7f0 unknown :-1 s=CLOSED pgs=1916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 >> 172.21.15.119:0/11020654 conn(0x7f497801e270 msgr2=0x7f49780dd5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 shutdown_connections 2024-09-13T16:36:34.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.042+0000 7f497f59e700 1 -- 172.21.15.119:0/11020654 wait complete. 2024-09-13T16:36:34.056 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:34.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5c56700 1 Processor -- start 2024-09-13T16:36:34.175 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5c56700 1 -- start start 2024-09-13T16:36:34.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5c56700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fded00cf190 0x7fded013ff50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5c56700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fded006aa60 0x7fded015d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5c56700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded006cc00 con 0x7fded015d560 2024-09-13T16:36:34.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5c56700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded006cee0 con 0x7fded00cf190 2024-09-13T16:36:34.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5c56700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded006d1c0 con 0x7fded006aa60 2024-09-13T16:36:34.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded4c54700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fded00cf190 0x7fded013ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fdecffff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fded015d560 0x7fded006a350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56088/0 (socket says 172.21.15.119:56088) 2024-09-13T16:36:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fdecffff700 1 -- 172.21.15.119:0/2318864715 learned_addr learned my addr 172.21.15.119:0/2318864715 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded5455700 1 --2- 172.21.15.119:0/2318864715 >> v2:172.21.15.145:3300/0 conn(0x7fded006aa60 0x7fded015d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded4c54700 1 -- 172.21.15.119:0/2318864715 >> v2:172.21.15.145:3300/0 conn(0x7fded006aa60 msgr2=0x7fded015d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded4c54700 1 --2- 172.21.15.119:0/2318864715 >> v2:172.21.15.145:3300/0 conn(0x7fded006aa60 0x7fded015d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded4c54700 1 -- 172.21.15.119:0/2318864715 >> v1:172.21.15.119:6789/0 conn(0x7fded015d560 legacy=0x7fded006a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded4c54700 1 -- 172.21.15.119:0/2318864715 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded006d4a0 con 0x7fded00cf190 2024-09-13T16:36:34.179 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.174+0000 7fded4c54700 1 --2- 172.21.15.119:0/2318864715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded013ff50 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fded01430a0 tx=0x7fdec4009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fdc548474f2ab52 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:34.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fdecf7fe700 1 -- 172.21.15.119:0/2318864715 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4011830 con 0x7fded00cf190 2024-09-13T16:36:34.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fdecf7fe700 1 -- 172.21.15.119:0/2318864715 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec4011990 con 0x7fded00cf190 2024-09-13T16:36:34.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fdecf7fe700 1 -- 172.21.15.119:0/2318864715 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec4011d40 con 0x7fded00cf190 2024-09-13T16:36:34.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 -- 172.21.15.119:0/2318864715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 msgr2=0x7fded013ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 --2- 172.21.15.119:0/2318864715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded013ff50 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7fded01430a0 tx=0x7fdec4009b30 comp rx=0 tx=0).stop 2024-09-13T16:36:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 -- 172.21.15.119:0/2318864715 shutdown_connections 2024-09-13T16:36:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 --2- 172.21.15.119:0/2318864715 >> v2:172.21.15.145:3300/0 conn(0x7fded006aa60 0x7fded015d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 --2- 172.21.15.119:0/2318864715 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded013ff50 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 -- 172.21.15.119:0/2318864715 >> 172.21.15.119:0/2318864715 conn(0x7fded006b460 msgr2=0x7fded006b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 -- 172.21.15.119:0/2318864715 shutdown_connections 2024-09-13T16:36:34.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 -- 172.21.15.119:0/2318864715 wait complete. 2024-09-13T16:36:34.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 Processor -- start 2024-09-13T16:36:34.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.178+0000 7fded5c56700 1 -- start start 2024-09-13T16:36:34.184 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fded006aa60 0x7fded013f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded01526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fded015d560 0x7fded0152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded01e7420 con 0x7fded00cf190 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded0154730 con 0x7fded006aa60 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded013f370 con 0x7fded015d560 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded01526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded01526a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46164/0 (socket says 172.21.15.119:46164) 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 -- 172.21.15.119:0/1798416491 learned_addr learned my addr 172.21.15.119:0/1798416491 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded4c54700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fded006aa60 0x7fded013f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5455700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fded015d560 0x7fded0152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.185 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 -- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fded015d560 msgr2=0x7fded0152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fded015d560 0x7fded0152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 -- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fded006aa60 msgr2=0x7fded013f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fded006aa60 0x7fded013f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 -- 172.21.15.119:0/1798416491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec40094b0 con 0x7fded00cf190 2024-09-13T16:36:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecffff700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded01526a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fdebc001eb0 tx=0x7fdebc00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecd7fa700 1 -- 172.21.15.119:0/1798416491 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc005d90 con 0x7fded00cf190 2024-09-13T16:36:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecd7fa700 1 -- 172.21.15.119:0/1798416491 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdebc0070d0 con 0x7fded00cf190 2024-09-13T16:36:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded0156f70 con 0x7fded00cf190 2024-09-13T16:36:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fdecd7fa700 1 -- 172.21.15.119:0/1798416491 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdebc007440 con 0x7fded00cf190 2024-09-13T16:36:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.182+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded0154450 con 0x7fded00cf190 2024-09-13T16:36:34.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.186+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded0157100 con 0x7fded00cf190 2024-09-13T16:36:34.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.186+0000 7fdecd7fa700 1 -- 172.21.15.119:0/1798416491 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fdebc0075a0 con 0x7fded00cf190 2024-09-13T16:36:34.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.186+0000 7fdecd7fa700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdeb4066730 0x7fdeb4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.186+0000 7fdecd7fa700 1 -- 172.21.15.119:0/1798416491 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fdebc083510 con 0x7fded00cf190 2024-09-13T16:36:34.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.186+0000 7fded4c54700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdeb4066730 0x7fdeb4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.186+0000 7fded4c54700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdeb4066730 0x7fdeb4068bf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdec40090b0 tx=0x7fdec4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:34.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.194+0000 7fdecd7fa700 1 -- 172.21.15.119:0/1798416491 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fdebc050780 con 0x7fded00cf190 2024-09-13T16:36:34.373 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.370+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fded015d9f0 con 0x7fded00cf190 2024-09-13T16:36:34.374 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.370+0000 7fdecd7fa700 1 -- 172.21.15.119:0/1798416491 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdebc053da0 con 0x7fded00cf190 2024-09-13T16:36:34.374 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdeb4066730 msgr2=0x7fdeb4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdeb4066730 0x7fdeb4068bf0 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7fdec40090b0 tx=0x7fdec4008030 comp rx=0 tx=0).stop 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 msgr2=0x7fded01526a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded01526a0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7fdebc001eb0 tx=0x7fdebc00c7b0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 shutdown_connections 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fdeb4066730 0x7fdeb4068bf0 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fded015d560 0x7fded0152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fded00cf190 0x7fded01526a0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 --2- 172.21.15.119:0/1798416491 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fded006aa60 0x7fded013f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 >> 172.21.15.119:0/1798416491 conn(0x7fded006b460 msgr2=0x7fded015fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:34.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 shutdown_connections 2024-09-13T16:36:34.378 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.374+0000 7fded5c56700 1 -- 172.21.15.119:0/1798416491 wait complete. 2024-09-13T16:36:34.389 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:36:34.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb11a9700 1 Processor -- start 2024-09-13T16:36:34.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb11a9700 1 -- start start 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb11a9700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2fac0d2a30 0x7f2fac0d2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb11a9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0af670 0x7f2fac0ce4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb11a9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fac0ce030 con 0x7f2fac0afab0 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb11a9700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fac0b3980 con 0x7f2fac0d2a30 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb11a9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0cd9e0 con 0x7f2fac0af670 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb09a8700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2fac0afab0 0x7f2fac0b2c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56102/0 (socket says 172.21.15.119:56102) 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fb09a8700 1 -- 172.21.15.119:0/130388791 learned_addr learned my addr 172.21.15.119:0/130388791 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fabfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2fac0d2a30 0x7f2fac0d2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fab7fe700 1 --2- 172.21.15.119:0/130388791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0af670 0x7f2fac0ce4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2faaffd700 1 -- 172.21.15.119:0/130388791 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1154534158 0 0) 0x7f2fac0ce030 con 0x7f2fac0afab0 2024-09-13T16:36:34.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2faaffd700 1 -- 172.21.15.119:0/130388791 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f940035d0 con 0x7f2fac0afab0 2024-09-13T16:36:34.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fab7fe700 1 -- 172.21.15.119:0/130388791 >> v2:172.21.15.119:3300/0 conn(0x7f2fac0d2a30 msgr2=0x7f2fac0d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:34.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fab7fe700 1 --2- 172.21.15.119:0/130388791 >> v2:172.21.15.119:3300/0 conn(0x7f2fac0d2a30 0x7f2fac0d2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fab7fe700 1 -- 172.21.15.119:0/130388791 >> v1:172.21.15.119:6789/0 conn(0x7f2fac0afab0 legacy=0x7f2fac0b2c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.506+0000 7f2fab7fe700 1 -- 172.21.15.119:0/130388791 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fac0cdcc0 con 0x7f2fac0af670 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fab7fe700 1 --2- 172.21.15.119:0/130388791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0af670 0x7f2fac0ce4c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009850 tx=0x7f2f9c009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=939ca9b4d6231047 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2faaffd700 1 -- 172.21.15.119:0/130388791 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c011830 con 0x7f2fac0af670 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2faaffd700 1 -- 172.21.15.119:0/130388791 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f9c011990 con 0x7f2fac0af670 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2faaffd700 1 -- 172.21.15.119:0/130388791 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f9c011d00 con 0x7f2fac0af670 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 -- 172.21.15.119:0/130388791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0af670 msgr2=0x7f2fac0ce4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/130388791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0af670 0x7f2fac0ce4c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c009850 tx=0x7f2f9c009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 -- 172.21.15.119:0/130388791 shutdown_connections 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/130388791 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0af670 0x7f2fac0ce4c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/130388791 >> v2:172.21.15.119:3300/0 conn(0x7f2fac0d2a30 0x7f2fac0d2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 -- 172.21.15.119:0/130388791 >> 172.21.15.119:0/130388791 conn(0x7f2fac08c0f0 msgr2=0x7f2fac08c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:34.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 -- 172.21.15.119:0/130388791 shutdown_connections 2024-09-13T16:36:34.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 -- 172.21.15.119:0/130388791 wait complete. 2024-09-13T16:36:34.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 Processor -- start 2024-09-13T16:36:34.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.510+0000 7f2fb11a9700 1 -- start start 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fac0af670 0x7f2fac0b6640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0afab0 0x7f2fac0c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fac0d2a30 0x7f2fac14ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0ca0b0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0ca2e0 con 0x7f2fac0af670 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fac0ca5c0 con 0x7f2fac0afab0 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fac0d2a30 0x7f2fac14ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fac0d2a30 0x7f2fac14ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46182/0 (socket says 172.21.15.119:46182) 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 -- 172.21.15.119:0/4129536342 learned_addr learned my addr 172.21.15.119:0/4129536342 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:34.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fab7fe700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0afab0 0x7f2fac0c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fabfff700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fac0af670 0x7f2fac0b6640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 -- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0afab0 msgr2=0x7f2fac0c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0afab0 0x7f2fac0c9b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 -- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fac0af670 msgr2=0x7f2fac0b6640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fac0af670 0x7f2fac0b6640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 -- 172.21.15.119:0/4129536342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c0094b0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fabfff700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fac0af670 0x7f2fac0b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb09a8700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fac0d2a30 0x7f2fac14ce50 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400caa0 tx=0x7f2fa400cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fa8ff9700 1 -- 172.21.15.119:0/4129536342 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400a770 con 0x7f2fac0d2a30 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fac0ca8a0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fa8ff9700 1 -- 172.21.15.119:0/4129536342 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fa400a8d0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fac0cae60 con 0x7f2fac0d2a30 2024-09-13T16:36:34.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fa8ff9700 1 -- 172.21.15.119:0/4129536342 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fa400ac40 con 0x7f2fac0d2a30 2024-09-13T16:36:34.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fac0d2e10 con 0x7f2fac0d2a30 2024-09-13T16:36:34.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.514+0000 7f2fa8ff9700 1 -- 172.21.15.119:0/4129536342 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2fa401b3b0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.518+0000 7f2fa8ff9700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f98066730 0x7f2f98068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:34.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.518+0000 7f2fa8ff9700 1 -- 172.21.15.119:0/4129536342 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f2fa4083cb0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.518+0000 7f2fabfff700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f98066730 0x7f2f98068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:34.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.518+0000 7f2fabfff700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f98066730 0x7f2f98068bf0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06bd10 tx=0x7f2fa0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:34.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.522+0000 7f2fa8ff9700 1 -- 172.21.15.119:0/4129536342 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2fa4050eb0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.698+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f2fac0cce30 con 0x7f2fac0d2a30 2024-09-13T16:36:34.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fa8ff9700 1 -- 172.21.15.119:0/4129536342 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 2 entries v127) v1 ==== 80+0+124 (secure 0 0 0) 0x7f2fa40544d0 con 0x7f2fac0d2a30 2024-09-13T16:36:34.703 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6835/3901706437 2024-09-14T16:36:21.631788+0000 2024-09-13T16:36:34.703 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6834/3901706437 2024-09-14T16:36:21.631788+0000 2024-09-13T16:36:34.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f98066730 msgr2=0x7f2f98068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f98066730 0x7f2f98068bf0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f2fac06bd10 tx=0x7f2fa0008040 comp rx=0 tx=0).stop 2024-09-13T16:36:34.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fac0d2a30 msgr2=0x7f2fac14ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:34.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fac0d2a30 0x7f2fac14ce50 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f2fa400caa0 tx=0x7f2fa400cb80 comp rx=0 tx=0).stop 2024-09-13T16:36:34.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 shutdown_connections 2024-09-13T16:36:34.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f98066730 0x7f2f98068bf0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fac0d2a30 0x7f2fac14ce50 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fac0afab0 0x7f2fac0c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 --2- 172.21.15.119:0/4129536342 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fac0af670 0x7f2fac0b6640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:34.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 >> 172.21.15.119:0/4129536342 conn(0x7f2fac08c0f0 msgr2=0x7f2fac0cfe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:34.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 shutdown_connections 2024-09-13T16:36:34.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:34.702+0000 7f2fb11a9700 1 -- 172.21.15.119:0/4129536342 wait complete. 2024-09-13T16:36:34.707 INFO:teuthology.orchestra.run.smithi119.stderr:listed 2 entries 2024-09-13T16:36:34.718 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:36:34.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:34.718 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:34.732 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[54756]: fuse finished with error 0 and tester_r 0 2024-09-13T16:36:34.733 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:36:40.834 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:36:40.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:40.835 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:36:40.843 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:40.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.958+0000 7fd773fff700 1 Processor -- start 2024-09-13T16:36:40.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.958+0000 7fd773fff700 1 -- start start 2024-09-13T16:36:40.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd773fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd77415ca00 0x7fd7740ad260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:40.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd773fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd7740abf40 0x7fd7740ac3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:40.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd773fff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7740ac8e0 con 0x7fd77415cde0 2024-09-13T16:36:40.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd773fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd774160460 con 0x7fd7740abf40 2024-09-13T16:36:40.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd773fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd77406ca70 con 0x7fd77415ca00 2024-09-13T16:36:40.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7737fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd7740abf40 0x7fd7740ac3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:40.965 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd76a7fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd77415cde0 0x7fd77406c360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56114/0 (socket says 172.21.15.119:56114) 2024-09-13T16:36:40.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd76a7fc700 1 -- 172.21.15.119:0/653506536 learned_addr learned my addr 172.21.15.119:0/653506536 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:40.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd772ffd700 1 --2- 172.21.15.119:0/653506536 >> v2:172.21.15.145:3300/0 conn(0x7fd77415ca00 0x7fd7740ad260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:40.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7737fe700 1 -- 172.21.15.119:0/653506536 >> v2:172.21.15.145:3300/0 conn(0x7fd77415ca00 msgr2=0x7fd7740ad260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:40.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7737fe700 1 --2- 172.21.15.119:0/653506536 >> v2:172.21.15.145:3300/0 conn(0x7fd77415ca00 0x7fd7740ad260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:40.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7727fc700 1 -- 172.21.15.119:0/653506536 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087408818 0 0) 0x7fd7740ac8e0 con 0x7fd77415cde0 2024-09-13T16:36:40.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7737fe700 1 -- 172.21.15.119:0/653506536 >> v1:172.21.15.119:6789/0 conn(0x7fd77415cde0 legacy=0x7fd77406c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:40.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7737fe700 1 -- 172.21.15.119:0/653506536 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd77415d310 con 0x7fd7740abf40 2024-09-13T16:36:40.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7737fe700 1 --2- 172.21.15.119:0/653506536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7740abf40 0x7fd7740ac3a0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fd77406fa40 tx=0x7fd76c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821e9a5345d36601 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:40.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7727fc700 1 -- 172.21.15.119:0/653506536 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c00e070 con 0x7fd7740abf40 2024-09-13T16:36:40.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7727fc700 1 -- 172.21.15.119:0/653506536 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76c00bb20 con 0x7fd7740abf40 2024-09-13T16:36:40.967 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.962+0000 7fd7727fc700 1 -- 172.21.15.119:0/653506536 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76c012430 con 0x7fd7740abf40 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- 172.21.15.119:0/653506536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7740abf40 msgr2=0x7fd7740ac3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 --2- 172.21.15.119:0/653506536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7740abf40 0x7fd7740ac3a0 secure :-1 s=READY pgs=1923 cs=0 l=1 rev1=1 crypto rx=0x7fd77406fa40 tx=0x7fd76c00b330 comp rx=0 tx=0).stop 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- 172.21.15.119:0/653506536 shutdown_connections 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 --2- 172.21.15.119:0/653506536 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd7740abf40 0x7fd7740ac3a0 unknown :-1 s=CLOSED pgs=1923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 --2- 172.21.15.119:0/653506536 >> v2:172.21.15.145:3300/0 conn(0x7fd77415ca00 0x7fd7740ad260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- 172.21.15.119:0/653506536 >> 172.21.15.119:0/653506536 conn(0x7fd7740ae750 msgr2=0x7fd7740b0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- 172.21.15.119:0/653506536 shutdown_connections 2024-09-13T16:36:40.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- 172.21.15.119:0/653506536 wait complete. 2024-09-13T16:36:40.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 Processor -- start 2024-09-13T16:36:40.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- start start 2024-09-13T16:36:40.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd77415ca00 0x7fd7741f60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:40.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd77415cde0 0x7fd774157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:40.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd774099a50 0x7fd77409c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7741f65f0 con 0x7fd77415ca00 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7740b1260 con 0x7fd77415cde0 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd773fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd77409b190 con 0x7fd774099a50 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd772ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd77415ca00 0x7fd7741f60b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd772ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd77415ca00 0x7fd7741f60b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46202/0 (socket says 172.21.15.119:46202) 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.966+0000 7fd772ffd700 1 -- 172.21.15.119:0/385751287 learned_addr learned my addr 172.21.15.119:0/385751287 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd76a7fc700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd77415cde0 0x7fd774157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd7737fe700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd774099a50 0x7fd77409c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd772ffd700 1 -- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd774099a50 msgr2=0x7fd77409c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd772ffd700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd774099a50 0x7fd77409c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd772ffd700 1 -- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd77415cde0 msgr2=0x7fd774157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd772ffd700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd77415cde0 0x7fd774157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:40.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd772ffd700 1 -- 172.21.15.119:0/385751287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c009a70 con 0x7fd77415ca00 2024-09-13T16:36:40.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd76a7fc700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd77415cde0 0x7fd774157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:40.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd772ffd700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd77415ca00 0x7fd7741f60b0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fd76400e870 tx=0x7fd76400ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:40.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd770ff9700 1 -- 172.21.15.119:0/385751287 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7640095c0 con 0x7fd77415ca00 2024-09-13T16:36:40.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd770ff9700 1 -- 172.21.15.119:0/385751287 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd76400ee70 con 0x7fd77415ca00 2024-09-13T16:36:40.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd770ff9700 1 -- 172.21.15.119:0/385751287 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd76400ca70 con 0x7fd77415ca00 2024-09-13T16:36:40.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77409a0f0 con 0x7fd77415ca00 2024-09-13T16:36:40.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77409a6b0 con 0x7fd77415ca00 2024-09-13T16:36:40.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7740ad260 con 0x7fd77415ca00 2024-09-13T16:36:40.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd770ff9700 1 -- 172.21.15.119:0/385751287 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd76400cbd0 con 0x7fd77415ca00 2024-09-13T16:36:40.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd770ff9700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd754066730 0x7fd754068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:40.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd770ff9700 1 -- 172.21.15.119:0/385751287 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fd764084300 con 0x7fd77415ca00 2024-09-13T16:36:40.974 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.970+0000 7fd76a7fc700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd754066730 0x7fd754068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:40.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.974+0000 7fd76a7fc700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd754066730 0x7fd754068bf0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd77415eda0 tx=0x7fd75c007cc0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:40.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:40.978+0000 7fd770ff9700 1 -- 172.21.15.119:0/385751287 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd7640514b0 con 0x7fd77415ca00 2024-09-13T16:36:41.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.182+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd77406bd90 con 0x7fd77415ca00 2024-09-13T16:36:41.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd770ff9700 1 -- 172.21.15.119:0/385751287 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4741 (secure 0 0 0) 0x7fd764054ad0 con 0x7fd77415ca00 2024-09-13T16:36:41.187 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:41.187 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":98,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:25.670376+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":127,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8264":{"gid":8264,"name":"b","rank":0,"incarnation":95,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6835/3898509964","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":3898509964},{"type":"v1","addr":"172.21.15.145:6835","nonce":3898509964}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd754066730 msgr2=0x7fd754068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd754066730 0x7fd754068bf0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fd77415eda0 tx=0x7fd75c007cc0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd77415ca00 msgr2=0x7fd7741f60b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd77415ca00 0x7fd7741f60b0 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7fd76400e870 tx=0x7fd76400ec30 comp rx=0 tx=0).stop 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 shutdown_connections 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd754066730 0x7fd754068bf0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd774099a50 0x7fd77409c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd77415cde0 0x7fd774157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 --2- 172.21.15.119:0/385751287 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd77415ca00 0x7fd7741f60b0 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 >> 172.21.15.119:0/385751287 conn(0x7fd7740abf40 msgr2=0x7fd7740aca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 shutdown_connections 2024-09-13T16:36:41.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.186+0000 7fd773fff700 1 -- 172.21.15.119:0/385751287 wait complete. 2024-09-13T16:36:41.193 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 98 2024-09-13T16:36:41.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json tag path /parent mytag123 2024-09-13T16:36:41.334 INFO:teuthology.orchestra.run.smithi145.stdout:{"return_code":0,"scrub_tag":"mytag123","mode":"asynchronous"} 2024-09-13T16:36:41.334 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mode": "asynchronous", "return_code": 0, "scrub_tag": "mytag123" } 2024-09-13T16:36:41.334 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f7.00000000 scrub_tag 2024-09-13T16:36:41.379 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 Processor -- start 2024-09-13T16:36:41.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 -- start start 2024-09-13T16:36:41.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845224f0 0x5593845fd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845228d0 0x5593845fafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845dfc20 0x5593845e0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593844b62f0 con 0x5593845224f0 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593844728e0 con 0x5593845dfc20 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cd5e8140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x559384472580 con 0x5593845228d0 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845224f0 0x5593845fd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845224f0 0x5593845fd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46228/0 (socket says 172.21.15.119:46228) 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 -- 172.21.15.119:0/3881415033 learned_addr learned my addr 172.21.15.119:0/3881415033 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb91e700 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845dfc20 0x5593845e0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7ca91c700 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845228d0 0x5593845fafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 -- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845228d0 msgr2=0x5593845fafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845228d0 0x5593845fafc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 -- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845dfc20 msgr2=0x5593845e0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845dfc20 0x5593845e0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.377+0000 7fa7cb11d700 1 -- 172.21.15.119:0/3881415033 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7b4005de0 con 0x5593845224f0 2024-09-13T16:36:41.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cb11d700 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845224f0 0x5593845fd4e0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00a530 tx=0x7fa7bc00a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4632929a2bd8cd7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7c991a700 1 -- 172.21.15.119:0/3881415033 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc0115f0 con 0x5593845224f0 2024-09-13T16:36:41.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7c991a700 1 -- 172.21.15.119:0/3881415033 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7bc011750 con 0x5593845224f0 2024-09-13T16:36:41.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7c991a700 1 -- 172.21.15.119:0/3881415033 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7bc011ae0 con 0x5593845224f0 2024-09-13T16:36:41.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/3881415033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845224f0 msgr2=0x5593845fd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.385 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845224f0 0x5593845fd4e0 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc00a530 tx=0x7fa7bc00a840 comp rx=0 tx=0).stop 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/3881415033 shutdown_connections 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845dfc20 0x5593845e0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845228d0 0x5593845fafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/3881415033 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845224f0 0x5593845fd4e0 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/3881415033 >> 172.21.15.119:0/3881415033 conn(0x559384523b10 msgr2=0x559384555ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/3881415033 shutdown_connections 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/3881415033 wait complete. 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 Processor -- start 2024-09-13T16:36:41.386 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.381+0000 7fa7cd5e8140 1 -- start start 2024-09-13T16:36:41.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845224f0 0x5593845fc730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845228d0 0x559384555c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.387 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845dfc20 0x5593845561d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x5593844488d0 con 0x5593845dfc20 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55938456cf00 con 0x5593845228d0 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55938456c280 con 0x5593845224f0 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845dfc20 0x5593845561d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845dfc20 0x5593845561d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46242/0 (socket says 172.21.15.119:46242) 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7ca91c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845228d0 0x559384555c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 -- 172.21.15.119:0/533715601 learned_addr learned my addr 172.21.15.119:0/533715601 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.388 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb11d700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845224f0 0x5593845fc730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 -- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845224f0 msgr2=0x5593845fc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.389 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x5593845224f0 0x5593845fc730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 -- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845228d0 msgr2=0x559384555c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845228d0 0x559384555c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 -- 172.21.15.119:0/533715601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7bc00d040 con 0x5593845dfc20 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7ca91c700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x5593845228d0 0x559384555c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cb91e700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845dfc20 0x5593845561d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400ef60 tx=0x7fa7c400ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7c8918700 1 -- 172.21.15.119:0/533715601 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c4002b60 con 0x5593845dfc20 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7c8918700 1 -- 172.21.15.119:0/533715601 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7c4008030 con 0x5593845dfc20 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x559384460b00 con 0x5593845dfc20 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7c8918700 1 -- 172.21.15.119:0/533715601 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7c400cc50 con 0x5593845dfc20 2024-09-13T16:36:41.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.385+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559384453f40 con 0x5593845dfc20 2024-09-13T16:36:41.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7c8918700 1 -- 172.21.15.119:0/533715601 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa7c4011450 con 0x5593845dfc20 2024-09-13T16:36:41.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7c8918700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa7ac065090 0x7fa7ac067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.392 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7c8918700 1 -- 172.21.15.119:0/533715601 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fa7c4082d80 con 0x5593845dfc20 2024-09-13T16:36:41.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55938456bad0 0x55938469abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 18.36 18:6fe38dc2:::100000001f7.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e127) v8 -- 0x55938469b130 con 0x55938456bad0 2024-09-13T16:36:41.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cb11d700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa7ac065090 0x7fa7ac067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7ca91c700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55938456bad0 0x55938469abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7ca91c700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55938456bad0 0x55938469abf0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cb11d700 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa7ac065090 0x7fa7ac067550 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f80 tx=0x7fa7bc011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7ca91c700 1 -- 172.21.15.119:0/533715601 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 100000001f7.00000000 [getxattr] v0'0 uv0 ondisk = -61 ((61) No data available)) v8 ==== 164+0+0 (crc 0 0 0) 0x7fa7b4019040 con 0x55938456bad0 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:error getting xattr cephfs_data/100000001f7.00000000/scrub_tag: (61) No data available 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55938456bad0 msgr2=0x55938469abf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55938456bad0 0x55938469abf0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa7ac065090 msgr2=0x7fa7ac067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa7ac065090 0x7fa7ac067550 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fa7bc000f80 tx=0x7fa7bc011000 comp rx=0 tx=0).stop 2024-09-13T16:36:41.395 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845dfc20 msgr2=0x5593845561d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 --2- 172.21.15.119:0/533715601 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x5593845dfc20 0x5593845561d0 secure :-1 s=READY pgs=1927 cs=0 l=1 rev1=1 crypto rx=0x7fa7c400ef60 tx=0x7fa7c400ef90 comp rx=0 tx=0).stop 2024-09-13T16:36:41.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cb91e700 1 -- 172.21.15.119:0/533715601 reap_dead start 2024-09-13T16:36:41.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 shutdown_connections 2024-09-13T16:36:41.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.389+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 >> 172.21.15.119:0/533715601 conn(0x559384523b10 msgr2=0x559384523ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.393+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 shutdown_connections 2024-09-13T16:36:41.396 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.393+0000 7fa7cd5e8140 1 -- 172.21.15.119:0/533715601 wait complete. 2024-09-13T16:36:41.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:41.398 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000002.00000000 scrub_tag 2024-09-13T16:36:41.436 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 Processor -- start 2024-09-13T16:36:41.437 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 -- start start 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77e7f300 0x557c77ec6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77f55100 0x557c77eadd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 0x557c77f56960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c77e15a00 con 0x557c77e7f300 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c77e112f0 con 0x557c77f55100 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa5330fd140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c77dcd8e0 con 0x557c77f554e0 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530c32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77e7f300 0x557c77ec6090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530431700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77f55100 0x557c77eadd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530c32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77e7f300 0x557c77ec6090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46256/0 (socket says 172.21.15.119:46256) 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530c32700 1 -- 172.21.15.119:0/417025050 learned_addr learned my addr 172.21.15.119:0/417025050 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa531433700 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 0x557c77f56960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530431700 1 -- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 msgr2=0x557c77f56960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530431700 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 0x557c77f56960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530431700 1 -- 172.21.15.119:0/417025050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77e7f300 msgr2=0x557c77ec6090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530431700 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77e7f300 0x557c77ec6090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530431700 1 -- 172.21.15.119:0/417025050 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa528005d20 con 0x557c77f55100 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.433+0000 7fa530431700 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77f55100 0x557c77eadd90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa518001b80 tx=0x7fa51800b2f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fbcfcbf81f8c8b7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa52f42f700 1 -- 172.21.15.119:0/417025050 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51800bbb0 con 0x557c77f55100 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa52f42f700 1 -- 172.21.15.119:0/417025050 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa51800bd10 con 0x557c77f55100 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa52f42f700 1 -- 172.21.15.119:0/417025050 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa51800e6b0 con 0x557c77f55100 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77f55100 msgr2=0x557c77eadd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.440 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77f55100 0x557c77eadd90 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7fa518001b80 tx=0x7fa51800b2f0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- 172.21.15.119:0/417025050 shutdown_connections 2024-09-13T16:36:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 0x557c77f56960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77f55100 0x557c77eadd90 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 --2- 172.21.15.119:0/417025050 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77e7f300 0x557c77ec6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- 172.21.15.119:0/417025050 >> 172.21.15.119:0/417025050 conn(0x557c77e80110 msgr2=0x557c77ec8bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- 172.21.15.119:0/417025050 shutdown_connections 2024-09-13T16:36:41.441 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- 172.21.15.119:0/417025050 wait complete. 2024-09-13T16:36:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 Processor -- start 2024-09-13T16:36:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- start start 2024-09-13T16:36:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77e7f300 0x557c77ead2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77f55100 0x557c77f4d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 0x557c77f451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c77dcd580 con 0x557c77f55100 2024-09-13T16:36:41.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c77da38d0 con 0x557c77e7f300 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa5330fd140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x557c77f50150 con 0x557c77f554e0 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77f55100 0x557c77f4d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77f55100 0x557c77f4d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46260/0 (socket says 172.21.15.119:46260) 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 -- 172.21.15.119:0/3938217 learned_addr learned my addr 172.21.15.119:0/3938217 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530c32700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77e7f300 0x557c77ead2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa531433700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 0x557c77f451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 -- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 msgr2=0x557c77f451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557c77f554e0 0x557c77f451c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 -- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77e7f300 msgr2=0x557c77ead2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77e7f300 0x557c77ead2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 -- 172.21.15.119:0/3938217 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c001e10 con 0x557c77f55100 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530c32700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557c77e7f300 0x557c77ead2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:41.444 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.437+0000 7fa530431700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77f55100 0x557c77f4d730 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fa518004170 tx=0x7fa518000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa52e42d700 1 -- 172.21.15.119:0/3938217 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518023640 con 0x557c77f55100 2024-09-13T16:36:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa52e42d700 1 -- 172.21.15.119:0/3938217 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5180237c0 con 0x557c77f55100 2024-09-13T16:36:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa52e42d700 1 -- 172.21.15.119:0/3938217 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa518023b70 con 0x557c77f55100 2024-09-13T16:36:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557c77daef40 con 0x557c77f55100 2024-09-13T16:36:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c77dae9a0 con 0x557c77f55100 2024-09-13T16:36:41.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa52e42d700 1 -- 172.21.15.119:0/3938217 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa518019440 con 0x557c77f55100 2024-09-13T16:36:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa52e42d700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa510065090 0x7fa510067550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa52e42d700 1 -- 172.21.15.119:0/3938217 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fa518084c30 con 0x557c77f55100 2024-09-13T16:36:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.441+0000 7fa5330fd140 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x557c77f45700 0x557c77f58030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa530c32700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa510065090 0x7fa510067550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:1 18.12 18:4bec3886:::10000000002.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e127) v8 -- 0x557c77f4dc70 con 0x557c77f45700 2024-09-13T16:36:41.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa531433700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x557c77f45700 0x557c77f58030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa530c32700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa510065090 0x7fa510067550 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa520000f80 tx=0x7fa520006a60 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa531433700 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x557c77f45700 0x557c77f58030 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa531433700 1 -- 172.21.15.119:0/3938217 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== osd_op_reply(1 10000000002.00000000 [getxattr (12) out=12b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7fa528008960 con 0x557c77f45700 2024-09-13T16:36:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x557c77f45700 msgr2=0x557c77f58030 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x557c77f45700 0x557c77f58030 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa510065090 msgr2=0x7fa510067550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa510065090 0x7fa510067550 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fa520000f80 tx=0x7fa520006a60 comp rx=0 tx=0).stop 2024-09-13T16:36:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77f55100 msgr2=0x557c77f4d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 --2- 172.21.15.119:0/3938217 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557c77f55100 0x557c77f4d730 secure :-1 s=READY pgs=1930 cs=0 l=1 rev1=1 crypto rx=0x7fa518004170 tx=0x7fa518000bc0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa531433700 1 -- 172.21.15.119:0/3938217 reap_dead start 2024-09-13T16:36:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 shutdown_connections 2024-09-13T16:36:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 >> 172.21.15.119:0/3938217 conn(0x557c77e80110 msgr2=0x557c77eadd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 shutdown_connections 2024-09-13T16:36:41.449 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.445+0000 7fa5330fd140 1 -- 172.21.15.119:0/3938217 wait complete. 2024-09-13T16:36:41.451 INFO:teuthology.orchestra.run.smithi119.stdout:mytag123 2024-09-13T16:36:41.451 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 100000001f8.00000000 scrub_tag 2024-09-13T16:36:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 Processor -- start 2024-09-13T16:36:41.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 -- start start 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd4f3300 0x55dffd5c5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd5c9100 0x55dffd5371c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c94e0 0x55dffd537700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dffd4852f0 con 0x55dffd5c9100 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dffd4418e0 con 0x55dffd5c94e0 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff451386140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dffd441580 con 0x55dffd4f3300 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff447fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd5c9100 0x55dffd5371c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff447fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd5c9100 0x55dffd5371c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46266/0 (socket says 172.21.15.119:46266) 2024-09-13T16:36:41.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.489+0000 7ff447fff700 1 -- 172.21.15.119:0/1084311188 learned_addr learned my addr 172.21.15.119:0/1084311188 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff447fff700 1 -- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd4f3300 msgr2=0x55dffd5c5e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff44eebb700 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd4f3300 0x55dffd5c5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff44f6bc700 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c94e0 0x55dffd537700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff447fff700 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd4f3300 0x55dffd5c5e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff447fff700 1 -- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c94e0 msgr2=0x55dffd537700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff447fff700 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c94e0 0x55dffd537700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff447fff700 1 -- 172.21.15.119:0/1084311188 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44800a100 con 0x55dffd5c9100 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff44f6bc700 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c94e0 0x55dffd537700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff447fff700 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd5c9100 0x55dffd5371c0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b240 tx=0x7ff43800b550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b8854c1d8c87608 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff44deb9700 1 -- 172.21.15.119:0/1084311188 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff438014070 con 0x55dffd5c9100 2024-09-13T16:36:41.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff44deb9700 1 -- 172.21.15.119:0/1084311188 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff43800f3f0 con 0x55dffd5c9100 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff44deb9700 1 -- 172.21.15.119:0/1084311188 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff43800f7a0 con 0x55dffd5c9100 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 -- 172.21.15.119:0/1084311188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd5c9100 msgr2=0x55dffd5371c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd5c9100 0x55dffd5371c0 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7ff43800b240 tx=0x7ff43800b550 comp rx=0 tx=0).stop 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 -- 172.21.15.119:0/1084311188 shutdown_connections 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c94e0 0x55dffd537700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd5c9100 0x55dffd5371c0 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 --2- 172.21.15.119:0/1084311188 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd4f3300 0x55dffd5c5e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 -- 172.21.15.119:0/1084311188 >> 172.21.15.119:0/1084311188 conn(0x55dffd4f4110 msgr2=0x55dffd5c68b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 -- 172.21.15.119:0/1084311188 shutdown_connections 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 -- 172.21.15.119:0/1084311188 wait complete. 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 Processor -- start 2024-09-13T16:36:41.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.493+0000 7ff451386140 1 -- start start 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd4f3300 0x55dffd652560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c9100 0x55dffd654440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd5c94e0 0x55dffd654980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dffd4178d0 con 0x55dffd4f3300 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dffd654050 con 0x55dffd5c9100 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dffd6567e0 con 0x55dffd5c94e0 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd4f3300 0x55dffd652560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd4f3300 0x55dffd652560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46280/0 (socket says 172.21.15.119:46280) 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 -- 172.21.15.119:0/992204173 learned_addr learned my addr 172.21.15.119:0/992204173 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff447fff700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c9100 0x55dffd654440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44f6bc700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd5c94e0 0x55dffd654980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 -- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd5c94e0 msgr2=0x55dffd654980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd5c94e0 0x55dffd654980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 -- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c9100 msgr2=0x55dffd654440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c9100 0x55dffd654440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 -- 172.21.15.119:0/992204173 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff438001b50 con 0x55dffd4f3300 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44f6bc700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55dffd5c94e0 0x55dffd654980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff447fff700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55dffd5c9100 0x55dffd654440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44eebb700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd4f3300 0x55dffd652560 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7ff4400084d0 tx=0x7ff4400087e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44ceb7700 1 -- 172.21.15.119:0/992204173 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440013760 con 0x55dffd4f3300 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44ceb7700 1 -- 172.21.15.119:0/992204173 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff440010040 con 0x55dffd4f3300 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55dffd42fb00 con 0x55dffd4f3300 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff44ceb7700 1 -- 172.21.15.119:0/992204173 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440013ae0 con 0x55dffd4f3300 2024-09-13T16:36:41.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.497+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dffd422f40 con 0x55dffd4f3300 2024-09-13T16:36:41.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff44ceb7700 1 -- 172.21.15.119:0/992204173 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff440005020 con 0x55dffd4f3300 2024-09-13T16:36:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff44ceb7700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff430065060 0x7ff430067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff44ceb7700 1 -- 172.21.15.119:0/992204173 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7ff440082c30 con 0x55dffd4f3300 2024-09-13T16:36:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55dffd5c2a70 0x55dffd654ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:1 18.19 18:997b7129:::100000001f8.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e127) v8 -- 0x55dffd5c2500 con 0x55dffd5c2a70 2024-09-13T16:36:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff447fff700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff430065060 0x7ff430067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff44f6bc700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55dffd5c2a70 0x55dffd654ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff447fff700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff430065060 0x7ff430067520 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff438000c00 tx=0x7ff438008000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff44f6bc700 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55dffd5c2a70 0x55dffd654ec0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff44f6bc700 1 -- 172.21.15.119:0/992204173 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(1 100000001f8.00000000 [getxattr (12) out=12b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7ff448017040 con 0x55dffd5c2a70 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55dffd5c2a70 msgr2=0x55dffd654ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55dffd5c2a70 0x55dffd654ec0 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff430065060 msgr2=0x7ff430067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff430065060 0x7ff430067520 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7ff438000c00 tx=0x7ff438008000 comp rx=0 tx=0).stop 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd4f3300 msgr2=0x55dffd652560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 --2- 172.21.15.119:0/992204173 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55dffd4f3300 0x55dffd652560 secure :-1 s=READY pgs=1933 cs=0 l=1 rev1=1 crypto rx=0x7ff4400084d0 tx=0x7ff4400087e0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff44f6bc700 1 -- 172.21.15.119:0/992204173 reap_dead start 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 shutdown_connections 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 >> 172.21.15.119:0/992204173 conn(0x55dffd4f4110 msgr2=0x55dffd5c5e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 shutdown_connections 2024-09-13T16:36:41.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.501+0000 7ff451386140 1 -- 172.21.15.119:0/992204173 wait complete. 2024-09-13T16:36:41.511 INFO:teuthology.orchestra.run.smithi119.stdout:mytag123 2024-09-13T16:36:41.511 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados --cluster ceph --pool cephfs_data getxattr 10000000004.00000000 scrub_tag 2024-09-13T16:36:41.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 Processor -- start 2024-09-13T16:36:41.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 -- start start 2024-09-13T16:36:41.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d55e80 0x560f57ddf530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d567a0 0x560f57d6e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d26640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57cb82f0 con 0x560f57dfa620 2024-09-13T16:36:41.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57c748e0 con 0x560f57d567a0 2024-09-13T16:36:41.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c646b140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f57c74580 con 0x560f57d55e80 2024-09-13T16:36:41.554 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d26640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c379f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d567a0 0x560f57d6e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c3fa0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d55e80 0x560f57ddf530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d26640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46292/0 (socket says 172.21.15.119:46292) 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 -- 172.21.15.119:0/524281674 learned_addr learned my addr 172.21.15.119:0/524281674 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 -- 172.21.15.119:0/524281674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d55e80 msgr2=0x560f57ddf530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d55e80 0x560f57ddf530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 -- 172.21.15.119:0/524281674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d567a0 msgr2=0x560f57d6e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d567a0 0x560f57d6e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 -- 172.21.15.119:0/524281674 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac005d20 con 0x560f57dfa620 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c379f700 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d567a0 0x560f57d6e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c47a1700 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d26640 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b290 tx=0x7fe4b400b5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743a6a9b7da467 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c279d700 1 -- 172.21.15.119:0/524281674 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4014070 con 0x560f57dfa620 2024-09-13T16:36:41.555 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c279d700 1 -- 172.21.15.119:0/524281674 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b400be70 con 0x560f57dfa620 2024-09-13T16:36:41.556 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.549+0000 7fe4c279d700 1 -- 172.21.15.119:0/524281674 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b400f5f0 con 0x560f57dfa620 2024-09-13T16:36:41.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- 172.21.15.119:0/524281674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 msgr2=0x560f57d26640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.558 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d26640 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b290 tx=0x7fe4b400b5a0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- 172.21.15.119:0/524281674 shutdown_connections 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d26640 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d567a0 0x560f57d6e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 --2- 172.21.15.119:0/524281674 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d55e80 0x560f57ddf530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- 172.21.15.119:0/524281674 >> 172.21.15.119:0/524281674 conn(0x560f57d27110 msgr2=0x560f57de0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- 172.21.15.119:0/524281674 shutdown_connections 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- 172.21.15.119:0/524281674 wait complete. 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 Processor -- start 2024-09-13T16:36:41.559 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- start start 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d55e80 0x560f57de7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d567a0 0x560f57d48760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d48ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57c4a8d0 con 0x560f57dfa620 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f57de6a20 con 0x560f57d55e80 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x560f57de6cb0 con 0x560f57d567a0 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d48ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d48ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46308/0 (socket says 172.21.15.119:46308) 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 -- 172.21.15.119:0/618265637 learned_addr learned my addr 172.21.15.119:0/618265637 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c379f700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d567a0 0x560f57d48760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c3fa0700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d55e80 0x560f57de7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.560 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 -- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d567a0 msgr2=0x560f57d48760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d567a0 0x560f57d48760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 -- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d55e80 msgr2=0x560f57de7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d55e80 0x560f57de7930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 -- 172.21.15.119:0/618265637 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f57c7cca0 con 0x560f57dfa620 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c379f700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x560f57d567a0 0x560f57d48760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c3fa0700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x560f57d55e80 0x560f57de7930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c47a1700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d48ca0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000f60 tx=0x7fe4b4000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c179b700 1 -- 172.21.15.119:0/618265637 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4014030 con 0x560f57dfa620 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c179b700 1 -- 172.21.15.119:0/618265637 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b401b860 con 0x560f57dfa620 2024-09-13T16:36:41.561 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b4001b90 con 0x560f57dfa620 2024-09-13T16:36:41.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c179b700 1 -- 172.21.15.119:0/618265637 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b401bc10 con 0x560f57dfa620 2024-09-13T16:36:41.562 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.553+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f57c62b00 con 0x560f57dfa620 2024-09-13T16:36:41.563 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c179b700 1 -- 172.21.15.119:0/618265637 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe4b400d040 con 0x560f57dfa620 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c179b700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4a4065060 0x7fe4a4067520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c179b700 1 -- 172.21.15.119:0/618265637 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fe4b400a040 con 0x560f57dfa620 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c646b140 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x560f57d68710 0x560f57ddda70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 18.5 18:a367a3e8:::10000000004.00000000:head [getxattr scrub_tag in=9b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e127) v8 -- 0x560f57dddfb0 con 0x560f57d68710 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c3fa0700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4a4065060 0x7fe4a4067520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c379f700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x560f57d68710 0x560f57ddda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c379f700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x560f57d68710 0x560f57ddda70 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c3fa0700 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4a4065060 0x7fe4a4067520 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005fd0 tx=0x7fe4bc007040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.557+0000 7fe4c379f700 1 -- 172.21.15.119:0/618265637 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 10000000004.00000000 [getxattr (12) out=12b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+12 (crc 0 0 0) 0x7fe4ac019040 con 0x560f57d68710 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x560f57d68710 msgr2=0x560f57ddda70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x560f57d68710 0x560f57ddda70 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4a4065060 msgr2=0x7fe4a4067520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe4a4065060 0x7fe4a4067520 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005fd0 tx=0x7fe4bc007040 comp rx=0 tx=0).stop 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 msgr2=0x560f57d48ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 --2- 172.21.15.119:0/618265637 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x560f57dfa620 0x560f57d48ca0 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fe4b4000f60 tx=0x7fe4b4000ed0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c47a1700 1 -- 172.21.15.119:0/618265637 reap_dead start 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 shutdown_connections 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 >> 172.21.15.119:0/618265637 conn(0x560f57d27110 msgr2=0x560f57ddf530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 shutdown_connections 2024-09-13T16:36:41.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.561+0000 7fe4c646b140 1 -- 172.21.15.119:0/618265637 wait complete. 2024-09-13T16:36:41.567 INFO:teuthology.orchestra.run.smithi119.stdout:mytag123 2024-09-13T16:36:41.567 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.8:0 flush journal 2024-09-13T16:36:41.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.689+0000 7f04449ab700 1 Processor -- start 2024-09-13T16:36:41.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.689+0000 7f04449ab700 1 -- start start 2024-09-13T16:36:41.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f044015cc00 0x7f04400d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04400d0c20 0x7f04400d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04400cdb00 con 0x7f04400ce8c0 2024-09-13T16:36:41.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04400cdde0 con 0x7f04400d0c20 2024-09-13T16:36:41.696 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04400ce0c0 con 0x7f044015cc00 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f0436ffd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f04400d0c20 0x7f04400d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043ffff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f04400ce8c0 0x7f0440160880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:49910/0 (socket says 172.21.15.119:49910) 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043ffff700 1 -- 172.21.15.119:0/1610330584 learned_addr learned my addr 172.21.15.119:0/1610330584 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043f7fe700 1 --2- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f044015cc00 0x7f04400d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043effd700 1 -- 172.21.15.119:0/1610330584 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792341249 0 0) 0x7f04400cdb00 con 0x7f04400ce8c0 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043effd700 1 -- 172.21.15.119:0/1610330584 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f042c0035d0 con 0x7f04400ce8c0 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f0436ffd700 1 -- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f044015cc00 msgr2=0x7f04400d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f0436ffd700 1 --2- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f044015cc00 0x7f04400d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f0436ffd700 1 -- 172.21.15.119:0/1610330584 >> v1:172.21.15.145:6789/0 conn(0x7f04400ce8c0 legacy=0x7f0440160880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.697 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f0436ffd700 1 -- 172.21.15.119:0/1610330584 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04400ce3a0 con 0x7f04400d0c20 2024-09-13T16:36:41.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f0436ffd700 1 --2- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400d0c20 0x7f04400d10d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0428009850 tx=0x7f0428009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48ae72a2c7179b9f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043effd700 1 -- 172.21.15.119:0/1610330584 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428011830 con 0x7f04400d0c20 2024-09-13T16:36:41.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043effd700 1 -- 172.21.15.119:0/1610330584 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0428011990 con 0x7f04400d0c20 2024-09-13T16:36:41.699 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f043effd700 1 -- 172.21.15.119:0/1610330584 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428011d40 con 0x7f04400d0c20 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400d0c20 msgr2=0x7f04400d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 --2- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400d0c20 0x7f04400d10d0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f0428009850 tx=0x7f0428009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- 172.21.15.119:0/1610330584 shutdown_connections 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 --2- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400d0c20 0x7f04400d10d0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 --2- 172.21.15.119:0/1610330584 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f044015cc00 0x7f04400d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- 172.21.15.119:0/1610330584 >> 172.21.15.119:0/1610330584 conn(0x7f044008c080 msgr2=0x7f04400d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- 172.21.15.119:0/1610330584 shutdown_connections 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.693+0000 7f04449ab700 1 -- 172.21.15.119:0/1610330584 wait complete. 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 Processor -- start 2024-09-13T16:36:41.700 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 -- start start 2024-09-13T16:36:41.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400ce8c0 0x7f0440155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.702 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04400d0c20 0x7f044006bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f044015cc00 0x7f044006c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440156ac0 con 0x7f044015cc00 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0440156da0 con 0x7f04400ce8c0 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0440157080 con 0x7f04400d0c20 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f044015cc00 0x7f044006c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f044015cc00 0x7f044006c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46320/0 (socket says 172.21.15.119:46320) 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 -- 172.21.15.119:0/4264493241 learned_addr learned my addr 172.21.15.119:0/4264493241 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f0436ffd700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04400d0c20 0x7f044006bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043f7fe700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400ce8c0 0x7f0440155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 -- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04400d0c20 msgr2=0x7f044006bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04400d0c20 0x7f044006bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 -- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400ce8c0 msgr2=0x7f0440155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400ce8c0 0x7f0440155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 -- 172.21.15.119:0/4264493241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04280094b0 con 0x7f044015cc00 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f0436ffd700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04400d0c20 0x7f044006bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043ffff700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f044015cc00 0x7f044006c180 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f043800caa0 tx=0x7f043800cb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043cff9700 1 -- 172.21.15.119:0/4264493241 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043800a7b0 con 0x7f044015cc00 2024-09-13T16:36:41.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043cff9700 1 -- 172.21.15.119:0/4264493241 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043800a910 con 0x7f044015cc00 2024-09-13T16:36:41.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f043cff9700 1 -- 172.21.15.119:0/4264493241 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043800aca0 con 0x7f044015cc00 2024-09-13T16:36:41.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0440157360 con 0x7f044015cc00 2024-09-13T16:36:41.705 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.697+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0440157880 con 0x7f044015cc00 2024-09-13T16:36:41.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.701+0000 7f043cff9700 1 -- 172.21.15.119:0/4264493241 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f043801b3b0 con 0x7f044015cc00 2024-09-13T16:36:41.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.701+0000 7f043cff9700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0420066760 0x7f0420068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.701+0000 7f043cff9700 1 -- 172.21.15.119:0/4264493241 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f0438083d40 con 0x7f044015cc00 2024-09-13T16:36:41.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.701+0000 7f043f7fe700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0420066760 0x7f0420068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.701+0000 7f043f7fe700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0420066760 0x7f0420068c20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f04400b7ad0 tx=0x7f0430008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.701+0000 7f0448ebe740 1 Processor -- start 2024-09-13T16:36:41.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.701+0000 7f0448ebe740 1 -- start start 2024-09-13T16:36:41.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x1872810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1871310 0x189a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18bc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1803010 con 0x1871310 2024-09-13T16:36:41.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1790f40 con 0x18bc170 2024-09-13T16:36:41.709 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x17e3e30 con 0x18bbd90 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1871310 0x189a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1871310 0x189a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46328/0 (socket says 172.21.15.119:46328) 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f043f7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x1872810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 -- 172.21.15.119:0/1664356091 learned_addr learned my addr 172.21.15.119:0/1664356091 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f043ffff700 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18bc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 -- 172.21.15.119:0/1664356091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 msgr2=0x1872810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x1872810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 -- 172.21.15.119:0/1664356091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 msgr2=0x18bc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18bc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 -- 172.21.15.119:0/1664356091 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x164fa40 con 0x1871310 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0436ffd700 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1871310 0x189a520 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f04280090b0 tx=0x7f0428010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3aeb2f99618d23e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.710 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f04357fa700 1 -- 172.21.15.119:0/1664356091 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428011d00 con 0x1871310 2024-09-13T16:36:41.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f04357fa700 1 -- 172.21.15.119:0/1664356091 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0428004040 con 0x1871310 2024-09-13T16:36:41.711 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f04357fa700 1 -- 172.21.15.119:0/1664356091 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04280043f0 con 0x1871310 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- 172.21.15.119:0/1664356091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1871310 msgr2=0x189a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1871310 0x189a520 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f04280090b0 tx=0x7f0428010000 comp rx=0 tx=0).stop 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- 172.21.15.119:0/1664356091 shutdown_connections 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18bc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1871310 0x189a520 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 --2- 172.21.15.119:0/1664356091 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x1872810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- 172.21.15.119:0/1664356091 >> 172.21.15.119:0/1664356091 conn(0x16045b0 msgr2=0x1875b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- 172.21.15.119:0/1664356091 shutdown_connections 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.705+0000 7f0448ebe740 1 -- 172.21.15.119:0/1664356091 wait complete. 2024-09-13T16:36:41.712 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 -- start start 2024-09-13T16:36:41.713 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 Processor -- start 2024-09-13T16:36:41.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1871310 0x1895210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x18bc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc170 0x1899030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x17df150 con 0x18bc170 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x17dceb0 con 0x1871310 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x179a8e0 con 0x18bbd90 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 10 client.0 _ll_register_callbacks cb 0x1779670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc170 0x1899030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc170 0x1899030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46336/0 (socket says 172.21.15.119:46336) 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 -- 172.21.15.119:0/2261818286 learned_addr learned my addr 172.21.15.119:0/2261818286 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0436ffd700 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x18bc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043f7fe700 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1871310 0x1895210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 -- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 msgr2=0x18bc550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x18bc550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 -- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1871310 msgr2=0x1895210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1871310 0x1895210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 -- 172.21.15.119:0/2261818286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04400ce3a0 con 0x18bc170 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f043ffff700 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc170 0x1899030 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f043800cf80 tx=0x7f0438055100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3c3b3cb1bb7fd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f041e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f041e7fc700 1 -- 172.21.15.119:0/2261818286 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04380589f0 con 0x18bc170 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f041e7fc700 1 -- 172.21.15.119:0/2261818286 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f043805cfb0 con 0x18bc170 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x16112c0 con 0x18bc170 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f041e7fc700 1 -- 172.21.15.119:0/2261818286 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0438054700 con 0x18bc170 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 10 client.10071 fetch_fsmap learned FSMap version 98 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x17a3290 con 0x18bc170 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f041e7fc700 1 -- 172.21.15.119:0/2261818286 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438070070 con 0x18bc170 2024-09-13T16:36:41.717 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f041e7fc700 1 -- 172.21.15.119:0/2261818286 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 98) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0438055bd0 con 0x18bc170 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 10 client.10071 fetch_fsmap finished waiting for FSMap version 98 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 10 client.10071 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 20 client.10071 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x1896f60 0x18937f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 4 client.10071 mds_command: new command op to 8264 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.709+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 --> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x165b540 con 0x1896f60 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f043f7fe700 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x1896f60 0x18937f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f043f7fe700 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x1896f60 0x18937f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3678ffe7aa233fe server_cookie=cdef61e2864c96b3 in_seq=0 out_seq=0 2024-09-13T16:36:41.718 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f041e7fc700 10 client.10071 ms_handle_connect on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:41.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f041e7fc700 1 -- 172.21.15.119:0/2261818286 <== mds.0 v2:172.21.15.145:6834/3898509964 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x165b540 con 0x1896f60 2024-09-13T16:36:41.719 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f041e7fc700 10 client.10071 handle_command_reply: tid=0 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 client.10071 shutdown 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc170 msgr2=0x1899030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc170 0x1899030 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f043800cf80 tx=0x7f0438055100 comp rx=0 tx=0).stop 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 shutdown_connections 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x1896f60 0x18937f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bc170 0x1899030 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x18bbd90 0x18bc550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2261818286 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1871310 0x1895210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f041e7fc700 0 client.10071 ms_handle_reset on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 >> 172.21.15.119:0/2261818286 conn(0x16045b0 msgr2=0x189a4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 shutdown_connections 2024-09-13T16:36:41.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 1 -- 172.21.15.119:0/2261818286 wait complete. 2024-09-13T16:36:41.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.713+0000 7f0448ebe740 20 client.10071 trim_cache size 0 max 16384 2024-09-13T16:36:41.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 Processor -- start 2024-09-13T16:36:41.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- start start 2024-09-13T16:36:41.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x1913950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x1875750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x1875c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1688b80 con 0x18bbd90 2024-09-13T16:36:41.733 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x17dbb60 con 0x18bc170 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x179d260 con 0x1871310 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x1875750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x1875750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46344/0 (socket says 172.21.15.119:46344) 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 -- 172.21.15.119:0/2493600689 learned_addr learned my addr 172.21.15.119:0/2493600689 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f043ffff700 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x1875c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f043f7fe700 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x1913950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 -- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 msgr2=0x1913950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x1913950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 -- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 msgr2=0x1875c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x1875c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 -- 172.21.15.119:0/2493600689 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x17894c0 con 0x18bbd90 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f043f7fe700 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x1913950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:41.734 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0436ffd700 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x1875750 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0428009eb0 tx=0x7f0428011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff49e2d09a87b17 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0435ffb700 1 -- 172.21.15.119:0/2493600689 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0428011d00 con 0x18bbd90 2024-09-13T16:36:41.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0435ffb700 1 -- 172.21.15.119:0/2493600689 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0428009330 con 0x18bbd90 2024-09-13T16:36:41.735 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0435ffb700 1 -- 172.21.15.119:0/2493600689 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04280043d0 con 0x18bbd90 2024-09-13T16:36:41.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- 172.21.15.119:0/2493600689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 msgr2=0x1875750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x1875750 secure :-1 s=READY pgs=1939 cs=0 l=1 rev1=1 crypto rx=0x7f0428009eb0 tx=0x7f0428011000 comp rx=0 tx=0).stop 2024-09-13T16:36:41.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- 172.21.15.119:0/2493600689 shutdown_connections 2024-09-13T16:36:41.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x1875c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x1875750 unknown :-1 s=CLOSED pgs=1939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.738 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 --2- 172.21.15.119:0/2493600689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x1913950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- 172.21.15.119:0/2493600689 >> 172.21.15.119:0/2493600689 conn(0x16045b0 msgr2=0x1895120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- 172.21.15.119:0/2493600689 shutdown_connections 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.729+0000 7f0448ebe740 1 -- 172.21.15.119:0/2493600689 wait complete. 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 -- start start 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 Processor -- start 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x18948b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x18d26f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18d18f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x1894df0 con 0x18bbd90 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x18d30d0 con 0x18bc170 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x18b99f0 con 0x1871310 2024-09-13T16:36:41.739 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 10 client.0 _ll_register_callbacks cb 0x17db610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x18d26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x18d26f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46352/0 (socket says 172.21.15.119:46352) 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 -- 172.21.15.119:0/3133452106 learned_addr learned my addr 172.21.15.119:0/3133452106 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f043f7fe700 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x18948b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f043ffff700 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18d18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 -- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 msgr2=0x18948b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x18948b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 -- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 msgr2=0x18d18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18d18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 -- 172.21.15.119:0/3133452106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x164fa40 con 0x18bbd90 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0436ffd700 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x18d26f0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f0428011960 tx=0x7f04280047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4570898728e923d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.740 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f041e7fc700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f041e7fc700 1 -- 172.21.15.119:0/3133452106 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04280036c0 con 0x18bbd90 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f041e7fc700 1 -- 172.21.15.119:0/3133452106 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7f0428003820 con 0x18bbd90 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x16802c0 con 0x18bbd90 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f041e7fc700 1 -- 172.21.15.119:0/3133452106 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0428003a40 con 0x18bbd90 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 10 client.10077 fetch_fsmap learned FSMap version 98 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x177d790 con 0x18bbd90 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f041e7fc700 1 -- 172.21.15.119:0/3133452106 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x16802c0 con 0x18bbd90 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f041e7fc700 1 -- 172.21.15.119:0/3133452106 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 98) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0428003ba0 con 0x18bbd90 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 10 client.10077 fetch_fsmap finished waiting for FSMap version 98 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 10 client.10077 resolve_mds: resolved 8:0 to role '8:0' aka daemon mds.b 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 20 client.10077 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x18d2c30 0x1912410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.741 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 4 client.10077 mds_command: new command op to 8264 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-09-13T16:36:41.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 --> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x18d1e30 con 0x18d2c30 2024-09-13T16:36:41.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f043f7fe700 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x18d2c30 0x1912410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f043f7fe700 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x18d2c30 0x1912410 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8db1fee50ae53a15 server_cookie=dc912fe60d808dc in_seq=0 out_seq=0 2024-09-13T16:36:41.742 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.733+0000 7f041e7fc700 10 client.10077 ms_handle_connect on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:41.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f041e7fc700 1 -- 172.21.15.119:0/3133452106 <== mds.0 v2:172.21.15.145:6834/3898509964 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x18d1e30 con 0x18d2c30 2024-09-13T16:36:41.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f041e7fc700 10 client.10077 handle_command_reply: tid=0 2024-09-13T16:36:41.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 client.10077 shutdown 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 msgr2=0x18d26f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x18d26f0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f0428011960 tx=0x7f04280047e0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 shutdown_connections 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x18bc170 0x18d18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:6834/3898509964,v1:172.21.15.145:6835/3898509964] conn(0x18d2c30 0x1912410 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x18bbd90 0x18d26f0 unknown :-1 s=CLOSED pgs=1940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f041e7fc700 0 client.10077 ms_handle_reset on v2:172.21.15.145:6834/3898509964 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 --2- 172.21.15.119:0/3133452106 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1871310 0x18948b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 >> 172.21.15.119:0/3133452106 conn(0x16045b0 msgr2=0x18b8bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 shutdown_connections 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 1 -- 172.21.15.119:0/3133452106 wait complete. 2024-09-13T16:36:41.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f0448ebe740 20 client.10077 trim_cache size 0 max 16384 2024-09-13T16:36:41.759 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:41.759 INFO:teuthology.orchestra.run.smithi119.stdout: "message": "", 2024-09-13T16:36:41.759 INFO:teuthology.orchestra.run.smithi119.stdout: "return_code": 0 2024-09-13T16:36:41.759 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0420066760 msgr2=0x7f0420068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0420066760 0x7f0420068c20 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f04400b7ad0 tx=0x7f0430008040 comp rx=0 tx=0).stop 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f044015cc00 msgr2=0x7f044006c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f044015cc00 0x7f044006c180 secure :-1 s=READY pgs=1936 cs=0 l=1 rev1=1 crypto rx=0x7f043800caa0 tx=0x7f043800cb80 comp rx=0 tx=0).stop 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 shutdown_connections 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0420066760 0x7f0420068c20 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f044015cc00 0x7f044006c180 unknown :-1 s=CLOSED pgs=1936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f04400d0c20 0x7f044006bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 --2- 172.21.15.119:0/4264493241 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f04400ce8c0 0x7f0440155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 >> 172.21.15.119:0/4264493241 conn(0x7f044008c080 msgr2=0x7f04400b82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 shutdown_connections 2024-09-13T16:36:41.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.753+0000 7f04449ab700 1 -- 172.21.15.119:0/4264493241 wait complete. 2024-09-13T16:36:41.771 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:36:41.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 Processor -- start 2024-09-13T16:36:41.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- start start 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc950144180 0x7fc95006cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95006fcc0 con 0x7fc9500ce040 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc95006d590 con 0x7fc9500cbe70 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc95006de30 con 0x7fc950144180 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94effd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc950144180 0x7fc95006cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc9500ce040 0x7fc950070150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:56144/0 (socket says 172.21.15.119:56144) 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94f7fe700 1 -- 172.21.15.119:0/2232943869 learned_addr learned my addr 172.21.15.119:0/2232943869 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:41.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94effd700 1 -- 172.21.15.119:0/2232943869 >> v1:172.21.15.145:6789/0 conn(0x7fc9500cbe70 legacy=0x7fc9500cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94effd700 1 -- 172.21.15.119:0/2232943869 >> v1:172.21.15.119:6789/0 conn(0x7fc9500ce040 legacy=0x7fc950070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94effd700 1 -- 172.21.15.119:0/2232943869 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc95006d870 con 0x7fc950144180 2024-09-13T16:36:41.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94dffb700 1 -- 172.21.15.119:0/2232943869 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673481645 0 0) 0x7fc95006fcc0 con 0x7fc9500ce040 2024-09-13T16:36:41.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94effd700 1 --2- 172.21.15.119:0/2232943869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc95006cf70 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc940009850 tx=0x7fc940009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e88acc3d3ea6014c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94dffb700 1 -- 172.21.15.119:0/2232943869 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940011830 con 0x7fc950144180 2024-09-13T16:36:41.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94dffb700 1 -- 172.21.15.119:0/2232943869 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc940011990 con 0x7fc950144180 2024-09-13T16:36:41.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94dffb700 1 -- 172.21.15.119:0/2232943869 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc940011d20 con 0x7fc950144180 2024-09-13T16:36:41.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- 172.21.15.119:0/2232943869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 msgr2=0x7fc95006cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2232943869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc95006cf70 secure :-1 s=READY pgs=1942 cs=0 l=1 rev1=1 crypto rx=0x7fc940009850 tx=0x7fc940009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:41.887 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- 172.21.15.119:0/2232943869 shutdown_connections 2024-09-13T16:36:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2232943869 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc95006cf70 unknown :-1 s=CLOSED pgs=1942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- 172.21.15.119:0/2232943869 >> 172.21.15.119:0/2232943869 conn(0x7fc95006a350 msgr2=0x7fc95006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- 172.21.15.119:0/2232943869 shutdown_connections 2024-09-13T16:36:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.881+0000 7fc94ffff700 1 -- 172.21.15.119:0/2232943869 wait complete. 2024-09-13T16:36:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 Processor -- start 2024-09-13T16:36:41.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 -- start start 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9500cbe70 0x7fc95015a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9500ce040 0x7fc9500ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc9501f0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500abb90 con 0x7fc950144180 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc95015a670 con 0x7fc9500cbe70 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9500aaf80 con 0x7fc9500ce040 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc9501f0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc9501f0510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46372/0 (socket says 172.21.15.119:46372) 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 -- 172.21.15.119:0/2721958135 learned_addr learned my addr 172.21.15.119:0/2721958135 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94e7fc700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9500ce040 0x7fc9500ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94effd700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9500cbe70 0x7fc95015a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 -- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9500ce040 msgr2=0x7fc9500ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9500ce040 0x7fc9500ac020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 -- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9500cbe70 msgr2=0x7fc95015a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9500cbe70 0x7fc95015a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 -- 172.21.15.119:0/2721958135 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400094b0 con 0x7fc950144180 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94f7fe700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc9501f0510 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc94800ccd0 tx=0x7fc94800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc9377fe700 1 -- 172.21.15.119:0/2721958135 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc94800ab30 con 0x7fc950144180 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc9377fe700 1 -- 172.21.15.119:0/2721958135 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc94800ac90 con 0x7fc950144180 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9500ab260 con 0x7fc950144180 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc9377fe700 1 -- 172.21.15.119:0/2721958135 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480070d0 con 0x7fc950144180 2024-09-13T16:36:41.891 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.885+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9500ab820 con 0x7fc950144180 2024-09-13T16:36:41.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.889+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9500abd40 con 0x7fc950144180 2024-09-13T16:36:41.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.889+0000 7fc9377fe700 1 -- 172.21.15.119:0/2721958135 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc948007230 con 0x7fc950144180 2024-09-13T16:36:41.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.889+0000 7fc9377fe700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc93c066730 0x7fc93c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:41.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.889+0000 7fc9377fe700 1 -- 172.21.15.119:0/2721958135 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6115+0+0 (secure 0 0 0) 0x7fc9480841a0 con 0x7fc950144180 2024-09-13T16:36:41.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.889+0000 7fc94effd700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc93c066730 0x7fc93c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:41.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.889+0000 7fc94effd700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc93c066730 0x7fc93c068bf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc9400090b0 tx=0x7fc94001c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:41.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:41.893+0000 7fc9377fe700 1 -- 172.21.15.119:0/2721958135 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc9500abd40 con 0x7fc950144180 2024-09-13T16:36:42.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.097+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fc9500abd40 con 0x7fc950144180 2024-09-13T16:36:42.102 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:36:42.097+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:36:42.119 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc9377fe700 1 -- 172.21.15.119:0/2721958135 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v99) v1 ==== 161+0+0 (secure 0 0 0) 0x7fc9500abd40 con 0x7fc950144180 2024-09-13T16:36:42.122 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc93c066730 msgr2=0x7fc93c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc93c066730 0x7fc93c068bf0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7fc9400090b0 tx=0x7fc94001c040 comp rx=0 tx=0).stop 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 msgr2=0x7fc9501f0510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc9501f0510 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7fc94800ccd0 tx=0x7fc94800a060 comp rx=0 tx=0).stop 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 shutdown_connections 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc93c066730 0x7fc93c068bf0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc950144180 0x7fc9501f0510 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc9500ce040 0x7fc9500ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 --2- 172.21.15.119:0/2721958135 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc9500cbe70 0x7fc95015a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.117+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 >> 172.21.15.119:0/2721958135 conn(0x7fc95006a350 msgr2=0x7fc95015a850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.121+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 shutdown_connections 2024-09-13T16:36:42.123 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.121+0000 7fc94ffff700 1 -- 172.21.15.119:0/2721958135 wait complete. 2024-09-13T16:36:42.124 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:36:42.124 INFO:tasks.ceph.mds.b.smithi145.stderr: -2293> 2024-09-13T16:36:27.661+0000 7fdde7ae0700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:36:42.137 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-journal-tool --debug-mds=20 --debug-ms=1 --debug-objecter=1 --rank cephfs:0 journal reset --force --yes-i-really-really-mean-it 2024-09-13T16:36:42.157 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:36:42.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 -- start start 2024-09-13T16:36:42.176 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 Processor -- start 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f17029400 0x557f170ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f17029800 0x557f170cf080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f17029c00 0x557f170cf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f1705fe00 con 0x557f17029800 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f170aaa00 con 0x557f17029400 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa780 con 0x557f17029c00 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f17029800 0x557f170cf080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f17029800 0x557f170cf080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46386/0 (socket says 172.21.15.119:46386) 2024-09-13T16:36:42.177 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 -- 172.21.15.119:0/1361302265 learned_addr learned my addr 172.21.15.119:0/1361302265 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36b63700 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f17029400 0x557f170ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.178 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa37364700 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f17029c00 0x557f170cf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 -- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f17029c00 msgr2=0x557f170cf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f17029c00 0x557f170cf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.180 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 -- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f17029400 msgr2=0x557f170ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f17029400 0x557f170ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 -- 172.21.15.119:0/1361302265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f1704c9c0 con 0x557f17029800 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa36362700 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f17029800 0x557f170cf080 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x557f17131410 tx=0x557f17eac120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d67194601bde91c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa3435e700 1 -- 172.21.15.119:0/1361302265 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f17079600 con 0x557f17029800 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa3435e700 1 -- 172.21.15.119:0/1361302265 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557f17074480 con 0x557f17029800 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 -- 172.21.15.119:0/1361302265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f1704cb60 con 0x557f17029800 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 1 -- 172.21.15.119:0/1361302265 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x557f1704c340 con 0x557f17029800 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 4 waiting for MDS map... 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa3435e700 1 -- 172.21.15.119:0/1361302265 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f170791e0 con 0x557f17029800 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa3435e700 1 -- 172.21.15.119:0/1361302265 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x557f17062c40 con 0x557f17029800 2024-09-13T16:36:42.181 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa3435e700 1 -- 172.21.15.119:0/1361302265 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 99) v1 ==== 2573+0+0 (secure 0 0 0) 0x557f17eb0900 con 0x557f17029800 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.173+0000 7faa383a9500 4 Got MDS map 99 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 10 main: JournalTool::main 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 4 main: JournalTool: connecting to RADOS... 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 Processor -- start 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- start start 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731a800 0x557f170cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731ac00 0x557f170d0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa640 con 0x557f1731b000 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa500 con 0x557f1731a800 2024-09-13T16:36:42.182 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f170aaa00 con 0x557f1731ac00 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46396/0 (socket says 172.21.15.119:46396) 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 -- 172.21.15.119:0/3757464187 learned_addr learned my addr 172.21.15.119:0/3757464187 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 -- 172.21.15.119:0/3757464187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731ac00 msgr2=0x557f170d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 --2- 172.21.15.119:0/3757464187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731ac00 0x557f170d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 -- 172.21.15.119:0/3757464187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731a800 msgr2=0x557f170cfb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 --2- 172.21.15.119:0/3757464187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731a800 0x557f170cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 -- 172.21.15.119:0/3757464187 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f1704c000 con 0x557f1731b000 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa36362700 1 --2- 172.21.15.119:0/3757464187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731b000 0x557f170d0680 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x557f17131a10 tx=0x557f17eac060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef90469893d83973 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa3235a700 1 -- 172.21.15.119:0/3757464187 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f17078f20 con 0x557f1731b000 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa3235a700 1 -- 172.21.15.119:0/3757464187 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557f17e10780 con 0x557f1731b000 2024-09-13T16:36:42.183 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa3235a700 1 -- 172.21.15.119:0/3757464187 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f17079600 con 0x557f1731b000 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- 172.21.15.119:0/3757464187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731b000 msgr2=0x557f170d0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 --2- 172.21.15.119:0/3757464187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731b000 0x557f170d0680 secure :-1 s=READY pgs=1946 cs=0 l=1 rev1=1 crypto rx=0x557f17131a10 tx=0x557f17eac060 comp rx=0 tx=0).stop 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- 172.21.15.119:0/3757464187 shutdown_connections 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 --2- 172.21.15.119:0/3757464187 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=CLOSED pgs=1946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 --2- 172.21.15.119:0/3757464187 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731ac00 0x557f170d0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 --2- 172.21.15.119:0/3757464187 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731a800 0x557f170cfb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- 172.21.15.119:0/3757464187 >> 172.21.15.119:0/3757464187 conn(0x557f1731a400 msgr2=0x557f170e5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- 172.21.15.119:0/3757464187 shutdown_connections 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- 172.21.15.119:0/3757464187 wait complete. 2024-09-13T16:36:42.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 Processor -- start 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.177+0000 7faa383a9500 1 -- start start 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731a800 0x557f170cfb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731ac00 0x557f170d0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa3c0 con 0x557f1731a800 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa280 con 0x557f1731ac00 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa140 con 0x557f1731b000 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731a800 0x557f170cfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731a800 0x557f170cfb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46400/0 (socket says 172.21.15.119:46400) 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 -- 172.21.15.119:0/2835438083 learned_addr learned my addr 172.21.15.119:0/2835438083 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa36362700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa36b63700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731ac00 0x557f170d0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 -- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731b000 msgr2=0x557f170d0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 -- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731ac00 msgr2=0x557f170d0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731ac00 0x557f170d0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 -- 172.21.15.119:0/2835438083 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f1704c9c0 con 0x557f1731a800 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa37364700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731a800 0x557f170cfb80 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x557f1732f650 tx=0x557f170ef6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa30b57700 1 -- 172.21.15.119:0/2835438083 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f17078f20 con 0x557f1731a800 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa30b57700 1 -- 172.21.15.119:0/2835438083 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557f17e11b00 con 0x557f1731a800 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x557f1704cb60 con 0x557f1731a800 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa30b57700 1 -- 172.21.15.119:0/2835438083 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f17079600 con 0x557f1731a800 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f1704c340 con 0x557f1731a800 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa383a9500 4 main: JournalTool: resolving pool 17 2024-09-13T16:36:42.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.181+0000 7faa30b57700 1 -- 172.21.15.119:0/2835438083 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x557f17088b00 con 0x557f1731a800 2024-09-13T16:36:42.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa30b57700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x557f1731b800 0x557f170d1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa30b57700 1 -- 172.21.15.119:0/2835438083 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x557f17062a80 con 0x557f1731a800 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x557f1731b800 0x557f170d1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 4 main: JournalTool: creating IoCtx.. 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 4 main: Executing for rank 0 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 -- start start 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 Processor -- start 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x557f1731b800 0x557f170d1700 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x557f171311d0 tx=0x557f170ff860 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f174d4000 0x557f170d0c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f174d4400 0x557f170d1180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f174d4800 0x557f174de000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa000 con 0x557f174d4400 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa140 con 0x557f174d4000 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x557f170aa280 con 0x557f174d4800 2024-09-13T16:36:42.193 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f174d4400 0x557f170d1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36362700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f174d4800 0x557f174de000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f174d4400 0x557f170d1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46404/0 (socket says 172.21.15.119:46404) 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa37364700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f174d4000 0x557f170d0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 -- 172.21.15.119:0/1627946627 learned_addr learned my addr 172.21.15.119:0/1627946627 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f174d4800 msgr2=0x557f174de000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f174d4800 0x557f174de000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f174d4000 msgr2=0x557f170d0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f174d4000 0x557f170d0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557f173d3a00 con 0x557f174d4400 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa36b63700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f174d4400 0x557f170d1180 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x557f173cc540 tx=0x557f170fe180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90666653a3f8ae4c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa2db51700 1 -- 172.21.15.119:0/1627946627 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f17079080 con 0x557f174d4400 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa2db51700 1 -- 172.21.15.119:0/1627946627 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x557f17e10180 con 0x557f174d4400 2024-09-13T16:36:42.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa2db51700 1 -- 172.21.15.119:0/1627946627 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x557f170786e0 con 0x557f174d4400 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557f173d3860 con 0x557f174d4400 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x557f173d36c0 con 0x557f174d4400 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 4 waiting for MDS map... 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa2db51700 1 -- 172.21.15.119:0/1627946627 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x557f170628c0 con 0x557f174d4400 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa2db51700 1 -- 172.21.15.119:0/1627946627 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 99) v1 ==== 2573+0+0 (secure 0 0 0) 0x557f174e2300 con 0x557f174d4400 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 4 Got MDS map 99 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.185+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x557f174d4c00 0x557f174de580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.189+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:1 17.f 17:f1f69726:::400.00000000:head [writefull 0~22 in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f174d5000 con 0x557f174d4c00 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.189+0000 7faa37364700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x557f174d4c00 0x557f174de580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.189+0000 7faa37364700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x557f174d4c00 0x557f174de580 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=594dedddb140be60 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.189+0000 7faa37364700 1 -- 172.21.15.119:0/1627946627 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(1 400.00000000 [writefull 0~22] v128'2 uv2 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0240 con 0x557f174d4c00 2024-09-13T16:36:42.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.189+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x557f174d5400 0x557f174deb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.189+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:2 17.14 17:292cf221:::200.00000000:head [writefull 0~90 in=90b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f174d5c00 con 0x557f174d5400 2024-09-13T16:36:42.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.189+0000 7faa36362700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x557f174d5400 0x557f174deb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.193+0000 7faa36362700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x557f174d5400 0x557f174deb00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=255666de019e832 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== osd_op_reply(2 200.00000000 [writefull 0~90] v128'40 uv40 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0000 con 0x557f174d5400 2024-09-13T16:36:42.201 INFO:teuthology.orchestra.run.smithi119.stdout:writing EResetJournal entry 2024-09-13T16:36:42.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:3 17.14 17:2e2fa760:::200.00000001:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f17500000 con 0x557f174d5400 2024-09-13T16:36:42.202 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x557f17500400 0x557f174df080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:4 17.1b 17:ddb4e4d7:::200.00000002:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f17500800 con 0x557f17500400 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x557f17500c00 0x557f174df600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:5 17.b 17:d5c7a900:::200.00000003:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f17501000 con 0x557f17500c00 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x557f17501400 0x557f174dfb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:6 17.1f 17:f83d65d8:::200.00000004:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f17501800 con 0x557f17501400 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:7 17.1a 17:5e41ab69:::200.00000005:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f17501c00 con 0x557f17500c00 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:8 17.1e 17:7bf7e7b5:::200.00000006:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f1750a000 con 0x557f174d4c00 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa36362700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x557f17501400 0x557f174dfb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa37364700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x557f17500400 0x557f174df080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa36b63700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x557f17500c00 0x557f174df600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa36362700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x557f17501400 0x557f174dfb80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=159b214f20e365a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.197+0000 7faa36b63700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x557f17500c00 0x557f174df600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=46bda5c27ea5c0e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa37364700 1 -- 172.21.15.119:0/1627946627 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== osd_op_reply(8 200.00000006 [delete] v128'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0240 con 0x557f174d4c00 2024-09-13T16:36:42.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa37364700 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x557f17500400 0x557f174df080 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=9d7bbb45d69b6151 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== osd_op_reply(3 200.00000001 [delete] v128'41 uv41 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0000 con 0x557f174d5400 2024-09-13T16:36:42.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa36b63700 1 -- 172.21.15.119:0/1627946627 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(5 200.00000003 [delete] v128'6 uv5 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0fc0 con 0x557f17500c00 2024-09-13T16:36:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa36b63700 1 -- 172.21.15.119:0/1627946627 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(7 200.00000005 [delete] v128'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0fc0 con 0x557f17500c00 2024-09-13T16:36:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(6 200.00000004 [delete] v128'9 uv8 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0000 con 0x557f17501400 2024-09-13T16:36:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa37364700 1 -- 172.21.15.119:0/1627946627 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(4 200.00000002 [delete] v128'3 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0240 con 0x557f17500400 2024-09-13T16:36:42.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.201+0000 7faa2c34e700 1 -- 172.21.15.119:0/1627946627 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:9 17.14 17:2e2fa760:::200.00000001:head [write 0~48 in=48b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x557f1750a000 con 0x557f174d5400 2024-09-13T16:36:42.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 <== osd.3 v2:172.21.15.119:6808/3803418857 3 ==== osd_op_reply(9 200.00000001 [write 0~48] v128'42 uv42 ondisk = 0) v8 ==== 156+0+0 (crc 0 0 0) 0x557f17ea0000 con 0x557f174d5400 2024-09-13T16:36:42.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 4 Successfully wrote new journal pointer and header for rank 8:0 2024-09-13T16:36:42.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x557f17500c00 msgr2=0x557f174df600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x557f17500c00 0x557f174df600 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x557f174d4c00 msgr2=0x557f174de580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x557f174d4c00 0x557f174de580 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x557f174d5400 msgr2=0x557f174deb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x557f174d5400 0x557f174deb00 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x557f17500400 msgr2=0x557f174df080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x557f17500400 0x557f174df080 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x557f17501400 msgr2=0x557f174dfb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x557f17501400 0x557f174dfb80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f174d4400 msgr2=0x557f170d1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1627946627 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f174d4400 0x557f170d1180 secure :-1 s=READY pgs=1948 cs=0 l=1 rev1=1 crypto rx=0x557f173cc540 tx=0x557f170fe180 comp rx=0 tx=0).stop 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 reap_dead start 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 reap_dead start 2024-09-13T16:36:42.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 reap_dead start 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa36362700 1 -- 172.21.15.119:0/1627946627 reap_dead start 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 shutdown_connections 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 >> 172.21.15.119:0/1627946627 conn(0x557f1731bc00 msgr2=0x557f170e5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 shutdown_connections 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1627946627 wait complete. 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x557f1731b800 msgr2=0x557f170d1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x557f1731b800 0x557f170d1700 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x557f171311d0 tx=0x557f170ff860 comp rx=0 tx=0).stop 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731a800 msgr2=0x557f170cfb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731a800 0x557f170cfb80 secure :-1 s=READY pgs=1947 cs=0 l=1 rev1=1 crypto rx=0x557f1732f650 tx=0x557f170ef6e0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 shutdown_connections 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x557f1731b800 0x557f170d1700 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f1731b000 0x557f170d0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f1731ac00 0x557f170d0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/2835438083 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f1731a800 0x557f170cfb80 unknown :-1 s=CLOSED pgs=1947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 >> 172.21.15.119:0/2835438083 conn(0x557f1731a400 msgr2=0x557f170e6000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 shutdown_connections 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/2835438083 wait complete. 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1361302265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f17029800 msgr2=0x557f170cf080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f17029800 0x557f170cf080 secure :-1 s=READY pgs=1945 cs=0 l=1 rev1=1 crypto rx=0x557f17131410 tx=0x557f17eac120 comp rx=0 tx=0).stop 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1361302265 shutdown_connections 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x557f17029c00 0x557f170cf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x557f17029800 0x557f170cf080 unknown :-1 s=CLOSED pgs=1945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 --2- 172.21.15.119:0/1361302265 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x557f17029400 0x557f170ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1361302265 >> 172.21.15.119:0/1361302265 conn(0x557f17028c00 msgr2=0x557f170e5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1361302265 shutdown_connections 2024-09-13T16:36:42.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.205+0000 7faa383a9500 1 -- 172.21.15.119:0/1361302265 wait complete. 2024-09-13T16:36:42.214 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-journal-tool', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', '--rank', 'cephfs:0', 'journal', 'reset', '--force', '--yes-i-really-really-mean-it'] in time 0:00:00.077365, result: writing EResetJournal entry 2024-09-13T16:36:42.215 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_extents cephfs_data 2024-09-13T16:36:42.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.245+0000 7f6cbad8f9c0 1 -- start start 2024-09-13T16:36:42.250 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.245+0000 7f6cbad8f9c0 1 Processor -- start 2024-09-13T16:36:42.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53003d800 0x55a5300e2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.251 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53003d400 0x55a5300e2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53003d000 0x55a5300e3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5300bea00 con 0x55a53003d800 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5300be780 con 0x55a53003d400 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5300be640 con 0x55a53003d000 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53003d800 0x55a5300e2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9d4a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53003d000 0x55a5300e3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53003d800 0x55a5300e2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46410/0 (socket says 172.21.15.119:46410) 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 -- 172.21.15.119:0/724559992 learned_addr learned my addr 172.21.15.119:0/724559992 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53003d400 0x55a5300e2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 -- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53003d000 msgr2=0x55a5300e3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53003d000 0x55a5300e3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 -- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53003d400 msgr2=0x55a5300e2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.252 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53003d400 0x55a5300e2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 -- 172.21.15.119:0/724559992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5300609c0 con 0x55a53003d800 2024-09-13T16:36:42.253 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53003d400 0x55a5300e2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:42.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb9549700 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53003d800 0x55a5300e2580 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x55a5300fd830 tx=0x55a530e26060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ecff40c32010d80 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb6d44700 1 -- 172.21.15.119:0/724559992 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a53008d080 con 0x55a53003d800 2024-09-13T16:36:42.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb6d44700 1 -- 172.21.15.119:0/724559992 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a530e28480 con 0x55a53003d800 2024-09-13T16:36:42.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/724559992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a5300601a0 con 0x55a53003d800 2024-09-13T16:36:42.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/724559992 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55a5300609c0 con 0x55a53003d800 2024-09-13T16:36:42.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 4 waiting for MDS map... 2024-09-13T16:36:42.254 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb6d44700 1 -- 172.21.15.119:0/724559992 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a53008d600 con 0x55a53003d800 2024-09-13T16:36:42.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb6d44700 1 -- 172.21.15.119:0/724559992 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x55a530076c40 con 0x55a53003d800 2024-09-13T16:36:42.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cb6d44700 1 -- 172.21.15.119:0/724559992 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 99) v1 ==== 2573+0+0 (secure 0 0 0) 0x55a53007ef00 con 0x55a53003d800 2024-09-13T16:36:42.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 4 Got MDS map 99 2024-09-13T16:36:42.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 4 datascan.main: Using metadata pool output 2024-09-13T16:36:42.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 4 datascan.main: connecting to RADOS... 2024-09-13T16:36:42.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 Processor -- start 2024-09-13T16:36:42.255 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.249+0000 7f6cbad8f9c0 1 -- start start 2024-09-13T16:36:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032c800 0x55a5300e3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032cc00 0x55a5300e3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032d000 0x55a5300e4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a530073e00 con 0x55a53032cc00 2024-09-13T16:36:42.257 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5300be780 con 0x55a53032c800 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5300bea00 con 0x55a53032d000 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032cc00 0x55a5300e3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9d4a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032c800 0x55a5300e3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032cc00 0x55a5300e3b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46420/0 (socket says 172.21.15.119:46420) 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 -- 172.21.15.119:0/191956084 learned_addr learned my addr 172.21.15.119:0/191956084 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb8d48700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032d000 0x55a5300e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 -- 172.21.15.119:0/191956084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032d000 msgr2=0x55a5300e4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032d000 0x55a5300e4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 -- 172.21.15.119:0/191956084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032c800 msgr2=0x55a5300e3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032c800 0x55a5300e3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 -- 172.21.15.119:0/191956084 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a530060b60 con 0x55a53032cc00 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032c800 0x55a5300e3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:42.258 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb9549700 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032cc00 0x55a5300e3b80 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x55a530e1a600 tx=0x55a530e26180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71ee0254b9ffe8ae server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb4d40700 1 -- 172.21.15.119:0/191956084 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a53008d1e0 con 0x55a53032cc00 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb4d40700 1 -- 172.21.15.119:0/191956084 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a530e28480 con 0x55a53032cc00 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cb4d40700 1 -- 172.21.15.119:0/191956084 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a53008cf20 con 0x55a53032cc00 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/191956084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032cc00 msgr2=0x55a5300e3b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032cc00 0x55a5300e3b80 secure :-1 s=READY pgs=1950 cs=0 l=1 rev1=1 crypto rx=0x55a530e1a600 tx=0x55a530e26180 comp rx=0 tx=0).stop 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/191956084 shutdown_connections 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032d000 0x55a5300e4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032cc00 0x55a5300e3b80 unknown :-1 s=CLOSED pgs=1950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/191956084 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032c800 0x55a5300e3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/191956084 >> 172.21.15.119:0/191956084 conn(0x55a53003cc00 msgr2=0x55a5300f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/191956084 shutdown_connections 2024-09-13T16:36:42.259 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.253+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/191956084 wait complete. 2024-09-13T16:36:42.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 Processor -- start 2024-09-13T16:36:42.260 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 -- start start 2024-09-13T16:36:42.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032c800 0x55a5300e5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032cc00 0x55a5300e4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032d000 0x55a5300e4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5300be500 con 0x55a53032d000 2024-09-13T16:36:42.262 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a5300be3c0 con 0x55a53032cc00 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a5300be280 con 0x55a53032c800 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032d000 0x55a5300e4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032d000 0x55a5300e4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:46424/0 (socket says 172.21.15.119:46424) 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 learned_addr learned my addr 172.21.15.119:0/3528725878 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb9549700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032cc00 0x55a5300e4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032c800 0x55a5300e5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032c800 msgr2=0x55a5300e5180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032c800 0x55a5300e5180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032cc00 msgr2=0x55a5300e4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53032cc00 0x55a5300e4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a5300601a0 con 0x55a53032d000 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53032c800 0x55a5300e5180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:42.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032d000 0x55a5300e4680 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x55a530e1bad0 tx=0x55a53033ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb353d700 1 -- 172.21.15.119:0/3528725878 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a53008d1e0 con 0x55a53032d000 2024-09-13T16:36:42.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb353d700 1 -- 172.21.15.119:0/3528725878 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a530e29980 con 0x55a53032d000 2024-09-13T16:36:42.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55a5300609c0 con 0x55a53032d000 2024-09-13T16:36:42.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cb353d700 1 -- 172.21.15.119:0/3528725878 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55a53008cf20 con 0x55a53032d000 2024-09-13T16:36:42.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a530060b60 con 0x55a53032d000 2024-09-13T16:36:42.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.257+0000 7f6cbad8f9c0 4 datascan.init: resolving metadata pool 17 2024-09-13T16:36:42.264 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cb353d700 1 -- 172.21.15.119:0/3528725878 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x55a53009c840 con 0x55a53032d000 2024-09-13T16:36:42.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cb353d700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55a5303a6800 0x55a5300e5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cb353d700 1 -- 172.21.15.119:0/3528725878 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x55a530076a80 con 0x55a53032d000 2024-09-13T16:36:42.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cbad8f9c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-09-13T16:36:42.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cbad8f9c0 4 datascan.main: data pool 'cephfs_data' has ID 18 2024-09-13T16:36:42.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cbad8f9c0 4 datascan.main: opening data pool 'cephfs_data' 2024-09-13T16:36:42.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55a53032d400 0x55a5300e4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55a5303a6800 0x55a5300e5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:1 18.0 18:00000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032d800 con 0x55a53032d400 2024-09-13T16:36:42.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cb9549700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55a53032d400 0x55a5300e4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cb9549700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55a53032d400 0x55a5300e4100 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.261+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55a5303a6800 0x55a5300e5700 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x55a5300fce70 tx=0x55a53033d2c0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.265+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(1 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.265+0000 7f6cb453f700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55a5303a6c00 0x55a530498000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.267 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.265+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:2 18.20 18:04000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7000 con 0x55a5303a6c00 2024-09-13T16:36:42.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.265+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55a5303a6c00 0x55a530498000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.268 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.265+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55a5303a6c00 0x55a530498000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.269+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(2 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.269+0000 7f6cb4d40700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55a5303a7800 0x55a530498580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.269+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:3 18.10 18:08000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7c00 con 0x55a5303a7800 2024-09-13T16:36:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.269+0000 7f6cb9549700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55a5303a7800 0x55a530498580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.269+0000 7f6cb9549700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55a5303a7800 0x55a530498580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.269+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(3 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5303a7800 2024-09-13T16:36:42.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.269+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:4 18.30 18:0c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7400 con 0x55a53032d400 2024-09-13T16:36:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(4 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:5 18.8 18:10000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7c00 con 0x55a53032d400 2024-09-13T16:36:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 3 ==== osd_op_reply(5 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:6 18.28 18:14000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa000 con 0x55a5303a6c00 2024-09-13T16:36:42.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== osd_op_reply(6 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb4d40700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55a5303a7400 0x55a530498b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:7 18.18 18:18000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032d800 con 0x55a5303a7400 2024-09-13T16:36:42.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55a5303a7400 0x55a530498b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55a5303a7400 0x55a530498b00 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(7 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb453f700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55a5304aa400 0x55a530499600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:8 18.38 18:1c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa800 con 0x55a5304aa400 2024-09-13T16:36:42.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.273+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55a5304aa400 0x55a530499600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55a5304aa400 0x55a530499600 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== osd_op_reply(8 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:9 18.4 18:20000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa000 con 0x55a53032d400 2024-09-13T16:36:42.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 4 ==== osd_op_reply(9 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:10 18.24 18:24000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304ab400 con 0x55a53032d400 2024-09-13T16:36:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 5 ==== osd_op_reply(10 [pgnls start_epoch 128 out=49b] v126'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:11 18.14 18:28000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aac00 con 0x55a5303a7800 2024-09-13T16:36:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(11 [pgnls start_epoch 128 out=49b] v126'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5303a7800 2024-09-13T16:36:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:12 18.34 18:2c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304ab800 con 0x55a53032d400 2024-09-13T16:36:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 6 ==== osd_op_reply(12 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:13 18.c 18:30000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304ab000 con 0x55a5303a6c00 2024-09-13T16:36:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 3 ==== osd_op_reply(13 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb453f700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55a5304abc00 0x55a530499b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.282 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.277+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:14 18.2c 18:34000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa000 con 0x55a5304abc00 2024-09-13T16:36:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb9549700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55a5304abc00 0x55a530499b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb9549700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55a5304abc00 0x55a530499b80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== osd_op_reply(14 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5304abc00 2024-09-13T16:36:42.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:15 18.1c 18:38000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7000 con 0x55a5303a7400 2024-09-13T16:36:42.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== osd_op_reply(15 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.284 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb453f700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55a5304ab400 0x55a53049a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:16 18.3c 18:3c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aac00 con 0x55a5304ab400 2024-09-13T16:36:42.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55a5304ab400 0x55a53049a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb8d48700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55a5304ab400 0x55a53049a100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(16 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.281+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:17 18.2 18:40000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aac00 con 0x55a5304ab400 2024-09-13T16:36:42.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== osd_op_reply(17 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:18 18.22 18:44000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7c00 con 0x55a5304aa400 2024-09-13T16:36:42.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== osd_op_reply(18 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.287 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:19 18.12 18:48000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a6400 con 0x55a5304abc00 2024-09-13T16:36:42.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== osd_op_reply(19 [pgnls start_epoch 128 out=81b] v127'22 uv22 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5304abc00 2024-09-13T16:36:42.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:20 18.12 18:4bec3886:::10000000002.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032dc00 con 0x55a5304abc00 2024-09-13T16:36:42.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.2 v2:172.21.15.119:6824/3093261484 3 ==== osd_op_reply(20 10000000002.00000000 [stat out=16b] v0'0 uv22 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5304abc00 2024-09-13T16:36:42.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cbad8f9c0 10 datascan.operator(): handling object 1099511627778.0 2024-09-13T16:36:42.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.285+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:21 18.12 18:4bec3886:::10000000002.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7c00 con 0x55a5304abc00 2024-09-13T16:36:42.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.2 v2:172.21.15.119:6824/3093261484 4 ==== osd_op_reply(21 10000000002.00000000 [call] v128'23 uv23 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5304abc00 2024-09-13T16:36:42.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:22 18.32 18:4c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa800 con 0x55a5304aa400 2024-09-13T16:36:42.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 3 ==== osd_op_reply(22 [pgnls start_epoch 128 out=49b] v126'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:23 18.a 18:50000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7000 con 0x55a5304aa400 2024-09-13T16:36:42.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 4 ==== osd_op_reply(23 [pgnls start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cb453f700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55a5303a6400 0x55a53049a680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:24 18.2a 18:54000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7c00 con 0x55a5303a6400 2024-09-13T16:36:42.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.289+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55a5303a6400 0x55a53049a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb9d4a700 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55a5303a6400 0x55a53049a680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(24 [pgnls start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:25 18.1a 18:58000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032d800 con 0x55a5304aa400 2024-09-13T16:36:42.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 5 ==== osd_op_reply(25 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:26 18.3a 18:5c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032dc00 con 0x55a53032d400 2024-09-13T16:36:42.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 7 ==== osd_op_reply(26 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:27 18.6 18:60000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032d800 con 0x55a5304ab400 2024-09-13T16:36:42.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 3 ==== osd_op_reply(27 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:28 18.26 18:64000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304ab800 con 0x55a5304abc00 2024-09-13T16:36:42.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.2 v2:172.21.15.119:6824/3093261484 5 ==== osd_op_reply(28 [pgnls start_epoch 128 out=49b] v126'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5304abc00 2024-09-13T16:36:42.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.293+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:29 18.16 18:68000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7000 con 0x55a5304aa400 2024-09-13T16:36:42.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.297+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 6 ==== osd_op_reply(29 [pgnls start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.297+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:30 18.36 18:6c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa000 con 0x55a5303a7400 2024-09-13T16:36:42.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.297+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 3 ==== osd_op_reply(30 [pgnls start_epoch 128 out=81b] v126'33 uv17 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.297+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:31 18.36 18:6fe38dc2:::100000001f7.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304ab000 con 0x55a5303a7400 2024-09-13T16:36:42.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.297+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 4 ==== osd_op_reply(31 100000001f7.00000000 [stat out=16b] v0'0 uv17 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.297+0000 7f6cbad8f9c0 10 datascan.operator(): handling object 1099511628279.0 2024-09-13T16:36:42.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.297+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:32 18.36 18:6fe38dc2:::100000001f7.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2000 con 0x55a5303a7400 2024-09-13T16:36:42.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 5 ==== osd_op_reply(32 100000001f7.00000000 [call] v128'34 uv34 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:33 18.e 18:70000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2400 con 0x55a53032d400 2024-09-13T16:36:42.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 8 ==== osd_op_reply(33 [pgnls start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:34 18.2e 18:74000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2000 con 0x55a5304aa400 2024-09-13T16:36:42.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 7 ==== osd_op_reply(34 [pgnls start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:35 18.1e 18:78000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7c00 con 0x55a5303a6c00 2024-09-13T16:36:42.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 4 ==== osd_op_reply(35 [pgnls start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:36 18.3e 18:7c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aac00 con 0x55a5304aa400 2024-09-13T16:36:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 8 ==== osd_op_reply(36 [pgnls start_epoch 128 out=49b] v126'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:37 18.1 18:80000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa800 con 0x55a5304aa400 2024-09-13T16:36:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 9 ==== osd_op_reply(37 [pgnls start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.301+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:38 18.21 18:84000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa000 con 0x55a5303a6c00 2024-09-13T16:36:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 5 ==== osd_op_reply(38 [pgnls start_epoch 128 out=49b] v126'24 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.306 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:39 18.11 18:88000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aac00 con 0x55a5303a7400 2024-09-13T16:36:42.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 6 ==== osd_op_reply(39 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:40 18.31 18:8c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304ab000 con 0x55a5303a6400 2024-09-13T16:36:42.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== osd_op_reply(40 [pgnls start_epoch 128 out=81b] v126'15 uv8 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:41 18.31 18:8de29a03:::10000000004.00000001:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2800 con 0x55a5303a6400 2024-09-13T16:36:42.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 3 ==== osd_op_reply(41 10000000004.00000001 [stat out=16b] v0'0 uv8 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cbad8f9c0 10 datascan.operator(): handling object 1099511627780.1 2024-09-13T16:36:42.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.305+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:42 18.5 18:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2c00 con 0x55a5303a7400 2024-09-13T16:36:42.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 7 ==== osd_op_reply(42 10000000004.00000000 [call] v128'14 uv14 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:43 18.9 18:90000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3000 con 0x55a5304aa400 2024-09-13T16:36:42.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 10 ==== osd_op_reply(43 [pgnls start_epoch 128 out=49b] v126'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.312 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:44 18.29 18:94000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032d800 con 0x55a5304ab400 2024-09-13T16:36:42.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 4 ==== osd_op_reply(44 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.313 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:45 18.19 18:98000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aa800 con 0x55a5303a6400 2024-09-13T16:36:42.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 4 ==== osd_op_reply(45 [pgnls start_epoch 128 out=81b] v127'23 uv23 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.314 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.309+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:46 18.19 18:997b7129:::100000001f8.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3400 con 0x55a5303a6400 2024-09-13T16:36:42.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.313+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 5 ==== osd_op_reply(46 100000001f8.00000000 [stat out=16b] v0'0 uv23 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.313+0000 7f6cbad8f9c0 10 datascan.operator(): handling object 1099511628280.0 2024-09-13T16:36:42.315 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.313+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:47 18.19 18:997b7129:::100000001f8.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3800 con 0x55a5303a6400 2024-09-13T16:36:42.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 6 ==== osd_op_reply(47 100000001f8.00000000 [call] v128'24 uv24 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:48 18.39 18:9c000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3c00 con 0x55a53032d400 2024-09-13T16:36:42.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 9 ==== osd_op_reply(48 [pgnls start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.319 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:49 18.5 18:a0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7000 con 0x55a5303a7400 2024-09-13T16:36:42.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 8 ==== osd_op_reply(49 [pgnls start_epoch 128 out=81b] v128'14 uv14 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:50 18.5 18:a367a3e8:::10000000004.00000000:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2800 con 0x55a5303a7400 2024-09-13T16:36:42.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 9 ==== osd_op_reply(50 10000000004.00000000 [stat out=16b] v0'0 uv14 ondisk = 0) v8 ==== 164+0+16 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cbad8f9c0 10 datascan.operator(): handling object 1099511627780.0 2024-09-13T16:36:42.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.317+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:51 18.5 18:a367a3e8:::10000000004.00000000:head [call cephfs.accumulate_inode_metadata in=112b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3000 con 0x55a5303a7400 2024-09-13T16:36:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 10 ==== osd_op_reply(51 10000000004.00000000 [call] v128'15 uv15 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:52 18.25 18:a4000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3400 con 0x55a5303a7800 2024-09-13T16:36:42.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(52 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5303a7800 2024-09-13T16:36:42.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:53 18.15 18:a8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7c00 con 0x55a5303a6c00 2024-09-13T16:36:42.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 6 ==== osd_op_reply(53 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:54 18.35 18:ac000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2c00 con 0x55a5303a6400 2024-09-13T16:36:42.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 7 ==== osd_op_reply(54 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.321+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:55 18.d 18:b0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2000 con 0x55a5303a6400 2024-09-13T16:36:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 8 ==== osd_op_reply(55 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.327 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:56 18.2d 18:b4000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2800 con 0x55a5303a7800 2024-09-13T16:36:42.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.6 v2:172.21.15.145:6800/3152548227 4 ==== osd_op_reply(56 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5303a7800 2024-09-13T16:36:42.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:57 18.1d 18:b8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2400 con 0x55a53032d400 2024-09-13T16:36:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.0 v2:172.21.15.119:6800/2655437631 10 ==== osd_op_reply(57 [pgnls start_epoch 128 out=49b] v126'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a53032d400 2024-09-13T16:36:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:58 18.3d 18:bc000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3000 con 0x55a5304aa400 2024-09-13T16:36:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.3 v2:172.21.15.119:6808/3803418857 11 ==== osd_op_reply(58 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5304aa400 2024-09-13T16:36:42.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:59 18.3 18:c0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3c00 con 0x55a5303a7400 2024-09-13T16:36:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 11 ==== osd_op_reply(59 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.325+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:60 18.23 18:c4000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5303a7000 con 0x55a5304abc00 2024-09-13T16:36:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.2 v2:172.21.15.119:6824/3093261484 6 ==== osd_op_reply(60 [pgnls start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5304abc00 2024-09-13T16:36:42.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:61 18.13 18:c8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3400 con 0x55a5304ab400 2024-09-13T16:36:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 5 ==== osd_op_reply(61 [pgnls start_epoch 128 out=49b] v126'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:62 18.33 18:cc000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032d800 con 0x55a5303a7400 2024-09-13T16:36:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 12 ==== osd_op_reply(62 [pgnls start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:63 18.b 18:d0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2800 con 0x55a5303a6c00 2024-09-13T16:36:42.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 7 ==== osd_op_reply(63 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:64 18.2b 18:d4000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304aac00 con 0x55a5304ab400 2024-09-13T16:36:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 6 ==== osd_op_reply(64 [pgnls start_epoch 128 out=49b] v126'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.329+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:65 18.1b 18:d8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304ab800 con 0x55a5303a7400 2024-09-13T16:36:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 13 ==== osd_op_reply(65 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:66 18.3b 18:dc000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304da000 con 0x55a5303a6400 2024-09-13T16:36:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 9 ==== osd_op_reply(66 [pgnls start_epoch 128 out=49b] v126'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:67 18.7 18:e0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a53032dc00 con 0x55a5304abc00 2024-09-13T16:36:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb9549700 1 -- 172.21.15.119:0/3528725878 <== osd.2 v2:172.21.15.119:6824/3093261484 7 ==== osd_op_reply(67 [pgnls start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ab40 con 0x55a5304abc00 2024-09-13T16:36:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:68 18.27 18:e4000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304da400 con 0x55a5303a6c00 2024-09-13T16:36:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 8 ==== osd_op_reply(68 [pgnls start_epoch 128 out=49b] v126'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:69 18.17 18:e8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304da000 con 0x55a5303a6c00 2024-09-13T16:36:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 9 ==== osd_op_reply(69 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.333+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:70 18.37 18:ec000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304da800 con 0x55a5304ab400 2024-09-13T16:36:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 7 ==== osd_op_reply(70 [pgnls start_epoch 128 out=49b] v126'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.339 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:71 18.f 18:f0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d3800 con 0x55a5303a6c00 2024-09-13T16:36:42.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.5 v2:172.21.15.145:6824/4068956621 10 ==== osd_op_reply(71 [pgnls start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6c00 2024-09-13T16:36:42.340 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:72 18.2f 18:f4000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304dac00 con 0x55a5303a7400 2024-09-13T16:36:42.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.1 v2:172.21.15.119:6816/1495422598 14 ==== osd_op_reply(72 [pgnls start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5303a7400 2024-09-13T16:36:42.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb4d40700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:73 18.1f 18:f8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304d2c00 con 0x55a5304ab400 2024-09-13T16:36:42.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 <== osd.7 v2:172.21.15.145:6808/1049327118 8 ==== osd_op_reply(73 [pgnls start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2afc0 con 0x55a5304ab400 2024-09-13T16:36:42.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.337+0000 7f6cb453f700 1 -- 172.21.15.119:0/3528725878 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:74 18.3f 18:fc000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55a5304db000 con 0x55a5303a6400 2024-09-13T16:36:42.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cb9d4a700 1 -- 172.21.15.119:0/3528725878 <== osd.4 v2:172.21.15.145:6816/4125431312 10 ==== osd_op_reply(74 [pgnls start_epoch 128 out=49b] v126'17 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55a530e2ad80 con 0x55a5303a6400 2024-09-13T16:36:42.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55a53032d400 msgr2=0x55a5300e4100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55a53032d400 0x55a5300e4100 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55a5303a7400 msgr2=0x55a530498b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.344 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55a5303a7400 0x55a530498b00 crc :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55a5304abc00 msgr2=0x55a530499b80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55a5304abc00 0x55a530499b80 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55a5304aa400 msgr2=0x55a530499600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55a5304aa400 0x55a530499600 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55a5303a6400 msgr2=0x55a53049a680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55a5303a6400 0x55a53049a680 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55a5303a6c00 msgr2=0x55a530498000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 reap_dead start 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55a5303a6c00 0x55a530498000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55a5303a7800 msgr2=0x55a530498580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55a5303a7800 0x55a530498580 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55a5304ab400 msgr2=0x55a53049a100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55a5304ab400 0x55a53049a100 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55a5303a6800 msgr2=0x55a5300e5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55a5303a6800 0x55a5300e5700 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x55a5300fce70 tx=0x55a53033d2c0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032d000 msgr2=0x55a5300e4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/3528725878 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53032d000 0x55a5300e4680 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x55a530e1bad0 tx=0x55a53033ce40 comp rx=0 tx=0).stop 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cb8d48700 1 -- 172.21.15.119:0/3528725878 reap_dead start 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 shutdown_connections 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 >> 172.21.15.119:0/3528725878 conn(0x55a53003cc00 msgr2=0x55a5300f9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 shutdown_connections 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/3528725878 wait complete. 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/724559992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53003d800 msgr2=0x55a5300e2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53003d800 0x55a5300e2580 secure :-1 s=READY pgs=1949 cs=0 l=1 rev1=1 crypto rx=0x55a5300fd830 tx=0x55a530e26060 comp rx=0 tx=0).stop 2024-09-13T16:36:42.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/724559992 shutdown_connections 2024-09-13T16:36:42.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55a53003d000 0x55a5300e3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55a53003d400 0x55a5300e2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 --2- 172.21.15.119:0/724559992 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55a53003d800 0x55a5300e2580 unknown :-1 s=CLOSED pgs=1949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/724559992 >> 172.21.15.119:0/724559992 conn(0x55a53003dc00 msgr2=0x55a5300f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/724559992 shutdown_connections 2024-09-13T16:36:42.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.341+0000 7f6cbad8f9c0 1 -- 172.21.15.119:0/724559992 wait complete. 2024-09-13T16:36:42.350 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_extents', 'cephfs_data'] in time 0:00:00.135578, result: 2024-09-13T16:36:42.350 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_inodes --filter-tag mytag123 cephfs_data 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.433+0000 7f34652d49c0 1 -- start start 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.433+0000 7f34652d49c0 1 Processor -- start 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df5490d800 0x55df549b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df5490d400 0x55df549b3600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df5490d000 0x55df549b3080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5498ea00 con 0x55df5490d000 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5498e640 con 0x55df5490d400 2024-09-13T16:36:42.445 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df5498e500 con 0x55df5490d800 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df5490d000 0x55df549b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df5490d000 0x55df549b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55328/0 (socket says 172.21.15.119:55328) 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 -- 172.21.15.119:0/3249332284 learned_addr learned my addr 172.21.15.119:0/3249332284 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346328d700 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df5490d400 0x55df549b3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f3463a8e700 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df5490d800 0x55df549b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 -- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df5490d800 msgr2=0x55df549b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df5490d800 0x55df549b3b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 -- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df5490d400 msgr2=0x55df549b3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df5490d400 0x55df549b3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 -- 172.21.15.119:0/3249332284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df549309c0 con 0x55df5490d000 2024-09-13T16:36:42.446 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f346428f700 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df5490d000 0x55df549b3080 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x55df549cd860 tx=0x55df556ec000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80572d9f358ddc26 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f3461289700 1 -- 172.21.15.119:0/3249332284 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55df5495d080 con 0x55df5490d000 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f3461289700 1 -- 172.21.15.119:0/3249332284 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55df54958900 con 0x55df5490d000 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f3461289700 1 -- 172.21.15.119:0/3249332284 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55df5495d600 con 0x55df5490d000 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 -- 172.21.15.119:0/3249332284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df549301a0 con 0x55df5490d000 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 1 -- 172.21.15.119:0/3249332284 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x55df549309c0 con 0x55df5490d000 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 4 waiting for MDS map... 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f3461289700 1 -- 172.21.15.119:0/3249332284 <== mon.0 v2:172.21.15.119:3300/0 4 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x55df54946c40 con 0x55df5490d000 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f3461289700 1 -- 172.21.15.119:0/3249332284 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 99) v1 ==== 2573+0+0 (secure 0 0 0) 0x55df5494ef00 con 0x55df5490d000 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 4 Got MDS map 99 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 10 datascan.parse_kwarg: Applying tag filter: 'mytag123' 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 4 datascan.main: Using metadata pool output 2024-09-13T16:36:42.447 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.437+0000 7f34652d49c0 4 datascan.main: connecting to RADOS... 2024-09-13T16:36:42.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 Processor -- start 2024-09-13T16:36:42.448 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 -- start start 2024-09-13T16:36:42.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 0x55df549b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00800 0x55df549b2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df54943e00 con 0x55df54c00800 2024-09-13T16:36:42.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5498e500 con 0x55df54c00c00 2024-09-13T16:36:42.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f34652d49c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df5498e640 con 0x55df54c00400 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00800 0x55df549b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f346328d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 0x55df549b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00800 0x55df549b2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55340/0 (socket says 172.21.15.119:55340) 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 -- 172.21.15.119:0/126453870 learned_addr learned my addr 172.21.15.119:0/126453870 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f346328d700 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 -- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 msgr2=0x55df549b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 0x55df549b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.453 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 -- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00c00 msgr2=0x55df549b4100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 -- 172.21.15.119:0/126453870 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df54930b60 con 0x55df54c00800 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f346328d700 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.441+0000 7f3463a8e700 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00800 0x55df549b2580 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x55df556ee7b0 tx=0x55df54c10d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907677806167ac5f server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f345f285700 1 -- 172.21.15.119:0/126453870 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55df5495d080 con 0x55df54c00800 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f345f285700 1 -- 172.21.15.119:0/126453870 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55df54bf8300 con 0x55df54c00800 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f345f285700 1 -- 172.21.15.119:0/126453870 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55df5495d600 con 0x55df54c00800 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- 172.21.15.119:0/126453870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00800 msgr2=0x55df549b2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00800 0x55df549b2580 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x55df556ee7b0 tx=0x55df54c10d20 comp rx=0 tx=0).stop 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- 172.21.15.119:0/126453870 shutdown_connections 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00800 0x55df549b2580 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 --2- 172.21.15.119:0/126453870 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 0x55df549b2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- 172.21.15.119:0/126453870 >> 172.21.15.119:0/126453870 conn(0x55df5490cc00 msgr2=0x55df549c9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- 172.21.15.119:0/126453870 shutdown_connections 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- 172.21.15.119:0/126453870 wait complete. 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 Processor -- start 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- start start 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 0x55df549b2b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00800 0x55df549b2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5498e3c0 con 0x55df54c00c00 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5498e280 con 0x55df54c00800 2024-09-13T16:36:42.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f34652d49c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x55df5498e140 con 0x55df54c00400 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f346328d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f346328d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55354/0 (socket says 172.21.15.119:55354) 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 learned_addr learned my addr 172.21.15.119:0/1300697221 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f3463a8e700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00800 0x55df549b2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.445+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 0x55df549b2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 msgr2=0x55df549b2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df54c00400 0x55df549b2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00800 msgr2=0x55df549b2580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df54c00800 0x55df549b2580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df54930340 con 0x55df54c00c00 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00c00 0x55df549b4100 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x55df556ef9b0 tx=0x55df556ec240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f345da82700 1 -- 172.21.15.119:0/1300697221 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55df5495d080 con 0x55df54c00c00 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f345da82700 1 -- 172.21.15.119:0/1300697221 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55df54bf9680 con 0x55df54c00c00 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55df54930000 con 0x55df54c00c00 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f345da82700 1 -- 172.21.15.119:0/1300697221 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x55df5495d600 con 0x55df54c00c00 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df54930ea0 con 0x55df54c00c00 2024-09-13T16:36:42.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f34652d49c0 4 datascan.init: resolving metadata pool 17 2024-09-13T16:36:42.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f345da82700 1 -- 172.21.15.119:0/1300697221 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x55df5496c840 con 0x55df54c00c00 2024-09-13T16:36:42.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f345da82700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55df54d74000 0x55df54d7a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.449+0000 7f345da82700 1 -- 172.21.15.119:0/1300697221 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x55df54946a80 con 0x55df54c00c00 2024-09-13T16:36:42.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 4 datascan.main: data pool 'cephfs_data' has ID 18 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 4 datascan.main: opening data pool 'cephfs_data' 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55df54c01000 0x55df549b4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:1 17.14 17:287034c4:::1.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54c01400 con 0x55df54c01000 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f3463a8e700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55df54d74000 0x55df54d7a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55df54c01000 0x55df549b4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55df54c01000 0x55df549b4680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f3463a8e700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55df54d74000 0x55df54d7a000 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x55df556ee7b0 tx=0x55df556ec300 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== osd_op_reply(1 1.00000000.inode [stat out=16b] v0'0 uv36 ondisk = 0) v8 ==== 160+0+16 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55df54c01800 0x55df549b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:2 17.1f 17:fe07bba1:::100.00000000.inode:head [stat] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d74400 con 0x55df54c01800 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f346428f700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55df54c01800 0x55df549b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f346428f700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55df54c01800 0x55df549b4c00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(2 100.00000000.inode [stat out=16b] v0'0 uv3 ondisk = 0) v8 ==== 162+0+16 (crc 0 0 0) 0x55df556f8d80 con 0x55df54c01800 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55df54d74800 0x55df549b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:3 18.0 18:00000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d74c00 con 0x55df54d74800 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f3463a8e700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55df54d74800 0x55df549b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.453+0000 7f3463a8e700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55df54d74800 0x55df549b5180 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(3 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f345ea84700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55df54d8e000 0x55df54d7a580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:4 18.20 18:04000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e400 con 0x55df54d8e000 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55df54d8e000 0x55df54d7a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55df54d8e000 0x55df54d7a580 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(4 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:5 18.10 18:08000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8ec00 con 0x55df54c01800 2024-09-13T16:36:42.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(5 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54c01800 2024-09-13T16:36:42.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:6 18.30 18:0c000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e800 con 0x55df54d74800 2024-09-13T16:36:42.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(6 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.457+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:7 18.8 18:10000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e800 con 0x55df54d74800 2024-09-13T16:36:42.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 3 ==== osd_op_reply(7 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:8 18.28 18:14000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f000 con 0x55df54d8e000 2024-09-13T16:36:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== osd_op_reply(8 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f345f285700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55df54d74c00 0x55df54d7ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:9 18.18 18:18000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e400 con 0x55df54d74c00 2024-09-13T16:36:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f346428f700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55df54d74c00 0x55df54d7ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.465 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f346428f700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55df54d74c00 0x55df54d7ab00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(9 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:10 18.38 18:1c000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f400 con 0x55df54c01000 2024-09-13T16:36:42.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== osd_op_reply(10 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.466 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:11 18.4 18:20000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f000 con 0x55df54d74800 2024-09-13T16:36:42.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 4 ==== osd_op_reply(11 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:12 18.24 18:24000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f800 con 0x55df54d74800 2024-09-13T16:36:42.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 5 ==== osd_op_reply(12 [pgnls-filter start_epoch 128 out=49b] v126'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.467 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.461+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:13 18.14 18:28000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54c01400 con 0x55df54c01800 2024-09-13T16:36:42.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(13 [pgnls-filter start_epoch 128 out=49b] v126'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54c01800 2024-09-13T16:36:42.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:14 18.34 18:2c000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e800 con 0x55df54d74800 2024-09-13T16:36:42.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 6 ==== osd_op_reply(14 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:15 18.c 18:30000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8ec00 con 0x55df54d8e000 2024-09-13T16:36:42.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 3 ==== osd_op_reply(15 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f345ea84700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55df54d8f000 0x55df54d7b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:16 18.2c 18:34000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f800 con 0x55df54d8f000 2024-09-13T16:36:42.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f3463a8e700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55df54d8f000 0x55df54d7b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.469 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f3463a8e700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55df54d8f000 0x55df54d7b080 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== osd_op_reply(16 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d8f000 2024-09-13T16:36:42.470 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.465+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:17 18.1c 18:38000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e400 con 0x55df54d74c00 2024-09-13T16:36:42.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== osd_op_reply(17 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f345ea84700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55df54d74400 0x55df54d7b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:18 18.3c 18:3c000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0000 con 0x55df54d74400 2024-09-13T16:36:42.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55df54d74400 0x55df54d7b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.471 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f346328d700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55df54d74400 0x55df54d7b600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(18 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.472 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:19 18.2 18:40000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8fc00 con 0x55df54d74400 2024-09-13T16:36:42.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== osd_op_reply(19 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.473 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:20 18.22 18:44000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0400 con 0x55df54c01000 2024-09-13T16:36:42.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 3 ==== osd_op_reply(20 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:21 18.12 18:48000000::::head [pgnls-filter start_epoch 128 in=89b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0000 con 0x55df54d8f000 2024-09-13T16:36:42.474 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== osd_op_reply(21 [pgnls-filter start_epoch 128 out=81b] v128'23 uv23 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d8f000 2024-09-13T16:36:42.475 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.469+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:22 18.0 18:00000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75000 con 0x55df54d74800 2024-09-13T16:36:42.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 7 ==== osd_op_reply(22 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:23 18.20 18:04000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1400 con 0x55df54d8e000 2024-09-13T16:36:42.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 4 ==== osd_op_reply(23 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:24 18.10 18:08000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0800 con 0x55df54c01800 2024-09-13T16:36:42.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.6 v2:172.21.15.145:6800/3152548227 4 ==== osd_op_reply(24 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54c01800 2024-09-13T16:36:42.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:25 18.30 18:0c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f400 con 0x55df54d74800 2024-09-13T16:36:42.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 8 ==== osd_op_reply(25 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.477 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:26 18.8 18:10000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0c00 con 0x55df54d74800 2024-09-13T16:36:42.478 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 9 ==== osd_op_reply(26 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:27 18.28 18:14000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8ec00 con 0x55df54d8e000 2024-09-13T16:36:42.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 5 ==== osd_op_reply(27 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.473+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:28 18.18 18:18000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1000 con 0x55df54d74c00 2024-09-13T16:36:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 3 ==== osd_op_reply(28 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:29 18.38 18:1c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8fc00 con 0x55df54c01000 2024-09-13T16:36:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 4 ==== osd_op_reply(29 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.481 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:30 18.4 18:20000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0800 con 0x55df54d74800 2024-09-13T16:36:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 10 ==== osd_op_reply(30 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:31 18.24 18:24000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1000 con 0x55df54d74800 2024-09-13T16:36:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 11 ==== osd_op_reply(31 [pgnls-filter start_epoch 128 out=49b] v126'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.482 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:32 18.14 18:28000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75000 con 0x55df54c01800 2024-09-13T16:36:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.6 v2:172.21.15.145:6800/3152548227 5 ==== osd_op_reply(32 [pgnls-filter start_epoch 128 out=49b] v126'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54c01800 2024-09-13T16:36:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:33 18.34 18:2c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0000 con 0x55df54d74800 2024-09-13T16:36:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 12 ==== osd_op_reply(33 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.477+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:34 18.c 18:30000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e400 con 0x55df54d8e000 2024-09-13T16:36:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 6 ==== osd_op_reply(34 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.483 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:35 18.2c 18:34000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1800 con 0x55df54d8f000 2024-09-13T16:36:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.2 v2:172.21.15.119:6824/3093261484 3 ==== osd_op_reply(35 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d8f000 2024-09-13T16:36:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:36 18.1c 18:38000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e800 con 0x55df54d74c00 2024-09-13T16:36:42.484 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 4 ==== osd_op_reply(36 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:37 18.3c 18:3c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1c00 con 0x55df54d74400 2024-09-13T16:36:42.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 3 ==== osd_op_reply(37 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.485 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:38 18.2 18:40000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f800 con 0x55df54d74400 2024-09-13T16:36:42.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 4 ==== osd_op_reply(38 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.481+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:39 18.22 18:44000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0400 con 0x55df54c01000 2024-09-13T16:36:42.486 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 5 ==== osd_op_reply(39 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:40 18.12 18:48000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54c01400 con 0x55df54d8f000 2024-09-13T16:36:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.2 v2:172.21.15.119:6824/3093261484 4 ==== osd_op_reply(40 [pgnls-filter start_epoch 128 out=49b] v128'23 uv23 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d8f000 2024-09-13T16:36:42.487 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:41 18.32 18:4c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1400 con 0x55df54c01000 2024-09-13T16:36:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 6 ==== osd_op_reply(41 [pgnls-filter start_epoch 128 out=49b] v126'15 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.488 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:42 18.a 18:50000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0800 con 0x55df54c01000 2024-09-13T16:36:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 7 ==== osd_op_reply(42 [pgnls-filter start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f345f285700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55df54da0c00 0x55df54d7bb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:43 18.2a 18:54000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8ec00 con 0x55df54da0c00 2024-09-13T16:36:42.489 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f346428f700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55df54da0c00 0x55df54d7bb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.490 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f346428f700 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55df54da0c00 0x55df54d7bb80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(43 [pgnls-filter start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54da0c00 2024-09-13T16:36:42.491 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.485+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:44 18.1a 18:58000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1000 con 0x55df54c01000 2024-09-13T16:36:42.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 8 ==== osd_op_reply(44 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:45 18.3a 18:5c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8fc00 con 0x55df54d74800 2024-09-13T16:36:42.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 13 ==== osd_op_reply(45 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.492 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:46 18.6 18:60000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0000 con 0x55df54d74400 2024-09-13T16:36:42.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 5 ==== osd_op_reply(46 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:47 18.26 18:64000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e400 con 0x55df54d8f000 2024-09-13T16:36:42.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.2 v2:172.21.15.119:6824/3093261484 5 ==== osd_op_reply(47 [pgnls-filter start_epoch 128 out=49b] v126'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d8f000 2024-09-13T16:36:42.494 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:48 18.16 18:68000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1c00 con 0x55df54c01000 2024-09-13T16:36:42.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 9 ==== osd_op_reply(48 [pgnls-filter start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.489+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:49 18.36 18:6c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f400 con 0x55df54d74c00 2024-09-13T16:36:42.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 5 ==== osd_op_reply(49 [pgnls-filter start_epoch 128 out=81b] v128'34 uv34 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 20 datascan.forall_objects: OSD matched oid 100000001f7.00000000 2024-09-13T16:36:42.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 10 datascan.operator(): handling object 100000001f7.0 2024-09-13T16:36:42.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:50 18.36 18:6fe38dc2:::100000001f7.00000000:head [getxattr scan_ceiling in=12b,getxattr scan_max_size in=13b,getxattr scan_max_mtime in=14b,getxattr scan_pool_id in=12b,getxattr parent in=6b,getxattr layout in=6b,getxattr symlink in=7b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75400 con 0x55df54d74c00 2024-09-13T16:36:42.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 6 ==== osd_op_reply(50 100000001f7.00000000 [getxattr (22) out=22b,getxattr (8) out=8b,getxattr (8) out=8b,getxattr,getxattr (126) out=126b,getxattr (30) out=30b,getxattr] v0'0 uv34 ondisk = 0) v8 ==== 416+0+194 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 10 datascan.inject_with_backtrace: inode: 0x0x100000001f7 2024-09-13T16:36:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 10 datascan.inject_with_backtrace: backptr: 0x0x10000000001/bravo 2024-09-13T16:36:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 20 datascan.get_frag_of: dirino=0x10000000001 target_dname=bravo 2024-09-13T16:36:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:51 17.7 17:e2e07930:::10000000001.00000000:head [getxattr parent in=6b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75800 con 0x55df54d74800 2024-09-13T16:36:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 14 ==== osd_op_reply(51 10000000001.00000000 [getxattr (95) out=95b] v0'0 uv7 ondisk = 0) v8 ==== 164+0+95 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 20 datascan.get_frag_of: got backtrace parent 0x10000000000/flushed 2024-09-13T16:36:42.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:52 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals-by-keys in=20b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75c00 con 0x55df54c01000 2024-09-13T16:36:42.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 10 ==== osd_op_reply(52 10000000000.00000000 [omap-get-vals-by-keys out=524b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+524 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 20 datascan.read_dentry: oid=10000000000.00000000 dname=flushed frag=*, r=0 2024-09-13T16:36:42.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 20 datascan.get_frag_of: fast path, fragtree is fragtree_t() 2024-09-13T16:36:42.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 20 datascan.get_frag_of: frag is * 2024-09-13T16:36:42.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:53 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-header] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75000 con 0x55df54d74800 2024-09-13T16:36:42.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 15 ==== osd_op_reply(53 10000000001.00000000 [omap-get-header out=274b] v0'0 uv7 ondisk = 0) v8 ==== 164+0+274 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 10 datascan.find_or_create_dirfrag: read_version = 7 2024-09-13T16:36:42.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 20 datascan.find_or_create_dirfrag: Dirfrag already exists: 0x0x10000000001 * 2024-09-13T16:36:42.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.493+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:54 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75400 con 0x55df54d74800 2024-09-13T16:36:42.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 16 ==== osd_op_reply(54 10000000001.00000000 [omap-get-vals-by-keys out=4b] v0'0 uv7 ondisk = 0) v8 ==== 164+0+4 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f34652d49c0 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-09-13T16:36:42.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f34652d49c0 20 datascan.read_dentry: bravo_head not found in result 2024-09-13T16:36:42.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f34652d49c0 10 datascan.inject_with_backtrace: Linking inode 0x0x100000001f7 at 0x0x10000000001/bravo with size=5 bytes 2024-09-13T16:36:42.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:55 17.7 17:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e800 con 0x55df54d74800 2024-09-13T16:36:42.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 17 ==== osd_op_reply(55 10000000001.00000000 [omap-set-vals] v128'9 uv9 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f34652d49c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-09-13T16:36:42.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.497+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:56 18.e 18:70000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8ec00 con 0x55df54d74800 2024-09-13T16:36:42.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 18 ==== osd_op_reply(56 [pgnls-filter start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:57 18.2e 18:74000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75800 con 0x55df54c01000 2024-09-13T16:36:42.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 11 ==== osd_op_reply(57 [pgnls-filter start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:58 18.1e 18:78000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0400 con 0x55df54d8e000 2024-09-13T16:36:42.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 7 ==== osd_op_reply(58 [pgnls-filter start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:59 18.3e 18:7c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75000 con 0x55df54c01000 2024-09-13T16:36:42.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 12 ==== osd_op_reply(59 [pgnls-filter start_epoch 128 out=49b] v126'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:60 18.1 18:80000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1400 con 0x55df54c01000 2024-09-13T16:36:42.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 13 ==== osd_op_reply(60 [pgnls-filter start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.501+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:61 18.21 18:84000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8fc00 con 0x55df54d8e000 2024-09-13T16:36:42.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 8 ==== osd_op_reply(61 [pgnls-filter start_epoch 128 out=49b] v126'24 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:62 18.11 18:88000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0800 con 0x55df54d74c00 2024-09-13T16:36:42.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 7 ==== osd_op_reply(62 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:63 18.31 18:8c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8e400 con 0x55df54da0c00 2024-09-13T16:36:42.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== osd_op_reply(63 [pgnls-filter start_epoch 128 out=49b] v126'15 uv8 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54da0c00 2024-09-13T16:36:42.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:64 18.9 18:90000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1000 con 0x55df54c01000 2024-09-13T16:36:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 14 ==== osd_op_reply(64 [pgnls-filter start_epoch 128 out=49b] v126'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:65 18.29 18:94000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54c01400 con 0x55df54d74400 2024-09-13T16:36:42.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 6 ==== osd_op_reply(65 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.505+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:66 18.19 18:98000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8f800 con 0x55df54da0c00 2024-09-13T16:36:42.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.4 v2:172.21.15.145:6816/4125431312 3 ==== osd_op_reply(66 [pgnls-filter start_epoch 128 out=49b] v128'24 uv24 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54da0c00 2024-09-13T16:36:42.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:67 18.39 18:9c000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1800 con 0x55df54d74800 2024-09-13T16:36:42.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 19 ==== osd_op_reply(67 [pgnls-filter start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:68 18.5 18:a0000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0000 con 0x55df54d74c00 2024-09-13T16:36:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 8 ==== osd_op_reply(68 [pgnls-filter start_epoch 128 out=49b] v128'15 uv15 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:69 18.25 18:a4000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dba000 con 0x55df54c01800 2024-09-13T16:36:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.6 v2:172.21.15.145:6800/3152548227 6 ==== osd_op_reply(69 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54c01800 2024-09-13T16:36:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.509+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:70 18.15 18:a8000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1c00 con 0x55df54d8e000 2024-09-13T16:36:42.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 9 ==== osd_op_reply(70 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:71 18.35 18:ac000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dba400 con 0x55df54da0c00 2024-09-13T16:36:42.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.4 v2:172.21.15.145:6816/4125431312 4 ==== osd_op_reply(71 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54da0c00 2024-09-13T16:36:42.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:72 18.d 18:b0000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0400 con 0x55df54da0c00 2024-09-13T16:36:42.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.4 v2:172.21.15.145:6816/4125431312 5 ==== osd_op_reply(72 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54da0c00 2024-09-13T16:36:42.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:73 18.2d 18:b4000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dba800 con 0x55df54c01800 2024-09-13T16:36:42.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.6 v2:172.21.15.145:6800/3152548227 7 ==== osd_op_reply(73 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54c01800 2024-09-13T16:36:42.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:74 18.1d 18:b8000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1400 con 0x55df54d74800 2024-09-13T16:36:42.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.0 v2:172.21.15.119:6800/2655437631 20 ==== osd_op_reply(74 [pgnls-filter start_epoch 128 out=49b] v126'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d74800 2024-09-13T16:36:42.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.513+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:75 18.3d 18:bc000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbac00 con 0x55df54c01000 2024-09-13T16:36:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.3 v2:172.21.15.119:6808/3803418857 15 ==== osd_op_reply(75 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54c01000 2024-09-13T16:36:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:76 18.3 18:c0000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75c00 con 0x55df54d74c00 2024-09-13T16:36:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 9 ==== osd_op_reply(76 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:77 18.23 18:c4000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbb000 con 0x55df54d8f000 2024-09-13T16:36:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.2 v2:172.21.15.119:6824/3093261484 6 ==== osd_op_reply(77 [pgnls-filter start_epoch 128 out=49b] v126'14 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d8f000 2024-09-13T16:36:42.520 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:78 18.13 18:c8000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75800 con 0x55df54d74400 2024-09-13T16:36:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 7 ==== osd_op_reply(78 [pgnls-filter start_epoch 128 out=49b] v126'30 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:79 18.33 18:cc000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbb400 con 0x55df54d74c00 2024-09-13T16:36:42.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 10 ==== osd_op_reply(79 [pgnls-filter start_epoch 128 out=49b] v126'20 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.517+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:80 18.b 18:d0000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d75000 con 0x55df54d8e000 2024-09-13T16:36:42.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 10 ==== osd_op_reply(80 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:81 18.2b 18:d4000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbb800 con 0x55df54d74400 2024-09-13T16:36:42.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 8 ==== osd_op_reply(81 [pgnls-filter start_epoch 128 out=49b] v126'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:82 18.1b 18:d8000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1000 con 0x55df54d74c00 2024-09-13T16:36:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 11 ==== osd_op_reply(82 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:83 18.3b 18:dc000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbbc00 con 0x55df54da0c00 2024-09-13T16:36:42.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.4 v2:172.21.15.145:6816/4125431312 6 ==== osd_op_reply(83 [pgnls-filter start_epoch 128 out=49b] v126'8 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54da0c00 2024-09-13T16:36:42.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:84 18.7 18:e0000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da1c00 con 0x55df54d8f000 2024-09-13T16:36:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 <== osd.2 v2:172.21.15.119:6824/3093261484 7 ==== osd_op_reply(84 [pgnls-filter start_epoch 128 out=49b] v126'10 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8fc0 con 0x55df54d8f000 2024-09-13T16:36:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.521+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:85 18.27 18:e4000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbb000 con 0x55df54d8e000 2024-09-13T16:36:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 11 ==== osd_op_reply(85 [pgnls-filter start_epoch 128 out=49b] v126'4 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.527 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:86 18.17 18:e8000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0800 con 0x55df54d8e000 2024-09-13T16:36:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 12 ==== osd_op_reply(86 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:87 18.37 18:ec000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbac00 con 0x55df54d74400 2024-09-13T16:36:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 9 ==== osd_op_reply(87 [pgnls-filter start_epoch 128 out=49b] v126'22 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:88 18.f 18:f0000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54da0000 con 0x55df54d8e000 2024-09-13T16:36:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.5 v2:172.21.15.145:6824/4068956621 13 ==== osd_op_reply(88 [pgnls-filter start_epoch 128 out=49b] v126'18 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d8e000 2024-09-13T16:36:42.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:89 18.2f 18:f4000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dbb800 con 0x55df54d74c00 2024-09-13T16:36:42.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.1 v2:172.21.15.119:6816/1495422598 12 ==== osd_op_reply(89 [pgnls-filter start_epoch 128 out=49b] v126'16 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54d74c00 2024-09-13T16:36:42.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.525+0000 7f345ea84700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:90 18.1f 18:f8000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54d8fc00 con 0x55df54d74400 2024-09-13T16:36:42.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f346328d700 1 -- 172.21.15.119:0/1300697221 <== osd.7 v2:172.21.15.145:6808/1049327118 10 ==== osd_op_reply(90 [pgnls-filter start_epoch 128 out=49b] v126'12 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8b40 con 0x55df54d74400 2024-09-13T16:36:42.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f345f285700 1 -- 172.21.15.119:0/1300697221 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:91 18.3f 18:fc000000::::head [pgnls-filter start_epoch 128 in=93b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x55df54dba000 con 0x55df54da0c00 2024-09-13T16:36:42.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f346428f700 1 -- 172.21.15.119:0/1300697221 <== osd.4 v2:172.21.15.145:6816/4125431312 7 ==== osd_op_reply(91 [pgnls-filter start_epoch 128 out=49b] v126'17 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x55df556f8d80 con 0x55df54da0c00 2024-09-13T16:36:42.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55df54d74800 msgr2=0x55df549b5180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x55df54d74800 0x55df549b5180 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55df54d74c00 msgr2=0x55df54d7ab00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x55df54d74c00 0x55df54d7ab00 crc :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55df54d8f000 msgr2=0x55df54d7b080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x55df54d8f000 0x55df54d7b080 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55df54c01000 msgr2=0x55df549b4680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x55df54c01000 0x55df549b4680 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55df54da0c00 msgr2=0x55df54d7bb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x55df54da0c00 0x55df54d7bb80 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55df54d8e000 msgr2=0x55df54d7a580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x55df54d8e000 0x55df54d7a580 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 reap_dead start 2024-09-13T16:36:42.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55df54c01800 msgr2=0x55df549b4c00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55df54c01800 0x55df549b4c00 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 reap_dead start 2024-09-13T16:36:42.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f3463a8e700 1 -- 172.21.15.119:0/1300697221 reap_dead start 2024-09-13T16:36:42.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55df54d74400 msgr2=0x55df54d7b600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55df54d74400 0x55df54d7b600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55df54d74000 msgr2=0x55df54d7a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55df54d74000 0x55df54d7a000 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x55df556ee7b0 tx=0x55df556ec300 comp rx=0 tx=0).stop 2024-09-13T16:36:42.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00c00 msgr2=0x55df549b4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00c00 0x55df549b4100 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x55df556ef9b0 tx=0x55df556ec240 comp rx=0 tx=0).stop 2024-09-13T16:36:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 shutdown_connections 2024-09-13T16:36:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x55df54d74400 0x55df54d7b600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x55df54c01800 0x55df549b4c00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x55df54d74000 0x55df54d7a000 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/1300697221 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df54c00c00 0x55df549b4100 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 >> 172.21.15.119:0/1300697221 conn(0x55df5490cc00 msgr2=0x55df549c9800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 shutdown_connections 2024-09-13T16:36:42.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/1300697221 wait complete. 2024-09-13T16:36:42.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/3249332284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df5490d000 msgr2=0x55df549b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df5490d000 0x55df549b3080 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x55df549cd860 tx=0x55df556ec000 comp rx=0 tx=0).stop 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/3249332284 shutdown_connections 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x55df5490d000 0x55df549b3080 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x55df5490d400 0x55df549b3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 --2- 172.21.15.119:0/3249332284 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x55df5490d800 0x55df549b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/3249332284 >> 172.21.15.119:0/3249332284 conn(0x55df5490dc00 msgr2=0x55df549c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/3249332284 shutdown_connections 2024-09-13T16:36:42.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.529+0000 7f34652d49c0 1 -- 172.21.15.119:0/3249332284 wait complete. 2024-09-13T16:36:42.543 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_inodes', '--filter-tag', 'mytag123', 'cephfs_data'] in time 0:00:00.192193, result: 2024-09-13T16:36:42.543 DEBUG:teuthology.orchestra.run.smithi119:> cephfs-data-scan --debug-mds=20 --debug-ms=1 --debug-objecter=1 scan_links 2024-09-13T16:36:42.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 -- start start 2024-09-13T16:36:42.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 Processor -- start 2024-09-13T16:36:42.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016adf9800 0x56016ae9e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016adf9400 0x56016ae9eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016adf9000 0x56016ae9f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7aa00 con 0x56016adf9800 2024-09-13T16:36:42.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7a780 con 0x56016adf9400 2024-09-13T16:36:42.585 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7a640 con 0x56016adf9000 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25de791700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016adf9800 0x56016ae9e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016adf9400 0x56016ae9eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016adf9400 0x56016ae9eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:54472/0 (socket says 172.21.15.119:54472) 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 -- 172.21.15.119:0/1944704566 learned_addr learned my addr 172.21.15.119:0/1944704566 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25def92700 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016adf9000 0x56016ae9f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 -- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016adf9000 msgr2=0x56016ae9f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016adf9000 0x56016ae9f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 -- 172.21.15.119:0/1944704566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016adf9800 msgr2=0x56016ae9e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016adf9800 0x56016ae9e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.586 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 -- 172.21.15.119:0/1944704566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56016ae1c9c0 con 0x56016adf9400 2024-09-13T16:36:42.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25ddf90700 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016adf9400 0x56016ae9eb00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x56016aeb98f0 tx=0x56016aeac840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c493a518602c5b10 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dbf8c700 1 -- 172.21.15.119:0/1944704566 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56016ae49600 con 0x56016adf9400 2024-09-13T16:36:42.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dbf8c700 1 -- 172.21.15.119:0/1944704566 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56016ae44180 con 0x56016adf9400 2024-09-13T16:36:42.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dbf8c700 1 -- 172.21.15.119:0/1944704566 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56016ae491e0 con 0x56016adf9400 2024-09-13T16:36:42.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 -- 172.21.15.119:0/1944704566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56016ae1cb60 con 0x56016adf9400 2024-09-13T16:36:42.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 1 -- 172.21.15.119:0/1944704566 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({fsmap=0}) v3 -- 0x56016ae1c340 con 0x56016adf9400 2024-09-13T16:36:42.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dffd79c0 4 waiting for MDS map... 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.581+0000 7f25dbf8c700 1 -- 172.21.15.119:0/1944704566 <== mon.1 v2:172.21.15.145:3300/0 4 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x56016ae32c40 con 0x56016adf9400 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dbf8c700 1 -- 172.21.15.119:0/1944704566 <== mon.1 v2:172.21.15.145:3300/0 5 ==== fsmap(e 99) v1 ==== 2573+0+0 (secure 0 0 0) 0x56016af5a300 con 0x56016adf9400 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 4 Got MDS map 99 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 4 datascan.main: Using metadata pool output 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 4 datascan.main: connecting to RADOS... 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 Processor -- start 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 -- start start 2024-09-13T16:36:42.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec400 0x56016ae9f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec800 0x56016ae9fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016ae2fe00 con 0x56016b0ecc00 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7aa00 con 0x56016b0ec400 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7a500 con 0x56016b0ec800 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25def92700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec400 0x56016ae9f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55368/0 (socket says 172.21.15.119:55368) 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 -- 172.21.15.119:0/3321760611 learned_addr learned my addr 172.21.15.119:0/3321760611 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25de791700 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec800 0x56016ae9fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 -- 172.21.15.119:0/3321760611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec800 msgr2=0x56016ae9fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec800 0x56016ae9fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 -- 172.21.15.119:0/3321760611 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec400 msgr2=0x56016ae9f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec400 0x56016ae9f600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 -- 172.21.15.119:0/3321760611 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56016ae1c000 con 0x56016b0ecc00 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25ddf90700 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0100 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x56016aeb9dd0 tx=0x56016ae668a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43ac83aed741a7f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25d9f88700 1 -- 172.21.15.119:0/3321760611 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56016ae48f20 con 0x56016b0ecc00 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25d9f88700 1 -- 172.21.15.119:0/3321760611 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56016ae44480 con 0x56016b0ecc00 2024-09-13T16:36:42.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.585+0000 7f25d9f88700 1 -- 172.21.15.119:0/3321760611 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56016ae49600 con 0x56016b0ecc00 2024-09-13T16:36:42.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- 172.21.15.119:0/3321760611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 msgr2=0x56016aea0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0100 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x56016aeb9dd0 tx=0x56016ae668a0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.596 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- 172.21.15.119:0/3321760611 shutdown_connections 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0100 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec800 0x56016ae9fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/3321760611 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec400 0x56016ae9f600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- 172.21.15.119:0/3321760611 >> 172.21.15.119:0/3321760611 conn(0x56016adf8c00 msgr2=0x56016aeb5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- 172.21.15.119:0/3321760611 shutdown_connections 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- 172.21.15.119:0/3321760611 wait complete. 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 Processor -- start 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- start start 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec400 0x56016aea1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec800 0x56016aea0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7a3c0 con 0x56016b0ecc00 2024-09-13T16:36:42.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7a280 con 0x56016b0ec800 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25dffd79c0 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x56016ae7a140 con 0x56016b0ec400 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25ddf90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25ddf90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55372/0 (socket says 172.21.15.119:55372) 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 learned_addr learned my addr 172.21.15.119:0/420463354 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec400 0x56016aea1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.589+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec800 0x56016aea0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec400 msgr2=0x56016aea1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016b0ec400 0x56016aea1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec800 msgr2=0x56016aea0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016b0ec800 0x56016aea0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56016ae1c9c0 con 0x56016b0ecc00 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0680 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x56016af5c960 tx=0x56016bc80c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.598 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25d8785700 1 -- 172.21.15.119:0/420463354 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56016ae491e0 con 0x56016b0ecc00 2024-09-13T16:36:42.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25d8785700 1 -- 172.21.15.119:0/420463354 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56016ae44180 con 0x56016b0ecc00 2024-09-13T16:36:42.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x56016ae1cb60 con 0x56016b0ecc00 2024-09-13T16:36:42.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25d8785700 1 -- 172.21.15.119:0/420463354 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x56016ae48dc0 con 0x56016b0ecc00 2024-09-13T16:36:42.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56016ae1c340 con 0x56016b0ecc00 2024-09-13T16:36:42.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25dffd79c0 4 datascan.init: resolving metadata pool 17 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25d8785700 1 -- 172.21.15.119:0/420463354 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x56016ae58840 con 0x56016b0ecc00 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25d8785700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56016b250000 0x56016aea1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25d8785700 1 -- 172.21.15.119:0/420463354 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x56016ae32a80 con 0x56016b0ecc00 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25dffd79c0 4 datascan.init: found metadata pool 'cephfs_metadata' 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25dffd79c0 4 datascan.main: resolving metadata pool 17 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56016b250000 0x56016aea1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56016b0ed000 0x56016aea0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.593+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:1 17.0 17:00000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b0ed400 con 0x56016b0ed000 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56016b0ed000 0x56016aea0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56016b250000 0x56016aea1700 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x56016aeb8450 tx=0x56016af5e060 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56016b0ed000 0x56016aea0100 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== osd_op_reply(1 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x56016b0ed400 0x56016b25e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:2 17.10 17:08000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b250400 con 0x56016b0ed400 2024-09-13T16:36:42.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x56016b0ed400 0x56016b25e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x56016b0ed400 0x56016b25e000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== osd_op_reply(2 [pgnls start_epoch 128 out=177b] v127'16 uv16 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:3 17.8 17:10000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b250400 con 0x56016b0ed000 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== osd_op_reply(3 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x56016b250400 0x56016b25e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:4 17.18 17:18000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b250800 con 0x56016b250400 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.597+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x56016b250400 0x56016b25e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x56016b250400 0x56016b25e580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== osd_op_reply(4 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:5 17.4 17:20000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b250800 con 0x56016b250400 2024-09-13T16:36:42.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== osd_op_reply(5 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:6 17.14 17:28000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b250800 con 0x56016b0ed400 2024-09-13T16:36:42.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== osd_op_reply(6 [pgnls start_epoch 128 out=149b] v128'42 uv42 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:7 17.c 17:30000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b250800 con 0x56016b0ed000 2024-09-13T16:36:42.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 3 ==== osd_op_reply(7 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56016b250800 0x56016b25eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:8 17.1c 17:38000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b250c00 con 0x56016b250800 2024-09-13T16:36:42.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56016b250800 0x56016b25eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56016b250800 0x56016b25eb00 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== osd_op_reply(8 [pgnls start_epoch 128 out=73b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x56016b250c00 0x56016b25f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.601+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:9 17.2 17:40000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251000 con 0x56016b250c00 2024-09-13T16:36:42.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x56016b250c00 0x56016b25f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x56016b250c00 0x56016b25f080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== osd_op_reply(9 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b250c00 2024-09-13T16:36:42.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:10 17.12 17:48000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251000 con 0x56016b0ed400 2024-09-13T16:36:42.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 3 ==== osd_op_reply(10 [pgnls start_epoch 128 out=81b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:11 17.a 17:50000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251000 con 0x56016b0ed000 2024-09-13T16:36:42.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 4 ==== osd_op_reply(11 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.609 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:12 17.1a 17:58000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251000 con 0x56016b250400 2024-09-13T16:36:42.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 3 ==== osd_op_reply(12 [pgnls start_epoch 128 out=49b] v128'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x56016b251000 0x56016b25f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:13 17.6 17:60000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251c00 con 0x56016b251000 2024-09-13T16:36:42.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.605+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x56016b251000 0x56016b25f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x56016b251000 0x56016b25f600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== osd_op_reply(13 [pgnls start_epoch 128 out=77b] v127'9 uv9 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25de791700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x56016b251c00 0x56016b25fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:14 17.16 17:68000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27a000 con 0x56016b251c00 2024-09-13T16:36:42.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x56016b251c00 0x56016b25fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x56016b251c00 0x56016b25fb80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== osd_op_reply(14 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25ddf90700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x56016b27a000 0x56016b260680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:42.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:15 17.e 17:70000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b27a000 2024-09-13T16:36:42.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x56016b27a000 0x56016b260680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:42.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.609+0000 7f25def92700 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x56016b27a000 0x56016b260680 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:42.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== osd_op_reply(15 [pgnls start_epoch 128 out=73b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78d80 con 0x56016b27a000 2024-09-13T16:36:42.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:16 17.1e 17:78000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b250800 2024-09-13T16:36:42.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== osd_op_reply(16 [pgnls start_epoch 128 out=49b] v128'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:17 17.1 17:80000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b250400 2024-09-13T16:36:42.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 4 ==== osd_op_reply(17 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:18 17.11 17:88000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b0ed400 2024-09-13T16:36:42.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 4 ==== osd_op_reply(18 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:19 17.9 17:90000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b0ed400 2024-09-13T16:36:42.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 5 ==== osd_op_reply(19 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:20 17.19 17:98000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b251c00 2024-09-13T16:36:42.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== osd_op_reply(20 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.613+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:21 17.5 17:a0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b251000 2024-09-13T16:36:42.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== osd_op_reply(21 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:22 17.15 17:a8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b251c00 2024-09-13T16:36:42.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 3 ==== osd_op_reply(22 [pgnls start_epoch 128 out=97b] v127'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:23 17.d 17:b0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b0ed000 2024-09-13T16:36:42.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 5 ==== osd_op_reply(23 [pgnls start_epoch 128 out=74b] v125'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:24 17.1d 17:b8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b0ed000 2024-09-13T16:36:42.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 6 ==== osd_op_reply(24 [pgnls start_epoch 128 out=73b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:25 17.3 17:c0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b250c00 2024-09-13T16:36:42.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== osd_op_reply(25 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b250c00 2024-09-13T16:36:42.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.617+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:26 17.13 17:c8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b0ed400 2024-09-13T16:36:42.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 6 ==== osd_op_reply(26 [pgnls start_epoch 128 out=98b] v127'5 uv5 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:27 17.b 17:d0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b250400 2024-09-13T16:36:42.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 5 ==== osd_op_reply(27 [pgnls start_epoch 128 out=76b] v128'6 uv5 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.624 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:28 17.1b 17:d8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b250c00 2024-09-13T16:36:42.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.4 v2:172.21.15.145:6816/4125431312 3 ==== osd_op_reply(28 [pgnls start_epoch 128 out=49b] v128'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b250c00 2024-09-13T16:36:42.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:29 17.7 17:e0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b250400 2024-09-13T16:36:42.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 6 ==== osd_op_reply(29 [pgnls start_epoch 128 out=105b] v128'9 uv9 ondisk = 1) v8 ==== 144+0+105 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.625 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:30 17.17 17:e8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b251000 2024-09-13T16:36:42.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 3 ==== osd_op_reply(30 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.626 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.621+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:31 17.f 17:f0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b250800 2024-09-13T16:36:42.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 3 ==== osd_op_reply(31 [pgnls start_epoch 128 out=73b] v128'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.627 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:32 17.1f 17:f8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b251000 2024-09-13T16:36:42.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 4 ==== osd_op_reply(32 [pgnls start_epoch 128 out=101b] v128'9 uv8 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 601.00000000 2024-09-13T16:36:42.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:33 17.10 17:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b0ed800 con 0x56016b0ed400 2024-09-13T16:36:42.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 7 ==== osd_op_reply(33 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 602.00000000 2024-09-13T16:36:42.629 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:34 17.10 17:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b000 con 0x56016b0ed400 2024-09-13T16:36:42.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 8 ==== osd_op_reply(34 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 10000000000.00000000 2024-09-13T16:36:42.630 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.625+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:35 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b400 con 0x56016b0ed400 2024-09-13T16:36:42.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 9 ==== osd_op_reply(35 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+1047 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 600.00000000 2024-09-13T16:36:42.631 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:36 17.10 17:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b800 con 0x56016b0ed400 2024-09-13T16:36:42.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 10 ==== osd_op_reply(36 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 603.00000000 2024-09-13T16:36:42.632 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:37 17.10 17:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27bc00 con 0x56016b0ed400 2024-09-13T16:36:42.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 11 ==== osd_op_reply(37 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 1.00000000.inode 2024-09-13T16:36:42.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-09-13T16:36:42.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 200.00000000 2024-09-13T16:36:42.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-09-13T16:36:42.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 200.00000001 2024-09-13T16:36:42.633 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-09-13T16:36:42.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 606.00000000 2024-09-13T16:36:42.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:38 17.14 17:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b400 con 0x56016b0ed400 2024-09-13T16:36:42.634 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 12 ==== osd_op_reply(38 606.00000000 [omap-get-vals out=5b] v0'0 uv37 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 607.00000000 2024-09-13T16:36:42.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.629+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:39 17.1c 17:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b800 con 0x56016b250800 2024-09-13T16:36:42.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 4 ==== osd_op_reply(39 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 10000000003.00000000 2024-09-13T16:36:42.635 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:40 17.12 17:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27bc00 con 0x56016b0ed400 2024-09-13T16:36:42.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 13 ==== osd_op_reply(40 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+523 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object mds0_openfiles.0 2024-09-13T16:36:42.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-09-13T16:36:42.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 608.00000000 2024-09-13T16:36:42.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:41 17.e 17:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b000 con 0x56016b27a000 2024-09-13T16:36:42.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== osd_op_reply(41 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b27a000 2024-09-13T16:36:42.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 604.00000000 2024-09-13T16:36:42.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:42 17.15 17:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b0ed800 con 0x56016b251c00 2024-09-13T16:36:42.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 4 ==== osd_op_reply(42 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 500.00000000 2024-09-13T16:36:42.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-09-13T16:36:42.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object mds_snaptable 2024-09-13T16:36:42.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-09-13T16:36:42.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 605.00000000 2024-09-13T16:36:42.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:43 17.1d 17:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284000 con 0x56016b0ed000 2024-09-13T16:36:42.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 7 ==== osd_op_reply(43 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object mds0_inotable 2024-09-13T16:36:42.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-09-13T16:36:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.633+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 100.00000000 2024-09-13T16:36:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:44 17.13 17:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284400 con 0x56016b0ed400 2024-09-13T16:36:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 14 ==== osd_op_reply(44 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object mds0_sessionmap 2024-09-13T16:36:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-09-13T16:36:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 609.00000000 2024-09-13T16:36:42.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:45 17.7 17:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284800 con 0x56016b250400 2024-09-13T16:36:42.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 7 ==== osd_op_reply(45 609.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 10000000001.00000000 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:46 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284c00 con 0x56016b250400 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 8 ==== osd_op_reply(46 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+1545 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 20 datascan.scan_links: injected ino detected 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 400.00000000 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 100.00000000.inode 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-09-13T16:36:42.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 10 datascan.scan_links: step 1: handling object 1.00000000 2024-09-13T16:36:42.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:47 17.1f 17:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285000 con 0x56016b251000 2024-09-13T16:36:42.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 5 ==== osd_op_reply(47 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+524 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.637+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:48 17.0 17:00000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed000 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 8 ==== osd_op_reply(48 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:49 17.10 17:08000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 15 ==== osd_op_reply(49 [pgnls start_epoch 128 out=177b] v127'16 uv16 ondisk = 1) v8 ==== 144+0+177 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:50 17.8 17:10000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed000 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 9 ==== osd_op_reply(50 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:51 17.18 17:18000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 9 ==== osd_op_reply(51 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:52 17.4 17:20000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 10 ==== osd_op_reply(52 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:53 17.14 17:28000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 16 ==== osd_op_reply(53 [pgnls start_epoch 128 out=149b] v128'42 uv42 ondisk = 1) v8 ==== 144+0+149 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.641+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:54 17.c 17:30000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed000 2024-09-13T16:36:42.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 10 ==== osd_op_reply(54 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:55 17.1c 17:38000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250800 2024-09-13T16:36:42.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 5 ==== osd_op_reply(55 [pgnls start_epoch 128 out=73b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:56 17.2 17:40000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250c00 2024-09-13T16:36:42.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.4 v2:172.21.15.145:6816/4125431312 4 ==== osd_op_reply(56 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b250c00 2024-09-13T16:36:42.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:57 17.12 17:48000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed400 2024-09-13T16:36:42.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 17 ==== osd_op_reply(57 [pgnls start_epoch 128 out=81b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+81 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:58 17.a 17:50000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed000 2024-09-13T16:36:42.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 11 ==== osd_op_reply(58 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.645+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:59 17.1a 17:58000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250400 2024-09-13T16:36:42.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 11 ==== osd_op_reply(59 [pgnls start_epoch 128 out=49b] v128'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:60 17.6 17:60000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b251000 2024-09-13T16:36:42.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 6 ==== osd_op_reply(60 [pgnls start_epoch 128 out=77b] v127'9 uv9 ondisk = 1) v8 ==== 144+0+77 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:61 17.16 17:68000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b251c00 2024-09-13T16:36:42.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 5 ==== osd_op_reply(61 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:62 17.e 17:70000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b27a000 2024-09-13T16:36:42.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.2 v2:172.21.15.119:6824/3093261484 3 ==== osd_op_reply(62 [pgnls start_epoch 128 out=73b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78d80 con 0x56016b27a000 2024-09-13T16:36:42.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:63 17.1e 17:78000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250800 2024-09-13T16:36:42.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 6 ==== osd_op_reply(63 [pgnls start_epoch 128 out=49b] v128'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.649+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:64 17.1 17:80000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250400 2024-09-13T16:36:42.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 12 ==== osd_op_reply(64 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:65 17.11 17:88000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed400 2024-09-13T16:36:42.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 18 ==== osd_op_reply(65 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:66 17.9 17:90000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed400 2024-09-13T16:36:42.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 19 ==== osd_op_reply(66 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:67 17.19 17:98000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b251c00 2024-09-13T16:36:42.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 6 ==== osd_op_reply(67 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:68 17.5 17:a0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b251000 2024-09-13T16:36:42.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 7 ==== osd_op_reply(68 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:69 17.15 17:a8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b251c00 2024-09-13T16:36:42.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 7 ==== osd_op_reply(69 [pgnls start_epoch 128 out=97b] v127'4 uv4 ondisk = 1) v8 ==== 144+0+97 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.658 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.653+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:70 17.d 17:b0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed000 2024-09-13T16:36:42.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 12 ==== osd_op_reply(70 [pgnls start_epoch 128 out=74b] v125'1 uv1 ondisk = 1) v8 ==== 144+0+74 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.659 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:71 17.1d 17:b8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed000 2024-09-13T16:36:42.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 13 ==== osd_op_reply(71 [pgnls start_epoch 128 out=73b] v127'3 uv3 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.660 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:72 17.3 17:c0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250c00 2024-09-13T16:36:42.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.4 v2:172.21.15.145:6816/4125431312 5 ==== osd_op_reply(72 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b250c00 2024-09-13T16:36:42.661 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:73 17.13 17:c8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b0ed400 2024-09-13T16:36:42.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 20 ==== osd_op_reply(73 [pgnls start_epoch 128 out=98b] v127'5 uv5 ondisk = 1) v8 ==== 144+0+98 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.657+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:74 17.b 17:d0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250400 2024-09-13T16:36:42.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 13 ==== osd_op_reply(74 [pgnls start_epoch 128 out=76b] v128'6 uv5 ondisk = 1) v8 ==== 144+0+76 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.663 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- osd_op(unknown.0.0:75 17.1b 17:d8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250c00 2024-09-13T16:36:42.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.4 v2:172.21.15.145:6816/4125431312 6 ==== osd_op_reply(75 [pgnls start_epoch 128 out=49b] v128'3 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78d80 con 0x56016b250c00 2024-09-13T16:36:42.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:76 17.7 17:e0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250400 2024-09-13T16:36:42.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 14 ==== osd_op_reply(76 [pgnls start_epoch 128 out=105b] v128'9 uv9 ondisk = 1) v8 ==== 144+0+105 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.664 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:77 17.17 17:e8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b251000 2024-09-13T16:36:42.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 8 ==== osd_op_reply(77 [pgnls start_epoch 128 out=49b] v0'0 uv0 ondisk = 1) v8 ==== 144+0+49 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.665 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:78 17.f 17:f0000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b250800 2024-09-13T16:36:42.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 7 ==== osd_op_reply(78 [pgnls start_epoch 128 out=73b] v128'2 uv2 ondisk = 1) v8 ==== 144+0+73 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.666 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.661+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:79 17.1f 17:f8000000::::head [pgnls start_epoch 128 in=39b] snapc 0=[] ondisk+read+ignore_overlay+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285400 con 0x56016b251000 2024-09-13T16:36:42.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 9 ==== osd_op_reply(79 [pgnls start_epoch 128 out=101b] v128'9 uv8 ondisk = 1) v8 ==== 144+0+101 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 601.00000000 2024-09-13T16:36:42.667 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:80 17.10 17:09401a81:::601.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285800 con 0x56016b0ed400 2024-09-13T16:36:42.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 21 ==== osd_op_reply(80 601.00000000 [omap-get-vals out=5b] v0'0 uv13 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 602.00000000 2024-09-13T16:36:42.668 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:81 17.10 17:0bd6d154:::602.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285c00 con 0x56016b0ed400 2024-09-13T16:36:42.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 22 ==== osd_op_reply(81 602.00000000 [omap-get-vals out=5b] v0'0 uv15 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 10000000000.00000000 2024-09-13T16:36:42.669 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:82 17.10 17:0cf6ad0f:::10000000000.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284000 con 0x56016b0ed400 2024-09-13T16:36:42.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 23 ==== osd_op_reply(82 10000000000.00000000 [omap-get-vals out=1047b] v0'0 uv12 ondisk = 0) v8 ==== 164+0+1047 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 600.00000000 2024-09-13T16:36:42.670 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.665+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:83 17.10 17:0d82a743:::600.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284800 con 0x56016b0ed400 2024-09-13T16:36:42.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 24 ==== osd_op_reply(83 600.00000000 [omap-get-vals out=5b] v0'0 uv14 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 603.00000000 2024-09-13T16:36:42.671 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:84 17.10 17:0d89b25e:::603.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284400 con 0x56016b0ed400 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 25 ==== osd_op_reply(84 603.00000000 [omap-get-vals out=5b] v0'0 uv16 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 1.00000000.inode 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: '1.00000000.inode' 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 200.00000000 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000000' 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 200.00000001 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '200.00000001' 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 606.00000000 2024-09-13T16:36:42.672 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:85 17.14 17:2f55791f:::606.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27ac00 con 0x56016b0ed400 2024-09-13T16:36:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 26 ==== osd_op_reply(85 606.00000000 [omap-get-vals out=5b] v0'0 uv37 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 607.00000000 2024-09-13T16:36:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- osd_op(unknown.0.0:86 17.1c 17:3ed09add:::607.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27a800 con 0x56016b250800 2024-09-13T16:36:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.1 v2:172.21.15.119:6816/1495422598 8 ==== osd_op_reply(86 607.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b250800 2024-09-13T16:36:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 10000000003.00000000 2024-09-13T16:36:42.674 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:87 17.12 17:4def72d4:::10000000003.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27a400 con 0x56016b0ed400 2024-09-13T16:36:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 27 ==== osd_op_reply(87 10000000003.00000000 [omap-get-vals out=523b] v0'0 uv3 ondisk = 0) v8 ==== 164+0+523 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object mds0_openfiles.0 2024-09-13T16:36:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds0_openfiles.0' 2024-09-13T16:36:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 608.00000000 2024-09-13T16:36:42.675 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- osd_op(unknown.0.0:88 17.e 17:717a0223:::608.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b400 con 0x56016b27a000 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.669+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.2 v2:172.21.15.119:6824/3093261484 4 ==== osd_op_reply(88 608.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78d80 con 0x56016b27a000 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 604.00000000 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- osd_op(unknown.0.0:89 17.15 17:a93a17c2:::604.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27bc00 con 0x56016b251c00 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25ddf90700 1 -- 172.21.15.119:0/420463354 <== osd.5 v2:172.21.15.145:6824/4068956621 8 ==== osd_op_reply(89 604.00000000 [omap-get-vals out=5b] v0'0 uv4 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78fc0 con 0x56016b251c00 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 500.00000000 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '500.00000000' 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object mds_snaptable 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds_snaptable' 2024-09-13T16:36:42.676 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 605.00000000 2024-09-13T16:36:42.677 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:90 17.1d 17:b871830b:::605.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b000 con 0x56016b0ed000 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 14 ==== osd_op_reply(90 605.00000000 [omap-get-vals out=5b] v0'0 uv3 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object mds0_inotable 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds0_inotable' 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 100.00000000 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:91 17.13 17:cd5a64a3:::100.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b251400 con 0x56016b0ed400 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 28 ==== osd_op_reply(91 100.00000000 [omap-get-vals out=5195b] v0'0 uv5 ondisk = 0) v8 ==== 156+0+5195 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object mds0_sessionmap 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: 'mds0_sessionmap' 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 609.00000000 2024-09-13T16:36:42.678 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:92 17.7 17:e0b41d06:::609.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285800 con 0x56016b250400 2024-09-13T16:36:42.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 15 ==== osd_op_reply(92 609.00000000 [omap-get-vals out=5b] v0'0 uv8 ondisk = 0) v8 ==== 156+0+5 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 10000000001.00000000 2024-09-13T16:36:42.679 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.673+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:93 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b285c00 con 0x56016b250400 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 16 ==== osd_op_reply(93 10000000001.00000000 [omap-get-vals out=1545b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+1545 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 20 datascan.scan_links: injected ino detected 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 20 datascan.scan_links: adding 0x100000001f7 for future processing to fix dnfirst 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 400.00000000 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag (invalid ino): '400.00000000' 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 100.00000000.inode 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: Not a dirfrag: '100.00000000.inode' 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: step 2: handling object 1.00000000 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- osd_op(unknown.0.0:94 17.1f 17:ff5b34d6:::1.00000000:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284000 con 0x56016b251000 2024-09-13T16:36:42.681 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.6 v2:172.21.15.145:6800/3152548227 10 ==== osd_op_reply(94 1.00000000 [omap-get-vals out=524b] v0'0 uv8 ondisk = 0) v8 ==== 154+0+524 (crc 0 0 0) 0x56016bc78b40 con 0x56016b251000 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: processing 0 dup_primaries, 0 remote_links 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: removing dup dentries from 0 objects 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: processing 0 bad_nlink_inos 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: processing 1 injected_inos 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 10 datascan.scan_links: handling injected_ino 0x100000001f7 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 20 datascan.scan_links: fixing linkage (dnfirst) of 0x10000000001:bravo 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:95 17.7 17:e2e07930:::10000000001.00000000:head [omap-get-vals-by-keys in=18b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284800 con 0x56016b250400 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 17 ==== osd_op_reply(95 10000000001.00000000 [omap-get-vals-by-keys out=506b] v0'0 uv9 ondisk = 0) v8 ==== 164+0+506 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 20 datascan.read_dentry: oid=10000000001.00000000 dname=bravo frag=*, r=0 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 20 datascan.scan_links: first is now 2 2024-09-13T16:36:42.682 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.677+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- osd_op(unknown.0.0:96 17.7 17:e2e07930:::10000000001.00000000:head [omap-set-vals in=506b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b284400 con 0x56016b250400 2024-09-13T16:36:42.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.0 v2:172.21.15.119:6800/2655437631 18 ==== osd_op_reply(96 10000000001.00000000 [omap-set-vals] v128'10 uv10 ondisk = 0) v8 ==== 164+0+0 (crc 0 0 0) 0x56016bc78b40 con 0x56016b250400 2024-09-13T16:36:42.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25dffd79c0 20 datascan.inject_linkage: Injected dentry 0x0x10000000001/bravo pointing to 0x0x100000001f7 2024-09-13T16:36:42.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25dffd79c0 10 datascan.scan_links: updating inotable 2024-09-13T16:36:42.683 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:97 17.13 17:c91d4a1d:::mds0_inotable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27b800 con 0x56016b0ed400 2024-09-13T16:36:42.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 29 ==== osd_op_reply(97 mds0_inotable [read 0~50 out=50b] v0'0 uv4 ondisk = 0) v8 ==== 157+0+50 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25dffd79c0 10 mds.0.inotable: skip_inos was [0x10000000005~0x1f2,0x100000001f9~0xfffffffe07] 2024-09-13T16:36:42.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25dffd79c0 10 mds.0.inotable: skip_inos now [0x100000001f9~0xfffffffe07] 2024-09-13T16:36:42.684 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.681+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- osd_op(unknown.0.0:98 17.13 17:c91d4a1d:::mds0_inotable:head [writefull 0~34 in=34b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27a800 con 0x56016b0ed400 2024-09-13T16:36:42.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25def92700 1 -- 172.21.15.119:0/420463354 <== osd.3 v2:172.21.15.119:6808/3803418857 30 ==== osd_op_reply(98 mds0_inotable [writefull 0~34] v128'6 uv6 ondisk = 0) v8 ==== 157+0+0 (crc 0 0 0) 0x56016bc78d80 con 0x56016b0ed400 2024-09-13T16:36:42.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 10 datascan.scan_links: updating snaptable 2024-09-13T16:36:42.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- osd_op(unknown.0.0:99 17.d 17:b50e409b:::mds_snaptable:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e128) v8 -- 0x56016b27bc00 con 0x56016b0ed000 2024-09-13T16:36:42.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 <== osd.7 v2:172.21.15.145:6808/1049327118 15 ==== osd_op_reply(99 mds_snaptable [read 0~70 out=70b] v0'0 uv1 ondisk = 0) v8 ==== 157+0+70 (crc 0 0 0) 0x56016bc78b40 con 0x56016b0ed000 2024-09-13T16:36:42.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x56016b250400 msgr2=0x56016b25e580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.690 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x56016b250400 0x56016b25e580 crc :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56016b250800 msgr2=0x56016b25eb00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x56016b250800 0x56016b25eb00 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x56016b27a000 msgr2=0x56016b260680 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x56016b27a000 0x56016b260680 crc :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x56016b0ed400 msgr2=0x56016b25e000 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x56016b0ed400 0x56016b25e000 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 reap_dead start 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x56016b250c00 msgr2=0x56016b25f080 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x56016b250c00 0x56016b25f080 crc :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 reap_dead start 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x56016b251c00 msgr2=0x56016b25fb80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x56016b251c00 0x56016b25fb80 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x56016b251000 msgr2=0x56016b25f600 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x56016b251000 0x56016b25f600 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56016b0ed000 msgr2=0x56016aea0100 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x56016b0ed000 0x56016aea0100 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56016b250000 msgr2=0x56016aea1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x56016b250000 0x56016aea1700 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x56016aeb8450 tx=0x56016af5e060 comp rx=0 tx=0).stop 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 msgr2=0x56016aea0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/420463354 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016b0ecc00 0x56016aea0680 secure :-1 s=READY pgs=1957 cs=0 l=1 rev1=1 crypto rx=0x56016af5c960 tx=0x56016bc80c00 comp rx=0 tx=0).stop 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25de791700 1 -- 172.21.15.119:0/420463354 reap_dead start 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 shutdown_connections 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 >> 172.21.15.119:0/420463354 conn(0x56016adf8c00 msgr2=0x56016aeb5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 shutdown_connections 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/420463354 wait complete. 2024-09-13T16:36:42.693 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016adf9400 msgr2=0x56016ae9eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016adf9400 0x56016ae9eb00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x56016aeb98f0 tx=0x56016aeac840 comp rx=0 tx=0).stop 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/1944704566 shutdown_connections 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x56016adf9000 0x56016ae9f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x56016adf9400 0x56016ae9eb00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 --2- 172.21.15.119:0/1944704566 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x56016adf9800 0x56016ae9e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/1944704566 >> 172.21.15.119:0/1944704566 conn(0x56016adf9c00 msgr2=0x56016aeb5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/1944704566 shutdown_connections 2024-09-13T16:36:42.694 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:42.685+0000 7f25dffd79c0 1 -- 172.21.15.119:0/1944704566 wait complete. 2024-09-13T16:36:42.696 DEBUG:tasks.cephfs.filesystem:Ran ['cephfs-data-scan', '--debug-mds=20', '--debug-ms=1', '--debug-objecter=1', 'scan_links'] in time 0:00:00.152698, result: 2024-09-13T16:36:42.697 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:42.700 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.733 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.734 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.735 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.736 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-09-13T16:36:42.737 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-09-13T16:36:42.738 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:42.739 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-09-13T16:36:42.740 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.742 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:42.743 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-09-13T16:36:42.743 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:42.743 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-09-13T16:36:42.743 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:42.743 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.743 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:42.743 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-09-13T16:36:42.744 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.744 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.744 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:42.744 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:42.744 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:42.745 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.745 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:42.745 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:42.745 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:42.745 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.745 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-09-13T16:36:42.745 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: mon op complaint time = 120 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on pool no app = False 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-09-13T16:36:42.746 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: client mount timeout = 600 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = "" 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-09-13T16:36:42.747 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds = 20 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds balancer = 20 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug frag = True 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug scatterstat = False 2024-09-13T16:36:42.748 INFO:teuthology.orchestra.run.smithi119.stdout: mds op complaint time = 180 2024-09-13T16:36:42.749 INFO:teuthology.orchestra.run.smithi119.stdout: mds verify scatter = True 2024-09-13T16:36:42.749 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:42.749 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.749 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.751 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-09-13T16:36:42.751 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-09-13T16:36:42.751 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:42.751 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-09-13T16:36:42.752 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.752 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-09-13T16:36:42.752 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.752 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:42.752 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.752 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:42.752 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:42.753 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:42.754 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.754 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-09-13T16:36:42.754 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.754 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:42.754 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-09-13T16:36:42.754 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-09-13T16:36:42.755 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:42.756 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-09-13T16:36:42.757 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.758 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2024-09-13T16:36:42.759 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.760 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.761 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.762 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-09-13T16:36:42.763 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:42.763 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.763 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:42.763 INFO:teuthology.orchestra.run.smithi145.stdout: mon op complaint time = 120 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on pool no app = False 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:42.764 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout: client mount timeout = 600 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = "" 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-09-13T16:36:42.765 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout:[mds] 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds = 20 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds balancer = 20 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug frag = True 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug scatterstat = False 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout: mds op complaint time = 180 2024-09-13T16:36:42.766 INFO:teuthology.orchestra.run.smithi145.stdout: mds verify scatter = True 2024-09-13T16:36:42.767 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:42.767 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.767 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.768 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:42.797 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-13T16:36:42.830 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2024-09-13T16:36:42.830 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2024-09-13T16:36:42.831 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:42.832 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2024-09-13T16:36:42.833 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.834 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2024-09-13T16:36:42.835 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 20 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 20 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2024-09-13T16:36:42.836 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 10 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.837 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2024-09-13T16:36:42.838 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:42.839 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: mon op complaint time = 120 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on pool no app = False 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2024-09-13T16:36:42.840 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout: client mount timeout = 600 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = "" 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2024-09-13T16:36:42.841 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds = 20 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds balancer = 20 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug frag = True 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug scatterstat = True 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: mds op complaint time = 180 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: mds verify scatter = True 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.842 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.849 INFO:teuthology.orchestra.run.smithi145.stdout:[global] 2024-09-13T16:36:42.849 INFO:teuthology.orchestra.run.smithi145.stdout: chdir = "" 2024-09-13T16:36:42.854 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-13T16:36:42.854 INFO:teuthology.orchestra.run.smithi145.stdout: auth supported = cephx 2024-09-13T16:36:42.854 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.854 INFO:teuthology.orchestra.run.smithi145.stdout: filestore xattr use omap = true 2024-09-13T16:36:42.854 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: mon clock drift allowed = 1.000 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: osd crush chooseleaf type = 0 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: auth debug = true 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on old message = true 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: ms die on bug = true 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.855 INFO:teuthology.orchestra.run.smithi145.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: mon pg warn max object skew = 0 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: # disable pg_autoscaler by default for new pools 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default size = 2 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow primary affinity = true 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd allow pg remap = true 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on legacy crush tunables = false 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on crush straw calc version zero = false 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on no sortbitwise = false 2024-09-13T16:36:42.856 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on osd down out interval zero = false 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on too few osds = false 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: mon_allow_pool_size_one = true 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: osd default data pool replay window = 5 2024-09-13T16:36:42.857 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: mon allow pool delete = true 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: mon cluster log file level = debug 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: debug asserts on shutdown = true 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: mon health detail to clog = false 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: mon host = "172.21.15.119,172.21.15.145,[v2:172.21.15.145:3301,v1:172.21.15.145:6790]" 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: fsid = 01de39e4-2965-4196-9df7-d7f8aeef36df 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout:[osd] 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: osd journal size = 100 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub load threshold = 5.0 2024-09-13T16:36:42.858 INFO:teuthology.orchestra.run.smithi145.stdout: osd scrub max interval = 600 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd mclock profile = high_recovery_ops 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd recover clone overlap = true 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd recovery max chunk = 1048576 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug shutdown = true 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug op order = true 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug verify stray on activate = true 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd open classes on start = true 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug pg log writeout = true 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: osd deep scrub update digest min age = 30 2024-09-13T16:36:42.859 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: osd map max advance = 10 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: journal zero on create = true 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: filestore ondisk finisher threads = 3 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: filestore apply finisher threads = 3 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: bdev debug aio = true 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: osd debug misdirected ops = true 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore block size = 96636764160 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: bluestore fsck on mount = True 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluefs = 20 2024-09-13T16:36:42.860 INFO:teuthology.orchestra.run.smithi145.stdout: debug bluestore = 20 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: debug osd = 20 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: debug rocksdb = 10 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd backfillfull_ratio = 0.85 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd full ratio = 0.9 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd nearfull ratio = 0.8 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: osd failsafe full ratio = 0.95 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: osd objectstore = bluestore 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout:[mgr] 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: debug mgr = 20 2024-09-13T16:36:42.861 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: mgr/telemetry/nag = false 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout:[mon] 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: debug mon = 20 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: debug paxos = 20 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: debug auth = 20 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: mon data avail warn = 5 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: mon mgr mkfs grace = 240 2024-09-13T16:36:42.862 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min pgs per osd = 4 2024-09-13T16:36:42.863 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd reporter subtree level = osd 2024-09-13T16:36:42.863 INFO:teuthology.orchestra.run.smithi145.stdout: mon osd prime pg temp = true 2024-09-13T16:36:42.863 INFO:teuthology.orchestra.run.smithi145.stdout: mon reweight min bytes per osd = 10 2024-09-13T16:36:42.863 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: auth mon ticket ttl = 660 # 11m 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: auth service ticket ttl = 240 # 4m 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: # don't complain about insecure global_id in the test suite 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: # 1m isn't quite enough 2024-09-13T16:36:42.864 INFO:teuthology.orchestra.run.smithi145.stdout: mon_down_mkfs_grace = 2m 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon_warn_on_filestore_osds = false 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon op complaint time = 120 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: mon warn on pool no app = False 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout:[client] 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: rgw cache enabled = true 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable ops log = true 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: rgw enable usage log = true 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: client mount timeout = 600 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: debug client = 20 2024-09-13T16:36:42.865 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout: pid file = "" 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.a] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.b] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mon.c] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.a] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.c] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.b] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mds.d] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout:[mds] 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds = 20 2024-09-13T16:36:42.866 INFO:teuthology.orchestra.run.smithi145.stdout: debug mds balancer = 20 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: debug ms = 1 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug frag = True 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: mds debug scatterstat = True 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: mds op complaint time = 180 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: mds verify scatter = True 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: osd op complaint time = 180 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: rados mon op timeout = 900 2024-09-13T16:36:42.867 INFO:teuthology.orchestra.run.smithi145.stdout: rados osd op timeout = 900 2024-09-13T16:36:42.868 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs joinable true 2024-09-13T16:36:43.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2da67700 1 Processor -- start 2024-09-13T16:36:43.037 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2da67700 1 -- start start 2024-09-13T16:36:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2da67700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffa2815cad0 0x7ffa280d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2da67700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280ce0d0 0x7ffa280ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2da67700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa280d05b0 con 0x7ffa280cf220 2024-09-13T16:36:43.038 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2da67700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa280d0870 con 0x7ffa2815cad0 2024-09-13T16:36:43.039 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2da67700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280d0b50 con 0x7ffa280ce0d0 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa27fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280ce0d0 0x7ffa280ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa27fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280ce0d0 0x7ffa280ce510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:57118/0 (socket says 172.21.15.119:57118) 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa2ca65700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ffa2815cad0 0x7ffa280d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.033+0000 7ffa27fff700 1 -- 172.21.15.119:0/4054833765 learned_addr learned my addr 172.21.15.119:0/4054833765 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa27fff700 1 -- 172.21.15.119:0/4054833765 >> v2:172.21.15.145:3300/0 conn(0x7ffa2815cad0 msgr2=0x7ffa280d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa27fff700 1 --2- 172.21.15.119:0/4054833765 >> v2:172.21.15.145:3300/0 conn(0x7ffa2815cad0 0x7ffa280d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa27fff700 1 -- 172.21.15.119:0/4054833765 >> v1:172.21.15.145:6789/0 conn(0x7ffa280cf220 legacy=0x7ffa280b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:43.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa27fff700 1 -- 172.21.15.119:0/4054833765 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa280d0e30 con 0x7ffa280ce0d0 2024-09-13T16:36:43.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa27fff700 1 --2- 172.21.15.119:0/4054833765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280ce0d0 0x7ffa280ce510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009910 tx=0x7ffa18009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf10a9274162ed18 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:43.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa277fe700 1 -- 172.21.15.119:0/4054833765 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18011830 con 0x7ffa280ce0d0 2024-09-13T16:36:43.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa277fe700 1 -- 172.21.15.119:0/4054833765 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa18011990 con 0x7ffa280ce0d0 2024-09-13T16:36:43.041 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa277fe700 1 -- 172.21.15.119:0/4054833765 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa18011d40 con 0x7ffa280ce0d0 2024-09-13T16:36:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 -- 172.21.15.119:0/4054833765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280ce0d0 msgr2=0x7ffa280ce510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 --2- 172.21.15.119:0/4054833765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280ce0d0 0x7ffa280ce510 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffa18009910 tx=0x7ffa18009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 -- 172.21.15.119:0/4054833765 shutdown_connections 2024-09-13T16:36:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 --2- 172.21.15.119:0/4054833765 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280ce0d0 0x7ffa280ce510 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 --2- 172.21.15.119:0/4054833765 >> v2:172.21.15.145:3300/0 conn(0x7ffa2815cad0 0x7ffa280d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 -- 172.21.15.119:0/4054833765 >> 172.21.15.119:0/4054833765 conn(0x7ffa2808bf70 msgr2=0x7ffa280d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:43.042 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 -- 172.21.15.119:0/4054833765 shutdown_connections 2024-09-13T16:36:43.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.037+0000 7ffa2da67700 1 -- 172.21.15.119:0/4054833765 wait complete. 2024-09-13T16:36:43.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 Processor -- start 2024-09-13T16:36:43.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 -- start start 2024-09-13T16:36:43.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa280ce0d0 0x7ffa28152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280cf220 0x7ffa281f3350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa2815cad0 0x7ffa280c5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280c68f0 con 0x7ffa2815cad0 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280c6bd0 con 0x7ffa280ce0d0 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa280c6eb0 con 0x7ffa280cf220 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa2815cad0 0x7ffa280c5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa2815cad0 0x7ffa280c5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55378/0 (socket says 172.21.15.119:55378) 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 -- 172.21.15.119:0/297353688 learned_addr learned my addr 172.21.15.119:0/297353688 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa27fff700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280cf220 0x7ffa281f3350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:43.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2ca65700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa280ce0d0 0x7ffa28152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 -- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280cf220 msgr2=0x7ffa281f3350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280cf220 0x7ffa281f3350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 -- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa280ce0d0 msgr2=0x7ffa28152a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa280ce0d0 0x7ffa28152a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 -- 172.21.15.119:0/297353688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa18009570 con 0x7ffa2815cad0 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2d266700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa2815cad0 0x7ffa280c5ff0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000cc50 tx=0x7ffa2000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa257fa700 1 -- 172.21.15.119:0/297353688 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa20003c20 con 0x7ffa2815cad0 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa257fa700 1 -- 172.21.15.119:0/297353688 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa20003d80 con 0x7ffa2815cad0 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa280c7190 con 0x7ffa2815cad0 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa257fa700 1 -- 172.21.15.119:0/297353688 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa2000aab0 con 0x7ffa2815cad0 2024-09-13T16:36:43.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.041+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa280c6530 con 0x7ffa2815cad0 2024-09-13T16:36:43.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.045+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa280c76b0 con 0x7ffa2815cad0 2024-09-13T16:36:43.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.045+0000 7ffa257fa700 1 -- 172.21.15.119:0/297353688 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ffa2000ac10 con 0x7ffa2815cad0 2024-09-13T16:36:43.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.045+0000 7ffa257fa700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa14066730 0x7ffa14068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:43.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.045+0000 7ffa257fa700 1 -- 172.21.15.119:0/297353688 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7ffa20084240 con 0x7ffa2815cad0 2024-09-13T16:36:43.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.045+0000 7ffa2ca65700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa14066730 0x7ffa14068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:43.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.049+0000 7ffa2ca65700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa14066730 0x7ffa14068bf0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009b50 tx=0x7ffa1c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:43.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.053+0000 7ffa257fa700 1 -- 172.21.15.119:0/297353688 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ffa280c76b0 con 0x7ffa2815cad0 2024-09-13T16:36:43.263 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:43.261+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"} v 0) v1 -- 0x7ffa280c76b0 con 0x7ffa2815cad0 2024-09-13T16:36:44.131 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.129+0000 7ffa257fa700 1 -- 172.21.15.119:0/297353688 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "joinable", "val": "true"}]=0 cephfs marked joinable; MDS may join as newly active. v101) v1 ==== 162+0+0 (secure 0 0 0) 0x7ffa280c76b0 con 0x7ffa2815cad0 2024-09-13T16:36:44.135 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa14066730 msgr2=0x7ffa14068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa14066730 0x7ffa14068bf0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009b50 tx=0x7ffa1c009240 comp rx=0 tx=0).stop 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa2815cad0 msgr2=0x7ffa280c5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa2815cad0 0x7ffa280c5ff0 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000cc50 tx=0x7ffa2000cf60 comp rx=0 tx=0).stop 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 shutdown_connections 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ffa14066730 0x7ffa14068bf0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ffa2815cad0 0x7ffa280c5ff0 unknown :-1 s=CLOSED pgs=1958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ffa280cf220 0x7ffa281f3350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 --2- 172.21.15.119:0/297353688 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ffa280ce0d0 0x7ffa28152a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 >> 172.21.15.119:0/297353688 conn(0x7ffa2808bf70 msgr2=0x7ffa2806a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 shutdown_connections 2024-09-13T16:36:44.136 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.133+0000 7ffa2da67700 1 -- 172.21.15.119:0/297353688 wait complete. 2024-09-13T16:36:44.137 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked joinable; MDS may join as newly active. 2024-09-13T16:36:44.153 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:44.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 Processor -- start 2024-09-13T16:36:44.270 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 -- start start 2024-09-13T16:36:44.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe0340b1910 0x7fe034156530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:44.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe03406dcc0 0x7fe03406bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:44.271 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe0340b2320 0x7fe0340ad710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340b2c70 con 0x7fe0340b1910 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0340b2f50 con 0x7fe0340b2320 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.265+0000 7fe038b84700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0340b3230 con 0x7fe03406dcc0 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe0340b1910 0x7fe034156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe0340b1910 0x7fe034156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55394/0 (socket says 172.21.15.119:55394) 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 -- 172.21.15.119:0/3751997761 learned_addr learned my addr 172.21.15.119:0/3751997761 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe032ffd700 1 --2- 172.21.15.119:0/3751997761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe03406dcc0 0x7fe03406bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe033fff700 1 --2- 172.21.15.119:0/3751997761 >> v2:172.21.15.145:3300/0 conn(0x7fe0340b2320 0x7fe0340ad710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 -- 172.21.15.119:0/3751997761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe03406dcc0 msgr2=0x7fe03406bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 --2- 172.21.15.119:0/3751997761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe03406dcc0 0x7fe03406bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 -- 172.21.15.119:0/3751997761 >> v2:172.21.15.145:3300/0 conn(0x7fe0340b2320 msgr2=0x7fe0340ad710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:44.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 --2- 172.21.15.119:0/3751997761 >> v2:172.21.15.145:3300/0 conn(0x7fe0340b2320 0x7fe0340ad710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 -- 172.21.15.119:0/3751997761 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe034070c80 con 0x7fe0340b1910 2024-09-13T16:36:44.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0337fe700 1 --2- 172.21.15.119:0/3751997761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0340b1910 0x7fe034156530 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fe0340ae410 tx=0x7fe028009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fa23989087b7f0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0327fc700 1 -- 172.21.15.119:0/3751997761 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028011830 con 0x7fe0340b1910 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0327fc700 1 -- 172.21.15.119:0/3751997761 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe028011990 con 0x7fe0340b1910 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe0327fc700 1 -- 172.21.15.119:0/3751997761 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028011d20 con 0x7fe0340b1910 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 -- 172.21.15.119:0/3751997761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0340b1910 msgr2=0x7fe034156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 --2- 172.21.15.119:0/3751997761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0340b1910 0x7fe034156530 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fe0340ae410 tx=0x7fe028009bf0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 -- 172.21.15.119:0/3751997761 shutdown_connections 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 --2- 172.21.15.119:0/3751997761 >> v2:172.21.15.145:3300/0 conn(0x7fe0340b2320 0x7fe0340ad710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 --2- 172.21.15.119:0/3751997761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe03406dcc0 0x7fe03406bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 --2- 172.21.15.119:0/3751997761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe0340b1910 0x7fe034156530 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 -- 172.21.15.119:0/3751997761 >> 172.21.15.119:0/3751997761 conn(0x7fe03406a350 msgr2=0x7fe03406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:44.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 -- 172.21.15.119:0/3751997761 shutdown_connections 2024-09-13T16:36:44.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.269+0000 7fe038b84700 1 -- 172.21.15.119:0/3751997761 wait complete. 2024-09-13T16:36:44.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 Processor -- start 2024-09-13T16:36:44.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 -- start start 2024-09-13T16:36:44.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe03406dcc0 0x7fe03415aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0340b1910 0x7fe034158d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0340b2320 0x7fe03415c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03415b110 con 0x7fe03406dcc0 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe03415a740 con 0x7fe0340b1910 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe034159280 con 0x7fe0340b2320 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe03406dcc0 0x7fe03415aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe03406dcc0 0x7fe03415aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55404/0 (socket says 172.21.15.119:55404) 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 -- 172.21.15.119:0/1688375315 learned_addr learned my addr 172.21.15.119:0/1688375315 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe032ffd700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0340b1910 0x7fe034158d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe033fff700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0340b2320 0x7fe03415c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:44.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 -- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0340b2320 msgr2=0x7fe03415c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:44.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0340b2320 0x7fe03415c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 -- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0340b1910 msgr2=0x7fe034158d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:44.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0340b1910 0x7fe034158d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 -- 172.21.15.119:0/1688375315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028009570 con 0x7fe03406dcc0 2024-09-13T16:36:44.279 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe0337fe700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe03406dcc0 0x7fe03415aa10 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fe028000c00 tx=0x7fe028004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:44.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe01bfff700 1 -- 172.21.15.119:0/1688375315 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe028004450 con 0x7fe03406dcc0 2024-09-13T16:36:44.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe01bfff700 1 -- 172.21.15.119:0/1688375315 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0280045b0 con 0x7fe03406dcc0 2024-09-13T16:36:44.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe03415a180 con 0x7fe03406dcc0 2024-09-13T16:36:44.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe01bfff700 1 -- 172.21.15.119:0/1688375315 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0280038e0 con 0x7fe03406dcc0 2024-09-13T16:36:44.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.273+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0341538d0 con 0x7fe03406dcc0 2024-09-13T16:36:44.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.277+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe03406bb90 con 0x7fe03406dcc0 2024-09-13T16:36:44.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.277+0000 7fe01bfff700 1 -- 172.21.15.119:0/1688375315 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe028021070 con 0x7fe03406dcc0 2024-09-13T16:36:44.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.277+0000 7fe01bfff700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe020066730 0x7fe020068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:44.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.277+0000 7fe01bfff700 1 -- 172.21.15.119:0/1688375315 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7fe028085750 con 0x7fe03406dcc0 2024-09-13T16:36:44.285 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.277+0000 7fe032ffd700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe020066730 0x7fe020068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:44.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.281+0000 7fe032ffd700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe020066730 0x7fe020068bf0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe03414cbb0 tx=0x7fe024009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:44.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.285+0000 7fe01bfff700 1 -- 172.21.15.119:0/1688375315 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe0280528c0 con 0x7fe03406dcc0 2024-09-13T16:36:44.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0341450b0 con 0x7fe03406dcc0 2024-09-13T16:36:44.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe01bfff700 1 -- 172.21.15.119:0/1688375315 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fe028055ee0 con 0x7fe03406dcc0 2024-09-13T16:36:44.496 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:44.496 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":102,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":102,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:44.131780+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9646":{"gid":9646,"name":"b","rank":0,"incarnation":102,"state":"up:replay","state_seq":1,"addr":"172.21.15.145:6835/2905269684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2905269684},{"type":"v1","addr":"172.21.15.145:6835","nonce":2905269684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:44.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe020066730 msgr2=0x7fe020068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:44.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe020066730 0x7fe020068bf0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fe03414cbb0 tx=0x7fe024009240 comp rx=0 tx=0).stop 2024-09-13T16:36:44.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe03406dcc0 msgr2=0x7fe03415aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:44.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe03406dcc0 0x7fe03415aa10 secure :-1 s=READY pgs=1960 cs=0 l=1 rev1=1 crypto rx=0x7fe028000c00 tx=0x7fe028004770 comp rx=0 tx=0).stop 2024-09-13T16:36:44.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 shutdown_connections 2024-09-13T16:36:44.499 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe020066730 0x7fe020068bf0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe0340b2320 0x7fe03415c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe0340b1910 0x7fe034158d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 --2- 172.21.15.119:0/1688375315 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe03406dcc0 0x7fe03415aa10 unknown :-1 s=CLOSED pgs=1960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 >> 172.21.15.119:0/1688375315 conn(0x7fe03406a350 msgr2=0x7fe0341442c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 shutdown_connections 2024-09-13T16:36:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:44.493+0000 7fe038b84700 1 -- 172.21.15.119:0/1688375315 wait complete. 2024-09-13T16:36:44.500 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 102 2024-09-13T16:36:44.920 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 102, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:35:55.244835+0000', 'modified': '2024-09-13T16:36:44.131780+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 128, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9646}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9646': {'gid': 9646, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:replay', 'state_seq': 1, 'addr': '172.21.15.145:6835/2905269684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2905269684}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 2905269684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:36:44.920 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9646:up:replay 2024-09-13T16:36:45.920 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:46.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 Processor -- start 2024-09-13T16:36:46.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- start start 2024-09-13T16:36:46.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5c4815a620 0x7f5c480b3720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5c481612f0 0x7f5c480ada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c480ac960 con 0x7f5c480adf40 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c480acc40 con 0x7f5c4815a620 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c480acf20 con 0x7f5c481612f0 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5c4815a620 0x7f5c480b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5c4815a620 0x7f5c480b3720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55416/0 (socket says 172.21.15.119:55416) 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 -- 172.21.15.119:0/2107748176 learned_addr learned my addr 172.21.15.119:0/2107748176 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c477fe700 1 --2- 172.21.15.119:0/2107748176 >> v2:172.21.15.145:3300/0 conn(0x7f5c481612f0 0x7f5c480ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 -- 172.21.15.119:0/2107748176 >> v2:172.21.15.145:3300/0 conn(0x7f5c481612f0 msgr2=0x7f5c480ada00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 --2- 172.21.15.119:0/2107748176 >> v2:172.21.15.145:3300/0 conn(0x7f5c481612f0 0x7f5c480ada00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 -- 172.21.15.119:0/2107748176 >> v1:172.21.15.119:6789/0 conn(0x7f5c480adf40 legacy=0x7f5c4806fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:46.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 -- 172.21.15.119:0/2107748176 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c480ad200 con 0x7f5c4815a620 2024-09-13T16:36:46.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c47fff700 1 --2- 172.21.15.119:0/2107748176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c4815a620 0x7f5c480b3720 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806cde0 tx=0x7f5c4000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f355b02faff107f7 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:46.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c46ffd700 1 -- 172.21.15.119:0/2107748176 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000e070 con 0x7f5c4815a620 2024-09-13T16:36:46.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c46ffd700 1 -- 172.21.15.119:0/2107748176 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4000b9c0 con 0x7f5c4815a620 2024-09-13T16:36:46.046 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c46ffd700 1 -- 172.21.15.119:0/2107748176 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4000bd70 con 0x7f5c4815a620 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- 172.21.15.119:0/2107748176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c4815a620 msgr2=0x7f5c480b3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 --2- 172.21.15.119:0/2107748176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c4815a620 0x7f5c480b3720 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f5c4806cde0 tx=0x7f5c4000b330 comp rx=0 tx=0).stop 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- 172.21.15.119:0/2107748176 shutdown_connections 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 --2- 172.21.15.119:0/2107748176 >> v2:172.21.15.145:3300/0 conn(0x7f5c481612f0 0x7f5c480ada00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 --2- 172.21.15.119:0/2107748176 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c4815a620 0x7f5c480b3720 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- 172.21.15.119:0/2107748176 >> 172.21.15.119:0/2107748176 conn(0x7f5c4806a350 msgr2=0x7f5c4806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- 172.21.15.119:0/2107748176 shutdown_connections 2024-09-13T16:36:46.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.041+0000 7f5c4e379700 1 -- 172.21.15.119:0/2107748176 wait complete. 2024-09-13T16:36:46.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 Processor -- start 2024-09-13T16:36:46.048 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 -- start start 2024-09-13T16:36:46.050 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c480adf40 0x7f5c4815d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c4815a620 0x7f5c481568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c481612f0 0x7f5c48154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480cc6a0 con 0x7f5c480adf40 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480cb6a0 con 0x7f5c481612f0 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4815c710 con 0x7f5c4815a620 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c480adf40 0x7f5c4815d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c480adf40 0x7f5c4815d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55424/0 (socket says 172.21.15.119:55424) 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 -- 172.21.15.119:0/785243800 learned_addr learned my addr 172.21.15.119:0/785243800 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c477fe700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c4815a620 0x7f5c481568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4c916700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c481612f0 0x7f5c48154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 -- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c4815a620 msgr2=0x7f5c481568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:46.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c4815a620 0x7f5c481568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 -- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c481612f0 msgr2=0x7f5c48154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c481612f0 0x7f5c48154460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 -- 172.21.15.119:0/785243800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c40009b30 con 0x7f5c480adf40 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c47fff700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c480adf40 0x7f5c4815d1f0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f5c480721f0 tx=0x7f5c40004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c44ff9700 1 -- 172.21.15.119:0/785243800 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c400099b0 con 0x7f5c480adf40 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c44ff9700 1 -- 172.21.15.119:0/785243800 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c400092e0 con 0x7f5c480adf40 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.045+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4815c9f0 con 0x7f5c480adf40 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.049+0000 7f5c44ff9700 1 -- 172.21.15.119:0/785243800 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4001c660 con 0x7f5c480adf40 2024-09-13T16:36:46.052 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.049+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c480cb980 con 0x7f5c480adf40 2024-09-13T16:36:46.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.049+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c48067760 con 0x7f5c480adf40 2024-09-13T16:36:46.053 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.049+0000 7f5c44ff9700 1 -- 172.21.15.119:0/785243800 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5c4001c7c0 con 0x7f5c480adf40 2024-09-13T16:36:46.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.049+0000 7f5c44ff9700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5c28066760 0x7f5c28068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:46.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.049+0000 7f5c44ff9700 1 -- 172.21.15.119:0/785243800 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f5c40085fd0 con 0x7f5c480adf40 2024-09-13T16:36:46.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.049+0000 7f5c477fe700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5c28066760 0x7f5c28068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:46.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.053+0000 7f5c477fe700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5c28066760 0x7f5c28068c20 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5c48156670 tx=0x7f5c30008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:46.062 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.057+0000 7f5c44ff9700 1 -- 172.21.15.119:0/785243800 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5c400533e0 con 0x7f5c480adf40 2024-09-13T16:36:46.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.273+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c4815ceb0 con 0x7f5c480adf40 2024-09-13T16:36:46.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.273+0000 7f5c44ff9700 1 -- 172.21.15.119:0/785243800 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f5c40056a00 con 0x7f5c480adf40 2024-09-13T16:36:46.276 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:46.276 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":104,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:46.137436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9646":{"gid":9646,"name":"b","rank":0,"incarnation":102,"state":"up:rejoin","state_seq":3,"addr":"172.21.15.145:6835/2905269684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2905269684},{"type":"v1","addr":"172.21.15.145:6835","nonce":2905269684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5c28066760 msgr2=0x7f5c28068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5c28066760 0x7f5c28068c20 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f5c48156670 tx=0x7f5c30008040 comp rx=0 tx=0).stop 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c480adf40 msgr2=0x7f5c4815d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c480adf40 0x7f5c4815d1f0 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f5c480721f0 tx=0x7f5c40004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 shutdown_connections 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5c28066760 0x7f5c28068c20 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5c481612f0 0x7f5c48154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5c4815a620 0x7f5c481568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.280 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 --2- 172.21.15.119:0/785243800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5c480adf40 0x7f5c4815d1f0 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:46.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 >> 172.21.15.119:0/785243800 conn(0x7f5c4806a350 msgr2=0x7f5c48070ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:46.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 shutdown_connections 2024-09-13T16:36:46.281 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:46.277+0000 7f5c4e379700 1 -- 172.21.15.119:0/785243800 wait complete. 2024-09-13T16:36:46.281 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 104 2024-09-13T16:36:46.699 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 104, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:35:55.244835+0000', 'modified': '2024-09-13T16:36:46.137436+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 128, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9646}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9646': {'gid': 9646, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:rejoin', 'state_seq': 3, 'addr': '172.21.15.145:6835/2905269684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2905269684}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 2905269684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:36:46.700 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_9646:up:rejoin 2024-09-13T16:36:47.701 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:47.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8b2d0700 1 Processor -- start 2024-09-13T16:36:47.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8b2d0700 1 -- start start 2024-09-13T16:36:47.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8b2d0700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fce8406d3e0 0x7fce8406ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:47.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8b2d0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce840cfc20 0x7fce8406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:47.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8b2d0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8415aa80 con 0x7fce841576c0 2024-09-13T16:36:47.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8b2d0700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce840d0020 con 0x7fce8406d3e0 2024-09-13T16:36:47.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8b2d0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce84157aa0 con 0x7fce840cfc20 2024-09-13T16:36:47.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8906c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fce841576c0 0x7fce8406ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:52378/0 (socket says 172.21.15.119:52378) 2024-09-13T16:36:47.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8906c700 1 -- 172.21.15.119:0/591850408 learned_addr learned my addr 172.21.15.119:0/591850408 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:47.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8986d700 1 --2- 172.21.15.119:0/591850408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce840cfc20 0x7fce8406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:47.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce8886b700 1 --2- 172.21.15.119:0/591850408 >> v2:172.21.15.145:3300/0 conn(0x7fce8406d3e0 0x7fce8406ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:47.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce7bfff700 1 -- 172.21.15.119:0/591850408 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789096880 0 0) 0x7fce8415aa80 con 0x7fce841576c0 2024-09-13T16:36:47.827 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.821+0000 7fce7bfff700 1 -- 172.21.15.119:0/591850408 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce700035d0 con 0x7fce841576c0 2024-09-13T16:36:47.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce7bfff700 1 -- 172.21.15.119:0/591850408 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2986724318 0 0) 0x7fce700035d0 con 0x7fce841576c0 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8986d700 1 -- 172.21.15.119:0/591850408 >> v2:172.21.15.145:3300/0 conn(0x7fce8406d3e0 msgr2=0x7fce8406ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8986d700 1 --2- 172.21.15.119:0/591850408 >> v2:172.21.15.145:3300/0 conn(0x7fce8406d3e0 0x7fce8406ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8986d700 1 -- 172.21.15.119:0/591850408 >> v1:172.21.15.119:6789/0 conn(0x7fce841576c0 legacy=0x7fce8406ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8986d700 1 -- 172.21.15.119:0/591850408 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840cf790 con 0x7fce840cfc20 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8886b700 1 --2- 172.21.15.119:0/591850408 >> v2:172.21.15.145:3300/0 conn(0x7fce8406d3e0 0x7fce8406ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8986d700 1 --2- 172.21.15.119:0/591850408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce840cfc20 0x7fce8406c6f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00baf0 tx=0x7fce7c00be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa2a494cd3bdcfc server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce7bfff700 1 -- 172.21.15.119:0/591850408 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce800019f0 con 0x7fce840cfc20 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce7bfff700 1 -- 172.21.15.119:0/591850408 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c00d970 con 0x7fce840cfc20 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce7bfff700 1 -- 172.21.15.119:0/591850408 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7c00dce0 con 0x7fce840cfc20 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 -- 172.21.15.119:0/591850408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce840cfc20 msgr2=0x7fce8406c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/591850408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce840cfc20 0x7fce8406c6f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fce7c00baf0 tx=0x7fce7c00be00 comp rx=0 tx=0).stop 2024-09-13T16:36:47.829 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 -- 172.21.15.119:0/591850408 shutdown_connections 2024-09-13T16:36:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/591850408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce840cfc20 0x7fce8406c6f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/591850408 >> v2:172.21.15.145:3300/0 conn(0x7fce8406d3e0 0x7fce8406ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 -- 172.21.15.119:0/591850408 >> 172.21.15.119:0/591850408 conn(0x7fce840b3fd0 msgr2=0x7fce840b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 -- 172.21.15.119:0/591850408 shutdown_connections 2024-09-13T16:36:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 -- 172.21.15.119:0/591850408 wait complete. 2024-09-13T16:36:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 Processor -- start 2024-09-13T16:36:47.830 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.825+0000 7fce8b2d0700 1 -- start start 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce8406d3e0 0x7fce841f2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce840cfc20 0x7fce8414d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce841576c0 0x7fce84154cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce8414d910 con 0x7fce840cfc20 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce841f2c30 con 0x7fce8406d3e0 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce841f2f10 con 0x7fce841576c0 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce840cfc20 0x7fce8414d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce840cfc20 0x7fce8414d3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55440/0 (socket says 172.21.15.119:55440) 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 -- 172.21.15.119:0/3162162274 learned_addr learned my addr 172.21.15.119:0/3162162274 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:47.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8986d700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce841576c0 0x7fce84154cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:47.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8906c700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce8406d3e0 0x7fce841f2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:47.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 -- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce841576c0 msgr2=0x7fce84154cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:47.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce841576c0 0x7fce84154cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:47.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 -- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce8406d3e0 msgr2=0x7fce841f2640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:47.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce8406d3e0 0x7fce841f2640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:47.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 -- 172.21.15.119:0/3162162274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce740094b0 con 0x7fce840cfc20 2024-09-13T16:36:47.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8886b700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce840cfc20 0x7fce8414d3d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fce7400ebe0 tx=0x7fce7400eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce79ffb700 1 -- 172.21.15.119:0/3162162274 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7400c850 con 0x7fce840cfc20 2024-09-13T16:36:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce79ffb700 1 -- 172.21.15.119:0/3162162274 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7400c9b0 con 0x7fce840cfc20 2024-09-13T16:36:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce7c00b750 con 0x7fce840cfc20 2024-09-13T16:36:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce841f32a0 con 0x7fce840cfc20 2024-09-13T16:36:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce79ffb700 1 -- 172.21.15.119:0/3162162274 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7400cd40 con 0x7fce840cfc20 2024-09-13T16:36:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce64005310 con 0x7fce840cfc20 2024-09-13T16:36:47.834 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.829+0000 7fce79ffb700 1 -- 172.21.15.119:0/3162162274 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fce74013470 con 0x7fce840cfc20 2024-09-13T16:36:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.833+0000 7fce79ffb700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce68066760 0x7fce68068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.833+0000 7fce79ffb700 1 -- 172.21.15.119:0/3162162274 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7fce74084310 con 0x7fce840cfc20 2024-09-13T16:36:47.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.833+0000 7fce8906c700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce68066760 0x7fce68068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:47.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.833+0000 7fce8906c700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce68066760 0x7fce68068c20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fce84154920 tx=0x7fce80002120 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:47.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:47.837+0000 7fce79ffb700 1 -- 172.21.15.119:0/3162162274 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fce74051490 con 0x7fce840cfc20 2024-09-13T16:36:48.043 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce640055c0 con 0x7fce840cfc20 2024-09-13T16:36:48.044 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce79ffb700 1 -- 172.21.15.119:0/3162162274 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fce7400cea0 con 0x7fce840cfc20 2024-09-13T16:36:48.044 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:48.044 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:47.145416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9646":{"gid":9646,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6835/2905269684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2905269684},{"type":"v1","addr":"172.21.15.145:6835","nonce":2905269684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce68066760 msgr2=0x7fce68068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce68066760 0x7fce68068c20 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fce84154920 tx=0x7fce80002120 comp rx=0 tx=0).stop 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce840cfc20 msgr2=0x7fce8414d3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce840cfc20 0x7fce8414d3d0 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7fce7400ebe0 tx=0x7fce7400eef0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 shutdown_connections 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce68066760 0x7fce68068c20 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce841576c0 0x7fce84154cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce840cfc20 0x7fce8414d3d0 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 --2- 172.21.15.119:0/3162162274 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce8406d3e0 0x7fce841f2640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.041+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 >> 172.21.15.119:0/3162162274 conn(0x7fce840b3fd0 msgr2=0x7fce84099360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.045+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 shutdown_connections 2024-09-13T16:36:48.047 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.045+0000 7fce8b2d0700 1 -- 172.21.15.119:0/3162162274 wait complete. 2024-09-13T16:36:48.048 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 105 2024-09-13T16:36:48.130 INFO:tasks.ceph.mds.b.smithi145.stderr:2024-09-13T16:36:48.125+0000 7fa615736700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:36:48.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 105, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-09-13T16:35:55.244835+0000', 'modified': '2024-09-13T16:36:47.145416+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 128, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9646}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9646': {'gid': 9646, 'name': 'b', 'rank': 0, 'incarnation': 102, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.145:6835/2905269684', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.145:6834', 'nonce': 2905269684}, {'type': 'v1', 'addr': '172.21.15.145:6835', 'nonce': 2905269684}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-09-13T16:36:48.461 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-09-13T16:36:48.462 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-09-13T16:36:48.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.585+0000 7f0819b68700 1 Processor -- start 2024-09-13T16:36:48.590 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.585+0000 7f0819b68700 1 -- start start 2024-09-13T16:36:48.591 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f080c033ac0 0x7f080c009ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f080c0eb420 0x7f080c04ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f080c01cfa0 con 0x7f080c007260 2024-09-13T16:36:48.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c01c670 con 0x7f080c0eb420 2024-09-13T16:36:48.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f080c01c950 con 0x7f080c033ac0 2024-09-13T16:36:48.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0813fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f080c0eb420 0x7f080c04ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819367700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f080c007260 0x7f080c01d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:52380/0 (socket says 172.21.15.119:52380) 2024-09-13T16:36:48.592 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819367700 1 -- 172.21.15.119:0/3421924690 learned_addr learned my addr 172.21.15.119:0/3421924690 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:48.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0818b66700 1 --2- 172.21.15.119:0/3421924690 >> v2:172.21.15.145:3300/0 conn(0x7f080c033ac0 0x7f080c009ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0813fff700 1 -- 172.21.15.119:0/3421924690 >> v2:172.21.15.145:3300/0 conn(0x7f080c033ac0 msgr2=0x7f080c009ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.593 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0813fff700 1 --2- 172.21.15.119:0/3421924690 >> v2:172.21.15.145:3300/0 conn(0x7f080c033ac0 0x7f080c009ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0813fff700 1 -- 172.21.15.119:0/3421924690 >> v1:172.21.15.119:6789/0 conn(0x7f080c007260 legacy=0x7f080c01d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0813fff700 1 -- 172.21.15.119:0/3421924690 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f080c01cc30 con 0x7f080c0eb420 2024-09-13T16:36:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0818b66700 1 --2- 172.21.15.119:0/3421924690 >> v2:172.21.15.145:3300/0 conn(0x7f080c033ac0 0x7f080c009ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0813fff700 1 --2- 172.21.15.119:0/3421924690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0eb420 0x7f080c04ec10 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f080c01e110 tx=0x7f0808009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2129f07e0c688b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f08137fe700 1 -- 172.21.15.119:0/3421924690 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808011830 con 0x7f080c0eb420 2024-09-13T16:36:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f08137fe700 1 -- 172.21.15.119:0/3421924690 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808011990 con 0x7f080c0eb420 2024-09-13T16:36:48.594 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f08137fe700 1 -- 172.21.15.119:0/3421924690 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808011d40 con 0x7f080c0eb420 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- 172.21.15.119:0/3421924690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0eb420 msgr2=0x7f080c04ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 --2- 172.21.15.119:0/3421924690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0eb420 0x7f080c04ec10 secure :-1 s=READY pgs=1967 cs=0 l=1 rev1=1 crypto rx=0x7f080c01e110 tx=0x7f0808009b30 comp rx=0 tx=0).stop 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- 172.21.15.119:0/3421924690 shutdown_connections 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 --2- 172.21.15.119:0/3421924690 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0eb420 0x7f080c04ec10 unknown :-1 s=CLOSED pgs=1967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 --2- 172.21.15.119:0/3421924690 >> v2:172.21.15.145:3300/0 conn(0x7f080c033ac0 0x7f080c009ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- 172.21.15.119:0/3421924690 >> 172.21.15.119:0/3421924690 conn(0x7f080c036140 msgr2=0x7f080c036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- 172.21.15.119:0/3421924690 shutdown_connections 2024-09-13T16:36:48.595 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.589+0000 7f0819b68700 1 -- 172.21.15.119:0/3421924690 wait complete. 2024-09-13T16:36:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.593+0000 7f0819b68700 1 Processor -- start 2024-09-13T16:36:48.597 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.593+0000 7f0819b68700 1 -- start start 2024-09-13T16:36:48.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080c007260 0x7f080c039890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f080c033ac0 0x7f080c039dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.599 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0374a0 0x7f080c037b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c049820 con 0x7f080c0374a0 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080c049b00 con 0x7f080c033ac0 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f080c049de0 con 0x7f080c007260 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0374a0 0x7f080c037b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0374a0 0x7f080c037b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55458/0 (socket says 172.21.15.119:55458) 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 -- 172.21.15.119:0/850663776 learned_addr learned my addr 172.21.15.119:0/850663776 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0818b66700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080c007260 0x7f080c039890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.600 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0813fff700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f080c033ac0 0x7f080c039dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 -- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080c007260 msgr2=0x7f080c039890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080c007260 0x7f080c039890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 -- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f080c033ac0 msgr2=0x7f080c039dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.601 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f080c033ac0 0x7f080c039dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 -- 172.21.15.119:0/850663776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08080094b0 con 0x7f080c0374a0 2024-09-13T16:36:48.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819367700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0374a0 0x7f080c037b00 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f081404f480 tx=0x7f0814067560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f08117fa700 1 -- 172.21.15.119:0/850663776 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0814073060 con 0x7f080c0374a0 2024-09-13T16:36:48.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f08117fa700 1 -- 172.21.15.119:0/850663776 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08140731c0 con 0x7f080c0374a0 2024-09-13T16:36:48.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080c04a0c0 con 0x7f080c0374a0 2024-09-13T16:36:48.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f08117fa700 1 -- 172.21.15.119:0/850663776 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0814072390 con 0x7f080c0374a0 2024-09-13T16:36:48.602 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.597+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c0494f0 con 0x7f080c0374a0 2024-09-13T16:36:48.603 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f08117fa700 1 -- 172.21.15.119:0/850663776 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f08140724f0 con 0x7f080c0374a0 2024-09-13T16:36:48.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f08117fa700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0800066730 0x7f0800068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f08117fa700 1 -- 172.21.15.119:0/850663776 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f08140fdb90 con 0x7f080c0374a0 2024-09-13T16:36:48.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0818b66700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0800066730 0x7f0800068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.604 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 Processor -- start 2024-09-13T16:36:48.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 -- start start 2024-09-13T16:36:48.605 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0818b66700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0800066730 0x7f0800068bf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f080400ac00 tx=0x7f0804008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.606 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1acf610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1a97890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1acfb50 0x1a97dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x19f6d80 con 0x1aade90 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19f4510 con 0x1acfb50 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x19efe00 con 0x1acf230 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1a97890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1a97890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55460/0 (socket says 172.21.15.119:55460) 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 -- 172.21.15.119:0/1706369495 learned_addr learned my addr 172.21.15.119:0/1706369495 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0818b66700 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1acf610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0819367700 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1acfb50 0x1a97dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 -- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 msgr2=0x1acf610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.607 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1acf610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 -- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1acfb50 msgr2=0x1a97dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1acfb50 0x1a97dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.601+0000 7f0813fff700 1 -- 172.21.15.119:0/1706369495 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x1862960 con 0x1aade90 2024-09-13T16:36:48.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0813fff700 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1a97890 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f0808013000 tx=0x7f0808010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84f3150ed20fe97 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f07fa7fc700 1 -- 172.21.15.119:0/1706369495 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808020070 con 0x1aade90 2024-09-13T16:36:48.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f07fa7fc700 1 -- 172.21.15.119:0/1706369495 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808011b00 con 0x1aade90 2024-09-13T16:36:48.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f07fa7fc700 1 -- 172.21.15.119:0/1706369495 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808004040 con 0x1aade90 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- 172.21.15.119:0/1706369495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 msgr2=0x1a97890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1a97890 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f0808013000 tx=0x7f0808010000 comp rx=0 tx=0).stop 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- 172.21.15.119:0/1706369495 shutdown_connections 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1acfb50 0x1a97dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1a97890 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 --2- 172.21.15.119:0/1706369495 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1acf610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- 172.21.15.119:0/1706369495 >> 172.21.15.119:0/1706369495 conn(0x18174d0 msgr2=0x1ab0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- 172.21.15.119:0/1706369495 shutdown_connections 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- 172.21.15.119:0/1706369495 wait complete. 2024-09-13T16:36:48.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- start start 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 Processor -- start 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ac78e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1a87a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1acfb50 0x1a89c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x19ad730 con 0x1acfb50 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19b35b0 con 0x1aade90 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x189baa0 con 0x1acf230 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f081e07b740 10 client.0 _ll_register_callbacks cb 0x19e9610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1acfb50 0x1a89c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1acfb50 0x1a89c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55468/0 (socket says 172.21.15.119:55468) 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 -- 172.21.15.119:0/1164023105 learned_addr learned my addr 172.21.15.119:0/1164023105 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:48.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0813fff700 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1a87a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0818b66700 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ac78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 -- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 msgr2=0x1a87a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1a87a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 -- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 msgr2=0x1ac78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ac78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 -- 172.21.15.119:0/1164023105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f080c01cc30 con 0x1acfb50 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f0819367700 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1acfb50 0x1a89c10 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f08140ce760 tx=0x7f08140ce490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f41eaf0e4476d23c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.605+0000 7f07f37fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 1 -- 172.21.15.119:0/1164023105 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f08140d6ca0 con 0x1acfb50 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 1 -- 172.21.15.119:0/1164023105 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f08140e7d60 con 0x1acfb50 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x18241e0 con 0x1acfb50 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 1 -- 172.21.15.119:0/1164023105 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f08140eb230 con 0x1acfb50 2024-09-13T16:36:48.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 10 client.10176 fetch_fsmap learned FSMap version 105 2024-09-13T16:36:48.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) v3 -- 0x19b6220 con 0x1acfb50 2024-09-13T16:36:48.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 1 -- 172.21.15.119:0/1164023105 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x18241e0 con 0x1acfb50 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 1 -- 172.21.15.119:0/1164023105 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 105) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f08140e6100 con 0x1acfb50 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 10 client.10176 fetch_fsmap finished waiting for FSMap version 105 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 10 client.10176 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 20 client.10176 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1a87fc0 0x1a84f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 4 client.10176 mds_command: new command op to 9646 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 --> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x186e460 con 0x1a87fc0 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f0818b66700 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1a87fc0 0x1a84f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 10 client.10176 ms_handle_connect on v2:172.21.15.145:6834/2905269684 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f0818b66700 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1a87fc0 0x1a84f10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9930ae9ab16e327c server_cookie=b888724c29f0fa17 in_seq=0 out_seq=0 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 1 -- 172.21.15.119:0/1164023105 <== mds.0 v2:172.21.15.145:6834/2905269684 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x186e460 con 0x1a87fc0 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f07f37fe700 10 client.10176 handle_command_reply: tid=0 2024-09-13T16:36:48.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 1 client.10176 shutdown 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1acfb50 msgr2=0x1a89c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.609+0000 7f081e07b740 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1acfb50 0x1a89c10 secure :-1 s=READY pgs=1970 cs=0 l=1 rev1=1 crypto rx=0x7f08140ce760 tx=0x7f08140ce490 comp rx=0 tx=0).stop 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 shutdown_connections 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1a87fc0 0x1a84f10 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1acfb50 0x1a89c10 unknown :-1 s=CLOSED pgs=1970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f07f37fe700 0 client.10176 ms_handle_reset on v2:172.21.15.145:6834/2905269684 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1a87a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 --2- 172.21.15.119:0/1164023105 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ac78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 >> 172.21.15.119:0/1164023105 conn(0x18174d0 msgr2=0x1ac8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 shutdown_connections 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 1 -- 172.21.15.119:0/1164023105 wait complete. 2024-09-13T16:36:48.617 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.613+0000 7f081e07b740 20 client.10176 trim_cache size 0 max 16384 2024-09-13T16:36:48.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 Processor -- start 2024-09-13T16:36:48.636 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 -- start start 2024-09-13T16:36:48.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a87fc0 0x1ae70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1adefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1adf4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x19e7b70 con 0x1aade90 2024-09-13T16:36:48.637 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x19f06c0 con 0x1a87fc0 2024-09-13T16:36:48.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a1eec0 con 0x1acf230 2024-09-13T16:36:48.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1adefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1adefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55478/0 (socket says 172.21.15.119:55478) 2024-09-13T16:36:48.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 -- 172.21.15.119:0/1637147182 learned_addr learned my addr 172.21.15.119:0/1637147182 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:48.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0819367700 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1adf4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0818b66700 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a87fc0 0x1ae70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 -- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 msgr2=0x1adf4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1adf4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 -- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a87fc0 msgr2=0x1ae70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a87fc0 0x1ae70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 -- 172.21.15.119:0/1637147182 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x199c450 con 0x1aade90 2024-09-13T16:36:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.633+0000 7f0813fff700 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1adefb0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f0808029040 tx=0x7f0808011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dc95d5f46fd4cb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.639 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f07faffd700 1 -- 172.21.15.119:0/1637147182 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808020070 con 0x1aade90 2024-09-13T16:36:48.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f07faffd700 1 -- 172.21.15.119:0/1637147182 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808006b10 con 0x1aade90 2024-09-13T16:36:48.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f07faffd700 1 -- 172.21.15.119:0/1637147182 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0808004040 con 0x1aade90 2024-09-13T16:36:48.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 -- 172.21.15.119:0/1637147182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 msgr2=0x1adefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.640 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1adefb0 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7f0808029040 tx=0x7f0808011000 comp rx=0 tx=0).stop 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 -- 172.21.15.119:0/1637147182 shutdown_connections 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1adf4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1aade90 0x1adefb0 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 --2- 172.21.15.119:0/1637147182 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1a87fc0 0x1ae70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 -- 172.21.15.119:0/1637147182 >> 172.21.15.119:0/1637147182 conn(0x18174d0 msgr2=0x1aac900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 -- 172.21.15.119:0/1637147182 shutdown_connections 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 -- 172.21.15.119:0/1637147182 wait complete. 2024-09-13T16:36:48.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 -- start start 2024-09-13T16:36:48.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.637+0000 7f081e07b740 1 Processor -- start 2024-09-13T16:36:48.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a87fc0 0x1ae4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ae2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1b2f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x19a9c30 con 0x1a87fc0 2024-09-13T16:36:48.644 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x1ac7990 con 0x1aade90 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x1ae6d30 con 0x1acf230 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 10 client.0 _ll_register_callbacks cb 0x19a1ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a87fc0 0x1ae4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a87fc0 0x1ae4ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55492/0 (socket says 172.21.15.119:55492) 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 -- 172.21.15.119:0/1340335079 learned_addr learned my addr 172.21.15.119:0/1340335079 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0819367700 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1b2f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0813fff700 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ae2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 -- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 msgr2=0x1b2f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1b2f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 -- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 msgr2=0x1ae2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ae2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 -- 172.21.15.119:0/1340335079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x1862960 con 0x1a87fc0 2024-09-13T16:36:48.645 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0818b66700 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a87fc0 0x1ae4ba0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f08040284a0 tx=0x7f08040047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f500d4723f0e5f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f07f37fe700 10 client.0 ms_handle_connect on v2:172.21.15.119:3300/0 2024-09-13T16:36:48.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f07f37fe700 1 -- 172.21.15.119:0/1340335079 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0804026040 con 0x1a87fc0 2024-09-13T16:36:48.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0808011b00 con 0x1a87fc0 2024-09-13T16:36:48.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f07f37fe700 1 -- 172.21.15.119:0/1340335079 <== mon.0 v2:172.21.15.119:3300/0 2 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f080401a040 con 0x1a87fc0 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f07f37fe700 1 -- 172.21.15.119:0/1340335079 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_get_version_reply(handle=1 version=105) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0804028d70 con 0x1a87fc0 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 10 client.10182 fetch_fsmap learned FSMap version 105 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,fsmap=105,monmap=2+}) v3 -- 0x1990720 con 0x1a87fc0 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f07f37fe700 1 -- 172.21.15.119:0/1340335079 <== mon.0 v2:172.21.15.119:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0808011b00 con 0x1a87fc0 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f07f37fe700 1 -- 172.21.15.119:0/1340335079 <== mon.0 v2:172.21.15.119:3300/0 5 ==== fsmap(e 105) v1 ==== 3114+0+0 (secure 0 0 0) 0x7f0804005a40 con 0x1a87fc0 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 10 client.10182 fetch_fsmap finished waiting for FSMap version 105 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 10 client.10182 resolve_mds: resolved name 'b' to daemon mds.b 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 20 client.10182 populate_metadata read hostname 'smithi119' 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1acfb50 0x1ae54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 4 client.10182 mds_command: new command op to 9646 tid=0 multi_id=0 [{"prefix": "status"}] 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 --> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] -- command(tid 0: {"prefix": "status"}) v1 -- 0x18931e0 con 0x1acfb50 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.641+0000 7f0813fff700 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1acfb50 0x1ae54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f07f37fe700 10 client.10182 ms_handle_connect on v2:172.21.15.145:6834/2905269684 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0813fff700 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1acfb50 0x1ae54e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f91b33186d06203d server_cookie=4b719e3af2d1420a in_seq=0 out_seq=0 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f07f37fe700 1 -- 172.21.15.119:0/1340335079 <== mds.0 v2:172.21.15.145:6834/2905269684 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+334 (crc 0 0 0) 0x18931e0 con 0x1acfb50 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f07f37fe700 10 client.10182 handle_command_reply: tid=0 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 client.10182 shutdown 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a87fc0 msgr2=0x1ae4ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a87fc0 0x1ae4ba0 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7f08040284a0 tx=0x7f08040047e0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 shutdown_connections 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:6834/2905269684,v1:172.21.15.145:6835/2905269684] conn(0x1acfb50 0x1ae54e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x1acf230 0x1b2f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x1aade90 0x1ae2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 --2- 172.21.15.119:0/1340335079 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x1a87fc0 0x1ae4ba0 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 >> 172.21.15.119:0/1340335079 conn(0x18174d0 msgr2=0x1aca240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f07f37fe700 0 client.10182 ms_handle_reset on v2:172.21.15.145:6834/2905269684 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 shutdown_connections 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 1 -- 172.21.15.119:0/1340335079 wait complete. 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f081e07b740 20 client.10182 trim_cache size 0 max 16384 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: "cluster_fsid": "01de39e4-2965-4196-9df7-d7f8aeef36df", 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: "whoami": 0, 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9646, 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: "want_state": "up:active", 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: "state": "up:active", 2024-09-13T16:36:48.651 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs", 2024-09-13T16:36:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: "rank_uptime": 4.5126189380000001, 2024-09-13T16:36:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: "mdsmap_epoch": 105, 2024-09-13T16:36:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch": 128, 2024-09-13T16:36:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: "osdmap_epoch_barrier": 128, 2024-09-13T16:36:48.652 INFO:teuthology.orchestra.run.smithi119.stdout: "uptime": 6.4877058659999998 2024-09-13T16:36:48.652 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0800066730 msgr2=0x7f0800068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0800066730 0x7f0800068bf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f080400ac00 tx=0x7f0804008040 comp rx=0 tx=0).stop 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0374a0 msgr2=0x7f080c037b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0374a0 0x7f080c037b00 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f081404f480 tx=0x7f0814067560 comp rx=0 tx=0).stop 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 shutdown_connections 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0800066730 0x7f0800068bf0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f080c0374a0 0x7f080c037b00 unknown :-1 s=CLOSED pgs=1968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f080c033ac0 0x7f080c039dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 --2- 172.21.15.119:0/850663776 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f080c007260 0x7f080c039890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.653 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 >> 172.21.15.119:0/850663776 conn(0x7f080c036140 msgr2=0x7f080c035930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 shutdown_connections 2024-09-13T16:36:48.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.645+0000 7f0819b68700 1 -- 172.21.15.119:0/850663776 wait complete. 2024-09-13T16:36:48.666 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-09-13T16:36:48.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 Processor -- start 2024-09-13T16:36:48.787 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- start start 2024-09-13T16:36:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f5f0815e840 0x7f5f08070b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f0815ec20 0x7f5f08161f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.788 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f08071210 con 0x7f5f08164a70 2024-09-13T16:36:48.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f080714f0 con 0x7f5f0815e840 2024-09-13T16:36:48.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080717d0 con 0x7f5f0815ec20 2024-09-13T16:36:48.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f077fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5f08164a70 0x7f5f080ad500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:52386/0 (socket says 172.21.15.119:52386) 2024-09-13T16:36:48.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f077fe700 1 -- 172.21.15.119:0/3149179145 learned_addr learned my addr 172.21.15.119:0/3149179145 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:48.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f067fc700 1 --2- 172.21.15.119:0/3149179145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f0815ec20 0x7f5f08161f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.789 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f06ffd700 1 --2- 172.21.15.119:0/3149179145 >> v2:172.21.15.145:3300/0 conn(0x7f5f0815e840 0x7f5f08070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.790 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f067fc700 1 -- 172.21.15.119:0/3149179145 >> v2:172.21.15.145:3300/0 conn(0x7f5f0815e840 msgr2=0x7f5f08070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f05ffb700 1 -- 172.21.15.119:0/3149179145 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745146049 0 0) 0x7f5f08071210 con 0x7f5f08164a70 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f067fc700 1 --2- 172.21.15.119:0/3149179145 >> v2:172.21.15.145:3300/0 conn(0x7f5f0815e840 0x7f5f08070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f067fc700 1 -- 172.21.15.119:0/3149179145 >> v1:172.21.15.119:6789/0 conn(0x7f5f08164a70 legacy=0x7f5f080ad500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f067fc700 1 -- 172.21.15.119:0/3149179145 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f08071ab0 con 0x7f5f0815ec20 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f067fc700 1 --2- 172.21.15.119:0/3149179145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f0815ec20 0x7f5f08161f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009850 tx=0x7f5efc009b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f281126625bf9d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f05ffb700 1 -- 172.21.15.119:0/3149179145 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc011830 con 0x7f5f0815ec20 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f05ffb700 1 -- 172.21.15.119:0/3149179145 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc011990 con 0x7f5f0815ec20 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f05ffb700 1 -- 172.21.15.119:0/3149179145 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc011d40 con 0x7f5f0815ec20 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- 172.21.15.119:0/3149179145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f0815ec20 msgr2=0x7f5f08161f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/3149179145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f0815ec20 0x7f5f08161f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009850 tx=0x7f5efc009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:48.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- 172.21.15.119:0/3149179145 shutdown_connections 2024-09-13T16:36:48.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/3149179145 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f0815ec20 0x7f5f08161f00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/3149179145 >> v2:172.21.15.145:3300/0 conn(0x7f5f0815e840 0x7f5f08070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- 172.21.15.119:0/3149179145 >> 172.21.15.119:0/3149179145 conn(0x7f5f0806a350 msgr2=0x7f5f0806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:48.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- 172.21.15.119:0/3149179145 shutdown_connections 2024-09-13T16:36:48.792 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.785+0000 7f5f0d55b700 1 -- 172.21.15.119:0/3149179145 wait complete. 2024-09-13T16:36:48.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 Processor -- start 2024-09-13T16:36:48.793 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 -- start start 2024-09-13T16:36:48.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f0815e840 0x7f5f080a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f08164a70 0x7f5f08159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f081f7230 0x7f5f080ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0815a080 con 0x7f5f08164a70 2024-09-13T16:36:48.795 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080ae590 con 0x7f5f0815e840 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f0d55b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f080a8f60 con 0x7f5f081f7230 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f067fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f08164a70 0x7f5f08159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f06ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f0815e840 0x7f5f080a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f077fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f081f7230 0x7f5f080ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.789+0000 7f5f067fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f08164a70 0x7f5f08159b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:55494/0 (socket says 172.21.15.119:55494) 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f067fc700 1 -- 172.21.15.119:0/2520124530 learned_addr learned my addr 172.21.15.119:0/2520124530 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f067fc700 1 -- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f081f7230 msgr2=0x7f5f080ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f067fc700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f081f7230 0x7f5f080ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f067fc700 1 -- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f0815e840 msgr2=0x7f5f080a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f067fc700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f0815e840 0x7f5f080a84b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f067fc700 1 -- 172.21.15.119:0/2520124530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5efc0094b0 con 0x7f5f08164a70 2024-09-13T16:36:48.796 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f067fc700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f08164a70 0x7f5f08159b40 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f5efc000c00 tx=0x7f5efc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5eef7fe700 1 -- 172.21.15.119:0/2520124530 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc004550 con 0x7f5f08164a70 2024-09-13T16:36:48.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5eef7fe700 1 -- 172.21.15.119:0/2520124530 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5efc01f070 con 0x7f5f08164a70 2024-09-13T16:36:48.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f080a9240 con 0x7f5f08164a70 2024-09-13T16:36:48.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5eef7fe700 1 -- 172.21.15.119:0/2520124530 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc003900 con 0x7f5f08164a70 2024-09-13T16:36:48.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f080ae830 con 0x7f5f08164a70 2024-09-13T16:36:48.798 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.793+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f08067760 con 0x7f5f08164a70 2024-09-13T16:36:48.799 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.797+0000 7f5eef7fe700 1 -- 172.21.15.119:0/2520124530 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5efc003a60 con 0x7f5f08164a70 2024-09-13T16:36:48.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.797+0000 7f5eef7fe700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ef4066730 0x7f5ef4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:48.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.797+0000 7f5eef7fe700 1 -- 172.21.15.119:0/2520124530 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f5efc021070 con 0x7f5f08164a70 2024-09-13T16:36:48.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.797+0000 7f5f06ffd700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ef4066730 0x7f5ef4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:48.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.797+0000 7f5f06ffd700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ef4066730 0x7f5ef4068bf0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80113e0 tx=0x7f5ef800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:48.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:48.801+0000 7f5eef7fe700 1 -- 172.21.15.119:0/2520124530 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5efc052d50 con 0x7f5f08164a70 2024-09-13T16:36:49.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.009+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f5f080a9700 con 0x7f5f08164a70 2024-09-13T16:36:49.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5eef7fe700 1 -- 172.21.15.119:0/2520124530 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v105) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5efc015020 con 0x7f5f08164a70 2024-09-13T16:36:49.015 INFO:teuthology.orchestra.run.smithi119.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-09-13T16:36:49.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ef4066730 msgr2=0x7f5ef4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:49.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ef4066730 0x7f5ef4068bf0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80113e0 tx=0x7f5ef800b040 comp rx=0 tx=0).stop 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f08164a70 msgr2=0x7f5f08159b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f08164a70 0x7f5f08159b40 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f5efc000c00 tx=0x7f5efc004770 comp rx=0 tx=0).stop 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 shutdown_connections 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5ef4066730 0x7f5ef4068bf0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5f081f7230 0x7f5f080ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5f08164a70 0x7f5f08159b40 unknown :-1 s=CLOSED pgs=1974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 --2- 172.21.15.119:0/2520124530 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5f0815e840 0x7f5f080a84b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 >> 172.21.15.119:0/2520124530 conn(0x7f5f0806a350 msgr2=0x7f5f081607a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 shutdown_connections 2024-09-13T16:36:49.018 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:49.013+0000 7f5f0d55b700 1 -- 172.21.15.119:0/2520124530 wait complete. 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi119.front.sepia.ceph.com 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-09-13T16:36:49.030 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:49.030 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.030 DEBUG:teuthology.orchestra.run.smithi119:> ip addr 2024-09-13T16:36:49.038 INFO:teuthology.orchestra.run.smithi119.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-09-13T16:36:49.038 INFO:teuthology.orchestra.run.smithi119.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: inet 127.0.0.1/8 scope host lo 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 ::1/128 scope host 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout:2: eno1: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:62 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout:3: eno2: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:d8:da:63 brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout:4: enp3s0f0: mtu 1500 qdisc noop state DOWN group default qlen 1000 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ee brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 0c:c4:7a:8f:d5:ef brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.21.15.119/20 brd 172.21.15.255 scope global dynamic enp3s0f1 2024-09-13T16:36:49.039 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft 41932sec preferred_lft 41932sec 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::ec4:7aff:fe8f:d5ef/64 scope link 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout:6: docker0: mtu 1500 qdisc noqueue state DOWN group default 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether 02:42:cb:3a:84:bd brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout:9: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::5029:8fff:fed6:20f4/64 scope link 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.040 INFO:teuthology.orchestra.run.smithi119.stdout:10: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether fa:be:bd:fb:83:73 brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::f8be:bdff:fefb:8373/64 scope link 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run.smithi119.stdout:11: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run.smithi119.stdout: link/ether a2:d6:65:75:2e:ab brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run.smithi119.stdout: inet6 fe80::a0d6:65ff:fe75:2eab/64 scope link 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run.smithi119.stdout: valid_lft forever preferred_lft forever 2024-09-13T16:36:49.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.041 DEBUG:teuthology.orchestra.run.smithi119:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-09-13T16:36:49.101 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:36:49.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.103 DEBUG:teuthology.orchestra.run.smithi119:> route 2024-09-13T16:36:49.112 INFO:teuthology.orchestra.run.smithi119.stdout:Kernel IP routing table 2024-09-13T16:36:49.112 INFO:teuthology.orchestra.run.smithi119.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-09-13T16:36:49.113 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 0 0 0 enp3s0f1 2024-09-13T16:36:49.113 INFO:teuthology.orchestra.run.smithi119.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-09-13T16:36:49.113 INFO:teuthology.orchestra.run.smithi119.stdout:172.17.0.0 0.0.0.0 255.255.0.0 U 0 0 0 docker0 2024-09-13T16:36:49.113 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 0 0 0 enp3s0f1 2024-09-13T16:36:49.113 INFO:teuthology.orchestra.run.smithi119.stdout:_gateway 0.0.0.0 255.255.255.255 UH 100 0 0 enp3s0f1 2024-09-13T16:36:49.113 INFO:teuthology.orchestra.run.smithi119.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-09-13T16:36:49.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.113 DEBUG:teuthology.orchestra.run.smithi119:> (cd / && exec bash -c ' 2024-09-13T16:36:49.113 DEBUG:teuthology.orchestra.run.smithi119:> set -e 2024-09-13T16:36:49.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-09-13T16:36:49.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-09-13T16:36:49.114 DEBUG:teuthology.orchestra.run.smithi119:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-09-13T16:36:49.114 DEBUG:teuthology.orchestra.run.smithi119:> ') 2024-09-13T16:36:49.153 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.153 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:36:49.160 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:36:49.160 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:36:49.160 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:49.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.160 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link set brx.0 up 2024-09-13T16:36:49.176 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-09-13T16:36:49.176 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:49.176 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:49.182 INFO:teuthology.orchestra.run.smithi119.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-09-13T16:36:49.183 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-09-13T16:36:49.183 DEBUG:teuthology.orchestra.run.smithi119:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:49.233 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:49.248 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:49.249 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:49.259 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:49.260 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:49.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.261 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:49.268 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-09-13T16:36:49.268 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-09-13T16:36:49.312 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:49.329 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:49.329 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:49.340 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:49.341 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:49.342 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:49.342 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:49.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:2024-09-13T16:36:49.397+0000 7fc76393f640 -1 init, newargv = 0x556272580390 newargc=15 2024-09-13T16:36:49.401 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[55653]: starting ceph client 2024-09-13T16:36:49.411 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[55653]: starting fuse 2024-09-13T16:36:50.448 DEBUG:teuthology.orchestra.run.smithi119:> sudo modprobe fuse 2024-09-13T16:36:50.467 INFO:teuthology.orchestra.run:Running command with timeout 30 2024-09-13T16:36:50.467 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2024-09-13T16:36:50.483 INFO:teuthology.orchestra.run.smithi119.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2024-09-13T16:36:50.483 DEBUG:teuthology.orchestra.run:got remote process result: 32 2024-09-13T16:36:50.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:50.484 DEBUG:teuthology.orchestra.run.smithi119:> ls /sys/fs/fuse/connections 2024-09-13T16:36:50.490 INFO:teuthology.orchestra.run.smithi119.stdout:52 2024-09-13T16:36:50.491 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> import glob 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> import re 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> import subprocess 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> def _find_admin_socket(client_name): 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> files = glob.glob(asok_path) 2024-09-13T16:36:50.491 DEBUG:teuthology.orchestra.run.smithi119:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> # Given a non-glob path, it better be there 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> if "*" not in asok_path: 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> assert(len(files) == 1) 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> return files[0] 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> for f in files: 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> if os.path.exists("/proc/{0}".format(pid)): 2024-09-13T16:36:50.492 DEBUG:teuthology.orchestra.run.smithi119:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-09-13T16:36:50.493 DEBUG:teuthology.orchestra.run.smithi119:> contents = proc_f.read() 2024-09-13T16:36:50.493 DEBUG:teuthology.orchestra.run.smithi119:> if mountpoint in contents: 2024-09-13T16:36:50.493 DEBUG:teuthology.orchestra.run.smithi119:> return f 2024-09-13T16:36:50.493 DEBUG:teuthology.orchestra.run.smithi119:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-09-13T16:36:50.493 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:50.493 DEBUG:teuthology.orchestra.run.smithi119:> print(_find_admin_socket("client.0")) 2024-09-13T16:36:50.493 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:50.577 INFO:teuthology.orchestra.run.smithi119.stdout:/var/run/ceph/ceph-client.0.55653.asok 2024-09-13T16:36:50.761 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.55653.asok 2024-09-13T16:36:50.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:50.761 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.55653.asok status 2024-09-13T16:36:50.909 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "metadata": { 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_sha1": "50bbe7f395997dcd871813949c49addbbf630361", 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "ceph_version": "ceph version 18.2.4-850-g50bbe7f3 (50bbe7f395997dcd871813949c49addbbf630361) reef (stable)", 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "entity_id": "0", 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "hostname": "smithi119", 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "pid": "55653", 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "root": "/" 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_count": 0, 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "dentry_pinned_count": 0, 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "id": 9760, 2024-09-13T16:36:50.910 INFO:teuthology.orchestra.run.smithi119.stdout: "inst": { 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "name": { 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "client", 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "num": 9760 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2513316827 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: } 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": { 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "type": "v1", 2024-09-13T16:36:50.911 INFO:teuthology.orchestra.run.smithi119.stdout: "addr": "192.168.0.1:0", 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "nonce": 2513316827 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: }, 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "inst_str": "client.9760 192.168.0.1:0/2513316827", 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "addr_str": "192.168.0.1:0/2513316827", 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "inode_count": 1, 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "mds_epoch": 105, 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch": 128, 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "osd_epoch_barrier": 0, 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "blocklisted": false, 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout: "fs_name": "cephfs" 2024-09-13T16:36:50.912 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:36:50.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:50.920 DEBUG:teuthology.orchestra.run.smithi119:> stat --file-system '--printf=%T 2024-09-13T16:36:50.920 DEBUG:teuthology.orchestra.run.smithi119:> ' -- /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:50.926 INFO:teuthology.orchestra.run.smithi119.stdout:fuseblk 2024-09-13T16:36:50.926 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:50.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:50.926 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:50.936 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:50.937 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest/mnt.0 && exec find ./) 2024-09-13T16:36:51.013 INFO:teuthology.orchestra.run.smithi119.stdout:./ 2024-09-13T16:36:51.013 INFO:teuthology.orchestra.run.smithi119.stdout:./parent 2024-09-13T16:36:51.013 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed 2024-09-13T16:36:51.013 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/charlie 2024-09-13T16:36:51.014 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/alpha 2024-09-13T16:36:51.014 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/flushed/bravo 2024-09-13T16:36:51.014 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/unflushed 2024-09-13T16:36:51.014 INFO:teuthology.orchestra.run.smithi119.stdout:./parent/unflushed/jfile 2024-09-13T16:36:51.015 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:51.015 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:51.015 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:51.015 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:51.015 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./").st_ino) 2024-09-13T16:36:51.015 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:51.085 INFO:teuthology.orchestra.run.smithi119.stdout:1 2024-09-13T16:36:51.276 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:51.276 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:51.276 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:51.276 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:51.276 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent").st_ino) 2024-09-13T16:36:51.277 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:51.338 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627776 2024-09-13T16:36:51.527 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:51.527 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:51.527 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:51.528 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:51.528 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed").st_ino) 2024-09-13T16:36:51.528 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:51.596 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627777 2024-09-13T16:36:51.787 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:51.788 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:51.788 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:51.788 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:51.788 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/charlie").st_ino) 2024-09-13T16:36:51.788 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:51.855 INFO:teuthology.orchestra.run.smithi119.stdout:1099511628280 2024-09-13T16:36:52.043 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:52.043 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:52.043 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:52.043 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:52.043 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/alpha").st_ino) 2024-09-13T16:36:52.043 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:52.113 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627778 2024-09-13T16:36:52.302 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:52.302 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:52.303 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:52.303 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:52.303 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/flushed/bravo").st_ino) 2024-09-13T16:36:52.303 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:52.411 INFO:teuthology.orchestra.run.smithi119.stdout:1099511628279 2024-09-13T16:36:52.600 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:52.600 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:52.600 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:52.600 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:52.600 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed").st_ino) 2024-09-13T16:36:52.600 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:52.663 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627779 2024-09-13T16:36:52.853 DEBUG:teuthology.orchestra.run.smithi119:> adjust-ulimits daemon-helper kill python3 -c ' 2024-09-13T16:36:52.853 DEBUG:teuthology.orchestra.run.smithi119:> import os 2024-09-13T16:36:52.853 DEBUG:teuthology.orchestra.run.smithi119:> import stat 2024-09-13T16:36:52.854 DEBUG:teuthology.orchestra.run.smithi119:> 2024-09-13T16:36:52.854 DEBUG:teuthology.orchestra.run.smithi119:> print(os.stat("/home/ubuntu/cephtest/mnt.0/./parent/unflushed/jfile").st_ino) 2024-09-13T16:36:52.854 DEBUG:teuthology.orchestra.run.smithi119:> ' 2024-09-13T16:36:52.921 INFO:teuthology.orchestra.run.smithi119.stdout:1099511627780 2024-09-13T16:36:53.110 INFO:teuthology.nuke:Clearing teuthology firewall rules... 2024-09-13T16:36:53.111 DEBUG:teuthology.orchestra.run.smithi119:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:36:53.132 DEBUG:teuthology.orchestra.run.smithi145:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-09-13T16:36:53.154 INFO:teuthology.nuke:Cleared teuthology firewall rules. 2024-09-13T16:36:53.154 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:53.154 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:53.182 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:53.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 Processor -- start 2024-09-13T16:36:53.299 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- start start 2024-09-13T16:36:53.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c15b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab3c1444b0 0x7fab3c06d080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab3c06bd60 con 0x7fab3c0cbb10 2024-09-13T16:36:53.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab3c15bf10 con 0x7fab3c1444b0 2024-09-13T16:36:53.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c06bfe0 con 0x7fab3c1440d0 2024-09-13T16:36:53.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3a7fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fab3c1444b0 0x7fab3c06d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3affd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c15b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3b7fe700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fab3c0cbb10 0x7fab3c0cbf30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40412/0 (socket says 172.21.15.119:40412) 2024-09-13T16:36:53.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3b7fe700 1 -- 172.21.15.119:0/1708424419 learned_addr learned my addr 172.21.15.119:0/1708424419 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3a7fc700 1 -- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 msgr2=0x7fab3c15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3a7fc700 1 --2- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c15b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3a7fc700 1 -- 172.21.15.119:0/1708424419 >> v1:172.21.15.145:6789/0 conn(0x7fab3c0cbb10 legacy=0x7fab3c0cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3a7fc700 1 -- 172.21.15.119:0/1708424419 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab3c06c2c0 con 0x7fab3c1444b0 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3affd700 1 --2- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c15b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab3a7fc700 1 --2- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c1444b0 0x7fab3c06d080 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009850 tx=0x7fab2c009b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b10e1e15e77cb72a server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab39ffb700 1 -- 172.21.15.119:0/1708424419 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c011830 con 0x7fab3c1444b0 2024-09-13T16:36:53.303 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab39ffb700 1 -- 172.21.15.119:0/1708424419 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab2c011990 con 0x7fab3c1444b0 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab39ffb700 1 -- 172.21.15.119:0/1708424419 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab2c011d40 con 0x7fab3c1444b0 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c1444b0 msgr2=0x7fab3c06d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 --2- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c1444b0 0x7fab3c06d080 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fab2c009850 tx=0x7fab2c009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- 172.21.15.119:0/1708424419 shutdown_connections 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 --2- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c1444b0 0x7fab3c06d080 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 --2- 172.21.15.119:0/1708424419 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c15b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- 172.21.15.119:0/1708424419 >> 172.21.15.119:0/1708424419 conn(0x7fab3c06a350 msgr2=0x7fab3c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- 172.21.15.119:0/1708424419 shutdown_connections 2024-09-13T16:36:53.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.297+0000 7fab41734700 1 -- 172.21.15.119:0/1708424419 wait complete. 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 Processor -- start 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 -- start start 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c0cbb10 0x7fab3c1f1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c0ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab3c1444b0 0x7fab3c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c14e690 con 0x7fab3c1444b0 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c13b300 con 0x7fab3c0cbb10 2024-09-13T16:36:53.307 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab41734700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3c14dd60 con 0x7fab3c1440d0 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab3c1444b0 0x7fab3c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3a7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c0ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab3c1444b0 0x7fab3c0ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56916/0 (socket says 172.21.15.119:56916) 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 -- 172.21.15.119:0/3954627310 learned_addr learned my addr 172.21.15.119:0/3954627310 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3affd700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c0cbb10 0x7fab3c1f1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 -- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 msgr2=0x7fab3c0ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c0ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 -- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c0cbb10 msgr2=0x7fab3c1f1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c0cbb10 0x7fab3c1f1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 -- 172.21.15.119:0/3954627310 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab240094b0 con 0x7fab3c1444b0 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab3b7fe700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab3c1444b0 0x7fab3c0ace00 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fab3400caa0 tx=0x7fab3400cad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.308 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab337fe700 1 -- 172.21.15.119:0/3954627310 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34007480 con 0x7fab3c1444b0 2024-09-13T16:36:53.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.301+0000 7fab337fe700 1 -- 172.21.15.119:0/3954627310 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab340075e0 con 0x7fab3c1444b0 2024-09-13T16:36:53.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab2c0094b0 con 0x7fab3c1444b0 2024-09-13T16:36:53.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab337fe700 1 -- 172.21.15.119:0/3954627310 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab34007970 con 0x7fab3c1444b0 2024-09-13T16:36:53.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab3c14e320 con 0x7fab3c1444b0 2024-09-13T16:36:53.309 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab3c067760 con 0x7fab3c1444b0 2024-09-13T16:36:53.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab337fe700 1 -- 172.21.15.119:0/3954627310 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fab3401b3b0 con 0x7fab3c1444b0 2024-09-13T16:36:53.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab337fe700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab1c066730 0x7fab1c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab337fe700 1 -- 172.21.15.119:0/3954627310 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7fab34083c80 con 0x7fab3c1444b0 2024-09-13T16:36:53.310 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.305+0000 7fab3affd700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab1c066730 0x7fab1c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.311 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.309+0000 7fab3affd700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab1c066730 0x7fab1c068bf0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fab24009d80 tx=0x7fab24009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.320 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.317+0000 7fab337fe700 1 -- 172.21.15.119:0/3954627310 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fab34050d80 con 0x7fab3c1444b0 2024-09-13T16:36:53.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.493+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab3c14e7d0 con 0x7fab3c1444b0 2024-09-13T16:36:53.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab337fe700 1 -- 172.21.15.119:0/3954627310 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17107 (secure 0 0 0) 0x7fab340543a0 con 0x7fab3c1444b0 2024-09-13T16:36:53.500 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:53.500 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":128,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:36:42.101635+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6835/3901706437":"2024-09-14T16:36:21.631788+0000","172.21.15.145:6834/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6834/3901706437":"2024-09-14T16:36:21.631788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:53.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab1c066730 msgr2=0x7fab1c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab1c066730 0x7fab1c068bf0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fab24009d80 tx=0x7fab24009240 comp rx=0 tx=0).stop 2024-09-13T16:36:53.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab3c1444b0 msgr2=0x7fab3c0ace00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab3c1444b0 0x7fab3c0ace00 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7fab3400caa0 tx=0x7fab3400cad0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 shutdown_connections 2024-09-13T16:36:53.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fab1c066730 0x7fab1c068bf0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fab3c1444b0 0x7fab3c0ace00 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.497+0000 7fab41734700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fab3c1440d0 0x7fab3c0ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.501+0000 7fab41734700 1 --2- 172.21.15.119:0/3954627310 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fab3c0cbb10 0x7fab3c1f1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.501+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 >> 172.21.15.119:0/3954627310 conn(0x7fab3c06a350 msgr2=0x7fab3c066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:53.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.501+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 shutdown_connections 2024-09-13T16:36:53.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.501+0000 7fab41734700 1 -- 172.21.15.119:0/3954627310 wait complete. 2024-09-13T16:36:53.516 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:53.516 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:53.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.637+0000 7f94724b2700 1 Processor -- start 2024-09-13T16:36:53.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.637+0000 7f94724b2700 1 -- start start 2024-09-13T16:36:53.646 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.637+0000 7f94724b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 0x7f946c0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.637+0000 7f94724b2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0d05b0 con 0x7f946c0ce0d0 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.637+0000 7f94724b2700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946c0d0870 con 0x7f946c0cf930 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.637+0000 7f94724b2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0d0b50 con 0x7f946c15cad0 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.637+0000 7f9470caf700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f946c0ce0d0 0x7f946c0cf220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49596/0 (socket says 172.21.15.119:49596) 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 -- 172.21.15.119:0/3055469448 learned_addr learned my addr 172.21.15.119:0/3055469448 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94714b0700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 0x7f946c0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476583845 0 0) 0x7f946c0d05b0 con 0x7f946c0ce0d0 2024-09-13T16:36:53.647 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94580035d0 con 0x7f946c0ce0d0 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2365229080 0 0) 0x7f94580035d0 con 0x7f946c0ce0d0 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 msgr2=0x7f946c0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 0x7f946c0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 >> v1:172.21.15.145:6789/0 conn(0x7f946c0cf930 legacy=0x7f946c0b5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c0d0e30 con 0x7f946c0ce0d0 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f9468001670 con 0x7f946c0ce0d0 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 >> v1:172.21.15.119:6789/0 conn(0x7f946c0ce0d0 legacy=0x7f946c0cf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.648 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9458003a50 0x7f9458003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9458004870 0x7f9458004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.649 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 0x7f9458005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0d0b50 con 0x7f9458004870 2024-09-13T16:36:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f945800bae0 con 0x7f9458003a50 2024-09-13T16:36:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f945800bca0 con 0x7f946c15cad0 2024-09-13T16:36:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9458004870 0x7f9458004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94714b0700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9458003a50 0x7f9458003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9471cb1700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 0x7f9458005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.650 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 -- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 msgr2=0x7f9458005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 0x7f9458005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 -- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9458003a50 msgr2=0x7f9458003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9458003a50 0x7f9458003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 -- 172.21.15.119:0/3055469448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580035d0 con 0x7f9458004870 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9471cb1700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c15cad0 0x7f9458005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9470caf700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9458004870 0x7f9458004cf0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f946c0c60f0 tx=0x7f94580058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=245e17ca295fe6fe server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468007c30 con 0x7f9458004870 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f946800ce20 con 0x7f9458004870 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9463fff700 1 -- 172.21.15.119:0/3055469448 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468005bc0 con 0x7f9458004870 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94724b2700 1 -- 172.21.15.119:0/3055469448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9458004870 msgr2=0x7f9458004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94724b2700 1 --2- 172.21.15.119:0/3055469448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9458004870 0x7f9458004cf0 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7f946c0c60f0 tx=0x7f94580058a0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f9471cb1700 1 -- 172.21.15.119:0/3055469448 reap_dead start 2024-09-13T16:36:53.651 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94724b2700 1 -- 172.21.15.119:0/3055469448 shutdown_connections 2024-09-13T16:36:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94724b2700 1 -- 172.21.15.119:0/3055469448 >> 172.21.15.119:0/3055469448 conn(0x7f946c08bf70 msgr2=0x7f946c0d29e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94724b2700 1 -- 172.21.15.119:0/3055469448 shutdown_connections 2024-09-13T16:36:53.652 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.641+0000 7f94724b2700 1 -- 172.21.15.119:0/3055469448 wait complete. 2024-09-13T16:36:53.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 Processor -- start 2024-09-13T16:36:53.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 -- start start 2024-09-13T16:36:53.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f946c14e050 0x7f946c0b6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c0b6da0 0x7f946c0b7220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.654 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c1f6000 0x7f946c1f6480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c0ca700 con 0x7f946c14e050 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946c09ca20 con 0x7f946c0b6da0 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946c09cd00 con 0x7f946c1f6000 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f946c14e050 0x7f946c0b6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f946c14e050 0x7f946c0b6860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56934/0 (socket says 172.21.15.119:56934) 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 -- 172.21.15.119:0/3179506884 learned_addr learned my addr 172.21.15.119:0/3179506884 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f9470caf700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c0b6da0 0x7f946c0b7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f9471cb1700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c1f6000 0x7f946c1f6480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 -- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c1f6000 msgr2=0x7f946c1f6480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.655 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c1f6000 0x7f946c1f6480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 -- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c0b6da0 msgr2=0x7f946c0b7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c0b6da0 0x7f946c0b7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 -- 172.21.15.119:0/3179506884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946c09e6e0 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f9470caf700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c0b6da0 0x7f946c0b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94714b0700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f946c14e050 0x7f946c0b6860 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f945c011a20 tx=0x7f945c011de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f9461ffb700 1 -- 172.21.15.119:0/3179506884 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c00fa00 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f9461ffb700 1 -- 172.21.15.119:0/3179506884 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c00fb60 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f946c09e9c0 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.645+0000 7f9461ffb700 1 -- 172.21.15.119:0/3179506884 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c014440 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.649+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f946c14e430 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.649+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946c0ce8a0 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.649+0000 7f9461ffb700 1 -- 172.21.15.119:0/3179506884 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f945c0145a0 con 0x7f946c14e050 2024-09-13T16:36:53.656 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.649+0000 7f9461ffb700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9450066760 0x7f9450068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.649+0000 7f9461ffb700 1 -- 172.21.15.119:0/3179506884 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f945c084af0 con 0x7f946c14e050 2024-09-13T16:36:53.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.649+0000 7f9470caf700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9450066760 0x7f9450068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.657 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.653+0000 7f9470caf700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9450066760 0x7f9450068c20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9468000c00 tx=0x7f9468007000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.662 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.657+0000 7f9461ffb700 1 -- 172.21.15.119:0/3179506884 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f945c051d30 con 0x7f946c14e050 2024-09-13T16:36:53.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.829+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f946c0d38f0 con 0x7f946c14e050 2024-09-13T16:36:53.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.829+0000 7f9461ffb700 1 -- 172.21.15.119:0/3179506884 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v128) v1 ==== 93+0+10 (secure 0 0 0) 0x7f945c055350 con 0x7f946c14e050 2024-09-13T16:36:53.833 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:53.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.829+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9450066760 msgr2=0x7f9450068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9450066760 0x7f9450068c20 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9468000c00 tx=0x7f9468007000 comp rx=0 tx=0).stop 2024-09-13T16:36:53.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f946c14e050 msgr2=0x7f946c0b6860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f946c14e050 0x7f946c0b6860 secure :-1 s=READY pgs=1978 cs=0 l=1 rev1=1 crypto rx=0x7f945c011a20 tx=0x7f945c011de0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 shutdown_connections 2024-09-13T16:36:53.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9450066760 0x7f9450068c20 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f946c1f6000 0x7f946c1f6480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f946c0b6da0 0x7f946c0b7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 --2- 172.21.15.119:0/3179506884 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f946c14e050 0x7f946c0b6860 unknown :-1 s=CLOSED pgs=1978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 >> 172.21.15.119:0/3179506884 conn(0x7f946c08bf70 msgr2=0x7f946c06a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:53.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 shutdown_connections 2024-09-13T16:36:53.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.833+0000 7f94724b2700 1 -- 172.21.15.119:0/3179506884 wait complete. 2024-09-13T16:36:53.847 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:53.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.961+0000 7f2f36011700 1 Processor -- start 2024-09-13T16:36:53.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.961+0000 7f2f36011700 1 -- start start 2024-09-13T16:36:53.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f36011700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300add60 0x7f2f300ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f36011700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2f300b3260 0x7f2f300b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f36011700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f300cd4d0 con 0x7f2f3015dbe0 2024-09-13T16:36:53.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f36011700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f30071be0 con 0x7f2f300b3260 2024-09-13T16:36:53.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f36011700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f30159af0 con 0x7f2f300add60 2024-09-13T16:36:53.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f35810700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2f300b3260 0x7f2f300b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f3500f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300add60 0x7f2f300ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f3480e700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f2f3015dbe0 0x7f2f300cd960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40442/0 (socket says 172.21.15.119:40442) 2024-09-13T16:36:53.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f3480e700 1 -- 172.21.15.119:0/3976422808 learned_addr learned my addr 172.21.15.119:0/3976422808 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:53.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f35810700 1 -- 172.21.15.119:0/3976422808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300add60 msgr2=0x7f2f300ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:53.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f35810700 1 --2- 172.21.15.119:0/3976422808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300add60 0x7f2f300ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.969 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f35810700 1 -- 172.21.15.119:0/3976422808 >> v1:172.21.15.145:6789/0 conn(0x7f2f3015dbe0 legacy=0x7f2f300cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f27fff700 1 -- 172.21.15.119:0/3976422808 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992028073 0 0) 0x7f2f300cd4d0 con 0x7f2f3015dbe0 2024-09-13T16:36:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f35810700 1 -- 172.21.15.119:0/3976422808 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3015e130 con 0x7f2f300b3260 2024-09-13T16:36:53.970 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f35810700 1 --2- 172.21.15.119:0/3976422808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 0x7f2f300b36f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009d90 tx=0x7f2f2800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e5e9972afde47166 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f27fff700 1 -- 172.21.15.119:0/3976422808 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2800e070 con 0x7f2f300b3260 2024-09-13T16:36:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f27fff700 1 -- 172.21.15.119:0/3976422808 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2800bb20 con 0x7f2f300b3260 2024-09-13T16:36:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.965+0000 7f2f27fff700 1 -- 172.21.15.119:0/3976422808 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f28012430 con 0x7f2f300b3260 2024-09-13T16:36:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 -- 172.21.15.119:0/3976422808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 msgr2=0x7f2f300b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 --2- 172.21.15.119:0/3976422808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 0x7f2f300b36f0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f2f28009d90 tx=0x7f2f2800b330 comp rx=0 tx=0).stop 2024-09-13T16:36:53.971 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 -- 172.21.15.119:0/3976422808 shutdown_connections 2024-09-13T16:36:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 --2- 172.21.15.119:0/3976422808 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 0x7f2f300b36f0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 --2- 172.21.15.119:0/3976422808 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f300add60 0x7f2f300ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 -- 172.21.15.119:0/3976422808 >> 172.21.15.119:0/3976422808 conn(0x7f2f300b0980 msgr2=0x7f2f300b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 -- 172.21.15.119:0/3976422808 shutdown_connections 2024-09-13T16:36:53.972 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 -- 172.21.15.119:0/3976422808 wait complete. 2024-09-13T16:36:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 Processor -- start 2024-09-13T16:36:53.973 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.969+0000 7f2f36011700 1 -- start start 2024-09-13T16:36:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300add60 0x7f2f301f31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 0x7f2f3015c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f3015dbe0 0x7f2f30066090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f30152530 con 0x7f2f300add60 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f301f3720 con 0x7f2f300b3260 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f301f2590 con 0x7f2f3015dbe0 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300add60 0x7f2f301f31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300add60 0x7f2f301f31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56940/0 (socket says 172.21.15.119:56940) 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 -- 172.21.15.119:0/2099543593 learned_addr learned my addr 172.21.15.119:0/2099543593 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f35810700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f3015dbe0 0x7f2f30066090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3480e700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 0x7f2f3015c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 -- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f3015dbe0 msgr2=0x7f2f30066090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f3015dbe0 0x7f2f30066090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 -- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 msgr2=0x7f2f3015c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 0x7f2f3015c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 -- 172.21.15.119:0/2099543593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f28009a70 con 0x7f2f300add60 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f3500f700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300add60 0x7f2f301f31e0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00e840 tx=0x7f2f2c00eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f25ffb700 1 -- 172.21.15.119:0/2099543593 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0095a0 con 0x7f2f300add60 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f25ffb700 1 -- 172.21.15.119:0/2099543593 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f2c013680 con 0x7f2f300add60 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f301f2870 con 0x7f2f300add60 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f25ffb700 1 -- 172.21.15.119:0/2099543593 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f2c0139f0 con 0x7f2f300add60 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.973+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f301f2e30 con 0x7f2f300add60 2024-09-13T16:36:53.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.977+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f300af2c0 con 0x7f2f300add60 2024-09-13T16:36:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.977+0000 7f2f25ffb700 1 -- 172.21.15.119:0/2099543593 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2f2c013b50 con 0x7f2f300add60 2024-09-13T16:36:53.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.977+0000 7f2f25ffb700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f14066730 0x7f2f14068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.977+0000 7f2f25ffb700 1 -- 172.21.15.119:0/2099543593 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f2f2c084170 con 0x7f2f300add60 2024-09-13T16:36:53.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.977+0000 7f2f3480e700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f14066730 0x7f2f14068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:53.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.977+0000 7f2f3480e700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f14066730 0x7f2f14068bf0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2f20007b30 tx=0x7f2f20007a80 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:53.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:53.985+0000 7f2f25ffb700 1 -- 172.21.15.119:0/2099543593 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2f2c0512f0 con 0x7f2f300add60 2024-09-13T16:36:54.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.153+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2f300ac960 con 0x7f2f300add60 2024-09-13T16:36:54.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.153+0000 7f2f25ffb700 1 -- 172.21.15.119:0/2099543593 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2f2c054910 con 0x7f2f300add60 2024-09-13T16:36:54.158 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:54.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f14066730 msgr2=0x7f2f14068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f14066730 0x7f2f14068bf0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f2f20007b30 tx=0x7f2f20007a80 comp rx=0 tx=0).stop 2024-09-13T16:36:54.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300add60 msgr2=0x7f2f301f31e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300add60 0x7f2f301f31e0 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c00e840 tx=0x7f2f2c00eb50 comp rx=0 tx=0).stop 2024-09-13T16:36:54.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 shutdown_connections 2024-09-13T16:36:54.160 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2f14066730 0x7f2f14068bf0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2f3015dbe0 0x7f2f30066090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2f300b3260 0x7f2f3015c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 --2- 172.21.15.119:0/2099543593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2f300add60 0x7f2f301f31e0 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 >> 172.21.15.119:0/2099543593 conn(0x7f2f300b0980 msgr2=0x7f2f3006fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:54.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 shutdown_connections 2024-09-13T16:36:54.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.157+0000 7f2f36011700 1 -- 172.21.15.119:0/2099543593 wait complete. 2024-09-13T16:36:54.173 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:54.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.281+0000 7fa1dd2a4700 1 Processor -- start 2024-09-13T16:36:54.286 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.281+0000 7fa1dd2a4700 1 -- start start 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d815cae0 0x7fa1d80ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa1d80d0fb0 0x7fa1d80d1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d80cd600 con 0x7fa1d80cff60 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1d80ce1d0 con 0x7fa1d80d0fb0 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d80d19a0 con 0x7fa1d815cae0 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dcaa3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa1d80d0fb0 0x7fa1d80d1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1d77fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fa1d80cff60 0x7fa1d80d05c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49606/0 (socket says 172.21.15.119:49606) 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1d77fe700 1 -- 172.21.15.119:0/502857970 learned_addr learned my addr 172.21.15.119:0/502857970 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:54.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1d7fff700 1 --2- 172.21.15.119:0/502857970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d815cae0 0x7fa1d80ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dcaa3700 1 -- 172.21.15.119:0/502857970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d815cae0 msgr2=0x7fa1d80ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dcaa3700 1 --2- 172.21.15.119:0/502857970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d815cae0 0x7fa1d80ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dcaa3700 1 -- 172.21.15.119:0/502857970 >> v1:172.21.15.119:6789/0 conn(0x7fa1d80cff60 legacy=0x7fa1d80d05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dcaa3700 1 -- 172.21.15.119:0/502857970 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d80d0dd0 con 0x7fa1d80d0fb0 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1d7fff700 1 --2- 172.21.15.119:0/502857970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d815cae0 0x7fa1d80ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dcaa3700 1 --2- 172.21.15.119:0/502857970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d80d0fb0 0x7fa1d80d1460 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009d90 tx=0x7fa1d000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14865538ac70e289 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1d6ffd700 1 -- 172.21.15.119:0/502857970 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d000e070 con 0x7fa1d80d0fb0 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1d6ffd700 1 -- 172.21.15.119:0/502857970 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d000bbd0 con 0x7fa1d80d0fb0 2024-09-13T16:36:54.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1d6ffd700 1 -- 172.21.15.119:0/502857970 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0012430 con 0x7fa1d80d0fb0 2024-09-13T16:36:54.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/502857970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d80d0fb0 msgr2=0x7fa1d80d1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/502857970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d80d0fb0 0x7fa1d80d1460 secure :-1 s=READY pgs=1981 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009d90 tx=0x7fa1d000b330 comp rx=0 tx=0).stop 2024-09-13T16:36:54.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/502857970 shutdown_connections 2024-09-13T16:36:54.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/502857970 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d80d0fb0 0x7fa1d80d1460 unknown :-1 s=CLOSED pgs=1981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/502857970 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d815cae0 0x7fa1d80ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/502857970 >> 172.21.15.119:0/502857970 conn(0x7fa1d80d29f0 msgr2=0x7fa1d80d2e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:54.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/502857970 shutdown_connections 2024-09-13T16:36:54.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.285+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/502857970 wait complete. 2024-09-13T16:36:54.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 Processor -- start 2024-09-13T16:36:54.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 -- start start 2024-09-13T16:36:54.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d80cff60 0x7fa1d81f2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1d80d0fb0 0x7fa1d8153e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d815cae0 0x7fa1d80b5840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d80c9730 con 0x7fa1d815cae0 2024-09-13T16:36:54.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d80b7cd0 con 0x7fa1d80d0fb0 2024-09-13T16:36:54.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d815d170 con 0x7fa1d80cff60 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d815cae0 0x7fa1d80b5840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d815cae0 0x7fa1d80b5840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56962/0 (socket says 172.21.15.119:56962) 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 -- 172.21.15.119:0/1470135167 learned_addr learned my addr 172.21.15.119:0/1470135167 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1d7fff700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d80cff60 0x7fa1d81f2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1d77fe700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1d80d0fb0 0x7fa1d8153e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 -- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d80cff60 msgr2=0x7fa1d81f2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d80cff60 0x7fa1d81f2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 -- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1d80d0fb0 msgr2=0x7fa1d8153e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1d80d0fb0 0x7fa1d8153e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 -- 172.21.15.119:0/1470135167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d0009a70 con 0x7fa1d815cae0 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dcaa3700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d815cae0 0x7fa1d80b5840 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0014040 tx=0x7fa1d0004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1d4ff9700 1 -- 172.21.15.119:0/1470135167 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0012430 con 0x7fa1d815cae0 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1d4ff9700 1 -- 172.21.15.119:0/1470135167 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d0020070 con 0x7fa1d815cae0 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d815d450 con 0x7fa1d815cae0 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1d4ff9700 1 -- 172.21.15.119:0/1470135167 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0004280 con 0x7fa1d815cae0 2024-09-13T16:36:54.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.289+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d81f31c0 con 0x7fa1d815cae0 2024-09-13T16:36:54.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.293+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d80d03a0 con 0x7fa1d815cae0 2024-09-13T16:36:54.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.293+0000 7fa1d4ff9700 1 -- 172.21.15.119:0/1470135167 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa1d00043e0 con 0x7fa1d815cae0 2024-09-13T16:36:54.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.293+0000 7fa1d4ff9700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.293+0000 7fa1d4ff9700 1 -- 172.21.15.119:0/1470135167 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7fa1d0022070 con 0x7fa1d815cae0 2024-09-13T16:36:54.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.293+0000 7fa1d7fff700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.298 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.293+0000 7fa1d7fff700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc006fd0 tx=0x7fa1cc008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:54.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.301+0000 7fa1d4ff9700 1 -- 172.21.15.119:0/1470135167 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa1d00533f0 con 0x7fa1d815cae0 2024-09-13T16:36:54.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.473+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1d80744c0 con 0x7fa1d815cae0 2024-09-13T16:36:54.476 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.473+0000 7fa1d4ff9700 1 -- 172.21.15.119:0/1470135167 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1d0056a10 con 0x7fa1d815cae0 2024-09-13T16:36:54.477 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:54.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 msgr2=0x7fa1c4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.479 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc006fd0 tx=0x7fa1cc008040 comp rx=0 tx=0).stop 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d815cae0 msgr2=0x7fa1d80b5840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d815cae0 0x7fa1d80b5840 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0014040 tx=0x7fa1d0004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 shutdown_connections 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa1c4066730 0x7fa1c4068bf0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa1d815cae0 0x7fa1d80b5840 unknown :-1 s=CLOSED pgs=1982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa1d80d0fb0 0x7fa1d8153e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 --2- 172.21.15.119:0/1470135167 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa1d80cff60 0x7fa1d81f2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 >> 172.21.15.119:0/1470135167 conn(0x7fa1d80d29f0 msgr2=0x7fa1d80d2dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 shutdown_connections 2024-09-13T16:36:54.480 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.477+0000 7fa1dd2a4700 1 -- 172.21.15.119:0/1470135167 wait complete. 2024-09-13T16:36:54.491 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-09-13T16:36:54.610 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.605+0000 7fbeb648d700 1 Processor -- start 2024-09-13T16:36:54.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.605+0000 7fbeb648d700 1 -- start start 2024-09-13T16:36:54.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbeb0159800 0x7fbeb015d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.611 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb00b3830 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb00b2c20 con 0x7fbeb006cbb0 2024-09-13T16:36:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb00b2f00 con 0x7fbeb0159800 2024-09-13T16:36:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb5c8c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fbeb00b3cc0 0x7fbeb00b4150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49618/0 (socket says 172.21.15.119:49618) 2024-09-13T16:36:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb548b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fbeb0159800 0x7fbeb015d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.612 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb5c8c700 1 -- 172.21.15.119:0/4141640445 learned_addr learned my addr 172.21.15.119:0/4141640445 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbea3fff700 1 -- 172.21.15.119:0/4141640445 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3863203984 0 0) 0x7fbeb00b3830 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.613 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbea3fff700 1 -- 172.21.15.119:0/4141640445 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9c0035d0 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb548b700 1 -- 172.21.15.119:0/4141640445 >> v1:172.21.15.145:6789/0 conn(0x7fbeb006cbb0 legacy=0x7fbeb015d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb548b700 1 -- 172.21.15.119:0/4141640445 >> v1:172.21.15.119:6789/0 conn(0x7fbeb00b3cc0 legacy=0x7fbeb00b4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb548b700 1 -- 172.21.15.119:0/4141640445 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb00b31e0 con 0x7fbeb0159800 2024-09-13T16:36:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb548b700 1 --2- 172.21.15.119:0/4141640445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb0159800 0x7fbeb015d240 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009850 tx=0x7fbeac009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4274cd87faf14c14 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbea3fff700 1 -- 172.21.15.119:0/4141640445 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac011830 con 0x7fbeb0159800 2024-09-13T16:36:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbea3fff700 1 -- 172.21.15.119:0/4141640445 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeac011990 con 0x7fbeb0159800 2024-09-13T16:36:54.614 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbea3fff700 1 -- 172.21.15.119:0/4141640445 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeac011d40 con 0x7fbeb0159800 2024-09-13T16:36:54.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- 172.21.15.119:0/4141640445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb0159800 msgr2=0x7fbeb015d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 --2- 172.21.15.119:0/4141640445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb0159800 0x7fbeb015d240 secure :-1 s=READY pgs=1984 cs=0 l=1 rev1=1 crypto rx=0x7fbeac009850 tx=0x7fbeac009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:54.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- 172.21.15.119:0/4141640445 shutdown_connections 2024-09-13T16:36:54.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 --2- 172.21.15.119:0/4141640445 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb0159800 0x7fbeb015d240 unknown :-1 s=CLOSED pgs=1984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- 172.21.15.119:0/4141640445 >> 172.21.15.119:0/4141640445 conn(0x7fbeb00b17a0 msgr2=0x7fbeb00b1bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:54.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- 172.21.15.119:0/4141640445 shutdown_connections 2024-09-13T16:36:54.615 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.609+0000 7fbeb648d700 1 -- 172.21.15.119:0/4141640445 wait complete. 2024-09-13T16:36:54.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 Processor -- start 2024-09-13T16:36:54.616 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 -- start start 2024-09-13T16:36:54.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbeb006cbb0 0x7fbeb0133480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb00b3cc0 0x7fbeb00ad620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbeb0159800 0x7fbeb0142f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0143900 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0143be0 con 0x7fbeb006cbb0 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb0143ec0 con 0x7fbeb0159800 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb00b3cc0 0x7fbeb00ad620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb00b3cc0 0x7fbeb00ad620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:56976/0 (socket says 172.21.15.119:56976) 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 -- 172.21.15.119:0/1133332987 learned_addr learned my addr 172.21.15.119:0/1133332987 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb5c8c700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbeb0159800 0x7fbeb0142f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb548b700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbeb006cbb0 0x7fbeb0133480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 -- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbeb0159800 msgr2=0x7fbeb0142f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbeb0159800 0x7fbeb0142f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 -- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbeb006cbb0 msgr2=0x7fbeb0133480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbeb006cbb0 0x7fbeb0133480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 -- 172.21.15.119:0/1133332987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeac0094b0 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb4c8a700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb00b3cc0 0x7fbeb00ad620 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fbea400c500 tx=0x7fbea400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbea1ffb700 1 -- 172.21.15.119:0/1133332987 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea40052d0 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbea1ffb700 1 -- 172.21.15.119:0/1133332987 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea4005430 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb01441a0 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.613+0000 7fbea1ffb700 1 -- 172.21.15.119:0/1133332987 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbea40055a0 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.620 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.617+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb0143660 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.617+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb00b4150 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.617+0000 7fbea1ffb700 1 -- 172.21.15.119:0/1133332987 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fbea40038e0 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.617+0000 7fbea1ffb700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe94066730 0x7fbe94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:54.621 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.617+0000 7fbea1ffb700 1 -- 172.21.15.119:0/1133332987 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7fbea40836d0 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.617+0000 7fbeb548b700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe94066730 0x7fbe94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:54.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.621+0000 7fbeb548b700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe94066730 0x7fbe94068bf0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0090b0 tx=0x7fbeac008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:54.628 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.625+0000 7fbea1ffb700 1 -- 172.21.15.119:0/1133332987 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fbea4050850 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.809+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbeb0071f80 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.809+0000 7fbea1ffb700 1 -- 172.21.15.119:0/1133332987 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 4 entries v128) v1 ==== 80+0+248 (secure 0 0 0) 0x7fbea4053e70 con 0x7fbeb00b3cc0 2024-09-13T16:36:54.814 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6835/3898509964 2024-09-14T16:36:42.101563+0000 2024-09-13T16:36:54.814 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6835/3901706437 2024-09-14T16:36:21.631788+0000 2024-09-13T16:36:54.814 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6834/3898509964 2024-09-14T16:36:42.101563+0000 2024-09-13T16:36:54.814 INFO:teuthology.orchestra.run.smithi119.stdout:172.21.15.145:6834/3901706437 2024-09-14T16:36:21.631788+0000 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe94066730 msgr2=0x7fbe94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe94066730 0x7fbe94068bf0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7fbeac0090b0 tx=0x7fbeac008030 comp rx=0 tx=0).stop 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb00b3cc0 msgr2=0x7fbeb00ad620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb00b3cc0 0x7fbeb00ad620 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fbea400c500 tx=0x7fbea400c810 comp rx=0 tx=0).stop 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 shutdown_connections 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fbe94066730 0x7fbe94068bf0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fbeb0159800 0x7fbeb0142f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fbeb00b3cc0 0x7fbeb00ad620 unknown :-1 s=CLOSED pgs=1985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 --2- 172.21.15.119:0/1133332987 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fbeb006cbb0 0x7fbeb0133480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:54.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 >> 172.21.15.119:0/1133332987 conn(0x7fbeb00b17a0 msgr2=0x7fbeb014d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:54.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 shutdown_connections 2024-09-13T16:36:54.818 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:54.813+0000 7fbeb648d700 1 -- 172.21.15.119:0/1133332987 wait complete. 2024-09-13T16:36:54.818 INFO:teuthology.orchestra.run.smithi119.stderr:listed 4 entries 2024-09-13T16:36:54.828 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi119.front.sepia.ceph.com... 2024-09-13T16:36:54.828 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:54.828 DEBUG:teuthology.orchestra.run.smithi119:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-09-13T16:36:54.842 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi119.stderr:ceph-fuse[55653]: fuse finished with error 0 and tester_r 0 2024-09-13T16:36:54.843 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:36:54.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:54.843 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-09-13T16:36:54.967 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:36:54.967 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:54.975 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:36:54.975 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:36:54.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:54.976 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-09-13T16:36:55.024 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-09-13T16:36:55.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:55.025 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:36:55.026 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:55.033 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:36:55.033 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:36:55.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:55.034 DEBUG:teuthology.orchestra.run.smithi119:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-09-13T16:36:55.080 INFO:teuthology.orchestra.run.smithi119.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-09-13T16:36:55.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:55.082 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:36:55.082 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:36:55.089 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:36:55.089 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:36:55.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:36:55.089 DEBUG:teuthology.orchestra.run.smithi145:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-09-13T16:36:55.096 INFO:teuthology.orchestra.run.smithi145.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-09-13T16:36:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:36:55.097 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:55.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c711c700 1 Processor -- start 2024-09-13T16:36:55.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c711c700 1 -- start start 2024-09-13T16:36:55.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c711c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fb5c00cd8d0 0x7fb5c006cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c711c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb5c00b2200 0x7fb5c00b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c711c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5c00705d0 con 0x7fb5c00b17f0 2024-09-13T16:36:55.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c711c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c00cbf10 con 0x7fb5c00b2200 2024-09-13T16:36:55.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c711c700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5c00b2bb0 con 0x7fb5c00cd8d0 2024-09-13T16:36:55.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c5919700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb5c00b17f0 0x7fb5c00cb800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49624/0 (socket says 172.21.15.119:49624) 2024-09-13T16:36:55.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c5919700 1 -- 172.21.15.119:0/464460584 learned_addr learned my addr 172.21.15.119:0/464460584 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:55.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c691b700 1 --2- 172.21.15.119:0/464460584 >> v2:172.21.15.119:3300/0 conn(0x7fb5c00b2200 0x7fb5c00b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c611a700 1 --2- 172.21.15.119:0/464460584 >> v2:172.21.15.145:3300/0 conn(0x7fb5c00cd8d0 0x7fb5c006cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c691b700 1 -- 172.21.15.119:0/464460584 >> v2:172.21.15.145:3300/0 conn(0x7fb5c00cd8d0 msgr2=0x7fb5c006cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c691b700 1 --2- 172.21.15.119:0/464460584 >> v2:172.21.15.145:3300/0 conn(0x7fb5c00cd8d0 0x7fb5c006cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c691b700 1 -- 172.21.15.119:0/464460584 >> v1:172.21.15.119:6789/0 conn(0x7fb5c00b17f0 legacy=0x7fb5c00cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c691b700 1 -- 172.21.15.119:0/464460584 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c006d0d0 con 0x7fb5c00b2200 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c691b700 1 --2- 172.21.15.119:0/464460584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c00b2670 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0071850 tx=0x7fb5b8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=350a6fc1ac354dc9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c5118700 1 -- 172.21.15.119:0/464460584 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8011830 con 0x7fb5c00b2200 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c5118700 1 -- 172.21.15.119:0/464460584 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b8011990 con 0x7fb5c00b2200 2024-09-13T16:36:55.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.213+0000 7fb5c5118700 1 -- 172.21.15.119:0/464460584 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b8011d20 con 0x7fb5c00b2200 2024-09-13T16:36:55.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- 172.21.15.119:0/464460584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 msgr2=0x7fb5c00b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 --2- 172.21.15.119:0/464460584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c00b2670 secure :-1 s=READY pgs=1987 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0071850 tx=0x7fb5b8009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:55.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- 172.21.15.119:0/464460584 shutdown_connections 2024-09-13T16:36:55.220 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 --2- 172.21.15.119:0/464460584 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c00b2670 unknown :-1 s=CLOSED pgs=1987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 --2- 172.21.15.119:0/464460584 >> v2:172.21.15.145:3300/0 conn(0x7fb5c00cd8d0 0x7fb5c006cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- 172.21.15.119:0/464460584 >> 172.21.15.119:0/464460584 conn(0x7fb5c006a350 msgr2=0x7fb5c006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:55.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- 172.21.15.119:0/464460584 shutdown_connections 2024-09-13T16:36:55.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- 172.21.15.119:0/464460584 wait complete. 2024-09-13T16:36:55.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 Processor -- start 2024-09-13T16:36:55.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- start start 2024-09-13T16:36:55.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c00b17f0 0x7fb5c015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c0151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c00cd8d0 0x7fb5c00ad790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c015e840 con 0x7fb5c00b2200 2024-09-13T16:36:55.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c015ce40 con 0x7fb5c00b17f0 2024-09-13T16:36:55.222 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c711c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5c01519d0 con 0x7fb5c00cd8d0 2024-09-13T16:36:55.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c5919700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c0151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c5919700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c0151490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57000/0 (socket says 172.21.15.119:57000) 2024-09-13T16:36:55.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c5919700 1 -- 172.21.15.119:0/967634520 learned_addr learned my addr 172.21.15.119:0/967634520 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:55.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c691b700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c00cd8d0 0x7fb5c00ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.223 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.217+0000 7fb5c611a700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c00b17f0 0x7fb5c015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c5919700 1 -- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c00cd8d0 msgr2=0x7fb5c00ad790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c5919700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c00cd8d0 0x7fb5c00ad790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.224 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c5919700 1 -- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c00b17f0 msgr2=0x7fb5c015aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c5919700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c00b17f0 0x7fb5c015aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c5919700 1 -- 172.21.15.119:0/967634520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b80094b0 con 0x7fb5c00b2200 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c5919700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c0151490 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000cb40 tx=0x7fb5b000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5beffd700 1 -- 172.21.15.119:0/967634520 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0007480 con 0x7fb5c00b2200 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5beffd700 1 -- 172.21.15.119:0/967634520 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5b00075e0 con 0x7fb5c00b2200 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5c015e2a0 con 0x7fb5c00b2200 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5beffd700 1 -- 172.21.15.119:0/967634520 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5b0007970 con 0x7fb5c00b2200 2024-09-13T16:36:55.225 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5c00ad270 con 0x7fb5c00b2200 2024-09-13T16:36:55.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5c00ad0d0 con 0x7fb5c00b2200 2024-09-13T16:36:55.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.221+0000 7fb5beffd700 1 -- 172.21.15.119:0/967634520 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fb5b0012400 con 0x7fb5c00b2200 2024-09-13T16:36:55.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.225+0000 7fb5beffd700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5ac066730 0x7fb5ac068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.225+0000 7fb5beffd700 1 -- 172.21.15.119:0/967634520 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7fb5b0083ba0 con 0x7fb5c00b2200 2024-09-13T16:36:55.227 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.225+0000 7fb5c611a700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5ac066730 0x7fb5ac068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.228 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.225+0000 7fb5c611a700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5ac066730 0x7fb5ac068bf0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009b50 tx=0x7fb5b4009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.229+0000 7fb5beffd700 1 -- 172.21.15.119:0/967634520 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fb5c00ad0d0 con 0x7fb5c00b2200 2024-09-13T16:36:55.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.433+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5c00ad0d0 con 0x7fb5c00b2200 2024-09-13T16:36:55.439 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5beffd700 1 -- 172.21.15.119:0/967634520 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4744 (secure 0 0 0) 0x7fb5c00ad0d0 con 0x7fb5c00b2200 2024-09-13T16:36:55.440 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:55.440 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:47.145416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9646":{"gid":9646,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6835/2905269684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2905269684},{"type":"v1","addr":"172.21.15.145:6835","nonce":2905269684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:55.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5ac066730 msgr2=0x7fb5ac068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5ac066730 0x7fb5ac068bf0 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7fb5b4009b50 tx=0x7fb5b4009240 comp rx=0 tx=0).stop 2024-09-13T16:36:55.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 msgr2=0x7fb5c0151490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c0151490 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fb5b000cb40 tx=0x7fb5b000cb70 comp rx=0 tx=0).stop 2024-09-13T16:36:55.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 shutdown_connections 2024-09-13T16:36:55.442 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fb5ac066730 0x7fb5ac068bf0 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fb5c00cd8d0 0x7fb5c00ad790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb5c00b2200 0x7fb5c0151490 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 --2- 172.21.15.119:0/967634520 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fb5c00b17f0 0x7fb5c015aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 >> 172.21.15.119:0/967634520 conn(0x7fb5c006a350 msgr2=0x7fb5c006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:55.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 shutdown_connections 2024-09-13T16:36:55.443 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.437+0000 7fb5c711c700 1 -- 172.21.15.119:0/967634520 wait complete. 2024-09-13T16:36:55.443 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 105 2024-09-13T16:36:55.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01759e700 1 Processor -- start 2024-09-13T16:36:55.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01759e700 1 -- start start 2024-09-13T16:36:55.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff018158b30 0x7ff018071030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01806de00 con 0x7ff01806d540 2024-09-13T16:36:55.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01806db20 con 0x7ff018158f10 2024-09-13T16:36:55.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01806c2a0 con 0x7ff018158b30 2024-09-13T16:36:55.569 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff016d9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff01806d540 0x7ff01806bb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49638/0 (socket says 172.21.15.119:49638) 2024-09-13T16:36:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff016d9d700 1 -- 172.21.15.119:0/3579123390 learned_addr learned my addr 172.21.15.119:0/3579123390 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01659c700 1 --2- 172.21.15.119:0/3579123390 >> v2:172.21.15.119:3300/0 conn(0x7ff018158b30 0x7ff018071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01559a700 1 -- 172.21.15.119:0/3579123390 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590056492 0 0) 0x7ff01806de00 con 0x7ff01806d540 2024-09-13T16:36:55.570 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01559a700 1 -- 172.21.15.119:0/3579123390 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0040035d0 con 0x7ff01806d540 2024-09-13T16:36:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01659c700 1 -- 172.21.15.119:0/3579123390 >> v1:172.21.15.145:6789/0 conn(0x7ff018158f10 legacy=0x7ff01806ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01659c700 1 -- 172.21.15.119:0/3579123390 >> v1:172.21.15.119:6789/0 conn(0x7ff01806d540 legacy=0x7ff01806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01559a700 1 -- 172.21.15.119:0/3579123390 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2966449796 0 0) 0x7ff0040035d0 con 0x7ff01806d540 2024-09-13T16:36:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01659c700 1 -- 172.21.15.119:0/3579123390 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0180b1920 con 0x7ff018158b30 2024-09-13T16:36:55.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01659c700 1 --2- 172.21.15.119:0/3579123390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff018158b30 0x7ff018071030 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7ff000009850 tx=0x7ff000009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a7dac87d7b86f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01559a700 1 -- 172.21.15.119:0/3579123390 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010001670 con 0x7ff018158b30 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01559a700 1 -- 172.21.15.119:0/3579123390 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff000004030 con 0x7ff018158b30 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.565+0000 7ff01559a700 1 -- 172.21.15.119:0/3579123390 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0000043a0 con 0x7ff018158b30 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- 172.21.15.119:0/3579123390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff018158b30 msgr2=0x7ff018071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 --2- 172.21.15.119:0/3579123390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff018158b30 0x7ff018071030 secure :-1 s=READY pgs=1990 cs=0 l=1 rev1=1 crypto rx=0x7ff000009850 tx=0x7ff000009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- 172.21.15.119:0/3579123390 shutdown_connections 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 --2- 172.21.15.119:0/3579123390 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff018158b30 0x7ff018071030 unknown :-1 s=CLOSED pgs=1990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- 172.21.15.119:0/3579123390 >> 172.21.15.119:0/3579123390 conn(0x7ff01806a350 msgr2=0x7ff01806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- 172.21.15.119:0/3579123390 shutdown_connections 2024-09-13T16:36:55.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- 172.21.15.119:0/3579123390 wait complete. 2024-09-13T16:36:55.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 Processor -- start 2024-09-13T16:36:55.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- start start 2024-09-13T16:36:55.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff01806d540 0x7ff018147f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff018158b30 0x7ff018144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff018158f10 0x7ff0180a9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff018146300 con 0x7ff01806d540 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff018145470 con 0x7ff018158b30 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff018148450 con 0x7ff018158f10 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff01806d540 0x7ff018147f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01659c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff01806d540 0x7ff018147f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57020/0 (socket says 172.21.15.119:57020) 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.569+0000 7ff01659c700 1 -- 172.21.15.119:0/2693606518 learned_addr learned my addr 172.21.15.119:0/2693606518 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff016d9d700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff018158f10 0x7ff0180a9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff015d9b700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff018158b30 0x7ff018144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01659c700 1 -- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff018158f10 msgr2=0x7ff0180a9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01659c700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff018158f10 0x7ff0180a9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.576 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01659c700 1 -- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff018158b30 msgr2=0x7ff018144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01659c700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff018158b30 0x7ff018144c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01659c700 1 -- 172.21.15.119:0/2693606518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0000094b0 con 0x7ff01806d540 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01659c700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff01806d540 0x7ff018147f10 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7ff0180b3ee0 tx=0x7ff000004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff00effd700 1 -- 172.21.15.119:0/2693606518 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000003ea0 con 0x7ff01806d540 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff00effd700 1 -- 172.21.15.119:0/2693606518 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff00001f070 con 0x7ff01806d540 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff00effd700 1 -- 172.21.15.119:0/2693606518 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff000011a90 con 0x7ff01806d540 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff018145180 con 0x7ff01806d540 2024-09-13T16:36:55.577 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0181459d0 con 0x7ff01806d540 2024-09-13T16:36:55.578 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01806bb90 con 0x7ff01806d540 2024-09-13T16:36:55.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff00effd700 1 -- 172.21.15.119:0/2693606518 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff000011bf0 con 0x7ff01806d540 2024-09-13T16:36:55.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff00effd700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff8066730 0x7feff8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff00effd700 1 -- 172.21.15.119:0/2693606518 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7ff000021070 con 0x7ff01806d540 2024-09-13T16:36:55.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.573+0000 7ff015d9b700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff8066730 0x7feff8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.579 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.577+0000 7ff015d9b700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff8066730 0x7feff8068bf0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff0181538b0 tx=0x7ff008009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.584 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.581+0000 7ff00effd700 1 -- 172.21.15.119:0/2693606518 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7ff00008c080 con 0x7ff01806d540 2024-09-13T16:36:55.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.749+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff018071160 con 0x7ff01806d540 2024-09-13T16:36:55.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff00effd700 1 -- 172.21.15.119:0/2693606518 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17107 (secure 0 0 0) 0x7ff00002b090 con 0x7ff01806d540 2024-09-13T16:36:55.756 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:55.756 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":128,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:36:42.101635+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6835/3901706437":"2024-09-14T16:36:21.631788+0000","172.21.15.145:6834/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6834/3901706437":"2024-09-14T16:36:21.631788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff8066730 msgr2=0x7feff8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff8066730 0x7feff8068bf0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7ff0181538b0 tx=0x7ff008009040 comp rx=0 tx=0).stop 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff01806d540 msgr2=0x7ff018147f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff01806d540 0x7ff018147f10 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7ff0180b3ee0 tx=0x7ff000004770 comp rx=0 tx=0).stop 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 shutdown_connections 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7feff8066730 0x7feff8068bf0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff018158f10 0x7ff0180a9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff018158b30 0x7ff018144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 --2- 172.21.15.119:0/2693606518 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff01806d540 0x7ff018147f10 unknown :-1 s=CLOSED pgs=1991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 >> 172.21.15.119:0/2693606518 conn(0x7ff01806a350 msgr2=0x7ff01815bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:55.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 shutdown_connections 2024-09-13T16:36:55.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.753+0000 7ff01759e700 1 -- 172.21.15.119:0/2693606518 wait complete. 2024-09-13T16:36:55.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-13T16:25:11.481827+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2024-09-13T16:35:53.564659+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2024-09-13T16:35:54.247608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '126', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-13T16:36:55.772 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-13T16:36:55.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88ca09e700 1 Processor -- start 2024-09-13T16:36:55.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88ca09e700 1 -- start start 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88ca09e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f88c40add60 0x7f88c40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88ca09e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c415dbe0 0x7f88c40cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88ca09e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c40cd4d0 con 0x7f88c40b3260 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88ca09e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c4071be0 con 0x7f88c40add60 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88ca09e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c4159af0 con 0x7f88c415dbe0 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c989d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f88c40b3260 0x7f88c406fb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49654/0 (socket says 172.21.15.119:49654) 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c989d700 1 -- 172.21.15.119:0/4091839761 learned_addr learned my addr 172.21.15.119:0/4091839761 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c909c700 1 --2- 172.21.15.119:0/4091839761 >> v2:172.21.15.119:3300/0 conn(0x7f88c40add60 0x7f88c40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c889b700 1 --2- 172.21.15.119:0/4091839761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c415dbe0 0x7f88c40cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88b7fff700 1 -- 172.21.15.119:0/4091839761 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598215524 0 0) 0x7f88c40cd4d0 con 0x7f88c40b3260 2024-09-13T16:36:55.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88b7fff700 1 -- 172.21.15.119:0/4091839761 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b00035d0 con 0x7f88c40b3260 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c909c700 1 -- 172.21.15.119:0/4091839761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c415dbe0 msgr2=0x7f88c40cd960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88b7fff700 1 -- 172.21.15.119:0/4091839761 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3309082165 0 0) 0x7f88b00035d0 con 0x7f88c40b3260 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c909c700 1 --2- 172.21.15.119:0/4091839761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c415dbe0 0x7f88c40cd960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c909c700 1 -- 172.21.15.119:0/4091839761 >> v1:172.21.15.119:6789/0 conn(0x7f88c40b3260 legacy=0x7f88c406fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c909c700 1 -- 172.21.15.119:0/4091839761 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c415e130 con 0x7f88c40add60 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88c909c700 1 --2- 172.21.15.119:0/4091839761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c40add60 0x7f88c40ce0f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009850 tx=0x7f88b8009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d5211bd98ced1a8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.889+0000 7f88b7fff700 1 -- 172.21.15.119:0/4091839761 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc001670 con 0x7f88c40add60 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88b7fff700 1 -- 172.21.15.119:0/4091839761 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8004030 con 0x7f88c40add60 2024-09-13T16:36:55.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88b7fff700 1 -- 172.21.15.119:0/4091839761 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b80043a0 con 0x7f88c40add60 2024-09-13T16:36:55.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- 172.21.15.119:0/4091839761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c40add60 msgr2=0x7f88c40ce0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4091839761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c40add60 0x7f88c40ce0f0 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009850 tx=0x7f88b8009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- 172.21.15.119:0/4091839761 shutdown_connections 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4091839761 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c415dbe0 0x7f88c40cd960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4091839761 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c40add60 0x7f88c40ce0f0 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- 172.21.15.119:0/4091839761 >> 172.21.15.119:0/4091839761 conn(0x7f88c40b0980 msgr2=0x7f88c40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- 172.21.15.119:0/4091839761 shutdown_connections 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- 172.21.15.119:0/4091839761 wait complete. 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 Processor -- start 2024-09-13T16:36:55.898 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- start start 2024-09-13T16:36:55.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c40add60 0x7f88c41503d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.900 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c40b3260 0x7f88c415a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c415dbe0 0x7f88c40a84b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c4130de0 con 0x7f88c415dbe0 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c41310c0 con 0x7f88c40b3260 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88ca09e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c41313a0 con 0x7f88c40add60 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88c989d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c415dbe0 0x7f88c40a84b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88c989d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c415dbe0 0x7f88c40a84b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57036/0 (socket says 172.21.15.119:57036) 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.893+0000 7f88c989d700 1 -- 172.21.15.119:0/4205699557 learned_addr learned my addr 172.21.15.119:0/4205699557 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c909c700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c40add60 0x7f88c41503d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c889b700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c40b3260 0x7f88c415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c989d700 1 -- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c40add60 msgr2=0x7f88c41503d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.901 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c989d700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c40add60 0x7f88c41503d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c989d700 1 -- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c40b3260 msgr2=0x7f88c415a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c989d700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c40b3260 0x7f88c415a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c989d700 1 -- 172.21.15.119:0/4205699557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b80094b0 con 0x7f88c415dbe0 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88c989d700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c415dbe0 0x7f88c40a84b0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f88bc00c520 tx=0x7f88bc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88b5ffb700 1 -- 172.21.15.119:0/4205699557 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc004ca0 con 0x7f88c415dbe0 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c4131680 con 0x7f88c415dbe0 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88b5ffb700 1 -- 172.21.15.119:0/4205699557 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88bc004e40 con 0x7f88c415dbe0 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c4150910 con 0x7f88c415dbe0 2024-09-13T16:36:55.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88b5ffb700 1 -- 172.21.15.119:0/4205699557 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88bc0072d0 con 0x7f88c415dbe0 2024-09-13T16:36:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c4131ba0 con 0x7f88c415dbe0 2024-09-13T16:36:55.903 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88b5ffb700 1 -- 172.21.15.119:0/4205699557 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f88bc007430 con 0x7f88c415dbe0 2024-09-13T16:36:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88b5ffb700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f88a8066730 0x7f88a8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.897+0000 7f88b5ffb700 1 -- 172.21.15.119:0/4205699557 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f88bc084060 con 0x7f88c415dbe0 2024-09-13T16:36:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.901+0000 7f88c909c700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f88a8066730 0x7f88a8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:55.904 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.901+0000 7f88c909c700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f88a8066730 0x7f88a8068bf0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f88b80090b0 tx=0x7f88b8008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:55.907 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:55.905+0000 7f88b5ffb700 1 -- 172.21.15.119:0/4205699557 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f88c4131ba0 con 0x7f88c415dbe0 2024-09-13T16:36:56.074 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.069+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f88c4131ba0 con 0x7f88c415dbe0 2024-09-13T16:36:56.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88b5ffb700 1 -- 172.21.15.119:0/4205699557 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v128) v1 ==== 93+0+10 (secure 0 0 0) 0x7f88c4131ba0 con 0x7f88c415dbe0 2024-09-13T16:36:56.075 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 1 2024-09-13T16:36:56.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f88a8066730 msgr2=0x7f88a8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f88a8066730 0x7f88a8068bf0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f88b80090b0 tx=0x7f88b8008030 comp rx=0 tx=0).stop 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c415dbe0 msgr2=0x7f88c40a84b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c415dbe0 0x7f88c40a84b0 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f88bc00c520 tx=0x7f88bc00c8e0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 shutdown_connections 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f88a8066730 0x7f88a8068bf0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f88c415dbe0 0x7f88c40a84b0 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f88c40b3260 0x7f88c415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 --2- 172.21.15.119:0/4205699557 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f88c40add60 0x7f88c41503d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 >> 172.21.15.119:0/4205699557 conn(0x7f88c40b0980 msgr2=0x7f88c4151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 shutdown_connections 2024-09-13T16:36:56.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.073+0000 7f88ca09e700 1 -- 172.21.15.119:0/4205699557 wait complete. 2024-09-13T16:36:56.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-09-13T16:36:56.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 Processor -- start 2024-09-13T16:36:56.203 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- start start 2024-09-13T16:36:56.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da406b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5da406b8a0 0x7f5da40cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5da40ccd70 con 0x7f5da415cad0 2024-09-13T16:36:56.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5da40ac280 con 0x7f5da406b8a0 2024-09-13T16:36:56.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da406bd30 con 0x7f5da40abe60 2024-09-13T16:36:56.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da9eb5700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5da406b8a0 0x7f5da40cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da8eb3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da406b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da96b4700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f5da415cad0 0x7f5da4071450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40470/0 (socket says 172.21.15.119:40470) 2024-09-13T16:36:56.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da96b4700 1 -- 172.21.15.119:0/1634051979 learned_addr learned my addr 172.21.15.119:0/1634051979 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:56.207 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da9eb5700 1 -- 172.21.15.119:0/1634051979 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 msgr2=0x7f5da406b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da9eb5700 1 --2- 172.21.15.119:0/1634051979 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da406b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da9eb5700 1 -- 172.21.15.119:0/1634051979 >> v1:172.21.15.145:6789/0 conn(0x7f5da415cad0 legacy=0x7f5da4071450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da9eb5700 1 -- 172.21.15.119:0/1634051979 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da4071ea0 con 0x7f5da406b8a0 2024-09-13T16:36:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5d9bfff700 1 -- 172.21.15.119:0/1634051979 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021910326 0 0) 0x7f5da40ccd70 con 0x7f5da415cad0 2024-09-13T16:36:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5da9eb5700 1 --2- 172.21.15.119:0/1634051979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da40cd4e0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009850 tx=0x7f5da0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac13b625d415eba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5d9bfff700 1 -- 172.21.15.119:0/1634051979 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da0011830 con 0x7f5da406b8a0 2024-09-13T16:36:56.208 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5d9bfff700 1 -- 172.21.15.119:0/1634051979 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da0011990 con 0x7f5da406b8a0 2024-09-13T16:36:56.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5d9bfff700 1 -- 172.21.15.119:0/1634051979 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da0011d40 con 0x7f5da406b8a0 2024-09-13T16:36:56.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- 172.21.15.119:0/1634051979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 msgr2=0x7f5da40cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/1634051979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da40cd4e0 secure :-1 s=READY pgs=1995 cs=0 l=1 rev1=1 crypto rx=0x7f5da0009850 tx=0x7f5da0009b60 comp rx=0 tx=0).stop 2024-09-13T16:36:56.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- 172.21.15.119:0/1634051979 shutdown_connections 2024-09-13T16:36:56.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/1634051979 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da40cd4e0 unknown :-1 s=CLOSED pgs=1995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/1634051979 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da406b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.209 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- 172.21.15.119:0/1634051979 >> 172.21.15.119:0/1634051979 conn(0x7f5da40cb8f0 msgr2=0x7f5da40cbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:56.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- 172.21.15.119:0/1634051979 shutdown_connections 2024-09-13T16:36:56.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.201+0000 7f5daa6b6700 1 -- 172.21.15.119:0/1634051979 wait complete. 2024-09-13T16:36:56.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 Processor -- start 2024-09-13T16:36:56.210 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 -- start start 2024-09-13T16:36:56.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da41f2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da40caa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5da415cad0 0x7f5da40c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41528c0 con 0x7f5da406b8a0 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da40c8be0 con 0x7f5da415cad0 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5da41f23a0 con 0x7f5da40abe60 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da96b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da41f2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da8eb3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da40caa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5da415cad0 0x7f5da40c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da96b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da41f2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57048/0 (socket says 172.21.15.119:57048) 2024-09-13T16:36:56.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5da415cad0 0x7f5da40c8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:43994/0 (socket says 172.21.15.119:43994) 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 -- 172.21.15.119:0/3740158102 learned_addr learned my addr 172.21.15.119:0/3740158102 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da8eb3700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da40caa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:50144/0 (socket says 172.21.15.119:50144) 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 -- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 msgr2=0x7f5da40caa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da40caa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 -- 172.21.15.119:0/3740158102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 msgr2=0x7f5da41f2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da41f2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 -- 172.21.15.119:0/3740158102 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da00094b0 con 0x7f5da415cad0 2024-09-13T16:36:56.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5da9eb5700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5da415cad0 0x7f5da40c8200 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5da00090b0 tx=0x7f5da0004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5d99ffb700 1 -- 172.21.15.119:0/3740158102 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da001a990 con 0x7f5da415cad0 2024-09-13T16:36:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5d99ffb700 1 -- 172.21.15.119:0/3740158102 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5da001f070 con 0x7f5da415cad0 2024-09-13T16:36:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da41f2680 con 0x7f5da415cad0 2024-09-13T16:36:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5d99ffb700 1 -- 172.21.15.119:0/3740158102 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5da001ad30 con 0x7f5da415cad0 2024-09-13T16:36:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.205+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da40c8ec0 con 0x7f5da415cad0 2024-09-13T16:36:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.209+0000 7f5d99ffb700 1 -- 172.21.15.119:0/3740158102 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5da001b430 con 0x7f5da415cad0 2024-09-13T16:36:56.215 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.209+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da41f2ba0 con 0x7f5da415cad0 2024-09-13T16:36:56.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.209+0000 7f5d99ffb700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d88066730 0x7f5d88068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.209+0000 7f5d99ffb700 1 -- 172.21.15.119:0/3740158102 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f5da0085330 con 0x7f5da415cad0 2024-09-13T16:36:56.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.209+0000 7f5da96b4700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d88066730 0x7f5d88068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.209+0000 7f5da96b4700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d88066730 0x7f5d88068bf0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5d94000f60 tx=0x7f5d94007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.221 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.217+0000 7f5d99ffb700 1 -- 172.21.15.119:0/3740158102 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f5da0052520 con 0x7f5da415cad0 2024-09-13T16:36:56.390 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.385+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5da40cc770 con 0x7f5da415cad0 2024-09-13T16:36:56.391 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5d99ffb700 1 -- 172.21.15.119:0/3740158102 <== mon.1 v2:172.21.15.145:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v128) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5da0055b40 con 0x7f5da415cad0 2024-09-13T16:36:56.391 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 32 2024-09-13T16:36:56.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d88066730 msgr2=0x7f5d88068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d88066730 0x7f5d88068bf0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5d94000f60 tx=0x7f5d94007e90 comp rx=0 tx=0).stop 2024-09-13T16:36:56.393 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5da415cad0 msgr2=0x7f5da40c8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5da415cad0 0x7f5da40c8200 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f5da00090b0 tx=0x7f5da0004770 comp rx=0 tx=0).stop 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 shutdown_connections 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5d88066730 0x7f5d88068bf0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f5da415cad0 0x7f5da40c8200 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f5da40abe60 0x7f5da40caa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 --2- 172.21.15.119:0/3740158102 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5da406b8a0 0x7f5da41f2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 >> 172.21.15.119:0/3740158102 conn(0x7f5da40cb8f0 msgr2=0x7f5da40aead0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 shutdown_connections 2024-09-13T16:36:56.394 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.389+0000 7f5daa6b6700 1 -- 172.21.15.119:0/3740158102 wait complete. 2024-09-13T16:36:56.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-09-13T16:36:56.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.521+0000 7f768359e700 1 Processor -- start 2024-09-13T16:36:56.526 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.521+0000 7f768359e700 1 -- start start 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f768415cc00 0x7f76840d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840d0c20 0x7f76840d10d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f76840ce8c0 0x7f76840d1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76840cdb00 con 0x7f76840ce8c0 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76840cdde0 con 0x7f768415cc00 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76840ce0c0 con 0x7f76840d0c20 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7682d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f76840ce8c0 0x7f76840d1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7682d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f76840ce8c0 0x7f76840d1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57052/0 (socket says 172.21.15.119:57052) 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7682d9d700 1 -- 172.21.15.119:0/4126307374 learned_addr learned my addr 172.21.15.119:0/4126307374 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:56.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7681d9b700 1 --2- 172.21.15.119:0/4126307374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840d0c20 0x7f76840d10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.529 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768259c700 1 --2- 172.21.15.119:0/4126307374 >> v2:172.21.15.145:3300/0 conn(0x7f768415cc00 0x7f76840d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7681d9b700 1 -- 172.21.15.119:0/4126307374 >> v2:172.21.15.145:3300/0 conn(0x7f768415cc00 msgr2=0x7f76840d06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:56.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7681d9b700 1 --2- 172.21.15.119:0/4126307374 >> v2:172.21.15.145:3300/0 conn(0x7f768415cc00 0x7f76840d06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7681d9b700 1 -- 172.21.15.119:0/4126307374 >> v2:172.21.15.119:3300/0 conn(0x7f76840ce8c0 msgr2=0x7f76840d1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:56.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7681d9b700 1 --2- 172.21.15.119:0/4126307374 >> v2:172.21.15.119:3300/0 conn(0x7f76840ce8c0 0x7f76840d1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7681d9b700 1 -- 172.21.15.119:0/4126307374 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76840ce3a0 con 0x7f76840d0c20 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f7681d9b700 1 --2- 172.21.15.119:0/4126307374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840d0c20 0x7f76840d10d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f76840b5610 tx=0x7f7678009b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ad18ce7dbaec668 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768159a700 1 -- 172.21.15.119:0/4126307374 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678011830 con 0x7f76840d0c20 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768159a700 1 -- 172.21.15.119:0/4126307374 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7678011990 con 0x7f76840d0c20 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768159a700 1 -- 172.21.15.119:0/4126307374 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7678011d20 con 0x7f76840d0c20 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- 172.21.15.119:0/4126307374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840d0c20 msgr2=0x7f76840d10d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 --2- 172.21.15.119:0/4126307374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840d0c20 0x7f76840d10d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f76840b5610 tx=0x7f7678009b30 comp rx=0 tx=0).stop 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- 172.21.15.119:0/4126307374 shutdown_connections 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 --2- 172.21.15.119:0/4126307374 >> v2:172.21.15.119:3300/0 conn(0x7f76840ce8c0 0x7f76840d1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 --2- 172.21.15.119:0/4126307374 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840d0c20 0x7f76840d10d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 --2- 172.21.15.119:0/4126307374 >> v2:172.21.15.145:3300/0 conn(0x7f768415cc00 0x7f76840d06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- 172.21.15.119:0/4126307374 >> 172.21.15.119:0/4126307374 conn(0x7f768408c080 msgr2=0x7f76840d2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:56.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- 172.21.15.119:0/4126307374 shutdown_connections 2024-09-13T16:36:56.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.525+0000 7f768359e700 1 -- 172.21.15.119:0/4126307374 wait complete. 2024-09-13T16:36:56.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 Processor -- start 2024-09-13T16:36:56.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 -- start start 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840ce8c0 0x7f76841f1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76840d0c20 0x7f76840c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f768415cc00 0x7f7684155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76840c88e0 con 0x7f768415cc00 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76841f2430 con 0x7f76840d0c20 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76840c7cd0 con 0x7f76840ce8c0 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f768415cc00 0x7f7684155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f768415cc00 0x7f7684155980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57064/0 (socket says 172.21.15.119:57064) 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 -- 172.21.15.119:0/1684894592 learned_addr learned my addr 172.21.15.119:0/1684894592 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:56.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f768259c700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840ce8c0 0x7f76841f1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7681d9b700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76840d0c20 0x7f76840c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 -- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840ce8c0 msgr2=0x7f76841f1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840ce8c0 0x7f76841f1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 -- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76840d0c20 msgr2=0x7f76840c7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76840d0c20 0x7f76840c7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 -- 172.21.15.119:0/1684894592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76780094b0 con 0x7f768415cc00 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.529+0000 7f7682d9d700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f768415cc00 0x7f7684155980 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f767c00ef00 tx=0x7f767c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f7672ffd700 1 -- 172.21.15.119:0/1684894592 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c010040 con 0x7f768415cc00 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f7672ffd700 1 -- 172.21.15.119:0/1684894592 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767c003c40 con 0x7f768415cc00 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f7672ffd700 1 -- 172.21.15.119:0/1684894592 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c009e60 con 0x7f768415cc00 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76840c7fb0 con 0x7f768415cc00 2024-09-13T16:36:56.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76840c8570 con 0x7f768415cc00 2024-09-13T16:36:56.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76840c8a90 con 0x7f768415cc00 2024-09-13T16:36:56.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f7672ffd700 1 -- 172.21.15.119:0/1684894592 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f767c00b870 con 0x7f768415cc00 2024-09-13T16:36:56.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f7672ffd700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7664066760 0x7f7664068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f7672ffd700 1 -- 172.21.15.119:0/1684894592 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f767c0840e0 con 0x7f768415cc00 2024-09-13T16:36:56.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.533+0000 7f768259c700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7664066760 0x7f7664068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.537+0000 7f768259c700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7664066760 0x7f7664068c20 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f7674009b50 tx=0x7f7674009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.541+0000 7f7672ffd700 1 -- 172.21.15.119:0/1684894592 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f76840c8a90 con 0x7f768415cc00 2024-09-13T16:36:56.715 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f76840c8a90 con 0x7f768415cc00 2024-09-13T16:36:56.716 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f7672ffd700 1 -- 172.21.15.119:0/1684894592 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v128) v1 ==== 100+0+11 (secure 0 0 0) 0x7f76840c8a90 con 0x7f768415cc00 2024-09-13T16:36:56.716 INFO:teuthology.orchestra.run.smithi119.stdout:pg_num: 64 2024-09-13T16:36:56.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7664066760 msgr2=0x7f7664068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7664066760 0x7f7664068c20 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f7674009b50 tx=0x7f7674009240 comp rx=0 tx=0).stop 2024-09-13T16:36:56.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f768415cc00 msgr2=0x7f7684155980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f768415cc00 0x7f7684155980 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7f767c00ef00 tx=0x7f767c00b390 comp rx=0 tx=0).stop 2024-09-13T16:36:56.720 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 shutdown_connections 2024-09-13T16:36:56.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f7664066760 0x7f7664068c20 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f768415cc00 0x7f7684155980 unknown :-1 s=CLOSED pgs=1996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f76840d0c20 0x7f76840c7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 --2- 172.21.15.119:0/1684894592 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f76840ce8c0 0x7f76841f1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 >> 172.21.15.119:0/1684894592 conn(0x7f768408c080 msgr2=0x7f76840b4600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:56.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 shutdown_connections 2024-09-13T16:36:56.722 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.713+0000 7f768359e700 1 -- 172.21.15.119:0/1684894592 wait complete. 2024-09-13T16:36:56.731 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:56.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 Processor -- start 2024-09-13T16:36:56.844 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 -- start start 2024-09-13T16:36:56.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f83f006e6a0 0x7f83f006ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006be70 0x7f83f00b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f83f00b4160 0x7f83f00b4610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f006cff0 con 0x7f83f00b4160 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83f006d480 con 0x7f83f006e6a0 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f52fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f006efc0 con 0x7f83f006be70 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f83f00b4160 0x7f83f00b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f83f00b4160 0x7f83f00b4610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57072/0 (socket says 172.21.15.119:57072) 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 -- 172.21.15.119:0/2338861585 learned_addr learned my addr 172.21.15.119:0/2338861585 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83effff700 1 --2- 172.21.15.119:0/2338861585 >> v2:172.21.15.145:3300/0 conn(0x7f83f006e6a0 0x7f83f006ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83ef7fe700 1 --2- 172.21.15.119:0/2338861585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006be70 0x7f83f00b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 -- 172.21.15.119:0/2338861585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006be70 msgr2=0x7f83f00b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 --2- 172.21.15.119:0/2338861585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006be70 0x7f83f00b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 -- 172.21.15.119:0/2338861585 >> v2:172.21.15.145:3300/0 conn(0x7f83f006e6a0 msgr2=0x7f83f006ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 --2- 172.21.15.119:0/2338861585 >> v2:172.21.15.145:3300/0 conn(0x7f83f006e6a0 0x7f83f006ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 -- 172.21.15.119:0/2338861585 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83f006c3e0 con 0x7f83f00b4160 2024-09-13T16:36:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83effff700 1 --2- 172.21.15.119:0/2338861585 >> v2:172.21.15.145:3300/0 conn(0x7f83f006e6a0 0x7f83f006ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83f4afd700 1 --2- 172.21.15.119:0/2338861585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f00b4610 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009e50 tx=0x7f83e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b3152bed99b6ed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83eeffd700 1 -- 172.21.15.119:0/2338861585 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e800e070 con 0x7f83f00b4160 2024-09-13T16:36:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83eeffd700 1 -- 172.21.15.119:0/2338861585 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e800b980 con 0x7f83f00b4160 2024-09-13T16:36:56.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.841+0000 7f83eeffd700 1 -- 172.21.15.119:0/2338861585 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e800bd30 con 0x7f83f00b4160 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 -- 172.21.15.119:0/2338861585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 msgr2=0x7f83f00b4610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2338861585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f00b4610 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7f83e8009e50 tx=0x7f83e800b330 comp rx=0 tx=0).stop 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 -- 172.21.15.119:0/2338861585 shutdown_connections 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2338861585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f00b4610 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2338861585 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006be70 0x7f83f00b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2338861585 >> v2:172.21.15.145:3300/0 conn(0x7f83f006e6a0 0x7f83f006ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 -- 172.21.15.119:0/2338861585 >> 172.21.15.119:0/2338861585 conn(0x7f83f00b5d20 msgr2=0x7f83f00b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:56.849 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 -- 172.21.15.119:0/2338861585 shutdown_connections 2024-09-13T16:36:56.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 -- 172.21.15.119:0/2338861585 wait complete. 2024-09-13T16:36:56.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 Processor -- start 2024-09-13T16:36:56.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.845+0000 7f83f52fe700 1 -- start start 2024-09-13T16:36:56.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83f006be70 0x7f83f009daa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006e6a0 0x7f83f0153760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f01f2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f01f2460 con 0x7f83f00b4160 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83f009e0d0 con 0x7f83f006be70 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83f009e3b0 con 0x7f83f006e6a0 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f01f2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f01f2bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57084/0 (socket says 172.21.15.119:57084) 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 -- 172.21.15.119:0/2414224692 learned_addr learned my addr 172.21.15.119:0/2414224692 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83ef7fe700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006e6a0 0x7f83f0153760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83effff700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83f006be70 0x7f83f009daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 -- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006e6a0 msgr2=0x7f83f0153760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006e6a0 0x7f83f0153760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 -- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83f006be70 msgr2=0x7f83f009daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:56.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83f006be70 0x7f83f009daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:56.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 -- 172.21.15.119:0/2414224692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e8009b30 con 0x7f83f00b4160 2024-09-13T16:36:56.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f4afd700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f01f2bd0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f83f00d18f0 tx=0x7f83e8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83ecff9700 1 -- 172.21.15.119:0/2414224692 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8004450 con 0x7f83f00b4160 2024-09-13T16:36:56.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83ecff9700 1 -- 172.21.15.119:0/2414224692 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83e80045b0 con 0x7f83f00b4160 2024-09-13T16:36:56.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83f009e690 con 0x7f83f00b4160 2024-09-13T16:36:56.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83ecff9700 1 -- 172.21.15.119:0/2414224692 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83e8004960 con 0x7f83f00b4160 2024-09-13T16:36:56.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.849+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83f0157260 con 0x7f83f00b4160 2024-09-13T16:36:56.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.853+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83f0074560 con 0x7f83f00b4160 2024-09-13T16:36:56.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.853+0000 7f83ecff9700 1 -- 172.21.15.119:0/2414224692 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f83e8025460 con 0x7f83f00b4160 2024-09-13T16:36:56.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.853+0000 7f83ecff9700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f83dc066730 0x7f83dc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:56.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.853+0000 7f83ecff9700 1 -- 172.21.15.119:0/2414224692 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f83e80861c0 con 0x7f83f00b4160 2024-09-13T16:36:56.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.853+0000 7f83effff700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f83dc066730 0x7f83dc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:56.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.853+0000 7f83effff700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f83dc066730 0x7f83dc068bf0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f83e000c960 tx=0x7f83e0008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:56.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:56.861+0000 7f83ecff9700 1 -- 172.21.15.119:0/2414224692 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f83e8053350 con 0x7f83f00b4160 2024-09-13T16:36:57.073 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.069+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83f01f28f0 con 0x7f83f00b4160 2024-09-13T16:36:57.075 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83ecff9700 1 -- 172.21.15.119:0/2414224692 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f83f01f28f0 con 0x7f83f00b4160 2024-09-13T16:36:57.075 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:57.075 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:47.145416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9646":{"gid":9646,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6835/2905269684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2905269684},{"type":"v1","addr":"172.21.15.145:6835","nonce":2905269684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:57.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f83dc066730 msgr2=0x7f83dc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f83dc066730 0x7f83dc068bf0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f83e000c960 tx=0x7f83e0008040 comp rx=0 tx=0).stop 2024-09-13T16:36:57.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 msgr2=0x7f83f01f2bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f01f2bd0 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7f83f00d18f0 tx=0x7f83e8004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 shutdown_connections 2024-09-13T16:36:57.077 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f83dc066730 0x7f83dc068bf0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f83f00b4160 0x7f83f01f2bd0 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f83f006e6a0 0x7f83f0153760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 --2- 172.21.15.119:0/2414224692 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f83f006be70 0x7f83f009daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 >> 172.21.15.119:0/2414224692 conn(0x7f83f00b5d20 msgr2=0x7f83f015bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:57.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 shutdown_connections 2024-09-13T16:36:57.078 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.073+0000 7f83f52fe700 1 -- 172.21.15.119:0/2414224692 wait complete. 2024-09-13T16:36:57.079 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 105 2024-09-13T16:36:57.090 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:57.204 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f640cd06700 1 Processor -- start 2024-09-13T16:36:57.205 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f640cd06700 1 -- start start 2024-09-13T16:36:57.211 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f640cd06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6408159930 0x7f6408070360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f640cd06700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f640806e610 0x7f640806d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f640cd06700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64080cc080 con 0x7f640806ea70 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f640cd06700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f640806cc60 con 0x7f640806e610 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f640cd06700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f640806e290 con 0x7f6408159930 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.201+0000 7f6406ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f640806e610 0x7f640806d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f64077fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6408159930 0x7f6408070360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6406ffd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f640806e610 0x7f640806d640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57090/0 (socket says 172.21.15.119:57090) 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6407fff700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7f640806ea70 0x7f640806db80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40474/0 (socket says 172.21.15.119:40474) 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6406ffd700 1 -- 172.21.15.119:0/1753548424 learned_addr learned my addr 172.21.15.119:0/1753548424 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:57.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f64077fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6408159930 0x7f6408070360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:50180/0 (socket says 172.21.15.119:50180) 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6406ffd700 1 -- 172.21.15.119:0/1753548424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6408159930 msgr2=0x7f6408070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6406ffd700 1 --2- 172.21.15.119:0/1753548424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6408159930 0x7f6408070360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6406ffd700 1 -- 172.21.15.119:0/1753548424 >> v1:172.21.15.145:6789/0 conn(0x7f640806ea70 legacy=0x7f640806db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6406ffd700 1 -- 172.21.15.119:0/1753548424 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640806d1b0 con 0x7f640806e610 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f6406ffd700 1 --2- 172.21.15.119:0/1753548424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806e610 0x7f640806d640 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009910 tx=0x7f63fc009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6286218f9df3cf49 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f64067fc700 1 -- 172.21.15.119:0/1753548424 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc011830 con 0x7f640806e610 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f64067fc700 1 -- 172.21.15.119:0/1753548424 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc011990 con 0x7f640806e610 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f64067fc700 1 -- 172.21.15.119:0/1753548424 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc011d40 con 0x7f640806e610 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 -- 172.21.15.119:0/1753548424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806e610 msgr2=0x7f640806d640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 --2- 172.21.15.119:0/1753548424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806e610 0x7f640806d640 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7f63fc009910 tx=0x7f63fc009c20 comp rx=0 tx=0).stop 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 -- 172.21.15.119:0/1753548424 shutdown_connections 2024-09-13T16:36:57.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 --2- 172.21.15.119:0/1753548424 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806e610 0x7f640806d640 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 --2- 172.21.15.119:0/1753548424 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6408159930 0x7f6408070360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 -- 172.21.15.119:0/1753548424 >> 172.21.15.119:0/1753548424 conn(0x7f64080ae4b0 msgr2=0x7f64080b0bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:57.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 -- 172.21.15.119:0/1753548424 shutdown_connections 2024-09-13T16:36:57.214 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.205+0000 7f640cd06700 1 -- 172.21.15.119:0/1753548424 wait complete. 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 Processor -- start 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 -- start start 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f640806e610 0x7f64081f0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806ea70 0x7f6408152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6408159930 0x7f64081dedc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6408152d80 con 0x7f640806ea70 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64081f0a60 con 0x7f6408159930 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.209+0000 7f640cd06700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64081f0d40 con 0x7f640806e610 2024-09-13T16:36:57.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f64077fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f640806e610 0x7f64081f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6406ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806ea70 0x7f6408152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f64077fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f640806e610 0x7f64081f0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:50182/0 (socket says 172.21.15.119:50182) 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f64077fe700 1 -- 172.21.15.119:0/2840343967 learned_addr learned my addr 172.21.15.119:0/2840343967 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6407fff700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6408159930 0x7f64081dedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6406ffd700 1 -- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f640806e610 msgr2=0x7f64081f0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6406ffd700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f640806e610 0x7f64081f0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6406ffd700 1 -- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6408159930 msgr2=0x7f64081dedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6406ffd700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6408159930 0x7f64081dedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6406ffd700 1 -- 172.21.15.119:0/2840343967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63fc009570 con 0x7f640806ea70 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6407fff700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6408159930 0x7f64081dedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f6406ffd700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806ea70 0x7f6408152840 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f63fc016000 tx=0x7f63fc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f63effff700 1 -- 172.21.15.119:0/2840343967 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc004590 con 0x7f640806ea70 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f63effff700 1 -- 172.21.15.119:0/2840343967 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63fc01f070 con 0x7f640806ea70 2024-09-13T16:36:57.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64081f1020 con 0x7f640806ea70 2024-09-13T16:36:57.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f63effff700 1 -- 172.21.15.119:0/2840343967 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63fc003900 con 0x7f640806ea70 2024-09-13T16:36:57.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f640809ab40 con 0x7f640806ea70 2024-09-13T16:36:57.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f63effff700 1 -- 172.21.15.119:0/2840343967 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f63fc003a60 con 0x7f640806ea70 2024-09-13T16:36:57.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f640809ad30 con 0x7f640806ea70 2024-09-13T16:36:57.218 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f63effff700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f63f4066730 0x7f63f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f63effff700 1 -- 172.21.15.119:0/2840343967 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f63fc021070 con 0x7f640806ea70 2024-09-13T16:36:57.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.213+0000 7f64077fe700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f63f4066730 0x7f63f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.219 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.217+0000 7f64077fe700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f63f4066730 0x7f63f4068bf0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6408158cb0 tx=0x7f63f8009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.226 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.221+0000 7f63effff700 1 -- 172.21.15.119:0/2840343967 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f63fc052980 con 0x7f640806ea70 2024-09-13T16:36:57.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.397+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f64080cbda0 con 0x7f640806ea70 2024-09-13T16:36:57.403 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f63effff700 1 -- 172.21.15.119:0/2840343967 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17107 (secure 0 0 0) 0x7f63fc055fa0 con 0x7f640806ea70 2024-09-13T16:36:57.404 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:57.404 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":128,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:36:42.101635+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6835/3901706437":"2024-09-14T16:36:21.631788+0000","172.21.15.145:6834/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6834/3901706437":"2024-09-14T16:36:21.631788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:57.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f63f4066730 msgr2=0x7f63f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f63f4066730 0x7f63f4068bf0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f6408158cb0 tx=0x7f63f8009240 comp rx=0 tx=0).stop 2024-09-13T16:36:57.406 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806ea70 msgr2=0x7f6408152840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806ea70 0x7f6408152840 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7f63fc016000 tx=0x7f63fc004770 comp rx=0 tx=0).stop 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 shutdown_connections 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f63f4066730 0x7f63f4068bf0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6408159930 0x7f64081dedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f640806ea70 0x7f6408152840 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 --2- 172.21.15.119:0/2840343967 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f640806e610 0x7f64081f0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 >> 172.21.15.119:0/2840343967 conn(0x7f64080ae4b0 msgr2=0x7f64080b2410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 shutdown_connections 2024-09-13T16:36:57.407 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.401+0000 7f640cd06700 1 -- 172.21.15.119:0/2840343967 wait complete. 2024-09-13T16:36:57.419 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:36:57.420 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:57.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.525+0000 7f4fe8f34700 1 Processor -- start 2024-09-13T16:36:57.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.525+0000 7f4fe8f34700 1 -- start start 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f4fe40b03e0 0x7f4fe40aee20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4fe40b4440 0x7f4fe40b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40b1560 con 0x7f4fe415ab50 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe40b4df0 con 0x7f4fe40b4440 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe40b0950 con 0x7f4fe40b03e0 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4fe40b4440 0x7f4fe40b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4fe40b4440 0x7f4fe40b48b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57104/0 (socket says 172.21.15.119:57104) 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 -- 172.21.15.119:0/3676440342 learned_addr learned my addr 172.21.15.119:0/3676440342 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe2ffd700 1 --2- 172.21.15.119:0/3676440342 >> v2:172.21.15.145:3300/0 conn(0x7f4fe40b03e0 0x7f4fe40aee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 -- 172.21.15.119:0/3676440342 >> v2:172.21.15.145:3300/0 conn(0x7f4fe40b03e0 msgr2=0x7f4fe40aee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 --2- 172.21.15.119:0/3676440342 >> v2:172.21.15.145:3300/0 conn(0x7f4fe40b03e0 0x7f4fe40aee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 -- 172.21.15.119:0/3676440342 >> v1:172.21.15.145:6789/0 conn(0x7f4fe415ab50 legacy=0x7f4fe40b19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 -- 172.21.15.119:0/3676440342 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40b0c30 con 0x7f4fe40b4440 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe3fff700 1 --2- 172.21.15.119:0/3676440342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe40b48b0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009e50 tx=0x7f4fdc00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c36b7ff6825895 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe27fc700 1 -- 172.21.15.119:0/3676440342 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00e070 con 0x7f4fe40b4440 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe27fc700 1 -- 172.21.15.119:0/3676440342 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc00b980 con 0x7f4fe40b4440 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe27fc700 1 -- 172.21.15.119:0/3676440342 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00bd30 con 0x7f4fe40b4440 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3676440342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 msgr2=0x7f4fe40b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3676440342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe40b48b0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009e50 tx=0x7f4fdc00b330 comp rx=0 tx=0).stop 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3676440342 shutdown_connections 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3676440342 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe40b48b0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3676440342 >> v2:172.21.15.145:3300/0 conn(0x7f4fe40b03e0 0x7f4fe40aee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3676440342 >> 172.21.15.119:0/3676440342 conn(0x7f4fe4072370 msgr2=0x7f4fe4099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:57.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3676440342 shutdown_connections 2024-09-13T16:36:57.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.529+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3676440342 wait complete. 2024-09-13T16:36:57.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 Processor -- start 2024-09-13T16:36:57.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 -- start start 2024-09-13T16:36:57.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fe40b03e0 0x7f4fe41f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe409e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fe415ab50 0x7f4fe4146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe409aa00 con 0x7f4fe40b4440 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe409ace0 con 0x7f4fe40b03e0 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe409afc0 con 0x7f4fe415ab50 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe409e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe409e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57110/0 (socket says 172.21.15.119:57110) 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 -- 172.21.15.119:0/3942929461 learned_addr learned my addr 172.21.15.119:0/3942929461 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe3fff700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fe415ab50 0x7f4fe4146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe37fe700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fe40b03e0 0x7f4fe41f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 -- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fe415ab50 msgr2=0x7f4fe4146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fe415ab50 0x7f4fe4146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 -- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fe40b03e0 msgr2=0x7f4fe41f2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fe40b03e0 0x7f4fe41f2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 -- 172.21.15.119:0/3942929461 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc009b30 con 0x7f4fe40b4440 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe2ffd700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe409e080 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400e760 tx=0x7f4fd400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fcbfff700 1 -- 172.21.15.119:0/3942929461 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4013610 con 0x7f4fe40b4440 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fcbfff700 1 -- 172.21.15.119:0/3942929461 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fd4013770 con 0x7f4fe40b4440 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe409b2a0 con 0x7f4fe40b4440 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fcbfff700 1 -- 172.21.15.119:0/3942929461 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fd4013ae0 con 0x7f4fe40b4440 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.533+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe409e5c0 con 0x7f4fe40b4440 2024-09-13T16:36:57.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.537+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe40b48b0 con 0x7f4fe40b4440 2024-09-13T16:36:57.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.537+0000 7f4fcbfff700 1 -- 172.21.15.119:0/3942929461 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4fd4013c40 con 0x7f4fe40b4440 2024-09-13T16:36:57.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.537+0000 7f4fcbfff700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4fd0066730 0x7f4fd0068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.537+0000 7f4fcbfff700 1 -- 172.21.15.119:0/3942929461 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f4fd4083f80 con 0x7f4fe40b4440 2024-09-13T16:36:57.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.537+0000 7f4fe37fe700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4fd0066730 0x7f4fd0068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.541 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.537+0000 7f4fe37fe700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4fd0066730 0x7f4fd0068bf0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8001640 tx=0x7f4fd8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.548 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.545+0000 7f4fcbfff700 1 -- 172.21.15.119:0/3942929461 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4fd40b5db0 con 0x7f4fe40b4440 2024-09-13T16:36:57.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4fe406e530 con 0x7f4fe40b4440 2024-09-13T16:36:57.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fcbfff700 1 -- 172.21.15.119:0/3942929461 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f4fd4051190 con 0x7f4fe40b4440 2024-09-13T16:36:57.760 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:57.760 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:47.145416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9646":{"gid":9646,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6835/2905269684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2905269684},{"type":"v1","addr":"172.21.15.145:6835","nonce":2905269684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4fd0066730 msgr2=0x7f4fd0068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4fd0066730 0x7f4fd0068bf0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8001640 tx=0x7f4fd8009040 comp rx=0 tx=0).stop 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 msgr2=0x7f4fe409e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe409e080 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400e760 tx=0x7f4fd400eb20 comp rx=0 tx=0).stop 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 shutdown_connections 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4fd0066730 0x7f4fd0068bf0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4fe415ab50 0x7f4fe4146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4fe40b4440 0x7f4fe409e080 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 --2- 172.21.15.119:0/3942929461 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4fe40b03e0 0x7f4fe41f2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 >> 172.21.15.119:0/3942929461 conn(0x7f4fe4072370 msgr2=0x7f4fe4158390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:57.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 shutdown_connections 2024-09-13T16:36:57.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.757+0000 7f4fe8f34700 1 -- 172.21.15.119:0/3942929461 wait complete. 2024-09-13T16:36:57.764 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 105 2024-09-13T16:36:57.775 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:36:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fe04b700 1 Processor -- start 2024-09-13T16:36:57.884 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fe04b700 1 -- start start 2024-09-13T16:36:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fe04b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80adb30 0x7f79f80cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fe04b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f79f806bb90 0x7f79f80cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fe04b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79f806cf60 con 0x7f79f80cde70 2024-09-13T16:36:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fe04b700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79f80b3390 con 0x7f79f806bb90 2024-09-13T16:36:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fe04b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f806c350 con 0x7f79f80adb30 2024-09-13T16:36:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd84a700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f79f80cde70 0x7f79f815e2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49656/0 (socket says 172.21.15.119:49656) 2024-09-13T16:36:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd84a700 1 -- 172.21.15.119:0/2031299545 learned_addr learned my addr 172.21.15.119:0/2031299545 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd049700 1 --2- 172.21.15.119:0/2031299545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80adb30 0x7f79f80cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.886 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fc848700 1 --2- 172.21.15.119:0/2031299545 >> v2:172.21.15.145:3300/0 conn(0x7f79f806bb90 0x7f79f80cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79f7fff700 1 -- 172.21.15.119:0/2031299545 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383331581 0 0) 0x7f79f806cf60 con 0x7f79f80cde70 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd049700 1 -- 172.21.15.119:0/2031299545 >> v2:172.21.15.145:3300/0 conn(0x7f79f806bb90 msgr2=0x7f79f80cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd049700 1 --2- 172.21.15.119:0/2031299545 >> v2:172.21.15.145:3300/0 conn(0x7f79f806bb90 0x7f79f80cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd049700 1 -- 172.21.15.119:0/2031299545 >> v1:172.21.15.119:6789/0 conn(0x7f79f80cde70 legacy=0x7f79f815e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd049700 1 -- 172.21.15.119:0/2031299545 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79f806c630 con 0x7f79f80adb30 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fc848700 1 --2- 172.21.15.119:0/2031299545 >> v2:172.21.15.145:3300/0 conn(0x7f79f806bb90 0x7f79f80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.881+0000 7f79fd049700 1 --2- 172.21.15.119:0/2031299545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80adb30 0x7f79f80cc060 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009d90 tx=0x7f79e800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ae4eea3cc4cffc6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79f7fff700 1 -- 172.21.15.119:0/2031299545 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e800e070 con 0x7f79f80adb30 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79f7fff700 1 -- 172.21.15.119:0/2031299545 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e800bbd0 con 0x7f79f80adb30 2024-09-13T16:36:57.888 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79f7fff700 1 -- 172.21.15.119:0/2031299545 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79e8012430 con 0x7f79f80adb30 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 -- 172.21.15.119:0/2031299545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80adb30 msgr2=0x7f79f80cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 --2- 172.21.15.119:0/2031299545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80adb30 0x7f79f80cc060 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f79e8009d90 tx=0x7f79e800b330 comp rx=0 tx=0).stop 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 -- 172.21.15.119:0/2031299545 shutdown_connections 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 --2- 172.21.15.119:0/2031299545 >> v2:172.21.15.145:3300/0 conn(0x7f79f806bb90 0x7f79f80cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 --2- 172.21.15.119:0/2031299545 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80adb30 0x7f79f80cc060 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 -- 172.21.15.119:0/2031299545 >> 172.21.15.119:0/2031299545 conn(0x7f79f806a350 msgr2=0x7f79f806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 -- 172.21.15.119:0/2031299545 shutdown_connections 2024-09-13T16:36:57.889 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 -- 172.21.15.119:0/2031299545 wait complete. 2024-09-13T16:36:57.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 Processor -- start 2024-09-13T16:36:57.890 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.885+0000 7f79fe04b700 1 -- start start 2024-09-13T16:36:57.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79f806bb90 0x7f79f80a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79f80adb30 0x7f79f813ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80cde70 0x7f79f80ac440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80c3960 con 0x7f79f80adb30 2024-09-13T16:36:57.892 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80a9bb0 con 0x7f79f806bb90 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79f80a9e90 con 0x7f79f80cde70 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79f80adb30 0x7f79f813ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79f80adb30 0x7f79f813ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57120/0 (socket says 172.21.15.119:57120) 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 -- 172.21.15.119:0/1424079630 learned_addr learned my addr 172.21.15.119:0/1424079630 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fd84a700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80cde70 0x7f79f80ac440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fd049700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79f806bb90 0x7f79f80a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 -- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80cde70 msgr2=0x7f79f80ac440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80cde70 0x7f79f80ac440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 -- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79f806bb90 msgr2=0x7f79f80a95d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79f806bb90 0x7f79f80a95d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:57.893 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 -- 172.21.15.119:0/1424079630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e8009a70 con 0x7f79f80adb30 2024-09-13T16:36:57.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fc848700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79f80adb30 0x7f79f813ebe0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00e880 tx=0x7f79ec00eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79f5ffb700 1 -- 172.21.15.119:0/1424079630 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec013610 con 0x7f79f80adb30 2024-09-13T16:36:57.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79f5ffb700 1 -- 172.21.15.119:0/1424079630 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ec013770 con 0x7f79f80adb30 2024-09-13T16:36:57.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79f80aa170 con 0x7f79f80adb30 2024-09-13T16:36:57.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79f5ffb700 1 -- 172.21.15.119:0/1424079630 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79ec013b00 con 0x7f79f80adb30 2024-09-13T16:36:57.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79f80aa730 con 0x7f79f80adb30 2024-09-13T16:36:57.894 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.889+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79f806d3d0 con 0x7f79f80adb30 2024-09-13T16:36:57.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.893+0000 7f79f5ffb700 1 -- 172.21.15.119:0/1424079630 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f79ec013c60 con 0x7f79f80adb30 2024-09-13T16:36:57.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.893+0000 7f79f5ffb700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f79e4066730 0x7f79e4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:57.895 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.893+0000 7f79f5ffb700 1 -- 172.21.15.119:0/1424079630 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f79ec083ed0 con 0x7f79f80adb30 2024-09-13T16:36:57.896 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.893+0000 7f79fd049700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f79e4066730 0x7f79e4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:57.897 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.893+0000 7f79fd049700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f79e4066730 0x7f79e4068bf0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f79e8014040 tx=0x7f79e8011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:57.902 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:57.897+0000 7f79f5ffb700 1 -- 172.21.15.119:0/1424079630 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f79f806d3d0 con 0x7f79f80adb30 2024-09-13T16:36:58.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79f806d3d0 con 0x7f79f80adb30 2024-09-13T16:36:58.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79f5ffb700 1 -- 172.21.15.119:0/1424079630 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 105 v105) v1 ==== 95+0+4744 (secure 0 0 0) 0x7f79f806d3d0 con 0x7f79f80adb30 2024-09-13T16:36:58.108 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:58.108 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":105,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":105,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-09-13T16:35:55.244835+0000","modified":"2024-09-13T16:36:47.145416+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":128,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":9646},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9646":{"gid":9646,"name":"b","rank":0,"incarnation":102,"state":"up:active","state_seq":4,"addr":"172.21.15.145:6835/2905269684","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2905269684},{"type":"v1","addr":"172.21.15.145:6835","nonce":2905269684}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2024-09-13T16:36:58.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f79e4066730 msgr2=0x7f79e4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f79e4066730 0x7f79e4068bf0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f79e8014040 tx=0x7f79e8011000 comp rx=0 tx=0).stop 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79f80adb30 msgr2=0x7f79f813ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79f80adb30 0x7f79f813ebe0 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7f79ec00e880 tx=0x7f79ec00eb90 comp rx=0 tx=0).stop 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 shutdown_connections 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f79e4066730 0x7f79e4068bf0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f79f80cde70 0x7f79f80ac440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f79f80adb30 0x7f79f813ebe0 unknown :-1 s=CLOSED pgs=2004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 --2- 172.21.15.119:0/1424079630 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f79f806bb90 0x7f79f80a95d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 >> 172.21.15.119:0/1424079630 conn(0x7f79f806a350 msgr2=0x7f79f815c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 shutdown_connections 2024-09-13T16:36:58.111 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.105+0000 7f79fe04b700 1 -- 172.21.15.119:0/1424079630 wait complete. 2024-09-13T16:36:58.112 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 105 2024-09-13T16:36:58.123 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:36:58.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 Processor -- start 2024-09-13T16:36:58.231 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- start start 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1d500afae0 0x7f1d500b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d500ce030 con 0x7f1d500af670 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d500b3980 con 0x7f1d500d2a30 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d500cd9e0 con 0x7f1d500afae0 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56229700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1d500afae0 0x7f1d500b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56229700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1d500afae0 0x7f1d500b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57124/0 (socket says 172.21.15.119:57124) 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56229700 1 -- 172.21.15.119:0/1074546283 learned_addr learned my addr 172.21.15.119:0/1074546283 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56229700 1 -- 172.21.15.119:0/1074546283 >> v1:172.21.15.145:6789/0 conn(0x7f1d500d2a30 legacy=0x7f1d500d2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:58.233 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56229700 1 -- 172.21.15.119:0/1074546283 >> v1:172.21.15.119:6789/0 conn(0x7f1d500af670 legacy=0x7f1d500ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56229700 1 -- 172.21.15.119:0/1074546283 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d500cdcc0 con 0x7f1d500afae0 2024-09-13T16:36:58.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56229700 1 --2- 172.21.15.119:0/1074546283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500afae0 0x7f1d500b2c50 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009e50 tx=0x7f1d4800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da535593c0e4e938 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d54a26700 1 -- 172.21.15.119:0/1074546283 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d4800e070 con 0x7f1d500afae0 2024-09-13T16:36:58.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d54a26700 1 -- 172.21.15.119:0/1074546283 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d48003a40 con 0x7f1d500afae0 2024-09-13T16:36:58.234 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d54a26700 1 -- 172.21.15.119:0/1074546283 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48003df0 con 0x7f1d500afae0 2024-09-13T16:36:58.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- 172.21.15.119:0/1074546283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500afae0 msgr2=0x7f1d500b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/1074546283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500afae0 0x7f1d500b2c50 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7f1d48009e50 tx=0x7f1d4800b330 comp rx=0 tx=0).stop 2024-09-13T16:36:58.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- 172.21.15.119:0/1074546283 shutdown_connections 2024-09-13T16:36:58.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/1074546283 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500afae0 0x7f1d500b2c50 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- 172.21.15.119:0/1074546283 >> 172.21.15.119:0/1074546283 conn(0x7f1d5008c0f0 msgr2=0x7f1d5008c500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:58.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- 172.21.15.119:0/1074546283 shutdown_connections 2024-09-13T16:36:58.235 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.229+0000 7f1d56a2a700 1 -- 172.21.15.119:0/1074546283 wait complete. 2024-09-13T16:36:58.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 Processor -- start 2024-09-13T16:36:58.236 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 -- start start 2024-09-13T16:36:58.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d500af670 0x7f1d50153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d500afae0 0x7f1d501e7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500d2a30 0x7f1d500b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d501e8130 con 0x7f1d500d2a30 2024-09-13T16:36:58.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d50153eb0 con 0x7f1d500af670 2024-09-13T16:36:58.239 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56a2a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d500b5a90 con 0x7f1d500afae0 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500d2a30 0x7f1d500b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500d2a30 0x7f1d500b19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57134/0 (socket says 172.21.15.119:57134) 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 -- 172.21.15.119:0/4012529635 learned_addr learned my addr 172.21.15.119:0/4012529635 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d55a28700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d500af670 0x7f1d50153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d55227700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d500afae0 0x7f1d501e7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 -- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d500afae0 msgr2=0x7f1d501e7680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d500afae0 0x7f1d501e7680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 -- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d500af670 msgr2=0x7f1d50153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d500af670 0x7f1d50153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 -- 172.21.15.119:0/4012529635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d48009b30 con 0x7f1d500d2a30 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.233+0000 7f1d56229700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500d2a30 0x7f1d500b19c0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f1d500d05a0 tx=0x7f1d48004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.240 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d467fc700 1 -- 172.21.15.119:0/4012529635 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48004440 con 0x7f1d500d2a30 2024-09-13T16:36:58.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d467fc700 1 -- 172.21.15.119:0/4012529635 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d480045a0 con 0x7f1d500d2a30 2024-09-13T16:36:58.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d500b5d70 con 0x7f1d500d2a30 2024-09-13T16:36:58.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d467fc700 1 -- 172.21.15.119:0/4012529635 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d48004950 con 0x7f1d500d2a30 2024-09-13T16:36:58.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d500b6330 con 0x7f1d500d2a30 2024-09-13T16:36:58.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d500b4370 con 0x7f1d500d2a30 2024-09-13T16:36:58.241 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d467fc700 1 -- 172.21.15.119:0/4012529635 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f1d48012430 con 0x7f1d500d2a30 2024-09-13T16:36:58.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d467fc700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d34066730 0x7f1d34068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d467fc700 1 -- 172.21.15.119:0/4012529635 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f1d48085740 con 0x7f1d500d2a30 2024-09-13T16:36:58.242 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.237+0000 7f1d55a28700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d34066730 0x7f1d34068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.243 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.241+0000 7f1d55a28700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d34066730 0x7f1d34068bf0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c005150 tx=0x7f1d4c007ee0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.247 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.245+0000 7f1d467fc700 1 -- 172.21.15.119:0/4012529635 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f1d480563e0 con 0x7f1d500d2a30 2024-09-13T16:36:58.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.421+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d500ce4a0 con 0x7f1d500d2a30 2024-09-13T16:36:58.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.421+0000 7f1d467fc700 1 -- 172.21.15.119:0/4012529635 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17107 (secure 0 0 0) 0x7f1d48055f70 con 0x7f1d500d2a30 2024-09-13T16:36:58.426 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:36:58.426 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":128,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:36:42.101635+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2024-09-13T16:35:53.564659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_data","create_time":"2024-09-13T16:35:54.247608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"126","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6835/3901706437":"2024-09-14T16:36:21.631788+0000","172.21.15.145:6834/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6834/3901706437":"2024-09-14T16:36:21.631788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d34066730 msgr2=0x7f1d34068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d34066730 0x7f1d34068bf0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f1d4c005150 tx=0x7f1d4c007ee0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500d2a30 msgr2=0x7f1d500b19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500d2a30 0x7f1d500b19c0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7f1d500d05a0 tx=0x7f1d48004af0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 shutdown_connections 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f1d34066730 0x7f1d34068bf0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1d500d2a30 0x7f1d500b19c0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f1d500afae0 0x7f1d501e7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 --2- 172.21.15.119:0/4012529635 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f1d500af670 0x7f1d50153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 >> 172.21.15.119:0/4012529635 conn(0x7f1d5008c0f0 msgr2=0x7f1d5015d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 shutdown_connections 2024-09-13T16:36:58.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.425+0000 7f1d56a2a700 1 -- 172.21.15.119:0/4012529635 wait complete. 2024-09-13T16:36:58.442 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-09-13T16:36:58.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd759e700 1 Processor -- start 2024-09-13T16:36:58.557 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd759e700 1 -- start start 2024-09-13T16:36:58.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 0x7f3bd0009ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd001cfa0 con 0x7f3bd00eb420 2024-09-13T16:36:58.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd001c670 con 0x7f3bd004f320 2024-09-13T16:36:58.564 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd001c950 con 0x7f3bd0033ac0 2024-09-13T16:36:58.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd5d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3bd00eb420 0x7f3bd004ec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49668/0 (socket says 172.21.15.119:49668) 2024-09-13T16:36:58.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.553+0000 7f3bd5d9b700 1 -- 172.21.15.119:0/3585779069 learned_addr learned my addr 172.21.15.119:0/3585779069 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:58.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd659c700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 0x7f3bd0009ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.565 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 166712338 0 0) 0x7f3bd001cfa0 con 0x7f3bd00eb420 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc40035d0 con 0x7f3bd00eb420 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645210794 0 0) 0x7f3bd001c670 con 0x7f3bd004f320 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd001cfa0 con 0x7f3bd004f320 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2524128537 0 0) 0x7f3bc40035d0 con 0x7f3bd00eb420 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 msgr2=0x7f3bd0009ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 0x7f3bd0009ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 >> v1:172.21.15.145:6789/0 conn(0x7f3bd004f320 legacy=0x7f3bd001d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd001cc30 con 0x7f3bd00eb420 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f3bc0001670 con 0x7f3bd00eb420 2024-09-13T16:36:58.566 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 >> v1:172.21.15.119:6789/0 conn(0x7f3bd00eb420 legacy=0x7f3bd004ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 0x7f3bc4003be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bc4004140 0x7f3bc4004870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bc4004db0 0x7f3bc4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd001c950 con 0x7f3bc4004db0 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc400ba60 con 0x7f3bc4004140 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc400bc20 con 0x7f3bd0033ac0 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bc4004db0 0x7f3bc4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd659c700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 0x7f3bc4003be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd5d9b700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bc4004140 0x7f3bc4004870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 msgr2=0x7f3bc4003be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd0033ac0 0x7f3bc4003be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bc4004140 msgr2=0x7f3bc4004870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:58.567 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bc4004140 0x7f3bc4004870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/3585779069 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc40035d0 con 0x7f3bc4004db0 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bc4004db0 0x7f3bc4005230 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f3bd804ffd0 tx=0x7f3bc40058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c0f38e1e68275c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8067480 con 0x7f3bc4004db0 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd80675e0 con 0x7f3bc4004db0 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd559a700 1 -- 172.21.15.119:0/3585779069 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8067bd0 con 0x7f3bc4004db0 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd759e700 1 -- 172.21.15.119:0/3585779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bc4004db0 msgr2=0x7f3bc4005230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd759e700 1 --2- 172.21.15.119:0/3585779069 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bc4004db0 0x7f3bc4005230 secure :-1 s=READY pgs=2009 cs=0 l=1 rev1=1 crypto rx=0x7f3bd804ffd0 tx=0x7f3bc40058a0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/3585779069 reap_dead start 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd759e700 1 -- 172.21.15.119:0/3585779069 shutdown_connections 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd759e700 1 -- 172.21.15.119:0/3585779069 >> 172.21.15.119:0/3585779069 conn(0x7f3bd0036140 msgr2=0x7f3bd0036550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd759e700 1 -- 172.21.15.119:0/3585779069 shutdown_connections 2024-09-13T16:36:58.568 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.557+0000 7f3bd759e700 1 -- 172.21.15.119:0/3585779069 wait complete. 2024-09-13T16:36:58.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 Processor -- start 2024-09-13T16:36:58.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 -- start start 2024-09-13T16:36:58.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bd00dad70 0x7f3bd0034190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd00e52e0 0x7f3bd00346d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bd004b690 0x7f3bd0180280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd0037e00 con 0x7f3bd004b690 2024-09-13T16:36:58.571 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd00374a0 con 0x7f3bd00dad70 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd0038d60 con 0x7f3bd00e52e0 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bd004b690 0x7f3bd0180280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd6d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bd004b690 0x7f3bd0180280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57148/0 (socket says 172.21.15.119:57148) 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/468837234 learned_addr learned my addr 172.21.15.119:0/468837234 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd659c700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bd00dad70 0x7f3bd0034190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.561+0000 7f3bd5d9b700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd00e52e0 0x7f3bd00346d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd00e52e0 msgr2=0x7f3bd00346d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd6d9d700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd00e52e0 0x7f3bd00346d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bd00dad70 msgr2=0x7f3bd0034190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd6d9d700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bd00dad70 0x7f3bd0034190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd6d9d700 1 -- 172.21.15.119:0/468837234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd0037740 con 0x7f3bd004b690 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd6d9d700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bd004b690 0x7f3bd0180280 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8069bc0 tx=0x7f3bd8068560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.572 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bceffd700 1 -- 172.21.15.119:0/468837234 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd809a070 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bceffd700 1 -- 172.21.15.119:0/468837234 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd8073090 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd0037a20 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bceffd700 1 -- 172.21.15.119:0/468837234 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd8072390 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd0038af0 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd0037f40 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bceffd700 1 -- 172.21.15.119:0/468837234 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f3bd80724f0 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bceffd700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3bb8066730 0x7f3bb8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.565+0000 7f3bceffd700 1 -- 172.21.15.119:0/468837234 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6201+0+0 (secure 0 0 0) 0x7f3bd80fe050 con 0x7f3bd004b690 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.569+0000 7f3bd659c700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3bb8066730 0x7f3bb8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.573 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.569+0000 7f3bd659c700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3bb8066730 0x7f3bb8068bf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009b70 tx=0x7f3bc8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.575 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.573+0000 7f3bceffd700 1 -- 172.21.15.119:0/468837234 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f3bd0037f40 con 0x7f3bd004b690 2024-09-13T16:36:58.778 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.773+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3bd0037f40 con 0x7f3bd004b690 2024-09-13T16:36:58.780 INFO:tasks.ceph.mon.a.smithi119.stderr:2024-09-13T16:36:58.777+0000 7fed920dc700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-13T16:36:58.797 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.793+0000 7f3bceffd700 1 -- 172.21.15.119:0/468837234 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v106) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3bd0037f40 con 0x7f3bd004b690 2024-09-13T16:36:58.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3bb8066730 msgr2=0x7f3bb8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3bb8066730 0x7f3bb8068bf0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f3bc8009b70 tx=0x7f3bc8008040 comp rx=0 tx=0).stop 2024-09-13T16:36:58.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bd004b690 msgr2=0x7f3bd0180280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bd004b690 0x7f3bd0180280 secure :-1 s=READY pgs=2010 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8069bc0 tx=0x7f3bd8068560 comp rx=0 tx=0).stop 2024-09-13T16:36:58.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 shutdown_connections 2024-09-13T16:36:58.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f3bb8066730 0x7f3bb8068bf0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3bd004b690 0x7f3bd0180280 unknown :-1 s=CLOSED pgs=2010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.801 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3bd00e52e0 0x7f3bd00346d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 --2- 172.21.15.119:0/468837234 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3bd00dad70 0x7f3bd0034190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 >> 172.21.15.119:0/468837234 conn(0x7f3bd0036140 msgr2=0x7f3bd0009c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:58.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 shutdown_connections 2024-09-13T16:36:58.802 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.797+0000 7f3bd759e700 1 -- 172.21.15.119:0/468837234 wait complete. 2024-09-13T16:36:58.802 INFO:tasks.ceph.mds.b.smithi145.stderr: -1377> 2024-09-13T16:36:48.125+0000 7fa615736700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-09-13T16:36:58.803 INFO:teuthology.orchestra.run.smithi119.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-09-13T16:36:58.815 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-09-13T16:36:58.834 INFO:tasks.ceph.mds.b.smithi145.stdout:starting mds.b at 2024-09-13T16:36:58.845 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:36:58.847 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:36:58.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.929+0000 7fc716bfe700 1 Processor -- start 2024-09-13T16:36:58.934 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.929+0000 7fc716bfe700 1 -- start start 2024-09-13T16:36:58.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc710144180 0x7fc71006c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71006cb40 con 0x7fc7100cc010 2024-09-13T16:36:58.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc71006ce20 con 0x7fc7100ce1d0 2024-09-13T16:36:58.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7100b1c50 con 0x7fc710144180 2024-09-13T16:36:58.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc7153fb700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fc7100cc010 0x7fc7100cdac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:49674/0 (socket says 172.21.15.119:49674) 2024-09-13T16:36:58.935 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc7153fb700 1 -- 172.21.15.119:0/544003206 learned_addr learned my addr 172.21.15.119:0/544003206 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:36:58.936 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc715bfc700 1 --2- 172.21.15.119:0/544003206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc710144180 0x7fc71006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc714bfa700 1 -- 172.21.15.119:0/544003206 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165550525 0 0) 0x7fc71006cb40 con 0x7fc7100cc010 2024-09-13T16:36:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc714bfa700 1 -- 172.21.15.119:0/544003206 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6f80035d0 con 0x7fc7100cc010 2024-09-13T16:36:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc714bfa700 1 -- 172.21.15.119:0/544003206 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157945314 0 0) 0x7fc71006ce20 con 0x7fc7100ce1d0 2024-09-13T16:36:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc714bfa700 1 -- 172.21.15.119:0/544003206 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc71006cb40 con 0x7fc7100ce1d0 2024-09-13T16:36:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc715bfc700 1 -- 172.21.15.119:0/544003206 >> v1:172.21.15.145:6789/0 conn(0x7fc7100ce1d0 legacy=0x7fc7100b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc715bfc700 1 -- 172.21.15.119:0/544003206 >> v1:172.21.15.119:6789/0 conn(0x7fc7100cc010 legacy=0x7fc7100cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.937 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc715bfc700 1 -- 172.21.15.119:0/544003206 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc71006d100 con 0x7fc710144180 2024-09-13T16:36:58.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc715bfc700 1 --2- 172.21.15.119:0/544003206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc710144180 0x7fc71006c480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc704009880 tx=0x7fc704009b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f30966de10834a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc714bfa700 1 -- 172.21.15.119:0/544003206 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7000019f0 con 0x7fc710144180 2024-09-13T16:36:58.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc714bfa700 1 -- 172.21.15.119:0/544003206 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc704010590 con 0x7fc710144180 2024-09-13T16:36:58.938 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc714bfa700 1 -- 172.21.15.119:0/544003206 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc704010940 con 0x7fc710144180 2024-09-13T16:36:58.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- 172.21.15.119:0/544003206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc710144180 msgr2=0x7fc71006c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 --2- 172.21.15.119:0/544003206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc710144180 0x7fc71006c480 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc704009880 tx=0x7fc704009b90 comp rx=0 tx=0).stop 2024-09-13T16:36:58.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- 172.21.15.119:0/544003206 shutdown_connections 2024-09-13T16:36:58.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 --2- 172.21.15.119:0/544003206 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc710144180 0x7fc71006c480 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- 172.21.15.119:0/544003206 >> 172.21.15.119:0/544003206 conn(0x7fc7100b0980 msgr2=0x7fc7100b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:58.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- 172.21.15.119:0/544003206 shutdown_connections 2024-09-13T16:36:58.939 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.933+0000 7fc716bfe700 1 -- 172.21.15.119:0/544003206 wait complete. 2024-09-13T16:36:58.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 Processor -- start 2024-09-13T16:36:58.940 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 -- start start 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7100cc010 0x7fc71013f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7100ce1d0 0x7fc710140a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc710144180 0x7fc71015a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc710140f80 con 0x7fc710144180 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc71013cab0 con 0x7fc7100ce1d0 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc71015a860 con 0x7fc7100cc010 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc710144180 0x7fc71015a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc710144180 0x7fc71015a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57164/0 (socket says 172.21.15.119:57164) 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 -- 172.21.15.119:0/4271679363 learned_addr learned my addr 172.21.15.119:0/4271679363 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc715bfc700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7100cc010 0x7fc71013f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.942 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7153fb700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7100ce1d0 0x7fc710140a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 -- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7100cc010 msgr2=0x7fc71013f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7100cc010 0x7fc71013f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 -- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7100ce1d0 msgr2=0x7fc710140a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7100ce1d0 0x7fc710140a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 -- 172.21.15.119:0/4271679363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7040094b0 con 0x7fc710144180 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc7163fd700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc710144180 0x7fc71015a320 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fc70800c500 tx=0x7fc70800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc70e7fc700 1 -- 172.21.15.119:0/4271679363 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7080052d0 con 0x7fc710144180 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc70e7fc700 1 -- 172.21.15.119:0/4271679363 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc708005430 con 0x7fc710144180 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7101411b0 con 0x7fc710144180 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc70e7fc700 1 -- 172.21.15.119:0/4271679363 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7080055a0 con 0x7fc710144180 2024-09-13T16:36:58.943 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.937+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc71013cd00 con 0x7fc710144180 2024-09-13T16:36:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.941+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc71006c480 con 0x7fc710144180 2024-09-13T16:36:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.941+0000 7fc70e7fc700 1 -- 172.21.15.119:0/4271679363 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc7080038e0 con 0x7fc710144180 2024-09-13T16:36:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.941+0000 7fc70e7fc700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6fc066730 0x7fc6fc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:58.944 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.941+0000 7fc70e7fc700 1 -- 172.21.15.119:0/4271679363 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6287+0+0 (secure 0 0 0) 0x7fc708083860 con 0x7fc710144180 2024-09-13T16:36:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.941+0000 7fc715bfc700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6fc066730 0x7fc6fc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:58.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.941+0000 7fc715bfc700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6fc066730 0x7fc6fc068bf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc7040097d0 tx=0x7fc704010000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:58.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:58.949+0000 7fc70e7fc700 1 -- 172.21.15.119:0/4271679363 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc708050ae0 con 0x7fc710144180 2024-09-13T16:36:59.157 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.153+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fc7100cb870 con 0x7fc710144180 2024-09-13T16:36:59.800 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.797+0000 7fc70e7fc700 1 -- 172.21.15.119:0/4271679363 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v107) v1 ==== 104+0+0 (secure 0 0 0) 0x7fc708054100 con 0x7fc710144180 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6fc066730 msgr2=0x7fc6fc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6fc066730 0x7fc6fc068bf0 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7fc7040097d0 tx=0x7fc704010000 comp rx=0 tx=0).stop 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc710144180 msgr2=0x7fc71015a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc710144180 0x7fc71015a320 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7fc70800c500 tx=0x7fc70800c8c0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 shutdown_connections 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc6fc066730 0x7fc6fc068bf0 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc710144180 0x7fc71015a320 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc7100ce1d0 0x7fc710140a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 --2- 172.21.15.119:0/4271679363 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc7100cc010 0x7fc71013f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 >> 172.21.15.119:0/4271679363 conn(0x7fc7100b0980 msgr2=0x7fc71015cdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:59.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 shutdown_connections 2024-09-13T16:36:59.806 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.801+0000 7fc716bfe700 1 -- 172.21.15.119:0/4271679363 wait complete. 2024-09-13T16:36:59.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-09-13T16:36:59.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.945+0000 7f9ba7fff700 1 Processor -- start 2024-09-13T16:36:59.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.945+0000 7f9ba7fff700 1 -- start start 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f9ba8159720 0x7f9ba80b1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ba80cd8a0 0x7f9ba80cdd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba80ce250 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ba806ffa0 con 0x7f9ba8159720 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba8070260 con 0x7f9ba80cc230 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ba80cd8a0 0x7f9ba80cdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9ba80cd8a0 0x7f9ba80cdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57168/0 (socket says 172.21.15.119:57168) 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 -- 172.21.15.119:0/3894613124 learned_addr learned my addr 172.21.15.119:0/3894613124 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:59.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba67fc700 1 --2- 172.21.15.119:0/3894613124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba6ffd700 1 --2- 172.21.15.119:0/3894613124 >> v2:172.21.15.145:3300/0 conn(0x7f9ba8159720 0x7f9ba80b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 -- 172.21.15.119:0/3894613124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 msgr2=0x7f9ba8159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 --2- 172.21.15.119:0/3894613124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 -- 172.21.15.119:0/3894613124 >> v2:172.21.15.145:3300/0 conn(0x7f9ba8159720 msgr2=0x7f9ba80b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 --2- 172.21.15.119:0/3894613124 >> v2:172.21.15.145:3300/0 conn(0x7f9ba8159720 0x7f9ba80b1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 -- 172.21.15.119:0/3894613124 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba8070540 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba6ffd700 1 --2- 172.21.15.119:0/3894613124 >> v2:172.21.15.145:3300/0 conn(0x7f9ba8159720 0x7f9ba80b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba77fe700 1 --2- 172.21.15.119:0/3894613124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba80cdd10 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80ce460 tx=0x7f9ba000d430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ebaf8912f42da9 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba5ffb700 1 -- 172.21.15.119:0/3894613124 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0010070 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba5ffb700 1 -- 172.21.15.119:0/3894613124 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba000dce0 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba5ffb700 1 -- 172.21.15.119:0/3894613124 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ba000b310 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3894613124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 msgr2=0x7f9ba80cdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3894613124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba80cdd10 secure :-1 s=READY pgs=2014 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80ce460 tx=0x7f9ba000d430 comp rx=0 tx=0).stop 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3894613124 shutdown_connections 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3894613124 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba80cdd10 unknown :-1 s=CLOSED pgs=2014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3894613124 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3894613124 >> v2:172.21.15.145:3300/0 conn(0x7f9ba8159720 0x7f9ba80b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3894613124 >> 172.21.15.119:0/3894613124 conn(0x7f9ba806a350 msgr2=0x7f9ba806a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3894613124 shutdown_connections 2024-09-13T16:36:59.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3894613124 wait complete. 2024-09-13T16:36:59.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 Processor -- start 2024-09-13T16:36:59.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.949+0000 7f9ba7fff700 1 -- start start 2024-09-13T16:36:59.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8149440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba81ee250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ba8159720 0x7f9ba8153ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba81ee8a0 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba81eeb80 con 0x7f9ba8159720 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ba81eee60 con 0x7f9ba80cc230 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba81ee250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba81ee250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57176/0 (socket says 172.21.15.119:57176) 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 -- 172.21.15.119:0/3879537414 learned_addr learned my addr 172.21.15.119:0/3879537414 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba6ffd700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8149440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba77fe700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ba8159720 0x7f9ba8153ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:59.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 -- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 msgr2=0x7f9ba8149440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8149440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 -- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ba8159720 msgr2=0x7f9ba8153ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ba8159720 0x7f9ba8153ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 -- 172.21.15.119:0/3879537414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba000bd40 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba6ffd700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba77fe700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ba8159720 0x7f9ba8153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba67fc700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba81ee250 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800e940 tx=0x7f9b9800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9b8ffff700 1 -- 172.21.15.119:0/3879537414 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b980096a0 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9b8ffff700 1 -- 172.21.15.119:0/3879537414 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b9800c850 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba81ef140 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9b8ffff700 1 -- 172.21.15.119:0/3879537414 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b9800cbe0 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba8154420 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9b8ffff700 1 -- 172.21.15.119:0/3879537414 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9b9800cd40 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ba806fc00 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.953+0000 7f9b8ffff700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b94066730 0x7f9b94068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:36:59.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.957+0000 7f9b8ffff700 1 -- 172.21.15.119:0/3879537414 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6287+0+0 (secure 0 0 0) 0x7f9b98084460 con 0x7f9ba80cd8a0 2024-09-13T16:36:59.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.957+0000 7f9ba6ffd700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b94066730 0x7f9b94068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:36:59.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.957+0000 7f9ba6ffd700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b94066730 0x7f9b94068bf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80b03e0 tx=0x7f9b9c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:36:59.968 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:36:59.965+0000 7f9b8ffff700 1 -- 172.21.15.119:0/3879537414 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9ba806fc00 con 0x7f9ba80cd8a0 2024-09-13T16:37:00.140 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.137+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9ba806fc00 con 0x7f9ba80cd8a0 2024-09-13T16:37:00.805 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.801+0000 7f9b8ffff700 1 -- 172.21.15.119:0/3879537414 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9ba806fc00 con 0x7f9ba80cd8a0 2024-09-13T16:37:00.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b94066730 msgr2=0x7f9b94068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b94066730 0x7f9b94068bf0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f9ba80b03e0 tx=0x7f9b9c008040 comp rx=0 tx=0).stop 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 msgr2=0x7f9ba81ee250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba81ee250 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f9b9800e940 tx=0x7f9b9800ed00 comp rx=0 tx=0).stop 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 shutdown_connections 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9b94066730 0x7f9b94068bf0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9ba8159720 0x7f9ba8153ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9ba80cd8a0 0x7f9ba81ee250 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 --2- 172.21.15.119:0/3879537414 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9ba80cc230 0x7f9ba8149440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 >> 172.21.15.119:0/3879537414 conn(0x7f9ba806a350 msgr2=0x7f9ba8070f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 shutdown_connections 2024-09-13T16:37:00.812 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.809+0000 7f9ba7fff700 1 -- 172.21.15.119:0/3879537414 wait complete. 2024-09-13T16:37:00.813 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_metadata' removed 2024-09-13T16:37:00.827 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-09-13T16:37:00.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.941+0000 7fd10ad76700 1 Processor -- start 2024-09-13T16:37:00.946 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.941+0000 7fd10ad76700 1 -- start start 2024-09-13T16:37:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc008170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd0fc055e70 0x7fd0fc056270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0fc00b160 con 0x7fd0fc04fd20 2024-09-13T16:37:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0fc009170 con 0x7fd0fc055e70 2024-09-13T16:37:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc008bb0 con 0x7fd0fc0df850 2024-09-13T16:37:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc008170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:00.947 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109573700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fd0fc055e70 0x7fd0fc056270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc008170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:50282/0 (socket says 172.21.15.119:50282) 2024-09-13T16:37:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 -- 172.21.15.119:0/4124030670 learned_addr learned my addr 172.21.15.119:0/4124030670 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:00.948 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10a575700 1 --1- 172.21.15.119:0/4124030670 >> v1:172.21.15.145:6789/0 conn(0x7fd0fc04fd20 0x7fd0fc0501b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:40514/0 (socket says 172.21.15.119:40514) 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 -- 172.21.15.119:0/4124030670 >> v2:172.21.15.145:3300/0 conn(0x7fd0fc055e70 msgr2=0x7fd0fc056270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 --2- 172.21.15.119:0/4124030670 >> v2:172.21.15.145:3300/0 conn(0x7fd0fc055e70 0x7fd0fc056270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd108d72700 1 -- 172.21.15.119:0/4124030670 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362468882 0 0) 0x7fd0fc00b160 con 0x7fd0fc04fd20 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 -- 172.21.15.119:0/4124030670 >> v1:172.21.15.145:6789/0 conn(0x7fd0fc04fd20 legacy=0x7fd0fc0501b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 -- 172.21.15.119:0/4124030670 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc008e90 con 0x7fd0fc0df850 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd109d74700 1 --2- 172.21.15.119:0/4124030670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc008170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd100009910 tx=0x7fd100009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=39ead67ae9a3af43 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd108d72700 1 -- 172.21.15.119:0/4124030670 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd100005e50 con 0x7fd0fc0df850 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd108d72700 1 -- 172.21.15.119:0/4124030670 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd100006030 con 0x7fd0fc0df850 2024-09-13T16:37:00.949 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd108d72700 1 -- 172.21.15.119:0/4124030670 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1000063e0 con 0x7fd0fc0df850 2024-09-13T16:37:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- 172.21.15.119:0/4124030670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 msgr2=0x7fd0fc008170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 --2- 172.21.15.119:0/4124030670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc008170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd100009910 tx=0x7fd100009c20 comp rx=0 tx=0).stop 2024-09-13T16:37:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- 172.21.15.119:0/4124030670 shutdown_connections 2024-09-13T16:37:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 --2- 172.21.15.119:0/4124030670 >> v2:172.21.15.145:3300/0 conn(0x7fd0fc055e70 0x7fd0fc056270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.950 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 --2- 172.21.15.119:0/4124030670 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc008170 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- 172.21.15.119:0/4124030670 >> 172.21.15.119:0/4124030670 conn(0x7fd0fc01eaf0 msgr2=0x7fd0fc055760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:00.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- 172.21.15.119:0/4124030670 shutdown_connections 2024-09-13T16:37:00.951 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.945+0000 7fd10ad76700 1 -- 172.21.15.119:0/4124030670 wait complete. 2024-09-13T16:37:00.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 Processor -- start 2024-09-13T16:37:00.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 -- start start 2024-09-13T16:37:00.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0fc04fd20 0x7fd0fc0dc0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:00.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0fc055e70 0x7fd0fc0d73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc006650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0d8e20 con 0x7fd0fc055e70 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0d8210 con 0x7fd0fc04fd20 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10ad76700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0fc0d84f0 con 0x7fd0fc0df850 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109d74700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0fc04fd20 0x7fd0fc0dc0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109573700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0fc055e70 0x7fd0fc0d73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109d74700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0fc04fd20 0x7fd0fc0dc0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:44166/0 (socket says 172.21.15.119:44166) 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109d74700 1 -- 172.21.15.119:0/2271744783 learned_addr learned my addr 172.21.15.119:0/2271744783 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd10a575700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc006650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109573700 1 -- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 msgr2=0x7fd0fc006650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:00.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109573700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc006650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109573700 1 -- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0fc04fd20 msgr2=0x7fd0fc0dc0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109573700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0fc04fd20 0x7fd0fc0dc0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.949+0000 7fd109573700 1 -- 172.21.15.119:0/2271744783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd100009520 con 0x7fd0fc055e70 2024-09-13T16:37:00.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd109573700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0fc055e70 0x7fd0fc0d73f0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400e8d0 tx=0x7fd0f400ebe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd0fa7fc700 1 -- 172.21.15.119:0/2271744783 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4013680 con 0x7fd0fc055e70 2024-09-13T16:37:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd0fa7fc700 1 -- 172.21.15.119:0/2271744783 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f40137e0 con 0x7fd0fc055e70 2024-09-13T16:37:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0fc0d87d0 con 0x7fd0fc055e70 2024-09-13T16:37:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd0fa7fc700 1 -- 172.21.15.119:0/2271744783 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4013b50 con 0x7fd0fc055e70 2024-09-13T16:37:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0fc0dc620 con 0x7fd0fc055e70 2024-09-13T16:37:00.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0fc0d8fd0 con 0x7fd0fc055e70 2024-09-13T16:37:00.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd0fa7fc700 1 -- 172.21.15.119:0/2271744783 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fd0f4013cb0 con 0x7fd0fc055e70 2024-09-13T16:37:00.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.953+0000 7fd0fa7fc700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd0e8066730 0x7fd0e8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:00.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.957+0000 7fd0fa7fc700 1 -- 172.21.15.119:0/2271744783 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 5849+0+0 (secure 0 0 0) 0x7fd0f40840a0 con 0x7fd0fc055e70 2024-09-13T16:37:00.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.957+0000 7fd109d74700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd0e8066730 0x7fd0e8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:00.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.957+0000 7fd109d74700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd0e8066730 0x7fd0e8068bf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd100009c60 tx=0x7fd100008070 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:00.966 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:00.961+0000 7fd0fa7fc700 1 -- 172.21.15.119:0/2271744783 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fd0fc0d8fd0 con 0x7fd0fc055e70 2024-09-13T16:37:01.129 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.125+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd0fc056270 con 0x7fd0fc055e70 2024-09-13T16:37:01.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.805+0000 7fd0fa7fc700 1 -- 172.21.15.119:0/2271744783 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd0f4051370 con 0x7fd0fc055e70 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd0e8066730 msgr2=0x7fd0e8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd0e8066730 0x7fd0e8068bf0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd100009c60 tx=0x7fd100008070 comp rx=0 tx=0).stop 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0fc055e70 msgr2=0x7fd0fc0d73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0fc055e70 0x7fd0fc0d73f0 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400e8d0 tx=0x7fd0f400ebe0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 shutdown_connections 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fd0e8066730 0x7fd0e8068bf0 secure :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd100009c60 tx=0x7fd100008070 comp rx=0 tx=0).stop 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fd0fc0df850 0x7fd0fc006650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd0fc055e70 0x7fd0fc0d73f0 unknown :-1 s=CLOSED pgs=2016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 --2- 172.21.15.119:0/2271744783 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fd0fc04fd20 0x7fd0fc0dc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 >> 172.21.15.119:0/2271744783 conn(0x7fd0fc01eaf0 msgr2=0x7fd0fc04bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:01.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.809+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 shutdown_connections 2024-09-13T16:37:01.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.813+0000 7fd10ad76700 1 -- 172.21.15.119:0/2271744783 wait complete. 2024-09-13T16:37:01.816 INFO:teuthology.orchestra.run.smithi119.stderr:pool 'cephfs_data' removed 2024-09-13T16:37:01.830 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-09-13T16:37:01.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e06ec4700 1 Processor -- start 2024-09-13T16:37:01.945 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e06ec4700 1 -- start start 2024-09-13T16:37:01.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e06ec4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6e0015ccb0 0x7f6e000d0860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e06ec4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e000b5670 0x7f6e0015e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e06ec4700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e000b5bb0 con 0x7f6e000d0da0 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e06ec4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e000d16c0 con 0x7f6e0015ccb0 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e06ec4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e000d1210 con 0x7f6e000b5670 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e05ec2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6e0015ccb0 0x7f6e000d0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e05ec2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f6e0015ccb0 0x7f6e000d0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57200/0 (socket says 172.21.15.119:57200) 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.941+0000 7f6e05ec2700 1 -- 172.21.15.119:0/3696749473 learned_addr learned my addr 172.21.15.119:0/3696749473 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e066c3700 1 --2- 172.21.15.119:0/3696749473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e000b5670 0x7f6e0015e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e05ec2700 1 -- 172.21.15.119:0/3696749473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e000b5670 msgr2=0x7f6e0015e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e05ec2700 1 --2- 172.21.15.119:0/3696749473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e000b5670 0x7f6e0015e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.953 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e05ec2700 1 -- 172.21.15.119:0/3696749473 >> v1:172.21.15.145:6789/0 conn(0x7f6e000d0da0 legacy=0x7f6e000b4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e05ec2700 1 -- 172.21.15.119:0/3696749473 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e0015d3e0 con 0x7f6e0015ccb0 2024-09-13T16:37:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e05ec2700 1 --2- 172.21.15.119:0/3696749473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e0015ccb0 0x7f6e000d0860 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009940 tx=0x7f6df0009c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d208298ae3a18b1c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e04ec0700 1 -- 172.21.15.119:0/3696749473 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6df0011830 con 0x7f6e0015ccb0 2024-09-13T16:37:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e04ec0700 1 -- 172.21.15.119:0/3696749473 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df0011990 con 0x7f6e0015ccb0 2024-09-13T16:37:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- 172.21.15.119:0/3696749473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e0015ccb0 msgr2=0x7f6e000d0860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/3696749473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e0015ccb0 0x7f6e000d0860 secure :-1 s=READY pgs=2017 cs=0 l=1 rev1=1 crypto rx=0x7f6df0009940 tx=0x7f6df0009c50 comp rx=0 tx=0).stop 2024-09-13T16:37:01.954 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- 172.21.15.119:0/3696749473 shutdown_connections 2024-09-13T16:37:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/3696749473 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e000b5670 0x7f6e0015e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/3696749473 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e0015ccb0 0x7f6e000d0860 unknown :-1 s=CLOSED pgs=2017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- 172.21.15.119:0/3696749473 >> 172.21.15.119:0/3696749473 conn(0x7f6e00072240 msgr2=0x7f6e00099a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- 172.21.15.119:0/3696749473 shutdown_connections 2024-09-13T16:37:01.955 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- 172.21.15.119:0/3696749473 wait complete. 2024-09-13T16:37:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 Processor -- start 2024-09-13T16:37:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- start start 2024-09-13T16:37:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e000b5670 0x7f6e00150fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e000d0da0 0x7f6e001f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e0015ccb0 0x7f6e000b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:01.956 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0009b780 con 0x7f6e000d0da0 2024-09-13T16:37:01.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0009ab70 con 0x7f6e000b5670 2024-09-13T16:37:01.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.945+0000 7f6e06ec4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e0009ae50 con 0x7f6e0015ccb0 2024-09-13T16:37:01.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e000d0da0 0x7f6e001f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:01.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e000d0da0 0x7f6e001f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57204/0 (socket says 172.21.15.119:57204) 2024-09-13T16:37:01.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 -- 172.21.15.119:0/2906980593 learned_addr learned my addr 172.21.15.119:0/2906980593 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:01.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e05ec2700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e000b5670 0x7f6e00150fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:01.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e066c3700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e0015ccb0 0x7f6e000b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:01.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 -- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e0015ccb0 msgr2=0x7f6e000b4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:01.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e0015ccb0 0x7f6e000b4410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 -- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e000b5670 msgr2=0x7f6e00150fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:01.958 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e000b5670 0x7f6e00150fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 -- 172.21.15.119:0/2906980593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6df0009570 con 0x7f6e000d0da0 2024-09-13T16:37:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e056c1700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e000d0da0 0x7f6e001f26b0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00c500 tx=0x7f6dfc00c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6df67fc700 1 -- 172.21.15.119:0/2906980593 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0052d0 con 0x7f6e000d0da0 2024-09-13T16:37:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6df67fc700 1 -- 172.21.15.119:0/2906980593 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dfc005430 con 0x7f6e000d0da0 2024-09-13T16:37:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6df67fc700 1 -- 172.21.15.119:0/2906980593 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dfc0055a0 con 0x7f6e000d0da0 2024-09-13T16:37:01.959 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e0009b130 con 0x7f6e000d0da0 2024-09-13T16:37:01.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e00151510 con 0x7f6e000d0da0 2024-09-13T16:37:01.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e000b3d80 con 0x7f6e000d0da0 2024-09-13T16:37:01.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6df67fc700 1 -- 172.21.15.119:0/2906980593 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f6dfc0038e0 con 0x7f6e000d0da0 2024-09-13T16:37:01.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6df67fc700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6de4066760 0x7f6de4068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:01.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.949+0000 7f6df67fc700 1 -- 172.21.15.119:0/2906980593 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f6dfc0833e0 con 0x7f6e000d0da0 2024-09-13T16:37:01.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.953+0000 7f6e05ec2700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6de4066760 0x7f6de4068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:01.962 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.953+0000 7f6e05ec2700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6de4066760 0x7f6de4068c20 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6df0014040 tx=0x7f6df0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:01.963 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:01.957+0000 7f6df67fc700 1 -- 172.21.15.119:0/2906980593 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f6dfc050870 con 0x7f6e000d0da0 2024-09-13T16:37:02.127 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.125+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6e000d0860 con 0x7f6e000d0da0 2024-09-13T16:37:02.134 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6df67fc700 1 -- 172.21.15.119:0/2906980593 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6dfc053e90 con 0x7f6e000d0da0 2024-09-13T16:37:02.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6de4066760 msgr2=0x7f6de4068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6de4066760 0x7f6de4068c20 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6df0014040 tx=0x7f6df0008030 comp rx=0 tx=0).stop 2024-09-13T16:37:02.137 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e000d0da0 msgr2=0x7f6e001f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e000d0da0 0x7f6e001f26b0 secure :-1 s=READY pgs=2018 cs=0 l=1 rev1=1 crypto rx=0x7f6dfc00c500 tx=0x7f6dfc00c810 comp rx=0 tx=0).stop 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 shutdown_connections 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6de4066760 0x7f6de4068c20 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f6e0015ccb0 0x7f6e000b4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f6e000d0da0 0x7f6e001f26b0 unknown :-1 s=CLOSED pgs=2018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 --2- 172.21.15.119:0/2906980593 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f6e000b5670 0x7f6e00150fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 >> 172.21.15.119:0/2906980593 conn(0x7f6e00072240 msgr2=0x7f6e001585c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 shutdown_connections 2024-09-13T16:37:02.138 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.133+0000 7f6e06ec4700 1 -- 172.21.15.119:0/2906980593 wait complete. 2024-09-13T16:37:02.152 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan' 2024-09-13T16:37:02.265 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fbfff700 1 Processor -- start 2024-09-13T16:37:02.266 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fbfff700 1 -- start start 2024-09-13T16:37:02.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fbfff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc2fc15aab0 0x7fc2fc06efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:02.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fbfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc06f810 0x7fc2fc0d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:02.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fbfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2fc15ae90 con 0x7fc2fc0ce480 2024-09-13T16:37:02.272 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fbfff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2fc06fe80 con 0x7fc2fc15aab0 2024-09-13T16:37:02.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fbfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc06e210 con 0x7fc2fc06f810 2024-09-13T16:37:02.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2fa7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc06f810 0x7fc2fc0d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:02.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.261+0000 7fc2faffd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fc2fc15aab0 0x7fc2fc06efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:02.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fa7fc700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc06f810 0x7fc2fc0d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:50314/0 (socket says 172.21.15.119:50314) 2024-09-13T16:37:02.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fa7fc700 1 -- 172.21.15.119:0/2244178555 learned_addr learned my addr 172.21.15.119:0/2244178555 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:02.273 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fa7fc700 1 -- 172.21.15.119:0/2244178555 >> v2:172.21.15.145:3300/0 conn(0x7fc2fc15aab0 msgr2=0x7fc2fc06efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:02.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fa7fc700 1 --2- 172.21.15.119:0/2244178555 >> v2:172.21.15.145:3300/0 conn(0x7fc2fc15aab0 0x7fc2fc06efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fa7fc700 1 -- 172.21.15.119:0/2244178555 >> v1:172.21.15.145:6789/0 conn(0x7fc2fc0ce480 legacy=0x7fc2fc0ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.274 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fa7fc700 1 -- 172.21.15.119:0/2244178555 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2fc06e4f0 con 0x7fc2fc06f810 2024-09-13T16:37:02.275 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2faffd700 1 --2- 172.21.15.119:0/2244178555 >> v2:172.21.15.145:3300/0 conn(0x7fc2fc15aab0 0x7fc2fc06efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:02.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fa7fc700 1 --2- 172.21.15.119:0/2244178555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc06f810 0x7fc2fc0d0f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009910 tx=0x7fc2ec009c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc8325ba3a3afb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:02.276 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2f9ffb700 1 -- 172.21.15.119:0/2244178555 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec011830 con 0x7fc2fc06f810 2024-09-13T16:37:02.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2f9ffb700 1 -- 172.21.15.119:0/2244178555 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2ec011990 con 0x7fc2fc06f810 2024-09-13T16:37:02.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2f9ffb700 1 -- 172.21.15.119:0/2244178555 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec011d40 con 0x7fc2fc06f810 2024-09-13T16:37:02.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 -- 172.21.15.119:0/2244178555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc06f810 msgr2=0x7fc2fc0d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/2244178555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc06f810 0x7fc2fc0d0f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009910 tx=0x7fc2ec009c20 comp rx=0 tx=0).stop 2024-09-13T16:37:02.277 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 -- 172.21.15.119:0/2244178555 shutdown_connections 2024-09-13T16:37:02.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/2244178555 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc06f810 0x7fc2fc0d0f20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/2244178555 >> v2:172.21.15.145:3300/0 conn(0x7fc2fc15aab0 0x7fc2fc06efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 -- 172.21.15.119:0/2244178555 >> 172.21.15.119:0/2244178555 conn(0x7fc2fc099a50 msgr2=0x7fc2fc099e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:02.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 -- 172.21.15.119:0/2244178555 shutdown_connections 2024-09-13T16:37:02.278 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.265+0000 7fc2fbfff700 1 -- 172.21.15.119:0/2244178555 wait complete. 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.269+0000 7fc2fbfff700 1 Processor -- start 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.269+0000 7fc2fbfff700 1 -- start start 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.269+0000 7fc2fbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2fc06f810 0x7fc2fc1452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.269+0000 7fc2fbfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2fc0ce480 0x7fc2fc15c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.269+0000 7fc2fbfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc15aab0 0x7fc2fc14ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fbfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc15d830 con 0x7fc2fc06f810 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fbfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc1457e0 con 0x7fc2fc0ce480 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fbfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2fc15d270 con 0x7fc2fc15aab0 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2fc06f810 0x7fc2fc1452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:02.288 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2fc06f810 0x7fc2fc1452a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:57208/0 (socket says 172.21.15.119:57208) 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 -- 172.21.15.119:0/3297428048 learned_addr learned my addr 172.21.15.119:0/3297428048 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fa7fc700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2fc0ce480 0x7fc2fc15c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fb7fe700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc15aab0 0x7fc2fc14ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 -- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc15aab0 msgr2=0x7fc2fc14ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc15aab0 0x7fc2fc14ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 -- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2fc0ce480 msgr2=0x7fc2fc15c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2fc0ce480 0x7fc2fc15c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 -- 172.21.15.119:0/3297428048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2f00094b0 con 0x7fc2fc06f810 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2faffd700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2fc06f810 0x7fc2fc1452a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000ebe0 tx=0x7fc2f000eef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:02.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2e37fe700 1 -- 172.21.15.119:0/3297428048 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0009770 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2e37fe700 1 -- 172.21.15.119:0/3297428048 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2f000c850 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec009570 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2e37fe700 1 -- 172.21.15.119:0/3297428048 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2f000cbe0 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2fc0afc10 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2fc0ccf10 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.273+0000 7fc2e37fe700 1 -- 172.21.15.119:0/3297428048 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fc2f000cd40 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.277+0000 7fc2e37fe700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2e8066730 0x7fc2e8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.277+0000 7fc2e37fe700 1 -- 172.21.15.119:0/3297428048 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fc2f0083ff0 con 0x7fc2fc06f810 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.277+0000 7fc2fa7fc700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2e8066730 0x7fc2e8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.277+0000 7fc2fa7fc700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2e8066730 0x7fc2e8068bf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec000c00 tx=0x7fc2ec008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:02.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.285+0000 7fc2e37fe700 1 -- 172.21.15.119:0/3297428048 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fc2f0051480 con 0x7fc2fc06f810 2024-09-13T16:37:02.519 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.517+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]} v 0) v1 -- 0x7fc2fc159310 con 0x7fc2fc06f810 2024-09-13T16:37:02.809 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.805+0000 7fc2e37fe700 1 -- 172.21.15.119:0/3297428048 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_forward_scrub.TestForwardScrub.test_orphan_scan"]}]=0 v542) v1 ==== 144+0+0 (secure 0 0 0) 0x7fc2f0054aa0 con 0x7fc2fc06f810 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2e8066730 msgr2=0x7fc2e8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2e8066730 0x7fc2e8068bf0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec000c00 tx=0x7fc2ec008030 comp rx=0 tx=0).stop 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2fc06f810 msgr2=0x7fc2fc1452a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2fc06f810 0x7fc2fc1452a0 secure :-1 s=READY pgs=2019 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000ebe0 tx=0x7fc2f000eef0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 shutdown_connections 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fc2e8066730 0x7fc2e8068bf0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fc2fc15aab0 0x7fc2fc14ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fc2fc0ce480 0x7fc2fc15c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 --2- 172.21.15.119:0/3297428048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc2fc06f810 0x7fc2fc1452a0 unknown :-1 s=CLOSED pgs=2019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 >> 172.21.15.119:0/3297428048 conn(0x7fc2fc099a50 msgr2=0x7fc2fc157c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:02.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 shutdown_connections 2024-09-13T16:37:02.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:02.809+0000 7fc2fbfff700 1 -- 172.21.15.119:0/3297428048 wait complete. 2024-09-13T16:37:02.831 INFO:tasks.cephfs_test_runner:test_orphan_scan (tasks.cephfs.test_forward_scrub.TestForwardScrub) ... ok 2024-09-13T16:37:02.832 INFO:tasks.cephfs_test_runner: 2024-09-13T16:37:02.832 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-09-13T16:37:02.832 INFO:tasks.cephfs_test_runner:Ran 7 tests in 660.286s 2024-09-13T16:37:02.832 INFO:tasks.cephfs_test_runner: 2024-09-13T16:37:02.832 INFO:tasks.cephfs_test_runner:OK 2024-09-13T16:37:02.833 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-09-13T16:37:02.841 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-09-13T16:37:02.905 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-09-13T16:37:02.906 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:37:02.906 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:37:02.914 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:37:02.914 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2024-09-13T16:37:02.914 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:37:02.922 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18959 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-09-13T16:37:02.923 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:37:02.923 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:37:02.931 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:37:02.931 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-09-13T16:37:02.932 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/proc/self/mounts of=/dev/stdout 2024-09-13T16:37:02.940 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0 proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 udev /dev devtmpfs rw,nosuid,noexec,relatime,size=16346292k,nr_inodes=4086573,mode=755 0 0 devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 /dev/sda1 / ext4 rw,relatime,errors=remount-ro 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0 tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k 0 0 tmpfs /sys/fs/cgroup tmpfs ro,nosuid,nodev,noexec,mode=755 0 0 cgroup2 /sys/fs/cgroup/unified cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate 0 0 cgroup /sys/fs/cgroup/systemd cgroup rw,nosuid,nodev,noexec,relatime,xattr,name=systemd 0 0 pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0 none /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 cgroup /sys/fs/cgroup/rdma cgroup rw,nosuid,nodev,noexec,relatime,rdma 0 0 cgroup /sys/fs/cgroup/net_cls,net_prio cgroup rw,nosuid,nodev,noexec,relatime,net_cls,net_prio 0 0 cgroup /sys/fs/cgroup/freezer cgroup rw,nosuid,nodev,noexec,relatime,freezer 0 0 cgroup /sys/fs/cgroup/cpuset cgroup rw,nosuid,nodev,noexec,relatime,cpuset 0 0 cgroup /sys/fs/cgroup/devices cgroup rw,nosuid,nodev,noexec,relatime,devices 0 0 cgroup /sys/fs/cgroup/perf_event cgroup rw,nosuid,nodev,noexec,relatime,perf_event 0 0 cgroup /sys/fs/cgroup/memory cgroup rw,nosuid,nodev,noexec,relatime,memory 0 0 cgroup /sys/fs/cgroup/pids cgroup rw,nosuid,nodev,noexec,relatime,pids 0 0 cgroup /sys/fs/cgroup/hugetlb cgroup rw,nosuid,nodev,noexec,relatime,hugetlb 0 0 cgroup /sys/fs/cgroup/cpu,cpuacct cgroup rw,nosuid,nodev,noexec,relatime,cpu,cpuacct 0 0 cgroup /sys/fs/cgroup/blkio cgroup rw,nosuid,nodev,noexec,relatime,blkio 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11874 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0 sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0 nfsd /proc/fs/nfsd nfsd rw,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=3275484k,mode=700,uid=1000,gid=1267 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=3275488k,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-09-13T16:37:02.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:02.940 DEBUG:teuthology.orchestra.run.smithi119:> ip netns list 2024-09-13T16:37:02.973 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-09-13T16:37:02.974 INFO:teuthology.orchestra.run.smithi119.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-09-13T16:37:02.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:02.974 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-09-13T16:37:02.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:02.987 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-09-13T16:37:03.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:03.000 DEBUG:teuthology.orchestra.run.smithi119:> sudo ip link delete ceph-brx 2024-09-13T16:37:03.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:03.173 DEBUG:teuthology.orchestra.run.smithi145:> ip netns list 2024-09-13T16:37:03.180 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-09-13T16:37:03.180 INFO:teuthology.orchestra.run.smithi145.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-09-13T16:37:03.181 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:03.181 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-09-13T16:37:03.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:03.196 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-09-13T16:37:03.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-09-13T16:37:03.211 DEBUG:teuthology.orchestra.run.smithi145:> sudo ip link delete ceph-brx 2024-09-13T16:37:03.396 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-13T16:37:03.406 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-13T16:37:03.406 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:37:03.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 Processor -- start 2024-09-13T16:37:03.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- start start 2024-09-13T16:37:03.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.531 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fad300cbb10 0x7fad300cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad300cc470 con 0x7fad301444b0 2024-09-13T16:37:03.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad3006bd60 con 0x7fad300cbb10 2024-09-13T16:37:03.532 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3015bf10 con 0x7fad301440d0 2024-09-13T16:37:03.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2e7fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fad301444b0 0x7fad3006d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41246/0 (socket says 172.21.15.119:41246) 2024-09-13T16:37:03.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2e7fc700 1 -- 172.21.15.119:0/1954183788 learned_addr learned my addr 172.21.15.119:0/1954183788 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:03.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2effd700 1 --2- 172.21.15.119:0/1954183788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2f7fe700 1 --2- 172.21.15.119:0/1954183788 >> v2:172.21.15.145:3300/0 conn(0x7fad300cbb10 0x7fad300cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2effd700 1 -- 172.21.15.119:0/1954183788 >> v2:172.21.15.145:3300/0 conn(0x7fad300cbb10 msgr2=0x7fad300cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:03.533 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2effd700 1 --2- 172.21.15.119:0/1954183788 >> v2:172.21.15.145:3300/0 conn(0x7fad300cbb10 0x7fad300cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2effd700 1 -- 172.21.15.119:0/1954183788 >> v1:172.21.15.119:6789/0 conn(0x7fad301444b0 legacy=0x7fad3006d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2effd700 1 -- 172.21.15.119:0/1954183788 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad3006d790 con 0x7fad301440d0 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2f7fe700 1 --2- 172.21.15.119:0/1954183788 >> v2:172.21.15.145:3300/0 conn(0x7fad300cbb10 0x7fad300cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2dffb700 1 -- 172.21.15.119:0/1954183788 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698303059 0 0) 0x7fad300cc470 con 0x7fad301444b0 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2effd700 1 --2- 172.21.15.119:0/1954183788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015b9d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fad28009d90 tx=0x7fad2800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=752ab56d39313a95 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2dffb700 1 -- 172.21.15.119:0/1954183788 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad2800e070 con 0x7fad301440d0 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2dffb700 1 -- 172.21.15.119:0/1954183788 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad2800bb20 con 0x7fad301440d0 2024-09-13T16:37:03.534 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad2dffb700 1 -- 172.21.15.119:0/1954183788 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28012430 con 0x7fad301440d0 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- 172.21.15.119:0/1954183788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 msgr2=0x7fad3015b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 --2- 172.21.15.119:0/1954183788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015b9d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fad28009d90 tx=0x7fad2800b330 comp rx=0 tx=0).stop 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- 172.21.15.119:0/1954183788 shutdown_connections 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 --2- 172.21.15.119:0/1954183788 >> v2:172.21.15.145:3300/0 conn(0x7fad300cbb10 0x7fad300cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 --2- 172.21.15.119:0/1954183788 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015b9d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- 172.21.15.119:0/1954183788 >> 172.21.15.119:0/1954183788 conn(0x7fad3006a350 msgr2=0x7fad3006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- 172.21.15.119:0/1954183788 shutdown_connections 2024-09-13T16:37:03.535 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.529+0000 7fad35911700 1 -- 172.21.15.119:0/1954183788 wait complete. 2024-09-13T16:37:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 Processor -- start 2024-09-13T16:37:03.536 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 -- start start 2024-09-13T16:37:03.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad300cbb10 0x7fad3013e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015d2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad301444b0 0x7fad3012fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3013e1a0 con 0x7fad300cbb10 2024-09-13T16:37:03.537 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad3015aba0 con 0x7fad301444b0 2024-09-13T16:37:03.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad35911700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad3015cb30 con 0x7fad301440d0 2024-09-13T16:37:03.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad300cbb10 0x7fad3013e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad300cbb10 0x7fad3013e3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44348/0 (socket says 172.21.15.119:44348) 2024-09-13T16:37:03.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 -- 172.21.15.119:0/2585564483 learned_addr learned my addr 172.21.15.119:0/2585564483 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:03.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2e7fc700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015d2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.538 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2f7fe700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad301444b0 0x7fad3012fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 -- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 msgr2=0x7fad3015d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015d2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 -- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad301444b0 msgr2=0x7fad3012fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad301444b0 0x7fad3012fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 -- 172.21.15.119:0/2585564483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad200094b0 con 0x7fad300cbb10 2024-09-13T16:37:03.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad2effd700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad300cbb10 0x7fad3013e3a0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fad28014040 tx=0x7fad28004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:03.539 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.533+0000 7fad17fff700 1 -- 172.21.15.119:0/2585564483 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28012430 con 0x7fad300cbb10 2024-09-13T16:37:03.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.537+0000 7fad17fff700 1 -- 172.21.15.119:0/2585564483 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad28020070 con 0x7fad300cbb10 2024-09-13T16:37:03.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.537+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad28009a70 con 0x7fad300cbb10 2024-09-13T16:37:03.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.537+0000 7fad17fff700 1 -- 172.21.15.119:0/2585564483 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad28004280 con 0x7fad300cbb10 2024-09-13T16:37:03.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.537+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad301419f0 con 0x7fad300cbb10 2024-09-13T16:37:03.540 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.537+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad3006c2a0 con 0x7fad300cbb10 2024-09-13T16:37:03.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.537+0000 7fad17fff700 1 -- 172.21.15.119:0/2585564483 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fad280043e0 con 0x7fad300cbb10 2024-09-13T16:37:03.542 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.541+0000 7fad17fff700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad1c066730 0x7fad1c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.541+0000 7fad17fff700 1 -- 172.21.15.119:0/2585564483 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fad28022070 con 0x7fad300cbb10 2024-09-13T16:37:03.543 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.541+0000 7fad2e7fc700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad1c066730 0x7fad1c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.544 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.541+0000 7fad2e7fc700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad1c066730 0x7fad1c068bf0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fad30066ab0 tx=0x7fad24007e90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:03.552 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.549+0000 7fad17fff700 1 -- 172.21.15.119:0/2585564483 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fad3006c2a0 con 0x7fad300cbb10 2024-09-13T16:37:03.721 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.717+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad3015e4d0 con 0x7fad1c066730 2024-09-13T16:37:03.727 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad17fff700 1 -- 172.21.15.119:0/2585564483 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+49032 (secure 0 0 0) 0x7fad3015e4d0 con 0x7fad1c066730 2024-09-13T16:37:03.728 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:37:03.729 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad1c066730 msgr2=0x7fad1c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad1c066730 0x7fad1c068bf0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fad30066ab0 tx=0x7fad24007e90 comp rx=0 tx=0).stop 2024-09-13T16:37:03.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad300cbb10 msgr2=0x7fad3013e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.730 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad300cbb10 0x7fad3013e3a0 secure :-1 s=READY pgs=2021 cs=0 l=1 rev1=1 crypto rx=0x7fad28014040 tx=0x7fad28004af0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 shutdown_connections 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fad1c066730 0x7fad1c068bf0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fad301444b0 0x7fad3012fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fad301440d0 0x7fad3015d2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 --2- 172.21.15.119:0/2585564483 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fad300cbb10 0x7fad3013e3a0 unknown :-1 s=CLOSED pgs=2021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 >> 172.21.15.119:0/2585564483 conn(0x7fad3006a350 msgr2=0x7fad30142a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 shutdown_connections 2024-09-13T16:37:03.731 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.725+0000 7fad35911700 1 -- 172.21.15.119:0/2585564483 wait complete. 2024-09-13T16:37:03.732 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:37:03.744 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-13T16:37:03.674820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279056,"kb_used_data":30132,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754695664,"statfs":{"total":773094113280,"available":772808359936,"internally_reserved":0,"allocated":30855168,"data_stored":23906615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390726,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-262,"num_write":-174,"num_write_kb":-8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001861"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T16:36:42.206130+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:36:42.206130+0000","last_peered":"2024-09-13T16:36:42.206130+0000","last_clean":"2024-09-13T16:36:42.206130+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:36:42.206130+0000","last_undegraded":"2024-09-13T16:36:42.206130+0000","last_fullsized":"2024-09-13T16:36:42.206130+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542289,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33280,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338560,"statfs":{"total":96636764160,"available":96602685440,"internally_reserved":0,"allocated":2215936,"data_stored":1335026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.80300000000000005,"15min":0.83699999999999997},"min":{"1min":0.51800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.198,"5min":1.262,"15min":1.357},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.52000000000000002,"15min":0.47699999999999998},"max":{"1min":1.1759999999999999,"5min":1.214,"15min":1.3460000000000001},"last":1.0209999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.56000000000000005,"15min":0.501},"max":{"1min":1.115,"5min":1.212,"15min":1.569},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.89100000000000001,"15min":0.86499999999999999},"min":{"1min":0.63600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.115,"5min":1.2290000000000001,"15min":1.4219999999999999},"last":0.876}]},{"osd":2,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.874},"min":{"1min":0.56699999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.113,"5min":1.1919999999999999,"15min":1.242},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.89100000000000001,"15min":0.87},"min":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.53900000000000003},"max":{"1min":1.238,"5min":1.24,"15min":1.339},"last":1.044}]},{"osd":3,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91300000000000003,"15min":0.95699999999999996},"min":{"1min":0.65300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3,"5min":1.3,"15min":1.476},"last":0.996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.998,"15min":1},"min":{"1min":0.73599999999999999,"5min":0.69599999999999995,"15min":0.65700000000000003},"max":{"1min":1.28,"5min":1.2969999999999999,"15min":1.3939999999999999},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.878,"15min":0.874},"min":{"1min":0.71399999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.089,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.94399999999999995,"15min":1.006},"min":{"1min":0.73299999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.216,"5min":1.381,"15min":1.589},"last":0.68799999999999994}]},{"osd":5,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.98199999999999998,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2230000000000001,"5min":1.444,"15min":1.444},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0720000000000001,"15min":1.091},"min":{"1min":0.99399999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.278,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0369999999999999,"15min":1.0529999999999999},"min":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2330000000000001,"5min":1.72,"15min":1.72},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.044,"15min":1.1080000000000001},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.375,"5min":1.542,"15min":1.6779999999999999},"last":0.90500000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542288,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5664,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5799936,"data_stored":4938761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.84499999999999997,"15min":0.84099999999999997},"min":{"1min":0.53300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.85799999999999998,"15min":0.878},"min":{"1min":0.55800000000000005,"5min":0.309,"15min":0.309},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.4410000000000001},"last":1.129}]},{"osd":1,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":0.84599999999999997},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0509999999999999,"5min":1.153,"15min":1.2030000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.88200000000000001,"15min":0.90800000000000003},"min":{"1min":0.57699999999999996,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1599999999999999,"5min":1.411,"15min":1.411},"last":0.65300000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.86299999999999999,"15min":0.86899999999999999},"min":{"1min":0.57899999999999996,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.262,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.874},"min":{"1min":0.53000000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0940000000000001,"5min":1.399,"15min":1.399},"last":0.747}]},{"osd":3,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.92100000000000004,"15min":0.94899999999999995},"min":{"1min":0.58099999999999996,"5min":0.379,"15min":0.379},"max":{"1min":1.1870000000000001,"5min":1.544,"15min":1.544},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.97499999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.21,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.1859999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95599999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1599999999999999,"5min":1.508,"15min":1.508},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.65500000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.343,"5min":1.55,"15min":1.6140000000000001},"last":0.82699999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.92300000000000004,"15min":0.93100000000000005},"min":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.214,"5min":1.339,"15min":1.339},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0580000000000001,"15min":1.089},"min":{"1min":0.70299999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2829999999999999,"5min":1.615,"15min":1.6439999999999999},"last":1.1020000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.048,"15min":1.087},"min":{"1min":0.72799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.522},"last":1.04},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.052,"15min":1.0589999999999999},"min":{"1min":0.68899999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.427},"last":1.0649999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542290,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":5804032,"data_stored":4938339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.58299999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.1899999999999999,"15min":1.411},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.83399999999999996,"15min":0.83899999999999997},"min":{"1min":0.53100000000000003,"5min":0.497,"15min":0.41699999999999998},"max":{"1min":1.046,"5min":1.3029999999999999,"15min":1.675},"last":0.89200000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.60399999999999998,"5min":0.51900000000000002,"15min":0.46600000000000003},"max":{"1min":1.0840000000000001,"5min":1.2190000000000001,"15min":1.4079999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83199999999999996,"15min":0.83899999999999997},"min":{"1min":0.61299999999999999,"5min":0.53700000000000003,"15min":0.36499999999999999},"max":{"1min":1.085,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.90400000000000003,"15min":0.91600000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.625},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.97599999999999998,"15min":0.97099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.159,"5min":1.331,"15min":1.552},"last":0.97099999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90500000000000003,"15min":0.879},"min":{"1min":0.57199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.23,"5min":1.23,"15min":1.3919999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.88600000000000001,"15min":0.94199999999999995},"min":{"1min":0.70199999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.286,"5min":1.2929999999999999,"15min":3.8460000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88200000000000001,"15min":0.96799999999999997},"min":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.56999999999999995},"max":{"1min":1.2030000000000001,"5min":1.3540000000000001,"15min":3.6869999999999998},"last":1.103},{"interface":"front","average":{"1min":1.073,"5min":1.0669999999999999,"15min":1.0640000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.57499999999999996},"max":{"1min":1.331,"5min":1.365,"15min":1.601},"last":1.0349999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.91800000000000004,"15min":0.94599999999999995},"min":{"1min":0.64300000000000002,"5min":0.61899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.995,"15min":1.0389999999999999},"min":{"1min":0.65800000000000003,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.1499999999999999,"5min":1.3740000000000001,"15min":2.21},"last":1.181}]},{"osd":7,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.008,"15min":1.0269999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.45500000000000002},"max":{"1min":1.4039999999999999,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0429999999999999,"15min":1.1220000000000001},"min":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":0.68799999999999994},"max":{"1min":1.556,"5min":1.556,"15min":3.9830000000000001},"last":1.079}]}]},{"osd":4,"up_from":14,"seq":60129542288,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":5816320,"data_stored":4939494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86099999999999999,"15min":0.84499999999999997},"min":{"1min":0.60999999999999999,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.206,"5min":1.21,"15min":1.411},"last":1.254},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.56499999999999995,"5min":0.54900000000000004,"15min":0.44600000000000001},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.899},"last":1.014}]},{"osd":1,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88400000000000001,"15min":0.90700000000000003},"min":{"1min":0.59099999999999997,"5min":0.53800000000000003,"15min":0.46200000000000002},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":3.085},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86499999999999999,"15min":0.90000000000000002},"min":{"1min":0.54200000000000004,"5min":0.52800000000000002,"15min":0.497},"max":{"1min":1.169,"5min":1.27,"15min":1.5720000000000001},"last":1.196}]},{"osd":2,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.874,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.433},"max":{"1min":1.258,"5min":1.258,"15min":1.4419999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.90300000000000002,"15min":0.94799999999999995},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3220000000000001,"5min":1.3640000000000001,"15min":3.2160000000000002},"last":0.91700000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.93600000000000005},"min":{"1min":0.65900000000000003,"5min":0.60399999999999998,"15min":0.47999999999999998},"max":{"1min":1.2749999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.875},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93200000000000005,"15min":0.94199999999999995},"min":{"1min":0.62,"5min":0.61499999999999999,"15min":0.52200000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.577},"last":1.044}]},{"osd":5,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91000000000000003,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.49299999999999999},"max":{"1min":1.2390000000000001,"5min":1.252,"15min":2.996},"last":1.075},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98199999999999998,"15min":1.0589999999999999},"min":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.495},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":2.8860000000000001},"last":1.0960000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.66700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.234,"5min":1.522,"15min":1.6499999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.085,"5min":1.0700000000000001,"15min":1.099},"min":{"1min":0.77200000000000002,"5min":0.63600000000000001,"15min":0.55500000000000005},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.1469999999999998},"last":1.127}]},{"osd":7,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.81499999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":2.1230000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.105,"5min":1.079,"15min":1.0980000000000001},"min":{"1min":0.78300000000000003,"5min":0.72599999999999998,"15min":0.67000000000000004},"max":{"1min":1.413,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":0.93899999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542289,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1617920,"data_stored":745212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69999999999999996,"15min":0.67800000000000005},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.64900000000000002},"min":{"1min":0.433,"5min":0.40799999999999997,"15min":0.35499999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.115,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71099999999999997,"15min":0.70199999999999996},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.131,"5min":1.133,"15min":1.27},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.80200000000000005,"15min":0.77900000000000003},"min":{"1min":0.57299999999999995,"5min":0.379,"15min":0.379},"max":{"1min":1.1299999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.73999999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.92400000000000004},"min":{"1min":0.74199999999999999,"5min":0.69999999999999996,"15min":0.60499999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.8460000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93999999999999995,"15min":0.94899999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.59399999999999997},"max":{"1min":1.256,"5min":1.3360000000000001,"15min":1.782},"last":1.038}]},{"osd":5,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.59499999999999997},"max":{"1min":1.1990000000000001,"5min":1.397,"15min":1.6299999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.0269999999999999,"15min":1.0600000000000001},"min":{"1min":0.76300000000000001,"5min":0.66900000000000004,"15min":0.56899999999999995},"max":{"1min":1.3049999999999999,"5min":1.581,"15min":1.8140000000000001},"last":1.0109999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0640000000000001,"15min":1.0900000000000001},"min":{"1min":0.76400000000000001,"5min":0.61599999999999999,"15min":0.58699999999999997},"max":{"1min":1.3380000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.0669999999999999,"15min":1.0469999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"max":{"1min":1.54,"5min":1.7030000000000001,"15min":1.95},"last":1.1970000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.093,"15min":1.0820000000000001},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.71399999999999997},"max":{"1min":1.573,"5min":1.657,"15min":1.889},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.294,"5min":1.6240000000000001,"15min":1.756},"last":1.141}]}]},{"osd":2,"up_from":14,"seq":60129542290,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1605632,"data_stored":743880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.62,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.93700000000000006,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.169,"15min":1.169},"last":0.69599999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.88600000000000001,"5min":1.147,"15min":1.147},"last":0.79600000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72099999999999997,"15min":0.68999999999999995},"min":{"1min":0.45200000000000001,"5min":0.39000000000000001,"15min":0.38100000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.746},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.57199999999999995,"5min":0.37,"15min":0.37},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2330000000000001},"last":0.93700000000000006}]},{"osd":4,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93600000000000005,"15min":0.94399999999999995},"min":{"1min":0.69799999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.171,"5min":1.448,"15min":1.5149999999999999},"last":1.014},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.96799999999999997},"min":{"1min":0.69399999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.8799999999999999},"last":1.0629999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95299999999999996,"15min":0.96899999999999997},"min":{"1min":0.63500000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.375,"5min":1.413,"15min":1.4750000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.96799999999999997,"15min":0.96399999999999997},"min":{"1min":0.66500000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.284,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.2529999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.74299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":2.024},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.044,"15min":1.052},"min":{"1min":0.82799999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.452,"15min":1.6359999999999999},"last":1.1020000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0529999999999999,"15min":1.0620000000000001},"min":{"1min":0.76400000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.27,"5min":1.357,"15min":1.4810000000000001},"last":1.202},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0569999999999999,"15min":1.054},"min":{"1min":0.56699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.482,"5min":1.587,"15min":1.587},"last":0.98399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542289,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":5804032,"data_stored":4938206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.23300000000000001},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":1.0880000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.751,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38800000000000001},"max":{"1min":1.0349999999999999,"5min":1.089,"15min":1.2010000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67900000000000005,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.38300000000000001,"15min":0.26700000000000002},"max":{"1min":0.94999999999999996,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73699999999999999,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.29699999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.458},"last":0.95799999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.75800000000000001,"15min":0.72899999999999998},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":1.052,"5min":1.1359999999999999,"15min":1.4019999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.39600000000000002},"max":{"1min":0.89100000000000001,"5min":1.175,"15min":1.403},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.95699999999999996,"15min":0.93100000000000005},"min":{"1min":0.75600000000000001,"5min":0.67800000000000005,"15min":0.505},"max":{"1min":1.212,"5min":1.2190000000000001,"15min":1.546},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.51700000000000002},"max":{"1min":1.1459999999999999,"5min":1.1559999999999999,"15min":2.2040000000000002},"last":0.93500000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96999999999999997,"15min":0.95999999999999996},"min":{"1min":0.70299999999999996,"5min":0.68799999999999994,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.3819999999999999,"15min":1.389},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.93500000000000005,"15min":0.94599999999999995},"min":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.52000000000000002},"max":{"1min":1.1759999999999999,"5min":1.238,"15min":2.3029999999999999},"last":0.98699999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.01,"15min":1.0049999999999999},"min":{"1min":0.77700000000000002,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.3009999999999999,"5min":1.4330000000000001,"15min":1.613},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.056,"15min":1.03},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.59299999999999997},"max":{"1min":1.278,"5min":1.667,"15min":1.698},"last":1.278}]},{"osd":7,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.0129999999999999,"15min":1.008},"min":{"1min":0.74199999999999999,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":1.2609999999999999,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0649999999999999,"15min":1.0569999999999999},"min":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.59299999999999997},"max":{"1min":1.2290000000000001,"5min":1.3779999999999999,"15min":1.7689999999999999},"last":1.222}]}]},{"osd":0,"up_from":14,"seq":60129542289,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33252,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":2191360,"data_stored":1327697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.52700000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68600000000000005,"15min":0.67300000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.101,"5min":1.1559999999999999,"15min":1.157},"last":0.95499999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70999999999999996,"15min":0.71599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"max":{"1min":0.94099999999999995,"5min":1.125,"15min":1.202},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39800000000000002},"max":{"1min":1.0249999999999999,"5min":1.0780000000000001,"15min":1.2629999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.312},"max":{"1min":0.92800000000000005,"5min":1.1259999999999999,"15min":1.4830000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.85499999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.097,"5min":1.155,"15min":1.6100000000000001},"last":0.77300000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.95999999999999996,"15min":0.95799999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57699999999999996},"max":{"1min":1.3080000000000001,"5min":1.3899999999999999,"15min":1.486},"last":1.107},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.95299999999999996},"min":{"1min":0.70399999999999996,"5min":0.64700000000000002,"15min":0.59999999999999998},"max":{"1min":0.97999999999999998,"5min":1.274,"15min":1.704},"last":1.0640000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":0.996},"min":{"1min":0.80500000000000005,"5min":0.67100000000000004,"15min":0.59599999999999997},"max":{"1min":1.3720000000000001,"5min":1.532,"15min":1.575},"last":1.091},{"interface":"front","average":{"1min":0.996,"5min":0.99199999999999999,"15min":0.98999999999999999},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.443},"max":{"1min":1.399,"5min":1.742,"15min":1.742},"last":1.252}]},{"osd":6,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.97599999999999998},"min":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.628},"max":{"1min":1.1020000000000001,"5min":1.4099999999999999,"15min":1.944},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.0940000000000001,"15min":1.075},"min":{"1min":0.81100000000000005,"5min":0.77300000000000002,"15min":0.58899999999999997},"max":{"1min":1.46,"5min":1.53,"15min":1.5780000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.095,"15min":1.0720000000000001},"min":{"1min":0.86699999999999999,"5min":0.76300000000000001,"15min":0.61699999999999999},"max":{"1min":1.3759999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.169},{"interface":"front","average":{"1min":1.042,"5min":1.073,"15min":1.071},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.65400000000000003},"max":{"1min":1.266,"5min":1.381,"15min":2.0139999999999998},"last":1.1299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:37:03.746 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:37:03.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0759e700 1 Processor -- start 2024-09-13T16:37:03.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0759e700 1 -- start start 2024-09-13T16:37:03.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9e081594f0 0x7f9e0806d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080b2280 con 0x7f9e0806da70 2024-09-13T16:37:03.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e080cc890 con 0x7f9e080b3800 2024-09-13T16:37:03.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0806dee0 con 0x7f9e081594f0 2024-09-13T16:37:03.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e05d9b700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f9e0806da70 0x7f9e080b30f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41254/0 (socket says 172.21.15.119:41254) 2024-09-13T16:37:03.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e05d9b700 1 -- 172.21.15.119:0/1453607597 learned_addr learned my addr 172.21.15.119:0/1453607597 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:03.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0659c700 1 --2- 172.21.15.119:0/1453607597 >> v2:172.21.15.119:3300/0 conn(0x7f9e081594f0 0x7f9e0806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0559a700 1 -- 172.21.15.119:0/1453607597 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739523168 0 0) 0x7f9e080b2280 con 0x7f9e0806da70 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0559a700 1 -- 172.21.15.119:0/1453607597 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df40035d0 con 0x7f9e0806da70 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0659c700 1 -- 172.21.15.119:0/1453607597 >> v1:172.21.15.145:6789/0 conn(0x7f9e080b3800 legacy=0x7f9e0806fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0659c700 1 -- 172.21.15.119:0/1453607597 >> v1:172.21.15.119:6789/0 conn(0x7f9e0806da70 legacy=0x7f9e080b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0659c700 1 -- 172.21.15.119:0/1453607597 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e080b1970 con 0x7f9e081594f0 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.857+0000 7f9e0659c700 1 --2- 172.21.15.119:0/1453607597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e0806d530 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009850 tx=0x7f9df0009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e5170a7b0325261 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0559a700 1 -- 172.21.15.119:0/1453607597 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0011830 con 0x7f9e081594f0 2024-09-13T16:37:03.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0559a700 1 -- 172.21.15.119:0/1453607597 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df0011990 con 0x7f9e081594f0 2024-09-13T16:37:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0559a700 1 -- 172.21.15.119:0/1453607597 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df0011d00 con 0x7f9e081594f0 2024-09-13T16:37:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0759e700 1 -- 172.21.15.119:0/1453607597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 msgr2=0x7f9e0806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0759e700 1 --2- 172.21.15.119:0/1453607597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e0806d530 secure :-1 s=READY pgs=2023 cs=0 l=1 rev1=1 crypto rx=0x7f9df0009850 tx=0x7f9df0009b60 comp rx=0 tx=0).stop 2024-09-13T16:37:03.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0759e700 1 -- 172.21.15.119:0/1453607597 shutdown_connections 2024-09-13T16:37:03.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0759e700 1 --2- 172.21.15.119:0/1453607597 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e0806d530 unknown :-1 s=CLOSED pgs=2023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0759e700 1 -- 172.21.15.119:0/1453607597 >> 172.21.15.119:0/1453607597 conn(0x7f9e0806a350 msgr2=0x7f9e0806a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:03.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0759e700 1 -- 172.21.15.119:0/1453607597 shutdown_connections 2024-09-13T16:37:03.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.861+0000 7f9e0759e700 1 -- 172.21.15.119:0/1453607597 wait complete. 2024-09-13T16:37:03.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 Processor -- start 2024-09-13T16:37:03.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 -- start start 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e0806da70 0x7f9e081529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e080b3800 0x7f9e0814da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e08158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0814d640 con 0x7f9e081594f0 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08142210 con 0x7f9e0806da70 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e081426b0 con 0x7f9e080b3800 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e08158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e08158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44368/0 (socket says 172.21.15.119:44368) 2024-09-13T16:37:03.870 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 -- 172.21.15.119:0/734983120 learned_addr learned my addr 172.21.15.119:0/734983120 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e05d9b700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e080b3800 0x7f9e0814da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0659c700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e0806da70 0x7f9e081529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 -- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e080b3800 msgr2=0x7f9e0814da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e080b3800 0x7f9e0814da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 -- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e0806da70 msgr2=0x7f9e081529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e0806da70 0x7f9e081529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 -- 172.21.15.119:0/734983120 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00094b0 con 0x7f9e081594f0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e06d9d700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e08158240 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000ee70 tx=0x7f9e0000c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9dfeffd700 1 -- 172.21.15.119:0/734983120 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00003a20 con 0x7f9e081594f0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9dfeffd700 1 -- 172.21.15.119:0/734983120 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e00003b80 con 0x7f9e081594f0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0814c6e0 con 0x7f9e081594f0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9dfeffd700 1 -- 172.21.15.119:0/734983120 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e000050d0 con 0x7f9e081594f0 2024-09-13T16:37:03.871 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.865+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0814cca0 con 0x7f9e081594f0 2024-09-13T16:37:03.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.869+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e080b26f0 con 0x7f9e081594f0 2024-09-13T16:37:03.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.869+0000 7f9dfeffd700 1 -- 172.21.15.119:0/734983120 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f9e00005230 con 0x7f9e081594f0 2024-09-13T16:37:03.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.869+0000 7f9dfeffd700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9de8066730 0x7f9de8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:03.872 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.869+0000 7f9dfeffd700 1 -- 172.21.15.119:0/734983120 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f9e00086740 con 0x7f9e081594f0 2024-09-13T16:37:03.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.869+0000 7f9e0659c700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9de8066730 0x7f9de8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:03.873 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.869+0000 7f9e0659c700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9de8066730 0x7f9de8068bf0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9df00090b0 tx=0x7f9df0008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:03.879 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:03.877+0000 7f9dfeffd700 1 -- 172.21.15.119:0/734983120 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f9e080b26f0 con 0x7f9e081594f0 2024-09-13T16:37:04.049 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.041+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9e080b1f10 con 0x7f9de8066730 2024-09-13T16:37:04.051 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9dfeffd700 1 -- 172.21.15.119:0/734983120 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+49032 (secure 0 0 0) 0x7f9df0011730 con 0x7f9de8066730 2024-09-13T16:37:04.051 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:37:04.054 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9de8066730 msgr2=0x7f9de8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9de8066730 0x7f9de8068bf0 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f9df00090b0 tx=0x7f9df0008030 comp rx=0 tx=0).stop 2024-09-13T16:37:04.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 msgr2=0x7f9e08158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e08158240 secure :-1 s=READY pgs=2024 cs=0 l=1 rev1=1 crypto rx=0x7f9e0000ee70 tx=0x7f9e0000c550 comp rx=0 tx=0).stop 2024-09-13T16:37:04.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 shutdown_connections 2024-09-13T16:37:04.055 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f9de8066730 0x7f9de8068bf0 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9e081594f0 0x7f9e08158240 unknown :-1 s=CLOSED pgs=2024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f9e080b3800 0x7f9e0814da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 --2- 172.21.15.119:0/734983120 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f9e0806da70 0x7f9e081529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 >> 172.21.15.119:0/734983120 conn(0x7f9e0806a350 msgr2=0x7f9e0814b9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:04.056 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 shutdown_connections 2024-09-13T16:37:04.057 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.049+0000 7f9e0759e700 1 -- 172.21.15.119:0/734983120 wait complete. 2024-09-13T16:37:04.057 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:37:04.067 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-13T16:37:03.674820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279056,"kb_used_data":30132,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754695664,"statfs":{"total":773094113280,"available":772808359936,"internally_reserved":0,"allocated":30855168,"data_stored":23906615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390726,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-262,"num_write":-174,"num_write_kb":-8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001861"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T16:36:42.206130+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:36:42.206130+0000","last_peered":"2024-09-13T16:36:42.206130+0000","last_clean":"2024-09-13T16:36:42.206130+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:36:42.206130+0000","last_undegraded":"2024-09-13T16:36:42.206130+0000","last_fullsized":"2024-09-13T16:36:42.206130+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542289,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33280,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338560,"statfs":{"total":96636764160,"available":96602685440,"internally_reserved":0,"allocated":2215936,"data_stored":1335026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.80300000000000005,"15min":0.83699999999999997},"min":{"1min":0.51800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.198,"5min":1.262,"15min":1.357},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.52000000000000002,"15min":0.47699999999999998},"max":{"1min":1.1759999999999999,"5min":1.214,"15min":1.3460000000000001},"last":1.0209999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.56000000000000005,"15min":0.501},"max":{"1min":1.115,"5min":1.212,"15min":1.569},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.89100000000000001,"15min":0.86499999999999999},"min":{"1min":0.63600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.115,"5min":1.2290000000000001,"15min":1.4219999999999999},"last":0.876}]},{"osd":2,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.874},"min":{"1min":0.56699999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.113,"5min":1.1919999999999999,"15min":1.242},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.89100000000000001,"15min":0.87},"min":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.53900000000000003},"max":{"1min":1.238,"5min":1.24,"15min":1.339},"last":1.044}]},{"osd":3,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91300000000000003,"15min":0.95699999999999996},"min":{"1min":0.65300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3,"5min":1.3,"15min":1.476},"last":0.996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.998,"15min":1},"min":{"1min":0.73599999999999999,"5min":0.69599999999999995,"15min":0.65700000000000003},"max":{"1min":1.28,"5min":1.2969999999999999,"15min":1.3939999999999999},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.878,"15min":0.874},"min":{"1min":0.71399999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.089,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.94399999999999995,"15min":1.006},"min":{"1min":0.73299999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.216,"5min":1.381,"15min":1.589},"last":0.68799999999999994}]},{"osd":5,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.98199999999999998,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2230000000000001,"5min":1.444,"15min":1.444},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0720000000000001,"15min":1.091},"min":{"1min":0.99399999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.278,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0369999999999999,"15min":1.0529999999999999},"min":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2330000000000001,"5min":1.72,"15min":1.72},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.044,"15min":1.1080000000000001},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.375,"5min":1.542,"15min":1.6779999999999999},"last":0.90500000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542288,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5664,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5799936,"data_stored":4938761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.84499999999999997,"15min":0.84099999999999997},"min":{"1min":0.53300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.85799999999999998,"15min":0.878},"min":{"1min":0.55800000000000005,"5min":0.309,"15min":0.309},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.4410000000000001},"last":1.129}]},{"osd":1,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":0.84599999999999997},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0509999999999999,"5min":1.153,"15min":1.2030000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.88200000000000001,"15min":0.90800000000000003},"min":{"1min":0.57699999999999996,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1599999999999999,"5min":1.411,"15min":1.411},"last":0.65300000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.86299999999999999,"15min":0.86899999999999999},"min":{"1min":0.57899999999999996,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.262,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.874},"min":{"1min":0.53000000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0940000000000001,"5min":1.399,"15min":1.399},"last":0.747}]},{"osd":3,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.92100000000000004,"15min":0.94899999999999995},"min":{"1min":0.58099999999999996,"5min":0.379,"15min":0.379},"max":{"1min":1.1870000000000001,"5min":1.544,"15min":1.544},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.97499999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.21,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.1859999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95599999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1599999999999999,"5min":1.508,"15min":1.508},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.65500000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.343,"5min":1.55,"15min":1.6140000000000001},"last":0.82699999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.92300000000000004,"15min":0.93100000000000005},"min":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.214,"5min":1.339,"15min":1.339},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0580000000000001,"15min":1.089},"min":{"1min":0.70299999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2829999999999999,"5min":1.615,"15min":1.6439999999999999},"last":1.1020000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.048,"15min":1.087},"min":{"1min":0.72799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.522},"last":1.04},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.052,"15min":1.0589999999999999},"min":{"1min":0.68899999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.427},"last":1.0649999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542290,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":5804032,"data_stored":4938339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.58299999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.1899999999999999,"15min":1.411},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.83399999999999996,"15min":0.83899999999999997},"min":{"1min":0.53100000000000003,"5min":0.497,"15min":0.41699999999999998},"max":{"1min":1.046,"5min":1.3029999999999999,"15min":1.675},"last":0.89200000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.60399999999999998,"5min":0.51900000000000002,"15min":0.46600000000000003},"max":{"1min":1.0840000000000001,"5min":1.2190000000000001,"15min":1.4079999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83199999999999996,"15min":0.83899999999999997},"min":{"1min":0.61299999999999999,"5min":0.53700000000000003,"15min":0.36499999999999999},"max":{"1min":1.085,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.90400000000000003,"15min":0.91600000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.625},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.97599999999999998,"15min":0.97099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.159,"5min":1.331,"15min":1.552},"last":0.97099999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90500000000000003,"15min":0.879},"min":{"1min":0.57199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.23,"5min":1.23,"15min":1.3919999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.88600000000000001,"15min":0.94199999999999995},"min":{"1min":0.70199999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.286,"5min":1.2929999999999999,"15min":3.8460000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88200000000000001,"15min":0.96799999999999997},"min":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.56999999999999995},"max":{"1min":1.2030000000000001,"5min":1.3540000000000001,"15min":3.6869999999999998},"last":1.103},{"interface":"front","average":{"1min":1.073,"5min":1.0669999999999999,"15min":1.0640000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.57499999999999996},"max":{"1min":1.331,"5min":1.365,"15min":1.601},"last":1.0349999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.91800000000000004,"15min":0.94599999999999995},"min":{"1min":0.64300000000000002,"5min":0.61899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.995,"15min":1.0389999999999999},"min":{"1min":0.65800000000000003,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.1499999999999999,"5min":1.3740000000000001,"15min":2.21},"last":1.181}]},{"osd":7,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.008,"15min":1.0269999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.45500000000000002},"max":{"1min":1.4039999999999999,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0429999999999999,"15min":1.1220000000000001},"min":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":0.68799999999999994},"max":{"1min":1.556,"5min":1.556,"15min":3.9830000000000001},"last":1.079}]}]},{"osd":4,"up_from":14,"seq":60129542288,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":5816320,"data_stored":4939494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86099999999999999,"15min":0.84499999999999997},"min":{"1min":0.60999999999999999,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.206,"5min":1.21,"15min":1.411},"last":1.254},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.56499999999999995,"5min":0.54900000000000004,"15min":0.44600000000000001},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.899},"last":1.014}]},{"osd":1,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88400000000000001,"15min":0.90700000000000003},"min":{"1min":0.59099999999999997,"5min":0.53800000000000003,"15min":0.46200000000000002},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":3.085},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86499999999999999,"15min":0.90000000000000002},"min":{"1min":0.54200000000000004,"5min":0.52800000000000002,"15min":0.497},"max":{"1min":1.169,"5min":1.27,"15min":1.5720000000000001},"last":1.196}]},{"osd":2,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.874,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.433},"max":{"1min":1.258,"5min":1.258,"15min":1.4419999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.90300000000000002,"15min":0.94799999999999995},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3220000000000001,"5min":1.3640000000000001,"15min":3.2160000000000002},"last":0.91700000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.93600000000000005},"min":{"1min":0.65900000000000003,"5min":0.60399999999999998,"15min":0.47999999999999998},"max":{"1min":1.2749999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.875},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93200000000000005,"15min":0.94199999999999995},"min":{"1min":0.62,"5min":0.61499999999999999,"15min":0.52200000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.577},"last":1.044}]},{"osd":5,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91000000000000003,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.49299999999999999},"max":{"1min":1.2390000000000001,"5min":1.252,"15min":2.996},"last":1.075},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98199999999999998,"15min":1.0589999999999999},"min":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.495},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":2.8860000000000001},"last":1.0960000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.66700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.234,"5min":1.522,"15min":1.6499999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.085,"5min":1.0700000000000001,"15min":1.099},"min":{"1min":0.77200000000000002,"5min":0.63600000000000001,"15min":0.55500000000000005},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.1469999999999998},"last":1.127}]},{"osd":7,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.81499999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":2.1230000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.105,"5min":1.079,"15min":1.0980000000000001},"min":{"1min":0.78300000000000003,"5min":0.72599999999999998,"15min":0.67000000000000004},"max":{"1min":1.413,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":0.93899999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542289,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1617920,"data_stored":745212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69999999999999996,"15min":0.67800000000000005},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.64900000000000002},"min":{"1min":0.433,"5min":0.40799999999999997,"15min":0.35499999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.115,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71099999999999997,"15min":0.70199999999999996},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.131,"5min":1.133,"15min":1.27},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.80200000000000005,"15min":0.77900000000000003},"min":{"1min":0.57299999999999995,"5min":0.379,"15min":0.379},"max":{"1min":1.1299999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.73999999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.92400000000000004},"min":{"1min":0.74199999999999999,"5min":0.69999999999999996,"15min":0.60499999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.8460000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93999999999999995,"15min":0.94899999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.59399999999999997},"max":{"1min":1.256,"5min":1.3360000000000001,"15min":1.782},"last":1.038}]},{"osd":5,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.59499999999999997},"max":{"1min":1.1990000000000001,"5min":1.397,"15min":1.6299999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.0269999999999999,"15min":1.0600000000000001},"min":{"1min":0.76300000000000001,"5min":0.66900000000000004,"15min":0.56899999999999995},"max":{"1min":1.3049999999999999,"5min":1.581,"15min":1.8140000000000001},"last":1.0109999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0640000000000001,"15min":1.0900000000000001},"min":{"1min":0.76400000000000001,"5min":0.61599999999999999,"15min":0.58699999999999997},"max":{"1min":1.3380000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.0669999999999999,"15min":1.0469999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"max":{"1min":1.54,"5min":1.7030000000000001,"15min":1.95},"last":1.1970000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.093,"15min":1.0820000000000001},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.71399999999999997},"max":{"1min":1.573,"5min":1.657,"15min":1.889},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.294,"5min":1.6240000000000001,"15min":1.756},"last":1.141}]}]},{"osd":2,"up_from":14,"seq":60129542290,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1605632,"data_stored":743880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.62,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.93700000000000006,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.169,"15min":1.169},"last":0.69599999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.88600000000000001,"5min":1.147,"15min":1.147},"last":0.79600000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72099999999999997,"15min":0.68999999999999995},"min":{"1min":0.45200000000000001,"5min":0.39000000000000001,"15min":0.38100000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.746},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.57199999999999995,"5min":0.37,"15min":0.37},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2330000000000001},"last":0.93700000000000006}]},{"osd":4,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93600000000000005,"15min":0.94399999999999995},"min":{"1min":0.69799999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.171,"5min":1.448,"15min":1.5149999999999999},"last":1.014},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.96799999999999997},"min":{"1min":0.69399999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.8799999999999999},"last":1.0629999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95299999999999996,"15min":0.96899999999999997},"min":{"1min":0.63500000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.375,"5min":1.413,"15min":1.4750000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.96799999999999997,"15min":0.96399999999999997},"min":{"1min":0.66500000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.284,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.2529999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.74299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":2.024},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.044,"15min":1.052},"min":{"1min":0.82799999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.452,"15min":1.6359999999999999},"last":1.1020000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0529999999999999,"15min":1.0620000000000001},"min":{"1min":0.76400000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.27,"5min":1.357,"15min":1.4810000000000001},"last":1.202},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0569999999999999,"15min":1.054},"min":{"1min":0.56699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.482,"5min":1.587,"15min":1.587},"last":0.98399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542289,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":5804032,"data_stored":4938206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.23300000000000001},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":1.0880000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.751,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38800000000000001},"max":{"1min":1.0349999999999999,"5min":1.089,"15min":1.2010000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67900000000000005,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.38300000000000001,"15min":0.26700000000000002},"max":{"1min":0.94999999999999996,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73699999999999999,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.29699999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.458},"last":0.95799999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.75800000000000001,"15min":0.72899999999999998},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":1.052,"5min":1.1359999999999999,"15min":1.4019999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.39600000000000002},"max":{"1min":0.89100000000000001,"5min":1.175,"15min":1.403},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.95699999999999996,"15min":0.93100000000000005},"min":{"1min":0.75600000000000001,"5min":0.67800000000000005,"15min":0.505},"max":{"1min":1.212,"5min":1.2190000000000001,"15min":1.546},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.51700000000000002},"max":{"1min":1.1459999999999999,"5min":1.1559999999999999,"15min":2.2040000000000002},"last":0.93500000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96999999999999997,"15min":0.95999999999999996},"min":{"1min":0.70299999999999996,"5min":0.68799999999999994,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.3819999999999999,"15min":1.389},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.93500000000000005,"15min":0.94599999999999995},"min":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.52000000000000002},"max":{"1min":1.1759999999999999,"5min":1.238,"15min":2.3029999999999999},"last":0.98699999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.01,"15min":1.0049999999999999},"min":{"1min":0.77700000000000002,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.3009999999999999,"5min":1.4330000000000001,"15min":1.613},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.056,"15min":1.03},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.59299999999999997},"max":{"1min":1.278,"5min":1.667,"15min":1.698},"last":1.278}]},{"osd":7,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.0129999999999999,"15min":1.008},"min":{"1min":0.74199999999999999,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":1.2609999999999999,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0649999999999999,"15min":1.0569999999999999},"min":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.59299999999999997},"max":{"1min":1.2290000000000001,"5min":1.3779999999999999,"15min":1.7689999999999999},"last":1.222}]}]},{"osd":0,"up_from":14,"seq":60129542289,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33252,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":2191360,"data_stored":1327697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.52700000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68600000000000005,"15min":0.67300000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.101,"5min":1.1559999999999999,"15min":1.157},"last":0.95499999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70999999999999996,"15min":0.71599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"max":{"1min":0.94099999999999995,"5min":1.125,"15min":1.202},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39800000000000002},"max":{"1min":1.0249999999999999,"5min":1.0780000000000001,"15min":1.2629999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.312},"max":{"1min":0.92800000000000005,"5min":1.1259999999999999,"15min":1.4830000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.85499999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.097,"5min":1.155,"15min":1.6100000000000001},"last":0.77300000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.95999999999999996,"15min":0.95799999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57699999999999996},"max":{"1min":1.3080000000000001,"5min":1.3899999999999999,"15min":1.486},"last":1.107},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.95299999999999996},"min":{"1min":0.70399999999999996,"5min":0.64700000000000002,"15min":0.59999999999999998},"max":{"1min":0.97999999999999998,"5min":1.274,"15min":1.704},"last":1.0640000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":0.996},"min":{"1min":0.80500000000000005,"5min":0.67100000000000004,"15min":0.59599999999999997},"max":{"1min":1.3720000000000001,"5min":1.532,"15min":1.575},"last":1.091},{"interface":"front","average":{"1min":0.996,"5min":0.99199999999999999,"15min":0.98999999999999999},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.443},"max":{"1min":1.399,"5min":1.742,"15min":1.742},"last":1.252}]},{"osd":6,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.97599999999999998},"min":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.628},"max":{"1min":1.1020000000000001,"5min":1.4099999999999999,"15min":1.944},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.0940000000000001,"15min":1.075},"min":{"1min":0.81100000000000005,"5min":0.77300000000000002,"15min":0.58899999999999997},"max":{"1min":1.46,"5min":1.53,"15min":1.5780000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.095,"15min":1.0720000000000001},"min":{"1min":0.86699999999999999,"5min":0.76300000000000001,"15min":0.61699999999999999},"max":{"1min":1.3759999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.169},{"interface":"front","average":{"1min":1.042,"5min":1.073,"15min":1.071},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.65400000000000003},"max":{"1min":1.266,"5min":1.381,"15min":2.0139999999999998},"last":1.1299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:37:04.069 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-13T16:37:04.069 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:37:04.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.181+0000 7f2fd20af700 1 Processor -- start 2024-09-13T16:37:04.187 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- start start 2024-09-13T16:37:04.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2fcc0cbb10 0x7f2fcc0cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc06bd60 con 0x7f2fcc1444b0 2024-09-13T16:37:04.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fcc06d790 con 0x7f2fcc1440d0 2024-09-13T16:37:04.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fcc0cc4c0 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fcaffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2fcc1444b0 0x7f2fcc06d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41262/0 (socket says 172.21.15.119:41262) 2024-09-13T16:37:04.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fcaffd700 1 -- 172.21.15.119:0/189742338 learned_addr learned my addr 172.21.15.119:0/189742338 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:04.188 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fcbfff700 1 --2- 172.21.15.119:0/189742338 >> v2:172.21.15.145:3300/0 conn(0x7f2fcc0cbb10 0x7f2fcc0cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39643930 0 0) 0x7f2fcc06bd60 con 0x7f2fcc1444b0 2024-09-13T16:37:04.189 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fb40035d0 con 0x7f2fcc1444b0 2024-09-13T16:37:04.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284775008 0 0) 0x7f2fcc06d790 con 0x7f2fcc1440d0 2024-09-13T16:37:04.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fcc06bd60 con 0x7f2fcc1440d0 2024-09-13T16:37:04.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fcbfff700 1 -- 172.21.15.119:0/189742338 >> v1:172.21.15.145:6789/0 conn(0x7f2fcc1440d0 legacy=0x7f2fcc15b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2233287839 0 0) 0x7f2fb40035d0 con 0x7f2fcc1444b0 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fcbfff700 1 -- 172.21.15.119:0/189742338 >> v1:172.21.15.119:6789/0 conn(0x7f2fcc1444b0 legacy=0x7f2fcc06d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fcbfff700 1 -- 172.21.15.119:0/189742338 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc06bfe0 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fcbfff700 1 --2- 172.21.15.119:0/189742338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc0cbb10 0x7f2fcc0cbf80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009d90 tx=0x7f2fc400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=965148238b6b6a7e server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc400e070 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc4003a40 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fca7fc700 1 -- 172.21.15.119:0/189742338 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc001670 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- 172.21.15.119:0/189742338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc0cbb10 msgr2=0x7f2fcc0cbf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 --2- 172.21.15.119:0/189742338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc0cbb10 0x7f2fcc0cbf80 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009d90 tx=0x7f2fc400b330 comp rx=0 tx=0).stop 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- 172.21.15.119:0/189742338 shutdown_connections 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 --2- 172.21.15.119:0/189742338 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc0cbb10 0x7f2fcc0cbf80 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- 172.21.15.119:0/189742338 >> 172.21.15.119:0/189742338 conn(0x7f2fcc06a350 msgr2=0x7f2fcc06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:04.191 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- 172.21.15.119:0/189742338 shutdown_connections 2024-09-13T16:37:04.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.185+0000 7f2fd20af700 1 -- 172.21.15.119:0/189742338 wait complete. 2024-09-13T16:37:04.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 Processor -- start 2024-09-13T16:37:04.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 -- start start 2024-09-13T16:37:04.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fcc0cbb10 0x7f2fcc15a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc1440d0 0x7f2fcc1f04b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fcc1444b0 0x7f2fcc13c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15cdf0 con 0x7f2fcc1444b0 2024-09-13T16:37:04.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15c500 con 0x7f2fcc1440d0 2024-09-13T16:37:04.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fd20af700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fcc15c7a0 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.194 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fcc1444b0 0x7f2fcc13c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fcc0cbb10 0x7f2fcc15a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fcc0cbb10 0x7f2fcc15a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:39466/0 (socket says 172.21.15.119:39466) 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcbfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fcc1444b0 0x7f2fcc13c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44378/0 (socket says 172.21.15.119:44378) 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 -- 172.21.15.119:0/3469602816 learned_addr learned my addr 172.21.15.119:0/3469602816 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcaffd700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc1440d0 0x7f2fcc1f04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 -- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc1440d0 msgr2=0x7f2fcc1f04b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc1440d0 0x7f2fcc1f04b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 -- 172.21.15.119:0/3469602816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fcc1444b0 msgr2=0x7f2fcc13c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fcc1444b0 0x7f2fcc13c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.189+0000 7f2fcb7fe700 1 -- 172.21.15.119:0/3469602816 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc4009a70 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fcb7fe700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fcc0cbb10 0x7f2fcc15a6c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000c520 tx=0x7f2fc000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:04.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fb3fff700 1 -- 172.21.15.119:0/3469602816 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00052d0 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fb3fff700 1 -- 172.21.15.119:0/3469602816 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc0005450 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fcc15ca80 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fb3fff700 1 -- 172.21.15.119:0/3469602816 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc00038e0 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fcc13bc10 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fcc067760 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fb3fff700 1 -- 172.21.15.119:0/3469602816 <== mon.2 v2:172.21.15.145:3301/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2fc0003a40 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fb3fff700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2fb8066730 0x7f2fb8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.198 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.193+0000 7f2fb3fff700 1 -- 172.21.15.119:0/3469602816 <== mon.2 v2:172.21.15.145:3301/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f2fc00837c0 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.199 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.197+0000 7f2fcaffd700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2fb8066730 0x7f2fb8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.200 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.197+0000 7f2fcaffd700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2fb8066730 0x7f2fb8068bf0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc15e240 tx=0x7f2fbc001690 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:04.206 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.205+0000 7f2fb3fff700 1 -- 172.21.15.119:0/3469602816 <== mon.2 v2:172.21.15.145:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2fc0050db0 con 0x7f2fcc0cbb10 2024-09-13T16:37:04.372 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.369+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2fcc0cb970 con 0x7f2fb8066730 2024-09-13T16:37:04.377 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.373+0000 7f2fb3fff700 1 -- 172.21.15.119:0/3469602816 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+49032 (secure 0 0 0) 0x7f2fcc0cb970 con 0x7f2fb8066730 2024-09-13T16:37:04.379 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:37:04.380 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2fb8066730 msgr2=0x7f2fb8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2fb8066730 0x7f2fb8068bf0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc15e240 tx=0x7f2fbc001690 comp rx=0 tx=0).stop 2024-09-13T16:37:04.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fcc0cbb10 msgr2=0x7f2fcc15a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fcc0cbb10 0x7f2fcc15a6c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2fc000c520 tx=0x7f2fc000c830 comp rx=0 tx=0).stop 2024-09-13T16:37:04.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 shutdown_connections 2024-09-13T16:37:04.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2fb8066730 0x7f2fb8068bf0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2fcc1444b0 0x7f2fcc13c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2fcc1440d0 0x7f2fcc1f04b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 --2- 172.21.15.119:0/3469602816 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2fcc0cbb10 0x7f2fcc15a6c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 >> 172.21.15.119:0/3469602816 conn(0x7f2fcc06a350 msgr2=0x7f2fcc15e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:04.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 shutdown_connections 2024-09-13T16:37:04.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.377+0000 7f2fd20af700 1 -- 172.21.15.119:0/3469602816 wait complete. 2024-09-13T16:37:04.383 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:37:04.395 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":481,"stamp":"2024-09-13T16:37:03.674820+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":279056,"kb_used_data":30132,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754695664,"statfs":{"total":773094113280,"available":772808359936,"internally_reserved":0,"allocated":30855168,"data_stored":23906615,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8390726,"num_objects":-30,"num_object_clones":0,"num_object_copies":-60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-30,"num_whiteouts":0,"num_read":-258,"num_read_kb":-262,"num_write":-174,"num_write_kb":-8293,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-17,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001861"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":261,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-13T16:36:42.206130+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:36:42.206130+0000","last_peered":"2024-09-13T16:36:42.206130+0000","last_clean":"2024-09-13T16:36:42.206130+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:36:42.206130+0000","last_undegraded":"2024-09-13T16:36:42.206130+0000","last_fullsized":"2024-09-13T16:36:42.206130+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542289,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33280,"kb_used_data":2164,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338560,"statfs":{"total":96636764160,"available":96602685440,"internally_reserved":0,"allocated":2215936,"data_stored":1335026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.80300000000000005,"15min":0.83699999999999997},"min":{"1min":0.51800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.198,"5min":1.262,"15min":1.357},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.52000000000000002,"15min":0.47699999999999998},"max":{"1min":1.1759999999999999,"5min":1.214,"15min":1.3460000000000001},"last":1.0209999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.56000000000000005,"15min":0.501},"max":{"1min":1.115,"5min":1.212,"15min":1.569},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.89100000000000001,"15min":0.86499999999999999},"min":{"1min":0.63600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.115,"5min":1.2290000000000001,"15min":1.4219999999999999},"last":0.876}]},{"osd":2,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.874},"min":{"1min":0.56699999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.113,"5min":1.1919999999999999,"15min":1.242},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.89100000000000001,"15min":0.87},"min":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.53900000000000003},"max":{"1min":1.238,"5min":1.24,"15min":1.339},"last":1.044}]},{"osd":3,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91300000000000003,"15min":0.95699999999999996},"min":{"1min":0.65300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3,"5min":1.3,"15min":1.476},"last":0.996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.998,"15min":1},"min":{"1min":0.73599999999999999,"5min":0.69599999999999995,"15min":0.65700000000000003},"max":{"1min":1.28,"5min":1.2969999999999999,"15min":1.3939999999999999},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.878,"15min":0.874},"min":{"1min":0.71399999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.089,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.94399999999999995,"15min":1.006},"min":{"1min":0.73299999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.216,"5min":1.381,"15min":1.589},"last":0.68799999999999994}]},{"osd":5,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.98199999999999998,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2230000000000001,"5min":1.444,"15min":1.444},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0720000000000001,"15min":1.091},"min":{"1min":0.99399999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.278,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0369999999999999,"15min":1.0529999999999999},"min":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2330000000000001,"5min":1.72,"15min":1.72},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.044,"15min":1.1080000000000001},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.375,"5min":1.542,"15min":1.6779999999999999},"last":0.90500000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542288,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36780,"kb_used_data":5664,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335060,"statfs":{"total":96636764160,"available":96599101440,"internally_reserved":0,"allocated":5799936,"data_stored":4938761,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.84499999999999997,"15min":0.84099999999999997},"min":{"1min":0.53300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.85799999999999998,"15min":0.878},"min":{"1min":0.55800000000000005,"5min":0.309,"15min":0.309},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.4410000000000001},"last":1.129}]},{"osd":1,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":0.84599999999999997},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0509999999999999,"5min":1.153,"15min":1.2030000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.88200000000000001,"15min":0.90800000000000003},"min":{"1min":0.57699999999999996,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1599999999999999,"5min":1.411,"15min":1.411},"last":0.65300000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.86299999999999999,"15min":0.86899999999999999},"min":{"1min":0.57899999999999996,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.262,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.93899999999999995},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.874},"min":{"1min":0.53000000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0940000000000001,"5min":1.399,"15min":1.399},"last":0.747}]},{"osd":3,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.92100000000000004,"15min":0.94899999999999995},"min":{"1min":0.58099999999999996,"5min":0.379,"15min":0.379},"max":{"1min":1.1870000000000001,"5min":1.544,"15min":1.544},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.97499999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.21,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":1.1859999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95599999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1599999999999999,"5min":1.508,"15min":1.508},"last":1.0069999999999999},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.65500000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.343,"5min":1.55,"15min":1.6140000000000001},"last":0.82699999999999996}]},{"osd":5,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.92300000000000004,"15min":0.93100000000000005},"min":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.214,"5min":1.339,"15min":1.339},"last":0.97499999999999998},{"interface":"front","average":{"1min":1.052,"5min":1.0580000000000001,"15min":1.089},"min":{"1min":0.70299999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2829999999999999,"5min":1.615,"15min":1.6439999999999999},"last":1.1020000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.048,"15min":1.087},"min":{"1min":0.72799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.522},"last":1.04},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.052,"15min":1.0589999999999999},"min":{"1min":0.68899999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.427},"last":1.0649999999999999}]}]},{"osd":5,"up_from":14,"seq":60129542290,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":5804032,"data_stored":4938339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.58299999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.1899999999999999,"15min":1.411},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.83399999999999996,"15min":0.83899999999999997},"min":{"1min":0.53100000000000003,"5min":0.497,"15min":0.41699999999999998},"max":{"1min":1.046,"5min":1.3029999999999999,"15min":1.675},"last":0.89200000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.60399999999999998,"5min":0.51900000000000002,"15min":0.46600000000000003},"max":{"1min":1.0840000000000001,"5min":1.2190000000000001,"15min":1.4079999999999999},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83199999999999996,"15min":0.83899999999999997},"min":{"1min":0.61299999999999999,"5min":0.53700000000000003,"15min":0.36499999999999999},"max":{"1min":1.085,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.61399999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.90400000000000003,"15min":0.91600000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.625},"last":0.93000000000000005},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.97599999999999998,"15min":0.97099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.159,"5min":1.331,"15min":1.552},"last":0.97099999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90500000000000003,"15min":0.879},"min":{"1min":0.57199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.23,"5min":1.23,"15min":1.3919999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.88600000000000001,"15min":0.94199999999999995},"min":{"1min":0.70199999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.286,"5min":1.2929999999999999,"15min":3.8460000000000001},"last":0.80600000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88200000000000001,"15min":0.96799999999999997},"min":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.56999999999999995},"max":{"1min":1.2030000000000001,"5min":1.3540000000000001,"15min":3.6869999999999998},"last":1.103},{"interface":"front","average":{"1min":1.073,"5min":1.0669999999999999,"15min":1.0640000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.57499999999999996},"max":{"1min":1.331,"5min":1.365,"15min":1.601},"last":1.0349999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.91800000000000004,"15min":0.94599999999999995},"min":{"1min":0.64300000000000002,"5min":0.61899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0069999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.995,"15min":1.0389999999999999},"min":{"1min":0.65800000000000003,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.1499999999999999,"5min":1.3740000000000001,"15min":2.21},"last":1.181}]},{"osd":7,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.008,"15min":1.0269999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.45500000000000002},"max":{"1min":1.4039999999999999,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0429999999999999,"15min":1.1220000000000001},"min":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":0.68799999999999994},"max":{"1min":1.556,"5min":1.556,"15min":3.9830000000000001},"last":1.079}]}]},{"osd":4,"up_from":14,"seq":60129542288,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36796,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335044,"statfs":{"total":96636764160,"available":96599085056,"internally_reserved":0,"allocated":5816320,"data_stored":4939494,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86099999999999999,"15min":0.84499999999999997},"min":{"1min":0.60999999999999999,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.206,"5min":1.21,"15min":1.411},"last":1.254},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.56499999999999995,"5min":0.54900000000000004,"15min":0.44600000000000001},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.899},"last":1.014}]},{"osd":1,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88400000000000001,"15min":0.90700000000000003},"min":{"1min":0.59099999999999997,"5min":0.53800000000000003,"15min":0.46200000000000002},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":3.085},"last":0.84399999999999997},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86499999999999999,"15min":0.90000000000000002},"min":{"1min":0.54200000000000004,"5min":0.52800000000000002,"15min":0.497},"max":{"1min":1.169,"5min":1.27,"15min":1.5720000000000001},"last":1.196}]},{"osd":2,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.874,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.433},"max":{"1min":1.258,"5min":1.258,"15min":1.4419999999999999},"last":1.1559999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.90300000000000002,"15min":0.94799999999999995},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3220000000000001,"5min":1.3640000000000001,"15min":3.2160000000000002},"last":0.91700000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.93600000000000005},"min":{"1min":0.65900000000000003,"5min":0.60399999999999998,"15min":0.47999999999999998},"max":{"1min":1.2749999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.875},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93200000000000005,"15min":0.94199999999999995},"min":{"1min":0.62,"5min":0.61499999999999999,"15min":0.52200000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.577},"last":1.044}]},{"osd":5,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91000000000000003,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.49299999999999999},"max":{"1min":1.2390000000000001,"5min":1.252,"15min":2.996},"last":1.075},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98199999999999998,"15min":1.0589999999999999},"min":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.495},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":2.8860000000000001},"last":1.0960000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.66700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.234,"5min":1.522,"15min":1.6499999999999999},"last":0.80800000000000005},{"interface":"front","average":{"1min":1.085,"5min":1.0700000000000001,"15min":1.099},"min":{"1min":0.77200000000000002,"5min":0.63600000000000001,"15min":0.55500000000000005},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.1469999999999998},"last":1.127}]},{"osd":7,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.81499999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":2.1230000000000002},"last":0.96899999999999997},{"interface":"front","average":{"1min":1.105,"5min":1.079,"15min":1.0980000000000001},"min":{"1min":0.78300000000000003,"5min":0.72599999999999998,"15min":0.67000000000000004},"max":{"1min":1.413,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":0.93899999999999995}]}]},{"osd":3,"up_from":14,"seq":60129542289,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32696,"kb_used_data":1580,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339144,"statfs":{"total":96636764160,"available":96603283456,"internally_reserved":0,"allocated":1617920,"data_stored":745212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69999999999999996,"15min":0.67800000000000005},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.64900000000000002}]},{"osd":1,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.64900000000000002},"min":{"1min":0.433,"5min":0.40799999999999997,"15min":0.35499999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.68899999999999995},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.115,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.79600000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71099999999999997,"15min":0.70199999999999996},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.131,"5min":1.133,"15min":1.27},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.80200000000000005,"15min":0.77900000000000003},"min":{"1min":0.57299999999999995,"5min":0.379,"15min":0.379},"max":{"1min":1.1299999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":0.73999999999999999}]},{"osd":4,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.92400000000000004},"min":{"1min":0.74199999999999999,"5min":0.69999999999999996,"15min":0.60499999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.8460000000000001},"last":0.90600000000000003},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93999999999999995,"15min":0.94899999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.59399999999999997},"max":{"1min":1.256,"5min":1.3360000000000001,"15min":1.782},"last":1.038}]},{"osd":5,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.59499999999999997},"max":{"1min":1.1990000000000001,"5min":1.397,"15min":1.6299999999999999},"last":0.93100000000000005},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.0269999999999999,"15min":1.0600000000000001},"min":{"1min":0.76300000000000001,"5min":0.66900000000000004,"15min":0.56899999999999995},"max":{"1min":1.3049999999999999,"5min":1.581,"15min":1.8140000000000001},"last":1.0109999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0640000000000001,"15min":1.0900000000000001},"min":{"1min":0.76400000000000001,"5min":0.61599999999999999,"15min":0.58699999999999997},"max":{"1min":1.3380000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.034,"5min":1.0669999999999999,"15min":1.0469999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"max":{"1min":1.54,"5min":1.7030000000000001,"15min":1.95},"last":1.1970000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.093,"15min":1.0820000000000001},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.71399999999999997},"max":{"1min":1.573,"5min":1.657,"15min":1.889},"last":1.2509999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.294,"5min":1.6240000000000001,"15min":1.756},"last":1.141}]}]},{"osd":2,"up_from":14,"seq":60129542290,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32684,"kb_used_data":1568,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339156,"statfs":{"total":96636764160,"available":96603295744,"internally_reserved":0,"allocated":1605632,"data_stored":743880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.62,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.93700000000000006,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.169,"15min":1.169},"last":0.69599999999999995}]},{"osd":1,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.88600000000000001,"5min":1.147,"15min":1.147},"last":0.79600000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72099999999999997,"15min":0.68999999999999995},"min":{"1min":0.45200000000000001,"5min":0.39000000000000001,"15min":0.38100000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.746},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.57199999999999995,"5min":0.37,"15min":0.37},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2330000000000001},"last":0.93700000000000006}]},{"osd":4,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93600000000000005,"15min":0.94399999999999995},"min":{"1min":0.69799999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.171,"5min":1.448,"15min":1.5149999999999999},"last":1.014},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.96799999999999997},"min":{"1min":0.69399999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.8799999999999999},"last":1.0629999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95299999999999996,"15min":0.96899999999999997},"min":{"1min":0.63500000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.375,"5min":1.413,"15min":1.4750000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.96799999999999997,"15min":0.96399999999999997},"min":{"1min":0.66500000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.284,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.2529999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.74299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":2.024},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.044,"15min":1.052},"min":{"1min":0.82799999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.452,"15min":1.6359999999999999},"last":1.1020000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0529999999999999,"15min":1.0620000000000001},"min":{"1min":0.76400000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.27,"5min":1.357,"15min":1.4810000000000001},"last":1.202},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0569999999999999,"15min":1.054},"min":{"1min":0.56699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.482,"5min":1.587,"15min":1.587},"last":0.98399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542289,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36784,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94335056,"statfs":{"total":96636764160,"available":96599097344,"internally_reserved":0,"allocated":5804032,"data_stored":4938206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.23300000000000001},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":1.0880000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.751,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38800000000000001},"max":{"1min":1.0349999999999999,"5min":1.089,"15min":1.2010000000000001},"last":0.90400000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67900000000000005,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.38300000000000001,"15min":0.26700000000000002},"max":{"1min":0.94999999999999996,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73699999999999999,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.29699999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.458},"last":0.95799999999999996}]},{"osd":3,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.75800000000000001,"15min":0.72899999999999998},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":1.052,"5min":1.1359999999999999,"15min":1.4019999999999999},"last":0.82599999999999996},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.39600000000000002},"max":{"1min":0.89100000000000001,"5min":1.175,"15min":1.403},"last":0.77600000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.95699999999999996,"15min":0.93100000000000005},"min":{"1min":0.75600000000000001,"5min":0.67800000000000005,"15min":0.505},"max":{"1min":1.212,"5min":1.2190000000000001,"15min":1.546},"last":1.0429999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.51700000000000002},"max":{"1min":1.1459999999999999,"5min":1.1559999999999999,"15min":2.2040000000000002},"last":0.93500000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96999999999999997,"15min":0.95999999999999996},"min":{"1min":0.70299999999999996,"5min":0.68799999999999994,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.3819999999999999,"15min":1.389},"last":1.1259999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.93500000000000005,"15min":0.94599999999999995},"min":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.52000000000000002},"max":{"1min":1.1759999999999999,"5min":1.238,"15min":2.3029999999999999},"last":0.98699999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.01,"15min":1.0049999999999999},"min":{"1min":0.77700000000000002,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.3009999999999999,"5min":1.4330000000000001,"15min":1.613},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.056,"15min":1.03},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.59299999999999997},"max":{"1min":1.278,"5min":1.667,"15min":1.698},"last":1.278}]},{"osd":7,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.0129999999999999,"15min":1.008},"min":{"1min":0.74199999999999999,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":1.2609999999999999,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0649999999999999,"15min":1.0569999999999999},"min":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.59299999999999997},"max":{"1min":1.2290000000000001,"5min":1.3779999999999999,"15min":1.7689999999999999},"last":1.222}]}]},{"osd":0,"up_from":14,"seq":60129542289,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33252,"kb_used_data":2140,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338588,"statfs":{"total":96636764160,"available":96602714112,"internally_reserved":0,"allocated":2191360,"data_stored":1327697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.52700000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68600000000000005,"15min":0.67300000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.101,"5min":1.1559999999999999,"15min":1.157},"last":0.95499999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70999999999999996,"15min":0.71599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"max":{"1min":0.94099999999999995,"5min":1.125,"15min":1.202},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39800000000000002},"max":{"1min":1.0249999999999999,"5min":1.0780000000000001,"15min":1.2629999999999999},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.312},"max":{"1min":0.92800000000000005,"5min":1.1259999999999999,"15min":1.4830000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.85499999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.097,"5min":1.155,"15min":1.6100000000000001},"last":0.77300000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.95999999999999996,"15min":0.95799999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57699999999999996},"max":{"1min":1.3080000000000001,"5min":1.3899999999999999,"15min":1.486},"last":1.107},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.95299999999999996},"min":{"1min":0.70399999999999996,"5min":0.64700000000000002,"15min":0.59999999999999998},"max":{"1min":0.97999999999999998,"5min":1.274,"15min":1.704},"last":1.0640000000000001}]},{"osd":5,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":0.996},"min":{"1min":0.80500000000000005,"5min":0.67100000000000004,"15min":0.59599999999999997},"max":{"1min":1.3720000000000001,"5min":1.532,"15min":1.575},"last":1.091},{"interface":"front","average":{"1min":0.996,"5min":0.99199999999999999,"15min":0.98999999999999999},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.443},"max":{"1min":1.399,"5min":1.742,"15min":1.742},"last":1.252}]},{"osd":6,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.97599999999999998},"min":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.628},"max":{"1min":1.1020000000000001,"5min":1.4099999999999999,"15min":1.944},"last":0.91400000000000003},{"interface":"front","average":{"1min":1.115,"5min":1.0940000000000001,"15min":1.075},"min":{"1min":0.81100000000000005,"5min":0.77300000000000002,"15min":0.58899999999999997},"max":{"1min":1.46,"5min":1.53,"15min":1.5780000000000001},"last":0.98999999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.095,"15min":1.0720000000000001},"min":{"1min":0.86699999999999999,"5min":0.76300000000000001,"15min":0.61699999999999999},"max":{"1min":1.3759999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.169},{"interface":"front","average":{"1min":1.042,"5min":1.073,"15min":1.071},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.65400000000000003},"max":{"1min":1.266,"5min":1.381,"15min":2.0139999999999998},"last":1.1299999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:37:04.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-13T16:37:04.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fb2fe700 1 Processor -- start 2024-09-13T16:37:04.512 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fb2fe700 1 -- start start 2024-09-13T16:37:04.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fb2fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa3f406e6a0 0x7fa3f406ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fb2fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406be70 0x7fa3f40b5430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fb2fe700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3f406cff0 con 0x7fa3f40b4160 2024-09-13T16:37:04.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fb2fe700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3f406d480 con 0x7fa3f406e6a0 2024-09-13T16:37:04.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fb2fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f406efc0 con 0x7fa3f406be70 2024-09-13T16:37:04.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3f9afb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406be70 0x7fa3f40b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fa2fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fa3f406e6a0 0x7fa3f406ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3f9afb700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406be70 0x7fa3f40b5430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:39468/0 (socket says 172.21.15.119:39468) 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3f9afb700 1 -- 172.21.15.119:0/261769362 learned_addr learned my addr 172.21.15.119:0/261769362 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3faafd700 1 --1- 172.21.15.119:0/261769362 >> v1:172.21.15.145:6789/0 conn(0x7fa3f40b4160 0x7fa3f40b4610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:34186/0 (socket says 172.21.15.119:34186) 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fa2fc700 1 -- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406be70 msgr2=0x7fa3f40b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fa2fc700 1 --2- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406be70 0x7fa3f40b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fa2fc700 1 -- 172.21.15.119:0/261769362 >> v1:172.21.15.145:6789/0 conn(0x7fa3f40b4160 legacy=0x7fa3f40b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.509+0000 7fa3fa2fc700 1 -- 172.21.15.119:0/261769362 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f406c3e0 con 0x7fa3f406e6a0 2024-09-13T16:37:04.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3f9afb700 1 --2- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406be70 0x7fa3f40b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:04.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fa2fc700 1 --2- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406e6a0 0x7fa3f406ea80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009880 tx=0x7fa3e4009b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=878743a364b88e53 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3f92fa700 1 -- 172.21.15.119:0/261769362 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4011830 con 0x7fa3f406e6a0 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3f92fa700 1 -- 172.21.15.119:0/261769362 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e4011990 con 0x7fa3f406e6a0 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3f92fa700 1 -- 172.21.15.119:0/261769362 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e4011d20 con 0x7fa3f406e6a0 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406e6a0 msgr2=0x7fa3f406ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406e6a0 0x7fa3f406ea80 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fa3e4009880 tx=0x7fa3e4009b90 comp rx=0 tx=0).stop 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/261769362 shutdown_connections 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406be70 0x7fa3f40b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/261769362 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406e6a0 0x7fa3f406ea80 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/261769362 >> 172.21.15.119:0/261769362 conn(0x7fa3f40b5d20 msgr2=0x7fa3f40b6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/261769362 shutdown_connections 2024-09-13T16:37:04.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/261769362 wait complete. 2024-09-13T16:37:04.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 Processor -- start 2024-09-13T16:37:04.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.513+0000 7fa3fb2fe700 1 -- start start 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406be70 0x7fa3f40c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406e6a0 0x7fa3f40cbbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3f40b4160 0x7fa3f40cc120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f40ccfe0 con 0x7fa3f40b4160 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f40cc740 con 0x7fa3f406be70 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3f40cca20 con 0x7fa3f406e6a0 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3f40b4160 0x7fa3f40cc120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3f40b4160 0x7fa3f40cc120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44390/0 (socket says 172.21.15.119:44390) 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 -- 172.21.15.119:0/675062319 learned_addr learned my addr 172.21.15.119:0/675062319 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3f9afb700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406e6a0 0x7fa3f40cbbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fa2fc700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406be70 0x7fa3f40c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 -- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406e6a0 msgr2=0x7fa3f40cbbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406e6a0 0x7fa3f40cbbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 -- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406be70 msgr2=0x7fa3f40c9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406be70 0x7fa3f40c9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 -- 172.21.15.119:0/675062319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f0009570 con 0x7fa3f40b4160 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fa2fc700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406be70 0x7fa3f40c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3faafd700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3f40b4160 0x7fa3f40cc120 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00cbc0 tx=0x7fa3ec00ced0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3eaffd700 1 -- 172.21.15.119:0/675062319 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00aaa0 con 0x7fa3f40b4160 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3eaffd700 1 -- 172.21.15.119:0/675062319 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec00ac00 con 0x7fa3f40b4160 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e40094b0 con 0x7fa3f40b4160 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3f4152240 con 0x7fa3f40b4160 2024-09-13T16:37:04.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.517+0000 7fa3eaffd700 1 -- 172.21.15.119:0/675062319 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec007070 con 0x7fa3f40b4160 2024-09-13T16:37:04.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.521+0000 7fa3eaffd700 1 -- 172.21.15.119:0/675062319 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fa3ec0071d0 con 0x7fa3f40b4160 2024-09-13T16:37:04.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.521+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3f4074560 con 0x7fa3f40b4160 2024-09-13T16:37:04.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.521+0000 7fa3eaffd700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa3d8066730 0x7fa3d8068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:04.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.521+0000 7fa3eaffd700 1 -- 172.21.15.119:0/675062319 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fa3ec0839b0 con 0x7fa3f40b4160 2024-09-13T16:37:04.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.521+0000 7fa3fa2fc700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa3d8066730 0x7fa3d8068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:04.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.521+0000 7fa3fa2fc700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa3d8066730 0x7fa3d8068bf0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40090b0 tx=0x7fa3e4008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:04.528 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.525+0000 7fa3eaffd700 1 -- 172.21.15.119:0/675062319 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fa3ec050dc0 con 0x7fa3f40b4160 2024-09-13T16:37:04.701 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.697+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa3f41520a0 con 0x7fa3f40b4160 2024-09-13T16:37:04.703 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3eaffd700 1 -- 172.21.15.119:0/675062319 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+13527 (secure 0 0 0) 0x7fa3f41520a0 con 0x7fa3f40b4160 2024-09-13T16:37:04.703 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:37:04.703 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":131,"fsid":"01de39e4-2965-4196-9df7-d7f8aeef36df","created":"2024-09-13T16:25:00.053707+0000","modified":"2024-09-13T16:37:01.798329+0000","last_up_change":"2024-09-13T16:25:10.404143+0000","last_in_change":"2024-09-13T16:25:02.142889+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-13T16:25:11.481827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"bfde5ac7-b275-4f11-a763-08c219fbfcdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6801","nonce":2655437631}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6802","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6803","nonce":2655437631}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6806","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6807","nonce":2655437631}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":2655437631},{"type":"v1","addr":"172.21.15.119:6805","nonce":2655437631}]},"public_addr":"172.21.15.119:6801/2655437631","cluster_addr":"172.21.15.119:6803/2655437631","heartbeat_back_addr":"172.21.15.119:6807/2655437631","heartbeat_front_addr":"172.21.15.119:6805/2655437631","state":["exists","up"]},{"osd":1,"uuid":"a2427b3c-c6b9-4481-9816-e6b9a0a8ac96","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6817","nonce":1495422598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6818","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6819","nonce":1495422598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6822","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6823","nonce":1495422598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":1495422598},{"type":"v1","addr":"172.21.15.119:6821","nonce":1495422598}]},"public_addr":"172.21.15.119:6817/1495422598","cluster_addr":"172.21.15.119:6819/1495422598","heartbeat_back_addr":"172.21.15.119:6823/1495422598","heartbeat_front_addr":"172.21.15.119:6821/1495422598","state":["exists","up"]},{"osd":2,"uuid":"a6fbd8ac-061c-4d6a-a81e-bd888adca00d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6825","nonce":3093261484}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6826","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6827","nonce":3093261484}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6830","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6831","nonce":3093261484}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":3093261484},{"type":"v1","addr":"172.21.15.119:6829","nonce":3093261484}]},"public_addr":"172.21.15.119:6825/3093261484","cluster_addr":"172.21.15.119:6827/3093261484","heartbeat_back_addr":"172.21.15.119:6831/3093261484","heartbeat_front_addr":"172.21.15.119:6829/3093261484","state":["exists","up"]},{"osd":3,"uuid":"0ab479b1-9888-4ffd-b4f3-c4c2873accb1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6809","nonce":3803418857}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6810","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6811","nonce":3803418857}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6814","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6815","nonce":3803418857}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":3803418857},{"type":"v1","addr":"172.21.15.119:6813","nonce":3803418857}]},"public_addr":"172.21.15.119:6809/3803418857","cluster_addr":"172.21.15.119:6811/3803418857","heartbeat_back_addr":"172.21.15.119:6815/3803418857","heartbeat_front_addr":"172.21.15.119:6813/3803418857","state":["exists","up"]},{"osd":4,"uuid":"d0b1aaff-0f4b-4f31-9a0c-520e64952b8e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6817","nonce":4125431312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6819","nonce":4125431312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6823","nonce":4125431312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":4125431312},{"type":"v1","addr":"172.21.15.145:6821","nonce":4125431312}]},"public_addr":"172.21.15.145:6817/4125431312","cluster_addr":"172.21.15.145:6819/4125431312","heartbeat_back_addr":"172.21.15.145:6823/4125431312","heartbeat_front_addr":"172.21.15.145:6821/4125431312","state":["exists","up"]},{"osd":5,"uuid":"651f46b8-548f-48f0-a80d-1e52f8bfe475","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6825","nonce":4068956621}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6827","nonce":4068956621}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6831","nonce":4068956621}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":4068956621},{"type":"v1","addr":"172.21.15.145:6829","nonce":4068956621}]},"public_addr":"172.21.15.145:6825/4068956621","cluster_addr":"172.21.15.145:6827/4068956621","heartbeat_back_addr":"172.21.15.145:6831/4068956621","heartbeat_front_addr":"172.21.15.145:6829/4068956621","state":["exists","up"]},{"osd":6,"uuid":"c1438d19-cd27-4b99-ac1d-e07136e7ad46","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6801","nonce":3152548227}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6803","nonce":3152548227}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6807","nonce":3152548227}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3152548227},{"type":"v1","addr":"172.21.15.145:6805","nonce":3152548227}]},"public_addr":"172.21.15.145:6801/3152548227","cluster_addr":"172.21.15.145:6803/3152548227","heartbeat_back_addr":"172.21.15.145:6807/3152548227","heartbeat_front_addr":"172.21.15.145:6805/3152548227","state":["exists","up"]},{"osd":7,"uuid":"fc8cd1d4-48c7-4624-9510-d7e0971ea575","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":124,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6809","nonce":1049327118}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6811","nonce":1049327118}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6815","nonce":1049327118}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":1049327118},{"type":"v1","addr":"172.21.15.145:6813","nonce":1049327118}]},"public_addr":"172.21.15.145:6809/1049327118","cluster_addr":"172.21.15.145:6811/1049327118","heartbeat_back_addr":"172.21.15.145:6815/1049327118","heartbeat_front_addr":"172.21.15.145:6813/1049327118","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.155911+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:06.913471+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.621013+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.419620+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.476664+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.605889+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.390924+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-13T16:25:07.351867+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.145:6835/2905269684":"2024-09-14T16:36:58.779368+0000","172.21.15.145:6834/2905269684":"2024-09-14T16:36:58.779368+0000","172.21.15.145:6835/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6835/3901706437":"2024-09-14T16:36:21.631788+0000","172.21.15.145:6834/3898509964":"2024-09-14T16:36:42.101563+0000","172.21.15.145:6834/3901706437":"2024-09-14T16:36:21.631788+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-13T16:37:04.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa3d8066730 msgr2=0x7fa3d8068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa3d8066730 0x7fa3d8068bf0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa3e40090b0 tx=0x7fa3e4008030 comp rx=0 tx=0).stop 2024-09-13T16:37:04.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3f40b4160 msgr2=0x7fa3f40cc120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:04.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3f40b4160 0x7fa3f40cc120 secure :-1 s=READY pgs=2026 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00cbc0 tx=0x7fa3ec00ced0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.706 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 shutdown_connections 2024-09-13T16:37:04.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fa3d8066730 0x7fa3d8068bf0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa3f40b4160 0x7fa3f40cc120 unknown :-1 s=CLOSED pgs=2026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fa3f406e6a0 0x7fa3f40cbbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 --2- 172.21.15.119:0/675062319 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fa3f406be70 0x7fa3f40c9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:04.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 >> 172.21.15.119:0/675062319 conn(0x7fa3f40b5d20 msgr2=0x7fa3f40b4570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:04.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 shutdown_connections 2024-09-13T16:37:04.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:04.701+0000 7fa3fb2fe700 1 -- 172.21.15.119:0/675062319 wait complete. 2024-09-13T16:37:05.719 INFO:tasks.ceph:Scrubbing osd.0 2024-09-13T16:37:05.719 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.849+0000 7f485a516700 1 Processor -- start 2024-09-13T16:37:05.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.849+0000 7f485a516700 1 -- start start 2024-09-13T16:37:05.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.849+0000 7f485a516700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f485415a730 0x7f48540706c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:05.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.849+0000 7f485a516700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f48540cb300 0x7f48540cb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:05.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.849+0000 7f485a516700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4854160770 con 0x7f48540ac8c0 2024-09-13T16:37:05.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.849+0000 7f485a516700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4854160560 con 0x7f485415a730 2024-09-13T16:37:05.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.849+0000 7f485a516700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48540cbcd0 con 0x7f48540cb300 2024-09-13T16:37:05.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f485415a730 0x7f48540706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:05.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4858d13700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f48540cb300 0x7f48540cb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f485415a730 0x7f48540706c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44392/0 (socket says 172.21.15.119:44392) 2024-09-13T16:37:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 -- 172.21.15.119:0/2332868951 learned_addr learned my addr 172.21.15.119:0/2332868951 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 -- 172.21.15.119:0/2332868951 >> v2:172.21.15.145:3300/0 conn(0x7f48540cb300 msgr2=0x7f48540cb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 --2- 172.21.15.119:0/2332868951 >> v2:172.21.15.145:3300/0 conn(0x7f48540cb300 0x7f48540cb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 -- 172.21.15.119:0/2332868951 >> v1:172.21.15.145:6789/0 conn(0x7f48540ac8c0 legacy=0x7f48540acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 -- 172.21.15.119:0/2332868951 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4854160a50 con 0x7f485415a730 2024-09-13T16:37:05.856 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f4859514700 1 --2- 172.21.15.119:0/2332868951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f485415a730 0x7f48540706c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f485406b330 tx=0x7f4844009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a82f12ccb442b44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:05.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f484bfff700 1 -- 172.21.15.119:0/2332868951 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844011830 con 0x7f485415a730 2024-09-13T16:37:05.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f484bfff700 1 -- 172.21.15.119:0/2332868951 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4844011990 con 0x7f485415a730 2024-09-13T16:37:05.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f484bfff700 1 -- 172.21.15.119:0/2332868951 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844011d20 con 0x7f485415a730 2024-09-13T16:37:05.857 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 -- 172.21.15.119:0/2332868951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f485415a730 msgr2=0x7f48540706c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 --2- 172.21.15.119:0/2332868951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f485415a730 0x7f48540706c0 secure :-1 s=READY pgs=2027 cs=0 l=1 rev1=1 crypto rx=0x7f485406b330 tx=0x7f4844009bf0 comp rx=0 tx=0).stop 2024-09-13T16:37:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 -- 172.21.15.119:0/2332868951 shutdown_connections 2024-09-13T16:37:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 --2- 172.21.15.119:0/2332868951 >> v2:172.21.15.145:3300/0 conn(0x7f48540cb300 0x7f48540cb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 --2- 172.21.15.119:0/2332868951 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f485415a730 0x7f48540706c0 unknown :-1 s=CLOSED pgs=2027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 -- 172.21.15.119:0/2332868951 >> 172.21.15.119:0/2332868951 conn(0x7f485406a350 msgr2=0x7f485406a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 -- 172.21.15.119:0/2332868951 shutdown_connections 2024-09-13T16:37:05.858 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.853+0000 7f485a516700 1 -- 172.21.15.119:0/2332868951 wait complete. 2024-09-13T16:37:05.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 Processor -- start 2024-09-13T16:37:05.860 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 -- start start 2024-09-13T16:37:05.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48540ac8c0 0x7f485415c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:05.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48540cb300 0x7f485414eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:05.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f485415a730 0x7f48540b0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48540c4880 con 0x7f48540ac8c0 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48540c4b60 con 0x7f485415a730 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f485a516700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48540c4e40 con 0x7f48540cb300 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f4859514700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48540ac8c0 0x7f485415c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f4859514700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48540ac8c0 0x7f485415c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44396/0 (socket says 172.21.15.119:44396) 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.857+0000 7f4859514700 1 -- 172.21.15.119:0/3968407060 learned_addr learned my addr 172.21.15.119:0/3968407060 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4858d13700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48540cb300 0x7f485414eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859d15700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f485415a730 0x7f48540b0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859514700 1 -- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48540cb300 msgr2=0x7f485414eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859514700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f48540cb300 0x7f485414eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859514700 1 -- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f485415a730 msgr2=0x7f48540b0780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:05.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859514700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f485415a730 0x7f48540b0780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859514700 1 -- 172.21.15.119:0/3968407060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4844009570 con 0x7f48540ac8c0 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859d15700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f485415a730 0x7f48540b0780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4859514700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48540ac8c0 0x7f485415c910 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f4854079960 tx=0x7f4844004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844004550 con 0x7f48540ac8c0 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48540c5120 con 0x7f48540ac8c0 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f484401f070 con 0x7f48540ac8c0 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844003900 con 0x7f48540ac8c0 2024-09-13T16:37:05.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f485415ce50 con 0x7f48540ac8c0 2024-09-13T16:37:05.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f48541612f0 con 0x7f48540ac8c0 2024-09-13T16:37:05.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4844003a60 con 0x7f48540ac8c0 2024-09-13T16:37:05.866 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4849ffb700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4838066730 0x7f4838068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:05.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.861+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f4844021070 con 0x7f48540ac8c0 2024-09-13T16:37:05.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4849ffb700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f483806cb70 0x7f483806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:05.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f483806f680 con 0x7f483806cb70 2024-09-13T16:37:05.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4844087940 con 0x7f48540ac8c0 2024-09-13T16:37:05.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4858d13700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4838066730 0x7f4838068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:05.867 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4859d15700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f483806cb70 0x7f483806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:05.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4859d15700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f483806cb70 0x7f483806efb0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:05.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4858d13700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4838066730 0x7f4838068bf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4854159b00 tx=0x7f4850009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:05.868 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.865+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== osd.0 v2:172.21.15.119:6800/2655437631 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f483806f680 con 0x7f483806cb70 2024-09-13T16:37:05.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 --> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4854161010 con 0x7f483806cb70 2024-09-13T16:37:05.883 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f4849ffb700 1 -- 172.21.15.119:0/3968407060 <== osd.0 v2:172.21.15.119:6800/2655437631 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f484c003a20 con 0x7f483806cb70 2024-09-13T16:37:05.883 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:05.884 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:05.884 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f483806cb70 msgr2=0x7f483806efb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:6800/2655437631,v1:172.21.15.119:6801/2655437631] conn(0x7f483806cb70 0x7f483806efb0 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4838066730 msgr2=0x7f4838068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4838066730 0x7f4838068bf0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f4854159b00 tx=0x7f4850009240 comp rx=0 tx=0).stop 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48540ac8c0 msgr2=0x7f485415c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 --2- 172.21.15.119:0/3968407060 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f48540ac8c0 0x7f485415c910 secure :-1 s=READY pgs=2028 cs=0 l=1 rev1=1 crypto rx=0x7f4854079960 tx=0x7f4844004770 comp rx=0 tx=0).stop 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f4859d15700 1 -- 172.21.15.119:0/3968407060 reap_dead start 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 shutdown_connections 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 >> 172.21.15.119:0/3968407060 conn(0x7f485406a350 msgr2=0x7f485415eb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 shutdown_connections 2024-09-13T16:37:05.885 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:05.881+0000 7f485a516700 1 -- 172.21.15.119:0/3968407060 wait complete. 2024-09-13T16:37:05.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-13T16:37:06.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.013+0000 7f664649c700 1 Processor -- start 2024-09-13T16:37:06.019 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 -- start start 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f66400adb30 0x7f66400cc060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f664006bb90 0x7f66400cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f66400cde70 0x7f664015e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664006cf60 con 0x7f66400adb30 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664006c350 con 0x7f66400cde70 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664006c630 con 0x7f664006bb90 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f66400adb30 0x7f66400cc060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f66400adb30 0x7f66400cc060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44404/0 (socket says 172.21.15.119:44404) 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f6645c9b700 1 --2- 172.21.15.119:0/524184170 >> v2:172.21.15.145:3300/0 conn(0x7f66400cde70 0x7f664015e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 -- 172.21.15.119:0/524184170 learned_addr learned my addr 172.21.15.119:0/524184170 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f6644c99700 1 --2- 172.21.15.119:0/524184170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f664006bb90 0x7f66400cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 -- 172.21.15.119:0/524184170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f664006bb90 msgr2=0x7f66400cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 --2- 172.21.15.119:0/524184170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f664006bb90 0x7f66400cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 -- 172.21.15.119:0/524184170 >> v2:172.21.15.145:3300/0 conn(0x7f66400cde70 msgr2=0x7f664015e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 --2- 172.21.15.119:0/524184170 >> v2:172.21.15.145:3300/0 conn(0x7f66400cde70 0x7f664015e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 -- 172.21.15.119:0/524184170 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664006c910 con 0x7f66400adb30 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f6645c9b700 1 --2- 172.21.15.119:0/524184170 >> v2:172.21.15.145:3300/0 conn(0x7f66400cde70 0x7f664015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:06.022 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664549a700 1 --2- 172.21.15.119:0/524184170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66400adb30 0x7f66400cc060 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f66400cb1e0 tx=0x7f663c009bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc1fb9273ceddc24 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f6637fff700 1 -- 172.21.15.119:0/524184170 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c011830 con 0x7f66400adb30 2024-09-13T16:37:06.023 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f6637fff700 1 -- 172.21.15.119:0/524184170 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c011990 con 0x7f66400adb30 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f6637fff700 1 -- 172.21.15.119:0/524184170 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c011d20 con 0x7f66400adb30 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.017+0000 7f664649c700 1 -- 172.21.15.119:0/524184170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66400adb30 msgr2=0x7f66400cc060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 --2- 172.21.15.119:0/524184170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66400adb30 0x7f66400cc060 secure :-1 s=READY pgs=2029 cs=0 l=1 rev1=1 crypto rx=0x7f66400cb1e0 tx=0x7f663c009bf0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 -- 172.21.15.119:0/524184170 shutdown_connections 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 --2- 172.21.15.119:0/524184170 >> v2:172.21.15.145:3300/0 conn(0x7f66400cde70 0x7f664015e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 --2- 172.21.15.119:0/524184170 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f664006bb90 0x7f66400cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 --2- 172.21.15.119:0/524184170 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f66400adb30 0x7f66400cc060 unknown :-1 s=CLOSED pgs=2029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 -- 172.21.15.119:0/524184170 >> 172.21.15.119:0/524184170 conn(0x7f664006a350 msgr2=0x7f664006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 -- 172.21.15.119:0/524184170 shutdown_connections 2024-09-13T16:37:06.024 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 -- 172.21.15.119:0/524184170 wait complete. 2024-09-13T16:37:06.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 Processor -- start 2024-09-13T16:37:06.025 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 -- start start 2024-09-13T16:37:06.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664006bb90 0x7f664013c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.027 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.021+0000 7f664649c700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66400adb30 0x7f66400ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664649c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66400cde70 0x7f664013e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664649c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66400af9e0 con 0x7f664006bb90 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664649c700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66400af6b0 con 0x7f66400adb30 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664649c700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66400aeb90 con 0x7f66400cde70 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664006bb90 0x7f664013c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664006bb90 0x7f664013c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44408/0 (socket says 172.21.15.119:44408) 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 -- 172.21.15.119:0/753181640 learned_addr learned my addr 172.21.15.119:0/753181640 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f6645c9b700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66400cde70 0x7f664013e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f6644c99700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66400adb30 0x7f66400ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 -- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66400cde70 msgr2=0x7f664013e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.028 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66400cde70 0x7f664013e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 -- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66400adb30 msgr2=0x7f66400ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66400adb30 0x7f66400ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.029 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 -- 172.21.15.119:0/753181640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6638009a70 con 0x7f664006bb90 2024-09-13T16:37:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664549a700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664006bb90 0x7f664013c540 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f663c000c00 tx=0x7f663c004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f6635ffb700 1 -- 172.21.15.119:0/753181640 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c004240 con 0x7f664006bb90 2024-09-13T16:37:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f6635ffb700 1 -- 172.21.15.119:0/753181640 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c01f070 con 0x7f664006bb90 2024-09-13T16:37:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f663c009570 con 0x7f664006bb90 2024-09-13T16:37:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f6635ffb700 1 -- 172.21.15.119:0/753181640 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c0045c0 con 0x7f664006bb90 2024-09-13T16:37:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66400af150 con 0x7f664006bb90 2024-09-13T16:37:06.030 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.025+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66400adf10 con 0x7f664006bb90 2024-09-13T16:37:06.031 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.029+0000 7f6635ffb700 1 -- 172.21.15.119:0/753181640 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f663c021030 con 0x7f664006bb90 2024-09-13T16:37:06.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.029+0000 7f6635ffb700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6624066730 0x7f6624068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.029+0000 7f6635ffb700 1 -- 172.21.15.119:0/753181640 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f663c085180 con 0x7f664006bb90 2024-09-13T16:37:06.032 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.029+0000 7f6644c99700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6624066730 0x7f6624068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.033 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.029+0000 7f6644c99700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6624066730 0x7f6624068bf0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f66400af640 tx=0x7f6630009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.040 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.037+0000 7f6635ffb700 1 -- 172.21.15.119:0/753181640 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f663c0525f0 con 0x7f664006bb90 2024-09-13T16:37:06.212 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.209+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f664006d3f0 con 0x7f6624066730 2024-09-13T16:37:06.213 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.209+0000 7f6635ffb700 1 -- 172.21.15.119:0/753181640 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6630011830 con 0x7f6624066730 2024-09-13T16:37:06.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6624066730 msgr2=0x7f6624068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6624066730 0x7f6624068bf0 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f66400af640 tx=0x7f6630009240 comp rx=0 tx=0).stop 2024-09-13T16:37:06.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664006bb90 msgr2=0x7f664013c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664006bb90 0x7f664013c540 secure :-1 s=READY pgs=2030 cs=0 l=1 rev1=1 crypto rx=0x7f663c000c00 tx=0x7f663c004770 comp rx=0 tx=0).stop 2024-09-13T16:37:06.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 shutdown_connections 2024-09-13T16:37:06.216 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f6624066730 0x7f6624068bf0 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f66400cde70 0x7f664013e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f66400adb30 0x7f66400ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 --2- 172.21.15.119:0/753181640 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f664006bb90 0x7f664013c540 unknown :-1 s=CLOSED pgs=2030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 >> 172.21.15.119:0/753181640 conn(0x7f664006a350 msgr2=0x7f6640066090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 shutdown_connections 2024-09-13T16:37:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.213+0000 7f664649c700 1 -- 172.21.15.119:0/753181640 wait complete. 2024-09-13T16:37:06.217 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 0 to deep-scrub 2024-09-13T16:37:06.228 INFO:tasks.ceph:Scrubbing osd.1 2024-09-13T16:37:06.228 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:06.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.341+0000 7f8da56a9700 1 Processor -- start 2024-09-13T16:37:06.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.341+0000 7f8da56a9700 1 -- start start 2024-09-13T16:37:06.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da00cb740 0x7f8da015b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da00b2d10 con 0x7f8da00cb300 2024-09-13T16:37:06.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8da00b1e20 con 0x7f8da015a7d0 2024-09-13T16:37:06.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da00ce1c0 con 0x7f8da00cb740 2024-09-13T16:37:06.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d977fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f8da00cb300 0x7f8da015abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41272/0 (socket says 172.21.15.119:41272) 2024-09-13T16:37:06.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d977fe700 1 -- 172.21.15.119:0/1160602422 learned_addr learned my addr 172.21.15.119:0/1160602422 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:06.348 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da4ea8700 1 --2- 172.21.15.119:0/1160602422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da00cb740 0x7f8da015b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d9f7fe700 1 -- 172.21.15.119:0/1160602422 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557959008 0 0) 0x7f8da00b2d10 con 0x7f8da00cb300 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d9f7fe700 1 -- 172.21.15.119:0/1160602422 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d8c0035d0 con 0x7f8da00cb300 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da4ea8700 1 -- 172.21.15.119:0/1160602422 >> v1:172.21.15.145:6789/0 conn(0x7f8da015a7d0 legacy=0x7f8da006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d9f7fe700 1 -- 172.21.15.119:0/1160602422 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942235459 0 0) 0x7f8da00b1e20 con 0x7f8da015a7d0 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da4ea8700 1 -- 172.21.15.119:0/1160602422 >> v1:172.21.15.119:6789/0 conn(0x7f8da00cb300 legacy=0x7f8da015abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da4ea8700 1 -- 172.21.15.119:0/1160602422 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da00b2100 con 0x7f8da00cb740 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da4ea8700 1 --2- 172.21.15.119:0/1160602422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da00cb740 0x7f8da015b2c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009dc0 tx=0x7f8d9800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb865fe877ef86d8 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d9f7fe700 1 -- 172.21.15.119:0/1160602422 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800e070 con 0x7f8da00cb740 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d9f7fe700 1 -- 172.21.15.119:0/1160602422 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d9800bb20 con 0x7f8da00cb740 2024-09-13T16:37:06.349 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8d9f7fe700 1 -- 172.21.15.119:0/1160602422 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9800bed0 con 0x7f8da00cb740 2024-09-13T16:37:06.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- 172.21.15.119:0/1160602422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da00cb740 msgr2=0x7f8da015b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 --2- 172.21.15.119:0/1160602422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da00cb740 0x7f8da015b2c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009dc0 tx=0x7f8d9800b330 comp rx=0 tx=0).stop 2024-09-13T16:37:06.350 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- 172.21.15.119:0/1160602422 shutdown_connections 2024-09-13T16:37:06.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 --2- 172.21.15.119:0/1160602422 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da00cb740 0x7f8da015b2c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- 172.21.15.119:0/1160602422 >> 172.21.15.119:0/1160602422 conn(0x7f8da006a350 msgr2=0x7f8da006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- 172.21.15.119:0/1160602422 shutdown_connections 2024-09-13T16:37:06.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.345+0000 7f8da56a9700 1 -- 172.21.15.119:0/1160602422 wait complete. 2024-09-13T16:37:06.351 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 Processor -- start 2024-09-13T16:37:06.352 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 -- start start 2024-09-13T16:37:06.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8da00cb300 0x7f8da01497a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8da00cb740 0x7f8da014a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da015a7d0 0x7f8da0140060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.353 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da014a380 con 0x7f8da00cb300 2024-09-13T16:37:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da01523d0 con 0x7f8da00cb740 2024-09-13T16:37:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da56a9700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8da0151a50 con 0x7f8da015a7d0 2024-09-13T16:37:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8da00cb300 0x7f8da01497a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8da00cb300 0x7f8da01497a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44420/0 (socket says 172.21.15.119:44420) 2024-09-13T16:37:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8da4ea8700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da015a7d0 0x7f8da0140060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 -- 172.21.15.119:0/3371646768 learned_addr learned my addr 172.21.15.119:0/3371646768 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:06.354 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d977fe700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8da00cb740 0x7f8da014a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 -- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da015a7d0 msgr2=0x7f8da0140060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f8da015a7d0 0x7f8da0140060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 -- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8da00cb740 msgr2=0x7f8da014a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.355 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8da00cb740 0x7f8da014a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 -- 172.21.15.119:0/3371646768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d98009a70 con 0x7f8da00cb300 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d977fe700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f8da00cb740 0x7f8da014a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.349+0000 7f8d9ffff700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8da00cb300 0x7f8da01497a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000c520 tx=0x7f8d9000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.353+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d900054e0 con 0x7f8da00cb300 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.353+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d900036a0 con 0x7f8da00cb300 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.353+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da014ad10 con 0x7f8da00cb300 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.353+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d90003a50 con 0x7f8da00cb300 2024-09-13T16:37:06.356 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.353+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da0151ff0 con 0x7f8da00cb300 2024-09-13T16:37:06.357 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.353+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8da0066f60 con 0x7f8da00cb300 2024-09-13T16:37:06.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.353+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f8d90003bb0 con 0x7f8da00cb300 2024-09-13T16:37:06.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d9d7fa700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8d80066730 0x7f8d80068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f8d90083370 con 0x7f8da00cb300 2024-09-13T16:37:06.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d9d7fa700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7f8d8006cb70 0x7f8d8006efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d8006f680 con 0x7f8d8006cb70 2024-09-13T16:37:06.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d90085a30 con 0x7f8da00cb300 2024-09-13T16:37:06.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d977fe700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8d80066730 0x7f8d80068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8da4ea8700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7f8d8006cb70 0x7f8d8006efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8da4ea8700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7f8d8006cb70 0x7f8d8006efb0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d977fe700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8d80066730 0x7f8d80068bf0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8da0066b10 tx=0x7f8d88009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.357+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== osd.1 v2:172.21.15.119:6816/1495422598 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f8d8006f680 con 0x7f8d8006cb70 2024-09-13T16:37:06.381 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 --> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8da00b29a0 con 0x7f8d8006cb70 2024-09-13T16:37:06.382 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8d9d7fa700 1 -- 172.21.15.119:0/3371646768 <== osd.1 v2:172.21.15.119:6816/1495422598 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f8d980098f0 con 0x7f8d8006cb70 2024-09-13T16:37:06.382 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:06.382 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:06.382 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:06.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7f8d8006cb70 msgr2=0x7f8d8006efb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:6816/1495422598,v1:172.21.15.119:6817/1495422598] conn(0x7f8d8006cb70 0x7f8d8006efb0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8d80066730 msgr2=0x7f8d80068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f8d80066730 0x7f8d80068bf0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f8da0066b10 tx=0x7f8d88009040 comp rx=0 tx=0).stop 2024-09-13T16:37:06.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8da00cb300 msgr2=0x7f8da01497a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.383 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 --2- 172.21.15.119:0/3371646768 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8da00cb300 0x7f8da01497a0 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000c520 tx=0x7f8d9000c830 comp rx=0 tx=0).stop 2024-09-13T16:37:06.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da4ea8700 1 -- 172.21.15.119:0/3371646768 reap_dead start 2024-09-13T16:37:06.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 shutdown_connections 2024-09-13T16:37:06.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 >> 172.21.15.119:0/3371646768 conn(0x7f8da006a350 msgr2=0x7f8da00c4210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 shutdown_connections 2024-09-13T16:37:06.384 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.377+0000 7f8da56a9700 1 -- 172.21.15.119:0/3371646768 wait complete. 2024-09-13T16:37:06.398 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-13T16:37:06.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a467c0700 1 Processor -- start 2024-09-13T16:37:06.513 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a467c0700 1 -- start start 2024-09-13T16:37:06.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a467c0700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4a401594f0 0x7f4a4006d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a467c0700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a400b2280 con 0x7f4a400b3800 2024-09-13T16:37:06.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a467c0700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a400cc890 con 0x7f4a4006da70 2024-09-13T16:37:06.514 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a467c0700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a4006dee0 con 0x7f4a401594f0 2024-09-13T16:37:06.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a457be700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4a401594f0 0x7f4a4006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a45fbf700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4a400b3800 0x7f4a4006fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41288/0 (socket says 172.21.15.119:41288) 2024-09-13T16:37:06.515 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.509+0000 7f4a45fbf700 1 -- 172.21.15.119:0/1552114309 learned_addr learned my addr 172.21.15.119:0/1552114309 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a457be700 1 -- 172.21.15.119:0/1552114309 >> v1:172.21.15.145:6789/0 conn(0x7f4a4006da70 legacy=0x7f4a400b30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a33fff700 1 -- 172.21.15.119:0/1552114309 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388467664 0 0) 0x7f4a400b2280 con 0x7f4a400b3800 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a457be700 1 -- 172.21.15.119:0/1552114309 >> v1:172.21.15.119:6789/0 conn(0x7f4a400b3800 legacy=0x7f4a4006fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a457be700 1 -- 172.21.15.119:0/1552114309 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a400b1970 con 0x7f4a401594f0 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a457be700 1 --2- 172.21.15.119:0/1552114309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401594f0 0x7f4a4006d530 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009850 tx=0x7f4a3c009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae51fd9c587d24bd server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a33fff700 1 -- 172.21.15.119:0/1552114309 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c011830 con 0x7f4a401594f0 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a33fff700 1 -- 172.21.15.119:0/1552114309 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a3c011990 con 0x7f4a401594f0 2024-09-13T16:37:06.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a33fff700 1 -- 172.21.15.119:0/1552114309 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3c011d20 con 0x7f4a401594f0 2024-09-13T16:37:06.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 -- 172.21.15.119:0/1552114309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401594f0 msgr2=0x7f4a4006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1552114309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401594f0 0x7f4a4006d530 secure :-1 s=READY pgs=2034 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c009850 tx=0x7f4a3c009b60 comp rx=0 tx=0).stop 2024-09-13T16:37:06.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 -- 172.21.15.119:0/1552114309 shutdown_connections 2024-09-13T16:37:06.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1552114309 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a401594f0 0x7f4a4006d530 unknown :-1 s=CLOSED pgs=2034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 -- 172.21.15.119:0/1552114309 >> 172.21.15.119:0/1552114309 conn(0x7f4a4006a350 msgr2=0x7f4a4006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 -- 172.21.15.119:0/1552114309 shutdown_connections 2024-09-13T16:37:06.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 -- 172.21.15.119:0/1552114309 wait complete. 2024-09-13T16:37:06.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 Processor -- start 2024-09-13T16:37:06.518 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.513+0000 7f4a467c0700 1 -- start start 2024-09-13T16:37:06.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a4006da70 0x7f4a400ac640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a400b3800 0x7f4a4015b250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a401594f0 0x7f4a4015b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.521 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4013e770 con 0x7f4a400b3800 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a4013de60 con 0x7f4a401594f0 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a400a98f0 con 0x7f4a4006da70 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a400b3800 0x7f4a4015b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a400b3800 0x7f4a4015b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44438/0 (socket says 172.21.15.119:44438) 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 -- 172.21.15.119:0/1048602323 learned_addr learned my addr 172.21.15.119:0/1048602323 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a457be700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a4006da70 0x7f4a400ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a45fbf700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a401594f0 0x7f4a4015b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 -- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a4006da70 msgr2=0x7f4a400ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a4006da70 0x7f4a400ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 -- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a401594f0 msgr2=0x7f4a4015b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a401594f0 0x7f4a4015b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 -- 172.21.15.119:0/1048602323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a3c0094b0 con 0x7f4a400b3800 2024-09-13T16:37:06.522 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a44fbd700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a400b3800 0x7f4a4015b250 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c530 tx=0x7f4a3400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a31ffb700 1 -- 172.21.15.119:0/1048602323 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a3400cea0 con 0x7f4a400b3800 2024-09-13T16:37:06.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a31ffb700 1 -- 172.21.15.119:0/1048602323 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a340052d0 con 0x7f4a400b3800 2024-09-13T16:37:06.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a31ffb700 1 -- 172.21.15.119:0/1048602323 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a340036a0 con 0x7f4a400b3800 2024-09-13T16:37:06.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a40140cd0 con 0x7f4a400b3800 2024-09-13T16:37:06.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.517+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a4013e390 con 0x7f4a400b3800 2024-09-13T16:37:06.523 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.521+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a400b26f0 con 0x7f4a400b3800 2024-09-13T16:37:06.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.521+0000 7f4a31ffb700 1 -- 172.21.15.119:0/1048602323 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4a34003800 con 0x7f4a400b3800 2024-09-13T16:37:06.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.521+0000 7f4a31ffb700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a24066760 0x7f4a24068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.524 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.521+0000 7f4a31ffb700 1 -- 172.21.15.119:0/1048602323 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f4a34083390 con 0x7f4a400b3800 2024-09-13T16:37:06.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.521+0000 7f4a457be700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a24066760 0x7f4a24068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.525 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.521+0000 7f4a457be700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a24066760 0x7f4a24068c20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0090b0 tx=0x7f4a3c008030 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.530 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.525+0000 7f4a31ffb700 1 -- 172.21.15.119:0/1048602323 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f4a400b26f0 con 0x7f4a400b3800 2024-09-13T16:37:06.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.701+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a400b1f10 con 0x7f4a24066760 2024-09-13T16:37:06.704 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.701+0000 7f4a31ffb700 1 -- 172.21.15.119:0/1048602323 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4a3c011750 con 0x7f4a24066760 2024-09-13T16:37:06.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a24066760 msgr2=0x7f4a24068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a24066760 0x7f4a24068c20 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f4a3c0090b0 tx=0x7f4a3c008030 comp rx=0 tx=0).stop 2024-09-13T16:37:06.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a400b3800 msgr2=0x7f4a4015b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a400b3800 0x7f4a4015b250 secure :-1 s=READY pgs=2035 cs=0 l=1 rev1=1 crypto rx=0x7f4a3400c530 tx=0x7f4a3400c840 comp rx=0 tx=0).stop 2024-09-13T16:37:06.707 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 shutdown_connections 2024-09-13T16:37:06.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4a24066760 0x7f4a24068c20 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4a401594f0 0x7f4a4015b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4a400b3800 0x7f4a4015b250 unknown :-1 s=CLOSED pgs=2035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 --2- 172.21.15.119:0/1048602323 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4a4006da70 0x7f4a400ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 >> 172.21.15.119:0/1048602323 conn(0x7f4a4006a350 msgr2=0x7f4a400afaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 shutdown_connections 2024-09-13T16:37:06.708 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.705+0000 7f4a467c0700 1 -- 172.21.15.119:0/1048602323 wait complete. 2024-09-13T16:37:06.709 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 1 to deep-scrub 2024-09-13T16:37:06.720 INFO:tasks.ceph:Scrubbing osd.2 2024-09-13T16:37:06.720 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.837+0000 7f4d671a2700 1 Processor -- start 2024-09-13T16:37:06.842 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.837+0000 7f4d671a2700 1 -- start start 2024-09-13T16:37:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 0x7f4d6006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4d600cba10 0x7f4d6015abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d600b2d10 con 0x7f4d6006bee0 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d600ce1c0 con 0x7f4d600cba10 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6015c100 con 0x7f4d6015a7d0 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d6599f700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f4d6006bee0 0x7f4d600cb300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41294/0 (socket says 172.21.15.119:41294) 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d669a1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f4d600cba10 0x7f4d6015abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d6599f700 1 -- 172.21.15.119:0/3241267984 learned_addr learned my addr 172.21.15.119:0/3241267984 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d661a0700 1 --2- 172.21.15.119:0/3241267984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 0x7f4d6006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d669a1700 1 -- 172.21.15.119:0/3241267984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 msgr2=0x7f4d6006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d669a1700 1 --2- 172.21.15.119:0/3241267984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 0x7f4d6006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d669a1700 1 -- 172.21.15.119:0/3241267984 >> v1:172.21.15.119:6789/0 conn(0x7f4d6006bee0 legacy=0x7f4d600cb300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d669a1700 1 -- 172.21.15.119:0/3241267984 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d600b1e20 con 0x7f4d600cba10 2024-09-13T16:37:06.846 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d6519e700 1 -- 172.21.15.119:0/3241267984 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954022079 0 0) 0x7f4d600b2d10 con 0x7f4d6006bee0 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d669a1700 1 --2- 172.21.15.119:0/3241267984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6015abb0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009880 tx=0x7f4d58009b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8086794676f02f9c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d6519e700 1 -- 172.21.15.119:0/3241267984 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58011830 con 0x7f4d600cba10 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d6519e700 1 -- 172.21.15.119:0/3241267984 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d58011990 con 0x7f4d600cba10 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d6519e700 1 -- 172.21.15.119:0/3241267984 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d58011d20 con 0x7f4d600cba10 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- 172.21.15.119:0/3241267984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 msgr2=0x7f4d6015abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 --2- 172.21.15.119:0/3241267984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6015abb0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f4d58009880 tx=0x7f4d58009b90 comp rx=0 tx=0).stop 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- 172.21.15.119:0/3241267984 shutdown_connections 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 --2- 172.21.15.119:0/3241267984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6015abb0 unknown :-1 s=CLOSED pgs=2037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 --2- 172.21.15.119:0/3241267984 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 0x7f4d6006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- 172.21.15.119:0/3241267984 >> 172.21.15.119:0/3241267984 conn(0x7f4d6006a350 msgr2=0x7f4d6006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- 172.21.15.119:0/3241267984 shutdown_connections 2024-09-13T16:37:06.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- 172.21.15.119:0/3241267984 wait complete. 2024-09-13T16:37:06.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 Processor -- start 2024-09-13T16:37:06.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.841+0000 7f4d671a2700 1 -- start start 2024-09-13T16:37:06.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6006bee0 0x7f4d600c94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6014f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 0x7f4d601f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.850 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c7870 con 0x7f4d600cba10 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d60153800 con 0x7f4d6006bee0 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d600c7d30 con 0x7f4d6015a7d0 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6014f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6014f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44456/0 (socket says 172.21.15.119:44456) 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 -- 172.21.15.119:0/2902512448 learned_addr learned my addr 172.21.15.119:0/2902512448 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d661a0700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6006bee0 0x7f4d600c94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d669a1700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 0x7f4d601f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 -- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 msgr2=0x7f4d601f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f4d6015a7d0 0x7f4d601f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 -- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6006bee0 msgr2=0x7f4d600c94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f4d6006bee0 0x7f4d600c94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.851 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 -- 172.21.15.119:0/2902512448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d580094b0 con 0x7f4d600cba10 2024-09-13T16:37:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d6599f700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6014f220 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000cc70 tx=0x7f4d5000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d5000aaa0 con 0x7f4d600cba10 2024-09-13T16:37:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d5000ac20 con 0x7f4d600cba10 2024-09-13T16:37:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d600c6300 con 0x7f4d600cba10 2024-09-13T16:37:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d500070d0 con 0x7f4d600cba10 2024-09-13T16:37:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d601508c0 con 0x7f4d600cba10 2024-09-13T16:37:06.852 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.845+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4d600b26c0 con 0x7f4d600cba10 2024-09-13T16:37:06.853 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f4d50007230 con 0x7f4d600cba10 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d5effd700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066730 0x7f4d4c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f4d50083cf0 con 0x7f4d600cba10 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d661a0700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066730 0x7f4d4c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d5effd700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f4d4c06cb70 0x7f4d4c06efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4d4c06f680 con 0x7f4d4c06cb70 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4d500863b0 con 0x7f4d600cba10 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d669a1700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f4d4c06cb70 0x7f4d4c06efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:06.854 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d661a0700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066730 0x7f4d4c068bf0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4d60152490 tx=0x7f4d54008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.849+0000 7f4d669a1700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f4d4c06cb70 0x7f4d4c06efb0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:06.855 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.853+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== osd.2 v2:172.21.15.119:6824/3093261484 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f4d4c06f680 con 0x7f4d4c06cb70 2024-09-13T16:37:06.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 --> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4d600b23e0 con 0x7f4d4c06cb70 2024-09-13T16:37:06.875 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d5effd700 1 -- 172.21.15.119:0/2902512448 <== osd.2 v2:172.21.15.119:6824/3093261484 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f4d58022040 con 0x7f4d4c06cb70 2024-09-13T16:37:06.876 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:06.876 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:06.876 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:06.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f4d4c06cb70 msgr2=0x7f4d4c06efb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:6824/3093261484,v1:172.21.15.119:6825/3093261484] conn(0x7f4d4c06cb70 0x7f4d4c06efb0 crc :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:06.877 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066730 msgr2=0x7f4d4c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f4d4c066730 0x7f4d4c068bf0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f4d60152490 tx=0x7f4d54008040 comp rx=0 tx=0).stop 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 msgr2=0x7f4d6014f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 --2- 172.21.15.119:0/2902512448 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f4d600cba10 0x7f4d6014f220 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7f4d5000cc70 tx=0x7f4d5000cf80 comp rx=0 tx=0).stop 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d669a1700 1 -- 172.21.15.119:0/2902512448 reap_dead start 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 shutdown_connections 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 >> 172.21.15.119:0/2902512448 conn(0x7f4d6006a350 msgr2=0x7f4d60155990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 shutdown_connections 2024-09-13T16:37:06.878 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.873+0000 7f4d671a2700 1 -- 172.21.15.119:0/2902512448 wait complete. 2024-09-13T16:37:06.889 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-13T16:37:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fee90700 1 Processor -- start 2024-09-13T16:37:07.000 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fee90700 1 -- start start 2024-09-13T16:37:07.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fee90700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9f815aac0 0x7fe9f80cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fee90700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe9f806a350 0x7fe9f806a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fee90700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9f806deb0 con 0x7fe9f815aea0 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fee90700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f806ad40 con 0x7fe9f815aac0 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fee90700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9f80cf1c0 con 0x7fe9f806a350 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:06.997+0000 7fe9fde8e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9f815aac0 0x7fe9f80cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fe68f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fe9f806a350 0x7fe9f806a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fde8e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fe9f815aac0 0x7fe9f80cc6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44470/0 (socket says 172.21.15.119:44470) 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fd68d700 1 --1- 172.21.15.119:0/3508417844 >> v1:172.21.15.145:6789/0 conn(0x7fe9f815aea0 0x7fe9f806d680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:34220/0 (socket says 172.21.15.119:34220) 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fde8e700 1 -- 172.21.15.119:0/3508417844 learned_addr learned my addr 172.21.15.119:0/3508417844 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fde8e700 1 -- 172.21.15.119:0/3508417844 >> v2:172.21.15.145:3300/0 conn(0x7fe9f806a350 msgr2=0x7fe9f806a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fde8e700 1 --2- 172.21.15.119:0/3508417844 >> v2:172.21.15.145:3300/0 conn(0x7fe9f806a350 0x7fe9f806a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fde8e700 1 -- 172.21.15.119:0/3508417844 >> v1:172.21.15.145:6789/0 conn(0x7fe9f815aea0 legacy=0x7fe9f806d680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.004 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fce8c700 1 -- 172.21.15.119:0/3508417844 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902008507 0 0) 0x7fe9f806deb0 con 0x7fe9f815aea0 2024-09-13T16:37:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fde8e700 1 -- 172.21.15.119:0/3508417844 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f815b2e0 con 0x7fe9f815aac0 2024-09-13T16:37:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fe68f700 1 --2- 172.21.15.119:0/3508417844 >> v2:172.21.15.145:3300/0 conn(0x7fe9f806a350 0x7fe9f806a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:07.005 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fde8e700 1 --2- 172.21.15.119:0/3508417844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f815aac0 0x7fe9f80cc6e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80d08a0 tx=0x7fe9e800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f287ed7fad8f1b server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fce8c700 1 -- 172.21.15.119:0/3508417844 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e800e070 con 0x7fe9f815aac0 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fce8c700 1 -- 172.21.15.119:0/3508417844 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e800b910 con 0x7fe9f815aac0 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fce8c700 1 -- 172.21.15.119:0/3508417844 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e800bcc0 con 0x7fe9f815aac0 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 -- 172.21.15.119:0/3508417844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f815aac0 msgr2=0x7fe9f80cc6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 --2- 172.21.15.119:0/3508417844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f815aac0 0x7fe9f80cc6e0 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fe9f80d08a0 tx=0x7fe9e800b330 comp rx=0 tx=0).stop 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 -- 172.21.15.119:0/3508417844 shutdown_connections 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 --2- 172.21.15.119:0/3508417844 >> v2:172.21.15.145:3300/0 conn(0x7fe9f806a350 0x7fe9f806a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 --2- 172.21.15.119:0/3508417844 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f815aac0 0x7fe9f80cc6e0 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 -- 172.21.15.119:0/3508417844 >> 172.21.15.119:0/3508417844 conn(0x7fe9f806b460 msgr2=0x7fe9f806b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 -- 172.21.15.119:0/3508417844 shutdown_connections 2024-09-13T16:37:07.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 -- 172.21.15.119:0/3508417844 wait complete. 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 Processor -- start 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.001+0000 7fe9fee90700 1 -- start start 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f806a350 0x7fe9f8145890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9f815aac0 0x7fe9f8142210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9f815aea0 0x7fe9f814e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f8145400 con 0x7fe9f806a350 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f81447f0 con 0x7fe9f815aac0 2024-09-13T16:37:07.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f8144ad0 con 0x7fe9f815aea0 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f806a350 0x7fe9f8145890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f806a350 0x7fe9f8145890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44482/0 (socket says 172.21.15.119:44482) 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 -- 172.21.15.119:0/308395535 learned_addr learned my addr 172.21.15.119:0/308395535 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fe68f700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9f815aea0 0x7fe9f814e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fd68d700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9f815aac0 0x7fe9f8142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 -- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9f815aea0 msgr2=0x7fe9f814e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9f815aea0 0x7fe9f814e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 -- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9f815aac0 msgr2=0x7fe9f8142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.010 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9f815aac0 0x7fe9f8142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 -- 172.21.15.119:0/308395535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00094b0 con 0x7fe9f806a350 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fe68f700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9f815aea0 0x7fe9f814e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fde8e700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f806a350 0x7fe9f8145890 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800b370 tx=0x7fe9e8004af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9f67fc700 1 -- 172.21.15.119:0/308395535 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e80099b0 con 0x7fe9f806a350 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9f67fc700 1 -- 172.21.15.119:0/308395535 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9e80092e0 con 0x7fe9f806a350 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e8009b30 con 0x7fe9f806a350 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9f67fc700 1 -- 172.21.15.119:0/308395535 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9e801c660 con 0x7fe9f806a350 2024-09-13T16:37:07.011 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f8145090 con 0x7fe9f806a350 2024-09-13T16:37:07.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f80677a0 con 0x7fe9f806a350 2024-09-13T16:37:07.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.005+0000 7fe9f67fc700 1 -- 172.21.15.119:0/308395535 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fe9e801c7c0 con 0x7fe9f806a350 2024-09-13T16:37:07.012 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.009+0000 7fe9f67fc700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9e4066730 0x7fe9e4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.009+0000 7fe9f67fc700 1 -- 172.21.15.119:0/308395535 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fe9e80856b0 con 0x7fe9f806a350 2024-09-13T16:37:07.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.009+0000 7fe9fd68d700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9e4066730 0x7fe9e4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.013 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.009+0000 7fe9fd68d700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9e4066730 0x7fe9e4068bf0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8150ad0 tx=0x7fe9ec007c00 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.013+0000 7fe9f67fc700 1 -- 172.21.15.119:0/308395535 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fe9e8052b30 con 0x7fe9f806a350 2024-09-13T16:37:07.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.181+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9f8071e10 con 0x7fe9e4066730 2024-09-13T16:37:07.186 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9f67fc700 1 -- 172.21.15.119:0/308395535 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fe9ec013420 con 0x7fe9e4066730 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9e4066730 msgr2=0x7fe9e4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9e4066730 0x7fe9e4068bf0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe9f8150ad0 tx=0x7fe9ec007c00 comp rx=0 tx=0).stop 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f806a350 msgr2=0x7fe9f8145890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f806a350 0x7fe9f8145890 secure :-1 s=READY pgs=2040 cs=0 l=1 rev1=1 crypto rx=0x7fe9e800b370 tx=0x7fe9e8004af0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 shutdown_connections 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fe9e4066730 0x7fe9e4068bf0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fe9f815aea0 0x7fe9f814e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fe9f815aac0 0x7fe9f8142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 --2- 172.21.15.119:0/308395535 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fe9f806a350 0x7fe9f8145890 unknown :-1 s=CLOSED pgs=2040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 >> 172.21.15.119:0/308395535 conn(0x7fe9f806b460 msgr2=0x7fe9f806d450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 shutdown_connections 2024-09-13T16:37:07.190 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.185+0000 7fe9fee90700 1 -- 172.21.15.119:0/308395535 wait complete. 2024-09-13T16:37:07.191 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 2 to deep-scrub 2024-09-13T16:37:07.201 INFO:tasks.ceph:Scrubbing osd.3 2024-09-13T16:37:07.202 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537759e700 1 Processor -- start 2024-09-13T16:37:07.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537759e700 1 -- start start 2024-09-13T16:37:07.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f537815cc40 0x7f53780cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.325 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537759e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53780d0720 0x7f53780d1130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537759e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53780d3260 con 0x7f53780d1670 2024-09-13T16:37:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537759e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53780d1b20 con 0x7f53780d0720 2024-09-13T16:37:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780d0130 con 0x7f537815cc40 2024-09-13T16:37:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53780d0720 0x7f53780d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f537659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f537815cc40 0x7f53780cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f53780d0720 0x7f53780d1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:53548/0 (socket says 172.21.15.119:53548) 2024-09-13T16:37:07.326 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 -- 172.21.15.119:0/2090763212 learned_addr learned my addr 172.21.15.119:0/2090763212 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:07.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 -- 172.21.15.119:0/2090763212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f537815cc40 msgr2=0x7f53780cfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:07.328 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 --2- 172.21.15.119:0/2090763212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f537815cc40 0x7f53780cfbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 -- 172.21.15.119:0/2090763212 >> v1:172.21.15.145:6789/0 conn(0x7f53780d1670 legacy=0x7f5378160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 -- 172.21.15.119:0/2090763212 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53780cd640 con 0x7f53780d0720 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.321+0000 7f5375d9b700 1 --2- 172.21.15.119:0/2090763212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53780d0720 0x7f53780d1130 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5360009940 tx=0x7f5360009c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2195b0decec7a190 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537559a700 1 -- 172.21.15.119:0/2090763212 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360011830 con 0x7f53780d0720 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537559a700 1 -- 172.21.15.119:0/2090763212 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5360011990 con 0x7f53780d0720 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537559a700 1 -- 172.21.15.119:0/2090763212 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360011d20 con 0x7f53780d0720 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 -- 172.21.15.119:0/2090763212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53780d0720 msgr2=0x7f53780d1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 --2- 172.21.15.119:0/2090763212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53780d0720 0x7f53780d1130 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f5360009940 tx=0x7f5360009c50 comp rx=0 tx=0).stop 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 -- 172.21.15.119:0/2090763212 shutdown_connections 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 --2- 172.21.15.119:0/2090763212 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f53780d0720 0x7f53780d1130 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.329 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 --2- 172.21.15.119:0/2090763212 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f537815cc40 0x7f53780cfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 -- 172.21.15.119:0/2090763212 >> 172.21.15.119:0/2090763212 conn(0x7f537809be40 msgr2=0x7f53780d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 -- 172.21.15.119:0/2090763212 shutdown_connections 2024-09-13T16:37:07.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 -- 172.21.15.119:0/2090763212 wait complete. 2024-09-13T16:37:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 Processor -- start 2024-09-13T16:37:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.325+0000 7f537759e700 1 -- start start 2024-09-13T16:37:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53780d0720 0x7f53780c9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53780d1670 0x7f53781f27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f537815cc40 0x7f53780c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53781f3810 con 0x7f53780d1670 2024-09-13T16:37:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53780c7b60 con 0x7f537815cc40 2024-09-13T16:37:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53780c7550 con 0x7f53780d0720 2024-09-13T16:37:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53780d1670 0x7f53781f27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537659c700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53780d0720 0x7f53780c9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53780d1670 0x7f53781f27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44492/0 (socket says 172.21.15.119:44492) 2024-09-13T16:37:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5376d9d700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f537815cc40 0x7f53780c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 -- 172.21.15.119:0/1843202512 learned_addr learned my addr 172.21.15.119:0/1843202512 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 -- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53780d0720 msgr2=0x7f53780c9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f53780d0720 0x7f53780c9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 -- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f537815cc40 msgr2=0x7f53780c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f537815cc40 0x7f53780c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 -- 172.21.15.119:0/1843202512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5360009570 con 0x7f53780d1670 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f5375d9b700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53780d1670 0x7f53781f27b0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f5360009fd0 tx=0x7f5360004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360004510 con 0x7f53780d1670 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53600036c0 con 0x7f53780d1670 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53781f3250 con 0x7f53780d1670 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360003a30 con 0x7f53780d1670 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53780c77f0 con 0x7f53780d1670 2024-09-13T16:37:07.334 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.329+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f53780d3770 con 0x7f53780d1670 2024-09-13T16:37:07.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5360003b90 con 0x7f53780d1670 2024-09-13T16:37:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f536effd700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5358066730 0x7f5358068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f5360085680 con 0x7f53780d1670 2024-09-13T16:37:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f536effd700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7f535806cb70 0x7f535806efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f535806f680 con 0x7f535806cb70 2024-09-13T16:37:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f53600859f0 con 0x7f53780d1670 2024-09-13T16:37:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f537659c700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5358066730 0x7f5358068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f5376d9d700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7f535806cb70 0x7f535806efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f5376d9d700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7f535806cb70 0x7f535806efb0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f537659c700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5358066730 0x7f5358068bf0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f53680113e0 tx=0x7f536800b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.333+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== osd.3 v2:172.21.15.119:6808/3803418857 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f535806f680 con 0x7f535806cb70 2024-09-13T16:37:07.358 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.353+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 --> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f53780d3490 con 0x7f535806cb70 2024-09-13T16:37:07.359 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f536effd700 1 -- 172.21.15.119:0/1843202512 <== osd.3 v2:172.21.15.119:6808/3803418857 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5370001960 con 0x7f535806cb70 2024-09-13T16:37:07.359 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:07.359 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:07.359 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:07.360 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7f535806cb70 msgr2=0x7f535806efb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:6808/3803418857,v1:172.21.15.119:6809/3803418857] conn(0x7f535806cb70 0x7f535806efb0 crc :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5358066730 msgr2=0x7f5358068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f5358066730 0x7f5358068bf0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f53680113e0 tx=0x7f536800b040 comp rx=0 tx=0).stop 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53780d1670 msgr2=0x7f53781f27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 --2- 172.21.15.119:0/1843202512 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53780d1670 0x7f53781f27b0 secure :-1 s=READY pgs=2041 cs=0 l=1 rev1=1 crypto rx=0x7f5360009fd0 tx=0x7f5360004770 comp rx=0 tx=0).stop 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f5376d9d700 1 -- 172.21.15.119:0/1843202512 reap_dead start 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 shutdown_connections 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 >> 172.21.15.119:0/1843202512 conn(0x7f537809be40 msgr2=0x7f53780cf420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 shutdown_connections 2024-09-13T16:37:07.361 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.357+0000 7f537759e700 1 -- 172.21.15.119:0/1843202512 wait complete. 2024-09-13T16:37:07.372 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-13T16:37:07.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.489+0000 7f45d69a4700 1 Processor -- start 2024-09-13T16:37:07.493 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.489+0000 7f45d69a4700 1 -- start start 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.489+0000 7f45d69a4700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f45d006cf70 0x7f45d00cd930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.489+0000 7f45d69a4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45d00cde70 0x7f45d0070150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.489+0000 7f45d69a4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45d006fcc0 con 0x7f45d0144180 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.489+0000 7f45d69a4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d006de30 con 0x7f45d00cde70 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.489+0000 7f45d69a4700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45d006d3b0 con 0x7f45d006cf70 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d61a3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f45d00cde70 0x7f45d0070150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d59a2700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f45d0144180 0x7f45d00cbe70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41310/0 (socket says 172.21.15.119:41310) 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d59a2700 1 -- 172.21.15.119:0/1962115251 learned_addr learned my addr 172.21.15.119:0/1962115251 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:07.495 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d51a1700 1 --2- 172.21.15.119:0/1962115251 >> v2:172.21.15.145:3300/0 conn(0x7f45d006cf70 0x7f45d00cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d61a3700 1 -- 172.21.15.119:0/1962115251 >> v2:172.21.15.145:3300/0 conn(0x7f45d006cf70 msgr2=0x7f45d00cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:07.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d61a3700 1 --2- 172.21.15.119:0/1962115251 >> v2:172.21.15.145:3300/0 conn(0x7f45d006cf70 0x7f45d00cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d61a3700 1 -- 172.21.15.119:0/1962115251 >> v1:172.21.15.119:6789/0 conn(0x7f45d0144180 legacy=0x7f45d00cbe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.496 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d61a3700 1 -- 172.21.15.119:0/1962115251 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45d006d590 con 0x7f45d00cde70 2024-09-13T16:37:07.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d49a0700 1 -- 172.21.15.119:0/1962115251 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989804987 0 0) 0x7f45d006fcc0 con 0x7f45d0144180 2024-09-13T16:37:07.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d61a3700 1 --2- 172.21.15.119:0/1962115251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d00cde70 0x7f45d0070150 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f45d00cb460 tx=0x7f45c800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b175bd6d0bfa4bba server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d49a0700 1 -- 172.21.15.119:0/1962115251 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c800e070 con 0x7f45d00cde70 2024-09-13T16:37:07.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d49a0700 1 -- 172.21.15.119:0/1962115251 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c800bb20 con 0x7f45d00cde70 2024-09-13T16:37:07.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d49a0700 1 -- 172.21.15.119:0/1962115251 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8012430 con 0x7f45d00cde70 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 -- 172.21.15.119:0/1962115251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d00cde70 msgr2=0x7f45d0070150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 --2- 172.21.15.119:0/1962115251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d00cde70 0x7f45d0070150 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7f45d00cb460 tx=0x7f45c800b330 comp rx=0 tx=0).stop 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 -- 172.21.15.119:0/1962115251 shutdown_connections 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 --2- 172.21.15.119:0/1962115251 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d00cde70 0x7f45d0070150 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 --2- 172.21.15.119:0/1962115251 >> v2:172.21.15.145:3300/0 conn(0x7f45d006cf70 0x7f45d00cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 -- 172.21.15.119:0/1962115251 >> 172.21.15.119:0/1962115251 conn(0x7f45d006a350 msgr2=0x7f45d006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 -- 172.21.15.119:0/1962115251 shutdown_connections 2024-09-13T16:37:07.498 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.493+0000 7f45d69a4700 1 -- 172.21.15.119:0/1962115251 wait complete. 2024-09-13T16:37:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 Processor -- start 2024-09-13T16:37:07.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 -- start start 2024-09-13T16:37:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d006cf70 0x7f45d01371a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45d00cde70 0x7f45d01357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45d0144180 0x7f45d013a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d015dbb0 con 0x7f45d006cf70 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45d0137a80 con 0x7f45d0144180 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d69a4700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45d01377a0 con 0x7f45d00cde70 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d59a2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d006cf70 0x7f45d01371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d59a2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d006cf70 0x7f45d01371a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44512/0 (socket says 172.21.15.119:44512) 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d59a2700 1 -- 172.21.15.119:0/319604689 learned_addr learned my addr 172.21.15.119:0/319604689 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:07.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d61a3700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45d0144180 0x7f45d013a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.504 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.497+0000 7f45d51a1700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45d00cde70 0x7f45d01357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d61a3700 1 -- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45d00cde70 msgr2=0x7f45d01357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d61a3700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45d00cde70 0x7f45d01357a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d61a3700 1 -- 172.21.15.119:0/319604689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d006cf70 msgr2=0x7f45d01371a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d61a3700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d006cf70 0x7f45d01371a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d61a3700 1 -- 172.21.15.119:0/319604689 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45c8009a70 con 0x7f45d0144180 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d59a2700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d006cf70 0x7f45d01371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d61a3700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45d0144180 0x7f45d013a280 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f45c8016000 tx=0x7f45c8004af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45c67fc700 1 -- 172.21.15.119:0/319604689 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8012430 con 0x7f45d0144180 2024-09-13T16:37:07.505 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45c67fc700 1 -- 172.21.15.119:0/319604689 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45c8020070 con 0x7f45d0144180 2024-09-13T16:37:07.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45d00af710 con 0x7f45d0144180 2024-09-13T16:37:07.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45c67fc700 1 -- 172.21.15.119:0/319604689 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45c8004280 con 0x7f45d0144180 2024-09-13T16:37:07.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45d00afcd0 con 0x7f45d0144180 2024-09-13T16:37:07.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.501+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45d00cd930 con 0x7f45d0144180 2024-09-13T16:37:07.506 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.505+0000 7f45c67fc700 1 -- 172.21.15.119:0/319604689 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f45c80043e0 con 0x7f45d0144180 2024-09-13T16:37:07.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.505+0000 7f45c67fc700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45bc066730 0x7f45bc068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.505+0000 7f45c67fc700 1 -- 172.21.15.119:0/319604689 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f45c8022070 con 0x7f45d0144180 2024-09-13T16:37:07.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.505+0000 7f45d59a2700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45bc066730 0x7f45bc068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.505+0000 7f45d59a2700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45bc066730 0x7f45bc068bf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f45c0000f60 tx=0x7f45c0007f50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.516 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.513+0000 7f45c67fc700 1 -- 172.21.15.119:0/319604689 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f45c8053520 con 0x7f45d0144180 2024-09-13T16:37:07.687 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f45d00ce300 con 0x7f45bc066730 2024-09-13T16:37:07.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45c67fc700 1 -- 172.21.15.119:0/319604689 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f45c0007120 con 0x7f45bc066730 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45bc066730 msgr2=0x7f45bc068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45bc066730 0x7f45bc068bf0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7f45c0000f60 tx=0x7f45c0007f50 comp rx=0 tx=0).stop 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45d0144180 msgr2=0x7f45d013a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45d0144180 0x7f45d013a280 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f45c8016000 tx=0x7f45c8004af0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 shutdown_connections 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f45bc066730 0x7f45bc068bf0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f45d0144180 0x7f45d013a280 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f45d00cde70 0x7f45d01357a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 --2- 172.21.15.119:0/319604689 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f45d006cf70 0x7f45d01371a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 >> 172.21.15.119:0/319604689 conn(0x7f45d006a350 msgr2=0x7f45d00b2600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.691 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.685+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 shutdown_connections 2024-09-13T16:37:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.689+0000 7f45d69a4700 1 -- 172.21.15.119:0/319604689 wait complete. 2024-09-13T16:37:07.692 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 3 to deep-scrub 2024-09-13T16:37:07.703 INFO:tasks.ceph:Scrubbing osd.4 2024-09-13T16:37:07.703 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:07.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.821+0000 7fae47fff700 1 Processor -- start 2024-09-13T16:37:07.826 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.821+0000 7fae47fff700 1 -- start start 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fae480d35a0 0x7fae480b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae480d0650 con 0x7fae480b26e0 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae480d39e0 con 0x7fae480d35a0 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae480ce5a0 con 0x7fae4815c900 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fae480d35a0 0x7fae480b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae46ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fae480d35a0 0x7fae480b21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:53582/0 (socket says 172.21.15.119:53582) 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae477fe700 1 --1- >> v1:172.21.15.145:6789/0 conn(0x7fae480b26e0 0x7fae480d10a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.145:6789/0 says I am v1:172.21.15.119:34236/0 (socket says 172.21.15.119:34236) 2024-09-13T16:37:07.831 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 -- 172.21.15.119:0/672991270 learned_addr learned my addr 172.21.15.119:0/672991270 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae46ffd700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:39596/0 (socket says 172.21.15.119:39596) 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 -- 172.21.15.119:0/672991270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 msgr2=0x7fae4815cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 --2- 172.21.15.119:0/672991270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 -- 172.21.15.119:0/672991270 >> v1:172.21.15.145:6789/0 conn(0x7fae480b26e0 legacy=0x7fae480d10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae45ffb700 1 -- 172.21.15.119:0/672991270 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577832451 0 0) 0x7fae480d0650 con 0x7fae480b26e0 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 -- 172.21.15.119:0/672991270 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae480ce880 con 0x7fae480d35a0 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae467fc700 1 --2- 172.21.15.119:0/672991270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480d35a0 0x7fae480b21a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fae38009910 tx=0x7fae38009c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd03ab6d43cc0959 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae45ffb700 1 -- 172.21.15.119:0/672991270 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38011830 con 0x7fae480d35a0 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae45ffb700 1 -- 172.21.15.119:0/672991270 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae38011990 con 0x7fae480d35a0 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae45ffb700 1 -- 172.21.15.119:0/672991270 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae38011d40 con 0x7fae480d35a0 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- 172.21.15.119:0/672991270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480d35a0 msgr2=0x7fae480b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 --2- 172.21.15.119:0/672991270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480d35a0 0x7fae480b21a0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fae38009910 tx=0x7fae38009c20 comp rx=0 tx=0).stop 2024-09-13T16:37:07.832 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- 172.21.15.119:0/672991270 shutdown_connections 2024-09-13T16:37:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 --2- 172.21.15.119:0/672991270 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480d35a0 0x7fae480b21a0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 --2- 172.21.15.119:0/672991270 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- 172.21.15.119:0/672991270 >> 172.21.15.119:0/672991270 conn(0x7fae480d2a80 msgr2=0x7fae480d2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- 172.21.15.119:0/672991270 shutdown_connections 2024-09-13T16:37:07.833 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.825+0000 7fae47fff700 1 -- 172.21.15.119:0/672991270 wait complete. 2024-09-13T16:37:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 Processor -- start 2024-09-13T16:37:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 -- start start 2024-09-13T16:37:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480b26e0 0x7fae4814ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.835 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae480d35a0 0x7fae480c8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815bae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae4815df80 con 0x7fae480d35a0 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae480c92f0 con 0x7fae480b26e0 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae47fff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae480c95d0 con 0x7fae4815c900 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae480d35a0 0x7fae480c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae46ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480b26e0 0x7fae4814ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae477fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae480d35a0 0x7fae480c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44516/0 (socket says 172.21.15.119:44516) 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae46ffd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480b26e0 0x7fae4814ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:53592/0 (socket says 172.21.15.119:53592) 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 -- 172.21.15.119:0/1341679048 learned_addr learned my addr 172.21.15.119:0/1341679048 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 -- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 msgr2=0x7fae4815bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fae4815c900 0x7fae4815bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.836 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 -- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480b26e0 msgr2=0x7fae4814ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480b26e0 0x7fae4814ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae467fc700 1 -- 172.21.15.119:0/1341679048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae38009570 con 0x7fae480d35a0 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.829+0000 7fae46ffd700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fae480b26e0 0x7fae4814ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae467fc700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae480d35a0 0x7fae480c8880 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fae38013000 tx=0x7fae38004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3800b760 con 0x7fae480d35a0 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3801f070 con 0x7fae480d35a0 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae4815e220 con 0x7fae480d35a0 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae3800bac0 con 0x7fae480d35a0 2024-09-13T16:37:07.837 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae480c98b0 con 0x7fae480d35a0 2024-09-13T16:37:07.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fae38021030 con 0x7fae480d35a0 2024-09-13T16:37:07.838 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.833+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fae480b1e70 con 0x7fae480d35a0 2024-09-13T16:37:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae2f7fe700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae34066760 0x7fae34068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fae38085300 con 0x7fae480d35a0 2024-09-13T16:37:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae2f7fe700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fae3406cba0 0x7fae3406efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:07.839 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fae3406f6b0 con 0x7fae3406cba0 2024-09-13T16:37:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7fae380879c0 con 0x7fae480d35a0 2024-09-13T16:37:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae46ffd700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae34066760 0x7fae34068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae477fe700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fae3406cba0 0x7fae3406efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae46ffd700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae34066760 0x7fae34068c20 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0112c0 tx=0x7fae3c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.840 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae477fe700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fae3406cba0 0x7fae3406efe0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:07.841 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.837+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== osd.4 v2:172.21.15.145:6816/4125431312 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7fae3406f6b0 con 0x7fae3406cba0 2024-09-13T16:37:07.861 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.857+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 --> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fae480d0b80 con 0x7fae3406cba0 2024-09-13T16:37:07.862 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.857+0000 7fae2f7fe700 1 -- 172.21.15.119:0/1341679048 <== osd.4 v2:172.21.15.145:6816/4125431312 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7fae400071c0 con 0x7fae3406cba0 2024-09-13T16:37:07.862 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:07.862 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:07.863 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:07.863 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fae3406cba0 msgr2=0x7fae3406efe0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6816/4125431312,v1:172.21.15.145:6817/4125431312] conn(0x7fae3406cba0 0x7fae3406efe0 crc :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:07.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae34066760 msgr2=0x7fae34068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fae34066760 0x7fae34068c20 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fae3c0112c0 tx=0x7fae3c009240 comp rx=0 tx=0).stop 2024-09-13T16:37:07.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae480d35a0 msgr2=0x7fae480c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:07.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 --2- 172.21.15.119:0/1341679048 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fae480d35a0 0x7fae480c8880 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fae38013000 tx=0x7fae38004770 comp rx=0 tx=0).stop 2024-09-13T16:37:07.864 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae477fe700 1 -- 172.21.15.119:0/1341679048 reap_dead start 2024-09-13T16:37:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 shutdown_connections 2024-09-13T16:37:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 >> 172.21.15.119:0/1341679048 conn(0x7fae480d2a80 msgr2=0x7fae4815b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 shutdown_connections 2024-09-13T16:37:07.865 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.861+0000 7fae47fff700 1 -- 172.21.15.119:0/1341679048 wait complete. 2024-09-13T16:37:07.876 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-13T16:37:07.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.993+0000 7f241cf41700 1 Processor -- start 2024-09-13T16:37:07.998 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.993+0000 7f241cf41700 1 -- start start 2024-09-13T16:37:08.001 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f241cf41700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f241815c8c0 0x7f2418066330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f241cf41700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f24180b2600 0x7f241806a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f241cf41700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24180b41b0 con 0x7f2418070360 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f241cf41700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24180ad170 con 0x7f24180b2600 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f241cf41700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24180b3b80 con 0x7f241815c8c0 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f24177fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f241815c8c0 0x7f2418066330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f24180b2600 0x7f241806a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f24180b2600 0x7f241806a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44518/0 (socket says 172.21.15.119:44518) 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 -- 172.21.15.119:0/2210950860 learned_addr learned my addr 172.21.15.119:0/2210950860 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:08.002 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f24177fe700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f241815c8c0 0x7f2418066330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3300/0 says I am v2:172.21.15.119:53602/0 (socket says 172.21.15.119:53602) 2024-09-13T16:37:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 -- 172.21.15.119:0/2210950860 >> v2:172.21.15.145:3300/0 conn(0x7f241815c8c0 msgr2=0x7f2418066330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 --2- 172.21.15.119:0/2210950860 >> v2:172.21.15.145:3300/0 conn(0x7f241815c8c0 0x7f2418066330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 -- 172.21.15.119:0/2210950860 >> v1:172.21.15.145:6789/0 conn(0x7f2418070360 legacy=0x7f24180b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 -- 172.21.15.119:0/2210950860 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24180b2a30 con 0x7f24180b2600 2024-09-13T16:37:08.003 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:07.997+0000 7f2417fff700 1 --2- 172.21.15.119:0/2210950860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24180b2600 0x7f241806a630 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f241806bef0 tx=0x7f241000b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6468550869ad5b8c server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.006 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.001+0000 7f24167fc700 1 -- 172.21.15.119:0/2210950860 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241000e070 con 0x7f24180b2600 2024-09-13T16:37:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.001+0000 7f24167fc700 1 -- 172.21.15.119:0/2210950860 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f241000b910 con 0x7f24180b2600 2024-09-13T16:37:08.007 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.001+0000 7f24167fc700 1 -- 172.21.15.119:0/2210950860 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f241000bcc0 con 0x7f24180b2600 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 -- 172.21.15.119:0/2210950860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24180b2600 msgr2=0x7f241806a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 --2- 172.21.15.119:0/2210950860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24180b2600 0x7f241806a630 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f241806bef0 tx=0x7f241000b330 comp rx=0 tx=0).stop 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 -- 172.21.15.119:0/2210950860 shutdown_connections 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 --2- 172.21.15.119:0/2210950860 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f24180b2600 0x7f241806a630 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 --2- 172.21.15.119:0/2210950860 >> v2:172.21.15.145:3300/0 conn(0x7f241815c8c0 0x7f2418066330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 -- 172.21.15.119:0/2210950860 >> 172.21.15.119:0/2210950860 conn(0x7f24180cef00 msgr2=0x7f24180cf2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 -- 172.21.15.119:0/2210950860 shutdown_connections 2024-09-13T16:37:08.008 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 -- 172.21.15.119:0/2210950860 wait complete. 2024-09-13T16:37:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 Processor -- start 2024-09-13T16:37:08.009 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.005+0000 7f241cf41700 1 -- start start 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2418070360 0x7f24180cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24180b2600 0x7f2418099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f241815c8c0 0x7f241809ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24180c9460 con 0x7f2418070360 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241809a430 con 0x7f24180b2600 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241809a710 con 0x7f241815c8c0 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2418070360 0x7f24180cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2418070360 0x7f24180cdbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44520/0 (socket says 172.21.15.119:44520) 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f2417fff700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f241815c8c0 0x7f241809ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 -- 172.21.15.119:0/3566690821 learned_addr learned my addr 172.21.15.119:0/3566690821 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:08.014 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f2416ffd700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24180b2600 0x7f2418099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 -- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f241815c8c0 msgr2=0x7f241809ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f241815c8c0 0x7f241809ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 -- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24180b2600 msgr2=0x7f2418099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24180b2600 0x7f2418099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 -- 172.21.15.119:0/3566690821 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2410009b30 con 0x7f2418070360 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f2417fff700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f241815c8c0 0x7f241809ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f24177fe700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2418070360 0x7f24180cdbd0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f240800e840 tx=0x7f240800eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f23fffff700 1 -- 172.21.15.119:0/3566690821 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24080095a0 con 0x7f2418070360 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f23fffff700 1 -- 172.21.15.119:0/3566690821 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2408013680 con 0x7f2418070360 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241809a9f0 con 0x7f2418070360 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f23fffff700 1 -- 172.21.15.119:0/3566690821 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24080139f0 con 0x7f2418070360 2024-09-13T16:37:08.015 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241809afb0 con 0x7f2418070360 2024-09-13T16:37:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241806edb0 con 0x7f2418070360 2024-09-13T16:37:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.009+0000 7f23fffff700 1 -- 172.21.15.119:0/3566690821 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2408013b50 con 0x7f2418070360 2024-09-13T16:37:08.016 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.013+0000 7f23fffff700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2404066730 0x7f2404068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.013+0000 7f23fffff700 1 -- 172.21.15.119:0/3566690821 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f2408083e00 con 0x7f2418070360 2024-09-13T16:37:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.013+0000 7f2416ffd700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2404066730 0x7f2404068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.017 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.013+0000 7f2416ffd700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2404066730 0x7f2404068bf0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f241809c8c0 tx=0x7f240c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.021 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.017+0000 7f23fffff700 1 -- 172.21.15.119:0/3566690821 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2408051310 con 0x7f2418070360 2024-09-13T16:37:08.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.189+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f24180b3e40 con 0x7f2404066730 2024-09-13T16:37:08.192 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.189+0000 7f23fffff700 1 -- 172.21.15.119:0/3566690821 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f240c0015e0 con 0x7f2404066730 2024-09-13T16:37:08.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2404066730 msgr2=0x7f2404068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2404066730 0x7f2404068bf0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f241809c8c0 tx=0x7f240c009040 comp rx=0 tx=0).stop 2024-09-13T16:37:08.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2418070360 msgr2=0x7f24180cdbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.195 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2418070360 0x7f24180cdbd0 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f240800e840 tx=0x7f240800eb50 comp rx=0 tx=0).stop 2024-09-13T16:37:08.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 shutdown_connections 2024-09-13T16:37:08.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2404066730 0x7f2404068bf0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f241815c8c0 0x7f241809ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f24180b2600 0x7f2418099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 --2- 172.21.15.119:0/3566690821 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2418070360 0x7f24180cdbd0 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 >> 172.21.15.119:0/3566690821 conn(0x7f24180cef00 msgr2=0x7f24181601f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.196 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 shutdown_connections 2024-09-13T16:37:08.197 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.193+0000 7f241cf41700 1 -- 172.21.15.119:0/3566690821 wait complete. 2024-09-13T16:37:08.197 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 4 to deep-scrub 2024-09-13T16:37:08.208 INFO:tasks.ceph:Scrubbing osd.5 2024-09-13T16:37:08.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:08.330 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.325+0000 7f02d9faa700 1 Processor -- start 2024-09-13T16:37:08.331 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.325+0000 7f02d9faa700 1 -- start start 2024-09-13T16:37:08.332 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d9faa700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d415cc40 0x7f02d40cfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d9faa700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d40d3260 con 0x7f02d40d0720 2024-09-13T16:37:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d9faa700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02d40cd640 con 0x7f02d40d1840 2024-09-13T16:37:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d9faa700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40d0130 con 0x7f02d415cc40 2024-09-13T16:37:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d3fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f02d40d0720 0x7f02d40d1130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41318/0 (socket says 172.21.15.119:41318) 2024-09-13T16:37:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d3fff700 1 -- 172.21.15.119:0/1641356868 learned_addr learned my addr 172.21.15.119:0/1641356868 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:08.333 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d8fa8700 1 --2- 172.21.15.119:0/1641356868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d415cc40 0x7f02d40cfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d37fe700 1 -- 172.21.15.119:0/1641356868 <== mon.1 v1:172.21.15.145:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2867877252 0 0) 0x7f02d40cd640 con 0x7f02d40d1840 2024-09-13T16:37:08.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d37fe700 1 -- 172.21.15.119:0/1641356868 --> v1:172.21.15.145:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02c00035d0 con 0x7f02d40d1840 2024-09-13T16:37:08.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d37fe700 1 -- 172.21.15.119:0/1641356868 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3233703162 0 0) 0x7f02d40d3260 con 0x7f02d40d0720 2024-09-13T16:37:08.335 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d37fe700 1 -- 172.21.15.119:0/1641356868 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02d40cd640 con 0x7f02d40d0720 2024-09-13T16:37:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d8fa8700 1 -- 172.21.15.119:0/1641356868 >> v1:172.21.15.145:6789/0 conn(0x7f02d40d1840 legacy=0x7f02d40cdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d8fa8700 1 -- 172.21.15.119:0/1641356868 >> v1:172.21.15.119:6789/0 conn(0x7f02d40d0720 legacy=0x7f02d40d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d8fa8700 1 -- 172.21.15.119:0/1641356868 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02d40d0c70 con 0x7f02d415cc40 2024-09-13T16:37:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d8fa8700 1 --2- 172.21.15.119:0/1641356868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d415cc40 0x7f02d40cfbf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0098b0 tx=0x7f02bc009bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92022734ceb1d5a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d37fe700 1 -- 172.21.15.119:0/1641356868 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc011830 con 0x7f02d415cc40 2024-09-13T16:37:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d37fe700 1 -- 172.21.15.119:0/1641356868 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02bc011990 con 0x7f02d415cc40 2024-09-13T16:37:08.336 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.329+0000 7f02d37fe700 1 -- 172.21.15.119:0/1641356868 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02bc011d20 con 0x7f02d415cc40 2024-09-13T16:37:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 -- 172.21.15.119:0/1641356868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d415cc40 msgr2=0x7f02d40cfbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 --2- 172.21.15.119:0/1641356868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d415cc40 0x7f02d40cfbf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0098b0 tx=0x7f02bc009bc0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 -- 172.21.15.119:0/1641356868 shutdown_connections 2024-09-13T16:37:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 --2- 172.21.15.119:0/1641356868 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d415cc40 0x7f02d40cfbf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 -- 172.21.15.119:0/1641356868 >> 172.21.15.119:0/1641356868 conn(0x7f02d409be40 msgr2=0x7f02d40d2b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 -- 172.21.15.119:0/1641356868 shutdown_connections 2024-09-13T16:37:08.337 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 -- 172.21.15.119:0/1641356868 wait complete. 2024-09-13T16:37:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 Processor -- start 2024-09-13T16:37:08.338 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 -- start start 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.333+0000 7f02d9faa700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d40d0720 0x7f02d415ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02d40d1840 0x7f02d415efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d415cc40 0x7f02d41f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b3e70 con 0x7f02d40d1840 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b32f0 con 0x7f02d415cc40 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02d40b35d0 con 0x7f02d40d0720 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02d40d1840 0x7f02d415efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02d40d1840 0x7f02d415efa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44528/0 (socket says 172.21.15.119:44528) 2024-09-13T16:37:08.341 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 -- 172.21.15.119:0/1129828526 learned_addr learned my addr 172.21.15.119:0/1129828526 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d8fa8700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d40d0720 0x7f02d415ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d97a9700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d415cc40 0x7f02d41f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 -- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d40d0720 msgr2=0x7f02d415ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f02d40d0720 0x7f02d415ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 -- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d415cc40 msgr2=0x7f02d41f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f02d415cc40 0x7f02d41f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 -- 172.21.15.119:0/1129828526 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02bc0094b0 con 0x7f02d40d1840 2024-09-13T16:37:08.342 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d3fff700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02d40d1840 0x7f02d415efa0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f02c4005930 tx=0x7f02c400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c4005d90 con 0x7f02d40d1840 2024-09-13T16:37:08.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02d40b38b0 con 0x7f02d40d1840 2024-09-13T16:37:08.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02c40070d0 con 0x7f02d40d1840 2024-09-13T16:37:08.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02d41535d0 con 0x7f02d40d1840 2024-09-13T16:37:08.343 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02c40012d0 con 0x7f02d40d1840 2024-09-13T16:37:08.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f02d408c9b0 con 0x7f02d40d1840 2024-09-13T16:37:08.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.337+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f02c4007440 con 0x7f02d40d1840 2024-09-13T16:37:08.345 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d17fa700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02b4066730 0x7f02b4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f02c40832f0 con 0x7f02d40d1840 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d17fa700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f02b406cb70 0x7f02b406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f02b406f680 con 0x7f02b406cb70 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f02c40859b0 con 0x7f02d40d1840 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d8fa8700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02b4066730 0x7f02b4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d97a9700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f02b406cb70 0x7f02b406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d8fa8700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02b4066730 0x7f02b4068bf0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0090b0 tx=0x7f02bc01c040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.346 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d97a9700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f02b406cb70 0x7f02b406efb0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.347 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.341+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== osd.5 v2:172.21.15.145:6824/4068956621 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f02b406f680 con 0x7f02b406cb70 2024-09-13T16:37:08.363 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 --> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f02d40b89d0 con 0x7f02b406cb70 2024-09-13T16:37:08.364 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d17fa700 1 -- 172.21.15.119:0/1129828526 <== osd.5 v2:172.21.15.145:6824/4068956621 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f02d40b89d0 con 0x7f02b406cb70 2024-09-13T16:37:08.364 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:08.364 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:08.365 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:08.365 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f02b406cb70 msgr2=0x7f02b406efb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6824/4068956621,v1:172.21.15.145:6825/4068956621] conn(0x7f02b406cb70 0x7f02b406efb0 crc :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02b4066730 msgr2=0x7f02b4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f02b4066730 0x7f02b4068bf0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f02bc0090b0 tx=0x7f02bc01c040 comp rx=0 tx=0).stop 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02d40d1840 msgr2=0x7f02d415efa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 --2- 172.21.15.119:0/1129828526 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f02d40d1840 0x7f02d415efa0 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7f02c4005930 tx=0x7f02c400c9e0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d97a9700 1 -- 172.21.15.119:0/1129828526 reap_dead start 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 shutdown_connections 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 >> 172.21.15.119:0/1129828526 conn(0x7f02d409be40 msgr2=0x7f02d406bbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.366 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 shutdown_connections 2024-09-13T16:37:08.367 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.361+0000 7f02d9faa700 1 -- 172.21.15.119:0/1129828526 wait complete. 2024-09-13T16:37:08.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-13T16:37:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.493+0000 7f0cc9e06700 1 Processor -- start 2024-09-13T16:37:08.497 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.493+0000 7f0cc9e06700 1 -- start start 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.493+0000 7f0cc9e06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4144330 0x7f0cc40cda50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.493+0000 7f0cc9e06700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0cc40b3410 0x7f0cc40cdf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.493+0000 7f0cc9e06700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc40ce4d0 con 0x7f0cc4143f50 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.493+0000 7f0cc9e06700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc40b1c40 con 0x7f0cc40b3410 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.493+0000 7f0cc9e06700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc406cdf0 con 0x7f0cc4144330 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9605700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0cc40b3410 0x7f0cc40cdf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc8e04700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f0cc4143f50 0x7f0cc406c5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41326/0 (socket says 172.21.15.119:41326) 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9605700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f0cc40b3410 0x7f0cc40cdf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44536/0 (socket says 172.21.15.119:44536) 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc8e04700 1 -- 172.21.15.119:0/2220991577 learned_addr learned my addr 172.21.15.119:0/2220991577 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:08.500 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc3fff700 1 --2- 172.21.15.119:0/2220991577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4144330 0x7f0cc40cda50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9605700 1 -- 172.21.15.119:0/2220991577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4144330 msgr2=0x7f0cc40cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9605700 1 --2- 172.21.15.119:0/2220991577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4144330 0x7f0cc40cda50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9605700 1 -- 172.21.15.119:0/2220991577 >> v1:172.21.15.119:6789/0 conn(0x7f0cc4143f50 legacy=0x7f0cc406c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9605700 1 -- 172.21.15.119:0/2220991577 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc406d0d0 con 0x7f0cc40b3410 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc3fff700 1 --2- 172.21.15.119:0/2220991577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4144330 0x7f0cc40cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9605700 1 --2- 172.21.15.119:0/2220991577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc40b3410 0x7f0cc40cdf90 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009850 tx=0x7f0cbc009b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2dc4e56cfc79cbf server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc37fe700 1 -- 172.21.15.119:0/2220991577 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc011830 con 0x7f0cc40b3410 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc37fe700 1 -- 172.21.15.119:0/2220991577 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc011990 con 0x7f0cc40b3410 2024-09-13T16:37:08.501 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc37fe700 1 -- 172.21.15.119:0/2220991577 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc011d20 con 0x7f0cc40b3410 2024-09-13T16:37:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 -- 172.21.15.119:0/2220991577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc40b3410 msgr2=0x7f0cc40cdf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/2220991577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc40b3410 0x7f0cc40cdf90 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc009850 tx=0x7f0cbc009b60 comp rx=0 tx=0).stop 2024-09-13T16:37:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 -- 172.21.15.119:0/2220991577 shutdown_connections 2024-09-13T16:37:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/2220991577 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc40b3410 0x7f0cc40cdf90 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.502 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/2220991577 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4144330 0x7f0cc40cda50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 -- 172.21.15.119:0/2220991577 >> 172.21.15.119:0/2220991577 conn(0x7f0cc406a350 msgr2=0x7f0cc406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 -- 172.21.15.119:0/2220991577 shutdown_connections 2024-09-13T16:37:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.497+0000 7f0cc9e06700 1 -- 172.21.15.119:0/2220991577 wait complete. 2024-09-13T16:37:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 Processor -- start 2024-09-13T16:37:08.503 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 -- start start 2024-09-13T16:37:08.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b3410 0x7f0cc415a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4143f50 0x7f0cc40ab110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc4144330 0x7f0cc40ab650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc415aa00 con 0x7f0cc4144330 2024-09-13T16:37:08.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc40abb90 con 0x7f0cc40b3410 2024-09-13T16:37:08.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9e06700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc40ace70 con 0x7f0cc4143f50 2024-09-13T16:37:08.507 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc4144330 0x7f0cc40ab650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc4144330 0x7f0cc40ab650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44544/0 (socket says 172.21.15.119:44544) 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 -- 172.21.15.119:0/1712532561 learned_addr learned my addr 172.21.15.119:0/1712532561 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc3fff700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4143f50 0x7f0cc40ab110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc8e04700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b3410 0x7f0cc415a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 -- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4143f50 msgr2=0x7f0cc40ab110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4143f50 0x7f0cc40ab110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 -- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b3410 msgr2=0x7f0cc415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b3410 0x7f0cc415a4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 -- 172.21.15.119:0/1712532561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc0094b0 con 0x7f0cc4144330 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc3fff700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4143f50 0x7f0cc40ab110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.501+0000 7f0cc9605700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc4144330 0x7f0cc40ab650 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc014040 tx=0x7f0cbc004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.508 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc17fa700 1 -- 172.21.15.119:0/1712532561 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc0036c0 con 0x7f0cc4144330 2024-09-13T16:37:08.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc17fa700 1 -- 172.21.15.119:0/1712532561 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cbc01e070 con 0x7f0cc4144330 2024-09-13T16:37:08.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc40abdd0 con 0x7f0cc4144330 2024-09-13T16:37:08.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc17fa700 1 -- 172.21.15.119:0/1712532561 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc003a60 con 0x7f0cc4144330 2024-09-13T16:37:08.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc40ac390 con 0x7f0cc4144330 2024-09-13T16:37:08.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc40cd070 con 0x7f0cc4144330 2024-09-13T16:37:08.509 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc17fa700 1 -- 172.21.15.119:0/1712532561 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f0cbc003bc0 con 0x7f0cc4144330 2024-09-13T16:37:08.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc17fa700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ca4066730 0x7f0ca4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.505+0000 7f0cc17fa700 1 -- 172.21.15.119:0/1712532561 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f0cbc0850a0 con 0x7f0cc4144330 2024-09-13T16:37:08.510 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.509+0000 7f0cc8e04700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ca4066730 0x7f0ca4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.511 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.509+0000 7f0cc8e04700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ca4066730 0x7f0ca4068bf0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4005930 tx=0x7f0cb4009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.517 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.513+0000 7f0cc17fa700 1 -- 172.21.15.119:0/1712532561 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f0cbc052510 con 0x7f0cc4144330 2024-09-13T16:37:08.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.681+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cc40b1ee0 con 0x7f0ca4066730 2024-09-13T16:37:08.685 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.681+0000 7f0cc17fa700 1 -- 172.21.15.119:0/1712532561 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0cb4005960 con 0x7f0ca4066730 2024-09-13T16:37:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ca4066730 msgr2=0x7f0ca4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ca4066730 0x7f0ca4068bf0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4005930 tx=0x7f0cb4009040 comp rx=0 tx=0).stop 2024-09-13T16:37:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc4144330 msgr2=0x7f0cc40ab650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc4144330 0x7f0cc40ab650 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc014040 tx=0x7f0cbc004770 comp rx=0 tx=0).stop 2024-09-13T16:37:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 shutdown_connections 2024-09-13T16:37:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f0ca4066730 0x7f0ca4068bf0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.688 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f0cc4144330 0x7f0cc40ab650 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f0cc4143f50 0x7f0cc40ab110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 --2- 172.21.15.119:0/1712532561 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f0cc40b3410 0x7f0cc415a4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 >> 172.21.15.119:0/1712532561 conn(0x7f0cc406a350 msgr2=0x7f0cc406a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 shutdown_connections 2024-09-13T16:37:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.685+0000 7f0cc9e06700 1 -- 172.21.15.119:0/1712532561 wait complete. 2024-09-13T16:37:08.689 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 5 to deep-scrub 2024-09-13T16:37:08.699 INFO:tasks.ceph:Scrubbing osd.6 2024-09-13T16:37:08.700 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:08.810 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0bce9f700 1 Processor -- start 2024-09-13T16:37:08.811 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0bce9f700 1 -- start start 2024-09-13T16:37:08.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0bce9f700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7ff0b80ad9c0 0x7ff0b80adda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0bce9f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80ae2e0 0x7ff0b80af000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0bce9f700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b80aeb70 con 0x7ff0b8072300 2024-09-13T16:37:08.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0bce9f700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b8074110 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.813 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0bce9f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b806c060 con 0x7ff0b80ae2e0 2024-09-13T16:37:08.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b7fff700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7ff0b8072300 0x7ff0b80af540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41342/0 (socket says 172.21.15.119:41342) 2024-09-13T16:37:08.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b7fff700 1 -- 172.21.15.119:0/1131888320 learned_addr learned my addr 172.21.15.119:0/1131888320 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:08.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b77fe700 1 --2- 172.21.15.119:0/1131888320 >> v2:172.21.15.145:3300/0 conn(0x7ff0b80ad9c0 0x7ff0b80adda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.814 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b6ffd700 1 --2- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80ae2e0 0x7ff0b80af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b67fc700 1 -- 172.21.15.119:0/1131888320 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3739817489 0 0) 0x7ff0b80aeb70 con 0x7ff0b8072300 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b67fc700 1 -- 172.21.15.119:0/1131888320 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a00035d0 con 0x7ff0b8072300 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b77fe700 1 -- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80ae2e0 msgr2=0x7ff0b80af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b77fe700 1 --2- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80ae2e0 0x7ff0b80af000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b77fe700 1 -- 172.21.15.119:0/1131888320 >> v1:172.21.15.119:6789/0 conn(0x7ff0b8072300 legacy=0x7ff0b80af540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b77fe700 1 -- 172.21.15.119:0/1131888320 --> v2:172.21.15.145:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b8073b50 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.809+0000 7ff0b77fe700 1 --2- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b80adda0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009d90 tx=0x7ff0a800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=31113e0501b93213 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0b67fc700 1 -- 172.21.15.119:0/1131888320 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800e070 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0b67fc700 1 -- 172.21.15.119:0/1131888320 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a800bb20 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.815 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0b67fc700 1 -- 172.21.15.119:0/1131888320 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800be90 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 -- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ad9c0 msgr2=0x7ff0b80adda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 --2- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b80adda0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8009d90 tx=0x7ff0a800b330 comp rx=0 tx=0).stop 2024-09-13T16:37:08.816 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 -- 172.21.15.119:0/1131888320 shutdown_connections 2024-09-13T16:37:08.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 --2- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b80ae2e0 0x7ff0b80af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 --2- 172.21.15.119:0/1131888320 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b80adda0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 -- 172.21.15.119:0/1131888320 >> 172.21.15.119:0/1131888320 conn(0x7ff0b80717e0 msgr2=0x7ff0b8071bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 -- 172.21.15.119:0/1131888320 shutdown_connections 2024-09-13T16:37:08.817 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.813+0000 7ff0bce9f700 1 -- 172.21.15.119:0/1131888320 wait complete. 2024-09-13T16:37:08.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 Processor -- start 2024-09-13T16:37:08.819 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 -- start start 2024-09-13T16:37:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b8072300 0x7ff0b8098910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b806db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ae2e0 0x7ff0b81f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b81f0d10 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8155d80 con 0x7ff0b80ae2e0 2024-09-13T16:37:08.820 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b8150160 con 0x7ff0b8072300 2024-09-13T16:37:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b806db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b806db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44546/0 (socket says 172.21.15.119:44546) 2024-09-13T16:37:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 -- 172.21.15.119:0/833900210 learned_addr learned my addr 172.21.15.119:0/833900210 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b77fe700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b8072300 0x7ff0b8098910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.821 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b7fff700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ae2e0 0x7ff0b81f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 -- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b8072300 msgr2=0x7ff0b8098910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7ff0b8072300 0x7ff0b8098910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 -- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ae2e0 msgr2=0x7ff0b81f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7ff0b80ae2e0 0x7ff0b81f07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 -- 172.21.15.119:0/833900210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0a8009a70 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0b6ffd700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b806db70 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009fd0 tx=0x7ff0ac00eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac0095a0 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.822 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac013680 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b806e270 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b81534e0 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.817+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac0139f0 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff0b80b50f0 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.823 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7ff0ac013b50 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff09ffff700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff0a4066730 0x7ff0a4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7ff0ac0842b0 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff09ffff700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7ff0a406cb70 0x7ff0a406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff0a406f680 con 0x7ff0a406cb70 2024-09-13T16:37:08.824 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff0ac086890 con 0x7ff0b80ad9c0 2024-09-13T16:37:08.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff0b77fe700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff0a4066730 0x7ff0a4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff0b7fff700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7ff0a406cb70 0x7ff0a406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff0b77fe700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff0a4066730 0x7ff0a4068bf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8014040 tx=0x7ff0a8011000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.825 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.821+0000 7ff0b7fff700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7ff0a406cb70 0x7ff0a406efb0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.828 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.825+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== osd.6 v2:172.21.15.145:6800/3152548227 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7ff0a406f680 con 0x7ff0a406cb70 2024-09-13T16:37:08.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 --> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff0b80ae800 con 0x7ff0a406cb70 2024-09-13T16:37:08.845 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff09ffff700 1 -- 172.21.15.119:0/833900210 <== osd.6 v2:172.21.15.145:6800/3152548227 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7ff0b0004890 con 0x7ff0a406cb70 2024-09-13T16:37:08.846 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:08.846 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:08.846 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7ff0a406cb70 msgr2=0x7ff0a406efb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6800/3152548227,v1:172.21.15.145:6801/3152548227] conn(0x7ff0a406cb70 0x7ff0a406efb0 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff0a4066730 msgr2=0x7ff0a4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7ff0a4066730 0x7ff0a4068bf0 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7ff0a8014040 tx=0x7ff0a8011000 comp rx=0 tx=0).stop 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80ad9c0 msgr2=0x7ff0b806db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 --2- 172.21.15.119:0/833900210 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff0b80ad9c0 0x7ff0b806db70 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac009fd0 tx=0x7ff0ac00eb20 comp rx=0 tx=0).stop 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0b7fff700 1 -- 172.21.15.119:0/833900210 reap_dead start 2024-09-13T16:37:08.847 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 shutdown_connections 2024-09-13T16:37:08.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 >> 172.21.15.119:0/833900210 conn(0x7ff0b80717e0 msgr2=0x7ff0b815b410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 shutdown_connections 2024-09-13T16:37:08.848 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.841+0000 7ff0bce9f700 1 -- 172.21.15.119:0/833900210 wait complete. 2024-09-13T16:37:08.858 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-13T16:37:08.975 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7f59e700 1 Processor -- start 2024-09-13T16:37:08.976 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7f59e700 1 -- start start 2024-09-13T16:37:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7f59e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2a800b2c10 0x7f2a800b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7f59e700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f2a80158f10 0x7f2a8015c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7f59e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a800b3530 con 0x7f2a80070980 2024-09-13T16:37:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7f59e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a800b2940 con 0x7f2a800b2c10 2024-09-13T16:37:08.977 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7f59e700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a800b2380 con 0x7f2a80158f10 2024-09-13T16:37:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2a800b2c10 0x7f2a800b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2a800b2c10 0x7f2a800b2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44552/0 (socket says 172.21.15.119:44552) 2024-09-13T16:37:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 -- 172.21.15.119:0/721801380 learned_addr learned my addr 172.21.15.119:0/721801380 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7dd9b700 1 --2- 172.21.15.119:0/721801380 >> v2:172.21.15.145:3300/0 conn(0x7f2a80158f10 0x7f2a8015c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 -- 172.21.15.119:0/721801380 >> v2:172.21.15.145:3300/0 conn(0x7f2a80158f10 msgr2=0x7f2a8015c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 --2- 172.21.15.119:0/721801380 >> v2:172.21.15.145:3300/0 conn(0x7f2a80158f10 0x7f2a8015c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 -- 172.21.15.119:0/721801380 >> v1:172.21.15.145:6789/0 conn(0x7f2a80070980 legacy=0x7f2a8015cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 -- 172.21.15.119:0/721801380 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a800b2660 con 0x7f2a800b2c10 2024-09-13T16:37:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7e59c700 1 --2- 172.21.15.119:0/721801380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a800b2c10 0x7f2a800b2ff0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f2a80071da0 tx=0x7f2a7000be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=108c14a3f70e3608 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7d59a700 1 -- 172.21.15.119:0/721801380 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7000dd30 con 0x7f2a800b2c10 2024-09-13T16:37:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7d59a700 1 -- 172.21.15.119:0/721801380 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7000de90 con 0x7f2a800b2c10 2024-09-13T16:37:08.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.973+0000 7f2a7d59a700 1 -- 172.21.15.119:0/721801380 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70010690 con 0x7f2a800b2c10 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 -- 172.21.15.119:0/721801380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a800b2c10 msgr2=0x7f2a800b2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/721801380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a800b2c10 0x7f2a800b2ff0 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f2a80071da0 tx=0x7f2a7000be90 comp rx=0 tx=0).stop 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 -- 172.21.15.119:0/721801380 shutdown_connections 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/721801380 >> v2:172.21.15.145:3300/0 conn(0x7f2a80158f10 0x7f2a8015c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/721801380 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a800b2c10 0x7f2a800b2ff0 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 -- 172.21.15.119:0/721801380 >> 172.21.15.119:0/721801380 conn(0x7f2a8006a350 msgr2=0x7f2a8006a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 -- 172.21.15.119:0/721801380 shutdown_connections 2024-09-13T16:37:08.980 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 -- 172.21.15.119:0/721801380 wait complete. 2024-09-13T16:37:08.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 Processor -- start 2024-09-13T16:37:08.981 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.977+0000 7f2a7f59e700 1 -- start start 2024-09-13T16:37:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a80070980 0x7f2a8014d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a800b2c10 0x7f2a800cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a80158f10 0x7f2a800cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a800a8ac0 con 0x7f2a80070980 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8015b4a0 con 0x7f2a800b2c10 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8015b780 con 0x7f2a80158f10 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a80070980 0x7f2a8014d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a80070980 0x7f2a8014d080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44560/0 (socket says 172.21.15.119:44560) 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 -- 172.21.15.119:0/3687755781 learned_addr learned my addr 172.21.15.119:0/3687755781 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7ed9d700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a80158f10 0x7f2a800cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7dd9b700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a800b2c10 0x7f2a800cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 -- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a80158f10 msgr2=0x7f2a800cd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a80158f10 0x7f2a800cd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.984 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 -- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a800b2c10 msgr2=0x7f2a800cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:08.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a800b2c10 0x7f2a800cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:08.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 -- 172.21.15.119:0/3687755781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7000b810 con 0x7f2a80070980 2024-09-13T16:37:08.985 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7e59c700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a80070980 0x7f2a8014d080 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f2a800b3ee0 tx=0x7f2a70004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a76ffd700 1 -- 172.21.15.119:0/3687755781 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7001f070 con 0x7f2a80070980 2024-09-13T16:37:08.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a76ffd700 1 -- 172.21.15.119:0/3687755781 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a700038d0 con 0x7f2a80070980 2024-09-13T16:37:08.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8015ba60 con 0x7f2a80070980 2024-09-13T16:37:08.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a76ffd700 1 -- 172.21.15.119:0/3687755781 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a70003c60 con 0x7f2a80070980 2024-09-13T16:37:08.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8015c020 con 0x7f2a80070980 2024-09-13T16:37:08.986 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.981+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a800ad9c0 con 0x7f2a80070980 2024-09-13T16:37:08.987 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.985+0000 7f2a76ffd700 1 -- 172.21.15.119:0/3687755781 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f2a70019400 con 0x7f2a80070980 2024-09-13T16:37:08.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.985+0000 7f2a76ffd700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a60066730 0x7f2a60068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:08.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.985+0000 7f2a76ffd700 1 -- 172.21.15.119:0/3687755781 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f2a70087710 con 0x7f2a80070980 2024-09-13T16:37:08.988 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.985+0000 7f2a7dd9b700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a60066730 0x7f2a60068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:08.989 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.985+0000 7f2a7dd9b700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a60066730 0x7f2a60068bf0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2a8006a900 tx=0x7f2a68009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:08.990 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:08.989+0000 7f2a76ffd700 1 -- 172.21.15.119:0/3687755781 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f2a70054b80 con 0x7f2a80070980 2024-09-13T16:37:09.158 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.153+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a8015d500 con 0x7f2a60066730 2024-09-13T16:37:09.159 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a76ffd700 1 -- 172.21.15.119:0/3687755781 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2a68004020 con 0x7f2a60066730 2024-09-13T16:37:09.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a60066730 msgr2=0x7f2a60068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a60066730 0x7f2a60068bf0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f2a8006a900 tx=0x7f2a68009240 comp rx=0 tx=0).stop 2024-09-13T16:37:09.161 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a80070980 msgr2=0x7f2a8014d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a80070980 0x7f2a8014d080 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7f2a800b3ee0 tx=0x7f2a70004770 comp rx=0 tx=0).stop 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 shutdown_connections 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f2a60066730 0x7f2a60068bf0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f2a80158f10 0x7f2a800cd630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f2a800b2c10 0x7f2a800cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 --2- 172.21.15.119:0/3687755781 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2a80070980 0x7f2a8014d080 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 >> 172.21.15.119:0/3687755781 conn(0x7f2a8006a350 msgr2=0x7f2a8014f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 shutdown_connections 2024-09-13T16:37:09.162 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.157+0000 7f2a7f59e700 1 -- 172.21.15.119:0/3687755781 wait complete. 2024-09-13T16:37:09.163 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 6 to deep-scrub 2024-09-13T16:37:09.174 INFO:tasks.ceph:Scrubbing osd.7 2024-09-13T16:37:09.174 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-13T16:37:09.290 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.285+0000 7f59171ba700 1 Processor -- start 2024-09-13T16:37:09.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- start start 2024-09-13T16:37:09.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f591015a7d0 0x7f591006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f591006bee0 0x7f59100cb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59100cb840 0x7f59100cc790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.293 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59100ce1c0 con 0x7f591006bee0 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59100cbcb0 con 0x7f591015a7d0 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59100b2d10 con 0x7f59100cb840 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f591006bee0 0x7f59100cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f591006bee0 0x7f59100cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44572/0 (socket says 172.21.15.119:44572) 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 -- 172.21.15.119:0/2450902408 learned_addr learned my addr 172.21.15.119:0/2450902408 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59169b9700 1 --2- 172.21.15.119:0/2450902408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59100cb840 0x7f59100cc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59161b8700 1 --2- 172.21.15.119:0/2450902408 >> v2:172.21.15.145:3300/0 conn(0x7f591015a7d0 0x7f591006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 -- 172.21.15.119:0/2450902408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59100cb840 msgr2=0x7f59100cc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 --2- 172.21.15.119:0/2450902408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59100cb840 0x7f59100cc790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 -- 172.21.15.119:0/2450902408 >> v2:172.21.15.145:3300/0 conn(0x7f591015a7d0 msgr2=0x7f591006b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 --2- 172.21.15.119:0/2450902408 >> v2:172.21.15.145:3300/0 conn(0x7f591015a7d0 0x7f591006b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 -- 172.21.15.119:0/2450902408 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59100b1e20 con 0x7f591006bee0 2024-09-13T16:37:09.294 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59159b7700 1 --2- 172.21.15.119:0/2450902408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f591006bee0 0x7f59100cb300 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f5910071f50 tx=0x7f5904009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86908ba45d3f9efa server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59151b6700 1 -- 172.21.15.119:0/2450902408 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590400ba20 con 0x7f591006bee0 2024-09-13T16:37:09.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59151b6700 1 -- 172.21.15.119:0/2450902408 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590400bb80 con 0x7f591006bee0 2024-09-13T16:37:09.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59151b6700 1 -- 172.21.15.119:0/2450902408 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904009e60 con 0x7f591006bee0 2024-09-13T16:37:09.295 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- 172.21.15.119:0/2450902408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f591006bee0 msgr2=0x7f59100cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 --2- 172.21.15.119:0/2450902408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f591006bee0 0x7f59100cb300 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7f5910071f50 tx=0x7f5904009c20 comp rx=0 tx=0).stop 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- 172.21.15.119:0/2450902408 shutdown_connections 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 --2- 172.21.15.119:0/2450902408 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f59100cb840 0x7f59100cc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 --2- 172.21.15.119:0/2450902408 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f591006bee0 0x7f59100cb300 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 --2- 172.21.15.119:0/2450902408 >> v2:172.21.15.145:3300/0 conn(0x7f591015a7d0 0x7f591006b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- 172.21.15.119:0/2450902408 >> 172.21.15.119:0/2450902408 conn(0x7f591006a350 msgr2=0x7f591006a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- 172.21.15.119:0/2450902408 shutdown_connections 2024-09-13T16:37:09.296 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.289+0000 7f59171ba700 1 -- 172.21.15.119:0/2450902408 wait complete. 2024-09-13T16:37:09.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 Processor -- start 2024-09-13T16:37:09.297 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 -- start start 2024-09-13T16:37:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591006bee0 0x7f59101f6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59100cb840 0x7f591013eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591015a7d0 0x7f591006fc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.300 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910070470 con 0x7f59100cb840 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910070750 con 0x7f591006bee0 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59171ba700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5910070a30 con 0x7f591015a7d0 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59100cb840 0x7f591013eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59100cb840 0x7f591013eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44584/0 (socket says 172.21.15.119:44584) 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 -- 172.21.15.119:0/856001620 learned_addr learned my addr 172.21.15.119:0/856001620 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59169b9700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591015a7d0 0x7f591006fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59161b8700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591006bee0 0x7f59101f6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 -- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591015a7d0 msgr2=0x7f591006fc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591015a7d0 0x7f591006fc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 -- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591006bee0 msgr2=0x7f59101f6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591006bee0 0x7f59101f6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.301 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59159b7700 1 -- 172.21.15.119:0/856001620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5904009570 con 0x7f59100cb840 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59169b9700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f591015a7d0 0x7f591006fc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.293+0000 7f59161b8700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f591006bee0 0x7f59101f6380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f59159b7700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59100cb840 0x7f591013eb00 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f5904009c60 tx=0x7f5904004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5904004480 con 0x7f59100cb840 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590401e070 con 0x7f59100cb840 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5910070d10 con 0x7f59100cb840 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59040038e0 con 0x7f59100cb840 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5910071510 con 0x7f59100cb840 2024-09-13T16:37:09.302 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f59100b26c0 con 0x7f59100cb840 2024-09-13T16:37:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f5904020030 con 0x7f59100cb840 2024-09-13T16:37:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.297+0000 7f5902ffd700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f58f4066730 0x7f58f4068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f5904085760 con 0x7f59100cb840 2024-09-13T16:37:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f59161b8700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f58f4066730 0x7f58f4068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f5902ffd700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7f58f406cb70 0x7f58f406efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f58f406f680 con 0x7f58f406cb70 2024-09-13T16:37:09.304 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_get_version_reply(handle=1 version=131) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5904085bb0 con 0x7f59100cb840 2024-09-13T16:37:09.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f59169b9700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7f58f406cb70 0x7f58f406efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f59161b8700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f58f4066730 0x7f58f4068bf0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f590c009990 tx=0x7f590c008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f59169b9700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7f58f406cb70 0x7f58f406efb0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.305 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.301+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== osd.7 v2:172.21.15.145:6808/1049327118 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (crc 0 0 0) 0x7f58f406f680 con 0x7f58f406cb70 2024-09-13T16:37:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 --> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f59100b23e0 con 0x7f58f406cb70 2024-09-13T16:37:09.321 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f5902ffd700 1 -- 172.21.15.119:0/856001620 <== osd.7 v2:172.21.15.145:6808/1049327118 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (crc 0 0 0) 0x7f5908009c30 con 0x7f58f406cb70 2024-09-13T16:37:09.322 INFO:teuthology.orchestra.run.smithi119.stdout:{ 2024-09-13T16:37:09.322 INFO:teuthology.orchestra.run.smithi119.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-13T16:37:09.322 INFO:teuthology.orchestra.run.smithi119.stdout:} 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7f58f406cb70 msgr2=0x7f58f406efb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6808/1049327118,v1:172.21.15.145:6809/1049327118] conn(0x7f58f406cb70 0x7f58f406efb0 crc :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f58f4066730 msgr2=0x7f58f4068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f58f4066730 0x7f58f4068bf0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f590c009990 tx=0x7f590c008040 comp rx=0 tx=0).stop 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59100cb840 msgr2=0x7f591013eb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 --2- 172.21.15.119:0/856001620 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f59100cb840 0x7f591013eb00 secure :-1 s=READY pgs=2057 cs=0 l=1 rev1=1 crypto rx=0x7f5904009c60 tx=0x7f5904004770 comp rx=0 tx=0).stop 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59169b9700 1 -- 172.21.15.119:0/856001620 reap_dead start 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 shutdown_connections 2024-09-13T16:37:09.323 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.317+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 >> 172.21.15.119:0/856001620 conn(0x7f591006a350 msgr2=0x7f59100ca410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:09.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.321+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 shutdown_connections 2024-09-13T16:37:09.324 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.321+0000 7f59171ba700 1 -- 172.21.15.119:0/856001620 wait complete. 2024-09-13T16:37:09.334 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-13T16:37:09.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf30e83700 1 Processor -- start 2024-09-13T16:37:09.452 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf30e83700 1 -- start start 2024-09-13T16:37:09.454 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf30e83700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fcf2c143f50 0x7fcf2c06c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf30e83700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcf2c0b3410 0x7fcf2c164330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf30e83700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf2c0b1c40 con 0x7fcf2c144330 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf30e83700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c06cdf0 con 0x7fcf2c0b3410 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf30e83700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf2c06d0d0 con 0x7fcf2c143f50 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcf2c0b3410 0x7fcf2c164330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2affd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcf2c144330 0x7fcf2c0cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41356/0 (socket says 172.21.15.119:41356) 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2affd700 1 -- 172.21.15.119:0/2888786388 learned_addr learned my addr 172.21.15.119:0/2888786388 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:09.455 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcf2c0b3410 0x7fcf2c164330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44596/0 (socket says 172.21.15.119:44596) 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2b7fe700 1 --2- 172.21.15.119:0/2888786388 >> v2:172.21.15.145:3300/0 conn(0x7fcf2c143f50 0x7fcf2c06c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2bfff700 1 -- 172.21.15.119:0/2888786388 >> v2:172.21.15.145:3300/0 conn(0x7fcf2c143f50 msgr2=0x7fcf2c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2bfff700 1 --2- 172.21.15.119:0/2888786388 >> v2:172.21.15.145:3300/0 conn(0x7fcf2c143f50 0x7fcf2c06c5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2a7fc700 1 -- 172.21.15.119:0/2888786388 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1078088852 0 0) 0x7fcf2c0b1c40 con 0x7fcf2c144330 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2bfff700 1 -- 172.21.15.119:0/2888786388 >> v1:172.21.15.119:6789/0 conn(0x7fcf2c144330 legacy=0x7fcf2c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2bfff700 1 -- 172.21.15.119:0/2888786388 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf2c06d3b0 con 0x7fcf2c0b3410 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.449+0000 7fcf2bfff700 1 --2- 172.21.15.119:0/2888786388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c0b3410 0x7fcf2c164330 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c06b1f0 tx=0x7fcf24009b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae1e6b2265a05f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.456 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2a7fc700 1 -- 172.21.15.119:0/2888786388 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24011830 con 0x7fcf2c0b3410 2024-09-13T16:37:09.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2a7fc700 1 -- 172.21.15.119:0/2888786388 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf24011990 con 0x7fcf2c0b3410 2024-09-13T16:37:09.457 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2a7fc700 1 -- 172.21.15.119:0/2888786388 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24011d20 con 0x7fcf2c0b3410 2024-09-13T16:37:09.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- 172.21.15.119:0/2888786388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c0b3410 msgr2=0x7fcf2c164330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2888786388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c0b3410 0x7fcf2c164330 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c06b1f0 tx=0x7fcf24009b30 comp rx=0 tx=0).stop 2024-09-13T16:37:09.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- 172.21.15.119:0/2888786388 shutdown_connections 2024-09-13T16:37:09.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2888786388 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c0b3410 0x7fcf2c164330 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2888786388 >> v2:172.21.15.145:3300/0 conn(0x7fcf2c143f50 0x7fcf2c06c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.460 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- 172.21.15.119:0/2888786388 >> 172.21.15.119:0/2888786388 conn(0x7fcf2c06a350 msgr2=0x7fcf2c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- 172.21.15.119:0/2888786388 shutdown_connections 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- 172.21.15.119:0/2888786388 wait complete. 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 Processor -- start 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- start start 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf2c0b3410 0x7fcf2c0c97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf2c143f50 0x7fcf2c15a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c144330 0x7fcf2c0c7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c87e0 con 0x7fcf2c144330 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c7950 con 0x7fcf2c143f50 2024-09-13T16:37:09.461 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf30e83700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c0c7bd0 con 0x7fcf2c0b3410 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c144330 0x7fcf2c0c7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2b7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf2c0b3410 0x7fcf2c0c97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c144330 0x7fcf2c0c7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44608/0 (socket says 172.21.15.119:44608) 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2b7fe700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf2c0b3410 0x7fcf2c0c97c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:39696/0 (socket says 172.21.15.119:39696) 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2b7fe700 1 -- 172.21.15.119:0/2412239261 learned_addr learned my addr 172.21.15.119:0/2412239261 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2affd700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf2c143f50 0x7fcf2c15a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2bfff700 1 -- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf2c0b3410 msgr2=0x7fcf2c0c97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2bfff700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf2c0b3410 0x7fcf2c0c97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2bfff700 1 -- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf2c143f50 msgr2=0x7fcf2c15a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2bfff700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf2c143f50 0x7fcf2c15a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.453+0000 7fcf2bfff700 1 -- 172.21.15.119:0/2412239261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf240094b0 con 0x7fcf2c144330 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf2bfff700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c144330 0x7fcf2c0c7410 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0cd590 tx=0x7fcf24004770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.462 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf23fff700 1 -- 172.21.15.119:0/2412239261 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf240044d0 con 0x7fcf2c144330 2024-09-13T16:37:09.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf23fff700 1 -- 172.21.15.119:0/2412239261 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf2401e070 con 0x7fcf2c144330 2024-09-13T16:37:09.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c0c7eb0 con 0x7fcf2c144330 2024-09-13T16:37:09.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf23fff700 1 -- 172.21.15.119:0/2412239261 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24003900 con 0x7fcf2c144330 2024-09-13T16:37:09.463 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c0c8470 con 0x7fcf2c144330 2024-09-13T16:37:09.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c0c8990 con 0x7fcf2c144330 2024-09-13T16:37:09.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf23fff700 1 -- 172.21.15.119:0/2412239261 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcf24003a60 con 0x7fcf2c144330 2024-09-13T16:37:09.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf23fff700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf0c066730 0x7fcf0c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf23fff700 1 -- 172.21.15.119:0/2412239261 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fcf240850b0 con 0x7fcf2c144330 2024-09-13T16:37:09.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf2b7fe700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf0c066730 0x7fcf0c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.464 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.457+0000 7fcf2b7fe700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf0c066730 0x7fcf0c068bf0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009a40 tx=0x7fcf1c00b040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.468 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.465+0000 7fcf23fff700 1 -- 172.21.15.119:0/2412239261 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcf2c0c8990 con 0x7fcf2c144330 2024-09-13T16:37:09.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.633+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf2c0a8bc0 con 0x7fcf0c066730 2024-09-13T16:37:09.638 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf23fff700 1 -- 172.21.15.119:0/2412239261 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fcf2c0a8bc0 con 0x7fcf0c066730 2024-09-13T16:37:09.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf0c066730 msgr2=0x7fcf0c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf0c066730 0x7fcf0c068bf0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c009a40 tx=0x7fcf1c00b040 comp rx=0 tx=0).stop 2024-09-13T16:37:09.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c144330 msgr2=0x7fcf2c0c7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c144330 0x7fcf2c0c7410 secure :-1 s=READY pgs=2060 cs=0 l=1 rev1=1 crypto rx=0x7fcf2c0cd590 tx=0x7fcf24004770 comp rx=0 tx=0).stop 2024-09-13T16:37:09.641 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 shutdown_connections 2024-09-13T16:37:09.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf0c066730 0x7fcf0c068bf0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcf2c144330 0x7fcf2c0c7410 unknown :-1 s=CLOSED pgs=2060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcf2c143f50 0x7fcf2c15a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 --2- 172.21.15.119:0/2412239261 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcf2c0b3410 0x7fcf2c0c97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 >> 172.21.15.119:0/2412239261 conn(0x7fcf2c06a350 msgr2=0x7fcf2c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:09.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 shutdown_connections 2024-09-13T16:37:09.642 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.637+0000 7fcf30e83700 1 -- 172.21.15.119:0/2412239261 wait complete. 2024-09-13T16:37:09.643 INFO:teuthology.orchestra.run.smithi119.stderr:instructed osd(s) 7 to deep-scrub 2024-09-13T16:37:09.654 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:37:09.769 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.765+0000 7f3891098700 1 Processor -- start 2024-09-13T16:37:09.770 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.765+0000 7f3891098700 1 -- start start 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.765+0000 7f3891098700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7f388c0b3410 0x7f388c165340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.765+0000 7f3891098700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c0b1c40 con 0x7f388c144330 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.765+0000 7f3891098700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f388c06cdf0 con 0x7f388c143f50 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.765+0000 7f3891098700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f388c06d0d0 con 0x7f388c0b3410 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f388c144330 0x7f388c0cda50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:41364/0 (socket says 172.21.15.119:41364) 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 -- 172.21.15.119:0/434945352 learned_addr learned my addr 172.21.15.119:0/434945352 (peer_addr_for_me v1:172.21.15.119:0/0) 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3890897700 1 --2- 172.21.15.119:0/434945352 >> v2:172.21.15.145:3300/0 conn(0x7f388c0b3410 0x7f388c165340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 <== mon.0 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1610234481 0 0) 0x7f388c0b1c40 con 0x7f388c144330 2024-09-13T16:37:09.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38780035d0 con 0x7f388c144330 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 <== mon.0 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209744022 0 0) 0x7f38780035d0 con 0x7f388c144330 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 >> v2:172.21.15.145:3300/0 conn(0x7f388c0b3410 msgr2=0x7f388c165340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 --2- 172.21.15.119:0/434945352 >> v2:172.21.15.145:3300/0 conn(0x7f388c0b3410 0x7f388c165340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 >> v1:172.21.15.145:6789/0 conn(0x7f388c143f50 legacy=0x7f388c06c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c06d3b0 con 0x7f388c144330 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 <== mon.0 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1942907670 0 0) 0x7f387c001670 con 0x7f388c144330 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 >> v1:172.21.15.119:6789/0 conn(0x7f388c144330 legacy=0x7f388c0cda50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878003a20 0x7f3878003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3878004870 0x7f3878004cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.774 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878005230 0x7f387800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c06d0d0 con 0x7f3878004870 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f387800c1d0 con 0x7f3878003a20 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f387800c390 con 0x7f3878005230 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3878004870 0x7f3878004cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3890897700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878005230 0x7f387800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388bfff700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878003a20 0x7f3878003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 -- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878005230 msgr2=0x7f387800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878005230 0x7f387800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 -- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878003a20 msgr2=0x7f3878003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878003a20 0x7f3878003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3890897700 1 -- 172.21.15.119:0/434945352 reap_dead start 2024-09-13T16:37:09.775 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 -- 172.21.15.119:0/434945352 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38780035d0 con 0x7f3878004870 2024-09-13T16:37:09.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388bfff700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f3878003a20 0x7f3878003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-13T16:37:09.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388b7fe700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3878004870 0x7f3878004cf0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f387c00c2a0 tx=0x7f38780058a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca09c3e2e6132859 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c007c30 con 0x7f3878004870 2024-09-13T16:37:09.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c00cd80 con 0x7f3878004870 2024-09-13T16:37:09.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f388affd700 1 -- 172.21.15.119:0/434945352 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c005bc0 con 0x7f3878004870 2024-09-13T16:37:09.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3891098700 1 -- 172.21.15.119:0/434945352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3878004870 msgr2=0x7f3878004cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.776 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3891098700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3878004870 0x7f3878004cf0 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7f387c00c2a0 tx=0x7f38780058a0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3891098700 1 -- 172.21.15.119:0/434945352 shutdown_connections 2024-09-13T16:37:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3891098700 1 --2- 172.21.15.119:0/434945352 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3878004870 0x7f3878004cf0 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3891098700 1 -- 172.21.15.119:0/434945352 >> 172.21.15.119:0/434945352 conn(0x7f388c06a350 msgr2=0x7f388c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.769+0000 7f3891098700 1 -- 172.21.15.119:0/434945352 shutdown_connections 2024-09-13T16:37:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 -- 172.21.15.119:0/434945352 wait complete. 2024-09-13T16:37:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 Processor -- start 2024-09-13T16:37:09.777 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 -- start start 2024-09-13T16:37:09.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878004870 0x7f388c15a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f388c0c3f70 0x7f388c0c43f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f388c0aa0c0 0x7f388c0aa540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0c4930 con 0x7f388c0c3f70 2024-09-13T16:37:09.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0aae30 con 0x7f388c0aa0c0 2024-09-13T16:37:09.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.773+0000 7f3891098700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f388c0ab5f0 con 0x7f3878004870 2024-09-13T16:37:09.779 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f388c0c3f70 0x7f388c0c43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f388c0c3f70 0x7f388c0c43f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:44626/0 (socket says 172.21.15.119:44626) 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 -- 172.21.15.119:0/695744977 learned_addr learned my addr 172.21.15.119:0/695744977 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388bfff700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878004870 0x7f388c15a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f3890897700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f388c0aa0c0 0x7f388c0aa540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 -- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878004870 msgr2=0x7f388c15a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878004870 0x7f388c15a330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 -- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f388c0aa0c0 msgr2=0x7f388c0aa540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f388c0aa0c0 0x7f388c0aa540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.780 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 -- 172.21.15.119:0/695744977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f388c0ab8d0 con 0x7f388c0c3f70 2024-09-13T16:37:09.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f388b7fe700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f388c0c3f70 0x7f388c0c43f0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f387c00c890 tx=0x7f388c0b1dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f3888ff9700 1 -- 172.21.15.119:0/695744977 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c001630 con 0x7f388c0c3f70 2024-09-13T16:37:09.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f3888ff9700 1 -- 172.21.15.119:0/695744977 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f387c0070d0 con 0x7f388c0c3f70 2024-09-13T16:37:09.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c0abbb0 con 0x7f388c0c3f70 2024-09-13T16:37:09.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f3888ff9700 1 -- 172.21.15.119:0/695744977 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f387c007440 con 0x7f388c0c3f70 2024-09-13T16:37:09.781 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c15a870 con 0x7f388c0c3f70 2024-09-13T16:37:09.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.777+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c06c630 con 0x7f388c0c3f70 2024-09-13T16:37:09.782 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.781+0000 7f3888ff9700 1 -- 172.21.15.119:0/695744977 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7f387c0075a0 con 0x7f388c0c3f70 2024-09-13T16:37:09.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.781+0000 7f3888ff9700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f386c066730 0x7f386c068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:09.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.781+0000 7f3888ff9700 1 -- 172.21.15.119:0/695744977 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7f387c085270 con 0x7f388c0c3f70 2024-09-13T16:37:09.783 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.781+0000 7f388bfff700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f386c066730 0x7f386c068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:09.784 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.781+0000 7f388bfff700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f386c066730 0x7f386c068bf0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3874007fd0 tx=0x7f3874009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:09.791 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.789+0000 7f3888ff9700 1 -- 172.21.15.119:0/695744977 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7f387c052820 con 0x7f388c0c3f70 2024-09-13T16:37:09.952 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.949+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f388c141610 con 0x7f386c066730 2024-09-13T16:37:09.957 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.953+0000 7f3888ff9700 1 -- 172.21.15.119:0/695744977 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+49043 (secure 0 0 0) 0x7f388c141610 con 0x7f386c066730 2024-09-13T16:37:09.958 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:37:09.960 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f386c066730 msgr2=0x7f386c068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f386c066730 0x7f386c068bf0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f3874007fd0 tx=0x7f3874009040 comp rx=0 tx=0).stop 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f388c0c3f70 msgr2=0x7f388c0c43f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f388c0c3f70 0x7f388c0c43f0 secure :-1 s=READY pgs=2063 cs=0 l=1 rev1=1 crypto rx=0x7f387c00c890 tx=0x7f388c0b1dd0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 shutdown_connections 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7f386c066730 0x7f386c068bf0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7f388c0aa0c0 0x7f388c0aa540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f388c0c3f70 0x7f388c0c43f0 unknown :-1 s=CLOSED pgs=2063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 --2- 172.21.15.119:0/695744977 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7f3878004870 0x7f388c15a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 >> 172.21.15.119:0/695744977 conn(0x7f388c06a350 msgr2=0x7f388c0a84b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 shutdown_connections 2024-09-13T16:37:09.961 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:09.957+0000 7f3891098700 1 -- 172.21.15.119:0/695744977 wait complete. 2024-09-13T16:37:09.962 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:37:09.974 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":484,"stamp":"2024-09-13T16:37:09.676464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":31,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262868,"kb_used_data":13928,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754711852,"statfs":{"total":773094113280,"available":772824936448,"internally_reserved":0,"allocated":14262272,"data_stored":7232152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8388625,"num_objects":-5,"num_object_clones":0,"num_object_copies":-10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-5,"num_whiteouts":0,"num_read":-32,"num_read_kb":-13,"num_write":-29,"num_write_kb":-8195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.877039"},"pg_stats":[{"pgid":"1.0","version":"17'117","reported_seq":262,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-13T16:36:59.086750+0000","last_change":"2024-09-13T16:25:13.156476+0000","last_active":"2024-09-13T16:36:59.086750+0000","last_peered":"2024-09-13T16:36:59.086750+0000","last_clean":"2024-09-13T16:36:59.086750+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:36:59.086750+0000","last_undegraded":"2024-09-13T16:36:59.086750+0000","last_fullsized":"2024-09-13T16:36:59.086750+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-13T16:25:11.599970+0000","last_clean_scrub_stamp":"2024-09-13T16:25:11.599970+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T18:01:53.429537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542290,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33292,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338548,"statfs":{"total":96636764160,"available":96602673152,"internally_reserved":0,"allocated":2228224,"data_stored":1346795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.80300000000000005,"15min":0.83699999999999997},"min":{"1min":0.51800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.198,"5min":1.262,"15min":1.357},"last":0.81999999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.52000000000000002,"15min":0.47699999999999998},"max":{"1min":1.1759999999999999,"5min":1.214,"15min":1.3460000000000001},"last":1.0209999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.56000000000000005,"15min":0.501},"max":{"1min":1.115,"5min":1.212,"15min":1.569},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.89100000000000001,"15min":0.86499999999999999},"min":{"1min":0.63600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.115,"5min":1.2290000000000001,"15min":1.4219999999999999},"last":0.876}]},{"osd":2,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.874},"min":{"1min":0.56699999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.113,"5min":1.1919999999999999,"15min":1.242},"last":1.0700000000000001},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.89100000000000001,"15min":0.87},"min":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.53900000000000003},"max":{"1min":1.238,"5min":1.24,"15min":1.339},"last":1.044}]},{"osd":3,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91300000000000003,"15min":0.95699999999999996},"min":{"1min":0.65300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3,"5min":1.3,"15min":1.476},"last":0.996},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.998,"15min":1},"min":{"1min":0.73599999999999999,"5min":0.69599999999999995,"15min":0.65700000000000003},"max":{"1min":1.28,"5min":1.2969999999999999,"15min":1.3939999999999999},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.878,"15min":0.874},"min":{"1min":0.71399999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.089,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.93400000000000005},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.94399999999999995,"15min":1.006},"min":{"1min":0.73299999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.216,"5min":1.381,"15min":1.589},"last":0.68799999999999994}]},{"osd":5,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.98199999999999998,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2230000000000001,"5min":1.444,"15min":1.444},"last":0.95599999999999996},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0720000000000001,"15min":1.091},"min":{"1min":0.99399999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.278,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":0.65400000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0369999999999999,"15min":1.0529999999999999},"min":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2330000000000001,"5min":1.72,"15min":1.72},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.044,"15min":1.1080000000000001},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.375,"5min":1.542,"15min":1.6779999999999999},"last":0.90500000000000003}]}]},{"osd":6,"up_from":14,"seq":60129542289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.84499999999999997,"15min":0.84099999999999997},"min":{"1min":0.53300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.85799999999999998,"15min":0.878},"min":{"1min":0.55800000000000005,"5min":0.309,"15min":0.309},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.4410000000000001},"last":0.80600000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":0.84599999999999997},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0509999999999999,"5min":1.153,"15min":1.2030000000000001},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.88200000000000001,"15min":0.90800000000000003},"min":{"1min":0.57699999999999996,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1599999999999999,"5min":1.411,"15min":1.411},"last":0.77200000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.86299999999999999,"15min":0.86899999999999999},"min":{"1min":0.57899999999999996,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.262,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.874},"min":{"1min":0.53000000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0940000000000001,"5min":1.399,"15min":1.399},"last":0.73599999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.92100000000000004,"15min":0.94899999999999995},"min":{"1min":0.58099999999999996,"5min":0.379,"15min":0.379},"max":{"1min":1.1870000000000001,"5min":1.544,"15min":1.544},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.97499999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.21,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.68200000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95599999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1599999999999999,"5min":1.508,"15min":1.508},"last":0.70599999999999996},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.65500000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.343,"5min":1.55,"15min":1.6140000000000001},"last":1.0169999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.92300000000000004,"15min":0.93100000000000005},"min":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.214,"5min":1.339,"15min":1.339},"last":0.91500000000000004},{"interface":"front","average":{"1min":1.052,"5min":1.0580000000000001,"15min":1.089},"min":{"1min":0.70299999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2829999999999999,"5min":1.615,"15min":1.6439999999999999},"last":0.88800000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.048,"15min":1.087},"min":{"1min":0.72799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.522},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.052,"15min":1.0589999999999999},"min":{"1min":0.68899999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.427},"last":1.234}]}]},{"osd":5,"up_from":14,"seq":60129542291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.83199999999999996,"5min":0.84299999999999997,"15min":0.84799999999999998},"min":{"1min":0.58299999999999996,"5min":0.441,"15min":0.441},"max":{"1min":1.0589999999999999,"5min":1.1899999999999999,"15min":1.411},"last":0.87},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.83399999999999996,"15min":0.83899999999999997},"min":{"1min":0.53100000000000003,"5min":0.497,"15min":0.41699999999999998},"max":{"1min":1.046,"5min":1.3029999999999999,"15min":1.675},"last":1.04}]},{"osd":1,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.84799999999999998,"15min":0.86199999999999999},"min":{"1min":0.60399999999999998,"5min":0.51900000000000002,"15min":0.46600000000000003},"max":{"1min":1.0840000000000001,"5min":1.2190000000000001,"15min":1.4079999999999999},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.83199999999999996,"15min":0.83899999999999997},"min":{"1min":0.61299999999999999,"5min":0.53700000000000003,"15min":0.36499999999999999},"max":{"1min":1.085,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.80100000000000005}]},{"osd":2,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.90400000000000003,"15min":0.91600000000000004},"min":{"1min":0.441,"5min":0.441,"15min":0.441},"max":{"1min":1.4379999999999999,"5min":1.4379999999999999,"15min":1.625},"last":0.629},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.97599999999999998,"15min":0.97099999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.159,"5min":1.331,"15min":1.552},"last":1.0740000000000001}]},{"osd":3,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.90500000000000003,"15min":0.879},"min":{"1min":0.57199999999999995,"5min":0.5,"15min":0.5},"max":{"1min":1.23,"5min":1.23,"15min":1.3919999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.89000000000000001,"5min":0.88600000000000001,"15min":0.94199999999999995},"min":{"1min":0.70199999999999996,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.286,"5min":1.2929999999999999,"15min":3.8460000000000001},"last":0.89000000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.88600000000000001,"5min":0.88200000000000001,"15min":0.96799999999999997},"min":{"1min":0.61199999999999999,"5min":0.58199999999999996,"15min":0.56999999999999995},"max":{"1min":1.2030000000000001,"5min":1.3540000000000001,"15min":3.6869999999999998},"last":0.97599999999999998},{"interface":"front","average":{"1min":1.073,"5min":1.0669999999999999,"15min":1.0640000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.57499999999999996},"max":{"1min":1.331,"5min":1.365,"15min":1.601},"last":1.141}]},{"osd":6,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.91800000000000004,"15min":0.94599999999999995},"min":{"1min":0.64300000000000002,"5min":0.61899999999999999,"15min":0.60899999999999999},"max":{"1min":1.5069999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.0049999999999999},{"interface":"front","average":{"1min":0.95999999999999996,"5min":0.995,"15min":1.0389999999999999},"min":{"1min":0.65800000000000003,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.1499999999999999,"5min":1.3740000000000001,"15min":2.21},"last":1.175}]},{"osd":7,"last update":"Fri Sep 13 16:36:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":1.008,"15min":1.0269999999999999},"min":{"1min":0.629,"5min":0.629,"15min":0.45500000000000002},"max":{"1min":1.4039999999999999,"5min":1.4279999999999999,"15min":1.4530000000000001},"last":1.228},{"interface":"front","average":{"1min":0.98499999999999999,"5min":1.0429999999999999,"15min":1.1220000000000001},"min":{"1min":0.79400000000000004,"5min":0.76100000000000001,"15min":0.68799999999999994},"max":{"1min":1.556,"5min":1.556,"15min":3.9830000000000001},"last":1.105}]}]},{"osd":4,"up_from":14,"seq":60129542289,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86099999999999999,"15min":0.84499999999999997},"min":{"1min":0.60999999999999999,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.206,"5min":1.21,"15min":1.411},"last":0.95999999999999996},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.56499999999999995,"5min":0.54900000000000004,"15min":0.44600000000000001},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.899},"last":1.0640000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88400000000000001,"15min":0.90700000000000003},"min":{"1min":0.59099999999999997,"5min":0.53800000000000003,"15min":0.46200000000000002},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":3.085},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86499999999999999,"15min":0.90000000000000002},"min":{"1min":0.54200000000000004,"5min":0.52800000000000002,"15min":0.497},"max":{"1min":1.169,"5min":1.27,"15min":1.5720000000000001},"last":0.85299999999999998}]},{"osd":2,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.874,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.433},"max":{"1min":1.258,"5min":1.258,"15min":1.4419999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.90300000000000002,"15min":0.94799999999999995},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3220000000000001,"5min":1.3640000000000001,"15min":3.2160000000000002},"last":1.161}]},{"osd":3,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.93600000000000005},"min":{"1min":0.65900000000000003,"5min":0.60399999999999998,"15min":0.47999999999999998},"max":{"1min":1.2749999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":0.747},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93200000000000005,"15min":0.94199999999999995},"min":{"1min":0.62,"5min":0.61499999999999999,"15min":0.52200000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.577},"last":0.99199999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91000000000000003,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.49299999999999999},"max":{"1min":1.2390000000000001,"5min":1.252,"15min":2.996},"last":1.2769999999999999},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98199999999999998,"15min":1.0589999999999999},"min":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.495},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":2.8860000000000001},"last":1.0409999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.66700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.234,"5min":1.522,"15min":1.6499999999999999},"last":1.208},{"interface":"front","average":{"1min":1.085,"5min":1.0700000000000001,"15min":1.099},"min":{"1min":0.77200000000000002,"5min":0.63600000000000001,"15min":0.55500000000000005},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.1469999999999998},"last":1.228}]},{"osd":7,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.81499999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":2.1230000000000002},"last":1.105},{"interface":"front","average":{"1min":1.105,"5min":1.079,"15min":1.0980000000000001},"min":{"1min":0.78300000000000003,"5min":0.72599999999999998,"15min":0.67000000000000004},"max":{"1min":1.413,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":1.1240000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542290,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32716,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339124,"statfs":{"total":96636764160,"available":96603262976,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.63200000000000001,"15min":0.625},"min":{"1min":0.441,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.79900000000000004,"5min":2.2050000000000001,"15min":2.2050000000000001},"last":0.749},{"interface":"front","average":{"1min":0.72699999999999998,"5min":0.69999999999999996,"15min":0.67800000000000005},"min":{"1min":0.503,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.6180000000000001,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":1.0189999999999999}]},{"osd":1,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.65600000000000003,"15min":0.64900000000000002},"min":{"1min":0.433,"5min":0.40799999999999997,"15min":0.35499999999999998},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.73199999999999998,"5min":0.80400000000000005,"15min":0.79300000000000004},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.115,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":0.92200000000000004}]},{"osd":2,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.71099999999999997,"15min":0.70199999999999996},"min":{"1min":0.40400000000000003,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.131,"5min":1.133,"15min":1.27},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.80200000000000005,"15min":0.77900000000000003},"min":{"1min":0.57299999999999995,"5min":0.379,"15min":0.379},"max":{"1min":1.1299999999999999,"5min":1.2490000000000001,"15min":1.2490000000000001},"last":1.095}]},{"osd":4,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.92400000000000004},"min":{"1min":0.74199999999999999,"5min":0.69999999999999996,"15min":0.60499999999999998},"max":{"1min":1.6020000000000001,"5min":1.6020000000000001,"15min":1.8460000000000001},"last":1.139},{"interface":"front","average":{"1min":0.94199999999999995,"5min":0.93999999999999995,"15min":0.94899999999999995},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.59399999999999997},"max":{"1min":1.256,"5min":1.3360000000000001,"15min":1.782},"last":0.68899999999999995}]},{"osd":5,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":0.95799999999999996,"15min":0.96199999999999997},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.59499999999999997},"max":{"1min":1.1990000000000001,"5min":1.397,"15min":1.6299999999999999},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.97699999999999998,"5min":1.0269999999999999,"15min":1.0600000000000001},"min":{"1min":0.76300000000000001,"5min":0.66900000000000004,"15min":0.56899999999999995},"max":{"1min":1.3049999999999999,"5min":1.581,"15min":1.8140000000000001},"last":1.1919999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.0109999999999999,"5min":1.0640000000000001,"15min":1.0900000000000001},"min":{"1min":0.76400000000000001,"5min":0.61599999999999999,"15min":0.58699999999999997},"max":{"1min":1.3380000000000001,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.27},{"interface":"front","average":{"1min":1.034,"5min":1.0669999999999999,"15min":1.0469999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.57499999999999996},"max":{"1min":1.54,"5min":1.7030000000000001,"15min":1.95},"last":0.81200000000000006}]},{"osd":7,"last update":"Fri Sep 13 16:36:23 2024","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.093,"15min":1.0820000000000001},"min":{"1min":0.752,"5min":0.74199999999999999,"15min":0.71399999999999997},"max":{"1min":1.573,"5min":1.657,"15min":1.889},"last":1.1799999999999999},{"interface":"front","average":{"1min":1.052,"5min":1.0600000000000001,"15min":1.0569999999999999},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.294,"5min":1.6240000000000001,"15min":1.756},"last":0.84599999999999997}]}]},{"osd":2,"up_from":14,"seq":60129542291,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.62,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.93700000000000006,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.169,"15min":1.169},"last":0.65400000000000003}]},{"osd":1,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.88600000000000001,"5min":1.147,"15min":1.147},"last":0.86699999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72099999999999997,"15min":0.68999999999999995},"min":{"1min":0.45200000000000001,"5min":0.39000000000000001,"15min":0.38100000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.57199999999999995,"5min":0.37,"15min":0.37},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2330000000000001},"last":0.89800000000000002}]},{"osd":4,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93600000000000005,"15min":0.94399999999999995},"min":{"1min":0.69799999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.171,"5min":1.448,"15min":1.5149999999999999},"last":1.097},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.96799999999999997},"min":{"1min":0.69399999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.8799999999999999},"last":0.93500000000000005}]},{"osd":5,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95299999999999996,"15min":0.96899999999999997},"min":{"1min":0.63500000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.375,"5min":1.413,"15min":1.4750000000000001},"last":1.0760000000000001},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.96799999999999997,"15min":0.96399999999999997},"min":{"1min":0.66500000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.284,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":1.0089999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.74299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":2.024},"last":1.052},{"interface":"front","average":{"1min":1.071,"5min":1.044,"15min":1.052},"min":{"1min":0.82799999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.452,"15min":1.6359999999999999},"last":1.0289999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0529999999999999,"15min":1.0620000000000001},"min":{"1min":0.76400000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.27,"5min":1.357,"15min":1.4810000000000001},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0569999999999999,"15min":1.054},"min":{"1min":0.56699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.482,"5min":1.587,"15min":1.587},"last":0.98399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542290,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.23300000000000001},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":1.0880000000000001},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.751,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38800000000000001},"max":{"1min":1.0349999999999999,"5min":1.089,"15min":1.2010000000000001},"last":0.89800000000000002}]},{"osd":2,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67900000000000005,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.38300000000000001,"15min":0.26700000000000002},"max":{"1min":0.94999999999999996,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73699999999999999,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.29699999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.458},"last":0.69899999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.75800000000000001,"15min":0.72899999999999998},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":1.052,"5min":1.1359999999999999,"15min":1.4019999999999999},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.39600000000000002},"max":{"1min":0.89100000000000001,"5min":1.175,"15min":1.403},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.95699999999999996,"15min":0.93100000000000005},"min":{"1min":0.75600000000000001,"5min":0.67800000000000005,"15min":0.505},"max":{"1min":1.212,"5min":1.2190000000000001,"15min":1.546},"last":1.1950000000000001},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.51700000000000002},"max":{"1min":1.1459999999999999,"5min":1.1559999999999999,"15min":2.2040000000000002},"last":1.111}]},{"osd":5,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96999999999999997,"15min":0.95999999999999996},"min":{"1min":0.70299999999999996,"5min":0.68799999999999994,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.3819999999999999,"15min":1.389},"last":1.1479999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.93500000000000005,"15min":0.94599999999999995},"min":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.52000000000000002},"max":{"1min":1.1759999999999999,"5min":1.238,"15min":2.3029999999999999},"last":1.02}]},{"osd":6,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.01,"15min":1.0049999999999999},"min":{"1min":0.77700000000000002,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.3009999999999999,"5min":1.4330000000000001,"15min":1.613},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.032,"5min":1.056,"15min":1.03},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.59299999999999997},"max":{"1min":1.278,"5min":1.667,"15min":1.698},"last":1.0820000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.0129999999999999,"15min":1.008},"min":{"1min":0.74199999999999999,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":1.2609999999999999,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0649999999999999,"15min":1.0569999999999999},"min":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.59299999999999997},"max":{"1min":1.2290000000000001,"5min":1.3779999999999999,"15min":1.7689999999999999},"last":1.0529999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542290,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33300,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338540,"statfs":{"total":96636764160,"available":96602664960,"internally_reserved":0,"allocated":2228224,"data_stored":1346795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.52700000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68600000000000005,"15min":0.67300000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.101,"5min":1.1559999999999999,"15min":1.157},"last":0.53500000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70999999999999996,"15min":0.71599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"max":{"1min":0.94099999999999995,"5min":1.125,"15min":1.202},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39800000000000002},"max":{"1min":1.0249999999999999,"5min":1.0780000000000001,"15min":1.2629999999999999},"last":0.48999999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.312},"max":{"1min":0.92800000000000005,"5min":1.1259999999999999,"15min":1.4830000000000001},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.85499999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.097,"5min":1.155,"15min":1.6100000000000001},"last":0.50600000000000001}]},{"osd":4,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.95999999999999996,"15min":0.95799999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57699999999999996},"max":{"1min":1.3080000000000001,"5min":1.3899999999999999,"15min":1.486},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.95299999999999996},"min":{"1min":0.70399999999999996,"5min":0.64700000000000002,"15min":0.59999999999999998},"max":{"1min":0.97999999999999998,"5min":1.274,"15min":1.704},"last":0.72099999999999997}]},{"osd":5,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":0.996},"min":{"1min":0.80500000000000005,"5min":0.67100000000000004,"15min":0.59599999999999997},"max":{"1min":1.3720000000000001,"5min":1.532,"15min":1.575},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.996,"5min":0.99199999999999999,"15min":0.98999999999999999},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.443},"max":{"1min":1.399,"5min":1.742,"15min":1.742},"last":0.73599999999999999}]},{"osd":6,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.97599999999999998},"min":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.628},"max":{"1min":1.1020000000000001,"5min":1.4099999999999999,"15min":1.944},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.115,"5min":1.0940000000000001,"15min":1.075},"min":{"1min":0.81100000000000005,"5min":0.77300000000000002,"15min":0.58899999999999997},"max":{"1min":1.46,"5min":1.53,"15min":1.5780000000000001},"last":0.70399999999999996}]},{"osd":7,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.095,"15min":1.0720000000000001},"min":{"1min":0.86699999999999999,"5min":0.76300000000000001,"15min":0.61699999999999999},"max":{"1min":1.3759999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":0.68300000000000005},{"interface":"front","average":{"1min":1.042,"5min":1.073,"15min":1.071},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.65400000000000003},"max":{"1min":1.266,"5min":1.381,"15min":2.0139999999999998},"last":0.752}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:37:09.979 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-13T16:25:11.599970+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=25, tm_sec=11, tm_wday=4, tm_yday=257, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=13, tm_hour=16, tm_min=37, tm_sec=4, tm_wday=4, tm_yday=257, tm_isdst=0) 2024-09-13T16:37:09.979 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-13T16:37:28.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:37:28.904 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:37:29.980 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-13T16:37:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 Processor -- start 2024-09-13T16:37:30.099 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- start start 2024-09-13T16:37:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fce3c0cdeb0 0x7fce3c06c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.100 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cd200 0x7fce3c15a760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce3c0ce2d0 con 0x7fce3c06c5f0 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- --> v2:172.21.15.145:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce3c06cf10 con 0x7fce3c0cdeb0 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c0ce540 con 0x7fce3c0cd200 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cd200 0x7fce3c15a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3affd700 1 --2- >> v2:172.21.15.145:3300/0 conn(0x7fce3c0cdeb0 0x7fce3c06c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cd200 0x7fce3c15a760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.145:3301/0 says I am v2:172.21.15.119:59980/0 (socket says 172.21.15.119:59980) 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 -- 172.21.15.119:0/984220056 learned_addr learned my addr 172.21.15.119:0/984220056 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 -- 172.21.15.119:0/984220056 >> v2:172.21.15.145:3300/0 conn(0x7fce3c0cdeb0 msgr2=0x7fce3c06c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 --2- 172.21.15.119:0/984220056 >> v2:172.21.15.145:3300/0 conn(0x7fce3c0cdeb0 0x7fce3c06c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 -- 172.21.15.119:0/984220056 >> v1:172.21.15.145:6789/0 conn(0x7fce3c06c5f0 legacy=0x7fce3c15a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.101 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 -- 172.21.15.119:0/984220056 --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce3c0cc720 con 0x7fce3c0cd200 2024-09-13T16:37:30.102 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3bfff700 1 --2- 172.21.15.119:0/984220056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cd200 0x7fce3c15a760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fce34009e50 tx=0x7fce3400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6376d773909c1028 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3a7fc700 1 -- 172.21.15.119:0/984220056 <== mon.2 v2:172.21.15.145:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3400e070 con 0x7fce3c0cd200 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3a7fc700 1 -- 172.21.15.119:0/984220056 <== mon.2 v2:172.21.15.145:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3400b910 con 0x7fce3c0cd200 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce3a7fc700 1 -- 172.21.15.119:0/984220056 <== mon.2 v2:172.21.15.145:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3400bcc0 con 0x7fce3c0cd200 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- 172.21.15.119:0/984220056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cd200 msgr2=0x7fce3c15a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 --2- 172.21.15.119:0/984220056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cd200 0x7fce3c15a760 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fce34009e50 tx=0x7fce3400b330 comp rx=0 tx=0).stop 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- 172.21.15.119:0/984220056 shutdown_connections 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 --2- 172.21.15.119:0/984220056 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cd200 0x7fce3c15a760 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 --2- 172.21.15.119:0/984220056 >> v2:172.21.15.145:3300/0 conn(0x7fce3c0cdeb0 0x7fce3c06c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- 172.21.15.119:0/984220056 >> 172.21.15.119:0/984220056 conn(0x7fce3c06a350 msgr2=0x7fce3c06a760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- 172.21.15.119:0/984220056 shutdown_connections 2024-09-13T16:37:30.103 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.096+0000 7fce41b3a700 1 -- 172.21.15.119:0/984220056 wait complete. 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 Processor -- start 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 -- start start 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce3c06c5f0 0x7fce3c151610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce3c0cd200 0x7fce3c0c6200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cdeb0 0x7fce3c070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c159c20 con 0x7fce3c06c5f0 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c159e70 con 0x7fce3c0cd200 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce3c15a150 con 0x7fce3c0cdeb0 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce3c06c5f0 0x7fce3c151610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.105 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3affd700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce3c0cd200 0x7fce3c0c6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce3c06c5f0 0x7fce3c151610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42190/0 (socket says 172.21.15.119:42190) 2024-09-13T16:37:30.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 -- 172.21.15.119:0/4179473782 learned_addr learned my addr 172.21.15.119:0/4179473782 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:30.106 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3bfff700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cdeb0 0x7fce3c070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 -- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cdeb0 msgr2=0x7fce3c070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cdeb0 0x7fce3c070310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 -- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce3c0cd200 msgr2=0x7fce3c0c6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce3c0cd200 0x7fce3c0c6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 -- 172.21.15.119:0/4179473782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34009b30 con 0x7fce3c06c5f0 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce3b7fe700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce3c06c5f0 0x7fce3c151610 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c500 tx=0x7fce3000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce23fff700 1 -- 172.21.15.119:0/4179473782 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce300052d0 con 0x7fce3c06c5f0 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce23fff700 1 -- 172.21.15.119:0/4179473782 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce30005430 con 0x7fce3c06c5f0 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3c15a430 con 0x7fce3c06c5f0 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce23fff700 1 -- 172.21.15.119:0/4179473782 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce300055a0 con 0x7fce3c06c5f0 2024-09-13T16:37:30.107 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.100+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3c15a9f0 con 0x7fce3c06c5f0 2024-09-13T16:37:30.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.104+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3c067760 con 0x7fce3c06c5f0 2024-09-13T16:37:30.108 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.104+0000 7fce23fff700 1 -- 172.21.15.119:0/4179473782 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fce300038e0 con 0x7fce3c06c5f0 2024-09-13T16:37:30.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.104+0000 7fce23fff700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce28066730 0x7fce28068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.104+0000 7fce23fff700 1 -- 172.21.15.119:0/4179473782 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fce300833a0 con 0x7fce3c06c5f0 2024-09-13T16:37:30.109 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.104+0000 7fce3affd700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce28066730 0x7fce28068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.110 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.104+0000 7fce3affd700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce28066730 0x7fce28068bf0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0728b0 tx=0x7fce2c009240 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.117 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.112+0000 7fce23fff700 1 -- 172.21.15.119:0/4179473782 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fce30050810 con 0x7fce3c06c5f0 2024-09-13T16:37:30.283 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.280+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 --> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fce3c0cc3d0 con 0x7fce28066730 2024-09-13T16:37:30.289 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.284+0000 7fce23fff700 1 -- 172.21.15.119:0/4179473782 <== mgr.4103 v2:172.21.15.145:6832/2154272207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+49057 (secure 0 0 0) 0x7fce2c0118c0 con 0x7fce28066730 2024-09-13T16:37:30.290 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:37:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce28066730 msgr2=0x7fce28068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.291 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce28066730 0x7fce28068bf0 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0728b0 tx=0x7fce2c009240 comp rx=0 tx=0).stop 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce3c06c5f0 msgr2=0x7fce3c151610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce3c06c5f0 0x7fce3c151610 secure :-1 s=READY pgs=2064 cs=0 l=1 rev1=1 crypto rx=0x7fce3000c500 tx=0x7fce3000c810 comp rx=0 tx=0).stop 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 shutdown_connections 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fce28066730 0x7fce28068bf0 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fce3c0cdeb0 0x7fce3c070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fce3c0cd200 0x7fce3c0c6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 --2- 172.21.15.119:0/4179473782 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fce3c06c5f0 0x7fce3c151610 unknown :-1 s=CLOSED pgs=2064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 >> 172.21.15.119:0/4179473782 conn(0x7fce3c06a350 msgr2=0x7fce3c0aef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 shutdown_connections 2024-09-13T16:37:30.292 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.288+0000 7fce41b3a700 1 -- 172.21.15.119:0/4179473782 wait complete. 2024-09-13T16:37:30.293 INFO:teuthology.orchestra.run.smithi119.stderr:dumped all 2024-09-13T16:37:30.305 INFO:teuthology.orchestra.run.smithi119.stdout:{"pg_ready":true,"pg_map":{"version":494,"stamp":"2024-09-13T16:37:29.681828+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":262852,"kb_used_data":13928,"kb_used_omap":12,"kb_used_meta":248819,"kb_avail":754711868,"statfs":{"total":773094113280,"available":772824952832,"internally_reserved":0,"allocated":14262272,"data_stored":7232152,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12743,"internal_metadata":254791225},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003225"},"pg_stats":[{"pgid":"1.0","version":"131'119","reported_seq":277,"reported_epoch":131,"state":"active+clean","last_fresh":"2024-09-13T16:37:13.202643+0000","last_change":"2024-09-13T16:37:13.202643+0000","last_active":"2024-09-13T16:37:13.202643+0000","last_peered":"2024-09-13T16:37:13.202643+0000","last_clean":"2024-09-13T16:37:13.202643+0000","last_became_active":"2024-09-13T16:25:13.154518+0000","last_became_peered":"2024-09-13T16:25:13.154518+0000","last_unstale":"2024-09-13T16:37:13.202643+0000","last_undegraded":"2024-09-13T16:37:13.202643+0000","last_fullsized":"2024-09-13T16:37:13.202643+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"131'119","last_scrub_stamp":"2024-09-13T16:37:13.202397+0000","last_deep_scrub":"131'119","last_deep_scrub_stamp":"2024-09-13T16:37:13.202397+0000","last_clean_scrub_stamp":"2024-09-13T16:37:13.202397+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-14T22:18:38.505073+0000","scrub_duration":0.0095831419999999994,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":14,"seq":60129542294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2228224,"data_stored":1346795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.80300000000000005,"15min":0.83699999999999997},"min":{"1min":0.51800000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":1.198,"5min":1.262,"15min":1.357},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.85199999999999998,"15min":0.85499999999999998},"min":{"1min":0.628,"5min":0.52000000000000002,"15min":0.47699999999999998},"max":{"1min":1.1759999999999999,"5min":1.214,"15min":1.3460000000000001},"last":0.67800000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.83299999999999996,"15min":0.84499999999999997},"min":{"1min":0.58399999999999996,"5min":0.56000000000000005,"15min":0.501},"max":{"1min":1.115,"5min":1.212,"15min":1.569},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.89100000000000001,"15min":0.86499999999999999},"min":{"1min":0.63600000000000001,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.115,"5min":1.2290000000000001,"15min":1.4219999999999999},"last":1.036}]},{"osd":2,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.874},"min":{"1min":0.56699999999999995,"5min":0.441,"15min":0.441},"max":{"1min":1.113,"5min":1.1919999999999999,"15min":1.242},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.94399999999999995,"5min":0.89100000000000001,"15min":0.87},"min":{"1min":0.64900000000000002,"5min":0.60499999999999998,"15min":0.53900000000000003},"max":{"1min":1.238,"5min":1.24,"15min":1.339},"last":1.077}]},{"osd":3,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.91300000000000003,"15min":0.95699999999999996},"min":{"1min":0.65300000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":1.3,"5min":1.3,"15min":1.476},"last":1.2070000000000001},{"interface":"front","average":{"1min":0.98199999999999998,"5min":0.998,"15min":1},"min":{"1min":0.73599999999999999,"5min":0.69599999999999995,"15min":0.65700000000000003},"max":{"1min":1.28,"5min":1.2969999999999999,"15min":1.3939999999999999},"last":1.175}]},{"osd":4,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.878,"15min":0.874},"min":{"1min":0.71399999999999997,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":1.089,"5min":1.4650000000000001,"15min":1.4650000000000001},"last":0.79300000000000004},{"interface":"front","average":{"1min":0.97199999999999998,"5min":0.94399999999999995,"15min":1.006},"min":{"1min":0.73299999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.216,"5min":1.381,"15min":1.589},"last":1.2549999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.98199999999999998,"15min":0.98999999999999999},"min":{"1min":0.65400000000000003,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.2230000000000001,"5min":1.444,"15min":1.444},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.1299999999999999,"5min":1.0720000000000001,"15min":1.091},"min":{"1min":0.99399999999999999,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.278,"5min":1.5760000000000001,"15min":1.5760000000000001},"last":1.1120000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:36:31 2024","interfaces":[{"interface":"back","average":{"1min":1.0800000000000001,"5min":1.0369999999999999,"15min":1.0529999999999999},"min":{"1min":0.76500000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"max":{"1min":1.2330000000000001,"5min":1.72,"15min":1.72},"last":0.90700000000000003},{"interface":"front","average":{"1min":1.05,"5min":1.044,"15min":1.1080000000000001},"min":{"1min":0.746,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.375,"5min":1.542,"15min":1.6779999999999999},"last":1.139}]}]},{"osd":6,"up_from":14,"seq":60129542293,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.84499999999999997,"15min":0.84099999999999997},"min":{"1min":0.53300000000000003,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.1819999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"last":1.046},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.85799999999999998,"15min":0.878},"min":{"1min":0.55800000000000005,"5min":0.309,"15min":0.309},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.4410000000000001},"last":0.625}]},{"osd":1,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.85699999999999998,"15min":0.84599999999999997},"min":{"1min":0.44600000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.0509999999999999,"5min":1.153,"15min":1.2030000000000001},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.88200000000000001,"15min":0.90800000000000003},"min":{"1min":0.57699999999999996,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":1.1599999999999999,"5min":1.411,"15min":1.411},"last":1.1000000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.86299999999999999,"15min":0.86899999999999999},"min":{"1min":0.57899999999999996,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.262,"5min":1.4359999999999999,"15min":1.4359999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.79900000000000004,"5min":0.88800000000000001,"15min":0.874},"min":{"1min":0.53000000000000003,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.0940000000000001,"5min":1.399,"15min":1.399},"last":0.81599999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.90800000000000003,"5min":0.92100000000000004,"15min":0.94899999999999995},"min":{"1min":0.58099999999999996,"5min":0.379,"15min":0.379},"max":{"1min":1.1870000000000001,"5min":1.544,"15min":1.544},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.0029999999999999,"5min":0.97499999999999998,"15min":0.98199999999999998},"min":{"1min":0.754,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.21,"5min":1.6339999999999999,"15min":1.6339999999999999},"last":0.78200000000000003}]},{"osd":4,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95599999999999996},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.1599999999999999,"5min":1.508,"15min":1.508},"last":1.147},{"interface":"front","average":{"1min":1.0349999999999999,"5min":0.95799999999999996,"15min":0.96299999999999997},"min":{"1min":0.65500000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":1.343,"5min":1.55,"15min":1.6140000000000001},"last":1.0149999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.92300000000000004,"15min":0.93100000000000005},"min":{"1min":0.59499999999999997,"5min":0.55700000000000005,"15min":0.55700000000000005},"max":{"1min":1.214,"5min":1.339,"15min":1.339},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.052,"5min":1.0580000000000001,"15min":1.089},"min":{"1min":0.70299999999999996,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.2829999999999999,"5min":1.615,"15min":1.6439999999999999},"last":1.1870000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:32 2024","interfaces":[{"interface":"back","average":{"1min":1.105,"5min":1.048,"15min":1.087},"min":{"1min":0.72799999999999998,"5min":0.51700000000000002,"15min":0.51700000000000002},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.522},"last":0.92800000000000005},{"interface":"front","average":{"1min":1.0640000000000001,"5min":1.052,"15min":1.0589999999999999},"min":{"1min":0.68899999999999995,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.4139999999999999,"5min":1.4139999999999999,"15min":1.427},"last":1.0780000000000001}]}]},{"osd":5,"up_from":14,"seq":60129542295,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.871,"5min":0.84699999999999998,"15min":0.84799999999999998},"min":{"1min":0.60399999999999998,"5min":0.441,"15min":0.441},"max":{"1min":1.131,"5min":1.169,"15min":1.411},"last":1.131},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.82999999999999996,"15min":0.83899999999999997},"min":{"1min":0.65300000000000002,"5min":0.497,"15min":0.41699999999999998},"max":{"1min":1.2290000000000001,"5min":1.3029999999999999,"15min":1.675},"last":1.0660000000000001}]},{"osd":1,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.85999999999999999,"15min":0.86099999999999999},"min":{"1min":0.68600000000000005,"5min":0.60299999999999998,"15min":0.46600000000000003},"max":{"1min":1.135,"5min":1.2190000000000001,"15min":1.4079999999999999},"last":1.044},{"interface":"front","average":{"1min":0.875,"5min":0.83599999999999997,"15min":0.84299999999999997},"min":{"1min":0.60999999999999999,"5min":0.53700000000000003,"15min":0.36499999999999999},"max":{"1min":1.1599999999999999,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":1.1599999999999999}]},{"osd":2,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.90700000000000003,"15min":0.91300000000000003},"min":{"1min":0.625,"5min":0.441,"15min":0.441},"max":{"1min":1.113,"5min":1.4379999999999999,"15min":1.625},"last":1.024},{"interface":"front","average":{"1min":1.0069999999999999,"5min":0.97699999999999998,"15min":0.97199999999999998},"min":{"1min":0.70899999999999996,"5min":0.505,"15min":0.505},"max":{"1min":1.1950000000000001,"5min":1.331,"15min":1.552},"last":0.98799999999999999}]},{"osd":3,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91200000000000003,"15min":0.88500000000000001},"min":{"1min":0.75700000000000001,"5min":0.5,"15min":0.5},"max":{"1min":1.2130000000000001,"5min":1.23,"15min":1.3919999999999999},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.94699999999999995,"5min":0.90800000000000003,"15min":0.93500000000000005},"min":{"1min":0.72599999999999998,"5min":0.56899999999999995,"15min":0.56899999999999995},"max":{"1min":1.165,"5min":1.2929999999999999,"15min":3.8460000000000001},"last":0.93500000000000005}]},{"osd":4,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.88800000000000001,"15min":0.95699999999999996},"min":{"1min":0.58599999999999997,"5min":0.58199999999999996,"15min":0.56999999999999995},"max":{"1min":1.1639999999999999,"5min":1.3540000000000001,"15min":3.6869999999999998},"last":0.84299999999999997},{"interface":"front","average":{"1min":1.069,"5min":1.0669999999999999,"15min":1.0640000000000001},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.57499999999999996},"max":{"1min":1.3029999999999999,"5min":1.365,"15min":1.601},"last":1.2490000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.92200000000000004,"15min":0.94399999999999995},"min":{"1min":0.64400000000000002,"5min":0.61899999999999999,"15min":0.60899999999999999},"max":{"1min":1.1679999999999999,"5min":1.5069999999999999,"15min":1.5069999999999999},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.075,"5min":1.012,"15min":1.0389999999999999},"min":{"1min":0.73999999999999999,"5min":0.63800000000000001,"15min":0.61199999999999999},"max":{"1min":1.2729999999999999,"5min":1.3740000000000001,"15min":2.21},"last":1.1919999999999999}]},{"osd":7,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":1.115,"5min":1.0229999999999999,"15min":1.032},"min":{"1min":0.82199999999999995,"5min":0.629,"15min":0.45500000000000002},"max":{"1min":1.3420000000000001,"5min":1.4099999999999999,"15min":1.4530000000000001},"last":1.321},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0449999999999999,"15min":1.1100000000000001},"min":{"1min":0.78900000000000003,"5min":0.76100000000000001,"15min":0.68799999999999994},"max":{"1min":1.26,"5min":1.556,"15min":3.9830000000000001},"last":1.2170000000000001}]}]},{"osd":4,"up_from":14,"seq":60129542293,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.85999999999999999,"5min":0.86099999999999999,"15min":0.84499999999999997},"min":{"1min":0.60999999999999999,"5min":0.51000000000000001,"15min":0.34599999999999997},"max":{"1min":1.206,"5min":1.21,"15min":1.411},"last":1.1359999999999999},{"interface":"front","average":{"1min":0.88500000000000001,"5min":0.88100000000000001,"15min":0.875},"min":{"1min":0.56499999999999995,"5min":0.54900000000000004,"15min":0.44600000000000001},"max":{"1min":1.1699999999999999,"5min":1.256,"15min":2.899},"last":0.751}]},{"osd":1,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.88400000000000001,"15min":0.90700000000000003},"min":{"1min":0.59099999999999997,"5min":0.53800000000000003,"15min":0.46200000000000002},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":3.085},"last":1.1819999999999999},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.86499999999999999,"15min":0.90000000000000002},"min":{"1min":0.54200000000000004,"5min":0.52800000000000002,"15min":0.497},"max":{"1min":1.169,"5min":1.27,"15min":1.5720000000000001},"last":0.78000000000000003}]},{"osd":2,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.874,"15min":0.85099999999999998},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.433},"max":{"1min":1.258,"5min":1.258,"15min":1.4419999999999999},"last":1.0489999999999999},{"interface":"front","average":{"1min":0.83799999999999997,"5min":0.90300000000000002,"15min":0.94799999999999995},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.498},"max":{"1min":1.3220000000000001,"5min":1.3640000000000001,"15min":3.2160000000000002},"last":0.83999999999999997}]},{"osd":3,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.92000000000000004,"15min":0.93600000000000005},"min":{"1min":0.65900000000000003,"5min":0.60399999999999998,"15min":0.47999999999999998},"max":{"1min":1.2749999999999999,"5min":1.4750000000000001,"15min":1.4750000000000001},"last":1.01},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.93200000000000005,"15min":0.94199999999999995},"min":{"1min":0.62,"5min":0.61499999999999999,"15min":0.52200000000000002},"max":{"1min":1.431,"5min":1.431,"15min":1.577},"last":0.86599999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91000000000000003,"15min":0.95099999999999996},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.49299999999999999},"max":{"1min":1.2390000000000001,"5min":1.252,"15min":2.996},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.94299999999999995,"5min":0.98199999999999998,"15min":1.0589999999999999},"min":{"1min":0.61499999999999999,"5min":0.61099999999999999,"15min":0.495},"max":{"1min":1.4890000000000001,"5min":1.4890000000000001,"15min":2.8860000000000001},"last":0.90000000000000002}]},{"osd":6,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.93799999999999994,"15min":0.93999999999999995},"min":{"1min":0.66700000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.234,"5min":1.522,"15min":1.6499999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.085,"5min":1.0700000000000001,"15min":1.099},"min":{"1min":0.77200000000000002,"5min":0.63600000000000001,"15min":0.55500000000000005},"max":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":3.1469999999999998},"last":1.081}]},{"osd":7,"last update":"Fri Sep 13 16:36:38 2024","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.0580000000000001,"15min":1.1000000000000001},"min":{"1min":0.81499999999999995,"5min":0.505,"15min":0.505},"max":{"1min":1.6100000000000001,"5min":1.6100000000000001,"15min":2.1230000000000002},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.105,"5min":1.079,"15min":1.0980000000000001},"min":{"1min":0.78300000000000003,"5min":0.72599999999999998,"15min":0.67000000000000004},"max":{"1min":1.413,"5min":1.6240000000000001,"15min":3.1070000000000002},"last":1.1639999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32716,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339124,"statfs":{"total":96636764160,"available":96603262976,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31848904},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61299999999999999,"15min":0.623},"min":{"1min":0.47299999999999998,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.83599999999999997,"5min":1.0900000000000001,"15min":2.2050000000000001},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.71799999999999997,"15min":0.68600000000000005},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.2769999999999999,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.72399999999999998}]},{"osd":1,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.65500000000000003,"15min":0.65000000000000002},"min":{"1min":0.41399999999999998,"5min":0.40799999999999997,"15min":0.35499999999999998},"max":{"1min":0.78600000000000003,"5min":1.008,"15min":1.008},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.81200000000000006,"5min":0.79100000000000004,"15min":0.79400000000000004},"min":{"1min":0.41999999999999998,"5min":0.373,"15min":0.373},"max":{"1min":1.0589999999999999,"5min":1.179,"15min":2.0750000000000002},"last":0.76500000000000001}]},{"osd":2,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.70699999999999996,"15min":0.70399999999999996},"min":{"1min":0.44500000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.27},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.86199999999999999,"5min":0.81100000000000005,"15min":0.78700000000000003},"min":{"1min":0.48799999999999999,"5min":0.379,"15min":0.379},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2490000000000001},"last":0.85099999999999998}]},{"osd":4,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.98999999999999999,"5min":0.95899999999999996,"15min":0.93000000000000005},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.60499999999999998},"max":{"1min":1.5289999999999999,"5min":1.6020000000000001,"15min":1.8460000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.92300000000000004,"5min":0.93500000000000005,"15min":0.94699999999999995},"min":{"1min":0.68899999999999995,"5min":0.65800000000000003,"15min":0.59399999999999997},"max":{"1min":1.1419999999999999,"5min":1.3360000000000001,"15min":1.782},"last":1.024}]},{"osd":5,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.95499999999999996,"15min":0.96099999999999997},"min":{"1min":0.66200000000000003,"5min":0.64700000000000002,"15min":0.59499999999999997},"max":{"1min":1.202,"5min":1.397,"15min":1.6299999999999999},"last":0.93799999999999994},{"interface":"front","average":{"1min":1.0620000000000001,"5min":1.026,"15min":1.0569999999999999},"min":{"1min":0.82599999999999996,"5min":0.66900000000000004,"15min":0.56899999999999995},"max":{"1min":1.3160000000000001,"5min":1.367,"15min":1.8140000000000001},"last":1.0680000000000001}]},{"osd":6,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":1.0609999999999999,"5min":1.052,"15min":1.0840000000000001},"min":{"1min":0.69899999999999995,"5min":0.61599999999999999,"15min":0.58699999999999997},"max":{"1min":1.4450000000000001,"5min":1.4450000000000001,"15min":2.1539999999999999},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.083,"5min":1.052,"15min":1.048},"min":{"1min":0.81200000000000006,"5min":0.58699999999999997,"15min":0.57499999999999996},"max":{"1min":1.5529999999999999,"5min":1.5529999999999999,"15min":1.95},"last":1.208}]},{"osd":7,"last update":"Fri Sep 13 16:37:24 2024","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.077,"15min":1.0840000000000001},"min":{"1min":0.79600000000000004,"5min":0.74199999999999999,"15min":0.71399999999999997},"max":{"1min":1.5,"5min":1.573,"15min":1.889},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.0609999999999999,"5min":1.0600000000000001,"15min":1.056},"min":{"1min":0.84599999999999997,"5min":0.68300000000000005,"15min":0.68300000000000005},"max":{"1min":1.242,"5min":1.6240000000000001,"15min":1.756},"last":1.1220000000000001}]}]},{"osd":2,"up_from":14,"seq":60129542295,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31848903},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.62,"15min":0.60399999999999998},"min":{"1min":0.433,"5min":0.34799999999999998,"15min":0.33100000000000002},"max":{"1min":0.93700000000000006,"5min":1.1100000000000001,"15min":1.1100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.67000000000000004,"15min":0.68799999999999994},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.93700000000000006,"5min":1.169,"15min":1.169},"last":0.55700000000000005}]},{"osd":1,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.68700000000000006,"15min":0.69299999999999995},"min":{"1min":0.41499999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.68600000000000005,"5min":0.71199999999999997,"15min":0.69799999999999995},"min":{"1min":0.33700000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.88600000000000001,"5min":1.147,"15min":1.147},"last":0.65700000000000003}]},{"osd":3,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.72099999999999997,"15min":0.68999999999999995},"min":{"1min":0.45200000000000001,"5min":0.39000000000000001,"15min":0.38100000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.1739999999999999},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.91700000000000004,"5min":0.84099999999999997,"15min":0.81999999999999995},"min":{"1min":0.57199999999999995,"5min":0.37,"15min":0.37},"max":{"1min":1.1539999999999999,"5min":1.1539999999999999,"15min":1.2330000000000001},"last":0.95999999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.93600000000000005,"15min":0.94399999999999995},"min":{"1min":0.69799999999999995,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.171,"5min":1.448,"15min":1.5149999999999999},"last":0.93600000000000005},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.96799999999999997},"min":{"1min":0.69399999999999995,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.4219999999999999,"5min":1.4219999999999999,"15min":1.8799999999999999},"last":0.80000000000000004}]},{"osd":5,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.95299999999999996,"15min":0.96899999999999997},"min":{"1min":0.63500000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.375,"5min":1.413,"15min":1.4750000000000001},"last":0.86299999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":0.96799999999999997,"15min":0.96399999999999997},"min":{"1min":0.66500000000000004,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.284,"5min":1.4039999999999999,"15min":1.5720000000000001},"last":0.71999999999999997}]},{"osd":6,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":0.97599999999999998,"5min":0.99199999999999999,"15min":0.99299999999999999},"min":{"1min":0.74299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"max":{"1min":1.3700000000000001,"5min":1.3700000000000001,"15min":2.024},"last":0.98399999999999999},{"interface":"front","average":{"1min":1.071,"5min":1.044,"15min":1.052},"min":{"1min":0.82799999999999996,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.452,"5min":1.452,"15min":1.6359999999999999},"last":1.0700000000000001}]},{"osd":7,"last update":"Fri Sep 13 16:36:37 2024","interfaces":[{"interface":"back","average":{"1min":1.0469999999999999,"5min":1.0529999999999999,"15min":1.0620000000000001},"min":{"1min":0.76400000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.27,"5min":1.357,"15min":1.4810000000000001},"last":0.83599999999999997},{"interface":"front","average":{"1min":1.0469999999999999,"5min":1.0569999999999999,"15min":1.054},"min":{"1min":0.56699999999999995,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.482,"5min":1.587,"15min":1.587},"last":1.0429999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542294,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":32712,"kb_used_data":1596,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339128,"statfs":{"total":96636764160,"available":96603267072,"internally_reserved":0,"allocated":1634304,"data_stored":756427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.63600000000000001,"15min":0.61799999999999999},"min":{"1min":0.44,"5min":0.44,"15min":0.23300000000000001},"max":{"1min":0.86899999999999999,"5min":0.88900000000000001,"15min":1.0880000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.72899999999999998,"5min":0.751,"15min":0.745},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.38800000000000001},"max":{"1min":1.0349999999999999,"5min":1.089,"15min":1.2010000000000001},"last":0.75}]},{"osd":2,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.68000000000000005,"5min":0.67900000000000005,"15min":0.65200000000000002},"min":{"1min":0.51100000000000001,"5min":0.38300000000000001,"15min":0.26700000000000002},"max":{"1min":0.94999999999999996,"5min":1.0489999999999999,"15min":1.0629999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.74199999999999999,"5min":0.73699999999999999,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.29699999999999999},"max":{"1min":1.044,"5min":1.044,"15min":1.458},"last":0.54900000000000004}]},{"osd":3,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.76800000000000002,"5min":0.75800000000000001,"15min":0.72899999999999998},"min":{"1min":0.5,"5min":0.41099999999999998,"15min":0.35399999999999998},"max":{"1min":1.052,"5min":1.1359999999999999,"15min":1.4019999999999999},"last":0.66800000000000004},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.68700000000000006,"15min":0.68600000000000005},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.39600000000000002},"max":{"1min":0.89100000000000001,"5min":1.175,"15min":1.403},"last":0.83299999999999996}]},{"osd":4,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.96299999999999997,"5min":0.95699999999999996,"15min":0.93100000000000005},"min":{"1min":0.75600000000000001,"5min":0.67800000000000005,"15min":0.505},"max":{"1min":1.212,"5min":1.2190000000000001,"15min":1.546},"last":0.86499999999999999},{"interface":"front","average":{"1min":0.86499999999999999,"5min":0.91400000000000003,"15min":0.91700000000000004},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.51700000000000002},"max":{"1min":1.1459999999999999,"5min":1.1559999999999999,"15min":2.2040000000000002},"last":1.0169999999999999}]},{"osd":5,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.95199999999999996,"5min":0.96999999999999997,"15min":0.95999999999999996},"min":{"1min":0.70299999999999996,"5min":0.68799999999999994,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.3819999999999999,"15min":1.389},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.93500000000000005,"15min":0.94599999999999995},"min":{"1min":0.70699999999999996,"5min":0.66800000000000004,"15min":0.52000000000000002},"max":{"1min":1.1759999999999999,"5min":1.238,"15min":2.3029999999999999},"last":0.91200000000000003}]},{"osd":6,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":1.01,"15min":1.0049999999999999},"min":{"1min":0.77700000000000002,"5min":0.64200000000000002,"15min":0.626},"max":{"1min":1.3009999999999999,"5min":1.4330000000000001,"15min":1.613},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.032,"5min":1.056,"15min":1.03},"min":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.59299999999999997},"max":{"1min":1.278,"5min":1.667,"15min":1.698},"last":0.96199999999999997}]},{"osd":7,"last update":"Fri Sep 13 16:36:33 2024","interfaces":[{"interface":"back","average":{"1min":0.94399999999999995,"5min":1.0129999999999999,"15min":1.008},"min":{"1min":0.74199999999999999,"5min":0.71599999999999997,"15min":0.62},"max":{"1min":1.2609999999999999,"5min":1.5960000000000001,"15min":2.3999999999999999},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.0189999999999999,"5min":1.0649999999999999,"15min":1.0569999999999999},"min":{"1min":0.74099999999999999,"5min":0.67200000000000004,"15min":0.59299999999999997},"max":{"1min":1.2290000000000001,"5min":1.3779999999999999,"15min":1.7689999999999999},"last":1.0840000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":33288,"kb_used_data":2176,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94338552,"statfs":{"total":96636764160,"available":96602677248,"internally_reserved":0,"allocated":2228224,"data_stored":1346795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.65600000000000003,"5min":0.64500000000000002,"15min":0.65400000000000003},"min":{"1min":0.52700000000000002,"5min":0.38700000000000001,"15min":0.35299999999999998},"max":{"1min":0.94499999999999995,"5min":1.0089999999999999,"15min":1.1100000000000001},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.68600000000000005,"15min":0.67300000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.101,"5min":1.1559999999999999,"15min":1.157},"last":0.70299999999999996}]},{"osd":2,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.72599999999999998,"5min":0.70999999999999996,"15min":0.71599999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"max":{"1min":0.94099999999999995,"5min":1.125,"15min":1.202},"last":0.753},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.73199999999999998,"15min":0.71099999999999997},"min":{"1min":0.46400000000000002,"5min":0.443,"15min":0.39800000000000002},"max":{"1min":1.0249999999999999,"5min":1.0780000000000001,"15min":1.2629999999999999},"last":0.56499999999999995}]},{"osd":3,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.67300000000000004,"5min":0.71999999999999997,"15min":0.69799999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.312},"max":{"1min":0.92800000000000005,"5min":1.1259999999999999,"15min":1.4830000000000001},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.876,"15min":0.85499999999999998},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.097,"5min":1.155,"15min":1.6100000000000001},"last":0.871}]},{"osd":4,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.95999999999999996,"15min":0.95799999999999996},"min":{"1min":0.58599999999999997,"5min":0.58599999999999997,"15min":0.57699999999999996},"max":{"1min":1.3080000000000001,"5min":1.3899999999999999,"15min":1.486},"last":0.90700000000000003},{"interface":"front","average":{"1min":0.85599999999999998,"5min":0.93500000000000005,"15min":0.95299999999999996},"min":{"1min":0.70399999999999996,"5min":0.64700000000000002,"15min":0.59999999999999998},"max":{"1min":0.97999999999999998,"5min":1.274,"15min":1.704},"last":0.84699999999999998}]},{"osd":5,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.0309999999999999,"5min":1.0049999999999999,"15min":0.996},"min":{"1min":0.80500000000000005,"5min":0.67100000000000004,"15min":0.59599999999999997},"max":{"1min":1.3720000000000001,"5min":1.532,"15min":1.575},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.996,"5min":0.99199999999999999,"15min":0.98999999999999999},"min":{"1min":0.65600000000000003,"5min":0.61199999999999999,"15min":0.443},"max":{"1min":1.399,"5min":1.742,"15min":1.742},"last":0.81499999999999995}]},{"osd":6,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":0.91000000000000003,"5min":0.95999999999999996,"15min":0.97599999999999998},"min":{"1min":0.72299999999999998,"5min":0.70599999999999996,"15min":0.628},"max":{"1min":1.1020000000000001,"5min":1.4099999999999999,"15min":1.944},"last":1.04},{"interface":"front","average":{"1min":1.115,"5min":1.0940000000000001,"15min":1.075},"min":{"1min":0.81100000000000005,"5min":0.77300000000000002,"15min":0.58899999999999997},"max":{"1min":1.46,"5min":1.53,"15min":1.5780000000000001},"last":1.115}]},{"osd":7,"last update":"Fri Sep 13 16:36:44 2024","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.095,"15min":1.0720000000000001},"min":{"1min":0.86699999999999999,"5min":0.76300000000000001,"15min":0.61699999999999999},"max":{"1min":1.3759999999999999,"5min":1.6639999999999999,"15min":1.6639999999999999},"last":1.018},{"interface":"front","average":{"1min":1.042,"5min":1.073,"15min":1.071},"min":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.65400000000000003},"max":{"1min":1.266,"5min":1.381,"15min":2.0139999999999998},"last":1.0860000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-13T16:37:30.307 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-13T16:37:30.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 Processor -- start 2024-09-13T16:37:30.419 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- start start 2024-09-13T16:37:30.420 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb40ce0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcfb415dbe0 0x7fcfb40cd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfb40cd4d0 con 0x7fcfb40b3260 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfb4071be0 con 0x7fcfb415dbe0 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4159af0 con 0x7fcfb40add60 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcfb415dbe0 0x7fcfb40cd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fcfb415dbe0 0x7fcfb40cd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42204/0 (socket says 172.21.15.119:42204) 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 -- 172.21.15.119:0/612645679 learned_addr learned my addr 172.21.15.119:0/612645679 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb259c700 1 --2- 172.21.15.119:0/612645679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb40ce0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 -- 172.21.15.119:0/612645679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 msgr2=0x7fcfb40ce0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 --2- 172.21.15.119:0/612645679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb40ce0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 -- 172.21.15.119:0/612645679 >> v1:172.21.15.145:6789/0 conn(0x7fcfb40b3260 legacy=0x7fcfb406fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.421 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 -- 172.21.15.119:0/612645679 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb415e130 con 0x7fcfb415dbe0 2024-09-13T16:37:30.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb1d9b700 1 --2- 172.21.15.119:0/612645679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40cd960 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009910 tx=0x7fcfa8009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e03536f38a843501 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb159a700 1 -- 172.21.15.119:0/612645679 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa800ba60 con 0x7fcfb415dbe0 2024-09-13T16:37:30.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb159a700 1 -- 172.21.15.119:0/612645679 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa800bbc0 con 0x7fcfb415dbe0 2024-09-13T16:37:30.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb159a700 1 -- 172.21.15.119:0/612645679 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8009e60 con 0x7fcfb415dbe0 2024-09-13T16:37:30.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- 172.21.15.119:0/612645679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 msgr2=0x7fcfb40cd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.422 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 --2- 172.21.15.119:0/612645679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40cd960 secure :-1 s=READY pgs=2065 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8009910 tx=0x7fcfa8009c20 comp rx=0 tx=0).stop 2024-09-13T16:37:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- 172.21.15.119:0/612645679 shutdown_connections 2024-09-13T16:37:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 --2- 172.21.15.119:0/612645679 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40cd960 unknown :-1 s=CLOSED pgs=2065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 --2- 172.21.15.119:0/612645679 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb40ce0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- 172.21.15.119:0/612645679 >> 172.21.15.119:0/612645679 conn(0x7fcfb40b0980 msgr2=0x7fcfb40b0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- 172.21.15.119:0/612645679 shutdown_connections 2024-09-13T16:37:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.416+0000 7fcfb359e700 1 -- 172.21.15.119:0/612645679 wait complete. 2024-09-13T16:37:30.423 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 Processor -- start 2024-09-13T16:37:30.424 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 -- start start 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb41f6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfb40b3260 0x7fcfb40ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb41f3100 con 0x7fcfb415dbe0 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4142c30 con 0x7fcfb40b3260 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4142f10 con 0x7fcfb40add60 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb2d9d700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40ace00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42210/0 (socket says 172.21.15.119:42210) 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb2d9d700 1 -- 172.21.15.119:0/2493183985 learned_addr learned my addr 172.21.15.119:0/2493183985 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:30.425 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb1d9b700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfb40b3260 0x7fcfb40ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.426 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb259c700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb41f6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb1d9b700 1 -- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 msgr2=0x7fcfb41f6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb1d9b700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb41f6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb1d9b700 1 -- 172.21.15.119:0/2493183985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 msgr2=0x7fcfb40ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb1d9b700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb1d9b700 1 -- 172.21.15.119:0/2493183985 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8009570 con 0x7fcfb40b3260 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb1d9b700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfb40b3260 0x7fcfb40ac8c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8016000 tx=0x7fcfa8004770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8004510 con 0x7fcfb40b3260 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb41431f0 con 0x7fcfb40b3260 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb41f6680 con 0x7fcfb40b3260 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfa801e070 con 0x7fcfb40b3260 2024-09-13T16:37:30.427 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.420+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa8003900 con 0x7fcfb40b3260 2024-09-13T16:37:30.428 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.424+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf98005310 con 0x7fcfb40b3260 2024-09-13T16:37:30.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.424+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7fcfa8003a60 con 0x7fcfb40b3260 2024-09-13T16:37:30.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.424+0000 7fcfa2ffd700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf94066760 0x7fcf94068c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.429 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.424+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7fcfa8020070 con 0x7fcfb40b3260 2024-09-13T16:37:30.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.424+0000 7fcfb259c700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf94066760 0x7fcf94068c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.430 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.428+0000 7fcfb259c700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf94066760 0x7fcf94068c20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4006b40 tx=0x7fcfa4008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.438 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.436+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7fcfa8052d10 con 0x7fcfb40b3260 2024-09-13T16:37:30.608 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.604+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7fcf98005890 con 0x7fcfb40b3260 2024-09-13T16:37:30.618 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.612+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fcfa8085030 con 0x7fcfb40b3260 2024-09-13T16:37:30.619 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfa2ffd700 1 -- 172.21.15.119:0/2493183985 <== mon.1 v2:172.21.15.145:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v24)=0 v24) v1 ==== 123+0+0 (secure 0 0 0) 0x7fcfa802a0c0 con 0x7fcfb40b3260 2024-09-13T16:37:30.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf94066760 msgr2=0x7fcf94068c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf94066760 0x7fcf94068c20 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fcfa4006b40 tx=0x7fcfa4008040 comp rx=0 tx=0).stop 2024-09-13T16:37:30.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfb40b3260 msgr2=0x7fcfb40ac8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfb40b3260 0x7fcfb40ac8c0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fcfa8016000 tx=0x7fcfa8004770 comp rx=0 tx=0).stop 2024-09-13T16:37:30.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 shutdown_connections 2024-09-13T16:37:30.622 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7fcf94066760 0x7fcf94068c20 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcfb415dbe0 0x7fcfb40ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7fcfb40b3260 0x7fcfb40ac8c0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 --2- 172.21.15.119:0/2493183985 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7fcfb40add60 0x7fcfb41f6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 >> 172.21.15.119:0/2493183985 conn(0x7fcfb40b0980 msgr2=0x7fcfb40b0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:30.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 shutdown_connections 2024-09-13T16:37:30.623 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.616+0000 7fcfb359e700 1 -- 172.21.15.119:0/2493183985 wait complete. 2024-09-13T16:37:30.636 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-09-13T16:37:30.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-09-13T16:37:30.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8bfff700 1 Processor -- start 2024-09-13T16:37:30.753 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8bfff700 1 -- start start 2024-09-13T16:37:30.754 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8bfff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd8c0b2200 0x7efd8c0b2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8bfff700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8c0705d0 con 0x7efd8c0cd8d0 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8bfff700 1 -- --> v1:172.21.15.145:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd8c06fca0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8bfff700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd8c0cbf10 con 0x7efd8c0b2200 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8b7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd8c0b2200 0x7efd8c0b2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8b7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efd8c0b2200 0x7efd8c0b2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42226/0 (socket says 172.21.15.119:42226) 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.748+0000 7efd8b7fe700 1 -- 172.21.15.119:0/912888529 learned_addr learned my addr 172.21.15.119:0/912888529 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8b7fe700 1 -- 172.21.15.119:0/912888529 >> v1:172.21.15.145:6789/0 conn(0x7efd8c0b17f0 legacy=0x7efd8c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8b7fe700 1 -- 172.21.15.119:0/912888529 >> v1:172.21.15.119:6789/0 conn(0x7efd8c0cd8d0 legacy=0x7efd8c06cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8b7fe700 1 -- 172.21.15.119:0/912888529 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c06ff80 con 0x7efd8c0b2200 2024-09-13T16:37:30.755 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8b7fe700 1 --2- 172.21.15.119:0/912888529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b2200 0x7efd8c0b2670 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7efd84009910 tx=0x7efd84009c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6327ec099754c533 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd89ffb700 1 -- 172.21.15.119:0/912888529 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd8400ba60 con 0x7efd8c0b2200 2024-09-13T16:37:30.756 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd89ffb700 1 -- 172.21.15.119:0/912888529 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7efd8400bbc0 con 0x7efd8c0b2200 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd89ffb700 1 -- 172.21.15.119:0/912888529 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd84009e60 con 0x7efd8c0b2200 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 -- 172.21.15.119:0/912888529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b2200 msgr2=0x7efd8c0b2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 --2- 172.21.15.119:0/912888529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b2200 0x7efd8c0b2670 secure :-1 s=READY pgs=2067 cs=0 l=1 rev1=1 crypto rx=0x7efd84009910 tx=0x7efd84009c20 comp rx=0 tx=0).stop 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 -- 172.21.15.119:0/912888529 shutdown_connections 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 --2- 172.21.15.119:0/912888529 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b2200 0x7efd8c0b2670 unknown :-1 s=CLOSED pgs=2067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 -- 172.21.15.119:0/912888529 >> 172.21.15.119:0/912888529 conn(0x7efd8c06a350 msgr2=0x7efd8c06a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 -- 172.21.15.119:0/912888529 shutdown_connections 2024-09-13T16:37:30.757 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 -- 172.21.15.119:0/912888529 wait complete. 2024-09-13T16:37:30.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 Processor -- start 2024-09-13T16:37:30.758 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.752+0000 7efd8bfff700 1 -- start start 2024-09-13T16:37:30.759 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b17f0 0x7efd8c0aaf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 --2- >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd8c0b2200 0x7efd8c15aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 --2- >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd8c0cd8d0 0x7efd8c0a8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c0a9bc0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 -- --> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c15cc70 con 0x7efd8c0b2200 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 -- --> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c15d4d0 con 0x7efd8c0cd8d0 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b17f0 0x7efd8c0aaf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b17f0 0x7efd8c0aaf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:42240/0 (socket says 172.21.15.119:42240) 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 -- 172.21.15.119:0/958566209 learned_addr learned my addr 172.21.15.119:0/958566209 (peer_addr_for_me v2:172.21.15.119:0/0) 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8b7fe700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd8c0cd8d0 0x7efd8c0a8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.760 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8a7fc700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd8c0b2200 0x7efd8c15aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.761 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 -- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd8c0cd8d0 msgr2=0x7efd8c0a8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd8c0cd8d0 0x7efd8c0a8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 -- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd8c0b2200 msgr2=0x7efd8c15aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd8c0b2200 0x7efd8c15aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 -- 172.21.15.119:0/958566209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84009570 con 0x7efd8c0b17f0 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8affd700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b17f0 0x7efd8c0aaf90 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7efd8000c500 tx=0x7efd8000c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd737fe700 1 -- 172.21.15.119:0/958566209 <== mon.0 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd800052d0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd737fe700 1 -- 172.21.15.119:0/958566209 <== mon.0 v2:172.21.15.119:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7efd80005430 con 0x7efd8c0b17f0 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8c0ab4d0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd737fe700 1 -- 172.21.15.119:0/958566209 <== mon.0 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd800055a0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.762 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.756+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8c15ce80 con 0x7efd8c0b17f0 2024-09-13T16:37:30.763 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.760+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8c0b2670 con 0x7efd8c0b17f0 2024-09-13T16:37:30.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.760+0000 7efd737fe700 1 -- 172.21.15.119:0/958566209 <== mon.0 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 5) v1 ==== 86092+0+0 (secure 0 0 0) 0x7efd80003900 con 0x7efd8c0b17f0 2024-09-13T16:37:30.764 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.760+0000 7efd737fe700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd78066730 0x7efd78068bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-13T16:37:30.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.760+0000 7efd737fe700 1 -- 172.21.15.119:0/958566209 <== mon.0 v2:172.21.15.119:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5435+0+0 (secure 0 0 0) 0x7efd800833e0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.760+0000 7efd8a7fc700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd78066730 0x7efd78068bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-13T16:37:30.765 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.760+0000 7efd8a7fc700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd78066730 0x7efd78068bf0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0ad990 tx=0x7efd7c009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-13T16:37:30.771 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.768+0000 7efd737fe700 1 -- 172.21.15.119:0/958566209 <== mon.0 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175519 (secure 0 0 0) 0x7efd800507d0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.978 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.972+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd8c070c00 con 0x7efd8c0b17f0 2024-09-13T16:37:30.979 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd737fe700 1 -- 172.21.15.119:0/958566209 <== mon.0 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+3534 (secure 0 0 0) 0x7efd80017de0 con 0x7efd8c0b17f0 2024-09-13T16:37:30.979 INFO:teuthology.orchestra.run.smithi119.stdout: 2024-09-13T16:37:30.979 INFO:teuthology.orchestra.run.smithi119.stdout:{"epoch":107,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":8039,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/707361651","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":707361651},{"type":"v1","addr":"172.21.15.119:6833","nonce":707361651}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8045,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/18043130","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":18043130},{"type":"v1","addr":"172.21.15.119:6835","nonce":18043130}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":8048,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6837/1397099563","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6836","nonce":1397099563},{"type":"v1","addr":"172.21.15.145:6837","nonce":1397099563}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":87},{"gid":9859,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.145:6835/2045201471","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6834","nonce":2045201471},{"type":"v1","addr":"172.21.15.145:6835","nonce":2045201471}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":107}],"filesystems":[]} 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd78066730 msgr2=0x7efd78068bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd78066730 0x7efd78068bf0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7efd8c0ad990 tx=0x7efd7c009040 comp rx=0 tx=0).stop 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b17f0 msgr2=0x7efd8c0aaf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b17f0 0x7efd8c0aaf90 secure :-1 s=READY pgs=2068 cs=0 l=1 rev1=1 crypto rx=0x7efd8000c500 tx=0x7efd8000c810 comp rx=0 tx=0).stop 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 shutdown_connections 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:6832/2154272207,v1:172.21.15.145:6833/2154272207] conn(0x7efd78066730 0x7efd78068bf0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3301/0,v1:172.21.15.145:6790/0] conn(0x7efd8c0cd8d0 0x7efd8c0a8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.982 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] conn(0x7efd8c0b2200 0x7efd8c15aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 --2- 172.21.15.119:0/958566209 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0b17f0 0x7efd8c0aaf90 unknown :-1 s=CLOSED pgs=2068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-13T16:37:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 >> 172.21.15.119:0/958566209 conn(0x7efd8c06a350 msgr2=0x7efd8c163710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-13T16:37:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 shutdown_connections 2024-09-13T16:37:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:2024-09-13T16:37:30.976+0000 7efd8bfff700 1 -- 172.21.15.119:0/958566209 wait complete. 2024-09-13T16:37:30.983 INFO:teuthology.orchestra.run.smithi119.stderr:dumped fsmap epoch 107 2024-09-13T16:37:30.994 DEBUG:tasks.cephfs.filesystem:already dead... 2024-09-13T16:37:30.994 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-13T16:37:30.994 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-09-13T16:37:30.995 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:31.008 INFO:tasks.ceph.mds.a:Stopped 2024-09-13T16:37:31.008 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-09-13T16:37:31.008 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:31.070 INFO:tasks.ceph.mds.c:Stopped 2024-09-13T16:37:31.070 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-09-13T16:37:31.070 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:31.085 INFO:tasks.ceph.mds.b:Stopped 2024-09-13T16:37:31.085 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-09-13T16:37:31.085 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:31.099 INFO:tasks.ceph.mds.d:Stopped 2024-09-13T16:37:31.099 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-13T16:37:31.100 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-13T16:37:31.100 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:31.195 INFO:tasks.ceph.osd.0:Stopped 2024-09-13T16:37:31.195 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-13T16:37:31.195 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:31.290 INFO:tasks.ceph.osd.1:Stopped 2024-09-13T16:37:31.290 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-13T16:37:31.290 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:33.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 is failed for ~0s 2024-09-13T16:37:37.392 INFO:tasks.ceph.osd.2:Stopped 2024-09-13T16:37:37.393 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-13T16:37:37.393 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:39.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-09-13T16:37:39.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.2 has been restored 2024-09-13T16:37:43.495 INFO:tasks.ceph.osd.3:Stopped 2024-09-13T16:37:43.496 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-13T16:37:43.496 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:43.583 INFO:tasks.ceph.osd.4:Stopped 2024-09-13T16:37:43.583 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-13T16:37:43.583 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:45.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-09-13T16:37:45.364 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-09-13T16:37:49.685 INFO:tasks.ceph.osd.5:Stopped 2024-09-13T16:37:49.685 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-13T16:37:49.685 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:49.771 INFO:tasks.ceph.osd.6:Stopped 2024-09-13T16:37:49.771 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-13T16:37:49.772 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:49.871 INFO:tasks.ceph.osd.7:Stopped 2024-09-13T16:37:49.871 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-13T16:37:49.872 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-13T16:37:49.872 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:49.904 INFO:tasks.ceph.mgr.y:Stopped 2024-09-13T16:37:49.904 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-13T16:37:49.905 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:49.937 INFO:tasks.ceph.mgr.x:Stopped 2024-09-13T16:37:49.938 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-13T16:37:49.938 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-13T16:37:49.938 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:49.986 INFO:tasks.ceph.mon.a:Stopped 2024-09-13T16:37:49.986 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-13T16:37:49.986 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:50.024 INFO:tasks.ceph.mon.b:Stopped 2024-09-13T16:37:50.025 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-13T16:37:50.025 INFO:teuthology.orchestra.run:waiting for 300 2024-09-13T16:37:50.118 INFO:tasks.ceph.mon.c:Stopped 2024-09-13T16:37:50.118 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-13T16:37:50.118 DEBUG:teuthology.orchestra.run.smithi119:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'online, but wants' | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'inode wrongly marked free' | egrep -v 'bad backtrace on inode' | egrep -v 'inode table repaired for inode' | egrep -v 'Scrub error on inode' | egrep -v 'Scrub error on dir' | egrep -v 'Metadata damage detected' | egrep -v 'object missing on disk' | egrep -v 'Invalid tag char' | head -n 1 2024-09-13T16:37:50.143 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:37:50.143 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-13T16:37:50.338 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:37:50.338 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-13T16:37:50.365 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-09-13T16:37:50.567 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:37:50.567 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-13T16:37:50.757 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:37:50.757 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-13T16:37:50.967 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:37:50.967 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-13T16:37:51.248 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:37:51.248 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-13T16:37:51.427 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:37:51.427 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-13T16:37:51.615 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:37:51.615 DEBUG:teuthology.orchestra.run.smithi145:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-13T16:37:51.786 INFO:tasks.ceph:Archiving mon data... 2024-09-13T16:37:51.787 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/data/mon.a.tgz 2024-09-13T16:37:51.788 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2024-09-13T16:37:51.794 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.IGEr8Z0urI 2024-09-13T16:37:51.795 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IGEr8Z0urI 2024-09-13T16:37:51.945 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.IGEr8Z0urI 2024-09-13T16:37:51.968 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.IGEr8Z0urI is 441KB 2024-09-13T16:37:52.033 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.IGEr8Z0urI 2024-09-13T16:37:52.040 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/data/mon.b.tgz 2024-09-13T16:37:52.040 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-09-13T16:37:52.046 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.Lis7LNR8RU 2024-09-13T16:37:52.047 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.Lis7LNR8RU 2024-09-13T16:37:52.156 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.Lis7LNR8RU 2024-09-13T16:37:52.179 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.Lis7LNR8RU is 461KB 2024-09-13T16:37:52.241 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.Lis7LNR8RU 2024-09-13T16:37:52.247 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/data/mon.c.tgz 2024-09-13T16:37:52.248 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-09-13T16:37:52.254 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.f9dmrhaocX 2024-09-13T16:37:52.254 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.f9dmrhaocX 2024-09-13T16:37:52.365 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.f9dmrhaocX 2024-09-13T16:37:52.389 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.f9dmrhaocX is 461KB 2024-09-13T16:37:52.408 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.f9dmrhaocX 2024-09-13T16:37:52.413 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-13T16:37:52.413 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T16:37:52.416 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-13T16:37:52.470 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-13T16:37:52.470 DEBUG:teuthology.orchestra.run.smithi119:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T16:37:52.473 DEBUG:teuthology.orchestra.run.smithi145:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-13T16:37:52.495 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T16:37:52.506 INFO:teuthology.orchestra.run.smithi145.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-13T16:37:52.512 INFO:tasks.ceph:Archiving crash dumps... 2024-09-13T16:37:52.514 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/remote/smithi119/crash 2024-09-13T16:37:52.515 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T16:37:52.531 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/crash to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/remote/smithi145/crash 2024-09-13T16:37:52.532 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-13T16:37:52.547 INFO:tasks.ceph:Shutting down logrotate 2024-09-13T16:37:52.547 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:37:52.550 DEBUG:teuthology.orchestra.run.smithi145:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-09-13T16:37:52.636 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T16:37:52.652 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-09-13T16:37:52.667 INFO:tasks.ceph:Compressing logs... 2024-09-13T16:37:52.668 DEBUG:teuthology.orchestra.run.smithi119:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:37:52.670 DEBUG:teuthology.orchestra.run.smithi145:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:37:59.334 INFO:tasks.ceph:Archiving logs... 2024-09-13T16:37:59.335 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/remote/smithi119/log 2024-09-13T16:37:59.336 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T16:38:00.823 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/remote/smithi145/log 2024-09-13T16:38:00.824 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-13T16:38:01.677 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-13T16:38:01.686 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-13T16:38:01.686 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T16:38:01.689 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-13T16:38:01.777 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-13T16:38:01.786 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-13T16:38:01.786 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:38:01.789 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-13T16:38:01.812 INFO:teuthology.orchestra.run.smithi119.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T16:38:01.813 INFO:teuthology.orchestra.run.smithi119.stdout:============================================================================== 2024-09-13T16:38:01.813 INFO:teuthology.orchestra.run.smithi119.stdout:*hv01.front.sepi 67.205.162.81 3 u 11 64 377 0.071 -0.544 0.269 2024-09-13T16:38:01.813 INFO:teuthology.orchestra.run.smithi119.stdout:+hv02.front.sepi 63.231.80.2 3 u 6 64 377 0.093 -0.407 0.293 2024-09-13T16:38:01.813 INFO:teuthology.orchestra.run.smithi119.stdout:+hv03.front.sepi 74.6.168.72 3 u 6 64 377 0.088 -0.334 0.280 2024-09-13T16:38:01.813 INFO:teuthology.orchestra.run.smithi119.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-13T16:38:01.814 INFO:teuthology.orchestra.run.smithi145.stdout: remote refid st t when poll reach delay offset jitter 2024-09-13T16:38:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:============================================================================== 2024-09-13T16:38:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:+hv01.front.sepi 67.205.162.81 3 u 9 64 377 0.071 -0.176 0.147 2024-09-13T16:38:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:*hv02.front.sepi 63.231.80.2 3 u 13 64 377 0.081 -0.040 0.092 2024-09-13T16:38:01.814 INFO:teuthology.orchestra.run.smithi145.stdout:+hv03.front.sepi 74.6.168.72 3 u 11 64 377 0.089 0.267 0.163 2024-09-13T16:38:01.814 INFO:teuthology.orchestra.run.smithi145.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 0.000 0.000 2024-09-13T16:38:01.814 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-13T16:38:01.839 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-13T16:38:01.839 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-13T16:38:01.850 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-13T16:38:01.862 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-13T16:38:01.872 INFO:teuthology.task.internal:Duration was 1333.377890 seconds 2024-09-13T16:38:01.873 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-13T16:38:01.883 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-13T16:38:01.883 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T16:38:01.886 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-13T16:38:01.938 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-13T16:38:01.938 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2024-09-13T16:38:01.938 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T16:38:01.949 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-09-13T16:38:01.949 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-13T16:38:01.961 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-13T16:38:01.961 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:38:01.963 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-13T16:38:02.005 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-13T16:38:02.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T16:38:02.048 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-13T16:38:02.144 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-13T16:38:02.154 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-13T16:38:02.154 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T16:38:02.170 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-13T16:38:02.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-13T16:38:02.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:38:02.202 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:38:02.214 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2024-09-13T16:38:02.217 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-09-13T16:38:02.230 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:38:02.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:38:02.280 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-13T16:38:02.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-13T16:38:02.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-13T16:38:02.304 INFO:teuthology.task.internal:Transferring archived files... 2024-09-13T16:38:02.305 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/remote/smithi119 2024-09-13T16:38:02.305 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T16:38:02.335 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-09-13_14:36:41-fs-wip-rishabh-testing-20240910.095512-reef-distro-default-smithi/7903804/remote/smithi145 2024-09-13T16:38:02.335 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-13T16:38:02.361 INFO:teuthology.task.internal:Removing archive directory... 2024-09-13T16:38:02.361 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T16:38:02.363 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-13T16:38:02.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-13T16:38:02.380 INFO:teuthology.task.internal:Not uploading archives. 2024-09-13T16:38:02.380 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-13T16:38:02.455 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-13T16:38:02.455 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T16:38:02.457 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-13T16:38:02.464 INFO:teuthology.orchestra.run.smithi119.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 16:38 /home/ubuntu/cephtest 2024-09-13T16:38:02.465 INFO:teuthology.orchestra.run.smithi145.stdout: 924112 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 13 16:38 /home/ubuntu/cephtest 2024-09-13T16:38:02.467 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-13T16:38:02.479 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-13T16:38:02.512 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{ubuntu_20.04} mount/fuse objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v2 tasks/forward-scrub} duration: 1333.3778896331787 flavor: default owner: scheduled_rishabh@teuthology success: true 2024-09-13T16:38:02.512 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-13T16:38:02.599 INFO:teuthology.run:pass